[07:22:43] 10Analytics-Tech-community-metrics, 10Bugzilla-Migration, 10Phabricator, 10DevRel-November-2015: Closed tickets in Bugzilla migrated without closing event? - https://phabricator.wikimedia.org/T107254#4005100 (10greg) >>! In T107254#4000910, @Aklapper wrote: > For the records, the [[ https://phabricator.wik... [08:02:52] Hi team [08:06:00] 10Analytics, 10Operations, 10User-Elukey: Import some Analytics git puppet submodules to operations/puppet - https://phabricator.wikimedia.org/T188377#4005170 (10elukey) [08:06:12] hey joal [08:07:25] elukey: I know you know :) [08:07:36] elukey: ealy christmas for cluster lovers ! [08:12:00] joal: what happened?? [08:12:37] elukey: brand new NODES :) [08:12:47] ahhhh!!! [08:12:49] yes :) [08:13:29] joal: we discussed with Andrew a test to do, namely installing debian stretch on them [08:13:41] using the current cdh jessie packages [08:13:53] elukey: super cool [08:14:06] elukey: I recall this was the plan [08:14:09] I am still wondering if it makes sense [08:14:17] in theory those packages should be compatible [08:14:24] especially now that we are on java 8 [08:14:36] elukey: if everything works fine with stretch, we can just upgrade the old nodes after [08:14:49] hm - I have no idea about package compatibility though [08:17:45] the alternative is being conservative and wait for cdh's official support for stretch [08:21:09] 10Analytics, 10Analytics-Kanban: Provide active editor metric for "all-projects"? - https://phabricator.wikimedia.org/T188194#4005190 (10Neil_P._Quinn_WMF) @fdans, what makes you say this is invalid? As understand it, this is about aggregate numbers for, say, all Wikipedias or all Wikivoyages in Wikistats 2.0,... [08:47:37] 10Analytics: Provide edit tags in the Data Lake edit data - https://phabricator.wikimedia.org/T161149#4005213 (10Neil_P._Quinn_WMF) [08:52:37] 10Analytics-Kanban, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410#4005226 (10Neil_P._Quinn_WMF) >>! In T172410#4001345, @elukey wrote: > @mpopov @chelsyx @Neil_P._Quinn_WMF - Hi! An... [09:17:30] 10Analytics-Kanban, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410#4005257 (10elukey) >>! In T172410#4005226, @Neil_P._Quinn_WMF wrote: > Could we hold off on this change until edit... [09:20:54] 10Analytics-Kanban, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410#4005262 (10Neil_P._Quinn_WMF) >>! In T172410#4005257, @elukey wrote: > Of course, we are currently not rushing any... [09:51:21] 10Analytics-Kanban, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410#4005280 (10jcrespo) > we will need to know which wikis will be on which hosts Most likely, you will have to connec... [10:41:23] 10Analytics, 10Analytics-Kanban: Provide active editor metric for "all-projects"? - https://phabricator.wikimedia.org/T188194#4005410 (10fdans) @Neil_P._Quinn_WMF it was marked as invalida because two tasks were open with exactly the same content. We're tracking this in T188265 [10:51:34] all right it seems that deployment-eventlogging05 is working as expected with systemd/stretch [10:52:23] going to ask a review ottomata this afternoon :) [10:52:28] *from [10:52:47] also, I was wondering the best way to migrate eventlog1001 to eventlog1002 [10:53:06] without loosing events etc.. [10:53:37] and in theory (I might be wrong) we could just spin up eventlog1002, and then wait for all the daemons to join their consumer groups [10:53:58] at any given point in time there will be only one active daemon for a specific partition [10:54:19] so simply turning off eventlog1001 (after 1002 is up) should do the trick [10:54:28] (that would be really awesome) [11:20:40] 10Analytics, 10Analytics-Kanban: Provide active editor metric for "all-projects"? - https://phabricator.wikimedia.org/T188194#4005505 (10Neil_P._Quinn_WMF) >>! In T188194#4005410, @fdans wrote: > @Neil_P._Quinn_WMF it was marked as invalida because two tasks were open with exactly the same content. > > We're... [11:21:04] 10Analytics, 10Analytics-Kanban: Provide active editor metric for "all-projects"? - https://phabricator.wikimedia.org/T188194#4005507 (10Neil_P._Quinn_WMF) [11:21:06] 10Analytics, 10Analytics-Wikistats: Active Editors metric for all projects - https://phabricator.wikimedia.org/T188265#4005510 (10Neil_P._Quinn_WMF) [11:59:03] going to be afk for ~1h to commute to the co-working (I left my car in there due to the snow, going to take the train :) [12:41:21] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): One account (in "gerrit_top_developers" widget) counted as two accounts (in "gerrit_main_numbers" widget) - https://phabricator.wikimedia.org/T184741#4005705 (10Qgil) p:05Normal>03High [13:05:51] 10Analytics-Tech-community-metrics, 10Bugzilla-Migration, 10Phabricator, 10DevRel-November-2015: Closed tickets in Bugzilla migrated without closing event? - https://phabricator.wikimedia.org/T107254#4005767 (10Aklapper) >>! In T107254#4005100, @greg wrote: > Now we have a good place for the data, do we ha... [13:47:04] 10Analytics, 10EventBus, 10Services (next): EventBus rejecting events because of malformed characters in the comment - https://phabricator.wikimedia.org/T184698#4005889 (10Pchelolo) Found another one in the logs today: ``` { "rev_content_changed": true, "performer": { "user_is_bot": false, "user_... [14:23:58] mooorniiing ateam [14:25:47] o/ [14:28:10] hello [14:35:22] elukey: i'm doing emails now, wanna do upload in a bit :O [14:35:24] ? [14:42:40] ottomata: sure! [14:43:38] ottomata: I was checking how to bundle new deps for Burrow 1.0 but they moved from GoDeps to .toml files, that require 'dep' afacs [14:43:41] *afaics [14:43:54] ha, i don't even know what .toml is! [14:44:20] it seems a more flexible way to specify go pkg deps [14:44:40] you can put pkg name + tag, etc. [14:44:46] there is also https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=888673 [14:45:06] he mentions the fact that burrow might get into debian soonish [14:45:34] we'd need to move to Burrow 1.0 sooner I think since 0.1.0 is not (of course) compatible with Jumbo [14:45:46] and we'll lose EL lag monitoring [14:46:34] * elukey coffee [14:49:40] aye [14:56:18] 10Analytics, 10Operations, 10User-Elukey: Import some Analytics git puppet submodules to operations/puppet - https://phabricator.wikimedia.org/T188377#4006202 (10Ottomata) All for jmxtrans, varnishkafka, and kafkatee. Might want to keep zookeeper as a submodule. Just because there aren't many contributions... [14:58:08] 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install analytics107[0-7] - https://phabricator.wikimedia.org/T188294#4006219 (10Ottomata) > We still haven't tested Hadoop packages on stretch We kinda have, just not services. stat1005 is a Stretch Hadoop clien... [15:01:45] 10Analytics, 10Operations, 10User-Elukey: Import some Analytics git puppet submodules to operations/puppet - https://phabricator.wikimedia.org/T188377#4006234 (10elukey) >>! In T188377#4006202, @Ottomata wrote: > All for jmxtrans, varnishkafka, and kafkatee. Might want to keep zookeeper as a submodule. Jus... [15:04:07] 10Analytics, 10Operations, 10User-Elukey: Import some Analytics git puppet submodules to operations/puppet - https://phabricator.wikimedia.org/T188377#4006258 (10Ottomata) stars: https://github.com/wikimedia/puppet-zookeeper/stargazers watchers: https://github.com/wikimedia/puppet-zookeeper/watchers forks: h... [15:04:40] elukey: prefer this to erik's patch: https://gerrit.wikimedia.org/r/#/c/415015/ whatchu think? [15:04:43] 10Analytics, 10EventBus, 10MediaWiki-General-or-Unknown, 10Services (watching): It should be possible to understand the reason of revision creation from RevisionRecordInserted hook - https://phabricator.wikimedia.org/T188396#4006259 (10Pchelolo) p:05Triage>03Normal [15:05:34] (03PS8) 10Mforns: [WIP] Add EL and whitelist sanitization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/412939 (https://phabricator.wikimedia.org/T181064) [15:05:45] 10Analytics, 10EventBus, 10MediaWiki-General-or-Unknown, 10Services (watching): It should be possible to understand the reason of revision creation from RevisionRecordInserted hook - https://phabricator.wikimedia.org/T188396#4006259 (10Pchelolo) p:05Normal>03Triage [15:09:44] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Move webrequest varnishkafka and consumers to Kafka jumbo cluster. - https://phabricator.wikimedia.org/T185136#4006294 (10Ottomata) [15:09:55] elukey: ready when you are: ^ [15:14:02] ottomata: I am :) let me warn traffic people first [15:14:08] ah you already did [15:14:10] <3 [15:14:14] bcave? [15:14:41] sure! [15:22:39] (03PS9) 10Mforns: [WIP] Add EL and whitelist sanitization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/412939 (https://phabricator.wikimedia.org/T181064) [15:37:50] Semantic UI made me found my new favourite Wikipedia article [15:37:51] https://en.wikipedia.org/wiki/Defenestration [15:37:56] cc milimetric [15:39:55] heh, I had to spell defenestration for my spelling bee in 6th grade [15:40:19] of course you were in a spelling bee [15:44:32] elukey, ottomata - I'm off, but thinking of you and kafka :) [15:47:43] o/ [15:50:15] :) [15:54:13] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Move webrequest varnishkafka and consumers to Kafka jumbo cluster. - https://phabricator.wikimedia.org/T185136#4006418 (10Ottomata) [15:56:11] vk is now pushing webrequest_upload to jumbo! [16:21:03] (03CR) 10Ottomata: "Reply to Joal: Yah, especially since we are going to have to change the puppet cron to run this stuff. I'll look into making a patch to " [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/413633 (https://phabricator.wikimedia.org/T186602) (owner: 10Ottomata) [16:29:41] (03PS15) 10Milimetric: Refactor sqoop jobs and add from_timestamp [analytics/refinery] - 10https://gerrit.wikimedia.org/r/408848 (https://phabricator.wikimedia.org/T184759) [16:31:31] joal: nuria makes a very good point that we should merge this before the end of the month to make the next sqoop safer ^. That means tomorrow. If we can't, it's ok, I'll just watch the jobs [16:32:40] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Move webrequest varnishkafka and consumers to Kafka jumbo cluster. - https://phabricator.wikimedia.org/T185136#4006541 (10Ottomata) @Jeff_Green, I think we are ready for webrequest_text. Can we find a time to do this together Tuesday March 6? [16:44:26] (03PS1) 10Addshore: Use Output::outputMessage in BF counts.php [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/415040 [16:44:38] (03PS1) 10Addshore: Use Output::outputMessage in BF counts.php [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/415041 [16:44:47] (03CR) 10Addshore: [C: 032] Use Output::outputMessage in BF counts.php [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/415041 (owner: 10Addshore) [16:44:49] (03CR) 10Addshore: [C: 032] Use Output::outputMessage in BF counts.php [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/415040 (owner: 10Addshore) [16:44:55] (03Merged) 10jenkins-bot: Use Output::outputMessage in BF counts.php [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/415041 (owner: 10Addshore) [16:44:58] (03Merged) 10jenkins-bot: Use Output::outputMessage in BF counts.php [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/415040 (owner: 10Addshore) [17:02:29] (03CR) 10Ottomata: Add RefineMonitor job (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/413633 (https://phabricator.wikimedia.org/T186602) (owner: 10Ottomata) [17:03:58] 10Analytics-Kanban, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410#4006697 (10Nuria) @Neil_P._Quinn_WMF : " I don't do cross-wiki joins, but I do aggregate data from all wikis into a... [17:18:27] 10Analytics-Kanban, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410#4006749 (10Addshore) >>! In T172410#4006697, @Nuria wrote: > @Neil_P._Quinn_WMF : " I don't do cross-wiki joins, bu... [17:19:01] 10Analytics, 10Analytics-Cluster: Migrate mjoiner Kafka clients to use Kafka jumbo - https://phabricator.wikimedia.org/T188408#4006750 (10Ottomata) p:05Triage>03Normal [17:24:37] (03PS1) 10Ladsgroup: Fix entity usage script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/415048 [17:25:01] (03CR) 10Ladsgroup: [C: 032] Fix entity usage script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/415048 (owner: 10Ladsgroup) [17:25:10] (03Merged) 10jenkins-bot: Fix entity usage script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/415048 (owner: 10Ladsgroup) [17:28:37] (03PS1) 10Ladsgroup: Fix entity usage script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/415050 [17:29:01] (03Abandoned) 10Ladsgroup: Fix entity usage script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/415050 (owner: 10Ladsgroup) [17:33:48] (03PS1) 10Ladsgroup: Fix entity usage script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/415052 [17:34:08] (03CR) 10Ladsgroup: [C: 032] Fix entity usage script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/415052 (owner: 10Ladsgroup) [17:34:16] (03Merged) 10jenkins-bot: Fix entity usage script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/415052 (owner: 10Ladsgroup) [17:48:06] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Add the prometheus jmx agent to AQS Cassandra - https://phabricator.wikimedia.org/T184795#4006912 (10elukey) Sad news: we had to rollback due to an issue with the Cassandra 2.2.x startup script: https://issues.apache.org/jira/browse/CASSANDRA-7254 http... [17:51:39] aaand of course cassandra 2.2 startup script doesn't work with the jmx agent joal :( --^ [17:52:26] * elukey off! [17:54:47] 10Analytics-Kanban, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410#4006937 (10chelsyx) >>! In T172410#4005226, @Neil_P._Quinn_WMF wrote: >>>! In T172410#4003480, @chelsyx wrote: >> N... [18:09:26] 10Analytics, 10Analytics-Cluster: Migrate mjoiner Kafka clients to use Kafka jumbo - https://phabricator.wikimedia.org/T188408#4007014 (10mforns) p:05Normal>03High [18:09:37] me testing [18:10:03] 10Analytics, 10Analytics-Cluster: Migrate mjoiner Kafka clients to use Kafka jumbo - https://phabricator.wikimedia.org/T188408#4006750 (10mforns) p:05High>03Normal [18:11:16] 10Analytics-Kanban, 10EventBus, 10Services (done): Investigate why disk usage on Kafka nodes is 2 times lower in codfw - https://phabricator.wikimedia.org/T187554#4007039 (10Pchelolo) 05Open>03Invalid I've checked some of the topics for the approximate date-time for the messages at the beginning of the t... [18:14:58] 10Analytics: Piwiki for new wikimedia foundation site - https://phabricator.wikimedia.org/T188419#4007049 (10Nuria) [18:22:07] 10Analytics: Test - https://phabricator.wikimedia.org/T188420#4007066 (10mforns) [18:22:27] 10Analytics: Test - https://phabricator.wikimedia.org/T188420#4007066 (10mforns) p:05Triage>03Low [18:25:38] 10Analytics: Piwiki for new wikimedia foundation site - https://phabricator.wikimedia.org/T188419#4007086 (10Nuria) I *think* traffic is about 150k pageviews per day last month: https://tinyurl.com/ydcgapbx Seems quite a bit for piwik actually. [18:27:38] 10Analytics: Piwik to measure traffic for new wikimedia foundation site - https://phabricator.wikimedia.org/T188419#4007119 (10Nuria) [18:29:30] 10Analytics-Kanban, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410#4007145 (10Ottomata) > So I sqoop them to Hadoop and use Spark to speed up. I LIKE YOUR STYYYYYYLE! [19:04:45] 10Analytics-Kanban, 10Cloud-VPS, 10EventBus, 10Patch-For-Review, 10Services (watching): Add page-related topics to EventStreams - https://phabricator.wikimedia.org/T187241#3968945 (10Ottomata) [19:12:22] !log updating spark2-* CLIs to spark 2.2.1: T185581 [19:12:24] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:12:25] T185581: Upgrade spark2 .deb to spark 2.2.1 - https://phabricator.wikimedia.org/T185581 [19:12:55] 10Analytics-Kanban, 10Cloud-VPS, 10EventBus, 10Patch-For-Review, 10Services (watching): Add page-related topics to EventStreams - https://phabricator.wikimedia.org/T187241#4007353 (10Ottomata) https://stream.wikimedia.org/?doc They are live :) [19:23:46] 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install analytics107[0-7] - https://phabricator.wikimedia.org/T188294#4007407 (10Cmjohnson) [19:25:11] 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install analytics107[0-7] - https://phabricator.wikimedia.org/T188294#4002756 (10Cmjohnson) a:03RobH @robh handing this off to you to finish installs and turnover to Team Analytics. [19:25:28] milimetric: i stand corrected , the most popular page for wikimedia foundation site is the privacy policy, terms of use and home [19:25:35] (03PS9) 10Ottomata: Add RefineMonitor job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/413633 (https://phabricator.wikimedia.org/T186602) [19:36:40] nuria_: yeah, so other sites link to it, then it's kind of unpredictablew [19:39:02] fdans: ok, i am moving to my next task, do you want me to help with the topic explorer work? (fine to say no) [21:21:18] 10Analytics, 10Cloud-VPS, 10EventBus, 10Services: Set up a Cloud VPS Kafka Cluster with replicated eventbus production data - https://phabricator.wikimedia.org/T187225#4007952 (10Pchelolo) [21:21:22] 10Analytics-Kanban, 10Cloud-VPS, 10EventBus, 10Services (done): Add page-related topics to EventStreams - https://phabricator.wikimedia.org/T187241#4007948 (10Pchelolo) 05Open>03Resolved a:03Pchelolo One more smallish bug with CSP headers was discovered, so I've deployed again. Resolving. [22:24:09] (03PS2) 10Nuria: [WIP] Changing initialization of QTree to work arround precision Bug [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/408329 [22:25:42] (03PS1) 10Ottomata: [WIP] Use properties instead of scopt to read Refine parameters [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/415174 [22:27:13] (03CR) 10Ottomata: "Super messy right now. I think I will move the case classes back into their respective job classes, but this works! It auto converts hyp" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/415174 (owner: 10Ottomata) [22:32:35] nuria_: that task is pretty simple, each question in the topic explorer should use the key name of the metric instead of inferring it by converting the title to kebab case [22:33:23] go ahead if you want to work on that, it's unrelated to what I'm doing right now so I wasn't planning to touch it soon [22:34:05] fdans: ok, i will do that after I am done with this bug i need to fix in one of our jobs [22:34:16] fdans: where you able to test the localization patch? [22:34:50] nuria_: not yet, will do that tomorrow morning [22:36:18] (03PS3) 10Nuria: [WIP] Changing initialization of QTree to work arround precision Bug [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/408329 [22:39:14] fdans: ok [22:43:38] ottomata: all of a sudden spark shell not working ? [22:43:43] ottomata: i was doing: [22:43:45] spark2-shell --jars /srv/deployment/analytics/refinery/artifacts/refinery-job-spark-2.1.jar,/srv/deployment/analytics/refinery/artifacts/refinery-core.jar --master yarn [22:52:31] 10Analytics, 10Analytics-EventLogging, 10Cognate, 10Collaboration-Team-Triage, and 14 others: Possible PHP7 compatibility issues in WMF-deployed extensions - https://phabricator.wikimedia.org/T173850#4008482 (10Jdforrester-WMF) [23:05:09] (03PS10) 10Mforns: Add EL and whitelist sanitization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/412939 (https://phabricator.wikimedia.org/T181064) [23:24:44] 10Analytics, 10EventBus, 10Services, 10Availability (Multiple-active-datacenters), 10User-mobrovac: WANObjectCache relay daemon or mcrouter support - https://phabricator.wikimedia.org/T97562#4008612 (10aaron) >>! In T97562#3977706, @Joe wrote: > I'm reopening this since the status of the FLOSS mcrouter p... [23:27:43] bye teaaam [23:28:59] 10Analytics, 10EventBus, 10Services, 10Availability (Multiple-active-datacenters), 10User-mobrovac: WANObjectCache relay daemon or mcrouter support - https://phabricator.wikimedia.org/T97562#4008640 (10aaron) For reference, there is T156938 , for evaluating dynomite.