[00:14:27] (03CR) 10Sahil505: "- I believe there is slight readability issues with the content inside the popup with large height bars. It would be great if can have sli" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423904 (https://phabricator.wikimedia.org/T188277) (owner: 10Fdans) [00:16:30] 10Analytics, 10Analytics-Wikistats: Beta: Y-axis units and rounding issues - https://phabricator.wikimedia.org/T187429#4128763 (10sahil505) @mforns @Nuria - I feel like parts of this task are repeated in various other tasks. Please see T184011 & T188277. Thanks. [01:29:35] 10Analytics-Kanban, 10Patch-For-Review: Refresh SWAP notebook hardware - https://phabricator.wikimedia.org/T183145#4128850 (10Neil_P._Quinn_WMF) >>! In T183145#4127859, @Ottomata wrote: > I had the same errors you guys saw with impyla. After parsing a few of those tickets you linked to, this invocation seems... [07:12:25] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10MediaWiki-extensions-Translate, and 2 others: Unable to mark pages for translation in Meta - https://phabricator.wikimedia.org/T192107#4129090 (10Pginer-WMF) [07:35:02] (03CR) 10Hoo man: [C: 031] Track number of constraint definition snaks over time (032 comments) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/425872 (https://phabricator.wikimedia.org/T191891) (owner: 10Ladsgroup) [07:46:28] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: [EL sanitization] Ensure presence of EL YAML whitelist in analytics1003 - https://phabricator.wikimedia.org/T189691#4050215 (10elukey) ``` elukey@analytics1003:~$ ls -l /etc/analytics/sanitization/eventlogging_p... [07:51:03] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: [EL sanitization] Modify mysql purging script to read from the new YAML whitelist - https://phabricator.wikimedia.org/T189692#4050230 (10elukey) ``` elukey@db1108:~$ sudo -u eventlogcleaner crontab -l 0 11 * * * /usr/bin/flock --verbose -n /var/lock/eventl... [07:53:05] mforns: o/ - yaml purging script deployed everywhere! [07:53:17] also the yaml whitelist is on analytics1003 [07:53:35] let me know if anything is wrong/missing (just updated all the tasks) [08:40:33] nuria_: hello [08:40:57] nuria_: FYI I've checked the geomapping of Opera Mini's XCIP, they're all correct (singapore -> singapore, amsterdam -> amsterdam, US -> US) [09:03:57] (03PS1) 10Addshore: RC: track max edit rate of fastest user [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/425978 [09:06:39] (03CR) 10Hoo man: [C: 031] RC: track max edit rate of fastest user [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/425978 (owner: 10Addshore) [09:06:44] (03CR) 10Addshore: [C: 032] RC: track max edit rate of fastest user [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/425978 (owner: 10Addshore) [09:06:50] (03PS1) 10Addshore: RC: track max edit rate of fastest user [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/425979 [09:06:53] (03Merged) 10jenkins-bot: RC: track max edit rate of fastest user [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/425978 (owner: 10Addshore) [09:06:55] (03CR) 10Addshore: [C: 032] RC: track max edit rate of fastest user [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/425979 (owner: 10Addshore) [09:07:03] (03Merged) 10jenkins-bot: RC: track max edit rate of fastest user [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/425979 (owner: 10Addshore) [10:04:31] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Refresh zookeeper nodes in eqiad - https://phabricator.wikimedia.org/T182924#4129301 (10elukey) >>! In T182924#4126533, @elukey wrote: > > But then zk1-1's state is not, since it keeps repeating the following: > > ``` > 2018-04-12 10:07:55,347 - INFO... [10:09:14] elukey: about output of the bash script, do you want to pipe the output to a log in /var/log? [10:10:23] fdans: hola! No that was a generic question, not sure if it was in the requirements or not. It might be good to inspect how things are going [10:14:20] fdans: bare in mind that what I wrote in there is not a diktat but rather a discussion about how to proceed etc.. [10:14:37] as all code reviews should be :( [10:14:40] err :) [10:20:16] elukey: yeah, thanks for the detailed CR by the way [10:21:02] elukey: the task put emphasis on notifying analytics alerts, but nothing about logging it locally [10:26:09] (03PS1) 10Addshore: Add script for wikibase docker pull stats [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/425996 [10:30:23] (03PS2) 10Addshore: Add script for wikibase docker pull stats [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/425996 [10:31:07] (03PS3) 10Addshore: Add script for wikibase docker pull stats [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/425996 [10:34:14] * elukey lunch + errand! brb in ~2h [10:41:24] (03CR) 10Addshore: [C: 032] Add script for wikibase docker pull stats [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/425996 (owner: 10Addshore) [10:41:29] (03PS1) 10Addshore: Add script for wikibase docker pull stats [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/426000 [10:41:34] (03CR) 10Addshore: [C: 032] Add script for wikibase docker pull stats [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/426000 (owner: 10Addshore) [10:41:36] (03Merged) 10jenkins-bot: Add script for wikibase docker pull stats [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/425996 (owner: 10Addshore) [10:41:41] (03Merged) 10jenkins-bot: Add script for wikibase docker pull stats [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/426000 (owner: 10Addshore) [12:47:07] * joal is super angry at spark ( ..•˘___˘• .. ) [12:49:26] nooooo [12:49:29] how come? [12:49:35] you and spark are friends! [12:49:40] * elukey runs away [12:50:08] elukey: I've spent again a full day since yesterday trying to understand why what was succeeding in test last week now fails in prod - without success [12:50:15] * joal is super angry [12:51:59] 10Analytics, 10Services (watching): Update to latest kafkacat - https://phabricator.wikimedia.org/T182163#4129563 (10faidon) @ottomata pinged me last week about that, I guess I hadn't seen this task or forgot about it entirely, sorry about that! I just built a stretch backport for kafkacat 1.3.1, built agains... [13:16:16] * mforns steps away from joal [13:16:21] :] [13:16:30] hey joal :] [13:16:50] if you want to change subject, you can help me with an oozie error I can not fix... :D [13:17:15] Hi mforns - I'll never even think to approach you - I know too much you are a Jujistsu and philipino-fencing master ;) [13:17:20] 10Analytics, 10Analytics-Kanban, 10Wikipedia-iOS-App-Backlog: Decrease the request from iOS app to bohrium - https://phabricator.wikimedia.org/T190566#4129597 (10elukey) @chelsyx so now the infrastructure that runs the bohrium host (and hence piwik) is much more stable, we hope to have solved the issues that... [13:17:35] xD [13:18:21] mforns: Will need to drop soon to catch the kids - After standup? [13:18:34] ok, no problem, will try here [13:18:39] joal, ^ [13:22:01] joal: I'd roll restart all the jvm things on druid100*, anything against it? [13:22:28] nope elukey - Please go ahead [13:23:26] (03CR) 10Lucas Werkmeister (WMDE): "Don’t you need to add the script to the cron file as well?" (034 comments) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/425872 (https://phabricator.wikimedia.org/T191891) (owner: 10Ladsgroup) [13:35:06] starting with druid100[456] [13:38:50] druid1004 done (depooled, restarted, re-pooled) [13:48:06] aaand druid1005/6 done [13:49:46] this is so cool : https://stats.wikimedia.org/v2/#/it.wikipedia.org/reading/top-viewed-articles [13:49:53] with the links to the pages <# [13:49:54] <3 [13:52:34] proceeding with the druid100[123] cluster then [13:53:10] so, druid1001 is the overlord manager [13:53:23] so I disabled the middle manager on druid1002 to start [13:53:37] now, I am going to roll restart all the zookeeper daemons on all the hosts [13:53:46] then I'll proceed with all the druid ones on 1002 [13:53:48] then 1003 [13:53:51] and finally 1001 [13:54:07] I am going to check of course superset/pivot configs to avoid user breakages [13:56:00] ok zookeeper restarted [13:56:44] ah the overlord manager shifted to druid1003 [13:57:04] so in theory now we should be able to see druid1003 to drain from banner activity [13:57:12] without fireworks [14:40:13] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Upgrading Wikistats 2.0 footer UI/design - https://phabricator.wikimedia.org/T191672#4129767 (10mforns) @sahil505 Looks good to me overall! I think we can now dive into detail and start implementation. I have a couple more specific comments: - In my pri... [14:58:20] druid1001's middle manager is disabled [15:14:19] (03CR) 10Ladsgroup: Track number of constraint definition snaks over time (036 comments) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/425872 (https://phabricator.wikimedia.org/T191891) (owner: 10Ladsgroup) [15:14:31] (03PS2) 10Ladsgroup: Track number of constraint definition snaks over time [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/425872 (https://phabricator.wikimedia.org/T191891) [15:15:56] ema: nice [15:16:06] ema: did you restaured the trusted proxy list? [15:23:02] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10New-Readers, and 4 others: Proxies information gone from Zero portal - https://phabricator.wikimedia.org/T187014#4129858 (10Nuria) Could we restore proxies now that the nice opera folks gave us their list? Clearly we also need to look into why/h... [15:24:55] druid1001's daemons just restarted [15:24:57] 10Analytics, 10Analytics-Kanban, 10Wikipedia-iOS-App-Backlog: Decrease the request from iOS app to bohrium - https://phabricator.wikimedia.org/T190566#4129873 (10Nuria) Per our latest conversation a wider dispatch interval will not change the amount of events sent, thus on our end we are in the same place, w... [15:25:21] now I have disabled druid1003's middle manager and then in ~1h I'll restart the last daemons in there (when it will be drained [15:26:04] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10New-Readers, and 4 others: Proxies information gone from Zero portal - https://phabricator.wikimedia.org/T187014#4129878 (10BBlack) Yeah, ema and I discussed this after the meeting the other day. I'm not sure whether or how we can look into th... [15:27:51] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10New-Readers, and 4 others: Proxies information gone from Zero portal - https://phabricator.wikimedia.org/T187014#4129884 (10Nuria) >we're planning to just stop pulling that empty data from them, and replace it with a private file that's puppet-ma... [15:29:28] nuria_: not yet, but I see that you've seen the ticket updates :) [15:29:47] ema: yayaya got it , seems like we will have a better system going forward [15:31:01] 10Analytics, 10Analytics-Wikistats: Beta: Y-axis units and rounding issues - https://phabricator.wikimedia.org/T187429#4129888 (10Nuria) Seems like only last point needs addressing and that is a duplicate of other tickets so this task can probably be closed as a duplicate. [15:35:51] 10Analytics, 10ChangeProp, 10EventBus, 10RESTBase-API, 10Services (doing): Re-render wiktionary definitions on user purges - https://phabricator.wikimedia.org/T192157#4129917 (10mobrovac) p:05Triage>03High [15:36:12] 10Analytics, 10ChangeProp, 10EventBus, 10RESTBase-API, 10Services (doing): Re-render wiktionary definitions on user purges - https://phabricator.wikimedia.org/T192157#4129929 (10mobrovac) [15:38:58] elukey: do you have a second in the batcave? [15:39:35] sure [15:40:49] fdans: I am in bc [15:42:51] milimetric: fyi https://phabricator.wikimedia.org/T189283#4128960, let us know when data is being generated and we can turn the rsyncs on [16:00:28] screen --help [16:00:33] oops :) [16:02:02] ping mforns standduppp [16:14:23] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Check detached accounts in DB with same username for "mediawiki" and "phab" sources but different uuid's (and merge if connected) - https://phabricator.wikimedia.org/T170091#4129973 (10Aklapper) [16:14:25] 10Analytics-Tech-community-metrics, 10DevRel-February-2016, 10DevRel-March-2016, 10Developer-Relations (Jan-Mar-2017): Investigate why there is a mismatch between six names and certain email address in mediawiki-identities data - https://phabricator.wikimedia.org/T123643#4129974 (10Aklapper) [16:14:29] 10Analytics-Tech-community-metrics, 10Developer-Relations (Apr-Jun-2018): Provide Hatstall to fix syncing / updating of identities on the wikimedia.biterg.io production instance - https://phabricator.wikimedia.org/T157898#4129969 (10Aklapper) 05Open>03Resolved Since today Bitergia allows us to access a bet... [16:16:43] 10Analytics-Tech-community-metrics, 10Developer-Relations (Oct-Dec 2017): Fix duplicated enrollments in database - https://phabricator.wikimedia.org/T176786#4129978 (10Aklapper) [16:16:47] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Check detached accounts in DB with same username for "mediawiki" and "phab" sources but different uuid's (and merge if connected) - https://phabricator.wikimedia.org/T170091#4129980 (10Aklapper) [16:16:50] 10Analytics-Tech-community-metrics, 10DevRel-February-2016, 10DevRel-March-2016, 10Developer-Relations (Jan-Mar-2017): Investigate why there is a mismatch between six names and certain email address in mediawiki-identities data - https://phabricator.wikimedia.org/T123643#4129981 (10Aklapper) [16:16:52] 10Analytics-Tech-community-metrics, 10Developer-Relations (Apr-Jun-2018): Provide Hatstall to fix syncing / updating of identities on the wikimedia.biterg.io production instance - https://phabricator.wikimedia.org/T157898#4129977 (10Aklapper) [16:52:59] joal, :S can you enlighten me with oozie? [16:53:25] mforns: hm - oozie enlightment may not be on my skillset - But I can have a look :) [16:53:26] it probably will take you 19 seconds [16:53:29] Batcave !!!!! [16:53:32] k :] [17:02:45] nuria_: holaaa 1 on 1? [17:03:24] fdans: yes onmw one sec [17:06:37] * elukey off! [17:18:52] 10Analytics, 10Analytics-Kanban, 10Wikipedia-iOS-App-Backlog: Decrease the request from iOS app to bohrium - https://phabricator.wikimedia.org/T190566#4130127 (10chelsyx) Thanks @elukey for the information about bohrium infrastructure! As @Nuria said, a wider dispatch interval won't solve the problem. I thin... [18:15:10] (03PS4) 10Mforns: Add job and query for page previews aggregation [analytics/refinery] - 10https://gerrit.wikimedia.org/r/425281 (https://phabricator.wikimedia.org/T186728) [18:26:09] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10New-Readers, and 4 others: Proxies information gone from Zero portal - https://phabricator.wikimedia.org/T187014#4130381 (10atgo) @Nuria @BBlack thanks for all the work on this! Once it's resolved, will the data for the time window that this was... [18:27:54] hello analytics! I have a licensing question. If someone were to share a screenshot of the Pageviews Analysis tool, ( https://tools.wmflabs.org/pageviews ), are they bound the REST API licenses described at https://wikimedia.org/api/rest_v1/ ? [18:28:02] I assume so but wanted to make sure [18:34:11] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10New-Readers, and 4 others: Proxies information gone from Zero portal - https://phabricator.wikimedia.org/T187014#4130422 (10Nuria) @atgo, It cannot be, we no longer have the original Ips of the records that are wrongly labeled. [18:34:54] (03CR) 10Mforns: "OK! It seems to work now. Tested it outputting on mforns.virtualpageviews_hourly. Ready for CR." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/425281 (https://phabricator.wikimedia.org/T186728) (owner: 10Mforns) [18:36:21] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10New-Readers, and 4 others: Proxies information gone from Zero portal - https://phabricator.wikimedia.org/T187014#4130431 (10atgo) Ok, thanks @nuria [18:48:07] (03CR) 10Nuria: [C: 031] Add job and query for page previews aggregation (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/425281 (https://phabricator.wikimedia.org/T186728) (owner: 10Mforns) [18:50:34] 10Analytics-Kanban, 10Analytics-Wikistats: friendly HTML titles for Wikistats 2.0 - https://phabricator.wikimedia.org/T182718#4130470 (10Nuria) [18:51:35] 10Analytics-Kanban, 10Analytics-Wikistats: friendly HTML titles for Wikistats 2.0 - https://phabricator.wikimedia.org/T182718#3832561 (10Nuria) SEO will not see updated titles if done via javascript, but regardless better titles are beneficial from a Ux perspective. [19:15:30] 10Analytics: Wikistats2 and SEO - https://phabricator.wikimedia.org/T192172#4130550 (10Nuria) [19:26:58] python [19:57:15] (03CR) 10Mforns: Add job and query for page previews aggregation (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/425281 (https://phabricator.wikimedia.org/T186728) (owner: 10Mforns) [21:45:43] 10Analytics: Wikistats2 and SEO - https://phabricator.wikimedia.org/T192172#4130961 (10mforns) p:05Triage>03Normal [21:45:49] 10Analytics: Wikistats2 and SEO - https://phabricator.wikimedia.org/T192172#4130550 (10mforns) p:05Normal>03Triage