[07:16:51] Hi team - I have an electricity outage this morning - Will launch a job before the cut, and will be back before lunh normally [07:24:16] morning joal! [07:24:18] ack [07:24:43] morning elukey :) [07:52:31] shuting down everything before the cut - later [09:40:33] 10Analytics: Status of analytics/limn-*-data git repositories? - https://phabricator.wikimedia.org/T221064 (10hashar) [10:10:14] 10Analytics, 10Analytics-EventLogging, 10QuickSurveys, 10Readers-Web-Backlog (Tracking): QuickSurveys EventLogging missing ~10% of interactions - https://phabricator.wikimedia.org/T220627 (10ovasileva) [10:59:33] 10Analytics, 10DC-Ops, 10Operations, 10decommission, 10ops-eqiad: Decommission analytics1003 - https://phabricator.wikimedia.org/T206524 (10elukey) [10:59:43] 10Analytics, 10Operations, 10decommission, 10ops-eqiad: Decommission analytics100[1,2] - https://phabricator.wikimedia.org/T205507 (10elukey) [11:01:57] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Move away from jmxtrans in favor of prometheus jmx_exporter - https://phabricator.wikimedia.org/T175344 (10elukey) 05Open→03Resolved [11:48:26] elukey: can you take a quick look at https://gerrit.wikimedia.org/r/c/operations/puppet/+/504303 would like to role out puppet5 and facter3 to some of your test machines [11:49:21] jbond42: feel free to go! [11:50:17] great thanks will probably do all the roles in analytics_test_cluster, you happy for me to just push them as and when or do you want me to ping you for each one? [11:50:35] jbond42: nono you can go ahead with testing cluster, no need to ping me [11:50:49] elukey: great thanks [12:05:27] \o/ ! I'm back :) [12:07:49] :) [12:07:59] * elukey lunch! [13:17:39] 10Analytics: Status of analytics/limn-*-data git repositories? - https://phabricator.wikimedia.org/T221064 (10Milimetric) hashar: these actually don't have anything to do with limn, they all hold configuration for reportupdater reports. We would rename them but that's not possible in gerrit, right? We could st... [13:25:28] Awesome explanation of manifolds here: https://bastian.rieck.me/blog/posts/2019/manifold/ [13:52:13] (03PS6) 10Mforns: Make saltrotate store salts with timestamps as file name. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/484250 (https://phabricator.wikimedia.org/T212014) [13:59:50] (03PS7) 10Mforns: Make saltrotate store salts with timestamps as file name. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/484250 (https://phabricator.wikimedia.org/T212014) [14:20:07] hello people [14:20:16] I am going to gently roll restart the druid clusters [14:20:23] k elukey - watching [14:20:25] to pick up openjdk updates [14:21:15] elukey: order would be overlord, historical, broker, coordinator? [14:22:33] joal: I am starting with the historicals (all of them, one at the time) [14:22:40] ack! [14:28:05] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: cergen: exceptions trying to add alt_name - https://phabricator.wikimedia.org/T220591 (10ema) 05Open→03Resolved @Ottomata thanks! The new error message is helpful, and the proposed solution works. [14:36:08] (03CR) 10Milimetric: [C: 03+2] "one question, but more something I'm wondering about. Code looks good." (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/497604 (https://phabricator.wikimedia.org/T218463) (owner: 10Joal) [14:40:05] (03CR) 10Milimetric: Fix for null-timestamps in checker (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/499914 (owner: 10Joal) [14:41:36] (03Merged) 10jenkins-bot: Update mw user-history timestamps [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/497604 (https://phabricator.wikimedia.org/T218463) (owner: 10Joal) [14:44:43] (03CR) 10Milimetric: Update mediawiki-history user bot fields (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/504025 (https://phabricator.wikimedia.org/T219177) (owner: 10Joal) [14:46:05] (03CR) 10Joal: Update mw user-history timestamps (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/497604 (https://phabricator.wikimedia.org/T218463) (owner: 10Joal) [14:47:35] (03Merged) 10jenkins-bot: Correct names in mediawiki-history sql package [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/498861 (owner: 10Joal) [14:47:58] Ah! It got merged :) [14:50:32] joal: any preference of the name of the user that should replace hdfs? [14:50:57] not really :) - analytics-runner is ok, except for being long when sudoing :) [14:51:16] elukey: --^ [14:52:27] even if the name doesn't convince me yet [14:52:34] (03Merged) 10jenkins-bot: Fix mediawiki-history-checker after field renamed [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/499527 (https://phabricator.wikimedia.org/T219484) (owner: 10Joal) [14:52:48] anyway, I tried to think about the procedure to swap the users [14:52:52] and it is a ton of work :D [14:53:02] I can imagine elukey :S [14:53:23] (03CR) 10Milimetric: [C: 04-1] "My only blocking request is that we use "name" instead of "name_regex"" (034 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/504025 (https://phabricator.wikimedia.org/T219177) (owner: 10Joal) [14:54:40] (03PS3) 10Joal: Fix null-timestamps in checker [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/499914 [14:54:49] milimetric: fixed --^ [14:54:59] milimetric: thanks a lot for all the reviews :) [14:55:08] (03CR) 10Milimetric: [C: 03+2] Fix null-timestamps in checker [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/499914 (owner: 10Joal) [14:55:53] joal: of course [14:58:59] druid clusters restarted [15:00:24] +1 to using reportupdater-queries, marcel [15:00:34] oops mforns ^ [15:00:55] sure, or we can change the name! [15:03:01] ottomata: o/ going to roll restart kafka on jumbo, good for you? [15:03:24] (03Merged) 10jenkins-bot: Fix null-timestamps in checker [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/499914 (owner: 10Joal) [15:04:47] (03CR) 10Joal: Update mediawiki-history user bot fields (034 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/504025 (https://phabricator.wikimedia.org/T219177) (owner: 10Joal) [15:05:25] (03PS2) 10Joal: Update mediawiki-history user bot fields [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/504025 (https://phabricator.wikimedia.org/T219177) [15:06:11] very interesting [15:06:12] https://grafana.wikimedia.org/d/000000027/kafka?panelId=54&fullscreen&orgId=1&from=1555289826908&to=1555427150198 [15:07:10] let's see what happens with a preferred replica election [15:07:23] elukey: ok! [15:19:42] elukey, is there a way to hdfs dfs -put with the rsync --delete functionality? [15:20:07] updating a whole directory with the contents of a given source directory? [15:20:28] even if this requires to delete some files that ceased to exist in the source dir? [15:23:13] elukey, or else, can I do a regular rsync and use the cdh::hadoop::mount::mount_point? [15:23:16] mforns: not that I know of [15:23:30] nope, the mount point is read only [15:23:33] what is your use case? [15:23:52] I have this directory containing salt files, and I want to sync it to hdfs [15:24:00] the source dir is in sn-coord1001 [15:24:06] *an-coord1001 [15:24:22] and it should go to /user/hdfs/eventlogging-salts [15:24:28] mforns: there is no 'rsync' for hdfs, its just joal's custom python code [15:24:40] i dton' think he has --delete support in there [15:25:01] maybe a hdfs -rm && rsync is the way to go? [15:26:15] a bit brutal but it should work [15:26:43] ottomata, yes, is there a possibility that the sanitization jobs are launched while the salt files are deleted and waiting to restored? [15:27:32] sanitization job runs every hour, and salt rotation happens every midnight [15:28:03] I can delay the sanitization by 1-2 minutes [15:30:40] do you need to delete the old ones from the dir? [15:30:46] right away? [15:32:11] ottomata, saltrotate will determine when to delete the expired salt files and delete them in the local folder [15:32:22] at that point, salt files should be deleted from hdfs as well [15:32:23] ah ya, hm. so maybe [15:33:14] hdfs dfs -put local_dir /tmp/dir.XXX; hdfs dfs -rm $curr_dir; hdfs dfs -mv /tmp/dir.XXX $curr_dir? [15:33:29] same problem i guess [15:35:33] ottomata, I think delete salt dir and replace it with new one will take seconds. if we just delay the sanitization job by 1-2 mins we should be good no? [15:36:13] ya likely! [15:36:43] ok, will try that and let you guys review [16:01:59] ping milimetric standdduppp [16:07:06] (03PS3) 10Joal: Update mediawiki-history user bot fields [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/504025 (https://phabricator.wikimedia.org/T219177) [16:07:54] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): EventGate service runner worker occasionally killed, usually during higher load - https://phabricator.wikimedia.org/T220661 (10Ottomata) [16:48:58] * elukey ofF! [16:50:36] 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics, 10Patch-For-Review: "Edit" equivalent of pageviews daily available to use in Turnilo and Superset - https://phabricator.wikimedia.org/T211173 (10MNeisler) Thanks @mforns! And sorry for the delay. I'm reviewing the edits_hourly... [17:11:34] 10Analytics: Decide: start_timestamp for mediawiki history - https://phabricator.wikimedia.org/T220507 (10Nuria) Just my usual plug to ask to please document in wikitech decision taken. [17:18:32] 10Analytics: Improve mediawiki-user-history bot-by-name regex - https://phabricator.wikimedia.org/T221114 (10JAllemandou) [17:21:41] (03PS1) 10Joal: Fix mediawiki-user-history writing filter [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/504382 [17:21:48] milimetric: the said fix [17:21:50] --^ [18:31:41] 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics, 10Patch-For-Review: "Edit" equivalent of pageviews daily available to use in Turnilo and Superset - https://phabricator.wikimedia.org/T211173 (10mforns) Thanks @MNeisler, Cool! Glad that you guys liked it. Yes, I left user_tenur... [19:36:36] (03PS1) 10Milimetric: Fix problem with breakdown [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/504417 [20:22:55] ottomata: how can i know what bastion can i use to ssh? Could not resolve hostname bast2001.wikimedia.org? (maybe there was an announcement about this that I totally missed) [20:23:02] joal: Hello! A quick question your Wikidata JSON dump copies in hdfs, may I? [20:23:24] nuria ya you missed an announcement [20:23:26] nuria: There was an announcement on bast2001, I think today. [20:23:28] try 2002 [20:23:32] ottomata: where was it? [20:24:11] not sure what you are asking GoranSM1 , the dumps are in hdfs for the last import under /joals user dir [20:24:13] hm i see it in operations mailing list [20:24:32] ottomata: yes in the operations mailing list [20:24:48] but also [20:24:48] https://wikitech.wikimedia.org/wiki/Bastion [20:24:50] looks up to date [20:25:00] ottomata: ah yess [20:25:32] nuria: Thanks, I am using them already. do you happen to know if these dumps - since joal says that they are not production ready yet - are complete? [20:26:50] (03CR) 10Nuria: Fix mediawiki-user-history writing filter (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/504382 (owner: 10Joal) [20:27:21] GoranSM1: they are complete but not updated regularly, they will not be for a while [20:27:50] nuria: Thank you! [20:58:03] (03PS1) 10Milimetric: [WIP] Change projection of world map to eckert 3 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/504429 (https://phabricator.wikimedia.org/T218045) [21:14:24] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): EventGate service runner worker occasionally killed, usually during higher load - https://phabricator.wikimedia.org/T220661 (10Ottomata) Been doing a lot to get more data, including enabling node profiling and connect... [23:07:54] 10Analytics, 10Education-Program-Dashboard, 10Event Metrics: I want WikiMetrics integration with the education dashboard that lets you easily pull reports about courses, institutions, etc. - https://phabricator.wikimedia.org/T92454 (10jmatazzoni)