[01:17:13] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Fundraising-Backlog, and 3 others: Fix EventLogging schemas that use array for items type - https://phabricator.wikimedia.org/T218617 (10Samwilson) I've made [[https://meta.wikimedia.org/w/index.php?title=Schema%3ATemplateWizard&type=revision&dif... [07:06:32] Good morning [07:06:57] o/ [07:07:25] elukey: I wonder about deploying new aqs datasource :S [07:10:10] anything problematic? [07:10:33] elukey: change on data, therefore some big shifts in some metrics [07:10:39] big = -10% [07:11:02] (03PS2) 10Joal: Fix oozie banner monthly job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508358 (https://phabricator.wikimedia.org/T222603) [07:11:30] (03CR) 10Joal: Fix oozie banner monthly job (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508358 (https://phabricator.wikimedia.org/T222603) (owner: 10Joal) [07:11:46] do we have any data on how filters on recent changes and watchlists are being used? [07:13:28] groceryheist: I have no idea :( [07:14:41] joal: another parallel thing that I wanted to ask you is if I can allow the 'analytics' user to push data to druid from oozie [07:15:03] (in the daily coord I mean) [07:15:33] elukey: of course yes :) [07:15:51] elukey: I need to read how it's done (low-tech regexp IIRC) [07:16:28] yeah it is in the email [07:16:32] to internal@ [07:16:50] hm [07:17:14] I think we're not on the same page - Let's dig a bit [07:17:22] (03CR) 10Joal: Remove wikipedia-zero as program is over (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507933 (https://phabricator.wikimedia.org/T213770) (owner: 10Joal) [07:18:14] elukey: https://github.com/wikimedia/analytics-refinery/blob/master/oozie/util/druid/load/workflow.xml#L69 [07:18:32] joal: yes it is the bit that I added in the email [07:18:59] (I replied to Fatal Error - Oozie Job banner_activity-druid-daily-wf-2019-5-6 etc..) [07:19:07] Arf elukey - Didn't get "the email" thing - I'm gonna look at emails you sent ... MOAR COFFEE [07:19:15] ahhhh okok! [07:19:48] I'll add it to https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/508283/ [07:23:08] elukey: about AQS, let's wait for Dan to be here, and cross-vaidate :) [07:23:44] ack [07:26:41] (03PS2) 10Elukey: banner_activity: move oozie coordinators to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508283 (https://phabricator.wikimedia.org/T220971) [07:26:43] (03PS2) 10Elukey: browser-general-coord: move oozie job to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508307 (https://phabricator.wikimedia.org/T220971) [07:26:45] (03PS3) 10Elukey: Move oozie coords interlanguage/mediacounts to the new user analytics [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508314 (https://phabricator.wikimedia.org/T220971) [07:26:47] (03PS2) 10Elukey: Move the oozie's apis-coord to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508330 (https://phabricator.wikimedia.org/T220971) [07:27:07] done :) [07:27:44] elukey: this also means we need to deploy all druid jobs to analytics once that one is deployed :) [07:28:02] elukey: should be ok, but just triple checking [07:28:22] yep yep makes sense [07:28:36] I can add or hdfs or analytics for the moment [07:29:06] (as interim solution) [07:29:20] 10Analytics, 10Analytics-Kanban: Decide: start_timestamp for mediawiki history - https://phabricator.wikimedia.org/T220507 (10JAllemandou) [07:29:22] 10Analytics, 10Analytics-Kanban: Mediawiki-history release - Snapshot 2019-05 - https://phabricator.wikimedia.org/T221825 (10JAllemandou) [07:29:28] elukey: a bit cumbersome, but would help to test and all [07:30:57] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: 15.wikipedia.org missclassified as a pageview, same for query.wikidata.org - https://phabricator.wikimedia.org/T222460 (10JAllemandou) Another row: ` spark.sql("select uri_host, uri_path, uri_query from wmf.webrequest where webrequest_source = 'text' and y... [07:32:34] (03PS3) 10Elukey: banner_activity: move oozie coordinators to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508283 (https://phabricator.wikimedia.org/T220971) [07:32:36] (03PS3) 10Elukey: browser-general-coord: move oozie job to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508307 (https://phabricator.wikimedia.org/T220971) [07:32:36] joal: yeah I agree but it is probably the safer option [07:32:38] (03PS4) 10Elukey: Move oozie coords interlanguage/mediacounts to the new user analytics [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508314 (https://phabricator.wikimedia.org/T220971) [07:32:40] (03PS3) 10Elukey: Move the oozie's apis-coord to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508330 (https://phabricator.wikimedia.org/T220971) [07:33:31] (03CR) 10Joal: banner_activity: move oozie coordinators to the analytics user (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508283 (https://phabricator.wikimedia.org/T220971) (owner: 10Elukey) [07:36:11] (03CR) 10Joal: "Another question: Shouldn't we also add those changes to the prod list? And therefore double check the presence of databases in there as w" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507355 (https://phabricator.wikimedia.org/T220456) (owner: 10Fdans) [07:37:47] elukey: need to go for ~1h - will update the sqoop patch when back [07:38:10] joal: sure! sorry for the hassle :( [07:38:53] elukey: no problem! /me likes to learn :) [07:46:03] !log temporary override of oozie/util/druid/load/workflow.xml in HDFS's refinery to allow the analytics user to push data to druid from oozie [07:46:05] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:54:37] (03PS4) 10Elukey: banner_activity: move oozie coordinators to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508283 (https://phabricator.wikimedia.org/T220971) [07:54:39] (03PS4) 10Elukey: browser-general-coord: move oozie job to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508307 (https://phabricator.wikimedia.org/T220971) [07:54:41] (03PS5) 10Elukey: Move oozie coords interlanguage/mediacounts to the new user analytics [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508314 (https://phabricator.wikimedia.org/T220971) [07:54:43] (03PS4) 10Elukey: Move the oozie's apis-coord to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508330 (https://phabricator.wikimedia.org/T220971) [09:18:55] fdans, joal - shall we move the cassandra-daily-coord-local_group_default_T_unique_devices and monthly coords to the analytics user? [09:19:07] didn't get if they are testing ones or not [09:21:29] awight: o/ - qq: I can see that you are still running a oozie coordinator for ores, that keeps failing.. Is it meant to run or can we kill it? [09:21:34] elukey: Normally we run those through bundle - coordinators are here to facilitate stopping/running single instances since the bundle needs to be started on 1st of month [09:22:53] joal: ah so we are waiting to move them to the cassandra bundle? [09:23:22] elukey: I think they should have yes [09:23:55] elukey: Oh! Please do drive a wooden stake through that, apologies. [09:24:05] awight: thanks! np :) [09:24:23] fdans: weren't we supposed to restart the cassandra bundle after the uniques-job refator? [09:24:54] !log kill ores-revision-scores-public-coord via hue (not used anymore) [09:24:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:25:04] Thanks elukey anf awight :) [09:25:12] FWIW I'm still interested in Analytics volunteer work, in general or to push the "ML features in Hadoop" project forward, please ping me if anyone wants to coordinate. [09:25:36] I have 20% time at WMDE <3 [09:26:00] Hi awight - Thanks for that :) I'll try to get better at coordinating, and help :) [09:26:22] I could grab random tasks, but it might be better to focus... [09:31:13] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Change permissions for daily traffic anomaly reports on stat1007 - https://phabricator.wikimedia.org/T219546 (10elukey) @ssingh very happy to help! We should be ok to close the task right? I can see that @Jdcc-berkman is still an active user - shall I keep the... [09:32:38] joal: my next two targets for the analytics move are mediacounts-archive and clickstream.. afaics both write only to the archive dir, so in theory I'd just need to 1) kill 2) chown those files 3) restart with new user [09:33:08] Sounds good elukey :) [09:37:52] (03PS1) 10Elukey: Move the clickstream oozie coordinator to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508525 (https://phabricator.wikimedia.org/T220971) [09:38:34] !log kill clickstream-coord, chown /wmf/data/archive/clickstream to analytics:analytics, restart the job with the analytics user override [09:38:35] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:43:32] elukey: I confirm I have right to sudo as analytics :) [09:43:34] \o/ [09:44:41] \o/ [09:49:54] Now it's a matter of changing my habits elukey :) [09:50:05] :) [09:50:11] I am merging your patch for puppet [09:50:21] Plenty thanks elukey [09:50:47] elukey: is there an easy way to test it [09:50:48] ? [09:51:10] to test the puppet catalog diff or the script? [09:51:24] hm, possibly both? [09:51:47] the catalog diff is added to the code review, I ran pcc [09:51:59] executing the script is simply a matter of starting the unit [09:52:07] but it will trigger jobs etc.. no? [09:52:24] yeah - Ah - They should fail, they can't overright [09:52:33] Shall we do that? [09:52:48] just to confirm [09:52:48] elukey@an-coord1001:~$ systemctl list-timers | grep sqoop [09:52:48] Sat 2019-06-01 00:00:00 UTC 3 weeks 3 days left n/a n/a refinery-sqoop-whole-mediawiki.timer refinery-sqoop-whole-mediawiki.service [09:52:52] Sun 2019-06-02 00:00:00 UTC 3 weeks 4 days left Thu 2019-05-02 00:00:02 UTC 5 days ago refinery-sqoop-mediawiki-private.timer refinery-sqoop-mediawiki-private.service [09:52:56] these are the only two left [09:53:01] great :) [09:53:28] in this case, if you want to run the timer, you need to pick the .service unit [09:53:31] and do [09:53:38] sudo systemctl start refinery-sqoop-whole-mediawiki.service [09:53:46] sounds good [09:53:57] no need to sudo -u analytics since if you check in the unit there is User=bla.. [09:54:06] will triple check parameters to make sure it doesn't overright, and possibly try [09:54:14] systemctl cat refinery-sqoop-whole-mediawiki.service [09:54:27] (without sudo) [09:55:06] PROBLEM - Check the last execution of refinery-sqoop-mediawiki on an-coord1001 is CRITICAL: NRPE: Command check_check_refinery-sqoop-mediawiki_status not defined [09:56:47] ah yes makes sense [09:57:19] need to run puppet on the icinga server to clear those out [09:57:48] PROBLEM - Check the last execution of refinery-sqoop-mediawiki-production on an-coord1001 is CRITICAL: NRPE: Command check_check_refinery-sqoop-mediawiki-production_status not defined [09:58:00] yep yep we know icinga [09:58:54] elukey: no --force in the prod script - Asking for permission to manually launch the service for testn [09:59:21] you have the powa now, please go ahead :) [10:00:53] brb [10:01:21] elukey: I think we miss writing logs rights in an-coord1001 [10:20:21] joal: what is the error? (sorry just got back) [10:20:31] no error actually! [10:20:44] service got launched, but didn't see anything else [10:20:59] I see that refinery-sqoop-whole-mediawiki.service failed [10:21:06] did you check journalctl? [10:21:18] ah PermissionError: [Errno 13] Permission denied: '/var/log/refinery/sqoop-mediawiki.log' [10:21:34] man - Got that without looking at the log :) [10:22:06] PROBLEM - Check the last execution of refinery-sqoop-whole-mediawiki on an-coord1001 is CRITICAL: CRITICAL: Status of the systemd unit refinery-sqoop-whole-mediawiki [10:23:31] elukey: I think analytics user needs to be in group analytics-admins [10:24:10] joal: I am a bit reluctant to add it to that group since it shouldn't have all the powers that we have [10:24:24] ah right - makes sense [10:24:28] possibly better to bend /var/log/refinery/'s perms [10:24:39] elukey@an-coord1001:~$ ls -ld /var/log/refinery/ [10:24:39] drwxrwsr-x 20 hdfs analytics-admins 20480 May 7 06:25 /var/log/refinery/ [10:31:19] joal: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/508534/1/modules/profile/manifests/analytics/refinery.pp - what do you think? [10:32:17] elukey: works for me if analytics is superseded by analytics-admins [10:37:59] joal: done! Can you re-run the job? [10:39:29] elukey: testing! [10:39:47] success ! [10:41:44] joal: nice! next step, if you want to clear the alert, is to systemctl reset-failed refinery-sqoop-whole-mediawiki.service [10:42:44] RECOVERY - Check the last execution of refinery-sqoop-whole-mediawiki on an-coord1001 is OK: OK: Status of the systemd unit refinery-sqoop-whole-mediawiki [10:43:12] \o/ [10:43:31] elukey: should fail again soon [10:44:08] elukey: first job failed with expected error now (i had to change ownership of temporary folder in hdfs) [10:44:54] elukey: everything looks good :) Thanks a lot for your help and support :) [10:46:56] elukey: With your permission, I'll chown analytics:analytics /wmf/data/raw/mediawiki/tables [10:47:09] Those are written by sqoop, therefore by ananlytics now [10:47:24] (same for /wmf/data/raw/mediawiki_private/tables) [10:49:47] +1! [10:50:14] elukey: and while at it, we could change ownership of the mediawiki_project_namespace_map script [10:50:27] yep! [10:50:33] I am currently checking the perms in /wmf/data/archive/mediacounts [10:50:42] that are a bit of a mess afaics [10:50:53] :S [10:51:27] but no idea how they should look like :( [10:51:52] elukey: I think it must come from the fact that some files are computed outside the cluster :S [10:52:36] are those completely public? [10:52:45] I mean without PII data etc.. [10:52:49] elukey: yes [10:53:18] elukey: mediacount-archives archive are like pageviews - to be sync as dumps files [10:53:22] ok so probably they don't need analytics-privatedata-users [10:53:44] and yeah we'd also need to change Erik's jobs to push files as analytics [10:53:52] ok I'll leave this for last :P [10:54:02] elukey: I think we should have 2 different folders for archives - One for external archives, one for internal [10:54:12] even better yes [10:54:47] anyway [11:13:18] joal: https://gerrit.wikimedia.org/r/508536 for namespace map :) [11:13:42] Ah elukey - Was doing it !! [11:14:18] elukey: Doing xmldumps import? [11:15:58] elukey: changing perms for namespace_map? [11:16:36] what sorry? :D [11:16:46] huhu :) [11:17:03] I created https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/504861/ a while ago [11:17:14] elukey: I'm about to send a patch for import_xmldumps to use analytics user [11:17:36] elukey: And I'm also about to change the permissions of existing namespace_map data to analytics [11:17:39] ok for you? [11:18:11] ahh okok thanks for the --verbose [11:18:15] please go ahead! thanks for the help! [11:18:28] :) [11:19:06] !log Updating /wmf/data/raw/mediawiki/tables to be owned by analytics:analytics [11:19:08] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:19:16] !log Updating /wmf/data/raw/mediawiki/project_namespace_map to be owned by analytics:analytics [11:19:18] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:19:51] !log Updating /wmf/data/raw/mediawiki/xmldumps to be owned by analytics:analytics [11:19:52] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:20:13] elukey: only 2 lefts are used by oozie, so not touching [11:21:34] ack! [11:21:50] in /wmf/data/raw/mediawiki I meant :) [11:23:11] !log Updating /wmf/data/raw/mediawiki_private/tables to be owned by analytics:analytics [11:23:12] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:25:16] ok my next target(s) are mediawiki-geoeditros [11:25:19] *geoeditros [11:25:21] uff [11:25:26] today is difficult :D [11:28:43] it seems a bit intricated, will move away from mediawiki and proceed with the rest [11:31:17] (03PS1) 10Elukey: oozie/mobileapps: move coords to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508541 (https://phabricator.wikimedia.org/T220971) [11:46:53] !log kill mobile apps coordinators + chown analytics:analytics /wmf/data/archive/mobile_apps, /wmf/data/wmf/mobile_apps + restart of the coordinators with user analytics [11:46:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:48:33] all right going to lunch :) [11:49:18] hey team :] [12:02:34] 10Analytics, 10EventBus, 10Operations, 10observability, and 3 others: Upgrade statsd_exporter to 0.9 - https://phabricator.wikimedia.org/T220709 (10fgiunchedi) Looks good so far in deployment-prep, there's a deb on `boron` for testing `/var/cache/pbuilder/result/stretch-amd64/prometheus-statsd-exporter_0.9... [12:04:45] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10akosiaris) >>! In T218346#5161770, @Ottomata wrote: > @akosiaris I think https:/... [13:13:13] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Fundraising-Backlog, and 3 others: Fix EventLogging schemas that use array for items type - https://phabricator.wikimedia.org/T218617 (10Ottomata) Thank you! [13:13:50] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Fundraising-Backlog, and 3 others: Fix EventLogging schemas that use array for items type - https://phabricator.wikimedia.org/T218617 (10Ottomata) [13:15:01] 10Analytics, 10EventBus, 10Operations, 10observability, and 3 others: Upgrade statsd_exporter to 0.9 - https://phabricator.wikimedia.org/T220709 (10Ottomata) Great! I guess it just needs to go into the WMF base docker image somehow? [13:20:42] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10Ottomata) Ok, sounds good. I'll be deploying a new 'service' to the eventgate-a... [13:23:39] joal: hellooo, starting my day now (8am in tx) [13:23:57] I'm pretty sure I restarted the cassandra bundle when we did that [13:33:22] good morning fdans :) [13:52:45] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10Ottomata) [14:00:19] (03PS7) 10Fdans: Replace time range selector [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/499968 (https://phabricator.wikimedia.org/T219112) [14:00:27] (03CR) 10jerkins-bot: [V: 04-1] Replace time range selector [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/499968 (https://phabricator.wikimedia.org/T219112) (owner: 10Fdans) [14:21:00] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10Ottomata) Ok, working on two remaining tasks here: 1. Replace chart=eventgate-a... [14:22:59] (03PS1) 10Elukey: oozie/pageview: move all coordinators to the new analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508585 (https://phabricator.wikimedia.org/T220971) [14:23:42] joal: ready to move the pageviews coord to the analytics user [14:24:08] afaics I'd need to chown /wmf/data/archive/pageview (for the legacy ones) and /wmf/data/wmf/pageview [14:24:17] (stop 4 coords, chown, restart) [14:24:32] o/ joal also q for you about eventlogging refine and nullable stuff when you get back :) [14:37:10] !log kill pageview oozie coord (4 in total) + chown analytics:analytics /wmf/data/wmf/pageview /wmf/data/archive/pageview + restart of the coordinators with the analytics user [14:37:12] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:39:18] 10Analytics, 10Analytics-Kanban, 10Discovery-Analysis, 10EventBus, 10Product-Analytics: Fix active EventLogging schemas that added backwards incompatable required fields. - https://phabricator.wikimedia.org/T222656 (10EBernhardson) I might suggest adding a validation step to saving schema's on mw.org th... [14:40:54] 10Analytics, 10Analytics-Kanban, 10Discovery-Analysis, 10EventBus, 10Product-Analytics: Fix active EventLogging schemas that added backwards incompatable required fields. - https://phabricator.wikimedia.org/T222656 (10Ottomata) On mw.org will be hard, but for Modern Event Platfrom schemaa repositories, f... [14:51:15] (03CR) 10Nuria: [C: 03+1] Fix oozie banner monthly job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508358 (https://phabricator.wikimedia.org/T222603) (owner: 10Joal) [14:51:26] (03CR) 10Nuria: [V: 03+2 C: 03+2] Fix oozie banner monthly job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508358 (https://phabricator.wikimedia.org/T222603) (owner: 10Joal) [14:56:27] (03PS3) 10Milimetric: Change projection of world map to eckert 3 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/504429 (https://phabricator.wikimedia.org/T218045) [15:00:39] (03PS4) 10Milimetric: Change projection of world map to eckert 3 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/504429 (https://phabricator.wikimedia.org/T218045) [15:01:54] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Wikistats: Change Mercator Projection to Eckert III - https://phabricator.wikimedia.org/T218045 (10Milimetric) [15:05:58] (03CR) 10Nuria: banner_activity: move oozie coordinators to the analytics user (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508283 (https://phabricator.wikimedia.org/T220971) (owner: 10Elukey) [15:06:34] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Wikistats: Change Mercator Projection to Eckert III - https://phabricator.wikimedia.org/T218045 (10Milimetric) Decided on eckert 3 over eckert 4, as it looks a lot better and gets closer to equal area than Mercator (it's not quite equal area, Africa still... [15:14:04] ottomata: did you tweak that centralnoticeimpression schema? [15:15:38] 10Analytics, 10Analytics-Kanban: Decide: start_timestamp for mediawiki history - https://phabricator.wikimedia.org/T220507 (10Milimetric) a:05Milimetric→03JAllemandou [15:32:25] nuria, joal: retro? [15:32:31] Cc: milimetric [15:32:41] argh , wait, I am in teh middle of managers meeting [15:32:55] and I'm spacing out [15:33:57] elukey: ok, coming [15:34:25] ping ottomata mforns milimetric joal elukey fdans RETRO [15:34:47] nuria: we are in the bc [15:34:52] oohhh [15:35:49] (03PS1) 10Milimetric: Make success email a little friendlier [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508602 (https://phabricator.wikimedia.org/T206894) [15:40:08] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10fsero) @Ottomata sure, i can help you with that [15:50:17] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Change permissions for daily traffic anomaly reports on stat1007 - https://phabricator.wikimedia.org/T219546 (10Jdcc-berkman) If this is successfully running elsewhere now, I don't need the data in my home directory anymore nor my account. I've backed up the va... [16:22:44] 10Analytics: Move reportupdater queries from limn-* repositories to reportupdater-queries - https://phabricator.wikimedia.org/T222739 (10Milimetric) [16:23:10] 10Analytics: Move reportupdater queries from limn-* repositories to reportupdater-queries - https://phabricator.wikimedia.org/T222739 (10Milimetric) p:05Triage→03Normal [16:25:34] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10fsero) @Ottomata i've created the namespaces and the users for eventgate-main yo... [16:32:28] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Change permissions for daily traffic anomaly reports on stat1007 - https://phabricator.wikimedia.org/T219546 (10elukey) @Jdcc-berkman since you have access to the host, can you please clean up those files? So I'll not accidentally delete anything valuable etc..... [16:35:20] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10Ottomata) Awesome thank you! I'll get that set up today, and we can talk about... [16:39:28] mforns: can you deploy aqs with chu chu train and thus enable in wikistats the new snapshot (cc joal, milimetric ) [16:39:57] nuria, tomorrow? [16:40:11] mforns: yes tomorrow [16:40:19] nuria, sure [16:40:43] mforns: I'm gonna send a patch with the new snapshot-update for aqs [16:40:56] (03CR) 10Nuria: "I am a bit confused on how does this interact with our code that restricts ingestions to druid to be performed by hdfs user alone." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508585 (https://phabricator.wikimedia.org/T220971) (owner: 10Elukey) [16:42:52] mforns: we have been documenting weekly deploys here so the deploy person has a template of what needs to be done [16:42:54] https://etherpad.wikimedia.org/p/analytics-weekly-train [16:43:54] (03CR) 10Nuria: [C: 03+2] "Looks good, merging. Let's document on chu chu train how to update table definitions after we deploy this. Sounds like several jobs need t" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507933 (https://phabricator.wikimedia.org/T213770) (owner: 10Joal) [16:44:14] (03CR) 10Nuria: [C: 03+2] Move the clickstream oozie coordinator to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508525 (https://phabricator.wikimedia.org/T220971) (owner: 10Elukey) [16:45:23] nuria, ok [16:46:53] fdans: please see comment of joal on https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/507355/, i think for this change to be complete you need to also add those wikis to the prod list [16:47:36] nuria: yeah I’m addressing that now [16:47:45] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Many small wikis missing from mediawiki_history dataset - https://phabricator.wikimedia.org/T220456 (10Nuria) ping @fdans let's make sure the prod list and lab list match as every snapshot scoops from both [16:48:43] * elukey afk for a bit! [16:51:15] fdans: ping also on the review for python-druid please :) [16:52:03] yessir [16:54:15] 10Analytics, 10Growth-Team, 10Product-Analytics: Update ServerSideAccountCreation schema whitelist - https://phabricator.wikimedia.org/T222101 (10nettrom_WMF) p:05Triage→03Normal [16:58:40] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: Possibly faulty BBU on analytics1029 - https://phabricator.wikimedia.org/T178742 (10Dzahn) Is the issue back or is this known ? https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=analytics1029&service=Device+not+healthy+-SMART- [17:04:01] (03CR) 10Nuria: [C: 03+2] oozie/mobileapps: move coords to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508541 (https://phabricator.wikimedia.org/T220971) (owner: 10Elukey) [17:04:23] a-team: I'm working on a patch to update the ServerSideAccountCreation schema. What's the current best practices on storing tokens, is that to use "hash" for them instead of just storing them directly? [17:04:33] analytics1029 has problems. unsure if it's known. notifications are disabled but that could be old from previous issue [17:04:49] did not see anything in SAL or phab though [17:05:06] Nettrom, yes please, this way we ensure they can not be used to collect user browsing histories [17:05:29] Nettrom, you don't need to instrument the hashing, though [17:05:43] mforns: wonderful, thanks! I'll change that in this patch too then [17:05:45] Nettrom, just use the keyword "hash" in the EL sanitization white-list [17:06:28] mforns: got it [17:07:41] milimetric: gotta say that map looks real nice [17:08:26] ty :) [17:08:42] nuria: ,mforns i'll probably want a refinery-source deploy soon [17:08:55] i can do it tho, should I add it to the analytics-weekly-train? [17:09:06] ottomata: let's just have 1 deploy [17:09:11] k [17:09:13] ottomata: ya, chu chu train [17:09:14] when is the next deploy then? [17:09:15] ottomata, you want to do it today? tomorrow is train [17:09:18] oh! [17:09:22] tomorrow will be fine [17:09:24] ok! [17:09:27] i'll make sure my patch gets in [17:09:28] I will do [17:09:31] right [17:14:11] (03PS1) 10Nettrom: Update ServerSideAccountCreation whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508626 (https://phabricator.wikimedia.org/T222101) [17:23:57] (03CR) 10Neil P. Quinn-WMF: [C: 03+1] "Thank you, Dan! 😁" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508602 (https://phabricator.wikimedia.org/T206894) (owner: 10Milimetric) [17:33:25] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Fix jobs after mediawiki-history refactor - https://phabricator.wikimedia.org/T222425 (10JAllemandou) [17:43:28] mforns: as FYI I'd need to merge/deploy this set of patches https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/508283/ [17:43:34] but I have already restarted the coordinators [17:43:38] so no action needed [17:43:39] (03PS8) 10Fdans: Replace time range selector [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/499968 (https://phabricator.wikimedia.org/T219112) [17:43:42] (only for consistency) [17:44:09] elukey, ? [17:44:17] should I deploy those tomorrow? [17:44:35] (03CR) 10Joal: [V: 03+2 C: 03+2] "LGTM - Merging! Thanks Dan :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508602 (https://phabricator.wikimedia.org/T206894) (owner: 10Milimetric) [17:44:36] yeah they'd need to be reviewed/merged before the train [17:44:44] very trivial changes [17:44:54] (it is for the hdfs -> analytics user transition) [17:45:04] I have restarted the coordinators already using -D user=analytics [17:45:20] what I meant to ask is please do not deploy before those are merged :) [17:46:01] (03CR) 10jerkins-bot: [V: 04-1] Replace time range selector [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/499968 (https://phabricator.wikimedia.org/T219112) (owner: 10Fdans) [17:48:59] elukey, sure! [17:49:10] elukey, but I'll need to restart the coordinators again no? [17:49:35] elukey, mforns - I think it'll be great if you guys sync tomorrow for the restart of jobs, as some of them need to be restarted with user analytics [17:49:49] joal, elukey sure [17:50:31] mforns: I have documented my stuff (and the one from milimetric I merged) in the tchu-pad [17:50:37] mforns: https://etherpad.wikimedia.org/p/analytics-weekly-train [17:50:50] mforns: sorry - hard deploy :S [17:51:38] mforns: no no need to restart any of those, I have already done it using the -D user=etc.. [17:51:52] (03CR) 10Joal: banner_activity: move oozie coordinators to the analytics user (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508283 (https://phabricator.wikimedia.org/T220971) (owner: 10Elukey) [17:51:52] joal: which ones need to be restarted? [17:52:07] elukey: plenty ... :S [17:52:19] elukey: https://etherpad.wikimedia.org/p/analytics-weekly-train [17:52:43] tchu-pad? xDDDD [17:52:52] joal: ah but not for my changes [17:52:53] okok [17:53:06] elukey, joal, tomorrow I'll start working at 13h our time [17:53:17] elukey: but coordination is needed, as some need sudo -u analytics, and other sudo -u hdfs :( [17:53:18] and will start with deployment [17:53:48] ok mforns - I'll be on and off tomorrow, bank holiday in France, but will keep an eye in case you ping [17:53:53] joal: yep yep now I got it [17:53:59] (03PS1) 10Ottomata: Refine - Make all fields not required when reading data using JSONSchema [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/508655 (https://phabricator.wikimedia.org/T215442) [17:54:03] sorry for the big mess lads :( [17:54:10] nah it will be fun! :P [17:54:30] elukey: you know my words ;) [17:54:49] hehe [17:54:56] mforns: I'll be around but the only nit to check is that some coords have already been migrated to the analytics user, so you will need to check and use sudo -u analytics vs sudo -u hdfs when using the oozie cli [17:55:07] aha [17:55:10] no problemo [17:56:01] mforns: if feasible I'd also like https://phabricator.wikimedia.org/T220111 to be merged for tomorrow - Waiting on review for some time now [17:56:12] going off! [17:56:14] ok [17:56:18] Bye elukey [17:56:18] bye elukey ! [17:56:39] same for me - gone for diner - Will check pings after [17:56:49] byeeee [17:56:55] mforns: Don't hesitate if the stuff I wrote in the pad is not clear :) [17:57:15] thank you joal :] [17:57:39] hi bearloga :) [17:58:04] bearloga: would you mind putting some executor-constraint on your SparkR jobs please? [17:58:18] It currently eats most of the cluster resources :) [17:59:10] 10Analytics, 10Analytics-Kanban, 10Discovery-Analysis, 10EventBus, and 2 others: Fix active EventLogging schemas that added backwards incompatable required fields. - https://phabricator.wikimedia.org/T222656 (10Ottomata) So, I was able to find a way to make the Refine work without requiring that we alter a... [17:59:33] (03CR) 10Nuria: "Seems worth adding a test for this case?" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/508655 (https://phabricator.wikimedia.org/T215442) (owner: 10Ottomata) [18:00:51] (03CR) 10Joal: "> Patch Set 1:" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/508655 (https://phabricator.wikimedia.org/T215442) (owner: 10Ottomata) [18:01:17] gone for diner for real :) [18:01:19] nuria: I agree, but it is pretty hard to add tests for stuff like that. [18:13:57] 10Analytics, 10Analytics-Kanban, 10Discovery-Analysis, 10EventBus, and 2 others: Fix active EventLogging schemas that added backwards incompatable required fields. - https://phabricator.wikimedia.org/T222656 (10Dbrant) > The most egregious offense! Type changes are definitely not allowed! Mea culpa! But I... [18:17:19] 10Analytics, 10Analytics-Kanban, 10Discovery-Analysis, 10EventBus, and 2 others: Fix active EventLogging schemas that added backwards incompatable required fields. - https://phabricator.wikimedia.org/T222656 (10Ottomata) Ah sorry! You are right, I read the diff hastily (sorry was reading lots of diffs yes... [19:53:16] milimetric: wanna to centralanotice tbale? [19:53:29] man, ever since getting this new keyboard I my typo rate has really increased [19:53:31] omw cave [19:53:33] new macbook [20:04:30] (03PS9) 10Fdans: Replace time range selector [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/499968 (https://phabricator.wikimedia.org/T219112) [20:05:08] ottomata: aren't your fingertips flatter too? [20:05:39] CTS incoming [20:07:15] (03CR) 10jerkins-bot: [V: 04-1] Replace time range selector [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/499968 (https://phabricator.wikimedia.org/T219112) (owner: 10Fdans) [20:07:46] (╯°□°）╯︵ ┻━┻ [20:08:17] xD [20:10:10] terrific [20:10:17] https://usercontent.irccloud-cdn.com/file/TRIwK2j1/Screen%20Shot%202019-05-07%20at%203.10.03%20PM.png [20:11:19] and moments later, same code: [20:11:27] https://usercontent.irccloud-cdn.com/file/sgSlALzk/Screen%20Shot%202019-05-07%20at%203.11.08%20PM.png [20:13:39] mforns: tienes un momento para la baticueva? [20:13:48] fdans, yessss [20:13:50] omw [20:14:20] mforns: #2 [20:14:24] k [20:14:57] fdans, https://hangouts.google.com/hangouts/_/wikimedia.org/a-batcave-2 [20:16:24] 10Analytics, 10Analytics-Kanban, 10Fundraising-Backlog: CentralNoticeImpression refined impressionEventSampleRate is int instead of double - https://phabricator.wikimedia.org/T217109 (10Milimetric) for the record, we made a teeny mistake the first time we did this and the useragent field had a bad schema. S... [20:44:25] (03CR) 10Milimetric: "Initial comments are in only a few categories, I just tried to find all the instances to make it easier:" (0328 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/499968 (https://phabricator.wikimedia.org/T219112) (owner: 10Fdans) [20:50:36] (03CR) 10Ottomata: [C: 03+2] Refine - Make all fields not required when reading data using JSONSchema [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/508655 (https://phabricator.wikimedia.org/T215442) (owner: 10Ottomata) [21:09:19] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: 15.wikipedia.org missclassified as a pageview, same for query.wikidata.org - https://phabricator.wikimedia.org/T222460 (10mforns) OK, I think I have some conclusions. We always got some traffic from '15.wikipedia.org' and 'query.wikidata.org'. ` select ur... [21:18:30] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review: Make Refine use JSONSchemas of event data to support Map types and proper types for integers vs decimals - https://phabricator.wikimedia.org/T215442 (10Ottomata) Ok, with the latest patches, I seem to be able to use the JSONSchemas to read Ev... [21:40:03] (03PS2) 10Mforns: Filter out 15.wikipedia.org and query.wikidata.org from pageview definition [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/508418 (https://phabricator.wikimedia.org/T222460) [21:56:37] milimetric: here https://gerrit.wikimedia.org/r/#/c/analytics/wikistats2/+/499968/8/src/components/detail/Detail.vue you mean adjusting to granularity as part of the state mutation? [21:56:48] (sorry if you're out already) [22:29:09] (03CR) 10Nuria: [C: 03+2] Filter out 15.wikipedia.org and query.wikidata.org from pageview definition [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/508418 (https://phabricator.wikimedia.org/T222460) (owner: 10Mforns) [23:53:45] (03PS10) 10Fdans: Replace time range selector [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/499968 (https://phabricator.wikimedia.org/T219112) [23:54:40] (03CR) 10Fdans: "@milimetric: I'm still fighting with a couple tests but all your comments have been addressed" (0326 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/499968 (https://phabricator.wikimedia.org/T219112) (owner: 10Fdans) [23:56:09] (03CR) 10jerkins-bot: [V: 04-1] Replace time range selector [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/499968 (https://phabricator.wikimedia.org/T219112) (owner: 10Fdans)