[06:11:57] 10Analytics, 10DBA: Upgrade analytics dbstore databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254870 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin2001.codfw.wmnet for hosts: ` ['dbstore1005.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto... [06:30:34] 10Analytics, 10DBA: Upgrade analytics dbstore databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254870 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbstore1005.eqiad.wmnet'] ` and were **ALL** successful. [07:01:00] 10Analytics, 10DBA, 10Patch-For-Review: Upgrade analytics dbstore databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254870 (10Marostegui) [07:01:41] 10Analytics, 10DBA, 10Patch-For-Review: Upgrade analytics dbstore databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254870 (10Marostegui) 05Open→03Resolved a:03Marostegui All done! [07:05:19] Good morning [07:39:07] gooood morning [07:39:56] today I had to take the car in the morning and of course I had a flat tire (puncture) [07:40:19] perfect way to start the week [07:41:46] :S [07:52:44] so I am going to change the tire (I have the small/temp one now), will probably be afk until after lunch time, will take half a day [07:52:56] joal: everything looks good right? [07:53:20] in any case, if needed ping me on the phone :) [07:55:33] np elukey - all good so far :) [07:56:52] super [07:57:00] good luck :S [07:57:17] thanks! :D [08:43:08] back! [08:43:17] should get the new tire this aftertoon or tomorrow [08:46:23] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban: Add new kafka brokers kafka-jumbo100[789] to the jumbo-eqiad Kafka cluster - https://phabricator.wikimedia.org/T252675 (10elukey) I had a chat with Andrew, this task is now to be considered done. We'll move/shuffle partitions for optimal balance across br... [08:47:05] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban: Add new kafka brokers kafka-jumbo100[789] to the jumbo-eqiad Kafka cluster - https://phabricator.wikimedia.org/T252675 (10elukey) [08:49:09] 10Analytics, 10Analytics-Cluster: Balance Kafka topic partitions on Kafka Jumbo to take advantage of the new brokers - https://phabricator.wikimedia.org/T255973 (10elukey) [09:01:48] (03PS3) 10Joal: Update clickstream and interlanguage jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606449 [09:17:30] (03CR) 10Joal: [V: 03+2] "Tested on cluster!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606449 (owner: 10Joal) [09:18:34] (03PS2) 10Joal: Correct bug in webrequest host normalization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/606752 (https://phabricator.wikimedia.org/T255779) [09:19:39] (03CR) 10Joal: [V: 03+2] "To be synchronized with:" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606449 (owner: 10Joal) [10:07:29] I have split the cookbooks for the upgrade in two [10:07:29] https://gerrit.wikimedia.org/r/#/c/operations/cookbooks/+/606736/ [10:07:33] - stop-cluster.py [10:07:38] - change-distro.py [10:07:50] tested some commands and they seem good [10:24:18] going afk for a quick lunch! [12:56:34] will m_forns be around today or still off? (just trying to plan when to look at certain things) [12:59:32] I think he'll be off today :( [13:01:14] 10Analytics-EventLogging, 10Analytics-Radar, 10NewcomerTasks 1.2, 10Product-Analytics, and 2 others: NewcomerTask EventLogging schema has invalid array items type specification - https://phabricator.wikimedia.org/T255597 (10Ottomata) > that documentation should probably be more specific about what is acce... [13:04:06] ok, np :) I just wont look at the task at all then (yet) :D [13:20:55] 10Analytics-EventLogging, 10Analytics-Radar, 10NewcomerTasks 1.2, 10Product-Analytics, and 2 others: NewcomerTask EventLogging schema has invalid array items type specification - https://phabricator.wikimedia.org/T255597 (10Tgr) >>! In T255597#6244375, @Ottomata wrote: > Ok, updated https://wikitech.wikime... [13:43:42] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230 (10Ottomata) @Samwilson @Niharika Hello! I'm looking for a candidate EventLogging schema stream to migra... [14:13:46] 10Analytics-Radar, 10AbuseFilter, 10Cognate, 10ConfirmEdit (CAPTCHA extension), and 28 others: Replace PageContent(Insert|Save)Complete hooks - https://phabricator.wikimedia.org/T250566 (10DannyS712) @Ostrzyciel the work to use EditResult shouldn't be tagged as part of this task [14:15:06] ottomata: good morning :) Let me know when ou have a minute to talk about normalization deduplication :) [14:17:52] ok joal gimme 10 mins! [14:18:15] sure ottomata [14:28:43] ok joal in bc [14:28:50] joining ot! [14:29:02] (03CR) 10Nuria: Correct bug in webrequest host normalization (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/606752 (https://phabricator.wikimedia.org/T255779) (owner: 10Joal) [14:34:20] 10Analytics-Radar, 10AbuseFilter, 10Cognate, 10ConfirmEdit (CAPTCHA extension), and 28 others: Replace PageContent(Insert|Save)Complete hooks - https://phabricator.wikimedia.org/T250566 (10Ostrzyciel) >>! In T250566#6244591, @DannyS712 wrote: > @Ostrzyciel the work to use EditResult shouldn't be tagged as... [14:35:45] 10Analytics-Radar, 10AbuseFilter, 10Cognate, 10ConfirmEdit (CAPTCHA extension), and 28 others: Replace PageContent(Insert|Save)Complete hooks - https://phabricator.wikimedia.org/T250566 (10DannyS712) >>! In T250566#6244668, @Ostrzyciel wrote: >>>! In T250566#6244591, @DannyS712 wrote: >> @Ostrzyciel the wo... [14:56:38] (03PS3) 10Joal: Correct bug in webrequest host normalization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/606752 (https://phabricator.wikimedia.org/T255779) [14:56:45] (03CR) 10Joal: Correct bug in webrequest host normalization (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/606752 (https://phabricator.wikimedia.org/T255779) (owner: 10Joal) [15:14:21] 10Analytics, 10User-Ladsgroup: Add shnwiktionary to analytics whitelist - https://phabricator.wikimedia.org/T256013 (10Nintendofan885) [15:42:28] 10Analytics, 10Analytics-Cluster, 10Research: Can we have more RAM on stat machines - https://phabricator.wikimedia.org/T255716 (10elukey) Opened T256017 to SRE :) [16:07:42] 10Analytics, 10Product-Analytics: Remove COUNT(*) from datasets when not useful in Superset & Turnilo - https://phabricator.wikimedia.org/T255725 (10LGoto) p:05Triage→03Medium [16:08:49] 10Analytics, 10Product-Analytics: Remove COUNT(*) from datasets when not useful in Superset & Turnilo - https://phabricator.wikimedia.org/T255725 (10mpopov) It can be removed manually in Superset (and has been for some datasets), but not sure if it gets automatically re-added when new data is ingested or when... [16:12:03] ping fdans [16:12:21] fdans: groominggg [16:17:31] 10Analytics-Radar, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review, 10Product-Analytics (Kanban): Creation of a new POSIX group and system user for the Product Analytics team - https://phabricator.wikimedia.org/T255039 (10mpopov) a:03mpopov [16:19:07] 10Analytics: Add `latest_page_event` and `latest_user_event` fields in mediawiki_history - https://phabricator.wikimedia.org/T255890 (10Milimetric) p:05Triage→03Medium [16:20:15] 10Analytics, 10Analytics-Kanban, 10Event-Platform: EventStreamConfig's auto-topics config is incorrect - https://phabricator.wikimedia.org/T255888 (10Milimetric) p:05Triage→03High [16:21:19] 10Analytics, 10Design: Broken icons on https://analytics.wikimedia.org/ - https://phabricator.wikimedia.org/T255840 (10Milimetric) p:05Triage→03High [16:21:29] 10Analytics-Radar, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review, 10Product-Analytics (Kanban): Creation of a new POSIX group and system user for the Product Analytics team - https://phabricator.wikimedia.org/T255039 (10mpopov) @elukey: so who needs to do what? it looks like SRE needs to review... [16:21:40] 10Analytics: Refine drops $schema field values - https://phabricator.wikimedia.org/T255818 (10Milimetric) p:05Triage→03High [16:24:33] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 2 others: EventLogging MEP Upgrade - https://phabricator.wikimedia.org/T238544 (10mpopov) a:05jlinehan→03mpopov [16:37:45] 10Analytics, 10Product-Analytics: Check Product Analytics team's standard datasets and remove COUNT(*) - https://phabricator.wikimedia.org/T256025 (10kzimmerman) p:05Triage→03Medium [16:52:38] milimetric, nuria - I have 3 patches I'd like to deploy this week - 2 of them have already been reviewed by nuria (https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/603590/ and https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/606752/), last one not yet (oozie change - https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/606449/) - Many thanks :) [16:55:44] 10Analytics: Add `is_last_page_event` and `is_last_user_event` fields in mediawiki_history - https://phabricator.wikimedia.org/T255890 (10Milimetric) [16:56:19] joal: updated description, take a look ^ and I'll look at that last patch you mention [16:57:35] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban: Add new kafka brokers kafka-jumbo100[789] to the jumbo-eqiad Kafka cluster - https://phabricator.wikimedia.org/T252675 (10Nuria) And, to triple confirm, all new hosts have been added to analytics VLAN? [17:01:29] 10Analytics-EventLogging, 10Analytics-Radar, 10NewcomerTasks 1.2, 10Product-Analytics, and 2 others: NewcomerTask EventLogging schema has invalid array items type specification - https://phabricator.wikimedia.org/T255597 (10Tgr) That only happens because the page doesn't exist, which won't happen in produc... [17:04:47] (03PS4) 10Milimetric: Update clickstream and interlanguage jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606449 (https://phabricator.wikimedia.org/T255779) (owner: 10Joal) [17:06:22] (03CR) 10Milimetric: "This change also depends on https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/606443/, otherwise +2, I'll update the commit m" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606449 (https://phabricator.wikimedia.org/T255779) (owner: 10Joal) [17:06:58] (03PS5) 10Milimetric: Update clickstream and interlanguage jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606449 (https://phabricator.wikimedia.org/T255779) (owner: 10Joal) [17:07:06] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Update clickstream and interlanguage jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606449 (https://phabricator.wikimedia.org/T255779) (owner: 10Joal) [17:12:27] * elukey off! [17:37:08] (03CR) 10Jenniferwang: "Hi," [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/606734 (https://phabricator.wikimedia.org/T247417) (owner: 10Nuria) [17:48:45] (03CR) 10Nuria: [WIP] Usage of commons files for tech tunning session metrics (031 comment) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/606734 (https://phabricator.wikimedia.org/T247417) (owner: 10Nuria) [17:53:57] (03PS2) 10Nuria: eventlogging: Remove unused props from PrefUpdate [analytics/refinery] - 10https://gerrit.wikimedia.org/r/588105 (https://phabricator.wikimedia.org/T249894) (owner: 10Krinkle) [18:01:56] 10Analytics-EventLogging, 10Analytics-Radar, 10NewcomerTasks 1.2, 10Product-Analytics, and 2 others: NewcomerTask EventLogging schema has invalid array items type specification - https://phabricator.wikimedia.org/T255597 (10Etonkovidova) 05Open→03Resolved >>! In T255597#6245490, @Tgr wrote: > That only... [18:42:04] 10Analytics, 10Analytics-Kanban, 10User-Ladsgroup: Add shnwiktionary to analytics whitelist - https://phabricator.wikimedia.org/T256013 (10JAllemandou) [18:44:51] (03PS1) 10Joal: Add shn.wiktionary to pageview whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/607088 (https://phabricator.wikimedia.org/T256013) [18:45:07] milimetric: another one if you have a minute please --^ [18:48:23] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Add shn.wiktionary to pageview whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/607088 (https://phabricator.wikimedia.org/T256013) (owner: 10Joal) [18:49:19] \o/ - sending a sqoop one in a minute, and then I'll be done :) [18:50:57] !log Manually update pageview whitelist adding shnwiktionary [18:50:58] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:57:11] (03PS1) 10Joal: Update sqoop list [analytics/refinery] - 10https://gerrit.wikimedia.org/r/607092 (https://phabricator.wikimedia.org/T256013) [19:10:17] (03PS1) 10Joal: Bump dataquality refinery-job jar version [analytics/refinery] - 10https://gerrit.wikimedia.org/r/607094 (https://phabricator.wikimedia.org/T251542) [19:15:56] 10Analytics-EventLogging, 10Analytics-Radar, 10NewcomerTasks 1.2, 10Product-Analytics, and 2 others: NewcomerTask EventLogging schema has invalid array items type specification - https://phabricator.wikimedia.org/T255597 (10Ottomata) FYI, this was not a JSONSchema validation error, it was a Hive ingestion... [19:20:31] elukey: should we remove this section? [19:20:31] https://wikitech.wikimedia.org/wiki/Analytics/Data_access#Host_access_granted [19:20:44] now that all groups give access to all stat boxes, and notebook* are gone? [19:22:49] 10Analytics, 10Analytics-Kanban: Aggregate pageview from pageview_actor_hourly - https://phabricator.wikimedia.org/T256049 (10JAllemandou) [19:22:58] 10Analytics, 10Analytics-Kanban: Aggregate pageview from pageview_actor_hourly - https://phabricator.wikimedia.org/T256049 (10JAllemandou) a:03JAllemandou [19:23:18] (03PS1) 10Joal: Aggregate pageview_hourly from pageview_actor_hourly [analytics/refinery] - 10https://gerrit.wikimedia.org/r/607096 (https://phabricator.wikimedia.org/T256049) [19:25:35] (03PS2) 10Joal: Aggregate pageview_hourly from pageview_actor_hourly [analytics/refinery] - 10https://gerrit.wikimedia.org/r/607096 (https://phabricator.wikimedia.org/T256049) [19:26:39] (03PS3) 10Joal: Aggregate pageview_hourly from pageview_actor_hourly [analytics/refinery] - 10https://gerrit.wikimedia.org/r/607096 (https://phabricator.wikimedia.org/T256049) [19:40:11] 10Analytics, 10Product-Analytics: Add dimensions to editors_daily dataset - https://phabricator.wikimedia.org/T256050 (10cchen) [19:40:23] 10Analytics, 10Product-Analytics: Add dimensions to editors_daily dataset - https://phabricator.wikimedia.org/T256050 (10cchen) p:05Triage→03Medium [19:40:51] (03CR) 10Joal: [V: 03+2] "Tested on cluster" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/607096 (https://phabricator.wikimedia.org/T256049) (owner: 10Joal) [19:46:53] Gone for tonight [20:05:28] (03PS6) 10Nuria: [WIP] Usage of commons files for tech tunning session metrics [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/606734 (https://phabricator.wikimedia.org/T247417) [20:17:24] (03PS7) 10Nuria: Usage of commons files for tech tunning session metrics [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/606734 (https://phabricator.wikimedia.org/T247417) [20:19:35] (03PS8) 10Nuria: Usage of commons files for tech tunning session metrics [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/606734 (https://phabricator.wikimedia.org/T247417) [20:56:45] 10Analytics, 10Analytics-Kanban: Spike, see how easy/hard is to scoop all tables from Eventlogging log database - https://phabricator.wikimedia.org/T250709 (10Nuria) Spot checked also TaskRecommendation_9266319, WikimediaBlogVisit_5308166, ServerSideAccountCreation_17706338, QuickSurveysResponses_15266417, Upl... [21:08:49] 10Analytics, 10Analytics-Kanban: Spike, see how easy/hard is to scoop all tables from Eventlogging log database - https://phabricator.wikimedia.org/T250709 (10Milimetric) This one wasn't bad! If we ever need to sqoop a bunch of tables, the python script I wrote should be almost-generic. Basically, it seems w... [21:11:07] 10Analytics-Kanban, 10Analytics-Radar, 10Product-Analytics, 10Patch-For-Review: Definition of not text content metrics for tuning session (rich media, images, etc) - https://phabricator.wikimedia.org/T247417 (10Milimetric) [22:03:39] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Ladsgroup: Add shnwiktionary to analytics whitelist - https://phabricator.wikimedia.org/T256013 (10Nintendofan885) It's now on line 699 of https://gerrit.wikimedia.org/g/analytics/refinery/+/master/static_data/pageview/whitelist/whitelist.tsv [23:15:47] 10Analytics-Kanban, 10Analytics-Radar, 10Privacy Engineering, 10Privacy, and 3 others: Identify pending analyses needing access to data older than 90 days - https://phabricator.wikimedia.org/T250857 (10nettrom_WMF) [23:17:30] 10Analytics-Kanban, 10Analytics-Radar, 10Privacy Engineering, 10Privacy, and 3 others: Identify pending analyses needing access to data older than 90 days - https://phabricator.wikimedia.org/T250857 (10nettrom_WMF) Since it'll take a little while longer to complete the data analysis in T230174, I updated t...