[05:26:20] !log re-run manually pageview-druid-hourly 29/09T22:00 [05:26:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:02:46] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic, 10WMDE-New-Editors-Banner-Campaigns (Reportings), 10Wikimedia-database-error: No access to mysql from stat1007 - https://phabricator.wikimedia.org/T234160 (10elukey) Interesting, thanks for the report. I am able to connect fine on dbsto... [07:12:38] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic, 10WMDE-New-Editors-Banner-Campaigns (Reportings), 10Wikimedia-database-error: No access to mysql from stat1007 - https://phabricator.wikimedia.org/T234160 (10elukey) Tried also to raise temporarily the connection_timeout global variable... [07:33:40] 10Analytics, 10User-Elukey: Port IRCRecentChanges to Kafka - https://phabricator.wikimedia.org/T232483 (10elukey) [07:34:16] 10Analytics, 10User-Elukey: Port IRCRecentChanges to Kafka - https://phabricator.wikimedia.org/T232483 (10elukey) p:05Normal→03Unbreak! [07:34:58] 10Analytics, 10User-Elukey: Port IRCRecentChanges to Kafka - https://phabricator.wikimedia.org/T232483 (10elukey) p:05Unbreak!→03Normal [07:35:16] 10Analytics, 10User-Elukey: Port IRCRecentChanges to Kafka - https://phabricator.wikimedia.org/T232483 (10elukey) p:05Normal→03Triage [07:35:28] uff fail in setting the priority [08:14:43] Hi team - Family is on the recovery path but I'll still be off today - Hopefully back tomorrow - I'll try to be there for standup say hello [08:22:41] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic, 10WMDE-New-Editors-Banner-Campaigns (Reportings), 10Wikimedia-database-error: No access to mysql from stat1007 - https://phabricator.wikimedia.org/T234160 (10elukey) As quick workaround, please use stat1004. stat1007 seems under heavy l... [08:24:22] joal: o/ <3 [08:57:39] Amir1: o/ as FYI I killed /srv/analytics-wmde/graphite/src/scripts/src/betafeatures/counts.php on stat1007, there was a load issue and the process was one of the top talkers [08:57:53] it didn't solve the issue that I was investigating, so it should be ok [08:58:04] if you want I can re-run it [10:24:57] elukey: thanks for the heads up. It will be reworked tomorrow [10:25:31] ack! [10:30:29] * elukey lunch! [12:46:34] 10Analytics, 10Fundraising-Backlog, 10Operations, 10SRE-Access-Requests: Banner History and page view data access for fundraising analysts - Jerrie and Erin - https://phabricator.wikimedia.org/T233636 (10EYener) Hi @Nuria I have created a Wikitech account. My username is my full name, Erin Yener. Please le... [12:48:09] 10Analytics, 10Fundraising-Backlog, 10Operations, 10SRE-Access-Requests: Banner History and page view data access for fundraising analysts - Jerrie and Erin - https://phabricator.wikimedia.org/T233636 (10jkumalah) HI @Nuria my wikitech is aslo my full name, Jerrie Kumalah [13:27:29] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: High volume mediawiki analytics events camus import is lagging - https://phabricator.wikimedia.org/T233718 (10Ottomata) Everything looks back to normal! @EBernhardson how's it look to you? [13:34:12] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Scoring-platform-team, 10Patch-For-Review: Change event.mediawiki_revision_score schema to use map types - https://phabricator.wikimedia.org/T225211 (10Ottomata) [13:36:20] ottomata: o/ [13:51:06] elukey: when were you trying to free up mysql space (eg. T233891)? [13:51:07] T233891: Drop Navigationtiming data entirely from mysql storage? - https://phabricator.wikimedia.org/T233891 [13:51:26] I can help by moving NavigationTiming to Hadoop [13:55:26] milimetric: hello :) [13:55:44] oh I'm reading T231858 now elukey and that kind of takes it in a different direction [13:55:45] T231858: Archive data on eventlogging MySQL to analytics replica before decomisioning - https://phabricator.wikimedia.org/T231858 [13:55:52] :) [13:56:33] I have an idea for how to make a generic sqoop of all the tables, so that they're 99% likely to be imported correctly, but nuria's right that we'd still have to check each table to be absolutely sure [13:58:07] we'd need to understand who will need that data queriable during the next months I think [13:58:19] it might be that the data in hadoop is sufficient [13:58:27] and we just need to archive it in a .sql format [13:59:26] yeah, I was thinking this could be another strategy [13:59:36] dump each table individually as a separate .sql file [13:59:58] and make a little script to load it into a mysql instance on demand [14:00:26] and maybe the script pings us every time it's used, and we can kind of figure out if this is too painful and come up with a better strategy [14:00:36] but for NavigationTiming, I think we should still move that data, it's used a lot [14:01:02] my proposal in the task is just to keep db1108 as read only db, and archive the current status of the data [14:01:16] so db1107 (the master) will be given back to sre [14:01:32] and eventually (possibly end of fiscal) even db1108 if we don't see people asking for data [14:09:33] elukey: yeah, that makes sense, we could still keep per-table backups going forward just in case. It has happened that many months pass before someone decides to go do data archaeology. [14:15:25] mforns did such a nice job with https://phabricator.wikimedia.org/T224459#5490949 [14:29:40] 10Analytics, 10CheckUser, 10DBA, 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Schema changes for `cu_changes` and `cu_log` table - https://phabricator.wikimedia.org/T233004 (10Milimetric) [14:31:30] 10Analytics, 10CheckUser, 10Core Platform Team: Refactor Comment fields for CheckUser Component - https://phabricator.wikimedia.org/T232531 (10Milimetric) @Anomie: thanks. Importantly, I tagged T233004 with #Analytics as well, because we need to be in this loop. Closing this task had taken us out of the lo... [14:34:14] hello teammm [14:35:15] 10Analytics: Sqoop: remove cuc_comment and join to comment table - https://phabricator.wikimedia.org/T217848 (10Milimetric) I fixed a broken link, the task we need to follow is now T233004, and it looks like work is going forward. So we'll need a patch here. I'm happy to take this but will wait for grooming to... [14:46:19] mforns: o/ [14:46:32] I have a couple of questions for you about the current rsyncs when you have time [14:46:37] hey luca [14:46:39] :] [14:46:40] yes? [14:47:21] do you know more or less when it started to rsync? [14:47:41] yes [14:48:20] it started last friday 5 am I think [14:48:27] UTC [14:48:48] ah ok ok [14:49:07] because this morning I was investigating why mysql tools seems to to connect to mysql dbs from stat1007 [14:49:10] https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=stat1007&var-datasource=eqiad%20prometheus%2Fops&var-cluster=analytics&from=now-24h&to=now [14:49:15] and I noticed that the host is a bit under pressure [14:49:51] uptime is a lot over what it should be [14:49:51] elukey@stat1007:~$ uptime [14:49:51] 14:49:29 up 66 days, 20:55, 6 users, load average: 68.47, 85.18, 78.10 [14:49:55] but not super horrible [14:50:01] aha [14:51:25] elukey, but why mysql dbs? the dumps should go to dumps.wikimedia.org, whis is in WMCS no? [14:52:04] I think it may be a side effect of having a ton of sockets for the rsyncs [14:52:05] *which [14:52:10] but I am still not sure [14:52:13] I see [14:52:28] it is more a suspicion than a proper theory [14:53:10] it can be, because I see 2 types of noy-yet-uploaded files in https://dumps.wikimedia.org/other/mediawiki_history/2019-08 [14:53:25] type 1) when you click, you see an empty folder [14:53:28] will it be the same every month, or only this one is special since it is the first? [14:53:46] type 2) when you click, you see a 403 [14:53:58] I think it will be the same! [14:54:30] I believe type 1) might be all files that have an open socket [14:55:18] (03PS9) 10Fdans: Add oozie job to load top mediarequests data [analytics/refinery] - 10https://gerrit.wikimedia.org/r/538880 (https://phabricator.wikimedia.org/T233717) [14:55:37] (03CR) 10Fdans: [V: 03+1] Add oozie job to load top mediarequests data (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/538880 (https://phabricator.wikimedia.org/T233717) (owner: 10Fdans) [14:57:06] 10Analytics, 10Fundraising-Backlog, 10Operations, 10SRE-Access-Requests: Banner History and page view data access for fundraising analysts - Jerrie and Erin - https://phabricator.wikimedia.org/T233636 (10Nuria) ping @herron I guess jerrie can be added to wmf and Erin to nda groups in ldap? [14:58:36] 10Analytics, 10Analytics-EventLogging: Remove ad-hoc UA logging from existing schemas - https://phabricator.wikimedia.org/T61832 (10Nuria) 05Open→03Resolved [14:59:19] 10Analytics, 10Analytics-EventLogging: Remove ad-hoc UA logging from existing schemas - https://phabricator.wikimedia.org/T61832 (10Nuria) UA logging is not happening per schema since a while back , some older data will still have schemas. [15:02:05] 10Analytics, 10Cloud-Services, 10Developer-Advocacy (Oct-Dec 2019): Develop a tool or integrate feature in existing one to visualize WMCS edits data - https://phabricator.wikimedia.org/T226663 (10Nuria) [15:08:10] 10Analytics, 10Fundraising-Backlog, 10Operations, 10SRE-Access-Requests: Banner History and page view data access for fundraising analysts - Jerrie and Erin - https://phabricator.wikimedia.org/T233636 (10EYener) Hi @Nuria I do have an LDAP account: eyener-ctr I'm not certain if/what groups I belong to, how... [15:25:19] 10Analytics: shorten the time it takes to move files from hadoop to dump hosts - https://phabricator.wikimedia.org/T234229 (10Nuria) [15:27:01] 10Analytics: shorten the time it takes to move files from hadoop to dump hosts by Kerberinzing the dump hosts - https://phabricator.wikimedia.org/T234229 (10Nuria) [15:29:09] 10Analytics, 10Research: Parse wikidumps and extract redirect information for 1 small wiki, romanian - https://phabricator.wikimedia.org/T232123 (10MGerlach) === matching data to mediawiki-history table === The historical redirect table is extracted from wmf.mediawiki_wikitext_history The above code extracts... [15:40:53] 10Analytics, 10Research: Parse wikidumps and extract redirect information for 1 small wiki, romanian - https://phabricator.wikimedia.org/T232123 (10JAllemandou) Hi @MGerlach and @leila - kids and I have been sick almost full last week, explaining me not answering fast. I have spent time trying to get a precis... [15:40:58] 10Analytics: shorten the time it takes to move files from hadoop to dump hosts by Kerberizing the dump hosts - https://phabricator.wikimedia.org/T234229 (10Nuria) [15:41:43] 10Analytics, 10Analytics-Kanban: shorten the time it takes to move files from hadoop to dump hosts by Kerberizing the dump hosts - https://phabricator.wikimedia.org/T234229 (10fdans) [15:46:18] 10Analytics, 10User-Elukey: Port IRCRecentChanges to Kafka - https://phabricator.wikimedia.org/T232483 (10Nuria) [15:46:27] 10Analytics, 10Code-Stewardship-Reviews, 10Operations, 10Tools, 10Wikimedia-IRC-RC-Server: IRC RecentChanges feed: code stewardship request - https://phabricator.wikimedia.org/T185319 (10Nuria) [15:50:13] 10Analytics, 10User-Elukey: Architecture of recent changes on top of kafka. Produce Design Document. - https://phabricator.wikimedia.org/T234234 (10Nuria) [15:50:47] 10Analytics, 10User-Elukey: Port IRCRecentChanges to Kafka - https://phabricator.wikimedia.org/T232483 (10fdans) p:05Triage→03Normal [15:51:20] 10Analytics, 10User-Elukey: Architecture of recent changes on top of kafka. Produce Design Document. - https://phabricator.wikimedia.org/T234234 (10fdans) p:05Triage→03Normal [15:51:59] 10Analytics, 10Analytics-Kanban: shorten the time it takes to move files from hadoop to dump hosts by Kerberizing the dump hosts - https://phabricator.wikimedia.org/T234229 (10fdans) p:05Triage→03High [15:52:55] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic, 10WMDE-New-Editors-Banner-Campaigns (Reportings), 10Wikimedia-database-error: No access to mysql from stat1007 - https://phabricator.wikimedia.org/T234160 (10fdans) p:05Triage→03High [15:53:31] 10Analytics, 10Analytics-Kanban, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic, and 2 others: No access to mysql from stat1007 - https://phabricator.wikimedia.org/T234160 (10fdans) a:03elukey [15:55:03] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: drop CitatitionUsage data on mysql - https://phabricator.wikimedia.org/T233893 (10fdans) p:05Triage→03High [15:57:11] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Services (watching): Add cassandra loading job for top mediarequests - https://phabricator.wikimedia.org/T233717 (10fdans) [15:57:17] 10Analytics, 10Analytics-Kanban, 10Services (watching): Create mediarequests top files AQS endpoint - https://phabricator.wikimedia.org/T233716 (10fdans) [15:57:39] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Services (watching): Add mediarequests per referer endpoint to AQS - https://phabricator.wikimedia.org/T232857 (10fdans) p:05Triage→03High [15:57:47] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Services (watching): Add cassandra loading job for top mediarequests - https://phabricator.wikimedia.org/T233717 (10fdans) p:05Triage→03High [15:59:21] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: Drop page create event data on mysql - https://phabricator.wikimedia.org/T233892 (10fdans) p:05Triage→03High [15:59:56] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: Drop page create event data on mysql - https://phabricator.wikimedia.org/T233892 (10fdans) +1 to deletion [16:00:00] elukey: I almost forgot! Happy Birthday :) [16:00:26] 10Analytics: Check home leftovers of smalyshev - https://phabricator.wikimedia.org/T231861 (10EBernhardson) I don't see anything in here that we would be losing, this is safe to delete. [16:01:36] milimetric: ahahaha thanks <3 [16:02:50] elukey: cumpleaniossssss felisssssss cumpleaniosss felisssss [16:05:03] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Performance-Team: Drop Navigationtiming data entirely from mysql storage? - https://phabricator.wikimedia.org/T233891 (10fdans) Dropping 2017 and 2018 data (hive records) [16:05:21] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Performance-Team: Drop Navigationtiming data entirely from mysql storage? - https://phabricator.wikimedia.org/T233891 (10fdans) p:05Triage→03High [16:08:54] (it was last friday but thanks all :) [16:16:40] 10Analytics: Enable geoeditors_daily deletion - https://phabricator.wikimedia.org/T234238 (10Nuria) [16:16:58] 10Analytics: Enable geoeditors_daily deletion - https://phabricator.wikimedia.org/T234238 (10Milimetric) p:05Triage→03High [16:17:04] 10Analytics, 10Analytics-Kanban: Enable geoeditors_daily deletion - https://phabricator.wikimedia.org/T234238 (10Milimetric) [16:22:27] 10Analytics, 10Cloud-Services, 10Developer-Advocacy (Oct-Dec 2019): Develop a tool or integrate feature in existing one to visualize WMCS edits data - https://phabricator.wikimedia.org/T226663 (10Milimetric) a:05srishakatux→03JAllemandou [16:22:41] 10Analytics, 10Analytics-Kanban, 10Cloud-Services, 10Developer-Advocacy (Oct-Dec 2019): Develop a tool or integrate feature in existing one to visualize WMCS edits data - https://phabricator.wikimedia.org/T226663 (10Milimetric) p:05Normal→03High [16:30:00] 10Analytics, 10Operations, 10Traffic, 10observability: Publish tls related info to webrequest via varnish - https://phabricator.wikimedia.org/T233661 (10Milimetric) p:05Normal→03High [16:31:25] 10Analytics, 10EventBus, 10Product-Analytics: Review draft Modern Event Platform schema guidelines - https://phabricator.wikimedia.org/T233329 (10Milimetric) a:05Neil_P._Quinn_WMF→03Ottomata [16:32:44] 10Analytics, 10Analytics-Kanban: shorten the time it takes to move files from hadoop to dump hosts by Kerberizing the dump hosts - https://phabricator.wikimedia.org/T234229 (10Milimetric) a:03Milimetric [16:58:56] * elukey off! [17:19:55] 10Analytics, 10Desktop Improvements, 10EventBus, 10Readers-Web-Backlog (Kanbanana-2019-20-Q2): [SPIKE 8hrs] How will the changes to eventlogging affect desktop improvements - https://phabricator.wikimedia.org/T233824 (10MBinder_WMF) [17:26:41] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: High volume mediawiki analytics events camus import is lagging - https://phabricator.wikimedia.org/T233718 (10EBernhardson) Data looks to have backfilled appropriately, thanks! [17:27:00] Is this normal? [17:27:01] ladsgroup@stat1007:~$ mysql -h s8-analytics-replica.eqiad.wmnet -P 3318 -A [17:27:01] ERROR 2013 (HY000): Lost connection to MySQL server at 'reading authorization packet', system error: 2 "No such file or directory" [17:49:45] Amir1: there is a ticket for stat1007, something going on on mysql [17:50:11] Amir1: https://phabricator.wikimedia.org/T234160 [17:51:04] thanks. As long as it's on the radar it's fine [18:12:38] nuria, re. reportupdater jobs migrated to hive [18:13:04] right now all the reports in a report folder like reportupdater-queries/flow [18:13:14] are executed together in the same machine [18:13:29] and they are either all mysql or all hive [18:14:02] in the case of flow, they are all mysql until now [18:14:49] should I split that folder 'flow' into say... flow-mysql and flow-hive? [18:14:54] milimetric, cc ^ [18:16:49] or should I change RU so that it accepts a flag (-e type:script), that only executes a subset of the scripts, so that we can configure that frok puppet? [18:17:18] 10Analytics, 10Research: Parse wikidumps and extract redirect information for 1 small wiki, romanian - https://phabricator.wikimedia.org/T232123 (10leila) @JAllemandou thanks for expanding. I've moved this task to the Done lane in the Research board. I'll also remove MGerlach as the assignee per what you descr... [18:17:19] meaning: -e type:sql executes the reports that are configured as such [18:17:29] 10Analytics, 10Research: Parse wikidumps and extract redirect information for 1 small wiki, romanian - https://phabricator.wikimedia.org/T232123 (10leila) a:05MGerlach→03None [18:17:32] and: -e type:script the ones that are hive scripts [18:17:49] mforns: right now it always takes /config.yaml or can you specify a different config file, I forget [18:18:03] one option would be to have config-mysql.yaml and config-hive.yaml [18:18:05] you can specify it [18:18:09] 10Analytics, 10Research: Parse wikidumps and extract redirect information for 1 small wiki, romanian - https://phabricator.wikimedia.org/T232123 (10leila) @MGerlach congratulations on finishing your first task. :) [18:18:17] yea! that's a great idea :D [18:18:31] k, because it's kind of temporary, so it's ok if it's a little messy [18:18:34] good milimetric will do that [18:18:40] thanks [18:19:05] ofc [19:17:10] 10Analytics, 10Research-Backlog, 10Wikidata: Copy Wikidata dumps to HDFs - https://phabricator.wikimedia.org/T209655 (10GoranSMilovanovic) @JAllemandou Would it be possible to have another update (beyond the most recent `20190603`) of the dump in hdfs? I would like to present some of the analytical systems b... [19:20:29] 10Analytics, 10Fundraising-Backlog, 10Operations, 10SRE-Access-Requests: Banner History and page view data access for fundraising analysts - Jerrie and Erin - https://phabricator.wikimedia.org/T233636 (10herron) 05Open→03Resolved a:03herron `jkumalah` has been added to ldap group `wmf`, `eyener` has... [19:22:09] 10Analytics, 10Fundraising-Backlog, 10Operations, 10SRE-Access-Requests: Banner History and page view data access for fundraising analysts - Jerrie and Erin - https://phabricator.wikimedia.org/T233636 (10herron) [19:59:49] 10Analytics, 10Fundraising-Backlog, 10Operations, 10SRE-Access-Requests: Banner History and page view data access for fundraising analysts - Jerrie and Erin - https://phabricator.wikimedia.org/T233636 (10Nuria) Please make sure you can access https://turnilo.wikimedia.org [20:09:39] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Performance-Team (Radar): Drop Navigationtiming data entirely from mysql storage? - https://phabricator.wikimedia.org/T233891 (10Gilles) [20:12:16] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Performance-Team (Radar): Drop Navigationtiming data entirely from mysql storage? - https://phabricator.wikimedia.org/T233891 (10Gilles) @fdans do you mean that you're going to drop SQL records where we have equivalent records in Hadoop by compar... [20:14:02] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Performance-Team (Radar): Drop Navigationtiming data entirely from mysql storage? - https://phabricator.wikimedia.org/T233891 (10Nuria) @Gilles Correction: in this case data drop is only 2018 year [20:31:09] 10Analytics, 10Fundraising-Backlog, 10Operations, 10SRE-Access-Requests: Banner History and page view data access for fundraising analysts - Jerrie and Erin - https://phabricator.wikimedia.org/T233636 (10jkumalah) @Nuria I have access. Thank you! [21:05:27] !log rolling restart of hdfs namenode and hdfs resourcemanager to take presto proxy user settings [21:05:29] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:43:11] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Performance-Team, 10Patch-For-Review: EventLogging needs to enque events to avoid draining users' battery on mobile - https://phabricator.wikimedia.org/T225578 (10Krinkle) [21:43:52] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team Legacy (Watching / External), 10Services (watching): Modern Event Platform: Schema Registry - https://phabricator.wikimedia.org/T201063 (10Milimetric) While I would love to argue for a .NET deployment, so me and everyone I love can e... [21:49:08] (03CR) 10Milimetric: "Thanks very much, joal" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/532974 (https://phabricator.wikimedia.org/T215655) (owner: 10Joal) [22:20:14] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team Legacy (Watching / External), 10Services (watching): Modern Event Platform: Schema Registry - https://phabricator.wikimedia.org/T201063 (10Ottomata) Just the UI