[01:18:07] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [01:23:07] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [06:08:54] (03PS1) 10Conniecc1: add translation_difficulty_levelto allow list [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1207489 [09:35:28] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07OKR-Work, 13Patch-For-Review: Provide a Spark production access for dbt with Airflow - https://phabricator.wikimedia.org/T410017#11391174 (10JAllemandou) [12:01:29] 06Data-Engineering, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 13Patch-For-Review, 07Sustainability (Incident Followup): Add monitoring / alerting on the number of MySQL queries done by Hive - https://phabricator.wikimedia.org/T410528#11391525 (10GGoncalves-WMF) Sorry I'm a bit late to this - I just ca... [12:04:59] 10Data-Engineering-Roadmap, 07Epic, 07OKR-Work: Analyze JA3N data and generate JA3N-UA table - https://phabricator.wikimedia.org/T409577#11391555 (10mforns) a:05mforns→03Hghani [12:05:41] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Data-Engineering-Roadmap, 07Epic, 07OKR-Work: Analyze JA3N data and generate JA3N-UA table - https://phabricator.wikimedia.org/T409577#11391558 (10mforns) [14:01:27] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Make the sic_url_identifier index unique and remove the default from the sic_url_identifier column in cusi_case on WMF wikis - https://phabricator.wikimedia.org/T410631 (10Dreamy_Jazz) 03NEW [14:04:36] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Make the sic_url_identifier index unique and remove the default from the sic_url_identifier column in cusi_case on WMF wikis - https://phabricator.wikimedia.org/T410631#11392091 (10Ladsgroup) Can this go now? [14:05:22] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Make the sic_url_identifier index unique and remove the default from the sic_url_identifier column in cusi_case on WMF wikis - https://phabricator.wikimedia.org/T410631#11392103 (10Dreamy_Jazz) Just checking production DBs to ensure that the values ar... [14:06:43] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07OKR-Work, 13Patch-For-Review: Provide a Spark production access for dbt with Airflow - https://phabricator.wikimedia.org/T410017#11392106 (10BTullis) [14:08:16] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Make the sic_url_identifier index unique and remove the default from the sic_url_identifier column in cusi_case on WMF wikis - https://phabricator.wikimedia.org/T410631#11392108 (10Dreamy_Jazz) So far we have rows in a few production wikis. I checked... [14:09:52] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Make the sic_url_identifier index unique and remove the default from the sic_url_identifier column in cusi_case on WMF wikis - https://phabricator.wikimedia.org/T410631#11392115 (10Dreamy_Jazz) [14:10:28] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Make the sic_url_identifier index unique and remove the default from the sic_url_identifier column in cusi_case on WMF wikis - https://phabricator.wikimedia.org/T410631#11392120 (10Dreamy_Jazz) [14:15:10] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Make the sic_url_identifier index unique and remove the default from the sic_url_identifier column in cusi_case on WMF wikis - https://phabricator.wikimedia.org/T410631#11392145 (10Ladsgroup) a:03Ladsgroup Ran it on testwiki and test2wiki. I'll wait... [14:15:18] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Make the sic_url_identifier index unique and remove the default from the sic_url_identifier column in cusi_case on WMF wikis - https://phabricator.wikimedia.org/T410631#11392148 (10Ladsgroup) p:05Triage→03Medium [14:24:01] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Make the sic_url_identifier index unique and remove the default from the sic_url_identifier column in cusi_case on WMF wikis - https://phabricator.wikimedia.org/T410631#11392163 (10Ladsgroup) 05Open→03Resolved [14:26:37] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Make the sic_url_identifier index unique and remove the default from the sic_url_identifier column in cusi_case on WMF wikis - https://phabricator.wikimedia.org/T410631#11392168 (10Dreamy_Jazz) Thanks! [15:37:22] (03CR) 10Aleksandar Mastilovic: Add HQL for pageviews_per_editor and pageview_top_pages_per_editor (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1206879 (https://phabricator.wikimedia.org/T410289) (owner: 10Snwachukwu) [15:47:33] (03PS4) 10Aleksandar Mastilovic: Add HQL for pageviews_per_editor and pageview_top_pages_per_editor [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1206879 (https://phabricator.wikimedia.org/T410289) (owner: 10Snwachukwu) [15:51:36] !log Starting refinery deploy regular deployment train for 4df475f3ab6ac9f43f9cd9f63008496fbc9e7c16 [15:51:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:56:32] (03CR) 10Aleksandar Mastilovic: Add HQL for pageviews_per_editor and pageview_top_pages_per_editor (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1206879 (https://phabricator.wikimedia.org/T410289) (owner: 10Snwachukwu) [16:00:38] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Update MediaWiki Content History SLO draft for SRE review - https://phabricator.wikimedia.org/T401892#11392780 (10APizzata-WMF) updated the document: https://wikitech.wikimedia.org/wiki/SLO/MediaWiki_Content_History_Table [16:01:09] (03PS5) 10Aleksandar Mastilovic: Add HQL for pageviews_per_editor and pageview_top_pages_per_editor [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1206879 (https://phabricator.wikimedia.org/T410289) (owner: 10Snwachukwu) [16:03:17] (03CR) 10Aleksandar Mastilovic: [V:03+2 C:03+2] Add HQL for pageviews_per_editor and pageview_top_pages_per_editor (037 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1206879 (https://phabricator.wikimedia.org/T410289) (owner: 10Snwachukwu) [16:11:18] 06Data-Engineering, 06Experimentation Lab, 10MediaWiki-extensions-EventLogging, 07Epic, 07Essential-Work: [EPIC] Deprecate and remove mw.eventLog.id - https://phabricator.wikimedia.org/T408177#11392830 (10JVanderhoop-WMF) [16:12:39] 06Data-Engineering, 06Experimentation Lab, 10MediaWiki-extensions-EventLogging, 07Essential-Work: Remove mw.eventLog.id - https://phabricator.wikimedia.org/T408179#11392881 (10JVanderhoop-WMF) [16:16:41] (03PS6) 10Aleksandar Mastilovic: Add HQL for pageviews_per_editor and pageview_top_pages_per_editor [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1206879 (https://phabricator.wikimedia.org/T410289) (owner: 10Snwachukwu) [16:19:10] !log Finished refinery deploy regular deployment train for 4df475f3ab6ac9f43f9cd9f63008496fbc9e7c16 [16:19:12] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:21:48] 06Data-Engineering, 06Experimentation Lab, 10Event-Platform: Client-side error logging should use Elastic Common Schema (ECS) fields when possible - https://phabricator.wikimedia.org/T267602#11392972 (10Milimetric) [16:38:44] 06Data-Engineering, 06Experimentation Lab, 10Event-Platform: Source geolocation directly rather than using IP in schema - https://phabricator.wikimedia.org/T290014#11393133 (10Milimetric) worth thinking about in our new context [16:45:33] btullis: sorry about the delay on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1199763 [16:45:48] that's my bad. I haven't found the time and while not an excuse, I will take care of it today [16:48:07] 06Data-Engineering, 10DiscussionTools, 10MediaWiki-extensions-EventLogging, 07Wikimedia-production-error: PHP Deprecated: Invalid characters passed for attempted conversion, these have been ignored - https://phabricator.wikimedia.org/T410668#11393291 (10brennen) [16:50:28] 06Data-Engineering, 06Experimentation Lab, 10MediaWiki-extensions-EventLogging, 07Essential-Work, 07Technical-Debt: Stop adding user-agent details to http.request_header.user-agent directly via EventLogging - https://phabricator.wikimedia.org/T408719#11393320 (10JVanderhoop-WMF) [18:08:05] !log Test Kitchen edge-unique experiments (poll 1) - adds: logged-out-retention-round2, fy25-26-we-4-2-hcaptcha-editing, image-browsing-enwiki, fy2025-26-we3.1-image-browsing-ab-test, hcaptcha-on-french-wikipedia; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [18:08:07] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:08:15] !log Test Kitchen mw-user experiment (poll 1) - adds: we-3-3-4-reading-list-test1, we-3-3-4-reading-list-test1-en, growthexperiments-get-started-notification; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [18:08:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:35:13] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Druid mediawiki_history_reduced changes - https://phabricator.wikimedia.org/T406069#11393933 (10JAllemandou) Here is the proposed solution we came with in today's meeting: * We are... [19:58:54] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Implement a new pipeline and table with reconciled historical revision data - https://phabricator.wikimedia.org/T410688 (10xcollazo) 03NEW [19:59:07] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Druid mediawiki_history_reduced changes - https://phabricator.wikimedia.org/T406069#11394002 (10xcollazo) [19:59:08] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Implement a new pipeline and table with reconciled historical revision data - https://phabricator.wikimedia.org/T410688#11394001 (10xcollazo) [20:12:17] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Implement a new pipeline and table with reconciled historical revision data - https://phabricator.wikimedia.org/T410688#11394042 (10xcollazo) Meta: Renamed relevant Gitlab project from: https://gitlab.wikimedia.org/repos/data-engineering/dumps/mediawi... [20:17:01] ^^^^^^ looks like some reboots in wmcs. the last log entry for the previous job was from 2025-11-20 18:00:02 (and 1800 was the last modified time of the file) and then after apparent reboots, the restarted job's new log's first entry was 2025-11-20 18:07:40 [21:53:35] 06Data-Engineering, 06Data-Platform-SRE, 06Movement-Insights, 07Epic: Provide a dbt-core development environment and production setup in the data-platform - https://phabricator.wikimedia.org/T406764#11394249 (10Ahoelzl) Moving this to Data Engineering Roadmap board [21:56:27] 10Data-Engineering-Roadmap, 06Data-Platform-SRE, 06Movement-Insights, 07Epic: Provide a dbt-core development environment and production setup in the data-platform - https://phabricator.wikimedia.org/T406764#11394270 (10Ahoelzl) [23:15:27] 06Data-Engineering, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Drop rc_type from recentchanges in wmf production - https://phabricator.wikimedia.org/T410531#11394520 (10Zabe) 05Stalled→03Open @Marostegui this is can be done now [23:41:03] 06Data-Engineering, 10Pageviews-Anomaly, 07SecTeam-Processed, 07Security: Views data integrity compromised by entity running up fake views - https://phabricator.wikimedia.org/T366554#11394565 (10Stevietheman) To follow up, I haven't seen any Wikipedia ads in The Verge newsletters I've received, so I am una...