[01:27:15] (03CR) 10Aleksandar Mastilovic: [C:03+2] "LGTM apart from one nitpick - you can choose if you want to implement it or not." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196049 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [01:40:41] (03Merged) 10jenkins-bot: Update mediawiki_history job for rev_sha1 DB removal [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196049 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [08:45:52] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE, 10Event-Platform: Avoid accepting Kafka messages with whacky timestamps - https://phabricator.wikimedia.org/T282887#11275507 (10JMonton-WMF) After some conversations, we have decide to not continue with this ticket until the cl... [08:54:43] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Create dbt folder structure - https://phabricator.wikimedia.org/T407322 (10JMonton-WMF) 03NEW [09:55:48] (03CR) 10Joal: "Some changes for the SQL part. Almost there. I can help with those tonight." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1194951 (https://phabricator.wikimedia.org/T406263) (owner: 10Aleksandar Mastilovic) [11:32:08] (03CR) 10Joal: "2 nits - I have not reviewed every detail, but overall looks good. If you're happy to use my sha1 implementation, please do :)" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1195330 (https://phabricator.wikimedia.org/T384945) (owner: 10Xcollazo) [12:58:59] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 3 others: Global Editor Metrics - HTTP API endpoints - https://phabricator.wikimedia.org/T405041#11276523 (10Ottomata) We are waffling bikeshedding around 'user' v... [13:08:51] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 3 others: Global Editor Metrics - HTTP API endpoints - https://phabricator.wikimedia.org/T405041#11276568 (10Ottomata) Indeed! If we could refactor AQS and make ed... [13:13:49] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 3 others: Global Editor Metrics - HTTP API endpoints - https://phabricator.wikimedia.org/T405041#11276591 (10Ottomata) Looking a bit more at existent Analytics API... [13:29:50] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Create a new gitlab repository for use with dbt - https://phabricator.wikimedia.org/T406765#11276663 (10Ottomata) Suggestion: instead of naming the repo 'dbt' which looks a bit more like a fork of 'dbt', name it: data-engineering/dbt-jobs Or somethin... [13:31:36] 06Data-Engineering, 06Data-Platform-SRE, 10observability, 06serviceops-radar, and 3 others: Upgrade Kafka to from 1.x to later version - https://phabricator.wikimedia.org/T300102#11276703 (10Ottomata) [13:37:14] 06Data-Engineering, 06Data-Platform-SRE, 10observability, 06serviceops-radar, and 3 others: Upgrade Kafka to from 1.x to later version - https://phabricator.wikimedia.org/T300102#11276724 (10brouberol) I don't know why I haven't posted it here, but I should have posted this [Kafka upgrade plan](https://doc... [13:38:32] 06Data-Engineering, 06Data-Platform-SRE, 10observability, 06serviceops-radar, and 3 others: Upgrade Kafka to from 1.x to later version - https://phabricator.wikimedia.org/T300102#11276731 (10elukey) We should form a working group to get this done, maybe in two quarters starting from the next one? One for t... [13:42:51] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11276746 (10Ottomata) Thanks @achou I really... [13:53:16] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11276791 (10dcausse) >>! In T401021#11274339... [13:58:12] 06Data-Engineering, 06Data-Platform-SRE, 10observability, 06serviceops-radar, and 3 others: Upgrade Kafka to from 1.x to later version - https://phabricator.wikimedia.org/T300102#11276818 (10brouberol) Agreed [14:01:49] (03CR) 10Xcollazo: "> If you're happy to use my sha1 implementation, please do 😊" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1195330 (https://phabricator.wikimedia.org/T384945) (owner: 10Xcollazo) [14:17:32] 06Data-Engineering, 10observability, 10Observability-Logging: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11277037 (10colewhite) a:03herron [14:18:06] 06Data-Engineering, 10Observability-Logging: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11277041 (10colewhite) [14:34:16] (03PS1) 10Joal: Fix mediawiki_history bug from previous patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196469 (https://phabricator.wikimedia.org/T406000) [14:43:07] 06Data-Engineering, 10Observability-Logging, 06serviceops: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11277182 (10Ottomata) Tagging #serviceops for kafka main rebalancing. [14:45:57] 06Data-Engineering, 10Observability-Logging, 06serviceops: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11277204 (10Clement_Goubert) `kafka-main` hosts also need to be rebooted soon-ish, is this something that should be done before, after, or it doesn't matter? [14:58:16] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Add user_central_id to mediawiki_content_history_v1 (and mediawiki_content_current_v1) - https://phabricator.wikimedia.org/T406515#11277273 (10Ottomata) Thanks to @tchin, `user_central_id` is now in `media... [14:58:59] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Add user_central_id to mediawiki_content_history_v1 (and mediawiki_content_current_v1) - https://phabricator.wikimedia.org/T406515#11277285 (10Ottomata) [15:01:11] (03CR) 10Aqu: [C:03+1] Fix mediawiki_history bug from previous patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196469 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [15:03:57] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11277317 (10Ottomata) > vs setting up someth... [15:08:05] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11277351 (10Ottomata) BTW, if we ever do {T2... [15:16:18] (03CR) 10Joal: MW Dumper: Add support for Multi-content Revisions (MCR) (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1195330 (https://phabricator.wikimedia.org/T384945) (owner: 10Xcollazo) [15:18:31] !log Deploying Refinery at 94efa6e8221602a331c19c39ea909eeaa90d98b4 for T405533 unique devices domains [15:18:34] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:18:35] T405533: Unique devices data uses non-standard domains for Wikidata, Wikifunctions, and MediaWiki.org - https://phabricator.wikimedia.org/T405533 [15:27:51] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE, 10Event-Platform: Avoid accepting Kafka messages with whacky timestamps - https://phabricator.wikimedia.org/T282887#11277463 (10JMonton-WMF) 05In progress→03Open [15:36:17] (03CR) 10Xcollazo: [C:03+2] Fix mediawiki_history bug from previous patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196469 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [15:47:22] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Wikimedia Enterprise, 10Wikimedia Enterprise - Content Integrity, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work: Implement an Airflow operator for moving data from point A t... - https://phabricator.wikimedia.org/T405360#11277591 [15:49:46] (03Merged) 10jenkins-bot: Fix mediawiki_history bug from previous patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196469 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [15:54:03] 06Data-Engineering, 10Observability-Logging, 06serviceops: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11277638 (10brouberol) Before, after, but not in-between, I'd say. [16:01:04] 06Data-Engineering, 06Data-Platform-SRE, 10Observability-Logging, 06serviceops: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11277673 (10Ottomata) [16:01:07] 06Data-Engineering, 06Data-Platform-SRE, 10Observability-Logging, 06serviceops: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11277677 (10Ottomata) p:05Triage→03Medium [16:01:21] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE, 10Observability-Logging, 06serviceops: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11277679 (10Ottomata) [16:05:56] !log Finished deploying Refinery at 94efa6e8221602a331c19c39ea909eeaa90d98b4 for T405533 unique devices domains [16:05:59] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:06:00] T405533: Unique devices data uses non-standard domains for Wikidata, Wikifunctions, and MediaWiki.org - https://phabricator.wikimedia.org/T405533 [16:10:33] (03PS1) 10Joal: Fix mediawiki_history bug from previous patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196485 (https://phabricator.wikimedia.org/T406000) [16:14:19] 06Data-Engineering, 10Dumps-Generation, 06Wikibase Reuse Team, 10Wikidata, and 3 others: No Wikidata dumps for Week 40 of 2025 (recurring issue) - https://phabricator.wikimedia.org/T406429#11277786 (10Ottomata) @BTullis how to you think #data-engineering should best help here? Who knows the most and might... [16:23:34] (03CR) 10Xcollazo: [C:03+1] "LGTM" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196485 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [16:34:53] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Wikibase Pingback, 10WMF-General-or-Unknown, and 2 others: PHP Warning: EventLoggingLegacyConverter: Failed proxying legacy EventLogging event query string to WMF Event Platform JSON: UnexpectedV... - https://phabricator.wikimedia.org/T406763#11277938 [16:35:17] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Wikibase Pingback, 10WMF-General-or-Unknown, and 2 others: PHP Warning: EventLoggingLegacyConverter: Failed proxying legacy EventLogging event query string to WMF Event Platform JSON: UnexpectedV... - https://phabricator.wikimedia.org/T406763#11277945 [16:43:36] (03CR) 10Aleksandar Mastilovic: Add user_central_id to the mediawiki_history dataset(s) (036 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1194951 (https://phabricator.wikimedia.org/T406263) (owner: 10Aleksandar Mastilovic) [16:49:39] (03PS5) 10Aleksandar Mastilovic: Add user_central_id to the mediawiki_history dataset(s) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1194951 (https://phabricator.wikimedia.org/T406263) [16:54:27] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07OKR-Work: SDS 1.3.2 Conduct Analysis on Alerting for changes in automated traffic distribution - https://phabricator.wikimedia.org/T406882#11278076 (10Snwachukwu) A quick summary of 5 weeks data: History table used: Pageview hourly Proposed Monitor... [16:57:51] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07OKR-Work: SDS 1.3.2 Conduct Analysis on Alerting for changes in automated traffic distribution - https://phabricator.wikimedia.org/T406882#11278104 (10Snwachukwu) [16:59:15] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07OKR-Work: SDS 1.3.2 Conduct Analysis on Alerting for changes in automated traffic distribution - https://phabricator.wikimedia.org/T406882#11278117 (10Snwachukwu) 05Open→03In progress [17:12:53] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 3 others: Global Editor Metrics - HTTP API endpoints - https://phabricator.wikimedia.org/T405041#11278192 (10Ottomata) pageviews/aggregate/per-editor endpoint is r... [17:17:17] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11278239 (10achou) > Q: "feed the paragrap... [17:33:58] (03PS4) 10Xcollazo: MW Dumper: Add support for Multi-content Revisions (MCR) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1195330 (https://phabricator.wikimedia.org/T384945) [17:36:56] (03CR) 10Xcollazo: "Modified 1196049 a bit to be able to use it outside of `Row`s context. LMK what you think." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1195330 (https://phabricator.wikimedia.org/T384945) (owner: 10Xcollazo) [17:52:26] (03CR) 10Xcollazo: [C:03+2] Fix mediawiki_history bug from previous patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196485 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [17:55:52] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 3 others: Global Editor Metrics - HTTP API endpoints - https://phabricator.wikimedia.org/T405041#11278524 (10Sfaci) > Looking a bit more at existent Analytics API... [18:05:32] (03Merged) 10jenkins-bot: Fix mediawiki_history bug from previous patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196485 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [18:11:23] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 3 others: Global Editor Metrics - HTTP API endpoints - https://phabricator.wikimedia.org/T405041#11278588 (10Ottomata) Discussed with @mforns today, and we aren't... [18:20:45] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11278612 (10Ottomata) > We want to move to F... [18:25:50] (03CR) 10Joal: [C:03+1] "Let's test this!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1194951 (https://phabricator.wikimedia.org/T406263) (owner: 10Aleksandar Mastilovic) [19:10:58] (03PS6) 10Aleksandar Mastilovic: Add user_central_id to the mediawiki_history dataset(s) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1194951 (https://phabricator.wikimedia.org/T406263) [19:18:26] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11278787 (10achou) Regarding the timeline, G... [19:43:16] (03PS1) 10Joal: Improve mediawiki_history previous patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196518 (https://phabricator.wikimedia.org/T406000) [19:43:52] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Serve mobile and desktop variants through the same URL (unified mobile routing) - https://phabricator.wikimedia.org/T214998#11278865 (10TheDJ) One more thing that I think we should consider in this entire story.. navboxes. As go... [19:52:10] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.6 First analysis review - https://phabricator.wikimedia.org/T407103#11278890 (10Hghani) Thanks for reviewing, I have summarised the observations so far below and I have added a new [[ https://gitlab.wikimedia.org/hghani/movement-insights-re... [19:52:49] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.6 First analysis review - https://phabricator.wikimedia.org/T407103#11278893 (10Hghani) [19:52:55] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.6 First analysis review - https://phabricator.wikimedia.org/T407103#11278894 (10Hghani) [20:10:16] (03CR) 10Joal: [C:03+1] "Sounds good to me :) Thanks for this!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1195330 (https://phabricator.wikimedia.org/T384945) (owner: 10Xcollazo) [20:13:20] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Serve mobile and desktop variants through the same URL (unified mobile routing) - https://phabricator.wikimedia.org/T214998#11278940 (10Izno) That was done some time ago, and isn't particularly related to this task. {T198949} Th... [20:17:55] (03CR) 10Xcollazo: [C:03+2] "Thanks for the review @joal@wikimedia.org!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1195330 (https://phabricator.wikimedia.org/T384945) (owner: 10Xcollazo) [20:27:37] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07OKR-Work: SDS 1.3.2 Conduct Analysis on Alerting for changes in automated traffic distribution - https://phabricator.wikimedia.org/T406882#11279023 (10Snwachukwu) I applied the suggested thresholds above to the old incident data found in wmf.pagevi... [20:29:56] (03Merged) 10jenkins-bot: MW Dumper: Add support for Multi-content Revisions (MCR) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1195330 (https://phabricator.wikimedia.org/T384945) (owner: 10Xcollazo) [20:34:27] I have a zookeeper server (installed by puppet) and a zookeeper client, kazoo (python, comes with zuul). They have trouble communicating and I finally found out this is about the "jute.maxbuffer" size. There is constantly a Java exception. "Len error". message is too large to process. A standard fix would be to increase the jute.maxbuffer. So I put a much larger value in the zookeeper [20:34:33] server config.. restarted it.. but it's like that changes nothing. the error message still and always contains. "length is greater than jute.maxbuffer=1048575 which is the default of only 1 Megabyte. There is also no java arg for it in the command line. I just dont get why I cant change it with the normal config. [21:14:35] ah.. it looks like I have to put a JVMFLAGS= line into /etc/zookeeper/conf/environment .. not in the zoo.cfg [21:16:38] or not.. still doesnt change at all. :/ checking again later [21:27:33] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Commons Impact Metrics has no data for September snapshot - https://phabricator.wikimedia.org/T406509#11279322 (10mforns) After some troubleshooting I saw that, when we added the linktarget table as a datasource for Commons Impact Metrics, we forgot to... [21:32:13] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07OKR-Work: SDS 1.3.2 Conduct Analysis on Alerting for changes in automated traffic distribution - https://phabricator.wikimedia.org/T406882#11279327 (10Hghani) @Snwachukwu Thanks for testing these methods. The pageview_hourly_backup_2025 data sta... [21:48:32] (03PS7) 10Aleksandar Mastilovic: Add user_central_id to the mediawiki_history dataset(s) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1194951 (https://phabricator.wikimedia.org/T406263) [23:41:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:46:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly