[00:31:50] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07Essential-Work, 10Event-Platform, 13Patch-For-Review: Upgrade mediawiki-event-enrichment jobs to >= Flink 1.20.2 and Java 17 - https://phabricator.wikimedia.org/T408918#11711934 (10Ottomata) Thank you! [00:58:52] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:58:52] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [03:54:16] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API, 10Tool-Pageviews: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857#11712004 (10Hawkeye7) Massviews was working on 7 December 2023. I am sure it was working in 2024. [04:58:52] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [04:58:52] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [08:22:18] !log Test Kitchen edge-unique experiments (poll 260174) - adds: synth-aa-test-traffic-impact-1; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [08:22:20] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:22:39] !log Test Kitchen edge-unique experiments (poll 260175) - adds: synth-aa-test-traffic-impact-3, synth-aa-test-traffic-impact-2; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [08:22:40] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:58:52] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [08:58:52] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [09:05:51] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 07Essential-Work, 06Test Kitchen (Experiment Platform Sprint 21): Deluge of inactionable console warnings - https://phabricator.wikimedia.org/T419481#11712369 (10Sfaci) [09:16:24] 06Data-Engineering, 06ServiceOps new, 10ServiceOps-Mediawiki, 10ServiceOps-Upgrades-Hardware, 07Schema-change: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980#11712392 (10MLechvien-WMF) p:05Triage→03High [10:41:03] 06Data-Engineering, 06Data-Persistence, 06ServiceOps new, 10ServiceOps-Mediawiki, and 2 others: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980#11712880 (10MLechvien-WMF) [11:13:37] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [11:13:37] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [11:18:37] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [11:18:37] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [11:51:35] 06Data-Engineering, 06Data-Persistence, 06ServiceOps new, 10ServiceOps-Mediawiki, and 2 others: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980#11713167 (10Ladsgroup) Two notes: - I think this needs to be done gradually. We have never done this in production and it c... [12:38:27] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Technical-Debt: Deprecate and remove mw.eventLog.submitClick() - https://phabricator.wikimedia.org/T415210#11713389 (10Sfaci) [12:43:26] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 07Essential-Work, 06Test Kitchen (Experiment Platform Sprint 21): Deluge of inactionable console warnings - https://phabricator.wikimedia.org/T419481#11713412 (10Sfaci) a:03Sfaci [13:00:11] 14Analytics, 06Data-Engineering, 10Data-Engineering-Wikistats: Wikidata unique devices statistics are obviously wrong - https://phabricator.wikimedia.org/T420210 (10TTWIDEE) 03NEW [13:07:53] 06Data-Engineering, 10Data-Services, 06cloud-services-team (FY2025/2026-Q3-Q4), 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11713534 (10BTullis) >>! In T417492#11703318, @fnegri wrote: > I ran th... [13:08:44] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: Audit and fix observability (logging and metrics) for pyflink jobs - https://phabricator.wikimedia.org/T418996#11713538 (10Ottomata) [13:08:53] 06Data-Engineering, 10Data-Services, 06cloud-services-team (FY2025/2026-Q3-Q4), 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11713539 (10fnegri) 05In progress→03Resolved a:03fnegri [13:09:56] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Research-engineering, 06Research-Freezer, 10Event-Platform, 13Patch-For-Review: Productionized Edit Types - https://phabricator.wikimedia.org/T351225#11713546 (10Ottomata) [13:10:02] 06Data-Engineering, 10Data-Services, 06cloud-services-team (FY2025/2026-Q3-Q4), 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11713550 (10fnegri) a:05fnegri→03Zabe [13:13:03] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Technical-Debt: Deprecate and remove mw.eventLog.submitClick() - https://phabricator.wikimedia.org/T415210#11713571 (10seanleong-WMDE) [13:13:40] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Technical-Debt: Deprecate and remove mw.eventLog.submitClick() - https://phabricator.wikimedia.org/T415210#11713575 (10seanleong-WMDE) Update: T417510 is scheduled to go on prod by 19 March for all wikis... [13:23:09] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Research-engineering, 06Research-Freezer, 10Event-Platform, 13Patch-For-Review: Productionized Edit Types - https://phabricator.wikimedia.org/T351225#11713597 (10Ottomata) [13:34:04] 06Data-Engineering, 10Data-Engineering-Wikistats, 10Wikidata: Wikidata unique devices statistics are obviously wrong - https://phabricator.wikimedia.org/T420210#11713639 (10A_smart_kitten) [13:40:10] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Research-engineering, 06Research-Freezer, 10Event-Platform, 13Patch-For-Review: Productionized Edit Types - https://phabricator.wikimedia.org/T351225#11713660 (10Ottomata) [14:03:03] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06MW-Interfaces-Team, 10Event-Platform: mediawiki.page_change.v1 event stream - Investigate mismatched meta.dt and dt (and rev_dt) fields - https://phabricator.wikimedia.org/T409105#11713771 (10Ottomata) [14:07:50] 06Data-Engineering, 10MediaWiki-extensions-WikimediaEvents, 10Event-Platform: Tests depending on EventBus fail when EventStreamConfig is not loaded - https://phabricator.wikimedia.org/T410122#11713783 (10Ottomata) 05Open→03Resolved @hashar thank you for you work! I did not follow this closely, but... [14:12:35] 06Data-Engineering, 10Event-Platform: mediawiki-event-enrichment page_html_change job should use X-MediaWiki-Render-Id header instead of etag - https://phabricator.wikimedia.org/T420217 (10Ottomata) 03NEW [14:13:50] 06Data-Engineering, 10Event-Platform: [NEEDS INVESTIGATION][BUG] eventutilities_python operator metrics - https://phabricator.wikimedia.org/T373112#11713844 (10Ottomata) [14:13:50] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: Audit and fix observability (logging and metrics) for pyflink jobs - https://phabricator.wikimedia.org/T418996#11713845 (10Ottomata) [14:15:28] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Content-Transform-Team, 06MW-Interfaces-Team, 10Event-Platform: Common event data model for data derived from parsed page revision html (and more!) - https://phabricator.wikimedia.org/T415158#11713853 (10Ottomata) p:05Triage→03High [14:17:03] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Research-engineering, 06Research-Freezer, 10Event-Platform, 13Patch-For-Review: Productionized Edit Types - https://phabricator.wikimedia.org/T351225#11713856 (10Ahoelzl) p:05Triage→03High [14:24:42] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Do multiple code and data clean ups for content tables - https://phabricator.wikimedia.org/T418754#11713906 (10xcollazo) >I do expect to pay a penalty when running table maintenance, so let's wait until that triggers before closing t... [14:24:59] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Do multiple code and data clean ups for content tables - https://phabricator.wikimedia.org/T418754#11713907 (10xcollazo) [14:25:40] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10AQS2.0, 13Patch-For-Review: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11713910 (10Snwachukwu) Thank you @Eevans . I have left you a comment on your patch [14:26:47] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Do multiple code and data clean ups for content tables - https://phabricator.wikimedia.org/T418754#11713914 (10xcollazo) As of now the only downstream consumer of `mediawiki_revision_history_v1 ` is [[ https://airflow.wikimedia.org/d... [14:27:18] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Modify legacy dumps to specify what artifacts are deprecated - https://phabricator.wikimedia.org/T416180#11713916 (10xcollazo) 05In progress→03Resolved [14:32:35] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635#11713949 (10Ladsgroup) We are about to go to maint freeze for the dc switchover, I'd wait a bit before starting to drop s... [14:42:34] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Technical-Debt: Deprecate and remove mw.eventLog.submitClick() - https://phabricator.wikimedia.org/T415210#11713979 (10Sfaci) [15:18:37] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:18:37] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:23:16] 06Data-Engineering, 10ChangeProp, 06MW-Interfaces-Team, 10Observability-Tracing, and 2 others: Implement tracing across changeprop-jobqueue, kafka, eventgate - https://phabricator.wikimedia.org/T395038#11714120 (10Scott_French) [15:23:41] 06Data-Engineering, 10ChangeProp, 06MW-Interfaces-Team, 10Observability-Tracing, and 2 others: Implement tracing across changeprop-jobqueue, kafka, eventgate - https://phabricator.wikimedia.org/T395038#11714128 (10Scott_French) [15:29:17] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10AQS2.0, 13Patch-For-Review: Alter AQS Cassandra tables in support of video plays endpoints - https://phabricator.wikimedia.org/T420008#11714173 (10Eevans) @Snwachukwu let me know when you're ready to have this applied to staging. [15:40:18] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API, 10Tool-Pageviews: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857#11714261 (10daniel) >>! In T219857#11704747, @MusikAnimal wrote: > It looks like the rate limiting... [15:54:34] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026-03-06 - 2026-03-27), 13Patch-For-Review: Deploy turnilo to dse-k8s-eqiad - https://phabricator.wikimedia.org/T416113#11714326 (10mforns) a:05JAllemandou→03None [15:57:23] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07Essential-Work, 10Event-Platform, 13Patch-For-Review: Upgrade mediawiki-event-enrichment jobs to >= Flink 1.20.3 and Java 17 - https://phabricator.wikimedia.org/T408918#11714343 (10mforns) [16:05:03] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635#11714406 (10FCeratto-WMF) @Ladsgroup ok, with the rollout to start after the freeze [16:07:51] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team: CentralAuth's localuser table contains many nulls and duplicate mappings - https://phabricator.wikimedia.org/T411116#11714414 (10APizzata-WMF) [16:16:48] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team: CentralAuth's localuser table contains many nulls and duplicate mappings - https://phabricator.wikimedia.org/T411116#11714477 (10APizzata-WMF) possible cause: https://issues.apache.org/jir... [16:18:14] !log Test Kitchen edge-unique experiments (poll 261578) - adds: none; removes: none; fields: share-highlight-baseline - xLab/MPIC/TK tips at https://w.wiki/FwuD [16:18:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:57:13] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: Linter for mediawiki-event-enrichment - https://phabricator.wikimedia.org/T418889#11714668 (10AKhatun_WMF) 05Open→03Resolved [16:57:50] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform, 13Patch-For-Review: Adatp HTML pipeline to the new diffs schema - https://phabricator.wikimedia.org/T419258#11714674 (10AKhatun_WMF) 05Open→03Resolved [17:06:00] !log Test Kitchen edge-unique experiments (poll 261719) - adds: growthexperiments-editattempt-anonwarning; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [17:06:01] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:10:56] 06Data-Engineering, 06Growth-Team, 10Image-Suggestions: Section Image Suggestions no longer available? - https://phabricator.wikimedia.org/T420244 (10Michael) 03NEW [17:28:37] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [17:28:37] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [17:33:37] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [17:33:37] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [17:48:25] 06Data-Engineering, 10Data Pipelines: Add user_central_id to mediawiki_history and mediawiki_history_reduced Hive tables - https://phabricator.wikimedia.org/T365648#11714962 (10Ottomata) user_central_id was added to MW History in this patch https://gerrit.wikimedia.org/r/c/analytics/refinery/+/1195348 [18:21:35] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform, 13Patch-For-Review: Fix mediawiki event enrichment to work with newest version of Blubber - https://phabricator.wikimedia.org/T406872#11715165 (10Ottomata) a:03JMonton-WMF [18:27:14] 06Data-Engineering, 10Wikidata, 10Wikidata-Query-Service, 10Event-Platform: WDQS sparql query event generator should not set meta.dt - https://phabricator.wikimedia.org/T405949#11715205 (10Ottomata) @BTracy-WMF @gmodena would love to resolve the tech debt on our side before we totally forget about it! Wou... [18:30:17] 06Data-Engineering, 10ChangeProp, 10EventStreams, 06MediaWiki-Engineering, and 15 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11715215 (10Ottomata) [18:30:48] 06Data-Engineering, 10Event-Platform: Upgrade eventstreams and eventstreams-internal to node22 - https://phabricator.wikimedia.org/T420257 (10Ottomata) 03NEW [18:40:54] 06Data-Engineering, 10Wikidata, 06Wikidata Platform Team, 10Wikidata-Query-Service, 10Event-Platform: WDQS sparql query event generator should not set meta.dt - https://phabricator.wikimedia.org/T405949#11715283 (10gmodena) [18:49:25] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API, 10Tool-Pageviews: 429 Too Many Requests hit despite making requests synchronously - https://phabricator.wikimedia.org/T219857#11715341 (10MusikAnimal) [18:52:51] 06Data-Engineering, 10Wikidata, 06Wikidata Platform Team, 10Wikidata-Query-Service, 10Event-Platform: WDQS sparql query event generator should not set meta.dt - https://phabricator.wikimedia.org/T405949#11715364 (10gmodena) >>! In T405949#11715205, @Ottomata wrote: > @BTracy-WMF @gmodena would love to re... [18:53:55] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API, 10Tool-Pageviews: massviews hits 429 Too Many Requests despite making requests synchronously - https://phabricator.wikimedia.org/T219857#11715367 (10daniel) [18:56:27] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API, 10Tool-Pageviews: massviews hits 429 Too Many Requests despite making requests synchronously - https://phabricator.wikimedia.org/T219857#11715374 (10daniel) @MusikAnimal how many requests does this tool need to make to pro... [19:02:17] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API, 10Tool-Pageviews: massviews hits 429 Too Many Requests despite making requests synchronously - https://phabricator.wikimedia.org/T219857#11715391 (10MusikAnimal) >>! In T219857#11715374, @daniel wrote: > @MusikAnimal how m... [19:03:13] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API, 10Tool-Pageviews: massviews hits 429 Too Many Requests despite making requests synchronously - https://phabricator.wikimedia.org/T219857#11715405 (10MusikAnimal) And heck, for Massviews specifically, maybe it's not too muc... [19:06:28] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API, 10Tool-Pageviews: massviews hits 429 Too Many Requests despite making requests synchronously - https://phabricator.wikimedia.org/T219857#11715406 (10Hawkeye7) Can we roll back this 429 change? I know I only use it once a... [19:57:25] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API, 10Tool-Pageviews: massviews hits 429 Too Many Requests despite making requests synchronously - https://phabricator.wikimedia.org/T219857#11715558 (10MusikAnimal) >>! In T219857#11715391, @MusikAnimal wrote: > … we could si... [20:10:20] 06Data-Engineering, 10ChangeProp, 10EventStreams, 06MediaWiki-Engineering, and 15 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11715627 (10Ottomata) [20:15:32] 06Data-Engineering, 10Event-Platform: Upgrade eventstreams and eventstreams-internal to node22 - https://phabricator.wikimedia.org/T420257#11715652 (10tchin) [20:38:37] RESOLVED: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [20:38:37] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:01:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:01:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:03:51] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07Essential-Work, 10Event-Platform, 13Patch-For-Review: Upgrade mediawiki-event-enrichment jobs to >= Flink 1.20.3 and Java 17 - https://phabricator.wikimedia.org/T408918#11715866 (10xcollazo) Folks I just did a deployment for `mw-content-history-re... [21:06:36] RESOLVED: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:06:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:08:06] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:08:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:10:15] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07Essential-Work, 10Event-Platform, 13Patch-For-Review: Upgrade mediawiki-event-enrichment jobs to >= Flink 1.20.3 and Java 17 - https://phabricator.wikimedia.org/T408918#11715885 (10xcollazo) Being bold and reverting changes to `mw-page-content-cha... [21:11:51] FIRING: [4x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:11:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:20:37] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API, 10Tool-Pageviews: massviews hits 429 Too Many Requests despite making requests synchronously - https://phabricator.wikimedia.org/T219857#11715954 (10daniel) >>! In T219857#11715405, @MusikAnimal wrote: > And heck, for Mass... [21:28:18] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API, 10Tool-Pageviews: massviews hits 429 Too Many Requests despite making requests synchronously - https://phabricator.wikimedia.org/T219857#11715985 (10MusikAnimal) Thanks so much for the help! >> And heck, for Massviews spe... [21:38:31] 06Data-Engineering, 10Event-Platform: Upgrade eventstreams and eventstreams-internal to node22 - https://phabricator.wikimedia.org/T420257#11716049 (10MoritzMuehlenhoff) I recently added node24 images: https://phabricator.wikimedia.org/T418440 So moving directly to Node 24 would also be an option?