[00:58:52] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:58:52] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [02:51:11] 10Analytics-Canonical-Data, 06Movement-Insights (FY25-26 H2): Update canonical wiki dataset generation based on changes in language name data - https://phabricator.wikimedia.org/T420126#11710781 (10BucheonFC) 05Open→03Declined [03:27:52] 10Analytics-Canonical-Data, 06Movement-Insights (FY25-26 H2): Update canonical wiki dataset generation based on changes in language name data - https://phabricator.wikimedia.org/T420126#11710919 (10AlexisJazz) 05Declined→03Open @nshahquinn-wmf [03:38:25] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API, 10Tool-Pageviews: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857#11710943 (10MusikAnimal) From preliminary tests on ~8-10K requests: * Making calls synchronously (f... [03:38:55] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API, 10Tool-Pageviews: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857#11710945 (10MusikAnimal) p:05Medium→03High [04:58:52] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [04:58:52] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:32:39] 06Data-Engineering, 10ChangeProp, 10EventStreams, 06MediaWiki-Engineering, and 15 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11711052 (10Kgarcia181) 05Open→03In progress p:05Medium→03Unbreak! a:03Kgarcia181 [06:35:54] 06Data-Engineering, 10ChangeProp, 10EventStreams, 06MediaWiki-Engineering, and 15 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11711055 (10Peachey88) 05In progress→03Open p:05Unbreak!→03Medium a:05Kgarcia181→03None [08:58:52] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [08:58:52] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [09:43:23] 10Analytics-Canonical-Data, 06Movement-Insights (FY25-26 H2): Update canonical wiki dataset generation based on changes in language name data - https://phabricator.wikimedia.org/T420126#11711190 (10TheBucheon) 05Open→03Resolved a:03TheBucheon [09:45:00] 10Analytics-Canonical-Data, 06Movement-Insights (FY25-26 H2): Update canonical wiki dataset generation based on changes in language name data - https://phabricator.wikimedia.org/T420126#11711198 (10Novem_Linguae) 05Resolved→03Open a:05TheBucheon→03None [12:58:52] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [12:58:52] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:33:13] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06Editing-team, and 2 others: EditPage save hooks pass an entire `EditPage` object - https://phabricator.wikimedia.org/T251588#11711434 (10Tacsipacsi) >>! In T251588#6580279, @DannyS712 wrote: > For now, the constraint system is completely i... [13:58:03] 06Data-Engineering, 06Data-Engineering-Radar, 06Movement-Insights, 06Traffic: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463#11711524 (10dr0ptp4kt) 05Open→03Resolved This is done. [14:09:35] 06Data-Engineering: 21 new wikis missing from mediawiki_history - https://phabricator.wikimedia.org/T386649#11711564 (10dr0ptp4kt) a:05dr0ptp4kt→03None [14:16:34] 06Data-Engineering, 10Data Pipelines, 07Performance Issue: Potentially increase resource limits for pageview_actor_hourly Airflow job - https://phabricator.wikimedia.org/T400209#11711592 (10dr0ptp4kt) [14:17:21] 06Data-Engineering, 10Data Pipelines, 07Performance Issue: Potentially increase resource limits for pageview_actor_hourly Airflow job - https://phabricator.wikimedia.org/T400209#11711593 (10dr0ptp4kt) 05Open→03Declined This is now stale. There are other things around this kind of stuff. Marking as De... [16:58:52] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [16:58:52] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [17:37:10] 06Data-Engineering, 10Data-Services, 06cloud-services-team (FY2025/2026-Q3-Q4), 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11711769 (10taavi) [20:58:52] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [20:58:52] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag