[00:04:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [00:15:25] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Add support for repository artifacts in Airflow - https://phabricator.wikimedia.org/T322690#11033412 (10amastilovic) >>! In T322690#11005129, @Ottomata wrote: > @amastilovic should we resolve or decline this? Thanks! This should already be su... [00:16:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [00:21:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [00:36:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [00:56:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [01:01:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [01:36:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [01:42:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [01:57:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [02:00:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [02:02:04] milimetric: What's the correct URL to change https://stats.wikimedia.org/index-v1.html to? [02:02:21] https://stats.wikimedia.org/ > https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Wikistats [02:02:21] https://phabricator.wikimedia.org/T389107#10733171 [02:02:47] I checked https://analytics.wikimedia.org/published/datasets/ and https://dumps.wikimedia.org/other/ and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1039246 but could not find it. [02:05:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [02:09:32] Also checked https://meta.wikimedia.org/wiki/Statistics [02:26:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [02:31:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [02:44:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [02:49:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [06:30:42] 06Data-Engineering, 06Data-Engineering-Radar, 10CheckUser, 06DBA, and 2 others: Add '*_actor_ip_hex_time' indexes to 'cu_changes', 'cu_log_event', and 'cu_private_event' on WMF wikis - https://phabricator.wikimedia.org/T399728#11033639 (10FCeratto-WMF) [07:06:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [07:16:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [07:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [11:15:13] 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117#11034302 (10Fabfur) 05In progress→03Resolved [11:15:59] 06Data-Engineering, 06Data-Engineering-Radar, 10HaproxyKafka, 06Traffic, and 2 others: New software: haproxykafka - https://phabricator.wikimedia.org/T370668#11034306 (10Fabfur) 05In progress→03Resolved Closing as we can use the HaproxyKafka component to keep track of related tasks [11:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [11:26:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:29:15] Krinkle, sorry we took that offline and never did the optional part to surface it. It hadn't been updated in over five years so we figured nobody was using it. The backup is on hdfs (https://phabricator.wikimedia.org/T389107#10743982) [11:29:52] But let's talk, happy to help get what you or others need out of it [11:31:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [13:49:33] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 07Schema-change, and 2 others: AbuseFilter abuse_filter_log table: Store IP addresses as hex values - https://phabricator.wikimedia.org/T395612#11034667 (10OKryva-WMF) [14:06:50] 10Data-Engineering (Q4 2025 April 1st - June 30th), 07Essential-Work, 13Patch-For-Review: Spike: Figure out a strategy to use Airflow's ExternalTaskMarker for our webrequest pipeline - https://phabricator.wikimedia.org/T399203#11034748 (10BTullis) Hmm. We do have the `airflow-production-tls-proxy` container... [14:34:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [14:39:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [14:43:38] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add cl_timestamp_id index to categorylinks table - https://phabricator.wikimedia.org/T399249#11034868 (10Marostegui) [14:44:13] FIRING: HaproxykafkaNoMessages: [WARNING] - Haproxykafka on cp3066 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/HAProxyKafka - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-datasource=esams%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp3066 - https://alerts.wikimedia.org/?q=alertname%3DHaproxykafkaNoMessages [14:44:14] FIRING: [2x] HaproxyKafkaDeliveryErrors: haproxykafka has cache_text saturation errors on cp5018:9341 - https://wikitech.wikimedia.org/wiki/HAProxyKafka - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaDeliveryErrors [14:49:13] RESOLVED: [2x] HaproxykafkaNoMessages: [WARNING] - Haproxykafka on cp3066 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/HAProxyKafka - https://alerts.wikimedia.org/?q=alertname%3DHaproxykafkaNoMessages [14:49:13] RESOLVED: [3x] HaproxyKafkaDeliveryErrors: haproxykafka has cache_text saturation errors on cp3066:9341 - https://wikitech.wikimedia.org/wiki/HAProxyKafka - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaDeliveryErrors [14:52:51] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Dumps-Generation, 10Wikidata, 10Data-Platform-SRE (2025.07.05 - 2025.07.25), and 2 others: wikidata-20250707-all.json.gz is corrupted - https://phabricator.wikimedia.org/T399077#11034928 (10DVrandecic) Not sure if this is related, but the latest Lexemes... [14:54:28] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10HaproxyKafka, 06Traffic, 13Patch-For-Review: Haproxykafka silently stops sending request data to kafka - https://phabricator.wikimedia.org/T400039#11034940 (10Fabfur) Note that this happened again ~2025-07-24 14:43 on cp3071, same host [15:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [16:04:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [16:07:57] 10Data-Engineering (Q4 2025 April 1st - June 30th), 07Essential-Work, 13Patch-For-Review: Spike: Figure out a strategy to use Airflow's ExternalTaskMarker for our webrequest pipeline - https://phabricator.wikimedia.org/T399203#11035235 (10xcollazo) Thanks @BTullis.We'd need to bump that to make this feature... [16:08:31] 10Data-Engineering (Q4 2025 April 1st - June 30th), 07Essential-Work, 13Patch-For-Review: Spike: Figure out a strategy to use Airflow's ExternalTaskMarker for our webrequest pipeline - https://phabricator.wikimedia.org/T399203#11035236 (10xcollazo) [16:19:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [16:26:27] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031#11035301 (10xcollazo) [16:35:49] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031#11035363 (10xcollazo) Removed remaining Dumps1 XML artifacts from HDFS: ` $ hostname -... [16:36:40] !log removed remaining raw Dumps1 XML files from HDFS. See T396031#11035363 for details. [16:36:43] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:36:44] T396031: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031 [16:49:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [16:51:55] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Experimentation Lab, 10Event-Platform: EventGate: Add Prometheus metric for hoisting errors - https://phabricator.wikimedia.org/T398922#11035404 (10dr0ptp4kt) Same as @phuedx , a new metric sounds good to me. Adding another label would be fine, but this... [16:52:59] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Experimentation Lab, 10Event-Platform: EventGate: Log unparseable X-Experiment-Enrollments headers to a distinct error stream - https://phabricator.wikimedia.org/T396359#11035406 (10dr0ptp4kt) [17:09:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:19:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:24:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:43:10] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Structured-Data-Backlog: Bump memory to enable large artifacts sync on HDFS - https://phabricator.wikimedia.org/T348958#11035519 (10xcollazo) a:05xcollazo→03None [17:47:37] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Dumps-Generation, 10Wikidata, 10Data-Platform-SRE (2025.07.05 - 2025.07.25), 07Essential-Work: wikidata-20250707-all.json.gz is corrupted - https://phabricator.wikimedia.org/T399077#11035522 (10BTullis) >>! In T399077#11034928, @DVrandecic wrote: > No... [17:51:32] 06Data-Engineering, 10DPE-Mediawiki-Content, 07Epic: Production-level file export (aka dump) of MW Content in XML - https://phabricator.wikimedia.org/T384382#11035538 (10xcollazo) 05Open→03In progress p:05Triage→03High a:03xcollazo [17:52:13] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Dumps-Generation: wikidatawiki fails dumps of the wbt_* tables, also lagging on XML Dumps - https://phabricator.wikimedia.org/T396125#11035547 (10BTullis) OK, interesting. I'm not sure that it's related to Airflow. The tables don't appear on the dumps that... [17:53:08] 06Data-Engineering, 10DPE-Mediawiki-Content, 07Epic: Production-level file export (aka dump) of MW Content in XML - https://phabricator.wikimedia.org/T384382#11035548 (10xcollazo) [17:55:25] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031#11035573 (10xcollazo) [17:55:27] 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 07Epic: Dumps 2.0 Phase III: Production level dumps - https://phabricator.wikimedia.org/T366752#11035574 (10xcollazo) [17:55:58] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031#11035578 (10xcollazo) [17:56:01] 06Data-Engineering, 10DPE-Mediawiki-Content, 07Epic: Production-level file export (aka dump) of MW Content in XML - https://phabricator.wikimedia.org/T384382#11035579 (10xcollazo) [17:57:01] 06Data-Engineering, 10DPE-Mediawiki-Content, 07Epic: Production-level file export (aka dump) of MW Content in XML - https://phabricator.wikimedia.org/T384382#11035582 (10xcollazo) [17:57:02] 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 07Epic: Dumps 2.0 Phase III: Production level dumps - https://phabricator.wikimedia.org/T366752#11035583 (10xcollazo) [17:57:41] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Structured-Data-Backlog: Bump memory to enable large artifacts sync on HDFS - https://phabricator.wikimedia.org/T348958#11035584 (10amastilovic) a:03amastilovic [17:59:20] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Add metrics for monthly reconciles - https://phabricator.wikimedia.org/T388439#11035596 (10xcollazo) @tchin are we done here? If so, can we tick the boxes and close? [18:01:04] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Structured-Data-Backlog: Bump memory to enable large artifacts sync on HDFS - https://phabricator.wikimedia.org/T348958#11035609 (10fkaelin) If there are no changes required on the airflow dag itself, it is preferable to not have to do another review/deploy... [18:01:07] 06Data-Engineering, 10DPE-Mediawiki-Content: Go over tasks in #dumps-generation and figure what makes sense to fix in Dumps 2.0 (DPE-Mediawiki-Content) - https://phabricator.wikimedia.org/T379410#11035610 (10xcollazo) [18:01:09] 06Data-Engineering, 10DPE-Mediawiki-Content, 07Epic: Production-level file export (aka dump) of MW Content in XML - https://phabricator.wikimedia.org/T384382#11035611 (10xcollazo) [18:01:10] 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 07Epic: Dumps 2.0 Phase III: Production level dumps - https://phabricator.wikimedia.org/T366752#11035612 (10xcollazo) [18:03:49] 06Data-Engineering, 10DPE-Mediawiki-Content: [SPIKE] Benchmark the run time of batch processing - https://phabricator.wikimedia.org/T379365#11035620 (10xcollazo) This work can definitely inform our hopes to have SLOs. CC @GGoncalves-WMF [18:16:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:21:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:33:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:33:46] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Add metrics for monthly reconciles - https://phabricator.wikimedia.org/T388439#11035695 (10tchin) Yeah it can be closed out [18:34:16] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Add metrics for monthly reconciles - https://phabricator.wikimedia.org/T388439#11035696 (10tchin) 05Open→03Resolved [18:38:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:09:15] 10Data-Engineering (Q4 2025 April 1st - June 30th): Spike: Assess feasibility of integration the new file export with the legacy Dumps1 UI - https://phabricator.wikimedia.org/T400507 (10xcollazo) 03NEW [19:09:45] 10Data-Engineering (Q4 2025 April 1st - June 30th): Spike: Assess feasibility of integration the new file export with the legacy Dumps1 UI - https://phabricator.wikimedia.org/T400507#11035817 (10xcollazo) [19:09:47] 06Data-Engineering, 10DPE-Mediawiki-Content, 07Epic: Production-level file export (aka dump) of MW Content in XML - https://phabricator.wikimedia.org/T384382#11035818 (10xcollazo) [19:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [19:46:44] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Experimentation Lab, 10Event-Platform: EventGate: Log unparseable X-Experiment-Enrollments headers to a distinct error stream - https://phabricator.wikimedia.org/T396359#11035912 (10dr0ptp4kt) @phuedx do you have a strong feeling we still need the raw hea... [20:11:38] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031#11035976 (10xcollazo) [20:13:08] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031#11035979 (10xcollazo) The only remaining task here is to DROP the tables themselves. Wi... [22:51:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [22:54:57] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): Spike: Assess feasibility of integration the new file export with the legacy Dumps1 UI - https://phabricator.wikimedia.org/T400507#11036194 (10Ahoelzl) [22:55:00] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): Enable forced cache warmup option for airflow-dags blunderbuss integration - https://phabricator.wikimedia.org/T400411#11036195 (10Ahoelzl) [22:55:01] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): Streamline and clean up airflow-dags gitlab-ci.yaml CI/CD pipelines - https://phabricator.wikimedia.org/T400283#11036196 (10Ahoelzl) [22:55:04] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 06Experimentation Lab, 10Event-Platform: EventGate: Log unparseable X-Experiment-Enrollments headers to a distinct error stream - https://phabricator.wikimedia.org/T396359#11036197 (10Ahoelzl) [22:55:07] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10MediaWiki-DomainEvents: Finalize and move Cross-Service Integration events design document to mediawiki.org - https://phabricator.wikimedia.org/T400095#11036200 (10Ahoelzl) [22:55:10] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Inconsistent state visibility for Flink deployments on dse-k8s-eqiad - https://phabricator.wikimedia.org/T395984#11036198 (10Ahoelzl) [22:55:11] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 06Structured-Data-Backlog: Bump memory to enable large artifacts sync on HDFS - https://phabricator.wikimedia.org/T348958#11036201 (10Ahoelzl) [22:55:12] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10CheckUser, 06Trust and Safety Product Team: Inform Data-Engineering about removal of cuc_ip, cule_ip, and cupe_ip - https://phabricator.wikimedia.org/T399958#11036199 (10Ahoelzl) [22:55:16] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): Implement full parity between HiveSensor and RESTExternalTaskSensor - https://phabricator.wikimedia.org/T384726#11036203 (10Ahoelzl) [22:55:20] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): Manage druid `webrequest_sampled_live` data size - https://phabricator.wikimedia.org/T398236#11036205 (10Ahoelzl) [22:55:24] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10Dumps-Generation: wikidatawiki fails dumps of the wbt_* tables, also lagging on XML Dumps - https://phabricator.wikimedia.org/T396125#11036202 (10Ahoelzl) [22:55:29] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 06Experimentation Lab: NEW/CHANGE FEATURE REQUEST: make available the centralauth.globaluser table in Data Lake - https://phabricator.wikimedia.org/T389666#11036206 (10Ahoelzl) [22:55:33] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 06Movement-Insights: event.editattemptstep is not logging some revisions that appear in mediawiki_history - https://phabricator.wikimedia.org/T394961#11036208 (10Ahoelzl) [22:55:37] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10Cassandra: Cassandra multi-tenant access configuration - https://phabricator.wikimedia.org/T318407#11036211 (10Ahoelzl) [22:55:43] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 07Essential-Work, 10Event-Platform: Gobblin-wmf Gitlab migration and maintenance - https://phabricator.wikimedia.org/T370368#11036210 (10Ahoelzl) [22:55:47] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: When doing ADD COLUMN to a struct under a map, Iceberg fails to SELECT it - https://phabricator.wikimedia.org/T388793#11036213 (10Ahoelzl) [22:55:51] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 07Essential-Work, 13Patch-For-Review: [Data Quality] Implement wiki completeness check for MediaWiki History - https://phabricator.wikimedia.org/T365203#11036209 (10Ahoelzl) [22:55:55] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): [Refine Simplification] Remove Schema Merging in Refine Process by Enforcing Backward Compatibility - https://phabricator.wikimedia.org/T381072#11036215 (10Ahoelzl) [22:55:59] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10Observability-Tracing, 10Event-Platform, 13Patch-For-Review: EventGate: Enable OpenTelemetry Propagation - https://phabricator.wikimedia.org/T391353#11036212 (10Ahoelzl) [22:56:03] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): Reduce `refine_to_hive_hourly` airflow task number - https://phabricator.wikimedia.org/T380856#11036216 (10Ahoelzl) [22:56:07] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): Move more of refine_hive_hourly dag logic into RefineConfiguration - https://phabricator.wikimedia.org/T375064#11036217 (10Ahoelzl) [22:56:11] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 06Trust and Safety Product Team, 13Patch-For-Review, 10Product-Analytics (Kanban): Add mediawiki_product_metrics_incident_reporting_system_interaction to the sanitization allowlist - https://phabricator.wikimedia.org/T384650#11036214 (10Ahoelzl) [22:56:15] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): test_produced_by_config SLA miss configured to be too small for upstream dataset run time - https://phabricator.wikimedia.org/T388861#11036218 (10Ahoelzl) [22:56:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [22:56:23] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): Review Image Suggestion pipeline SLOs - https://phabricator.wikimedia.org/T400282#11036228 (10Ahoelzl) [22:56:28] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): Adapt Sqoop for categorylinks schema change - https://phabricator.wikimedia.org/T397923#11036232 (10Ahoelzl) [22:56:32] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10EventStreams, 10Event-Platform: EventStreams: duplicate events from double compute (wdqs/rdf) streams - https://phabricator.wikimedia.org/T396564#11036234 (10Ahoelzl) [22:56:36] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10HaproxyKafka, 06Traffic, 13Patch-For-Review: Haproxykafka silently stops sending request data to kafka - https://phabricator.wikimedia.org/T400039#11036230 (10Ahoelzl) [22:56:42] 07Analytics-Data-Problem, 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 06Movement-Insights: NEW BUG REPORT <"Domain" field issue: some domains have trailing dots> - https://phabricator.wikimedia.org/T395963#11036240 (10Ahoelzl) [22:56:48] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10Dumps-Generation, 10Wikidata, 10Data-Platform-SRE (2025.07.05 - 2025.07.25), 07Essential-Work: wikidata-20250707-all.json.gz is corrupted - https://phabricator.wikimedia.org/T399077#11036236 (10Ahoelzl) [22:56:54] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): Refine to Hive with Airflow – Post-Migration Cleanup - https://phabricator.wikimedia.org/T392698#11036246 (10Ahoelzl) [22:56:58] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 06Experimentation Lab, 10Event-Platform: EventGate: Add Prometheus metric for hoisting errors - https://phabricator.wikimedia.org/T398922#11036244 (10Ahoelzl) [22:57:02] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): Define Event Platform Essential Work FY26 - https://phabricator.wikimedia.org/T399661#11036248 (10Ahoelzl) [22:57:06] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): Revive data engineering alert metrics dashboard - https://phabricator.wikimedia.org/T399518#11036250 (10Ahoelzl) [22:57:10] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031#11036252 (10Ahoelzl) [22:57:14] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 13Patch-For-Review: NEW BUG REPORT wmf.interlanguage_navigation missing mobile data - https://phabricator.wikimedia.org/T396514#11036242 (10Ahoelzl) [22:57:18] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 06Movement-Insights, 06Traffic: NEW BUG REPORT: Investigate rise in May 2025 Reader metrics - https://phabricator.wikimedia.org/T395934#11036256 (10Ahoelzl) [22:57:24] 14Analytics, 07Analytics-Data-Problem, 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10Data-Engineering-Wikistats, 06Movement-Insights: Sharp spike in unique devices for past month on all projects - https://phabricator.wikimedia.org/T395727#11036254 (10Ahoelzl) [22:57:28] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): Refine to Hive with Airflow – Update Refine Documentation on Wikitech - https://phabricator.wikimedia.org/T392697#11036262 (10Ahoelzl) [22:57:32] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): Investigate mw-page-content-change memory alerts - https://phabricator.wikimedia.org/T397336#11036264 (10Ahoelzl) [22:57:36] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 13Patch-For-Review: Refine to Hive with Airflow – Handle Late-Arrived Events - https://phabricator.wikimedia.org/T370665#11036258 (10Ahoelzl) [22:57:40] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 13Patch-For-Review: [Refine Refactoring] Refine jobs should be scheduled by Airflow: deployment - https://phabricator.wikimedia.org/T369845#11036260 (10Ahoelzl) [22:57:48] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: MediaWiki Content History alerts too much for minor reconcile issues - https://phabricator.wikimedia.org/T395139#11036286 (10Ahoelzl) [22:57:52] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): Refine to Hive with Airflow – Kubernetes Resource Optimization - https://phabricator.wikimedia.org/T392668#11036288 (10Ahoelzl) [22:57:56] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Investigate reasons for remaining inconsistencies - https://phabricator.wikimedia.org/T385112#11036281 (10Ahoelzl) [22:58:00] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 13Patch-For-Review: Enable Spark data lineage for all Airflow instances - https://phabricator.wikimedia.org/T386862#11036279 (10Ahoelzl) [22:58:04] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): Analytics Cluster Dataset Usage Discovery Task - https://phabricator.wikimedia.org/T389903#11036290 (10Ahoelzl) [22:58:16] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10Event-Platform: Add alerting to eventbus and eventgate for drastic changes in event rate production. - https://phabricator.wikimedia.org/T398437#11036297 (10Ahoelzl) [22:58:20] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10EventStreams, 06SRE Observability, 10Event-Platform: Eventstreams 'assignments' logstash field type - https://phabricator.wikimedia.org/T390140#11036301 (10Ahoelzl) [22:58:24] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10Event-Platform, 10MW-1.45-notes (1.45.0-wmf.12; 2025-07-29), 13Patch-For-Review: Update event-producing tools to overwrite `meta.dt` - https://phabricator.wikimedia.org/T376026#11036299 (10Ahoelzl) [22:58:28] 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: [Event Platform] eventutilites-python: improve consistency guarantees of async process functions - https://phabricator.wikimedia.org/T347282#11036305 (10Ahoelzl) [23:03:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:05:19] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Data-Engineering-Radar, 10Event-Platform, 07Wikimedia-production-error: eventgate-analytics has stopped producing events since 2025-06-25 - https://phabricator.wikimedia.org/T398187#11036312 (10Ahoelzl) 05Open→03Resolved [23:05:23] 10Data-Engineering (Q4 2025 April 1st - June 30th), 07Essential-Work: Druid job of mediawiki_history_reduced overwhelms the cluster, using 85%+ of its capacity - https://phabricator.wikimedia.org/T399013#11036315 (10Ahoelzl) 05Open→03Resolved [23:05:26] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10EventStreams: Figure out how Eventstreams connected client metrics went negative - https://phabricator.wikimedia.org/T398325#11036314 (10Ahoelzl) 05Open→03Resolved [23:05:28] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Diffusion-Repository-Administrators, 10Projects-Cleanup: Archive the `reportupdater` and `reportupdater-queries` gerrit repos - https://phabricator.wikimedia.org/T397922#11036317 (10Ahoelzl) 05Open→03Resolved [23:05:29] 10Data-Engineering (Q4 2025 April 1st - June 30th), 07Essential-Work, 13Patch-For-Review: Spike: Figure out a strategy to use Airflow's ExternalTaskMarker for our webrequest pipeline - https://phabricator.wikimedia.org/T399203#11036316 (10Ahoelzl) 05Open→03Resolved [23:05:32] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 07Epic, 10Event-Platform, and 3 others: [SPIKE] PoC to implement an example pipeline for bringing data into MediaWiki - https://phabricator.wikimedia.org/T396892#11036318 (10Ahoelzl) 05Open→03Resolved [23:05:34] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 07Epic, 10Event-Platform, and 2 others: PageChangeEventSerializer should deprecate WikiPage and adopt PageIdentity - https://phabricator.wikimedia.org/T396453#11036320 (10Ahoelzl) 05Open→03Resolved [23:05:35] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 07Epic, 10Event-Platform, and 2 others: EventBus: replace ArticleRevisionVisibilitySetHook with a RevisionVisibilityChangedEvent listener - https://phabricator.wikimedia.org/T396213#11036322 (10Ahoelzl) 05Open→03Resolved [23:05:41] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Event-Platform: mediawiki.page_change.v1 should not contain events for undelete into existing pages. - https://phabricator.wikimedia.org/T395327#11036324 (10Ahoelzl) 05Open→03Resolved [23:05:57] 10Data-Engineering (Q4 2025 April 1st - June 30th): Spike on choosing a solution for DagProperties - https://phabricator.wikimedia.org/T394541#11036331 (10Ahoelzl) 05Open→03Resolved [23:06:01] 10Data-Engineering (Q4 2025 April 1st - June 30th): Determine how many admins are there in English Wikipedia and French Wikipedia from sub-Saharan Africa - https://phabricator.wikimedia.org/T395279#11036330 (10Ahoelzl) 05Open→03Resolved [23:06:05] 10Data-Engineering (Q4 2025 April 1st - June 30th): Modify scap config files so that we pull artifacts from main rather than deprecated analytics config - https://phabricator.wikimedia.org/T394343#11036334 (10Ahoelzl) 05Open→03Resolved [23:06:10] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, 10Event-Platform: PagedMovedEvent should carry reason data - https://phabricator.wikimedia.org/T394046#11036336 (10Ahoelzl) 05Open→03Resolved [23:06:16] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, 10Event-Platform: PageMovedEvent should carry article redirect data - https://phabricator.wikimedia.org/T394049#11036335 (10Ahoelzl) 05Open→03Resolved [23:06:22] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 07Epic, 10Event-Platform, and 3 others: Testing the domain event refactoring with production data - https://phabricator.wikimedia.org/T394899#11036332 (10Ahoelzl) 05Open→03Resolved [23:06:28] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 07Epic, 10Event-Platform, and 2 others: EventBus: replace PageMoveCompleteHook with PageMovedEvent - https://phabricator.wikimedia.org/T393890#11036340 (10Ahoelzl) 05Open→03Resolved [23:06:34] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 07Epic, 10Event-Platform, and 3 others: EventBus: Replace PageUndeleteCompleteHook with PageRevisionUpdated - https://phabricator.wikimedia.org/T393891#11036339 (10Ahoelzl) 05Open→03Resolved [23:06:40] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Event-Platform: [BUG} PageEntitySerializer does not handle redirects correctly on deletion - https://phabricator.wikimedia.org/T393757#11036344 (10Ahoelzl) 05Open→03Resolved [23:06:44] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, 10Event-Platform: PageDeleted event should contain outgoing redirect target information - https://phabricator.wikimedia.org/T393633#11036345 (10Ahoelzl) 05Open→03Resolved [23:06:50] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 07Epic, 10Event-Platform, and 2 others: EventBus: replace PageDeleteCompleteHook with PageDeletedListener - https://phabricator.wikimedia.org/T392205#11036346 (10Ahoelzl) 05Open→03Resolved [23:07:02] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 10Event-Platform: EventBus: replace PageSaveCompleteHook with PageRevisionUpdateListener - https://phabricator.wikimedia.org/T390970#11036348 (10Ahoelzl) 05Open→03Resolved [23:07:08] 10Data-Engineering (Q4 2025 April 1st - June 30th): Analyze impact for webrequest and unique devices pipelines to derive access_method without m-dot domain - https://phabricator.wikimedia.org/T389696#11036350 (10Ahoelzl) 05Open→03Resolved [23:07:12] 10Data-Engineering (Q4 2025 April 1st - June 30th): Provide Data Engineering Q4 draft - https://phabricator.wikimedia.org/T387385#11036352 (10Ahoelzl) 05Open→03Resolved [23:07:16] 10Data-Engineering (Q4 2025 April 1st - June 30th): Warning of mismatch in declarations of Webrequest schema - https://phabricator.wikimedia.org/T380916#11036354 (10Ahoelzl) 05Open→03Resolved [23:07:20] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Commons-Impact-Metrics: [Commons Impact Metrics] Add page wiki to the corresponding top endpoints - https://phabricator.wikimedia.org/T372805#11036356 (10Ahoelzl) 05Open→03Resolved [23:07:24] 10Data-Engineering (Q4 2025 April 1st - June 30th): Switch webrequest dataset to feed from HAProxy instead of VarnishKafka - https://phabricator.wikimedia.org/T386177#11036353 (10Ahoelzl) 05Open→03Resolved [23:07:28] 10Data-Engineering (Q4 2025 April 1st - June 30th), 07Essential-Work: Remove sqoop code for wikibase term storage - https://phabricator.wikimedia.org/T391006#11036358 (10Ahoelzl) 05Open→03Resolved [23:07:32] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Commons-Impact-Metrics, 13Patch-For-Review: Update Commons Impact Metrics to account for new File table - https://phabricator.wikimedia.org/T389800#11036359 (10Ahoelzl) 05Open→03Resolved [23:07:42] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: List out all migration candidates for mediawiki_content_history - https://phabricator.wikimedia.org/T386757#11036363 (10Ahoelzl) 05Open→03Resolved [23:07:46] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 10MW-Interfaces-Team (MWI-Roadmap): DomainEvents - [Hypothesis] WE5.2.6 Event Broadcasting Discovery & Design - https://phabricator.wikimedia.org/T384874#11036364 (10Ahoelzl) 05Open→03Resolved [23:07:54] 07Analytics-Data-Problem, 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Event-Platform, 10MediaWiki-Platform-Team (Radar), 05SUL3: Some events in mediawiki.page_change.v1 refers to auth.wikimedia.org in meta.uri and meta.domain - https://phabricator.wikimedia.org/T388825#11036362 (10Ahoelzl)... [23:08:00] 10Data-Engineering (Q4 2025 April 1st - June 30th): 2025-04-01 run of mediawiki_wikitext_history is stuck (20d running) - https://phabricator.wikimedia.org/T394954#11036366 (10Ahoelzl) 05Open→03Resolved [23:13:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [23:46:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:51:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly