[00:02:17] FIRING: [6x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [00:07:17] FIRING: [7x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [00:12:17] FIRING: [6x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [00:17:17] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [00:22:17] FIRING: [6x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [00:27:17] FIRING: [6x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [00:32:17] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [00:37:17] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [00:42:17] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [00:47:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [00:52:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [00:57:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [01:02:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [01:07:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [01:12:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [01:17:17] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [01:22:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [01:27:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [01:42:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [01:47:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [01:52:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [02:17:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [02:22:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [02:27:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [02:32:17] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [02:37:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [02:57:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:02:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:12:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:17:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [03:22:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:27:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:32:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:37:17] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:42:17] FIRING: [6x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:47:17] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:52:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:57:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:07:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:12:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:17:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:22:17] RESOLVED: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:32:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:37:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:42:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:47:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:52:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:57:47] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [05:02:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [05:07:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [05:12:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [05:17:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [05:22:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [05:32:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [05:37:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [05:47:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [05:52:47] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [05:57:47] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [06:02:47] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [06:07:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [06:12:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [06:17:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [06:27:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [06:32:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [06:42:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [07:02:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [07:10:57] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add cl_timestamp_id index to categorylinks table - https://phabricator.wikimedia.org/T399249#11029978 (10Marostegui) [07:17:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [07:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [07:22:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [07:27:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [07:32:47] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [07:37:47] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [07:42:47] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [07:52:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [07:58:16] 06Data-Engineering, 06Data-Engineering-Radar, 10CheckUser, 06DBA, and 3 others: Add '*_actor_ip_hex_time' indexes to 'cu_changes', 'cu_log_event', and 'cu_private_event' on WMF wikis - https://phabricator.wikimedia.org/T399728#11030089 (10FCeratto-WMF) [08:02:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [08:07:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [08:12:47] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [08:17:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [08:22:47] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [08:27:47] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [08:31:57] !log roll restart druid analytics cluster to pickup new zookeeper hosts an-druid100[4-5] [08:31:59] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:32:47] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [08:37:47] RESOLVED: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [08:42:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [08:47:47] FIRING: [6x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [08:52:47] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:02:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:07:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:12:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:17:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:27:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:32:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:37:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-main in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:42:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:43:02] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:47:47] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:52:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:03:02] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:09:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:14:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:19:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:34:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:44:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:44:20] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Structured-Data-Backlog: Bump memory to enable large artifacts sync on HDFS - https://phabricator.wikimedia.org/T348958#11030463 (10mfossati) >>! In T348958#11021599, @xcollazo wrote: > > can you please try to repro and confirm whether we can close this ti... [10:48:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:53:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:07:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-analytics - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:12:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:17:17] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [11:22:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:27:47] RESOLVED: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:32:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:37:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:42:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:47:47] RESOLVED: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:52:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [12:02:47] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [12:05:14] 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 10Data-Platform-SRE (2025.07.05 - 2025.07.25), 10Discovery-Search (2025.07.04 - 2025.07.25): Flink: Update k8s operator to 1.12.0 - https://phabricator.wikimedia.org/T398162#11030747 (10BTullis) [12:17:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [12:41:00] 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 10Data-Platform-SRE (2025.07.05 - 2025.07.25), 10Discovery-Search (2025.07.04 - 2025.07.25): Flink: Update k8s operator to 1.12.0 - https://phabricator.wikimedia.org/T398162#11030924 (10BTullis) a:03BTullis [12:43:23] 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 10Data-Platform-SRE (2025.07.05 - 2025.07.25), 10Discovery-Search (2025.07.04 - 2025.07.25): Flink: Update k8s operator to 1.12.0 - https://phabricator.wikimedia.org/T398162#11030932 (10BTullis) This is becoming increasingly important, as we have an inci... [12:52:47] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [13:10:52] 06Data-Engineering, 06Data-Engineering-Radar, 10CampaignEvents, 10EntitySchema, and 18 others: Add namespace descriptions for Special:NamespaceInfo in WMF-deployed extensions - https://phabricator.wikimedia.org/T373070#11031059 (10A_smart_kitten) [13:18:30] (03PS1) 10Aqu: Refine: Fix location of refined data-empty partitions [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1172318 (https://phabricator.wikimedia.org/T369845) [13:25:06] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Structured-Data-Backlog: Bump memory to enable large artifacts sync on HDFS - https://phabricator.wikimedia.org/T348958#11031102 (10fkaelin) Research does not bump versions for minor things, e.g. the [[ https://gitlab.wikimedia.org/repos/data-engineering/ai... [13:30:06] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031#11031119 (10xcollazo) [13:31:45] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031#11031140 (10xcollazo) [13:33:03] 06Data-Engineering, 06Traffic: Fix Hive event.development_network_probe table - https://phabricator.wikimedia.org/T400360 (10Ottomata) 03NEW [13:33:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [13:33:29] 06Data-Engineering, 06Traffic: Fix Hive event.development_network_probe table - https://phabricator.wikimedia.org/T400360#11031165 (10Ottomata) [13:34:00] 06Data-Engineering, 06Traffic: Fix Hive event.development_network_probe table - https://phabricator.wikimedia.org/T400360#11031167 (10Ottomata) [13:38:06] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031#11031175 (10xcollazo) Manually deleted the db representation for... [13:38:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [14:13:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [14:23:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [14:31:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [15:01:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [15:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [15:40:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [15:55:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [16:26:28] 10Data-Engineering (Q4 2025 April 1st - June 30th), 07Essential-Work, 13Patch-For-Review: Spike: Figure out a strategy to use Airflow's ExternalTaskMarker for our webrequest pipeline - https://phabricator.wikimedia.org/T399203#11032100 (10xcollazo) The latest version of the `ExternalTaskMarker` experiment at... [16:35:35] 10Data-Engineering (Q4 2025 April 1st - June 30th), 07Essential-Work, 13Patch-For-Review: Spike: Figure out a strategy to use Airflow's ExternalTaskMarker for our webrequest pipeline - https://phabricator.wikimedia.org/T399203#11032174 (10xcollazo) Further: When clearing a full day from the point of view of... [16:36:47] 10Data-Engineering (Q4 2025 April 1st - June 30th), 07Essential-Work, 13Patch-For-Review: Spike: Figure out a strategy to use Airflow's ExternalTaskMarker for our webrequest pipeline - https://phabricator.wikimedia.org/T399203#11032184 (10mforns) That's amazing, Xabriel! :-) so cool Maybe we can clear a time... [16:47:21] 06Data-Engineering, 06Data-Platform-SRE: FAIL: refinery-drop-raw-event alerting - https://phabricator.wikimedia.org/T400393 (10dr0ptp4kt) 03NEW [16:49:48] 06Data-Engineering, 06Data-Platform-SRE: FAIL: refinery-drop-raw-event alerting - https://phabricator.wikimedia.org/T400393#11032228 (10amastilovic) Copy-pasting the actual error message: ` ERROR: ('Command: hdfs dfs -rm -R -skipTrash /wmf/data/raw/event/codfw.eventgate-analytics.test.event/year=2025/month=04... [17:35:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:40:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:57:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:02:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:08:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:13:40] 10Data-Engineering (Q4 2025 April 1st - June 30th), 07Essential-Work, 13Patch-For-Review: Spike: Figure out a strategy to use Airflow's ExternalTaskMarker for our webrequest pipeline - https://phabricator.wikimedia.org/T399203#11032451 (10xcollazo) >>! In T399203#11032184, @mforns wrote: > That's amazing, Xa... [18:18:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:24:25] 06Data-Engineering, 10Data-Services: Create a view for existencelinks table - https://phabricator.wikimedia.org/T394898#11032509 (10taavi) a:03Ladsgroup https://gerrit.wikimedia.org/r/c/operations/puppet/+/1163846 seems to have done this. [18:24:30] 06Data-Engineering, 10Data-Services: Create a view for existencelinks table - https://phabricator.wikimedia.org/T394898#11032511 (10taavi) 05Open→03Resolved [18:25:34] 10Data-Engineering (Q4 2025 April 1st - June 30th), 07Essential-Work, 13Patch-For-Review: Spike: Figure out a strategy to use Airflow's ExternalTaskMarker for our webrequest pipeline - https://phabricator.wikimedia.org/T399203#11032514 (10xcollazo) For Airflow UI to calculate and present the list of 6420 tas... [18:29:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:34:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [19:55:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:57:13] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Structured-Data-Backlog: Bump memory to enable large artifacts sync on HDFS - https://phabricator.wikimedia.org/T348958#11032756 (10amastilovic) >>! In T348958#11031102, @fkaelin wrote: > Research does not bump versions for minor things, e.g. the [[ https:/... [20:15:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [20:26:44] 10Data-Engineering (Q4 2025 April 1st - June 30th): Enable forced cache warmup option for airflow-dags blunderbuss integration - https://phabricator.wikimedia.org/T400411 (10amastilovic) 03NEW [20:39:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [20:49:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [20:57:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:03:37] 06Data-Engineering, 10Data-Services: Create a view for existencelinks table - https://phabricator.wikimedia.org/T394898#11032974 (10Tacsipacsi) Indeed, thanks for noticing this! I’m still eagerly waiting for the API, but at least there is now a workaround for WMF production wikis. [21:07:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:15:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:20:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:54:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [22:10:47] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [22:17:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [22:22:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [22:33:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [22:43:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [23:59:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly