[01:04:43] !log refine_eventlogging_legacy mediawikipingback 2025-05-15T04 2025-05-23T00 [01:04:45] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [01:27:53] !log refine_eventlogging_legacy mediawikipingback 2025-07-15T04 2025-07-23T00 [01:27:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [03:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [06:47:10] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Event-Platform, 13Patch-For-Review: Add alerting to eventbus and eventgate for drastic changes in event rate production. - https://phabricator.wikimedia.org/T398437#11026637 (10fgiunchedi) >>! In T398437#11025615, @Ottomata wrote: > @gmodena +1 on both pa... [07:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [09:07:50] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Experimentation Lab, 10Event-Platform: EventGate: Add Prometheus metric for hoisting errors - https://phabricator.wikimedia.org/T398922#11026916 (10phuedx) > I think I'm inclined to make a new metric. I think that perhaps it would be better if HoistingErr... [10:00:58] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 07Schema-change, and 2 others: AbuseFilter abuse_filter_log table: Store IP addresses as hex values - https://phabricator.wikimedia.org/T395612#11027123 (10Tchanders) Having spoken with @STran, this belongs in "Epics in Progress", since the accep... [10:05:31] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Structured-Data-Backlog: Bump memory to enable large artifacts sync on HDFS - https://phabricator.wikimedia.org/T348958#11027140 (10mfossati) >>! In T348958#11021599, @xcollazo wrote: > > On {T392244}, we migrated Airflow artifact deployment from scap to a... [10:50:24] 06Data-Engineering: ERROR AsyncEventQueue: Listener DatahubSparkListener threw an exception - https://phabricator.wikimedia.org/T400207#11027292 (10dr0ptp4kt) Just a hunch, but this may also pertain to DataHub sometimes being down. There are `data-platform-alerts` intermittently indicating `DatahubNextServiceUna... [11:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [13:31:54] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Structured-Data-Backlog: Bump memory to enable large artifacts sync on HDFS - https://phabricator.wikimedia.org/T348958#11027796 (10xcollazo) >>! In T348958#11027140, @mfossati wrote: >>>! In T348958#11021599, @xcollazo wrote: >> >> On {T392244}, we migrat... [15:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [15:27:46] 06Data-Engineering, 10HaproxyKafka, 06Traffic, 13Patch-For-Review: Haproxykafka silently stops sending request data to kafka - https://phabricator.wikimedia.org/T400039#11028321 (10Fabfur) [15:28:23] 06Data-Engineering, 10HaproxyKafka, 06Traffic, 13Patch-For-Review: Haproxykafka silently stops sending request data to kafka - https://phabricator.wikimedia.org/T400039#11028327 (10Fabfur) Adding DE team too, considering that two hosts didn't sent messages for a long time and this could be impacting on the... [15:34:55] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Structured-Data-Backlog: Bump memory to enable large artifacts sync on HDFS - https://phabricator.wikimedia.org/T348958#11028385 (10amastilovic) >>! In T348958#11027140, @mfossati wrote: > Does this mean that a given artifact is automatically deployed as so... [15:36:02] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Experimentation Lab, 10Event-Platform: EventGate: Add Prometheus metric for hoisting errors - https://phabricator.wikimedia.org/T398922#11028392 (10Ottomata) T396359, Right okay! > Would that event include the header that caused the hoisting error? Hm,... [15:36:59] 06Data-Engineering, 06Experimentation Lab, 10Event-Platform: EventGate: Log unparseable X-Experiment-Enrollments headers to a distinct error stream - https://phabricator.wikimedia.org/T396359#11028400 (10Ottomata) The code to modify is the same as in {T398922}. Let's work on these in tandem. [15:54:20] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 07Schema-change, and 2 others: AbuseFilter abuse_filter_log table: Store IP addresses as hex values - https://phabricator.wikimedia.org/T395612#11028476 (10STran) [15:54:46] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 07Schema-change, and 2 others: AbuseFilter abuse_filter_log table: Store IP addresses as hex values - https://phabricator.wikimedia.org/T395612#11028480 (10STran) [15:55:16] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 07Schema-change, and 2 others: AbuseFilter abuse_filter_log table: Store IP addresses as hex values - https://phabricator.wikimedia.org/T395612#11028487 (10STran) Filed {T400280} to better track the last task in our sprints. [16:03:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [16:04:20] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10HaproxyKafka, 06Traffic, 13Patch-For-Review: Haproxykafka silently stops sending request data to kafka - https://phabricator.wikimedia.org/T400039#11028530 (10Ahoelzl) [16:04:35] 06Data-Engineering, 06Data-Platform-SRE: ERROR AsyncEventQueue: Listener DatahubSparkListener threw an exception - https://phabricator.wikimedia.org/T400207#11028534 (10Milimetric) cc @BTullis Andrew says might be a problem with k8s networking [16:05:28] 07Analytics-Data-Problem, 06Data-Engineering, 10Event-Platform, 10MediaWiki-Platform-Team (Radar), 05SUL3: [EventBus] Fix incorrect CentralAuth meta.domain values for relevant streams - https://phabricator.wikimedia.org/T400093#11028538 (10Milimetric) p:05Triage→03Medium [16:05:30] 06Data-Engineering, 06Data-Engineering-Radar, 06Growth-Team, 10GrowthExperiments-NewcomerTasks, and 3 others: Error for mediawiki.cirrussearch-request: '' should NOT have additional properties - https://phabricator.wikimedia.org/T399965#11028540 (10Milimetric) [16:07:34] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10CheckUser, 06Trust and Safety Product Team: Inform Data-Engineering about removal of cuc_ip, cule_ip, and cupe_ip - https://phabricator.wikimedia.org/T399958#11028551 (10Milimetric) [16:07:39] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10CheckUser, 06Trust and Safety Product Team: Inform Data-Engineering about removal of cuc_ip, cule_ip, and cupe_ip - https://phabricator.wikimedia.org/T399958#11028554 (10Milimetric) p:05Triage→03High To Do: * Update sqoop code * Update hql scripts me... [16:08:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [16:08:27] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10CheckUser, 06Trust and Safety Product Team: Inform Data-Engineering about removal of cuc_ip, cule_ip, and cupe_ip - https://phabricator.wikimedia.org/T399958#11028561 (10Ottomata) @amastilovic since you are looking into {T397923}, could you take this too?... [16:11:01] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10CheckUser, 06Trust and Safety Product Team: Inform Data-Engineering about removal of cuc_ip, cule_ip, and cupe_ip - https://phabricator.wikimedia.org/T399958#11028568 (10Dreamy_Jazz) Thanks for looking into this. Let me know if there is anything I can ass... [16:11:40] 10Data-Engineering (Q4 2025 April 1st - June 30th): NEW BUG REPORT wmf.interlanguage_navigation missing mobile data - https://phabricator.wikimedia.org/T396514#11028570 (10Milimetric) p:05Triage→03High a:03Milimetric I gotcha [16:13:17] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [16:15:23] 06Data-Engineering, 10Data-Engineering-Jupyter, 06Data-Platform-SRE: Conda-Analytics has package conflict when trying to install R with key packages (R-Arrow and R-Stringi) - https://phabricator.wikimedia.org/T391911#11028579 (10Milimetric) p:05Triage→03Low Data Engineering is assuming the work-around wi... [16:16:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [16:21:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [16:25:19] 10Data-Engineering (Q4 2025 April 1st - June 30th): Review Image Suggestion pipeline SLOs - https://phabricator.wikimedia.org/T400282 (10Ahoelzl) 03NEW [16:26:17] RESOLVED: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [16:28:07] 10Data-Engineering (Q4 2025 April 1st - June 30th): Streamline and clean up airflow-dags gitlab-ci.yaml CI/CD pipelines - https://phabricator.wikimedia.org/T400283 (10amastilovic) 03NEW [16:28:41] 10Data-Engineering (Q4 2025 April 1st - June 30th): Streamline and clean up airflow-dags gitlab-ci.yaml CI/CD pipelines - https://phabricator.wikimedia.org/T400283#11028639 (10amastilovic) [16:29:39] 06Data-Engineering, 06SRE: WE 5.4 FY 25/26: Improve automata detection at the edge and pass it to the refinery pipeline - https://phabricator.wikimedia.org/T396562#11028642 (10Milimetric) Data Engineering is ready to do or help with this work whenever you need. [16:30:10] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Event-Platform, 13Patch-For-Review: Inconsistent state visibility for Flink deployments on dse-k8s-eqiad - https://phabricator.wikimedia.org/T395984#11028647 (10Milimetric) [16:30:26] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 10MediaWiki-Page-derived-data, and 2 others: Normalize categorylinks table - https://phabricator.wikimedia.org/T299951#11028649 (10Milimetric) [16:30:27] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Experimentation Lab, 10Event-Platform: EventGate: Log unparseable X-Experiment-Enrollments headers to a distinct error stream - https://phabricator.wikimedia.org/T396359#11028652 (10Milimetric) [16:30:58] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Experimentation Lab, 10Event-Platform: EventGate: Log unparseable X-Experiment-Enrollments headers to a distinct error stream - https://phabricator.wikimedia.org/T396359#11028653 (10Milimetric) a:03Ottomata [16:38:06] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 07Schema-change, and 2 others: AbuseFilter abuse_filter_log table: Store IP addresses as hex values - https://phabricator.wikimedia.org/T395612#11028686 (10STran) [16:41:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [16:46:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:06:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:10:36] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Remove m-dot subdomain, serve mobile and desktop variants through the same URL - https://phabricator.wikimedia.org/T214998#11028781 (10Krinkle) a:03Krinkle [17:16:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:21:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:26:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-main in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:31:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-main in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:35:26] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10CheckUser, 06Trust and Safety Product Team: Inform Data-Engineering about removal of cuc_ip, cule_ip, and cupe_ip - https://phabricator.wikimedia.org/T399958#11028915 (10amastilovic) >>! In T399958#11028559, @Ottomata wrote: > @amastilovic since you are l... [17:35:42] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10CheckUser, 06Trust and Safety Product Team: Inform Data-Engineering about removal of cuc_ip, cule_ip, and cupe_ip - https://phabricator.wikimedia.org/T399958#11028916 (10amastilovic) a:03amastilovic [17:41:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-main in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:46:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-main in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:51:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-main in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:52:03] 06Data-Engineering, 10ChangeProp, 06MW-Interfaces-Team, 10Observability-Tracing, and 2 others: Implement tracing across changeprop-jobqueue - https://phabricator.wikimedia.org/T395038#11028940 (10Ottomata) Hm, in beta eventgate-main I now get: ` MaxListenersExceededWarning: Possible EventEmitter memory le... [17:56:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-main in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:07:10] 06Data-Engineering, 10ChangeProp, 06MW-Interfaces-Team, 10Observability-Tracing, and 2 others: Implement tracing across changeprop-jobqueue, kafka, eventgate - https://phabricator.wikimedia.org/T395038#11029015 (10Ottomata) [18:11:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:16:37] 06Data-Engineering, 10ChangeProp, 06MW-Interfaces-Team, 10Observability-Tracing, and 2 others: Implement tracing across changeprop-jobqueue, kafka, eventgate - https://phabricator.wikimedia.org/T395038#11029037 (10Ottomata) @Mate, I [[ https://gitlab.wikimedia.org/repos/data-engineering/eventgate-wikimedia... [18:21:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:31:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:36:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:38:18] (03PS1) 10Milimetric: WIP - migrate, update, and test interlanguage_navigation data [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1172085 (https://phabricator.wikimedia.org/T396514) [18:41:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:42:50] 10Data-Engineering (Q4 2025 April 1st - June 30th), 13Patch-For-Review: NEW BUG REPORT wmf.interlanguage_navigation missing mobile data - https://phabricator.wikimedia.org/T396514#11029139 (10Milimetric) K, @CMyrick-WMF and @Isaac what I'm thinking here is: * add a navigation_type column of the form `previous... [18:46:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:51:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:56:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:01:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:06:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:16:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [19:26:47] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:31:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:36:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:40:53] 06Data-Engineering, 10Event-Platform: Flink images should be build on top of openjdk - https://phabricator.wikimedia.org/T400296 (10gmodena) 03NEW [19:41:47] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:44:04] 06Data-Engineering, 10Event-Platform: Flink images should be build on top of openjdk - https://phabricator.wikimedia.org/T400296#11029359 (10bking) @gmodena it looks like we have a few images with openJDK, such as [[ https://docker-registry.wikimedia.org/openjdk-11-jdk/tags/ | this one ]] . Which version of Ja... [19:56:47] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:56:57] 14Analytics, 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: Adopt conventions for server receive and client/event timestamps in non analytics event schemas - https://phabricator.wikimedia.org/T267648#11029389 (10Ottomata) I'm not totally sure who to notify of this, so @Joe can maybe help? [Eve... [20:02:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [20:17:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [20:24:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [20:29:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-main in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [20:37:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [20:42:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [20:47:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [20:52:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [20:57:17] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:02:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:07:17] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:12:17] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:17:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:27:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:29:22] 06Data-Engineering-Radar, 06Data-Platform-SRE, 10Fundraising analytics stack, 07Epic: [Trino] Connect to Google Sheets - https://phabricator.wikimedia.org/T385748#11029531 (10AStein-WMF) 05Open→03Resolved [21:29:32] 06Data-Engineering-Radar, 06Data-Platform-SRE, 10Fundraising analytics stack, 07Epic: [Trino] Connect to Google Sheets - https://phabricator.wikimedia.org/T385748#11029533 (10AStein-WMF) we decided to use dlt instead of the trino connector [21:42:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:52:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:57:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [22:02:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [22:12:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [22:22:17] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [22:27:22] FIRING: [6x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [22:32:17] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [22:37:17] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [22:57:17] FIRING: [5x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:07:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:12:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:17:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [23:22:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:37:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:42:17] FIRING: [4x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:57:17] FIRING: [6x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-5%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly