[00:43:35] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [04:43:35] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [07:01:01] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for Neslihan_Turan_WMDE - https://phabricator.wikimedia.org/T394395#10882658 (10SLyngshede-WMF) 05Stalled→03Resolved [07:08:34] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, 06Privacy Engineering: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#10882664 (10Gehel) [07:33:54] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-user for Anton Kokh (WMDE) - https://phabricator.wikimedia.org/T395917#10882723 (10SLyngshede-WMF) We need an approval from: @Ottomata @Ahoelzl or @Milimetric for access to analytics-privatedata-... [08:02:10] 06Data-Engineering, 06Data-Persistence, 10Data-Platform, 10Data-Services, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10882766 (10Marostegui) We are going to start with s6 [08:43:35] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [08:57:46] 06Data-Engineering, 06Data-Persistence, 10Data-Platform, 10Data-Services, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10882911 (10fnegri) [09:09:04] 06Data-Engineering, 06Data-Persistence, 10Data-Platform, 10Data-Services, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10882935 (10fnegri) [09:13:37] 06Data-Engineering, 06Data-Persistence, 10Data-Platform, 10Data-Services, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10882975 (10fnegri) > We are going to start with s6 Ok, then I will start with clouddb1015 and clouddb1019 which are the two clouddbs... [09:17:54] 06Data-Engineering, 06Data-Persistence, 10Data-Platform, 10Data-Services, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10882978 (10Marostegui) The steps look good to me, thank you [11:38:33] 06Data-Engineering, 10AbuseFilter, 06Data-Persistence, 06Trust and Safety Product Team, and 4 others: Create afl_ip_hex column in the abuse_filter_log table - https://phabricator.wikimedia.org/T395613#10883372 (10Dreamy_Jazz) a:03Dreamy_Jazz [12:43:35] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [13:58:14] 06Data-Engineering, 10DPE-Mediawiki-Content: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031 (10xcollazo) 03NEW [14:25:43] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031#10884214 (10Ahoelzl) [14:46:29] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031#10884321 (10xcollazo) [14:46:33] 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 07Epic: Dumps 2.0 Phase III: Production level dumps - https://phabricator.wikimedia.org/T366752#10884322 (10xcollazo) [14:51:37] 06Data-Engineering, 06Data-Platform-SRE, 10Event-Platform: Inconsistent state visibility for Flink deployments on dse-k8s-eqiad - https://phabricator.wikimedia.org/T395984#10884341 (10bking) 05Open→03In progress p:05Triage→03Medium [14:52:07] 06Data-Engineering, 10Data-Platform-SRE (2025.05.24 - 2025.06.13), 10Event-Platform: Inconsistent state visibility for Flink deployments on dse-k8s-eqiad - https://phabricator.wikimedia.org/T395984#10884343 (10bking) [15:06:48] 06Data-Engineering, 10Data-Platform-SRE (2025.05.24 - 2025.06.13), 10Event-Platform: Inconsistent state visibility for Flink deployments on dse-k8s-eqiad - https://phabricator.wikimedia.org/T395984#10884403 (10BTullis) p:05Medium→03High a:03BTullis [15:11:58] 06Data-Engineering, 10Data-Platform-SRE (2025.05.24 - 2025.06.13), 10Event-Platform: Inconsistent state visibility for Flink deployments on dse-k8s-eqiad - https://phabricator.wikimedia.org/T395984#10884447 (10xcollazo) Seems like production Flink apps like [[ https://grafana.wikimedia.org/d/K9x0c4aVk/flink-... [15:21:35] 06Data-Engineering, 10Data-Platform-SRE (2025.05.24 - 2025.06.13), 10Event-Platform: Inconsistent state visibility for Flink deployments on dse-k8s-eqiad - https://phabricator.wikimedia.org/T395984#10884484 (10bking) a:05BTullis→03bking [15:23:34] 06Data-Engineering, 10Data Pipelines: Provide an easy way for MediaWiki to fetch aggregate statistics from the data lake - https://phabricator.wikimedia.org/T341649#10884487 (10BTullis) >>! In T341649#10659194, @Ottomata wrote: > BTW I gave a [[ https://docs.google.com/presentation/d/1cum7oFGN_pdx3za3hHgYYRkqO... [16:03:45] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-user for Anton Kokh (WMDE) - https://phabricator.wikimedia.org/T395917#10884678 (10Ahoelzl) Approved. [16:04:01] 06Data-Engineering, 06Data-Engineering-Radar, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-user for Anton Kokh (WMDE) - https://phabricator.wikimedia.org/T395917#10884680 (10Ahoelzl) [16:08:28] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, 06Privacy Engineering: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#10884693 (10VirginiaPoundstone) @fnegri can you remind me what initiative this supports? Better yet add the context to... [16:10:10] 14Analytics, 06Data-Engineering, 10Data-Engineering-Wikistats, 06Movement-Insights: 0 unique devices for past month on all projects - https://phabricator.wikimedia.org/T395727#10884710 (10Ahoelzl) @OSefu-WMF can you investigate? [16:10:13] 14Analytics, 06Data-Engineering, 10Data-Engineering-Wikistats, 06Movement-Insights: 0 unique devices for past month on all projects - https://phabricator.wikimedia.org/T395727#10884711 (10VirginiaPoundstone) [16:10:43] 14Analytics, 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Engineering-Wikistats, 06Movement-Insights: 0 unique devices for past month on all projects - https://phabricator.wikimedia.org/T395727#10884714 (10Ahoelzl) [16:11:00] 14Analytics, 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Engineering-Wikistats, 06Movement-Insights: 0 unique devices for past month on all projects - https://phabricator.wikimedia.org/T395727#10884716 (10Ahoelzl) p:05Triage→03High [16:13:32] 06Data-Engineering, 06Data-Engineering-Radar, 10CheckUser, 06Trust and Safety Product Team, 07Schema-change: Add '*_actor_ip_hex_time' indexes to 'cu_changes', 'cu_log_event', and 'cu_private_event' - https://phabricator.wikimedia.org/T395683#10884726 (10Ahoelzl) [16:16:15] 06Data-Engineering, 06Data-Engineering-Radar, 10CheckUser, 06Trust and Safety Product Team, 07Schema-change: Add '*_actor_ip_hex_time' indexes to 'cu_changes', 'cu_log_event', and 'cu_private_event' - https://phabricator.wikimedia.org/T395683#10884750 (10Ahoelzl) @mforns can you evaluate if removing the... [16:17:38] 06Data-Engineering, 06DBA, 10GlobalBlocking, 06Trust and Safety Product Team, 07Schema-change: globalblocks table: SQL in extension and production have different type for gb_address - https://phabricator.wikimedia.org/T395669#10884777 (10Ahoelzl) @JAllemandou does this have an impact on DE? [16:22:30] 14Analytics, 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Engineering-Wikistats, 06Movement-Insights: 0 unique devices for past month on all projects - https://phabricator.wikimedia.org/T395727#10884822 (10OSefu-WMF) Thanks @Alien333 and @Ahoelzl - We have been tracking this and are investig... [16:27:26] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list May 2025 - https://phabricator.wikimedia.org/T395639#10884875 (10Ahoelzl) [16:27:59] 06Data-Engineering, 10Dumps-Generation: Migrate format of pageview dumps to replace two or more consecutive zeros by char caret and then the count of the zeros - https://phabricator.wikimedia.org/T395649#10884884 (10VirginiaPoundstone) 05Open→03Declined @Dusan_Krehel Thank you for sharing the D0CMF pag... [16:30:12] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 06Trust and Safety Product Team, and 2 others: AbuseFilter abuse_filter_log table: Store IP addresses as hex values - https://phabricator.wikimedia.org/T395612#10884905 (10Ahoelzl) [16:35:54] 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team, 06serviceops: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10884949 (10Jdforrester-WMF) [16:39:28] 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team, 06serviceops: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10884963 (10Jdforrester-WMF) [16:40:12] 06Data-Engineering: Airflow: MariaDB to Hive to operator - https://phabricator.wikimedia.org/T395554#10884969 (10Ahoelzl) p:05Triage→03Medium [16:42:37] 06Data-Engineering: AlertLintProblem - https://phabricator.wikimedia.org/T395539#10884985 (10Ahoelzl) @gmodena relevant to DE? [16:43:35] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [16:48:07] 14Analytics, 07Analytics-Data-Problem, 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Engineering-Wikistats, 06Movement-Insights: 0 unique devices for past month on all projects - https://phabricator.wikimedia.org/T395727#10885007 (10nshahquinn-wmf) [16:52:42] 14Analytics, 07Analytics-Data-Problem, 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Engineering-Wikistats, 06Movement-Insights: 0 unique devices for past month on all projects - https://phabricator.wikimedia.org/T395727#10885027 (10Mayakp.wiki) I checked wikistats and I am not seeing zero u... [17:00:29] 10Data-Engineering-Roadmap, 07Epic: [Epic] Instrument pageviews using events, instead of webrequests - https://phabricator.wikimedia.org/T371321#10885039 (10Ahoelzl) [17:34:38] 06Data-Engineering, 10Data-Platform-SRE (2025.05.24 - 2025.06.13), 10Event-Platform, 13Patch-For-Review: Inconsistent state visibility for Flink deployments on dse-k8s-eqiad - https://phabricator.wikimedia.org/T395984#10885231 (10BTullis) I'm going to try to redeploy the flink-operator. There is some docum... [17:38:04] 06Data-Engineering, 10Data-Platform-SRE (2025.05.24 - 2025.06.13), 10Event-Platform, 13Patch-For-Review: Inconsistent state visibility for Flink deployments on dse-k8s-eqiad - https://phabricator.wikimedia.org/T395984#10885236 (10BTullis) I have destroyed and re-applied it. ` root@deploy1003:/srv/deploymen... [17:39:01] 06Data-Engineering: AlertLintProblem - https://phabricator.wikimedia.org/T395539#10885238 (10gmodena) Looks related to this ops week report from @dr0ptp4kt https://wikimedia.slack.com/archives/C055QGPTC69/p1747860685199499?thread_ts=1747860523.871779&cid=C055QGPTC69 [17:44:41] 10Analytics-Canonical-Data, 06Data-Engineering, 06Data-Engineering-Icebox, 06Movement-Insights: Automatically update the canonical data tables - https://phabricator.wikimedia.org/T339928#10885253 (10mpopov) For wikis.tsv, what about moving that out of the GitLab repo and into analytics.wikimedia.org/publis... [17:47:58] 07Analytics-Data-Problem, 06Trust and Safety Product Team, 10Product-Analytics (Kanban): Rerun trust_safety_metrics queries for March 2025 - https://phabricator.wikimedia.org/T396060 (10nettrom_WMF) 03NEW [17:55:26] 10Analytics-Canonical-Data, 06Data-Engineering, 06Data-Engineering-Icebox, 06Movement-Insights: Automatically update the canonical data tables - https://phabricator.wikimedia.org/T339928#10885331 (10nshahquinn-wmf) >>! In T339928#10885253, @mpopov wrote: > For wikis.tsv, what about moving that out of the G... [17:59:28] 07Analytics-Data-Problem, 06Trust and Safety Product Team, 10Product-Analytics (Kanban): Rerun trust_safety_metrics queries for March 2025 - https://phabricator.wikimedia.org/T396060#10885343 (10nettrom_WMF) [18:00:26] 10Analytics-Canonical-Data, 06Data-Engineering, 06Data-Engineering-Icebox, 06Movement-Insights: Automatically update the canonical data tables - https://phabricator.wikimedia.org/T339928#10885346 (10mpopov) Yep! See `publish_tsv` task in https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/bl... [18:00:27] 07Analytics-Data-Problem, 06Trust and Safety Product Team, 10Product-Analytics (Kanban): Rerun trust_safety_metrics queries for March 2025 - https://phabricator.wikimedia.org/T396060#10885347 (10nettrom_WMF) [18:00:49] 07Analytics-Data-Problem, 06Trust and Safety Product Team, 10Product-Analytics (Kanban): Rerun trust_safety_metrics queries for March 2025 - https://phabricator.wikimedia.org/T396060#10885353 (10nettrom_WMF) Might be easier to do this through Airflow. Waiting for the DAG errors to go away first, though. [18:01:48] 06Data-Engineering, 10Data Pipelines: Provide an easy way for MediaWiki to fetch aggregate statistics from the data lake - https://phabricator.wikimedia.org/T341649#10885356 (10HNordeenWMF) [18:13:26] 14Analytics, 07Analytics-Data-Problem, 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Engineering-Wikistats, 06Movement-Insights: Sharp spike in unique devices for past month on all projects - https://phabricator.wikimedia.org/T395727#10885385 (10VirginiaPoundstone) [18:14:34] 14Analytics, 07Analytics-Data-Problem, 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Engineering-Wikistats, 06Movement-Insights: Sharp spike in unique devices for past month on all projects - https://phabricator.wikimedia.org/T395727#10885389 (10VirginiaPoundstone) Changes task name to "Shar... [19:15:51] 14Analytics, 07Analytics-Data-Problem, 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Engineering-Wikistats, 06Movement-Insights: Sharp spike in unique devices for past month on all projects - https://phabricator.wikimedia.org/T395727#10885576 (10Alien333) (For some projects, 0 was reported,... [20:43:35] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [20:53:49] 07Analytics-Data-Problem, 06Trust and Safety Product Team, 10Product-Analytics (Kanban): Rerun trust_safety_metrics queries for March 2025 - https://phabricator.wikimedia.org/T396060#10885860 (10nettrom_WMF) [20:56:45] 07Analytics-Data-Problem, 06Trust and Safety Product Team, 10Product-Analytics (Kanban): Rerun trust_safety_metrics queries for March 2025 - https://phabricator.wikimedia.org/T396060#10885865 (10nettrom_WMF) 05Open→03Resolved This has been completed by rerunning the Airflow DAGs for that snapshot. Th... [21:26:43] 10Data-Engineering (Q4 2025 April 1st - June 30th): Define SDS 2.1.11 hypothesis - ExP threats and mitigations - https://phabricator.wikimedia.org/T396078 (10Ahoelzl) 03NEW [21:51:29] 10Data-Engineering (Q4 2025 April 1st - June 30th): Define SDS 2.1.11 hypothesis - ExP threats and mitigations - https://phabricator.wikimedia.org/T396078#10886035 (10Ahoelzl) [22:28:38] 10Data-Engineering (Q4 2025 April 1st - June 30th): Define SDS 2.1.11 hypothesis - ExP threats and mitigations - https://phabricator.wikimedia.org/T396078#10886088 (10Ahoelzl) For current data platform access control see: https://wikitech.wikimedia.org/wiki/Data_Platform/Data_access > limit experiment tables to...