[00:22:14] 06Data-Engineering, 06DBA, 10MediaWiki-Core-Revision-backend, 07Schema-change: Rethink rev_sha1 field - https://phabricator.wikimedia.org/T389026 (10Ladsgroup) 03NEW [00:37:36] 06Data-Engineering, 06DBA, 10MediaWiki-Core-Revision-backend, 07Schema-change: Rethink rev_sha1 field - https://phabricator.wikimedia.org/T389026#10640183 (10Bugreporter) [03:38:45] 06Data-Engineering, 10MediaWiki-Blocks, 10Multiblocks, 06Trust and Safety Product Team, and 2 others: Add a unique index to the block_target table - https://phabricator.wikimedia.org/T389028 (10tstarling) 03NEW [05:02:04] (03CR) 10TChin: Add columns to data_quality_alerts to support inserting ResultKey (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1127967 (https://phabricator.wikimedia.org/T384962) (owner: 10TChin) [06:45:26] 06Data-Engineering: test_produced_by_config SLA miss configured to be too small for upstream dataset run time - https://phabricator.wikimedia.org/T388861#10640318 (10Aklapper) [removing archived project tag and inactive subscribers] [07:33:20] bking: we have an ongoing HelmfileAdminNGPendingChangesDseEqiad alert for a change that would remove all cloudelastic ES services. Is that alright to apply? [08:19:37] 06Data-Engineering, 10MediaWiki-Blocks, 10Multiblocks, 06Trust and Safety Product Team, and 3 others: Add a unique index to the block_target table - https://phabricator.wikimedia.org/T389028#10640547 (10tstarling) [09:10:53] 06Data-Engineering, 10MediaWiki-Blocks, 10Multiblocks, 06Trust and Safety Product Team, and 3 others: Add a unique index to the block_target table - https://phabricator.wikimedia.org/T389028#10640653 (10tstarling) [11:43:39] 06Data-Engineering, 06DBA: Deploy new file tables in production - https://phabricator.wikimedia.org/T383491#10641248 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views run by ladsgroup: Started updating wiki replica views [11:52:42] 06Data-Engineering, 06DBA: Deploy new file tables in production - https://phabricator.wikimedia.org/T383491#10641286 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views started by ladsgroup completed: - an-redacteddb1001.eqiad.wmnet (**PASS**) - Ran Puppet agent - Ran 'maintain-view... [12:16:18] 06Data-Engineering: test_produced_by_config SLA miss configured to be too small for upstream dataset run time - https://phabricator.wikimedia.org/T388861#10641349 (10dr0ptp4kt) Thanks @Aklapper ! @VirginiaPoundstone @Ahoelzl @Gehel the backlink from https://www.mediawiki.org/wiki/Data_Platform_Engineering/Intak... [12:36:58] 06Data-Engineering, 10DPE-Mediawiki-Content, 10Dumps-Generation, 06SRE, and 2 others: Dumps generation cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#10641387 (10BTullis) Hello. Just FYI, we are planning to switch snapshot1016 back to using the core database serv... [13:10:00] 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team, 06serviceops: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10641561 (10Jdforrester-WMF) [13:12:05] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 6 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10641571 (10Lydia_Pintscher) Thank you for furthe... [13:29:02] 06Data-Engineering, 06DBA, 10MediaWiki-Core-Revision-backend, 07Schema-change: Rethink rev_sha1 field - https://phabricator.wikimedia.org/T389026#10641664 (10Ladsgroup) This could be particularly useful for wikidata given revision table growth rate there and how massive already is. The save for wikidata wo... [13:54:18] (03CR) 10Xcollazo: Add columns to data_quality_alerts to support inserting ResultKey (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1127967 (https://phabricator.wikimedia.org/T384962) (owner: 10TChin) [14:00:37] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users group for DSantamaria - https://phabricator.wikimedia.org/T388693#10641782 (10Ottomata) Hi @BCornwall ! group owner approval for analytics-privatedata-users [[ https://github.com/wikimedi... [14:06:13] 06Data-Engineering, 06DBA, 10MediaWiki-Core-Revision-backend, 07Schema-change: Rethink rev_sha1 field - https://phabricator.wikimedia.org/T389026#10641813 (10Ottomata) As this is being considered, please keep in mind that rev_sha1 is used in downstream data pipelines: in the WMF Data Platform, for training... [14:47:15] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users group for DSantamaria - https://phabricator.wikimedia.org/T388693#10642034 (10ssingh) From https://wikitech.wikimedia.org/wiki/SRE/Clinic_Duty/Access_requests#analytics-privatedata-users,... [14:47:31] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users group for DSantamaria - https://phabricator.wikimedia.org/T388693#10642035 (10ssingh) [15:14:59] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06Product-Analytics, 10Event-Platform: [BUG] new eventgate-wikimedia header enrich config loses client set headers - https://phabricator.wikimedia.org/T387908#10642221 (10Ottomata) Note: this task is currently blocking {T383814} as the node 20 upgrade wa... [15:19:33] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06Product-Analytics, 10Event-Platform: [BUG] new eventgate-wikimedia header enrich config loses client set headers - https://phabricator.wikimedia.org/T387908#10642253 (10Ahoelzl) a:03Ottomata [15:24:00] (03PS6) 10Aqu: Extract refinery canaryevents into a separate module [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1120953 [15:38:48] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users group for DSantamaria - https://phabricator.wikimedia.org/T388693#10642387 (10BCornwall) Ugh, sorry about that. But we do still nee @ATsay-WMF to approve, right? [15:41:10] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users group for DSantamaria - https://phabricator.wikimedia.org/T388693#10642395 (10ssingh) >>! In T388693#10642387, @BCornwall wrote: > Ugh, sorry about that. But we do still nee @ATsay-WMF to... [16:05:50] 10Data-Engineering (Q3 2025 January 1st - March 31th), 07Essential-Work: [Data Quality] Implement wiki completeness check for MediaWiki History - https://phabricator.wikimedia.org/T365203#10642473 (10Ottomata) [16:31:41] (03CR) 10Joal: "I like the idea of a "refinery-jvm-jobs" module (scala + java)" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1120953 (owner: 10Aqu) [16:35:07] 06Data-Engineering, 06Discovery-Search, 06serviceops, 10Data-Platform-SRE (2025.03.01 - 2025.03.21): Search Update Pipeline requests to Action API are logged as coming from 127.0.0.1 - https://phabricator.wikimedia.org/T388855#10642657 (10Gehel) [17:11:20] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10MediaWiki-DomainEvents, 10MW-Interfaces-Team (MWI-Roadmap): DomainEvents - [Hypothesis] WE5.2.6 Event Broadcasting Discovery & Design - https://phabricator.wikimedia.org/T384874#10642962 (10Ottomata) Status update: [[ https://docs.google.com/document/d... [17:27:29] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users group for DSantamaria - https://phabricator.wikimedia.org/T388693#10643107 (10ATsay-WMF) Approved [17:28:11] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users group for DSantamaria - https://phabricator.wikimedia.org/T388693#10643112 (10ssingh) [17:40:08] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 6 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10643183 (10xcollazo) >>! In T386255#10641571, @L... [17:51:16] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 6 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10643250 (10Lydia_Pintscher) Thank you! [18:05:05] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users group for DSantamaria - https://phabricator.wikimedia.org/T388693#10643304 (10ssingh) 05In progress→03Resolved @DSantamaria: This has been merged, please try accessing Superset ~30... [18:23:09] 06Data-Engineering, 10MediaWiki-Blocks, 10Multiblocks, 06Trust and Safety Product Team, and 3 others: Add a unique index to the block_target table - https://phabricator.wikimedia.org/T389028#10643354 (10Ladsgroup) From schema point of view, since the table is small. Adding more indexes should be fine. I do... [18:40:51] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 10Charts (Sprint 18), and 2 others: Deploy patch-gjlw_namespace_text.sql on x1.commonswiki for JsonConfig - https://phabricator.wikimedia.org/T385917#10643430 (10Jdlrobson-WMF) This will be deployed later today. [19:21:07] 06Data-Engineering, 10DPE-Mediawiki-Content: Consider whether we want to dump private and/or closed wikis - https://phabricator.wikimedia.org/T371509#10643586 (10xcollazo) [19:23:58] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content: Investigate and fix duplicate data on wmf_content.mediawiki_content_history_v1 for muswiki - https://phabricator.wikimedia.org/T388715#10643605 (10xcollazo) Upon further inspection, `muswiki` is actually closed: ` xcollazo@stat1011... [19:29:24] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content: Investigate and fix duplicate data on wmf_content.mediawiki_content_history_v1 for muswiki - https://phabricator.wikimedia.org/T388715#10643613 (10xcollazo) ` $ whoami analytics $ hostname -f an-launcher1002.eqiad.wmnet $ spark3-s... [19:30:16] !log Ran "DELETE FROM wmf_content.mediawiki_content_history_v1 WHERE wiki_id = 'muswiki'" to delete bad data. T388715. [19:30:19] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:30:19] T388715: Investigate and fix duplicate data on wmf_content.mediawiki_content_history_v1 for muswiki - https://phabricator.wikimedia.org/T388715 [19:39:21] an-druid1003 is almost out of disk space on /srv. A quick look shows that there isn't anything extraneous that can be deleted and that it's all taken up by druid. Anyone already know about this? [19:51:16] 06Data-Engineering, 06Movement-Insights: WMF-last-access cookie sets wrong domain for static assets on commons - https://phabricator.wikimedia.org/T367346#10643698 (10nshahquinn-wmf) p:05Triage→03Low If I'm understanding this correctly, this should not have any impact on our unique devices except in extrem... [19:51:26] 06Data-Engineering, 06Movement-Insights: WMF-last-access cookie sets wrong domain for static assets on commons - https://phabricator.wikimedia.org/T367346#10643701 (10nshahquinn-wmf) Side note: I tried replicating this on en.wikibooks.org (using [this asset](https://en.wikibooks.org/static/images/mobile/copyri... [19:51:42] 06Data-Engineering: WMF-last-access cookie sets wrong domain for static assets on commons - https://phabricator.wikimedia.org/T367346#10643702 (10nshahquinn-wmf) [19:57:31] PROBLEM - Webrequests Varnishkafka log producer on cp4051 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:58:31] RECOVERY - Webrequests Varnishkafka log producer on cp4051 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [20:00:33] 06Data-Engineering: WMF-Last-Access-Global cookie set on wrong domain when accessing static assets - https://phabricator.wikimedia.org/T367346#10643739 (10nshahquinn-wmf) [20:26:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [20:26:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [20:41:54] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE HAProxy Migration: [HAProxy migration] Compile expected migration delta, switch over plan and communicate - https://phabricator.wikimedia.org/T387750#10643992 (10Antoine_Quhen) Some notebooks compiling studies about Pageviews & Unique devices: https:... [20:46:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [20:46:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:35:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:35:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:50:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:50:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag