[00:47:38] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [04:47:38] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [06:06:46] 10Quarry: Quarry down - web service unreachable - https://phabricator.wikimedia.org/T395201#10894364 (10Liz) 05Resolved→03Open It's happening again! The exact same thing is happening again. [06:14:57] 10Quarry: [bug] Quarry down? - https://phabricator.wikimedia.org/T396295#10894368 (10SD0001) →14Duplicate dup:03T395201 [06:14:58] 10Quarry: Quarry down - web service unreachable - https://phabricator.wikimedia.org/T395201#10894370 (10SD0001) [08:47:38] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [08:58:28] I'm going to merge this -> https://gerrit.wikimedia.org/r/c/operations/puppet/+/1148373 [08:58:46] it shouldn't have any impact but better to notify [09:18:52] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [09:28:52] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [11:22:15] 06Data-Engineering, 06Diffusion-Repository-Administrators, 10Dumps-Generation, 10Projects-Cleanup, and 2 others: Undeploy and archive ActiveAbstract - https://phabricator.wikimedia.org/T382069#10895047 (10Ahtula) I'm presently reviewing [[ https://github.com/validator/validator | validator.nu ]] (html5 val... [12:05:33] 06Data-Engineering, 06Data-Persistence, 10Data-Platform, 10Data-Services, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10895105 (10fnegri) 05Open→03In progress [12:20:21] 06Data-Engineering, 06Diffusion-Repository-Administrators, 10Dumps-Generation, 10Projects-Cleanup, and 2 others: Undeploy and archive ActiveAbstract - https://phabricator.wikimedia.org/T382069#10895126 (10Ahtula) Could you by the way confirm what the license of the files computationally derived from abstra... [12:28:11] 06Data-Engineering, 06Data-Persistence, 10Data-Platform, 10Data-Services, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10895147 (10fnegri) [13:14:49] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10895285 (10Marostegui) [13:29:46] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [13:41:27] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06cloud-services-team, 10Cloud-VPS, 07IPv6: Add new WMCS IP ranges to analytics - https://phabricator.wikimedia.org/T392468#10895435 (10taavi) May I ask what's the status of this? The task itself is open and the checkboxes in the description are still mar... [13:51:36] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Movement-Insights: Investigate rise in May 2025 unique devices and reader metrics - https://phabricator.wikimedia.org/T395934#10895461 (10OSefu-WMF) [[ https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(miscellaneous)#c-Cremastra-20250607012600-Siteviews... [13:53:14] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Movement-Insights: Investigate rise in May 2025 unique devices and reader metrics - https://phabricator.wikimedia.org/T395934#10895467 (10OSefu-WMF) Some results from our initial analysis [14:22:23] 06Data-Engineering, 10Dumps-Generation: wikidatawiki fails dumps of the wbt_* tables - https://phabricator.wikimedia.org/T396125#10895545 (10xcollazo) This did not recover, but we continue to get the email warnings for old files. Thus removed them: ` ssh snapshot1011.eqiad.wmnet sudo -u dumpsgen bash cd /mn... [14:31:57] 06Data-Engineering, 06Data-Persistence, 10Data-Platform, 10Data-Services, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10895604 (10fnegri) [14:42:08] 06Data-Engineering, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10895702 (10Marostegui) [14:42:53] 06Data-Engineering, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10895707 (10Marostegui) [15:02:04] 06Data-Engineering, 06Data-Engineering-Radar, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-user for Anton Kokh (WMDE) - https://phabricator.wikimedia.org/T395917#10895822 (10elukey) @Anton.Kokh Hi! To keep archives happy - are you going to follow up with @K... [15:07:32] 10Data-Engineering (Q4 2025 April 1st - June 30th): Determine how many admins are there in English Wikipedia and French Wikipedia from sub-Saharan Africa - https://phabricator.wikimedia.org/T395279#10895841 (10mforns) a:03mforns [16:03:46] 06Data-Engineering, 06Data-Persistence, 10Data-Platform, 10Data-Services, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10896236 (10fnegri) clouddb1015 is upgraded and repooled! I will continue with clouddb1019 tomorrow. [16:40:21] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Movement-Insights, 06Traffic: Investigate rise in May 2025 unique devices and reader metrics - https://phabricator.wikimedia.org/T395934#10896375 (10sbassett) [17:33:13] 10Data-Engineering (Q4 2025 April 1st - June 30th): Determine how many admins are there in English Wikipedia and French Wikipedia from sub-Saharan Africa - https://phabricator.wikimedia.org/T395279#10896564 (10Asaf) (We have since come up with a couple of ENWP admins who self-report as being from sub-Saharan Afr... [17:33:29] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [18:00:09] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Movement-Insights, 06Traffic: NEW BUG REPORT: Investigate rise in May 2025 Reader metrics - https://phabricator.wikimedia.org/T395934#10896672 (10Mayakp.wiki) p:05Medium→03High [18:04:23] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10896717 (10Marostegui) [18:04:48] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10896719 (10Marostegui) [18:41:38] 06Data-Engineering, 06Diffusion-Repository-Administrators, 10Dumps-Generation, 10Projects-Cleanup, and 3 others: Undeploy and archive ActiveAbstract - https://phabricator.wikimedia.org/T382069#10896921 (10Novem_Linguae) [18:50:37] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Movement-Insights, 06Traffic: NEW BUG REPORT: Investigate rise in May 2025 Reader metrics - https://phabricator.wikimedia.org/T395934#10896970 (10OSefu-WMF) **Retention window extension** - In anticipation of the need for a backfill and to give teams more... [19:00:07] 06Data-Engineering, 06Diffusion-Repository-Administrators, 10Dumps-Generation, 10Projects-Cleanup, and 3 others: Undeploy and archive ActiveAbstract - https://phabricator.wikimedia.org/T382069#10896989 (10Pppery) [19:11:48] 06Data-Engineering, 06Diffusion-Repository-Administrators, 10Dumps-Generation, 10Projects-Cleanup, and 3 others: Undeploy and archive ActiveAbstract - https://phabricator.wikimedia.org/T382069#10896999 (10Novem_Linguae) [19:14:50] 06Data-Engineering, 06Diffusion-Repository-Administrators, 10Dumps-Generation, 10Projects-Cleanup, and 3 others: Undeploy and archive ActiveAbstract - https://phabricator.wikimedia.org/T382069#10897001 (10Novem_Linguae) I think the last 3 unmarked items in the checklist at the top of this ticket need someo... [20:26:51] 06Data-Engineering, 06Data-Engineering-Radar, 10Data-Services, 06DBA, 06Privacy Engineering: Create views for SecurePoll db tables in Toolforge replicas - https://phabricator.wikimedia.org/T381197#10897305 (10Novem_Linguae) [21:33:29] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem