[05:22:33] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Degraded RAID on es2026 - https://phabricator.wikimedia.org/T263837 (10Marostegui) Just for the record, the rebuilt process for the new disk finished correctly: ` root@es2026:~# megacli -LDInfo -L0 -a0 Adapter 0 -- Virtual Drive Information: Virtual D... [06:25:57] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Marostegui) [06:26:20] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Marostegui) a:03Marostegui [06:41:27] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Marostegui) The event scheduler has been checked across all hosts in eqiad, and it was ON everywhere. [06:41:45] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Marostegui) [07:05:55] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Marostegui) events are enabled everywhere within eqiad [07:06:06] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Marostegui) [07:56:06] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Marostegui) Notifications are enabled everywhere within eqiad [07:56:17] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Marostegui) [09:34:13] 10DBA, 10Data-Services, 10MediaWiki-General, 10Security: Make (redacted) log_search table available on Wiki Replicas - https://phabricator.wikimedia.org/T85756 (10LSobanski) We need to check what has been done here and what actions are pending. [09:38:34] 10DBA, 10Privacy Engineering, 10WMF-Legal, 10Performance Issue, and 2 others: Deploy access to performance_schema/sys for the administrative mediawiki account (mediawiki deployers) - https://phabricator.wikimedia.org/T195578 (10LSobanski) Pending sign-off from #security-team. [09:54:20] 10DBA, 10AbuseFilter, 10Stewards-and-global-tools, 10Patch-For-Review, 10User-Daimona: Addition of last hit date to Special:AbuseFilter table - https://phabricator.wikimedia.org/T93564 (10LSobanski) 05Open→03Resolved a:03LSobanski Resolving. Please reopen when a #blocked-on-schema-change ticket is... [09:55:42] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): enwiki_p logging vs logging_userindex returning dramatically different results - https://phabricator.wikimedia.org/T168349 (10Marostegui) [10:35:13] I don't know who added the sections to motd, but that is awsome!!! [10:35:32] Stevie did [10:36:42] jynus: glad you like it :) [10:37:11] we lost that when we move sections out of the role name [10:37:29] *had lost *moved [10:37:30] yeah exactly. and it never covered multi-instance hosts either [10:39:32] does dev.mysql.org work for you? [10:39:50] it does not [10:40:05] NXDOMAIN [10:40:22] sorry, I meant .com [10:40:28] but it seams it is back now [10:40:28] works for me [10:40:32] it timed out before [10:40:44] probably a schema change [10:42:08] nah, I think they serve those pages with oracle (not kidding) [12:10:09] 10DBA, 10Data-Services: Prepare and check storage layer for smnwiki - https://phabricator.wikimedia.org/T264900 (10Urbanecm) @Marostegui @LSobanski Hello, I've just created the database. Can you please do the needful? [12:10:43] 10DBA, 10Data-Services: Prepare and check storage layer for smnwiki - https://phabricator.wikimedia.org/T264900 (10Marostegui) a:03Marostegui Thank you. I will apply the alter, sanitization etc Thanks! [12:28:29] 10DBA, 10Data-Services: Prepare and check storage layer for smnwiki - https://phabricator.wikimedia.org/T264900 (10Marostegui) Alter done. ` root@db1082.eqiad.wmnet[smnwiki]> show create table ipblocks_restrictions; +-----------------------+----------------------------------------------------------------------... [13:15:23] 10DBA, 10Operations, 10User-Kormat: Convert role::mariadb::misc to profile - https://phabricator.wikimedia.org/T265900 (10Kormat) [13:21:27] 10DBA, 10Operations, 10User-Kormat: Clean up role::mariadb::ferm and profile::mariadb::ferm - https://phabricator.wikimedia.org/T265901 (10Kormat) [13:34:33] 10DBA, 10Data-Persistence, 10Operations, 10User-Kormat: Clean up role::mariadb::ferm and profile::mariadb::ferm - https://phabricator.wikimedia.org/T265901 (10Marostegui) p:05Triage→03Medium [13:55:45] 10DBA, 10Operations, 10Patch-For-Review, 10User-Kormat: Convert role::mariadb::misc to profile - https://phabricator.wikimedia.org/T265900 (10LSobanski) p:05Triage→03Medium [13:56:08] 10DBA, 10Data-Services: Prepare and check storage layer for smnwiki - https://phabricator.wikimedia.org/T264900 (10LSobanski) p:05Triage→03Medium [13:56:24] 10DBA, 10Data-Persistence, 10Growth-Structured-Tasks, 10Growth-Team: Add a link engineering: Determine format for accessing and storing link recommendations - https://phabricator.wikimedia.org/T261411 (10LSobanski) p:05Triage→03Medium [14:07:40] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for smnwiki - https://phabricator.wikimedia.org/T264900 (10Marostegui) check_private_data came back clean. The following has been done: - created `smnwiki_p` - added grants to labsdbuser for `smnwiki_p`. This is ready fo... [16:01:31] 10DBA, 10AbuseFilter, 10Stewards-and-global-tools, 10Patch-For-Review, 10User-Daimona: Addition of last hit date to Special:AbuseFilter table - https://phabricator.wikimedia.org/T93564 (10Daimona) 05Resolved→03Open @LSobanski I'm sorry, this wasn't meant as a #schema-change task, but rather as a feat... [16:08:10] PROBLEM - MariaDB sustained replica lag on db1081 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [16:08:54] PROBLEM - MariaDB sustained replica lag on db2073 is CRITICAL: 3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2073&var-port=9104 [16:09:37] 10DBA, 10AbuseFilter, 10Stewards-and-global-tools, 10Patch-For-Review, 10User-Daimona: Addition of last hit date to Special:AbuseFilter table - https://phabricator.wikimedia.org/T93564 (10Daimona) [16:11:19] we had a spike on s4 ^ [16:14:30] PROBLEM - MariaDB sustained replica lag on db1121 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [16:14:30] RECOVERY - MariaDB sustained replica lag on db2073 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2073&var-port=9104 [16:15:28] 10DBA, 10AbuseFilter, 10Stewards-and-global-tools, 10Patch-For-Review, 10User-Daimona: Addition of last hit date to Special:AbuseFilter table - https://phabricator.wikimedia.org/T93564 (10Daimona) 05Open→03Stalled a:05LSobanski→03Daimona Based on some quick experiments, using the afl_filter_id +... [16:16:00] RECOVERY - MariaDB sustained replica lag on db1121 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [16:16:42] RECOVERY - MariaDB sustained replica lag on db1081 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [16:18:24] 10DBA, 10AbuseFilter, 10Stewards-and-global-tools, 10Patch-For-Review, 10User-Daimona: Addition of last hit date to Special:AbuseFilter table - https://phabricator.wikimedia.org/T93564 (10Marostegui) @Daimona do we have a schema change task for that? I have not been able to find it under #blocked-on-sche... [16:20:29] it was an increase in deletes? [16:20:52] https://grafana.wikimedia.org/d/000000273/mysql?viewPanel=3&orgId=1&var-server=db2073&var-port=9104 [16:21:32] we've had that for the last few days [16:21:42] oh, and do you know why? [16:21:45] from the start of past week I reckon [16:21:47] maintenance? [16:22:00] I think there was some clean up going on [16:23:14] user-driven or devel-driven? [16:24:30] user [16:24:58] ok, thanks [16:27:38] it could have been: https://commons.wikimedia.org/wiki/Category:Creative_Commons_Attribution_2.0_missing_SDC_copyright_license [16:27:55] that would fit with "renaming a category with lots of pages" [16:29:47] if it keeps happening frequently I will report it and create a task [16:54:40] 10DBA, 10AbuseFilter, 10Stewards-and-global-tools, 10Patch-For-Review, 10User-Daimona: Addition of last hit date to Special:AbuseFilter table - https://phabricator.wikimedia.org/T93564 (10Daimona) >>! In T93564#6561104, @Marostegui wrote: > @Daimona do we have a schema change task for that? I have not be... [16:55:44] 10DBA, 10AbuseFilter, 10Stewards-and-global-tools, 10Patch-For-Review, 10User-Daimona: Addition of last hit date to Special:AbuseFilter table - https://phabricator.wikimedia.org/T93564 (10Marostegui) >>! In T93564#6561295, @Daimona wrote: >>>! In T93564#6561104, @Marostegui wrote: >> @Daimona do we have... [20:07:45] PROBLEM - MariaDB sustained replica lag on db1081 is CRITICAL: 82.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [20:10:51] PROBLEM - MariaDB sustained replica lag on db1138 is CRITICAL: 20.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1138&var-port=9104 [20:15:13] PROBLEM - MariaDB sustained replica lag on db1121 is CRITICAL: 50.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [20:16:03] PROBLEM - MariaDB sustained replica lag on db1143 is CRITICAL: 52.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1143&var-port=9104 [20:16:57] RECOVERY - MariaDB sustained replica lag on db1121 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [20:17:43] RECOVERY - MariaDB sustained replica lag on db1138 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1138&var-port=9104 [20:17:47] RECOVERY - MariaDB sustained replica lag on db1143 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1143&var-port=9104 [20:33:43] RECOVERY - MariaDB sustained replica lag on db1081 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [21:23:21] 10DBA, 10MediaWiki-Parser, 10Parsoid, 10serviceops, 10Platform Team Workboards (Green): CAPEX for ParserCache for Parsoid - https://phabricator.wikimedia.org/T263587 (10Pchelolo) I guess we have to begin here. TLDR of the problem is that we will not have enough space in MySQL for ParserCache for transi...