[01:53:00] 10DBA, 10GrowthExperiments, 10Growth-Team (Current Sprint), 10Patch-For-Review, and 2 others: Slow load times for Special:Homepage on cswiki - https://phabricator.wikimedia.org/T267216 (10Tgr) The code generating these queries has been removed. [02:29:33] PROBLEM - MariaDB sustained replica lag on db1089 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1089&var-port=9104 [02:40:13] RECOVERY - MariaDB sustained replica lag on db1089 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1089&var-port=9104 [04:03:05] 10DBA, 10Add-Link, 10Growth-Structured-Tasks, 10Growth-Team (Current Sprint), 10Patch-For-Review: Add Link engineering: Provide a mechanism for storing data about which link recommendations were rejected by the user - https://phabricator.wikimedia.org/T266446 (10Tgr) Flagging this for DBA review. The tab... [06:11:31] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) [06:11:49] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) 05Openβ†’03Resolved All done [06:26:28] 10DBA, 10Patch-For-Review: Test upgrading sanitarium hosts to Buster + 10.4 - https://phabricator.wikimedia.org/T268742 (10Marostegui) No errors on db1155:3317, compression still on-going [06:26:35] 10DBA, 10Patch-For-Review: Test upgrading sanitarium hosts to Buster + 10.4 - https://phabricator.wikimedia.org/T268742 (10Marostegui) [07:18:32] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in logging table - https://phabricator.wikimedia.org/T270620 (10Marostegui) @Ladsgroup I have found one of these indexes being forced on code, can you double check that piece of code? `page_time`: https://github.com/examknow/CentralC... [09:02:15] 10DBA, 10GrowthExperiments, 10Growth-Team (Current Sprint), 10Patch-For-Review, and 2 others: Slow load times for Special:Homepage on cswiki - https://phabricator.wikimedia.org/T267216 (10Marostegui) @Tgr is that removal live or pending to be pushed? I just saw the same query just now on logstash ` 2021-0... [09:16:31] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) [09:16:53] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) I recall having issues with this table and metadata locking in the past, so we'll see how it goes for the most busy wikis. [09:20:18] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) I have altered db1096:3316 and db1098:3316 first, to see if we run into any indexes issue there. Will leave it running for a couple of days. s6 eqiad [] labs... [09:20:45] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) [09:27:13] 10DBA, 10GrowthExperiments, 10Growth-Team (Current Sprint), 10Patch-For-Review, and 2 others: Slow load times for Special:Homepage on cswiki - https://phabricator.wikimedia.org/T267216 (10kostajh) It’s on the train and should be on cswiki tomorrow. [09:28:52] 10DBA, 10GrowthExperiments, 10Growth-Team (Current Sprint), 10Patch-For-Review, and 2 others: Slow load times for Special:Homepage on cswiki - https://phabricator.wikimedia.org/T267216 (10Marostegui) Great! Thank you [09:37:36] 10Blocked-on-schema-change, 10DBA: Increase size of slot_roles.role_id - https://phabricator.wikimedia.org/T270054 (10Marostegui) a:03Marostegui [09:39:14] 10Blocked-on-schema-change, 10DBA: Increase size of slot_roles.role_id - https://phabricator.wikimedia.org/T270054 (10Marostegui) [09:39:16] 10Blocked-on-schema-change, 10DBA: Increase size of slot_roles.role_id - https://phabricator.wikimedia.org/T270054 (10Marostegui) [09:44:13] 10Blocked-on-schema-change, 10DBA: Increase size of slot_roles.role_id - https://phabricator.wikimedia.org/T270054 (10Marostegui) s6 is codfw is done, leaving it for a few hours before going for eqiad ` /home/marostegui/section s6 | grep codfw | while read host port; do echo "$host:$port"; mysql.py -h$host:$p... [09:45:03] 10Blocked-on-schema-change, 10DBA: Increase size of slot_roles.role_id - https://phabricator.wikimedia.org/T270054 (10Marostegui) [09:49:10] 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host on candidate masters - https://phabricator.wikimedia.org/T271106 (10Marostegui) a:03Marostegui [09:57:17] 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host on candidate masters - https://phabricator.wikimedia.org/T271106 (10Marostegui) enabled on all candidate masters at codfw ` db2080 +--------------------+ | @@report_host | +--------------------+ | db2080.codfw.wmnet | +--------------------+ db2103... [09:57:38] 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host on candidate masters - https://phabricator.wikimedia.org/T271106 (10Marostegui) [10:02:24] 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host on candidate masters - https://phabricator.wikimedia.org/T271106 (10Marostegui) ` root@es1020:~# mysql -e "select @@report_host" +--------------------+ | @@report_host | +--------------------+ | es1020.eqiad.wmnet | +--------------------+ ` [10:02:31] 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host on candidate masters - https://phabricator.wikimedia.org/T271106 (10Marostegui) [10:54:13] 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) es4 eqiad [] es1022 [] es1021 [x] es1020 [10:54:45] 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) [11:31:24] 10DBA, 10SRE: Refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) These is a recap of the new servers we can use to replace the old ones and their rack - I will do a 1:1 replacing planning once I have checked where the servers we have to replace are ra... [12:05:11] marostegui, T270877 is health now, should I resolve or want to do some additional work/checks? [12:05:12] T270877: db2078 m1 mysqld process crashed - https://phabricator.wikimedia.org/T270877 [12:11:13] jynus: let's resolve it! [12:11:21] cool, thanks! [12:14:22] 10DBA: db2078 m1 mysqld process crashed - https://phabricator.wikimedia.org/T270877 (10jcrespo) 05Openβ†’03Resolved a:05jcrespoβ†’03Marostegui Instance m1 on db2078 has been reloaded with data from its master + dumping grants reloaded. Old version is still available on /srv/sqldata.m1.bak. Replication seems... [12:36:34] 10Data-Persistence-Backup: Restart backup related hosts - https://phabricator.wikimedia.org/T271913 (10jcrespo) [14:20:09] Fire alarm went off on my building, may be afk for a while [14:20:22] take care [15:16:29] Back, turned out to be a false alarm :) [15:20:37] cool [15:21:43] Exactly! [15:48:18] sobanski: next time, to be safe, start a small fire yourself on the way out. at least then you'll know it was worth evacuating. [15:48:42] it also stops the fire department from feeling teased [15:49:20] I usually just use my katana to cut the surrounding grass [15:50:24] This way the fire can go around me [15:50:33] smart. :) [15:50:36] plasterer working here told a story that his plaster work once triggered the smoked detectors in a building and dispatched the fire dept [15:50:42] which was Not Pleased when he was laughing hard [17:32:45] 10DBA, 10Add-Link, 10Growth-Structured-Tasks, 10Growth-Team (Current Sprint), 10Patch-For-Review: Add Link engineering: Provide a mechanism for storing data about which link recommendations were rejected by the user - https://phabricator.wikimedia.org/T266446 (10kostajh) >>! In T266446#6742131, @Tgr wrot... [18:25:26] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Ladsgroup) It looks good. Thank you! >>! In T270621#6742756, @Marostegui wrote: > I recall having issues with this table and metadata locking in the past, so we'll see ho... [18:26:54] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) Now that I think about it, maybe it was site_stats [18:37:48] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Ladsgroup) Yeah that table is being read A LOT and I think it needs a rewrite but that's waay down the road [19:06:27] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in logging table - https://phabricator.wikimedia.org/T270620 (10Ladsgroup) The first link doesn't matter as it's not deployed in our systems AFAIK The second one actually has changed to accommodates this change. See https://gerrit.w... [19:29:27] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in logging table - https://phabricator.wikimedia.org/T270620 (10RhinosF1) I'll contact Manual re the first one shortly. [19:29:37] marostegui: mind a PM? [19:36:28] I know it's probably out of office hours for you so please drop me an email (rhinosf1[at]gmail.com) if I'm not online when you reply