[02:23:37] PROBLEM - MariaDB sustained replica lag on db1089 is CRITICAL: 2.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1089&var-port=9104 [02:27:47] RECOVERY - MariaDB sustained replica lag on db1089 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1089&var-port=9104 [05:48:36] 10Blocked-on-schema-change, 10DBA: Increase size of content_models.model_id - https://phabricator.wikimedia.org/T270053 (10Marostegui) [05:48:48] 10Blocked-on-schema-change, 10DBA: Increase size of content_models.model_id - https://phabricator.wikimedia.org/T270053 (10Marostegui) 05Open→03Resolved All done [05:56:45] 10DBA, 10GrowthExperiments, 10Growth-Team (Current Sprint), 10Patch-For-Review, and 2 others: Slow load times for Special:Homepage on cswiki - https://phabricator.wikimedia.org/T267216 (10Marostegui) >>! In T267216#6730372, @Etonkovidova wrote: >>>! In T267216#6728387, @Marostegui wrote: >> We just had ano... [05:57:19] 10DBA, 10SRE, 10ops-codfw: db2140 crashed due to HW memory errors - https://phabricator.wikimedia.org/T271084 (10Marostegui) Thank you Papaul - once it arrives, feel free to replace the DIMM (the host is off) and power it back on. [06:01:19] 10DBA: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10Marostegui) Thanks for the heads up. It looks it went a lot better this time, there was an initial big spike but too worrying and within reasonable values. I see no contention or blockages during the imports. I am happy with... [06:08:52] 10DBA, 10Growth-Team, 10MediaWiki-Watchlist, 10MW-1.36-notes (1.36.0-wmf.25; 2021-01-05), 10Wikimedia-production-error: ClearUserWatchlistJob/WatchedItemStore::removeWatchBatchForUser bad database peformance on enwiki and others, causing database lag - https://phabricator.wikimedia.org/T270481 (10Marosteg... [06:25:02] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) [06:33:23] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) s2 eqiad [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [] db1155 [x] db1146 [x] db1129 [] db1125 [] db1122 [x] db1105 [] db1095 [x] d... [06:38:56] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) [06:54:29] 10DBA, 10Patch-For-Review: Test upgrading sanitarium hosts to Buster + 10.4 - https://phabricator.wikimedia.org/T268742 (10Marostegui) s2 compression on db1155 finished [06:54:37] 10DBA, 10Patch-For-Review: Test upgrading sanitarium hosts to Buster + 10.4 - https://phabricator.wikimedia.org/T268742 (10Marostegui) On-going transfer from db1085 to db1155 s6 replication positions: {P13668} [07:21:48] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) [07:29:30] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) [08:08:59] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) s5 eqiad [x] labsdb1012 [x] labsdb1011 [x] labsdb1010 [x] labsdb1009 [x] dbstore1003 [x] db1154 [x] db1150 [x] db1145 [x] db1144 [x] db1130 [x] db1124 [x] db1... [08:09:01] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) [08:12:55] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) [08:20:32] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) s4 eqiad [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [x] db1150 [] db1149 [] db1148 [] db1147 [x] db1146 [x] db1145 [] db1144 [] d... [09:29:50] I am going to restart tendril [09:29:57] I mean, db1115 [09:32:13] marostegui: restart everything tendril knows about, too, just to be safe [10:33:39] 10DBA, 10Patch-For-Review: Test upgrading sanitarium hosts to Buster + 10.4 - https://phabricator.wikimedia.org/T268742 (10Marostegui) s6 started on db1155, so far no InnoDB errors after the PII removal. Replication is working fine. Triggers also working fine and check_private_data was clean. I am compressing... [11:05:48] 10DBA, 10Patch-For-Review: Test upgrading sanitarium hosts to Buster + 10.4 - https://phabricator.wikimedia.org/T268742 (10Marostegui) [12:11:34] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) [12:25:04] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) a:03Marostegui [13:31:24] 10DBA: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10Marostegui) [13:33:11] 10DBA: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10Marostegui) @akosiaris @jcrespo what about **Thursday 21 at 09:00AM UTC** for the mysql restart of this host? @jcrespo, if this doesn't work cause it can mess up with backup times, let me know so we can look for a more s... [13:33:27] 10DBA: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10Marostegui) p:05Triage→03Medium [13:50:49] 10DBA: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10Marostegui)