[07:06:03] 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 10 others: Restart x1 database master (db1103) - https://phabricator.wikimedia.org/T273758 (10Marostegui) This was done: Master down: 07:00:09 Master up: 07:01:24 [07:06:42] 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 10 others: Restart x1 database master (db1103) - https://phabricator.wikimedia.org/T273758 (10Marostegui) [07:13:12] 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 10 others: Restart x1 database master (db1103) - https://phabricator.wikimedia.org/T273758 (10Marostegui) Closing this as fixed: ` # mysql -e "select @@report_host" +--------------------+ | @@report_host | +--------------------+ | db1103.eqiad... [07:13:18] 10DBA, 10Orchestrator, 10Patch-For-Review, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) [07:13:20] 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 10 others: Restart x1 database master (db1103) - https://phabricator.wikimedia.org/T273758 (10Marostegui) 05Open→03Resolved [07:13:59] 10DBA, 10Orchestrator, 10Patch-For-Review, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) [07:17:27] x1 is now on orchestrator [08:05:38] Nice [08:06:44] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) [08:47:27] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) [09:14:43] no dump failures this week [09:25:08] marostegui: \o/ [09:25:38] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) s4 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [] db1160 [] db1155 [] db1150 [] db1149 [x] db1148 [] db1147... [10:29:07] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) [10:54:05] 10Data-Persistence-Backup, 10SRE, 10SRE-swift-storage, 10Traffic, 10netops: Depool codfw swift cluster - https://phabricator.wikimedia.org/T267338 (10jcrespo) Adding local dc ops on CC of this ticket- things would have to go really bad to needing him for this test (this should be a relatively boring proc... [11:20:50] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) [11:21:45] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) s8 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1005 [] db1172 [] db1154 [] db1126 [] db1124 [x] db1116 [] db1114... [11:54:09] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) [12:40:49] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [12:41:02] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) db1172 is now being automatically pooled into s8 [12:42:29] 10DBA, 10decommission-hardware: decommission db1092.eqiad.wmnet - https://phabricator.wikimedia.org/T275019 (10Marostegui) [12:43:23] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [13:42:55] 10DBA, 10SRE, 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), and 2 others: Create integration test env for wmfmariadbpy - https://phabricator.wikimedia.org/T265266 (10Kormat) Current status: - `integration-env` script created to build docker image, download & cache bin... [13:43:45] 10DBA, 10SRE, 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), and 2 others: Create integration test env for wmfmariadbpy - https://phabricator.wikimedia.org/T265266 (10Kormat) What's not integration-tested yet: - db-compare - db-stop-in-sync - db-switchover [14:22:06] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) [14:31:30] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) s7 progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003 [] db1174 [x] db1170 [] db1155 [] db1136 [x] db1127 [] db1125 [x] d... [14:46:09] 10Blocked-on-schema-change, 10DBA: Schema change for renaming name_title_timestamp on archive table - https://phabricator.wikimedia.org/T273359 (10Marostegui) [20:24:35] 10DBA, 10DC-Ops, 10SRE, 10ops-codfw: (Need By: TBD) rack/setup/install db21[45-52] - https://phabricator.wikimedia.org/T273568 (10Papaul) [22:23:56] PROBLEM - MariaDB sustained replica lag on db1119 is CRITICAL: 4.5 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1119&var-port=9104 [22:31:00] PROBLEM - MariaDB sustained replica lag on db1119 is CRITICAL: 3.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1119&var-port=9104 [22:39:38] RECOVERY - MariaDB sustained replica lag on db1119 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1119&var-port=9104 [22:51:46] PROBLEM - MariaDB sustained replica lag on db1119 is CRITICAL: 2.25 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1119&var-port=9104 [22:55:14] RECOVERY - MariaDB sustained replica lag on db1119 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1119&var-port=9104 [23:51:11] please review that this is correct: https://wikitech.wikimedia.org/w/index.php?title=MariaDB&type=revision&diff=1899258&oldid=1896626