[05:36:15] 10Blocked-on-schema-change, 10DBA: Schema change to drop three indexes from wb_changes - https://phabricator.wikimedia.org/T264109 (10Marostegui) 05Open→03Stalled All done except codfw master (db2079) as it is the active one. I will complete that one once we are back in eqiad. Stalling until then. ` root@c... [06:17:39] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [06:22:56] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [06:33:18] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) [06:33:20] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) @Ladsgroup I believe this is all done, but I would love if you can re-run the script again for s3, as there were so many different wikis and drifts that I could have easily missed some. [07:47:43] 10DBA, 10decommission-hardware: decommission es2017.codfw.wmnet - https://phabricator.wikimedia.org/T264386 (10Marostegui) [11:51:36] PROBLEM - MariaDB sustained replica lag on db1081 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [11:55:06] RECOVERY - MariaDB sustained replica lag on db1081 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [12:47:46] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Ladsgroup) >>! In T260111#6516106, @Marostegui wrote: > @Ladsgroup I believe this is all done, but I would love if you can re-run the script again for s3, as there were so many different wikis and drifts tha... [12:48:12] 10DBA, 10Epic: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) >>! In T260111#6517158, @Ladsgroup wrote: >>>! In T260111#6516106, @Marostegui wrote: >> @Ladsgroup I believe this is all done, but I would love if you can re-run the script again for s3, as ther...