[06:08:54] 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [06:08:56] 10DBA, 10Epic, 10Patch-For-Review: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10Marostegui) [06:08:58] 10DBA, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10Marostegui) [06:09:13] 10DBA, 10Patch-For-Review: Test upgrading sanitarium hosts to Buster + 10.4 - https://phabricator.wikimedia.org/T268742 (10Marostegui) 05Open→03Resolved db1155 ran its automatic data check with no issues. So everything is completed. Next step is to move all the new clouddb hosts under these new sanitariums... [06:10:49] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) Schema change started on s3 master, it will take around 15 hours to get completed. [06:14:06] 10DBA, 10Wikimedia-Etherpad: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10Marostegui) All hosts but the master have `report_host` enabled. ` ./section m1 | while read host port; do echo "=== $port:$port ===";mysql.py -h$host:$port -e "select @@report_host";done === 3306... [07:09:29] 10DBA: Switchover s4 (commonswiki) from db1081 to db1138 - https://phabricator.wikimedia.org/T271427 (10Marostegui) Upgraded the kernel on db1138 as part of: T272255 [07:10:05] 10DBA, 10SRE: Refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [08:16:36] 10DBA, 10cloud-services-team (Kanban): Move wikireplicas under the new sanitarium hosts (db1154, db1155) - https://phabricator.wikimedia.org/T272008 (10Marostegui) a:03Marostegui [08:46:51] 10DBA, 10Orchestrator, 10SRE, 10CAS-SSO, 10User-Kormat: orchestrator: Support SSO - https://phabricator.wikimedia.org/T266106 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by kormat@cumin1001 for hosts: `dborch1001.eqiad.wmnet` - dborch1001.eqiad.wmnet (**PASS**) - Downtimed host on... [08:55:02] 10DBA, 10cloud-services-team (Kanban): Move wikireplicas under the new sanitarium hosts (db1154, db1155) - https://phabricator.wikimedia.org/T272008 (10Marostegui) [09:19:45] 10DBA, 10Orchestrator, 10SRE, 10CAS-SSO, 10User-Kormat: orchestrator: Support SSO - https://phabricator.wikimedia.org/T266106 (10Kormat) [09:41:46] should I downtime lag on db1125 ? [09:42:46] it will recover in a sec [09:42:52] ok [09:43:30] 10DBA, 10cloud-services-team (Kanban): Move wikireplicas under the new sanitarium hosts (db1154, db1155) - https://phabricator.wikimedia.org/T272008 (10Marostegui) [09:46:23] 10DBA, 10cloud-services-team (Kanban): Move wikireplicas under the new sanitarium hosts (db1154, db1155) - https://phabricator.wikimedia.org/T272008 (10Marostegui) I have moved 4 out of the new 16 instances under the new hosts. Won't move more today, to make sure everything runs fine. [10:33:23] 10DBA, 10SRE: Refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [10:55:21] 10DBA, 10Orchestrator, 10SRE, 10CAS-SSO, 10User-Kormat: orchestrator: Support SSO - https://phabricator.wikimedia.org/T266106 (10Kormat) [11:42:07] I think I have you both access to the doc, shout if you still cannot access [11:42:52] jynus: 👍 [11:43:07] I have access too, thanks [11:43:26] yeah, zarcillo is something with lots of possiblities, but worried we may want to kill it instead [11:43:40] yep [11:43:43] which may be blocked on how we use orchestator [11:43:52] exactly, yeah [11:44:34] kormat, you are working a lot on it this Q, right? [11:44:43] (orchestator) [11:44:55] jynus: i'd have to consult my OKRs, which aren't in betterworks yet i believe ;) [11:44:59] lol [11:45:04] i'm definitely doing some work on it, yeah [11:45:07] in any way [11:45:21] we could reevaluate at end of Q the doc manuel wrote [11:45:32] to see how we will move forward [12:56:00] * sobanski stepping out for a while [13:09:52] 10DBA, 10mariadb-optimizer-bug: Investigate possible optimizer regression on 10.4.17 with DELETE statements - https://phabricator.wikimedia.org/T268457 (10Marostegui) I am going to install 10.4.17 on db2071 to see the effect there [14:26:33] 10DBA, 10Orchestrator, 10SRE, 10CAS-SSO, 10User-Kormat: orchestrator: Support SSO - https://phabricator.wikimedia.org/T266106 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by kormat@cumin1001 for hosts: `dborch1001.eqiad.wmnet` - dborch1001.eqiad.wmnet (**PASS**) - Downtimed host on... [15:00:52] 10DBA, 10mariadb-optimizer-bug: Investigate possible optimizer regression on 10.4.17 with DELETE statements - https://phabricator.wikimedia.org/T268457 (10Marostegui) For now I haven't seen bad regressions on the enwiki slave I put 10.4.17 on. So far it looks specifically related to that DELETE happening on pa...