[04:16:44] 10DBA, 10Data-Services, 10Operations, 10Patch-For-Review: Prepare and check storage layer for nqowiki - https://phabricator.wikimedia.org/T230543 (10Marostegui) a:03Marostegui [05:10:53] 10DBA, 10Operations: Switchover s4 (commonswiki) primary database master db1081 -> db1138 - 26th Sept @05:00 UTC - https://phabricator.wikimedia.org/T230784 (10Marostegui) The switchover was done successfully Read-only start: 05:00:51 Read-only stop: 05:01:42 Total read-only time: 51 seconds [05:30:56] 10DBA, 10Operations: Switchover s4 (commonswiki) primary database master db1081 -> db1138 - 26th Sept @05:00 UTC - https://phabricator.wikimedia.org/T230784 (10Marostegui) 05Open→03Resolved [06:16:01] 10Blocked-on-schema-change, 10GlobalBlocking: Alter gbw_reason/gb_reason/gbw_by_text on WMF production - https://phabricator.wikimedia.org/T231172 (10Marostegui) [06:16:09] 10Blocked-on-schema-change, 10GlobalBlocking: Alter gbw_reason/gb_reason/gbw_by_text on WMF production - https://phabricator.wikimedia.org/T231172 (10Marostegui) 05Open→03Resolved s3 finished, so all done [06:31:14] 10DBA: Change PK and remove partitions from the logging table - https://phabricator.wikimedia.org/T233625 (10Marostegui) [06:32:19] 10DBA: Change PK and remove partitions from the logging table - https://phabricator.wikimedia.org/T233625 (10Marostegui) [06:33:29] 10DBA: Change PK and remove partitions from the logging table - https://phabricator.wikimedia.org/T233625 (10Marostegui) [06:43:52] 10DBA, 10Data-Services, 10Operations, 10Patch-For-Review: Prepare and check storage layer for nqowiki - https://phabricator.wikimedia.org/T230543 (10Marostegui) I have sanitized both db1124:3313 and db2094:3313 (sanitariums). All the users have been sanitized correctly and my username was created correctl... [06:53:31] ok to merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/537132/ now? (given that Manuel renamed the old DB tables they can't be in use any further) [06:54:04] moritzm: yep [06:54:18] ack [07:04:56] jynus: ok to power off db1114? [07:05:02] I think you were doing some work with it? [07:06:00] I am now done [07:06:13] but I put mysql back up already [07:07:56] jynus: ok, thanks! I will power it down for the mainboard replacement [07:08:05] thanks to you [07:11:19] 10DBA, 10Operations, 10ops-eqiad: db1114 crashed due to memory issues (server under warranty) - https://phabricator.wikimedia.org/T229452 (10Marostegui) @Cmjohnson db1114 is now off, so the mainboard can be replaced anytime. [08:19:29] 10DBA, 10conftool: specify group (api/vslow/etc) weights in terms of 0..100 instead of 0..1 - https://phabricator.wikimedia.org/T231018 (10Marostegui) [08:19:44] 10DBA, 10conftool: specify group (api/vslow/etc) weights in terms of 0..100 instead of 0..1 - https://phabricator.wikimedia.org/T231018 (10Marostegui) [08:22:59] 10DBA, 10conftool: specify group (api/vslow/etc) weights in terms of 0..100 instead of 0..1 - https://phabricator.wikimedia.org/T231018 (10Marostegui) [12:18:29] 10DBA, 10Patch-For-Review: Change PK and remove partitions from the logging table - https://phabricator.wikimedia.org/T233625 (10Marostegui) [13:08:54] marostegui: no issues with dbctl this morning? [13:09:30] cdanis: meeting - but all good [14:42:27] marostegui: jynus: I'm doing a silly schema migration procedure now, don't use dbctl for a few minutes [14:43:03] ack [14:43:07] cdanis: sure, no problem, we are in a meeting [14:43:08] thanks [15:00:13] ok all is back to normal [15:08:48] and one last note: dbctl will run a lot more quickly if you run it on cumin2001 vs cumin1001 [15:59:11] cdanis: oh, that is good to know, in the morning the commits normally takes like 10 seconds or so [15:59:20] so we can reduce the switchover even more then! :) [15:59:22] yeah should be much faster on cumin2001 [15:59:29] as the etcd cluster is in codfw [15:59:33] just realized this today [15:59:51] no worries, we have switchovers to do! :) [15:59:54] anyways, I am off o/ [17:16:45] 10DBA, 10conftool: #dbctl: add 'comment'/'description' metadata to instances - https://phabricator.wikimedia.org/T229677 (10CDanis) 05Open→03Resolved a:03CDanis released [17:16:58] 10DBA, 10Operations, 10ops-eqiad: es1019 IPMI and its management interface are unresponsive (again2) - https://phabricator.wikimedia.org/T233698 (10Cmjohnson) @Marostegui Can you depool it leave it for us to do when we get a free moment. It's an easy thing to do but may not happen until later in the day aft... [18:20:07] 10DBA: Drop 'designate_pool_manager' database from m5 and remove associated grants - https://phabricator.wikimedia.org/T233978 (10Andrew) [20:14:25] Well, at this point I just saw labsdb1011 and then labsdb1010 crash one after another. It has to be some kind of user interaction? https://phabricator.wikimedia.org/T233986