[04:52:10] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1131 - https://phabricator.wikimedia.org/T257253 (10Marostegui) I will failover db1131 to db1093 on Tuesday 14th at 05:00 AM UTC [04:56:41] 10DBA, 10Patch-For-Review: Compress enwiki InnoDB tables - https://phabricator.wikimedia.org/T254462 (10Marostegui) [07:31:10] 10DBA, 10Patch-For-Review: Upgrade x1 databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254871 (10Kormat) Ok. [07:31:30] 10DBA, 10Patch-For-Review, 10User-Kormat: Upgrade x1 databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254871 (10Kormat) [08:26:19] 10DBA, 10Epic, 10Patch-For-Review, 10User-Kormat: Upgrade es4 to debian buster + mariadb 10.4 - https://phabricator.wikimedia.org/T257284 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts: ` ['es1021.eqiad.wmnet'] ` The log can be found in `/var/log/wm... [08:35:55] 10DBA, 10MediaWiki-General, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), 10Patch-For-Review, and 2 others: Normalise MW Core database language fields length - https://phabricator.wikimedia.org/T253276 (10Marostegui) [08:40:37] huh. es1021 failed to boot into the pxe env. it's instead back in the on-disk OS [08:41:07] 10DBA, 10Epic, 10Patch-For-Review, 10User-Kormat: Upgrade es4 to debian buster + mariadb 10.4 - https://phabricator.wikimedia.org/T257284 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts: ` ['es1021.eqiad.wmnet'] ` The log can be found in `/var/log/wm... [08:41:09] you can try setting it up manually via ipmi-tool locally, to see if it fails [08:41:45] mm, ok. i'll try that if it fails again. i'm attached to the console this time [08:42:08] ipmitool -I lanplus -H "$hostname" -U root -E chassis bootdev pxe [08:42:12] https://wikitech.wikimedia.org/wiki/Management_Interfaces#Other_useful_ipmitool_commands [08:44:48] hmm, no, that's not the issue. it looks like it does _try_ to boot off pxe [08:45:11] and if you force it on the console manually? as in pressing f12 when it asks? [08:45:32] i'll try that now [08:47:13] if it try and fails it could be some misconfiguration on network or something else [08:47:43] that happens to me frequently when the wrong ip/dns has been configured, but it is weird that it happens on a newly setup host [08:47:51] *non newly setup [08:48:11] maybe get some logs from dhcp server? [08:52:12] aaand it worked this time, for no apparent reason [08:57:05] as far as i can tell, the dhcp servers never received a request until 08:51 UTC for es1021 [08:57:37] temporary network blip? [08:57:44] could be, yeah [08:59:41] that's weird [09:10:03] 10DBA, 10Epic, 10Patch-For-Review, 10User-Kormat: Upgrade es4 to debian buster + mariadb 10.4 - https://phabricator.wikimedia.org/T257284 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['es1021.eqiad.wmnet'] ` and were **ALL** successful. [10:22:47] 10DBA, 10Epic, 10Patch-For-Review, 10User-Kormat: Upgrade es4 to debian buster + mariadb 10.4 - https://phabricator.wikimedia.org/T257284 (10Kormat) The codfw nodes are done, as is es1021. On monday i'll upgrade es1022, and then we'll need to look at scheduling a master switchover. [10:49:52] 10DBA, 10Core Platform Team, 10Schema-change, 10User-DannyS712: slot_revision_origin_role should be a UNIQUE INDEX - https://phabricator.wikimedia.org/T256841 (10Marostegui) 05Open→03Declined [13:43:39] oh. Ohh. the existing mysql_legacy spicerack module doesn't support multiinstance hosts [13:43:48] and in fact a cumin-based approach cannot [13:44:16] because all cumin can give you is a hostname [13:44:22] it can't even tell you which role that host has [13:44:46] wonderful [13:48:02] marostegui: change of plans! i think i need to focus on ensuring zarcillo is consistent with puppet, and then having the new spicerack mysql module query zarcillo [13:51:40] kormat: let's chat about it on the monday meeting I think [13:51:45] so we can get more background and all that [13:52:13] looking forward to it :) [14:05:24] kormat: you reached the same conclusion I did some time ago :-D [14:06:27] host inventory vs instance inventory [14:11:21] one disclaimer: for me zarcillo is more of an idea than what we have now [14:11:40] could be a better db model or could be something else [14:12:24] ack [14:17:53] but I am happy we reached similar conclusions: https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/570161 [14:29:14] :1 [14:29:15] +1 [14:30:45] 10DBA, 10Operations, 10User-Kormat: Port DBA dashboards to thanos - https://phabricator.wikimedia.org/T256730 (10jcrespo)