[04:23:05] <wikibugs>	 10DBA: db2097 (backup source) restarted itself - https://phabricator.wikimedia.org/T252492 (10Marostegui)
[04:41:16] <wikibugs>	 10DBA: db2097 (backup source) restarted itself - https://phabricator.wikimedia.org/T252492 (10Marostegui) HW logs: ` /system1/log1/record38   Targets   Properties     number=38     severity=Critical     date=05/12/2020     time=01:50:28     description=Uncorrectable Machine Check Exception (Processor 2, APIC ID...
[04:45:50] <wikibugs>	 10DBA, 10cloud-services-team (Kanban): Reimage labsdb1011 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T249188 (10Marostegui) The import has finished - I am going to stop MySQL and copy the content to backup1001 and then I'll attempt to configure replication.
[04:51:36] <wikibugs>	 10DBA, 10Data-Services: database query timeouts since April 26th 2020 - https://phabricator.wikimedia.org/T252209 (10Marostegui) 05Open→03Declined Declining as nothing can be done at the moment to mitigate the current timeouts. We have finished reimporting all the data into labsdb1011, if all goes well, we...
[05:07:01] <wikibugs>	 10DBA, 10Operations, 10User-notice: Upgrade and restart s4 (commonswiki) primary database master: Tue 12th May - https://phabricator.wikimedia.org/T251502 (10Marostegui) 05Open→03Resolved This has been done. RO starts: 05:00:54 RO stops: 05:03:40
[05:07:04] <wikibugs>	 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb daemons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui)
[05:07:38] <wikibugs>	 10DBA, 10Operations, 10User-notice: Upgrade and restart s4 (commonswiki) primary database master: Tue 12th May - https://phabricator.wikimedia.org/T251502 (10Marostegui)
[05:07:56] <wikibugs>	 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb daemons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui)
[05:36:41] <wikibugs>	 10DBA: db2097 (backup source) restarted itself - https://phabricator.wikimedia.org/T252492 (10jcrespo) The same thing happened at T225378#5303653   ` root@db2097:~$ date Tue May 12 05:36:21 UTC 2020 root@db2097:~$ free -m               total        used        free      shared  buff/cache   available Mem:...
[05:41:43] <wikibugs>	 10DBA: db2097 (backup source) restarted itself - https://phabricator.wikimedia.org/T252492 (10jcrespo) ` 462 - Uncorrectable Memory Error Threshold Exceeded (Processor 2, DIMM 4).  The DIMM is mapped out and is currently not available. Action: Take corrective action for the failing DIMM. Re-map all DIMMs back in...
[07:12:39] <jynus>	 there is some ongoing lagging on some multiinstance s6 hosts since yesterday
[08:11:22] <jynus>	 db1113 lagging for 5 minutes already
[08:12:31] <marostegui>	 ok I will take a look, I am busy with clinic duty stuff
[08:13:03] <jynus>	 could be HTMLCacheUpdateJob::invalidateTitles ?
[08:15:41] <marostegui>	 big spike on writes on all the sections 
[08:15:45] <marostegui>	 https://grafana.wikimedia.org/d/000000278/mysql-aggregated?panelId=7&fullscreen&orgId=1
[08:16:05] <jynus>	 I think it is mostly s6
[08:16:15] <jynus>	 see https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1113&var-port=13316&from=1589270050436&to=1589271318426&panelId=6&fullscreen
[08:16:38] <jynus>	 lots of inserts in a single transaction
[08:16:55] <marostegui>	 https://grafana.wikimedia.org/d/kUxjfbeWk/mysql-kormat?panelId=2&fullscreen&orgId=1&from=now-24h&to=now&var-dc=eqiad%20prometheus%2Fops&var-server=db1131&var-port=9104
[08:17:08] <marostegui>	 right after a deploy
[08:17:33] <jynus>	 not hw, the other instance on the same host did not lag
[08:18:06] <jynus>	 although it is true there is weird traffic patterns
[08:18:19] <marostegui>	 17:04 brennen@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.31
[08:18:25] <marostegui>	 right after that the master spiked on writes
[08:18:44] <jynus>	 but that is yesterday, right?
[08:18:54] <marostegui>	 yes, but the master is still having lots more INSERTs
[08:18:58] <jynus>	 I see
[08:19:06] <marostegui>	 https://grafana.wikimedia.org/d/kUxjfbeWk/mysql-kormat?panelId=2&fullscreen&orgId=1&from=now-2d&to=now&var-dc=eqiad%20prometheus%2Fops&var-server=db1131&var-port=9104
[08:19:16] <marostegui>	 I am going to create a task for that version
[08:19:22] <marostegui>	 Will capture a bit of the traffic on the master
[08:19:34] <marostegui>	 but it is very suspiciuous it started right after the deployment
[08:19:35] <jynus>	 I think it is the invalidate stuff
[08:19:43] <jynus>	 I mentioned before
[08:19:45] <marostegui>	 have you checked binlogs?
[08:19:48] <jynus>	 no
[08:19:53] <marostegui>	 let me take a look
[08:19:56] <jynus>	 it is a guess
[08:20:03] <jynus>	 although it could be a synthom, not a cause
[08:20:14] <jynus>	 marostegui: https://grafana.wikimedia.org/d/000000303/mysql-replication-lag?panelId=6&fullscreen&orgId=1&from=1589269504265&to=1589271510297&var-dc=eqiad%20prometheus%2Fops
[08:20:19] <jynus>	 if you filter out the labs hosts
[08:20:39] <jynus>	 it points to an app issue (all hosts affected at the same time)
[08:20:59] <jynus>	 from 7:57
[08:22:07] <marostegui>	 On the master i see most of the writes like INSERT /* Wikimedia\DependencyStore\SqlModuleDependencyStore::storeMulti  */ INTO  `module_deps` (md_module,md_skin,md_deps)
[08:22:12] <jynus>	 oh
[08:22:25] <jynus>	 that was a previous issue
[08:22:30] <jynus>	 let me find it
[08:22:46] <marostegui>	 I am not seenig that huge amount on previous binlogs (from before the release)
[08:22:59] <jynus>	 was that happening now or yesterday?
[08:23:07] <jynus>	 https://phabricator.wikimedia.org/T247028
[08:23:10] <marostegui>	 This is affecting everything https://grafana.wikimedia.org/d/000000273/mysql?panelId=2&fullscreen&orgId=1&from=now-24h&to=now&var-dc=eqiad%20prometheus%2Fops&var-server=db1083&var-port=9104
[08:23:12] <marostegui>	 That's enwiki master
[08:23:15] <marostegui>	 same huge increase
[08:23:28] <jynus>	 see the task
[08:23:31] <marostegui>	 Let me comment on that task yeah
[08:23:42] <jynus>	 we ended up lowering importance
[08:23:48] <jynus>	 because back then it didn't cause lag
[08:23:52] <marostegui>	 I will raise it again
[08:23:55] <jynus>	 so it was not ideal but manageable
[08:24:07] <jynus>	 now it is causing lag, so it should be high
[08:24:32] <jynus>	 ping krinke, as he was around last time, he may want to know
[08:24:48] <marostegui>	 yeah, he's subscribed
[08:25:56] <jynus>	 "The table stores absolute paths which means when a wmf-branch roll over, it loses track of some files, thus causing a needless cache invalidation"
[08:26:08] <jynus>	 that seems to be fixed
[08:26:19] <jynus>	 but confirms there is a relation between the invalidation and that table
[08:41:18] <wikibugs>	 10DBA: Productionize db114[1-9] - https://phabricator.wikimedia.org/T252512 (10Marostegui)
[08:41:45] <wikibugs>	 10DBA: Productionize db114[1-9] - https://phabricator.wikimedia.org/T252512 (10Marostegui) p:05Triage→03Medium
[09:18:11] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: db2097 (backup source) restarted itself - https://phabricator.wikimedia.org/T252492 (10Marostegui) This host is under warranty from what I can see, so maybe we should get a new memory DIMM from HP? That is what we did when it happened at T225378
[09:21:29] <jynus>	 ^man that is the plan :-D but I want to solve the sw issue first
[09:21:38] <jynus>	 in particular, the failed dumps
[09:22:01] <jynus>	 I am worried about the 15% less data on dumps, that smells corruption on backup taking
[09:47:28] <marostegui>	 eh, ok, sorry for posting
[09:48:09] <jynus>	 oh, don't be sorry
[09:48:14] <jynus>	 it was just a FYI
[09:48:23] <jynus>	 that I am on top of it :-D
[09:49:48] <marostegui>	 ok thanks
[09:50:44] <jynus>	 man == manuel, not Man!!! :-D
[10:25:35] <kormat>	 marostegui: tendril shows that pc1 currently has 4 hosts, but both pc2 and pc3 only have 2 each
[10:26:03] <kormat>	 that seems odd
[10:27:05] <kormat>	 ohh. i'm misreading my own notes
[10:27:34] <kormat>	 we have 3 sections, 1 machine per DC per section
[10:27:37] <kormat>	 plus some spares
[10:30:34] <kormat>	 T252182 says pc2010 has been upgraded, but it has not
[10:30:35] <stashbot>	 T252182: Upgrade parsercache to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T252182
[10:37:02] <marostegui>	 kormat: yeah, looks like I didn't upgrade pc2010 even though I thought I did
[10:37:56] <kormat>	 should i do that before the production machines?
[10:38:23] <marostegui>	 yeah please
[10:40:10] <kormat>	 grand, on it.
[10:40:46] <wikibugs>	 10DBA: Upgrade parsercache to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T252182 (10Kormat) Correction: pc2010 was not upgraded.
[10:40:50] <marostegui>	 thank you
[10:40:56] <kormat>	 np :)
[10:42:19] <kormat>	 marostegui: as i'm not (yet) touching the master, this is just the normal reimage procedure, right?
[10:42:33] <marostegui>	 for pc2010 yep
[10:42:34] <kormat>	 (i know that touching the master requires changing the mw config, and reordering replication)
[10:42:36] <marostegui>	 and for pc2007 also
[10:42:47] <marostegui>	 no need to re-order replication for this "quick" task
[10:42:58] <marostegui>	 but we will get there once you touch pc1007 or pc1009
[10:43:06] <marostegui>	 if pc2XXX that means it is not active master
[10:43:06] <kormat>	 ack :)
[10:43:14] <marostegui>	 the ones that indeed require MW depooling are pc1XXX
[10:43:22] * kormat nods
[10:46:33] <kormat>	 marostegui: for you: https://gerrit.wikimedia.org/r/c/operations/puppet/+/595901
[10:59:32] <wikibugs>	 10DBA, 10Patch-For-Review: Upgrade parsercache to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T252182 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts: ` ['pc2010.codfw.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202005121...
[12:12:49] <kormat>	 hurm. puppet agent is erroring out after reimage, because it can't connect to mariadb ☹️
[12:13:06] <marostegui>	 ah yes, because of pt-hearbeat
[12:13:07] <marostegui>	 right?
[12:13:11] <kormat>	 yep
[12:13:29] <marostegui>	 so, mount /srv and all that, start mysql, run puppet, stop mysql and let the installation finish :)
[12:13:51] <kormat>	 i.. wow.
[12:13:51] <marostegui>	 if you don't do that, it will finally fail and say that the installation never finished correctly, but it is a lie :)
[12:13:54] <marostegui>	 it actually did
[12:13:59] <marostegui>	 it is just a race condition
[12:14:10] <marostegui>	 you can leave it fail too, it doesn't matter at that point
[12:14:18] <kormat>	 it's a shit condition. :)
[12:14:45] <marostegui>	 yeah, for those that run pt-heartbeat yes
[12:15:18] <wikibugs>	 10DBA: Upgrade parsercache to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T252182 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['pc2010.codfw.wmnet'] `  and were **ALL** successful.
[12:17:54] <kormat>	 do we have a task for this?
[12:18:34] <marostegui>	 about that race condition?
[12:18:37] <marostegui>	 no that I know of
[12:18:59] <kormat>	 ok, i'll file one for tracking purposes
[12:19:08] <marostegui>	 but yeah, when mysql is stopped, pt-kill and pt-heartbeat will fail
[12:19:18] <marostegui>	 and as they are suppsoed to be started by puppet..
[12:19:44] <kormat>	 why does puppet manage services? isn't that systemd's job?
[12:20:14] <marostegui>	 yeah, sorry, for pt-kill it is not 
[12:20:26] <marostegui>	 I don't remember what we did actually
[12:20:28] <marostegui>	 I can check later
[12:20:42] <kormat>	 ok :) it's not important
[12:20:51] <marostegui>	 you can check on labsdb1009
[12:20:57] <marostegui>	 those are the only hosts that run pt kill
[12:23:51] <wikibugs>	 10DBA, 10Operations: wmf-auto-reinstall fails on hosts that run pt-heartbeat - https://phabricator.wikimedia.org/T252528 (10Kormat)
[12:23:54] <kormat>	 ^ created
[12:24:07] <marostegui>	 thanks - can you move it into the backlog column too?
[12:26:27] <wikibugs>	 10DBA, 10Operations: wmf-auto-reinstall fails on hosts that run pt-heartbeat - https://phabricator.wikimedia.org/T252528 (10Marostegui) p:05Triage→03Medium This is "expected" as the host doesn't have MySQL up and running. This is pretty much the last step of the script, so even if the installation reports...
[12:30:32] <kormat>	 done
[12:31:52] <marostegui>	 thanks :*
[12:41:39] <kormat>	 `REVOKE DELETE HISTORY ON *.* from 'dump'@'X.X.X.X';` - is that a literal command to run? or does the ip need to be specified?
[12:41:59] <kormat>	 (mm. we probably don't bother backing up pc, do we?)
[12:42:09] <marostegui>	 First check if the user exists, which I doubt on the parsercache hosts
[12:42:43] <marostegui>	 you can do a quick check with select user,host from mysql.user where user like 'dump';
[12:43:00] <kormat>	 thanks <3
[12:43:03] <kormat>	 nope, doesn't exist
[12:43:09] <kormat>	 (on pc2010)
[12:43:25] <marostegui>	 cool, we don't backup pc of course :)
[12:53:12] <kormat>	 marostegui: it looks like it's going to take a Long time for pc2010 to catch back up with replication - is that expected?
[12:53:16] <kormat>	 https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-dc=codfw%20prometheus%2Fops&var-server=pc2010&var-port=9104&from=now-1h&to=now
[12:53:56] <kormat>	 hmm. i'd estimate that it's i/o bound writing to disk
[12:54:41] <marostegui>	 yes, it is expected
[12:54:47] <marostegui>	 it usually takes a bit
[12:54:59] <kormat>	 alright. i'll continue with a production host in the meantime then?
[12:56:30] <marostegui>	 which one?
[12:56:48] <kormat>	 i was going to close my eyes and just stab my finger at tendril
[12:56:54] <kormat>	 but, i guess, pc2007?
[12:56:54] <marostegui>	 :)
[12:57:02] <marostegui>	 cool, sounds good
[12:57:13] <marostegui>	 keep in mind that as it is codfw, it doesn't receive any traffic 
[12:57:59] <kormat>	 we should make a 'lonely' puppet role for these poor hosts
[13:00:59] <kormat>	 marostegui: this (how long it will take for pc2010 to resync) is a very clear reason for why we disable notifications for reimaging db hosts. i'll bear that in mind for future discussions with volans ;)
[13:01:33] <marostegui>	 yeah, in this case pc2010 wouldn't page (because it is codfw), but with any other eqiad host, it would, that's why :)
[13:01:45] <kormat>	 right, good point :)
[13:04:06] <volans>	 kormat: although that's debatable
[13:04:15] * kormat dons his debating gloves
[13:04:35] <volans>	 a host that is up and running and just re-syncing, should be healthy apart for 1~2 checks
[13:04:54] <marostegui>	 volans: yeah, but replication would still page
[13:04:55] <volans>	 right now pc2010 has only a Lag red alert in icinga
[13:05:28] <volans>	 so would be better to have all checks running as normal, not downtimed, not silenced and just the expected ones (lag) that clearly are not ready downtimed/disabled
[13:05:30] <kormat>	 marostegui: https://gerrit.wikimedia.org/r/c/operations/puppet/+/595922 for you (i used a separate line for pc2007 in the hopes that i'll still be able to do an auto git revert for the pc2010 change)
[13:05:31] <marostegui>	 volans: I think the point is that the host might get out of downtime if for any reason the catching up takes long
[13:05:55] <volans>	 marostegui: sure, I'm saying that we should have something dedicated in addition to the simple short downtime for the reimage
[13:06:00] <volans>	 that covers only the bits we're interested
[13:06:34] <volans>	 for example, wouldn't you want to know if RAID fails on that host during the resync before putting it back in production? :D
[13:07:46] <volans>	 what I'm trying to say, is that after a reimage a host should be healthy, with maybe some service and related check not yet happy (and that's ok and we should aim at those instead of the whole host)
[13:10:59] <kormat>	 volans: are you saying that wmf-auto-reimage could have per-service knowledge about what to downtime after a host gets reimaged?
[13:11:38] <kormat>	 host comes up -> downtime everything until after initial puppet runs -> downtime only services that might need some time (maybe a few hours) to recover?
[13:12:21] <volans>	 kormat: depends, wmf-auto-reimage already downtimes the whole host for 2h (IIRC), given it's not the one taking care of any post-install action
[13:12:41] <volans>	 I'd say that is more the post-install action that should be able to downtime what it needs
[13:12:51] <volans>	 like the resync downtiming the lag for examaple
[13:12:54] <volans>	 *example
[13:13:21] <volans>	 the only case not covered in this case is a reimage where the post-actions are not started within 2h from the reimage
[13:13:38] <volans>	 that seems unlikely, but YMMV
[13:14:17] <kormat>	 volans: so wmf-auto-reimage doesn't undowntime when it finishes currently?
[13:14:23] <volans>	 nope
[13:14:30] <kormat>	 huh
[13:14:45] <kormat>	 that sounds like something you're arguing against above, too
[13:15:04] <volans>	 it might in the future once converted to cookbook (the helper functions for that are in spicerack)
[13:15:12] <volans>	 but most likely will do that only if it's all green
[13:15:39] <volans>	 (we have something to check that, not yet in prod)
[13:17:02] <volans>	 the reason why it's not doing that was lack of support for checking state and downtime removal. don't forget that wmf-auto-reimage is "legacy" as in pre-spicerack
[13:17:09] <kormat>	 gotcha
[13:17:27] <kormat>	 well, that sounds like something our Future And Amazing Automation could work with
[13:18:07] <volans>	 ™
[13:18:08] <volans>	 :D
[13:29:19] <kormat>	 marostegui: i have some heresy for you: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/595928
[13:29:35] <kormat>	 (rather than making a CR per host that gets switched to buster, i've made one just switching all of the pc* hosts to buster)
[13:30:11] <volans>	 +1 fwiw :
[13:30:13] <volans>	 :)
[13:31:39] <kormat>	 merging quickly before marostegui changes his mind :)
[13:32:28] <marostegui>	 xddddd
[13:32:36] <marostegui>	 I did +1
[13:39:21] <wikibugs>	 10DBA, 10Patch-For-Review: Upgrade parsercache to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T252182 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts: ` ['pc2007.codfw.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202005121...
[13:52:23] <kormat>	 marostegui: er. pc2007 has no link
[13:53:17] <marostegui>	 woot?
[13:53:20] <marostegui>	 for pxe?
[13:53:23] <kormat>	 yep
[13:53:57] <marostegui>	 maybe reboot back with disk and check if the MAC that shows on the dhcp file is the one that asks for link
[13:54:38] <kormat>	 i can confirm that eno1 has the same mac address as is in the dhcpd config
[13:55:22] <marostegui>	 and the same that asks for PXE?
[13:55:37] <kormat>	 that i'm trying to figure out how to check
[13:56:38] <marostegui>	 you can check the dhcp logs
[13:56:56] <kormat>	 i'm on install2003 - is that the right server?
[13:57:05] <marostegui>	 yeah, or install1003
[13:57:13] <marostegui>	 they are the dhcp/tftp 
[13:57:21] <kormat>	 a codfw host could end up talking to install1003?
[13:57:29] <marostegui>	 yes, whichever responds faster
[13:57:33] <kormat>	 ohh. i see.
[13:59:05] <kormat>	 `May 12 13:43:45 install2003 dhcpd[2808]: DHCPACK on 10.192.0.104 to d0:94:66:86:2e:4d via 10.192.0.3`
[13:59:14] <kormat>	 that says the right mac address got the right ip..
[14:00:17] <marostegui>	 yeah, that looks like the one on the dhcpd file yeah
[14:00:34] <marostegui>	 so maybe you can open a ticket to DCOPs and CC netops to see if they can give you a hint
[14:00:44] <marostegui>	 that would be: netops tag and ops-codfw tag
[14:02:33] <kormat>	 there's something weird going on here. i can bring the interface up manually, and it does have an ip configured on it
[14:03:56] <marostegui>	 maybe the switch is doing something funky?
[14:03:57] <kormat>	 aaand now it's all mysteriously working
[14:04:07] <marostegui>	 loose cable?
[14:05:19] <kormat>	 i dunno, it feels like an issue with d-i
[14:05:30] <kormat>	 (but maybe that's because the more i learn about d-i the less i trust the fucker :)
[14:05:53] <marostegui>	 but this isn't the first pc2XXX we reimage
[14:08:30] <kormat>	 the install is running fine. i'll see if i can have a look through the logs when it comes back up
[14:24:22] <kormat>	 hmm. well, it _was_. then it hung. it turns out it's trying to install stretch. but also it looks like there's a bug in the preseed config we use:
[14:24:37] <kormat>	 `/var/lib/dpkg/info/network-preseed.postinst: eval: line 1: syntax error: unexpected ";;"`
[14:25:42] <kormat>	 agh. that's my fault. i have a typo in my netboot.cfg change.
[14:28:03] <marostegui>	 and mine for not catching it
[14:29:47] <volans>	 kormat: wanna know a fun fact about that typo?
[14:30:36] <kormat>	 marostegui: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/595945
[14:30:40] <kormat>	 volans: go on :)
[14:31:10] <volans>	 that breaks the reimage from your host and all the following in the case/esac list, but works fine for all the ones defined before the typo
[14:31:37] <kormat>	 volans: oh ouch
[14:31:48] <volans>	 joys of bash :D
[14:32:10] <kormat>	 the sooner we remove humans from editing that the better :)
[14:32:35] <volans>	 yeah
[14:47:22] <jynus>	 lag on pc2010, I am guessing expected
[14:47:31] <marostegui>	 yes
[14:47:43] <wikibugs>	 10DBA, 10Patch-For-Review: Upgrade parsercache to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T252182 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts: ` ['pc2007.codfw.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202005121...
[14:47:54] <wikibugs>	 10DBA, 10Patch-For-Review: Upgrade parsercache to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T252182 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['pc2007.codfw.wmnet'] `  Of which those **FAILED**: ` ['pc2007.codfw.wmnet'] `
[14:48:33] <kormat>	 `14:47:39 | pc2007.codfw.wmnet | Unable to run wmf-auto-reimage-host: Failed to icinga_downtime`
[14:48:42] <marostegui>	 kormat: use --no-downtime
[14:48:55] <kormat>	 cheers
[14:49:16] <wikibugs>	 10DBA, 10Patch-For-Review: Upgrade parsercache to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T252182 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts: ` ['pc2007.codfw.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202005121...
[14:51:18] <volans>	 kormat: pro-tip: for codfw hosts is quicker to launch it from cumin2001 ;)
[14:52:18] <kormat>	 how much quicker are we talking?
[14:53:23] <volans>	 my guess is not that much, just few round trips, but the whole first puppet run has to go cross-dc
[14:53:29] <volans>	 (the log of it)
[14:53:39] <jynus>	 I would do it more for reliability too
[14:54:27] <kormat>	 ok, added to notes. *sobs*
[14:55:14] <jynus>	 it is not the first time that we run a 4-hour alter table and in the middle of it a host it loses connection for a second
[14:55:33] <jynus>	 since then I tend to run them -if long- from localhost
[14:56:08] <jynus>	 don't think just network- e.g. "we need to reboot cumin" or whatever
[14:58:41] <marostegui>	 I am going offline, I started at 6.30am
[14:58:44] <marostegui>	 More than enough
[14:59:10] <jynus>	 have a nice day
[14:59:13] <marostegui>	 labsdb1011 -> backup1001 transfer still ongoing
[14:59:15] <marostegui>	 2TB or so pending
[14:59:17] <jynus>	 ok
[14:59:24] <jynus>	 will not touch it
[14:59:30] <jynus>	 have my own list of things to fix :-D
[14:59:46] <marostegui>	 tomorrow I will do the start+mysql_upgrade+replication
[15:00:05] * jynus crosses fingers
[15:00:20] * marostegui too and looks at kormat's
[15:00:43] * kormat is worried marostegui might be hungry, and hides his fingers
[15:00:57] <marostegui>	 XDD
[15:01:03] <marostegui>	 I am actually quite hungry
[15:01:14] <kormat>	 well, in that case,
[15:01:19] * kormat gives marostegui the finger
[15:01:26] <kormat>	 (not sure if that needs translation :)
[15:02:09] <marostegui>	 XDDDD
[15:02:12] <marostegui>	 no need to
[15:04:31] <volans>	 lol
[15:25:02] <wikibugs>	 10DBA: Upgrade parsercache to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T252182 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['pc2007.codfw.wmnet'] `  and were **ALL** successful.
[17:24:42] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: db2097 (backup source) restarted itself - https://phabricator.wikimedia.org/T252492 (10jcrespo) I resetup the host from backups. I am going to generate a logical backup (and a snapshot will be also generated later this day) and then send this to dc ops.
[19:35:57] <wikibugs>	 10DBA, 10cloud-services-team (Kanban): Reimage labsdb1011 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T249188 (10Marostegui) The transfer has finished.  I have started mysql on labsdb1011 and ran a mysql_upgrade --force as it wasn't run before. No errors were shown during startup or during t...
[20:21:24] <wikibugs>	 10DBA, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata-Campsite, and 4 others: [Story] Monitor size of some Wikidata database tables - https://phabricator.wikimedia.org/T68025 (10Addshore)
[20:22:42] <wikibugs>	 10DBA, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata-Campsite, and 4 others: [Story] Monitor size of some Wikidata database tables - https://phabricator.wikimedia.org/T68025 (10Addshore)