[07:06:56] <marostegui>	 I've refreshed ms3 codfw master with a new host, if you see something weird let me know, as the process is very delicate
[07:12:53] <jynus>	 hi, marostegui, may you double check my rebase is correct to current status?
[07:12:56] <jynus>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1276382
[07:13:04] <marostegui>	 checking
[07:13:34] <marostegui>	 +1ed again
[07:13:38] <marostegui>	 looks good
[07:14:00] <jynus>	 just making sure you were not in the mean time working with db2253
[07:14:14] <jynus>	 will merge as is now
[07:14:18] <marostegui>	 nop, not today probably
[07:14:20] <marostegui>	 thanks though
[07:14:39] <jynus>	 no problem
[07:24:01] <jinxer-wm>	 FIRING: SystemdUnitFailed: prometheus-mysqld-exporter.service on db2250:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:18:16] <federico3>	 Amir1, marostegui: can I start the rolling reboot on es6 and then es7 using rolling_restart.py ? Any blocker or glitch to be aware of?
[08:18:57] <marostegui>	 federico3: double check with jaime to make sure no backups are running
[08:19:46] <federico3>	 the rolling_restart.py scripts checks for running backups and skips hosts as needed
[08:20:04] <marostegui>	 nice then
[08:20:25] <federico3>	 (you mean jynus?)
[08:20:54] <marostegui>	 yes
[08:21:17] <jynus>	 no ongoing backups now, how long does it take? backups run the monday to tuesday night
[08:25:19] <federico3>	 should be done well before night
[08:36:59] <jynus>	 👍
[09:10:20] <jynus>	 FYI: Apr 23 09:09:02 prometheus2006 mysqld_exporter_config.py[734533]: Writing mysql-dbstore_codfw.yaml
[09:10:48] <jynus>	 if there has been any schema change in the last 24 hours for s1, it may need reappying for db2250:s1
[09:44:01] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: prometheus-mysqld-exporter.service on db2250:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:20:10] <marostegui>	 I've refreshed pc2 codfw master with a new host, if you see something weird let me know, as the process is very delicate
[15:15:57] <jynus>	 so you know if there is more backup sources or backup1 dbs programmed to be installed this Q?
[15:16:52] <marostegui>	 jynus: https://phabricator.wikimedia.org/T412415 those have arrived, not sure if in the refreshes (listed there too) are backup sources
[15:17:02] <jynus>	 I will check myself
[15:17:38] <jynus>	 is there a list of decoms, or just first + number of hosts?
[15:18:53] <marostegui>	 This is covered by "Refresh of db11[50-82,84]"
[15:19:02] <marostegui>	 Those are the hosts that are being refreshed
[15:19:02] <jynus>	 thanks, that I couldn't find
[15:19:06] <jynus>	 I will check those
[15:19:17] <marostegui>	 jynus: 34 will be decommissioned, but we bought only 26
[15:19:25] <jynus>	 oh
[15:19:31] <marostegui>	 if there are backup sources involved, those will get a host, so no worries
[15:19:47] <jynus>	 a worry for other time, anyway
[15:20:13] <marostegui>	 yep :)
[15:20:32] <marostegui>	 I will create the "productionize task" soon, so I will ping you there too with the host that will be "yours"
[15:23:11] <jynus>	 https://phabricator.wikimedia.org/P91376
[15:24:13] <jynus>	 db1150 & db1171 for backup sources
[15:24:35] <marostegui>	 I will assign some of the new hosts to those :)
[15:30:53] <jynus>	 I will be leaving a backup running to test the new host- it will have a 50% chance of failing, because I always forget some minor thing on setup (which gets thoroughly checked on backup), so you can ignore if you see some job error tommorrow morning