[08:16:28] <federico3>	 Amir1: can I start the schema change in s3 codfw? Also the setup of es2049?
[08:16:53] <Amir1>	 federico3: go for it
[08:17:23] <federico3>	 both things?
[08:47:35] <federico3>	 Amir1: ah the clone cookbook is for s* section, it might needs tweaking for es*. Is there a documented process to set up es* nodes?
[08:51:54] <federico3>	 cookb
[09:21:55] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: swift_dispersion_stats.service on ms-fe1009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:24:44] <Emperor>	 probably a side-effect of the envoy upgrade, but I'll have a quick look
[09:25:42] <Emperor>	 yeah, fixed now
[09:27:10] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: swift_dispersion_stats.service on ms-fe1009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:31:55] <jinxer-wm>	 RESOLVED: [2x] SystemdUnitFailed: swift_dispersion_stats.service on ms-fe1009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:16:55] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: swift_dispersion_stats.service on thanos-fe1004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:21:32] <Emperor>	 fixed (again, consequence of the envoy upgrade, apologies for the noise)
[10:22:10] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: swift_dispersion_stats.service on thanos-fe1004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:22:27] <Emperor>	 grafana is lagging, it's fine on the host.
[10:23:31] <Emperor>	 Can I get a +1 on two CRs, please? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1183627/ to remove 3 drained hosts for disk controller swap ; and then https://gerrit.wikimedia.org/r/c/operations/puppet/+/1183628/ to re-load those and drain the next batch
[10:25:42] <jynus>	 going for a coffee outside
[10:26:55] <jinxer-wm>	 RESOLVED: [2x] SystemdUnitFailed: swift_dispersion_stats.service on thanos-fe1004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:26:58] <federico3>	 Emperor: d'you mind putting the hostnames in the description just for extra safety?
[10:28:50] <Emperor>	 federico3: like that?
[10:29:10] <federico3>	 yes thx
[10:29:49] <federico3>	 ah I wanted to +1 but Jaime did it already
[10:36:10] <Emperor>	 well, thanks to both of you :)
[12:03:31] <Amir1>	 https://www.irccloud.com/pastebin/j1AObd21/
[12:03:37] <Amir1>	 how
[12:07:39] <Amir1>	 this is the same range of commons extlinks and commons has +100M pages
[12:22:48] <Amir1>	 aaaaaa
[12:22:51] <Amir1>	 https://www.irccloud.com/pastebin/grQ4PBMR/
[12:23:07] <Amir1>	 27M links to wikimedia.org
[12:23:23] <federico3>	 Amir1: I think the copy of the es host needs a bit of a bespoke process for cloning so I have a copy of the clone cookbook with the replication checks disabled https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1183646 - can I give it a shot?
[12:26:16] <Amir1>	 federico3: 1- it still tries to set replication. 2- I suggest to add a flag to the existing coobook
[12:26:32] <Amir1>	 for 1, line 639
[12:29:22] <federico3>	 i could integrate it in the existing cookbook but i would do it only after getting the second one tested few times 
[12:31:10] <federico3>	 also the CHANGE MASTER TO few lines above
[12:44:15] <Emperor>	 Are we having a team meeting today? AFAICT Kwaku, Kavitha, Giuseppe, and Eric are all OoO
[12:47:13] <federico3>	 good point, we can skip it and sync up here
[12:47:50] <Amir1>	 I always appreciate less meetings
[13:01:24] <jynus>	 oh
[13:03:02] <sobanski>	 Good to know, but I'm by no means against ;)
[13:03:43] * Emperor content either way, it just seemed worth asking with so many people away
[15:06:36] <federico3>	 Amir1: we spoke about automating some parts of the master flips - but there's also T196366  ( See https://phabricator.wikimedia.org/T196366#10479471 ) and I'm taking a look at https://gitlab.wikimedia.org/repos/sre/wmfmariadbpy/-/merge_requests/3/diffs - maybe I can help with the new script without touching the existing one?
[15:06:37] <stashbot>	 T196366: Implement (or refactor) a script to move slaves when the master is not available - https://phabricator.wikimedia.org/T196366
[15:38:48] <Amir1>	 I'd pause more long term improvements until Manuel is back
[15:44:01] <federico3>	 ok
[17:04:45] <federico3>	 Amir1: the schema change on s3 is done, can I start s7?
[21:56:55] <jinxer-wm>	 FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on es2026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[22:14:04] <Amir1>	 That was supposed to be the source for cloning the new host, downtime expired?