[08:17:37] FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s3.service on db2239:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:20:42] I'll downtime this host for a couple of weeks as I'm not sure Jaime will have time to handle it soon [09:19:49] Can I get a +1 on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1085617 please? We're going to moving our first production user to the apus cluster soon (probably this week), so it needs to be p.aging now [09:20:17] {done} [09:21:00] Thanks. I think I can mark T279621 done ... [09:21:00] T279621: Set up new S3-level replicated storage cluster "apus" - https://phabricator.wikimedia.org/T279621 [09:21:21] 🎉 [09:21:54] nice nice... [11:37:03] It is great to see that we have a cookbook to sanitize wikis! Can someone update https://wikitech.wikimedia.org/wiki/MariaDB/PII with the new workflow/commands? [12:08:22] btullis: https://phabricator.wikimedia.org/T367856#10288226 [12:09:06] marostegui: Thanks. I'll check now and update the ticket. [12:12:49] thanks [14:22:37] FIRING: PuppetFailure: Puppet has failed on ms-be1058:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:23:47] marostegui: the wiki pii cleaner has not yet been merged, I'll update the docs right after, good catch [14:23:55] excellent thanks [14:28:28] ^-- that's the silence expiring again, maybe sometime soon I'll be able to actually use the replacement hardware and get rid of this :-/