[08:21:49] Hi folks - could I get a +1 on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1295804 please? The integration tests of our rewrite middleware were failing and using a non-standard thumb size. [08:53:33] looking [08:54:39] Emperor: I have no context on the change, what do you want me to review? [09:00:09] this is an integration test for our swift rewrite middleware, which I've fixed (because it was using a non-standard thumb size); you could check that 960px is a standard thumb size, and that the change otherwise makes sense https://www.mediawiki.org/wiki/Common_thumbnail_sizes [09:01:45] You _could_ download the thumbnail and check it's the size I've said it is, but if it weren't the tests would fail (and I've put test output in the CR text), so I'd think that was being a bit keen :) [09:34:29] thanks :) [10:48:06] I am swithcing es1 eqiad and codfw "masters" (RO standalone section) [15:41:33] that's odd, I ran sudo cookbook sre.hosts.remove-downtime db1224.eqiad.wmnet and puppet-merged the change to enable notifications a while ago but it's still showing as disabled [15:41:47] do i need to trigger a run of puppet manually? [15:46:55] Puppet says: 2026-06-01T15:40:06.788315+00:00 db1224 puppet-agent[78323]: Applied catalog in 19.70 seconds [15:48:39] federico3: so, maybe you were just a little too early with the check? [15:54:39] yes, the pool script could be a bit more clear about it [15:54:44] btw is https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-job=%24__all&var-server=db1224&var-port=9104&from=now-5m&to=now&timezone=utc&refresh=1m loading for you? [15:55:17] Yes, I see populated charts for the last 5m [15:55:18] ok it took 2 mins now it loads [17:02:21] we have a bug between rolling restarts and auto schema that makes it crash out [19:17:42] federico3: puppet needs to run on icinga host for notifications to get enabled [20:28:56] FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed