[00:10:25] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:11:55] FIRING: [7x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [01:15:25] FIRING: [3x] SystemdUnitFailed: haproxy_stek_job.service on cp2059:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:11:55] FIRING: [7x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [05:15:25] FIRING: [3x] SystemdUnitFailed: haproxy_stek_job.service on cp2059:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:25:41] 10netops, 06Infrastructure-Foundations: Junos: ae speed not reported corectly through gNMI - https://phabricator.wikimedia.org/T439202 (10ayounsi) 03NEW [06:22:48] FIRING: PuppetFailure: Puppet has failed on durum5003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [06:27:48] RESOLVED: PuppetFailure: Puppet has failed on durum5003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [07:53:07] 06Traffic, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Provide a scheduled data download service from Google Cloud Storage - https://phabricator.wikimedia.org/T427457#12362749 (10Gehel) [09:11:55] FIRING: [7x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [09:15:25] FIRING: [3x] SystemdUnitFailed: haproxy_stek_job.service on cp2059:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:20:24] 06Traffic, 10observability, 06SRE: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12363311 (10dkertesz) [10:52:05] 06Traffic: Refactoring and some other work on purged - https://phabricator.wikimedia.org/T350396#12363458 (10Vgutierrez) [10:59:32] 06Traffic: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828#12363466 (10Vgutierrez) [11:06:38] 06Traffic: Avoid heap allocations related to prometheus metrics on purged hot path - https://phabricator.wikimedia.org/T439234 (10Vgutierrez) 03NEW [13:11:55] FIRING: [7x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [13:15:25] FIRING: [3x] SystemdUnitFailed: haproxy_stek_job.service on cp2059:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:28:21] inflatador: welcome back, there is an outstanding issue with cirrussearch1120 ^ [13:28:47] XioNoX ACK, that sounds like a hangover from the IP change. Will take a look now [13:28:49] 06Traffic: Refactoring and some other work on purged - https://phabricator.wikimedia.org/T350396#12364115 (10Vgutierrez) [13:28:52] yep [13:29:29] FIRING: HAProxyRestarted: HAProxy server restarted on cp2059:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=codfw%20prometheus/ops&var-instance=cp2059&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [13:29:55] FIRING: HaproxyKafkaRestarted: HaproxyKafka restarted on cp2059:9100 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaRestarted - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2059 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaRestarted [13:30:25] FIRING: [3x] SystemdUnitFailed: haproxy_stek_job.service on cp2059:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:31:00] 06Traffic: Reduce heap allocations on purged TCPPurger.Send() - https://phabricator.wikimedia.org/T439254 (10Vgutierrez) 03NEW [13:31:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2059 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2059 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [13:32:42] cp2059 is running puppter after being rebooted.. it should come back soon(TM) [13:33:06] thanksvg [13:33:40] *puppet :) one week and my fingers haven't get used to the keyboard again /o\ [13:34:29] RESOLVED: HAProxyRestarted: HAProxy server restarted on cp2059:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=codfw%20prometheus/ops&var-instance=cp2059&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [13:34:39] OK, I pooled `cirrussearch1120`, but if memory serves, those PyBal alerts won't go away until PyBal is restarted. LMK if I need to do that [13:34:43] RESOLVED: HaproxyKafkaRestarted: HaproxyKafka restarted on cp2059:9100 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaRestarted - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2059 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaRestarted [13:35:24] Sep 25 13:35:14 lvs1020 pybal[756662]: [search-https_9243 ProxyFetch] WARN: cirrussearch1120.eqiad.wmnet (enabled/down/not pooled): Fetch failed (https://localhost/_cluster/health?timeout=5s), 3.056 s [13:35:25] FIRING: [5x] SystemdUnitFailed: haproxy.service on cp2059:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:35:50] inflatador: services need to be healthy for healthchecks to be happy :) [13:36:31] vgutierrez the service is healthy, but the host had a recent IP change [13:36:43] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2059 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2059 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [13:36:55] FIRING: [7x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [13:37:22] I don't think PyBal knows about the new IP [13:37:41] should I take care of the restarts? [13:38:59] vgutierrez sure, whenever you have the cycles [13:40:25] FIRING: [5x] SystemdUnitFailed: haproxy.service on cp2059:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:40:40] FIRING: [7x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [13:41:28] hmm that haproxy.service@cp2059 alert is stale, the host is fine [13:47:41] inflatador: done.. cirrussearch1120 should be back in business [13:49:27] vgutierrez ACK, thanks for the assist! [13:50:40] RESOLVED: [7x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [14:14:30] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12364294 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir2001.codfw.wmnet with OS trixie [14:57:47] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12364495 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir2001.codfw.wmnet with OS trixie completed: - ncredir2001 (**WARN**) - Downtimed on Icin... [15:05:09] 06Traffic, 10observability, 06SRE, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12364518 (10dkertesz) Something weird happened: we decided to move the stats file to `/run` so that it //wouldn't// survive reboots, so I prepared a [[ https://gerrit.... [15:08:08] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12364550 (10CDobbins) [15:33:11] 06Traffic, 10observability, 06SRE, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12364687 (10Vgutierrez) it definitely looks like you're hitting the same issue mentioned on https://github.com/haproxy/haproxy/issues/2483#issuecomment-2579845321, def... [15:45:59] 06Traffic, 06DC-Ops: Figuring out the equivalent of Dell's `racadm getsel' in Supermicro - https://phabricator.wikimedia.org/T438989#12364737 (10ssingh) Thanks both. @Dzahn: I tried to parse through the output and I am having a hard time doing that as compared to the racadm one. So I wonder if there is a bet... [16:11:26] 06Traffic, 06DC-Ops: Figuring out the equivalent of Dell's `racadm getsel' in Supermicro - https://phabricator.wikimedia.org/T438989#12364884 (10Dzahn) To be honest I think parsing that output should be for some combination of dcops and the vendor itself. Like we typically only look at this log when a server... [17:40:25] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:46:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on durum5003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [17:47:12] looking [17:48:28] hm failing to connect to puppetserver weird [17:49:04] sukhe: I think there might be some flakiness with teh codfw puppetservers [17:49:28] slow runs were being discussed yesterday I think [17:49:48] cdanis: interstingly enough, this is connecting the one in eqiad [17:49:54] hm! [17:50:03] > Request to https://puppetserver1001.eqiad.wmnet:8140/puppet/v3/catalog/durum5003.eqsin.wmnet?environment=production failed after 0.2 second [17:50:04] maybe we didn't repool those? [17:50:29] please tell me `0.2 second` isn't the deadline [17:51:11] er [17:51:19] codfw.wmnet. [17:51:19] ; puppet SRV records [17:51:19] _x-puppet._tcp 5M IN SRV 0 5 8140 puppetserver1001.eqiad.wmnet. [17:51:22] _x-puppet._tcp 5M IN SRV 0 5 8140 puppetserver1002.eqiad.wmnet. [17:51:25] _x-puppet._tcp 5M IN SRV 0 5 8140 puppetserver1003.eqiad.wmnet. [17:51:33] yeah Iea236f5de0ecb1b0ce41ced135338f8db89f5215 yesterday [17:51:50] reverting [17:52:48] 13:51:48 <+wikibugs> (PS1) Ssingh: Revert "Failover Puppet to eqiad" [dns] - https://gerrit.wikimedia.org/r/1345186 [17:52:51] 13:52:20 <+wikibugs> (PS1) CDanis: Revert "Failover Puppet to eqiad" [dns] - https://gerrit.wikimedia.org/r/1345188 [17:52:53] ahaha [17:52:54] abandoning mine [17:53:07] +1ed yours [17:53:24] so yeah, I guess we missed this revert when we focussed on the etcd one [17:56:07] yeah, it wasn't critical [17:56:24] I think we did check that the puppetserver2* gits got caught back up, but then didn't do the next step [18:03:57] cool, reran puppet on cp5017, after confirming the new record was visible, and successful and no diffs [18:04:33] thanks! [18:04:49] interestingly enough this alert fired on perhaps the least critical host, durum :P [18:04:55] so there is value [18:05:31] wow I had never logged into a durum before apparently [18:05:33] cool motd [18:06:22] thanks! I like wheat, durum is named after that, so had to do a wheat sheaf :) [18:06:39] doh* have a cool one as well, which jbond colourized to perfection [18:07:24] (that's because we are running auditd on them. so bblack and I decided we had to say something about that :) [18:12:31] hahahaha [18:12:37] cooooool [18:13:06] :D [18:30:47] 06Traffic, 10Observability-Alerting, 13Patch-For-Review: urldownloader service should be a paging alert - https://phabricator.wikimedia.org/T435648#12365226 (10ssingh) Thanks for the feedback folks. @MatthewVernon: This has now been fixed as the text has been updated, since the idea is that this should be a... [18:31:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on durum5003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:40:25] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:38:14] 06Traffic, 06Commons, 10MediaWiki-File-management, 06SRE, and 3 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12365931 (10BBlack) >>! In T425216#12360080, @Krinkle wrote: > [,,, re: canonical ...] > Is this an inten...