[01:35:40] FIRING: VarnishChildRestarted: varnish-text restarted on cp3071 - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000330/varnish-machine-stats?orgId=1&viewPanel=66&var-server=cp3071&datasource=esams%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DVarnishChildRestarted [05:35:40] FIRING: VarnishChildRestarted: varnish-text restarted on cp3071 - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000330/varnish-machine-stats?orgId=1&viewPanel=66&var-server=cp3071&datasource=esams%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DVarnishChildRestarted [06:57:43] FIRING: [4x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [07:02:43] FIRING: [17x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [07:12:43] RESOLVED: [17x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [08:25:40] RESOLVED: VarnishChildRestarted: varnish-text restarted on cp3071 - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000330/varnish-machine-stats?orgId=1&viewPanel=66&var-server=cp3071&datasource=esams%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DVarnishChildRestarted [10:03:18] 10netops, 06Infrastructure-Foundations, 06SRE: Network: gnmic monitoring issues and improvements - https://phabricator.wikimedia.org/T441780 (10cmooney) 03NEW p:05Triage→03Medium [10:04:28] 10netops, 06Infrastructure-Foundations: Junos 25 and gNMI metrics - https://phabricator.wikimedia.org/T439166#12453414 (10cmooney) [10:04:30] 10netops, 06Infrastructure-Foundations, 06SRE: Network: gnmic monitoring issues and improvements - https://phabricator.wikimedia.org/T441780#12453413 (10cmooney) [10:04:31] 10netops, 06Infrastructure-Foundations, 06SRE: Missing series for BGP session_state from eqiad CRs since upgrade to 23.4R2-S8.7 - https://phabricator.wikimedia.org/T435909#12453415 (10cmooney) [10:04:37] 10netops, 06Infrastructure-Foundations, 06SRE: Export network fixed-limit usage stats in gnmi - https://phabricator.wikimedia.org/T395998#12453416 (10cmooney) [10:04:43] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Enable gNMI on SRX devices and fasw - https://phabricator.wikimedia.org/T390052#12453417 (10cmooney) [10:04:49] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 13Patch-For-Review: Migrate network icinga alerts to gNMI/prometheus - https://phabricator.wikimedia.org/T388641#12453418 (10cmooney) [10:22:25] FIRING: SystemdUnitFailed: prometheus_liberica_cp_checks.service on lvs7001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:27:10] ^^ lvs7001 is being rebooted [10:27:28] Aaah [10:30:55] RESOLVED: SystemdUnitFailed: prometheus_liberica_cp_checks.service on lvs7001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:25:28] FIRING: KeyholderUnarmed: 1 unarmed Keyholder key(s) on acmechief1002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [11:30:28] FIRING: [2x] KeyholderUnarmed: 1 unarmed Keyholder key(s) on acmechief1002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [11:35:24] FIRING: SystemdUnitFailed: acme-chief-certs-sync.service on acmechief1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:40:28] FIRING: [2x] KeyholderUnarmed: 1 unarmed Keyholder key(s) on acmechief1002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [11:45:28] RESOLVED: [2x] KeyholderUnarmed: 1 unarmed Keyholder key(s) on acmechief1002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [11:50:24] FIRING: [3x] SystemdUnitFailed: acme-chief-certs-sync.service on acmechief1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:51:32] hmm looking [11:53:08] ok.. that got solved by s.lyngs rearming the keyholder [11:53:12] it should be fine [11:55:24] RESOLVED: [3x] SystemdUnitFailed: acme-chief-certs-sync.service on acmechief1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:26:35] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12454528 (10ssingh) [13:28:21] vgutierrez: o/ I ran puppet on cp7001 and checked the webrequest_frontend_text topic on kafka jumbo (with latest events) and I see hostname: cp7001.magru.wmnet [13:28:58] and haproxykafka's unit started 8 mins ago [13:29:10] yep... it restarted as expected [13:29:17] so I'd be inclined to go ahead with all magru, ok? [13:29:22] ack [13:36:48] FIRING: PuppetFailure: Puppet has failed on cp5031:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [13:37:53] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12454587 (10Vgutierrez) [13:39:55] 06Traffic, 10Liberica: Get ready to upgrade liberica LBs to trixie - https://phabricator.wikimedia.org/T401824#12454609 (10Vgutierrez) 05Open→03Resolved [13:40:40] so kcat -t webrequest_frontend_text -o latest -b localhost:9092 | egrep 'cp7[0-9]*.magru.wmnet' | jq '.' on kafka-jumbo1010 looks good [13:40:47] and all magru has been moved to the new ca [13:40:53] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12454614 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1004 for host lvs7003.magru.wmnet with OS trixie [13:43:44] and I also verified the certs in /etc/haproxykafka/ssl/kafka__haproxykafka_kafka_11.chained.pem, it looks good [13:44:34] vgutierrez, slyngs - ok to re-enable puppet on all cp hosts? Anything that you want to double check etc..? [13:45:36] Let me just check something really quickly [13:46:13] We're good [13:46:39] super, re-enabling puppet on cp* [13:46:50] Thanks [13:59:26] 06Traffic, 06Collaboration-Services, 10Infrastructure Security, 06Infrastructure-Foundations, and 7 others: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838 (10bking) 03NEW [14:00:10] 06Traffic, 06Collaboration-Services, 10Infrastructure Security, 06Infrastructure-Foundations, and 7 others: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838#12454745 (10bking) 05Open→03In progress p:05Triage→03Medium a:03bking [14:16:48] RESOLVED: PuppetFailure: Puppet has failed on cp5031:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:40:09] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12455019 (10Vgutierrez) [14:40:52] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12455027 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1004 for host lvs7003.magru.wmnet with OS trixie completed: - lvs7003 (**WARN**) - Downtimed on Icinga/Ale... [14:45:10] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12455058 (10Vgutierrez) [14:53:03] 06Traffic: Upgrade to HAProxy 3.2.26 - https://phabricator.wikimedia.org/T441435#12455103 (10dkertesz) [14:54:39] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12455107 (10Vgutierrez) [15:03:25] FIRING: SystemdUnitFailed: prometheus_liberica_cp_checks.service on lvs7002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:07:01] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12455193 (10Vgutierrez) [15:18:25] RESOLVED: SystemdUnitFailed: prometheus_liberica_cp_checks.service on lvs7002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:57:13] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Network: gnmic monitoring issues and improvements - https://phabricator.wikimedia.org/T441780#12455759 (10cmooney) @ayounsi it strikes me we might need better profiling of devices than simply basing things on the manufacturer. For instance... [16:58:07] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: WMCS switch refresh 2026 (C8 & D5) - https://phabricator.wikimedia.org/T440305#12455778 (10cmooney) [20:09:25] FIRING: SystemdUnitFailed: haproxy.service on cp7016:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:14:25] RESOLVED: SystemdUnitFailed: haproxy.service on cp7016:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed