[00:01:25] FIRING: [3x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:01:58] RESOLVED: [3x] HaproxyKafkaExporterDown: HaproxyKafka on cp2054 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [00:03:28] RESOLVED: HaproxyKafkaRestarted: HaproxyKafka restarted on cp2054:9100 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaRestarted - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2054 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaRestarted [00:06:25] FIRING: [7x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:10:20] FIRING: DnsboxServiceMismatch: Service ntp-b state mismatch on dns2005:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=codfw&var-instance=dns2005:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [00:11:25] FIRING: [15x] SystemdUnitFailed: haproxy.service on cp3069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:15:20] RESOLVED: DnsboxServiceMismatch: Service ntp-b state mismatch on dns2005:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=codfw&var-instance=dns2005:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [00:16:25] FIRING: [15x] SystemdUnitFailed: haproxy.service on cp3069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:17:00] 06Traffic, 10HaproxyKafka, 13Patch-For-Review: haproxykafka fails to start after a system reboot - https://phabricator.wikimedia.org/T444888#12557950 (10Vgutierrez) [00:28:59] 06Traffic, 10HaproxyKafka, 13Patch-For-Review: haproxykafka fails to start after a system reboot - https://phabricator.wikimedia.org/T444888#12557967 (10Vgutierrez) p:05Triage→03Medium [00:36:25] FIRING: [3x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:41:25] FIRING: [11x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:46:25] FIRING: [13x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:59:13] FIRING: [3x] HaproxyKafkaExporterDown: HaproxyKafka on cp2058 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [01:01:58] RESOLVED: [2x] HaproxyKafkaExporterDown: HaproxyKafka on cp2058 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [01:04:20] FIRING: DnsboxServiceMismatch: Service ntp-a state mismatch on dns3003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=esams&var-instance=dns3003:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [01:06:25] FIRING: [4x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:09:20] RESOLVED: DnsboxServiceMismatch: Service ntp-a state mismatch on dns3003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=esams&var-instance=dns3003:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [01:11:25] FIRING: [7x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:14:13] FIRING: [7x] HaproxyKafkaExporterDown: HaproxyKafka on cp1115 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [01:16:25] FIRING: [13x] SystemdUnitFailed: haproxy.service on cp3071:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:16:58] RESOLVED: [6x] HaproxyKafkaExporterDown: HaproxyKafka on cp1115 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [01:21:25] FIRING: [15x] SystemdUnitFailed: haproxy.service on cp3071:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:37:13] FIRING: [2x] HaproxyKafkaExporterDown: HaproxyKafka on cp2051 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [01:41:20] FIRING: DnsboxServiceMismatch: Service ntp-b state mismatch on dns3004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=esams&var-instance=dns3004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [01:41:25] FIRING: [3x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:41:58] RESOLVED: [5x] HaproxyKafkaExporterDown: HaproxyKafka on cp1108 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [01:46:20] RESOLVED: DnsboxServiceMismatch: Service ntp-b state mismatch on dns3004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=esams&var-instance=dns3004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [01:51:25] FIRING: [5x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:11:25] FIRING: [5x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:16:25] FIRING: [7x] SystemdUnitFailed: haproxy.service on cp3073:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:26:25] FIRING: [9x] SystemdUnitFailed: haproxy.service on cp3073:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:41:25] RESOLVED: [8x] SystemdUnitFailed: haproxy.service on cp4045:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:52:55] FIRING: [7x] SystemdUnitFailed: haproxy.service on cp4045:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:54:20] FIRING: DnsboxServiceMismatch: Service ntp-a state mismatch on dns5003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqsin&var-instance=dns5003:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [02:57:55] FIRING: [12x] SystemdUnitFailed: haproxy.service on cp2057:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:59:20] RESOLVED: DnsboxServiceMismatch: Service ntp-a state mismatch on dns5003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqsin&var-instance=dns5003:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [03:12:55] FIRING: [5x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:27:55] FIRING: [5x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:54:20] FIRING: DnsboxServiceMismatch: Service ntp-a state mismatch on dns6001:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=drmrs&var-instance=dns6001:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [03:59:20] RESOLVED: DnsboxServiceMismatch: Service ntp-a state mismatch on dns6001:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=drmrs&var-instance=dns6001:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [04:03:48] FIRING: PuppetFailure: Puppet has failed on cp3074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [04:27:55] FIRING: [2x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:22:55] FIRING: [2x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:03:48] FIRING: PuppetFailure: Puppet has failed on cp3074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:22:55] FIRING: SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:03:48] FIRING: PuppetFailure: Puppet has failed on cp3074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:28:33] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12559448 (10ssingh) FWIW, we saw this again today withe the reboots: 09:25:40 <+jinxer-wm> FIRING: [18x] BFDdown: BFD session down between cr1-codfw... [17:21:18] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error on Wikidata - https://phabricator.wikimedia.org/T421642#12559732 (10connorshea) see also https://phabricator.wikimedia.org/T359602 which is about Wikidata's REST API not having maxlag support, which is part of the... [20:13:51] 06Traffic, 07Voice & Tone: Status code 429 response does not print contact information wrapped in angle brackets - https://phabricator.wikimedia.org/T440648#12560139 (10ssingh) >>! In T440648#12554310, @Aklapper wrote: > @ssingh: Would you know where that error message string is located which omits outputting...