[03:45:49] FIRING: PuppetFailure: Puppet has failed on cp3074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [03:46:25] FIRING: SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:45:49] FIRING: PuppetFailure: Puppet has failed on cp3074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [07:46:25] FIRING: SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:34:43] 06Traffic, 07Voice & Tone: Status code 429 response does not print contact information wrapped in angle brackets - https://phabricator.wikimedia.org/T440648#12554311 (10Aklapper) @ssingh: Would you know where that error message string is located which omits outputting the URI? [10:58:41] 10netops, 06Infrastructure-Foundations, 06SRE: Core router upgrades 2026 #2 - https://phabricator.wikimedia.org/T431748#12554584 (10ayounsi) 05Open→03Resolved a:03ayounsi [11:45:49] FIRING: PuppetFailure: Puppet has failed on cp3074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [11:46:25] FIRING: SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:49:00] 06Traffic, 06Collaboration-Services, 10MediaWiki Train Development Environment: train-dev build fails to build Gerrit image: cannot fetch Git LFS objects - https://phabricator.wikimedia.org/T443677#12555569 (10ABran-WMF) >>! In T443677#12550835, @dancy wrote: > T441387#12482937 says that the CDN changes from... [15:45:49] FIRING: PuppetFailure: Puppet has failed on cp3074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:46:25] FIRING: SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:45:49] FIRING: PuppetFailure: Puppet has failed on cp3074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [19:46:25] FIRING: SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:57:02] 06Traffic, 06Collaboration-Services, 10MediaWiki Train Development Environment, 13Patch-For-Review: train-dev build fails to build Gerrit image: cannot fetch Git LFS objects - https://phabricator.wikimedia.org/T443677#12556820 (10dancy) >>! In T443677#12555569, @ABran-WMF wrote: >>>! In T443677#12550835, @... [20:58:09] 06Traffic, 06Collaboration-Services, 10MediaWiki Train Development Environment, 13Patch-For-Review: train-dev build fails to build Gerrit image: cannot fetch Git LFS objects - https://phabricator.wikimedia.org/T443677#12556822 (10dancy) @Southparkfan I merged a workaround to the train-dev repo. Please giv... [23:01:25] FIRING: [5x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:06:25] FIRING: [15x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:30:43] FIRING: [2x] HaproxyKafkaRestarted: HaproxyKafka restarted on cp4049:9100 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaRestarted - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaRestarted [23:35:43] RESOLVED: [3x] HaproxyKafkaRestarted: HaproxyKafka restarted on cp4049:9100 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaRestarted - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaRestarted [23:36:25] FIRING: [5x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:41:25] FIRING: [9x] SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:45:49] FIRING: PuppetFailure: Puppet has failed on cp3074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [23:52:39] 06Traffic, 10HaproxyKafka: haproxykafka fails to start after a system reboot - https://phabricator.wikimedia.org/T444888 (10Vgutierrez) 03NEW [23:57:13] FIRING: [2x] HaproxyKafkaExporterDown: HaproxyKafka on cp2054 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [23:58:28] FIRING: HaproxyKafkaRestarted: HaproxyKafka restarted on cp2054:9100 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaRestarted - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2054 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaRestarted [23:59:13] FIRING: [2x] HaproxyKafkaRestarted: HaproxyKafka restarted on cp1109:9100 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaRestarted - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaRestarted