[10:33:40] FIRING: [5x] VarnishPrometheusExporterDown: Varnish Exporter on instance cp3067:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [10:33:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp3071 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3071 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [10:34:43] FIRING: [3x] HaproxyKafkaExporterDown: HaproxyKafka on cp3077 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [11:53:12] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Setup new cr1-esams - https://phabricator.wikimedia.org/T442691#12538696 (10ayounsi) [12:37:10] FIRING: VarnishPrometheusExporterDown: Varnish Exporter on instance cp3074:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [14:24:00] 14Varnish, 06Fundraising-Backlog, 10FY26-27 ST6.4 - Logged-in performance, 10MediaWiki-extensions-CentralNotice, and 5 others: Special:HideBanners is not really cacheable - https://phabricator.wikimedia.org/T256447#12539382 (10Krinkle) >>! In T285210#12539348, @Krinkle wrote: > […] I'll work on making this... [14:27:10] RESOLVED: VarnishPrometheusExporterDown: Varnish Exporter on instance cp3074:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [14:31:03] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp3074 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3074 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [14:38:47] 06Traffic, 10Citoid, 06Editing-team, 10RESTBase-Sunsetting, and 5 others: Switch from restbase to rest-gateway for Citoid - https://phabricator.wikimedia.org/T361576#12539550 (10Mvolz) >>! In T361576#12484285, @Jdforrester-WMF wrote: > For the "move docs" step, we nowadays have https://www.mediawiki.org/w/... [14:39:15] 06Traffic, 10Citoid, 06Editing-team, 10RESTBase-Sunsetting, and 5 others: Switch from restbase to rest-gateway for Citoid - https://phabricator.wikimedia.org/T361576#12539553 (10Mvolz) [15:31:24] FIRING: SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:50:48] FIRING: PuppetFailure: Puppet has failed on cp3074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:51:20] FIRING: DnsboxServiceMismatch: Service ntp-a state mismatch on dns4003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=ulsfo&var-instance=dns4003:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [15:56:20] RESOLVED: DnsboxServiceMismatch: Service ntp-a state mismatch on dns4003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=ulsfo&var-instance=dns4003:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [16:31:35] 10netops, 06Infrastructure-Foundations: Setup new cr1-esams - https://phabricator.wikimedia.org/T442691#12540474 (10ayounsi) 05Open→03Resolved a:03ayounsi All done. [19:31:25] FIRING: SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:50:48] FIRING: PuppetFailure: Puppet has failed on cp3074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [23:31:25] FIRING: SystemdUnitFailed: cpupower.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:50:48] FIRING: PuppetFailure: Puppet has failed on cp3074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure