[00:05:24] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:56:55] FIRING: [3x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [04:05:39] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:00:15] 06Traffic, 10requestctl: Re-think how external_cloud_vendors works - https://phabricator.wikimedia.org/T438658#12353355 (10Joe) 05Open→03Resolved a:03Fabfur [06:40:25] RESOLVED: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:57:10] FIRING: [3x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [08:58:01] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw during switchover resulting from increased thumb.wm.o cache pressure - https://phabricator.wikimedia.org/T438896#12353833 (10MLechvien-WMF) #traffic could you help us looking into this ? [09:46:45] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover, 13Patch-For-Review: Investigate high saturation in codfw during switchover resulting from increased thumb.wm.o cache pressure - https://phabricator.wikimedia.org/T438896#12353980 (10Joe) As an action to at least partially... [10:14:59] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover, 13Patch-For-Review: Investigate high saturation in codfw during switchover resulting from increased thumb.wm.o cache pressure - https://phabricator.wikimedia.org/T438896#12354054 (10JMeybohm) >>! In T438896#12353980, @Joe... [10:57:10] FIRING: [3x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [10:57:40] 06Traffic, 06DBA, 06Infrastructure-Foundations: Move orchestrator (dborch) to private ipaddrs + CDN - https://phabricator.wikimedia.org/T317179#12354219 (10Marostegui) >>! In T317179#12352214, @ssingh wrote: >>>! In T317179#12349393, @Marostegui wrote: >> If I recall correctly this needs #traffic and as far... [11:00:26] XioNoX: it looks like you moved cirrussearch1120 to another vlan yesterday but the host is still pooled, can we depool it? [11:00:47] https://www.irccloud.com/pastebin/mJVGT0Gh/ [11:01:01] confctl reports the host as pooled and it's unreachable by pybal [11:39:12] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw during switchover resulting from increased thumb.wm.o cache pressure - https://phabricator.wikimedia.org/T438896#12354518 (10JMeybohm) eqiad has double the memcached capacity codf... [12:37:18] vgutierrez: 302 inflatador [12:45:08] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12354779 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=7687fe60-8be3-4891-bb62-41d39b74f427) set by cmooney@cumin1004 for 2:00:00 on 3... [12:52:58] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw during switchover resulting from increased thumb.wm.o cache pressure - https://phabricator.wikimedia.org/T438896#12354818 (10CDanis) Discussing it on IRC some this morning, we decided tha... [12:55:07] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw during switchover resulting from increased thumb.wm.o cache pressure - https://phabricator.wikimedia.org/T438896#12354836 (10CDanis) eqiad has been pooled for a little over half an hour,... [13:01:41] 06Traffic, 10observability, 06SRE, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12354871 (10Vgutierrez) 05Stalled→03In progress [13:31:49] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12354947 (10CDobbins) [13:36:04] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12354978 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir1002.eqiad.wmnet with OS trixie [14:17:49] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12355201 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir1002.eqiad.wmnet with OS trixie completed: - ncredir1002 (**WARN**) - Downtimed on Icin... [14:41:07] 06Traffic: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828#12355309 (10Vgutierrez) [14:42:27] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12355317 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir7004.magru.wmnet with OS trixie [14:43:22] inflatador: any chance that you could depool cirrussearch1120 till it's ready to serve traffic? [14:52:44] 06Traffic, 06DBA, 06Infrastructure-Foundations: Move orchestrator (dborch) to private ipaddrs + CDN - https://phabricator.wikimedia.org/T317179#12355387 (10ssingh) >>! In T317179#12354219, @Marostegui wrote: >>>! In T317179#12352214, @ssingh wrote: >>>>! In T317179#12349393, @Marostegui wrote: >>> If I recal... [14:57:10] FIRING: [3x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [15:23:44] 06Traffic: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828#12355601 (10Vgutierrez) [15:33:57] 06Traffic, 06Commons, 10MediaWiki-File-management, 06SRE, and 2 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12355691 (10AFBorchert) I run into this problem when trying to reupload [[ https://commons.wikimedia.org/... [15:42:31] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12355758 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir7004.magru.wmnet with OS trixie completed: - ncredir7004 (**WARN**) - Downtimed on Icin... [15:51:44] 06Traffic, 06DC-Ops: Figuring out the equiavlent of Dell's `racadm getsel' in Supermicro - https://phabricator.wikimedia.org/T438989 (10ssingh) 03NEW [15:56:33] 06Traffic: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828#12355900 (10Vgutierrez) [16:01:13] 06Traffic, 06DC-Ops: Figuring out the equivalent of Dell's `racadm getsel' in Supermicro - https://phabricator.wikimedia.org/T438989#12355954 (10ssingh) [16:02:32] 06Traffic, 06DC-Ops: Figuring out the equivalent of Dell's `racadm getsel' in Supermicro - https://phabricator.wikimedia.org/T438989#12355958 (10Dzahn) ` -> cd /system1/logs1/log1 /system1/logs1/log1 -> show /system1/logs1/log1 Targets : record001 record002 record003 record004 ... -> show... [16:03:43] 06Traffic, 06DC-Ops: Figuring out the equivalent of Dell's `racadm getsel' in Supermicro - https://phabricator.wikimedia.org/T438989#12355968 (10Dzahn) It seems very tedious though because there are over 30 records and seems like you can't show them all at once. [16:07:41] 06Traffic, 06DC-Ops: Figuring out the equivalent of Dell's `racadm getsel' in Supermicro - https://phabricator.wikimedia.org/T438989#12355986 (10BLiviero-WMF) Apparently Supermicro has their own "utilities" that run ipmi commands under the hood, similar to dell - at least, that's what Claude says. are these p... [16:29:38] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12356080 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir6001.drmrs.wmnet with OS trixie [16:57:48] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw during switchover resulting from increased thumb.wm.o cache pressure - https://phabricator.wikimedia.org/T438896#12356408 (10JMeybohm) >>! In T438896#12353980, @Joe wrote: > it is a very... [17:05:39] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw during switchover resulting from increased thumb.wm.o cache pressure - https://phabricator.wikimedia.org/T438896#12356442 (10JMeybohm) Things that where found in mw traces: - Long POSTs t... [17:12:40] FIRING: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2049:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [17:12:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2049 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2049 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [17:12:46] ^ depooled [17:26:42] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12356573 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir6001.drmrs.wmnet with OS trixie completed: - ncredir6001 (**WARN**) - Downtimed on Icin... [17:27:25] FIRING: NELNotReported: NEL metrics not reported - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELNotReported [17:27:29] FIRING: NELByCountryNotReported: NEL metrics by country not reported - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryNotReported [17:28:04] FIRING: [2x] LibericaEtcdErrors: Liberica instance lvs4008:3003 is experiencing etcd issues - https://wikitech.wikimedia.org/wiki/Liberica#LibericaEtcdErrors - https://grafana.wikimedia.org/d/fa4de97a-7114-48c7-a91a-f56089ef554f/liberica?orgId=1&var-site=ulsfo&var-instance=lvs4008&viewPanel=11 - https://alerts.wikimedia.org/?q=alertname%3DLibericaEtcdErrors [17:28:36] FIRING: [13x] PurgedHighEventLag: High event process lag with purged on cp1106:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [17:28:54] FIRING: [3x] LibericaDiffFPCheck: Liberica instance lvs4010:9100 control plane status doesn't match with forwarding plane status - https://wikitech.wikimedia.org/wiki/Liberica#LibericaDiffFPCheck - https://alerts.wikimedia.org/?q=alertname%3DLibericaDiffFPCheck [17:29:50] RESOLVED: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2049:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [17:29:55] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2049 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2049 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [17:32:08] FIRING: [12x] LibericaEtcdErrors: Liberica instance lvs4008:3003 is experiencing etcd issues - https://wikitech.wikimedia.org/wiki/Liberica#LibericaEtcdErrors - https://alerts.wikimedia.org/?q=alertname%3DLibericaEtcdErrors [17:32:19] FIRING: [95x] PurgedHighEventLag: High event process lag with purged on cp1100:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [17:32:55] FIRING: [5x] LibericaDiffFPCheck: Liberica instance lvs4008:9100 control plane status doesn't match with forwarding plane status - https://wikitech.wikimedia.org/wiki/Liberica#LibericaDiffFPCheck - https://alerts.wikimedia.org/?q=alertname%3DLibericaDiffFPCheck [17:35:04] RESOLVED: NELNotReported: NEL metrics not reported - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELNotReported [17:35:08] RESOLVED: NELByCountryNotReported: NEL metrics by country not reported - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryNotReported [17:36:00] FIRING: [158x] PurgedHighEventLag: High event process lag with purged on cp1100:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [18:06:30] RESOLVED: [5x] LibericaDiffFPCheck: Liberica instance lvs4008:9100 control plane status doesn't match with forwarding plane status - https://wikitech.wikimedia.org/wiki/Liberica#LibericaDiffFPCheck - https://alerts.wikimedia.org/?q=alertname%3DLibericaDiffFPCheck [18:10:30] RESOLVED: [12x] LibericaEtcdErrors: Liberica instance lvs4008:3003 is experiencing etcd issues - https://wikitech.wikimedia.org/wiki/Liberica#LibericaEtcdErrors - https://alerts.wikimedia.org/?q=alertname%3DLibericaEtcdErrors [18:18:08] FIRING: [54x] PurgedHighEventLag: High event process lag with purged on cp4037:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [18:19:08] FIRING: [6x] SystemdUnitCrashLoop: purged.service crashloop on cp4047:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [18:19:12] FIRING: [14x] SystemdUnitCrashLoop: purged.service crashloop on cp4037:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [18:21:08] FIRING: [16x] LibericaEtcdErrors: Liberica instance lvs4008:3003 is experiencing etcd issues - https://wikitech.wikimedia.org/wiki/Liberica#LibericaEtcdErrors - https://alerts.wikimedia.org/?q=alertname%3DLibericaEtcdErrors [18:22:29] FIRING: [8x] SystemdUnitCrashLoop: purged.service crashloop on cp4047:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [18:26:08] FIRING: [14x] LibericaEtcdErrors: Liberica instance lvs4008:3003 is experiencing etcd issues - https://wikitech.wikimedia.org/wiki/Liberica#LibericaEtcdErrors - https://alerts.wikimedia.org/?q=alertname%3DLibericaEtcdErrors [18:31:10] FIRING: NELByCountryNotReported: NEL metrics by country not reported - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryNotReported [18:31:19] FIRING: [11x] ProbeDown: Service ncredir2001:443 has failed probes (tcp_ncredir_non_canonical_redirect_20_ip4) - https://wikitech.wikimedia.org/wiki/Ncredir - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:31:37] FIRING: NELNotReported: NEL metrics not reported - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELNotReported [18:32:19] FIRING: [4x] PurgedHighEventLag: High event process lag with purged on cp4039:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [18:32:36] FIRING: [54x] ProbeDown: Service ncredir2001:443 has failed probes (tcp_ncredir_non_canonical_redirect_13_ip4) - https://wikitech.wikimedia.org/wiki/Ncredir - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:34:10] FIRING: [54x] PurgedHighEventLag: High event process lag with purged on cp4037:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [19:02:11] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2048 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2048 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [19:02:16] FIRING: [2x] HaproxyKafkaExporterDown: HaproxyKafka on cp2043 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [19:03:34] FIRING: [3x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns2004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [19:06:47] FIRING: KeyholderUnarmed: 1 unarmed Keyholder key(s) on acmechief2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [19:07:04] brett: which is the primary acme-chief? [19:07:13] we should move that over to eqiad [19:07:25] ah, yeah [19:07:53] FIRING: [14x] SystemdUnitFailed: haproxy.service on cp2043:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:08:33] RESOLVED: NELByCountryNotReported: NEL metrics by country not reported - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryNotReported [19:08:47] RESOLVED: [156x] ProbeDown: Service ncredir2001:443 has failed probes (tcp_ncredir_non_canonical_redirect_10_ip4) - https://wikitech.wikimedia.org/wiki/Ncredir - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:09:01] RESOLVED: NELNotReported: NEL metrics not reported - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELNotReported [19:09:05] brett: please patch this profile::acme_chief::active: acmechief2002.codfw.wmnet [19:09:14] ack, on it [19:10:06] FIRING: [11x] PurgedHighEventLag: High event process lag with purged on cp2056:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [19:10:32] FIRING: [156x] ProbeDown: Service ncredir2001:443 has failed probes (tcp_ncredir_non_canonical_redirect_10_ip4) - https://wikitech.wikimedia.org/wiki/Ncredir - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:32:05] RESOLVED: [10x] LibericaEtcdErrors: Liberica instance lvs4008:3003 is experiencing etcd issues - https://wikitech.wikimedia.org/wiki/Liberica#LibericaEtcdErrors - https://alerts.wikimedia.org/?q=alertname%3DLibericaEtcdErrors [19:32:34] FIRING: [10x] LibericaEtcdErrors: Liberica instance lvs4008:3003 is experiencing etcd issues - https://wikitech.wikimedia.org/wiki/Liberica#LibericaEtcdErrors - https://alerts.wikimedia.org/?q=alertname%3DLibericaEtcdErrors [19:36:09] RESOLVED: [2x] LibericaEtcdErrors: Liberica instance lvs4008:3003 is experiencing etcd issues - https://wikitech.wikimedia.org/wiki/Liberica#LibericaEtcdErrors - https://grafana.wikimedia.org/d/fa4de97a-7114-48c7-a91a-f56089ef554f/liberica?orgId=1&var-site=ulsfo&var-instance=lvs4008&viewPanel=11 - https://alerts.wikimedia.org/?q=alertname%3DLibericaEtcdErrors [19:40:09] FIRING: [5x] HaproxyKafkaExporterDown: HaproxyKafka on cp2043 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [19:40:59] FIRING: [3x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns2004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [19:44:15] FIRING: [14x] SystemdUnitFailed: haproxy.service on cp2043:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:44:42] RESOLVED: [156x] ProbeDown: Service ncredir2001:443 has failed probes (tcp_ncredir_non_canonical_redirect_10_ip4) - https://wikitech.wikimedia.org/wiki/Ncredir - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:45:08] FIRING: [14x] SystemdUnitFailed: haproxy.service on cp2043:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:45:37] FIRING: [21x] PurgedHighEventLag: High event process lag with purged on cp2050:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [19:46:34] FIRING: [24x] PurgedHighEventLag: High event process lag with purged on cp2050:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [19:49:00] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12357046 (10CDobbins) [19:52:36] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12357049 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir5004.eqsin.wmnet with OS trixie [19:52:46] RESOLVED: [8x] SystemdUnitCrashLoop: purged.service crashloop on cp4047:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [19:52:58] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12357050 (10CDobbins) [19:53:45] RESOLVED: [14x] SystemdUnitCrashLoop: purged.service crashloop on cp4037:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [19:56:05] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2048 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2048 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [19:56:09] RESOLVED: [5x] HaproxyKafkaExporterDown: HaproxyKafka on cp2043 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [19:56:34] RESOLVED: [3x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns2004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [19:57:31] RESOLVED: [14x] SystemdUnitFailed: haproxy.service on cp2043:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:58:31] RESOLVED: [16x] PurgedHighEventLag: High event process lag with purged on cp2050:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [19:58:49] FIRING: [8x] PurgedHighBacklogQueue: Large backlog queue for purged on cp2043:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [20:02:04] RESOLVED: [20x] PurgedHighBacklogQueue: Large backlog queue for purged on cp2043:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [20:03:59] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw during switchover resulting from increased thumb.wm.o cache pressure - https://phabricator.wikimedia.org/T438896#12357070 (10jasmine_) [20:08:02] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw possibily resulting from increased thumb.wm.o cache pressure - https://phabricator.wikimedia.org/T438896#12357086 (10jasmine_) [20:08:30] FIRING: [3x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [20:09:43] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw possibily resulting from increased thumb.wm.o cache pressure - https://phabricator.wikimedia.org/T438896#12357089 (10jasmine_) [20:12:07] FIRING: SystemdUnitFailed: acme-chief-certs-sync.service on acmechief2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:24:01] FIRING: [6x] SystemdUnitFailed: acme-chief-certs-sync.service on acmechief2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:38:39] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12357173 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir5004.eqsin.wmnet with OS trixie executed with errors: - ncredir5004 (**FAIL**) - Downti... [20:39:51] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12357174 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir5004.eqsin.wmnet with OS trixie [21:22:08] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12357279 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir5004.eqsin.wmnet with OS trixie executed with errors: - ncredir5004 (**FAIL**) - Remove... [23:01:55] FIRING: [3x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown