[00:11:45] !log [WDQS] T430880 Repooled wdqs1014.eqiad.wmnet and wdqs2008.codfw.wmnet after Bookworm reimage, transfer, and postflight; wdqs2008 is serving, while wdqs1014 will remain outside of service until a pybal restart next monday [00:11:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:11:49] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [00:24:11] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs2023.codfw.wmnet with OS bookworm [00:24:29] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs1016.eqiad.wmnet with OS bookworm [00:27:41] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs2023 [00:27:53] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs1016 [00:27:53] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1016 [00:30:44] ryankemper@cumin2003 reimage (PID 158393) is awaiting input [00:31:09] !log ryankemper@cumin2003 START - Cookbook sre.dns.netbox [00:37:18] ryankemper@cumin2003 reimage (PID 158393) is awaiting input [00:43:54] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1016.eqiad.wmnet with reason: host reimage [00:49:21] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1016.eqiad.wmnet with reason: host reimage [01:03:24] 10ops-eqsin, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12155396 (10Papaul) While troubleshooting the loop issue today remote hands said : ` Dear Customer, Please note that on ASW-0604-EQSIN (Juniper EX4600... [01:12:54] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1316331 [01:12:54] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1316331 (owner: 10TrainBranchBot) [01:13:57] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:15:10] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1016.eqiad.wmnet with OS bookworm [01:24:58] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1316331 (owner: 10TrainBranchBot) [01:43:13] FIRING: CertAlmostExpired: Certificate for service lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [01:44:26] FIRING: [14x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [02:00:44] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:07:11] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 06m 26s) [02:08:57] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:13:57] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:18:57] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:24:26] FIRING: [4x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549) {#12267}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [02:25:24] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [02:29:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [02:30:02] !log ryankemper@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2023 - ryankemper@cumin2003" [02:30:07] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2023 - ryankemper@cumin2003" [02:30:07] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [02:30:08] !log ryankemper@cumin2003 START - Cookbook sre.dns.wipe-cache wdqs2023.codfw.wmnet 35.0.192.10.in-addr.arpa 5.3.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [02:30:11] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2023.codfw.wmnet 35.0.192.10.in-addr.arpa 5.3.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [02:30:12] !log ryankemper@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wdqs2023 [02:33:04] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2023 [02:33:04] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2023 [02:33:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:36:14] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [02:38:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:39:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [02:56:03] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2023.codfw.wmnet with reason: host reimage [02:59:40] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2023.codfw.wmnet with reason: host reimage [03:26:59] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2023.codfw.wmnet with OS bookworm [03:36:24] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [03:37:18] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [04:14:02] 10SRE-swift-storage, 10EasyTimeline, 07Regression: "Timeline error. Could not store output files" - https://phabricator.wikimedia.org/T428063#12155498 (10Ipr1) Making any edit on the timeline results in error : https://fi.wikipedia.org/wiki/Arch_Enemy Trying to import newer version from another wikipedia al... [04:19:19] 10SRE-swift-storage, 10EasyTimeline, 07Regression: "Timeline error. Could not store output files" - https://phabricator.wikimedia.org/T428063#12155500 (10Ipr1) Ok, so looks like timeline breaks when using swedish capital letter 'Å' in name: bar:Fredrik text:"Fredrik Åkesson". If you mangle the name into usi... [05:06:43] !log ryankemper@deploy1003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host [05:06:48] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [05:06:50] !log ryankemper@deploy1003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host (duration: 00m 06s) [05:07:00] !log ryankemper@deploy1003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host [05:07:07] !log ryankemper@deploy1003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host (duration: 00m 07s) [05:19:55] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards [05:19:59] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [05:20:26] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs1024.eqiad.wmnet -> wdqs2023.codfw.wmnet, repooling source-only afterwards [05:43:13] FIRING: CertAlmostExpired: Certificate for service lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [05:44:26] FIRING: [14x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [06:13:14] (03CR) 10Zaidusyy: "The voting Verified job passed (+2). The experimental catalog-compiler run" [puppet] - 10https://gerrit.wikimedia.org/r/1316125 (https://phabricator.wikimedia.org/T429901) (owner: 10Zaidusyy) [06:17:06] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs1024.eqiad.wmnet -> wdqs2023.codfw.wmnet, repooling source-only afterwards [06:17:10] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [06:24:26] FIRING: [4x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549) {#12267}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [06:35:36] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards [06:35:41] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [06:38:57] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:54:24] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [06:57:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [06:58:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:04:14] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [07:07:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [07:08:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [08:53:24] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [08:57:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [08:58:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:04:14] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:07:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [09:08:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:11:38] PROBLEM - SSH on arclamp2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:15:01] FIRING: [2x] RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [09:18:57] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:22:28] RECOVERY - SSH on arclamp2001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:23:57] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:25:01] RESOLVED: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [09:43:13] FIRING: CertAlmostExpired: Certificate for service lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [09:44:26] FIRING: [14x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:24:26] FIRING: [4x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549) {#12267}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [10:26:10] FIRING: BFDdown: BFD session down between cr2-magru and fe80::ee38:73ff:fee8:9c58 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:31:10] RESOLVED: BFDdown: BFD session down between cr2-magru and fe80::ee38:73ff:fee8:9c58 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:33:10] PROBLEM - OSPF status on cr1-drmrs is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [11:33:32] PROBLEM - OSPF status on cr2-eqiad is CRITICAL: OSPFv2: 10/11 UP : OSPFv3: 10/11 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [11:34:10] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 185.15.58.139 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:34:39] FIRING: CoreBGPDown: Core BGP session down between cr1-drmrs and cr2-eqiad (185.15.58.138) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=drmrs&var-device=cr1-drmrs:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [11:39:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-drmrs and cr2-eqiad (185.15.58.138) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=drmrs&var-device=cr1-drmrs:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [12:12:03] (03PS1) 10Jforrester: Drop old support for non-temporary accounts wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316832 [13:05:19] (03CR) 10Dreamy Jazz: [C:03+1] Drop old support for non-temporary accounts wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316832 (owner: 10Jforrester) [13:23:57] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:43:13] FIRING: CertAlmostExpired: Certificate for service lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:44:26] FIRING: [14x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:07:30] 06SRE, 06Editing-team, 06Fundraising-Backlog, 06Traffic-Icebox, and 5 others: RFC: Serve Main Page of Wikimedia wikis from a consistent URL - https://phabricator.wikimedia.org/T120085#12156032 (10Izno) Is this task done? :) [14:11:47] 06SRE, 06Editing-team, 06Fundraising-Backlog, 06Traffic-Icebox, and 5 others: RFC: Serve Main Page of Wikimedia wikis from a consistent URL - https://phabricator.wikimedia.org/T120085#12156035 (10Krinkle) This task hasn't begun. No WMF wikis use a root main page. The fixcopyrightwiki experiment ended when... [14:13:34] 06SRE, 06Editing-team, 06Fundraising-Backlog, 06Traffic-Icebox, and 5 others: RFC: Serve Main Page of Wikimedia wikis from a consistent URL - https://phabricator.wikimedia.org/T120085#12156039 (10Izno) Ah, I was confusing it for the m. work which was obviously elsewhere. Sorry about the noise. [14:24:26] FIRING: [4x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549) {#12267}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:53:57] FIRING: [15x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:10:36] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, July 27 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307270 (https://phabricator.wikimedia.org/T431060) (owner: 10Novem Linguae) [15:34:25] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 185.15.58.139 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:35:24] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [15:37:14] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [15:39:54] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-drmrs and cr2-eqiad (185.15.58.138) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=drmrs&var-device=cr1-drmrs:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [16:08:57] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:13:57] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:43:13] FIRING: CertAlmostExpired: Certificate for service lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [17:43:56] !log ryankemper@cumin2003 conftool action : set/pooled=yes; selector: name=wdqs1016.eqiad.wmnet [17:44:24] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [17:45:45] !log ryankemper@cumin2003 conftool action : set/pooled=yes; selector: name=wdqs2023.codfw.wmnet [17:48:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [17:48:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:49:16] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [17:53:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [17:53:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:05:57] !log [WDQS] T430880 Reimaged `wdqs1016` and `wdqs2023` to Bookworm with `--move-vlan`, restored main and scholarly data, validated postflights, and repooled both hosts. Confirmed PyBal rebuilt both backends with their new addresses [18:06:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:06:01] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [18:14:11] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs2024.codfw.wmnet with OS bookworm [18:14:13] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs1017.eqiad.wmnet with OS bookworm [18:17:31] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs2024 [18:17:44] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs1017 [18:18:07] !log ryankemper@cumin2003 START - Cookbook sre.dns.netbox [18:23:49] !log ryankemper@cumin2003 START - Cookbook sre.dns.netbox [18:24:25] !log ryankemper@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1017 - ryankemper@cumin2003" [18:24:26] FIRING: [4x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549) {#12267}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [18:24:30] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1017 - ryankemper@cumin2003" [18:24:30] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [18:24:31] !log ryankemper@cumin2003 START - Cookbook sre.dns.wipe-cache wdqs1017.eqiad.wmnet 238.48.64.10.in-addr.arpa 8.3.2.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [18:24:34] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1017.eqiad.wmnet 238.48.64.10.in-addr.arpa 8.3.2.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [18:24:35] !log ryankemper@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wdqs1017 [18:26:54] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [18:26:55] !log ryankemper@cumin2003 START - Cookbook sre.dns.wipe-cache wdqs2024.codfw.wmnet 58.16.192.10.in-addr.arpa 8.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [18:26:58] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2024.codfw.wmnet 58.16.192.10.in-addr.arpa 8.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [18:26:59] !log ryankemper@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wdqs2024 [18:30:03] ryankemper@cumin2003 reimage (PID 358308) is awaiting input [18:30:23] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1017 [18:30:24] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1017 [18:30:25] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2024 [18:30:25] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2024 [18:37:46] (03CR) 10Ryan Kemper: [C:03+2] pontoon: add rkemper-kafka stack [puppet] - 10https://gerrit.wikimedia.org/r/1308355 (https://phabricator.wikimedia.org/T276088) (owner: 10Ryan Kemper) [18:52:14] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1017.eqiad.wmnet with reason: host reimage [18:53:57] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2024.codfw.wmnet with reason: host reimage [18:54:26] FIRING: [15x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [18:58:46] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1017.eqiad.wmnet with reason: host reimage [19:02:10] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2024.codfw.wmnet with reason: host reimage [19:23:39] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1017.eqiad.wmnet with OS bookworm [19:26:52] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2024.codfw.wmnet with OS bookworm [19:34:25] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 185.15.58.139 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [19:39:54] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-drmrs and cr2-eqiad (185.15.58.138) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=drmrs&var-device=cr1-drmrs:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [19:57:20] !log ryankemper@deploy1003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host [19:57:24] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [19:57:27] !log ryankemper@deploy1003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host (duration: 00m 07s) [19:57:32] !log ryankemper@deploy1003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host [19:57:38] !log ryankemper@deploy1003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host (duration: 00m 05s) [20:08:57] FIRING: [4x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549) {#12267}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [20:09:43] (03PS1) 10Ryan Kemper: sre.wdqs.data-transfer: abort on failed transfer [cookbooks] - 10https://gerrit.wikimedia.org/r/1317127 (https://phabricator.wikimedia.org/T430880) [20:09:45] (03PS1) 10Ryan Kemper: sre.wdqs: stop transferpy burying cookbook output [cookbooks] - 10https://gerrit.wikimedia.org/r/1317128 (https://phabricator.wikimedia.org/T430880) [20:15:11] !log ssastry@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [20:15:14] !log ssastry@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [20:15:15] !log ssastry@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [20:15:18] !log ssastry@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [20:18:21] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs1024.eqiad.wmnet -> wdqs2024.codfw.wmnet, repooling source-only afterwards [20:18:25] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [20:18:32] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs1017.eqiad.wmnet, repooling source-only afterwards [20:52:45] !log ssastry@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [20:52:47] !log ssastry@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [20:52:49] !log ssastry@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [20:52:52] !log ssastry@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [21:06:50] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs1024.eqiad.wmnet -> wdqs2024.codfw.wmnet, repooling source-only afterwards [21:06:54] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [21:26:45] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs1017.eqiad.wmnet, repooling source-only afterwards [21:26:50] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [21:43:13] FIRING: CertAlmostExpired: Certificate for service lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [21:53:57] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [22:03:33] !log ryankemper@cumin2003 conftool action : set/pooled=yes; selector: name=wdqs1017.eqiad.wmnet [22:04:46] !log ryankemper@cumin2003 conftool action : set/pooled=yes; selector: name=wdqs2024.codfw.wmnet [22:06:17] !log T430880 [WDQS] Repooled `wdqs1017` and `wdqs2024` after reimaging to bookworm, scap deploying, and data xfering [22:06:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:06:21] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [22:13:57] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549) {#12267}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [22:24:10] FIRING: [4x] BFDdown: BFD session down between cr1-codfw and fe80::5e5e:abff:fe3d:8198 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [22:54:26] FIRING: [15x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:04:23] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [23:08:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [23:08:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:09:13] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [23:13:31] RECOVERY - OSPF status on cr2-eqiad is OK: OSPFv2: 11/11 UP : OSPFv3: 11/11 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [23:13:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [23:13:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:14:11] RECOVERY - OSPF status on cr1-drmrs is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [23:19:10] RESOLVED: [2x] BFDdown: BFD session down between cr2-eqiad and 185.15.58.139 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [23:19:39] RESOLVED: [2x] CoreBGPDown: Core BGP session down between cr1-drmrs and cr2-eqiad (185.15.58.138) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=drmrs&var-device=cr1-drmrs:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [23:23:37] PROBLEM - SSH on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [23:23:37] PROBLEM - librenms.wikimedia.org tls expiry on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [23:23:41] PROBLEM - librenms.wikimedia.org requires authentication on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [23:24:37] RECOVERY - librenms.wikimedia.org tls expiry on netmon2002 is OK: OK - Certificate librenms.wikimedia.org will expire on Thu 10 Sep 2026 02:24:49 AM GMT +0000. https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [23:27:27] RECOVERY - SSH on netmon2002 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u10 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [23:27:31] RECOVERY - librenms.wikimedia.org requires authentication on netmon2002 is OK: HTTP OK: Status line output matched HTTP/1.1 302 - 701 bytes in 0.131 second response time https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [23:28:57] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:36:37] PROBLEM - SSH on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [23:36:37] PROBLEM - librenms.wikimedia.org tls expiry on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [23:36:41] PROBLEM - librenms.wikimedia.org requires authentication on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [23:38:57] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-codfw:xe-1/1/1:0 (Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294) {#12252_12295-1}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [23:42:11] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1317235 [23:42:12] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1317235 (owner: 10TrainBranchBot) [23:43:57] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:43:57] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:xe-1/1/1:0 (Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294) {#12252_12295-1}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [23:53:57] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:xe-1/1/1:0 (Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294) {#12252_12295-1}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [23:54:48] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1317235 (owner: 10TrainBranchBot) [23:57:23] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring