[00:00:04] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [00:00:14] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [00:00:19] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [00:00:29] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [00:00:32] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [00:00:39] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [00:04:29] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:39:03] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1369157 [00:39:03] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1369157 (owner: 10TrainBranchBot) [00:46:22] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1369157 (owner: 10TrainBranchBot) [01:09:04] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1369159 [01:09:04] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1369159 (owner: 10TrainBranchBot) [01:16:41] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1369159 (owner: 10TrainBranchBot) [01:22:00] FIRING: PingLossPercent: Blackbox probe packet loss 4.375% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [01:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.57% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [02:05:25] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [02:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [02:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [02:57:45] FIRING: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [03:06:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 29d 15h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [03:15:24] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [03:52:00] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [04:04:29] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:07:01] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.958% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [04:12:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [04:52:58] ACKNOWLEDGEMENT - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1194 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 1 Failed : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T444742 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [04:53:04] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1194 - https://phabricator.wikimedia.org/T444742 (10ops-monitoring-bot) 03NEW [05:09:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12553523 (10phaultfinder) [05:34:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12553536 (10phaultfinder) [05:35:25] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-eqdfw:ae0 (External: Facebook) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqdfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [05:42:41] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-eqdfw:ae0 (External: Facebook) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqdfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [05:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.6% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [05:54:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12553541 (10phaultfinder) [06:05:25] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:24:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12553562 (10phaultfinder) [06:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:45:42] FIRING: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [06:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [07:00:04] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261113T0700) [07:00:42] RESOLVED: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:06:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 29d 11h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [07:12:01] RESOLVED: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [07:15:24] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [07:20:09] (03PS1) 10Kevin Bazira: ml-services: deploy tts-section-generator server that supports paragraph-level captions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369265 (https://phabricator.wikimedia.org/T444576) [07:29:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12553603 (10phaultfinder) [08:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261113T0800) [08:04:29] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:06:09] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission clouddb1014.eqiad.wmnet - https://phabricator.wikimedia.org/T441395#12553643 (10VRiley-WMF) a:03VRiley-WMF [08:06:31] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission clouddb1015.eqiad.wmnet - https://phabricator.wikimedia.org/T443768#12553644 (10VRiley-WMF) a:03VRiley-WMF [08:07:16] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1194 - https://phabricator.wikimedia.org/T443980#12553648 (10VRiley-WMF) a:03VRiley-WMF [08:07:24] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for backup1010.mgmt:22 - https://phabricator.wikimedia.org/T443829#12553649 (10VRiley-WMF) a:03VRiley-WMF [08:07:40] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for an-worker1147.mgmt:22 - https://phabricator.wikimedia.org/T443870#12553650 (10VRiley-WMF) a:03VRiley-WMF [08:07:55] (03PS2) 10Majavah: dnsrecursor: Use a proper data type for forward zone data [puppet] - 10https://gerrit.wikimedia.org/r/1250626 [08:08:10] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-a4-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T444641#12553651 (10VRiley-WMF) a:03VRiley-WMF [08:09:17] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1194 - https://phabricator.wikimedia.org/T443980#12553653 (10VRiley-WMF) 05Open→03Resolved Clsoing as a duplicate of T444742 [08:09:23] (03PS3) 10Majavah: dnsrecursor: Use a proper data type for forward zone data [puppet] - 10https://gerrit.wikimedia.org/r/1250626 [08:09:46] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1194 - https://phabricator.wikimedia.org/T444742#12553660 (10VRiley-WMF) a:03VRiley-WMF [08:10:24] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install (2) new puppetservers - https://phabricator.wikimedia.org/T432998#12553661 (10Jhancock.wm) a:05Jhancock.wm→03MoritzMuehlenhoff got this error on both reimages Attempt to run 'cookbooks.sre.hosts.reimage.ReimageRunner._p... [08:11:01] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (DIFF 1 CORE_DIFF 3): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/" [puppet] - 10https://gerrit.wikimedia.org/r/1250626 (owner: 10Majavah) [08:11:19] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install (2) new puppetservers - https://phabricator.wikimedia.org/T432998#12553663 (10Jhancock.wm) [08:11:30] (03PS4) 10Majavah: dnsrecursor: Use a proper data type for forward zone data [puppet] - 10https://gerrit.wikimedia.org/r/1250626 [08:13:19] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install (2) new puppetservers - https://phabricator.wikimedia.org/T432998#12553665 (10Jhancock.wm) a:05MoritzMuehlenhoff→03Jhancock.wm coffee kicked in. found the discrepancy [08:13:35] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (DIFF 4): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9654/console" [puppet] - 10https://gerrit.wikimedia.org/r/1250626 (owner: 10Majavah) [08:13:48] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install (2) new puppetservers - https://phabricator.wikimedia.org/T432998#12553667 (10Jhancock.wm) [08:14:53] (03CR) 10Majavah: [V:03+1] dnsrecursor: Use a proper data type for forward zone data (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1250626 (owner: 10Majavah) [08:14:58] (03CR) 10Bartosz Wójtowicz: [C:03+1] ml-services: deploy tts-section-generator server that supports paragraph-level captions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369265 (https://phabricator.wikimedia.org/T444576) (owner: 10Kevin Bazira) [08:15:03] !log jhancock@cumin2003 START - Cookbook sre.dns.netbox [08:17:27] !log jhancock@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [08:19:36] 10ops-codfw, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T444749 (10phaultfinder) 03NEW [08:21:01] (03Abandoned) 10Majavah: P:base: Make nftables::set resources always defined [puppet] - 10https://gerrit.wikimedia.org/r/1266205 (owner: 10Majavah) [08:28:10] PROBLEM - Host an-worker1194 is DOWN: PING CRITICAL - Packet loss = 100% [08:29:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12553707 (10phaultfinder) [08:32:25] (03PS1) 10Majavah: monitoring: Do not provision Icinga hosts without services [puppet] - 10https://gerrit.wikimedia.org/r/1369312 [08:32:25] (03PS1) 10Majavah: hieradata: Remove check_ssh from cloudcumin2001 [puppet] - 10https://gerrit.wikimedia.org/r/1369313 [08:32:52] (03CR) 10Kevin Bazira: [C:03+2] ml-services: deploy tts-section-generator server that supports paragraph-level captions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369265 (https://phabricator.wikimedia.org/T444576) (owner: 10Kevin Bazira) [08:33:45] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9655/console" [puppet] - 10https://gerrit.wikimedia.org/r/1369313 (owner: 10Majavah) [08:34:51] (03Merged) 10jenkins-bot: ml-services: deploy tts-section-generator server that supports paragraph-level captions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369265 (https://phabricator.wikimedia.org/T444576) (owner: 10Kevin Bazira) [08:35:17] (03CR) 10CI reject: [V:04-1] monitoring: Do not provision Icinga hosts without services [puppet] - 10https://gerrit.wikimedia.org/r/1369312 (owner: 10Majavah) [08:39:21] FIRING: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 168.8888888888889 times in the last 10 minutes due to memory usage (mw@codfw to codfw) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [08:39:30] !log kevinbazira@deploy2003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [08:41:16] !log kevinbazira@deploy2003 helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . [08:42:32] !log kevinbazira@deploy2003 helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [08:43:11] (03CR) 10Atsuko: [C:03+1] global_config: drop the gitlab-wikimedia external services policy [puppet] - 10https://gerrit.wikimedia.org/r/1368995 (https://phabricator.wikimedia.org/T444600) (owner: 10Brouberol) [08:43:26] (03CR) 10Brouberol: [C:03+2] global_config: drop the gitlab-wikimedia external services policy [puppet] - 10https://gerrit.wikimedia.org/r/1368995 (https://phabricator.wikimedia.org/T444600) (owner: 10Brouberol) [08:46:13] (03PS1) 10Bartosz Wójtowicz: kserve-llm-inference: use vLLM 0.32 as the default image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369316 (https://phabricator.wikimedia.org/T444464) [08:51:56] (03PS2) 10Majavah: monitoring: Do not provision Icinga hosts without services [puppet] - 10https://gerrit.wikimedia.org/r/1369312 [08:51:56] (03PS2) 10Majavah: hieradata: Remove check_ssh from cloudcumin2001 [puppet] - 10https://gerrit.wikimedia.org/r/1369313 [08:53:42] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9656/console" [puppet] - 10https://gerrit.wikimedia.org/r/1369313 (owner: 10Majavah) [08:54:30] (03CR) 10CI reject: [V:04-1] monitoring: Do not provision Icinga hosts without services [puppet] - 10https://gerrit.wikimedia.org/r/1369312 (owner: 10Majavah) [08:56:38] RECOVERY - Host an-worker1194 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [08:57:37] !log jhancock@cumin2003 START - Cookbook sre.hosts.reimage for host puppetserve2005.codfw.wmnet with OS bookworm [08:57:47] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host puppetserve2005.codfw.wmnet (push config only) [08:57:50] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install (2) new puppetservers - https://phabricator.wikimedia.org/T432998#12553789 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2003 for host puppetserve2005.codfw.wmnet with OS bookworm [08:57:51] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2005.codfw.wmnet (push config only) [08:58:13] !log jhancock@cumin2003 START - Cookbook sre.hosts.reimage for host puppetserve2006.codfw.wmnet with OS bookworm [08:58:20] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install (2) new puppetservers - https://phabricator.wikimedia.org/T432998#12553790 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2003 for host puppetserve2006.codfw.wmnet with OS bookworm [08:58:23] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host puppetserve2006.codfw.wmnet (push config only) [08:58:27] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2006.codfw.wmnet (push config only) [08:58:42] (03PS1) 10Arnaudb: stewards: Do not show diffs of the onboarder config [puppet] - 10https://gerrit.wikimedia.org/r/1369332 [08:58:45] (03PS1) 10Arnaudb: releases: Do not show diffs of the Jenkins release secrets [puppet] - 10https://gerrit.wikimedia.org/r/1369331 [08:58:46] (03PS1) 10Arnaudb: vrts: Do not show diffs of Config.pm [puppet] - 10https://gerrit.wikimedia.org/r/1369328 [08:58:48] (03PS1) 10Arnaudb: doc: Do not show diffs of the rsync secrets [puppet] - 10https://gerrit.wikimedia.org/r/1369330 [08:58:49] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1194 - https://phabricator.wikimedia.org/T444742#12553791 (10VRiley-WMF) 05Open→03Resolved Updated the firmeare of the Bios. That seemed to correct the issue with the harddrive. Closing this ticket. [08:58:50] (03PS1) 10Arnaudb: phabricator: Do not show diffs of files that hold secrets [puppet] - 10https://gerrit.wikimedia.org/r/1369329 [08:58:53] (03PS1) 10Arnaudb: etherpad: Do not show diffs of settings.json [puppet] - 10https://gerrit.wikimedia.org/r/1369327 [08:58:56] (03PS1) 10Arnaudb: stewards: Keep the users DB token out of the git URL [puppet] - 10https://gerrit.wikimedia.org/r/1369323 [08:58:58] (03PS1) 10Arnaudb: mailman3: Do not show diffs of the configs with secrets [puppet] - 10https://gerrit.wikimedia.org/r/1369326 [08:59:01] (03PS1) 10Arnaudb: gerrit: Do not show diffs of secure.config [puppet] - 10https://gerrit.wikimedia.org/r/1369325 [08:59:05] (03PS1) 10Arnaudb: gitlab: Do not show diffs of files that hold secrets [puppet] - 10https://gerrit.wikimedia.org/r/1369324 [08:59:09] (03PS1) 10Arnaudb: lists: Alert on a Mailman outbound queue backlog [puppet] - 10https://gerrit.wikimedia.org/r/1369322 [08:59:13] (03PS1) 10Arnaudb: vrts: Stop the install and upgrade scripts on a bad download [puppet] - 10https://gerrit.wikimedia.org/r/1369320 [08:59:17] (03PS1) 10Arnaudb: ci: Do not show diffs of the docker-pkg configs [puppet] - 10https://gerrit.wikimedia.org/r/1369318 [08:59:20] RESOLVED: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 24.44453703755144 times in the last 10 minutes due to memory usage (mw@codfw to codfw) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [08:59:21] (03PS1) 10Arnaudb: zuul: Do not show diffs of files that hold secrets [puppet] - 10https://gerrit.wikimedia.org/r/1369317 [08:59:25] (03PS1) 10Arnaudb: jenkins: Back up the Jenkins controllers [puppet] - 10https://gerrit.wikimedia.org/r/1369319 [08:59:29] (03PS1) 10Arnaudb: phabricator: Keep the envoy auto-restart where aphlict is absent [puppet] - 10https://gerrit.wikimedia.org/r/1369315 [08:59:33] (03PS1) 10Arnaudb: gitlab: Read the backup file argument of gitlab-restore -f [puppet] - 10https://gerrit.wikimedia.org/r/1369314 [08:59:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12553795 (10phaultfinder) [09:00:19] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host puppetserve2005.codfw.wmnet (remove config only) [09:00:23] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2005.codfw.wmnet (remove config only) [09:00:25] !log jhancock@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host puppetserve2005.codfw.wmnet with OS bookworm [09:00:33] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install (2) new puppetservers - https://phabricator.wikimedia.org/T432998#12553796 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2003 for host puppetserve2005.codfw.wmnet with OS bookworm execut... [09:01:40] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host puppetserve2006.codfw.wmnet (remove config only) [09:01:44] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2006.codfw.wmnet (remove config only) [09:01:46] !log jhancock@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host puppetserve2006.codfw.wmnet with OS bookworm [09:01:57] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install (2) new puppetservers - https://phabricator.wikimedia.org/T432998#12553798 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2003 for host puppetserve2006.codfw.wmnet with OS bookworm execut... [09:02:34] (03PS3) 10Majavah: monitoring: Do not provision Icinga hosts without services [puppet] - 10https://gerrit.wikimedia.org/r/1369312 [09:02:34] (03PS3) 10Majavah: hieradata: Remove check_ssh from cloudcumin2001 [puppet] - 10https://gerrit.wikimedia.org/r/1369313 [09:02:50] RECOVERY - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1194 is OK: communication: 0 OK : controller: 0 OK : physical_disk: 0 OK : virtual_disk: 0 OK : bbu: 0 OK : enclosure: 0 OK https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [09:04:31] !log jhancock@cumin2003 START - Cookbook sre.dns.netbox [09:04:45] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (DIFF 3): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9657/console" [puppet] - 10https://gerrit.wikimedia.org/r/1369313 (owner: 10Majavah) [09:06:40] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [09:07:41] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [09:08:12] !log brouberol@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [09:08:27] !log jhancock@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding puppetserve2005 to codfw - jhancock@cumin2003" [09:08:31] !log jhancock@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding puppetserve2005 to codfw - jhancock@cumin2003" [09:08:31] !log jhancock@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [09:08:46] !log brouberol@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [09:11:36] (03CR) 10Ladsgroup: varnish: Don't strip browser Cache-Control:maxage from action=raw (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1364773 (https://phabricator.wikimedia.org/T279120) (owner: 10Krinkle) [09:15:36] (03CR) 10Muehlenhoff: [C:03+1] "Looks good, thanks" [puppet] - 10https://gerrit.wikimedia.org/r/1369015 (https://phabricator.wikimedia.org/T442758) (owner: 10Daniel Kinzler) [09:15:38] (03CR) 10Muehlenhoff: [C:03+2] admin: Remove old SSH key for DKinzler [puppet] - 10https://gerrit.wikimedia.org/r/1369015 (https://phabricator.wikimedia.org/T442758) (owner: 10Daniel Kinzler) [09:20:52] (03PS1) 10Trueg: WDQSv2: Update wdqs-qlever to 0.10.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369337 (https://phabricator.wikimedia.org/T444329) [09:21:05] (03PS1) 10Muehlenhoff: Remove legacy key for fab [puppet] - 10https://gerrit.wikimedia.org/r/1369338 (https://phabricator.wikimedia.org/T442665) [09:22:46] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy key for fab [puppet] - 10https://gerrit.wikimedia.org/r/1369338 (https://phabricator.wikimedia.org/T442665) (owner: 10Muehlenhoff) [09:23:57] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for fab - https://phabricator.wikimedia.org/T442665#12553838 (10MoritzMuehlenhoff) 05Stalled→03Resolved a:03MoritzMuehlenhoff >>! In T442665#12550093, @fkaelin wrote: > Sorry for the delay - I confirm that the yubikey works... [09:24:56] (03PS1) 10Muehlenhoff: Remove legacy key for cklimas [puppet] - 10https://gerrit.wikimedia.org/r/1369339 (https://phabricator.wikimedia.org/T442758) [09:28:08] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy key for cklimas [puppet] - 10https://gerrit.wikimedia.org/r/1369339 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [09:31:46] (03PS1) 10Arnaudb: planet: Delete theme files and templates that nothing uses [puppet] - 10https://gerrit.wikimedia.org/r/1369341 [09:31:49] (03PS1) 10Arnaudb: releases: Delete the reprepro files [puppet] - 10https://gerrit.wikimedia.org/r/1369343 [09:31:51] (03PS1) 10Arnaudb: microsites: Delete the os-reports vhost and the static RT index [puppet] - 10https://gerrit.wikimedia.org/r/1369344 [09:31:53] (03PS1) 10Arnaudb: lists: Delete the unused Mailman 3 exim templates [puppet] - 10https://gerrit.wikimedia.org/r/1369345 [09:31:56] (03PS1) 10Arnaudb: mailman3: Remove the Mailman 2 migration script [puppet] - 10https://gerrit.wikimedia.org/r/1369346 [09:31:59] (03PS1) 10Arnaudb: phabricator: Delete unused files, parameters and an exec [puppet] - 10https://gerrit.wikimedia.org/r/1369348 [09:32:06] (03PS1) 10Arnaudb: planet: Delete the strings and the logo of the old index page [puppet] - 10https://gerrit.wikimedia.org/r/1369342 [09:32:08] (03PS1) 10Arnaudb: contint: Delete the unused rspec scaffolding [puppet] - 10https://gerrit.wikimedia.org/r/1369349 [09:32:10] (03PS1) 10Arnaudb: vrts: Delete unused files and finished cleanups [puppet] - 10https://gerrit.wikimedia.org/r/1369347 [09:50:51] 10ops-eqsin, 06SRE: eqsin power feed down Nov 2026 - https://phabricator.wikimedia.org/T444574#12553890 (10ayounsi) 05Open→03Resolved a:03ayounsi That came back on its own at ~8am [09:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.62% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [09:56:27] (03PS1) 10Jelto: admin: Add FIDO SSH key for jgiannelos [puppet] - 10https://gerrit.wikimedia.org/r/1369353 (https://phabricator.wikimedia.org/T442758) [09:58:18] (03PS1) 10Dzahn: phabricator: remove the unused local aphlict path [puppet] - 10https://gerrit.wikimedia.org/r/1369354 [10:02:58] 10ops-eqiad, 06SRE, 06DC-Ops: Check list of PXE miss-configs for eqiad - https://phabricator.wikimedia.org/T401441#12553927 (10brouberol) @VRiley-WMF `dse-k8s-worker` hosts can be operated upon at any point. The following command should be run from a cumin host: ` brouberol@cumin1004:~$ cookbook sre.k8s.pool... [10:05:25] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:08:44] (03PS2) 10Arnaudb: collab services: fix typos in comments and docs [puppet] - 10https://gerrit.wikimedia.org/r/1369037 (https://phabricator.wikimedia.org/T444758) (owner: 10Dzahn) [10:08:51] (03PS2) 10Arnaudb: gitlab: Read the backup file argument of gitlab-restore -f [puppet] - 10https://gerrit.wikimedia.org/r/1369314 (https://phabricator.wikimedia.org/T444758) [10:08:57] (03PS2) 10Arnaudb: phabricator: Keep the envoy auto-restart where aphlict is absent [puppet] - 10https://gerrit.wikimedia.org/r/1369315 (https://phabricator.wikimedia.org/T444758) [10:09:04] (03PS2) 10Arnaudb: zuul: Do not show diffs of files that hold secrets [puppet] - 10https://gerrit.wikimedia.org/r/1369317 (https://phabricator.wikimedia.org/T444758) [10:09:10] (03PS2) 10Arnaudb: ci: Do not show diffs of the docker-pkg configs [puppet] - 10https://gerrit.wikimedia.org/r/1369318 (https://phabricator.wikimedia.org/T444758) [10:09:17] (03PS2) 10Arnaudb: jenkins: Back up the Jenkins controllers [puppet] - 10https://gerrit.wikimedia.org/r/1369319 (https://phabricator.wikimedia.org/T444758) [10:09:23] (03PS2) 10Arnaudb: vrts: Stop the install and upgrade scripts on a bad download [puppet] - 10https://gerrit.wikimedia.org/r/1369320 (https://phabricator.wikimedia.org/T444758) [10:09:29] (03PS2) 10Arnaudb: lists: Alert on a Mailman outbound queue backlog [puppet] - 10https://gerrit.wikimedia.org/r/1369322 (https://phabricator.wikimedia.org/T444758) [10:09:36] (03PS2) 10Arnaudb: stewards: Keep the users DB token out of the git URL [puppet] - 10https://gerrit.wikimedia.org/r/1369323 (https://phabricator.wikimedia.org/T444758) [10:09:42] (03PS2) 10Arnaudb: gitlab: Do not show diffs of files that hold secrets [puppet] - 10https://gerrit.wikimedia.org/r/1369324 (https://phabricator.wikimedia.org/T444758) [10:09:50] (03PS2) 10Arnaudb: gerrit: Do not show diffs of secure.config [puppet] - 10https://gerrit.wikimedia.org/r/1369325 (https://phabricator.wikimedia.org/T444758) [10:09:51] (03PS1) 10Jelto: admin: Add FIDO SSH key for stran [puppet] - 10https://gerrit.wikimedia.org/r/1369357 (https://phabricator.wikimedia.org/T442758) [10:09:57] (03PS2) 10Arnaudb: mailman3: Do not show diffs of the configs with secrets [puppet] - 10https://gerrit.wikimedia.org/r/1369326 (https://phabricator.wikimedia.org/T444758) [10:10:03] (03PS2) 10Arnaudb: etherpad: Do not show diffs of settings.json [puppet] - 10https://gerrit.wikimedia.org/r/1369327 (https://phabricator.wikimedia.org/T444758) [10:10:09] (03PS2) 10Arnaudb: vrts: Do not show diffs of Config.pm [puppet] - 10https://gerrit.wikimedia.org/r/1369328 (https://phabricator.wikimedia.org/T444758) [10:10:16] (03PS2) 10Arnaudb: phabricator: Do not show diffs of files that hold secrets [puppet] - 10https://gerrit.wikimedia.org/r/1369329 (https://phabricator.wikimedia.org/T444758) [10:10:22] (03PS2) 10Arnaudb: doc: Do not show diffs of the rsync secrets [puppet] - 10https://gerrit.wikimedia.org/r/1369330 (https://phabricator.wikimedia.org/T444758) [10:10:29] (03PS2) 10Arnaudb: releases: Do not show diffs of the Jenkins release secrets [puppet] - 10https://gerrit.wikimedia.org/r/1369331 (https://phabricator.wikimedia.org/T444758) [10:10:34] (03PS2) 10Arnaudb: stewards: Do not show diffs of the onboarder config [puppet] - 10https://gerrit.wikimedia.org/r/1369332 (https://phabricator.wikimedia.org/T444758) [10:10:41] (03PS2) 10Arnaudb: planet: Delete theme files and templates that nothing uses [puppet] - 10https://gerrit.wikimedia.org/r/1369341 (https://phabricator.wikimedia.org/T444758) [10:10:47] (03PS2) 10Arnaudb: planet: Delete the strings and the logo of the old index page [puppet] - 10https://gerrit.wikimedia.org/r/1369342 (https://phabricator.wikimedia.org/T444758) [10:10:52] 06SRE, 10LDAP-Access-Requests: Grant Access to nda LDAP group for Monrac5 - https://phabricator.wikimedia.org/T443971#12553978 (10Dzahn) Thanks! I added monrac5 to the `nda` LDAP group. [10:10:53] (03PS2) 10Arnaudb: releases: Delete the reprepro files [puppet] - 10https://gerrit.wikimedia.org/r/1369343 (https://phabricator.wikimedia.org/T444758) [10:10:59] (03PS2) 10Arnaudb: microsites: Delete the os-reports vhost and the static RT index [puppet] - 10https://gerrit.wikimedia.org/r/1369344 (https://phabricator.wikimedia.org/T444758) [10:11:00] 06SRE, 10LDAP-Access-Requests: Grant Access to nda LDAP group for Monrac5 - https://phabricator.wikimedia.org/T443971#12553980 (10Dzahn) 05Open→03Resolved a:03Dzahn [10:11:06] (03PS2) 10Arnaudb: lists: Delete the unused Mailman 3 exim templates [puppet] - 10https://gerrit.wikimedia.org/r/1369345 (https://phabricator.wikimedia.org/T444758) [10:11:13] (03PS2) 10Arnaudb: mailman3: Remove the Mailman 2 migration script [puppet] - 10https://gerrit.wikimedia.org/r/1369346 (https://phabricator.wikimedia.org/T444758) [10:11:18] (03PS2) 10Arnaudb: vrts: Delete unused files and finished cleanups [puppet] - 10https://gerrit.wikimedia.org/r/1369347 (https://phabricator.wikimedia.org/T444758) [10:11:26] (03PS2) 10Arnaudb: phabricator: Delete unused files, parameters and an exec [puppet] - 10https://gerrit.wikimedia.org/r/1369348 (https://phabricator.wikimedia.org/T444758) [10:11:32] (03PS2) 10Arnaudb: contint: Delete the unused rspec scaffolding [puppet] - 10https://gerrit.wikimedia.org/r/1369349 (https://phabricator.wikimedia.org/T444758) [10:12:15] (03PS1) 10Dzahn: phabricator: remove the unused local aphlict path [puppet] - 10https://gerrit.wikimedia.org/r/1369358 (https://phabricator.wikimedia.org/T444758) [10:12:17] (03PS1) 10Dzahn: phabricator: remove the unused aphlict TLS settings [puppet] - 10https://gerrit.wikimedia.org/r/1369359 (https://phabricator.wikimedia.org/T444758) [10:12:19] (03PS1) 10Dzahn: phabricator: remove unused params from profile::phabricator::main [puppet] - 10https://gerrit.wikimedia.org/r/1369360 (https://phabricator.wikimedia.org/T444758) [10:12:21] (03PS1) 10Dzahn: phabricator: remove the old ssh-phab leftovers from phabricator::vcs [puppet] - 10https://gerrit.wikimedia.org/r/1369361 (https://phabricator.wikimedia.org/T444758) [10:12:23] (03PS1) 10Dzahn: phabricator: remove two unreferenced files [puppet] - 10https://gerrit.wikimedia.org/r/1369362 (https://phabricator.wikimedia.org/T444758) [10:12:25] (03PS1) 10Dzahn: phabricator: remove the disabled mfa_check logmail [puppet] - 10https://gerrit.wikimedia.org/r/1369363 (https://phabricator.wikimedia.org/T444758) [10:12:27] (03PS1) 10Dzahn: phabricator: remove the unused phabricator::conf_env define [puppet] - 10https://gerrit.wikimedia.org/r/1369364 (https://phabricator.wikimedia.org/T444758) [10:12:29] (03PS1) 10Dzahn: phabricator: remove unused params from the phabricator class [puppet] - 10https://gerrit.wikimedia.org/r/1369365 (https://phabricator.wikimedia.org/T444758) [10:12:31] (03PS1) 10Dzahn: phabricator: remove the apt_update_php exec that nothing triggers [puppet] - 10https://gerrit.wikimedia.org/r/1369366 (https://phabricator.wikimedia.org/T444758) [10:12:34] (03PS1) 10Dzahn: phabricator: pass the mysql admin user/pass to config directly [puppet] - 10https://gerrit.wikimedia.org/r/1369367 (https://phabricator.wikimedia.org/T444758) [10:12:39] (03PS1) 10Dzahn: phabricator: always manage the aphlict config with phabricator::config [puppet] - 10https://gerrit.wikimedia.org/r/1369368 (https://phabricator.wikimedia.org/T444758) [10:12:43] (03PS1) 10Dzahn: phabricator: remove two hiera keys without effect from the role [puppet] - 10https://gerrit.wikimedia.org/r/1369369 (https://phabricator.wikimedia.org/T444758) [10:12:47] (03PS1) 10Dzahn: phabricator: remove mysql ports equal to the defaults [puppet] - 10https://gerrit.wikimedia.org/r/1369370 (https://phabricator.wikimedia.org/T444758) [10:12:51] (03PS1) 10Dzahn: phabricator: remove hosts/aphlict1002.yaml, it repeats the role [puppet] - 10https://gerrit.wikimedia.org/r/1369371 (https://phabricator.wikimedia.org/T444758) [10:12:55] (03PS1) 10Dzahn: phabricator: remove unused mailrelay params [puppet] - 10https://gerrit.wikimedia.org/r/1369372 (https://phabricator.wikimedia.org/T444758) [10:12:59] (03PS1) 10Dzahn: phabricator: default logmail month/monthday to '*' directly [puppet] - 10https://gerrit.wikimedia.org/r/1369373 (https://phabricator.wikimedia.org/T444758) [10:13:03] (03PS1) 10Dzahn: phabricator: remove an unused variable, inline the httpd modules [puppet] - 10https://gerrit.wikimedia.org/r/1369374 (https://phabricator.wikimedia.org/T444758) [10:13:07] (03PS1) 10Dzahn: phabricator: remove a duplicate and an unread key from devtools hiera [puppet] - 10https://gerrit.wikimedia.org/r/1369375 (https://phabricator.wikimedia.org/T444758) [10:13:11] (03PS1) 10Dzahn: phabricator: fix the sndr_address lookup key in logmail [puppet] - 10https://gerrit.wikimedia.org/r/1369376 (https://phabricator.wikimedia.org/T444758) [10:20:24] (03PS1) 10Jelto: admin: Add FIDO SSH key for mmartorana [puppet] - 10https://gerrit.wikimedia.org/r/1369377 (https://phabricator.wikimedia.org/T442758) [10:22:14] (03PS1) 10Dzahn: gerrit: remove params the gerrit class takes but never uses [puppet] - 10https://gerrit.wikimedia.org/r/1369378 (https://phabricator.wikimedia.org/T444758) [10:22:16] (03PS1) 10Dzahn: gerrit: remove unused params from profile::gerrit::proxy [puppet] - 10https://gerrit.wikimedia.org/r/1369379 (https://phabricator.wikimedia.org/T444758) [10:22:18] (03PS1) 10Dzahn: gerrit: remove role::gerrit::migration and migration_base [puppet] - 10https://gerrit.wikimedia.org/r/1369380 (https://phabricator.wikimedia.org/T444758) [10:22:20] (03PS1) 10Dzahn: gerrit: remove host hiera that repeats the common and role values [puppet] - 10https://gerrit.wikimedia.org/r/1369381 (https://phabricator.wikimedia.org/T444758) [10:22:22] (03PS1) 10Dzahn: gerrit: take the replica/spare replication URLs from hiera only [puppet] - 10https://gerrit.wikimedia.org/r/1369382 (https://phabricator.wikimedia.org/T444758) [10:22:24] (03PS1) 10Dzahn: gerrit: remove hiera values that equal the defaults or are never read [puppet] - 10https://gerrit.wikimedia.org/r/1369383 (https://phabricator.wikimedia.org/T444758) [10:22:26] (03PS1) 10Dzahn: gerrit: drop the duplicate include of profile::java from the role [puppet] - 10https://gerrit.wikimedia.org/r/1369384 (https://phabricator.wikimedia.org/T444758) [10:22:29] (03PS1) 10Dzahn: gerrit: drop "!= undef" checks on params that cannot be undef [puppet] - 10https://gerrit.wikimedia.org/r/1369385 (https://phabricator.wikimedia.org/T444758) [10:22:33] (03PS1) 10Dzahn: gerrit: always use gerrit.config.erb, drop the $config param [puppet] - 10https://gerrit.wikimedia.org/r/1369386 (https://phabricator.wikimedia.org/T444758) [10:22:37] (03PS1) 10Dzahn: gerrit: use bool2str for the motd script, fix the file name header [puppet] - 10https://gerrit.wikimedia.org/r/1369387 (https://phabricator.wikimedia.org/T444758) [10:22:41] (03PS1) 10Dzahn: gerrit: remove three unread gerrit keys from devtools hiera [puppet] - 10https://gerrit.wikimedia.org/r/1369388 (https://phabricator.wikimedia.org/T444758) [10:26:51] (03PS1) 10Dzahn: releases: remove the unused reprepro files [puppet] - 10https://gerrit.wikimedia.org/r/1369390 (https://phabricator.wikimedia.org/T444758) [10:26:54] (03PS1) 10Dzahn: releases: remove params the releases class does not use [puppet] - 10https://gerrit.wikimedia.org/r/1369391 (https://phabricator.wikimedia.org/T444758) [10:26:56] (03PS1) 10Dzahn: releases: set the jenkins auto_restarts ensure to absent directly [puppet] - 10https://gerrit.wikimedia.org/r/1369392 (https://phabricator.wikimedia.org/T444758) [10:26:58] (03PS1) 10Dzahn: releases: remove an unused param and a commented block from common [puppet] - 10https://gerrit.wikimedia.org/r/1369393 (https://phabricator.wikimedia.org/T444758) [10:27:00] (03PS1) 10Dzahn: releases: drop the vhost-level IndexIgnore/HeaderName duplicates [puppet] - 10https://gerrit.wikimedia.org/r/1369394 (https://phabricator.wikimedia.org/T444758) [10:29:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12554273 (10phaultfinder) [10:33:12] (03PS1) 10Gmodena: wdqs: add discovery records for the wdqs2 internal releases [dns] - 10https://gerrit.wikimedia.org/r/1369395 (https://phabricator.wikimedia.org/T436375) [10:33:56] (03PS1) 10Gmodena: wdqs: deploy internal releases on topolvm-balanced [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369389 (https://phabricator.wikimedia.org/T436375) [10:34:45] (03PS2) 10Gmodena: dse-k8s: raise the wdqs namespace CPU quota [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369355 (https://phabricator.wikimedia.org/T436375) [10:34:45] (03CR) 10CI reject: [V:04-1] releases: remove params the releases class does not use [puppet] - 10https://gerrit.wikimedia.org/r/1369391 (https://phabricator.wikimedia.org/T444758) (owner: 10Dzahn) [10:36:27] !log javiermonton@deploy2003 Started scap deploy-service pageview-trending-relative: Deploying due to pending changes from SRE on Envoy [10:36:27] !log javiermonton@deploy2003 scap deploy-service pageview-trending-relative: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying pageview-trending-relative [10:37:04] !log javiermonton@deploy2003 Started scap deploy-service pageview-trending-relative-next: Deploying due to pending changes from SRE on Envoy [10:37:04] !log javiermonton@deploy2003 scap deploy-service pageview-trending-relative-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying pageview-trending-relative-next [10:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:42:34] !log javiermonton@deploy2003 Finished scap deploy-service pageview-trending-relative-next: Deploying due to pending changes from SRE on Envoy (duration: 05m 30s) [10:42:35] !log javiermonton@deploy2003 Finished scap deploy-service pageview-trending-relative: Deploying due to pending changes from SRE on Envoy (duration: 06m 08s) [10:43:07] !log hashar@deploy2003 Started deploy [integration/docroot@069e018]: build: Updating postcss-selector-parser to 6.1.6 [10:43:21] !log hashar@deploy2003 Finished deploy [integration/docroot@069e018]: build: Updating postcss-selector-parser to 6.1.6 (duration: 00m 14s) [10:45:05] !log javiermonton@deploy2003 Started scap deploy-service mw-page-html-feature-counts-change-enrich: Deploying due to pending changes from SRE on Envoy [10:45:06] !log javiermonton@deploy2003 scap deploy-service mw-page-html-feature-counts-change-enrich: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying mw-page-html-feature-counts-change-enrich [10:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [10:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [10:51:46] !log javiermonton@deploy2003 Finished scap deploy-service mw-page-html-feature-counts-change-enrich: Deploying due to pending changes from SRE on Envoy (duration: 06m 40s) [10:52:18] !log javiermonton@deploy2003 Started scap deploy-service mw-page-html-content-change-enrich: Deploying due to pending changes from SRE on Envoy [10:52:18] !log javiermonton@deploy2003 scap deploy-service mw-page-html-content-change-enrich: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying mw-page-html-content-change-enrich [10:52:58] !log javiermonton@deploy2003 Started scap deploy-service webrequest-pageview: Deploying due to pending changes from SRE on Envoy [10:52:58] !log javiermonton@deploy2003 scap deploy-service webrequest-pageview: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying webrequest-pageview [10:55:29] !log javiermonton@deploy2003 Finished scap deploy-service mw-page-html-content-change-enrich: Deploying due to pending changes from SRE on Envoy (duration: 03m 11s) [10:56:48] !log javiermonton@deploy2003 Finished scap deploy-service webrequest-pageview: Deploying due to pending changes from SRE on Envoy (duration: 03m 50s) [10:58:37] 06SRE, 06Infrastructure-Foundations, 10netops: Core router upgrades 2026 #2 - https://phabricator.wikimedia.org/T431748#12554584 (10ayounsi) 05Open→03Resolved a:03ayounsi [10:59:27] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply [10:59:44] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply [11:00:38] 06SRE, 06DC-Ops, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12554621 (10fnegri) [11:01:49] !log javiermonton@deploy2003 Started scap deploy-service mw-content-history-reconcile-enrich: Deploying due to pending changes from SRE on Envoy [11:01:49] !log javiermonton@deploy2003 scap deploy-service mw-content-history-reconcile-enrich: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying mw-content-history-reconcile-enrich [11:04:51] !log javiermonton@deploy2003 Finished scap deploy-service mw-content-history-reconcile-enrich: Deploying due to pending changes from SRE on Envoy (duration: 03m 02s) [11:06:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 29d 7h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [11:11:13] 06SRE, 10MediaWiki-General, 06ServiceOps, 07Epic: Introduce a global load shedding level - https://phabricator.wikimedia.org/T443236#12554722 (10Ladsgroup) After talking to @Joe in SRE summit. He suggested that we rename LSL to Quality of Service. So in longer-term that can be set on the edge during outage... [11:15:24] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:19:12] (03PS1) 10Majavah: team-wmcs: Exclude Heat from API latency metrics [alerts] - 10https://gerrit.wikimedia.org/r/1369400 [11:19:40] (03CR) 10Majavah: [C:03+2] "virtual +1 from Filippo" [alerts] - 10https://gerrit.wikimedia.org/r/1369400 (owner: 10Majavah) [11:20:13] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: apply [11:21:36] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: apply [11:22:19] (03Merged) 10jenkins-bot: team-wmcs: Exclude Heat from API latency metrics [alerts] - 10https://gerrit.wikimedia.org/r/1369400 (owner: 10Majavah) [11:41:00] FIRING: PingLossPercent: Blackbox probe packet loss 4.375% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [11:54:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [12:00:04] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [12:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261113T0800) [12:00:05] jelto, arnoldokoth, mutante, and arnaudb: May I have your attention please! GitLab version upgrades. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261113T1200) [12:00:14] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [12:00:19] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [12:00:29] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [12:00:32] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [12:00:39] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [12:04:29] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:04:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [12:11:31] (03CR) 10Filippo Giunchedi: [C:03+1] hieradata: Remove check_ssh from cloudcumin2001 [puppet] - 10https://gerrit.wikimedia.org/r/1369313 (owner: 10Majavah) [12:15:26] (03CR) 10Filippo Giunchedi: [C:03+1] "Looks good to me, should be fairly easy to test at deploy time when puppet exported resources have converged" [puppet] - 10https://gerrit.wikimedia.org/r/1369312 (owner: 10Majavah) [12:23:00] (03CR) 10Filippo Giunchedi: [C:03+1] P:openstack: Remove pointless inline_template() usage [puppet] - 10https://gerrit.wikimedia.org/r/1347510 (owner: 10Majavah) [12:23:44] (03CR) 10Filippo Giunchedi: [C:03+1] cloudnfs: Replace use of inline_template [puppet] - 10https://gerrit.wikimedia.org/r/1347511 (owner: 10Majavah) [12:25:39] (03PS1) 10Dzahn: planet: remove the bookworm-only xml settings from the feed configs [puppet] - 10https://gerrit.wikimedia.org/r/1369412 (https://phabricator.wikimedia.org/T444758) [12:30:03] (03PS1) 10Dzahn: doc: drop the pre-bookworm PHP branch, fix the class name in admin [puppet] - 10https://gerrit.wikimedia.org/r/1369413 (https://phabricator.wikimedia.org/T444758) [12:30:05] (03PS1) 10Dzahn: doc: remove the unused gitlab_runner_hosts hiera key [puppet] - 10https://gerrit.wikimedia.org/r/1369414 (https://phabricator.wikimedia.org/T444758) [12:30:07] (03PS1) 10Dzahn: doc: write the doc path directly, drop the unused fallback [puppet] - 10https://gerrit.wikimedia.org/r/1369415 (https://phabricator.wikimedia.org/T444758) [12:30:09] (03PS1) 10Dzahn: peopleweb: remove the unused deployment_server param [puppet] - 10https://gerrit.wikimedia.org/r/1369416 (https://phabricator.wikimedia.org/T444758) [12:30:11] (03PS1) 10Dzahn: peopleweb: drop the extra rsync package and double blank lines [puppet] - 10https://gerrit.wikimedia.org/r/1369417 (https://phabricator.wikimedia.org/T444758) [12:30:13] (03PS1) 10Dzahn: insetup: remove role::insetup::collaboration_services_ferm [puppet] - 10https://gerrit.wikimedia.org/r/1369418 (https://phabricator.wikimedia.org/T444758) [12:33:46] (03CR) 10Arnaudb: [C:03+1] planet: remove the bookworm-only xml settings from the feed configs [puppet] - 10https://gerrit.wikimedia.org/r/1369412 (https://phabricator.wikimedia.org/T444758) (owner: 10Dzahn) [12:39:22] (03CR) 10Muehlenhoff: [C:03+1] "Key has been verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1369014 (https://phabricator.wikimedia.org/T442758) (owner: 10Daniel Kinzler) [12:39:48] (03CR) 10Muehlenhoff: [C:03+2] admin: Add Yubikey for DKinzler [puppet] - 10https://gerrit.wikimedia.org/r/1369014 (https://phabricator.wikimedia.org/T442758) (owner: 10Daniel Kinzler) [12:41:44] !log jhancock@cumin2003 START - Cookbook sre.hosts.provision for host ganeti2051.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [12:42:02] !log jhancock@cumin2003 START - Cookbook sre.hosts.provision for host ganeti2052.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [12:42:02] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2051.codfw.wmnet (push config only) [12:42:07] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2051.codfw.wmnet (push config only) [12:42:14] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2052.codfw.wmnet (push config only) [12:42:18] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2052.codfw.wmnet (push config only) [12:42:23] !log jhancock@cumin2003 START - Cookbook sre.hosts.provision for host ganeti2053.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [12:42:35] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2053.codfw.wmnet (push config only) [12:42:39] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2053.codfw.wmnet (push config only) [12:43:58] (03PS2) 10Dzahn: gerrit: remove params the gerrit class takes but never uses [puppet] - 10https://gerrit.wikimedia.org/r/1369378 (https://phabricator.wikimedia.org/T444758) [12:43:58] (03PS2) 10Dzahn: gerrit: remove unused params from profile::gerrit::proxy [puppet] - 10https://gerrit.wikimedia.org/r/1369379 (https://phabricator.wikimedia.org/T444758) [12:43:58] (03PS2) 10Dzahn: gerrit: remove role::gerrit::migration and migration_base [puppet] - 10https://gerrit.wikimedia.org/r/1369380 (https://phabricator.wikimedia.org/T444758) [12:43:59] (03PS2) 10Dzahn: gerrit: remove host hiera that repeats the common and role values [puppet] - 10https://gerrit.wikimedia.org/r/1369381 (https://phabricator.wikimedia.org/T444758) [12:44:00] (03PS2) 10Dzahn: gerrit: take the replica/spare replication URLs from hiera only [puppet] - 10https://gerrit.wikimedia.org/r/1369382 (https://phabricator.wikimedia.org/T444758) [12:44:01] (03PS2) 10Dzahn: gerrit: remove hiera values that equal the defaults or are never read [puppet] - 10https://gerrit.wikimedia.org/r/1369383 (https://phabricator.wikimedia.org/T444758) [12:44:05] (03PS2) 10Dzahn: gerrit: drop the duplicate include of profile::java from the role [puppet] - 10https://gerrit.wikimedia.org/r/1369384 (https://phabricator.wikimedia.org/T444758) [12:44:09] (03PS2) 10Dzahn: gerrit: drop "!= undef" checks on params that cannot be undef [puppet] - 10https://gerrit.wikimedia.org/r/1369385 (https://phabricator.wikimedia.org/T444758) [12:44:13] (03PS2) 10Dzahn: gerrit: always use gerrit.config.erb, drop the $config param [puppet] - 10https://gerrit.wikimedia.org/r/1369386 (https://phabricator.wikimedia.org/T444758) [12:44:17] (03PS2) 10Dzahn: gerrit: use bool2str for the motd script, fix the file name header [puppet] - 10https://gerrit.wikimedia.org/r/1369387 (https://phabricator.wikimedia.org/T444758) [12:44:21] (03PS2) 10Dzahn: gerrit: remove three unread gerrit keys from devtools hiera [puppet] - 10https://gerrit.wikimedia.org/r/1369388 (https://phabricator.wikimedia.org/T444758) [12:44:25] (03PS1) 10Dzahn: gerrit: remove the devtools gerrit hiera and the certbot branch [puppet] - 10https://gerrit.wikimedia.org/r/1369421 (https://phabricator.wikimedia.org/T444758) [12:44:29] (03PS1) 10Dzahn: gerrit: remove GWT-era site files nothing reads any more [puppet] - 10https://gerrit.wikimedia.org/r/1369422 (https://phabricator.wikimedia.org/T444758) [12:44:33] (03PS1) 10Clément Goubert: Lift IP cap for WikiIndaba 2026 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369423 (https://phabricator.wikimedia.org/T444794) [12:48:16] (03CR) 10Ssingh: Lift IP cap for WikiIndaba 2026 (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369423 (https://phabricator.wikimedia.org/T444794) (owner: 10Clément Goubert) [12:49:03] (03PS2) 10Clément Goubert: Lift IP cap for WikiIndaba 2026 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369423 (https://phabricator.wikimedia.org/T444794) [12:49:11] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2051.codfw.wmnet (remove config only) [12:49:16] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2051.codfw.wmnet (remove config only) [12:49:22] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2052.codfw.wmnet (remove config only) [12:49:27] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2052.codfw.wmnet (remove config only) [12:49:30] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2053.codfw.wmnet (remove config only) [12:49:34] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2053.codfw.wmnet (remove config only) [12:49:35] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2051.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [12:49:40] (03CR) 10Ssingh: [C:03+1] Lift IP cap for WikiIndaba 2026 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369423 (https://phabricator.wikimedia.org/T444794) (owner: 10Clément Goubert) [12:49:47] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2052.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [12:49:54] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2053.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [12:49:58] (03CR) 10Clément Goubert: Lift IP cap for WikiIndaba 2026 (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369423 (https://phabricator.wikimedia.org/T444794) (owner: 10Clément Goubert) [12:50:45] FIRING: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [12:52:54] !log jhancock@cumin2003 START - Cookbook sre.hosts.reimage for host ganeti2051.codfw.wmnet with OS bookworm [12:53:05] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2051.codfw.wmnet (push config only) [12:53:09] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2051.codfw.wmnet (push config only) [12:53:11] !log jhancock@cumin2003 START - Cookbook sre.hosts.reimage for host ganeti2052.codfw.wmnet with OS bookworm [12:53:21] !log jhancock@cumin2003 START - Cookbook sre.hosts.reimage for host ganeti2053.codfw.wmnet with OS bookworm [12:53:22] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2052.codfw.wmnet (push config only) [12:53:26] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2052.codfw.wmnet (push config only) [12:53:31] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2053.codfw.wmnet (push config only) [12:53:35] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2053.codfw.wmnet (push config only) [12:54:44] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cgoubert@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369423 (https://phabricator.wikimedia.org/T444794) (owner: 10Clément Goubert) [12:56:43] (03Merged) 10jenkins-bot: Lift IP cap for WikiIndaba 2026 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369423 (https://phabricator.wikimedia.org/T444794) (owner: 10Clément Goubert) [12:57:20] !log cgoubert@deploy2003 Started scap sync-world: Backport for [[gerrit:1369423|Lift IP cap for WikiIndaba 2026 (T444794)]] [12:57:23] T444794: IP throttle exemption for WikiIndaba 2026 - https://phabricator.wikimedia.org/T444794 [12:59:51] !log cgoubert@deploy2003 cgoubert: Backport for [[gerrit:1369423|Lift IP cap for WikiIndaba 2026 (T444794)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:00:27] !log cgoubert@deploy2003 cgoubert: Continuing with deployment [13:00:32] PROBLEM - OSPF status on cr2-eqiad is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [13:00:44] PROBLEM - OSPF status on cr2-codfw is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [13:01:10] FIRING: BFDdown: BFD session down between cr2-codfw and 185.15.58.159 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:01:39] FIRING: CoreBGPDown: Core BGP session down between cr2-codfw and cr2-drmrs (185.15.58.159) - group Confed_drmrs - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr2-codfw:9804&var-bgp_group=Confed_drmrs&var-bgp_neighbor=cr2-drmrs - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:02:33] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2051.codfw.wmnet (remove config only) [13:02:37] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2052.codfw.wmnet (remove config only) [13:02:38] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2051.codfw.wmnet (remove config only) [13:02:41] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-drmrs:et-0/0/0 (Transport: Arelion (IC-398708) [20Gbps]) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-drmrs:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [13:02:44] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2052.codfw.wmnet (remove config only) [13:02:51] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2053.codfw.wmnet (remove config only) [13:02:56] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2053.codfw.wmnet (remove config only) [13:04:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr1-drmrs and Arelion (2001:2035:0:cf1::1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [13:05:11] !log jhancock@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2051.codfw.wmnet with reason: host reimage [13:05:12] !log jhancock@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2052.codfw.wmnet with reason: host reimage [13:05:30] !log jhancock@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2053.codfw.wmnet with reason: host reimage [13:06:10] FIRING: [2x] BFDdown: BFD session down between cr2-codfw and 185.15.58.159 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:06:39] FIRING: [3x] CoreBGPDown: Core BGP session down between cr2-codfw and cr2-drmrs (185.15.58.159) - group Confed_drmrs - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:06:48] !log cgoubert@deploy2003 Finished scap sync-world: Backport for [[gerrit:1369423|Lift IP cap for WikiIndaba 2026 (T444794)]] (duration: 09m 28s) [13:06:52] T444794: IP throttle exemption for WikiIndaba 2026 - https://phabricator.wikimedia.org/T444794 [13:09:13] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2051.codfw.wmnet with reason: host reimage [13:10:44] (03PS1) 10Dpogorzelski: liftwing-studio: bump librechat to node 24.22.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369426 [13:11:10] FIRING: [3x] BFDdown: BFD session down between cr1-codfw and fe80::669:8f07:ece:f4e7 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:11:11] (03CR) 10Dpogorzelski: [C:03+2] liftwing-studio: bump librechat to node 24.22.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369426 (owner: 10Dpogorzelski) [13:11:13] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] liftwing-studio: bump librechat to node 24.22.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369426 (owner: 10Dpogorzelski) [13:12:17] !log dpogorzelski@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [13:12:50] PROBLEM - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1194 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 1 Failed : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [13:12:51] ACKNOWLEDGEMENT - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1194 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 1 Failed : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T444799 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [13:13:15] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2052.codfw.wmnet with reason: host reimage [13:13:15] !log dpogorzelski@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [13:13:32] (03PS1) 10Cwhite: hiera: clean up decomissioned logstash host-specific configs [puppet] - 10https://gerrit.wikimedia.org/r/1369427 [13:16:03] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2053.codfw.wmnet with reason: host reimage [13:26:20] !log jhancock@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003" [13:29:24] jhancock@cumin2003 reimage (PID 1656304) is awaiting input [13:31:33] !log jhancock@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003" [13:32:26] !log jhancock@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003" [13:32:27] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2051.codfw.wmnet with OS bookworm [13:32:29] !log jhancock@cumin2003 END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003" [13:32:31] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2052.codfw.wmnet with OS bookworm [13:33:30] !log jhancock@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003" [13:34:35] !log jhancock@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003" [13:34:37] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2053.codfw.wmnet with OS bookworm [13:36:33] !log jhancock@cumin2003 START - Cookbook sre.hosts.provision for host ganeti2054.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [13:36:47] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2054.codfw.wmnet (push config only) [13:36:51] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2054.codfw.wmnet (push config only) [13:36:56] !log jhancock@cumin2003 START - Cookbook sre.hosts.provision for host ganeti2055.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [13:37:21] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2055.codfw.wmnet (push config only) [13:37:25] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2055.codfw.wmnet (push config only) [13:37:25] !log jhancock@cumin2003 START - Cookbook sre.hosts.provision for host ganeti2056.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [13:37:40] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2056.codfw.wmnet (push config only) [13:37:44] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2056.codfw.wmnet (push config only) [13:41:02] (03PS1) 10Bking: Add FIDO SSH key for amastilovic [puppet] - 10https://gerrit.wikimedia.org/r/1369432 (https://phabricator.wikimedia.org/T442758) [13:41:16] (03PS1) 10Tryvix1509: dewiki: Add als, bar and nds to import sources [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369433 (https://phabricator.wikimedia.org/T444759) [13:42:42] (03PS2) 10Daniel Kinzler: admin: Remove old SSH key for DKinzler [puppet] - 10https://gerrit.wikimedia.org/r/1369015 (https://phabricator.wikimedia.org/T442758) [13:43:42] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2054.codfw.wmnet (remove config only) [13:43:47] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2054.codfw.wmnet (remove config only) [13:43:59] (03PS1) 10Brouberol: test-kitchen: upgade the application image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369434 [13:44:01] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2055.codfw.wmnet (remove config only) [13:44:06] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2054.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [13:44:07] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2055.codfw.wmnet (remove config only) [13:44:20] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2056.codfw.wmnet (remove config only) [13:44:25] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2056.codfw.wmnet (remove config only) [13:44:25] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2055.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [13:44:36] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2056.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [13:44:39] (03PS2) 10Brouberol: test-kitchen: upgrade the application image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369434 [13:45:44] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install ganeti205[1-6] - https://phabricator.wikimedia.org/T434698#12555332 (10Jhancock.wm) [13:45:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [13:46:15] (03CR) 10Santiago Faci: [C:03+1] test-kitchen: upgrade the application image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369434 (owner: 10Brouberol) [13:47:08] (03PS1) 10Brouberol: growthbook: upgrade the application image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369435 [13:49:23] (03CR) 10Brouberol: [C:03+1] "Public key checks out with https://phabricator.wikimedia.org/T442758#12550843" [puppet] - 10https://gerrit.wikimedia.org/r/1369432 (https://phabricator.wikimedia.org/T442758) (owner: 10Bking) [13:50:11] (03CR) 10Santiago Faci: [C:03+1] growthbook: upgrade the application image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369435 (owner: 10Brouberol) [13:50:52] (03CR) 10Brouberol: [C:03+2] test-kitchen: upgrade the application image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369434 (owner: 10Brouberol) [13:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.63% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [13:52:34] (03CR) 10Brouberol: [C:03+2] growthbook: upgrade the application image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369435 (owner: 10Brouberol) [13:52:40] (03CR) 10Bking: [C:04-1] "Do not merge until the key is verified out of band." [puppet] - 10https://gerrit.wikimedia.org/r/1369432 (https://phabricator.wikimedia.org/T442758) (owner: 10Bking) [13:53:47] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply [13:54:47] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply [13:55:25] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply [13:56:31] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply [13:57:15] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply [13:57:41] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply [13:57:54] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply [13:58:22] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply [13:59:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12555393 (10phaultfinder) [14:05:25] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:11:40] (03PS1) 10Aqu: spark35: Turn off Iceberg vectorized reads [puppet] - 10https://gerrit.wikimedia.org/r/1369440 (https://phabricator.wikimedia.org/T441704) [14:19:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12555475 (10phaultfinder) [14:20:37] (03CR) 10Dzahn: [C:03+2] admin: Add deployment-jenkins to the release-engineering description [puppet] - 10https://gerrit.wikimedia.org/r/1369128 (owner: 10Ahmon Dancy) [14:31:01] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.333% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [14:38:43] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for FSesini-WMF - https://phabricator.wikimedia.org/T444520#12555518 (10Dzahn) a:03thcipriani [14:38:49] (03PS1) 10Cwhite: hiera: removed logstash::java_package setting [puppet] - 10https://gerrit.wikimedia.org/r/1369446 [14:39:37] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for mazevedo - https://phabricator.wikimedia.org/T444497#12555520 (10Dzahn) a:03thcipriani [14:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:41:22] (03CR) 10CI reject: [V:04-1] hiera: removed logstash::java_package setting [puppet] - 10https://gerrit.wikimedia.org/r/1369446 (owner: 10Cwhite) [14:42:04] (03PS1) 10Jelto: admin: Add FIDO SSH key for gkyziridis [puppet] - 10https://gerrit.wikimedia.org/r/1369449 (https://phabricator.wikimedia.org/T442758) [14:43:35] (03PS1) 10Brouberol: eventstreams: upgrade from 0.22 to 0.23 to run on the newest node image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369450 [14:44:34] (03CR) 10Dzahn: [C:04-1] "deleting GPL is wrong" [puppet] - 10https://gerrit.wikimedia.org/r/1369341 (https://phabricator.wikimedia.org/T444758) (owner: 10Arnaudb) [14:44:40] (03PS1) 10Jelto: admin: Add FIDO SSH key for karen [puppet] - 10https://gerrit.wikimedia.org/r/1369451 (https://phabricator.wikimedia.org/T442762) [14:46:09] (03PS1) 10Brouberol: eventgate: upgrade from 1.38 to 1.39 to run on the newest node image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369454 [14:46:45] (03CR) 10Dzahn: "does not seem like simplification or reducing complexity - more like adding it to improve security" [puppet] - 10https://gerrit.wikimedia.org/r/1369323 (https://phabricator.wikimedia.org/T444758) (owner: 10Arnaudb) [14:46:47] (03PS2) 10Jelto: admin: Add FIDO SSH key for karen [puppet] - 10https://gerrit.wikimedia.org/r/1369451 (https://phabricator.wikimedia.org/T442762) [14:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [14:48:41] (03CR) 10Brouberol: [C:03+2] eventstreams: upgrade from 0.22 to 0.23 to run on the newest node image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369450 (owner: 10Brouberol) [14:48:56] (03CR) 10Brouberol: [C:03+2] eventgate: upgrade from 1.38 to 1.39 to run on the newest node image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369454 (owner: 10Brouberol) [14:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [14:49:14] (03CR) 10Jelto: [C:03+2] admin: Add FIDO SSH key for karen [puppet] - 10https://gerrit.wikimedia.org/r/1369451 (https://phabricator.wikimedia.org/T442762) (owner: 10Jelto) [14:51:17] 10SRE-SLO, 06Collaboration-Services, 10Gerrit: Add a SSH probe to test a connection to Gerrit - https://phabricator.wikimedia.org/T444800#12555580 (10ABran-WMF) [14:51:27] 10SRE-SLO, 06Collaboration-Services, 10Gerrit: Add a SSH probe to test a connection to Gerrit - https://phabricator.wikimedia.org/T444800#12555583 (10ABran-WMF) p:05Triage→03Medium [14:51:37] !log brouberol@deploy2003 helmfile [staging] START helmfile.d/services/eventstreams: apply [14:52:24] (03CR) 10Dzahn: [C:03+2] collab services: fix typos in comments and docs [puppet] - 10https://gerrit.wikimedia.org/r/1369037 (https://phabricator.wikimedia.org/T444758) (owner: 10Dzahn) [14:53:36] (03PS1) 10Brouberol: eventstreams-internal: upgrade from 0.22 to 0.23 to run on the newest node image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369456 [14:54:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12555597 (10phaultfinder) [14:55:28] (03CR) 10Dzahn: [C:03+2] jenkins: Back up the Jenkins controllers [puppet] - 10https://gerrit.wikimedia.org/r/1369319 (https://phabricator.wikimedia.org/T444758) (owner: 10Arnaudb) [14:56:24] (03CR) 10Arnaudb: "thanks for the merge!" [puppet] - 10https://gerrit.wikimedia.org/r/1369319 (https://phabricator.wikimedia.org/T444758) (owner: 10Arnaudb) [14:56:45] (03PS1) 10Jelto: admin: Add FIDO SSH key for ifrahkh [puppet] - 10https://gerrit.wikimedia.org/r/1369457 (https://phabricator.wikimedia.org/T442763) [14:57:27] (03CR) 10Brouberol: [C:03+2] eventstreams-internal: upgrade from 0.22 to 0.23 to run on the newest node image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369456 (owner: 10Brouberol) [14:58:34] !log brouberol@deploy2003 helmfile [staging] START helmfile.d/services/eventgate-analytics: apply [14:58:59] !log brouberol@deploy2003 helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply [14:59:55] (03CR) 10Jelto: [C:03+2] admin: Add FIDO SSH key for ifrahkh [puppet] - 10https://gerrit.wikimedia.org/r/1369457 (https://phabricator.wikimedia.org/T442763) (owner: 10Jelto) [15:00:37] !log brouberol@deploy2003 helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply [15:01:24] !log brouberol@deploy2003 helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply [15:02:35] !log brouberol@deploy2003 helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply [15:03:19] !log brouberol@deploy2003 helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply [15:03:26] !log brouberol@deploy2003 helmfile [staging] START helmfile.d/services/eventgate-analytics-external: apply [15:03:38] !log brouberol@deploy2003 helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: apply [15:03:55] !log brouberol@deploy2003 helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply [15:04:37] !log brouberol@deploy2003 helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply [15:04:43] !log brouberol@deploy2003 helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply [15:05:10] !log brouberol@deploy2003 helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply [15:05:30] !log brouberol@deploy2003 helmfile [staging] START helmfile.d/services/eventgate-logging-external: apply [15:05:54] (03CR) 10Dzahn: [C:03+1] microsites: Delete the os-reports vhost and the static RT index [puppet] - 10https://gerrit.wikimedia.org/r/1369344 (https://phabricator.wikimedia.org/T444758) (owner: 10Arnaudb) [15:05:55] !log brouberol@deploy2003 helmfile [staging] DONE helmfile.d/services/eventgate-logging-external: apply [15:06:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 29d 3h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [15:06:06] (03CR) 10Jelto: [C:03+2] admin: Add FIDO SSH key for jgiannelos [puppet] - 10https://gerrit.wikimedia.org/r/1369353 (https://phabricator.wikimedia.org/T442758) (owner: 10Jelto) [15:06:07] (03PS3) 10Arnaudb: planet: Delete theme files and templates that nothing uses [puppet] - 10https://gerrit.wikimedia.org/r/1369341 (https://phabricator.wikimedia.org/T444758) [15:06:19] !log brouberol@deploy2003 helmfile [eqiad] START helmfile.d/services/eventgate-logging-external: apply [15:06:36] (03CR) 10Arnaudb: "fixed!" [puppet] - 10https://gerrit.wikimedia.org/r/1369341 (https://phabricator.wikimedia.org/T444758) (owner: 10Arnaudb) [15:06:45] !log brouberol@deploy2003 helmfile [eqiad] DONE helmfile.d/services/eventgate-logging-external: apply [15:06:53] !log brouberol@deploy2003 helmfile [codfw] START helmfile.d/services/eventgate-logging-external: apply [15:07:19] !log brouberol@deploy2003 helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: apply [15:07:27] !log brouberol@deploy2003 helmfile [staging] START helmfile.d/services/eventgate-main: apply [15:07:30] (03CR) 10Dzahn: [C:03+2] microsites: Delete the os-reports vhost and the static RT index [puppet] - 10https://gerrit.wikimedia.org/r/1369344 (https://phabricator.wikimedia.org/T444758) (owner: 10Arnaudb) [15:07:38] !log brouberol@deploy2003 helmfile [staging] DONE helmfile.d/services/eventgate-main: apply [15:07:59] !log brouberol@deploy2003 helmfile [eqiad] START helmfile.d/services/eventgate-main: apply [15:08:13] !log brouberol@deploy2003 helmfile [eqiad] DONE helmfile.d/services/eventgate-main: apply [15:08:22] !log brouberol@deploy2003 helmfile [codfw] START helmfile.d/services/eventgate-main: apply [15:08:44] !log brouberol@deploy2003 helmfile [codfw] DONE helmfile.d/services/eventgate-main: apply [15:08:45] (03CR) 10Jelto: [C:03+2] admin: Add FIDO SSH key for stran [puppet] - 10https://gerrit.wikimedia.org/r/1369357 (https://phabricator.wikimedia.org/T442758) (owner: 10Jelto) [15:09:32] (03PS1) 10Brouberol: eventstreams: upgrade from 0.23 to 0.24 to solve the dependency runtime issue [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369460 [15:09:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12555670 (10phaultfinder) [15:12:48] (03PS2) 10Snwachukwu: Rewrite edit per editor v3 gateway requests to edit per editor v4 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368714 (https://phabricator.wikimedia.org/T434136) [15:14:08] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply [15:14:23] (03CR) 10Jelto: [C:03+2] admin: Add FIDO SSH key for mmartorana [puppet] - 10https://gerrit.wikimedia.org/r/1369377 (https://phabricator.wikimedia.org/T442758) (owner: 10Jelto) [15:14:33] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply [15:15:02] PROBLEM - Host ps1-e2-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:15:02] PROBLEM - Host ps1-e7-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:15:02] PROBLEM - Host ps1-f2-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:15:02] PROBLEM - Host ps1-f7-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:15:02] PROBLEM - Host ps1-e6-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:15:03] PROBLEM - Host ps1-f6-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:15:03] PROBLEM - Host ps1-e4-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:15:04] PROBLEM - Host ps1-f8-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:15:04] PROBLEM - Host ps1-e3-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:15:05] PROBLEM - Host ps1-f3-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:15:05] PROBLEM - Host ps1-e5-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:15:06] PROBLEM - Host ps1-e9-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:15:06] PROBLEM - Host ps1-f4-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:15:07] PROBLEM - Host ps1-e8-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:15:24] FIRING: [11x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:15:28] PROBLEM - Host mr1-eqiad IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [15:15:32] PROBLEM - Host asw2-a-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:15:36] PROBLEM - Host asw2-b-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:15:56] PROBLEM - Host ps1-a1-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:15:56] PROBLEM - Host ps1-a2-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:16:02] PROBLEM - Host ps1-e10-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:16:02] PROBLEM - Host ps1-e1-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:16:02] PROBLEM - Host ps1-d6-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:16:02] PROBLEM - Host ps1-e15-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:16:02] PROBLEM - Host ps1-b6-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:16:03] PROBLEM - Host ps1-c1-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:16:03] PROBLEM - Host ps1-d3-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:16:04] PROBLEM - Host ps1-d1-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:16:04] PROBLEM - Host ps1-c6-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:16:05] PROBLEM - Host ps1-e12-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:16:05] PROBLEM - Host ps1-b3-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:16:06] PROBLEM - Host ps1-d2-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:16:06] PROBLEM - Host ps1-d4-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [15:16:13] (03CR) 10Brouberol: [C:03+2] eventstreams: upgrade from 0.23 to 0.24 to solve the dependency runtime issue [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369460 (owner: 10Brouberol) [15:16:31] (03CR) 10Dzahn: [C:04-1] "still needed for hosts using docker" [puppet] - 10https://gerrit.wikimedia.org/r/1369418 (https://phabricator.wikimedia.org/T444758) (owner: 10Dzahn) [15:16:44] (03PS1) 10Aqu: analytics: Deploy conda-analytics-next 0.0.50 [puppet] - 10https://gerrit.wikimedia.org/r/1369461 (https://phabricator.wikimedia.org/T441704) [15:16:54] RECOVERY - Host ps1-a6-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.13 ms [15:16:54] RECOVERY - Host ps1-c4-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.69 ms [15:16:54] RECOVERY - Host ps1-a1-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.66 ms [15:16:54] RECOVERY - Host ps1-e10-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.45 ms [15:16:54] RECOVERY - Host ps1-d8-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.05 ms [15:17:01] (03CR) 10Jelto: [C:03+2] admin: Add FIDO SSH key for gkyziridis [puppet] - 10https://gerrit.wikimedia.org/r/1369449 (https://phabricator.wikimedia.org/T442758) (owner: 10Jelto) [15:17:35] (03CR) 10CI reject: [V:04-1] analytics: Deploy conda-analytics-next 0.0.50 [puppet] - 10https://gerrit.wikimedia.org/r/1369461 (https://phabricator.wikimedia.org/T441704) (owner: 10Aqu) [15:18:49] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to releasers-mediawiki for osleger - https://phabricator.wikimedia.org/T443655#12555722 (10Dzahn) This is waiting for @MSantos for approval on `releasers-mediawiki` group. [15:19:10] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to releasers-mediawiki for osleger - https://phabricator.wikimedia.org/T443655#12555723 (10Dzahn) a:03MSantos [15:19:39] !log brouberol@deploy2003 helmfile [staging] START helmfile.d/services/eventstreams: apply [15:20:30] RECOVERY - Host mr1-eqiad IPv6 is UP: PING OK - Packet loss = 0%, RTA = 1.08 ms [15:20:51] !log brouberol@deploy2003 helmfile [staging] START helmfile.d/services/eventstreams: apply [15:21:11] !log brouberol@deploy2003 helmfile [staging] DONE helmfile.d/services/eventstreams: apply [15:21:26] !log brouberol@deploy2003 helmfile [eqiad] START helmfile.d/services/eventstreams: apply [15:21:56] !log brouberol@deploy2003 helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply [15:22:17] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for kartik - https://phabricator.wikimedia.org/T443182#12555734 (10Dzahn) a:03KartikMistry [15:22:20] !log brouberol@deploy2003 helmfile [codfw] START helmfile.d/services/eventstreams: apply [15:22:47] !log brouberol@deploy2003 helmfile [codfw] DONE helmfile.d/services/eventstreams: apply [15:22:52] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for aranyap - https://phabricator.wikimedia.org/T443041#12555746 (10Dzahn) a:03aranyap [15:23:50] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for xcollazo - https://phabricator.wikimedia.org/T442741#12555752 (10Dzahn) a:03xcollazo [15:24:45] (03PS1) 10Trueg: WDQSv2: Raise wdqs-streaming-consumer memory limit to 4Gi [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369465 (https://phabricator.wikimedia.org/T444632) [15:25:08] 06SRE, 10SRE-Access-Requests, 06Wikimedia Enterprise: Request for additional rights for SanchezA-WMF to manage analytics - https://phabricator.wikimedia.org/T443636#12555756 (10Dzahn) I don't think it worked as intended since this ticket is still missing the template info. Let me edit it to add it now. [15:25:23] 06SRE, 10SRE-Access-Requests, 06Wikimedia Enterprise: Request for additional rights for SanchezA-WMF to manage analytics - https://phabricator.wikimedia.org/T443636#12555757 (10Dzahn) [15:25:29] 06SRE, 10SRE-Access-Requests, 06SRE Observability, 13Patch-For-Review: Add FIDO-backed SSH keys for gkyziridis - https://phabricator.wikimedia.org/T444318#12555758 (10Jelto) [15:26:19] (03Abandoned) 10Gkyziridis: admin: Add FIDO SSH key for gkyziridis [puppet] - 10https://gerrit.wikimedia.org/r/1368303 (https://phabricator.wikimedia.org/T442758) (owner: 10Gkyziridis) [15:26:23] (03CR) 10Brouberol: [C:03+1] "LGTM, although it's the first time I'm reviewing anything AQS-related. I suggest @eevans@wikimedia.org has a look as well, as a precaution" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368714 (https://phabricator.wikimedia.org/T434136) (owner: 10Snwachukwu) [15:27:08] 06SRE, 10SRE-Access-Requests, 06SRE Observability, 13Patch-For-Review: Add FIDO-backed SSH keys for gkyziridis - https://phabricator.wikimedia.org/T444318#12555767 (10gkyziridis) I am closing this task since there was a patch already merged: [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1369449 |... [15:27:14] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for cooltey - https://phabricator.wikimedia.org/T444236#12555775 (10Jelto) [15:27:28] 06SRE, 10SRE-Access-Requests, 06SRE Observability, 13Patch-For-Review: Add FIDO-backed SSH keys for gkyziridis - https://phabricator.wikimedia.org/T444318#12555783 (10gkyziridis) 05Open→03Declined [15:27:30] (03CR) 10Gmodena: [C:03+1] WDQSv2: Raise wdqs-streaming-consumer memory limit to 4Gi [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369465 (https://phabricator.wikimedia.org/T444632) (owner: 10Trueg) [15:29:24] (03PS1) 10Aqu: spark35: Turn Iceberg vectorized reads back on [puppet] - 10https://gerrit.wikimedia.org/r/1369466 (https://phabricator.wikimedia.org/T441704) [15:29:29] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for xcollazo - https://phabricator.wikimedia.org/T442741#12555790 (10Jelto) [15:29:47] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH key for sbassett - https://phabricator.wikimedia.org/T443508#12555795 (10Jelto) [15:30:10] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for kartik - https://phabricator.wikimedia.org/T443182#12555798 (10Jelto) [15:30:40] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for aranyap - https://phabricator.wikimedia.org/T443041#12555800 (10Jelto) [15:32:43] (03CR) 10Trueg: [C:03+2] WDQSv2: Raise wdqs-streaming-consumer memory limit to 4Gi [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369465 (https://phabricator.wikimedia.org/T444632) (owner: 10Trueg) [15:34:34] (03Merged) 10jenkins-bot: WDQSv2: Raise wdqs-streaming-consumer memory limit to 4Gi [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369465 (https://phabricator.wikimedia.org/T444632) (owner: 10Trueg) [15:36:02] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [15:36:10] (03CR) 10Snwachukwu: [C:03+2] Rewrite edit per editor v3 gateway requests to edit per editor v4 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368714 (https://phabricator.wikimedia.org/T434136) (owner: 10Snwachukwu) [15:37:21] (03CR) 10Snwachukwu: [V:03+2 C:03+2] Rewrite edit per editor v3 gateway requests to edit per editor v4 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368714 (https://phabricator.wikimedia.org/T434136) (owner: 10Snwachukwu) [15:39:24] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [15:40:29] !log ebysans@deploy2003 helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [15:40:48] !log ebysans@deploy2003 helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [15:41:33] !log ebysans@deploy2003 helmfile [codfw] START helmfile.d/services/rest-gateway: apply [15:41:50] !log ebysans@deploy2003 helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [15:41:56] (03PS1) 10Cwhite: logstash: clean up unused logstash defines [puppet] - 10https://gerrit.wikimedia.org/r/1369468 [15:41:56] (03PS1) 10Cwhite: cleanup: remove unused logstash hiera globals [puppet] - 10https://gerrit.wikimedia.org/r/1369469 [15:41:56] (03PS1) 10Cwhite: varnish,mjolnir: removed unused logstash options [puppet] - 10https://gerrit.wikimedia.org/r/1369470 [15:41:57] (03PS1) 10Cwhite: logstash: remove unused drop_ecs filter [puppet] - 10https://gerrit.wikimedia.org/r/1369471 [15:41:58] (03PS1) 10Cwhite: logstash: remove unused apifeatureusage filter [puppet] - 10https://gerrit.wikimedia.org/r/1369472 [15:42:54] !log trueg@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [15:45:10] (03PS1) 10Jelto: admin: Remove non-FIDO SSH key for stran and jgiannelos [puppet] - 10https://gerrit.wikimedia.org/r/1369474 (https://phabricator.wikimedia.org/T442758) [15:46:12] !log trueg@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [15:47:05] (03CR) 10Jelto: [C:03+2] admin: Remove non-FIDO SSH key for stran and jgiannelos [puppet] - 10https://gerrit.wikimedia.org/r/1369474 (https://phabricator.wikimedia.org/T442758) (owner: 10Jelto) [16:04:29] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:08:38] (03CR) 10Daniel Kinzler: [C:03+1] "I tried the fido key, it works. The old one can be removed." [puppet] - 10https://gerrit.wikimedia.org/r/1369015 (https://phabricator.wikimedia.org/T442758) (owner: 10Daniel Kinzler) [16:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:22:13] (03PS1) 10Greg Grossmeier: admin: Add FIDO keys for gjg [puppet] - 10https://gerrit.wikimedia.org/r/1369485 (https://phabricator.wikimedia.org/T442758) [16:26:12] (03CR) 10Lerickson: "Thanks! I assume --persist-updates and --enable-metrics don't need to be specified anymore; can you explain why in the MR description?" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369337 (https://phabricator.wikimedia.org/T444329) (owner: 10Trueg) [16:28:07] (03PS2) 10Trueg: WDQSv2: Update wdqs-qlever to 0.10.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369337 (https://phabricator.wikimedia.org/T444329) [16:28:46] (03CR) 10Clare Ming: [C:03+1] Add growthbook kubernetes service to spiderpig [puppet] - 10https://gerrit.wikimedia.org/r/1368697 (https://phabricator.wikimedia.org/T444502) (owner: 10Santiago Faci) [16:33:08] (03CR) 10Lerickson: [C:03+1] WDQSv2: Update wdqs-qlever to 0.10.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369337 (https://phabricator.wikimedia.org/T444329) (owner: 10Trueg) [16:37:08] (03CR) 10Krinkle: varnish: Don't strip browser Cache-Control:maxage from action=raw (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1364773 (https://phabricator.wikimedia.org/T279120) (owner: 10Krinkle) [16:45:27] (03PS1) 10DCausse: Reapply "search: add pool counter rules and dedicated pool for WDQS" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369489 (https://phabricator.wikimedia.org/T442824) [16:46:39] (03CR) 10CI reject: [V:04-1] Reapply "search: add pool counter rules and dedicated pool for WDQS" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369489 (https://phabricator.wikimedia.org/T442824) (owner: 10DCausse) [16:50:30] (03PS2) 10DCausse: Reapply "search: add pool counter rules and dedicated pool for WDQS" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369489 (https://phabricator.wikimedia.org/T442824) [17:04:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr1-drmrs and Arelion (2001:2035:0:cf1::1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [17:05:25] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-drmrs:et-0/0/0 (Transport: Arelion (IC-398708) [20Gbps]) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-drmrs:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [17:06:39] FIRING: [3x] CoreBGPDown: Core BGP session down between cr2-codfw and cr2-drmrs (185.15.58.159) - group Confed_drmrs - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [17:11:10] FIRING: [2x] BFDdown: BFD session down between cr2-codfw and 185.15.58.159 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [17:14:36] (03CR) 10Santiago Faci: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1368697 (https://phabricator.wikimedia.org/T444502) (owner: 10Santiago Faci) [17:21:17] (03CR) 10Bking: [C:03+2] "Verified via Slack. Merging..." [puppet] - 10https://gerrit.wikimedia.org/r/1369432 (https://phabricator.wikimedia.org/T442758) (owner: 10Bking) [17:39:38] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369494 [17:52:07] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.63% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [17:54:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12556372 (10phaultfinder) [18:02:24] !log jhancock@cumin2003 START - Cookbook sre.hosts.reimage for host ganeti2054.codfw.wmnet with OS bookworm [18:02:31] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install ganeti205[1-6] - https://phabricator.wikimedia.org/T434698#12556396 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2003 for host ganeti2054.codfw.wmnet with OS bookworm [18:02:35] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2054.codfw.wmnet (push config only) [18:02:39] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2054.codfw.wmnet (push config only) [18:03:02] !log jhancock@cumin2003 START - Cookbook sre.hosts.reimage for host ganeti2055.codfw.wmnet with OS bookworm [18:03:10] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install ganeti205[1-6] - https://phabricator.wikimedia.org/T434698#12556397 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2003 for host ganeti2055.codfw.wmnet with OS bookworm [18:03:12] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2055.codfw.wmnet (push config only) [18:03:16] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2055.codfw.wmnet (push config only) [18:03:36] !log jhancock@cumin2003 START - Cookbook sre.hosts.reimage for host ganeti2056.codfw.wmnet with OS bookworm [18:03:42] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install ganeti205[1-6] - https://phabricator.wikimedia.org/T434698#12556398 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2003 for host ganeti2056.codfw.wmnet with OS bookworm [18:03:46] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2056.codfw.wmnet (push config only) [18:03:50] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2056.codfw.wmnet (push config only) [18:05:25] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:11:53] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2054.codfw.wmnet (remove config only) [18:11:58] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2054.codfw.wmnet (remove config only) [18:12:50] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2055.codfw.wmnet (remove config only) [18:12:55] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2055.codfw.wmnet (remove config only) [18:13:04] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host ganeti2056.codfw.wmnet (remove config only) [18:13:09] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2056.codfw.wmnet (remove config only) [18:14:10] !log jhancock@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2054.codfw.wmnet with reason: host reimage [18:15:21] !log jhancock@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2055.codfw.wmnet with reason: host reimage [18:15:51] !log jhancock@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2056.codfw.wmnet with reason: host reimage [18:18:12] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2054.codfw.wmnet with reason: host reimage [18:21:24] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2056.codfw.wmnet with reason: host reimage [18:24:47] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2055.codfw.wmnet with reason: host reimage [18:25:08] hi ops folks, I'm pondering deploying a very small CentralNotice fix but it's Friday (the 13th no less). The recent addition of 'script-src-attr: none' to the CSP has broken banner previews due to our extra bit of code alerting on any CSP violation ( https://phabricator.wikimedia.org/T444656 ) [18:25:38] the fix is just to avoid the alert when the blockedUri is 'inline': https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralNotice/+/1369504 [18:26:51] there's also a fix for another regression already merged to CentralNotice's deploy branch: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralNotice/+/1368712 [18:27:17] so the easiest thing to do would be to just cherry-pick the CSP fix over to wmf_deploy and get those both out at once [18:31:31] anyway, i'm willing to take the risk and deploy today. Will be around for at least a few hours to check that no new errors crop up as a result [18:32:57] oh i see the IRC message template [18:34:17] thcipriani and / or brennen would it be OK if I deployed at least the CSP fix in the next hour? [18:34:44] I think the banner team is working over the weekend and I'd like to let them continue previewing banners [18:34:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12556493 (10phaultfinder) [18:35:05] the alternative would be a partial rollback of config patch 05f58b995e1411cc544a08c54b7a738b3137cf0e [18:35:07] !log jhancock@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003" [18:35:17] just the part that adds + "script-src-attr 'none'", [18:35:24] !log jhancock@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003" [18:35:26] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2054.codfw.wmnet with OS bookworm [18:35:39] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install ganeti205[1-6] - https://phabricator.wikimedia.org/T434698#12556494 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2003 for host ganeti2054.codfw.wmnet with OS bookworm completed: - ganet... [18:38:21] !log jhancock@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003" [18:39:02] !log jhancock@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003" [18:39:04] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2056.codfw.wmnet with OS bookworm [18:39:17] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install ganeti205[1-6] - https://phabricator.wikimedia.org/T434698#12556499 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2003 for host ganeti2056.codfw.wmnet with OS bookworm completed: - ganet... [18:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:42:45] !log jhancock@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003" [18:43:46] !log jhancock@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003" [18:43:47] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2055.codfw.wmnet with OS bookworm [18:43:56] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install ganeti205[1-6] - https://phabricator.wikimedia.org/T434698#12556511 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2003 for host ganeti2055.codfw.wmnet with OS bookworm completed: - ganet... [18:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [18:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [18:54:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12556519 (10phaultfinder) [19:00:28] ok operations folks, I guess the CN issues don't rise to the level of an emergency deploy. I will schedule a regular deploy slot for Monday [19:01:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2141.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2159.codfw.wmnet, wikikube-work [19:01:13] odfw.wmnet, wikikube-worker2179.codfw.wmnet, wikikube-worker2296.codfw.wmnet, wikikube-worker2018.codfw.wmnet, wikikube-worker2333.codfw.wmnet, wikikube-worker2049.codfw.wmnet, wikikube-worker2212.codfw.wmnet, wikikube-worker2275.codfw.wmnet, wikikube-worker2284.codfw.wmnet, wikikube-worker2354.codfw.wmnet, wikikube-worker2129.codfw.wmnet, wikikube-worker2167.codfw.wmnet, wikikube-worker2074.codfw.wmnet, wikikube-worker2310.codfw.wmnet, w [19:01:13] worker2145.codfw.wmnet, wikikube-worker2359.codfw.wmnet, wikikube-worker2131.codfw.wmnet, wikikube-worker2288.codfw.wmnet, wikikube-worker2338.codfw.wmnet, wikikube-worker2250.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [19:02:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:05:23] (03PS1) 10Ejegg: Update CentralNotice for preview bugfixes [core] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1369512 [19:06:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 28d 23h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [19:06:50] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 16 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [core] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1369512 (owner: 10Ejegg) [19:15:24] FIRING: [11x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:15:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12556541 (10phaultfinder) [19:17:04] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:20:15] PROBLEM - OSPF status on cr2-drmrs is CRITICAL: OSPFv2: 2/4 UP : OSPFv3: 2/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [19:20:25] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr2-drmrs:et-0/0/0 (Transport: Arelion (IC-398708) [20Gbps]) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-drmrs:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [19:22:04] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:24:39] RESOLVED: [2x] TransitBGPDown: Transit BGP session down between cr1-drmrs and Arelion (2001:2035:0:cf1::1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [19:28:43] RECOVERY - OSPF status on cr2-codfw is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [19:29:09] !log dancy@deploy2003 Installing scap version "4.314.0" for 154 host(s) [19:29:15] RECOVERY - OSPF status on cr2-drmrs is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [19:29:31] RECOVERY - OSPF status on cr2-eqiad is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [19:31:10] RESOLVED: [2x] BFDdown: BFD session down between cr2-codfw and 185.15.58.159 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [19:31:39] RESOLVED: [3x] CoreBGPDown: Core BGP session down between cr2-codfw and cr2-drmrs (185.15.58.159) - group Confed_drmrs - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [19:33:11] !log dancy@deploy2003 Installation of scap version "4.314.0" completed for 154 hosts [19:34:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12556567 (10phaultfinder) [19:38:00] ejegg: apologies for missing the earlier ping; working-from-coffeeshop and missed the notification. i don't think there's an objection to a deploy from releng side if you still wanted to get one going, provided there are SREs about (cc: Emperor). [19:40:02] thanks brennen ! I'll hold off till monday, the banner team realized they could preview on latin-wiki for the next couple days :) [19:40:15] ejegg: cool, sounds good. :) [19:48:33] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install ganeti205[1-6] - https://phabricator.wikimedia.org/T434698#12556621 (10Jhancock.wm) [19:49:35] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install ganeti205[1-6] - https://phabricator.wikimedia.org/T434698#12556622 (10Jhancock.wm) 05Open→03Resolved finished! [20:04:29] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:10:39] PROBLEM - MariaDB Replica Lag: s1 on db2250 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 631.21 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:16:15] PROBLEM - OSPF status on cr2-drmrs is CRITICAL: OSPFv2: 2/4 UP : OSPFv3: 2/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [20:16:33] PROBLEM - OSPF status on cr2-eqiad is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [20:16:43] PROBLEM - OSPF status on cr2-codfw is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [20:17:39] FIRING: CoreBGPDown: Core BGP session down between cr2-codfw and cr2-drmrs (185.15.58.159) - group Confed_drmrs - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr2-codfw:9804&var-bgp_group=Confed_drmrs&var-bgp_neighbor=cr2-drmrs - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [20:18:10] FIRING: [4x] BFDdown: BFD session down between cr2-codfw and 185.15.58.159 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [20:19:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12556711 (10phaultfinder) [20:22:39] FIRING: [3x] CoreBGPDown: Core BGP session down between cr2-codfw and cr2-drmrs (185.15.58.159) - group Confed_drmrs - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [20:23:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr1-drmrs and Arelion (2001:2035:0:cf1::1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [20:41:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:44:15] RECOVERY - OSPF status on cr2-drmrs is OK: OSPFv2: 2/2 UP : OSPFv3: 2/2 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [20:46:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:47:41] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-drmrs:et-0/0/0 (Transport: Arelion (IC-398708) [20Gbps]) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-drmrs:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [20:53:10] FIRING: [4x] BFDdown: BFD session down between cr2-codfw and 185.15.58.159 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [20:59:15] PROBLEM - OSPF status on cr2-drmrs is CRITICAL: OSPFv2: 2/4 UP : OSPFv3: 2/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [21:00:25] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr2-drmrs:et-0/0/0 (Transport: Arelion (IC-398708) [20Gbps]) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-drmrs:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [21:00:39] RECOVERY - MariaDB Replica Lag: s1 on db2250 is OK: OK slave_sql_lag Replication lag: 0.02 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:01:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:06:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:18:39] RESOLVED: [2x] TransitBGPDown: Transit BGP session down between cr1-drmrs and Arelion (2001:2035:0:cf1::1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [21:27:15] RECOVERY - OSPF status on cr2-drmrs is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [21:27:33] RECOVERY - OSPF status on cr2-eqiad is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [21:27:43] RECOVERY - OSPF status on cr2-codfw is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [21:28:10] FIRING: [2x] BFDdown: BFD session down between cr2-codfw and 185.15.58.159 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [21:29:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12556897 (10phaultfinder) [21:31:32] !log slyngshede@cumin1004 START - Cookbook sre.hosts.reboot-single for host idp1005.wikimedia.org [21:32:39] RESOLVED: [3x] CoreBGPDown: Core BGP session down between cr2-codfw and cr2-drmrs (185.15.58.159) - group Confed_drmrs - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [21:33:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:33:10] RESOLVED: [2x] BFDdown: BFD session down between cr2-codfw and 185.15.58.159 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [21:33:24] (03PS1) 10Slyngshede: IDP switch over [dns] - 10https://gerrit.wikimedia.org/r/1369521 [21:35:27] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp1005.wikimedia.org [21:36:31] PROBLEM - MariaDB Replica Lag: s4 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 637.42 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:36:34] (03CR) 10Slyngshede: [C:03+2] IDP switch over [dns] - 10https://gerrit.wikimedia.org/r/1369521 (owner: 10Slyngshede) [21:36:42] !log slyngshede@dns1004 START - running authdns-update [21:38:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:39:00] !log slyngshede@dns1004 END - running authdns-update [21:41:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:45:17] !log slyngshede@cumin1004 START - Cookbook sre.hosts.reboot-single for host idp2005.wikimedia.org [21:46:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:49:03] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp2005.wikimedia.org [21:52:07] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.64% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [21:53:25] PROBLEM - Host cloudlb2002-dev is DOWN: PING CRITICAL - Packet loss = 100% [21:54:13] PROBLEM - BFD status on cloudsw1-b1-codfw.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [21:54:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cloudsw1-b1-codfw and cloudlb2002-dev (172.20.5.3) - group cloud_host - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [21:54:55] RECOVERY - Host cloudlb2002-dev is UP: PING OK - Packet loss = 0%, RTA = 31.62 ms [21:59:10] FIRING: [4x] BFDdown: BFD session down between cloudsw1-b1-codfw and 172.20.5.4 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cloudsw1-b1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [21:59:39] RESOLVED: [4x] CoreBGPDown: Core BGP session down between cloudsw1-b1-codfw and cloudlb2002-dev (172.20.5.3) - group cloud_host - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [21:59:41] PROBLEM - MariaDB Replica Lag: x4 on db2250 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 614.22 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:01:13] RECOVERY - BFD status on cloudsw1-b1-codfw.mgmt is OK: UP: 10 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [22:04:10] RESOLVED: [6x] BFDdown: BFD session down between cloudsw1-b1-codfw and 172.20.5.4 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cloudsw1-b1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [22:05:25] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:08:45] !log vgutierrez@cumin1004 START - Cookbook sre.dns.roll-reboot rolling reboot on A:dnsbox [22:08:45] !log vgutierrez@cumin1004 cookbooks.sre.dns.roll-reboot begin reboot of dns1004.wikimedia.org [22:10:09] FIRING: [8x] CoreBGPDown: Core BGP session down between cloudsw1-b1-codfw and cloudlb2002-dev (172.20.5.3) - group cloud_host - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [22:14:21] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling reboot on A:ncredir [22:14:25] FIRING: [10x] BFDdown: BFD session down between cloudsw1-b1-codfw and 172.20.5.4 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [22:14:54] RESOLVED: [8x] CoreBGPDown: Core BGP session down between cloudsw1-b1-codfw and cloudlb2002-dev (172.20.5.3) - group cloud_host - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [22:15:40] FIRING: [10x] BFDdown: BFD session down between cloudsw1-b1-codfw and 172.20.5.4 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [22:17:19] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_magru [22:17:22] !log vgutierrez@cumin1004 cookbooks.sre.dns.roll-reboot finished rebooting dns1004.wikimedia.org [22:17:45] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_magru [22:18:54] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_ulsfo [22:19:30] (03PS1) 10Ahmon Dancy: beta: Disable the shell cgroup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369524 (https://phabricator.wikimedia.org/T435866) [22:19:33] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_ulsfo [22:20:25] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqsin [22:21:01] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqsin [22:21:52] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_codfw [22:22:19] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_codfw [22:24:09] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_drmrs [22:24:38] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_drmrs [22:25:31] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_esams [22:25:58] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_esams [22:26:04] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:27:05] !log vgutierrez@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp7011.magru.wmnet