[00:00:06] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [00:00:13] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [00:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:05:24] RESOLVED: [2x] SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:13:51] (03PS4) 10RLazarus: services: Remove codfw from sites for wikifunctions-python-evaluator-staging [puppet] - 10https://gerrit.wikimedia.org/r/1364993 (https://phabricator.wikimedia.org/T424193) [00:15:07] (03CR) 10RLazarus: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1364993 (https://phabricator.wikimedia.org/T424193) (owner: 10RLazarus) [00:16:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.75% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [00:17:23] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [00:27:01] (03PS5) 10RLazarus: services: Remove codfw from sites for wikifunctions-python-evaluator-staging [puppet] - 10https://gerrit.wikimedia.org/r/1364993 (https://phabricator.wikimedia.org/T424193) [00:30:14] (03CR) 10RLazarus: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1364993 (https://phabricator.wikimedia.org/T424193) (owner: 10RLazarus) [00:35:01] (03CR) 10RLazarus: [C:03+2] services: Remove codfw from sites for wikifunctions-python-evaluator-staging [puppet] - 10https://gerrit.wikimedia.org/r/1364993 (https://phabricator.wikimedia.org/T424193) (owner: 10RLazarus) [00:39:02] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1365098 [00:39:02] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1365098 (owner: 10TrainBranchBot) [00:45:09] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1365098 (owner: 10TrainBranchBot) [00:51:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [00:52:46] FIRING: [6x] CertAlmostExpired: Certificate for service kubestagemaster2003:6443 is about to expire - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [01:07:31] FIRING: [2x] Not accepting/receiving prefixes from anycast BGP peer: Alert for device asw1-b3-magru.mgmt.magru.wmnet - Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [01:08:00] FIRING: PingLossPercent: Blackbox probe packet loss 4.167% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [01:09:10] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1365147 [01:09:10] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1365147 (owner: 10TrainBranchBot) [01:15:19] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1365147 (owner: 10TrainBranchBot) [01:43:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.333% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [01:44:56] (03PS9) 10Krinkle: varnish: Don't strip browser Cache-Control:maxage from action=raw [puppet] - 10https://gerrit.wikimedia.org/r/1364773 (https://phabricator.wikimedia.org/T279120) [01:47:53] 07sre-alert-triage, 06Abstract Wikipedia team, 10Wikifunctions: Alert in need of triage: ProbeDown (instance wikifunctions-python-evaluator-staging:30443) - https://phabricator.wikimedia.org/T429678#12525583 (10RLazarus) 05Open→03Resolved Alert's no longer firing -- looks like that took care of it. [01:52:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.958% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [01:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.02% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [02:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:22:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.333% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [02:28:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.333% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [02:33:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [02:39:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.542% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [02:54:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [03:00:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.75% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [04:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:05:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [04:17:23] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [04:24:57] (03PS2) 10JHathaway: Add a script to set the in_service_date [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1364871 (https://phabricator.wikimedia.org/T441507) [04:26:56] (03CR) 10CI reject: [V:04-1] Add a script to set the in_service_date [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1364871 (https://phabricator.wikimedia.org/T441507) (owner: 10JHathaway) [04:32:35] (03PS3) 10JHathaway: Add a script to set the in_service_date [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1364871 (https://phabricator.wikimedia.org/T441507) [04:42:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.333% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [04:47:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [04:52:46] FIRING: [6x] CertAlmostExpired: Certificate for service kubestagemaster2003:6443 is about to expire - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [04:53:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.75% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [04:56:45] (03PS1) 10Jasmine: pontoon, admin: remove references to parsoidtest [puppet] - 10https://gerrit.wikimedia.org/r/1365619 (https://phabricator.wikimedia.org/T433239) [05:07:31] FIRING: [2x] Not accepting/receiving prefixes from anycast BGP peer: Alert for device asw1-b3-magru.mgmt.magru.wmnet - Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [05:17:24] (03PS1) 10Jasmine: trafficserver: remove testreduce mapping, stop incoming traffic to backend [puppet] - 10https://gerrit.wikimedia.org/r/1365659 (https://phabricator.wikimedia.org/T436917) [05:21:13] (03Abandoned) 10Jasmine: trafficserver: remove testreduce mapping, stop incoming traffic to backend [puppet] - 10https://gerrit.wikimedia.org/r/1365659 (https://phabricator.wikimedia.org/T436917) (owner: 10Jasmine) [05:48:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [05:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.07% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [05:54:00] FIRING: PingLossPercent: Blackbox probe packet loss 4.167% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [06:14:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [06:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [06:33:03] (03PS1) 10Dzahn: codesearch: mount the hound gitconfig as /etc/gitconfig [puppet] - 10https://gerrit.wikimedia.org/r/1365703 (https://phabricator.wikimedia.org/T268199) [06:52:58] !log push pfw policies - T443852 [06:52:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261106T0700) [07:14:30] 10ops-eqiad, 06DC-Ops: Unresponsive management for an-worker1147.mgmt:22 - https://phabricator.wikimedia.org/T443870 (10phaultfinder) 03NEW [07:28:08] (03PS1) 10Muehlenhoff: Remove legacy keys for dsantamaria, bd808 and rscout [puppet] - 10https://gerrit.wikimedia.org/r/1365785 (https://phabricator.wikimedia.org/T442758) [07:30:04] (03PS1) 10Hashar: ci: upgrade QEmu image from Bullseye to Trixie [puppet] - 10https://gerrit.wikimedia.org/r/1365786 [07:32:46] FIRING: [6x] CertAlmostExpired: Certificate for service kubestagemaster2003:6443 is about to expire - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [07:34:32] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy keys for dsantamaria, bd808 and rscout [puppet] - 10https://gerrit.wikimedia.org/r/1365785 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [07:38:12] FIRING: [2x] ProbeDown: Service kubestagemaster2005:6443 has failed probes (http_staging_codfw_kube_apiserver_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#kubestagemaster2005:6443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:40:53] (03PS2) 10Hashar: ci: upgrade QEmu image from Bullseye to Trixie [puppet] - 10https://gerrit.wikimedia.org/r/1365786 (https://phabricator.wikimedia.org/T443872) [07:44:41] (03PS2) 10Kevin Bazira: airflow-ml: enable the urldownloader [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364448 (https://phabricator.wikimedia.org/T441987) [07:45:58] (03CR) 10Muehlenhoff: [C:03+1] "Key has been verified out of band." [puppet] - 10https://gerrit.wikimedia.org/r/1364857 (https://phabricator.wikimedia.org/T442758) (owner: 10Dr0ptp4kt) [07:46:04] (03CR) 10Muehlenhoff: [C:03+2] Add yubikey 1 ssh pub key for dr0ptp4kt [puppet] - 10https://gerrit.wikimedia.org/r/1364857 (https://phabricator.wikimedia.org/T442758) (owner: 10Dr0ptp4kt) [07:53:52] (03PS2) 10Gehel: admin: adding Leon Ziemba as contact for sd [puppet] - 10https://gerrit.wikimedia.org/r/1364022 (https://phabricator.wikimedia.org/T442756) [07:54:44] (03PS3) 10Gehel: admin: adding MusikAnimal as contact for sd [puppet] - 10https://gerrit.wikimedia.org/r/1364022 (https://phabricator.wikimedia.org/T442756) [07:55:28] (03CR) 10Gehel: admin: adding MusikAnimal as contact for sd (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1364022 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [07:55:44] (03PS4) 10Gehel: admin: adding MusikAnimal as contact for sd [puppet] - 10https://gerrit.wikimedia.org/r/1364022 (https://phabricator.wikimedia.org/T442756) [07:57:15] (03CR) 10Gehel: [C:03+2] admin: adding MusikAnimal as contact for sd [puppet] - 10https://gerrit.wikimedia.org/r/1364022 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [08:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261106T0800) [08:00:11] (03PS1) 10Gehel: admin: adding FIDO/SSH keys for sd [puppet] - 10https://gerrit.wikimedia.org/r/1365868 (https://phabricator.wikimedia.org/T442756) [08:03:36] (03PS1) 10Gehel: admin: adding FIDO/SSH keys for hghani, radimer [puppet] - 10https://gerrit.wikimedia.org/r/1365871 (https://phabricator.wikimedia.org/T442756) [08:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:08:02] (03PS2) 10Arendpieter: apereo_cas: Allow CAS to redirect clients after logout [puppet] - 10https://gerrit.wikimedia.org/r/1341188 (https://phabricator.wikimedia.org/T392350) [08:08:02] (03PS2) 10Arendpieter: apereo_cas: Allow services to declare a logout endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1341189 (https://phabricator.wikimedia.org/T233941) [08:13:53] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 06Infrastructure-Foundations: Reclaim parsoidtest1001 as a Ganeti node - https://phabricator.wikimedia.org/T421484#12525990 (10JMeybohm) >>! In T421484#12524140, @Dzahn wrote: > The host still appeared in other configs including CDN which is why it h... [08:17:23] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [08:22:13] (03CR) 10Btullis: [C:03+1] admin: adding FIDO/SSH keys for hghani, radimer [puppet] - 10https://gerrit.wikimedia.org/r/1365871 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [08:35:06] (03PS1) 10Jelto: wmf-navigator: decrease data-sync frequency to daily sync [deployment-charts] - 10https://gerrit.wikimedia.org/r/1365921 (https://phabricator.wikimedia.org/T443826) [08:37:04] (03CR) 10Lucian: [C:04-1] "Will wait with this until I get the hardware-backed keys." [puppet] - 10https://gerrit.wikimedia.org/r/1364304 (owner: 10Lucian) [08:37:10] (03PS2) 10Effie Mouzeli: maps: Disable replication timers in eqiad/bookworm [puppet] - 10https://gerrit.wikimedia.org/r/1364328 (https://phabricator.wikimedia.org/T443438) [08:40:55] (03PS1) 10Effie Mouzeli: (DNM) maps: Disable replication timers in codfw/bookworm [puppet] - 10https://gerrit.wikimedia.org/r/1365926 (https://phabricator.wikimedia.org/T443438) [08:43:33] (03CR) 10Jelto: [C:03+2] wmf-navigator: decrease data-sync frequency to daily sync [deployment-charts] - 10https://gerrit.wikimedia.org/r/1365921 (https://phabricator.wikimedia.org/T443826) (owner: 10Jelto) [08:45:04] (03Merged) 10jenkins-bot: wmf-navigator: decrease data-sync frequency to daily sync [deployment-charts] - 10https://gerrit.wikimedia.org/r/1365921 (https://phabricator.wikimedia.org/T443826) (owner: 10Jelto) [08:46:18] (03CR) 10MVernon: [C:04-1] "Hi," [puppet] - 10https://gerrit.wikimedia.org/r/1364304 (owner: 10Lucian) [08:47:20] (03CR) 10Gehel: [C:03+2] admin: adding FIDO/SSH keys for hghani, radimer [puppet] - 10https://gerrit.wikimedia.org/r/1365871 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [08:47:54] !log mvernon@cumin2003 START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe [08:48:05] !log mvernon@cumin1004 START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe [08:52:05] (03CR) 10Vgutierrez: "key confirmed OOB using Slack" [puppet] - 10https://gerrit.wikimedia.org/r/1364221 (https://phabricator.wikimedia.org/T443758) (owner: 10Vgutierrez) [08:52:05] !log mvernon@cumin2003 END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe [08:52:38] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for ecarg - https://phabricator.wikimedia.org/T443758#12526063 (10Vgutierrez) 05Stalled→03In progress key confirmed OOB using Slack [08:53:23] !log jelto@deploy2003 Started scap deploy-service miscweb-aux: Deploy to fix T443826 [08:53:23] !log jelto@deploy2003 scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb [08:56:05] !log jelto@deploy2003 scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb [08:57:35] !log jelto@deploy2003 Finished scap deploy-service miscweb-aux: Deploy to fix T443826 (duration: 04m 11s) [08:59:08] (03CR) 10Effie Mouzeli: [C:03+1] imposm: Skip water lines import because source mirror is down [puppet] - 10https://gerrit.wikimedia.org/r/1364444 (https://phabricator.wikimedia.org/T443438) (owner: 10Jgiannelos) [09:01:35] (03PS1) 10Gehel: admin: adding FIDO/SSH keys for dsaez [puppet] - 10https://gerrit.wikimedia.org/r/1365945 (https://phabricator.wikimedia.org/T442756) [09:04:06] !log mvernon@cumin1004 END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe [09:07:12] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12526102 (10CWilliams-WMF) @Marostegui the charts for this box suggest that it has been coping with the workload since being repooled, so I will close it later today if that persists. [09:07:31] FIRING: [2x] Not accepting/receiving prefixes from anycast BGP peer: Alert for device asw1-b3-magru.mgmt.magru.wmnet - Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [09:08:17] (03PS3) 10Hashar: ci: upgrade QEmu image from Bullseye to Trixie [puppet] - 10https://gerrit.wikimedia.org/r/1365786 (https://phabricator.wikimedia.org/T443872) [09:08:28] (03CR) 10Effie Mouzeli: [C:03+2] imposm: Skip water lines import because source mirror is down [puppet] - 10https://gerrit.wikimedia.org/r/1364444 (https://phabricator.wikimedia.org/T443438) (owner: 10Jgiannelos) [09:10:23] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12526104 (10Marostegui) +1 thanks [09:14:04] (03PS1) 10Effie Mouzeli: (DNM) tegola: switch to new swift container (eqiad) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1365961 (https://phabricator.wikimedia.org/T443438) [09:14:06] (03PS1) 10Effie Mouzeli: (DNM) tegola: switch to new swift container (codfw) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1365962 (https://phabricator.wikimedia.org/T443438) [09:17:05] (03PS4) 10Hashar: ci: upgrade QEmu image from Bullseye to Trixie [puppet] - 10https://gerrit.wikimedia.org/r/1365786 (https://phabricator.wikimedia.org/T443872) [09:23:08] (03CR) 10Btullis: [C:03+1] admin: adding FIDO/SSH keys for dsaez [puppet] - 10https://gerrit.wikimedia.org/r/1365945 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [09:28:51] (03PS1) 10Muehlenhoff: Add FIDO keys for ejegg, trueg and sbisson [puppet] - 10https://gerrit.wikimedia.org/r/1365965 (https://phabricator.wikimedia.org/T442758) [09:37:57] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO keys for ejegg, trueg and sbisson [puppet] - 10https://gerrit.wikimedia.org/r/1365965 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [09:43:06] (03PS1) 10Cathal Mooney: SR-Linux: add description to breakout parent ports [homer/public] - 10https://gerrit.wikimedia.org/r/1365971 [09:44:52] (03CR) 10Ayounsi: [C:03+1] SR-Linux: add description to breakout parent ports [homer/public] - 10https://gerrit.wikimedia.org/r/1365971 (owner: 10Cathal Mooney) [09:45:13] (03CR) 10Cathal Mooney: [C:03+2] SR-Linux: add description to breakout parent ports [homer/public] - 10https://gerrit.wikimedia.org/r/1365971 (owner: 10Cathal Mooney) [09:45:34] (03PS1) 10Jelto: wmf-navigator: update container images to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1365974 (https://phabricator.wikimedia.org/T443357) [09:46:03] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1364261 (owner: 10L10n-bot) [09:46:22] (03Merged) 10jenkins-bot: SR-Linux: add description to breakout parent ports [homer/public] - 10https://gerrit.wikimedia.org/r/1365971 (owner: 10Cathal Mooney) [09:46:28] (03PS1) 10Aqu: admin: Add FIDO SSH key for aqu [puppet] - 10https://gerrit.wikimedia.org/r/1365975 (https://phabricator.wikimedia.org/T442758) [09:47:15] (03CR) 10Jelto: [C:03+1] "lgtm" [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/1364330 (https://phabricator.wikimedia.org/T443455) (owner: 10JMeybohm) [09:49:18] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888 (10MatthewVernon) 03NEW [09:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.21% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [09:52:23] RESOLVED: CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [09:53:34] (03Abandoned) 10Muehlenhoff: admin: add fido key for sbisson [puppet] - 10https://gerrit.wikimedia.org/r/1364574 (https://phabricator.wikimedia.org/T442758) (owner: 10Sbisson) [09:55:26] (03CR) 10Volans: [C:04-1] "Good question, I agree that it should live with the netbox ganeti scripts, if we want to keep all of them here or in puppet, no strong pre" [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1364871 (https://phabricator.wikimedia.org/T441507) (owner: 10JHathaway) [10:01:54] (03PS1) 10Marostegui: mariadb grants: Limit icinga to 10 connections. [puppet] - 10https://gerrit.wikimedia.org/r/1365999 (https://phabricator.wikimedia.org/T442995) [10:03:46] (03PS1) 10Ayounsi: Report test_enabled_not_connected - ignore mark_connected interfaces [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1366003 [10:04:18] (03CR) 10Cathal Mooney: [C:03+1] "LGTM!" [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1366003 (owner: 10Ayounsi) [10:06:06] (03CR) 10Ayounsi: [C:03+2] Report test_enabled_not_connected - ignore mark_connected interfaces [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1366003 (owner: 10Ayounsi) [10:06:39] (03CR) 10CWilliams: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1365999 (https://phabricator.wikimedia.org/T442995) (owner: 10Marostegui) [10:07:35] (03Merged) 10jenkins-bot: Report test_enabled_not_connected - ignore mark_connected interfaces [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1366003 (owner: 10Ayounsi) [10:08:31] !log ayounsi@cumin1004 START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary [10:08:33] (03CR) 10Marostegui: [C:03+2] mariadb grants: Limit icinga to 10 connections. [puppet] - 10https://gerrit.wikimedia.org/r/1365999 (https://phabricator.wikimedia.org/T442995) (owner: 10Marostegui) [10:08:51] (03PS1) 10Clément Goubert: mw-api-ext: Scale next if needed for 8.5 migration [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366008 (https://phabricator.wikimedia.org/T439955) [10:08:52] (03CR) 10Btullis: [C:03+2] wdqs: Remove hosts that we will rename from firewall lists [puppet] - 10https://gerrit.wikimedia.org/r/1364508 (https://phabricator.wikimedia.org/T443792) (owner: 10Btullis) [10:11:47] ayounsi@cumin1004 update-extras (PID 973270) is awaiting input [10:11:51] !log fabfur@cumin1004 conftool action : set/pooled=yes; selector: dc=magru,cluster=dnsbox [10:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [10:14:28] (03CR) 10Btullis: [C:03+2] dse_k8s: Add site.pp and preseed entries for six new workers [puppet] - 10https://gerrit.wikimedia.org/r/1364509 (https://phabricator.wikimedia.org/T443792) (owner: 10Btullis) [10:16:04] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary [10:16:12] !log ayounsi@cumin1004 START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox [10:16:40] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox [10:17:31] RESOLVED: [2x] Not accepting/receiving prefixes from anycast BGP peer: Device asw1-b3-magru.mgmt.magru.wmnet recovered from Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [10:22:01] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888#12526410 (10MatthewVernon) We can find where the container dbs should be: ` /usr/bin/swift-get-nodes /etc/swift/container.ring.gz AUTH_tegola tegola-swift-eqiad-v003 ` which tells us (after some hostname loo... [10:22:21] (03PS1) 10Muehlenhoff: Add FIDO keys for hoo [puppet] - 10https://gerrit.wikimedia.org/r/1366024 (https://phabricator.wikimedia.org/T442758) [10:23:10] (03CR) 10CI reject: [V:04-1] Add FIDO keys for hoo [puppet] - 10https://gerrit.wikimedia.org/r/1366024 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [10:23:40] (03CR) 10Jelto: [C:03+2] wmf-navigator: update container images to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1365974 (https://phabricator.wikimedia.org/T443357) (owner: 10Jelto) [10:25:14] (03Merged) 10jenkins-bot: wmf-navigator: update container images to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1365974 (https://phabricator.wikimedia.org/T443357) (owner: 10Jelto) [10:29:08] (03PS2) 10Muehlenhoff: Add FIDO keys for hoo [puppet] - 10https://gerrit.wikimedia.org/r/1366024 (https://phabricator.wikimedia.org/T442758) [10:29:11] (03CR) 10Effie Mouzeli: [C:03+2] maps: Disable replication timers in eqiad/bookworm [puppet] - 10https://gerrit.wikimedia.org/r/1364328 (https://phabricator.wikimedia.org/T443438) (owner: 10Effie Mouzeli) [10:29:13] !log Limit icinga db user connections to 10 in ms* and pc* T442995 [10:29:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:29:16] T442995: pc5 and pc7 issues - https://phabricator.wikimedia.org/T442995 [10:31:15] !log jelto@deploy2003 Started scap deploy-service miscweb-aux: Deploy to fix T443357 [10:31:15] !log jelto@deploy2003 scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb [10:32:09] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO keys for hoo [puppet] - 10https://gerrit.wikimedia.org/r/1366024 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [10:32:10] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888#12526446 (10MatthewVernon) The quarantined copy on thanos-be2008 is quite badly corrupted: ` root@thanos-be2008:/srv/swift-storage/accounts0/quarantined/containers/4f0ba9617366341a8939cbe7634ba9e8-b94dccb673... [10:33:39] !log jelto@deploy2003 scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb [10:34:54] !log jelto@deploy2003 Finished scap deploy-service miscweb-aux: Deploy to fix T443357 (duration: 03m 39s) [10:39:51] (03CR) 10Clément Goubert: trafficserver: route WDQSv2 through REST Gateway (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363813 (https://phabricator.wikimedia.org/T443756) (owner: 10Gmodena) [10:41:50] jouncebot: now [10:41:50] For the next 21 hour(s) and 18 minute(s): No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261106T0800) [10:42:27] !log depool kartotherian from eqiad (ie maps) - T443438 [10:42:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:42:29] T443438: Plan, kickoff and monitor the Maps planet import - https://phabricator.wikimedia.org/T443438 [10:44:24] !log jiji@cumin1004 START - Cookbook sre.discovery.service-route depool kartotherian in eqiad: maintenance [10:49:28] !log jiji@cumin1004 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool kartotherian in eqiad: maintenance [10:53:40] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888#12526520 (10MatthewVernon) Thanos-be1005's copy is less damaged, but still probably not useful ` root@thanos-be1005:/srv/swift-storage/accounts1/quarantined/containers/4f0ba9617366341a8939cbe7634ba9e8# sqlit... [10:54:40] (03CR) 10Vgutierrez: [C:03+2] admin: Add FIDO-backed SSH key for ecarg [puppet] - 10https://gerrit.wikimedia.org/r/1364221 (https://phabricator.wikimedia.org/T443758) (owner: 10Vgutierrez) [10:57:03] (03PS1) 10Muehlenhoff: Add FIDO keys for ppenloglu and nahidunlimited [puppet] - 10https://gerrit.wikimedia.org/r/1366038 (https://phabricator.wikimedia.org/T442760) [10:57:22] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for ecarg - https://phabricator.wikimedia.org/T443758#12526537 (10Vgutierrez) 05In progress→03Stalled FIDO key deployed, please let us know if it works as expected and we can remove the legacy one @ecarg [11:00:48] !log btullis@cumin1004 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1030.eqiad.wmnet [11:01:42] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO keys for ppenloglu and nahidunlimited [puppet] - 10https://gerrit.wikimedia.org/r/1366038 (https://phabricator.wikimedia.org/T442760) (owner: 10Muehlenhoff) [11:04:31] btullis@cumin1004 upgrade-firmware (PID 980710) is awaiting input [11:07:57] (03PS1) 10Daniel Kertesz: sre.dns.admin: log usage of the --depool-dns-hosts flag [cookbooks] - 10https://gerrit.wikimedia.org/r/1366053 [11:08:02] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet [11:08:05] (03CR) 10Muehlenhoff: [C:03+1] "Key has been verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1365975 (https://phabricator.wikimedia.org/T442758) (owner: 10Aqu) [11:08:08] (03CR) 10Muehlenhoff: [C:03+2] admin: Add FIDO SSH key for aqu [puppet] - 10https://gerrit.wikimedia.org/r/1365975 (https://phabricator.wikimedia.org/T442758) (owner: 10Aqu) [11:09:02] (03PS3) 10Clément Goubert: bento-mw-accesslog-metrics: set up kubeconfig file [puppet] - 10https://gerrit.wikimedia.org/r/1354784 (https://phabricator.wikimedia.org/T371214) (owner: 10Tiziano Fogli) [11:09:02] (03PS3) 10Clément Goubert: dmarc-tools: add kubeconfig [puppet] - 10https://gerrit.wikimedia.org/r/1363303 [11:09:02] (03PS2) 10Clément Goubert: ncmonitor: add kubeconfig [puppet] - 10https://gerrit.wikimedia.org/r/1363304 [11:09:03] (03PS2) 10Clément Goubert: pampinus: add kubeconfig [puppet] - 10https://gerrit.wikimedia.org/r/1363305 [11:09:28] !log btullis@cumin1004 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet [11:09:52] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wdqs2025.codfw.wmnet [11:12:53] (03PS1) 10Muehlenhoff: Add FIDO key for ngkountas [puppet] - 10https://gerrit.wikimedia.org/r/1366061 (https://phabricator.wikimedia.org/T442758) [11:17:22] (03CR) 10Clément Goubert: [C:03+2] pampinus: add kubeconfig [puppet] - 10https://gerrit.wikimedia.org/r/1363305 (owner: 10Clément Goubert) [11:17:35] (03CR) 10Fabfur: [C:03+1] sre.dns.admin: log usage of the --depool-dns-hosts flag [cookbooks] - 10https://gerrit.wikimedia.org/r/1366053 (owner: 10Daniel Kertesz) [11:18:01] (03CR) 10Clément Goubert: [C:03+2] ncmonitor: add kubeconfig [puppet] - 10https://gerrit.wikimedia.org/r/1363304 (owner: 10Clément Goubert) [11:18:12] (03CR) 10Muehlenhoff: [C:03+1] "Key has been verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1366061 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [11:18:14] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for ngkountas [puppet] - 10https://gerrit.wikimedia.org/r/1366061 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [11:18:20] (03CR) 10Clément Goubert: [C:03+2] dmarc-tools: add kubeconfig [puppet] - 10https://gerrit.wikimedia.org/r/1363303 (owner: 10Clément Goubert) [11:18:26] (03CR) 10Clément Goubert: [C:03+2] bento-mw-accesslog-metrics: set up kubeconfig file [puppet] - 10https://gerrit.wikimedia.org/r/1354784 (https://phabricator.wikimedia.org/T371214) (owner: 10Tiziano Fogli) [11:18:33] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet [11:18:34] !log btullis@cumin1004 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1030.eqiad.wmnet [11:18:59] (03PS2) 10Daniel Kertesz: sre.dns.admin: log usage of the --depool-dns-hosts flag [cookbooks] - 10https://gerrit.wikimedia.org/r/1366053 [11:19:00] !log btullis@cumin1004 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1031.eqiad.wmnet [11:20:06] (03CR) 10Vgutierrez: [C:03+1] sre.dns.admin: log usage of the --depool-dns-hosts flag [cookbooks] - 10https://gerrit.wikimedia.org/r/1366053 (owner: 10Daniel Kertesz) [11:21:25] (03PS5) 10Clément Goubert: basic-app: Introduce basic chart for basic apps [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362085 (https://phabricator.wikimedia.org/T436785) [11:21:57] (03CR) 10Clément Goubert: [C:03+2] bento-mw-accesslog-metrics: add namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1354785 (https://phabricator.wikimedia.org/T371214) (owner: 10Tiziano Fogli) [11:22:02] (03CR) 10Clément Goubert: [C:03+2] pampinus: add namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363315 (owner: 10Clément Goubert) [11:22:05] btullis@cumin1004 upgrade-firmware (PID 984509) is awaiting input [11:22:07] (03CR) 10Clément Goubert: [C:03+2] ncmonitor: add namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363316 (owner: 10Clément Goubert) [11:22:11] (03CR) 10Clément Goubert: [C:03+2] dmarc-tools: add namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363317 (owner: 10Clément Goubert) [11:25:38] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12526580 (10cmooney) [11:28:26] 10ops-eqiad, 06SRE, 06DC-Ops: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12526584 (10cmooney) p:05Medium→03Low a:05cmooney→03None The work here is complete from the LVS/network configuration side of things. DC-Ops when you have time we can remo... [11:29:39] (03Merged) 10jenkins-bot: bento-mw-accesslog-metrics: add namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1354785 (https://phabricator.wikimedia.org/T371214) (owner: 10Tiziano Fogli) [11:29:57] (03Merged) 10jenkins-bot: pampinus: add namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363315 (owner: 10Clément Goubert) [11:29:58] (03Merged) 10jenkins-bot: ncmonitor: add namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363316 (owner: 10Clément Goubert) [11:29:59] (03Merged) 10jenkins-bot: dmarc-tools: add namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363317 (owner: 10Clément Goubert) [11:30:00] !log a-pizzata@deploy2003 Started deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] [11:30:01] 10ops-eqiad, 06SRE, 06DC-Ops: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12526612 (10cmooney) [11:30:01] 06SRE, 06Infrastructure-Foundations, 10netops: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12526611 (10cmooney) [11:30:05] 06SRE, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh configuration - https://phabricator.wikimedia.org/T439535#12526613 (10cmooney) [11:30:46] !log a-pizzata@deploy2003 Finished deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] (duration: 00m 46s) [11:31:21] !log a-pizzata@deploy2003 Started deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] [11:31:47] !log btullis@cumin1004 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet [11:32:02] (03PS1) 10Marostegui: team-data-persistence: Alert on high running thread [alerts] - 10https://gerrit.wikimedia.org/r/1366067 (https://phabricator.wikimedia.org/T442995) [11:32:26] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet [11:32:39] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet [11:32:46] FIRING: [4x] CertAlmostExpired: Certificate for service kubestagemaster2003:6443 is about to expire - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:32:58] (03CR) 10Marostegui: "I checked the average values at https://thanos.wikimedia.org/graph?g0.expr=max_over_time(mysql_global_status_threads_running%7Binstance%3D" [alerts] - 10https://gerrit.wikimedia.org/r/1366067 (https://phabricator.wikimedia.org/T442995) (owner: 10Marostegui) [11:36:12] !log a-pizzata@deploy2003 Finished deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] (duration: 04m 50s) [11:36:23] (03PS1) 10Muehlenhoff: Add FIDO key for ep1c [puppet] - 10https://gerrit.wikimedia.org/r/1366069 (https://phabricator.wikimedia.org/T442760) [11:36:58] (03CR) 10CI reject: [V:04-1] team-data-persistence: Alert on high running thread [alerts] - 10https://gerrit.wikimedia.org/r/1366067 (https://phabricator.wikimedia.org/T442995) (owner: 10Marostegui) [11:37:11] (03PS1) 10Muehlenhoff: Add FIDO key for melos [puppet] - 10https://gerrit.wikimedia.org/r/1366070 (https://phabricator.wikimedia.org/T442760) [11:38:12] FIRING: [2x] ProbeDown: Service kubestagemaster2005:6443 has failed probes (http_staging_codfw_kube_apiserver_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#kubestagemaster2005:6443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:39:36] (03PS2) 10Marostegui: team-data-persistence: Alert on high running thread [alerts] - 10https://gerrit.wikimedia.org/r/1366067 (https://phabricator.wikimedia.org/T442995) [11:39:52] !log installing haproxykafka 0.4.0 on cp7001 for testing (T443632) (T443640) [11:39:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:39:57] T443632: Support x-kapow-score header in HaproxyKafka - https://phabricator.wikimedia.org/T443632 [11:39:57] T443640: Upgrade HaproxyKafka to go1.27 - https://phabricator.wikimedia.org/T443640 [11:40:21] (03CR) 10Ladsgroup: [C:03+1] "confirmed oob" [puppet] - 10https://gerrit.wikimedia.org/r/1366069 (https://phabricator.wikimedia.org/T442760) (owner: 10Muehlenhoff) [11:41:23] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for ep1c [puppet] - 10https://gerrit.wikimedia.org/r/1366069 (https://phabricator.wikimedia.org/T442760) (owner: 10Muehlenhoff) [11:42:37] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet [11:42:39] !log btullis@cumin1004 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1031.eqiad.wmnet [11:43:08] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet [11:43:10] !log btullis@cumin1004 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2025.codfw.wmnet [11:45:46] (03CR) 10Vgutierrez: [C:03+1] trafficserver: remove support for running multiple instances (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1363267 (https://phabricator.wikimedia.org/T443596) (owner: 10Daniel Kertesz) [11:47:29] !log btullis@cumin1004 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1032.eqiad.wmnet [11:47:47] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888#12526691 (10MatthewVernon) Thanos-be2008 filled up again, due to a combination of tmp and quarantined entries. I've cleared those all out again. [11:48:27] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet [11:48:47] (03PS1) 10Ayounsi: Interfaces valdator: access mode but not untagged_vlan [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1366074 (https://phabricator.wikimedia.org/T443893) [11:49:22] (03PS2) 10Ayounsi: Interfaces valdator: access mode but not untagged_vlan [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1366074 (https://phabricator.wikimedia.org/T443893) [11:49:53] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888#12526694 (10MatthewVernon) Likewise thanos-be1005 [11:49:56] !log cgoubert@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [11:50:14] (03CR) 10Cathal Mooney: [C:03+1] "LGTM! I'll take a look at the puppetdb import script too to try to prevent it doing what it did the other day, and also get it to run the" [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1366074 (https://phabricator.wikimedia.org/T443893) (owner: 10Ayounsi) [11:51:37] !log cgoubert@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [11:52:07] !log cgoubert@deploy2003 helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. [11:52:55] !log cgoubert@deploy2003 helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. [11:58:01] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1032.eqiad.wmnet [11:58:02] !log btullis@cumin1004 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1032.eqiad.wmnet [12:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261106T0800) [12:00:05] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [12:00:05] jelto, arnoldokoth, mutante, and arnaudb: It is that lovely time of the day again! You are hereby commanded to deploy GitLab version upgrades. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261106T1200). [12:00:10] (03CR) 10Ayounsi: [C:03+2] Interfaces valdator: access mode but not untagged_vlan [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1366074 (https://phabricator.wikimedia.org/T443893) (owner: 10Ayounsi) [12:00:15] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [12:00:19] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [12:00:29] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [12:00:32] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [12:00:39] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [12:00:58] (03CR) 10CWilliams: [C:03+1] team-data-persistence: Alert on high running thread [alerts] - 10https://gerrit.wikimedia.org/r/1366067 (https://phabricator.wikimedia.org/T442995) (owner: 10Marostegui) [12:01:33] (03CR) 10Marostegui: "Thanks - will deploy next week to avoid noise during the weekend/travel days." [alerts] - 10https://gerrit.wikimedia.org/r/1366067 (https://phabricator.wikimedia.org/T442995) (owner: 10Marostegui) [12:02:01] (03Merged) 10jenkins-bot: Interfaces valdator: access mode but not untagged_vlan [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1366074 (https://phabricator.wikimedia.org/T443893) (owner: 10Ayounsi) [12:02:43] !log ayounsi@cumin1004 START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary [12:03:18] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary [12:03:32] !log ayounsi@cumin1004 START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox [12:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:04:48] (03Abandoned) 10Ayounsi: Nokia: handle access interface with no untagged_vlan [homer/public] - 10https://gerrit.wikimedia.org/r/1362812 (owner: 10Ayounsi) [12:06:30] (03CR) 10Clément Goubert: "I'll just merge this and mesh.all so we can use them for the summit, we can iterate on finer points and new features later." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362085 (https://phabricator.wikimedia.org/T436785) (owner: 10Clément Goubert) [12:06:37] (03CR) 10Clément Goubert: [C:03+2] basic-app: Introduce basic chart for basic apps [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362085 (https://phabricator.wikimedia.org/T436785) (owner: 10Clément Goubert) [12:06:41] (03CR) 10Clément Goubert: [C:03+2] modules.mesh: Introduce mesh.all [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362123 (owner: 10Clément Goubert) [12:07:45] (03CR) 10Abijeet Patro: [C:04-1] ULS: Use the new language selector on Timeless on testwiki and mediawiki.org (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363037 (https://phabricator.wikimedia.org/T439715) (owner: 10Huei Tan) [12:08:25] (03Merged) 10jenkins-bot: modules.mesh: Introduce mesh.all [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362123 (owner: 10Clément Goubert) [12:08:26] (03Merged) 10jenkins-bot: basic-app: Introduce basic chart for basic apps [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362085 (https://phabricator.wikimedia.org/T436785) (owner: 10Clément Goubert) [12:12:14] !log imported openjdk-21 21.0.13+9-1~deb13u2~wmf12u1 to component/jdk21 for bookworm-wikimedia [12:12:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:14:35] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366088 [12:23:07] ayounsi@cumin1004 update-extras (PID 988960) is awaiting input [12:28:10] (03PS3) 10Vgutierrez: varnish: Add /esi/ URL rewrites [puppet] - 10https://gerrit.wikimedia.org/r/1366066 (https://phabricator.wikimedia.org/T443703) [12:29:12] !log btullis@cumin1004 START - Cookbook sre.hosts.rename from wdqs1030 to dse-k8s-worker1046 [12:29:16] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox [12:29:34] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [12:29:59] !log btullis@cumin1004 START - Cookbook sre.hosts.rename from wdqs1031 to dse-k8s-worker1047 [12:30:19] !log btullis@cumin1004 START - Cookbook sre.hosts.rename from wdqs1032 to dse-k8s-worker1048 [12:31:37] !log btullis@cumin1004 START - Cookbook sre.hosts.rename from wdqs2025 to dse-k8s-worker2013 [12:33:19] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004" [12:33:37] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004" [12:33:37] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:33:37] !log btullis@cumin1004 START - Cookbook sre.dns.wipe-cache dse-k8s-worker1046 on all recursors [12:33:40] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1046 on all recursors [12:33:41] !log btullis@cumin1004 START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1046 [12:33:41] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [12:34:20] !log btullis@cumin1004 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1046 [12:34:57] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1030 to dse-k8s-worker1046 [12:38:44] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004" [12:38:48] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004" [12:38:48] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:38:48] !log btullis@cumin1004 START - Cookbook sre.dns.wipe-cache dse-k8s-worker1048 on all recursors [12:38:51] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1048 on all recursors [12:38:51] !log btullis@cumin1004 START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1048 [12:39:07] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [12:40:11] !log btullis@cumin1004 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1048 [12:40:49] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1032 to dse-k8s-worker1048 [12:41:32] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:41:32] !log btullis@cumin1004 START - Cookbook sre.dns.wipe-cache dse-k8s-worker1047 on all recursors [12:41:35] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1047 on all recursors [12:41:35] !log btullis@cumin1004 START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1047 [12:42:00] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [12:42:46] FIRING: [4x] CertAlmostExpired: Certificate for service kubestagemaster2003:6443 is about to expire - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:44:12] (03PS4) 10Vgutierrez: varnish: Add /esi/ URL rewrites [puppet] - 10https://gerrit.wikimedia.org/r/1366066 (https://phabricator.wikimedia.org/T443703) [12:44:17] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:44:17] !log btullis@cumin1004 START - Cookbook sre.dns.wipe-cache dse-k8s-worker2013 on all recursors [12:44:20] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2013 on all recursors [12:44:21] !log btullis@cumin1004 START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2013 [12:44:41] btullis@cumin1004 rename (PID 992438) is awaiting input [12:45:39] !log btullis@cumin1004 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2013 [12:46:15] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs2025 to dse-k8s-worker2013 [12:46:34] !log btullis@cumin1004 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1047 [12:47:10] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1031 to dse-k8s-worker1047 [12:48:12] FIRING: [4x] ProbeDown: Service kubestagemaster2003:6443 has failed probes (http_staging_codfw_kube_apiserver_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:49:04] (03PS5) 10Vgutierrez: varnish: Add /esi/ URL rewrites [puppet] - 10https://gerrit.wikimedia.org/r/1366066 (https://phabricator.wikimedia.org/T443703) [12:49:31] (03CR) 10Vgutierrez: "varnishtests are happy: `0 tests failed, 0 tests skipped, 45 tests passed`" [puppet] - 10https://gerrit.wikimedia.org/r/1366066 (https://phabricator.wikimedia.org/T443703) (owner: 10Vgutierrez) [12:50:16] (03PS2) 10Btullis: wdqs: Remove the wdqs::test role [puppet] - 10https://gerrit.wikimedia.org/r/1364510 (https://phabricator.wikimedia.org/T443792) [12:53:12] (03CR) 10Btullis: [C:03+2] wdqs: Remove the wdqs::test role [puppet] - 10https://gerrit.wikimedia.org/r/1364510 (https://phabricator.wikimedia.org/T443792) (owner: 10Btullis) [12:54:02] !log btullis@cumin1004 START - Cookbook sre.hosts.provision for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:54:10] (03PS1) 10Ladsgroup: Thumbor: Remove VIPS for jpg [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366145 (https://phabricator.wikimedia.org/T220171) [12:54:25] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm [12:54:34] !log btullis@cumin1004 START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (push config only) [12:54:38] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (push config only) [12:54:52] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm [12:54:55] (03CR) 10AOkoth: "Thank you." [dns] - 10https://gerrit.wikimedia.org/r/1361944 (https://phabricator.wikimedia.org/T430113) (owner: 10AOkoth) [12:54:59] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm [12:55:02] !log btullis@cumin1004 START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (push config only) [12:55:06] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (push config only) [12:55:09] !log btullis@cumin1004 START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (push config only) [12:55:13] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (push config only) [12:55:14] (03CR) 10JMeybohm: [C:04-1] sre.k8s.upgrade-apiservers-inplace: hold kubernetes-node and kubernetes-master (032 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1363285 (https://phabricator.wikimedia.org/T437480) (owner: 10Jelto) [12:55:59] (03CR) 10Ladsgroup: [C:03+2] Thumbor: Remove VIPS for jpg [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366145 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup) [12:56:18] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888#12526906 (10MatthewVernon) Thanos-be2008 now has a copy, which is good: ` mvernon@thanos-be2008:~$ sudo sqlite3 --readonly /srv/swift-storage/accounts0/containers/20235/9e8/4f0ba9617366341a8939cbe7634ba9e8/4... [12:57:33] (03Merged) 10jenkins-bot: Thumbor: Remove VIPS for jpg [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366145 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup) [12:58:55] (03PS1) 10Muehlenhoff: Add FIDO key for daimona [puppet] - 10https://gerrit.wikimedia.org/r/1366153 (https://phabricator.wikimedia.org/T442762) [12:59:44] (03CR) 10CI reject: [V:04-1] Add FIDO key for daimona [puppet] - 10https://gerrit.wikimedia.org/r/1366153 (https://phabricator.wikimedia.org/T442762) (owner: 10Muehlenhoff) [13:00:37] !log ladsgroup@deploy2003 helmfile [staging] START helmfile.d/services/thumbor: apply [13:00:53] !log ladsgroup@deploy2003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [13:02:16] !log ladsgroup@deploy2003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [13:02:44] !log ladsgroup@deploy2003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [13:03:26] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:04:37] (03PS2) 10Muehlenhoff: Add FIDO key for daimona [puppet] - 10https://gerrit.wikimedia.org/r/1366153 (https://phabricator.wikimedia.org/T442762) [13:04:46] 07sre-alert-triage, 06Abstract Wikipedia team, 10Wikifunctions: Alert in need of triage: ProbeDown (instance wikifunctions-python-evaluator-staging:30443) - https://phabricator.wikimedia.org/T429678#12526972 (10cmassaro) >>! In T429678#12525583, @RLazarus wrote: > Alert's no longer firing -- looks like t... [13:05:06] (03PS2) 10Jelto: sre.k8s.upgrade-apiservers-inplace: hold kubernetes-node and kubernetes-master [cookbooks] - 10https://gerrit.wikimedia.org/r/1363285 (https://phabricator.wikimedia.org/T437480) [13:06:03] (03CR) 10Jelto: sre.k8s.upgrade-apiservers-inplace: hold kubernetes-node and kubernetes-master (032 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1363285 (https://phabricator.wikimedia.org/T437480) (owner: 10Jelto) [13:07:30] !log ladsgroup@deploy2003 helmfile [codfw] START helmfile.d/services/thumbor: apply [13:08:02] !log ladsgroup@deploy2003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [13:09:00] (03CR) 10Muehlenhoff: [C:03+1] "Verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1366153 (https://phabricator.wikimedia.org/T442762) (owner: 10Muehlenhoff) [13:09:02] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for daimona [puppet] - 10https://gerrit.wikimedia.org/r/1366153 (https://phabricator.wikimedia.org/T442762) (owner: 10Muehlenhoff) [13:09:38] (03PS11) 10Jelto: sre.k8s: add upgrade-kubelets-inplace cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1346036 (https://phabricator.wikimedia.org/T437480) [13:12:10] !log btullis@cumin1004 START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (remove config only) [13:12:14] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (remove config only) [13:12:47] !log btullis@cumin1004 START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (remove config only) [13:12:51] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (remove config only) [13:13:18] !log btullis@cumin1004 START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (remove config only) [13:13:22] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (remove config only) [13:14:31] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage [13:15:05] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage [13:15:28] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage [13:18:19] !log installing linux 6.12.115 on Trixie hosts [13:18:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:18:25] (03CR) 10MSantos: "This might need to be reverted so we can deal with T271001#12525392" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311062 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [13:18:40] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage [13:18:49] (03Abandoned) 10Kevin Bazira: airflow-ml: enable the urldownloader [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364448 (https://phabricator.wikimedia.org/T441987) (owner: 10Kevin Bazira) [13:22:46] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage [13:25:27] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host dse-k8s-worker2013.codfw.wmnet with OS bookworm [13:25:35] !log btullis@cumin1004 START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (push config only) [13:25:40] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (push config only) [13:25:42] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage [13:27:57] (03CR) 10Gehel: [C:03+2] admin: adding FIDO/SSH keys for dsaez [puppet] - 10https://gerrit.wikimedia.org/r/1365945 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [13:29:45] (03PS1) 10Gehel: admin: adding FIDO/SSH keys for ttaylor [puppet] - 10https://gerrit.wikimedia.org/r/1366188 (https://phabricator.wikimedia.org/T442756) [13:32:01] (03PS1) 10Muehlenhoff: Record LDAP access for mtozzi [puppet] - 10https://gerrit.wikimedia.org/r/1366189 [13:33:36] (03CR) 10Muehlenhoff: [C:03+2] Record LDAP access for mtozzi [puppet] - 10https://gerrit.wikimedia.org/r/1366189 (owner: 10Muehlenhoff) [13:34:28] (03CR) 10Jelto: [C:03+1] "lgtm" [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/1364546 (https://phabricator.wikimedia.org/T443455) (owner: 10JMeybohm) [13:35:42] (03CR) 10Klausman: [C:03+1] admin: adding FIDO/SSH keys for ttaylor [puppet] - 10https://gerrit.wikimedia.org/r/1366188 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [13:36:47] (03PS1) 10Muehlenhoff: Convert access for anakanishi from shell to LDAP [puppet] - 10https://gerrit.wikimedia.org/r/1366193 (https://phabricator.wikimedia.org/T442756) [13:37:20] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm [13:38:25] 10SRE-Access-Requests: Remove ANakanishi_WMF from production access - https://phabricator.wikimedia.org/T443913 (10Gehel) 03NEW [13:38:53] (03CR) 10Gehel: [C:03+2] admin: adding FIDO/SSH keys for ttaylor [puppet] - 10https://gerrit.wikimedia.org/r/1366188 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [13:42:15] !log btullis@cumin1004 START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (remove config only) [13:42:20] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (remove config only) [13:42:54] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm [13:43:28] (03PS1) 10Gehel: admin: remove non FIDO SSH keys for novemlinguae [puppet] - 10https://gerrit.wikimedia.org/r/1366198 (https://phabricator.wikimedia.org/T442756) [13:44:40] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage [13:45:32] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm [13:48:38] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage [13:50:45] (03CR) 10JMeybohm: [C:03+2] k8s.pool-depool-node: Fix repool message [cookbooks] - 10https://gerrit.wikimedia.org/r/1361391 (https://phabricator.wikimedia.org/T440696) (owner: 10JMeybohm) [13:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.19% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [13:52:46] RESOLVED: [2x] CertAlmostExpired: Certificate for service kubestagemaster2004:6443 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#kubestagemaster2004:6443 - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:54:06] (03Merged) 10jenkins-bot: k8s.pool-depool-node: Fix repool message [cookbooks] - 10https://gerrit.wikimedia.org/r/1361391 (https://phabricator.wikimedia.org/T440696) (owner: 10JMeybohm) [13:54:26] (03Abandoned) 10Hashar: ci: upgrade QEmu image from Bullseye to Trixie [puppet] - 10https://gerrit.wikimedia.org/r/1365786 (https://phabricator.wikimedia.org/T443872) (owner: 10Hashar) [13:58:12] FIRING: [6x] ProbeDown: Service kubestagemaster2003:6443 has failed probes (http_staging_codfw_kube_apiserver_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:00:18] (03CR) 10Btullis: [C:03+1] admin: remove non FIDO SSH keys for novemlinguae [puppet] - 10https://gerrit.wikimedia.org/r/1366198 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [14:00:36] (03PS4) 10JMeybohm: kubelet: --register-schedulable flag has been removed in 1.32 [puppet] - 10https://gerrit.wikimedia.org/r/1362069 (https://phabricator.wikimedia.org/T427069) [14:02:25] (03PS5) 10JMeybohm: kubelet: --register-schedulable flag has been removed in 1.32 [puppet] - 10https://gerrit.wikimedia.org/r/1362069 (https://phabricator.wikimedia.org/T427069) [14:02:32] (03CR) 10Huei Tan: ULS: Use the new language selector on Timeless on testwiki and mediawiki.org (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363037 (https://phabricator.wikimedia.org/T439715) (owner: 10Huei Tan) [14:02:34] (03CR) 10JMeybohm: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1362069 (https://phabricator.wikimedia.org/T427069) (owner: 10JMeybohm) [14:07:13] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet with OS bookworm [14:09:59] (03CR) 10MSantos: "Decision still needs to be made as we evaluate the situation." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311062 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [14:10:27] (03CR) 10JMeybohm: [C:03+2] Set Node.Spec.Unschedulable registering with unschedulable taint [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/1364546 (https://phabricator.wikimedia.org/T443455) (owner: 10JMeybohm) [14:10:30] (03CR) 10JMeybohm: [C:03+2] Update debian/ to gbp/pbuilder builds [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/1364330 (https://phabricator.wikimedia.org/T443455) (owner: 10JMeybohm) [14:10:34] (03CR) 10JMeybohm: [C:03+2] Add debian packaging from v1.34 branch [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/1364329 (https://phabricator.wikimedia.org/T443455) (owner: 10JMeybohm) [14:12:29] (03CR) 10Gehel: [C:03+2] admin: remove non FIDO SSH keys for novemlinguae [puppet] - 10https://gerrit.wikimedia.org/r/1366198 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [14:13:12] (03PS1) 10Muehlenhoff: Remove legacy key for sbisson [puppet] - 10https://gerrit.wikimedia.org/r/1366222 (https://phabricator.wikimedia.org/T442758) [14:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [14:14:46] (03PS1) 10JMeybohm: New upstream version 1.34.13 [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/1366223 (https://phabricator.wikimedia.org/T439103) [14:16:28] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy key for sbisson [puppet] - 10https://gerrit.wikimedia.org/r/1366222 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [14:16:40] (03CR) 10Daniel Kertesz: [C:03+1] "+1 (but I'm a varnish newbie)" [puppet] - 10https://gerrit.wikimedia.org/r/1366066 (https://phabricator.wikimedia.org/T443703) (owner: 10Vgutierrez) [14:17:08] PROBLEM - mysqld processes on dbstore1009 is CRITICAL: PROCS CRITICAL: 3 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [14:17:10] PROBLEM - MariaDB Replica Lag: staging on dbstore1009 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [14:17:26] PROBLEM - MariaDB Replica SQL: staging on dbstore1009 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [14:17:26] PROBLEM - MariaDB Replica IO: staging on dbstore1009 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [14:17:30] PROBLEM - MariaDB read only staging on dbstore1009 is CRITICAL: Could not connect to localhost:3350 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [14:18:27] (03CR) 10JMeybohm: [C:03+2] New upstream version 1.34.13 [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/1366223 (https://phabricator.wikimedia.org/T439103) (owner: 10JMeybohm) [14:19:40] (03PS1) 10Muehlenhoff: Add FIDO key for itamar [puppet] - 10https://gerrit.wikimedia.org/r/1366232 (https://phabricator.wikimedia.org/T442763) [14:22:47] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for itamar [puppet] - 10https://gerrit.wikimedia.org/r/1366232 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [14:26:36] (03PS1) 10Jgiannelos: thumbor: Disable async loading on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366245 [14:26:59] (03CR) 10Hashar: "Done" [puppet] - 10https://gerrit.wikimedia.org/r/1356233 (https://phabricator.wikimedia.org/T440177) (owner: 10Arnaudb) [14:27:28] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12527305 (10CWilliams-WMF) 05Open→03Resolved [14:27:29] (03CR) 10Jgiannelos: [C:03+2] thumbor: Disable async loading on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366245 (owner: 10Jgiannelos) [14:27:34] (03PS1) 10Muehlenhoff: Remove legacy keys for a-pizzata and daimona [puppet] - 10https://gerrit.wikimedia.org/r/1366246 (https://phabricator.wikimedia.org/T442762) [14:28:53] (03Merged) 10jenkins-bot: thumbor: Disable async loading on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366245 (owner: 10Jgiannelos) [14:30:08] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy keys for a-pizzata and daimona [puppet] - 10https://gerrit.wikimedia.org/r/1366246 (https://phabricator.wikimedia.org/T442762) (owner: 10Muehlenhoff) [14:30:33] !log jgiannelos@deploy2003 helmfile [staging] START helmfile.d/services/thumbor: apply [14:30:44] !log jgiannelos@deploy2003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [14:38:26] (03PS3) 10Gmodena: trafficserver: route WDQSv2 through REST Gateway [puppet] - 10https://gerrit.wikimedia.org/r/1363813 (https://phabricator.wikimedia.org/T443756) [14:40:21] (03CR) 10Gmodena: trafficserver: route WDQSv2 through REST Gateway (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363813 (https://phabricator.wikimedia.org/T443756) (owner: 10Gmodena) [14:40:26] (03PS2) 10Btullis: dse_k8s: Add four former wdqs hosts to the clusters [puppet] - 10https://gerrit.wikimedia.org/r/1364511 (https://phabricator.wikimedia.org/T443792) [14:40:49] (03PS5) 10Hashar: gerrit: fix the readonly plugin config [puppet] - 10https://gerrit.wikimedia.org/r/1356233 (https://phabricator.wikimedia.org/T440177) (owner: 10Arnaudb) [14:41:32] (03CR) 10Hashar: [C:03+1] "I have removed the misleading comment from the commit message:" [puppet] - 10https://gerrit.wikimedia.org/r/1356233 (https://phabricator.wikimedia.org/T440177) (owner: 10Arnaudb) [14:42:40] (03PS3) 10Hashar: gerrit: allow `gerrit index` in readonly plugin [puppet] - 10https://gerrit.wikimedia.org/r/1358246 (https://phabricator.wikimedia.org/T440177) [14:43:30] (03CR) 10Btullis: [C:03+2] dse_k8s: Add four former wdqs hosts to the clusters [puppet] - 10https://gerrit.wikimedia.org/r/1364511 (https://phabricator.wikimedia.org/T443792) (owner: 10Btullis) [14:47:33] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366275 [14:50:09] (03CR) 10Hashar: [C:03+1] "I wasn't aware of the `--reuse` command. I guess it addresses the concern I had 😊 Thx for having testing it!" [puppet] - 10https://gerrit.wikimedia.org/r/1358246 (https://phabricator.wikimedia.org/T440177) (owner: 10Hashar) [14:56:06] 10SRE-Access-Requests: Request to retire ppenloglou prod SSH access - https://phabricator.wikimedia.org/T443922 (10ppenloglou) 03NEW [14:57:20] (03PS1) 10Jgiannelos: Revert "thumbor: Disable async loading on staging" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366285 [14:57:32] (03CR) 10Jgiannelos: [C:03+2] Revert "thumbor: Disable async loading on staging" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366285 (owner: 10Jgiannelos) [14:59:11] (03Merged) 10jenkins-bot: Revert "thumbor: Disable async loading on staging" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366285 (owner: 10Jgiannelos) [15:00:03] !log jgiannelos@deploy2003 helmfile [staging] START helmfile.d/services/thumbor: apply [15:00:14] !log jgiannelos@deploy2003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [15:01:43] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1046.eqiad.wmnet [15:03:10] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366293 [15:03:11] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366294 [15:03:50] (03CR) 10Mark Bergsma: "Approved for Lucian to be added to ops-limited, but indeed let's merge with hardware backed keys." [puppet] - 10https://gerrit.wikimedia.org/r/1364304 (owner: 10Lucian) [15:04:20] 07Puppet, 06SRE, 06Release-Engineering-Team, 10Continuous-Integration-Config, 13Patch-For-Review: Upgrade operations/puppet CI from Bullseye to Bookworm - https://phabricator.wikimedia.org/T443798#12527456 (10Jdforrester-WMF) [15:06:20] (03CR) 10Jelto: [C:04-1] "two comments in-line" [puppet] - 10https://gerrit.wikimedia.org/r/1362069 (https://phabricator.wikimedia.org/T427069) (owner: 10JMeybohm) [15:09:14] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet [15:09:53] (03PS6) 10JMeybohm: kubelet: --register-schedulable flag has been removed in 1.32 [puppet] - 10https://gerrit.wikimedia.org/r/1362069 (https://phabricator.wikimedia.org/T427069) [15:10:36] (03CR) 10JMeybohm: kubelet: --register-schedulable flag has been removed in 1.32 (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1362069 (https://phabricator.wikimedia.org/T427069) (owner: 10JMeybohm) [15:11:25] !log ssastry@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [15:11:58] !log ssastry@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [15:14:10] (03CR) 10Jelto: [C:03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/1362069 (https://phabricator.wikimedia.org/T427069) (owner: 10JMeybohm) [15:16:22] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366310 [15:16:39] (03CR) 10Clément Goubert: trafficserver: route WDQSv2 through REST Gateway (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363813 (https://phabricator.wikimedia.org/T443756) (owner: 10Gmodena) [15:17:22] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1295888 (owner: 10PipelineBot) [15:17:28] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1304792 (owner: 10PipelineBot) [15:17:31] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306262 (owner: 10PipelineBot) [15:17:35] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326266 (owner: 10PipelineBot) [15:17:38] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1328581 (owner: 10PipelineBot) [15:17:42] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1330403 (owner: 10PipelineBot) [15:17:45] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341844 (owner: 10PipelineBot) [15:17:49] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341980 (owner: 10PipelineBot) [15:17:55] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344776 (owner: 10PipelineBot) [15:17:58] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345998 (owner: 10PipelineBot) [15:18:01] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346586 (owner: 10PipelineBot) [15:18:05] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1350394 (owner: 10PipelineBot) [15:18:09] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352012 (owner: 10PipelineBot) [15:18:13] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352290 (owner: 10PipelineBot) [15:18:16] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360397 (owner: 10PipelineBot) [15:18:20] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366275 (owner: 10PipelineBot) [15:18:23] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366293 (owner: 10PipelineBot) [15:18:36] (03Abandoned) 10Jforrester: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366294 (owner: 10PipelineBot) [15:18:43] (03CR) 10JMeybohm: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1362069 (https://phabricator.wikimedia.org/T427069) (owner: 10JMeybohm) [15:19:11] (03Abandoned) 10Jforrester: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359904 (owner: 10PipelineBot) [15:19:15] (03Abandoned) 10Jforrester: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360499 (owner: 10PipelineBot) [15:19:18] (03Abandoned) 10Jforrester: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361531 (owner: 10PipelineBot) [15:19:21] (03Abandoned) 10Jforrester: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364126 (owner: 10PipelineBot) [15:19:25] (03Abandoned) 10Jforrester: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364325 (owner: 10PipelineBot) [15:20:02] (03Abandoned) 10Jforrester: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357352 (owner: 10PipelineBot) [15:20:06] (03Abandoned) 10Jforrester: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353471 (owner: 10PipelineBot) [15:20:08] (03Abandoned) 10Jforrester: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352011 (owner: 10PipelineBot) [15:20:10] (03Abandoned) 10Jforrester: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1350395 (owner: 10PipelineBot) [15:20:13] (03Abandoned) 10Jforrester: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346593 (owner: 10PipelineBot) [15:20:56] James_F: if you feel adventurous, you could even delete those obsolete changes :] [15:21:05] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1047.eqiad.wmnet [15:21:59] which makes me wonder whether that pipeline promote bot could maybe reuse an existing change when one exists already [15:22:24] hashar: Yeah, or at least use a unique gerrit topic per service, so the clashes can be auto-discovered and dropped. [15:22:28] anyway that is not that many change given the 1.3M changes we have in the Gerrit db [15:22:38] yeah or a hashtag :) [15:22:41] Pipeline-promote-citoid not just pipeline-promote. [15:23:02] But eh, we’re trying to move everything using pipeline off gerrit and onto GitLab, so… [15:27:17] (03CR) 10JMeybohm: [C:03+2] kubelet: --register-schedulable flag has been removed in 1.32 [puppet] - 10https://gerrit.wikimedia.org/r/1362069 (https://phabricator.wikimedia.org/T427069) (owner: 10JMeybohm) [15:28:31] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet [15:31:00] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1048.eqiad.wmnet [15:32:08] (03PS1) 10JMeybohm: Revert^2 "Update staging-codfw to kubernetes 1.34" [puppet] - 10https://gerrit.wikimedia.org/r/1366331 (https://phabricator.wikimedia.org/T427069) [15:33:14] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366335 [15:33:32] 07Puppet, 06SRE, 06Release-Engineering-Team, 10Continuous-Integration-Config, 13Patch-For-Review: Upgrade operations/puppet CI from Bullseye to Bookworm - https://phabricator.wikimedia.org/T443798#12527623 (10hashar) [15:38:35] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet [15:40:31] !log btullis@puppetserver1001 conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet [15:40:42] !log btullis@puppetserver1001 conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet [15:40:51] !log btullis@puppetserver1001 conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet [15:40:55] !log btullis@puppetserver1001 conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet [15:41:03] !log btullis@puppetserver1001 conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet [15:41:10] !log btullis@puppetserver1001 conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet [15:41:34] !log btullis@puppetserver1001 conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2013.codfw.wmnet [15:41:54] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2013.codfw.wmnet [15:45:05] (03PS3) 10JMeybohm: sre.k8s.upgrade-apiservers-inplace: hold kubernetes-node and kubernetes-master [cookbooks] - 10https://gerrit.wikimedia.org/r/1363285 (https://phabricator.wikimedia.org/T437480) (owner: 10Jelto) [15:45:06] (03PS1) 10JMeybohm: sre.k8s.upgrade-apiservers-inplace: Validate systemctl services [cookbooks] - 10https://gerrit.wikimedia.org/r/1366351 (https://phabricator.wikimedia.org/T437480) [15:48:22] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366357 [15:48:35] (03CR) 10Ahmon Dancy: "@jhathaway@wikimedia.org Are you ok with this?" [puppet] - 10https://gerrit.wikimedia.org/r/1361577 (https://phabricator.wikimedia.org/T443373) (owner: 10Ahmon Dancy) [15:49:31] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet [15:54:26] (03CR) 10Gmodena: trafficserver: route WDQSv2 through REST Gateway (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363813 (https://phabricator.wikimedia.org/T443756) (owner: 10Gmodena) [15:58:12] RESOLVED: [6x] ProbeDown: Service kubestagemaster2003:6443 has failed probes (http_staging_codfw_kube_apiserver_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:01:18] (03PS1) 10Gehel: admin: adding FIDO/SSH keys for rmaung [puppet] - 10https://gerrit.wikimedia.org/r/1366363 (https://phabricator.wikimedia.org/T442756) [16:01:40] 10ops-codfw, 06SRE, 06DC-Ops: Degraded RAID on dse-k8s-wdqs-test2001 - https://phabricator.wikimedia.org/T442163#12527746 (10Jhancock.wm) 05Open→03Declined [16:03:05] !log jayme@cumin1004 START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13 [16:03:15] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet [16:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:04:59] (03CR) 10Btullis: [C:03+1] admin: adding FIDO/SSH keys for rmaung [puppet] - 10https://gerrit.wikimedia.org/r/1366363 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [16:05:35] (03CR) 10Gehel: [C:03+2] admin: adding FIDO/SSH keys for rmaung [puppet] - 10https://gerrit.wikimedia.org/r/1366363 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [16:06:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [16:07:08] !log jayme@cumin1004 END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet [16:07:11] !log jayme@cumin1004 END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.13 [16:07:16] !log jayme@cumin1004 START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13 [16:07:26] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet [16:07:28] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet [16:07:41] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366366 [16:08:44] !log jayme@cumin1004 END (ERROR) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=97) Upgrade apiservers of staging-codfw to 1.34.13 [16:08:50] !log jayme@cumin1004 START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13 [16:09:00] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet [16:09:02] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet [16:09:31] (03CR) 10JMeybohm: [C:03+2] Revert^2 "Update staging-codfw to kubernetes 1.34" [puppet] - 10https://gerrit.wikimedia.org/r/1366331 (https://phabricator.wikimedia.org/T427069) (owner: 10JMeybohm) [16:09:34] (03PS1) 10Gehel: admin: adding FIDO/SSH keys for pcoombe [puppet] - 10https://gerrit.wikimedia.org/r/1366370 (https://phabricator.wikimedia.org/T442756) [16:11:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [16:11:34] (03CR) 10Btullis: [C:03+1] admin: adding FIDO/SSH keys for pcoombe [puppet] - 10https://gerrit.wikimedia.org/r/1366370 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [16:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:12:04] jayme@cumin1004 upgrade-apiservers-inplace (PID 1031852) is awaiting input [16:12:14] (03CR) 10JMeybohm: [C:04-1] sre.k8s: Add cookbook for apiserver inplace upgrades (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1358223 (https://phabricator.wikimedia.org/T437480) (owner: 10JMeybohm) [16:12:37] (03CR) 10Gehel: [C:03+2] admin: adding FIDO/SSH keys for pcoombe [puppet] - 10https://gerrit.wikimedia.org/r/1366370 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [16:15:20] (03CR) 10JMeybohm: [C:04-1] sre.k8s: Add cookbook for apiserver inplace upgrades (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1358223 (https://phabricator.wikimedia.org/T437480) (owner: 10JMeybohm) [16:15:40] (03PS1) 10Muehlenhoff: Add FIDO key for damilare [puppet] - 10https://gerrit.wikimedia.org/r/1366375 (https://phabricator.wikimedia.org/T442758) [16:16:14] (03PS1) 10Dr0ptp4kt: Remove old non-yubikey prod ssh pub key for dr0ptp4kt [puppet] - 10https://gerrit.wikimedia.org/r/1366376 [16:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:18:05] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet [16:18:08] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet [16:18:11] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.13 [16:18:57] (03CR) 10Dr0ptp4kt: "Thanks in advance! Have a good Friday and weekend, and a fun next week." [puppet] - 10https://gerrit.wikimedia.org/r/1366376 (owner: 10Dr0ptp4kt) [16:20:49] (03CR) 10Lizimedia: [C:03+2] Add prometheus-atlas-exporter for bookworm and staging [puppet] - 10https://gerrit.wikimedia.org/r/1361580 (https://phabricator.wikimedia.org/T429672) (owner: 10Lizimedia) [16:20:51] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366378 [16:21:31] (03PS1) 10Muehlenhoff: Add second FIDO key for andrewtavis-wmde [puppet] - 10https://gerrit.wikimedia.org/r/1366379 (https://phabricator.wikimedia.org/T442763) [16:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:21:45] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for damilare [puppet] - 10https://gerrit.wikimedia.org/r/1366375 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [16:22:18] (03Abandoned) 10Dbrant: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366335 (owner: 10PipelineBot) [16:22:22] (03Abandoned) 10Dbrant: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366357 (owner: 10PipelineBot) [16:22:27] (03Abandoned) 10Dbrant: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366366 (owner: 10PipelineBot) [16:24:59] (03CR) 10Muehlenhoff: [C:03+2] Add second FIDO key for andrewtavis-wmde [puppet] - 10https://gerrit.wikimedia.org/r/1366379 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [16:28:25] (03CR) 10Muehlenhoff: [C:03+1] "Thanks for making a patch" [puppet] - 10https://gerrit.wikimedia.org/r/1366376 (owner: 10Dr0ptp4kt) [16:28:30] (03CR) 10Muehlenhoff: [C:03+2] Remove old non-yubikey prod ssh pub key for dr0ptp4kt [puppet] - 10https://gerrit.wikimedia.org/r/1366376 (owner: 10Dr0ptp4kt) [16:29:43] !log jayme@cumin1004 START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13 [16:29:49] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2003.codfw.wmnet [16:29:50] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2003.codfw.wmnet [16:30:01] (03CR) 10JMeybohm: [C:03+1] sre.k8s: add upgrade-kubelets-inplace cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1346036 (https://phabricator.wikimedia.org/T437480) (owner: 10Jelto) [16:30:28] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2003.codfw.wmnet [16:30:30] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2003.codfw.wmnet [16:30:35] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2004.codfw.wmnet [16:30:36] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2004.codfw.wmnet [16:31:14] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2004.codfw.wmnet [16:31:16] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2004.codfw.wmnet [16:31:21] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet [16:31:22] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet [16:31:59] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet [16:32:01] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet [16:32:01] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13 [16:33:07] (03CR) 10JMeybohm: [C:04-1] sre.k8s: add upgrade-kubelets-inplace cookbook (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1346036 (https://phabricator.wikimedia.org/T437480) (owner: 10Jelto) [16:33:34] (03CR) 10Jelto: sre.k8s: add upgrade-kubelets-inplace cookbook (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1346036 (https://phabricator.wikimedia.org/T437480) (owner: 10Jelto) [16:34:48] (03PS1) 10Muehlenhoff: Add FIDO key for cparle [puppet] - 10https://gerrit.wikimedia.org/r/1366390 (https://phabricator.wikimedia.org/T442758) [16:36:46] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for cparle [puppet] - 10https://gerrit.wikimedia.org/r/1366390 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [16:37:55] !log jayme@cumin1004 START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13 [16:38:02] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet [16:43:05] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet [16:43:30] (03PS1) 10Muehlenhoff: Remove shell access for ppenloglou [puppet] - 10https://gerrit.wikimedia.org/r/1366397 (https://phabricator.wikimedia.org/T443922) [16:43:41] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet [16:43:43] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet [16:43:43] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13 [16:44:56] (03CR) 10CI reject: [V:04-1] Remove shell access for ppenloglou [puppet] - 10https://gerrit.wikimedia.org/r/1366397 (https://phabricator.wikimedia.org/T443922) (owner: 10Muehlenhoff) [16:46:10] (03PS2) 10Muehlenhoff: Remove shell access for ppenloglou [puppet] - 10https://gerrit.wikimedia.org/r/1366397 (https://phabricator.wikimedia.org/T443922) [16:48:00] (03CR) 10Muehlenhoff: [C:03+2] Remove shell access for ppenloglou [puppet] - 10https://gerrit.wikimedia.org/r/1366397 (https://phabricator.wikimedia.org/T443922) (owner: 10Muehlenhoff) [16:48:48] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Request to retire ppenloglou prod SSH access - https://phabricator.wikimedia.org/T443922#12528069 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff Shell access has been retired. [16:51:19] (03CR) 10MusikAnimal: [C:03+1] admin: adding FIDO/SSH keys for sd [puppet] - 10https://gerrit.wikimedia.org/r/1365868 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [16:53:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [16:54:53] (03PS1) 10Nathillard: wmf-navigator: update container images to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366410 (https://phabricator.wikimedia.org/T443937) [16:55:49] (03CR) 10Mforns: "Let's discuss about passing the estimate bot/human label instead of the estimate score insite the header." [puppet] - 10https://gerrit.wikimedia.org/r/1363149 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [16:58:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [17:03:18] (03CR) 10Jelto: wmf-navigator: update container images to newest version (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366410 (https://phabricator.wikimedia.org/T443937) (owner: 10Nathillard) [17:04:57] (03CR) 10Ottomata: "> increased load during the switchover" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362092 (https://phabricator.wikimedia.org/T443442) (owner: 10Ottomata) [17:05:38] !log jayme@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{kubestagemaster2005.codfw.wmnet} and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw) [17:05:41] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet [17:05:43] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet [17:09:31] (03PS1) 10Nathillard: updating other images [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366438 [17:09:33] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [17:10:44] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet [17:10:46] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet [17:10:46] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{kubestagemaster2005.codfw.wmnet} and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw) [17:14:33] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [17:19:46] (03PS1) 10Muehlenhoff: Add FIDO key for sfaci [puppet] - 10https://gerrit.wikimedia.org/r/1366455 (https://phabricator.wikimedia.org/T442758) [17:21:06] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for sfaci [puppet] - 10https://gerrit.wikimedia.org/r/1366455 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [17:33:31] (03Abandoned) 10Nathillard: updating other images [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366438 (owner: 10Nathillard) [17:38:57] (03PS2) 10Nathillard: wmf-navigator: update container images to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366410 (https://phabricator.wikimedia.org/T443937) [17:48:38] (03CR) 10SBassett: wmf-navigator: update container images to newest version (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366410 (https://phabricator.wikimedia.org/T443937) (owner: 10Nathillard) [17:49:11] (03CR) 10SBassett: [C:03+2] wmf-navigator: update container images to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366410 (https://phabricator.wikimedia.org/T443937) (owner: 10Nathillard) [17:50:43] (03Merged) 10jenkins-bot: wmf-navigator: update container images to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366410 (https://phabricator.wikimedia.org/T443937) (owner: 10Nathillard) [17:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.17% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [17:55:34] (03PS1) 10Bartosz Dziewoński: mediawiki.org/keys: Change recommended import command [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1366494 [18:01:31] (03CR) 10Bartosz Dziewoński: "We probably should keep the keys, but you can correct the name and email in the .html and .txt files. They're just informational not used " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/981583 (owner: 10Brion VIBBER) [18:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [18:16:08] (03CR) 10Jsn.sherman: [C:03+1] "lgtm; thanks for your work!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356990 (https://phabricator.wikimedia.org/T441729) (owner: 10Dillon) [18:18:34] (03CR) 10ArielGlenn: "There's a phab task about this too, see https://phabricator.wikimedia.org/T405165" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1366494 (owner: 10Bartosz Dziewoński) [18:25:28] (03PS3) 10Southparkfan: profile::pki::client: Properly update system trust store [puppet] - 10https://gerrit.wikimedia.org/r/1363421 (https://phabricator.wikimedia.org/T436562) [18:28:17] !log aokoth@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply [18:29:01] !log aokoth@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply [18:29:23] !log aokoth@deploy2003 helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply [18:30:12] !log aokoth@deploy2003 helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply [18:41:25] (03PS1) 10KineticPelagic: REST: Correct test server hostname to test.wikipedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1366554 (https://phabricator.wikimedia.org/T408816) [18:46:17] (03CR) 10ArielGlenn: "The option is only available in gpg 2.59 and later, which isn't readily available. Sigh." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1366494 (owner: 10Bartosz Dziewoński) [18:54:52] 10SRE-Access-Requests, 10LDAP-Access-Requests, 06Wikimedia Enterprise: Requesting Ops level access to the 'platform_eng' Airflow Instance for Federico Ceratto - https://phabricator.wikimedia.org/T443966#12528728 (10FCeratto-WMF) [19:27:09] 06SRE, 10LDAP-Access-Requests: Grant Access to nda LDAP group for Monrac5 - https://phabricator.wikimedia.org/T443971 (10Monrac5) 03NEW [19:50:42] 06SRE: Upgrade deps to Trixie - https://phabricator.wikimedia.org/T439497#12528894 (10Aklapper) [20:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:10:36] PROBLEM - MariaDB Replica Lag: s1 on db2250 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 621.22 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:32:16] PROBLEM - MegaRAID on an-worker1151 is CRITICAL: CRITICAL: 11 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [21:01:36] RECOVERY - MariaDB Replica Lag: s1 on db2250 is OK: OK slave_sql_lag Replication lag: 0.21 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:03:39] (03PS1) 10Pppery: Update translations [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1366738 [21:10:47] (03PS1) 10Pppery: Update source strings an extra time for good measure [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1366746 [21:11:02] (03PS2) 10Pppery: Update translations [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1366738 [21:11:06] (03PS2) 10Pppery: Update source strings an extra time for good measure [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1366746 [21:11:23] (03CR) 10Jdrewniak: Enable ReadingLists custom lists on all Wikipedias. (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364782 (https://phabricator.wikimedia.org/T441372) (owner: 10Jdrewniak) [21:42:30] PROBLEM - MariaDB Replica Lag: s4 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 654.21 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.27% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [21:58:36] PROBLEM - MariaDB Replica Lag: x4 on db2250 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 656.22 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:12:50] PROBLEM - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1194 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 1 Failed : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [22:12:52] ACKNOWLEDGEMENT - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1194 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 1 Failed : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T443980 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [22:12:58] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1194 - https://phabricator.wikimedia.org/T443980 (10ops-monitoring-bot) 03NEW [22:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [22:15:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.333% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [22:20:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.333% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [22:22:02] (03PS1) 10Dzahn: codesearch: tolerate git clones removed during delete-zombie-locks [puppet] - 10https://gerrit.wikimedia.org/r/1366842 (https://phabricator.wikimedia.org/T443874) [22:24:16] (03CR) 10Dzahn: [C:03+2] codesearch: tolerate git clones removed during delete-zombie-locks [puppet] - 10https://gerrit.wikimedia.org/r/1366842 (https://phabricator.wikimedia.org/T443874) (owner: 10Dzahn) [22:25:18] lizimedia: hi, would you like to merge or wait on the puppetmaster? [22:25:49] mutante: merge what, sorry? [22:26:02] lizimedia: Lizimedia: Add prometheus-atlas-exporter for bookworm and staging (bda8fd4c3e) [22:26:32] on puppetserver1001 at puppet-merge [22:27:12] is there a manual step I need to do? All I did was hit the merge button in gerrit [22:27:50] lizimedia: yes, there is. after merging changes in operations/puppet in gerrit you are expected to go to a puppetserver and type "puppet-merge" and review / approve it [22:28:10] every once in a while we have a conflict when we merge right after another [22:28:14] ugh I'm sorry [22:28:15] then we ping each other like this [22:28:28] no worries at all [22:28:52] want me to run it or try yourself right away? [22:29:10] I am only asking because I have an unrelated change. but there is no rush. [22:29:13] do whatever you need to unblock yourself, it turns out I missed a file anyway [22:30:52] looking at the diff, determining worst it can break is new packages dont build yet. deploying [22:31:25] the puppet-merge command is now syincing it to the other puppetservers [22:31:51] and then the next puppet agent run on the apt* servers applies the APT repo config change [22:32:15] you can either just wait or run puppet agent on the apt* now to see yourself [22:33:49] I think I needed to update `modules/aptrepo/files/updates` too so it won't do anything [22:34:23] all is clear now, and clearing up this type of "merge conflict" is not uncommon at all [22:35:06] so I need to manually run puppet-merge after every change that gets merged? [22:35:18] some repos fully self-deploy when hitting +2 and CI submits, but operations/puppet is different like that [22:35:23] yes [22:35:30] kk [22:35:48] merge in gerrit -> merge on master -> run a client against master and see the change with your own eyes [22:35:53] ideally [22:36:21] well, depending how complex it is I guess [22:37:58] hmm, no gerrit bot [22:38:43] meaning.. normally we see right here when stuff gets merged. but now we didnt. [22:43:55] the puppetserver1001 MOTD says the last puppet commit was 6dee072925 [22:44:30] RECOVERY - MariaDB Replica Lag: s4 on db2239 is OK: OK slave_sql_lag Replication lag: 0.03 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:44:49] (03PS1) 10SBassett: Content Security Policy: add stricter report-only policy [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1366869 (https://phabricator.wikimedia.org/T419612) [22:45:03] that's before you, and I was already after you: https://gerrit.wikimedia.org/r/q/project:operations/puppet+status:merged [22:45:19] but how often is the motd updated [22:45:26] lets see git log in the real git dir [22:46:08] (03CR) 10SBassett: [C:04-2] "Hold for future config deploy." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1366869 (https://phabricator.wikimedia.org/T419612) (owner: 10SBassett) [22:46:36] RECOVERY - MariaDB Replica Lag: x4 on db2250 is OK: OK slave_sql_lag Replication lag: 0.22 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:47:06] [puppetserver1001:~] $ sudo git -C /srv/git/operations/puppet log --oneline -20 | head -n3 [22:47:09] 445d92e3b2 codesearch: tolerate git clones removed during delete-zombie-locks [22:47:12] bda8fd4c3e Add prometheus-atlas-exporter for bookworm and staging [22:47:14] 6dee072925 Add FIDO key for sfaci [22:47:28] lizimedia: ^ puppetserver and gerrit agree, just the motd is behind [22:47:33] ah okay [22:48:10] I don't see anyone in here named gerrit-anything [22:48:52] my first thought is marked as "obsolete". https://wikitech.wikimedia.org/wiki/Obsolete:Grrrit-wm [22:49:12] https://wikitech.wikimedia.org/wiki/Gerrit/Phabricator_integration [22:49:23] it's the gerrit plugin.. hrmm [22:50:01] no, this is not the IRC bot.. hold on [22:52:08] we want "the gerrit job of the wikibugs tool on Toolforge" [22:52:49] ever used toolforge.org ? [22:52:52] gerrit isn't really involved at this point right? [22:53:13] mutante: no! only tangentially aware of it [22:53:52] it should not be, this IRC bot running on toolforge should read stream-events from Gerrit [22:55:27] lizimedia: let me just ask in #wikimedia-cloud for someone to restart it. there is a "become" command there. [23:03:27] (03PS2) 10Dzahn: codesearch: mount the hound gitconfig as /etc/gitconfig on codesearch2001 [puppet] - 10https://gerrit.wikimedia.org/r/1365703 (https://phabricator.wikimedia.org/T268199) [23:04:47] lizimedia: pinging that channel worked. and wikibugs looks back ^ [23:05:44] a mystery! [23:06:39] 23:02 < lucaswerkmeister> ERROR: JobsCliError: Failed to create a job, likely an internal bug in the jobs framework. [23:06:42] 23:03 < lucaswerkmeister> ERROR: Please report this issue to the Toolforge admins if it persists: https://w.wiki/6Zuu [23:06:45] 23:03 < lucaswerkmeister> mutante: should I just try a `kubectl rollout restart deployment gerrit`? [23:06:48] this :) [23:08:52] (03CR) 10Dzahn: [C:03+2] codesearch: mount the hound gitconfig as /etc/gitconfig on codesearch2001 [puppet] - 10https://gerrit.wikimedia.org/r/1365703 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [23:15:04] (03PS1) 10Dzahn: codesearch: keep codesearch1001 on Wikimedia-hosted repos for now [puppet] - 10https://gerrit.wikimedia.org/r/1366876 (https://phabricator.wikimedia.org/T268199) [23:16:14] (03CR) 10Dzahn: [C:03+2] "re-enabling puppet on codesearch1001" [puppet] - 10https://gerrit.wikimedia.org/r/1366876 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [23:22:30] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 654.16 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [23:50:03] FIRING: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [23:55:03] RESOLVED: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [23:57:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures