[00:00:05] !log root@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: sync [00:00:19] !log root@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: sync [00:00:22] !log root@deploy2003 helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: sync [00:00:31] !log root@deploy2003 helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: sync [00:00:37] !log root@deploy2003 helmfile [eqiad] START helmfile.d/services/shellbox: sync [00:00:41] (03PS2) 10Dduvall: mathoid: Bump from 2026-10-27-201041-production to 2026-11-19-232014-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371148 (https://phabricator.wikimedia.org/T445371) [00:01:02] !log root@deploy2003 helmfile [eqiad] DONE helmfile.d/services/shellbox: sync [00:01:05] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox: sync [00:01:24] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox: sync [00:01:29] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [00:01:39] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [00:01:42] !log root@deploy2003 helmfile [eqiad] START helmfile.d/services/shellbox-timeline: sync [00:01:59] !log root@deploy2003 helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: sync [00:02:02] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [00:02:13] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [00:02:16] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [00:02:21] !log Running `/usr/local/bin/foreachwikiindblist "mediamoderation-continuous-scan.dblist" extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` for T439783 [00:02:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:02:25] T439783: MediaModerationImageContentsLookup uses original file mime type not the thumbnail mime type - https://phabricator.wikimedia.org/T439783 [00:04:19] !log Stopped that run [00:04:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:04:29] !log Running `/usr/local/bin/foreachwikiindblist "mediamoderation-continuous-scan.dblist" extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=15 --verbose` for T439783 in a tmux session [00:04:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [00:11:10] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:23:49] (03PS1) 10Dillon: admin: Add FIDO SSH keys for dhardy [puppet] - 10https://gerrit.wikimedia.org/r/1372260 (https://phabricator.wikimedia.org/T445598) [00:28:56] PROBLEM - Blazegraph Port for wdqs-blazegraph on wdqs1022 is CRITICAL: connect to address 127.0.0.1 and port 9999: Connection refused https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [00:29:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12585543 (10phaultfinder) [00:30:56] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs1022 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [00:38:22] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for dhardy to restore deployment access - https://phabricator.wikimedia.org/T445598#12585564 (10Dillon) [00:39:08] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1372261 [00:39:08] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1372261 (owner: 10TrainBranchBot) [00:44:30] (03PS2) 10Dillon: admin: Add FIDO SSH keys for dhardy [puppet] - 10https://gerrit.wikimedia.org/r/1372260 (https://phabricator.wikimedia.org/T445598) [00:46:33] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1372261 (owner: 10TrainBranchBot) [00:50:30] FIRING: KubernetesDeploymentUnavailableReplicas: ... [00:50:30] Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s&var-namespace=editcheck-headless&var-deployment=editcheck-headless-main - ... [00:50:30] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [01:09:28] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1372279 [01:09:28] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1372279 (owner: 10TrainBranchBot) [01:16:14] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1372279 (owner: 10TrainBranchBot) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:19:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12585747 (10phaultfinder) [02:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:36:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [02:37:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:50:18] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [02:53:58] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1033.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [02:54:04] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1034.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1033.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [02:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [02:55:58] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [02:56:04] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:14:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12585796 (10phaultfinder) [03:14:50] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 22d 15h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [03:17:46] (03PS1) 10Dwisehaupt: Shift frbast alias to codfw bastion [dns] - 10https://gerrit.wikimedia.org/r/1372373 [03:19:09] (03CR) 10Dwisehaupt: [C:03+2] Shift frbast alias to codfw bastion [dns] - 10https://gerrit.wikimedia.org/r/1372373 (owner: 10Dwisehaupt) [03:19:39] !log dwisehaupt@dns1004 START - running authdns-update [03:21:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [03:21:59] !log dwisehaupt@dns1004 END - running authdns-update [03:30:05] FIRING: HelmReleaseBadStatus: Helm release wdqs/main-internal on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [03:39:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12585834 (10phaultfinder) [03:48:14] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2278.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2148.codfw.wmnet, wikikube-worker2033.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-work [03:48:14] odfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2276.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2358.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-worker2177.codfw.wmnet, w [03:48:14] worker2325.codfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker2273.codfw.wmnet, wikikube-worker2213.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [03:49:14] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:54:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12585837 (10phaultfinder) [03:57:37] (03CR) 10Kevin Bazira: [C:03+2] ml-serve: bump cpu for semantic-highlighting-staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371871 (owner: 10DCausse) [03:59:22] (03Merged) 10jenkins-bot: ml-serve: bump cpu for semantic-highlighting-staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371871 (owner: 10DCausse) [04:02:15] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [04:03:10] !log kevinbazira@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [04:03:16] !log kevinbazira@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' . [04:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [04:06:24] (03CR) 10Kevin Bazira: [C:03+2] "Thank you for working on this David :)" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371871 (owner: 10DCausse) [04:11:10] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:24:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12585871 (10phaultfinder) [04:40:00] !log ssastry@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [04:41:19] !log ssastry@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [04:50:30] FIRING: KubernetesDeploymentUnavailableReplicas: ... [04:50:30] Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s&var-namespace=editcheck-headless&var-deployment=editcheck-headless-main - ... [04:50:30] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [05:15:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.72% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:20:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.72% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:31:16] PROBLEM - Blazegraph Port for wdqs-blazegraph on wdqs1020 is CRITICAL: connect to address 127.0.0.1 and port 9999: Connection refused https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [05:32:14] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs1020 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [05:49:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12585937 (10phaultfinder) [06:17:16] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for dhardy to restore deployment access - https://phabricator.wikimedia.org/T445598#12585966 (10Marostegui) p:05Triage→03Medium a:03Marostegui Waiting for manager approval and out of band verification, [06:24:16] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for toyofuku - https://phabricator.wikimedia.org/T445548#12585996 (10Marostegui) Thanks! Would you mind also your manager to rubber stamp this request? Thank you! [06:24:34] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for toyofuku - https://phabricator.wikimedia.org/T445548#12585998 (10Marostegui) 05Stalled→03Open [06:26:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.88% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:26:15] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2157.codfw.wmnet with reason: Maintenance [06:31:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.88% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:31:34] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1159.eqiad.wmnet with reason: Maintenance [06:31:36] !log marostegui@cumin1004 dbctl commit (dc=all): 'Depooling db1159 (T445307)', diff saved to https://phabricator.wikimedia.org/P97625 and previous config saved to /var/cache/conftool/dbconfig/20261120-063135-marostegui.json [06:31:39] T445307: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307 [06:34:57] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1159 (T445307)', diff saved to https://phabricator.wikimedia.org/P97626 and previous config saved to /var/cache/conftool/dbconfig/20261120-063456-marostegui.json [06:37:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:42:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.36% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:43:50] 06SRE, 10SRE-Access-Requests, 06Product Safety and Integrity, 06Security-Team, 07SecTeam-Processed: Request IRC #mediawiki_security access for Novem Linguae - https://phabricator.wikimedia.org/T442747#12586037 (10Novem_Linguae) Thanks! [06:47:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 19.82% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:48:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.57% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:49:59] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97627 and previous config saved to /var/cache/conftool/dbconfig/20261120-064959-marostegui.json [06:50:18] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [06:57:04] (03CR) 10Marostegui: "That's exactly what I meant, to run a PCC in general to make sure this won't break anything else that we didn't have in mind. You can run " [puppet] - 10https://gerrit.wikimedia.org/r/1371837 (owner: 10Ladsgroup) [06:58:38] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 25 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364354 (https://phabricator.wikimedia.org/T439718) (owner: 10Huei Tan) [06:58:46] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 23 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1370984 (https://phabricator.wikimedia.org/T123063) (owner: 10Abijeet Patro) [06:59:33] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 25 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363074 (https://phabricator.wikimedia.org/T439718) (owner: 10Huei Tan) [07:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261120T0700) [07:00:40] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 26 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369726 (https://phabricator.wikimedia.org/T439851) (owner: 10Huei Tan) [07:02:00] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 25 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362108 (https://phabricator.wikimedia.org/T442735) (owner: 10Kosta Harlan) [07:03:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.91% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:04:38] 10ops-eqiad, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T445619 (10phaultfinder) 03NEW [07:05:01] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97628 and previous config saved to /var/cache/conftool/dbconfig/20261120-070501-marostegui.json [07:07:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.32% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:12:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.78% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:14:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.52% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:14:50] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 22d 11h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [07:19:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.1% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:20:04] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1159 (T445307)', diff saved to https://phabricator.wikimedia.org/P97629 and previous config saved to /var/cache/conftool/dbconfig/20261120-072003-marostegui.json [07:20:07] T445307: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307 [07:20:19] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1161.eqiad.wmnet with reason: Maintenance [07:20:21] !log marostegui@cumin1004 dbctl commit (dc=all): 'Depooling db1161 (T445307)', diff saved to https://phabricator.wikimedia.org/P97630 and previous config saved to /var/cache/conftool/dbconfig/20261120-072021-marostegui.json [07:23:40] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1161 (T445307)', diff saved to https://phabricator.wikimedia.org/P97631 and previous config saved to /var/cache/conftool/dbconfig/20261120-072340-marostegui.json [07:28:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.43% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:30:05] FIRING: HelmReleaseBadStatus: Helm release wdqs/main-internal on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [07:37:24] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1371859 (https://phabricator.wikimedia.org/T445375) (owner: 10Marostegui) [07:37:40] (03CR) 10Marostegui: [C:03+2] data.yaml: Add daimona to analytics-privatedata [puppet] - 10https://gerrit.wikimedia.org/r/1371859 (https://phabricator.wikimedia.org/T445375) (owner: 10Marostegui) [07:38:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.21% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:38:34] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for daimona - https://phabricator.wikimedia.org/T445375#12586114 (10Marostegui) 05Open→03Resolved Change has been merged. Please allow 20 minutes for the change to spread across all the servers. [07:38:42] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97632 and previous config saved to /var/cache/conftool/dbconfig/20261120-073842-marostegui.json [07:40:23] (03PS1) 10Marostegui: db2219: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1372588 (https://phabricator.wikimedia.org/T445479) [07:40:33] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2219.codfw.wmnet with reason: down [07:41:24] (03CR) 10Marostegui: [C:03+2] db2219: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1372588 (https://phabricator.wikimedia.org/T445479) (owner: 10Marostegui) [07:44:35] 06SRE, 10SRE-Access-Requests: Requesting access to Analytics Data Lake for sancheza/SanchezA-WMF - https://phabricator.wikimedia.org/T443850#12586122 (10Marostegui) 05Open→03Stalled p:05Triage→03Medium [07:45:20] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for toyofuku - https://phabricator.wikimedia.org/T445548#12586124 (10Marostegui) p:05Triage→03Medium a:03Marostegui [07:51:04] (03CR) 10Ryan Kemper: [C:03+1] "looks great!" [puppet] - 10https://gerrit.wikimedia.org/r/1364521 (https://phabricator.wikimedia.org/T443798) (owner: 10Hashar) [07:53:44] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97633 and previous config saved to /var/cache/conftool/dbconfig/20261120-075344-marostegui.json [07:57:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [08:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261120T0800) [08:02:15] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [08:04:08] (03PS1) 10Jelto: aptrepo: update gitlab-ce package to 19.4 [puppet] - 10https://gerrit.wikimedia.org/r/1372594 (https://phabricator.wikimedia.org/T445485) [08:04:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12586140 (10phaultfinder) [08:05:19] (03CR) 10Arnaudb: [C:03+1] "lgtm!" [puppet] - 10https://gerrit.wikimedia.org/r/1372594 (https://phabricator.wikimedia.org/T445485) (owner: 10Jelto) [08:05:38] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1372594 (https://phabricator.wikimedia.org/T445485) (owner: 10Jelto) [08:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [08:07:30] (03CR) 10Jelto: [C:03+2] aptrepo: update gitlab-ce package to 19.4 [puppet] - 10https://gerrit.wikimedia.org/r/1372594 (https://phabricator.wikimedia.org/T445485) (owner: 10Jelto) [08:08:46] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1161 (T445307)', diff saved to https://phabricator.wikimedia.org/P97634 and previous config saved to /var/cache/conftool/dbconfig/20261120-080846-marostegui.json [08:08:50] T445307: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307 [08:09:02] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1164.eqiad.wmnet with reason: Maintenance [08:09:50] (03PS1) 10Ammarpad: admin: Add FIDO SSH key for ammarpad [puppet] - 10https://gerrit.wikimedia.org/r/1372596 (https://phabricator.wikimedia.org/T442758) [08:10:31] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1185.eqiad.wmnet with reason: Maintenance [08:10:33] !log marostegui@cumin1004 dbctl commit (dc=all): 'Depooling db1185 (T445307)', diff saved to https://phabricator.wikimedia.org/P97635 and previous config saved to /var/cache/conftool/dbconfig/20261120-081032-marostegui.json [08:11:10] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:12:28] (03PS1) 10Arnaudb: gerrit: remove high 4xxratio noisy alert [alerts] - 10https://gerrit.wikimedia.org/r/1372598 [08:13:38] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1185 (T445307)', diff saved to https://phabricator.wikimedia.org/P97636 and previous config saved to /var/cache/conftool/dbconfig/20261120-081337-marostegui.json [08:14:03] (03CR) 10Ammarpad: "I have added the key to my userpage: https://www.mediawiki.org/w/index.php?title=User:Ammarpad&diff=prev&oldid=8735801" [puppet] - 10https://gerrit.wikimedia.org/r/1372596 (https://phabricator.wikimedia.org/T442758) (owner: 10Ammarpad) [08:28:14] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2141.codfw.wmnet, wikikube-worker2283.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-work [08:28:14] odfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker2345.codfw.wmnet, wikikube-worker2289.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2060.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2301.codfw.wmnet, wikikube-worker2002.codfw.wmnet, wikikube-worker2206.codfw.wmnet, wikikube-worker2263.codfw.wmnet, wikikube-worker2305.codfw.wmnet, wikikube-worker2089.codfw.wmnet, w [08:28:14] worker2272.codfw.wmnet, wikikube-worker2339.codfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2105.codfw.wmnet, wikikube-worker2324.codfw.wmnet, wikikube-worker2344.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [08:28:40] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97637 and previous config saved to /var/cache/conftool/dbconfig/20261120-082839-marostegui.json [08:29:14] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [08:30:20] !log jelto@cumin1004 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: version upgrade [08:33:20] !log brouberol@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [08:34:18] !log brouberol@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [08:34:34] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for gaiarajan - https://phabricator.wikimedia.org/T444625#12586227 (10Marostegui) 05Open→03Stalled p:05Triage→03Medium a:03gaiarajan [08:34:56] !log brouberol@deploy2003 helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. [08:35:30] !log brouberol@deploy2003 helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. [08:36:41] !log brouberol@deploy2003 helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [08:36:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv4 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140314 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [08:37:15] !log brouberol@deploy2003 helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [08:38:20] !log brouberol@deploy2003 helmfile [staging-codfw] START helmfile.d/admin 'apply'. [08:38:31] !log brouberol@deploy2003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [08:39:39] !log brouberol@deploy2003 helmfile [codfw] START helmfile.d/admin 'apply'. [08:39:39] (03PS1) 10Nik Gkountas: Enable the editing signals on Special:Saved on the Beta Cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372604 (https://phabricator.wikimedia.org/T445624) [08:40:01] !log brouberol@deploy2003 helmfile [codfw] DONE helmfile.d/admin 'apply'. [08:40:58] !log brouberol@deploy2003 helmfile [eqiad] START helmfile.d/admin 'apply'. [08:41:37] !log jelto@cumin1004 END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: version upgrade [08:41:41] !log brouberol@deploy2003 helmfile [eqiad] DONE helmfile.d/admin 'apply'. [08:42:37] !log jelto@cumin1004 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: version upgrade [08:43:07] (03PS1) 10Tryvix1509: itwiki: Add 'autopatrol' to 'mover' user group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372605 (https://phabricator.wikimedia.org/T445622) [08:43:11] !log brouberol@deploy2003 helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [08:43:41] !log brouberol@deploy2003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [08:43:42] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97638 and previous config saved to /var/cache/conftool/dbconfig/20261120-084341-marostegui.json [08:44:14] !log brouberol@deploy2003 helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [08:44:45] !log brouberol@deploy2003 helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [08:45:21] !log brouberol@deploy2003 helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [08:46:13] !log brouberol@deploy2003 helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [08:47:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [08:50:30] FIRING: KubernetesDeploymentUnavailableReplicas: ... [08:50:30] Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s&var-namespace=editcheck-headless&var-deployment=editcheck-headless-main - ... [08:50:30] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [08:52:48] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 2 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12586245 (10brouberol) `admin_ng` actually needed to be applied to all clusters, as we have the mirrormaker instances running in aux. All done. [08:52:49] (03PS1) 10Arnaudb: sre.hosts.upgrade-packages-collab: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1372606 [08:53:32] !log jelto@cumin1004 END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: version upgrade [08:54:57] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for marktozzi - https://phabricator.wikimedia.org/T445145#12586246 (10Marostegui) >>! In T445145#12580266, @Marostegui wrote: > Waiting for the out of band ssh key verification This was done yesterday - now waiting on approvals. [08:55:21] (03CR) 10JMeybohm: [C:03+2] upgrade-firmware: Allow updates of NonRAID controllers [cookbooks] - 10https://gerrit.wikimedia.org/r/1354829 (https://phabricator.wikimedia.org/T438999) (owner: 10JMeybohm) [08:56:09] !log jelto@cumin1004 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade [08:57:26] !log depool dns7001 for troubleshooting - T434806 [08:57:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:57:30] T434806: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806 [08:57:45] !log ayounsi@cumin1004 conftool action : set/pooled=no; selector: name=dns7001.* [08:58:01] (03Merged) 10jenkins-bot: upgrade-firmware: Allow updates of NonRAID controllers [cookbooks] - 10https://gerrit.wikimedia.org/r/1354829 (https://phabricator.wikimedia.org/T438999) (owner: 10JMeybohm) [08:58:44] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1185 (T445307)', diff saved to https://phabricator.wikimedia.org/P97639 and previous config saved to /var/cache/conftool/dbconfig/20261120-085843-marostegui.json [08:58:48] T445307: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307 [08:58:48] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1200.eqiad.wmnet with reason: Maintenance [08:58:50] !log marostegui@cumin1004 dbctl commit (dc=all): 'Depooling db1200 (T445307)', diff saved to https://phabricator.wikimedia.org/P97640 and previous config saved to /var/cache/conftool/dbconfig/20261120-085850-marostegui.json [09:01:19] (03PS1) 10Marostegui: data.yaml: Add mtozzi to analytics-privatedata [puppet] - 10https://gerrit.wikimedia.org/r/1372609 (https://phabricator.wikimedia.org/T445145) [09:01:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv4 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140314 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [09:01:59] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1200 (T445307)', diff saved to https://phabricator.wikimedia.org/P97641 and previous config saved to /var/cache/conftool/dbconfig/20261120-090158-marostegui.json [09:02:53] !log ayounsi@cumin1004 START - Cookbook sre.hosts.reboot-single for host dns7001.wikimedia.org [09:03:14] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586274 (10ops-monitoring-bot) Host dns7001* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [09:04:37] (03PS1) 10Nik Gkountas: Enable the worklist card view in production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372611 (https://phabricator.wikimedia.org/T445217) [09:04:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12586286 (10phaultfinder) [09:05:32] PROBLEM - BFD status on asw1-b3-magru.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [09:06:32] RECOVERY - BFD status on asw1-b3-magru.mgmt is OK: UP: 2 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [09:07:10] FIRING: [2x] BFDdown: BFD session down between asw1-b3-magru and 195.200.68.4 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:12:10] RESOLVED: [2x] BFDdown: BFD session down between asw1-b3-magru and 195.200.68.4 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:12:28] !log arnaudb@cumin1004 DONE (FAIL) - Cookbook sre.hosts.upgrade-packages-collab (exit_code=99) list of the available upgrades on aphlict2001.codfw.wmnet,aphlict1002.eqiad.wmnet,phab2003.codfw.wmnet,phab1005.eqiad.wmnet [09:15:07] jelto@cumin1004 jelto: The backup on gitlab1004 is complete, ready to proceed with upgrade. [09:16:30] (03PS1) 10Dpogorzelski: team-data-platform: alert on missing AMD GPUs [alerts] - 10https://gerrit.wikimedia.org/r/1372614 (https://phabricator.wikimedia.org/T445526) [09:17:01] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97642 and previous config saved to /var/cache/conftool/dbconfig/20261120-091700-marostegui.json [09:19:16] (03CR) 10CI reject: [V:04-1] team-data-platform: alert on missing AMD GPUs [alerts] - 10https://gerrit.wikimedia.org/r/1372614 (https://phabricator.wikimedia.org/T445526) (owner: 10Dpogorzelski) [09:19:35] (03PS2) 10Dpogorzelski: team-data-platform: alert on missing AMD GPUs [alerts] - 10https://gerrit.wikimedia.org/r/1372614 (https://phabricator.wikimedia.org/T445526) [09:21:56] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1092 bytes in 0.055 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [09:22:34] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dns7001.wikimedia.org [09:22:58] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 32493 bytes in 1.973 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [09:23:12] (03PS2) 10Arnaudb: sre.hosts.upgrade-packages-collab: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1372606 [09:23:58] !log arnaudb@cumin1004 DONE (PASS) - Cookbook sre.hosts.upgrade-packages-collab (exit_code=0) list of the available upgrades on aphlict2001.codfw.wmnet,aphlict1002.eqiad.wmnet,phab2003.codfw.wmnet,phab1005.eqiad.wmnet [09:26:12] !log jelto@cumin1004 END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade [09:27:01] !log ayounsi@cumin1004 conftool action : set/pooled=yes; selector: name=dns7001.*,service=authdns-update [09:27:20] !log ayounsi@dns1004 START - running authdns-update [09:27:32] !log arnaudb@cumin1004 START - Cookbook sre.hosts.upgrade-packages-collab upgrade of openssl libssl3 envoyproxy on aphlict2001.codfw.wmnet [09:28:05] !log repool dns7001 - T434806 [09:28:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:28:08] T434806: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806 [09:28:42] !log arnaudb@cumin1004 END (PASS) - Cookbook sre.hosts.upgrade-packages-collab (exit_code=0) upgrade of openssl libssl3 envoyproxy on aphlict2001.codfw.wmnet [09:29:33] !log ayounsi@dns1004 END - running authdns-update [09:29:51] !log ayounsi@cumin1004 conftool action : set/pooled=yes; selector: name=dns7001.* [09:32:03] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97643 and previous config saved to /var/cache/conftool/dbconfig/20261120-093202-marostegui.json [09:32:22] (03PS3) 10Arnaudb: sre.hosts.upgrade-packages-collab: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1372606 [09:37:36] !log depool doh1002 for troubleshooting - T434806 [09:37:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:37:40] T434806: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806 [09:38:12] !log ayounsi@cumin1004 START - Cookbook sre.hosts.reboot-single for host doh1002.wikimedia.org [09:38:40] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586369 (10ops-monitoring-bot) Host doh1002* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [09:38:47] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host doh1002.wikimedia.org [09:39:16] (03CR) 10Aghirelli: [C:03+1] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1371731 (https://phabricator.wikimedia.org/T445484) (owner: 10Daniel Kinzler) [09:40:09] !log trueg@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [09:41:10] FIRING: BFDdown: BFD session down between cr2-eqiad and 2620:0:861:4:208:80:155:112 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:44:53] !log ayounsi@cumin1004 START - Cookbook sre.hosts.reboot-single for host doh1002.wikimedia.org [09:45:15] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586373 (10ops-monitoring-bot) Host doh1002* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [09:45:41] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host doh1002.wikimedia.org [09:46:10] FIRING: [2x] BFDdown: BFD session down between cr1-eqiad and 2620:0:861:4:208:80:155:112 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:47:05] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1200 (T445307)', diff saved to https://phabricator.wikimedia.org/P97644 and previous config saved to /var/cache/conftool/dbconfig/20261120-094704-marostegui.json [09:47:09] T445307: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307 [09:47:20] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1207.eqiad.wmnet with reason: Maintenance [09:47:22] !log marostegui@cumin1004 dbctl commit (dc=all): 'Depooling db1207 (T445307)', diff saved to https://phabricator.wikimedia.org/P97645 and previous config saved to /var/cache/conftool/dbconfig/20261120-094722-marostegui.json [09:50:30] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1207 (T445307)', diff saved to https://phabricator.wikimedia.org/P97646 and previous config saved to /var/cache/conftool/dbconfig/20261120-095029-marostegui.json [09:56:06] !log ayounsi@cumin1004 START - Cookbook sre.hosts.reboot-single for host doh1002.wikimedia.org [09:56:29] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586406 (10ops-monitoring-bot) Host doh1002* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [09:56:44] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host doh1002.wikimedia.org [09:57:26] !log trueg@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [09:57:29] !log ayounsi@cumin1004 START - Cookbook sre.hosts.reboot-single for host doh1002.wikimedia.org [09:57:53] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586408 (10ops-monitoring-bot) Host doh1002* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [09:58:14] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host doh1002.wikimedia.org [09:58:42] (03PS2) 10Tryvix1509: itwiki: Add 'autopatrol' to 'mover' user group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372605 (https://phabricator.wikimedia.org/T445622) [09:59:50] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [10:01:10] RESOLVED: [2x] BFDdown: BFD session down between cr1-eqiad and 2620:0:861:4:208:80:155:112 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:02:22] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586416 (10ayounsi) Test is successful. # rebooted twice doh1002 as a control. Both time BFD stayed down. # Created a file `/etc/sysctl.d/99-tes... [10:02:57] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for mgerlach - https://phabricator.wikimedia.org/T443038#12586421 (10MGerlach) 05Resolved→03Open Hi, re-opening as I would like to add a second key for backup as recommended in the [[ https://wikitech.wikimedia.org/wiki/Hardware-backed_SSH_keys | docs... [10:05:32] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97647 and previous config saved to /var/cache/conftool/dbconfig/20261120-100531-marostegui.json [10:09:50] FIRING: [3x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [10:16:51] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1372609 (https://phabricator.wikimedia.org/T445145) (owner: 10Marostegui) [10:17:02] (03CR) 10Marostegui: [C:03+2] data.yaml: Add mtozzi to analytics-privatedata [puppet] - 10https://gerrit.wikimedia.org/r/1372609 (https://phabricator.wikimedia.org/T445145) (owner: 10Marostegui) [10:19:01] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1006.eqiad.wmnet [10:19:17] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for marktozzi - https://phabricator.wikimedia.org/T445145#12586467 (10Marostegui) analytics-privatedata-users has been granted (as doesn't require approval). You should've received an email from kerberos to set... [10:19:35] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for marktozzi - https://phabricator.wikimedia.org/T445145#12586469 (10Marostegui) [10:19:37] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1006.eqiad.wmnet [10:19:50] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [10:20:10] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for marktozzi - https://phabricator.wikimedia.org/T445145#12586474 (10Marostegui) 05Open→03Stalled [10:20:34] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97648 and previous config saved to /var/cache/conftool/dbconfig/20261120-102033-marostegui.json [10:24:23] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 2 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12586490 (10brouberol) As we agreed, I'm going to reclaim all available space on all kafka jumbo hosts by running: ` $ sudo lvextend --resizefs -l +... [10:28:18] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply [10:28:32] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply [10:29:38] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply [10:29:48] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply [10:29:53] (03PS1) 10Ladsgroup: mediawiki: Drop mediawiki-firejail-ffmpeg.py file [puppet] - 10https://gerrit.wikimedia.org/r/1372637 (https://phabricator.wikimedia.org/T172298) [10:30:31] (03CR) 10Muehlenhoff: [C:03+1] "Look good, thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1372596 (https://phabricator.wikimedia.org/T442758) (owner: 10Ammarpad) [10:30:34] (03CR) 10Muehlenhoff: [C:03+2] admin: Add FIDO SSH key for ammarpad [puppet] - 10https://gerrit.wikimedia.org/r/1372596 (https://phabricator.wikimedia.org/T442758) (owner: 10Ammarpad) [10:31:07] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply [10:31:12] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply [10:31:43] !log blake@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker1006.eqiad.wmnet with reason: T445629 [10:31:46] T445629: Validate 1G->10G migration steps - https://phabricator.wikimedia.org/T445629 [10:32:38] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply [10:32:42] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 2 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12586541 (10brouberol) {F116167570} We're now ready to rebalance. I'm going to close this. [10:32:47] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 2 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12586544 (10brouberol) 05In progress→03Resolved [10:32:48] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply [10:34:12] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for mgerlach - https://phabricator.wikimedia.org/T443038#12586549 (10Marostegui) key verified out of band [10:34:55] (03PS1) 10Marostegui: data.yaml: Add a second key for mgerlach [puppet] - 10https://gerrit.wikimedia.org/r/1372639 (https://phabricator.wikimedia.org/T443038) [10:35:35] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1207 (T445307)', diff saved to https://phabricator.wikimedia.org/P97649 and previous config saved to /var/cache/conftool/dbconfig/20261120-103535-marostegui.json [10:35:38] (03CR) 10Muehlenhoff: "Per Antoine it's still used in deployment-prep" [puppet] - 10https://gerrit.wikimedia.org/r/1372637 (https://phabricator.wikimedia.org/T172298) (owner: 10Ladsgroup) [10:35:39] T445307: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307 [10:35:51] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1216.eqiad.wmnet with reason: Maintenance [10:35:55] !log blake@cumin1004 START - Cookbook sre.hosts.decommission for hosts wikikube-worker1006.eqiad.wmnet [10:37:16] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1230.eqiad.wmnet with reason: Maintenance [10:37:17] (03CR) 10Ladsgroup: "Maybe I'm misunderstanding them but beta is also using shellbox too. Also https://wikimedia.slack.com/archives/C0BGTS44K5W/p17950375935664" [puppet] - 10https://gerrit.wikimedia.org/r/1372637 (https://phabricator.wikimedia.org/T172298) (owner: 10Ladsgroup) [10:37:18] !log marostegui@cumin1004 dbctl commit (dc=all): 'Depooling db1230 (T445307)', diff saved to https://phabricator.wikimedia.org/P97650 and previous config saved to /var/cache/conftool/dbconfig/20261120-103717-marostegui.json [10:37:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:37:47] (03CR) 10Muehlenhoff: [C:03+1] data.yaml: Add a second key for mgerlach [puppet] - 10https://gerrit.wikimedia.org/r/1372639 (https://phabricator.wikimedia.org/T443038) (owner: 10Marostegui) [10:38:06] (03CR) 10Marostegui: [C:03+2] data.yaml: Add a second key for mgerlach [puppet] - 10https://gerrit.wikimedia.org/r/1372639 (https://phabricator.wikimedia.org/T443038) (owner: 10Marostegui) [10:38:44] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for mgerlach - https://phabricator.wikimedia.org/T443038#12586567 (10Marostegui) 05Open→03Resolved key added, please allow 20 minutes for the change to spread across all the servers. [10:38:56] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 2 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12586575 (10Gehel) [10:39:14] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Degraded RAID on an-worker1194 - https://phabricator.wikimedia.org/T444799#12586583 (10Gehel) [10:40:24] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1230 (T445307)', diff saved to https://phabricator.wikimedia.org/P97651 and previous config saved to /var/cache/conftool/dbconfig/20261120-104023-marostegui.json [10:40:35] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-11-20 - 2026-12-11): Q1:rack/setup/install 10 new ceph nodes - https://phabricator.wikimedia.org/T438213#12586616 (10Gehel) [10:41:52] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Eqiad worker nodes should be connected to 10G NICs - https://phabricator.wikimedia.org/T440696#12586649 (10Blake) I've created https://phabricator.wikimedia.org/T445629 to test the process described [[ https://phabric... [10:42:12] 10SRE-SLO, 10observability, 10Wikidata, 06Wikidata Platform Team, and 3 others: Update WDQS SLOs to reflect graph split changes - https://phabricator.wikimedia.org/T393966#12586654 (10Gehel) [10:42:37] !log blake@cumin1004 START - Cookbook sre.dns.netbox [10:43:58] 06SRE, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07SecTeam-Processed: Build/deploy new Flink OCI (docker) image - https://phabricator.wikimedia.org/T443032#12586704 (10Gehel) [10:45:07] 06SRE, 06Data-Engineering, 10Kafka-Infrastructure, 06serviceops-radar, and 3 others: Configuration Management for Kafka settings - https://phabricator.wikimedia.org/T276088#12586742 (10Gehel) [10:47:07] !log blake@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker1006.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - blake@cumin1004" [10:47:30] !log blake@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker1006.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - blake@cumin1004" [10:47:30] !log blake@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [10:47:31] !log blake@cumin1004 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wikikube-worker1006.eqiad.wmnet [10:47:51] 06SRE, 10Kafka-Infrastructure, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Monitor kafka-configurator runs and pending changes - https://phabricator.wikimedia.org/T439506#12586791 (10Gehel) [10:49:48] 10ops-codfw, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Q1:rack/setup/install cirrussearch21[16-20] - https://phabricator.wikimedia.org/T436287#12586826 (10Gehel) [10:50:15] (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] wikidata-query-gui: bump to latest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371809 (https://phabricator.wikimedia.org/T441707) (owner: 10Arthur taylor) [10:50:19] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:50:19] (03PS2) 10Ladsgroup: vips: Use thumbnail command instead of shrink [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1358872 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup-claude) [10:50:23] (03PS3) 10Ladsgroup: vips: Use thumbnail command instead of shrink [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1358872 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup-claude) [10:50:49] (03PS2) 10Hnowlan: deployment: add mazevedo [puppet] - 10https://gerrit.wikimedia.org/r/1371075 (https://phabricator.wikimedia.org/T444497) [10:53:20] (03PS2) 10Ladsgroup: Remove the core_multiinstance role [puppet] - 10https://gerrit.wikimedia.org/r/1371837 [10:53:28] (03CR) 10Ladsgroup: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1371837 (owner: 10Ladsgroup) [10:53:32] 10ops-eqiad, 06DC-Ops, 06ServiceOps: Modify cabling of wikikube-worker1006 - https://phabricator.wikimedia.org/T445632 (10Blake) 03NEW [10:54:05] 10ops-eqiad, 06DC-Ops, 06ServiceOps: Modify cabling of wikikube-worker1006 - https://phabricator.wikimedia.org/T445632#12586860 (10Blake) p:05Triage→03Medium [10:55:26] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97652 and previous config saved to /var/cache/conftool/dbconfig/20261120-105525-marostegui.json [10:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [11:00:13] !log ayounsi@cumin1004 START - Cookbook sre.hosts.reboot-single for host doh1002.wikimedia.org [11:00:36] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586867 (10ops-monitoring-bot) Host doh1002* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [11:00:58] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host doh1002.wikimedia.org [11:02:54] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-pageview: apply [11:03:03] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-pageview: apply [11:03:17] (03PS3) 10Ladsgroup: Remove the core_multiinstance role [puppet] - 10https://gerrit.wikimedia.org/r/1371837 [11:03:23] (03CR) 10Ladsgroup: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1371837 (owner: 10Ladsgroup) [11:06:38] (03CR) 10Jgiannelos: vips: Use thumbnail command instead of shrink (031 comment) [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1358872 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup-claude) [11:06:55] (03PS4) 10Ladsgroup: Remove the core_multiinstance role [puppet] - 10https://gerrit.wikimedia.org/r/1371837 [11:07:01] (03PS5) 10Ladsgroup: Remove the core_multiinstance role [puppet] - 10https://gerrit.wikimedia.org/r/1371837 [11:07:09] (03PS6) 10Ladsgroup: Remove the core_multiinstance role [puppet] - 10https://gerrit.wikimedia.org/r/1371837 [11:07:17] (03CR) 10Ladsgroup: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1371837 (owner: 10Ladsgroup) [11:07:25] !log jelto@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on gitlab1004.wikimedia.org with reason: service restart [11:08:31] (03CR) 10Muehlenhoff: [C:03+1] "Key has been verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1371891 (https://phabricator.wikimedia.org/T442758) (owner: 10Tchanders) [11:08:34] (03CR) 10Muehlenhoff: [C:03+2] admin: Add second FIDO SSH key for tchanders [puppet] - 10https://gerrit.wikimedia.org/r/1371891 (https://phabricator.wikimedia.org/T442758) (owner: 10Tchanders) [11:09:27] (03CR) 10Ladsgroup: vips: Use thumbnail command instead of shrink (031 comment) [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1358872 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup-claude) [11:10:28] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97653 and previous config saved to /var/cache/conftool/dbconfig/20261120-111027-marostegui.json [11:11:42] !log marostegui@cumin1004 DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on db2219.codfw.wmnet with reason: down [11:14:50] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 22d 7h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [11:17:19] (03CR) 10Ladsgroup: "https://puppet-compiler.wmflabs.org/output/1371837/8424/ PCC picked one host at random from each role." [puppet] - 10https://gerrit.wikimedia.org/r/1371837 (owner: 10Ladsgroup) [11:17:25] FIRING: [3x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:18:07] (03CR) 10Marostegui: [C:03+1] Remove the core_multiinstance role [puppet] - 10https://gerrit.wikimedia.org/r/1371837 (owner: 10Ladsgroup) [11:18:37] (03PS7) 10Ladsgroup: Remove the core_multiinstance role [puppet] - 10https://gerrit.wikimedia.org/r/1371837 [11:18:41] (03CR) 10Ladsgroup: [V:03+2 C:03+2] Remove the core_multiinstance role [puppet] - 10https://gerrit.wikimedia.org/r/1371837 (owner: 10Ladsgroup) [11:21:59] (03CR) 10Jgiannelos: vips: Use thumbnail command instead of shrink (031 comment) [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1358872 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup-claude) [11:24:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12586905 (10phaultfinder) [11:25:29] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1230 (T445307)', diff saved to https://phabricator.wikimedia.org/P97654 and previous config saved to /var/cache/conftool/dbconfig/20261120-112529-marostegui.json [11:25:33] T445307: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307 [11:25:45] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1274.eqiad.wmnet with reason: Maintenance [11:25:47] !log marostegui@cumin1004 dbctl commit (dc=all): 'Depooling db1274 (T445307)', diff saved to https://phabricator.wikimedia.org/P97655 and previous config saved to /var/cache/conftool/dbconfig/20261120-112546-marostegui.json [11:28:20] (03CR) 10Jgiannelos: [C:03+1] vips: Use thumbnail command instead of shrink [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1358872 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup-claude) [11:28:52] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1274 (T445307)', diff saved to https://phabricator.wikimedia.org/P97656 and previous config saved to /var/cache/conftool/dbconfig/20261120-112851-marostegui.json [11:34:45] FIRING: ProbeDown: Service etherpad1004:7443 has failed probes (http_etherpad_envoy_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#etherpad1004:7443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:43:54] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1274', diff saved to https://phabricator.wikimedia.org/P97657 and previous config saved to /var/cache/conftool/dbconfig/20261120-114353-marostegui.json [11:44:56] (03CR) 10Hashar: [C:03+1] "Moritz and I talked about the mediawiki-firejail scripts on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1364525/comments/0e7f72c8" [puppet] - 10https://gerrit.wikimedia.org/r/1372637 (https://phabricator.wikimedia.org/T172298) (owner: 10Ladsgroup) [11:52:01] !log awight@deploy2003 helmfile [codfw] START helmfile.d/services/mw-experimental: apply [11:52:53] !log awight@deploy2003 helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply [11:58:56] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1274', diff saved to https://phabricator.wikimedia.org/P97658 and previous config saved to /var/cache/conftool/dbconfig/20261120-115855-marostegui.json [12:00:04] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [12:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261120T0800) [12:00:05] jelto, arnoldokoth, mutante, and arnaudb: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for GitLab version upgrades deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261120T1200). [12:00:14] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [12:00:18] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [12:00:29] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [12:02:15] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [12:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [12:11:11] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:13:57] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1274 (T445307)', diff saved to https://phabricator.wikimedia.org/P97659 and previous config saved to /var/cache/conftool/dbconfig/20261120-121357-marostegui.json [12:14:02] T445307: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307 [12:14:15] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1275.eqiad.wmnet with reason: Maintenance [12:14:24] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on 6 hosts with reason: Maintenance [12:14:27] !log marostegui@cumin1004 dbctl commit (dc=all): 'Depooling db1275 (T445307)', diff saved to https://phabricator.wikimedia.org/P97660 and previous config saved to /var/cache/conftool/dbconfig/20261120-121426-marostegui.json [12:17:37] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1275 (T445307)', diff saved to https://phabricator.wikimedia.org/P97661 and previous config saved to /var/cache/conftool/dbconfig/20261120-121736-marostegui.json [12:32:38] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1275', diff saved to https://phabricator.wikimedia.org/P97662 and previous config saved to /var/cache/conftool/dbconfig/20261120-123238-marostegui.json [12:35:00] (03PS3) 10Dpogorzelski: team-data-platform: alert on missing AMD GPUs [alerts] - 10https://gerrit.wikimedia.org/r/1372614 (https://phabricator.wikimedia.org/T445526) [12:36:40] (03PS1) 10Jelto: etherpad: disable etherpad service for now [puppet] - 10https://gerrit.wikimedia.org/r/1372689 (https://phabricator.wikimedia.org/T445634) [12:37:35] !log arnaudb@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on etherpad2002.codfw.wmnet,etherpad1004.eqiad.wmnet with reason: maintenance [12:38:35] (03CR) 10Jelto: [V:03+1] "PCC SUCCESS (CORE_DIFF 1 NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/" [puppet] - 10https://gerrit.wikimedia.org/r/1372689 (https://phabricator.wikimedia.org/T445634) (owner: 10Jelto) [12:39:07] (03PS1) 10Mpostoronca: WikimediaAntiAbuse: Add wikimediaantiabuse-abusereview dblist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372688 (https://phabricator.wikimedia.org/T443206) [12:39:41] (03PS4) 10Dpogorzelski: team-data-platform: alert on missing AMD GPUs [alerts] - 10https://gerrit.wikimedia.org/r/1372614 (https://phabricator.wikimedia.org/T445526) [12:43:27] (03CR) 10JMeybohm: [C:04-1] keda: add WMF changes to the upstream chart (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367745 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [12:43:32] (03CR) 10JMeybohm: [C:04-1] "Please keep in mind that deploying it like this will grants broad permissions to the operator across the whole cluster. I would suggest li" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364011 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [12:44:10] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for mgerlach - https://phabricator.wikimedia.org/T443038#12587097 (10MGerlach) Confirmed that the new key works. Thank you. [12:46:08] (03CR) 10Jelto: [V:03+1 C:03+2] etherpad: disable etherpad service for now [puppet] - 10https://gerrit.wikimedia.org/r/1372689 (https://phabricator.wikimedia.org/T445634) (owner: 10Jelto) [12:47:41] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1275', diff saved to https://phabricator.wikimedia.org/P97663 and previous config saved to /var/cache/conftool/dbconfig/20261120-124740-marostegui.json [12:48:28] (03PS2) 10Mpostoronca: WikimediaAntiAbuse: Add wikimediaantiabuse-abusereview dblist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372688 (https://phabricator.wikimedia.org/T443206) [12:50:30] FIRING: KubernetesDeploymentUnavailableReplicas: ... [12:50:30] Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s&var-namespace=editcheck-headless&var-deployment=editcheck-headless-main - ... [12:50:30] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [12:58:19] (03PS2) 10Ladsgroup: base: phaste: authenticate with a Conduit API token [puppet] - 10https://gerrit.wikimedia.org/r/1372691 (owner: 10Ladsgroup-claude) [12:58:19] (03CR) 10Ladsgroup: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1372691 (owner: 10Ladsgroup-claude) [12:58:25] (03PS3) 10Ladsgroup: base: phaste: authenticate with a Conduit API token [puppet] - 10https://gerrit.wikimedia.org/r/1372691 (owner: 10Ladsgroup-claude) [12:59:24] (03PS1) 10Ladsgroup: Dummy conduit token for ProdPasteBot [labs/private] - 10https://gerrit.wikimedia.org/r/1372694 [12:59:25] (03CR) 10Mpostoronca: "l" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372688 (https://phabricator.wikimedia.org/T443206) (owner: 10Mpostoronca) [13:01:37] (03PS1) 10Dpogorzelski: prometheus: export AMD GPU RAS error counters [puppet] - 10https://gerrit.wikimedia.org/r/1372697 (https://phabricator.wikimedia.org/T445526) [13:01:42] (03PS1) 10Dpogorzelski: team-data-platform: alert on AMD GPU uncorrectable errors [alerts] - 10https://gerrit.wikimedia.org/r/1372698 (https://phabricator.wikimedia.org/T445526) [13:01:42] FIRING: JobUnavailable: Reduced availability for job etherpad in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:01:47] (03PS3) 10Mpostoronca: tables-catalog: Add AbuseReview tables [puppet] - 10https://gerrit.wikimedia.org/r/1364499 (https://phabricator.wikimedia.org/T443206) [13:02:42] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1275 (T445307)', diff saved to https://phabricator.wikimedia.org/P97664 and previous config saved to /var/cache/conftool/dbconfig/20261120-130242-marostegui.json [13:02:46] T445307: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307 [13:02:58] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance [13:04:40] (03PS2) 10Dpogorzelski: prometheus: export AMD GPU RAS error counters [puppet] - 10https://gerrit.wikimedia.org/r/1372697 (https://phabricator.wikimedia.org/T445526) [13:04:47] (03PS4) 10Mpostoronca: tables-catalog: Add AbuseReview tables [puppet] - 10https://gerrit.wikimedia.org/r/1364499 (https://phabricator.wikimedia.org/T443206) [13:07:21] (03CR) 10Mpostoronca: tables-catalog: Add AbuseReview tables (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1364499 (https://phabricator.wikimedia.org/T443206) (owner: 10Mpostoronca) [13:14:00] (03CR) 10Ladsgroup: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1372691 (owner: 10Ladsgroup-claude) [13:14:15] (03CR) 10Ladsgroup: [V:03+2 C:03+2] Dummy conduit token for ProdPasteBot [labs/private] - 10https://gerrit.wikimedia.org/r/1372694 (owner: 10Ladsgroup) [13:15:21] (03CR) 10Arnaudb: [C:03+1] "lgtm!" [puppet] - 10https://gerrit.wikimedia.org/r/1372691 (owner: 10Ladsgroup-claude) [13:15:43] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 23 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372688 (https://phabricator.wikimedia.org/T443206) (owner: 10Mpostoronca) [13:19:15] !log ammarpad@deploy2003 mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=itwiki --logwiki=metawiki 'Nikolay Omonov' User3761 # T445620 [13:19:19] T445620: Unblock stuck global rename of User3761 - https://phabricator.wikimedia.org/T445620 [13:19:38] (03PS4) 10Ladsgroup: base: phaste: authenticate with a Conduit API token [puppet] - 10https://gerrit.wikimedia.org/r/1372691 (owner: 10Ladsgroup-claude) [13:19:44] (03CR) 10Ladsgroup: [V:03+2 C:03+2] base: phaste: authenticate with a Conduit API token [puppet] - 10https://gerrit.wikimedia.org/r/1372691 (owner: 10Ladsgroup-claude) [13:19:55] (03CR) 10Klausman: [C:03+1] prometheus: export AMD GPU RAS error counters [puppet] - 10https://gerrit.wikimedia.org/r/1372697 (https://phabricator.wikimedia.org/T445526) (owner: 10Dpogorzelski) [13:20:19] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1210.eqiad.wmnet with reason: Maintenance [13:21:25] (03CR) 10Klausman: [C:03+1] team-data-platform: alert on missing AMD GPUs [alerts] - 10https://gerrit.wikimedia.org/r/1372614 (https://phabricator.wikimedia.org/T445526) (owner: 10Dpogorzelski) [13:25:27] !log ammarpad@deploy2003 mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki metawiki 'Wiki Loves Children' 'Wiki Loves Childhood' Ammarpad --reason 'Requested at [[:phab:T445242|T445242]]' [13:25:31] T445242: Request to move translatable page: Wiki Loves Children - https://phabricator.wikimedia.org/T445242 [13:30:01] !log ayounsi@cumin1004 START - Cookbook sre.hosts.reboot-single for host netflow4003.ulsfo.wmnet [13:31:54] (03PS2) 10Ladsgroup: mediawiki: Drop mediawiki-firejail-ffmpeg.py file [puppet] - 10https://gerrit.wikimedia.org/r/1372637 (https://phabricator.wikimedia.org/T172298) [13:32:34] (03CR) 10Ladsgroup: [V:03+2 C:03+2] mediawiki: Drop mediawiki-firejail-ffmpeg.py file [puppet] - 10https://gerrit.wikimedia.org/r/1372637 (https://phabricator.wikimedia.org/T172298) (owner: 10Ladsgroup) [13:34:08] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow4003.ulsfo.wmnet [13:36:02] (03CR) 10Ladsgroup: "maybe something has changed recently but dblist-index should also be updated and the CI should fail otherwise. I think something is missin" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372688 (https://phabricator.wikimedia.org/T443206) (owner: 10Mpostoronca) [13:37:02] (03PS1) 10Jelto: etherpad: update offline page, remove nodejs incident link [puppet] - 10https://gerrit.wikimedia.org/r/1372702 [13:37:40] (03CR) 10Marostegui: [C:03+1] etherpad: update offline page, remove nodejs incident link [puppet] - 10https://gerrit.wikimedia.org/r/1372702 (owner: 10Jelto) [13:38:14] !log ayounsi@cumin1004 START - Cookbook sre.hosts.reboot-single for host netflow4003.ulsfo.wmnet [13:41:30] (03CR) 10Jelto: [C:03+2] etherpad: update offline page, remove nodejs incident link [puppet] - 10https://gerrit.wikimedia.org/r/1372702 (owner: 10Jelto) [13:42:21] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow4003.ulsfo.wmnet [13:47:14] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2170.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2148.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-work [13:47:14] odfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker2345.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2139.codfw.wmnet, w [13:47:14] worker2315.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2322.codfw.wmnet, wikikube-worker2060.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2125.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [13:48:02] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2364.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-work [13:48:02] odfw.wmnet, wikikube-worker2041.codfw.wmnet, wikikube-worker2147.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2114.codfw.wmnet, wikikube-worker2324.codfw.wmnet, wikikube-worker2075.codfw.wmnet, wikikube-worker2104.codfw.wmnet, wikikube-worker2035.codfw.wmnet, wikikube-worker2267.codfw.wmnet, wikikube-worker2321.codfw.wmnet, wikikube-worker2351.codfw.wmnet, wikikube-worker2352.codfw.wmnet, wikikube-worker2275.codfw.wmnet, w [13:48:02] worker2284.codfw.wmnet, wikikube-worker2163.codfw.wmnet, wikikube-worker2167.codfw.wmnet, wikikube-worker2095.codfw.wmnet, wikikube-worker2318.codfw.wmnet, wikikube-worker2309.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [13:48:14] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [13:49:02] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [13:57:40] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), and 2 others: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12587452 (10Gehel) [13:57:50] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Q1:rack/setup/install cirrussearch11[26-30] - https://phabricator.wikimedia.org/T436285#12587459 (10Gehel) [13:57:58] (03PS1) 10Andrew Bogott: Revert "[wmcs::kubeadm::core] remove kubeadm-flags.env" [puppet] - 10https://gerrit.wikimedia.org/r/1372707 [13:58:49] (03CR) 10CI reject: [V:04-1] Revert "[wmcs::kubeadm::core] remove kubeadm-flags.env" [puppet] - 10https://gerrit.wikimedia.org/r/1372707 (owner: 10Andrew Bogott) [14:01:09] 06SRE, 06DC-Ops, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12587546 (10Gehel) [14:02:50] (03PS2) 10Andrew Bogott: Revert "[wmcs::kubeadm::core] remove kubeadm-flags.env" [puppet] - 10https://gerrit.wikimedia.org/r/1372707 (https://phabricator.wikimedia.org/T370245) [14:03:00] (03CR) 10Majavah: [C:03+1] Revert "[wmcs::kubeadm::core] remove kubeadm-flags.env" [puppet] - 10https://gerrit.wikimedia.org/r/1372707 (https://phabricator.wikimedia.org/T370245) (owner: 10Andrew Bogott) [14:05:53] (03CR) 10Andrew Bogott: [C:03+2] Revert "[wmcs::kubeadm::core] remove kubeadm-flags.env" [puppet] - 10https://gerrit.wikimedia.org/r/1372707 (https://phabricator.wikimedia.org/T370245) (owner: 10Andrew Bogott) [14:08:56] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for dhardy to restore deployment access - https://phabricator.wikimedia.org/T445598#12587649 (10Marostegui) Out of band verified. [14:11:28] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for dhardy to restore deployment access - https://phabricator.wikimedia.org/T445598#12587665 (10DMburugu) I approve [14:16:49] (03PS1) 10Marostegui: data.yaml: Add ssh keys for dhardy [puppet] - 10https://gerrit.wikimedia.org/r/1372711 (https://phabricator.wikimedia.org/T445598) [14:17:41] (03CR) 10Marostegui: "keys have been verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1372711 (https://phabricator.wikimedia.org/T445598) (owner: 10Marostegui) [14:18:57] !log ayounsi@cumin1004 START - Cookbook sre.hosts.reboot-single for host netflow4003.ulsfo.wmnet [14:19:00] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1372711 (https://phabricator.wikimedia.org/T445598) (owner: 10Marostegui) [14:19:13] (03CR) 10Marostegui: [C:03+2] data.yaml: Add ssh keys for dhardy [puppet] - 10https://gerrit.wikimedia.org/r/1372711 (https://phabricator.wikimedia.org/T445598) (owner: 10Marostegui) [14:20:05] FIRING: HelmReleaseBadStatus: Helm release wdqs/main-internal on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [14:20:32] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for dhardy to restore deployment access - https://phabricator.wikimedia.org/T445598#12587783 (10Marostegui) 05Open→03Resolved Patch merged - please allow 20 minutes for the change to spread across production. [14:22:49] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow4003.ulsfo.wmnet [14:23:04] !log ayounsi@cumin1004 START - Cookbook sre.hosts.reboot-single for host netflow4003.ulsfo.wmnet [14:24:02] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2174.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker2345.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-work [14:24:02] odfw.wmnet, wikikube-worker2002.codfw.wmnet, wikikube-worker2062.codfw.wmnet, wikikube-worker2339.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2014.codfw.wmnet, wikikube-worker2300.codfw.wmnet, wikikube-worker2169.codfw.wmnet, wikikube-worker2214.codfw.wmnet, wikikube-worker2370.codfw.wmnet, wikikube-worker2266.codfw.wmnet, wikikube-worker2104.codfw.wmnet, wikikube-worker2321.codfw.wmnet, wikikube-worker2138.codfw.wmnet, w [14:24:02] worker2351.codfw.wmnet, wikikube-worker2088.codfw.wmnet, wikikube-worker2037.codfw.wmnet, wikikube-worker2012.codfw.wmnet, wikikube-worker2286.codfw.wmnet, wikikube-worker2207.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [14:25:02] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [14:26:53] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for dhardy to restore deployment access - https://phabricator.wikimedia.org/T445598#12587838 (10Dillon) >>! In T445598#12587783, @Marostegui wrote: > Patch merged - please allow 20 minutes for the change to spread across productio... [14:27:04] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow4003.ulsfo.wmnet [14:31:02] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2033.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2289.codfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-worker2055.codfw.wmnet, wikikube-work [14:31:02] odfw.wmnet, wikikube-worker2344.codfw.wmnet, wikikube-worker2332.codfw.wmnet, wikikube-worker2176.codfw.wmnet, wikikube-worker2094.codfw.wmnet, wikikube-worker2189.codfw.wmnet, wikikube-worker2137.codfw.wmnet, wikikube-worker2316.codfw.wmnet, wikikube-worker2186.codfw.wmnet, wikikube-worker2068.codfw.wmnet, wikikube-worker2201.codfw.wmnet, wikikube-worker2188.codfw.wmnet, wikikube-worker2057.codfw.wmnet, wikikube-worker2145.codfw.wmnet, w [14:31:02] worker2290.codfw.wmnet, wikikube-worker2038.codfw.wmnet, wikikube-worker2256.codfw.wmnet, wikikube-worker2365.codfw.wmnet, wikikube-worker2069.codfw.wmnet, wikikube-worker2078.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [14:32:02] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [14:34:55] (03PS1) 10Majavah: kubeadm: Plumb in current pause_image to init config [puppet] - 10https://gerrit.wikimedia.org/r/1372712 [14:37:03] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-pageview: apply [14:37:12] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-pageview: apply [14:38:24] !log javiermonton@deploy2003 Started scap deploy-service mw-page-html-content-change-enrich: scaling down to its regular number of replicas after incident. [14:38:24] !log javiermonton@deploy2003 scap deploy-service mw-page-html-content-change-enrich: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying mw-page-html-content-change-enrich [14:39:23] FIRING: GnmiInterfaceCountersDrop: ... [14:39:23] cr3-ulsfo is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=cr3-ulsfo:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:40:23] !log javiermonton@deploy2003 Finished scap deploy-service mw-page-html-content-change-enrich: scaling down to its regular number of replicas after incident. (duration: 01m 59s) [14:41:46] (03PS1) 10Krinkle: php: Remove php_apcu_cache_ops{type=entries} pseudo counter [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1372714 (https://phabricator.wikimedia.org/T433312) [14:42:47] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9691/co" [puppet] - 10https://gerrit.wikimedia.org/r/1372712 (owner: 10Majavah) [14:48:00] (03PS3) 10Majavah: icinga: Ignore hosts not present in Icinga [software/spicerack] - 10https://gerrit.wikimedia.org/r/1369967 (https://phabricator.wikimedia.org/T445087) [14:48:34] (03CR) 10Ladsgroup: [C:04-1] "I see why it's not added to DB_LISTS, it's for documentation only but that doesn't set well with me. Because documentations easily drift a" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372688 (https://phabricator.wikimedia.org/T443206) (owner: 10Mpostoronca) [14:49:29] !log forcing major compaction, sessionstore1004-a (diagnostic) — T445661 [14:49:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:49:33] T445661: sessionstore: rapid increase in utilization - https://phabricator.wikimedia.org/T445661 [14:50:19] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:53:51] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for toyofuku - https://phabricator.wikimedia.org/T445548#12588007 (10Jdrewniak) request approved [14:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [14:59:15] 06SRE, 06Fundraising-Backlog, 10FY26-27 ST6.4 - Logged-in performance, 10MediaWiki-extensions-CentralNotice, and 4 others: Special:HideBanners is not really cacheable - https://phabricator.wikimedia.org/T256447#12588067 (10Krinkle) a:03Krinkle @Ejegg Yes and no. As it stand today, I think we can rely on... [15:05:38] 06SRE, 06Infrastructure-Foundations, 10netops: QFX5120 reporting 0 for neighbor bgp prefix counters on 23.4R2-S8.7 - https://phabricator.wikimedia.org/T433490#12588126 (10cmooney) So annoying: ` Hi Cathel, I have been working with our engineering teams and was informed that this feature is not fully suppo... [15:06:02] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2177.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-work [15:06:02] odfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2282.codfw.wmnet, wikikube-worker2307.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2013.codfw.wmnet, wikikube-worker2056.codfw.wmnet, wikikube-worker2208.codfw.wmnet, wikikube-worker2279.codfw.wmnet, wikikube-worker2351.codfw.wmnet, wikikube-worker2088.codfw.wmnet, wikikube-worker2037.codfw.wmnet, wikikube-worker2286.codfw.wmnet, wikikube-worker2260.codfw.wmnet, w [15:06:02] worker2199.codfw.wmnet, wikikube-worker2016.codfw.wmnet, wikikube-worker2137.codfw.wmnet, wikikube-worker2348.codfw.wmnet, wikikube-worker2074.codfw.wmnet, wikikube-worker2323.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [15:07:02] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [15:07:04] (03CR) 10Hashar: [C:04-1] "I'll amend and sync the other templates 😎" [puppet] - 10https://gerrit.wikimedia.org/r/1371818 (owner: 10Paladox) [15:07:30] 06SRE, 06Fundraising-Backlog, 10FY26-27 ST6.4 - Logged-in performance, 10MediaWiki-extensions-CentralNotice, and 4 others: Special:HideBanners is not really cacheable - https://phabricator.wikimedia.org/T256447#12588129 (10Ejegg) Interesting that the cookie would work for a few days at least. I guess that'... [15:08:28] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12588133 (10ayounsi) Unfortunately it's more complicated than expected... Setting up that configuration option would trigger more of the issue docume... [15:09:43] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy2003 using scap backport" [core] (wmf/1.48.0-wmf.5) - 10https://gerrit.wikimedia.org/r/1370680 (https://phabricator.wikimedia.org/T445147) (owner: 10Krinkle) [15:13:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12588158 (10Jhancock.wm) last attempt at power balancing to fix. will need to move a wikikube after this. [15:14:04] PROBLEM - Host logging-sd2001 is DOWN: PING CRITICAL - Packet loss = 100% [15:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 22d 3h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [15:15:02] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2274.codfw.wmnet, wikikube-worker2314.codfw.wmnet, wikikube-worker2140.codfw.wmnet, wikikube-worker2050.codfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2151.codfw.wmnet, wikikube-worker2156.codfw.wmnet, wikikube-worker2324.codfw.wmnet, wikikube-worker2087.codfw.wmnet, wikikube-work [15:15:02] odfw.wmnet, wikikube-worker2267.codfw.wmnet, wikikube-worker2283.codfw.wmnet, wikikube-worker2298.codfw.wmnet, wikikube-worker2317.codfw.wmnet, wikikube-worker2204.codfw.wmnet, wikikube-worker2016.codfw.wmnet, wikikube-worker2194.codfw.wmnet, wikikube-worker2137.codfw.wmnet, wikikube-worker2167.codfw.wmnet, wikikube-worker2316.codfw.wmnet, wikikube-worker2265.codfw.wmnet, wikikube-worker2188.codfw.wmnet, wikikube-worker2178.codfw.wmnet, w [15:15:02] worker2306.codfw.wmnet, wikikube-worker2072.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [15:16:02] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [15:17:01] (03Merged) 10jenkins-bot: MediaWikiEntryPoint: Introduce unsampled post.log [core] (wmf/1.48.0-wmf.5) - 10https://gerrit.wikimedia.org/r/1370680 (https://phabricator.wikimedia.org/T445147) (owner: 10Krinkle) [15:17:16] RECOVERY - Host logging-sd2001 is UP: PING OK - Packet loss = 0%, RTA = 31.71 ms [15:17:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:18:07] !log krinkle@deploy2003 Started scap sync-world: Backport for [[gerrit:1370680|MediaWikiEntryPoint: Introduce unsampled post.log (T445147)]] (1.48.0-wmf.5) [15:18:10] T445147: Introduce unsampled post.log - https://phabricator.wikimedia.org/T445147 [15:20:42] !log krinkle@deploy2003 krinkle: Backport for [[gerrit:1370680|MediaWikiEntryPoint: Introduce unsampled post.log (T445147)]] (1.48.0-wmf.5) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:22:46] !log krinkle@deploy2003 krinkle: Continuing with deployment [15:23:00] PROBLEM - Blazegraph Port for wdqs-blazegraph on wdqs1019 is CRITICAL: connect to address 127.0.0.1 and port 9999: Connection refused https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [15:24:00] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs1019 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [15:27:40] (03PS1) 10Ladsgroup: beta: Point to a different font in beta cluster for timeline [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372781 [15:28:54] !log krinkle@deploy2003 Finished scap sync-world: Backport for [[gerrit:1370680|MediaWikiEntryPoint: Introduce unsampled post.log (T445147)]] (1.48.0-wmf.5) (duration: 10m 48s) [15:28:58] T445147: Introduce unsampled post.log - https://phabricator.wikimedia.org/T445147 [15:29:38] (03CR) 10Ladsgroup: [C:03+2] beta: Point to a different font in beta cluster for timeline [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372781 (owner: 10Ladsgroup) [15:30:39] (03Merged) 10jenkins-bot: beta: Point to a different font in beta cluster for timeline [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372781 (owner: 10Ladsgroup) [15:34:38] 10ops-eqiad, 06DC-Ops: Alert for device ps1-a4-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T445684 (10phaultfinder) 03NEW [15:36:51] (03PS4) 10Hashar: gerrit: sync soy templates with version 3.11 [puppet] - 10https://gerrit.wikimedia.org/r/1371818 (https://phabricator.wikimedia.org/T43608) (owner: 10Paladox) [15:39:14] (03CR) 10Hashar: [C:03+1] "I have verified the other soy templates and amended the commit message with a rationale for each o them." [puppet] - 10https://gerrit.wikimedia.org/r/1371818 (https://phabricator.wikimedia.org/T43608) (owner: 10Paladox) [15:39:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12588328 (10phaultfinder) [15:40:28] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for daimona - https://phabricator.wikimedia.org/T445375#12588335 (10Daimona) Thank you :) I confirm that I can SSH to stat hosts (although I don't really need that, and I only need GrowthBook access which I requested separately,... [15:43:39] (03PS2) 10Bking: opensearch: Refresh images (disable hardware-accelerated base64) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1370674 (https://phabricator.wikimedia.org/T445270) [15:44:28] (03PS4) 10Krinkle: Discontinue xff.log [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361194 (https://phabricator.wikimedia.org/T288223) [15:46:08] (03CR) 10Daimona Eaytoy: [C:03+1] Enable the editing signals on Special:Saved on the Beta Cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372604 (https://phabricator.wikimedia.org/T445624) (owner: 10Nik Gkountas) [15:46:16] (03PS3) 10Bking: opensearch: Refresh images [deployment-charts] - 10https://gerrit.wikimedia.org/r/1370674 (https://phabricator.wikimedia.org/T445270) [15:46:30] (03CR) 10Daimona Eaytoy: [C:03+1] Enable the worklist card view in production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372611 (https://phabricator.wikimedia.org/T445217) (owner: 10Nik Gkountas) [15:47:15] (03PS2) 10Krinkle: RawAction: Reduce action=raw maxage without CSS/JSON/JS ctype to 5s [core] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1369119 [15:47:25] (03Abandoned) 10Krinkle: RawAction: Reduce action=raw maxage without CSS/JSON/JS ctype to 5s [core] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1369119 (owner: 10Krinkle) [15:48:08] (03CR) 10Klausman: [C:03+1] opensearch: Refresh images [deployment-charts] - 10https://gerrit.wikimedia.org/r/1370674 (https://phabricator.wikimedia.org/T445270) (owner: 10Bking) [15:49:51] (03CR) 10Atsuko: [C:03+1] opensearch: Refresh images [deployment-charts] - 10https://gerrit.wikimedia.org/r/1370674 (https://phabricator.wikimedia.org/T445270) (owner: 10Bking) [15:51:34] (03PS1) 10AKhatun: deployment_server::deploy_service::service_catalog: Add revision-feature-counts [puppet] - 10https://gerrit.wikimedia.org/r/1372871 (https://phabricator.wikimedia.org/T445665) [15:51:38] (03CR) 10Bking: [C:03+2] opensearch: Refresh images [deployment-charts] - 10https://gerrit.wikimedia.org/r/1370674 (https://phabricator.wikimedia.org/T445270) (owner: 10Bking) [15:52:13] !log bking@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-apifeatureusage: apply [15:52:20] !log bking@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-apifeatureusage: apply [15:52:54] !log bking@deploy2003 `charlie --services_dir dse-k8s-services -s opensearch-* -e dse-k8s-* apply` T445272 [15:52:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:52:57] !log bking@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-apifeatureusage: apply [15:53:01] !log bking@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-apifeatureusage: apply [15:53:23] !log bking@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-apifeatureusage-test: apply [15:53:30] !log bking@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-apifeatureusage-test: apply [15:53:39] (03CR) 10AKhatun: deployment_server::deploy_service::service_catalog: Add revision-feature-counts (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1372871 (https://phabricator.wikimedia.org/T445665) (owner: 10AKhatun) [15:53:45] (03PS1) 10Brouberol: kafka-jumbo-eqiad: Upscale the network_flows_internal partition count [puppet] - 10https://gerrit.wikimedia.org/r/1372876 [15:54:02] !log bking@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-apifeatureusage-test: apply [15:54:06] !log bking@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-apifeatureusage-test: apply [15:54:13] !log bking@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply [15:54:21] !log bking@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply [15:54:30] !log bking@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply [15:54:35] !log bking@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply [15:54:35] 10ops-eqiad, 06DC-Ops: Alert for device ps1-a4-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T445684#12588417 (10phaultfinder) [15:54:40] !log bking@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [15:54:47] !log bking@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [15:55:39] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: db2219 is not coming back online after reboot - https://phabricator.wikimedia.org/T445479#12588431 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm @Ladsgroup server has had the firmware updated and relavent hardware reseated. should be good. lmk if it gets w... [15:55:48] !log bking@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [15:55:52] !log bking@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [15:56:04] !log bking@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply [15:56:11] !log bking@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply [15:56:17] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T444749#12588435 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [15:56:20] !log bking@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-test: apply [15:56:25] !log bking@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-test: apply [15:56:43] !log bking@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply [15:56:50] !log bking@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply [15:58:59] !log bking@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply [15:59:03] !log bking@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply [15:59:08] !log bking@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply [15:59:15] !log bking@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply [15:59:20] !log bking@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply [15:59:24] !log bking@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply [15:59:45] !log bking@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply [15:59:52] !log bking@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply [15:59:58] !log bking@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply [16:00:02] !log bking@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply [16:00:08] !log bking@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply [16:00:16] !log bking@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply [16:00:21] !log bking@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply [16:00:25] !log bking@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply [16:02:16] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [16:03:17] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: db2219 is not coming back online after reboot - https://phabricator.wikimedia.org/T445479#12588454 (10Ladsgroup) Thanks. I just started the mariadb and it's catching up. [16:03:41] 10ops-codfw, 06SRE, 06DC-Ops, 10procurement, 06ServiceOps (Next quarter): decommission deploy2002.codfw.wmnet - https://phabricator.wikimedia.org/T426222#12588458 (10Jhancock.wm) 05In progress→03Resolved a:03Jhancock.wm [16:04:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12588480 (10phaultfinder) [16:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [16:10:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2170.codfw.wmnet, wikikube-worker2262.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2141.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-work [16:10:13] odfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2307.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2185.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-worker2177.codfw.wmnet, w [16:10:13] worker2311.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2273.codfw.wmnet, wikikube-worker2213.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [16:11:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2364.codfw.wmnet, wikikube-worker2033.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2312.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-work [16:11:03] odfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2255.codfw.wmnet, wikikube-worker2263.codfw.wmnet, wikikube-worker2302.codfw.wmnet, wikikube-worker2134.codfw.wmnet, wikikube-worker2151.codfw.wmnet, wikikube-worker2133.codfw.wmnet, wikikube-worker2327.codfw.wmnet, wikikube-worker2106.codfw.wmnet, wikikube-worker2266.codfw.wmnet, wikikube-worker2279.codfw.wmnet, wikikube-worker2144.codfw.wmnet, wikikube-worker2351.codfw.wmnet, w [16:11:03] worker2348.codfw.wmnet, wikikube-worker2342.codfw.wmnet, wikikube-worker2187.codfw.wmnet, wikikube-worker2309.codfw.wmnet, wikikube-worker2277.codfw.wmnet, wikikube-worker2038.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [16:11:11] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:11:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [16:11:42] FIRING: [3x] JobUnavailable: Reduced availability for job etherpad in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:12:22] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for gaiarajan - https://phabricator.wikimedia.org/T444625#12588489 (10gaiarajan) [16:12:52] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for gaiarajan - https://phabricator.wikimedia.org/T444625#12588498 (10gaiarajan) Hi, I just switched to a hardware key and updated the task description. Is there any other info I can provide? Thank you! [16:13:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [16:13:22] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for gaiarajan - https://phabricator.wikimedia.org/T444625#12588499 (10gaiarajan) 05Stalled→03In progress a:05gaiarajan→03hnowlan [16:14:44] (03PS1) 10Btullis: hadoop: Exit and restart the NodeManager on OutOfMemoryError [puppet] - 10https://gerrit.wikimedia.org/r/1372894 (https://phabricator.wikimedia.org/T445683) [16:15:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.35% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:16:42] FIRING: [3x] JobUnavailable: Reduced availability for job etherpad in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:20:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.35% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:21:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2306.codfw.wmnet, wikikube-worker2187.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-work [16:21:03] odfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2141.codfw.wmnet, wikikube-worker2276.codfw.wmnet, wikikube-worker2312.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2287.codfw.wmnet, w [16:21:03] worker2161.codfw.wmnet, wikikube-worker2345.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2041.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [16:23:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [16:26:25] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361194 (https://phabricator.wikimedia.org/T288223) (owner: 10Krinkle) [16:26:50] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:27:09] (03Merged) 10jenkins-bot: Discontinue xff.log [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361194 (https://phabricator.wikimedia.org/T288223) (owner: 10Krinkle) [16:27:26] !log krinkle@deploy2003 Started scap sync-world: Backport for [[gerrit:1361194|Discontinue xff.log (T288223)]] [16:27:30] T288223: Discontinue xff.log (remove $wgExtensionFunctions from wmf-config) - https://phabricator.wikimedia.org/T288223 [16:28:39] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [16:31:21] !log krinkle@deploy2003 krinkle: Backport for [[gerrit:1361194|Discontinue xff.log (T288223)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:32:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2033.codfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-work [16:32:03] odfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2261.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2311.codfw.wmnet, w [16:32:03] worker2248.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2273.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2289.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [16:32:15] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Q1:rack/setup/install mc20[56-73] - https://phabricator.wikimedia.org/T436272#12588552 (10Jhancock.wm) reminder to myself that i need to upload the license to these servers. [16:33:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [16:33:34] (03CR) 10Xcollazo: [C:03+1] "LGTM." [puppet] - 10https://gerrit.wikimedia.org/r/1372894 (https://phabricator.wikimedia.org/T445683) (owner: 10Btullis) [16:35:44] !log krinkle@deploy2003 krinkle: Continuing with deployment [16:41:48] !log krinkle@deploy2003 Finished scap sync-world: Backport for [[gerrit:1361194|Discontinue xff.log (T288223)]] (duration: 14m 22s) [16:41:52] T288223: Discontinue xff.log (remove $wgExtensionFunctions from wmf-config) - https://phabricator.wikimedia.org/T288223 [16:42:57] 06SRE, 06Infrastructure-Foundations, 10netops: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12588598 (10cmooney) 05Resolved→03Open p:05Medium→03Low Nokia support actually came back, and seem to have found a sequence of operations that can get the switch in... [16:44:41] (03CR) 10Krinkle: "I've deactivated the channel in wmf-config a few minutes ago with https://gerrit.wikimedia.org/r/c/1361194." [puppet] - 10https://gerrit.wikimedia.org/r/1370696 (https://phabricator.wikimedia.org/T288223) (owner: 10Krinkle) [16:45:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2033.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2176.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2345.codfw.wmnet, wikikube-work [16:45:03] odfw.wmnet, wikikube-worker2273.codfw.wmnet, wikikube-worker2124.codfw.wmnet, wikikube-worker2159.codfw.wmnet, wikikube-worker2206.codfw.wmnet, wikikube-worker2263.codfw.wmnet, wikikube-worker2305.codfw.wmnet, wikikube-worker2089.codfw.wmnet, wikikube-worker2302.codfw.wmnet, wikikube-worker2134.codfw.wmnet, wikikube-worker2307.codfw.wmnet, wikikube-worker2339.codfw.wmnet, wikikube-worker2179.codfw.wmnet, wikikube-worker2346.codfw.wmnet, w [16:45:03] worker2156.codfw.wmnet, wikikube-worker2018.codfw.wmnet, wikikube-worker2327.codfw.wmnet, wikikube-worker2087.codfw.wmnet, wikikube-worker2056.codfw.wmnet, wikikube-worker2370.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [16:48:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [16:50:30] FIRING: KubernetesDeploymentUnavailableReplicas: ... [16:50:30] Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s&var-namespace=editcheck-headless&var-deployment=editcheck-headless-main - ... [16:50:30] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [16:51:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2174.codfw.wmnet, wikikube-worker2283.codfw.wmnet, wikikube-worker2185.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-work [16:51:13] odfw.wmnet, wikikube-worker2305.codfw.wmnet, wikikube-worker2302.codfw.wmnet, wikikube-worker2140.codfw.wmnet, wikikube-worker2369.codfw.wmnet, wikikube-worker2114.codfw.wmnet, wikikube-worker2285.codfw.wmnet, wikikube-worker-exp2001.codfw.wmnet, wikikube-worker2344.codfw.wmnet, wikikube-worker2321.codfw.wmnet, wikikube-worker2352.codfw.wmnet, wikikube-worker2348.codfw.wmnet, wikikube-worker2115.codfw.wmnet, wikikube-worker2074.codfw.wmne [16:51:13] ube-worker2184.codfw.wmnet, wikikube-worker2152.codfw.wmnet, wikikube-worker2249.codfw.wmnet, wikikube-worker2366.codfw.wmnet, wikikube-worker2290.codfw.wmnet, wikikube-worker2015.codfw https://wikitech.wikimedia.org/wiki/PyBal [16:52:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [16:53:01] PROBLEM - Blazegraph Port for wdqs-blazegraph on wdqs1012 is CRITICAL: connect to address 127.0.0.1 and port 9999: Connection refused https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [16:54:01] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs1012 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [16:55:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2364.codfw.wmnet, wikikube-worker2141.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-work [16:55:03] odfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2176.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2198.codfw.wmnet, w [16:55:03] worker2287.codfw.wmnet, wikikube-worker2345.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2315.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [16:55:14] (03PS4) 10Krinkle: Simplify Math config, reduce mention of Mathoid, prepare for removal [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311063 (https://phabricator.wikimedia.org/T432511) [16:56:29] !log lerickson@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [16:57:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [17:00:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2170.codfw.wmnet, wikikube-worker2033.codfw.wmnet, wikikube-worker2243.codfw.wmnet, wikikube-worker2282.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2307.codfw.wmnet, wikikube-work [17:00:13] odfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2345.codfw.wmnet, wikikube-worker2273.codfw.wmnet, wikikube-worker2198.codfw.wmnet, w [17:00:13] worker2315.codfw.wmnet, wikikube-worker2322.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2041.codfw.wmnet, wikikube-worker2293.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [17:00:19] (03PS5) 10Krinkle: CommonSettings: Remove obsolete config for Math extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311063 (https://phabricator.wikimedia.org/T121100) [17:01:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [17:05:05] (03PS1) 10Danielyepezgarces: cowikimedia: Create staff-wmco user group for CampaignEvents [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372899 (https://phabricator.wikimedia.org/T445690) [17:05:55] (03PS6) 10Krinkle: CommonSettings: Remove obsolete config for Math extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311063 (https://phabricator.wikimedia.org/T121100) [17:08:13] PROBLEM - Blazegraph Port for wdqs-blazegraph on wdqs1020 is CRITICAL: connect to address 127.0.0.1 and port 9999: Connection refused https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [17:09:13] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs1020 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [17:10:13] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311063 (https://phabricator.wikimedia.org/T121100) (owner: 10Krinkle) [17:12:12] (03Merged) 10jenkins-bot: CommonSettings: Remove obsolete config for Math extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311063 (https://phabricator.wikimedia.org/T121100) (owner: 10Krinkle) [17:12:26] !log krinkle@deploy2003 Started scap sync-world: Backport for [[gerrit:1311063|CommonSettings: Remove obsolete config for Math extension (T121100 T432511)]] [17:12:31] T121100: Move research and experimental code from Math to MathSearch extension - https://phabricator.wikimedia.org/T121100 [17:12:31] T432511: Remove Mathoid/RESTBase integration from Math extension - https://phabricator.wikimedia.org/T432511 [17:14:26] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 23 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372899 (https://phabricator.wikimedia.org/T445690) (owner: 10Danielyepezgarces) [17:16:30] !log krinkle@deploy2003 krinkle: Backport for [[gerrit:1311063|CommonSettings: Remove obsolete config for Math extension (T121100 T432511)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [17:18:22] (03PS1) 10Audrey Penven: Stop skipping captcha consequence for wbsetclaim [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372900 (https://phabricator.wikimedia.org/T445575) [17:23:42] !log krinkle@deploy2003 krinkle: Continuing with deployment [17:26:01] (03PS2) 10Btullis: hadoop: Exit and restart the NodeManager on OutOfMemoryError [puppet] - 10https://gerrit.wikimedia.org/r/1372894 (https://phabricator.wikimedia.org/T445683) [17:26:01] (03PS1) 10Btullis: hadoop: Keep the YARN containers when a test NodeManager restarts [puppet] - 10https://gerrit.wikimedia.org/r/1372901 (https://phabricator.wikimedia.org/T445683) [17:26:03] (03PS1) 10Btullis: hadoop: Keep the YARN containers when a NodeManager restarts [puppet] - 10https://gerrit.wikimedia.org/r/1372902 (https://phabricator.wikimedia.org/T445683) [17:26:33] (03PS1) 10Btullis: spark: Limit the shuffle blocks in flight on the test cluster [puppet] - 10https://gerrit.wikimedia.org/r/1372903 (https://phabricator.wikimedia.org/T445683) [17:26:35] (03PS1) 10Btullis: spark: Limit the shuffle blocks in flight on analytics-hadoop [puppet] - 10https://gerrit.wikimedia.org/r/1372904 (https://phabricator.wikimedia.org/T445683) [17:28:39] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [17:28:39] (03CR) 10Btullis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1372894 (https://phabricator.wikimedia.org/T445683) (owner: 10Btullis) [17:29:46] !log krinkle@deploy2003 Finished scap sync-world: Backport for [[gerrit:1311063|CommonSettings: Remove obsolete config for Math extension (T121100 T432511)]] (duration: 17m 20s) [17:29:51] T121100: Move research and experimental code from Math to MathSearch extension - https://phabricator.wikimedia.org/T121100 [17:29:51] T432511: Remove Mathoid/RESTBase integration from Math extension - https://phabricator.wikimedia.org/T432511 [17:33:17] (03CR) 10BCornwall: [C:04-2] "I think this changes a little more than you meant to change!" [dns] - 10https://gerrit.wikimedia.org/r/1368683 (https://phabricator.wikimedia.org/T444019) (owner: 10Dzahn) [17:33:19] 10SRE-swift-storage, 10Ceph, 06Infrastructure-Foundations: Create new S3 backends for the Docker Registry service - https://phabricator.wikimedia.org/T427175#12588898 (10Ladsgroup) This is done I assume? [17:34:49] (03CR) 10BCornwall: [C:03+1] wmnet: Move the s2, s3, s4 and x4 analytics replicas to dbstore1010 [dns] - 10https://gerrit.wikimedia.org/r/1371876 (https://phabricator.wikimedia.org/T424003) (owner: 10Btullis) [17:34:53] 10SRE-swift-storage: 2024-2025 ms swift capacity - https://phabricator.wikimedia.org/T359077#12588907 (10Ladsgroup) This probably should be closed. [17:36:11] (03CR) 10BCornwall: [V:03+2 C:03+2] Additional CAA record for wikipedia.org to permit digicert certificates [dns] - 10https://gerrit.wikimedia.org/r/1370560 (https://phabricator.wikimedia.org/T440185) (owner: 10Jgreen) [17:36:24] !log brett@dns4003 START - running authdns-update [17:37:38] (03PS1) 10Krinkle: InitialiseSettings: Remove unused wmgUseRestbaseVRS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372908 (https://phabricator.wikimedia.org/T436054) [17:37:55] (03CR) 10BCornwall: [C:03+2] delete comcom.wikimedia.org, unused [dns] - 10https://gerrit.wikimedia.org/r/1368688 (https://phabricator.wikimedia.org/T444019) (owner: 10Dzahn) [17:38:28] (03CR) 10BCornwall: [V:03+1] "PCC SUCCESS (NOOP 1 CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/" [puppet] - 10https://gerrit.wikimedia.org/r/1370050 (https://phabricator.wikimedia.org/T50835) (owner: 10Krinkle) [17:38:31] (03CR) 10JHathaway: [C:03+2] rake_modules: run rubocop when .rubocop.yml changes [puppet] - 10https://gerrit.wikimedia.org/r/1348717 (owner: 10Hashar) [17:38:43] !log brett@dns4003 END - running authdns-update [17:39:04] (03CR) 10BCornwall: [C:03+1] Add new discovery records for multiple services [dns] - 10https://gerrit.wikimedia.org/r/1363323 (owner: 10Clément Goubert) [17:39:50] (03CR) 10BCornwall: [C:03+1] wdqs: add discovery records for the wdqs2 internal releases [dns] - 10https://gerrit.wikimedia.org/r/1369395 (https://phabricator.wikimedia.org/T436375) (owner: 10Gmodena) [17:40:05] (03CR) 10Btullis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1372903 (https://phabricator.wikimedia.org/T445683) (owner: 10Btullis) [17:40:08] (03CR) 10Btullis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1372904 (https://phabricator.wikimedia.org/T445683) (owner: 10Btullis) [17:40:51] (03PS2) 10Krinkle: InitialiseSettings: Remove unused wmgUseRestbaseVRS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372908 (https://phabricator.wikimedia.org/T262315) [17:40:53] (03PS1) 10Krinkle: Remove unused "restbase" service from $wmgLocalServices [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372909 (https://phabricator.wikimedia.org/T262315) [17:41:45] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [17:44:22] (03CR) 10Krinkle: "https://codesearch.wmcloud.org/search/?q=%28LocalServices%7CAllServices%7CMasterServices%29.*restbase&files=&excludeFiles=&repos=" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372909 (https://phabricator.wikimedia.org/T262315) (owner: 10Krinkle) [17:44:43] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372908 (https://phabricator.wikimedia.org/T262315) (owner: 10Krinkle) [17:44:44] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372909 (https://phabricator.wikimedia.org/T262315) (owner: 10Krinkle) [17:45:05] Krinkle: On a Friday? [17:46:16] (03Merged) 10jenkins-bot: InitialiseSettings: Remove unused wmgUseRestbaseVRS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372908 (https://phabricator.wikimedia.org/T262315) (owner: 10Krinkle) [17:46:20] (03Merged) 10jenkins-bot: Remove unused "restbase" service from $wmgLocalServices [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372909 (https://phabricator.wikimedia.org/T262315) (owner: 10Krinkle) [17:46:38] !log krinkle@deploy2003 Started scap sync-world: Backport for [[gerrit:1372908|InitialiseSettings: Remove unused wmgUseRestbaseVRS (T262315 T436054)]], [[gerrit:1372909|Remove unused "restbase" service from $wmgLocalServices (T262315 T436054)]] [17:46:43] T262315: Sunset RESTBase (tracking) - https://phabricator.wikimedia.org/T262315 [17:46:44] T436054: Remove $wgVirtualRestConfig from MediaWiki - https://phabricator.wikimedia.org/T436054 [17:47:16] (03CR) 10Dzahn: [C:04-2] "ugh yea. thanks. will amend" [dns] - 10https://gerrit.wikimedia.org/r/1368683 (https://phabricator.wikimedia.org/T444019) (owner: 10Dzahn) [17:47:49] (03CR) 10Audrey Penven: [C:04-1] "The first patchset was written under the mistaken assumption that with hcaptcha support being added to wikidata, there is no reason to pre" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372900 (https://phabricator.wikimedia.org/T445575) (owner: 10Audrey Penven) [17:48:01] PROBLEM - Blazegraph Port for wdqs-blazegraph on wdqs1011 is CRITICAL: connect to address 127.0.0.1 and port 9999: Connection refused https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [17:49:01] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs1011 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [17:51:03] !log krinkle@deploy2003 krinkle: Backport for [[gerrit:1372908|InitialiseSettings: Remove unused wmgUseRestbaseVRS (T262315 T436054)]], [[gerrit:1372909|Remove unused "restbase" service from $wmgLocalServices (T262315 T436054)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [17:52:28] !log krinkle@deploy2003 krinkle: Continuing with deployment [17:52:50] James_F: one var unused for 2 months, the other unused for 2 weeks. pretty straight forward yeah. [17:53:08] someone else can have the honours of shutting down mathoid and restbase itself. [17:53:17] slated for 10 days from now [17:54:32] (03CR) 10Klausman: [C:03+1] hadoop: Exit and restart the NodeManager on OutOfMemoryError [puppet] - 10https://gerrit.wikimedia.org/r/1372894 (https://phabricator.wikimedia.org/T445683) (owner: 10Btullis) [17:54:41] (03CR) 10Klausman: [C:03+1] hadoop: Keep the YARN containers when a NodeManager restarts [puppet] - 10https://gerrit.wikimedia.org/r/1372902 (https://phabricator.wikimedia.org/T445683) (owner: 10Btullis) [17:54:50] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs/main-internal on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [17:55:46] (03CR) 10Klausman: [C:03+1] hadoop: Keep the YARN containers when a test NodeManager restarts [puppet] - 10https://gerrit.wikimedia.org/r/1372901 (https://phabricator.wikimedia.org/T445683) (owner: 10Btullis) [17:56:09] (03CR) 10Klausman: [C:03+1] wmnet: Move the s2, s3, s4 and x4 analytics replicas to dbstore1010 [dns] - 10https://gerrit.wikimedia.org/r/1371876 (https://phabricator.wikimedia.org/T424003) (owner: 10Btullis) [17:58:28] !log krinkle@deploy2003 Finished scap sync-world: Backport for [[gerrit:1372908|InitialiseSettings: Remove unused wmgUseRestbaseVRS (T262315 T436054)]], [[gerrit:1372909|Remove unused "restbase" service from $wmgLocalServices (T262315 T436054)]] (duration: 11m 50s) [17:58:33] T262315: Sunset RESTBase (tracking) - https://phabricator.wikimedia.org/T262315 [17:58:34] T436054: Remove $wgVirtualRestConfig from MediaWiki - https://phabricator.wikimedia.org/T436054 [18:00:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2251.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2187.codfw.wmnet, wikikube-worker2329.codfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-work [18:00:13] odfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2033.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2273.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2314.codfw.wmnet, w [18:00:13] worker2125.codfw.wmnet, wikikube-worker2301.codfw.wmnet, wikikube-worker2355.codfw.wmnet, wikikube-worker2002.codfw.wmnet, wikikube-worker2159.codfw.wmnet, wikikube-worker2062.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [18:01:08] Krinkle: Dropping services requires SRE to kill off the alerting, sadly, otherwise I’d have just done it. [18:01:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:07:49] (03CR) 10Audrey Penven: "@dreamyjazzwikipedia@gmail.com - the goal is to allow AbuseFilter's `showcaptcha` to have an effect on Wikidata (`wikidatawiki` and `testw" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372900 (https://phabricator.wikimedia.org/T445575) (owner: 10Audrey Penven) [18:09:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2177.codfw.wmnet, wikikube-worker2187.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2261.codfw.wmnet, wikikube-work [18:09:13] odfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2289.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2125.codfw.wmnet, wikikube-worker2301.codfw.wmnet, wikikube-worker2159.codfw.wmnet, wikikube-worker2255.codfw.wmnet, wikikube-worker2090.codfw.wmnet, wikikube-worker2305.codfw.wmnet, wikikube-worker2134.codfw.wmnet, w [18:09:13] worker2272.codfw.wmnet, wikikube-worker2140.codfw.wmnet, wikikube-worker2143.codfw.wmnet, wikikube-worker2050.codfw.wmnet, wikikube-worker2307.codfw.wmnet, wikikube-worker2296.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [18:10:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2170.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2249.codfw.wmnet, wikikube-worker2255.codfw.wmnet, wikikube-work [18:10:03] odfw.wmnet, wikikube-worker2134.codfw.wmnet, wikikube-worker2257.codfw.wmnet, wikikube-worker2143.codfw.wmnet, wikikube-worker2329.codfw.wmnet, wikikube-worker2014.codfw.wmnet, wikikube-worker2326.codfw.wmnet, wikikube-worker2332.codfw.wmnet, wikikube-worker2266.codfw.wmnet, wikikube-worker2283.codfw.wmnet, wikikube-worker2292.codfw.wmnet, wikikube-worker2279.codfw.wmnet, wikikube-worker2298.codfw.wmnet, wikikube-worker2163.codfw.wmnet, w [18:10:03] worker2074.codfw.wmnet, wikikube-worker2130.codfw.wmnet, wikikube-worker2160.codfw.wmnet, wikikube-worker2142.codfw.wmnet, wikikube-worker2265.codfw.wmnet, wikikube-worker2006.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [18:10:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:11:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:17:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2174.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2261.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2041.codfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-work [18:17:13] odfw.wmnet, wikikube-worker2055.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-worker2106.codfw.wmnet, wikikube-worker2344.codfw.wmnet, wikikube-worker2208.codfw.wmnet, wikikube-worker2104.codfw.wmnet, wikikube-worker2146.codfw.wmnet, wikikube-worker2267.codfw.wmnet, wikikube-worker2012.codfw.wmnet, wikikube-worker2260.codfw.wmnet, wikikube-worker2189.codfw.wmnet, wikikube-worker2371.codfw.wmnet, wikikube-worker2194.codfw.wmnet, w [18:17:13] worker2137.codfw.wmnet, wikikube-worker2354.codfw.wmnet, wikikube-worker2074.codfw.wmnet, wikikube-worker2152.codfw.wmnet, wikikube-worker2108.codfw.wmnet, wikikube-worker2291.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [18:17:20] (03PS2) 10Stoyofuku-wmf: admin: Add FIDO SSH keys for toyofuku [puppet] - 10https://gerrit.wikimedia.org/r/1371861 (https://phabricator.wikimedia.org/T445548) [18:17:30] (03CR) 10Marostegui: "Key has been verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1371861 (https://phabricator.wikimedia.org/T445548) (owner: 10Stoyofuku-wmf) [18:18:30] (03CR) 10Marostegui: [C:03+2] admin: Add FIDO SSH keys for toyofuku [puppet] - 10https://gerrit.wikimedia.org/r/1371861 (https://phabricator.wikimedia.org/T445548) (owner: 10Stoyofuku-wmf) [18:19:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:19:48] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for toyofuku - https://phabricator.wikimedia.org/T445548#12589104 (10Marostegui) 05Open→03Resolved I've merged the change. Please allow around 20 minutes for the change to spread across production. [18:20:04] (03PS1) 10Bking: opensearch: Fix invalid image link [deployment-charts] - 10https://gerrit.wikimedia.org/r/1372921 (https://phabricator.wikimedia.org/T445272) [18:28:52] (03CR) 10Bking: [C:03+2] opensearch: Fix invalid image link [deployment-charts] - 10https://gerrit.wikimedia.org/r/1372921 (https://phabricator.wikimedia.org/T445272) (owner: 10Bking) [18:28:58] (03CR) 10Btullis: [C:03+1] opensearch: Fix invalid image link [deployment-charts] - 10https://gerrit.wikimedia.org/r/1372921 (https://phabricator.wikimedia.org/T445272) (owner: 10Bking) [18:29:41] (03PS1) 10Dzahn: etherpad: scale etherpad-next down to 0 replicas [deployment-charts] - 10https://gerrit.wikimedia.org/r/1372924 [18:30:13] (03CR) 10Dzahn: [C:03+2] etherpad: scale etherpad-next down to 0 replicas [deployment-charts] - 10https://gerrit.wikimedia.org/r/1372924 (owner: 10Dzahn) [18:30:54] (03Merged) 10jenkins-bot: opensearch: Fix invalid image link [deployment-charts] - 10https://gerrit.wikimedia.org/r/1372921 (https://phabricator.wikimedia.org/T445272) (owner: 10Bking) [18:32:09] !log bking@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [18:32:14] (03Merged) 10jenkins-bot: etherpad: scale etherpad-next down to 0 replicas [deployment-charts] - 10https://gerrit.wikimedia.org/r/1372924 (owner: 10Dzahn) [18:32:22] !log bking@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [18:34:17] !log dzahn@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply [18:34:32] !log dzahn@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply [18:34:35] !log dzahn@deploy2003 helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/etherpad: apply [18:34:46] !log dzahn@deploy2003 helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/etherpad: apply [18:35:53] PROBLEM - Blazegraph Port for wdqs-blazegraph on wdqs1016 is CRITICAL: connect to address 127.0.0.1 and port 9999: Connection refused https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:36:53] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs1016 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:38:42] !log dzahn@deploy2003 helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/etherpad: apply [18:38:46] !log dzahn@deploy2003 helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/etherpad: apply [18:39:37] FIRING: GnmiInterfaceCountersDrop: ... [18:39:38] cr3-ulsfo is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=cr3-ulsfo:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [18:41:14] !log disabled etherpad-next.wikimedia.org by setting "replicas: 0" on it on aux-k8s cluster. (https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1372924) [18:41:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:41:46] 06SRE, 10SRE-Access-Requests: Requesting access to Analytics Data Lake for sancheza/SanchezA-WMF - https://phabricator.wikimedia.org/T443850#12589188 (10SanchezA-WMF) If https://phabricator.wikimedia.org/T443636 is enough to get access to Turnillo and Superset we can close this ticket. [18:46:32] !log dzahn@dns1004 START - running authdns-update [18:47:02] !log DNS deleting comcom.wikimedia.org ( T444019) - deploy https://gerrit.wikimedia.org/r/c/operations/dns/+/1368688 [18:47:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:47:06] T444019: Cleanup unused DNS subdomains - https://phabricator.wikimedia.org/T444019 [18:48:12] 06SRE, 10DNS, 06Traffic, 13Patch-For-Review: Cleanup unused DNS subdomains - https://phabricator.wikimedia.org/T444019#12589212 (10Dzahn) [18:48:51] !log dzahn@dns1004 END - running authdns-update [18:50:19] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:52:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2036.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2279.codfw.wmnet, wikikube-worker2345.codfw.wmnet, wikikube-worker2041.codfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-worker2134.codfw.wmnet, wikikube-worker2272.codfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-work [18:52:03] odfw.wmnet, wikikube-worker2105.codfw.wmnet, wikikube-worker2285.codfw.wmnet, wikikube-worker2087.codfw.wmnet, wikikube-worker2259.codfw.wmnet, wikikube-worker2333.codfw.wmnet, wikikube-worker2332.codfw.wmnet, wikikube-worker2126.codfw.wmnet, wikikube-worker2013.codfw.wmnet, wikikube-worker2176.codfw.wmnet, wikikube-worker2260.codfw.wmnet, wikikube-worker2194.codfw.wmnet, wikikube-worker2294.codfw.wmnet, wikikube-worker2074.codfw.wmnet, w [18:52:03] worker2187.codfw.wmnet, wikikube-worker2304.codfw.wmnet, wikikube-worker2360.codfw.wmnet, wikikube-worker2042.codfw.wmnet, wikikube-worker2093.codfw.wmnet, wikikube-worker2128.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [18:52:07] PROBLEM - PyBal IPVS diff check on lvs1019 is CRITICAL: CRITICAL: Mismatch between IPVS and PyBal https://wikitech.wikimedia.org/wiki/PyBal [18:52:58] (03PS3) 10Dzahn: Revert "Add DNS entries for initiativeswiki" [dns] - 10https://gerrit.wikimedia.org/r/1368683 (https://phabricator.wikimedia.org/T444019) [18:53:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:53:19] !log bking@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply [18:53:27] (03CR) 10Dzahn: [C:03+1] "should be ok now but no rush" [dns] - 10https://gerrit.wikimedia.org/r/1368683 (https://phabricator.wikimedia.org/T444019) (owner: 10Dzahn) [18:53:29] !log bking@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply [18:54:11] (03Abandoned) 10Dzahn: insetup: remove role::insetup::collaboration_services_ferm [puppet] - 10https://gerrit.wikimedia.org/r/1369418 (https://phabricator.wikimedia.org/T444758) (owner: 10Dzahn) [18:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [18:55:54] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [18:56:08] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [18:56:14] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [18:57:05] RECOVERY - PyBal IPVS diff check on lvs1019 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [18:57:17] (03CR) 10Dzahn: [C:03+1] "docker::gc is only declared in profile::gitlab::runner. other services not affected." [puppet] - 10https://gerrit.wikimedia.org/r/1372039 (https://phabricator.wikimedia.org/T443833) (owner: 10Ahmon Dancy) [19:03:47] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply [19:03:52] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply [19:04:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12589273 (10phaultfinder) [19:06:22] (03CR) 10Dzahn: [C:03+2] "fixes a real bug - mostly for shared runners to not keep losing their job caches" [puppet] - 10https://gerrit.wikimedia.org/r/1372039 (https://phabricator.wikimedia.org/T443833) (owner: 10Ahmon Dancy) [19:07:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2364.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2273.codfw.wmnet, wikikube-worker2164.codfw.wmnet, wikikube-worker2326.codfw.wmnet, wikikube-work [19:07:13] odfw.wmnet, wikikube-worker2351.codfw.wmnet, wikikube-worker2352.codfw.wmnet, wikikube-worker2154.codfw.wmnet, wikikube-worker2203.codfw.wmnet, wikikube-worker2359.codfw.wmnet, wikikube-worker2256.codfw.wmnet, wikikube-worker2365.codfw.wmnet, wikikube-worker2015.codfw.wmnet, wikikube-worker2325.codfw.wmnet, wikikube-worker2058.codfw.wmnet, wikikube-worker2250.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [19:08:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:09:50] FIRING: [3x] HelmReleaseBadStatus: Helm release wdqs/main-internal on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [19:11:58] 06SRE, 10DNS, 06Traffic, 13Patch-For-Review: Cleanup unused DNS subdomains - https://phabricator.wikimedia.org/T444019#12589287 (10Dzahn) @ssingh The current way where I upload changes, got a review and then deployed myself was perfect. That being said I only do the ones that are obvious to me and are wik... [19:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 21d 23h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [19:15:16] (03CR) 10Dzahn: "The commit message says Comment.soy gets removed, but PS4 only changes ChangeSubject.soy. Is the removal still meant to be part of this ch" [puppet] - 10https://gerrit.wikimedia.org/r/1371818 (https://phabricator.wikimedia.org/T43608) (owner: 10Paladox) [19:17:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:21:02] (03CR) 10Dzahn: [C:03+2] zuul: Update image refs for 14.3.0-1 release [puppet] - 10https://gerrit.wikimedia.org/r/1371904 (https://phabricator.wikimedia.org/T445248) (owner: 10Dduvall) [19:23:28] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [19:23:38] James_F: I'll make sure to invite you to the bonfire :) [19:24:50] FIRING: [3x] HelmReleaseBadStatus: Helm release wdqs/main-internal on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [19:34:33] !log zuul1004 - sudo systemctl restart zuul-scheduler zuul-launcher zuul-web [19:34:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:34:44] !log zuul1005 - sudo systemctl restart zuul-executor [19:34:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:37:26] !log forcing major compaction, sessionstore[2004-2006].codfw.wmnet,sessionstore[1004-1006].eqiad.wmnet — T445661 [19:37:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:37:31] T445661: sessionstore: rapid increase in utilization - https://phabricator.wikimedia.org/T445661 [19:39:03] !log zuul1005: mv /var/lib/zuul/ansible-bin to ansible-bin.py311 (old python3.11 venvs broke zuul-executor on the 14.3.0-1 image), restarted zuul-executor - T445248 [19:39:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:39:07] T445248: Build and deploy Zuul 14.3.0 - https://phabricator.wikimedia.org/T445248 [19:44:50] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs/main-internal on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [19:45:44] (03CR) 10Dzahn: [C:03+2] "Deployed the 14.3.0-1 images (1371904) on zuul1004 and zuul1005." [puppet] - 10https://gerrit.wikimedia.org/r/1371904 (https://phabricator.wikimedia.org/T445248) (owner: 10Dduvall) [19:46:34] (03CR) 10JHathaway: [C:03+2] rubocop: enable Lint/UnneededDisable [puppet] - 10https://gerrit.wikimedia.org/r/1348795 (https://phabricator.wikimedia.org/T441847) (owner: 10Hashar) [19:47:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2033.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2328.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-work [19:47:03] odfw.wmnet, wikikube-worker2179.codfw.wmnet, wikikube-worker2042.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2273.codfw.wmnet, w [19:47:03] worker2315.codfw.wmnet, wikikube-worker2289.codfw.wmnet, wikikube-worker2060.codfw.wmnet, wikikube-worker2159.codfw.wmnet, wikikube-worker2355.codfw.wmnet, wikikube-worker2255.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [19:47:08] (03CR) 10JHathaway: [C:03+2] rubocop: upgrade and target ruby 2.7 [puppet] - 10https://gerrit.wikimedia.org/r/1348797 (https://phabricator.wikimedia.org/T441847) (owner: 10Hashar) [19:49:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:51:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2191.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2328.codfw.wmnet, wikikube-worker2185.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2125.codfw.wmnet, wikikube-worker2002.codfw.wmnet, wikikube-worker2159.codfw.wmnet, wikikube-work [19:51:13] odfw.wmnet, wikikube-worker2089.codfw.wmnet, wikikube-worker2062.codfw.wmnet, wikikube-worker2164.codfw.wmnet, wikikube-worker2282.codfw.wmnet, wikikube-worker2307.codfw.wmnet, wikikube-worker2369.codfw.wmnet, wikikube-worker2268.codfw.wmnet, wikikube-worker2075.codfw.wmnet, wikikube-worker2127.codfw.wmnet, wikikube-worker2087.codfw.wmnet, wikikube-worker2259.codfw.wmnet, wikikube-worker2333.codfw.wmnet, wikikube-worker2267.codfw.wmnet, w [19:51:13] worker2283.codfw.wmnet, wikikube-worker2176.codfw.wmnet, wikikube-worker2109.codfw.wmnet, wikikube-worker2212.codfw.wmnet, wikikube-worker2371.codfw.wmnet, wikikube-worker2194.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [19:52:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:54:50] FIRING: [3x] HelmReleaseBadStatus: Helm release wdqs/main-internal on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [19:55:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2033.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2283.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-work [19:55:13] odfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker2345.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2289.codfw.wmnet, w [19:55:13] worker2314.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2196.codfw.wmnet, wikikube-worker2124.codfw.wmnet, wikikube-worker2206.codfw.wmnet, wikikube-worker2089.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [19:56:15] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:00:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2356.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2143.codfw.wmnet, wikikube-worker2329.codfw.wmnet, wikikube-worker2307.codfw.wmnet, wikikube-worker2179.codfw.wmnet, wikikube-work [20:00:03] odfw.wmnet, wikikube-worker2104.codfw.wmnet, wikikube-worker2294.codfw.wmnet, wikikube-worker2095.codfw.wmnet, wikikube-worker2310.codfw.wmnet, wikikube-worker2145.codfw.wmnet, wikikube-worker2277.codfw.wmnet, wikikube-worker2131.codfw.wmnet, wikikube-worker2148.codfw.wmnet, wikikube-worker2077.codfw.wmnet, wikikube-worker2001.codfw.wmnet, wikikube-worker2070.codfw.wmnet, wikikube-worker2250.codfw.wmnet, wikikube-worker2061.codfw.wmnet ar [20:00:03] down but pooled https://wikitech.wikimedia.org/wiki/PyBal [20:01:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:02:16] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [20:02:45] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade rubocop ruby linter on operations/puppet - https://phabricator.wikimedia.org/T441847#12589461 (10jhathaway) [20:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [20:10:37] PROBLEM - MariaDB Replica Lag: s1 on db2250 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 622.48 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:11:03] PROBLEM - MariaDB Replica Lag: s8 on db2198 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 648.13 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:11:11] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:11:38] (03CR) 10Bking: [C:03+1] hadoop: Keep the YARN containers when a test NodeManager restarts [puppet] - 10https://gerrit.wikimedia.org/r/1372901 (https://phabricator.wikimedia.org/T445683) (owner: 10Btullis) [20:12:43] (03CR) 10Bking: [C:03+1] hadoop: Keep the YARN containers when a NodeManager restarts [puppet] - 10https://gerrit.wikimedia.org/r/1372902 (https://phabricator.wikimedia.org/T445683) (owner: 10Btullis) [20:13:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2273.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2322.codfw.wmnet, wikikube-worker2060.codfw.wmnet, wikikube-work [20:13:13] odfw.wmnet, wikikube-worker2002.codfw.wmnet, wikikube-worker2159.codfw.wmnet, wikikube-worker2164.codfw.wmnet, wikikube-worker2302.codfw.wmnet, wikikube-worker2339.codfw.wmnet, wikikube-worker2327.codfw.wmnet, wikikube-worker2259.codfw.wmnet, wikikube-worker2106.codfw.wmnet, wikikube-worker2013.codfw.wmnet, wikikube-worker2283.codfw.wmnet, wikikube-worker2138.codfw.wmnet, wikikube-worker2144.codfw.wmnet, wikikube-worker2109.codfw.wmnet, w [20:13:13] worker2367.codfw.wmnet, wikikube-worker2034.codfw.wmnet, wikikube-worker2095.codfw.wmnet, wikikube-worker2342.codfw.wmnet, wikikube-worker2152.codfw.wmnet, wikikube-worker2360.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [20:14:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:16:15] (03CR) 10JHathaway: [C:03+2] ipmi: ruby: stop using then for multiline when statement [puppet] - 10https://gerrit.wikimedia.org/r/1348873 (https://phabricator.wikimedia.org/T441847) (owner: 10Hashar) [20:16:40] 06SRE, 10Wikimedia-Mailing-lists: New WikiLatinos Mailing List - https://phabricator.wikimedia.org/T437742#12589542 (10Dzahn) Thank you, Ladsgroup :) [20:17:17] 06SRE, 10SRE-Access-Requests: Requesting access to releasers-mediawiki for osleger - https://phabricator.wikimedia.org/T443655#12589543 (10Dzahn) Thanks for closing this out, Manuel. [20:24:02] (03PS5) 10JHathaway: rubocop: upgrade to 0.82.0 [puppet] - 10https://gerrit.wikimedia.org/r/1348876 (https://phabricator.wikimedia.org/T443798) (owner: 10Hashar) [20:24:05] (03CR) 10JHathaway: [C:03+2] rubocop: upgrade to 0.82.0 [puppet] - 10https://gerrit.wikimedia.org/r/1348876 (https://phabricator.wikimedia.org/T443798) (owner: 10Hashar) [20:28:58] FIRING: [2x] ProbeDown: Service text-https:443 has failed probes (http_text-https_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:28:58] FIRING: [2x] ProbeDown: Service text-https:443 has failed probes (http_text-https_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:29:49] what's happening [20:30:13] couple people on Discord reported general slowness starting 4 minutes ago [20:32:25] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:33:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2170.codfw.wmnet, wikikube-worker2262.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2307.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-work [20:33:13] odfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker2345.codfw.wmnet, w [20:33:13] worker2213.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2289.codfw.wmnet, wikikube-worker2314.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [20:33:45] AntiComposite: thanks, and yes [20:33:57] RESOLVED: [5x] ProbeDown: Service text-https:443 has failed probes (http_text-https_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:34:01] RESOLVED: [3x] ProbeDown: Service commons.wikimedia.org:443 has failed probes (http_commons_wikimedia_org_ip4) #page - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:34:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:35:50] !incidents [20:35:51] 8757 (ACKED, 26h 21m old) Manual (paged) by Catrope (roan@wikimedia.org): T445505 [20:35:51] 8758 (ACKED) Manual (paged) by rsilvola (riku.silvola@wikimedia.org): see security@ [20:35:51] 8759 (RESOLVED) [2x] ProbeDown sre (text-https:443 probes/service drmrs) [20:37:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2141.codfw.wmnet, wikikube-worker2033.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-work [20:37:03] odfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2289.codfw.wmnet, wikikube-worker2314.codfw.wmnet, wikikube-worker2160.codfw.wmnet, wikikube-worker2355.codfw.wmnet, wikikube-worker2263.codfw.wmnet, wikikube-worker2305.codfw.wmnet, wikikube-worker2322.codfw.wmnet, wikikube-worker2302.codfw.wmnet, w [20:37:03] worker2134.codfw.wmnet, wikikube-worker2257.codfw.wmnet, wikikube-worker2272.codfw.wmnet, wikikube-worker2143.codfw.wmnet, wikikube-worker2050.codfw.wmnet, wikikube-worker2149.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [20:38:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:39:04] 06SRE, 10MediaWiki-General, 06ServiceOps, 07Epic, 13Patch-For-Review: Introduce a global load shedding level - https://phabricator.wikimedia.org/T443236#12589630 (10Ladsgroup) The patch above basically explains how I'm thinking of implementing it. It has claude on it but I basically spelled out the whole... [20:40:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2364.codfw.wmnet, wikikube-worker2141.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2283.codfw.wmnet, wikikube-work [20:40:13] odfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2322.codfw.wmnet, w [20:40:13] worker2314.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2060.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2125.codfw.wmnet, wikikube-worker2196.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [20:42:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:42:25] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:46:30] (03CR) 10Ryan Kemper: [C:03+1] "LGTM, one commit msg nit" [puppet] - 10https://gerrit.wikimedia.org/r/1372902 (https://phabricator.wikimedia.org/T445683) (owner: 10Btullis) [20:48:48] (03CR) 10Ryan Kemper: [C:03+1] kafka-jumbo-eqiad: Upscale the network_flows_internal partition count [puppet] - 10https://gerrit.wikimedia.org/r/1372876 (owner: 10Brouberol) [20:50:30] FIRING: KubernetesDeploymentUnavailableReplicas: ... [20:50:30] Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s&var-namespace=editcheck-headless&var-deployment=editcheck-headless-main - ... [20:50:30] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [20:51:53] (03CR) 10Ryan Kemper: [C:03+2] kafka-jumbo-eqiad: Upscale the network_flows_internal partition count [puppet] - 10https://gerrit.wikimedia.org/r/1372876 (owner: 10Brouberol) [20:52:41] 06SRE, 10DNS, 06Traffic, 13Patch-For-Review: Cleanup unused DNS subdomains - https://phabricator.wikimedia.org/T444019#12589661 (10bking) `matomo-next.wikimedia.org` will soon be used, see T436003. [[ https://gerrit.wikimedia.org/r/c/operations/dns/+/1352139 | The Gerrit change that added this record ]] h... [21:00:37] RECOVERY - MariaDB Replica Lag: s1 on db2250 is OK: OK slave_sql_lag Replication lag: 0.40 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:05:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2141.codfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2185.codfw.wmnet, wikikube-worker2312.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-work [21:05:13] odfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2314.codfw.wmnet, wikikube-worker2041.codfw.wmnet, wikikube-worker2206.codfw.wmnet, wikikube-worker2089.codfw.wmnet, wikikube-worker2062.codfw.wmnet, w [21:05:13] worker2140.codfw.wmnet, wikikube-worker2143.codfw.wmnet, wikikube-worker2050.codfw.wmnet, wikikube-worker2147.codfw.wmnet, wikikube-worker2369.codfw.wmnet, wikikube-worker2151.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [21:06:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2170.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2306.codfw.wmnet, wikikube-worker2199.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-work [21:06:03] odfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2345.codfw.wmnet, wikikube-worker2157.codfw.wmnet, w [21:06:03] worker2198.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2289.codfw.wmnet, wikikube-worker2314.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [21:07:03] RECOVERY - MariaDB Replica Lag: s8 on db2198 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:07:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [21:08:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [21:09:32] 06SRE, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work, 07SecTeam-Processed: Build/deploy new Flink OCI (docker) image - https://phabricator.wikimedia.org/T443032#12589810 (10Gehel) [21:10:43] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Q1:rack/setup/install 10 new ceph nodes - https://phabricator.wikimedia.org/T438213#12589832 (10Gehel) [21:11:30] (03CR) 10Dzahn: [V:03+1 C:03+2] "https://puppet-compiler.wmflabs.org/output/1369416/8428/" [puppet] - 10https://gerrit.wikimedia.org/r/1369416 (https://phabricator.wikimedia.org/T444758) (owner: 10Dzahn) [21:11:36] (03PS2) 10Dzahn: peopleweb: remove the unused deployment_server param [puppet] - 10https://gerrit.wikimedia.org/r/1369416 (https://phabricator.wikimedia.org/T444758) [21:14:31] 06SRE, 10DNS, 06Traffic, 13Patch-For-Review: Cleanup unused DNS subdomains - https://phabricator.wikimedia.org/T444019#12589853 (10Dzahn) Thanks! I am going to check it off on the list. [21:14:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12589854 (10phaultfinder) [21:15:07] 06SRE, 10DNS, 06Traffic, 13Patch-For-Review: Cleanup unused DNS subdomains - https://phabricator.wikimedia.org/T444019#12589855 (10Dzahn) [21:17:05] (03CR) 10Dzahn: [C:03+2] peopleweb: remove the unused deployment_server param [puppet] - 10https://gerrit.wikimedia.org/r/1369416 (https://phabricator.wikimedia.org/T444758) (owner: 10Dzahn) [21:21:19] PROBLEM - MariaDB Replica Lag: x3 on db2200 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 651.11 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:31:19] RECOVERY - MariaDB Replica Lag: x3 on db2200 is OK: OK slave_sql_lag Replication lag: 0.02 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:33:02] (03CR) 10Dzahn: [C:03+2] peopleweb: drop the extra rsync package and double blank lines [puppet] - 10https://gerrit.wikimedia.org/r/1369417 (https://phabricator.wikimedia.org/T444758) (owner: 10Dzahn) [21:33:10] (03PS2) 10Dzahn: peopleweb: drop the extra rsync package and double blank lines [puppet] - 10https://gerrit.wikimedia.org/r/1369417 (https://phabricator.wikimedia.org/T444758) [21:35:53] PROBLEM - MariaDB Replica Lag: s4 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 640.30 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:36:51] (03CR) 10Dzahn: [C:03+2] peopleweb: drop the extra rsync package and double blank lines [puppet] - 10https://gerrit.wikimedia.org/r/1369417 (https://phabricator.wikimedia.org/T444758) (owner: 10Dzahn) [21:38:17] (03CR) 10BCornwall: [C:03+1] varnish: Add /esi/ URL rewrites [puppet] - 10https://gerrit.wikimedia.org/r/1366066 (https://phabricator.wikimedia.org/T443703) (owner: 10Vgutierrez) [21:41:56] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [21:42:07] (03CR) 10Dzahn: [C:03+1] "please check again :)" [dns] - 10https://gerrit.wikimedia.org/r/1368683 (https://phabricator.wikimedia.org/T444019) (owner: 10Dzahn) [21:42:17] (03PS1) 10DCausse: admin: add spare fido key for dcausse [puppet] - 10https://gerrit.wikimedia.org/r/1372981 [21:43:38] (03CR) 10Dzahn: [C:03+1] etherpad: update offline page, remove nodejs incident link [puppet] - 10https://gerrit.wikimedia.org/r/1372702 (owner: 10Jelto) [21:44:50] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs/main-internal on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [21:52:49] (03PS2) 10Dzahn: admin: add spare fido key for dcausse [puppet] - 10https://gerrit.wikimedia.org/r/1372981 (https://phabricator.wikimedia.org/T442758) (owner: 10DCausse) [21:54:36] (03CR) 10Dzahn: [C:03+2] ""No change expected" is wrong, clearly. But doesn't mean the change itself isn't good." [puppet] - 10https://gerrit.wikimedia.org/r/1369412 (https://phabricator.wikimedia.org/T444758) (owner: 10Dzahn) [21:54:50] FIRING: [3x] HelmReleaseBadStatus: Helm release wdqs/main-internal on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [21:57:37] PROBLEM - MariaDB Replica Lag: x4 on db2250 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 621.45 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:09:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2353.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2185.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-work [22:09:13] odfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-worker2002.codfw.wmnet, wikikube-worker2090.codfw.wmnet, wikikube-worker2050.codfw.wmnet, wikikube-worker2214.codfw.wmnet, wikikube-worker2252.codfw.wmnet, wikikube-worker2013.codfw.wmnet, wikikube-worker2106.codfw.wmnet, wikikube-worker2344.codfw.wmnet, wikikube-worker2283.codfw.wmnet, wikikube-worker2258.codfw.wmnet, wikikube-worker2129.codfw.wmnet, wikikube-worker2316.codfw.wmnet, w [22:09:13] worker2342.codfw.wmnet, wikikube-worker2341.codfw.wmnet, wikikube-worker2264.codfw.wmnet, wikikube-worker2291.codfw.wmnet, wikikube-worker2193.codfw.wmnet, wikikube-worker2188.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [22:10:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:11:13] (03PS1) 10Cwhite: logstash: Move all ecs logs onto default partition [puppet] - 10https://gerrit.wikimedia.org/r/1372986 (https://phabricator.wikimedia.org/T445720) [22:11:15] (03PS1) 10Cwhite: logstash: Switch ecs logs to use daily indexes [puppet] - 10https://gerrit.wikimedia.org/r/1372987 (https://phabricator.wikimedia.org/T445720) [22:18:36] (03PS1) 10Dzahn: Revert "planet: remove the bookworm-only xml settings from the feed configs" [puppet] - 10https://gerrit.wikimedia.org/r/1372988 [22:20:28] (03CR) 10Dzahn: [C:03+2] Revert "planet: remove the bookworm-only xml settings from the feed configs" [puppet] - 10https://gerrit.wikimedia.org/r/1372988 (owner: 10Dzahn) [22:32:43] (03CR) 10Dzahn: [C:03+2] "it was wrong and broke things. when called out it claimed it never checked ://" [puppet] - 10https://gerrit.wikimedia.org/r/1369412 (https://phabricator.wikimedia.org/T444758) (owner: 10Dzahn) [22:33:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2276.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2141.codfw.wmnet, wikikube-worker2033.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2046.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-work [22:33:13] odfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2345.codfw.wmnet, wikikube-worker2157.codfw.wmnet, w [22:33:13] worker2273.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2289.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [22:34:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:39:38] FIRING: GnmiInterfaceCountersDrop: ... [22:39:38] cr3-ulsfo is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=cr3-ulsfo:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [22:43:53] RECOVERY - MariaDB Replica Lag: s4 on db2239 is OK: OK slave_sql_lag Replication lag: 0.24 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:50:19] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [22:50:37] RECOVERY - MariaDB Replica Lag: x4 on db2250 is OK: OK slave_sql_lag Replication lag: 0.37 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [23:13:39] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 611.41 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [23:14:47] (03CR) 10Rafka Aditia: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361803 (https://phabricator.wikimedia.org/T443174) (owner: 10Rafka Aditia) [23:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 21d 19h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [23:21:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.333% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [23:40:30] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [23:59:39] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.41 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response