[00:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T0000) [00:00:05] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [00:00:15] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [00:00:18] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [00:00:25] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [00:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:05:29] jouncebot: nowandnext [00:05:30] For the next 0 hour(s) and 54 minute(s): Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T0000) [00:05:30] In 1 hour(s) and 54 minute(s): Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T0200) [00:06:01] (03CR) 10Zabe: [C:03+2] Disable automatic user page watching on loginwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360928 (https://phabricator.wikimedia.org/T442779) (owner: 10Zabe) [00:06:48] (03Merged) 10jenkins-bot: Disable automatic user page watching on loginwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360928 (https://phabricator.wikimedia.org/T442779) (owner: 10Zabe) [00:07:15] !log zabe@deploy2003 Started scap sync-world: Backport for [[gerrit:1360928|Disable automatic user page watching on loginwiki (T442779)]] [00:07:19] T442779: Stop users watching their own userpage on creation on loginwiki - https://phabricator.wikimedia.org/T442779 [00:10:31] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 623.30 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:15:22] !log zabe@deploy2003 zabe: Backport for [[gerrit:1360928|Disable automatic user page watching on loginwiki (T442779)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [00:15:26] T442779: Stop users watching their own userpage on creation on loginwiki - https://phabricator.wikimedia.org/T442779 [00:16:51] !log zabe@deploy2003 zabe: Continuing with deployment [00:24:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12536751 (10phaultfinder) [00:25:40] !log zabe@deploy2003 Finished scap sync-world: Backport for [[gerrit:1360928|Disable automatic user page watching on loginwiki (T442779)]] (duration: 18m 24s) [00:25:43] T442779: Stop users watching their own userpage on creation on loginwiki - https://phabricator.wikimedia.org/T442779 [00:36:47] (03PS1) 10Zabe: Move reads/writes of wbc_entity_usage to x1 for wikidatawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367981 (https://phabricator.wikimedia.org/T439753) [00:38:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [00:39:01] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1367983 [00:39:01] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1367983 (owner: 10TrainBranchBot) [00:41:22] (03CR) 10Zabe: [C:03+2] Move reads/writes of wbc_entity_usage to x1 for enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361059 (https://phabricator.wikimedia.org/T439753) (owner: 10Zabe) [00:42:21] (03Merged) 10jenkins-bot: Move reads/writes of wbc_entity_usage to x1 for enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361059 (https://phabricator.wikimedia.org/T439753) (owner: 10Zabe) [00:42:58] !log zabe@deploy2003 Started scap sync-world: Backport for [[gerrit:1361059|Move reads/writes of wbc_entity_usage to x1 for enwiki (T439753)]] [00:43:01] T439753: Switch reads/writes of wbc_entity_usage from local cluster to x1 - https://phabricator.wikimedia.org/T439753 [00:43:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [00:46:36] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1367983 (owner: 10TrainBranchBot) [00:47:05] !log zabe@deploy2003 zabe: Backport for [[gerrit:1361059|Move reads/writes of wbc_entity_usage to x1 for enwiki (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [00:47:35] !log zabe@deploy2003 zabe: Continuing with deployment [00:48:31] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.26 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:54:45] !log zabe@deploy2003 Finished scap sync-world: Backport for [[gerrit:1361059|Move reads/writes of wbc_entity_usage to x1 for enwiki (T439753)]] (duration: 11m 47s) [00:54:48] T439753: Switch reads/writes of wbc_entity_usage from local cluster to x1 - https://phabricator.wikimedia.org/T439753 [01:06:48] !log run populateLuAttachmentMethod.php # T442322 [01:06:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:06:51] T442322: Run populateLuAttachmentMethod.php in wmf production - https://phabricator.wikimedia.org/T442322 [01:08:59] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1367989 [01:08:59] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1367989 (owner: 10TrainBranchBot) [01:09:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12536860 (10phaultfinder) [01:15:01] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1367989 (owner: 10TrainBranchBot) [01:28:33] RECOVERY - OSPF status on cr2-magru is OK: OSPFv2: 3/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [01:39:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12536927 (10phaultfinder) [01:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.17% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [02:00:04] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T0200) [02:00:51] !log mwpresync@deploy2003 Started scap build-images: Publishing wmf/next image [02:08:27] !log mwpresync@deploy2003 Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) [02:08:45] (03PS1) 10TrainBranchBot: Branch commit for wmf/1.48.0-wmf.4 [core] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1367994 (https://phabricator.wikimedia.org/T442222) [02:08:47] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/1.48.0-wmf.4 [core] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1367994 (https://phabricator.wikimedia.org/T442222) (owner: 10TrainBranchBot) [02:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:14:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12536968 (10phaultfinder) [02:15:11] (03Merged) 10jenkins-bot: Branch commit for wmf/1.48.0-wmf.4 [core] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1367994 (https://phabricator.wikimedia.org/T442222) (owner: 10TrainBranchBot) [02:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:34:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12536993 (10phaultfinder) [02:41:07] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [02:47:06] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [02:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [02:54:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12537005 (10phaultfinder) [03:00:05] Deploy window Automatic branching of MediaWiki, extensions, skins, and vendor – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T0300) [03:00:34] (03PS2) 10Anzx: tcywiki: revert 10 anniversary logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367997 (https://phabricator.wikimedia.org/T434176) [03:01:42] (03PS1) 10Anzx: bjnwiktionary: update logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367999 (https://phabricator.wikimedia.org/T444085) [03:01:54] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 10 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367999 (https://phabricator.wikimedia.org/T444085) (owner: 10Anzx) [03:02:41] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 10 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367997 (https://phabricator.wikimedia.org/T434176) (owner: 10Anzx) [03:15:23] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [03:19:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12537008 (10phaultfinder) [03:54:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12537023 (10phaultfinder) [04:00:05] Deploy window Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T0400) [04:02:04] (03PS1) 10TrainBranchBot: testwikis to 1.48.0-wmf.4 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368007 (https://phabricator.wikimedia.org/T442222) [04:02:06] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by mwpresync@deploy2003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368007 (https://phabricator.wikimedia.org/T442222) (owner: 10TrainBranchBot) [04:02:52] (03Merged) 10jenkins-bot: testwikis to 1.48.0-wmf.4 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368007 (https://phabricator.wikimedia.org/T442222) (owner: 10TrainBranchBot) [04:03:21] !log mwpresync@deploy2003 Started scap sync-world: testwikis to 1.48.0-wmf.4 refs T442222 [04:03:25] T442222: 1.48.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T442222 [04:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:09:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12537030 (10phaultfinder) [04:39:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12537055 (10phaultfinder) [04:42:39] !log mwpresync@deploy2003 Finished scap sync-world: testwikis to 1.48.0-wmf.4 refs T442222 (duration: 39m 18s) [04:42:42] T442222: 1.48.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T442222 [05:00:04] Deploy window Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T0500) [05:02:28] !log mwpresync@deploy2003 Pruned MediaWiki: 1.47.0-wmf.24 (duration: 02m 26s) [05:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.21% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [06:41:07] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:47:06] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [06:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [06:52:18] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12537283 (10Jhancock.wm) the power is not distributing between between redundant and primary power evenly. even though all the servers in this rack should be. I'll... [06:52:21] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12537284 (10Jhancock.wm) a:03Jhancock.wm [07:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T0700) [07:00:05] marostegui, cezmunsta, and federico3: It is that lovely time of the day again! You are hereby commanded to deploy Primary database switchover. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T0700). [07:15:23] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [07:30:41] (03PS3) 10Anzx: tcywiki: revert 10 anniversary logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367997 (https://phabricator.wikimedia.org/T434176) [07:40:06] (03PS1) 10Ozge: ml-serve: Deny KServe model management paths at the ingress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368150 (https://phabricator.wikimedia.org/T443921) [07:45:18] (03CR) 10CI reject: [V:04-1] ml-serve: Deny KServe model management paths at the ingress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368150 (https://phabricator.wikimedia.org/T443921) (owner: 10Ozge) [07:52:19] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1367800 (owner: 10L10n-bot) [07:54:43] hey folks I need to start my deploy a little early today, hope that's ok [07:55:05] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdrewniak@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367950 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [07:55:29] (03PS1) 10Ozge: ml-serve: Deny KServe model management paths at the ingress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368162 (https://phabricator.wikimedia.org/T443921) [07:55:54] (03Merged) 10jenkins-bot: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367950 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [07:56:13] jan_drewniak: will you be doing the whole window or just your patch? [07:56:37] chlod: hi sorry I only have time to do my patch [07:56:48] all good :) i'll handle the rest then [07:56:54] !log jdrewniak@deploy2003 Started scap sync-world: Backport for [[gerrit:1367950|Bumping portals to master (T128546)]] [07:56:57] T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics - https://phabricator.wikimedia.org/T128546 [07:57:14] (03Abandoned) 10Ozge: ml-serve: Deny KServe model management paths at the ingress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368150 (https://phabricator.wikimedia.org/T443921) (owner: 10Ozge) [08:00:05] Amir1, urbanecm, awight, and chlod: #bothumor My software never has bugs. It just develops random features. Rise for UTC morning backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T0800). [08:00:05] jan_drewniak and anzx: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [08:00:26] o/ (I too will have a patch, but working out CI issues still) [08:03:26] !log jdrewniak@deploy2003 jdrewniak: Backport for [[gerrit:1367950|Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:03:29] T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics - https://phabricator.wikimedia.org/T128546 [08:03:54] (03PS1) 10Matthias Mullie: Squashed diff to master [extensions/ReaderExperiments] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1368165 [08:04:24] !log jdrewniak@deploy2003 jdrewniak: Continuing with deployment [08:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:04:49] (03PS1) 10Matthias Mullie: Squashed diff to master [extensions/ReaderExperiments] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368166 [08:06:10] (03CR) 10CI reject: [V:04-1] Squashed diff to master [extensions/ReaderExperiments] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368166 (owner: 10Matthias Mullie) [08:09:04] chlod: o/ [08:09:10] hello hello [08:09:17] waiting for an ongoing deploy to finish [08:14:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12537406 (10phaultfinder) [08:15:19] !log jdrewniak@deploy2003 Finished scap sync-world: Backport for [[gerrit:1367950|Bumping portals to master (T128546)]] (duration: 18m 26s) [08:15:23] T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics - https://phabricator.wikimedia.org/T128546 [08:15:55] chlod: I'm done [08:16:01] awesome, thanks! [08:16:02] anzx: i see you submitted these patches together so i'm also deploying them together; that also probably prevents a merge conflict [08:16:32] Sure [08:17:01] (03CR) 10TrainBranchBot: [C:03+2] "Approved by chlod@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367997 (https://phabricator.wikimedia.org/T434176) (owner: 10Anzx) [08:17:02] (03CR) 10TrainBranchBot: [C:03+2] "Approved by chlod@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367999 (https://phabricator.wikimedia.org/T444085) (owner: 10Anzx) [08:17:51] (03Merged) 10jenkins-bot: tcywiki: revert 10 anniversary logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367997 (https://phabricator.wikimedia.org/T434176) (owner: 10Anzx) [08:23:11] urk, gate-and-submit isn't running for one of them [08:23:52] (03CR) 10Chlod Alejandro: [C:03+2] bjnwiktionary: update logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367999 (https://phabricator.wikimedia.org/T444085) (owner: 10Anzx) [08:25:28] hmm, i'll rebase [08:25:52] (03PS2) 10Anzx: bjnwiktionary: update logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367999 (https://phabricator.wikimedia.org/T444085) [08:27:00] (03CR) 10TrainBranchBot: "Approved by chlod@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367999 (https://phabricator.wikimedia.org/T444085) (owner: 10Anzx) [08:27:48] (03Merged) 10jenkins-bot: bjnwiktionary: update logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367999 (https://phabricator.wikimedia.org/T444085) (owner: 10Anzx) [08:28:13] !log chlod@deploy2003 Started scap sync-world: Backport for [[gerrit:1367997|tcywiki: revert 10 anniversary logo (T434176)]], [[gerrit:1367999|bjnwiktionary: update logos (T444085)]] [08:28:18] T434176: Requesting temporary logo change for tcy.wikipedia.org (WP10) - https://phabricator.wikimedia.org/T434176 [08:28:19] T444085: Implement Wiktionary Banjar wordmark/tagline for bjn.wiktionary.org - https://phabricator.wikimedia.org/T444085 [08:35:26] !log chlod@deploy2003 chlod, anzx: Backport for [[gerrit:1367997|tcywiki: revert 10 anniversary logo (T434176)]], [[gerrit:1367999|bjnwiktionary: update logos (T444085)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:35:30] T434176: Requesting temporary logo change for tcy.wikipedia.org (WP10) - https://phabricator.wikimedia.org/T434176 [08:35:31] T444085: Implement Wiktionary Banjar wordmark/tagline for bjn.wiktionary.org - https://phabricator.wikimedia.org/T444085 [08:35:32] Looking [08:35:37] ack [08:36:38] chlod: looks good, ok to continue [08:36:42] awesome :) [08:36:47] !log chlod@deploy2003 chlod, anzx: Continuing with deployment [08:38:01] (03CR) 10Jgiannelos: [C:03+2] thumbor: Pin async image in staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367869 (https://phabricator.wikimedia.org/T431767) (owner: 10Jgiannelos) [08:39:11] (03Merged) 10jenkins-bot: thumbor: Pin async image in staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367869 (https://phabricator.wikimedia.org/T431767) (owner: 10Jgiannelos) [08:39:12] i think the minor hiccup earlier was caused by one of the patches relying on an older patchset of the other [08:39:28] i should probably write that down somewhere if it isn't already written down [08:41:14] !log jgiannelos@deploy2003 helmfile [staging] START helmfile.d/services/thumbor: apply [08:41:25] !log jgiannelos@deploy2003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [08:42:40] !log ayounsi@cumin1004 START - Cookbook sre.dns.admin DNS admin: depool esams [reason: network upgrade, T440349] [08:42:44] T440349: esams router and switch maint - https://phabricator.wikimedia.org/T440349 [08:42:47] !log ayounsi@cumin1004 END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool esams [reason: network upgrade, T440349] [08:43:39] !log ayounsi@cumin1004 START - Cookbook sre.dns.admin DNS admin: depool esams [reason: network upgrade, T440349] [08:43:45] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: network upgrade, T440349] [08:43:49] (03CR) 10MVernon: [C:03+1] Icinga: added myself to the following groups, as part of on-boarding: [puppet] - 10https://gerrit.wikimedia.org/r/1367843 (owner: 10Lucian) [08:43:57] !log chlod@deploy2003 Finished scap sync-world: Backport for [[gerrit:1367997|tcywiki: revert 10 anniversary logo (T434176)]], [[gerrit:1367999|bjnwiktionary: update logos (T444085)]] (duration: 15m 43s) [08:44:01] T434176: Requesting temporary logo change for tcy.wikipedia.org (WP10) - https://phabricator.wikimedia.org/T434176 [08:44:01] T444085: Implement Wiktionary Banjar wordmark/tagline for bjn.wiktionary.org - https://phabricator.wikimedia.org/T444085 [08:44:09] one sec while i purge some of the old logos [08:44:15] chlod: logo looks good without debug, thanks for deploying [08:44:24] np :) [08:44:55] (03CR) 10MVernon: [C:03+1] "ssh key confirmed out of band." [puppet] - 10https://gerrit.wikimedia.org/r/1364304 (owner: 10Lucian) [08:46:20] matthiasmullie: all yours, sorry for the long wait! 🙇 [08:47:20] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy2003 using scap backport" [extensions/ReaderExperiments] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1368165 (owner: 10Matthias Mullie) [08:47:26] haha no worries - thanks! [08:48:40] (03Merged) 10jenkins-bot: Squashed diff to master [extensions/ReaderExperiments] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1368165 (owner: 10Matthias Mullie) [08:48:47] !log mlitn@deploy2003 Started scap sync-world: Backport for [[gerrit:1368165|Squashed diff to master]] (1.48.0-wmf.3) [08:53:37] (03CR) 10MVernon: [C:03+2] Icinga: added myself to the following groups, as part of on-boarding: [puppet] - 10https://gerrit.wikimedia.org/r/1367843 (owner: 10Lucian) [08:55:07] !log mlitn@deploy2003 mlitn: Backport for [[gerrit:1368165|Squashed diff to master]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:55:53] !log mlitn@deploy2003 mlitn: Continuing with deployment [08:59:22] (03CR) 10Jgiannelos: [C:03+2] Keep temp files in a directory that only exists at runtime [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1364701 (https://phabricator.wikimedia.org/T431768) (owner: 10Ladsgroup-claude) [09:00:05] andre and brennen: gettimeofday() says it's time for MediaWiki train - Utc-0+Utc-7 Version. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T0900) [09:00:11] o/ [09:00:13] no hurry [09:01:53] !log mlitn@deploy2003 Finished scap sync-world: Backport for [[gerrit:1368165|Squashed diff to master]] (1.48.0-wmf.3) (duration: 13m 06s) [09:02:52] !log UTC morning backport window done [09:02:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:03:03] yay, thanks [09:09:03] (03Merged) 10jenkins-bot: Keep temp files in a directory that only exists at runtime [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1364701 (https://phabricator.wikimedia.org/T431768) (owner: 10Ladsgroup-claude) [09:09:04] (03PS1) 10Muehlenhoff: Record LDAP access for fsesini [puppet] - 10https://gerrit.wikimedia.org/r/1368220 [09:10:23] (03CR) 10Muehlenhoff: [C:03+2] Record LDAP access for fsesini [puppet] - 10https://gerrit.wikimedia.org/r/1368220 (owner: 10Muehlenhoff) [09:15:35] (03PS1) 10TrainBranchBot: group0 to 1.48.0-wmf.4 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368224 (https://phabricator.wikimedia.org/T442222) [09:15:38] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by aklapper@deploy2003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368224 (https://phabricator.wikimedia.org/T442222) (owner: 10TrainBranchBot) [09:16:22] (03Merged) 10jenkins-bot: group0 to 1.48.0-wmf.4 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368224 (https://phabricator.wikimedia.org/T442222) (owner: 10TrainBranchBot) [09:17:53] (03PS1) 10Jelto: Add FIDO spare key for milimetric [puppet] - 10https://gerrit.wikimedia.org/r/1368225 (https://phabricator.wikimedia.org/T442758) [09:18:50] (03PS5) 10Gmodena: wdqs: add WDQS v2 (qlever) alerts [alerts] - 10https://gerrit.wikimedia.org/r/1360953 (https://phabricator.wikimedia.org/T436018) [09:19:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12537848 (10phaultfinder) [09:21:09] (03CR) 10Jelto: [C:03+2] Add FIDO spare key for milimetric [puppet] - 10https://gerrit.wikimedia.org/r/1368225 (https://phabricator.wikimedia.org/T442758) (owner: 10Jelto) [09:21:49] (03PS1) 10Ozge: admin: Add FIDO SSH key for ozge [puppet] - 10https://gerrit.wikimedia.org/r/1368227 [09:22:10] !log set graceful shutdown on cr1-esams transit/ix bgp - T440349 [09:22:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:22:13] T440349: esams router and switch maint - https://phabricator.wikimedia.org/T440349 [09:24:59] (03CR) 10Jelto: [C:03+2] Remove old ssh key for cparle [puppet] - 10https://gerrit.wikimedia.org/r/1367886 (https://phabricator.wikimedia.org/T442758) (owner: 10Cparle) [09:25:24] (03CR) 10Jelto: [C:03+2] admin: removed non FIDO SSH key for esanders [puppet] - 10https://gerrit.wikimedia.org/r/1367891 (https://phabricator.wikimedia.org/T442758) (owner: 10Esanders) [09:27:40] (03CR) 10Gmodena: wdqs: add WDQS v2 (qlever) alerts (032 comments) [alerts] - 10https://gerrit.wikimedia.org/r/1360953 (https://phabricator.wikimedia.org/T436018) (owner: 10Gmodena) [09:28:55] !log aklapper@deploy2003 rebuilt and synchronized wikiversions files: group0 to 1.48.0-wmf.4 refs T442222 [09:28:58] T442222: 1.48.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T442222 [09:34:36] (03PS2) 10Ozge: admin: Add FIDO SSH key for ozge [puppet] - 10https://gerrit.wikimedia.org/r/1368227 (https://phabricator.wikimedia.org/T442758) [09:36:08] powering down cr1-esams very soon [09:37:25] (03CR) 10Jgiannelos: [C:03+2] Revert "Build image with async support in thumbor" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1367864 (owner: 10Jgiannelos) [09:39:29] jouncebot: nowandnext [09:39:29] For the next 1 hour(s) and 20 minute(s): MediaWiki train - Utc-0+Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T0900) [09:39:29] In 1 hour(s) and 20 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1100) [09:39:42] I wait for now then [09:45:02] PROBLEM - OSPF status on cr2-esams is CRITICAL: OSPFv2: 2/4 UP : OSPFv3: 2/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:45:30] PROBLEM - Host re0.cr1-esams.mgmt is DOWN: PING CRITICAL - Packet loss = 100% [09:45:32] PROBLEM - OSPF status on cr1-eqiad is CRITICAL: OSPFv2: 6/7 UP : OSPFv3: 6/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:45:34] PROBLEM - OSPF status on cr1-codfw is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:45:34] (03CR) 10Trueg: [C:03+1] wdqs: add WDQS v2 (qlever) alerts (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1360953 (https://phabricator.wikimedia.org/T436018) (owner: 10Gmodena) [09:46:10] FIRING: [2x] BFDdown: BFD session down between cr1-codfw and 185.15.59.139 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:46:35] PROBLEM - Host cr1-esams is DOWN: PING CRITICAL - Packet loss = 100% [09:46:39] FIRING: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-esams (185.15.59.139) - group Confed_esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_esams&var-bgp_neighbor=cr1-esams - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [09:46:50] I think that's intentional? [09:46:59] esams is depooled last I saw [09:47:10] https://grafana.wikimedia.org/d/000000093/cdn-frontend-network?from=now-2d&to=now&timezone=utc&var-site=$__all [09:47:23] !incidents [09:47:23] 8731 (UNACKED) Host cr1-esams [09:47:35] I've just powered off my laptop because my taxi's due soonish. Are we OK? [09:47:35] XioNoX: okay if I ack it? [09:47:38] topranks, XioNoX [09:47:56] they should be working on esams today [09:48:00] yeah, you can ack it it's "expected" [09:48:02] !ack 8731 [09:48:02] 8731 (ACKED) Host cr1-esams [09:48:08] PROBLEM - Host cr1-esams IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [09:48:17] I added a silence to all the paging alert in esams [09:48:25] we all got paged :) [09:48:27] but I guess that's icinga so it didn't get the silence..... [09:48:57] anyway, continuing the work [09:50:57] If we get more pages on esams, I'll ack it. Don't worry [09:51:00] FIRING: [2x] PingLossPercent: Blackbox probe packet loss 12.5% from codfw to esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=esams - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [09:51:10] (03Merged) 10jenkins-bot: Revert "Build image with async support in thumbor" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1367864 (owner: 10Jgiannelos) [09:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.24% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [10:10:34] RECOVERY - OSPF status on cr1-eqiad is OK: OSPFv2: 7/7 UP : OSPFv3: 7/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:10:34] RECOVERY - OSPF status on cr1-codfw is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:10:47] RECOVERY - Host cr1-esams is UP: PING OK - Packet loss = 0%, RTA = 77.47 ms [10:11:10] FIRING: [2x] BFDdown: BFD session down between cr1-codfw and 185.15.59.139 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:11:39] RESOLVED: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-esams (185.15.59.139) - group Confed_esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_esams&var-bgp_neighbor=cr1-esams - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [10:13:50] (03PS1) 10Jelto: Remove old ssh key for ebysans, mfossati, sfaci and ecarg [puppet] - 10https://gerrit.wikimedia.org/r/1368244 (https://phabricator.wikimedia.org/T442758) [10:13:58] RECOVERY - Host cr1-esams IPv6 is UP: PING OK - Packet loss = 0%, RTA = 83.01 ms [10:16:10] RESOLVED: [2x] BFDdown: BFD session down between cr1-codfw and 185.15.59.139 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:16:40] (03CR) 10Jelto: [C:03+2] Remove old ssh key for ebysans, mfossati, sfaci and ecarg [puppet] - 10https://gerrit.wikimedia.org/r/1368244 (https://phabricator.wikimedia.org/T442758) (owner: 10Jelto) [10:16:48] (03PS5) 10Hashar: cinderutils: honor mount_options when mountpoints exist [puppet] - 10https://gerrit.wikimedia.org/r/1343081 [10:16:49] (03PS7) 10Hashar: cinderutils: support appending mount options [puppet] - 10https://gerrit.wikimedia.org/r/1343082 [10:16:49] (03PS4) 10Hashar: ci: set noatime on castor /srv partition [puppet] - 10https://gerrit.wikimedia.org/r/1343083 [10:18:09] (03CR) 10CI reject: [V:04-1] cinderutils: honor mount_options when mountpoints exist [puppet] - 10https://gerrit.wikimedia.org/r/1343081 (owner: 10Hashar) [10:18:32] PROBLEM - OSPF status on cr1-eqiad is CRITICAL: OSPFv2: 7/7 UP : OSPFv3: 6/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:18:34] PROBLEM - OSPF status on cr1-codfw is CRITICAL: OSPFv2: 8/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:19:10] FIRING: [2x] BFDdown: BFD session down between cr1-codfw and 185.15.59.139 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:19:20] PROBLEM - Host tcp-proxy3002 is DOWN: PING CRITICAL - Packet loss = 100% [10:19:26] PROBLEM - Host install3004 is DOWN: PING CRITICAL - Packet loss = 100% [10:19:26] PROBLEM - Host tcp-proxy3001 is DOWN: PING CRITICAL - Packet loss = 100% [10:19:34] RECOVERY - OSPF status on cr1-eqiad is OK: OSPFv2: 7/7 UP : OSPFv3: 7/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:19:34] RECOVERY - OSPF status on cr1-codfw is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:19:54] (03CR) 10CI reject: [V:04-1] cinderutils: support appending mount options [puppet] - 10https://gerrit.wikimedia.org/r/1343082 (owner: 10Hashar) [10:20:10] PROBLEM - Host doh3005 is DOWN: PING CRITICAL - Packet loss = 100% [10:20:10] PROBLEM - Host durum3006 is DOWN: PING CRITICAL - Packet loss = 100% [10:20:16] PROBLEM - Host durum3005 is DOWN: PING CRITICAL - Packet loss = 100% [10:20:16] PROBLEM - Host ncredir3006 is DOWN: PING CRITICAL - Packet loss = 100% [10:20:16] PROBLEM - Host netflow3004 is DOWN: PING CRITICAL - Packet loss = 100% [10:20:16] PROBLEM - Host ncredir3005 is DOWN: PING CRITICAL - Packet loss = 100% [10:20:20] PROBLEM - Host asw1-by27-esams is DOWN: PING CRITICAL - Packet loss = 100% [10:20:24] PROBLEM - Host hcaptcha-proxy3001 is DOWN: PING CRITICAL - Packet loss = 100% [10:20:28] PROBLEM - Host hcaptcha-proxy3002 is DOWN: PING CRITICAL - Packet loss = 100% [10:20:36] RECOVERY - Host doh3005 is UP: PING OK - Packet loss = 0%, RTA = 88.29 ms [10:20:38] RECOVERY - Host asw1-by27-esams is UP: PING OK - Packet loss = 0%, RTA = 95.63 ms [10:20:38] RECOVERY - Host durum3006 is UP: PING OK - Packet loss = 0%, RTA = 88.30 ms [10:20:44] RECOVERY - Host durum3005 is UP: PING OK - Packet loss = 0%, RTA = 88.41 ms [10:20:46] RECOVERY - Host tcp-proxy3002 is UP: PING OK - Packet loss = 0%, RTA = 88.43 ms [10:20:46] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for ecarg - https://phabricator.wikimedia.org/T443758#12538195 (10Jelto) 05Stalled→03Resolved a:03Jelto This is resolved, old key was removed in T442758 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1368244. [10:20:48] PROBLEM - Host 2a02:ec80:300:2:185:15:59:34 is DOWN: PING CRITICAL - Packet loss = 100% [10:20:52] RECOVERY - Host install3004 is UP: PING OK - Packet loss = 0%, RTA = 88.47 ms [10:20:54] RECOVERY - Host tcp-proxy3001 is UP: PING OK - Packet loss = 0%, RTA = 88.51 ms [10:20:56] RECOVERY - Host hcaptcha-proxy3002 is UP: PING OK - Packet loss = 0%, RTA = 88.37 ms [10:20:56] RECOVERY - Host hcaptcha-proxy3001 is UP: PING OK - Packet loss = 0%, RTA = 88.49 ms [10:21:20] RECOVERY - Host netflow3004 is UP: PING OK - Packet loss = 0%, RTA = 88.48 ms [10:21:20] RECOVERY - Host ncredir3006 is UP: PING OK - Packet loss = 0%, RTA = 88.35 ms [10:21:20] RECOVERY - Host ncredir3005 is UP: PING OK - Packet loss = 0%, RTA = 88.37 ms [10:22:26] PROBLEM - Host asw1-by27-esams IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [10:22:35] FIRING: Not accepting/receiving prefixes from anycast BGP peer: Alert for device asw1-by27-esams.mgmt.esams.wmnet - Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [10:23:58] RECOVERY - Host 2a02:ec80:300:2:185:15:59:34 is UP: PING OK - Packet loss = 0%, RTA = 87.97 ms [10:24:10] RESOLVED: [2x] BFDdown: BFD session down between cr1-codfw and 185.15.59.139 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:26:55] (03CR) 10Jelto: [C:03+2] Remove old ssh key for dwisehaupt [puppet] - 10https://gerrit.wikimedia.org/r/1367907 (https://phabricator.wikimedia.org/T442759) (owner: 10Dwisehaupt) [10:27:28] RECOVERY - Host asw1-by27-esams IPv6 is UP: PING OK - Packet loss = 0%, RTA = 88.35 ms [10:27:31] FIRING: [2x] Not accepting/receiving prefixes from anycast BGP peer: Alert for device asw1-bw27-esams.mgmt.esams.wmnet - Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [10:27:47] !incidents [10:27:47] 8731 (RESOLVED) Host cr1-esams [10:28:02] RECOVERY - OSPF status on cr2-esams is OK: OSPFv2: 3/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:29:16] PROBLEM - Host durum3005 is DOWN: PING CRITICAL - Packet loss = 100% [10:29:16] PROBLEM - Host tcp-proxy3002 is DOWN: PING CRITICAL - Packet loss = 100% [10:29:18] PROBLEM - Host doh3005 is DOWN: PING CRITICAL - Packet loss = 100% [10:29:24] PROBLEM - Host install3004 is DOWN: PING CRITICAL - Packet loss = 100% [10:29:26] PROBLEM - Host tcp-proxy3001 is DOWN: PING CRITICAL - Packet loss = 100% [10:29:26] PROBLEM - Host hcaptcha-proxy3002 is DOWN: PING CRITICAL - Packet loss = 100% [10:29:26] PROBLEM - Host hcaptcha-proxy3001 is DOWN: PING CRITICAL - Packet loss = 100% [10:29:30] PROBLEM - Host asw1-by27-esams is DOWN: PING CRITICAL - Packet loss = 100% [10:29:52] PROBLEM - Host ncredir3006 is DOWN: PING CRITICAL - Packet loss = 100% [10:29:52] PROBLEM - Host netflow3004 is DOWN: PING CRITICAL - Packet loss = 100% [10:29:52] PROBLEM - Host ncredir3005 is DOWN: PING CRITICAL - Packet loss = 100% [10:30:02] PROBLEM - Host bast3007 is DOWN: PING CRITICAL - Packet loss = 100% [10:30:08] PROBLEM - Host durum3006 is DOWN: PING CRITICAL - Packet loss = 100% [10:30:16] PROBLEM - Router interfaces on mr1-esams is CRITICAL: CRITICAL: host 185.15.59.130, interfaces up: 34, down: 1, dormant: 0, excluded: 0, unused: 0: https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [10:32:30] PROBLEM - Host asw1-by27-esams.mgmt is DOWN: PING CRITICAL - Packet loss = 100% [10:33:50] PROBLEM - Host asw1-by27-esams IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [10:35:55] !log ayounsi@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 39 hosts with reason: Switch maintenance [10:37:29] !log ayounsi@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt with reason: Switch maintenance [10:37:54] (03CR) 10Mark Bergsma: [C:03+1] Add lucian to the ops-limited group [puppet] - 10https://gerrit.wikimedia.org/r/1364304 (owner: 10Lucian) [10:41:07] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:42:24] FIRING: SystemdUnitFailed: netbox_ganeti_esams03_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:46:28] RECOVERY - Host asw1-by27-esams.mgmt is UP: PING OK - Packet loss = 0%, RTA = 88.31 ms [10:47:06] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [10:47:31] RESOLVED: Not accepting/receiving prefixes from anycast BGP peer: Device asw1-by27-esams.mgmt.esams.wmnet recovered from Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [10:47:50] RECOVERY - Host asw1-by27-esams is UP: PING OK - Packet loss = 0%, RTA = 92.19 ms [10:48:16] RECOVERY - Router interfaces on mr1-esams is OK: OK: host 185.15.59.130, interfaces up: 35, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [10:48:40] RECOVERY - Host doh3005 is UP: PING WARNING - Packet loss = 77%, RTA = 88.39 ms [10:48:40] RECOVERY - Host durum3006 is UP: PING OK - Packet loss = 0%, RTA = 88.39 ms [10:48:43] FIRING: [5x] RipeAtlasAnchorUnreachable: ipv4 ping to esams RIPE Atlas anchor: failures over threshold for measurement 59935536 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [10:48:46] RECOVERY - Host durum3005 is UP: PING OK - Packet loss = 0%, RTA = 88.20 ms [10:48:46] RECOVERY - Host tcp-proxy3002 is UP: PING OK - Packet loss = 0%, RTA = 88.45 ms [10:48:52] RECOVERY - Host install3004 is UP: PING OK - Packet loss = 0%, RTA = 88.41 ms [10:48:54] RECOVERY - Host tcp-proxy3001 is UP: PING OK - Packet loss = 0%, RTA = 88.61 ms [10:48:56] RECOVERY - Host hcaptcha-proxy3002 is UP: PING OK - Packet loss = 0%, RTA = 88.44 ms [10:48:56] RECOVERY - Host hcaptcha-proxy3001 is UP: PING OK - Packet loss = 0%, RTA = 92.79 ms [10:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [10:49:12] RECOVERY - Host ncredir3006 is UP: PING OK - Packet loss = 0%, RTA = 88.47 ms [10:49:16] RECOVERY - Host asw1-by27-esams IPv6 is UP: PING OK - Packet loss = 0%, RTA = 91.65 ms [10:49:20] RECOVERY - Host netflow3004 is UP: PING OK - Packet loss = 0%, RTA = 88.33 ms [10:49:20] RECOVERY - Host ncredir3005 is UP: PING OK - Packet loss = 0%, RTA = 88.38 ms [10:49:30] RECOVERY - Host bast3007 is UP: PING OK - Packet loss = 0%, RTA = 89.93 ms [10:50:02] PROBLEM - OSPF status on cr2-esams is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:51:00] RESOLVED: [2x] PingLossPercent: Blackbox probe packet loss 3.75% from codfw to esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=esams - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [10:52:00] (03PS2) 10Btullis: spark-operator: Watch the analytics and team Spark namespaces [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367770 (https://phabricator.wikimedia.org/T432322) [10:53:16] PROBLEM - Router interfaces on mr1-esams is CRITICAL: CRITICAL: host 185.15.59.130, interfaces up: 34, down: 1, dormant: 0, excluded: 0, unused: 0: https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [10:53:43] FIRING: [8x] RipeAtlasAnchorUnreachable: ipv4 ping to esams RIPE Atlas anchor: failures over threshold for measurement 59935536 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [10:54:41] !log ayounsi@cumin1004 START - Cookbook sre.dns.netbox [10:57:19] !log ayounsi@cumin1004 END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [10:57:24] RESOLVED: SystemdUnitFailed: netbox_ganeti_esams03_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:58:43] RESOLVED: [8x] RipeAtlasAnchorUnreachable: ipv4 ping to esams RIPE Atlas anchor: failures over threshold for measurement 59935536 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [11:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1100) [11:05:16] RECOVERY - Router interfaces on mr1-esams is OK: OK: host 185.15.59.130, interfaces up: 35, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [11:09:29] (03CR) 10Btullis: [C:03+2] deployment_server: Add the kubeconfigs of the team Spark namespaces [puppet] - 10https://gerrit.wikimedia.org/r/1367772 (https://phabricator.wikimedia.org/T432322) (owner: 10Btullis) [11:10:38] (03PS6) 10Hashar: cinderutils: honor mount_options when mountpoints exist [puppet] - 10https://gerrit.wikimedia.org/r/1343081 [11:11:38] PROBLEM - Host asw1-by27-esams is DOWN: PING CRITICAL - Packet loss = 100% [11:11:43] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv4 ping to esams RIPE Atlas anchor: failures over threshold for measurement 59935536 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [11:13:04] RECOVERY - Host asw1-by27-esams is UP: PING OK - Packet loss = 0%, RTA = 91.85 ms [11:13:42] (03PS8) 10Hashar: cinderutils: support appending mount options [puppet] - 10https://gerrit.wikimedia.org/r/1343082 [11:14:35] (03Abandoned) 10Daimona Eaytoy: build: Upgrade mocha to 12+ to make sure this works in Node 26+ [extensions/CampaignEvents] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358770 (https://phabricator.wikimedia.org/T442318) (owner: 10Jforrester) [11:15:23] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:15:59] (03PS9) 10Hashar: cinderutils: support appending mount options [puppet] - 10https://gerrit.wikimedia.org/r/1343082 [11:16:43] FIRING: [8x] RipeAtlasAnchorUnreachable: ipv4 ping to esams RIPE Atlas anchor: failures over threshold for measurement 59935536 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [11:17:33] (03PS5) 10Hashar: ci: set noatime on castor /srv partition [puppet] - 10https://gerrit.wikimedia.org/r/1343083 [11:18:23] jouncebot: nowandnext [11:18:23] For the next 0 hour(s) and 41 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1100) [11:18:23] In 1 hour(s) and 41 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1300) [11:21:43] RESOLVED: [8x] RipeAtlasAnchorUnreachable: ipv4 ping to esams RIPE Atlas anchor: failures over threshold for measurement 59935536 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [11:25:36] (03PS1) 10Urbanecm: cswiki: Remove celebration logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368266 (https://phabricator.wikimedia.org/T442387) [11:26:40] (03CR) 10CI reject: [V:04-1] cswiki: Remove celebration logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368266 (https://phabricator.wikimedia.org/T442387) (owner: 10Urbanecm) [11:27:08] (03CR) 10Klausman: [V:03+2 C:03+2] ml-serve: Deny KServe model management paths at the ingress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368162 (https://phabricator.wikimedia.org/T443921) (owner: 10Ozge) [11:33:00] (03Merged) 10jenkins-bot: ml-serve: Deny KServe model management paths at the ingress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368162 (https://phabricator.wikimedia.org/T443921) (owner: 10Ozge) [11:35:56] !log ayounsi@cumin1004 START - Cookbook sre.dns.netbox [11:41:18] !log ayounsi@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new cr1-esams - ayounsi@cumin1004" [11:42:41] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new cr1-esams - ayounsi@cumin1004" [11:42:41] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [11:44:54] (03PS1) 10Ayounsi: update cr1-esams mgmt IP in monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1368274 (https://phabricator.wikimedia.org/T442691) [11:46:01] (03CR) 10Cathal Mooney: [C:03+1] update cr1-esams mgmt IP in monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1368274 (https://phabricator.wikimedia.org/T442691) (owner: 10Ayounsi) [11:49:00] !log jhancock@cumin2003 START - Cookbook sre.hosts.provision for host cp3074.mgmt.esams.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:51:25] (03PS1) 10Ayounsi: New cr1-esams update interface names [homer/public] - 10https://gerrit.wikimedia.org/r/1368277 (https://phabricator.wikimedia.org/T442691) [11:51:41] (03CR) 10Ayounsi: [C:03+2] update cr1-esams mgmt IP in monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1368274 (https://phabricator.wikimedia.org/T442691) (owner: 10Ayounsi) [11:52:20] (03CR) 10Cathal Mooney: [C:03+1] New cr1-esams update interface names [homer/public] - 10https://gerrit.wikimedia.org/r/1368277 (https://phabricator.wikimedia.org/T442691) (owner: 10Ayounsi) [11:53:31] FIRING: Not accepting/receiving prefixes from anycast BGP peer: Alert for device asw1-by27-esams.mgmt.esams.wmnet - Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [11:54:43] (03PS2) 10Ayounsi: New cr1-esams update interface names [homer/public] - 10https://gerrit.wikimedia.org/r/1368277 (https://phabricator.wikimedia.org/T442691) [11:55:47] !log gmodena@deploy2003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [11:55:59] !log gmodena@deploy2003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [11:56:13] (03CR) 10Ayounsi: [C:03+2] New cr1-esams update interface names [homer/public] - 10https://gerrit.wikimedia.org/r/1368277 (https://phabricator.wikimedia.org/T442691) (owner: 10Ayounsi) [11:57:27] (03Merged) 10jenkins-bot: New cr1-esams update interface names [homer/public] - 10https://gerrit.wikimedia.org/r/1368277 (https://phabricator.wikimedia.org/T442691) (owner: 10Ayounsi) [12:00:04] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [12:00:14] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [12:00:19] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [12:00:25] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [12:03:10] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp3074.mgmt.esams.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:04:55] !log gmodena@deploy2003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [12:04:58] !log gmodena@deploy2003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [12:05:14] !log gmodena@deploy2003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [12:05:25] !log gmodena@deploy2003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [12:05:29] !log gmodena@deploy2003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [12:05:40] !log gmodena@deploy2003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [12:29:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [12:30:49] (03CR) 10Jelto: [C:03+1] "verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1368227 (https://phabricator.wikimedia.org/T442758) (owner: 10Ozge) [12:31:53] (03CR) 10Jelto: [C:03+2] admin: Add FIDO SSH key for ozge [puppet] - 10https://gerrit.wikimedia.org/r/1368227 (https://phabricator.wikimedia.org/T442758) (owner: 10Ozge) [12:34:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [12:35:10] PROBLEM - Host cp3074 is DOWN: PING CRITICAL - Packet loss = 100% [12:37:14] (03PS1) 10Gmodena: Reapply "rest-gateway: set SNI on WDQSv2 upstream clusters" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368284 (https://phabricator.wikimedia.org/T436778) [12:38:31] FIRING: [2x] Not accepting/receiving prefixes from anycast BGP peer: Alert for device asw1-bw27-esams.mgmt.esams.wmnet - Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [12:41:00] jouncebot: nowandnext [12:41:00] No deployments scheduled for the next 0 hour(s) and 18 minute(s) [12:41:00] In 0 hour(s) and 18 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1300) [12:41:22] proposing to sync a wmf.3 patch unless there are any objections [12:42:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy2003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367904 (https://phabricator.wikimedia.org/T442170) (owner: 10Kosta Harlan) [12:45:43] (03Merged) 10jenkins-bot: AbuseReview: Delay the link spam model check job [extensions/WikimediaAntiAbuse] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367904 (https://phabricator.wikimedia.org/T442170) (owner: 10Kosta Harlan) [12:46:11] !log kharlan@deploy2003 Started scap sync-world: Backport for [[gerrit:1367904|AbuseReview: Delay the link spam model check job (T442170)]] (1.48.0-wmf.3) [12:46:15] T442170: AbuseReview: Queue job for scoring edits that add external links - https://phabricator.wikimedia.org/T442170 [12:47:39] (03CR) 10Clément Goubert: [C:03+1] Reapply "rest-gateway: set SNI on WDQSv2 upstream clusters" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368284 (https://phabricator.wikimedia.org/T436778) (owner: 10Gmodena) [12:50:37] !log kharlan@deploy2003 kharlan: Backport for [[gerrit:1367904|AbuseReview: Delay the link spam model check job (T442170)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:52:32] !log kharlan@deploy2003 kharlan: Continuing with deployment [12:53:03] (03PS2) 10Gmodena: Reapply "rest-gateway: set SNI on WDQSv2 upstream clusters" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368284 (https://phabricator.wikimedia.org/T436778) [12:53:03] (03PS1) 10Gmodena: rest-gateway: add smoke test for the WDQSv2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368286 (https://phabricator.wikimedia.org/T436778) [12:58:32] !log kharlan@deploy2003 Finished scap sync-world: Backport for [[gerrit:1367904|AbuseReview: Delay the link spam model check job (T442170)]] (1.48.0-wmf.3) (duration: 12m 21s) [12:58:35] T442170: AbuseReview: Queue job for scoring edits that add external links - https://phabricator.wikimedia.org/T442170 [13:00:05] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1300) [13:06:14] kostajh: if you're done, can I deploy stuff? [13:11:31] Amir1: yes, go ahead [13:11:40] thanks! [13:13:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [13:18:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [13:18:16] (03PS1) 10Ladsgroup: DASH: Add "auto" resolution as default and still show other resolutions [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368287 (https://phabricator.wikimedia.org/T443728) [13:18:41] (03PS1) 10Ladsgroup: DASH: Add "auto" resolution as default and still show other resolutions [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1368288 (https://phabricator.wikimedia.org/T443728) [13:19:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy2003 using scap backport" [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368287 (https://phabricator.wikimedia.org/T443728) (owner: 10Ladsgroup) [13:19:20] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy2003 using scap backport" [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1368288 (https://phabricator.wikimedia.org/T443728) (owner: 10Ladsgroup) [13:20:34] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 10 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362142 (https://phabricator.wikimedia.org/T440359) (owner: 10Daimona Eaytoy) [13:23:48] (03PS1) 10Dreamy Jazz: Follow-ups for CheckUser AbuseFilter variables [extensions/CheckUser] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368290 (https://phabricator.wikimedia.org/T234155) [13:24:23] (03Merged) 10jenkins-bot: DASH: Add "auto" resolution as default and still show other resolutions [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368287 (https://phabricator.wikimedia.org/T443728) (owner: 10Ladsgroup) [13:25:31] (03CR) 10EarlyWarningBot: "[Failed command](https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-vendor-mysql-php83/68825/consoleFull): `composer r" [extensions/CheckUser] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368290 (https://phabricator.wikimedia.org/T234155) (owner: 10Dreamy Jazz) [13:27:55] (03Merged) 10jenkins-bot: DASH: Add "auto" resolution as default and still show other resolutions [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1368288 (https://phabricator.wikimedia.org/T443728) (owner: 10Ladsgroup) [13:28:25] !log ladsgroup@deploy2003 Started scap sync-world: Backport for DASH: Add "auto" resolution as default and still show other resolutions (T443728) ([[gerrit:1368287|1.48.0-wmf.4]], [[gerrit:1368288|1.48.0-wmf.3]]) [13:28:28] T443728: DASH resolution selector - https://phabricator.wikimedia.org/T443728 [13:29:21] (03CR) 10CI reject: [V:04-1] Follow-ups for CheckUser AbuseFilter variables [extensions/CheckUser] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368290 (https://phabricator.wikimedia.org/T234155) (owner: 10Dreamy Jazz) [13:31:26] (03PS1) 10Dreamy Jazz: Enable CheckUser AbuseFilter variables on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368292 (https://phabricator.wikimedia.org/T234155) [13:31:43] jouncebot: nowandnext [13:31:43] For the next 0 hour(s) and 28 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1300) [13:31:44] In 0 hour(s) and 28 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1400) [13:32:44] !log ladsgroup@deploy2003 ladsgroup: Backport for DASH: Add "auto" resolution as default and still show other resolutions (T443728) ([[gerrit:1368287|1.48.0-wmf.4]], [[gerrit:1368288|1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:33:09] (03CR) 10Dreamy Jazz: "recheck" [extensions/CheckUser] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368290 (https://phabricator.wikimedia.org/T234155) (owner: 10Dreamy Jazz) [13:35:41] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 10 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/CheckUser] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368290 (https://phabricator.wikimedia.org/T234155) (owner: 10Dreamy Jazz) [13:36:01] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 10 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368292 (https://phabricator.wikimedia.org/T234155) (owner: 10Dreamy Jazz) [13:40:44] !log ladsgroup@deploy2003 ladsgroup: Continuing with deployment [13:46:43] !log ladsgroup@deploy2003 Finished scap sync-world: Backport for DASH: Add "auto" resolution as default and still show other resolutions (T443728) ([[gerrit:1368287|1.48.0-wmf.4]], [[gerrit:1368288|1.48.0-wmf.3]]) (duration: 18m 18s) [13:46:47] T443728: DASH resolution selector - https://phabricator.wikimedia.org/T443728 [13:47:08] (03Abandoned) 10Nikerabbit: Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1361381 (owner: 10L10n-bot) [13:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.25% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [14:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: Your horoscope predicts another UTC afternoon backport window deploy. May Zuul be (nice) with you. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1400). [14:00:05] Daimona and Dreamy_Jazz: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [14:00:16] \o [14:00:18] o/ [14:01:32] (03PS1) 10WMDE-Fisch: Enable Citoid PDF fallback on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368298 (https://phabricator.wikimedia.org/T443469) [14:01:56] (03CR) 10CI reject: [V:04-1] Enable Citoid PDF fallback on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368298 (https://phabricator.wikimedia.org/T443469) (owner: 10WMDE-Fisch) [14:02:12] Is Daimona here? [14:02:29] If not, I can go first [14:02:52] go ahead, I think [14:03:41] (03PS2) 10WMDE-Fisch: Enable Citoid PDF fallback on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368298 (https://phabricator.wikimedia.org/T443469) [14:04:27] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy2003 using scap backport" [extensions/CheckUser] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368290 (https://phabricator.wikimedia.org/T234155) (owner: 10Dreamy Jazz) [14:04:28] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368292 (https://phabricator.wikimedia.org/T234155) (owner: 10Dreamy Jazz) [14:04:53] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 11 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368298 (https://phabricator.wikimedia.org/T443469) (owner: 10WMDE-Fisch) [14:05:14] (03Merged) 10jenkins-bot: Enable CheckUser AbuseFilter variables on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368292 (https://phabricator.wikimedia.org/T234155) (owner: 10Dreamy Jazz) [14:05:41] 06SRE, 10SRE-Access-Requests, 06SRE Observability: Add FIDO-backed SSH keys for gkyziridis - https://phabricator.wikimedia.org/T444318 (10gkyziridis) 03NEW [14:06:01] (03Merged) 10jenkins-bot: Follow-ups for CheckUser AbuseFilter variables [extensions/CheckUser] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368290 (https://phabricator.wikimedia.org/T234155) (owner: 10Dreamy Jazz) [14:06:27] !log dreamyjazz@deploy2003 Started scap sync-world: Backport for [[gerrit:1368290|Follow-ups for CheckUser AbuseFilter variables (T234155)]] (1.48.0-wmf.4), [[gerrit:1368292|Enable CheckUser AbuseFilter variables on testwiki (T234155)]] [14:06:30] T234155: Create CheckUser-level abuse filters - https://phabricator.wikimedia.org/T234155 [14:10:13] (03PS1) 10Santiago Faci: Upgrade GrowthBook beyond 5.2.0 release [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368302 (https://phabricator.wikimedia.org/T443793) [14:10:57] oh, apparently Daimona_ is here, actually [14:11:02] (03PS2) 10Santiago Faci: Upgrade GrowthBook beyond 5.2.0 release [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368302 (https://phabricator.wikimedia.org/T443793) [14:11:08] !log dreamyjazz@deploy2003 dreamyjazz: Backport for [[gerrit:1368290|Follow-ups for CheckUser AbuseFilter variables (T234155)]] (1.48.0-wmf.4), [[gerrit:1368292|Enable CheckUser AbuseFilter variables on testwiki (T234155)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:11:19] Oh hi! Yes I'm here [14:11:29] alright, you’re up next after Dreamy_Jazz then :) [14:11:41] Sounds good, thank you :) [14:14:03] !log dreamyjazz@deploy2003 dreamyjazz: Continuing with deployment [14:17:34] (03PS1) 10Gkyziridis: admin: Add FIDO SSH key for gkyziridis [puppet] - 10https://gerrit.wikimedia.org/r/1368303 (https://phabricator.wikimedia.org/T444318) [14:20:04] !log dreamyjazz@deploy2003 Finished scap sync-world: Backport for [[gerrit:1368290|Follow-ups for CheckUser AbuseFilter variables (T234155)]] (1.48.0-wmf.4), [[gerrit:1368292|Enable CheckUser AbuseFilter variables on testwiki (T234155)]] (duration: 13m 37s) [14:20:07] T234155: Create CheckUser-level abuse filters - https://phabricator.wikimedia.org/T234155 [14:20:51] Daimona: I think scap will complain about your Depends-On (I don’t remember if the complaint can be bypassed or not) – is it okay to deploy the config change even when not all wikis have the CampaignEvents change yet? [14:21:34] Yup totally fine, the depends-on was there mostly to say that I wanted CR on the approach first, but that has already happened [14:21:48] ok [14:21:56] Dreamy_Jazz: are you done deploying? [14:22:00] Yes [14:22:07] ok, I’ll take over then [14:22:31] (03CR) 10WMDE-Fisch: [C:03+1] "Can be dropped. AFAIK it's not used anywhere anymore." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1126134 (https://phabricator.wikimedia.org/T258917) (owner: 10Reedy) [14:22:32] bleh, the error is not bypassable [14:22:47] “To avoid this error you will need to cherry-pick and merge the dependency into that branch” [14:22:56] or just remove the Depends-On from the commit message [14:23:07] (or turn it into some other trailer. idk, See-Also?) [14:23:34] or we actually backport it, I suppose (fortunately it doesn’t touch i18n) [14:23:36] Daimona: wdyt? [14:23:54] I can just drop the depends-on I think [14:24:00] 06SRE, 06Fundraising-Backlog, 10FY26-27 ST6.4 - Logged-in performance, 10MediaWiki-extensions-CentralNotice, and 4 others: Special:HideBanners is not really cacheable - https://phabricator.wikimedia.org/T256447#12539382 (10Krinkle) >>! In T285210#12539348, @Krinkle wrote: > […] I'll work on making this cac... [14:24:13] (03PS2) 10Daimona Eaytoy: CommonSettings: set $wgCampaignEventsArticleQualitySupportedWikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362142 (https://phabricator.wikimedia.org/T440359) [14:24:19] Aaaaand done [14:24:38] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362142 (https://phabricator.wikimedia.org/T440359) (owner: 10Daimona Eaytoy) [14:25:29] (03Merged) 10jenkins-bot: CommonSettings: set $wgCampaignEventsArticleQualitySupportedWikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362142 (https://phabricator.wikimedia.org/T440359) (owner: 10Daimona Eaytoy) [14:25:48] RECOVERY - Host cp3074 is UP: PING OK - Packet loss = 0%, RTA = 97.79 ms [14:25:50] PROBLEM - HAProxy HTTPS wikiworkshop.org ECDSA on cp3074 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS [14:25:51] !log lucaswerkmeister-wmde@deploy2003 Started scap sync-world: Backport for [[gerrit:1362142|CommonSettings: set $wgCampaignEventsArticleQualitySupportedWikis (T440359)]] [14:25:52] PROBLEM - HAProxy HTTPS wikipedia25.org ECDSA on cp3074 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS [14:25:54] T440359: Worklist quality signals are wrong on non-Wikipedia wikis - https://phabricator.wikimedia.org/T440359 [14:26:00] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp3074 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS [14:26:02] PROBLEM - haproxy process on cp3074 is CRITICAL: PROCS CRITICAL: 0 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy [14:26:34] PROBLEM - OSPF status on cr1-eqiad is CRITICAL: OSPFv2: 6/7 UP : OSPFv3: 6/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:26:34] PROBLEM - OSPF status on cr1-codfw is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:27:34] RECOVERY - OSPF status on cr1-eqiad is OK: OSPFv2: 7/7 UP : OSPFv3: 7/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:27:34] RECOVERY - OSPF status on cr1-codfw is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:28:00] PROBLEM - Host hcaptcha-proxy3001 is DOWN: PING CRITICAL - Packet loss = 100% [14:28:00] PROBLEM - Host hcaptcha-proxy3002 is DOWN: PING CRITICAL - Packet loss = 100% [14:28:10] FIRING: BFDdown: BFD session down between cr1-codfw and 185.15.59.139 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:28:20] PROBLEM - Host netflow3004 is DOWN: PING CRITICAL - Packet loss = 100% [14:28:20] PROBLEM - Host ncredir3006 is DOWN: PING CRITICAL - Packet loss = 100% [14:28:20] PROBLEM - Host ncredir3005 is DOWN: PING CRITICAL - Packet loss = 100% [14:28:30] PROBLEM - Host bast3007 is DOWN: PING CRITICAL - Packet loss = 100% [14:28:30] PROBLEM - Host doh3005 is DOWN: PING CRITICAL - Packet loss = 100% [14:28:36] PROBLEM - Host durum3006 is DOWN: PING CRITICAL - Packet loss = 100% [14:28:44] PROBLEM - Host durum3005 is DOWN: PING CRITICAL - Packet loss = 100% [14:28:46] PROBLEM - Host tcp-proxy3002 is DOWN: PING CRITICAL - Packet loss = 100% [14:28:53] hmm, what’s happening with esams [14:28:54] RECOVERY - Host tcp-proxy3002 is UP: PING OK - Packet loss = 0%, RTA = 88.45 ms [14:28:54] RECOVERY - Host durum3005 is UP: PING OK - Packet loss = 0%, RTA = 88.50 ms [14:28:56] RECOVERY - Host hcaptcha-proxy3001 is UP: PING OK - Packet loss = 0%, RTA = 88.33 ms [14:28:56] RECOVERY - Host hcaptcha-proxy3002 is UP: PING OK - Packet loss = 0%, RTA = 88.42 ms [14:29:24] RECOVERY - Host netflow3004 is UP: PING OK - Packet loss = 0%, RTA = 88.67 ms [14:29:24] RECOVERY - Host ncredir3006 is UP: PING OK - Packet loss = 0%, RTA = 88.39 ms [14:29:24] RECOVERY - Host ncredir3005 is UP: PING OK - Packet loss = 0%, RTA = 88.48 ms [14:29:30] RECOVERY - Host bast3007 is UP: PING OK - Packet loss = 0%, RTA = 88.24 ms [14:29:32] RECOVERY - Host doh3005 is UP: PING OK - Packet loss = 0%, RTA = 88.50 ms [14:29:38] RECOVERY - Host durum3006 is UP: PING OK - Packet loss = 0%, RTA = 88.32 ms [14:29:42] PROBLEM - Host 2a02:ec80:300:2:185:15:59:34 is DOWN: PING CRITICAL - Packet loss = 100% [14:29:48] is esams still supposed to be down for T440349? [14:29:49] T440349: esams router and switch maint - https://phabricator.wikimedia.org/T440349 [14:29:50] RECOVERY - HAProxy HTTPS wikiworkshop.org ECDSA on cp3074 is OK: SSL OK - Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2027-01-08 02:27:11 +0000 (expires in 58 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:29:54] RECOVERY - HAProxy HTTPS wikipedia25.org ECDSA on cp3074 is OK: SSL OK - Certificate wikipedia25.org contains all required SANs:Certificate wikipedia25.org (ECDSA) valid until 2027-02-01 04:33:45 +0000 (expires in 82 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:30:00] RECOVERY - haproxy process on cp3074 is OK: PROCS OK: 2 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy [14:30:02] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp3074 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2027-01-02 18:34:50 +0000 (expires in 53 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:30:40] PROBLEM - Wikidough DoT Check -IPv6- on doh3005 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikidough/Monitoring%23Wikidough_Basic_Check [14:30:58] RECOVERY - Host 2a02:ec80:300:2:185:15:59:34 is UP: PING OK - Packet loss = 0%, RTA = 88.01 ms [14:31:05] (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] "Obsolete per T258917#10622678. 👍" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1126134 (https://phabricator.wikimedia.org/T258917) (owner: 10Reedy) [14:31:30] RECOVERY - Wikidough DoT Check -IPv6- on doh3005 is OK: TCP OK - 0.190 second response time on 2a02:ec80:300:3:185:15:59:98 port 853 https://wikitech.wikimedia.org/wiki/Wikidough/Monitoring%23Wikidough_Basic_Check [14:32:02] RECOVERY - OSPF status on cr2-esams is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:32:03] !log lucaswerkmeister-wmde@deploy2003 lucaswerkmeister-wmde, daimona: Backport for [[gerrit:1362142|CommonSettings: set $wgCampaignEventsArticleQualitySupportedWikis (T440359)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:32:05] FIRING: [2x] PingLossPercent: Blackbox probe packet loss 5.417% from codfw to esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=esams - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [14:32:06] T440359: Worklist quality signals are wrong on non-Wikipedia wikis - https://phabricator.wikimedia.org/T440359 [14:32:42] Daimona: please test on some group0 wikis :) [14:33:10] RESOLVED: BFDdown: BFD session down between cr1-codfw and 185.15.59.139 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:33:12] PROBLEM - Host cp3079 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:12] PROBLEM - Host cp3081 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:12] PROBLEM - Host cp3077 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:12] PROBLEM - Host ganeti3007 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:12] PROBLEM - Host cp3071 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:15] I can't test it, as the feature it tweaks is disabled on all of production :D Sorry I should have realized/stated that sooner [14:33:18] PROBLEM - Host cp3069 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:20] PROBLEM - Host cp3067 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:20] PROBLEM - Host cp3073 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:20] PROBLEM - Host cp3075 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:20] PROBLEM - Host ganeti3005 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:20] PROBLEM - Host lvs3009 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:22] ah ok [14:33:32] PROBLEM - Host netflow3004 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:32] PROBLEM - Host ncredir3006 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:32] PROBLEM - Host ncredir3005 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:48] PROBLEM - Host durum3006 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:56] PROBLEM - Host durum3005 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:56] PROBLEM - Host cp3070 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:56] PROBLEM - Host cp3078 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:56] PROBLEM - Host cp3080 is DOWN: PING CRITICAL - Packet loss = 100% [14:33:58] PROBLEM - Host tcp-proxy3002 is DOWN: PING CRITICAL - Packet loss = 100% [14:34:02] PROBLEM - Host cp3072 is DOWN: PING CRITICAL - Packet loss = 100% [14:34:04] PROBLEM - Host prometheus3004 is DOWN: PING CRITICAL - Packet loss = 100% [14:34:10] PROBLEM - Host cp3068 is DOWN: PING CRITICAL - Packet loss = 100% [14:34:12] !log lucaswerkmeister-wmde@deploy2003 lucaswerkmeister-wmde, daimona: Continuing with deployment [14:34:14] PROBLEM - Host tcp-proxy3001 is DOWN: PING CRITICAL - Packet loss = 100% [14:34:37] (I’m assuming the esams stuff in icinga doesn’t concern me, let me know otherwise) [14:34:50] RECOVERY - Host cp3073 is UP: PING OK - Packet loss = 0%, RTA = 77.14 ms [14:34:50] RECOVERY - Host cp3067 is UP: PING OK - Packet loss = 0%, RTA = 77.12 ms [14:34:50] RECOVERY - Host ganeti3005 is UP: PING OK - Packet loss = 0%, RTA = 77.38 ms [14:34:52] RECOVERY - Host cp3080 is UP: PING OK - Packet loss = 0%, RTA = 77.15 ms [14:34:52] RECOVERY - Host durum3005 is UP: PING OK - Packet loss = 0%, RTA = 77.52 ms [14:34:52] RECOVERY - Host cp3075 is UP: PING OK - Packet loss = 0%, RTA = 77.10 ms [14:34:52] RECOVERY - Host cp3069 is UP: PING OK - Packet loss = 0%, RTA = 77.18 ms [14:34:52] RECOVERY - Host cp3071 is UP: PING OK - Packet loss = 0%, RTA = 77.07 ms [14:34:53] RECOVERY - Host cp3079 is UP: PING OK - Packet loss = 0%, RTA = 77.11 ms [14:34:54] RECOVERY - Host cp3070 is UP: PING OK - Packet loss = 0%, RTA = 77.13 ms [14:34:54] RECOVERY - Host tcp-proxy3002 is UP: PING OK - Packet loss = 0%, RTA = 77.69 ms [14:34:54] RECOVERY - Host cp3077 is UP: PING OK - Packet loss = 0%, RTA = 77.18 ms [14:34:55] RECOVERY - Host cp3078 is UP: PING OK - Packet loss = 0%, RTA = 77.34 ms [14:34:55] RECOVERY - Host tcp-proxy3001 is UP: PING OK - Packet loss = 0%, RTA = 77.61 ms [14:34:56] RECOVERY - Host cp3081 is UP: PING OK - Packet loss = 0%, RTA = 77.15 ms [14:34:56] RECOVERY - Host cp3068 is UP: PING OK - Packet loss = 0%, RTA = 77.21 ms [14:34:57] RECOVERY - Host ganeti3007 is UP: PING OK - Packet loss = 0%, RTA = 78.12 ms [14:35:00] RECOVERY - Host cp3072 is UP: PING OK - Packet loss = 0%, RTA = 77.13 ms [14:35:04] RECOVERY - Host lvs3009 is UP: PING OK - Packet loss = 0%, RTA = 77.10 ms [14:35:24] RECOVERY - Host netflow3004 is UP: PING OK - Packet loss = 0%, RTA = 77.75 ms [14:35:24] RECOVERY - Host ncredir3006 is UP: PING OK - Packet loss = 0%, RTA = 77.58 ms [14:35:24] RECOVERY - Host ncredir3005 is UP: PING OK - Packet loss = 0%, RTA = 77.58 ms [14:35:32] RECOVERY - Host prometheus3004 is UP: PING OK - Packet loss = 0%, RTA = 77.75 ms [14:35:38] RECOVERY - Host durum3006 is UP: PING OK - Packet loss = 0%, RTA = 77.53 ms [14:37:34] PROBLEM - OSPF status on cr2-eqiad is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:37:44] PROBLEM - OSPF status on cr2-codfw is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:38:34] RECOVERY - OSPF status on cr2-eqiad is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:38:44] RECOVERY - OSPF status on cr2-codfw is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:38:54] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-esams (185.15.59.139) - group Confed_esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:39:09] RESOLVED: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-esams (185.15.59.139) - group Confed_esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:40:13] !log lucaswerkmeister-wmde@deploy2003 Finished scap sync-world: Backport for [[gerrit:1362142|CommonSettings: set $wgCampaignEventsArticleQualitySupportedWikis (T440359)]] (duration: 14m 22s) [14:40:16] T440359: Worklist quality signals are wrong on non-Wikipedia wikis - https://phabricator.wikimedia.org/T440359 [14:40:25] !log UTC afternoon backport+config window done [14:40:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:41:12] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:47:06] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [14:48:23] (03PS1) 10Xcollazo: admin: Add spare FIDO key for xcollazo [puppet] - 10https://gerrit.wikimedia.org/r/1368313 (https://phabricator.wikimedia.org/T442741) [14:48:27] (03PS1) 10Gehel: admin: removed non FIDO SSH keys for sd, cmyrick [puppet] - 10https://gerrit.wikimedia.org/r/1368314 (https://phabricator.wikimedia.org/T442756) [14:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [14:49:32] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for xcollazo - https://phabricator.wikimedia.org/T442741#12539622 (10xcollazo) 05Resolved→03Open a:05Dzahn→03None Reopening to add spare FIDO key, and to remove the soft key. [14:51:49] (03PS1) 10Ayounsi: esams: interface changes [homer/public] - 10https://gerrit.wikimedia.org/r/1368316 [14:56:04] (03PS1) 10Gehel: admin: adding FIDO/SSH keys for paragon, dsaez, tandic [puppet] - 10https://gerrit.wikimedia.org/r/1368318 (https://phabricator.wikimedia.org/T442756) [15:00:05] Deploy window Test Kitchen UI Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1500) [15:01:50] (03CR) 10Gmodena: [C:03+2] Reapply "rest-gateway: set SNI on WDQSv2 upstream clusters" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368284 (https://phabricator.wikimedia.org/T436778) (owner: 10Gmodena) [15:02:05] (03CR) 10Klausman: [V:03+2 C:03+2] admin: removed non FIDO SSH keys for sd, cmyrick [puppet] - 10https://gerrit.wikimedia.org/r/1368314 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [15:03:36] (03Merged) 10jenkins-bot: Reapply "rest-gateway: set SNI on WDQSv2 upstream clusters" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368284 (https://phabricator.wikimedia.org/T436778) (owner: 10Gmodena) [15:05:15] (03CR) 10Klausman: [V:03+2 C:03+2] admin: adding FIDO/SSH keys for paragon, dsaez, tandic [puppet] - 10https://gerrit.wikimedia.org/r/1368318 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [15:07:00] RESOLVED: [2x] PingLossPercent: Blackbox probe packet loss 4.375% from codfw to esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=esams - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [15:08:02] (03PS1) 10Dpogorzelski: service: exclude ml-serve from DC switchovers [puppet] - 10https://gerrit.wikimedia.org/r/1368320 (https://phabricator.wikimedia.org/T442866) [15:08:02] (03Abandoned) 10Dpogorzelski: amd_gpu: set the compute partition mode through sysfs [puppet] - 10https://gerrit.wikimedia.org/r/1344685 (https://phabricator.wikimedia.org/T436928) (owner: 10Dpogorzelski) [15:10:04] (03PS2) 10Dpogorzelski: service: exclude ml-serve from DC switchovers [puppet] - 10https://gerrit.wikimedia.org/r/1368320 (https://phabricator.wikimedia.org/T442866) [15:15:23] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:23:47] !log gmodena@deploy2003 helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [15:24:07] !log gmodena@deploy2003 helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [15:28:38] (03PS2) 10DCausse: Search: add dedicated poolcounter for WDQS search use-cases [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360927 (https://phabricator.wikimedia.org/T442824) [15:28:38] (03PS1) 10DCausse: search: add pool counter rules [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368323 [15:29:32] (03CR) 10CI reject: [V:04-1] search: add pool counter rules [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368323 (owner: 10DCausse) [15:29:56] !log gmodena@deploy2003 helmfile [codfw] START helmfile.d/services/rest-gateway: apply [15:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1530) [15:30:09] !log gmodena@deploy2003 helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [15:30:39] (03PS8) 10Brouberol: migrate the sre.etcd.roll-restart-reboot cookbook under sre.k8s [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) [15:31:01] (03PS3) 10DCausse: Search: add dedicated poolcounter for WDQS search use-cases [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360927 (https://phabricator.wikimedia.org/T442824) [15:31:01] (03PS2) 10DCausse: search: add pool counter rules [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368323 (https://phabricator.wikimedia.org/T442824) [15:31:20] (03PS1) 10Seanleong-wmde: Switch site lookup to sitematrix lookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368324 (https://phabricator.wikimedia.org/T438982) [15:33:17] (03CR) 10CI reject: [V:04-1] migrate the sre.etcd.roll-restart-reboot cookbook under sre.k8s [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) (owner: 10Brouberol) [15:34:07] (03PS9) 10Brouberol: migrate the sre.etcd.roll-restart-reboot cookbook under sre.k8s [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) [15:35:15] (03CR) 10Reedy: Drop TemplateData EventStreams/EventLogging config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1126134 (https://phabricator.wikimedia.org/T258917) (owner: 10Reedy) [15:35:26] (03PS3) 10Reedy: Drop TemplateData EventStreams/EventLogging config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1126134 (https://phabricator.wikimedia.org/T258917) [15:37:22] FIRING: GnmiInterfaceCountersDrop: ... [15:37:22] cr1-esams is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=cr1-esams:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [15:38:16] (03PS10) 10Brouberol: migrate the sre.etcd.roll-restart-reboot cookbook under sre.k8s [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) [15:39:09] (03PS11) 10Brouberol: migrate the sre.etcd.roll-restart-reboot cookbook under sre.k8s [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) [15:39:47] (03PS12) 10Brouberol: migrate the sre.etcd.roll-restart-reboot cookbook under sre.k8s [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) [15:41:09] (03PS13) 10Brouberol: migrate the sre.etcd.roll-restart-reboot cookbook under sre.k8s [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) [15:41:54] (03PS14) 10Brouberol: migrate the sre.etcd.roll-restart-reboot cookbook under sre.k8s [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) [15:43:10] (03CR) 10Brouberol: "Done" [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) (owner: 10Brouberol) [15:46:57] (03CR) 10WMDE-Fisch: "I could just deploy it tomorrow, I scheduled another config patch anyway. - Or is there something I need to look out for specifically @ree" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1126134 (https://phabricator.wikimedia.org/T258917) (owner: 10Reedy) [15:49:08] (03CR) 10DLynch: [C:03+1] VisualEditor: add image for Suggestion Mode welcome dialog [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367522 (https://phabricator.wikimedia.org/T442024) (owner: 10Medelius) [15:50:37] (03PS4) 10DCausse: search: add pool counter rules and dedicated pool for WDQS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360927 (https://phabricator.wikimedia.org/T442824) [15:50:41] (03CR) 10Tobias Gritschacher: [C:03+1] Enable Citoid PDF fallback on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368298 (https://phabricator.wikimedia.org/T443469) (owner: 10WMDE-Fisch) [15:51:16] (03Abandoned) 10DCausse: search: add pool counter rules [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368323 (https://phabricator.wikimedia.org/T442824) (owner: 10DCausse) [15:58:42] (03PS1) 10Mvolz: admin: Add usb A key & revoke non-hardware backed key for mvolz [puppet] - 10https://gerrit.wikimedia.org/r/1368328 (https://phabricator.wikimedia.org/T442756) [16:00:04] jelto, arnoldokoth, mutante, and arnaudb: How many deployers does it take to do SRE Collaboration Services office hours deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1600). [16:00:47] 06SRE, 06DC-Ops, 06tools-infrastructure-team, 06cloud-services-team (Hardware), and 2 others: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12540066 (10xcollazo) Ok, sorry for the delay. This is my pitch to go with RAID6: We have indeed lately been deleting and... [16:01:15] (03CR) 10Ebernhardson: [C:03+1] search: add pool counter rules and dedicated pool for WDQS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360927 (https://phabricator.wikimedia.org/T442824) (owner: 10DCausse) [16:02:20] FIRING: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 180.38055333333332 times in the last 10 minutes due to memory usage (mw@codfw to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [16:03:50] (03CR) 10Ayounsi: [C:03+2] esams: interface changes [homer/public] - 10https://gerrit.wikimedia.org/r/1368316 (owner: 10Ayounsi) [16:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:05:04] (03Merged) 10jenkins-bot: esams: interface changes [homer/public] - 10https://gerrit.wikimedia.org/r/1368316 (owner: 10Ayounsi) [16:09:24] (03PS1) 10STran: Enable Special:SuggestedInvestigations on ptwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368331 (https://phabricator.wikimedia.org/T442582) [16:10:44] (03CR) 10STran: Enable Special:SuggestedInvestigations on ptwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368331 (https://phabricator.wikimedia.org/T442582) (owner: 10STran) [16:11:28] !log ayounsi@cumin1004 START - Cookbook sre.dns.admin DNS admin: pool esams [reason: network upgrade, T440349] [16:11:31] T440349: esams router and switch maint - https://phabricator.wikimedia.org/T440349 [16:11:33] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool esams [reason: network upgrade, T440349] [16:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:13:22] jouncebot: nowandnext [16:13:22] For the next 0 hour(s) and 46 minute(s): SRE Collaboration Services office hours (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1600) [16:13:22] In 0 hour(s) and 46 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1700) [16:13:58] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:17:34] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [16:19:24] !log trueg@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [16:22:56] !log trueg@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [16:23:19] (03PS1) 10Gehel: admin: removed non FIDO SSH keys for hghani [puppet] - 10https://gerrit.wikimedia.org/r/1368333 (https://phabricator.wikimedia.org/T442756) [16:24:50] RESOLVED: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 60 times in the last 10 minutes due to memory usage (mw@codfw to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [16:25:18] (03CR) 10Gehel: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1368320 (https://phabricator.wikimedia.org/T442866) (owner: 10Dpogorzelski) [16:28:30] (03CR) 10Mszwarc: [C:03+1] Enable Special:SuggestedInvestigations on ptwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368331 (https://phabricator.wikimedia.org/T442582) (owner: 10STran) [16:29:14] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 11 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368331 (https://phabricator.wikimedia.org/T442582) (owner: 10STran) [16:31:32] !log sukhe@puppetserver1001 conftool action : set/pooled=yes; selector: name=dns3.*,service=authdns-update [16:31:45] !log sukhe@dns1004 START - running authdns-update [16:33:55] !log sukhe@dns1004 END - running authdns-update [16:35:01] !log sukhe@puppetserver1001 conftool action : set/pooled=yes; selector: name=dns3.* [16:38:31] RESOLVED: [2x] Not accepting/receiving prefixes from anycast BGP peer: Device asw1-bw27-esams.mgmt.esams.wmnet recovered from Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [16:43:46] (03PS1) 10Dreamy Jazz: tables-catalog: Document missing GlobalBlocking tables [puppet] - 10https://gerrit.wikimedia.org/r/1368338 (https://phabricator.wikimedia.org/T437037) [16:44:40] (03CR) 10CI reject: [V:04-1] tables-catalog: Document missing GlobalBlocking tables [puppet] - 10https://gerrit.wikimedia.org/r/1368338 (https://phabricator.wikimedia.org/T437037) (owner: 10Dreamy Jazz) [16:48:15] (03PS2) 10Dreamy Jazz: tables-catalog: Document missing GlobalBlocking tables [puppet] - 10https://gerrit.wikimedia.org/r/1368338 (https://phabricator.wikimedia.org/T437037) [16:49:09] (03CR) 10CI reject: [V:04-1] tables-catalog: Document missing GlobalBlocking tables [puppet] - 10https://gerrit.wikimedia.org/r/1368338 (https://phabricator.wikimedia.org/T437037) (owner: 10Dreamy Jazz) [16:54:48] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 2 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12540635 (10wiki_willy) a:05brouberol→03VRiley-WMF Moving over to @VRiley-WMF, so we can get these going after the SRE offsite. [16:55:38] (03PS3) 10Dreamy Jazz: tables-catalog: Document missing GlobalBlocking tables [puppet] - 10https://gerrit.wikimedia.org/r/1368338 (https://phabricator.wikimedia.org/T437037) [16:56:03] 10ops-magru: Outbound errors on interface cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://phabricator.wikimedia.org/T444127#12540638 (10RobH) a:05RobH→03ayounsi > Dear Customer, good morning. > > The QSFP module on router port 0/1 has been replaced as requested. Please find the photos of the ac... [16:57:01] 10ops-magru: Outbound errors on interface cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://phabricator.wikimedia.org/T444127#12540647 (10RobH) Photos from work: {F113859819}{F113859820}{F113859821}{F113859822}{F113859823} [17:00:05] jhathaway and rzl: Time to do the Puppet request window deploy. Don't look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1700). [17:00:05] No Gerrit patches in the queue for this window AFAICS. [17:01:30] (03PS1) 10Bking: WIP: Improve validation of conftool data [puppet] - 10https://gerrit.wikimedia.org/r/1368341 (https://phabricator.wikimedia.org/T444321) [17:02:31] (03CR) 10Btullis: [C:03+1] admin: removed non FIDO SSH keys for hghani [puppet] - 10https://gerrit.wikimedia.org/r/1368333 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [17:04:04] (03CR) 10Dpogorzelski: [C:03+2] service: exclude ml-serve from DC switchovers [puppet] - 10https://gerrit.wikimedia.org/r/1368320 (https://phabricator.wikimedia.org/T442866) (owner: 10Dpogorzelski) [17:04:58] (03PS1) 10Urbanecm: [Growth] beta: Enable Special:Home as a beta feature on all beta wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368342 (https://phabricator.wikimedia.org/T443176) [17:06:07] (03CR) 10Gabriel Birke: [C:03+1] citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366088 (owner: 10PipelineBot) [17:08:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:09:06] (03CR) 10Michael Große: [C:03+1] [Growth] beta: Enable Special:Home as a beta feature on all beta wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368342 (https://phabricator.wikimedia.org/T443176) (owner: 10Urbanecm) [17:13:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:14:55] (03CR) 10Klausman: [V:03+2 C:03+2] admin: removed non FIDO SSH keys for hghani [puppet] - 10https://gerrit.wikimedia.org/r/1368333 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [17:16:02] (03PS1) 10Urbanecm: beta: Enable PersonalDashboard on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368344 (https://phabricator.wikimedia.org/T444353) [17:16:22] (03PS5) 10Michael Große: Growth: enable home beta on pilot and innovation wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361861 (https://phabricator.wikimedia.org/T443176) [17:16:28] (03PS6) 10Michael Große: Growth: enable home beta on pilot and innovation wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361861 (https://phabricator.wikimedia.org/T443176) [17:16:34] (03PS2) 10Urbanecm: [Growth] beta: Enable Special:Home as a beta feature on all beta wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368342 (https://phabricator.wikimedia.org/T443176) [17:17:09] jouncebot: nowandnext [17:17:09] For the next 0 hour(s) and 42 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1700) [17:17:09] In 0 hour(s) and 42 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1800) [17:17:18] (03CR) 10Urbanecm: [C:03+2] beta: Enable PersonalDashboard on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368344 (https://phabricator.wikimedia.org/T444353) (owner: 10Urbanecm) [17:17:24] (03CR) 10Urbanecm: [C:03+2] [Growth] beta: Enable Special:Home as a beta feature on all beta wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368342 (https://phabricator.wikimedia.org/T443176) (owner: 10Urbanecm) [17:18:09] (03Merged) 10jenkins-bot: beta: Enable PersonalDashboard on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368344 (https://phabricator.wikimedia.org/T444353) (owner: 10Urbanecm) [17:18:32] (03Merged) 10jenkins-bot: [Growth] beta: Enable Special:Home as a beta feature on all beta wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368342 (https://phabricator.wikimedia.org/T443176) (owner: 10Urbanecm) [17:20:45] !log urbanecm@deploy2003 Started scap sync-world: Backport for [[gerrit:1368344|beta: Enable PersonalDashboard on all wikis (T444353)]], [[gerrit:1368342|[Growth] beta: Enable Special:Home as a beta feature on all beta wikis (T443176)]] [17:20:49] T444353: Enable PersonalDashboard on all beta wikis - https://phabricator.wikimedia.org/T444353 [17:20:50] T443176: Beta Feature deployment of Home at pilot wikis - https://phabricator.wikimedia.org/T443176 [17:25:11] !log urbanecm@deploy2003 urbanecm: Backport for [[gerrit:1368344|beta: Enable PersonalDashboard on all wikis (T444353)]], [[gerrit:1368342|[Growth] beta: Enable Special:Home as a beta feature on all beta wikis (T443176)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [17:25:52] !log urbanecm@deploy2003 urbanecm: Continuing with deployment [17:28:47] (03PS1) 10Urbanecm: beta: Disable PersonalDashboard on lockeddown wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368346 (https://phabricator.wikimedia.org/T444353) [17:30:56] FIRING: TransportLinksInUsageNoRedundancy: magru inbound transports link usage is at redundancy capacity - https://wikitech.wikimedia.org/wiki/Network_monitoring#TransportLinksInUsageNoRedundancy - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/bandwidth-site-totals?var-site=magru - https://alerts.wikimedia.org/?q=alertname%3DTransportLinksInUsageNoRedundancy [17:31:53] !log urbanecm@deploy2003 Finished scap sync-world: Backport for [[gerrit:1368344|beta: Enable PersonalDashboard on all wikis (T444353)]], [[gerrit:1368342|[Growth] beta: Enable Special:Home as a beta feature on all beta wikis (T443176)]] (duration: 11m 08s) [17:31:58] T444353: Enable PersonalDashboard on all beta wikis - https://phabricator.wikimedia.org/T444353 [17:31:59] T443176: Beta Feature deployment of Home at pilot wikis - https://phabricator.wikimedia.org/T443176 [17:32:09] (03CR) 10Urbanecm: [C:03+2] beta: Disable PersonalDashboard on lockeddown wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368346 (https://phabricator.wikimedia.org/T444353) (owner: 10Urbanecm) [17:32:23] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368346 (https://phabricator.wikimedia.org/T444353) (owner: 10Urbanecm) [17:32:52] (03Merged) 10jenkins-bot: beta: Disable PersonalDashboard on lockeddown wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368346 (https://phabricator.wikimedia.org/T444353) (owner: 10Urbanecm) [17:35:53] 06SRE, 10LDAP-Access-Requests: Grant Access to ciadmin for Peter Wangai - https://phabricator.wikimedia.org/T443855#12540862 (10pwangai) Hi all, a small nudge on this one. I would appreciate getting this resolved soon, because we are closing the hypothesis this Friday and need a day or two to create the new ag... [17:43:02] (03PS1) 10Jsn.sherman: Enable PersonalDashboard on Home pilot wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368350 (https://phabricator.wikimedia.org/T444355) [17:44:37] (03PS2) 10Jsn.sherman: Enable PersonalDashboard on Home pilot wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368350 (https://phabricator.wikimedia.org/T444355) [17:45:02] (03PS7) 10Michael Große: Growth: enable home beta on pilot and innovation wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361861 (https://phabricator.wikimedia.org/T443176) [17:45:23] (03CR) 10Urbanecm: [C:04-1] Growth: enable home beta on pilot and innovation wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361861 (https://phabricator.wikimedia.org/T443176) (owner: 10Michael Große) [17:45:48] (03CR) 10CI reject: [V:04-1] Growth: enable home beta on pilot and innovation wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361861 (https://phabricator.wikimedia.org/T443176) (owner: 10Michael Große) [17:48:10] (03PS8) 10Michael Große: Growth: enable home beta on pilot and innovation wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361861 (https://phabricator.wikimedia.org/T443176) [17:48:42] (03CR) 10Clare Ming: [C:03+2] Upgrade GrowthBook beyond 5.2.0 release [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368302 (https://phabricator.wikimedia.org/T443793) (owner: 10Santiago Faci) [17:50:23] (03Merged) 10jenkins-bot: Upgrade GrowthBook beyond 5.2.0 release [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368302 (https://phabricator.wikimedia.org/T443793) (owner: 10Santiago Faci) [17:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.28% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [17:58:13] (03CR) 10MusikAnimal: [C:03+1] Enable PersonalDashboard on Home pilot wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368350 (https://phabricator.wikimedia.org/T444355) (owner: 10Jsn.sherman) [18:00:00] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - thanos-query_443: Servers titan1002.eqiad.wmnet are marked down but pooled: thanos-web_443: Servers titan1001.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [18:00:03] FIRING: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1800) [18:00:57] FIRING: ProbeDown: Service thanos-query:443 has failed probes (http_thanos-query_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#thanos-query:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:01:00] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:01:30] !ack [18:01:31] All incidents are already acked. [18:01:45] hey [18:01:56] !incidents [18:01:57] 8732 (ACKED) ProbeDown sre (10.2.2.53 ip4 thanos-query:443 probes/service http_thanos-query_ip4 eqiad) [18:01:57] 8731 (RESOLVED) Host cr1-esams [18:02:10] Here, very close to the hotel. [18:02:20] I'll look at it. [18:03:04] (03CR) 10Dillon: [C:03+1] "LGTM, thanks!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368350 (https://phabricator.wikimedia.org/T444355) (owner: 10Jsn.sherman) [18:03:39] it was recommended to restart the service on the titan hosts by my dinner group [18:04:47] "Engineering by dinner group" [18:04:50] (03CR) 10Michael Große: Growth: enable home beta on pilot and innovation wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361861 (https://phabricator.wikimedia.org/T443176) (owner: 10Michael Große) [18:04:59] thanos-query looks ok [18:05:03] RESOLVED: [3x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:05:57] RESOLVED: ProbeDown: Service thanos-query:443 has failed probes (http_thanos-query_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#thanos-query:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:06:37] !incidents [18:06:37] 8732 (RESOLVED) ProbeDown sre (10.2.2.53 ip4 thanos-query:443 probes/service http_thanos-query_ip4 eqiad) [18:06:37] 8731 (RESOLVED) Host cr1-esams [18:06:48] \o/ [18:06:59] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 10 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368350 (https://phabricator.wikimedia.org/T444355) (owner: 10Jsn.sherman) [18:07:07] thanos-query logs indicate a possible failure in connectivity to the backend prometheus hosts [18:07:22] or, the prometheus hosts were overwhelmed and stopped responding to thanos [18:08:17] found a huge spike in RX on titan1002 - looks like possible QoD [18:08:47] I'll go offline again, metrics for thanos-query look better again [18:09:00] 10Gb RX - yeah way too much data [18:09:40] (03PS1) 10DLynch: editcheck-headless: Increase to 14 replicas and stop pinning testwiki [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368354 (https://phabricator.wikimedia.org/T432286) [18:09:42] (03PS1) 10DLynch: changeprop-jobqueue: Increase edit_suggestions_counts prefetch concurrency to 13 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368355 (https://phabricator.wikimedia.org/T432286) [18:11:04] Page is self-resolved and the services look happy after coughing a bit on that big bite. /me goes back to doing other things [18:12:11] (03PS2) 10DLynch: editcheck-headless: Increase to 14 replicas and stop pinning testwiki [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368354 (https://phabricator.wikimedia.org/T432286) [18:12:11] (03PS2) 10DLynch: changeprop-jobqueue: Increase edit_suggestions_counts prefetch concurrency to 13 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368355 (https://phabricator.wikimedia.org/T432286) [18:12:11] (03PS1) 10DLynch: editcheck-headless: upgrade image to 2026-11-05-212150-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368356 (https://phabricator.wikimedia.org/T432286) [18:27:19] (03PS2) 10DLynch: editcheck-headless: upgrade image to 2026-11-05-212150-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368356 (https://phabricator.wikimedia.org/T442767) [18:27:21] (03PS3) 10DLynch: editcheck-headless: Increase to 14 replicas and stop pinning testwiki [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368354 (https://phabricator.wikimedia.org/T442767) [18:27:23] (03PS3) 10DLynch: changeprop-jobqueue: Increase edit_suggestions_counts prefetch concurrency to 13 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368355 (https://phabricator.wikimedia.org/T442767) [18:29:17] (03CR) 10TrainBranchBot: [C:03+2] "Approved by tchin@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367743 (https://phabricator.wikimedia.org/T444095) (owner: 10Aqu) [18:30:03] (03Merged) 10jenkins-bot: EventStreamConfig: Refine app_base as medium job [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367743 (https://phabricator.wikimedia.org/T444095) (owner: 10Aqu) [18:30:07] (03PS2) 10Xcollazo: data_purge: Drop raw HDFS fsimage files older than 56 days [puppet] - 10https://gerrit.wikimedia.org/r/1360975 (https://phabricator.wikimedia.org/T438056) [18:30:27] !log tchin@deploy2003 Started scap sync-world: Backport for [[gerrit:1367743|EventStreamConfig: Refine app_base as medium job (T444095)]] [18:30:30] T444095: Refine: product_metrics.app_base hourly jobs failing, bump ingestion scale to medium - https://phabricator.wikimedia.org/T444095 [18:30:56] (03CR) 10CI reject: [V:04-1] data_purge: Drop raw HDFS fsimage files older than 56 days [puppet] - 10https://gerrit.wikimedia.org/r/1360975 (https://phabricator.wikimedia.org/T438056) (owner: 10Xcollazo) [18:32:27] (03PS3) 10Xcollazo: data_purge: Drop raw HDFS fsimage files older than 56 days [puppet] - 10https://gerrit.wikimedia.org/r/1360975 (https://phabricator.wikimedia.org/T438056) [18:32:45] !log tchin@deploy2003 tchin, aqu: Backport for [[gerrit:1367743|EventStreamConfig: Refine app_base as medium job (T444095)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [18:35:31] (03CR) 10Xcollazo: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360975 (https://phabricator.wikimedia.org/T438056) (owner: 10Xcollazo) [18:38:07] !log tchin@deploy2003 tchin, aqu: Continuing with deployment [18:41:07] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:44:08] !log tchin@deploy2003 Finished scap sync-world: Backport for [[gerrit:1367743|EventStreamConfig: Refine app_base as medium job (T444095)]] (duration: 13m 41s) [18:44:11] T444095: Refine: product_metrics.app_base hourly jobs failing, bump ingestion scale to medium - https://phabricator.wikimedia.org/T444095 [18:47:06] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [18:48:27] (03CR) 10Xcollazo: "PPC LGTM." [puppet] - 10https://gerrit.wikimedia.org/r/1360975 (https://phabricator.wikimedia.org/T438056) (owner: 10Xcollazo) [18:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [19:00:04] andre and brennen: MediaWiki train - Utc-0+Utc-7 Version (secondary timeslot) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T1900). Please do the needful. [19:00:12] nah [19:12:35] andre: jouncebot said please though... [19:14:30] jouncebot very friendly but not me [19:15:23] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:21:39] (03CR) 10Aqu: [C:03+1] data_purge: Drop raw HDFS fsimage files older than 56 days [puppet] - 10https://gerrit.wikimedia.org/r/1360975 (https://phabricator.wikimedia.org/T438056) (owner: 10Xcollazo) [19:33:34] (03PS1) 10Codename Noreste: Implement multiple proposals for Wikimedia Outreach [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368372 (https://phabricator.wikimedia.org/T444368) [19:37:22] FIRING: GnmiInterfaceCountersDrop: ... [19:37:22] cr1-esams is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=cr1-esams:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [19:49:04] (03CR) 10DLynch: [C:03+2] editcheck-headless: upgrade image to 2026-11-05-212150-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368356 (https://phabricator.wikimedia.org/T442767) (owner: 10DLynch) [19:50:46] (03Merged) 10jenkins-bot: editcheck-headless: upgrade image to 2026-11-05-212150-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368356 (https://phabricator.wikimedia.org/T442767) (owner: 10DLynch) [19:51:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:52:00] !log kemayo@deploy2003 helmfile [staging] START helmfile.d/services/editcheck-headless: apply [19:52:37] !log kemayo@deploy2003 helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply [19:54:33] (03CR) 10DLynch: [C:03+2] editcheck-headless: Increase to 14 replicas and stop pinning testwiki [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368354 (https://phabricator.wikimedia.org/T442767) (owner: 10DLynch) [19:55:38] (03Merged) 10jenkins-bot: editcheck-headless: Increase to 14 replicas and stop pinning testwiki [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368354 (https://phabricator.wikimedia.org/T442767) (owner: 10DLynch) [19:56:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:56:43] !log kemayo@deploy2003 helmfile [staging] START helmfile.d/services/editcheck-headless: apply [19:57:00] !log kemayo@deploy2003 helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply [19:57:52] !log kemayo@deploy2003 helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply [19:58:52] !log kemayo@deploy2003 helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply [20:01:00] !log kemayo@deploy2003 helmfile [codfw] START helmfile.d/services/editcheck-headless: apply [20:01:47] !log kemayo@deploy2003 helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply [20:03:33] (03CR) 10DLynch: [C:03+2] changeprop-jobqueue: Remove editSuggestionCountsPrecompute rule [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362096 (https://phabricator.wikimedia.org/T443442) (owner: 10Ottomata) [20:03:45] (03CR) 10DLynch: [C:03+2] changeprop-jobqueue: Increase edit_suggestions_counts prefetch concurrency to 13 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368355 (https://phabricator.wikimedia.org/T442767) (owner: 10DLynch) [20:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:04:44] (03Merged) 10jenkins-bot: changeprop-jobqueue: Remove editSuggestionCountsPrecompute rule [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362096 (https://phabricator.wikimedia.org/T443442) (owner: 10Ottomata) [20:05:23] (03Merged) 10jenkins-bot: changeprop-jobqueue: Increase edit_suggestions_counts prefetch concurrency to 13 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368355 (https://phabricator.wikimedia.org/T442767) (owner: 10DLynch) [20:08:14] !log kemayo@deploy2003 helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply [20:08:26] !log kemayo@deploy2003 helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply [20:08:53] !log kemayo@deploy2003 helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply [20:09:58] !log kemayo@deploy2003 helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [20:10:12] !log kemayo@deploy2003 helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [20:10:38] PROBLEM - MariaDB Replica Lag: s1 on db2250 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 624.07 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:11:29] !log kemayo@deploy2003 helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [20:17:29] (03PS1) 10DLynch: VisualEditor: Precompute edit suggestion counts on 100% of enwiki, jawiki, arwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368379 (https://phabricator.wikimedia.org/T442767) [20:17:47] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 10 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368379 (https://phabricator.wikimedia.org/T442767) (owner: 10DLynch) [20:44:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:49:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:56:48] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 10 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368372 (https://phabricator.wikimedia.org/T444368) (owner: 10Codename Noreste) [20:58:50] I have a patch to check in 2 minutes, for wikimedia outreach [21:00:05] RoanKattouw, urbanecm, TheresNoTime, and cjming: UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T2100). Please do the needful. [21:00:05] sbassett, JSherman, kemayo, and codenamenoreste: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [21:00:18] here [21:00:38] RECOVERY - MariaDB Replica Lag: s1 on db2250 is OK: OK slave_sql_lag Replication lag: 0.10 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:01:16] hehehe I cant wait for le sticker [21:01:53] o/ [21:01:53] Do we have a deployer on hand? I can self deploy if not. [21:02:02] o/ [21:02:07] I can deploy as well - I just have a config patch [21:02:12] me first whose patch should be deployed [21:02:23] I can self-deploy as well. Also, my patch has nothing I can test, so it can get bundled in with other config if people don't mind. [21:02:35] looks like it's all config? [21:02:42] Looks like. Do it all in one deploy? [21:02:47] sure, why not [21:02:50] fine by me [21:03:12] yep. i don’t think mine should explode the logs this time. [21:03:47] I'm happy to push the button for us, unless someone else would rather [21:03:50] i just added a user group for wikimedia outreach and a user right for admins as well as the block action for the abuse filter [21:04:00] go ahead, no worries, it wouldn't break probably anything [21:04:15] Go for it [21:04:57] i gotta go to my phone becausse i will not use this laptop in a while and can't test these new changes, thanks [21:05:13] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jsn@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368350 (https://phabricator.wikimedia.org/T444355) (owner: 10Jsn.sherman) [21:05:14] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jsn@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363494 (https://phabricator.wikimedia.org/T393212) (owner: 10SBassett) [21:05:14] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jsn@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368379 (https://phabricator.wikimedia.org/T442767) (owner: 10DLynch) [21:05:15] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jsn@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368372 (https://phabricator.wikimedia.org/T444368) (owner: 10Codename Noreste) [21:05:42] hmm; immediate errors. taking a look. [21:06:14] sbassett: you have a -2? [21:06:15] (03Merged) 10jenkins-bot: Enable PersonalDashboard on Home pilot wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368350 (https://phabricator.wikimedia.org/T444355) (owner: 10Jsn.sherman) [21:06:18] (03Merged) 10jenkins-bot: VisualEditor: Precompute edit suggestion counts on 100% of enwiki, jawiki, arwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368379 (https://phabricator.wikimedia.org/T442767) (owner: 10DLynch) [21:06:21] (03Merged) 10jenkins-bot: Implement multiple proposals for Wikimedia Outreach [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368372 (https://phabricator.wikimedia.org/T444368) (owner: 10Codename Noreste) [21:06:23] oh, ha, let me remove that [21:06:30] (03CR) 10SBassett: [C:03+1] Enable DBSC report-only mode on all Wikimedia wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363494 (https://phabricator.wikimedia.org/T393212) (owner: 10SBassett) [21:06:42] removed [21:06:53] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jsn@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363494 (https://phabricator.wikimedia.org/T393212) (owner: 10SBassett) [21:07:12] looking happier this time around [21:08:24] (03Merged) 10jenkins-bot: Enable DBSC report-only mode on all Wikimedia wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363494 (https://phabricator.wikimedia.org/T393212) (owner: 10SBassett) [21:08:53] !log jsn@deploy2003 Started scap sync-world: Backport for [[gerrit:1368350|Enable PersonalDashboard on Home pilot wikis (T444355)]], [[gerrit:1363494|Enable DBSC report-only mode on all Wikimedia wikis (T393212)]], [[gerrit:1368379|VisualEditor: Precompute edit suggestion counts on 100% of enwiki, jawiki, arwiki (T442767)]], [[gerrit:1368372|Implement multiple proposals for Wikimedia Outreach (T444368)]] [21:09:02] T444355: Deploy PersonalDashboard extension to Home pilot wikis - https://phabricator.wikimedia.org/T444355 [21:09:03] T393212: Implement Device Bound Session Credentials (DBSC) in MediaWiki - https://phabricator.wikimedia.org/T393212 [21:09:03] T442767: Deploy Headless VE to more traffic - https://phabricator.wikimedia.org/T442767 [21:09:03] T444368: Multiple proposals awaiting implementation on Wikimedia Outreach - https://phabricator.wikimedia.org/T444368 [21:09:42] no localization cache rebuild, woohoo [21:11:12] !log jsn@deploy2003 jsn, sbassett, codenamenoreste, kemayo: Backport for [[gerrit:1368350|Enable PersonalDashboard on Home pilot wikis (T444355)]], [[gerrit:1363494|Enable DBSC report-only mode on all Wikimedia wikis (T393212)]], [[gerrit:1368379|VisualEditor: Precompute edit suggestion counts on 100% of enwiki, jawiki, arwiki (T442767)]], [[gerrit:1368372|Implement multiple proposals for Wikimedia Outreach (T444368)]] sy [21:11:12] nced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:11:26] alrighty, please test everybody [21:11:30] (that is able) [21:14:24] mine is all good, just hit a few new wikis and they logged dbsc attempts for me [21:16:20] I remain untestable. 🥲 [21:17:09] mine is good [21:17:40] codenamenoreste: ? [21:19:21] ah, I see they said they can't test theirs [21:19:27] If they've vanished, it does look like everything in their patch is confined to outreach.wikimedia.org, so just rolling ahead with it is going to quite contained regardless. [21:19:36] agreed [21:19:43] !log jsn@deploy2003 jsn, sbassett, codenamenoreste, kemayo: Continuing with deployment [21:25:17] !log tchin@deploy2003 Started deploy [analytics/refinery@23a3975] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@23a39754] [21:25:48] !log jsn@deploy2003 Finished scap sync-world: Backport for [[gerrit:1368350|Enable PersonalDashboard on Home pilot wikis (T444355)]], [[gerrit:1363494|Enable DBSC report-only mode on all Wikimedia wikis (T393212)]], [[gerrit:1368379|VisualEditor: Precompute edit suggestion counts on 100% of enwiki, jawiki, arwiki (T442767)]], [[gerrit:1368372|Implement multiple proposals for Wikimedia Outreach (T444368)]] (duration: 16m 5 [21:25:48] 5s) [21:25:56] T444355: Deploy PersonalDashboard extension to Home pilot wikis - https://phabricator.wikimedia.org/T444355 [21:25:56] T393212: Implement Device Bound Session Credentials (DBSC) in MediaWiki - https://phabricator.wikimedia.org/T393212 [21:25:56] T442767: Deploy Headless VE to more traffic - https://phabricator.wikimedia.org/T442767 [21:25:57] T444368: Multiple proposals awaiting implementation on Wikimedia Outreach - https://phabricator.wikimedia.org/T444368 [21:26:02] !log tchin@deploy2003 Finished deploy [analytics/refinery@23a3975] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@23a39754] (duration: 00m 45s) [21:26:13] alrighty, we're deployed. Thanks all for being willing to consolidate! [21:26:13] Thanks for getting it all! [21:26:27] !log tchin@deploy2003 Started deploy [analytics/refinery@23a3975]: Regular analytics weekly train [analytics/refinery@23a39754] [21:30:56] FIRING: TransportLinksInUsageNoRedundancy: magru inbound transports link usage is at redundancy capacity - https://wikitech.wikimedia.org/wiki/Network_monitoring#TransportLinksInUsageNoRedundancy - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/bandwidth-site-totals?var-site=magru - https://alerts.wikimedia.org/?q=alertname%3DTransportLinksInUsageNoRedundancy [21:33:30] PROBLEM - MariaDB Replica Lag: s4 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 628.20 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:35:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:35:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker2015:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker2015 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [21:37:00] !log tchin@deploy2003 Finished deploy [analytics/refinery@23a3975]: Regular analytics weekly train [analytics/refinery@23a39754] (duration: 10m 33s) [21:40:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:44:22] can i squeeze in a backport? [21:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.32% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [21:56:38] PROBLEM - MariaDB Replica Lag: x4 on db2250 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 605.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:59:50] is anyone still using the window? [22:00:04] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261110T2200) [22:00:56] RESOLVED: TransportLinksInUsageNoRedundancy: magru inbound transports link usage is at redundancy capacity - https://wikitech.wikimedia.org/wiki/Network_monitoring#TransportLinksInUsageNoRedundancy - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/bandwidth-site-totals?var-site=magru - https://alerts.wikimedia.org/?q=alertname%3DTransportLinksInUsageNoRedundancy [22:06:58] dear Readers - can i use your deployment window to backport a fix to Test Kitchen for one of your (or RG's) experiments? [22:07:48] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368390 [22:09:33] (03PS1) 10Clare Ming: Force absent request headers to null [extensions/TestKitchen] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1368391 (https://phabricator.wikimedia.org/T444205) [22:09:47] (03PS1) 10Clare Ming: Force absent request headers to null [extensions/TestKitchen] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368392 (https://phabricator.wikimedia.org/T444205) [22:10:16] (03PS2) 10Clare Ming: Force absent request headers to null [extensions/TestKitchen] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368392 (https://phabricator.wikimedia.org/T444205) [22:11:00] i am going to be bold and run some backports - should be quick [22:11:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:12:06] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cjming@deploy2003 using scap backport" [extensions/TestKitchen] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1368391 (https://phabricator.wikimedia.org/T444205) (owner: 10Clare Ming) [22:12:06] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cjming@deploy2003 using scap backport" [extensions/TestKitchen] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368392 (https://phabricator.wikimedia.org/T444205) (owner: 10Clare Ming) [22:16:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:16:47] (03Merged) 10jenkins-bot: Force absent request headers to null [extensions/TestKitchen] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1368391 (https://phabricator.wikimedia.org/T444205) (owner: 10Clare Ming) [22:16:49] (03Merged) 10jenkins-bot: Force absent request headers to null [extensions/TestKitchen] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368392 (https://phabricator.wikimedia.org/T444205) (owner: 10Clare Ming) [22:17:16] !log cjming@deploy2003 Started scap sync-world: Backport for Force absent request headers to null (T444205) ([[gerrit:1368391|1.48.0-wmf.3]], [[gerrit:1368392|1.48.0-wmf.4]]) [22:17:20] T444205: [Semantic Search] log an exposure server-side at the conditional where $semanticEnabled is checked - https://phabricator.wikimedia.org/T444205 [22:19:36] !log cjming@deploy2003 cjming: Backport for Force absent request headers to null (T444205) ([[gerrit:1368391|1.48.0-wmf.3]], [[gerrit:1368392|1.48.0-wmf.4]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:20:16] !log cjming@deploy2003 cjming: Continuing with deployment [22:26:20] !log cjming@deploy2003 Finished scap sync-world: Backport for Force absent request headers to null (T444205) ([[gerrit:1368391|1.48.0-wmf.3]], [[gerrit:1368392|1.48.0-wmf.4]]) (duration: 09m 03s) [22:26:23] T444205: [Semantic Search] log an exposure server-side at the conditional where $semanticEnabled is checked - https://phabricator.wikimedia.org/T444205 [22:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [22:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [22:47:30] RECOVERY - MariaDB Replica Lag: s4 on db2239 is OK: OK slave_sql_lag Replication lag: 0.38 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [22:57:38] RECOVERY - MariaDB Replica Lag: x4 on db2250 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [23:02:04] (03CR) 10Cooltey: [C:03+2] wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368390 (owner: 10PipelineBot) [23:03:55] !log bking@cumin2003 `curl -XDELETE "https://search.svc.codfw.wmnet:9242/enwiki_general_1794311678" T444363 [23:03:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:03:58] T444363: Cirrussearch: Investigate CirrusSearchBackendMemoryIssue alerts in CODFW - https://phabricator.wikimedia.org/T444363 [23:05:23] FIRING: [8x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:10:23] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:10:42] FIRING: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:12:30] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 658.11 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [23:13:15] (03Merged) 10jenkins-bot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368390 (owner: 10PipelineBot) [23:15:23] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:15:42] RESOLVED: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:16:01] !log cooltey@deploy2003 helmfile [staging] START helmfile.d/services/wikifeeds: apply [23:16:24] !log cooltey@deploy2003 helmfile [staging] DONE helmfile.d/services/wikifeeds: apply [23:17:25] !log cooltey@deploy2003 helmfile [eqiad] START helmfile.d/services/wikifeeds: apply [23:17:53] !log cooltey@deploy2003 helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply [23:17:57] !log cooltey@deploy2003 helmfile [codfw] START helmfile.d/services/wikifeeds: apply [23:18:08] 10ops-codfw, 06SRE, 06DC-Ops: SM rail swap - sretest2010 - https://phabricator.wikimedia.org/T443460#12542413 (10RobH) FedEx tracking 500245945553 shows delivered on Monday. Papaul is out and Jenn is back next week. [23:18:20] !log cooltey@deploy2003 helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply [23:20:23] FIRING: [11x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:22:15] jouncebot: nowandnext [23:22:15] No deployments scheduled for the next 2 hour(s) and 37 minute(s) [23:22:16] In 2 hour(s) and 37 minute(s): Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T0200) [23:23:10] (03PS2) 10Zabe: Move reads/writes of wbc_entity_usage to x1 for wikidatawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367981 (https://phabricator.wikimedia.org/T439753) [23:25:35] (03CR) 10Zabe: [C:03+2] Move reads/writes of wbc_entity_usage to x1 for wikidatawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367981 (https://phabricator.wikimedia.org/T439753) (owner: 10Zabe) [23:26:36] (03Merged) 10jenkins-bot: Move reads/writes of wbc_entity_usage to x1 for wikidatawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367981 (https://phabricator.wikimedia.org/T439753) (owner: 10Zabe) [23:28:12] !log zabe@deploy2003 Started scap sync-world: Backport for [[gerrit:1367981|Move reads/writes of wbc_entity_usage to x1 for wikidatawiki (T439753)]] [23:28:15] T439753: Switch reads/writes of wbc_entity_usage from local cluster to x1 - https://phabricator.wikimedia.org/T439753 [23:30:29] !log zabe@deploy2003 zabe: Backport for [[gerrit:1367981|Move reads/writes of wbc_entity_usage to x1 for wikidatawiki (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [23:32:32] !log zabe@deploy2003 zabe: Continuing with deployment [23:34:15] FIRING: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-jobrunner - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [23:37:23] FIRING: GnmiInterfaceCountersDrop: ... [23:37:23] cr1-esams is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=cr1-esams:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [23:38:30] !log zabe@deploy2003 Finished scap sync-world: Backport for [[gerrit:1367981|Move reads/writes of wbc_entity_usage to x1 for wikidatawiki (T439753)]] (duration: 10m 18s) [23:38:34] T439753: Switch reads/writes of wbc_entity_usage from local cluster to x1 - https://phabricator.wikimedia.org/T439753 [23:39:15] RESOLVED: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-jobrunner - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [23:39:52] (03PS1) 10Dreamy Jazz: Set $wgGlobalBlockingCentralWikiContentLanguage to 'en' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368407 (https://phabricator.wikimedia.org/T444367) [23:40:35] jouncebot: nowandnext [23:40:35] No deployments scheduled for the next 2 hour(s) and 19 minute(s) [23:40:35] In 2 hour(s) and 19 minute(s): Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T0200) [23:40:52] zabe: You finished? [23:41:01] Dreamy_Jazz: yes [23:41:04] Thanks [23:41:34] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368407 (https://phabricator.wikimedia.org/T444367) (owner: 10Dreamy Jazz) [23:42:28] (03Merged) 10jenkins-bot: Set $wgGlobalBlockingCentralWikiContentLanguage to 'en' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368407 (https://phabricator.wikimedia.org/T444367) (owner: 10Dreamy Jazz) [23:42:52] !log dreamyjazz@deploy2003 Started scap sync-world: Backport for [[gerrit:1368407|Set $wgGlobalBlockingCentralWikiContentLanguage to 'en' (T444367)]] [23:42:55] T444367: Global autoblock language is in language of wiki the autoblock is created - https://phabricator.wikimedia.org/T444367 [23:45:09] !log dreamyjazz@deploy2003 dreamyjazz: Backport for [[gerrit:1368407|Set $wgGlobalBlockingCentralWikiContentLanguage to 'en' (T444367)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [23:46:57] !log dreamyjazz@deploy2003 dreamyjazz: Continuing with deployment [23:52:53] !log dreamyjazz@deploy2003 Finished scap sync-world: Backport for [[gerrit:1368407|Set $wgGlobalBlockingCentralWikiContentLanguage to 'en' (T444367)]] (duration: 10m 02s) [23:52:57] T444367: Global autoblock language is in language of wiki the autoblock is created - https://phabricator.wikimedia.org/T444367 [23:57:30] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.19 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response