[00:02:28] (03PS1) 10ArielGlenn: Changeprop beta: replace appservers.svc (mediawiki13) with new instance [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353510 [00:04:48] (03PS1) 10Ncmonitor: DNSRepository: Automated MarkMonitor domain sync [dns] - 10https://gerrit.wikimedia.org/r/1353513 [00:04:51] (03PS1) 10Ncmonitor: NCRedirRedirects: Automated MarkMonitor domain sync [puppet] - 10https://gerrit.wikimedia.org/r/1353514 [00:04:54] (03PS1) 10Ncmonitor: ACMEChiefConfig: Automated MarkMonitor domain sync [puppet] - 10https://gerrit.wikimedia.org/r/1353515 [00:17:35] 06SRE, 06Collaboration-Services, 06Traffic, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12414362 (10cmooney) The issue is the speed at which esams is pulling the file in from codfw at, which happens every time. Tbh it doesn't make much se... [00:38:18] 06SRE, 06Collaboration-Services, 06Traffic, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12414384 (10Bewfip) According to https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/bandwidth-site-totals, esams and drmrs should not... [00:39:22] 06SRE, 06Collaboration-Services, 06Traffic, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12414391 (10Dzahn) more notes (that I did not get to save yet, from earlier) - ATS isn't the cause, it's just slow without it (from curl testing from... [01:07:41] (03PS1) 10DLynch: BaseEditCheck: don't use toDataElement when looking for categories [extensions/VisualEditor] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353543 (https://phabricator.wikimedia.org/T440669) [01:08:48] Squeaking in one last backport before Friday. [01:09:11] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1353544 [01:09:11] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1353544 (owner: 10TrainBranchBot) [01:14:34] !log releases1003 - /usr/sbin/sysctl -w net.ipv4.tcp_congestion_control=bbr T440068 - downloads 120x times faster [01:14:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:14:38] T440068: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068 [01:14:51] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kemayo@deploy1003 using scap backport" [extensions/VisualEditor] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353543 (https://phabricator.wikimedia.org/T440669) (owner: 10DLynch) [01:16:04] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1353544 (owner: 10TrainBranchBot) [01:16:07] (03Merged) 10jenkins-bot: BaseEditCheck: don't use toDataElement when looking for categories [extensions/VisualEditor] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353543 (https://phabricator.wikimedia.org/T440669) (owner: 10DLynch) [01:17:35] 06SRE, 06Collaboration-Services, 06Traffic, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12414454 (10Dzahn) - Ganeti host isnt dropping packets either - roughly 0.8% loss is on the network beyond ganeti1033: the switches, the eqiad↔ulsfo... [01:17:54] 06SRE, 06Collaboration-Services, 06Traffic, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12414456 (10Dzahn) How about now, everybody? [01:20:15] !log kemayo@deploy1003 Started scap sync-world: Backport for [[gerrit:1353543|BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] [01:20:19] T440669: Checks using inCategory/notInCategory config are causing JS errors - https://phabricator.wikimedia.org/T440669 [01:24:14] !log kemayo@deploy1003 kemayo: Backport for [[gerrit:1353543|BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [01:24:25] FIRING: KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1081.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [01:25:03] !log kemayo@deploy1003 kemayo: Continuing with deployment [01:30:21] !log kemayo@deploy1003 Finished scap sync-world: Backport for [[gerrit:1353543|BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] (duration: 10m 10s) [01:30:25] T440669: Checks using inCategory/notInCategory config are causing JS errors - https://phabricator.wikimedia.org/T440669 [01:42:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:51:37] !log ssastry@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [01:52:05] !log ssastry@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [01:52:06] !log ssastry@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [01:52:38] !log ssastry@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [01:53:41] 06SRE, 06Collaboration-Services, 06Traffic, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12414546 (10Bewfip) Has improved for me. Though I’m curious why those dedicated inter-DC links have a relatively high loss rate. [02:00:31] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:08:09] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) [02:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [04:19:14] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:14:14] RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:24:25] FIRING: KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1081.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:30:59] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1019.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [05:31:59] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [05:42:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:42:50] !log oblivian@cumin1004 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet [05:42:57] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet [05:45:46] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet [05:45:54] !log oblivian@cumin1004 END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) pool for host wikikube-worker1057.eqiad.wmnet [05:50:57] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet [05:50:58] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1057.eqiad.wmnet [05:51:40] !log oblivian@cumin1004 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261009T0600) [06:00:11] (03PS1) 10Marostegui: monitoring.yaml: Add eqiad x4 [puppet] - 10https://gerrit.wikimedia.org/r/1353750 (https://phabricator.wikimedia.org/T437279) [06:04:59] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1014.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [06:05:59] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [06:20:59] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [06:21:59] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [06:24:39] !log oblivian@cumin1004 END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet [06:25:23] (03CR) 10Filippo Giunchedi: [C:03+1] Alert on node_textfile scrape error [alerts] - 10https://gerrit.wikimedia.org/r/1353299 (https://phabricator.wikimedia.org/T312516) (owner: 10Hnowlan) [06:28:28] 10ops-eqiad, 06DC-Ops: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1081.eqiad.wmnet - https://phabricator.wikimedia.org/T440683 (10Joe) 03NEW [06:37:54] (03CR) 10Marostegui: [C:03+2] monitoring.yaml: Add eqiad x4 [puppet] - 10https://gerrit.wikimedia.org/r/1353750 (https://phabricator.wikimedia.org/T437279) (owner: 10Marostegui) [06:41:13] (03CR) 10Huei Tan: [C:03+1] ULS: Define LanguageSelectorV2SupportedSkins config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353160 (https://phabricator.wikimedia.org/T439707) (owner: 10Abijeet Patro) [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261009T0700) [07:11:02] oblivian@cumin1004 enable_virtualization (PID 4131551) is awaiting input [07:15:03] (03PS1) 10Ozge: ml-services: Scale llm-qwen3-14b on concurrency [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353777 (https://phabricator.wikimedia.org/T440588) [07:21:13] (03CR) 10Ozge: "Hey, I ended up testing it with llm-qwen3-14b and we may move it to ml-serve1015 later." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353777 (https://phabricator.wikimedia.org/T440588) (owner: 10Ozge) [07:24:54] !log oblivian@cumin1004 START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1084-1087,1093-1095].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [07:24:57] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1084.eqiad.wmnet [07:25:30] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1084.eqiad.wmnet [07:25:50] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1084.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:29:52] (03PS11) 10Elukey: sre.hosts.reimage: refactor DHCP configs as cookbook calls [cookbooks] - 10https://gerrit.wikimedia.org/r/1348725 [07:30:32] (03CR) 10Slyngshede: [C:03+1] "Looks good, assuming that you trust RADOS to do the group check." [puppet] - 10https://gerrit.wikimedia.org/r/1352486 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [07:34:41] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1084.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:34:59] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1084.eqiad.wmnet [07:36:56] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1084.eqiad.wmnet [07:37:09] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1085.eqiad.wmnet [07:37:44] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1085.eqiad.wmnet [07:38:03] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1085.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:38:26] (03CR) 10JMeybohm: [C:03+2] thumbor: Add priorityClassName value [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353198 (https://phabricator.wikimedia.org/T420223) (owner: 10JMeybohm) [07:39:14] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:39:57] (03CR) 10Slyngshede: [C:03+1] "Please allow Moritz time to review as well. Otherwise okay from me." [puppet] - 10https://gerrit.wikimedia.org/r/1352484 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [07:39:58] (03Merged) 10jenkins-bot: thumbor: Add priorityClassName value [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353198 (https://phabricator.wikimedia.org/T420223) (owner: 10JMeybohm) [07:40:11] (03CR) 10Slyngshede: [C:03+1] idp_test: Accept wmf_s3_login requests only from the Ceph admin hosts [puppet] - 10https://gerrit.wikimedia.org/r/1352485 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [07:46:54] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1085.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:47:05] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1085.eqiad.wmnet [07:47:06] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1085.eqiad.wmnet [07:47:20] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1086.eqiad.wmnet [07:47:29] (03CR) 10JMeybohm: "Sorry, I don't recall really. Doing them one-by-one will be fine as well. But please be aware of https://phabricator.wikimedia.org/T382630" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346712 (https://phabricator.wikimedia.org/T438375) (owner: 10Herron) [07:47:52] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1086.eqiad.wmnet [07:48:12] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1086.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:48:25] (03CR) 10Arnaudb: gerrit: allow puppetservers to reach the envoy tlsproxy (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1328152 (https://phabricator.wikimedia.org/T420184) (owner: 10Arnaudb) [07:49:14] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:50:06] (03CR) 10Arnaudb: [C:03+2] "rolling this out with puppet disabled on primary at first" [puppet] - 10https://gerrit.wikimedia.org/r/1328152 (https://phabricator.wikimedia.org/T420184) (owner: 10Arnaudb) [07:51:43] (03CR) 10Muehlenhoff: gerrit: allow puppetservers to reach the envoy tlsproxy (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1328152 (https://phabricator.wikimedia.org/T420184) (owner: 10Arnaudb) [07:53:49] (03PS1) 10Muehlenhoff: Extend access for lwilson-ctr [puppet] - 10https://gerrit.wikimedia.org/r/1353832 [07:57:34] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1086.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:57:45] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1086.eqiad.wmnet [07:58:16] (03CR) 10Arnaudb: [C:03+2] "`" [puppet] - 10https://gerrit.wikimedia.org/r/1328152 (https://phabricator.wikimedia.org/T420184) (owner: 10Arnaudb) [07:59:00] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1086.eqiad.wmnet [07:59:14] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:59:14] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1087.eqiad.wmnet [07:59:47] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1087.eqiad.wmnet [08:00:07] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1087.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:02:34] 07Puppet, 06Collaboration-Services, 10Gerrit, 06Infrastructure-Foundations, 13Patch-For-Review: Change puppet-merge git origin to use gerrit.discovery.wmnet instead of gerrit.wikimedia.org - https://phabricator.wikimedia.org/T420184#12415017 (10ABran-WMF) [08:04:14] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:08:56] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1087.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:09:08] (03PS10) 10Elukey: sre.hosts.provision: move DHCP configs to the DHCP cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1347466 [08:09:08] (03PS12) 10Elukey: sre.hosts.reimage: refactor DHCP configs as cookbook calls [cookbooks] - 10https://gerrit.wikimedia.org/r/1348725 [08:09:15] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1087.eqiad.wmnet [08:09:25] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:09:37] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1087.eqiad.wmnet [08:09:51] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1093.eqiad.wmnet [08:10:23] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1093.eqiad.wmnet [08:10:44] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1093.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:12:22] (03PS1) 10Brouberol: ceph::admin: update CI secrets to reflect recent changes [labs/private] - 10https://gerrit.wikimedia.org/r/1353839 [08:12:32] (03CR) 10Brouberol: [C:03+2] ceph::admin: update CI secrets to reflect recent changes [labs/private] - 10https://gerrit.wikimedia.org/r/1353839 (owner: 10Brouberol) [08:12:34] (03CR) 10Brouberol: [V:03+2 C:03+2] ceph::admin: update CI secrets to reflect recent changes [labs/private] - 10https://gerrit.wikimedia.org/r/1353839 (owner: 10Brouberol) [08:14:14] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:17:30] 06SRE, 06Collaboration-Services, 06Traffic, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12415042 (10cmooney) >>! In T440068#12414454, @Dzahn wrote: > - roughly 0.8% loss is on the network beyond ganeti1033: the switches, the eqiad↔ulsfo >... [08:18:33] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1093.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:18:39] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1093.eqiad.wmnet [08:19:14] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:21:09] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1093.eqiad.wmnet [08:21:23] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1094.eqiad.wmnet [08:21:59] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1094.eqiad.wmnet [08:22:19] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1094.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:23:08] 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12415061 (10cmooney) @Jclark-ctr moved the uplink to the adjacent rack and everything is now working fine. I'll keep this task open... [08:24:14] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:30:10] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1094.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:30:16] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1094.eqiad.wmnet [08:32:14] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1094.eqiad.wmnet [08:32:28] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1095.eqiad.wmnet [08:33:04] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1095.eqiad.wmnet [08:33:23] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1095.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:34:14] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:34:31] (03PS1) 10Kevin Bazira: ml-services: deploy clef model-server [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353849 (https://phabricator.wikimedia.org/T440223) [08:34:35] (03CR) 10JMeybohm: [C:03+1] bgp: add port number to var-device. [alerts] - 10https://gerrit.wikimedia.org/r/1353249 (https://phabricator.wikimedia.org/T439944) (owner: 10Blake) [08:35:03] (03Abandoned) 10Cathal Mooney: network data: increase size of public1-ulsfo IPv4 range [puppet] - 10https://gerrit.wikimedia.org/r/1205135 (https://phabricator.wikimedia.org/T410047) (owner: 10Cathal Mooney) [08:35:30] 06SRE, 10Infrastructure Security, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07SecTeam-Processed, and 2 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12415084 (10Gehel) [08:37:18] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Q1:rack/setup/install 10 new ceph nodes - https://phabricator.wikimedia.org/T438213#12415143 (10Gehel) [08:39:00] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 2 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12415181 (10Gehel) [08:39:33] 10SRE-SLO, 10observability, 10Wikidata, 06Wikidata Platform Team, and 3 others: Update WDQS SLOs to reflect graph split changes - https://phabricator.wikimedia.org/T393966#12415195 (10Gehel) [08:40:04] (03CR) 10Ozge: [C:03+1] ml-services: deploy clef model-server [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353849 (https://phabricator.wikimedia.org/T440223) (owner: 10Kevin Bazira) [08:40:56] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12415233 (10Gehel) [08:41:08] 06SRE, 06Data-Engineering, 10Kafka-Infrastructure, 06serviceops-radar, and 4 others: Configuration Management for Kafka settings - https://phabricator.wikimedia.org/T276088#12415231 (10Gehel) [08:41:13] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1095.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:41:24] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1095.eqiad.wmnet [08:42:21] 06SRE, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Site: EQIAD VM request for Kerberos - https://phabricator.wikimedia.org/T438229#12415255 (10Gehel) [08:42:40] 06SRE, 10Kafka-Infrastructure, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Monitor kafka-configurator runs and pending changes - https://phabricator.wikimedia.org/T439506#12415260 (10Gehel) [08:42:45] RECOVERY - dump of s5 in codfw on backupmon1001 is OK: Last dump for s5 at codfw (db2201) taken on 2026-10-09 08:10:09 (53 GiB, +0.1 %) https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Rerun_a_failed_backup [08:43:11] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1095.eqiad.wmnet [08:43:12] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[1084-1087,1093-1095].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [08:44:02] 10ops-codfw, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Q1:rack/setup/install cirrussearch21[16-20] - https://phabricator.wikimedia.org/T436287#12415291 (10Gehel) [08:44:14] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:44:25] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:45:05] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [08:45:14] 10ops-eqiad, 06SRE, 06DC-Ops, 06Machine-Learning-Team, and 2 others: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12415323 (10Gehel) [08:45:30] (03CR) 10Kevin Bazira: [C:03+2] ml-services: deploy clef model-server [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353849 (https://phabricator.wikimedia.org/T440223) (owner: 10Kevin Bazira) [08:45:51] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Q1:rack/setup/install cirrussearch11[26-30] - https://phabricator.wikimedia.org/T436285#12415337 (10Gehel) [08:48:18] (03Merged) 10jenkins-bot: ml-services: deploy clef model-server [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353849 (https://phabricator.wikimedia.org/T440223) (owner: 10Kevin Bazira) [08:51:20] (03PS5) 10Jelto: sre.k8s: move ensure_k8s_services_active into init method [cookbooks] - 10https://gerrit.wikimedia.org/r/1353201 (https://phabricator.wikimedia.org/T437480) [08:51:26] (03PS1) 10Ladsgroup: Reduce the chunk size to 16MB [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1353853 (https://phabricator.wikimedia.org/T420223) [08:59:44] oblivian@cumin1004 enable_virtualization (PID 4154494) is awaiting input [09:04:07] (03CR) 10JMeybohm: wikikube: Add high-priority priority class (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353199 (https://phabricator.wikimedia.org/T420223) (owner: 10JMeybohm) [09:04:47] (03CR) 10JMeybohm: [C:03+1] Reduce the chunk size to 16MB [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1353853 (https://phabricator.wikimedia.org/T420223) (owner: 10Ladsgroup) [09:16:21] (03CR) 10Federico Ceratto: mysql: Fix fleetwide RO/RW setting (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1351376 (https://phabricator.wikimedia.org/T438831) (owner: 10Federico Ceratto) [09:18:27] (03CR) 10Ladsgroup: [C:03+2] "Let's go" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1353853 (https://phabricator.wikimedia.org/T420223) (owner: 10Ladsgroup) [09:18:51] !log jmm@cumin2003 START - Cookbook sre.hosts.decommission for hosts cuminunpriv1001.eqiad.wmnet [09:19:22] (03CR) 10Gabriel Birke: [C:03+1] Enable sub-references on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353180 (https://phabricator.wikimedia.org/T440243) (owner: 10WMDE-Fisch) [09:20:07] (03CR) 10Gabriel Birke: [C:03+1] Enable better autonames CommunityConfiguration on group0 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353173 (https://phabricator.wikimedia.org/T440357) (owner: 10WMDE-Fisch) [09:23:15] 10ops-codfw, 06SRE, 10database-backups, 06DBA, 06DC-Ops: db2201 (backup source s5 and x1) crashed - https://phabricator.wikimedia.org/T440283#12415537 (10Marostegui) s5 logical/snapshot were fixed too. [09:23:49] !log jmm@cumin2003 START - Cookbook sre.dns.netbox [09:25:17] (03Merged) 10jenkins-bot: Reduce the chunk size to 16MB [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1353853 (https://phabricator.wikimedia.org/T420223) (owner: 10Ladsgroup) [09:28:23] (03PS1) 10Elukey: profile::docker::reporter: update rules exclusions [puppet] - 10https://gerrit.wikimedia.org/r/1353857 (https://phabricator.wikimedia.org/T440697) [09:28:28] (03PS1) 10Gkyziridis: ml-services: Deploy latest version of revertrisk-wikidata on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353858 (https://phabricator.wikimedia.org/T433699) [09:28:48] (03Abandoned) 10Ladsgroup: [WIP] Read originals off the event loop in the file loader [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337314 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup-claude) [09:29:19] jmm@cumin2003 decommission (PID 319913) is awaiting input [09:33:44] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1353857 (https://phabricator.wikimedia.org/T440697) (owner: 10Elukey) [09:34:08] (03CR) 10Elukey: [C:03+2] profile::docker::reporter: update rules exclusions [puppet] - 10https://gerrit.wikimedia.org/r/1353857 (https://phabricator.wikimedia.org/T440697) (owner: 10Elukey) [09:35:32] 10ops-eqiad, 06DC-Ops, 06ServiceOps: Eqiad worker nodes should be connected to 10G NICs - https://phabricator.wikimedia.org/T440696#12415564 (10JMeybohm) [09:35:51] 10ops-eqiad, 06DC-Ops, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Eqiad worker nodes should be connected to 10G NICs - https://phabricator.wikimedia.org/T440696#12415565 (10JMeybohm) [09:36:24] (03PS1) 10Gkyziridis: ml-services: Remove remove llm-qwen36 and add llm-qwen38-27b in experimental ns [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353861 (https://phabricator.wikimedia.org/T440307) [09:37:13] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host druid1010.eqiad.wmnet with OS bookworm [09:40:11] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Eqiad worker nodes should be connected to 10G NICs - https://phabricator.wikimedia.org/T440696#12415570 (10JMeybohm) [09:40:46] !log Updated the Wikidata property suggester with data from 20261005's JSON dump [09:40:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:42:07] (03PS1) 10Ladsgroup: thumbor: Bump to head [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353862 (https://phabricator.wikimedia.org/T420223) [09:42:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:43:17] RECOVERY - Host druid1010 is UP: PING OK - Packet loss = 0%, RTA = 0.20 ms [09:44:00] 07sre-alert-triage, 06Infrastructure-Foundations, 10Kafka-Infrastructure, 06ServiceOps, and 2 others: Alert in need of triage: PKICertificateExpiry (instance pki2002:9100) - https://phabricator.wikimedia.org/T438703#12415591 (10LSobanski) a:03elukey @elukey will handle this early next week, thanks for th... [09:45:54] (03CR) 10Ladsgroup: [C:03+2] thumbor: Bump to head [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353862 (https://phabricator.wikimedia.org/T420223) (owner: 10Ladsgroup) [09:47:26] (03Merged) 10jenkins-bot: thumbor: Bump to head [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353862 (https://phabricator.wikimedia.org/T420223) (owner: 10Ladsgroup) [09:51:19] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [09:51:36] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Eqiad worker nodes should be connected to 10G NICs - https://phabricator.wikimedia.org/T440696#12415602 (10MLechvien-WMF) [09:52:37] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [09:53:38] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage [09:54:24] (03CR) 10AikoChou: [C:03+1] ml-services: Scale llm-qwen3-14b on concurrency (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353777 (https://phabricator.wikimedia.org/T440588) (owner: 10Ozge) [09:57:18] !log ladsgroup@deploy1003 helmfile [codfw] START helmfile.d/services/thumbor: apply [09:57:52] (03CR) 10Atsuko: [C:03+2] airflow3: tuning dag-processor [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353296 (https://phabricator.wikimedia.org/T435480) (owner: 10Atsuko) [09:58:01] (03CR) 10Btullis: [C:03+2] dumps: Keep pages-meta-history only in the 3 newest runs on replicas [puppet] - 10https://gerrit.wikimedia.org/r/1346629 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [09:58:56] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage [09:59:01] (03CR) 10AikoChou: [C:03+1] ml-services: Deploy latest version of revertrisk-wikidata on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353858 (https://phabricator.wikimedia.org/T433699) (owner: 10Gkyziridis) [09:59:22] !log ladsgroup@deploy1003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [09:59:38] (03CR) 10Btullis: [C:03+2] dumps: Keep pages-meta-history only in the 3 newest runs on replicas (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1346629 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [09:59:39] (03Merged) 10jenkins-bot: airflow3: tuning dag-processor [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353296 (https://phabricator.wikimedia.org/T435480) (owner: 10Atsuko) [10:01:06] !log jmm@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cuminunpriv1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" [10:02:22] !log jmm@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cuminunpriv1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" [10:02:23] !log jmm@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [10:02:24] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cuminunpriv1001.eqiad.wmnet [10:02:30] 06SRE, 06Infrastructure-Foundations: Sunset rootless Cumin - https://phabricator.wikimedia.org/T439522#12415616 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2003 for hosts: `cuminunpriv1001.eqiad.wmnet` - cuminunpriv1001.eqiad.wmnet (**PASS**) - Downtimed host on Icinga/Alertm... [10:02:51] !log ladsgroup@deploy1003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [10:04:03] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [10:04:15] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [10:04:37] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [10:05:51] !log ladsgroup@deploy1003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [10:11:46] (03PS6) 10Jelto: sre.k8s: move ensure_k8s_services_active into k8s module [cookbooks] - 10https://gerrit.wikimedia.org/r/1353201 (https://phabricator.wikimedia.org/T437480) [10:14:34] (03CR) 10JMeybohm: [C:03+1] sre.k8s: move ensure_k8s_services_active into k8s module [cookbooks] - 10https://gerrit.wikimedia.org/r/1353201 (https://phabricator.wikimedia.org/T437480) (owner: 10Jelto) [10:16:38] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on lswtest-d8-eqiad,lswtest-d8-eqiad IPv6,ssw1-d[1,8]-eqiad with reason: take rail kit from test switch [10:16:41] (03CR) 10Gkyziridis: [C:03+2] ml-services: Deploy latest version of revertrisk-wikidata on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353858 (https://phabricator.wikimedia.org/T433699) (owner: 10Gkyziridis) [10:17:10] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on sretest1006.eqiad.wmnet with reason: move sretest1006 to row A [10:18:13] (03Merged) 10jenkins-bot: ml-services: Deploy latest version of revertrisk-wikidata on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353858 (https://phabricator.wikimedia.org/T433699) (owner: 10Gkyziridis) [10:20:20] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host druid1010.eqiad.wmnet with OS bookworm [10:24:12] !log gkyziridis@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [10:26:08] (03CR) 10Jelto: [C:03+2] sre.k8s: move ensure_k8s_services_active into k8s module [cookbooks] - 10https://gerrit.wikimedia.org/r/1353201 (https://phabricator.wikimedia.org/T437480) (owner: 10Jelto) [10:28:25] (03Merged) 10jenkins-bot: sre.k8s: move ensure_k8s_services_active into k8s module [cookbooks] - 10https://gerrit.wikimedia.org/r/1353201 (https://phabricator.wikimedia.org/T437480) (owner: 10Jelto) [10:35:43] (03CR) 10Ozge: [C:03+2] ml-services: Scale llm-qwen3-14b on concurrency (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353777 (https://phabricator.wikimedia.org/T440588) (owner: 10Ozge) [10:36:19] (03CR) 10Vgutierrez: cache:haproxy: deprecating mathoid as a service endpoint (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1353052 (https://phabricator.wikimedia.org/T431372) (owner: 10Fabfur) [10:37:23] (03Merged) 10jenkins-bot: ml-services: Scale llm-qwen3-14b on concurrency [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353777 (https://phabricator.wikimedia.org/T440588) (owner: 10Ozge) [10:41:36] (03PS1) 10Gkyziridis: ml-services: Add gRPC deployment for revertrisk-wikidata on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353885 (https://phabricator.wikimedia.org/T433699) [10:47:11] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [10:48:18] (03CR) 10Gkyziridis: [C:03+2] ml-services: Remove remove llm-qwen36 and add llm-qwen38-27b in experimental ns [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353861 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [10:50:28] (03PS1) 10Muehlenhoff: Remove last bits of cuminunpriv [puppet] - 10https://gerrit.wikimedia.org/r/1353890 (https://phabricator.wikimedia.org/T439422) [10:50:41] (03Merged) 10jenkins-bot: ml-services: Remove remove llm-qwen36 and add llm-qwen38-27b in experimental ns [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353861 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [10:51:49] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [10:54:49] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [10:59:11] (03CR) 10AikoChou: [C:03+1] ml-services: Add gRPC deployment for revertrisk-wikidata on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353885 (https://phabricator.wikimedia.org/T433699) (owner: 10Gkyziridis) [11:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261009T0700) [11:00:05] jelto, arnoldokoth, mutante, and arnaudb: #bothumor I � Unicode. All rise for GitLab version upgrades deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261009T1100). [11:02:46] (03CR) 10Jelto: "one question in line" [debs/kubernetes] (v1.34) - 10https://gerrit.wikimedia.org/r/1346480 (owner: 10JMeybohm) [11:05:05] (03CR) 10Gkyziridis: [C:03+2] ml-services: Add gRPC deployment for revertrisk-wikidata on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353885 (https://phabricator.wikimedia.org/T433699) (owner: 10Gkyziridis) [11:06:50] (03Merged) 10jenkins-bot: ml-services: Add gRPC deployment for revertrisk-wikidata on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353885 (https://phabricator.wikimedia.org/T433699) (owner: 10Gkyziridis) [11:07:55] !log gkyziridis@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [11:08:18] (03CR) 10Ozge: [C:03+2] ml-services: Scale llm-qwen3-14b on concurrency (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353777 (https://phabricator.wikimedia.org/T440588) (owner: 10Ozge) [11:18:35] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.6 point update - https://phabricator.wikimedia.org/T434866#12415892 (10MoritzMuehlenhoff) [11:21:09] (03PS1) 10Zabe: Move reads/writes of wbc_entity_usage to x1 for ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353911 (https://phabricator.wikimedia.org/T439753) [11:21:25] (03CR) 10Muehlenhoff: [C:03+2] Remove last bits of cuminunpriv [puppet] - 10https://gerrit.wikimedia.org/r/1353890 (https://phabricator.wikimedia.org/T439422) (owner: 10Muehlenhoff) [11:23:13] (03CR) 10AikoChou: [C:03+1] ml-services: Scale llm-qwen3-14b on concurrency (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353777 (https://phabricator.wikimedia.org/T440588) (owner: 10Ozge) [11:24:45] 06SRE, 06Infrastructure-Foundations: Sunset rootless Cumin - https://phabricator.wikimedia.org/T439522#12415910 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff rootless cumin has been retired, but if we ever want it back we can simply restore https://gerrit.wikimedia.org/r/c/operations/... [11:25:24] (03CR) 10Ozge: [C:03+2] ml-services: Scale llm-qwen3-14b on concurrency (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353777 (https://phabricator.wikimedia.org/T440588) (owner: 10Ozge) [11:27:04] (03PS1) 10Btullis: jupyterhub: Show support details on the error pages [puppet] - 10https://gerrit.wikimedia.org/r/1353912 (https://phabricator.wikimedia.org/T403863) [11:28:50] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: Eqiad worker nodes should be connected to 10G NICs - https://phabricator.wikimedia.org/T440696#12415932 (10JMeybohm) @Ladsgroup mentioned {T399927} for a process to move to 10G without reimage [11:29:35] (03PS1) 10Ozge: ml-services: Raise llm-qwen3-14b concurrency target to 16 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353913 (https://phabricator.wikimedia.org/T440588) [11:29:48] (03PS2) 10Btullis: jupyterhub: Show support details on the error pages [puppet] - 10https://gerrit.wikimedia.org/r/1353912 (https://phabricator.wikimedia.org/T403863) [11:30:33] (03PS3) 10Btullis: jupyterhub: Show support details on the error pages [puppet] - 10https://gerrit.wikimedia.org/r/1353912 (https://phabricator.wikimedia.org/T403863) [11:31:01] (03CR) 10Ozge: "Hey Aiko," [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353913 (https://phabricator.wikimedia.org/T440588) (owner: 10Ozge) [11:32:13] (03PS5) 10Brouberol: ceph::admin: configure tf env vars properly and clone the dpe-infra repo [puppet] - 10https://gerrit.wikimedia.org/r/1353844 (https://phabricator.wikimedia.org/T435600) [11:33:29] !log fceratto@cumin1004 END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2901.codfw.wmnet onto db2902.codfw.wmnet [11:33:43] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db1901: test [11:33:46] !log fceratto@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1901: test [11:35:23] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db1902: test [11:35:26] !log fceratto@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1902: test [11:35:34] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db1903: test [11:35:37] !log fceratto@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1903: test [11:37:05] (03CR) 10AikoChou: [C:03+1] ml-services: Raise llm-qwen3-14b concurrency target to 16 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353913 (https://phabricator.wikimedia.org/T440588) (owner: 10Ozge) [11:37:21] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db2901: test [11:38:17] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db2902: test [11:38:34] (03CR) 10Ozge: [V:03+2 C:03+2] ml-services: Raise llm-qwen3-14b concurrency target to 16 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353913 (https://phabricator.wikimedia.org/T440588) (owner: 10Ozge) [11:40:16] (03Merged) 10jenkins-bot: ml-services: Raise llm-qwen3-14b concurrency target to 16 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353913 (https://phabricator.wikimedia.org/T440588) (owner: 10Ozge) [11:42:24] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [11:42:47] (03PS1) 10Muehlenhoff: Blocklisting more unused packet mangling/network scheduler modules [puppet] - 10https://gerrit.wikimedia.org/r/1353917 [11:43:41] (03PS1) 10AikoChou: ml-services: use 2 fixed replicas for qwen3-14b benchmarking [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353919 (https://phabricator.wikimedia.org/T431553) [11:46:43] (03CR) 10Btullis: ceph::admin: configure tf env vars properly and clone the dpe-infra repo (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1353844 (https://phabricator.wikimedia.org/T435600) (owner: 10Brouberol) [11:46:45] (03CR) 10Muehlenhoff: gerrit: allow puppetservers to reach the envoy tlsproxy (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1328152 (https://phabricator.wikimedia.org/T420184) (owner: 10Arnaudb) [11:51:27] (03PS1) 10Gkyziridis: ml-services: Remove Qwen38-27b from experimental ns. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353922 (https://phabricator.wikimedia.org/T440307) [11:52:26] !log fceratto@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2901: test [11:52:30] (03PS1) 10Trueg: WDQS: Raise the max size of a form parameter for wdqs-proxy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353923 (https://phabricator.wikimedia.org/T440699) [11:53:19] (03CR) 10Gkyziridis: [C:03+2] ml-services: Remove Qwen38-27b from experimental ns. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353922 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [11:53:22] !log fceratto@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2902: test [11:54:19] (03CR) 10Ozge: [C:03+1] ml-services: use 2 fixed replicas for qwen3-14b benchmarking [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353919 (https://phabricator.wikimedia.org/T431553) (owner: 10AikoChou) [11:54:45] (03Merged) 10jenkins-bot: ml-services: Remove Qwen38-27b from experimental ns. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353922 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [11:56:08] (03CR) 10AikoChou: [C:03+2] ml-services: use 2 fixed replicas for qwen3-14b benchmarking [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353919 (https://phabricator.wikimedia.org/T431553) (owner: 10AikoChou) [11:56:28] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [11:57:37] (03Merged) 10jenkins-bot: ml-services: use 2 fixed replicas for qwen3-14b benchmarking [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353919 (https://phabricator.wikimedia.org/T431553) (owner: 10AikoChou) [11:58:54] !log aikochou@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [12:00:08] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox: sync [12:00:11] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [12:00:14] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [12:00:28] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [12:05:22] (03PS2) 10JMeybohm: k8s: Remove LVS L2 adjacency handling [cookbooks] - 10https://gerrit.wikimedia.org/r/1351949 (https://phabricator.wikimedia.org/T420729) [12:05:28] (03CR) 10JMeybohm: k8s: Remove LVS L2 adjacency handling (032 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1351949 (https://phabricator.wikimedia.org/T420729) (owner: 10JMeybohm) [12:06:52] (03CR) 10JMeybohm: [C:03+2] conftool: Add remaining k8s nodes to conftool-data [puppet] - 10https://gerrit.wikimedia.org/r/1351940 (https://phabricator.wikimedia.org/T420729) (owner: 10JMeybohm) [12:11:57] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: Eqiad worker nodes should be connected to 10G NICs - https://phabricator.wikimedia.org/T440696#12416046 (10Jclark-ctr) Row A and B have not been upgraded to 10g yet | Server | Row | Rack | Position | Racking ticket | |---|---|---|---:|---| |... [12:12:19] (03PS6) 10Brouberol: ceph::admin: configure tf env vars properly and clone the dpe-infra repo [puppet] - 10https://gerrit.wikimedia.org/r/1353844 (https://phabricator.wikimedia.org/T435600) [12:13:27] !log oblivian@cumin1004 START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1113-1120].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [12:13:31] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1113.eqiad.wmnet [12:14:38] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1113.eqiad.wmnet [12:14:58] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1113.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:16:18] (03PS7) 10Brouberol: ceph::admin: configure tf env vars properly and clone the dpe-infra repo [puppet] - 10https://gerrit.wikimedia.org/r/1353844 (https://phabricator.wikimedia.org/T435600) [12:16:58] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 3 others: wikikube-worker1152.eqiad.wmnet networking issue - https://phabricator.wikimedia.org/T438819#12416054 (10Jclark-ctr) >>! In T438819#12409234, @Jclark-ctr wrote: > I think the board has other issues passed the nic. i am unable to use keyboard... [12:22:49] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1113.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:23:06] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1113.eqiad.wmnet [12:24:14] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:24:32] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1113.eqiad.wmnet [12:24:46] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1114.eqiad.wmnet [12:25:22] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1114.eqiad.wmnet [12:25:24] !log jayme@cumin1004 conftool action : set/weight=10; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:25:41] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1114.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:25:53] !log jayme@cumin1004 conftool action : set/weight=10; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:26:08] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1353917 (owner: 10Muehlenhoff) [12:26:38] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:39] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:39] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1329.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:39] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1330.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:40] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1331.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:40] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1332.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:40] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1333.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:40] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1334.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:41] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1347.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:41] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1362.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:42] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1363.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:42] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1364.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:43] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1365.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:43] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1366.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:44] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1367.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:44] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1368.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:45] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1369.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:45] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1370.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:46] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1373.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:46] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1374.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc [12:27:47] (03CR) 10Elukey: "I have reimaged a lot of sretest hosts, so far everything went really smoothly! Ready for review :)" [cookbooks] - 10https://gerrit.wikimedia.org/r/1348725 (owner: 10Elukey) [12:28:11] 06SRE, 10Maps, 06Traffic: Possibility to allow Wikimedia Maps usage on all Wikibase Cloud instances - https://phabricator.wikimedia.org/T429191#12416069 (10Anton.Kokh) @MSantos Could you please check if this could be approved? Thank you! [12:28:14] (03CR) 10Brouberol: ceph::admin: configure tf env vars properly and clone the dpe-infra repo (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1353844 (https://phabricator.wikimedia.org/T435600) (owner: 10Brouberol) [12:29:14] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:29:24] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2332.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:25] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2333.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:25] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2334.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:25] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2335.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:25] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2336.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:26] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2337.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:26] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2338.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:26] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2339.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:27] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2340.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:27] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2341.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:28] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2342.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:28] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2343.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:29] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2344.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:29] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2345.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:30] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2346.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:30] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2347.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:31] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2348.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:31] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2349.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:32] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2350.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:32] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2351.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:33] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2352.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:33] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2353.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:34] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2354.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:34] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2355.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:35] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2356.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:35] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2357.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:36] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2358.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:36] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2359.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:37] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2360.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:37] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2361.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:38] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2362.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:38] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2363.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:39] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2364.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:39] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2365.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:40] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2366.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:40] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2367.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:41] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2368.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:41] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2369.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:42] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2370.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:42] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2371.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:43] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2372.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:43] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2373.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:29:44] !log jayme@cumin1004 conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2374.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc [12:31:40] !log jayme@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1081.eqiad.wmnet with reason: hardware issues [12:31:40] (03PS1) 10Muehlenhoff: Blacklist udf [puppet] - 10https://gerrit.wikimedia.org/r/1353943 [12:31:44] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1081.eqiad.wmnet - https://phabricator.wikimedia.org/T440683#12416090 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=272a5094-1a1e-44a6-8311-3f5498247cda) set by jayme@cumin1004 for 14 days, 0:... [12:31:53] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, 10ServiceOps-Upgrades-Hardware: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1081.eqiad.wmnet - https://phabricator.wikimedia.org/T440683#12416093 (10JMeybohm) [12:33:31] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1114.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:33:43] 06SRE, 10SRE-Access-Requests: Setting up Yubikey for foks - https://phabricator.wikimedia.org/T440598#12416103 (10elukey) Reached out to Joe on Slack to get a confirmation about the new public key :) [12:33:49] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1114.eqiad.wmnet [12:34:14] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1113.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:35:29] (03CR) 10Elukey: "This needs to be discussed in next SRE I/F's meeting due to the sudo requests. I'll report back when done!" [puppet] - 10https://gerrit.wikimedia.org/r/1352546 (https://phabricator.wikimedia.org/T440471) (owner: 10Dr0ptp4kt) [12:35:47] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1114.eqiad.wmnet [12:36:01] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1115.eqiad.wmnet [12:36:12] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to dumps-roots for analytics_admins_members (people associated with analytics-admins) - https://phabricator.wikimedia.org/T440471#12416131 (10elukey) Hi folks! My understanding is that https://gerrit.wikimedia.org/r/c/operations/puppet/+/135... [12:36:34] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1115.eqiad.wmnet [12:36:54] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1115.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:39:14] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker1113.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:42:58] Hi there! I'd need to run a query on x1.wikishared to manually fix 3 rows. The query and details are in https://phabricator.wikimedia.org/T440557#12411029. I'll run it in a bit if there are no objections. [12:44:46] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1115.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:45:04] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1115.eqiad.wmnet [12:45:41] (03CR) 10Btullis: [C:03+1] ceph::admin: configure tf env vars properly and clone the dpe-infra repo [puppet] - 10https://gerrit.wikimedia.org/r/1353844 (https://phabricator.wikimedia.org/T435600) (owner: 10Brouberol) [12:45:46] RECOVERY - Host wikikube-worker1081 is UP: PING OK - Packet loss = 0%, RTA = 3.53 ms [12:46:30] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1115.eqiad.wmnet [12:46:44] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1116.eqiad.wmnet [12:47:16] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1116.eqiad.wmnet [12:47:36] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1116.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:47:57] (03CR) 10DCausse: [C:03+1] cirrussearch: Absent icinga-based masters-eligible checks [puppet] - 10https://gerrit.wikimedia.org/r/1352523 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [12:48:27] (03CR) 10DCausse: [C:03+1] cirrussearch: Remove icinga-based checks [puppet] - 10https://gerrit.wikimedia.org/r/1352543 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [12:50:04] (03PS8) 10Brouberol: ceph::admin: configure tf env vars properly and clone the dpe-infra repo [puppet] - 10https://gerrit.wikimedia.org/r/1353844 (https://phabricator.wikimedia.org/T435600) [12:54:36] (03CR) 10Btullis: [C:03+1] ceph::admin: configure tf env vars properly and clone the dpe-infra repo (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1353844 (https://phabricator.wikimedia.org/T435600) (owner: 10Brouberol) [12:54:55] !log cmooney@cumin1004 START - Cookbook sre.dns.netbox [12:55:59] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1116.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:56:16] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1116.eqiad.wmnet [12:58:03] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1116.eqiad.wmnet [12:58:18] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1117.eqiad.wmnet [12:58:38] !log cmooney@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries for sretest1006 - cmooney@cumin1004" [12:58:52] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1117.eqiad.wmnet [12:58:55] !log cmooney@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries for sretest1006 - cmooney@cumin1004" [12:58:55] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:59:14] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1117.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:59:27] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need a few more things in place before we can reimage it in new location [13:02:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:03:55] (03PS1) 10Elukey: admin: add full account and deployment access for dhardy [puppet] - 10https://gerrit.wikimedia.org/r/1353960 (https://phabricator.wikimedia.org/T440467) [13:05:50] (03CR) 10Arnaudb: [C:03+2] gerrit: allow puppetservers to reach the envoy tlsproxy (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1328152 (https://phabricator.wikimedia.org/T420184) (owner: 10Arnaudb) [13:06:08] (03PS1) 10Arnaudb: gerrit: track connections from puppetservers to envoy [puppet] - 10https://gerrit.wikimedia.org/r/1353962 (https://phabricator.wikimedia.org/T420184) [13:06:10] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 2 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12416187 (10brouberol) @cmooney Do we have a sense of when we should be ready to start? The kafka-jumbo1018 broker is now at 82% used storage (~+20%... [13:07:04] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1117.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:07:17] (03CR) 10Dr0ptp4kt: "Thank you, much appreciated @ltoscano@wikimedia.org" [puppet] - 10https://gerrit.wikimedia.org/r/1352546 (https://phabricator.wikimedia.org/T440471) (owner: 10Dr0ptp4kt) [13:07:31] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1117.eqiad.wmnet [13:08:05] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users + Kerberos for OBenhmida - https://phabricator.wikimedia.org/T440464#12416204 (10elukey) @OBenhmida Hi! Could you pleae check https://wikitech.wikimedia.org/wiki/Data_Platform/Data_access#What_access_should_I_request? and let me kno... [13:08:57] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.7 point update - https://phabricator.wikimedia.org/T437715#12416206 (10MoritzMuehlenhoff) [13:08:57] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1117.eqiad.wmnet [13:09:00] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 2 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12416207 (10cmooney) @brouberol we were delayed waiting on dc-ops to get the cabling done but we’ve made progress on that this week so things are fi... [13:09:11] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1118.eqiad.wmnet [13:09:14] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1116.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:09:25] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker1116.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:09:43] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1118.eqiad.wmnet [13:10:03] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1118.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:11:01] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 2 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12416210 (10brouberol) I //think// it should be fine, and I can reduce retention in the meantime, if we ever get to 90%. [13:11:46] (03PS1) 10Daniel Kinzler: REST Gateway: add trust expiry for rlc claims. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353970 (https://phabricator.wikimedia.org/T440060) [13:12:02] (03CR) 10CI reject: [V:04-1] REST Gateway: add trust expiry for rlc claims. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353970 (https://phabricator.wikimedia.org/T440060) (owner: 10Daniel Kinzler) [13:16:33] (03CR) 10Brouberol: [C:03+2] ceph::admin: configure tf env vars properly and clone the dpe-infra repo (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1353844 (https://phabricator.wikimedia.org/T435600) (owner: 10Brouberol) [13:17:06] (03CR) 10Scott French: docroot/wikimedia.org: update assetlink.json for credential sharing on root domain (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335409 (https://phabricator.wikimedia.org/T427929) (owner: 10Jasmine) [13:17:55] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1118.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:18:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [13:18:13] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1118.eqiad.wmnet [13:18:14] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1118.eqiad.wmnet [13:18:28] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1119.eqiad.wmnet [13:18:57] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, 10ServiceOps-Upgrades-Hardware: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1081.eqiad.wmnet - https://phabricator.wikimedia.org/T440683#12416226 (10Jclark-ctr) 05Open→03Resolved a:03Jclark-ctr Updated all firmwares this serv... [13:19:04] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1119.eqiad.wmnet [13:19:24] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1119.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:19:47] !log elukey@cumin1004 START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:19:49] (03PS1) 10Jelto: deployment_server::deploy_service::service_catalog: add miscweb-aux [puppet] - 10https://gerrit.wikimedia.org/r/1353973 (https://phabricator.wikimedia.org/T412941) [13:21:51] (03PS2) 10Jelto: deployment_server::deploy_service::service_catalog: add miscweb-aux [puppet] - 10https://gerrit.wikimedia.org/r/1353973 (https://phabricator.wikimedia.org/T412941) [13:23:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [13:23:56] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, 10ServiceOps-Upgrades-Hardware: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1081.eqiad.wmnet - https://phabricator.wikimedia.org/T440683#12416230 (10Jclark-ctr) 05Resolved→03Open [13:27:14] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1119.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:27:32] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1119.eqiad.wmnet [13:29:10] (03PS1) 10Brouberol: ceph/admin: clone the dpe-infra before generating the config file [puppet] - 10https://gerrit.wikimedia.org/r/1353978 (https://phabricator.wikimedia.org/T435600) [13:29:14] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1118.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:30:02] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1119.eqiad.wmnet [13:30:16] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1120.eqiad.wmnet [13:30:34] (03PS2) 10Daniel Kinzler: REST Gateway: add trust expiry for rlc claims. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353970 (https://phabricator.wikimedia.org/T440060) [13:30:47] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1120.eqiad.wmnet [13:31:07] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1120.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:31:37] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1353962 (https://phabricator.wikimedia.org/T420184) (owner: 10Arnaudb) [13:34:14] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker1118.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:35:16] (03CR) 10Atsuko: [C:03+1] jupyterhub: Show support details on the error pages [puppet] - 10https://gerrit.wikimedia.org/r/1353912 (https://phabricator.wikimedia.org/T403863) (owner: 10Btullis) [13:37:27] elukey@cumin1004 provision (PID 44056) is awaiting input [13:38:13] !log elukey@cumin1004 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:38:28] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1120.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:38:47] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1120.eqiad.wmnet [13:39:14] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1119.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:39:20] (03CR) 10Btullis: [C:03+1] ceph/admin: clone the dpe-infra before generating the config file [puppet] - 10https://gerrit.wikimedia.org/r/1353978 (https://phabricator.wikimedia.org/T435600) (owner: 10Brouberol) [13:39:52] !log elukey@cumin1004 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest2003.codfw.wmnet'] [13:40:24] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1120.eqiad.wmnet [13:40:25] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[1113-1120].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [13:41:06] (03CR) 10Brouberol: [C:03+2] ceph/admin: clone the dpe-infra before generating the config file [puppet] - 10https://gerrit.wikimedia.org/r/1353978 (https://phabricator.wikimedia.org/T435600) (owner: 10Brouberol) [13:42:28] (03PS1) 10Elukey: admin: add new ssh key for foks [puppet] - 10https://gerrit.wikimedia.org/r/1353985 (https://phabricator.wikimedia.org/T440598) [13:43:08] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1081.eqiad.wmnet - https://phabricator.wikimedia.org/T440683#12416285 (10Jclark-ctr) 05Open→03Resolved [13:44:14] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker1119.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:45:06] (03CR) 10Gmodena: [C:03+1] "+1 to unblock testing. I would challenge the idea that sending 50k bytes in urlencoded parameters is a good approach." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353923 (https://phabricator.wikimedia.org/T440699) (owner: 10Trueg) [13:45:56] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad: WMCS switch refresh 2026 - rack new switches - https://phabricator.wikimedia.org/T440313#12416304 (10Jclark-ctr) Both switches have been Racked in C8 and D5 [13:46:47] (03CR) 10Btullis: [C:03+2] jupyterhub: Show support details on the error pages [puppet] - 10https://gerrit.wikimedia.org/r/1353912 (https://phabricator.wikimedia.org/T403863) (owner: 10Btullis) [13:48:04] (03CR) 10Lerickson: [C:03+1] WDQS: Raise the max size of a form parameter for wdqs-proxy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353923 (https://phabricator.wikimedia.org/T440699) (owner: 10Trueg) [13:50:47] (03CR) 10FNegri: [C:03+2] toolhub: Update urldownloader endpoints [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352561 (https://phabricator.wikimedia.org/T439732) (owner: 10FNegri) [13:52:22] (03Merged) 10jenkins-bot: toolhub: Update urldownloader endpoints [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352561 (https://phabricator.wikimedia.org/T439732) (owner: 10FNegri) [13:53:43] !log fnegri@deploy1003 helmfile [staging] START helmfile.d/services/toolhub: apply [13:54:14] !log fnegri@deploy1003 helmfile [staging] DONE helmfile.d/services/toolhub: apply [13:55:50] !log fnegri@deploy1003 helmfile [codfw] START helmfile.d/services/toolhub: apply [13:56:46] !log fnegri@deploy1003 helmfile [codfw] DONE helmfile.d/services/toolhub: apply [13:56:49] !log fnegri@deploy1003 helmfile [eqiad] START helmfile.d/services/toolhub: apply [13:57:09] !log fnegri@deploy1003 helmfile [eqiad] DONE helmfile.d/services/toolhub: apply [13:57:44] (03CR) 10Trueg: [C:03+2] WDQS: Raise the max size of a form parameter for wdqs-proxy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353923 (https://phabricator.wikimedia.org/T440699) (owner: 10Trueg) [13:59:17] (03Merged) 10jenkins-bot: WDQS: Raise the max size of a form parameter for wdqs-proxy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353923 (https://phabricator.wikimedia.org/T440699) (owner: 10Trueg) [13:59:58] !log elukey@cumin1004 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['sretest2003.codfw.wmnet'] [14:08:12] (03PS1) 10Brouberol: ceph/admin: harmonize where the tf code gets pull to with WMCS [puppet] - 10https://gerrit.wikimedia.org/r/1353992 (https://phabricator.wikimedia.org/T435600) [14:10:37] (03CR) 10Brouberol: [C:03+2] ceph/admin: harmonize where the tf code gets pull to with WMCS [puppet] - 10https://gerrit.wikimedia.org/r/1353992 (https://phabricator.wikimedia.org/T435600) (owner: 10Brouberol) [14:16:14] !log elukey@cumin1004 START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:19:11] (03CR) 10Scott French: [C:03+1] deployment_server::deploy_service::service_catalog: add miscweb-aux [puppet] - 10https://gerrit.wikimedia.org/r/1353973 (https://phabricator.wikimedia.org/T412941) (owner: 10Jelto) [14:19:36] (03PS1) 10Brouberol: ceph/admin: only keep a single account for now [puppet] - 10https://gerrit.wikimedia.org/r/1353997 (https://phabricator.wikimedia.org/T435600) [14:20:00] !log ammarpad@deploy1003 mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Abstract Wikipedia team' 'Abstract Wikipedia' Ammarpad --reason 'requested at [[:phab:T440606|T440606]]' [14:20:04] T440606: Request to move translatable page: Abstract Wikipedia team - https://phabricator.wikimedia.org/T440606 [14:20:17] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.7 point update - https://phabricator.wikimedia.org/T437715#12416397 (10MoritzMuehlenhoff) [14:22:22] (03CR) 10Brouberol: [C:03+2] ceph/admin: only keep a single account for now [puppet] - 10https://gerrit.wikimedia.org/r/1353997 (https://phabricator.wikimedia.org/T435600) (owner: 10Brouberol) [14:24:32] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [14:25:01] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [14:26:02] (03PS1) 10FNegri: toolhub: fix urldownloader endpoints [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353999 (https://phabricator.wikimedia.org/T439732) [14:27:12] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [14:30:51] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [14:32:09] !log elukey@cumin1004 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:32:21] !log elukey@cumin1004 START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:33:04] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:36:49] (03CR) 10Andrew Bogott: [C:03+1] toolhub: fix urldownloader endpoints [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353999 (https://phabricator.wikimedia.org/T439732) (owner: 10FNegri) [14:37:12] (03CR) 10FNegri: [C:03+2] toolhub: fix urldownloader endpoints [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353999 (https://phabricator.wikimedia.org/T439732) (owner: 10FNegri) [14:38:41] (03Merged) 10jenkins-bot: toolhub: fix urldownloader endpoints [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353999 (https://phabricator.wikimedia.org/T439732) (owner: 10FNegri) [14:39:21] !log fnegri@deploy1003 helmfile [staging] START helmfile.d/services/toolhub: apply [14:39:45] !log fnegri@deploy1003 helmfile [staging] DONE helmfile.d/services/toolhub: apply [14:40:07] !log fnegri@deploy1003 helmfile [codfw] START helmfile.d/services/toolhub: apply [14:40:23] !log fnegri@deploy1003 helmfile [codfw] DONE helmfile.d/services/toolhub: apply [14:40:27] !log fnegri@deploy1003 helmfile [eqiad] START helmfile.d/services/toolhub: apply [14:40:39] !log fnegri@deploy1003 helmfile [eqiad] DONE helmfile.d/services/toolhub: apply [14:40:40] (03PS1) 10Elukey: installserver: move sretest2003 to legacy BIOS [puppet] - 10https://gerrit.wikimedia.org/r/1354004 (https://phabricator.wikimedia.org/T440711) [14:42:57] (03PS1) 10Jdlrobson: Re-Display success message if user has already consented [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1354005 (https://phabricator.wikimedia.org/T438248) [14:45:53] (03PS1) 10Brouberol: ceph/admin: replace analytics id placeholder by real one [puppet] - 10https://gerrit.wikimedia.org/r/1354006 (https://phabricator.wikimedia.org/T435600) [14:46:39] (03PS2) 10Brouberol: ceph/admin: replace analytics id placeholder by real one [puppet] - 10https://gerrit.wikimedia.org/r/1354006 (https://phabricator.wikimedia.org/T435600) [14:48:49] (03PS3) 10Brouberol: ceph/admin: replace analytics id placeholder by real one [puppet] - 10https://gerrit.wikimedia.org/r/1354006 (https://phabricator.wikimedia.org/T435600) [14:49:22] (03CR) 10Btullis: [C:03+1] ceph/admin: replace analytics id placeholder by real one [puppet] - 10https://gerrit.wikimedia.org/r/1354006 (https://phabricator.wikimedia.org/T435600) (owner: 10Brouberol) [14:50:21] (03CR) 10Bking: [C:03+2] cirrussearch: Absent icinga-based masters-eligible checks [puppet] - 10https://gerrit.wikimedia.org/r/1352523 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [14:55:55] (03CR) 10Ahmon Dancy: Changeprop beta: replace appservers.svc (mediawiki13) with new instance (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353510 (owner: 10ArielGlenn) [14:57:30] (03PS1) 10Mark Bergsma: wmf-update-ssh-config: Don't hardcode UID 1000 in agent socket paths [debs/wmf-laptop] - 10https://gerrit.wikimedia.org/r/1354008 [14:57:30] (03PS1) 10Mark Bergsma: wmf-update-ssh-config: Allow for an alternate template path [debs/wmf-laptop] - 10https://gerrit.wikimedia.org/r/1354009 [14:57:30] (03PS1) 10Mark Bergsma: wmf-update-ssh-config: Enable multi-platform ssh-agent activation [debs/wmf-laptop] - 10https://gerrit.wikimedia.org/r/1354010 [14:57:31] (03PS1) 10Mark Bergsma: wmf-update-ssh-config: Use ~/.ssh/agent for agent sockets on macOS [debs/wmf-laptop] - 10https://gerrit.wikimedia.org/r/1354011 [14:57:32] (03PS1) 10Mark Bergsma: Add a Homebrew formula for macOS [debs/wmf-laptop] - 10https://gerrit.wikimedia.org/r/1354012 [14:57:33] (03PS1) 10Mark Bergsma: wmf-update-ssh-config: Start the ssh-agents with launchd on macOS [debs/wmf-laptop] - 10https://gerrit.wikimedia.org/r/1354013 [14:57:37] (03PS1) 10Mark Bergsma: Formula: Run the known hosts refresh as a brew service [debs/wmf-laptop] - 10https://gerrit.wikimedia.org/r/1354014 [14:57:41] (03PS1) 10Mark Bergsma: Document the macOS setup [debs/wmf-laptop] - 10https://gerrit.wikimedia.org/r/1354015 [15:01:39] FIRING: [2x] RejectingBGPPrefixes: Rejecting 55 BGP prefixes - ssw1-a1-eqiad (10.66.0.25 - group Switch) - https://wikitech.wikimedia.org/wiki/Network_monitoring#RejectingBGPPrefixes - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqiad&var-device=cr1-eqiad:9804&var-bgp_group=Switch&var-bgp_neighbor=ssw1-a1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DRejectingBGPPrefixes [15:04:20] (03CR) 10Ahmon Dancy: Changeprop beta: replace appservers.svc (mediawiki13) with new instance (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353510 (owner: 10ArielGlenn) [15:05:20] (03CR) 10CDanis: [C:03+1] wmf-update-ssh-config: Don't hardcode UID 1000 in agent socket paths [debs/wmf-laptop] - 10https://gerrit.wikimedia.org/r/1354008 (owner: 10Mark Bergsma) [15:06:32] (03CR) 10Brouberol: [C:03+2] ceph/admin: replace analytics id placeholder by real one [puppet] - 10https://gerrit.wikimedia.org/r/1354006 (https://phabricator.wikimedia.org/T435600) (owner: 10Brouberol) [15:07:09] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12416592 (10fnegri) I think this task can be resolved, no references are left (only comment lines): `lang=shell-session... [15:09:25] 06SRE, 06Infrastructure-Foundations: Provide a live Debian image to act as a console - https://phabricator.wikimedia.org/T78135#12416604 (10fgiunchedi) [15:10:11] (03CR) 10ArielGlenn: Changeprop beta: replace appservers.svc (mediawiki13) with new instance (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353510 (owner: 10ArielGlenn) [15:10:26] (03CR) 10CDanis: wmf-update-ssh-config: Use ~/.ssh/agent for agent sockets on macOS (031 comment) [debs/wmf-laptop] - 10https://gerrit.wikimedia.org/r/1354011 (owner: 10Mark Bergsma) [15:10:37] (03PS1) 10Jdlrobson: Hide "maybe later" button for campaign URL [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1354020 (https://phabricator.wikimedia.org/T438251) [15:11:37] (03CR) 10CDanis: wmf-update-ssh-config: Enable multi-platform ssh-agent activation (031 comment) [debs/wmf-laptop] - 10https://gerrit.wikimedia.org/r/1354010 (owner: 10Mark Bergsma) [15:17:09] (03PS1) 10Cathal Mooney: SR-Linux: Set MED value on exported routes to IGP cost [homer/public] - 10https://gerrit.wikimedia.org/r/1354024 (https://phabricator.wikimedia.org/T439535) [15:18:30] (03PS2) 10Cathal Mooney: SR-Linux: Set MED value on exported routes to IGP cost [homer/public] - 10https://gerrit.wikimedia.org/r/1354024 (https://phabricator.wikimedia.org/T439535) [15:18:39] 06SRE, 06Collaboration-Services, 06Traffic, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12416633 (10Dzahn) >>! In T440068#12414362, @cmooney wrote: > The issue is the speed at which esams is pulling the file from codfw, which happens every... [15:22:59] (03CR) 10Mark Bergsma: wmf-update-ssh-config: Use ~/.ssh/agent for agent sockets on macOS (031 comment) [debs/wmf-laptop] - 10https://gerrit.wikimedia.org/r/1354011 (owner: 10Mark Bergsma) [15:26:27] (03PS1) 10Hnowlan: alertmanager: Handle DataSourceNoData error in Grafana [puppet] - 10https://gerrit.wikimedia.org/r/1354026 (https://phabricator.wikimedia.org/T391793) [15:36:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [15:36:14] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.7 point update - https://phabricator.wikimedia.org/T437715#12416717 (10MoritzMuehlenhoff) [15:38:41] 10ops-eqiad, 06SRE, 06DC-Ops, 10procurement, 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Investigate sizing and pricing options for dse-k8s expansion - https://phabricator.wikimedia.org/T440717 (10Gehel) 03NEW [15:38:57] (03CR) 10Cathal Mooney: [C:03+2] Interface validator: check Nokia breakout ports have parent [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1347593 (https://phabricator.wikimedia.org/T439537) (owner: 10Cathal Mooney) [15:40:39] (03Merged) 10jenkins-bot: Interface validator: check Nokia breakout ports have parent [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1347593 (https://phabricator.wikimedia.org/T439537) (owner: 10Cathal Mooney) [15:40:48] (03CR) 10Andrew Bogott: [C:03+1] openstack: enable performance governor in nova::common [puppet] - 10https://gerrit.wikimedia.org/r/1352452 (https://phabricator.wikimedia.org/T439729) (owner: 10Filippo Giunchedi) [15:40:49] (03PS1) 10Jdlrobson: Drop MediaViewerMobileBeta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1348770 (https://phabricator.wikimedia.org/T436799) [15:41:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [15:42:18] !log oblivian@cumin1004 START - Cookbook sre.hosts.remove-downtime for wikikube-worker1057.eqiad.wmnet [15:42:18] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1057.eqiad.wmnet [15:42:32] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1081.eqiad.wmnet [15:42:33] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1081.eqiad.wmnet [15:42:42] !log oblivian@cumin1004 START - Cookbook sre.hosts.remove-downtime for wikikube-worker1081.eqiad.wmnet [15:42:43] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1081.eqiad.wmnet [15:42:58] !log cmooney@cumin1004 START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary [15:44:26] !log cmooney@cumin1004 END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary [15:44:30] !log cmooney@cumin1004 START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox [15:46:54] !log cmooney@cumin1004 START - Cookbook sre.dns.netbox [15:49:28] (03PS1) 10Giuseppe Lavagetto: tox: update style checkers [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1354031 [15:49:28] (03PS1) 10Giuseppe Lavagetto: Version bump [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1354032 [15:49:28] (03PS1) 10Giuseppe Lavagetto: Convert project to use pyproject.toml [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1354033 [15:49:42] (03PS2) 10Jdlrobson: EditWatchlist: use same view on mobile and desktop [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347658 (https://phabricator.wikimedia.org/T363032) [15:49:46] (03CR) 10CI reject: [V:04-1] Version bump [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1354032 (owner: 10Giuseppe Lavagetto) [15:49:48] (03CR) 10CI reject: [V:04-1] Convert project to use pyproject.toml [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1354033 (owner: 10Giuseppe Lavagetto) [15:50:40] !log cmooney@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge new dns entries for eqiad ab links - cmooney@cumin1004" [15:50:43] !log cmooney@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge new dns entries for eqiad ab links - cmooney@cumin1004" [15:50:43] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [15:51:23] (03CR) 10Giuseppe Lavagetto: [C:03+2] tox: update style checkers [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1354031 (owner: 10Giuseppe Lavagetto) [15:53:15] (03Merged) 10jenkins-bot: tox: update style checkers [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1354031 (owner: 10Giuseppe Lavagetto) [15:58:15] 06SRE, 06Collaboration-Services, 06Traffic, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12416816 (10cmooney) >>! In T440068#12416633, @Dzahn wrote: > Just checking on this part. This should be about "esams <-> eqiad" link. the currently ac... [16:03:53] 06SRE, 06Collaboration-Services, 06Traffic, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12416845 (10Dzahn) 05Open→03Resolved a:03Dzahn Thank you very much. Let me close this out once the change is properly in puppet. [16:04:00] cmooney@cumin1004 update-extras (PID 63213) is awaiting input [16:04:06] 06SRE, 06Collaboration-Services, 06Traffic, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12416848 (10Dzahn) 05Resolved→03Open [16:05:31] !log cmooney@cumin1004 END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox [16:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:20:53] (03PS1) 10Cathal Mooney: SR-Linux: add basic support for breakout ports [homer/public] - 10https://gerrit.wikimedia.org/r/1354034 (https://phabricator.wikimedia.org/T439537) [16:21:53] (03CR) 10CI reject: [V:04-1] SR-Linux: add basic support for breakout ports [homer/public] - 10https://gerrit.wikimedia.org/r/1354034 (https://phabricator.wikimedia.org/T439537) (owner: 10Cathal Mooney) [16:22:44] (03PS2) 10Cathal Mooney: SR-Linux: add basic support for breakout ports [homer/public] - 10https://gerrit.wikimedia.org/r/1354034 (https://phabricator.wikimedia.org/T439537) [16:27:16] (03PS1) 10Cathal Mooney: Eqiad row a/b: add definitions for new switches for migration [homer/public] - 10https://gerrit.wikimedia.org/r/1354035 (https://phabricator.wikimedia.org/T439535) [16:28:15] (03CR) 10Cathal Mooney: Eqiad row a/b: add definitions for new switches for migration (031 comment) [homer/public] - 10https://gerrit.wikimedia.org/r/1354035 (https://phabricator.wikimedia.org/T439535) (owner: 10Cathal Mooney) [16:30:34] (03Abandoned) 10ArielGlenn: Changeprop beta: replace appservers.svc (mediawiki13) with new instance [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353510 (owner: 10ArielGlenn) [16:36:10] (03PS1) 10Dzahn: releases: use BBR TCP congestion control [puppet] - 10https://gerrit.wikimedia.org/r/1354036 (https://phabricator.wikimedia.org/T440068) [16:37:31] (03PS1) 10Dzahn: gerrit: use BBR TCP congestion control [puppet] - 10https://gerrit.wikimedia.org/r/1354037 (https://phabricator.wikimedia.org/T440068) [16:43:00] (03CR) 10Dzahn: [C:03+2] releases: use BBR TCP congestion control [puppet] - 10https://gerrit.wikimedia.org/r/1354036 (https://phabricator.wikimedia.org/T440068) (owner: 10Dzahn) [16:43:40] (03CR) 10Dzahn: [C:03+2] "for the prod machine - just puppetizing manual hack already applied since yesterday" [puppet] - 10https://gerrit.wikimedia.org/r/1354036 (https://phabricator.wikimedia.org/T440068) (owner: 10Dzahn) [16:44:53] (03CR) 10Dzahn: [C:03+2] "modules/base/manifests/sysctl.pp: $use_bbr = lookup('bbr_congestion_control', {'default_value' => false})" [puppet] - 10https://gerrit.wikimedia.org/r/1354036 (https://phabricator.wikimedia.org/T440068) (owner: 10Dzahn) [16:47:03] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on netbox1003.eqiad.wmnet with reason: Restoring DB from backup on netboxdb1003 [16:47:22] (03CR) 10Dzahn: [C:03+2] "noop on releases1003, added on inactive host" [puppet] - 10https://gerrit.wikimedia.org/r/1354036 (https://phabricator.wikimedia.org/T440068) (owner: 10Dzahn) [16:48:00] 06SRE, 06Collaboration-Services, 06Traffic, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12417085 (10Dzahn) 05Open→03Resolved [16:48:55] 06SRE, 06Collaboration-Services, 06Traffic, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12417086 (10Dzahn) We might also want to do this for certain other services. (gerrit, gitlab, people?) that offer large downloads to end users and... [16:50:46] (03CR) 10Herron: [C:03+1] eventgate-logging-external: replace kafka-logging200[1-3] with 200[6-8] [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351204 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [16:50:55] 10SRE-Access-Requests: Set up new SSH public key for production access for jgiannelos - https://phabricator.wikimedia.org/T440729 (10Jgiannelos) 03NEW [16:52:09] (03PS3) 10Bking: cirrussearch: Remove icinga-based checks [puppet] - 10https://gerrit.wikimedia.org/r/1352543 (https://phabricator.wikimedia.org/T431304) [16:53:05] (03CR) 10Herron: [C:03+1] team-sre: add exporter-based host_down check [alerts] - 10https://gerrit.wikimedia.org/r/1351431 (https://phabricator.wikimedia.org/T395448) (owner: 10Hnowlan) [16:53:32] (03CR) 10Bking: [C:03+2] cirrussearch: Remove icinga-based checks [puppet] - 10https://gerrit.wikimedia.org/r/1352543 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [16:54:03] 10SRE-Access-Requests: Set up new SSH public key for production access for jgiannelos - https://phabricator.wikimedia.org/T440729#12417108 (10Jgiannelos) [16:54:11] (03CR) 10Herron: [C:03+1] alertmanager: Handle DataSourceNoData error in Grafana [puppet] - 10https://gerrit.wikimedia.org/r/1354026 (https://phabricator.wikimedia.org/T391793) (owner: 10Hnowlan) [17:03:44] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12417125 (10cmooney) >>! In T418012#12411083, @Jclark-ctr wrote: > Finished cabling for the A/B upgrade. Everything is complete except for **1 cross-connect... [17:04:10] (03PS1) 10Bking: cirrussearch: finish remove icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/1354041 (https://phabricator.wikimedia.org/T431304) [17:04:25] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1354041 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [17:05:28] (03PS1) 10Subramanya Sastry: Undeploy Parsoid readviews on frwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1354042 (https://phabricator.wikimedia.org/T440695) [17:05:36] (03PS2) 10Bking: cirrussearch: finish remove icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/1354041 (https://phabricator.wikimedia.org/T431304) [17:06:35] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1354041 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [17:07:05] (03CR) 10Dzahn: "tested from esams and ulsfo at about 4–5 MB/s with cubic, limited by archive generation, so not needed." [puppet] - 10https://gerrit.wikimedia.org/r/1354037 (https://phabricator.wikimedia.org/T440068) (owner: 10Dzahn) [17:07:17] (03Abandoned) 10Dzahn: gerrit: use BBR TCP congestion control [puppet] - 10https://gerrit.wikimedia.org/r/1354037 (https://phabricator.wikimedia.org/T440068) (owner: 10Dzahn) [17:08:45] 06SRE, 06Collaboration-Services, 06Traffic, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12417164 (10Dzahn) Follow-up tests, all with cubic (default), 100 MB / 42 MB files: * people2004 (codfw) -> cp3070 (esams), direct: 24 MB/s... [17:21:39] RESOLVED: [2x] RejectingBGPPrefixes: Rejecting 55 BGP prefixes - ssw1-a1-eqiad (10.66.0.25 - group Switch) - https://wikitech.wikimedia.org/wiki/Network_monitoring#RejectingBGPPrefixes - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqiad&var-device=cr1-eqiad:9804&var-bgp_group=Switch&var-bgp_neighbor=ssw1-a1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DRejectingBGPPrefix [17:22:03] (03CR) 10Dzahn: [C:03+1] "can confirm that file does not exist anymore: modules/icinga/files/elasticsearch/check_masters_eligible.py: cannot open `modules/icinga/fi" [puppet] - 10https://gerrit.wikimedia.org/r/1354041 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [17:22:10] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad: WMCS switch refresh 2026 - rack new switches - https://phabricator.wikimedia.org/T440313#12417216 (10Jclark-ctr) @cmooney I moved the existing SCS cables from cloudsw1 to the cloudsw3 switches. I verified the SCS connections, u... [17:23:21] (03CR) 10Bking: [C:03+2] cirrussearch: finish remove icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/1354041 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [17:35:55] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad: WMCS switch refresh 2026 - rack new switches - https://phabricator.wikimedia.org/T440313#12417289 (10cmooney) Cool thanks. If we can run the following links from the Juniper switches in those racks to the new ones that'd be gre... [17:38:46] !log Run database query from T440557#12411029 in x1.wikishared [17:38:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:38:50] T440557: Restore three participants accidentally removed from a completed Event Registration - https://phabricator.wikimedia.org/T440557 [18:11:04] (03PS1) 10Dduvall: dockerfile: Add verify_stage template function [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1354049 (https://phabricator.wikimedia.org/T439490) [18:18:33] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 10Ceph, and 2 others: Q1:rack/setup/install apus-be100[7-9] - https://phabricator.wikimedia.org/T436181#12417517 (10Jclark-ctr) @cmooney I have these three servers I'd like to rack in A3, B3, and C1 using the new Nokia switches. If you have any concerns, let me know... [18:18:50] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 10Ceph, and 2 others: Q1:rack/setup/install apus-be100[7-9] - https://phabricator.wikimedia.org/T436181#12417518 (10Jclark-ctr) [18:20:34] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 10Ceph, and 2 others: Q1:rack/setup/install apus-be100[7-9] - https://phabricator.wikimedia.org/T436181#12417533 (10cmooney) >>! In T436181#12417516, @Jclark-ctr wrote: > @cmooney I have these three servers I'd like to rack in A3, B3, and C1 using the new Nokia swit... [18:21:01] (03PS2) 10Dduvall: dockerfile: Add verify_stage template function [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1354049 (https://phabricator.wikimedia.org/T439490) [18:21:57] (03CR) 10Dduvall: "My first Claude assisted patch. I feel a little sick but here we are. :/" [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1354049 (https://phabricator.wikimedia.org/T439490) (owner: 10Dduvall) [18:23:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:24:23] (03CR) 10C. Scott Ananian: [C:03+1] Undeploy Parsoid readviews on frwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1354042 (https://phabricator.wikimedia.org/T440695) (owner: 10Subramanya Sastry) [18:26:40] wpao, thcipriani: hey SREs, content platform team would like to do an Emergency Deploy of https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1354042 to address T440695 (cite breakage on fr.wikisource.org when using ). we can self-deploy with spiderpig. [18:26:41] T440695: Deactivate Parsoid from fr.wikisource - https://phabricator.wikimedia.org/T440695 [18:28:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:28:32] (03PS3) 10Dduvall: dockerfile: Add verify_stage template function [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1354049 (https://phabricator.wikimedia.org/T439490) [18:28:46] I need an emergency deploy for https://gerrit.wikimedia.org/r/1354042 -- context is T440695, are SRE ok with a deployment? (cc: thcipriani wpao brennen). I already have someone to deploy. (sorry, repost to use the official template) [18:29:24] also, andre aklapper is the train conductor but i don't think they are online here and/or i don't know how to map phabricator usernames to irc usernames. brennen is listed as the backup so I cc'ed you above, sorry. [18:30:47] hey cscott - i think this is ok from a releng / train perspective. [18:31:11] cscott: Fine with me (releng) I'd say, if you'll still be around for a few hours afterwards and watch Logstash (because... it's Friday) [18:32:32] *un*deploying parsoid should be very safe as config changes go. the major risk would be cache churn, but for a small wiki like this it won't be a problem. i'll keep an eye on grafana to be safe, and us east coast CPTers will be around for hours yet. [18:32:35] FYI, IRC Nick is shown on the Phab user profile (if the user entered that value in their Phab profile) [18:33:14] ah, ok, good to know. i was looking in the ak* in the nick list and forgot to check andre* :) [18:34:02] i think i need an ok from thcipriani and/or wpao before proceeding, but wpao is also not online? [18:34:28] oh, sorry, i was reading this wrong again. it's the SREs on call, that's elukey and fabfur, not the manager on call which is wpao. [18:34:41] elukey, fabfur: ^ emergency deploy request [18:35:39] wpao would be w.iki_willy in here (added a dot not to ping) [18:36:24] the room topic should probably give the irc nick then :) [18:39:41] (03PS1) 10CDanis: ats: multi-dc: test domain override edge case [puppet] - 10https://gerrit.wikimedia.org/r/1354056 [18:39:41] (03PS1) 10CDanis: trafficserver: multi-dc: fix key inheritance [puppet] - 10https://gerrit.wikimedia.org/r/1354057 [18:39:57] (03PS2) 10CDanis: trafficserver: multi-dc: test domain override edge case [puppet] - 10https://gerrit.wikimedia.org/r/1354056 [18:39:57] (03PS2) 10CDanis: trafficserver: multi-dc: fix key inheritance [puppet] - 10https://gerrit.wikimedia.org/r/1354057 [18:40:26] cscott: go for it [18:41:11] (03PS4) 10Dduvall: Refactor tests to run inline with build [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1345668 (https://phabricator.wikimedia.org/T439490) [18:42:05] ok thanks! [18:42:56] (03CR) 10CI reject: [V:04-1] trafficserver: multi-dc: test domain override edge case [puppet] - 10https://gerrit.wikimedia.org/r/1354056 (owner: 10CDanis) [18:43:02] (03PS3) 10CDanis: trafficserver: multi-dc: fix key inheritance [puppet] - 10https://gerrit.wikimedia.org/r/1354057 [18:43:14] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cscott@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1354042 (https://phabricator.wikimedia.org/T440695) (owner: 10Subramanya Sastry) [18:43:21] (03PS3) 10CDanis: trafficserver: multi-dc: test domain override edge case [puppet] - 10https://gerrit.wikimedia.org/r/1354056 [18:43:29] (03PS4) 10CDanis: trafficserver: multi-dc: fix key inheritance [puppet] - 10https://gerrit.wikimedia.org/r/1354057 [18:43:29] (03CR) 10CI reject: [V:04-1] trafficserver: multi-dc: fix key inheritance [puppet] - 10https://gerrit.wikimedia.org/r/1354057 (owner: 10CDanis) [18:44:28] (03Merged) 10jenkins-bot: Undeploy Parsoid readviews on frwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1354042 (https://phabricator.wikimedia.org/T440695) (owner: 10Subramanya Sastry) [18:44:45] !log cscott@deploy1003 Started scap sync-world: Backport for [[gerrit:1354042|Undeploy Parsoid readviews on frwikisource (T440695)]] [18:44:48] T440695: Deactivate Parsoid from fr.wikisource - https://phabricator.wikimedia.org/T440695 [18:45:52] (03CR) 10CI reject: [V:04-1] trafficserver: multi-dc: test domain override edge case [puppet] - 10https://gerrit.wikimedia.org/r/1354056 (owner: 10CDanis) [18:48:45] !log cscott@deploy1003 cscott, ssastry: Backport for [[gerrit:1354042|Undeploy Parsoid readviews on frwikisource (T440695)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [18:48:49] cscott: ok for me, sorry it didn't paged [18:49:23] !log cscott@deploy1003 cscott, ssastry: Continuing with deployment [18:49:32] (03CR) 10JHathaway: [C:03+1] profile::puppetserver::volatile: Remove GeoIP fetch without private access [puppet] - 10https://gerrit.wikimedia.org/r/1353306 (owner: 10Ahmon Dancy) [18:54:43] !log cscott@deploy1003 Finished scap sync-world: Backport for [[gerrit:1354042|Undeploy Parsoid readviews on frwikisource (T440695)]] (duration: 09m 58s) [18:54:47] T440695: Deactivate Parsoid from fr.wikisource - https://phabricator.wikimedia.org/T440695 [18:58:18] great all done, thanks everyone. [19:08:44] (03PS1) 10JHathaway: facts: remove top scope :: [puppet] - 10https://gerrit.wikimedia.org/r/1354061 (https://phabricator.wikimedia.org/T438912) [19:09:04] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1354061 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [19:16:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:19:11] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad: WMCS switch refresh 2026 - rack new switches - https://phabricator.wikimedia.org/T440313#12417704 (10Jclark-ctr) These cables have been connected >>! In T440313#12417289, @cmooney wrote: > Cool thanks. If we can run the foll... [19:20:45] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 10Ceph, and 2 others: Q1:rack/setup/install apus-be100[7-9] - https://phabricator.wikimedia.org/T436181#12417709 (10Jclark-ctr) >>! In T436181#12417533, @cmooney wrote: >>>! In T436181#12417516, @Jclark-ctr wrote: >> @cmooney I have these three servers I'd like to r... [19:21:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:33:23] (03PS1) 10Bking: cirrussearch: Absent unneeded shard size check. [puppet] - 10https://gerrit.wikimedia.org/r/1354068 (https://phabricator.wikimedia.org/T431304) [19:39:51] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1354068 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [19:40:02] (03PS1) 10Bking: cirrussearch: Remove unneeded shard size check and related script [puppet] - 10https://gerrit.wikimedia.org/r/1354070 (https://phabricator.wikimedia.org/T431304) [19:41:48] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1354070 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [20:06:12] (03PS2) 10JHathaway: facts: remove top scope :: [puppet] - 10https://gerrit.wikimedia.org/r/1354061 (https://phabricator.wikimedia.org/T438912) [20:06:20] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1354061 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [20:14:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:19:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:30:11] (03CR) 10JHathaway: [C:03+2] facts: remove top scope :: [puppet] - 10https://gerrit.wikimedia.org/r/1354061 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [21:04:05] (03PS1) 10Scott French: hieradata: Divert testwiki traffic to mw-pretrain for mw-jobrunner [puppet] - 10https://gerrit.wikimedia.org/r/1346775 (https://phabricator.wikimedia.org/T427666) [21:04:07] (03PS16) 10Scott French: hieradata: Divert testwiki traffic to mw-pretrain for mw-api-int [puppet] - 10https://gerrit.wikimedia.org/r/1328249 (https://phabricator.wikimedia.org/T427666) [21:07:47] (03PS1) 10Scott French: debug.json: Add k8s-mw-pretrain debug backends [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346796 (https://phabricator.wikimedia.org/T427666) [21:22:13] (03PS1) 10Bking: kerberos: Add new VM replica [puppet] - 10https://gerrit.wikimedia.org/r/1354085 (https://phabricator.wikimedia.org/T438229) [21:25:12] (03PS1) 10Bking: WIP: kerberos: Present newly-provisioned host to clients [puppet] - 10https://gerrit.wikimedia.org/r/1354088 (https://phabricator.wikimedia.org/T438229) [21:25:37] (03PS2) 10Bking: kerberos: Present newly-provisioned host to clients [puppet] - 10https://gerrit.wikimedia.org/r/1354088 (https://phabricator.wikimedia.org/T438229) [21:26:21] (03CR) 10Bking: [C:04-1] "Do not merge until manual provisioning steps described above are complete." [puppet] - 10https://gerrit.wikimedia.org/r/1354088 (https://phabricator.wikimedia.org/T438229) (owner: 10Bking) [22:47:17] 06SRE: Bing Webmaster Tools access request for Ash Han - https://phabricator.wikimedia.org/T439646#12418500 (10bcampbell) 05Declined→03Open I’m reopening this, since ITS can’t grant BWT access. ITS took over Google Search Console provisioning in Sept 2025, but BWT wasn’t part of that handoff. [[ https://wiki... [22:59:45] (03PS4) 10Dzahn: site/codesearch: test codesearch puppet role [puppet] - 10https://gerrit.wikimedia.org/r/1342800 (https://phabricator.wikimedia.org/T268199) [22:59:45] (03CR) 10Dzahn: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342800 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [23:00:59] (03PS5) 10Dzahn: site/codesearch: test codesearch puppet role [puppet] - 10https://gerrit.wikimedia.org/r/1342800 (https://phabricator.wikimedia.org/T268199) [23:02:15] (03CR) 10Dzahn: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342800 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [23:17:58] (03PS1) 10Dzahn: codesearch: add envoy to the role [puppet] - 10https://gerrit.wikimedia.org/r/1354098 [23:18:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:39:01] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1354100 [23:39:01] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1354100 (owner: 10TrainBranchBot) [23:45:24] (03CR) 10CI reject: [V:04-1] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1354100 (owner: 10TrainBranchBot) [23:53:41] RECOVERY - snapshot of x4 in codfw on backupmon1001 is OK: Last snapshot for x4 at codfw (db2250) taken on 2026-10-09 23:07:48 (1057 GiB, +0.1 %) https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Rerun_a_failed_backup