[00:00:07] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [00:00:17] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [00:00:21] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [00:00:31] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [00:00:34] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [00:00:41] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [00:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:05:23] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [00:30:05] (03Abandoned) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1367525 (owner: 10TrainBranchBot) [00:38:54] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1367598 [00:38:54] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1367598 (owner: 10TrainBranchBot) [00:45:33] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1367598 (owner: 10TrainBranchBot) [00:45:44] PROBLEM - Host titan1002 is DOWN: PING CRITICAL - Packet loss = 100% [00:45:50] RECOVERY - Host titan1002 is UP: PING OK - Packet loss = 0%, RTA = 290.34 ms [00:47:03] FIRING: ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:52:03] RESOLVED: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:08:55] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1367605 [01:08:55] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1367605 (owner: 10TrainBranchBot) [01:14:02] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1367605 (owner: 10TrainBranchBot) [01:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 98.92% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [02:00:04] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261108T0800) [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261109T0200) [02:00:19] !log mwpresync@deploy2003 Started scap build-images: Publishing wmf/next image [02:08:20] !log mwpresync@deploy2003 Finished scap build-images: Publishing wmf/next image (duration: 08m 01s) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:14:24] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:47:06] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [02:50:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [02:55:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [04:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:05:23] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [05:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 98.99% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [06:14:24] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [06:47:06] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [07:25:23] (03CR) 10Jelto: [C:03+2] Add FIDO SSH key for eileen [puppet] - 10https://gerrit.wikimedia.org/r/1363505 (https://phabricator.wikimedia.org/T442759) (owner: 10Dwisehaupt) [08:00:05] Amir1, urbanecm, awight, and chlod: gettimeofday() says it's time for UTC morning backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261109T0800) [08:00:05] matthiasmullie and TausheefHassan: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [08:00:05] o/ [08:00:16] o/ [08:00:18] o/ [08:00:19] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy2003 using scap backport" [extensions/ReaderExperiments] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367552 (owner: 10Matthias Mullie) [08:01:21] (03Merged) 10jenkins-bot: Squashed diff to master [extensions/ReaderExperiments] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367552 (owner: 10Matthias Mullie) [08:01:52] need a deployer, TausheefHassan? i see matthiasmullie is already underway with theirs [08:02:12] I do need a deployer [08:02:22] I'm around as well. Ping me if you need extra hands [08:02:42] TausheefHassan: gotcha, i can deploy, Amir1: ack, thanks! [08:02:58] !log mlitn@deploy2003 Started scap sync-world: Backport for [[gerrit:1367552|Squashed diff to master]] (1.48.0-wmf.3) [08:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:05:23] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [08:07:08] (03PS4) 10Hashar: gerrit: allow `gerrit index` in readonly plugin [puppet] - 10https://gerrit.wikimedia.org/r/1358246 (https://phabricator.wikimedia.org/T440177) [08:13:11] (03CR) 10Arnaudb: [C:03+2] gerrit: fix the readonly plugin config [puppet] - 10https://gerrit.wikimedia.org/r/1356233 (https://phabricator.wikimedia.org/T440177) (owner: 10Arnaudb) [08:13:23] (03CR) 10Arnaudb: [C:03+2] gerrit: allow `gerrit index` in readonly plugin [puppet] - 10https://gerrit.wikimedia.org/r/1358246 (https://phabricator.wikimedia.org/T440177) (owner: 10Hashar) [08:13:35] (03PS5) 10Hashar: gerrit: allow `gerrit index` in readonly plugin [puppet] - 10https://gerrit.wikimedia.org/r/1358246 (https://phabricator.wikimedia.org/T440177) [08:13:39] (03CR) 10Arnaudb: [C:03+2] gerrit: allow `gerrit index` in readonly plugin [puppet] - 10https://gerrit.wikimedia.org/r/1358246 (https://phabricator.wikimedia.org/T440177) (owner: 10Hashar) [08:14:36] (03CR) 10Arnaudb: [C:03+2] "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1358246 (https://phabricator.wikimedia.org/T440177) (owner: 10Hashar) [08:14:49] (03CR) 10Arnaudb: gerrit: allow `gerrit index` in readonly plugin [puppet] - 10https://gerrit.wikimedia.org/r/1358246 (https://phabricator.wikimedia.org/T440177) (owner: 10Hashar) [08:15:33] (03PS6) 10Arnaudb: gerrit: allow `gerrit index` in readonly plugin [puppet] - 10https://gerrit.wikimedia.org/r/1358246 (https://phabricator.wikimedia.org/T440177) (owner: 10Hashar) [08:16:54] (03CR) 10Arnaudb: [C:03+2] gerrit: allow `gerrit index` in readonly plugin [puppet] - 10https://gerrit.wikimedia.org/r/1358246 (https://phabricator.wikimedia.org/T440177) (owner: 10Hashar) [08:23:44] !log mlitn@deploy2003 mlitn: Backport for [[gerrit:1367552|Squashed diff to master]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:24:21] !log mlitn@deploy2003 mlitn: Continuing with deployment [08:38:40] chlod: I can also take care of TausheefHassan's patch if you want? [08:38:51] (almost there btw - its taking its sweet time) [08:39:03] up to you :) [08:39:22] i can wait [08:39:35] !log mlitn@deploy2003 Finished scap sync-world: Backport for [[gerrit:1367552|Squashed diff to master]] (1.48.0-wmf.3) (duration: 36m 37s) [08:39:45] Ah, might as well take care of another while I'm at it and free you up! [08:39:58] alrighty, thanks! [08:40:27] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367544 (https://phabricator.wikimedia.org/T444031) (owner: 10Tausheef Hassan) [08:40:46] TausheefHassan: starting! [08:41:04] matthiasmullie: thanks, ready to test! [08:41:11] (03Merged) 10jenkins-bot: Enable {{USERLANGUAGE}} on bdwikimedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367544 (https://phabricator.wikimedia.org/T444031) (owner: 10Tausheef Hassan) [08:41:28] !log mlitn@deploy2003 Started scap sync-world: Backport for [[gerrit:1367544|Enable {{USERLANGUAGE}} on bdwikimedia (T444031)]] [08:41:31] T444031: Enable USERLANGUAGE magic word for bd.wikimedia.org - https://phabricator.wikimedia.org/T444031 [08:43:37] (03PS1) 10Jgiannelos: thumbor: Make haproxy worker maxconn configurable [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367731 [08:47:39] !log mlitn@deploy2003 mlitn, tausheefhassan: Backport for [[gerrit:1367544|Enable {{USERLANGUAGE}} on bdwikimedia (T444031)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:48:09] TausheefHassan: ready to test - LMK when you're good to move forward! [08:49:13] matthiasmullie: works perfectly on mwdebug. good to go! thanks [08:49:20] !log mlitn@deploy2003 mlitn, tausheefhassan: Continuing with deployment [08:49:59] (03CR) 10Vgutierrez: [C:04-1] trafficserver: remove support for running multiple instances (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363267 (https://phabricator.wikimedia.org/T443596) (owner: 10Daniel Kertesz) [08:54:44] (03PS1) 10Jelto: Remove legacy key for ngkountas [puppet] - 10https://gerrit.wikimedia.org/r/1367735 (https://phabricator.wikimedia.org/T442762) [08:58:03] !log mlitn@deploy2003 Finished scap sync-world: Backport for [[gerrit:1367544|Enable {{USERLANGUAGE}} on bdwikimedia (T444031)]] (duration: 16m 35s) [08:58:09] TausheefHassan: done! [08:58:37] !log UTC morning backport window done [08:58:47] (03CR) 10Ladsgroup: [C:03+1] thumbor: Make haproxy worker maxconn configurable [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367731 (owner: 10Jgiannelos) [08:59:15] (03CR) 10Jelto: [C:03+2] Remove legacy key for ngkountas [puppet] - 10https://gerrit.wikimedia.org/r/1367735 (https://phabricator.wikimedia.org/T442762) (owner: 10Jelto) [08:59:19] matthiasmullie: confirmed working in production, thank you so much! [08:59:43] (03CR) 10Abijeet Patro: [C:03+1] ULS: Use the new language selector on Timeless on testwiki and mediawiki.org (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363037 (https://phabricator.wikimedia.org/T439715) (owner: 10Huei Tan) [09:01:30] (03CR) 10Jgiannelos: [C:03+2] thumbor: Make haproxy worker maxconn configurable [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367731 (owner: 10Jgiannelos) [09:01:39] FIRING: CoreBGPDown: Core BGP session down between cr1-magru and asw1-b3-magru (2a02:ec80:700:fe06::2) - group Switch - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Switch&var-bgp_neighbor=asw1-b3-magru - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [09:02:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [09:03:29] (03Merged) 10jenkins-bot: thumbor: Make haproxy worker maxconn configurable [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367731 (owner: 10Jgiannelos) [09:04:27] (03PS1) 10Jelto: Remove legacy key of andrew-wmde [puppet] - 10https://gerrit.wikimedia.org/r/1367736 (https://phabricator.wikimedia.org/T442763) [09:06:39] RESOLVED: CoreBGPDown: Core BGP session down between cr1-magru and asw1-b3-magru (2a02:ec80:700:fe06::2) - group Switch - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Switch&var-bgp_neighbor=asw1-b3-magru - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [09:06:42] (03CR) 10Jelto: [C:03+2] Remove legacy key of andrew-wmde [puppet] - 10https://gerrit.wikimedia.org/r/1367736 (https://phabricator.wikimedia.org/T442763) (owner: 10Jelto) [09:08:10] FIRING: BFDdown: BFD session down between cr2-magru and 195.200.68.160 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:10:06] !log jgiannelos@deploy2003 helmfile [staging] START helmfile.d/services/thumbor: apply [09:10:22] !log jgiannelos@deploy2003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [09:13:10] RESOLVED: BFDdown: BFD session down between cr2-magru and 195.200.68.160 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:14:18] (03PS1) 10Jgiannelos: thumbor: Disable async engine on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367739 [09:16:49] (03CR) 10Jgiannelos: [C:03+2] thumbor: Disable async engine on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367739 (owner: 10Jgiannelos) [09:18:24] (03Merged) 10jenkins-bot: thumbor: Disable async engine on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367739 (owner: 10Jgiannelos) [09:19:58] !log jgiannelos@deploy2003 helmfile [staging] START helmfile.d/services/thumbor: apply [09:20:17] !log jgiannelos@deploy2003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [09:21:51] !log jgiannelos@deploy2003 helmfile [staging] START helmfile.d/services/thumbor: apply [09:21:55] !log jgiannelos@deploy2003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [09:27:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [09:28:20] (03CR) 10Jelto: [C:03+1] "lgtm" [cookbooks] - 10https://gerrit.wikimedia.org/r/1366351 (https://phabricator.wikimedia.org/T437480) (owner: 10JMeybohm) [09:29:57] (03CR) 10Jelto: [C:03+1] "confirmed out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1367557 (owner: 10Gergő Tisza) [09:30:48] (03PS2) 10Jelto: admin: Add FIDO SSH keys for tgr [puppet] - 10https://gerrit.wikimedia.org/r/1367557 (https://phabricator.wikimedia.org/T442758) (owner: 10Gergő Tisza) [09:31:37] 10SRE-swift-storage, 10Thumbor: Gradually drop all thumbnails as a one-off clean up - https://phabricator.wikimedia.org/T379942#12532169 (10Ladsgroup) [09:33:15] (03CR) 10Jelto: [C:03+2] admin: Add FIDO SSH keys for tgr [puppet] - 10https://gerrit.wikimedia.org/r/1367557 (https://phabricator.wikimedia.org/T442758) (owner: 10Gergő Tisza) [09:35:36] 10SRE-swift-storage, 10Thumbor: Gradually drop all thumbnails as a one-off clean up - https://phabricator.wikimedia.org/T379942#12532189 (10Ladsgroup) 05Open→03Resolved The clean up is now officially done. We went from 2.5B thumbnails to 500M and from half a PB to 100TB now. We still need to do one mor... [09:35:59] (03PS1) 10Aqu: EventStreamConfig: Refine app_base as medium job [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367743 (https://phabricator.wikimedia.org/T444095) [09:45:10] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888#12532248 (10MatthewVernon) This recurred over the weekend. ` /dev/sdx4 186G 186G 141M 100% /srv/swift-storage/accounts0 ` And it's the same container database ` mvernon@thanos-be2008:~$ sudo ls -l /s... [09:51:24] (03PS2) 10Dpogorzelski: keda: import the upstream KEDA 2.21.0 chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364019 (https://phabricator.wikimedia.org/T436650) [09:51:24] (03PS3) 10Dpogorzelski: admin_ng: add KEDA, on ml-staging-codfw only [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364011 (https://phabricator.wikimedia.org/T436650) [09:51:24] (03PS1) 10Dpogorzelski: keda: add WMF changes to the upstream chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367745 (https://phabricator.wikimedia.org/T436650) [09:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.06% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [09:54:54] (03PS1) 10Jgiannelos: Revert "thumbor: Disable async engine on staging" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367746 [09:55:10] (03CR) 10Jgiannelos: [C:03+2] Revert "thumbor: Disable async engine on staging" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367746 (owner: 10Jgiannelos) [09:57:29] (03Merged) 10jenkins-bot: Revert "thumbor: Disable async engine on staging" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367746 (owner: 10Jgiannelos) [09:57:39] (03PS1) 10Ladsgroup: Keep temp files in a directory that only exists at runtime [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1364701 (https://phabricator.wikimedia.org/T431768) (owner: 10Ladsgroup-claude) [09:58:24] (03PS1) 10Bartosz Wójtowicz: admin: Replace SSH key for bwojtowicz with a FIDO key [puppet] - 10https://gerrit.wikimedia.org/r/1367747 (https://phabricator.wikimedia.org/T442758) [09:58:58] (03CR) 10Dpogorzelski: "Done" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364019 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [09:59:22] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888#12532390 (10MatthewVernon) Possibly also of note - we got cron mail over the weekend from some backends: `Exception during recon-cron while accessing devices: 10 seconds: /var/lock/swift-recon-object-cron/.... [10:00:50] !log jgiannelos@deploy2003 helmfile [staging] START helmfile.d/services/thumbor: apply [10:01:02] !log jgiannelos@deploy2003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [10:02:34] PROBLEM - OSPF status on cr2-magru is CRITICAL: OSPFv2: 3/3 UP : OSPFv3: 2/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:03:34] RECOVERY - OSPF status on cr2-magru is OK: OSPFv2: 3/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:05:10] FIRING: BFDdown: BFD session down between cr2-magru and fe80::b6f9:5d02:e530:c943 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:05:20] I'm going to do a private code deploy [10:05:36] if nobody objects in the next couple of minutes [10:10:10] RESOLVED: BFDdown: BFD session down between cr2-magru and fe80::b6f9:5d02:e530:c943 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:10:25] (03PS1) 10Tchanders: Take Special:AbuseReview vandalism queue out of alpha on frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367749 (https://phabricator.wikimedia.org/T443815) [10:12:02] (03CR) 10Kosta Harlan: [C:03+1] Take Special:AbuseReview vandalism queue out of alpha on frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367749 (https://phabricator.wikimedia.org/T443815) (owner: 10Tchanders) [10:12:18] (03PS2) 10Dpogorzelski: keda: add WMF changes to the upstream chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367745 (https://phabricator.wikimedia.org/T436650) [10:12:18] (03PS4) 10Dpogorzelski: admin_ng: add KEDA, on ml-staging-codfw only [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364011 (https://phabricator.wikimedia.org/T436650) [10:13:10] FIRING: BFDdown: BFD session down between cr2-magru and fe80::b6f9:5d02:e530:c943 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:14:24] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [10:15:40] (03PS3) 10Dpogorzelski: keda: add WMF changes to the upstream chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367745 (https://phabricator.wikimedia.org/T436650) [10:15:40] (03PS5) 10Dpogorzelski: admin_ng: add KEDA, on ml-staging-codfw only [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364011 (https://phabricator.wikimedia.org/T436650) [10:18:10] RESOLVED: [2x] BFDdown: BFD session down between cr2-magru and fe80::b6f9:5d02:e530:c943 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:18:28] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888#12532522 (10MatthewVernon) Just double-checking, the other quarantined containers on thanos-be2008 - `/srv/swift-storage/accounts0/quarantined/containers/4f0ba9617366341a8939cbe7634ba9e8/4f0ba9617366341a8939... [10:18:39] (03PS1) 10Gehel: admin: removed non FIDO SSH keys for pcoombe, ttaylor, rmaung, radimer [puppet] - 10https://gerrit.wikimedia.org/r/1367750 (https://phabricator.wikimedia.org/T442756) [10:21:31] (03CR) 10Klausman: [C:03+1] admin_ng: add KEDA, on ml-staging-codfw only [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364011 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [10:21:46] (03CR) 10Btullis: [C:03+1] admin: removed non FIDO SSH keys for pcoombe, ttaylor, rmaung, radimer [puppet] - 10https://gerrit.wikimedia.org/r/1367750 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [10:22:11] (03CR) 10Klausman: [C:03+1] admin: removed non FIDO SSH keys for pcoombe, ttaylor, rmaung, radimer [puppet] - 10https://gerrit.wikimedia.org/r/1367750 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [10:23:32] (03CR) 10Gehel: [C:03+2] admin: adding FIDO/SSH keys for sd [puppet] - 10https://gerrit.wikimedia.org/r/1365868 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [10:24:03] (03PS7) 10Krinkle: Remove unused "TitleBlacklist-cache" channel [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1354558 (https://phabricator.wikimedia.org/T85428) [10:24:11] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 09 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1354558 (https://phabricator.wikimedia.org/T85428) (owner: 10Krinkle) [10:24:42] (03CR) 10TrainBranchBot: [C:03+2] "Approved by tchanders@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367749 (https://phabricator.wikimedia.org/T443815) (owner: 10Tchanders) [10:24:47] (03PS1) 10Krinkle: tests: Cover updateCSS.php and updateLexerList.php maint scripts [extensions/SyntaxHighlight_GeSHi] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367751 [10:25:01] (03PS1) 10Krinkle: Always return bundled from getLexers, remove "fetch_lexers" shellbox call [extensions/SyntaxHighlight_GeSHi] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367752 (https://phabricator.wikimedia.org/T292663) [10:25:14] (03PS1) 10Krinkle: Always use bundled CSS, remove "generated_css" shellbox call [extensions/SyntaxHighlight_GeSHi] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367753 (https://phabricator.wikimedia.org/T292663) [10:25:28] (03PS1) 10Krinkle: Change getVersion() to use lockTSE and busyValue to avoid fatal error [extensions/SyntaxHighlight_GeSHi] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367754 (https://phabricator.wikimedia.org/T292663) [10:25:40] (03Merged) 10jenkins-bot: Take Special:AbuseReview vandalism queue out of alpha on frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367749 (https://phabricator.wikimedia.org/T443815) (owner: 10Tchanders) [10:25:44] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 09 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/SyntaxHighlight_GeSHi] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367752 (https://phabricator.wikimedia.org/T292663) (owner: 10Krinkle) [10:25:56] (03CR) 10Gehel: [C:03+2] admin: removed non FIDO SSH keys for pcoombe, ttaylor, rmaung, radimer [puppet] - 10https://gerrit.wikimedia.org/r/1367750 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [10:25:57] !log tchanders@deploy2003 Started scap sync-world: Backport for [[gerrit:1367749|Take Special:AbuseReview vandalism queue out of alpha on frwiki (T443815)]] [10:26:01] T443815: Open up the vandalism classifier on frwiki to rollbackers and above, and lower threshold slightly - https://phabricator.wikimedia.org/T443815 [10:26:18] (03PS1) 10Jelto: Add FIDO key for jakob [puppet] - 10https://gerrit.wikimedia.org/r/1367755 (https://phabricator.wikimedia.org/T442763) [10:31:52] (03CR) 10Muehlenhoff: [C:03+1] "Key has been verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1367755 (https://phabricator.wikimedia.org/T442763) (owner: 10Jelto) [10:32:15] !log tchanders@deploy2003 tchanders: Backport for [[gerrit:1367749|Take Special:AbuseReview vandalism queue out of alpha on frwiki (T443815)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [10:32:18] T443815: Open up the vandalism classifier on frwiki to rollbackers and above, and lower threshold slightly - https://phabricator.wikimedia.org/T443815 [10:33:23] (03CR) 10Jelto: [C:03+2] Add FIDO key for jakob [puppet] - 10https://gerrit.wikimedia.org/r/1367755 (https://phabricator.wikimedia.org/T442763) (owner: 10Jelto) [10:33:44] !log tchanders@deploy2003 tchanders: Continuing with deployment [10:41:27] !log tchanders@deploy2003 Finished scap sync-world: Backport for [[gerrit:1367749|Take Special:AbuseReview vandalism queue out of alpha on frwiki (T443815)]] (duration: 15m 29s) [10:41:30] T443815: Open up the vandalism classifier on frwiki to rollbackers and above, and lower threshold slightly - https://phabricator.wikimedia.org/T443815 [10:42:47] !log Updated WikimediaAntiAbuse AbuseReview private code [10:42:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:43:08] I'm finished with my private code changes [10:44:42] (03PS1) 10Jelto: Add FIDO key for sihe [puppet] - 10https://gerrit.wikimedia.org/r/1367761 (https://phabricator.wikimedia.org/T442763) [10:45:55] (03PS2) 10Urbanecm: GrowthExperiments: Remove settings that nothing reads [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363328 (https://phabricator.wikimedia.org/T443648) [10:45:59] (03CR) 10Urbanecm: [C:03+2] GrowthExperiments: Remove settings that nothing reads [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363328 (https://phabricator.wikimedia.org/T443648) (owner: 10Urbanecm) [10:46:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363328 (https://phabricator.wikimedia.org/T443648) (owner: 10Urbanecm) [10:46:43] (03Merged) 10jenkins-bot: GrowthExperiments: Remove settings that nothing reads [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363328 (https://phabricator.wikimedia.org/T443648) (owner: 10Urbanecm) [10:46:46] (03PS1) 10Ladsgroup: varnish: Increase rate limits on DASH files [puppet] - 10https://gerrit.wikimedia.org/r/1367759 (https://phabricator.wikimedia.org/T373546) (owner: 10Ladsgroup-claude) [10:46:46] (03CR) 10Ladsgroup: "I don't have an easy way to test the .vtc file. If you want to, feel free to remove it." [puppet] - 10https://gerrit.wikimedia.org/r/1367759 (https://phabricator.wikimedia.org/T373546) (owner: 10Ladsgroup-claude) [10:46:58] !log urbanecm@deploy2003 Started scap sync-world: Backport for [[gerrit:1363328|GrowthExperiments: Remove settings that nothing reads (T443648)]] [10:47:01] T443648: Remove easily removeable code from GrowthExperiments - https://phabricator.wikimedia.org/T443648 [10:47:01] (03CR) 10Muehlenhoff: [C:03+1] "Key has been verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1367761 (https://phabricator.wikimedia.org/T442763) (owner: 10Jelto) [10:47:06] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [10:47:46] (03CR) 10Jelto: [C:03+2] Add FIDO key for sihe [puppet] - 10https://gerrit.wikimedia.org/r/1367761 (https://phabricator.wikimedia.org/T442763) (owner: 10Jelto) [10:53:18] !log urbanecm@deploy2003 urbanecm: Backport for [[gerrit:1363328|GrowthExperiments: Remove settings that nothing reads (T443648)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [10:53:22] T443648: Remove easily removeable code from GrowthExperiments - https://phabricator.wikimedia.org/T443648 [10:53:56] !log urbanecm@deploy2003 urbanecm: Continuing with deployment [10:55:28] (03CR) 10Cparle: [C:03+1] varnish: Increase rate limits on DASH files [puppet] - 10https://gerrit.wikimedia.org/r/1367759 (https://phabricator.wikimedia.org/T373546) (owner: 10Ladsgroup-claude) [10:55:39] FIRING: CoreBGPDown: Core BGP session down between cr1-magru and asw1-b3-magru (2a02:ec80:700:fe06::2) - group Switch - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Switch&var-bgp_neighbor=asw1-b3-magru - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [10:57:10] FIRING: BFDdown: BFD session down between cr2-magru and fe80::b6f9:5d02:e530:c943 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:57:52] PROBLEM - OSPF status on cr1-magru is CRITICAL: OSPFv2: 4/5 UP : OSPFv3: 4/4 UP : 5 v2 P2P interfaces vs. 4 v3 P2P interfaces https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:58:38] !log btullis@cumin1004 conftool action : set/pooled=inactive; selector: name=dse-k8s-wdqs-test1001.eqiad.wmnet [10:58:40] RECOVERY - OSPF status on cr1-magru is OK: OSPFv2: 5/5 UP : OSPFv3: 5/5 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:58:47] !log btullis@cumin1004 conftool action : set/pooled=inactive; selector: name=dse-k8s-wdqs-test2001.codfw.wmnet [10:59:52] !log urbanecm@deploy2003 Finished scap sync-world: Backport for [[gerrit:1363328|GrowthExperiments: Remove settings that nothing reads (T443648)]] (duration: 12m 54s) [10:59:55] T443648: Remove easily removeable code from GrowthExperiments - https://phabricator.wikimedia.org/T443648 [11:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261109T1100) [11:00:39] RESOLVED: CoreBGPDown: Core BGP session down between cr1-magru and asw1-b3-magru (2a02:ec80:700:fe06::2) - group Switch - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Switch&var-bgp_neighbor=asw1-b3-magru - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [11:01:34] PROBLEM - OSPF status on cr2-magru is CRITICAL: OSPFv2: 3/3 UP : OSPFv3: 2/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [11:02:10] RESOLVED: BFDdown: BFD session down between cr2-magru and fe80::b6f9:5d02:e530:c943 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:03:10] FIRING: [2x] BFDdown: BFD session down between cr2-magru and 195.200.68.160 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:03:34] RECOVERY - OSPF status on cr2-magru is OK: OSPFv2: 3/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [11:03:39] FIRING: [3x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (2a02:ec80:700:fe06::1) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [11:04:14] (03PS1) 10Urbanecm: GrowthExperiments: Remove the $wgGECampaignTopics setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367767 (https://phabricator.wikimedia.org/T444124) [11:04:56] (03CR) 10Btullis: [C:03+2] dse_k8s: Remove the wdqs test nodes from the clusters [puppet] - 10https://gerrit.wikimedia.org/r/1364512 (https://phabricator.wikimedia.org/T443792) (owner: 10Btullis) [11:06:17] (03PS5) 10Gmodena: rest-gateway: set wdqs v2 GA rate limits (shadow mode). [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358194 (https://phabricator.wikimedia.org/T437892) [11:06:36] (03CR) 10Gmodena: rest-gateway: set wdqs v2 GA rate limits (shadow mode). (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358194 (https://phabricator.wikimedia.org/T437892) (owner: 10Gmodena) [11:06:48] PROBLEM - OSPF status on cr1-magru is CRITICAL: OSPFv2: 5/5 UP : OSPFv3: 4/4 UP : 5 v2 P2P interfaces vs. 4 v3 P2P interfaces https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [11:07:25] FIRING: [4x] BFDdown: BFD session down between cr1-magru and fe80::b6f9:5d02:e530:e143 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:07:42] RECOVERY - OSPF status on cr1-magru is OK: OSPFv2: 5/5 UP : OSPFv3: 5/5 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [11:08:10] FIRING: [5x] BFDdown: BFD session down between cr1-magru and fe80::b6f9:5d02:e530:e143 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:08:39] RESOLVED: [3x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (2a02:ec80:700:fe06::1) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [11:08:55] 10ops-magru: Outbound errors on interface cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://phabricator.wikimedia.org/T444127 (10phaultfinder) 03NEW [11:10:12] !log btullis@cumin1004 START - Cookbook sre.hosts.rename from dse-k8s-wdqs-test1001 to dse-k8s-worker1049 [11:10:23] !log btullis@cumin1004 START - Cookbook sre.hosts.rename from dse-k8s-wdqs-test2001 to dse-k8s-worker2012 [11:10:23] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [11:10:24] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [11:11:06] (03CR) 10Gmodena: rest-gateway: set wdqs v2 GA rate limits (shadow mode). (039 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358194 (https://phabricator.wikimedia.org/T437892) (owner: 10Gmodena) [11:12:25] RESOLVED: [4x] BFDdown: BFD session down between cr1-magru and fe80::b6f9:5d02:e530:e143 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:12:34] (03PS1) 10Btullis: admin_ng: Add a Spark namespace for each team on dse-k8s-eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367769 (https://phabricator.wikimedia.org/T432322) [11:12:34] PROBLEM - OSPF status on cr2-magru is CRITICAL: OSPFv2: 2/3 UP : OSPFv3: 2/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [11:12:36] (03PS1) 10Btullis: spark-operator: Watch the analytics and team Spark namespaces [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367770 (https://phabricator.wikimedia.org/T432322) [11:12:39] FIRING: [4x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (195.200.68.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [11:13:10] FIRING: [5x] BFDdown: BFD session down between cr1-magru and fe80::b6f9:5d02:e530:e143 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:13:28] (03PS1) 10Btullis: deployment_server: Restrict the spark namespaces to analytics-deployers [puppet] - 10https://gerrit.wikimedia.org/r/1367771 (https://phabricator.wikimedia.org/T429558) [11:13:28] (03PS1) 10Btullis: deployment_server: Add the kubeconfigs of the team Spark namespaces [puppet] - 10https://gerrit.wikimedia.org/r/1367772 (https://phabricator.wikimedia.org/T432322) [11:13:31] (03PS1) 10Btullis: ceph/admin/eqiad: Add an account for each Spark namespace [puppet] - 10https://gerrit.wikimedia.org/r/1367773 (https://phabricator.wikimedia.org/T435470) [11:13:33] (03PS1) 10Btullis: ceph/admin/codfw: Fix the trust policy of the analytics writer role [puppet] - 10https://gerrit.wikimedia.org/r/1367774 (https://phabricator.wikimedia.org/T435470) [11:15:09] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888#12532873 (10MatthewVernon) Interestingly (and reminding me of T377827), if I VACUUM the quarantined-but-good db on thanos-be2008, it takes about 5 minutes, but the result is 18G, quite a substantial reductio... [11:15:24] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [11:15:51] btullis@cumin1004 rename (PID 1498360) is awaiting input [11:16:15] (03CR) 10Trueg: [C:03+1] wdqs: version bump health container 0.0.2 -> 0.0.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360943 (https://phabricator.wikimedia.org/T442744) (owner: 10Gmodena) [11:16:15] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [11:16:27] (03CR) 10Trueg: [C:03+1] wdqs: version bump proxy 0.14.2 -> 0.14.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360941 (https://phabricator.wikimedia.org/T442744) (owner: 10Gmodena) [11:16:49] (03CR) 10Trueg: [C:03+1] wdqs: version bump streaming consumer 0.3.1 -> 0.4.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360944 (https://phabricator.wikimedia.org/T442744) (owner: 10Gmodena) [11:17:25] RESOLVED: [5x] BFDdown: BFD session down between cr1-magru and fe80::b6f9:5d02:e530:e143 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:17:39] FIRING: [4x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (195.200.68.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [11:18:10] FIRING: [6x] BFDdown: BFD session down between cr1-magru and fe80::b6f9:5d02:e530:e143 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:18:54] FIRING: [4x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (195.200.68.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [11:20:24] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [11:21:10] !log btullis@cumin1004 END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [11:21:34] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [11:21:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [11:22:21] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming dse-k8s-wdqs-test1001 to dse-k8s-worker1049 - btullis@cumin1004" [11:22:25] FIRING: [6x] BFDdown: BFD session down between cr1-magru and fe80::b6f9:5d02:e530:e143 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:22:39] FIRING: [4x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (195.200.68.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [11:23:10] RESOLVED: [6x] BFDdown: BFD session down between cr1-magru and fe80::b6f9:5d02:e530:e143 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:24:27] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming dse-k8s-wdqs-test1001 to dse-k8s-worker1049 - btullis@cumin1004" [11:24:27] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [11:24:27] !log btullis@cumin1004 START - Cookbook sre.dns.wipe-cache dse-k8s-worker1049 on all recursors [11:24:29] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1049 on all recursors [11:24:30] !log btullis@cumin1004 START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1049 [11:24:56] !log btullis@cumin1004 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1049 [11:25:29] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming dse-k8s-wdqs-test2001 to dse-k8s-worker2012 - btullis@cumin1004" [11:25:32] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from dse-k8s-wdqs-test1001 to dse-k8s-worker1049 [11:25:32] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming dse-k8s-wdqs-test2001 to dse-k8s-worker2012 - btullis@cumin1004" [11:25:33] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [11:25:33] !log btullis@cumin1004 START - Cookbook sre.dns.wipe-cache dse-k8s-worker2012 on all recursors [11:25:35] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2012 on all recursors [11:25:36] !log btullis@cumin1004 START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2012 [11:26:21] (03PS1) 10Krinkle: RawAction: Improve test coverage [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367785 [11:26:24] !log btullis@cumin1004 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2012 [11:26:32] !log btullis@cumin1004 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dse-k8s-worker1049.eqiad.wmnet'] [11:26:39] (03PS1) 10Krinkle: RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367786 (https://phabricator.wikimedia.org/T279120) [11:26:48] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 09 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367786 (https://phabricator.wikimedia.org/T279120) (owner: 10Krinkle) [11:26:51] RESOLVED: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [11:27:00] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from dse-k8s-wdqs-test2001 to dse-k8s-worker2012 [11:28:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [11:30:40] FIRING: [4x] BFDdown: BFD session down between cr2-magru and 195.200.68.160 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:30:58] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host dse-k8s-worker2012.codfw.wmnet with OS bookworm [11:31:06] !log btullis@cumin1004 START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2012.codfw.wmnet (push config only) [11:31:11] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2012.codfw.wmnet (push config only) [11:33:31] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:33:45] !log btullis@cumin1004 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['dse-k8s-worker1049.eqiad.wmnet'] [11:33:51] RESOLVED: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [11:34:39] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host dse-k8s-worker1049.eqiad.wmnet with OS bookworm [11:34:48] !log btullis@cumin1004 START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1049.eqiad.wmnet (push config only) [11:34:51] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1049.eqiad.wmnet (push config only) [11:35:40] RESOLVED: [3x] BFDdown: BFD session down between cr2-magru and 195.200.68.160 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:38:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [11:41:00] (03CR) 10Clément Goubert: [C:03+1] service::node: remove statsd configuration [puppet] - 10https://gerrit.wikimedia.org/r/1359819 (https://phabricator.wikimedia.org/T435340) (owner: 10Hnowlan) [11:45:41] (03PS1) 10Jgiannelos: Build image with async support in thumbor [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1367794 [11:45:59] !log btullis@cumin1004 START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2012.codfw.wmnet (remove config only) [11:46:04] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2012.codfw.wmnet (remove config only) [11:47:12] (03CR) 10Jgiannelos: "Turns out, I built an image using the forked thumbor for testing on staging BUT never built an image with async support in that fork. No w" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1367794 (owner: 10Jgiannelos) [11:48:00] FIRING: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker1049 has a BGP session which is not in the 'established' state. [11:48:38] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2012.codfw.wmnet with reason: host reimage [11:48:40] (03CR) 10Jgiannelos: [C:04-1] "Blocking this while i verify things locally." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1367794 (owner: 10Jgiannelos) [11:48:51] RESOLVED: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [11:49:44] (03Abandoned) 10Hashar: build: Upgrade eslint-config-wikimedia from 0.23.0 to 0.32.3 and make pass [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1249368 (owner: 10Jforrester) [11:50:24] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [11:51:10] FIRING: BFDdown: BFD session down between cr2-magru and 195.200.68.160 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:51:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [11:52:35] !log btullis@cumin1004 START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1049.eqiad.wmnet (remove config only) [11:52:40] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [11:52:40] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1049.eqiad.wmnet (remove config only) [11:53:54] (03CR) 10Clément Goubert: [C:03+2] eventschemas: allow CORS requests from any origin [puppet] - 10https://gerrit.wikimedia.org/r/1354903 (https://phabricator.wikimedia.org/T438209) (owner: 10Aghirelli) [11:54:00] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2012.codfw.wmnet with reason: host reimage [11:54:08] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888#12533007 (10MatthewVernon) So, the handoff copies of this container are (per `swift-get-nodes /etc/swift/container.ring.gz AUTH_tegola tegola-swift-eqiad-v003`): ` thanos-be2007 accounts1/containers/20235/9e... [11:55:01] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1049.eqiad.wmnet with reason: host reimage [11:56:10] RESOLVED: BFDdown: BFD session down between cr2-magru and 195.200.68.160 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:58:49] (03CR) 10Vgutierrez: [C:03+1] trafficserver::backend: Remove apis XWD routing [puppet] - 10https://gerrit.wikimedia.org/r/1358726 (https://phabricator.wikimedia.org/T428909) (owner: 10Clément Goubert) [11:59:12] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1049.eqiad.wmnet with reason: host reimage [11:59:15] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888#12533034 (10MatthewVernon) I stopped the container replicator on thanos-be2006, restarted rsync and swift-* on thanos-be2008 and thanos-be1005. ` Nov 09 11:39:32 thanos-be2006 container-replicator[3094326]:... [12:00:04] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [12:00:14] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [12:00:19] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [12:00:28] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [12:00:31] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [12:00:38] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [12:01:50] (03CR) 10Clément Goubert: [C:03+2] trafficserver::backend: Remove apis XWD routing [puppet] - 10https://gerrit.wikimedia.org/r/1358726 (https://phabricator.wikimedia.org/T428909) (owner: 10Clément Goubert) [12:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:05:23] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:05:53] (03PS1) 10Tchanders: Take Special:AbuseReview vandalism queue out of alpha on eswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367806 (https://phabricator.wikimedia.org/T443052) [12:08:15] !log stop swift on thanos-be2006 to VACUUM 27G container DB T443888 [12:08:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:08:19] T443888: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888 [12:11:17] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2012.codfw.wmnet with OS bookworm [12:16:51] RESOLVED: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [12:17:17] !log VACUUM 27G container database on thanos-be2008 T443888 [12:17:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:17:20] T443888: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888 [12:19:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [12:19:57] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1049.eqiad.wmnet with OS bookworm [12:28:30] (03PS1) 10Hnowlan: ircecho: add basic throttling to messaging [puppet] - 10https://gerrit.wikimedia.org/r/1367813 (https://phabricator.wikimedia.org/T442719) [12:29:28] (03CR) 10CI reject: [V:04-1] ircecho: add basic throttling to messaging [puppet] - 10https://gerrit.wikimedia.org/r/1367813 (https://phabricator.wikimedia.org/T442719) (owner: 10Hnowlan) [12:36:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [12:41:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [12:41:32] (03PS1) 10Sergio Gimeno: admin: add FIDO ssh key for sgimeno [puppet] - 10https://gerrit.wikimedia.org/r/1367818 (https://phabricator.wikimedia.org/T442758) [12:43:47] (03PS2) 10Hnowlan: ircecho: add basic throttling to messaging [puppet] - 10https://gerrit.wikimedia.org/r/1367813 (https://phabricator.wikimedia.org/T442719) [12:44:51] RESOLVED: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [12:46:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [12:49:01] (03PS4) 10Hashar: cinderutils: honor mount_options when mountpoints exist [puppet] - 10https://gerrit.wikimedia.org/r/1343081 [12:50:03] (03PS4) 10Michael Große: Growth: enable home beta on pilot and innovation wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361861 (https://phabricator.wikimedia.org/T443176) [12:50:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [12:50:29] (03CR) 10CI reject: [V:04-1] cinderutils: honor mount_options when mountpoints exist [puppet] - 10https://gerrit.wikimedia.org/r/1343081 (owner: 10Hashar) [12:54:15] (03PS2) 10Btullis: dse_k8s: Add the two former wdqs test hosts to the clusters [puppet] - 10https://gerrit.wikimedia.org/r/1364513 (https://phabricator.wikimedia.org/T443792) [12:57:38] (03CR) 10Jelto: [C:03+1] "verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1367818 (https://phabricator.wikimedia.org/T442758) (owner: 10Sergio Gimeno) [13:00:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [13:00:29] (03CR) 10Btullis: [C:03+2] dse_k8s: Add the two former wdqs test hosts to the clusters [puppet] - 10https://gerrit.wikimedia.org/r/1364513 (https://phabricator.wikimedia.org/T443792) (owner: 10Btullis) [13:05:31] (03PS6) 10Hashar: cinderutils: support appending mount options [puppet] - 10https://gerrit.wikimedia.org/r/1343082 [13:06:45] (03PS3) 10Hashar: ci: set noatime on castor /srv partition [puppet] - 10https://gerrit.wikimedia.org/r/1343083 [13:07:44] (03PS1) 10Esanders: Add FIDO SSH keys for esanders [puppet] - 10https://gerrit.wikimedia.org/r/1367822 (https://phabricator.wikimedia.org/T442758) [13:08:28] (03CR) 10CI reject: [V:04-1] cinderutils: support appending mount options [puppet] - 10https://gerrit.wikimedia.org/r/1343082 (owner: 10Hashar) [13:11:51] RESOLVED: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [13:12:54] (03CR) 10Jelto: [C:03+2] admin: add FIDO ssh key for sgimeno [puppet] - 10https://gerrit.wikimedia.org/r/1367818 (https://phabricator.wikimedia.org/T442758) (owner: 10Sergio Gimeno) [13:15:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [13:17:07] (03CR) 10Jgiannelos: Build image with async support in thumbor [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1367794 (owner: 10Jgiannelos) [13:19:12] (03CR) 10Jgiannelos: [C:03+2] "This is only for staging testing and its going to be reverted either way, so self +2'ing." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1367794 (owner: 10Jgiannelos) [13:22:16] (03CR) 10Jelto: [C:03+1] "verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1367747 (https://phabricator.wikimedia.org/T442758) (owner: 10Bartosz Wójtowicz) [13:26:53] 10ops-magru: Outbound errors on interface cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://phabricator.wikimedia.org/T444127#12533464 (10ayounsi) p:05Triage→03High a:03RobH This looks like {T436675} happening again. After flapping, link has been down since 2026-11-09 12:41:43 UTC interface is... [13:30:08] (03PS1) 10Gehel: admin: removed non FIDO SSH keys for dsaez [puppet] - 10https://gerrit.wikimedia.org/r/1367827 (https://phabricator.wikimedia.org/T442756) [13:30:38] (03CR) 10Jelto: [C:03+2] "The key is replaced instead of a new key added and removed. But I confirmed with @bwojtowicz@wikimedia.org that 30 minutes of migration ti" [puppet] - 10https://gerrit.wikimedia.org/r/1367747 (https://phabricator.wikimedia.org/T442758) (owner: 10Bartosz Wójtowicz) [13:32:39] (03Merged) 10jenkins-bot: Build image with async support in thumbor [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1367794 (owner: 10Jgiannelos) [13:33:20] (03CR) 10Klausman: [C:03+1] admin: removed non FIDO SSH keys for dsaez [puppet] - 10https://gerrit.wikimedia.org/r/1367827 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [13:39:54] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1367800 (owner: 10L10n-bot) [13:44:27] (03PS1) 10Jelto: Remove legacy key of sihe and hoo. [puppet] - 10https://gerrit.wikimedia.org/r/1367831 (https://phabricator.wikimedia.org/T442763) [13:45:51] RESOLVED: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [13:48:46] (03PS1) 10Matthias Mullie: [DNM] ICE: Disable Special:SemanticSearch entirely [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367832 (https://phabricator.wikimedia.org/T437396) [13:49:08] (03CR) 10Matthias Mullie: [C:04-2] "DNM unless in case of emergency" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367832 (https://phabricator.wikimedia.org/T437396) (owner: 10Matthias Mullie) [13:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.09% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [13:52:46] (03CR) 10Jelto: [C:03+2] Remove legacy key of sihe and hoo. [puppet] - 10https://gerrit.wikimedia.org/r/1367831 (https://phabricator.wikimedia.org/T442763) (owner: 10Jelto) [13:52:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [13:56:03] (03PS1) 10Hashar: ci: ci-build-images: add bullseye-security [puppet] - 10https://gerrit.wikimedia.org/r/1367835 [13:57:33] (03PS2) 10Hashar: ci: ci-build-images: add bullseye-security [puppet] - 10https://gerrit.wikimedia.org/r/1367835 [13:59:35] (03PS2) 10Vgutierrez: varnish: Increase rate limits on DASH files [puppet] - 10https://gerrit.wikimedia.org/r/1367759 (https://phabricator.wikimedia.org/T373546) (owner: 10Ladsgroup-claude) [14:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for UTC afternoon backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261109T1400). [14:00:05] jan_drewniak and Krinkle: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [14:00:22] (03PS1) 10Filippo Giunchedi: team-wmcs: decrease BackupCoverageDrop threshold to 80% [alerts] - 10https://gerrit.wikimedia.org/r/1367838 (https://phabricator.wikimedia.org/T443945) [14:00:31] o/ [14:01:07] FIRING: [2x] KubernetesCalicoDown: dse-k8s-worker1049.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:01:09] (03CR) 10Jdrewniak: Enable ReadingLists custom lists on all Wikipedias. (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364782 (https://phabricator.wikimedia.org/T441372) (owner: 10Jdrewniak) [14:01:19] o/ [14:01:21] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2012.codfw.wmnet [14:01:28] jan_drewniak, Krinkle: I guess you can both self-service? [14:01:30] o// [14:01:31] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1049.eqiad.wmnet [14:02:04] (03CR) 10Vgutierrez: [C:03+1] "VTCs are happy: `0 tests failed, 0 tests skipped, 22 tests passed`" [puppet] - 10https://gerrit.wikimedia.org/r/1367759 (https://phabricator.wikimedia.org/T373546) (owner: 10Ladsgroup-claude) [14:03:15] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdrewniak@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364782 (https://phabricator.wikimedia.org/T441372) (owner: 10Jdrewniak) [14:04:04] o/ [14:04:19] (03Merged) 10jenkins-bot: Enable ReadingLists custom lists on all Wikipedias. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364782 (https://phabricator.wikimedia.org/T441372) (owner: 10Jdrewniak) [14:04:35] !log jdrewniak@deploy2003 Started scap sync-world: Backport for [[gerrit:1364782|Enable ReadingLists custom lists on all Wikipedias. (T441372)]] [14:04:38] T441372: Roll out custom lists to projects - https://phabricator.wikimedia.org/T441372 [14:05:03] (03PS1) 10Filippo Giunchedi: wmcs: log backup deletion only when needed [puppet] - 10https://gerrit.wikimedia.org/r/1367840 (https://phabricator.wikimedia.org/T443945) [14:05:04] Lucas_WMDE: yes, I'll self-servie thank you [14:05:24] ok, jan_drewniak please ping Krinkle when done :) [14:06:11] (03CR) 10Ssingh: [C:03+1] "Looks good -- I haven't verified the path but I trust you both to have done that, and the regex looks good, even though we can tighten it " [puppet] - 10https://gerrit.wikimedia.org/r/1367759 (https://phabricator.wikimedia.org/T373546) (owner: 10Ladsgroup-claude) [14:07:28] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1049.eqiad.wmnet [14:08:28] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2012.codfw.wmnet [14:10:05] (03CR) 10Vgutierrez: [C:03+1] "yep, I've tested the paths in production using https://test.wikipedia.org/w/index.php?title=User:Ladsgroup/sandbox&oldid=782822" [puppet] - 10https://gerrit.wikimedia.org/r/1367759 (https://phabricator.wikimedia.org/T373546) (owner: 10Ladsgroup-claude) [14:11:00] !log jdrewniak@deploy2003 jdrewniak: Backport for [[gerrit:1364782|Enable ReadingLists custom lists on all Wikipedias. (T441372)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:11:03] T441372: Roll out custom lists to projects - https://phabricator.wikimedia.org/T441372 [14:11:33] (03PS3) 10Hashar: ci: ci-build-images: add bullseye-security [puppet] - 10https://gerrit.wikimedia.org/r/1367835 [14:11:54] (03PS1) 10Lucian: Icinga: added myself to the following groups, as part of on-boarding: [puppet] - 10https://gerrit.wikimedia.org/r/1367843 [14:14:24] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [14:16:50] !log jdrewniak@deploy2003 jdrewniak: Continuing with deployment [14:17:25] (03PS4) 10Hashar: ci: ci-build-images: use Bullseye snapshot apt repo [puppet] - 10https://gerrit.wikimedia.org/r/1367835 (https://phabricator.wikimedia.org/T443872) [14:17:58] RESOLVED: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [14:20:45] Hi Lucas_WMDE ! I have some things to deploy when backport window is done. Could you let me know when we are all clear? TY! :) [14:20:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [14:21:10] Krinkle: can you ping ottomata when you’re done? ^^ [14:21:19] ottomata: ETA 30min [14:22:54] !log jdrewniak@deploy2003 Finished scap sync-world: Backport for [[gerrit:1364782|Enable ReadingLists custom lists on all Wikipedias. (T441372)]] (duration: 18m 18s) [14:22:57] T441372: Roll out custom lists to projects - https://phabricator.wikimedia.org/T441372 [14:23:36] (03CR) 10Krinkle: [C:03+2] RawAction: Improve test coverage [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367785 (owner: 10Krinkle) [14:23:39] (03CR) 10Krinkle: [C:03+2] RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367786 (https://phabricator.wikimedia.org/T279120) (owner: 10Krinkle) [14:23:51] (03CR) 10Krinkle: [C:03+2] tests: Cover updateCSS.php and updateLexerList.php maint scripts [extensions/SyntaxHighlight_GeSHi] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367751 (owner: 10Krinkle) [14:23:54] (03CR) 10Krinkle: [C:03+2] Always return bundled from getLexers, remove "fetch_lexers" shellbox call [extensions/SyntaxHighlight_GeSHi] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367752 (https://phabricator.wikimedia.org/T292663) (owner: 10Krinkle) [14:23:58] (03CR) 10Krinkle: [C:03+2] Always use bundled CSS, remove "generated_css" shellbox call [extensions/SyntaxHighlight_GeSHi] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367753 (https://phabricator.wikimedia.org/T292663) (owner: 10Krinkle) [14:24:01] (03CR) 10Krinkle: [C:03+2] Change getVersion() to use lockTSE and busyValue to avoid fatal error [extensions/SyntaxHighlight_GeSHi] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367754 (https://phabricator.wikimedia.org/T292663) (owner: 10Krinkle) [14:26:27] Krinkle: TY ! [14:26:47] (03PS3) 10Ottomata: Prefetch edit_suggestions_counts with EventBus LinkedArtifacts [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362093 (https://phabricator.wikimedia.org/T443442) [14:28:03] (03Merged) 10jenkins-bot: RawAction: Improve test coverage [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367785 (owner: 10Krinkle) [14:28:09] (03Merged) 10jenkins-bot: RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367786 (https://phabricator.wikimedia.org/T279120) (owner: 10Krinkle) [14:28:13] (03Merged) 10jenkins-bot: tests: Cover updateCSS.php and updateLexerList.php maint scripts [extensions/SyntaxHighlight_GeSHi] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367751 (owner: 10Krinkle) [14:28:13] (03Merged) 10jenkins-bot: Always return bundled from getLexers, remove "fetch_lexers" shellbox call [extensions/SyntaxHighlight_GeSHi] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367752 (https://phabricator.wikimedia.org/T292663) (owner: 10Krinkle) [14:28:15] (03Merged) 10jenkins-bot: Always use bundled CSS, remove "generated_css" shellbox call [extensions/SyntaxHighlight_GeSHi] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367753 (https://phabricator.wikimedia.org/T292663) (owner: 10Krinkle) [14:28:33] (03PS1) 10Jelto: Remove legacy key for sgimeno [puppet] - 10https://gerrit.wikimedia.org/r/1367847 (https://phabricator.wikimedia.org/T442758) [14:29:33] (03CR) 10TChin: [C:03+1] EventStreamConfig: Refine app_base as medium job [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367743 (https://phabricator.wikimedia.org/T444095) (owner: 10Aqu) [14:30:33] (03Merged) 10jenkins-bot: Change getVersion() to use lockTSE and busyValue to avoid fatal error [extensions/SyntaxHighlight_GeSHi] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367754 (https://phabricator.wikimedia.org/T292663) (owner: 10Krinkle) [14:30:34] !log krinkle@deploy2003 Started scap sync-world: Backport for [[gerrit:1367786|RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs (T279120)]] (1.48.0-wmf.3), [[gerrit:1367752|Always return bundled from getLexers, remove "fetch_lexers" shellbox call (T292663)]] (1.48.0-wmf.3) [14:30:39] T279120: Make index.php?action=raw CDN cacheable for both logged-in and logged-out for import script use case - https://phabricator.wikimedia.org/T279120 [14:30:39] T292663: Pygments intermittenly fatals load.php, pageviews, and edits (Shellbox server returned status code 503) - https://phabricator.wikimedia.org/T292663 [14:34:36] !log krinkle@deploy2003 krinkle: Backport for [[gerrit:1367786|RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs (T279120)]] (1.48.0-wmf.3), [[gerrit:1367752|Always return bundled from getLexers, remove "fetch_lexers" shellbox call (T292663)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:34:43] (03CR) 10Jelto: [C:03+2] Remove legacy key for sgimeno [puppet] - 10https://gerrit.wikimedia.org/r/1367847 (https://phabricator.wikimedia.org/T442758) (owner: 10Jelto) [14:35:24] (03CR) 10Gmodena: [C:03+2] wdqs: version bump proxy 0.14.2 -> 0.14.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360941 (https://phabricator.wikimedia.org/T442744) (owner: 10Gmodena) [14:35:31] (03CR) 10Gmodena: [C:03+2] wdqs: version bump health container 0.0.2 -> 0.0.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360943 (https://phabricator.wikimedia.org/T442744) (owner: 10Gmodena) [14:35:37] (03CR) 10Gmodena: [C:03+2] wdqs: version bump streaming consumer 0.3.1 -> 0.4.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360944 (https://phabricator.wikimedia.org/T442744) (owner: 10Gmodena) [14:37:01] (03Merged) 10jenkins-bot: wdqs: version bump proxy 0.14.2 -> 0.14.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360941 (https://phabricator.wikimedia.org/T442744) (owner: 10Gmodena) [14:37:31] (03Merged) 10jenkins-bot: wdqs: version bump health container 0.0.2 -> 0.0.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360943 (https://phabricator.wikimedia.org/T442744) (owner: 10Gmodena) [14:37:32] (03Merged) 10jenkins-bot: wdqs: version bump streaming consumer 0.3.1 -> 0.4.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360944 (https://phabricator.wikimedia.org/T442744) (owner: 10Gmodena) [14:38:26] !log krinkle@deploy2003 krinkle: Continuing with deployment [14:38:46] !log btullis@puppetserver1001 conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2012.codfw.wmnet [14:38:47] (03CR) 10Hashar: "So last Friday I went to try to switch the VM image from Bullseye to Trixie (T443872) and eventually lost patience with a change in the De" [puppet] - 10https://gerrit.wikimedia.org/r/1367835 (https://phabricator.wikimedia.org/T443872) (owner: 10Hashar) [14:38:53] !log btullis@puppetserver1001 conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2012.codfw.wmnet [14:38:57] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888#12533972 (10MatthewVernon) So much for verbose logging ` mvernon@thanos-be2006:~$ sudo -u swift /usr/bin/swift-container-replicator /etc/swift/container-server.conf -v -o container-replicator: Starting 32214... [14:39:12] !log btullis@puppetserver1001 conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1049.eqiad.wmnet [14:39:20] !log btullis@puppetserver1001 conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1049.eqiad.wmnet [14:40:55] (03CR) 10Ladsgroup: [C:03+1] varnish: Increase rate limits on DASH files [puppet] - 10https://gerrit.wikimedia.org/r/1367759 (https://phabricator.wikimedia.org/T373546) (owner: 10Ladsgroup-claude) [14:41:07] FIRING: [2x] KubernetesCalicoDown: dse-k8s-worker1049.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:41:24] (03PS1) 10Jelto: Remove legacy key for tgr [puppet] - 10https://gerrit.wikimedia.org/r/1367850 (https://phabricator.wikimedia.org/T442758) [14:42:40] (03PS1) 10Marco Fossati: admin: Add FIDO SSH key for mfossati [puppet] - 10https://gerrit.wikimedia.org/r/1367851 (https://phabricator.wikimedia.org/T442758) [14:43:00] RESOLVED: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker1049 has a BGP session which is not in the 'established' state. [14:44:13] (03CR) 10Jelto: [C:03+2] Remove legacy key for tgr [puppet] - 10https://gerrit.wikimedia.org/r/1367850 (https://phabricator.wikimedia.org/T442758) (owner: 10Jelto) [14:44:25] !log krinkle@deploy2003 Finished scap sync-world: Backport for [[gerrit:1367786|RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs (T279120)]] (1.48.0-wmf.3), [[gerrit:1367752|Always return bundled from getLexers, remove "fetch_lexers" shellbox call (T292663)]] (1.48.0-wmf.3) (duration: 13m 51s) [14:44:30] T279120: Make index.php?action=raw CDN cacheable for both logged-in and logged-out for import script use case - https://phabricator.wikimedia.org/T279120 [14:44:30] T292663: Pygments intermittenly fatals load.php, pageviews, and edits (Shellbox server returned status code 503) - https://phabricator.wikimedia.org/T292663 [14:45:51] RESOLVED: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [14:46:26] !log krinkle@deploy2003 Started scap sync-world: Backport for [[gerrit:1367786|RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs (T279120)]] (1.48.0-wmf.3) [14:47:03] FIRING: ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:47:06] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [14:47:07] !log krinkle@deploy2003 sync-world aborted: Backport for [[gerrit:1367786|RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs (T279120)]] (1.48.0-wmf.3) (duration: 00m 41s) [14:47:14] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1354558 (https://phabricator.wikimedia.org/T85428) (owner: 10Krinkle) [14:48:04] (03Merged) 10jenkins-bot: Remove unused "TitleBlacklist-cache" channel [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1354558 (https://phabricator.wikimedia.org/T85428) (owner: 10Krinkle) [14:48:09] (03PS1) 10Jelto: Add FIDO key for emc-wmf [puppet] - 10https://gerrit.wikimedia.org/r/1367854 (https://phabricator.wikimedia.org/T442758) [14:48:19] !log krinkle@deploy2003 Started scap sync-world: Backport for [[gerrit:1354558|Remove unused "TitleBlacklist-cache" channel (T85428 T441284)]] [14:48:23] T85428: Title blacklist intermittently failing, allowing users to edit things they shouldn't be able to - https://phabricator.wikimedia.org/T85428 [14:48:24] T441284: TitleBlacklist generates a lot of logs in production - https://phabricator.wikimedia.org/T441284 [14:48:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [14:50:32] (03CR) 10Jelto: [C:03+2] "verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1367854 (https://phabricator.wikimedia.org/T442758) (owner: 10Jelto) [14:51:06] (03CR) 10Btullis: [C:03+1] dse-k8s-eqiad: Add ATS mapping for matomo-next [puppet] - 10https://gerrit.wikimedia.org/r/1364735 (https://phabricator.wikimedia.org/T436186) (owner: 10Bking) [14:51:42] (03PS1) 10Sergio Gimeno: GrowthExperiments: Scale early onboarding on ar/en/es/frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367855 [14:52:03] RESOLVED: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:52:11] (03PS2) 10Sergio Gimeno: GrowthExperiments: Scale early onboarding on ar/en/es/frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367855 [14:52:22] !log krinkle@deploy2003 krinkle: Backport for [[gerrit:1354558|Remove unused "TitleBlacklist-cache" channel (T85428 T441284)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:53:01] (03CR) 10Sergio Gimeno: [C:04-1] "Not yet, requires coordination with experimentation platform to turn off the experiment in a safe way that ensures no experiment data poll" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367855 (owner: 10Sergio Gimeno) [14:53:12] (03PS1) 10Muehlenhoff: Convert antonkokhwmde from SSH to LDAP access [puppet] - 10https://gerrit.wikimedia.org/r/1367856 (https://phabricator.wikimedia.org/T442763) [14:53:40] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888#12534088 (10MatthewVernon) Looking through `swift/common/db_replicator.py`: `lang=python elif response.status == HTTP_INSUFFICIENT_STORAGE: raise DriveNotMounted() ` Which explains some o... [14:53:43] !log krinkle@deploy2003 krinkle: Continuing with deployment [14:57:18] (03CR) 10Vgutierrez: [C:03+2] varnish: Increase rate limits on DASH files [puppet] - 10https://gerrit.wikimedia.org/r/1367759 (https://phabricator.wikimedia.org/T373546) (owner: 10Ladsgroup-claude) [14:58:30] (03CR) 10Jelto: [C:03+1] "confirmed out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1367851 (https://phabricator.wikimedia.org/T442758) (owner: 10Marco Fossati) [14:59:18] (03CR) 10Bking: [C:03+2] dse-k8s-eqiad: Add ATS mapping for matomo-next [puppet] - 10https://gerrit.wikimedia.org/r/1364735 (https://phabricator.wikimedia.org/T436186) (owner: 10Bking) [14:59:31] (03CR) 10Jelto: [C:03+2] admin: Add FIDO SSH key for mfossati [puppet] - 10https://gerrit.wikimedia.org/r/1367851 (https://phabricator.wikimedia.org/T442758) (owner: 10Marco Fossati) [14:59:39] !log krinkle@deploy2003 Finished scap sync-world: Backport for [[gerrit:1354558|Remove unused "TitleBlacklist-cache" channel (T85428 T441284)]] (duration: 11m 20s) [14:59:43] T85428: Title blacklist intermittently failing, allowing users to edit things they shouldn't be able to - https://phabricator.wikimedia.org/T85428 [14:59:43] ottomata: done [14:59:44] T441284: TitleBlacklist generates a lot of logs in production - https://phabricator.wikimedia.org/T441284 [15:00:58] inflatador : can I also merge your change? dse-k8s-eqiad: Add ATS mapping for matomo-next (8936b62c76) [15:02:03] jelto sure! Thank you [15:04:19] (03CR) 10BPirkle: [C:03+1] "Looks good to me per discussion on the change" [puppet] - 10https://gerrit.wikimedia.org/r/1363813 (https://phabricator.wikimedia.org/T443756) (owner: 10Gmodena) [15:04:19] Krinkle: ty! [15:04:47] (03CR) 10Ottomata: [C:03+2] changeprop-jobqueue: Add rule for edit_suggestions_counts LAC prefetch [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362092 (https://phabricator.wikimedia.org/T443442) (owner: 10Ottomata) [15:05:48] (03CR) 10Jelto: [C:03+1] "confirmed out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1367822 (https://phabricator.wikimedia.org/T442758) (owner: 10Esanders) [15:06:42] (03Merged) 10jenkins-bot: changeprop-jobqueue: Add rule for edit_suggestions_counts LAC prefetch [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362092 (https://phabricator.wikimedia.org/T443442) (owner: 10Ottomata) [15:07:16] (03CR) 10Ladsgroup: [C:03+1] Build image with async support in thumbor [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1367794 (owner: 10Jgiannelos) [15:07:17] !log otto@deploy2003 helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply [15:07:26] (03PS6) 10Gmodena: rest-gateway: set wdqs v2 GA rate limits [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358194 (https://phabricator.wikimedia.org/T437892) [15:07:34] (03CR) 10Jelto: [C:03+2] Add FIDO SSH keys for esanders [puppet] - 10https://gerrit.wikimedia.org/r/1367822 (https://phabricator.wikimedia.org/T442758) (owner: 10Esanders) [15:07:36] !log deploying changeprop-jobqueue to pick up new queue for T443442 [15:07:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:07:40] T443442: Edit Suggestion Counts - use LinkedArtifacts config - https://phabricator.wikimedia.org/T443442 [15:07:52] !log otto@deploy2003 helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply [15:08:28] !log otto@deploy2003 helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply [15:08:38] (03PS1) 10DCausse: admin: Add FIDO SSH keys for dcausse [puppet] - 10https://gerrit.wikimedia.org/r/1367860 [15:09:30] (03CR) 10Vgutierrez: [C:03+1] trafficserver: route WDQSv2 through REST Gateway [puppet] - 10https://gerrit.wikimedia.org/r/1363813 (https://phabricator.wikimedia.org/T443756) (owner: 10Gmodena) [15:09:59] jouncebot: nowandnext [15:09:59] No deployments scheduled for the next 0 hour(s) and 20 minute(s) [15:09:59] In 0 hour(s) and 20 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261109T1530) [15:10:03] !log otto@deploy2003 helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [15:10:25] !log otto@deploy2003 helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [15:11:41] !log otto@deploy2003 helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [15:12:00] (03PS1) 10Ladsgroup: Enable MPEG-DASH everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367862 (https://phabricator.wikimedia.org/T373546) [15:12:57] (03CR) 10TrainBranchBot: [C:03+2] "Approved by otto@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362093 (https://phabricator.wikimedia.org/T443442) (owner: 10Ottomata) [15:12:58] (03PS1) 10Jelto: Remove legacy key for jakob [puppet] - 10https://gerrit.wikimedia.org/r/1367863 (https://phabricator.wikimedia.org/T442763) [15:13:51] RESOLVED: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [15:14:14] (03Merged) 10jenkins-bot: Prefetch edit_suggestions_counts with EventBus LinkedArtifacts [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362093 (https://phabricator.wikimedia.org/T443442) (owner: 10Ottomata) [15:14:27] !log otto@deploy2003 Started scap sync-world: Backport for [[gerrit:1362093|Prefetch edit_suggestions_counts with EventBus LinkedArtifacts (T443442)]] [15:14:30] T443442: Edit Suggestion Counts - use LinkedArtifacts config - https://phabricator.wikimedia.org/T443442 [15:16:31] !log otto@deploy2003 otto: Backport for [[gerrit:1362093|Prefetch edit_suggestions_counts with EventBus LinkedArtifacts (T443442)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:16:46] (03CR) 10Jelto: [C:03+2] Remove legacy key for jakob [puppet] - 10https://gerrit.wikimedia.org/r/1367863 (https://phabricator.wikimedia.org/T442763) (owner: 10Jelto) [15:17:05] !log otto@deploy2003 otto: Continuing with deployment [15:17:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [15:19:04] (03PS2) 10Jelto: admin: Add FIDO SSH keys for dcausse [puppet] - 10https://gerrit.wikimedia.org/r/1367860 (https://phabricator.wikimedia.org/T442758) (owner: 10DCausse) [15:20:31] (03PS1) 10Jgiannelos: Revert "Build image with async support in thumbor" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1367864 [15:21:00] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367865 [15:21:30] (03PS1) 10Btullis: postgresql-airflow-dumps: Fix the values file name and grow the data volume [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367866 [15:22:33] !log klausman@deploy2003 helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [15:22:39] FIRING: [4x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (195.200.68.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:23:07] !log otto@deploy2003 Finished scap sync-world: Backport for [[gerrit:1362093|Prefetch edit_suggestions_counts with EventBus LinkedArtifacts (T443442)]] (duration: 08m 40s) [15:23:11] T443442: Edit Suggestion Counts - use LinkedArtifacts config - https://phabricator.wikimedia.org/T443442 [15:23:34] (03CR) 10Dbrant: [C:03+2] wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367865 (owner: 10PipelineBot) [15:25:03] (03Merged) 10jenkins-bot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367865 (owner: 10PipelineBot) [15:25:24] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367862 (https://phabricator.wikimedia.org/T373546) (owner: 10Ladsgroup) [15:26:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [15:26:15] (03Merged) 10jenkins-bot: Enable MPEG-DASH everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367862 (https://phabricator.wikimedia.org/T373546) (owner: 10Ladsgroup) [15:26:28] !log ladsgroup@deploy2003 Started scap sync-world: Backport for [[gerrit:1367862|Enable MPEG-DASH everywhere (T373546)]] [15:26:31] T373546: Migrate off HLS mov/mp4 experiment to chunked WebM MPEG-DASH - https://phabricator.wikimedia.org/T373546 [15:27:07] !log dbrant@deploy2003 helmfile [staging] START helmfile.d/services/wikifeeds: apply [15:27:19] !log klausman@deploy2003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [15:27:29] !log dbrant@deploy2003 helmfile [staging] DONE helmfile.d/services/wikifeeds: apply [15:27:39] !log dbrant@deploy2003 helmfile [eqiad] START helmfile.d/services/wikifeeds: apply [15:28:05] !log dbrant@deploy2003 helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply [15:28:13] !log dbrant@deploy2003 helmfile [codfw] START helmfile.d/services/wikifeeds: apply [15:28:32] !log ladsgroup@deploy2003 ladsgroup: Backport for [[gerrit:1367862|Enable MPEG-DASH everywhere (T373546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:28:36] !log dbrant@deploy2003 helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply [15:28:50] !log ayounsi@cumin1004 START - Cookbook sre.network.peering with action 'email' for AS: 267201 [15:29:08] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 267201 [15:29:19] !log ladsgroup@deploy2003 ladsgroup: Continuing with deployment [15:30:01] (03CR) 10Btullis: [C:03+2] postgresql-airflow-dumps: Fix the values file name and grow the data volume [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367866 (owner: 10Btullis) [15:30:04] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261109T1530) [15:31:45] (03Merged) 10jenkins-bot: postgresql-airflow-dumps: Fix the values file name and grow the data volume [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367866 (owner: 10Btullis) [15:32:50] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [15:33:30] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [15:34:56] (03PS1) 10Bking: matomo-next: Register service in IDP and configure CDN [puppet] - 10https://gerrit.wikimedia.org/r/1367868 (https://phabricator.wikimedia.org/T436186) [15:35:16] !log ladsgroup@deploy2003 Finished scap sync-world: Backport for [[gerrit:1367862|Enable MPEG-DASH everywhere (T373546)]] (duration: 08m 48s) [15:35:19] T373546: Migrate off HLS mov/mp4 experiment to chunked WebM MPEG-DASH - https://phabricator.wikimedia.org/T373546 [15:35:21] (03CR) 10Bking: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1367868 (https://phabricator.wikimedia.org/T436186) (owner: 10Bking) [15:36:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [15:36:06] (03PS1) 10Jgiannelos: thumbor: Pin async image in staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367869 [15:36:12] (03CR) 10Vgutierrez: [C:03+2] trafficserver: route WDQSv2 through REST Gateway [puppet] - 10https://gerrit.wikimedia.org/r/1363813 (https://phabricator.wikimedia.org/T443756) (owner: 10Gmodena) [15:37:35] (03PS2) 10Jgiannelos: thumbor: Pin async image in staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367869 (https://phabricator.wikimedia.org/T431767) [15:42:17] (03PS2) 10Lucian: Add lucian to the ops-limited group [puppet] - 10https://gerrit.wikimedia.org/r/1364304 [15:42:17] (03PS1) 10Lucian: Admin: add a hardware-backed ssh pubkey for lucian [puppet] - 10https://gerrit.wikimedia.org/r/1367870 [15:43:30] (03CR) 10DLynch: [C:03+1] changeprop-jobqueue: Remove editSuggestionCountsPrecompute rule [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362096 (https://phabricator.wikimedia.org/T443442) (owner: 10Ottomata) [15:45:45] 10ops-magru: Outbound errors on interface cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://phabricator.wikimedia.org/T444127#12534504 (10RobH) Chatted in IRC about this, current plan of action: * I'll file remote hands for this work ** swap the QSFP in the router with the QSFP set on the servers fro... [15:46:10] (03PS3) 10Lucian: Add lucian to the ops-limited group [puppet] - 10https://gerrit.wikimedia.org/r/1364304 [15:47:27] (03Abandoned) 10Lucian: Admin: add a hardware-backed ssh pubkey for lucian [puppet] - 10https://gerrit.wikimedia.org/r/1367870 (owner: 10Lucian) [15:47:51] RESOLVED: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [15:48:37] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-dumps: apply [15:48:47] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-dumps: apply [15:49:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [15:53:06] (03PS2) 10Ottomata: [Beta EventBus] Add revision_tags_change and log_tags_change dev0 streams, enable in beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357611 (https://phabricator.wikimedia.org/T434143) (owner: 10TChin) [15:53:39] (03CR) 10TrainBranchBot: [C:03+2] "Approved by otto@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357611 (https://phabricator.wikimedia.org/T434143) (owner: 10TChin) [15:54:50] (03Merged) 10jenkins-bot: [Beta EventBus] Add revision_tags_change and log_tags_change dev0 streams, enable in beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357611 (https://phabricator.wikimedia.org/T434143) (owner: 10TChin) [15:55:04] !log otto@deploy2003 Started scap sync-world: Backport for [[gerrit:1357611|[Beta EventBus] Add revision_tags_change and log_tags_change dev0 streams, enable in beta (T434143)]] [15:55:07] T434143: Capture relevant mediawiki tags changes in event stream(s) - https://phabricator.wikimedia.org/T434143 [15:59:19] !log otto@deploy2003 tchin, otto: Backport for [[gerrit:1357611|[Beta EventBus] Add revision_tags_change and log_tags_change dev0 streams, enable in beta (T434143)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:00:49] !log otto@deploy2003 tchin, otto: Continuing with deployment [16:01:48] !log klausman@deploy2003 helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [16:01:56] (03CR) 10Jelto: [C:03+2] Convert antonkokhwmde from SSH to LDAP access [puppet] - 10https://gerrit.wikimedia.org/r/1367856 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [16:02:08] !log klausman@deploy2003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [16:03:01] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [16:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:05:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [16:05:23] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:05:36] !log jelto@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply [16:06:06] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [16:07:03] !log otto@deploy2003 Finished scap sync-world: Backport for [[gerrit:1357611|[Beta EventBus] Add revision_tags_change and log_tags_change dev0 streams, enable in beta (T434143)]] (duration: 11m 59s) [16:07:06] T434143: Capture relevant mediawiki tags changes in event stream(s) - https://phabricator.wikimedia.org/T434143 [16:08:56] !log klausman@deploy2003 helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [16:09:24] RESOLVED: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [16:10:51] !log klausman@deploy2003 helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [16:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:11:54] (03PS1) 10Bking: IDP: Add dummy secret for staging Matomo [labs/private] - 10https://gerrit.wikimedia.org/r/1367873 (https://phabricator.wikimedia.org/T436186) [16:12:34] (03PS2) 10Bking: matomo-next: Register service in IDP and configure CDN [puppet] - 10https://gerrit.wikimedia.org/r/1367868 (https://phabricator.wikimedia.org/T436186) [16:12:54] (03PS2) 10Gehel: admin: removed non FIDO SSH keys for dsaez [puppet] - 10https://gerrit.wikimedia.org/r/1367827 (https://phabricator.wikimedia.org/T442756) [16:12:54] (03PS1) 10Gehel: admin: adding FIDO/SSH keys for eniroomand [puppet] - 10https://gerrit.wikimedia.org/r/1367874 (https://phabricator.wikimedia.org/T442756) [16:13:19] (03CR) 10Bking: [V:03+2 C:03+2] IDP: Add dummy secret for staging Matomo [labs/private] - 10https://gerrit.wikimedia.org/r/1367873 (https://phabricator.wikimedia.org/T436186) (owner: 10Bking) [16:13:40] (03CR) 10Bking: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1367868 (https://phabricator.wikimedia.org/T436186) (owner: 10Bking) [16:14:05] (03PS2) 10Gehel: admin: adding FIDO/SSH keys for eniroomand, cmyrick [puppet] - 10https://gerrit.wikimedia.org/r/1367874 (https://phabricator.wikimedia.org/T442756) [16:14:23] (03CR) 10Gehel: [C:03+2] admin: removed non FIDO SSH keys for dsaez [puppet] - 10https://gerrit.wikimedia.org/r/1367827 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [16:15:59] !log jelto@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply [16:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:17:24] (03CR) 10Btullis: [C:03+1] admin: adding FIDO/SSH keys for eniroomand, cmyrick [puppet] - 10https://gerrit.wikimedia.org/r/1367874 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [16:18:17] (03CR) 10Gehel: [C:03+2] admin: adding FIDO/SSH keys for eniroomand, cmyrick [puppet] - 10https://gerrit.wikimedia.org/r/1367874 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [16:20:17] (03CR) 10Jelto: [C:03+1] "verified out of band and confirmed that 30 minute delay for access is fine" [puppet] - 10https://gerrit.wikimedia.org/r/1367860 (https://phabricator.wikimedia.org/T442758) (owner: 10DCausse) [16:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:21:58] (03CR) 10Jelto: [C:03+2] admin: Add FIDO SSH keys for dcausse [puppet] - 10https://gerrit.wikimedia.org/r/1367860 (https://phabricator.wikimedia.org/T442758) (owner: 10DCausse) [16:29:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12534791 (10phaultfinder) [16:29:53] (03PS1) 10Jelto: Add FIDO key for ebysans [puppet] - 10https://gerrit.wikimedia.org/r/1367876 (https://phabricator.wikimedia.org/T442758) [16:30:05] jan_drewniak: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for Wikimedia Portals Update deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261109T1630). [16:30:08] (03PS1) 10Trueg: WDQSv2: Remove node affinitity [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367877 (https://phabricator.wikimedia.org/T443792) [16:32:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [16:32:53] (03CR) 10Jelto: [C:03+2] Add FIDO key for ebysans [puppet] - 10https://gerrit.wikimedia.org/r/1367876 (https://phabricator.wikimedia.org/T442758) (owner: 10Jelto) [16:37:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [16:43:42] !log klausman@deploy2003 helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [16:45:41] !log klausman@deploy2003 helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [16:53:28] (03PS1) 10Cparle: Remove old ssh key for cparle [puppet] - 10https://gerrit.wikimedia.org/r/1367886 (https://phabricator.wikimedia.org/T442758) [17:00:05] brennen: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for Release Engineering Clinic Duty . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261109T1700). [17:04:56] (03CR) 10Michael Große: [C:03+1] GrowthExperiments: Remove the $wgGECampaignTopics setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367767 (https://phabricator.wikimedia.org/T444124) (owner: 10Urbanecm) [17:06:13] (03CR) 10Trueg: [C:03+2] WDQSv2: Remove node affinitity [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367877 (https://phabricator.wikimedia.org/T443792) (owner: 10Trueg) [17:07:40] (03Merged) 10jenkins-bot: WDQSv2: Remove node affinitity [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367877 (https://phabricator.wikimedia.org/T443792) (owner: 10Trueg) [17:08:53] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [17:10:20] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888#12535080 (10MatthewVernon) So. I have done some bodging. After problems with rsync transferring invalid files (which doesn't always happen, but does sometimes - swift upstream might like to consider https:/... [17:15:59] (03PS1) 10Esanders: admin: removed non FIDO SSH key for esanders [puppet] - 10https://gerrit.wikimedia.org/r/1367891 (https://phabricator.wikimedia.org/T442758) [17:20:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:23:30] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [17:23:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [17:25:04] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [17:28:21] !log T444015 start full-cluster reindex in eqiad [17:28:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:28:24] T444015: CirrusSearchIndexTooOld - https://phabricator.wikimedia.org/T444015 [17:28:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [17:35:12] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [17:36:06] !log trueg@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [17:40:31] !log T444015 start cirrussearch full-cluster reindex in codfw and cloudelastic [17:40:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:40:34] T444015: CirrusSearchIndexTooOld - https://phabricator.wikimedia.org/T444015 [17:42:26] jouncebot: nowandnext [17:42:27] For the next 0 hour(s) and 17 minute(s): Release Engineering Clinic Duty (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261109T1700) [17:42:27] In 0 hour(s) and 17 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261109T1800) [17:42:27] In 0 hour(s) and 17 minute(s): Wikidata Query Service weekly deploy (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261109T1800) [17:44:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12535207 (10phaultfinder) [17:45:42] !log trueg@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [17:48:27] (03PS1) 10Kosta Harlan: AbuseReview: Delay the link spam model check job [extensions/WikimediaAntiAbuse] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1367904 (https://phabricator.wikimedia.org/T442170) [17:48:58] !log cgoubert@deploy2003 helmfile [codfw] START helmfile.d/services/rest-gateway: apply [17:49:32] !log cgoubert@deploy2003 helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [17:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.11% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [17:54:31] !log cgoubert@deploy2003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [17:54:44] !log cgoubert@deploy2003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [17:58:33] (03PS1) 10Dwisehaupt: Remove old ssh key for dwisehaupt [puppet] - 10https://gerrit.wikimedia.org/r/1367907 (https://phabricator.wikimedia.org/T442759) [18:00:02] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888#12535269 (10MatthewVernon) Actually, I was wrong, thanos-be1006 isn't a handoff host. Inspecting the rsync command that was running during a container-sync run showed me ` rsync --quiet --no-motd --timeout=... [18:00:04] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261109T1800) [18:00:04] ryankemper: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for Wikidata Query Service weekly deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261109T1800). [18:01:33] (03CR) 10Btullis: [C:03+2] deployment_server: Restrict the spark namespaces to analytics-deployers [puppet] - 10https://gerrit.wikimedia.org/r/1367771 (https://phabricator.wikimedia.org/T429558) (owner: 10Btullis) [18:06:02] !log cgoubert@deploy2003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [18:06:08] (03CR) 10Btullis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1367772 (https://phabricator.wikimedia.org/T432322) (owner: 10Btullis) [18:06:13] !log cgoubert@deploy2003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [18:06:35] !log cgoubert@deploy2003 helmfile [codfw] START helmfile.d/services/rest-gateway: apply [18:06:48] !log cgoubert@deploy2003 helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [18:16:52] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 12 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1366869 (https://phabricator.wikimedia.org/T419612) (owner: 10SBassett) [18:36:25] 06SRE, 10SRE-swift-storage: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888#12535393 (10MatthewVernon) I've been alerted to a [[ https://docs.google.com/document/d/1mCY_KDHNlgWLPegySsMKn3g-7l370jfRgDoiPcZDQc4/edit?pli=1&tab=t.0#heading=h.wcknp1nj0lhz | disk wear analysis document ]]... [18:38:09] (03PS1) 10Gmodena: rest-gateway: set SNI on WDQSv2 upstream clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367912 (https://phabricator.wikimedia.org/T436778) [18:40:34] (03PS1) 10Btullis: profile::ceph::admin: take snapshots of the OpenTofu state [puppet] - 10https://gerrit.wikimedia.org/r/1367913 (https://phabricator.wikimedia.org/T440891) [18:40:36] (03PS1) 10Btullis: profile::ceph::admin: copy the OpenTofu state snapshots between hosts [puppet] - 10https://gerrit.wikimedia.org/r/1367914 (https://phabricator.wikimedia.org/T440891) [18:40:38] (03PS1) 10Btullis: profile::ceph::admin: back up the OpenTofu state snapshots with Bacula [puppet] - 10https://gerrit.wikimedia.org/r/1367915 (https://phabricator.wikimedia.org/T440891) [18:40:40] (03PS1) 10Btullis: profile::ceph::admin: check the OpenTofu plan every day [puppet] - 10https://gerrit.wikimedia.org/r/1367916 (https://phabricator.wikimedia.org/T440891) [18:40:54] (03PS1) 10Btullis: team-data-platform: alert on pending OpenTofu changes [alerts] - 10https://gerrit.wikimedia.org/r/1367917 (https://phabricator.wikimedia.org/T440891) [18:41:07] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:41:48] (03PS1) 10Xcollazo: dumps: Fetch File Exports every 4 hours [puppet] - 10https://gerrit.wikimedia.org/r/1367918 (https://phabricator.wikimedia.org/T440383) [18:44:08] (03CR) 10CI reject: [V:04-1] dumps: Fetch File Exports every 4 hours [puppet] - 10https://gerrit.wikimedia.org/r/1367918 (https://phabricator.wikimedia.org/T440383) (owner: 10Xcollazo) [18:46:56] (03PS2) 10Xcollazo: dumps: Fetch File Exports every 4 hours [puppet] - 10https://gerrit.wikimedia.org/r/1367918 (https://phabricator.wikimedia.org/T440383) [18:47:06] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [18:47:19] (03PS1) 10Btullis: airflow-dumps: Parse the DAG files every 300 seconds again [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367919 (https://phabricator.wikimedia.org/T435478) [18:49:09] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [18:50:56] (03CR) 10Clément Goubert: [C:03+1] rest-gateway: set SNI on WDQSv2 upstream clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367912 (https://phabricator.wikimedia.org/T436778) (owner: 10Gmodena) [18:51:16] (03CR) 10Xcollazo: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1367918 (https://phabricator.wikimedia.org/T440383) (owner: 10Xcollazo) [18:52:43] (03CR) 10Btullis: [C:03+2] dumps: Fetch File Exports every 4 hours [puppet] - 10https://gerrit.wikimedia.org/r/1367918 (https://phabricator.wikimedia.org/T440383) (owner: 10Xcollazo) [18:56:48] (03CR) 10Btullis: [C:03+2] airflow-dumps: Parse the DAG files every 300 seconds again [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367919 (https://phabricator.wikimedia.org/T435478) (owner: 10Btullis) [18:58:06] (03Merged) 10jenkins-bot: airflow-dumps: Parse the DAG files every 300 seconds again [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367919 (https://phabricator.wikimedia.org/T435478) (owner: 10Btullis) [19:00:11] FIRING: Temperature: Temp issue on wdqs2023:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=wdqs2023 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [19:00:12] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [19:01:34] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [19:05:11] RESOLVED: Temperature: Temp issue on wdqs2023:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=wdqs2023 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [19:06:11] FIRING: Temperature: Temp issue on wdqs2023:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=wdqs2023 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [19:10:26] RESOLVED: Temperature: Temp issue on wdqs2023:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=wdqs2023 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [19:15:23] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:29:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12535581 (10phaultfinder) [19:29:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [19:31:58] (03CR) 10Ryan Kemper: [C:03+1] deployment_server: Add the kubeconfigs of the team Spark namespaces [puppet] - 10https://gerrit.wikimedia.org/r/1367772 (https://phabricator.wikimedia.org/T432322) (owner: 10Btullis) [19:32:04] (03CR) 10Ryan Kemper: [C:03+1] admin_ng: Add a Spark namespace for each team on dse-k8s-eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367769 (https://phabricator.wikimedia.org/T432322) (owner: 10Btullis) [19:32:09] (03CR) 10Ryan Kemper: [C:03+1] spark-operator: Watch the analytics and team Spark namespaces [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367770 (https://phabricator.wikimedia.org/T432322) (owner: 10Btullis) [19:32:31] FIRING: Traffic bill over quota: Alert for device cr2-eqiad.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [19:33:16] (03CR) 10BPirkle: [C:03+2] rest-gateway: set SNI on WDQSv2 upstream clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367912 (https://phabricator.wikimedia.org/T436778) (owner: 10Gmodena) [19:35:23] (03Merged) 10jenkins-bot: rest-gateway: set SNI on WDQSv2 upstream clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367912 (https://phabricator.wikimedia.org/T436778) (owner: 10Gmodena) [19:37:31] FIRING: [2x] Traffic bill over quota: Alert for device cr1-codfw.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [19:44:37] !log gmodena@deploy2003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [19:44:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12535624 (10phaultfinder) [19:44:48] !log gmodena@deploy2003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [19:52:31] FIRING: [2x] Traffic bill over quota: Alert for device cr1-codfw.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [19:54:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [19:57:31] RESOLVED: Traffic bill over quota: Alert for device cr1-codfw.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [20:00:08] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [20:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:08:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:13:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:14:52] (03PS1) 10Gmodena: Revert "rest-gateway: set SNI on WDQSv2 upstream clusters" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367937 [20:15:43] (03CR) 10Gaiarajan: [C:03+1] Revert "rest-gateway: set SNI on WDQSv2 upstream clusters" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367937 (owner: 10Gmodena) [20:16:04] (03CR) 10BPirkle: [C:03+2] Revert "rest-gateway: set SNI on WDQSv2 upstream clusters" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367937 (owner: 10Gmodena) [20:17:06] (03PS1) 10Aqu: admin: Add FIDO SSH keys for aqu [puppet] - 10https://gerrit.wikimedia.org/r/1367939 (https://phabricator.wikimedia.org/T442758) [20:17:42] (03Merged) 10jenkins-bot: Revert "rest-gateway: set SNI on WDQSv2 upstream clusters" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367937 (owner: 10Gmodena) [20:19:17] !log gmodena@deploy2003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [20:19:28] !log gmodena@deploy2003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [20:46:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:50:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [20:51:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:59:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12535900 (10phaultfinder) [21:00:05] RoanKattouw, urbanecm, TheresNoTime, and cjming: It is that lovely time of the day again! You are hereby commanded to deploy UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261109T2100). [21:00:05] EricGardner: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [21:01:58] I see that I'm the only person with a patch for the UTC Late Backport window. I can go ahead and deploy it myself unless someone else is actively deploying [21:04:22] (03CR) 10TrainBranchBot: [C:03+2] "Approved by egardner@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358854 (https://phabricator.wikimedia.org/T439162) (owner: 10Eric Gardner) [21:05:16] (03Merged) 10jenkins-bot: Enable the ShareCard feature on all Wikipedias (Minerva only) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358854 (https://phabricator.wikimedia.org/T439162) (owner: 10Eric Gardner) [21:05:33] !log egardner@deploy2003 Started scap sync-world: Backport for [[gerrit:1358854|Enable the ShareCard feature on all Wikipedias (Minerva only) (T439162)]] [21:05:36] T439162: ShareCard: Launch Checklist - https://phabricator.wikimedia.org/T439162 [21:11:38] !log egardner@deploy2003 egardner: Backport for [[gerrit:1358854|Enable the ShareCard feature on all Wikipedias (Minerva only) (T439162)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:11:41] T439162: ShareCard: Launch Checklist - https://phabricator.wikimedia.org/T439162 [21:12:18] !log egardner@deploy2003 egardner: Continuing with deployment [21:12:38] (03CR) 10AKhatun: [C:03+2] revision feature counts lac-lambda: reduce cpu and memory [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362476 (https://phabricator.wikimedia.org/T442914) (owner: 10AKhatun) [21:12:53] 10ops-magru: Outbound errors on interface cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://phabricator.wikimedia.org/T444127#12536000 (10RobH) CS1376707 submitted with the directions linked above, they'll review and reach out to schedule the service according to the portal. [21:13:16] (03CR) 10AKhatun: [C:03+2] revision feature counts lac-lambda: Tune alerts and add 2 alerts [alerts] - 10https://gerrit.wikimedia.org/r/1362469 (https://phabricator.wikimedia.org/T442914) (owner: 10AKhatun) [21:14:25] (03Merged) 10jenkins-bot: revision feature counts lac-lambda: reduce cpu and memory [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362476 (https://phabricator.wikimedia.org/T442914) (owner: 10AKhatun) [21:15:12] (03PS1) 10Jdrewniak: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367950 (https://phabricator.wikimedia.org/T128546) [21:16:52] !log akhatun@deploy2003 helmfile [staging] START helmfile.d/services/lac-lambdas: apply [21:17:00] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 10 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367950 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [21:17:10] !log akhatun@deploy2003 helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply [21:17:49] (03Merged) 10jenkins-bot: revision feature counts lac-lambda: Tune alerts and add 2 alerts [alerts] - 10https://gerrit.wikimedia.org/r/1362469 (https://phabricator.wikimedia.org/T442914) (owner: 10AKhatun) [21:18:21] !log egardner@deploy2003 Finished scap sync-world: Backport for [[gerrit:1358854|Enable the ShareCard feature on all Wikipedias (Minerva only) (T439162)]] (duration: 12m 48s) [21:18:24] T439162: ShareCard: Launch Checklist - https://phabricator.wikimedia.org/T439162 [21:19:23] !log akhatun@deploy2003 helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply [21:19:45] !log akhatun@deploy2003 helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply [21:21:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [21:22:36] !log akhatun@deploy2003 helmfile [codfw] START helmfile.d/services/lac-lambdas: apply [21:23:02] !log akhatun@deploy2003 helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply [21:28:43] (03Abandoned) 10Cwhite: ncredir: add wikimediastatus.net funnel [puppet] - 10https://gerrit.wikimedia.org/r/1242499 (https://phabricator.wikimedia.org/T419887) (owner: 10Cwhite) [21:46:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [21:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.13% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [21:53:09] 10SRE-Access-Requests: Add FIDO-backed SSH keys for cooltey - https://phabricator.wikimedia.org/T444236 (10cooltey) 03NEW [21:54:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12536175 (10phaultfinder) [21:56:23] (03PS1) 10Southparkfan: {echo,session}store: Use external-services for egress policies [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367957 (https://phabricator.wikimedia.org/T365172) [22:00:04] alexsanford, Reedy, sbassett, Maryum, and manfredi: May I have your attention please! Weekly Security deployment window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261109T2200) [22:05:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:06:01] is anyone doing deployments? preparing to do a security deploy [22:08:49] (03PS1) 10Cooltey: admin: Add FIDO SSH keys for cooltey [puppet] - 10https://gerrit.wikimedia.org/r/1367964 [22:09:36] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12536209 (10phaultfinder) [22:09:44] (03PS2) 10Cooltey: admin: Add FIDO SSH keys for cooltey [puppet] - 10https://gerrit.wikimedia.org/r/1367964 (https://phabricator.wikimedia.org/T444236) [22:10:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:27:40] (03PS1) 10Ladsgroup: mw-videoscaler: Expand the normal pool but allow stealing from prio queue [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367963 (https://phabricator.wikimedia.org/T444235) (owner: 10Ladsgroup-claude) [22:29:56] preparing to run scap [22:41:07] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [22:47:06] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [22:49:09] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [22:49:42] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Requesting access to Analytics Data Lake for Federico Ceratto - https://phabricator.wikimedia.org/T443235#12536370 (10Ahoelzl) [22:54:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12536413 (10phaultfinder) [23:11:00] !log Deployed security fix for T443539, T440392, T435882, T442600 [23:11:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:15:23] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:49:00] 06SRE, 10MediaWiki-Core-HTTP-Cache, 06Traffic-Icebox, 07Wikimedia-Performance-recommendation: Separate Cache-Control header for proxy and client - https://phabricator.wikimedia.org/T50835#12536717 (10Krinkle) This came up again during {T279120} at https://gerrit.wikimedia.org/r/c/operations/puppet/+/136477...