[00:00:07] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox: sync [00:00:14] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [00:00:18] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [00:00:25] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [00:02:42] 06SRE, 10Wikimedia-Mailing-lists, 07Chinese-Sites: Alternative mailing lists for Chinese ArbCom - https://phabricator.wikimedia.org/T442488#12493011 (10Ladsgroup) Honestly, there should be a better way to do this but okay. I'll do my best to create them soon. [00:05:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:07:25] FIRING: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:07:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr2-drmrs and Orange (193.251.154.145) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [00:10:10] (03CR) 10Dzahn: "thank you. did the partial revert and kept the other part. for the "real fix" I think maybe we should get it blocked on the webproxies?" [puppet] - 10https://gerrit.wikimedia.org/r/1359404 (https://phabricator.wikimedia.org/T439046) (owner: 10Majavah) [00:10:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:12:25] RESOLVED: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:17:39] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for snowick - https://phabricator.wikimedia.org/T442951#12493047 (10Dzahn) Hi @SNowick_WMF do you already have a Yubikey or do you first need hardware? https://office.wikimedia.org/wiki/Security/Training/Staff_Basics#YubiKey_Security_Keys If you alread... [00:18:54] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for kemayo - https://phabricator.wikimedia.org/T442902#12493053 (10Dzahn) Hi! do you already have a Yubikey or do you first need hardware? https://office.wikimedia.org/wiki/Security/Training/Staff_Basics#YubiKey_Security_Keys If yo... [00:19:30] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for kemayo - https://phabricator.wikimedia.org/T442902#12493055 (10DLynch) Keys are generated and in the attached patch. [00:19:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12493067 (10phaultfinder) [00:19:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12493068 (10phaultfinder) [00:21:33] 06SRE, 10SRE-Access-Requests: Add yubikey SSH key for urbanecm - https://phabricator.wikimedia.org/T442901#12493072 (10Dzahn) Sent an email for out-of-band key verification. [00:22:06] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for kevinbazira - https://phabricator.wikimedia.org/T442865#12493074 (10Dzahn) 05Open→03In progress [00:22:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:22:49] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for kevinbazira - https://phabricator.wikimedia.org/T442865#12493078 (10Dzahn) Hi @kevinbazira please let us know if the new key works and after that we can remove the old key. [00:24:50] (03PS1) 101Veertje: Update Citoid's User-Agent to a current Chrome version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360574 (https://phabricator.wikimedia.org/T442971) [00:25:26] 06SRE, 10SRE-Access-Requests: Request IRC #mediawiki_security access for Novem Linguae - https://phabricator.wikimedia.org/T442747#12493086 (10Dzahn) @LSobanski Any thoughts on who gets to approve IRC access requests? [00:26:42] (03CR) 10Dzahn: [C:03+2] admin: Add one FIDO key for xcollazo [puppet] - 10https://gerrit.wikimedia.org/r/1359924 (https://phabricator.wikimedia.org/T442741) (owner: 10Xcollazo) [00:26:47] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for xcollazo - https://phabricator.wikimedia.org/T442741#12493088 (10Dzahn) I was about to mail you but since the key was already verified out-of-band by @SLyngshede-WMF (per Gerrit comment), I am going ahead and merging this. [00:27:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:27:30] (03CR) 10Dzahn: [C:03+2] "only yubikey and already verified. going ahead." [puppet] - 10https://gerrit.wikimedia.org/r/1359924 (https://phabricator.wikimedia.org/T442741) (owner: 10Xcollazo) [00:28:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [00:28:13] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for xcollazo - https://phabricator.wikimedia.org/T442741#12493090 (10Dzahn) @xcollazo Please give it up to 30 min to propagate. Then let us know if it works and when we can remove the old key. [00:29:01] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for xcollazo - https://phabricator.wikimedia.org/T442741#12493091 (10Dzahn) 05Open→03In progress [00:30:22] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-drmrs:xe-0/1/2 (Transit: Orange (LD019029)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-drmrs:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [00:31:05] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [00:34:40] (03PS1) 10Dzahn: admin: document Kerberos identity for rsilvola [puppet] - 10https://gerrit.wikimedia.org/r/1360575 (https://phabricator.wikimedia.org/T442695) [00:34:57] FIRING: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:35:05] (03CR) 10Dzahn: [C:03+2] admin: document Kerberos identity for rsilvola [puppet] - 10https://gerrit.wikimedia.org/r/1360575 (https://phabricator.wikimedia.org/T442695) (owner: 10Dzahn) [00:36:18] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for mpostoronca - https://phabricator.wikimedia.org/T442390#12493111 (10Dzahn) 05Stalled→03In progress [00:36:24] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for aikochou - https://phabricator.wikimedia.org/T442367#12493114 (10Dzahn) 05Stalled→03In progress [00:38:04] 06SRE, 10SRE-Access-Requests, 06Product Safety and Integrity, 06Security-Team, 13Patch-For-Review: Replace SSH key with hw backed SSH keys for rsilvola - https://phabricator.wikimedia.org/T442092#12493117 (10Dzahn) 05Stalled→03In progress [00:39:00] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add new hardware backed key for osefu - https://phabricator.wikimedia.org/T442183#12493118 (10Dzahn) Hi @OSefu-WMF do you have a Yubikey and could use that for the new keys? [00:39:08] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add new hardware backed key for osefu - https://phabricator.wikimedia.org/T442183#12493119 (10Dzahn) 05Stalled→03Open [00:39:57] RESOLVED: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:41:07] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 13Patch-For-Review: Requesting Kerberos identity for rsilvola - https://phabricator.wikimedia.org/T442695#12493127 (10Dzahn) @Rsilvola Please look out for an email about the creation of your Kerberos principal. It will give you a temp. password and exp... [00:41:14] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 13Patch-For-Review: Requesting Kerberos identity for rsilvola - https://phabricator.wikimedia.org/T442695#12493129 (10Dzahn) 05Open→03Resolved a:03Dzahn [00:42:22] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1360577 [00:42:23] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1360577 (owner: 10TrainBranchBot) [00:45:22] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr2-drmrs:xe-0/1/2 (Transit: Orange (LD019029)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-drmrs:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [00:48:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [00:48:53] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1360577 (owner: 10TrainBranchBot) [01:04:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12493184 (10phaultfinder) [01:09:53] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12493205 (10phaultfinder) [01:10:22] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-drmrs:xe-0/1/2 (Transit: Orange (LD019029)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-drmrs:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [01:12:16] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1360580 [01:12:16] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1360580 (owner: 10TrainBranchBot) [01:19:05] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1360580 (owner: 10TrainBranchBot) [01:20:42] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [01:25:22] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr2-drmrs:xe-0/1/2 (Transit: Orange (LD019029)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-drmrs:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [01:35:45] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add new hardware backed key for osefu - https://phabricator.wikimedia.org/T442183#12493248 (10OSefu-WMF) Not yet. I have requested one via the process that was emailed out today. Perhaps we should closed this request and I can manage key enrollment through th... [01:42:39] RESOLVED: [2x] TransitBGPDown: Transit BGP session down between cr2-drmrs and Orange (193.251.154.145) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [01:47:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [02:01:11] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:06:46] 10ops-eqiad, 06DC-Ops: Physical Checking - https://phabricator.wikimedia.org/T442911#12493293 (10Aklapper) [02:09:02] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 51s) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:22:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [02:37:57] FIRING: ProbeDown: Service mw-api-int:4446 has failed probes (http_mw-api-int_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-int:4446 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [02:42:57] RESOLVED: ProbeDown: Service mw-api-int:4446 has failed probes (http_mw-api-int_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-int:4446 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:07:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [03:12:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [03:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:19:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12493484 (10phaultfinder) [04:24:22] 06SRE, 10SRE-Access-Requests, 06Product Safety and Integrity, 06Security-Team: Request IRC #mediawiki_security access for Novem Linguae - https://phabricator.wikimedia.org/T442747#12493487 (10Aklapper) [04:31:05] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [04:34:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12493519 (10phaultfinder) [04:36:22] !log T442431 [airflow-experiment-platform] starting upgrade to Airflow 3.3; pausing its DAGs [04:36:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:36:28] T442431: Upgrade airflow-experiment-platform to 3.3.x - https://phabricator.wikimedia.org/T442431 [04:39:05] (03CR) 10Ryan Kemper: [C:03+2] kafka-jumbo: declare Data Engineering topics [puppet] - 10https://gerrit.wikimedia.org/r/1355378 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [04:39:10] (03CR) 10Ryan Kemper: [C:03+2] kafka-jumbo: declare Search topics [puppet] - 10https://gerrit.wikimedia.org/r/1355379 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [04:39:13] (03CR) 10Ryan Kemper: [C:03+2] kafka-jumbo: declare Wikidata Platform topics [puppet] - 10https://gerrit.wikimedia.org/r/1355380 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [04:39:16] (03CR) 10Ryan Kemper: [C:03+2] kafka-jumbo: declare Experiment Platform topics [puppet] - 10https://gerrit.wikimedia.org/r/1355381 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [04:39:19] (03CR) 10Ryan Kemper: [C:03+2] kafka-jumbo: declare Reader Experience topics [puppet] - 10https://gerrit.wikimedia.org/r/1355382 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [04:39:22] (03CR) 10Ryan Kemper: [C:03+2] kafka-jumbo: declare Editing topics [puppet] - 10https://gerrit.wikimedia.org/r/1355383 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [04:39:26] (03CR) 10Ryan Kemper: [C:03+2] kafka-jumbo: declare Infra Foundations topics [puppet] - 10https://gerrit.wikimedia.org/r/1355384 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [04:39:29] (03CR) 10Ryan Kemper: [C:03+2] kafka-jumbo: declare Observability topics [puppet] - 10https://gerrit.wikimedia.org/r/1355385 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [04:43:31] (03CR) 10Ryan Kemper: [C:03+2] airflow-experiment-platform: upgrade to Airflow 3.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360245 (https://phabricator.wikimedia.org/T442431) (owner: 10Ryan Kemper) [04:44:54] (03Merged) 10jenkins-bot: airflow-experiment-platform: upgrade to Airflow 3.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360245 (https://phabricator.wikimedia.org/T442431) (owner: 10Ryan Kemper) [04:45:29] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [04:46:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [04:49:34] !log ryankemper@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply [04:50:23] !log ryankemper@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply [04:51:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [04:51:22] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add new hardware backed key for osefu - https://phabricator.wikimedia.org/T442183#12493554 (10Dzahn) 05Open→03Stalled @OSefu-WMF Ah! I understand. Yea, no worries at all. We can still use this ticket to get it done but there is no rush. [05:06:32] !log T442431 [airflow-experiment-platform] completed upgrade to Airflow 3.3; unpaused its DAGs and verified successful growthbook ldap sync dag runs for both staging and prod. this is done [05:06:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:06:38] T442431: Upgrade airflow-experiment-platform to 3.3.x - https://phabricator.wikimedia.org/T442431 [05:06:57] FIRING: ProbeDown: Service mw-api-int:4446 has failed probes (http_mw-api-int_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-int:4446 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [05:11:57] RESOLVED: ProbeDown: Service mw-api-int:4446 has failed probes (http_mw-api-int_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-int:4446 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [05:14:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12493585 (10phaultfinder) [05:20:42] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [05:35:29] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [05:37:38] (03PS1) 10Ryan Kemper: kafka-jumbo: sync wdqs partition counts [puppet] - 10https://gerrit.wikimedia.org/r/1360786 (https://phabricator.wikimedia.org/T435284) [05:38:21] (03CR) 10Ryan Kemper: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360786 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [05:59:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12493613 (10phaultfinder) [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261030T0600) [06:16:00] 10ops-eqiad, 06SRE, 06DC-Ops: eqiad: audit mgmt switches - https://phabricator.wikimedia.org/T441671#12493619 (10ayounsi) [06:16:02] 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12493620 (10ayounsi) [06:17:23] (03PS1) 10Kevin Bazira: admin: Remove non-FIDO SSH key for kevinbazira [puppet] - 10https://gerrit.wikimedia.org/r/1360789 (https://phabricator.wikimedia.org/T442865) [06:20:37] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for kevinbazira - https://phabricator.wikimedia.org/T442865#12493624 (10kevinbazira) >>! In T442865#12493074, @Dzahn wrote: > Hi @kevinbazira > > please let us know if the new key works and after that we can remove the old key. Hi... [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261030T0700) [07:16:27] (03CR) 10Slyngshede: [C:03+1] admin: Remove non-FIDO SSH key for kevinbazira [puppet] - 10https://gerrit.wikimedia.org/r/1360789 (https://phabricator.wikimedia.org/T442865) (owner: 10Kevin Bazira) [07:16:33] (03CR) 10Slyngshede: [C:03+2] admin: Remove non-FIDO SSH key for kevinbazira [puppet] - 10https://gerrit.wikimedia.org/r/1360789 (https://phabricator.wikimedia.org/T442865) (owner: 10Kevin Bazira) [07:24:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12493674 (10phaultfinder) [07:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:46:29] (03PS6) 10Slyngshede: P:idp Java 25 and Tomcat 11 [puppet] - 10https://gerrit.wikimedia.org/r/1346531 (https://phabricator.wikimedia.org/T439509) [07:49:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12493693 (10phaultfinder) [07:54:42] (03Abandoned) 10Arthur taylor: Add smartcard SSH key for arthurtaylor [puppet] - 10https://gerrit.wikimedia.org/r/1119482 (owner: 10Arthur taylor) [08:01:56] (03PS1) 10Kevin Bazira: ml-services: update tts-section-generator to split long sections across multiple requests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360843 (https://phabricator.wikimedia.org/T441983) [08:04:38] (03CR) 10Kevin Bazira: [C:03+2] ml-services: update tts-section-generator to split long sections across multiple requests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360843 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [08:06:06] (03Merged) 10jenkins-bot: ml-services: update tts-section-generator to split long sections across multiple requests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360843 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [08:08:30] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [08:10:28] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . [08:11:46] !log kevinbazira@deploy1003 helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [08:12:42] (03CR) 10Filippo Giunchedi: [V:03+2 C:03+2] rbd2backy2: Detect missing volumes from rbd stderr (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359781 (https://phabricator.wikimedia.org/T442690) (owner: 10Filippo Giunchedi) [08:14:27] (03PS1) 10Vgutierrez: hieradata: Let lvs3010 handle traffic during reimages [puppet] - 10https://gerrit.wikimedia.org/r/1360844 (https://phabricator.wikimedia.org/T442859) [08:16:15] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360844 (https://phabricator.wikimedia.org/T442859) (owner: 10Vgutierrez) [08:18:54] (03PS1) 10Vgutierrez: hieradata: Depool lvs3009 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1360845 (https://phabricator.wikimedia.org/T401832) [08:19:38] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360845 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [08:19:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12493776 (10phaultfinder) [08:29:19] (03CR) 10Vgutierrez: [C:03+2] hieradata: Let lvs3010 handle traffic during reimages [puppet] - 10https://gerrit.wikimedia.org/r/1360844 (https://phabricator.wikimedia.org/T442859) (owner: 10Vgutierrez) [08:29:44] The deployment server, and SpiderPig will be unavailable for ~2 hours, possibly less, while I move the active server to codfw [08:31:05] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:32:26] (03PS1) 10Novem Linguae: data: add novemlinguae's yubikey [puppet] - 10https://gerrit.wikimedia.org/r/1360846 (https://phabricator.wikimedia.org/T442756) [08:32:36] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3010.*} and A:liberica [08:32:37] !log slyngshede@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet,releases1003.eqiad.wmnet with reason: Deployment server switchover [08:32:54] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3010.*} and A:liberica [08:34:35] (03CR) 10Slyngshede: [C:03+2] wmnet: update deployment CNAME record to deploy2003 [dns] - 10https://gerrit.wikimedia.org/r/1319825 (https://phabricator.wikimedia.org/T436630) (owner: 10Slyngshede) [08:34:48] !log slyngshede@dns1004 START - running authdns-update [08:37:24] !log slyngshede@dns1004 END - running authdns-update [08:39:29] (03CR) 10Slyngshede: [C:03+2] hieradata: update deployment_server to deploy2003 [puppet] - 10https://gerrit.wikimedia.org/r/1319826 (https://phabricator.wikimedia.org/T436630) (owner: 10Slyngshede) [08:43:10] !log Moved deployment.eqiad.wmnet to deploy2003.codfw.wmnet - T436630 [08:43:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:43:16] T436630: Deployment server switch over - https://phabricator.wikimedia.org/T436630 [08:43:44] (03CR) 10Daniel Kertesz: [C:03+1] hieradata: Depool lvs3009 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1360845 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [08:48:27] (03PS2) 10Mvolz: Bump chrome version in user-agent in citoid [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358340 (https://phabricator.wikimedia.org/T442971) [08:49:56] (03CR) 10Vgutierrez: [C:03+2] hieradata: Depool lvs3009 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1360845 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [08:51:09] (03Abandoned) 10Mvolz: Update Citoid's User-Agent to a current Chrome version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360574 (https://phabricator.wikimedia.org/T442971) (owner: 101Veertje) [08:53:29] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3009.*} and A:liberica [08:53:48] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3009.*} and A:liberica [08:57:51] (03Abandoned) 10Mpostoronca: admin: Add FIDO SSH key for mpostoronca [puppet] - 10https://gerrit.wikimedia.org/r/1358758 (https://phabricator.wikimedia.org/T442390) (owner: 10Mpostoronca) [08:58:33] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for mpostoronca - https://phabricator.wikimedia.org/T442390#12493924 (10MPostoronca-WMF) 05In progress→03Invalid Closed because of https://phabricator.wikimedia.org/T442758 [08:59:30] !log slyngshede@deploy2003 Started scap sync-world: Test deployment to validate deployment server switchover - T436630 [08:59:36] T436630: Deployment server switch over - https://phabricator.wikimedia.org/T436630 [09:00:01] !log asw1-bw27-esams# set protocols bgp group PyBal multipath - T401832 [09:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261030T0700) [09:00:05] slyngs: Deployment server switchover (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261030T0900). Please do the needful. [09:00:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:00:06] T401832: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832 [09:02:31] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reimage for host lvs3009.esams.wmnet with OS trixie [09:03:22] (03PS4) 10CWilliams: zarcillo: enabling egress to config-master [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360470 (https://phabricator.wikimedia.org/T442904) [09:03:38] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs3009.esams.wmnet (push config only) [09:03:44] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs3009.esams.wmnet (push config only) [09:07:37] (03PS1) 10Jelto: phabricator: add ensure to phd auto restart [puppet] - 10https://gerrit.wikimedia.org/r/1360848 (https://phabricator.wikimedia.org/T442892) [09:10:12] (03CR) 10CWilliams: [C:03+2] zarcillo: enabling egress to config-master [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360470 (https://phabricator.wikimedia.org/T442904) (owner: 10CWilliams) [09:11:36] (03Merged) 10jenkins-bot: zarcillo: enabling egress to config-master [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360470 (https://phabricator.wikimedia.org/T442904) (owner: 10CWilliams) [09:14:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12494014 (10phaultfinder) [09:14:56] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12494015 (10phaultfinder) [09:20:08] (03CR) 10Lucas Werkmeister (WMDE): "So, uh, was anyone gonna tell us (WMDE) about this? Because we both need these statistics and tried to migrate to the new stats system alr" [puppet] - 10https://gerrit.wikimedia.org/r/1357500 (owner: 10Ladsgroup) [09:20:15] (03PS2) 10Jelto: phabricator: add ensure to phd auto restart [puppet] - 10https://gerrit.wikimedia.org/r/1360848 (https://phabricator.wikimedia.org/T442892) [09:20:42] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [09:21:41] (03CR) 10Jelto: [V:03+1] "PCC SUCCESS (NOOP 2 CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/" [puppet] - 10https://gerrit.wikimedia.org/r/1360848 (https://phabricator.wikimedia.org/T442892) (owner: 10Jelto) [09:22:39] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs3009.esams.wmnet (remove config only) [09:22:45] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs3009.esams.wmnet (remove config only) [09:25:44] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on lvs3009.esams.wmnet with reason: host reimage [09:27:40] (03PS1) 10Vgutierrez: Revert "lvs: Bring high-traffic1 services to lvs3009" [puppet] - 10https://gerrit.wikimedia.org/r/1360851 [09:27:46] (03CR) 10Lucas Werkmeister (WMDE): "Also, I don’t see any remaining code in there that might try to write to Graphite; do you have a copy of any errors you saw? (They’re gone" [puppet] - 10https://gerrit.wikimedia.org/r/1357500 (owner: 10Ladsgroup) [09:28:06] (03CR) 10CI reject: [V:04-1] Revert "lvs: Bring high-traffic1 services to lvs3009" [puppet] - 10https://gerrit.wikimedia.org/r/1360851 (owner: 10Vgutierrez) [09:29:12] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs3009.esams.wmnet with reason: host reimage [09:29:56] (03PS2) 10Vgutierrez: Revert "lvs: Bring high-traffic1 services to lvs3009" [puppet] - 10https://gerrit.wikimedia.org/r/1360851 (https://phabricator.wikimedia.org/T442859) [09:34:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12494081 (10phaultfinder) [09:35:18] !log slyngshede@deploy2003 Finished scap sync-world: Test deployment to validate deployment server switchover - T436630 (duration: 35m 48s) [09:35:24] T436630: Deployment server switch over - https://phabricator.wikimedia.org/T436630 [09:37:09] Deployment server switch over is completed. Please note that deploy2003.codfw.wmnet is now active. Please update your known ssh key as connecting to deployment.eqiad.wmnet will now present the host key for deploy2003, not deploy1003. [09:37:12] 10ops-eqiad, 06SRE, 06DC-Ops, 07Essential-Work, 06Machine-Learning-Team (Q2 FY2026-27): QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12494083 (10isarantopoulos) [09:37:15] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), and 2 others: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12494084 (10isarantopoulos) [09:39:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12494109 (10phaultfinder) [09:45:55] (03Abandoned) 10Ozge: knative-serving: Allow port 8112 in the activator ingress policy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360372 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [09:48:37] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs3009.esams.wmnet with OS trixie [09:49:02] (03CR) 10Daniel Kertesz: [C:03+1] Revert "lvs: Bring high-traffic1 services to lvs3009" [puppet] - 10https://gerrit.wikimedia.org/r/1360851 (https://phabricator.wikimedia.org/T442859) (owner: 10Vgutierrez) [09:49:21] (03CR) 10Vgutierrez: [C:03+2] Revert "lvs: Bring high-traffic1 services to lvs3009" [puppet] - 10https://gerrit.wikimedia.org/r/1360851 (https://phabricator.wikimedia.org/T442859) (owner: 10Vgutierrez) [09:53:25] (03PS1) 10Vgutierrez: Revert "hieradata: Depool lvs3009 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360852 [09:53:41] (03PS2) 10Vgutierrez: Revert "hieradata: Depool lvs3009 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360852 (https://phabricator.wikimedia.org/T401832) [09:53:48] (03CR) 10CI reject: [V:04-1] Revert "hieradata: Depool lvs3009 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360852 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [09:54:16] (03CR) 10CI reject: [V:04-1] Revert "hieradata: Depool lvs3009 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360852 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [09:55:00] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3009.*} and A:liberica [09:55:18] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3009.*} and A:liberica [09:56:46] (03PS3) 10Vgutierrez: Revert "hieradata: Depool lvs3009 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360852 (https://phabricator.wikimedia.org/T401832) [09:56:54] (03CR) 10Ayounsi: [C:04-1] "That change needs PCC checked across the whole infra and not just one server as it's used in a lot of places." [puppet] - 10https://gerrit.wikimedia.org/r/1359145 (https://phabricator.wikimedia.org/T442497) (owner: 10Ayounsi) [09:57:38] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360852 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [09:59:20] (03CR) 10Daniel Kertesz: [C:03+1] Revert "hieradata: Depool lvs3009 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360852 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [09:59:34] (03CR) 10Vgutierrez: [C:03+2] Revert "hieradata: Depool lvs3009 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360852 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [10:01:34] (03CR) 10Ayounsi: "Change LGTM thx, will leave it to Jesse for a final review, please run PCC as well to make sure it does the right things." [puppet] - 10https://gerrit.wikimedia.org/r/1360559 (https://phabricator.wikimedia.org/T442063) (owner: 10BCornwall) [10:04:59] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3009.*} and A:liberica [10:05:17] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3009.*} and A:liberica [10:06:44] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [10:07:21] (03PS1) 10Arthur taylor: termbox: bump to latest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360854 (https://phabricator.wikimedia.org/T442990) [10:08:02] (03PS1) 10Vgutierrez: hieradata: Depool lvs3008 for reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1360855 (https://phabricator.wikimedia.org/T401832) [10:08:36] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360855 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [10:08:51] !log mvernon@cumin2003 START - Cookbook sre.hosts.reboot-single for host sretest2010.codfw.wmnet [10:11:17] (03CR) 10Daniel Kertesz: [C:03+1] hieradata: Depool lvs3008 for reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1360855 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [10:11:42] (03CR) 10Vgutierrez: [C:03+2] hieradata: Depool lvs3008 for reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1360855 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [10:11:44] RESOLVED: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [10:11:46] (03PS1) 10Kevin Bazira: ml-services: raise tts-section-generator mesh upstream timeout [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360856 (https://phabricator.wikimedia.org/T441983) [10:13:46] (03CR) 10Kevin Bazira: [C:03+2] ml-services: raise tts-section-generator mesh upstream timeout [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360856 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [10:14:20] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3008.*} and A:liberica [10:14:38] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3008.*} and A:liberica [10:15:09] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2010.codfw.wmnet [10:15:18] (03Merged) 10jenkins-bot: ml-services: raise tts-section-generator mesh upstream timeout [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360856 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [10:18:22] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [10:19:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12494247 (10phaultfinder) [10:20:14] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reimage for host lvs3008.esams.wmnet with OS trixie [10:20:23] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . [10:20:43] (03PS1) 10Jelto: aptrepo: update gitlab-runner to 19.5 [puppet] - 10https://gerrit.wikimedia.org/r/1360857 (https://phabricator.wikimedia.org/T442992) [10:21:10] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs3008.esams.wmnet (push config only) [10:21:16] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs3008.esams.wmnet (push config only) [10:21:30] !log kevinbazira@deploy1003 helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [10:22:31] (03PS1) 10Vgutierrez: Revert "hieradata: Let lvs3008 handle upload while lvs3010 is down" [puppet] - 10https://gerrit.wikimedia.org/r/1360858 [10:22:33] (03Restored) 10Ozge: knative-serving: Allow port 8112 in the activator ingress policy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360372 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [10:22:57] (03CR) 10CI reject: [V:04-1] Revert "hieradata: Let lvs3008 handle upload while lvs3010 is down" [puppet] - 10https://gerrit.wikimedia.org/r/1360858 (owner: 10Vgutierrez) [10:24:08] (03PS2) 10Vgutierrez: Revert "hieradata: Let lvs3008 handle upload while lvs3010 is down" [puppet] - 10https://gerrit.wikimedia.org/r/1360858 (https://phabricator.wikimedia.org/T442859) [10:26:29] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), and 2 others: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12494305 (10Gehel) [10:28:28] !log cwilliams@deploy2003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [10:29:27] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Q1:rack/setup/install cirrussearch11[26-30] - https://phabricator.wikimedia.org/T436285#12494390 (10Gehel) [10:30:20] 10ops-codfw, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Q1:rack/setup/install cirrussearch21[16-20] - https://phabricator.wikimedia.org/T436287#12494420 (10Gehel) [10:30:48] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Degraded RAID controller on an-presto1015 - https://phabricator.wikimedia.org/T442340#12494430 (10Gehel) [10:31:00] (03PS1) 10Ozge: ml-services: Use for qwen38-27b in llm-batch [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360859 (https://phabricator.wikimedia.org/T442150) [10:32:32] 06SRE, 10Kafka-Infrastructure, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Monitor kafka-configurator runs and pending changes - https://phabricator.wikimedia.org/T439506#12494457 (10Gehel) [10:32:40] (03CR) 10Daniel Kertesz: [C:03+1] Revert "hieradata: Let lvs3008 handle upload while lvs3010 is down" [puppet] - 10https://gerrit.wikimedia.org/r/1360858 (https://phabricator.wikimedia.org/T442859) (owner: 10Vgutierrez) [10:33:19] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Degraded RAID on an-presto1006 - https://phabricator.wikimedia.org/T441831#12494467 (10Gehel) [10:33:37] (03CR) 10Klausman: [C:03+1] knative-serving: Allow port 8112 in the activator ingress policy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360372 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [10:34:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [10:36:08] (03PS8) 10Ayounsi: Add cookbook to sync BGP sessions with Netbox [cookbooks] - 10https://gerrit.wikimedia.org/r/1347441 (https://phabricator.wikimedia.org/T439408) [10:37:42] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs3008.esams.wmnet (remove config only) [10:37:49] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs3008.esams.wmnet (remove config only) [10:40:38] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on lvs3008.esams.wmnet with reason: host reimage [10:40:48] 10SRE-swift-storage, 10Ceph, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Test the Docker Registry's GC with Releng/MediaWiki images - https://phabricator.wikimedia.org/T437297#12494531 (10LSobanski) p:05Medium→03Triage [10:41:44] (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] termbox: bump to latest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360854 (https://phabricator.wikimedia.org/T442990) (owner: 10Arthur taylor) [10:42:59] !log jelto@cumin1004 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: version upgrade [10:44:31] (03CR) 10Vgutierrez: [C:03+2] Revert "hieradata: Let lvs3008 handle upload while lvs3010 is down" [puppet] - 10https://gerrit.wikimedia.org/r/1360858 (https://phabricator.wikimedia.org/T442859) (owner: 10Vgutierrez) [10:44:49] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs3008.esams.wmnet with reason: host reimage [10:47:45] (03PS2) 10Ozge: (DON'T MERGE YET) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360859 (https://phabricator.wikimedia.org/T442150) [10:47:55] (03PS3) 10Ozge: ml-services: Use for qwen38-27b in llm-batch [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360859 (https://phabricator.wikimedia.org/T442150) [10:53:19] !log jelto@cumin1004 END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: version upgrade [10:54:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12494574 (10phaultfinder) [10:54:39] !log jelto@cumin1004 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: version upgrade [10:59:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [11:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261030T0700) [11:00:05] jelto, arnoldokoth, mutante, and arnaudb: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) GitLab version upgrades deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261030T1100). [11:03:54] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs3008.esams.wmnet with OS trixie [11:04:40] !log jelto@cumin1004 END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: version upgrade [11:05:00] (03PS1) 10Vgutierrez: Revert "hieradata: Depool lvs3008 for reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360863 [11:05:13] (03PS2) 10Vgutierrez: Revert "hieradata: Depool lvs3008 for reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360863 (https://phabricator.wikimedia.org/T401832) [11:05:15] !log jelto@cumin1004 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade [11:05:38] (03CR) 10CI reject: [V:04-1] Revert "hieradata: Depool lvs3008 for reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360863 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [11:06:38] (03PS3) 10Vgutierrez: Revert "hieradata: Depool lvs3008 for reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360863 (https://phabricator.wikimedia.org/T401832) [11:08:08] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360863 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [11:09:43] (03CR) 10Kevin Bazira: [C:03+1] ml-services: Use for qwen38-27b in llm-batch [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360859 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [11:10:39] (03CR) 10Daniel Kertesz: [C:03+1] Revert "hieradata: Depool lvs3008 for reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360863 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [11:11:08] (03CR) 10Vgutierrez: [C:03+2] Revert "hieradata: Depool lvs3008 for reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360863 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [11:12:43] (03PS1) 10Vgutierrez: Revert "hieradata: Let lvs3010 handle traffic during reimages" [puppet] - 10https://gerrit.wikimedia.org/r/1360864 [11:12:54] (03PS4) 10Ozge: ml-services: Use llmisvc in llm-batch [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360859 (https://phabricator.wikimedia.org/T442150) [11:13:11] (03PS2) 10Vgutierrez: Revert "hieradata: Let lvs3010 handle traffic during reimages" [puppet] - 10https://gerrit.wikimedia.org/r/1360864 (https://phabricator.wikimedia.org/T442859) [11:13:38] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360864 (https://phabricator.wikimedia.org/T442859) (owner: 10Vgutierrez) [11:14:44] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [11:14:57] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3008.*} and A:liberica [11:15:06] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host wikikube-worker1298.eqiad.wmnet [11:15:16] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3008.*} and A:liberica [11:16:01] (03CR) 10Daniel Kertesz: [C:03+1] Revert "hieradata: Let lvs3010 handle traffic during reimages" [puppet] - 10https://gerrit.wikimedia.org/r/1360864 (https://phabricator.wikimedia.org/T442859) (owner: 10Vgutierrez) [11:16:06] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [11:16:31] (03CR) 10Vgutierrez: [C:03+2] Revert "hieradata: Let lvs3010 handle traffic during reimages" [puppet] - 10https://gerrit.wikimedia.org/r/1360864 (https://phabricator.wikimedia.org/T442859) (owner: 10Vgutierrez) [11:17:39] (03PS1) 10Ilias Sarantopoulos: admin: Remove non-FIDO SSH key for isaranto [puppet] - 10https://gerrit.wikimedia.org/r/1360865 (https://phabricator.wikimedia.org/T442176) [11:18:48] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3010.*} and A:liberica [11:19:07] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3010.*} and A:liberica [11:19:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [11:21:56] (03CR) 10Muehlenhoff: [C:03+2] admin: Remove non-FIDO SSH key for isaranto [puppet] - 10https://gerrit.wikimedia.org/r/1360865 (https://phabricator.wikimedia.org/T442176) (owner: 10Ilias Sarantopoulos) [11:22:07] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1298.eqiad.wmnet [11:24:33] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host wikikube-worker1300.eqiad.wmnet [11:25:08] jelto@cumin1004 jelto: The backup on gitlab1004 is complete, ready to proceed with upgrade. [11:25:38] !log jelto@cumin1004 END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade [11:26:53] (03CR) 10JMeybohm: [C:03+2] kubernetes::node: Fix zone detection logic [puppet] - 10https://gerrit.wikimedia.org/r/1360488 (https://phabricator.wikimedia.org/T442874) (owner: 10JMeybohm) [11:29:02] !log jelto@cumin1004 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade [11:29:09] !log jelto@cumin1004 END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade [11:31:02] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1300.eqiad.wmnet [11:31:05] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1300.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:31:51] (03PS1) 10Giuseppe Lavagetto: deployment_server: temporarily button down access [puppet] - 10https://gerrit.wikimedia.org/r/1360869 (https://phabricator.wikimedia.org/T442994) [11:32:04] !log jelto@cumin1004 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade [11:32:09] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12494654 (10Gehel) [11:32:11] (03CR) 10Clément Goubert: [C:03+1] deployment_server: temporarily button down access [puppet] - 10https://gerrit.wikimedia.org/r/1360869 (https://phabricator.wikimedia.org/T442994) (owner: 10Giuseppe Lavagetto) [11:32:45] 10SRE-SLO, 10observability, 10Wikidata, 06Wikidata Platform Team, and 3 others: Update WDQS SLOs to reflect graph split changes - https://phabricator.wikimedia.org/T393966#12494674 (10Gehel) [11:33:10] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 2 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12494691 (10Gehel) [11:33:23] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for isaranto - https://phabricator.wikimedia.org/T442176#12494693 (10isarantopoulos) Everything works great, and thanks to Moritz we also removed the old key. Thanks a lot! [11:33:31] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1300.eqiad.wmnet [11:33:33] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1300.eqiad.wmnet [11:33:43] (03CR) 10Giuseppe Lavagetto: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9582/co" [puppet] - 10https://gerrit.wikimedia.org/r/1360869 (https://phabricator.wikimedia.org/T442994) (owner: 10Giuseppe Lavagetto) [11:33:58] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1298.eqiad.wmnet [11:34:01] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1298.eqiad.wmnet [11:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:34:43] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Q1:rack/setup/install 10 new ceph nodes - https://phabricator.wikimedia.org/T438213#12494725 (10Gehel) [11:35:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12494751 (10phaultfinder) [11:51:32] jelto@cumin1004 jelto: The backup on gitlab1004 is complete, ready to proceed with upgrade. [11:54:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [11:56:52] !log jelto@cumin1004 END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade [12:00:05] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [12:00:15] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [12:03:09] (03CR) 10Ozge: [C:03+2] ml-services: Use llmisvc in llm-batch [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360859 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [12:04:20] (03CR) 10Kevin Bazira: [C:03+1] ml-services: Deploy revise-tone-task-generator on 64G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360424 (https://phabricator.wikimedia.org/T442877) (owner: 10Gkyziridis) [12:04:37] (03Merged) 10jenkins-bot: ml-services: Use llmisvc in llm-batch [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360859 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [12:04:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12494825 (10phaultfinder) [12:05:03] (03PS1) 10Muehlenhoff: Add FIDO key for ihurbain [puppet] - 10https://gerrit.wikimedia.org/r/1360872 (https://phabricator.wikimedia.org/T442758) [12:06:30] (03CR) 10Jelto: [C:03+2] aptrepo: update gitlab-runner to 19.5 [puppet] - 10https://gerrit.wikimedia.org/r/1360857 (https://phabricator.wikimedia.org/T442992) (owner: 10Jelto) [12:09:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [12:10:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [12:12:01] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for ihurbain [puppet] - 10https://gerrit.wikimedia.org/r/1360872 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [12:13:03] (03PS1) 10FNegri: toolhub: Bump container to 2026-10-30-120558-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360874 (https://phabricator.wikimedia.org/T442900) [12:13:31] !log ozge@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' . [12:13:44] (03CR) 10Thibaut Le Page: [C:03+1] toolhub: Bump container to 2026-10-30-120558-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360874 (https://phabricator.wikimedia.org/T442900) (owner: 10FNegri) [12:14:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12494854 (10phaultfinder) [12:15:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [12:15:27] (03CR) 10FNegri: [C:03+2] toolhub: Bump container to 2026-10-30-120558-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360874 (https://phabricator.wikimedia.org/T442900) (owner: 10FNegri) [12:16:06] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [12:16:50] (03Merged) 10jenkins-bot: toolhub: Bump container to 2026-10-30-120558-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360874 (https://phabricator.wikimedia.org/T442900) (owner: 10FNegri) [12:17:12] 10SRE-SLO, 06SRE Observability (FY2025/2026-Q4): Grafana: deploy grafana-dashboard-reporter-app - https://phabricator.wikimedia.org/T425795#12494859 (10hnowlan) 05Open→03Resolved a:03hnowlan I believe everything here is complete - any outstanding work should be in the parent task. [12:17:28] !log fnegri@deploy2003 helmfile [staging] START helmfile.d/services/toolhub: apply [12:18:33] !log asw1-bw27-esams# delete protocols bgp group PyBal multipath - T401832 [12:18:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:18:39] T401832: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832 [12:18:51] !log fnegri@deploy2003 helmfile [staging] DONE helmfile.d/services/toolhub: apply [12:19:36] !log fnegri@deploy2003 helmfile [eqiad] START helmfile.d/services/toolhub: apply [12:20:08] !log ozge@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'llmisvc' . [12:20:31] !log fnegri@deploy2003 helmfile [eqiad] DONE helmfile.d/services/toolhub: apply [12:20:44] !log ozge@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' . [12:20:48] !log ozge@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'llmisvc' . [12:20:52] !log fnegri@deploy2003 helmfile [codfw] START helmfile.d/services/toolhub: apply [12:21:39] !log fnegri@deploy2003 helmfile [codfw] DONE helmfile.d/services/toolhub: apply [12:24:36] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 02 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355410 (https://phabricator.wikimedia.org/T440832) (owner: 10Mahmoud-abdelsattar) [12:24:54] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 02 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355507 (https://phabricator.wikimedia.org/T440832) (owner: 10Mahmoud-abdelsattar) [12:26:13] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host puppetboard2003.codfw.wmnet [12:29:50] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard2003.codfw.wmnet [12:30:03] (03PS1) 10Hnowlan: arclamp: define arclamp systemd service to control restarts [puppet] - 10https://gerrit.wikimedia.org/r/1360876 (https://phabricator.wikimedia.org/T441510) [12:30:05] 10SRE-tools, 06Infrastructure-Foundations: Update sre.network.tls cookbook to deal with Python datetime warning - https://phabricator.wikimedia.org/T442883#12494904 (10Aklapper) [12:30:40] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host puppetboard1003.eqiad.wmnet [12:34:08] 06SRE, 10SRE-Access-Requests: Add yubikey SSH key for urbanecm - https://phabricator.wikimedia.org/T442901#12494932 (10Urbanecm_WMF) Uploaded key to production for verification purposes: ` [urbanecm@bast6003 ~]$ ls id_ed25519_sk_yubikey_usbc_main.pub [urbanecm@bast6003 ~]$ cat id_ed25519_sk_yubikey_usbc_main.... [12:34:42] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard1003.eqiad.wmnet [12:35:23] (03CR) 10Gkyziridis: [C:03+2] ml-services: Deploy revise-tone-task-generator on 64G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360424 (https://phabricator.wikimedia.org/T442877) (owner: 10Gkyziridis) [12:36:53] (03Merged) 10jenkins-bot: ml-services: Deploy revise-tone-task-generator on 64G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360424 (https://phabricator.wikimedia.org/T442877) (owner: 10Gkyziridis) [12:43:26] !log gkyziridis@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . [12:43:39] !log gkyziridis@deploy2003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . [12:44:03] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user niharika29 [12:44:13] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user niharika29 [12:48:44] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user iflorez [12:50:35] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user iflorez [12:58:36] (03CR) 10Svantje Lilienthal: [C:03+1] Enable sub-references globally [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359782 (https://phabricator.wikimedia.org/T440304) (owner: 10WMDE-Fisch) [12:59:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [12:59:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12495022 (10phaultfinder) [13:04:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:10:22] (03PS1) 10Muehlenhoff: remove legacy key for ihurbain [puppet] - 10https://gerrit.wikimedia.org/r/1360884 (https://phabricator.wikimedia.org/T442758) [13:12:04] (03CR) 10Isabelle Hurbain-Palatin: [C:03+1] remove legacy key for ihurbain [puppet] - 10https://gerrit.wikimedia.org/r/1360884 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [13:12:51] (03CR) 10Muehlenhoff: [C:03+2] remove legacy key for ihurbain [puppet] - 10https://gerrit.wikimedia.org/r/1360884 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [13:14:44] RESOLVED: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:14:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12495112 (10phaultfinder) [13:15:09] (03CR) 10Alex.sanford: [C:03+2] Bump security landing page image version to 2026-10-08-214827 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360557 (owner: 10SBassett) [13:16:52] (03Merged) 10jenkins-bot: Bump security landing page image version to 2026-10-08-214827 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360557 (owner: 10SBassett) [13:17:58] (03PS3) 10Hnowlan: miscweb: add apps.wikipedia.org configuration [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356269 (https://phabricator.wikimedia.org/T440988) [13:19:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [13:19:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12495118 (10phaultfinder) [13:20:42] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [13:30:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:34:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12495165 (10phaultfinder) [13:36:46] (03CR) 10Ladsgroup: [V:03+2 C:03+2] "Sorry, I had a draft message written but got lost during all of the mayhem of the incident. I am sure I saw code related to statsd and I'm" [puppet] - 10https://gerrit.wikimedia.org/r/1357500 (owner: 10Ladsgroup) [13:38:41] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user sguebo [13:38:55] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user sguebo [13:39:04] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris [13:39:13] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris [13:39:34] 06SRE, 10SRE-Access-Requests: Add yubikey SSH key for urbanecm - https://phabricator.wikimedia.org/T442901#12495198 (10Dzahn) [x] verified out-of-band [13:39:45] 06SRE, 10SRE-Access-Requests: Add yubikey SSH key for urbanecm - https://phabricator.wikimedia.org/T442901#12495199 (10Dzahn) 05Open→03In progress [13:39:57] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hokwelum [13:40:06] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hokwelum [13:46:18] (03CR) 10Jelto: [V:03+1 C:03+2] phabricator: add ensure to phd auto restart [puppet] - 10https://gerrit.wikimedia.org/r/1360848 (https://phabricator.wikimedia.org/T442892) (owner: 10Jelto) [13:49:22] (03CR) 10Dzahn: [C:03+1] "thanks! that's what I wanted just ran out of time yesterday" [puppet] - 10https://gerrit.wikimedia.org/r/1360848 (https://phabricator.wikimedia.org/T442892) (owner: 10Jelto) [13:56:13] (03CR) 10Jelto: [C:03+2] spiderpig: Add pageview-trending-relative [puppet] - 10https://gerrit.wikimedia.org/r/1359776 (https://phabricator.wikimedia.org/T442341) (owner: 10JavierMonton) [13:56:17] (03CR) 10Cathal Mooney: "Logic looks good to me. It will include all the networks from the "frack" definition here: https://w.wiki/fPvk" [puppet] - 10https://gerrit.wikimedia.org/r/1360559 (https://phabricator.wikimedia.org/T442063) (owner: 10BCornwall) [13:56:32] (03CR) 10TrainBranchBot: [C:03+2] "Approved by otto@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360516 (https://phabricator.wikimedia.org/T436695) (owner: 10Ottomata) [13:57:18] (03CR) 10Ladsgroup: [V:03+2 C:03+2] "I checked a couple of times and it'll all hit graphite which was made read only for a year now. If you want to follow. The config for stat" [puppet] - 10https://gerrit.wikimedia.org/r/1357500 (owner: 10Ladsgroup) [13:57:40] (03Merged) 10jenkins-bot: EventStreamConfig: Add revision_html_feature_counts_change.dev0 stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360516 (https://phabricator.wikimedia.org/T436695) (owner: 10Ottomata) [13:58:51] !log otto@deploy2003 Started scap sync-world: Backport for [[gerrit:1360516|EventStreamConfig: Add revision_html_feature_counts_change.dev0 stream (T436695)]] [13:58:58] T436695: Linked Artifact Cache - Automated MediaWiki JobQueue prefetch - https://phabricator.wikimedia.org/T436695 [14:00:34] 06SRE, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10MediaWiki-extensions-CentralNotice, 06Traffic: Set expiry time for GeoIP cookies - https://phabricator.wikimedia.org/T122097#12495291 (10SBurnett-WMF) @ssingh - would Q3 work for this? [14:04:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12495301 (10phaultfinder) [14:06:03] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for xcollazo - https://phabricator.wikimedia.org/T442741#12495302 (10xcollazo) >>! In T442741#12493090, @Dzahn wrote: > @xcollazo Please give it up to 30 min to propagate. Then let us know if it works and when we can remove the old key. Thanks @Dzahn. I h... [14:14:49] !log cmooney@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1240.eqiad.wmnet [14:14:49] !log depool wikikube-worker1240 to trial 1G -> 10G migration process for eqiad A/B wikikubes T440696 [14:14:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:14:58] T440696: Eqiad worker nodes should be connected to 10G NICs - https://phabricator.wikimedia.org/T440696 [14:15:40] !log cmooney@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1240.eqiad.wmnet [14:15:46] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Eqiad worker nodes should be connected to 10G NICs - https://phabricator.wikimedia.org/T440696#12495341 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by cmooney@cumin1004 depool for host w... [14:18:05] (03CR) 10Ladsgroup: [V:03+2 C:03+2] "The removal of DNS record was reverted cause it cause issues in restbase, that's why it still resolves but it will start triggering fatals" [puppet] - 10https://gerrit.wikimedia.org/r/1357500 (owner: 10Ladsgroup) [14:20:37] (03CR) 10Jelto: [C:03+2] Add Test Kitchen service to SpiderPig (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1357383 (https://phabricator.wikimedia.org/T442889) (owner: 10Clare Ming) [14:21:28] !log otto@deploy2003 otto: Backport for [[gerrit:1360516|EventStreamConfig: Add revision_html_feature_counts_change.dev0 stream (T436695)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:21:33] T436695: Linked Artifact Cache - Automated MediaWiki JobQueue prefetch - https://phabricator.wikimedia.org/T436695 [14:22:26] (03PS4) 10Rsilvola: admin: replace rsilvola SSH key for a hw backed SSH key [puppet] - 10https://gerrit.wikimedia.org/r/1358237 (https://phabricator.wikimedia.org/T442092) [14:23:12] !log otto@deploy2003 otto: Continuing with deployment [14:23:33] (03CR) 10Rsilvola: "Replaced with a new hw key for which I have an attestation." [puppet] - 10https://gerrit.wikimedia.org/r/1358237 (https://phabricator.wikimedia.org/T442092) (owner: 10Rsilvola) [14:24:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12495373 (10phaultfinder) [14:28:56] 06SRE, 10SRE-Access-Requests, 06Product Safety and Integrity, 06Security-Team, 13Patch-For-Review: Replace SSH key with hw backed SSH keys for rsilvola - https://phabricator.wikimedia.org/T442092#12495387 (10Rsilvola) I've uploaded the new patch set on gerrit and uploaded the challenge and attestation to... [14:29:24] (03CR) 10Aklapper: [V:03+2 C:03+2] "Thank you, applies cleanly here" [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1357551 (owner: 10Pppery) [14:34:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12495399 (10phaultfinder) [14:34:44] 06SRE, 10SRE-Access-Requests, 06Data-Engineering: Requesting Kerberos identity for rsilvola - https://phabricator.wikimedia.org/T442695#12495400 (10Rsilvola) Done. Thanks! [14:35:39] !log otto@deploy2003 Finished scap sync-world: Backport for [[gerrit:1360516|EventStreamConfig: Add revision_html_feature_counts_change.dev0 stream (T436695)]] (duration: 36m 47s) [14:35:45] T436695: Linked Artifact Cache - Automated MediaWiki JobQueue prefetch - https://phabricator.wikimedia.org/T436695 [14:36:23] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-b8-codfw.mgmt.codfw.wmnet is going to expire in 6d 23h 54m 7s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:40:06] (03PS1) 10Kevin Bazira: ml-services: stop envoy cutting off long tts sections at 5 minutes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360919 (https://phabricator.wikimedia.org/T441983) [14:41:22] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-b2-codfw.mgmt.codfw.wmnet is going to expire in 6d 23h 54m 7s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:42:09] (03CR) 10Kevin Bazira: [C:03+2] ml-services: stop envoy cutting off long tts sections at 5 minutes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360919 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [14:43:37] (03Merged) 10jenkins-bot: ml-services: stop envoy cutting off long tts sections at 5 minutes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360919 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [14:43:46] !log cmooney@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1240.eqiad.wmnet [14:43:48] !log cmooney@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1240.eqiad.wmnet [14:43:54] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Eqiad worker nodes should be connected to 10G NICs - https://phabricator.wikimedia.org/T440696#12495436 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by cmooney@cumin1004 pool for host wik... [14:44:47] !log kevinbazira@deploy2003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [14:46:24] !log kevinbazira@deploy2003 helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . [14:47:15] !log kevinbazira@deploy2003 helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [14:47:56] (03CR) 10Vgutierrez: [C:03+1] wikimedia.org: add CNAME for apps microsite [dns] - 10https://gerrit.wikimedia.org/r/1356256 (https://phabricator.wikimedia.org/T440988) (owner: 10Hnowlan) [14:54:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12495511 (10phaultfinder) [14:57:28] 06SRE, 10SRE-Access-Requests, 06Product Safety and Integrity, 06Security-Team, 07SecTeam-Processed: Request IRC #mediawiki_security access for Novem Linguae - https://phabricator.wikimedia.org/T442747#12495533 (10Rscout) a:03sbassett [14:59:36] 06SRE, 10SRE-Access-Requests, 06Product Safety and Integrity, 06Security-Team, 07SecTeam-Processed: Request IRC #mediawiki_security access for Novem Linguae - https://phabricator.wikimedia.org/T442747#12495542 (10Rscout) @sbassett I think helped me set up mine and seems to have worked with you before. I'... [15:00:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [15:01:22] FIRING: [7x] CertAlmostExpired: gNMI TLS certificate for lsw1-a4-codfw.mgmt.codfw.wmnet is going to expire in 6d 23h 54m 7s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:02:58] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [15:03:05] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [15:03:58] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Degraded RAID controller on an-presto1015 - https://phabricator.wikimedia.org/T442340#12495584 (10Jclark-ctr) 05Open→03Resolved [15:04:37] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [15:04:56] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [15:06:22] FIRING: [8x] CertAlmostExpired: gNMI TLS certificate for lsw1-a4-codfw.mgmt.codfw.wmnet is going to expire in 6d 23h 50m 7s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:07:59] (03PS1) 10DCausse: Search: add dedicated poolcounter for WDQS search use-cases [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360927 (https://phabricator.wikimedia.org/T442824) [15:09:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12495633 (10phaultfinder) [15:10:15] (03PS1) 10Zabe: Disable automatic user page watching on loginwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360928 (https://phabricator.wikimedia.org/T442779) [15:11:46] (03PS2) 10Zabe: Disable automatic user page watching on loginwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360928 (https://phabricator.wikimedia.org/T442779) [15:14:30] !log javiermonton@deploy2003 Started scap deploy-service pageview-trending-relative-next: Test spiderpig with Flink checks. This is a staging release. [15:14:30] !log javiermonton@deploy2003 scap deploy-service pageview-trending-relative-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying pageview-trending-relative-next [15:16:48] !log javiermonton@deploy2003 Finished scap deploy-service pageview-trending-relative-next: Test spiderpig with Flink checks. This is a staging release. (duration: 02m 17s) [15:20:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [15:29:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12495705 (10phaultfinder) [15:31:05] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [15:34:26] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:34:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12495736 (10phaultfinder) [15:36:19] (03CR) 10JMeybohm: sre.k8s: add upgrade-kubelets-inplace cookbook (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1346036 (https://phabricator.wikimedia.org/T437480) (owner: 10Jelto) [15:39:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12495754 (10phaultfinder) [15:54:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12495828 (10phaultfinder) [15:54:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12495829 (10phaultfinder) [15:55:03] 06SRE, 10SRE-Access-Requests, 06Product Safety and Integrity, 06Security-Team, 13Patch-For-Review: Replace SSH key with hw backed SSH keys for rsilvola - https://phabricator.wikimedia.org/T442092#12495830 (10Rsilvola) a:05Rsilvola→03None [16:01:39] !log otto@deploy2003 helmfile [staging] START helmfile.d/services/eventgate-main: sync [16:01:48] !log otto@deploy2003 helmfile [staging] DONE helmfile.d/services/eventgate-main: sync [16:02:01] !log roll restarting eventgate main to pick up EventStreamConfig: Add revision_html_feature_counts_change.dev0 stream (operations/mediawiki-config~1360516) [16:02:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:02:06] (03PS1) 10JavierMonton: spiderpig: add mediawiki-event-enrichment deployments [puppet] - 10https://gerrit.wikimedia.org/r/1360938 (https://phabricator.wikimedia.org/T442341) [16:02:08] !log otto@deploy2003 helmfile [eqiad] START helmfile.d/services/eventgate-main: sync [16:02:33] !log otto@deploy2003 helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync [16:02:57] !log otto@deploy2003 helmfile [codfw] START helmfile.d/services/eventgate-main: sync [16:03:19] !log otto@deploy2003 helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync [16:05:42] (03PS3) 10JMeybohm: sre.k8s: Add cookbook for apiserver inplace upgrades [cookbooks] - 10https://gerrit.wikimedia.org/r/1358223 (https://phabricator.wikimedia.org/T437480) [16:05:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [16:06:00] (03CR) 10JMeybohm: sre.k8s: Add cookbook for apiserver inplace upgrades (034 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1358223 (https://phabricator.wikimedia.org/T437480) (owner: 10JMeybohm) [16:08:15] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-b8-codfw [16:08:26] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b8-codfw [16:08:28] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-b6-codfw [16:08:40] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b6-codfw [16:08:42] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-b2-codfw [16:08:54] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b2-codfw [16:08:56] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a4-codfw [16:09:08] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a4-codfw [16:09:10] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device ssw1-f1-eqiad [16:09:17] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-f1-eqiad [16:09:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12495918 (10phaultfinder) [16:11:17] (03CR) 10EMcFarland: Home Experience: Feature flag to enable Home (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352515 (https://phabricator.wikimedia.org/T439738) (owner: 10EMcFarland) [16:11:22] FIRING: [8x] CertAlmostExpired: gNMI TLS certificate for lsw1-a4-codfw.mgmt.codfw.wmnet is going to expire in 6d 22h 46m 7s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:15:42] RESOLVED: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [16:16:19] (03PS1) 10Gmodena: wdqs: version bump proxy 0.14.2 -> 0.14.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360941 (https://phabricator.wikimedia.org/T442744) [16:17:02] (03PS1) 10Bking: flink: bump image version to pick up changes from java image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360942 (https://phabricator.wikimedia.org/T443032) [16:17:56] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [16:19:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12495964 (10phaultfinder) [16:20:30] (03PS1) 10Gmodena: wdqs: version bump health container 0.0.2 -> 0.0.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360943 (https://phabricator.wikimedia.org/T442744) [16:20:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [16:27:26] (03PS1) 10Gmodena: wdqs: version bump streaming consumer 0.3.1 -> 0.4.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360944 (https://phabricator.wikimedia.org/T442744) [16:31:40] (03CR) 10Gmodena: wdqs: version bump streaming consumer 0.3.1 -> 0.4.0 (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360944 (https://phabricator.wikimedia.org/T442744) (owner: 10Gmodena) [16:33:47] (03CR) 10Atsuko: [C:03+1] flink: bump image version to pick up changes from java image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360942 (https://phabricator.wikimedia.org/T443032) (owner: 10Bking) [16:36:03] (03CR) 10Gmodena: [C:03+1] flink: bump image version to pick up changes from java image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360942 (https://phabricator.wikimedia.org/T443032) (owner: 10Bking) [16:37:03] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for snowick - https://phabricator.wikimedia.org/T442951#12496052 (10SNowick_WMF) Hi Have the hardware and have generated keys, I have uploaded the the challenge and attestation files using the form from the Wikitech page. Attached is my public key. {F1114... [16:40:07] 10SRE-Access-Requests: Add FIDO-backed SSH keys for mgerlach - https://phabricator.wikimedia.org/T443038 (10MGerlach) 03NEW [16:46:23] FIRING: [5x] CertAlmostExpired: gNMI TLS certificate for lsw1-a5-codfw.mgmt.codfw.wmnet is going to expire in 6d 22h 10m 7s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:48:13] hey ops - volans, federico3, bliviero -- i'm about to start parsoid RT testing again. Last time the parser cache melted down just *before* we started RT testing, and I think we established that our RT testing was just a coincidence. But I'll be watching here for any sign of parser cache unhappiness. [16:49:13] OK, good luck [16:49:22] cscott: ack, please don't melt parser cache if possible, it's friday evening around here ... :D [16:49:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12496143 (10phaultfinder) [16:49:53] yeah, we're hoping to get a good test run in so we've got something to release on monday :) [16:51:23] FIRING: [6x] CertAlmostExpired: gNMI TLS certificate for lsw1-a5-codfw.mgmt.codfw.wmnet is going to expire in 6d 22h 4m 7s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:54:14] 10SRE-Access-Requests: Add FIDO-backed SSH keys for aranyap - https://phabricator.wikimedia.org/T443041 (10aranyap) 03NEW [16:54:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12496177 (10phaultfinder) [16:55:27] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12496181 (10cfoster) @Dzahn bumping this - would love to get started on some data analysis today or early nex... [16:56:03] 10ops-codfw, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: wikikube-worker2280 unreachable - https://phabricator.wikimedia.org/T423395#12496185 (10JMeybohm) a:05JMeybohm→03Jhancock.wm [16:56:23] FIRING: [8x] CertAlmostExpired: gNMI TLS certificate for lsw1-a5-codfw.mgmt.codfw.wmnet is going to expire in 6d 22h 0m 7s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [17:00:07] (03CR) 10Andrew Bogott: [C:03+2] Add profile for backing up capi worker nodes [puppet] - 10https://gerrit.wikimedia.org/r/1355194 (https://phabricator.wikimedia.org/T440363) (owner: 10Andrew Bogott) [17:01:23] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-a5-codfw.mgmt.codfw.wmnet is going to expire in 6d 21h 54m 7s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [17:05:44] FIRING: [3x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [17:12:40] (03PS1) 10Majavah: P:wmcs::novaproxy: Allow loading error page assets on disabled hosts [puppet] - 10https://gerrit.wikimedia.org/r/1360950 (https://phabricator.wikimedia.org/T442870) [17:14:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12496276 (10phaultfinder) [17:14:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12496277 (10phaultfinder) [17:15:02] (03CR) 10FNegri: [C:03+1] P:wmcs::novaproxy: Allow loading error page assets on disabled hosts [puppet] - 10https://gerrit.wikimedia.org/r/1360950 (https://phabricator.wikimedia.org/T442870) (owner: 10Majavah) [17:15:31] (03CR) 10Majavah: [C:03+2] P:wmcs::novaproxy: Allow loading error page assets on disabled hosts [puppet] - 10https://gerrit.wikimedia.org/r/1360950 (https://phabricator.wikimedia.org/T442870) (owner: 10Majavah) [17:19:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12496322 (10phaultfinder) [17:19:56] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12496323 (10phaultfinder) [17:24:24] (03CR) 10BCornwall: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9584/co" [puppet] - 10https://gerrit.wikimedia.org/r/1360559 (https://phabricator.wikimedia.org/T442063) (owner: 10BCornwall) [17:25:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [17:29:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12496361 (10phaultfinder) [17:30:45] (03PS1) 10Gmodena: WIP: wdqs2 alerts [alerts] - 10https://gerrit.wikimedia.org/r/1360953 (https://phabricator.wikimedia.org/T436018) [17:40:38] (03PS2) 10Dzahn: admin: create user for Corbin Foster, analytics-privatedata [puppet] - 10https://gerrit.wikimedia.org/r/1359965 (https://phabricator.wikimedia.org/T442648) [17:41:20] (03CR) 10Dzahn: [C:03+2] "moving forward with just the part that is already approved, skipping analytics-product-users for later" [puppet] - 10https://gerrit.wikimedia.org/r/1359965 (https://phabricator.wikimedia.org/T442648) (owner: 10Dzahn) [17:44:34] 10ops-eqiad, 06SRE, 06DC-Ops, 10ServiceOps-Upgrades-Hardware, 06ServiceOps (Next quarter): Q1:rack/setup/install conf101[0-2] - https://phabricator.wikimedia.org/T435426#12496392 (10Jclark-ctr) [17:45:29] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install puppetserver100[45] and ganeti106[0-4] - https://phabricator.wikimedia.org/T434699#12496397 (10Jclark-ctr) [17:46:17] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Q1:rack/setup/install cirrussearch11[26-30] - https://phabricator.wikimedia.org/T436285#12496398 (10Jclark-ctr) a:03Jclark-ctr [17:46:22] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Q1:rack/setup/install cirrussearch11[26-30] - https://phabricator.wikimedia.org/T436285#12496402 (10Jclark-ctr) These have arrived on site [17:48:22] (03CR) 10Dzahn: [C:03+1] aptrepo: update gitlab-runner to 19.5 [puppet] - 10https://gerrit.wikimedia.org/r/1360857 (https://phabricator.wikimedia.org/T442992) (owner: 10Jelto) [17:51:21] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12496420 (10Dzahn) Hi @cfoster sorry, busy days over here. Here you go! Please keep an eye out for an em... [17:52:52] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for aranyap - https://phabricator.wikimedia.org/T443041#12496424 (10Dzahn) Hi @aranyap do you already have a Yubikey or do you first need hardware? https://office.wikimedia.org/wiki/Security/Training/Staff_Basics#YubiKey_Security_Keys Best, Daniel [17:54:32] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for mgerlach - https://phabricator.wikimedia.org/T443038#12496431 (10Dzahn) I sent an email for out-of-band verification. [17:55:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [17:59:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12496438 (10phaultfinder) [18:03:27] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for snowick - https://phabricator.wikimedia.org/T442951#12496443 (10Dzahn) I'm sorry, I have no information on challenge and attestation or a new form. But it's ok, I sent you an email to verify the new key out-of-band. [18:03:50] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for snowick - https://phabricator.wikimedia.org/T442951#12496444 (10Dzahn) 05Open→03In progress [18:06:55] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for mpostoronca - https://phabricator.wikimedia.org/T442390#12496458 (10Dzahn) You didn't have to close it. It's ok to keep it open but stalled until you have actually acquired a Yubikey. But up to you! Can also be a new one later. Ok either way. htt... [18:10:07] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for isaranto - https://phabricator.wikimedia.org/T442176#12496462 (10Dzahn) 05In progress→03Resolved a:03Dzahn Great! Closing as resolved. [18:11:53] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for kevinbazira - https://phabricator.wikimedia.org/T442865#12496467 (10Dzahn) 05In progress→03Resolved a:03Dzahn Thanks! I see the patch to remove is already merged as well. closing out [18:12:58] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for aranyap - https://phabricator.wikimedia.org/T443041#12496470 (10aranyap) Hi @Dzahn I currently have a YubiKey 5C NFC [18:14:10] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for xcollazo - https://phabricator.wikimedia.org/T442741#12496471 (10Dzahn) 05In progress→03Stalled Alright, sure, sounds good to me! I'll just set this to stalled for now. It's fine. [18:14:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12496473 (10phaultfinder) [18:19:31] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for kemayo - https://phabricator.wikimedia.org/T442902#12496480 (10Dzahn) ACK! Sorry, too many tickets, my bad. Verifying out-of-band. [18:19:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12496482 (10phaultfinder) [18:21:57] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-b5-codfw [18:22:09] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b5-codfw [18:22:12] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a5-codfw [18:22:23] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a5-codfw [18:25:03] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for aikochou - https://phabricator.wikimedia.org/T442367#12496497 (10Dzahn) Hello @achou The " Secure Enclave" docs have been removed meanwhile. Are the keys in your patch Yubikey key nows unrelated to that process? Cheers Daniel [18:25:34] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for aikochou - https://phabricator.wikimedia.org/T442367#12496498 (10Dzahn) verifying out-of-band [18:26:23] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-a5-codfw.mgmt.codfw.wmnet is going to expire in 6d 20h 32m 7s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [18:26:31] (03CR) 10Dzahn: [V:03+1] "verified new key out-of-band" [puppet] - 10https://gerrit.wikimedia.org/r/1358237 (https://phabricator.wikimedia.org/T442092) (owner: 10Rsilvola) [18:26:56] (03CR) 10Dzahn: [V:03+1 C:03+2] admin: replace rsilvola SSH key for a hw backed SSH key [puppet] - 10https://gerrit.wikimedia.org/r/1358237 (https://phabricator.wikimedia.org/T442092) (owner: 10Rsilvola) [18:26:56] (03PS1) 10Volans: wmcs root keys: add FIDO keys for volans [puppet] - 10https://gerrit.wikimedia.org/r/1360965 (https://phabricator.wikimedia.org/T443002) [18:28:07] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a6-codfw [18:28:19] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a6-codfw [18:29:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12496522 (10phaultfinder) [18:31:23] RESOLVED: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-a5-codfw.mgmt.codfw.wmnet is going to expire in 6d 20h 32m 7s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [18:31:46] 06SRE, 10SRE-Access-Requests, 06Product Safety and Integrity, 06Security-Team, 13Patch-For-Review: Replace SSH key with hw backed SSH keys for rsilvola - https://phabricator.wikimedia.org/T442092#12496528 (10Dzahn) new key verified out-of-band patch deployed [18:31:52] 06SRE, 10SRE-Access-Requests, 06Product Safety and Integrity, 06Security-Team: Replace SSH key with hw backed SSH keys for rsilvola - https://phabricator.wikimedia.org/T442092#12496529 (10Dzahn) 05In progress→03Resolved a:03Dzahn [18:39:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12496558 (10phaultfinder) [18:42:41] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for aranyap - https://phabricator.wikimedia.org/T443041#12496579 (10Dzahn) Hi @aranyap great! Once you have generated a new SSH key using it please paste the public part here on the ticket. Cheers [18:44:25] (03CR) 10Dzahn: [V:03+1 C:03+2] "verified out-of-band" [puppet] - 10https://gerrit.wikimedia.org/r/1360467 (https://phabricator.wikimedia.org/T442902) (owner: 10DLynch) [18:45:24] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2013.codfw.wmnet, wdqs2021.codfw.wmnet, wdqs2014.codfw.wmnet, wdqs2008.codfw.wmnet, wdqs2022.codfw.wmnet, wdqs2010.codfw.wmnet, wdqs2011.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [18:45:46] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2013.codfw.wmnet, wdqs2007.codfw.wmnet, wdqs2010.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [18:46:24] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:46:34] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for kemayo - https://phabricator.wikimedia.org/T442902#12496589 (10Dzahn) New key deployed. Please give it max 30 min and it should all work with the new key. Let us know once you are ready to remove the old key. [18:46:41] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for kemayo - https://phabricator.wikimedia.org/T442902#12496593 (10Dzahn) 05Open→03In progress [18:46:46] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:49:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:49:46] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2013.codfw.wmnet, wdqs2015.codfw.wmnet, wdqs2007.codfw.wmnet, wdqs2022.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [18:50:23] !log cmooney@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1250.eqiad.wmnet [18:50:24] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2012.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [18:50:48] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:50:59] !log cmooney@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1250.eqiad.wmnet [18:51:06] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Eqiad worker nodes should be connected to 10G NICs - https://phabricator.wikimedia.org/T440696#12496615 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by cmooney@cumin1004 depool for host w... [18:51:24] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:51:50] (03PS1) 10Dzahn: admin: adding yubikey key for urbanecm [puppet] - 10https://gerrit.wikimedia.org/r/1360968 (https://phabricator.wikimedia.org/T442901) [18:52:05] (03PS2) 10Dzahn: admin: add yubikey key for urbanecm [puppet] - 10https://gerrit.wikimedia.org/r/1360968 (https://phabricator.wikimedia.org/T442901) [18:52:36] (03CR) 10Dzahn: [V:03+1 C:03+2] admin: add yubikey key for urbanecm [puppet] - 10https://gerrit.wikimedia.org/r/1360968 (https://phabricator.wikimedia.org/T442901) (owner: 10Dzahn) [18:54:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:54:38] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add yubikey SSH key for urbanecm - https://phabricator.wikimedia.org/T442901#12496636 (10Dzahn) yubikey added and deployed. please give it max 30 minutes and it should work on all relevant machines. Let us know once you are ready to remove the old key. [18:56:28] (03PS1) 10Dzahn: admin: add yubikey key for mgerlach [puppet] - 10https://gerrit.wikimedia.org/r/1360970 (https://phabricator.wikimedia.org/T443038) [18:57:21] (03CR) 10Dzahn: [V:03+1] "verified" [puppet] - 10https://gerrit.wikimedia.org/r/1360970 (https://phabricator.wikimedia.org/T443038) (owner: 10Dzahn) [19:00:18] (03CR) 10Dzahn: [V:03+1 C:03+2] admin: add yubikey key for mgerlach [puppet] - 10https://gerrit.wikimedia.org/r/1360970 (https://phabricator.wikimedia.org/T443038) (owner: 10Dzahn) [19:01:22] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for mgerlach - https://phabricator.wikimedia.org/T443038#12496676 (10Dzahn) yubikey added and deployed! Please give it max 30 minutes and it should work on all relevant machines. Let us know once you are ready to remove the old key. [19:08:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:13:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:16:27] 06SRE, 10LDAP-Access-Requests: Grant Access to Logstash for SomeRandomDeveloper - https://phabricator.wikimedia.org/T442855#12496779 (10Dzahn) Hi! No worries, we can add you with realname "known to legal" so it won't become public. There is precedence for that. But Legal itself needs to know it and go throug... [19:18:07] 06SRE, 10LDAP-Access-Requests: Grant Access to Logstash for SomeRandomDeveloper - https://phabricator.wikimedia.org/T442855#12496782 (10Urbanecm) @DZahn Just noting there should already be an NDA here (they already have #acl_security). Not sure if there needs to be a separate/new one? [19:21:34] (03PS1) 10Urbanecm: zookeeper: Add slf4j and log4j jars to the CLASSPATH on bullseye [puppet] - 10https://gerrit.wikimedia.org/r/1360972 (https://phabricator.wikimedia.org/T436481) [19:22:40] (03CR) 10CI reject: [V:04-1] zookeeper: Add slf4j and log4j jars to the CLASSPATH on bullseye [puppet] - 10https://gerrit.wikimedia.org/r/1360972 (https://phabricator.wikimedia.org/T436481) (owner: 10Urbanecm) [19:24:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12496788 (10phaultfinder) [19:28:45] (03CR) 10Bking: [C:03+2] flink: bump image version to pick up changes from java image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360942 (https://phabricator.wikimedia.org/T443032) (owner: 10Bking) [19:28:47] (03CR) 10Bking: [V:03+2 C:03+2] flink: bump image version to pick up changes from java image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360942 (https://phabricator.wikimedia.org/T443032) (owner: 10Bking) [19:29:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12496800 (10phaultfinder) [19:31:05] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [19:32:28] (03PS1) 10Xcollazo: data_purge: Drop raw HDFS fsimage files older than 56 days [puppet] - 10https://gerrit.wikimedia.org/r/1360975 (https://phabricator.wikimedia.org/T438056) [19:33:38] (03Abandoned) 10Dzahn: admin: add new hardware backed key for osefu [puppet] - 10https://gerrit.wikimedia.org/r/1358308 (https://phabricator.wikimedia.org/T442183) (owner: 10Dzahn) [19:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:40:20] 06SRE, 10LDAP-Access-Requests: Grant Access to Logstash for SomeRandomDeveloper - https://phabricator.wikimedia.org/T442855#12496815 (10Dzahn) @Urbanecm I don't know. The entire history of https://phabricator.wikimedia.org/project/manage/30/ does not seem to show me when they were added. I do see T428003 but... [19:41:35] 06SRE, 10LDAP-Access-Requests: Grant Access to Logstash for SomeRandomDeveloper - https://phabricator.wikimedia.org/T442855#12496816 (10SomeRandomDeveloper) >>! In T442855#12496815, @Dzahn wrote: > @Urbanecm I don't know. The entire history of https://phabricator.wikimedia.org/project/manage/30/ does not seem... [19:41:49] 06SRE, 10LDAP-Access-Requests: Grant Access to Logstash for SomeRandomDeveloper - https://phabricator.wikimedia.org/T442855#12496818 (10Urbanecm) >>! In T442855#12496815, @Dzahn wrote: > @Urbanecm I don't know. The entire history of https://phabricator.wikimedia.org/project/manage/30/ does not seem to show me... [19:42:37] (03CR) 10Krinkle: [C:04-1] "Does not seem to work." [puppet] - 10https://gerrit.wikimedia.org/r/1360876 (https://phabricator.wikimedia.org/T441510) (owner: 10Hnowlan) [19:42:58] 06SRE, 10LDAP-Access-Requests: Grant Access to Logstash for SomeRandomDeveloper - https://phabricator.wikimedia.org/T442855#12496823 (10Dzahn) Sorry that it's so complicated but I have no relation to the process that gives access to security tickets. [19:43:03] 06SRE, 10SRE-Access-Requests: Add yubikey SSH key for urbanecm - https://phabricator.wikimedia.org/T442901#12496824 (10Urbanecm_WMF) @Dzahn Confirmed it works, old keys now can be removed. I'll eventually(TM) add a backup key, but that can wait. Thank you! [19:43:43] 06SRE, 10LDAP-Access-Requests: Grant Access to Logstash for SomeRandomDeveloper - https://phabricator.wikimedia.org/T442855#12496825 (10Dzahn) I left a comment on the old ticket to ask about the spreadsheet. [19:44:24] (03PS1) 10Dzahn: admin: remove old keys for urbanecm [puppet] - 10https://gerrit.wikimedia.org/r/1360978 (https://phabricator.wikimedia.org/T442901) [19:44:44] !log bking@build2004 docker-pkg -c /etc/production-images/config.yaml build images/ --select '*flink* T443032 [19:44:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:44:50] T443032: Build/deploy new Flink OCI (docker) image - https://phabricator.wikimedia.org/T443032 [19:44:54] (03PS2) 10Dzahn: admin: remove old keys for urbanecm [puppet] - 10https://gerrit.wikimedia.org/r/1360978 (https://phabricator.wikimedia.org/T442901) [19:45:24] (03CR) 10Dzahn: [V:03+1 C:03+2] "https://phabricator.wikimedia.org/T442901#12496824" [puppet] - 10https://gerrit.wikimedia.org/r/1360978 (https://phabricator.wikimedia.org/T442901) (owner: 10Dzahn) [19:47:32] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add yubikey SSH key for urbanecm - https://phabricator.wikimedia.org/T442901#12496839 (10Dzahn) 05In progress→03Resolved a:03Dzahn Thanks for confirming. Removing old keys. This should be done. [19:57:24] 06SRE, 06Collaboration-Services, 06SRE Observability: create a new place for prometheus/alertmanager checks not tied to physical machines - https://phabricator.wikimedia.org/T397264#12496873 (10Dzahn) We were affected by this again over at T361090#12488647 [20:05:52] 06SRE, 10SRE-Access-Requests, 06Product Safety and Integrity, 06Security-Team, 07SecTeam-Processed: Request IRC #mediawiki_security access for Novem Linguae - https://phabricator.wikimedia.org/T442747#12496907 (10Novem_Linguae) >>! In T442747#12495542, @Rscout wrote: > Do make sure you have registered yo... [20:29:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12496973 (10phaultfinder) [20:30:10] (03PS3) 10Ladsgroup-claude: Clean up temp files when a request fails part way [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1357427 (https://phabricator.wikimedia.org/T431768) [20:30:15] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for aranyap - https://phabricator.wikimedia.org/T443041#12496974 (10aranyap) @Dzahn The public key is sk-ssh-ed25519@openssh.com AAAAGnNrLXNzaC1lZDI1NTE5QG9wZW5zc2guY29tAAAAIIk/2bICCtJx1vCapmcc6J7vzJv3xLKXWnGKzX2qcDOJAAAABHNzaDo= primary Thank you! [20:31:52] (03CR) 10Ladsgroup: "recheck" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1357427 (https://phabricator.wikimedia.org/T431768) (owner: 10Ladsgroup-claude) [20:36:31] (03CR) 10Krinkle: "If I do just a service restart via Scap, then it does work." [puppet] - 10https://gerrit.wikimedia.org/r/1360876 (https://phabricator.wikimedia.org/T441510) (owner: 10Hnowlan) [20:37:02] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064 (10cmooney) 03NEW p:05Triage→03High [20:37:41] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12497024 (10cmooney) [20:37:43] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Eqiad worker nodes should be connected to 10G NICs - https://phabricator.wikimedia.org/T440696#12497025 (10cmooney) [20:38:34] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12497027 (10cmooney) [20:43:47] (03CR) 10Krinkle: [C:03+1] "The lack of restart is consistent. I think on beta it's because I'm doing no-op deploys and Scap knows the code revision hasn't changed." [puppet] - 10https://gerrit.wikimedia.org/r/1360876 (https://phabricator.wikimedia.org/T441510) (owner: 10Hnowlan) [20:44:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12497082 (10phaultfinder) [20:44:45] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12497083 (10cmooney) @JMeybohm I've done this now for wikikube-worker1250 and things seem to be ok. I have lef... [20:44:49] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12497094 (10cmooney) @VRiley-WMF when you are able to if we can get the 10G DACs connected for the 9 hosts in r... [20:48:08] (03Abandoned) 10Catrope: Drop $wgOATHUserHandlesTable [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1248663 (https://phabricator.wikimedia.org/T416544) (owner: 10Catrope) [20:49:18] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12497109 (10cmooney) [20:49:50] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy2003 using scap backport" [extensions/PageImages] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358456 (https://phabricator.wikimedia.org/T441253) (owner: 10Krinkle) [20:52:10] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12497126 (10cmooney) [20:55:38] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12497150 (10cmooney) [20:57:09] (03Merged) 10jenkins-bot: Batch file retrievals for search result thumbnails [extensions/PageImages] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358456 (https://phabricator.wikimedia.org/T441253) (owner: 10Krinkle) [20:57:31] !log krinkle@deploy2003 Started scap sync-world: Backport for [[gerrit:1358456|Batch file retrievals for search result thumbnails (T441253)]] (1.48.0-wmf.1) [20:57:37] T441253: ST6.4.10 Increase MediaWiki backend capacity for logged-in pageviews (Oct-Nov-Dec 2026) - https://phabricator.wikimedia.org/T441253 [21:00:42] (03PS1) 10Volans: openstack API: re-enable security group APIs [puppet] - 10https://gerrit.wikimedia.org/r/1360994 [21:01:42] !log krinkle@deploy2003 krinkle: Backport for [[gerrit:1358456|Batch file retrievals for search result thumbnails (T441253)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:02:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [21:02:34] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12497194 (10cmooney) [21:04:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12497203 (10phaultfinder) [21:05:29] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12497208 (10cmooney) [21:09:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12497213 (10phaultfinder) [21:10:10] (03CR) 10Dzahn: [C:03+2] "does not fail anymore now:" [puppet] - 10https://gerrit.wikimedia.org/r/1355152 (https://phabricator.wikimedia.org/T441067) (owner: 10Dzahn) [21:14:55] !log krinkle@deploy2003 krinkle: Continuing with deployment [21:22:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [21:22:19] (03PS1) 10Cathal Mooney: Update common.py Importer logic to skip all IPIP/Tunnel ints [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1361004 [21:22:53] !log krinkle@deploy2003 Finished scap sync-world: Backport for [[gerrit:1358456|Batch file retrievals for search result thumbnails (T441253)]] (1.48.0-wmf.1) (duration: 25m 22s) [21:22:59] T441253: ST6.4.10 Increase MediaWiki backend capacity for logged-in pageviews (Oct-Nov-Dec 2026) - https://phabricator.wikimedia.org/T441253 [21:24:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12497251 (10phaultfinder) [21:44:19] (03CR) 10Volans: [C:03+2] "Agreed with Belinda to merge to unblock the related use-cases." [puppet] - 10https://gerrit.wikimedia.org/r/1360994 (owner: 10Volans) [21:46:16] PROBLEM - MariaDB Replica Lag: s4 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 631.28 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:48:50] 10SRE-tools, 06Infrastructure-Foundations, 10Packaging: Upgrade prometheus-atlas-exporter - https://phabricator.wikimedia.org/T429672#12497414 (10Lizimedia) The current version of Go in Debian Trixie is 1.24. The most recent version of this package that supports go 1.24 is [[https://github.com/czerwonk/atlas... [21:55:03] (03PS1) 10Jdlrobson: Fast forward wmf.1 branch to latest master [extensions/ReadingLists] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360551 (https://phabricator.wikimedia.org/T440916) [21:55:03] (03CR) 10Jdlrobson: "We decided that the a11y issue is not a blocker." [extensions/ReadingLists] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360551 (https://phabricator.wikimedia.org/T440916) (owner: 10Jdlrobson) [22:29:34] 10SRE-swift-storage, 10MediaWiki-File-management, 06MediaWiki-Media-Platform-Team: Unknown error while undeleting a file on Wikimedia Commons - https://phabricator.wikimedia.org/T437114#12497522 (10Sreejithk2000) Here is another example. I am unable to restore this file as well. https://commons.wikimedia.or... [22:39:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12497593 (10phaultfinder) [22:50:16] RECOVERY - MariaDB Replica Lag: s4 on db2239 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [23:00:06] 06SRE, 10SRE-Access-Requests, 06Product Safety and Integrity, 06Security-Team, 07SecTeam-Processed: Request IRC #mediawiki_security access for Novem Linguae - https://phabricator.wikimedia.org/T442747#12497674 (10LSobanski) +1 on behalf of SRE. [23:14:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12497695 (10phaultfinder) [23:17:36] (03PS1) 10Lizimedia: Move to gitlab [debs/prometheus-atlas-exporter] - 10https://gerrit.wikimedia.org/r/1361028 (https://phabricator.wikimedia.org/T429672) [23:18:51] (03PS2) 10Jdlrobson: Fast forward wmf.1 branch to 0626e6 [extensions/ReadingLists] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360551 (https://phabricator.wikimedia.org/T440916) [23:21:08] 10SRE-tools, 06Infrastructure-Foundations, 10Packaging, 13Patch-For-Review: Upgrade prometheus-atlas-exporter - https://phabricator.wikimedia.org/T429672#12497716 (10Lizimedia) I managed to talk Trixie into using the right version of Golang! I've tested the build and I know it compiles, at least. Just need... [23:21:18] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 604.65 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [23:21:23] (03PS2) 10Jdlrobson: Enable Vector CTA on all wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359878 (https://phabricator.wikimedia.org/T441705) [23:31:05] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [23:34:26] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:34:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12497765 (10phaultfinder) [23:54:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12497793 (10phaultfinder) [23:58:00] FIRING: PingLossPercent: Blackbox probe packet loss 4.375% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent