[00:59:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker1381:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1381 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [01:00:05] (03Abandoned) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1312266 (owner: 10TrainBranchBot) [01:04:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1381:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1381 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [01:14:59] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1312289 [01:14:59] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1312289 (owner: 10TrainBranchBot) [01:23:59] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1312289 (owner: 10TrainBranchBot) [01:57:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker1381:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1381 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [02:00:27] !log mwpresync@deploy2003 Started scap build-images: Publishing wmf/next image [02:02:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1381:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1381 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [02:07:30] !log mwpresync@deploy2003 Finished scap build-images: Publishing wmf/next image (duration: 07m 02s) [02:10:44] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:15:44] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:05:40] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [04:15:45] FIRING: CirrusStreamingUpdaterRateTooLow: CirrusSearch update rate from flink-app-consumer-search-backfill is critically low - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterRateTooLow [04:16:41] PROBLEM - SSH on arclamp2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [04:20:31] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [04:20:44] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [04:25:45] RESOLVED: CirrusStreamingUpdaterRateTooLow: CirrusSearch update rate from flink-app-consumer-search-backfill is critically low - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterRateTooLow [04:28:51] !log kevinbazira@deploy2003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [04:33:39] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [04:40:31] FIRING: [2x] RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [04:40:44] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [04:48:31] RECOVERY - SSH on arclamp2001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [04:50:31] FIRING: [2x] RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [04:50:44] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [04:59:31] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [05:00:31] RESOLVED: [2x] RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [05:00:44] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [05:14:34] (03PS1) 10Kevin Bazira: ml-services: Enable staging ingress for internal cluster access of tts-section-generator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312294 (https://phabricator.wikimedia.org/T432308) [05:18:09] (03CR) 10Kevin Bazira: [C:03+2] ml-services: Enable staging ingress for internal cluster access of tts-section-generator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312294 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [05:21:02] (03Merged) 10jenkins-bot: ml-services: Enable staging ingress for internal cluster access of tts-section-generator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312294 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [05:25:30] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on db2207.codfw.wmnet with reason: Host down [05:25:56] !log kevinbazira@deploy2003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [05:41:52] (03PS1) 10Kevin Bazira: ml-services: Enable prod ingress for internal cluster access of tts-section-generator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312296 (https://phabricator.wikimedia.org/T432308) [05:43:10] FIRING: BFDdown: BFD session down between cr2-eqdfw and fe80::b6f9:5dff:fe30:e538 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqdfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [05:45:29] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns7002 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is , dns.git is 56640a07d3f676819bc26a1b5e3c641c92dce74e) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [05:45:50] (03CR) 10Kevin Bazira: [C:03+2] ml-services: Enable prod ingress for internal cluster access of tts-section-generator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312296 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [05:48:10] RESOLVED: BFDdown: BFD session down between cr2-eqdfw and fe80::b6f9:5dff:fe30:e538 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqdfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [05:48:22] (03Merged) 10jenkins-bot: ml-services: Enable prod ingress for internal cluster access of tts-section-generator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312296 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [05:50:24] !log kevinbazira@deploy2003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [05:51:56] !log kevinbazira@deploy2003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [05:57:48] (03PS1) 10Marostegui: db2209: Add ticket number [puppet] - 10https://gerrit.wikimedia.org/r/1312297 [05:59:58] (03CR) 10Marostegui: [C:03+2] db2209: Add ticket number [puppet] - 10https://gerrit.wikimedia.org/r/1312297 (owner: 10Marostegui) [06:05:41] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [06:09:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [06:10:44] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:11:41] PROBLEM - SSH on arclamp2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [06:12:52] (03CR) 10Marostegui: "A DC master of a RW section cannot be depooled. It would need to be the whole section in that case." [cookbooks] - 10https://gerrit.wikimedia.org/r/1311405 (https://phabricator.wikimedia.org/T430769) (owner: 10Federico Ceratto) [06:14:10] (03CR) 10Marostegui: mysql: update replication source (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [06:15:44] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:19:31] FIRING: [2x] RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [06:25:31] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [06:25:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker1249:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1249 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [06:29:31] FIRING: [2x] RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [06:30:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1249:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1249 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [06:30:44] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:31:07] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1311443 (owner: 10L10n-bot) [06:45:31] RECOVERY - SSH on arclamp2001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [06:49:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [06:50:44] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:54:21] (03CR) 10Nikerabbit: [C:03+1] Article Guidance: migrate wikidata config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311489 (https://phabricator.wikimedia.org/T421250) (owner: 10Sbisson) [07:00:05] Amir1, urbanecm, and awight: #bothumor I � Unicode. All rise for UTC morning backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260720T0700). [07:00:05] DreamRimmer: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:05:40] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [07:06:03] Any deployers around? [07:09:25] (03PS1) 10Kevin Bazira: ml-services: Enable MediaWiki API proxy for tts-section-generator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312301 (https://phabricator.wikimedia.org/T432308) [07:12:37] (03CR) 10Kevin Bazira: [C:03+2] ml-services: Enable MediaWiki API proxy for tts-section-generator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312301 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [07:15:10] (03Merged) 10jenkins-bot: ml-services: Enable MediaWiki API proxy for tts-section-generator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312301 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [07:15:36] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-be2067.codfw.wmnet with OS trixie [07:15:44] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12135590 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-be2067.codfw.wmnet with OS trixie [07:17:23] !log kevinbazira@deploy2003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [07:30:01] !log brouberol@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [07:30:21] !log mvernon@cumin1003 START - Cookbook sre.hosts.reimage for host ms-be1072.eqiad.wmnet with OS trixie [07:30:29] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12135605 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-be1072.eqiad.wmnet with OS trixie [07:30:30] !log brouberol@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [07:31:40] (03PS1) 10Ladsgroup: Disable MJPEG, enable MPEG-4 Part II [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1312302 (https://phabricator.wikimedia.org/T358266) [07:35:43] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2067.codfw.wmnet with reason: host reimage [07:37:25] (03PS1) 10Kevin Bazira: ml-services: add envoy services-proxy listener for mw-api-int in tts-section-generator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312303 (https://phabricator.wikimedia.org/T432308) [07:39:48] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2067.codfw.wmnet with reason: host reimage [07:40:58] (03CR) 10Kevin Bazira: [C:03+2] ml-services: add envoy services-proxy listener for mw-api-int in tts-section-generator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312303 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [07:43:53] (03Merged) 10jenkins-bot: ml-services: add envoy services-proxy listener for mw-api-int in tts-section-generator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312303 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [07:44:43] (03CR) 10Ladsgroup: [C:03+2] swift: Migrate storage of results to asyncio [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1310618 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup) [07:44:49] (03CR) 10Ladsgroup: [C:03+2] images: Move loading of memcached key to asyncio [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1310623 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup) [07:45:26] !log kevinbazira@deploy2003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [07:45:42] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [07:45:51] !log kevinbazira@deploy2003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [07:49:02] !log mvernon@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1072.eqiad.wmnet with reason: host reimage [07:49:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [07:50:34] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [07:50:44] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:54:01] hi! I'm working on the Wikidata Query Service (the `wikidata-query-gui` service in helm / kubernetes) and we're trying to work out how our team can access the staging service. I see in the docs (https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters#staging) that I should be able to access the service at "https://staging.svc.eqiad.wmnet:{port}" but... that's only so helpful. Can anyone here help? [07:54:07] (03Merged) 10jenkins-bot: swift: Migrate storage of results to asyncio [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1310618 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup) [07:54:09] (03Merged) 10jenkins-bot: images: Move loading of memcached key to asyncio [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1310623 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup) [07:54:24] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1072.eqiad.wmnet with reason: host reimage [07:54:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [07:55:25] (03CR) 10Tiziano Fogli: [C:03+2] sloth: add rule to detect missing slo:sli_error:ratio_rate metric [alerts] - 10https://gerrit.wikimedia.org/r/1310604 (https://phabricator.wikimedia.org/T432140) (owner: 10Tiziano Fogli) [07:55:44] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:59:19] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2067.codfw.wmnet with OS trixie [07:59:26] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12135685 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-be2067.codfw.wmnet with OS trixie completed: - ms-be2067 (**PASS*... [08:00:09] (03PS5) 10A-pizzata: Daily sqoops for logging and cu_log tables [puppet] - 10https://gerrit.wikimedia.org/r/1311397 (https://phabricator.wikimedia.org/T432208) [08:01:11] (03CR) 10CI reject: [V:04-1] Daily sqoops for logging and cu_log tables [puppet] - 10https://gerrit.wikimedia.org/r/1311397 (https://phabricator.wikimedia.org/T432208) (owner: 10A-pizzata) [08:02:51] (03CR) 10A-pizzata: Daily sqoops for logging and cu_log tables (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1311397 (https://phabricator.wikimedia.org/T432208) (owner: 10A-pizzata) [08:12:59] (03PS6) 10A-pizzata: Daily sqoops for logging and cu_log tables [puppet] - 10https://gerrit.wikimedia.org/r/1311397 (https://phabricator.wikimedia.org/T432208) [08:15:59] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1072.eqiad.wmnet with OS trixie [08:16:13] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12135808 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-be1072.eqiad.wmnet with OS trixie completed: - ms-be1072 (**PASS*... [08:16:23] (03PS7) 10A-pizzata: Daily sqoops for logging and cu_log tables [puppet] - 10https://gerrit.wikimedia.org/r/1311397 (https://phabricator.wikimedia.org/T432208) [08:17:58] (03CR) 10Federico Ceratto: "When .depool is called on the primary master we instead set the master RO and leave everything pooled." [cookbooks] - 10https://gerrit.wikimedia.org/r/1311405 (https://phabricator.wikimedia.org/T430769) (owner: 10Federico Ceratto) [08:22:06] (03CR) 10Brouberol: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311397 (https://phabricator.wikimedia.org/T432208) (owner: 10A-pizzata) [08:25:11] mvernon@cumin2003 reimage (PID 3058370) is awaiting input [08:27:29] (03CR) 10TheDJ: [C:03+1] Disable MJPEG, enable MPEG-4 Part II [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1312302 (https://phabricator.wikimedia.org/T358266) (owner: 10Ladsgroup) [08:32:25] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-be2068.codfw.wmnet with OS trixie [08:32:40] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12135887 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-be2068.codfw.wmnet with OS trixie [08:45:15] (03PS1) 10Kevin Bazira: ml-services: update tts-section-generator image to server that uses WMF CA bundle for isvc TLS verification [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312307 (https://phabricator.wikimedia.org/T432308) [08:48:40] (03CR) 10Kevin Bazira: [C:03+2] ml-services: update tts-section-generator image to server that uses WMF CA bundle for isvc TLS verification [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312307 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [08:50:45] !log kevinbazira@deploy2003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [08:51:21] (03Merged) 10jenkins-bot: ml-services: update tts-section-generator image to server that uses WMF CA bundle for isvc TLS verification [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312307 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [08:51:35] (03CR) 10Brouberol: [C:03+1] Daily sqoops for logging and cu_log tables [puppet] - 10https://gerrit.wikimedia.org/r/1311397 (https://phabricator.wikimedia.org/T432208) (owner: 10A-pizzata) [08:52:38] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2068.codfw.wmnet with reason: host reimage [08:53:55] (03CR) 10Brouberol: [C:03+2] Daily sqoops for logging and cu_log tables [puppet] - 10https://gerrit.wikimedia.org/r/1311397 (https://phabricator.wikimedia.org/T432208) (owner: 10A-pizzata) [08:56:29] (03CR) 10Klausman: [C:03+1] stat hosts: Set up S3 config for search platform team. [puppet] - 10https://gerrit.wikimedia.org/r/1311880 (https://phabricator.wikimedia.org/T432249) (owner: 10Bking) [08:56:44] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [08:57:08] (03PS6) 10Samwilson: Deny access to some particular format conversions [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1311830 (https://phabricator.wikimedia.org/T430528) [08:59:15] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2068.codfw.wmnet with reason: host reimage [09:00:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [09:00:44] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:05:36] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:10:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [09:10:44] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:11:20] (03CR) 10Blake: [C:03+2] kubernetes: Add a debug deployment for mw-pretrain. [puppet] - 10https://gerrit.wikimedia.org/r/1311049 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [09:11:42] PROBLEM - SSH on arclamp2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:12:44] (03PS1) 10Ozge: ml-services: Switch llm-qwen3-14b to FP8 on a single 24GB partition [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312310 (https://phabricator.wikimedia.org/T431852) [09:13:33] !log blake@deploy2003 Started scap sync-world: Non-deployment scap run to populate new release values [09:13:36] !log blake@deploy2003 sync-world aborted: Non-deployment scap run to populate new release values (duration: 00m 02s) [09:14:40] (will wait until the infra window to do that) [09:15:44] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:16:07] !log mvernon@cumin1003 START - Cookbook sre.hosts.reimage for host ms-be1073.eqiad.wmnet with OS trixie [09:16:16] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12136032 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-be1073.eqiad.wmnet with OS trixie [09:17:01] FIRING: [2x] RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [09:19:01] (03CR) 10Marostegui: "I think the logic needs to follow what is described on the task, let me know if it is not clear enough and I need to elaborate a bit more " [cookbooks] - 10https://gerrit.wikimedia.org/r/1311405 (https://phabricator.wikimedia.org/T430769) (owner: 10Federico Ceratto) [09:19:09] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2068.codfw.wmnet with OS trixie [09:19:23] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12136036 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-be2068.codfw.wmnet with OS trixie completed: - ms-be2068 (**PASS*... [09:21:58] (03Abandoned) 10Hashar: gerrit: limit the volume of returnable pages [puppet] - 10https://gerrit.wikimedia.org/r/1308081 (https://phabricator.wikimedia.org/T431398) (owner: 10Arnaudb) [09:24:37] (03PS1) 10VadymTS1: Modify user groups rights in English Wikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1312423 (https://phabricator.wikimedia.org/T432557) [09:29:14] 06SRE, 10hCaptcha, 06Product Safety and Integrity, 06ServiceOps new, and 2 others: memcached errors seen in hCaptcha health checks - https://phabricator.wikimedia.org/T430340#12136060 (10MLechvien-WMF) [09:30:10] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, July 20 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1312423 (https://phabricator.wikimedia.org/T432557) (owner: 10VadymTS1) [09:32:19] (03PS1) 10Kevin Bazira: ml-services: update tts-section-generator image to server that uses system CA bundle for isvc TLS verification [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312431 (https://phabricator.wikimedia.org/T432308) [09:34:51] !log mvernon@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1073.eqiad.wmnet with reason: host reimage [09:35:15] (03CR) 10Kevin Bazira: [C:03+2] ml-services: update tts-section-generator image to server that uses system CA bundle for isvc TLS verification [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312431 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [09:35:54] (03PS1) 10Brouberol: Add 2TB to the dumps volume [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312437 [09:37:33] (03Merged) 10jenkins-bot: ml-services: update tts-section-generator image to server that uses system CA bundle for isvc TLS verification [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312431 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [09:37:51] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 8 hosts [09:38:43] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1073.eqiad.wmnet with reason: host reimage [09:39:16] (03CR) 10Bartosz Wójtowicz: "Thank you, left 1 comment" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312310 (https://phabricator.wikimedia.org/T431852) (owner: 10Ozge) [09:39:35] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 8 hosts [09:39:44] !log kevinbazira@deploy2003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [09:42:20] (03PS2) 10Ozge: ml-services: Switch llm-qwen3-14b to FP8 on a single 24GB partition [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312310 (https://phabricator.wikimedia.org/T431852) [09:42:51] (03CR) 10Ozge: ml-services: Switch llm-qwen3-14b to FP8 on a single 24GB partition (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312310 (https://phabricator.wikimedia.org/T431852) (owner: 10Ozge) [09:43:32] RECOVERY - SSH on arclamp2001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:45:44] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:47:00] (03CR) 10Bartosz Wójtowicz: [C:03+1] ml-services: Switch llm-qwen3-14b to FP8 on a single 24GB partition [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312310 (https://phabricator.wikimedia.org/T431852) (owner: 10Ozge) [09:47:01] RESOLVED: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [09:56:47] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 324 hosts [09:58:41] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to Superset for jniren-ctr - https://phabricator.wikimedia.org/T432273#12136108 (10Jniren-ctr) HI @Aklapper - think thats done now [10:00:04] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260720T1000) [10:00:32] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1073.eqiad.wmnet with OS trixie [10:00:40] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12136128 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-be1073.eqiad.wmnet with OS trixie completed: - ms-be1073 (**PASS*... [10:04:27] going to start a non-deployment scap run to sync release values momentarily [10:04:44] (adding a debug release for mw-pretrain) [10:05:42] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-be2069.codfw.wmnet with OS trixie [10:05:50] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12136142 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-be2069.codfw.wmnet with OS trixie [10:06:40] !log blake@deploy2003 Started scap sync-world: Non-deployment scap run to populate new release values [10:06:58] !log blake@deploy2003 Stopping before sync operations [10:08:24] (03PS5) 10Trueg: WDQSv2: Several small adjustments [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311841 [10:10:14] PROBLEM - Host cloudlb1001 is DOWN: PING CRITICAL - Packet loss = 100% [10:10:26] PROBLEM - BFD status on cloudsw1-c8-eqiad.mgmt is CRITICAL: Down: 1 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [10:10:34] (03CR) 10Federico Ceratto: sre.mysql: Reorder imports (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1311404 (owner: 10Federico Ceratto) [10:11:27] RECOVERY - BFD status on cloudsw1-c8-eqiad.mgmt is OK: UP: 16 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [10:11:29] RECOVERY - Host cloudlb1001 is UP: PING OK - Packet loss = 0%, RTA = 2.21 ms [10:11:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cloudsw1-c8-eqiad and cloudlb1001 (172.20.1.2) - group cloud_host - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [10:12:51] PROBLEM - Host cloudlb1002 is DOWN: PING CRITICAL - Packet loss = 100% [10:13:21] RECOVERY - Host cloudlb1002 is UP: PING OK - Packet loss = 0%, RTA = 0.38 ms [10:16:39] RESOLVED: [3x] CoreBGPDown: Core BGP session down between cloudsw1-c8-eqiad and cloudlb1001 (172.20.1.2) - group cloud_host - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [10:19:44] (03PS1) 10Trueg: WDQSv2: Use TopoLVM as the storage solution for the Qlever DB [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312456 (https://phabricator.wikimedia.org/T432029) [10:23:47] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [10:26:33] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2069.codfw.wmnet with reason: host reimage [10:27:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [10:30:31] !log mvernon@cumin1003 START - Cookbook sre.hosts.reimage for host ms-be1074.eqiad.wmnet with OS trixie [10:30:41] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12136208 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-be1074.eqiad.wmnet with OS trixie [10:30:44] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:30:44] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2069.codfw.wmnet with reason: host reimage [10:34:39] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [10:35:44] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:37:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [10:41:42] PROBLEM - SSH on arclamp2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [10:42:22] (03CR) 10Btullis: WDQSv2: Use TopoLVM as the storage solution for the Qlever DB (033 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312456 (https://phabricator.wikimedia.org/T432029) (owner: 10Trueg) [10:45:01] FIRING: [2x] RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [10:45:16] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [10:45:44] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:49:12] !log mvernon@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1074.eqiad.wmnet with reason: host reimage [10:50:01] FIRING: [2x] RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [10:50:40] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2069.codfw.wmnet with OS trixie [10:50:50] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12136293 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-be2069.codfw.wmnet with OS trixie completed: - ms-be2069 (**PASS*... [10:53:53] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1074.eqiad.wmnet with reason: host reimage [10:54:19] (03CR) 10Btullis: [C:03+1] "Looks good to me." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312437 (owner: 10Brouberol) [11:00:40] (03CR) 10Btullis: [C:03+1] "Great, thanks." [puppet] - 10https://gerrit.wikimedia.org/r/1311899 (https://phabricator.wikimedia.org/T432148) (owner: 10Bking) [11:05:40] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:06:05] (03CR) 10Ozge: [V:03+2 C:03+2] ml-services: Switch llm-qwen3-14b to FP8 on a single 24GB partition [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312310 (https://phabricator.wikimedia.org/T431852) (owner: 10Ozge) [11:06:16] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 324 hosts [11:08:36] (03Merged) 10jenkins-bot: ml-services: Switch llm-qwen3-14b to FP8 on a single 24GB partition [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312310 (https://phabricator.wikimedia.org/T431852) (owner: 10Ozge) [11:09:13] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 6 hosts [11:09:38] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 6 hosts [11:09:56] (03CR) 10Effie Mouzeli: api-gateway: remove DNS for api-gateway service. (032 comments) [dns] - 10https://gerrit.wikimedia.org/r/1311856 (https://phabricator.wikimedia.org/T432445) (owner: 10Blake) [11:10:21] (03CR) 10Effie Mouzeli: "We also probably should remove the entry from utils/mock_etc/discovery-geo-resources" [dns] - 10https://gerrit.wikimedia.org/r/1311856 (https://phabricator.wikimedia.org/T432445) (owner: 10Blake) [11:10:53] (03CR) 10Trueg: WDQSv2: Use TopoLVM as the storage solution for the Qlever DB (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312456 (https://phabricator.wikimedia.org/T432029) (owner: 10Trueg) [11:14:54] !log ozge@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [11:15:20] (03CR) 10Brouberol: [C:03+2] Add 2TB to the dumps volume [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312437 (owner: 10Brouberol) [11:16:57] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply [11:17:06] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply [11:17:22] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1074.eqiad.wmnet with OS trixie [11:17:28] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12136330 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-be1074.eqiad.wmnet with OS trixie completed: - ms-be1074 (**PASS*... [11:20:36] (03PS1) 10Michael Große: postEdit experiment: enroll control users by same criteria [core] (wmf/1.47.0-wmf.11) - 10https://gerrit.wikimedia.org/r/1312471 [11:20:51] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, July 20 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [core] (wmf/1.47.0-wmf.11) - 10https://gerrit.wikimedia.org/r/1312471 (owner: 10Michael Große) [11:21:27] PROBLEM - BFD status on cloudsw1-d5-eqiad.mgmt is CRITICAL: Down: 1 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [11:22:27] RECOVERY - BFD status on cloudsw1-d5-eqiad.mgmt is OK: UP: 16 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [11:23:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cloudsw1-d5-eqiad and cloudservices1005 (172.20.2.4) - group cloud_host - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [11:24:27] PROBLEM - BFD status on cloudsw1-c8-eqiad.mgmt is CRITICAL: Down: 1 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [11:25:27] RECOVERY - BFD status on cloudsw1-c8-eqiad.mgmt is OK: UP: 16 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [11:28:39] RESOLVED: [4x] CoreBGPDown: Core BGP session down between cloudsw1-c8-eqiad and cloudservices1006 (172.20.1.5) - group cloud_host - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [11:37:57] PROBLEM - Host cloudgw1003 is DOWN: PING CRITICAL - Packet loss = 100% [11:39:00] (03PS1) 10Ozge: ml-services: Tune llm-qwen3-14b to fit KV cache on 24GB partition [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312475 (https://phabricator.wikimedia.org/T431852) [11:39:25] RECOVERY - Host cloudgw1003 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms [11:39:45] the cloud-* alerts is me, trixie reboots [11:40:21] (03CR) 10Ozge: "fix attempt for the No available memory for the cache blocks in https://phabricator.wikimedia.org/T431852#12136380" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312475 (https://phabricator.wikimedia.org/T431852) (owner: 10Ozge) [11:42:57] PROBLEM - Host cloudgw1004 is DOWN: PING CRITICAL - Packet loss = 100% [11:45:25] RECOVERY - Host cloudgw1004 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [11:48:41] (03CR) 10CWilliams: sre.mysql: Reorder imports (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1311404 (owner: 10Federico Ceratto) [11:49:14] (03PS1) 10Ayounsi: Add subrated circuit rate to interface descriptions [software/homer/deploy] - 10https://gerrit.wikimedia.org/r/1312476 (https://phabricator.wikimedia.org/T374614) [11:50:16] 06SRE, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Create alerting for saturation on sub-rated interfaces - https://phabricator.wikimedia.org/T374614#12136425 (10ayounsi) `lang=diff Changes for 1 devices: ['cr2-drmrs.wikimedia.org'] [edit interfaces et-0/0/0] - description "Transport: Ar... [11:50:29] (03CR) 10CI reject: [V:04-1] Add subrated circuit rate to interface descriptions [software/homer/deploy] - 10https://gerrit.wikimedia.org/r/1312476 (https://phabricator.wikimedia.org/T374614) (owner: 10Ayounsi) [11:50:45] (03CR) 10Michael Große: [C:03+1] "This seems indeed like the only remaining place that uses that stream:" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311851 (https://phabricator.wikimedia.org/T428265) (owner: 10EMcFarland) [11:51:49] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-be2070.codfw.wmnet with OS trixie [11:51:49] !log mvernon@cumin1003 START - Cookbook sre.hosts.reimage for host ms-be1075.eqiad.wmnet with OS trixie [11:51:56] (03PS1) 10Jelto: Revert "gerrit: filter paths on httpd" [puppet] - 10https://gerrit.wikimedia.org/r/1312477 (https://phabricator.wikimedia.org/T431398) [11:51:58] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12136434 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-be2070.codfw.wmnet with OS trixie [11:51:59] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12136435 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-be1075.eqiad.wmnet with OS trixie [11:52:18] (03CR) 10CI reject: [V:04-1] Revert "gerrit: filter paths on httpd" [puppet] - 10https://gerrit.wikimedia.org/r/1312477 (https://phabricator.wikimedia.org/T431398) (owner: 10Jelto) [11:54:54] (03PS2) 10Ozge: ml-services: Tune llm-qwen3-14b to fit KV cache on 24GB partition [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312475 (https://phabricator.wikimedia.org/T431852) [11:57:02] (03PS2) 10Ayounsi: Add subrated circuit rate to interface descriptions [software/homer/deploy] - 10https://gerrit.wikimedia.org/r/1312476 (https://phabricator.wikimedia.org/T374614)