[00:52:47] 10Striker, 06tools-platform-team, 10Charts, 06MediaWiki-API-Platform-Team, and 10 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12579362 (10Jdforrester-WMF) [00:54:35] 10Striker, 06tools-platform-team, 10Citoid, 10CXServer, and 21 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12579364 (10Jdforrester-WMF) p:05Triage→03Medium [02:57:54] (03open) 10ibrahemqasim: Refactor job syncers and admin views [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/46 [02:59:31] (03update) 10ibrahemqasim: Refactor job syncers and admin views [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/46 [02:59:58] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v2.19.1 [03:00:00] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v2.19.1 [03:01:41] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics/kube-state-metrics:v2.19.1 [03:01:50] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/kube-state-metrics/kube-state-metrics:v2.19.1 [03:02:34] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v2.19.1 [03:02:37] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/kube-state-metrics:v2.19.1 [05:08:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:13:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:38:24] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44466124665 (https://github.com/cluebotng/component-configs/commits/95ede68face1e762eed077032e4e876c8c44a581) [05:39:15] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44466124641 (https://github.com/cluebotng/component-configs/commits/95ede68face1e762eed077032e4e876c8c44a581) [06:03:43] 10Striker, 06tools-platform-team, 10Citoid, 10CXServer, and 21 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12580218 (10KartikMistry) [06:04:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:09:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:11:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:16:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:49:49] !log tools.cluebot-documentation Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44474696258 (https://github.com/cluebotng/component-configs/commits/dcf2a4e2b6aaebd5e761a7ecc06e7b6279c46ba8) [06:50:17] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44474696515 (https://github.com/cluebotng/component-configs/commits/dcf2a4e2b6aaebd5e761a7ecc06e7b6279c46ba8) [06:50:18] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44474696433 (https://github.com/cluebotng/component-configs/commits/dcf2a4e2b6aaebd5e761a7ecc06e7b6279c46ba8) [06:50:35] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44474696571 (https://github.com/cluebotng/component-configs/commits/dcf2a4e2b6aaebd5e761a7ecc06e7b6279c46ba8) [06:50:35] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44474696560 (https://github.com/cluebotng/component-configs/commits/dcf2a4e2b6aaebd5e761a7ecc06e7b6279c46ba8) [06:50:45] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44474696508 (https://github.com/cluebotng/component-configs/commits/dcf2a4e2b6aaebd5e761a7ecc06e7b6279c46ba8) [06:51:15] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44474696427 (https://github.com/cluebotng/component-configs/commits/dcf2a4e2b6aaebd5e761a7ecc06e7b6279c46ba8) [06:51:23] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/44474696596 (https://github.com/cluebotng/component-configs/commits/dcf2a4e2b6aaebd5e761a7ecc06e7b6279c46ba8) [06:52:36] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44474696230 (https://github.com/cluebotng/component-configs/commits/dcf2a4e2b6aaebd5e761a7ecc06e7b6279c46ba8) [07:56:09] (03open) 10kevinpayravi: Speed up first load (narrow PetScan query / background refresh / smaller thumbs) [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/41 [07:58:04] (03update) 10kevinpayravi: Speed up first load (narrow PetScan query / background refresh / smaller thumbs) [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/41 [08:12:28] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations: The Cloud intermediate CA is going to expire on Dec 12th - https://phabricator.wikimedia.org/T442012#12580390 (10LSobanski) p:05High→03Triage [08:44:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:49:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:04:10] 10Striker, 06tools-platform-team, 10Charts, 06MediaWiki-API-Platform-Team, and 10 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12580510 (10aputhin) Just to reiterate: #Striker is already on Node 26, but the script doesn't capture that since it does... [09:41:44] 06tools-infrastructure-team: Decom old cloudcephosd hardware - https://phabricator.wikimedia.org/T440684#12580655 (10fgiunchedi) I started the cloudcephosd1025 drain via cookbook yesterday, draining a single osd fully, though I noticed a fair amount of network traffic *into* 1025 which I didn't expect nor we wou... [09:44:14] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44498684542 (https://github.com/cluebotng/component-configs/commits/b740e79851900a7193ec4ebf1095dd80c90d4b5b) [09:44:25] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44498684658 (https://github.com/cluebotng/component-configs/commits/b740e79851900a7193ec4ebf1095dd80c90d4b5b) [09:50:15] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44499450421 (https://github.com/cluebotng/component-configs/commits/397027bdf00cb3bc48057a4b0376c0ea5b049b23) [10:10:06] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [10:13:06] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance tools-test-nfs on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:21:25] (03update) 10atitkov: Point OPS deployment commands at the games-native-wikirun tool [toolforge-repos/wikirun-game] (native-integration) - 10https://gitlab.wikimedia.org/toolforge-repos/wikirun-game/-/merge_requests/5 [10:25:06] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [11:13:19] (03update) 10atitkov: Point OPS deployment commands at the games-native-wikirun tool [toolforge-repos/wikirun-game] (native-integration) - 10https://gitlab.wikimedia.org/toolforge-repos/wikirun-game/-/merge_requests/5 [11:36:01] (03update) 10atitkov: Point OPS deployment commands at the games-native-wikirun tool [toolforge-repos/wikirun-game] (native-integration) - 10https://gitlab.wikimedia.org/toolforge-repos/wikirun-game/-/merge_requests/5 [11:40:01] (03merge) 10atitkov: Point OPS deployment commands at the games-native-wikirun tool [toolforge-repos/wikirun-game] (native-integration) - 10https://gitlab.wikimedia.org/toolforge-repos/wikirun-game/-/merge_requests/5 [12:06:20] (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/81 [12:06:22] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/55 [12:47:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:59:40] 10Striker, 06tools-platform-team, 10Charts, 06MediaWiki-API-Platform-Team, and 10 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12582020 (10Jdforrester-WMF) [13:00:39] 10Striker, 06tools-platform-team, 10Citoid, 10CXServer, and 21 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12582043 (10Jdforrester-WMF) [13:00:43] 10Striker, 06tools-platform-team, 10Charts, 06MediaWiki-API-Platform-Team, and 10 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12582054 (10Jdforrester-WMF) >>! In T393434#12580510, @aputhin wrote: > Just to reiterate: #Striker is already on Node 26... [13:02:03] (03approved) 10collins: Search in Language Dropdowns (Source/Target) only matches localized script, not English names. [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/120 (https://phabricator.wikimedia.org/T123456 https://phabricator.wikimedia.org/T422997) (owner: 10juniorb8) [13:02:15] (03merge) 10collins: Search in Language Dropdowns (Source/Target) only matches localized script, not English names. [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/120 (https://phabricator.wikimedia.org/T123456 https://phabricator.wikimedia.org/T422997) (owner: 10juniorb8) [13:07:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:07:54] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44474696596 (https://github.com/cluebotng/component-configs/commits/dcf2a4e2b6aaebd5e761a7ecc06e7b6279c46ba8) [13:09:58] 10Striker, 06tools-platform-team, 10Charts, 06MediaWiki-API-Platform-Team, and 10 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12582101 (10Nikerabbit) FYI {T445348} we are trying to get it undeployed to avoid future confusion. [13:19:41] (03update) 10aghirelli: chore: run on Node.js 22 via the Toolforge Build Service [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/28 (https://phabricator.wikimedia.org/T442177) [13:20:20] (03update) 10collins: Link to the volunteer's own edits on My contributions [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/121 (https://phabricator.wikimedia.org/T445005) (owner: 10juniorb8) [13:21:21] 10Striker, 06tools-platform-team, 10Charts, 06MediaWiki-API-Platform-Team, and 10 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12582160 (10AGhirelli-WMF) [13:32:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:37:01] 10Cloud-VPS, 06tools-infrastructure-team, 06Test Platform (Aktau 32): Test/benchmark virtio multiqueue for integration-cache-proxy - https://phabricator.wikimedia.org/T440418#12582246 (10AMarkossyan-WMF) [13:37:03] 10Cloud-VPS, 06tools-infrastructure-team, 06Test Platform (Aktau 32): Evaluate virtio-net multiqueue for Cloud VPS instances - https://phabricator.wikimedia.org/T443396#12582247 (10AMarkossyan-WMF) [13:38:02] 10Tool-inteGraality: Replace the free-form `url` parameter with a wiki identity - https://phabricator.wikimedia.org/T438455#12582262 (10JeanFred) Updated both Property dashboard and Integraality cell templates to pass `wiki` instead of `url` [13:56:14] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: Move cloudinfra & Toolforge container registries behind HAProxy - https://phabricator.wikimedia.org/T445537 (10taavi) 03NEW [13:56:44] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloud-vps: run unattended upgrades more often - https://phabricator.wikimedia.org/T445398#12582388 (10Volans) @Andrew sorry jusdt noticed you opened the task yesterday, I have sent this today: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1... [14:02:09] (03open) 10hashar: gerrit: ignore PCC triggering message [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/83 [14:03:56] (03approved) 10collins: Link to the volunteer's own edits on My contributions [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/121 (https://phabricator.wikimedia.org/T445005) (owner: 10juniorb8) [14:04:07] (03update) 10collins: Link to the volunteer's own edits on My contributions [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/121 (https://phabricator.wikimedia.org/T445005) (owner: 10juniorb8) [14:04:17] (03update) 10hashar: gerrit: ignore PCC triggering message [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/83 [14:05:00] (03update) 10hashar: gerrit: ignore PCC triggering message [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/83 [14:07:21] (03merge) 10collins: Link to the volunteer's own edits on My contributions [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/121 (https://phabricator.wikimedia.org/T445005) (owner: 10juniorb8) [14:22:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:23:35] 06tools-infrastructure-team: Decom old cloudcephosd hardware - https://phabricator.wikimedia.org/T440684#12582564 (10Andrew) >>! In T440684#12580655, @fgiunchedi wrote: > I started the cloudcephosd1025 drain via cookbook yesterday, draining a single osd fully, though I noticed a fair amount of network traffic *i... [15:12:24] 10Toolforge, 06tools-platform-team: Remove jessie + stretch + buster + bullseye containers from Toolforge - https://phabricator.wikimedia.org/T286798#12582810 (10taavi) [15:25:20] 10Toolforge, 06tools-infrastructure-team: Toolforge: Ensure long-running Kubernetes pods get container updates applied - https://phabricator.wikimedia.org/T314705#12582919 (10taavi) p:05Low→03High a:03taavi [15:28:09] (03update) 10bd808: gerrit: ignore ScheduleDeploymentBot [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/81 (owner: 10hashar) [15:28:09] (03approved) 10bd808: gerrit: ignore ScheduleDeploymentBot [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/81 (owner: 10hashar) [15:28:12] (03merge) 10bd808: gerrit: ignore ScheduleDeploymentBot [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/81 (owner: 10hashar) [15:35:40] (03update) 10bd808: gerrit: ignore PCC triggering message [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/83 (owner: 10hashar) [15:39:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:40:37] 10Toolforge, 06tools-platform-team: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559 (10aputhin) 03NEW [15:40:52] (03update) 10bd808: gerrit: ignore PCC triggering message [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/83 (owner: 10hashar) [15:40:53] (03approved) 10bd808: gerrit: ignore PCC triggering message [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/83 (owner: 10hashar) [15:40:57] (03merge) 10bd808: gerrit: ignore PCC triggering message [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/83 (owner: 10hashar) [15:44:10] 10Toolforge, 06tools-platform-team: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12583112 (10aputhin) p:05Triage→03High a:03fnegri [15:44:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:46:06] 10Toolforge, 06tools-platform-team: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12583138 (10lukastesar) Does this apply to [[ https://hub.docker.com/_/mediawiki | mediawiki docker hub image ]] or should I create a new separate task for that? [15:47:35] 10Toolforge, 06tools-platform-team: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12583164 (10taavi) >>! In T445559#12583138, @lukastesar wrote: > Does this also apply to [[ https://hub.docker.com/_/mediawiki | mediawiki docker hub image ]] or should I create a new separate task for... [15:49:12] 10Toolforge, 06tools-platform-team: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12583171 (10lukastesar) oh I see, thanks [15:49:25] 10Toolforge, 06tools-platform-team: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12583175 (10fnegri) [15:51:39] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:51:58] (03merge) 10taavi: Remove outdated names and links [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/15 (owner: 10fnegri) [15:56:39] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:01:39] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:06:39] RESOLVED: [2x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:18:19] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: Move cloudinfra & Toolforge container registries behind HAProxy - https://phabricator.wikimedia.org/T445537#12583480 (10Volans) +1 [16:21:13] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: Move cloudinfra & Toolforge container registries behind HAProxy - https://phabricator.wikimedia.org/T445537#12583489 (10taavi) [16:22:11] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: Move cloudinfra & Toolforge container registries behind HAProxy - https://phabricator.wikimedia.org/T445537#12583503 (10taavi) [16:22:49] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloud-vps: run unattended upgrades more often - https://phabricator.wikimedia.org/T445398#12583504 (10Volans) 05Open→03Resolved p:05Triage→03Medium Merged and deployed, unattended upgrades are now run hourly [16:49:50] (03open) 10taavi: Rest of the owl [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/1 (https://phabricator.wikimedia.org/T314705) [16:50:24] (03update) 10taavi: Rest of the owl [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/1 (https://phabricator.wikimedia.org/T314705) [16:54:57] (03update) 10taavi: Rest of the owl [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/1 (https://phabricator.wikimedia.org/T314705) [16:55:19] (03update) 10taavi: Rest of the owl [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/1 (https://phabricator.wikimedia.org/T314705) [16:56:34] (03merge) 10taavi: Rest of the owl [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/1 (https://phabricator.wikimedia.org/T314705) [16:59:05] (03open) 10taavi: Remove .gitreview [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/2 [16:59:57] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [17:00:01] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [17:01:20] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [17:01:24] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [17:02:54] 10Toolforge, 06tools-platform-team: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12583801 (10aputhin) [17:03:46] (03update) 10taavi: Remove .gitreview [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/2 [17:05:27] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [17:06:04] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [17:07:02] (03update) 10taavi: Remove .gitreview [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/2 [17:08:06] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [17:08:37] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [17:11:03] (03update) 10taavi: Remove .gitreview [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/2 [17:16:36] (03merge) 10taavi: Remove .gitreview [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/2 [17:22:39] (03open) 10fnegri: Update Heroku builder image [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1532 (https://phabricator.wikimedia.org/T445559) [17:22:42] (03update) 10fnegri: Update Heroku builder image [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1532 (https://phabricator.wikimedia.org/T445559) [17:23:54] !log fnegri@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [17:26:33] (03open) 10taavi: Add kube-container-updater [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1533 (https://phabricator.wikimedia.org/T314705) [17:28:29] (03open) 10tlepage: openapi: don't publish the alerts API [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/116 (https://phabricator.wikimedia.org/T440736) [17:28:33] !log fnegri@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [17:31:38] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [17:32:54] (03open) 10taavi: chart: Fix ServiceAccount definition [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/3 [17:32:59] 10Striker, 06tools-platform-team, 10Citoid, 10CXServer, and 20 others: Migrate Event Platform to node26 - https://phabricator.wikimedia.org/T445574 (10tchin) 03NEW [17:36:35] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [17:38:56] (03update) 10taavi: chart: Fix ServiceAccount definition [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/3 [17:43:11] (03open) 10tlepage: stop echoing request headers in the /auth response [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/117 (https://phabricator.wikimedia.org/T440736) [17:48:33] (03open) 10piastu: MPD-12; adding WDQSv2 endpoint (and its tests) [toolforge-repos/memento-pd] - 10https://gitlab.wikimedia.org/toolforge-repos/memento-pd/-/merge_requests/1 [17:48:52] (03merge) 10piastu: MPD-12; adding WDQSv2 endpoint (and its tests) [toolforge-repos/memento-pd] - 10https://gitlab.wikimedia.org/toolforge-repos/memento-pd/-/merge_requests/1 [17:51:11] 10Striker, 06tools-platform-team, 10Charts, 06MediaWiki-API-Platform-Team, and 10 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12584049 (10Jdforrester-WMF) [17:51:30] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12584060 (10fnegri) [17:51:51] 10Striker, 06tools-platform-team, 10Citoid, 10CXServer, and 21 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12584059 (10Jdforrester-WMF) [17:52:00] 10Striker, 06tools-platform-team, 10Charts, 06MediaWiki-API-Platform-Team, and 10 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12584066 (10Jdforrester-WMF) [17:52:14] 10Striker, 06tools-platform-team, 10Citoid, 10CXServer, and 21 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12584064 (10Jdforrester-WMF) [17:54:10] (03update) 10taavi: chart: Fix ServiceAccount definition [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/3 [17:54:16] (03open) 10tlepage: alerts-api: correct cluster domain in certificate [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1534 (https://phabricator.wikimedia.org/T440736) [17:56:06] (03open) 10fnegri: Remove outdated warning from README [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/100 (https://phabricator.wikimedia.org/T442364) [17:58:16] (03update) 10taavi: chart: Fix ServiceAccount definition [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/3 [17:59:37] (03update) 10fnegri: Remove outdated warning from README [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/100 (https://phabricator.wikimedia.org/T442364 https://phabricator.wikimedia.org/T444615) [17:59:44] (03update) 10taavi: Fixes to run in Toolsbeta [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/3 [18:00:32] (03update) 10fnegri: Update Heroku builder image [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1532 (https://phabricator.wikimedia.org/T444615 https://phabricator.wikimedia.org/T445559) [18:01:14] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [18:01:15] (03update) 10fnegri: Update Heroku builder image [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1532 (https://phabricator.wikimedia.org/T444615 https://phabricator.wikimedia.org/T445559) [18:01:17] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [18:01:27] (03update) 10fnegri: Update Heroku builder image [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1532 (https://phabricator.wikimedia.org/T444615 https://phabricator.wikimedia.org/T445559) [18:01:36] (03update) 10fnegri: Update Heroku builder image [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1532 (https://phabricator.wikimedia.org/T444615 https://phabricator.wikimedia.org/T445559) [18:02:31] (03open) 10andrew: Downgrade deps to k8s1.34.0 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/41 [18:03:31] (03open) 10tlepage: Draft: api-gateway: route /alerts to alerts-api [repos/cloud/toolforge/toolforge-deploy] (correct-alerts-api-cert-domain) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1535 (https://phabricator.wikimedia.org/T440736) [18:03:51] (03update) 10tlepage: Draft: api-gateway: route /alerts to alerts-api [repos/cloud/toolforge/toolforge-deploy] (correct-alerts-api-cert-domain) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1535 (https://phabricator.wikimedia.org/T440736) [18:05:32] (03update) 10bd808: Set bot mode (+B) when connecting [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/82 (https://phabricator.wikimedia.org/T445359) (owner: 10hashar) [18:06:34] (03merge) 10taavi: Fixes to run in Toolsbeta [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/3 [18:07:39] (03merge) 10andrew: Downgrade deps to k8s1.34.0 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/41 [18:12:27] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: envvars-admission: bump to 0.0.46-20261119180753-f1e9e478 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1536 [18:12:28] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: envvars-admission: bump to 0.0.46-20261119180753-f1e9e478 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1536 [18:12:40] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [18:16:05] (03update) 10taavi: Add kube-container-updater [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1533 (https://phabricator.wikimedia.org/T314705) [18:17:41] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12584181 (10DamianZaremba) I see the builder change waa deployed, but can we also get the MR merged. In leu of T422046 that file is the only way to pick up the image externally, s... [18:17:43] (03update) 10fnegri: Remove outdated warning from README [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/100 (https://phabricator.wikimedia.org/T444615) [18:18:24] 10Toolforge, 06tools-platform-team: [builds-api] Remove redundant image tags - https://phabricator.wikimedia.org/T444615#12584186 (10fnegri) a:03fnegri [18:21:51] (03approved) 10taavi: Update Heroku builder image [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1532 (https://phabricator.wikimedia.org/T444615 https://phabricator.wikimedia.org/T445559) (owner: 10fnegri) [18:22:14] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [18:22:41] (03merge) 10fnegri: Update Heroku builder image [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1532 (https://phabricator.wikimedia.org/T444615 https://phabricator.wikimedia.org/T445559) [18:22:45] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44586505718 (https://github.com/cluebotng/component-configs/commits/397027bdf00cb3bc48057a4b0376c0ea5b049b23) [18:22:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [18:23:26] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12584203 (10fnegri) > I see the builder change waa deployed, but can we also get the MR merged. Done! [18:25:16] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12584205 (10fnegri) [18:25:52] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44586542810 (https://github.com/cluebotng/component-configs/commits/397027bdf00cb3bc48057a4b0376c0ea5b049b23) [18:25:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [18:26:06] 10Toolforge, 06tools-platform-team: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12584208 (10fnegri) [18:30:45] (03approved) 10fnegri: Add kube-container-updater [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1533 (https://phabricator.wikimedia.org/T314705) (owner: 10taavi) [18:32:15] (03update) 10taavi: Add kube-container-updater [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1533 (https://phabricator.wikimedia.org/T314705) [18:32:57] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics [18:34:04] (03merge) 10taavi: Add kube-container-updater [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1533 (https://phabricator.wikimedia.org/T314705) [18:34:25] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component kube-container-updater [18:34:40] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kube-container-updater [18:39:43] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44589673118 (https://github.com/cluebotng/component-configs/commits/29d0d4824983a5248e149d281ec0c0abbcd1a942) [18:39:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [18:40:14] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44589515234 (https://github.com/cluebotng/component-configs/commits/33776fc9780cb659714b69bf0245893a5f63a149) [18:40:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [18:44:29] (03open) 10tlepage: chart: pin nginx to 1.30.6 [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/118 (https://phabricator.wikimedia.org/T440736) [18:50:46] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics [18:52:47] 10Toolforge, 06tools-platform-team: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12584275 (10DamianZaremba) >>! In T445559#12584203, @fnegri wrote: > Done! Thanks - all green on my end + version pins deployed for cluebot images. [18:56:44] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [18:59:35] (03update) 10taavi: Add .gitreview [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/4 [18:59:36] (03update) 10taavi: chart: Use tools-harbor by default [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/5 [18:59:41] (03open) 10taavi: Add .gitreview [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/4 [18:59:43] (03open) 10taavi: chart: Use tools-harbor by default [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/5 [19:02:16] (03update) 10taavi: build: Disable golangci-lint [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/6 [19:02:17] (03update) 10taavi: Add .gitreview [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/4 [19:02:18] (03update) 10taavi: chart: Use tools-harbor by default [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/5 [19:02:32] (03open) 10taavi: build: Disable golangci-lint [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/6 [19:02:36] (03update) 10taavi: Add .gitreview [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/4 [19:02:38] (03update) 10taavi: chart: Use tools-harbor by default [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/5 [19:02:40] (03update) 10taavi: build: Disable golangci-lint [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/6 [19:03:00] (03approved) 10fnegri: chart: Use tools-harbor by default [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/5 (owner: 10taavi) [19:04:13] (03open) 10tlepage: auth: superuser patterns must match the whole path [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/119 (https://phabricator.wikimedia.org/T440736) [19:05:29] (03approved) 10fnegri: Add .gitreview [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/4 (owner: 10taavi) [19:05:37] (03approved) 10fnegri: build: Disable golangci-lint [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/6 (owner: 10taavi) [19:06:23] (03update) 10taavi: build: Disable golangci-lint [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/6 [19:06:25] (03merge) 10taavi: build: Disable golangci-lint [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/6 [19:06:27] (03update) 10taavi: Add .gitreview [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/4 [19:07:19] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [19:09:14] (03merge) 10taavi: Add .gitreview [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/4 [19:09:16] (03update) 10taavi: chart: Use tools-harbor by default [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/5 [19:10:08] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: kube-container-updater: bump to 0.0.17-20261119190642-2c5ae9fe [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1537 [19:13:43] 10Toolforge, 06tools-platform-team: Define a safe process for mass-restarting tools - https://phabricator.wikimedia.org/T445580 (10fnegri) 03NEW [19:14:30] 10Toolforge, 06tools-platform-team: Define a safe process for mass-restarting tools - https://phabricator.wikimedia.org/T445580#12584361 (10fnegri) [19:14:45] (03merge) 10taavi: chart: Use tools-harbor by default [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/5 [19:15:18] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: kube-container-updater: bump to 0.0.18-20261119190933-a1056c37 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1537 [19:15:23] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: kube-container-updater: bump to 0.0.18-20261119190933-a1056c37 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1537 [19:18:22] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: kube-container-updater: bump to 0.0.19-20261119191459-27628cd0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1537 [19:18:26] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: kube-container-updater: bump to 0.0.19-20261119191459-27628cd0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1537 [19:18:36] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component kube-container-updater [19:26:48] 10Cloud-VPS, 06tools-infrastructure-team, 10Continuous-Integration-Infrastructure, 06Infrastructure-Foundations, and 2 others: Slow connections from Cloud VPS to cdn.cypress.io make CI builds take up to 50 minutes - https://phabricator.wikimedia.org/T445584 (10pwangai) 03NEW [19:31:59] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/81 (owner: 10l10n-bot) [19:32:13] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/81 (owner: 10l10n-bot) [19:32:55] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/55 (owner: 10l10n-bot) [19:33:18] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/55 (owner: 10l10n-bot) [19:34:16] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kube-container-updater [19:34:49] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component kube-container-updater [19:52:37] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kube-container-updater [19:55:35] (03open) 10tlepage: nginx: make /auth internal [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/120 (https://phabricator.wikimedia.org/T440736) [19:57:27] (03merge) 10taavi: kube-container-updater: bump to 0.0.19-20261119191459-27628cd0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1537 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:03:55] (03merge) 10ibrahemqasim: Refactor job syncers and admin views [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/46 [20:19:04] (03open) 10sabrsl: ui: refonte interface et correctifs moteur [toolforge-repos/ovixbot] - 10https://gitlab.wikimedia.org/toolforge-repos/ovixbot/-/merge_requests/3 [20:22:39] (03merge) 10sabrsl: ui: refonte interface et correctifs moteur [toolforge-repos/ovixbot] - 10https://gitlab.wikimedia.org/toolforge-repos/ovixbot/-/merge_requests/3 [20:32:14] 10Cloud-VPS, 06tools-infrastructure-team, 10Continuous-Integration-Infrastructure, 06Infrastructure-Foundations, and 2 others: Slow connections from Cloud VPS to cdn.cypress.io make CI builds take up to 50 minutes - https://phabricator.wikimedia.org/T445584#12584573 (10pwangai) 05Open→03Invalid CI... [20:38:38] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.34.11 (T414595) [20:38:38] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade to 1.34.11 (T414595) [20:38:42] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [20:39:11] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.34.12 (T414595) [20:56:34] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade to 1.34.12 (T414595) [20:56:39] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [21:02:37] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [21:02:42] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [21:05:01] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [21:05:04] (03open) 10tlepage: auth: let trusted services authenticate with a bearer token [repos/cloud/toolforge/api-gateway] (fullmatch-superuser-patterns) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/121 (https://phabricator.wikimedia.org/T440736) [21:08:07] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [21:08:12] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [21:08:28] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: api-gateway: Support bearer-token auth for Striker - https://phabricator.wikimedia.org/T440736#12584701 (10TLepage-WMF) In **bold** the MRs that block this task: - **[[ https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merg... [21:08:30] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [21:10:29] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [21:10:53] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [21:12:11] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico [21:12:15] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico [21:12:29] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico [21:13:28] 10Toolforge, 06tools-platform-team: Define a safe process for mass-restarting tools - https://phabricator.wikimedia.org/T445580#12584733 (10fnegri) I am restarting 16 buildpack-based tools because of {T445559}. Roughly I'm following this procedure, which is usually working fine, but has lots of edge cases wher... [21:29:21] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico [21:30:11] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [21:30:15] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [21:30:35] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [21:32:42] (03update) 10sonali-7: working db functions for adding links [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/37 [21:41:44] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [21:41:48] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [21:42:08] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [22:06:12] 10Toolforge, 06tools-platform-team: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12584983 (10fnegri) > tools using PHP buildpack restarted/upgraded There are currently 16 tools using buildpacks and running PHP, all were running PHP 8.5.12 until this morning. I am upgrading all of... [22:20:58] 10Toolforge, 06tools-platform-team: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12585037 (10DamianZaremba) >>! In T445559#12584983, @fnegri wrote: > @DamianZaremba I haven't touched `cluebotng` and `cluebot3` as I saw you already redeployed them. However, there's a single pod in `c... [22:21:44] 10Toolforge, 06tools-platform-team: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12585039 (10fnegri) > That is running a back-fill job which I already restarted once today, it should finish overnight, if not then I can create an updated job in the morning. That works, thanks! [23:58:29] (03open) 10ibrahemqasim: Add per-article translation leaderboard pages [toolforge-repos/mdwiki] (up) - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/47 [23:59:04] (03open) 10ibrahemqasim: Track detailed outcomes for view update jobs [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/48