[00:06:55] (03update) 10ibrahemqasim: Track detailed outcomes for view update jobs [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/48 [00:07:22] (03merge) 10ibrahemqasim: Track detailed outcomes for view update jobs [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/48 [00:07:33] (03update) 10ibrahemqasim: Add per-article translation leaderboard pages [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/47 [00:08:55] (03update) 10ibrahemqasim: Add per-article translation leaderboard pages [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/47 [00:21:42] (03update) 10bd808: Set bot mode (+B) when connecting [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/82 (https://phabricator.wikimedia.org/T445359) (owner: 10hashar) [00:31:15] 10Toolforge, 06tools-platform-team: Define a safe process for mass-restarting tools - https://phabricator.wikimedia.org/T445580#12585548 (10bd808) >>! In T445580#12584733, @fnegri wrote: > * `toolforge build list` > ** trigger a new build with `toolforge build start` and the same Git repo used by the most rece... [00:35:23] (03update) 10bd808: Set bot mode (+B) when connecting [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/82 (https://phabricator.wikimedia.org/T445359) (owner: 10hashar) [00:35:25] (03approved) 10bd808: Set bot mode (+B) when connecting [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/82 (https://phabricator.wikimedia.org/T445359) (owner: 10hashar) [00:35:37] (03merge) 10bd808: Set bot mode (+B) when connecting [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/82 (https://phabricator.wikimedia.org/T445359) (owner: 10hashar) [00:41:58] (03update) 10bd808: phorge: Replace frozen maniphest.info with maniphest.search [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/72 (https://phabricator.wikimedia.org/T402454) (owner: 10h2o) [01:03:32] (03open) 10bd808: channels: Add Development-Metrics to #wikimedia-releng [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/84 [01:08:59] (03merge) 10bd808: channels: Add Development-Metrics to #wikimedia-releng [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/84 [01:31:28] (03update) 10ibrahemqasim: Add per-article translation leaderboard pages [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/47 [01:44:50] (03update) 10ibrahemqasim: Add per-article translation leaderboard pages [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/47 [02:13:32] (03update) 10ibrahemqasim: Add per-article translation leaderboard pages [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/47 [02:32:27] (03update) 10ibrahemqasim: Add per-article translation leaderboard pages [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/47 [02:32:33] (03update) 10ibrahemqasim: Add per-article translation leaderboard pages [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/47 [02:32:45] (03approved) 10ibrahemqasim: Add per-article translation leaderboard pages [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/47 [02:34:58] (03merge) 10ibrahemqasim: Add per-article translation leaderboard pages [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/47 [02:50:54] (03open) 10ibrahemqasim: Process pageview updates one title at a time [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/49 [04:50:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:50:44] 10Tool-glam-tool-hospital, 10MediaWiki-extensions-Page_Forms: Add new input type for storing Wikidata/Wikibase QID (while user selects a human-readable label) - https://phabricator.wikimedia.org/T430098#12585881 (10Yaron_Koren) [05:20:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:43:28] (03update) 10h2o: phorge: Replace frozen maniphest.info with maniphest.search [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/72 (https://phabricator.wikimedia.org/T402454) [05:45:43] (03update) 10h2o: phorge: Replace frozen maniphest.info with maniphest.search [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/72 (https://phabricator.wikimedia.org/T402454) [05:46:20] (03update) 10h2o: phorge: Replace frozen maniphest.info with maniphest.search [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/72 (https://phabricator.wikimedia.org/T402454) [07:59:10] 10Toolforge, 06tools-infrastructure-team: Toolforge: Ensure long-running Kubernetes pods get container updates applied - https://phabricator.wikimedia.org/T314705#12586131 (10taavi) The first dry-run on `tools` crashed: `lang=shell-session taavi@tools-bastion-14:~ $ kubectl get pod -n kube-container-updater N... [08:13:26] 10Toolforge, 06tools-platform-team: [sample apps] update components config to expected standard - https://phabricator.wikimedia.org/T444784#12586149 (10aputhin) p:05Triage→03Medium [08:13:54] 10Toolforge, 06tools-platform-team: [sample-complex-app-backend] failing to deploy since last builder update - https://phabricator.wikimedia.org/T444783#12586154 (10aputhin) p:05Triage→03Medium a:03komla [08:17:06] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44700842720 (https://github.com/cluebotng/component-configs/commits/60257f6606490e1c119184ff1c4f389c4c653843) [08:17:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [08:17:49] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44700842665 (https://github.com/cluebotng/component-configs/commits/60257f6606490e1c119184ff1c4f389c4c653843) [08:17:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [08:18:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:23:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:55:47] (03open) 10taavi: Ignore pods without pod status data [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/7 [09:00:14] (03merge) 10taavi: Ignore pods without pod status data [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/7 [09:04:10] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: kube-container-updater: bump to 0.0.20-20261120090032-df6f2606 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1538 [09:04:14] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: kube-container-updater: bump to 0.0.20-20261120090032-df6f2606 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1538 [09:04:18] 10Striker, 06tools-platform-team, 03WMDE-TechWish-Kanban: Unable to request Toolforge membership - https://phabricator.wikimedia.org/T444782#12586285 (10Tobi_WMDE_SW) a:05fnegri→03None [09:05:03] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component kube-container-updater [09:05:10] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kube-container-updater [09:06:04] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component kube-container-updater [09:06:11] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kube-container-updater [09:06:18] (03merge) 10taavi: kube-container-updater: bump to 0.0.20-20261120090032-df6f2606 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1538 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:17:41] (03open) 10shivnagi: T445001: Website: Use the shared Button component everywhere [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/122 (https://phabricator.wikimedia.org/T445001) [09:18:33] (03open) 10taavi: Improve logging and error handling [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/8 [09:31:26] (03merge) 10taavi: Improve logging and error handling [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/8 [09:32:31] (03update) 10shivnagi: T445001: Website: Use the shared Button component everywhere [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/122 (https://phabricator.wikimedia.org/T445001) [09:32:58] (03update) 10shivnagi: T445001: Website: Use the shared Button component everywhere [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/122 (https://phabricator.wikimedia.org/T445001) [09:35:26] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: kube-container-updater: bump to 0.0.20-20261120093137-b20be480 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1539 [09:35:30] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: kube-container-updater: bump to 0.0.20-20261120093137-b20be480 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1539 [09:36:59] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component kube-container-updater [09:37:06] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kube-container-updater [09:37:13] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component kube-container-updater [09:37:21] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kube-container-updater [09:38:25] (03merge) 10taavi: kube-container-updater: bump to 0.0.20-20261120093137-b20be480 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1539 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:42:21] (03open) 10taavi: Handle pods that failed to pull the specified image [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/9 [09:51:22] (03merge) 10taavi: Handle pods that failed to pull the specified image [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/9 [09:55:03] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: kube-container-updater: bump to 0.0.20-20261120095135-fd0a3712 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1540 [09:55:13] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component kube-container-updater [09:55:20] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kube-container-updater [09:55:23] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component kube-container-updater [09:55:30] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kube-container-updater [09:59:17] (03merge) 10taavi: kube-container-updater: bump to 0.0.20-20261120095135-fd0a3712 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1540 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:01:12] (03update) 10taavi: kube-container-updater: Turn off dry-run mode in toolsbeta [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1541 (https://phabricator.wikimedia.org/T314705) [10:01:13] (03update) 10taavi: kube-container-updater: Turn off dry-run mode in tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1542 (https://phabricator.wikimedia.org/T314705) [10:01:21] (03open) 10taavi: kube-container-updater: Turn off dry-run mode in toolsbeta [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1541 (https://phabricator.wikimedia.org/T314705) [10:01:23] (03update) 10taavi: kube-container-updater: Turn off dry-run mode in toolsbeta [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1541 (https://phabricator.wikimedia.org/T314705) [10:01:23] (03open) 10taavi: kube-container-updater: Turn off dry-run mode in tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1542 (https://phabricator.wikimedia.org/T314705) [10:05:06] 10Toolforge, 06tools-infrastructure-team, 13Patch-For-Review: Toolforge: Ensure long-running Kubernetes pods get container updates applied - https://phabricator.wikimedia.org/T314705#12586430 (10taavi) Dry-run on `tools` is now completing without errors. [10:19:27] (03open) 10taavi: Avoid bringing down all replicas of a single Deployment at once [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/10 [10:28:37] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate etytree away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435770#12586513 (10Epantaleo) Hello, Ferdinando Traversa helped solve this issue (ferdi2005). Thanks a lot to everyone who helped and was so kind. Let me know I need to do... [10:38:34] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate etytree away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435770#12586564 (10taavi) >>! In T435770#12586513, @Epantaleo wrote: > Hello, Ferdinando Traversa helped solve this issue (ferdi2005). Thanks a > lot to everyone who helped... [10:40:25] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Engineering, and 4 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#12586612 (10Gehel) [10:40:58] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Engineering, and 4 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#12586624 (10Gehel) [10:41:06] 10Cloud Services Proposals, 10Data-Services, 06tools-platform-team, 06Data-Persistence, and 2 others: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#12586626 (10Gehel) [10:49:55] 10Data-Services, 06tools-platform-team, 10BetaFeatures, 06Data-Engineering, and 5 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#12586830 (10Gehel) [11:01:45] FIRING: [3x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_toolserver_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:06:45] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:09:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:14:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:23:46] 10Toolforge, 06tools-platform-team: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12586902 (10aputhin) [11:24:49] (03update) 10tlepage: api: /firing returns the alerts firing now [repos/cloud/toolforge/alerts-api] (reduce-prometheus-calls) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/22 (https://phabricator.wikimedia.org/T441144) [11:25:51] 10Toolforge, 06tools-platform-team, 07Essential-Work: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12586910 (10aputhin) 05Open→03Resolved Marking this as Resolved, since we took the planned actions on the platform side. If any problems arise, please reach out. [11:51:45] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:56:45] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:01:17] (03merge) 10taavi: Avoid bringing down all replicas of a single Deployment at once [repos/cloud/toolforge/kube-container-updater] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/10 [12:05:11] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: kube-container-updater: bump to 0.0.20-20261120120134-1b4c192b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1543 [12:05:36] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component kube-container-updater [12:05:43] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kube-container-updater [12:05:48] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component kube-container-updater [12:05:56] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kube-container-updater [12:06:04] (03merge) 10taavi: kube-container-updater: bump to 0.0.20-20261120120134-1b4c192b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1543 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:06:39] (03update) 10taavi: kube-container-updater: Turn off dry-run mode in tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1542 (https://phabricator.wikimedia.org/T314705) [12:06:40] (03update) 10taavi: kube-container-updater: Turn off dry-run mode in toolsbeta [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1541 (https://phabricator.wikimedia.org/T314705) [12:24:34] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component kube-container-updater [12:24:36] (03merge) 10taavi: kube-container-updater: Turn off dry-run mode in toolsbeta [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1541 (https://phabricator.wikimedia.org/T314705) [12:24:42] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kube-container-updater [12:24:44] (03update) 10taavi: kube-container-updater: Turn off dry-run mode in tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1542 (https://phabricator.wikimedia.org/T314705) [12:25:02] 10Toolforge, 06tools-platform-team, 07Essential-Work: Upgrade Toolforge PHP images - https://phabricator.wikimedia.org/T445559#12587053 (10DamianZaremba) >>! In T445559#12585039, @fnegri wrote: > That works, thanks! Script finished and job is now deleted. [12:27:23] 10Toolforge, 06tools-infrastructure-team, 13Patch-For-Review: Toolforge: Ensure long-running Kubernetes pods get container updates applied - https://phabricator.wikimedia.org/T314705#12587055 (10taavi) Non-dry-run tested to work as expected on `toolsbeta` now. I plan to enable it for real on `tools` on Monday. [12:40:08] (03open) 10tlepage: api: a header naming another tool is 403, not 401 [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/23 [12:59:24] (03PS1) 10Ladsgroup: Dummy conduit token for ProdPasteBot [labs/private] - 10https://gerrit.wikimedia.org/r/1372694 [13:09:13] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [13:09:18] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [13:09:38] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [13:12:23] 10Toolforge, 06tools-platform-team: Define a safe process for mass-restarting tools - https://phabricator.wikimedia.org/T445580#12587212 (10DamianZaremba) >>! In T445580#12585548, @bd808 wrote: > It seems like a forced container rebuild like this should use the same git commit as the target. For tools that are... [13:14:15] (03CR) 10Ladsgroup: [V:03+2 C:03+2] Dummy conduit token for ProdPasteBot [labs/private] - 10https://gerrit.wikimedia.org/r/1372694 (owner: 10Ladsgroup) [13:28:50] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [13:28:55] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [13:29:43] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [13:45:03] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [13:45:08] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [13:45:38] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [13:48:37] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [13:49:19] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [14:01:05] 06tools-infrastructure-team, 06DC-Ops, 06SRE, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12587546 (10Gehel) [14:02:34] 10Toolforge, 06tools-platform-team: Retire wikidata-geo-dumps - https://phabricator.wikimedia.org/T443820#12587586 (10aputhin) [14:03:04] 10Toolforge, 06tools-platform-team, 07Documentation: Toolforge API documentation - API requests failing - https://phabricator.wikimedia.org/T445658 (10KBach) 03NEW [14:09:27] 10Tool-wikinewsie: Make Wikinewsie related-articles feature more Wikidata-driven - https://phabricator.wikimedia.org/T444346#12587651 (10ZhaoFJx) p:05Triage→03Medium [14:11:50] 10Toolforge, 06tools-platform-team, 07Documentation: Toolforge API documentation - API requests failing - https://phabricator.wikimedia.org/T445658#12587671 (10DamianZaremba) Unfortunately, you can't really 'try it out' as the api gateway requires certificate authentication for any useful endpoint (/builds/v... [14:16:48] 10Tool-wikinewsie: Wikinewsie history page for each article card - https://phabricator.wikimedia.org/T444224#12587748 (10ZhaoFJx) p:05Triage→03Medium [14:24:11] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [14:24:16] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [14:25:13] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [14:32:26] 10Tool-wikinewsie: Wikinewsie language selector fails to show full language names on mobile - https://phabricator.wikimedia.org/T439939#12587859 (10ZhaoFJx) a:03Tduk [14:32:48] 10Tool-wikinewsie: Wikinewsie language selector fails to show full language names on mobile - https://phabricator.wikimedia.org/T439939#12587864 (10ZhaoFJx) 05Open→03In progress [14:37:05] 10Tool-wikinewsie: Wikinewsie language selector fails to show full language names on mobile - https://phabricator.wikimedia.org/T439939#12587892 (10Tduk) Seems it may be a problem in desktop too. Investigating what triggers it. [14:37:16] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [14:37:21] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [14:38:03] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.34.12 (T414595) [14:45:57] (03update) 10zhaofjx: Speed up first load (narrow PetScan query / background refresh / smaller thumbs) [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/41 (owner: 10kevinpayravi) [14:50:06] 10Tool-wikinewsie: Extend time windows to 7d or longer on Wikinewsie Front Page - https://phabricator.wikimedia.org/T444223#12588000 (10ZhaoFJx) 05Resolved→03In progress [14:50:27] 10Tool-wikinewsie: Extend time windows to 7d or longer on Wikinewsie Front Page - https://phabricator.wikimedia.org/T444223#12588002 (10ZhaoFJx) Re-open for patch issues. [14:51:49] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-10 [14:51:59] 10Tool-wikinewsie: Dropdown selection type for flexible Wikinewsie subportals - https://phabricator.wikimedia.org/T434608#12588005 (10ZhaoFJx) 05Stalled→03In progress [14:53:26] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=0) for toolsbeta-test-k8s-control-10 [14:57:26] (03update) 10zhaofjx: T436544: Article linking API over the links data layer [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/40 (owner: 10kitkat1100) [14:57:48] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.33.13 to 1.34.12 (T414595) [14:57:53] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [14:59:17] (03PS1) 10Majavah: toolforge: k8s: Check for broken static restart attempts [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1372717 [14:59:17] (03PS1) 10Majavah: wmcs_libs: Ensure all cookbook runs use a durable shell [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1372718 [15:02:49] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.33.13 to 1.34.12 (T414595) [15:03:03] 10Tool-wikinewsie: Deployment Failure after Wikinewsie Localization Update - https://phabricator.wikimedia.org/T445172#12588093 (10ZhaoFJx) a:03ZhaoFJx [15:03:06] 10Tool-wikinewsie: Deployment Failure after Wikinewsie Localization Update - https://phabricator.wikimedia.org/T445172#12588095 (10ZhaoFJx) 05Open→03In progress [15:04:57] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.33.13 to 1.34.12 (T414595) [15:05:02] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [15:08:26] 10Tool-wikinewsie: Wikinewsie language selector fails to show full language names on mobile - https://phabricator.wikimedia.org/T439939#12588130 (10Tduk) Apparently there is already code to sort of address this, a function called Intl.DisplayNames (which may not be ours?) isn't always returning the right thing.... [15:09:11] 10Tool-wikinewsie: Wikinewsie language selector fails to show full language names - https://phabricator.wikimedia.org/T439939#12588141 (10Tduk) [15:10:29] 10Tool-wikinewsie: Wikinewsie language selector fails to show full language names - https://phabricator.wikimedia.org/T439939#12588148 (10Tduk) note: in my case I could only show that skr-arab did not map to Saraiki, could not reproduce the two languages mentioned here. that may have been an intermittent problem... [15:11:04] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.33.13 to 1.34.12 (T414595) [15:11:08] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [15:11:25] 10Tool-wikinewsie: Dropdown selection type for flexible Wikinewsie subportals - https://phabricator.wikimedia.org/T434608#12588154 (10ZhaoFJx) [15:16:45] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:16:49] 10Toolforge: Retire wikidata-geo-dumps - https://phabricator.wikimedia.org/T443820#12588175 (10aputhin) [15:17:23] 06tools-platform-team, 06Product Safety and Integrity, 06Security-Team, 10Toolhub, and 3 others: Revert action bypasses object authorization and republishes suppressed revisions in Toolhub - https://phabricator.wikimedia.org/T442629#12588178 (10aputhin) [15:21:45] RESOLVED: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:23:14] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T414595) [15:23:19] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [15:23:27] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T414595) [15:24:11] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T414595) [15:24:47] 06Toolforge-standards-committee, 10Tools, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): Right to fork request: https://wikimap.toolforge.org/ - https://phabricator.wikimedia.org/T440145#12588211 (10DB111) bd808: The comparison is a little flawed, not I consumed resources. T... [15:25:40] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T414595) [15:26:21] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T414595) [15:30:51] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T414595) [15:30:56] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [15:33:21] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-1 from 1.33.13 to 1.34.12 (T414595) [15:33:34] 10Tool-wikinewsie: Wikinewsie language selector fails to show full language names on android - https://phabricator.wikimedia.org/T439939#12588297 (10Tduk) [15:33:52] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-gateway-1 from 1.33.13 to 1.34.12 (T414595) [15:34:06] 10Tool-wikinewsie: Wikinewsie language selector fails to show full language names on android - https://phabricator.wikimedia.org/T439939#12588299 (10Tduk) two unrelated issues, ska-arab is a known issue. this issue only affects android, investigating [15:36:14] FIRING: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-gateway-1.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [15:37:06] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-1 from 1.33.13 to 1.34.12 (T414595) [15:37:11] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [15:37:37] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-1 from 1.33.13 to 1.34.12 (T414595) [15:39:03] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-2 from 1.33.13 to 1.34.12 (T414595) [15:39:44] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-2 from 1.33.13 to 1.34.12 (T414595) [15:40:49] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-3 from 1.33.13 to 1.34.12 (T414595) [15:41:14] RESOLVED: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-gateway-1.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDow [15:41:29] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-3 from 1.33.13 to 1.34.12 (T414595) [15:47:58] FIRING: JobsEmailerNoEmails: No emails sent in the last hour - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/JobsEmailerNoEmails - https://prometheus-alerts.wmcloud.org/?q=alertname%3DJobsEmailerNoEmails [16:07:58] RESOLVED: JobsEmailerNoEmails: No emails sent in the last hour - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/JobsEmailerNoEmails - https://prometheus-alerts.wmcloud.org/?q=alertname%3DJobsEmailerNoEmails [16:26:49] (03open) 10tlepage: api: /history returns past alerts [repos/cloud/toolforge/alerts-api] (add-firing-endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/24 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [17:09:07] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595#12588675 (10Andrew) Toolsbeta is now running k8s 1.34.12 and tests are passing. Before upgrading toolforge k8s I need to upgra... [17:27:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:32:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:41:03] 10Cloud-VPS, 06tools-infrastructure-team: Cannot create EC2 credentials in OS - https://phabricator.wikimedia.org/T444826#12588929 (10Andrew) It sounds like the feature we relied on (s3 tokens bound to an app credential trust) is not coming back. My thoughts are... * Just use the swift API (which can use open... [17:42:18] 10Cloud-VPS, 06tools-infrastructure-team: Cannot create EC2 credentials in OS - https://phabricator.wikimedia.org/T444826#12588934 (10taavi) If we could generate these credentials via Horizon then I think that'd be even better than the previous status. [18:06:26] (03PS5) 10Hridyesh_Gupta: Fix the typo in the "no participants" message on campaign pages [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1053046 (https://phabricator.wikimedia.org/T358396) [18:13:18] (03CR) 10BryanDavis: [C:04-2] "test" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [18:17:01] (03CR) 10Hridyesh_Gupta: "Patch set 5 rebases this onto the current main. Since the campaign page was split up for pagination, the message is used in campaign_table" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1053046 (https://phabricator.wikimedia.org/T358396) (owner: 10Hridyesh_Gupta) [18:32:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:47:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:06:36] 10Tool-wikinewsie: Default time windows on Wikinewsie proportionate to edits/day on the language and portal - https://phabricator.wikimedia.org/T437956#12589473 (10Tduk) do we want to have the bigger timespans available for all languages? or custom spans based on language? how do we want to be able to configure... [20:13:41] (03open) 10tlepage: api: add readiness & liveness probes [repos/cloud/toolforge/alerts-api] (add-history-endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/25 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145 https://phabricator.wikimedia.org/T445148) [20:13:52] 10Tool-wikinewsie: Default time windows on Wikinewsie proportionate to edits/day on the language and portal - https://phabricator.wikimedia.org/T437956#12589525 (10ZhaoFJx) a:03Tduk [20:13:57] 10Tool-wikinewsie: Default time windows on Wikinewsie proportionate to edits/day on the language and portal - https://phabricator.wikimedia.org/T437956#12589529 (10ZhaoFJx) 05Open→03In progress [20:29:55] (03update) 10tlepage: api: add readiness & liveness probes [repos/cloud/toolforge/alerts-api] (add-history-endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/25 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145 https://phabricator.wikimedia.org/T445148) [20:41:29] (03open) 10tlepage: api: serve Prometheus metrics on plain-HTTP 9000 [repos/cloud/toolforge/alerts-api] (readiness-liveness-probes) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/26 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [20:44:43] (03PS1) 10Gerrit Patch Uploader: feat: add dark theme support Change-Id: Ie456ac72048275c6a31d0aaf725f147b9a6317b6 [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1372956 [20:44:43] (03CR) 10Gerrit Patch Uploader: "This commit was uploaded using the Gerrit Patch Uploader [1]." [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1372956 (owner: 10Gerrit Patch Uploader) [20:52:12] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.3 Implement Toolforge UI MVP - https://phabricator.wikimedia.org/T440589#12589650 (10TLepage-WMF) Weekly Asana updates: ## Nov "13th" ### Summary **This week** (phase //Development//): * Facilitated peer review of the open patches for T441144... [20:59:25] 10Tool-itwiki, 10Tool-itwikibot-markadmins-bot: Migrate itwiki's "markadmins" job from tool itwiki to dedicated tool - https://phabricator.wikimedia.org/T442501#12589669 (10valerio.bozzolan) Created tool, and authorized previous maintainers: https://toolsadmin.wikimedia.org/tools/id/itwikibot-markadmins-bot... [21:16:42] 10Tool-itwiki, 10Tool-itwikibot-markadmins-bot: Migrate itwiki's "markadmins" job from tool itwiki to dedicated tool - https://phabricator.wikimedia.org/T442501#12589863 (10valerio.bozzolan) Waiting for authorization: https://it.wikipedia.org/wiki/Wikipedia:Richieste_agli_amministratori/Richieste_correnti#c-V... [21:36:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:36:46] 10Tool-itwiki, 10Tool-itwiki-markadmins-bot: Migrate itwiki's "markadmins" job from tool itwiki to dedicated tool - https://phabricator.wikimedia.org/T442501#12589935 (10valerio.bozzolan) Whopsie, typo. Fixed: https://toolsadmin.wikimedia.org/tools/id/itwiki-markadmins-bot [21:44:24] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace or decommission RESTBase hosts: deployment-restbase(05|-bullseye) (Bullseye deprecation) - https://phabricator.wikimedia.org/T436477#12590041 (10Southparkfan) After fixing RESTBase - only recommendation-api is broken now, but I d... [21:49:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [21:56:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:59:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [22:03:13] 10Tool-itwiki, 10Tool-itwiki-markadmins-bot: Migrate itwiki's "markadmins" job from tool itwiki to dedicated tool - https://phabricator.wikimedia.org/T442501#12590112 (10valerio.bozzolan) Dedicated account created: https://it.wikipedia.org/wiki/Utente:MarkAdmins_Bot Set bot passwords in the tool. I need a c... [22:10:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:10:53] 10Tool-containers, 10Wikibugs: Replace tool-wikibugs/znccontainer with tool-containers/bnc container - https://phabricator.wikimedia.org/T366966#12590147 (10bd808) 05Open→03In progress a:03bd808 [22:14:08] 10Tool-containers, 10Wikibugs: Replace tool-wikibugs/znccontainer with tool-containers/bnc container - https://phabricator.wikimedia.org/T366966#12590167 (10bd808) > the bnc container needs a few more set. `lang=shell-session tools.wikibugs@tools-bastion-14:~$ toolforge envvars create BNC_USER wikibugs name... [22:17:16] 10Tool-containers, 10Wikibugs: Replace tool-wikibugs/znccontainer with tool-containers/bnc container - https://phabricator.wikimedia.org/T366966#12590183 (10bd808) >>! In T366966#11882581, @bd808 wrote: > It would also be neat to figure out how to support pulling in config using the envvars that wikibugs2 uses... [22:17:34] (03update) 10emj912: Add QID lookup helper, extend CI, update test plan [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/39 [22:20:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:17:41] (03merge) 10ibrahemqasim: Process pageview updates one title at a time [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/49 [23:19:19] (03open) 10ibrahemqasim: Fix MediaWiki continuation handling [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/50 [23:24:18] (03merge) 10ibrahemqasim: Fix MediaWiki continuation handling [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/50 [23:55:23] (03open) 10ibrahemqasim: Fix MediaWiki query pagination and errors [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/51 [23:56:30] (03PS1) 10Wandji collins: OAuth: fix the redirect to front end after login [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1372993 (https://phabricator.wikimedia.org/T445421) [23:56:46] (03CR) 10CI reject: [V:04-1] OAuth: fix the redirect to front end after login [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1372993 (https://phabricator.wikimedia.org/T445421) (owner: 10Wandji collins) [23:58:26] (03update) 10ibrahemqasim: Fix MediaWiki query pagination and errors [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/51