[00:08:16] 10Tool-campwiz-nxt: CampWiz Nxt Redesign: Root Path - https://phabricator.wikimedia.org/T415408#11716531 (10Ayaan-mi) Hi @Nokib_Sarkar and @Tiven2240 I have completed the microtask of migrating the CampaignEditForm component to the new structure PR Link : https://github.com/Open-Knowledge-Impact-Foundation/cam... [00:17:09] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' (T406516) [00:17:17] T406516: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516 [00:19:42] 10Tool-delintbot: Use REST API to check if any lint errors were actually fixed before saving edit - https://phabricator.wikimedia.org/T420294 (10Redmin) 03NEW [00:23:44] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' (T406516) [00:23:45] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' (T406516) [00:23:49] T406516: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516 [00:27:24] 10Tool-Pageviews: Data problems with dumps and siteviews tool - https://phabricator.wikimedia.org/T384636#11716591 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal Closing per above. For future reference, if there's missing data, you can file the task under #data-engineering and not #tool-pageviews,... [00:28:13] 10Tool-delintbot: Explore visual regression testing - https://phabricator.wikimedia.org/T415882#11716598 (10Redmin) [00:29:32] 10Tool-Pageviews: Require login for mass-querying tools like Massviews - https://phabricator.wikimedia.org/T420295 (10MusikAnimal) 03NEW p:05Triage→03Medium [00:30:18] 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API: massviews hits 429 Too Many Requests despite making requests synchronously - https://phabricator.wikimedia.org/T219857#11716619 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal For most users, things should be back... [00:30:27] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' (T406516) [00:30:28] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' (T406516) [00:30:35] T406516: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516 [00:37:41] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' (T406516) [00:37:42] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' (T406516) [00:37:49] T406516: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516 [00:44:27] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' (T406516) [00:44:28] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' (T406516) [00:44:34] T406516: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516 [00:50:54] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' (T406516) [00:50:55] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' (T406516) [00:51:00] T406516: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516 [00:52:56] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:57:29] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' (T406516) [00:57:30] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1046.eqiad.wmnet' (T406516) [00:57:35] T406516: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516 [00:57:56] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:04:15] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' (T406516) [01:04:16] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' (T406516) [01:04:22] T406516: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516 [01:11:04] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' (T406516) [01:11:05] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' (T406516) [01:11:09] T406516: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516 [01:17:55] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' (T406516) [01:17:56] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' (T406516) [01:18:02] T406516: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516 [01:25:02] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' (T406516) [01:25:09] T406516: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516 [01:28:58] 10Tool-delintbot: Evaluate integrating a HTML5 linter - https://phabricator.wikimedia.org/T420300 (10Redmin) 03NEW [01:29:30] 10Tool-delintbot: Evaluate integrating a HTML5 linter - https://phabricator.wikimedia.org/T420300#11716774 (10Redmin) [02:43:15] 06cloud-services-team, 10Cloud-VPS: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516#11716855 (10Andrew) This is all done except for the cloudvirtlocal hosts. [03:02:18] 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API: massviews hits 429 Too Many Requests despite making requests synchronously - https://phabricator.wikimedia.org/T219857#11716872 (10Hawkeye7) @MusikAnimal: I tried to run a massviews and got: https://en.wikipedia.org is no... [06:00:28] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1253474 (owner: 10L10n-bot) [07:00:41] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: cloudcumin not able to communicate with openstack.eqiad1.wikimediacloud.org:25000 anymore - https://phabricator.wikimedia.org/T419996#11717076 (10ayounsi) Thanks, patch lgtm ! and indeed using the proxies seems the best here. [07:30:23] 06cloud-services-team, 10Toolforge, 10Ultraviolet: RedWarn/Ultraviolet vandalism statistics show "INF" RPM occasionally - https://phabricator.wikimedia.org/T420314 (10OzmoOzmo) 03NEW [07:44:37] 10Tool-delintbot: Evaluate integrating a HTML5 validator - https://phabricator.wikimedia.org/T420300#11717163 (10Redmin) [08:45:14] 10Tool-campwiz-nxt: Migration of CampWiz NXT to toolforge - https://phabricator.wikimedia.org/T394515#11717374 (10programmer-olashow) @Nokib_Sarkar Hello, my name is Illias Olanrewaju, and I’m a GSoC 2026 applicant interested in contributing to the CampWiz NxT project. I’ve been reviewing the project and would... [09:07:17] FIRING: JobUnavailable: Reduced availability for job cloudlb-haproxy in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:12:17] RESOLVED: JobUnavailable: Reduced availability for job cloudlb-haproxy in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:27:23] 10Tool-inteGraality: Table not updating - https://phabricator.wikimedia.org/T302314#11717684 (10JeanFred) I switched the dashboard to use QLever, dashboard is now updating https://www.wikidata.org/wiki/Special:Diff/2472881133 [10:27:25] 10Tool-inteGraality: Table not updating - https://phabricator.wikimedia.org/T302314#11717685 (10JeanFred) 05Invalid→03Resolved [10:58:46] 10Cloud-Services, 06tools-platform-team: Unlink accounts - https://phabricator.wikimedia.org/T420329 (10mickeybarber) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific pr... [11:22:44] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Permanently set 'noout' for cloudceph - https://phabricator.wikimedia.org/T419877#11717883 (10dcaro) For the osd down, we can have a non-critical alert so the oncall of the week will handle eventually (or automatically create a task out of it to handle). [11:47:03] 10Tool-wd-image-positions: wd-image-positions tool failing to load - https://phabricator.wikimedia.org/T420332 (10taavi) 03NEW [12:28:05] 10Tool-wiktlexbot: Properly retrieve lemmas for sign languages - https://phabricator.wikimedia.org/T418890#11718203 (10Redmin) [12:28:35] 10Cloud-Services, 06tools-platform-team: Unlink accounts - https://phabricator.wikimedia.org/T420329#11718204 (10Yug) [12:29:04] 10Tool-wiktlexbot: Properly retrieve lemmas for sign languages - https://phabricator.wikimedia.org/T418890#11718206 (10Redmin) a:03Redmin [12:29:43] 10Cloud-Services, 06tools-platform-team: Unlink accounts - https://phabricator.wikimedia.org/T420329#11718207 (10Yug) [12:31:02] 10Cloud-Services, 06tools-platform-team: Unlink accounts - https://phabricator.wikimedia.org/T420329#11718208 (10Yug) [12:32:01] 10Cloud-Services, 06tools-platform-team: Unlink WM and Phab accounts from Dev accounts - https://phabricator.wikimedia.org/T420329#11718211 (10Yug) [12:32:40] 10Cloud-Services, 06tools-platform-team: Unlink WM and Phab accounts from Dev accounts - https://phabricator.wikimedia.org/T420329#11718213 (10Yug) [12:34:09] 10Cloud-Services, 06tools-platform-team: Unlink WM and Phab accounts from Dev accounts - https://phabricator.wikimedia.org/T420329#11718220 (10Yug) [12:39:28] 10Cloud-Services, 06tools-platform-team: Unlink WM and Phab accounts from Dev accounts - https://phabricator.wikimedia.org/T420329#11718248 (10Yug) >>! In T420329#11717825, @Herald wrote: > The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia... [12:41:23] 06tools-platform-team, 10LDAP-Access-Requests, 07LDAP: Unlink WM and Phab accounts from Dev accounts - https://phabricator.wikimedia.org/T420329#11718260 (10Yug) [12:41:44] 06tools-platform-team, 10LDAP-Access-Requests, 07LDAP: Unlink WM and Phab accounts from Dev accounts - https://phabricator.wikimedia.org/T420329#11718268 (10Yug) [12:42:31] 06cloud-services-team, 10Toolforge: Upgrade or retire tools-package-builder-04 - https://phabricator.wikimedia.org/T401819#11718274 (10taavi) [12:42:32] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 10Toolforge: [infra] Toolforge: migrate to Debian Bookworm or later - https://phabricator.wikimedia.org/T387005#11718275 (10taavi) [12:42:41] 06cloud-services-team, 10Toolforge: Retire tools-package-builder-04 - https://phabricator.wikimedia.org/T401819#11718277 (10taavi) [12:43:07] 06cloud-services-team, 10Toolforge: Retire tools-package-builder-04 - https://phabricator.wikimedia.org/T401819#11718281 (10taavi) a:03taavi Looking at `last` it seems pretty safe to just retire it entirely. [12:57:37] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' (T406516) [12:57:42] T406516: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516 [13:00:35] 06cloud-services-team, 10Cloud-VPS: cloudcumin not able to communicate with openstack.eqiad1.wikimediacloud.org:25000 anymore - https://phabricator.wikimedia.org/T419996#11718352 (10cmooney) FWIW I agree it'd be better if the web proxy could be used here, as conceptually this is "private WMF host needs access... [13:05:00] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' (T406516) [13:05:06] T406516: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516 [13:16:05] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' (T406516) [13:16:11] T406516: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516 [13:20:44] (03CR) 10Andrew Bogott: [C:03+2] Remove cloudweb.set_maintenance and cloudweb_unset_maintenance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1253627 (owner: 10Andrew Bogott) [13:23:19] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' (T406516) [13:23:26] T406516: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516 [13:25:21] (03Merged) 10jenkins-bot: Remove cloudweb.set_maintenance and cloudweb_unset_maintenance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1253627 (owner: 10Andrew Bogott) [13:25:26] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' (T406516) [13:33:02] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' (T406516) [13:33:12] T406516: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516 [13:47:57] 06cloud-services-team, 10Horizon, 13Patch-For-Review: Update our Horizon release to 2025.2 - https://phabricator.wikimedia.org/T405117#11718630 (10Andrew) 05Open→03Resolved [13:59:54] 10Cloud-VPS (Project-requests), 10Lingua-Libre: Clean up WMFR sysop triple idm account - https://phabricator.wikimedia.org/T420358 (10Yug) 03NEW [14:05:16] 06cloud-services-team, 10Cloud-VPS: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516#11718733 (10Andrew) 05Open→03Resolved [14:06:12] 06cloud-services-team, 10Cloud-VPS: cloudcumin not able to communicate with openstack.eqiad1.wikimediacloud.org:25000 anymore - https://phabricator.wikimedia.org/T419996#11718738 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This is fixed! Thank you all for your help, and will follow up with anothe... [14:18:14] 06cloud-services-team, 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T420153#11718796 (10taavi) [14:18:25] 06cloud-services-team, 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T420152#11718798 (10taavi) [14:44:59] (03open) 10kavaljeetsingh: Fix self closing tags [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/9 [14:45:40] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#11718947 (10dcaro) Created the entry in wikitech: https://wikitech.wikime... [14:46:31] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node [14:46:35] 10Tool-delintbot: Fix cases of tags not being closed correctly - https://phabricator.wikimedia.org/T417483#11718954 (10Kavaljeet_Singh) Hi @Redmin, I’ve implemented the fix for self closing tags along with namespace handling and ensured all tests pass. Please let me know if any changes are needed. -thanks [14:48:31] (03update) 10dcaro: build: add use_deprecated_versions [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/155 (https://phabricator.wikimedia.org/T380127) [14:49:59] (03update) 10dcaro: build: add use_deprecated_versions [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/155 (https://phabricator.wikimedia.org/T380127) [14:58:46] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) [15:00:17] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [15:00:52] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [15:04:07] (03update) 10r4356th: Fix self closing tags [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/9 (https://phabricator.wikimedia.org/T417483) (owner: 10kavaljeetsingh) [15:04:17] FIRING: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:08:30] 06cloud-services-team, 10Cloud-VPS: Support connections from bastion to other hosts - https://phabricator.wikimedia.org/T103552#11719105 (10taavi) 05Open→03Declined Boldly declining. [15:09:17] RESOLVED: [3x] JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:09:27] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [15:09:47] FIRING: [3x] JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:10:00] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [15:10:55] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [15:11:28] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [15:12:27] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [15:12:42] 06cloud-services-team, 10Cloud-VPS: Deprecate and remove 'bastion-restricted' hosts - https://phabricator.wikimedia.org/T420213#11719145 (10taavi) FWIW I see some value in having the cumin authorized_keys entries have an IP restriction on a host not accessible to most people, but otherwise agree with the propo... [15:14:47] RESOLVED: [3x] JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:15:41] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [15:21:00] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node [15:28:32] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on toolsbeta-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [15:31:38] 06cloud-services-team, 10Striker: Unlink WM and Phab accounts from Dev accounts - https://phabricator.wikimedia.org/T420329#11719265 (10bd808) This data lives in the local database of #Striker. [15:32:11] 06cloud-services-team, 10Striker: Unlink SUL and Phab accounts from Michabarber in Striker - https://phabricator.wikimedia.org/T420329#11719285 (10bd808) [15:32:54] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) [15:33:46] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [15:37:00] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [15:41:45] 06cloud-services-team, 10Cloud-VPS, 10Lingua-Libre: Configure vanity domain for lingualibre - https://phabricator.wikimedia.org/T419525#11719329 (10dcaro) Hi @Yug, can you follow the steps detailed here https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_VPS_servers_from_the_internet#Va... [15:44:58] (03update) 10dcaro: builds-api: update buildpacks to 24_0.20.7/24_0.21.5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1175 (https://phabricator.wikimedia.org/T380127) [15:45:11] (03update) 10dcaro: builds-api: update buildpacks to 24_0.20.7/24_0.21.5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1175 [15:45:28] 10Tool-delintbot, 13Patch-For-Review: Fix cases of tags not being closed correctly - https://phabricator.wikimedia.org/T417483#11719363 (10Redmin) [15:45:44] 10Tool-delintbot, 13Patch-For-Review: Fix cases of tags not being closed correctly - https://phabricator.wikimedia.org/T417483#11719364 (10Redmin) p:05Low→03Medium [15:46:00] 10Tool-delintbot, 13Patch-For-Review: Fix cases of tags not being closed correctly - https://phabricator.wikimedia.org/T417483#11719365 (10Redmin) 05Open→03In progress [15:49:06] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node [16:05:45] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) [16:14:34] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node [16:18:32] RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on toolsbeta-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [16:33:39] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) [16:36:14] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11719821 (10A_smart_kitten) >>! In T388022#11715419, @Dzahn wrote: > @Aklapper The last question above was about a configur... [16:43:24] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node [16:50:32] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on toolsbeta-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [16:59:25] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#11720045 (10dcaro) Sent the email to announce: https://lists.wikimedia.or... [17:01:12] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) [17:09:49] 10Tools: bldrwnsch.toolforge.org needs migration of cl_to to linktarget - https://phabricator.wikimedia.org/T420145#11720096 (10simon04) 05Open→03Resolved Awesome @Ladsgroup, thank you for your help. 🤗 Fixed via https://github.com/simon04/bldrwnsch/compare/a636e49afe5eb8de236930fa333e70d25d602e37...e4f... [17:10:29] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#11720115 (10aputhin) 05Open→03In progress [17:15:41] 06cloud-services-team, 10Striker: Unlink SUL and Phab accounts from Michabarber in Striker - https://phabricator.wikimedia.org/T420329#11720160 (10Yug) [17:16:11] 06cloud-services-team, 10Striker: Unlink SUL and Phab accounts from Michabarber in Striker - https://phabricator.wikimedia.org/T420329#11720162 (10Yug) [17:18:49] 06cloud-services-team, 10Striker: Unlink SUL and Phab accounts from Michabarber in Striker - https://phabricator.wikimedia.org/T420329#11720171 (10Yug) [17:20:55] 06cloud-services-team, 10Striker: Unlink SUL and Phab accounts from Michabarber in Striker - https://phabricator.wikimedia.org/T420329#11720176 (10Yug) [17:22:13] 10Tool-wd-image-positions: wd-image-positions tool failing to load - https://phabricator.wikimedia.org/T420332#11720182 (10LucasWerkmeister) Should be fixed now, thanks for the report! The perils of having tools translated into more than 50 languages :) ([Link to the fix](https://gitlab.wikimedia.org/toolforge-... [17:22:21] 10Tool-wd-image-positions: wd-image-positions tool failing to load - https://phabricator.wikimedia.org/T420332#11720190 (10LucasWerkmeister) 05Open→03Resolved a:03LucasWerkmeister [17:22:27] 06cloud-services-team, 10Striker: Unlink SUL and Phab accounts from Michabarber in Striker - https://phabricator.wikimedia.org/T420329#11720192 (10Yug) [17:24:17] 06cloud-services-team, 10Striker: Unlink SUL and Phab accounts from Michabarber in Striker - https://phabricator.wikimedia.org/T420329#11720196 (10Yug) [17:33:28] 10Cloud-VPS (Project-requests), 10Lingua-Libre: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11720275 (10Yug) 05Resolved→03Open Hello Everyone, We just found out Michael has a trio of private/private/professional LDAP accounts and we are currently cleaning this u... [17:39:28] 10Cloud-VPS (Project-requests), 10Lingua-Libre: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11720291 (10taavi) 05Open→03Resolved Please do not hijack this task, those are not related to creating the lingualibre project. [17:44:46] 10Tool-openstack-browser, 07Performance Issue: openstack-browser's proxy listing takes a long time to load - https://phabricator.wikimedia.org/T420400 (10A_smart_kitten) 03NEW [17:54:46] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 10Ceph: Permanently set 'noout' for cloudceph - https://phabricator.wikimedia.org/T419877#11720374 (10BTullis) [17:56:32] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-test-k8s-etcd-34 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:05:17] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [18:06:32] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-test-k8s-etcd-34 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:13:12] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [18:13:50] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [18:21:54] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) [18:22:14] 06cloud-services-team, 10Striker: Unlink SUL and Phab accounts from Michabarber in Striker - https://phabricator.wikimedia.org/T420329#11720482 (10bd808) 05Open→03Resolved a:03bd808 `lang=mysql striker_admin@m5-master.eqiad.wmnet(striker)> UPDATE labsauth_labsuser SET sulid=null, sulname=null, sulema... [18:23:05] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node [18:43:32] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) [18:44:22] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [18:52:43] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [18:52:56] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [19:02:13] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) [19:04:14] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node [19:15:32] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance toolsbeta-test-k8s-etcd-35 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [19:23:20] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) [19:25:09] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [19:28:02] RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on toolsbeta-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [19:30:21] FIRING: MaintainKubeusersHang: maintain-kubeusers last finished run is 29.56M minutes old - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersHang [19:34:03] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [19:35:32] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance toolsbeta-test-k8s-etcd-35 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [19:37:48] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [19:40:21] RESOLVED: MaintainKubeusersHang: maintain-kubeusers last finished run is 29.56M minutes old - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersHang [19:41:21] FIRING: MaintainKubeusersHang: maintain-kubeusers last finished run is 29.56M minutes old - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersHang [19:46:37] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) [19:47:25] 10Tool-openstack-browser, 07Performance Issue: openstack-browser's proxy listing takes a long time to load - https://phabricator.wikimedia.org/T420400#11720676 (10A_smart_kitten) ..... and now I load it just now and it takes less than half-a-second. Huh. 🤷‍♀️ I feel like this has also happened to me before tod... [19:47:35] 10Tool-openstack-browser, 07Performance Issue: openstack-browser's proxy listing sometimes takes a long time to load - https://phabricator.wikimedia.org/T420400#11720677 (10A_smart_kitten) [19:50:36] RESOLVED: MaintainKubeusersHang: maintain-kubeusers last finished run is 29.56M minutes old - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersHang [19:53:21] FIRING: MaintainKubeusersHang: maintain-kubeusers last finished run is 29.56M minutes old - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersHang [19:58:12] 10Tool-wikiloves: wikiloves updates fail with "Unknown column 'cl_to' in 'SELECT'" - https://phabricator.wikimedia.org/T420408 (10JeanFred) 03NEW [20:02:58] FIRING: JobsEmailerNoEmails: No emails sent in the last hour - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/JobsEmailerNoEmails - https://prometheus-alerts.wmcloud.org/?q=alertname%3DJobsEmailerNoEmails [20:03:21] RESOLVED: MaintainKubeusersHang: maintain-kubeusers last finished run is 29.56M minutes old - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersHang [20:04:21] FIRING: MaintainKubeusersHang: maintain-kubeusers last finished run is 29.56M minutes old - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersHang [20:18:55] 10Tool-wikimonitor: Unable to Edit SpEL Condition from Mobile - https://phabricator.wikimedia.org/T420409 (10Gerges) 03NEW [20:19:50] 10Tool-wikimonitor, 07patch-welcome: Unable to Edit SpEL Condition from Mobile - https://phabricator.wikimedia.org/T420409#11720773 (10Gerges) p:05Triage→03Medium [20:20:28] 10Tool-wikimonitor, 07good first task, 07patch-welcome: Unable to Edit SpEL Condition from Mobile - https://phabricator.wikimedia.org/T420409#11720776 (10Gerges) Thank you for tagging this task with #good_first_task for Wikimedia newcomers! Newcomers often may not be aware of things that may seem obvious to... [20:23:36] RESOLVED: MaintainKubeusersHang: maintain-kubeusers last finished run is 29.56M minutes old - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersHang [20:27:21] FIRING: MaintainKubeusersHang: maintain-kubeusers last finished run is 29.56M minutes old - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersHang [20:34:32] 10Tool-phab-ban: Consider enabling permanent sessions for clients with poor session scoped cookie handling - https://phabricator.wikimedia.org/T420147#11720835 (10A_smart_kitten) >>! In T420147#11715697, @bd808 wrote: > I'm pretty sure the behavior described in the use case is a broken user-agent or a user-agent... [20:34:36] 10wikitech.wikimedia.org, 10Wikidata: Allow accessing Wikidata data in Wikitech - https://phabricator.wikimedia.org/T171143#11720837 (10ExE-Boss) [20:34:42] 10wikitech.wikimedia.org, 10Wikidata, 10Wikimedia-Interwiki-links, 13Patch-For-Review, 10Wikidata Integration in Wikimedia projects (Kanban Board): Enable interwiki links to/from Wikitech - https://phabricator.wikimedia.org/T290147#11720836 (10ExE-Boss) [20:37:58] RESOLVED: JobsEmailerNoEmails: No emails sent in the last hour - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/JobsEmailerNoEmails - https://prometheus-alerts.wmcloud.org/?q=alertname%3DJobsEmailerNoEmails [20:43:52] 10wikitech.wikimedia.org, 10Wikidata: Allow accessing Wikidata data in Wikitech - https://phabricator.wikimedia.org/T171143#11720855 (10taavi) [20:43:55] 10wikitech.wikimedia.org, 10Wikidata, 10Wikimedia-Interwiki-links, 13Patch-For-Review, 10Wikidata Integration in Wikimedia projects (Kanban Board): Enable interwiki links to/from Wikitech - https://phabricator.wikimedia.org/T290147#11720856 (10taavi) [21:05:02] (03CR) 10Arendpieter: "When you have time, I would greatly appreciate your review and any comments." [labs/striker] - 10https://gerrit.wikimedia.org/r/1250537 (https://phabricator.wikimedia.org/T359554) (owner: 10Arendpieter) [21:05:41] (03CR) 10Arendpieter: "This is the second attempt, where I made several different choices compared to the previously abandoned patch. I chose django-cas-ng inste" [labs/striker] - 10https://gerrit.wikimedia.org/r/1250537 (https://phabricator.wikimedia.org/T359554) (owner: 10Arendpieter) [21:33:59] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23217273642 (https://github.com/cluebotng/component-configs/commits/4fe6cb3d8ad39b60746b3b6bd2f83c4d05a82d6b) [21:34:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [22:17:21] RESOLVED: MaintainKubeusersHang: maintain-kubeusers last finished run is 29.56M minutes old - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersHang [22:18:21] FIRING: MaintainKubeusersHang: maintain-kubeusers last finished run is 29.56M minutes old - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersHang [22:52:36] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#11721215 (10bd808) @dcaro, I see the `--use-latest-versions` version of h... [23:24:24] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11721280 (10Alaaess) Hi I'm Alaa. I'm interested in contributing to CampWiz NxT for GSoC 2026. I have experience with React and frontend development, and I'm excited about this projec... [23:32:36] RESOLVED: MaintainKubeusersHang: maintain-kubeusers last finished run is 29.56M minutes old - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersHang [23:35:21] FIRING: MaintainKubeusersHang: maintain-kubeusers last finished run is 29.56M minutes old - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersHang