[00:02:29] 06cloud-services-team, 10Toolforge, 10Tools: Geohack tool frequently triggers the Toolforge front proxy's per-tool rate limit due to too much traffic - https://phabricator.wikimedia.org/T409185#11352090 (10bd808) [03:05:08] 10Tool-Pageviews: Time period cannot be selected on pageviews.wmcloud.org on mobile - https://phabricator.wikimedia.org/T367626#11352305 (10SD0001) [04:11:14] 10Tool-wmf-openapi-linter, 03[MWI] FY2025-26 Q2, 06MW-Interfaces-Team (MWI-Sprint-21 (2025-10-21 to 2025-11-04)), 07OKR-Work: Create a tool to lint OpenAPI spec based on Wikimedia ruleset - https://phabricator.wikimedia.org/T406879#11352362 (10HCoplin-WMF) 05Open→03Resolved [06:02:36] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1202693 (owner: 10L10n-bot) [06:23:25] 06cloud-services-team, 10Data-Services, 06DBA: Set up replication on new hosts clouddb102[2-5] - https://phabricator.wikimedia.org/T408692#11352425 (10Marostegui) p:05Triage→03Medium a:03Marostegui >>! In T408692#11351084, @fnegri wrote: > @Marostegui no rush at all, but when you have some time you can... [06:44:40] (03CR) 10Lokal Profil: [C:03+2] Extract Toolforge Docker images to use as default variables [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202813 (owner: 10Jean-Frédéric) [06:54:45] (03CR) 10Lokal Profil: [C:03+2] Extract method to create a Toolforge job over the API [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202811 (owner: 10Jean-Frédéric) [07:03:15] (03CR) 10Lokal Profil: [C:03+1] "would it be pedagogical to give the payloads different variable names (in fhe fill... script)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202810 (owner: 10Jean-Frédéric) [07:12:05] (03CR) 10Lokal Profil: [C:03+2] Fix default variables sourcing in deploy-to-toolforge Shell script [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202812 (owner: 10Jean-Frédéric) [07:28:52] 06cloud-services-team, 10Tool-iw, 10Toolforge: Toolforge interwikli link handling no longer strips URL-encoding before redirecting when it previously did, breaking existing on-wiki links - https://phabricator.wikimedia.org/T409493#11352501 (10Tamzin) If this has to be a breaking change, I think that's okay,... [07:29:02] 06cloud-services-team, 10Tool-iw, 10Toolforge: Toolforge interwiki link handling no longer strips URL-encoding before redirecting when it previously did, breaking existing on-wiki links - https://phabricator.wikimedia.org/T409493#11352503 (10Tamzin) [07:31:20] (03PS2) 10Aklapper: Set all IO to none and hidden for internal URIs of Diffusion repos [labs/striker] - 10https://gerrit.wikimedia.org/r/1202259 (https://phabricator.wikimedia.org/T409341) [10:12:32] 10Tool-paulina: Implement consistent loading states for asynchronous actions across Paulina - https://phabricator.wikimedia.org/T409535 (10Oluwatumininu.m) 03NEW [10:23:13] 10Tool-schedule-deployment: Schedule button should disabled after first click - https://phabricator.wikimedia.org/T409536 (10Esanders) 03NEW [10:34:24] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Unplanned, 13Patch-For-Review: [promethus,haproxy] Move to haproxy internal metrics from haproxy_exporter - https://phabricator.wikimedia.org/T343885#11352916 (10taavi) a:05dcaro→03taavi [10:39:36] (03PS1) 10Tiziano Fogli: metamonitoring/icinga/ext-mon: add dummy basic auth info [labs/private] - 10https://gerrit.wikimedia.org/r/1202990 (https://phabricator.wikimedia.org/T397003) [10:39:42] (03PS1) 10Majavah: Add dummy toolviews hash salt [labs/private] - 10https://gerrit.wikimedia.org/r/1202991 [10:40:01] (03CR) 10Tiziano Fogli: [C:03+2] metamonitoring/icinga/ext-mon: add dummy basic auth info [labs/private] - 10https://gerrit.wikimedia.org/r/1202990 (https://phabricator.wikimedia.org/T397003) (owner: 10Tiziano Fogli) [10:40:06] (03CR) 10Tiziano Fogli: [V:03+2 C:03+2] metamonitoring/icinga/ext-mon: add dummy basic auth info [labs/private] - 10https://gerrit.wikimedia.org/r/1202990 (https://phabricator.wikimedia.org/T397003) (owner: 10Tiziano Fogli) [10:40:40] (03CR) 10Majavah: [V:03+2 C:03+2] Add dummy toolviews hash salt [labs/private] - 10https://gerrit.wikimedia.org/r/1202991 (owner: 10Majavah) [10:47:09] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11353005 (10fnegri) Rsync completed in 5h8m, but of course there's a snatch: I thought I could upgrade to trixie, but mariadb 1... [10:47:59] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Fix MTU on single-NIC Ceph nodes - https://phabricator.wikimedia.org/T409294#11353008 (10taavi) 05Open→03Invalid Huh, TIL, thanks! And sorry for me panicing over nothing in that case. [10:56:50] (03PS1) 10Vgutierrez: secrets: Mock of trusted_proxies.map [labs/private] - 10https://gerrit.wikimedia.org/r/1202994 [11:05:27] 06cloud-services-team, 10Toolforge, 10Tools: Geohack tool frequently triggers the Toolforge front proxy's per-tool rate limit due to too much traffic - https://phabricator.wikimedia.org/T409185#11353073 (10taavi) Some of those 6 replicas were hitting the half a core CPU limit. I've bumped the CPU limit to a... [11:05:58] RESOLVED: TargetDown: Job toolsdb-mariadb is unreachable in project tools instance tools-db-7 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [11:06:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance tools-puppetserver-01 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [11:07:09] (03CR) 10Vgutierrez: [V:03+2 C:03+2] secrets: Mock of trusted_proxies.map [labs/private] - 10https://gerrit.wikimedia.org/r/1202994 (owner: 10Vgutierrez) [11:12:47] PuppetSyncFailure is me [11:16:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance tools-puppetserver-01 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [11:33:02] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.vps.remove_instance for instance tools-db-7 [11:33:05] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=1) for instance tools-db-7 [11:33:27] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.vps.remove_instance for instance tools-db-7 (T409287) [11:33:30] T409287: [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287 [11:34:14] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-7 (T409287) [11:35:49] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' (T409287) [11:42:41] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' (T409287) [11:42:46] T409287: [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287 [11:44:28] FIRING: TargetDown: Job toolsdb-mariadb is unreachable in project tools instance tools-db-7 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [11:45:27] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11353147 (10fnegri) Created new snapshot with the position below: ` MariaDB [(none)]> SELECT @@gtid_current_pos; +------------... [11:45:56] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-7.tools.eqiad1.wikimedia.cloud (T409287) [11:47:29] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-7.tools.eqiad1.wikimedia.cloud (T409287) [11:53:14] (03PS2) 10Jean-Frédéric: Extract JSON payloads to Toolforge API as their own variables [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202810 [11:53:14] (03PS2) 10Jean-Frédéric: Extract method to create a Toolforge job over the API [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202811 [11:53:14] (03PS2) 10Jean-Frédéric: Fix default variables sourcing in deploy-to-toolforge Shell script [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202812 [11:53:15] (03PS2) 10Jean-Frédéric: Extract Toolforge Docker images to use as default variables [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202813 [11:57:49] (03CR) 10Lokal Profil: [C:03+2] Extract JSON payloads to Toolforge API as their own variables [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202810 (owner: 10Jean-Frédéric) [11:59:11] (03CR) 10Lokal Profil: [C:03+2] Extract method to create a Toolforge job over the API [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202811 (owner: 10Jean-Frédéric) [11:59:28] RESOLVED: TargetDown: Job toolsdb-mariadb is unreachable in project tools instance tools-db-7 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [11:59:45] (03Merged) 10jenkins-bot: Extract JSON payloads to Toolforge API as their own variables [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202810 (owner: 10Jean-Frédéric) [11:59:58] 10Toolforge, 06tools-infrastructure-team: Reduce tool breakage over new ingress-nginx annotation validation rules - https://phabricator.wikimedia.org/T409474#11353177 (10taavi) I checked and the proxy use case seems to still work fine in Toolsbeta. So I think this is the real list of tools this would affect, g... [12:00:52] (03Merged) 10jenkins-bot: Extract method to create a Toolforge job over the API [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202811 (owner: 10Jean-Frédéric) [12:00:57] (03Merged) 10jenkins-bot: Fix default variables sourcing in deploy-to-toolforge Shell script [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202812 (owner: 10Jean-Frédéric) [12:01:05] (03Merged) 10jenkins-bot: Extract Toolforge Docker images to use as default variables [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202813 (owner: 10Jean-Frédéric) [12:06:48] 06cloud-services-team, 10Cloud-VPS: Investigate and enable jumbo frames in cloudvirt nodes - https://phabricator.wikimedia.org/T273596#11353192 (10taavi) Merging to newer T330075 as that has more discussion about it. [12:06:51] 06cloud-services-team, 10Cloud-VPS: Investigate and enable jumbo frames in cloudvirt nodes - https://phabricator.wikimedia.org/T273596#11353195 (10taavi) →14Duplicate dup:03T330075 [12:06:53] 06cloud-services-team, 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Unplanned: [cloudvirt] Move to jumbo frames - https://phabricator.wikimedia.org/T330075#11353197 (10taavi) [12:06:59] 06cloud-services-team, 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Unplanned: [cloudvirt] Move to jumbo frames - https://phabricator.wikimedia.org/T330075#11353199 (10taavi) a:03taavi [12:07:46] 06cloud-services-team, 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Unplanned: [cloudvirt] Enable jumbo frames on cloud-hosts/cloud-private interfaces - https://phabricator.wikimedia.org/T330075#11353200 (10taavi) [12:08:46] 06cloud-services-team, 10Cloud-VPS: Tell Neutron the physical network has jumbo frames enabled - https://phabricator.wikimedia.org/T409544 (10taavi) 03NEW p:05Triage→03High [12:08:52] 06cloud-services-team, 10Cloud-VPS: Tell Neutron the physical network has jumbo frames enabled - https://phabricator.wikimedia.org/T409544#11353216 (10taavi) p:05High→03Medium [12:09:06] 06cloud-services-team, 10Cloud-VPS: Tell Neutron the physical network has jumbo frames enabled - https://phabricator.wikimedia.org/T409544#11353217 (10taavi) [12:09:08] 06cloud-services-team, 10Cloud-VPS: MTU setting in IPv6 VMs causes issues with Docker - https://phabricator.wikimedia.org/T408543#11353218 (10taavi) [12:09:15] 06cloud-services-team, 10Cloud-VPS: Tell Neutron the physical network has jumbo frames enabled - https://phabricator.wikimedia.org/T409544#11353219 (10taavi) [12:09:17] 06cloud-services-team, 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Unplanned: [cloudvirt] Enable jumbo frames on cloud-hosts/cloud-private interfaces - https://phabricator.wikimedia.org/T330075#11353220 (10taavi) [12:28:54] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11353253 (10fnegri) >>! In T409287#11353005, @fnegri wrote: > But maybe there are some good news: it looks like recent versions... [13:31:01] 10Cloud Services Proposals, 06cloud-services-team (FY2025/26-Q1), 10Data-Services, 06Data-Persistence, and 2 others: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#11353410 (10Gehel) [13:31:15] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, and 3 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#11353414 (10Gehel) [13:35:11] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#11353512 (10Gehel) [13:36:38] 06cloud-services-team, 10Data-Services, 10BetaFeatures, 06Data-Engineering, and 2 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#11353553 (10Gehel) [13:39:57] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407485#11353612 (10Gehel) [13:40:05] 10Toolforge (Toolforge iteration 25): [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images - https://phabricator.wikimedia.org/T409191#11353617 (10Raymond_Ndibe) >>! In T409191#11349652, @dcaro wrote: > Also, what does you mean with `doesn't exist in the toollabs-images rep... [13:54:40] (03open) 10taavi: kubernetes: Update HAProxy metrics for the new exporter [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/46 (https://phabricator.wikimedia.org/T343885) [13:54:42] (03update) 10taavi: kubernetes: Update HAProxy metrics for the new exporter [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/46 (https://phabricator.wikimedia.org/T343885) [14:13:20] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#11353776 (10Andrew) For my reference, the following will be the redundant pairs according to T401295 clouddb1013 & clouddb1017 clouddb1014 & clouddb1018 cloudd... [14:23:52] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#11353826 (10Andrew) [14:28:12] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#11353839 (10Andrew) a:05Andrew→03None [14:34:56] (03open) 10fnegri: Stop managing tools-db data volumes with tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/101 [14:35:00] (03update) 10fnegri: Stop managing tools-db data volumes with tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/101 [14:47:08] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#11353915 (10Marostegui) @Andrew I believe you also have to do the puppet patches. [14:48:30] (03update) 10fnegri: Stop managing tools-db-7-data with tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/101 [14:48:31] (03open) 10fnegri: Stop managing all tools-db volumes with tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/102 [14:48:31] (03update) 10fnegri: Stop managing all tools-db volumes with tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/102 [14:48:38] (03update) 10fnegri: Stop managing tools-db-7-data with tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/101 [14:48:38] (03update) 10fnegri: Stop managing all tools-db volumes with tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/102 [14:49:23] 06cloud-services-team, 10Data-Services, 06DBA: Set up replication on new hosts clouddb102[2-5] - https://phabricator.wikimedia.org/T408692#11353918 (10Marostegui) Ha, I just realised the hosts aren't installed yet [14:57:29] (03approved) 10taavi: Stop managing tools-db-7-data with tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/101 (owner: 10fnegri) [14:57:41] (03merge) 10fnegri: Stop managing tools-db-7-data with tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/101 [14:57:43] (03update) 10fnegri: Stop managing all tools-db volumes with tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/102 [14:58:50] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#11353954 (10Marostegui) [14:59:01] 06cloud-services-team, 10Data-Services, 06DBA: Set up replication on new hosts clouddb102[2-5] - https://phabricator.wikimedia.org/T408692#11353956 (10Marostegui) [15:02:05] 10Cloud-VPS (Quota-requests), 13Patch-For-Review: Grant zuul project access to `fast-iops` volume type and `4xiops` instance flavor - https://phabricator.wikimedia.org/T409365#11353966 (10fnegri) @bd808 do you also need high-iops Cinder volumes, or the root volumes are enough? [15:02:27] 10Cloud-VPS (Quota-requests), 13Patch-For-Review: Grant zuul project access to `fast-iops` volume type and `4xiops` instance flavor - https://phabricator.wikimedia.org/T409365#11353967 (10fnegri) a:03fnegri [15:03:57] (03open) 10oluwatumininu: feat(loading): add consistent loading states for major async actions [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/168 (https://phabricator.wikimedia.org/T409535) [15:15:46] 06cloud-services-team, 10Data-Services, 06DBA: Set up replication on new hosts clouddb102[2-5] - https://phabricator.wikimedia.org/T408692#11354004 (10Marostegui) After talking to @fnegri the confusion was cleared up, these hosts are, the ones that aren't are the refreshes. I created a task with all the host... [15:20:33] 06cloud-services-team, 10Data-Services: Add support for x1 and x4 sections on wiki replicas on the load balancer layer - https://phabricator.wikimedia.org/T409560 (10taavi) 03NEW [15:20:39] 06cloud-services-team, 10Data-Services: Add support for x1 and x4 sections on wiki replicas on the load balancer layer - https://phabricator.wikimedia.org/T409560#11354034 (10taavi) [15:20:42] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#11354035 (10taavi) [15:32:49] 06cloud-services-team (FY2025/26-Q1), 10Toolforge: [toolsdb] Add users and grants for IPv6, remove obsolete ones - https://phabricator.wikimedia.org/T409563 (10fnegri) 03NEW [15:41:27] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11354133 (10fnegri) Next steps (I will do these next week): * {T409563} * Move DNS for `tools-readonly.db.svc.wikimedia.cloud`... [16:14:02] 10Cloud-VPS (Quota-requests), 13Patch-For-Review: Grant zuul project access to `fast-iops` volume type and `4xiops` instance flavor - https://phabricator.wikimedia.org/T409365#11354267 (10bd808) >>! In T409365#11353966, @fnegri wrote: > @bd808 do you also need high-iops Cinder volumes, or the root volumes are... [17:11:16] 06cloud-services-team, 06SRE: latest Trixie image (as of 2025-10-16) grub failure on R450 hardware - https://phabricator.wikimedia.org/T407586#11354501 (10Andrew) I've just noticed that there are quite a few 2-drive r450s that reimaged without trouble, for example cloudrabbit200[234]-dev. [17:23:05] FIRING: [2x] HostBGPDown: BGP session for cloudlb2002-dev (172.20.5.3) is down - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DHostBGPDown [17:24:14] 10Tool-paulina: Add field "inception" to work page - https://phabricator.wikimedia.org/T406949#11354545 (10Sopzy) >>! In T406949#11335865, @Pepe_piton wrote: > @Afanyulionel @Sopzy Excellent work. You catched the logic of the task. > > @Dipanshu1223 You also did a good job, but it seems to me that when you adde... [17:35:36] 10Tool-paulina: Proof of concept for editing statements in Paulina - https://phabricator.wikimedia.org/T405412#11354569 (10Sopzy) >>! In T405412#11336032, @Pepe_piton wrote: > @Sopzy You did a great job implementing everything we asked for in a very straightforward way. Perhaps the only thing missing is the abil... [18:36:35] 06cloud-services-team, 10Tool-iw, 10Toolforge: Toolforge interwiki link handling no longer strips URL-encoding before redirecting when it previously did, breaking existing on-wiki links - https://phabricator.wikimedia.org/T409493#11354664 (10bd808) >>! In T409493#11352501, @Tamzin wrote: > I'm wondering if t... [18:39:22] 06cloud-services-team, 10Cloud-VPS: Upgrade cloud-vps hosts to Debian Trixie - https://phabricator.wikimedia.org/T409579 (10Andrew) 03NEW [18:40:42] 06cloud-services-team, 10Cloud-VPS: Prepare cloud-vps haproxy configs to work on debian trixie - https://phabricator.wikimedia.org/T409580 (10Andrew) 03NEW [18:44:02] 06cloud-services-team, 10Cloud-VPS: Prepare cloud-vps haproxy configs to work on debian trixie - https://phabricator.wikimedia.org/T409580#11354714 (10Andrew) An example of an offending line is ` option httpchk GET / HTTP/1.0\r\nUser-agent:\ haproxy-healthcheck ` "hiding headers or body at the end of th... [18:45:15] 10Toolforge, 06tools-infrastructure-team: Reduce tool breakage over new ingress-nginx annotation validation rules - https://phabricator.wikimedia.org/T409474#11354718 (10bd808) I would say this result points to a need for a "simple" way to replace the https://wikitech.wikimedia.org/wiki/User:BryanDavis/Kuberne... [18:47:29] 06cloud-services-team, 10Cloud-VPS: Prepare cloud-vps haproxy configs to work on debian trixie - https://phabricator.wikimedia.org/T409580#11354724 (10Andrew) https://www.claudiokuenzler.com/blog/1498/haproxy-option-httpchk-headers-body-end-version-string-unsupported shows a before: ` option httpchk GET /api... [19:34:48] FIRING: PuppetFailure: Puppet has failed on cloudlb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [19:58:05] RESOLVED: [2x] HostBGPDown: BGP session for cloudlb2002-dev (172.20.5.3) is down - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DHostBGPDown [19:59:48] RESOLVED: PuppetFailure: Puppet has failed on cloudlb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [20:24:59] 06cloud-services-team, 10Tool-iw, 10Toolforge: Toolforge interwiki link handling no longer strips URL-encoding before redirecting when it previously did, breaking existing on-wiki links - https://phabricator.wikimedia.org/T409493#11354949 (10Ahecht) I cleaned up all the templates on enwiki that were broken b... [20:51:46] 10Tool-schedule-deployment: Disable "Schedule" button after first click - https://phabricator.wikimedia.org/T409536#11355011 (10bd808) [20:58:40] 10Tool-schedule-deployment: Disable "Schedule" button after first click - https://phabricator.wikimedia.org/T409536#11355033 (10bd808) @Esanders Since you said button I assumed that you mean the form submit buttons at https://schedule-deployment.toolforge.org/ and in the detail screen for a given window, but I r... [21:53:04] 10Tool-schedule-deployment: Disable "Schedule" button after first click - https://phabricator.wikimedia.org/T409536#11355170 (10Esanders) Yes - that's what I mean. I think onclick->disable is a reasonable pattern here. If you prefer you could do it in the backend using a form token to prevent multiple submission... [22:25:16] 10Tool-schedule-deployment: Disable "Schedule" button after first click - https://phabricator.wikimedia.org/T409536#11355235 (10bd808) p:05Triage→03Medium [22:53:02] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#11355319 (10Andrew) a:03Andrew You're right!