[00:03:51] <wikibugs>	 10Tool-campwiz-nxt: CampWiz Nxt Redesign: Root Path - https://phabricator.wikimedia.org/T415408#11700391 (10Henryl746) Hi, just wanted to update those of concern; I've been digging more into the codebase and made a bugfix + implemented the custom session context hook it looks like was planning to be added as a p...
[00:03:55] <wikibugs>	 10Tool-containers: rproxy running on Toolforge fails to successfully proxy requests to musicbrainz.org - https://phabricator.wikimedia.org/T419777#11700392 (10bd808) `lang=irc,name=#musicbrainz [23:37]  <    bd808> I have been troubleshooting a mysterious error with the MusicBrainz API and a reverse proxy. I thi...
[00:23:42] <jinxer-wm>	 RESOLVED: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[00:24:38] <wikibugs>	 10Tool-containers: rproxy running on Toolforge fails to successfully proxy requests to musicbrainz.org - https://phabricator.wikimedia.org/T419777#11700403 (10bd808) `lang=irc,name=musicbrainz [00:06]  <julian45[m]> [bd808 (IRC)](https://matrix.to/#/@irc_bd808:chatbrainz.org): please put this over in #metabrain...
[00:53:42] <jinxer-wm>	 FIRING: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[01:07:03] <jinxer-wm>	 FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudgw2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[01:14:44] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[01:23:42] <jinxer-wm>	 RESOLVED: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[01:53:42] <jinxer-wm>	 FIRING: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[02:23:42] <jinxer-wm>	 RESOLVED: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[02:53:42] <jinxer-wm>	 FIRING: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[03:23:42] <jinxer-wm>	 RESOLVED: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[03:53:42] <jinxer-wm>	 FIRING: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[04:23:42] <jinxer-wm>	 RESOLVED: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[04:53:42] <jinxer-wm>	 FIRING: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[05:07:03] <jinxer-wm>	 FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudgw2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[05:14:59] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[05:23:42] <jinxer-wm>	 RESOLVED: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[05:53:42] <jinxer-wm>	 FIRING: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[06:23:42] <jinxer-wm>	 RESOLVED: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[06:53:42] <jinxer-wm>	 FIRING: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[07:23:42] <jinxer-wm>	 RESOLVED: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[07:53:42] <jinxer-wm>	 FIRING: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[08:23:42] <jinxer-wm>	 RESOLVED: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[08:53:42] <jinxer-wm>	 FIRING: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[09:07:03] <jinxer-wm>	 FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudgw2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[09:14:59] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[09:23:42] <jinxer-wm>	 RESOLVED: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[09:53:42] <jinxer-wm>	 FIRING: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[10:23:42] <jinxer-wm>	 RESOLVED: AlertLintProblem: Linting problems found for NeutronAgentAdminDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[10:44:26] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Carry out controlled network switch down tests in cloud - https://phabricator.wikimedia.org/T417393#11701437 (10fgiunchedi) >>! In T417393#11698352, @Andrew wrote: >>>! In T417393#11696203, @fgiunchedi wrote: >> Plan is to grab another announced maint win...
[10:47:55] <wikibugs>	 06cloud-services-team, 10Toolforge: Add new k8s toolforge workers to cater for memory requests - https://phabricator.wikimedia.org/T419824 (10fgiunchedi) 03NEW
[11:02:14] <wikibugs>	 06cloud-services-team, 10Toolforge: Add new k8s toolforge workers to cater for memory requests - https://phabricator.wikimedia.org/T419824#11701502 (10fgiunchedi) Following the docs at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes#Building_new_nodes this is what I'm planning on running,...
[11:12:01] <wikibugs>	 06cloud-services-team, 10Toolforge: Add new k8s toolforge workers to cater for memory requests - https://phabricator.wikimedia.org/T419824#11701536 (10fgiunchedi) I'm also assuming that the most requests come from nfs workers above, to be verified once a nfs worker is added and how it changes memory requests %
[11:21:14] <wikibugs>	 06cloud-services-team, 10Toolforge: Add new k8s toolforge workers to cater for memory requests - https://phabricator.wikimedia.org/T419824#11701552 (10dcaro) >>! In T419824#11701502, @fgiunchedi wrote: > Following the docs at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes#Building_new_no...
[12:25:25] <wikibugs>	 (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/61
[12:25:26] <wikibugs>	 (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/33
[13:11:00] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services: [wikireplicas] Create views for new wiki kaiwiki - https://phabricator.wikimedia.org/T414240#11701997 (10fnegri) 05Open→03In progress
[13:20:15] <wikibugs>	 (03open) 10filippo: flavors: add g4.cores8.ram32.disk20.ephem140 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/301 (https://phabricator.wikimedia.org/T419824)
[13:20:49] <wikibugs>	 (03update) 10filippo: flavors: add g4.cores8.ram32.disk20.ephem140 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/301 (https://phabricator.wikimedia.org/T419824)
[13:26:48] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services: [wikireplicas] Create views for new wiki kaiwiki - https://phabricator.wikimedia.org/T414240#11702068 (10fnegri) a:05fnegri→03BTullis Views were created on all `clouddb*`, but the cookbook failed halfway through, so I ran manually `maintain-meta_p...
[13:27:50] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Platform-SRE: [wikireplicas] Create views for new wiki kaiwiki - https://phabricator.wikimedia.org/T414240#11702072 (10fnegri)
[13:37:34] <wikibugs>	 (03approved) 10dcaro: flavors: add g4.cores8.ram32.disk20.ephem140 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/301 (https://phabricator.wikimedia.org/T419824) (owner: 10filippo)
[13:41:04] <wikibugs>	 (03update) 10filippo: flavors: add g4.cores8.ram32.disk20.ephem140 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/301 (https://phabricator.wikimedia.org/T419824)
[13:41:23] <wikibugs>	 (03merge) 10filippo: flavors: add g4.cores8.ram32.disk20.ephem140 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/301 (https://phabricator.wikimedia.org/T419824)
[13:43:05] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
[13:44:40] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
[13:45:31] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
[13:46:17] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
[13:46:25] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Carry out controlled network switch down tests in cloud - https://phabricator.wikimedia.org/T417393#11702131 (10Andrew) >>! In T417393#11701437, @fgiunchedi wrote: >  > Thank you, I looked at `cloudvirt.drain` though I couldn't find an option specifically...
[13:50:57] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76
[13:55:00] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76
[14:08:19] <wikibugs>	 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11702308 (10Dipanshu1223) @Nokib_Sarkar  and @Tiven2240  I just need to ask about something about this project  Next.js project need to be converted pure react.js to avoid dedicated N...
[14:53:22] <wikibugs>	 10Tool-containers: rproxy running on Toolforge fails to successfully proxy requests to musicbrainz.org - https://phabricator.wikimedia.org/T419777#11702597 (10bd808) `lang=irc,name=#metabrainz [00:33]  <julian45[m]> [bd808 (IRC)](https://matrix.to/#/@irc_bd808:chatbrainz.org): this would be the place! thanks for...
[14:57:09] <wikibugs>	 06cloud-services-team, 10Toolforge: [logs-api] Using --latest N gives the first N lines since the last hour, not the last - https://phabricator.wikimedia.org/T419863 (10dcaro) 03NEW
[14:57:14] <wikibugs>	 06cloud-services-team, 10Toolforge: [logs-api] Using --latest N gives the first N lines since the last hour, not the last - https://phabricator.wikimedia.org/T419863#11702625 (10dcaro) p:05Triage→03Medium
[15:20:08] <wikibugs>	 10VPS-project-Codesearch: Server timeout waiting for the HTTP request from the client. - https://phabricator.wikimedia.org/T419869 (10Gerges) 03NEW
[15:36:28] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 azwikimedia START - Cookbook wmcs.openstack.quota_increase by 1 floating-ips (T419582)
[15:36:30] <stashbot>	 T419582: Add floating IP and vanity domain for azwikimedia project - https://phabricator.wikimedia.org/T419582
[15:36:35] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 azwikimedia END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 1 floating-ips (T419582)
[15:37:26] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests): Add floating IP and vanity domain for azwikimedia project - https://phabricator.wikimedia.org/T419582#11702942 (10dcaro) Just added the floating ip :)  For the vanity domain, you have to follow the steps here: https://wikitech.wikimedia.org/wiki/Help:Using_a...
[15:39:47] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Carry out controlled network switch down tests in cloud - https://phabricator.wikimedia.org/T417393#11702963 (10fgiunchedi) >>! In T417393#11702131, @Andrew wrote: >>>! In T417393#11701437, @fgiunchedi wrote: >>  >> Thank you, I looked at `cloudvirt.drain...
[15:43:28] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Permanently set 'noout' for cloudceph - https://phabricator.wikimedia.org/T419877 (10fgiunchedi) 03NEW
[15:46:03] <wikibugs>	 10VPS-project-Codesearch: Server timeout waiting for the HTTP request from the client. - https://phabricator.wikimedia.org/T419869#11703006 (10Dzahn) Hi @Gerges it works for me and 172.17.0.2 looks like a Verizon IP, not a Wikimedia IP.
[15:47:26] <wikibugs>	 10VPS-project-Codesearch: Server timeout waiting for the HTTP request from the client. - https://phabricator.wikimedia.org/T419869#11703026 (10taavi) >>! In T419869#11703006, @Dzahn wrote: > Hi @Gerges it works for me and 172.17.0.2 looks like a Verizon IP, not a Wikimedia IP.  This is RFC 1918 space, and in par...
[15:48:56] <wikibugs>	 10VPS-project-Codesearch: Server timeout waiting for the HTTP request from the client. - https://phabricator.wikimedia.org/T419869#11703050 (10Dzahn) Yea, that's correct. It's the Docker default network. I would have to check why that resolves to something in myvzw.com on my local machine.  Nevertheless, search...
[15:50:07] <wikibugs>	 10VPS-project-Codesearch: Codesearch is down/unreachable (2026-01-16) - https://phabricator.wikimedia.org/T414776#11703052 (10A_smart_kitten)
[15:50:45] <wikibugs>	 10VPS-project-Codesearch: Server timeout waiting for the HTTP request from the client. - https://phabricator.wikimedia.org/T419869#11703056 (10A_smart_kitten) FWIW Codesearch also works for me at the moment. https://codesearch.wmcloud.org/_health also shows all as 'up' for me currently. (Maybe a temporary blip?...
[15:52:14] <wikibugs>	 06cloud-services-team, 10decommission-hardware: decommission cloudgw2002-dev - https://phabricator.wikimedia.org/T419738#11703084 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by andrew@cumin2002 for hosts: `cloudgw2002-dev.codfw.wmnet` - cloudgw2002-dev.codfw.wmnet (**PASS**)   - Downtimed...
[15:52:40] <wikibugs>	 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team: Add tests to ensure consistency between OAD example and OpenAPI linter - https://phabricator.wikimedia.org/T419576#11703086 (10HCoplin-WMF) **Notes from Estimation**:  * For now, we will reference a copy of the Wikimedia Pet Store OAD into testing instead of ref...
[15:53:18] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-codfw: decommission cloudgw2002-dev - https://phabricator.wikimedia.org/T419738#11703088 (10Andrew) a:05Andrew→03None
[15:55:19] <wikibugs>	 (03update) 10dcaro: toolforge_deploy: add restore-all [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/314
[16:08:28] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Collect access metrics from cloud-vps web proxy - https://phabricator.wikimedia.org/T371382#11703154 (10Andrew) 05Open→03Declined It's been a long time since we discussed this and no one is working on it. The privacy implications are a bit messy...
[16:09:43] <wikibugs>	 10VPS-project-Codesearch: Server timeout waiting for the HTTP request from the client. - https://phabricator.wikimedia.org/T419869#11703160 (10Gerges) search works for me now, I searched for `WQ`, it took a long time and I didn't get any results, before it crashed.
[16:18:56] <wikibugs>	 06cloud-services-team, 06collaboration-services, 06SRE Observability, 13Patch-For-Review: Move https://status.wikimedia.org/ away from rackspace - https://phabricator.wikimedia.org/T414098#11703202 (10Dzahn) curl to the k8s ingress asking for status.wikimedia.org works :)  ` [deploy1003:~] $ curl --resolve...
[16:28:31] <wikibugs>	 (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/61 (owner: 10l10n-bot)
[16:28:35] <wikibugs>	 (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/61 (owner: 10l10n-bot)
[16:28:57] <wikibugs>	 10VPS-project-Codesearch: Server timeout waiting for the HTTP request from the client. - https://phabricator.wikimedia.org/T419869#11703235 (10A_smart_kitten) 05Open→03Resolved Glad to hear :)
[16:29:45] <wikibugs>	 10VPS-project-Codesearch: Server timeout waiting for the HTTP request from the client (2026-03-12) - https://phabricator.wikimedia.org/T419869#11703241 (10A_smart_kitten)
[16:29:49] <wikibugs>	 (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/33 (owner: 10l10n-bot)
[16:29:53] <wikibugs>	 (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/33 (owner: 10l10n-bot)
[16:45:26] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Engineering, 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11703287 (10fnegri) 05Open→03In progress p:05Triage→03Medium a:...
[16:53:17] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Engineering, 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11703319 (10fnegri) I ran this on all clouddbs:  ` sudo maintain-views...
[16:54:41] <wikibugs>	 06cloud-services-team, 06collaboration-services, 06SRE Observability, 13Patch-For-Review: Move https://status.wikimedia.org/ away from rackspace - https://phabricator.wikimedia.org/T414098#11703325 (10Dzahn) @Andrew The status.wikimedia.org page has moved away from Rackspace :)   ` status.wikimedia.org is...
[16:55:02] <wikibugs>	 10Tool-containers: rproxy running on Toolforge fails to successfully proxy requests to musicbrainz.org - https://phabricator.wikimedia.org/T419777#11703326 (10bd808) 05Open→03Resolved a:03bd808 https://bd808-test.toolforge.org/musicbrainz/ws/2/artist/5b11f4ce-a62d-471e-81fc-a69a8278c7da?fmt=json works...
[16:55:10] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Engineering, 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11703330 (10fnegri) a:05fnegri→03None
[16:55:20] <wikibugs>	 06cloud-services-team, 06collaboration-services, 06SRE Observability, 13Patch-For-Review: Move https://status.wikimedia.org/ away from rackspace - https://phabricator.wikimedia.org/T414098#11703331 (10Dzahn) 05In progress→03Resolved
[16:56:10] <wikibugs>	 10Tool-containers: Create a reusable container to replace nginx ingress anonymizing reverse proxy setups - https://phabricator.wikimedia.org/T414836#11703337 (10bd808)
[16:56:15] <wikibugs>	 10Tool-containers: rproxy running on Toolforge fails to successfully proxy requests to musicbrainz.org - https://phabricator.wikimedia.org/T419777#11703338 (10bd808)
[16:57:23] <wikibugs>	 10Tool-containers: Create a reusable container to replace nginx ingress anonymizing reverse proxy setups - https://phabricator.wikimedia.org/T414836#11703344 (10bd808) The musicbrainz mystery has been solved (Toolforge/Cloud VPS egress IP was blocked by them and is now unblocked). I will review the changes I was...
[17:32:53] <wikibugs>	 06cloud-services-team, 06collaboration-services, 06SRE Observability, 13Patch-For-Review: Move https://status.wikimedia.org/ away from rackspace - https://phabricator.wikimedia.org/T414098#11703467 (10colewhite) 05Resolved→03Open a:05Dzahn→03colewhite Last thing the old rackspace host handles is `[...
[17:36:04] <wikibugs>	 06cloud-services-team, 06SRE Observability, 06Traffic: Move wikimediastatus.net 301 to ncredir - https://phabricator.wikimedia.org/T419887 (10colewhite) 03NEW
[17:36:30] <wikibugs>	 06cloud-services-team, 06SRE Observability, 06Traffic: Move wikimediastatus.net 301 to ncredir - https://phabricator.wikimedia.org/T419887#11703499 (10colewhite)
[17:37:20] <wikibugs>	 06cloud-services-team, 06collaboration-services, 06SRE Observability, 13Patch-For-Review: Move https://status.wikimedia.org/ away from rackspace - https://phabricator.wikimedia.org/T414098#11703506 (10colewhite) 05Open→03Resolved a:05colewhite→03Dzahn >>! In T414098#11703467, @colewhite wrote:...
[17:41:35] <wikibugs>	 06cloud-services-team, 06SRE Observability, 06Traffic, 13Patch-For-Review: Move wikimediastatus.net 301 to ncredir - https://phabricator.wikimedia.org/T419887#11703537 (10Aklapper)
[17:41:36] <wikibugs>	 06cloud-services-team, 06collaboration-services, 06SRE Observability, 13Patch-For-Review: Move https://status.wikimedia.org/ away from rackspace - https://phabricator.wikimedia.org/T414098#11703536 (10Aklapper)
[17:41:54] <wikibugs>	 06cloud-services-team, 06SRE Observability, 06Traffic, 13Patch-For-Review: Move wikimediastatus.net 301 to ncredir - https://phabricator.wikimedia.org/T419887#11703554 (10Aklapper)
[17:41:57] <wikibugs>	 06cloud-services-team, 06collaboration-services, 06SRE Observability, 13Patch-For-Review: Move https://status.wikimedia.org/ away from rackspace - https://phabricator.wikimedia.org/T414098#11703553 (10Aklapper)
[18:29:23] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests): Add floating IP and vanity domain for azwikimedia project - https://phabricator.wikimedia.org/T419582#11703725 (10Andrew) This was discussed and approved during today's weekly meeting.
[19:22:07] <wikibugs>	 (03approved) 10pepepiton: Add missing information for author and work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/183 (owner: 10nurahwakili)
[19:22:27] <wikibugs>	 (03merge) 10pepepiton: Add missing information for author and work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/183 (owner: 10nurahwakili)
[21:02:17] <wikibugs>	 10Tool-campwiz-nxt: CampWiz Nxt Redesign: Root Path - https://phabricator.wikimedia.org/T415408#11704397 (10Dipanshu1223) @Nokib_Sarkar  and @Tiven2240  I just need to ask about something about this project  Next.js project need to be converted pure react.js to avoid dedicated Node.js server so, this website hav...
[22:07:12] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Collect access metrics from cloud-vps web proxy - https://phabricator.wikimedia.org/T371382#11704608 (10bd808) >>! In T371382#11703154, @Andrew wrote: > The privacy implications are a bit messy so I'm just closing.  Porting the per backend request count tracking used wit...
[22:24:42] <wikibugs>	 (03open) 10bd808: Draft: Big pile of debug and tuning changes [toolforge-repos/containers-rproxy] - 10https://gitlab.wikimedia.org/toolforge-repos/containers-rproxy/-/merge_requests/5
[22:38:34] <wikibugs>	 (03update) 10bd808: Draft: Big pile of debug and tuning changes [toolforge-repos/containers-rproxy] - 10https://gitlab.wikimedia.org/toolforge-repos/containers-rproxy/-/merge_requests/5
[22:39:38] <wikibugs>	 (03update) 10bd808: Big pile of debug and tuning changes [toolforge-repos/containers-rproxy] - 10https://gitlab.wikimedia.org/toolforge-repos/containers-rproxy/-/merge_requests/5
[22:40:34] <wikibugs>	 (03merge) 10bd808: Big pile of debug and tuning changes [toolforge-repos/containers-rproxy] - 10https://gitlab.wikimedia.org/toolforge-repos/containers-rproxy/-/merge_requests/5
[23:07:16] <wikibugs>	 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857#11704747 (10MusikAnimal) In recent tests, it seems to start giving 429s after exactly 1000 requests. That's with each req...
[23:09:51] <wikibugs>	 10Tool-delintbot: Make lint category an optional argument - https://phabricator.wikimedia.org/T416792#11704751 (10Redmin)
[23:10:29] <wikibugs>	 10Tool-delintbot: Make lint category an optional argument - https://phabricator.wikimedia.org/T416792#11704752 (10Redmin) p:05Medium→03Low
[23:12:53] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: decommission cloudgw2002-dev - https://phabricator.wikimedia.org/T419738#11704769 (10Papaul) @Jhancock.wm try to delete the mgmt ip address and run the script again  https://netbox.wikimedia.org/dcim/devices/3026/interfaces/
[23:16:21] <wikibugs>	 10Tool-containers: Create a reusable container to replace nginx ingress anonymizing reverse proxy setups - https://phabricator.wikimedia.org/T414836#11704772 (10bd808) 05In progress→03Resolved
[23:25:24] <wikibugs>	 06cloud-services-team, 10Toolforge: Remove remaining uses of ingress-nginx specific annotations - https://phabricator.wikimedia.org/T414674#11704780 (10bd808) T414836#11609516 should be a pretty good example of how to migrate existing Ingress-only reverse proxies to the new [[https://wikitech.wikimedia.org/wik...
[23:26:43] <wikibugs>	 10Tool-delintbot: Make edit requests when protected pages need to be edited - https://phabricator.wikimedia.org/T416790#11704782 (10Redmin)
[23:44:44] <wikibugs>	 10Tool-delintbot: Handle cases of end tags being missing in headings - https://phabricator.wikimedia.org/T416082#11704837 (10Redmin)
[23:48:24] <wikibugs>	 06cloud-services-team, 10Toolforge: Remove remaining uses of ingress-nginx specific annotations - https://phabricator.wikimedia.org/T414674#11704866 (10taavi) >>! In T414674#11704780, @bd808 wrote: > # Profit!! Unfortunately this is missing the step of adding a Gateway API HTTPRoute object so that this doesn't...