[07:04:23] <wikibugs>	 10Cloud-Services, 06Infrastructure-Foundations, 10netops: codfw: Upgrade cloudsw1-b1-codfw (2026) - https://phabricator.wikimedia.org/T416443 (10ayounsi) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/...
[07:09:36] <wikibugs>	 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: codfw: Upgrade cloudsw1-b1-codfw (2026) - https://phabricator.wikimedia.org/T416443#11582002 (10ayounsi)
[07:09:46] <wikibugs>	 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: codfw: Upgrade cloudsw1-b1-codfw (2026) - https://phabricator.wikimedia.org/T416443#11582003 (10ayounsi)
[07:22:36] <wikibugs>	 10Tool-curator: Add an option to deactivate categorization of photographs by date in Curator - https://phabricator.wikimedia.org/T414560#11582016 (10DaxServer) 05In progress→03Resolved a:03DaxServer
[08:11:14] <wikibugs>	 (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1235786 (owner: 10L10n-bot)
[08:46:12] <wikibugs>	 (03open) 10r4356th: Turn sensitive configuration variables into environment variables [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/8 (https://phabricator.wikimedia.org/T416071)
[08:49:16] <wikibugs>	 (03update) 10r4356th: Turn sensitive configuration variables into environment variables [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/8 (https://phabricator.wikimedia.org/T416071)
[08:50:41] <wikibugs>	 (03merge) 10r4356th: Turn sensitive configuration variables into environment variables [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/8 (https://phabricator.wikimedia.org/T416071)
[08:51:26] <wikibugs>	 10Tool-redminbot, 13Patch-For-Review: Turn sensitive configuration variables into environment variables - https://phabricator.wikimedia.org/T416071#11582182 (10Redmin) 05Open→03Resolved
[10:39:40] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Upgrade cloudcumin hosts to bookworm - https://phabricator.wikimedia.org/T403153#11582516 (10MoritzMuehlenhoff) a:03MoritzMuehlenhoff
[10:59:47] <wikibugs>	 10Tool-bub2: Bulk Upload Functionality - https://phabricator.wikimedia.org/T344118#11582588 (10Aklapper) a:05Akanksha.t05→03None Removing task assignee as this open task has been assigned for more than two years - See the email sent to task assignee on 2025-11-25. Please assign this task to yourself again if...
[11:29:29] <wmcs-alerts>	 FIRING: [6x] PuppetCertificateAboutToExpire: Puppet CA certificate pki-db.pki.eqiad1.wikimedia.cloud is about to expire in 25d 23h 32m 33s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire
[11:30:24] <logmsgbot_cloud>	 !log volans@cloudcumin2001 bastioninfra-codfw1dev START - Cookbook wmcs.vps.add_user_to_project for user 'volans' in role 'member'
[11:30:26] <stashbot>	 volans@cloudcumin2001: Unknown project "bastioninfra-codfw1dev"
[11:30:27] <wikibugs>	 10Tools: 'hoiscript' tool uses an unreasonable amount of disk space - https://phabricator.wikimedia.org/T349913#11582726 (10Aklapper) 05Open→03Resolved Assuming this is resolved. If not, please reopen.
[11:30:27] <logmsgbot_cloud>	 !log volans@cloudcumin2001 bastioninfra-codfw1dev END (FAIL) - Cookbook wmcs.vps.add_user_to_project (exit_code=99) for user 'volans' in role 'member'
[11:30:27] <stashbot>	 volans@cloudcumin2001: Unknown project "bastioninfra-codfw1dev"
[11:43:40] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: codfw: Upgrade cloudsw1-b1-codfw (2026) - https://phabricator.wikimedia.org/T416443#11582777 (10taavi)
[11:47:05] <wikibugs>	 10Tool-redminbot: Explore visual regression testing - https://phabricator.wikimedia.org/T415882#11582796 (10Redmin)
[11:53:45] <wikibugs>	 10Tool-redminbot: Handle div-span-flip errors - https://phabricator.wikimedia.org/T416463 (10Redmin) 03NEW
[11:54:07] <wikibugs>	 10Tool-redminbot: Handle div-span-flip errors - https://phabricator.wikimedia.org/T416463#11582818 (10Redmin) p:05Triage→03High
[12:13:35] <wikibugs>	 (03Abandoned) 10Alexandros Kosiaris: Remove obsolete dummy certs for api and parsoid [labs/private] - 10https://gerrit.wikimedia.org/r/1059029 (https://phabricator.wikimedia.org/T360636) (owner: 10Clément Goubert)
[13:42:11] <wikibugs>	 06Toolforge-standards-committee: Adoption request for "request" tool - https://phabricator.wikimedia.org/T389540#11583272 (10Tkarcher) @LucasWerkmeister : I still hesitate to complete the adoption, as most parts of the bots still run smoothly as of today and would immediately stop once you delete the confidentia...
[14:07:03] <wikibugs>	 (03PS1) 10Majavah: Foramt with latest black [labs/striker] - 10https://gerrit.wikimedia.org/r/1236751
[14:08:02] <wikibugs>	 (03PS2) 10Majavah: Format with latest black [labs/striker] - 10https://gerrit.wikimedia.org/r/1236751
[14:09:34] <wikibugs>	 (03CR) 10Majavah: [C:03+2] Format with latest black [labs/striker] - 10https://gerrit.wikimedia.org/r/1236751 (owner: 10Majavah)
[14:13:43] <wikibugs>	 (03Merged) 10jenkins-bot: Format with latest black [labs/striker] - 10https://gerrit.wikimedia.org/r/1236751 (owner: 10Majavah)
[14:36:58] <wikibugs>	 06cloud-services-team, 10Toolforge: `elph` tool stuck in disable process since 2025-04-06 - https://phabricator.wikimedia.org/T415683#11583512 (10taavi) ` Jan 28 10:58:01 cloudcontrol1007 disable_tool.py[2537979]: INFO __main__: Tool elph is expired but home dir not yet archived. Postponing ldap deletion. ` `...
[14:43:19] <wikibugs>	 06cloud-services-team, 10Toolforge: `elph` tool stuck in disable process since 2025-04-06 - https://phabricator.wikimedia.org/T415683#11583544 (10taavi) 05Open→03Resolved a:03taavi
[14:46:27] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Adding a new user to codfw1dev is messed up - https://phabricator.wikimedia.org/T416483 (10Andrew) 03NEW
[14:46:48] <wikibugs>	 (03open) 10matthieulec: Update list of channels following our changes in Phabricator workboard [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/65
[14:48:19] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Adding a new user to codfw1dev is messed up - https://phabricator.wikimedia.org/T416483#11583616 (10Andrew) The 'Access -> Project Access' horizon dashboard in codfw1dev is also broken, likely the same issue as #1
[14:48:41] <wikibugs>	 (03update) 10matthieulec: Update list of channels following our changes in Phabricator workboard [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/65 (https://phabricator.wikimedia.org/T415352)
[14:50:45] <wikibugs>	 10Tools: [FNBot] Store user review statistics in gender neutral "Sichtungstabelle" - https://phabricator.wikimedia.org/T416487 (10Tkarcher) 03NEW
[14:51:32] <wikibugs>	 10Tools: [FNBot] Store user review statistics in gender neutral "Sichtungstabelle" - https://phabricator.wikimedia.org/T416487#11583651 (10Tkarcher) a:03Tkarcher
[14:51:33] <wikibugs>	 10Wikibugs, 06ServiceOps new: Add serviceops new to the monitored channels on IRC bot/wikibugs - https://phabricator.wikimedia.org/T415352#11583652 (10MLechvien-WMF) During last week summit we decided to take that opportunity to route automated updates to a new channel `#wikimedia-serviceops-feed`.   The inten...
[14:51:55] <wikibugs>	 10VPS-project-Codesearch: Codesearch down/unreachable (2026-02-04) - https://phabricator.wikimedia.org/T416488 (10SomeRandomDeveloper) 03NEW
[14:52:24] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki urwikisource - https://phabricator.wikimedia.org/T415977#11583669 (10taavi) 05Open→03Stalled p:05Triage→03Medium Stalled until the wiki is created.
[14:53:15] <wikibugs>	 10Wikibugs, 06ServiceOps new: Add serviceops new to the monitored channels on IRC bot/wikibugs - https://phabricator.wikimedia.org/T415352#11583679 (10MLechvien-WMF)
[14:53:17] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Documentation: Update and improve: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Java - https://phabricator.wikimedia.org/T245314#11583678 (10taavi) p:05Triage→03Medium
[14:54:17] <wikibugs>	 06cloud-services-team, 10Striker: Allow maintainers of tools that are maintainers of another tool to modify the inherited tool's maintainers list - https://phabricator.wikimedia.org/T415963#11583681 (10taavi) p:05Triage→03Low
[14:54:32] <wikibugs>	 (03open) 10r1f4t: Update [toolforge-repos/ztools] - 10https://gitlab.wikimedia.org/toolforge-repos/ztools/-/merge_requests/4
[14:54:38] <wikibugs>	 06cloud-services-team, 10Toolforge: "toolforge build logs" shows error when logs have expired - https://phabricator.wikimedia.org/T415324#11583683 (10fnegri) p:05Triage→03Medium
[14:54:40] <wikibugs>	 06cloud-services-team, 10Toolforge: Upgrade Toolforge ingress-nginx to v1.14 - https://phabricator.wikimedia.org/T416366#11583684 (10taavi) p:05Triage→03Medium
[14:54:45] <wikibugs>	 (03merge) 10r1f4t: Update [toolforge-repos/ztools] - 10https://gitlab.wikimedia.org/toolforge-repos/ztools/-/merge_requests/4
[14:55:13] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.14.3
[14:55:27] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.14.3
[14:55:43] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Adding a new user to codfw1dev is messed up - https://phabricator.wikimedia.org/T416483#11583686 (10taavi) p:05Triage→03Low
[14:55:45] <wikibugs>	 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#11583687 (10fnegri) p:05Triage→03Medium
[14:55:58] <wikibugs>	 06cloud-services-team, 10Toolforge: Upgrade Toolforge ingress-nginx to v1.14 - https://phabricator.wikimedia.org/T416366#11583689 (10taavi) ` v1.14.3: digest: sha256:8bc6fcbcca76baa27777a09a82ce7143c8ae6831f0a9a6aae3a4acf868f6bffb size: 3469                             `
[14:56:30] <wikibugs>	 06cloud-services-team, 10Data-Services: Deprecate /data/scratch volume - https://phabricator.wikimedia.org/T415819#11583690 (10fnegri) p:05Triage→03Medium
[15:00:51] <wikibugs>	 10Tool-redminbot: Edit across all namespaces in one go based on a sensible list - https://phabricator.wikimedia.org/T415890#11583729 (10Redmin)
[15:08:46] <wikibugs>	 (03open) 10r1f4t: update [toolforge-repos/ztools] - 10https://gitlab.wikimedia.org/toolforge-repos/ztools/-/merge_requests/5
[15:08:52] <wikibugs>	 (03merge) 10r1f4t: update [toolforge-repos/ztools] - 10https://gitlab.wikimedia.org/toolforge-repos/ztools/-/merge_requests/5
[15:18:15] <wikibugs>	 (03open) 10taavi: ingress-nginx: Upgrade ingress-nginx to 1.14.3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1133 (https://phabricator.wikimedia.org/T416366)
[15:18:17] <wikibugs>	 (03update) 10taavi: ingress-nginx: Upgrade ingress-nginx to 1.14.3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1133 (https://phabricator.wikimedia.org/T416366)
[15:18:28] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
[15:18:54] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Upgrade Toolforge ingress-nginx to v1.14 - https://phabricator.wikimedia.org/T416366#11583832 (10taavi) a:03taavi
[15:20:33] <wikibugs>	 (03update) 10taavi: ingress-nginx: Upgrade ingress-nginx to 1.14.3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1133 (https://phabricator.wikimedia.org/T416366)
[15:20:55] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
[15:21:10] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
[15:23:46] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
[15:29:24] <wikibugs>	 (03approved) 10fnegri: ingress-nginx: Upgrade ingress-nginx to 1.14.3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1133 (https://phabricator.wikimedia.org/T416366) (owner: 10taavi)
[15:30:29] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
[15:32:08] <wmcs-alerts>	 FIRING: [2x] IngressPodMisplaced: ingress-nginx-gen2 pod misplaced - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/IngressPodMisplaced  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DIngressPodMisplaced
[15:33:08] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
[15:35:22] <wikibugs>	 (03merge) 10taavi: ingress-nginx: Upgrade ingress-nginx to 1.14.3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1133 (https://phabricator.wikimedia.org/T416366)
[15:37:02] <wikibugs>	 10VPS-project-Codesearch: Codesearch down/unreachable (2026-02-04) - https://phabricator.wikimedia.org/T416488#11583923 (10SomeRandomDeveloper) 05Open→03Resolved Seems to be resolved.
[15:37:08] <wmcs-alerts>	 FIRING: [3x] IngressPodMisplaced: ingress-nginx-gen2 pod misplaced - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/IngressPodMisplaced  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DIngressPodMisplaced
[15:40:40] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Upgrade Toolforge ingress-nginx to v1.14 - https://phabricator.wikimedia.org/T416366#11583928 (10taavi) 05Open→03Resolved
[15:47:08] <wmcs-alerts>	 RESOLVED: [3x] IngressPodMisplaced: ingress-nginx-gen2 pod misplaced - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/IngressPodMisplaced  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DIngressPodMisplaced
[16:16:13] <wikibugs>	 (03PS1) 10Majavah: vps: delete_project: Be careful about project name/ID separation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236773
[16:16:13] <wikibugs>	 (03PS1) 10Majavah: vps: delete_project: Delete DNS zones [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236774
[16:18:03] <wikibugs>	 (03PS2) 10Majavah: vps: delete_project: Be careful about project name/ID separation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236773
[16:18:03] <wikibugs>	 (03PS2) 10Majavah: vps: delete_project: Delete DNS zones [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236774
[16:18:10] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 k8splay START - Cookbook wmcs.vps.delete_project for project k8splay in eqiad1
[16:18:13] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 k8splay END (FAIL) - Cookbook wmcs.vps.delete_project (exit_code=99) for project k8splay in eqiad1
[16:21:31] <wikibugs>	 (03CR) 10CI reject: [V:04-1] vps: delete_project: Delete DNS zones [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236774 (owner: 10Majavah)
[16:22:08] <wikibugs>	 (03CR) 10CI reject: [V:04-1] vps: delete_project: Be careful about project name/ID separation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236773 (owner: 10Majavah)
[16:28:18] <wikibugs>	 (03PS3) 10Majavah: vps: delete_project: Be careful about project name/ID separation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236773
[16:28:18] <wikibugs>	 (03PS3) 10Majavah: vps: delete_project: Delete DNS zones [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236774
[16:28:18] <wikibugs>	 (03PS1) 10Majavah: vps: delete_project: Delete floating IPs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236776
[16:30:54] <wikibugs>	 (03CR) 10CI reject: [V:04-1] vps: delete_project: Delete floating IPs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236776 (owner: 10Majavah)
[16:31:00] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Deleting a project should delete associated sudo rules in LDAP - https://phabricator.wikimedia.org/T416497 (10taavi) 03NEW
[16:31:03] <wikibugs>	 (03CR) 10CI reject: [V:04-1] vps: delete_project: Delete DNS zones [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236774 (owner: 10Majavah)
[16:32:03] <wikibugs>	 (03PS4) 10Majavah: vps: delete_project: Be careful about project name/ID separation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236773
[16:32:03] <wikibugs>	 (03PS4) 10Majavah: vps: delete_project: Delete DNS zones [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236774
[16:32:03] <wikibugs>	 (03PS2) 10Majavah: vps: delete_project: Delete floating IPs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236776
[16:32:04] <wikibugs>	 (03PS1) 10Majavah: build: Bump to Python 3.11 [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236778
[16:33:07] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 k8splay START - Cookbook wmcs.vps.delete_project for project k8splay in eqiad1
[16:33:16] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 k8splay END (FAIL) - Cookbook wmcs.vps.delete_project (exit_code=99) for project k8splay in eqiad1
[16:35:10] <wikibugs>	 (03PS3) 10Majavah: vps: delete_project: Delete floating IPs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236776
[16:35:19] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 k8splay START - Cookbook wmcs.vps.delete_project for project k8splay in eqiad1
[16:35:27] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 k8splay END (FAIL) - Cookbook wmcs.vps.delete_project (exit_code=99) for project k8splay in eqiad1
[16:36:12] <wikibugs>	 (03CR) 10CI reject: [V:04-1] vps: delete_project: Delete floating IPs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236776 (owner: 10Majavah)
[16:36:33] <wikibugs>	 (03PS4) 10Majavah: vps: delete_project: Delete floating IPs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236776
[16:36:41] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 k8splay START - Cookbook wmcs.vps.delete_project for project k8splay in eqiad1
[16:36:54] <wikibugs>	 (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project k8splay [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/288
[16:38:19] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 k8splay END (FAIL) - Cookbook wmcs.vps.delete_project (exit_code=99) for project k8splay in eqiad1
[16:40:19] <wikibugs>	 (03CR) 10CI reject: [V:04-1] vps: delete_project: Delete floating IPs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236776 (owner: 10Majavah)
[16:41:52] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Adding a new user to codfw1dev is messed up - https://phabricator.wikimedia.org/T416483#11584223 (10Andrew) ` 2026-02-04 16:40:46.922 500255 ERROR keystone.server.flask.application [None req-92a79b9d-c3f8-45d9-940f-3337cd379c89 novaadmin admin - - default default] Invalid in...
[16:42:09] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Adding a new user to a codfw1dev project is messed up - https://phabricator.wikimedia.org/T416483#11584226 (10Andrew)
[16:42:14] <wikibugs>	 (03PS5) 10Majavah: vps: delete_project: Delete DNS zones [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236774
[16:42:14] <wikibugs>	 (03PS5) 10Majavah: vps: delete_project: Delete floating IPs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236776
[16:42:14] <wikibugs>	 (03PS1) 10Majavah: vps: delete_project: Ask what to do on Tofu failures [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236782
[16:42:14] <wikibugs>	 (03PS1) 10Majavah: vps: delete_project: Only run Tofu in relevant cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236783
[16:42:35] <wikibugs>	 (03close) 10taavi: projects: delete project k8splay [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/288 (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461)
[16:43:14] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 k8splay START - Cookbook wmcs.vps.delete_project for project k8splay in eqiad1
[16:43:27] <wikibugs>	 (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project k8splay [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/289
[16:43:53] <wikibugs>	 (03merge) 10taavi: projects: delete project k8splay [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/289 (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461)
[16:44:18] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 k8splay END (PASS) - Cookbook wmcs.vps.delete_project (exit_code=0) for project k8splay in eqiad1
[16:46:26] <wikibugs>	 (03PS6) 10Majavah: vps: delete_project: Delete floating IPs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236776 (https://phabricator.wikimedia.org/T392680)
[16:48:12] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 media-streaming START - Cookbook wmcs.vps.delete_project for project media-streaming in eqiad1
[16:48:37] <wikibugs>	 (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project media-streaming [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/290
[16:48:41] <wikibugs>	 (03CR) 10FNegri: [C:03+1] build: Bump to Python 3.11 [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236778 (owner: 10Majavah)
[16:49:48] <wikibugs>	 (03merge) 10taavi: projects: delete project media-streaming [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/290 (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461)
[16:49:58] <wikibugs>	 (03CR) 10Majavah: [C:03+2] build: Bump to Python 3.11 [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236778 (owner: 10Majavah)
[16:50:14] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 media-streaming END (PASS) - Cookbook wmcs.vps.delete_project (exit_code=0) for project media-streaming in eqiad1
[16:50:16] <stashbot>	 taavi@cloudcumin1001: Unknown project "media-streaming"
[16:51:47] <wikibugs>	 (03CR) 10FNegri: [C:03+1] vps: delete_project: Only run Tofu in relevant cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236783 (owner: 10Majavah)
[16:52:04] <wikibugs>	 (03CR) 10FNegri: [C:03+1] vps: delete_project: Ask what to do on Tofu failures [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236782 (owner: 10Majavah)
[16:52:50] <wikibugs>	 (03CR) 10FNegri: [C:03+1] vps: delete_project: Be careful about project name/ID separation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236773 (owner: 10Majavah)
[16:53:28] <wikibugs>	 (03Merged) 10jenkins-bot: build: Bump to Python 3.11 [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236778 (owner: 10Majavah)
[16:54:02] <wikibugs>	 (03CR) 10Majavah: [C:03+2] vps: delete_project: Be careful about project name/ID separation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236773 (owner: 10Majavah)
[16:54:07] <wikibugs>	 (03CR) 10Majavah: [C:03+2] vps: delete_project: Ask what to do on Tofu failures [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236782 (owner: 10Majavah)
[16:54:11] <wikibugs>	 (03CR) 10Majavah: [C:03+2] vps: delete_project: Only run Tofu in relevant cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236783 (owner: 10Majavah)
[16:55:06] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 sso START - Cookbook wmcs.vps.delete_project for project sso in eqiad1
[16:55:25] <wikibugs>	 (03update) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project sso [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/291
[16:55:28] <wikibugs>	 (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project sso [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/291
[16:56:32] <wikibugs>	 (03merge) 10taavi: projects: delete project sso [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/291 (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461)
[16:56:38] <wikibugs>	 (03CR) 10FNegri: [C:03+1] vps: delete_project: Delete floating IPs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236776 (https://phabricator.wikimedia.org/T392680) (owner: 10Majavah)
[16:57:10] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 sso END (PASS) - Cookbook wmcs.vps.delete_project (exit_code=0) for project sso in eqiad1
[16:57:12] <stashbot>	 taavi@cloudcumin1001: Unknown project "sso"
[16:57:32] <wikibugs>	 (03Merged) 10jenkins-bot: vps: delete_project: Be careful about project name/ID separation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236773 (owner: 10Majavah)
[16:57:32] <wikibugs>	 (03Merged) 10jenkins-bot: vps: delete_project: Ask what to do on Tofu failures [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236782 (owner: 10Majavah)
[16:57:35] <wikibugs>	 (03CR) 10FNegri: [C:03+1] vps: delete_project: Delete DNS zones [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236774 (owner: 10Majavah)
[16:57:36] <wikibugs>	 (03Merged) 10jenkins-bot: vps: delete_project: Only run Tofu in relevant cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236783 (owner: 10Majavah)
[17:05:15] <wikibugs>	 (03PS1) 10Majavah: vps: delete_project: Check for volumes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236785
[17:05:15] <wikibugs>	 (03PS1) 10Majavah: vps: delete_project: Check for database instances [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236786
[17:05:37] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 vuessr START - Cookbook wmcs.vps.delete_project for project vuessr in eqiad1
[17:05:38] <stashbot>	 taavi@cloudcumin1001: Unknown project "vuessr"
[17:05:44] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Adding a new user to a codfw1dev project is messed up - https://phabricator.wikimedia.org/T416483#11584284 (10Andrew) upstream but: https://bugs.launchpad.net/keystone/+bug/2140354
[17:06:05] <wikibugs>	 (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project vuessr [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/292
[17:06:35] <wikibugs>	 (03merge) 10taavi: projects: delete project vuessr [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/292 (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461)
[17:06:36] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Upstream: Adding a new user to a codfw1dev project is messed up - https://phabricator.wikimedia.org/T416483#11584286 (10taavi)
[17:07:04] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 vuessr END (PASS) - Cookbook wmcs.vps.delete_project (exit_code=0) for project vuessr in eqiad1
[17:07:04] <stashbot>	 taavi@cloudcumin1001: Unknown project "vuessr"
[17:08:31] <wikibugs>	 (03CR) 10CI reject: [V:04-1] vps: delete_project: Check for volumes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236785 (owner: 10Majavah)
[17:08:58] <wikibugs>	 (03CR) 10CI reject: [V:04-1] vps: delete_project: Check for database instances [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236786 (owner: 10Majavah)
[17:11:25] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 checkuser-beta-wiki START - Cookbook wmcs.vps.delete_project for project checkuser-beta-wiki in eqiad1
[17:11:56] <wikibugs>	 (03update) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project checkuser-beta-wiki [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/293
[17:11:58] <wikibugs>	 (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project checkuser-beta-wiki [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/293
[17:12:24] <wikibugs>	 (03merge) 10taavi: projects: delete project checkuser-beta-wiki [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/293 (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461)
[17:12:42] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 checkuser-beta-wiki END (PASS) - Cookbook wmcs.vps.delete_project (exit_code=0) for project checkuser-beta-wiki in eqiad1
[17:13:06] <wikibugs>	 (03PS2) 10Majavah: vps: delete_project: Check for volumes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236785
[17:13:06] <wikibugs>	 (03PS2) 10Majavah: vps: delete_project: Check for database instances [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236786
[17:14:56] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 duct START - Cookbook wmcs.vps.delete_project for project duct in eqiad1
[17:14:58] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 duct END (FAIL) - Cookbook wmcs.vps.delete_project (exit_code=99) for project duct in eqiad1
[17:15:43] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 duct START - Cookbook wmcs.vps.delete_project for project duct in eqiad1
[17:15:46] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 duct END (FAIL) - Cookbook wmcs.vps.delete_project (exit_code=99) for project duct in eqiad1
[17:16:42] <wikibugs>	 (03CR) 10CI reject: [V:04-1] vps: delete_project: Check for database instances [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236786 (owner: 10Majavah)
[17:28:41] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 wmf-dumps-playground START - Cookbook wmcs.vps.delete_project for project wmf-dumps-playground in eqiad1
[17:29:10] <wikibugs>	 (03update) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project wmf-dumps-playground [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/294
[17:29:11] <wikibugs>	 (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project wmf-dumps-playground [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/294
[17:29:44] <wikibugs>	 (03merge) 10taavi: projects: delete project wmf-dumps-playground [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/294 (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461)
[17:30:18] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 wmf-dumps-playground END (PASS) - Cookbook wmcs.vps.delete_project (exit_code=0) for project wmf-dumps-playground in eqiad1
[17:35:49] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 duct START - Cookbook wmcs.vps.delete_project for project duct in eqiad1
[17:35:54] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 duct END (FAIL) - Cookbook wmcs.vps.delete_project (exit_code=99) for project duct in eqiad1
[17:36:21] <wikibugs>	 (03PS3) 10Majavah: vps: delete_project: Check for database instances [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236786
[17:36:56] <jinxer-wm>	 FIRING: [2x] SystemdUnitDown: The service unit labs-ip-alias-dump.service is in failed status on host cloudservices1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown  - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[17:38:08] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 duct START - Cookbook wmcs.vps.delete_project for project duct in eqiad1
[17:38:35] <wikibugs>	 (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project duct [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/295
[17:38:59] <wikibugs>	 (03merge) 10taavi: projects: delete project duct [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/295 (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461)
[17:39:32] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 duct END (PASS) - Cookbook wmcs.vps.delete_project (exit_code=0) for project duct in eqiad1
[18:31:56] <jinxer-wm>	 RESOLVED: [2x] SystemdUnitDown: The service unit labs-ip-alias-dump.service is in failed status on host cloudservices1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown  - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[18:54:48] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Upstream: Adding a new user to a codfw1dev project is messed up - https://phabricator.wikimedia.org/T416483#11584751 (10Andrew) At least one upstream dev has asserted that this is a bug fix and not a regression, which would mean that we've been setting the 'enabled' flag...
[19:00:46] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Upstream: Adding a new user to a codfw1dev project is messed up - https://phabricator.wikimedia.org/T416483#11584796 (10Andrew) p:05Low→03Medium I'm elevating the priority because this is a blocker for upgrading eqiad1 to flamingo
[19:28:17] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[19:38:17] <jinxer-wm>	 RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[21:51:24] <wikibugs>	 (03PS1) 10Andrew Bogott: Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841
[21:54:50] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841 (owner: 10Andrew Bogott)
[22:28:28] <wikibugs>	 (03update) 10bd808: Update list of channels following our changes in Phabricator workboard [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/65 (https://phabricator.wikimedia.org/T415352) (owner: 10matthieulec)
[22:37:33] <wikibugs>	 (03merge) 10bd808: Update list of channels following our changes in Phabricator workboard [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/65 (https://phabricator.wikimedia.org/T415352) (owner: 10matthieulec)
[22:43:46] <wikibugs>	 10Wikibugs, 06ServiceOps new: Add serviceops new to the monitored channels on IRC bot/wikibugs - https://phabricator.wikimedia.org/T415352#11585494 (10bd808) Config change was merged and has been pushed to the wikibugs api server. Does this comment lead to the bot joining `#wikimedia-serviceops-feed` and annou...
[22:44:48] <wikibugs>	 10Wikibugs, 06ServiceOps new: Add serviceops new to the monitored channels on IRC bot/wikibugs - https://phabricator.wikimedia.org/T415352#11585495 (10bd808) 05Open→03Resolved `lang=irc,name=#wikimedia-serviceops-feed [22:43]  wikibugs (~wikibugs@wikimedia/bot/pywikibugs) joined the channel [22:43]  <...
[23:17:53] <wikibugs>	 10Tool-campwiz-nxt, 10Google-Summer-of-Code (Google Summer of Code (2026)): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11585568 (10LGoto) Hi @Nokib_Sarkar GSoC contacted us to note that this project is listed as 120 hours, which is not allowed. Please update it to 90 or 175 or...