[02:37:34] <wikibugs>	 10Tool-inteGraality: Error (first time using inteGraality) - https://phabricator.wikimedia.org/T415138#11585825 (10RobinIsadorable) Not exactly sure how, but figured it out. Reformatted based on templates.
[04:40:39] <wikibugs>	 10Tool-redminbot: Explore visual regression testing - https://phabricator.wikimedia.org/T415882#11585912 (10Redmin)
[05:54:07] <wikibugs>	 10Tool-redminbot: Handle div-span-flip errors - https://phabricator.wikimedia.org/T416463#11585985 (10Redmin)
[05:59:33] <wikibugs>	 (03open) 10r4356th: Handle div-span-flip errors [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/9 (https://phabricator.wikimedia.org/T416463)
[06:00:45] <wikibugs>	 (03update) 10r4356th: Handle div-span-flip errors [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/9 (https://phabricator.wikimedia.org/T416463)
[06:37:24] <wikibugs>	 (03update) 10r4356th: Handle div-span-flip errors [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/9 (https://phabricator.wikimedia.org/T416463)
[06:41:57] <wikibugs>	 (03update) 10r4356th: Handle div-span-flip errors [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/9 (https://phabricator.wikimedia.org/T416463)
[06:42:46] <wikibugs>	 (03update) 10r4356th: Handle div-span-flip errors [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/9 (https://phabricator.wikimedia.org/T416463)
[06:45:49] <wikibugs>	 10Tool-redminbot, 13Patch-For-Review: Handle div-span-flip errors - https://phabricator.wikimedia.org/T416463#11586033 (10Redmin)
[06:50:11] <wikibugs>	 (03update) 10r4356th: Handle div-span-flip errors [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/9 (https://phabricator.wikimedia.org/T416463)
[07:04:28] <wikibugs>	 (03update) 10r4356th: Handle div-span-flip errors [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/9 (https://phabricator.wikimedia.org/T416463)
[07:05:39] <wikibugs>	 (03update) 10r4356th: Handle div-span-flip errors [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/9 (https://phabricator.wikimedia.org/T416463)
[07:07:54] <wikibugs>	 (03merge) 10r4356th: Handle div-span-flip errors [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/9 (https://phabricator.wikimedia.org/T416463)
[07:08:16] <wikibugs>	 10Tool-redminbot, 13Patch-For-Review: Handle div-span-flip errors - https://phabricator.wikimedia.org/T416463#11586046 (10Redmin) 05Open→03Resolved a:03Redmin
[07:08:32] <wikibugs>	 10Tool-redminbot: Handle div-span-flip errors - https://phabricator.wikimedia.org/T416463#11586048 (10Redmin)
[07:24:35] <wikibugs>	 (03open) 10driedmueller: Improved logging and html converter [toolforge-repos/cr-grants-team-metasync] - 10https://gitlab.wikimedia.org/toolforge-repos/cr-grants-team-metasync/-/merge_requests/4
[07:50:44] <wikibugs>	 10Tool-redminbot: Add ability to only enable or disable certain fixes - https://phabricator.wikimedia.org/T416550 (10Redmin) 03NEW
[08:35:04] <wikibugs>	 (03merge) 10dsavuljesku: Improved logging and html converter [toolforge-repos/cr-grants-team-metasync] - 10https://gitlab.wikimedia.org/toolforge-repos/cr-grants-team-metasync/-/merge_requests/4 (owner: 10driedmueller)
[09:17:48] <wikibugs>	 10Tools: [FNBot] Store user review statistics in gender neutral "Sichtungstabelle" - https://phabricator.wikimedia.org/T416487#11586282 (10Tkarcher)
[10:36:08] <wikibugs>	 10Tool-redminbot: Handle cases of multiple colons being used to escape wiki links - https://phabricator.wikimedia.org/T416083#11586605 (10Redmin)
[10:37:54] <wikibugs>	 10Tool-bub2, 13Patch-For-Review: Add docker support to Tool-Bub2 - https://phabricator.wikimedia.org/T344286#11586611 (10Aklapper)
[10:46:59] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Documentation: Document how to add SSH keys in codfw1dev LDAP deployment - https://phabricator.wikimedia.org/T416568 (10taavi) 03NEW
[10:47:12] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Upstream: openstack flamingo: "'enabled' is a required property" for LDAP-managed users - https://phabricator.wikimedia.org/T416483#11586680 (10taavi)
[10:47:20] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Upstream: openstack flamingo: "'enabled' is a required property" for LDAP-managed users - https://phabricator.wikimedia.org/T416483#11586684 (10taavi) I have split the SSH key documentation part to {T416568}.
[11:04:16] <wikibugs>	 (03update) 10fnegri: images: support harbor-based pre-built images [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/252 (https://phabricator.wikimedia.org/T409727) (owner: 10dcaro)
[11:05:05] <wikibugs>	 (03update) 10fnegri: core: add prometheus counter for jobs synced from runtime [repos/cloud/toolforge/jobs-api] (use_custom_resources_in_code) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/253 (owner: 10dcaro)
[11:12:08] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Upgrade cloudcumin hosts to bookworm - https://phabricator.wikimedia.org/T403153#11586788 (10taavi)
[11:12:24] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Data-Platform-SRE: Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#11586793 (10taavi)
[11:12:31] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Reimage cloudgw hosts to Trixie - https://phabricator.wikimedia.org/T401899#11586795 (10taavi)
[11:16:28] <jinxer-wm>	 FIRING: KeyholderUnarmed: 2 unarmed Keyholder key(s) on cloudcumin1001:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed
[11:18:31] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Data-Platform-SRE: Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#11586810 (10fgiunchedi) I spoke with @brouberol and @BTullis at the SRE Summit about this: clouddumps hosts host a variety of dumps types, including some from HDFS,...
[11:21:28] <jinxer-wm>	 RESOLVED: KeyholderUnarmed: 1 unarmed Keyholder key(s) on cloudcumin1001:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed
[11:29:29] <wmcs-alerts>	 FIRING: [6x] PuppetCertificateAboutToExpire: Puppet CA certificate pki-db.pki.eqiad1.wikimedia.cloud is about to expire in 24d 23h 32m 33s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire
[11:32:35] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Upgrade cloudcumin hosts to bookworm - https://phabricator.wikimedia.org/T403153#11586860 (10MoritzMuehlenhoff) 05Open→03Resolved Both cloudcumin nodes were upgraded to Bookworm, which means that we no longer need to build Cumin/Spicerack for Bullseye.
[11:33:22] <wikibugs>	 06cloud-services-team, 10Data-Services, 10Datasets-General-or-Unknown, 13Patch-For-Review: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#11586864 (10taavi)
[11:48:42] <wikibugs>	 06cloud-services-team, 10Data-Services, 10Datasets-General-or-Unknown, 13Patch-For-Review: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#11586934 (10taavi) a:03taavi
[12:00:26] <wikibugs>	 06cloud-services-team, 10Data-Services, 13Patch-For-Review: Add support for x1 and x4 sections on wiki replicas on the load balancer layer - https://phabricator.wikimedia.org/T409560#11587013 (10taavi) 05Open→03Stalled a:05taavi→03None Unclaiming and stalling until x1 is unblocked via `T415219`.
[12:06:33] <wikibugs>	 (03CR) 10Majavah: "recheck" [labs/striker] - 10https://gerrit.wikimedia.org/r/1229802 (owner: 10Libraryupgrader)
[12:09:37] <wikibugs>	 (03CR) 10Majavah: [C:03+2] build: Updating lodash to 4.17.23 [labs/striker] - 10https://gerrit.wikimedia.org/r/1229802 (owner: 10Libraryupgrader)
[12:10:19] <wikibugs>	 (03CR) 10Majavah: [C:03+2] vps: delete_project: Delete floating IPs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236776 (https://phabricator.wikimedia.org/T392680) (owner: 10Majavah)
[12:10:20] <wikibugs>	 (03CR) 10Majavah: [C:03+2] vps: delete_project: Delete DNS zones [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236774 (owner: 10Majavah)
[12:10:52] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Deleting a project does not release floating IPs for that project - https://phabricator.wikimedia.org/T392680#11587105 (10taavi) 05Open→03Resolved a:03taavi
[12:13:41] <wikibugs>	 (03Merged) 10jenkins-bot: vps: delete_project: Delete DNS zones [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236774 (owner: 10Majavah)
[12:14:19] <wikibugs>	 (03Merged) 10jenkins-bot: vps: delete_project: Delete floating IPs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236776 (https://phabricator.wikimedia.org/T392680) (owner: 10Majavah)
[12:27:03] <wikibugs>	 (03open) 10raymond-ndibe: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T348755)
[12:30:05] <wikibugs>	 (03update) 10mercy-o: Fix bugs and improve UX in microtask generator [toolforge-repos/microtask-generator] - 10https://gitlab.wikimedia.org/toolforge-repos/microtask-generator/-/merge_requests/3
[12:33:25] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Deleting ENC projects is broken - https://phabricator.wikimedia.org/T416588 (10taavi) 03NEW
[12:34:03] <wikibugs>	 (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] (replace_job_images_with_web_images) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T388092)
[12:34:22] <wikibugs>	 (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T348755)
[12:34:51] <wikibugs>	 (03update) 10raymond-ndibe: replace job images with web images [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/263 (https://phabricator.wikimedia.org/T415322)
[12:35:20] <wikibugs>	 (03update) 10raymond-ndibe: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T348755)
[12:35:20] <wikibugs>	 (03update) 10raymond-ndibe: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T348755)
[12:41:21] <wikibugs>	 (03PS1) 10Majavah: openstack: Drop OVS migration cookbooks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1237214
[13:02:29] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590 (10Odinaldo) 03NEW
[13:03:59] <wikibugs>	 06cloud-services-team (FY2025/2026-Q1-Q2), 10Cloud-VPS (Project-requests): CloudVPS instance for ProVe - https://phabricator.wikimedia.org/T408387#11587328 (10Odinaldo) Thank you very much! I have just opened a new task with details and diagrams and will communicate via other tasks leaving this closed.  Re...
[13:09:01] <wikibugs>	 06cloud-services-team, 10Data-Services, 10Datasets-General-or-Unknown, 13Patch-For-Review: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#11587354 (10taavi)
[13:16:38] <wikibugs>	 10Tool-redminbot: Edit across all namespaces in one go based on a sensible list - https://phabricator.wikimedia.org/T415890#11587392 (10Redmin)
[13:19:07] <wikibugs>	 (03update) 10raymond-ndibe: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T348755)
[13:19:27] <wikibugs>	 (03update) 10raymond-ndibe: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T348755)
[14:02:55] <wikibugs>	 (03open) 10raymond-ndibe: jobs-api: test for proper handling of the diff variations of the --image argument [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_webservice) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1134 (https://phabricator.wikimedia.org/T348755 https://phabricator.wikimedia.org/T414978)
[14:04:46] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#11587535 (10Aklapper) For reference and context: {T408387}
[14:05:02] <wikibugs>	 (03update) 10raymond-ndibe: jobs-api: test for proper handling of the diff variations of the --image argument [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_webservice) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1134 (https://phabricator.wikimedia.org/T348755 https://phabricator.wikimedia.org/T414978)
[14:06:44] <wikibugs>	 (03update) 10raymond-ndibe: jobs-api: test for proper handling of the diff variations of the --image argument [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_webservice) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1134 (https://phabricator.wikimedia.org/T348755 https://phabricator.wikimedia.org/T414978)
[14:08:41] <wikibugs>	 (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_publish) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T348755)
[14:09:35] <wikibugs>	 (03update) 10raymond-ndibe: jobs-api: test for proper handling of the diff variations of the --image argument [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_webservice) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1134 (https://phabricator.wikimedia.org/T348755 https://phabricator.wikimedia.org/T414978)
[14:10:22] <wikibugs>	 (03update) 10raymond-ndibe: jobs-api: test for proper handling of the diff variations of the --image argument [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_webservice) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1134 (https://phabricator.wikimedia.org/T348755 https://phabricator.wikimedia.org/T414978)
[14:11:05] <wikibugs>	 (03update) 10raymond-ndibe: test for jobs-api backend in webservice-cli [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_webservice) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1134 (https://phabricator.wikimedia.org/T348755)
[14:14:58] <wikibugs>	 (03update) 10raymond-ndibe: replace job images with web images [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/263 (https://phabricator.wikimedia.org/T415322)
[14:17:00] <wikibugs>	 (03update) 10raymond-ndibe: replace job images with web images [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/263 (https://phabricator.wikimedia.org/T415322)
[14:18:21] <wikibugs>	 (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] (replace_job_images_with_web_images) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T388092)
[14:19:48] <wikibugs>	 (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T348755)
[14:20:28] <wikibugs>	 (03update) 10raymond-ndibe: replace job images with web images [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/263 (https://phabricator.wikimedia.org/T415322)
[14:20:48] <wikibugs>	 (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] (replace_job_images_with_web_images) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T388092)
[14:21:02] <wikibugs>	 (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T348755)
[14:33:59] <wikibugs>	 (03update) 10raymond-ndibe: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T348755)
[14:38:55] <wikibugs>	 (03update) 10raymond-ndibe: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T348755)
[14:46:48] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Complete upgrading WMCS bare metal hosts to Trixie - https://phabricator.wikimedia.org/T375217#11587736 (10taavi)
[14:46:49] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Upgrade cloudcumin hosts to bookworm - https://phabricator.wikimedia.org/T403153#11587735 (10taavi)
[14:46:49] <wikibugs>	 (03update) 10raymond-ndibe: test for jobs-api backend in webservice-cli [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_webservice) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1134 (https://phabricator.wikimedia.org/T348755)
[14:46:57] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Data-Platform-SRE: Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#11587737 (10taavi)
[14:47:02] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Reimage cloudgw hosts to Trixie - https://phabricator.wikimedia.org/T401899#11587739 (10taavi)
[14:47:20] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Upgrade cloudcumin hosts to bookworm - https://phabricator.wikimedia.org/T403153#11587741 (10taavi)
[14:54:46] <wikibugs>	 (03update) 10raymond-ndibe: test for jobs-api backend in webservice-cli [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_webservice) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1134 (https://phabricator.wikimedia.org/T348755)
[14:56:57] <wikibugs>	 (03update) 10raymond-ndibe: test for jobs-api backend in webservice-cli [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_webservice) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1134 (https://phabricator.wikimedia.org/T348755)
[15:20:49] <wikibugs>	 (03update) 10raymond-ndibe: test for jobs-api backend in webservice-cli [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_webservice) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1134 (https://phabricator.wikimedia.org/T348755)
[15:23:47] <wikibugs>	 (03update) 10raymond-ndibe: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T348755)
[15:33:40] <wikibugs>	 10Tool-erinnermich: [ErinnerMich] Add a blocklist for sites where requests should be rejected - https://phabricator.wikimedia.org/T410993#11587995 (10Tkarcher) a:03Tkarcher
[15:38:10] <wikibugs>	 10Tool-wosretbot: [Wosretbot] Current logic for checking and replacing daily headers may delete content - https://phabricator.wikimedia.org/T400742#11588020 (10Tkarcher) a:03Tkarcher
[15:39:45] <wikibugs>	 10Tool-wosretbot: [WosretBot] Deletion of empty daily headers on WP:FZW - https://phabricator.wikimedia.org/T392902#11588029 (10Tkarcher) a:03Tkarcher
[15:44:27] <wikibugs>	 10VPS-project-Codesearch, 10MediaWiki-libs-HTTP, 10MediaWiki-libs-utfnormal, 13Patch-For-Review, 07PHP 8.5 support: Deprecation Notice: Function curl_close() is deprecated since 8.5, as it has no effect since PHP 8.0 - https://phabricator.wikimedia.org/T413538#11588043 (10cscott)
[16:23:27] <wikibugs>	 10VPS-project-Codesearch: Codesearch down/unreachable (2026-02-05) - https://phabricator.wikimedia.org/T416614 (10SomeRandomDeveloper) 03NEW
[16:31:51] <wikibugs>	 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#11588287 (10bd808) >>! In T416373#11580024, @KCVelaga_WMF wrote: > Perhaps just keeping it on maintenance mode, with no further additions could be considered.  Unmaintained projects are not id...
[16:32:45] <wikibugs>	 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#11588290 (10Samwalton9-WMF) Might it be possible to enable superset.wikimedia.org dashboards to be publicly viewable?
[16:33:40] <wikibugs>	 (03open) 10bd808: channels: Add comment about case-insensitive display name matching [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/66
[16:39:46] <wikibugs>	 (03merge) 10bd808: channels: Add comment about case-insensitive display name matching [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/66
[16:45:05] <wikibugs>	 (03update) 10raymond-ndibe: test for jobs-api backend in webservice-cli [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_webservice) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1134 (https://phabricator.wikimedia.org/T348755)
[16:50:43] <wikibugs>	 06cloud-services-team, 10Data-Services, 10Datasets-General-or-Unknown, 06Infrastructure-Foundations, and 3 others: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#11588444 (10taavi) This is still primarily blocked on {T415193} (for DE).  In the meantime, so...
[17:13:35] <wikibugs>	 10VPS-project-Codesearch: Codesearch down/unreachable (2026-02-05) - https://phabricator.wikimedia.org/T416614#11588517 (10A_smart_kitten) Now up for me, https://codesearch.wmcloud.org/_health/ shows all indexes up except for 'search' (which shows 'starting up')
[17:15:13] <wikibugs>	 10VPS-project-Codesearch: Codesearch down/unreachable (2026-02-05) - https://phabricator.wikimedia.org/T416614#11588536 (10SomeRandomDeveloper) 05Open→03Resolved >>! In T416614#11588517, @A_smart_kitten wrote: > Now up for me, https://codesearch.wmcloud.org/_health/ shows all indexes up except for 'searc...
[17:16:19] <wikibugs>	 (03open) 10taavi: Fix Loki metrics scraping [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1135 (https://phabricator.wikimedia.org/T407852)
[17:16:21] <wikibugs>	 (03update) 10taavi: Fix Loki metrics scraping [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1135 (https://phabricator.wikimedia.org/T407852)
[17:19:42] <wikibugs>	 (03update) 10taavi: Fix Loki metrics scraping [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1135 (https://phabricator.wikimedia.org/T407852)
[17:20:52] <wikibugs>	 (03update) 10taavi: Fix Loki metrics scraping [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1135 (https://phabricator.wikimedia.org/T407852)
[17:22:34] <wikibugs>	 (03update) 10taavi: Fix Loki metrics scraping [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1135 (https://phabricator.wikimedia.org/T407852)
[17:28:22] <wikibugs>	 (03update) 10taavi: Fix Loki metrics scraping [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1135 (https://phabricator.wikimedia.org/T407852)
[17:33:11] <wikibugs>	 10VPS-project-Codesearch: Codesearch down/unreachable (2026-02-05) - https://phabricator.wikimedia.org/T416614#11588646 (10Ladsgroup) My guess is some sort of scraper. I‌ haven't looked into it though.
[17:36:59] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 spacemedia START - Cookbook wmcs.openstack.quota_increase by 32768 ram
[17:37:06] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 spacemedia END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 32768 ram
[17:39:09] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Tool-spacemedia: Request quota increase for spacemedia project - https://phabricator.wikimedia.org/T416073#11588715 (10Andrew) 05Open→03Resolved a:03Andrew done. Sorry for the slow response; lots of travel and illness lately.
[17:41:38] <wikibugs>	 10VPS-project-Codesearch: Codesearch down/unreachable (2026-02-05) - https://phabricator.wikimedia.org/T416614#11588739 (10brennen) > My guess is some sort of scraper. I‌ haven't looked into it though.  Based on everything else lately that seems like a pretty good guess.  Anyway, seems possible the service i...
[17:44:43] <wikibugs>	 (03close) 10taavi: Fix Loki metrics scraping [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1135 (https://phabricator.wikimedia.org/T407852)
[18:03:34] <wikibugs>	 (03PS2) 10Andrew Bogott: Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841
[18:07:18] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841 (owner: 10Andrew Bogott)
[18:08:07] <wikibugs>	 (03PS3) 10Andrew Bogott: Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841
[18:11:18] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841 (owner: 10Andrew Bogott)
[18:22:24] <wikibugs>	 (03open) 10volans: infra-tracing: bump loki query limits [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1136 (https://phabricator.wikimedia.org/T415199)
[18:23:02] <wikibugs>	 (03update) 10volans: infra-tracing: bump loki query limits [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1136 (https://phabricator.wikimedia.org/T415199)
[18:26:13] <wikibugs>	 (03approved) 10taavi: infra-tracing: bump loki query limits [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1136 (https://phabricator.wikimedia.org/T415199) (owner: 10volans)
[18:28:42] <logmsgbot_cloud>	 !log volans@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing
[18:46:33] <logmsgbot_cloud>	 !log volans@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing
[18:48:07] <logmsgbot_cloud>	 !log volans@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing
[19:08:23] <logmsgbot_cloud>	 !log volans@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing
[19:11:46] <wikibugs>	 (03merge) 10volans: infra-tracing: bump loki query limits [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1136 (https://phabricator.wikimedia.org/T415199)
[19:15:06] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Tool-spacemedia: Request quota increase for spacemedia project - https://phabricator.wikimedia.org/T416073#11589065 (10Don-vip) Thank you! No worry, I hope you're all okay.
[19:22:13] <wikibugs>	 (03PS4) 10Andrew Bogott: Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841
[19:25:28] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence: Add datetime versions of timestamp fields to Wikireplica databases - https://phabricator.wikimedia.org/T414199#11589123 (10Huji) I created https://meta.wikimedia.org/wiki/Community_Wishlist/W509 accordingly
[19:26:08] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841 (owner: 10Andrew Bogott)
[20:30:53] <wikibugs>	 (03PS5) 10Andrew Bogott: Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841
[20:43:39] <wikibugs>	 (03PS6) 10Andrew Bogott: Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841
[20:43:46] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev
[20:43:50] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=99) on deployment codfw1dev
[20:44:54] <wikibugs>	 (03PS7) 10Andrew Bogott: Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841
[20:44:58] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev
[20:45:08] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=99) on deployment codfw1dev
[20:46:08] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev
[20:46:18] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=99) on deployment codfw1dev
[20:46:26] <wikibugs>	 (03PS8) 10Andrew Bogott: Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841
[20:46:30] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev
[20:47:22] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=99) on deployment codfw1dev
[20:48:17] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job rabbitmq in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[20:48:23] <wikibugs>	 (03PS9) 10Andrew Bogott: Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841
[20:48:26] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev
[20:49:04] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=99) on deployment codfw1dev
[20:51:50] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841 (owner: 10Andrew Bogott)
[20:55:35] <wikibugs>	 (03PS10) 10Andrew Bogott: Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841
[20:55:42] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev
[20:56:29] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=99) on deployment codfw1dev
[20:57:12] <wikibugs>	 (03PS11) 10Andrew Bogott: Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841
[20:57:15] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev
[20:58:02] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=99) on deployment codfw1dev
[21:00:39] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841 (owner: 10Andrew Bogott)
[21:01:13] <wikibugs>	 (03PS12) 10Andrew Bogott: Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841
[21:01:17] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev
[21:02:43] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#11589535 (10bd808)
[21:03:45] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=0) on deployment codfw1dev
[21:07:30] <wikibugs>	 (03PS13) 10Andrew Bogott: Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841
[21:07:43] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
[21:08:17] <jinxer-wm>	 RESOLVED: JobUnavailable: Reduced availability for job rabbitmq in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[21:11:47] <jinxer-wm>	 FIRING: [2x] JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[21:12:23] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
[21:16:47] <jinxer-wm>	 RESOLVED: [2x] JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[21:49:22] <wikibugs>	 (03PS1) 10Dzahn: zookeeper: add fake TLS password to match private repo [labs/private] - 10https://gerrit.wikimedia.org/r/1237343 (https://phabricator.wikimedia.org/T395938)
[21:49:51] <wikibugs>	 (03PS2) 10Dzahn: zookeeper: add fake TLS password to match private repo [labs/private] - 10https://gerrit.wikimedia.org/r/1237343 (https://phabricator.wikimedia.org/T395938)
[21:50:47] <wikibugs>	 (03PS3) 10Dzahn: zookeeper: add fake TLS password to match private repo [labs/private] - 10https://gerrit.wikimedia.org/r/1237343 (https://phabricator.wikimedia.org/T395938)
[21:51:04] <wikibugs>	 (03CR) 10Dzahn: [V:03+2 C:03+2] zookeeper: add fake TLS password to match private repo [labs/private] - 10https://gerrit.wikimedia.org/r/1237343 (https://phabricator.wikimedia.org/T395938) (owner: 10Dzahn)
[21:56:50] <wikibugs>	 10Tool-campwiz-nxt, 10Google-Summer-of-Code (Google Summer of Code (2026)): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11589726 (10LGoto) Hi @Nokib_Sarkar @Tiven2240 please see my previous comment, this is an urgent matter. Please let me know when you have updated the number of...
[21:57:39] <wikibugs>	 (03PS1) 10Dzahn: zuul: fix renamed password variable [labs/private] - 10https://gerrit.wikimedia.org/r/1237345 (https://phabricator.wikimedia.org/T395938)
[22:00:02] <wikibugs>	 (03PS2) 10Dzahn: zuul: fix renamed password variable [labs/private] - 10https://gerrit.wikimedia.org/r/1237345 (https://phabricator.wikimedia.org/T395938)
[22:01:27] <wikibugs>	 (03CR) 10Dzahn: [V:03+2 C:03+2] zuul: fix renamed password variable [labs/private] - 10https://gerrit.wikimedia.org/r/1237345 (https://phabricator.wikimedia.org/T395938) (owner: 10Dzahn)
[22:07:20] <wikibugs>	 06Toolforge-standards-committee: Adoption request for "request" tool - https://phabricator.wikimedia.org/T389540#11589746 (10bd808) >>! In T389540#11583272, @Tkarcher wrote: > Would it be possible to share the code (without the confidental data) with me prior to the adoption  This sounds like a https://wikitech....
[22:28:07] <wikibugs>	 10VPS-project-Phabricator, 10Bitu, 06collaboration-services, 06Infrastructure-Foundations: Allow Bitu to link Phabricator account - https://phabricator.wikimedia.org/T406495#11589845 (10bd808) The new Phabricator account linking feature that is live on https://idm.wikimedia.org/ldapbackend/properties/ seem...