[02:14:15] 06cloud-services-team, 10Toolforge: Add support for Python 3.13 - https://phabricator.wikimedia.org/T381899#10395699 (10Bugreporter) Note if you must use Python 3.13, you can create a Bookworm Kubernetes shell using `toolforge webservice python3.11 shell` (note you can no longer use Bastion for this since deco... [05:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:01:44] 10Cloud-VPS (Quota-requests), 06collaboration-services: cloud VPS project codesearch or devtools - quota increase request for storage volumes - https://phabricator.wikimedia.org/T381911#10395914 (10dcaro) > Is this considered a lot or not at all compared to what you have available for all of cloud VPS? It's... [08:12:41] !log fnegri@cloudcumin1001 codesearch START - Cookbook wmcs.openstack.quota_increase (T381911) [08:12:45] T381911: cloud VPS project codesearch or devtools - quota increase request for storage volumes - https://phabricator.wikimedia.org/T381911 [08:12:49] !log fnegri@cloudcumin1001 codesearch END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T381911) [08:13:37] 10Cloud-VPS (Quota-requests), 06collaboration-services: cloud VPS project codesearch or devtools - quota increase request for storage volumes - https://phabricator.wikimedia.org/T381911#10396001 (10fnegri) 05Open→03Resolved a:03fnegri Quota increased, it's now 230GB. [08:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:12:37] (03update) 10sstefanova: functional tests: add components-api tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/631 (https://phabricator.wikimedia.org/T379092) [09:18:19] (03update) 10sstefanova: functional tests: add components-api tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/631 (https://phabricator.wikimedia.org/T379092) [09:28:39] (03update) 10sstefanova: functional tests: add components-api tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/631 (https://phabricator.wikimedia.org/T379092) [09:38:52] (03update) 10sstefanova: functional tests: add components-api tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/631 (https://phabricator.wikimedia.org/T379092) [09:43:01] 10Tool-lexeme-forms, 06translatewiki.net, 10LPL Essential (LPL Essential 2024 Nov-Dec), 13Patch-For-Review, 07Unplanned-Sprint-Work: translatewiki export for Wikidata Lexeme Forms tries to remove sh-latn translations - https://phabricator.wikimedia.org/T379188#10396265 (10abi_) We've disabled translation... [09:44:09] (03update) 10sstefanova: functional tests: add components-api tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/631 (https://phabricator.wikimedia.org/T379092) [09:55:32] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Trove DB full - https://phabricator.wikimedia.org/T381745#10396283 (10Magnus) Yes. No joy. [10:18:34] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Trove DB full - https://phabricator.wikimedia.org/T381745#10396385 (10Andrew) @Magnus sorry for the hangup, looks like Trove automation isn't great at this yet. I did a bit of hands-on repair and I think you're good to go now (at least, all the status flags... [10:26:48] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Trove volume resize doesnt always (ever?) work - https://phabricator.wikimedia.org/T381959 (10Andrew) 03NEW [10:27:08] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Trove volume resize doesnt always (ever?) work - https://phabricator.wikimedia.org/T381959#10396426 (10Andrew) p:05Triage→03Medium a:05rook→03None [10:27:52] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Trove volume resize doesnt always (ever?) work - https://phabricator.wikimedia.org/T381959#10396429 (10Andrew) Oh, I should add that there was a roadbump with running resize2fs because the volume was mounted both on the VM OS level and also within the docker... [10:28:46] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10396430 (10Andrew) >>! In T376267#10394651, @taavi wrote: >> I was just looking over @Sarai-WMF's shoulder and wikitech is for some reason not editable for her. > What exactly does this mean... [10:30:03] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudcephmon100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T380893#10396433 (10Andrew) These hosts have a somewhat unusual vlan setup, so my guess is something is tripping on that -- paging @cmooney for m... [10:43:37] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Trove DB full - https://phabricator.wikimedia.org/T381745#10396494 (10Magnus) @Andrew Works again, thanks! [10:58:00] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Trove DB full - https://phabricator.wikimedia.org/T381745#10396529 (10Andrew) 05Open→03Resolved awesome [11:30:53] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS (Quota-requests): Subdomain for catalyst-dev project - https://phabricator.wikimedia.org/T381508#10396641 (10bd808) Horizon presented error: ` Error: Unable to create proxy: 500 Internal Server Error

In... [12:39:25] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10396858 (10jhsoby) [13:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:24:00] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10397194 [14:35:59] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 06Data-Persistence: [wikireplicas] Route alerts to WMCS team - https://phabricator.wikimedia.org/T381589#10397263 (10fnegri) We discussed this today with @Marostegui and @joanna_borun and we agreed that the following alerts should exist and should b... [14:53:32] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10397336 (10Leaderboard) Can you force-create a local account for Leaderbot? It fails when I try to login, telling me that I am not authorised to perform the action. [15:11:41] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10397411 (10ppenloglou) Hey all! I'm able to login using my Developer account and using Special:PasswordReset but I'm having trouble trying to log in with my SUL account. When I enter my SUL... [15:19:49] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for idwikivoyage - https://phabricator.wikimedia.org/T381079#10397462 (10Marostegui) a:03fnegri Per our chat in the offsite, assigning to @fnegri [15:24:56] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10397494 [15:48:05] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10397569 (10Reedy) ` reedy@deploy2002:~$ mwscript extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=labswiki "Leaderbot" DEPRECATION WARNING: Maintenance scripts are moving to... [16:05:27] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Trove volume resize doesnt always (ever?) work - https://phabricator.wikimedia.org/T381959#10397636 (10JJMC89) [17:06:47] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for idwikivoyage - https://phabricator.wikimedia.org/T381079#10398051 (10fnegri) [17:32:41] 10Tool-wikiqanda, 06Future-Audiences: Add testing capabilities to Slack bot - https://phabricator.wikimedia.org/T379029#10398194 (10etz) a:05etz→03None [17:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:52:52] 10Tool-ducttape, 06Abstract Wikipedia team, 10function-orchestrator: function-orchestrator publish-dev-image step not running - https://phabricator.wikimedia.org/T347487#10398326 (10Jdforrester-WMF) 05Open→03Invalid [17:56:09] 10Tool-wikiqanda, 06Future-Audiences, 07Spike: [Spike] Explore integrating with/using WikiChat methodology to serve bot Q&A - https://phabricator.wikimedia.org/T382020 (10Maryana) 03NEW [17:56:33] 10Tool-wikiqanda, 06Future-Audiences, 07Spike: [Spike] Explore integrating with/using WikiChat methodology to serve bot Q&A - https://phabricator.wikimedia.org/T382020#10398361 (10Maryana) [17:56:36] 10Tool-wikiqanda, 06Future-Audiences, 07Epic: [Epic] Discord Q&A bot Milestone 3 - https://phabricator.wikimedia.org/T378126#10398362 (10Maryana) [17:57:34] 10Tool-wikiqanda, 06Future-Audiences, 07Spike: [Spike] Explore integrating with/using WikiChat methodology to serve bot Q&A - https://phabricator.wikimedia.org/T382020#10398372 (10Maryana) [18:07:13] 10Tool-wikiqanda, 06Future-Audiences, 07Spike: [Spike] Explore integrating with/using WikiChat methodology to serve bot Q&A - https://phabricator.wikimedia.org/T382020#10398397 (10Maryana) [18:08:11] 10Tool-wikiqanda, 06Future-Audiences, 07Spike: [Spike] Explore integrating with/using WikiChat methodology to serve bot Q&A - https://phabricator.wikimedia.org/T382020#10398401 (10derenrich) wikichat - as deployed it is using a static index. but they let you plug in a different retrieval system that we could... [18:16:05] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10398428 (10ppenloglou) @Reedy Confirming I can login with my SUL on Wikitech now! Thank you so much! [18:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:34:06] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10398765 (10GiovanniPen) Hello, when I try to [[https://wikitech.wikimedia.org/wiki/Special:UserLogin|log]] I receive this error: "Auto-creation of a local account failed: You are not allowed... [19:41:03] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10398773 (10Reedy) ` reedy@deploy2002:~$ mwscript extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=labswiki "GiovanniPen" DEPRECATION WARNING: Maintenance scripts are moving t... [20:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:29:34] 10Tool-ducttape, 06Abstract Wikipedia team, 10Wikifunctions: Run DUCT's end-to-end testing system on new function-orchestrator patches on GitLab, like we do on WikiLambda on gerrit - https://phabricator.wikimedia.org/T333191#10398969 (10Jdforrester-WMF) 05Open→03Declined This has been replaced by the... [21:19:13] 06cloud-services-team, 10Toolforge, 10wikitech.wikimedia.org, 10Diffusion, and 2 others: Document diffusion->github mirroring to https://github.com/toolforge/ on wikitech - https://phabricator.wikimedia.org/T361859#10399125 (10Aklapper) [21:59:25] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Kernel error Server cloudvirt1061 may have kernel errors - https://phabricator.wikimedia.org/T380673#10399164 (10Jclark-ctr) i have updated firmwares and dell sees no issues. these where ordered with 512 memory and are listing the correct amou... [22:19:07] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Kernel error Server cloudvirt1061 may have kernel errors - https://phabricator.wikimedia.org/T380673#10399201 (10wiki_willy) @Jclark-ctr - there's nothing that I'm aware of. If there's no additional info in the original procurement task or an... [23:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks