[00:50:40] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11026451 (10Scott_French) Alright, MediaWiki deployments should no longer be at risk: the php8.1 production images have been rebuilt on `docker-registry.dis... [01:12:34] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11026454 (10Scott_French) [02:09:31] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11026493 (10Scott_French) [07:06:34] 06serviceops, 06cloud-services-team, 10Toolforge: Shield wmcloud.org and toolforge.org​ against crawler traffic - https://phabricator.wikimedia.org/T400212#11026650 (10PerfektesChaos) >>! In T400212#11025942, @bd808 wrote: > Possible duplicate of {T226688}. Nope. I am requesting harsh action for both wmclou... [07:15:29] 06serviceops, 06cloud-services-team, 10Toolforge: Shield wmcloud.org and toolforge.org​ against crawler traffic - https://phabricator.wikimedia.org/T400212#11026653 (10PerfektesChaos) >>! In T400212#11025977, @bd808 wrote: > A robots.txt file is only advisory guidance for well-behaved web bots: The request... [07:31:36] 06serviceops, 06cloud-services-team, 10Toolforge: Shield wmcloud.org and toolforge.org​ against crawler traffic - https://phabricator.wikimedia.org/T400212#11026723 (10PerfektesChaos) The defense strategy does need a restart, since our tools, services and usage of BETA suffer from significant limitations in... [08:52:30] 06serviceops, 06cloud-services-team, 10Toolforge: Shield wmcloud.org and toolforge.org​ against crawler traffic - https://phabricator.wikimedia.org/T400212#11026858 (10taavi) >>! In T400212#11026650, @PerfektesChaos wrote: > T226688 is dealing with beta.wmcloud.org only. I do not see a single mention of bet... [08:52:41] 06serviceops, 06cloud-services-team, 10Toolforge: Shield wmcloud.org and toolforge.org​ against crawler traffic - https://phabricator.wikimedia.org/T400212#11026860 (10taavi) →14Duplicate dup:03T226688 [09:21:55] 06serviceops, 10Cassandra, 06Growth-Team, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#11027008 (10Michael) >>! In T368096#11025294, @Scott_French wrote: > The data-gateway listener is now available (though unused) in production Me... [09:43:06] 06serviceops, 06MW-Interfaces-Team, 07OKR-Work, 13Patch-For-Review: Improved API rerouting strategy for REST gateway - https://phabricator.wikimedia.org/T400131#11027062 (10Clement_Goubert) 05Open→03In progress p:05Triage→03Medium a:03Clement_Goubert [10:32:29] 06serviceops, 06Commons, 10Thumbor, 13Patch-For-Review, 07Wikimedia-production-error: No thumbnail available for File:Himalaya,_Indian_Atlas,_sheet_66_(15219000).jpg - Error: 429, Too Many Requests - https://phabricator.wikimedia.org/T392348#11027269 (10Clement_Goubert) We have rolled out a first change... [11:18:48] 06serviceops, 06Commons, 10Thumbor, 13Patch-For-Review, 07Wikimedia-production-error: No thumbnail available for File:Himalaya,_Indian_Atlas,_sheet_66_(15219000).jpg - Error: 429, Too Many Requests - https://phabricator.wikimedia.org/T392348#11027377 (10hnowlan) I've checked each of the images mentioned... [12:32:34] 06serviceops, 13Patch-For-Review: Publish Wikimedia trixie base Docker image - https://phabricator.wikimedia.org/T393173#11027619 (10taavi) [13:58:31] 06serviceops, 10Cassandra, 06Growth-Team, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#11027903 (10Eevans) >>! In T368096#11027008, @Michael wrote: >>>! In T368096#11025294, @Scott_French wrote: >> The data-gateway listener is now... [14:01:31] 06serviceops: ☂️ [FY2025-26][Hypothesis] WE6.2.1 Production Readiness Checklist - https://phabricator.wikimedia.org/T400263 (10jijiki) 03NEW [14:35:19] 06serviceops: Informal chats with software engineers around wmf - https://phabricator.wikimedia.org/T400272 (10jijiki) 03NEW [14:35:27] 06serviceops: Informal chats with software engineers around wmf - https://phabricator.wikimedia.org/T400272#11028135 (10jijiki) 05Open→03In progress p:05Triage→03Medium [14:35:45] 06serviceops: Informal chats with software engineers around wmf - https://phabricator.wikimedia.org/T400272#11028138 (10jijiki) [14:43:42] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11028166 (10dancy) Thanks for the fixes @Scott_French ! [14:55:02] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11028177 (10Scott_French) [15:20:20] 06serviceops, 06Commons, 10Thumbor, 07Wikimedia-production-error: No thumbnail available for File:Himalaya,_Indian_Atlas,_sheet_66_(15219000).jpg - Error: 429, Too Many Requests - https://phabricator.wikimedia.org/T392348#11028304 (10MatthewVernon) 05Open→03Resolved a:03MatthewVernon Thanks @hnow... [15:28:54] 06serviceops, 06MW-Interfaces-Team, 07OKR-Work: Identify and configure rest.php routes in REST gateway - https://phabricator.wikimedia.org/T400132#11028333 (10hnowlan) Dump of sandbox keys from testwiki - there may be other keys on other wikis based on what's enabled I believe? This is good enough to make a... [15:40:57] 06serviceops, 06Commons, 10Thumbor, 07Wikimedia-production-error: No thumbnail available for File:Himalaya,_Indian_Atlas,_sheet_66_(15219000).jpg - Error: 429, Too Many Requests - https://phabricator.wikimedia.org/T392348#11028427 (10Yann) Wow, great! Thanks to you all! [15:49:54] 06serviceops, 06Infrastructure-Foundations, 06SRE: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11028456 (10Ottomata) Should we perhaps use `latest` tag for Gitlab CI images? I suppose other things could break if the base image is silently upgraded between different pipeli... [15:56:29] 06serviceops, 06MediaWiki-Engineering, 13Patch-For-Review, 10Release-Engineering-Team (Radar): Deprecate mwdebugXXXX hosts - https://phabricator.wikimedia.org/T397498#11028494 (10thcipriani) [16:06:01] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11028543 (10Scott_French) @Ottomata - So, image build workflows in CI that use the `latest` tag would still have been affected by this, but they would have... [16:17:01] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11028586 (10Scott_French) [17:10:26] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11028782 (10Ottomata) FTR, went with versioned tag for repeatability. [17:28:16] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade httpd images to bullseye or bookworm - https://phabricator.wikimedia.org/T378128#11028898 (10Scott_French) 05Open→03Resolved The tags have been deleted for all three transitional images, which should b... [19:12:02] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11029224 (10Jdforrester-WMF) >>! In T383557#11026158, @Scott_French wrote: > I'm no longer seeing any references to bullseye-backports in puppet, so I belie... [19:16:58] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11029230 (10Scott_French) @Jdforrester-WMF - Basically, the rebuilds would need to start at the first image that depends on `docker-registry.discovery.wmnet... [19:20:12] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11029245 (10dancy) @Jdforrester-WMF I'll do the docker-pkg stuff and pass it by you for review. [19:34:23] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11029288 (10dancy) docker-registry.wikimedia.org/python3-devel:latest is another image that needs a rebuild. [21:06:11] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11029488 (10Scott_French) [22:16:44] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11029582 (10Scott_French) [22:42:36] 06serviceops, 10MW-on-K8s: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#11029608 (10RLazarus) @Krinkle Thanks for the report (and sorry for the long wait, I'm just back from sabbatical yesterday). This should now work: ` $ mwscript-k8s --dblist=s1 --follow -- mysql.php --... [22:58:48] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11029630 (10Scott_French)