[09:16:31] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Fix dependencies between admin_ng deployments - https://phabricator.wikimedia.org/T389080#10886863 (10JMeybohm) a:03JMeybohm [09:18:49] 06serviceops, 06Security-Team, 10MW-1.45-notes (1.45.0-wmf.3; 2025-05-27), 13Patch-For-Review, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10886866 (10Clement_Goubert) ` swift stat global-data-captcha-render | grep Objects Objects:... [09:30:30] FYI, kubestagemaster2005 will briefly go down for a reboot [09:41:14] 06serviceops, 10MediaWiki-extensions-ReadingLists, 06MW-Interfaces-Team, 10RESTBase Sunsetting, and 3 others: Switchover plan from RESTbase to REST Gateway for Reading Lists endpoints - https://phabricator.wikimedia.org/T384891#10886963 (10hnowlan) Restbaseless reading list APIs are being rolled out for ca... [10:12:03] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Fix dependencies between admin_ng deployments - https://phabricator.wikimedia.org/T389080#10887016 (10JMeybohm) I made a small script to visualize dependencies while fixing them, not sure where to put it so: `lang=python, name=depend... [10:30:31] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Make it possible to bootstrap a production linke k8s cluster locally - https://phabricator.wikimedia.org/T396107 (10JMeybohm) 03NEW [10:36:05] 06serviceops, 06Security-Team, 10MW-1.45-notes (1.45.0-wmf.3; 2025-05-27), 13Patch-For-Review, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10887146 (10Clement_Goubert) 05In progress→03Resolved a:03Clement_Goubert ` extensions/ConfirmEdi... [10:36:15] 06serviceops, 06Security-Team, 10MW-1.45-notes (1.45.0-wmf.3; 2025-05-27), 13Patch-For-Review, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10887150 (10Clement_Goubert) [10:37:36] 06serviceops, 10Deployments, 10Release-Engineering-Team (Radar), 07Wikimedia-production-error: httpb sometimes fails upon deployment with a HTTP 503 - https://phabricator.wikimedia.org/T380958#10887157 (10akosiaris) >>! In T380958#10885815, @cjming wrote: > encountered this deploying today - https://spider... [10:38:03] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Implement periodic maintenance scripts for mw-on-k8s - https://phabricator.wikimedia.org/T341555#10887160 (10Clement_Goubert) 05In progress→03Resolved All jobs have been migrated \o/ [10:40:38] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Fix dependencies between admin_ng deployments - https://phabricator.wikimedia.org/T389080#10887173 (10Clement_Goubert) >>! In T389080#10887016, @JMeybohm wrote: > I made a small script to visualize dependencies w... [10:49:53] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s: growthexperiments-deleteoldsurveys OOMKilled during June 1st run - https://phabricator.wikimedia.org/T395893#10887201 (10Clement_Goubert) Limits have been reintroduced. [10:51:16] 06serviceops, 10Deployments, 10Release-Engineering-Team (Radar), 07Wikimedia-production-error: httpb sometimes fails upon deployment with a HTTP 503 - https://phabricator.wikimedia.org/T380958#10887212 (10akosiaris) I 'll be hammering mwdebug2001 with siege for the next 30 minutes in an effort to reproduce... [10:53:20] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10887218 (10Clement_Goubert) [10:56:02] 06serviceops, 10MW-on-K8s: Limit CPU usage for mw-on-k8s cli deployments - https://phabricator.wikimedia.org/T395436#10887230 (10Clement_Goubert) Limited resources for mediawiki containers created in these namespaces: CPU request/limit to 1 RAM request/limit to 1000Mi/2000Mi [11:05:00] 06serviceops, 06collaboration-services, 10GitLab, 07Kubernetes: Trusted gitlab runner containers need access to staging k8s cluster - https://phabricator.wikimedia.org/T325385#10887284 (10JMeybohm) 05Resolved→03Open We dismantled the CI access to k8s staging in {T288629}, so access from Gitlab runners... [11:18:09] 06serviceops, 06Security-Team, 10MW-1.45-notes (1.45.0-wmf.3; 2025-05-27), 13Patch-For-Review, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10887317 (10A_smart_kitten) >>! In T388531#10856208, @Clement_Goubert wrote: >>>! In T388531#10856196,... [12:25:30] claime: With your check-off of the last use in T319432, are we OK to disable PHP 7.4 testing in CI? [13:07:06] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 07Wikimedia-production-error: growthexperiments-fixlinkrecommendationdata-dryrun 'link-recommendation' is not a link recommendation task type error - https://phabricator.wikimedia.org/T395249#10887601 (10Michael) >>! In T395249#10874954, @Cl... [13:13:05] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, and 2 others: growthexperiments-fixlinkrecommendationdata-dryrun 'link-recommendation' is not a link recommendation task type error - https://phabricator.wikimedia.org/T395249#10887615 (10Michael) >>! In T395249#10887601, @Michael wrote: > * `... [13:18:06] James_F: let me check with swfrench-wmf first but I think so [13:28:28] 06serviceops, 10GrowthExperiments, 10MW-on-K8s, 10Growth-Team (Current Sprint), and 2 others: growthexperiments-fixlinkrecommendationdata-dryrun 'link-recommendation' is not a link recommendation task type error - https://phabricator.wikimedia.org/T395249#10887659 (10Michael) >>! In T395249#10874954, @Cle... [13:28:40] 06serviceops, 10GrowthExperiments, 10MW-on-K8s, 10Growth-Team (Current Sprint), and 2 others: growthexperiments-fixlinkrecommendationdata-dryrun 'link-recommendation' is not a link recommendation task type error - https://phabricator.wikimedia.org/T395249#10887661 (10Michael) a:03Michael [13:30:59] 06serviceops, 10GrowthExperiments, 10MW-on-K8s, 10Growth-Team (Current Sprint), and 2 others: growthexperiments-fixlinkrecommendationdata-dryrun 'link-recommendation' is not a link recommendation task type error - https://phabricator.wikimedia.org/T395249#10887666 (10Clement_Goubert) Maybe #sre_observabil... [14:07:30] 06serviceops, 06Security-Team, 10MW-1.45-notes (1.45.0-wmf.3; 2025-05-27), 13Patch-For-Review, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10887779 (10sbassett) >>! In T388531#10887317, @A_smart_kitten wrote: >>> Slight side note (I mentioned... [14:39:02] 06serviceops, 10Deployments, 10Release-Engineering-Team (Radar), 07Wikimedia-production-error: httpb sometimes fails upon deployment with a HTTP 503 - https://phabricator.wikimedia.org/T380958#10887906 (10akosiaris) Close to 100k requests later to `Main_Page` of `spcom.wikimedia.org`, 1 error. This is belo... [17:58:58] I have a new shellbox version to push out for T364249 (unblocked by dropping PHP 7.4 tests earlier today). It has been like a year and a half since I did a shellbox deployment. Is it still considered best practice to update all of the deployments after bumping the image version in helmfile.d/services/shellbox/global.yaml? [18:15:49] 06serviceops, 13Patch-For-Review: Build php-uuid package, and add to WMF production and CI - https://phabricator.wikimedia.org/T373752#10888653 (10Jdforrester-WMF) [18:59:38] 06serviceops, 10decommission-hardware, 13Patch-For-Review: decommission mw135[8-9], mw136[4-6], mw137[2-3], mw140[0-4], mw1406, mw14[11-13] - https://phabricator.wikimedia.org/T383227#10888788 (10jasmine_) [19:01:22] 06serviceops, 10decommission-hardware, 13Patch-For-Review: decommission mw135[8-9], mw136[4-6], mw137[2-3], mw140[0-4], mw1406, mw14[11-13] - https://phabricator.wikimedia.org/T383227#10888793 (10jasmine_) [19:11:23] 06serviceops, 13Patch-For-Review: Build php-uuid package, and add to WMF production and CI - https://phabricator.wikimedia.org/T373752#10888814 (10Jdforrester-WMF) 05Open→03Resolved [19:58:27] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10888893 (10Krinkle) I'm trying to migrate the following idiom from mwmaint (using mwscript or foreachwikiindblist) to the deployment host (using mwscript-k8s). ` $ foreachwikiin... [20:46:08] 06serviceops, 13Patch-For-Review: Publish Wikimedia trixie base Docker image - https://phabricator.wikimedia.org/T393173#10888999 (10Jdforrester-WMF) [21:46:11] 06serviceops, 06Release-Engineering-Team, 10Scap: Scap failing in beta with `Error: You are using an unsupported PHP version (PHP 7.4.33)` - https://phabricator.wikimedia.org/T396158 (10thcipriani) 03NEW [21:47:39] 06serviceops, 06Release-Engineering-Team, 10Scap: Scap failing in beta with `Error: You are using an unsupported PHP version (PHP 7.4.33)` - https://phabricator.wikimedia.org/T396158#10889270 (10thcipriani) I also wonder if this is related to {T380958} as `php_fpm` is the version passed to a script to gracef... [22:05:13] 06serviceops, 06Release-Engineering-Team, 10Scap: Scap failing in beta with `Error: You are using an unsupported PHP version (PHP 7.4.33)` - https://phabricator.wikimedia.org/T396158#10889380 (10thcipriani) >>! In T396158#10889270, @thcipriani wrote: > I also wonder if this is related to {T380958} as `php_fp... [22:25:42] 06serviceops, 13Patch-For-Review: Converge / update image version across shellbox service instances (cleanup) - https://phabricator.wikimedia.org/T388260#10889448 (10bd808) `lang=irc [17:58] < bd808> I have a new shellbox version to push out for T364249 (unblocked by dropping PHP 7.4 tests earlier today).... [22:46:57] 06serviceops, 13Patch-For-Review: Converge / update image version across shellbox service instances (cleanup) - https://phabricator.wikimedia.org/T388260#10889477 (10bd808) [22:49:22] 06serviceops, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 10Scap, 13Patch-For-Review: Scap failing in beta with `Error: You are using an unsupported PHP version (PHP 7.4.33)` - https://phabricator.wikimedia.org/T396158#10889479 (10bd808) [23:25:58] 06serviceops, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 10Scap, 13Patch-For-Review: Scap failing in beta with `Error: You are using an unsupported PHP version (PHP 7.4.33)` - https://phabricator.wikimedia.org/T396158#10889516 (10thcipriani) 05Open→03Resolved a:03thcipriani @Sco... [23:59:00] 06serviceops, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 10Scap, 07Essential-Work: Scap failing in beta with `Error: You are using an unsupported PHP version (PHP 7.4.33)` - https://phabricator.wikimedia.org/T396158#10889621 (10thcipriani)