[07:06:27] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Remove docker related parts from kubernetes puppet code - https://phabricator.wikimedia.org/T395870 (10JMeybohm) 03NEW [07:21:24] hello folks! [07:21:25] https://alerts.wikimedia.org/?q=%40state%3Dactive&q=%40cluster%3Dwikimedia.org&q=alertname%3DPuppetCertificateAboutToExpire [07:21:36] there are some old puppet-ca-based certs to drop [07:21:48] rec-api, proton and mobileapps [07:22:02] anybody that could double check before I drop them? [07:48:53] rec-api is blocked at the edge, pending RESTBase endpoint removal before full undeployment, see https://phabricator.wikimedia.org/T338430#10718106 [07:49:40] but for all 3, the old cergen stuff doesn't apply anymore anyway. Feel free to clean them up. [09:02:02] ooook [09:21:44] all removed [09:39:34] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Members of restricted cannot use --dblist - https://phabricator.wikimedia.org/T395521#10878775 (10Clement_Goubert) >>! In T395521#10877883, @Scott_French wrote: > Amazing - thank you very for confirming, @jrbs. I believe https://gerrit.wikimedia.org/r/1152854 sh... [10:01:34] 06serviceops, 10Wikidata, 10Wikidata-Query-Service: wikidata-updatequeryservicelag cron job failing - https://phabricator.wikimedia.org/T395814#10878854 (10Lucas_Werkmeister_WMDE) >>! In T395814#10876051, @Clement_Goubert wrote: >> Which I guess turns this task from “how do we stop it from failing” into “how... [10:07:27] 06serviceops, 10Wikidata: Improve alerting for flaky mw-cron jobs - https://phabricator.wikimedia.org/T395885 (10Clement_Goubert) 03NEW [10:07:56] 06serviceops, 10Wikidata: Improve alerting for flaky mw-cron jobs - https://phabricator.wikimedia.org/T395885#10878874 (10Clement_Goubert) 05Open→03In progress p:05Triage→03Medium [10:09:33] 06serviceops, 10Wikidata: Improve alerting for flaky mw-cron jobs - https://phabricator.wikimedia.org/T395885#10878880 (10Clement_Goubert) [10:09:34] 06serviceops, 10MW-on-K8s, 10Observability-Alerting, 10SRE Observability (FY2024/2025-Q3): Periodic job alerting - https://phabricator.wikimedia.org/T385709#10878881 (10Clement_Goubert) [10:29:32] 06serviceops, 10MediaWiki-extensions-ReadingLists, 06MW-Interfaces-Team, 10RESTBase Sunsetting, and 3 others: Switchover plan from RESTbase to REST Gateway for Reading Lists endpoints - https://phabricator.wikimedia.org/T384891#10878924 (10hnowlan) Are we ready to proceed with the quasi-group1 migration at... [10:59:47] 06serviceops, 10DiscussionTools, 10MediaWiki-REST-API, 06MW-Interfaces-Team, and 3 others: Reroute RESTbase Endpoints for Linter Errors to MediaWiki REST Endpoints - https://phabricator.wikimedia.org/T384216#10878997 (10hnowlan) >>! In T384216#10864899, @daniel wrote: >>>! In T384216#10863969, @hnowlan wro... [11:19:19] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s: growthexperiments-deleteoldsurveys OOMKilled during June 1st run - https://phabricator.wikimedia.org/T395893 (10hnowlan) 03NEW [11:42:19] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s: growthexperiments-deleteoldsurveys OOMKilled during June 1st run - https://phabricator.wikimedia.org/T395893#10879226 (10hnowlan) a:05hnowlan→03None Growth looks fairly linear over time: {F61372606} I see that batch size is set [[ https:... [11:42:43] 06serviceops, 06Language and Product Localization: Migrate language_and_product_localization jobs to mw-cron - https://phabricator.wikimedia.org/T388539#10879230 (10Nikerabbit) I noticed {T395892} was tagged with #lpl_essential, but that's not where triage incoming tasks regularly. How would I go about cha... [11:45:33] 06serviceops, 06Language and Product Localization: Migrate language_and_product_localization jobs to mw-cron - https://phabricator.wikimedia.org/T388539#10879234 (10Clement_Goubert) >>! In T388539#10879230, @Nikerabbit wrote: > I noticed {T395892} was tagged with #lpl_essential, but that's not where triage... [11:49:31] 06serviceops, 10LPL Essential (LPL Essential 2025 Apr-Jun: CX): MediaWiki periodic job purge-old-cx-drafts failed - https://phabricator.wikimedia.org/T395892#10879256 (10hnowlan) This job failed because the job attempted to run on a wiki without the translation extension: ` test2wiki The "ContentTranslation"... [11:52:54] 06serviceops, 10LPL Essential (LPL Essential 2025 Apr-Jun: CX): MediaWiki periodic job purge-old-cx-drafts failed - https://phabricator.wikimedia.org/T395892#10879268 (10Nikerabbit) The ideal solution would be to skip running the script on test2wiki (or make test2wiki CX enabled for consistency though that mig... [12:09:36] 06serviceops, 10LPL Essential (LPL Essential 2025 Apr-Jun: CX): MediaWiki periodic job purge-old-cx-drafts failed - https://phabricator.wikimedia.org/T395892#10879306 (10hnowlan) This is happening because although the job excludes `dblists/testwikis.dblist`, test2wiki itself is not seen as an official test wik... [14:10:12] 06serviceops, 10MediaWiki-extensions-ReadingLists, 06MW-Interfaces-Team, 10RESTBase Sunsetting, and 3 others: Switchover plan from RESTbase to REST Gateway for Reading Lists endpoints - https://phabricator.wikimedia.org/T384891#10879603 (10MSantos) >>! In T384891#10811349, @HCoplin-WMF wrote: > @hnowlan --... [15:29:49] 06serviceops, 10LPL Essential (LPL Essential 2025 Apr-Jun: CX): MediaWiki periodic job purge-old-cx-drafts failed - https://phabricator.wikimedia.org/T395892#10880018 (10A_smart_kitten) [16:10:16] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Members of restricted cannot use --dblist - https://phabricator.wikimedia.org/T395521#10880555 (10Scott_French) 05Open→03Resolved a:03Scott_French With https://gerrit.wikimedia.org/r/1152854 merged, I believe this problem should be fixed. Thanks again... [16:41:36] 06serviceops, 06LPL Essential, 13Patch-For-Review: MediaWiki periodic job purge-old-cx-drafts failed - https://phabricator.wikimedia.org/T395892#10880702 (10Clement_Goubert) Please leave the #lpl_essential tag until the PHID has been changed to avoid duplicate issues being created. [16:42:15] 06serviceops, 10LPL Essential (LPL Essential 2025 Apr-Jun: CX), 13Patch-For-Review: MediaWiki periodic job purge-old-cx-drafts failed - https://phabricator.wikimedia.org/T395892#10880706 (10Clement_Goubert) [16:42:45] 06serviceops, 06LPL Essential, 13Patch-For-Review: MediaWiki periodic job purge-old-cx-drafts failed - https://phabricator.wikimedia.org/T395892#10880707 (10Clement_Goubert) Of course the two tags are incompatible. [16:51:24] 06serviceops, 06LPL Essential, 13Patch-For-Review: MediaWiki periodic job purge-old-cx-drafts failed - https://phabricator.wikimedia.org/T395892#10880732 (10Pppery) #lpl-current is a milestone of #lpl-essential so a task can't be in both. It's a bug in phaultfinder that it can't find tasks in a milestone of... [17:39:32] 06serviceops, 06LPL Essential, 13Patch-For-Review: MediaWiki periodic job purge-old-cx-drafts failed - https://phabricator.wikimedia.org/T395892#10881009 (10A_smart_kitten) ^ above issue regarding duplicate @phaultfinder task creations split to {T395942} [21:03:14] 06serviceops, 06MediaWiki-Engineering, 13Patch-For-Review: Clean up UcfirstOverrides.php following PHP 7.4 -> 8.1 transition - https://phabricator.wikimedia.org/T394556#10881833 (10Scott_French) Alright, I was able to run `uppercaseTitlesForUnicodeTransition.php` across all wikis in the default dry-run mode... [21:19:50] 06serviceops, 10Deployments, 10Release-Engineering-Team (Radar), 07Wikimedia-production-error: httpb sometimes fails upon deployment with a HTTP 503 - https://phabricator.wikimedia.org/T380958#10881891 (10Mstyles) ` 21:11:43 Started check-testservers 21:11:43 Executing check 'check_testservers_baremetal-1_... [22:04:58] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10882040 (10Scott_French) @Zabe - There will be an additional announcement soon, but similar to the guidance around other not-yet-supported use cases like `sql.php` in [[ https://...