[02:22:29] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow using helper scripts inside of mwscript-k8s - https://phabricator.wikimedia.org/T378479#10842481 (10RLazarus) 05Open→03Resolved a:03RLazarus This is now supported via the --dblist flag! See https://wikitech.wikimedia.org/wiki/Maintenance_scripts#... [07:45:51] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10842714 (10akosiaris) >>! In T352956#10839001, @akosiaris wrote: > #### Long-term > > We probably want to mini... [08:21:52] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10842876 (10Vgutierrez) >>! In T352956#10839348, @akosiaris wrote: > `lang=bash > nobody@wmfdebug:/$ ip link > 1... [08:53:52] 06serviceops, 10Prod-Kubernetes, 06SRE, 07Kubernetes: improve docker registry architecture - https://phabricator.wikimedia.org/T209271#10843046 (10elukey) 05Open→03Resolved a:03elukey [08:54:12] 06serviceops, 10Prod-Kubernetes, 06SRE, 07Kubernetes: Set up a local redis proxy since docker-registry can only connect to one redis instance for caching - https://phabricator.wikimedia.org/T215809#10843055 (10elukey) 05Open→03Resolved a:03elukey Already implemented. [09:08:54] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10843107 (10akosiaris) staging-eqiad with an MTU of 1460 as well. `lang=bash 2: eth0@if841: 06serviceops, 13Patch-For-Review: Migrate etcd::tlsproxy Nginx certs and etcd itself to PKI - https://phabricator.wikimedia.org/T352245#10843210 (10Vgutierrez) >>! In T352245#10808382, @Scott_French wrote: > Alright, after a quick review, the main thing that's changed since November is the migration of all (hi... [09:45:21] 06serviceops, 10MediaWiki-Special-pages, 13Patch-For-Review: Migrate updatequerypages/update_special_pages/initsitestats jobs to mw-cron - https://phabricator.wikimedia.org/T388534#10843280 (10hnowlan) [09:51:48] 06serviceops, 10MediaWiki-extensions-ReadingLists, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 13Patch-For-Review: Switchover plan from RESTbase to REST Gateway for Reading Lists endpoints - https://phabricator.wikimedia.org/T384891#10843298 (10hnowlan) The lists API on testwiki is now routed via the re... [10:05:24] 06serviceops, 06Commons, 10MediaWiki-File-management, 10MW-on-K8s, 13Patch-For-Review: Migrate cleanupUploadStash job to mw-cron - https://phabricator.wikimedia.org/T385868#10843341 (10hnowlan) 05In progress→03Resolved a:05Clement_Goubert→03hnowlan [10:24:54] 06serviceops, 06DBA, 10Editing-team (Tracking), 10MW-1.45-notes (1.45.0-wmf.1; 2025-05-13), and 2 others: Fatal exception of type "DBUnexpectedError: Database servers in extension1 are overloaded." affecting page views - https://phabricator.wikimedia.org/T393513#10843412 (10Ladsgroup) Eventgate slowdown ma... [10:34:47] nemo-yiannis: we should probably redirect here, seems like it's not a traffic issue [10:34:50] https://phabricator.wikimedia.org/P76361 [10:55:43] ok, thanks! [10:55:59] 06serviceops, 06DBA, 10Editing-team (Tracking), 10MW-1.45-notes (1.45.0-wmf.1; 2025-05-13), and 2 others: Fatal exception of type "DBUnexpectedError: Database servers in extension1 are overloaded." affecting page views - https://phabricator.wikimedia.org/T393513#10843528 (10Ladsgroup) It looks like kafka-m... [10:58:00] 06serviceops, 06DBA, 10Editing-team (Tracking), 10MW-1.45-notes (1.45.0-wmf.1; 2025-05-13), and 2 others: Fatal exception of type "DBUnexpectedError: Database servers in extension1 are overloaded." affecting page views - https://phabricator.wikimedia.org/T393513#10843529 (10Ladsgroup) That looks like swapp... [11:59:27] 06serviceops, 10Discovery-Search (2025.05.02 - 2025.05.23), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10843822 (10Clement_Goubert) >>! In T388538#10840891, @EBernhardson wrote: >>>! In T388538#10839770, @Clement_Goubert wrote: >> Coming back to t... [12:42:00] 06serviceops, 06DBA, 10Editing-team (Tracking), 10MW-1.45-notes (1.45.0-wmf.1; 2025-05-13), and 2 others: Fatal exception of type "DBUnexpectedError: Database servers in extension1 are overloaded." affecting page views - https://phabricator.wikimedia.org/T393513#10843920 (10Ladsgroup) >>! In T393513#108435... [13:05:49] 06serviceops, 06DBA, 10Editing-team (Tracking), 10MW-1.45-notes (1.45.0-wmf.1; 2025-05-13), and 2 others: Fatal exception of type "DBUnexpectedError: Database servers in extension1 are overloaded." affecting page views - https://phabricator.wikimedia.org/T393513#10844067 (10Ladsgroup) It might be the reaso... [13:27:33] 06serviceops, 06DBA, 10Editing-team (Tracking), 10MW-1.45-notes (1.45.0-wmf.1; 2025-05-13), and 2 others: Fatal exception of type "DBUnexpectedError: Database servers in extension1 are overloaded." affecting page views - https://phabricator.wikimedia.org/T393513#10844208 (10Ladsgroup) After switching the c... [14:15:07] 06serviceops, 10MW-on-K8s: Investigate startingDeadlineSeconds setting for kubernetes CronJobs - https://phabricator.wikimedia.org/T394423#10844399 (10Clement_Goubert) 05In progress→03Resolved Jobs that need to complete and not be restarted if they run longer should set `concurrency_policy: Forbid` and... [14:41:09] 06serviceops, 10MW-on-K8s, 06Trust and Safety Product Team, 10MediaModeration (MediaModeration 2.1), 13Patch-For-Review: Migrate MediaModeration jobs to mw-cron - https://phabricator.wikimedia.org/T385799#10844561 (10hnowlan) >>! In T385799#10531958, @Dreamy_Jazz wrote: >>>! In T385799#10531677, @Clement... [14:41:45] 06serviceops, 10MW-on-K8s, 06Trust and Safety Product Team, 10MediaModeration (MediaModeration 2.1), 13Patch-For-Review: Migrate MediaModeration jobs to mw-cron - https://phabricator.wikimedia.org/T385799#10844570 (10Dreamy_Jazz) >>! In T385799#10844561, @hnowlan wrote: >>>! In T385799#10531958, @Dreamy_... [15:45:32] 06serviceops, 10MW-on-K8s, 06Trust and Safety Product Team, 10MediaModeration (MediaModeration 2.1): Migrate MediaModeration jobs to mw-cron - https://phabricator.wikimedia.org/T385799#10844880 (10hnowlan) >>! In T385799#10844570, @Dreamy_Jazz wrote: >>>! In T385799#10844561, @hnowlan wrote: >>>>! In T3857... [16:30:05] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 13Patch-For-Review: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10845115 (10hnowlan) >>! In T385782#10839937, @Michael wrote: >>>! In T385782#10839778, @hnowlan wrote: >> Now that we've la... [16:49:10] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 13Patch-For-Review: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10845163 (10hnowlan) [17:33:09] 06serviceops, 10MW-on-K8s, 06Trust and Safety Product Team, 10MediaModeration (MediaModeration 2.1): Migrate MediaModeration jobs to mw-cron - https://phabricator.wikimedia.org/T385799#10845271 (10Dreamy_Jazz) >>! In T385799#10844880, @hnowlan wrote: >>>! In T385799#10844570, @Dreamy_Jazz wrote: >>>>! In T... [17:33:55] 06serviceops, 10MW-on-K8s, 06Trust and Safety Product Team, 10MediaModeration (MediaModeration 2.1): Migrate MediaModeration jobs to mw-cron - https://phabricator.wikimedia.org/T385799#10845272 (10Dreamy_Jazz) I'm going to start a different scan of all wikis because we have improved the success rate of sca... [17:38:33] 06serviceops, 10MediaWiki-Special-pages: Migrate updatequerypages/update_special_pages/initsitestats jobs to mw-cron - https://phabricator.wikimedia.org/T388534#10845286 (10Scott_French) [17:39:52] 06serviceops, 10MediaWiki-Special-pages: Migrate updatequerypages/update_special_pages/initsitestats jobs to mw-cron - https://phabricator.wikimedia.org/T388534#10845294 (10Scott_French) Remaining update-special-pages jobs have been migrated, following a successful manual test of the s6 shard earlier today. Fi... [18:27:15] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10845430 (10cmooney) What might be worth testing is if PMTUD works. i.e. send a UDP packet to a POD IP of 1500... [19:46:26] 06serviceops, 13Patch-For-Review: Turn down MediaWiki image builds for PHP 7.4 - https://phabricator.wikimedia.org/T391057#10845618 (10Scott_French) [19:47:33] 06serviceops, 13Patch-For-Review: Turn down MediaWiki image builds for PHP 7.4 - https://phabricator.wikimedia.org/T391057#10845625 (10Scott_French) Last step here is to merge https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/172 and `scap sync-world` to integrate it. I'll aim to do that eithe... [22:58:43] 06serviceops: Clean up UcfirstOverrides.php following PHP 7.4 -> 8.1 transition - https://phabricator.wikimedia.org/T394556#10846104 (10Scott_French) After spending some time reading through T292552 and the code in `uppercaseTitlesForUnicodeTransition.php` [0], I think I better understand the process here. One... [23:21:13] 06serviceops, 13Patch-For-Review: Migrate etcd::tlsproxy Nginx certs and etcd itself to PKI - https://phabricator.wikimedia.org/T352245#10846183 (10Scott_French) Thank you very much, @Vgutierrez - this is great, and thank you for your offer to assist with validation. So, I think the two phases of the migratio...