[07:06:15] hey folks, morning [07:06:50] so the maps tegola's cache is still not warmed up, the elements were 90M (!!) and it is still regenerating tiles [07:07:11] (basically the process is to enqueue in a kafka topic the request to regeneration, and then the k8s cron picks it up at midnight) [07:07:25] I'll do some tests later on but I think we are still ready to go [07:07:56] the only doubt that I have is if the new stack will absorb all the requests (eqiad+codfw) [07:54:12] 06serviceops, 10Page Content Service, 06Traffic: Block traffic to RESTBase /page/talk endpoint and sunset it - https://phabricator.wikimedia.org/T401895#11232159 (10akosiaris) [09:40:31] jayme: jelto: so we depool Toolhub so we fail fast on failoid? [09:40:55] I'd say yes [09:41:09] it's the cleaner approach [09:41:09] Cool, editing the task [09:41:14] +1 [09:41:15] Agreed [09:46:14] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11232564 (10Clement_Goubert) [10:05:35] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11232622 (10Jelto) The update is blocked by T406094 currently [11:05:17] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11232755 (10JMeybohm) Zotero was updated to `docker-registry.discovery.wmnet/repos/mediawiki/services/... [11:15:31] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11232778 (10JMeybohm) [11:19:54] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11232793 (10JMeybohm) [11:30:13] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11232817 (10JMeybohm) [11:36:30] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11232828 (10JMeybohm) [11:50:11] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11232848 (10JMeybohm) [13:04:19] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11233088 (10hashar) [13:13:44] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11233119 (10Clement_Goubert) 05Open→03In progress [13:51:39] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11233313 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=c05638d5-ce6f-47ba-a690-9... [13:52:08] 06serviceops, 06collaboration-services, 10envoy, 06SRE, 13Patch-For-Review: Upgrade Envoy to v1.29.12 - https://phabricator.wikimedia.org/T403663#11233317 (10Eevans) The RESTBase cluster has been upgraded to v1.29.12 (sorry for the delay, I was out all last week and missed the message). [13:55:28] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11233333 (10JMeybohm) [14:02:22] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11233363 (10JMeybohm) [14:09:36] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11233419 (10Clement_Goubert) Followup: - `charlie` feature request: Just do it and don't show me any d... [14:11:40] sorry I missed some announcement or something, when are we planning to repool eqiad? I have some fun db maint I want to finish [14:13:28] Amir1: tomorrow [14:13:31] 1500UTCV [14:13:33] -V [14:15:13] cool cool [14:15:29] I have enough time to have fun [14:45:38] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11233555 (10Jelto) @BTullis @bking there is an undeployed change for flink in the `dse-k8s-eqiad` clus... [14:48:59] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11233565 (10BTullis) @Jelto thanks and apologies for that. The pending `admin_ng` diff should be mainl... [14:49:41] 06serviceops, 06MediaWiki-Engineering, 07Epic, 07Performance Issue: Limit the number of expensive API queries a user can perform - https://phabricator.wikimedia.org/T405472#11233579 (10daniel) Tagging #serviceops for awareness. [15:13:42] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11233640 (10Jelto) [15:37:09] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11233763 (10Clement_Goubert) 05In progress→03Resolved [15:43:23] 06serviceops, 10Prod-Kubernetes, 13Patch-For-Review: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#11233785 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert This has now been fixed with the upgrade in T405703 [18:03:04] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 3 others: Update wikikube eqiad to kubernetes 1.31 - https://phabricator.wikimedia.org/T405703#11234563 (10RLazarus) >>! In T405703#11233419, @Clement_Goubert wrote: > Followup: > - `charlie` f... [18:54:27] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.3 production workload migration - https://phabricator.wikimedia.org/T405955#11234718 (10Scott_French) [20:55:22] 06serviceops, 06Abstract Wikipedia team, 10function-evaluator, 07Essential-Work: Provide a dedicated for Abstract Wikipedia Rust image - https://phabricator.wikimedia.org/T380807#11235104 (10Jdforrester-WMF)