[12:54:06] 06serviceops, 06Content-Transform-Team, 06Growth-Team, 06MediaWiki-Platform-Team, and 2 others: Determine the source of internal requests going through the API gateway. - https://phabricator.wikimedia.org/T410198#11387937 (10Clement_Goubert) Tagging: - #content-transform-team for Wikifeeds - #growth-team... [12:54:53] 06serviceops, 06Content-Transform-Team, 06Growth-Team, 06MediaWiki-Platform-Team, and 2 others: Determine the source of internal requests going through the API gateway. - https://phabricator.wikimedia.org/T410198#11387940 (10hnowlan) Seems like these cases should be changed to query the [[ https://wikitech... [13:34:53] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work, 13Patch-For-Review: api rate limiting: Assign ratelimit class based on IP range - https://phabricator.wikimedia.org/T410273#11388064 (10daniel) [17:07:08] 06serviceops, 06Content-Transform-Team, 10Wikifeeds, 06Wikipedia-Android-App-Backlog (Android Release - FY2025-26): Significant increase in wikifeeds latency and mobileapps error rate since 2025/11/13 - https://phabricator.wikimedia.org/T410296#11389052 (10hnowlan) [17:35:34] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11389152 (10RobH) Depooling wikikube in rack C6: sudo cookbook sre.k8s.pool-depool-node -t T405950 -r 'network migration' --k8s-cluster wikikube-eqiad depool wikikube-worker1... [17:38:59] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11389160 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by robh@cumin2002 depool for host wikikube-worker[1260-1269].eqiad.wmnet completed: - wikikub... [17:42:58] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11389174 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by robh@cumin2002 depool for host wikikube-worker[1036,1051-1052,1054-1055,1083].eqiad.wmnet... [17:50:22] 06serviceops, 06Content-Transform-Team, 10Wikifeeds, 06Wikipedia-Android-App-Backlog (Android Release - FY2025-26): Significant increase in wikifeeds latency and mobileapps error rate since 2025/11/13 - https://phabricator.wikimedia.org/T410296#11389201 (10Dbrant) ^ after deploying the above patch, the 504... [17:58:29] 06serviceops, 06Content-Transform-Team, 10Wikifeeds, 06Wikipedia-Android-App-Backlog (Android Release - FY2025-26): Significant increase in wikifeeds latency and mobileapps error rate since 2025/11/13 - https://phabricator.wikimedia.org/T410296#11389223 (10hnowlan) >>! In T410296#11389201, @Dbrant wrote: >... [18:04:08] 06serviceops, 06Infrastructure-Foundations, 10SRE-tools: Add a --rack flag to sre.k8s.pool-depool-node - https://phabricator.wikimedia.org/T410537 (10RLazarus) 03NEW p:05Triage→03Medium [18:08:31] 06serviceops, 06Infrastructure-Foundations, 10SRE-tools: Add a --rack flag to sre.k8s.pool-depool-node - https://phabricator.wikimedia.org/T410537#11389289 (10RLazarus) (I'm not married to the specific CLI syntax in the example. Among other things, making it an --optional-flag means that the positional `host... [18:24:00] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11389329 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by robh@cumin2002 pool for host wikikube-worker[1260-1269].eqiad.wmnet completed: - wikikube-... [18:24:03] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11389330 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by robh@cumin2002 pool for host wikikube-worker[1036,1051-1052,1054-1055,1083].eqiad.wmnet co... [18:28:26] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11389339 (10RobH) Ran: sudo cookbook sre.k8s.pool-depool-node -t T405950 -r 'network migration' --k8s-cluster wikikube-eqiad pool wikikube-worker126[0-9].eqiad.wmnet sudo cookb... [18:39:31] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11389435 (10RobH) Going to depool wikikube in rack eqiad D1 for port migrations. sudo cookbook sre.k8s.pool-depool-node -t T405950 -r 'network migration' --k8s-cluster wikikube... [18:43:29] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11389453 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by robh@cumin2002 depool for host wikikube-worker[1140-1141,1160-1161].eqiad.wmnet completed:... [18:50:14] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11389493 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by robh@cumin2002 depool for host wikikube-worker[1270-1275].eqiad.wmnet completed: - wikikub... [19:13:21] 06serviceops, 06Content-Transform-Team, 10Wikifeeds, 06Wikipedia-Android-App-Backlog (Android Release - FY2025-26): Significant increase in wikifeeds latency and mobileapps error rate since 2025/11/13 - https://phabricator.wikimedia.org/T410296#11389599 (10Scott_French) It looks like mobileapps in codfw is... [19:21:18] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11389632 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by robh@cumin2002 pool for host wikikube-worker[1140-1141,1160-1161].eqiad.wmnet completed: -... [19:21:24] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11389633 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by robh@cumin2002 pool for host wikikube-worker[1270-1275].eqiad.wmnet completed: - wikikube-... [19:31:26] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11389672 (10RobH) a:05RobH→03brouberol @brouberol, you were tagged into this task by T405950#11236474 but I don't have any feedback on the migration details for kafka-main1... [19:48:49] 06serviceops: Improve detection of - https://phabricator.wikimedia.org/T410552 (10Scott_French) 03NEW [19:49:38] 06serviceops: Improve detection of kafka-main broker certification rotation work - https://phabricator.wikimedia.org/T410552#11389725 (10Scott_French) [19:50:14] 06serviceops: Improve detection of kafka-main broker certification rotation work - https://phabricator.wikimedia.org/T410552#11389728 (10Scott_French) p:05Triage→03Low [19:56:49] 06serviceops: Improve detection of kafka-main broker TLS certificate rotations - https://phabricator.wikimedia.org/T410552#11389733 (10Scott_French) [20:19:37] 06serviceops, 07OKR-Work: Publish redioscope image to wmf registry for deployment on the aux cluster - https://phabricator.wikimedia.org/T410559 (10daniel) 03NEW [20:21:29] 06serviceops, 06Content-Transform-Team, 10Wikifeeds, 06Wikipedia-Android-App-Backlog (Android Release - FY2025-26): Significant increase in wikifeeds latency and mobileapps error rate since 2025/11/13 - https://phabricator.wikimedia.org/T410296#11389871 (10Dbrant) I wouldn't have an issue with (temporarily... [22:00:04] 06serviceops, 06Infrastructure-Foundations, 10SRE-tools: Add a --rack flag to sre.k8s.pool-depool-node - https://phabricator.wikimedia.org/T410537#11390175 (10Volans) Just for context referencing past ideas on the topic: T327300 [22:25:29] 06serviceops, 10Infrastructure Security, 06SRE, 10SRE-swift-storage, and 5 others: October 2025 Bullseye reboots: Search Platform-owned hosts - https://phabricator.wikimedia.org/T410573 (10bking) 03NEW [22:29:54] 06serviceops, 10Infrastructure Security, 06SRE, 10SRE-swift-storage, and 5 others: October 2025 Bullseye reboots: Search Platform-owned hosts - https://phabricator.wikimedia.org/T410573#11390286 (10bking) [22:54:09] 06serviceops, 06Content-Transform-Team, 10Wikifeeds, 06Wikipedia-Android-App-Backlog, 13Patch-For-Review: Significant increase in wikifeeds latency and mobileapps error rate since 2025/11/13 - https://phabricator.wikimedia.org/T410296#11390379 (10Dbrant) [23:00:42] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11390409 (10RobH) >>! In T405950#11238805, @Scott_French wrote: > conf1009 is (1) a member of eqiad main-etcd cluster, so clients will attempt to issue writes to it, (2) the ups...