[05:23:34] 06serviceops, 10MediaWiki-Core-JobQueue, 10WMF-JobQueue, 13Patch-Needs-Improvement: Find a way to set elevated timeouts for job running - https://phabricator.wikimedia.org/T247114#11357196 (10Pppery) [11:14:15] 06serviceops, 06MediaWiki-Platform-Team, 06Traffic, 07OKR-Work: api-gateway helm chart: rest routes should return retry-after when a rate limit applies. - https://phabricator.wikimedia.org/T405636#11357980 (10Clement_Goubert) After thinking about it a bit, I tend to agree with @pmiazga Adding #traffic for... [15:21:12] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11359142 (10herron) a:05herron→03None FWIW T326419 has some details about the last rebalance on kafka-logging [16:10:19] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11359365 (10brouberol) a:03brouberol [16:36:33] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11359541 (10brouberol) The leadership skew is expected, as it is a result of //storage// rebalancing. Some brokers are leaders of few large... [16:36:56] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11359542 (10brouberol) Which I'll run on wednesday, as tomorrow is a bank holiday in France. [18:58:02] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.3 production workload migration - https://phabricator.wikimedia.org/T405955#11360350 (10Scott_French) Updates for today: * As of ~ 18:05 - 18:20 UTC, we're now serving 50% of residual external web and API traffic on PHP 8.3. * As of ~ 18:35 UTC, mw-videosca... [19:08:19] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.3 production workload migration - https://phabricator.wikimedia.org/T405955#11360387 (10Scott_French) [19:43:27] 06serviceops, 13Patch-For-Review: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245#11360536 (10Scott_French) The pilot on conf2006 today went smoothly. No issues encountered following the switch to the new cfssl/pki-based certs. Total elapsed from SIGTERM to the new... [21:11:57] 06serviceops, 13Patch-For-Review: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245#11360864 (10ssingh) >>! In T352245#11360536, @Scott_French wrote: > For PyBals located in codfw, I'm on the fence. If we consider (clean) PyBal restarts to be trivially safe, then it... [23:00:35] Hey, could someone merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1202872 for me?