[09:10:37] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker11XX - https://phabricator.wikimedia.org/T408749#11365411 (10Clement_Goubert) a:05Clement_Goubert→03Jhancock.wm Puppet updated [09:10:53] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker1335-59 - https://phabricator.wikimedia.org/T408752#11365413 (10Clement_Goubert) a:03Jhancock.wm Puppet updated [09:11:09] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11365415 (10Clement_Goubert) a:05Clement_Goubert→03Jhancock.wm Puppet updated [09:12:50] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker2332-56 - https://phabricator.wikimedia.org/T408757#11365423 (10Clement_Goubert) Puppet updated, but we've got some work to do so the hosts can be racked in E/F (see https://phabricator.wikimedia.org/T405... [09:32:03] 06serviceops, 07sre-alert-triage: Alert in need of triage: SystemdUnitFailed (instance registry1005:9100) - https://phabricator.wikimedia.org/T409817#11365493 (10Clement_Goubert) @Blake will handle this one [09:46:53] 06serviceops, 07sre-alert-triage: Alert in need of triage: SystemdUnitFailed (instance registry1005:9100) - https://phabricator.wikimedia.org/T409817#11365534 (10Blake) 05Open→03In progress p:05Triage→03Low a:03Blake [10:32:13] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work, 13Patch-For-Review: Test api rate limiting on production cluster - https://phabricator.wikimedia.org/T406498#11365700 (10daniel) [10:34:47] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work, 13Patch-For-Review: Test api rate limiting on production cluster - https://phabricator.wikimedia.org/T406498#11365705 (10daniel) [10:59:51] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work, 13Patch-For-Review: Test api rate limiting on production cluster - https://phabricator.wikimedia.org/T406498#11365817 (10daniel) [11:17:17] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work, 13Patch-For-Review: Test api rate limiting on production cluster - https://phabricator.wikimedia.org/T406498#11365887 (10daniel) [11:47:42] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work: Test api rate limiting on production cluster - https://phabricator.wikimedia.org/T406498#11366012 (10daniel) [12:02:40] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work: Test api rate limiting on production cluster - https://phabricator.wikimedia.org/T406498#11366066 (10daniel) 05Open→03Resolved This is done, @Clement_Goubert deployed API rate limiting for all users of all APIs today (in "shadow mode", so nothing i... [12:39:05] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11366251 (10brouberol) I'll probably wait until https://phabricator.wikimedia.org/T409800 is figured out first. [14:57:24] 06serviceops, 13Patch-For-Review, 07sre-alert-triage: Alert in need of triage: SystemdUnitFailed (instance registry1005:9100) - https://phabricator.wikimedia.org/T409817#11366931 (10Blake) 05In progress→03Resolved Merged and deployed - thanks @Clement_Goubert! [15:27:29] 06serviceops, 07Sustainability (Incident Followup): Expand upon Kask/Sessionstore documentation - https://phabricator.wikimedia.org/T320398#11367168 (10LSobanski) 05Open→03Resolved a:03LSobanski Here are the changes to the documentation that happened since the last comment: - https://www.mediawiki.or... [16:10:02] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: hw troubleshooting: host unresponsive for wikikube-worker2203.codfw.wmnet - https://phabricator.wikimedia.org/T408004#11367385 (10Jhancock.wm) 05Open→03Resolved @Raine ran into a secondary issue with the backplane, but it's fixed now. let us you know if y... [17:26:56] 06serviceops, 13Patch-For-Review: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245#11367783 (10Scott_French) Thanks to @ssingh and @MoritzMuehlenhoff for the reviews. Alas, the work described in T352245#11360536 could not happen today due to last-minute conflicts.... [19:20:41] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.3 production workload migration - https://phabricator.wikimedia.org/T405955#11368240 (10Scott_French) Between 18:40 and 19:00 UTC today, all remaining external API and web traffic served on mw-api-ext and mw-web, respectively, was migrated to PHP 8.3. This... [19:21:44] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.3 production workload migration - https://phabricator.wikimedia.org/T405955#11368241 (10Scott_French) [19:33:10] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.3 production workload migration - https://phabricator.wikimedia.org/T405955#11368272 (10Scott_French)