[01:33:20] 06serviceops, 10MediaWiki-extensions-CentralAuth, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Migrate CentralAuth maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385866#10803010 (10Scott_French) I've posted a handful of patches to migrate the periodic jobs tracked... [07:42:42] o/ I'll be switching search traffic from eqiad to codfw (opensearch), I'll do some warmup but we could expect perturbation while opensearch@codfw gets fully hot [09:20:30] 06serviceops, 06Trust and Safety Product Team: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10803611 (10hnowlan) >>! In T388542#10802622, @Dreamy_Jazz wrote: >>>! In T388542#10799442, @hnowlan wrote: >> The `purgeExpiredBlocks.php` job has been migrated to k... [09:34:20] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 13Patch-For-Review: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10803640 (10hnowlan) listTaskCounts has been migrated to Kubernetes, and the task is exiting 0 which is a success as far as... [09:37:12] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 13Patch-For-Review: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10803645 (10hnowlan) [10:12:11] 06serviceops, 10MW-on-K8s: Create mw-cron dashboards - https://phabricator.wikimedia.org/T393680 (10jijiki) 03NEW [10:12:56] 06serviceops, 10MW-on-K8s: Create mw-cron dashboards - https://phabricator.wikimedia.org/T393680#10803725 (10jijiki) a:05Clement_Goubert→03jijiki [10:25:06] 06serviceops, 10envoy, 10Observability-Metrics, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q4): Revisit default envoy histogram buckets - https://phabricator.wikimedia.org/T391333#10803765 (10elukey) I met today with Filippo and this is the proposed plan: - Create a new envoy Docker image with... [10:29:59] 06serviceops, 10envoy, 10Observability-Metrics, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q4): Revisit default envoy histogram buckets - https://phabricator.wikimedia.org/T391333#10803768 (10elukey) Of course the above doesn't work, since we override envoy.yaml via the mesh config. This requir... [10:46:29] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 13Patch-For-Review: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10803817 (10Sgs) >>! In T385782#10803640, @hnowlan wrote: > listTaskCounts has been migrated to Kubernetes, and the task is... [11:23:04] 06serviceops, 06Data-Persistence, 10MW-on-K8s: Migrate ParserCachePurging jobs to mw-cron - https://phabricator.wikimedia.org/T385800#10803920 (10hnowlan) [12:08:01] 06serviceops, 06Data-Platform-SRE, 06Machine-Learning-Team, 10Prod-Kubernetes, 07Kubernetes: Update kserve to v0.13.0 on ML clusters - https://phabricator.wikimedia.org/T380722#10804075 (10isarantopoulos) [12:08:09] 06serviceops, 06Data-Platform-SRE, 06Machine-Learning-Team, 10Prod-Kubernetes, and 2 others: Update knative-serving+net-istio to v1.12.x on ML clusters - https://phabricator.wikimedia.org/T380723#10804074 (10isarantopoulos) [13:44:00] 06serviceops, 06Release-Engineering-Team, 07Essential-Work: train presync failed - https://phabricator.wikimedia.org/T387823#10804450 (10thcipriani) [13:45:23] 06serviceops, 06SRE, 07Essential-Work, 10Release-Engineering-Team (Radar), 05Train Deployments: MW script "eval.php" failing for "testcommonswiki" during train operations - https://phabricator.wikimedia.org/T379628#10804452 (10thcipriani) [13:55:03] 06serviceops, 10CampaignEvents, 06Campaigns-Product-Team, 10MW-on-K8s: Migrate CampaignEvents jobs to mw-cron - https://phabricator.wikimedia.org/T385867#10804539 (10kamila) [13:55:24] 06serviceops, 10CampaignEvents, 06Campaigns-Product-Team, 10MW-on-K8s: Migrate CampaignEvents jobs to mw-cron - https://phabricator.wikimedia.org/T385867#10804547 (10kamila) 05In progress→03Resolved All jobs are migrated and running successfully. [13:58:11] 06serviceops: httpbb bookworm support - https://phabricator.wikimedia.org/T393711 (10Volans) 03NEW [14:02:39] 06serviceops, 06Trust and Safety Product Team: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10804631 (10kamila) [14:05:17] 06serviceops, 10RESTBase, 10RESTBase Sunsetting, 06Traffic, 10Content-Transform-Team (Work In Progress): Block external traffic to RESTBase /page/data-parsoid endpoint and investigate internal usage - https://phabricator.wikimedia.org/T393557#10804651 (10MSantos) [14:05:29] 06serviceops, 10RESTBase, 10RESTBase Sunsetting, 06Traffic, and 2 others: Block external traffic to RESTBase /page/data-parsoid endpoint and investigate internal usage - https://phabricator.wikimedia.org/T393557#10804652 (10MSantos) [14:12:40] 06serviceops, 06Trust and Safety Product Team: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10804697 (10kamila) I attempted to migrate the two `wikimediaevents` jobs, and while the global one runs successfully, the per-wiki one logs ` The "WikimediaEvents"... [14:35:43] 06serviceops, 10MediaWiki-extensions-CentralAuth, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Migrate CentralAuth maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385866#10804870 (10Tgr) Yeah, those jobs don't presuppose anything about how often they are run. The e... [16:28:15] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install aux-k8s-worker100[6-9].eqiad.wmnet - https://phabricator.wikimedia.org/T393053#10805256 (10akosiaris) >>! In T393053#10782038, @RobH wrote: > Alex, > > We didn't get racking details on the ordering task T392715, so we need to get them from... [16:30:34] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install aux-k8s-worker200[6-9].codfw.wmnet - https://phabricator.wikimedia.org/T393054#10805262 (10akosiaris) a:05akosiaris→03None >>! In T393054#10782085, @RobH wrote: > Alex, > > We didn't get racking details on the ordering task T392714, so... [16:30:45] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install aux-k8s-worker100[6-9].eqiad.wmnet - https://phabricator.wikimedia.org/T393053#10805266 (10akosiaris) a:05akosiaris→03None [21:59:55] 06serviceops, 10MediaWiki-extensions-CentralAuth, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Migrate CentralAuth maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385866#10806110 (10Scott_French) Great, thanks for confirming, @tgr - I'll get started migrating these... [23:07:48] 06serviceops, 13Patch-For-Review: Turn down MediaWiki image builds for PHP 7.4 - https://phabricator.wikimedia.org/T391057#10806197 (10Scott_French)