[08:49:28] <_joe_> cdanis: if you are deploying a mediawiki-config patch, that should go in a backport window, typically [08:49:48] <_joe_> those infra windows are actually guaranteed to be backports-free so we can change stuff for k8s [08:50:27] <_joe_> so your change would be more appropriate in a backport window, but I don't think that's so important tbh :) [09:56:24] quick heads up that I'll restart eventgate-main@codfw (needs to be restarted to pick new streams config) [09:56:32] cc gmodena [10:04:06] done [10:40:53] 06serviceops, 07Kubernetes, 13Patch-For-Review: Add 2 more nodes per DC to wikikube-staging - https://phabricator.wikimedia.org/T380043#10343181 (10ops-monitoring-bot) depool host kubernetes[1007-1008].eqiad.wmnet by jayme@cumin2002 with reason: None [10:42:04] 06serviceops, 07Kubernetes, 13Patch-For-Review: Add 2 more nodes per DC to wikikube-staging - https://phabricator.wikimedia.org/T380043#10343183 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin2002 depool for host kubernetes[1007-1008].eqiad.wmnet completed: - kuberne... [11:23:30] 06serviceops, 10Citoid, 06Editing-team, 10RESTBase Sunsetting, 06Traffic: Switchover plan from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10343341 (10MSantos) [11:31:04] 06serviceops, 13Patch-For-Review: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343369 (10Clement_Goubert) [11:31:06] 06serviceops, 13Patch-For-Review: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343371 (10Clement_Goubert) 05Open→03In progress [11:39:58] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: hw troubleshooting: Link down for wikikube-worker2140.codfw.wmnet - https://phabricator.wikimedia.org/T380265#10343404 (10Clement_Goubert) I have the same issue on `wikikube-worker2157.codfw.wmnet`, the interface in netbox is `eno12409np1` but it has no link, whe... [12:07:11] 06serviceops, 10VisualEditor, 10Parsoid (Tracking): Preemptively warm caches for Parsoid output - https://phabricator.wikimedia.org/T301371#10343517 (10MSantos) 05Open→03Resolved a:03MSantos @daniel as far as I can remember this work is complete, I'll be bold and close but please re-open it and let... [12:14:00] 06serviceops, 13Patch-For-Review: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343620 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2156.codfw.wmnet with OS bookworm [12:14:06] 06serviceops, 13Patch-For-Review: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343599 (10Clement_Goubert) `wikikube-worker2157.codfw.wmnet` has the same issue as {T380265} [12:14:08] 06serviceops, 13Patch-For-Review: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343622 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2158.codfw.wmnet with OS bookworm [12:16:08] 06serviceops, 13Patch-For-Review: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343627 (10Clement_Goubert) `wikikube-worker2159.codfw.wmnet` is in `C4` and blocked by management switch being down [12:16:50] 06serviceops, 13Patch-For-Review: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343629 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2161.codfw.wmnet with OS bookworm [12:17:20] 06serviceops, 13Patch-For-Review: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343630 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2160.codfw.wmnet with OS bookworm [12:17:53] 06serviceops, 13Patch-For-Review: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343632 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2162.codfw.wmnet with OS bookworm [12:18:30] 06serviceops, 13Patch-For-Review: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343634 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2163.codfw.wmnet with OS bookworm [12:19:07] 06serviceops, 13Patch-For-Review: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343635 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2164.codfw.wmnet with OS bookworm [12:19:39] 06serviceops, 13Patch-For-Review: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343645 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2165.codfw.wmnet with OS bookworm [12:21:13] 06serviceops, 13Patch-For-Review: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343660 (10Clement_Goubert) [12:55:42] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343778 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2156.codfw.wmnet with OS bookworm completed: - wikikube-worker2156 (**PASS**) - D... [12:56:33] 06serviceops, 10Citoid, 06Editing-team, 10RESTBase Sunsetting, 06Traffic: Switchover plan from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10343780 (10Mvolz) Once we do the switchover, Citoid endpoint: /api?format=&search= Becomes optional, as that's... [12:58:58] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343786 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2161.codfw.wmnet with OS bookworm completed: - wikikube-worker2161 (**PASS**) - D... [13:01:16] 06serviceops, 06Content-Transform-Team-WIP, 10CX-cxserver, 06Language and Product Localization, and 2 others: Decommission cxserver endpoints from RESTBase - https://phabricator.wikimedia.org/T372753#10343793 (10Nikerabbit) [13:02:36] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343796 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2158.codfw.wmnet with OS bookworm completed: - wikikube-worker2158 (**PASS**) - D... [13:05:45] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343797 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2163.codfw.wmnet with OS bookworm completed: - wikikube-worker2163 (**PASS**) - D... [13:10:33] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343803 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2165.codfw.wmnet with OS bookworm completed: - wikikube-worker2165 (**PASS**) - D... [13:12:05] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343809 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2162.codfw.wmnet with OS bookworm completed: - wikikube-worker2162 (**PASS**) - D... [13:12:23] 06serviceops, 06Content-Transform-Team-WIP, 10RESTBase, 10RESTBase Sunsetting, and 2 others: Wikifeeds increase on 500 errors after switchover to core page HTML - https://phabricator.wikimedia.org/T354657#10343816 (10MSantos) [13:13:17] 06serviceops, 06Content-Transform-Team-WIP, 10RESTBase, 10RESTBase Sunsetting, and 2 others: Wikifeeds increase on 500 errors after switchover to core page HTML - https://phabricator.wikimedia.org/T354657#10343806 (10MSantos) This doesn't seem to be related to the switchover of Wikifeeds or RESTBase Sunset... [13:16:17] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343820 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2164.codfw.wmnet with OS bookworm completed: - wikikube-worker2164 (**PASS**) - D... [13:18:38] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343824 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2160.codfw.wmnet with OS bookworm completed: - wikikube-worker2160 (**PASS**) - D... [13:25:06] 06serviceops, 07Kubernetes, 13Patch-For-Review: Add 2 more nodes per DC to wikikube-staging - https://phabricator.wikimedia.org/T380043#10343847 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin2002 from kubernetes1007 to kubestage1005 completed: - kubernetes1007 (**PASS**)... [13:26:08] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Update kubeconform schema and CI checks to new target Kubernetes version - https://phabricator.wikimedia.org/T379919#10343845 (10Jelto) adding our team tag for visibility [13:28:06] 06serviceops, 07Kubernetes, 13Patch-For-Review: Add 2 more nodes per DC to wikikube-staging - https://phabricator.wikimedia.org/T380043#10343853 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin2002 for host kubestage1005.eqiad.wmnet with OS bookworm [13:31:23] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Update Kubernetes clusters to >1.25 - https://phabricator.wikimedia.org/T341984#10343855 (10Jelto) adding our team tag for visibility [13:33:23] 06serviceops, 07Kubernetes, 13Patch-For-Review: Add 2 more nodes per DC to wikikube-staging - https://phabricator.wikimedia.org/T380043#10343858 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin2002 from kubernetes1008 to kubestage1006 completed: - kubernetes1008 (**PASS**)... [13:34:38] 06serviceops, 07Kubernetes, 13Patch-For-Review: Add 2 more nodes per DC to wikikube-staging - https://phabricator.wikimedia.org/T380043#10343865 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin2002 for host kubestage1006.eqiad.wmnet with OS bookworm [14:04:18] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343959 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2166.codfw.wmnet with OS bookworm [14:04:49] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2167.codfw.wmnet with OS bookworm [14:04:51] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343963 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2168.codfw.wmnet with OS bookworm [14:05:08] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343964 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2169.codfw.wmnet with OS bookworm [14:06:20] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10343979 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2170.codfw.wmnet with OS bookworm [14:06:57] 06serviceops, 07Kubernetes, 13Patch-For-Review: Add 2 more nodes per DC to wikikube-staging - https://phabricator.wikimedia.org/T380043#10343984 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin2002 for host kubestage1005.eqiad.wmnet with OS bookworm completed: - kubestage100... [14:11:22] 06serviceops, 07Kubernetes, 13Patch-For-Review: Add 2 more nodes per DC to wikikube-staging - https://phabricator.wikimedia.org/T380043#10343995 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin2002 for host kubestage1006.eqiad.wmnet with OS bookworm completed: - kubestage100... [14:12:23] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10344000 (10Clement_Goubert) [14:13:50] 06serviceops: Decommission parse20[01-20] - https://phabricator.wikimedia.org/T380473 (10Clement_Goubert) 03NEW [14:15:15] 06serviceops: Decommission parse20[01-20] - https://phabricator.wikimedia.org/T380473#10344020 (10Clement_Goubert) p:05Triage→03Low [14:21:16] 06serviceops, 13Patch-For-Review: Decommission kubernetes10[09-14] - https://phabricator.wikimedia.org/T380027#10344037 (10Clement_Goubert) [14:23:50] 06serviceops: wikikube-worker13[13-28] implementation tracking - https://phabricator.wikimedia.org/T380350#10344058 (10Clement_Goubert) [14:36:48] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: hw troubleshooting: Link down for wikikube-worker2140.codfw.wmnet - https://phabricator.wikimedia.org/T380265#10344103 (10Papaul) @Clement_Goubert for 2140 you can re-image nothing else needs to be done on our end. I also update 2157. Let me know if you have any... [14:37:14] l/16 [14:39:18] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10344109 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2140.codfw.wmnet with OS bookworm [14:46:21] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10344125 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2157.codfw.wmnet with OS bookworm [14:46:22] 06serviceops: kafka-main100[6789] and kafka-main1010 implementation tracking - https://phabricator.wikimedia.org/T363214#10344126 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=9f9d188a-551c-412a-8d68-ca67db96a150) set by jynus@cumin1002 for 7 days, 0:00:00 on 1 host(s) and their services wi... [14:47:51] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10344135 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2166.codfw.wmnet with OS bookworm completed: - wikikube-worker2166 (**PASS**) - D... [14:47:53] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10344136 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2140.codfw.wmnet with OS bookworm executed with errors: - wikikube-worker2140 (**FA... [14:49:27] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10344142 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2167.codfw.wmnet with OS bookworm completed: - wikikube-worker2167 (**PASS**) - D... [14:51:19] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10344148 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2170.codfw.wmnet with OS bookworm completed: - wikikube-worker2170 (**PASS**) - D... [14:53:29] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10344154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2168.codfw.wmnet with OS bookworm completed: - wikikube-worker2168 (**PASS**) - D... [14:54:00] 06serviceops, 06Data Products, 06Data-Platform-SRE, 10Dumps-Generation, and 3 others: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650#10344155 (10BTullis) We have now created a cephfs file system for dumps. ` btullis@cephosd1001:~$ sudo ceph... [14:56:48] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10344162 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2140.codfw.wmnet with OS bookworm [14:56:49] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10344163 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2169.codfw.wmnet with OS bookworm completed: - wikikube-worker2169 (**PASS**) - D... [14:59:02] 06serviceops, 06Data Products, 06Data-Platform-SRE, 10Dumps-Generation, and 3 others: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650#10344183 (10BTullis) Currently, they are both assigned the `hdd` CRUSH rule. ` btullis@cephosd1001:~$ sudo... [15:09:03] 06serviceops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 4 others: Reimage one of the wikikube-worker1240 to wikikube-worker1304 node in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10344210 (10VRiley-WMF) A 10G connection has been placed into port 25... [15:16:49] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10344257 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2140.codfw.wmnet with OS bookworm executed with errors: - wikikube-worker2140 (**FA... [15:16:52] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10344258 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2140.codfw.wmnet with OS bookworm [15:23:29] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10344306 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2140.codfw.wmnet with OS bookworm executed with errors: - wikikube-worker2140 (**FA... [15:24:05] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10344312 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2140.codfw.wmnet with OS bookworm [15:29:17] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 06MediaWiki-Platform-Team, and 3 others: Transition parsoidtest1001 to PHP 8.1 - https://phabricator.wikimedia.org/T380485 (10cscott) 03NEW [15:32:40] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Update Kubernetes clusters to >1.25 - https://phabricator.wikimedia.org/T341984#10344420 (10Jelto) [16:03:16] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10344636 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2140.codfw.wmnet with OS bookworm executed with errors: - wikikube-worker2140 (**FA... [16:04:23] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10344638 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2140.codfw.wmnet with OS bookworm [16:20:26] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10344712 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2157.codfw.wmnet with OS bookworm executed with errors: - wikikube-worker2157 (**FA... [16:21:21] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10344739 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2157.codfw.wmnet with OS bookworm [16:47:15] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10344961 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2140.codfw.wmnet with OS bookworm completed: - wikikube-worker2140 (**PASS**) - D... [17:03:40] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10345104 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2157.codfw.wmnet with OS bookworm completed: - wikikube-worker2157 (**PASS**) - R... [17:04:29] 06serviceops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 4 others: Reimage one of the wikikube-worker1240 to wikikube-worker1304 node in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10345111 (10cmooney) >>! In T379790#10344210, @VRiley-WMF wrote: > A 1... [18:44:36] 06serviceops, 13Patch-For-Review: Monitoring to surface "low-traffic" jobs isolation failure - https://phabricator.wikimedia.org/T378609#10345905 (10Scott_French) 05In progress→03Resolved The latency-impact alert is now live, and based on initial testing with short trigger duration, appears to work as... [18:47:26] 06serviceops, 10FlaggedRevs, 10WMF-JobQueue: Spike in JobQueue job backlog time (500ms -> 4-8 minutes) - https://phabricator.wikimedia.org/T378385#10345915 (10Scott_French) 05Open→03Resolved Monitoring for sustained latency impact on low-traffic jobs is now live. [19:13:05] 06serviceops, 06Content-Transform-Team-WIP, 10Push-Notification-Service, 06Wikipedia-Android-App-Backlog, 07Essential-Work: Timeout errors when making requests to Firebase for push notifications - https://phabricator.wikimedia.org/T379647#10346044 (10Dbrant) We deployed the [[ https://gerrit.wikimedia.or... [19:46:37] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 10MW-1.44-notes (1.44.0-wmf.4; 2024-11-19), and 2 others: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10346179 (10BPirkle) The redirect parameter f... [20:08:02] 06serviceops, 06DC-Ops, 10ops-eqiad: Degraded RAID on wikikube-worker1256 - https://phabricator.wikimedia.org/T379454#10346245 (10VRiley-WMF) This drive has been replaced. Please let us know if anything else is needed. [21:41:03] 06serviceops, 10WMF-JobQueue: Temporarily run more refreshLinks jobs on Commons - https://phabricator.wikimedia.org/T380544 (10LucasWerkmeister) 03NEW [21:43:55] 06serviceops, 10WMF-JobQueue: Temporarily run more refreshLinks jobs on Commons - https://phabricator.wikimedia.org/T380544#10346534 (10LucasWerkmeister) p:05Triage→03High Per IRC discussion, marking as High priority. @AntiCompositeNumber reports that this results in category changes being slow to propagat... [21:45:45] 06serviceops, 06Commons, 10WMF-JobQueue: Temporarily run more refreshLinks jobs on Commons - https://phabricator.wikimedia.org/T380544#10346554 (10AntiCompositeNumber) [21:51:17] 06serviceops, 06Commons, 10WMF-JobQueue: Temporarily run more refreshLinks jobs on Commons - https://phabricator.wikimedia.org/T380544#10346561 (10LucasWerkmeister) [22:03:44] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: hw troubleshooting: Link down for wikikube-worker2140.codfw.wmnet - https://phabricator.wikimedia.org/T380265#10346623 (10Papaul) 05Open→03Resolved This is fix [22:30:08] 06serviceops, 06Commons, 10WMF-JobQueue: Temporarily run more refreshLinks jobs on Commons - https://phabricator.wikimedia.org/T380544#10346771 (10LucasWerkmeister) > Currently, the number of links to [Template:SDC statement has value](https://commons.wikimedia.org/wiki/Template:SDC_statement_has_value) (as... [23:11:58] 06serviceops, 06Commons, 10WMF-JobQueue: Temporarily run more refreshLinks jobs on Commons - https://phabricator.wikimedia.org/T380544#10346887 (10Platonides) I have processed those editrequests, so at least when a page contains multiple license templates, it will only need to be reparsed once. [23:12:49] 06serviceops, 06Commons, 10WMF-JobQueue: Temporarily run more refreshLinks jobs on Commons - https://phabricator.wikimedia.org/T380544#10346893 (10LucasWerkmeister) [23:12:59] 06serviceops, 06Commons, 10WMF-JobQueue: Temporarily run more refreshLinks jobs on Commons - https://phabricator.wikimedia.org/T380544#10346894 (10LucasWerkmeister) That’s great, thank you! [23:35:39] 06serviceops, 06Commons, 10WMF-JobQueue: Temporarily run more refreshLinks jobs on Commons - https://phabricator.wikimedia.org/T380544#10346937 (10Scott_French) Indeed, it looks like the `refreshLinks_partitioner` rule is [[ https://grafana.wikimedia.org/goto/xT8E4FnNR?orgId=1 | easily keeping up ]] with the... [23:45:36] 06serviceops, 10MW-on-K8s: Functional replacement for importImages.php on Kubernetes - https://phabricator.wikimedia.org/T377497#10346951 (10Platonides) I am old enough to remember the days when only Special:Upload existed. The only way to upload any relatively large file was a server-side import. Nowadays, w...