[01:07:37] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: Decommission mc-gp100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T381173#10378133 (10VRiley-WMF) [01:24:13] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: Decommission mc-gp100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T381173#10378138 (10VRiley-WMF) 05Open→03Resolved [02:18:43] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 3 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T381268#10378161 (10VRiley-WMF) [02:32:00] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 3 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T381268#10378169 (10VRiley-WMF) [02:32:17] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 3 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T381268#10378170 (10VRiley-WMF) 05Open→03Resolved [09:05:49] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10378579 (10ops-monitoring-bot) depool host mw[2440,2442-2444].codfw.wmnet by jayme@cumin2002 with reason: None [09:08:05] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10378584 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin2002 depool for host mw[2440,2442-2444].codfw.wmnet completed: - mw[2440,2442-2444]... [09:45:45] 06serviceops, 06SRE, 13Patch-For-Review: mw2420-mw2451 do have unnecessary raid controllers (configured) - https://phabricator.wikimedia.org/T358489#10378675 (10JMeybohm) [09:57:40] 06serviceops, 06Data Products, 06Data-Platform, 06Data-Platform-SRE, and 4 others: Generate a dumps-enabled mediawiki image - https://phabricator.wikimedia.org/T381473 (10Joe) 03NEW [10:03:44] 06serviceops: Handle expiring puppet certificates - https://phabricator.wikimedia.org/T381474 (10JMeybohm) 03NEW [10:05:03] 06serviceops: Handle expiring puppet certificates - https://phabricator.wikimedia.org/T381474#10378756 (10JMeybohm) [10:08:41] 06serviceops: Handle expiring puppet certificates - https://phabricator.wikimedia.org/T381474#10378760 (10MoritzMuehlenhoff) eventschema was moved to cfssl by @btullis in https://github.com/wikimedia/operations-puppet/commit/f9ce640346be05b8b00f37716f38dc09d43bbc87 [10:22:39] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10378800 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin2002 from mw2442 to wikikube-worker20160 completed: - mw2442 (**WARN*... [10:24:09] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10378805 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin2002 from mw2442 to wikikube-worker2016 completed: - mw2442 (**WARN**... [10:27:50] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10378810 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin2002 from mw2442 to wikikube-worker2016 completed: - mw2442 (**WARN**... [10:28:06] 06serviceops, 06Data Products, 06Data-Platform-SRE, 10Dumps-Generation, and 3 others: Migrate current-generation dumps to run on kubernetes - https://phabricator.wikimedia.org/T352650#10378803 (10Joe) [10:30:56] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10378817 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin2002 from mw2444 to wikikube-worker2172 completed: - mw2444 (**PASS**) - ✔️ Downtimed hos... [10:35:56] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10378837 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin2002 from mw2443 to wikikube-worker2171 completed: - mw2443 (**PASS**) - ✔️ Downtimed hos... [10:36:09] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10378838 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin2002 for host wikikube-worker2172.codfw.wmnet with OS bookworm [10:39:32] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10378841 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin2002 for host wikikube-worker2171.codfw.wmnet with OS bookworm [10:39:43] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10378851 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin2002 from mw2440 to wikikube-worker2015 completed: - mw2440 (**PASS**) - ✔️ Downtimed hos... [10:40:05] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10378878 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin2002 for host wikikube-worker2015.codfw.wmnet with OS bookworm [10:47:57] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10378894 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin2002 from mw2442 to wikikube-worker2016 completed: - mw2442 (**PASS**) - ✔️ Downtimed hos... [10:47:58] 06serviceops, 06DC-Ops, 10ops-codfw: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T381478 (10JMeybohm) 03NEW [10:48:37] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10378908 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin2002 for host wikikube-worker2016.codfw.wmnet with OS bookworm [10:49:38] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10378911 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin2002 for host wikikube-worker2016.codfw.wmnet with OS bookworm executed with errors: - wik... [10:52:02] 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, 10Wikidata, and 3 others: [SW] [WBQC] shellbox-constraints returning 500 on preg_match error - https://phabricator.wikimedia.org/T362084#10378918 (10ItamarWMDE) [10:54:14] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10378928 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin2002 for host wikikube-worker2016.codfw.wmnet with OS bookworm [11:36:53] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10379064 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin2002 for host wikikube-worker2171.codfw.wmnet with OS bookworm completed: - wikikube-worke... [11:38:28] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10379071 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin2002 for host wikikube-worker2172.codfw.wmnet with OS bookworm executed with errors: - wik... [11:39:59] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10379072 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin2002 for host wikikube-worker2172.codfw.wmnet with OS bookworm [12:03:46] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10379202 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin2002 for host wikikube-worker2016.codfw.wmnet with OS bookworm completed: - wikikube-worke... [12:22:28] 06serviceops, 06Content-Transform-Team, 10WMDE-TechWish-Maintenance, 07Epic, 10Maps (Kartotherian): Allow Kartotherian to use a local HTTP proxy - https://phabricator.wikimedia.org/T381257#10379284 (10elukey) I think I was wrong, this bit inside geoshape's index.js should work: ` // TODO: we might wan... [12:22:36] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10379285 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin2002 for host wikikube-worker2172.codfw.wmnet with OS bookworm completed: - wikikube-worke... [12:55:04] 06serviceops: kafka-main100[6789] and kafka-main1010 implementation tracking - https://phabricator.wikimedia.org/T363214#10379474 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=1e93a56a-8513-43f0-ac1d-4277501c6e39) set by jiji@cumin1002 for 1 day, 0:00:00 on 2 host(s) and their services with... [13:30:58] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10379634 (10ops-monitoring-bot) depool host kubernetes[1023-1024].eqiad.wmnet by jelto@cumin1002 with reason: Renaming nodes [13:32:09] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10379635 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host kubernetes[1... [13:35:26] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10379645 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin2002 for host wikikube-worker2015.codfw.wmnet with OS bookworm executed with errors: - wik... [13:35:52] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10379647 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin2002 for host wikikube-worker2015.codfw.wmnet with OS bookworm [13:41:21] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10379662 (10ops-monitoring-bot) pool host wikikube-worker[2016,2171-2172].codfw.wmnet by jayme@cumin2002 with reason: None [13:41:24] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10379663 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin2002 pool for host wikikube-worker[2016,2171-2172].codfw.wmnet completed: - wikikub... [13:49:41] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10379724 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes1023 to wikikube-work... [13:50:15] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10379728 (10ops-monitoring-bot) depool host mw[2445-2447].codfw.wmnet by jayme@cumin2002 with reason: None [13:52:02] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10379733 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin2002 depool for host mw[2445-2447].codfw.wmnet completed: - mw[2445-2447].codfw.wmn... [13:56:36] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10379745 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes1024 to wikikube-work... [14:07:38] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10379801 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker1036.eq... [14:19:54] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10379856 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin2002 for host wikikube-worker2015.codfw.wmnet with OS bookworm completed: - wikikube-worke... [14:22:27] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 10MW-1.44-notes (1.44.0-wmf.4; 2024-11-19), and 3 others: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10379872 (10akosiaris) Change merged and depl... [14:25:17] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 10MW-1.44-notes (1.44.0-wmf.4; 2024-11-19), and 3 others: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10379881 (10akosiaris) Moving to `stalled` on... [14:46:46] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10379957 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker1036.eqiad.wmnet with OS bookworm... [14:47:27] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10379960 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker1037.eqiad.wmnet with OS book... [14:55:07] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10379976 (10ops-monitoring-bot) pool host wikikube-worker2015.codfw.wmnet by jayme@cumin2002 with reason: None [14:55:08] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10379977 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin2002 pool for host wikikube-worker2015.codfw.wmnet completed: - wikikube-worker2015... [15:21:18] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10380105 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin2002 from mw2445 to wikikube-worker2173 completed: - mw2445 (**PASS**... [15:22:55] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10380110 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin2002 from mw2446 to wikikube-worker2174 completed: - mw2446 (**PASS**... [15:23:24] 06serviceops, 06SRE, 13Patch-For-Review: mw2420-mw2451 do have unnecessary raid controllers (configured) - https://phabricator.wikimedia.org/T358489#10380111 (10JMeybohm) [15:24:08] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T381478#10380114 (10JMeybohm) [15:25:31] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10380118 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin2002 from mw2447 to wikikube-worker2175 completed: - mw2447 (**PASS**... [15:26:32] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10380124 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin2002 for host wikikube-worker2173.codfw.wmnet with OS bookworm [15:26:58] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 06MW-Interfaces-Team, and 3 others: Testing and verification of MediaWiki on PHP 8.1 in mwdebug-next - https://phabricator.wikimedia.org/T379986#10380121 (10MSantos) a:03MSantos Since I'm doing the coordination for this work I'm assigning i... [15:27:01] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10380126 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker1037.eqiad.wmnet with OS bookworm... [15:27:53] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10380128 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin2002 for host wikikube-worker2174.codfw.wmnet with OS bookworm [15:28:39] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10380130 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin2002 for host wikikube-worker2175.codfw.wmnet with OS bookworm [15:41:39] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10380204 (10ops-monitoring-bot) pool host wikikube-worker[1036-1037].eqiad.wmnet by jelto@cumin1002 with reason: None [15:41:42] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10380205 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker[1036-1037].eqiad.wmn... [15:47:46] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 2 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T381504 (10Jelto) 03NEW [16:14:00] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10380306 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin2002 for host wikikube-worker2173.codfw.wmnet with OS bookworm completed: - wikikube-worke... [16:14:21] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T381478#10380295 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [16:21:42] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10380328 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin2002 for host wikikube-worker2174.codfw.wmnet with OS bookworm completed: - wikikube-worke... [16:33:04] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10380369 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin2002 for host wikikube-worker2175.codfw.wmnet with OS bookworm completed: - wikikube-worke... [16:35:25] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10380376 (10ops-monitoring-bot) pool host wikikube-worker[2173-2175].codfw.wmnet by jayme@cumin2002 with reason: None [16:35:35] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10380378 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin2002 pool for host wikikube-worker[2173-2175].codfw.wmnet completed: - wikikube-wor... [16:53:57] 06serviceops: kafka-main100[6789] and kafka-main1010 implementation tracking - https://phabricator.wikimedia.org/T363214#10380429 (10jijiki) [17:43:48] 06serviceops, 06Data Products, 06Data-Platform, 06Data-Platform-SRE, and 4 others: Generate a dumps-enabled mediawiki image - https://phabricator.wikimedia.org/T381473#10380690 (10BTullis) Thanks for creating this ticket. I noticed a few other things that our image will need, based on: [[https://github.co... [18:11:57] can I trouble someone for a small deployment-charts/modules/app review? [18:15:53] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1100515 [18:38:16] hiii [18:47:24] 06serviceops, 06Data Products, 06Data-Platform, 06Data-Platform-SRE, and 4 others: Generate a dumps-enabled mediawiki image - https://phabricator.wikimedia.org/T381473#10380942 (10hashar) You can retrieve a tarball directly from Gerrit using eg: Direct commit: https://gerrit.wikimedia.org/g/operations/dum... [19:19:25] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381089 (10ops-monitoring-bot) depool host mw[1491-1496].eqiad.wmnet by kamila@cumin1002 with reason: Renaming nodes [19:23:03] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381092 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 depool for host mw[1491-149... [19:28:31] ✅ [19:29:05] thanks! [19:30:38] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381102 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1491 to wikikube-worker1038... [19:35:06] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381121 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1492 to wikikube-worker1039 completed: - mw1492 (*... [19:39:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381135 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1493 to wikikube-worker1040 completed: - mw1493 (*... [19:44:17] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381148 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1494 to wikikube-worker1041 completed: - mw1494 (*... [19:51:13] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381168 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1495 to wikikube-worker1042 completed: - mw1495 (*... [19:55:01] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381174 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1496 to wikikube-worker1043 completed: - mw1496 (*... [20:09:48] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381232 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1038.eqiad.wmnet with OS boo... [20:10:29] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381235 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1039.eqiad.wmnet with OS boo... [20:25:00] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 10MW-1.44-notes (1.44.0-wmf.4; 2024-11-19), and 2 others: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10381290 (10HCoplin-WMF) 05Open→03In progr... [20:27:44] hi folks. I see a bunch of failures on lvs1020 to wikikube-worker1319 and some mw* hosts [20:27:47] is that known? [20:30:30] ah my bad, I see it in SAL. sorry about that [20:32:54] sukhe: I'm renaming them [20:33:10] yep just saw it in SAL [20:33:23] is it in the way? [20:33:48] basically Dec 04 20:33:10 lvs2013 pybal[1590965]: [shellbox-syntaxhighlight_4014 IdleConnection] WARN: wikikube-worker2136.codfw.wmnet (enabled/down/not pooled): Connection to 10.192.11.13:4014 failed. [20:34:00] and similarly for others [20:34:30] uh codfw isn't me [20:36:04] it seems like a network issue, we are looking at in -traffic [20:36:29] I'm only touching eqiad right now [20:36:44] yeah, not related to you most likely. we can't seem to ping other hosts from that lvs [20:36:52] (for sure not related to you) [20:37:07] ok [20:37:10] thanks [20:47:54] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381348 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1038.eqiad.wmnet with OS bookwor... [20:51:32] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381356 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1039.eqiad.wmnet with OS bookwor... [21:06:27] 06serviceops, 06Content-Transform-Team-WIP, 10Push-Notification-Service, 06Wikipedia-Android-App-Backlog, 07Essential-Work: Timeout errors when making requests to Firebase for push notifications - https://phabricator.wikimedia.org/T379647#10381410 (10Dbrant) Can confirm that this is now resolved. Many th... [21:18:16] 06serviceops, 06Content-Transform-Team-WIP, 10Push-Notification-Service, 06Wikipedia-Android-App-Backlog, 07Essential-Work: Timeout errors when making requests to Firebase for push notifications - https://phabricator.wikimedia.org/T379647#10381504 (10CDanis) 05Open→03Resolved a:03CDanis [22:13:40] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381667 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1040.eqiad.wmnet with OS boo... [22:13:53] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381668 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1041.eqiad.wmnet with OS boo... [22:34:33] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381693 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1042.eqiad.wmnet with OS boo... [22:34:55] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381695 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1043.eqiad.wmnet with OS boo... [22:51:39] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381722 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1040.eqiad.wmnet with OS bookwor... [22:55:04] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381727 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1041.eqiad.wmnet with OS bookwor... [23:13:56] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381788 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1042.eqiad.wmnet with OS bookwor... [23:19:21] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10381804 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1043.eqiad.wmnet with OS bookwor...