[08:35:28] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Install (2) 960GB SSDs each in kafka-main10[06-10] - https://phabricator.wikimedia.org/T371422#10060634 (10JMeybohm) Correct. Anything that is at least as big as the ~900G of the four currently installed SSDs will be fine. Thanks! [09:03:56] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423#10060810 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kafka-main2010.codfw.wmnet with OS bu... [09:16:34] 06serviceops, 07Kubernetes: Remove deprecated cloudnative-pg charts from chart-museum - https://phabricator.wikimedia.org/T371667#10060828 (10JMeybohm) 05Open→03Resolved a:03JMeybohm >>! In T371667#10058085, @brouberol wrote: > Yes, that's perfect. Ok, done. Documentation is now available at https:/... [09:17:32] 06serviceops: kafka-main200[6789] and kafka-main2010 implementation tracking - https://phabricator.wikimedia.org/T363210#10060833 (10JMeybohm) 05Stalled→03In progress [09:55:00] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423#10060991 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kafka-main2010.codfw.wmnet with OS bullseye executed with error... [11:09:14] 06serviceops, 06Content-Transform-Team, 10WMDE-TechWish-Maintenance, 07Epic, and 2 others: Move Kartotherian to Kubernetes - https://phabricator.wikimedia.org/T216826#10061160 (10MSantos) In the past, baremetal OS upgrades for Maps required a full re-sync of the maps data {T299049}. Or at least that used t... [11:28:38] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423#10061225 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kafka-main2010.codfw.wmnet with OS bu... [11:59:29] 06serviceops, 07Kubernetes: Remove deprecated cloudnative-pg charts from chart-museum - https://phabricator.wikimedia.org/T371667#10061340 (10brouberol) Thank you! [12:18:44] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423#10061430 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kafka-main2010.codfw.wmnet with OS bullseye executed with error... [12:37:58] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423#10061457 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kafka-main2009.codfw.wmnet with OS bullseye [12:40:05] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423#10061463 (10JMeybohm) @Jhancock.wm could you please check kafka-main2010 again? After trying to re-image I now only see 5 disks in iDRAC. [13:25:51] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423#10061619 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kafka-main2009.codfw.wmnet with OS bullseye executed with error... [13:26:32] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423#10061621 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kafka-main2009.codfw.wmnet with OS bullseye [13:36:05] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423#10061680 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kafka-main2009.codfw.wmnet with OS bullseye executed with error... [13:38:24] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423#10061682 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kafka-main2009.codfw.wmnet with OS bullseye [14:09:09] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423#10061804 (10Jhancock.wm) I located the missing disk and reseated it. it's showing as having a size of 0.94 GB. Not sure if it's bad or needs to be reformatted. lmk and... [14:17:42] 06serviceops, 10conftool, 06Infrastructure-Foundations: Adding a new conftool schema type might require a manual `conftool-merge` invocation - https://phabricator.wikimedia.org/T372408#10061842 (10CDanis) [14:25:21] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423#10061858 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kafka-main2009.codfw.wmnet with OS bullseye executed with error... [14:27:35] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423#10061861 (10JMeybohm) >>! In T371423#10061804, @Jhancock.wm wrote: > I located the missing disk and reseated it. it's showing as having a size of 0.94 GB. Not sure if i... [14:36:23] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423#10061922 (10JMeybohm) Oh, wait. 0.9GB - I totally misread. That is obviously not okay :D Tried rescanning the drives without luck. I would assume it's broken. [18:34:24] 06serviceops, 10MediaWiki-Platform-Team (Radar): Some mw-api-int traffic is going cross-DC - https://phabricator.wikimedia.org/T368064#10062626 (10CDanis) [20:10:59] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: wikitech self-auth: Allow wikitech to use its own internal authentication - https://phabricator.wikimedia.org/T371588#10062856 (10bd808)