[00:04:15] 10Traffic, 10DNS, 10Operations, 10Technical blog, and 2 others: Setup DNS to direct techblog.wikimedia.org to new Wordpress VIP hosting - https://phabricator.wikimedia.org/T246507 (10bd808) 05Open→03Resolved [08:58:21] 10netops, 10Operations: Configure conditional advertising in eqdfw and knams - https://phabricator.wikimedia.org/T236785 (10ayounsi) Mystery solved. The reason for the route to not be accepted was: > Inactive reason: Unusable path This was due to: `rib inet6.0 aggregate route 2620:0:862:ed1a::/64` Causing th... [09:09:44] 10netops, 10Operations: Configure conditional advertising in eqdfw and knams - https://phabricator.wikimedia.org/T236785 (10ayounsi) 05Open→03Resolved All done! [09:23:06] 10Traffic, 10MediaWiki-Debug-Logger, 10Operations, 10Developer Productivity: noc.wikimedia.org doesn't route to the docroot when WikimediaDebug browser extension is live - https://phabricator.wikimedia.org/T245552 (10ema) >>! In T245552#5895210, @Krinkle wrote: > This has regressed last month as well and w... [10:26:40] 10netops, 10Operations, 10Patch-For-Review: esams/knams: advertise 185.15.58.0/23 instead of 185.15.56.0/22 - https://phabricator.wikimedia.org/T207753 (10ayounsi) 05Open→03Resolved Got the OK during a previous meeting. Done and verified: ROAs are good /23 is advertised as expected WMCS /24s are reachable [10:40:22] 10netops, 10Operations, 10Wikimedia-Incident: Add linecard diversity to the router-to-router interconnect in codfw - https://phabricator.wikimedia.org/T248506 (10ayounsi) As data point, FPC0 got purchased on 2014 and FPC5 in 2013 so it's also time to replace them. [11:21:26] 10Traffic, 10netops, 10Operations: BGP: Investigate isolating codfw and eqiad - https://phabricator.wikimedia.org/T246721 (10ayounsi) Next steps are: * Check ROAs - DONE * Advertise `208.80.154.0/23` (compete with CF) and `2620:0:861::/48` from eqiad/eqord * Check that they are being advertised as expected (... [12:41:27] 10netops, 10Operations: Configure management-instance on routers with Junos > 17.3 - https://phabricator.wikimedia.org/T247073 (10ayounsi) [12:41:30] 10netops, 10Operations, 10Wikimedia-Incident: Juniper HA audit - https://phabricator.wikimedia.org/T191667 (10ayounsi) [13:06:38] 10netops, 10Operations, 10Wikimedia-Incident: Juniper HA audit - https://phabricator.wikimedia.org/T191667 (10ayounsi) [13:53:25] 10Traffic, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10Papaul) [13:58:21] 10Traffic, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` cp2027.codfw.wmnet ` The log can be found in `/var... [14:06:58] 10netops, 10Operations: roll out sensible flow-table-sizes to Juniper core routers with sampling enabled - https://phabricator.wikimedia.org/T248394 (10CDanis) Druid disk usage is not greatly increased, routers seem happy. Will reconfigure another router or two, and work on Homer-izing the change, today [14:30:53] 10Traffic, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2027.codfw.wmnet'] ` and were **ALL** successful. [14:31:45] 10Traffic, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` cp2028.codfw.wmnet ` The log can be found in `/var... [14:52:04] ema: he, ok. So we flip flopped between two broken setups. One where noc can't be debugged to one where noc can't be accessed while debugging. [14:52:27] I guess the problem now is that mw debug is missing the relevant Apache configuration [14:52:36] I didn't realize they were separate [14:56:02] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2028.codfw.wmnet'] ` and were **ALL** successful. [14:56:41] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` cp2029.codfw.wmnet ` The log... [15:01:21] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` cp2030.codfw.wmnet ` The log... [15:13:53] bblack: if ok for you (and you're around JIC anything unexpected happen), I'd go ahead and test the deploy-check on dns4002, to be precise I'd test: [15:13:59] cd /srv/authdns/git && utils/deploy-check.py -g /srv/git/netbox_dns_snippets --deploy [15:14:09] as it will be executed by the cookbook [15:16:09] volans: ack sounds ok [15:18:14] any specific way to check all is good? aprt the output of the script and checking the dns works afterwards? [15:19:55] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2029.codfw.wmnet'] ` and were **ALL** successful. [15:19:59] 10Traffic, 10Analytics, 10Analytics-Kanban, 10EventStreams, and 2 others: EventStreams drops the connection after 15 minutes, which makes it unreliable - https://phabricator.wikimedia.org/T242767 (10stjn) 05Open→03Resolved Frequent disconnects stopped after 25th March, 15:30 UTC, so yes. Thank you for... [15:25:14] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2030.codfw.wmnet'] ` and were **ALL** successful. [15:25:16] bblack: all good also: [15:25:16] No action needed, zones and config files unchanged [15:25:26] so truly noop [15:26:41] 10Traffic, 10Operations, 10SRE-tools, 10Goal, and 3 others: Automate generation of Management DNS records from Netbox - https://phabricator.wikimedia.org/T233183 (10Volans) In relation to https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/583109 I've just tested the command on a single host, this is t... [15:26:43] full output in https://phabricator.wikimedia.org/T233183#6002124 [15:28:57] if it looks good for you too I'll go ahead, merge the cookbook and run it [15:30:44] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` cp2031.codfw.wmnet ` The log... [15:32:24] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` cp2032.codfw.wmnet ` The log... [15:36:20] volans: yeah looks good, go ahead [15:37:19] ack, thanks (cc chaomodus ) [15:40:25] cool cool [15:42:23] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: BGP: Investigate isolating codfw and eqiad - https://phabricator.wikimedia.org/T246721 (10ayounsi) ` Prefix Nexthop MED Lclpref AS path * 2620:0:860::/46 Self I * 2620:0:861::/48 Sel... [15:50:10] ema, bblack https://phabricator.wikimedia.org/P10789 [15:51:26] looking good :) [15:53:23] * volans firing up the sre.dns.netbox, should be a noop in terms of gdnsd [15:53:51] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2031.codfw.wmnet'] ` and were **ALL** successful. [15:54:06] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` cp2033.codfw.wmnet ` The log... [15:54:09] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: BGP: Investigate isolating codfw and eqiad - https://phabricator.wikimedia.org/T246721 (10ayounsi) Both v4 and v6 are seen as expected in LG: https://stat.ripe.net/widget/looking-glass#w.resource=208.80.154.0/23 https://stat.ripe.net/widget/looking-gla... [15:58:30] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2032.codfw.wmnet'] ` and were **ALL** successful. [15:58:49] {done} [15:58:57] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` cp2034.codfw.wmnet ` The log... [15:59:03] volans: everything is on fire [16:00:26] vgutierrez: what? [16:00:36] volans: just kidding <3 [16:00:48] you owe me 2y of life [16:00:54] ahahah [16:01:13] 🍻 [16:01:27] well played! [16:01:28] :D [16:01:52] "due to an unfortunate bug in the netbox update script, coupled with some unfortunate software automation from valentin aimed at ncredir issues, your change has accidentally unregistered every domain we own and released them to the public" [16:02:06] looool [16:02:12] rotfl [16:02:58] I'm currently debating with myself if the cookbook should fetch the specific SHA1 instead of git pull in the netbox directory. Given that it had pushed a new HEAD just seconds before. I think I might add that to be extra safe [16:03:27] s/given that/although/ [16:05:07] yeah it's tricky I guess [16:05:25] but since you happen to know what you just pushed, being explicit isn't an awful idea [16:05:57] at the end of the day, a random authdns-update might pull mixed HEADs anyways, so it's not like there's any real defense on that. [16:06:06] it's eventual (but fairly rapid) consistency all the way down [16:06:32] yeah, I'll send the patch to be explicit, I already have the SHA1, so not a big deal [16:07:29] authdns-update pulling whatever the latest master tip is... seems fine design-wise in a world where netbox updates result in automatic dns updates too [16:07:54] it's a little more dicey if we're still cookbooking netbox updates manually because we don't trust things or need to carefully roll out transitional changes [16:07:59] which HEAD are you referring to now, the netbox repo or the manual repo? [16:08:04] the netbox repo [16:08:18] authdns-update is not pulling it at all [16:08:26] it uses what the hosts have [16:10:05] oh right, it has -g [16:10:09] hmmm [16:11:18] so, ready to try flipping on one of the includes? [16:11:41] volans: I'd like to witness a debate between you and riccardo [16:12:08] volans: cool! [16:13:13] ema: lol [16:13:18] well I mean i've been ready to CI one of the includes for a while, seems like we're just being fussy at this point [16:14:01] chaomodus: let's start with a PoP? maybe ulsfo? [16:14:15] that'd be godo probably in case of destruction [16:14:17] good* [16:14:24] is there an offline pop right now ? :) [16:14:29] nope [16:14:43] but really for the pure cache pops, the worst-case probably isn't that bad [16:15:03] right and we're like 89.75% sure nothing will go wrong [16:15:25] * volans too curious to know how you came up with that number :D [16:15:30] I'm 99.99% certain that there's only an 5% chance we blow up 100% of the things [16:15:51] 50% chance 100% of the time [16:15:59] etc. [16:16:14] you know like 90% of statistics people just throw around are made up [16:16:39] 90% of the time someone says that, they're just using it to justify poor quality statistics work! [16:17:19] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2033.codfw.wmnet'] ` and were **ALL** successful. [16:17:49] so, we still have the issue that zone_validator counts will go up, because mgmt moved and host ip didn't, right? [16:18:06] yes afaik [16:18:15] but i think the consensus was to ignore that and patch it out eventually [16:18:21] ok [16:18:22] and/or check it a different way [16:18:50] well I think it's reasonable to remove that check in the long run (from zone_validator anyways; you might want some equivalent on the generation side of the netbox stuff) [16:19:17] but so long as we're in the interim state of generating mgmt but not host IPs, it will be a blind spot about any mismatches [16:20:03] multiple checks are still WARNING only and don't make CI fail, so we're still blind in a lot of things :D [16:20:26] true :) [16:20:28] we shall have a report on missing mgmt interfaces also [16:20:29] but yeah agree [16:20:39] we could also alarm on missing fqdns for those interfaces [16:24:30] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2034.codfw.wmnet'] ` and were **ALL** successful. [16:25:15] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` cp2035.codfw.wmnet ` The log... [16:31:25] here we go: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/583676 [16:32:30] * volans be back in ~10 [16:34:51] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` cp2036.codfw.wmnet ` The log... [16:45:12] 10netops, 10Operations: IRR updates needed - https://phabricator.wikimedia.org/T235886 (10ayounsi) [16:45:18] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: BGP: Investigate isolating codfw and eqiad - https://phabricator.wikimedia.org/T246721 (10ayounsi) [16:45:26] 10netops, 10Operations: IRR updates needed - https://phabricator.wikimedia.org/T235886 (10ayounsi) [16:46:05] 10netops, 10Operations: esams/knams: advertise 185.15.58.0/23 instead of 185.15.56.0/22 - https://phabricator.wikimedia.org/T207753 (10ayounsi) [16:49:14] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2035.codfw.wmnet'] ` and were **ALL** successful. [16:49:22] 10netops, 10Operations: IRR updates needed - https://phabricator.wikimedia.org/T235886 (10ayounsi) p:05Low→03High a:03ayounsi [16:49:42] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: BGP: Investigate isolating codfw and eqiad - https://phabricator.wikimedia.org/T246721 (10ayounsi) This is all done. Last step is to update IRRs, tracked in T235886. [16:58:43] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2036.codfw.wmnet'] ` and were **ALL** successful. [17:07:21] 10netops, 10Operations: esams/knams: advertise 185.15.58.0/23 instead of 185.15.56.0/22 - https://phabricator.wikimedia.org/T207753 (10ayounsi) IRR objects created. [17:36:54] 10netops, 10Operations, 10ops-eqiad: Three ports on asw2-d-eqiad are not working as expected - https://phabricator.wikimedia.org/T247881 (10wiki_willy) a:03Cmjohnson [18:44:31] 10Traffic, 10Operations, 10Performance-Team: Production load.php spends ~ 10% time doing output compression within PHP - https://phabricator.wikimedia.org/T242478 (10Krinkle) Summary from {T247020}: * The fix for this task (T242478) made it so that handle compression at the edge again, instead of at the app... [18:47:50] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` cp2037.codfw.wmnet ` The log... [18:57:16] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` cp2038.codfw.wmnet ` The log... [18:57:40] 10netops, 10Operations: roll out sensible flow-table-sizes to Juniper core routers with sampling enabled - https://phabricator.wikimedia.org/T248394 (10CDanis) Oh, for posterity, it definitely worked -- here's bytes + packets reported by netflow with dst IP == any eqsin loadbalancer IP{: {F31703601} [19:10:43] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2037.codfw.wmnet'] ` and were **ALL** successful. [19:20:10] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2038.codfw.wmnet'] ` and were **ALL** successful. [19:20:17] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` cp2039.codfw.wmnet ` The log... [19:20:49] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` cp2040.codfw.wmnet ` The log... [19:44:12] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2039.codfw.wmnet'] ` and were **ALL** successful. [19:45:30] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2040.codfw.wmnet'] ` and were **ALL** successful. [21:11:17] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` cp2041.codfw.wmnet ` The log... [21:15:10] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` cp2042.codfw.wmnet ` The log... [21:16:52] 10netops, 10Operations, 10Patch-For-Review: roll out sensible flow-table-sizes to Juniper core routers with sampling enabled - https://phabricator.wikimedia.org/T248394 (10CDanis) [21:37:06] 10netops, 10Operations, 10Patch-For-Review: roll out sensible flow-table-sizes to Juniper core routers with sampling enabled - https://phabricator.wikimedia.org/T248394 (10CDanis) [21:37:11] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2041.codfw.wmnet'] ` and were **ALL** successful. [21:38:11] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10Papaul) [21:41:13] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2042.codfw.wmnet'] ` and were **ALL** successful. [21:47:07] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10Papaul) 05Open→03Resolved @BBlack servers are ready for service.