[06:12:57] ema: around? [07:21:06] 10Traffic, 10DNS, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install dns200[12].wikimedia.org - https://phabricator.wikimedia.org/T196493#4310383 (10ayounsi) [07:21:08] 10Traffic, 10netops, 10DNS, 10Operations, 10ops-codfw: switch port configuration for dns200[1-2] - https://phabricator.wikimedia.org/T197697#4310380 (10ayounsi) 05Open>03Resolved a:03ayounsi Switch ports enabled and in the public vlans. [07:23:36] nice [07:23:49] XioNoX, dns2001 and 2002 should be ready for installation, right? [07:25:00] I think so [07:25:14] I'll handle that later then :D [10:32:38] 10Traffic, 10DNS, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install dns200[12].wikimedia.org - https://phabricator.wikimedia.org/T196493#4258543 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts: ``` dns2001.wikimedia.org ``` The... [10:43:39] partman is missing... [10:43:42] * vgutierrez newbie [10:46:00] 10Traffic, 10DNS, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install dns200[12].wikimedia.org - https://phabricator.wikimedia.org/T196493#4311005 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['dns2001.wikimedia.org'] ``` Of which those **FAILED**: ``` ['dns2001.wikimedia.or... [10:59:23] 10Traffic, 10DNS, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install dns200[12].wikimedia.org - https://phabricator.wikimedia.org/T196493#4311071 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts: ``` dns2001.wikimedia.org ``` The... [11:20:43] 10Traffic, 10DNS, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install dns200[12].wikimedia.org - https://phabricator.wikimedia.org/T196493#4311178 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['dns2001.wikimedia.org'] ``` and were **ALL** successful. [11:23:09] 10Traffic, 10DNS, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install dns200[12].wikimedia.org - https://phabricator.wikimedia.org/T196493#4311186 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts: ``` dns2002.wikimedia.org ``` The... [11:23:53] 10Traffic, 10DNS, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install dns200[12].wikimedia.org - https://phabricator.wikimedia.org/T196493#4311189 (10Vgutierrez) [11:44:02] 10Traffic, 10DNS, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install dns200[12].wikimedia.org - https://phabricator.wikimedia.org/T196493#4311259 (10Vgutierrez) [11:44:26] 10Traffic, 10DNS, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install dns200[12].wikimedia.org - https://phabricator.wikimedia.org/T196493#4311260 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['dns2002.wikimedia.org'] ``` and were **ALL** successful. [12:41:27] 10Traffic, 10DNS, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install dns200[12].wikimedia.org - https://phabricator.wikimedia.org/T196493#4311555 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts: ``` dns2001.wikimedia.org ``` The... [12:54:04] 10Traffic, 10DNS, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install dns200[12].wikimedia.org - https://phabricator.wikimedia.org/T196493#4311589 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['dns2001.wikimedia.org'] ``` Of which those **FAILED**: ``` ['dns2001.wikimedia.or... [13:15:50] bblack: I need to merge this https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/441860/ to be able to run puppet on dns200[12] with the role set to recursor [13:17:45] bblack: I've to take something into account to not affect existing ntp servers? [13:29:03] 10Traffic, 10netops, 10Operations, 10ops-ulsfo: troubleshoot cr3/cr4 link - https://phabricator.wikimedia.org/T196030#4311726 (10ayounsi) I've been going back and forth with JTAC. The next physical change we need to try is a "loop test", for example connect cr3:et-0/0/1 to cr3:et-0/0/2 and see if the links... [13:31:47] 10Traffic, 10netops, 10Operations, 10ops-ulsfo, 10Patch-For-Review: Rack/cable/configure ulsfo MX204 - https://phabricator.wikimedia.org/T189552#4311756 (10ayounsi) [14:08:32] 10netops, 10Operations: Rack/setup cr2-eqdfw - https://phabricator.wikimedia.org/T196941#4311985 (10ayounsi) As the current MX80 uses XFP-10G-LR optics and the MX204 uses EX-SFP-10GE-LR we're going to need 5*EX-SFP-10GE-LR optics (+ at least one spare). @Papaul how many EX-SFP-10GE-LR optics do you have? I'll... [14:36:28] 10Traffic, 10DNS, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install dns200[12].wikimedia.org - https://phabricator.wikimedia.org/T196493#4312121 (10Papaul) a:05Papaul>03Vgutierrez [14:40:19] vgutierrez: should be ok. technically it will give some other servers too many peers for their other settings, but it will "work" for the duration to get them online and take out the others. [14:45:53] bblack: ack [15:25:38] bblack: does this https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/441891/ make sense to you? [15:36:31] _joe_: I could get your input as well, dunno if it's the best approach to replace a server on conftool [15:38:41] <_joe_> vgutierrez: it works, of course you will need to pool it afterwards [15:39:17] _joe_: should I depool achernar before? [15:39:23] or conftool is smart enough? [15:39:51] vgutierrez: I usually do those in two steps (add, then remove) [15:40:18] when you add new things, they'll start out in the depooled state. Then you can manage (via conftool) runtime pooling of the new and depooling of the old, then remove from the puppet data definition after it's depooled [15:40:28] ack [15:40:45] then it makes sense to add the new servers at once, and then remove the old ones [15:41:06] now that I looked at the rest, I would definitely hold off the nameservers changes too [15:41:29] on the loadbalancers you mean? [15:41:33] profile::base::nameservers or whatever. the purpose of that is to redefine resolv.conf entries across a bunch of client hosts. [15:41:44] right [15:41:46] get the new server sane and tested before puppeting that part of the change :) [15:41:55] (and pooled I guess) [15:42:00] ack [15:42:10] <_joe_> vgutierrez: conftool is smart enough [15:42:26] <_joe_> but do whatever you prefer [15:49:23] I'm going with bblack approach [15:49:32] for obvious reasons :) [16:02:25] dns2001 is getting some love already :) [16:02:47] https://grafana.wikimedia.org/dashboard/db/dns-recursors?orgId=1&var-datasource=codfw%20prometheus%2Fops&var-server=All&from=now-5m&to=now [16:05:37] and now dns2002 [16:18:23] 10Traffic, 10DNS, 10Operations, 10Patch-For-Review: rack/setup/install dns200[12].wikimedia.org - https://phabricator.wikimedia.org/T196493#4312463 (10RobH) [16:36:33] 10netops, 10Operations, 10ops-codfw: switch port configuration for graphite2003 - https://phabricator.wikimedia.org/T198119#4312529 (10Papaul) p:05Triage>03Normal [16:47:50] 10netops, 10Operations: Rack/setup cr2-eqdfw - https://phabricator.wikimedia.org/T196941#4312600 (10Papaul) @ayounsi I have none. I had 12 left but I used them to connect the lvs2009 and lvs2010 [16:58:15] bblack: all set... we just need to remove the old recdns from the loadbalancers and so on :) [16:59:18] and from smokeping [17:08:25] XioNoX: let me know what you think about https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/441919/ [17:09:46] vgutierrez: dns2001 is now in C5, and 2002 in D5 [17:09:46] D5: Ok so I hacked up ssh.py to use mozprocess - https://phabricator.wikimedia.org/D5 [17:09:59] ignore dat bot :) [17:26:25] I am adding more alarms for vk instances with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/441808/ - anything against it? [17:30:16] 10netops, 10Operations, 10ops-codfw: Swith port information for authdns2001 - https://phabricator.wikimedia.org/T198126#4312781 (10Papaul) p:05Triage>03Normal [18:44:47] 10Traffic, 10Operations, 10Performance-Team, 10Patch-For-Review, 10Wikimedia-Incident: Collect Backend-Timing in Prometheus - https://phabricator.wikimedia.org/T131894#4313027 (10Gilles) 05Open>03stalled Stuck in review since May [23:05:13] 10Traffic, 10Analytics, 10Operations: Size of headers processed by varnish? - https://phabricator.wikimedia.org/T198152#4313704 (10Nuria)