[06:42:34] <vgutierrez>	 Krenair: AES128-SHA TLS ciphersuite deprecation
[06:59:17] <vgutierrez>	 _joe_: so... I've been debugging the June 26th pybal/scap issue
[06:59:34] <_joe_>	 vgutierrez: oh? do tell
[06:59:52] <vgutierrez>	 _joe_: at some point.. something (I'm assuming scap) asked for a 100% depool via etcd
[07:00:09] <_joe_>	 yes
[07:00:21] <_joe_>	 that's known, see the incident report
[07:00:42] <vgutierrez>	 yup
[07:00:44] <vgutierrez>	 Jun 26 09:33:35 service: api_80: 0 enabled 46 disabled: 0% enabled
[07:00:53] <vgutierrez>	 that's the timestamp for api_80 for instance
[07:00:57] <vgutierrez>	 at least from pybal point of view
[07:01:12] <vgutierrez>	 and what I'm seeing right now on pybal code
[07:01:14] <_joe_>	 https://wikitech.wikimedia.org/wiki/Incident_documentation/20180626-LoadBalancers see here
[07:01:29] <vgutierrez>	 yup, I've read that :)
[07:01:39] <_joe_>	 the issue is that T184715 is not really resovled
[07:01:40] <stashbot>	 T184715: pybal's "can-depool" logic only takes downServers into account - https://phabricator.wikimedia.org/T184715
[07:02:00] <vgutierrez>	 pybal trusts the config enough to depool servers without going through canDepool() logic
[07:02:26] <_joe_>	 yeah, which was ok when using files
[07:02:30] <_joe_>	 now it's not
[07:02:35] <_joe_>	 so we wanted to change that
[07:02:39] <_joe_>	 but we apparently failed
[07:02:47] <vgutierrez>	 https://github.com/wikimedia/PyBal/blob/1.15-stretch/pybal/coordinator.py#L110
[07:03:08] <_joe_>	 I know
[07:12:11] <vgutierrez>	 _joe_: hmmm well... T184715 refers to depooling when a server is detected as down
[07:12:12] <stashbot>	 T184715: pybal's "can-depool" logic only takes downServers into account - https://phabricator.wikimedia.org/T184715
[07:12:23] <vgutierrez>	 _joe_: and that's properly done
[07:14:08] <_joe_>	 vgutierrez: well, in the discussion in the ticket https://phabricator.wikimedia.org/T184715#3896284
[07:14:30] <_joe_>	 but that's hard to solve in the current code, I know
[07:14:49] <vgutierrez>	 yey... your comment is changing the scope of the task
[07:15:10] <vgutierrez>	 "So when we want to depool a server, either logically or because of monitoring, we should first check that we have enough pooled servers to be able to depool the current machine." :)
[07:35:04] <vgutierrez>	 mark: I've two tests in pybal currently failing in master
[07:35:13] <vgutierrez>	 pybal.test.test_monitors.UDPMonitoringProtocolTestCase.testInit
[07:35:21] <vgutierrez>	 pybal.test.test_monitors.UDPMonitoringProtocolTestCase.testRun
[07:36:22] <vgutierrez>	 exceptions.AttributeError: 'UDPMonitoringProtocol' object has no attribute 'interval'
[07:36:27] <vgutierrez>	 exceptions.AttributeError: 'UDPMonitoringProtocol' object has no attribute 'loop'
[07:40:50] <vgutierrez>	 forget about it...
[07:40:54] <vgutierrez>	 rusty pyc files :/
[07:41:15] <vgutierrez>	 that actually makes sense 
[07:41:20] <ema>	 hey, I'm merging https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/440157/ with puppet disabled on text hosts 
[07:42:40] * vgutierrez hides
[07:42:45] <vgutierrez>	 ema: good luck <3
[07:46:04] <ema>	 !log install misc VCL on a text host (cp3030) T164609
[07:46:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:46:08] <stashbot>	 T164609: Merge cache_misc into cache_text functionally - https://phabricator.wikimedia.org/T164609
[07:48:07] <elukey>	 ema: does it mean that the misc merge to text happens today? (just to alert my team)
[07:48:16] <ema>	 elukey: nope
[07:48:33] <elukey>	 ah only the multiple VCL thing
[07:54:51] <ema>	 elukey: yes :)
[07:55:15] <ema>	 this is about installing misc-specific VCL files on text hosts and do auto-reloads in case they change
[07:56:04] <ema>	 VCL switching already does work and is tested: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/443573/2/modules/varnish/files/tests/text/22-cache_misc-vcl-switch.vtc
[07:56:12] <ema>	 we just don't do it yet 
[07:56:32] <elukey>	 nice
[07:57:49] <wikibugs>	 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Review analytics-in4 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10Gehel) For WDQS, we should keep access to at least 2 nodes in both eqiad and codfw. I propose:  wdqs1003: 10.64.0.14 wdqs1004: 10.64.0.17 wdqs2001: 10.192.32.1...
[07:58:55] <ema>	 !log install misc VCL on all text hosts T164609
[07:58:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:58:58] <stashbot>	 T164609: Merge cache_misc into cache_text functionally - https://phabricator.wikimedia.org/T164609
[08:10:04] <wikibugs>	 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Review analytics-in4 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10elukey) First batch of changes:  ``` delete firewall family inet filter analytics-in4 term logstash delete firewall family inet filter analytics-in4 term event...
[08:29:03] <moritzm>	 ema: 4.9.107~wmf1 works fine on multatuli, uploading to apt.wikimedia.org
[08:29:17] <moritzm>	 (but also needs an update of linux-meta as the ABI was bumped)
[08:29:45] <ema>	 moritzm: great, thanks!
[09:26:40] <moritzm>	 ema: meta packages now also upgraded, "apt-get -y install linux-meta will do the right thing", maybe start with a misc host, though
[09:53:46] * mark is nearby esams
[09:53:51] <mark>	 i will eat something and then head over
[09:55:30] <ema>	 nice, I'm about to have lunch too :)
[09:56:49] * mark grins
[09:56:56] <mark>	 i just found myself ordering food in english
[09:57:03] <moritzm>	 haha
[09:57:46] <vgutierrez>	 pretty common brain bug
[09:57:54] <vgutierrez>	 unable to write in one language and speak in other
[10:02:46] <mark>	 so i'm actually sitting at the beach
[10:03:16] <mark>	 https://www.google.com/maps/place/Het+Veerkwartier/@52.3877732,4.6722825,3a,75y,90t/data=!3m8!1e2!3m6!1sAF1QipPFfn4p9LB0Csl8QC2mMp0Llcrw2J22ozfiNIDd!2e10!3e12!6shttps:%2F%2Flh5.googleusercontent.com%2Fp%2FAF1QipPFfn4p9LB0Csl8QC2mMp0Llcrw2J22ozfiNIDd%3Dw114-h86-k-no!7i4032!8i3024!4m5!3m4!1s0x47c5e5853612c017:0x454ab64f9b0e37e!8m2!3d52.3875056!4d4.673433
[10:03:55] <mark>	 it's really the wrong weather to work in a hot data center, but oh well ;-)
[10:07:43] <ema>	 mark: cool place!
[10:07:59] <mark>	 yeah pretty good
[10:14:37] <volans>	 where's the sea?
[10:14:51] * volans associate beach with sea ;)
[10:16:12] <mark>	 actuallly not far either ;)
[10:17:51] <mark>	 so esams will definitely get flooded some day with rising sea levels hehe
[10:19:00] <jynus>	 isn't esams below see level?
[10:19:05] <mark>	 yes
[10:19:42] <volans>	 I was thinking the same ;)
[10:19:42] <jynus>	 well, you would get free cooling!
[10:19:57] <mark>	 and great interconnection
[10:26:43] <moritzm>	 Microsoft already does that: https://natick.research.microsoft.com/
[10:28:41] <mark>	 huh, our data center, evoswitch, apparently has been acquired by iron mountain
[10:28:43] <mark>	 i missed that
[10:30:04] <mark>	 not quite sure what is 'mountain' about this very flat place below sea level ;p
[10:31:28] <mark>	 s/esams/imams/ then
[10:31:34] <mark>	 :P
[10:32:02] <mark>	 i'm done with food, gonna pack up and head over
[10:51:23] <ema>	 _joe_: possibly related to the traffic anomalies in codfw, look at this IP on turnilo: https://bit.ly/2tUeyah
[10:56:36] <mark>	 ema: so I'd like to start cannibilizing cp3048
[10:56:42] <mark>	 we can probably get it back up again afterwards,
[10:56:48] <mark>	 but it won't be any more reliable than it already was ;)
[10:57:17] <mark>	 or was not, hehe
[10:58:35] <mark>	 hm
[10:58:43] <mark>	 actually I only need one of its SSDs I think...
[10:59:59] <ema>	 mark: alright, the host is depooled and the most recent hardware task is T190607
[10:59:59] <stashbot>	 T190607: cp3048 hardware issues - https://phabricator.wikimedia.org/T190607
[11:00:24] <ema>	 for the record :)
[11:00:38] <mark>	 yeah, i'm working off the ops-esams workboard, "next visit" column
[11:01:01] <mark>	 so cp3043 has a disk failure, I will steal an SSD from cp3048
[11:02:01] <mark>	 sdb...
[11:02:21] <ema>	 mark: sounds good, I'll depool cp3043 whenever you're ready to begin working on it
[11:02:32] <mark>	 go ahead
[11:03:01] <mark>	 I see cp3048 has varnish running too
[11:03:16] * mark stops it
[11:03:40] <ema>	 !log depool cp3043 (cache_upload) for hardware maintenance T179953 
[11:03:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:03:44] <stashbot>	 T179953: cp3043 disk failure - https://phabricator.wikimedia.org/T179953
[11:03:52] <jynus>	 can I downtime it on icinga?
[11:04:07] <ema>	 jynus: sure, thanks :)
[11:04:16] <jynus>	 43 or 48?
[11:04:40] <jynus>	 I guess 43?
[11:04:45] <ema>	 both
[11:04:52] <jynus>	 ok
[11:04:54] <jynus>	 doing
[11:05:50] <ema>	 you can see which hosts are pooled here https://config-master.wikimedia.org/pybal/esams/upload
[11:06:19] <jynus>	 Iv'e down them for 24 hours
[11:07:06] <mark>	 ok
[11:07:12] <mark>	 i'm going to pull sdb from cp3048 now
[11:08:01] <ema>	 ok
[11:09:35] <jynus>	 sorry if it still complained on irc, some errors were already soft when I downtimed them
[11:11:24] <mark>	 ema: could you shutdown processes on cp3043?
[11:11:29] <mark>	 some stuff is still accessing its cp3043
[11:11:32] <ema>	 mark: sure
[11:11:33] <mark>	 probably varnish, because it's out of the raid
[11:11:49] <mark>	 its sdb I mean
[11:13:05] <mark>	 lmk when done
[11:13:41] <ema>	 mark: varnishes stopped, we can also shutdown the host altogether if you prefer
[11:13:50] <mark>	 well
[11:13:53] <mark>	 i would prefer to rebuild the raid first
[11:14:02] <mark>	 not entirely sure if it wouldn't pick cp3048's otherwise ;)
[11:14:05] <mark>	 since that is the new drive
[11:14:10] <ema>	 k
[11:14:14] <mark>	 but after that we should reboot yes
[11:14:23] <mark>	 i will pull cp3043 sdb now
[11:14:28] <mark>	 and put the new drive in
[11:14:38] <ema>	 ok
[11:16:28] <mark>	 it detected the new one as sdc but I think that will correct after reboot
[11:17:48] <mark>	 ok, raid1 rebuilt, I will now reboot?
[11:17:55] <ema>	 go ahead
[11:18:07] <mark>	 damn yuviguard
[11:19:06] <ema>	 haha is that a new molly-guard?
[11:19:26] <ema>	 I'll ACK the IPSec alerts meanwhile
[11:19:51] <mark>	 ;)
[11:21:28] <ema>	 hey that was a fast reboot
[11:21:37] <mark>	 ok
[11:21:48] <mark>	 yeah
[11:21:55] <mark>	 everything looks good?
[11:22:04] <ema>	 looks good I see sda1 and sdb1 in the raid
[11:22:09] <mark>	 yes
[11:22:16] <mark>	 if varnish backend works then I suppose we're done
[11:22:38] <ema>	 varnish-be is up
[11:22:54] <mark>	 cool, then I will resolve the ticket
[11:23:05] <ema>	 \o/
[11:23:48] <vgutierrez>	 nice
[11:27:24] <mark>	 ok next
[11:27:25] <mark>	 cp3034
[11:27:32] <mark>	 memory error
[11:27:40] <mark>	 so yeah, I will probably have to steal cp3048's memory then
[11:27:44] <mark>	 just swap it altogether
[11:28:04] <mark>	 or I could just do the one dimm, says it lists it as B3...
[11:28:25] <mark>	 ema: could you depool/shutdown cp3034 entirely so I can open it up?
[11:28:49] <ema>	 mark: 1 sec
[11:28:59] <mark>	 and the same for cp3048 I guess ;)
[11:29:32] <ema>	 sure
[11:30:17] <ema>	 !log shutdown cp3048 and cp3034 (both already depooled) for hardware maintenance T190607 T189305
[11:30:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:30:22] <stashbot>	 T190607: cp3048 hardware issues - https://phabricator.wikimedia.org/T190607
[11:30:22] <stashbot>	 T189305: cp3034: Uncorrectable Memory Error - https://phabricator.wikimedia.org/T189305
[11:33:01] <ema>	 mark: done, cp3034 downtimed for 2h
[11:33:07] <mark>	 ok
[11:36:55] <mark>	 cp3034 DIMM b3 removed...
[11:37:00] <mark>	 will now pull the same one from cp3048
[11:37:22] <ema>	 ok
[11:44:01] <mark>	 swapped, powering on cp3034
[11:45:31] <mark>	 i don't see anything on the console yet...
[11:45:40] <ema>	 mmh
[11:47:06] <mark>	 i'll try a power cycle
[11:48:30] <ema>	 ok, I've canceled downtime for cp3043 which looks good 
[11:49:00] <mark>	 Message PR1: Replaced part detected for device: DDR4 DIMM(Socket B3).
[11:49:05] <mark>	 smart servers
[11:49:22] <mark>	 linux booting
[11:49:26] <ema>	 yay
[11:49:58] <mark>	 up, looking good
[11:50:09] <mark>	 i will put the bad dimm in cp3048
[11:50:14] <ema>	 k
[11:50:20] <mark>	 it may just work without issue after a reseat
[11:51:43] <mark>	 there are no other servers I need cp3048 parts for, right?
[11:51:46] <mark>	 cp3009 is older hardware...
[11:52:40] <ema>	 possibly bast3002?
[11:52:44] <mark>	 much older
[11:52:47] <ema>	 ok
[11:52:50] <mark>	 different drives also
[11:52:54] <mark>	 i will look at what I can do for it in a bit
[11:53:05] <mark>	 ok then I will connect cp3048 back up and we'll see how it does, but I wouldn't pool it ;p
[11:53:11] <mark>	 it's lacking a drive also hehe
[11:53:52] <ema>	 poor fella
[11:54:11] <ema>	 ok repooling cp3043
[11:54:44] <ema>	 !log repool cp3043 after hardware maintenance T179953
[11:54:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:54:47] <stashbot>	 T179953: cp3043 disk failure - https://phabricator.wikimedia.org/T179953
[11:57:55] <mark>	 UEFI0081: Memory configuration has changed from the last time the system was
[11:57:55] <mark>	 started.
[11:57:55] <mark>	 If the change is expected, no action is necessary. Otherwise, check the DIMM
[11:57:55] <mark>	 population inside the system and memory settings in System Setup.
[11:57:56] <mark>	 UEFI0081: Memory configuration has changed from the last time the system was
[11:57:56] <mark>	 started.
[11:57:58] <mark>	 If the change is expected, no action is necessary. Otherwise, check the DIMM
[11:58:00] <mark>	 population inside the system and memory settings in System Setup.
[11:58:05] <mark>	 that must have been why cp3034 didn't boot the first time, too
[11:59:23] <mark>	 cp3048 is booting but is missing its sdb
[11:59:29] <mark>	 it will continue after 90s
[12:01:41] <mark>	 went into emergency shell
[12:02:45] <ema>	 I wouldn't spend too much time on it, it's not that we're gonna pool it anyways in its sorry state
[12:03:07] <mark>	 exactly
[12:03:12] <mark>	 so I'll leave it in that state
[12:03:15] <ema>	 +1
[12:03:26] * mark checks the SEL
[12:03:43] <mark>	 -------------------------------------------------------------------------------
[12:03:43] <mark>	 Record:      108
[12:03:43] <mark>	 Date/Time:   07/04/2018 11:55:40
[12:03:43] <mark>	 Source:      system
[12:03:43] <mark>	 Severity:    Critical
[12:03:43] <mark>	 Description: The chassis is open while the power is off.
[12:03:46] <mark>	 sneaky buggers
[12:04:18] <mark>	 heh so
[12:04:23] <mark>	 it actually reports all sensors as good
[12:04:25] <mark>	 atm
[12:04:32] <mark>	 that may or may not change again
[12:04:50] <mark>	 so if we can get it a different SSD it /may/ be worth trying again, possibly with a cpu reseat/swap
[12:04:51] <mark>	 but maybe not now
[12:04:58] <mark>	 we can check on my next visit
[12:05:06] <mark>	 i left a comment on the task with what i've done to it
[12:05:27] <mark>	 next up are bast3002 and cp3009
[12:05:31] <mark>	 but I need a drink
[12:05:36] <mark>	 it's super hot and very dry here
[12:05:39] <mark>	 bbi 10
[12:05:47] <ema>	 enjoy
[12:14:48] <mark>	 back
[12:15:43] <mark>	 ok so bast3002
[12:15:50] <mark>	 it doesn't have hotswap drive, so i will need to shut it down
[12:15:55] <mark>	 and find a donating server as well
[12:17:52] <mark>	 ...and first I need to figure out which server is bast3002 as it has not been physically relabeled here ;)
[12:18:34] <mark>	 hooft... ok
[12:18:34] <ema>	 ha
[12:18:41] <mark>	 hooft was the original bastion
[12:19:01] * mark makes a ticket to relabel it
[12:21:39] <godog>	 we've come full circle!
[12:21:58] <vgutierrez>	 /o\
[12:22:03] <vgutierrez>	 \o/
[12:22:27] <mark>	 yeah
[12:22:27] <godog>	 hahaha lovely, thanks vgutierrez 
[12:22:29] <mark>	 so there's a bunch of similar boxes, unused
[12:22:34] <mark>	 I will steal a drive from amslvs3
[12:22:43] <mark>	 which is powered off already
[12:22:57] <mark>	 time to get new hardware in soon
[12:23:09] <mark>	 bast3002 downtime will mean prometheus downtime I think?
[12:23:22] <godog>	 correct yeah
[12:24:17] <godog>	 and a bunch of sre with higher latency ssh towards the cluster
[12:25:06] <vgutierrez>	 actually every SRE that's working today
[12:25:18] * volans not ;0
[12:25:19] <volans>	 ;)
[12:25:40] <vgutierrez>	 hahaha.. iron for you
[12:25:40] <volans>	 some of us are using iron 
[12:25:52] <volans>	 so higher latency by default :-P
[12:26:35] <mark>	 yes
[12:26:37] <mark>	 so i'll try to be quick
[12:26:43] <mark>	 if someone can shut down bast3002, then I can swap its sdb
[12:26:52] <volans>	 btw it's also athe install server, but I guess no reimages going on in esams today
[12:27:03] <mark>	 even if there were they would be out of luck :P
[12:27:23] <mark>	 i need to make my very scarce time here count ;)
[12:27:40] * volans downtiming bast3002 on icinga
[12:27:42] <mark>	 is anyone shutting bast3002 down?
[12:27:45] <mark>	 or I can
[12:28:15] <volans>	 godog: anything special to do?
[12:28:18] <volans>	 for prometheus
[12:28:40] <volans>	 downtimed already
[12:31:29] <godog>	 just shutting prometheus is fine yeah
[12:31:41] <godog>	 volans: ^
[12:31:55] <volans>	 godog: ack, mark I'm stopping prom. and shutting it down
[12:32:01] <mark>	 cool
[12:33:56] <volans>	 mark: going down...
[12:34:00] <mark>	 ok
[12:34:02] <mark>	 swapping drive...
[12:34:11] <volans>	 check the leds
[12:35:42] <ema>	 I've canceled downtime for cp3034 too, pooling it back shortly
[12:39:03] <mark>	 bast3002 powered back up
[12:39:08] <ema>	 !log cp3034 repooled after hw maintenance T189305
[12:39:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:39:11] <stashbot>	 T189305: cp3034: Uncorrectable Memory Error - https://phabricator.wikimedia.org/T189305
[12:39:29] <mark>	 i'm blind without console access now
[12:39:37] <mark>	 hopefully it just boots back up ;)
[12:39:43] <mark>	 or I will need to go find a console cart
[12:40:15] <volans>	 mark:  why no mgmt? I'm in
[12:40:32] <volans>	 is booting
[12:40:34] <mark>	 because to access the management network... we need to go via the bastion
[12:40:39] <mark>	 maybe it works via iron these days actually
[12:40:45] <volans>	 at login
[12:40:46] <mark>	 i'm still thinking of the old situation without full global connectivity heh
[12:40:56] <volans>	 and I can ssh into the host
[12:41:04] <mark>	 yeah nice
[12:41:14] <mark>	 so the drive sdb will need repartitioning and raid rebuild
[12:41:19] <mark>	 is one of you able to do that so I can focus on dc stuff?
[12:41:34] <mark>	 the drive is from amslvs3 which will have had very different partitions
[12:41:48] <volans>	 any of you do you have at hand the commands? I'm sure you've already done that on the other hosts
[12:42:08] <mark>	 i've done the raid rebuild today, not the repartitioning
[12:42:13] <mark>	 but it basically means playing around with parted
[12:42:19] <mark>	 and copying the partition table from sda to sdb
[12:42:31] <mark>	 after that, mdadm /dev/md0 --add /dev/sdb1 etc
[12:42:49] <godog>	 sure I can take that, I usually use sfdisk -d | sfdisk
[12:42:58] <mark>	 here's our swift man
[12:43:02] <mark>	 not afraid of a few partitions
[12:43:03] <mark>	 ;)
[12:43:08] <godog>	 lol, indeed
[12:43:08] <volans>	 thanks godog!
[12:43:10] <vgutierrez>	 hahahah
[12:43:22] <mark>	 godog eats drive labels for breakfast
[12:43:40] <mark>	 ty!
[12:43:42] <mark>	 so next up, cp3009
[12:43:48] <godog>	 gotta get those vitamins going in the morning
[12:43:50] <mark>	 which has been broken for nearly 2 years, heh
[12:43:53] <mark>	 bad memory
[12:44:05] <mark>	 we have a lot of decom'ed servers from the same hw batch, except most of them will have less memory
[12:44:10] <mark>	 so i'm not sure we actually have a parts donor there
[12:44:16] <mark>	 and apparently it was 1 out of 4 misc varnish servers
[12:44:26] <mark>	 ema: do you feel it's worth repairing cp3009, considering that?
[12:44:30] <mark>	 or is 3 servers enough? ;)
[12:45:26] <ema>	 mark: 3 is enough I think
[12:45:56] <mark>	 ok
[12:46:31] <ema>	 especially considering the ongoing misc<->text merging work
[12:46:34] <mark>	 ema: do you want to decom it then?
[12:46:35] <mark>	 yeah
[12:46:48] <mark>	 not sure what exactly needs doing for that
[12:47:32] <volans>	 godog: anything to do to restart prometheus? the process seems up but I'm still not seeing data from other esams hosts in grafana
[12:47:43] <ema>	 mark: so, the host is not in etcd
[12:48:46] <volans>	 mmmh some graphs have data, mybe I just need to wait few more minutes
[12:48:49] <godog>	 volans: should be unattended, perhaps in a minute or two
[12:48:50] <godog>	 yeah
[12:49:05] <volans>	 yeah seems so, I'll keep an eye, thanks
[12:49:12] <godog>	 I see the unknowns in icinga recovering too
[12:49:29] <godog>	       [>....................]  recovery =  1.0% (4485312/438449152) finish=292.5min speed=24722K/sec
[12:50:18] <volans>	 will take a while :D
[12:51:19] <godog>	 heheh indeed, so happy we're defaulting to ssds nowadays
[12:51:37] <volans>	 grafana seems ok, all graphs that I've checked have new datapoints
[12:51:50] <mark>	 ema: i don't really know what that means, may or may not have been fully decom'ed?
[12:51:55] <volans>	 ntp will take a while too but usually recovers by itself
[12:53:36] <ema>	 mark: partially decom'ed, I'll search and removing the remaining things
[12:53:48] <ema>	 *remove
[12:54:00] <mark>	 ty!
[12:54:02] <mark>	 ok
[12:54:14] <vgutierrez>	 volans: yup... sometimes needs an extra help with a manual restart fo the service though
[12:54:15] <mark>	 i am now going to fetch packages from evoswitch, they were a bit unhappy with me not coming by to pick stuff up ;p
[12:54:17] <vgutierrez>	 s/fo/of
[15:20:48] <mark>	 ema: could you clarify the status of some esams cp systems?
[15:20:54] <mark>	 i have a whole bunch listed for removal soon
[15:21:00] <mark>	 https://phabricator.wikimedia.org/T184063
[15:21:07] <mark>	 but looking at puppet site.pp, some are still in there
[15:21:17] <mark>	 node /^cp300[3-6]\.esams\.wmnet$/ {
[15:21:17] <mark>	     # ex-cache_maps, to be decommed
[15:21:17] <mark>	     role(spare::system)
[15:21:17] <mark>	 }
[15:21:26] <mark>	 so not in use, but not really decom'ed either
[15:21:30] <mark>	 can I disconnect them?
[15:21:45] <mark>	 and why is this one special:
[15:21:46] <mark>	 node 'cp3022.esams.wmnet' {
[15:21:46] <mark>	     include ::standard
[15:21:46] <mark>	 }
[15:23:48] <ema>	 so for the ex-cache_maps systems (cp300[3-6]) I see there's a decom task -> T167376
[15:23:49] <stashbot>	 T167376: Decommission cp300[3456] - https://phabricator.wikimedia.org/T167376
[15:24:45] <mark>	 would you be able to do all the things one can do online for those so I can focus on physical stuff now?
[15:25:16] <ema>	 yeah
[15:25:33] <mark>	 awesome
[15:25:38] <ema>	 as for why is cp3022 not role spare I'm not sure, but it's part of T130883
[15:25:38] <stashbot>	 T130883: decom cp3011-22 (12 machines) - https://phabricator.wikimedia.org/T130883
[15:25:58] <mark>	 ok that task is further ahead
[15:26:06] <mark>	 so I will proceed with removing cabling for cp3011-3022
[15:26:11] <mark>	 and will check back in with you after that
[15:27:23] <ema>	 ok!
[15:35:11] <mark>	 btw, what do you think is the timeline for folding cache misc into text?
[15:35:23] <mark>	 wondering if I should expect those remaining 3 misc servers here to also get recycled
[15:35:29] <mark>	 (no problem if not though, they can go in the next round :)
[15:35:52] <ema>	 surely no merge before the next round :)
[15:36:03] <mark>	 ok
[15:36:25] <mark>	 I -think- cp3003-3006 are identical hardware to those (including cp3009) btw, if we want more spares ;)
[15:39:43] <ema>	 so, cp3003-3006:
[15:39:58] <ema>	 1) icinga checks disabled
[15:40:11] <ema>	 2) puppet disabled
[15:40:29] <ema>	 3) puppet patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/443845/
[15:40:36] <ema>	 4) dns patch https://gerrit.wikimedia.org/r/#/c/operations/dns/+/443846/
[15:41:25] <ema>	 I can now proceed with merging the puppet change and powering down the hosts
[15:43:24] <mark>	 cool
[15:43:29] <mark>	 then i will remove their cabling
[15:43:34] <mark>	 and their drives will get destroyed like all others
[15:46:07] <ema>	 mark: alright, powering them off
[15:47:53] <mark>	 ty!
[15:48:24] <ema>	 mark: done! Do you know offhand how to disable the switch ports? 
[15:51:41] <mark>	 yeah I can do that
[15:51:57] <ema>	 ty
[15:52:10] <ema>	 !log cp300[3-6]: puppet node clean/deactivate T167376
[15:52:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:52:13] <stashbot>	 T167376: Decommission cp300[3456] - https://phabricator.wikimedia.org/T167376
[15:58:21] <mark>	 mark@csw2-esams# set interfaces interface-range disabled-ports member-range xe-5/0/0 to  xe-5/0/15
[16:02:10] <ema>	 prod dns entries removed
[16:02:47] <mark>	 cables removed
[16:03:20] <ema>	 I've prepared https://gerrit.wikimedia.org/r/#/c/operations/dns/+/443851/ to remove the mgmt entries too once we're done with everything else
[16:05:38] <ema>	 mark: anything else I can help with?
[16:06:20] <mark>	 I don't think so :)
[16:06:33] <ema>	 mark: cp3008 seems to be down
[16:06:38] <mark>	 checking
[16:07:03] <mark>	 fixed
[16:07:07] <mark>	 sorry about that :(
[16:08:30] <ema>	 mark: no big deal, luckily it's a misc host :)
[16:09:00] <mark>	 pybal would have caught it quickly I hope
[16:10:30] <ema>	 mark: it did, see https://grafana.wikimedia.org/dashboard/db/pybal?orgId=1&var-datasource=esams%20prometheus%2Fops&var-server=lvs3004&var-service=misc_weblb_443&from=1530718730844&to=1530720594489
[16:12:00] <ema>	 mark: I'm now going afk, but I'm at home. don't hesitate to call if you need anything
[16:14:57] <mark>	 thanks!
[16:15:06] <mark>	 i'll keep cleaning up a bit more here and then head home too
[16:15:11] <mark>	 getting hungry
[16:15:20] <mark>	 and before anything else, a large pile of hardware needs to go to the recycler...