[11:40:04] <moritzm>	 hashar: the Jenkins regression I mentioned recently (which breaks Jenkins if running on a Java with a four digit version scheme): https://issues.jenkins.io/browse/JENKINS-64212 is said to be fixed in the latest  2.263.1 LTS;
[11:40:36] <moritzm>	 from https://issues.jenkins.io/browse/JENKINS-64212: Mark Waite added a comment - 3 days ago Included in weekly release Jenkins 2.268. Will be included in LTS release Jenkins 2.263.1.
[12:56:51] <hashar>	 moritzm: great, I guess we can catch up with Jenkins LTS version now :]
[12:57:04] <hashar>	  2.263.1  got released today, I was waiting for that LTS to get out to upgrade our
[12:57:18] <hashar>	 cause last time we ended upgrading to some weekly release
[12:57:22] <moritzm>	 shall I port 2.263.1 to apt.wikimedia.org?
[12:57:30] <moritzm>	 shall I import 2.263.1 to apt.wikimedia.org?
[12:58:33] <hashar>	 moritzm: yes please, and there are the magic reprepro commands at  https://wikitech.wikimedia.org/wiki/Jenkins#Get_the_package
[12:58:56] <hashar>	 using the magic  -C thirdparty/ci --restrict=jenkins  options
[13:00:02] <moritzm>	 k, doing that in a bit
[13:01:23] <hashar>	 the alternative would be to use a  thirdparty/jenkins component, but I can't remember why we are not doing that ;]
[13:33:24] <elukey>	 hello people, I am working with John (DCOps) to move servers between racks to free 10g spots, and the next inline would be conf1005 and conf1006
[13:33:37] <elukey>	 (I already pinged Alex)
[13:33:57] <elukey>	 the zookeeper part is easy, but I have some doubts about the etcd part
[13:34:09] <elukey>	 more specifically
[13:34:10] <elukey>	 profile::etcd::replication::dst_url: https://conf1005.eqiad.wmnet:2379
[13:34:19] <elukey>	 profile::pybal::config_host: conf1006.eqiad.wmnet
[13:35:08] <volans>	 elukey: just different rack or also row (is any IP changing?)
[13:37:21] <elukey>	 nono same rack
[13:37:24] <elukey>	 err row
[13:38:17] <volans>	 see also https://wikitech.wikimedia.org/wiki/Service_restarts#etcd
[13:38:20] <elukey>	 so profile::etcd::replication::active: true is active only for conf2002, so I guess conf1005 should be fine
[13:38:34] <elukey>	 thanks :)
[13:38:40] <elukey>	 ahahhah I wrote it
[13:38:45] <volans>	 wrotfl
[13:38:55] <volans>	 LMWTFY
[13:39:03] <elukey>	 I always say  that myself from the past was wiser
[13:40:24] <akosiaris>	 o/
[13:40:28] <elukey>	 o/
[13:40:47] <akosiaris>	 ok, so pybal will continue working if the node it is contacting is down
[13:40:58] <akosiaris>	 but there is a catch
[13:41:08] <akosiaris>	 if it is restarted we would be in for a surprise
[13:41:28] <elukey>	 do you mean while conf1006 is down?
[13:41:34] <akosiaris>	 yes
[13:42:07] <akosiaris>	 so, depending on the amount of time the node is going to be down for, we either a) just do it, b) switch pybal to say conf1004
[13:42:42] <akosiaris>	 the nodes are identical in nature, it should be just fine (FLW)
[13:43:16] <elukey>	 it should be around 10 mins probably
[13:43:31] <elukey>	 maybe we can switch pybal to 1004 just to be safe
[13:43:49] <akosiaris>	 you know what, let's just switch to 1004
[13:43:51] <elukey>	 for conf1005 we should be ok, it is not replicating (2002 is from 1004)
[13:43:55] <elukey>	 yeah
[13:44:00] <akosiaris>	 I anyway need to bundle in an LVS removal
[13:44:09] <akosiaris>	 so now I got to reasons to restart pybal for
[13:44:16] <akosiaris>	 when is this due?
[13:44:24] <akosiaris>	 due for*
[13:44:36] <volans>	 elukey: is the bit about the mirroring still valid in the sense that 1005 is special?
[13:44:41] <elukey>	 John it is in the DC now, we can skip entirely or say only do conf1005
[13:45:11] <elukey>	 volans: so in theory the profile::etcd::replication::active: false prevents the etcd mirror from running (codfw -> eqiad)
[13:45:47] <elukey>	 ah no wait okok I got it
[13:45:52] <elukey>	 it could be a target for 2002
[13:46:10] <elukey>	 etcdmirror-conftool-eqiad-wmnet.service on 2002 points to conf1004
[13:48:32] <elukey>	 ah akosiaris conf1004 is also profile::pybal::config_host: conf1004.eqiad.wmnet, no problem I think but just mentioning
[13:49:04] <elukey>	 volans: so 1005 doesn't run a mirror and it is not a target for one afaics
[13:49:13] <volans>	 great, thanks for checking
[13:49:34] <elukey>	 elukey@conf2002:~$ sudo netstat -tuap | grep conf1004
[13:49:35] <elukey>	 tcp6       0      0 conf2002.codfw.wm:52728 conf1004.eqiad.wmn:4001 ESTABLISHED 7141/python
[13:49:40] <elukey>	 just to double check
[13:50:07] <elukey>	 (and 7141 is etcd mirror)
[13:50:24] <elukey>	 so to summarize:
[13:50:30] <elukey>	 - conf1005 seems doable anytime
[13:50:48] <elukey>	 - conf1006 needs some work beforehand (to move pybal to another target)
[13:51:05] <elukey>	 akosiaris: --^ is it a fair statement?
[13:51:40] <akosiaris>	 ah pybal in esams is conf1006 and pybal in eqiad conf1004
[13:51:49] <elukey>	 yes
[13:52:10] <volans>	 the SRV records don't need any change right?
[13:52:14] <akosiaris>	 yeah conf1005 seems doable anyway
[13:52:17] <akosiaris>	 no they don't
[13:57:04] <elukey>	 ok so I can ping John and do it later on (conf1005)
[14:16:14] <elukey>	 as FYI in ~15/20 mins I'll start the procedure to shutdown conf1005
[14:17:12] <elukey>	 (from b4 to b3)
[14:18:29] <elukey>	 1005 is the zookeeper leader but it will not be an issue
[14:18:50] <elukey>	 and etcdctl cluster-health looks good
[14:33:43] <elukey>	 akosiaris: starting to shutdown conf1005 as FYI
[14:34:36] <elukey>	 zookeeper down (new leader elected), etcd down
[14:35:41] <elukey>	 and etcd cluster health sees 1004 and 1006 as up
[14:36:26] <elukey>	 for the moment all good
[14:37:30] <akosiaris>	 ok
[15:06:14] <effie>	 sigh too many channels
[15:06:28] <effie>	 elukey: I got some confctl failures
[15:06:35] <effie>	 given the other too are happy, is that normal ?
[15:06:38] <effie>	 two*
[15:09:16] <elukey>	 effie: we moved conf1005 to another rack, it should be ok now (all good for both etcd and zookeeper0
[15:10:14] <effie>	 elukey: I got the errors 10' ago and the cluster looked healthy 40' ago
[15:10:24] <effie>	 that's why I am wondering
[15:10:28] <effie>	 ok I am running again
[15:10:34] <elukey>	 effie: can you give me more details ? :)
[15:10:45] <effie>	 no
[15:10:46] <effie>	 :p
[15:11:00] <effie>	 so I run cumin on parsoid servers in eqiad
[15:11:04] <elukey>	 ahhahaha okok
[15:11:08] <effie>	 depool ; something; pool
[15:11:36] <effie>	 so the first 2 got stuck on pool
[15:11:54] <effie>	 I got
[15:11:55] <effie>	 <head><title>502 Bad Gateway</title></head>
[15:12:16] <effie>	 thinks seem to work now, you think it is worth digging a bit more?
[15:12:25] <effie>	 I can have a look at the cumin logs
[15:12:30] <effie>	 things*
[15:13:23] <effie>	 nah not worth it
[15:17:27] <elukey>	 all right lemme know if anything is weird :)
[15:17:38] <effie>	 thank you
[16:16:53] <jbond42>	 just in case puppet wasn't ...fun... enough i just found that someone created a haskell implmentation http://lpuppet.banquise.net/blog/2012/07/04/the-puppet-resources-application/
[16:18:51] <moritzm>	 *phew*, makes you appreciate Ruby even :-)
[16:19:21] <jbond42>	 :) yep
[16:25:17] <hashar>	 next step: port the puppet syntax to yaml
[16:47:46] <paravoid>	 jbond42: oh I remember looking at that back in the day
[16:49:05] <jbond42>	 paravoid:  no envy here ;)
[20:30:58] <mutante>	 added new docs for "how to get a simple LAMP stack on a cloud VPS" (not related to prod appserver or beta, for volunteers and others who used the old "simplelamp" class to just get a quick LAMP but that got deprecated:  https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_project#How_to_automatically_setup_a_simple_LAMP_server_on_a_cloud_VPS_instance