[06:28:43] good morning [06:28:55] so the Zayo link between codfw and eqiad is still showing issues [06:29:44] in theory maintenance is over judging from the emails, they suggest to bounce the interface still down in case it is needed [06:33:42] ah no now I see the contact that Chris opened to them [06:33:55] (those emails are unreadable to me) [06:34:42] last update is still WIP on their side, ok [06:42:14] yep [06:44:30] XioNoX: hello :) How do somebody bounce an interface if needed? set it disabled then remove from disabled? [06:57:03] yep exactly [09:42:39] <_joe_> what happen when you have two softwares that speak https to each other and are both very good at keeping persistent connections to your LVS load-balancing? [09:42:48] <_joe_> the answer is: weird stuff [09:43:38] <_joe_> right now cpjobqueue is connecting to the jobqueue via envoy, which has persistent connections enabled, which means it's very easy to get to imbalances in load, as new connections are rare [09:44:08] <_joe_> so I have a server I pooled 6 minutes ago with full weight that still has 10 active connections (compared to the 50+ of the others) [09:49:03] <_joe_> also one gotta love the message ipvsadm gives you when you try to list a service that's not present. "Memory allocation problem" is not what you expect a program interacting with the kernel to tell you when you try to see something [10:47:13] hey, I need etcd >= 3.3, but I only see 3.2 for debian everywhere [10:47:40] I wonder if https://gerrit.wikimedia.org/r/admin/projects/operations/debs/etcd is the way to go, cc _joe_ [10:48:11] <_joe_> arturo: I'd start from the official debian package [10:48:19] <_joe_> that was created before it existed IIRC [10:48:26] I see [10:48:31] <_joe_> why do you need 3.3? [10:48:45] kubernetes 1.16 needs it [10:49:24] (apparently, per the release notes and per kubeadm info messages) [10:49:31] <_joe_> uh intersting [10:49:42] <_joe_> akosiaris: ^^ this might be interesting for us as well [10:50:10] <_joe_> arturo: so yeah, start from the current debian package and go from there [10:50:16] ack [10:50:22] will take a look [11:17:07] which k8s version do you run? [11:17:37] if you are running >= 1.16 perhaps you can confirm if this is just a soft dependency and I can ignore it :-) [11:28:04] ouch [11:28:14] I hope that's not a hard dependency [11:28:30] 3.3 is not even in buster-backports [11:29:04] FYI, I'm going to start ulsfo routers upgrades in ~15min [11:31:44] yeah, not sure how hard it is in practice, the Debian k8s package is at 1.18.2 and there's no bug about this kind of dependency on the etcd bug page either [11:58:42] is there a way to get cumin to select based in hieradata? [11:59:08] *based on [12:00:44] kormat: not directly [12:00:51] but you can based on puppet class parameters [12:01:13] like 'C:mariadb::config%replication_role = "standalone"' [12:01:31] (the double quotes are optional in this case) [12:03:09] mm, ok. i really need to make time to grok puppet first [12:12:38] s/class/class or define/ [15:06:06] Does pcc work for hosts in deployment-prep? [15:06:33] I'd assume not [15:11:11] hnowlan: yes it can [15:11:28] but you might need to sync the facts [15:11:42] see https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet-diffs#How_to_update_the_facts_for_cloud_VMs?_(e.g._INFO:_Unable_to_find_facts_for_host_util-abogott-stretch.testlabs.eqiad.wmflabs,_skipping) [15:12:12] I also think I recall hearing you can't mix prod and cloud hosts in the same pcc invocation [15:12:20] you just can't mix prod and cloud hosts in the same compile run, separately it works [15:12:28] heh [15:13:03] oh, nice! thanks [15:13:15] hnowlan: you can also specify a `Hosts: re: .*\.deployment-prep\.eqiad\.wmflabs` to test all the nodes with a rege [15:13:47] as to the cant run both at the same time that is definetly a limitation although i think its one that can probably be fixed but have not had a chance to take a proper look [15:14:52] jbond42: ah, cool [18:12:52] hi all i have a PS out to change to using hiera 5. at PS53 PCC now seems to be all good so would love to get some reviews. i have added a few people who i thought may be intrested or have worked on the backends before but would love feedback from anyone intrested https://gerrit.wikimedia.org/r/c/operations/puppet/+/566559 [18:14:50] make that PS54 noticed a debug file left over