[04:51:42] We are going to take over puppet and mediawiki deployment for the s4 failover, if you need to deploy please coordinate with us. I will communicate once it is all done and deployments can happen normally again [05:11:18] Failover was done, mediawiki and puppet deployments can happen as usual [05:29:42] $someday we need a way to make that doable by script (lock out other puppet deployments, notice to users on attempts to puppet-merge, etc) [05:29:54] plus maybe even script the irc channel announcements [05:30:01] and a log entry [05:30:21] congrats on the quick failover, sounds like it went without a hitch [06:07:28] thanks [06:07:45] it will be much faster once we have dbctl in place too [06:08:21] ooohhh [06:53:46] apergos: scap has a lock you can set for all deployers :) [06:54:06] that gets the scap side, indeed. I'm thinking of the puppet side though [06:59:29] ah yes, missed your parenthetical [11:27:00] hey _joe_ [11:27:28] I see this in `modules/profile/manifests/etcd.pp` [11:27:33] https://www.irccloud.com/pastebin/NoLZaKJZ/ [11:28:35] which end up being something like `$peer_url = "http://${host}:${peer_port}" # Peer TLS is currently broken?` in `modules/etcd/manifests/init.pp` [11:29:15] but but according to https://etcd.io/docs/v3.3.12/op-guide/configuration/#listen-peer-urls, using a FQDN is invalid, and we should use an IP address [11:29:45] in fact, I see this error in my testing environment: `etcd[16341]: error verifying flags, expected IP in URL for binding (http://toolsbeta-arturo-k8s-etcd-1.toolsbeta.eqiad.wmflabs:2380). See 'etcd --help'.` [11:32:25] <_joe_> arturo: I think we're running etcd 3.2 which supports that [11:32:53] I've got the error msg in `3.2.16-1` [11:33:00] <_joe_> arturo: you should use etcd::v3 as a class [11:33:39] <_joe_> arturo: it's running like that on conf1004-6 [11:34:11] <_joe_> but hah, we use the SRV record for discovery [11:34:26] <_joe_> so /that/ might be broken [11:34:43] <_joe_> but then I'm not sure the cert will work as expected, because puppet :/ [11:35:23] my current approach is to have `profile::toolforge::k8s::etcd` as a wrapper of `profile::etcd`. Would you recommend using directly the base etcd classes instead of wrapping the profile? [11:36:53] (this is in the early stages of development, so we have margin to rewrite as we see fit= [11:43:23] ping _joe_ [11:43:56] <_joe_> arturo: I think that's a bug in the base classes. I'll have to dig a bit [11:44:01] <_joe_> care to open a task? [11:44:14] sure [11:50:44] here you go https://phabricator.wikimedia.org/T226095 [11:51:34] <_joe_> thanks, I'll try to look into it this week. I'm pretty sure this will uncover some further limitations in our approach [11:51:34] I could even try sending a tentative fix patch [11:51:43] <_joe_> sure [11:52:45] the other question I'm interested now is which approach you recommend: [11:52:45] 1) profile toolforge --> profile etcd --> base etcd [11:52:45] 2) profile toolforge --> base etcd [11:58:33] my initial approach was 1), but I'm now leaning towards 2), because I don't think we need some of the stuff in profile::etcd [12:47:54] <_joe_> that makes sense, yes [12:48:00] <_joe_> 2) I mean [12:48:08] <_joe_> unless you have *lots* of duplication [12:48:31] also, I'm looking at v3 right now, so it will be `profile toolforge --> etcd v3` directly