[00:00:26] <Krenair>	 root@deployment-acme-chief04:/var/lib/puppet# ssh-keygen -lf /etc/ssh/userkeys/acme-chief 
[00:00:26] <Krenair>	 256 SHA256:pf9LDsLA+r2noYae+qmRv6uQL3kB6GGo/gU3BIqs8TA root@deployment-puppetmaster03 (ED25519)
[00:00:34] <Krenair>	 so the key matches
[00:00:46] <Krenair>	 "sign_and_send_pubkey: signing failed: agent refused operation"
[00:02:07] <Krenair>	 doesn't seem to matter whether I try to SSH out of root or acme-chief, `SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh acme-chief@deployment-acme-chief04` shows this
[00:05:29] <volans>	 Krenair: sudo keyholder status
[00:05:39] <volans>	 and restart the proxy one (if it's still there)
[00:05:40] <Krenair>	 root@deployment-acme-chief03:~# keyholder status
[00:05:41] <Krenair>	 keyholder-agent: active
[00:05:41] <Krenair>	 - 256 SHA256:pf9LDsLA+r2noYae+qmRv6uQL3kB6GGo/gU3BIqs8TA root@deployment-puppetmaster03 (ED25519)
[00:05:41] <Krenair>	 keyholder-proxy: active
[00:05:41] <Krenair>	 - 256 SHA256:pf9LDsLA+r2noYae+qmRv6uQL3kB6GGo/gU3BIqs8TA root@deployment-puppetmaster03 (ED25519)
[00:05:45] <Krenair>	 tried disarm/arm
[00:06:08] <Krenair>	 you think I should restart the keyholder-proxy service volans ?
[00:06:14] <volans>	 yep
[00:06:33] <volans>	 on first installation by puppet it's needed
[00:06:49] <volans>	 and I bet also in other occasions
[00:07:00] <Krenair>	 fun
[00:07:02] <Krenair>	 that did the trick
[00:07:06] <Krenair>	 thanks for that volans 
[00:07:17] <volans>	 yw :)
[00:08:16] <Krenair>	 now I have a new problem in the form of labs instance access restrictions, but at least I've dealt with that before
[01:32:40] <wikibugs>	 10Acme-chief, 10Beta-Cluster-Infrastructure: Write designate integration script for certcentral DNS challenges - https://phabricator.wikimedia.org/T206922 (10Krenair) I finally got around to writing the basic script today (copy at deployment-acme-chief03:/usr/local/bin/acme-chief-designate-sync.py) and it pret...
[02:02:00] * Krenair now has to remember how to package this
[02:47:26] * Krenair grumbles
[02:47:36] <Krenair>	 it's building *something* but not the version I want to test
[02:52:06] <Krenair>	 there we go, my debian branch was missing some stuff
[07:59:54] <wikibugs>	 10Traffic, 10Operations, 10Wikidata, 10Wikidata-Campsite, and 2 others: nuke_limit often reached on esams varnish frontends - https://phabricator.wikimedia.org/T216006 (10Addshore) 05Open→03Resolved a:03ema
[13:46:30] <vgutierrez>	 Krenair: we had a little bit of a mess this morning, glad to see that the API CR worked for you as well :D
[13:46:52] <Krenair>	 I saw
[13:47:01] <Krenair>	 they vandalised a lot of my changes
[13:47:05] <Krenair>	 but it looks like ops cleaned it up so thanks
[13:47:26] <vgutierrez>	 yeah.. we've been working for that during EU morning
[13:52:30] <Krenair>	 vgutierrez, I made some cherry-picks and a release commit ready
[13:52:35] <Krenair>	 in case you want to do a release with what we have
[13:53:04] <vgutierrez>	 I've seen that, I'll proceed soon
[13:53:06] <vgutierrez>	 thx
[13:54:57] <Krenair>	 I didn't go through the whole process of approving stuff in case you wanted to add more
[14:00:08] <wikibugs>	 10netops, 10Operations: eqiad - eqord Telia link down - IC-314533 - https://phabricator.wikimedia.org/T218307 (10ayounsi) Opened a ticket with Equinix to check the X-connect.
[14:03:24] <vgutierrez>	 Krenair: I'll hold that till 15 UTC, librenms renewal should be triggered by then on acmechief1001
[14:03:35] <Krenair>	 k
[14:03:52] <vgutierrez>	 let's see how it goes, and if it works as expected I'll release 0.13 with those two commits
[14:11:34] <Krenair>	 vgutierrez, oh I also did https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/497431/
[14:11:44] <vgutierrez>	 the stretch support one?
[14:11:47] <Krenair>	 yep
[14:12:07] <vgutierrez>	 yep.. I've seen that as soon as my gerrit dashboard got cleared :D
[14:12:12] <Krenair>	 and I tested with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/497430/
[14:12:13] <Krenair>	 cool
[15:05:39] <vgutierrez>	 Krenair: needs a small fix but overall it went as expected: https://phabricator.wikimedia.org/P8225
[15:07:47] <wikibugs>	 10Acme-chief: acme-chief creates absolute symlinks for "live" and relative for "new" on certificate renewal - https://phabricator.wikimedia.org/T218685 (10Vgutierrez) p:05Triage→03Normal
[15:10:54] <vgutierrez>	 let me see if I can submit a test for that as well
[15:17:53] <Krenair>	 did you test that change vgutierrez ?
[15:18:07] <Krenair>	 didn't you want abspath rather than basename?
[15:23:15] <vgutierrez>	 Krenair: nope, the goal is a relative symlink, not an absolute one
[15:23:37] <Krenair>	 oh right
[15:23:47] <vgutierrez>	 I added a test in PS2
[15:23:58] <vgutierrez>	 to be sure that the live symlink is relative
[15:24:04] <vgutierrez>	 I guess that I should test the new symlink as well :)
[15:28:17] <vgutierrez>	 Krenair: please revert that :(
[15:28:50] <Krenair>	 vgutierrez, ?
[15:29:05] <vgutierrez>	 let me submit PS3 with a more complete test
[15:29:10] <Krenair>	 ok
[15:30:39] <vgutierrez>	 done, let me know if you want me to amend your release commit or you'll do it
[15:31:49] <Krenair>	 I'll do it
[15:32:46] <vgutierrez>	 thx
[15:33:42] <wikibugs>	 10netops, 10Cognate, 10Growth-Team, 10Language-Team, and 6 others: Rack/cable/configure asw2-a-eqiad switch stack - https://phabricator.wikimedia.org/T187960 (10Marostegui) For the record, read only times on s2: Read only ON: 15:21:10 Read only OFF: 15:27:56
[15:35:50] <vgutierrez>	 Krenair: do you perform the cherry-pick action from the web UI or from git CLI?
[15:36:01] <vgutierrez>	 if the latter.. please add -x to get the commit id on the commit msg
[15:36:45] <Krenair>	 web ui
[15:36:48] <vgutierrez>	 ack
[15:38:06] <Krenair>	 vgutierrez, are the rights on the new repo broken?
[15:38:13] <Krenair>	 I can't appear to create tags in the web UI
[15:40:24] <vgutierrez>	 Krenair: you should be able to create an annotated tag
[15:41:22] <vgutierrez>	 it gets the permissions from the group operations/software/certcentral, and you are listed as the unique member there
[15:42:17] <Krenair>	 Maybe I'm misremembering how to do this in the UI
[15:42:22] <Krenair>	 I thought there was a form at https://gerrit.wikimedia.org/r/#/admin/projects/operations/software/acme-chief,tags ?
[15:42:45] <vgutierrez>	 dunno TBH, I go with "git tag -a..."
[15:42:50] <Krenair>	 ok
[15:44:04] <vgutierrez>	 let the release commit to be merged before tagging BTW
[15:44:12] <vgutierrez>	 it was missing a rebase
[15:45:20] <Krenair>	 yeah
[15:46:55] <Krenair>	 vgutierrez, https://gerrit.wikimedia.org/g/operations/software/acme-chief/+/refs/tags/0.13
[15:47:06] <Krenair>	 managed to do it via the CLI, mucking about with permissions didn't seem to reveal the web UI form
[15:47:51] <vgutierrez>	 hmmm
[15:47:57] <vgutierrez>	 did you push the branch and the tag I'm afraid
[15:48:23] <Krenair>	 ugh
[15:48:36] <vgutierrez>	 git push origin --tag 0.13 is the proper way
[15:48:44] <Krenair>	 vgutierrez, fixed
[15:48:46] <vgutierrez>	 thx :D
[15:48:47] <Krenair>	 no more branch
[15:54:47] <Krenair>	 vgutierrez, so all gerrit stuff done I think
[15:54:51] <vgutierrez>	 almost :)
[15:54:56] <vgutierrez>	 we're missing the changelog commit
[15:55:00] <vgutierrez>	 in the debian branc
[15:55:02] <vgutierrez>	 *branch
[15:55:30] <vgutierrez>	 something like https://gerrit.wikimedia.org/r/c/operations/software/acme-chief/+/497259
[15:55:47] <vgutierrez>	 wanna go for it or should I?
[15:57:55] <Krenair>	 vgutierrez, doing
[15:58:04] <bblack>	 XioNoX: what's the context on all the lvs100x disable/enable?
[15:58:04] <Krenair>	 gotta run, changelog commit is up for review
[15:58:20] <XioNoX>	 bblack: row A server move
[15:58:28] <XioNoX>	 https://phabricator.wikimedia.org/T187960
[15:58:58] <bblack>	 why disable pybal though?
[15:59:33] <vgutierrez>	 Krenair: ack
[16:00:37] <XioNoX>	 bblack: depool in case they are pooled (even though those are backup)
[16:01:01] <bblack>	 ok
[16:11:07] <vgutierrez>	 UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 1742: ordinal not in range(128)... FFS
[16:11:24] <vgutierrez>	 a damn unicode space character in the README breaks package building in boron
[16:11:31] * vgutierrez breaths up to 10....
[16:45:57] <wikibugs>	 10Acme-chief: acme-chief creates absolute symlinks for "live" and relative for "new" on certificate renewal - https://phabricator.wikimedia.org/T218685 (10Vgutierrez) 05Open→03Resolved
[18:00:40] <elukey>	 hello people, as FYI conf1004 (etcd/zookeeper) is going to have a brief network outage for the maintenance that Arzhel is doing. It should all be fine but let's be vigilant in case Pybal doesn't like it
[18:03:51] <wikibugs>	 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10Patch-For-Review: Remove wildcard vhost for *.wikimedia.org - https://phabricator.wikimedia.org/T192206 (10EddieGP) 05Open→03Declined
[18:46:04] <wikibugs>	 10netops, 10Cognate, 10Growth-Team, 10Language-Team, and 6 others: Rack/cable/configure asw2-a-eqiad switch stack - https://phabricator.wikimedia.org/T187960 (10ayounsi)
[20:17:18] <wikibugs>	 10netops, 10Cognate, 10Growth-Team, 10Language-Team, and 6 others: Rack/cable/configure asw2-a-eqiad switch stack - https://phabricator.wikimedia.org/T187960 (10ayounsi)
[20:17:25] <wikibugs>	 10netops, 10Operations: Increase network capacity (2018-19 Q3 Goal) - https://phabricator.wikimedia.org/T213122 (10ayounsi)
[20:17:36] <wikibugs>	 10netops, 10Cognate, 10Growth-Team, 10Language-Team, and 6 others: Rack/cable/configure asw2-a-eqiad switch stack - https://phabricator.wikimedia.org/T187960 (10ayounsi) 05Open→03Resolved Everything here is done, thank you all for your help!
[20:18:19] <wikibugs>	 10netops, 10Operations, 10decommission, 10ops-eqiad, 10Patch-For-Review: Decommission asw-c-eqiad - https://phabricator.wikimedia.org/T208734 (10ayounsi)
[20:18:35] <wikibugs>	 10netops, 10Cognate, 10Growth-Team, 10Language-Team, and 6 others: Rack/cable/configure asw2-a-eqiad switch stack - https://phabricator.wikimedia.org/T187960 (10ayounsi)
[20:19:08] <wikibugs>	 10netops, 10Operations, 10decommission, 10ops-eqiad, 10Patch-For-Review: Decommission asw-c-eqiad - https://phabricator.wikimedia.org/T208734 (10ayounsi)
[20:19:25] <wikibugs>	 10netops, 10Cognate, 10Growth-Team, 10Language-Team, and 6 others: Rack/cable/configure asw2-a-eqiad switch stack - https://phabricator.wikimedia.org/T187960 (10ayounsi)
[20:19:31] <wikibugs>	 10netops, 10Operations: Increase network capacity (2018-19 Q3 Goal) - https://phabricator.wikimedia.org/T213122 (10ayounsi) 05Open→03Resolved
[22:27:10] <wikibugs>	 10Acme-chief, 10Patch-For-Review: CN + SNI list on config file doesn't match issued certificate on some scenarios - https://phabricator.wikimedia.org/T218418 (10Krenair) 05Open→03Resolved