[02:13:20] <swfrench-wmf>	 many thanks R.aine for taking point on this and K.rinkle for reviewing the proposed updates! also, apologies for missing the mention while focused on T383047 earlier :)
[02:13:20] <stashbot>	 T383047: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047
[06:50:40] <marostegui>	 I've switched m5 master, which has mailman, if you see/notice something strange let me know
[08:01:51] <XioNoX>	 bjensen, marostegui, I'm going to depool drmrs shortly to upgrade the core routers for https://phabricator.wikimedia.org/T431749
[08:06:14] <marostegui>	 XioNoX: thanks for the heads up
[09:21:34] <marostegui>	 XioNoX: check -operations if you have a sec
[09:30:33] <XioNoX>	 repooling drmrs
[09:47:32] <Raine>	 oh no, T383047 is One Of Those (see last comment) :D 
[09:47:33] <stashbot>	 T383047: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047
[13:03:23] <godog>	 cezmunsta: merging your change too
[13:04:44] <godog>	 {{done}}
[13:12:56] <swfrench-wmf>	 FYI, in advance of maintenance on the conf* (e.g., conftool etcd) cluster in codfw, we're going to start depooling etcd client traffic there (in favor of reading from eqiad) shortly. this is similar to what we did a couple of weeks ago for a rack maintenance, and no impact is expected.
[13:12:56] <swfrench-wmf>	 cc: oncallers Raine, federico3
[13:13:10] <federico3>	 ack
[13:28:26] <XioNoX>	 federico3, Raine, I'm going to depool esams for a router upgrade - https://phabricator.wikimedia.org/T431751
[13:28:37] <federico3>	 ack
[13:34:24] <cezmunsta>	 godog: ty
[14:11:03] <XioNoX>	 repooling esams
[16:10:57] <swfrench-wmf>	 following up on the conf* work: etcd is back in action on conf2004 post-reimage. investigating some issues that are preventing zookeeper startup along with elukey.
[17:07:57] <swfrench-wmf>	 update: I was able to get zookeeper running again with some classpath hackery. I will follow with some puppet patches to persist it, along with other issues discovered.
[17:08:18] <cdanis>	 thanks <3
[17:47:21] <swfrench-wmf>	 anyone feeling particularly interested in reviewing [0] to lock in the classpath hack? as I mention in the comment on the patch [1]
[17:47:21] <swfrench-wmf>	 [0] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1318749
[17:47:25] <swfrench-wmf>	 [1] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1318749/comments/2305ae3d_a792a060
[17:47:33] <swfrench-wmf>	 whoops ... hit enter too soon :)
[17:48:09] <swfrench-wmf>	 as I mention in the comment on the patch, I suspect that PCC believes this to be a noop for conf2004 due to it only having very recently switched to bookworm.
[17:49:02] <rzl>	 almost certainly yeah - https://wikitech.wikimedia.org/wiki/Help:Puppet-compiler#Manually_update_production is the incantation if you wanted to wait for it
[17:49:46] <swfrench-wmf>	 thanks! I have had ... extremely mixed results with that actually working, so I *probably* won't wait :)
[17:50:01] <rzl>	 haha valid
[17:50:07] <rzl>	 (looking)
[17:53:19] <rzl>	 +1
[17:53:31] <swfrench-wmf>	 thank you!
[18:29:16] <swfrench-wmf>	 alright, next one: if anyone might be able to review [0], that would be greatly appreciated. this maintains the status quo in order to get things stable, though I would like to revisit whether we even want these nginx SSL params anymore, given the current state of the certificates we use.
[18:29:16] <swfrench-wmf>	 [0] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1318760
[18:54:22] <swfrench-wmf>	 And now one more [1], to fix the v2-API backup timer on etcd v3.4+ - if anyone might be able to review that and [0] above, that would be swell :)
[18:54:22] <swfrench-wmf>	 [1] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1318764
[18:58:24] <cdanis>	 looks good, thanks scott
[18:58:54] <swfrench-wmf>	 amazing, thanks cdanis!
[19:19:56] <swfrench-wmf>	 alright, I'm going to pause for a bit. I'll likely take up two more follow-up changes today, then call it a day and pick things up early tomorrow. thanks, all, for the reviews!
[19:37:21] <Raine>	 belated on-call handoff: boring \o/ (despite Sco.tt's best efforts ;-) )
[20:31:34] <jhathaway>	 swfrench-wmf: if you have a moment, I tracked down our email timeout, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1318778
[20:32:18] <swfrench-wmf>	 oh, that's awesome! I'm doing something mildly fiddly, but will take a look soon
[20:32:28] <jhathaway>	 thanks
[23:01:10] <swfrench-wmf>	 FYI, I'm hands-off on conf* hosts in codfw for the day. current status is summarized in [0] and I expect to pick things back up first thing tomorrow. in the meantime, etcd client traffic remains depooled from codfw.
[23:01:10] <swfrench-wmf>	 [0] https://phabricator.wikimedia.org/T428495#12165373