[13:36:16] topranks: sorry moving here operations is unusable rn [13:36:33] If it’s only mgmt net then no need to hold the switchover yeah? [13:37:04] But the potential alerting noise is a problem [13:37:25] Will you be done by the top of the hour ? [13:37:45] Cc slyngs [13:42:59] claime: yeah I should be. tbh these stupid mgmt link networks were an afterthought in my planning and the config is subtlely awful in terms of the migration [13:43:13] trying to find exactly what is wrong - but nothing has been touched for production networks [13:55:03] topranks: yeah ok. I’m just wary of alert flood during the switchover obscuring something actually wrong [13:58:20] claime: absolutely yeah things are stable now so go ahead [13:58:29] Thank you :-) [14:14:56] Jeff_Green: I'm about to merge a DNS change and I have an unrelated change to frbast showing up. Is this one of yours and safe to merge? If not, do you know whose it might be? [14:15:30] btullis: yup, I just saw the same thing and paused, yes please go ahead [14:15:51] Jeff_Green: Ack, merged, thanks. [14:15:56] thx! [14:33:36] I appreciate that our primary datacentre goes south for the winter [14:35:56] <_joe_> lizimedia: my slide on the topic at SRECon had migratory birds on the background :) [14:36:54] excellent [14:38:25] lizimedia: https://wikitech.wikimedia.org/wiki/Switch_Datacenter/Recurring,_Equinox-based,_Data_Center_Switchovers especially the FAQ is a nice read :D [14:39:45] Audio monitoring provided by https://listen.hatnote.com/#te,bn,et,hi,ru,en,fr,wikidata,es,de as usual (warning, the volume slider is automatically at full power I suggest putting it way down before clicking enable audio) [14:40:23] claime: well one of them is cyrusone-based [14:44:38] heeeeyyyy (☞゚ヮ゚)☞ [14:50:49] btw tmux is on cumin1004 sudo -i tmux -S /tmp/tmux_dc_switchover attach -t DCSwitchOver [14:51:03] that tmux version apparently doesn't support read only so keep your fingers to yourselves [14:52:44] der tmux ist nicht für der gefingerpoken und mittengraben! [14:52:55] it's a trixie host... [14:53:40] volans: they removed the -r cli flag in favour of user based access control basically [14:53:46] and we didn't set that up [14:54:02] Lucas_WMDE: mittengraben is a gut word [14:54:22] (I feel obliged to point out there’s nothing remotely german about it :P) [14:55:19] thats never stopped me in the past [14:55:32] all words are made up, etc etc [14:56:49] claime: sigh [14:58:03] Maintenance jobs are stopped [14:58:20] there is a an alert firing for poolcounter queues full, but I don't think it's a blocker [14:58:28] go/nogo for read-only please [14:58:33] cc slyngs [14:58:58] hnowlan jhathaway as well [14:59:09] thanks [14:59:48] ack, go from me [15:00:41] cezmunsta: good from db pov? [15:01:40] Yep, go ahead [15:02:12] go from me too [15:02:15] slyngs: proceed [15:02:47] https://listen.hatnote.com/ for newcomers [15:02:56] FYI I noticed the PC queue-full stuff in logstash last night too. The rate of those log messages became elevated after yesterday's switchover [15:03:00] we are silent [15:05:26] we are back [15:05:31] ding [15:05:34] <3 [15:05:34] sounds [15:05:47] :D [15:05:51] first try! [15:06:16] 2 minutes 50 seconds by my count [15:06:50] rzl: the real eternal september [15:08:43] haha I'm not sure all the helmfile output from mw-cron is still useful [15:08:54] what if there was a typo on line 2,194 [15:10:30] rzl: the horror [15:11:16] I could have sworn it said "mediakiwi" as it was scrolling past. you'd better scroll back up and check [15:11:37] (note to bystanders, this is a joke and claime knows it) [15:11:51] it's our new zealand specific version of mediawiki [15:11:55] it's ok [15:12:14] flightless :( [15:12:24] i will now never be able to get 'mediakiwi' out of my head, i blame rzl [15:12:39] honestly i can't believe we never made that joke before [15:12:51] mediaotearoa [15:13:13] claime: here at the kiwimedia foundation we know that we can do better. and we *will* do better. [15:18:35] tpakl dc when [15:19:31] how is https://meta.wikimedia.org/wiki/Kiwimedia not a redirect to https://meta.wikimedia.org/wiki/Wikimedia_Aotearoa_New_Zealand [15:23:26] anyway the actual concrete suggestion is we should consider `helmfile apply --suppress-diff` [15:24:10] it'd be nice to have `--just-tell-me-how-many-objects-are-affected` as a security blanket, but I don't think it's a requirement [15:24:39] and maybe there's even a `--diff-args` incantation for that [15:25:59] rzl: use --show-only-dangerous-diffs [15:26:11] oh man now you're talking [15:27:40] charlie --cowboy [15:28:19] --cowboy is implicit, if you try set --nocowboy it just refuses to run [15:29:08] I should add --yeehaw as an alias to --dangerously_fast but that name is already so much fun [15:43:34] rzl: would you accept a patch adding an alias --🤠 [15:43:37] we are done :) great job slyngs <3 [15:43:44] nice job!! [15:43:51] great job slyngs and everyone else! [15:43:56] Thank you jasmine_ and claime, you are wonderful :-) [15:43:58] thanks to jasmine_ for the support throughout the process especially <3 [15:44:39] great jobs yall :) [15:45:12] great! [15:45:37] btullis: ok to merge your puppet patch? ('dse-k8s-wdqs: Add dse-k8s-wdqs1004 and dse-k8s-wdqs2005') [15:45:50] Oh yes please. Sorry. [15:46:35] and to u all for hanging on for the longest sept. ever :') <3 [15:47:41] also pumpkin themed switchover def lived up to the all love no spooks [15:48:26] jasmine_: I will miss meeting up with you every week. We'll have to make up some other project