[05:53:42] <_joe_>	 Amir1: https://gitlab.wikimedia.org/repos/sre/conftool/-/merge_requests/37
[09:33:37] <Amir1>	 _joe_: thanks!
[10:11:34] <elukey>	 hey folks! If you run "tox" locally for the cookbooks repo please rm the .tox dir since we just merged https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1078427 (to fix some pylint issues).
[10:12:04] <elukey>	 I know it is not ideal and annoying so if you have anything against it please blame me and Riccardo
[10:12:13] <volans>	 rm -rf .tox/py*-prospector "is enough" :D
[10:12:24] <elukey>	 (signed: your dear automation folks)
[10:26:03] <_joe_>	 let me repeat again
[10:26:10] <_joe_>	 please freeze the prospector version
[10:26:18] <_joe_>	 both in cookbooks and spicerack
[10:26:20] <_joe_>	 don'
[10:26:26] <_joe_>	 t make me do it for you :)
[10:26:40] <_joe_>	 and upgrades should only happen explicitly
[10:26:50] <_joe_>	 signed: everyone in SRE not named volans :)
[10:27:06] <volans>	 it was frozen yesterday and has been upgraded explicitly today
[10:27:18] <volans>	 and due to tox inability to manage the venv it requires the rm
[10:32:12] <elukey>	 _joe_ I pulled the trigger on this change, sometimes we just need to upgrade, hence the msg :) The rest is ok don't worry :)
[10:33:31] <volans>	 and I plan to get rid of it soon~ish
[13:53:25] <arturo>	 do you know if we have prometheus-based alerts if a kernel panic happens on a server?
[14:02:22] <jynus>	 jhathaway, brett: calm morning, I only documented https://phabricator.wikimedia.org/T303534#10209698
[14:02:39] <jynus>	 sorry, that was for swfrench-wmf
[14:03:09] <swfrench-wmf>	 no worries, and thank you :)
[14:15:08] <jhathaway>	 thanks jynus 
[14:47:18] <sergi0>	 !log deployment-prep: `sgimeno@deployment-mwmaint03:~$ foreachwiki userOptions.php --delete --old=1 growthexperiments-tour-newimpact-discovery` (T376461)
[14:47:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:47:21] <stashbot>	 T376461: Remove unused user property growthexperiments-tour-newimpact-discovery - https://phabricator.wikimedia.org/T376461
[16:29:15] <moritzm>	 puppetserver1001 needs to be taken down for hardware maintenance, starting in 10 minutes, I would disable Puppet fleet-wide and for 15-20 mins reimages and decoms would fail
[16:29:32] <moritzm>	 unless it's bad time for anyone, then please let me know
[16:29:55] <cdanis>	 moritzm: I think swfrench-wmf is still doing some LVS restarts
[16:30:18] <swfrench-wmf>	 ah, good timing :)
[16:30:36] <swfrench-wmf>	 moritzm: I have not touched anything yet, so please proceed
[16:30:47] <swfrench-wmf>	 and thanks for flagging, cdanis
[16:30:50] <moritzm>	 ok, thanks. I'll ping you when we're done
[16:31:02] <moritzm>	 (but would still wait another 10mins for anyone to object)
[16:57:02] <moritzm>	 puppetserver1001 maintenance is complete and Puppet is re-enabled fleet-wide
[16:57:22] <moritzm>	 swfrench-wmf: you can proceed with pybal restart
[16:58:01] <moritzm>	 and for general context: we have been updating the RAM on puppet servers in eqiad (codfw to follow) to 128G
[16:58:39] <swfrench-wmf>	 mortizm: ack, and thanks for the heads-up - that was quick :)
[16:58:53] <moritzm>	 sacrifices to the Java gods essentially
[16:59:01] <swfrench-wmf>	 moritzm: ^ (apologies for typos, heh)
[16:59:30] <moritzm>	 np at all, I've seen every permutation of my name by now :-)
[17:00:44] <moritzm>	 and for comparison: the old puppet 5 masters (mod_passenger and Ruby) had 16G and we barely used half of it even when all servers were on Puppet 5
[17:01:06] <moritzm>	 and now jruby/clojure
[20:49:56] <urandom>	 mutante: either you can merge mine, or let me know if it's OK to merge yours :)
[20:50:18] <mutante>	 urandom: yea, I just noticed the same thing. it's ok if you type "multiple"
[20:50:25] <mutante>	 sometimes it can separate them and sometimes not
[20:50:38] <urandom>	 mutante: done.
[20:50:44] <mutante>	 I am checking my part isnt breaking anything. thank you
[22:32:05] <mutante>	 current number of active critical alerts: 76     (are they really that critical if we can ignore them?)
[22:32:30] <mutante>	 maybe a good chunk should actually just be warnings