[07:05:07] <elukey>	 goood morning
[07:06:05] <elukey>	 I see some errors for wmf auto restart and mcelog on some analytics nodes, I am wondering if anything changed
[07:11:10] <elukey>	 ah mcelog on the nodes with 4.19 seems not coming up
[07:13:14] <elukey>	             if versioncmp($::kernelversion, '4.12') >= 0 {
[07:13:14] <elukey>	                 systemd::mask { 'mcelog.service': }
[07:13:15] <elukey>	             }
[07:13:17] <elukey>	 perfect
[07:15:11] <elukey>	 but then in base::standard_packages we have
[07:15:11] <elukey>	           if os_version('debian == jessie') or os_version('debian == stretch') {
[07:15:14] <elukey>	               require_package('mcelog')
[07:15:17] <elukey>	               base::service_auto_restart { 'mcelog': }
[07:15:19] <elukey>	           }
[07:15:20] <elukey>	 and I have 4.19 on stretch
[07:25:49] <elukey>	 there is also a problem with the Zayo transport between eqiad and codfw
[07:26:14] <elukey>	 from the zayo correspondence I see that an issue was resolved 18h ago for the same link, but then nothing
[07:30:10] <elukey>	 optics looks good afaics from both sides
[07:32:42] <elukey>	 and from their last update there is a suggestion to disable/enable if the service is down, but this looks more another issue
[07:36:34] <elukey>	 cdanis: I saw all now your last email to zayo full of emojis, 10+
[07:36:43] <elukey>	 s/all/only
[07:37:23] * elukey sends an email to Zayo
[07:46:04] <elukey>	 done
[08:16:16] <XioNoX>	 thx!
[09:13:26] <jbond42>	 elukey: in relaion to the auto restarti mad an update yesterday to switch them all from using cron to using systemd::timer::job.  initially there was an issue with jessie not recognising the timespec interval however that should be resolved now.  i can take a look if you let me know a host thats having issues
[09:18:17] <moritzm>	 jbond42: that was related to mcelog, which doesn't work with Linux 4.19
[09:18:32] <jbond42>	 ahh ok thx
[09:18:51] <moritzm>	 and a few analytics servers are now using Stretch with 4.19, so the mcelog auto restart was failing there and the failure was simply no longer silenced
[09:19:00] <moritzm>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/636560/ will fix it
[09:21:17] <jbond42>	 ack thanks
[09:21:57] <jbond42>	 FYI all ripe81 starts today https://ripe81.ripe.net/programme/meeting-plan/
[09:39:02] <XioNoX>	 thx!
[11:34:29] <XioNoX>	 someone is familiar with Apache config? trying to figure out something easy I think
[14:14:25] <_joe_>	 XioNoX: that's a trick question right?
[14:14:35] <_joe_>	 anyone familiar with apache knows it's never easy
[14:15:13] <XioNoX>	 haha yeah, that's what I figured out
[17:25:24] <mutante>	 andrewbogott: can we merge my change related to restricted_to/from? compiler likes it and yesterday's problem is gone, right
[17:34:02] <andrewbogott>	 mutante: sure, you can merge that now if you want
[17:34:04] <andrewbogott>	 or I can
[17:34:54] <mutante>	 andrewbogott: ok, thanks. doing it
[17:36:03] <mutante>	 merged on master
[17:36:59] <mutante>	 all: this should remove noise from compiler output on all compiles. those lines at the top about "restricted_to"/"restricted_from" not being found will be gone
[17:37:05] <mutante>	 confirming it now
[17:38:54] <mutante>	 yep, looking good
[17:39:47] <dsaez_>	 ppl,  my nickname  was banned, I don't know why, do you know where/how I can try to solve this problem? I can't join most of the channels (because I-m not registered). That is why I-m asking here.
[17:41:07] <mutante>	 dsaez_: are you familiar with nickserv? make sure you are identified with it by password
[17:41:42] <dsaez_>	 mutante, I know nickserv, but I can't change my  nick to dsaez, because I recieve this error:
[17:41:54] <dsaez_>	 435 dsaez #wikimedia-office Cannot change nickname while banned/quieted on channel
[17:42:40] <mutante>	 dsaez_: ah, ok. try leaving these channels, change your nick, then rejoin them
[17:42:41] <cdanis>	 dsaez_: try doing this first: /msg nickserv identify dsaez <PASSWORD>
[17:42:41] <dsaez>	 aha!
[17:42:43] <cdanis>	 ah
[17:42:45] <cdanis>	 there you go
[17:43:34] <mutante>	 a lot of channels don't want you to change nick while you are in them. especially large channels like ##debian etc. the fix is to /part them, identify and /join again
[17:44:36] <mutante>	 they don't mean "banned" as in "cant come back"..it's a bit strange language
[17:52:33] <bblack>	 https://rachelbythebay.com/w/2020/10/26/num/ <- sounds cumin-applicable and not too difficult
[17:54:02] <paravoid>	 haha nice idea
[17:54:03] <paravoid>	 volans: ^^
[17:55:42] * volans reading
[18:05:14] <cdanis>	 haha didn't realize you followed her blog bblack :)
[18:12:14] <volans>	 bblack, paravoid: https://gerrit.wikimedia.org/r/c/operations/software/cumin/+/636729
[18:13:02] <paravoid>	 volans: maybe add the link to the commit msg to give credit where credit is due :)
[18:13:04] <cdanis>	 ahah
[18:13:51] <volans>	 {done}
[18:15:10] <bblack>	 nice :)
[18:15:29] <paravoid>	 nice indeed!
[18:15:37] <paravoid>	 thanks for the responsiveness, that's awesome :)
[18:15:38] <volans>	 I didn't apply the formatting thing, we rarely go over 1k
[18:15:45] <volans>	 and sounded a bit too much
[18:15:54] <volans>	 but happy to do if you're strong about it
[18:16:00] <volans>	 *you feel strongly about it
[18:16:18] <bblack>	 yeah my bikeshed thoughts are:
[18:16:59] <bblack>	 1) Yes, we could add the comma for the 1k+ case, maybe that's a nice additional circuitbreaker when affecting most of our environment, to avoid the copypaste reflex
[18:17:31] <bblack>	 2) For big lists, currently the total is only printed above the list:
[18:17:33] <bblack>	 1576 hosts will be targeted:
[18:17:37] <bblack>	 acmechief2001.codfw.wmnet, [...]
[18:17:47] <bblack>	 which means it almost always scrolls off for big cases
[18:17:50] <volans>	 yes, I didn't move it/repeated it on purpose ;)
[18:18:14] <bblack>	 so yeah, you could take the view that keeping it up top is the same kind of impediment as formatting, kinda
[18:18:44] <bblack>	 there's a few different ways you could go with both questions, but the way it is now makes for a very simple and minimal change, and isn't bad :)
[18:19:49] <volans>	 now just wait 6~12 months for a release :-P
[18:20:02] <volans>	 the upstream maintainer is a slacker
[18:20:07] <bblack>	 if you wanted to make the whole team curse you for all time, you could make the input string be the english version
[18:20:10] <rzl>	 please do actually print it below the list too
[18:20:20] <bblack>	 "One thousand four hundred sixty eight"
[18:20:21] <rzl>	 a speed bump is a good idea, but that doesn't mean you need to intentionally make it difficult ;)
[18:21:13] <bblack>	 ^ if you take rzl's position, a good trade is to then add the formatting, too, so as to preempty a copypaste reflex for the few very-large cases
[18:21:17] <volans>	 rzl: if your window terminal is 50 lines height it doesn't scroll :-P
[18:21:50] <rzl>	 "works on my machine"
[18:22:00] <volans>	 one of my original ideas back in the day was to randomly pick beteen a list of words like 'yes', 'go', 'proceed', et....
[18:22:07] <volans>	 I thought you would be all mad at me
[18:22:26] <rzl>	 You Don't Need To Intentionally Make It Difficult
[18:22:28] <Reedy>	 only in english? pfft
[18:22:30] <bblack>	 you could also do a low-end cutoff and just ask/accept yes like before if it's less than some cutoff lke 25
[18:22:38] <Reedy>	 write it in languages where it preferrably has non a-z characters
[18:22:39] <bblack>	 but that's less elegant to implelemt :)
[18:23:13] <volans>	 convert it to roman numbers
[18:23:21] <volans>	 that's so hard to implement :D
[18:25:48] <bblack>	 https://pypi.org/project/roman/
[18:25:59] <volans>	 yeah :)
[18:26:26] <mszabo>	 take it to the next level with https://web.eecs.umich.edu/~imarkov/Perligata.html ;)
[18:27:23] <bblack>	 ... and this is why nobody likes Perl anymore :)
[18:28:27] <rzl>	 two years later, onboarding a new SRE: "what's cumin?" "oh, we used to use that for orchestration but nobody knows how to use it anymore. here, let me show you how we just ssh in a for-loop, it's quicker"
[18:29:08] <wkandek>	 btw: https://techblog.wikimedia.org/2020/10/26/impact-of-using-http-connection-pooling-for-php-applications-at-scale/ is live
[18:32:52] <rzl>	 \o/
[18:35:56] <mutante>	 PHP is a "loved one", heh :)
[18:37:20] <wkandek>	 "i love php" - About 3,210,000,000 results (0.57 seconds)
[18:37:39] <wkandek>	 "i hate php" - About 307,000,000 results (0.54 seconds)
[18:38:15] <_joe_>	 in two lines - what's wrong with the internet.
[18:39:24] <mutante>	 hehe, there used to an IRC bot for this, !googlefight  (https://en.wikipedia.org/wiki/Googlefight)
[18:39:32] <cdanis>	 🎉
[18:39:39] <cdanis>	 _joe_: have you seen Slack's defense of PHP?
[18:39:52] <rzl>	 he links to it in that post :D
[18:39:59] <_joe_>	 cdanis: RTFA
[18:41:33] <mszabo>	 Slack's defense of s/PHP/Hack/g
[18:42:19] <mszabo>	 :D
[20:44:24] <razzi>	 I'm looking to publish my gpg key by following the instructions at https://wikitech.wikimedia.org/wiki/PGP_Keys, and I'm getting at `keyserver send failed: General error`. The issue may be with the hkps://hkps.pool.sks-keyservers.net keyserver; has anybody else had trouble with that?
[20:46:02] <cdanis>	 yeah, they are mega unreliable nowadays :(
[20:46:16] <cdanis>	 lots of endemic problems there, but the easiest thing to do is to use hkp://keyserver.ubuntu.com
[20:50:37] <razzi>	 Cool, that worked :)
[20:58:55] <mutante>	 I'll ACK the logstash codfw alerts because they have been unhandled CRIT with disabled notifications for > 18 days and it's a lot of noise
[21:02:08] <mutante>	 linking to https://phabricator.wikimedia.org/T263970 because that's the closest i see
[22:30:55] <shdubsh>	 mutante: thanks for pointing out the logstash issue.  looks like an unassigned shard was preventing an old index from reaching desired replica count on the new cluster.  Reduced replica count to 1 so the check would clear up.
[22:33:25] <mutante>	 shdubsh: ah! cool, thank you