[05:20:54] <elukey>	 wow all sessions migrated away from Redis! \o/
[05:28:57] <_joe_>	 elukey: not really
[05:29:03] <_joe_>	 centralauth is still there
[05:30:54] <elukey>	 _joe_ no idea what it does for mw, I saw the task for sessions migrated to kask though
[05:30:57] <elukey>	 namely https://phabricator.wikimedia.org/T243106
[05:31:05] <_joe_>	 sure
[05:31:24] <_joe_>	 so it's half of the sessions, see the discussion between me and Petr yesterday night
[05:31:34] <elukey>	 ah snap, reading
[05:31:35] <elukey>	 sigh
[05:39:16] <_joe_>	 heads up: I'm changing the users employed to pool/depool servers via etcd
[05:39:27] <_joe_>	 if you see any issues, please report them to me
[07:49:16] <mutante>	 What would be a good description of it.wikinews.org .. in Italian?
[07:49:26] <mutante>	 I see it is called "wikinotizie" in Italian
[07:49:45] <mutante>	 so "Wikinews Italiano" is probably wrong
[07:50:17] <kormat>	 mutante: italian is a fake language anyway, just make something up. ;)
[07:50:22] <mutante>	 being asked on a ticket to add it to the Google "Publisher Center" so that it gets indexed
[07:50:30] <mutante>	 haha kormat :)
[07:51:42] <_joe_>	 "wikinotizie" is a literal translation of wikinews
[07:54:05] <mutante>	 i used "Wikinotizie in italiano" so far
[07:54:32] <mutante>	 this ticket is kind of an issue anyways. there is much more to do there and it probably doesn't scale that we do this for all sites. it is also the first time though people ask about that Publisher Center
[07:54:51] <mutante>	 the alternative is to deal with delegated access to non-staff
[07:55:01] <_joe_>	 mutante: "wikinotizie" is enough, we would never expect that to be in english
[07:55:18] <_joe_>	 mutante: I don't think it's an SRE task then
[07:55:25] <mutante>	 _joe_: ok, thanks! primary language: Italian  but "worldwide" location
[07:55:29] <_joe_>	 what's the task?
[07:55:39] <mutante>	 https://phabricator.wikimedia.org/T253988
[07:55:52] <mutante>	 it's called "add to Search Console" but it turned out to be diffrent
[07:55:59] <mutante>	 it's the Publisher Center thing now
[07:56:28] <mutante>	 i am trying it this one time for just it.wikinews to get an idea...
[07:57:10] <mutante>	 but if they have a lot of follow-up or other languages we need to figure something out about access
[07:57:46] <_joe_>	 ok so
[07:57:52] <_joe_>	 this needs to be automated
[07:58:28] <_joe_>	 given a list of domains (that we procure somehow, more on that later), we should be able to use some google API to sync what we want published
[07:58:29] <mutante>	 yea.. every time i click next there is another screen, asking for logo now
[07:59:04] <mutante>	 this is something for a dedicated SEO person at WMF ?
[07:59:07] <_joe_>	 mutante: please open a task about automating the process
[07:59:11] <mutante>	 ok
[07:59:14] <_joe_>	 mutante: partly, yes
[07:59:24] <_joe_>	 I have no idea who would be the product owner for this tbh
[08:00:01] <mutante>	 yea..that is the issue.. even though ironically we have listed a whole handful of people outside SRE with access to that noc@ account that can do it
[08:00:27] <mutante>	 ticket was originally an SRE-access-request to let the volunteer have it too
[08:01:05] <mutante>	 at first seemed like we just have to do a few clicks to add a domain.. but it's more
[08:06:54] <_joe_>	 this needs to be escalated I think, talk with wolfgang :)
[08:08:20] <mutante>	 agreed.   "Google can sell ads to show in your publication in the Google News app. Control how much of your ad inventory you want Google to manage."  defaults to 30%... yea.. no
[08:13:04] <_joe_>	 mutante: I'm not sure you should make those decisions
[08:13:28] <_joe_>	 as in, you shouldn't be requested to
[08:14:13] <_joe_>	 and also, I would expect this needs a green light from legal?
[08:15:07] <mutante>	 _joe_: no worries, i am already creating a ticket for it and it is all in status 'draft' and not actually published because it's missing a bunch of stuff I am not going to do (contacts, square logo, the ads part.... )
[08:15:22] <_joe_>	 yeah indeed
[08:15:31] <mutante>	 also changed it to 0%
[08:15:54] <_joe_>	 I wasn't considering this would make contents be managed / owned by us on a google platform
[08:16:08] <_joe_>	 I'd want legal to sign off on any of that
[08:16:21] <mutante>	 for sure, i'll tag it with legal too
[08:16:59] <_joe_>	 kormat: re your comment on italian, that's not nearly as offensive to my culture as pineapple on pizza or ketchup on pasta
[08:17:08] <_joe_>	 both things I've seen doing in your motherland
[08:17:39] <kormat>	 :D
[08:18:07] <_joe_>	 there's nothing to laugh about.
[08:19:31] <kormat>	 _joe_: i used to work with an italian from bologna who would get agitated about "spaghetti bolognese"
[08:19:44] <_joe_>	 ahah poor soul
[08:25:14] <hashar>	 _joe_: good morning, would you be kind enough to rebuild the doxygen package for me please? I have to upgrade it further from 1.8.17 to 1.8.18  ( https://gerrit.wikimedia.org/r/#/c/operations/debs/doxygen/+/599094/ )
[08:25:38] <hashar>	 the 1.8.17 built a couple weeks ago segfault when generating the doc for Wikibase :-\
[08:25:50] <_joe_>	 hashar: we have mutante on clinic duty, maybe he could do it :)
[08:26:04] <hashar>	 yeah either way would work ;]
[10:12:17] <jbond42>	 Hi all, im about to deploy a fix to puppet-merge (https://gerrit.wikimedia.org/r/c/operations/puppet/+/601712) please ping if you see any issues
[10:20:33] <elukey>	 jbond42: o/ https://phabricator.wikimedia.org/P11394
[10:21:19] <jbond42>	 elukey: ack thanks will roll back in a sec just trying to debug a sec
[10:21:28] <elukey>	 sure sure no rush
[10:21:34] <elukey>	 the change is low priority
[10:22:45] <jbond42>	 acvk thanks
[10:39:56] <jbond42>	 elukey: your change should be deployed now
[10:40:43] <jbond42>	 i have finished my deploy now
[10:42:24] <vgutierrez>	 hmmm maybe I missed something, but confctl is missing from every cp server apparently
[10:43:07] <vgutierrez>	 we have a bunch of icinga UNKNOWNs due to that
[10:43:52] <jbond42>	 vgutierrez: could that be in anyway related to the puppet-merge upgrade?
[10:45:05] <vgutierrez>	 so puppet deployed a change on /usr/bin/ispooled
[10:45:10] <vgutierrez>	 Jun  4 10:23:30 cp3050 puppet-agent[11029]: (/Stage[main]/Conftool::Scripts/File[/usr/local/bin/ispooled]/content) content changed '{md5}bfbacdc16a47701a16b73443e919f77c' to '{md5}a4d0b7067549eee2427c6edbf7f0fb2f'
[10:46:27] <vgutierrez>	 _joe_, godog: https://gerrit.wikimedia.org/r/c/operations/puppet/+/599299
[10:46:56] <vgutierrez>	 so something is clearly wrong with that change
[10:46:58] <_joe_>	 vgutierrez: uh, confctl missing?
[10:47:02] <vgutierrez>	 not missing at all
[10:47:07] <vgutierrez>	 https://www.irccloud.com/pastebin/rySqcpJd/
[10:47:15] <vgutierrez>	 it's there
[10:47:27] <godog>	 sigh I'm dumb, the condition is wrong
[10:47:31] <godog>	 I'll fix it
[10:47:33] <_joe_>	 yes
[10:47:34] <vgutierrez>	 godog: <3
[10:47:40] <_joe_>	 gosh how could I not see it :P
[10:48:13] <_joe_>	 vgutierrez: although apparently it worked on some cp servers
[10:48:25] <_joe_>	 I used pool/depool manually earlier on cp2040
[10:49:00] <_joe_>	 oh it was merged now
[10:49:02] <_joe_>	 I get it
[10:49:03] <vgutierrez>	 :)
[10:49:51] <godog>	 https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/602334
[10:50:56] <vgutierrez>	 godog: that also covers pool/depool?
[10:51:00] <_joe_>	 yes
[10:51:13] <vgutierrez>	 nice
[10:52:12] <_joe_>	 btw, with this morning's changes, if your script for some reason do something wrong, they can only affect the pool your server is in, they're not allowed to delete the whole etcd tree in one go
[10:52:18] <_joe_>	 small progress :P
[10:52:35] <vgutierrez>	 tsk, that's no good
[10:52:44] <vgutierrez>	 I want my t-shirt :P
[10:52:49] <_joe_>	 you will have to go to cumin1001 to do so
[10:53:01] <godog>	 ok merged, I'll do a round of puppet runs on cp hosts
[11:07:31] <godog>	 vgutierrez: puppet run finished and icinga is recovering, thanks for the heads up!
[11:29:30] <vgutierrez>	 thx godog ❤️
[13:57:39] <cdanis>	 jbond42: one thing that might be smart to do in the future is to grab the expanded puppet-merge.sh.erb from PCC and run it through shellcheck
[13:57:49] <cdanis>	 curl https://puppet-compiler.wmflabs.org/compiler1003/22927/puppetmaster1002.eqiad.wmnet/change.puppetmaster1002.eqiad.wmnet.pson | jq '.resources[] | select(.title == "/usr/local/bin/puppet-merge") | .parameters.content' -r | shellcheck -
[13:58:08] <cdanis>	 In - line 181:
[13:58:10] <cdanis>	 if [ $LABS_PRIVATE -eq 1 -a ${LABS_EXIT} -ne 99]; then
[13:58:12] <cdanis>	 ^-- SC1009: The mentioned syntax error was in this if expression.
[13:58:14] <cdanis>	    ^-- SC1073: Couldn't parse this test expression. Fix to allow more checks.
[13:58:16] <cdanis>	                                                 ^-- SC1020: You need a space before the ].
[13:58:18] <cdanis>	                                                 ^-- SC1072: Missing space before ]. Fix any mentioned problems and try again.
[13:59:05] <cdanis>	 dunno if we could easily integrate "expand the erb and run it through shellcheck" with CI but it'd be neat
[14:01:01] <jbond42>	 cdanis: that would be good and i know simlar thing shave been asked for python.erb files as wel.  could you create a task and assign it tyo me, unfortunatly i broke cloud with that change so on a call trying to help fix
[14:01:16] <cdanis>	 sure!
[14:01:28] <cdanis>	 even just a procedure for manually checking given PCC output would be useful as a v0
[14:01:56] <cdanis>	 (I'll jot down the dumb oneliner I just devised)
[14:02:05] <jbond42>	 thanks
[15:00:57] <elukey>	 kormat: kudos for https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/601761/
[15:03:44] <kormat>	 elukey: well, i don't know whether i should get kudos or abuse for having done it, but i'll take the kudos :)
[15:04:53] <elukey>	 definitely kudos!
[15:04:57] <cdanis>	 kormat: we already had congratudolences, now we also have kudodemnation
[15:05:11] <kormat>	 hahahaha
[15:05:44] <cdanis>	 thumbs-up with one hand while shaking your fist with the other
[15:10:28] <rzl>	 setting off fireworks for you, but not being TOO careful where they're pointed
[15:10:43] <kormat>	 :D
[15:19:08] <_joe_>	 I think rzl nailed the metaphor
[15:19:42] <cdanis>	 he usually does
[15:19:45] <_joe_>	 233 lines and .sh
[15:20:04] <_joe_>	 the signs are all there
[15:20:09] <cdanis>	 _joe_: and actual sh, none of this 'bash' fad
[15:20:13] <kormat>	 _joe_: don't worry, i've just added 15 new lines
[15:20:37] <kormat>	 cdanis: sh not bash, _and_ busybox utilities only. it's the worst of the worst :)
[15:20:53] <_joe_>	 wow
[15:21:16] <_joe_>	 yeah we're not blaming you kormat
[15:21:26] <_joe_>	 anything at that stage of installation is evil
[15:21:30] <rzl>	 kormat: I've only heard them say that when they're blaming you, jfyi
[15:21:39] <kormat>	 rzl: *giggle*
[15:21:58] <_joe_>	 rzl: shhhh
[15:22:04] <_joe_>	 i almost got away with it
[15:22:44] <moritzm>	 the limitations within debian-installer are also totally self-imposed, it's sacrificing flexibility for development for very dubious efficiency gains in terms of image size
[15:22:55] <kormat>	 moritzm: +1
[15:23:16] <moritzm>	 as if it would matter with today's network/disks/RAM if the image avoids bash...
[15:23:34] <kormat>	 moritzm: it would change Everything if they'd just include `python3-minimal`
[15:24:05] <moritzm>	 kormat: defininitely, there's also the design choice that some applications build udebs
[15:24:25] <moritzm>	 effectively packages get built twice for providing a narrower set within d-i
[15:24:40] <kormat>	 mmm, right
[15:25:10] <moritzm>	 which also causes plenty of churn, limits flexibility for little again (effectively just saving some image size)
[15:25:35] <cdanis>	 what's the smallest computer that needs to run debian-installer anyway?? all the small systems get boot images generated on larger hosts anyway
[15:26:22] <moritzm>	 there's still some "enthusiasts" which port Debian to the Amiga/m68k
[15:26:34] <moritzm>	 but I suspect most of these actually run it in emulators, no idea
[15:32:14] <mutante>	 https://wiki.debian.org/RaspberryPi
[15:34:44] <cdanis>	 mutante: yes, that is one of the things I had in mind -- the official boot images (which you just put on an SD card) are all generated via cross-compiler
[15:35:21] <mutante>	 ah, right. ack
[15:35:46] <cdanis>	 also, even an original raspberry pi (I have one) has no trouble with python3-minimal ;)
[16:21:54] <kormat>	 nooooo. busybox doesn't have awk in d-i
[16:22:48] <cdanis>	 "initially developed in 1977"
[16:23:58] <kormat>	 cdanis: it's just too damn newfangled
[17:08:39] * bd808 misses his Amiga/m68k boxen
[17:59:23] <_joe_>	 moritzm: you'd be surprised
[17:59:37] <_joe_>	 I have a couple friends who run linux on amigas :)
[18:06:32] <_joe_>	 we should improve our interviewing process
[18:07:14] <_joe_>	 I should add "would you eat pizza with pineapple?" to my cultural fit questions
[18:07:28] * kormat grins
[18:07:59] <cdanis>	 _joe_: I guess I failed then
[18:08:08] <_joe_>	 many of you
[18:08:30] <rzl>	 _joe_: if you do that, I'm going to add "would you be offended by someone else's pizza?" to mine
[18:08:44] <_joe_>	 but I mean, I had a taco with onions and sweet pepper last week
[18:08:57] <cdanis>	 I think that is allowed
[18:08:57] <_joe_>	 rzl: well that's discriminatory against italians
[18:09:36] <_joe_>	 cdanis: I am not so sure, but it was really a ""taco""
[18:09:54] <_joe_>	 I just wrapped some stuff in a taco
[18:10:17] <andrewbogott>	 Is there a good way for me to test 'puppet agent --noop -tv' without actually enabling puppet and running the risk of a non-noop run?
[18:10:22] <mutante>	 Tacos Al Pastor with Grilled Pineapple Salsa
[18:10:35] <andrewbogott>	 mutante: I recently learned that that's lebanese/mexican fusion
[18:10:36] <_joe_>	 andrewbogott: nope
[18:10:43] <cdanis>	 pineapples aren't quite 'traditional' on al pastor but they're half-traditional
[18:10:47] <andrewbogott>	 alas!
[18:10:59] <mutante>	 andrewbogott: TIL, i did not know :)
[18:11:14] <andrewbogott>	 (which explains why an al pastor stand looks so much like a shawarma stand)
[18:11:27] <cdanis>	 mutante: if you offered 'tacos al arabes with pineapple' I think you'd be in the wrong, but, certainly not al pastor
[18:11:42] <cdanis>	 AIUI al arabes refers to the original lebanese/mexican fusion
[18:11:53] <andrewbogott>	 lots of Lebanese immigrants to Mexico back in the day, they figured out that pork was more popular than lamb at their shops, and they were mostly Christian refugees so no prohibition against pork
[18:12:08] <rzl>	 now I want a shawarma and you are all monsters
[18:12:12] <_joe_>	 I just had a pizza, and you managed to make me hungry again
[18:12:17] <cdanis>	 I want a torta
[18:12:19] <_joe_>	 yeah rzl tell them
[18:12:20] <andrewbogott>	 rzl, what is standing between you and getting a shawarma?
[18:12:29] <_joe_>	 andrewbogott: health
[18:12:35] <andrewbogott>	 bah!
[18:12:44] <jbond42>	 andrewbogott: i normaly do `sudo enaple-puppet 'msg'; run-puppet-agent ; disbale-puppet 'msg';
[18:12:52] <rzl>	 also, if I were going to get a good one, a quantity of ocean
[18:13:00] <jbond42>	 with extra sudo's not perfect but sohuld be good enough
[18:13:03] <_joe_>	 jbond42: he wants to do a noop run
[18:13:05] <andrewbogott>	 jbond42: and hope to get lucky, I take it?
[18:13:17] <rzl>	 there is good shawarma in the US but probably not in Tallahassee
[18:13:22] <_joe_>	 andrewbogott: in that case i just remove teh cronjob
[18:13:27] <jbond42>	 oh the yes puppet agent -t --noop in the middle instead
[18:13:43] <andrewbogott>	 ok, yes, I just thought of that.  It's still a cron and not a systemd timer right?
[18:14:01] <andrewbogott>	 should work, thanks all
[18:14:26] <_joe_>	 yes, still a cron
[18:14:54] <andrewbogott>	 Good, I'm more confident in my ability to effectively disable a cron.
[18:14:55] <cdanis>	 rzl: Sahara Cafe doesn't look terrible but they don't have the rotisserie up front, so minus points for that
[18:14:56] <andrewbogott>	 thx all
[18:14:57] <kormat>	 jbond42: are the typos part of what make it safe?
[18:15:04] <jbond42>	 :D
[18:15:38] * andrewbogott will maybe eat before diving back into this.  Not shawarma though
[18:17:45] <rzl>	 cdanis: yeah that doesn't sound real
[18:17:57] <rzl>	 also fair warning, star ratings on google maps etc are super inflated in Tally and I don't know why
[18:18:10] <rzl>	 just because a place has good reviews doesn't actually mean it's any good
[18:18:21] <cdanis>	 well I looked at the food photos, and they were somewhat convincing
[18:20:21] <rzl>	 their falafel doesn't look right IMO
[18:20:26] <rzl>	 I might try it but I'm skeptical
[18:20:59] <cdanis>	 oh I just saw the gyro meat
[18:21:54] <mutante>	 if you order gyros / shawarma / döner but they open a drawer to get the meat instead of cutting it from a rotisserie spit.. then that's a bad start
[18:22:13] <mutante>	 unfortunately happened a lot in California
[18:23:45] <rzl>	 yep
[18:27:52] <jbond42>	 cdanis: with shellcheck i see green, orange and red messages, do you know a switch to get things to just report on the red ones (assuming they are the most critical errors)?
[18:28:51] <cdanis>	 jbond42: I don't; we might have to switch output formats (there's a couple machine-readable ones) and do filtering ourselves
[18:29:18] <cdanis>	 the 'gcc' format might a happy medium of machine/human readable
[18:29:30] <jbond42>	 there is an an exclude option to reject specific tests
[18:29:44] <cdanis>	 yeah, that feels a bit like whack-a-mole though
[18:29:46] <jbond42>	 just looked like internaly they had some catagories
[18:29:49] <jbond42>	 yes
[18:32:20] <jbond42>	 ahh i see with gcc we it outputs nofe and error that should help thanks
[19:05:17] * bd808 now wants döner mit, and tacos al pastor
[19:07:11] <bd808>	 the best shawarma I've had in my WMF adventures was in Jerusalem. But also I was many, many units of alcohol beyond a recommended daily allowance so possibly not a super taster at that point
[20:17:54] <ryankemper>	 Anyone know what username I need to use for netbox? It says `Use LDAP uid field as Username` but I'm very confused about what the LDAP uid is
[20:19:06] <hashar>	 ryankemper: that would be your shell username
[20:20:07] <ryankemper>	 hashar: Awesome! That worked
[20:20:24] <hashar>	 when usually most apps use the 'cn' field (which stands for Common Name)
[20:20:43] <ryankemper>	 I was trying tons of `RKemper`,  `rkemper`, `Ryan Kemper`, `RKemper(WMF)` but not my shell username `ryankemper`, go figure xD
[20:21:25] <hashar>	 if you get access to ldap or bastion.wmflabs.org  you can get infos from there
[20:22:40] <hashar>	 ldapsearch -x uid=hashar
[20:22:41] <hashar>	 ;)
[20:23:26] <ryankemper>	 hashar: That's actually a separate question I have, do I need to do something to enable access to `bastion.wmflabs.org`?
[20:23:41] <ryankemper>	 Using my Gerrit SSH key (so not the public one) and shell username, I get permission denied
[20:23:47] <hashar>	 E_NOT_OUT_OF_BEER_CREDITS
[20:24:53] <hashar>	 hmm
[20:25:14] <hashar>	 so yeah should be a dedicated key, or at least not the same used for production access
[20:25:42] <hashar>	 but SRE have to use bastion-restricted.wmflabs.org  iirc
[20:27:54] <hashar>	 ryankemper: https://wikitech.wikimedia.org/wiki/Production_access#Advanced:_operations_config  and expand the green bar ;)
[20:29:35] <hashar>	 else #wikimedia-cloud-admin might be able to assist
[20:30:27] <ryankemper>	 Thanks, I'll adapt that config
[20:30:42] <ryankemper>	 And yeah when I had the problem a few days ago I was trying to connect to `restricted.bastion`
[20:33:26] <Reedy>	 SSH key into horizon or wikitech should get you onto the labs bastion
[20:38:32] <ryankemper>	 !log disabled puppet on `cloudelastic100[5,6]` which are two racked nodes that we are now bringing into service. Will re-enable after successful puppet-merge / elasticsearch cluster join
[20:38:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:38:42] <ryankemper>	 oops
[20:38:45] <ryankemper>	 Wrong channel
[21:31:40] <ryankemper>	 Another question! Does anyone know how `/etc/ssl/dhparam.pem` gets placed on new hosts? I'm hoping puppet does that somewhere and we're just missing a configuration value or something
[21:32:24] <ryankemper>	 Quick context: We just brought the previously-racked-but-not-in-service-node `cloudelastic1005` into service, but its `nginx` is failing to start because it's looking for an `/etc/ssl/dhparam.pem` which does not exist
[21:32:44] <ryankemper>	 I see on the old servers, `/etc/ssl/dhparam.pem` exists
[21:33:19] <rzl>	 ryankemper: I don't know anything about what you're doing or how it works, but modules/sslcert/manifests/dhparam.pp is a thing that exists in puppet
[21:33:29] <ryankemper>	 That makes two of us!
[21:33:33] <rzl>	 :D
[21:34:28] <ryankemper>	 Ah yes, looks like `modules/sslcert/files/dhparam.pem` is in the puppet repo
[21:34:35] <hashar>	 files in /var/lib/puppet/state sometime gives clues ;D
[21:35:27] <hashar>	 else move the file, run puppet -vt and see which class regenerate the file? (which is probably a bad trick to do on a production host)
[21:35:40] <ryankemper>	 Interesting, `ryankemper@cloudelastic1005:~$ sudo cat /var/lib/puppet/state/state.yaml` does not seem to have a file resource for `dhparam`
[21:36:08] <rzl>	 does it include that sslcert::dhparam class?
[21:37:52] <rzl>	 just to be totally clear, including sslcert::dhparam will install that dhparam.pem you're looking at, but I don't have anything for you as to whether it's the right file
[21:38:15] <ryankemper>	 Well, the file entirely does not exist on the new server so that's actually fine for my purposes
[21:38:23] <ryankemper>	 I'll worry about the content when there is content :D
[21:38:26] <ryankemper>	 https://www.irccloud.com/pastebin/GivPvsfl/
[21:38:37] <ryankemper>	 ^ These are the 4 places in the repo that seem to include that
[22:34:33] <ryankemper>	 ^ Circling back on the above: https://gerrit.wikimedia.org/r/c/operations/puppet/+/602520