[03:40:10] Anyone around for a quick +2 on https://gerrit.wikimedia.org/r/c/operations/puppet/+/687928/? [06:17:51] dpifke: o/ I checked the code review but I am a little confused, why is redis mentioned ? [06:18:24] I don't see Redis running on mwlog1002 (caffeine levels still low, apologies if I am missing something clear :D) [06:24:55] <_joe_> elukey: I can take a look [06:28:40] <_joe_> elukey: you don't see redis running on mwlog1002? [06:28:49] <_joe_> yeah go get coffee :P [06:29:49] ahahha yes it is the first, now I see it [06:30:15] seems easy enough, I can take it, I was reviewing the other alerts [10:33:18] Hey folks nice to be here! [10:33:31] I'm Cathal just started as an SRE within Infrastructure Foundations. [10:34:02] Hello to those of you I met during the invterview process and look forward to meeting those I didn't :) [10:34:23] welcome :-) (Moritz, we haven't met during the interview process) [10:34:25] Excited to be here, Murphy's law there is some issue with my LDAP creds but I will hopefully get that sorted soon. [10:35:11] Hi Moritz nice to meet you. Faidon tells me you're the Debian security don :) [10:38:00] ping me if you need help with the LDAP credentials, happy to have a look! [10:38:01] topranks: welcome (john, we did meet during the interview process :)) [10:38:58] Hey John yes hope all is good with you. [10:39:11] Plenty of interesting things going on from what I can make out. [10:40:07] yes all good here great to have you on board [10:40:55] topranks: Hello! (Luca, Machine Learning / Analytics) [10:44:47] elukey: Hey Luca thanks :) [10:48:50] Cathal will be focusing on networking, to be precise :) Welcome! [10:49:37] XioNoX: thanks! [10:50:47] and yes I am a network guy first and foremost, was working with Akamai until recently. [10:56:45] hello, topranks, and welcome. Which continent will you be working from? [10:58:53] topranks: welcome! [10:59:35] topranks: how should I read your nick: top ranks or to pranks? :) [10:59:58] top ranks will do for now... but yes watch out for the pranks haha [11:00:07] marostegui: nice to meet you :) [11:00:19] jynus: nice to meet you [11:00:38] i am based in Dublin, Ireland which is where I grew up [11:01:02] so "the edge of Europe" is the answer I guess :) [11:01:52] cool, topranks, Spain here. I will be the person to recover the things you accidentally delete :-) [11:02:10] ok.... I may need you on speed-dial in that case :-) [11:24:17] topranks: Welcome to the team! (/me Alex, based on the opposite side of Europe (Greece) from you) [11:25:57] akosiaris: Thanks! cool there are a lot of EU people here greetings from Dublin :) [11:52:59] effie: herron: xenon in mw wmf-config is the name for arclamp redis (actively used). Looks like the mwlogx002 update resulted in codfw no longer pointing at eqiad. That may need fixing if not (yet) intentional/planned/tested as such. The samples afaik are not active active yet. [12:18:13] Krinkle: can we do something to make this more evident ? [12:20:38] I will ping you later about it as I have to dash [12:51:18] volans: having cumin prompt for the number of hosts (a) is a good idea, (b) but hate it. :) [12:51:42] *but i hate it [12:53:05] kormat: are you testing the bullseye host cumin2002? [12:53:14] is not yet deployed to the official ones ;) [12:53:15] volans: nah this is in pontoon [12:53:20] ah [12:53:38] dunno why it ended up there if not officially released, though [12:54:23] the deb in apt.w.o [12:54:26] *is in [12:55:01] ahh, right. wmcs hosts auto-upgrade packages. this has bitten me before. [12:55:04] and it's good to go,I'm doing some more tests on cumin2002 to be on the safe side and will roll it in the next days [12:55:29] there are also other features [12:56:39] https://doc.wikimedia.org/cumin/master/release.html#v4-1-0-2021-05-03 if you're interested [12:56:48] volans: i saw - it finally has the one i've been whining for for aaages :D [12:56:58] (dry-run mode outputting hosts to stdout) [12:59:58] <_joe_> kormat: so you want the --bblack flag added to cumin [13:17:17] jbond42: our cloud-vps internal cumin servers have broken puppet due to [13:17:18] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Function lookup() did not find a value for the name 'profile::pki::client::auth_key' (file: /etc/puppet/modules/profile/manifests/pki/client.pp, line: 7) on node cloud-cumin-01.cloudinfra.eqiad.wmflabs [13:17:36] can you advise? Do I just need to add a dummy key somewhere? [13:18:31] andrewbogott: is labs/private up to date? https://github.com/wikimedia/labs-private/commit/1386c96c223c2d30e94246af1c601b5b62cbe5fd should have solved that [13:18:58] andrewbogott: yes like Majavah i thought that was fixed [13:19:05] checking [13:20:15] hm, merge conflicts on my puppetmaster, that's likely it [13:20:51] we should send nag emails for operations/puppet and labs/private merge conflicts like we do for failing puppet runs [13:22:38] yeah, better now, thanks all [13:24:17] great :) [13:25:57] Krinkle effie uploaded https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/688281/ for this [13:45:31] <_joe_> Majavah / jbond42 I just wrote https://gerrit.wikimedia.org/r/c/operations/puppet/+/688315 as an alternative approach at solving the services_proxy issue [13:45:50] <_joe_> now you can just declare an empty list of listeners and the profile will be effectively a noop [13:46:06] lol beet me https://gerrit.wikimedia.org/r/c/operations/puppet/+/688312/1 [13:46:47] <_joe_> yeah my change is smaller and is actually operating as intended I think in this case [13:47:06] <_joe_> although I am a big fan of not reusing production roles in beta, but rather assemble profiles [13:48:14] _joe_: i try not to touch beta tbh :) ill leave Majavah to comment if that ifxes the initial issue (i think it dose but dont rember) [13:49:13] ill abandon both of mine for now eitherway [14:38:25] since we're talking about deployment-prep breakage... there's also this https://gerrit.wikimedia.org/r/c/operations/puppet/+/684814 -- is that somehow the same issue as the one addressed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/688315 ? [14:40:53] _joe_: yeah, that should fix the services proxy issue. if there's a better way to approach it I can look at that too [14:41:10] andrewbogott: those are separate issues :/ [14:42:27] andrewbogott: in the comment you said "Because we don't have ipv4 on cloud-vps" im gussing you meant ipv6? [14:48:15] <_joe_> Majavah: we should probably maintain a service::catalog hiera key for beta too [14:49:03] _joe_: we can try, but I think the data structure gets annoying since there is no lvs on beta [14:52:35] jbond42: yes! [16:03:49] topranks: welcome! (Kunal, serviceops) [16:08:12] logoktm: thanks Kunal pleased to meet you :) [16:52:23] moritzm: base-files still not merged into bullseye? [16:59:01] andrewbogott: dosn;t look like it [16:59:02] cat /etc/debian_version [16:59:02] bullseye/sid [17:00:12] seems way overdue at this point, but... "when it's ready" I guess [17:04:36] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=988276 [17:06:11] Majavah: is that just someone else also wanting that merged? I'm not at all familiar with the cr process there so can't really understand that bug. [17:07:35] andrewbogott: you can replace the string with a number and run puppet again and there should be no difference [17:07:46] andrewbogott: that's the maintainer of the base-files package asking the release team to manually allow the migration from unstable to testing [17:12:29] mutante: thanks -- I'm using their daily builds as part of a multi-step build process so it isn't easy for me to change a file midway through [17:12:47] ah, *nod* [17:12:50] I can just wait -- originally this was expected on the 1st I think so I just assumed it would work. [17:15:51] yeah, what Majavah said, needs a review by the release team as filed in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=988276 [17:15:56] probably done by tomorrow [19:52:38] I have https://gerrit.wikimedia.org/r/688394 which will skip sending cross-validate-accounts emails on the weekend, given they aren't urgent enough to handle outside of working hours -- any objections? [19:53:07] (any problems that come up on a Saturday will still be there in the Monday email, so we don't lose any signal, we just deduplicate) [22:36:45] rzl: I like it. That is never so urgent, we would always do that on the following work day [22:37:11] at the earliest