[06:27:33] any idea where I can find the "NDA tracking sheet" mentioned in https://phabricator.wikimedia.org/T253676 ? [06:29:39] <_joe_> XioNoX: I did once upon a time but I hoped it moved on. But, Cindy is a WMF employee, and we assume by default they all signed an NDA [06:29:48] ok! [06:30:22] _joe_: do I have to wait for the 3 days for WMF employees too? [06:30:36] <_joe_> no idea! [06:30:45] <_joe_> I stopped following what [06:31:04] <_joe_> s going on with access as soon as it became too many steps [06:31:21] <_joe_> but I would assume so [06:31:58] yep, following the instructions [06:56:30] XioNoX: it's at https://docs.google.com/spreadsheets/d/1xQNx5s2yErvayCMzvk9VkIA2ZihFXSBEhT5Z5ziCsi4, but it's as Giuseppe said, if they are staff, they can be added right away, the gdoc is only needed for researchers and volunteers with an NDA [06:56:44] noted, thanks! [06:56:54] moritzm: should I add it to the doc? [06:57:05] please do! [06:57:13] to double-check whether someone is staff you can either do an ldapsearch -x uid=GMAILID on ldap-corp1001 [06:57:41] or simply follow the staff page (which is also being updated by T&C for departures and people joining) [06:59:17] I do the latter, gmail id is not always on the task [07:00:11] moritzm: is "Currently, you can assign tasks to @[[phab:p/KFrancis|KFrancis]] in phabricator for confirmation." still valid? [07:00:39] which page is that? let me check the context [07:01:18] moritzm: https://wikitech.wikimedia.org/wiki/SRE_Clinic_Duty#Access_requests [07:01:25] under ALL ACCESS REQUESTS REQUIRE AN NDA. [07:02:16] let me update that now and then you can have a quick doublechck whether it's now more clear [07:02:27] ok! [07:06:20] moritzm: can I get a +1 on https://gerrit.wikimedia.org/r/c/operations/puppet/+/599272 as well (I see that you're an auto-reviewer on that file) [07:12:15] +1d, I'll ping you when I'm done with the wikitech page [07:14:03] sure! [07:16:40] updated https://wikitech.wikimedia.org/wiki/SRE_Clinic_Duty#Access_requests, let me know if there's anything unclear left :-) [07:19:41] moritzm: some typoes "If that's not the add to" [07:20:34] but lgtm! thx [07:21:03] fixed the typo [07:23:48] "OSP crews are hands off at this time and all customers have confirmed restored. There will be no additional updates after this one. Thank you." [07:23:55] well, our circuit is still down [07:30:13] https://i.imgur.com/rMXQBp1.gifv [07:33:14] "and all customers have confirmed restored" [07:33:24] I think that's a lie [07:51:08] good morning. I had Doxygen debian package build earlier on and it turns out it tends to crashes when being run under Docker for some repositories [07:51:52] luckily the next version is not affected and Debian packaged it. So I have proposed a change to bump our fork and could use the package to be build / uploaded to apt.wikimedia.org . The merge change is https://gerrit.wikimedia.org/r/#/c/operations/debs/doxygen/+/599094 ;] [08:19:14] arturo: I have pooled a new host on the wikireplicas services, specifically on the analytics role, if you notice people complaining or something, please ping me [08:29:50] ack marostegui [08:36:02] I'm going to upgrade cr2-eqord shortly [08:48:35] mcrouter | 0.41.0-1 | buster-wikimedia | main | amd64 [08:48:40] jbond42: \o/ [08:49:06] sorry I learn things with a lot of lag but thanks for this -^ !! [08:49:25] :) no probs [08:54:33] <_joe_> jbond42: I have an interesting puppet issue [08:54:54] <_joe_> so I ran the puppet compiler on https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/597806/ and it looked good [08:55:23] <_joe_> but in production it seems my code can't find the apr1md5 library [08:55:51] ack let me take a look [08:55:52] <_joe_> is this the dreaded pluginsync horror? [08:56:07] <_joe_> like I need to actually run puppet on the puppetmasters before this works? [08:56:26] i would say so [08:56:54] from a high level 1m look at the patch that would be a good guess [08:59:07] <_joe_> yeah puppet is failing on many hosts though, so I might need to rollback if this doesn't fix the problem [08:59:35] can you give me a host its failing on and have you ran puppet on masters? [09:00:08] running on masters now [09:00:18] jbond42: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?panelId=5&fullscreen&orgId=1 [09:00:39] random one: mw1314.eqiad.wmnet [09:01:01] <_joe_> and yes jbond42 it's pluginsync [09:01:02] thx [09:01:35] <_joe_> if anyone is asking themselves "what the hell is pluginsync", my suggestion is to avoid researching further :P [09:02:06] lol cool, i wasn't 100% sure if it needed to be in a subdirectory i know puppet can be picky about file locations [09:02:27] let me know when it's under control, so I can proceed with my router reboot [09:02:49] ill kick of a run for failed hosts now [09:03:07] <_joe_> I did on two,, it works in both places [09:04:08] yes also worked on 1314, cumin is working through the other failed ones now [09:05:18] well that was quick when you said you had a strange puppet issue i expected to be down a rabbit hole for most the day ;) [09:05:51] hahaha [09:05:57] <_joe_> I frankly hoped it was just pluginsync being pluginsync [09:06:04] I'll resume my upgrades [09:06:07] yes im glad it was :) [09:06:22] yes puppet has ran on all the failed ones now [09:06:38] <_joe_> jbond42: otoh, we need to do this for basically every case where we declare some class in the same file as a function [09:06:49] <_joe_> I found out the class is *redeclared* every time you use the function [09:07:08] <_joe_> because the ruby code gets just included [09:07:24] <_joe_> so any case where we try to use memoization/caching just doesn't work [09:07:32] <_joe_> e.g. the ipresolve() function [09:07:50] <_joe_> I *think* this changed compared to puppet 3.8 [09:09:06] yes i saw your comment before but im not entierly familure with ipresolve however that is how i understand the behaviour i.e. there is an object cache just for the period of the catalouge compilation. i have very little experience on puppet < 4 so not sure about that [09:09:39] <_joe_> so turns out that it doesn't work like that [09:09:57] so not every time you use the function neccesarily but every time yu compile the catalouge, could be wrong on that, oh ok? [09:10:00] <_joe_> if you call ipresolve() 3 times in a row, the whole DNSCache class gets redefined [09:10:12] <_joe_> I have to confirm [09:10:22] <_joe_> but that was def. the case with htpasswd() [09:11:09] i have to admit i have never needed to debug this and im the stuff i rad was high level docs and im not sure how long ago so definetly could be wrong. [09:11:22] <_joe_> yeah I was sure as well [09:11:46] <_joe_> hence my suprise in getting warnings from ruby that some class constant was being defined again [09:12:08] ill see if i can dig into it tomorrow but you know what its like trying to find out stuff like this in puppet especially as it can change between minor version :@ [09:12:24] but please let me know if you find more [13:26:50] ema: looks like the debian glue job is broken :/ [13:27:19] the jenkins plugin that looks at the xunit file does not recognizes it anymore bah [13:27:59] hashar: was it upgraded/changed recently? [13:29:41] yeah the jenkins plugin [13:29:43] will roll it back [13:32:52] thanks! [13:34:05] stupid plugins ;D [13:55:25] ema: it worked :) [13:55:37] hashar: indeed! ty :) [14:30:30] mc1030 had some trouble, tx bandwitdh saturation [14:32:26] the gutter has worked as expected afaics [14:32:53] there is an increase in get traffic but nothing horrible afaics [14:33:11] I will not go into the rabbit hole of the keys etc.. seems not needed [14:40:55] <_joe_> elukey: you seem utterly satisfied with that [14:42:16] speaking of [14:42:18] _joe_ I have to try the new mw metrics for key size, IIRC we added some and it would be great to see if we can spot anomalies from there.. doing it manually is a little bit frustrating :D [14:42:26] SRE topic this coming monday? [14:43:15] elukey & _joe_ that is [14:43:28] (although I think _joe_ you said you are skipping) [14:43:28] <_joe_> paravoid: as I said, I'm off on monday [14:43:56] yes, the question really is if you want that to happen without you or if you'd rather propose it for the meeting after that :) [14:45:47] paravoid: I am in the Analytics team, I don't know anything about memcached, I'd ask to the service-ops team :D [14:46:03] hahaha [14:46:54] <_joe_> I'm very happy to let luca make the honours ofc [14:47:05] <_joe_> but he doesn't seem so happy about that [14:47:17] jokes aside, it is something that also Effie would be probably happy to present when back [14:47:50] I don't particularly love presentations :D [14:53:09] ok [14:53:53] I don't want to raise the bar too high in terms of preparedness or presentation material, I'd just like to spend half an hour every two weeks disseminating information, building bridges and talking about cool things [14:55:45] <_joe_> elukey: you can just evolve this schema https://people.wikimedia.org/~oblivian/arrow-hell.png to include also the gutter pool [14:58:00] :O [15:01:08] _joe_: i took a quick look at the class var think in puppet and it seems at least when using puppet apply class vars are kept for the entire compilation, although my brain is feeling a bit numb after navigating the Gsuite, service account, api key madness https://gist.github.com/b4ldr/a63a95399ef6b6afd1df620124f19195 [15:01:29] ...so i may have made an obvious mistake or missed someting [15:23:02] elukey: shall I merge 'Add page_restrictions to analytics sqooped tables'? [15:24:15] addshore: ooof, yes please, sorry [15:24:42] uff sorr addshore, difficult day [15:24:48] andrewbogott: ooof, yes please, sorry [15:24:49] :D [15:26:21] :D [15:26:30] not a difficult day for me then? fewwwwww [15:36:56] <_joe_> nah it seems correct [15:37:04] <_joe_> so [16:06:42] paravoid: I can prep something for monday :) [16:06:53] 20 mins more or less right? [16:07:08] yeah [16:07:35] ack [16:08:58] are you not off on monday? [16:26:20] paravoid: I am not, will be off Tue [16:32:01] oh thanks for merging that through andrew bogott, that's great