[06:44:25] hello hello, going to check the AQS alarms in a bit [11:01:23] XioNoX: what's the policy/procedure re: diffscan? searching on wikitech doesn't show anything (context: my new service showed up in the diffscan email a few minutes ago) [11:01:28] is there something i'm supposed to do? [11:01:55] kormat: no, it just registered that there is a new open service (host:port) [11:02:07] and if it's not intended alerts us to hve a look [11:02:16] and see if maybe some misconfiguration was deployed [11:02:32] in this case it looks ok as you just created a new service that is listening there AFAIK [11:02:48] so i'm not supposed to reply to the email? [11:03:05] kormat: nop [11:03:22] kormat: well, depends [11:03:25] ok. i guess everyone else will just assume it's legit? [11:03:47] kormat: or you can reply to root saying that it's normal if you think some people could get confused [11:08:35] <_joe_> kormat: are you saying it would be nice to have a way to let diffscan know that a port is expected to be open explicitly? [11:09:12] <_joe_> or you're just proposing people answer the email specifying "it's my service, nothing to see" so that noone else goes on and needs to check? [11:10:05] _joe_: i just ignore the emails as they seem to be noise. the only reason i knew this one was relevant to me was because volans mentioned it. if this is expected, then carry on [12:25:28] heads up, I noticed icinga check latency has gone up significantly, going to restart icinga in 5 min [14:17:49] kormat: they aren't noise, but they aren't not noisy :) [14:18:39] cdanis: uh huh :) [14:18:40] signal/noise has been pretty good for the people who care about them :) [14:19:03] the expected normal case of diffscan is empty [14:19:05] (of course it's subjective) [14:19:30] cdanis: but it shows that the tool didn't crash! [14:19:33] :) [14:19:43] 'empty' meaning "successful, no changes", yes [14:19:44] :) [14:38:34] Hi, anyone knows if homer diffs need to be manually deployed after merging on gerrit? The wiki seems to suggests some 12h check with email and such, we would appreciate something a bit faster though :/ [14:39:09] dcaro: yes, no automated deploy with homer is done. There is a 12h check to check for diffs with prod, and reports them [14:39:12] dcaro: yeah, you need to `homer commit` like the examples at https://wikitech.wikimedia.org/wiki/Homer#Running_Homer_from_cumin_hosts_(recommended) [14:39:27] so if the data is changed you need to deploy the changes to the affected devices [14:39:56] or check with our team of network engineer(s) ;) [14:40:12] hello [14:40:16] ack [14:40:17] :P [14:41:46] dcaro: we can follow along if you need any specific guidance ofc [14:43:36] left a message on -cloud-admins :) [17:25:25] heads up re: T272258 (lvs1015 network port issue) - will be stopping puppet+pybal on lvs1015 in ~25 minutes or so for hardware maint. This will fail over traffic to lvs1016 for internal services in eqiad. [17:25:26] T272258: lvs1015 interface errors - https://phabricator.wikimedia.org/T272258 [19:35:20] if you create new virtual disks in ganeti.. the progress is like 80% -> 20% -> 70% -> 10% ... - DONE :) weird, maybe it's for each node.. but looks weird. but works in the end so ..ok [19:45:47] we had multiple requests for LDAP group changes with missing information [19:46:04] I updated Engineering's handbook (with their permission) [19:46:17] and tried to update our description so it is more user-friendly [19:46:32] please feel free to improve upon my changes: https://phabricator.wikimedia.org/project/manage/1564/ [19:56:29] if that is where the "as part of my onboarding" stuff always came from.. thank you! [19:57:29] your edit looks good [20:00:42] mutante, it came from here: https://office.wikimedia.org/w/index.php?title=Guide_for_new_engineering_staff&diff=prev&oldid=284690 [20:01:12] I would link to the new template directly, but it will get outdated soon if I di [20:01:44] jynus: ACK. the part I did not like in the past was when they were just saying "as part of onboarding" but never had a reason what they _actually_ need. but that has been fixed now. so cool [20:02:23] so reviewing our guide, we don't require manager's ok for wmf group [20:02:32] it made it sound like membership in "wmf" is a flag that indicates "is employee" and is default for all, which it is not [20:02:53] jynus: oh. in that case, +1 and just merge it [20:03:02] but I wonder if there is a chance an unrelated account claims to be a wmf employee [20:03:05] everything else looked good and i checked it.. even corp LDAP [20:03:11] employee status [20:03:20] yes, of the actual employee [20:03:23] they could not fake that [20:03:29] mutante: the group should not be named that, then [20:03:43] cdanis: yes, i agree [20:03:53] but how do you trust the ldap and corp-ldap are the same person? [20:04:02] actual employee check: [ldap-corp1001:~] $ /usr/bin/ldapsearch -x "mail=thand*" | grep -E 'employee|mail|manager' [20:04:13] yes, that part I know [20:04:28] but what stops me from claiming I am your corp account? [20:04:51] and in in reality I am Mutante's fake wikitech account [20:05:01] see where I am going? [20:05:04] jynus: the way to verify a phab user is theoretically to follow back to the linked Mediawiki account and then who created that account and if that was OIT/ITS and it has "WMF" in the name then it must be real. regular users create their own users and can't use "WMF" string [20:05:14] Could you not look for a verified email etc? [20:05:37] I can check linked mail, but would it show as verified or not on the ldap? [20:06:06] just to be clear, I am tryint to poke a hole on the procedure [20:06:17] jynus: are you describing https://phabricator.wikimedia.org/T259746 ? [20:06:20] or more accurately, makins sure the procedure is complete [20:07:29] you can check if the email address used is the corporate @wikimeedia.org i nboth wikitech and corp LDAP, yea [20:07:56] they would have to be able to click a verification link when creating it, right? [20:08:11] but they could create it and then change email? [20:10:47] jynus: if they have wikitech or MediaWiki then there's a confirmed timestamp [20:10:59] Like MediaWiki as in normal sul [23:41:19] tried to reboot a VM.. was supposed to be "super quick and harmless". NOPE, did not come back, could not SSH to it, see on console it is actually up, login via console.. realize it's because there are no iptables rules, which is because ferm failed, which is because a DNS lookup for backup1001 failed.... grrrrmbl [23:41:53] and that host is also still there [23:46:39] it can't reach the nameserver because Network is unreachable, did not get an IP on the interface...