[06:39:19] can I install mtr in one of esams cp hosts? [06:43:35] nevermind, I will use the one on the bastion [07:09:23] and I just realized I am not savy enough to force mtr through a particular carrier [12:13:28] that needs to be done on the router, catching up on my email backlog :) [12:13:49] more exactly `traceroute XXX source YYY interface ZZZ` [12:16:24] I assumed so, too scary for me :-D [18:41:31] Can I borrow an SRE for a moment to clean up /srv/docroot 's git ownership on doc1001? I can't chown it back to nobody… [18:41:49] yeah I can take a peek [18:42:41] Thanks. [18:43:05] is it supposed to be "nobody"? that seems odd :) [18:43:22] I mean, I don't know, it seems odd to me too, but hashar isn't around to ask. [18:43:27] I assume it's basically public data right? [18:43:37] Yeah, all public. [18:43:50] The git repo is public and the contents are live on https://doc.wikimedia.org/ [18:44:39] done [18:44:45] Thank you! [18:48:34] ticket related to doc1001 permission issues https://phabricator.wikimedia.org/T237707 [18:48:42] it looked resolved but wasn't [18:51:20] Possibly `nobody` was also the wrong answer? [18:53:46] from: https://gerrit.wikimedia.org/r/c/operations/puppet/+/484304 [18:53:47] sudo -u doc-uploader chmod g+w /srv/docroot/org/wikimedia/doc [18:53:53] "The manual fix:" [18:55:32] i ran that to let wikidev group write to it [18:55:42] deployers are in wikidev group [19:00:03] `sudo -u doc-uploader git -C /srv/docroot/ pull` fails with `error: cannot open .git/FETCH_HEAD: Permission denied` for me. [19:04:52] hmm.. doc-uploader isn't in wikidev. [19:05:11] i guess the docs changed to .. fix this ..looks [19:05:19] heh [19:07:34] the user doc-uploader only has group doc-uploader, not wikidev [19:08:32] the puppetization does have the clone as nobody:wikidev though [19:09:04] yea, this is part of the bug i guess [19:09:05] It's probably intentionally different [19:09:14] and doc-uploader was introduced later [19:09:40] hmmm [19:09:47] there's also /srv/docroot/org/wikimedia/doc [19:09:54] in puppetization, with doc-uploader:wikidev as owner [19:09:58] curiouser and curiouser [19:10:00] Hmm. [19:10:20] So both can pull, and so it breaks when someone (hi!) does the pull from the wrong side? [19:10:35] seems like an odd smell, that there's puppetization of file resources underneath a puppetized git::clone destination [19:10:46] yea, so the 'manual fix' i ran applies to /srv/docroot/org/wikimedia/doc [19:10:50] James_F: yes [19:11:36] but the git pull command pasted by James_F above still fails the same way [19:11:40] we could put doc-uploader into wikidev group (like human users) or break it for human users who don not use doc-uploader ? [19:11:49] Let's break it for human users. [19:11:57] Make people do the 'right' thing. [19:12:00] last time on the ticket it was about running it as a human deployer user [19:12:01] does doc-uploader even belong in wikidev though? [19:12:13] wikidev has some broad access rights all around I think? [19:12:19] Probably a good idea to not give wikidev. [19:12:25] yea, wikidev is the default group for all shell users [19:12:38] so doc-uploader:nobody? [19:13:16] doc-uploader:doc-uploader is also possible [19:13:21] really nothing should be intentionally set to nobody [19:13:24] that's the whole point of nobody [19:14:19] chown -R doc-uploader:doc-uploader /srv/docroot ? [19:14:50] doesn't seem harmful, not sure if it will help [19:15:08] note the puppetization will change the group ownership of that one subdir back to wikidev on next run [19:15:21] ( /srv/docroot/org/wikimedia/doc ) [19:16:20] but in general this thing seems either broken by design, or it's that the design of the ownership/layout conflicts with the instructions on how to use it (or at least someone's idea of what the instructions are) [19:16:38] it probably needs more than yet another manual perms bugfix, in the long run [19:17:11] yea, ACK. that ticket is already reopened [19:17:18] i'll look at making a puppet change [19:17:28] ok awesome [19:18:07] Thanks, both. [19:20:54] James_F: i ran the pull command as doc-uploader and says Already up-to-date [19:21:29] Excellent. [19:21:40] So, working perfectly until puppet re-runs? [19:21:47] i disabled it for a moment [19:23:15] * James_F nods. [19:27:32] the git::clone in puppet uses "shared => true" (Enable git's core.sharedRepository=group setting for sharing the repository between several users, default: false" [19:27:43] but we should probably stop doing that then [19:28:27] if now we use doc-uploader instead and want to avoid what you described above (both can clone ...) [20:34:29] a second change .. there is more since wikidev was also used as the group for rsync. TLDR i think is we should just stop trying to use wikidev at all [20:34:54] and if all users sudo -u doc-uploader we don't need all the stuff for multi-user [20:35:19] (there is another puppet class just to change umask for wikidev to have it own newly generated files) [20:44:34] jaufrecht: I saw some recent edits from you on wikitech giving links to phab and instructions on how to fill out the task once you are there. You might find https://tools.wmflabs.org/phabulous/ useful for making "deep links" to phab task creation that put notes in the form itself for people to use. [20:45:10] bd808: thx [20:46:21] hypothesis: we have tools to do more things than we have people who know where all the tools are hiding [21:06:47] James_F: well.. i hope that's it. summarized here https://phabricator.wikimedia.org/T237707#5846006 all members of "contint-admins" can deploy with sudo -u doc-uploader. it's just an issue if we had doc deployers who are not contint-admins [21:08:07] Right. But I think only contint-admins and SREs should be able to ssh in anyway? [21:09:50] James_F: looking at /etc/ssh/userkeys.. yes. root and contint-admins should be all [21:10:14] In that case, that works for me. Thank you! [21:10:24] now we just need to change docs if they don't mention sudo [21:10:28] yw [21:12:57] "stop using wikidev" is probably a good thing in general [21:13:12] The docs say to sudo to doc-uploader. [21:13:19] But I'll do another audit to make sure. [21:13:20] ok, great [21:13:23] thanks [21:14:09] just adjusted reality to existing docs then. i like it [21:18:07] mutante: Faidon might put you in for some sort of SRE of the month/year award if you figure out how to "stop using wikidev" everywhere. ;) That grab-bag group has roots going all the way back to NFS in prod times. [21:19:30] bd808: heh, yea, that's REALLy old [23:30:00] ganeti VMs are not happy that there is no floppy disk drive at boot, hehe. it doesn't hurt but kind of funny. "floppy: error 10 while reading block 0" [23:30:48] I wonder how many decades that assumed boot target will still be around in pretty much everything. :-) [23:41:40] It amuses me that vmware kit still creates a floppy drive by default too [23:42:06] I see references to floppies and modems in random containers I run