[00:18:26] !log account-creation-assistance Unable to SSH in to accounts-database, issuing a reboot [00:18:28] Logged the message, Master [00:23:03] !log account-creation-assistance `apt-get update && apt-get upgrade -y` on all accounts-* [00:23:05] Logged the message, Master [00:24:17] !log account-creation-assistance accounts-application requesting reboot, fulfilling [00:24:18] Logged the message, Master [04:14:28] !log reportcard updated puppetmaster::self repo to head and force-runned puppet [04:14:29] Logged the message, Master [05:29:53] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Needed Toolserver features was modified, changed by MZMcBride link https://www.mediawiki.org/w/index.php?diff=667368 edit summary: [+38] /* Links */ +1 [07:56:00] [bz] (RESOLVED - created by: Antoine "hashar" Musso, priority: Low - enhancement) [Bug 46104] reduce the number of wiki on beta - https://bugzilla.wikimedia.org/show_bug.cgi?id=46104 [08:47:53] !ping [08:47:54] pong [08:49:20] petan: pm [13:34:39] @notify petan [13:34:39] This user is now online in #huggle so I will let you know when they show some activity (talk etc) [14:08:46] hi Coren :P [14:08:54] Heya. [15:54:04] !log tools giving sudo to Petrb in order to update qdisplay [15:54:08] Logged the message, Master [15:55:19] !log patched /usr/local/bin/qdisplay so that it can display jobs per node properly [15:55:20] patched is not a valid project. [15:55:24] !log tools patched /usr/local/bin/qdisplay so that it can display jobs per node properly [15:55:26] Logged the message, Master [15:59:36] Coren and what about the logging from terminal? should we have something like "log" command in bots? [16:00:28] petan: I don't feel strongly about it either way; I'm always on irc so using !log here is just as easy for me. Do you think it useful? [16:00:44] Coren, do you have cycles to talk about service groups a bit? [16:01:18] andrewbogott: If you give me 5, I'm all yours after that. [16:01:28] I don't really care either but I find it myself useful, because I am lazy :) I don't need to switch windows while I am in terminal and when I want to log some command I just did, I can scroll in history and prefix it with log just to log it to sal [16:01:30] ok. Time for me to make a cup of tea [16:01:44] I don't know about others [16:01:52] if it was just up to me I can install it to ~/bin [16:02:01] I don't know if other users want it or not... [16:02:23] I can search SAL to see how many users except for me were using it [16:04:18] !log tools petrb: installed log to /home/petrb/bin/ and testing it [16:04:20] Logged the message, Master [16:07:46] andrewbogott: Actually, I should probably nab a quick lunch; do you mind if I delay you my 15-20? [16:07:57] np [16:08:00] brb [16:29:38] * Coren is back. [16:29:42] andrewbogott: I'm all yours. [16:31:10] ok. firstly… I'm ready to turn on the service group code on wikitech whenever you are available to stand by and test it. [16:32:30] I'm standing by for you! [16:32:32] secondly… I'd like to set up a chown sudo rule and make sure it does what you're expecting. [16:33:12] Have you already written the code you need to implement the service groups/users out of ldap? (I'm not clear on what's needed on the instance level. Maybe nothing...) [16:34:28] andrewbogott: There isn't much to do on my side, mostly create directories and check a couple of permissions; I'll be able to adapt my existing creation code for it; I wanted to have actual service users in LDAP to work on it (which doubles as a happy fun test) [16:34:41] ok. So, here comes the merge. [16:35:35] Yeay. I see service groups. :-) [16:35:36] ok, the service group GUI is live on wikitech. Hopefully it's obvious how to operate. [16:36:18] It behaves in exactly the obvious way. [16:36:29] * Coren looks at the LDAPness. [16:38:57] Ah, right, we need to merge the addition to nslcd.conf first [16:40:17] What's the basedn of the per-project ou again? [16:40:18] I don't think I know what that is :) Is there a patch already written someplace? [16:40:33] andrewbogott: No, I'll make one once I test it live in 30s. :-) [16:41:14] ou=groups,cn=,ou=projects,dc=wikimedia,dc=org [16:41:35] and ou=people,cn=,ou=projects,dc=wikimedia,dc=org [16:42:57] I haz a suksess! [16:43:01] root@tools-login:/etc/ldap# id local-test [16:43:01] uid=40000(local-test) gid=40000(local-test) groups=40000(local-test) [16:43:01] root@tools-login:/etc/ldap# id marc [16:43:02] uid=2138(marc) gid=500(wikidev) groups=4(adm),8000(local-sample),8010(local-csbot),50062(project-bastion),50064(project-bots),50082(project-webtools),50380(project-tools),40000(local-test),500(wikidev) [16:43:18] cool [16:44:44] puppet is going to trample that away shortly, but I'll write a changeset for that. [16:45:58] andrewbogott: Do we have a puppet variable containing the project name automatically that you know of? [16:46:27] Hm… not sure. I think so. [16:46:28] * andrewbogott looks [16:49:34] Coren, looks like it is just $instanceproject. Does that work? [16:51:36] That'll work. I just found it myself in ldap/autofs.default.erb [16:59:18] git review is sooooo slow. [17:05:38] andrewbogott: https://gerrit.wikimedia.org/r/#/c/57082/ [17:09:01] Doesnt' that break existing uses of passwd, shadow, group? [17:10:30] No, it adds a new place to /also/ search, the result is the concatenation of all result sets. [17:10:43] * andrewbogott looks again [17:11:27] per nslcd.conf(5): [17:11:36] This option may be supplied multiple times and all specified bases will be searched. [17:11:56] Oh, ok. Weird! [17:12:23] No, it make sense - lots of orgs have users under more than one OU [17:13:53] Makes sense that it would take a list, but… I guess I'm not used to that kind of config format. [17:13:55] Anyway… merged :) [17:18:52] And we now have service groups. Now to plan a transition outage for tools to move the ad-hoc service groups to the new scheme and we be all set. Thanks a bundle, andrew. [17:23:12] Coren: So if I'm developing a new tool that will hopefully be developed by multiple users, should I request a new one? [17:23:44] legoktm: No, no, I'll convert the "ad-hoc" local tool users to the new ones transparently. [17:24:26] Er, so I should just set it up in legobot? [17:24:44] I want other users (when they volunteer) to be able to login and do stuff with it [17:24:52] Kinda like toolserver's MMP [17:24:58] ... wait, I think I misunderstood you now. You mean create a /different/ tool than legobot? [17:25:07] Basically [17:25:20] Because legobot is already setup to be multi-maintainer; but I can add a new one for you. :-) [17:25:32] Oh. [17:25:44] Well I have password files in legobot so I'd rather not share that with people... [17:26:16] legoktm: Well, you're the only maintainer until /you/ add more. :-) [17:26:25] Right [17:27:17] Hmm, I'll go with a new tool [17:27:23] Hopefully I can convince people to help me :/ [17:27:47] named "matilda" [17:30:37] legoktm: Give me a minute while I debug the new scheme. :-) [17:30:45] Sure [17:31:02] My code won't be ready until tonight, so no rush [17:31:31] !andrewbogott [17:31:37] yep [17:32:03] Can you make a manual change in LDAP for me? The home of local-matilda is set the the default; I forgot to change the pattern first. :-/ [17:32:26] sure. You want it changed to... [17:33:07] Coren, or, would it work to delete/recreate the group? That would test another code path [17:33:18] andrewbogott: Will I get the same uid back? [17:33:24] hm, nope. [17:33:43] Then /data/project/matilda if you please. I'll try the code path with my test group instead. :-) [17:33:49] ok [17:34:10] (stupid kvirc, ^W is werase not close-the-effing-window) [17:35:37] andrewbogott: delete/recreate did work with a different uid and the right home pattern. [17:35:51] great. [17:36:08] The code doesn't recycle uids, so we'll run out in 2085 [17:38:37] andrewbogott: Tell me when the LDAP entry has been changed? [17:39:16] yep, will take me a couple minutes to remember how to do this by hand [17:42:29] legoktm: FYI, you can manage the maintainers of the project through https://wikitech.wikimedia.org/wiki/Special:NovaProject [17:42:35] s/project/tool/ [17:42:52] oh [17:42:59] i see matilda but not legobot? [17:43:15] legoktm: That's because legobot hasn't been moved over to that new scheme yet. [17:43:20] gotcha [17:43:54] Coren, should be fixed now. [17:44:13] andrewbogott: Ayup. Danke [17:44:35] legoktm: And matilda should be all set for you too as a consequence. [17:44:41] great [17:45:46] legoktm: Actually, if you have no running job with legobot atm, I can move it over now. [17:46:04] lemme see if it finished what i started last night [17:46:09] kk [17:46:10] it had around 20k edits to make ;) [17:46:44] gah, it crashed [17:46:52] so no, nothings running right now [17:46:52] Aw. :-( [17:46:52] Coren: OK, so en route to automating this I want to set up the chmod sudo rule by hand for a service user. Is matilda a good candidate for that? [17:47:10] andrewbogott: Would be perfect. [17:48:26] ok, temporarily making myself a tools project admin [17:49:33] suddenly sudo appears to not be working on staging.wmflabs.org [17:49:54] all users typically with sudo are reporting seeing ;'awjrichards is not allowed to run sudo on staging. This incident will be reported.' [17:50:04] anyone know what might be going on? ^ [17:51:20] Coren: Hm, can't easily add it for the actual matilda user. Can I set it for you instead, and then remove your other sudo rights so we can test? [17:51:41] andrewbogott: ^^ [17:51:42] andrewbogott: No need to remove my other rights, sudo -l will enumerate the specific ones I get. [17:51:57] awjr: Nothing I'm doing. [17:52:16] Coren: OK… my specific question is whether the long rambling command survives quoting and such via ldap. So I want to verify that it really works, not just that it looks right [17:52:25] awjr: I can take a look in a few minutes [17:52:32] awesome thanks andrewbogott [17:52:36] andrewbogott: okay. Go ahead. [17:53:09] Coren: Thanks. Test now? [17:53:44] (root) NOPASSWD: chown -R local-sg10sgtest:local-sg10sgtest /data/project/local-sg10sgtest [17:54:02] Oops, copy/paste dumbness [17:54:23] Although that sudo rule would really need to use the actual configured home, not arbitrarily /data/project/foo :-) [17:54:45] Yeah, I just thought of that. [17:54:46] * andrewbogott makes note [17:55:26] OK, how 'bout now? [17:56:39] (root) NOPASSWD: chown -R local-matilda:local-matilda /data/project/matilda [17:56:47] This is full of joy. [17:57:01] OK, I'll fix that patch to use the actual homedir [17:57:17] um… after figuring out what's going on with awjr. [17:57:39] andrewbogott: Please to give me my normal sudo back? :-) [17:57:45] yep, done [17:57:53] Thanks. [17:58:44] awjr, what project and instance are we talking about? [17:59:54] mobile/staging andrewbogott [18:00:42] i haven't even hopped on there in a while, but i was starting to investigate some other weirdness - the MobileFrontend extension dir in our webroot is mysteriously empty [18:02:49] legoktm: legobot has been transitionned, but you will need to log out and back again before you can get to it. [18:03:37] will do [18:03:48] awjr: Did I change anything? [18:03:59] * awjr looks [18:04:38] no andrewbogott, and im still being prompted for pw (i think previously it was passwordless sudo) [18:05:01] yep, something is clearly broken. Can you check a different instance in the same project? [18:05:39] same problem on mobile-varnish [18:05:45] dang [18:06:11] andrewbogott: but mysteriously OK on mobile-osm2 [18:06:22] oh! curiouser [18:07:32] Coren: [18:07:33] Unable to run job: warning: local-legobot your job is not allowed to run in any queue [18:07:34] denied: no matching department for user "local-legobot" or group "local-legobot". [18:07:34] Exiting. [18:07:49] mobile-solr3, mobile-solr2 and mobile-b2g are all behaving fine as well, andrewbogott [18:08:03] I wonder how I restart the sudo service on an instance that I don't have sudo on... [18:08:14] lol [18:08:16] legoktm: Aw, crap. Forgot about that. Gimme a minute. [18:08:40] awjr, how would you feel about cycling power on staging? Will that mess with people? [18:09:01] i think that will be ok - gimme one sec to dbl check [18:09:33] It might not help, but I think the alternative is waiting for ryan who has root keys installed [18:10:07] andrewbogott: go for it [18:11:45] talk about timing [18:11:59] heh [18:12:43] awjr, no dice [18:12:44] what's up? [18:12:49] pssshhh [18:12:59] Ryan_Lane, sudo has stopped working on a couple of mobile instances. [18:13:01] e.g. staging [18:13:13] But not on /all/ the instances in the project [18:13:46] what specifically isn't working? [18:13:59] I think it isn't getting the policies from ldap. [18:14:08] So no one has sudo rights to do anything [18:14:17] Ryan_Lane: awjrichards@staging:~$ sudo su - [18:14:17] [sudo] password for awjrichards: [18:14:17] Sorry, try again. [18:14:17] [sudo] password for awjrichards: [18:14:17] awjrichards is not allowed to run sudo on staging. This incident will be reported. [18:14:23] ok. let me see [18:14:26] yeah, same story for all sudoers on staging [18:14:58] what's the actual instance name> [18:15:01] legoktm: Try again? [18:15:03] staging [18:15:06] ah [18:15:07] ok [18:15:08] will do [18:15:44] Coren: nope, same thing :( [18:15:51] * Coren grumbles. [18:16:04] Coren: is this a similar issue, or different? [18:16:30] Ryan_Lane: Completely different. Caused by SGE only looking at primary groups and service groups not having a common primary group. :-) [18:16:59] heh [18:18:01] Only workaround is disable queue access control. [18:20:03] why does SGE only look at primary groups? [18:20:48] Ryan_Lane: Because it was written by Sun in the days of SVR4 before additional groups even existed, I'm guessing. :-) [18:21:05] andrewbogott: so, ldap lookups are broken on this instance [18:21:31] let's see if the ldap servers are acting poorly [18:21:38] nope [18:22:23] RIP Sun :( [18:23:00] I never much likes Sun. I was a Digital weenie myself. :-) [18:23:12] liked* [18:24:04] strange getent passwd and getent group work [18:25:17] legoktm: Should work now, I disabled the check that failed. [18:25:25] ty [18:26:21] Coren: nope...... [18:26:33] ... message? [18:26:50] same [18:26:51] Unable to run job: denied: no matching department for user "local-legobot" or group "local-legobot". [18:26:51] Exiting. [18:27:00] Ohcrap. It's not even able to put you in a department at /all/. [18:27:10] I'll have to add every service user manually. [18:27:18] * Coren glares at Ryan_Lane [18:27:33] hey, don't glare at me for shitty software [18:28:30] do the open grid people have an irc channel? [18:28:48] ask them what you can do. this is a stupid requirement of the software [18:28:52] this isn't 1980 [18:30:29] legoktm: Try this. [18:30:33] OH [18:30:40] you changed the nslcd config [18:30:41] ok [18:30:42] guys. [18:30:49] this is broken [18:30:55] ...? [18:31:00] all of labs is now broken [18:31:00] wooh, it submitted [18:31:06] o.O [18:31:24] Ryan_Lane: Whu? Why would that have broken anything? [18:31:26] http://pastebin.com/ZXTkdmjf [18:31:55] Right, that looks okay to me. Why is it broken? [18:32:02] because lookups don't work [18:32:16] Apr 2 18:30:12 staging nslcd[4888]: [3c9869] ldap_result() failed: No such object [18:32:47] Coren: Is there a high load right now? Its still queued. [18:33:11] ... what? That stupid thing fails multiple basedn if /any/ of them fail? [18:33:37] no clue, but this config isn't correct [18:33:49] Ryan_Lane: It's working perfectly on tools. [18:34:02] We were thinking that it returns the union of all lookups [18:34:21] Ryan_Lane: and on bots [18:34:29] andrewbogott: It does! [18:34:44] andrewbogott: bots doesn't have the ou and it's working fine. [18:34:44] I bet it only works if the OU exists [18:35:23] Ryan_Lane: ^^ [18:35:35] you sure puppet ran on them>? [18:35:38] Ryan_Lane: There's the message in the logs, though. But sudo works 100%, and so to getent, id, etc [18:35:47] do: id laner [18:35:50] Ryan_Lane: So it's noisy if the OU isn't there, but works. [18:35:51] do all of my groups show up? [18:37:06] Nope. We apparently tickled a bug in nslcd. Damn. Rollback? [18:37:24] wait [18:37:28] Apparently it works fine except when collecting additional groups. [18:37:41] is this also a problem in tools? [18:38:07] Ryan_Lane: Nope, you got all your groups there. So it really is behaving badly when the OU isn't there at all. [18:38:16] yeah, so, let's populate the OUs [18:38:25] hahaha [18:38:28] That'd work. [18:38:28] it's even worse than you think [18:38:33] root@i-0000015e:~# id laner [18:38:34] id: laner: No such user [18:38:59] What's i-0000015e? [18:39:10] bots-labs [18:39:23] which is lucid [18:39:33] precise at least doesn't totally break [18:39:35] o_O? [18:39:37] lucid completely breaks [18:39:46] ok. let me make an ldif [18:39:55] So they only half-fixed the bug between releases. [18:40:05] it's not necessarily a bug [18:40:12] well [18:40:12] it is... [18:40:16] * legoktm slightly nudges Coren [18:40:27] my job is still listed as qw [18:40:30] legoktm: Hang on, you are number 2 on my list atm. :-) [18:40:36] ok :P [18:40:41] legoktm maybe I can help you? :) [18:41:29] legoktm: What's your job number? [18:41:39] Coren: 231 [18:41:42] petan: on tools :/ [18:42:21] legoktm: no problem [18:42:36] legoktm: Yeah, it's the same problem about permissions on queue for users without the right group; lemme just set all the ACLs to none for the moment. [18:42:39] but im about to queue 190 jobs on bots-gs [18:42:56] legoktm no problem as well [18:43:39] :) [18:43:59] legoktm: Can you cancel and restart it, please? [18:44:05] sure [18:44:56] legoktm: Yeah, still got the permission problem. Darn. Give me a few while I try to find a workaround. [18:45:44] things are also being added to ldap improperly [18:45:44] dn: ou=groups,cn=local-matilda,ou=groups,cn=tools,ou=projects,dc=wikimedia,dc=org [18:45:56] legoktm: There we go. It unstuck. [18:46:03] oh [18:46:03] waity [18:46:05] *wait [18:46:06] ignore me [18:46:13] that's the ldif generator I'm running [18:46:14] Ryan_Lane: Don't we always? :-) [18:46:15] :D [18:46:19] :) [18:47:52] Ryan_Lane: Well, so it's not a major problem because I can turn off access control and move the "problem" to a human and documentation one about what is proper to queue where; but I just read the documentation on SGE and know why it uses the primary group: by architecture, users need to be in exactly one department. [18:48:22] ok. ldap should now be fixed everywhere [18:48:30] Coren, Ryan_Lane, the sudo policy patch should be ready to go now. https://gerrit.wikimedia.org/r/#/c/56523/ [18:48:34] awjr: try to sudo now? [18:48:55] sudo on staging works for me now [18:48:57] Ryan_Lane: And the new code creates the OU on project creation now, IIRC, so it won't recur? [18:49:24] andrewbogott, Coren: let's schedule deploys like this in the future. I wanted to create the OUs for all projects before we pushed the change ;) [18:49:25] Coren, btw, adding the sudo policy will be kind of a pain for groups creasted before that patch lands. So... [18:49:50] Coren: yeah, project creation automatically creates the OUs [18:49:54] andrewbogott: That's okay, I can work around it. [18:50:07] Ryan_Lane: Yep, OK. The code I wrote creates the OUs anytime it needs them. I didn't know there would be other things that needed them :( [18:50:27] well, I wouldn't expect a problem like this [18:50:35] but I like to have the projects in a consistent state [18:50:46] so, I always make sure I run a maintenance script to update everything [18:51:10] which in this case was just writing a small python script to make an ldif [18:51:41] I'll write an outage email [18:54:53] did someone say outage here [18:54:59] :o [18:55:20] petan: It was an outage, it's an outrage! [18:55:31] yay [18:55:40] whatever - wm-bot survived [18:55:43] !ping [18:55:43] pong [18:56:07] I have a terrible leak in my irc client [18:56:11] petan: In theory, nothing that was actively running and not doing LDAP lookups (directly or indirectly) is likely to have been unaffected. [18:56:39] ... too many negatives. [18:56:39] nothing is going to be unaffected? :D [18:56:40] damn [18:56:48] that sounds like everything is down [18:56:52] :o [18:57:05] nothing not doing lookups is unlikely not not have been unaffected. :-) [18:57:17] aha now itś clean [18:57:19] * clear [18:58:06] Coren how are you handling outgoing mails [18:58:21] so that when some borked tool starts spamming root account or any other account [18:58:34] like the times when leslie come here totally pissed [18:58:44] threatening us with bringing labs down etc :D [18:58:56] Ryan_Lane, andrewbogott: sudo works now, thanks! [18:59:02] yw [18:59:51] petan: it likely didn't affect anything, but it was an outage, so it deserves a report [19:00:02] ok [19:00:51] petan: I'm not, yet, I'm waiting on Legal to finalize its position. [19:01:49] legals you say? remind me this: http://www.techdirt.com/articles/20130401/03025722522/is-golden-age-wikipedia-coming-to-end.shtml [19:02:08] wikipedia being sued :o [19:04:44] petan: Honestly, I'm surprised this doesn't occur more often. [19:12:01] @labs-project-instances tools [19:12:02] Following instances are in this project: tools-login, tools-puppet-test, tools-webproxy, tools-exec-01, tools-webserver-01, tools-master, tools-db, tools-exec-02, tools-shadow, [19:12:23] tools-shadow? [19:12:35] @labs-info tools-shadow [19:12:36] [Name tools-shadow doesn't exist but resolves to I-0000064d] I-0000064d is Nova Instance with name: tools-shadow, host: virt11, IP: 10.4.1.66 of type: m1.small, with number of CPUs: 1, RAM of this size: 2048M, member of project: tools, size of storage: 30 and with image ID: ubuntu-12.04-precise [19:12:46] petan: SGE shadow master. Not yet ready for primetime (waiting for NFS) [19:12:50] aha [19:13:17] @user-info dzahn [19:13:25] case sensitive [19:13:30] and itś @labs-user [19:13:31] @user-info Dzahn [19:13:39] @labs-user Dzahn [19:13:39] Dzahn is member of 21 projects: Bastion, Bots, Ceph, Commons-dev, Deployment-prep, Editor-engagement, Etherpad, Gareth, Greensmw, Haproxy, Integration, Mailman, Nagios, Pdbhandler, Phabricator, Planet, Rt-testing, Testlabs, Translation-memory, Visualeditor, Wikistats, [19:13:40] @labs-user Dzahn [19:13:40] Dzahn is member of 21 projects: Bastion, Bots, Ceph, Commons-dev, Deployment-prep, Editor-engagement, Etherpad, Gareth, Greensmw, Haproxy, Integration, Mailman, Nagios, Pdbhandler, Phabricator, Planet, Rt-testing, Testlabs, Translation-memory, Visualeditor, Wikistats, [19:13:41] thanks [19:13:44] :D [19:14:14] but most memberships were just for helping people to setup stuff ..hmm [19:14:27] Ryan_Lane, can I get a +2 on my sudo policy patch? https://gerrit.wikimedia.org/r/#/c/56523/ [19:14:28] is this apostropheś test displaying apostrophe properly or s with ´ above [19:14:42] looks good to me [19:14:45] GTK textview keep showing it as crap [19:14:52] petan: The second thing… looks to me like an s with an accent. [19:15:06] oh, yea, sorry, that [19:15:09] wait second thing? [19:15:12] I sent only 1 :D [19:15:21] ś [19:15:32] is this apostrophe and s or weird s [19:15:40] damn textview [19:15:47] < petan> ś [19:15:48] petan, my choices were 1) displaying apostrophe properly or 2) s with ´ above [19:15:51] And I choose 2 [19:16:13] oh right :D [19:17:27] andrewbogott: looking [19:17:40] array( 'chown -R ' . $groupName . ':' . $groupName . ' ' . $homeDir ), [19:17:59] hm [19:18:11] that used to be something else [19:18:12] this is better :) [19:20:46] heh in linux it works [19:20:53] and they say GTK is cross platform [19:22:17] andrewbogott: +2'd. good work [19:22:43] Ryan_Lane, Coren, I'm going to merge that patch into wikitech right now if there's no objection… will save us cleanup later on. [19:22:54] go for it [19:23:00] andrewbogott: Sounds good to me. [19:25:11] Coren, want to test and verify? [19:25:30] andrewbogott: Doing so now. [19:27:22]