[00:31:20] Warning: There is 1 user waiting for shell: شركت بنيان سازان شمال (waiting 0 minutes) [00:34:17] Coren: since I am still awake, I got a minor issue with the NFS mount in labs. [00:34:29] Pray tell [00:34:36] Coren: as root on the instance, I could not chown a file belonging to root ( no root squash ) [00:34:58] that was an issue with logrotate which is root [00:35:00] o_O? It's supposed to be no_root_squash for everyone. [00:35:06] it would zip the old files [00:35:22] then create a file with root:root and then attempt to restore the original ownership [00:35:22] What is the mount point? [00:35:24] which would fail [00:35:41] the workaround was to ask log rotate to not recreate the files and letting the daemon to do it :) [00:35:58] Oh, wait, that's not the problem you think. That's a /local/ user that owned the file, right? [00:36:13] I.e. One not in LDAP? [00:36:19] the local root [00:36:30] No, I mean the original owner. [00:36:46] ah yeah that was udp2log:udp2log which is a user local to the machine [00:36:56] * Coren ponders. [00:37:24] Can you cat /sys/module/nfs/parameters/nfs4_disable_idmapping on that box? [00:44:55] Warning: There is 1 user waiting for shell: شركت بنيان سازان شمال (waiting 13 minutes) [00:50:31] Coren: sorry got distracted [00:50:45] N [00:50:49] $ sudo cat /sys/module/nfs/parameters/nfs4_disable_idmapping [00:50:50] N [00:51:07] hashar: Hold that thought, doing something enwp-related with semi-hi emergency level. Will be back in 5 [00:55:05] Coren: it is not a big issue, I can fill a bug for it [00:55:28] 3am there so I am going to sleep soon [00:56:36] hashar: No, I'm back. [00:57:25] oook :-] [00:58:21] Warning: There is 1 user waiting for shell: شركت بنيان سازان شمال (waiting 27 minutes) [00:58:26] hashar: For some reason, the /etc/init/nfs-noidmap didn't run on that box. [00:58:39] hashar: Did you reboot it when you transitionned? [00:58:53] yes sir [00:58:57] o_O [00:59:10] want me to reboot it ? :D [00:59:21] check of the presence of /etc/init/nfs-noidmap [00:59:40] not there [00:59:57] dafu? Did you add the puppet class? [01:00:16] (What box is this?) [01:01:29] sorry I am slow [01:01:35] deployment-bastion.pmtpa.wmflabs [01:01:40] part of the deployment-prep project [01:02:09] https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000390 [01:02:14] Puppet Class [01:02:15] base, ldap::client::wmf-test-cluster, exim::simple-mail-sender, sudo::labs_project, jenkins::user, misc::deployment::scap_scripts, role::beta::autoupdater, role::beta::logging::mediawiki, role::beta::maintenance, role::labsnfs::client [01:02:22] well the one that matter is role::labsnfs::client [01:03:49] hmm maybe something is not applied [01:03:58] there is some puppet error err: /Stage[main]/Mediawiki::Users::L10nupdate/File[/home/l10nupdate/.ssh]/owner: change from 10002 to l10nupdate failed: Failed to set owner to '996': Invalid argument - /home/l10nupdate/.ssh [01:03:58] :( [01:11:58] Warning: There is 1 user waiting for shell: شركت بنيان سازان شمال (waiting 40 minutes) [01:13:25] Crap. That's a bit circular since the problem is preventing the puppet to run. [01:13:47] wtf is puppet trying to write in /home for? [01:14:20] There is an ugly workaround possible: [01:14:38] Add the following: [01:14:40] echo 1 >/sys/module/nfs/parameters/nfs4_disable_idmapping [01:14:45] At the end of /etc/rc.local [01:14:49] And reboot. [01:14:58] This will force the change, and allow puppet to run. [01:15:18] Which will install the nfs-noidmap which does it the "right" way. [01:15:55] It looks like the problem was caused by the puppet update; when you ran it originally, it must have failed partway. [01:16:53] * Coren does it [01:17:14] Can I reboot the box? [01:17:29] hashar: ^^ [01:17:37] sure [01:18:00] just one minor issue is that once rebooted you have to stop the udp2log service and start the udp2log-mw one [01:18:02] they conflict [01:18:07] !log deployment-prep rebooted deployment-bastion after manual workaround for a broken puppet run [01:18:09] Logged the message, Master [01:18:57] and the write in /home is caused by some wierd dependency in some role class :-] [01:19:42] root@i-00000390:~# cat /sys/module/nfs/parameters/nfs4_disable_idmapping [01:19:42] Y [01:19:42] is the expected result. [01:19:59] forcing a puppet run to see if that fixed it [01:21:02] notice: /Stage[main]/Role::Logging::Mediawiki/Misc::Udp2log::Instance[mw]/Service[udp2log-mw]/ensure: ensure changed 'stopped' to 'running' [01:23:18] Your puppet config on that box is seriously broken, man. It edits /etc/iptables-save one way, then undoes it immediately, at every run. [01:24:14] And it's not applying the role::labsnfs::client class completely? o_O [01:24:38] the rc.local fix will circumvent the problem for now, but we'll need to figure out what the root cause it. [01:25:13] :( [01:25:28] Warning: There is 1 user waiting for shell: شركت بنيان سازان شمال (waiting 54 minutes) [01:25:50] Coren: for beta I usually apply the production class [01:26:05] Coren: that tends to work until the prod classes are changed in someway or an other [01:27:13] Coren: the iptables issues, I have no idea [01:27:23] Coren: I think that is a problem with Augeas [01:29:03] The NFS chowns should work right, for now. But we need to eventually figure out why the class sin't getting applied right. [01:29:09] !log deployment-prep bastion: /etc/init.d/udp2log stop && /etc/init.d/udp2log-mw start [01:29:10] Logged the message, Master [01:29:25] Coren: so I have applied it on other instances as well [01:29:42] hashar: Lemme go see one of 'em to compare. Name? [01:29:45] deployment-apache32.pmtpa.wmflabs deployment-apache33.pmtpa.wmflabs [01:30:09] deployment-jobrunner08.pmtpa.wmflabs and finally deployment-video05.pmtpa.wmflabs [01:30:25] they all uses the MediaWiki files hosted on the NFS [01:32:14] Ah, fuuu. The bug seems to be in the puppet class. [01:32:31] It's the change Ryan requested. I blame Ryan. :-) [01:32:49] (I'll fix it for good tomorrow; it does mean you'll be stuck having to reboot 'em all) [01:33:06] will do that next week :D [01:33:12] I think I will sleep tomorrow [01:33:20] Go sleep, then. :-) [01:33:20] thanks for the debugging! [06:54:13] is there any facility similar to https://wiki.toolserver.org/view/Toolserver_database ? [08:08:00] hey [08:10:00] Warning: There is 1 user waiting for shell: Bjelleklang (waiting 0 minutes) [08:23:29] Warning: There is 1 user waiting for shell: Bjelleklang (waiting 13 minutes) [08:23:49] I'm here in case of any questions :) [08:24:00] hey [08:25:32] bjelleklang so, what exactly, beside shell access you need? :P [08:25:57] because you already have that [08:26:00] oh [08:26:06] If so I'm not really sure yet [08:26:28] ok let me know once you are ;) [08:26:34] I'm using the toolserver today, but that will go away sometime in the future, so thought that familiarizing myself with labs would be a good idea :) [08:26:48] Will do, thanks! [08:26:49] !docs [08:26:49] View complete documentation at https://labsconsole.wikimedia.org/wiki/Help:Contents [08:26:57] that is some boring reading [08:27:03] !toolsdocs [08:27:04] http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Help [08:27:08] that is less boring, but still [08:29:01] :) [08:29:14] I'll take a look at it and try not to fall asleep [08:43:49] Hi [08:45:23] I'm trying to login to brand new project instance and got stuck with login from bastion. [08:45:32] !access [08:45:32] https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [08:45:48] exbe are you sure that instance was built successfuly? [08:45:55] !console [08:45:55] in case you want to see what is happening on terminal of your vm, check console output [08:46:09] you better check :P [08:46:59] Not sure, security groups is empty, console says permission denied (publickey) [08:47:17] ok what about the !console [08:47:46] But I have checked web and default groups during creating [08:47:52] when you open the instance list [08:48:00] there is console output link [08:48:01] click it [08:48:03] read it [08:48:20] you may see that there was problem during creation of instance [08:48:27] @labs-user Exbe [08:48:28] Exbe is member of 2 projects: Bastion, Documentation, [08:48:51] exbe try to ssh to bastion2 [08:48:54] from bastion1 [08:49:04] Ok, one moment [08:49:05] if it won't work, you are not forwarding key [08:49:13] if it works, your newly created instance is broken [08:49:38] Nope, rejects me [08:50:05] from bastion2 [08:51:36] Instance's console is showing welcome msg [09:00:22] !forwarding | exbe [09:00:22] exbe: https://labsconsole.wikimedia.org/wiki/Access#Using_agent_forwarding [09:00:54] Reading... [09:05:33] well, I'm on putty with "allow redirect..." + console output is different in my case. [09:06:45] When security groups should be updated on the instanse? Is there a way to check it somehow? [09:17:05] * exbe updated my user page@wikitech. In case anybody will have a chance to look. [12:02:49] exbe you don't need to update security group unless you are installing some extra sw [12:22:32] morning all... any admins up? [12:23:04] i'd like a new instance [12:32:01] greenrosetta where? [12:32:07] tools [12:32:16] call it foobar pls... we can delete it later? [12:32:25] ok, what do you need that instance for? [12:35:19] tools-login [12:36:56] ? [12:37:09] I mean why do you need a new instance? what is purpose of it? [12:37:27] im testing a deployment process [12:37:41] Trying to create a project for someone to easily have a shell application working [12:38:03] there is bots project (staging area) for tests, tools project is supposed to be production like and only testing can be done tools-dev [12:38:21] i can use that [12:38:30] do I have to be a member of that project? [12:38:30] I don't think Coren|Sleep would agree with instance only for tests was created [12:38:35] bots? [12:38:43] everyone who is on tools has access to bots [12:38:43] not a bot [12:38:48] ah [12:38:58] bots-gs is alternative to tools-login [12:38:59] ill wait for Coren... ive got other stuff to do [12:39:01] for now [12:39:04] no biggie [12:51:48] so, what exactly you want to test? should I create that new instance in bots project? [12:52:10] I would like to know more about what you really need, maybe creating instance would not be enough, or maybe it's not even needed? [14:16:08] hello, i've got some questions.. i've checking the docs, and there are several projects, bots,webtools and tools. as far as i understood, tools is a kinda ts, so i can have a project there with bots and tools like in ts? [14:18:12] Alchimista: You can create a project on Tools for every tool or bot you need. [14:20:30] Alchimista: That's it, yes. If you give me you wikitech username, I can give you access. [14:22:36] Coren: so i'll have a /home/ where i can have my codes and don't care to make passwords and other sensitive stuff private? [14:24:14] Alchimista: Yes, but the "right" way to do things is to create tool accounts for your tools that have each their distinct homes. Check out https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Your_tool_account [14:24:57] Also, only tools have web stuff. [14:24:58] Credentials that need to remain private simply need to have their permissions set accordingly (generally, just deny from others does) [14:27:42] BTW, a common problem on Toolserver are users who make their ~/.my.cnf & Co. readable to the world :-(. [14:28:16] (Many editors don't preserve the permissions.) [14:28:27] I don't think we need to touch this file? [14:28:34] Coren: i prefer to go slowlly, first get adapted to labs environment and try it. if i get too many new things now, i'll lose myself [14:28:54] can i have several tools on the same tool account? [14:29:27] You can. [14:29:45] Darkdadaah: No, but enough users do. Just take a look at /home/*/.my.cnf on Toolserver. [14:32:03] (Database passwords aren't that sensitive, but leaking a bot's credentials can suck big time.) [14:35:09] Coren: my labs nick is alchimista,i already have access to bastion and a bot project. could you also create a tool account named alchtools ? [14:39:12] Alchimista: It's easier if you create it yourself because that automatically marks you down as its maintainer (and I want to test that it works properly if you make it yourself, too) :-) [14:39:19] Alchimista: You're in the project now. [14:41:33] addshore I released new version of pidgeon, it has extension for freenode as well, so that you can simply /kb users or trolls without need to use chanserv at all [14:41:51] it fixes most of issues I found so far + your bug [14:41:59] Coren: i've entered without probelm. Nice ASCII art on labs logo, by the way [14:42:15] Alchimista: Just goes to show how old school I am. :-) [14:42:17] greenrosetta btw Coren is here [14:42:24] greenrosetta so that you can discuss it with him [14:47:16] Coren: now i'm lost. to create a project, it's just going to https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProject&showmsg=setfilter and click on "add service group"? [14:47:46] Alchimista: In the tools project, yes. [14:47:55] Be back ina few minutes. [14:57:31] Alchimista: Did it work? [14:58:29] scfc_de: no, it says i'm not in the group, so i can't perform the action, yet, i'm in the left collum [14:58:46] in the member list. [14:59:26] Alchimista what is your labs name? what are you going to do? [14:59:33] @labs-user Alchimista [14:59:33] Alchimista is member of 3 projects: Bastion, Bots, Tools, [14:59:37] Alchimista: Yeah, that was a bug/misconfiguration a few days ago. Perhaps Coren was just overoptimistic :-). [15:00:28] nop, i was watting for him, while that, i'm on translatewiki, because the actual labs interface really really sucks [15:01:22] Alchimista: You mean wikitech.wikimedia.org? [15:02:19] scfc_de: yah, the special:nuovaProject. It's with portuguese and english all mixed up, so i'm making some translations [15:03:47] Alchimista: Ah, okay, I didn't ever change the language from English on wikitech :-). Let's see how German is doing ... [15:04:31] Actually, not too bad. [15:05:10] Alchimista if you tell me what you need I may be able to help you with it [15:05:38] Warning: There is 1 user waiting for shell: Happyharrythehorse (waiting 0 minutes) [15:06:09] petan: Coren: my labs nick is alchimista,i already have access to bastion and a bot project. could you also create a tool account named alchtools ? [15:06:33] petan: i was tryng to create a tool account by myself, so that coren could test if it's ok. but whille my user is on the members list, clicking on ad service group returns me a message that i can't create it, because i'm not a project member [15:08:51] ok I can either create it for you, or wait for Coren :P [15:09:01] I don't really know why it happens, might be a bug? [15:09:03] !rb [15:09:03] broken? report a bug: https://bugzilla.wikimedia.org/enter_bug.cgi?product=Wikimedia%20Labs [15:09:33] petan: yah, that's why coren didn't created already, so that he could test it [15:09:51] And apparently the test says: Still broken :-). [15:10:34] well, coren must have some things to do, or else whe would get bored :P [15:14:00] Alchimista Coren can test this on another project... if you really need to have your tool account somewhat now, I can create it, or if you don't need it now, you can of course wait... for ages :D [15:14:43] I am wondering who is making that interface [15:14:46] I think andrewbogott? [15:15:34] which interface? [15:15:43] wikitech [15:15:57] that one where you can create a service [15:16:01] It's some me and some Ryan. [15:16:03] like lical-blah [15:16:05] locak [15:16:08] * local-blah [15:16:15] Oh, on the manage projects page? [15:16:19] yes [15:16:34] That's mine. Probably toollabs will need its own suite of pages at some point though. [15:16:43] ok Alchimista reports it is broken [15:16:55] * andrewbogott scrolls up [15:17:07] I be back! [15:17:20] Coren you scroll up too :P [15:17:23] (It would be nice if on the Tools' page for this "Service groups" could be renamed "Tools".) [15:17:58] whoever create new images for ubuntu, it has invalid /etc/fstab [15:18:08] scfc_de: I think that Tool Labs is different enough conceptually that we'll probably want to make a Special:Toolproject that's more tailored. [15:19:04] Warning: There is 1 user waiting for shell: Happyharrythehorse (waiting 13 minutes) [15:19:48] Coren: Fine with me :-). An "s/^local-//;" would probably make sense then as well. [15:20:12] !log bots petrb: created new instance bots-login that will replace bots-gs in order to make things look like in tools [15:20:14] Coren, petan: Could one of you add Magnus_Manske to the Tools project, please? [15:20:15] Logged the message, Master [15:20:28] scfc_de even you can! [15:20:51] no need to be admin to add people to projects [15:20:59] petan: "You must be a member of the projectadmin role to perform this action." [15:21:05] o.O [15:21:09] that has had to change then [15:21:19] but I can add people to bastion project and I am not an admin there [15:21:22] Alchimista, can you try again? [15:21:42] sorry, i got out. let me try again.. [15:21:46] Failed to add Magnus_Manske to tools. [15:21:55] petan, yeah, I think Ryan changed some permissions to make a project admin more like a project admin. [15:21:59] scfc_de that is typical labs Error - with no clue why it happens [15:22:03] I can't create a tool either. Same error: You can not complete the action requested as your user account is not in the project requested. [15:22:15] it just "failed" :/ [15:22:32] andrewbogott: still the same error. i can't perform the action because i'm not on the solicitated proj [15:22:48] petan: Space, not underscore. [15:22:52] (I added it) [15:22:57] Failed to add Magnus Manske to tools. [15:22:58] aha [15:23:20] in both cases it should be failed to add because user was already in list / or whatever according to reason [15:23:28] I hate errors that say just: Error [15:23:31] Darkdadaah: it's a sys admins cabal XD [15:23:52] Alchimista: here too? [15:23:52] Darkdadaah I can create it for you until this problem is solved [15:24:03] Alchimista, petan, ok -- Ryan was messing with authentication on that page yesterday so it's best if we let him sort things out. He should be in in a couple of hours. [15:24:06] Coren, petan: Well, it worked in the end, so thanks anyway :-). [15:24:09] petan: No need, I was only testing. [15:24:10] Darkdadaah: they're cabal is everywhere :P [15:24:15] Meanwhile I can create Alchimista's tool if that gets him unstuck. [15:24:30] well, he just decided to be a guinea pig [15:24:35] Ah, ok. [15:24:35] doesn't want it :P [15:24:43] y'all could log a bug if you feel like it :) [15:24:54] !rb | Alchimista :P [15:24:54] Alchimista :P: broken? report a bug: https://bugzilla.wikimedia.org/enter_bug.cgi?product=Wikimedia%20Labs [15:24:57] yah, ig it'l take hours. i'll provably create another later [15:25:11] petan: "/!\ A problem has occured." :-) [15:25:31] that is microsoft alternative to Ryan's error message [15:25:54] Alchimista: Which name do you want for your tool? I'll add it. [15:26:16] o_O [15:26:19] Even I can't do it! [15:26:34] looool [15:26:47] Silly Ryan! :-P [15:27:17] He did say he was closing security holes. [15:27:36] why isnt this statement working? [15:27:37] PATH="$HOME/bin:$PATH" [15:27:43] it adds an extra // [15:28:00] greenrosetta: $HOME currently has a trailing / but it's harmless [15:28:07] [bz] (NEW - created by: Alchimista, priority: Unprioritized - normal) [Bug 48049] Error creating a tool project - https://bugzilla.wikimedia.org/show_bug.cgi?id=48049 [15:28:09] In Unix, // is exactly equivalent to / [15:28:15] it aint working though [15:28:24] nothing in bin runs [15:28:35] Did you /export/ your $PATH? :-) [15:28:43] export PATH="$HOME/bin:$PATH" [15:29:14] still doesnt work [15:29:36] greenrosetta what doesn't work? [15:29:40] pastebin what you are trying? [15:29:42] try echo $PATH [15:29:51] I have a script in the bin directory called "test" [15:30:00] it runs with ./test from that directory [15:30:01] does it have +x [15:30:04] it should run anywhere [15:30:24] greenrosetta: /your/ home? [15:31:02] the tools home [15:31:05] which resolves correctly [15:31:14] common-interests? [15:31:31] flask-stub [15:31:48] this is the $PATH /data/project/flask-stub//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/sbin:/bin:/usr/X11R6/bin [15:31:57] see the //? [15:32:04] Yeah, the // is harmless. [15:32:18] ok, but the script called "test" in that path doesnt run [15:32:32] You called your script test? :-) [15:32:33] lol [15:32:40] cos its just a test [15:32:41] Warning: There is 1 user waiting for shell: Happyharrythehorse (waiting 27 minutes) [15:32:41] You are aware there is a shell builtin named test? [15:32:51] im deleting it after it works [15:32:57] ah [15:33:00] that would be the rub [15:33:03] You need to name it something else. :-) [15:33:07] Coren, how are your apache sites-available skills? [15:33:10] or use command test [15:33:20] andrewbogott: Reasonably good. What can I do? [15:33:46] On https://openstack-role-dev4.pmtpa.wmflabs/ I get a circular redirect despite my best intentions... [15:33:58] * Coren goes check. [15:34:11] you should be able to log into openstack-role-dev4 and see the file. Do you mind spending a minute to tell me what I have wrong? [15:34:35] andrewbogott: wiki or default? [15:34:58] Right now I'm looking at the top level dir. [15:35:20] hm… same behavior at /wiki though [15:35:28] s/dir/url/ [15:36:12] andrewbogott: You have conflicting configs; you still have 'default' there which also configures *:80 and *:443 [15:37:17] nevermind, there's just one enabled. Sorry [15:37:25] exit vim? [15:37:38] exited :) [15:38:42] andrewbogott: What are you trying to do with the rewriterules? It helps if I know what your intent is. :-) [15:39:35] It's copy/pasted from nova-precise2, so I'm largely free of intent. The overall goal is to have http: get us https:/wiki [15:40:09] But also some of that is magic mediawiki stuff that I don't quite understand :( [15:40:21] You want it actually redirected or you want /foo to work like /wiki/foo? [15:40:35] the second thing. [15:41:11] Ah. Then your rules are all wrong. :-) Gimme a sec. [15:41:31] Probably worth looking at the rules on nova-precise2 since I'm using that as a model. [15:41:40] Basically, I want this site to work like that one. (only via puppet) [15:41:54] which, btw, that file is puppetized so you should copy before you edit. [15:42:13] There might be more going on at nova-precise2; there's some stuff there about a /view/ URL [15:42:41] nova-precise2 is for wikitech-testing [15:42:45] old wikitech used /view/ URLs [15:44:32] Okay, so that I'm clear: You want to be able to view pages with https:/ThePage directly, not have it redirect to /wiki/ThePage? [15:44:52] (Both are equally easy) [15:46:07] Warning: There is 1 user waiting for shell: Happyharrythehorse (waiting 40 minutes) [15:47:12] andrewbogott: ^^ ping? [15:47:28] Coren: I want it to act like nova-precise2 and/or wikitech. When I go to https://wikitech.wikimedia.org/bananas gets me a 404, but https://wikitech.wikimedia.org/wiki/Bananas gets me a wiki page. [15:47:48] The latter is, I think, actually displaying https://wikitech.wikimedia.org/wiki/index.php/Bananas or something like that. [15:47:50] Ah, okay, standard mediawiki short path. :-) [15:50:54] andrewbogott: {{done}} Check /tmp/wiki [15:51:00] thanks [15:53:27] * andrewbogott tries to puppetize [15:58:31] Coren, was it actually loading a wiki page for you? [15:58:38] how so I add an enviorment variable so my app sees it? I can add it from the terminal, but apache wont see it [15:58:55] eg, i can see my enviornement variable from the terminal [15:59:34] Warning: There is 1 user waiting for shell: Happyharrythehorse (waiting 54 minutes) [16:01:15] andrewbogott: Yep. [16:02:05] Still does. [16:02:33] at what url? [16:02:39] I just get 404s [16:02:46] http://wikitech.wikimedia.org/wiki/Bananas ... oh, what a moron I am. :-) [16:02:56] * Coren tested against the wrong URL. :-) [16:03:47] Ryan_Lane: I haven't investigated but I think your auth changes have some issues. Sounds like no one at all can create service groups anymore (including Coren) [16:04:03] Because of the Aliases I may have to be more explicit about the redirect. [16:04:03] oh? [16:04:16] Ryan_Lane: "Not a member of the project", it lies. [16:04:22] heh [16:05:13] you can add/remove members, though, right? [16:05:29] ah. I see why [16:05:42] try now [16:05:57] andrewbogott: {{fixed}}. I had to explicitely instanciate the path to /w and not rely on the alias [16:06:20] andrewbogott: http://wikitech.wikimedia.org/wiki/Bananas leads to the woki as expected [16:06:47] well, at least I know my change was working [16:07:10] Coren, the page I see has text but is missing logo and graphics and such, so some path is still messed up. [16:07:13] Same for you? [16:07:22] previously if you screwed up the variable assignment it would just let everyone do the action [16:07:29] -> Add service group <- \o/ [16:08:21] andrewbogott: Oh, yes, I didn't check images. You probably need a couple of aliases for /other/ paths since your wiki doesn't actually live at the documentroot. [16:08:47] andrewbogott: I can't think of a reason why it shouldn't actually. Lemme fix taht. [16:09:03] Coren, I don't htink it's just images -- all the formatting is messed up. [16:09:23] andrewbogott: Yeah, all the static bits. That's because of the missing documentroot. Fixing now. [16:12:45] andrewbogott: Try this? [16:13:18] andrewbogott: There's also a note I added in the vhost config, if you want to give access to the /dumps you'll need to give permission in the section I added with a comment. [16:13:31] Ryan_Lane: i was able to create a group, but i can't *become* it. -> alchimista@tools-login:~$ become alchimista -> sudo: sorry, a password is required to run sudo [16:13:41] Alchimista: Did you log off and back on? [16:13:51] Alchimista: Group membership is set when you log in [16:13:53] Coren, looks the same as before, roughly -- still no static content. [16:13:59] o_O [16:14:06] Coren: out of ssh? let m restart then.. [16:14:12] Is this visible from the outside? I'm using elinks to test. :-) [16:14:33] Coren: working! [16:15:08] Coren, I'm just visiting https://openstack-role-dev4.pmtpa.wmflabs/wiki/Main_Page in firefox [16:15:34] andrewbogott: I'm remote. I can't talk to inside from the outside. :-) [16:15:43] Coren: Is there a way to set enviornemnt variables? [16:16:04] Coren: I'm remote too but… I guess I have a proxy set up. [16:16:23] greenrosetta: Lots of ways. The most common is in the account's ~/.profile [16:16:39] Well, I did that. But the application won't see it [16:16:43] becase it isn't logging in [16:17:11] maybe the cgi script is involved? [16:17:43] greenrosetta: Oh, you mean /for the CGI/? [16:18:00] yes [16:18:30] SetEnv in your tool's .htaccess should do that for you. [16:18:37] http://httpd.apache.org/docs/2.2/mod/mod_env.html#setenv [16:21:11] andrewbogott: Gimme a minute to set a tunnel up and I'll debug this for you. [16:27:01] andrewbogott: The apache config is the least of your worries; that Mediawiki install has some path refering to the local filesystem, some to the wrong place, and a couple I don't even understand. :-) [16:30:58] andrewbogott: Aha! The $wgSctylePath is overriden somewhere to a filesystem path. It should be an URI [16:32:30] ... because orig/LocalSettings.php overrides $wgScriptPath with a filesystem path rather than an URL. There's your problem. :-) (line 29) [16:36:42] Coren, what's wrong with this? http://bpaste.net/show/rbbB8fcf292eDDxlnvoC/ [16:40:00] Coren: Ok, so what should scriptpath be? $hostname/wiki? [16:42:01] greenrosetta: Works for me. [16:51:04] scfc_de: do I need to do something with the permissions for htaccess? [16:51:59] greenrosetta: One moment, let me check. [16:53:21] greenrosetta: No. With your .htaccess and permissions 755, the environment is set correctly for me: http://tools.wmflabs.org/wikilint/test.php [17:01:14] greenrosetta: also having flask problems? [17:01:55] well, i'm having some kind of flup one: http://tools.wmflabs.org/alchimista/test.fcgi [17:03:40] Alchimista: I think you need to put CGIs in /cgi-bin. [17:04:17] scfc_de: i was about to try it, but fcgi should run out of /cgi [17:05:47] Alchimista: Hmmm? What do you mean? I believe under /public_html, *only* *.php and *.php5 are executed, the rest is served as is. [17:05:56] andrewbogott_afk: $wgScriptPath should just be "/w" [17:06:05] scfc_de: .py also [17:06:53] andrewbogott_afk: (Not /wiki, that's rewritten to /w/index.php) [17:07:24] andrewbogott_afk: $wgScriptPath should just be "/w" [17:07:29] yep [17:08:06] Coren: Okay (https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Tools/Help&diff=68950&oldid=68936) :-). [17:08:27] scfc_de: Bah. Documentation. :-P [17:08:31] (thanks) [17:13:12] scfc_de: this is not my beach, on the few times i've used fcgi, on ts and uni, it only worked on pub_html [17:13:57] but u're right in deed -> http://tools.wmflabs.org/alchimista/cgi-bin/test.fcgi/ [17:16:30] Alchimista: Really? I'm pretty certain that on Toolserver, CGIs have to reside in ~/public_html/cgi-bin as well. Perhaps .fcgi is treated there the same way as .php "here". I don't know if it makes sense to change the configuration. [17:18:24] scfc_de: cgi's yes, but fcgi not. I had some troubles understanding how to use flup there, and on ~/public_html/cgi-bin, in max it runs as a .cgi script [17:23:14] by the way, is it possible to use filezilla on tools projects? [17:27:32] Alchimista: I never used FCGIs on Toolserver once I read that the process get killed after x seconds, and so it was only useful for *heavily* used tools :-). [17:28:04] How big should I make the filesystem? It's growable, but more efficient if we don't end up having to often. It's provision thinly regardless of size. [17:28:16] for public datasets. it's huge [17:28:20] almost 10T [17:28:25] ssh keys are tiny [17:28:36] public repos.... [17:29:18] Ouch. Eating 10T out of our 36 so soon? [17:30:21] public repos are about 3GB [17:30:26] so I'd say give that about 20 [17:30:55] Do we do the datasets now or wait until we can reshuffle shelves around? Public repo + keys = trivial [17:31:09] reshuffle shelves? [17:31:37] Well, if the NFS server hosts all that data, then the shelves which currently host it can eventually be added to the NFS server, no? [17:32:07] Or do I just give that space for the datasets and we'll deal later? [17:32:18] well, it depends. are we going to take the shelves from labstore1/2 or are we going to set up another storage solution there? [17:32:23] for experimentation [17:33:01] we could also just have a second set of nfs servers, like these, that have different shares [17:33:08] I suppose it depends on our timeline. If we don't expect to swift to something new before pmtpa goes away, then there's no point it keeping [12] around [17:33:44] Further experiments can always take place in eqiad [17:33:56] and we could use the 2nd set for bulk storage like public dumps, keys and repos [17:34:04] or read-only storage [17:34:14] Ryan_Lane, Stefano will come here at the office. The only rooms available are in 6th florr. It might take us a bit to connect to the hangout. [17:34:25] qgil: ah. ok [17:34:27] That'd work, but then we'll have to wait until everything is moved to labnfs [17:34:45] true [17:35:09] I'd say let's put the keys on nfs now [17:35:18] can do the same with public datasets [17:35:21] And the repos; those are small. [17:35:28] and the repos too [17:35:38] we can move them all over to the 2nd set of servers after [17:35:44] * Coren nods. [17:35:54] we don't even have to move them, of course [17:36:40] just re-replicate the data and when that's done delete it from labstore3/4 [17:37:11] * Coren creates and exports the volumes. [17:38:18] hm. I should go somewhere with wifi for the hangout... [17:38:27] 20G for repo, 10T for datasets. Keys? [17:38:29] keys are tiny [17:38:30] 1G? [17:38:35] Works for me. [17:38:59] ok. back in a bit [17:39:03] need to relocate [17:57:49] qgil_away: ok. I'm in the hangout associated with the meeting, whenever you guys are ready [17:58:09] Ryan_Lane, still waiting. A bell rings... [17:58:21] * Ryan_Lane nods [18:01:02] Coren: what's the directory/hostname to use for replication? [18:01:20] I'll push in the change for gerrit [18:01:31] (for repos) [18:01:40] labnfs:/public/repo [18:02:11] !resource globaleducation [18:02:12] https://labsconsole.wikimedia.org/wiki/Nova_Resource:globaleducation [18:02:26] ragesoss: there's a list of members and admins [18:02:33] ragesoss: what's your wikitech username? [18:02:45] Ragesoss [18:02:54] ok. one sec [18:03:00] Ryan_Lane: All three are now available, and exported. [18:03:20] I.e.: /public/{datasets,keys,repo} [18:03:41] ragesoss: ok. I've added you as a project admin [18:03:58] ragesoss: which means you can add anyone into the project or as projectadmin, assuming they have shell access [18:04:07] which in general we clear that queue in hours [18:04:16] man. my mom's office really sucks to work from [18:04:26] the phone constantly rings and people constantly knock on the door [18:04:47] hehe [18:05:03] and she's not here. so people knock for like 5 minutes at a time [18:05:16] thanks. This means I can also create instances and that kind of thing, and that Jeroen can as well. correct? [18:05:21] yep [18:05:28] he was already doing development in this project [18:05:37] the sidebar has all the available actions [18:05:43] may want to read up on some of the docs [18:05:57] security groups are usually the biggest gotcha for folks [18:06:06] !mediawiki [18:06:10] bah [18:06:15] @search mediawiki [18:06:15] Results (Found 18): morebots, labs-home-wm, labs-nagios-wm, labs-morebots, gerrit-wm, extension, revision, info, bots, labs-project, openstack-manager, wl, deployment-prep, tl, coren, tooldocs, toolspad, hackaton, [18:06:30] * Ryan_Lane grumbles [18:06:33] !coren [18:06:33] The toolmeister: http://www.mediawiki.org/wiki/User:MPelletier_(WMF) [18:06:41] Ah, hence 'mediawiki' [18:06:59] !mediawiki is https://wikitech.wikimedia.org/wiki/Help:Single_Node_MediaWiki [18:06:59] Key was added [18:07:12] ragesoss: https://wikitech.wikimedia.org/wiki/Help:Single_Node_MediaWiki <— of particular interest [18:07:18] makes doing mediawiki dev far easier [18:12:12] labnfs.pmtpa.wmnet:/public/repo 20959232 33280 20925952 1% /mnt [18:12:23] That was easy. :-) [18:13:17] Ryan_Lane: Do you need some of them rw or will you rsync over ssh? [18:33:38] Coren: rsync? [18:33:48] Coren: the ssh keys will be generated on the nfs server [18:33:57] it should be read only [18:34:38] hm [18:34:47] I think even the public dumps can be rsync [18:34:55] I'm not sure which will be faster [18:34:59] let me ask ariel about that [18:35:58] right now it's writing into a gluster mount [18:36:11] but that may be because it was impossible to rsync before [18:36:27] Ryan_Lane: A monkey retyping from hardcopy would likely be faster than gluster. :-) [18:36:35] indeed [18:36:44] (If it's a fast hypermonkey) :-) [19:01:41] is mingle.corp.wikimedia.org hosted under labs? [19:03:40] * AzaToth pokes Ryan_Lane  [19:04:45] or Coren ツ [19:05:27] * Coren pokes back! Poink! [19:05:39] ツ [19:05:59] I just wondered if above site is hosted under labs or not [19:06:27] dunno how to find out by my self [19:06:35] I'm not sure I know how to answer that question definitively. [19:07:10] are they bound by the labs term of use [19:07:19] * Coren checks. [19:07:21] or similar [19:07:48] AzaToth: no [19:07:52] ok [19:08:13] all things .corp.wm.o are in the office [19:08:21] * AzaToth just dislike they using propietary software [19:08:23] I see [19:08:26] I agree [19:08:28] I dislike it too [19:08:34] and that's one reason it isn't in labs [19:08:41] the other reason is that it's a production app [19:08:52] wtf [19:08:56] azatoth@azaboxen:~$ host corp.wikimedia.org [19:08:56] corp.wikimedia.org mail is handled by 10 ASPMX.L.GOOGLE.COM. [19:09:08] we switched to google apps ages ago [19:09:12] ah [19:09:22] running the office doesn't really have to do with the projects [19:09:35] so we deem it an acceptable use of proprietary software [19:09:57] we use only open source for things that need to be forkable [19:09:58] except spiritual and moral preferences [19:10:04] I don't think anyone wants to fork the foundation ;) [19:10:04] only? [19:10:16] that sounded bad [19:10:17] they'd want to fork the projects [19:10:38] what? that we only use open source for our core infrastructure? [19:10:47] we use open source everywhere possible [19:11:14] it sounded like you only used open source, if, and only if it must be forkable [19:11:20] we could use an alternative to google apps, but for a lot of it the alternatives are terrible [19:11:51] we mandate open source for things that need to be forkable [19:11:53] google apps is ok, a bit unmaintainable sometimes though [19:12:17] with exceptions for things that are impossible (like network gear) [19:12:21] it's like trying to use AWS and find that the 0.1% you must have is missing [19:12:35] heh [19:12:57] we should be completely forkable at this point. media storage may still be an issue [19:13:02] so if there are two software available, one oss, and one not, and the OSS is better, you wil lchoose the non-OSS if forkability isn't needed? [19:13:21] the non-OSS would have to be considerably better [19:13:44] what you said sounded much inverse [19:13:50] heh. sorry [19:13:56] np [19:13:58] there's also personal preference too [19:14:03] many employees use os x [19:14:08] heh [19:14:13] designers... [19:14:19] I use os x ;) [19:14:24] most of the ops team does, in fact [19:15:18] most? really? [19:15:25] * greg-g is counting in his head [19:15:34] it's better to have productive employees than idealogy. if people find themselves more productive using something non-linux I think they should use that thing [19:15:34] if they use OS-X, it means they earn too much [19:15:58] hah. we're considerably underpaid compared to the average [19:16:06] ツ [19:16:39] well, as long people aint using outlook [19:16:44] heh [19:16:46] I doubt anyone is :) [19:17:01] I think we do have some folks using windows [19:17:12] ok. gotta run [19:17:19] k [19:17:19] back in a while [19:17:42] hmm [19:18:01] when I think about it, why is mobile under office?= [20:28:32] Ryan_Lane there is nothing bad on using comercial software, people need to live from something :P I prefer visual studio over any other open source IDE [20:28:54] but I am quite happy with linux [20:31:00] Coren: Can you setup a couple of more projects in tools for me? I want to test my instructions so people can get started quickly [20:31:16] greenrosetta I can do that too ;) [20:31:23] ah, didnt see u [20:31:34] "stub" and "dummy" [20:31:34] greenrosetta: Doing it yourself should work, now, but please don't go overboard in quantity. :-) [20:31:46] we can delete them after [20:32:41] what do you mean I can do it myself now? Did something change? [20:34:28] hg revert -r80 --all [20:40:27] Coren: Is there a process where I can DIY a tool? [20:45:10] greenrosetta: You should be able to create a new service group from the manage projects page now; I think Ryan_Lane fixed it. [20:46:07] URL? [20:46:10] Warning: There is 1 user waiting for shell: Jose.canciani (waiting 0 minutes) [20:46:20] Hard to find these links [20:46:37] greenrosetta which [20:46:44] manage projects page [20:46:48] greenrosetta wm-bot contains a lot of links and you can add them [20:47:03] !manage is https://wikitech.wikimedia.org/wiki/Special:NovaProject [20:47:03] Key was added [20:47:04] never used wm-bot [20:47:16] !manage [20:47:16] https://wikitech.wikimedia.org/wiki/Special:NovaProject [20:47:44] !wm-bot | greenrosetta [20:47:44] greenrosetta: http://meta.wikimedia.org/wiki/WM-Bot [20:48:52] ok, so I added a project called "stub" [20:48:57] now I just "become stub" [20:48:58] ? [20:48:58] !nyan [20:49:04] :( [20:49:16] greenrosetta yes [20:49:27] ah.. have to log out first [20:49:52] !nyan is ~=[,,_,,]:3 [20:49:52] Key was added [20:49:58] !nyan | petan [20:49:58] petan: ~=[,,_,,]:3 [20:52:56] Ryan_Lane ping [20:53:12] Ryan_Lane ca t ssh to instance :( [20:53:54] connection refused [20:54:02] but ping work [20:55:06] paravoid ^ [20:59:40] Warning: There is 1 user waiting for shell: Jose.canciani (waiting 13 minutes) [21:05:10] Coren: "ls -dl /data/project/wikidatastats": Seems some uid is screwed up. [21:13:15] Warning: There is 1 user waiting for shell: Jose.canciani (waiting 27 minutes) [21:26:19] petan: to which instance? [21:26:31] bots-login :/ [21:26:46] Warning: There is 1 user waiting for shell: Jose.canciani (waiting 40 minutes) [21:29:07] petan: I'm getting connection refused on 22 [21:29:13] I'd imagine that it OOM'd [21:29:15] same [21:29:16] reboot it [21:29:20] I did just reboot it [21:29:21] twice [21:29:25] oh? [21:29:47] there was nothing running [21:29:53] I just rebooted it and since then it happen [21:30:27] there is almost no console output now [21:30:29] dunno why [21:30:33] I just see login screen [21:30:42] there's nothing in the console log at all [21:30:49] why there is no full log? [21:30:51] did you change the kernel on this? [21:31:46] yes, it downloaded update in aptitude [21:31:56] I built instance, then I downloaded all updates [21:31:58] you accepted defaults? [21:32:06] yes [21:32:13] hm [21:32:13] I did just aptitude upgrade -y [21:32:17] * Ryan_Lane nods [21:32:43] how is kernel related to ssh o.O [21:32:48] that instance seems to work [21:32:50] I can ping it [21:33:01] I believe that local terminal is working as well [21:33:03] I have a feeling it didn't actually boot [21:33:06] just ssh got borked [21:33:07] and no, it's not [21:33:13] I can ping [21:33:32] now I can [21:33:34] cant [21:33:37] but before it worked [21:33:47] I just did a virsh destroy [21:33:48] don't reboot [21:33:52] or you'll corrupt it [21:33:57] I mounted the disk [21:34:50] ssh wasn't installed [21:34:51] weird [21:35:43] huh [21:35:51] but I did login to it before [21:36:01] that new image is kind of borked too [21:36:09] it has /dev/vda2 in /etc/fstab [21:36:14] but it doesn exist [21:36:40] oh? [21:36:46] please enter a bug [21:36:51] so it's missing the 2nd disk? [21:37:01] no [21:37:08] /dev/vdb is there [21:37:13] ah. ok [21:37:15] but /dev/vda2 is not [21:37:20] yeah, enter a bug [21:37:25] !br [21:37:25] oh no, so fix it! dam you [21:37:33] wtf [21:37:39] @infobot-detail br [21:37:39] There is no such a key [21:37:48] @infobot-detail !br [21:37:48] There is no such a key [21:37:58] !rb [21:37:58] broken? report a bug: https://bugzilla.wikimedia.org/enter_bug.cgi?product=Wikimedia%20Labs [21:38:02] oh lol [21:38:23] oh. bleh. ssh is a meta package [21:38:53] I don't think this booted properly [21:39:00] [bz] (NEW - created by: Peter Bena, priority: Unprioritized - normal) [Bug 48058] /etc/fstab contain non existent partition - https://bugzilla.wikimedia.org/show_bug.cgi?id=48058 [21:40:21] Warning: There is 1 user waiting for shell: Jose.canciani (waiting 54 minutes) [21:40:42] bots-secure:/mnt/bsql /mnt/secure nfs rw,addr=10.4.0.54 0 0 [21:40:46] is that working? [21:40:54] otherwise it won't boot [21:40:59] I'm going to comment it out for now [21:42:35] ah. shit. I changed the resolv.conf and forgot to change it back [21:43:03] aha... that might have been problem... [21:43:06] no [21:43:09] I just did that [21:43:24] so it wasnt nfs? [21:43:28] seems not [21:47:17] is this a new image? [21:47:27] ah. seems it is [21:48:05] I guess I should create an instance, upgrade the kernel and make sure it still works [21:49:03] hm [21:49:20] it says /dev/sda1 in the fstab and not vda1 [21:49:33] it also doesn't use a label [21:51:54] hm. weird [21:51:59] I bet it installed grub into the wrong place [21:53:47] Warning: There is 1 user waiting for shell: Jose.canciani (waiting 67 minutes) [21:57:08] upgrading the kernel on this new instance I created worked fine [21:58:48] I think cloud-init is fucking things up [21:59:20] ok so let me know when I can recreate it and be sure it will work :D [21:59:28] don't recreate it [21:59:45] well, it wouldnt be much work... but ok [21:59:53] I want to figure out what's wrong with it [22:00:04] I'm purging cloud-init from it [22:00:35] there's obviously some things to fix in this image [22:00:44] device names for sure [22:01:01] / is also an incorrect size [22:02:44] Coren/petan: Could you grep the httpd error log on Tools for "flickr2commons"? [22:02:55] sure [22:03:22] @labs-resolve tools- [22:03:22] I don't know this instance - aren't you are looking for: I-00000515 (webtools-odie), I-000005c9 (webtools-login), I-000005ca (webtools-apache-1), I-000005cb (webtools-rr), I-000005f9 (tools-login), I-00000600 (tools-webproxy), I-00000604 (tools-exec-01), I-00000608 (tools-webserver-01), [22:04:53] File does not exist: /data/project/flickr2commons/cgi-bin/upload_from_url1.php [22:05:45] [Fri May 03 21:49:25 2013] [error] [client 10.4.1.89] SoftException in Application.cpp:350: UID of script "/data/project/flickr2commons/cgi-bin/upload_from_url.php" is smaller than min_uid [22:06:23] there is a ton of errors, if you want I can grep them all to a file [22:06:32] scfc_de ^ [22:07:21] Warning: There is 1 user waiting for shell: Jose.canciani (waiting 81 minutes) [22:08:08] petan: Ah, that would be suphp barking. [22:08:14] Thanks! [22:08:23] you are welcome :)) [22:09:19] ok. fuck it [22:09:26] I can't figure out what's wrong with it [22:09:50] it's not logging to anything when it boots [22:09:54] something in the boot stage is hanging [22:09:57] no clue which service it is [22:10:28] did you remove that nfs part [22:10:31] yes [22:10:40] maybe it's the lvm stuff? [22:10:42] let me try that [22:11:03] hmm, it could be but... there is nothing important in vg, just swap [22:11:16] you can disable that as well, just to see [22:11:26] but on previous image lvm worked [22:11:27] wait [22:11:30] /dev/vg/lvol0 /mnt/share auto [22:11:36] that's missing stuff [22:12:05] oh true, I forgot to create fs there :x [22:12:06] I'm betting that's it [22:12:11] it's also missing info [22:12:17] you left out three columns [22:12:28] oh, wrong paste [22:12:30] I guess [22:12:34] swap is on /dev/vda2 [22:12:35] btw [22:12:36] petan: can you delete a couple of groups please? [22:12:39] 1GB of swap [22:12:44] I need to fix the fstab [22:12:49] for the new images [22:12:50] aha [22:12:54] didn't know [22:12:57] "local-foobar" and "local-stub" [22:12:57] did you look at /proc/partitions? :) [22:13:08] that was it [22:13:11] it was the fstab [22:13:21] unfortunately I think I destroyed its ssh keys [22:13:30] I purged and reinstalled ssh [22:13:54] so, lesson of the day. fucking up fstab will make systems not boot ;) [22:14:04] heh :) [22:14:14] well, it should have been in console ;) [22:14:23] I couldn't see it [22:14:24] no [22:14:33] why it doesn't write boot errors to console... [22:14:40] it'll never write that to the console [22:14:41] I thought that is what console is for [22:14:50] the console on the new images is also incorrect [22:14:53] couldn't figure that out [22:15:20] I think grub needs to be modified to use the right console, but it's not [22:15:24] greenrosetta I was about to go sleep [22:15:29] np [22:15:31] greenrosetta I can do that - but... [22:15:36] I can't figure out which console it is [22:15:44] won't be able to recreate them if you needed [22:15:49] because I won't be there then [22:15:50] no probelm, I have eveyrthing I need [22:18:00] Ryan_Lane can I put cloudinit back or not? [22:18:15] hold on [22:18:27] cloud-init and cloud-utils [22:18:32] I removed both [22:18:46] they don't actually do anything once the instance is up, but it's good to be consistent [22:19:33] ok [22:20:47] Warning: There is 1 user waiting for shell: Jose.canciani (waiting 94 minutes) [22:29:42] petan: is it possible to use filezilla on tools? [22:30:40] mysql is giving me an auth erro [22:31:10] ERROR 1045 (28000): Access denied for user 'foobar'@'tools-login.pmtpa.wmflabs' (using password: YES) [22:31:48] !manage [22:31:49] https://wikitech.wikimedia.org/wiki/Special:NovaProject [22:34:17] Warning: There is 1 user waiting for shell: Jose.canciani (waiting 108 minutes) [22:37:05] I created a new group called "fubar" and after becoming MySQL also gives an auth error, so there is either a delay or a bug [22:38:41] Coren: Is greenrosetta's problem re DB auth still the same as the other day? [22:39:27] All I did was create a new group, became, and did MySLQ [22:40:42] i'm with the same problem with myswl [22:40:47] *mysql [22:41:52] So it's probably still the same :-). [22:41:53] coren: can we set LC_ALL to en_US.UTF-8 on the grid nodes (i.e. the environment that is available for the jobs) as it is on -login? [22:43:36] oh, well it's LANG [22:47:39] Warning: There is 1 user waiting for shell: Jose.canciani (waiting 121 minutes) [23:01:01] Warning: There is 1 user waiting for shell: Jose.canciani (waiting 134 minutes) [23:14:31] Warning: There is 1 user waiting for shell: Jose.canciani (waiting 148 minutes) [23:27:58] Warning: There is 1 user waiting for shell: Jose.canciani (waiting 161 minutes) [23:36:19] maybe we can discuss that later, bye [23:41:24] Warning: There is 1 user waiting for shell: Jose.canciani (waiting 175 minutes) [23:54:54] Warning: There is 1 user waiting for shell: Jose.canciani (waiting 188 minutes)