[00:18:46] greenrosetta: The web server checks of accessibility using world read permission; only your scripts are allowed to use owner and group. [00:19:03] greenrosetta: This is to prevent making files accidentally accessible through the web that contain credentials. [00:22:41] Coren: check this out.... [00:22:43] http://tools.wmflabs.org/quickquery/ [00:22:49] run "select * from City" [00:32:29] allright im off for the night. [06:40:56] Hi, is it possible to rename my user on wikitech? ie. "Full Name" on gerrit is pretty ugly now :( [07:56:56] New patchset: ArielGlenn; "Added labs redis password" [labs/private] (master) - https://gerrit.wikimedia.org/r/64264 [07:58:16] Change merged: ArielGlenn; [labs/private] (master) - https://gerrit.wikimedia.org/r/64264 [09:28:39] Coren now I am here [09:35:39] Damianz there is folder mysql in bots project that has 205gb [09:35:41] what is it? [09:35:56] In /data/project/? [09:35:57] backups? [09:35:58] yes [09:35:59] Probably mysql backups [09:36:01] aha [09:36:03] ok [09:36:22] wow public_html has 4gb [12:52:27] Coren Coren [12:52:34] ? [12:52:36] hi :D [12:52:40] hoy! [12:52:42] I need you to merge patch [12:52:55] Linky? [12:52:57] !g 64285 [12:52:57] https://gerrit.wikimedia.org/r/#q,64285,n,z [12:53:30] commit-msg is shit but because it took me like 1 hour to actually execute that push and I had to write it many times [12:53:43] first one was beautiful :> [12:55:19] Hm. Actually, I'd really rather you didn't do this in profile.d [12:55:30] ok, where should it be then? :P [12:55:42] it needs to be executed as user who is logging in [12:55:47] motd is ran by root [12:55:48] That can mess up non-interactive login shells. [12:56:07] Do it in update-motd, just su to $LOGNAME. [12:56:09] I thought I somehow patched it against non-interactive logins... mhm [12:56:19] aha [13:02:09] Coren it doesn't work :) [13:02:10] :( [13:02:16] are you sure there is LOGNAME? [13:03:03] Oh, maybe not by the time update-motd is called. [13:03:14] But why run it as the user, incidentally? [13:04:09] because it needs to check a) if you are sysadmin b) if there is ~/.suppresstips [13:04:17] Hm. [13:04:39] Well, you can stuff it in profile.d iff you guard carefully for an interactive session. [13:04:43] no, I just checked there is no variable indicating who you are when motd is being created :/ [13:04:59] well /it is/ guarded, dunno if well, but somehow it is :P [13:05:05] you can try it so far it was there [13:05:11] I just inserted it to puppet now [13:05:22] it was on login for weeks, or at least 1 week :P [13:05:56] petan: That's because nobody yet reported a problem. Don't confuse non-interactive with just non-login. :-) [13:06:16] how do I check if login is non-interactive? is there a variable for that? [13:06:19] petan: Check for tty in profile-tool-labs.sh [13:06:24] man tty [13:06:46] Specifically, tty -s [13:07:13] if tty -s; then do_stuff;fi [13:07:18] Warning: There is 1 user waiting for shell: Liuxinyu970226 (waiting 0 minutes) [13:07:27] are you sure check of $SHLVL isn't enough? [13:07:51] hm, maybe not... [13:08:14] anyway - that ugly ssh banner is displaying on non-interactive as well and it suck :P [13:09:57] The /etc/issue? Yeah. That's a Labs requirement for bastions. It cuts down support requests by half. :-) [13:10:22] I'm mostly worried about stuff sent /between/ tools instances. [13:11:18] Coren, are all of my bots running in the correct location now? [13:12:48] Coren ok fixed - check the source: /data/project/.system/bin/tips.sh [13:12:52] Cyberpower678: I don't see any running in the wrong place, at least. [13:13:03] :) [13:13:05] the other script I pushed to gerrit is still same... no need to change that [13:14:01] petan: No, please check for tty in that script; I want to be 100% sure it isn't even /tried/ if there isn't a tty, in case changes in tips.sh cause output in the future. [13:19:57] done [13:20:42] Warning: There is 1 user waiting for shell: Liuxinyu970226 (waiting 13 minutes) [13:20:59] merged [13:23:22] thanks [13:23:53] is it safe to run puppetd on login? [13:24:00] I rather ask :D [13:24:17] It is, but some would say that patience is a virtue. :-) [13:25:27] well, I need to switch this with previous hardcoded line... so I would rather make it work without outages of script... [13:26:00] how often puppet run? [13:26:20] Alternately, just remove the hardcoded line everywhere. People can live a little bit without tips at worse. :-) It runs twice/hour [13:26:39] so far it was on login only... [13:26:40] and -de [13:26:41] v [13:26:45] Ah. :-) [13:26:57] are you sure people can live without my tips? ;P [13:27:49] I see only gifti has .suppresstips :P [13:27:57] other people clearly lovin' it [13:28:42] Or they don't know how to turn it off. :-) [13:28:43] but true reason why I made this project wide is that sysadmin motd, that could be useful in future [13:28:52] one of tips is how to turn tips off [13:29:51] it is less annoying than ssh banner which can't be turned off [13:30:14] and at least same useful :P [13:34:16] Warning: There is 1 user waiting for shell: Liuxinyu970226 (waiting 27 minutes) [13:36:18] !rq Liuxinyu970226 [13:36:19] https://wikitech.wikimedia.org/wiki/Shell_Request/?action=edit https://wikitech.wikimedia.org/wiki/User_talk:?action=edit§ion=new&preload=Template:ShellGranted https://wikitech.wikimedia.org/wiki/Special:UserRights/ [13:36:30] !rq Liuxinyu970226 [13:36:30] https://wikitech.wikimedia.org/wiki/Shell_Request/Liuxinyu970226?action=edit https://wikitech.wikimedia.org/wiki/User_talk:Liuxinyu970226?action=edit§ion=new&preload=Template:ShellGranted https://wikitech.wikimedia.org/wiki/Special:UserRights/Liuxinyu970226 [13:49:20] !bots [13:49:20] http://www.mediawiki.org/wiki/Wikimedia_Labs/Create_a_bot_running_infrastructure proposal for bots [14:43:24] Coren I think exec nodes would deserve /tmp on separate volume as well... maybe vdb could be mounted to /tmp on them as default? there is over 10k files and new images have quite small rootfs [14:44:04] well, actually 10gb, not really that small but idk if we really want to use them for /tmp [14:44:52] I've LVM'ed vdb on the exec nodes, we could add a lv for /tmp [14:45:01] great [14:45:17] I'll set something up so it comes back with it Monday after the reboot. [14:45:19] did you lvmed it in puppet? [14:45:30] ok [14:45:38] Of course not. Never ever do LVM stuff in puppet; that's not idempotent. [14:45:42] well, actually being able to resize on demand would be best [14:45:52] heh, I agree [14:47:54] hm, I don't see any LVM installed there... [14:47:56] Ohwait. I lied. I didn't lvm the swap, I created a physical partition. Same idea though. [14:48:25] * Coren is handling it. [14:49:05] it's not really critical, but anyway... there is some 80gb of unused disk space and I think we could use it somehow effectively :] [14:49:39] Yeah, I got vdb2 on /mnt atm; we could simply move it to /tmp for next reboot. [14:50:14] ok, but don't forget to chmod 1777 it [14:50:47] maybe it would be easiest if you chmod 1777 the current /mnt, umount it and update /etc/fstab so that next reboot it's there [14:50:48] Yes, I know how /tmp works. :-) [14:50:54] :P [14:51:53] Done. We get /tmp on next boot. [14:52:03] yay [14:53:01] jemand da der Deutsch Spricht? [14:53:52] Ich spreche ein wenig. [15:31:21] Warning: There is 1 user waiting for shell: Li3939108 (waiting 0 minutes) [15:44:51] Warning: There is 1 user waiting for shell: Li3939108 (waiting 13 minutes) [15:58:21] Warning: There is 1 user waiting for shell: Li3939108 (waiting 27 minutes) [16:52:32] we are having an issue with a new user logging into parsoid.wmflabs.org: Unable to create and initialize directory /home/smith [16:52:49] is gluster having issues again? [16:54:04] <^demon> gluster always has issues :( [16:54:31] let me try a reboot.. [16:54:38] <^demon> But yes, I believe something was said earlier about a brick or somesuch. [16:55:00] so is it a transient or permanent (for now) issue? [16:55:29] <^demon> No clue. Coren? [16:56:10] Yeah, filesystems becoming readonly are a the primary symptom of a failed gluster brick. [16:57:35] Which project is readonly? [17:00:10] gwicke, which project? [17:00:21] andrewbogott: visualeditor [17:02:40] gwicke: Any better now? [17:17:41] hashar, is integration-home performing any better now? [17:24:17] !tr test [17:24:17] request page: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/test?action=edit talk page: https://wikitech.wikimedia.org/wiki/User_talk:test?action=edit§ion=new&preload=Template:ToolsGranted link: https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProject&action=addmember&projectname=tools [17:28:24] !tr Merlijn van Deen [17:28:24] request page: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Merlijn?action=edit talk page: https://wikitech.wikimedia.org/wiki/User_talk:Merlijn?action=edit§ion=new&preload=Template:ToolsGranted link: https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProject&action=addmember&projectname=tools [17:28:34] hm, that looks broken [17:28:43] !tr Merlijn_van_Deen [17:28:43] request page: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Merlijn_van_Deen?action=edit talk page: https://wikitech.wikimedia.org/wiki/User_talk:Merlijn_van_Deen?action=edit§ion=new&preload=Template:ToolsGranted link: https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProject&action=addmember&projectname=tools [17:57:58] @notify giftpflanze [17:57:58] I'll let you know when I see giftpflanze around here [18:15:15] hi ! would it be possible to have the httplib2 python library on bots-login ? it is present at bots-gs but there's a warning there telling "*** THIS SERVER WAS REPLACED BY bots-login DO NOT USE IT ***" [18:22:01] andrewbogott: the login worked now, thanks! [18:22:07] cool [18:47:51] heya, Ryan_Lane, does Reassociate IP in labsconsole work? [18:48:12] ah, i see, nm, it told me it failed [18:48:17] but it just deassociated [18:48:27] re-associating it again manually seems to have worked [18:50:44] hm [18:50:51] that should have worked [18:50:52] weird [19:22:32] did someone uninstall mysql? [19:30:20] OK guys who broke tools ? [19:30:36] yeah, mysql isnt working [19:31:12] greenrosetta: ive got an IRC bot that doesnt use mysql that just died [19:31:27] cant help you there [19:31:36] but since you got me started on python... check his out [19:31:38] http://tools.wmflabs.org/quickquery/cgi.py/results?query=select+%2A+from+City%3B&csv=bc676.csv [19:32:09] Coren: ^^ [19:32:16] Coren: mysql broken in tools? [19:32:20] Betacommand: in which way is it broken? [19:32:38] Betacommand: the irc server was rebooted for a security upgrade [19:32:43] Ah [19:32:50] mysql shouldn't be broken, thoguh [19:32:52] *though [19:33:01] No, his query seems to work. [19:33:40] Ryan_Lane: Speaking of IRC, did you manage to corner the Freenode peeps about the I-line? [19:34:19] Coren: is there a command to kill all items in your queue? [19:35:22] qdel -u you_user_name [19:51:56] * Damianz looks at http://tools.wmflabs.org/quickquery/cgi.py/results?query=select+%2A+from+City%3B&csv=bc676.csv and shudders [19:56:26] Ryan_Lane: it was an issue with irc.wikimedia not being up all the way [19:57:38] ah irc.wikimedia was down? that explains random complaining in logs and 'IM DIAHINGAGAARGAHGHAG' emails [20:15:49] Ryan_Lane, the latest version of the instance status wiki updater uses setuptools/entrypoints. Is there a standard way of packaging something like that for puppet? Should I just make a debian by hand that calls setuptools after installing? [20:35:10] * Damianz pokes Coren [20:35:27] * Coren pokes back! Poink! [20:37:39] * Damianz wonders how sexytime betwean Coren and mysql replication is going [20:38:03] Oh noes! You broke my concentration! Now I have to start over again. [20:38:43] If you need distracting I'll just get the rainbow pony out ;) [20:42:02] My neighbour has a crappy taste in music tonight [20:42:21] <^demon> s/tonight// [20:42:27] <^demon> ^ That's what I have to deal with [21:20:32] andrewbogott_afk: <3 [21:33:09] i'm having a problem on grid engine. I'm submitting this jsub cmd -> http://pastebin.com/mrkt8nDj [21:34:02] still, -cwd seems not to work, since all data form other files isn't loaded, and the outputs are going to my top folder [21:50:11] andrewbogott_afk: hm. I'm not very sure [21:50:28] andrewbogott_afk: I've never actually done that in python [22:25:02] andrewbogott_afk: $hostArr = $this->getSudoHosts( $this->projectName, $sudoer ); [22:25:17] that gets the instances from nova [22:25:26] which also pulls their host entries [22:25:41] you don't need to pull the puppet variable [22:26:20] you can use the instance name returned by openstack [22:30:07] alternatively, both the i-xxx record and the instance name record are in LDAP in the same attribute [22:30:24] I'd rather not use puppet info if possible [22:33:13] petan: it's a kernel update [22:33:28] just make sure all of your linux boxes are running the newest available kernel from your distro [23:34:56] Alchimista: You need to put the jsub options (-cwd, -o, etc.) to the left of the script name (aleph.py). [23:44:30] scfc_de: thanks, i've moved to qsub instead of jsub, but basicly the same command, the only thing that doesn't work is the path, only main script is copied to execution server, and it fails because it can't import the otheres, shouldnt the use of --cwd plus_path be enought? [23:46:06] oops, it's basicly: "qsub -cwd -j y -o /data/project/alchimista/aleph/aleph.out aleph.py" [23:46:35] Alchimista: And your trying to import Python modules? [23:47:24] scfc_de: modules, scripts or .ini files [23:49:53] Alchimista: For modules, you need to set PYTHONPATH. It's not inherited from your shell or crontab, so easiest advice is to follow John's on toolserver-l and sys.path.append("PATH_TO_PYWIKIFOLDER") in aleph.py