[05:43:31] !log planet rebooting venus - it's unresponsive to ssh [05:43:32] Logged the message, Master [08:35:34] @notify ceradon [08:35:34] I will notify you, when I see ceradon around here [08:49:46] petan: ceradon == euphoria|jacked on IRC right now [08:49:59] @seen ceradon [08:49:59] petan: Last time I saw ceradon they were quitting the network with reason: Quit: http://www.kiwiirc.com/ - A hand crafted IRC client N/A at 3/20/2013 10:10:21 PM (10:39:38.5633840 ago) [08:50:05] really? [08:50:13] I don't see him [08:50:51] @seenrx euphoria [08:50:51] petan: Last time I saw euphoria they were changing the nickname to Guest82676, but Guest82676 is no longer in channel because he quitted the network 1.08:11:59.1560580 ago. The nick change was done in #wikimedia-office at 3/17/2013 2:22:11 AM (4.06:28:39.9859100 ago) (multiple results were found: euphoria|jacked) [08:52:31] he's in -en-afc [13:59:02] hey all! [13:59:12] hi [13:59:22] i have been away for a few days [13:59:25] I know [14:01:10] hmm, doing ls in my home directory seems to take an age :p [14:01:21] I have a feeling sge has been dumping files in it :/ [14:01:23] hehe [14:01:24] no wonder [14:01:35] right... [14:02:04] problem is I have no idea what the files are called to delete them ;p [14:04:04] petan: any idea? :/ [14:05:05] addshore need help removing them? [14:05:18] the files are named as the script you are running [14:10:50] !log bots petrb: sudo apt-get install uuid-dev libattr1-dev zlib1g-dev libacl1-dev e2fslibs-dev libblkid-dev liblzo2-dev [14:10:52] Logged the message, Master [14:13:00] !log bots root: scrubbing fs on bsql01 [14:13:02] Logged the message, Master [14:17:01] I should have know that selfhealing fs don't need fsck :P [14:38:33] addshore did you sort it out? [14:39:45] addshore you need some better irc client :P [15:00:38] petan: I don't know if you read the talk between Coren and me: I have created a list with question I have about Tools and volunteer admins: https://wikitech.wikimedia.org/wiki/User:Jan/Tools#Sysadmin_doc [15:00:44] Maybe you want to add something [15:03:21] Jan_Luca: Have you looked over my docs-in-progress? [15:03:50] Coren: When you mean https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Help then yes [15:04:02] Incidentally, I'm not saying anything on the "how-to-login" etc since I'm trying to find a better way to do it than the current bastion. [15:04:27] Jan_Luca no I haven't yet [15:04:31] Jan_Luca: That's what I meant, yes. I'm going to work on it some more today making sure your own Qs are answered too. [15:05:15] Coren: For a big project like Tools I think we could setup some direct login instance (e.g. tools-login.wmflabs.org) [15:05:45] Jan_Luca: That's my preferred solution. [15:07:06] petan: Feel free to add something :-) [15:18:52] petan not yet :P [15:18:56] I got distracted by food :) [15:19:03] k [15:22:21] andrewbogott, Ryan_Lane: https://bugzilla.wikimedia.org/show_bug.cgi?id=46356 [15:22:55] addshore@bots-bnr1:~$ rm -v wd.g.* [15:22:56] -bash: /bin/rm: Argument list too long [15:22:57] nice.. [15:23:35] addshore try: rm -v `ls | grep wd.g | head -n 800` [15:23:40] :P [15:23:45] petan: :( [15:23:48] that will remove first 800 :D [15:23:53] andrewbogott [15:23:54] ? [15:24:01] addshore or if you want I will delete them [15:24:03] Just, split brain :( [15:24:11] I know, but how to sort it out? [15:25:00] I've watched ryan do it a couple of times, lemme see if I remember [15:25:11] (after I finish breakfast) [15:27:22] petan, is it happening to new files as well, or just that one? [15:27:39] andrewbogott I didn't notice similar error recently [15:27:50] but I would like to have all files working [15:27:59] it's possible that once you fix this we discover another one [15:28:07] the job which is processing them die on first problem [15:29:40] ok [15:30:04] Coren what if multiple people want to operate one bot? [15:30:04] andrewbogott: Never switch the PC on when you want to have a nice breakfast ;-) [15:30:20] yeah, serves me right [15:30:36] petan: They can; the interface to do it from wikitech isn't there yet but they just have to be added to the right group. [15:30:43] k [15:39:29] Coren: How should Tools be integrated into Wikitech? With an extra special page? [15:40:23] Jan_Luca: Actually, we're in a sprint right now to code that in. It's a new part of Special:NovaProject. [15:40:45] (The mechanism for local service groups is useful enough in general that it will be generalized) [15:41:26] Coren: Oh, is that what Ryan, andrew and you talk about yesterday? [15:41:38] Jan_Luca: Yeppers. :-) [15:41:59] Coren: I only read something about NovaProject and a test instance :-) [15:44:07] Coren: I'm interested what will be the result ;-) [15:44:42] petan, I think it'll take me as long to figure this out as it'll take ryan to get to work… ping the two of us again when he shows up? (And then I'll pester him to document this time.) [15:45:08] @notify Ryan_Lane [15:45:08] This user is now online in #wikimedia-dev so I will let you know when they show some activity (talk etc) [15:45:09] andrewbogott: Do you have the bits to increment my floating address quota on tools? Leslie had to run away before she firgured out how. :-) [15:45:25] Coren, yes, how many do you need? [15:45:31] andrewbogott: Just the one. [15:45:44] Danke. [15:46:02] Coren: Oh we change the language :-P [15:46:39] Jan_Luca: I often randomize language of pleasantries. :-) [15:46:47] Coren: OK, should be set [15:47:51] Coren: Danks mir f�r de Erkl�rung :-P [15:48:06] andrewbogott: And it already floats no more. :-0 [15:48:26] Jan_Luca: ay ay. Your character encoding, she is broken! :-) [15:50:29] Coren: Does it now work: f�r [15:50:46] Jan_Luca: Not for me. Are you using UTF-8? [15:50:53] yes [15:50:57] Odd. [15:51:29] Do you see this properly? Marc-André, Gdańsk, 謝謝 [15:51:35] yes [15:51:52] für [15:52:04] Coren: I see no problem [15:52:15] Jan_Luca I don't see symbols from you either [15:52:23] That's really strange; yours have unicode substitution chars instead: f�r [15:52:47] I see a square with "?" [15:52:59] Coren: Ok, it seems my client can read your encoding but does not send the right [15:53:13] petan: Yeah, that's the "this wasn't a good UTF-8 codepoint" character [15:54:44] * hashar 🍺 time [15:55:21] It seems that I have to change something if I want to speak German via IRC (dat babbel klappt net :-( ) [15:56:26] How does ERC cope: äöü. [15:57:39] Coren: Maybe it works now: für [15:57:51] Jan_Luca: Yep; that works. [15:58:10] Coren: OK, I had to restart my client [15:59:56] There. tools-login.wmflabs.org [16:02:42] * Coren wishes there was IPv6 in the labs. [16:03:36] Coren: Oh this is almost a miracle: it works :-P [16:04:04] Jan_Luca: I would expect that having ssh working may be useful, but hardly miraculous. :-) [16:04:48] Coren: I mean that something works without problems is a miracle ;-) [16:11:07] addshore did you manage to delete these files? [16:11:20] you can also use midnight commander [16:11:28] it's on -gs [16:11:59] Oh wow. I haven't used mc in nearly ten years that I can recall. :-) [16:12:10] it's useful for some stuff [16:12:45] but the reason why it's installed is that toolserver people wished it [16:13:54] Coren, petan: I haven't known that tool till today... [16:15:33] !log bots Added tools-login.wmflabs.org public IP for the tools-login instance and allowed incoming ssh to it. [16:15:36] Logged the message, Master [16:15:44] Idiot. [16:15:49] !log tools Added tools-login.wmflabs.org public IP for the tools-login instance and allowed incoming ssh to it. [16:15:50] Logged the message, Master [16:21:40] Coren no worries, nobody except for me is reading sal anyway :D [16:22:01] unfortunatelly almost no one is writing to it as well [16:22:29] Coren: A question (I want to unterstand SGE ;-) ): Why do you set the h_vmem values of the queues and exec-server to infinity? When I unterstand that right, I could request e.g 20gb of memory [16:23:05] Jan_Luca yes, but that would mean your job would be never scheduled until there would be a single instance with 20gb of ram free [16:23:24] which wouldn't because so big instances are not available yet [16:23:53] petan: OK but the h_vmem of the exec server is infinity, too? [16:24:00] that doesn't matter [16:24:16] the job will not be submitted there unless the memory you request is available [16:24:52] at some point it matters [16:25:10] because there should be probably some limit like system_max - 400mb [16:25:20] given that system daemons will never need more than 400mb of ram [16:25:53] that's what I did on bots [16:26:15] petan: Thank you, now I understand SGE better [16:26:30] Coren believes there will be never problem with system daemons and if so, the system is supposed to die anyway :> [16:26:36] or that's how I understood it [16:26:54] despite I disagree with that... [16:27:06] I would rather heal that system before it dies... [16:28:19] petan: You could setup a instance with 20gb if you use swap ;-) [16:28:30] yes actually that's what we have on bots [16:28:51] in that case h_vmem should be replaced with h_mem I think or something like that [16:28:59] maybe pmem? :P [16:29:01] no idea [16:29:08] hard to find a good documentation for that [16:29:27] there is a config file that contains all these definitions [16:30:23] petan: qconf -mc [16:30:36] The complex configuration should not be accessed directly. [16:30:36] In order to add or modify complex entries, the qconf(1) [16:30:36] options -Mc and -mc should be used instead. While the -Mc [16:30:36] option takes a complex configuration file as an argument and [16:30:36] overrides the current configuration, the -mc option bring up [16:30:36] nope [16:30:37] an editor filled in with the current complex configuration. [16:30:50] there is another one in /etc [16:31:06] oh wait [16:31:08] yes that's it [16:32:02] petan: I remember that because I read parts of the SGE man yesterday [16:33:55] petan: When I unterstand that right you could define own attributes and attach them to hosts or queues [16:34:08] probably [16:35:42] petan: I think that is what the TS people do because there you can request access for user-SQL or wikimedia-SQL-replicas, .. [16:36:34] petan: https://wiki.toolserver.org/view/Job_scheduling#Optional_resources [16:36:59] mhmmm [16:51:17] * Coren returns from lunch. [16:52:02] Jan_Luca: What do you mean? The exec instance is at 7G [16:52:51] Coren: petan answered my question [16:56:02] complex_values h_vmem=7G,slots=32 [16:57:18] Coren: I mean: h_vmem INFINITY [16:57:42] Jan_Luca: You're not looking at the right config, I think. :-) [16:57:52] Jan_Luca: Are you looking at the queue? [16:58:05] qconf -me tools-exec-01 [16:58:09] Coren: oh, yes you're right [16:58:20] I don't like qconf [16:58:37] It's... very very Sun. :-) [16:59:29] I like it [16:59:34] Coren: Why they not use long parameter for qconf then it would be clearer [16:59:38] it just give you a conf file in editor :) better than shiny gui that doesn't allow you to do anything [17:00:07] petan: I mean the names of the parameters are sometimes very unclear [17:00:11] yes [17:00:13] that's true [17:00:25] it would be cool if there was a cool manual page for them [17:00:25] Jan_Luca: Wrong epoch. SGE was written, originally, by people from vax background who did not understand / did not care about Unix conventions all that much. [17:00:53] Jan_Luca if you read my source code you would find out I am from a similar epoch [17:01:02] my variable names are unclear even to me some times [17:01:21] petan: Oh, do you have some public code to look into :-P [17:01:23] I could find some nice example maybe on github... :P [17:01:29] me? tons [17:01:32] I'm also from that epoch, but SGE is out there even from me. :-) [17:01:52] Jan_Luca: https://github.com/benapetr [17:02:29] i gtg now [17:02:41] petan: My code is not the nicest, too: https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/tools/commonshelper2.git;a=tree;h=HEAD;hb=HEAD [17:04:05] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Help was modified, changed by MPelletier (WMF) link https://www.mediawiki.org/w/index.php?diff=662122 edit summary: [+13] + section [17:21:58] is there an interwiki for wikitech? [17:35:05] Coren: wikitech: [17:35:24] Coren: For all interwiki prefixes: https://www.mediawiki.org/wiki/Special:Interwiki [17:35:26] Found it since, thanks. :-) [18:01:09] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Help was modified, changed by Coren link https://www.mediawiki.org/w/index.php?diff=662140 edit summary: [+1067] /* Access */ (really) basic instructions [18:02:05] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Help was modified, changed by Coren link https://www.mediawiki.org/w/index.php?diff=662141 edit summary: [+9] /* Getting access to the Tool Labs */ +iw [19:07:15] * Coren eats his own dogfood: time to move CSBot to tools- [19:08:32] Coren: The dogfood is sometimes better than the human food ;-) [19:26:59] Coren: You should create a link in your help to https://wikitech.wikimedia.org/wiki/Help:Access and https://wikitech.wikimedia.org/wiki/Help:Getting_Started [19:28:09] so you don't have to write the complete access docu again ;-) [19:28:10] Jan_Luca: Not entirely sure why, but I know Ryan_Lane doesn't want to mess with docs on wikitech atm; I think that's because there's an impending copy that'll overwrite everything. [19:28:25] Oh, wait, I read you backwards. [19:28:29] Yes, I will. [19:30:03] Coren: Maybe we should create some kind of screenshot documation when the system is setted up [19:30:21] Is would be nice. Especially for the Putty users. [19:30:37] Coren: Oh, here is one :-) [19:31:00] there's a page for putty users [19:31:03] with screenshots [19:31:12] Ryan_Lane: Where? [19:31:35] lemme find it [19:32:27] Favorite 404.html for my server: [19:32:41] Twenty score and four errors! [19:32:41] [19:32:41] Thou must eye thine URL close, for mayhaps tis whyfore it eludest thee. [19:33:25] I just love faux-Victorian. :-) [19:33:25] https://wikitech.wikimedia.org/wiki/Help:Access_to_instances_with_PuTTY_and_WinSCP [19:33:50] Ryan_Lane: With plink, natch [19:33:54] linked to from here: https://wikitech.wikimedia.org/wiki/Help:Putty [19:34:15] Not that needed anymore given tools-login [19:34:23] that's linked to from access and getting started ;) [20:00:07] !putty [20:00:07] official site: http://www.chiark.greenend.org.uk/~sgtatham/putty/ | how to tunnel - http://oldsite.precedence.co.uk/nc/putty.html [20:00:43] !putty del [20:00:44] Successfully removed putty [20:00:55] !putty is https://wikitech.wikimedia.org/wiki/Help:Access_to_instances_with_PuTTY_and_WinSCP [20:00:55] Key was added [20:05:58] http://tools.wmflabs.org/csbot/csb.php [22:35:31] !log deployment-prep both apaches gives out Error 500 so beta is now serving blank pages. [22:35:33] Logged the message, Master [22:37:47] !log deployment-prep -bastion : stopping udp2log that prevents udp2log-mw from running :/ See {{bug|38995}} [22:37:50] Logged the message, Master [22:40:27] chrismcmahon: sorry I have no idea what might be wrong on beta [22:49:19] [2013-03-21 22:42:37.440262] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-deployment-prep-project-client-0: remote operation failed: Stale NFS file handle. Path: /apache/common-local/php-master/extensions/UploadWizard/UploadWizardHooks.php (1a42fed1-04b8-4dae-bb9f-593d41d097d0) [22:49:20] [2013-03-21 22:42:37.440326] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-deployment-prep-project-client-1: remote operation failed: Stale NFS file handle. Path: /apache/common-local/php-master/extensions/UploadWizard/UploadWizardHooks.php (1a42fed1-04b8-4dae-bb9f-593d41d097d0) [22:49:23] bohbohbbbhah [22:57:09] I am out [22:57:12] :( [23:05:30] chrismcmahon: I found the root cause :-] [23:05:31] I think [23:05:44] hashar: \0/ [23:06:07] Fatal Error : Failed opening required '/usr/local/apache/common-local/php-master/extensions/E3Experiments/Experiments.php' [23:06:13] I have sent a mail to ori about it [23:06:15] you are on cc [23:06:43] took me half an hour to find out where the PHP error log end up being logged :/ [23:07:27] chrismcmahon: will get fixed whenever ori comes back :-] [23:07:37] thanks hashar [23:07:58] I don't want to force merge something on E3Experiment extension nor revert their change [23:08:03] so yeah, that should self fix [23:08:16] at least beta catched a nasty fatal error [23:09:17] hashar: beta is getting better about that, thanks [23:09:54] I wish it displayed the fatal error [23:10:46] Mar 21 23:10:28 i-0000031a apache2: PHP Warning: require_once(/usr/local/apache/common-local/php-fatal-error.html) [function.require-once]: failed to open stream: No such file or directory in /data/project/apache/common-local/wmf-config/CommonSettings.php on line 2482 [23:10:51] ahh no php-fatal-error.html [23:12:31] ah that points to the doc hmm [23:12:33] or something [23:12:54] well bed time for now [23:12:56] * hashar waves [23:13:03] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Help was modified, changed by Dcoetzee link https://www.mediawiki.org/w/index.php?diff=662262 edit summary: [+0] /* Getting access to the Tool Labs */ labs, not lags [23:13:28] legoktm: You can now ssh straight to tools-login.wmflabs.org btw. :-) [23:13:38] Oh cool [23:13:51] I have proxycommand set up so it doesn't really make a difference :P [23:14:12] legoktm: You'll like the extra performance. I know I did. :-) [23:14:33] Oh but I can probably use a graphical sftp program now... [23:15:17] legoktm: Indeed you can. :-P [23:15:24] :D [23:30:20] !log tools turned on interpretation of .py as CGI by default on tools-webserver-* to parallel .php [23:30:22] Logged the message, Master [23:47:29] * Coren points at the new vict^H^H^H^H guinea pi^W^W tools dev! :-) [23:48:19] * Dcoetzee waves! [23:48:34] Might as well talk in here [23:48:41] Indeed. [23:48:48] Is Labs providing anything in the way of version control? [23:49:12] You can get a git repo on gerrit by filing a request [23:49:19] There's a gerrit made avaliable, but it's not automatic. [23:49:29] Automatic would be better [23:49:41] If they don't use it, no big deal [23:50:03] It would, but gerrit doesn't know how to speak to Openstack, and is a pain to manipulate from outside itself. [23:50:16] Integration woes [23:50:17] I get it [23:50:27] Maybe later [23:51:32] Hmm what's the easiest way to transfer files from my user account to a project account? Besides just chmoding them world readable? [23:52:06] ...apparently chmoding them world readable doesn't work, my home directory is modded private [23:52:25] Can try to move them through /tmp I guess [23:53:16] That worked but was awkward, project user doesn't have write access to /tmp [23:53:20] Which is also weird [23:53:38] So they wouldn't be able to move files back that way [23:53:55] Anyway http://tools.wmflabs.org/dupdet/ is up, enjoy :-) [23:54:03] Damnit wrong tool [23:54:05] I'm an idiot [23:54:07] Hold on [23:55:54] There http://tools.wmflabs.org/dupdet/ [23:55:55] Now is up [23:56:56] Ah, you used the happy fun DirectoryIndex :-) [23:57:19] Yeah, moving files around is something I'm not quite satisfied with atm. [23:58:06] Would be nice if there were a user group that all members of a project belonged to, and a directory they all had read/write access to [23:58:12] As your project user, you could have written the files to the tool's home, but then you'd have had to copy them again to change ownership. I'm probably going to make a suid chown-equivalent for that purpose though. [23:58:23] Dcoetzee: You do, and they do. [23:58:28] Ah I see, didn't know it'd let me do that [23:58:30] local-tool [23:58:39] and ~local-tool/ is group-writable [23:58:43] Aha [23:59:13] Yay works [23:59:22] But then the files are owned by the maintainers, which is problematic. You can mv-cp-rm around the ownership, but that is teh sux.