[02:59:29] Hello world! [04:01:56] Hello. The replicas are in a different datacenter than Labs, which has a major performance impact on crosswiki tools. Last month Marc mentioned plans to combine them in the medium term: http://lists.wikimedia.org/pipermail/labs-l/2013-September/001574.html . Do we know when that might occur, or is there a ticket I can watch? [04:28:31] Pathoschild: I'm not sure about the existence of a ticket (much internal ops stuff is tracked via RT instead) but the timeline for the move has labs moved before the end of the year. [04:29:41] Thanks. I just finished creating a tracking ticket we can link to: https://bugzilla.wikimedia.org/show_bug.cgi?id=55929 [04:33:56] (Actually, the most likely scenario is to create a new tool labs in eqiad then move the tools individually rather than attempt to move the current deployment -- net effect is the same but it means that tools can probably be moved early when it counts) [04:37:53] Sounds good. I can redeploy my tools when that happens, if you need a guinea pig. [08:55:40] tools is down [09:30:25] Steinsplitter: elaborate [09:37:47] https://bugzilla.wikimedia.org/show_bug.cgi?id=55933 [09:38:04] petan: ^ [10:12:10] yup, down. [12:09:58] Coren|Away, petan, https://tools.wmflabs.org 500's [12:10:35] oh, a bug has already been submitted [13:02:01] Only -02 was down, actually. Someone having fun with a DDOS [13:03:09] Something is hammering _hard_ on cluebot. [13:24:52] hello, How I can get access to dumps? Do I need to download them separately? [13:25:54] petan: ^ [13:25:56] hi! [13:32:15] never mind [13:32:17] solved [13:47:22] is there an equivalent of submit.toolserver.org where you're supposed to place your crontab? [13:47:51] in the tools project that is [13:50:23] johang, https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Scheduling_jobs_at_regular_intervals_with_cron [13:52:41] MartijnH: yes, but on which machine should I do this? tools-login? [13:53:24] johang: I think so. [13:53:24] I'm not 100% sure, but I think yeah, after you have become your toolname [13:53:51] I see [13:55:03] on toolserver the submit server was some kind of HA cluster so it was less likely crons on that machine would fail to launch. [13:58:11] I think Coren|Away is working on making cron distributed [14:01:52] (03CR) 10Hashar: [C: 032] Ping #wikimedia-dev and #wikimedia-qa when mediawiki/selenium is updated [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/90537 (owner: 10Zfilipin) [14:02:30] (03CR) 10Hashar: "Jenkins cant merge in that repo, needs Yuvi to submit and then update the labs instance :-]" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/90537 (owner: 10Zfilipin) [14:18:51] is it possible to request ubuntu packages to install? kind of missing ant. [14:26:15] give Coren, petan or YuviPanda a ping or request it on WikiTech [14:26:41] are you building for the JVM by the way? [14:26:57] yes, building some java code for jvm. [14:27:47] you might want to run it with -jamvm. The default jvm (OpenJDK 7) requiers absurd amounts of memory assigned to the job to start [14:29:20] I'll keep that in mind [14:31:34] I'm still trying to find my footing myself, but that worked well for small scale experiments [14:34:56] I went ahead and submitted a bugzilla ticket about ant. [14:36:27] oh, that should work too. Since it's availible in the official Ubuntu repos, it shouldn't be too much of a hassle [14:40:55] right. I don't think it adds any deps because java is already there [14:41:11] and javac [14:50:54] MaxSem: can please help me out with a puppet confusion i have? [14:51:06] sure [14:52:34] thanks. in manifests/misc/download.pp i see in line 203 and onward, two ensures, can you help me figure out what was inteneded there? [14:54:33] MaxSem: also, I can't understand why lighthhtpd is used, but that is a difernet question. [14:55:14] because this infrastructure is from aeons ago, when lighty was really faster for statics [14:56:48] as of the manifest, it just creates 1) symlink, 2) empty file [14:57:47] puppet complains about duplicate ensures, MaxSem [14:58:27] hmm, dunno [15:00:40] a question about dumps, is there anyone to answer? [15:01:11] thanks MaxSem, i'll try to re-write it in a sane way [15:02:14] MaxSem: the file is /data/xmldatadumps/public/other/kiwix and the link is /data/xmldatadumps/public/other/kiwix or vice versa? [15:03:10] ensure => "/data/xmldatadumps/public/other/kiwix"; is a (deprecated) link syntax [15:06:55] MaxSem: i didn't quite get this. what is the target of the link and what is the source? [15:07:35] iiuc /data/xmldatadumps/public/kiwix is the link pointing to /data/xmldatadumps/public/other/kiwix [15:07:36] link is /data/xmldatadumps/public/kiwix, target is /data/xmldatadumps/public/other/kiwix [15:08:19] thanks a lot MaxSem [15:09:41] welcome:) [16:20:10] Hi, everyone... I have a question about the best way to go about working on and running a maintenance script for checking and fixing DB tables, and how to maybe use labs for this [16:20:48] Anyone have a sec to point me in the right direction? [17:31:20] (03CR) 10Yuvipanda: "Done. I'll be grateful if someone else is willing to also help with these merges / deploys - it's trivial, and reduces bus factor :D" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/90537 (owner: 10Zfilipin) [17:34:04] (03CR) 10Yuvipanda: "Hmmm, this shows up as merged but 'fetch origin' doesn't see it yet :|" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/90537 (owner: 10Zfilipin) [17:35:11] johang: link to bug for ant? [17:41:22] johang: MartijnH ant is installed on both -login and -dev now [17:41:27] i'll add it to puppet in a bit [17:41:43] johang: MartijnH let me know if it should be on the exec nodes too, but i don't think that is necessary [17:46:10] YuviPanda: https://bugzilla.wikimedia.org/show_bug.cgi?id=55938 [17:47:10] YuviPanda: not needed on exec. I don't plan to use ant in SGE jobs. only interactively. [17:48:13] johang: yeah, should be installed and available for you now then :) [17:48:23] I've pushed the puppet changes as well. [17:48:28] sweet [17:49:33] johang: do cc me when you file bugs that need packages :) [17:49:43] got it [17:54:48] how do I login to the tools-dev instance? it rejects my public key. am able to login to tools-login. [17:54:57] johang: ssh tools-dev.wmflabs.org [17:54:57] ? [17:56:19] ah, it worked from my own machine, but not from tools-login. [17:57:17] still figuring out the architecture and the differences from toolserver :) [17:57:23] johang: :D [17:57:32] johang: host based auth is something that's still flakey. you should file a bug [17:57:35] that 'ought to work' [17:57:45] johang: also, if you don't use mosh, I highly reccomend it (as an alternative to ssh0 [17:57:46] ) [17:58:02] am I supposed to be able to login to exec instances too? [17:58:50] johang: you should be able to if needed, but don't [17:58:53] johang: use jsub [17:59:01] run things on the grid! [17:59:17] yes, the grid is nice [18:01:00] johang: indeed [18:01:25] !ping [18:01:25] !pong [18:03:52] YuviPanda: submitted a ticket: https://bugzilla.wikimedia.org/show_bug.cgi?id=55945 [18:03:58] ty [18:28:40] Did someone forget to feed the server hamsters? [18:30:43] Coren|Away: poke [18:31:10] Coren: servers are taking about 3 minutes to load pages [18:41:15] I can try to fix this, but I hope I won't break anything [18:41:34] !toolsadmin [18:41:35] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Documentation/Admin [18:41:48] haha [18:41:59] I wish we had more documentation about webservers than where logs are [18:42:17] it seems jstart doesn't support array jobs [18:42:35] (or jsub) [18:43:14] johang: you can use qsub directly [18:43:44] I tried that first, but wasn't able to get the memory limits right [18:44:13] couldn't even do "java -version" in my job without failing [18:46:03] is there are dry-run parameter to jstart that only prints the qsub command without actually running it? [18:47:25] Betacommand: can you give me example of what loads slowly? [18:47:46] hm everything I guess [18:49:13] !log tools petrb: installed links on -dev and going to investigate what is wrong with apaches, documentation, Coren, please update it [18:49:17] Logged the message, Master [18:50:30] ok, contacting webproxy from tools-dev take ages I guess problem is on our side Houston [18:51:25] !log tools petrb: restarting apache server on tools-webproxy [18:51:28] Logged the message, Master [18:52:17] !log tools petrb: everything looks better [18:52:20] Logged the message, Master [18:52:25] Betacommand: fixed? [18:54:08] johang: hello [18:54:13] johang: you can't ssh to -dev I see [18:57:23] yes, I can from my home machine, but not from tools-login [18:58:03] you need to forward your private key [18:58:16] @search forwarding [18:58:16] Results (Found 2): port-forwarding, forwarding, [18:58:21] !forwarding [18:58:21] https://labsconsole.wikimedia.org/wiki/Access#Using_agent_forwarding [18:58:54] ah, darn it. that's probably it. [18:58:58] thanks. [18:59:01] yw [19:08:42] johang: dry run feels like a good feature request. file a bug :) [19:08:52] johang: also, poke MartijnH, he had a way to work around the java problem [19:09:40] !ping [19:09:40] !pong [19:09:41] ok [19:10:08] YuviPanda: had to set h_vmem real high to make it work. 2G = OK, 1G = NOK. [19:10:22] johang: yeah [19:14:05] * MartijnH considers himself poked [19:15:13] johang, you should be able to get it somewhere sort of acceptable with java -jamvm -Xms512m -Xmx512m, if you have enough with 512 heap space [19:17:24] that ran with -l h_vmem 750m for me, but you might need some twidling [19:24:30] I'm doing -Xms32m -Xmx64m and h_vmem=2G for now. seems to work. [19:26:58] MartijnH: do you know which option to pass to qsub to make it remove empty log files after the job? [19:29:49] johang, no, not sure [20:22:43] (03CR) 10Hashar: "Yuvi : do write to us a step by step howto on https://wikitech.wikimedia.org/wiki/Grrrit and we will be able to help :-}" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/90537 (owner: 10Zfilipin) [20:26:25] (03CR) 10Yuvipanda: "https://wikitech.wikimedia.org/wiki/Grrrit-wm exists ;)" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/90537 (owner: 10Zfilipin)