[00:01:25] Cyberpower678: where is your bot source? [00:01:45] saper, /data/project/cyberbot/compat/rfppbot.py [00:01:54] sorry, don't have access to that [00:02:10] github? :P [00:02:29] addshore, no [00:02:32] not yet [00:02:46] saper, you don't have access to labs? [00:03:50] not yet [00:04:02] saper: why not? :O :P [00:04:13] saper, then how are you going to help me> [00:04:31] don't see your project in gerrit or something [00:04:32] There's nothing wrong with the source. [00:05:06] well, not really, if output is intercepted, or shebang line is wrong it might be [00:05:34] Nothing about the source has changed. But it stopped outputting just now. [00:06:08] I'm pretty sure print is not the wrong command. [00:06:08] sad story [00:06:28] Not even stderr is outputting. [00:06:48] Which is something Coren needs to look at. [00:06:57] Since I can't fix it myself. [00:07:21] not sure he can do much [00:07:41] He's the god of Labs. [00:07:45] I've seen some trouble running python scripts on grid engine [00:07:46] So yes he can. [00:08:08] saper, if you want my opinion, python is nothing but trouble. [00:08:10] I don't care about your religion, really... [00:08:17] so use something else [00:08:29] our users use ruby and lisp, among others [00:08:40] saper, I'm non-religious myself. [00:08:44] I use PHP [00:09:04] https://bitbucket.org/plwiki/bot/src <- clmw is in Common Lisp [00:10:27] sajklk [00:10:27] ]j [00:10:28] gh [00:10:31] ghkfoklksdrsdkjnm [00:10:33] kl;lkhjnb [00:10:34] hgkopfgufdindf'plh [00:10:35] ,j [00:10:36] ;i [00:10:38] ghmkfgojnfdon[lpi [00:10:39] 'io'lhj [00:10:42] 'jk [00:10:44] 'jk';lghlkfre;kkhj [00:10:48] jhkopnlflkmghj;'l,gh [00:10:50] fgm;lfgreklfgm;ln[khj [00:10:58] 'gh;mlghdfnlhgmgl;,j [00:11:04] jm;lklmfgmbijijp m,trpo [00:11:05] yt;lgfdlmkglkhg [00:11:11] ? [00:11:24] 'd;lnlknjsdskbskjwkgsdjhwejhbsesddfviyvjfg;mlj,/ilnjkgfuysacgebkgfmui;l,hj';lmlkfdnjdfjknh [00:11:33] ;yj;mlhgfklgfnjkdfsdsebhjdsjnfglkmfgn;lghjiokjndssjkbsdkjngblgf [00:11:42] Cyberpower678: ? [00:11:44] gfmh;lghl;hkdklsjnsnkanjdfknbklmhg''gh [00:11:51] fg;fkdflklfdklsdklmkgf [00:11:57] -.-\ [00:12:11] Parrot strikes again. [01:05:43] I was wondering if someone could help me connect to the boogs.wmflabs.org instance [02:43:01] !puppet del [02:43:02] Successfully removed puppet [02:51:25] anyone here? [02:52:06] I have problems in https://en.wikipedia.org/wiki/Wikipedia:HOTCAT [02:52:16] HotCat isn't working here.. [02:53:22] for me. [11:45:11] @seenrx damian [11:45:11] petan: I have never seen damian [11:45:15] @seenrx damianz [11:45:15] petan: I have never seen damianz [11:45:18] @seenrx Damianz [11:45:18] petan: Last time I saw Damianz they were quitting the network with reason: Ping timeout: 248 seconds at 11/19/2013 5:53:41 AM (2d5h51m36s ago) (multiple results were found: Damianz_, Damianz[Work]) [11:45:28] @seenrx [dD]amian[Zz] [11:45:29] petan: Last time I saw Damianz they were quitting the network with reason: Ping timeout: 248 seconds at 11/19/2013 5:53:41 AM (2d5h51m46s ago) (multiple results were found: Damianz_, Damianz[Work]) [11:59:09] morning YuviPanda [13:27:44] !ping [13:27:44] !pong [14:42:22] petan: can you login to the webserver? render has a runaway process that needs to be killed... [14:42:49] JohannesK_WMDE: on tools? [14:43:44] petan: yes [14:43:54] yes I can, which one is it [14:44:09] which user? [14:44:18] local-render [14:44:29] a python script [14:44:37] probably tlgwsgi.py [14:44:43] petan ^^ [14:45:14] I don't see any process of such user (id 50613) on neither of these 3 servers... [14:45:54] something is eating up all our database connections to eswiki [14:46:01] i assumed it was this script [14:46:37] can you check the db connections to eswiki itself? i can't even do "show processlist" because well... all db connections are used up [14:46:44] "User p50380g50613 already has more than 'max_user_connections' active connections" [14:47:12] http://tools.wmflabs.org/paste/view/39484f62 :( [14:47:30] no I have absolutely no access to sql servers [14:47:39] oh, we have a paste [14:49:02] petan: ok, thanks. so... any idea who i should ask? Coren? [14:49:06] yes [14:49:17] or bugzilla [14:50:16] JohannesK_WMDE: That's your user? [14:50:37] Coren: that's local-render, yes [14:52:15] giftpflanze: yes, with real time syntax highlight :) [14:52:27] and extremely easy to use [14:52:35] no need to register or fill in any captcha [14:53:28] JohannesK_WMDE: You should switch that to the new webserver scheme so that you can restart your own server at need. [14:54:01] !toolsdocs [14:54:01] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help [14:54:28] Coren: is that new scheme documented somewhere [14:54:31] !toolsadmin [14:54:32] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Documentation/Admin [14:54:44] !newweb [14:55:16] @info [14:55:16] http://bots.wmflabs.org/~wm-bot/dump/%23wikimedia-labs.htm [14:55:19] petan: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help/NewWeb [14:55:27] petan: Pointed to by the help doc [14:55:40] !newweb is https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help/NewWeb [14:55:40] Key was added [14:55:53] Coren: sounds good, going to do this when i get around to it [14:56:14] but not before wikicon i guess [14:56:46] Coren: these web servers are started on same boxes as apaches? [14:56:47] JohannesK_WMDE: You have no connection to eswiki. [14:56:56] petan: No. [14:57:10] aha webgrid [14:57:11] I see [14:57:29] Coren: http://tools.wmflabs.org/render/tlgbe/tlgwsgi.py?action=query&query=Europa&lang=es&querydepth=1&flaws=ALL&format=json&showthreads=true <-- see error at end of output [14:58:38] but i can 'sql eswiki' now... [14:58:40] strange. [14:58:52] yesterday even connection via shell wouldn't work [14:59:15] JohannesK_WMDE: Maybe yesterday you just had too many requests to your tool. :-) [14:59:46] Coren: no, it was like this for several days [15:00:03] JohannesK_WMDE: Is this just PHP in public_html? [15:00:11] python [15:00:18] Same thing. :-P [15:00:23] yeah. so? [15:00:31] Converting to the new web scheme is probably just "webservice start" [15:00:46] Unless you have .htaccess [15:00:47] the difference is, tool tries to open about 10 connections, while console needs only one [15:01:06] we have a connection limit of 512, so something must be using ~500 connections [15:06:07] Coren: any chance .htaccess will be supported in the new scheme? [15:06:46] JohannesK_WMDE: No; you need to convert what is done with .htaccess to lighttpd directives instead. [15:07:03] (The same functionality is provided, you just have to configure it differently) [15:07:11] okay [17:37:25] Hi. How to change my output (.e and .o) files' from qsub commands destination? by default they appear in home folder [17:44:09] alkamid: With the -o and -e options [17:44:43] alkamid: If the option parameter is a directory, it'll stuff the out and err files there. If they are filenames, it'll use that and append to them instead of creating new ones. [17:46:34] Hi! I try to install a MW installation on instance https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000993 using puppet and when I run "sudo puppetd -tv" just after having added role::mediawiki-install::labs I have the error: [17:46:37] info: Retrieving plugin [17:46:38] info: Loading facts in /var/lib/puppet/lib/facter/puppet_config_dir.rb [17:46:39] info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb [17:46:41] info: Loading facts in /var/lib/puppet/lib/facter/projectgid.rb [17:46:42] info: Loading facts in /var/lib/puppet/lib/facter/apt.rb [17:46:44] info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb [17:46:45] info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb [17:46:47] info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb [17:46:48] info: Loading facts in /var/lib/puppet/lib/facter/default_gateway.rb [17:46:49] info: Caching catalog for i-00000993.pmtpa.wmflabs [17:46:50] err: Failed to apply catalog: Cannot alias File[git_clone_mediawiki_directory] to ["/srv/mediawiki"] at /etc/puppet/modules/git/manifests/clone.pp:98; resource ["File", "/srv/mediawiki"] already defined at /etc/puppet/modules/apache/manifests/vhost.pp:93 [17:48:16] Tpt_, you couldn't pastebin that? [17:48:42] Sorry, I haven't thought to do that :-( [17:54:00] thanks Coren [17:55:14] Tpt_: looking... [17:55:35] andrewbogott: Thanks :-) [17:56:37] Tpt_: That's most likely my fault, I'm going to set up a new VM and see if I can reproduce the problem. [17:56:58] andrewbogott: Thanks [18:01:30] back in ten [18:38:28] Tpt_: Yes, definitely my fault, although I don't totally understand yet... [18:39:02] andrewbogott: Thanks for having investigate :-) [18:56:20] Tpt_: OK, I have a pending fix now but I want to run it by a coworker before it goes live, and he's at lunch. If you re-run puppet tomorrow I predict it will be magically fixed. [18:56:45] andrewbogott: Thanks a lot :-) [19:00:55] Tpt_: whoops, said coworker is out until Tuesday. Are you blocked by this or are you just doing cleanup in response to my recent, nagging email? [19:01:06] (If you're blocked I can figure out a short-term workaround.) [19:01:42] andrewbogott: I'm not really blocked so I can wait a week. [19:01:49] ok, thank you. [19:01:53] And, sorry :) [21:17:14] Any hint why a python line " inp = gzip.open(/public/pagecounts/.../file.gz)" works fine if I run "python script.py", and gives IOError if run with qsub? [21:17:32] exact qsub command: qsub -N 'visits.py' -b y -l h_rt=0:30:00 -l h_vmem=250M -e $HOME/output/ -o $HOME/output/ python $HOME/scripts/visits.py [21:18:26] alkamid: if the file didn't exist, would that be an example of an IOError? [21:18:52] andrewbogott, it would [21:18:53] alkamid: check the exception message to see what the exact error is? [21:19:09] alkamid: It appears that /public/pagecounts doesn't exist on at least tools-exec-04 [21:19:21] the cause could be a mount that is not available on the execution hosts ^^ that. [21:19:58] alkamid, background: when you use qsub your script is then run on one of a group of hosts, none of which is the one you're logged in to. [21:20:12] right, I understand this [21:20:18] 'k [21:20:25] Coren: Should /public/pagecounts exist on exec nodes? See scrollback for the past few minutes. [21:20:50] but I would suppose pagecounts to be accesible from execution hosts [21:20:51] anomie: It should, and indeed it should be automounted. [21:21:28] it's something the bots could possibly work on, not humans [21:22:09] but maybe there is a reason for them not being included? [21:22:27] anomie, Coren, from what you say, should I just wait patiently until it's mounted? [21:22:37] alkamid: There is a simple reason: autofs is a piece of crap that almost always fails to change its config without a reboot. [21:22:48] alkamid: I'll try to find a workaround in the meantime. [21:23:12] Coren, great, thanks [21:25:40] it doesn't look like that directory is kept up to date very well, the most recent file is from 14 hours ago [21:40:44] alkamid: Not fun news. Because there are already mounts in the directory where that new automount lives, autofs isn't able to cope with the addition until everything /else/ has been unmounted. I.e.: not workable without disruption. [21:41:06] alkamid: Best I can offer is to mount it somewhere else random until the next reboot cycle. [21:41:44] Coren, do you know when the next cycle will be? [21:42:44] alkamid: I can schedule a rolling restart of nodes with minimal disruption, but that'd take ~12h per node or so so not before 4 days at best, and I'd like to give a few day's notice before I start. [21:43:25] (Because to avoid disruption I'd first disable the nodes and let the jobs drain) [21:44:31] Coren, for me mounting it somewhere else is fine temporarily [21:45:15] I guess they will have to be restarted at some point and I can wait [21:49:11] alkamid: They're be available at /mnt/pagecounts [21:49:21] Coren, thanks a lot [21:49:26] alkamid: I'll make that into a symlink after reboot. [21:49:43] Coren, great [23:08:20] !log cvn The cvn-api web service from CVN at Toolserver is now in source control: https://github.com/countervandalism/cvn-api [23:08:22] Logged the message, Master [23:08:43] !log Installed the cvn-api web service on the cvn-apache2.wmflabs host http://cvn.wmflabs.org/api.php [23:08:43] Installed is not a valid project. [23:09:10] !log The http://cvn.weblabs.org entry point now supports HTTPS (thanks YuviPanda!) [23:09:10] The is not a valid project. [23:09:17] !log cvn Installed the cvn-api web service on the cvn-apache2.wmflabs host http://cvn.wmflabs.org/api.php [23:09:18] Logged the message, Master [23:09:20] !log cvn The http://cvn.weblabs.org entry point now supports HTTPS (thanks YuviPanda!) [23:09:22] Logged the message, Master