[02:23:47] "mediawiki doesn't have support for challenges" [02:24:03] i wonder if you could hook in to the same thing that was used for the forced passwd change? [02:24:07] wild guess :) [03:59:07] On wikitech, searching for anything that isn't a pagename results in "An error has occurred while searching: We could not complete your search due to a temporary problem. Please try again later." [04:14:58] Ryan_Lane: 19 03:59:07 < spagewmf> On wikitech, searching for anything that isn't a pagename results in "An error has occurred while searching: We could not complete your search due to a temporary problem. Please try again later." [04:15:08] is that the prod elasticsearch or a separate one? [04:15:36] * jeremyb has verified spagewmf's STR [04:22:48] Coren, I think labs is having a mild hiccup. [04:23:03] Cyberpower678: Define 'hiccup'? [04:23:12] Webservers have hung up. [04:23:33] and something caused a few of my scripts to randomly run OOM. [04:23:50] Works for me. What breaks exactly? [04:23:51] Nothing a reboot of the scripts couldn't fix though. [04:24:08] http://tools.wmflabs.org/cyberbot/spambotstatus.php [04:24:36] Webserver-01 seems to be running on red for quite some time. [04:24:36] Eeew. Autorefresh. [04:24:43] But I see it working fine. [04:24:56] It won't load on my end. [04:25:28] I just keep getting a spinning circle. [04:25:40] And it's obviously not my internet. [04:25:44] Works perfectly fine from here. [04:26:02] Can you have a look at Webserver-01 though? [04:26:05] obviously you should be looking at a lower level or a browser console then [04:26:37] curl, tcpdump, s_client [04:26:42] jeremyb, who was that directed to? [04:26:59] Cyberpower678: you :) (why is it obvious that your internet isn't at fault?) [04:27:05] oh, did i mention mtr? [04:27:28] jeremyb, well, since I'm able to communicate on IRC... [04:27:34] uhuh [04:27:42] the rest explains itself. [04:27:50] so, your internet very well maybe the problem [04:27:59] jeremyb, howso? [04:28:53] Coren, did you get my message about Webserver-01 running on high load? [04:29:34] webserver-01's problem was unrelated -- there was a stuck apport spinning. [04:30:07] Coren, it's been stuck for days now apparently. It was like that about 3 days ago or so. [04:30:32] Yeah; since that didn't wedge the webserver I didn't notice. I don't look at the pretty colored graphs unless something is visibly wrong. [04:30:36] :-) [04:30:58] Hmm... [04:31:02] Wikipedia loads. [04:31:12] Peachy wiki loads, so does Wikidata. [04:31:21] That one little tiny thing doesn't. [04:31:35] Coren, ^ what could be causing that. [04:31:40] Well, the webservers are fine and it loads for me. Swiftly, even. [04:31:53] jeremyb: What do you get when you look at Cyberpower678's URI? [04:32:14] URI? [04:32:33] 200, 200, 200. always <200ms [04:32:53] ? [04:32:57] and always with a protest from iceweasel about your lack of charset [04:33:12] Cyberpower678: It works fine for him too. [04:33:19] Coren the translate [04:33:22] translator* [04:33:27] Looks like the problem is on your end. [04:33:44] Grrr. [04:33:55] Everything but that link works on my end. [04:34:17] i did give you a good shopping list for tools to dig for more info about the problem [04:34:20] :-) [04:34:40] jeremyb, you did? [04:34:43] i did [04:35:05] jeremyb, well, can you point me to it? [04:35:09] curl, tcpdump, mtr. maybe an s_client too [04:35:22] ?? [04:35:29] and first of all: your browser's console [04:35:32] I know what curl is. [04:35:35] wherever you're seeing the issue [04:36:29] I'm getting nothing. [04:37:06] welp, i can't discuss it any more tonight, good luck [04:37:29] Good night. [04:37:35] I'm going to bed myself. [04:39:09] (one more thing for the list: a 2nd or third browser to compare results with. and even a 2nd machine) [04:39:12] welp, nvm then [04:39:23] (and 2nd uplink) [09:34:15] !tunnel [09:34:15] ssh -f user@bastion.wmflabs.org -L :server: -N Example for sftp "ssh chewbacca@bastion.wmflabs.org -L 6000:bots-1:22 -N" will open bots-1:22 as localhost:6000 [12:51:06] Coren, are you there? [13:17:15] petan: maybe you can do this... [13:17:25] hi [13:18:02] or Reedy. any tools admin? [13:19:34] JohannesK_WMDE: whats your issue ? :d [13:20:00] addshore: render has one or several runaway processes on one of the webserves [13:20:12] i need you to KILL ALL OF THEM DEAD [13:20:13] ;) [13:20:19] runaway process? :d petan can help you there :P [13:21:07] hm, ok [13:22:32] addshore: you can't login to the webservers? [13:22:37] nope :/ [13:22:50] ok, then i'll have to wait [13:25:02] JohannesK_WMDE: what sort of runaway process is it? :) [13:25:10] I would hope something will auto kill it eventually :) [13:28:51] addshore: a python wsgi process [13:29:20] and i was hoping that, but it has been running for days [13:29:27] ahhh :P [13:29:53] well, if no one picks itup in here then file a bug ;p people should be around soon enough though [13:30:04] k [13:54:06] I'm getting stuck on something trivial which has put me back a day. I've just set myself up on tools and installed Pywikibot using default settings so I can try running some of my scripts there, but I can't get path settings right and get errors like "AttributeError: 'module' object has no attribute 'argv'". I have set PYTHONPATH=~/pywikibot-core:~/pywikibot-core/scripts:~/pywikibot-core/pywikibot [13:54:08] - what am I overlooking? [14:03:34] andrewbogott_afk: Coren is mhoover new ops hire for labs? [14:03:59] YuviPanda: He's the contractor who'll be doing the bulk of the move to eqiad. [14:04:06] oooo nice! [14:04:17] close enough I guess :) [14:04:55] Fae: I'm the most non-python expert around; but I'm pretty sure you can find plenty of people to help here if they weren't all asleep. [14:06:30] I hope so. I'd just like to get pywikibot started remotely for Faebot, there are a few reports that I should be able to convert and then think about actual tools in a couple of months. [14:06:48] Fae: poke legoktm or valhallasw, I think they've both run pywikibot on labs before [14:07:01] toollabs, I mean [14:07:11] Coren: our terminology is confusing :( [14:07:33] (By 'convert' I mean I run an old version of pywikibot, so there are a few tweaks needed) YuviPanda Okay I'll see who's about a bit later. [14:14:57] Fae: does "import sys; print sys.argv" from python commandline work for you? [14:24:32] let me check... [14:25:52] Ah I can't get the python commandline, so something else look awry. My error is... [14:26:24] (after running 'python')Traceback (most recent call last): [14:26:25] File "/home/faebot/pywikibot-core/pywikibot/site.py", line 28, in [14:26:27] import pywikibot [14:26:29] File "/home/faebot/pywikibot-core/pywikibot/__init__.py", line 26, in [14:26:30] from pywikibot import config2 as config [14:26:32] File "/home/faebot/pywikibot-core/pywikibot/config2.py", line 160, in [14:26:33] _base_dir = _get_base_dir() [14:26:35] File "/home/faebot/pywikibot-core/pywikibot/config2.py", line 122, in _get_base_dir [14:26:36] for arg in __sys.argv[1:]: [14:26:38] AttributeError: 'module' object has no attribute 'argv' [14:28:48] This looks like one of those annoying install problems that I never really understand, even if I fix it :-) [14:34:48] Okay, after blanking PYTHONPATH, I can run import sys; print sys.argv. This gives me [''] [14:37:31] So I just have no idea how to set PYTHONPATH :-) [14:38:45] The next problem is getting "Http response status 500" for a non-trivial query. So /python pwb.py scripts/listpages.py -cat:"Uploads by Fæ"/ works but using /-catr/ gives me errors. [14:39:49] I would hazard a guess that this needs a tweak to user-config.py; or I'm going about this the wrong way. [14:58:57] Fae: Your better sources of information for hot Python on Tool labs action would be legoktm, valhallasw and magnus manske. [15:04:24] Okay, will follow up with those guys. It's embarrassingly trivial but it's these sorts of install tweaks that always trip me up. :-) [15:49:04] hi Coren I think you have done some work with Varnish on the beta cluster, have you noticed 503s there? I have seen a few of them this morning https://bugzilla.wikimedia.org/show_bug.cgi?id=57249 [15:51:38] chrismcmahon: Sorry no, I haven't gotten anywhere near beta. [15:54:08] yuvipanda: are you around? [16:03:11] hm, https://wikitech.wikimedia.org/wiki/Special:FormEdit/New_Project_Request is quite hard to find [16:03:15] is that on purpose?... [16:03:41] !log deployment-prep running puppet on elasticsearch machines and restarting elasticsearch to suck up new configuration [16:03:45] Logged the message, Master [16:04:22] DanielK_WMDE_: It's not on purpose, but it maybe should be :) Is there no link to it from the docs? [16:07:10] !log deployment-prep rebuilding elasticsearch indexes to suck up configuration changes [16:07:14] Logged the message, Master [16:17:09] !log deployment-prep applying role::ci::slave::labs::common class on deployment-parsoid2 [16:17:12] Logged the message, Master [17:11:17] aude: EntityInfoBuilder should work now [17:12:27] wrong channel :) [17:18:08] aude: bah, who cares :) [17:18:21] :D [17:18:30] did you get what you need for labs? [17:18:41] andrewbogott: there's a link from the help page. but not from the main page, or anywhere "in context" i was looking. [17:19:11] probably in a lot of cases, what people really want is something in tool labs [17:19:35] not an entire labs project, but both options should be easier to find [17:19:39] true. the distinction shoul dbe made very clear [17:19:46] yep [17:19:50] Yeah, wikitech needs lots of work in the area of 'what projects are there' and 'how do I join' and generally 'what should I do' [17:19:52] it's slightly confusing [17:20:09] it's slightly confusing to me, it must be a nightmare for a noob :) [17:21:22] hmmm, https://tools.wmflabs.org/ also looks confusing [18:15:47] petan: Poke? [19:04:46] Hey, we'd like to change the ee-flow labs instance from m1.small to m1.large for more widespread testing. [19:05:43] I assume I should create a new large instance and switch the well-known name to it. [19:09:05] Coren, petan, how do I stop an SGE job? qstop does not seem to be available on tools-login... [19:09:16] stop ==> kill [19:09:20] valhallasw: qdel? [19:09:33] ahhhh [19:09:41] That'd be because.... yeah, what a930913 [19:09:41] * valhallasw hits himself [19:09:44] said* [19:09:56] Sorry. [19:10:24] spagewmf: We did try the instance resize thing, but it makes ground meat of the instance half the time. Better off just making a new one. :-) [19:10:38] is there a way to submit a max runtime to jsub? [19:10:53] ok, thx [19:12:19] I just remembered why there are so few comprehensive web apps for editing wikipeidia, well after I committed myself to making one. The state of the web is s***. [19:13:01] valhallasw: Yes, you want a hard limit or a soft one? [19:13:20] valhallasw: Soft limit politely sends you a SIGUSR1. Hard limits kills you -9. [19:14:28] Coren: both are fine. [19:14:39] for qsub, it would be -l h_rt=1:00:00 [19:14:45] but I'm not sure how to tell that to jsub [19:15:25] valhallasw: jsub speaks qsub options. It's just a fancy wrapper. :-) [19:15:30] ah! [19:15:46] Well, /most/ qsub options. -l being one. ;-) [19:15:57] perfect [19:18:07] anomie: gerrit reviewer bot is running again! [19:18:18] valhallasw: \o/ [19:18:43] * anomie wonders if he will be getting large numbers of review requests now [19:19:05] somewhat, probably. It was down for approx. 12 hours [19:19:23] but it skips all merged/abandoned changesets, so it's at least not spamming you with useless requests ;-) [19:19:40] a930913: it will probably be easier if you just pretend IE8-9 don't exist [19:20:18] Nonono. Gotta support IE6. [19:21:23] Still got ~5% [19:21:28] * Coren headdesks. [19:22:18] Coren: Heh. I have to maintain a distinctly web 1.0 site, and it's a PITA to do that with Chrome, as the person paying me to do that uses IE6. [19:22:24] What, you want sensible CSS? NOPE. [19:23:42] Thankfully, I don't do that kind of thing anymore. Now if I have to do a website thing, I code against css3 and html5. If your browser doesn't render it right, then your browser is broken and you are SOL. :-) [19:25:08] valhallasw: have you convinced Coren about toolserver.org redirects yet? [19:26:15] jeremyb: see my mail on toolserver-l [19:26:17] Coren: you seemed to support the ability of individual maintainers to redirect their stuff with htaccess. the problem is that when your account expires then your htaccess don't work anymore. AFAIK [19:26:41] jeremyb: Nosy is taking a look at the possibility to support .htaccess after burnbridges [19:27:27] valhallasw: but i guess not further editing after that [19:27:40] jeremyb: sure, but you would just redirect everything to TL [19:27:44] and split it out from there [19:28:08] jeremyb: Nosy is looking an how to preserve redirect after bridge burning. [19:28:32] Coren: errr, what did valhallasw just say? ;-) [19:29:08] meh, ganglia with start/end point is fine and then clunky and then fine … [19:29:34] valhallasw: 1) i think nosy needs to join some channels. it's fine if she's not paying attention half the time but only joining channels once a month (or even once a week) even though she *is* logged in sucks :( [19:29:41] * jeremyb will tell her too [19:30:17] (she's not on right now but she often is) [19:31:48] jeremyb: use email -- not everyone finds IRC and effective way of communication [19:31:57] valhallasw: 2) my first thought is that expired accounts should fallback to an htaccess file from a public git repo that we can all work on (and then labs or whoever can eventually steal once toolserver.org ceases to exist). one file per user/tool/whatever in that repo [19:32:26] can steal that* [19:32:48] * jeremyb goes back to reading your mail [19:33:58] (or maybe a list of mappings each instead of a full blown htaccess) [19:35:23] jeremyb: and if you reply, change the subject -- I was too stupid to do that ;-) [19:35:32] hah [19:35:36] maybe why i didn't read it? :) [19:35:44] exactly :-) [21:23:17] MrZ-man|not_here: I thought that went without saying? It's nested ajax, or promises that I'm complaining about now.