[00:00:07] Betacommand: Moar detailz! [00:00:27] Coren: Mail delivery failed: returning message to sender [00:00:41] Ive gotten 4 in the last few hours [00:00:50] What was the intended destination, and what's the error? [00:01:05] Coren: its from jsub [00:01:09] and cron [00:01:28] Yeah, but what does the bounce say? What's the destination? [00:01:57] http://pastebin.com/Ye47fHAW [00:03:35] "retry timeout exceeded" isn't all that helpful. Hm. Wait, how did you receive the bounces at all? [00:03:43] * Coren boggles a little. [00:04:25] Coren: I dont know those just showed up in my email [00:05:04] Coren: would the job ID be useful? [00:05:31] Well it speaks of timeout and that'd normally happen in case of repeated transient failure. I'll keep an eye on the mail system, see if there's something odd beyond the changes I was doing this morning (which shouldn't affect inside mail) [00:06:24] Coren: I get my stuff forwarded to gmail [00:09:07] Coren: does that make a difference? [00:09:44] It really shouldn't. [00:10:45] Coren: *shouldnt* is a key word. Ive seen a small change in a non-connected system take down a network before [00:12:51] Coren: could it be tied to bug 59731 [00:16:22] That looks like a possibility, as my mailer is configured pretty much the same way, but I doubt tools volume are high enough. I'll dig into the logs to see whether there's a pattern based on destination; but given how many people use gmail as their wikitech email, I'd have thought there'd be more issues than just you. [00:16:44] (Or it /just/ got triggered) [00:20:49] Coren: or like normal I am just the first to notice and complain [00:21:27] Heh, or the only one plagues with wierd flukes and one-off bugs. You're like the Great Bug Attractor. :-) [00:21:34] plagued* [00:23:07] Coren: thats not cool [00:26:29] Coren: I just looked again and there where 2 more from 25 minutes ago [00:28:08] Aha. I see those. Lemme see... [00:31:11] got it [00:31:17] Hm. [00:31:48] AFAICT, everything works fine. Can you dpaste me a complete bounce? [00:32:41] just got 1 more [00:32:47] Bounce? [00:34:31] yeah [00:34:32] one sec [00:35:21] Coren: see PM [00:40:56] Huh ah; I was looking at the wrong place. It's not the mailserver that's unable to talk to google, it's -login that seems to have issues with the mail server. [00:43:23] Coren: gotta love the bugs I find [00:43:43] Ah, and I see a pattern. Looks like only tools with dashes in their names. [00:44:12] COREN... why are you making my life a headache? [01:01:29] Betacommand: You may or may not be amused to look at your inbox. [01:02:19] Although now apparently gmail is getting cross at me. [01:03:32] 450-4.2.1 The user you are trying to contact is receiving mail at a rate that\n450-4.2.1 prevents additional messages from being delivered. Please resend your\n450-4.2.1 message at a later time. If the user is able to receive mail at that\n450-4.2.1 time, your message will be delivered. [01:24:14] Hello I can't search for articles containing the ++ C familly languages operator in all language: https://en.wikipedia.org/w/index.php?title=Sp%C3%A9cial%3ARecherche&profile=default&search=%2B%2B&fulltext=Search&searchengineselect=mediawiki [01:26:57] ytrezq: The subject of this IRC channel is Wikimedia Labs; I think you should file a report in Bugzilla for that. [01:27:33] Coren: what was the hangup? [01:28:04] The relay was overly paranoid and didn't trust the login server. :-) [01:28:32] ..... [01:28:57] Coren: why do I get all of these bugs? [01:30:01] looking in my trash, 26 failed messages [01:33:37] Coren: please fix all the bugs before I have to discover them, This is getting old [01:34:46] scfc_de: Bugzilla isn't IRC and require account creation. Would there be better channnel for talking about this? [01:43:23] ytrezq: You could try #wikimedia-operations, but I don't know if somebody considers this important enough to require immediate action. [01:43:56] ytrezq: Oh, I see you tried that already. [04:55:01] hi, Coren, a quick labs question, not tools specific. I need to create an upstart job that does something as a specific user via 'sudo -u user'. The job is started 'on' network and local-filesystems. [04:55:16] But it seems to start so early that sudo does not yet know the user 'user' [04:55:35] I'm guessing I need to more services to start up, but which one? [04:57:34] Upstart log for that job says: "sudo: unknown user: *****" "sudo: unable to initialize policy plugin" [05:02:51] I'm guessing it has something to do with ldap [05:03:12] but why is the connection not working even after network and eth0 are up and running? [12:46:14] Coren: How do I use flask again? So I can have tool/script.py/foo/bar/?bah=baz [12:55:36] a930913: should be in the newweb docs [12:55:37] !newweb [12:55:37] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help/NewWeb [12:56:29] a930913: combine the "Example FCGI configuration" with the app.fcgi as in https://github.com/valhallasw/gerrit-patch-uploader/blob/master/app.fcgi [12:58:19] valhallasw: Thanks, I'll take a look when I get back :( [13:41:41] Coren: ping [15:36:01] hello. I'm working with mwparserfromhell. I want to change a parameter value like this: template.get("llengua").value=u"anglès" . However I get a UnicodeDecodeError: 'ascii' codec can't decode byte 0xe8 in position 4: ordinal not in range(128) [15:36:07] does anyone know how to solve this? [15:36:38] it's probably a noobis [15:36:44] noobish error :) [15:51:13] Arnaugir: This is probably more a Python/mwparserfromhell problem. Debugging this from afar is very complicated. [15:51:51] scfc_de: I know, but maybe someone has got an idea of a possible solution... [15:51:56] I'm kind of stuck there [15:54:21] I had a similar problem in Python once and it was related to not declaring that stdin was feeding utf-8. But without a deep look in mwparserfromhell's inner workings, it's impossible to tell. Have you contacted mwparserfromhell's author? [15:56:13] Arnaugir: could you post the code how you created the template object? [15:56:39] I shall contact the author. [15:56:56] Arnaugir: I'm thinking you may have parsed a str, and then you should also pass str when you assign something [15:58:04] valhallasw: i just tried to check the type with type(template.get("llengua").value) and I got [15:59:22] 16:56 < valhallasw> Arnaugir: could you post the code how you created the template object? [16:00:51] wikipage = page.Page(site, title="Usuari:Arnaugir/proves") [16:01:02] text = wikipage.getWikiText().decode('utf-8', 'replace') [16:01:09] code = mwparserfromhell.parse(text) [16:01:10] for template in code.filter_templates(): [16:01:11] etc. [16:01:20] valhallasw: is that what you're asking for? [16:01:34] Yes. [16:01:50] text = wikipage.getWikiText().decode('utf-8', 'replace') [16:01:55] ^ why are you running .decode(...) here? [16:02:16] I don't remember, but works everywhere else [16:02:23] in other scripts [16:02:26] should I remove...? [16:02:45] http://www.joelonsoftware.com/articles/Unicode.html < you should read that [16:03:05] in any case, either you should remove that [16:03:17] or assign u"anglès".encode('utf-8') instead [16:03:29] either work with unicode objects all the way, or work with str objects all the way [16:04:46] valhallasw: yes I should look deeper into codifications ;) I already tried u"anglès"..encode('utf-8') , not working [16:05:47] What is the error you get? [16:06:29] SyntaxError: (unicode error) 'utf8' codec can't decode byte 0xe8 in position 0: unexpected end of data [16:07:41] Arnaugir: so the next problem is that you declare your script file to be utf-8 (coding: utf-8) on top, while your editor saves it as latin-1 [16:13:54] valhallasw: so what should I do? [16:14:19] okay, I saved it as utf8 [16:14:37] cool, that worked [16:16:05] thanks a lot [17:02:32] valhallasw: I've got it working somehow, but it's not running in the virtual environment. [17:03:11] a930913: does the hashbang line in your app.fcgi refer to the correct python? i.e. the one in your venv? [17:03:36] valhallasw: Yeah. Well, my cgi.py. [17:03:47] err. [17:04:09] which project is this? and is it on eqiad? [17:04:17] valhallasw: Yeah, cluestuff. [17:04:55] a930913: errr. You're doing it the cgi-bin way instead of the fcgi way? [17:05:34] and which file are we talking about? [17:05:43] Erm, possibly. I followed the instructions that made the most sense :p [17:06:03] Yeah, except the flask instructions are for apache, and not for lighttpd, so I'm not sure how well they map [17:06:06] but ok [17:06:15] in any case [17:06:16] valhallasw: Atm, cgi.py bootstraps stats.py [17:06:27] I see no cgi.py [17:06:40] oh, in public_html [17:06:41] never mind [17:07:08] I see no reason why that would not use the venv, to be honest... [17:07:25] unless lighttpd is doing something special with .py files [17:07:47] try adding [17:07:48] import sys [17:07:48] print sys.executable [17:07:55] to see which python is being used... [17:08:07] valhallasw: I'm assuming it's not doing the venv, because the second import isn't working. [17:08:44] a930913: but stats is not in the venv, but in the local directory [17:09:12] valhallasw: Yeah, it's /usr/bin/python [17:09:32] valhallasw: But the hashbang? [17:09:42] ? [17:10:32] valhallasw: Nvm, it's running in the regular python. [17:10:55] Despite the hashbang. [17:11:49] :/ [17:13:53] valhallasw: Ok, I've moved it all into stats.py [17:17:24] hello, I'm confused about the tools migration. I run an instance under the Maps project. In the documentation about the move, is an instance the same as a "tool" ? [17:18:12] No. 'tools' are (sub)projects within the Tools project [17:18:30] You should check the generic Labs migration documentation. [17:19:11] ahh okay, that makes it more clear. Thanks. [17:27:32] so I think the docs would be here: https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration [17:27:57] Does each instance need to be migrated, or just the one project? [17:30:53] chippy: https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration_Howto [17:32:09] chippy: poking andrewbogott_afk is also a good option [17:32:22] Thanks valhallasw :-) [17:34:44] at any rate I know that there's none of the immediate urgency regarding the move for non Tools project instances [17:37:18] chippy: I still can't tell if you're talking about tools or projects :) [17:37:32] If you have instances that you care about, they need to be migrated soon. [17:37:35] hehe, projects (I think) [17:37:41] Are you subscribed to labs-l? [17:37:43] yeah it's an instance on the maps project [17:38:21] yup, subscribed. Maybe I should email to our maps-l list - there's a few instances under the project, I don't know who's doing what, but I'd imagine some of the steps to be the same [17:39:47] I have migrated maps-wma1. Nothing else. [17:40:01] Folks (including you) need to file bugs if you want me to migrate other instances. [17:40:25] And if that's not obvious from the emails I've already sent then I need to say something different I guess :( [17:41:03] andrewbogott: heya! [17:41:05] * andrewbogott is slightly cranky due to tiny airplane seat [17:41:15] andrewbogott: MaxSem might know gbetter about mobile project than me. [17:41:19] YuviPanda: hey -- are you the king of project mobile? [17:41:25] andrewbogott: MaxSem is more than me :) [17:41:37] andrewbogott: I haven't actually done anything there ever. it's more a mobile web thing [17:41:41] OK. By the way, is user dr0pwhatever Adam Baso? Or someone else? [17:41:49] andrewbogott: yes, that's Adam Baso [17:42:02] OK, good, that's one fewer person I have to keep track of! [17:42:10] I'll pass that email on to Max. Thanks for checking in. [17:42:22] andrewbogott: yw! :) [17:42:25] While you're here, do you have any other labs migration stuff that I should note down? [17:42:57] andrewbogott: You seem to have taken care of the proxy stuff well :) I don't maintain anything else, so I guess I'm fine. [17:43:12] Yep, setting up the eqiad proxy was surprisingly easy. [17:43:14] OK, cool. [17:43:19] andrewbogott: woo for puppetization :) [17:43:24] yeah! [17:43:24] andrewbogott: and thanks for helping fix the DNS issue! [17:43:41] YuviPanda: is it fixed? All I did was change a cache size and cross my fingers. [17:43:47] andrewbogott, thanks! it wasn't 100% clear - but I blame my newbieness for that. Anyhow this is what I love about IRC - it's much easier to clear things up. I will file a bug ticket. I'm not sure how much hand holding would be required, if puppet needs to be set up for the stuff on this instance etc, but we can get to that later [17:43:57] I guess I haven't heard any complaints for a few days. [17:43:59] valhallasw: I've got it to import from the venv, but it 500's when I use the import :/ [17:44:16] chippy: Most importantly, add a note on this page that you're on the case: https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration_Progress [17:44:27] That way I'll know not to start shutting stuff down next week! [17:44:27] will do [17:44:47] andrewbogott: no, we changed the DNS resolver, remmeber? [17:44:54] andrewbogott: nginx was trying to ping the old pmtpa DNS IP [17:44:57] oh, that -- two different dns issues. [17:45:20] andrewbogott: heh :) Yeah, that was the one place where the puppet manifest wasn't properly done, and that bit us in the migration :) [17:45:28] We've been having intermittent dns failures labs-wide. Or, at least, we were. Maybe that was fixed by the cache change. [17:45:30] * andrewbogott nods [17:45:37] andrewbogott: also sorry for not being able to help generally with migration. Been travelling. [17:46:17] no problem -- it's going fairly well so far. [17:46:30] andrewbogott: yeah, looks like it :) I'll try to migrate my tools in a day or so [17:46:40] cool [17:46:46] andrewbogott: :) [17:47:22] YuviPanda: you in Goa now? [17:47:30] andrewbogott: no, heading there next week :P [17:47:33] andrewbogott: in Bhopal now. [17:48:09] andrewbogott: a small town in the middle of nowhere. Visiting a friend who is volunteering at a clinic here (free clinic for victims of Dow Chemicals' Factory blowup here - India's "Chernobyl" equivalent) [17:48:12] Man, I don't know a damn thing about Bhopal except that… one thing. [17:48:16] andrewbogott: :D [17:48:26] OK, so, sounds like that's the main thing to know :( [17:48:35] andrewbogott: there isn't anything else :( [17:49:06] andrewbogott: the site still hasn't been fully cleaned up. Kids/People play cricket on the grounds. It's been 'reclaimed' by nature a fair bit. [17:49:41] That's ridiculous! It's been, what, 25 years? [17:50:24] andrewbogott: 30. 1984 [17:51:04] andrewbogott: they also denied ground water contamination until 10 years or so ago, and people were drinking it. [17:51:25] yikes [17:51:37] Friend you're visiting is a dr/nurse? Or some other kind of helping hand? [17:51:41] andrewbogott: Nurse. [17:51:51] andrewbogott: she's the 'Alice Coy' mentioned in http://www.independent.co.uk/news/world/middle-east/i-saw-israeli-bulldozer-kill-rachel-corrie-1919464.html [17:52:13] andrewbogott: ex-programmer who quit writing code to become a Nurse. Very inspiring people. [17:52:57] And drawn to trouble like a magnet, huh? [17:53:41] andrewbogott: heh :) No drama in Bhopal yet. The Palestine one was the only 'drama'. Today was the 11th anniversary of the event, so I came over to spend it with her. [17:54:34] * andrewbogott nods [17:54:39] a930913: as mentioned, I have no clue what exactly happens, and how lighttpd is running the cgi script. I only know the fcgi method works ;-) [17:54:59] andrewbogott: we've an app release next month. Hopefully I can spend more time on labs after that [17:56:12] valhallasw: Can you talk me through it, because I didn't understand that method? :3 [17:56:30] a930913: I can try. [17:57:12] basically, there are two steps. 1) create an fcgi wrapper, such as https://github.com/valhallasw/gerrit-patch-uploader/blob/master/app.fcgi [17:57:22] note the venv interpreter hashbang [17:57:39] secondly, create/adapt .lighttd.conf to be something like https://github.com/valhallasw/gerrit-patch-uploader/blob/master/lighttpd.conf [17:58:50] line 1 is the base path, and in your case should be something like /cluestuff/stats [17:59:12] then socket should be something like /tmp/cluestuff-stats.sock (the name does not matter as long as it doesn't exist) [17:59:27] bin-path should be the path to the fcgi wrapper [17:59:35] and that should be it, basically. [18:10:37] so I'm a bit clueless about puppet, self-hosted puppet and what I need to do re puppet and the software I apt-get installed onto the instance within the context of the migration. Is this type of question better in Bugzilla, mailing list or here? [18:16:04] mailing list unless anyone is specifically around in here [18:16:15] self-hosted puppet is the biggest hack ever [18:16:27] valhallasw: Done, how should I be able to access it? [18:16:59] Damianz, okay :) Thanks. [18:17:32] I need better home internet... trying to watch the CWIF finals while downloading RHEL isos.. not good [18:22:33] a930913: restart lighttpd [18:22:45] then access http://tools.wmflabs.org/cluestuff/stats [18:24:47] valhallasw: 404? [18:25:17] a930913: ah. fastcgi.server += ( "/stats" => [18:25:20] should be [18:25:24] fastcgi.server += ( "/cluestuff/stats" => [18:28:14] * Damianz wonders what cluestuff is since apparently he is a member of that project but has never touched it [18:28:28] * Damianz also notes he should move monitoring in the next few days [18:37:25] valhallasw: "not currently serviced." [18:45:28] andrewbogott_afk, YuviPanda, pong [18:46:20] MaxSem: plz2see email [19:19:48] Coren: ping [19:28:55] ping timeout [19:29:57] Damianz: :p [20:04:36] !ping [20:04:37] !pong [20:06:42] a930913: wait for the webserver to come back up. If it doesn't work, check error.log [20:10:32] Coren, is the webgrid-tomcat host supposed to service normal jobs? [20:10:43] valhallasw: No. [20:11:42] It's for the (soon to be available) tomcat webservice. [20:12:56] *shudder* [20:13:30] Coren: ok... /data/project/nlwikibots/bin/tvpupdater seems to have run on the tomcat host the last two days ( https://nl.wikipedia.org/wiki/Speciaal:Bijdragen/Valhallasw-toolserver-botje ) [20:14:00] valhallasw: o_O what. [20:14:57] which means that either a) the job ran on tools-webgrid-tomcat-eqiad, or b) socket.getfqdn() returned tools-webgrid-tomcat-eqiad on a host that was not tools-webgrid-tomcat-eqiad [20:17:18] tools-webgrid-tomcat-eqiad isn't a fqdn [20:18:20] sorry. tools-webgrid-tomcat.eqiad.wmflabs [20:20:16] and then you can complain about a trailing dot-or-not, but I'm don't really care about the semantics of fqdn's, to be honest. [20:22:49] valhallasw: I'll keep an eye on this. There may be a misconfigured bit of DNS; I doubt gridengine is easily confusable enough to mix queues up. [21:03:34] valhallasw: Got it working, but it throws 500s for websockets (flask-sockets) :/ [21:04:06] a930913_: apache is proxying, no websockets yet I think [21:04:30] Gah. Do I have to bodge? :( [21:04:35] a930913_: bodge? [21:04:49] a930913_: the nginx proxy should support some form of websockets, but I haven't had the time to make tools use it yet... :( [21:05:05] a930913_: currently, you either have to use something like SSE, or have your own instance :( [21:05:21] YuviPanda: Bodge as in stream by non "proper" means. [21:05:26] a930913_: ah, yes. [21:05:35] a930913_: although do check out SSE. if you only want server -> client stream that might work [21:05:43] a930913_: and it is fairly supported... [21:06:50] a930913: ugh. Not sure about websockets :/ [21:07:22] YuviPanda: SSE? [21:08:01] a930913_: http://www.html5rocks.com/en/tutorials/eventsource/basics/ [21:08:18] valhallasw: no websockets on tools yet. we need to switch the apache proxies to be dynamicproxy before that can happen [21:08:51] a930913_: support http://caniuse.com/#search=eventsou [21:09:28] An aside, how do I /ghost a930913? [21:09:54] a930913_: /msg NickServ ghost i think [21:11:15] Derp, I was trying chanserv :p [21:13:08] Double derp, never registered a930913 :p [21:16:31] a930913_: :D [22:41:37] (03PS1) 10Andrew Bogott: Added placeholder neutron password [labs/private] - 10https://gerrit.wikimedia.org/r/118991 [22:43:41] zz_yuvipanda: I got it pushing i++ \o/ Thanks :) [22:45:18] (03CR) 10Andrew Bogott: [C: 032 V: 032] Added placeholder neutron password [labs/private] - 10https://gerrit.wikimedia.org/r/118991 (owner: 10Andrew Bogott) [22:55:08] /usr/local/bin/migrate-tool: line 32: ...MIGRATE.STATUS: Permission denied [22:55:15] ? [23:06:37] Coren: around by chance? [23:29:06] petan: around? [23:33:17] rschen7754: whats up? [23:33:32] Betacommand: when i try and migrate i get the error /usr/local/bin/migrate-tool: line 32: ...MIGRATE.STATUS: Permission denied [23:34:04] rschen7754: under your account or the tools [23:34:16] under my account [23:34:30] have you migrated all your tools already? [23:34:54] all of mine, yes, and my account too, but today i got added to a service group to move it [23:35:35] rschen7754: service group? [23:35:40] stewardbots [23:35:43] Ah [23:35:54] did you "become" that tool? [23:36:22] wait never mind [23:36:33] I know what the issue is [23:36:37] it's supposed to be done from the user account not the tool [23:36:51] rschen7754: yeah, brain freeze [23:37:20] rschen7754: the issue is that new changes to group memberships are not being synced to tmpa [23:37:29] rschen7754: try to become the tool [23:37:48] well, i've become the tool to check the files and see what's going wrong [23:38:09] Ok, then thats not the issue [23:38:15] unless it's only updated in some places and not others [23:44:38] rschen7754: drop an email to the mailing list [23:44:40] rschen7754: Have you tried the work around to add someone to the tool's maintainer list, removing them immediately afterwards and wait for a few minutes? [23:45:16] scfc_de: yep [23:45:18] scfc_de: that issue is on the other end of the process not in tampa [23:46:51] Betacommand: Yes, but I don't know how/what the sync works. [23:47:12] and i can't tell what ...MIGRATE.STATUS file is being referred to [23:47:22] i rmed the one in my own directory [23:47:39] and didn't see one in the stewardbots tool [23:47:49] rschen7754: You already migrated your own user account? [23:48:04] scfc_de: yes, and i wonder if that is causing the problem [23:48:05] Anyhow, don't know specifics about the migration tools, Coren is probably the only one who could help. [23:48:23] So mail might be a good idea.