[00:12:19] petan: liangent: took a bit, but jobs have to be finished. this is the eqiad paste version with https https://tools.wmflabs.org/paste-test/ [00:12:37] petan: mission accomplished [00:18:41] Coren: my graph is looking interesting with everyone migrating to the new data centre ;p http://grab.by/uXZE [00:18:50] >> http://new.tinygrab.com/1becbe2e78a34f013af7ed25048712919f70d111db.png [00:18:52] night all! [00:26:13] Coren, ping [00:30:29] petan, ping [00:38:47] scfc_de, ping [00:39:31] !ask | CP678|Plane [00:39:31] CP678|Plane: Hi, how can we help you? Just ask your question. [00:39:59] scfc_de, I want to receive a response packet first. [00:40:29] scfc_de, so. When using the migration tool, why does it comment out the crontab? [00:41:15] CP678|Plane: If I remember Coren correctly, it's so that the maintainer can determine if everything has been set up correctly before cron jobs are started again. [00:41:44] Ah. [00:41:47] Makes sense. [02:11:49] does anyone else have problems with the webproxy for instances on eqiad? I got bad gateway errors, deleted the proxy entry and recreated it, then it worked for a while and now it is back to throwing 502s again [02:17:00] dschwen: I don't have experience with or access to that proxy, but "bad gateway" sounds like your webserver isn't delivering a response fast enough (or at all)? [02:17:11] curl fetches the page from tools-login.eqiad [02:17:26] works fine and fast [02:17:45] its just the proxy that complains [02:18:08] it is an exact copy of the config on pmtpa, where webserver and proxy still work fine [02:18:26] I adjusted the sec groups [02:18:55] and, since access from tools-login works I'm assuming the proxy should have access as well [02:19:14] aaaaaaand, deleting and recreating the proxy entry made it work for a few minutes [02:19:32] guess this is really a question for andrewbogott_afk [02:19:40] If it works some time and other times not, that sounds to me like a load issue. But I don't have any insight in the setup or could debug. [02:20:07] yeah, but if there is already a load issue on eqiad... [02:20:15] I sure hope there isn;t [02:23:43] I thought more about your webserver; queries filling up & Co. But that's much easier to debug with someone who has access to the proxy's logs :-). [02:26:37] andrewbogott: I have issues with search on wikitech (no results) i.e. https://wikitech.wikimedia.org/w/index.php?title=Special%3ASearch&profile=default&search=Fingerprint&fulltext=Search [02:27:55] hedonil: did that search turn up pages recently? [02:28:09] I think the search infrastructure was just upgraded, maybe they need to re-run an index. I'll send a note. [02:29:19] andrewbogott: 'k. just noticed that today [02:30:55] andrewbogott: I'm migrating another tool to tool labs, I suppose I might as well set it up in tools/eqiad right away instead of migrating it later [02:31:14] Krinkle: yep! [02:31:15] there seems to be something off about new service groups though, it's not showing up on eqiad [02:31:22] I just created a new tool 'krinklebot' [02:31:49] on pmtpa it is created, but I have no permission to it (I can't touch its files, and also can't migrate the empty tool account, permission denied) [02:32:00] and in eqiad the service group doesn't seem to got created [02:32:32] Krinkle: mentioned earlier here: new tools are not sync'd automatically [02:33:54] hi andrewbogott [02:34:14] did you see my issue with webproxy above? [02:34:27] Krinkle: I created a new tool today -> migrated it 'blank' to eqiad (from tools account) -> finished migration in eqiad (from user account) [02:34:55] from tools account? interesting [02:35:27] dschwen: I don't have time to work on this today, but the quick answer is: dns is flaky. I'm pretty sure the instance is 'working' properly but it doesn't always route to the desired host because it can't find it, due to partial DNS outage. [02:35:35] It's going to be a somewhat hard problem to fix, unfortunately. [02:35:41] Fatal error in defaults handling. Program aborted [02:35:48] ah, ok, DNS explains a lot of my other issues as well [02:36:00] Could not open required defaults file: /data/project/krinklebot/replica.my.cnf [02:36:20] that's ok, just good to know what is going on [02:36:30] I'll just wait for this to go away then [02:36:46] Krinkle: andrewbogott: last step was a chown (done by admin) of new directory /data/project/ (because this is initial broken with this way) [02:36:58] getting more errors [02:37:24] Krinkle: yep. database I assume. doesn't matter [02:37:47] rm: cannot remove `...REMOTE.READY': Permission denied [02:37:53] mv: cannot move `...DATA.public_html' to `public_html': Permission denied [02:37:53] . restoring crontab [02:37:54] rm: cannot remove `...DATA.crontab': Permission denied [02:38:01] Krinkle: yep yep [02:38:34] so an admin has to create that first I assume [02:38:45] the local-krinklebot acount and service group and me being in that group [02:39:05] I assume at least part of that will be picked up by ldap? [02:39:18] it'd get messy very soon otherwise [02:39:28] Krinkle: If the migration script is done in pmtpa it says all copied [02:39:36] Yes [02:39:45] This is in finish-migration on eqiad, the errors above [02:40:01] also still not listed on tools-eqiad.wmflabs.org so I'm assuming it didn't work [02:40:20] Krinkle: what's the tools name [02:40:24] krinklebot [02:40:30] created less than an hour ago [02:41:31] Krinkle: ok. directory /data/project/krinklebot has been created in eqiad [02:41:49] Krinkle: you should be able to become krinklebot [02:42:00] in eqiad [02:42:22] Do you know why the permissions are malformed on tools-login.pmtpa? I created that service group, sure I'm allowed to modify its files? I can do that with the other tools I created earlier fine. [02:42:25] Krinkle: and a $ls -lisa should list some files [02:42:28] Oy. I see what the issue is. [02:42:45] I need to disable new tool creation in pmtpa entirely. It copies a broken tool. [02:43:50] Coren: Can you inspect krinklebot on eqiad/pmtpa and make one of them work properly permission-wise etc.? [02:43:52] And trying to migrate that malformed tool will cause all sorts of isue. [02:44:00] issues. [02:44:19] I don't mind where it runs for now, it's a simple python process that will be kept runing on the grid [02:44:32] Coren: We did this twice today since you were away ;) [02:45:01] worked so far - we complain later about issues ;9 ;) [02:45:14] hedonil: migrate-tool specifically cannot work on anything that didn't exist before the migration. :-) [02:46:10] hedonil, Krinkle: you should have a fully operational batt-- tool. [02:46:32] in eqiad. [02:47:22] Coren: btw. the paste tool thingy has been fixed [02:47:31] hedonil: What was it? [02:47:47] Coren: about 3 changes in the config [02:47:58] 1. new database credentials [02:48:52] 2. .lighttpd url rewrite for RESTful style url [02:49:28] that was all [02:50:54] 3rd was enabling https (in https://tools.wmflabs.org/paste-test/) [02:55:21] hedonil: Isn't 3. Profit? [02:56:46] Coren: yep. liangent work on paste, I worked on paste-test, so https just works on paste-test right now [02:57:15] but you caan do the modification in paste (original) [02:57:17] Coren: Hm.. certificate error to git.wikimedia.org from tools-labs-eqiad? [02:57:24] (git clone https://git.wikimedia.org/git/pywikibot/compat.git pywiki-compat) [02:58:00] o_O That works in the other direction. [02:58:33] error: server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none while accessing https://git.wikimedia.org/git/pywikibot/compat.git/info/refs [02:59:40] My browser likes the certificate, at least [03:00:25] Coren: the webserver is fine, I think it's eqiad (or tools-login-eqiad) that's missing certs for it to work [03:00:25] And I see nothing wrong with it. [03:00:35] I get the error when runing git-clone from tools-login-eqiad for git.wikimedia.org [03:00:45] Try running $git clone https://git.wikimedia.org/git/pywikibot/compat.git pywiki-compat on a tmp dir there [03:00:49] (or home dir) [03:01:04] No, I believe you, I'm just a little puzzled. [03:01:38] or maybe its a bug with git.wikimedia.org and the login server being a bit stricter than usual [03:03:03] Both from on my home computer and on willow.toolserver.org , the same git clone shows no certificate errors [03:03:45] and works from tools-login.pmtpa [03:04:47] Damianz: in the future we'll use hiera and most of these problems will be gone [03:05:11] hierarchy could be common -> labs -> project -> instance [03:05:29] Coren: Krinkle: I did it from tools-dev-eqiad worked, tools-login-eqiad failed [03:06:21] Krinkle: maybe you want to login to tools-dev-eqiad until the bug is fixed [03:08:05] hedonil: How does that work? I mean, does it execute on the same grid? Same /data directory? [03:08:09] Or will it need to be migrated? [03:08:42] Krinkle: no it's just another login host, same grid [03:09:04] Krinkle: no copying, no migration [03:10:45] OK [03:11:05] There are no significant package difference between tools-dev-eqiad and tools-login-eqiad, so must be something sub-package level. [03:12:31] ah, (read the manual), it's not -dev as in beta or less production like environment,but the place to develop stuff [03:12:32] OK [03:12:41] a superset if anything [03:14:09] /etc/ssl/certs seem to be identical on tools-{dev,login}-eqiad as well. [03:15:28] scfc_de: Unsurprisingly, given that those are identical images with the same puppet config. [03:15:30] scfc_de: maybe it's trying to read the CRL which is not accessible [03:16:36] scfc_de: Coren: CRL read error sounds like proxy, firewall, NAT or something [03:17:59] /etc/ssl/certs/ca-certificates.crt has the same permissions and content on tools-{dev,login}-eqiad. Very strange. [03:19:17] Okay, I misread. [03:19:58] hmm CRL is http://rapidssl-crl.geotrust.com/crls/rapidssl.crl [03:20:20] which can be curl'd from tools-login [03:21:16] hedonil: Shouldn't this be local on the filesystem? [03:22:07] scfc_de: usually the CRL is provided by CA or Sub-CA to check for revokek certificates [03:26:26] *revoked [03:27:09] scfc_de: the 'modern' way is to pull that status from CA's webservice via OCSP [03:27:52] which is http://rapidssl-ocsp.geotrust.com [03:28:07] but also accessible fro tools-login [03:30:20] the $ git clone http://git.wikimedia.org/git/pywikibot/compat.git works with plain http even on tools-login-eqiad [03:34:42] http://stackoverflow.com/questions/7814423/ssl-works-with-browser-wget-and-curl-but-fails-with-git has some more hints; "GIT_CURL_VERBOSE=1 git clone ..." isn't verbose enough, though, and /etc/ssl/certs/ca-certificates.crt is the same on both. Too tired to debug further. Good night! [03:38:56] scfc_de: yeah sleep well [03:40:25] andrewbogott: can your magic fingers make wikilinks [[toollabs:]] work on wikitech. As on other wiki's like dewiki and enwiki? [03:41:04] hedonil: interwiki stuff? I can change the setting if you know what setting it is :) [03:41:13] Otherwise best to make me a bug and I'll learn how when I have a moment. [03:41:20] andrewbogott: I have no clue [03:41:57] andrewbogott: I really have no clue... so I'll file a bz ;) [03:41:58] "It turns out that this was a gnuTLS issue. gnuTLS is order sensitive, while openssl is not. I re-ordered the certificates in my intermediate cert file and the problem went away" [03:42:07] If that this, it's downright daft. [03:42:24] hedonil: ty. I'm focussed on a particular bug right now, trying to not get too distracted :) [03:42:43] Coren: applause [03:42:56] andrewbogott: no priority at all [03:43:46] Coren: the buggy GNUTLS (goto fail) hehe [03:51:15] Coren: if you can make a config change in /data/project/paste/public_html/application/config/config.php [03:51:35] Coren: $config['base_url'] = '//tools.wmflabs.org/paste'; [03:52:10] hedonil: I make a rule of not touching endusers' tools. Sorry. That way madness (and blame) lies. :-) [03:52:12] Coren: then paste works perfectly (with https) and paste-test can be vaporized [03:52:56] Coren: 'k fair enough. I'll send it to petan tomorrow [03:53:36] It's actually a lie; of course. I'll disable tools when needed. But you know what I mean. :-P [03:55:18] Coren: yeah, no blaming petan (sending him a message to one of his 200 consoles will do it) ;) [05:29:07] When I try to log in to a couple of my instances I'm getting "Unable to create and initialize directory '/home/madman'." and kicked the hell out. :( [05:35:34] Found a similar thread on labs-l. Is there any way for me to poke gluster without being able to log in via SSH? :x [05:50:55] Doesn't seem so. Should I e-mail labs-l? Create a bug? [10:14:14] Coren: Still getting certificate errors on tools-login-eqiad, now for other tools I already migrated earlier as well [10:14:25] e.g. /snapshots/ fetching mediawiki-core [10:14:33] Fetching origin [10:14:33] error: server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none while accessing https://git.wikimedia.org/git/mediawiki/core.git/info/refs [10:14:33] fatal: HTTP request failed [10:14:35] error: Could not fetch origin [10:14:49] andrewbogott_afk: was swamped with work, thus the late reply. i only used the puppet-cleanup project for testing puppet patches. i deleted my instance as I currently do not have much time to hack on the things I want to :( . the idea was anyway to automate what i did manually there. [10:15:06] So I guess I'll move that crontab job to tools-dev-eqiad also? [10:19:16] andrewbogott_afk: Coren: Filed as https://bugzilla.wikimedia.org/show_bug.cgi?id=62432 [13:07:16] Hello, I moved now with project templatetiger to eqiad, but can not connect to database: [13:07:59] tools.templatetiger@tools-login:~/public_html/dumps$ mysql [13:08:00] ERROR 1045 (28000): Access denied for user 'templatetiger'@'10.68.16.7' (using password: YES) [13:10:34] Kolossos: 'templatetiger' isn't your mysql username (nad .my.cnf doesn't exist) try: mysql --defaults-file=replica.my.cnf -h tools-db [13:12:26] Kolossos: will you use the new naming scheme? [13:13:17] I try: tools.templatetiger@tools-login:~/public_html/dumps$ mysql --defaults-file=replica.my.cnf -h tools-db [13:13:19] Could not open required defaults file: /data/project/templatetiger/public_html/dumps/replica.my.cnf [13:13:20] Fatal error in defaults handling. Program aborted [13:13:43] What's the new naming scheme? [13:15:01] Kolossos: sry I expected you to be in ~ try mysql --defaults-file=~/replica.my.cnf -h tools-db [13:15:56] Ok, than I see some databases. One of them is "s51071__templatetiger_p". Should I try this one? [13:16:24] Kolossos: sounds good [13:17:57] Kolossos: this is it [13:18:09] (the scheme) [13:19:05] database name is scary, and it's empty. But I can fill it again. Positive thing seems that the db is public now. [13:21:22] if we copy ~/replica.my.cnf to ~/.my.cnf this would be easier [14:57:44] hedonil: fixed [14:57:44] hedonil: ( https://tools.wmflabs.org/paste/ ) [14:58:12] liangent: yep, great. [15:03:40] Coren: /usr/bin/webservice script seems to be missing from tools-dev-eqiad [15:04:02] hedonil: how you fixed it? [15:04:13] hedonil: also it's still a bit fucked [15:04:24] open page 2 https://tools.wmflabs.org/paste/lists [15:04:44] err [15:04:51] here [15:04:52] https://tools.wmflabs.org/paste-test/lists [15:05:04] petan: we are pro's ;) [15:05:11] (03:47:47) hedonil: Coren: about 3 changes in the config [15:05:11] (03:47:58) hedonil: 1. new database credentials [15:05:11] (03:48:52) hedonil: 2. .lighttpd url rewrite for RESTful style url [15:05:11] (03:49:28) hedonil: that was all [15:05:11] (03:50:54) hedonil: 3rd was enabling https [15:05:19] hedonil: that's not how, also you aren't pros because this produces https://tools.wmflabs.org/tools.wmflabs.org/paste-test/lists/15 [15:05:37] hedonil: which changes in config? [15:05:42] that is what I want to know [15:05:58] petan: let me look again [15:06:03] have a diff? [15:06:41] petan: liangent: did the url rewrite in .lighttpd.conf I suppose [15:07:15] hedonil: I want to know what you did so that it stopped silently dying [15:07:53] * hedonil searches the thingy again [15:09:10] hedonil: there is still this link bug [15:09:16] it needs to be fixed [15:09:47] petan: 3. https was in /data/project/paste/public_html/application/config/config.php http://foo -> //foo [15:10:10] hedonil: that is the one I am not interested in, btw this config caused the new bug [15:10:20] hedonil: I am interested in fix that make it stop silently die [15:11:42] petan: 2. was change the db creds in /data/project/paste/public_html/application/config/stikked.php [15:11:52] petan: nothing else [15:12:04] come on lol [15:12:11] you fixed it and you don't even know how? [15:12:42] petan: I made it more verbose dring debugging by adding some random echo commands [15:13:04] ok that I did too yesterday and figured it crashes on line where it make new instance of a class [15:13:12] but how you fixed that crash huh? [15:13:19] ok look I have an idea [15:13:22] petan: well we still habe paste-test, so let's check it again [15:13:30] paste-test works [15:13:43] paste will not I will now make paste purposefuly broken as yesterday [15:13:47] petan: yea, bt I will break it now [15:13:49] and you fix it step by step XD [15:13:50] wait [15:13:55] no don't break paste-test [15:13:57] let it as it is [15:14:01] don't touch it [15:14:37] lol [15:14:42] you moved this fix to /paste already? [15:14:44] aha [15:14:55] petan: liangent did [15:15:03] damn [15:15:27] guys you can't fix thing like this, that could be just temporary error that can come back anytime if you don't know what it was [15:15:45] ok let's experiment on paste-test then [15:15:55] petan: let's test something [15:16:52] petan: 1.) in paste-test goto to ../stikked.php and change the db creds to nonsenes [15:17:12] petan: then err 500 shold appear again [15:17:12] they were nonsense yesterday? [15:17:32] petan: old creds from pmtpa [15:17:42] where is stikked.php [15:17:56] /data/project/paste/public_html/application/config/stikked.php [15:17:56] got it [15:19:03] petan: ok, now it' 500 as before [15:19:05] damn that is a stupid php application it silently die on wrong credentials without saying anything bah [15:19:09] yes I see [15:19:15] ok [15:19:22] now we need to fix that https bug [15:19:33] petan: now try to add wait a sec.. [15:19:39] if you put relative protocol there it seems that links are broken [15:22:03] petan: hmm, works for me, I can create new paste, browse menu etc. [15:22:30] http://tools.wmflabs.org/paste/ this doesn't even display css to me [15:22:55] petan: also works for me [15:23:11] petan: ok now fail [15:23:25] I think we need to use only 1 protocol [15:23:29] let's just enforce ssl [15:23:38] so that http redirects to https [15:24:25] petan: the problem is, that the browser blocks it as it's calling http resources from https (unsecure) [15:24:31] https://tools.wmflabs.org/paste/themes/default/css/reset.css is 404 [15:24:35] dunno why [15:24:48] hedonil I know [15:25:01] hedonil: that is why I say we should redirect http to https [15:25:05] so that user is always on https [15:25:12] wikitech is doing this too [15:25:20] it run on single webserver and has only https [15:26:03] petan: right now it doesn't work at all [15:26:10] indeed [15:26:20] I didn't change anything though [15:27:17] the files are there on fs [15:27:28] dunno why lighttpd doesn't show them [15:27:34] maybe the url rewrite bug? [15:28:19] petan: you f*ck'd it up :P [15:28:27] how? [15:28:37] petan: ...but I restored http://tools.wmflabs.org/paste-test/ [15:29:36] petan: my fault. I broke it when fixing favicon.ico [15:30:59] and now fixed [15:31:00] so it seems ... https://tools.wmflabs.org/paste/favicon.ico can't be handled by php but https://tools.wmflabs.org/paste/themes/* must be handled by php [15:31:57] petan: so you are not the one to blame ;) [15:32:14] am I ever? :P [15:32:45] * hedonil notices that https://tools.wmflabs.org/paste/ works totally fine now with http and https [17:35:10] hi. why doesn't wikimedia use SPDY instead of HTTP? [17:40:49] hi, I just got my project accepted on mediawiki/extensions/SimpleSamlAuth [17:40:53] the repository is already populated with one commit adding a .gitreview [17:41:08] but actually I wanted to push the Github repository that i've been using [17:41:09] I [17:42:01] I'm fine with adding .gitreview to this repository, but am I allowed to overwrite the previous commit, and by that introduce a lot of commits that don't contain .gitreview? [18:11:37] (03PS2) 10AzaToth: grrrit: Pass betacluster messages to QA [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/116997 [18:11:40] (03PS2) 10AzaToth: grrrit: Allow filtering based on branches [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/116996 [19:52:01] Anyone around? [19:52:42] Never mind. [20:42:12] * a930913 pets Damianz. [20:42:31] * Damianz screams [20:43:34] * a930913 whips out the choloroform. [20:44:08] Damianz: Any advancement on the redis? :3 [20:50:08] Give me a few min [21:04:19] Looking any better? [21:12:31] Coren, I'm back online and ready to assist in determining the MySQL bug. [21:12:45] Coren, what do you need me to do? [21:16:27] petan, ^ this is why I ping people first. :p [21:16:45] Cyberpower678: ping [21:16:51] petan, pong [21:17:24] ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, pong [21:18:28] pong, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, ping, pang [21:18:37] err, [21:18:53] pong, pong, pong, pong, pong, pong, pong, pong, pong, pong, pong, pong, pong, pong, pong, pong, pong, pong, pong, pang [21:53:45] (03CR) 10Hashar: grrrit: Allow filtering based on branches (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/116996 (owner: 10AzaToth) [21:57:16] (03PS3) 10AzaToth: grrrit: Pass betacluster messages to QA [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/116997 [21:57:19] (03PS3) 10AzaToth: grrrit: Allow filtering based on branches [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/116996 [21:58:27] !ping [21:58:27] !pong [22:02:00] !ping [22:02:00] !pong [22:02:16] AzaToth: login-eqiad slow? [22:02:32] (03PS1) 10AzaToth: grrrit: Send all bug commits to #wikimedia-dev [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/117662 [22:02:44] a930913: haven't tested [22:03:00] Oh, I thought you were pinging for the same reason as me :p [22:03:18] whoops [22:03:50] I was copying the uri, and had the topic marked... [22:04:26] a930913: no lag for me to login, and no percieved lag on login [22:05:28] a930913: you think https://gerrit.wikimedia.org/r/117662 is a good or really bad idea? [22:08:55] AzaToth: Wouldn't know. I avoid all the complicated bugzilla/git stuff as much as possible. :p [22:09:16] heh [22:09:21] Darkdadaah: ping [22:10:04] Darkdadaah: was doing a ps auxfww on tools-login-eqiad.wmflabs.org and notice you have a zombie shell running on pid 11863 [22:10:37] AzaToth: let me see [22:10:49] a930913: the change is to post all commits to branches starting with "bug/" to #wikimedia-dev [22:12:46] AzaToth: I was running an interactive pywikipedia job, could that be it? [22:13:27] Darkdadaah: it's marked as which afaik implies it is zombified [22:13:42] I don't see it :/ [22:13:54] it's not defunct anymore [22:14:01] did you enter it again? [22:14:27] I stopped my bot. [22:14:35] ah [22:14:43] Darkdadaah: http://paste.debian.net/86353/ was the list [22:15:04] so the bot probably was stalled [22:15:16] Coren, ping [22:15:26] It seemed to be working just fine [22:15:35] strange [22:16:00] I'll restart... [22:16:08] is it fully interactive? [22:16:43] yea, the defunct shell is back [22:17:08] but as you said, it seems not affect the execution of the main program [22:17:14] so it's probably in a own shell [22:17:48] s/shell/thread/ [22:19:31] Cyberpower678: I'm quite certain that's a weekend in Canada as well, so I think Coren's only available for emergencies. [22:20:05] By the way, is there a way to run an interactive job on tools? Right now I just run in directly on tools-login. [22:20:16] scfc_de, what's a weekend in Canada? [22:22:18] scfc_de, Coren's in Canada? [22:22:32] I thoought he was in Florida. [22:22:50] Cyberpower678: In Québec, https://fr.wikipedia.org/wiki/Week-end. Same time (and day) zone anyway. [22:24:10] Darkdadaah: Theoretically, you can use the grid with qrsh for that, but using tools-dev for resource hungry jobs is usually sufficient to not step on somebody's toes :-). [22:25:56] scfc_de: running qrsh tells me not to manually run my tool here :P [22:28:34] Darkdadaah: :-) The grid should manage the resources just fine, *but* the exec nodes do not have all installed packages of the login nodes, so that may be a problem for you. [22:30:06] Damianz: <3 [22:30:49] thought you'd died [22:31:32] Damianz: I was filling out an application form for longer than expected, then tried to get it to work. [22:32:11] Damianz: http://pastebin.com/scbLa0fQ [22:33:18] Damianz: Where does "Reverted before" come from, and why isn't edit_score there? [22:33:52] That's 'right' - but I've not really standardised a data struct. Some of that stuff should be removed and other bits restructured. All the things in the irc line should be in that dict [22:38:09] a930913: better? [22:38:16] * Damianz notes these classes need refactoring [22:38:54] Erm, lemme unchange my code :p [22:40:41] You should have a revert_reason (stupid name because it's also the not reverted reason... but that's what it is in code) and edit_score [22:40:56] edit_score should really be ann_score and might not be set [22:44:12] Damianz: Not set as in KeyError or not set as in "N/A"? [22:44:58] null probably or empty string.. it's an uninit'd var so guessing null. Can override it in the sanitize function if needed [22:45:48] Damianz: Why would it be null? [22:46:19] Edits that hit the pre filter (like not in main namespace) don't get scored [22:46:26] They show up as N/A in irc [22:48:35] Damianz: Yeah, so they'd be "N/A", no? That said, I just saw one that gave what looked like a python None, repeated with the real score. [22:49:49] ok... it's not very programatic... but now it's N/A [23:22:50] Betacommand: Available to help me with some python unicode troubles? [23:23:46] a930913: yeah [23:26:43] Betacommand: So I'm collecting strings from redis, loadsing the JSON, and using it to make some JSON that I push into another redis. [23:27:11] Betacommand: Webservice page fetches the second redis and prints it. [23:27:26] Somewhere I'm ending up with \uxxxx everywhere. [23:28:05] a930913: first thing to do is track down after what step you have the screwed up text [23:28:25] I tend to throw a lot of file logging in at this point [23:28:44] Betacommand: Yeah, but I have no idea wtf py2 does with unicode. [23:28:51] IE logging the same thing repeatedly to a file as it gets processed [23:29:18] a930913: the odds are its not a py issue, its something you screwed up in your code [23:29:43] throwing in the excessive logging should help [23:29:58] In the same way a MySQL hole in PHP is not a PHP issue? [23:31:10] No, the odds are you are either mixing up your encoding, or during the redis > python > redis process you are doing something wrong [23:31:37] a930913: if you want you can take the lazy mans process of logging [23:31:50] log the start, end, and middle [23:32:16] then log halfway between the last known good point and the known point of failure [23:32:22] Betacommand: I can't recognise what's what. [23:32:57] a930913: you have a string that you are working with correct? [23:33:14] Plus, there's a lot of data that doesn't cause this issue. [23:33:41] a930913: its just non ASCII text that messes up correct? [23:33:59] Betacommand: Think so. [23:34:38] Betacommand: Could json.dumps be doing it? [23:35:21] a930913: it shouldnt [23:35:37] let me check something [23:37:35] a930913: yeah it does [23:37:52] \o/ I'm not mad :D [23:37:58] thats how json is treated [23:38:13] Not according to spec. [23:38:21]