[01:24:11] I am having trouble with a cgi script not being executed as cgi : http://tools.wmflabs.org/veblenbot/cats/cats.cgi [01:26:08] how do I get the server to run .cgi files as cgi files? [07:10:32] is there limit to number of instances per project? [07:23:08] oh, Nikerabbit, was about to ask same ;) [07:23:44] project instancecounter [07:27:48] kart_: how long did you keep this doubt in your heart? :) [07:36:36] mutante: was that an answer? [07:38:52] Nemo_bis: heh. There was question in internal list and, I went to Lunch etc. [07:39:01] Nikerabbit: no, joking [07:43:10] I doubt there is a limit [07:43:18] Otherwise it would be documented, no? :P [07:51:50] nobody ever hit it [07:52:03] i'm sure there is _a_ limit but it might be very high [07:52:18] and if it's 65532 or whatnot [07:57:42] mutante: that helps. will check. [07:58:55] mutante: Runa was trying to create 11th instance for Language Team and it failed. [07:59:01] I'll ask for more details. [07:59:55] ok. Same for me. It says: Failed to create instance. [07:59:59] that's all. [08:00:33] kart_: i asked Ryan, confirmed it's either "no limit" or at least "nothing you would ever hit in a reasonable project" [08:01:37] now, that said, you'll need to have your quota increased [08:02:25] you can see your current quotas via the "Manage projects" interface [08:04:19] Ryan_Lane: yes. 10 it is. [08:04:29] yep. so, someone needs to increase your quota [08:04:49] mutante: can you help ^ [08:09:23] kart_: try again.. (docs are outdated it seems) [08:09:37] projectname "languageteam" right [08:14:40] !log languageteam upped instance quota from 10 to 20 [08:14:40] languageteam is not a valid project. [08:15:00] labs-morebots: nova thinks it is a valid tenant [08:15:00] I am a logbot running on tools-exec-08. [08:15:00] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [08:15:00] To log a message, type !log . [08:15:05] the project is just "language" [08:15:15] nova quota-show --tenant languageteam | grep instances [08:15:15] | instances | 20 | [08:15:28] (wikitech docs are for older openstack version) [08:15:34] this is from docs.openstack [08:15:54] nova quota-show --tenant language | grep instances [08:15:54] | instances | 10 | [08:16:11] | instances | 20 | [08:16:14] upped [08:16:34] !log language upped instance quota from 10 to 20 [08:16:36] Logged the message, Master [08:17:19] kart_: [08:26:17] mu [08:26:22] mutante: thanks [08:27:37] Need remote irrsi+notification now for myself. [10:07:14] How can I do RRD stuff? [10:19:32] Coren: I'm getting "sockets disabled, connection limit reached" errors for a tool used by many users. It doesn't seem to be possible to increase server.max-connections from the lighttpd conf file [10:22:49] a930913: what is RRD [10:23:02] apper: did you restart the service [10:23:21] I don't really know lighttpd, but it should be possible to adjust this [10:25:42] I tried this some days ago, so I don't know the correct error message but it was something like " server.max-connections is already set" [10:25:58] wait, I'll try it [10:27:43] "Duplicate config variable in conditional 0 global: server.max-connections" [10:29:17] apper: you can add additional params to your .lighttpd.conf ( += ), but afaik you can't override the 'global' default, if it's already set [10:29:22] apper: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Default_configuration [10:30:16] hedonil: I know this, "max-connections = 20" is very low [10:31:34] hedonil: the tool is called for every page view (for articles) for every user who activated it. So if there are 40 users parallel surfing the wikipedia, the problems occur [10:33:51] I also got "backend is overloaded" errors yesterday. The tool was promoted yesterday at de.wikipedia and there are several new users. Since then there are problems (escpecially in the evening) [10:34:41] apper: yeah, celebrity sucks some times :-D [10:36:08] apper: maybe with your own instance (or own, not shared) webservice you could do some tricks [10:38:14] maybe... ;) [10:38:51] But I thought using lighttpd on the grid should be scaleable [10:40:40] I don't really have the time to get into the whole labs stuff, moving tools from toolserver is time consuming enough ;) [10:43:11] apper: I feel with you! The other way would be (of course) just raising the limits ;) [10:43:25] hedonil: yes, that's why I'm here ;) [10:44:23] But Coren seems to be the only one able to do this [10:44:29] petan: Round robin database. [10:45:21] apper: Coren: raise da limit. and/or add not only one, but two new webgrid boxes :-D [10:45:24] moar [10:46:51] Ah, I have another question about lighttpd... error.log and access.log - the first sometimes have messages I read and corrected and the second one gets bigger and bigger. But when I just delete them they are not recreated by lighttpd until a server restart. Is there a proper way to null these files? [10:47:31] apper: just try $ >error.log [10:50:15] hedonil: hm... seems to work for error.log, but access.log is the same size as before, but filled with blanks... [10:50:44] * hedonil checks [10:50:57] ah, okay, it seems to be 0 for a short time, but then the server writes at position 15M for example [10:51:22] so the file is empty the first 15 MB and then the new log entries are added [10:52:35] o.O [10:52:41] the third time it worked... [10:53:01] apper: yeah, just typed while you answered ;) [10:53:05] apper: $ >access.log && >access.log && >access.log && >access.log [10:53:29] hedonil: okay, thanks! [10:53:34] apper: because maybe it's in use when you type it just once [10:54:01] hedonil: yes, that's possible :) [10:57:19] ah, maybe scfc_de is able to change configuration files ;) [10:57:40] scfc_de: are you able to increase lighttpd max-connections for a tool on tool labs? [11:01:13] apper: I think I can increase the "workers". What's the tool's name? [11:01:23] scfc_de: wikihistory [11:01:49] scfc_de: I think the max-connections value of 20 is very low, I got "sockets disabled, connection limit reached" erros [11:02:38] scfc_de: and "backend is overloaded" errors, too. Yesterday in the evening when a lot of users used it [11:03:21] apper: Let's see what I can change and what effect that has. [11:05:11] Hmmm. IIRC the default number of workers is 5, and /data/project/.system/config/geohack.workers has an exemption of 20 for geohack. So I already don't understand how you would get 20 in the first place :-). [11:05:37] workers is 5, max-connections is 20 [11:05:59] according to https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Default_configuration [11:08:28] Ah! Okay, now I see. lighttpd-starter sets workers to 5 and max-connections to workers * 4. I've now set /data/project/.system/config/wikihistory.workers to 20 which should give you max-connections = 80. Could you "webservice restart", please? [11:09:15] scfc_de: done [11:10:28] apper: Can you whether that worked? [11:10:28] scfc_de: thanks a lot, we'll see how it's doing this evening ;) [11:10:56] apper: k [11:11:06] scfc_de: I don't really know, how... I only have problems when there are a lot of calls [11:11:23] !log tools Set /data/project/.system/config/wikihistory.workers to 20 on apper's request [11:11:26] Logged the message, Master [11:12:09] scfc_de: there was an article in de:WP:Kurier about the tool yesterday and the number of users increased a lot, and so it was overloaded yesterday in the evening [11:13:14] That's not necessarily a bad thing :-). [11:14:57] scfc_de: more users are cool, unsatisfied users because it doesn't work are not cool ;) [11:15:46] apper: wikihistory looks great! congrats. How about wikis other than german ? [11:16:15] apper: and an API for all of that ;) [11:18:38] hedonil: the crucial part is the calculation of the text percentages for users: I have five queues for de.wikipedia, this works relatively smooth now... I will have to think about how to handle smaller wikis and of course en.wikipedia is very interesting, but I think, then I really should move to an own instance... [11:19:52] apper: would be great. your job is never done hehe [11:20:00] hedonil: calculation is maximum some seconds for most articles, but there are articles, where the calculation lasts hourts... [11:20:11] hours [11:20:35] apper: http://tools.wmflabs.org/wikihistory/wh.php?page_title=Der+Denver-Clan [11:20:54] hedonil: uuh, main author ;) [11:20:57] apper: looking at this, I can found my name in the graphics, but not in the table [11:21:22] hedonil: you have to re-order by "Textanteil" [11:21:46] hedonil: the table sorting javascript has a bug... I didn't really had time to get into this [11:21:47] apper: yeah, thought it was default .... [11:22:24] it's obviously called (the header is changed), but it doesn't sort the rows... [11:23:33] maybe I just shouldn't call it directly but with setinterval or something like this... I will have a look at this later [11:24:10] apper: fix *ALL* the bugs ;) [11:24:17] now I need a shower ;) [11:25:07] but great tool anyway. [11:25:24] * hedonil likes quick debug sessions on new, mindbreaking tools [11:26:10] apper: 404 on resource jquery-2.0.3.min.map [11:34:45] apper: Very neat tool. Now if these (and others) could make their way into a MediaWiki extension ... :-) I think that's probably not interesting only for Wikipedia. [11:40:12] hedonil: this 404 only comes from your debugger, it's the jquery map file [11:40:32] hedonil: I can put it there for better debugging, of course [11:41:08] apper: 'k. just mentioning ;) [11:41:26] * hedonil gives feedback if he finds other things [11:41:44] hedonil: thanks [12:51:11] scfc_de: apper: Yeah, that was exactly what you had to do. [12:51:21] * Coren needs to add a few nodes to the grid. [12:53:31] Coren: great, thanks [12:58:01] Moar powaaar! [13:17:53] Ping: Petan [13:17:53] ^^ It is possible to translate @seen to de? :) [13:22:52] Steinsplitter: How would the bot know what language the user speaks? [13:23:43] scfc_de:it is possible to change bot language [13:23:57] https://github.com/benapetr/wikimedia-bot/blob/master/Translation/english.txt [13:28:13] Steinsplitter: So you don't want to make wm-bot's answer depend on the user who asks, but on the channel? And you want wm-bot to be changed so that @seen /can/ be translated? [13:29:35] yes, seen schould /can/ be translated ;) [13:29:42] it is not on the list, see github [13:30:23] * Coren almost wishes he had prebaked exec node images because /man/ is that first puppet run long! [13:34:22] (AFKish) why would a cgi be downloaded/source spewed instead of executed? [13:34:23] I don't think you're supposed to stand by and watch :-). [13:35:38] a930913: Not recognized by the server as such? https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Web_services [13:37:30] scfc_de: It's not enough to just have a hashbang and mark it executable? [13:40:34] a930913: By default, I don't think so. There may be options to lighttpd to enable that, but I don't know them. [13:45:25] Look at the "apache-like cgi-bin" section of the help file. [14:48:00] Coren, minor lag found on fiwikisource.labsdb [14:48:09] Oh dear. The database appears to be lagging behind. I won't be able to present you with information newer than 12844 seconds. [14:49:00] Just thought I'd mention it. [15:01:13] Cyberpower678 seems to be gone, but last recentchanges on fiwikisource_p is 20140404111255 which is consistent with https://fi.wikisource.org/w/index.php?title=Toiminnot:Tuoreet_muutokset&hidewikidata=0&hidebots=0. "Oh, dear." [15:03:23] scfc_de, i'm still here [15:06:31] CP678: So? [15:06:55] scfc_de, you said Iwas gone [15:07:52] CP678: You said "minor lag found on fiwikisource.labsdb". [15:08:18] A 12844 lag. :p [15:09:49] Where is that lag? [15:26:14] anyone know if bits on beta labs is having problems right now? [15:48:36] !log tools Moar powar!!1!one: added two exec nodes (-09 -10) and one webgrid node (-02) [15:48:38] Logged the message, Master [16:00:02] hi Coren, hashar is not around, do you know if bits on beta labs has a problem at the moment? http://en.wikipedia.beta.wmflabs.org/ [16:01:02] chrismcmahon: I know so little about how beta works it's not even funny. As far as I can tell, the infrastructure is fine. [16:01:35] Coren: bummer :) [16:07:48] chrismcmahon: Do you know the name of the server that provides bits? [16:08:17] chrismcmahon: What's the problem? (Not that I would know how to help :-).) [16:08:17] http://bits.beta.wmflabs.org/en.wikipedia.beta.wmflabs.org/load.php?debug=false&lang=en&modules=ext.flaggedRevs.basic%7Cext.gadget.PagesForDeletion%2CwmfFR2011Style%7Cext.uls.nojs%7Cext.visualEditor.viewPageTarget.noscript%7Cext.wikihiero%7Cmediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.ui.button%7Cskins.common.interface%7Cskins.vector.styles%7Cwikibase.client.init&only=styles&skin=vector&* seems to work for example. [16:08:25] DB connection error: Unknown MySQL server host 'zhwiki.labsdb' (2) (zhwiki.labsdb) [16:08:32] in my tool's output [16:08:43] haven't diagnosed yet [16:09:18] liangent: Oh, I keep forgetting this. The hostname keeps being removed from hosts for some reason. [16:09:21] wow, scfc_de Coren all of a sudden beta looks normal again. [16:09:32] chrismcmahon: Not my doing. [16:09:56] Coren: so what should I do now? [16:10:01] I'm OK with magic, thanks [16:10:10] chrismcmahon: The jobs queue seems to be really active in beta right now. It may be eating all the apache workers. [16:10:17] liangent: Nothing; Coren will fix it in a moment. [16:11:44] thanks bd808 [16:11:44] scfc_de: not even restarting the script? [16:12:40] liangent: That's a good question :-); but you'll have to wait with that until hosts and iptables are up anyhow. [16:15:06] !log deployment-prep Performing a rolling restart of Elasticsearch nodes to pick up a new plugin [16:15:08] Logged the message, Master [16:16:45] liangent: Should work now? [16:18:14] Coren: it now says "DB connection error: Can't connect to MySQL server on 'zhwiki.labsdb' (4) (zhwiki.labsdb)" [16:18:18] (without a restart) [16:19:01] liangent: I can connect to it fine Where are you getting that error from? [16:19:09] chrismcmahon: Making direct curl requests on deployment-apache0[12] works for me, but I'm getting connection refused errors from my browser. Unfortunately I don't know how the varnish setup in beta works to debug that layer. [16:20:03] Coren: seems fine now [16:20:34] a bunch of "DB connection error: Unknown MySQL server host 'zhwiki.labsdb' (2) (zhwiki.labsdb)", followed by a single "DB connection error: Can't connect to MySQL server on 'zhwiki.labsdb' (4) (zhwiki.labsdb)", then followed by normal output text [16:22:50] liangent: Took a moment to update on the new exec nodes, I'm guessing. [16:23:32] Coren: Webservice not running, error.log: connect to portgranter: No such file or directory [16:24:32] a930913: Hm. Apparently, the new webgrid node is missing a puppet class. [16:25:08] Ah, no, it was applied but the daemon had not started. Try again? [16:26:29] Coren: (configfile.c.958) opening configfile /var/run/lighttpd/cluestuff.conf failed: Permission denied [16:27:09] Ah, I see why. Fixed manually, incoming puppet patch soon. [16:27:26] * a930913 nods slowly. [16:28:28] Coren: How soon is soon? [16:28:54] In a couple of minutes, but the manual fix should allow you to start your webservice. [16:29:59] Coren: As in manual fix should work for me already? [16:30:07] * Coren nods. [16:30:50] Ah, new errors now. [16:31:17] It now fails for the right reason? :-) [16:31:53] Coren: Nope, I commented out all my changes. [16:32:47] Coren: the fastcgi-backend /usr/bin/php-cgi failed to start: \n child exited with status 2 /usr/bin/php-cgi [16:33:14] The program 'php-cgi' is currently not installed. D'oh! [16:33:20] * Coren adds to the same patch. [16:33:37] If it's php, it's not my problem :p [16:33:53] a930913: No, but it still fails to start the backend and thus goes boom. [16:34:00] Try again? [16:34:07] Coren: As in it's not my fault. [16:34:21] * a930913 rolls a drum. [16:34:30] * Coren groans at Jenkins. [16:35:13] Coren: Yay, back to user errors I think. [16:35:44] Sorry about the inconvenience. Everytime I deploy a new server I notice something else I forgot to puppetize, it seems. :-) [16:36:12] I understand up to server :p [16:36:16] * Coren wonders where Jenkins has gone off to. [16:37:18] Coren: (configfile.c.912) source: /var/run/lighttpd/cluestuff.conf line: 585 pos: 1 parser failed somehow near here: (EOL) [16:37:31] Is that my problem or your's? [16:39:12] * a930913 thinks it's his. [16:40:09] No starting 585 is what comes from your .lighttpd.conf [16:40:13] * Coren takes a look. [16:40:26] Coren: I think I might be clashing with the default? [16:40:59] a930913: second last line [16:41:05] No, that wouldn't give you a 'parser failed'. [16:41:20] You know, I don't actually know that # is a comment marker in lighttpd config. [16:41:31] Coren: I'm trying to get /data/project/cluestuff/public_html/cgi-bin/vandalstat/index.cgi to work. [16:41:56] Coren: I think I was reading the docs when I started commenting. [16:42:16] It should be "#". [16:42:23] http://redmine.lighttpd.net/projects/1/wiki/Docs_Configuration [16:42:24] It might be, I just don't know that it is. :-) Lemme try and see what you're doing. [16:42:28] scfc_de: or += [16:42:45] hedonil: That's not a comment? [16:43:16] either uncomment all, or += [16:43:38] Hm. no. always += [16:43:48] a930913: The bit that fails is cgi.assign = ( ".cgi" => "" ) [16:43:58] For sure, you'd need a += [16:43:59] Coren: Yeah. [16:44:04] Trying now. [16:44:25] Holy **** it's working. [16:44:37] * a930913 changes some config. [16:49:34] Awesomeness, can now graph wiki stats over time :) Thanks guys. [16:51:57] Also, I managed to make install something locally and get it to work :D [16:59:17] Do beta labs have IRC rc channels (for testing)? [17:00:14] Coren: Can we restart a given job? I.e. kill it so it restarts and uses the latest version? [17:02:02] huh: no there is bug for that almost for 2 years [17:02:12] huh: Have you looked at the channel list at irc.wikimedia.org? [17:02:57] a930913: You can try "qmod -rj". [17:02:57] scfc_de: oh, thanks - they're there [17:03:22] sorry [17:03:24] Did someone here recommend Ganetti before? [17:08:31] hedonil, hi [17:09:10] and bye. Gotta go. [17:19:54] scfc_de: Thanks, it's now state Rr? [17:21:24] oh, that really works? [17:21:52] !log deployment-prep Forced puppet run on deployment-cache-bits01 [17:21:55] Logged the message, Master [17:22:06] a930913: That should be the status after rescheduling (AFAIUI). [17:22:52] !log deployment-prep Rebooting deployment-cache-bits01 [17:22:53] Logged the message, Master [17:23:46] scfc_de: Hmm, how would I know if it really is running the right source? [17:28:06] Wait for the effects I suppose. [17:28:14] a930913: Your logs? :-) "qmod -rj" doesn't preserve the process memory, so it should be equivalent to a new submission. Which source that pulls depends on what the submission call looks like. [17:28:22] Cyberpower678: oh, Hi CP. you fixed it. applause ;) [17:28:46] Thank you. The credit however has to go to a930913 [17:29:31] hedonil, ^ [17:29:49] Cyberpower678: so what was it? [17:29:51] !log deployment-prep Manually fixed puppet config on deployment-cache-text02 (the cert html error problem) [17:29:53] Logged the message, Master [17:30:03] go bd808 go :-) [17:30:10] hedonil, the tspan's dy parameter wasn't 0 [17:30:36] Cyberpower678: teh [17:30:43] Cyberpower678: It passed? [17:30:51] Oh, that fix? [17:31:02] passed? [17:31:05] !log deployment-prep Forced puppet run on deployment-cache-text02 [17:31:07] Logged the message, Master [17:31:14] The UAT [17:31:19] Yes [17:31:24] a930913, ^ [17:33:47] why is there in the help page the information that qdel only takes job numbers? it also takes job names [17:35:45] Because noone knew then and since? [17:36:53] Coren: Hmm. Don't know if you even noticed.. [17:37:12] someone has added moar power to the grid [17:37:25] three in numbers... [17:37:33] hedonil: it was coren [17:37:46] Moar powar! Muahahaha! [17:37:47] gifti: ;) [17:38:08] so it's time now to compute the Answer to The Ultimate Question of Life, the Universe, and Everything [17:40:07] gifti: btw. since when did you change your nick? [17:41:23] * hedonil finds it always suspicious if hair is cut, furniture is moved or nicks are changed .. :-) [17:41:28] since the pmtpa-eqiad move <.< >.> [17:41:45] Coren: I'm seeing a really confusing problem in the deployment-prep project. I can `curl -v http://deployment.wikimedia.beta.wmflabs.org/wiki/Main_Page` and get a response, but if I access the same URL from Chrome or Safari I'm getting a connection refused error. Any ideas what would cause that? [17:42:02] Curl and browser access are from the same source host/ip [17:42:19] gifti: this one sounds better [17:42:52] All I know about the inbound path is that this hostname is managed by [[Special:NovaAddress]] [17:42:54] bd808: Well, I'd suspect something a bit of javascript would be trying to do -- curl won't try that. [17:43:27] bd808: I can access that URL (from outside Labs). [17:43:31] bd808: Hm, actually, that works fine from here. [17:43:54] bd808: ... hey, maybe you have something odd in your local hosts file? Did you perhaps put an address in there during the migration to point at pmtpa? [17:44:16] That's possible; /me looks [17:46:20] Hi, everyone [17:46:33] I asked some questions yesterday and people were very helpful [17:46:37] now I have another question [17:46:52] I have a file with the results of a query to the replica databases [17:47:04] I would like to store the data in a local database so I can query that [17:47:09] how do I do that? [17:48:49] Bluma: 1st you need a databases on the replica (create it) [17:48:58] Coren: No /etc/hosts cruft. I think I see the difference though. Chrome is redirecting to https and that is the blackhole. [17:49:11] Bluma: 2nd: Use: INSERT into ... SELECT... [17:49:55] right [17:49:59] I know that part [17:50:08] problem is that I can't work out how to do step 1 :) [17:50:16] do I have to log in separately? [17:50:26] when I try to run create table, I get a permission denied error [17:50:43] Bluma: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Creating_new_databases [17:51:21] Bluma: before creating a table you must create your own database of course [17:51:57] Bluma: you can't create a table in the 'official' schema like enwiki [17:52:29] I saw that, too [17:52:45] but how do I get to tools-db [17:52:47] ? [17:53:01] Bluma: $ sql local [17:53:33] ah [17:53:40] and I create a new database there? [17:54:08] Bluma: but keep in mind. you can't use INSERT into ..SELECT from replica to tools-db [17:54:21] right [17:54:23] not a problem [17:54:29] I have the data in a tsv file [17:54:57] is mwparserfromhell installed on tool? [17:55:21] Bluma: pastebin some lines of that file [17:56:08] Bluma: you have 2 choices [17:56:20] 1. replica -> file -> tools-db [17:56:47] 2. replica -> replica (your db) [17:57:12] Coren: ^ [17:57:30] Betacommand: No. [17:57:51] * Betacommand goes to have it installed [17:57:54] http://pastebin.com/SQYtd8aK [17:57:55] what is Use of uninitialized value $prog in scalar chomp at /usr/bin/jsub line 112. // Use of uninitialized value $prog in -l at /usr/bin/jsub line 113. // Use of uninitialized value $prog in pattern match (m//) at /usr/bin/jsub line 127. // Use of uninitialized value $cwd in scalar chomp at /usr/bin/jsub line 129. // Use of uninitialized value $cwd in concatenation (.) or string at /usr/bin/jsub line 130. [17:58:00] (where //s are newlines) [17:58:31] got a mail 2 days ago with these lines in its body [17:59:17] Bluma: hm. how about choice #2 (direct on the replica) it's easier [17:59:48] liangent: Usually, that means the host executing jsub was out of memory and the call to "which" failed. Not your fault, nothing you can do about it. [18:00:30] Bluma: I'm in the right mood to talk you through this ;) [18:01:28] scfc_de: ok [18:02:02] hedonil: that's good :) [18:02:35] Bluma: ok. let's go for it [18:02:48] which wiki is the data from? [18:03:08] enwiki [18:03:29] Bluma: $ sql enwiki [18:04:06] hedonil: done [18:04:25] I have a maria-db prompt now [18:04:30] Bluma: select current_user(); [18:04:49] your MySQL username [18:04:53] yup [18:04:56] got it [18:05:14] what do you want your database to be named? [18:05:40] afcresearch [18:06:21] Bluma: > create database __afcresearch [18:07:32] ok, done [18:07:56] Bluma: > use u4817__afcresearch; [18:08:03] yes [18:08:45] Bluma: ok. now your query insert into Select ..... [18:09:46] I have it in a file; I can take it from there, right? Using load data infile? [18:10:17] Bluma: better > CREATE TABLE AS (SELECT ...); [18:10:36] you don't have a table yet ;) [18:11:02] hedonil: I don't have the query, just the resulting table data; I can reconstruct the query but it will take time [18:11:11] can I just create a table and then load the data in? [18:11:49] Bluma: either way. but the latter will create all columns & stuff [18:12:19] hedonil: is that a problem? isn't that a good thing? [18:12:37] Bluma: no it's even better [18:12:39] (I am assuming that by 'the latter' you mean loading the table from the file rather than selecting it) [18:13:06] Bluma: no the latter meant creat table from Select [18:13:22] Bluma: 'load data infile' requires a special right which you don't have. afaik [18:13:22] ah [18:13:28] I see [18:13:44] hedonil: ok, I will have to reconstruct the query then [18:14:03] I will take some time; I may come back later and ask whoever's around for help [18:14:07] * Betacommand waits for bug 63539 to be fixed [18:14:12] Bluma: np ;) [18:14:17] hedonil: thanks for what you've done so far [18:18:55] Bluma: btw. that's why I asked for a sample of your file, as it has to be rewritten as SQL dump & properly escaped & ... [18:26:42] Bluma what do you need? :o [18:32:38] petan: can you fix bug 63539 ? [18:33:35] I can but I am not allowed... [18:33:48] I can only do the same as you, add it to gerrit [18:33:53] and wait for someone to merge it [18:33:54] Reedy: Do you know what's causing "PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/fss.ini on line 1 in Unknown on line 0" in scripts for beta-config-eqiad? [18:34:01] I think we used to have that in production for a short while and then it got fixed [18:34:11] Would be nice to get rid of that in labs as well [18:34:11] petan: thats more than I can do [18:34:14] petan: [18:34:44] Krinkle: comments starting with # are deprecated? wow what a nice move :D [18:34:56] Betacommand: are you sure? :P [18:34:59] Jamhttps://integration.wikimedia.org/ci/job/beta-code-update-eqiad/1720/consoleFull [18:35:20] @search gerrit [18:35:21] Results (Found 6): gerrit, whitespace, git-puppet, gerritsearch, ryanland, gitweb, [18:35:26] !git-puppet [18:35:26] git clone ssh://gerrit.wikimedia.org:29418/operations/puppet.git [18:35:45] wm-bot: I would marry u if it was legal [18:35:45] Hi petan, there is some error, I am a stupid bot and I am not intelligent enough to hold a conversation with you :-) [18:36:00] petan: I have zero clue how to go about that. I zero about puppet and git drives me nuts [18:36:18] Betacommand: I can understand you more than you think [18:36:59] that's why I made this "git for idiots" guide, it's for people like me [18:37:37] petan: even then I use windows and its beyond me to figure out the clusterfuck thats git/gerrit [18:41:42] Betacommand: where is mwparserfromhell is it packaged? [18:42:28] I can make .deb packages but I don't know how to upload them to local repository, so, if it's not anywhere in wmf repo, I am not allowed to install it [19:00:36] petan: Just put them in /data/project/.system/deb and run update-repo.sh [19:01:52] petan: pip mwparserfromhell [19:02:45] thats how you install python packages on a normal computer :P [19:04:35] Betacommand: Doesn't work that well in a cluster with two dozen servers and hundreds of users :-). [19:08:16] scfc_de: running the same command across multi servers? [19:11:09] Betacommand: No; making sure that it is run again when tools-exec-11 is added; making sure that the dependencies are properly stated; making sure that it doesn't do something "funny"; etc. [19:20:37] scfc_de: pip takes care of any dependencies [19:22:21] scfc_de: puppet doesnt have a way to ensure that a particular command is run when its deployed to a new machine? [19:22:27] Betacommand: And what will it do if Python package A has already been installed as a .deb in version 1, but the package to be installed needs version 2? [19:23:00] Betacommand: Packaging a Python module as a .deb is a one-time process, and it has benefits for other Debian/Ubuntu users as well. [19:24:31] scfc_de: python packages dont use deb unless its the exception to the rule [19:25:06] scfc_de: pip is the python packaging tool [19:25:22] oh, a war [19:26:32] Betacommand: I'm well aware of that. And if the servers were only running Python, pip might be a good idea to manage them. [19:27:49] !python [19:27:49] http://1.bp.blogspot.com/_i9u1_OaCZM4/Sgmoh8J6K-I/AAAAAAAAD30/YV6FJkJ_v6A/s400/luanda2.jpg << python waiting to eat all your resources and you [19:29:00] petan: thats not python, thats the programmer [19:49:25] for my curiosity - https://gerrit.wikimedia.org/r/#/c/123902/1/maintain-replicas/maintain-replicas.pl doesn't have other wb_* eg wb_terms listed. how are they whitelisted for public view? [19:50:50] liangent: Aren't those the tables? [19:51:06] liangent: "@fullviews" [19:51:41] liangent: Or is wb_terms a table as well? [19:53:45] scfc_de: http://pastebin.com/KJDXKVki http://pastebin.com/wMmeC7xG [19:54:57] liangent: That's Toolserver? [19:55:40] scfc_de: labs. [19:55:51] thats current schema on tool labs: https://tools.wmflabs.org/tools-info/schemas.php?schema=wikidatawiki [19:55:54] toolserver is "mysql> " labs is "[MariaDB] " [19:57:36] liangent: Okay; for slow ones like me: So wb_terms *is* available on Labs, but isn't listed in maintain-replicas.pl? [19:59:27] scfc_de: yeah [19:59:39] and I'm curious where it came from [20:01:49] hedonil: is tools-info yours? [20:02:20] gifti: sí [20:02:51] i wonder if you should present tool directories to the public [20:03:16] liangent: Probably manually hot-fixed by Coren. You might want to file a bug so it isn't forgotten to hardwire this. [20:03:38] gifti: why not? It's OpenSource [20:04:27] and also more vulnerable if you forget to set the right permissions to your files [20:05:44] yeah we had this discussion. but by default permissions are ok, and security by obscurity is no promising way... [20:05:44] 3:1 :-). [20:05:51] scfc_de: ;) [20:06:52] It's not security by obscurity, it's not teasing script kiddies & Co. [20:07:34] scfc_de: Seriously, man, *anyone* can just create a labs account and go look. [20:07:55] scfc_de: I'm not sure how /useful/ it is, but it's not a security issue AFAIAC [20:08:00] maybe it's just unfamiliar to some lads. but if score outnumber 10:1 I'll surrender ;) [20:08:29] Coren: I didn't say it was a security issue. I said it attracted the wrong people. [20:12:25] Hi, I'm back [20:12:57] silly question: how do I get into the database I just created? [20:13:29] sql responds that it's not familiar with that database [20:13:29] Bluma: $ mysql --defaults-file=~/replica.my.cnf -h enwiki.labsdb u4817__afcresearch [20:13:34] ah, ah [20:13:36] thanks :0 [20:15:03] got in, thanks [20:17:39] Bluma: a more detailed version of 'how to MySQL' on Labs [20:17:45] !mysql [20:17:45] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Shared_Resources/MySQL_queries [20:31:40] thanks! [20:32:36] Bluma: my pleasure [20:44:25] springle: ore aynone who definetly knows: out of curiosity: how is wikitext stored and accessed in production? rumors say it's not in a 'regular' table. [20:49:35] hedonil: *dejavu* Didn't you say recently you knew that? #wikimedia-operations might be a better place to ask, but you can look in operations/mediawiki-config:wmf-config/db-eqiad.php that sets wgDefaultExternalStore and then follow the trail from https://www.mediawiki.org/wiki/Manual:$wgDefaultExternalStore. Caution: Very complex and seriously underdocumented. [20:50:17] scfc_de: hey, great. thx [21:03:30] So this external storage is just another set of MySQL databases with longblob & MyISAM Engine [21:11:37] scfc_de: all I knew was that (replica) text table holds no data. [21:12:10] scfc_de: but now, this can be tagged as solved. no mysteries there [21:13:36] hedonil: The replica text table holds no data just as the production one doesn't. [21:14:23] Coren: yeah, just read the docs and inspected some of the maintenance scripts. [21:14:55] It's a really, really ugly pile of hacks. [21:17:28] * hedonil feels the upcoming urge to request an instance for some hacks with Hadoop [21:20:49] hedonil: I'm not sure the Analytics team has succeeded in setting up an instance :-). You might want to check with Magnus to set an appropriate frustration level :-). [21:23:16] scfc_de: thanks for the hint. I haven a open eye on the analytics guys ;) [21:27:44] Coren: if text holds nothing -- any risk of exposing that table? [21:27:54] hedonil: ok, I think I made my best effort and am still stuck: [21:28:02] I can get into my db [21:28:26] Bluma: ok. pastebin your sql query [21:29:10] but I can't import data into it. I am trying to use mysqlimport [21:29:48] hedonil:it's not a query. I'm still stuck at the import. It's not possible for me to get the data by querying. I have to import it. So I'm trying to use mysqlimport. [21:31:10] mysqlimport -h enwiki.labsdb -u --ignore-lines=1 .tsv [21:31:56] Coren: scfc_de: but prior to check Hadoop, I would like to figure out the benefits of the the new PostgreSQL feature: Fast columnar store for analytics with PostgreSQL [21:33:10] Bluma: mysqlimport is the same as 'load data infile' - special rights required [21:33:53] Bluma: so you don't have the original SQL query at hand? [21:35:08] Bluma: what are you trying to query from revision table? maybe we can just create a fresh query on the fly. [21:35:43] hedonil: I see. No, I don't. The data files were created by Aaron Halfaker over a long period of time. I asked him for the query but it wasn't just a query - it took a lot of work. It won't be possible to recreate. [21:36:16] Maybe someone with the permissions can create it for me and give me access? Or is it a very time-iintensive process? [21:36:44] Bluma: ok. then we'll transform your file into an SQL dump and then import that. Should be few lines of sed [21:37:13] whats the name of your file? location? [21:37:33] hedonil: thank you so much [21:38:58] there are 3: they're all at stat1001.wikimedia.org/public-datasets/enwiki/etc [21:39:07] !log deployment-prep Restarted logstash; it stopped processing events again at 2014-04-04T19:56:46Z [21:39:07] the 3 files that start with nov13 [21:39:09] Logged the message, Master [21:39:35] if you don't want to download them, I have them in my account [21:41:34] Bluma: can you copy them over to /shared/bluma/ ? [21:41:42] sure [21:45:55] hedonil: I can't copy them; permission denied error. My user name is bgelley. [21:46:32] Bluma: can you try again? [21:49:59] Bluma: ok I can see 2 files now [21:50:09] hang on, the copying got messed up [21:50:17] I need to fix it - give me a few minutes. sorry [21:50:23] Bluma: 'k [21:57:15] Well, I'm off! See y'all in a week or so. [21:57:19] * Coren waves. [21:58:01] Bon voyage [21:58:11] meh [21:58:59] * hedonil notes that the cat's away now :-D [22:01:55] hedonil: ok, all files in folder [22:01:57] thank you so much [22:02:12] Bluma: ok, then. [22:02:22] * hedonil goes to work on this ;) [22:50:12] What the rmytop (or mytop) equivalent on Labs? [22:53:42] hedonil: I have to leave now. I really really appreciate your help. Please let me know what happens at my userpage Bluma.Gelley, or just leave a note for me in the /shared/bluma folder. thank you again!!!