[07:15:35] Warning: There is 1 user waiting for access to tools project: Pouyan (waiting 1 minutes) [07:19:12] petan: may 20th [07:19:13] aha... [07:19:13] see uptime [07:27:33] Coren the old /tmp wasn't cleaned on exec nodes I am cleaning it now [07:27:38] it just got mounted over [07:29:01] Warning: There is 1 user waiting for access to tools project: Pouyan (waiting 14 minutes) [07:42:26] Warning: There is 1 user waiting for access to tools project: Pouyan (waiting 27 minutes) [07:45:44] petan: how often does your bot nag about shell requests? [07:45:47] every 13 minutes? [07:45:54] idk [07:46:35] if (Shell.Count > 0 || Tools.Count > 0) [07:46:36] { [07:46:38] Thread.Sleep(800000); [07:46:53] 800 seconds... [07:47:04] https://github.com/benapetr/wikimedia-bot/blob/master/plugins/request/request/Request.cs [07:48:09] it seems you're wasting threads [07:48:21] (I don't know how expensive threads are in c#) [07:48:49] how would you do that? [07:49:03] I don't think that any thread which is sleeping is expensive at all [07:50:13] I may use a job queue scheduling mini = quick tasks to do in the future [07:50:19] and do them in one thread [07:50:33] this may reduce the chance of deadlock too [07:51:42] there is no chance for deadlock I know of [07:51:56] also, I don't think your version would be faster, in fact it would be slower [07:52:12] I am putting this thread to sleep, what would you do with execution context? [07:53:55] how did that deadlock happen yesterday [07:54:02] or the day before? [07:54:34] also are irc bots cpu-intensive? [07:54:53] so being faster or slower a little is critical [07:55:48] Warning: There is 1 user waiting for access to tools project: Pouyan (waiting 41 minutes) [07:59:04] about context, maybe I want to consider putting a closure into job queue. serialization is not needed I guess [07:59:36] liangent that was entirely different code which caused the deadlock, anyway - I see no point in rewriting code which works and is fast to something that is slower, even if it work as well. what is the point? [08:00:14] btw - there is a ton of stuff that could be improved in wm-bot I just have little time for that, you are welcome to improve it [08:00:27] it's open source in the end [08:00:49] I am working on other project right now which I spend most of my day on... [08:01:42] I'm not talking about rewriting the bot using my idea. I'm just explaining I may use another approach if I designed it [08:02:33] btw I know nothing about c# - and I remember there was some talk about developer capacity in different languages at the beginning [08:04:07] ok [08:09:23] Warning: There is 1 user waiting for access to tools project: Pouyan (waiting 54 minutes) [08:22:54] Warning: There is 1 user waiting for access to tools project: Pouyan (waiting 68 minutes) [08:27:48] !tr Pouyan [08:27:48] request page: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Pouyan?action=edit talk page: https://wikitech.wikimedia.org/wiki/User_talk:Pouyan?action=edit§ion=new&preload=Template:ToolsGranted link: https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProject&action=addmember&projectname=tools [08:28:36] !rq Pouyan [08:28:37] https://wikitech.wikimedia.org/wiki/Shell_Request/Pouyan?action=edit https://wikitech.wikimedia.org/wiki/User_talk:Pouyan?action=edit§ion=new&preload=Template:ShellGranted https://wikitech.wikimedia.org/wiki/Special:UserRights/Pouyan [08:31:53] Hello I made a job for my bot but it isn't working [08:32:06] ladsgroup@tools-login:~$ qstat [08:32:07] job-ID prior name user state submit/start at queue slots ja-task-ID [08:32:09] ----------------------------------------------------------------------------------------------------------------- [08:32:10] 68196 0.26753 python2 ladsgroup Rr 05/20/2013 21:08:40 continuous@tools-exec-02.pmtpa 1 [08:32:28] How i can see the job [08:32:34] like screen -r [08:32:41] I'm new :( [08:33:48] hi [08:33:53] hi [08:34:24] the output from your job should be in your home [08:35:06] there are 2 separate files, error output and std out [08:36:35] Oh [08:36:43] I get that [08:38:18] ok [08:38:44] Thank you [08:38:54] :) [08:52:40] sorry but my internet got crazy so i didn't see what you said [08:56:05] well, I didn't say anything :) there are logs of this channel [08:56:10] !logs [08:56:11] experimental: http://bots.wmflabs.org/~wm-bot/html/%23wikimedia-labs [08:56:18] oh.. [08:56:20] !logs del [08:56:20] Unable to find the specified key in db [08:56:25] o.O [08:56:29] !logs unalias [08:56:29] Alias removed! [08:56:40] !logs is http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs/ [08:56:41] Key was added [08:59:25] I said What I'm doing is ok or not ? Amir1 petan: what I'm doing is ok? jsub -continuous python /ladsgroup/pywikipedia/wdph50.py [08:59:37] ok I didn't see that [08:59:55] I don't think it is correct I see 2 mistakes [09:00:05] 1. you are executing it as you, instead of tool account [09:00:12] I think that wasn't sent [09:00:16] 2. you are using invalid path /ladsgroup/pywikipedia/wdph50.py [09:15:58] sorry :( my internet got really crazy. I said can you help me to write a correct job? [09:22:20] Amir1: sure [09:22:29] first of all, you should create an account for it [09:24:30] my account is ladsgroup [09:24:30] i have a tool account too, [09:24:30] I made a job named local-dexbot [09:24:37] ok [09:24:42] you need to type become dexbot [09:25:59] type what? [09:26:10] become dexbot [09:26:20] ok [09:26:24] let me do that [09:26:39] then you need to upload your bot to some location accessible by that user [09:28:43] see this:https://wikitech.wikimedia.org/wiki/Special:NovaProject [09:29:07] hm? [09:29:31] I had dexbot [09:29:38] now I am dexbot [09:29:45] ok [09:29:48] ladsgroup@tools-login:~$ become dexbot [09:29:51] local-dexbot@tools-login:~$ [09:30:05] now you need to copy the bot files to some folder [09:30:13] probably to /data/project/dexbot [09:30:13] so what's next [09:30:57] there is a program mc which can be used for that [09:31:04] it's easy to use file manager ;) [09:48:24] jsub -once python /data/project/dexbot/pywikipedia/wdph50jsub -continuous python /data/project/dexbot/pywikipedia/wdph50 [09:48:24] petan: is it correct now? [09:48:44] sorry I copied incorrectly jsub -once python /data/project/dexbot/pywikipedia/wdph50 [09:48:46] I am not really sure, but I need to go now... will check later :/ [09:48:50] !toolsdocs [09:48:51] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help [11:02:37] For some reasons I have problems to access my test server from outside [11:02:38] http://math-test.instance-proxy.wmflabs.org/srv/mediawiki/skins/common/images/labs_mediawiki_logo.png [11:03:16] the wgScript path is set to $wgScriptPath = "/srv/mediawiki"; [12:05:18] http://math-test.instance-proxy.wmflabs.org/wiki/Main_Page [12:05:50] He's gone, but it's setup wrong [12:05:52] http://math-test.instance-proxy.wmflabs.org/mediawiki/skins/common/images/poweredby_mediawiki_88x31.png [12:06:00] /srv needs removing [12:08:47] Reedy: yeah, the old labs mediawiki instance puppet configs were screwy [13:07:08] Oh, how fun. I seem to have some replication up. [13:10:07] Coren up for you or up for people [13:10:24] I'm people! [13:10:47] I'm about to document it but, sneak preview: mysql --defaults-file=replica.my.cnf -h nlwiki.labsdb nlwiki_p [13:11:07] Coren you know you didn't clean /tmp on these old boxes :P [13:11:11] I mean old /tmp [13:11:14] new boxes :D [13:11:27] after reboot of exec nodes it got mounted over the old tmp [13:11:40] but no worries I cleaned it on all of them... [13:11:52] petan: Thanks. I'd have gotten to it evenbtually. [13:12:07] well, it was hard to notice, I just thought it would happen [13:12:38] I knew it wouldn't: mounting /tmp takes place before the cleanup. [13:12:53] aha [13:13:43] btw we should write somewhere all these manual parts that aren't in puppet, like change /etc/fstab and so in case we needed to create more exec nodes [13:14:03] * Coren nods. [13:14:28] I think there isn't actually so much of that.. I think only swap and this /tmp [13:14:34] There aren't too many of them left except for the grid config. [13:14:45] grid config isn't in puppet as well? [13:14:46] aha [13:14:54] I didn't set up any exec node on tools so far [13:15:36] petan: Partially only; there is no clean interface to qconf in puppet so I need to finish the gridengine module. [13:18:05] petan: Between Amsterdan and Hong Kong [13:41:17] Coren: Re replication: \o/ Although next people will want to be able to do joins between tool tables and the replicas, and probably between enwiki_p and commonswiki_p, etc. [13:43:14] anomie: joins between commons/wikidata and others may come within a day or two with luck. [13:43:30] anomie: Everyone can create tables of the form username__xxx on the replicas though. [13:43:34] To join with [13:44:06] Oh? Does everyone see those tables then? [13:44:08] create databases* [13:45:22] Hm. In theory. I need to turn off read-only now, don't I. :-) [13:45:41] Coren, help! [13:45:57] http://tools.wmflabs.org/cyberbot/waca/acc.php [13:46:42] Cyberpower678: Did you check against the common errors in the docs? [13:46:52] what docs? [13:46:56] !tooldocs [13:46:56] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help [13:47:28] In particular, check out https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Logs [13:50:08] Coren, can you make Cyberbot own it for me? [13:50:25] Cyberpower678: take [13:50:31] (From the right account, of course) [13:50:46] Is that a new command? [13:51:32] !petan-build is make -j `getconf _NPROCESSORS_ONLN` deb-pkg LOCALVERSION=-custom [13:51:32] Key was added [13:51:44] I always forget this command [13:52:06] !petan-forgot is !petan-build [13:52:06] Key was added [13:52:08] Cyberpower678: It's been there a while, just not yet documented [13:52:18] Coren, no one ever told me. [13:52:36] And now you know -- and knowing is half the battle!™ [13:52:44] :-) [13:53:16] Umm...http://tools.wmflabs.org/acc.php?action=login&nocheck=1 [13:53:35] Coren, [13:53:36] Four hundred and four! [13:53:36] The URI you have requested, http://tools.wmflabs.org/acc.php?action=login&nocheck=1, doesn't seem to actually exist. [13:53:36] You might want to looks at the list of tools to find what you were looking for, or one of the links on the sidebar to the left. [13:54:06] Yes, impressive isn't it? If the URL you give is wrong, it returns an error. :-) Check your URL. [13:54:47] AKA the login button doesn't seem to want work. That seems to be a labs error. It's the exact code on my PC and toolserver and the work just fine. [13:55:08] Look at your URI. The URI is wrong. [13:55:18] if anything is broken coren did it!™ [13:55:32] What's wrong with it? [13:55:52] Cyberpower678 you are missing the tool name? [13:56:02] tools.wmflabs.org is a root of webserver [13:56:10] FAIL. [13:56:48] Cyberpower678: Were you added to the ACC devs? ACC needs IP information to work right, and I can't give that to random people who haven't identified to the Foundation. :-) [13:57:44] Not officially yet. I'm learning the code. But I am identified to the foundation. All tool users are. [13:58:00] o_O? What's your WM username? [13:58:00] o.O [13:58:13] all tool users are identified to foundation? wtf [13:58:28] Cyberpower678 [13:59:11] It's required because we have access to sensitive information. In some case more than a CU. [13:59:27] I don't see you on http://meta.wikimedia.org/wiki/Identification_noticeboard [13:59:38] Oh, now I do. [13:59:47] Yeah, clearly I need new glasses. :-) [14:00:15] LOL. Just in case. https://meta.wikimedia.org/wiki/?diff=3713947&diffonly=yes [14:00:23] Or, alternately, I need to not search for Cyverpower [14:01:03] ACC (and UTRS for that matter) will need special handling. Special handling that will only be available at or after the Hackaton. [14:01:37] Because right now I'm teh buzies. :-) [14:01:37] :D [14:16:19] addshore when you arrive to hostel? [14:16:27] I will be there on thursday evening [14:24:14] hi, pleas enable on commons.wikimedia.beta.wmflabs.org the translate extension. Thank yoz [14:24:16] *u [14:28:21] Coren, I'm having some issues here. Cyberbot owns the files now but gets a permission denied when I chmod 664 -R /data/project/cyberbot/public_html/waca [14:28:50] What could be causing that. [14:29:23] Well, first of, the -R should go before the 664. But more importantly, you shouldn't hardcode the mode like this because you'll mess some of the sgid on directories. What you want is: [14:29:30] chmod -R g+w /data/project/cyberbot/public_html/waca [14:29:50] I.e.: add group write to all [14:30:24] Directories you want to leave sgid because that allows the group to remain correct regardless of who edits files in them. [14:31:00] chmod: cannot access `/data/project/cyberbot/public_html/waca/includes': Permission denied [14:32:39] Coren, does it have something to do with the waca folder having been set to 2664 [14:34:13] Gah! Don't chmod a directory without x! [14:34:19] Yep, that was the culprit. [14:34:20] First fix: [14:34:27] chmod a+x /data/project/cyberbot/public_html/waca [14:34:34] :-) [14:34:45] Then do the chmod -R and that'll fix everything else [14:34:59] Coren, already done. That mod changed itself when I executed take for some reason. [14:35:25] No, it didn't. take does not touches modes ever. [14:35:35] That was your chmod 664 -R /data/project/cyberbot/public_html/waca [14:35:59] A directory shouldn't be 664. Ever. :-) [14:36:40] I still can't create folders. [14:37:30] Coren, ^ [14:37:36] Can I 774 everything. [14:38:43] Wait! Don't! That's also a bad mode. [14:38:56] I'm waiting. [14:39:16] Never, ever change recursively with an octal mode: good permissions for directories are not the same as for files. [14:39:25] 774 o.O [14:39:25] Who are you trying to create a directory as, and where? [14:40:20] Hrm. You didn't fix /data/project/cyberbot/public_html/waca right. chmod a+x /data/project/cyberbot/public_html/waca [14:40:42] There you go. [14:40:50] Coren, Done [14:41:04] * Cyberpower678 will get it right eventually. :p [14:41:21] Now your permissions seem okay. [14:56:34] petan: hello are you there? [15:10:21] I want to make two jobs (via jsub) but this doesn't work jsub -once python /data/..... [15:10:42] does anybody know that? [15:12:52] Warning: There is 1 user waiting for shell: Aandrei (waiting 0 minutes) [15:19:44] Amir1: what is the problem? [15:19:54] i.e. what doesn't work? [15:20:06] does the second job not start, do you get an error, ....? [15:20:34] Coren, can you add stwalkerster to tools? [15:21:25] Cyberpower678: I don't see his request? [15:21:29] PM [15:21:34] !toolsrequest is https://wikitech.wikimedia.org/wiki/Special:FormEdit/Tools_Access_Request [15:21:34] You are not authorized to perform this, sorry [15:21:37] :-( [15:21:50] I need to let him access local-cyberbot [15:22:05] To debug some stuff. [15:22:06] Amir1: If you didn't specify a name for you job, it takes the name of the program you ran. In your case, 'python'. [15:22:16] Cyberpower678: He needs to request it himself. [15:22:26] !toolsrequest is https://wikitech.wikimedia.org/wiki/Special:FormEdit/Tools_Access_Request [15:22:26] Key was added [15:22:54] Amir1: So since your other job is /also/ named python, -once (correctly) prevents it from starting. [15:23:05] Amir1: Name your jobs with the -N option, and that'll be fixed. :-) [15:24:43] What's the DB word? [15:25:02] greenrosetta, Bare minimum will be ready by Amsterdam. [15:25:54] whats bare minimum entail? en-wiki? [15:26:00] yes. [15:26:07] No archive yet I believe. [15:26:20] sorry i was away [15:27:05] Warning: There is 1 user waiting for access to tools project: Stwalkerster (waiting 0 minutes) [15:27:23] Coren, he's requesting it now. [15:27:30] Coren: Thank you but is ok doing this way: jsub -once -N:albumbox ....? [15:27:51] You want a space after -N, not a colon, but yeah. [15:28:00] jsub -once -N albumbox ... [15:28:07] oh [15:28:09] I get that [15:28:44] yes [15:28:47] It worked [15:28:49] Cyberpower678: {{done}} [15:28:51] thank you [15:28:58] I think we should update the manual [15:29:02] Amir1: That's what I'm here for. [15:29:23] Redactatron... heheh [15:29:25] Amir1: I thought it was pretty clear about job names, but if you find a way to make it clearer, go ahead: it's a wiki. :-) [15:29:35] Coren, thank you [15:29:57] It's strange [15:30:05] sounds like a character from a 70's exploitation flick [15:30:41] I thought that was like this jsub -once jobname php /data/project/tool/task/execute.php [15:30:45] (there wasn't -N) [15:30:55] so my bad [15:30:58] sorry [15:31:04] and thank you again [15:40:11] Successfuly added .*@wirenat2\.strw\.leidenuniv\.nl [15:40:11] @trustadd .*@wirenat2\.strw\.leidenuniv\.nl trusted [15:40:17] valhallasw try again [15:46:41] Oh, how interesting! There has been an update to https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Database_access [15:53:39] petan: heh, except that I switch domains often. I should probably just get a mask [15:54:11] indeed [15:57:58] petan: ^ fixed :-) [15:58:22] Successfuly added .*@unaffiliated/valhallasw [15:58:22] @trustadd .*@unaffiliated/valhallasw trusted [15:58:51] thanks :-) [16:02:38] kernel doesn't maintain any good backward compatibility.... [16:02:49] basically every new release breaks half of my drivers :/ [16:03:20] I wish they would actually deprecate variables and functions before they remove them from code [16:18:15] Krenair, are you able to access nova-precise2? [16:18:27] Um… via ssh, I mean? The web interface is working fine [16:18:57] no Permission denied (publickey). [16:21:16] Mind if I reboot? [16:23:50] Krenair, mind if I reboot? [16:24:27] andrewbogott, don't mind at all, I'm obviously not logged in at the moment [16:24:35] Ah, good point [16:40:21] Krenair, the disk is full… which I don't know how to fix without a login :( [16:54:09] Warning: There is 1 user waiting for shell: Moshe742 (waiting 0 minutes) [16:59:42] Funny, given how often people clamoured for replication I'd have expected half of -login's ps to be full of mysql. :-) [17:00:11] replication is public now? [17:01:57] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Database_access [17:03:48] Coren, I didn't see anything about this on the mailing list... [17:04:27] Krenair: Because I don't want to open the floodgates quite yet; "officially" that's for the Hackaton. [17:04:38] Some use until then would be good, though. [17:04:53] Coren, ... then why are you surprised that most people aren't aware of it? [17:05:10] I've been saying things about it on-channel all morning. :-) [17:05:30] I only got home at like 16:30 BST [17:05:38] Heh. Well, now you know! Have at it! [17:05:40] :-P [17:05:48] Do you need tools access to connect to it? [17:06:01] Krenair: Yes. It is only accessible from the tools project atm. [17:06:10] Can I be added to that project please? [17:06:19] (Well, also by deployment-prep but that's a side-effect) :-) [17:06:27] I'm in deployment-prep [17:06:46] You're in tools now. [17:06:57] thanks [17:07:08] The way to access them from deployment-prep wouldn't be the same anyways; the hacks to do it cleanly are, atm, only on tools [17:07:39] Warning: There is 1 user waiting for shell: Moshe742 (waiting 13 minutes) [17:09:19] Krenair: Hm. Need a reboot of the DBs. (See why I didn't widely publicize it?) [17:09:30] I understand that [17:09:31] Krenair: Downtime: ~2-3 minutes in 1-2 minutes [17:09:51] However, you seemed surprised that people were seemingly unaware of it [17:10:03] No, I wasn't really. I was dropping hints. :-) [17:10:26] And, indeed, it worked for you didn't it? :-) [17:11:02] Coren what [17:11:07] there is mysql you say? :D [17:11:10] anyway [17:11:41] what about the private dbs? are they still on that slow box? or we are getting dedicated one [17:13:08] petan: RTFWP. :-) [17:13:15] Coren, which instance should I connect to in tools to use this? [17:13:18] i'm going to keep tools-db up, provisionally. [17:13:42] Krenair: tools-login is the normal entry point for all things tools. Has a public IP too, since it's a bastion (tools-login.wmflabs.org) [17:13:44] Krenair -login? :P [17:13:57] hm [17:14:07] I was wondering if I shouldn't be using that to connect to mysql [17:14:19] since I don't have a ~/replica.my.cnf file there [17:14:41] Krenair: It takes a moment after creation for the file to appear; and the DB is not up. :-) [17:14:43] Coren what about the creating db [17:14:45] part [17:14:55] what is that for [17:14:57] petan: Read the fine Wiki Page, petan. :-) [17:15:03] I do [17:15:07] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Database_access [17:15:08] that is what confuse me [17:15:16] Creating databases [17:15:17] Users can create databases with names starting with username__ (where username is your credentials' user, which can be found in your ~/replica.my.cnf followed by two underscores), and have all privileges on it (with grant options). [17:15:35] I thought private databases are supposed to be on separate server [17:15:38] like tools-db [17:16:17] ... you couldn't join with the replica databases if that's the only place you could make them. [17:16:25] aha [17:16:31] Those databases can be created on the replica servers (as above), or on a database local to the tools project: tools-db. [17:16:37] Next paragraph. [17:16:52] so these are permanent databases? [17:17:13] * Coren notes Asher is finishing up maintenance on said DBs and they will be back up shortly. [17:18:41] Define "permanent"? They don't go away automatically if that's what you're asking. [17:20:06] yes :D [17:21:50] Krenair +1 I don t think how people would be spamming the labs with db requests if they dont know it :) you need to send announcement :D [17:22:00] s1 (enwiki) is back [17:22:05] So any minute now. [17:22:36] Coren that cnf file differs per user? [17:22:49] petan: Of course it does. [17:23:06] aha you said people can create db too [17:23:12] if it was read only it wouldnt need to [17:23:40] petan: You'd still want to be able to see who does what queries to know who to beat up when they make silly full scan cross-joins. :-) [17:23:50] ok [17:24:36] Coren what a creepy username [17:24:38] :D [17:25:41] DBs are back up. [17:25:50] Krenair: your creds should have arrived. [17:57:19] Coren: creds? en-wiki? [17:57:55] greenrosetta: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Database_access [17:58:37] * brion tossed up a quick project request at https://wikitech.wikimedia.org/wiki/New_Project_Request/SVG-dashboard \o/ [18:00:04] error: commlib error: access denied (client IP resolved to host name "bots-gs.pmtpa.wmflabs". This is not identical to clients host name "bots-gs") [18:00:05] error: unable to send message to qmaster using port 6444 on host "bots-gs.pmtpa.wmflabs": got send error [18:00:07] What does that mean? [18:01:12] Pyfisch: I'm not sure where it came from, but apparently the program you are using requires you using fully qualified names that match the reverse. If you give it the whole name, it should work. [18:01:29] brion: Not sure that needs a whole project. Lemme poke Ryan. [18:01:43] heh [18:01:45] hmmm [18:01:50] i'll take a vhost, just need storage :) [18:01:52] I wish we had a project for mediawiki that was easy to use [18:02:07] so that we didn't need to create a project for each one [18:02:11] let me look at the project list really quick [18:02:18] whee [18:02:19] Coren: I just try to execute qstat on bots-gs [18:03:44] Pyfisch: Oh, I'm not quite familiar with the setup of the gridengine on bots; you'll have to ask Petan. At first glance, he's configured some of the access control lists with just hostnames instead of FQDN and it breaks from some hosts. [18:04:01] petan: ^^ [18:05:00] Ryan_Lane: We should make a project where one can just deploy a WM install automatically per service group. Shouldn't be too hard to swing. [18:05:10] ah. true [18:05:21] service groups are handy [18:05:57] With a suitable apache config with nice rewrite rules, that'd be the idea spot for testing codebases. [18:06:10] indeed. that's what I wanted at first [18:06:18] never got a chance to work on it [18:07:07] for now I'll just create an svg project [18:07:45] brion: I created an svg project and added you as projectadmin [18:07:56] whee [18:07:58] brion: see: https://wikitech.wikimedia.org/wiki/Help:Single_Node_MediaWiki [18:08:05] it'll make the mediawiki install part quicker :) [18:09:55] "(In the homedir pattern, %u will be replaced by the username and %p with "local-".)" [18:09:58] * brion hmms :) [18:10:05] Coren: good job [18:11:15] brion: That's because usernames are local-$foo but it's useful to be able to set homes /data/project/$foo instead. [18:11:40] Coren: i was concerned more with the literal " " :) [18:11:42] greenrosetta: Much of the applause should be directed in binasher's direction. :-) [18:14:59] Well, whoever it deserves: Thanks! [18:15:55] Coren, are there any other db's available right now? dewiki_p for example? [18:16:06] * brion carefully follows directions [18:16:34] * brion waits for instance to build so he doesn't explode it by mistake :D [18:16:36] greenrosetta: The available list is on that page I linked. There's about 20 available now, including most of the bigger ones. [18:17:03] Pyfisch hi [18:17:12] brion: Lucky you. Not so very long ago the instance build process was painfully long. Now the image comes out of the box mostly prebuilt. :-) [18:17:15] Pyfisch why are u running anything on bots-gs? [18:17:19] yayy [18:17:40] not seeing a list of db's on https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Database_access [18:17:41] petan: because I was to lazy to move it to bots-login [18:17:53] nbm [18:17:57] nvm.. found it [18:18:12] Pyfisch what exactly do you need to move, beside cron? [18:18:40] petan: only some config files for pwb [18:18:54] Pyfisch did you store the config file locally? or shared [18:18:57] I believe shared [18:19:03] so you don really need to move anything [18:19:11] just cron [18:19:37] no config was is my home directory [18:19:42] instance active! [18:19:47] home directory is shared too [18:20:22] * petan hides before brion and his new instance explode [18:21:38] ok, now to wait for puppet [18:21:42] * brion wanders off in the meantime [18:21:55] petan thx [18:22:01] Pyfisch works? [18:35:59] why isn't this query returning results? It works when I remove the lower and capitialize "Fubar" [18:36:02] select distinct(rev_user_text) FROM revision LEFT JOIN page on revision.rev_page = page.page_id WHERE lower(page_title) = 'fubar'; [18:43:35] hmmm.. it appears the lower function doesnt work [18:44:07] greenrosetta: page_title is (presumably) a binary string, so lower() and upper() won't work. Try lower(convert(page_title using utf8)). [18:45:15] greenrosetta: Your query is going to be a LOT faster if you use revision_userindex too. I've yet to document that, but tables with revision deletion tend to have an alternate with rows elided but indices preserved. [18:46:01] Also, does MYSQL support ILIKE? [18:46:20] Coren, is it possible to USE INDEX on labs? [18:46:21] page_title ILIKE 'fubar' is probably going to be much faster too (using index) [18:46:33] anomie: that worked [18:46:45] Coren: im searching on article name, so I don't see how I can use the index [18:46:57] iIRC, there's an index on page_title [18:47:30] Platonides: Yes, but generally pointless to do so. [18:47:56] I mean, it's not possible in toolserver (due to the views) [18:48:34] Coren: I don't think MySQL has ILIKE. Instead, LIKE is case-insensitive unless one of your strings is a binary string. [18:49:04] is it normal that some libraries present on bots-gs are not anymore installed in bots-login ? [18:49:06] Or maybe just as long as both have a "_cs" collation. [18:49:13] anomie: I just checked and you're right. I'm a prosgres dude myself. :-) [18:49:29] Toto_Azero|away: bots-gs is deprecated. [18:49:55] yes I know :) that's why I was wondering if it is normal that some libraries are missing on bots-login [18:50:34] Coren: Me too. Although MariaDB doesn't suffer from one of my main objections to MySQL. [18:50:52] (which is... ?) [18:51:19] (for instance httplib2) [18:51:51] anomie: Well, my oldest objection to mysql hasn't been valid for quite some time, but for the longest time it confused the hell out of me why mysql was used at all, let alone that widely: how the hell can a "relational" database not support foreign keys -- that's what relational /means/! :-) [18:52:22] anyone want to see if this query can be optimized? http://bpaste.net/show/slsebeXhDfTFupUarsN8/ [18:52:47] I want to get all editors that have edited "fubar" (any case) [18:53:16] Coren: That's another one. Also, MyISAM tables don't even support transactions. Ugh! [18:54:00] greenrosetta: is it faster to get a list of all the page titles that actually match, and then eliminate the lower() call from that query? [18:54:18] Coren: Oh no, no EXPLAIN on the replicas? [18:54:25] also, what is the reason for the left join, if you don't want null rows anyway? [18:55:54] carl-cbm: It would be fastest if you get the list of all *page_id* whose title match. [18:56:39] Coren: yes, exactly [18:56:51] carl-cbm: The problem is that your case folding requires a full table scan. [18:56:56] carl-cbm: I dunno.. I wrote this a while ago... I thinki someone else hlped me with this query [18:57:54] Coren: which is faster, a full scan of the page table (actually, just the pages that start with 'F' or 'f'), or a scan of the entire revision table? I put my bet on the page table [18:58:03] select distinct(rev_user_text) FROM revision_userindex where rev_page in (select distinct page_id from page where lower(convert(page_title using utf8))='fubar'); -- is pretty much optimal, most of the query is spent in the subquery once just to get the page ids. [18:58:35] carl-cbm: You're not doing a full scan of the revision table since rev_page is indexed. [18:58:45] thats actually slower [18:59:40] ... takes about two seconds shorter for me. [18:59:51] this is the fastest: select distinct(rev_user_text) FROM revision LEFT JOIN page on revision.rev_page = page.page_id where convert(page_title using utf8)='fubar'; [18:59:57] revision_userindex is not a table [19:00:26] where are you running that? :S [19:00:31] Platonides: None of them are. They're all views. [19:01:01] so is there now access to the DB replicas on the tools lab? [19:03:10] greenrosetta: select distinct(rev_user_text) FROM revision_userindex where rev_page=11177; -- is near instant. Getting the list of page is the only expensive thing. [19:03:15] carl-cbm: There is. [19:03:30] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Database_access [19:03:31] Thanks - I managed to find the server name now [19:03:40] greenrosetta: There won't be streaming from the Amsterdam hackathon but there will be videorecording of some talks to be uploaded to Commons [19:03:53] sumanah: ty [19:04:19] * Damianz pokes Coren [19:04:38] * Coren pokes Damianz back! Poink! [19:05:02] Coren: Any chance of mysql replication this evening? :) [19:05:09] * Coren chuckles. [19:05:13] Coren: this runs for me 6 seconds faster than your original query: select distinct(rev_user_text) FROM revision_userindex where rev_page in (select distinct page_id from page where substring(page_title, 1,1) = 'F' and lower(convert(page_title using utf8))='fubar'); [19:05:27] Damianz: I would say that the probability is exactly 1. Since it's working. [19:05:32] :D [19:05:38] Can you add me to tools? [19:06:00] carl-cbm: Aha. Yes, since you filter a great part of the table first (and that does use the index) [19:06:07] yes, exactly [19:06:10] * Damianz scrolls up to read today [19:06:48] Damianz: What's your wikitech username? [19:06:53] greenrosetta: the point is to avoid the lower() call as much as possible. And the first character of every page title is uppercase [19:06:59] Coren: DamianZaremba [19:07:04] 53 messages... damn need to login more [19:07:24] Ooooh those icons are sexy [19:07:27] carl-cbm: but there are 5 articles with "fubar" [19:07:30] various cases [19:07:31] Successfully added DamianZaremba to tools. [19:07:34] Tyvm [19:08:07] greenrosetta: Actually, avoid *functions* as much as possible since they can't use indices. [19:08:28] greenrosetta: sure, but in the database the first character of every one of them is uppercase, all page titles are normalized that way in the database. The second part of the 'and' takes care of the rest of the title [19:08:31] im kinda stuck using at least convert here [19:08:50] greenrosetta: Hence /as much as possible/ :-) [19:10:38] greenrosetta: are you interested in all pages, or just in articles? [19:10:59] pages, because I'm letting the user search by namespace [19:11:18] so you know the namespace ahead of time? Because if you do, you can make it even faster like this [19:11:22] select distinct(rev_user_text) FROM revision_userindex where rev_page in (select distinct page_id from page where page_namespace = 0 and substring(page_title, 1,1) = 'F' and lower(convert(page_title using utf8))='fubar'); [19:11:36] that's a 4 second query for me [19:12:10] what does the substring part do? [19:12:19] ah, makes sure it is upper case [19:12:34] it makes sure the first letter is F before calling the lower() function [19:13:21] so that every other page that starts with a different letter does not have to call lower(). and since the substring is at the front of the title, the server should use the index for it if I understand things correctly [19:18:20] select distinct(rev_user_text) FROM revision_userindex where rev_page in (select distinct page_id from page where page_namespace = 0 and substring(page_title, 1,1) = 'F' and lower(convert(page_title using utf8))='fubar'); [19:18:24] select distinct(rev_user_text) FROM revision_userindex where rev_page in (select distinct page_id from page where page_namespace = 0 and substring(page_title, 1,1) = 'F' and lower(convert(page_title using utf8))='fubar'); [19:18:28] select distinct(rev_user_text) FROM revision_userindex where rev_page in (select distinct page_id from page where page_namespace = 0 and substring(page_title, 1,1) = 'F' and lower(convert(page_title using utf8))='fubar'); [19:18:32] select distinct(rev_user_text) FROM revision_userindex where rev_page in (select distinct page_id from page where page_namespace = 0 and substring(page_title, 1,1) = 'F' and lower(convert(page_title using utf8))='fubar'); [19:18:36] select distinct(rev_user_text) FROM revision_userindex where rev_page in (select distinct page_id from page where page_namespace = 0 and substring(page_title, 1,1) = 'F' and lower(convert(page_title using utf8))='fubar'); [19:18:40] select distinct(rev_user_text) FROM revision_userindex where rev_page in (select distinct page_id from page where page_namespace = 0 and substring(page_title, 1,1) = 'F' and lower(convert(page_title using utf8))='fubar'); [19:18:43] ew [19:18:44] select distinct(rev_user_text) FROM revision_userindex where rev_page in (select distinct page_id from page where page_namespace = 0 and substring(page_title, 1,1) = 'F' and lower(convert(page_title using utf8))='fubar'); [19:18:48] Huh - sorry :( [19:19:00] select distinct(rev_user_text) FROM revision_userindex where rev_page in (select distinct page_id from page where page_namespace = 0 and page_title like 'F%' and lower(convert(page_title using utf8))='fubar'); [19:19:06] * Damianz wonders how long before Coren stabs him [19:19:07] That last ^^ is near-instant [19:19:25] substring is fast, but doesn't use the index. LIKE does. [19:19:44] that only returns 889 rows however [19:20:05] ah wait [19:20:07] So does the one with substring. [19:20:18] You had included a non-article before. :-) [19:20:27] * petan eats substrings [19:20:44] no namespace its about 10 secs [19:20:47] still the best so far [19:21:04] * Damianz crosses fingers and hits start [19:21:23] Damianz start button? windows :o [19:21:24] cute [19:21:38] supervisord [19:21:48] :)) [19:22:12] Coren: plan on supporting custom domains on public_html by any chance? [19:22:23] * Damianz thinks he might get around to moving some stuff over to tools off bots [19:23:51] Damianz: Probably, at some point in the mid-term. [19:24:10] :) [19:24:17] First bits to move would be the backend anyway [19:24:23] * Damianz thinks he broke this already... damn [19:24:44] Does mariadb hate subselects? [19:24:53] I could probably make this sql less evil actually [19:25:39] Damianz: Not that I know of. Mariadb should be a strict superset of mysql. [19:26:55] query has been doing for 242 seconds [19:27:02] runs realy fast on ts [19:27:27] http://toolserver.org/~cobi/cb.php?user=86.0.162.74&ns=0&title=Louis+Cole×tamp=1369164148 [19:27:30] vs [19:27:33] http://tools.wmflabs.org/cluebot/cb.php?user=86.0.162.74&ns=0&title=Louis+Cole×tamp=1369164148 [19:27:37] like still running [19:27:40] vs instant [19:27:43] Hm. There are caveats about indexes on some columns you might need to use an alternate view. Can I see your query? [19:28:09] Oh, doing matches against usernames? [19:28:14] use revision_userindex [19:28:39] revision has the suprpessed rows with possible NULLs, and that renders the index nonoperative. [19:28:56] I.e.: full table scan on revision. :-) [19:29:12] http://pastebin.com/t8BGCwSg is the query before it gets parambinded [19:29:14] Damianz you broke wikipedia :) [19:29:16] :( [19:29:17] it's ugly... [19:29:24] but it's faster than pulling that from the api [19:29:27] I am trying to send a message and it is so slow [19:29:53] petan: Dial up [19:30:22] I did just break cluebot though lol [19:30:23] Damianz: Try to use revision_userindex [19:30:26] * Damianz pauses it to revise the query [19:30:45] That should give you about five orders of magnitude of improvement. :-) [19:31:10] Coren: That a table/column? I didn't write this specific query so I'm not 100% up on the db layout... I just know the ann barfs without it [19:31:19] * Damianz gets pizza out the oven and sits down [19:31:20] It's a table. [19:31:25] Cool [19:31:52] The rule is, whenever you match against rev_user or rev_user_text, you should use revision_userindex [19:32:24] revision has all the rows, but some of them are conditionally nulled which destroys index use. [19:32:54] revision_userindex doesn't have the rows where the user has been suprpessed or revdel'ed, but has the indices usable. [19:33:17] makes sense [19:33:50] Could totally run these as seperate sql querys too =\ [19:56:30] Are we having fun yet? [20:00:31] Coren: it looks to me like cross-db joins between enwiki_p and a user database on the enwiki.labsdb server are working. is there anything unusual that I need to be aware for user databases? [20:01:10] carl-cbm: Not that I know of. The databases are yours so you should know what you indexed or not. :-) [20:02:15] Coren: Hating sql :P [20:03:13] indeed. just trying to avoid falling into any traps. I would like to migrate the WP 1.0 database over to labs soon. But I first I have to get someone to dump it from the toolserver for me [20:03:44] Warning: There is 1 user waiting for shell: Heziresheff (waiting 0 minutes) [20:03:59] bleh [20:14:10] ERROR 2005 (HY000): Unknown MySQL server host 'enwiki.labsdb' (0) [20:14:20] Oh I'm running from the wrong instance. Oops. [20:17:13] Warning: There is 1 user waiting for shell: Heziresheff (waiting 13 minutes) [20:18:02] hmm [20:26:23] Coren should I distribute hosts to all instances? [20:26:47] we could eventually either put it into puppet (bad idea) or create some sync script (bad as well, but... who cares) [20:27:06] !log tools petrb: uploaded hosts to -dev [20:27:08] Logged the message, Master [20:27:45] lol [20:27:50] it was already there I see [20:28:12] never mind [20:28:25] or just put it in dns :P [20:29:25] It'll go in DNS in a bit, there's more backend stuff to handle before that. [20:29:47] Krenair: That should work from all tools exec environments. [20:30:25] Coren, yeah it was deployment-bastion in deployment-prep [20:30:36] which is why it didn't work :) [20:30:36] Coren: re revision_userindex - does it not store the first rev of a page? [20:30:46] Warning: There are 2 users waiting for shell, displaying last 2: Heziresheff (waiting 27 minutes) Zalak (waiting 6 minutes) [20:31:26] Damianz: It should; it's just a view into the revision table with WITH rev_deleted=0 [20:32:55] Coren: select rev_user_text, rev_timestamp from revision_userindex where rev_page = 'Louis_Cole' order by rev_text_id limit 1; I expect Yurishui and 1335183353, not what I get =\ [20:33:44] Hm. Interesting. lemme check. [20:33:48] in enwiki? [20:34:07] actually something's wrong there... that user never edited that page according to the history =\ [20:34:11] yeah enwiki [20:34:36] Db says Morty, wiki says Morty never touched the page... lol [20:34:40] Coren what is tools-mail for? how does it work [20:34:42] Hate un-picking sql [20:35:16] petan: It's a mail relay, unsurprisingly enough. It receives mail, then either delivers it locally, or transmits it to the appropriate MX. :-) [20:35:28] ah [20:35:38] I thought that each instance handle mail locally [20:35:46] actaully all the users returned by that have never touched the page [20:36:05] Ryan_Lane, do you have the necessary superpowers to empty /a on nova-precise2 even though it won't accept ssh-logins? [20:36:20] yep [20:36:34] I think you do too ;) [20:36:45] done [20:36:47] btw is that webserver 02 useful, Coren? [20:36:59] Coren: Nvm... I've got another query that's fast enough [20:37:00] SELECT `rev_timestamp`, `rev_user_text` FROM `page` JOIN `revision` ON `rev_page` = `page_id` WHERE `page_namespace` = '0' AND `page_title` = 'Louis_Cole' ORDER BY `rev_id` LIMIT 1 [20:37:01] I mean are we able to switch to latest version? or not [20:37:14] petan: It will be soon. [20:37:20] k [20:37:25] Ryan_Lane, what did you do? [20:37:36] Damianz: Does that give you the expected revisions? [20:37:47] Yeah that gives me the right timestamp and user [20:37:52] Not sure why the other doesn't [20:37:53] hm… still can't log in :( [20:38:00] andrewbogott: I ssh'd as root [20:38:19] Oh… I tried that and was refused. But maybe I didn't forward my key or something. [20:38:28] * Ryan_Lane nods [20:38:29] hm [20:38:30] Anyway… probably after a puppet run the system will recover [20:38:39] /home won't mount for gluster [20:39:13] The query with "rev_page = 'Louis_Cole'" earlier was giving odd results was because it was converting 'Louis_Cole' to an integer 0. [20:39:22] * Coren laughs. [20:39:33] Damianz: want to know why the other one didn't work? [20:39:39] Oh. Anomie ninja'd me. [20:39:44] andrewbogott: it'll work now [20:39:56] Rofl [20:39:57] it's absurd that gluster has to be able to write to its log file to mount [20:39:57] Ryan_Lane, can any project admin do that? [20:39:59] That would explain why [20:40:02] Krenair: nope [20:40:08] just labs cloudadmins? [20:40:09] Krenair: only people with super powers [20:40:13] Coren: Another thing on the list of "why MySQL sucks" [20:44:15] Warning: There are 2 users waiting for shell, displaying last 2: Heziresheff (waiting 40 minutes) Zalak (waiting 20 minutes) [20:44:52] !rq Zalak [20:44:52] https://wikitech.wikimedia.org/wiki/Shell_Request/Zalak?action=edit https://wikitech.wikimedia.org/wiki/User_talk:Zalak?action=edit§ion=new&preload=Template:ShellGranted https://wikitech.wikimedia.org/wiki/Special:UserRights/Zalak [20:45:30] !rq Heziresheff [20:45:30] https://wikitech.wikimedia.org/wiki/Shell_Request/Heziresheff?action=edit https://wikitech.wikimedia.org/wiki/User_talk:Heziresheff?action=edit§ion=new&preload=Template:ShellGranted https://wikitech.wikimedia.org/wiki/Special:UserRights/Heziresheff [20:57:44] Warning: There is 1 user waiting for shell: Shayt (waiting 2 minutes) [20:59:04] Coren: With the (I assume :-)) increased load on the database servers, does the replication lag increase? [21:03:54] Where can I find statuses? (As in, why is ClueBotNG down?) [21:04:28] scfc_de: probably will. yes [21:04:52] a930913: does it run in bots or has it been moved to tools? [21:04:55] a930913: http://tools.wmflabs.org/?status [21:05:08] that'll show you status for tools running in the tools project [21:06:05] it's in the list of bots: http://tools.wmflabs.org/?list [21:07:18] Damianz: ^^ [21:07:50] scfc_de: It should be fairly minor. [21:08:12] scfc_de: The heavy lifting is done by the prelabs replica, not the masters. [21:08:21] a930913: Because I'm re-writing the api script to migrate the last bit from toolserver to tools [21:08:34] Read as - it broke, fixing it, be back soon [21:08:54] Greetings Earthilings. [21:08:55] *Earthling [21:10:07] Coren: Data! Data! Data! :-) More than single-digit at the moment? [21:10:18] Damianz: But I was using it :'( [21:10:33] Using it for? [21:10:58] I could start it up pointing at ts...but I'm down to 1 bug [21:11:17] Warning: There is 1 user waiting for shell: Shayt (waiting 15 minutes) [21:11:22] Damianz: A few things it turns out, now that it's down :p [21:11:29] Damianz: Can I help debug? [21:11:37] You like sql? [21:11:50] Damianz: What flavour? [21:13:00] Bog standard sql [21:13:28] oh derp [21:13:31] actually typo [21:13:32] 1min [21:13:42] Damianz: Sure, fahadsadah can tell you about some of the monstrosities I've made :D [21:14:35] Damianz: Switch to cluenet? [21:14:47] Think I've got it, just confirming then I'll turn it on and see if the server falls over [21:16:47] somoene help me optimize my query? this is taking forever [21:16:48] http://bpaste.net/show/bLFeDjLaLXEeP9SxNUPD/ [21:20:46] greenrosetta: Perhaps try filter before the join? [21:21:08] And COUNT(1) though that should be optimised out I think. [21:22:24] not sure what you mean aobout filter before join [21:23:24] greenrosetta: You might also want to follow Coren's tip and use revision_userindex instead. [21:23:25] greenrosetta: This one looks most apt for you. http://www.mysqlperformanceblog.com/2007/04/06/using-delayed-join-to-optimize-count-and-limit-queries/ [21:24:45] Warning: There is 1 user waiting for shell: Shayt (waiting 29 minutes) [21:29:39] Damianz: I can't see any mainspace edits. Has ClueBot NG suffered brain damage? [21:31:32] Just giving it some therapy [21:34:38] Toolserver is sooo lagged even doing queries by hand to validate is impssoible [21:35:07] * a930913 blames the mortals. [21:36:01] Damianz: How does CBNG get its diffs? [21:36:16] The actual diff? API [21:36:44] Damianz: Yeah, so it still requests each edit, like back in the old days? [21:37:13] Every incoming requests causes a request to the api and a request to a helper script that pulls some meta data from the db [21:37:55] Damianz: So there is an API request for every edit, right? [21:38:15] If it passes the first checks (is it whitelisted, is it outside of main). [21:38:19] Warning: There is 1 user waiting for shell: Shayt (waiting 42 minutes) [21:40:19] I think that was the thing that impressed me most, when CBNG first came out :p I had tried to do that and it exploded in my face. [21:41:48] The code could be a lot better - I'd probably use a queue and throw it out in a worker model... the fork per edit causes random usage and sometimes it kills its self [21:42:16] Coren: DUDE [21:42:29] Damianz: Whazza? [21:42:33] Damianz: With generators, it's now trivial to download every edit. [21:42:37] What's the internal ip for the web proxy? [21:42:40] Our nat is retarded [21:43:33] Wait, missing teh context. 10.4.1.89 from the inside. But why? [21:44:04] tools. resolves to ext ip - you can't hit external ips from internal [21:44:28] echo '10.4.1.89 tools.wmflabs.org' >> /etc/hosts # and the bot comes to life [21:44:53] Guys... I still cant get this to work.... Coren, mind taking a peek? http://bpaste.net/show/bLFeDjLaLXEeP9SxNUPD/ [21:45:18] a930913: Should be making mainspace edits again [21:45:40] Damianz: \o/ [21:45:50] greenrosetta: Do you even know what "LEFT JOIN" means?! :-) [21:46:10] https://github.com/DamianZaremba/cluebotng/commit/a209e54c875a314f59013f8b3048846a2871fc4f That was painful [21:46:19] Damianz: Are you trying to hit the webserver from a tool? [21:46:27] Coren: Yes [21:46:30] Damianz: Then you can user tools-webproxy as the hostname [21:46:34] use* [21:46:43] Hmm yeah I could do that [21:46:43] its been a while, since I use simple queries. IIRC it means take all the rows from the left table [21:47:20] When I get around to moving the bot to tools fully, I might put this stuff directly back in it - but it's allways used http since it ran remotly rather than on ts [21:47:57] Coren: I hope your servers like queries =\ [21:47:59] Damianz: I can add to the hosts file, but that's a bit unmaintainable. Can you work it with the inner name or do you need it? [21:48:06] Damianz: They exist for it. :-) [21:48:15] I can use the inner name [21:48:45] greenrosetta: Connect to enwiki.labsdb, "SELECT * FROM revision LIMIT 1;" and look at the rev_user and rev_user_text columns for a start. [21:49:13] It's like less than 4 queries a second so should be fine *whistles* [21:49:21] greenrosetta: I'm in the middle of working the other replicas up atm; I can take a look later maybe? :-) [21:51:01] The replicas actually live on SSD storage, they should be fairly swift and scale a bit better than with spinning bits of metal. [21:51:28] Now I fixed the queries a little, it's pretty ruddy fast [21:51:33] Even running like 5 rather than 1 now [21:51:44] Warning: There is 1 user waiting for shell: Shayt (waiting 56 minutes) [21:53:49] Damianz: Are you using revision, ipblocks or logging [21:54:20] Damianz: All three of those tables have alternatives with more indexing when you don't care about some revdel fields. [21:55:50] Damianz: Specifically, revision_userindex and logging_userindex if you have any where clauses on _user or _user_text, and ipblocks_ipindex if you have where clauses on ips or _user and don't care about autoblocks. [21:55:56] yes [21:56:03] https://github.com/DamianZaremba/cluebotng/blob/master/bot/toolserver/cb.php < [21:56:09] mostly for pulling creation time [21:56:46] Ah, and you've already used _userindex. How smooth. :-) [21:57:08] The ones using joins onto revision are pretty fast right now tbh [21:57:20] I tried using some of the *index tables, but it gave funny stuff back [21:58:13] "funny stuff"? Do you have an example, because unless you're looking at deleted revisions the results should be the same. [21:59:05] But yeah, you can user revision in the joins since you're not trying to match against rev_user or rev_user_text (that aren't indexed in that one) [21:59:33] Coren: Are federated tables already around to play with? [21:59:49] Like the one that was convering to 0 and returning incorrect users [22:00:40] Deleted revs retain comments right? So there's 2 I can't get rid of [22:01:33] * Damianz notes that these servers seem far less lagged than ts [22:03:35] scfc_de: No. Probably will set them up during the hackaton. [22:04:30] Damianz: No, the revision_userindex only removes the ones where the /username/ has been supressed. Revdel'ed ec just show up as null. [22:04:38] that quyery took 5 mins to run :( [22:04:52] there's edits where the username is removed? never knew that [22:05:13] Warning: There is 1 user waiting for shell: Shayt (waiting 69 minutes) [22:05:32] I'll come back and do some sql improvement after I screw about with ltib to get it to compile my rpi image [22:06:52] Damianz: Well, it's running with reasonable performance now? [22:07:36] Yeah - script has a 10second timeout and doing a watch on mysql process list with a .5 second refresh doesn't keep the query up long enough to see the time... [22:08:04] It might be worse performance for a user with thousands of revisions I suppose, but I'm sure I'll find that out soon [22:09:26] greenrosetta: In your query the 'rev_user = ' bit was causing my speed issues, mostly [22:09:35] Join against the index table in the middle [22:09:40] im gonna try and use the id first [22:10:44] Coren: Since you're writing the access logs into home dirs, these have logrotate on them, right? [22:11:12] Damianz: ... that would be a good idea, wouldn't it? :-) [22:11:28] Considering I have a 300k log already, if not it's going to get fun [22:11:37] * Coren puts that near the top of his to-do. [22:12:01] Damianz: Meh. There's some 30T of free space. :-) [22:12:26] But yeah, that needs to be done. [22:12:49] Buuut ext3 only supports like 16TiB for a single file :( [22:12:56] s/ext3/ext4/ [22:13:00] Damianz: Thankfully, that's not ext4. [22:13:02] :-) [22:13:06] xfs? [22:13:14] * Coren nods. Think there's the same limit though. :-) [22:13:37] I think xfs is like 6 EB or something insane [22:13:53] Honestly, though, if your access log gets in the TB range, you and are are going to have some words. :-) [22:14:16] is it 32bit or 64bit? apparently 16tb on 32, 9AxaB 64 [22:14:24] This could be fun [22:14:28] s/Axab/ExaB/ [22:14:57] Damianz: so, we may be able to kill NAT [22:15:07] Ryan_Lane: :D With fire? [22:15:11] quantum supports mutli-nic [22:15:12] Ipv6 all the things [22:15:18] Oh cool [22:15:28] Awesome - can we do multiple security groups? [22:15:31] we may be able to replace floating IP with a second NIC for instances that want public IPs [22:15:31] per nic [22:15:42] probably not [22:15:47] Coren: Look on the bright side, I havn't created billions of small files in a directory ;) [22:15:48] the security groups aren't related to NIC [22:15:54] it's just IP based [22:15:58] Ah [22:16:00] or source group based [22:16:37] hmmm. toolserver gets this query pretty quickly [22:18:42] Warning: There is 1 user waiting for shell: Shayt (waiting 83 minutes) [22:19:58] oh. I should try resizes again, since we upgraded [22:20:01] Ryan_Lane: If we're still looking to move to l3 for floating ips... I wonder if we could just route them to the normal ip and not need a second int, would be cleaner I think... esp for failover/moving betwean zones... granted the boxes would have to be configured slightly differently probably or you'd start async routing the traffic back out of the nat'd ip... but we could maybe want that for sec [22:20:09] * Damianz goes away to think quietly [22:20:26] well, that's basically what happens now [22:20:37] but the NAT seems to be broken [22:20:49] and this is apparently a normal OpenStack infrastructure issue [22:22:50] actually yeah - it's arping out so l2, rather than routing but it's just a second ip so the gateway is the same... makes sense why nat is pissy, it would never get a proper entry in the table for translation [22:22:51] I hope resize is reliable now [22:23:58] Coren: Could you 'for JOBID in 80718 80719 80130 80715; do qacct -j "$JOBID" | mail -s "$JOBID" scfc; done' for me on somewhere-qacct-able? Thanks! Is petan able to do that as well? [22:24:03] Damianz: it's not ARP [22:24:10] we don't have a network for our floating ips [22:24:19] we route statically to the network node [22:25:04] is this the fastest way to get a user's edit count? [22:25:04] eventually we'll route via BGP to whichever network node owns the floating IP [22:25:05] oh yeah, I remember - this is why we wanted to bgp it, so it could go to multi node rather than using arp [22:25:07] select count(rev_id) from revision where rev_user=17310446; [22:25:31] well, we've never used arp for that :) [22:25:49] hm. I wonder if arp is actually what's fucking this up [22:25:50] no - just had the 1 network node, rather than network per compute [22:26:02] greenrosetta: SELECT `user_editcount` FROM `user` WHERE `user_name` = ? [22:26:04] the IP is bound to eth0 and not lo [22:26:05] is pretty fast [22:26:24] so it's still arping [22:26:32] even though we're doing static routes [22:26:38] scfc_de: {{done}}. Yes, petan could have done it. [22:26:46] Damianz: ah.. thats not in 3NF... [22:26:54] Coren: Thanks. [22:27:04] Someohow toolserver gets a users top X articles with reasonable speed [22:28:01] Well static routes doesn't make any different to arp... but if the boxes are arping out, that's a lot of spam arguing who owns the ip that you're trying to get traffic to [22:28:12] Could be an interesting tcpdump on the network node [22:28:17] well, we only have a single network node [22:28:23] so there's no arguing [22:29:51] I guess openstack has ebtables or such to stop the vms trying to arp out for ips they havn't been assigned, which would be the only source of arguing if they sit in the same l2 network [22:29:58] yep [22:30:04] it does [22:31:38] resizing takes fucking forever [22:31:53] it looks like this resize is working so far, though [22:32:15] Warning: There is 1 user waiting for shell: Shayt (waiting 96 minutes) [22:32:44] yep. it did indeed work [22:32:55] maybe I can finally enable it! :) [22:33:24] the interface for it will be kind of wonky, though [22:33:39] you need to do a resize, then once it's finished confirm it [22:33:47] Ryan_Lane: http://tools.wmflabs.org/common-interests [22:33:58] That part works nicely [22:34:03] greenrosetta: ? [22:34:16] someones gotta be the first guinea pig [22:34:19] :D [22:34:25] ah. this is pretty cool [22:35:05] If I could figure out how to get a users top X articles in a reasonable amout of time, I could finish this thing tonight [22:35:12] hell, if you made an api, then returned data as json, you could use this in a gadget on projects [22:35:38] Probably can do that [22:36:34] it could be interesting to have a gadget that says "based on my latest editing history, these other users share my interests" [22:37:09] That would require a date range/time period [22:37:36] well, latest being the user's last x articles modified [22:38:13] heh. this tool by itself is cool, though [22:38:57] still buggy [22:39:44] Damianz: hm. if we went with another NIC then the IPs would be randomly assigned. [22:39:47] :( [22:39:52] that's not gonna work [22:40:28] the positive of a floating IP is that it can move from instance to instance immediately [22:42:16] Hmm [22:43:06] I wonder if resize is still broken for floating IPs [22:44:52] I have a feeling even if we 'fix' the nat issue, we might still get weird things happening because of the way SNAT works... mostly betwean boxes on the same host if using the public ip internally [22:45:20] well, that *should* still work [22:45:25] but yeah, nat sucks [22:45:43] Warning: There is 1 user waiting for shell: Shayt (waiting 110 minutes) [22:45:51] We should just make labs ipv6 only and give everything a real ip :D [22:45:54] May not go down well [22:46:04] hahaha. good luck on that ;) [22:46:13] every instance *will* support ipv6. though [22:46:17] in eqiad [22:46:32] do security groups work properly for ipv6? [22:46:46] doubt it somehow [22:47:03] in folsom: "currently, ipv6 and other protocols cannot be managed with the security rules, making them permitted by default." [22:47:07] well, that's not going to work [22:47:19] same in grizzly [22:47:25] well, we won't be having ipv6 any time soon [22:47:28] * Ryan_Lane sighs [22:47:41] I do think a lot of stuff using public addresses doesn't need to be, like really beta just needs 1 ip for a squid/lvs box not like 6.. most wikis could use a proxy etc [22:48:01] yep [22:48:08] most things can use the instance proxy now [22:48:18] The funny thing is, my house has ipv6... there's 1 acl on the ipv6 interface on the router that denies all incoming traffic, yet ipv4 has bunchs of ports open :D [22:48:29] heh [22:49:14] resize -> m1.large worked and resize -> m1.medium worked [22:49:17] andrewbogott: so…… :) [22:49:38] shall we try resizing nova-precise2 to something larger? [22:49:48] Isn't this how that box died last time [22:49:50] actually I did large -> small [22:49:59] Damianz: nope. it died mysteriously last time [22:50:17] yeah, let's try it! It will definitely be more useful if it's large. [22:50:27] that's what she said [22:50:29] cool. I'll resize it to large [22:50:34] it'll be unavailable during that time [22:50:36] doing it now [22:51:12] Actually, andrewbogott reminds me [22:51:16] * Damianz opens up gerrit [22:51:46] ok. resizing [22:52:49] Hmm - actually how hard is it to rename vms? I thought they supported that now [22:53:02] From time to time I get "Unknown MySQL server host 'enwiki.labsdb'", so it seems DNS or whatever isn't rock-solid. [22:53:13] * Damianz doesn't like the idea of maintaining stuff like this as it's a slipperly slide [22:53:15] Damianz, I think it's 'supported' but doesn't work [22:53:25] Oh, like all the best features [22:53:49] scfc_de: I was fiddling with the entries as I added s3 and s6; that might have been my fault. Sowwy. [22:53:50] scfc_de: well, we're kind of doing this in a hackish way right now [22:54:09] No problem. [22:54:11] In that case, this is probably acceptable... but we're probably going to need a maint task to bulk load in the current hosts, since I'm pretty sure some have been broken for months [22:54:37] hm. nova show has the list of security groups [22:54:59] that didn't work in essex. I wonder how it works in folsom [22:56:00] if ( $name != $this->getHostName() ) { < I'm going to laugh the day something has 3 entries for some bug [22:56:04] Damianz: Cluebot is now running from tool labs now? [22:56:12] Coren: sorta-no [22:56:20] Aww. [22:56:51] Coren: ClueBotNG is made up of multiple bits; core/relay/bot/report interface runs on bots, funky api thing that use to be on ts is on tools [22:57:02] Going to look at moving cluebot3 to tools and eventually cluebotng (no mean feat) [22:57:23] Pretty sure from memory I gotta patch the core code before it will compile on a clean system properly [22:58:11] Though... if the security rules allow... I could move it bit by bit, but there's some tcp based stuff betwean report<>core<>bot<>relay that has to work [22:58:11] hm. it looks like it's returned in the regular cal... [22:58:13] *call [22:59:17] Warning: There is 1 user waiting for shell: Shayt (waiting 123 minutes) [22:59:47] Coren: Pretty sure if I ran cbng.php on tools you'd cry ;) [23:10:29] What is the URL to access the en-wiki user page of the user whose rev_user id = 2681322 [23:10:55] im getting "100%tequila" [23:11:00] but obviouslt that wont wokr in the browser [23:15:09] andrewbogott: looks like the resize worked [23:15:32] * andrewbogott tries to create a vm in a vm [23:16:57] Is it possible to have a % in a username? [23:17:16] greenrosetta: what kind of usernames? [23:17:21] run this query [23:17:23] select distinct(rev_user_text) as rut, rev_user FROM revision_userindex where rev_page in (select distinct page_id from page where page_title like 'W%' and lower(convert(page_title using latin1))='war' and page_namespace = 0) order by rut asc limit 10; [23:17:23] mediawiki usernames? [23:17:33] oh. % is like .* [23:17:40] the third entry is 100%tequilla [23:18:10] um… can't log in [23:20:47] im getting error 500's [23:20:53] something must be up [23:21:13] greenrosetta: 500s where? [23:21:19] http://tools.wmflabs.org/common-interests/cgi.py [23:21:22] andrewbogott: opendj doesn't seem to be running [23:21:34] * Ryan_Lane starts it [23:21:41] root@nova-precise2:~# /etc/init.d/opendj start [23:21:41] Data version does not match binaries. Run upgrade script to solve this [23:21:42] hahaha [23:21:44] dist-upgrade [23:21:50] forgot about that... [23:22:03] let me find the upgrade docs [23:22:32] yeah, both my projects went tits up [23:24:07] greenrosetta: eh? what I'm talking about with andrewbogott is unrelated [23:24:24] Coren: seems things in toollabs is broken [23:24:25] ok.. all i know is that neither of my projects will start now [23:24:29] Coren: http://tools.wmflabs.org/?status [23:24:46] thatst dead too [23:24:47] :D [23:24:58] that's why I gave him that link :) [23:25:51] tools-webserver-01 may be down? [23:26:29] Cannot allocate memory: couldn't create child process: /usr/lib/suphp/suphp for /data/project/cluebot/public_html/cb.php [23:26:50] *snort* [23:26:53] does everything use one shared memory pool? [23:26:54] nasty [23:27:08] 20912 ua31 20 0 145m 58m 4800 R 98.1 2.9 101:08.21 perl botr.pl < wth does it do [23:27:10] Coren: can we please use a virtual host and subdomains? :) [23:27:45] *virtual hosts and subdomains [23:29:32] loooooots of 500 errors [23:30:15] Warning: There is 1 user waiting for shell: RAN1 (waiting 0 minutes) [23:30:16] Ryan_Lane: I believe Coren wanted to partition the webservers by tools (cf. /data/project/.system/webservers). [23:33:27] Damianz: That's why I hate screen. It gives people an excuse to not run their job on the grid. [23:33:38] So does nohup [23:34:17] But you can't claim that your tool is interactive when you use nohup. [23:34:38] IPC! [23:34:47] :-) [23:40:42] Dafu?! [23:42:58] Something odd is going on. Cluebot has, like, eleventeen umtpillion stuck PHPs [23:43:35] Actually stuck? [23:43:44] Warning: There is 1 user waiting for shell: RAN1 (waiting 13 minutes) [23:43:52] its all gone off [23:43:58] Hm. [23:44:26] Well, CPU is not going up on 'em. [23:44:35] strace the pid [23:44:41] I guess it's blocked on a select() [23:44:44] which will be mysql [23:44:47] is apache down? [23:45:02] wait.. nvm.. cant be.. im getting an apache err msg [23:45:03] I wonder if they're the dodgy ones... though I doubt it hmm [23:45:04] No, they all be stuck on reads. [23:45:06] * Damianz hates php [23:45:44] The box isn't out of ram, nor loaded, or stuck in iowait. [23:46:28] But apache /thinks/ it's out of ram. Whu? [23:47:15] A swift kick in Apache's butt fixed it, but I have no idea what happened. [23:48:51] Apache sez: "Cannot allocate memory: couldn't create child process" but there was 75% free ram or so. [23:49:03] And the process table was, like, small [23:49:07] security limits? [23:51:40] I'm not seeing it. As far as I can tell, apache just went plain nuts. [23:51:53] * Coren activates webserver-02 and pushes cluebot on it. [23:51:58] I swaer it's not by script [23:52:04] Just sent like 500 requests to it and it works fine [23:52:34] Nevertheless, I want to isolate the problem; if it's not cluebot, then it'll be kept safe while whatever else goes crazy. :-) [23:53:18] If it is cluebot, we'll soon know... I'll have like 30 messages about why it's not editing when I get up :P [23:53:27] random 500 errors [23:53:35] now totally 500 errors [23:53:40] -.- [23:54:05] [Tue May 21 23:53:50 2013] [error] [client 10.4.1.89] (12)Cannot allocate memory: couldn't create child process: /usr/lib/suphp/suphp for /data/project/cluebot/public_html/cb.php [23:54:12] total used free shared buffers cached [23:54:12] Mem: 2051756 1218652 833104 0 168808 479232 [23:54:38] Barely 200 processes. [23:54:47] how many php? shouldn't be that many [23:55:02] 54 [23:55:17] andrewbogott: bleh. we may need to reinitialize the ldap server on nova-precise2 [23:55:19] All CB's? [23:55:20] hmm, there probably shouldn't be that many - but still [23:55:25] * Coren nods. [23:55:35] all cluebot. [23:55:42] And it's talking with itself? [23:55:47] Ryan_Lane, because of the resize or because of the kernel patch yesterday? [23:55:55] because of the dist-upgrade [23:56:01] and opendj failed to upgrade [23:56:22] Or are these external accesses? [23:56:23] scfc_de: Well, it's coming from the webproxy. [23:56:34] oh ffs [23:56:36] I see why [23:56:41] * Damianz stabs mysql [23:56:47] * Damianz then stabs php for ignoring its time limits [23:57:09] Warning: There is 1 user waiting for shell: RAN1 (waiting 26 minutes) [23:57:48] Apparently [23:57:49] select count(*) from (select distinct rev_page from revision_userindex where `rev_user_text` = 'Cydebot' [23:57:52] is really slow [23:58:35] Well, blame Coren :-). [23:58:45] andrewbogott: I take that back [23:58:47] ERROR 1345 (HY000): EXPLAIN/SHOW can not be issued; lacking privileges for underlying table < [23:58:51] * Damianz now stabs Coren [23:59:03] Damianz: Why in blazes aren't you getting user.user_editcount? [23:59:24] Still, CB shouldn't flood the server just due to that. [23:59:28] Because that query isn't for the user edit count - I am geting that for the edit count [23:59:37] That query is the number of PAGES a user has edited [23:59:56] andrewbogott: the upgrade did work