[00:00:22] indexes are important if you expect some performance from it [00:00:37] yep, especially when things get big :) [00:00:47] i.e. what has just happened to mine xD [00:01:06] it went from a few thousand to the above :/ [00:01:11] heh [00:01:18] but it shouldnt ever get bigger than this ;p [00:01:21] petan|wk: If you expect performance - CACHE EVERYTHING and don't touch the db [00:01:29] lol [00:01:33] text file and ssd? ;p [00:01:40] reminds me of idea of PC running from ramdisk [00:01:54] cbng had that problem until we lost the db and started again - hundreds of thousands of rows of revert data... slooow [00:02:16] buy 60gb of ram, create 40gb ramdisk - put all stuff on that including OS during boot [00:02:27] browse port safely [00:02:31] * porn [00:02:36] HAHA [00:03:00] ooooh... getting up to a million rows again now =D [00:06:34] petan|wk: http://ganglia.wmflabs.org/latest/graph.php?r=hour&z=xlarge&c=bots&h=bots-bsql01&v=20.3&m=part_max_used&jr=&js=&vl=%25&ti=Maximum+Disk+Space+Used [00:06:41] Yay :) [00:06:52] what is that [00:06:57] storage per project? [00:06:58] disk on bsql01 [00:06:59] xD [00:07:03] ah [00:07:11] AND ram on bsql1 http://ganglia.wmflabs.org/latest/graph.php?r=hour&z=xlarge&h=bots-bsql01&m=load_one&s=by+name&mc=2&g=mem_report&c=bots [00:07:21] cache cache cache ;p [00:07:39] addshore: you can't believe that [00:07:43] it takes all disks together [00:07:51] if u type df you will see 1% [00:09:36] 1GB used >.< [00:09:43] dam slow import [01:12:44] petan|wk: import done :D [01:12:53] * addshore is happy* [01:14:17] * Damianz gives addshore a unicorn [01:16:08] * addshore runs a speedy count [01:17:18] * addshore is a unicorn [01:34:16] now for the massive alter which broke last time :P [01:35:37] heh [01:35:40] gluck [01:39:00] legoktm: hu done yet? :P [01:39:14] i dont think i got approved on hu? [01:39:53] wait, I cant remmeber which it was, do I mean he? [01:40:02] the one you were talking about earlier :P [01:40:14] yeah he [01:40:18] my bot is still running [01:40:30] also petan|wk is there some sort of limit for transfer rate per instance to the network storage? [01:40:48] or infact on the network as a whole [01:41:33] the instances all currently have 100Mb network cards [01:41:44] we're hoping to change that to 1Gb cards next week [01:41:49] hmm [01:42:43] the ganglia graph for the net on bsql01 seems to have a rather low maximum usage, and as far as I can tell that would be the only thing slowing my current query down :/ [01:44:00] * addshore might get petan to fiddle with sqld tomorow :) [01:51:05] [bz] (8RESOLVED - created by: 2Addshore, priority: 4Unprioritized - 6normal) [Bug 45654] bots-sql3 needs more disk space - https://bugzilla.wikimedia.org/show_bug.cgi?id=45654 [01:52:55] erm [01:52:57] oh [01:53:05] those colors >.< [01:57:10] all looks good on my client except maybe yellow :P [01:57:32] its the yellow [01:57:40] i cant read it [02:05:14] legoktm: this is how I feel right now.. http://thebest404pageever.com/swf/processing.swf [02:15:59] heh [06:03:28] hashar: get some better sleep tonight! [06:03:38] I do need [06:03:52] some wild girls have been partying in the room next to me for the last two nights [06:04:09] I am just waiting for my wife to wake up for the daily conf call :_D [06:04:46] hah [06:10:25] * jeremyb_ sleeps [06:54:02] legoktm: your irc client suck [06:54:10] wat. [06:54:16] it just uses a white background [06:54:40] just try reading http://cl.ly/image/1U2F00263i27 [06:54:42] meh [06:55:15] you can add a background to the colors [06:55:20] that would be nicer [08:51:10] @notify addshore [08:51:10] I will notify you, when I see addshore around here [08:53:23] * legoktm wonders what addshore is in trouble for now [09:36:15] addshore hey let me know if there are still some [09:36:28] data on these boxes you were copying from [09:36:34] so that I can delete them once it's done [09:37:01] I suppose it finished [09:49:40] legoktm anyway, regarding irc clients, no matter of background it should display the colors so that you can see them [09:49:53] yes but [09:49:58] the color code stands for yellow, but it could be some more dark yellow :P [09:50:05] that [09:50:06] the bot should display colors for all users :P [09:50:16] colors usable for all users* [09:50:17] legoktm you know you can change the colors of bot? [09:50:18] :P [09:50:25] oh really? [09:50:26] how? [09:50:27] yes [09:50:34] I need to reed docs sec [09:50:35] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.10.6.8 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [09:50:38] :P [09:50:44] good, i'm going to change it to black. [09:50:55] or better, no colors at all! [09:50:57] style-rss string message template for rss item (variables: $name $author $title $link $description) [09:51:03] http://bots.wmflabs.org/~wm-bot/dump/%23wikimedia-labs.htm [09:51:17] meh [09:51:25] no way to retrieve current value :/ [09:51:39] * legoktm is lazy [09:52:11] ill just try and ignore new bugs :P [09:52:32] [bz] (8$bugzilla_status - created by: 2$author, priority: 4$bugzilla_priority - 6$bugzilla_severity) $title - $link [09:52:35] damn [09:52:42] it's html encoded version :D [09:53:29] @rss-setstyle [09:53:32] @rss-setstyle fdsga [09:53:32] I don't have this item in a db [09:53:39] @rss-setstyle bz test [09:53:39] I don't have this item in a db [09:53:44] @rss-setstyle bugzilla [09:53:45] Item now has a different style you can restore the default style by removing this value [09:54:36] @rss-setstyle bugzilla [bz] ($bugzilla_status; - created by: $author, priority: $bugzilla_priority - $bugzilla_severity) $title - $link [09:54:36] Item now has a different style you can restore the default style by removing this value [09:54:44] @rss-setstyle bugzilla [bz] ($bugzilla_status - created by: $author, priority: $bugzilla_priority - $bugzilla_severity) $title - $link [09:54:45] Item now has a different style you can restore the default style by removing this value [09:54:47] no colors :P [09:56:41] :))) [09:56:50] * legoktm goes to file a bug about wm-bot not having colors :PPPP [11:05:39] !log wikidata-dev wikidata-testrepo: Last week's attempt of puppet to downgrade php5-mysql (that Andrew fixed manually) was back today. Fixed i manually again. Hm... [11:05:41] Logged the message, Master [11:33:59] Hi there! Did you see my e-mail "how to puppetize solarium" on labs-l? Any opinions on that? [11:35:10] * saper does not subscribe, even. [11:35:52] tststs [11:36:51] petan: there probably is still data on them [11:36:55] I only removed my db [11:40:15] Silke_WMDE: is it one of those nicely worded, well-researched, repectful and polite emails that provide all the information and actually require some actual work to be done other than hitting the R button, that never get a reply? [11:41:45] What? I get replies to most of my e-mails on this list. [11:41:59] http://lists.wikimedia.org/pipermail/labs-l/2013-March/000932.html [11:43:04] :-) bad joke, sorry [11:43:17] It's a puppet "layout" question, don't know if it requires extra work. For people who know what the layout is supposed to look like, it's probably hitting R. [11:45:19] frankly it might be just a timezone issue [12:31:34] andrewbogott_afk: we toook the liberty of assuming that your IRC nick/cloak/etc. were spelled correctly. https://commons.wikimedia.org/w/index.php?title=File:Andrew_Bogott_staff_photo.jpg&diff=91912785&oldid=73770124 [12:31:41] legoktm: ^ [12:32:03] :) [12:32:34] legoktm: i think he's actually in your TZ [12:32:39] oh cool [12:33:00] I should probably be sleeping... [12:37:11] addshore ok, so can I safely remove all addbot db's now? [12:44:20] Silke_WMDE: for solarium where is the git repo? [12:46:47] ok, so it's not clear to me how cloning the repo is different from using composer [12:47:12] but probably if you want to use it with puppet then you need to: [12:47:31] make a mirror of the github repo on gerrit and maintain that mirror manually as needed [12:47:36] and have puppet clone from gerrit [12:47:41] .... [12:48:01] legoktm: ? [12:48:19] nvm i read that wrong [13:23:09] petan yes "addbot" and "addshore_dump" [13:23:59] addshored_dump is not copied [13:24:01] :/ [13:24:05] its empty ;p [13:24:08] ah [13:24:10] ok [13:24:13] hadnt started using it yet :P [13:24:27] been able to run all of my alters on the table now :) [13:24:30] * addshore is happy [13:25:31] you have superuser on mysql there so I suppose you can do that [13:25:34] :D [13:25:49] what kind of alters it were [13:28:44] adding another collum and altering an index [13:28:51] jeremyb_: composer downloads _and_ installs solarium [13:29:10] the collum took a few hours and the index took 15 mins :P [13:29:37] Silke_WMDE: what does install mean? [13:29:59] I think it tells php about the new library [13:33:27] huh [13:33:45] well i think i don't even want to know what that means in php [13:33:59] i kinda know the equivalent for java, perl and python [13:34:04] ?? [13:34:23] ??? [13:34:26] :) [13:34:30] :) [13:34:37] you are sceptic [13:34:56] i think you should figure out how to make it work without composer [13:35:11] I don't know how to tell php about it when I clone the git repo just like that. [13:35:27] well you can run composer and then figure out what it did [13:35:29] and copy that [13:36:17] ok [14:05:46] Damianz can you move cluebot data to bots-bsql01? [14:06:10] I would like to remove all sqlN instances [14:06:37] :D [14:11:04] addshore is your bot working now? [14:12:20] the db is working [14:12:28] gonna add some more stuff to it though ;p [14:13:02] 2 more collums whihc might take a while, so Im htinking carefull before I make any changes ;p [14:17:54] addshore I don't think adding columns take time [14:18:13] it's just a ddl command [14:18:21] won't really change any data [14:18:30] true, just last time I was adding data too ;p [14:21:58] I think a collum with the date the row was last edited and then a count of the number of links left after each check and potentialy a collum with the number of times it has been checked :/ [14:37:47] addshore: I admire your consistency, but I believe it's spelt "column" :-). [14:38:09] haha! indeed :) [14:41:28] lol :d [14:47:53] legoktm could you run the db parse on en for me? ;p [14:48:01] ok..... [14:48:11] which instance should i run it from? [14:48:18] bnr1 :) [14:48:44] legoktm@bastion1:~$ ssh legoktm@bots-bnr1 [14:48:44] If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [14:48:44] Permission denied (publickey). [14:48:48] petan: :( ^ [14:48:52] oh :/ [14:49:00] tell me what to type and where and I'll run it? ;p [14:49:06] legoktm o.O [14:49:14] @labs-user legoktm [14:49:15] That user is not a member of any project [14:49:20] @labs-user Legoktm [14:49:20] Legoktm is member of 4 projects: Bastion, Bots, Editor-engagement, Tools, [14:49:31] let me check [14:49:37] @labs-user addshore [14:49:37] That user is not a member of any project [14:49:40] >.< [14:49:42] @labs-user Addshore [14:49:42] Addshore is member of 6 projects: Bastion, Bots, Huggle, Hugglewa, Proposals, Tools, [14:49:57] * legoktm would like to see someone finish hugglewa [14:50:12] haha, we cant [14:50:23] until the whole openid login e.t.c is done [14:51:13] why not? [14:51:25] just use jquery [14:51:38] legoktm: first of all you shouldn't need to ssh there [14:51:47] oh? [14:51:48] you can launch mysql from any bots instance (application one) [14:51:54] BUT [14:51:54] nono [14:52:00] you should be able to do that [14:52:06] im running a script that outputs to a text file [14:52:10] legoktm what exactly you want to do? [14:52:18] log into bots-bnr1 :) [14:52:26] ok such a script can run on any instance? [14:52:29] yes [14:52:36] oh wait lol [14:52:42] but i can run faster when it has more resources [14:52:44] it* [14:52:48] I thought you are talking about bsql1 [14:52:50] nvm me [14:52:52] xD [14:53:15] * legoktm gives petan some coffee [14:54:44] legoktm: can you tell me the reason why you can't ssh [14:54:55] ssh -vvvv bots-bnr1 [14:55:32] there is a public key uploaded in your keys folder [14:55:38] http://dpaste.de/8hAwx/raw/ [14:55:39] so I don't really see a reason :/ [14:58:02] legoktm and you are trying it from bastion right? [14:58:04] other instance work? [14:58:07] legoktm: Eh, do you really store your private key on bastion? [14:58:17] scfc_de I do same [14:58:33] imho it's secure :> [14:58:50] as long as your private key is for labs only [14:58:55] hi yalls, anyone know much about connecting to http on labs instances? i've done it before but am currently having trouble [14:58:55] i do? [14:58:57] hmmm [14:59:18] legoktmfrom logs it looks like your ssh client actually doesn't see your keys [14:59:21] it can't read them [14:59:26] erm [14:59:27] ottomata hi [14:59:30] hiya [14:59:41] i'm just using a normal terminal window [14:59:47] @search proxy [14:59:47] Results (Found 1): socks-proxy, [14:59:50] scfc_de: er…where is it stored? [14:59:51] yeah [14:59:58] i've gotten that to work before [14:59:58] ottomata you mean connecting using proxy? [14:59:59] it just isn't right now [15:00:00] !socks [15:00:01] ssh @bastion.wmflabs.org -D ; # [15:00:02] so, simplest example, i have a reportcard.pmtpa.wmflabs [15:00:03] i should probably remove it [15:00:23] i can http request from the instance [15:00:24] but [15:00:26] legoktm remove what [15:00:29] even from a bastion [15:00:32] it times out [15:00:39] otto@bastion-restricted1:~$ curl -v http://reportcard.pmtpa.wmflabs:80 [15:00:39] * About to connect() to reportcard.pmtpa.wmflabs port 80 (#0) [15:00:39] * Trying 10.4.1.55... Connection timed out [15:00:44] petan: my private key [15:01:00] if you remove your private key then no wonder you won't auth... [15:01:05] oh [15:01:07] wait [15:01:11] * legoktm is confused now [15:01:18] so why cant i connect? [15:01:19] are you forwaring your keys? [15:01:26] i can get into normal instances just fine [15:01:27] yes [15:01:40] from my laptop terminal i type in [15:01:44] if you are forwarding it you can remove it from storage [15:01:46] on labs [15:01:47] ssh legoktm@bots-3.pmtpa.wmflabs [15:01:48] otto@reportcard:/etc/apache2/sites-enabled$ curl -v http://reportcard.pmtpa.wmflabs [15:01:48] * About to connect() to reportcard.pmtpa.wmflabs port 80 (#0) [15:01:48] * Trying 10.4.1.55... connected [15:01:51] that gets me in just fine [15:01:57] but not for bots-bnr1 [15:02:09] petan: legoktm: I just use ProxyCommand. No keys stored, no double logins, etc. [15:02:15] right, thats what i'm using too [15:02:21] scfc_de so? [15:02:53] scfc_de that still may be insecure at some point :P [15:03:09] petan: That means I can't help with legoktm's bastion login problem :-). [15:03:31] well i cant login from any instance to bots-bnr1 [15:03:39] i can get from bastion --> bots-3 for example [15:03:45] i can go from laptop --> bots-3 [15:03:51] just not to bnr1. [15:03:52] legoktm did you try using bastion3 or bastion2 [15:04:09] no, il ldo that now [15:04:40] legoktm I will check logs further but I don't see a reason why it doesn't work unless there is a brainsplit again and it show different pubkey on bnr than on other instances [15:04:47] the pubkey is readable on bnr1 [15:04:49] no on bastion2 [15:04:56] legoktm? [15:05:15] i tried from bastion2 and it didnt work [15:05:26] ok, did you change your key recently? [15:05:30] and didnt work on bastion3 [15:05:31] nope [15:05:39] same key since i signed up [15:05:40] mhm [15:05:56] whats wrong with this? O_o "ALTER TABLE iwlinked ADD links SMALLINT, ADD checked DATETIME DEFAULT ON UPDATE CURRENT_TIMESTAMP;" [15:05:57] ^demon did update my key in svn.wikimedia.org though [15:06:07] ok let me compare them [15:06:12] but that update is to the same key [15:06:55] addshore ADD links SMALLINT, checked... [15:06:56] ? [15:07:46] nop [15:08:04] somehting wrong with DATETIME DEFAULT ON UPDATE CURRENT_TIMESTAMP;" [15:09:17] legoktm can you retry now? [15:09:52] yay :) [15:09:54] i'm in! [15:10:24] legoktm I didn't do anything [15:10:29] :o [15:10:32] I just was tailing the log so that I see error [15:10:40] there was no error :/ [15:10:54] lolwut [15:11:06] Accepted publickey for legoktm from 10.4.0.54 port 44967 [15:11:15] can you try several more times? [15:11:19] heheh sure [15:11:20] just to make sure it works [15:11:50] well i just logged in + out 5 times [15:11:58] I see [15:11:59] ahh /me has to specificy default and onupdate [15:12:01] * legoktm thanks petan's magic :) [15:12:08] I think it was automount [15:12:16] how I tried to read your public key [15:12:22] maybe it somehow mounted what wasn't before [15:12:29] interesting [15:12:43] ok so where should i delete my private key from if i'm going to be using proxycommand? [15:12:55] from bastion [15:13:22] where in bastion? [15:13:26] its not in my .ssh [15:13:40] they you don't have the private key there... [15:13:45] oh [15:13:54] scfc_de: then what were you referring to? [15:14:27] petan, adding collums does take a long time :( [15:14:30] 1.64% [15:14:39] of what [15:14:53] im adding 2 collums, and it goes rather slowly :p [15:15:07] it depends if these columns are keys / indexed / have some default value other than NULL etc [15:15:08] legoktm: Hmmm. It looked to me as if you wanted to store the key on bastion, but according to your log you didn't. So everything seems to be fine. [15:15:34] ok :) [15:15:36] Traceback (most recent call last): [15:15:36] File "xmlscanner.py", line 27, in [15:15:36] import os [15:15:36] ImportError: No module named os [15:16:04] :/ [15:16:23] maybe virtualenv's dont carry accross instances? [15:16:38] virtual env? [15:16:39] what is that [15:16:50] you mean environment? it doesn't [15:16:58] virtualenv is a python thing [15:17:01] er its a python thing [15:17:03] oh [15:17:05] and surely they don't travel across instances [15:17:08] right? [15:17:12] why not? [15:17:26] the directory structure is all from the same place [15:17:28] i mean, it probably stores the env in your homedir…and homedirs are available across instnaces [15:17:35] right, but don't you have to initialize the virtualenv? [15:17:39] i did that [15:17:39] (i've only used it a few times) [15:17:40] ah [15:17:42] nm thne [15:17:44] i will hush [15:17:48] and mine is stored in /data/project/legoktm/py2 [15:17:51] :P [15:17:59] lemme see what happens if i create a new one [15:18:02] aye [15:18:10] petan, any insight on my curl issue? [15:18:12] lol [15:18:25] i can curl from localhost on my instance, but not from a bastion [15:18:26] ottomata mhm I forgot what it was :D [15:18:28] petan: can you install "python-virtualenv" on bots-bnr1? [15:18:35] ottomata of course [15:18:38] ottomata firewall [15:18:43] oh it comes installed? [15:18:47] did you open port 80? [15:18:55] yes it is on your project definition [15:19:02] !security [15:19:02] https://labsconsole.wikimedia.org/wiki/Help:Security_Groups [15:19:03] hmmmMMMMM [15:19:12] reading [15:21:14] petan these alter queries are really odd, the instance isnt at maxram or maxcpu or max anything but it still goes really slowly O_o [15:21:31] mhm [15:21:46] http://stackoverflow.com/questions/5677932/optimize-mysql-for-faster-alter-table-add-column [15:22:23] petan, if a group is in a project, all instances in that project are then in that group [15:22:25] correct? [15:22:30] (security group*) [15:22:43] no [15:22:50] you select group when you create instance [15:23:02] can you edit after it is created? [15:23:06] it wasn't possible to be changed later, dunno if it still isn't [15:23:14] AGHHHH [15:23:19] poop scoops [15:23:24] :P [15:23:31] that is mentioned in help [15:23:41] * legoktm nudges petan for "python-virtualenv" on bots-bnr1 ;) [15:23:54]