[00:05:20] There we go. fix't [01:11:33] right my db is actually very broken.. [01:12:38] no requests work :< [01:19:03] Coren: make me a bots-sql4? :D [01:22:56] or reboot sql-3? :/ [01:34:17] or anyone :/ [02:30:55] !log bots creating sql4 for heavy queries [02:30:57] Logged the message, Master [02:31:28] !log bots by heavy I mean addshore :))) [02:31:30] Logged the message, Master [02:31:46] HAH [02:32:06] its going to be fun shifting my db over ;p [02:33:07] Ryan_Lane: failed to create :( [02:33:13] no reason :/ [02:33:18] :/ [02:34:07] @notify Ryan_Lane [02:34:07] This user is now online in #wikimedia-dev so I will let you know when they show some activity (talk etc) [02:34:21] addshore we can go sleep now [02:34:43] HAHA, im still fixing some of my stuff ;p [02:34:57] so do [02:35:00] I lol [02:35:02] I am ;p [03:48:58] dam, petan now with what I am trying to do I get a lovely PROBLEM Disk Space is now: CRITICAL on bots-sql3.pmtpa.wmflabs 10.4.0.40 output: DISK CRITICAL - free space: /mnt 134 MB (0% inode=99%): [03:59:44] Coren: Could you install ack-grep on webtools-login, please? [04:39:15] [bz] (8NEW - created by: 2Adam Shorland, priority: 4Unprioritized - 6normal) [Bug 45654] bots-sql3 needs more disk space - https://bugzilla.wikimedia.org/show_bug.cgi?id=45654 [04:42:28] ^ That should have been tools-login, of course. [04:44:00] (Which also explains why there was a difference between a file in my open shell session to tools-login and my Emacs buffers for /webtools-login...: -- *argl*! PEBKAC) [15:42:44] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Notepad was modified, changed by MPelletier (WMF) link https://www.mediawiki.org/w/index.php?diff=654814 edit summary: [+9] [20:12:13] [bz] (8NEW - created by: 2Ryan Lane, priority: 4Unprioritized - 6normal) [Bug 45670] Set ulimits on bastion - https://bugzilla.wikimedia.org/show_bug.cgi?id=45670 [20:39:00] Ryan_Lane, I'm not actually sure how I would change that file to use the public address instead... [20:39:05] !ping [20:39:05] pong [20:39:16] Krenair: via controller_hostname [20:39:22] there's a possibility that could break other things, though [20:39:39] hm [20:39:39] no [20:39:39] it's only used in that one file [20:39:39] Krenair: should be able to just set it via "configure" for the instance [20:40:06] !ping [20:40:06] pong [20:40:08] lol [20:40:08] hahaha [20:40:08] 3min lag [20:40:16] why so long? [20:40:17] that's what I call slow boot up [20:40:17] it's loading... [20:40:17] ah [20:40:17] * Ryan_Lane nods [20:40:17] it resides in about 80 channels [20:40:17] that's why start is so slow [20:40:17] right [20:40:22] !ping [20:40:22] pong [20:41:07] today was funny [20:41:23] I bought a new laptop and wifi wasn't working on kernel I wanted [20:41:34] so i had to write own patch for driver :> [20:41:52] after like 10 Ooopses I finally got it working on 3.8.x kernel [20:42:19] :D [20:42:42] I am still afraid of rebooting it when it finally works [20:42:45] :D [20:42:57] Okay I set controller_hostname to the public host, now how do I get puppet to update that file? [21:01:23] Krenair: run: puppetd -tv [21:24:07] notice: Finished catalog run in 87.86 seconds [21:24:08] err: Could not run command from postrun_command: Cannot allocate memory - fork(2) [21:24:09] hm [21:32:10] I stopped some services to make puppet run [21:32:18] hopefully that'll let it complete a run [21:32:40] this instance needs more memory [21:35:12] well, it ran, but didn't fix the issue [21:35:16] I'll take a look at this soon [22:13:45] petan|wk / petan I saw a ping from you earlier but have now lost it, what did it say? :O [22:14:18] addshore that u can use sql2 that has space [22:14:38] more total space than sql3? [22:14:53] as I use most of the space there :/ thats the problem. [22:14:55] yes [22:15:00] 40gb or more [22:15:40] Ryan_Lane: we wanted to create some better sql server for bots project but I wasn't able to make it [22:15:49] failed to create [22:15:53] ah, you are probably hitting a quota [22:15:59] I need to pass those messages through [22:16:00] who knows [22:16:08] I wanted to remove some boxes [22:16:15] and use few huge instead of many small [22:16:22] that would save a lot of resources [22:16:27] because OS eats a lot of them [22:16:35] * Ryan_Lane nods [22:16:39] and having 1 OS instead of 6 would definitely save some ram and such [22:17:00] heh. there's 22 instances in bots [22:17:03] addshore: how urgent is your need for a better sql server [22:17:04] I'm sure you're hitting quotas [22:17:07] * addshore is going to eat sql2 and sql 3 for a while while he transfers his rather large db from one to another :/ [22:17:14] @labs-project-instances bots [22:17:14] Following instances are in this project: bots-2, bots-cb, bots-1, bots-sql2, bots-sql3, bots-sql1, bots-3, bots-4, bots-labs, bots-dev, bots-salebot, bots-apache-test, bots-nr1, bots-apache01, bots-abogott-devel, bots-analytics, bots-nr2, bots-liwa, bots-bnr1, [22:17:16] yay [22:17:36] petan|wk: well the biggest usage will be when wikidata rolls out onto other projects (a few days time) [22:17:38] addshore: maybe wait [22:17:44] if we create a new huge one [22:17:50] we could kill all sql1 2 and 3 [22:17:53] and move all db's there [22:18:10] * addshore thinks that would be a good idea [22:18:11] we can call it bots-bsql1 :D [22:18:14] we talked yesterday about how cool it would be to 'magically' add memory to a running instance [22:18:27] Platonides it would be amazing! [22:18:28] Platonides: heh it shouldn't be that hard [22:18:35] magically add another cpu ;p [22:18:53] actually vm-ware can do these things on the fly lol :D just guest systems can't survive that [22:19:08] well, not magically, but as it's virtualised, it's much easier than doing it on a real server :) [22:19:08] petan: it'll work now [22:19:13] ok [22:19:41] I think we already have a bug about passing through error messages, rather than giving a generic one [22:19:53] hmm petan|wk looking at sql3 and sql2 I had the problem on sql2 with 10 GB free and it appears sql2 has 2 GB free :/ [22:19:59] so that wouldnt actually fix my problem :P [22:20:06] I don't know if openstack supports it, or if the compiled kernels have support for hotswapping memory [22:20:15] addshore: what lol [22:20:24] addshore: if there is 2gb free that really suck [22:20:26] night [22:20:39] * addshore is going to login and do df now but ganglia doesnt show much room [22:20:45] Ryan_Lane: how large instance can we create then [22:20:53] as large as necessary [22:21:28] mhm all right [22:21:50] on other hand if I create 10gb inno cache I will save a lot of IO :P [22:22:10] !log bots created a huuge instance for addshore and his crap :) [22:22:18] damn bot [22:22:32] maybe I should rewrite it using supybot [22:22:54] rewrite the logs bot? :P [22:22:58] the bot's logic is amazingly small [22:23:00] heh that would be cool, though andrewbogott didn't like idea of rewriting something what is already written [22:23:20] well, it's a problem that it doesn't reconnect properly [22:23:31] we'd just be changing libraries, really [22:23:31] current library can't fix it? [22:23:39] not as far as I can tell [22:23:43] wm-bot actually has troubles with that as well, it's rather problem of ircd [22:23:46] I've tried numerous times [22:24:00] I know supybot handles this correctly [22:24:01] if connection die and you reconnect too fast, ircd put you on anti hammer [22:24:05] or something like that [22:24:07] hm [22:24:11] I wonder if that's the cause [22:24:22] either way, this is an annoying problem [22:24:28] I remember I sometimes had to shut down my irc client and then try to connect again to fix ti [22:24:30] it [22:24:47] I really need to change the mount timeout of gluster, too [22:24:52] that's also very annoying [22:25:15] labs-morebots: welcome back [22:25:16] I am a logbot running on i-0000015e. [22:25:16] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [22:25:16] To log a message, type !log . [22:26:22] we need to rewrite this anyway [22:26:30] I don't want it writing to mediawiki pages anymore [22:26:39] that wikitech import took an eternity [22:26:48] and now the xml dumps are giant and take forever [22:30:47] Ryan_Lane: did you import full history? [22:30:51] yes [22:30:55] btw that is something that needs to cahnge [22:31:02] the way how mediawiki stores stuff [22:31:06] why? [22:31:37] I believe there should be ability to create pages with no history for similar purposes, so that pages that are generated by bots like 100 times a day would eat no space in db [22:31:56] did you see the statistic pages on english wiki [22:32:17] each of them probably eats GB's of sql space which is absolutely for nothing [22:32:25] I was considering switching this out with http://www.mediawiki.org/wiki/Extension:EventLogging [22:32:28] huge pages that are generated by bots [22:32:35] or I remember huggle whitelist [22:32:40] that was one huge page [22:32:48] * Ryan_Lane nod [22:32:50] with hundreds of thousands revisions [22:32:51] *nods [22:32:59] yeah, it's kind of silly [22:43:36] addshore: your sql server is ready in few minutes :P [22:43:38] hold on [22:43:39] :D [22:43:42] :P [22:43:52] you can prepare some migration scripts :D [22:44:05] * addshore was just thinking how to move his db :P [22:44:28] I hope you won't eat these 160 gb too fast but I set up lvm so we can eventually mount another storage and extend it [22:44:41] I shouldnt :p [22:44:49] the db isnt going to get any bigger than it is now [22:44:59] and to manipulate it I need twice the size of the db :P [22:45:09] really? [22:45:10] why :D [22:45:22] you know you can mysqldump | mysql [22:45:38] petan|wk: 20,000,000 rows :/ [22:45:47] any size [22:45:54] * addshore may try it [22:45:55] that's how I imported simple wiki [22:46:04] was like 60gb [22:47:25] good :P Mine is only about 25GB :) [22:49:28] !log bots petrb: installing mariadb on bsql01 [22:49:30] Logged the message, Master [22:52:18] mysqldump -uuser -ppassword myDatabase | mysql -hremoteserver -uremoteuser -premoteserver :)) [22:52:27] woop woop petan|wk let me know when its ready =] [22:53:09] lets just hope the pipe doesnt collapse :P [22:53:30] sure [22:53:51] 82kb/s download [22:53:53] slow labs :/ [22:54:17] well I should have pick some american mirror heh [22:54:20] Ryan_Lane: Totally just shove stuff into redis :D [22:56:49] Damianz: yeah, that was an idea, too [23:15:18] addshore: it's all done [23:15:24] just eh, users lol [23:15:27] damn ldap [23:15:30] :D [23:15:36] pm me some password [23:16:55] right so in theroy this should work :P [23:16:55] mysqldump -u addshore -p addbot | mysql -h bots-bsql01 -u addshore -p [23:19:27] wait lol [23:19:32] addshore: did you try if you can login there? [23:19:41] you should be able to create users and dbs and all that crap [23:19:42] * addshore thinks it failed :P [23:19:49] I gave you admin rights [23:19:52] * addshore will try logging in first ;p [23:21:03] ERROR 2003 (HY000): Can't connect to MySQL server on 'bots-bsql01' (111) [23:23:15] ok try now [23:23:25] bingo :) [23:23:32] I forgot to listen on all ethernets [23:23:33] :D [23:23:47] now to try running this command ;p [23:24:02] okay db is initally optimized - like all buffers has 2000% of normal size :P [23:24:08] HAHA [23:24:12] I will do more tunning as I see the queries and how it run [23:24:45] * addshore is trying to import the db now :P [23:24:54] btw you can u htop ;) [23:25:40] hm /db used 1% and we just created users :P [23:26:10] how pretty ;p [23:26:18] petan|wk: my import might have lready started ;p [23:26:34] really? [23:26:38] I don't notice it [23:26:43] :D [23:26:44] hmm [23:26:52] and I didnt notice it on sql3 either O_o [23:27:09] I even doubt [23:27:15] there is no new database [23:27:16] :D [23:27:23] * Damianz wonders why petan is at work at midnight on a sunday [23:27:25] unless you import directly to mysql [23:27:35] Damianz: because I am at work all time [23:27:37] :D [23:27:40] would "mysqldump -u addshore -p addbot | mysql -h bots-bsql01 -u addshore -p" create the new db ..? [23:27:48] btw in 4am they woke me cuz some server was down [23:28:00] naaaw :P [23:28:03] I checked logs and responded: guys that server was down like 6 years [23:28:10] are you sure it just crashed? [23:28:10] HAHA! [23:28:15] lol [23:28:31] I have a nice vpn... so I'm at home, doing work... though I'm not actually on call for another hour. [23:28:35] they told me "sorry we did a wrong check, go sleep" [23:28:36] :D [23:29:02] Damianz: I bought a laptop and whole day I was patching driver for wifi :D [23:29:09] I had to write own patch cuz nobody did it yet [23:29:17] heh [23:29:25] that's when knowledge of c++ is good :D [23:29:39] though it was like fixing some renamed variables and structures only [23:29:40] I really need to learn c/c++... :( can never get the hang of it [23:29:47] this was fun [23:29:51] kernel devs are mad [23:29:51] Also win, I'm not on level 1/2 on-call until next week =D [23:30:18] I'd love to do driver work on the kernel... but that community isn't friendly to noobs [23:30:21] they like renamed variable 'information elements' to eis [23:30:30] probably it wasn't confusing enough [23:30:31] For good reason tbf - they're trying to keep it not-broken on hundreds of platforms. [23:30:59] * err ies [23:31:01] not eis [23:31:19] ERROR 1046 (3D000) at line 22: No database selected [23:31:22] I see [23:31:36] I would rewrite whole linux kernel to c++ :> [23:31:44] c++ has too much magic [23:31:46] c is nice but old [23:31:47] :D [23:31:51] I like structure [23:32:06] no objects make it incredible hard to read [23:32:22] like whole kernel is a bunch of functions and definitions [23:32:31] no objects or classes whatsoever [23:32:35] Hmm - if I'm going to build a 'custom distro' to pxe boot stuff into to for the purposes of adding to an auditing system should I actually write a custom distro or just take debian and throw a python script on it to take over the default tty and output the info? [23:33:01] Damianz: I always over take debian :P [23:33:05] simplest [23:33:19] but I recommend you to take debian sid :P [23:33:24] it will look more like you made it [23:33:29] it will also contain a lot of bugs [23:33:34] :) [23:33:37] lol [23:33:42] that makes it more authentic [23:33:48] I might use centos or debian stable... [23:33:52] like you see this distro is made by me - look how it crash [23:34:40] I am watching htop like if it was some kind of porn [23:34:45] Damianz: join us? :) [23:34:46] Not really bothered about it looking like I did it - just clear to noobs that I can say 'pxe boot the pc, wait until it says done' and have it pull the hardware and throw it at a rpc-json api without interaction :D [23:34:48] :D [23:34:58] Not really a fan of htop... [23:34:58] addshore: where you begin with that [23:35:09] better than glance on hp ux [23:35:35] * addshore has to make the db first [23:35:48] addshore: mysqldump can do that [23:35:53] apparently not [23:35:57] it typically has create db statements [23:36:03] Depends how you dump it [23:36:09] mhm [23:36:13] maybe [23:36:13] ERROR 1049 (42000): Unknown database 'addbot' [23:36:22] you can head it [23:36:25] * addshore goes to find a param for mysqldump that creates [23:36:33] mysqldump | head -n 200 [23:36:38] so that you can see if it creates it or nto [23:36:39] or just create database addbot in mysql [23:36:44] or that [23:36:57] or import all tables into mysql db [23:36:57] who cares [23:37:08] You can get MYSQLDump to drop the database before creating it by adding [23:37:08] --add-drop-database [23:37:12] :D [23:37:17] :P [23:37:24] can you also make it create it? :D [23:37:30] that would be cool as well [23:37:54] 800mb eaten from 32gb [23:37:54] mysqldump addbot | mysql addbot # wooooop [23:37:56] of ram :D [23:38:19] glusterfs eats it [23:38:20] :/ [23:38:23] I have oracle database servers with less than 32gb of ram :P [23:38:31] Damianz: really? :D [23:38:40] I actually have tons of oracle db's [23:38:51] I don't want to know how many dbs I have, it would make me sad. [23:39:04] same :D [23:39:23] I know of 8 seperate oracle db servers off the top of my head... only like 6 mysql though :( Damn working on silly oracle apps for the past month [23:39:24] I recently fucked my tnsnames and I was wondering if I should fix it or pretend all db's disappeared [23:39:46] * addshore is currently failing >.< [23:40:03] addshore: not enough beer in you [23:40:09] that's why [23:40:41] addshore: http://xkcd.com/323/ [23:40:42] :D [23:40:43] beer? whisky [23:43:02] right petan|wk I think it is doing it now ;p [23:44:06] escpecial if the volume I am looking at is /dev/mapper/vg-lvol1 [23:44:09] yes I can see :D [23:44:15] yes [23:44:20] that's db [23:44:26] :> [23:44:39] or /dev/vg/lvol1 [23:44:40] :P [23:44:50] * addshore is ahppy [23:44:53] *happy [23:44:55] :D [23:45:10] bnr1 and bsql01 :D [23:45:18] shoop da woop! [23:45:30] oh yes /db is growing [23:45:47] I am using tablespace per db [23:46:02] Isn't that standard? [23:46:04] so it shouldn't be problem to shrink dbfiles on deletions [23:46:09] Damianz: no [23:46:11] Since if you clear a table you can't shrink your ibdata files :( [23:46:19] standard is 1 monolithic datafile for everything [23:46:28] in mysql :D [23:46:31] crazy [23:46:32] * Damianz notes his standard is in puppet -D [23:47:00] I've taken a db from 800gb to 50gb by dumping and importing it due to that before now [23:47:11] heh [23:47:33] that's how I did shrink ext3 fs [23:47:42] I had to copy all files to new fs and then back [23:47:49] so that I could shrink [23:48:01] petan|wk: if only bnr1 was as big as bsql01 ;p [23:48:06] yeah... ext3 sucks like that [23:48:17] addshore: I would rather create bnr2 :P [23:48:27] mysql server is harder to exhaust than application [23:48:28] haha, understandable ;p [23:48:32] At least with ext4 it doesn't take 8894798724987249867298467246 hours to fsck when you've got tb drives [23:48:47] Damianz: with btrfs is takes 0 seconds :P [23:48:51] cuz you can fsck online [23:48:53] :D [23:49:10] I am using everywhere now [23:49:17] except for few production boxes [23:49:26] these usually run ext3 for compatibility reasons [23:49:37] Mosting still using ext3/4 atm... looking to use zfs for storage boxes though [23:49:50] zfs that is solaris fs? [23:49:54] yeah [23:49:58] yes that's cool [23:50:01] Turn atime off on ext3 and it works okish performance wise. [23:50:01] there's zfs for linux [23:50:05] err ext4 [23:50:06] similar to btrfs [23:50:22] Ryan_Lane: zfs for linux is still kinda new though... [23:50:25] but is it open source? [23:50:41] yeah - it's a port iirc [23:50:53] why in the world they started btrfs then [23:51:10] btrfs is more advanced [23:51:13] meh - let's all just use reiser fs! [23:51:27] likely yes but zfs has subvolumes and that as well [23:51:34] or I think so [23:51:36] btrfs (as the name implies) uses btrees [23:51:49] specifically a type of btree that didn't exist when zfs was written [23:51:55] ah [23:51:57] (a copy on write btree) [23:52:12] CoW gives you a few advantages [23:52:45] addshore: did you know you can run multiple mysql dumps at once :P [23:52:51] that machine is almost sleeping [23:52:57] it wants you to make it hot [23:57:47] petan|wk I dont need to [23:57:53] there is only 1 table with stuff in [23:58:06] mhm ok [23:58:08] lol :D [23:58:14] well partitioned schema [23:58:20] I think its the network thats slowing stuff down O_o or the mem on sql3 [23:58:31] 1 table which takes like 40gb and then nothing [23:58:31] :D [23:58:46] does it have 2 rows or something XD [23:58:46] yeh :P [23:58:54] no :p [23:59:04] 20000000 rows and a few fields [23:59:05] are you using indexes and keys? [23:59:06] :P [23:59:32] on that table [23:59:38] mysql is so damn simple [23:59:43] ya, and im trying to add more stuff to it, but I cant without the space to alter [23:59:50] I like how you can specify a different tablespace for indexes etc [23:59:54] on oracle