[00:00:01] * Damianz pats addsleep and un-kicks him [00:00:05] :) [00:00:11] night all :) [00:00:53] sorry for shocking you all ;p [00:03:33] Ruddy hell beetstra's bots eat a lot of bandwidth [00:03:42] how can you tell? :O [00:04:25] http://ganglia.wmflabs.org/latest/graph_all_periods.php?title=Bots+network+in&vl=&x=&n=&hreg%5B%5D=bots-.%2B&mreg%5B%5D=bytes_in>ype=stack&glegend=show&aggregate=1 < bots-liwa host the linkwatcher etc bots [00:04:54] and do all of bots instances run on virt9? [00:05:17] I think they're spread out a bit - most that traffic is to gluster [00:05:26] yep :/ [00:05:34] silly gluster [00:06:02] weird it does like 20mb of reading from disk... rather than writing and reading though [00:06:42] RX bytes:4185627807280 (4.1 TB) TX bytes:136730882171 (136.7 GB) < nice, considering it's only been up 5 days [00:07:25] haha Damianz yep! [00:08:17] anyway :P really sleeping now :p Night xD [00:09:20] * Damianz tucks addsleep in [00:09:38] do you have bandages for my bruises? ;p [00:09:55] bandages wouldn't help for bruises [00:10:12] i know :O But then I wouldnt have to look at them :P [00:10:45] * Damianz eyes the duct tape and grins [00:15:34] Damianz: Is there a way I can make sure a package is installed across all bots instances so when I use SGE it works? [00:16:14] not really right now - working in getting puppet classes merged in which we'll use for all nr instances [00:17:06] Is it just nr*, bnr* that the job will run on? [00:17:55] Oh just looks like bnr* [00:18:04] Just bnr I think [00:19:32] Could I get you to install python-requests, python-virtualenv, python-twisted on bnr2 then? [00:20:26] !log bots damian: apt-get install python-requests python-virtualenv python-twisted [00:20:28] Logged the message, Master [00:20:47] * Damianz notes to fix that script so it includes the instance name [00:32:20] I wonder, is it possible to specify a different executable depending on which server it runs on? [00:32:47] Actually, that might not be needed. [00:34:12] Ok, so is the syntax for SGE the same as the toolserver? [00:53:19] Hello, [00:53:40] I noticed the new public_html folders [00:55:22] What is the webpage that the things in public_html would be viewed at? [00:55:52] bots.wmflabs.org/~user [00:58:16] yays [01:21:55] Damianz: Could you install python-requests on bnr1 too? Apparently it doesnt have it [01:22:45] !log bots damian: apt-get install python-requests [01:22:56] ty :) [01:23:06] but we lost labs-morebots in the netsplit... [01:25:58] aw shit, its an old version of requests. [01:26:26] ugh.... [01:27:27] Damianz: sorry about all of this. Would it be possible to uninstall python-requests on bnr1,2 and run $ sudo python setup.py install in /data/project/legoktm/requests/ ? [01:27:54] the API changed with v1 onwards, and it seems ubuntu has v.0.8.2 [01:28:53] !log bots damian: apt-get remove python-requests -y [01:28:56] Logged the message, Master [01:29:07] !log bots damian: pip install requests [01:29:08] Logged the message, Master [01:29:17] ah that works too [01:37:36] it works! [01:37:40] * legoktm hugs Damianz [01:40:45] * Damianz notes legoktm has really long arms [01:50:41] :> [01:50:52] i queued like 10 jobs [01:50:57] all waiting :P [01:51:16] * Damianz tickles Ryan_Lane [02:02:20] ? [02:02:41] petan: you know coren has been setting up open grid in the tools project, right? [02:02:42] :) [02:03:29] tickles are awesome, no questioning the tickles [02:03:42] heh [02:04:03] petan, Damianz: you guys should coordinate efforts with Coren, since you're all working on similar things [02:05:14] I've never figured out why we have tools and bots, since bots have web stuff and web stuff has bots... we should just have some development project for bots/tools and some production project for running bots/tools *shrug* [02:24:43] do i only get 2 running jobs at a time? or is there another limit somewhere? [02:25:00] im not sure why only 2 are running right now... [02:26:41] Dunno how it's setup, probably 1 per host or something funny [02:27:44] 22 0.75000 am_rmv legoktm r 03/11/2013 01:36:24 main.q@bots-bnr2.pmtpa.wmflabs 1 [02:27:45] 24 0.29457 an_rmv legoktm r 03/11/2013 02:19:54 long@bots-bnr2.pmtpa.wmflabs 1 [02:27:57] and i have 25-36 all "qw" [02:37:29] http://bots.wmflabs.org/robots.txt [02:37:35] why? [02:38:25] because we never decided if allowing bots was a good idea [02:39:41] ok [02:39:52] i'll just override it [03:08:49] it's funny than we go back to the 'github vs gerrit' argument every month [03:12:49] Damianz: why not "hg vs git", just to discuss something different XD [03:13:48] There's not that much difference betwean hg and git really... [03:13:56] or "blue vs red" [03:15:36] but i must be honest, gerrit had some good layout improvments, but i still miss some code explorer feature [03:16:07] I never use to like gerrit but its workflow is pretty awesome when you get use to it [03:16:29] Damianz: coren is setting up a "tools" project that's going to be bots/tools combined [03:16:40] So I am. [03:17:06] Alchimista: use gitblit? [03:17:21] oh. did they turn that off temporarily? [03:17:25] Did tools never become webtools then? [03:17:31] Have you ever tried compiling gitblit? rofl [03:17:42] * Damianz notes to build the latest gerrit with gitblit tomorrow for his install [03:17:43] Damianz: it's a plugin for gerrit, now [03:17:56] Yeah - that plugin won't complile against stable [03:17:58] D: [03:18:01] heh [03:18:13] Damianz: Not sure I understand your question. [03:18:16] Ryan_Lane: i'm not a professional developer, it's just for fun, and in my contact to gerrit, not so permanent one, some things aren't quite easy [03:18:43] or at least percectible [03:20:17] Coren: 'tools' was going to be 'web tools' or something, that plant was doing though I think it never happended... so do we have bots and tools (which will end up being tools) or bots, webtools and tools (which will end up being tools) [03:20:20] well, gitblit will be the replacement for gitweb [03:20:31] Alchimista: and it'll be a code explorer [03:20:37] gitweb kind of is, too, but it sucks [03:21:07] yah, i've been watching the discussion, gitlib seemed a good replacement [03:21:21] Damianz: All three exist atm; tools is the ultimate destination, bots and webtools are left alone to avoid disrupting work. But eventually, there's going to be just the one (tools) once everyone moved over. [03:22:10] gitblit is awesome, just a pain to install [03:23:39] I'd like to see more integration between gitblit and gerrit [03:24:46] Coren: q about tools, which server should i submit jobs to, and do i need to ask you to install packages? or can i do it myself? [03:25:59] legoktm: atm, most of the management is manual, but I'm pretty fast about installing packages. But you don't submit jobs /to/ a server, that's what the queue system is there for. It picks. :-) You can submit jobs from -login or -webserver-* (in case you have something started by a web interface, say) [03:26:03] isnt a gitlib demo somewhere? [03:27:07] well, it was on our gerrit server [03:27:15] * Damianz wonders if he should work more on tools than bots or neither and work on the interesting stuff (underlying/supporting infrastructure) [03:27:17] Coren: Ok can I get python-twisted, python-virtualenv, python-pip, python-dev installed, then run $ pip install cython oursql requests [03:27:22] :) [03:28:26] legoktm: is oursql much more fast than sqlalch? [03:29:12] Alchimista: i've never used sqlalch, i just always use oursql for any mysql/python things. [03:31:11] so you're in my situation, i've never used oursql, i usualy use sqlAlchemy, but saw somewere that our was faster :S [03:31:45] :P [03:32:07] oursql handles unicode much better than MySQLdb, so thats why I originally switched [03:34:59] i've started using alchemy because it can manage mysql,sqlite mongo... , and specially in webtools, *prevents* sqlinjection [03:35:26] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Notepad was modified, changed by MPelletier (WMF) link https://www.mediawiki.org/w/index.php?diff=657724 edit summary: [+149] +more [03:36:36] legoktm: pip gets build fails during cython build (some missing deps). I'll look into the dependencies tomorrow and add them (and/or make a puppet Package provider for pip) [03:36:47] The others are already there. [03:36:55] Hmmmm [03:36:58] Did oursql install properly? [03:37:04] If so, then I don't need cython. [03:37:27] I thought cython was a dependency for ousql, but not always apparently. [03:37:47] legoktm: No, since it needs cython. lemme use the ubuntu-provided cython instead. [03:37:59] Ok that should be fine [03:44:19] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Notepad was modified, changed by MPelletier (WMF) link https://www.mediawiki.org/w/index.php?diff=657726 edit summary: [+41] fixy (use ubuntu's cython) [03:44:27] legoktm: u can haz oursql [03:44:37] I can haz sleepz? [03:44:39] :D [03:44:47] goodnight Coren [03:44:50] I'll try and not blow it up [03:45:04] er wait [03:45:08] No, actually, try to blow it up. We know it worked if you fail. :-) [03:45:10] Yes? [03:45:13] Did you install python-requests from apt? [03:45:15] Or pip? [03:45:18] apt [03:45:19] Because the one in apt is outdated [03:45:26] Its v0.8.2 [03:45:28] pip has v1 [03:45:51] Aaaah. [03:45:54] sleep? what is this nonsense [03:46:20] Hm. Then I *really* need to write/find a puppet package provider for pip. [03:46:40] So it'll have to wait until tomorrow if -request won't do. :-) [03:46:50] I mean 0.8.2 [03:47:00] See? See? Need sleep! [03:47:02] * Coren waves [03:47:08] v1 wasn backwards compatible so ill just wait [03:47:13] wasnt* [03:47:14] pip's in core iirc [03:47:36] Coren|Sleep: For tomorrow: http://projects.puppetlabs.com/issues/6527 [04:40:22] @notify petan [04:40:22] I doubt that anyone could have such a nick 'petan ' [04:40:28] @notify petan [04:40:28] This user is now online in #huggle so I will let you know when they show some activity (talk etc) [04:55:03] [bz] (RESOLVED - created by: Antoine "hashar" Musso, priority: High - normal) [Bug 45908] review lsearch-global.conf for beta context - https://bugzilla.wikimedia.org/show_bug.cgi?id=45908 [05:15:38] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Normal - normal) [Bug 45868] let search indexer access the squid public ip - https://bugzilla.wikimedia.org/show_bug.cgi?id=45868 [05:39:50] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Normal - major) [Bug 41132] live hack in beta mediawiki-config (tracking) - https://bugzilla.wikimedia.org/show_bug.cgi?id=41132 [05:40:02] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Normal - normal) [Bug 38995] [OPS] udp2log prevents udp2log-mw from starting - https://bugzilla.wikimedia.org/show_bug.cgi?id=38995 [05:41:03] [bz] (NEW - created by: T. Gries, priority: Unprioritized - normal) [Bug 45214] Suggestion: when installing instances, starting puppet runs etc.: ping the developer by mail about the status - https://bugzilla.wikimedia.org/show_bug.cgi?id=45214 [05:44:36] !log deployment-prep Running MediaWiki update.php on all databases [05:44:38] Logged the message, Master [07:04:49] Ryan_Lane: hi [07:05:43] I had a kind of Berlin Hackathon, and fixed many showstoppers in Extension:OpenID, have you seen ? [08:23:31] addsleep MAN [08:23:42] next time write an email before you reboot production boxes [08:23:47] ^^^ [08:23:50] hi petan [08:23:53] hi [08:24:05] i had a question about SGE [08:24:06] wtf [08:24:09] hi [08:24:19] bsql01 rebooted and yet it needs one more reboot [08:24:23] o.o [08:24:24] addsleep I wanted to install new kernel before [08:24:26] well [08:24:28] I already got it [08:24:33] just was scheduled for installation [08:24:40] for some reason only 2 jobs of mine run at a time [08:24:50] ohwut [08:24:53] legoktm but I don't see any job in queue of you [08:24:54] now 4 are running [08:25:09] what you see when you type qstat [08:25:12] http://dpaste.de/KU42g/raw/ [08:25:34] any chance we can get more resources added? [08:26:00] ganglia shows lots of unused cpu/memory [08:26:05] on both bnr1 and 2 [08:26:50] wow [08:26:53] let me check... [08:27:35] yes I see now [08:27:47] well, I just installed exec on 1 [08:27:54] maybe it's related to this, but it's weird [08:28:20] legoktm these jobs have different name all [08:28:26] why do you think they are all the same process? [08:28:39] err, they aren't? [08:28:55] an_rmw != bat_smg_rm [08:28:59] yeah [08:29:01] they're all different [08:29:08] but right now only 4 are running [08:29:09] so what's wrong [08:29:17] ah [08:29:25] yes that's true I need to find out why [08:29:32] I submitted some jobs as well and they didn't started [08:29:35] the other 4 are still queued, and there are free resources... [08:29:38] maybe there is some limit [08:29:46] earlier it was stuck at 2 [08:30:48] also not sure why some are running in the "main.q" and 2 others are in "long" [08:31:24] lol [08:31:31] there is limit in each queue [08:31:33] let me remove it [08:31:53] thanks [08:31:57] * legoktm goes to queue more jobs [08:32:30] just type qstat -j [08:32:35] you will see what is reason why it' [08:32:39] s [08:32:40] stuck [08:32:43] :o [08:32:49] ah [08:32:51] didnt know about that [08:34:33] Krenair: are you listening ? [08:34:41] lol [08:39:21] legoktm it runs now [08:39:34] woo :D [08:39:37] all my jobs are running [08:40:59] Coren|Sleep hi [08:41:05] Coren|Sleep is there any nice documentation for SGE? [08:52:01] addsleep never ever reboot sql server without consulting with others [08:54:13] Damianz did you restart your db import? [08:54:20] petan: It's done [08:54:21] or u gave up [08:54:23] ah fine [08:54:41] so cb is now running or still some imports? [08:54:50] legoktm what about your sql import [08:54:54] is it working? [08:56:20] well it got killed after the servers got rebooted [08:56:21] ill restart it in a bit [08:57:35] okay... [09:23:12] !log bots petrb: addshore to motd so that people know who to blame :) [09:23:14] Logged the message, Master [10:01:05] petan: running [10:01:21] yay [10:01:33] Damianz btw you have some experiences with sge? [10:01:40] I am trying to find a manual for queue parameters [10:02:00] Not really [10:04:29] petan: did you try https://wiki.toolserver.org/view/Job_scheduling ? [10:04:39] Darkdadaah I like sge more [10:05:04] or is that sge as well? [10:05:07] petan: that is SGE [10:05:10] oh lol [10:05:11] Well, yes. [10:05:29] but that's for end users [10:05:37] I need some manual for sge admins who creates queues [10:05:53] there is ton of parameters for each queue but no manual descibing them [10:06:10] Oh, then I can't help much :P [10:06:20] right now I am guessing instead of knowing what I do :P [10:06:54] Maybe ask DaB on the toolserver channel [10:07:30] Hm he's not there yet... [10:07:59] Merlissimo knows quite a bit about queues [10:08:08] what command are you using? [10:08:12] qconf [10:08:19] qconf -mq [10:08:36] it open vi editor with text file containing many lines - each line is one parameter [10:08:42] with no description of what they mean [10:08:48] I guess from name [10:09:13] for example I would like to find out how to allow job in long queue to run infinitely long [10:09:27] I belive it kills the job after some time [10:09:52] did you see qconf(1) and queue_conf(5) ? [10:09:54] yes [10:10:06] it contains parameters of qconf but not of that text file which gets opened [10:10:45] yay [10:10:47] man queue_conf :D [10:10:49] here we go [10:11:56] :) [10:22:39] legoktm I hope your job return 0 [10:22:41] as exit code [10:22:48] otherwise it will be restarted in long queue [10:22:48] they should [10:22:50] ok [10:23:05] long queue should be probably used for permanent jobs only [10:23:11] but we could change that behaviour [10:23:17] mine are right now set with a runtime of 7 days [10:23:19] we can create more queues for multiple purposes [10:23:25] ok [10:23:47] most will finish after 2-3 hours, some will run for a few days [10:23:52] i just gave it 7 days to be safe [10:25:32] ok [10:25:52] error: commlib error: access denied (client IP resolved to host name "bots-gs". This is not identical to clients host name "bots-gs.pmtpa.wmflabs") [10:25:52] Unable to run job: unable to send message to qmaster using port 6444 on host "bots-gs.pmtpa.wmflabs": got send error. [10:25:52] Exiting. [10:26:20] petan: ^ [10:27:00] legoktm@bots-gs:/data/project/legoktm/wd_removals$ qstat [10:27:00] error: commlib error: access denied (client IP resolved to host name "bots-gs". This is not identical to clients host name "bots-gs.pmtpa.wmflabs") [10:27:00] error: unable to send message to qmaster using port 6444 on host "bots-gs.pmtpa.wmflabs": got send error [10:29:15] yes, I see [10:29:17] fixed [10:29:29] I cleaned some mess in config and forgot to update hosts [10:29:44] works now, thanks [10:54:13] $ git pull --rebase [10:54:14] First, rewinding head to replay your work on top of it... [10:54:15] Applying: shit [10:54:16] XD [10:54:30] hehe [11:26:31] legoktm: can I point you to http://www.wikidata.org/wiki/User_talk:Addshore#(Can you help me?) (I am guessing you probably know the answer [11:26:38] looking [11:27:56] done [11:28:09] cheers :) [11:34:50] !logs [11:34:50] logs http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs [11:35:52] something somewhere has changed in the system that brings down my bot onbots- liwa ... while I was not there .. [11:36:01] communication with bots-sql2? [11:36:58] errr [11:37:08] that no longer exists i think. [11:37:12] everything got migrated into bots-bsql01 [11:37:25] kay! [11:37:27] fun .. [11:37:40] everything got migrated .. including my dbś ?? [11:37:50] and why do my bots not complain about not being able to connect to .. oh .. [11:37:53] i think so…? [11:38:12] I am on bots_sql2 .. the instance exists .. but is defunct .. [11:38:15] or something [11:38:22] * Beetstra investigates bots-bsql01 [11:39:15] * Beetstra curses [11:40:49] the 'i think so...?' .. maybe not - I don't seem to have an account on bsql01 [11:40:49] ? [11:40:54] oh [11:40:56] addsleep: ^ [11:40:59] give Beetstra an account [11:41:08] I mean .. in mysql .. [11:41:14] yeah [11:41:20] addsleep had to manually create mine too [11:41:31] oh .. aargh [11:41:52] and then I will probably also have to manually move the db from sql2 .. which is going to take ages [11:42:10] i thought petan had imported them [11:42:37] sorry i literally have to go now, how big is the db, shouldnt take that long, i will be back in a few hours, there should be a cron which will run and create your account, ttyl [11:42:59] OK [11:44:02] By then, I wilill then probably see you tomorrow [11:47:38] In case you see it .. I need the databases 'coibot', 'linkwatcher', and 'monitorbot' (including the accounts that can access them [11:50:40] * legoktm checks [11:51:44] and the linkwatcher db is huge, really huge [12:03:24] Beetstra nooo [12:03:32] Beetstra I don't think anyone migrated your bot [12:03:36] bots-sql2 exist for sure [12:03:49] Yeah, but where is then my problem .. :-( [12:04:06] what the problem is? [12:04:16] Suddenly the boxes can't keep up anymore [12:04:19] it worked fine just yesterday [12:04:23] nah [12:04:24] no [12:04:36] ok, Beetstra first of all, where your bots run? [12:04:36] instance name [12:04:40] bots-liwa and bots-nr1 [12:04:43] ok [12:04:58] can you move it from bots-nr1 either to gse or some bots-bnrX instances? [12:05:05] I recommend gse [12:05:18] it would be also cool to move your db's to bots-bsql01 [12:05:24] because it's faster than -sql2 [12:05:30] however you can keep it on -sql2 for now [12:05:33] ... or tools- [12:05:33] hi Coren [12:05:38] Coren read mails pls [12:05:40] :-) [12:05:40] the bot on bots-liwa constantly saves backlog files, as if it is too busy with something else [12:05:46] Hi Coren [12:05:47] Coren btw you have sql server on tools? [12:05:59] Ayup [12:06:12] Coren ok then proceed to reading mails :P [12:06:32] Beetstra okay, so the problem now is that you can't connect to sql2? [12:06:42] no, it seems to be connected [12:06:53] ok, so what is problem :P [12:06:56] Actually, I don't think that that is the problem .. except maybe that it stores to slow .. [12:07:10] what I say .. the bot was running fine until two days ago .. now it can't keep up at all [12:07:12] aha, the performance might be problem, that's why I suggest moving to bsql01 [12:07:19] uh [12:07:31] the sql server is not loaded at all [12:07:34] either the performance of sql2, or of the boxes itself [12:07:39] Yeah, but the instances seem to be [12:07:42] let me check the instance [12:07:46] But that was never a problem [12:08:05] yes liwa is kind of loaded [12:08:12] glusterfs is fucked [12:08:17] I think you should reboot it [12:08:21] that will fix it for sure [12:08:30] hmm .. could try .. [12:08:41] * Beetstra should figure out how to reboot it .. [12:08:52] first: shut down all your processes [12:08:55] then sudo reboot [12:09:14] but, I recommend you to move your bot to sge anyway :P unless you require a dedicated instance only for you [12:09:20] petan: Ima gonna reply on -labs, but you're missing the point. :-) (1) Most of tools- will be self-serve (i.e.: doesn't need sysadmin intervention), and (2) the objective is to also have volunteer roots eventually. [12:09:47] Coren well, so far you have none. There is always need for interventions [12:09:52] We earlier made the choice to have liwa on an own instance .. [12:10:05] I am not allowed to sudo on bots-liwa [12:10:05] machines are crashing, software needs to be updated or reinstalled [12:10:05] gluster is breaking [12:10:11] Coren ^ and tons of more issues [12:10:19] I was fixing stuff whole weekend [12:10:32] petan: In all fairness, gluster hasn't broken since the upgrade. [12:10:47] Coren well, it did broke like 2 times just yesterday [12:10:55] unless the upgrade was few minutes ago, I doubt [12:10:59] petan: On bots-? [12:11:01] yes [12:11:06] brain splits [12:11:19] Beetstra really? [12:11:24] Beetstra ok I will reboot it then [12:11:30] Beetstra let me know when I can do that [12:11:30] yep [12:11:37] And I thought I had root on that box ... [12:11:40] oh yes I know why you can't now [12:11:47] you can now .. [12:11:54] petan: Ah. No matter anyways: Gluster is gonna go away. [12:11:55] it's because of new sql server, I can give you root back but I will remove /mnt/secure [12:12:24] Coren whatever is going to replace it - you will still need some sysadmins who are available 24*7 every day for assistance to users [12:12:41] wmf-based project will always fail to meet that criteria [12:13:04] take a look at wikipedia - that is managed by volunteers, you will always find some wikipedia admin online any day and hour [12:13:50] !log bots petrb: rebooting -liwa per request [12:13:52]