[01:23:10] rsmhhhm [01:27:20] petan: is there by any chance a db backup of bsql01 that runs at 1am? [01:27:20] / midnight [01:27:36] yes [01:28:01] Im guessing that is why bsql01 seems to have ground to a halt ;p [01:28:23] ignoring that it's half 2 possibly [01:29:54] It ran 29 mins ago and finished 9 mins ago :) [01:30:34] mhm, it's not running now but the cpu is insane from your queries :P [01:30:35] but it seemed to cause every process I had running on the gridengine to stop [01:30:50] http://bots.wmflabs.org/~addbot/status [01:31:26] the gap from all of my log points started 5 mins after the db backup started and didnt manage to start again until just now [01:31:52] hmm, I didn't think the backup took that long [01:31:57] well [01:32:06] addshore's dbs are pretty big now [01:32:23] * Damianz blames addshore [01:32:38] they are a third of the size that they used to, just having to use some very inefficient queries right now [01:32:42] not what the db was designed for [01:32:58] We should look at using percona tools for backups or enabling binlogs and taking the dump off a slave [01:33:03] its almost like all of my processes paused for 15 mins.. [01:33:17] Damianz: yep, we definatly need to make a slave [01:33:24] your queries take like 15-30 seconds to run normally lol [01:33:34] well according to mysql anyway... not the most reliable dude [01:34:03] so odd [01:34:04] http://en.wikipedia.org/wiki/Special:Contributions/Addbot [01:34:13] literally everything everywhere just stopped for 15 mins.. [01:34:15] XtraBackup is suppose to be a hot copy... never actually tried it though hmm [01:34:24] * Damianz notes to think about a slave at some point [01:36:23] Damianz: most of my queries finish in a fraction of a second [01:36:42] except for my 40 processes that are currently having to do inefficient queries :P [01:36:51] actually, they do now [01:37:18] Bunch of inserts where at like 14, but selects are under a second now [01:37:22] just lots of them [01:37:25] yay buffers [01:37:32] where are you seeing this? :P [01:37:50] every massivly slowed down in the last 30 mins, but has suddenly got fast again [01:38:07] i.e. a script was taking 10 mins to run that normally runs in about 30 seconds xD [01:38:56] process on the server, time not 100% accurate as far as query goes but meh if I'm running explain on it [01:39:34] HAHA Damianz take a look at the top left graph on http://bots.wmflabs.org/~addbot/status , shows that 150k worth of processing seconds all just finished at the same time xd [01:40:04] lol [01:40:16] makes sense if they where hung on sql [01:40:21] yep :P [01:40:37] sill sql [01:40:37] I like those graphs but I think I'd still like a general graphite server around lol [01:41:05] also, can you get to google? ;p [01:42:29] office is a bit far at this time in the morning :P [01:43:08] somethign is rather messed up with my dns atm :/ [01:43:25] 8.8.8.8 8.8.4.4? [01:47:15] okay it is just generally my internet.. [01:52:04] wonder how fast i can make this :P [01:53:52] * addshore has come up with a brilliant plan [04:03:03] perfect :) [07:46:54] Change on 12mediawiki a page Wikimedia Labs/Tool Labs was modified, changed by Nemo bis link https://www.mediawiki.org/w/index.php?diff=668975 edit summary: [+106] And now also the third wiki joined us. [08:26:46] Damianz we HAVE binlogs :D :D [08:26:58] yes we can make backups from them [08:47:00] hi petan [08:47:07] heya [08:47:20] is there a place on bots lab where I can store private files and shared among nr instances? [08:47:31] yes [08:47:39] /mnt/secure [08:48:02] okay .. any docs page about similar stuff? [08:48:09] let me check... [08:48:15] so I don't need to bother you next time :) [08:48:24] however - since all instances are no-root atm basically even your home is safe [08:48:37] !botsdocs [08:48:37] https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots/Documentation [08:49:33] They can write to shared gluster storage at /data/project or shared NFS server at /mnt/secure [08:49:45] see Architecture [08:49:58] is gluster considered less stable? [08:50:05] yes [08:50:09] significantly [08:50:52] but /mnt/secure is place where mysql passwords are stored so it's considered "most secure" place [08:52:24] then any use case of gluster? huge files of large amount of small files ? [08:52:48] yes it's good for any data which aren't small, /mnt/secure is tiny [08:53:16] so anything but password files should live on gluster I guess... or /mnt/share which is local only [08:55:01] what about coredumps, if it accidentally happens ... :p [08:59:23] I don't expect password files to generate coredumps :P [08:59:40] anyway - if it happened worst that can happen is that the volume become full ;) [09:03:40] I mean .. apps generates dumps containing passwords in it [09:04:41] or should we always follow the pattern to destroy password in memory after using it .. [09:05:15] I might be thinking too much [09:07:51] liangent not all application do that, and - depends where your app run, even /data/project, /mnt/share are secure, nobody has root anymore [09:08:09] that /mnt/secure is from times where root boxes existed [09:08:26] now it's kept just because gluster is not reliable [09:08:39] and because the mysql passwords were generated there [09:09:22] you can even run your bot on tools project, but that doesn't have reliable sql server yet [09:09:40] and it's still using gluster as master storage [09:10:43] when was those root boxes removed? just for curiosity .. and they'll never appear again? [09:15:16] the boxes weren't removed but the root access was removed [09:15:29] some of them were removed of course, bots-2 and bots-3 [09:15:40] it was all announced in many emails [09:15:54] the reason is that tools and bots projects are going to be merged at some point [09:16:04] the bots project will be likely a staging area for tools [09:16:18] that means it should be designed in a similar way as tools project is [09:22:47] of course it's not going to be restricted so much, so basically anyone trusted can request root there... [09:22:59] but it's not going to be default for everyone [10:47:59] petan: did we ever get anything like qcronsub on bots? [10:48:04] so we dont duplicate jobs [10:55:31] legoktm: Maybe you could speak with Coren. He has created a tool for tools SGE, maybe petan could use it on Labs, too. [10:55:54] yeah i know tools has it, i was just wondering about bots [10:56:01] since it was discussed earlier [10:58:43] legoktm: Coren has a full-time-job to work on tools, petan do this when he has time [10:58:56] i know [10:59:41] legoktm: And sge is sometimes very ugly, this can need time ;-) [11:00:31] heh :P [11:26:59] hey I am back legoktm [11:27:07] hey [11:27:22] no there is no qcronsub in this moment but if that jsub is enough I can import it from tools [11:27:36] actually we might want to make a package from it [11:27:47] Coren: ping [11:27:54] no rush though :) [11:28:13] Coren: what about making a package for all these tools we have on tools [11:28:21] * tools / utilities whatever [11:28:27] like qtop etc [11:29:39] qtop? [11:29:51] as soon as there is a good sql server and stable filesystem on tools project I think we could be ready to start copying bots from tools, so that they exist on both [11:30:01] legoktm type "qtop" and you will see [11:30:06] * legoktm looks [11:30:31] oh cooool [11:30:38] it exist on tools and bots [11:30:48] was created by addshore and bit enhanced [11:31:09] !addshore [11:31:09] lolz [11:31:29] wm-bot: yay [11:31:29] Hi petan, there is some error, I am a stupid bot and I am not intelligent enough to hold a conversation with you :-) [12:12:07] heh, Coren I am wondering how are you going to run beetstra bot which need 18gb of virtual memory [12:12:23] in fact it uses some 3gb of resident [12:12:30] so it runs on bots-bnr3 [12:12:39] but... not even possible to make instance with so much ram [12:12:44] :) [12:13:03] maybe there is some option for perl so it eats less [12:13:12] @notify Beetstra [12:13:12] I will notify you, when I see Beetstra around here [12:13:21] Coren-the-perl-master will just fix it :P [12:13:33] ok but he can't just go and fix all bots... [12:13:42] I suppose there will many of them not just in perl [12:13:53] mono and java is having same problems as perl [12:14:12] @notify mabdul [12:14:12] I will notify you, when I see mabdul around here [12:14:28] I wish this bot could be in all wikimedia channels... [12:27:35] petan, 18GB of memory? [12:27:35] what the hell does that bot do? [12:27:40] Krenair virtual memory [12:27:50] Krenair it eats 2gb of resident [12:27:56] Krenair that is very normal amount for memory [12:28:10] for example my IDE eats also about 8gb of virtual memory [12:28:14] that is not REAL memory [12:28:36] 3gb rss for a bot? [12:28:38] basically every java / python etc application eats shittons of virtual memory [12:28:39] you're doing it wrong [12:28:48] paravoid maybe, but I didn't write that bot :> [12:28:55] btw it's doing some huge tasks [12:28:59] afaik [12:29:07] Beetstra would know more [12:29:18] I believe there is a leak in code [12:29:27] because it seems to keep allocating more and more [12:32:37] even wm-bot is eating 320mb of virtual memory [12:32:46] and I am trying to make it eat as less memory as possible [12:32:55] it eats less than 100mb of res [12:33:08] though it's still a lot for irc bot [12:33:22] but it's doing a lot of stuff as well [12:55:48] petan: Yeah, packaging the local/bin tools is on my todo, but this week my priority is on the gluster-replacement. [12:57:51] * Damianz pats Coren [13:02:49] ok [13:03:18] Coren I could actually make that package but I am not sure about the dependencies and location of all these scripts, also do you want to make package per script or 1 big package [13:03:39] what about manual pages? :P [13:03:44] we should make some [13:03:49] :)) [13:04:20] I made manual page even for my mono GUI application launcher even if it consisted of 2 lines :D [13:04:28] petan: Well j* would make one reasonable package (jobutils?); they're all in /usr/local/bin and have only perl and gridengine as dependencies. [13:04:40] ok [13:05:12] petan: I agree about man page; but it should be easy to make: all of the j* have a detailed usage when you invoke them without parameters. [13:05:20] ok [13:05:38] BTW, high-performance NFS server is working well in eqiad now. [13:06:48] man page? meh use perldoc [13:07:12] I think that control file of package should be actually somewhere in git [13:07:22] so that we can easily rebuild it [13:07:33] maybe whole tarball / structure [13:08:05] petan: I've been meaning to greate a labs git for config, tools, etc. Lemme do that now. [13:09:25] ok [13:10:40] version 1.0.0? [13:10:41] :P [13:12:19] anything to be executed in preinstall? [13:12:23] I don't think so [13:14:52] petan: http://en.wikipedia.org/wiki/User_talk:Coren [13:14:59] copypasta fail [13:15:04] :P [13:15:16] git clone https://gerrit.wikimedia.org/r/labs/toollabs [13:16:04] ok [13:16:14] btw what license you want for manual page? [13:16:21] Licensed under BSD-like License. [13:16:23] that's what I use [13:17:33] petan: is there a way to make wm-bot's infobox case-insensitive? [13:17:39] infobot* [13:17:41] yes [13:18:03] http://meta.wikimedia.org/wiki/WM-Bot#.40configure [13:18:11] infobot-case bool whether keys are case sensitive (default is true) [13:18:24] ty [13:18:26] petan: I think we want CC-BY-SA for documentation. [13:18:28] yu just do [13:18:35] Value true was stored into infobot-case to config [13:18:40] legoktm ^ [13:19:01] yup [13:19:02] thanks [13:20:31] Coren no manual for jstop :( [13:20:37] it just produces error with no parameter [13:21:01] petan: Ah. I suck. It's trivial: usage: jstop [13:21:08] deletes the task. [13:21:09] ok [13:21:09] :-) [13:21:35] Should prolly add usage too. I'll copypaste from your man page. :-) [13:25:56] ok Coren can you give me read / write git url for that new repo :D [13:26:03] https is read only [13:26:05] petan: For code I've written, I've asked legal and they have no boilerplate or requirement; it's work-for-hire though so the holder is the Foundation. GPL. [13:26:22] k [13:26:58] ssh://petrb@gerrit.wikimedia.org:29418/labs/toollabs [13:26:59] ssh://petrb@gerrit.wikimedia.org/r/labs/toollabs doesn't work :( [13:27:01] aha [13:27:02] port [13:27:20] And no /r/ [13:27:25] aha [13:27:57] make a directory like 'tools' to stuff tools in. :-) [13:28:05] I wish -v switch in git was useful [13:28:17] I'll probably add a /config/ and such [13:28:22] I hate how it write cloning... and all I know is how fast cursor is blinking [13:30:13] petanb@linux-dev:~$ git clone ssh://petrb@gerrit.wikimedia.org:29418/labs/toollabs -vvvvvvvv [13:30:14] Cloning into 'toollabs'... [13:30:15] ssh: connect to host gerrit.wikimedia.org port 29418: Connection timed out [13:30:16] fatal: The remote end hung up unexpectedly [13:30:40] ohhhhh [13:30:41] nvm [13:30:42] lol [13:30:46] I am in office [13:30:49] fucking firewall [13:30:54] * petan hacks [13:34:47] ok Coren now tell me how can I push into that :P [13:34:51] git push doesn't work [13:35:02] To ssh://petrb@gerrit.wikimedia.org:29418/labs/toollabs [13:35:03] ! [remote rejected] master -> master (prohibited by Gerrit) [13:35:11] You need to git review. gerrit workflow. :-) [13:35:13] * petan slaps gerrit [13:35:23] so I type "git review" ? [13:35:32] I do stuff otherwise in labs [13:35:39] I type git push-for-review-production [13:35:48] and on github it works [13:35:57] Yep. If that don't work, you just need to apt-get git-review [13:35:58] I am trying to avoid gerrit as much as I can [13:36:26] petan: Get used to gerrit, all of WMF-related stuff lives there. It's not so bad once you get use to it, and it has a good workflow [13:36:36] ok there is no git-review for this version of ubuntu :P [13:36:41] damn [13:36:42] o_O [13:36:47] What version are you using? [13:36:54] 10.04 LTS it's not really my pc [13:37:13] I decided to override the firewall in office just by sshing to one of my outside servers and push through that [13:37:18] I was pretty sure git-review was in Lucid. Odd. [13:37:18] I should have make a tunnel [13:37:53] petan: Ah, it's in /universe/ though. [13:38:11] maybe there is some no-debian package? or maybe tar.gz of that [13:38:31] I would prefer version I can just install for my user and no system wide [13:38:43] Lemme look it up. [13:39:19] I found this http://linux.softpedia.com/get/Programming/Version-Control/git-review-74740.shtml but reliabilty is questionable [13:39:23] https://pypi.python.org/pypi/git-review [13:39:36] yay [13:39:38] Looks like there's a tarball. [13:41:49] LOL it's using some python modules I don't have.. gonna find them too... [13:42:12] yup [13:42:18] these are in this one [13:42:23] repository [13:42:47] petanb@srv:~/toollabs$ git review [13:42:48] No '.gitreview' file found in this repository. [13:42:49] We don't know where your gerrit is. Please manually create [13:42:50] a remote named gerrit and try again. [13:43:08] what should be there? [13:43:33] demon really needs to make a guide for people who aren't running latest ubuntu or ubuntu at all [13:45:13] Coren do you have that file? [13:47:43] [gerrit] [13:47:43] host=gerrit.wikimedia.org [13:47:43] port=29418 [13:47:43] project=labs/toollabs.git [13:48:17] It would have been created automatically, IIRC, if you had git-review installed before you cloned. [13:49:03] New patchset: Petrb; "inserted a package structure for jobutils" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/57740 [13:50:10] petan: Did you try the packaging? [13:50:20] not yet [13:50:26] is that ok? [13:50:30] the control file and manual pages [13:50:38] category etc [13:50:58] tbh I think I will need to create blank preinst too [13:51:23] I could +2 it now, but I'd prefer you test that the package is buildable before. [13:51:43] You can amend the commit [13:51:55] sec [13:52:37] New review: coren; "As best practice, we should only merge completed packages that are known to build." [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/57740 [13:52:42] I probably need to install some tools to tools-login to be able to make package there [13:52:51] fakeroot etc [13:53:25] If you install packages on -login, make sure you add them to the list at http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Notepad [13:53:40] in the dev section [13:53:53] btw, I forgot a comma :D [13:54:11] That's why it's better to test the actual build. :-) [13:55:26] also, this of course is not buildeable just as it is - manpages should be gziped before [13:55:33] but I don't want to commit gzipped files to git [13:55:57] * Coren wonders. [13:56:08] I though there was a way to have the build gzip them. [13:56:44] The simplest thing, of course, is to have a Makefile that does exactly that. [13:56:47] ok it's built [13:56:52] so except the comma it's ok [13:57:08] hm... or maybe some build shell script [13:57:13] Coren: are you still OS on enwiki? [13:57:21] legoktm: Just CU [13:57:23] gah [13:58:35] !log bots petrb: install jobutils.deb [13:58:38] Logged the message, Master [13:58:54] LOL [13:58:59] installation took like 0.2 sec [13:59:21] and your name is wrong :D [13:59:24] ok 2 mistakes [13:59:45] petan: The "right" way to do this is to make sure that if you clone the repo, go into tools/jobutils, and run debuild, you get a .deb. :-) [14:00:41] petan: Yeah, My name is "Marc-André Pelletier" :-) [14:00:46] aha [14:00:52] yet another fix then... [14:01:33] Once you're done with the fixes, you can simply 'git commit --amend' and 'git review' again, it'll amend your changeset [14:03:26] ok I forgot to do that before so now I have 2 commits let's try... [14:03:32] New patchset: Petrb; "mark > marc + comma" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/57742 [14:06:59] OMG [14:07:03] haha [14:07:11] this is silly mistake really [14:07:45] Gerrit takes some getting used to; I think I did this at least twice. :-) [14:08:06] no I mean this what I am about to commit [14:08:18] New patchset: Petrb; "ok, and these manual pages of course should be in usr as well" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/57743 [14:08:52] Urk. [14:08:59] Now you have *3* changesets. :-) [14:09:12] but each is about something else [14:09:32] I can't just ammed 3 times 1 commit - where should I write what I change? [14:10:00] It's not important, because it's only the final /corrected/ changeset that gets merged. [14:10:06] wait [14:10:11] I still don't know if it's correct [14:11:14] yay [14:11:16] now it works :D [14:11:20] "Add the jobutils package" is one *changeset*; you amend it until it's right, and we merge that. The workflow allows you to see all the intermediate changes in gerrit. :-) [14:11:37] yes, but with no "commit message" [14:11:43] because I ammend still the same message [14:11:52] so I see that there were some extra changes but I don't see why [14:12:01] Well, you can tweak the commit message if you want; or comment in gerrit itself. [14:12:25] that would be ugly long and boring commit message containing informations for 3 commits... imagine I ammended 540 times :P [14:12:36] The problem with doing it with distinct changeset is that forces us to merge in a changeset we /know/ is wrong. [14:12:41] this way we could make whole software with 1 commit [14:12:56] ah [14:13:06] petan: But why would you want to amend your commit message? You are trying to do one thing; the small fixes that go in before merge are not interesting in the changelog. [14:13:35] mhm [14:13:45] What's important is describing what is being merged. We're just merging the final version of the changeset. :-) [14:14:02] ok you can consider the last commit to be final :P [14:14:54] Allright, what you need to do now is abandon your last two changesets, then squash your three commits. [14:15:05] You know how to squash? [14:15:07] no [14:15:29] You do: git rebase -i [14:15:38] btw it took me 10 minutes to make package and almost 1 hour to get it into gerrit :P there is clearly something overcomplicated on gerrit [14:16:02] Then you change your fixes in the editor from 'pick' to 'fixup' [14:16:15] petan: It's not complicated, it's just different and you're not used to it yet. :-) [14:16:25] git rebase -i print out a lot of stuff [14:16:35] usage: git-rebase [-i] [options] [--] [] [14:16:36] or: git-rebase [-i] (--continue | --abort | --skip) [14:17:44] Oh, sorry, thought you'd have known. The exact command you need is $ git rebase -i HEAD~3 [14:17:50] To edit your last 3 commits. [14:18:16] BTW, you normally don't have to do this; that's just to merge all your commits manually. :-) [14:19:15] Once you're done, you should have just the one commit left with all the changes. 'git review' that and it will update the original changeset on gerrit. [14:20:08] Change abandoned: coren; "Explained to Petr how amending changesets worked. :-)" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/57743 [14:20:17] Change abandoned: coren; "Explained to Petr how amending changesets worked. :-)" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/57742 [14:20:54] The workflow is a bit different from github's, but it works well once you're used to it. [14:21:01] addshore PROBLEM Current Load is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CRITICAL - load average: 31.81, 21.21, 19.54 [14:21:14] addshore can you check what's going on? [14:22:30] Coren I have no idea why but after doing git rebase all files just disappeared [14:22:39] ironically [14:22:40] [08:39:50 AM] also legoktm I finslly made my bot determine the number of articles it is going to try to take from the DB dependant upon the current number of queued db requests :P SO no more breaking the cluster :D [14:23:52] petan: In the editor, you had three lines - one for each commit, right? [14:24:04] Coren yes I had, but now it's all gone :( [14:24:06] petan: You changed the 'pick' from the first two to 'fixup'? [14:24:18] no I changed all to fixup [14:24:45] should I change just 2 of 3? o.o [14:24:55] ... oh. That's why I said " Then you change your fixes in the editor from 'pick' to 'fixup'" Only the fixes. :-) Now you squashed even the combined patch. [14:25:08] btw that effectively deleted all my commits [14:25:30] Right, because you squished all of them instead of the last two in the first. :-) [14:25:38] I should have make a backup... evil git [14:25:54] this could never happen with svn... [14:26:04] Git takes some getting used to. You can cherry pick your changes back off gerrit though. [14:26:14] well but now it's gone so no idea [14:26:25] it was only working copy I had [14:26:31] petan: Here's what you do: [14:26:48] (a) do a git status. Make sure you are at head. [14:27:20] petanb@srv:~/toollabs$ git status [14:27:21] # Not currently on any branch. [14:27:59] kk [14:28:11] git fetch ssh://petrb@gerrit.wikimedia.org:29418/labs/toollabs refs/changes/43/57743/1 && git cherry-pick FETCH_HEAD [14:28:50] Wait, that's the wrong one to start with. [14:29:29] git fetch ssh://petrb@gerrit.wikimedia.org:29418/labs/toollabs refs/changes/43/57740/1 && git cherry-pick FETCH_HEAD [14:29:36] git fetch ssh://petrb@gerrit.wikimedia.org:29418/labs/toollabs refs/changes/43/57742/1 && git cherry-pick FETCH_HEAD [14:29:39] git fetch ssh://petrb@gerrit.wikimedia.org:29418/labs/toollabs refs/changes/43/57743/1 && git cherry-pick FETCH_HEAD [14:29:52] That will cherry pick all three of your changes [14:30:36] It would, if I didn't edit them wrong. *sigh* [14:30:47] petanb@srv:~/toollabs$ git fetch ssh://petrb@gerrit.wikimedia.org:29418/labs/toollabs refs/changes/43/57740/1 && git cherry-pick FETCH_HEAD [14:30:48] fatal: Couldn't find remote ref refs/changes/43/57740/1 [14:30:55] git fetch ssh://petrb@gerrit.wikimedia.org:29418/labs/toollabs refs/changes/40/57740/1 && git cherry-pick FETCH_HEAD [14:31:14] git fetch ssh://petrb@gerrit.wikimedia.org:29418/labs/toollabs refs/changes/42/57742/1 && git cherry-pick FETCH_HEAD [14:31:31] git fetch ssh://petrb@gerrit.wikimedia.org:29418/labs/toollabs refs/changes/43/57743/1 && git cherry-pick FETCH_HEAD [14:31:51] Now you're back before your rebase. [14:31:52] :-) [14:31:57] ok [14:32:09]