[00:00:05] for number of instances, anyway [00:00:11] Nagios needs to diaf and be re-born as a princess [00:00:23] well, we're actually using icinga now, I think [00:00:46] btw: https://labsconsole.wikimedia.org/wiki/Special:Ask/-5B-5BCategory:Shell-20Access-20Requests-5D-5D-5B-5BIs-20Completed::false-5D-5D/-3FShell-20Justification/-3FModification-20date/format%3Drss/sort%3DModification-20date/order%3Dasc/headers%3Dshow/link%3Dall/searchlabel%3DOutstanding-20Requests/default%3D(no-20outstanding-20requests) [00:01:01] shell requests can be subscribed to via rss :) [00:01:05] I was thinking nagios.wmflabs but yeah it should go to icinga too [00:02:05] it's nice to have the infrastructure stable for a while. it lets me actually do work :) [00:02:43] Hmm I really want to switch this over to my version of the rebuild script but it needs more work first and someone else to check it's sane. [00:03:05] heh [00:04:08] hm. I'm going to enable scribunto [00:04:34] actually, before I go and change the main page back to sanity, I'll send petan an email asking him if there was a particular reason he did it like that - don't want to step on anyone's toes [00:04:41] Feels latin [00:04:53] Thehelpfulone: Stole from en mainpage [00:05:01] oh just that? [00:05:11] silly enwiki [00:05:14] * Damianz shrugs, that's his comment in one of the transcludes [00:06:51] Ryan_Lane, is there anyway to watchlist https://labsconsole.wikimedia.org/wiki/Special:Ask/-5B-5BCategory:Shell-20Access-20Requests-5D-5D-5B-5BIs-20Completed::false-5D-5D/-3FShell-20Justification/-3FModification-20date/format%3Dbroadtable/sort%3DModification-20date/order%3Dasc/headers%3Dshow/searchlabel%3DOutstanding-20Requests/default%3D(no-20outstanding-20requests)/offset%3D0 [00:11:47] Thehelpfulone: dunno if you can watchlist ti [00:11:48] *it [00:11:51] but you can rss feed it [00:12:03] every request is a new page [00:13:21] i like this extension http://www.mediawiki.org/wiki/Extension:WikiFeeds [00:13:39] it allows feeds "per category" [00:13:43] yeah [00:13:50] you can do that with SMW, though, too [00:13:53] ugh, of course it has a SQL injection warning on it:) [00:14:26] with SMW, you can do: {{#ask:[[Category:]]|format=rss}} [00:14:37] ah, cool [00:14:44] it's likely best to add a sort condition on modification date, though [00:15:22] Bleh, it's 1am and I'm halfway though the first email on my todo list [00:15:34] heh [00:15:36] that's rough [00:15:56] ok. captcha extensions. we'll need that [00:16:05] I haven't tested account creation with it, thougj [00:16:09] guess I'll need to do that [00:21:01] Is it too cheeky to imply you'd happily stay over a couple of days to 'get acquainted with the city' if a company is already paying for you to fly out an interview with them? [00:21:13] heh [00:21:37] well, usually you ask them to extend the flight date and pay for the extra days [00:21:45] but no, that's not abnormal [00:21:51] you need to know if you'd move to the city [00:23:09] Mhm [00:23:40] I like new places, usually get funny looks turning up with a camera and laptop in hand though o.0 [01:18:13] seems confirmedit works perfectly fine with ldap auth, openstackmanager and oath [01:18:19] I'm actually surprised [01:23:31] :) Now we just need a captcha that is like Asirra but has unicorns/horses instead of cats/dogs :) [01:24:46] heh [01:24:58] I'm using the same one that we use on the sites ;) [01:31:07] well, with that I think we're ready to open registration [01:35:30] We really should puppetize labs wiki config [01:38:11] I opened a bug about that today [01:38:15] it was a duplicate [01:38:19] I just marked it as so :) [01:38:24] heh [04:15:35] Change on 12mediawiki a page Developer access was modified, changed by LarryGilbert link https://www.mediawiki.org/w/index.php?diff=580530 edit summary: [06:05:13] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=580536 edit summary: /* User:LarryGilbert */ done [12:16:34] Change on 12mediawiki a page Developer access was modified, changed by Bryan link https://www.mediawiki.org/w/index.php?diff=580651 edit summary: [18:05:08] <^demon> https://gerrit.wikimedia.org/r/#/c/22924/ - this needs to go in right now. `git gc` (possibly) hosed operations/mediawiki-config last night, and I don't want it to run again until I figure out wtf happened. [18:05:25] <^demon> (Well, the repo *was* hosed, I just haven't determined why yet) [18:05:47] <^demon> Ryan_Lane: That was all for you, forgot to ping. [18:06:54] ^demon: bad permissions on some refs maybe? [18:07:14] <^demon> Not a clue, haven't dug very far yet. Everything in refs/* seems to have remained intact. [18:07:18] well, if that's the case, then it's possible a bunch of repos are hosed [18:07:37] <^demon> But master and refs/meta/config disappeared and git branch couldn't find them [18:07:53] well, that sucks [18:08:00] ^demon: Did you look at reflog? Did it still have the commits? [18:08:13] <^demon> Not before you pushed master back. [18:08:23] Hah [18:08:37] But it's still got the submitted-for-review commits though [18:08:53] <^demon> Right, everything in refs/changes/* definitely seems to be there. [18:10:07] <^demon> On the bright side: [18:10:13] <^demon> 154M /var/lib/gerrit2/review_site/git/mediawiki/core.git) [18:10:51] <^demon> Ok, step 1 is to figure out if any other repos are broken, and fix them if needed. [18:11:04] <^demon> Step 2 is to figure out what really happened, and if it was indeed git-gc's fault. [18:11:18] <^demon> Step 3 is to (hopefully) re-enable the gc cron, but not on the slave host. [18:11:32] how do you detect a broken repo? :P [18:11:46] <^demon> "Are branches missing?" [18:12:00] <^demon> I guess I could clone everything and see if it works. [18:12:04] * ^demon shudders [18:19:44] ^demon is going to have a fun evening [18:21:29] <^demon> Possibly operations/debs/mysqlatfacebook is broken--not sure if there's history there that's now missing or just no history yet. [18:21:41] <^demon> HEAD points to master, but no master. [18:22:32] <^demon> operations/debs/wikimedia-lvs-realserver [18:22:41] <^demon> operations/debs/wikimedia-search-qa [18:22:49] <^demon> operations/debs/wikistats [18:23:44] <^demon> operations/software is missing refs/meta/config, but not master. [18:25:07] <^demon> Ok, so I've checked everything other than mediawiki/extensions/* -- the 5 above + operations/mediawiki-config are the broken ones. [18:26:15] how to fix them> [18:27:18] <^demon> Working on that. For operations/mediawiki-config we just pushed the sources back from fenari. [18:28:20] ...What happened? [18:28:44] The pixies invaded and kicked the unicorns ass [18:28:53] but they're so cute :( [18:29:20] <^demon> dschoon: Something happened. Which did bad stuff. And it's quite probably my fault. [18:29:26] <^demon> And we have zero clue why yet. [18:30:17] <^demon> RoanKattouw: Ouch. https://gerrit.wikimedia.org/r/#/q/mysqlatfacebook,n,z - try to open any of the commits [18:30:31] Ouch [18:30:37] I wonder what happends if you run git gc in a dir above that of a repo... like you did when $d was interpritated as a puppet var [18:30:38] So the actual commits aren't there [18:31:26] <^demon> Damianz: Well, the find should've only gone into directories that ended in *.git [18:31:40] <^demon> (these are bare repos, btw) [18:32:20] OK seriously we need a real backup of our version control [18:32:28] This kind of crap should not be possible [18:32:47] * Damianz finds his replication isn't a backup solution line [18:32:52] As a temp hack, could we like copy the bare repo to a 3rd machine, or to a random directory on formey, or something [18:33:15] <^demon> RoanKattouw: We could rsync $gerrit_site/git off to some place. [18:33:18] <^demon> I'd be fine with that [18:33:25] Yeah let's do that [18:33:36] Because I'm starting to feel paranoid [18:33:40] Rsync would happily rsync broken repos though so it would be luck if it hadn't gotten overriden yet [18:33:48] <^demon> Oh yeah, truth. [18:34:09] cd $d; git should really have been cd $d && git which might have helped [18:34:56] <^demon> RoanKattouw: Well, we could just go back to using the normal replication, but be smarter about not running maintenance things like gc on the slave. [18:35:09] <^demon> I really don't like how the slave is a full gerrit instance right now. All it needs is ssh+git [18:35:25] Yeah, we should do that [18:35:38] AND we should make backups [18:35:43] <^demon> That too. [18:35:48] * Damianz thinks it should have the dir of git repos backed up in amanda so it's versioned [18:35:55] With git's data model, rsync without --delete is reasonably safe [18:36:05] But yeah backing it up in Amanda would be best [18:36:28] And if we lost gerrit the mysql db and git repo would match on restore and not have weird broken changes [18:37:10] Right, we need to back them up as a unit [18:39:03] <^demon> formey is a mess with svn on it too, with so much other junk. [18:39:07] <^demon> We should just set up a new slave. [18:39:23] opening labs registration = opening gerrit reg? [18:39:40] <^demon> Yes, they use the same auth [18:40:06] It could be interesting if we get spam via gerrit changes [18:40:26] * Damianz wonders how many ascii cats could fit in LocalSettings without making it any less useful [18:41:20] will review load increase somehow? [18:41:50] Probably not, hopefully though [18:50:32] ^demon: operations/puppet is broken too [19:04:22] ^demon: http://lists.openstack.org/pipermail/openstack-dev/2012-September/001102.html [19:07:25] <^demon> Ryan_Lane: Are they upstreaming that? [19:08:32] <^demon> Ahhh, I see how they do it. [19:08:38] <^demon> They use patchset-created and a hook [19:08:42] <^demon> I thought it was a core mod :) [19:11:41] <^demon> That probably looks doable to add to our patchset-created. Toss it in BZ so I remember to poke at it when I'm not putting out a fire. [19:12:27] Ryan_Lane: If I want to deploy a single python file, do you recommend that I make it into a .dpkg, or use pip, or just drop the raw file into puppet/files? [19:13:27] * ^demon knows Ryan will say "definitely not pip" or something to that effect [19:13:46] pip+virtualenv = <3 [19:14:24] <^demon> I hate pip almost as much as I hate pear. [19:20:43] pip is nice when you're deploying multiple python apps that require different versions of libs. Packages are nice for deploying stuff that works on system install of things. [19:23:07] I'm never quite clear how to get my libraries into the python path for other tools… pip seems to make that happen. [19:43:27] andrewbogott, in the other tool: [19:43:29] import sys [19:43:43] sys.path.append("/path/to/your/tool") [19:43:54] there's probably a better way, but that one works :P [19:44:07] Platonides: That presumes that I can modify the other tool... [19:44:20] I'm adding an extension, so it needs to go into the existing path. [19:49:30] Place in any of the following ['', '/usr/lib/python2.6', '/usr/lib/python2.6/plat-linux2', '/usr/lib/python2.6/lib-tk', '/usr/lib/python2.6/lib-old', '/usr/lib/python2.6/lib-dynload', '/usr/lib/python2.6/dist-packages', '/usr/lib/pymodules/python2.6', '/usr/lib/pymodules/python2.6/gtk-2.0', '/usr/lib/python2.6/dist-packages/wx-2.8-gtk2-unicode', '/usr/local/lib/python2.6/dist-packages']? [19:59:48] Damianz: Or subdirs of same? is the search recursive? [20:00:03] And, how do I subsequently refer to the module in import statements? [20:02:16] That's sys.path, normally you'd stick mymodule in /usr/lib/python2.6/dist-packages then you can just import mymodule. I believe that dir is a mix of .py, .pyc and .so . [20:02:42] I think if you write a setup.py you can use the dist folder to create a deb, it just needs some control files [20:03:07] http://ghantoos.org/2008/10/19/creating-a-deb-package-from-a-python-setuppy/ for example [20:15:39] * andrewbogott reads [20:16:27] I really really hate compiled extensions :( [20:47:02] andrewbogott: either package or puppet [20:47:03] not pit [20:47:04] pip [20:47:08] I hate pip [20:47:28] * Damianz puts Ryan_Lane on pypi [20:47:49] <^demon> Ryan_Lane, RoanKattouw: So, #gerrit is about 99% sure it's not git-gc's fault. Looking into other possibilities. [20:48:00] OK [20:49:04] <^demon> Ryan_Lane: It was suggested we double-check for a bad disk in manganese--can you do that? [20:49:17] <^demon> (I doubt it's the cause, but can't hurt to rule it out) [20:49:34] That would be rather interesting if it was the case but likely not [20:49:45] Ryan_Lane: Is it possible to do a test install of nova/essex on a labs machine? Or should I just write a perfect patch that goes straight to production? [20:50:14] ^demon: we do have a ton of kernel errors [20:50:45] <^demon> Hmm, that can't be good. [20:51:06] soft lockups on cpus [20:51:17] seems to be related to java, though [20:52:36] <^demon> Our repos are managed by jgit :\ [20:52:58] there's also an xfs error [20:53:13] <^demon> Would upgrading to precise help? [20:53:15] let me check mdadm [20:53:57] no failed devices in mdadm [20:55:53] <^demon> I'm going to go ahead and send out an e-mail to wikitech and other places telling people we're having some problems. [20:56:00] ok [20:56:00] <^demon> Those broken repos especially need to be left alone for now [21:01:16] <^demon> Are we getting similar kernel errors on formey? Some of the same repos are broken (and some of the same ones are fine, Comments.git for example is broken on mang. but not form.) [21:01:28] * ^demon is starting to suspect git-gc may at some level be at fault here. [21:02:29] ^demon: Cause you toally tested this in labs, with a clone of prod reports, right? :D [21:02:43] <^demon> I tested it in labs, but not with the full ~350 repos. [21:02:49] <^demon> I tested it on a smaller scale. [21:06:46] ^demon: nope. not getting those on formey [21:13:07] <^demon> http://p.defau.lt/?SzECcf3P5AGnk80uoDRLFQ - git fsck says one repo is midly fsck'd right now. [21:18:13] wonderful [21:22:02] <^demon> So, the objects are still there afaict. It's just a matter of reconstructing the history in some sane manner. [21:22:12] <^demon> Which I'm not 100% sure on yet, especially if it's an old repo with a long history [21:25:26] <^demon> I wonder--and bear with me here--if the issues with manganese (if kernel issues caused the problem) broke the repos, and the broken references were replicated. [21:25:49] <^demon> So you've got a repo that loses it refs/whatever, and then gerrit merrily replicated the ref changes. [21:26:00] <^demon> Leading to broken repos on both boxes. [21:29:49] I wonder if it was semi-broken but appeared to be working and git gc just went and cleaned up what looked broken and not in use but was actually not so broken. Coincidences are more annoying to solve. [21:30:19] <^demon> Indeed. Right now I'm just looking to find a semi-automated way to restore this. [21:40:07] <^demon> git-ls-tree seems to show a sane tree, so all the objects + their structure seem to be intact. [21:40:14] <^demon> It looks like it's just the refs that need rebuilding. [21:49:01] <^demon> Ryan_Lane: I'm going to step out for dinner for a bit before diving back into this. I've copied all of the affect repos *out* of gerrit's git directory, so in case anything else goes wonky, we've at least got copies as of now. [21:51:19] Enjoy f0000d [22:00:07] Ryan_Lane: I need a wiki account for Nova to use when updating instance pages. Should I just create an account for that, or do you already have something like that? [22:23:54] andrewbogott: I have one [22:24:15] we can use the same one that's used for logging [22:27:06] andrewbogott: see /etc/adminbot/config.py on bots-labs instance [22:27:11] labslogsbot user [22:31:57] ok! [23:00:11] Change on 12mediawiki a page Developer access was modified, changed by !Silent link https://www.mediawiki.org/w/index.php?diff=580801 edit summary: /* User:!Silent */ [23:00:59] Ryan_Lane: Are I-00000xxx style instance names some manner of Nova ID, or are they IDs that OSM generates? [23:01:13] BecauseI only see crazy alphanumeric guids w/in Nova. [23:03:37] Isn't i- ec2 style, crazy guids openstack style [23:04:28] Oh yeah, probably. So maybe I can convince Nova to tell me the ec2 id. [23:04:38] * andrewbogott reads yet more Nova source code [23:06:29] * andrewbogott decides pizza is more interesting [23:06:53] Mmm I had pizza earlier, debating about chinese now [23:08:14] Nova code is confusing in part, still toying with the idea of writing my own wrapper ontop of libvirt though so tend to avoid getting into the mood for contributing to it as there's some crazy stuff I'd like to try. [23:10:08] I've found the openstack/nova dev community to be generally pleasant. [23:10:21] And they use git and gerrit which you're probably used to by now [23:11:38] Yeah, though I thought they used github and then random bits on launchpad like gerrit. [23:15:56] I sorta like the way openstack is heading and the community, but things like Ganeti are interesting for the different approachs (though things like drdb betwean nodes could be rolled into a kvm plugin I guess) [23:16:54] VMWare is joining openstack, which either means victory or death for the project, I'm not sure which. [23:18:15] o.0 [23:18:57] VMWare have been doing a lot of pushing puppet based deployments of esx lately, that could be interesting. [23:19:13] Though if you look at xen and how it's all citrixish now it pretty much died [23:50:00] I want this in a labs project: https://github.com/cweiske/phorkie [23:50:04] though I want it puppetized [23:50:48] Apart from it saying php that looks pretty cool [23:50:49] and preferably without pear [23:51:07] let me remove the "preferably" part [23:51:09] no pear [23:51:09] :) [23:51:24] It's a crappy version of gists :D [23:51:26] it's a gist clone [23:51:28] yeah [23:51:37] Though thinking about it writing that in python should be pretty simple... [23:51:51] <^demon> It relies on a ton of pear libs, so you'd have to package them too (or pray someone already did) [23:51:59] <^demon> I. Hate. Pear. [23:52:03] yep [23:52:04] me too [23:52:10] I like pears, only when they're ripe though [23:52:18] * ^demon whacks Damianz [23:52:23] <^demon> No puns allowed. [23:52:29] Tbf I dislike installing php :P [23:52:45] meh. we use php a lot :) [23:52:47] <^demon> apt-get install php is so hard ;-) [23:53:03] Using it a lot doesn't make it good :P [23:53:23] https://github.com/justinvh/gitpaste [23:53:32] Hmmm if it uses git-deamon to serve up the repos wtf does it have *THAT MANY* deps [23:53:42] written in python [23:53:57] <^demon> If only we had some software where you could just write freeform text. [23:54:01] <^demon> And it stored versions of them [23:54:13] "just pip" [23:54:15] fuckers [23:54:38] mediawiki is terribly for stuff like this [23:54:42] unfortunately [23:54:48] It does only have 13 deps :P I wouldn't have used Django though... no db required, just use flask [23:55:04] Now being able to clone mediawiki pages udner git would be cool :P [23:56:07] I think I'd rather gitpaste than phorkie [23:56:48] http://www.gitpaste.com/ does actually look rather like gist... [23:57:06] Not sure why it has login stuff but heh it's a django app so hooking it up to ldap would be simples [23:57:21] <^demon> Damianz: People keep bringing up the idea of "let's move revisions to git" every couple of years, but never really think about it. [23:57:34] <^demon> They see a couple of features in git they'd like, but really it's not as simple as it sounds on the tin. [23:58:04] I'm gonna make a project [23:58:07] anyone else want to join? [23:58:10] For what mediawiki does git is overkill, it does require half the features and would just be bloated [23:58:17] doesn't* [23:58:29] meh. for a paste-bin like system git makes sense [23:58:33] it doesn't for mediawiki [23:58:36] For pastebin it does [23:59:36] hm [23:59:37] crap [23:59:44] hell this is backed by haystack o.0 [23:59:45] labsconsole is failing to add me to roles [23:59:53] I wonder if it failed to create the roles