[06:00:59] 07/30/2012 - 06:00:59 - User fastily may have been modified in LDAP or locally, updating key in project(s): bastion [06:01:21] 07/30/2012 - 06:01:21 - Updating keys for fastily at /export/keys/fastily [07:30:15] morning [08:13:28] Platonides: there it is easier for the ops :-) [08:13:43] nah, they're all sleeping now :) [09:01:35] petan: ping pong. if you are around, do you have anything on -bastion host? I would like to simply delete it [09:04:53] remember to document the steps :) [09:05:43] yes, we have have nfs::upload::labs [09:05:50] but puppet complained [09:06:08] I thought that perhaps nfs::upload::labs automatically inherited from nfs::upload [09:06:17] I don't really speak puppet [09:06:21] well not really [09:06:29] or we would need a nfs::upload::production [09:06:42] and have nfs::upload to include one or the other whith some if/else :-D [09:06:46] but ops don't want that [09:07:29] maybe it was bastion who was in both classes [09:07:37] might be [09:07:42] anyway I want to deleted that host :-) [09:08:51] we also need a bastion class I guess [09:09:13] probably [09:09:27] then we could create a new bastion just by using that class [09:10:22] btw, do we have a class which could be used to get a basic LAMP? [09:13:39] Platonides: if you or anyone gets around to doing that it would be great to get that setup working w/vagrant, which integrates with puppet [09:13:55] vagrant? [09:14:05] Platonides: there is ./role/labsmw.pp [09:14:19] ori-l, what's that? [09:14:19] Platonides: well that install mediawiki hmm [09:14:34] Platonides: webserver::php5-mysql or webserver::php5 probably [09:14:36] http://vagrantup.com/. basically spins up a local VM using virtualbox, but it's headless. "vagrant ssh" to ssh in, neat stuff like that [09:14:56] if you give it a puppet config it'll set up the VM accordingly. [12:12:08] Change on 12mediawiki a page OAuth was modified, changed by Dantman link https://www.mediawiki.org/w/index.php?diff=566815 edit summary: /* Previous discussions */ [12:21:30] hashar: heya [12:21:45] paravoid: hello :-) [12:22:47] paravoid: wanna chat about some ugly hack I have done in puppet? https://gerrit.wikimedia.org/r/#/c/16661/ [12:23:02] trying to resolve rsyslog / syslog-ng conflicting to each other [12:29:44] er [12:29:46] not really :P [12:30:43] more seriously [12:30:46] let me have a look [12:30:59] this is blocking you, right? [12:32:24] paravoid: well it prevents us from collecting syslog [12:32:31] that has been the case since day 1 anyway :-D [12:32:51] I thought about replacing syslog-ng by rsyslog as a main log collector [12:33:11] afaik, syslog-ng is historical and we only use it in production for syslog collecting [12:33:22] whereas rsyslog is installed everywhere else to send the log to a remote host [12:34:46] (or we can do it the other way around and have syslog-ng everwhere) [12:35:51] yes, that was my idea as well [12:36:08] the easiest is to keep syslog-ng since it got a large conf file [12:36:16] whereas rsyslog has very tiny configuration files [12:36:32] but then I think rsyslog has more features / is well maintained [12:36:46] + I think rsyslog is default on ubuntu [12:38:49] petan: still around? May I phase out deployment-bastion which is corrupted or is there anything you need there? [12:39:04] iok [12:39:06] ok [12:39:30] paravoid: rsyslog is installed by 'ubuntu-minimal' as of Precise. [12:39:36] petan: yeahhh [12:39:50] petan: it apparently caused some awoke this week-end :-) [12:40:11] how [12:40:38] !log deployment-prep Deleting -bastion , was corrupted. [12:40:40] Logged the message, Master [12:43:33] !log deployment-prep Recreating deployment-bastion using a Precise image and s1.small (1CPU, 1GB RAM, 80G storage) [12:43:34] Logged the message, Master [12:47:32] Change on 12mediawiki a page OAuth/Issues was modified, changed by Dantman link https://www.mediawiki.org/w/index.php?diff=566837 edit summary: /* Summary */ [12:48:18] !log deployment-prep Shutdowning deployment-nfs-memc for a while, will see if it is still needed around or if we can safely delete it (see {{bug|38084}}). All data should be on /data/project . [12:48:19] Logged the message, Master [13:00:00] <^demon> hashar: Happy monday :) [13:00:09] ^demon: you too :-] [13:00:11] <^demon> What do you think of new gerrit colors? http://gerrit-dev.wmflabs.org/r/#/q/status:merged,n,z [13:00:23] ahahah [13:00:28] I hate the blue :-] [13:00:47] <^demon> It's better than puke green :p [13:00:55] totally :-) [13:01:02] literal bikeshedding! [13:01:55] ^demon: could you make the table borders a bit darker ? #DBDCFF apparently [13:02:12] <^demon> No, I can't, that's CSS. [13:02:20] <^demon> This is all done with the theme settings. [13:02:26] ohhhhh [13:02:30] <^demon> We're trying to get a base color set that doesn't make you throw up :) [13:02:51] so go ahead with the blue. You have my support :)) [13:03:25] ^demon: also, could you customize the header to add the WMF logo ? [13:03:34] <^demon> We could. [13:03:51] ^demon: similarly can the header file be used to inject .css after the Gerrit css? [13:04:06] <^demon> Maybe with some Javascript trickery, been playing with it. [13:04:21] <^demon> I've been mainly trying to fix the underlying issue. Spent a lot of time Friday trying to dig into it. [13:05:19] <^demon> What I'd like to work on is getting gitlist packaged and deployed. [13:05:39] <^demon> That's a big thing. And since it has a sane repo listing, we could add a "Project list" link in the header along with the logo. [13:06:23] paravoid could set up a gitlist-source package [13:06:34] that would just install all the gitlist file somewhere like in /usr/share/site/gitlist or something [13:07:09] <^demon> paravoid: Is that something you could help me with? [13:07:31] it is, but I'm starting to get overbooked lately :) [13:07:42] <^demon> Join the club :) [13:07:46] welcome to the club:) [13:08:11] let me find out my notes about building a deb package [13:08:48] <^demon> paravoid: Basically, it's just a "copy some .tar.gz and extract it, plus some apache setup" No database or anything else fancy--but all the docs I read make it seem harder than it should be. [13:10:04] http://en.wikipedia.org/w/index.php?title=User_talk:Petrb&diff=next&oldid=504265243 O.O [13:10:16] don't put the apache setup in the .deb [13:10:30] <^demon> Yeah, was gonna do that bit in puppet. [13:10:33] wmf labs logo is political-religious propagandist WTF [13:10:54] test [13:11:01] eh, nvm that test [13:11:31] <^demon> paravoid: So really, it's just "copy this tar and extract it somewhere" [13:11:42] petan: just ask that person a reference about the unicorn being a political/religious logo :-) [13:11:53] there is actually article on ENWP [13:12:05] https://en.wikipedia.org/wiki/Invisible_Pink_Unicorn [13:12:17] ^demon: well, the deb shouldn't ship a tarball either :) [13:12:18] but it doesn't say it's propagandist logo [13:12:28] !log integration dist-upgrading psm-precise [13:12:29] Logged the message, Master [13:12:33] it just say that the religion uses similar unicorn as a logo [13:12:38] ^demon: you should probably ship the stuff as it is in the deb [13:13:01] petan: well just discard that message or reply asking him to argument about how it hurts anyone :-) [13:13:02] I didn't make that unicorn, I found it on commons :| [13:13:28] petan: well just reply that :) [13:13:42] <^demon> paravoid: The more I'm thinking, it might just be better to make puppet do it all rather than making a deb for it :p [13:14:09] puppet's fileserver is not really very good at shipping large or multiple files [13:14:33] <^demon> No, have puppet fetch the files. [13:14:49] from where? [13:15:15] <^demon> Either the upstream tar, or from a repo we own with Git::clone{} [13:15:27] the upstream tar is a bad idea [13:15:43] for security reasons [13:15:45] and imho :) [13:15:50] <^demon> Well, forking it to our repo might not be a bad idea. The code's not fantastic, so it'd give us a chance to make hacks. [13:16:11] <^demon> Then we control it, and puppet can just clone that. [13:17:15] <^demon> I could stash it in operations/software/gitlist or somesuch. [13:17:41] I guess that's sensible [13:17:45] or fork it on github? :-D [13:18:42] <^demon> Oh wow, he got really active the last couple of days https://github.com/klaussilveira/gitlist/commits/master :) [13:18:58] <^demon> Well, on the 25th. [13:22:05] so there is git-buildpackage [13:22:05] http://wiki.debian.org/PackagingWithGit/ [13:22:16] but it ask about a .dsc file which is supposed to describe a source package [13:22:22] no idea how to write such file :-D [13:38:20] paravoid: so what do we do for the rsyslog / syslog-ng stuff ? [13:38:42] paravoid: I guess the easiest is to apply the instancename hack [13:38:53] paravoid: and start up something to migrate from syslog-ng to rsyslog :) [13:41:15] ^demon: bha gitlist source code ship with Symfony / Silex / Twig and others! [13:42:58] <^demon> yeah, the code sucks. [13:44:37] I guess you could create an operations/debs/gitlist repo [13:47:37] <^demon> Ugh, the released version in the tar sucks. [13:48:04] <^demon> The version in git works. [13:48:14] Hrm. [13:48:25] ^demon: I am doing a basic package right now :-) [13:49:00] <^demon> We should just fork the damn thing. [13:49:08] fork what? [13:49:10] I'm trying to pull some github repos onto my labs instance, but I don't seem to have net access. [13:49:16] the whole gitlist in Gerrit ? :)) [13:49:46] jerith: well if you are connect to the instance, it get network access isn't it ? [13:49:57] jerith: maybe there is a default security rule that prevent instances to get out though [13:49:58] no idea [13:50:05] ^demon: dpkg-source: info: building gitlist using existing ./gitlist_0.2.orig.tar.gz [13:50:29] jerith@vumi:~/vumitest$ curl github.com [13:50:31] curl: (7) couldn't connect to host [13:52:01] I can connect to it through the bastion. [13:52:14] <^demon> hashar: That tar suckssss. [13:52:24] <^demon> It doesn't even recurse into the directory yet. [13:52:38] ^demon: make them tag a new version so ? :-] [13:52:46] though I could use some arbitrary sha1 [13:53:11] jerith: maybe GitHub has banned our labs IP address [13:53:24] hashar: I can't connect to google.com either. [13:53:38] ohh [13:53:52] let me try :) [13:53:59] <^demon> hashar: This is 0.2: http://gerrit-dev.wmflabs.org/repo/ [13:54:03] <^demon> Just finished installing by hand [13:54:31] jerith: yeah apparently my instance can't get out of labs too despite it having a rule to allow port 80 [13:55:00] though that might something else [13:55:02] I've definitely beeen able to connect to gtalk from there. [13:55:12] That's on a high port, though. [13:55:39] jerith: I have no idea how labs get out to the net. There must be some kind of shared PAT in front of it [13:55:55] no idea which box it lies on and I am pretty sure I don't have any access to it :-( [13:56:12] paravoid: apparently instances can not access the web on port 80 anymore (ex: curl -v www.yahoo.de ) [13:56:54] ^demon: so I build a debian package [13:57:00] ^demon: and it has no content :-) [13:57:00] Or 443 -- the github URL is on HTTPS. [13:57:19] jerith: do you have a bugzilla account? If so can you please open a bug? [13:57:25] I am not able to fix that issue for sure [13:57:26] I don't. [13:58:08] I'll poke preilly or someone later this evening. [14:00:06] hashar: works for me, it might be something with the new nodes and the network setup [14:00:24] <^demon> I'm not having any problems on the gerrit node, either [14:00:24] ...which I have no idea how it works [14:00:28] I can reproduce the issue on psm-precise a Precise box on 'integration' [14:01:35] yep, confirmed [14:01:50] jerith@vumi:~/vumitest$ telnet talk.google.com 5222 [14:01:50] Trying 74.125.134.125... [14:01:50] Trying 2001:4860:800a::7d... [14:01:51] telnet: Unable to connect to remote host: Network is unreachable [14:02:01] Yeah, that's definitely a new thing. [14:02:21] hm, psm-precise runs on virt1 though [14:02:31] that's strange [14:08:13] ^demon: I have asked for a new tagged version https://github.com/klaussilveira/gitlist/issues/123 [14:09:47] f*** Debian [14:09:50] grmblblb [14:10:00] hashar: that guy replied [14:10:01] http://en.wikipedia.org/w/index.php?title=User_talk:Petrb&diff=next&oldid=504931210 [14:10:06] <3 debian [14:10:18] I can't find the latest Standards-Version anywhere :( [14:10:24] wut [14:10:48] I never heard of Standards :P [14:10:53] petan: I would just ignore him, or ask him to send that to the labs-l mailing list [14:11:00] ok [14:11:05] petan: If you are really evil, I guess that could be asked on foundation-l :-))) [14:11:18] then that will ensure a LOT of drama about us needing to be NPOV and so on :) [14:11:21] going to be a lot of fun [14:11:32] meanwhile, you might want to find another unicorn picture :))) [14:11:40] or that [14:11:46] I liked that [14:11:50] or all the christians / theist / whatever we keep harassing you to get the invisible unicorn to get removed :-((( [14:12:01] but this one is nice [14:12:06] :/ [14:12:07] but I would definitely try to start a nice drama thread on foundation-l / labs-l whatever [14:12:13] maybe we could change the color or something [14:12:17] yeah [14:12:27] make it the not-pink-visible unicorn :-)))))) [14:13:48] petan: something like the green unicorn like http://www.colourlovers.com/color/00FFD5/Hue_170_Degrees :) [14:13:58] petan: or we could make one that use the WMF colors [14:14:22] or start up a new context to have a nice unicorn designed. I will be happy to give out some of my personal money to build a prize for a new unicorn :) [14:15:01] I can't really make pictures, I am a coder not a painter :o [14:15:16] I was rather thinking of looking for another one on commons [14:15:27] well code a tool to load the palette and change pink to green ? :-D [14:15:34] heh [14:15:51] it's fading color [14:16:00] ohh [14:16:01] wouldn't be that simple [14:16:05] it is also a SVG file [14:16:07] yes [14:16:11] so you can change the colors using vim :) [14:16:17] heh [14:16:34] I don't understand svg at all [14:16:36] and upload it back on commons as a derived work [14:16:42] hmm [14:16:53] with a filename like VisibleNotPinkUnicorn.svg :D [14:18:36] or NotAPropagandistUnicornWhichIsNotAffiliatedToAnyWeirdGroupOfWeirdos.svg [14:18:43] petan: apparently just s/#ffc0cb/#00FFD5/ [14:18:48] ok [14:18:49] wanna do it or should I ? [14:18:57] if you feel bold, do it :D [14:19:10] I can change it but I don't know how to test it [14:19:13] how do I open svg [14:19:24] vim :) [14:19:29] ohh sorry [14:19:36] most browsers should be able to read the svg [14:19:40] hm... [14:19:42] firefox and safari definitely can [14:19:55] ok [14:20:28] uploading :) [14:20:29] how I download source [14:20:34] firefox download it as png [14:23:46] petan: https://commons.wikimedia.org/wiki/File:Visible_Green_Unicorn.svg [14:24:54] my opinion is that you're taking that complain far too seriously [14:25:04] ok [14:25:10] but that's just my opinion [14:26:09] petan: so just replace it with the new green unicorn :-]]] [14:26:17] hashar: I did [14:26:24] well done :-]] [14:26:52] (of course, my green version will be deleted soon by the commonists) [14:27:12] why :D [14:27:17] I can protect it :P [14:27:19] cause I forgot something ? :-))) [14:27:23] but that likely won't help [14:27:26] ^demon: so hmm I get a very basic package :) [14:27:29] categories. [14:27:34] heh [14:27:51] ^demon: but based out of v0.2 [14:27:53] (it is already tagged as "media lacking categories") [14:27:59] :D [14:28:12] we need to create [[Category:Labs]] [14:28:54] wait, let them delete it, and you can use the "unicorn.svg.which.does.not.exist.svg" [14:29:14] <^demon> hashar: What I'm gonna do instead is clone upstream, do the install steps like I would manually (so we get lib/), then push that to gerrit for use. [14:29:46] ^demon: in which repo ? puppet? [14:30:23] <^demon> Na, I'll make operations/software/gitlist [14:30:28] <^demon> And puppet can clone it [14:30:35] fine :)) [14:31:22] catted [[Category:Invisible Pink Unicorn]] [14:44:19] Amgine: thanks! [14:47:40] !log deployment-prep rebooting -bastion [14:47:41] Logged the message, Master [14:56:59] 07/30/2012 - 14:56:59 - Created a home directory for hashar in project(s): gerrit [14:57:59] 07/30/2012 - 14:57:59 - User hashar may have been modified in LDAP or locally, updating key in project(s): gerrit [15:00:10] !log deployment-prep deployment-bastion does not let us log in despite being a fresh instance. Logged as {{bug|38846}} [15:00:11] Logged the message, Master [16:12:48] !log gerrit applying puppetmaster::self to gerrit-puppet-overhaul [16:12:49] Logged the message, Master [16:51:46] ack. the labs unicorn changed colors [16:51:53] it's blue now! [16:52:07] pink > blue [16:52:12] heh [16:52:20] Name says it's green anyway >.> [16:52:32] that's not blue! [16:52:39] it's blueish green [16:52:48] * paravoid passes his ColorHug to Ryan [16:52:48] aquamarine? [16:56:08] paravoid: euca-reboot-instances didn't output anything for you/ [16:56:11] weird [16:56:13] it did [16:56:18] INSTANCE i-... [16:56:21] that's about it [16:56:27] that's normal [16:56:30] that means it rebooted [16:56:34] no it didn't :) [16:56:43] that means the reboot command was sent ;) [16:57:04] okay, but nothing happened [16:57:12] you looked at nova-api on virt2, right? [16:57:26] virt2? why would I check on virt2? [16:57:33] because that's where it lives [16:57:46] what is? [16:57:52] the api [16:58:04] there's a historical reason behind that [16:58:20] the metadata service lives in the api service [16:58:34] (which is thankfully changing in folsom, or did it change in essex?) [16:58:36] and how do you check where the api service lives? [16:58:49] nova-manage service list [16:59:01] nope [16:59:02] I did that [16:59:06] before I asked [16:59:09] ah. right. api service isn't listed [16:59:11] puppet? [16:59:13] heh [16:59:27] "ask Ryan!" [16:59:27] :P [16:59:30] /etc/nova/nova.conf [16:59:50] hm. nope [16:59:52] ec2_url? [16:59:53] kind of [17:00:05] it's in nova.conf [17:00:11] ah [17:00:16] yeah, that's the one [17:00:23] what about the native openstack api? [17:00:31] we aren't using it yet [17:00:36] okay [17:00:36] I'm working on that right now [17:00:43] okay, so to answer your question [17:00:47] no, I didn't check virt2's logs :P [17:00:49] heh [17:00:59] let me check now. [17:01:01] I usually check the compute host's logs [17:01:10] to see if the compute host is attempting a reboot [17:01:25] I also see if the compute hosts' service is listed as down [17:01:40] because that would mean it isn't getting the notification [17:02:10] nova-compute shows as down on virt6 [17:02:46] some services will list down on occasion, this one shows every time I run the service list command [17:03:20] so, the nova-compute service should be restarted on that node [17:03:48] paravoid: btw, unfortunately for now, the instance pages won't get updated for the host [17:03:52] mediawiki is doing it. [17:04:09] andrewbogott wrote an openstack plugin for updating mediawiki, though [17:04:19] so, at some point openstack will take care of those pages [17:05:04] also, each instance uses its own network filterref in case you want to modify them per-instance [17:06:56] ok. heading to the office [17:33:05] ^demon: any idea what gerrit on formey is ? [17:33:22] ^demon: seems to be a different role than the one on manganese [17:38:59] hashar: cross-DC replica I'd presume [17:39:09] looks so [17:39:13] will investigate later :) [17:39:23] off for now :-D [17:56:00] 07/30/2012 - 17:56:00 - Created a home directory for lcarr in project(s): globaleducation [17:57:00] 07/30/2012 - 17:56:59 - User jeremyb may have been modified in LDAP or locally, updating key in project(s): globaleducation [18:15:21] 07/30/2012 - 18:15:21 - Creating a home directory for madman at /export/keys/madman [18:16:22] 07/30/2012 - 18:16:21 - Updating keys for madman at /export/keys/madman [18:30:55] Hey, all. Who should I talk to about joining the bastion and bots projects? :) [18:31:29] petan worked for me [18:31:53] Ryan_Lane or andrewbogott or ssmollett for bastion IIRC [18:32:02] should write that to the bots page [18:32:22] Yeah, I poked around the bots page but didn't see who were admins. [18:32:35] AMadman: We're all in a meeting, can you ping me in half an hour? [18:32:41] andrewbogott: Sure thing. :) [18:32:45] andrewbogott: you're optimistic :) [18:33:01] are you in a meeting or not? [18:33:12] we all are [18:33:14] So, if I want to set up an instance with the MediaWiki template (to test bug fixes, etc.) do I have to be a sysadmin of a project? There doesn't seem to be a general "tinkering with bugs" project. :p [18:33:28] Oh and parav, forget about him :) [18:33:45] AMadman: Yes [18:34:11] hashar: Btw did you see the BZ about thumbs on beta? [18:34:17] Damianz: yup [18:34:23] that has been broken since may or so [18:34:29] Ah [18:34:35] * Damianz kicks Ryan_Lane for saying it should work [18:34:40] (Also: The Next link doesn't work on List projects.) [18:34:42] that is because the files do not exist on the local wiki [18:35:02] I spent like an hour trying to make mw happy :( [18:35:12] So is it kosher to request a project just for myself so I can do that, or is there a larger project I'm missing? [18:35:21] I assume the thumbs shared folder will be the new location for them? [18:44:53] !log gerrit applied {{gerrit|13484}} patch #9 to gerrit-puppet-overhaul : cd /var/lib/git/operations/puppet && git fetch anonymous refs/changes/84/13484/9 && git checkout -b 13484/9 . Now manually running puppet and fixing the class :) [18:44:54] Logged the message, Master [18:50:34] does anyone have any idea about what /root/.my.cnf is for ? :-D [18:50:38] we don't have it on labs :-( [18:52:04] client config? [18:52:23] We do on some boxes, actually would be nice to randomise the root pass and stick it in there rather than using puppet [18:56:44] Ryan_Lane: I've made a basic puppet tutorial based on the DC session. [18:57:11] OrenBo: cool [18:57:16] Ryan_Lane: I'ts currently at http://meta.wikimedia.org/wiki/User:OrenBochman/Labs_2 [18:57:41] Ryan_Lane: I'll upload it to MediaWiki when its complete. [18:58:08] labsconsole is likely better than mediawiki.org [18:58:30] both! [18:59:06] Ryan_Lane there is a guy on enwiki who say we can't use that pink unicorn [18:59:20] because it's a logo of some religious group of people or what [18:59:29] hashar changed the color for now [18:59:44] If it's some anti-gay natzi party I vote we use it just to spite them [18:59:52] sort of [19:00:02] The Invisible Pink Unicorn. [19:00:07] yes [19:00:22] Kind of an anti-religious group, really. Or a mock-religious group. :p [19:01:06] o.0 [19:01:07] it's green atm [19:01:11] They sounds mental {{cn}} [19:01:19] so we don't match the color hehe [19:01:19] o.o [19:01:26] No, they just think religious people are mental. :p [19:01:46] I'm surprised more people have not heard of this. [19:01:47] http://en.wikipedia.org/wiki/Invisible_Pink_Unicorn [19:02:06] "The mutually exclusive attributes of pinkness and invisibility, coupled with the inability to disprove the IPU's existence, satirize properties that some theists attribute to a theistic deity." [19:02:52] But yeah, a bit political of a logo to use unmodified. :p [19:05:22] I just found it on commons and thought it's a good picture for logo [19:05:35] I didn't really look what it is used for [19:06:22] i want it pink! [19:06:47] the good thing is that it isn't our official logo, so I don't think its too big of a deal :) [19:08:00] Ryan_Lane can we make a bigger official logo [19:08:09] Yeah. No harm, no foul. [19:08:16] it's white and invisible and you can't proof that it isn't there, haha :) [19:08:19] the official logo needs to be trademarked [19:08:31] trademarks suck [19:08:38] the one in the top-left corner is the official one [19:08:39] just as software patents do [19:08:51] petan: tell the lawyers that, not me :) [19:08:56] heh [19:09:09] trademarks are good for one thing, though [19:09:21] hm, no one can steal your logo [19:09:22] to help shut down phishing sites [19:09:27] ah [19:09:28] Ryan seems to have a think for laywers [19:09:29] true [19:09:40] but it's hard to reuse it [19:09:45] for example on wikipedia [19:10:17] there is no picture of star wars in [[Star Wars]] because it's copyrighted [19:10:37] I kinda hate the wikipedia licencing [19:10:59] Like cc-by-sa is fine but hell if it's under an opensource license then why does it need to ALSO be under cc-by-sa -.- [19:15:44] yeah, everywhere on wikipedia you can read how bad are copyrights and wmf is licensing everything just so wrong as others heh [19:16:31] I really like these I don't give a fuck copyright [19:16:38] copyrights [19:16:48] that's what I would use for my software if there was no GPL [19:18:04] Most my stuff all just goes under GPL or WTFPL because really I don't give a damn [19:18:24] petan: well, you could use public domain [19:18:27] Beerware. \o/ [19:18:31] if you really hate copyright [19:18:36] that's probably WTFPL [19:18:44] public domain and that is same [19:18:49] AFAIK [19:19:03] not everywhere in the world has a concept of public domain, though [19:19:09] so CC0 is likely best [19:19:12] CC0. [19:19:20] Ah. Ryan beat me to it. :p [19:19:37] it sucks to have to know all this legal crap [19:19:47] Or any sort of similar disclaimer like the PD-self template has on Commons. [19:19:57] DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE [19:19:58] Version 2, December 2004 [19:19:59] Copyright (C) 2004 Sam Hocevar [19:19:59] WTFPL is basically CC0 [19:20:00] Everyone is permitted to copy and distribute verbatim or modified [19:20:02] copies of this license document, and changing it is allowed as long [19:20:03] as the name is changed. [19:20:04] DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE [19:20:05] TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION [19:20:06] 0. You just DO WHAT THE FUCK YOU WANT TO. [19:20:10] :D [19:20:23] I know I should use pastebin but you wouldn't click the link anyway... [19:20:37] "you're forced to read this" [19:20:43] hehe [19:20:58] GPL is slightly restrictive in some parts, LGPL fixes that a little [19:21:10] use apache or bsd or mit [19:21:30] I really want to see a court where they read WTFPL [19:21:49] :) [19:24:33] ^demon: so that gerrit overhaul is going to take a good amount of time :-D [19:24:48] ^demon: currently finding a way to generate a mysql root password when on labs :-] [19:25:38] hashar I just use puppet everywhere [19:25:39] :D [19:25:43] that's a good password [19:25:53] or [19:25:55] !password [19:25:56] gfgjoagaewhgAW#YAU_#Y$*U*U^*^%Q#Tqyhe [19:26:05] ^ totally secure [19:26:09] yes [19:26:19] not at all secure [19:26:29] I am using this password for my banking [19:26:31] for a human, you would have to store it somewhere insecurely [19:26:44] I would instead use PetanLikesPinkUnicorns [19:26:51] hold it in your mind [19:26:55] easy to remember :-D still high entropy [19:27:13] http://xkcd.com/936/ [19:27:14] ;) [19:27:36] hashar if someone decrypted it that guy would probably call you Satan [19:28:46] it is all about social engineering so :-D or finding out the unencrypted file [19:29:44] Crack an encrypted password? Just buy a 5� wrench and beat the guy who knows the password with it [19:31:46] heh [19:31:55] I think Ryan just quitted irc [19:32:04] because he found out a serious bussiness [19:32:27] selling password that give you powers over whole wikipedia XD [19:32:52] AMadman: Did you get your bastion account yet? If not, what's your username? [19:32:58] madman [19:36:00] 07/30/2012 - 19:36:00 - Created a home directory for madman in project(s): bastion [19:36:30] Thanks. :) [19:37:01] 07/30/2012 - 19:37:01 - User madman may have been modified in LDAP or locally, updating key in project(s): bastion [19:37:25] AMadman: sorry, didn't see this until now. could have poked me ;) [19:37:41] No problem. :) [19:37:59] I'm at work so in no hurry at all. [19:38:09] AMadman: Do you also want your own project, or are you teamed up with someone? (Or looking for someone to team up with?) [19:38:32] andrewbogott: bots [19:38:38] he said i think? [19:38:39] Well, that was my question earlier. Should I have my own project if I just want to create MediaWiki instances to test bug fixes? [19:39:01] Well, and bots, yes. I'm moving away from the Toolserver. [19:39:03] 30 18:30:55 < AMadman> Hey, all. Who should I talk to about joining the bastion and bots projects? :) [19:39:53] I wish i had an answer for this... > Should I have my own project if I just want to create MediaWiki instances to test bug fixes? [19:39:58] AMadman: jeremyb (or another bots admin) can add you there. [19:40:28] It's easy to create a project for you for debugging purposes… I feel like there are already a couple of projects for that purpose, but I'm not positive. [19:40:30] I feel like there should be a general MediaWiki project for that sort of thing. Does anyone have any objection to creating one? [19:40:42] AMadman++ [19:41:01] Yeah, I'm not positive either. I looked at the list of projects and didn't see any that stuck out like that. [19:41:16] (Also noted the Next link doesn't work on that resource query, at least.) [19:41:41] (true) [19:41:58] AMadman: confirmed next link is broke [19:42:01] https://labsconsole.wikimedia.org/w/index.php?title=Special:Ask&offset=50&limit=50&q=%5B%5BResource+Type%3A%3Aproject%5D%5D&p=mainlabel%3D-2D%2Fsearchlabel%3Dprojects%2Fformat%3Dbroadtable&po=%3F%0A%3FMember%0A%3FDescription%0A [19:42:17] So I just expanded to 250. [19:42:23] heh [19:42:29] AMadman: you should discuss bots a little with petan [19:42:45] But yeah, so either a self-project or a MediaWiki project would be good if admins are amenable. :) [19:42:56] anyway, you're a member of bots now [19:43:17] jeremyb, where'd that query come from? [19:43:19] Thanks much. :) [19:43:36] andrewbogott: left sidebar, "list projects" -> "Next" [19:43:38] AMadman: I'm happy to create a mediawiki-dev project if you like. [19:43:58] 07/30/2012 - 19:43:58 - Created a home directory for madman in project(s): bots [19:44:14] Okay. That's probably the most general solution. [19:45:00] 07/30/2012 - 19:45:00 - User madman may have been modified in LDAP or locally, updating key in project(s): bots [19:48:00] 07/30/2012 - 19:48:00 - Creating a project directory for mediawiki-dev [19:48:01] 07/30/2012 - 19:48:01 - Created a home directory for madman in project(s): mediawiki-dev [19:48:39] Yay. I'mma fix *all the things* {{citation needed}}. [19:48:59] 07/30/2012 - 19:48:59 - User madman may have been modified in LDAP or locally, updating key in project(s): mediawiki-dev [19:52:10] AMadman: Please add a description to your project so that the next time someone wants to fix mw bugs they find you :) [19:52:36] Will do. :) [19:59:44] (Amgine thot xe saw) AMadman: you should discuss bots with a little petan [19:59:59] sentence dyslexia is confusing. [20:00:04] Amgine: :p [20:40:52] <^demon> Ryan_Lane: So, we started poking gitlist this morning. The 0.2 tagged version totally sucks (it doesn't even recurse into sub-repos!), so hashar opened a bug asking him to tag a new release :p [20:41:05] cool [20:41:18] <^demon> http://gerrit-dev.wmflabs.org/repo/ - rough initial install :) [20:41:19] and I got a basic package on psm-precise instance (untested though [20:41:40] ^demon: aren't you going to fork the github repo under the Gerrit umbrella ? [20:41:44] if so I can abandon the package :-D [20:41:57] <^demon> Well, I was going to wait to see if he'll tag a new release. [20:42:00] <^demon> That would be better. [20:42:09] <^demon> If not, we'll hack it together ourselves, won't take long. [20:42:26] Ryan_Lane: I was wondering what the process *would be* regarding getting a public IP for a web instance, it's not urgent but if it's not going to happen / takes work I should probably factor that into my plans [20:42:31] maybe we should create an operations/debs/gitlist repo so I can put the packaging work there ? [20:42:53] <^demon> Oh, I see his response. [20:42:55] ^demon: I am on vacations for three weeks, the first without any internet connection (can't remember if I told you about it :((( ) [20:43:04] <^demon> He's waiting on improved windows support first? [20:43:05] <^demon> yuck. [20:43:11] <^demon> hashar: Yes, you told me :) [20:44:10] Ryan_Lane: do you know if it possible to save a puppet variable for a node? I am trying to generate a onetime password and would like puppetmasterd to magically remember about it :-D [20:44:42] you can't, but you can export a file [20:44:49] but we don't use exported resources a lot [20:44:57] ahh [20:45:00] among other things they're not very good performance-wise [20:45:10] and in labs we don't support exported resources, really [20:45:13] I was looking / rewriting the gerrit class that chad started on [20:45:20] oh [20:45:22] I need to review that [20:45:25] and hit an issue with /root/.my.cnf not being generated on labs [20:45:38] <^demon> Ryan_Lane: Yeah, I know you'd been busy so I asked hashar to take a crack at it :) [20:45:39] so I though I could generate() a password and save it somewhere [20:45:44] fuuuuuuuuuuck I have *so many review requests* [20:45:56] heh [20:46:16] if(!@@mysql_root_password) { @@mysql_root_password = generate( '/usr/bin/openssl rand -base64 16' ) } [20:46:16] were you popular in school? :) [20:46:19] that does not work :-) [20:46:24] of course not [20:46:35] that is all because you are an expert [20:46:43] to the super n00b I am, that looks beautiful :-) [20:46:47] (though totally non sense) [20:46:55] didn't I say that you can't do that? :) [20:47:07] you can only export resources, e.g. a file [20:47:12] <^demon> paravoid: It's what he gets for being user #1 :) [20:48:20] paravoid: look at the iowait on virt8 [20:48:28] Ryan_Lane: nova-api.log doesn't say anything [20:48:56] there's something going really bad on that box [20:49:06] ouch [20:49:19] *ouch* [20:49:56] I just installed iotp [20:50:00] *iotop [20:50:35] CONFIG_TASK_DELAY_ACCT not enabled in kernel, cannot determine SWAPIN and IO % [20:50:38] dammit [20:50:39] ^demon: so basically I got the gerrit class overhauled using roles just like on varnish [20:50:54] <^demon> followups to my change, or a new change? [20:50:55] ^demon: most of the code is still in manifests/gerrit.pp [20:51:08] any idea why *all* the VMs are writing stuff like crazy? [20:51:15] I have no clue [20:51:16] ^demon: as new patch sets to your change, but feel free to abandon it and open a new change using the latest patchset: ) [20:51:16] it's not much [20:51:18] that's really strange [20:51:25] it's like 20MB/s in total [20:51:34] even less on average [20:52:03] ^demon: sending a last patchset [20:52:04] virt7 has a similar number of instances [20:52:13] and way less writes [20:52:22] that's what I was just looking at [20:52:45] <^demon> hashar: Oh, that looks much better. Did you get it working on the new instance? [20:52:45] ^demon: maybe I should add some inline comments [20:52:59] paravoid: you rebooted i-0000020b ? [20:53:04] I tried to [20:53:10] 2012-07-30 20:47:44,737 AUDIT nova.api.cloud [2df1cfa8-f267-4028-b088-bf99ab1859d5 laner queue] Reboot instance [u'i-0000020b'] [20:53:15] in nova-api.log [20:53:36] ^demon: na it does not work on the gerrit-puppet-overhaul instance because installing the Gerrit database in mysql depends on a File["/root/.my.cnf"] [20:53:43] 2012-07-30 20:47:44,942 DEBUG nova.rpc [-] Making asynchronous cast on compute.virt6... from (pid=14054) cast /usr/lib/pymodules/python2.6/nova/rpc/impl_kombu.py:747 [20:53:46] <^demon> Ah yeah, that. [20:53:49] ^demon: in production that is provided using a template and a variable from the private repo [20:54:08] ^demon: so I thought on labs I could generate a onetime password and save that pass in puppet master [20:54:36] ^demon: then on a second run attempt to load the pass from master :D but puppet does not have a support for that [20:54:55] paravoid: I mentioned earlier that virt6's nova-compute process showed as down [20:55:02] <^demon> The other thought is maybe (for now), having the labs install use H2 instead of MySQL. [20:55:10] I just restarted it, it's coming up [20:55:17] which should make your reboot occur [20:55:19] <^demon> Ideally, we'd have some db9-esque service we can just point the labs instance to. [20:55:26] <^demon> Rather than having to install it on the same host. [20:55:28] oh? I didn't see that, very sorry [20:55:31] ^demon: note the instance is setup using puppet master self. /var/lib/git/operations/puppet has an https remote named "anonymous" for us to easily fetch a patchset from gerrit [20:55:33] paravoid: no worries [20:55:34] why is that though? [20:55:48] and shouldn't we have a nagios or something for that? [20:55:48] ^demon: we have plans for a database as a service service [20:55:54] ^demon: well you can still setup a gerrit-db instance :) [20:55:55] yeah, we should [20:56:03] okay, noted on my TODO [20:56:11] ^demon: we have a puppet class to install a mysql server on labs [20:56:11] paravoid: welcome to the wonderful world of my todo list being too long :) [20:56:22] hehe [20:56:48] <^demon> Ryan_Lane: I know. I'm just bellyaching that it's not available now :) [20:57:00] ^demon: heh [20:57:07] !log gerrit gerrit-puppet-overhaul : cd /var/lib/git/operations/puppet && git fetch anonymous refs/changes/84/13484/10 && git checkout -b 13484/10 [20:57:08] Logged the message, Master [20:57:10] <^demon> Also, I can't seem to delete instance i-00000335. I try, but it says it doesn't exist anymore. [20:57:18] <^demon> But it's stuck in my instance list :( [20:57:26] ^demon: if it was on virt6. that's why [20:57:28] ^demon: try now [20:57:52] <^demon> "The requested host does not exist." [20:58:18] <^demon> Oh, ok. Now it's gone from my list. [20:58:20] <^demon> Magic. [20:58:49] yeah, I just brought virt6 back up :) [20:59:09] !log gerrit {{gerrit|13484}} patchset 10 is currently stuck at "Could not find dependency File[/root/.my.cnf] for Exec[create_gerrit_db_user] at /etc/puppet/manifests/gerrit.pp:62". Caused by role::gerrit::labs when trying to apply the Gerrit changes. Maybe the role should be split and we could use a gerrit::role::database or something. [20:59:10] Logged the message, Master [20:59:31] magic indeed [20:59:41] Ryan_Lane: so, since that one's settled now [20:59:52] Ryan_Lane: I was thinking of doing some migrations this week [21:00:07] I started with one of my own, then I'll do all of the VMs belonging to ops members [21:00:16] and hashar with whom I speak every day :) [21:00:18] then what? [21:00:28] unannounced downtime? [21:00:33] any ideas? :) [21:00:50] I announced migrations [21:00:57] and that it would cause instance reboots [21:00:59] in general you mean [21:01:03] so, we should just do them [21:01:05] if you migrate the deployment-prep instances, please do so asap :-) [21:01:08] I am out by friday [21:01:17] we can do all of them right now, probably [21:01:24] it's scriptable [21:01:25] though they are all puppetized by know [21:01:30] beside video02 [21:01:38] I think they're already migrated [21:01:45] all of them? [21:01:48] they have a shit-ton [21:01:49] and only an integration thing is left [21:02:07] would be nice to do the bots cluster all as a group [21:02:13] I try to ssh @bastion.wmflabs.org -L 8080::80 and get the error open failed:administratively prohibited [21:02:16] sometime petan and Damianz are ready for it :) [21:02:21] any help is appreciated :) [21:02:30] no, some of them are not but they belong to petrb [21:02:38] * Ryan_Lane nods [21:02:40] I know [21:02:55] we can't contact everyone individually, though [21:03:18] sure [21:03:22] I really wish we had the damn email relay up [21:03:27] then we could email the project [21:03:35] anyway, maybe we should solve virt8 problems first... [21:03:39] yes [21:03:52] before I put more VMs there :) [21:04:14] indeed [21:04:18] let me check for a failed disk [21:05:22] hrm, I straced a VM and it just died [21:05:25] wow. [21:05:34] no failed [21:05:37] our KVM version must be horribly fucked up [21:05:44] well, it's lucid's [21:05:49] and it's old as hell [21:05:57] no failed drives [21:06:04] Ubuntu's ex-kvm maintainer was doing a very crappy job [21:06:12] :( [21:06:30] I think I complained *a lot*, I even complained on a conference on a session about collaboration between Debian-Ubuntu [21:06:38] I think you know him :P [21:06:46] it's Dustin Kierkland or something like that [21:06:55] he was at the openstack conference, you introduced me to him [21:07:00] (I think) [21:07:04] likely [21:07:41] I meet too many people there. I always have a hard time keeping track of who is doing exactly what outside of openstack itself [21:08:41] it's funny. I'm looking at some of the instances [21:08:48] and I don't see them actually doing anything [21:08:57] wow [21:09:01] we're not using lucid's kvm [21:09:10] oh. we're using the one from the ppa? [21:09:11] we're using a bastardized interim version that the diablo PPA shipped [21:09:25] 0.14 [21:09:35] bleh. we really need to upgrade [21:09:35] lucid had 0.12, precise has 1.0 [21:09:40] yeah [21:09:47] I can't wait to start using ubuntu's stable repo [21:10:01] i-00000080 is corrupt [21:10:23] I'm going to virsh destroy it [21:11:28] i-000000c1 is corrupt too [21:11:48] these were migrated instances [21:12:20] i-000000e7 isn't responding [21:12:47] looks like it had a segfault [21:13:04] I'm going to try rebooting it [21:13:51] pretty sure it's another corrupted one [21:13:54] lemme check the list [21:14:16] The corruption in labs is dreadful [21:14:16] :P [21:14:20] heh [21:14:29] well, these were because of the block migrations [21:14:31] JeroenDeDauw: are you following this? [21:14:55] * jeremyb gets ready to run out the door [21:15:12] Which reminds me, I need to god damn get around to writing my backup scripts [21:15:14] e7 ws indeed on the list [21:15:40] as mentioned in the labs-l post, I'm more than willing to pull any data, if you need it [21:15:42] bye! [21:16:41] i-000002dd is also on the list [21:17:12] so is -00000263 [21:18:20] ok. I killed all the corrupted ones [21:25:01] paravoid: so, whatever caused this happened sometime this morning: http://ganglia.wikimedia.org/latest/?r=day&cs=&ce=&c=Virtualization+cluster+pmtpa&h=virt8.pmtpa.wmnet&tab=m&vn=&mc=2&z=medium&metric_group=ALLGROUPS [21:25:32] maybe [21:25:34] I dunno [21:26:06] paravoid: all of the instances are basically idle, from the ones I'm checking [21:26:14] I've checked about half on the host [21:26:28] j^: which instance is this? [21:26:34] instancd id [21:26:36] *instance id [21:27:01] 36a [21:27:30] well, that's not causing our issue :) [21:27:57] yeah, seems unrelated [21:28:08] hrm [21:30:54] looks like i-0000036c failed to build [21:30:55] Change on 12mediawiki a page OAuth/status was modified, changed by RobLa-WMF link https://www.mediawiki.org/w/index.php?diff=566958 edit summary: new status update [21:31:26] OrenBo: around? I'd like to delete an instance you created that failed to build [21:31:31] unrelated to the problem, unfortunately [21:32:16] Jul 30 21:20:03 i-00000375 nslcd[13116]: [61c7ca] "Debian-exim": name denied by validnames option [21:32:17] bleh [21:45:18] paravoid: found it [21:45:22] what is it? [21:45:29] it's two instances [21:45:43] 00000375 and 00000322 [21:45:48] sorry for not helping more, it's getting late and I'm more capable of chatting rather than actually working right now [21:45:52] it's cool [21:45:59] 375 is an osm postgres node [21:46:04] yay [21:46:06] and it's loading a shit-ton of data into instance storage [21:46:52] Any problems? [21:47:04] apmon: yeah. it's causing issues on the host :( [21:47:15] :-( [21:47:18] iowait of 80% or so [21:47:53] Is it still causing problems? The import failed a couple of hours ago, so it shouldn't be doing anything anymore [21:48:04] yeah [21:48:10] it's writing a lot [21:48:19] That's odd. Let me check what it is still doing then [21:48:22] like 30M/s [21:48:54] Btw, is there a reason why / is mounted ext4 and /mnt mounted ext3? [21:49:14] because ext4 is the default for ubuntu precise [21:49:24] err [21:49:25] for ubuntu [21:49:36] and ext3 is what openstack chooses to use [21:49:46] I could probably switch that to xfs or ext4 [21:49:56] you can also reformat it however you'd like [21:50:41] postgres is definitely still doing things [21:50:56] is there any way you can load less data in the database, since this is for testing purposes? [21:51:15] I'm going to stop postgres [21:51:25] I am stoping postgres already [21:51:33] <3 xfs [21:51:44] the writer process is still going [21:51:45] heh [21:52:00] there it goes [21:52:24] and the iowait on virt8 is now 1% [21:52:24] heh [21:52:29] Sorry [21:52:39] no worries. I was able to track it down [21:52:49] Wonder if we could use cgroups to limit io per vm :D [21:52:49] databases in virtual machines are really bad at IO [21:53:05] With regard to large data import. The thing I was trying to test were optimisations that only have a gain with high data sizes [21:53:14] Ryan_Lane: how did you track it down btw? [21:53:18] It assumes a dense ID space, which isn't the case for smaller extracts [21:53:42] paravoid: iostat [21:53:47] heh [21:53:49] I sorted by writes [21:53:55] and made my font really small [21:54:02] so that I could see the instance name [21:54:02] I noticed that the kjournald and flush-XXX processes were using very high CPU load. Is that normal? [21:54:22] apmon: on a virtual machine doing a ton of io, likely [21:54:49] paravoid: we need to switch to virtio [21:54:52] why do they use more CPU on a vitual machine? [21:55:02] overhead [21:55:31] we don't use virtio? [21:55:34] that sucks [21:55:36] yep [21:55:44] although I'd probably wait until after the precise upgrade [21:55:51] it has gotten a lot better [21:55:56] indeed [21:56:00] that's what I was waiting for [21:56:13] Is there a way to shutdown an instance? I am only using the instances intermittently and so inbetween there is no need to use up RAM on the host [21:56:14] I was also waiting on the ciscos [21:56:23] since they have better hardware support for it [21:56:36] apmon: it's better to keep it running [21:56:51] apmon: it makes it easier for us to track memory usage [21:57:33] So weird that cisco do servers [21:57:39] an instance that's turned off is hidden memory :) [21:58:12] The alternative was to delete and recreate the instances. [21:58:19] no need [21:58:31] We're not /that/ short on ram :D [21:58:40] just keep it running and use it when you need it [21:58:44] delete it when you're totally done [21:58:59] OK. [21:58:59] hell, soon we'll have a *lot* of ram [21:59:02] IO is more of a problem [21:59:12] Yay, 128GB instances? [21:59:13] :D [21:59:26] Totally would solve io issues with mysql [21:59:31] I'm not sure I'm going to allow instance flavors that large ;) [21:59:51] a database service would fix that more than larger instances [22:00:02] Indeed [22:00:04] though we'd still likely run into issues like today with a database service [22:00:26] mhm [22:00:45] Would be interesting to have some form of monitor on mysql queries though [22:01:11] I'm thinking of using containers for the database service [22:01:14] Did you see the haproxy openstack service manager thing? Looked kinda cool [22:01:15] so we may be able to limit io [22:01:30] Damianz: you mean the new load balancer service written in python? [22:01:34] Yeah [22:01:39] Does like haproxy/cisco lbs [22:01:46] yeah. I'd like to write a pybal driver for it [22:02:08] we could definitely use haproxy or some other reverse proxy, though too [22:02:16] pybal would be kinda cool [22:02:28] Have a secret <3 for haproxy though, saves my ass so many times [22:02:56] well, I <3 pybal because it does the HA between lvs servers using BGP [22:03:04] which means no need for pacemaker/corosync [22:03:37] I had a patch in nova-network for using BGP to advertise floating IPs [22:03:46] That is the best thing about pybal [22:03:51] it got rejected. I need to wait till quantum has floating IP support [22:04:57] Pacemaker is freaking awesome, if you're dealing with stuff that needs STONITH kinda handling, loadbalencers a 5second outage for a re-arp is painful :( [22:05:21] it's configuration makes me want to punch kittens [22:05:48] hm. 12 people with 10+ nominations for openstack board now [22:06:12] I have a feeling it'll be 16-20 before nominations close [22:06:15] No like the xml managed via binaries? [22:06:26] Damianz: I despise it [22:06:38] Tbh I did prefer the heatbeat style of config [22:06:50] Pacemaker is so much more like RHCS though and much more powerful [22:06:58] RHCS is a real whore to manage [22:07:24] pybal is one of the easier ones I've used [22:07:48] and it's failover is much quicker than waiting on arp [22:11:13] BGP is pretty awesome, don't use it so much in production as our datacenters are connected over MPLS so the vlans that span both can just arp out. Would be interesting to balance/shift traffic cross POPs though to lbs. [22:28:41] Ryan_Lane: which instance [22:29:05] umm [22:29:20] I-0000036c [22:31:14] ok [22:31:27] I'll recreate it [22:31:31] cool [22:31:36] actually, can you delete it? [22:31:39] I'm not in the project [22:31:40] yes [22:37:14] sec - how is that possible! [22:38:15] ok it is deleted