[00:05:56] New patchset: Sara; "lvs for labs #5." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1585 [00:06:09] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/1585 [00:06:41] New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1585 [00:06:41] Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1585 [00:17:03] Ryan_Lane: did someone else puppetize adminbot while i was on hiatus? [00:43:57] New patchset: Ryan Lane; "Adding hack for array in ldap" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1586 [00:44:10] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/1586 [00:44:14] New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1586 [00:44:14] Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1586 [00:46:34] New patchset: Ryan Lane; "Put this in the right spot, this time" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1587 [00:46:46] New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1587 [00:46:46] Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1587 [00:46:46] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/1587 [00:48:03] New patchset: Ryan Lane; "Fix typo. Please kill me now." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1588 [00:48:15] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/1588 [00:48:19] New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1588 [00:48:19] Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1588 [00:50:00] hyperon: nope [00:50:05] it's still needing to be done :) [00:51:18] New patchset: Ryan Lane; "I swear, this is the last commit for this. If not, you'll find me hanging somewhere." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1589 [00:51:31] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/1589 [00:51:38] New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1589 [00:51:39] Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1589 [00:58:58] New patchset: Pyoungmeister; "more along the search overhaul path." [operations/puppet] (testlabs/searchoverhaul) - https://gerrit.wikimedia.org/r/1590 [00:59:09] New review: gerrit2; "Lint check passed." [operations/puppet] (testlabs/searchoverhaul); V: 1 - https://gerrit.wikimedia.org/r/1590 [18:30:01] petan: wm-bot is throwing exceptions in -operations :( [18:30:08] @commands [18:30:20] @drop #wikimedia-operations [21:36:31] ah. there it is [21:36:57] No, it just doesn't trust you [21:37:09] @commands [21:37:17] @logonn [21:37:22] weird [21:37:25] it should trust me cloak [21:42:31] @logon [21:43:52] @logon [21:43:57] aww me too [21:47:30] @search tunnel [21:48:05] !socks-proxy [21:48:17] !help [21:48:25] !wm-bot [21:52:18] Ryan_Lane: vvuksan is the developer for the ganglia web interface. [21:52:31] New patchset: Hashar; "basic class to install Android SDK prerequisites" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1614 [21:52:34] ? [21:52:39] (he just joined the channel) [21:52:43] ah [21:52:45] cool [21:52:47] (that was an introduction) [21:52:49] ;) [21:52:51] heh [21:52:53] :-) [21:53:08] vvuksan: Ryan_Lane is the lead labs dev. [21:53:16] sweet [21:53:36] so what would be cool is to have a VM that has access to the Production Ganglia feed [21:53:48] ie. ability to connect to prod gmonds [21:53:59] we were talking about using labs to test the ganglia web ui since it's hard to write for large sites without access to a large amount of data. [21:54:09] ah. cool [21:54:18] do you know if you can get to our aggregators from within labs? [21:54:23] I wonder if it would have access... [21:54:31] it just needs multicast access [21:54:46] I think it needs unicast access to the aggregators. [21:54:47] we may filter that traffic [21:54:53] not multicast access. [21:55:01] we likely do filter that traffic [21:55:05] LeslieCarr: ? ^^ [21:55:28] yeah [21:55:31] TCP [21:56:06] I'd imagine listening on the multicast address would be less of a headache than inbound access to the aggregators [21:56:31] ok [21:56:42] but I could be wrong [21:56:48] if i can see prod multicast [21:56:50] that's fine [21:56:52] mark and LeslieCarr would be good to get into this chat :) [21:57:02] I just tested - it does work. [21:57:08] :) [21:57:16] ah. cool [21:57:21] multicast? [21:57:21] (for an aggregator wiht a public IP, that is) [21:57:27] no, tcp. [21:57:27] yeah [21:57:31] public IPs will work [21:57:39] private will most likely not [21:57:49] confirmed that as well - private does not work. [21:58:07] security boundary :) [21:58:17] but of our aggregators, 15 clusters are public [21:58:23] yep [21:58:25] that's probably enough data for a good test set. [21:58:28] hey Ryan_Lane [21:58:32] howdy howdy [21:58:34] sorry, I ran off to get lunch [21:58:36] reading now [21:58:43] no worries [21:58:48] I'm supposed to be in a meeting right now [21:58:55] but no one is responding to me [21:59:01] so they may be SOL [21:59:26] hehe okay [21:59:45] so it looks like maplebed alredy tested that it would work for vvuksan… so that's good :) [21:59:54] for the public groups [22:00:13] yeah, we have a generic ban for the private groups on going to labs [22:00:27] we figure labs isn't a secure enough infrastructure to allow access to our soft chewy center [22:00:48] yep [22:01:02] vvuksan: one note though - labs instances don't have public IPs; the only way to get to a web ui is by tunneling through ssh. [22:01:12] ok [22:01:12] well, we can give out public ips [22:01:13] means it's good for your testing, not so good for exposing to other people. [22:01:16] that's doable [22:01:18] if it *really* needs it [22:01:31] oh, ok. nevermind. [22:02:01] we do have a few public ip's around.. just we try not to give them out very often since we're a bit low on ip space [22:02:10] but making ganglia more awesome would be a worthy goal [22:02:50] ;) [22:03:26] ok, back to swift. [22:04:13] if only people with labs accounts need to get to the interface, then we should keep it on a private IP and use a socks proxy [22:04:40] if everyone in the world should be able to see it, we can give it a public ip [22:06:46] vvuksan: let's get you set up with a labs account, though [22:06:51] sweet :-) [22:06:53] I can make you a project as well [22:07:24] btw do you guys use Nagios ? [22:07:33] Yes we do [22:07:41] http://nagios.wikimedia.org [22:07:50] I need the following from you: 1. your preferred wiki user name (this is also your git username, so if you like your git commits to use your realname, this should be your real name). 2. Your preferred shell account name (or your wikimedia svn account name, if you have one) 3. your preferred email address. [22:08:23] vvuksan: I've currently got ganglios instaled on our nagios server, though testing out the newer stuff in labs would be ossm. [22:09:01] dunno if you saw this [22:09:03] http://sourceforge.net/apps/trac/ganglia/wiki/ganglia_nagios_integration [22:09:22] yeah, I think that's what I mean by 'the newer stuff'. [22:09:22] i also have a Nagios bot :-) [22:10:02] cool [22:11:23] oooo… those new checks look nice [22:12:26] :-) [22:12:54] also jbuchbinder has Nagios API patches so we'll be bringing alerts onto graphs [22:13:03] and being a bit more intelligent with checks [22:13:10] labs-home-wm: ...? [22:13:11] 12/16/2011 - 22:13:10 - Creating a project directory for ganglia [22:13:13] heh [22:14:10] labs-home-wm: shouldn't you be creating me a homedir? [22:14:17] labs-home-wm: I will *stab* you [22:14:48] oh. wow. [22:14:53] there's no project members [22:14:56] that's a bug [22:15:11] 12/16/2011 - 22:15:10 - Creating a home directory for laner at /export/home/ganglia/laner [22:16:08] I guess that new feature I added for project creation isn't working correctly [22:16:10] 12/16/2011 - 22:16:10 - Updating keys for laner [22:20:59] vvuksan: so, once you log in, you'll be able to create instances in your project [22:21:08] k [22:21:10] make as many as you need [22:21:33] storage is slightly limited right now, and memory is our limiting factor, generally for how many instances we can have over all [22:21:39] so, take that in mind when creating things [22:21:55] but don't be too frugal. tiny instances are usually fairly useless [22:22:41] ok [22:23:58] we're planning on expanding the cluster based on usage anyway, so use what you need :) [22:24:29] i'm a light user :-) [22:25:28] heh. ok [22:30:27] PROBLEM host: firewallclient is DOWN address: firewallclient CRITICAL - Host Unreachable (firewallclient) [22:31:27] PROBLEM host: firewallserver is DOWN address: firewallserver CRITICAL - Host Unreachable (firewallserver) [22:32:34] hi ryan, i have another apache config problem :( (really sorry for bothering you with this kind of mundane stuff) [22:32:47] oh? what's up? [22:33:18] on pageviews, i get a 500 error saying it cannot import a file config, the file is in the folder and the rights are correct (i think) [22:33:41] tail /var/log/apache2/stats.grok.se_error.log [22:34:19] maybe you are missing a python module? [22:34:30] but it is in the folder [22:34:36] ah. right [22:34:51] I can indeed import it when in that directory [22:34:59] ah [22:35:01] I see [22:35:05] :( [22:35:10] how about now? [22:35:19] nope [22:36:22] WSGIScriptAlias / /srv/stats.grok.se/web/index.py [22:36:26] that doesn't seem right [22:36:52] maybe i should set the documentroot [22:37:01] that said, I don't know how to configure wsgi [22:37:29] i am pretty sure that WSGIScriptAlias is correct [22:37:33] because it can find index.py [22:37:44] but it cannot import config from index.py [22:38:35] I dunno [22:38:55] I think you need to set the documentroot [22:39:04] config.py doesn't need to be executable [22:39:13] and it definitely shouldn't be writable by www-data [22:39:20] none of the source code should be [22:39:26] that's dangerous [22:39:42] ok [22:39:44] yah, no reason for anything there to be writable. [22:39:49] indeed [22:39:58] henrik: documentroot is set [22:40:32] config.py no longer executable [22:40:33] writable code in the docroot can mean a web-server getting owned if there is a bug that can allow writes to the directory [22:40:43] sorry [22:41:33] I haven't gotten the new stats.grok.se-code up and running properly in my local wsgi though, I still think there are some more code tweaks that need to be done [22:43:16] i think i found the problem [22:43:36] let me tweak a bi [22:43:37] t [22:43:44] Change abandoned: Hashar; "nothing to do in test. this is intended for production." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1614 [22:51:02] Ryan_Lane: oh [22:51:04] sorry [22:54:18] @trusted [22:54:22] yay [22:54:38] @trustadd .*wikimedia/.* trusted [22:54:49] I will fix it, sorry, Ryan_Lane [22:56:20] drdee: just pushed another structuring commit. index.py has gone from 388 to 169 lines now, always great when you can remove more code than you add :) [23:01:07] PROBLEM host: firewallclient is DOWN address: firewallclient check_ping: Invalid hostname/address - firewallclient [23:01:47] PROBLEM host: firewallserver is DOWN address: firewallserver check_ping: Invalid hostname/address - firewallserver [23:03:55] petan: heh [23:27:20] @trusted [23:27:20] I trust: petan!.*@wikimedia/Petrb, [23:27:27] @commands [23:27:27] Commands: channellist, trusted, trustadd, trustdel, infobot-off, infobot-on, drop, add, reload, logon, logoff [23:27:36] @search labs [23:27:36] Results: puppet, instance, git, nagios, labs, bastion, projects, socks-proxy, monitor, alert, $realm, [23:27:40] ok [23:28:12] @trustadd .*@wikimedia/.* trusted [23:28:12] Successfuly added .*@wikimedia/.* [23:28:27] @trustadd .*@mediawiki/.* trusted [23:28:27] Successfuly added .*@mediawiki/.* [23:28:29] @trustadd .*@wikimedia/Ryan-lane admin [23:28:29] Successfuly added .*@wikimedia/Ryan-lane [23:28:49] thanks [23:30:32] @add #wikimedia-operations [23:31:07] PROBLEM host: firewallclient is DOWN address: firewallclient check_ping: Invalid hostname/address - firewallclient [23:31:47] PROBLEM host: firewallserver is DOWN address: firewallserver check_ping: Invalid hostname/address - firewallserver [23:33:10] New patchset: Lcarr; "fixing up ganglia.pp" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1616 [23:33:23] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/1616 [23:34:07] New review: Lcarr; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1616 [23:34:07] Change merged: Lcarr; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1616 [23:35:04] fix [23:35:04] !g test [23:35:04] https://gerrit.wikimedia.org/r/test [23:35:07] !g test [23:35:07] https://gerrit.wikimedia.org/r/test [23:35:11] good