[02:13:53] PROBLEM Current Load is now: CRITICAL on test2 test2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:14:33] PROBLEM Current Users is now: CRITICAL on test2 test2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:15:13] PROBLEM Disk Space is now: CRITICAL on test2 test2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:16:03] PROBLEM Free ram is now: CRITICAL on test2 test2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:17:23] PROBLEM Total Processes is now: CRITICAL on test2 test2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:18:13] PROBLEM dpkg-check is now: CRITICAL on test2 test2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:40:13] RECOVERY Disk Space is now: OK on test2 test2 output: DISK OK [02:41:03] RECOVERY Free ram is now: OK on test2 test2 output: OK: 64% free memory [02:42:23] RECOVERY Total Processes is now: OK on test2 test2 output: PROCS OK: 78 processes [02:43:13] RECOVERY dpkg-check is now: OK on test2 test2 output: All packages OK [02:43:53] RECOVERY Current Load is now: OK on test2 test2 output: OK - load average: 0.00, 0.06, 0.07 [02:44:33] RECOVERY Current Users is now: OK on test2 test2 output: USERS OK - 0 users currently logged in [05:26:02] PROBLEM dpkg-check is now: CRITICAL on bots-apache1 bots-apache1 output: CHECK_NRPE: Socket timeout after 10 seconds. [05:26:02] PROBLEM Free ram is now: CRITICAL on bots-apache1 bots-apache1 output: CHECK_NRPE: Socket timeout after 10 seconds. [12:04:11] petan: the IRC logs don't log channel joins and parts? [12:14:41] New patchset: Dzahn; "add SSLCACertificateFile to Apache config for valid SSL certificate chain" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1755 [12:14:59] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/1755 [12:16:18] New patchset: Dzahn; "add SSLCACertificateFile to Apache config for valid SSL certificate chain" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1755 [12:18:29] New review: Dzahn; "now just need to import the Labs CA into browser and all fine, no security exceptions needed." [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1755 [12:18:29] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1755 [12:43:25] New patchset: Dzahn; "configurable version for MariaDB - different approach" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1756 [12:43:38] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/1756 [12:44:00] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1756 [12:44:00] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1756 [12:46:26] yay [12:46:29] notice: /Stage[main]/Misc::Mariadb::Client/Package[mariadb-client-5.3]/ensure: ensure changed 'purged' to 'latest' [12:47:31] !log wikistats installed MariaDB server and client via new puppet class [12:47:32] Logged the message, Master [12:47:36] mutante, have you fallenout with mysql? [12:47:37] ;) [12:48:03] Reedy: yeah:) [12:48:09] and it was like a labs policy too [12:48:19] mysql yes, if we use maria [12:48:27] But Oracle! [12:48:32] hehe [12:49:10] there was also the "why not drizzle" question [12:49:35] but maria is really just "drop-in replacement", no code changes [12:49:40] I shall have words with Ryan for this heracy [12:49:49] :) [12:53:17] btw, using "system_role{.." adds messages to the server MOTD in production, it drops files into /etc/update-motd.d/ on labs too, just the default labs instances don't read from there [12:53:36] eh.or don't run the update.. i should check [12:58:29] disregard that completely. it works fine [13:01:43] RECOVERY Disk Space is now: OK on puppet-lucid puppet-lucid output: DISK OK [13:03:47] New patchset: Dzahn; "minor fix in system_role for MariaDB repo" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1757 [13:04:01] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/1757 [13:04:12] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1757 [13:04:12] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1757 [13:09:43] PROBLEM Disk Space is now: WARNING on puppet-lucid puppet-lucid output: DISK WARNING - free space: / 74 MB (5% inode=35%): [13:11:37] ok, i moved those MariaDB classes to the "DB" section [13:13:10] if you want to use it, include misc::mariadb::repository AND one or more of "client" and "server", set mariadb_version to "5.3". then it works for me fully puppetized, adds the repo, APT key, then installs packages [13:13:57] Finished catalog run .. root@i-00000042:~# mysql ... Welcome to the MariaDB monitor. [13:24:15] New patchset: Dzahn; "wikistats - get php5-mysql from generic class" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1758 [13:24:28] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (test); V: -1 - https://gerrit.wikimedia.org/r/1758 [13:25:46] New patchset: Dzahn; "wikistats - get php5-mysql from generic class" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1758 [13:26:01] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/1758 [13:26:07] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1758 [13:26:07] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1758 [14:35:18] New patchset: Dzahn; "do not require generic::mysql::client - this removes MariaDB now" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1759 [14:35:33] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/1759 [14:35:35] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1759 [14:35:36] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1759 [14:45:43] New patchset: Dzahn; "add an exec to ensure file permissions in doc root" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1760 [14:46:15] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1760 [14:46:16] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1760 [14:53:22] New patchset: Dzahn; "do not install index.php via puppet anymore - fix permission exec" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1761 [14:53:35] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/1761 [14:54:01] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1761 [14:54:01] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1761 [14:56:53] PROBLEM Disk Space is now: WARNING on nova-production1 nova-production1 output: DISK WARNING - free space: / 562 MB (5% inode=86%): [15:37:19] I learnt of your heracy today via mutante [15:59:32] Reedy? [15:59:40] Ryan_Lane, wtf is 'Delete on Instance Deletion' ? [15:59:49] where do you see this? [16:00:10] Openstackmanager-volumedeleteonvolumedelete message [16:00:20] ah [16:00:28] it makes sense in context ;) [16:00:49] "Do you want this volume to be deleted when its associated instance is deleted?" [16:01:34] thanks [16:01:36] yw [16:09:54] lol [16:12:15] is openstackmanager-novakey-info used anywhere? [16:12:20] only OpenStackManager.i18n.php seem to match [16:12:39] It's possible I deleted it [16:12:46] or that I didn't implement it [16:12:58] since we don't use keys that way [16:13:49] I'm sure there are a few messages that don't match. I removed some keys recently, and forgot to remove them from the i18n file [16:14:01] Reedy> I learnt of your heracy today via mutante [16:14:14] Bah, seemed I failed to even start to tab complete Ryan_Lane [16:14:14] Reedy: who's heracy? [16:14:18] :D [16:14:21] Yours [16:14:23] No MySQL!? [16:14:25] No Orcale? [16:14:27] How dare you. [16:14:30] eh? [16:14:34] MariaDB :p [16:14:42] mariadb is a dropin replacement for mysql [16:14:48] do you mean heresy? [16:14:54] Possibly [16:15:03] it has ldap support [16:15:08] My joke was still the same [16:15:24] Cause we all love oracle [16:15:25] so, instead of having to have yet another damn password, you can use the same one [16:15:42] So when you get one user hacked the entire cluster is theirs! [16:15:55] if the user gets hacked, they are fucked either way [16:16:05] lol [16:16:27] would they need to still store it in my.cnf? [16:16:39] or would the credential be "magically" passed? [16:16:42] seriously. If I wanted to know your mysql password, I'd just drop a keylogger into your account [16:16:59] they shouldn't store it at all, for their account [16:17:00] Actually that reminds me, I should change cluebots mysql pass [16:17:07] <^demon> Sqlite for everyone. No passwords needed! [16:17:23] Ryan_Lane: Thought of deploying krb out though? That would be awesomesauce [16:17:38] s/krb/kerb/ [16:17:39] you can authenticate by user when connecting using a fs socket [16:17:40] well, no databases have support for krb :( [16:17:57] Platonides: fs socket? [16:18:11] filesystem socket [16:18:18] Wth is a filesystem socket? [16:18:25] that assumes the database is local? [16:18:44] it would need it to be running in the same host [16:18:44] Ryan_Lane, people will use their LDAP passwords for MySQL? [16:19:02] MaxSem: people could, yes [16:19:04] what if it's a shared instance? [16:19:11] otherwise, you would need to begin with things like ident, which are much uglier [16:19:22] MaxSem: how's that a problem? [16:19:32] I guess we should require ssl :) [16:19:43] $wgDBuser = 'maxsem'; [16:19:52] $wgDBpassword = ??? [16:19:54] You'd create users still [16:20:06] Awww but I wanted to build a logging server to capture all your passwords :P [16:20:28] well, this is meant for databases that are central [16:20:35] not for local databases [16:20:55] I don't plan on having 1283478723894423 copies of enwiki around [16:21:40] wheee! [16:21:44] I'd be happy with one that works :P [16:21:44] <^demon> A could of enwikis :) [16:21:51] is there at least one? [16:21:54] Ruddy toolserver db [16:22:07] but anyway, this is for people doing queries against databases, not necessarily wikis [16:22:19] wikis should likely have their own accounts [16:22:33] with their own passwords, and they shouldn't be LDAP'd [16:22:47] I need a copy of enwiki, but I would also like to change its schema :) [16:23:28] why? [16:23:57] would replicating the tables with a view work for you? [16:24:04] I really need to package up this bot and write a decent puppet script for it for when the server implodes to hell :( [16:24:05] it's labs, after all:) [16:24:11] dunno about views [16:24:46] kind of a "symlink table" [16:24:52] I'd like to have short-lived read/write clones of databases [16:25:02] or long-lived clones that don't update [16:25:10] we'll have to see about that, though [16:25:26] does it support copy-on-write? [16:25:35] Damianz: oh, did I kill your bot last night? [16:25:45] a copy of enwiki is likely to be very big [16:25:48] I don't think so [16:25:49] Platonides: we might be able to use a filesystem that does. [16:25:53] even with text at ES [16:26:02] Or rather I didn't get spammed with emails, though the bot is on a nfs mount IIRC. [16:26:10] So you might have done and it got restarted before I got spammed [16:26:12] I rebooted a bunch of instances last night, due to problems on a node [16:26:14] * Damianz shrugs [16:26:33] one of the instances was the NFS home directory server [16:26:40] I've not re-written that gerrit ticket yet to use a misc class so it doesn't have proper monitoring. I'm not being chased with a pick axe though so it's all good. [16:26:42] so all of the instances had problems for a little while [16:26:58] Platonides: yeah, I was planning on COW [16:27:13] then only changes would eat up space [16:27:47] Ah some of the feeds are down [16:27:48] * Damianz restarts [16:28:02] we could put the original in hot copy mode, take a snapshot, then take it out of hot copy mode [16:28:18] then start the snapshot as a secondary database [16:29:58] Argh ffs [16:30:15] I hate irc.... why do I have to restar tthe whole bot to reconnect it to one server :( [16:31:40] Also as a side note udp socket servers in php for php to spam stuff to as a relay to irc cause hell [16:32:13] Damianz: you shouldn't need to restart the whole bot to reconnect it [16:32:33] my bots reconnect when disconnected [16:33:35] It's written around irc, so it can part/join channels when running but can't reconnect. [16:33:48] I'll get around to re-writing it using twisted and thread pools when I have a week free. [16:33:59] irclib has support for reconnecting [16:34:20] When it "tries" to fix it's self it actually causes issues sometimes because the udp binds hang :( [16:34:35] Yeah I know [16:34:39] The main bot is php though :'( [16:34:44] ahhhhh ok [16:34:50] Hence the recourse usage and a fork per edit [16:35:19] you should try evenlet rather than twisted [16:35:23] err [16:35:24] eventlet [16:35:35] Not seen that, I'll take a look. [16:35:49] I find twisted to be awkward to work in :) [16:35:57] and twisted takes ages to get support for things [16:36:05] like websockets, and ipv6 [16:36:08] I'm kinda wanting to take on improving cbng is a slightly selfish move that it would make a nice project to write stuff around/do a talk on xD [16:37:43] what's cbng? [16:37:55] en.wiki anti vandal bot [16:39:16] Ryan_Lane: I don't suppose you have a love for tomcat by any chance? [16:39:29] I don't feel strongly either way about it [16:41:11] It seems nice in some ways but it's also java =/ [16:41:24] it's just an appserver [16:41:58] an advantage is that it automatically daemonizes things for you [16:42:04] but, it's pretty easy to daemonize python [16:42:29] it's actually pretty annoyingly difficult to daemonize java without tomcat [16:43:52] I find java a bit of a weird lang to work with in many ways =/ Feel more love for Django than Wicket anyway. [16:44:09] I haven't used django yet [16:44:21] I haven't written any python web code at all :) [16:44:49] though I guess technically openstack software is http based [16:44:53] but it's api only [16:45:08] PHP frameworks make me want to punch kittens, Django + Celery just make sense for what I do with it which is mostly insane. [16:46:12] heh [16:46:30] I couldn't imagine trying to run something persistent in php [16:48:49] The most annoying thing about php is the amount of idiots there are that use it... like the guy who wondered why his site got exploited when a bunch of eval's+system's resulted in a wget, sh and his server brute forcing china *sigh* [16:49:21] :D [17:06:53] RECOVERY Disk Space is now: OK on nova-production1 nova-production1 output: DISK OK [20:01:12] petan: hello [20:09:01] cookies [23:39:02] I should probably sort out my labs account and my ssh keys later [23:42:19] Reedy: do you not have one yet? [23:42:32] I can link it for you right now if you'd like [23:43:18] I've got one setup [23:43:35] But I've not done anything with my ssh keys since your email saying not to use same key on cluster etc [23:46:17] Ryan_Lane: can you to add Accessing public and private instances some info on how to do it via windows?