[00:03:04] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [00:20:34] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [00:32:20] hashar: curl "https://gerrit.wikimedia.org/r/tools/hooks/commit-msg" > $TOPLEVEL/.git/hooks/commit-msg && chmod u+x $TOPLEVEL/.git/hooks/commit-msg [00:33:04] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [00:35:14] PROBLEM Puppet freshness is now: CRITICAL on mobile-feeds i-000000c1 output: Puppet has not run in last 20 hours [00:46:43] Can I get a beta cluster (labs) developer or steward please? [00:47:36] petan or petan_? [00:48:01] Thehelpfulone or THO|Cloud? [00:48:34] o_0 [00:49:09] What's up? [00:49:54] Reedy: We have a spammer on simple.wikipedia and I get a db error when I try to block. Petrb globally blocked then unblocked for some reason. [00:50:14] 10.4.0.17 [00:50:34] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [00:50:35] yep? [00:51:09] See above, Thehelpfulone. [00:51:28] The error is http://i.imgur.com/GKF72.png [00:52:23] works for me on deployment.wikimedia [00:53:36] where did you try to do it? [00:54:23] 10.4.0.17 sounds like a more local IP? [00:54:55] Thehelpfulone: I tried to block it locally. I said Petrb did this: http://meta.wikimedia.beta.wmflabs.org/w/index.php?title=Special%3ALog&type=gblblock&user=Petrb&page=User%3A10.4.0.17&year=&month=-1 [00:55:07] I have to go for a few mins, brb. [00:55:12] that's not a real IP though [00:58:23] 10.4.0.17 is a local one [00:58:24] RECOVERY Disk Space is now: OK on nagios 127.0.0.1 output: DISK OK [00:58:33] Sounds like MediaWiki doesn't know about the squids [00:59:10] wgSquidServersNoPurge [01:03:04] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [01:06:24] PROBLEM Disk Space is now: CRITICAL on nagios 127.0.0.1 output: DISK CRITICAL - free space: /home/dzahn 830 MB (4% inode=81%): [01:11:24] RECOVERY Disk Space is now: OK on nagios 127.0.0.1 output: DISK OK [01:13:59] I'm back [01:20:34] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [01:23:06] !log deployment-prep did I got fixed ? [01:33:04] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [01:50:34] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [02:00:24] RECOVERY Disk Space is now: OK on deployment-feed i-00000118 output: DISK OK [02:03:04] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [02:20:34] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [02:27:24] Ryan_Lane: I need the !log !!! ;:-D [02:30:59] He's ignoring you [02:33:04] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [02:48:58] RECOVERY Current Load is now: OK on deployment-apache08 i-0000025f output: OK - load average: 1.45, 1.65, 0.78 [02:49:28] RECOVERY Current Users is now: OK on deployment-apache10 i-00000260 output: USERS OK - 0 users currently logged in [02:49:28] RECOVERY Current Users is now: OK on deployment-apache08 i-0000025f output: USERS OK - 0 users currently logged in [02:49:58] PROBLEM Disk Space is now: WARNING on deployment-apache10 i-00000260 output: DISK WARNING - free space: / 76 MB (5% inode=59%): [02:50:28] PROBLEM Disk Space is now: WARNING on deployment-apache08 i-0000025f output: DISK WARNING - free space: / 75 MB (5% inode=59%): [02:50:38] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [02:50:48] RECOVERY Free ram is now: OK on deployment-apache10 i-00000260 output: OK: 62% free memory [02:50:58] RECOVERY Free ram is now: OK on deployment-apache08 i-0000025f output: OK: 62% free memory [02:52:38] RECOVERY Total Processes is now: OK on deployment-apache08 i-0000025f output: PROCS OK: 83 processes [02:52:43] RECOVERY Total Processes is now: OK on deployment-apache10 i-00000260 output: PROCS OK: 83 processes [02:53:18] RECOVERY dpkg-check is now: OK on deployment-apache10 i-00000260 output: All packages OK [02:53:38] RECOVERY dpkg-check is now: OK on deployment-apache08 i-0000025f output: All packages OK [02:53:48] RECOVERY Current Load is now: OK on deployment-apache10 i-00000260 output: OK - load average: 0.01, 0.59, 0.48 [02:54:18] PROBLEM Disk Space is now: CRITICAL on nagios 127.0.0.1 output: DISK CRITICAL - free space: /home/dzahn 828 MB (4% inode=81%): [03:01:08] RECOVERY Free ram is now: OK on demo-mysql1 i-00000256 output: OK: 93% free memory [03:03:08] RECOVERY Disk Space is now: OK on demo-mysql1 i-00000256 output: DISK OK [03:03:08] RECOVERY Total Processes is now: OK on demo-mysql1 i-00000256 output: PROCS OK: 95 processes [03:03:18] RECOVERY dpkg-check is now: OK on demo-mysql1 i-00000256 output: All packages OK [03:03:28] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [03:05:28] RECOVERY Current Load is now: OK on demo-mysql1 i-00000256 output: OK - load average: 0.02, 0.10, 0.07 [03:05:28] RECOVERY Current Users is now: OK on demo-mysql1 i-00000256 output: USERS OK - 0 users currently logged in [03:20:38] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [03:30:28] PROBLEM Disk Space is now: WARNING on deployment-apache09 i-0000025e output: DISK WARNING - free space: / 75 MB (5% inode=59%): [03:30:58] RECOVERY Free ram is now: OK on deployment-apache09 i-0000025e output: OK: 62% free memory [03:32:38] RECOVERY Total Processes is now: OK on deployment-apache09 i-0000025e output: PROCS OK: 83 processes [03:33:28] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [03:33:38] RECOVERY dpkg-check is now: OK on deployment-apache09 i-0000025e output: All packages OK [03:33:58] RECOVERY Current Load is now: OK on deployment-apache09 i-0000025e output: OK - load average: 0.01, 0.05, 0.08 [03:34:28] RECOVERY Current Users is now: OK on deployment-apache09 i-0000025e output: USERS OK - 0 users currently logged in [03:39:08] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 15% free memory [03:46:18] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 13% free memory [03:49:58] RECOVERY Disk Space is now: OK on deployment-apache10 i-00000260 output: DISK OK [03:50:38] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [03:53:58] RECOVERY Current Load is now: OK on deployment-apache06 i-0000025c output: OK - load average: 0.31, 0.13, 0.04 [03:54:28] RECOVERY Current Users is now: OK on deployment-apache06 i-0000025c output: USERS OK - 0 users currently logged in [03:55:28] PROBLEM Disk Space is now: WARNING on deployment-apache06 i-0000025c output: DISK WARNING - free space: / 75 MB (5% inode=59%): [03:55:58] RECOVERY Free ram is now: OK on deployment-apache06 i-0000025c output: OK: 61% free memory [03:57:38] RECOVERY Total Processes is now: OK on deployment-apache06 i-0000025c output: PROCS OK: 87 processes [03:57:58] PROBLEM Disk Space is now: WARNING on deployment-apache10 i-00000260 output: DISK WARNING - free space: / 75 MB (5% inode=59%): [03:58:38] RECOVERY dpkg-check is now: OK on deployment-apache06 i-0000025c output: All packages OK [03:59:08] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 3% free memory [04:03:28] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [04:03:28] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 16% free memory [04:04:08] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:06:08] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:06:18] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 13% free memory [04:14:18] RECOVERY Disk Space is now: OK on nagios 127.0.0.1 output: DISK OK [04:16:08] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [04:20:38] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [04:21:18] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 4% free memory [04:22:18] PROBLEM Disk Space is now: CRITICAL on nagios 127.0.0.1 output: DISK CRITICAL - free space: /home/dzahn 823 MB (4% inode=81%): [04:23:28] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 3% free memory [04:26:18] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 95% free memory [04:28:28] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 97% free memory [04:31:18] PROBLEM Free ram is now: WARNING on test3 i-00000093 output: Warning: 6% free memory [04:33:28] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [04:36:18] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [04:50:38] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [04:57:18] RECOVERY Disk Space is now: OK on nagios 127.0.0.1 output: DISK OK [05:03:28] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [05:20:38] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [05:33:28] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [05:50:38] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [06:03:31] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [06:03:45] hashar: why u created new apaches [06:21:51] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [06:27:41] PROBLEM Disk Space is now: WARNING on bz-dev i-000001db output: DISK WARNING - free space: / 40 MB (3% inode=43%): [06:33:43] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [06:33:49] PROBLEM dpkg-check is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:33:49] PROBLEM Disk Space is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:33:49] PROBLEM Current Users is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:33:49] PROBLEM Total Processes is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:33:54] PROBLEM Free ram is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:38:35] RECOVERY Disk Space is now: OK on incubator-bot2 i-00000252 output: DISK OK [06:38:35] RECOVERY dpkg-check is now: OK on incubator-bot2 i-00000252 output: All packages OK [06:38:35] RECOVERY Current Users is now: OK on incubator-bot2 i-00000252 output: USERS OK - 0 users currently logged in [06:38:35] RECOVERY Total Processes is now: OK on incubator-bot2 i-00000252 output: PROCS OK: 119 processes [06:38:43] RECOVERY Free ram is now: OK on incubator-bot2 i-00000252 output: OK: 47% free memory [06:38:58] hmm? [06:41:35] PROBLEM Disk Space is now: CRITICAL on nagios 127.0.0.1 output: DISK CRITICAL - free space: /home/dzahn 959 MB (5% inode=81%): [06:48:48] PROBLEM dpkg-check is now: CRITICAL on nova-precise1 i-00000236 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:51:31] PROBLEM Current Users is now: CRITICAL on pediapress-ocg1 i-00000233 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:51:32] PROBLEM dpkg-check is now: CRITICAL on pediapress-ocg1 i-00000233 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:52:45] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [06:53:45] RECOVERY dpkg-check is now: OK on nova-precise1 i-00000236 output: All packages OK [06:56:15] RECOVERY Current Users is now: OK on pediapress-ocg1 i-00000233 output: USERS OK - 0 users currently logged in [06:56:15] RECOVERY dpkg-check is now: OK on pediapress-ocg1 i-00000233 output: All packages OK [07:01:46] PROBLEM dpkg-check is now: CRITICAL on pediapress-ocg2 i-00000234 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:16] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [07:06:26] RECOVERY dpkg-check is now: OK on pediapress-ocg2 i-00000234 output: All packages OK [07:22:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [07:36:16] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [07:52:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [08:06:16] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [08:22:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [08:36:16] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [08:46:34] petan: 1 day ago – Nagios Core 3.4.0 was just released http://www.nagios.org/news [08:52:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [09:06:16] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [09:06:18] ok [09:09:54] mutante: not in repo [09:10:02] I would prefer to use .deb [09:18:06] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap1 i-000000df output: Puppet has not run in last 20 hours [09:19:55] petan: yea, that was just for your info [09:22:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [09:35:46] PROBLEM Total Processes is now: CRITICAL on labs-nfs1 i-0000005d output: CHECK_NRPE: Socket timeout after 10 seconds. [09:36:16] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [09:40:26] RECOVERY Disk Space is now: OK on deployment-apache06 i-0000025c output: DISK OK [09:40:36] RECOVERY Total Processes is now: OK on labs-nfs1 i-0000005d output: PROCS OK: 104 processes [09:48:26] PROBLEM Disk Space is now: WARNING on deployment-apache06 i-0000025c output: DISK WARNING - free space: / 75 MB (5% inode=59%): [09:52:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [10:06:16] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [10:22:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [10:36:06] PROBLEM Puppet freshness is now: CRITICAL on mobile-feeds i-000000c1 output: Puppet has not run in last 20 hours [10:36:16] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [10:51:50] !whitespace is don't want to see red in gerrit? | trailing with vim: :%s/\s\+$// | trailing with joe: \+ \$ | all with sed: sed 's/^[ \t]*//;s/[ \t]*$//' | ... [10:51:50] Key was added! [10:52:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [10:54:43] <^demon> mutante: You can also make git diff yell at you for it :) [10:57:18] ^demon: ah:) hi [10:57:32] ^demon: did you add the new project for apache config? [10:58:00] i shall look at the git diff yelling as well [10:58:16] <^demon> New project for apache config? [10:58:45] https://gerrit.wikimedia.org/r/#/q/status:open+project:operations/apache-config,n,z [10:59:02] <^demon> Nope, not me. [10:59:33] <^demon> I thought that apache config stuff was going to go in puppet? [10:59:37] <^demon> Why is it getting a new repo? [11:00:46] there is another new one for Mediawiki config [11:01:06] <^demon> That one's been there. [11:01:11] yea, your questions is totally valid, i don't know, but i do like it is in gerrit in _some_ way [11:01:26] Sam wrote about it like 3 hours ago though [11:01:37] <^demon> It's been empty since ~January ;-) [11:01:42] ok:) [11:02:01] <^demon> But re:apache....it's not *in* gerrit other than being committed. Until puppet deploys it it's absolutely useless sitting in its own repo there. [11:02:32] <^demon> And with it being in a non-puppet repo...how's puppet going to deploy it? [11:02:42] yes, that is true, but isnt it like chicken/egg problem [11:02:54] and at least you have the code review [11:03:16] instead of doing svn commit on fenari somebody could clone from repo and manually update it? [11:03:21] until puppet does it soon? [11:03:27] * ^demon shrugs [11:03:48] true about non-puppet repo [11:04:10] <^demon> I thought Mark H. had already pushed this stuff to puppet too and it was just pending review. [11:04:24] ack [11:04:37] the difference was he put in in apache/files/ in the operations/puppet [11:04:56] it sounded like there was discussion in between which ended in this new project being created [11:05:00] to me.. but i may be wrong [11:06:16] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [11:07:16] <^demon> Buttt, as far as who made this repo...it was anyone in https://gerrit.wikimedia.org/r/#/admin/groups/1,members [11:07:25] <^demon> (note the "included group" at the bottom too) [11:07:55] <^demon> My bets are on Roan or Antoine. [11:08:34] yea, and i could +1 but not +2, so i shall wait for more gerrit votes..hmmm [11:09:54] <^demon> That's cuz whoever set it up doesn't know that the 'wmf' group is really under-populated and is missing most staff/contractors. [11:10:01] <^demon> So granting rights to 'wmf' is near-useless :) [11:12:46] <^demon> Whoo, progress on plugins https://groups.google.com/d/msg/repo-discuss/2uRrpO40_Pw/a7Erwm36gKAJ :D [11:20:08] <^demon> mutante: I saw your latest e-mail, thanks for summarizing us :) [11:21:32] k:) [11:22:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [11:47:10] robiH: you have been confused with another robh ;) [11:47:13] thats why you werent on the project yet [11:48:34] !log wikistats added member RobiH (!= RobH) :p [11:48:34] That's what I thought in the first place, when I triegt to sign up. [11:48:34] 05/09/2012 - 11:48:03 - Creating a home directory for robih at /export/home/wikistats/robih [11:48:58] see the bot message, you got a home dir now [11:49:43] 05/09/2012 - 11:49:03 - Updating keys for robih [11:49:49] <^demon> Too many Robs around here. [11:55:17] Too many robs in the world [12:00:16] !log bots Restarted morebots due to some connection issue on Freenode servers. log-bottie remains down due to sudo policy (If only I had sudo access...) [12:01:40] !log bots Restarted morebots due to some connection issue on Freenode servers. log-bottie remains down due to sudo policy (If only I had sudo access...) [12:01:42] Logged the message, Master [12:03:58] !log wikistats running puppet and missing /home/robih [12:03:59] Logged the message, Master [12:10:19] What server is og-bottie on? [12:11:11] <^demon> localhost. [12:11:16] bots-labs [12:11:29] /etc/init.d/ircecho [12:12:09] !bot [12:12:10] http://meta.wikimedia.org/wiki/WM-Bot [12:12:12] !bots [12:12:13] http://www.mediawiki.org/wiki/Wikimedia_Labs/Create_a_bot_running_infrastructure proposal for bots [12:12:17] POWER UP [12:12:36] arr, where was the doc trigger again [12:12:37] * Damianz pats labs-logs-bottie and eats Hydriz, then returns to work. [12:12:42] !about [12:12:42] https://www.mediawiki.org/wiki/Wikimedia_Labs here you can find more [12:12:48] we have one for "bot down?" [12:12:55] i made it but forgot [12:12:57] SCREW, I WANT SUDO :( [12:13:11] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots/Documentation [12:13:14] @search bot [12:13:14] Results (found 9): morebots, bot, labs-morebots, keys, bots, cs, help, logs, db, [12:13:17] :) [12:13:24] !help [12:13:25] !documentation for labs !wm-bot for bot [12:13:38] !labs-morebots [12:13:38] adminbot: http://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/adminlogbot/ [12:16:45] Damianz: you know details about latest changes to the way home dirs are created? maybe i missed something [12:17:15] but i would have expected to see a new home dir on an instance after the bot said it updated keys [12:17:27] Not sure, there was talk about moving to gluster, also Ryan mentioned a script issue affecting something homedir related [12:18:09] Does the .ssh dir exists? I believe authorized_keys fails to be written if the dir doesn't exist and it's a known bug [12:18:52] no, it did not create /home/user at all [12:19:12] and i know there are changes to it [12:19:22] but i may not be on the latest updates [12:19:31] Oh, that's rather special. I assume you're looking on nfs? It might have moved to gluster with some automount crazyness, I've not been keeping up tbh.. [12:21:13] until now just looking on instance and trying to change to dir because of automount [12:22:09] ah, now! [12:22:10] weird [12:22:37] robiH: try again?:) [12:22:48] there was just some delay it seems [12:32:09] so now his home exists, his .ssh, his authorized_keys , he is forwarding agent, using the right IP to connect from bastion, and yet he gets permission denied public key and i don't even see him ever attempting to login in auth.log ? what the.. [12:32:30] permissions look right as well... ehmmm... [12:33:25] strategy: look again after fresh coffee .. [12:34:15] Hmm [12:48:26] RECOVERY Disk Space is now: OK on deployment-apache06 i-0000025c output: DISK OK [12:48:26] PROBLEM Disk Space is now: CRITICAL on labs-nfs1 i-0000005d output: CHECK_NRPE: Socket timeout after 10 seconds. [12:52:06] turns out it his something with his agent forwarding [12:52:52] connects to machine, looks for private keys on bastion, cant find them , gives up [12:53:16] PROBLEM Disk Space is now: WARNING on labs-nfs1 i-0000005d output: DISK WARNING - free space: /export 831 MB (4% inode=81%): /home/SAVE 831 MB (4% inode=81%): [12:53:56] I think my gluster just went busted [12:54:39] nevermind, maybe not [12:56:26] PROBLEM Disk Space is now: WARNING on deployment-apache06 i-0000025c output: DISK WARNING - free space: / 74 MB (5% inode=59%): [13:00:12] Damianz: Are you acquainted to putty? [13:00:23] not really, all unixified. [13:01:27] What we found out, that the agent forwarding option fails to convert my ppk into openssh format. [13:01:41] (obviously) [13:04:13] let me add i meant that unrelated: turn on forwarding option alone should do it (but doesnt), but if you wanted to put a key on another host you would have to convert it from ppk to openssh format [13:04:40] sorry need to go for the moment, maybe another putty user (there are some) could help.. i'll be back later [13:05:29] bye mutante. [13:12:46] PROBLEM Disk Space is now: CRITICAL on bz-dev i-000001db output: DISK CRITICAL - free space: / 39 MB (2% inode=43%): [13:20:58] New review: Dzahn; "pending https://bugzilla.wikimedia.org/show_bug.cgi?id=36352" [operations/puppet] (test); V: 0 C: 0; - https://gerrit.wikimedia.org/r/6118 [13:21:59] robiH: try without pagent.exe but instead loading the key in putty itself and turning on forwarding [13:23:06] This is what I always did. [13:23:15] I never used pagent. [13:26:35] then i turn it around:) try with pageant.exe, it turns into a tray icon, load the key there once, minimize to tray, start putty, dont care about key there, BUT turn on the forwarding option [13:31:54] huh? I'm in? [13:32:53] Issue solved! [13:33:19] heh:) [13:42:46] PROBLEM Disk Space is now: WARNING on bz-dev i-000001db output: DISK WARNING - free space: / 39 MB (3% inode=43%): [13:43:15] !log wikistats granting select on wikistats.* to robih @ localhost [13:43:16] Logged the message, Master [14:12:46] PROBLEM Disk Space is now: CRITICAL on bz-dev i-000001db output: DISK CRITICAL - free space: / 39 MB (2% inode=43%): [14:17:58] Re. [16:24:01] helllloouu [16:42:00] RECOVERY Current Load is now: OK on mobile-enwp i-000000ce output: OK - load average: 1.95, 0.55, 0.19 [16:42:00] RECOVERY dpkg-check is now: OK on mobile-enwp i-000000ce output: All packages OK [16:46:00] RECOVERY Free ram is now: OK on mobile-enwp i-000000ce output: OK: 71% free memory [18:10:24] hashar: . [18:10:37] hi! [18:10:39] why we have so many new apache which aren't working [18:10:50] work in progress :-] [18:10:52] why u deleted these 5 I configured XD [18:11:05] I want to have apache configured entirely from pupet [18:11:07] puppet [18:11:07] why they are so small [18:11:13] ok [18:11:14] small? [18:11:19] it looks to me [18:11:19] like there is five of them!!! [18:11:20] :-D [18:11:22] ok [18:11:26] small disk [18:11:29] ohh [18:11:39] probably did a mistake when creating them [18:11:44] anyway, that should n to be an issue [18:11:51] kk [18:11:52] I am not sure we need any disk space for apaches anyway [18:11:57] we will probably fine [18:11:57] we need ram [18:12:00] we will probably be fine [18:12:05] yeah ram, probably [18:12:10] hmm m1.small [18:12:13] 512MB ram [18:12:17] not that great [18:12:17] heh [18:12:22] 1gb was failing [18:12:27] we switched to 8gb [18:12:31] DOH [18:12:32] that is huge [18:12:33] now u created 512 :P [18:12:44] Ryan said we should use 8gb [18:12:56] anyway, the aim is to have them entirely setup from puppet [18:13:12] I think it is just going to be about adding a link for /etc/apache2/wmf [18:13:14] that should work [18:13:27] that is what was on the deployment-apache0{1,2,3,4} [18:13:36] need to document that in puppet [18:17:31] petan: also the morebot bot was broken yesterday [18:17:33] probably still [18:17:51] hashar: yeah, 8GB, like production [18:18:03] !log deployment-prep Made logs from wgCommandLine script to be redirected to /home/wikipedia/logs/cli.log instead of /home/wikipedia/logs/catchall.log [18:18:04] Logged the message, Master [18:18:04] I'm working on puppetizing morebots right now [18:18:23] that would be awesome [18:20:44] 05/09/2012 - 18:20:43 - Creating a project directory for wiki-migration [18:20:44] 05/09/2012 - 18:20:44 - Creating a home directory for laner at /export/home/wiki-migration/laner [18:21:45] 05/09/2012 - 18:21:45 - Updating keys for laner [18:22:45] hashar: it works [18:22:54] !log hashar is not a project [18:22:54] hashar is not a valid project. [18:23:00] yup got fixed somehow :-D [18:26:10] RECOVERY dpkg-check is now: OK on reportcard2 i-000001ea output: All packages OK [18:26:46] wiki-migration? [18:34:26] chrismcmahon: I thought we agreed that apache/mediawiki needs to be configured and run properly on imagescaler01 [18:34:53] chrismcmahon: and hashar is working on that? [18:35:09] Thehelpfulone: I'm helping the openstack people migrate from moinmoin to mediawiki [18:35:12] paravoid: chris is out to get some food [18:35:24] I am looking at having the imagescaler01 to be used :-D [18:35:33] ok :) [18:35:34] ah, okay [18:35:49] because in his mail he said that I'm working on getting thumbnails and I'm not [18:35:54] yet at least [18:35:55] oh really? [18:36:00] ;-] [18:36:10] so I wanted to be sure that there was no miscommunication here [18:36:40] will clarify with chris later on [18:38:15] 05/09/2012 - 18:38:14 - Creating a home directory for luca at /export/home/bastion/luca [18:39:16] 05/09/2012 - 18:39:15 - Updating keys for luca [18:44:26] PROBLEM Current Users is now: CRITICAL on migration1 i-00000261 output: Connection refused by host [18:45:06] PROBLEM Disk Space is now: CRITICAL on migration1 i-00000261 output: Connection refused by host [18:45:46] PROBLEM Free ram is now: CRITICAL on migration1 i-00000261 output: Connection refused by host [18:46:56] PROBLEM Total Processes is now: CRITICAL on migration1 i-00000261 output: Connection refused by host [18:47:36] PROBLEM dpkg-check is now: CRITICAL on migration1 i-00000261 output: Connection refused by host [18:48:46] PROBLEM Current Load is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:56:18] !log deployment-prep allocated three more IPs for upload, bits, and mobile [18:56:19] Logged the message, Master [18:56:33] !log deployment-prep added hostnames for associated IPs [18:56:34] Logged the message, Master [19:00:10] Ryan_Lane: let me know when I can test the upload extension (I presume that's what upload is for), I've got some big files :) [19:01:27] Ryan_Lane: is there anyway to rename an instance from the web interface ? [19:01:33] or should I just delete / recreate it ? [19:01:39] delete/recreate [19:01:39] (it is a new one) [19:01:42] k thx [19:01:44] renames aren't possible [19:06:26] RECOVERY Disk Space is now: OK on deployment-apache06 i-0000025c output: DISK OK [19:06:37] Change on 12mediawiki a page OAuth/User stories was modified, changed by CSteipp link https://www.mediawiki.org/w/index.php?diff=535261 edit summary: /* Huggle 3x */ [19:12:59] Ryan_Lane: https://gerrit.wikimedia.org/r/7073 trivial change to sort instance names in Associate IP :-D [19:13:08] sweet [19:13:26] getInstances() could be made to do that [19:13:31] yeah [19:13:41] 05/09/2012 - 19:13:41 - Creating a project directory for blamemaps [19:13:41] but I got lazy and just fixed the obvious case :-]]]]]]]]]]] [19:13:41] 05/09/2012 - 19:13:41 - Creating a home directory for luca at /export/home/blamemaps/luca [19:13:43] PROBLEM Current Load is now: CRITICAL on deployment-cache-upload i-00000263 output: Connection refused by host [19:13:49] * Ryan_Lane nods [19:14:13] hm. can I just pull on labsconsole, or do I have outstanding changes? :) [19:14:21] * Ryan_Lane wonders [19:14:23] PROBLEM Current Users is now: CRITICAL on deployment-cache-upload i-00000263 output: Connection refused by host [19:14:23] PROBLEM Disk Space is now: WARNING on deployment-apache06 i-0000025c output: DISK WARNING - free space: / 74 MB (5% inode=59%): [19:14:39] git remote update && git log master..origin/master # will tell for sure [19:14:42] lemme do a diff on fetch_head [19:14:43] 05/09/2012 - 19:14:43 - Updating keys for luca [19:15:05] PROBLEM Disk Space is now: CRITICAL on deployment-cache-upload i-00000263 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:15:32] quite a few [19:15:37] basically minor changes [19:15:43] PROBLEM Free ram is now: CRITICAL on deployment-cache-upload i-00000263 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:15:44] all minor changes [19:15:53] let's see if I break labsconsole [19:15:58] lol [19:16:53] PROBLEM Total Processes is now: CRITICAL on deployment-cache-upload i-00000263 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:17:33] PROBLEM dpkg-check is now: CRITICAL on deployment-cache-upload i-00000263 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:18:22] seems to be good :) [19:19:13] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap1 i-000000df output: Puppet has not run in last 20 hours [19:19:23] RECOVERY Disk Space is now: OK on deployment-apache06 i-0000025c output: DISK OK [19:24:32] New patchset: Hashar; "class to cleanup /var/cache/apt/archives" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/7075 [19:24:46] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/7075 [19:27:48] Ryan_Lane: got an issue trying to create an instance named deployment-cache-bits :( [19:27:57] oh? [19:28:00] just got me a "Failed to create instance" message [19:28:00] failed? [19:28:01] really [19:28:03] :_D [19:28:08] I wonder if you guys hit the quota again [19:36:22] paravoid sorry if that was confusing, I meant that *ultimately* what you're doing now will make thumbs dtrt. I probably didn't explain it well. [19:37:04] I don't mind, I was just worried that we miscommunicated and we'd have a deadlock [19:38:09] Ryan_Lane: fixed with https://gerrit.wikimedia.org/r/7078 [19:40:34] RECOVERY Disk Space is now: OK on deployment-apache09 i-0000025e output: DISK OK [19:40:38] hashar: http://php.net/manual/en/array.sorting.php [19:44:24] PROBLEM Current Users is now: CRITICAL on deployment-cache-bits i-00000264 output: Connection refused by host [19:45:31] Ryan_Lane: OSM seems to have duplicate i18n keys in en again [19:45:40] * Ryan_Lane mumbles [19:45:41] 'openstackmanager-securitygrouprule-fromport' => 'From port', [19:45:41] 'openstackmanager-securitygrouprule-toport' => 'To port', [19:45:41] 'openstackmanager-securitygrouprule-protocol' => 'Protocol', [19:45:47] 'openstackmanager-securitygrouprule-groups' => 'Security groups', [19:45:51] I thought I committed those.. [19:45:58]