[00:11:06] ^demon: heh [00:11:13] ^demon: that was in the bug :) [00:11:24] <^demon> Yes. [00:11:29] <^demon> I assumed refs/for/* would work. [00:11:43] * ^demon sighs [00:11:45] <^demon> Oh well :) [00:11:59] heh [00:29:52] New patchset: Sara; "First iteration of adding ganglia for labs." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2157 [00:30:03] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2157 [00:51:21] New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 1; - https://gerrit.wikimedia.org/r/2157 [01:49:38] PROBLEM Free ram is now: WARNING on mobile-enwp mobile-enwp output: Warning: 17% free memory [01:54:38] RECOVERY Free ram is now: OK on mobile-enwp mobile-enwp output: OK: 21% free memory [02:51:45] RECOVERY Current Load is now: OK on bots-dev bots-dev output: OK - load average: 0.08, 0.09, 0.03 [02:52:05] RECOVERY Current Users is now: OK on bots-dev bots-dev output: USERS OK - 0 users currently logged in [02:53:05] RECOVERY Disk Space is now: OK on bots-dev bots-dev output: DISK OK [02:53:35] RECOVERY Free ram is now: OK on bots-dev bots-dev output: OK: 88% free memory [02:54:55] RECOVERY Total Processes is now: OK on bots-dev bots-dev output: PROCS OK: 81 processes [02:55:45] RECOVERY dpkg-check is now: OK on bots-dev bots-dev output: All packages OK [03:31:05] PROBLEM Free ram is now: WARNING on mobile-enwp mobile-enwp output: Warning: 19% free memory [04:02:11] 03/10/2012 - 04:02:11 - Updating keys for fastily [04:16:45] PROBLEM Current Load is now: WARNING on bots-cb bots-cb output: WARNING - load average: 16.15, 13.77, 6.17 [04:21:45] RECOVERY Current Load is now: OK on bots-cb bots-cb output: OK - load average: 0.69, 5.30, 4.57 [04:44:19] la la la labs [04:51:06] RECOVERY Free ram is now: OK on mobile-enwp mobile-enwp output: OK: 25% free memory [06:34:48] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 5.49, 4.74, 2.64 [06:34:59] PROBLEM Total Processes is now: CRITICAL on orgcharts-dev orgcharts-dev output: CHECK_NRPE: Socket timeout after 10 seconds. [06:36:35] PROBLEM Current Load is now: WARNING on bots-sql3 bots-sql3 output: WARNING - load average: 12.80, 10.87, 7.06 [06:51:44] PROBLEM Current Load is now: WARNING on mobile-enwp mobile-enwp output: WARNING - load average: 15.27, 13.13, 8.48 [06:51:44] RECOVERY Total Processes is now: OK on orgcharts-dev orgcharts-dev output: PROCS OK: 84 processes [06:53:55] PROBLEM Current Load is now: WARNING on nova-production1 nova-production1 output: WARNING - load average: 7.47, 9.08, 6.44 [06:54:00] PROBLEM Free ram is now: CRITICAL on mobile-enwp mobile-enwp output: Critical: 3% free memory [06:55:01] PROBLEM Current Load is now: WARNING on orgcharts-dev orgcharts-dev output: WARNING - load average: 5.09, 5.70, 5.14 [06:56:51] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 2.89, 4.38, 4.04 [06:59:58] RECOVERY Current Load is now: OK on orgcharts-dev orgcharts-dev output: OK - load average: 0.20, 2.91, 4.19 [07:01:38] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 0.61, 2.35, 3.31 [07:01:38] PROBLEM Current Load is now: CRITICAL on mobile-enwp mobile-enwp output: CRITICAL - load average: 27.17, 26.53, 17.41 [07:03:58] RECOVERY Current Load is now: OK on nova-production1 nova-production1 output: OK - load average: 0.18, 2.29, 4.27 [07:03:58] RECOVERY Free ram is now: OK on mobile-enwp mobile-enwp output: OK: 35% free memory [07:06:38] PROBLEM Current Load is now: WARNING on mobile-enwp mobile-enwp output: WARNING - load average: 2.08, 12.83, 14.24 [07:31:38] RECOVERY Current Load is now: OK on bots-sql3 bots-sql3 output: OK - load average: 0.67, 2.05, 4.21 [08:41:28] PROBLEM Disk Space is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Socket timeout after 10 seconds. [08:41:28] PROBLEM SSH is now: CRITICAL on deployment-web deployment-web output: CRITICAL - Socket timeout after 10 seconds [08:44:35] PROBLEM Current Users is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Socket timeout after 10 seconds. [08:44:45] PROBLEM Free ram is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Socket timeout after 10 seconds. [08:44:45] PROBLEM Current Load is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Socket timeout after 10 seconds. [08:44:45] PROBLEM Total Processes is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Socket timeout after 10 seconds. [08:44:50] PROBLEM dpkg-check is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Socket timeout after 10 seconds. [08:54:35] PROBLEM Current Load is now: WARNING on bots-sql3 bots-sql3 output: WARNING - load average: 8.07, 7.41, 5.41 [09:04:37] RECOVERY Current Users is now: OK on deployment-web deployment-web output: USERS OK - 0 users currently logged in [09:06:57] PROBLEM Free ram is now: WARNING on mobile-enwp mobile-enwp output: Warning: 18% free memory [09:09:37] RECOVERY Free ram is now: OK on deployment-web deployment-web output: OK: 21% free memory [09:09:37] RECOVERY Total Processes is now: OK on deployment-web deployment-web output: PROCS OK: 146 processes [09:09:42] RECOVERY dpkg-check is now: OK on deployment-web deployment-web output: All packages OK [09:11:57] RECOVERY Free ram is now: OK on mobile-enwp mobile-enwp output: OK: 28% free memory [09:17:37] PROBLEM Free ram is now: WARNING on deployment-web deployment-web output: Warning: 17% free memory [09:29:37] RECOVERY Current Load is now: OK on bots-sql3 bots-sql3 output: OK - load average: 0.65, 2.03, 4.64 [09:34:37] PROBLEM Current Load is now: WARNING on deployment-web deployment-web output: WARNING - load average: 0.08, 1.62, 15.04 [09:47:38] I'm in Gerrit. Why I'm not "Ready for git?"? [09:54:37] RECOVERY Current Load is now: OK on deployment-web deployment-web output: OK - load average: 0.01, 0.07, 4.20 [11:29:37] PROBLEM Disk Space is now: WARNING on aggregator1 aggregator1 output: DISK WARNING - free space: / 542 MB (5% inode=94%): [11:54:38] PROBLEM Disk Space is now: CRITICAL on aggregator1 aggregator1 output: DISK CRITICAL - free space: / 285 MB (2% inode=94%): [12:01:58] PROBLEM Free ram is now: WARNING on mobile-enwp mobile-enwp output: Warning: 18% free memory [12:16:58] RECOVERY Free ram is now: OK on mobile-enwp mobile-enwp output: OK: 21% free memory [12:52:43] grr my bloody instance always fails during a global takedown of instances... [12:56:28] RECOVERY Total Processes is now: OK on prefixexport prefixexport output: PROCS OK: 109 processes [12:56:33] RECOVERY Current Load is now: OK on prefixexport prefixexport output: OK - load average: 0.46, 0.37, 0.15 [12:56:33] RECOVERY dpkg-check is now: OK on prefixexport prefixexport output: All packages OK [12:56:38] RECOVERY Current Users is now: OK on prefixexport prefixexport output: USERS OK - 1 users currently logged in [12:56:47] GREAT [12:57:20] * Damianz eats Hydriz [12:57:29] :( [12:57:37] I am hurt [12:58:28] RECOVERY Disk Space is now: OK on prefixexport prefixexport output: DISK OK [12:58:38] RECOVERY Free ram is now: OK on prefixexport prefixexport output: OK: 88% free memory [12:58:38] RECOVERY SSH is now: OK on prefixexport prefixexport output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [12:58:38] RECOVERY HTTP is now: OK on prefixexport prefixexport output: HTTP OK: HTTP/1.1 200 OK - 470 bytes in 0.029 second response time [12:59:12] I just wanna slap nagios for giving the obvious... :P [13:00:39] How I can get the "Ready for git" status? [13:01:04] IWorld: Explain [13:01:26] sec [13:01:32] http://svn.wikimedia.org/users.php [13:01:50] Oh [13:02:04] you need someone to add you to the git group of LDAP or something [13:02:13] wait no [13:02:22] I think its something about the email address or something [13:02:31] ah [13:02:40] !git [13:02:41] for more information about git on labs see https://labsconsole.wikimedia.org/wiki/Git [13:02:50] !git del [13:02:51] Successfully removed git [13:02:52] I have a labs account and I'm in Gerrit. [13:03:08] !git is For more information about git on labs see https://labsconsole.wikimedia.org/wiki/Help:Git [13:03:08] Key was added! [13:03:17] hmm [13:03:50] I think you got to ask around a bit [13:04:01] but today is Saturday, so try on the next working day [13:04:11] okay [13:04:27] do ask the operations team [13:04:31] or something [13:16:16] its more logical to be in deleted.dblist [13:16:21] * Hydriz checks the Toolserver... [13:16:53] wtf there is a database for bawiktionary [13:17:36] sorry [13:17:40] posted in wrong channel :P [13:44:04] !log incubator Deleted instances (incubator-nfs, -live, -dep and prefixexport) to do more like Wikimedia's configuration for better testing and usage of lesser resources [13:44:05] Logged the message, Master [13:49:41] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [13:49:41] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [13:49:41] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [13:54:09] PROBLEM dpkg-check is now: CRITICAL on incubator-squid incubator-squid output: Connection refused by host [13:54:49] PROBLEM Current Load is now: CRITICAL on incubator-squid incubator-squid output: Connection refused by host [13:55:34] PROBLEM Current Users is now: CRITICAL on incubator-squid incubator-squid output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:56:19] PROBLEM Disk Space is now: CRITICAL on incubator-squid incubator-squid output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:56:59] PROBLEM Free ram is now: CRITICAL on incubator-squid incubator-squid output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:59:09] RECOVERY dpkg-check is now: OK on incubator-squid incubator-squid output: All packages OK [13:59:49] RECOVERY Current Load is now: OK on incubator-squid incubator-squid output: OK - load average: 0.30, 0.80, 0.58 [14:00:29] RECOVERY Current Users is now: OK on incubator-squid incubator-squid output: USERS OK - 1 users currently logged in [14:01:19] RECOVERY Disk Space is now: OK on incubator-squid incubator-squid output: DISK OK [14:01:59] RECOVERY Free ram is now: OK on incubator-squid incubator-squid output: OK: 90% free memory [14:04:49] PROBLEM Current Load is now: CRITICAL on bots-cb bots-cb output: CRITICAL - load average: 69.42, 39.18, 16.37 [14:06:53] lol [14:07:55] 69.42?! [14:08:06] what an average [14:08:48] Quite normal for that bot :P [14:09:09] Might have another go at getting it working under HipHop and see if that helps the random php leakageness. [14:09:57] Oh yes, anyone has the permissions to assign public ips? [14:10:03] ops [14:10:21] yeah, I wonder which kind of ops [14:10:24] We where out of IPs last time I looked, that might have changed though. [14:10:45] * Damianz looks at huge defunt process list [14:11:02] looks changed [14:11:11] hugglewa was assigned one [14:11:16] a few days ago [14:13:20] I wonder how hard it is to make puppet modules behave properly [14:14:39] PROBLEM Current Load is now: WARNING on bots-cb bots-cb output: WARNING - load average: 0.54, 15.06, 17.72 [14:39:39] RECOVERY Current Load is now: OK on bots-cb bots-cb output: OK - load average: 0.50, 0.47, 3.82 [15:03:28] 03/10/2012 - 15:03:28 - Creating a project directory for php [15:03:28] 03/10/2012 - 15:03:28 - Creating a home directory for midom at /export/home/php/midom [15:04:27] 03/10/2012 - 15:04:27 - Updating keys for midom [15:07:33] hi [15:07:57] ih [15:13:54] PROBLEM dpkg-check is now: CRITICAL on php5builds php5builds output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:14:34] PROBLEM Current Load is now: CRITICAL on php5builds php5builds output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:15:19] PROBLEM Current Users is now: CRITICAL on php5builds php5builds output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:16:04] PROBLEM Disk Space is now: CRITICAL on php5builds php5builds output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:16:44] PROBLEM Free ram is now: CRITICAL on php5builds php5builds output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:18:02] 03/10/2012 - 15:18:02 - Creating a home directory for midom at /export/home/testlabs/midom [15:18:14] PROBLEM Total Processes is now: CRITICAL on php5builds php5builds output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:19:11] 03/10/2012 - 15:19:11 - Updating keys for midom [15:32:01] just got permission denied for root... [15:32:10] hah! [15:53:04] PROBLEM Free ram is now: WARNING on mobile-enwp mobile-enwp output: Warning: 18% free memory [15:58:04] RECOVERY Free ram is now: OK on mobile-enwp mobile-enwp output: OK: 36% free memory [20:01:30] If I have a Gerrit account, can I commit to that? [20:02:19] !git [20:02:19] For more information about git on labs see https://labsconsole.wikimedia.org/wiki/Help:Git [20:02:23] ^ Yes [20:02:34] To all repos? :O [20:02:58] Most you can push up to and it opens a review then once it's been reviewed it gets merged. [20:03:07] Gerrit has some fake branch stuff to handle reviewing [20:03:17] In some causes you can directly push up and bypass review though [20:03:30] Like ops can for the production branch of the puppet repo. [20:03:34] are you Wikimedia staff [20:03:36] ? [20:03:40] Nah [20:03:45] ? [20:03:59] [Yes|No] ;D [20:04:01] Not people friendly enough to work for MWF :P [20:04:22] MWF = MediaWiki Foundation :D [20:05:39] Hmm true, I suppose the foundation is technically seperate though I general class it as wikimedia. [20:05:53] ah [20:06:19] Eitherway I'm not an op, staff, wikipedia admin or out. Just a random crazy bot looker-after with a max of like rollback permissions :D [20:06:31] o.O [20:07:12] Damianz: this channel is logged. [20:07:41] Indeed it is [20:07:59] Which reminds me, I wonder if the logging bot has moved yet. If not I might get bored and move her off 2. [20:08:12] ah [20:25:37] Is a Wikimedia stuff here? [20:25:42] *staff [20:26:58] ssmollett might be, #wikimedia-tech is a better place to get ops to do stuff. Ryan is usually around but he appears to not be here (labs is his realm). [20:27:30] There's a few people but they are probably idle or drunk [20:30:52] o.O