[00:48:53] PROBLEM Current Load is now: CRITICAL on basictestsugar i-000004b2.pmtpa.wmflabs output: Connection refused by host [00:53:52] RECOVERY Current Load is now: OK on basictestsugar i-000004b2.pmtpa.wmflabs output: OK - load average: 0.28, 0.54, 0.41 [01:05:52] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 183 processes [01:10:47] 10/09/2012 - 01:10:47 - User Sugar has been renamed, moving home directory in project(s): sugarcrm [01:10:52] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 106 processes [01:15:48] 10/09/2012 - 01:15:48 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [01:16:01] !log sugarcrm set up Instance for basic sugar test install on basictestsugar. Php, apache and db setup [01:16:01] Logged the message, Master [01:20:48] 10/09/2012 - 01:20:48 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [01:25:50] 10/09/2012 - 01:25:50 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [01:30:48] 10/09/2012 - 01:30:47 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [01:34:52] PROBLEM Free ram is now: WARNING on deployment-integration i-0000034a.pmtpa.wmflabs output: 854180 [01:35:47] 10/09/2012 - 01:35:47 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [01:40:49] 10/09/2012 - 01:40:49 - User Sugar has been renamed, moving home directory in project(s): sugarcrm [01:45:46] 10/09/2012 - 01:45:46 - User Sugar has been renamed, moving home directory in project(s): sugarcrm [01:50:47] 10/09/2012 - 01:50:46 - User Sugar has been renamed, moving home directory in project(s): sugarcrm [01:55:43] 10/09/2012 - 01:55:43 - User Sugar has been renamed, moving home directory in project(s): sugarcrm [01:55:52] PROBLEM Total processes is now: UNKNOWN on dumps-bot2 i-000003f4.pmtpa.wmflabs output: NRPE: Call to fork() failed [01:55:52] PROBLEM Free ram is now: UNKNOWN on dumps-bot2 i-000003f4.pmtpa.wmflabs output: NRPE: Call to fork() failed [01:58:12] PROBLEM dpkg-check is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:00:45] 10/09/2012 - 02:00:44 - User Sugar has been renamed, moving home directory in project(s): sugarcrm [02:00:52] PROBLEM Total processes is now: WARNING on dumps-bot2 i-000003f4.pmtpa.wmflabs output: PROCS WARNING: 152 processes [02:00:52] RECOVERY Free ram is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: OK: 52% free memory [02:03:12] RECOVERY dpkg-check is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: All packages OK [02:05:50] 10/09/2012 - 02:05:50 - User sugar has been renamed, moving home directory in project(s): sugarcrm [02:29:52] RECOVERY Free ram is now: OK on deployment-integration i-0000034a.pmtpa.wmflabs output: 539348 [02:40:42] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory [02:53:43] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 7% free memory [03:00:46] 10/09/2012 - 03:00:46 - User diederik has a new uid, changing ownership in project(s): hadoop [03:02:52] PROBLEM SSH is now: CRITICAL on worker1 i-00000208.pmtpa.wmflabs output: Server answer: [03:40:50] 10/09/2012 - 03:40:50 - User louisdang may have been modified in LDAP or locally, updating key in project(s): hadoop [03:41:00] 10/09/2012 - 03:41:00 - Updating keys for louisdang at /export/keys/louisdang [03:53:53] PROBLEM Current Load is now: CRITICAL on sfero i-000004b3.pmtpa.wmflabs output: Connection refused by host [03:57:52] PROBLEM Free ram is now: WARNING on deployment-integration i-0000034a.pmtpa.wmflabs output: 854372 [03:58:52] RECOVERY Current Load is now: OK on sfero i-000004b3.pmtpa.wmflabs output: OK - load average: 0.02, 0.42, 0.39 [04:09:12] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Warning: 14% free memory [04:10:46] hi, i'd like to request an additional public IP be allocated for editor-engagement [04:12:52] RECOVERY Free ram is now: OK on deployment-integration i-0000034a.pmtpa.wmflabs output: 514580 [04:17:52] RECOVERY SSH is now: OK on worker1 i-00000208.pmtpa.wmflabs output: SSH OK - OpenSSH_5.8p1 Debian-7ubuntu1 (protocol 2.0) [04:25:10] louisdang: you're sudo changes are spamming us [04:25:30] oh sorry [04:29:12] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Critical: 5% free memory [04:39:12] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f.pmtpa.wmflabs output: OK: 95% free memory [04:52:23] louisdang: mind fixing it? [04:52:33] louisdang : /etc/sudoers.d/90-cloudimg-ubuntu is owned by uid 2574, should be 0 ; TTY=pts/0 ; PWD=/home/louisdang ; COMMAND=shel [04:53:03] ok [04:56:19] Ryan_Lane: I think I fixed it, used: chown 0 /etc/sudoers.d/90-cloudimg-ubuntu [04:56:32] why is that file even needed? [04:57:03] I just chown -R /etc/ without thinking [04:57:13] you chowned etc? [04:57:17] to who? [04:57:25] myself [04:57:32] why in the world would you do that? [04:57:42] you probably need to delete/recreate that instance now [04:58:08] can't I chown it back? [04:58:20] not everything in etc is owned by root [04:58:41] in general you should almost never chown anything in root [04:58:45] err [04:58:46] in /etc [05:00:17] ok, can you recreate it? [05:00:52] you can [05:33:52] PROBLEM Current Load is now: CRITICAL on worker1 i-000004b4.pmtpa.wmflabs output: Connection refused by host [05:34:32] PROBLEM Current Users is now: CRITICAL on worker1 i-000004b4.pmtpa.wmflabs output: Connection refused by host [05:38:52] RECOVERY Current Load is now: OK on worker1 i-000004b4.pmtpa.wmflabs output: OK - load average: 0.06, 0.44, 0.36 [05:39:32] RECOVERY Current Users is now: OK on worker1 i-000004b4.pmtpa.wmflabs output: USERS OK - 1 users currently logged in [05:52:52] PROBLEM Free ram is now: WARNING on dumps-bot3 i-000003ef.pmtpa.wmflabs output: Warning: 19% free memory [07:05:52] RECOVERY Free ram is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK: 24% free memory [08:37:52] PROBLEM dpkg-check is now: CRITICAL on worker1 i-000004b4.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [08:53:42] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 21% free memory [09:25:42] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 16% free memory [09:35:53] PROBLEM Free ram is now: WARNING on deployment-integration i-0000034a.pmtpa.wmflabs output: 852392 [10:20:53] RECOVERY Free ram is now: OK on deployment-integration i-0000034a.pmtpa.wmflabs output: 532456 [10:32:31] Change on 12mediawiki a page Developer access was modified, changed by Jayantanth link https://www.mediawiki.org/w/index.php?diff=591842 edit summary: [11:08:53] PROBLEM Free ram is now: WARNING on deployment-integration i-0000034a.pmtpa.wmflabs output: 851660 [11:48:52] RECOVERY Free ram is now: OK on deployment-integration i-0000034a.pmtpa.wmflabs output: 531792 [11:58:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 19% free memory [12:26:52] PROBLEM Free ram is now: WARNING on deployment-integration i-0000034a.pmtpa.wmflabs output: 852408 [13:02:10] guillom: all of a sudden the G+ plugin is giving me problems [13:02:16] guillom: mind if I call you via SIP instead? [13:03:02] sumanah: I don't have SIP set up on my desktop computer (OIT is supposed to send me new credentials). But I think I can call the US for free with my new internet line. Let me check quickly. [13:03:08] OK. [13:06:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [13:11:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 97 processes [13:21:53] RECOVERY Free ram is now: OK on deployment-integration i-0000034a.pmtpa.wmflabs output: 542816 [13:39:26] Change on 12mediawiki a page Developer access was modified, changed by Zarg404 link https://www.mediawiki.org/w/index.php?diff=591863 edit summary: [13:40:59] Change on 12mediawiki a page Developer access was modified, changed by Zarg404 link https://www.mediawiki.org/w/index.php?diff=591864 edit summary: [13:41:21] Change on 12mediawiki a page Developer access was modified, changed by Zarg404 link https://www.mediawiki.org/w/index.php?diff=591865 edit summary: [13:42:12] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 19% free memory [13:48:21] Change on 12mediawiki a page Developer access was modified, changed by Zarg404 link https://www.mediawiki.org/w/index.php?diff=591866 edit summary: /* User:Zarg404 */ [13:57:12] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 39% free memory [14:21:12] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 19% free memory [14:40:58] 10/09/2012 - 14:40:58 - Creating a home directory for lwelling at /export/keys/lwelling [14:45:59] 10/09/2012 - 14:45:59 - Updating keys for lwelling at /export/keys/lwelling [15:04:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [15:15:42] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory [15:23:43] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 17% free memory [15:29:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 97 processes [15:44:31] !ping [15:44:31] pong [15:50:27] Oh, one of this very useful functions ;-) [15:51:23] @rss-on [15:51:23] Rss feed has been enabled on channel [15:52:11] @rss+ bugzilla https://bugzilla.wikimedia.org/buglist.cgi?chfieldfrom=-4h&chfieldto=Now&list_id=151044&product=Wikimedia%20Labs&query_format=advanced&title=Bug%20List&ctype=atom [15:52:11] Item was inserted to feed [15:53:15] [bugzilla] [Bug 40895] This is a test bug, ignore me https://bugzilla.wikimedia.org/show_bug.cgi?id=40895 [15:53:57] petan: Nice feature [15:54:43] could be better [15:56:32] andrewbogott: Did you test the passfile option of install.php? I haven't time yet. [15:56:57] Jan_Luca: Not yet. Did it get merged? [15:57:14] Yes [15:57:27] https://gerrit.wikimedia.org/r/#/c/26894/ [15:57:34] ok, I'll rearrange my puppet class today and see how it goes [15:58:09] Thank you [15:58:26] Same to you! [16:08:52] PROBLEM Current Load is now: CRITICAL on mwreview-passfile i-000004b5.pmtpa.wmflabs output: Connection refused by host [16:13:52] RECOVERY Current Load is now: OK on mwreview-passfile i-000004b5.pmtpa.wmflabs output: OK - load average: 1.06, 0.95, 0.62 [16:38:13] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 17% free memory [16:43:12] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 41% free memory [16:56:13] Jan_Luca: Works great. https://gerrit.wikimedia.org/r/#/c/27304/ [17:16:12] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 16% free memory [17:17:06] andrewbogott: That's nice. Thank you! [17:17:29] Jan_Luca: I'm much less ashamed of that line now :) [17:17:42] :-D [18:56:52] !log wikistats - adding several W3C wikis to mediawiki table [18:56:54] Logged the message, Master [19:10:15] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 196 processes [19:12:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 108 processes [20:03:43] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory [20:05:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 198 processes [20:11:42] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 16% free memory [20:15:53] PROBLEM Free ram is now: WARNING on deployment-integration i-0000034a.pmtpa.wmflabs output: 854480 [20:35:52] RECOVERY Free ram is now: OK on deployment-integration i-0000034a.pmtpa.wmflabs output: 511408 [20:39:25] !log deployment-prep Updating all DB following the merge of content handler patch : foreachwiki update.php --quick [20:39:28] Logged the message, Master [20:39:32] oh [20:46:13] !log deployment-prep moving misc::beta::scripts from -integration to -dbdump [20:46:15] Logged the message, Master [21:13:12] PROBLEM dpkg-check is now: CRITICAL on master i-0000007a.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [21:20:12] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 19% free memory [21:35:12] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 42% free memory [21:38:12] RECOVERY dpkg-check is now: OK on master i-0000007a.pmtpa.wmflabs output: All packages OK [21:58:38] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=592029 edit summary: /* User:Jayantanth */ [21:59:15] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=592030 edit summary: /* User:Zarg404 */ [21:59:56] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=592032 edit summary: /* Requests */ archiving requests that are done [22:01:06] !log deployment-prep applying misc::beta::scripts on a precise instance: -jobrunner06 [22:01:08] Logged the message, Master [22:02:53] RECOVERY dpkg-check is now: OK on worker1 i-000004b4.pmtpa.wmflabs output: All packages OK [22:04:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [22:04:56] oh man..i gotta kill these procs [22:09:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 98 processes [22:33:47] [bugzilla] [Bug 36885] wmflabs.org does not resolve https://bugzilla.wikimedia.org/show_bug.cgi?id=36885 [23:11:22] PROBLEM Disk Space is now: WARNING on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DISK WARNING - free space: /export 615 MB (3% inode=59%): /home/SAVE 615 MB (3% inode=59%): [23:18:12] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 18% free memory [23:28:12] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 21% free memory