[03:42:31] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 15% free memory [03:45:31] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 14% free memory [03:45:41] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 16% free memory [03:48:31] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 16% free memory [04:00:31] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 5% free memory [04:02:31] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:05:41] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 3% free memory [04:08:31] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 4% free memory [04:10:31] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 95% free memory [04:10:41] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:12:31] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 93% free memory [04:13:31] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 96% free memory [04:22:31] PROBLEM Free ram is now: WARNING on test3 i-00000093 output: Warning: 8% free memory [04:27:31] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [05:48:05] Change on 12mediawiki a page Wikimedia Labs was modified, changed by 174.141.208.98 link https://www.mediawiki.org/w/index.php?diff=530434 edit summary: /* Test/Dev Labs */ [06:12:27] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Peachey88 link https://www.mediawiki.org/w/index.php?diff=530436 edit summary: Reverted edits by [[Special:Contributions/174.141.208.98|174.141.208.98]] ([[User talk:174.141.208.98|talk]]) to last revision by [[User:Eloquence|Eloquence]] [06:54:41] PROBLEM Current Load is now: WARNING on bots-sql3 i-000000b4 output: WARNING - load average: 5.99, 6.38, 5.41 [07:04:35] RECOVERY Current Load is now: OK on bots-sql3 i-000000b4 output: OK - load average: 1.95, 3.15, 4.37 [10:17:44] PROBLEM Disk Space is now: CRITICAL on labs-nfs1 i-0000005d output: DISK CRITICAL - free space: /export 514 MB (2% inode=81%): /home/SAVE 514 MB (2% inode=81%): [11:19:33] * Beetstra annoys the servers .. [11:20:46] hi Beetstra [11:20:49] did you get my email? [11:21:15] !log bots synchronizing databases 'coibot' and 'linkwatcher' from bots-sql3 to bots-sql2, adapting the bot-code to store/query on bots-sql2, will restart bots after transfer is complete [11:21:17] Logged the message, Master [11:21:33] mail? [11:21:37] * Beetstra looks around [11:21:57] Sent .. eh .. where? [11:23:11] Ohoh .. phpmyadmin is not listening to me anymore .. [11:27:24] Thehelpfulone, sorry, but I think I did not get your email :-) [11:33:28] Beetstra: no problem, see PM :) [18:18:35] All quiet in the world of Labs :P [18:22:35] * Damianz shoots methecooldude for making noise [19:33:13] PROBLEM Disk Space is now: CRITICAL on bz-dev i-000001db output: DISK CRITICAL - free space: / 31 MB (2% inode=43%): [19:38:13] PROBLEM Disk Space is now: WARNING on bz-dev i-000001db output: DISK WARNING - free space: / 52 MB (3% inode=43%): [22:08:13] PROBLEM Puppet freshness is now: CRITICAL on otrs-jgreen i-0000015a output: Puppet has not run in last 20 hours [22:08:29] Hi everyone... searching in Google "Wikimedia Labs" the first result is http://en.labs.wikimedia.org/wiki/Main_Page which says "closed, moved to http://labs.wikimedia.beta.wmflabs.org/wiki/ " which in turn is "404 Not Found" [22:08:51] So maybe someone can and wants to fix it :) [22:15:13] ah that's a cache error hmm [22:24:56] Ok, seems fixed now :) [22:25:28] See you soon (probably in other channels)... [22:39:38] It's probably a squid issue [22:42:06] nah Damianz it was that the main page was updated to the new link [22:42:20] but it had to be approved via pending changes it turns out