[00:34:57] <shinken-wm>	 PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[01:50:17] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[01:55:01] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 47221 bytes in 0.708 second response time  
[01:59:22] <grrrit-wm>	 (03CR) 10Awight: "ping!" [integration/config] - 10https://gerrit.wikimedia.org/r/195063 (https://phabricator.wikimedia.org/T91895) (owner: 10Awight)
[02:10:54] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<100.00%)  
[02:12:53] <greg-g>	 0bytes free on /var
[02:12:55] <greg-g>	 fucking...
[02:19:04] * greg-g goes back to being with family around a fire
[02:31:09] <grrrit-wm>	 (03CR) 10Awight: [C: 04-1] "/me facepalms. This doesn't do the trick, cos only one job at a time can safely own the $HOME/.amp directory." [integration/config] - 10https://gerrit.wikimedia.org/r/203187 (https://phabricator.wikimedia.org/T91895) (owner: 10Awight)
[07:50:30] <grrrit-wm>	 (03PS1) 10devunt: Add Josa extension to make-wmf-branch/default.conf [tools/release] - 10https://gerrit.wikimedia.org/r/203642 (https://phabricator.wikimedia.org/T15712) 
[12:33:18] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-bastion is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [43200.0]  
[12:37:35] <shinken-wm>	 PROBLEM - SSH on deployment-lucid-salt is CRITICAL: Connection refused  
[16:49:45] <wikibugs>	 10Beta-Cluster, 10MediaWiki-extensions-ContentTranslation, 10Wikimedia-Site-requests: make amire80 sysop and bureaucrat at http://ru.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T95862#1201867 (10Amire80) 3NEW
[16:49:46] <shinken-wm>	 PROBLEM - App Server bits response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[16:54:34] <shinken-wm>	 RECOVERY - App Server bits response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 3895 bytes in 0.009 second response time  
[16:57:43] <wikibugs>	 10Beta-Cluster: make amire80 sysop and bureaucrat at http://ru.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T95862#1201880 (10Krenair)
[17:00:19] <wikibugs>	 10Beta-Cluster: make amire80 sysop and bureaucrat at http://ru.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T95862#1201887 (10Krenair) 5Open>3Resolved a:3Krenair Done: http://deployment.wikimedia.beta.wmflabs.org/wiki/Special:Log/rights?page=User%3AAmire80%40*
[17:26:04] <Krenair>	 sigh
[17:26:11] <Krenair>	 something is up with upload.beta
[17:26:28] <Krenair>	 2015-04-12 17:06:31 deployment-mediawiki01 deploymentwiki: Thumbnail failed on deployment-mediawiki01: could not get local copy of "27px-Gnome-devel.png"
[17:52:16] <wikibugs>	 10Beta-Cluster: make amire80 sysop and bureaucrat at http://ru.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T95862#1201897 (10Amire80) Thanks!
[19:27:34] <shinken-wm>	 PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[19:29:15] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[19:32:25] <shinken-wm>	 RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 27735 bytes in 0.571 second response time  
[19:34:08] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 46931 bytes in 1.011 second response time  
[23:11:14] <greg-g>	 !log 0bytes left on /var on deployment-bastion
[23:11:23] <qa-morebots>	 Logged the message, Master
[23:48:53] <bd808>	 greg-g: The problem on deployment-bastion is that the l10nupdate role was enabled (or testing I guess) and it puts 861M of stuff in our tiny /var
[23:49:39] <bd808>	 The puppet classes for it create the directory which keeps us from symlinking to /srv where we have ample space.
[23:49:54] <bd808>	 We moved it once but then puppet put it right back
[23:50:16] <bd808>	 My suggestion would be to take l10nupdate off of the box
[23:58:34] <bd808>	 !log sudo ln -s /srv/l10nupdate/mediawiki /var/lib/l10nupdate/mediawiki on deployment-bastion
[23:58:37] <qa-morebots>	 Logged the message, Master