[04:54:05] <icinga2-wm>	 PROBLEM - check disk on ORES-web02.Experimental is WARNING: DISK WARNING - free space: / 1080 MB (5% inode=92%);
[06:24:08] <icinga2-wm>	 PROBLEM - check disk on ORES-web02.Experimental is WARNING: DISK WARNING - free space: / 1044 MB (5% inode=92%);
[06:26:05] <icinga2-wm>	 RECOVERY - check disk on ORES-web02.Experimental is OK: DISK OK
[13:38:40] <Zppix>	 halfak:  i've noticed a decent amount of alerts coming from icinga2-wm about your guy's experimental hosts, is this expected ?
[13:53:46] <halfak>	 Zppix, not sure.  Just got back from vacation. 
[13:53:50] * halfak checks the logs. 
[13:54:11] <halfak>	 Seems to be online. 
[13:54:22] <Zppix>	 halfak:  it was disk alerts
[13:54:24] <halfak>	 We often get alerts when anything else in labs hiccups. 
[13:54:28] <halfak>	 Gotcha. 
[13:54:36] <Zppix>	 for example PROBLEM - check disk on ORES-web02.Experimental is WARNING: DISK WARNING - free space: / 1044 MB (5% inode=92%);
[13:55:06] <Zppix>	 they recover almost instantly but still
[13:55:16] <halfak>	 Would you do me a favor and put that in a task with #ORES tagged? 
[13:55:31] <halfak>	 something like "Check on disk warnings in wmflabs" 
[13:55:51] <Zppix>	 Sure, want me to subscribe you?
[13:58:52] <wikibugs>	 10ORES, 10Scoring-platform-team, 10VPS-project-icinga2, 10User-Zppix: Look into increased wmflabs alerts from icinga2 - https://phabricator.wikimedia.org/T217832 (10Zppix)
[13:59:25] <Zppix>	 halfak:  ^
[14:00:05] <halfak>	 Thanks Zppix. 
[14:01:32] <wikibugs>	 10ORES, 10Scoring-platform-team, 10VPS-project-icinga2, 10User-Zppix: Look into increased wmflabs alerts from icinga2 - https://phabricator.wikimedia.org/T217832 (10Halfak) From #wikimedia-ai:  ` 2019-03-07 04:54:05 <icinga2-wm> PROBLEM - check disk on ORES-web02.Experimental is WARNING: DISK WARNING - fre...
[14:03:05] <Zppix>	 np
[17:00:56] <halfak>	 o/ hare  
[17:00:59] <halfak>	 Are we doing standup today?
[17:01:24] <hare>	 Yes!
[19:30:23] <hare>	 halfak: do you have anything interesting to talk with me about? if not, i'm probably going to continue focusing on this other work. (I haven't gotten to the ORES docs yet, so no interesting questions for you)
[22:19:18] <wikibugs>	 (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/495112 (owner: 10L10n-bot)