[00:13:50] <wmf-insecte>	 Yippee, build fixed!
[00:13:50] <wmf-insecte>	 Project beta-code-update-eqiad build #53275: FIXED in 49 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53275/
[00:33:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53277: FAILURE in 55 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53277/
[00:53:48] <wmf-insecte>	 Yippee, build fixed!
[00:53:49] <wmf-insecte>	 Project beta-code-update-eqiad build #53279: FIXED in 48 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53279/
[01:13:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53281: FAILURE in 60 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53281/
[01:33:48] <wmf-insecte>	 Yippee, build fixed!
[01:33:49] <wmf-insecte>	 Project beta-code-update-eqiad build #53283: FIXED in 48 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53283/
[01:53:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53285: FAILURE in 67 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53285/
[02:13:51] <wmf-insecte>	 Yippee, build fixed!
[02:13:52] <wmf-insecte>	 Project beta-code-update-eqiad build #53287: FIXED in 51 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53287/
[02:25:17] <wmf-insecte>	 Project browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce build #80: FAILURE in 3 min 16 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce/80/
[02:33:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53289: FAILURE in 0.23 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53289/
[02:53:50] <wmf-insecte>	 Yippee, build fixed!
[02:53:51] <wmf-insecte>	 Project beta-code-update-eqiad build #53291: FIXED in 50 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53291/
[03:08:53] <wmf-insecte>	 Yippee, build fixed!
[03:08:53] <wmf-insecte>	 Project browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce build #81: FIXED in 3 min 7 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce/81/
[03:13:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53293: FAILURE in 78 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53293/
[03:33:49] <wmf-insecte>	 Yippee, build fixed!
[03:33:50] <wmf-insecte>	 Project beta-code-update-eqiad build #53295: FIXED in 49 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53295/
[03:53:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53297: FAILURE in 74 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53297/
[04:13:51] <wmf-insecte>	 Yippee, build fixed!
[04:13:51] <wmf-insecte>	 Project beta-code-update-eqiad build #53299: FIXED in 50 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53299/
[04:33:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53301: FAILURE in 62 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53301/
[04:41:26] <wmf-insecte>	 Yippee, build fixed!
[04:41:27] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #425: FIXED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/425/
[04:53:50] <wmf-insecte>	 Yippee, build fixed!
[04:53:50] <wmf-insecte>	 Project beta-code-update-eqiad build #53303: FIXED in 49 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53303/
[05:06:46] <shinken-wm>	 PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[05:13:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53305: FAILURE in 49 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53305/
[05:33:49] <wmf-insecte>	 Yippee, build fixed!
[05:33:50] <wmf-insecte>	 Project beta-code-update-eqiad build #53307: FIXED in 49 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53307/
[05:53:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53309: FAILURE in 54 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53309/
[05:59:55] <shinken-wm>	 PROBLEM - SSH on integration-saltmaster is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[06:09:37] <thcipriani|afk>	 !log rm scap l10nfiles from /tmp on deployment-bastion root partition 100% again...
[06:09:42] <qa-morebots>	 Logged the message, Master
[06:13:47] <wmf-insecte>	 Yippee, build fixed!
[06:13:48] <wmf-insecte>	 Project beta-code-update-eqiad build #53311: FIXED in 47 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53311/
[06:19:39] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK  
[06:20:28] <wmf-insecte>	 Yippee, build fixed!
[06:20:29] <wmf-insecte>	 Project beta-update-databases-eqiad build #9184: FIXED in 28 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9184/
[06:24:51] <shinken-wm>	 RECOVERY - SSH on integration-saltmaster is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0)  
[06:35:55] <shinken-wm>	 PROBLEM - SSH on integration-saltmaster is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[06:45:18] <shinken-wm>	 PROBLEM - Puppet failure on integration-zuul-server is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[07:10:14] <shinken-wm>	 RECOVERY - Puppet failure on integration-zuul-server is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:16:23] <wikibugs>	 10Deployment-Systems: scap eats underlying commands output (such as maintenance script stacktrace) - https://phabricator.wikimedia.org/T97140#1236479 (10mmodell) as far as I could tell it should be working, verbose flag not necessary... but for some reason it does seem to eat the output - note I closely examined...
[08:35:09] <wikibugs>	 10Deployment-Systems, 6Services: Evaluate Ansible as a deployment tool - https://phabricator.wikimedia.org/T93433#1236512 (10mmodell) @RyanLane: Thanks for the feedback, and your extensive blog post was a really good read.
[12:11:49] <wikibugs>	 10Deployment-Systems, 6operations: [Trebuchet] Salt times out on parsoid restarts - https://phabricator.wikimedia.org/T63882#1236601 (10Aklapper) a:5RyanLane>3None
[13:07:22] <wmf-insecte>	 Yippee, build fixed!
[13:07:22] <wmf-insecte>	 Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #627: FIXED in 35 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/627/
[14:35:39] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace.root.byte_percentfree (<10.00%) WARN: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<100.00%)  
[15:13:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53365: FAILURE in 0.13 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53365/
[15:20:06] <wmf-insecte>	 Project beta-update-databases-eqiad build #9193: FAILURE in 6.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9193/
[15:23:27] <shinken-wm>	 PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[15:33:47] <wmf-insecte>	 Yippee, build fixed!
[15:33:48] <wmf-insecte>	 Project beta-code-update-eqiad build #53367: FIXED in 47 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53367/
[15:38:00] <shinken-wm>	 PROBLEM - Parsoid on deployment-parsoid05 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[15:52:52] <shinken-wm>	 RECOVERY - Parsoid on deployment-parsoid05 is OK: HTTP OK: HTTP/1.1 200 OK - 1086 bytes in 4.453 second response time  
[15:53:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53369: FAILURE in 59 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53369/
[16:13:48] <wmf-insecte>	 Yippee, build fixed!
[16:13:49] <wmf-insecte>	 Project beta-code-update-eqiad build #53371: FIXED in 48 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53371/
[16:29:06] <shinken-wm>	 PROBLEM - Puppet failure on deployment-zotero01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[16:32:25] <wikibugs>	 10Deployment-Systems: scap eats underlying commands output (such as maintenance script stacktrace) - https://phabricator.wikimedia.org/T97140#1236890 (10bd808) >>! In T97140#1236479, @mmodell wrote: > The only part I didn't examine thoroughly is the debug logger - I don't know how that is set up or whether it re...
[16:33:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53373: FAILURE in 50 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53373/
[16:53:48] <wmf-insecte>	 Yippee, build fixed!
[16:53:48] <wmf-insecte>	 Project beta-code-update-eqiad build #53375: FIXED in 47 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53375/
[17:13:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53377: FAILURE in 68 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53377/
[17:33:54] <wmf-insecte>	 Yippee, build fixed!
[17:33:54] <wmf-insecte>	 Project beta-code-update-eqiad build #53379: FIXED in 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53379/
[17:53:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53381: FAILURE in 52 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53381/
[18:13:54] <wmf-insecte>	 Yippee, build fixed!
[18:13:54] <wmf-insecte>	 Project beta-code-update-eqiad build #53383: FIXED in 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53383/
[18:33:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53385: FAILURE in 51 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53385/
[18:53:50] <wmf-insecte>	 Yippee, build fixed!
[18:53:51] <wmf-insecte>	 Project beta-code-update-eqiad build #53387: FIXED in 50 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53387/
[19:09:25] <wikibugs>	 10Beta-Cluster, 6operations: [OPS] exim config points to mchenry.wmflabs.org - https://phabricator.wikimedia.org/T38996#1237019 (10hashar)
[19:13:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53389: FAILURE in 69 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53389/
[19:33:50] <wmf-insecte>	 Yippee, build fixed!
[19:33:50] <wmf-insecte>	 Project beta-code-update-eqiad build #53391: FIXED in 49 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53391/
[19:53:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53393: FAILURE in 71 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53393/
[20:14:00] <wmf-insecte>	 Yippee, build fixed!
[20:14:01] <wmf-insecte>	 Project beta-code-update-eqiad build #53395: FIXED in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53395/
[20:30:29] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-eventlogging02 is CRITICAL: CRITICAL: deployment-prep.deployment-eventlogging02.diskspace._var.byte_percentfree (<100.00%)  
[20:33:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53397: FAILURE in 76 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53397/
[20:54:34] <wmf-insecte>	 Yippee, build fixed!
[20:54:34] <wmf-insecte>	 Project beta-code-update-eqiad build #53399: FIXED in 1 min 33 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53399/
[21:13:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53401: FAILURE in 52 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53401/
[21:15:44] <greg-g>	 ms?
[21:17:25] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-eventlogging02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]  
[21:17:56] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Deployment-Systems: Failed to create a temp file on /mnt/home/jenkins-deploy/workspace/beta-code-update-eqiad - https://phabricator.wikimedia.org/T97257#1237101 (10greg) 3NEW
[21:33:47] <wmf-insecte>	 Yippee, build fixed!
[21:33:48] <wmf-insecte>	 Project beta-code-update-eqiad build #53403: FIXED in 47 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53403/
[21:53:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53405: FAILURE in 52 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53405/
[22:13:48] <wmf-insecte>	 Yippee, build fixed!
[22:13:48] <wmf-insecte>	 Project beta-code-update-eqiad build #53407: FIXED in 47 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53407/
[22:33:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53409: FAILURE in 93 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53409/
[22:53:51] <wmf-insecte>	 Yippee, build fixed!
[22:53:52] <wmf-insecte>	 Project beta-code-update-eqiad build #53411: FIXED in 51 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53411/
[23:13:00] <wmf-insecte>	 Project beta-code-update-eqiad build #53413: FAILURE in 57 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53413/
[23:38:54] <wikibugs>	 10Deployment-Systems: scap eats underlying commands output (such as maintenance script stacktrace) - https://phabricator.wikimedia.org/T97140#1237275 (10mmodell) Ok that explains a lot. For some reason I was thinking about passing --verbose to the subcommand, not to scap it's self. Sorry for being dense. I'll tr...