[00:35:05] Yippee, build fixed! [00:35:05] Project beta-scap-eqiad build #50394: FIXED in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/50394/ [00:56:24] 10Deployment-Systems, 10MediaWiki-extensions-LocalisationUpdate, 7Epic, 7I18n: Localization Cache Redo - https://phabricator.wikimedia.org/T78802#1235661 (10greg) p:5Triage>3Normal [01:04:53] So… beta has been reliable today? I’m still seeing weird shinken alerts for some labs instances. [01:09:50] 6Release-Engineering, 10Staging, 3releng-201415-Q3, 3releng-201415-Q4: [Quarterly Success Metric] Green nightly builds on the staging cluster (tracking) - https://phabricator.wikimedia.org/T88701#1235673 (10greg) 5Open>3stalled [01:10:04] 6Release-Engineering, 10Staging, 3releng-201415-Q3, 3releng-201415-Q4: [Quarterly Success Metric] Green nightly builds on the staging cluster (tracking) - https://phabricator.wikimedia.org/T88701#1018466 (10greg) p:5Normal>3Low [01:14:18] 10Staging, 3releng-201415-Q3: [Quarterly Success Metric] Stable uptime metrics of the Staging cluster - https://phabricator.wikimedia.org/T88705#1235676 (10mmodell) [01:18:56] Yippee, build fixed! [01:18:56] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #233: FIXED in 55 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/233/ [01:24:52] 10Staging, 3releng-201415-Q3: [Quarterly Success Metric] Stable uptime metrics of the Staging cluster - https://phabricator.wikimedia.org/T88705#1235692 (10mmodell) Ok I updated the dashboard, improved the graphs just a little [01:55:27] Yippee, build fixed! [01:55:27] Project browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-os_x_10.10-iphone-sauce build #50: FIXED in 1 min 26 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-os_x_10.10-iphone-sauce/50/ [02:07:13] PROBLEM - Puppet staleness on deployment-elastic07 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [43200.0] [02:25:14] Yippee, build fixed! [02:25:15] Project browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce build #79: FIXED in 3 min 14 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce/79/ [02:35:36] Yippee, build fixed! [02:35:36] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #552: FIXED in 2 min 35 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/552/ [02:48:04] Yippee, build fixed! [02:48:05] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce build #33: FIXED in 55 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce/33/ [02:59:32] (03PS1) 10Krinkle: check-vars: Remove $wgRUstart (removed from MediaWiki) [tools/code-utils] - 10https://gerrit.wikimedia.org/r/206503 [02:59:53] (03CR) 10Krinkle: [C: 032] check-vars: Remove $wgRUstart (removed from MediaWiki) [tools/code-utils] - 10https://gerrit.wikimedia.org/r/206503 (owner: 10Krinkle) [02:59:55] (03Merged) 10jenkins-bot: check-vars: Remove $wgRUstart (removed from MediaWiki) [tools/code-utils] - 10https://gerrit.wikimedia.org/r/206503 (owner: 10Krinkle) [03:01:22] Yippee, build fixed! [03:01:22] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #233: FIXED in 52 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/233/ [03:29:18] PROBLEM - Puppet staleness on deployment-redis01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [43200.0] [04:25:53] Project beta-scap-eqiad build #50417: FAILURE in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/50417/ [04:30:14] Yippee, build fixed! [04:30:14] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #419: FIXED in 38 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/419/ [04:36:22] 10Browser-Tests, 6Collaboration-Team, 10Collaboration-Team-Sprint-A-2015-05-06, 10Flow, 5Patch-For-Review: A5. Fix failed Flow browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94153#1235859 (10Mattflaschen) They are now failing with: "timed out after 5 seconds, waiting for {:name=>"topicli... [04:37:10] 10Browser-Tests, 6Collaboration-Team, 10Collaboration-Team-Sprint-A-2015-05-06, 10Flow, 5Patch-For-Review: A5. Fix failed Flow browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94153#1235860 (10Mattflaschen) This is after Beta was fixed: https://integration.wikimedia.org/ci/job/browsertests-... [05:42:28] Yippee, build fixed! [05:42:29] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce build #50: FIXED in 26 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce/50/ [05:58:40] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<50.00%) [06:30:50] Yippee, build fixed! [06:30:51] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #593: FIXED in 11 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/593/ [06:53:40] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<20.00%) WARN: deployment-prep.deployment-bastion.diskspace.root.byte_percentfree (<100.00%) [07:13:00] Project beta-code-update-eqiad build #53173: FAILURE in 0.16 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53173/ [07:20:23] Project beta-update-databases-eqiad build #9161: FAILURE in 22 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9161/ [07:33:56] Yippee, build fixed! [07:33:56] Project beta-code-update-eqiad build #53175: FIXED in 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53175/ [07:42:44] Yippee, build fixed! [07:42:44] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce build #17: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce/17/ [07:53:00] Project beta-code-update-eqiad build #53177: FAILURE in 71 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53177/ [08:14:00] Yippee, build fixed! [08:14:00] Project beta-code-update-eqiad build #53179: FIXED in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53179/ [08:33:00] Project beta-code-update-eqiad build #53181: FAILURE in 57 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53181/ [08:53:57] Yippee, build fixed! [08:53:58] Project beta-code-update-eqiad build #53183: FIXED in 57 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53183/ [09:01:14] Yippee, build fixed! [09:01:14] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #580: FIXED in 51 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/580/ [09:13:00] Project beta-code-update-eqiad build #53185: FAILURE in 64 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53185/ [09:33:58] Yippee, build fixed! [09:33:59] Project beta-code-update-eqiad build #53187: FIXED in 58 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53187/ [09:53:00] Project beta-code-update-eqiad build #53189: FAILURE in 74 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53189/ [10:14:01] Yippee, build fixed! [10:14:01] Project beta-code-update-eqiad build #53191: FIXED in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53191/ [10:33:00] Project beta-code-update-eqiad build #53193: FAILURE in 60 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53193/ [10:53:56] Yippee, build fixed! [10:53:57] Project beta-code-update-eqiad build #53195: FIXED in 56 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53195/ [11:13:00] Project beta-code-update-eqiad build #53197: FAILURE in 64 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53197/ [11:34:00] Yippee, build fixed! [11:34:01] Project beta-code-update-eqiad build #53199: FIXED in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53199/ [11:53:00] Project beta-code-update-eqiad build #53201: FAILURE in 81 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53201/ [11:59:18] Yippee, build fixed! [11:59:18] Project browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #89: FIXED in 2 min 17 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/89/ [12:13:59] Yippee, build fixed! [12:13:59] Project beta-code-update-eqiad build #53203: FIXED in 58 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53203/ [12:33:00] Project beta-code-update-eqiad build #53205: FAILURE in 63 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53205/ [12:53:48] Yippee, build fixed! [12:53:48] Project beta-code-update-eqiad build #53207: FIXED in 47 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53207/ [13:06:46] Yippee, build fixed! [13:06:46] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #525: FIXED in 45 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/525/ [13:13:00] Project beta-code-update-eqiad build #53209: FAILURE in 51 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53209/ [13:33:48] Yippee, build fixed! [13:33:48] Project beta-code-update-eqiad build #53211: FIXED in 47 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53211/ [13:53:00] Project beta-code-update-eqiad build #53213: FAILURE in 61 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53213/ [14:14:02] Yippee, build fixed! [14:14:02] Project beta-code-update-eqiad build #53215: FIXED in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53215/ [14:33:00] Project beta-code-update-eqiad build #53217: FAILURE in 63 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53217/ [14:36:42] Yippee, build fixed! [14:36:42] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #99: FIXED in 8 min 41 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/99/ [14:54:00] Yippee, build fixed! [14:54:00] Project beta-code-update-eqiad build #53219: FIXED in 59 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53219/ [15:13:00] Project beta-code-update-eqiad build #53221: FAILURE in 69 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53221/ [15:15:59] !log clear /tmp/scap files on deployment-bastion, root partition at 100% [15:16:04] Logged the message, Master [15:20:15] Yippee, build fixed! [15:20:16] Project beta-update-databases-eqiad build #9169: FIXED in 15 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9169/ [15:23:48] Yippee, build fixed! [15:23:48] Project beta-code-update-eqiad build #53222: FIXED in 47 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53222/ [15:28:39] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [15:28:55] Yippee, build fixed! [15:28:56] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #514: FIXED in 55 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/514/ [16:00:35] !log manually ran logrotate on deployment-jobrunner01, root partition at 100% [16:00:37] Logged the message, Master [16:04:57] Yippee, build fixed! [16:04:57] Project beta-scap-eqiad build #50463: FIXED in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/50463/ [16:56:01] PROBLEM - Parsoid on deployment-parsoid05 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:00:50] RECOVERY - Parsoid on deployment-parsoid05 is OK: HTTP OK: HTTP/1.1 200 OK - 1086 bytes in 0.804 second response time [17:01:36] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:32] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 47285 bytes in 6.284 second response time [18:34:32] Project beta-scap-eqiad build #50478: FAILURE in 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/50478/ [18:49:38] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace.root.byte_percentfree (<50.00%) [19:53:00] Project beta-code-update-eqiad build #53249: FAILURE in 52 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53249/ [20:13:49] Yippee, build fixed! [20:13:50] Project beta-code-update-eqiad build #53251: FIXED in 49 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53251/ [20:20:02] Project beta-update-databases-eqiad build #9174: FAILURE in 2.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9174/ [20:33:00] Project beta-code-update-eqiad build #53253: FAILURE in 59 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53253/ [20:53:55] Yippee, build fixed! [20:53:55] Project beta-code-update-eqiad build #53255: FIXED in 54 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53255/ [21:13:00] Project beta-code-update-eqiad build #53257: FAILURE in 91 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53257/ [21:26:27] PROBLEM - Puppet failure on integration-saltmaster is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:34:40] Yippee, build fixed! [21:34:41] Project beta-code-update-eqiad build #53259: FIXED in 1 min 40 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53259/ [21:46:26] RECOVERY - Puppet failure on integration-saltmaster is OK: OK: Less than 1.00% above the threshold [0.0] [21:53:00] Project beta-code-update-eqiad build #53261: FAILURE in 53 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53261/ [22:13:48] Yippee, build fixed! [22:13:49] Project beta-code-update-eqiad build #53263: FIXED in 48 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53263/ [22:33:00] Project beta-code-update-eqiad build #53265: FAILURE in 72 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53265/ [22:53:50] Yippee, build fixed! [22:53:50] Project beta-code-update-eqiad build #53267: FIXED in 49 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53267/ [23:13:00] Project beta-code-update-eqiad build #53269: FAILURE in 57 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53269/ [23:31:19] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:31:24] PROBLEM - Puppet staleness on deployment-urldownloader is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [23:31:44] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:33:49] Yippee, build fixed! [23:33:50] Project beta-code-update-eqiad build #53271: FIXED in 49 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53271/ [23:53:00] Project beta-code-update-eqiad build #53273: FAILURE in 52 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/53273/