[00:22:33] 10Beta-Cluster, 5Patch-For-Review: 404 handler for bits.beta.wmflabs.org serves unparsed php - https://phabricator.wikimedia.org/T76810#1253896 (10Krinkle) I can still reproduce this. [00:27:45] 10Beta-Cluster: 404 handler for bits.beta.wmflabs.org serves unparsed php - https://phabricator.wikimedia.org/T76810#1253905 (10greg) [01:09:40] 10Beta-Cluster, 6operations, 7Puppet: Trebuchet on deployment-bastion: wrong group owner - https://phabricator.wikimedia.org/T97775#1253962 (10Dzahn) p:5Triage>3High [01:14:19] PROBLEM - Host deployment-boom-boom is DOWN: CRITICAL - Host Unreachable (10.68.16.34) [02:06:09] Project beta-scap-eqiad build #51196: FAILURE in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/51196/ [02:15:18] Yippee, build fixed! [02:15:19] Project beta-scap-eqiad build #51197: FIXED in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/51197/ [02:42:37] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:30] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 47406 bytes in 2.770 second response time [03:33:25] (03PS1) 10Catrope: Run jsduck for Flow [integration/config] - 10https://gerrit.wikimedia.org/r/208339 [03:38:44] (03PS1) 10Catrope: Enable npm for Echo [integration/config] - 10https://gerrit.wikimedia.org/r/208340 [03:52:09] (03PS1) 10Catrope: Enable npm for Thanks [integration/config] - 10https://gerrit.wikimedia.org/r/208342 [03:56:41] !log reset mediawiki-extensions-hhvm workspace on integration-slave-trusty-1015 (bad .git lock) [03:56:47] Logged the message, Master [03:57:21] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #426: FAILURE in 5 min 20 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/426/ [04:11:49] (03PS2) 10Legoktm: Enable npm for Thanks [integration/config] - 10https://gerrit.wikimedia.org/r/208342 (owner: 10Catrope) [04:11:51] (03PS2) 10Legoktm: Enable npm for Echo [integration/config] - 10https://gerrit.wikimedia.org/r/208340 (owner: 10Catrope) [04:12:09] (03CR) 10Legoktm: [C: 032] Enable npm for Echo [integration/config] - 10https://gerrit.wikimedia.org/r/208340 (owner: 10Catrope) [04:12:11] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #431: FAILURE in 5 min 10 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/431/ [04:12:14] (03CR) 10Legoktm: [C: 032] Enable npm for Thanks [integration/config] - 10https://gerrit.wikimedia.org/r/208342 (owner: 10Catrope) [04:13:48] (03Merged) 10jenkins-bot: Enable npm for Echo [integration/config] - 10https://gerrit.wikimedia.org/r/208340 (owner: 10Catrope) [04:13:58] (03Merged) 10jenkins-bot: Enable npm for Thanks [integration/config] - 10https://gerrit.wikimedia.org/r/208342 (owner: 10Catrope) [04:18:12] !log deploying https://gerrit.wikimedia.org/r/208342 and https://gerrit.wikimedia.org/r/208340 [04:18:16] Logged the message, Master [05:06:46] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:14:02] (03PS3) 10Legoktm: Use generic phpunit job for extensions with dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/207132 (https://phabricator.wikimedia.org/T96690) [05:21:20] (03CR) 10Legoktm: [C: 032] Use generic phpunit job for extensions with dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/207132 (https://phabricator.wikimedia.org/T96690) (owner: 10Legoktm) [05:24:09] (03Merged) 10jenkins-bot: Use generic phpunit job for extensions with dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/207132 (https://phabricator.wikimedia.org/T96690) (owner: 10Legoktm) [05:25:56] !log deploying https://gerrit.wikimedia.org/r/207132 [05:26:02] Logged the message, Master [05:42:42] (03PS1) 10Legoktm: Dependencies need to be prefixed by "mediawiki/extensions/" [integration/config] - 10https://gerrit.wikimedia.org/r/208358 [05:44:33] (03CR) 10jenkins-bot: [V: 04-1] Dependencies need to be prefixed by "mediawiki/extensions/" [integration/config] - 10https://gerrit.wikimedia.org/r/208358 (owner: 10Legoktm) [05:45:18] (03PS2) 10Legoktm: Dependencies need to be prefixed by "mediawiki/extensions/" [integration/config] - 10https://gerrit.wikimedia.org/r/208358 [05:45:49] (03CR) 10Legoktm: [C: 032] Dependencies need to be prefixed by "mediawiki/extensions/" [integration/config] - 10https://gerrit.wikimedia.org/r/208358 (owner: 10Legoktm) [05:49:12] (03Merged) 10jenkins-bot: Dependencies need to be prefixed by "mediawiki/extensions/" [integration/config] - 10https://gerrit.wikimedia.org/r/208358 (owner: 10Legoktm) [05:49:27] !log deploying https://gerrit.wikimedia.org/r/208358 [05:49:32] Logged the message, Master [06:16:16] (03PS1) 10Legoktm: ext_dependencies: Add missing Flow deps (VisualEditor & ConfirmEdit) [integration/config] - 10https://gerrit.wikimedia.org/r/208362 [06:16:49] (03CR) 10Legoktm: [C: 032] ext_dependencies: Add missing Flow deps (VisualEditor & ConfirmEdit) [integration/config] - 10https://gerrit.wikimedia.org/r/208362 (owner: 10Legoktm) [06:18:27] (03Merged) 10jenkins-bot: ext_dependencies: Add missing Flow deps (VisualEditor & ConfirmEdit) [integration/config] - 10https://gerrit.wikimedia.org/r/208362 (owner: 10Legoktm) [06:27:42] (03PS1) 10Legoktm: Handle VisualEditor submodule in prepare-mediawiki-zuul-project builder [integration/config] - 10https://gerrit.wikimedia.org/r/208363 [06:28:11] (03CR) 10Legoktm: [C: 032] Handle VisualEditor submodule in prepare-mediawiki-zuul-project builder [integration/config] - 10https://gerrit.wikimedia.org/r/208363 (owner: 10Legoktm) [06:28:54] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #600: FAILURE in 9 min 53 sec: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/600/ [06:30:49] (03Merged) 10jenkins-bot: Handle VisualEditor submodule in prepare-mediawiki-zuul-project builder [integration/config] - 10https://gerrit.wikimedia.org/r/208363 (owner: 10Legoktm) [06:41:52] (03PS1) 10Legoktm: Use generic phpunit job for AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/208364 [06:43:00] (03CR) 10Legoktm: [C: 032] Use generic phpunit job for AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/208364 (owner: 10Legoktm) [06:44:53] (03Merged) 10jenkins-bot: Use generic phpunit job for AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/208364 (owner: 10Legoktm) [06:45:27] !log deploying https://gerrit.wikimedia.org/r/208364 [06:45:37] Logged the message, Master [06:48:50] (03PS1) 10Legoktm: VectorBeta depends upon EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/208366 [06:49:44] (03CR) 10Legoktm: [C: 032] VectorBeta depends upon EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/208366 (owner: 10Legoktm) [06:51:26] (03Merged) 10jenkins-bot: VectorBeta depends upon EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/208366 (owner: 10Legoktm) [06:53:45] !log deploying https://gerrit.wikimedia.org/r/208366 [06:53:49] Logged the message, Master [06:57:33] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Have extensions with dependencies use the shared mwext-testextension-* job - https://phabricator.wikimedia.org/T96690#1254154 (10Legoktm) More patches: * {47541aa6de3e3b4db82ccd777b624fd826aa9a90} * {6956221963f4399ca698120619523c4b2ebec7c3} * {6b466c... [07:13:49] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce build #24: FAILURE in 4 min 47 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce/24/ [08:11:11] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #587: FAILURE in 1 min 10 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/587/ [08:21:12] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #569: FAILURE in 1 min 11 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/569/ [09:15:27] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:18:07] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [09:18:57] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:30:32] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:33:11] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [09:38:57] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:45:48] PROBLEM - Puppet staleness on deployment-restbase02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [11:29:30] 10Continuous-Integration-Infrastructure: PHP >5.3 support in phplint - https://phabricator.wikimedia.org/T97889#1254372 (10Nikerabbit) 3NEW [11:48:57] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce build #236: FAILURE in 56 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce/236/ [12:48:02] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #242: FAILURE in 1 min 1 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/242/ [13:06:41] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #532: FAILURE in 40 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/532/ [13:48:38] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<11.11%) [14:06:48] Project browsertests-Wikidata-PerformanceTests-linux-firefox-sauce build #235: FAILURE in 47 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/235/ [14:24:02] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #239: FAILURE in 7 min 1 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/239/ [14:36:23] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #106: FAILURE in 8 min 22 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/106/ [14:56:33] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [15:18:45] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce build #44: FAILURE in 43 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce/44/ [15:21:31] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [15:23:27] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:53:58] 10Beta-Cluster, 10Graphoid: Deploy Graphoid on Beta Cluster - https://phabricator.wikimedia.org/T97606#1254615 (10Yurik) I have added graphoid-beta.wmflabs.org:80 -> deployment-sca01:19000 [16:29:07] PROBLEM - Puppet failure on deployment-zotero01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:21:32] PROBLEM - Puppet failure on integration-zuul-packaged is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:30:49] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #305: FAILURE in 48 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/305/ [18:53:41] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #306: FAILURE in 40 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/306/ [19:07:04] (03PS1) 10Dduvall: Create ES index before importing mappings [integration/raita] - 10https://gerrit.wikimedia.org/r/208402 [19:09:01] (03CR) 10Dduvall: [C: 032] Create ES index before importing mappings [integration/raita] - 10https://gerrit.wikimedia.org/r/208402 (owner: 10Dduvall) [19:09:09] (03Merged) 10jenkins-bot: Create ES index before importing mappings [integration/raita] - 10https://gerrit.wikimedia.org/r/208402 (owner: 10Dduvall) [19:26:46] (03CR) 10Dduvall: [C: 032] Raita Elasticsearch logging [selenium] - 10https://gerrit.wikimedia.org/r/207324 (owner: 10Dduvall) [19:27:05] (03Merged) 10jenkins-bot: Raita Elasticsearch logging [selenium] - 10https://gerrit.wikimedia.org/r/207324 (owner: 10Dduvall) [19:29:06] (03PS1) 10Dduvall: Log browser test results to Raita [integration/config] - 10https://gerrit.wikimedia.org/r/208403 [21:17:24] PROBLEM - Puppet staleness on deployment-eventlogging02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [22:05:31] Project beta-scap-eqiad build #51316: FAILURE in 1 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/51316/ [22:15:12] Yippee, build fixed! [22:15:13] Project beta-scap-eqiad build #51317: FIXED in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/51317/ [22:59:56] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-firefox-sauce build #39: FAILURE in 55 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-firefox-sauce/39/ [23:56:44] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-chrome-sauce build #39: FAILURE in 43 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-chrome-sauce/39/