[01:35:47] Project beta-scap-eqiad build #43356: FAILURE in 1 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/43356/ [02:15:46] 01:45:21 rsync: write failed on "/srv/mediawiki/wikiversions-labs.cdb": No space left on device (28) [02:18:35] to deployment-jobrunner01.eqiad.wmflabs [02:33:32] (03CR) 10Legoktm: "Could the testing refactor be split out of this commit?" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/153399 (owner: 10Addshore) [02:35:28] (03CR) 10Legoktm: [C: 032] Fixed phpcs sniff verifies wg prefix against all global variables [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/192168 (https://phabricator.wikimedia.org/T90365) (owner: 10Phoenix303) [02:35:46] (03Merged) 10jenkins-bot: Fixed phpcs sniff verifies wg prefix against all global variables [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/192168 (https://phabricator.wikimedia.org/T90365) (owner: 10Phoenix303) [02:37:23] 10Continuous-Integration, 5Patch-For-Review: phpcs sniff should verify all global variables are prefixed with wg - https://phabricator.wikimedia.org/T90365#1076088 (10Legoktm) 5Open>3Resolved a:3Phoenix303 [02:37:58] 1.3G /home/bsimmers [02:39:50] (03PS5) 10Legoktm: Sniff to detect unused global variables [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/192248 (https://phabricator.wikimedia.org/T53279) (owner: 10Phoenix303) [02:42:40] -rw-r--r-- 1 root root 4.5G Mar 1 02:41 /var/log/mediawiki/jobrunner.log [02:43:52] 2015-03-01T02:23:09+0000: Runner loop 0 process in slot 1 gave status '0': [02:43:53] curl -XPOST -s -a 'http://127.0.0.1:9005/rpc/RunJobs.php?wiki=cirrusSearchLinksUpdatePrioritized%2Fenwiki&type=webVideoTranscode%2Fdewiki&maxtime=60&maxmem=300M' [02:44:12] followed by a full HTML body response including "Sorry, we were not able to work out what wiki you were trying to view. Please specify a valid Host header." [02:50:30] (03PS6) 10Legoktm: Sniff to detect unused global variables [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/192248 (https://phabricator.wikimedia.org/T53279) (owner: 10Phoenix303) [02:50:54] that request is broken [02:52:44] (03PS7) 10Legoktm: Sniff to detect unused global variables [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/192248 (https://phabricator.wikimedia.org/T53279) (owner: 10Phoenix303) [02:52:58] (03CR) 10Legoktm: [C: 032] "PS6-7: Fixed a little more code style and added two passing tests." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/192248 (https://phabricator.wikimedia.org/T53279) (owner: 10Phoenix303) [02:53:18] (03Merged) 10jenkins-bot: Sniff to detect unused global variables [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/192248 (https://phabricator.wikimedia.org/T53279) (owner: 10Phoenix303) [02:56:04] 10Continuous-Integration, 10MediaWiki-Codesniffer, 5Patch-For-Review: Install sniff to detect unused globals - https://phabricator.wikimedia.org/T53279#1076102 (10Legoktm) 5Open>3Resolved a:3Phoenix303 [02:59:33] (03PS4) 10Legoktm: Sniff to detect text before first opening php tag [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/192201 (https://phabricator.wikimedia.org/T46875) (owner: 10Sumit) [02:59:51] (03CR) 10jenkins-bot: [V: 04-1] Sniff to detect text before first opening php tag [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/192201 (https://phabricator.wikimedia.org/T46875) (owner: 10Sumit) [03:33:05] ["webVideoTranscode\/enwiki","webVideoTranscode\/dewiki"] [03:33:06] [1424895533,"webVideoTranscode\/enwiki"] [03:33:13] (03PS5) 10Legoktm: Sniff to detect text before first opening php tag [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/192201 (https://phabricator.wikimedia.org/T46875) (owner: 10Sumit) [03:33:34] Krenair: ? [03:33:53] so [ "$type/$wiki", "$type/$wiki" ] or [ "$something", "$type/$wiki" ] [03:34:11] it's supposed to be [ "$type", "$wiki" ] [03:34:48] legoktm, the job runner on deployment-jobrunner01 is spamming logs and caused deployments to break due to the disk being full [03:34:53] Yippee, build fixed! [03:34:53] Project beta-scap-eqiad build #43368: FIXED in 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/43368/ [03:34:58] o.O [03:35:07] (beta deployments) [03:35:38] I don't know why this is occurring in beta only. [03:36:32] RECOVERY - Free space - all mounts on deployment-jobrunner01 is OK: OK: All targets OK [03:37:43] (all OK because I deleted the completely fucked log so I could get it to produce new logs including a debug line I added) [03:41:39] https://github.com/wikimedia/mediawiki-services-jobrunner/blob/master/redisJobRunnerService#L879 [03:43:04] which can apparently be {"1":"cirrusSearchLinksUpdatePrioritized\/enwiki","2":1424895824,"3":"recentChangesUpdate\/enwiki","4":"webVideoTranscode\/dewiki","5":"webVideoTranscode\/commonswiki","6":1424895533,"8":"webVideoTranscode\/enwiki"} - wat [04:55:14] (03CR) 10Legoktm: [C: 032] Sniff to detect text before first opening php tag [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/192201 (https://phabricator.wikimedia.org/T46875) (owner: 10Sumit) [04:55:32] (03Merged) 10jenkins-bot: Sniff to detect text before first opening php tag [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/192201 (https://phabricator.wikimedia.org/T46875) (owner: 10Sumit) [04:56:38] 10Continuous-Integration, 10MediaWiki-Codesniffer, 5Patch-For-Review: Jenkins: Create phpcs sniff to detect text before the first 3Resolved a:3Sumit [05:17:12] (03CR) 10Legoktm: Sniff to detect unused global variables (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/192248 (https://phabricator.wikimedia.org/T53279) (owner: 10Phoenix303) [05:31:22] (03PS1) 10Legoktm: Handle interfaces or abstract functions in UnusedGlobalVariablesSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/193660 [05:33:07] (03CR) 10Phoenix303: Sniff to detect unused global variables (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/192248 (https://phabricator.wikimedia.org/T53279) (owner: 10Phoenix303) [05:33:19] (03CR) 10Legoktm: "Follow up: I3a9859aae5b55b5dec4b0f2c657d2e4883b526de" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/192248 (https://phabricator.wikimedia.org/T53279) (owner: 10Phoenix303) [05:46:38] (03CR) 10Phoenix303: [C: 031] "Yes it works. Thanks Legoktm. Please let me know if there are any additional cases." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/193660 (owner: 10Legoktm) [05:54:36] Project beta-scap-eqiad build #43382: FAILURE in 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/43382/ [06:09:13] (03CR) 10Phoenix303: "I think this is another check that should be added." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/193660 (owner: 10Legoktm) [06:14:45] Yippee, build fixed! [06:14:45] Project beta-scap-eqiad build #43384: FIXED in 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/43384/ [06:23:01] 10Deployment-Systems, 6Release-Engineering, 10Wikimedia-Hackathon-2015, 7HHVM: HHVM RepoAuthoritative Hackathon proof of concept - https://phabricator.wikimedia.org/T91074#1076192 (10greg) [06:46:38] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #1556: FAILURE in 37 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/1556/ [06:48:02] PROBLEM - App Server bits response on deployment-mediawiki01 is CRITICAL: Connection refused [06:50:57] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: Connection refused [07:05:58] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 48343 bytes in 0.539 second response time [07:08:02] RECOVERY - App Server bits response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 3895 bytes in 0.002 second response time [07:11:45] (03PS2) 10Phoenix303: Handle interfaces or abstract functions and heredoc in UnusedGlobalVariablesSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/193660 (owner: 10Legoktm) [07:12:43] (03CR) 10Phoenix303: [C: 031] Handle interfaces or abstract functions and heredoc in UnusedGlobalVariablesSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/193660 (owner: 10Legoktm) [07:56:34] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:07:18] PROBLEM - Puppet staleness on deployment-eventlogging02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [11:35:53] 10Continuous-Integration, 10Math: beta-recompile-math-texvc-eqiad fails - https://phabricator.wikimedia.org/T91191#1076362 (10Physikerwelt) 3NEW [12:14:46] Project beta-scap-eqiad build #43420: FAILURE in 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/43420/ [12:34:56] Yippee, build fixed! [12:34:56] Project beta-scap-eqiad build #43422: FIXED in 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/43422/ [12:37:35] PROBLEM - SSH on deployment-lucid-salt is CRITICAL: Connection refused [12:46:34] Yippee, build fixed! [12:46:34] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #1557: FIXED in 33 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/1557/ [13:52:32] PROBLEM - Free space - all mounts on deployment-jobrunner01 is CRITICAL: CRITICAL: deployment-prep.deployment-jobrunner01.diskspace.root.byte_percentfree.value (<50.00%) [16:55:47] Project beta-scap-eqiad build #43448: FAILURE in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/43448/ [18:06:10] Project browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs build #478: FAILURE in 1 min 8 sec: https://integration.wikimedia.org/ci/job/browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs/478/ [18:40:39] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #344: FAILURE in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/344/ [19:08:53] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #342: FAILURE in 38 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/342/ [19:44:01] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:45:35] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:54:20] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:58:55] 10Continuous-Integration: MediaWiki PHPUnit false negative: "Error creating thumbnail: Unable to save thumbnail to destination" - https://phabricator.wikimedia.org/T91208#1076683 (10Krinkle) 3NEW [20:01:35] 10Continuous-Integration, 6MediaWiki-Core-Team: /tmp/mw-Parser-*/thumb/*.jpg files left behind on Jenkins slaves in production - https://phabricator.wikimedia.org/T91209#1076690 (10Krinkle) 3NEW [20:08:37] 10Continuous-Integration, 6operations: gallium.wikimedia.org disk space running low - https://phabricator.wikimedia.org/T91211#1076705 (10Krinkle) 3NEW [20:34:10] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #497: FAILURE in 25 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/497/ [22:02:53] (03PS3) 10Legoktm: Handle interfaces or abstract functions and heredoc in UnusedGlobalVariablesSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/193660 [22:03:16] (03CR) 10Legoktm: "Thanks! PS3: added a missing newline." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/193660 (owner: 10Legoktm) [22:04:28] (03CR) 10Legoktm: [C: 032] Handle interfaces or abstract functions and heredoc in UnusedGlobalVariablesSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/193660 (owner: 10Legoktm) [22:04:53] (03Merged) 10jenkins-bot: Handle interfaces or abstract functions and heredoc in UnusedGlobalVariablesSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/193660 (owner: 10Legoktm) [22:14:03] (03PS2) 10Legoktm: Use composer for experimental mediawiki-phpunit-{phpflavor}-composer jobs [integration/config] - 10https://gerrit.wikimedia.org/r/193555 (https://phabricator.wikimedia.org/T90303) [22:20:29] (03CR) 10Legoktm: [C: 032] Use composer for experimental mediawiki-phpunit-{phpflavor}-composer jobs [integration/config] - 10https://gerrit.wikimedia.org/r/193555 (https://phabricator.wikimedia.org/T90303) (owner: 10Legoktm) [22:27:03] (03Merged) 10jenkins-bot: Use composer for experimental mediawiki-phpunit-{phpflavor}-composer jobs [integration/config] - 10https://gerrit.wikimedia.org/r/193555 (https://phabricator.wikimedia.org/T90303) (owner: 10Legoktm) [22:54:25] (03CR) 10Legoktm: "Hmm, zuul-cloner is cloning to "src" instead of ., so composer validate can't find it.." [integration/config] - 10https://gerrit.wikimedia.org/r/193555 (https://phabricator.wikimedia.org/T90303) (owner: 10Legoktm) [22:55:01] 10Beta-Cluster, 10MediaWiki-extensions-GettingStarted: Configure Beta Cluster to bypass the CAPTCHA when testing GettingStarted user registration - https://phabricator.wikimedia.org/T90451#1076818 (10Mattflaschen) 5Open>3Resolved a:3Mattflaschen I'm not sure what I was doing wrong before (maybe I put th... [23:00:20] 10Quality-Assurance, 6Release-Engineering, 10MediaWiki-extensions-GettingStarted, 5Patch-For-Review: Pass MEDIAWIKI_CAPTCHA_BYPASS_PASSWORD in on Jenkins so GettingStarted browser tests pass - https://phabricator.wikimedia.org/T91220#1076825 (10Mattflaschen) 3NEW [23:03:54] 10Quality-Assurance, 6Release-Engineering, 10MediaWiki-extensions-GettingStarted, 5Patch-For-Review: Pass MEDIAWIKI_CAPTCHA_BYPASS_PASSWORD in on Jenkins so GettingStarted browser tests pass - https://phabricator.wikimedia.org/T91220#1076835 (10Mattflaschen) I forgot to mention. The password is stored as... [23:06:11] (03PS1) 10Legoktm: Don't use zuul-cloner for mediawiki-phpunit-*-composer jobs [integration/config] - 10https://gerrit.wikimedia.org/r/193757 [23:06:36] (03CR) 10Legoktm: "https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm-composer/2/console" [integration/config] - 10https://gerrit.wikimedia.org/r/193757 (owner: 10Legoktm) [23:18:44] 6Release-Engineering, 5Patch-For-Review: Don't commit interwiki cdbs - https://phabricator.wikimedia.org/T75905#1076848 (10Reedy) p:5Triage>3Normal [23:19:03] 6Release-Engineering, 5Patch-For-Review: Don't commit interwiki cdbs - https://phabricator.wikimedia.org/T75905#784902 (10Reedy) p:5Normal>3Lowest