[00:06:59] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #259: FAILURE in 22 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/259/ [00:26:35] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #380: ABORTED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/380/ [00:27:42] (03PS1) 10Krinkle: Unlink "nightly" directory (not updated since mid-2013) [integration/docroot] - 10https://gerrit.wikimedia.org/r/170868 [00:27:52] (03CR) 10Krinkle: [C: 032] Unlink "nightly" directory (not updated since mid-2013) [integration/docroot] - 10https://gerrit.wikimedia.org/r/170868 (owner: 10Krinkle) [00:27:54] (03Merged) 10jenkins-bot: Unlink "nightly" directory (not updated since mid-2013) [integration/docroot] - 10https://gerrit.wikimedia.org/r/170868 (owner: 10Krinkle) [00:30:11] Krinkle: https://integration.wikimedia.org/ doesn't show that change yet, but the message on the change claims it should now be live? [00:30:43] James_F: Yeah, seems to take a while [00:30:45] works now though [00:30:58] Aha, yeah. [01:25:25] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #382: STILL FAILING in 53 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/382/ [08:02:29] Yippee, build fixed! [08:02:29] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #264: FIXED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/264/ [09:14:36] Yippee, build fixed! [09:14:37] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #109: FIXED in 42 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/109/ [10:34:48] (03Abandoned) 10Zfilipin: Fixed RuboCop offense Style/AndOr [selenium] - 10https://gerrit.wikimedia.org/r/168320 (https://bugzilla.wikimedia.org/63307) (owner: 10Zfilipin) [10:35:05] (03Abandoned) 10Zfilipin: Fixed RuboCop offense Lint/UselessAccessModifier [selenium] - 10https://gerrit.wikimedia.org/r/168318 (https://bugzilla.wikimedia.org/63307) (owner: 10Zfilipin) [10:35:11] (03Abandoned) 10Zfilipin: Fixed RuboCop offense Lint/UnusedMethodArgument [selenium] - 10https://gerrit.wikimedia.org/r/168316 (https://bugzilla.wikimedia.org/63307) (owner: 10Zfilipin) [10:35:14] (03Abandoned) 10Zfilipin: Fixed RuboCop offense Lint/UnusedBlockArgument [selenium] - 10https://gerrit.wikimedia.org/r/168314 (https://bugzilla.wikimedia.org/63307) (owner: 10Zfilipin) [10:35:47] Project beta-scap-eqiad build #28395: FAILURE in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/28395/ [10:42:18] (03Abandoned) 10Zfilipin: WIP better link to Sauce Labs job [selenium] - 10https://gerrit.wikimedia.org/r/137673 (owner: 10Zfilipin) [10:45:48] Yippee, build fixed! [10:45:48] Project beta-scap-eqiad build #28396: FIXED in 1 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/28396/ [10:55:37] (03CR) 10Zfilipin: [C: 031] "The idea sounds good to me. I am not familiar with zuul enough to give a proper code review." [integration/config] - 10https://gerrit.wikimedia.org/r/170320 (owner: 10Hashar) [10:57:25] zeljkof: merged :) [10:58:08] YuviPand: what is merged? sorry, reviewed a lot of commits today :) [10:59:38] zeljkof: https://gerrit.wikimedia.org/r/#/c/166046/ [10:59:55] wait, did I confuse you with someone else? [11:00:20] YuviPand: thanks! :) [11:00:44] no, I am the one that made the comment, but also another 10-20 in the last hour or two [11:01:02] so it was not easy for me to see what you were referring to :) [11:01:41] (03CR) 10Zfilipin: [C: 031] "The idea sounds good to me. I am not familiar with zuul enough to give a code review." [integration/config] - 10https://gerrit.wikimedia.org/r/170323 (owner: 10Hashar) [11:01:50] ah :) [11:01:51] ok [11:02:21] YuviPand: cleaning my gerrit dashboard, commenting, voting, abandoning commits left and right :) [11:02:27] :) [11:03:38] cleaning my tools, I have noticed some dust and rust [11:14:05] (03CR) 10Hashar: [C: 031] "The failing job is verifying that a all jobs triggerable by Zuul are actually deployed in Jenkins. Since the jobs do not exist yet (they " [integration/config] - 10https://gerrit.wikimedia.org/r/170447 (owner: 10Lewis Cawte) [11:17:55] 3Wikimedia / 3Continuous integration: mwext-browsertests-UniversalLanguageSelector-phantomjs job failing due to missing Psr\Log dependency - 10https://bugzilla.wikimedia.org/72925#c1 (10Antoine "hashar" Musso (WMF)) The mwext-browsertests-* jobs were meant to be triggered on patchset proposals. Apparently t... [11:25:30] Yippee, build fixed! [11:25:30] Project browsertests-VisualEditor-production-linux-firefox-sauce build #27: FIXED in 1 hr 34 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-production-linux-firefox-sauce/27/ [11:59:14] hi lubaochuan [11:59:25] hi [11:59:50] I am ready. [12:00:13] me too :) [12:00:19] I am joining the hangout now [12:00:30] Great. [12:00:46] did you already join it? [14:04:42] (03PS2) 10Hashar: Only test Jenkins jobs existence on gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/170320 [14:06:22] (03PS3) 10Hashar: Only test Jenkins jobs existence on gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/170320 [14:07:30] (03CR) 10Hashar: [C: 032] "The new job integration-zuul-layoutvalidation-gate was not added to the jjb project section." [integration/config] - 10https://gerrit.wikimedia.org/r/170320 (owner: 10Hashar) [14:09:50] (03CR) 10jenkins-bot: [V: 04-1] Only test Jenkins jobs existence on gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/170320 (owner: 10Hashar) [14:10:56] (03Merged) 10jenkins-bot: Only test Jenkins jobs existence on gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/170320 (owner: 10Hashar) [14:12:15] (03CR) 10Hashar: "Zuul deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/170320 (owner: 10Hashar) [14:12:31] (03PS3) 10Hashar: Tests for Bouquet skin. [integration/config] - 10https://gerrit.wikimedia.org/r/170447 (owner: 10Lewis Cawte) [14:22:56] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c12 (10Antoine "hashar" Musso (WMF)) Created attachment 17012 --> https://bugzilla.wikimedia.org/attachment.cgi?id=17012&action=edit Raw thread... [14:24:41] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c13 (10Antoine "hashar" Musso (WMF)) From Jenkins log https://integration.wikimedia.org/ci/log/Plugins%20-%20Gearman/ ( hudson.plugins.gearman.lo... [14:31:12] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c14 (10Antoine "hashar" Musso (WMF)) The rest of the executors are hold in lock with: "Gearman worker deployment-bastion.eqiad_exec-2" daemon... [14:47:59] hi zeljkof do you know what might be wrong with the Core builds? https://integration.wikimedia.org/ci/view/BrowserTests/view/-All/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/ [14:48:10] chrismcmahon: looking [14:48:24] zeljkof: the PDFHandler and MMV builds got fixed https://bugzilla.wikimedia.org/show_bug.cgi?id=72435 [14:48:32] 00:00:12.204 Could not locate Gemfile [14:48:35] but the Core build is still not right [14:48:49] yes, the commit that moved gemfile to root is not merged [14:48:50] this one [14:49:02] https://gerrit.wikimedia.org/r/#/c/167794/ [14:49:15] will work on the commit right now to take it mergable [14:49:24] while hashar is still online :) [14:50:19] zeljkof: oops, I just merged it [14:50:28] chrismcmahon: then thanks :) [14:50:28] should I undo that? [14:50:35] ask hashar :) [14:50:41] I thought the commit was fine [14:50:58] he wanted to make sure some folders were excluded from rubocop [14:51:04] but we can do that later [14:57:41] 3Wikimedia / 3Quality Assurance: pdfhandler and MMV builds are misconfigured - 10https://bugzilla.wikimedia.org/72435 (10Chris McMahon) 5NEW>3RESO/FIX [15:00:11] chrismcmahon: should I rerun the core job? [15:00:21] to see if the problem is fixed? [15:00:36] (reruning...) [15:03:28] job is looking good so far [15:03:28] https://integration.wikimedia.org/ci/view/BrowserTests/view/-All/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/272/console [15:11:47] zeljkof: after more than a year, the edit controls in VisualEditor no longer appear in iframes. I have been doing a ton of refactoring in there, with more to go. Page Objects are great! [15:11:58] chrismcmahon: :) [15:12:25] so many kluges to get rid of... [15:13:21] (03CR) 10Zfilipin: "Should be commit be abandoned? Or, is there something that we can do to?" [integration/config] - 10https://gerrit.wikimedia.org/r/166553 (https://bugzilla.wikimedia.org/72326) (owner: 10Zfilipin) [15:14:43] Yippee, build fixed! [15:14:43] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #272: FIXED in 14 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/272/ [15:15:29] (03CR) 10Zfilipin: "Is there anything preventing this commit from being merged into master branch?" [integration/config] - 10https://gerrit.wikimedia.org/r/168270 (https://bugzilla.wikimedia.org/69245) (owner: 10Zfilipin) [15:20:43] (03PS2) 10Zfilipin: WIP experiment to see if this will fix VisualEditor language screenshot job [integration/config] - 10https://gerrit.wikimedia.org/r/170023 [15:21:40] (03Abandoned) 10Zfilipin: WIP experiment to see if this will fix VisualEditor language screenshot job [integration/config] - 10https://gerrit.wikimedia.org/r/170023 (owner: 10Zfilipin) [15:59:19] zeljkof: I can probably fix the language screenshot job. VE just had a major architecture change, no more iframes [15:59:26] Yippee, build fixed! [15:59:26] Project browsertests-Wikidata-WikidataTests-linux-firefox-sauce build #29: FIXED in 1 hr 37 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-firefox-sauce/29/ [16:06:43] 3Wikimedia / 3Continuous integration: Jenkins: Create job for verifying comitted composer vendor directory - 10https://bugzilla.wikimedia.org/72952 (10Krinkle) 3NEW p:3Unprio s:3normal a:3None Let's add a job that verifies the commited vendor/ directory (in mediawiki/vendor.git) matches what composer... [16:17:05] chrismcmahon: great [16:17:27] Yippee, build fixed! [16:17:28] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #283: FIXED in 8 min 5 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/283/ [16:17:41] zeljkof: I wanted to make the regular build green first [16:18:05] great: Exception Caught: Unable to connect to redis server. (internal_api_error_JobQueueConnectionError) (MediawikiApi::ApiError) [16:18:14] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #384: ABORTED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/384/ [16:18:58] zeljkof: have you seen this error before? [16:19:03] $ bundle install [16:19:03] Fetching source index from https://rubygems.org/ [16:19:03] Network error while fetching https://rubygems.org/quick/Marshal.4.8/ast-2.0.0.gemspec.rz [16:20:14] chrismcmahon: try this [16:20:15] bundle install -V [16:20:18] that is verbose [16:20:32] but looks like you can not get to rubygems [16:20:37] can you open it in a browser? [16:20:48] zeljkof: that URL resolves and downloads for me, it's strange [16:21:22] not much help: [16:21:25] $ bundle install -V [16:21:25] Fetching from: https://bundler.rubygems.org/api/v1/dependencies [16:21:25] Fetching source index from https://rubygems.org/ [16:21:27] Fetching from: https://rubygems.org/quick/Marshal.4.8/ast-2.0.0.gemspec.rz [16:21:29] Network error while fetching https://rubygems.org/quick/Marshal.4.8/ast-2.0.0.gemspec.rz [16:21:51] does gem install work? [16:21:55] try installing any gem [16:22:19] Project beta-scap-eqiad build #28416: FAILURE in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/28416/ [16:25:08] Yippee, build fixed! [16:25:08] Project beta-scap-eqiad build #28417: FIXED in 1 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/28417/ [16:25:17] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c15 (10Antoine "hashar" Musso (WMF)) I disconnected the Jenkins Gearman client which is holding the lock. That unleashed the jobs that are meant... [16:27:28] "bundle update" did the job [16:30:37] hashar: since you're here: have you seen this thread? [Ops] Monitoring checks config for labs [16:31:07] greg-g: nop [16:31:29] it's regarind Shinken in labs, something we probably care about :) [16:31:33] regarding* [16:31:39] ah yeah could sync with yuvipanda a bit [16:31:59] I DIDN'T DO IT :) [16:32:10] greg-g: but yeah, in an ideal world all the checks in prod would also be in betalabs [16:32:44] if only we had a project in phabricator we could do regular checkins :-D [16:33:02] Real Soon Now(TM) [16:34:01] should we have a project for labs monitoring in phab? [16:34:21] or just one for monitoring, and just tag on the betalabs one? [16:35:55] seems there are two projects going [16:36:01] one to monitor all of labs instances [16:36:11] and another one to add specific monitors on the beta cluster [16:36:56] 3Wikimedia / 3Quality Assurance: Pending fails upon not finding an element (a failure case that should pass) - 10https://bugzilla.wikimedia.org/72379#c4 (10Jeff Hobson) Sorry for the long delay in response to this, somehow the email slipped by me. I've uploaded the following patch as an example case of this... [16:38:08] Jenkins restarted I guess [16:51:32] Jenkins is now 503 [16:51:43] chrismcmahon: yeah I am restarting it [16:52:03] there is a weird deadlock issue from time to time that prevents jobs to run on beta cluster [16:52:11] it is restarting [17:01:43] Reedy: meeting ping :) [17:01:59] I'm doing useful things! :P [17:01:59] https://github.com/mzmcbride/gerrit-reports/pull/10/files [17:02:09] be there in a min [17:05:04] greg-g: joining in a sec, hangouts is being difficult [17:36:46] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #332: ABORTED in 27 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/332/ [17:37:33] Krinkle: I was waiting to see.. ;) [17:37:39] yeah [17:37:57] Hes hardcore to the end. [17:38:11] A honey badger [18:32:50] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #333: STILL FAILING in 56 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/333/ [18:36:35] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #935: SUCCESS in 33 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/935/ [18:42:39] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #55: FAILURE in 40 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/55/ [19:04:27] 3Wikimedia / 3Continuous integration: Jenkins: Create job for verifying comitted composer vendor directory - 10https://bugzilla.wikimedia.org/72952 (10Greg Grossmeier) p:5Unprio>3Normal [19:28:50] Yippee, build fixed! [19:28:50] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce build #275: FIXED in 8 min 26 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce/275/ [19:56:41] 3Wikimedia / 3Continuous integration: Jenkins: Create job for verifying comitted composer vendor directory - 10https://bugzilla.wikimedia.org/72952 (10Sam Reed (reedy)) s:5normal>3enhanc [20:43:27] (03Abandoned) 10Hashar: ZeroPortal has browser tests [integration/config] - 10https://gerrit.wikimedia.org/r/166553 (https://bugzilla.wikimedia.org/72326) (owner: 10Zfilipin) [20:44:12] 3Wikimedia / 3Quality Assurance: ZeroPortal browsertests fails early due to auth issue with zero.wikimedia.org - 10https://bugzilla.wikimedia.org/72326 (10Antoine "hashar" Musso (WMF)) p:5Unprio>3Normal s:5normal>3enhanc [20:50:14] (03CR) 10Hashar: [C: 04-1] "Jenkins job created:" [integration/config] - 10https://gerrit.wikimedia.org/r/170447 (owner: 10Lewis Cawte) [20:55:33] Project beta-scap-eqiad build #28436: FAILURE in 1 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/28436/ [21:05:41] Yippee, build fixed! [21:05:41] Project beta-scap-eqiad build #28437: FIXED in 1 min 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/28437/ [21:21:45] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #387: STILL FAILING in 53 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/387/ [21:23:01] wat. both failures due to bad API behavior [21:29:44] 3Wikimedia Labs / 3deployment-prep (beta): API error: unable to connect to redis - 10https://bugzilla.wikimedia.org/72982 (10Chris McMahon) 3NEW p:3Unprio s:3normal a:3None Browser tests on beta labs have been getting a fair number of these in recent times: Exception Caught: Unable to connect to re... [21:38:04] Yippee, build fixed! [21:38:05] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #148: FIXED in 9 min 4 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/148/ [21:39:49] Project browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce build #243: FAILURE in 1 min 44 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce/243/ [22:11:41] 3Wikimedia / 3Continuous integration: Jenkins: Set up PHPUnit testing on PostgreSQL backend - 10https://bugzilla.wikimedia.org/37602#c18 (10Tim Landscheidt) I create a separate user and database for each test in my setup because I have some diffuse recollections of not every DB object identifier being schema... [22:50:57] 3Wikimedia Labs / 3deployment-prep (beta): API error: unable to connect to redis - 10https://bugzilla.wikimedia.org/72982#c1 (10Greg Grossmeier) Chris: can you see if this is related (or not) to runs of scap on Beta Cluster a la bug 72366? 503s are now suspicious to me :) [22:51:11] 3Wikimedia Labs / 3deployment-prep (beta): HHVM fcgi restart during scap runs cause 503s (and failed tests) - 10https://bugzilla.wikimedia.org/72366 (10Greg Grossmeier)