[05:34:39] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [06:03:39] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:36:29] Yippee, build fixed! [06:36:30] Project selenium-Wikibase » chrome,test,Linux,contintLabsSlave && UbuntuTrusty build #276: 09FIXED in 1 hr 56 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/276/ [06:44:05] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:19:04] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [08:34:28] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:10:37] 10Continuous-Integration-Infrastructure: Regenerate doxygen pages for mw core - https://phabricator.wikimedia.org/T158423#3039954 (10hashar) 05Open>03Resolved a:03hashar Did a few more tags: zuul enqueue-ref --trigger gerrit --pipeline publish --project mediawiki/core --ref refs/tags/1.28.0 --newrev ae893... [09:45:43] (03PS1) 10Hashar: [mediawiki/vagrant] drop erblint-HEAD [integration/config] - 10https://gerrit.wikimedia.org/r/338724 [09:45:59] (03PS2) 10Hashar: [mediawiki/vagrant] drop erblint-HEAD [integration/config] - 10https://gerrit.wikimedia.org/r/338724 (https://phabricator.wikimedia.org/T154894) [09:48:21] (03PS1) 10Hashar: Remove erblint-HEAD and pplint-HEAD [integration/config] - 10https://gerrit.wikimedia.org/r/338725 (https://phabricator.wikimedia.org/T154894) [09:52:26] 10Continuous-Integration-Config: Phase out jobs puppet-validate jobs - https://phabricator.wikimedia.org/T158544#3040019 (10hashar) [10:12:25] Hi releng, is there somebody who can deploy https://gerrit.wikimedia.org/r/#/c/338726/ ? It's a last-minute throttle rule. [10:17:29] Reedy, twentyafterfour zeljkof ^ [10:17:57] It's for T158432 [10:17:58] T158432: Lift IP registration cap for an event on 2017-02-20 [IP address currently unknown] - https://phabricator.wikimedia.org/T158432 [10:18:40] hashar: ^ [10:18:52] zeljkof, hashar is at a meeting as he said at -operations [10:19:10] I can deploy it [10:19:15] Thank you twentyafterfour [10:19:28] Urbanecm: I'm in a meeting with hashar :) [10:19:39] the other exceptions mention the task in a comment [10:19:44] zeljkof, don't know, sorry then :) [10:57:59] PROBLEM - Puppet run on deployment-mira is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:31:49] (03PS1) 10Hashar: Use puppet-syntax to check puppet manifests [integration/config] - 10https://gerrit.wikimedia.org/r/338746 (https://phabricator.wikimedia.org/T158544) [11:32:59] RECOVERY - Puppet run on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [11:36:19] 10Continuous-Integration-Config: operations/software/cumin should not run debian-glue job on master branch - https://phabricator.wikimedia.org/T158551#3040258 (10hashar) [11:39:42] 10Continuous-Integration-Config: Enhance debian-glue job packages validation - https://phabricator.wikimedia.org/T158553#3040285 (10hashar) [11:46:17] (03PS1) 10Hashar: cumin: skip debian-glue on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/338748 (https://phabricator.wikimedia.org/T158551) [11:47:59] (03CR) 10Hashar: [C: 032] cumin: skip debian-glue on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/338748 (https://phabricator.wikimedia.org/T158551) (owner: 10Hashar) [11:48:50] (03Merged) 10jenkins-bot: cumin: skip debian-glue on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/338748 (https://phabricator.wikimedia.org/T158551) (owner: 10Hashar) [11:52:25] 10Continuous-Integration-Config, 13Patch-For-Review: operations/software/cumin should not run debian-glue job on master branch - https://phabricator.wikimedia.org/T158551#3040326 (10hashar) 05Open>03Resolved a:03hashar Did a `recheck` on a change against the `master` branch ( https://gerrit.wikimedia.or... [11:55:31] 10Continuous-Integration-Config: Enhance debian-glue job packages validation - https://phabricator.wikimedia.org/T158553#3040333 (10Volans) Another couple of enhancement that could be done for lintian, looking at the source code at https://github.com/mika/jenkins-debian-glue/blob/16f0ba5565435e12cb211c686bd5a49c... [12:10:50] 10Continuous-Integration-Config, 13Patch-For-Review: operations/software/cumin should not run debian-glue job on master branch - https://phabricator.wikimedia.org/T158551#3040341 (10Volans) @hashar I'm wondering if it could be easier to do the opposite and run only if matches a specific branch, `debian` in my... [12:18:28] 10Continuous-Integration-Config, 13Patch-For-Review: operations/software/cumin should not run debian-glue job on master branch - https://phabricator.wikimedia.org/T158551#3040345 (10hashar) > I'm wondering if it could be easier to do the opposite and run only if matches a specific branch, debian in my case, bu... [12:20:36] (03PS1) 10Hashar: debian-glue: only lintian against *.changes [integration/config] - 10https://gerrit.wikimedia.org/r/338752 (https://phabricator.wikimedia.org/T158553) [12:22:12] 10Continuous-Integration-Config, 13Patch-For-Review: Enhance debian-glue job packages validation - https://phabricator.wikimedia.org/T158553#3040356 (10hashar) [12:27:15] (03CR) 10Volans: [C: 031] "LGTM" [integration/config] - 10https://gerrit.wikimedia.org/r/338752 (https://phabricator.wikimedia.org/T158553) (owner: 10Hashar) [12:36:26] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#3040424 (10zeljkofilipin) http://web.archive.org/web/20160713013212/http://blog.crowdint.com/2011/08/22/auto-retr... [12:36:49] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#3040425 (10zeljkofilipin) Testing: https://integration.wikimedia.org/ci/job/selenium-Math-338368/ [12:42:07] (03CR) 10Hashar: [C: 032] "Jobs updated :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/338752 (https://phabricator.wikimedia.org/T158553) (owner: 10Hashar) [12:45:41] (03Merged) 10jenkins-bot: debian-glue: only lintian against *.changes [integration/config] - 10https://gerrit.wikimedia.org/r/338752 (https://phabricator.wikimedia.org/T158553) (owner: 10Hashar) [12:45:57] 10MediaWiki-Codesniffer: Provide an MWCS rule to enforce "short" type definitions: int and bool, not integer and boolean - https://phabricator.wikimedia.org/T145162#3040468 (10Ricordisamoa) >>! In https://github.com/squizlabs/PHP_CodeSniffer/pull/1351#issuecomment-280510812, [[ https://github.com/gsherwood | gsh... [12:50:46] !log integration-slave-jessie-1001 downgraded cowbuilder to 0.73 from jessie to match integration-slave-jessie-1002 [12:50:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:04:01] Hi releng again, may I ask for adding next week to https://wikitech.wikimedia.org/wiki/Deployments? [13:06:49] it's US holiday, so less people are around [13:07:04] (03PS1) 10Hashar: debian-glue: add debc to dump packages content [integration/config] - 10https://gerrit.wikimedia.org/r/338760 (https://phabricator.wikimedia.org/T158553) [13:07:14] Nikerabbit, that means no deploys? [13:07:30] 10Continuous-Integration-Config, 13Patch-For-Review: Enhance debian-glue job packages validation - https://phabricator.wikimedia.org/T158553#3040512 (10hashar) [13:08:13] Urbanecm: I don't know where the base template is to generate the new deployments page [13:08:23] supposedly one has to include some template [13:08:53] Urbanecm: I guess we can just handle that via the IRC channel for now [13:09:07] Okay, so the time and other things are normal hashar? [13:09:14] nop [13:09:17] it is a us holiday [13:09:23] so supposedly we are not deploying stuff [13:09:35] then I don't think it makes sense to have such a restricting for the european swat [13:10:29] I don't understand you. One time you say we'll handle it via IRC channel for now and then that we're not deploying anything. [13:10:57] well [13:11:06] I have no idea how/who update the deployments page [13:11:12] supposedly that is done on friday US evening [13:11:25] that hasn't been done because it is a manual process [13:11:42] and I have no idea where the template is [13:11:58] then second topic is: today is an holiday in the USA [13:12:04] and we don't deploy during holiday [13:12:38] then for the mediawiki-config changes during European SWAt, I don't think we need anyone from the US [13:14:47] Yes, I understand this. According to the history of deployments calendar update is done by greg-g. But regarding the last message. Will I have chance to get my config changes deployed at 14:00 UTC? [13:17:07] (03CR) 10Volans: [C: 031] "LGTM and tested live" [integration/config] - 10https://gerrit.wikimedia.org/r/338760 (https://phabricator.wikimedia.org/T158553) (owner: 10Hashar) [13:17:27] (03CR) 10Hashar: [C: 032] "Thanks for the test :)" [integration/config] - 10https://gerrit.wikimedia.org/r/338760 (https://phabricator.wikimedia.org/T158553) (owner: 10Hashar) [13:18:56] (03Merged) 10jenkins-bot: debian-glue: add debc to dump packages content [integration/config] - 10https://gerrit.wikimedia.org/r/338760 (https://phabricator.wikimedia.org/T158553) (owner: 10Hashar) [13:20:53] Urbanecm: lets handle them purely via IRC [13:21:03] I don't want to waste half an hour figuring out how to update the Deployments page [13:21:08] but let me check with operations people first [13:22:09] Urbanecm: if that is just the throttle rule https://gerrit.wikimedia.org/r/#/c/338128/ [13:22:14] I can even deploy it right now [13:23:25] Yes, just this rule. [13:24:34] Thank you [13:26:21] Urbanecm: will do right now [13:26:27] Thank you [13:28:52] rebased it [13:44:11] zeljkof: for when you are around, I need your input :-} [13:44:24] I am trying to have different names for the various browser tests we run [13:44:24] hashar: around! [13:44:52] a first set are the daily browser tests that runs against SauceLabs , the jenkins slaves running it just need ruby (more or less) [13:45:08] the second step are the browser tests triggered on patch proposals eg mwext-mw-selenium or something [13:45:28] which run locally and need a web browser, xvfb, chromium, chromium-driver etc [13:45:33] hashar: any way to get more info for https://gerrit.wikimedia.org/r/#/c/337545/ ? -- I cannot reproduce locally [13:45:41] I would like to have each set to run on different slaves [13:45:57] and looking for good names cause using "browsertests" for both is really confusing me [13:46:12] Nikerabbit: what is up on that change? [13:46:29] Nikerabbit: ah that is qunit [13:46:44] Nikerabbit: have you tried running them with the same set of extensions? [13:47:47] zeljkof: so my goal is to have different puppet role classes for each set of slaves [13:48:09] hashar: not sure my development environment can handle for example scribunto [13:48:26] Nikerabbit: for qunit you would not need it most probably [13:48:50] "I am missing few extensions from the list: GuidedTour, Scribunto, Wikidata." [13:49:06] hashar: so, what is the question? how to name the two jobs? [13:49:08] PROBLEM - Puppet run on buildlog is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:49:21] selenium-sauce and selenium-local? [13:49:31] zeljkof: how to name the puppet role and instances that will run the daily browser tests on sauce labs :D [13:49:36] that is bike shedding really [13:50:00] right now we have role::ci::slave::browsertests [13:50:21] which installs all browsers, xvfb etc [13:51:37] role::ci::slave::selenium::local and role::ci::slave::selenium::saucelabs? [13:52:03] zeljkof: yeah sounds good [13:52:43] Nikerabbit: ext.cx.tools.template Equivalent template exists FAILED >>> Expected 1 assertions, but 0 were run [13:52:52] Nikerabbit: who knows why no assertions are run at all :( [13:56:43] hashar: yeah it's a mystery [13:58:10] hashar: I cannot think of anything else but riddling the source code with console.log debug statements and trying to run that via jenkins [13:58:14] zeljkof: so I am just going to name them saucelabs [13:58:25] hashar: sounds good [13:58:35] zeljkof: eg role::ci::slave::saucelabs and instances names saucelabs-##### [13:58:47] create a couple of small instances for redundancy and call it an end [13:58:57] +1 [13:59:10] then I hope to be able to attach them to a standalone Jenkins master that is dedicated to the daily browsertests [13:59:25] * hashar heads to refactor puppet [14:00:22] Nikerabbit: from a quick look there is nothing suspicious in the debug logs :( [14:02:52] hashar: I would love ajax network request log :D [14:03:22] Nikerabbit: I don't know whether karma dump the browser console log [14:03:58] one would have to dig in karma documentation and adjust mediawiki/core Gruntfile.js in consequence [14:28:14] zeljkof: https://gerrit.wikimedia.org/r/338770 contint: slave role for Saucelabs jobs :) [14:28:34] so I now get a very very simple role "role::ci::slave::saucelabs" [14:28:47] which just installs ruby and libav-tools [14:29:38] will create an instance that uses that [14:32:58] PROBLEM - Host browsertests-1001 is DOWN: CRITICAL - Host Unreachable (10.68.19.196) [14:33:17] hashar: we don't even need video recording for sauce jobs [14:33:22] it is done on sauce labs side [14:33:22] ^^^ I deleted it [14:33:23] oh man [14:33:24] right [14:33:52] so basically the slave would only have ruby right ? ;} [14:34:08] yes, ruby and bundler [14:34:23] hm, probably chromedriver too, for selenium 3 [14:34:28] I think we install bundler directly from gem [14:34:36] chromedriver ? [14:34:46] arent we just driving the sauce labs directly? [14:34:54] not sure how it works [14:35:14] I think the local client sends commands to the server [14:35:45] but I don't know if sauce has the browser driver (chromedriver) or if we have to have it locally [14:35:47] we can test [14:42:32] zeljkof: https://gerrit.wikimedia.org/r/338770 updated [14:42:40] basically just install ruby for now [14:43:07] hashar: commit message is missing "c" at the very beginning :) [14:43:23] yeah [14:43:25] and I have to install bundler I guess [14:43:36] the selenium jobs install it from ruby gems via a gem install [14:43:44] which was because trusty shipped an old version [14:43:52] I will move the jobs to Jessie in the process [14:43:58] yes, ruby and bundler, the dependencies are installed via bundler, probably even including chromedriver, if needed [14:44:14] oh but [14:44:21] bundler is installed by contint::packages::ruby :} [14:44:41] https://rubygems.org/gems/chromedriver-helper/versions/1.0.0 [14:45:52] yeah well will come to that later I guess [14:47:28] PROBLEM - Puppet run on deployment-logstash2 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:47:52] now the trick is figuring out how to add a class that is not merged in operations/puppet.git yet bah [14:49:35] PROBLEM - Puppet run on deployment-elastic06 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:52:11] bah [14:52:28] I even get Trebuchet installed for some reason [14:52:29] ... [14:53:43] !log integration: applying role::ci::slave::saucelabs to saucelabs-01 [14:53:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:56:26] bundler 1.7.4 [14:56:29] lets hope it is enough [15:04:06] (03Abandoned) 10Zfilipin: Move Rake and RuboCop dependency from repositories to mediawiki/selenium [selenium] - 10https://gerrit.wikimedia.org/r/338137 (https://phabricator.wikimedia.org/T158326) (owner: 10Zfilipin) [15:04:39] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 07Ruby, 15User-zeljkofilipin: Move Rake and RuboCop dependency from repositories to mediawiki/selenium - https://phabricator.wikimedia.org/T158326#3040940 (10zeljkofilipin) 05Open>03declined [15:20:55] hashar: regarding Zend 5.3 / CI, can I leave the task of reaching a decision/implementing to you, or do you need further input/herding from me? [15:27:29] RECOVERY - Puppet run on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [15:29:33] RECOVERY - Puppet run on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [15:33:22] andrewbogott: good morning :} [15:33:39] andrewbogott: can you poke the task instead please ? Supposedly I will raise the topic during the weekly releng meeting tomorrow [15:33:52] I don't even know how much refactoring will be needed in CI :/ [15:36:18] hashar hi, managed to build zuul from upstream. I backported the changes onto gerrit-test :). I built it using http://gerrit-jenkins.wmflabs.org/job/debian-glue-non-voting/14/ and have upgraded zuul to that version locally and it works :). I built https://gerrit.wikimedia.org/r/#/c/338620/ [15:41:20] (03PS1) 10Hashar: Migrate Selenium jobs to jessie/dedicated slaves [integration/config] - 10https://gerrit.wikimedia.org/r/338784 [15:45:32] (03PS1) 10Zfilipin: WIP Increase in failures caused by Saucelabs [selenium] - 10https://gerrit.wikimedia.org/r/338785 (https://phabricator.wikimedia.org/T152963) [15:59:24] Project selenium-Core » firefox,beta,Linux,BrowserTests build #322: 04FAILURE in 6 min 34 sec: https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/322/ [16:02:17] ^^^ that is me [16:02:22] 00:02:52.036 /tmp/hudson910171424565520258.sh: line 14: /srv/deployment/integration/slave-scripts/bin/mw-api-siteinfo.py: No such file or directory [16:02:22] bah [16:09:22] PROBLEM - Puppet run on saucelabs-01 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [0.0] [16:19:24] RECOVERY - Puppet run on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:28:20] zeljkof: it works https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/326/console ! :} [16:29:10] (03PS1) 10Umherirrender: [DeskMessMirrored] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/338791 [16:29:18] hashar: nice! [16:31:52] the puppet part is a bit dirty [16:32:48] Yippee, build fixed! [16:32:48] Project selenium-Core » firefox,beta,Linux,BrowserTests build #326: 09FIXED in 6 min 54 sec: https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/326/ [16:39:52] adding a second slave [16:53:33] zeljkof: I am going to test/migrate rest of the jobs [16:53:53] hashar: great [16:55:22] PROBLEM - Puppet run on saucelabs-01 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [16:56:59] Project selenium-CentralNotice » chrome,beta,Linux,BrowserTests build #311: 09SUCCESS in 1 min 34 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/311/ [16:57:05] Project selenium-CentralNotice » firefox,beta,Linux,BrowserTests build #311: 09SUCCESS in 1 min 40 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/311/ [16:57:33] the selenium-CirrusSearch will surely breaks though [16:58:08] Project selenium-CentralNotice » chrome,beta,OS X 10.9,BrowserTests build #311: 09SUCCESS in 2 min 43 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/311/ [16:58:38] Project selenium-CentralNotice » chrome,beta,Windows 7,BrowserTests build #311: 09SUCCESS in 3 min 13 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%207,label=BrowserTests/311/ [16:58:49] Project selenium-CentralNotice » firefox,beta,OS X 10.9,BrowserTests build #311: 09SUCCESS in 3 min 25 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/311/ [16:59:24] Project selenium-CentralNotice » firefox,beta,Windows 7,BrowserTests build #311: 09SUCCESS in 4 min 0 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%207,label=BrowserTests/311/ [16:59:39] zeljkof: not sure they should be alerting on a success -> success [16:59:39] :D [17:00:21] RECOVERY - Puppet run on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:00:29] Project selenium-CirrusSearch » firefox,beta,Linux,BrowserTests build #319: 09SUCCESS in 4 min 49 sec: https://integration.wikimedia.org/ci/job/selenium-CirrusSearch/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/319/ [17:00:47] Project selenium-CentralAuth » firefox,beta,Linux,BrowserTests build #311: 09SUCCESS in 5 min 31 sec: https://integration.wikimedia.org/ci/job/selenium-CentralAuth/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/311/ [17:09:23] Project selenium-Echo » chrome,beta,Linux,BrowserTests build #310: 09SUCCESS in 56 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/310/ [17:09:24] Project selenium-Echo » firefox,beta,Linux,BrowserTests build #310: 09SUCCESS in 1 min 8 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/310/ [17:09:25] Project selenium-Flow » chrome,beta,Linux,BrowserTests build #307: 09SUCCESS in 1 min 38 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/307/ [17:09:25] Project selenium-Flow » firefox,beta,Linux,BrowserTests build #307: 09SUCCESS in 1 min 55 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/307/ [17:09:26] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #313: 09SUCCESS in 2 min 3 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/313/ [17:09:26] Project selenium-Math » chrome,beta,Linux,BrowserTests build #308: 09SUCCESS in 1 min 49 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/308/ [17:09:27] Project selenium-Math » firefox,beta,Linux,BrowserTests build #308: 09SUCCESS in 2 min 10 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/308/ [17:13:55] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #304: 04FAILURE in 4 min 57 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/304/ [17:19:56] PROBLEM - Host saucelabs-03 is DOWN: CRITICAL - Host Unreachable (10.68.17.200) [17:33:05] Project selenium-MobileFrontend » chrome,beta,Linux,BrowserTests build #333: 09SUCCESS in 24 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/333/ [17:34:08] Project selenium-PageTriage » chrome,beta,Linux,BrowserTests build #307: 09SUCCESS in 24 min: https://integration.wikimedia.org/ci/job/selenium-PageTriage/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/307/ [17:34:57] RECOVERY - Host saucelabs-03 is UP: PING OK - Packet loss = 0%, RTA = 1.92 ms [17:35:09] Project selenium-PageTriage » firefox,beta,Linux,BrowserTests build #307: 09SUCCESS in 25 min: https://integration.wikimedia.org/ci/job/selenium-PageTriage/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/307/ [17:39:39] Project selenium-QuickSurveys » chrome,beta,Linux,BrowserTests build #318: 09SUCCESS in 28 min: https://integration.wikimedia.org/ci/job/selenium-QuickSurveys/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/318/ [17:40:22] Project selenium-RelatedArticles » chrome,beta-desktop,Linux,BrowserTests build #322: 09SUCCESS in 29 min: https://integration.wikimedia.org/ci/job/selenium-RelatedArticles/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta-desktop,PLATFORM=Linux,label=BrowserTests/322/ [17:41:09] Project selenium-RelatedArticles » chrome,beta-mobile,Linux,BrowserTests build #322: 09SUCCESS in 30 min: https://integration.wikimedia.org/ci/job/selenium-RelatedArticles/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta-mobile,PLATFORM=Linux,label=BrowserTests/322/ [17:42:09] Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #309: 09SUCCESS in 31 min: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/309/ [17:42:10] Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #333: 04FAILURE in 34 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/333/ [17:42:41] (03CR) 10Hashar: "I have migrated all the jobs to the new Jenkins slaves and triggered a build for each of them. Will check tonight whether all is fine." [integration/config] - 10https://gerrit.wikimedia.org/r/338784 (owner: 10Hashar) [17:43:30] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #313: 04FAILURE in 32 min: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/313/ [17:44:27] Project selenium-MultimediaViewer » firefox,mediawiki,Linux,BrowserTests build #305: 09SUCCESS in 28 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=mediawiki,PLATFORM=Linux,label=BrowserTests/305/ [17:51:46] (03CR) 10Hashar: [C: 032] [DeskMessMirrored] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/338791 (owner: 10Umherirrender) [17:52:56] (03Merged) 10jenkins-bot: [DeskMessMirrored] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/338791 (owner: 10Umherirrender) [17:54:25] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,BrowserTests build #305: 09SUCCESS in 38 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/305/ [18:05:47] Project selenium-MultimediaViewer » chrome,beta,OS X 10.9,BrowserTests build #305: 09SUCCESS in 50 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/305/ [18:21:44] 10Deployment-Systems, 10MediaWiki-Debug-Logger: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#3041383 (10Tgr) [18:31:06] 10Deployment-Systems, 10MediaWiki-Debug-Logger: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#3041403 (10Tgr) What's the current state of this task? Is it blocked in the log volume being too high? There are a few things we could do about that... [18:49:55] 10Continuous-Integration-Infrastructure: Remove throttling plugin from Jenkins - https://phabricator.wikimedia.org/T158596#3041464 (10hashar) [18:50:30] (03PS1) 10Hashar: Stop throttling mwext-Wikibase jobs [integration/config] - 10https://gerrit.wikimedia.org/r/338816 (https://phabricator.wikimedia.org/T158596) [18:54:11] 10Deployment-Systems, 10MediaWiki-Debug-Logger: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#3041483 (10bd808) >>! In T45086#3041403, @Tgr wrote: > What's the current state of this task? Is it blocked in the log volume being too high? There... [18:58:23] Yippee, build fixed! [18:58:23] Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #334: 09FIXED in 1 hr 14 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/334/ [18:58:42] \O/ [18:58:47] zeljkof: some tests are flappy :/ [19:02:11] 10Deployment-Systems, 10MediaWiki-Debug-Logger: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#3041502 (10Tgr) >>! In T158270#3041160, @bd808 wrote: > We [return false from the error handler] that because it's the correct way for a default `se... [19:07:53] (03CR) 10Hashar: [C: 032] "Jobs updated. Less cruft in the CI config :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/338816 (https://phabricator.wikimedia.org/T158596) (owner: 10Hashar) [19:09:07] 10Deployment-Systems, 10MediaWiki-Debug-Logger, 13Patch-For-Review: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#3041541 (10bd808) >>! In T45086#3041502, @Tgr wrote: >>>! In T158270#3041160, @bd808 wrote: >> We [return false from the error... [19:09:56] 10Deployment-Systems, 10MediaWiki-Debug-Logger, 13Patch-For-Review: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#3041542 (10tstarling) Is someone actually saying the log volume is too high? It looks to me like it's blocked on nobody giving... [19:10:08] (03Merged) 10jenkins-bot: Stop throttling mwext-Wikibase jobs [integration/config] - 10https://gerrit.wikimedia.org/r/338816 (https://phabricator.wikimedia.org/T158596) (owner: 10Hashar) [19:13:19] 10Deployment-Systems, 10MediaWiki-Debug-Logger, 13Patch-For-Review: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#3041544 (10Tgr) >>! In T45086#3041483, @bd808 wrote: > One way to provide granularity without creating N log channels would be... [19:20:44] 10Deployment-Systems, 10MediaWiki-Debug-Logger, 13Patch-For-Review: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#3041550 (10Tgr) >>! In T45086#3041541, @bd808 wrote: > By not returning false and instead handling everything internally in th... [19:20:51] lots of jobs queued @ zuul [19:21:39] :( [19:21:56] Yippee, build fixed! [19:21:57] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #306: 09FIXED in 54 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/306/ [19:22:02] there are a bunch of patches for the Popups mw extension [19:22:22] tabbycat: so that will take a while. And changes that receives CR+2 are run before the rest [19:23:11] hashar: okay, I though they got stalled [19:23:33] tabbycat: https://integration.wikimedia.org/zuul/ shows the queue [19:23:43] tabbycat: and on Jenkins on the left side of https://integration.wikimedia.org/ci/ [19:23:49] you get the builds going on per instances [19:24:11] tabbycat: https://integration.wikimedia.org/ci/label/ci-jessie-wikimedia/load-statistics might gives an idea of how busy it is [19:24:53] anyway I am off [19:27:24] 10Deployment-Systems, 10MediaWiki-Debug-Logger, 13Patch-For-Review: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#3041557 (10bd808) >>! In T45086#3041550, @Tgr wrote: >>>! In T45086#3041541, @bd808 wrote: >> By not returning false and inste... [19:32:25] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T157638#3041558 (10greg) a:03demon [19:32:42] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T157639#3041559 (10greg) a:03demon [19:36:13] 10Deployment-Systems, 10MediaWiki-Debug-Logger, 13Patch-For-Review: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#3041561 (10Tgr) >>! In T45086#3041557, @bd808 wrote: > How is raising an unhandled exception functionally different than passi... [19:40:47] 10Deployment-Systems, 10MediaWiki-Debug-Logger, 13Patch-For-Review: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#3041564 (10tstarling) >>! In T45086#3041550, @Tgr wrote: >>>! In T45086#3041541, @bd808 wrote: >> By not returning false and i... [19:45:21] bd808, tgr: I don't get this thing about logging suppressed errors [19:45:59] people couldn't even be bothered fixing actual non-suppressed errors for months and months (by people I mean me) [19:46:28] and suppressed errors are not bugs, errors are usually suppressed for a good reason [19:47:15] 10Deployment-Systems, 10MediaWiki-Debug-Logger, 13Patch-For-Review: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#3041566 (10bd808) >>! In T45086#3041561, @Tgr wrote: > The point is that the handler should not return false on errors which P... [19:47:37] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team: Investigate installing the zuul plugin for gerrit - https://phabricator.wikimedia.org/T158600#3041567 (10Paladox) [19:51:23] Project selenium-Wikibase » chrome,test,Linux,BrowserTests build #277: 09SUCCESS in 2 hr 9 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=BrowserTests/277/ [19:55:19] 10Continuous-Integration-Infrastructure: jsduck publish error: index-pack died of signal 15 - https://phabricator.wikimedia.org/T158601#3041581 (10Reedy) [19:57:23] 10Continuous-Integration-Config, 05Goal, 07I18n, 13Patch-For-Review: Configure banana checker for i18n files to run on all MediaWiki extensions and skins - https://phabricator.wikimedia.org/T94547#3041594 (10Umherirrender) [19:57:51] 10Continuous-Integration-Config, 05Goal, 07I18n, 13Patch-For-Review: Configure banana checker for i18n files to run on all MediaWiki extensions and skins - https://phabricator.wikimedia.org/T94547#2949515 (10Umherirrender) updated skins and extensions list [19:58:14] 10Continuous-Integration-Infrastructure, 10Gerrit: jsduck publish error: index-pack died of signal 15 - https://phabricator.wikimedia.org/T158601#3041597 (10Paladox) [19:58:44] 10Continuous-Integration-Infrastructure, 10Gerrit: jsduck publish error: index-pack died of signal 15 - https://phabricator.wikimedia.org/T158601#3041581 (10Paladox) That looks like a gerrit error. Though we could try deleting it's local clone on the zuul server to see if that fixes it? [20:00:20] Project selenium-Wikibase » chrome,beta,Linux,BrowserTests build #277: 09SUCCESS in 2 hr 18 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/277/ [20:01:19] 10Continuous-Integration-Config, 05Goal, 07I18n, 13Patch-For-Review: Configure banana checker for i18n files to run on all MediaWiki extensions and skins - https://phabricator.wikimedia.org/T94547#3041599 (10Umherirrender) [20:20:50] 10Continuous-Integration-Infrastructure, 10Gerrit: jsduck publish error: index-pack died of signal 15 - https://phabricator.wikimedia.org/T158601#3041608 (10Reedy) Certainly gerrit, and probably transient [20:21:15] (03PS2) 10Hashar: Migrate Selenium jobs to jessie/dedicated slaves [integration/config] - 10https://gerrit.wikimedia.org/r/338784 [20:21:46] (03CR) 10Hashar: [C: 032] "They all pass with the exception of VisualEditor but I am pretty sure it is unrelated :}" [integration/config] - 10https://gerrit.wikimedia.org/r/338784 (owner: 10Hashar) [20:23:41] (03Merged) 10jenkins-bot: Migrate Selenium jobs to jessie/dedicated slaves [integration/config] - 10https://gerrit.wikimedia.org/r/338784 (owner: 10Hashar) [20:25:02] TimStarling: sure, as long as we have unsuppressed warnings on production, let's focus on that [20:25:24] I want to send suppressed errors to a separate channel and blackhole it in WMF config [20:26:24] 10Continuous-Integration-Infrastructure, 10Gerrit: jsduck publish error: index-pack died of signal 15 - https://phabricator.wikimedia.org/T158601#3041614 (10hashar) [20:26:56] but most of it should be fixed eventually, it's the equivalent of an empty catch block [20:27:09] Is this both wfSuppressWarnings and also @ prefixes? [20:27:15] yes [20:27:45] and I guess everything else if some wiki admin sets a higher error_reporting [20:36:53] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#3041633 (10Aklapper) https://gerrit.wikimedia.org/r/#/c/326154/ and https://gerrit.wikimedia.org/r/#/c/326161/ were merged. https://g... [20:36:56] 10Continuous-Integration-Infrastructure, 06Operations, 10netops: jsduck publish error: index-pack died of signal 15 - https://phabricator.wikimedia.org/T158601#3041635 (10hashar) The jenkins jobs triggered by Zuul clones the repo from the zuul-merger instances on contint1001 / contint2001. They are being ser... [20:37:40] 10Continuous-Integration-Infrastructure, 06Operations, 10netops: git clone over EQIAD (wmflabs) CODFW timeout due to low bandwidth (~250 KiB/s) - https://phabricator.wikimedia.org/T158601#3041638 (10hashar) [20:40:20] 10Continuous-Integration-Infrastructure, 06Labs, 06Operations, 10netops: git clone over EQIAD (wmflabs) CODFW timeout due to low bandwidth (~250 KiB/s) - https://phabricator.wikimedia.org/T158601#3041644 (10Paladox) [20:41:10] 10Continuous-Integration-Infrastructure, 06Labs, 06Operations, 10netops: git clone over EQIAD (wmflabs) CODFW timeout due to low bandwidth (~250 KiB/s) - https://phabricator.wikimedia.org/T158601#3041581 (10Paladox) Is this happening to any other repos? Should we set this as normal or high priority?