[00:02:07] Yippee, build fixed! [00:02:08] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #207: FIXED in 2 min 47 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/207/ [03:15:30] Yippee, build fixed! [03:15:30] Project browsertests-TwnMainPage-sandbox.translatewiki.net-linux-firefox-sauce build #169: FIXED in 11 min: https://integration.wikimedia.org/ci/job/browsertests-TwnMainPage-sandbox.translatewiki.net-linux-firefox-sauce/169/ [04:34:13] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #36: FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/36/ [04:44:40] Yippee, build fixed! [04:44:41] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #103: FIXED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/103/ [05:29:16] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #89: FAILURE in 8 min 35 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/89/ [06:56:30] 3Wikimedia / 3Continuous integration: Write and implement tests for Wikimedia's Apache configuration (redirects.conf, etc.) - 10https://bugzilla.wikimedia.org/43266 (10Krinkle) [07:16:17] morning [07:29:02] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #252: FAILURE in 59 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/252/ [07:39:47] (03CR) 10Hashar: "I found such a use case for mediawiki/core . It more or less pass lenient thought not strict (there is still a lot of warnings). The chan" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161757 (https://bugzilla.wikimedia.org/48420) (owner: 10Jforrester) [07:47:30] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #49: FAILURE in 4 min 29 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/49/ [07:53:45] (03PS9) 10Hashar: Dedicated VisualEditor qunit job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 [07:59:31] (03PS1) 10Krinkle: [WIP] macro/npm: Set CHROME_BIN to chromium-browser [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164906 [08:00:05] hashar: I can cherry-pick the ops change (I've done this in the past, no problem), but when using a new role, how can I apply that? [08:00:23] It seems the wikitech interface only knows about roles that exist in operations puppet, it doesnt' look at the self puppetmaster [08:02:57] (03CR) 10Hashar: [V: 031] "I though upstream developers of VisualEditor/VisualEditor might be interesting to see the result of testing their patch with the head of m" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 (owner: 10Hashar) [08:03:19] (03CR) 10Krinkle: [C: 04-1] "This change doesn't seem to reflect the intent. It changes it from running non-voting on all branches to running only on the master branch" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/153575 (https://bugzilla.wikimedia.org/46500) (owner: 10Addshore) [08:05:01] (03PS1) 10Hashar: Enable VisualEditor-qunit [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/164907 [08:05:24] (03CR) 10Hashar: "Zuul configuration tweaking is done with https://gerrit.wikimedia.org/r/164907" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 (owner: 10Hashar) [08:05:52] Krinkle: good morning :] [08:06:57] Krinkle: I would just include the new role::ci::slave::localbrowser from role::ci::slave::labs [08:07:08] Krinkle: else you would have to edit in wikitech all the instances to add the new class [08:07:14] (03CR) 10Krinkle: "People don't have to install or locally per se. Announcement to wikitech-l is fine to inform that we've finally finished the pass and have" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/153575 (https://bugzilla.wikimedia.org/46500) (owner: 10Addshore) [08:07:18] role::ci::slave::labs is meant to be used as the entry point [08:07:24] hashar: k [08:07:43] though you only want to include it when the host is trusty [08:07:58] hashar: Did you see my review on https://gerrit.wikimedia.org/r/#/c/163837/ ? [08:08:04] hashar: Did someone ask for this job? [08:09:08] Krinkle: na I thought about it myself :D [08:09:38] I thought folks developing in upstream repo VE/VE would be interested in finding out what their patch do to the mediawiki extension qunit job [08:09:41] as a non voting hint [08:10:24] you explained pretty well that upstream can introduce breaking changes and the integration work is done when updating the submodule in mediawiki/extensions/VisualEditor [08:10:26] well, it can never be voting because it's downstream. Breaking changes are normal in iteration. Periodically when we update the submodule, that commit will naturally test the new result and require any updates to be made to the calling code if there are breaking changes. [08:10:35] so I submitted a new patch that remove all the shell if/else cruft and just update the submodule [08:10:46] yeah I got the point [08:11:01] I though VE/VE might be willing to maintain back compatibility or something :] [08:11:35] PS9 clears out [08:11:40] hashar: Hm.. [08:11:54] it is now just a copy of the existing mwext-{ext-name}-qunit job but with a builder to update the submodule [08:11:56] hashar: It should be triggered on ve/ve, right? not ve-mw [08:12:35] ve/ve is upstream isn't it ? And you said there was no point in running the qunit jobs there [08:12:43] -_- [08:12:47] :-D [08:12:51] jetlag! [08:13:12] Your idea is to test downstream with newer upstream version to see if there is an impact [08:13:17] yes [08:13:21] but can be figured out later I guess [08:13:22] e.g. changes to jQuery affecting MediaWiki (for comparison) [08:13:27] exactly [08:13:33] or change of oojs / ooui [08:13:35] Your commit currently does the revere which is useless [08:13:39] I think you just typoed it [08:13:44] to figure out whether the head of master still work with mediawiki/core@master [08:13:49] Right nwo it's triggering the job on mediawiki/extensions/VisualEditor [08:13:58] with doesn't make sense [08:14:23] yeah it is to replace the mwext-VisualEditor-qunit job which does not process submodules [08:14:39] well, that one is still in there [08:15:09] yup [08:15:18] wait, these are different thigns [08:15:20] I have added a comment mentioning it is not to be triggered [08:15:48] do you want to replace the regular qunit test (loading the submodule according to the commit, not latest upstream master), or add a test for downstream/upstream test. [08:15:49] the reason is that VE is a huge list of extensions which realize a bunch of tempates [08:16:19] ultimately I want to trigger a qunit job against mediawiki/extensions/VisualEditor which process the submodule in that repo [08:16:21] As it is https://gerrit.wikimedia.org/r/#/c/164907/1/layout.yaml looks confusing, can you confirm? [08:16:29] forget about testing downstream/upstream :] [08:16:40] ? [08:17:04] you just spent a fair amount working on that, and it seems useful as information to have when changing VE to know whether it breaks ve-mw intentionally or not. [08:17:22] yeah but I am willing to do that later on, can always be copy pasted from the previous patchset [08:17:37] what I need to be done is have the VE qunit job to run with zuul cloner [08:17:42] so it fetch mediawiki/vendor.git [08:17:42] It's not high prio or anything like that, but I'm confused you seem to drop the idea? [08:17:43] ok [08:17:51] so I am willing to drop the idea [08:18:06] well, then call it mwext-VisualEditor-qunit so it replaces/fixes the job [08:18:08] document it and propose it to the VE team to figure out whether it is a good thing to have (I think it is) [08:18:25] but testing downstream/upstream changes need a bit more thoughts [08:18:26] assuming it still has the same semantics (regular mwext with its submodules, no downstream/upstream latest magic) [08:18:29] sure [08:18:50] will look at having it named mwext-VisualEditor-qunit [08:18:55] 'VisualEditor-qunit' looks like a job for upstream visualeditor that tests something with downstream mediawiki [08:18:56] probably by defining a job template named mwext-VisualEditor-qunit :D [08:19:01] yeah that is confusing indeed [08:19:02] Exactly [08:19:18] point taken. refactoring! [08:19:24] and in the patch as it is, it runs both, one of which is broken according to you (doesn't run submodules), we don't need to run both. [08:19:26] OK :) [08:19:32] * Krinkle is on coffee [08:19:38] * hashar sends donuts [08:19:44] are you back in Europe? [08:19:47] Yep [08:19:53] As of last night [08:19:54] don't forget to rest a bit ! [08:20:17] I've slept the entire night, I'm all reset now :) [08:24:13] (03PS10) 10Hashar: Dedicated VisualEditor qunit job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 [08:24:58] updated mwext-VisualEditor-qunit [08:26:25] running on https://integration.wikimedia.org/ci/job/mwext-VisualEditor-qunit/11697/console [08:30:14] it pass! [08:30:16] (03PS2) 10Hashar: Remove VisualEditor-qunit [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/164907 [08:31:45] (03CR) 10Hashar: [V: 031] "PS10 renames the job mwext-VisualEditor-qunit to avoid confusion (VisualEditor-qunit would suggest it is testing VE/VE when this job reall" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 (owner: 10Hashar) [08:42:31] Krinkle: I think it is fine now :] [08:44:40] hashar: looks wrong [08:44:46] https://integration.wikimedia.org/ci/job/mwext-VisualEditor-qunit/11697/consoleFull [08:44:50] 08:28:54 Submodule path 'lib/ve': checked out 'a1c190c327541702cbae58b98b3f9670daff9909' [08:44:54] https://git.wikimedia.org/tree/mediawiki%2Fextensions%2FVisualEditor/e6f1f52a9c7b1feeca85cb2ea3bb9c66d5ca446a/lib [08:44:57] ve @ 3aafc81 [08:45:01] https://github.com/wikimedia/VisualEditor/commit/a1c190c327541702cbae58b98b3f9670daff9909 [08:45:04] https://github.com/wikimedia/VisualEditor/commit/3aafc81e1e6289df76760ead51e01426889aaa51 [08:45:06] 3 days ago [08:45:09] 9 days ago [08:45:22] I tested it with an old change probably [08:45:37] https://gerrit.wikimedia.org/r/#/c/163847/ is quite old rebasing [08:45:47] Why would it use a newer version? [08:46:09] Ah, it merges into master. It didn't use the newer version of upstream, but the newer version the extension was already using [08:46:12] maybe git submodule update --init is not enough [08:46:29] https://git.wikimedia.org/tree/mediawiki%2Fextensions%2FVisualEditor/3ae4e327e2f32776b5b4eb2a2b0bba5ee871e851/lib [08:46:31] Yeah, thats it [08:46:37] ve @ a1c190c [08:47:41] hashar: It seems odd we have to manually run that submodule update command though. You'd think this is basic 1-2-3 for upstream Git plugin and/or Zuul merger. Don't they have the same problem? [08:49:24] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #208: FAILURE in 45 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/208/ [08:51:22] (03PS1) 10Hashar: Fix jjb diff job not comparing parent patchset [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164915 [08:51:35] Krinkle: upstream don't use submodules at all [08:51:49] instead they use pip requirements, the equivalent of npm packages.json [08:51:50] openstack doesn't use submodules anywhere in any of their repos? [08:51:59] right, and release everything [08:52:13] so a downstream repo would just : require somelib<=1.0 [08:52:33] I think they have jobs that tests out patch sets with the tip of all branches though [08:53:02] Krinkle: rebased job yielded https://integration.wikimedia.org/ci/job/mwext-VisualEditor-qunit/11698/consoleFull [08:54:20] (03CR) 10Hashar: "recheck" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 (owner: 10Hashar) [08:54:28] (03CR) 10Hashar: "recheck" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161459 (owner: 10Hashar) [08:55:22] 08:45:52 + git submodule update --init [08:55:23] 08:45:52 Submodule 'lib/ve' () registered for path 'lib/ve' [08:55:23] 08:45:52 + popd [08:55:28] That looks suspicious [08:55:30] yeah [08:55:34] need to reset it somehow [08:55:44] git clean -ff [08:55:48] two f's [08:55:52] 'git reset' command no found :D [08:55:52] we fixed this already [08:56:06] double -ff ? [08:56:34] hashar: https://gerrit.wikimedia.org/r/#/c/163221/ [08:56:54] o [08:56:55] h [08:57:05] those scripts are no more used [08:57:07] this was causing issues a few weeks ago [08:57:10] though multigit.sh might still be [08:57:22] it turns out you bypassed that bug by using zuul-merger, so this patch was obsolete / no-op [08:57:31] but either way, I found the bug in the old system and fixed it [08:58:43] hashar: Where do we run the submodule update --init command? I mean, where does that script live in git? [09:03:56] Krinkle: in the job template I am proposing at https://gerrit.wikimedia.org/r/#/c/163837/10/mediawiki-extensions.yaml [09:04:08] that runs Zuul cloner to fetch VisualEditor / Parsoid [09:04:18] then just git submodule update --init [09:04:30] probably need to reset the submodules [09:05:03] hashar: each 'shell' builder gets a fresh environment with cwd in wordspace [09:05:07] cd is not preserved [09:05:15] workspace* [09:05:48] I found out when I tried to optimise the 'npm-in' macro by doing shell: cd {dir} and then builder -npm, but failed because it was reset in between. [09:05:54] so no pushd or cd - needed [09:06:20] yeah the popd/pushd are unneeded [09:06:38] don't we need that job to run on Trusty as well? [09:07:25] not per se no [09:07:41] in fact, it might fail because slave-scripts has compiled node_modules that were built on Precise [09:07:56] and since -qunit uses global wmfgrunt (not local), it shouldn't be run on precise [09:08:02] same goes for the legacy -jshint job [09:08:21] shoudn't be run on trusty* [09:08:44] it runs in phantomjs, which for our purposes is the same on precise/trusty [09:08:54] okk [09:09:13] cause git on Trusty can be forced to checkout the submodules [09:09:21] cool [09:09:22] something like git submodule update --init --force [09:15:39] (03PS2) 10Krinkle: [WIP] macro/npm: Set CHROME_BIN to chromium-browser [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164906 [09:17:26] (03PS2) 10Hashar: Fix jjb diff job not comparing parent patchset [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164915 [09:19:09] that is becoming crazy [09:21:52] (03Abandoned) 10Hashar: Fix jjb diff job not comparing parent patchset [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164915 (owner: 10Hashar) [09:26:44] (03PS11) 10Hashar: Switch extensions qunit jobs to Zuul cloner [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161459 [09:26:53] (03PS11) 10Hashar: Dedicated VisualEditor qunit job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 [09:27:51] (03PS12) 10Hashar: Dedicated VisualEditor qunit job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 [09:29:51] (03CR) 10Hashar: "Removed pushd/popd." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 (owner: 10Hashar) [09:30:49] (03CR) 10Krinkle: "Experimenting with oojs-core-npm and https://gerrit.wikimedia.org/r/#/c/164921/" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164906 (owner: 10Krinkle) [09:30:53] (03CR) 10Krinkle: [C: 04-2] [WIP] macro/npm: Set CHROME_BIN to chromium-browser [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164906 (owner: 10Krinkle) [09:33:54] Krinkle: I removed the pushd/popd cruft and added a 'git submodule status' [09:33:55] https://gerrit.wikimedia.org/r/#/c/163837/12/mediawiki-extensions.yaml,unified [09:35:36] zeljkof: gooood morning :] [09:35:45] hashar: morning :) [09:36:21] zeljkof: I probably found out why the local browser tests lost connection with firefox from time to time [09:36:41] hashar: I saw something in mail, did not have the time to test [09:36:46] zeljkof: they use the same xvfb display and the headless gem shutdown the display on completion. Race condition! [09:36:52] hashar: ha [09:37:01] did reusing the session help? [09:37:12] or starting separate sessions [09:37:13] ? [09:38:05] zeljkof: Dan reproduced the issue in mediawiki vagrant and proposed a patch at https://gerrit.wikimedia.org/r/#/c/164704/ [09:38:17] I am not quite happy with the patch though :D [09:38:45] I took a quick look but I am not that familiar with the headless gem [09:38:56] hashar: what were you not so happy about? [09:40:34] zeljkof: it overrides headless gems default and lack docs :D [09:40:39] I will comment I guess [09:41:12] hashar: please do [09:41:30] zeljkof: and dan mention the tests might just work in the same xvfb [09:41:51] though i suspect browsers sharing the same display might well have some race conditions [09:42:33] hashar: I have heard a talk about that problem yesterday on a local conference [09:44:10] zeljkof: I need to read headless code again :D [09:44:15] and figure out some sane default [09:44:30] hashar: great, thanks [09:44:43] headless defaults are: [09:44:44] * display port 99 [09:44:44] * auto picking disabled (since display port is set) [09:44:44] * reuses display [09:52:30] 3Wikimedia / 3Continuous integration: Jenkins: Zuul should not run jenkins-bot on changes for refs/meta/* - 10https://bugzilla.wikimedia.org/50389#c7 (10Krinkle) *** Bug 64678 has been marked as a duplicate of this bug. *** [09:52:32] 3Wikimedia / 3Continuous integration: Jenkins: jenkins-bot reports spurious merge error when pushing changes to one of the gerrit config branches - 10https://bugzilla.wikimedia.org/64678#c5 (10Krinkle) 5NEW>3RESO/DUP *** This bug has been marked as a duplicate of bug 50389 *** [09:57:49] hashar: zeljkof: While working on running unit tests on local browsers I ran into /bin/xvdb-run --auto-servernum which can be useful to create a temporary create a window with a server numb >99 (reserved) [09:58:16] alternatively, use a deamon with a server < 99 and set export DISPLAY=':xx' and never quit it (only quit the browsers) [09:59:15] 3Wikimedia / 3Continuous integration: Jenkins: Doxygen job shouldn't have extra "html/" subdirectory and publish error as artefact - 10https://bugzilla.wikimedia.org/71060 (10Krinkle) [10:04:44] (03CR) 10Hashar: [C: 04-1] "Thanks for reproducing and the preliminary patch. On MediaWiki vagrant there is a permanent xvfb on port 99 and that patch would end up k" (032 comments) [selenium] - 10https://gerrit.wikimedia.org/r/164704 (https://bugzilla.wikimedia.org/71602) (owner: 10Dduvall) [10:04:59] zeljkof: found some reply :] [10:05:35] Krinkle: the ruby gem we are using can auto allocate displays making sure they are fresh then shut them down [10:05:41] Krinkle: we will find :] [10:06:52] Yep, that's what xvfb-run does as well. [10:06:56] but for bash [10:07:03] comes with Xvfb on ubuntu [10:07:48] who is flimport ? [10:08:08] Krinkle: shah that is a nice command [10:08:19] Krinkle: turns out headless gem reimplements that auto display assignment :D [10:08:59] Krinkle: it iterates over all displays to find one which is available, and end up just invoking Xvfb :display [10:08:59] https://github.com/leonid-shevtsov/headless/blob/master/lib/headless.rb#L131-L150 [10:09:34] Krinkle: can check whether the new VE-qunit job is fine to you? Change: https://gerrit.wikimedia.org/r/#/c/163837/ JJB diff: https://integration.wikimedia.org/ci/job/integration-jjb-config-diff/1287/console [10:10:48] looks like the headless gem assumes it's the only thing on the server, blindly starts at a number and keeps incrementing [10:11:16] so if other code on the server uses Xvfb's own logic for auto assignment, or if anything on the server assigns a number >99 for anything, it'll clash [10:11:33] similar to server port listeners (eg. port 80 taken everywhere) [10:11:57] though with reuse => false, it iterates till it find an available display port to use [10:12:10] hashar: only for displays it started itself [10:12:29] if I start a display with 100, it'll try to spawn one with 100 eventually and fail I think [10:12:50] wihtout recovring in a good way [10:14:17] it attempts to CliUtil.read_pid(/tmp/.X#{display}-lock) [10:14:47] Which is created itself, right? [10:14:53] or by another user [10:14:58] Ah, seems that part is standardised [10:15:13] the lock file is not part of the gem but of xvdb [10:15:23] so at least it works realiably. [10:15:32] so potentially another process / user would end up having a /tmp/.X100-lock which would prevent the headless gem to start one on 100 [10:15:40] yeah [10:16:10] depends what read_pid() returns, I guess it attempts to open the file [10:16:14] wich might not be readable [10:17:50] heading lunch with wife and kid [10:18:16] Krinkle: please please give a look at https://gerrit.wikimedia.org/r/#/c/163837/ :] I would like it to be merged soonish [10:18:44] will look later after nap and lunch [10:33:03] Krinkle: xvfb-run has a similar logic, it iterates over /tmp/.X###.lock files [11:18:29] * zeljkof is out of lunch [11:25:28] bicycling to coworking place [12:12:12] Any idea why me or santhosh are not able to ssh, https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000582.eqiad.wmflabs [12:17:32] Nevermind. [12:18:11] (03CR) 10Hashar: [C: 032] Remove VisualEditor-qunit [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/164907 (owner: 10Hashar) [12:18:21] (03Merged) 10jenkins-bot: Remove VisualEditor-qunit [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/164907 (owner: 10Hashar) [12:20:14] re [12:33:34] (03CR) 10Hashar: [C: 031] [WIP] macro/npm: Set CHROME_BIN to chromium-browser (031 comment) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164906 (owner: 10Krinkle) [12:42:04] (03PS3) 10Tobias Gritschacher: Make repository host parameterizable [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164332 [12:42:31] (03CR) 10jenkins-bot: [V: 04-1] Make repository host parameterizable [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164332 (owner: 10Tobias Gritschacher) [12:43:59] (03PS4) 10Tobias Gritschacher: Make repository host parameterizable [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164332 [12:44:25] (03CR) 10jenkins-bot: [V: 04-1] Make repository host parameterizable [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164332 (owner: 10Tobias Gritschacher) [12:45:56] hashar: any idea why https://gerrit.wikimedia.org/r/#/c/164332/ does not work? [12:46:25] the default for repository_host is defined in the browsertests-defaults [12:46:43] which is the default for the job-templates [12:46:55] (03PS1) 10Hashar: mw-run-phpunit*: let us override PHPUNIT_DIR [integration/jenkins] - 10https://gerrit.wikimedia.org/r/164953 [12:46:56] which apply to the jobs defined in browsertests.yaml [12:47:15] but for some reason it still complains about that missing parameter [12:48:49] (03CR) 10Hashar: "That will let us run some integration tests when a change is proposed to integration/phpunit such as https://gerrit.wikimedia.org/r/#/c/16" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/164953 (owner: 10Hashar) [12:48:59] (03CR) 10Hashar: [C: 032] mw-run-phpunit*: let us override PHPUNIT_DIR [integration/jenkins] - 10https://gerrit.wikimedia.org/r/164953 (owner: 10Hashar) [12:49:01] (03Merged) 10jenkins-bot: mw-run-phpunit*: let us override PHPUNIT_DIR [integration/jenkins] - 10https://gerrit.wikimedia.org/r/164953 (owner: 10Hashar) [12:49:13] Tobi_WMDE_SWE: hold on deploying :) [12:49:36] hashar: ok [12:50:05] done :D [12:50:47] ah that is our discussion from last thursday [12:51:26] exactly [12:51:38] looking at https://gerrit.wikimedia.org/r/#/c/164332/4/job-templates-browsertests.yaml [12:51:49] seems variable set in the default are not applied :/ [12:52:32] hashar: thought after what you showed me last week, it should.. http://paste.openstack.org/show/117785/ [12:54:46] yeah :( [12:54:53] trying to reproduce it [12:55:50] reproduced http://paste.openstack.org/show/118969/ :D [12:56:07] the job template is given parameters: Given: {'': '', 'jobs': ['{name}-build'], 'name': 'myproject'} [12:56:16] which does not have the my variable one :D [12:56:53] that is because the variable is used in the default [12:57:07] and the default doesn't pass its own variables :/ [12:58:30] hashar: Î tried moving the variable to the job-templates and that did not work either.. [12:59:18] :( [12:59:56] hold on [13:00:09] my last test case is wrong, the default was not applied on the project [13:01:27] holy shit [13:02:21] what happend? [13:02:30] http://paste.openstack.org/show/118970/ [13:02:52] had to apply defaults: 'browsertests' on the project [13:03:02] in addition to applying it to the job template [13:03:20] what JJB is doing is that it read the project, load the variable then realize the default [13:03:24] hashar: why? [13:03:28] but the defaults does not apply variables to itself [13:03:36] hm [13:03:45] so if you apply the defaults at the project level, the variables are loaded and passed to the defaults: browertests [13:03:48] that is a bit cumbersome [13:04:07] ok [13:04:10] really [13:04:18] that is cumbersome [13:04:29] so either: we implement the feature in JJB [13:04:39] or add to each browser test project: defaults: browsertests [13:04:53] hashar: i would prefer the latter [13:05:07] though implementing it in jjb would be cleaner.. ;) [13:05:17] but i don't want to do that. [13:05:21] :-D [13:05:21] :) [13:05:25] so copy paste! [13:05:32] \o/ [13:05:41] hashar: should I do it in my change? [13:05:42] one day I will look at refactoring them [13:05:55] yeah do it in the same change [13:06:05] since that is the first that actually need that "feature" [13:06:15] hashar: ok [13:06:16] thx! [13:06:27] need to get used to this jjb more [13:19:08] (03PS1) 10Hashar: Zuul cloner now defaults --branch to $ZUUL_BRANCH [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164958 [13:22:29] (03PS5) 10Tobias Gritschacher: Make repository host parameterizable [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164332 [13:26:45] hashar: \o/ https://gerrit.wikimedia.org/r/#/c/164332/ [13:27:21] :-) [13:28:40] (03PS1) 10Hashar: Test integration/phpunit against mw core branches [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164959 [13:30:06] (03CR) 10jenkins-bot: [V: 04-1] Test integration/phpunit against mw core branches [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164959 (owner: 10Hashar) [13:30:11] ouch [13:32:39] (03PS2) 10Hashar: Test integration/phpunit against mw core branches [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164959 [13:35:23] (03PS1) 10Hashar: Test integration/phpunit against mw core branches [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/164960 [13:38:15] (03PS2) 10Hashar: Test integration/phpunit against mw core branches [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/164960 [13:44:56] (03PS3) 10Hashar: Test integration/phpunit against mw core branches [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/164960 [13:46:00] (03CR) 10Hashar: [C: 032] Test integration/phpunit against mw core branches [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/164960 (owner: 10Hashar) [13:46:09] (03Merged) 10jenkins-bot: Test integration/phpunit against mw core branches [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/164960 (owner: 10Hashar) [13:47:21] (03PS1) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [integration/phpunit] - 10https://gerrit.wikimedia.org/r/164962 [13:47:30] 3Wikimedia / 3Quality Assurance: Selenium tests sometimes fail with strange error messages (tracking) - 10https://bugzilla.wikimedia.org/60338 (10Željko Filipin) [13:47:32] 3Wikimedia / 3Quality Assurance: browser tests failing with "getaddrinfo: Name or service not known (SocketError)" - 10https://bugzilla.wikimedia.org/68125 (10Željko Filipin) [13:48:32] 3Wikimedia / 3Quality Assurance: Investigate how often browser tests fail because of Jenkins performance problems (tracking) - 10https://bugzilla.wikimedia.org/60338 (10Željko Filipin) [13:48:32] 3Wikimedia / 3Quality Assurance: browser tests failing with "getaddrinfo: Name or service not known (SocketError)" - 10https://bugzilla.wikimedia.org/68125 (10Željko Filipin) [13:55:44] hashar: what means "(pending—All nodes of label ‘ contintLabsSlave && UbuntuPrecise ’ are offline)" [13:55:55] ouch [13:56:02] that there is no slave available [13:56:21] in the case of browsertests, the message is misleading I believe [13:56:36] hmm [13:56:41] Tobi_WMDE_SWE: jenkins is fubar :( [13:56:50] hashar: it does only happen when I add a new project [13:57:08] it is not when I build a ob that is already there [13:57:13] *job [13:57:24] (03CR) 10Hashar: "recheck" [integration/phpunit] - 10https://gerrit.wikimedia.org/r/164683 (owner: 10BryanDavis) [13:57:35] hashar: see the spaces at the beginning ant the end? ‘ contintLabsSlave && UbuntuPrecise ’ [13:58:01] hashar: could be that my version of jjb is doing something strange [13:58:07] yeah [13:58:09] it is wrong at https://integration.wikimedia.org/ci/job/browsertests-WikidataTest-wikidata.beta.wmflabs.org-linux-firefox-sauce/configure [13:58:16] Label Expression: "\n contintLabsSlave && UbuntuPrecise\n " [13:58:21] literally, with double quotes! :d [13:58:23] that is wrong [13:58:31] There’s no slave/cloud that matches this assignment. Did you mean ‘contintLabsSlave’ instead of ‘ [13:58:31] contintLabsSlave && UbuntuPrecise [13:58:31] ’? [13:58:32] hehe [13:58:35] good catch [13:58:45] hashar: hm, that definitely has to do with my version of jjb [13:58:54] might be [13:59:03] attempt to update with the last version from integration/jenkins-job-builder [13:59:12] hashar: will do [13:59:38] or some yaml definition is horribly wrong :-/ [14:19:33] (03CR) 10Hashar: "recheck" [integration/phpunit] - 10https://gerrit.wikimedia.org/r/151252 (owner: 10EBernhardson) [14:19:37] (03CR) 10jenkins-bot: [V: 04-1] Update jenkins phpunit to 4.1.4 [integration/phpunit] - 10https://gerrit.wikimedia.org/r/151252 (owner: 10EBernhardson) [14:20:10] (03CR) 10Hashar: [C: 031] "I was too lazy to manually test the proposed patchset against each MediaWiki version we support so I made that an integration test. The ch" [integration/phpunit] - 10https://gerrit.wikimedia.org/r/164683 (owner: 10BryanDavis) [14:21:34] (03CR) 10Hashar: "To help detects potential problems when upgrading integration/phpunit to a later version, I have crafted a set of integration jobs that ru" [integration/phpunit] - 10https://gerrit.wikimedia.org/r/151252 (owner: 10EBernhardson) [14:22:41] (03CR) 10Hashar: "Note: Erik Bernhardson proposed a bump to PHPUnit 4.1.4 which I -1ed because some release branch would not support it ( https://gerrit.wik" [integration/phpunit] - 10https://gerrit.wikimedia.org/r/164683 (owner: 10BryanDavis) [14:24:09] (03Abandoned) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [integration/phpunit] - 10https://gerrit.wikimedia.org/r/164962 (owner: 10Hashar) [14:29:58] Tobi_WMDE_SWE: your repo change is fine though it has a side effect. browsertests-Wikidata-wikidata.beta.wmflabs.org-linux-firefox-sauce is slightly changed [14:31:36] hashar: ? [14:32:19] (03CR) 10Hashar: "Note that the diff in https://integration.wikimedia.org/ci/job/integration-jjb-config-diff/1291/console shows that Wikidata-wikidata.beta." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164332 (owner: 10Tobias Gritschacher) [14:32:42] apparently Wikidata-wikidata.beta.wmflabs.org-linux-firefox-sauce was not using our set of defaults [14:32:53] https://integration.wikimedia.org/ci/job/integration-jjb-config-diff/1291/console shows a bunch of differences [14:34:41] hashar: hm, yeah, could leave the defaults out for the wikidata job, since it is not a project yet [14:35:09] Tobi_WMDE_SWE: that is probably not an issue [14:35:15] hashar: there are other things I need to sort out before I can use the full template [14:37:16] it might need to be made a job-template [14:39:23] hashar: let's see. [14:39:54] I finally managed to install jjb from integration/jenkins-job-builder [14:40:33] hashar: but look waht it does with the configuration: https://integration.wikimedia.org/ci/view/BrowserTests/view/Wikidata/job/browsertests-WikidataTest-wikidata.beta.wmflabs.org-linux-firefox-sauce/jobConfigHistory/showDiffFiles?timestamp1=2014-10-06_14-26-49×tamp2=2014-10-06_14-39-12 [14:40:34] great [14:40:44] ah yeah [14:40:49] so, it formats the xml nicer [14:40:54] so JJB craft the XML using its own order [14:41:01] but that causes the problem with the nodes [14:41:01] which is different to whatever Jenkins ends up building [14:41:34] hashar: so it works with the left side, but does not with the right side [14:41:51] the right side being the one generated with JJB right ? [14:42:09] hashar: the left side too. but with an older version of jjb [14:42:11] seems there is too many blank lines being inserted :-/ [14:43:03] which sounds exactly like the issue you had with the labels having blank lines before/after [14:43:04] grmblbl [14:44:08] hashar: but line 18-20 on the right looks good, doesn't it? [14:44:22] iti sn't [14:44:26] different from line 53 on the left though [14:44:30] 3Wikimedia / 3Quality Assurance: browser tests failing with "getaddrinfo: Name or service not known (SocketError)" - 10https://bugzilla.wikimedia.org/68125 (10Željko Filipin) [14:44:32] 3Wikimedia / 3Quality Assurance: Investigate how often browser tests fail because of Jenkins performance problems (tracking) - 10https://bugzilla.wikimedia.org/60338 (10Željko Filipin) [14:44:43] the XML element assigned node contains: \n contintLabsSlave && UbuntuPrecise\n [14:44:50] all those blanks should not be there [14:45:10] hashar: but it works for you? [14:45:13] so either your jjb generate some bad configurationn [14:45:21] so, could it be I'm using jjb on windows? [14:45:23] or your jenkins has some issue [14:45:30] yeah that might be [14:45:35] since end of lines are slightly different [14:45:58] or it is a problem with the python version [14:46:00] though you ran the left side with windows as well haven't you? [14:46:06] hashar: yes [14:46:24] but with a different version of jjb (that doesn't apply the default variables..) [14:46:38] so I can have either the one or the other.. but not both [14:48:52] gotta craft a dummy job and see what your jjb outputs [14:49:05] or figure out the different between your two installs [14:51:20] hashar: yeah, seems so [14:51:42] also might be whatever python xml module which would cause issues [14:52:12] # Python 2.6's minidom toprettyxml produces broken output by adding extraneous [14:52:12] # whitespace around data. This patches the broken implementation with one taken [14:52:12] # from 2.7 [14:52:15] from JJB source code [14:52:31] which python version are you using? [14:52:32] I have 2.7 [14:52:36] ah [14:53:06] 2.7.2 [14:53:11] that should be fine, right? [14:53:15] 3Wikimedia / 3Quality Assurance: Compare screen shots from current and previous test run - 10https://bugzilla.wikimedia.org/63107#c1 (10Željko Filipin) It would also be interesting to compare screenshots taken in different browsers in the same test run. [14:54:09] ahh [14:54:12] Tobi_WMDE_SWE: yeah that might be the issue [14:54:17] https://review.openstack.org/#/c/93198/1/jenkins_jobs/builder.py [14:54:24] that is the patch that fixed it for python 2.6 [14:54:38] it uses to have a condition python <= 2.7.3 [14:54:51] which got changed to < 2.7.0 [14:54:57] so apparently anything in between is bugged as a result [14:55:14] hashar: hm, could try to update to 2.7.8 [14:55:22] if you don't mind [14:55:23] :-] [14:59:03] hashar: wait, should we all upgrade python? [14:59:15] looks like I have Python 2.7.5 [15:01:13] zeljkof: think it's only between 2.7.0 and 2.7.2 [15:01:24] Tobi_WMDE_SWE: ok, thanks [15:01:47] and you managed to have the exactly wrong version? :) [15:02:27] zeljkof: yeah. seems so. and I managed to have the right version of jjb for the wrong version of pthon. until today since I've updated jjb. [15:02:39] I am preparing a patch for upstream [15:02:51] but not sure yet if that solves my problem [15:03:04] you can try with python 2.6 :-D [15:04:25] hashar: haha! [15:04:30] seems that was it: https://integration.wikimedia.org/ci/view/BrowserTests/view/Wikidata/job/browsertests-WikidataTest-wikidata.beta.wmflabs.org-linux-firefox-sauce/jobConfigHistory/showDiffFiles?timestamp1=2014-10-06_14-39-12×tamp2=2014-10-06_15-03-57 [15:04:34] \O/ [15:04:39] -_- [15:04:48] for python 2.6.x JJB hijack the XML writer [15:06:00] Tobi_WMDE_SWE: if you feel brave you can apply a patch I proposed https://review.openstack.org/#/c/126318/ [15:06:05] and retry with python 2.7.2 :] [15:06:11] hashar: haha [15:06:21] git fetch https://review.openstack.org/openstack-infra/jenkins-job-builder refs/changes/18/126318/1 && git cherry-pick FETCH_HEAD [15:06:24] should give you the patch [15:06:36] I'm not trying out what my system does to me when I downgrade python now [15:07:02] not even sure whether gerrit still works as it uses python too [15:07:21] meh, I mean git-review [15:36:38] bd808: hello! :] Got a few things for you! [15:37:13] bd808: mediawiki/vendor is still blocked on the qunit job for VisualEditor . Timo and I worked a bit on it this morning but jet lag sent him to nap [15:37:17] hashar: hello to you too. [15:37:27] I think I got all the other jobs covered though [15:37:42] that sounds great. [15:38:01] I have been too lazy to *manually* test your proposal to Update phpunit 3.7.37 [15:38:03] I saw that you poked at that phpunit upgrade patch [15:38:18] I wanted to make sure it works on all branch of mw/core we support. So I created a few integration jobs for it [15:38:23] lengthy explanation on https://gerrit.wikimedia.org/r/#/c/164683/ [15:38:30] the change seems good to me :] [15:38:41] the new job tested what I would have tested manually anyway [15:38:58] that might break other repositories though ( Wikidata and extensions ) [15:39:32] It would be nice to move up to phpunit 4.x but I can see how we may have things that keep that from happening [15:40:48] Erik Bernhardson proposed a patch for 4.x [15:40:53] but it comes with a .phar apparently [15:41:07] and REL1_22 most definitely would not find phpunit :/ [15:41:20] the change is https://gerrit.wikimedia.org/r/#/c/151252/ [15:41:52] now that we have some integration test using the tip of some mw/core branches, I guess the patch can be moved again [15:42:05] and patches sent to release branches to make them work with both phpunit versions [15:42:48] hmm, is anyone else able to ssh into deployment-rsync02.eqiad.wmflabs? [15:43:02] it says 'temp failure in name resolution' [15:43:06] GroggyPanda: I deleted that instance [15:43:20] aaah, I see [15:43:21] GroggyPanda: It only existed for about 3 hours [15:43:25] ok [15:43:28] let me fix graphite then [15:43:36] I need to move us to shinken quicker so we don' have to do this manually [15:43:41] decomm is tricky in wikitech :( [15:43:47] yup [15:46:04] hashar: I'm sure it would be more complexity to manage, but I wonder if we shouldn't be thinking about versioning tools and jobs for maintenance branches. Maybe even to the point of having separate jenkins clusters for each release branch that is considered "alive". [15:46:08] bd808: I cleeard up the data, it should fix the icinga alert now [15:46:12] $wgHooks['OpenstackManagerInstanceDeleted'] = function () { /** fix graphite **/ }  [15:46:33] + fix salt + fix trebuchet + ... [15:47:49] bd808: for PHPUnit we could have the jobs to ask zuul-cloner to clone integration/phpunit . It would checkout the REL branch if it exists [15:48:07] * bd808 nods [15:48:10] bd808: so we could indeed freeze our PHPUnit to a specific version for REL1_22 by just creating a REL1_22 branch in there [15:48:12] that is doable [15:48:18] gotta refactor a few things again hehe [15:48:45] I think we are going to have php runtime version issues soon too. hhvm vs php5.5 vs php5.3 [15:48:56] yeah that one I have no idea [15:49:36] the Zend PHP we have is whatever Ubuntu provides, so all php related jobs are (should?) forced to run on Precise [15:49:44] for hhvm, we would need to duplicate the jobs I guess [15:51:25] It's a complex problem. If you try to do full matrix on all jobs you will need a lot more slaves. [15:52:05] I keep wondering if we shouldn't just be donating hardware to travisci and letting them worry about most of it. [15:52:39] I like the way that setting up jobs in travis works so much better than Jenkins. [15:56:19] (03CR) 10Hashar: "I forgot, to deploy this change:" [integration/phpunit] - 10https://gerrit.wikimedia.org/r/164683 (owner: 10BryanDavis) [15:56:38] bd808: ditto [15:59:13] bd808: that is been floating for quite sometime [15:59:51] bd808: will most probably end up writing a proposal to migrate to travis indeed [16:00:01] Krinkle is pushing for it, too [16:00:01] and thus never end up isolating our tests :] [16:00:42] yeah half of my time is probably not enough to set up an isolated infrastructure that would reuse Travis command line [16:01:24] One really nice thing about testing via Travis (or a travis like system) would be that the job specs live in the branch. So we can add say hhvm matrix testing and then have it stick with that branch where ever it goes. [16:03:21] And internal Travis install/clone would be ok too. I don't know how sad we might be is we had to wait on Travis slaves to merge things for swat deploys. [16:03:27] s/is/if/ [16:04:01] But job specs in the branch seems like the right direction to move [16:04:02] Timo does that for javascript related test [16:04:11] the -npm jobs just 'npm test' [16:04:25] up to the developers to define the tasks they want to execute. So that is a step forward [16:05:02] Tests at $DAYJOB-1 were `phing test-quick` and `phing test-nightly` [16:06:48] hehe [16:08:09] anyway commuting back home [16:08:29] We had shared phing build scripts across projects that could be tweaked to add/remove things that a particular project needed. I actually wrote the initial upstream code to allow for config inheritance in phing. [16:09:00] oh Make for PHP hehe [16:09:05] It was like most of phing basically a copy-paste from ant [16:09:26] Yeah phing is a port of the apache ant build tool to php [16:09:38] oh interesting [16:09:44] well we could adjust Zuul to trigger jobs at Travis-ci.org [16:09:56] giving it the list of repos to clone and the ref we want [16:10:15] it most probably has some kind of notification API to pass everything we would need [16:10:30] could let us drop Jenkins :] [16:10:33] and maybe even Gearman hehe [16:10:35] hashar: btw, while Krinkle|detached was here I asked him to write up an outline of his thoughts for how he wants to use Zuul (etc) in the future [16:11:31] * hashar greg-g: sounds good to me [16:11:43] cause my Gerrit dashboard and bug queue are over filled :-/ [16:12:20] rushing home, I am late [16:47:13] (03CR) 10Krinkle: [WIP] macro/npm: Set CHROME_BIN to chromium-browser (031 comment) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164906 (owner: 10Krinkle) [16:51:23] (03PS1) 10Krinkle: Add npm-set-env [integration/jenkins] - 10https://gerrit.wikimedia.org/r/165010 [16:51:43] (03PS3) 10Krinkle: [WIP] macro/npm: Set CHROME_BIN to chromium-browser [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164906 [16:51:51] (03PS2) 10Krinkle: Add npm-set-env [integration/jenkins] - 10https://gerrit.wikimedia.org/r/165010 [16:52:08] (03CR) 10Krinkle: [C: 032] Add npm-set-env [integration/jenkins] - 10https://gerrit.wikimedia.org/r/165010 (owner: 10Krinkle) [16:52:10] (03Merged) 10jenkins-bot: Add npm-set-env [integration/jenkins] - 10https://gerrit.wikimedia.org/r/165010 (owner: 10Krinkle) [16:54:43] (03PS4) 10Krinkle: macro/npm: Set CHROME_BIN to chromium-browser [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164906 [17:01:52] bd808: fyi: Travis only supports github.com, even their Pro product for non-opensource repos is GitHub-only. [17:02:29] Krinkle: Ah. I forgot about that bit. [17:02:35] and primarily post-merge for a linear branch. Except for pull requests, that's the only way it can run on a non-sequential branch [17:13:09] (03CR) 10Krinkle: [C: 032] "Published oojs-core-npm, oojs-ui-npm, unicodejs-npm, VisualEditor-npm. Will publish others as needed." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164906 (owner: 10Krinkle) [17:16:26] (03Merged) 10jenkins-bot: macro/npm: Set CHROME_BIN to chromium-browser [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164906 (owner: 10Krinkle) [17:23:23] (03PS1) 10Krinkle: misc: Re-order job definitions [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165020 [17:39:41] (03CR) 10Krinkle: [C: 032] "No effective changes." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165020 (owner: 10Krinkle) [17:42:45] (03Merged) 10jenkins-bot: misc: Re-order job definitions [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165020 (owner: 10Krinkle) [18:16:41] Project browsertests-TwnMainPage-sandbox.translatewiki.net-linux-firefox-sauce build #170: FAILURE in 12 min: https://integration.wikimedia.org/ci/job/browsertests-TwnMainPage-sandbox.translatewiki.net-linux-firefox-sauce/170/ [18:46:31] RECOVERY - CI: Puppet failure events on labmon1001 is OK: OK: All targets OK [19:23:16] Yippee, build fixed! [19:23:16] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #272: FIXED in 42 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/272/ [19:35:06] blohhh [19:42:30] 3Wikimedia / 3Continuous integration: Jenkins: browser test host performance issue for timed builds - 10https://bugzilla.wikimedia.org/66449#c3 (10Antoine "hashar" Musso) For an unrelated change, I have eventually found some time to look at Xvfb and had a look at the headless ruby gem. In short: we have a ra... [19:42:32] 3Wikimedia / 3Quality Assurance: mediawiki_selenium always use the same default xvfb display 99 - 10https://bugzilla.wikimedia.org/71602 (10Antoine "hashar" Musso) [19:42:56] greg-g: hello! The firefox browser tests failing turns out to be a race condition between jobs. They kill the X virtual frame buffer at end of execution :-] [19:43:09] :/ [19:43:31] well i found it and dan leet ruby skills already crafted a patch! [19:44:05] hashar: working on a follow-up atm [19:44:17] greg-g: and we had all the browser test job on a given instance to run on the same display port which might well cause some other race conditions [19:44:27] marxarelli: yeah I wrote some review early today in a hurry [19:44:38] marxarelli: zeljkof is happy with the idea. Hope my comment made sense [19:45:21] marxarelli: in short, I would like to set env variable just like if I was invoking headless.new() (i.e. still rely on whatever headless has for defaults) [19:45:31] but I can only read ruby :-( [19:45:47] hashar: yeah, it makes sense for the most part (though my mwv instance doesn't have xvfb running by default which is weird) [19:46:09] hashar: the defaults are retained with the patch afaict [19:46:29] but i changed it to only set the options if the respective env vars are present [19:46:44] Yippee, build fixed! [19:46:45] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #37: FIXED in 38 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/37/ [19:47:05] marxarelli: this way tests can be made in mediawiki vagrant to reuse 99 and never kill it [19:47:16] hashar: along the way i realized that the default behavior of headless could be better with respect to destroying upon exit by default, so i'm working on an upstream pull request [19:47:22] and on jenkins we can allocate ports in sequence and shut them down [19:47:37] great [19:47:54] there is another potential issue with upstream CliUtils.read_pid() method [19:48:02] i.e. if headless detects that xvfb is already running AND it's given the reuse option, it shouldn't kill the process at exit by default [19:48:29] it look at /tmp/.X42-local or something and attempt to read() the file. Though it might well not be readable if opened by another user which I think would make it consider that display port 42 is available [19:48:44] ah yeah that is good one [19:48:52] makes more sense [19:49:17] I am probablynot there for long, but will catchup tomorrow with Zeljkof [19:54:16] marxarelli: and if you feel brave, I guess mediawiki_selenium could use a few tests :] [19:55:00] hashar: yes! i'm implementing tests with the new environment abstraction layer [19:55:17] hashar: https://gerrit.wikimedia.org/r/#/c/159644/ [19:55:33] oh [19:55:51] how one runs them? [19:55:54] bundle install [19:56:00] bundle exec rspec [19:56:00] ? [19:56:04] bundle exec rspec [19:56:46] * hashar feels like an experimented rubyist [19:57:49] why the hell doesn't it run install for me automatically :D [20:01:08] hashar: it's bundler. don't ask questions, just be glad it works at all :) [20:01:16] :D [20:02:37] hashar: it seems that headless is one of those libraries where you wish you'd never looked under the hood at all. thar be demons! [20:02:39] (03PS1) 10Hashar: Create mediawiki-selenium-bundle-rspec [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165083 [20:03:04] marxarelli: maybe we can dish it out entirely and implement our own ? :D [20:03:22] though it has a few screenshoting utilities we might be relying on [20:04:19] hashar: better to just submit major pull requests i think. at least there are unit tests so one can refactor with confidence [20:05:09] (03PS1) 10Hashar: Experimental mediawiki-selenium-bundle-rspec [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/165084 [20:05:57] marxarelli: sounds good [20:07:04] hashar: one my favorite benefits of unit testing: gut the whole thing with confidence! :) [20:07:58] (03CR) 10Hashar: [C: 032] "Triggering the experimental job defined by https://gerrit.wikimedia.org/r/165083" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/165084 (owner: 10Hashar) [20:08:10] (03Merged) 10jenkins-bot: Experimental mediawiki-selenium-bundle-rspec [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/165084 (owner: 10Hashar) [20:09:26] (03CR) 10Hashar: "recheck" [selenium] - 10https://gerrit.wikimedia.org/r/159644 (owner: 10Dduvall) [20:09:29] (03CR) 10jenkins-bot: [V: 04-1] WIP Environment abstraction layer [selenium] - 10https://gerrit.wikimedia.org/r/159644 (owner: 10Dduvall) [20:10:09] marxarelli: I crafted a Jenkins job to run rspec but the change need a rebase :/ [20:10:31] and yeah unit testing with full coverage and well modularized code is nice for refactoring [20:11:56] (03CR) 10Hashar: "I have added it in Zuul experimental pipeline so we can test it on mediawiki_selenium change https://gerrit.wikimedia.org/r/#/c/159644/ b" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165083 (owner: 10Hashar) [20:17:12] Project browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce build #229: FAILURE in 1 hr 9 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce/229/ [20:21:33] I am off to bed, see you tomorrow! [20:25:35] marxarelli: the headless gem has been forked heavily at https://github.com/pgeraghty/headless/ [20:26:46] I hate people forking and not submitting back to upstream project :( [20:27:50] * hashar waves [20:28:29] it's the github failure, they promote the word 'fork' too much :/ [20:28:38] forks were an action of last resort before [20:28:42] * greg-g talks to himself [20:31:03] (03PS3) 10Dduvall: Additional headless environment variables [selenium] - 10https://gerrit.wikimedia.org/r/164704 (https://bugzilla.wikimedia.org/71602) [20:32:49] greg-g: werd. i've resorted to creating pull requests for some of my own projects after discovering forks that contain some really great improvements [20:35:11] Yippee, build fixed! [20:35:11] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #245: FIXED in 38 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/245/ [20:35:13] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #49: FAILURE in 2.3 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/49/ [20:35:41] marxarelli: oh man, wow [20:49:01] (03CR) 10Dduvall: "With the latest patch set, I've updated the documentation and changed the implementation to only pass each option if the respective enviro" [selenium] - 10https://gerrit.wikimedia.org/r/164704 (https://bugzilla.wikimedia.org/71602) (owner: 10Dduvall) [20:51:24] (03CR) 10Dduvall: "Oh Gerrit... (my inline comments aren't submitting):" [selenium] - 10https://gerrit.wikimedia.org/r/164704 (https://bugzilla.wikimedia.org/71602) (owner: 10Dduvall) [21:37:15] 3Wikimedia / 3Quality Assurance: QA: implement "login to " to speed up browser tests - 10https://bugzilla.wikimedia.org/71635#c2 (10Dan Duvall) (In reply to spage from comment #1) > (In reply to spage from comment #0) > > Is using > > on(APIPage).client.log_in(ENV["MEDIAWIKI_USER"], ENV["MEDIAWIKI_P... [21:50:18] 3Wikimedia Labs / 3deployment-prep (beta): image embedding seems to be broken on beta labs - 10https://bugzilla.wikimedia.org/71210#c9 (10Ryan Kaldari) 5NEW>3RESO/FIX Looks like this was caused by a core change. Fixed with https://gerrit.wikimedia.org/r/#/c/164543/. [22:35:40] Yippee, build fixed! [22:35:41] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #50: FIXED in 7 min 40 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/50/ [23:09:21] PROBLEM - BetaLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_avail.value (10.00%) [23:09:59] bd808: ^ I'm taking a look [23:10:22] bah. Stupid tiny /var partitions [23:12:35] bd808: yeah. [23:13:02] bd808: you should potentially apply the biglogs role, and then stop everything on the machine, force puppet run, then start everything back up again, and do a mount bind of old /var and delete everything.... [23:13:07] bd808: newer instances have a 10G /var, btw [23:13:13] newer as in (since last thursday) [23:13:56] bd808: I killed lotsa log files, should be fine now [23:14:43] anyway, off to try sleep now [23:14:51] I've a meeting in the *afternoon* with mark, hopefully I wake up in time... [23:15:37] YuviPanda: We can't use biglogs because we already mount the secondary lvs for other things [23:15:48] hmmm, for /mnt? [23:15:52] you can have both /srv and biglogs, no? [23:15:59] but I guess the ordering matters... [23:16:36] labs partitioning is a bit of a mess [23:16:39] heh, true [23:16:45] * bd808 shrugs and moves on [23:16:47] at least, from now on, things will be 10G in /var [23:16:55] bd808: hmm, deployment prep stuff is all precise? [23:17:01] * YuviPanda wonders when they'll move to Trusty [23:17:05] I guess the mw servers are HAT [23:17:10] we have 4-5 trusty images [23:17:29] but yea mostly boxes we built at the eqiad migration [23:17:39] hmm, right [23:17:51] * YuviPanda should set aside some time to migrate tools to Trusty... [23:20:21] RECOVERY - BetaLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [23:26:25] alright, that's green [23:26:28] I should go sleep now [23:42:47] Yippee, build fixed! [23:42:47] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #209: FIXED in 3 min 34 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/209/ [23:44:39] Project browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce build #227: FAILURE in 1 hr 8 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce/227/