[03:20:03] (03PS2) 10Krinkle: mwext-VisualEditor-doc-publish: Simplify rsync calls [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/125546 (owner: 10Jforrester) [03:21:58] (03CR) 10Krinkle: [C: 032] "Pushed 'mwext-VisualEditor-doc-publish'." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/125546 (owner: 10Jforrester) [03:23:00] (03Merged) 10jenkins-bot: mwext-VisualEditor-doc-publish: Simplify rsync calls [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/125546 (owner: 10Jforrester) [03:24:15] (03PS3) 10Krinkle: Create a general jsduck test task and use it for OOjs and OOjs UI [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/125541 (owner: 10Jforrester) [07:50:53] hashar: I will be late for the pairing today, I have to do some paperwork for my accountant [07:51:03] I will be ready in 20-30 minutes [07:51:11] can't you postpone the paperwork ? :D [07:51:21] I have meetings all day :( [07:51:25] hehe [07:51:40] literarly [07:51:41] I got a bunch of python to review, so it is fine [07:51:43] take your time! [07:51:50] thanks [07:52:05] it should not take more than 20-30 minutes [08:37:21] hashar: I am ready [08:37:34] thanks for waiting [08:37:41] I am joining the hangout [08:41:32] ah [08:41:41] zeljkof: joining [09:53:09] Project CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox build #1: ABORTED in 54 sec: https://integration.wikimedia.org/ci/job/CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox/1/ [09:53:35] :-] [09:56:53] Project CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox build #2: FAILURE in 1 min 4 sec: https://integration.wikimedia.org/ci/job/CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox/2/ [09:59:01] (03PS1) 10Hashar: Tie browsertests jobs to wmflabs slaves [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126943 [10:00:21] Project CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox build #3: STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox/3/ [10:00:43] (03PS1) 10Hashar: Remove ruby 2.1.1 when running on wmflabs [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126945 [10:05:06] (03PS1) 10Hashar: Install gems in a shared dir [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126946 [10:05:24] hashar: had any chance to look at the compoer patches yet? :) [10:05:42] Project CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox build #4: ABORTED in 2 min 12 sec: https://integration.wikimedia.org/ci/job/CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox/4/ [10:06:28] (03PS1) 10Hashar: Make bundle install verbose! [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126947 [10:07:25] (03PS2) 10Hashar: Make bundle install verbose! [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126947 [10:10:10] Project CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox build #5: STILL FAILING in 3 min 28 sec: https://integration.wikimedia.org/ci/job/CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox/5/ [10:19:06] (03PS1) 10Hashar: Explicitly use gem version 1.9.3 [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126948 [10:22:08] Project CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox build #6: STILL FAILING in 2 min 51 sec: https://integration.wikimedia.org/ci/job/CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox/6/ [10:31:22] Project CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox build #7: ABORTED in 1 min 39 sec: https://integration.wikimedia.org/ci/job/CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox/7/ [10:31:44] zeljkof: https://integration.wikimedia.org/ci/job/CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox/8/console :-] [10:32:09] ie fixed! [10:33:16] (03PS2) 10Hashar: Use gem 1.9.3 and the installed bundler version [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126948 [10:36:00] Project CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox build #8: STILL FAILING in 4 min 26 sec: https://integration.wikimedia.org/ci/job/CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox/8/ [10:37:29] (03PS2) 10Hashar: Tie browsertests jobs to wmflabs slaves [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126943 [10:37:32] (03PS3) 10Hashar: Use gem 1.9.3 and the installed bundler version [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126948 [10:37:35] (03PS2) 10Hashar: Install gems in a shared dir [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126946 [10:37:38] (03PS3) 10Hashar: Make bundle install verbose! [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126947 [10:37:41] (03PS2) 10Hashar: Remove ruby 2.1.1 when running on wmflabs [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126945 [10:39:32] hashar: great :) [10:40:53] zeljkof: I will let you +2 them :] [10:41:14] hashar: will do in a few minutes [10:41:40] meeting with Tobi is cancelled so I have another hour or so [10:43:36] notice: /Stage[main]/Contint::Browsertests/Package[ruby-bundler]/ensure: removed [10:43:42] the old bundle is gone!:-] [10:44:46] :) [10:50:14] (03PS1) 10Hashar: Make the ircbot publisher a macro [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126955 [10:52:22] (03CR) 10Hashar: "Should be a noop. The integration-jjb-config-diff job is bugged, it does the diff against the tip of the branch and not against the previo" [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126955 (owner: 10Hashar) [11:06:23] (03PS4) 10Zfilipin: Use gem 1.9.3 and the installed bundler version [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126948 (owner: 10Hashar) [11:07:02] (03CR) 10Zfilipin: [C: 032] Use gem 1.9.3 and the installed bundler version [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126948 (owner: 10Hashar) [11:07:05] (03CR) 10jenkins-bot: [V: 04-1] Use gem 1.9.3 and the installed bundler version [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126948 (owner: 10Hashar) [11:07:09] zeljkof: I broke the job :-] [11:07:21] hashar: which job? [11:07:33] the one that validates JJB configuration files [11:07:48] I see [11:07:58] the diff job failed [11:08:46] hashar: should I wait before merging the rest of the commits? [11:12:57] zeljkof: yeah fixing the job right now [11:13:07] hashar: great, ping me when done [11:14:23] zeljkof: should be fixed now :-] [11:14:30] hashar: will try [11:15:46] (03PS3) 10Zfilipin: Install gems in a shared dir [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126946 (owner: 10Hashar) [11:15:49] (03PS1) 10Hashar: JJB conf diff now done against parent commit [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/126959 [11:16:14] (03CR) 10Zfilipin: [C: 032] Install gems in a shared dir [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126946 (owner: 10Hashar) [11:16:39] hashar: works fine [11:17:11] gotta +2 the parents now :] [11:17:44] if you ever want to +2 again a change, you can remove yourself from the review list and revote CR+2 [11:17:59] (03PS3) 10Zfilipin: Tie browsertests jobs to wmflabs slaves [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126943 (owner: 10Hashar) [11:18:02] (03CR) 10Hashar: [C: 032] JJB conf diff now done against parent commit [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/126959 (owner: 10Hashar) [11:18:02] hashar: nice trick [11:18:33] (03PS4) 10Zfilipin: Make bundle install verbose! [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126947 (owner: 10Hashar) [11:18:36] (03Merged) 10jenkins-bot: JJB conf diff now done against parent commit [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/126959 (owner: 10Hashar) [11:18:45] (03PS3) 10Zfilipin: Remove ruby 2.1.1 when running on wmflabs [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126945 (owner: 10Hashar) [11:19:30] (03CR) 10Zfilipin: [C: 032] Make bundle install verbose! [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126947 (owner: 10Hashar) [11:19:58] (03CR) 10Zfilipin: [C: 032] Remove ruby 2.1.1 when running on wmflabs [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126945 (owner: 10Hashar) [11:20:24] zeljkof: some of the patches needs a rebase now :-] [11:20:33] hashar: argh :) [11:20:35] will do [11:20:37] anyway [11:20:39] just +2 https://gerrit.wikimedia.org/r/#/c/126943/2 [11:20:50] then press rebase and re +2 the child patches [11:21:39] (03CR) 10Zfilipin: [C: 032] Tie browsertests jobs to wmflabs slaves [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126943 (owner: 10Hashar) [11:21:57] Gerrit is a missing a way to easily see a dependency tree [11:22:17] (03CR) 10Zfilipin: [C: 032] Tie browsertests jobs to wmflabs slaves [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126943 (owner: 10Hashar) [11:22:43] \O/ [11:22:54] (03Merged) 10jenkins-bot: Tie browsertests jobs to wmflabs slaves [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126943 (owner: 10Hashar) [11:23:06] (03PS4) 10Zfilipin: Install gems in a shared dir [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126946 (owner: 10Hashar) [11:23:10] (03PS4) 10Hashar: Remove ruby 2.1.1 when running on wmflabs [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126945 [11:23:19] zeljkof: next is https://gerrit.wikimedia.org/r/#/c/126945/ :] [11:23:54] (03PS5) 10Hashar: Install gems in a shared dir [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126946 [11:23:57] then https://gerrit.wikimedia.org/r/#/c/126946/ [11:24:05] (03PS5) 10Hashar: Make bundle install verbose! [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126947 [11:24:07] and so on [11:24:13] (03PS5) 10Hashar: Use gem 1.9.3 and the installed bundler version [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126948 [11:24:17] (03PS2) 10Hashar: Make the ircbot publisher a macro [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126955 [11:26:09] hashar: I can only +1 this https://gerrit.wikimedia.org/r/#/c/126953/ [11:26:27] zeljkof: yeah that is on operations/puppet [11:26:44] argh [11:26:50] so a trivial rebase does not clear CR+2 :-D [11:26:58] cumbersome [11:27:15] (03CR) 10Zfilipin: [C: 032] Install gems in a shared dir [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126946 (owner: 10Hashar) [11:27:15] you gotta to remove yourself and revote CR+2 to trigger the gate/submit [11:27:36] (03Merged) 10jenkins-bot: Install gems in a shared dir [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126946 (owner: 10Hashar) [11:27:39] (03Merged) 10jenkins-bot: Make bundle install verbose! [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126947 (owner: 10Hashar) [11:27:42] (03Merged) 10jenkins-bot: Use gem 1.9.3 and the installed bundler version [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126948 (owner: 10Hashar) [11:28:30] hashar: I think everything is merged now [11:28:39] except one in puppet repo [11:28:45] one left is https://gerrit.wikimedia.org/r/#/c/126955/ [11:28:50] that remove duplicate code [11:28:59] the diff shows that nothing is modified : https://integration.wikimedia.org/ci/job/integration-jjb-config-diff/176/console [11:30:29] (03PS3) 10Zfilipin: Make the ircbot publisher a macro [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126955 (owner: 10Hashar) [11:30:49] the poor bug is being spammed :-] [11:31:08] hashar: sorry about that :) [11:31:22] yeah, maybe not everything is related to the bug [11:31:31] my bugzilla emails notifications are threaded so it is just about deleting them all with a single keystroke [11:31:50] (03CR) 10Zfilipin: [C: 032] Make the ircbot publisher a macro [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126955 (owner: 10Hashar) [11:32:25] (03Merged) 10jenkins-bot: Make the ircbot publisher a macro [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/126955 (owner: 10Hashar) [11:33:06] hashar: in gmail it is also threaded and just one click to delete/archive everything [11:33:13] :-) [11:42:19] * zeljkof is out to lunch [11:44:07] out for nap [12:04:33] yo ya zeljkof_ [12:04:48] aharoni: coming to the hangout [12:55:07] re [13:32:14] zeljkof: manybubbles sent us a complaint email. we spammed his mailbox earlier today :-] [13:32:27] hashar: just confused me [13:32:31] more then anything [13:32:48] we should disable the notifications [13:42:30] bd808: good morning :] [13:42:50] * bd808 waves and rubs sleep out of his eyes [13:43:13] * hashar raises donuts and coffee [13:43:22] I'm working on a little script to create the right directory structure [13:45:54] bd808: I tried to run salt again but with no luck http://paste.openstack.org/show/76113/ :( [13:46:07] I guess I use the wrong command: salt -G "*" cmd.run hostname [13:46:36] Ah. Try the same thing without `-G` [13:47:01] -G means the pattern will be a grain to target [13:47:31] With not specifier it means you want to target by hostname [13:47:36] my bad [13:47:48] and '*' is a magic hostname that means all hosts [13:48:57] There are some other modifiers too for using a glob match on hostnames and other things [13:49:05] and it shows me the ec2 id such as i-0000007e.eqiad.wmflabs: instead of the hostname :/ [13:49:28] Yeah. That's a lame thing about the salt setup in labs [13:50:09] I'd like to figure out how to change that [13:50:19] ideally one day [13:50:20] :-D [13:50:31] anything I can help to get videoscaler01 to migrate to scap ? [13:50:51] it seems scap itself is triggered as a sub job of the existing code update job [13:51:11] I think I'm ready to try switching the symlink, but I'm not sure I know how to test [13:51:19] well [13:51:20] switch [13:51:24] and see what is broken :-] [13:51:24] I guess I can use eval.php [13:51:36] and we can disable the Jenkins jobs as well [13:51:46] oh no we can't, cause that will broke everything else [13:52:01] yeah we don't want to mess with jenkins yet [13:52:34] When we do all we'll need to do is remove the l10n update step at the end of the current update script [13:59:02] hashar: videoscaler01 is switched [13:59:09] \O/` [14:04:42] hashar: I think I want to switch the apaches and see what happens. Sound crazy? [14:05:33] (03CR) 10Hashar: "The email-ext provides more settings http://ci.openstack.org/jenkins-job-builder/publishers.html#publishers.email-ext . It is probably a g" (033 comments) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/126870 (owner: 10BryanDavis) [14:05:40] bd808: not at all [14:05:52] * bd808 gives it a shot [14:06:02] craziness would be to migrate the prod cluster to php-fpm or something like that [14:06:44] I think Ori's working on that :) [14:07:00] yeah for hhvm [14:07:35] any clue where /usr/local/bin/wmf-beta-scap is ? [14:08:09] It's only installed on deployment-bastion [14:08:34] It comes from my WIP patch to add all the scap bits [14:08:38] ah it must be pending in some puppet change [14:08:41] \O/ [14:09:00] hashar: It's in https://gerrit.wikimedia.org/r/#/c/123674/ [14:09:25] https://gerrit.wikimedia.org/r/#/c/123674/14/modules/beta/files/wmf-beta-scap,unified [14:09:26] I havent looked at this one yet :-( [14:10:28] That patch is all the crazy stuff I've added to make scap work in beta [14:11:06] It's actually turned out to be cleaner than I'd imagined it would be when I started [14:11:57] we have a bunch of random sudo policy around as well https://wikitech.wikimedia.org/wiki/Special:NovaSudoer [14:12:23] I can't remember which one I created originally for the beta code update jenkins job [14:19:25] hashar: apaches are switched and things seem to be working. I got 3 errors about "Cannot redeclare class languageconverter" when I first switched but restarting apaches seems to have made that go away. [14:19:48] maybe a glitch with apc [14:19:56] Probably [14:20:10] I switched symlinks out from under it [14:21:10] So now there is a /usr/local/apache-local directory with the right symlinks inside it. /usr/local/apache links to apache-local instead of /data/project/apache [14:21:47] Should switching the jobrunner be just as easy? [14:22:04] zeljkof: you can add these to your report: https://gerrit.wikimedia.org/r/#/c/126985/ and https://gerrit.wikimedia.org/r/#/c/126974/ . [14:22:22] aharoni: thanks :) [14:22:36] bd808: yeah switch them all [14:23:01] bd808: I guess the jenkins job beta code update will have to be refreshing the repositories to some other place than the NFS dir right ? [14:23:53] Eventually I think that would be good just to get off of nfs all together. [14:24:07] right now that dir is treated like /a/common on tin [14:24:23] so Jenkins preps it and then scap copies it around via rsync [14:25:02] and advantage of /data/project is that it is redundant / backed up (I hope) [14:25:09] where as /srv/ is local to the instance [14:25:19] so if the instance dies/is deleted, the data are gone [14:25:29] yes, but I'm pretty sure it's the cause of the l10n updates being so slow [14:26:04] The l10n update code stats every i18n file multiple times. Over nfs that is really really slow [14:26:38] If we can move the primary checkout to local disk on deployment-bastion things should be much faster [14:26:49] s/things/l10n build/ [14:27:17] I see the use case [14:27:38] if the git repositories are lost, at worse we will have to reclone them [14:27:51] Yeah. REally it's all "backed up" in gerrit [14:28:17] There shouldn't be any local changes in the beta cluster [14:28:46] Ok, jobrunner is switched too [14:29:19] I'll write an email explaining what I did and how to undo it if things get weird [14:29:43] Which lists should I send that to? qa-l and eng-l? [14:30:45] so everything got switched right ? [14:31:03] I am not sure whether there is any need to undo [14:31:18] I think so. apache0[12], videoscaler01 and jobrunner01 [14:31:25] zeljkof: can you run one of the passing Cloudbees job against beta please? :-) [14:31:40] bd808: I think that covers everything [14:31:40] Good call. [14:31:53] hashar: sure [14:32:06] The only reason to roll back would be if we find out that scap isn't really updating things correctly [14:32:36] In that event we could roll back by fixing the symlinks and then figure out what's wrong with scap [14:40:17] hashar: In that JJB review, the upstream job woudn't fail if the downstream broke. [14:40:48] I think there is a plugin that can watch downstream status but I'd have to look it up [14:41:35] I'm pretty sure I have used that before to make complex job chains but I may have been using the pipeline plugin [14:42:43] sorry sorry [14:42:51] reviewing your huge scap change :° [14:43:15] bd808: ahh [14:43:41] maybe the trigger plugin as an option to fail the job if the downstream jobs are failing [14:44:34] * hashar tries [14:46:58] the child is triggered as a post build [14:47:04] and that is not changing the result grr [14:49:52] I think what I've done before is have a master pipeline build that fires childA, childB in parallel and then either fires childC or dies depending on the results of A & B [14:49:52] bd808: did review the huge scap change https://gerrit.wikimedia.org/r/#/c/123674/ [14:50:18] you can also trigger B as a build step of A [14:50:26] ie A ( B ) [14:50:34] and whenever B fails, A will be marked as a failure [14:50:48] https://integration.wikimedia.org/ci/job/test-parent/2/console :-D [14:51:06] test-parent invokes as a build step test-child-fails which exit 1 [14:51:25] looking for JJB conf [14:55:14] (03CR) 10Hashar: [C: 04-1] "The downstream job can be triggered as a build step of the parent job by using the parameterized trigger plugin http://ci.openstack.org/je" (031 comment) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/126870 (owner: 10BryanDavis) [14:55:17] there [14:55:51] Cool. I'll work on that later. [14:56:52] lovely output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/810/console :-] [14:58:00] maybe we can move the lame wmf-beta-autoupdater.py I wrote in a hurry from puppet to the scap project [14:59:49] I suppose we could. I wouldn't want that to end up running in prod but if we left 'beta' in the name that should warn folks [15:02:47] probably [15:03:15] is the l10n cache synced from the bastion /data/project to each application server to /srv/common-local/... ? [15:04:22] hashar: Yes [15:04:49] and hopefully the MediaWiki are using the new /srv/common-local [15:05:43] I am trying to find how we could get the cdb files generated on bastion to the local dir instead of to /data/project [15:06:08] bd808: mediawiki.org just died for me [15:06:25] * bd808 didn't do it [15:06:43] I see that error though chrismcmahon [15:07:01] chrismcmahon: that is a production issue. There is notification about it in #wikimedia-operations [15:07:10] chrismcmahon: the main text load balancer flapped [15:07:48] thanks [15:12:13] hashar: sorry, forgot to let you know about the job https://wmf.ci.cloudbees.com/view/cs/job/CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox/127/console [15:16:16] zeljkof: \O/ [15:23:42] hashar: I'm off to shower and have breakfast. Thanks for the moral support and reviews. [15:30:08] bd808|AWAY: you are the best! heading out to get my daughter back home [15:30:12] bd808|AWAY: congratulations! [15:40:52] Project CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox build #9: STILL FAILING in 3 min 27 sec: https://integration.wikimedia.org/ci/job/CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox/9/ [16:24:06] (03PS1) 10Zfilipin: Run browser tests on the local machine headlessly [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/127003 [18:08:54] chrismcmahon: strange browser failure https://wmf.ci.cloudbees.com/view/flow/job/Flow-en.wikipedia.beta.wmflabs.org-linux-firefox/lastFailedBuild/testReport/junit/%28root%29/Edit%20existing%20title/Edit_existing_post/ , seems Ruby lost connection to the driven browser [18:15:53] spagewmf: I saw that earlier today [18:16:07] spagewmf: actually, that [18:17:40] spagewmf: actually, that's not the strange one. that "connection reset" happens *all the time*, it seems to be a problem with internet between Cloudbees and Sauce Labs. Zeljko and I are working right now on being able to run builds in WMF Jenkins in headless Firefox to talk to beta labs directly. It's coming, and I can't wait. [18:18:22] spagewmf: this is the strange one: https://wmf.ci.cloudbees.com/job/Flow-en.wikipedia.beta.wmflabs.org-linux-firefox/371/ afaict, Sauce was just 404 for a short time. [18:29:50] (03PS2) 10BryanDavis: Run scap when config or code is updated [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/126870 [18:42:18] Project CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox build #10: STILL FAILING in 3 min 24 sec: https://integration.wikimedia.org/ci/job/CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox/10/ [21:19:34] (03Abandoned) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [qa/browsertests] - 10https://gerrit.wikimedia.org/r/57292 (owner: 10Hashar)