[00:09:32] (03CR) 10Krinkle: "Still broken per regression https://github.com/Medium/phantomjs/issues/235" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/166900 (owner: 10Krinkle) [03:43:10] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #72: FAILURE in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/72/ [04:12:52] Yippee, build fixed! [04:12:53] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #293: FIXED in 38 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/293/ [04:36:04] Yippee, build fixed! [04:36:04] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #58: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/58/ [04:44:16] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #70: FAILURE in 36 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/70/ [05:39:20] Yippee, build fixed! [05:39:21] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #340: FIXED in 1 hr 3 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/340/ [06:43:21] Yippee, build fixed! [06:43:21] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #274: FIXED in 1 hr 1 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/274/ [06:54:38] PROBLEM - BetaLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash1.diskspace._var.byte_avail.value (10.00%) [07:26:10] Yippee, build fixed! [07:26:10] Project browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce build #217: FIXED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce/217/ [07:49:17] (03CR) 10Zfilipin: [C: 032] Merge browsertest-website and browsertest-website-version builders [integration/config] - 10https://gerrit.wikimedia.org/r/166981 (owner: 10Zfilipin) [07:52:47] (03Merged) 10jenkins-bot: Merge browsertest-website and browsertest-website-version builders [integration/config] - 10https://gerrit.wikimedia.org/r/166981 (owner: 10Zfilipin) [08:08:22] (03PS2) 10Hashar: Add myself to trusted users list [integration/config] - 10https://gerrit.wikimedia.org/r/167107 (owner: 10John F. Lewis) [08:08:34] (03CR) 10jenkins-bot: [V: 04-1] Add myself to trusted users list [integration/config] - 10https://gerrit.wikimedia.org/r/167107 (owner: 10John F. Lewis) [08:09:08] (03CR) 10Hashar: [C: 032] "Deploying." [integration/config] - 10https://gerrit.wikimedia.org/r/167107 (owner: 10John F. Lewis) [08:09:32] zeljkof: I am flipping papers this morning for accounting purposes :D [08:10:06] hashar: good luck :) [08:10:20] I have managed to finish it yesterday [08:10:31] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #176: FAILURE in 8.9 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/176/ [08:11:52] !log deployment-logstash1 deleting files under /var/log/apache2/ gotta fill a bug to prevent access log from filling the partition [08:13:02] (03Merged) 10jenkins-bot: Add myself to trusted users list [integration/config] - 10https://gerrit.wikimedia.org/r/167107 (owner: 10John F. Lewis) [08:20:56] 3Wikimedia Labs / 3deployment-prep (beta): Diamond logstash monitor fills /var/log/apache2 access log - 10https://bugzilla.wikimedia.org/72175 (10Antoine "hashar" Musso (WMF)) 3NEW p:3Unprio s:3normal a:3None deployment-logstash1.eqiad.wmflabs ended up with a filled /var/ /var/log/apache2/other_vhos... [08:21:30] !log logstash1 instance being filled up is {{bug|72175}} probably caused by the Diamond collector spamming /server-status?auto [08:22:07] 3Wikimedia Labs / 3deployment-prep (beta): Automatic mobile redirects for MobileFrontend on Betalabs - 10https://bugzilla.wikimedia.org/46620 (10Yuvi Panda) [08:22:32] !log deployment-logstash1 unlocking puppet by deleting left over /var/lib/puppet/state/agent_catalog_run.lock [08:22:42] bah the logger is broken [08:22:43] ... [08:22:56] !sal [08:22:56] https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:25:34] RECOVERY - BetaLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [08:28:32] and accounting time [08:29:52] 3Wikimedia Labs / 3deployment-prep (beta): Diamond logstash monitor fills /var/log/apache2 access log - 10https://bugzilla.wikimedia.org/72175#c1 (10Antoine "hashar" Musso (WMF)) + Yuvi Panda: seems the diamond collector for logstash spams the apache access log with garbage requests such as ""\x16\x03\x01"... [09:14:37] 3Wikimedia Labs / 3deployment-prep (beta): Diamond logstash monitor fills /var/log/apache2 access log - 10https://bugzilla.wikimedia.org/72175#c2 (10Antoine "hashar" Musso (WMF)) That might be due to monitor fatal using an HTTPS connection: modules/beta/files/monitor_fatals.rb:27:https://logstash-beta.wmfla... [09:17:18] !log [09:17:24] !sal [09:17:24] https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:40:38] * zeljkof is out of lunch [11:20:18] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #15: STILL FAILING in 27 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/15/ [11:28:50] (03PS2) 10Tobias Gritschacher: Fix broken jobs when BROWSER_TIMEOUT is not set [integration/config] - 10https://gerrit.wikimedia.org/r/166547 (owner: 10Zfilipin) [11:30:41] (03PS3) 10Tobias Gritschacher: Fix broken jobs when BROWSER_TIMEOUT is not set [integration/config] - 10https://gerrit.wikimedia.org/r/166547 (owner: 10Zfilipin) [11:31:21] (03PS4) 10Tobias Gritschacher: Fix broken jobs when BROWSER_TIMEOUT is not set [integration/config] - 10https://gerrit.wikimedia.org/r/166547 (owner: 10Zfilipin) [11:36:20] https://gerrit.wikimedia.org/r/166547 needs to be merged to unbreak all browsertest jobs [13:17:20] bah more or less back [13:46:30] hashar: I think with reconfiguring the jobs (after the split of macro.yaml) the BROWSER_TIMEOUT bug came back.. [13:46:42] because the fix was not merged yet [13:47:01] I guess that all jobs will fail [13:47:07] zeljkof: ^^ [13:47:23] Tobi_WMDE_SW_NA: nice :) [13:47:37] will take a look [13:47:56] oh no [13:48:00] zeljkof: I rebased the change some hours ago [13:48:05] gotta rebase the timeout patch so [13:48:13] sorry should have caught it :( [13:48:18] was not without conflicts because of the changes to macr.yaml [13:52:02] * zeljkof brb [14:00:48] Tobi_WMDE_SW_NA: have you redeployed the jobs for the BROWSER_TIMEOUT fix ( https://gerrit.wikimedia.org/r/#/c/166547/4 ) ? [14:00:51] if so will +2 [14:01:24] hashar: no, did not du that [14:01:41] I will -:-) [14:01:48] I just did it for one wikidata job to verify that this is actually the issue [14:02:08] (03CR) 10Hashar: [C: 032] "Thanks for the rebase. Merging in and refreshing the jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/166547 (owner: 10Zfilipin) [14:03:13] I confirmed it reading the ruby related code [14:03:22] it sets a default of 0 if BROWSER_TIMEOUT='' [14:03:29] 0 meaning wait 0 seconds :D [14:06:07] (03Merged) 10jenkins-bot: Fix broken jobs when BROWSER_TIMEOUT is not set [integration/config] - 10https://gerrit.wikimedia.org/r/166547 (owner: 10Zfilipin) [14:06:52] Project browsertests-Wikidata-PerformanceTests-linux-firefox-sauce build #17: FAILURE in 51 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/17/ [14:14:00] hashar: yeah. discussed this with zeljkof yesterday [14:14:01] bah [14:14:08] yet another failure [14:14:33] hashar: unrelated [14:14:48] current build for wikidata is broken ob beta.. [14:25:03] :( [14:38:10] 3Wikimedia / 3Continuous integration: fix the qunit tests for wikidata: mwext-Wikibase-qunit - 10https://bugzilla.wikimedia.org/72184 (10Jan Zerebecki) 3NEW p:3Unprio s:3normal a:3None Fix the qunit tests for wikidata: mwext-Wikibase-qunit and after that make them voting. [14:48:14] I am off for this week [14:48:22] not planning to connect to IRC tonight :D got to focus [14:50:33] (03PS1) 10Zfilipin: WIP testing rubocop [ruby/api] - 10https://gerrit.wikimedia.org/r/167210 [14:51:00] (03CR) 10Zfilipin: "check experimental" [ruby/api] - 10https://gerrit.wikimedia.org/r/167210 (owner: 10Zfilipin) [14:56:42] (03PS2) 10Zfilipin: WIP testing rubocop [ruby/api] - 10https://gerrit.wikimedia.org/r/167210 [14:58:17] (03CR) 10Zfilipin: "check experimental" [ruby/api] - 10https://gerrit.wikimedia.org/r/167210 (owner: 10Zfilipin) [15:02:33] (03CR) 10Cmcmahon: [C: 032] ZeroPortal has browser tests [selenium] - 10https://gerrit.wikimedia.org/r/166549 (owner: 10Zfilipin) [15:02:55] (03Merged) 10jenkins-bot: ZeroPortal has browser tests [selenium] - 10https://gerrit.wikimedia.org/r/166549 (owner: 10Zfilipin) [15:09:07] 3Wikimedia / 3Quality Assurance: create custom RSpec matchers for when_present, when_not_present, etc - 10https://bugzilla.wikimedia.org/72149#c1 (10Chris McMahon) Zeljko and I dug into this and we discovered that refactoring has removed most instance of this sort of thing. But here is one example still in... [15:12:52] 3Wikimedia / 3Quality Assurance: create custom RSpec matchers for when_present, when_not_present, etc - 10https://bugzilla.wikimedia.org/72149#c2 (10Ċ½eljko Filipin) This should do the same thing: Then(/^I should see the topic reply form$/) do expect(on(FlowPage).new_reply_input_element.when_visible).t... [15:14:06] (03PS3) 10Zfilipin: WIP testing rubocop [ruby/api] - 10https://gerrit.wikimedia.org/r/167210 [15:15:08] 3Wikimedia / 3Quality Assurance: create custom RSpec matchers for when_present, when_not_present, etc - 10https://bugzilla.wikimedia.org/72149#c3 (10Chris McMahon) I think this should actually be Then(/^I should see the topic reply form$/) do expect(on(FlowPage).new_reply_input_element.when_present).to... [15:15:39] (03CR) 10Zfilipin: "check experimental" [ruby/api] - 10https://gerrit.wikimedia.org/r/167210 (owner: 10Zfilipin) [15:40:42] (03CR) 10Zfilipin: "Success! :)" [ruby/api] - 10https://gerrit.wikimedia.org/r/167210 (owner: 10Zfilipin) [15:51:53] 3Wikimedia / 3Continuous integration: fix the qunit tests for wikidata: mwext-Wikibase-qunit - 10https://bugzilla.wikimedia.org/72184 (10John F. Lewis) p:5Unprio>3Low [15:53:07] 3Wikimedia / 3Continuous integration: make wikidata-build instances work without puppet master self - 10https://bugzilla.wikimedia.org/71419 (10John F. Lewis) p:5Unprio>3Normal [16:02:09] 3Wikimedia Labs / 3deployment-prep (beta): Automatic mobile redirects for MobileFrontend on Betalabs - 10https://bugzilla.wikimedia.org/46620#c1 (10Greg Grossmeier) a:5Ryan Lane>3None Resseting assignee (from Ryan Lane). Kaldari: Is this still needed? (this was in the general WMF Labs queue, not Beta Cl... [17:57:01] hi marxarelli if you're up for more guerilla merging I have two more patches https://gerrit.wikimedia.org/r/#/c/167109/ and https://gerrit.wikimedia.org/r/#/c/167232/. They're trivial, I'm comfortable self-merging those but I wouldn't mind eyeballs. [17:59:59] chrismcmahon: sure thing [18:01:51] and with that I believe I am done with all of the easy (ha) refactoring in the Flow repo. there are still two tests that need some work, but I'll tackle those next week in SF [18:04:15] chrismcmahon: cool. looking forward to meeting you face to face, btw. we should get drinks or something [18:04:53] well, hangouts are face to face, i suppose. irl i guess :) [18:08:26] let's do that. I haven't been to SF in a long time [18:35:13] marxarelli: fixed, thanks [18:46:32] Yippee, build fixed! [18:46:33] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #73: FIXED in 25 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/73/ [18:56:22] 3Wikimedia / 3Continuous integration: fix the qunit tests for wikidata: mwext-Wikibase-qunit - 10https://bugzilla.wikimedia.org/72184 (10Lydia Pintscher) p:5Low>3Normal [18:59:15] James_F: /me shops around that bug via component moving :) [19:13:39] greg-g: Hadn't noticed. :-) [19:16:54] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #294: FAILURE in 36 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/294/ [19:27:05] Yippee, build fixed! [19:27:06] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #126: FIXED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/126/ [19:34:05] anyone know why beta labs is 503ing now and then for the past few days? [20:02:22] Yippee, build fixed! [20:02:23] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #268: FIXED in 35 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/268/ [20:05:32] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #341: FAILURE in 23 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/341/ [20:08:07] Project browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce build #206: FAILURE in 1 min 35 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce/206/ [20:31:02] !log beta: switching Parsoid config file to the one in mediawiki/services/parsoid/deploy.git instead of the puppet maintained config file https://gerrit.wikimedia.org/r/#/c/166610/ for subbu [20:33:18] ffff [20:33:23] !sal [20:33:23] https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:40:34] qa-morebots, feeling better? [20:40:34] I am a logbot running on tools-exec-14. [20:40:34] Messages are logged to https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL. [20:40:34] To log a message, type !log . [20:48:24] andrewbogott: you are a magician ! [20:48:32] !log qa-morebots is back [20:48:33] Logged the message, Master [20:48:38] Do you want me to show you how to do it? [20:48:57] * hashar ponders whether he is willing to learn the tools labs software stack [20:49:08] I am not in the tools.morebot group :/ [20:49:46] would you like to be? [20:50:20] The nice thing about 'become morebots' is that then everything that you might want to do is already in your bash history :) [20:50:40] yeah definitely add me in that group [20:50:58] earlier today I looked at the documentation on https://wikitech.wikimedia.org/wiki/Morebots [20:51:09] and that seems straightforward [20:51:44] !sal [20:51:44] https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:53:31] hashar: ok, added you [21:02:33] andrewbogott: are become / qstat / qdel etc commands that have been developed specially for tools labs? [21:02:46] oh and my access work :-] Thank you [21:03:03] I think that become is and that qstat and qdel are standad sge commands. I'm not sure though [21:03:06] *standard [21:13:36] oh Sun Grid Engine [21:14:44] should use that to replace Jenkins :D [21:14:53] no! [21:15:07] oh I am sure I will phase out Jenkins [21:15:14] at least for CI purposes [21:15:23] how so? [21:15:26] But Jenkins is so dapper [21:15:36] but we will definitely keep some Jenkins when it is actually useful such as for Maven projects or browsertests [21:15:46] hashar: what would you replace Jenkins with? [21:15:56] but to run php lint or PHPUnit ... There is not much added value [21:16:52] Gearman workers running in isolated VM and hooked directly with Zuul [21:16:52] hashar: (I was actually watching a thread about alternates to standard CI systems recently.) [21:17:23] so Zuul would send a work request such as: mediawiki-core-phplint and a worker grab it, executes, done [21:17:41] I am meeting the OpenStack infra folks beginning of November in Paris for a week [21:17:51] and they already started the effort to replace jenkins entirely [21:18:01] in their use case, it has no real added value [21:18:52] one even proposed to have JJB to support generating Ansible tasks [21:19:00] which are described in YAML, just like ... JJB ! :D [21:19:05] hashar: there is a guy on the QA mail list named Noah Sussman who was part of the Etsy DevOps team that built their Continuous Delivery system, he talks about this stuff a lot. (And he uses WMF Jenkins and beta labs in talks and training) [21:19:19] hire him! :D [21:19:35] I think he's happy being a consultant in NYC :-) [21:20:13] but he's someone who know about us who also knows about these systems, if you want to get ideas or anything [21:20:17] knows [21:20:20] ah yeah i found an email by him replying to me :D [21:20:49] well [21:21:02] I would love to send the console logs / junit reports to logstash [21:21:08] I met him at the Google Test Automatiion Conference last year. This year's is in 2 weeks I'll be there too. [21:22:36] chrismcmahon: there might be some OpenStack folks attending. If you see any you should ask them about how they handle qa / release / ci :-) [21:22:48] will do [21:24:09] I might trick one of their CI guys to come in the SF Office in January while I am there [21:24:18] depends on availability obviously [21:25:03] hashar, i am not sure if the logs are getting to beta logstash ... i wanted to login and check .. can i ssh onto the beta parsoid instance from deployment-bastion? is it deployment-parsoid04? [21:25:31] you should be able [21:26:02] through probably not from deployment-bastion since your ssh credentials are not there :D [21:26:26] got in. [21:26:34] yes, i was trying to deployment-bastion and it didn't work. [21:26:40] i logged out and tried direclty and i am in now. [21:27:01] subbu: deployment-bastion is really the equivalent of tin.eqiad.wmnet [21:27:05] it is misnamed :-D [21:27:13] ok. [21:31:51] ah .. the betalabs logstash patch hasn't been +2ed yet .. [21:32:06] ok, now to rebase and have it reviewd by arlo. [21:46:02] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #275: FAILURE in 1 hr 2 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/275/ [22:04:43] i broke beta labs parsoid with the logging patch .. will fix in a bit. [22:10:38] hashar, is it okay to live edit the config file and restart parsoid to do a quick test? or should i go through gerrit, merge, and auto-dpeloy? [22:11:29] subbu: Jenkins fetch new changes on postmerge [22:11:34] so that might override your live edit [22:11:39] that is fine. [22:11:43] but if it is quick enough and between two merges, you would be fine [22:11:48] i just want to verify that my fix will work. [22:12:02] make sure clear the repo and push to Gerrit after [22:12:12] will do. [22:12:12] or stuff will break :-] [22:12:14] k [22:12:32] I think the Jenkins job does a git clean then fetch && checkout [22:12:56] subbu: and if you want to keep a trace of what you did. you can !log in this channel [22:12:59] that is then logged to: [22:13:02] !sal | subbu [22:13:02] subbu: https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:13:40] for very important changes , folks mail the QA list [22:14:25] bed crashing [22:14:30] have a nice weekend everyone [22:14:35] ok, it is just a trivial fix to a bad config. [22:14:41] thanks. yo utoo. [22:14:45] will verify and push a gerrit fix. [22:15:31] thanks hashar [22:16:14] subbu: ho and on mediawiki-config is slightly more complicated because the jenkns job run every 10 minutes and we use scap to push config to a bunch of instances [22:16:39] hmm not every 10 minutes, it is on post merge as well [22:16:50] sleeping for real now [22:22:39] Project browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce build #218: FAILURE in 35 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce/218/ [22:34:38] !log live fixed bad logger config in /srv/deployment/parsoid/deploy/conf/wmf/betalabs.localsettings.js and verified that parsoid doesn't crash anymore -- fix now on gerrit and being merged [22:34:40] Logged the message, Master [22:43:29] chrismcmahon, do you know if I have the right servername/port for logstash on beta? [22:43:30] LOGSTASH_HOSTNAME='deployment-logstash1.eqiad.wmflabs'; [22:43:30] LOGSTASH_PORT=12201; [22:43:45] Yippee, build fixed! [22:43:45] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #231: FIXED in 17 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/231/ [22:44:18] subbu: sorry, I don't know. Reedy, do you? ^^ [22:44:23] i am not seeing the log events show up in https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/default [23:13:06] Yippee, build fixed! [23:13:07] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #177: FIXED in 1 min 0 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/177/ [23:16:56] Yippee, build fixed! [23:16:57] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #226: FIXED in 21 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/226/