[00:12:11] (03CR) 10Krinkle: "Pushed oojs-core-npm, oojs-ui-npm, unicodejs-npm." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159224 (https://bugzilla.wikimedia.org/66056) (owner: 10Krinkle) [00:15:36] (03CR) 10Krinkle: "Pushed mwext-TemplateData-npm and mwext-VisualEditor-npm." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159224 (https://bugzilla.wikimedia.org/66056) (owner: 10Krinkle) [00:15:51] (03PS5) 10Krinkle: job-templates: Move {name}-npm from Precise to Trusty [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159224 (https://bugzilla.wikimedia.org/66056) [00:16:18] 3Wikimedia / 3Continuous integration: Jenkins: Set up integration slaves with Ubuntu Trusty - 10https://bugzilla.wikimedia.org/68256 (10Krinkle) 5ASSI>3RESO/FIX [00:16:30] 3Wikimedia / 3Continuous integration: Jenkins: Upgrade nodejs from 0.8.x to 0.10.x on wmflabs integration slaves - 10https://bugzilla.wikimedia.org/66056 (10Krinkle) 5PATC>3RESO/FIX [00:18:39] grrrrrrrrr [00:51:40] !log deployment-pdf01 removed base::firewall (ldap via wikitech) [00:51:42] Logged the message, Master [00:54:40] deployment-pdf01 .. lol? [00:54:48] i hope that is OCG [00:54:54] and not actually pdf1 the old one [00:55:20] rotfl [00:55:23] i had the same thought [00:55:29] i needs some puppet help [00:55:47] let's see what quota's like [00:57:07] cscott: you there? [01:08:29] ok, pdf02 is "building"... [01:28:39] PROBLEM - BetaLabs: Puppet failure events on labmon1001 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash1.puppetagent.failed_events.value (100.00%) [01:30:16] (03Abandoned) 10Jforrester: browsertests: Make it possible to specify a browser target version [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/157479 (owner: 10Jforrester) [01:59:05] jeremyb: i'm here. [02:00:08] jeremyb: ocg hosts are listed in https://wikitech.wikimedia.org/wiki/OCG -- they are ocg100[123].eqiad.wmnet (production) and deployment-pdf01.eqiad.wmflabs (beta) [02:01:43] http://pdf2.wikimedia.org:8080/mw-serve/ is the tampa machine we're shutting down soon, and http://ocg.svc.eqiad.wmnet:80 is a round-robin for ocg100[123].eqiad.wmnet (i think) [02:01:43] 00 [02:03:26] cscott: well i just created a pdf02 [02:03:30] still puppeting [02:03:43] we'll see if it has the same problem as pdf01 [02:03:54] pdf01 is down in prod [02:04:01] only pdf02 and pdf03 are up [02:04:13] mutante: doesn't prod have no zeros? [02:04:22] * jeremyb is in labs [02:04:32] labs has a different naming scheme? :p [02:05:01] yea, no zeros [02:05:28] cscott: why don't we call it ocg btw? [02:11:30] (03PS1) 10Jforrester: parsoidsvc: Add an explict npm on node 0.10 and on node 0.8 [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160589 [02:21:15] jeremyb: ask mwalker [02:22:01] hah [02:23:53] cscott: how do i test if a pdf node is working right/ [02:23:56] ?* [02:25:03] jeremyb: good question ;) [02:27:03] jeremyb: there should be a web server on port 17080 i think. [02:27:05] (03CR) 10Krinkle: "Compiled and deployed:" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159224 (https://bugzilla.wikimedia.org/66056) (owner: 10Krinkle) [02:27:10] cscott: ^ [02:28:07] you mean 8000? [02:29:13] anyway, where does it log to? [02:29:48] oh, is this an upstart thing? eww [02:30:08] i think i do mean 17080. at least on localhost:17080 i see the ocg service. [02:30:40] jeremyb: you should be looking at https://wikitech.wikimedia.org/wiki/OCG probably [02:31:02] i couldn't finish today's OCG deploy on beta (deployment-prep) because of the salt issues, though. [02:31:24] (03PS6) 10Krinkle: job-templates: Move {name}-npm from Precise to Trusty [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159224 (https://bugzilla.wikimedia.org/66056) [02:31:45] cscott: i fixed most of the salt stuff [02:31:54] well not all [02:31:59] i should clean up the rest :P [02:33:06] $ sudo netstat -lptn | fgrep nodejs [02:33:06] tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 24843/nodejs-ocg [02:33:08] so what is that? [02:34:16] (03CR) 10Jforrester: [C: 031] Swap in the new parsoidsvc-(source|deploy) jobs. [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/156693 (owner: 10Cscott) [02:36:18] oh, hah [02:36:19] jeremyb: oh, it looks like it's configured to be port 8000 in production, even though the default port is 17080 [02:36:24] Error: Cannot find module '/srv/deployment/ocg/ocg/mw-ocg-service.js' [02:37:20] jeremyb: perhaps ocg has never been git deployed there? [02:38:00] and/or you've checked out git but not checked out the submodules (since mw-ocg-service.js is a link inside a submodule) [02:39:09] cscott: well it's a brand new instance [02:39:11] jeremyb@deployment-bastion:/mnt/srv/deployment/ocg/ocg$ git deploy start [02:39:12] A deployment has already been started for this repo by C. Scott Ananian. [02:39:57] for mediawiki, puppet will do the initial sync for you. i think [02:40:04] i figured maybe ocg was the same [02:42:05] oh, right, i did git deploy start, but salt crapped out so i left it there. i guess i need to clean up and/or complete that deploy? [02:42:17] whatever :) [02:43:41] 1/2 minions completed fetch [02:43:41] Details: [02:43:42] i-000005d2.eqiad.wmflabs: [02:43:42] fetch status: None [started: 23 mins ago, last-return: None mins ago] [02:57:21] cscott: so, logs? [03:12:31] jeremyb: they are in logstash for production, although i'm not certain that labs/beta also dumps to logstash [03:13:29] configuration in /etc/ocg/mw-ocg-service.js -- looks like they go to deployment-logstash1.eqiad.wmflabs ? [03:26:32] so, there's nothing in /var/log? [03:29:02] hah [03:29:04] > A username and password are being requested by https://logstash-beta.wmflabs.org. The site says: "Logstash (ssh deployment-bastion.eqiad.wmflabs sudo cat /root/secrets.txt)" [03:38:44] :) [03:43:38] RECOVERY - BetaLabs: Puppet failure events on labmon1001 is OK: OK: All targets OK [03:53:13] hrmmm, /home/bd808/cleanup-hhvm-cores [03:55:25] jeremyb: cleanup-hhvm-cores doing something bad? [03:55:35] It is a hack [03:59:28] bd808: /run was 99%. so i rm -rv /run/hhvm/cache and booted hhvm [03:59:46] !log deployment-mediawiki03: rm -rv /run/hhvm/cache && service hhvm restart [03:59:50] Logged the message, Master [03:59:57] !log deployment-mediawiki03: (/run was 99%) [04:00:01] Logged the message, Master [04:00:03] gah [04:00:05] damnit [04:00:13] those were 02 [04:00:16] i'll fix the wiki [04:00:38] oooh. That may be something we need to manage too. hhvm's bytecode cache [04:01:14] jeremyb: That may actually be worth opening a bug to track (periodic pruning of hhbc caches) [04:01:16] !log deployment-mediawiki02: salt was broken with a msgpack exception. mv -v /var/cache/salt{,.old} && service salt-minion restart fixed it. also did salt-call saltutil.sync_all [04:01:18] Logged the message, Master [04:02:04] what are hhvm cores? [04:03:08] hhvm will create a core dump when it has a fatal error. For a while that was happening often enough that we filled up the root partition on the mediawiki* boxes [04:03:34] So I made a cron to sweep them onto the nfs shared drive [04:05:06] bd808: hhbc are taking room too. but these were bigger (and it said cache explicitly in dir name) so i got them first [04:05:08] removed ‘/run/hhvm/cache/cli.hhbc.sq3’ [04:05:08] removed ‘/run/hhvm/cache/fcgi.hhbc.sq3’ [04:05:20] hhbc is still there [04:06:20] The hhbc files are sqlite databases that hhvm uses to store the php bytecode. Think of them as the APC cache but persistent between hhvm invocations. [04:06:50] We probably should be clearing them out somehow as a part of scap [04:06:54] (to clarify: 238636 /run/hhvm/hhvm.hhbc is still there) [04:06:56] that's kb [04:07:17] does it prune old stuff on its own? [04:07:28] I don't think it does [04:07:49] and each new hhvm binary will start all over again in the same db file :/ [04:08:10] When the internal hhvm schema changes it starts a new set of tables [04:08:37] FB ships their code with the hhbc precompiled on each deploy [04:08:49] we just have a big cache that we let grow organically [04:09:07] and at some point that will become a problem obviously [04:10:08] haha [04:10:20] how do they make sure everything's in the cache? [04:11:25] They build their deployments on a staging server and only copy the hhbc to the hhvm servers. No files on disk. [04:11:54] I have some testing scripts to do the same for MW, but we haven't tried it as the deploy method [04:12:12] ohh. so there's not even php on the hhvm boxes [04:12:14] huh [04:12:15] in part because rsync is horrible at shipping large binaries [04:12:43] i bet facebook don't use trebuchet :P [04:12:57] They use bitorrent [04:14:29] I think at this point in time WMF is the only place that uses Trebuchet. Ryan started a nice open project for it but last I heard was not using it himself at Lyft [04:16:30] The coolest thing about the FB deploy system is that they build a squshfs volume with all the parts in it (hhvm binary, hhbc cache, assets) and ship that to the hosts with bittorrent. [04:16:42] Then the node just unmounts the old version and mounts the new one [04:17:09] rollback is easy because they leave an old version or two on disk [04:17:33] even better than switching symlinks for atomic cutover [04:19:19] what about while it's unmounted? [04:20:08] They depool during deploy [04:20:44] so atomic isn't so important for them [04:21:07] but the "everything in a file" approach is very appealing to me [04:22:51] Thier process is something like: prepare image, torrent image to hosts, select a set of hosts to switch to new version, tell hhvm there to stop accepting new requests, wait for active requests to finish, shutdown hhvm, unmount, mount, run local sanity tests, repool [04:25:02] with salt? :) [04:26:05] I'm not sure what they use for command-and-control [04:26:29] salt is a great idea if you're looking to replace puppet [04:26:52] I'm still not sold on it as an addendum for puppet [04:27:18] Opinions differ but I'd pick mcollective if it was up to me [04:27:36] If only because it has native puppet intgration [04:27:50] instead of being something else entirely [04:30:31] well we don't need to keep puppet at all [04:36:53] Sure… we just need to retrain enough people in salt and then convert the whole cluster config to a new platform that works on slightly different basic principles. [04:37:03] But on the plus side, python! [04:37:28] bd808: so how do i check if there's anything missing in salt? [04:38:03] * bd808 looks in bash history for magic commands [04:38:05] salt-master doesn't have a way to know if nodes are missing? [04:38:50] sudo salt-run manage.down [04:38:55] aha [04:39:14] that will show hosts that have certs on the master but don't respond to a ping [04:39:25] sudo salt-run manage.up shows the ones that do [04:39:49] and salt-run manage.down still looks very sad [04:40:01] oh wait jsut a little sad now [04:40:59] I really wish salt knew the host by their pretty names instead of the i-* names [04:40:59] yes, i've been playing with dsh... [04:41:09] well that's configurable!! [04:41:20] we could use a different /etc/salt/minion [04:44:12] jeremyb: If you can figure out how to change the node ids in beta to be the dns names of the instances and make that happen automagically for new instances that are setup that would be a++ awesome. [04:44:28] I know just enough salt to get myself in trouble [04:46:22] jeremyb: of the 3 hosts showing down, deployment-mediawiki04 seems to be the only one that is still an actual instance. [04:46:36] The other two are just old keys it looks like [04:47:23] There's a handy script in ~bd808/downed-salt-hosts.php that Mukunda wrote (and I just remembered) that converts the i-* names to host names [04:47:55] hah, ancient history. |Instance Name=gluster-4 [04:50:14] how did rt-testing get into our salt? [04:50:21] also ancient history [04:51:24] bd808: i didn't see mediawiki04 down. [04:51:43] except supposedly all are down now [04:51:50] now we see another thing I don't love about salt [04:52:06] woot, all up [04:52:07] the answer to questions changes [04:52:27] because it is all about async results [04:52:30] it was just a timeout i'm sure [04:52:36] * bd808 nods [05:03:14] grrrrr [05:03:16] wtf [05:03:23] we reused instance IDs? [05:03:29] and changed the format of page names? [05:03:37] (was short hostname now is fqdn?) [05:03:56] https://wikitech.wikimedia.org/wiki/Nova_Resource:I-000005ba.eqiad.wmflabs vs. https://wikitech.wikimedia.org/wiki/Nova_Resource:I-000005ba [05:08:56] jeremyb: Probably pre/post eqiad being turned up I bet [05:10:34] bd808: yeah, but ewwww? id reuse?????? [05:10:52] bd808: salt should be all good now unless there's some instance ldap/wikitech doesn't know about [05:11:00] ec2 id which is probably realm dependent [05:11:10] jeremyb: w00t. Thanks for your hard work [05:24:22] what's soa-cache? [05:27:41] bd808: 3 hosts with no puppet nor salt in ldap. so not in manage.down/manage.status/manage.up [05:27:42] deployment-saio deployment-parsoidcache02 deployment-soa-cache01 [05:27:58] ldapsearch -x -b 'ou=hosts,dc=wikimedia,dc=org' '(&(!(puppetVar=puppetmaster=deployment-salt.eqiad.wmflabs))(puppetVar=instanceproject=deployment-prep))' associatedDomain puppetVar [05:28:06] ldapsearch -x -b 'ou=hosts,dc=wikimedia,dc=org' '(&(!(puppetVar=salt_master_override=deployment-salt.eqiad.wmflabs))(puppetVar=instanceproject=deployment-prep))' associatedDomain puppetVar [05:28:58] -saio is a testing box that ^d setup recently. The other 2 I don't know about. [05:30:43] well Roan recently killed parsoidcache01 and made parsoidcache02 [05:31:21] Ah. And he probably didn't follow the "new beta host instructions" when he did [05:31:47] It would be so nice to be able to configure some things per-project rather than per-host [05:32:47] soa-cache is may be varnish in front of the pdf renderer… guessing based on naming [05:33:03] or for mathoid [05:33:10] or both [05:39:19] right, i guessed that [05:41:42] ok, nacht [05:41:54] later [05:48:29] 3Wikimedia Labs / 3deployment-prep (beta): deployment-salt can't talk to itself, git deploy hangs - 10https://bugzilla.wikimedia.org/70868#c3 (10jeremyb) 5NEW>3RESO/FIX 2 prerequisites before booting salt-minion: * kill all the existing salt-minion/grain-ensure/salt-call/etc. procs * make sure /etc/salt... [06:21:29] 3Wikimedia / 3Continuous integration: Have Jenkins run a voting PHPUnit job for GettingStarted - 10https://bugzilla.wikimedia.org/70631#c3 (10Matthew Flaschen) It looks like the issue was in our extension (it was already running tests, presumably the core ones, but not our own). BTW, I made a minor change t... [07:37:44] 3Wikimedia Labs / 3deployment-prep (beta): Search is sometimes slow - 10https://bugzilla.wikimedia.org/70869#c4 (10Antoine "hashar" Musso) Created attachment 16480 --> https://bugzilla.wikimedia.org/attachment.cgi?id=16480&action=edit Elastic search instances load average [07:42:14] 3Wikimedia Labs / 3deployment-prep (beta): monitor unsigned salt keys - 10https://bugzilla.wikimedia.org/70862#c1 (10Antoine "hashar" Musso) Yuvi, I am not sure how familiar you are with diamond. Would it make sense to write a basic collector that list the rejected/unsigned keys on the salt master, send that... [07:44:59] 3Wikimedia Labs / 3deployment-prep (beta): Setup monitoring for Beta cluster - 10https://bugzilla.wikimedia.org/51497#c8 (10Antoine "hashar" Musso) Thank you Yuvi for the monitoring! Do we have a way to tweak the body of email notifications? I find them hard to read :-D [07:51:15] 3Wikimedia / 3Continuous integration: "Gate and submit" jobs take ages to run (30m+) - 10https://bugzilla.wikimedia.org/70256#c1 (10Antoine "hashar" Musso) 5NEW>3RESO/DUP That is an issue in the Jenkins gearman plugin which sometime forget about the jobs. Hence Zuul can no more trigger them and keep queu... [07:51:15] 3Wikimedia / 3Continuous integration: Jobs are sometime no more being triggered by Zuul / Jenkins - 10https://bugzilla.wikimedia.org/63760#c11 (10Antoine "hashar" Musso) *** Bug 70256 has been marked as a duplicate of this bug. *** [07:58:15] 3Wikimedia / 3Continuous integration: "Failed to archive artifacts" and "java.util.concurrent.ExecutionException" - 10https://bugzilla.wikimedia.org/70369#c1 (10Antoine "hashar" Musso) Created attachment 16481 --> https://bugzilla.wikimedia.org/attachment.cgi?id=16481&action=edit Console output for https:/... [08:07:44] 3Wikimedia / 3Continuous integration: "Failed to archive artifacts" and "java.util.concurrent.ExecutionException" - 10https://bugzilla.wikimedia.org/70369#c2 (10Antoine "hashar" Musso) 5NEW>3RESO/WOR I am not sure what happened. /log/ is where we have commands log to and it is deleted early on with: r... [08:10:14] 3Wikimedia Labs / 3deployment-prep (beta): deployment-salt can't talk to itself, git deploy hangs - 10https://bugzilla.wikimedia.org/70868#c4 (10Antoine "hashar" Musso) all are now salty except: * deployment-saio * deployment-parsoidcache02 * deployment-soa-cache01 Those instances haven't been migrated to t... [08:14:11] (03PS1) 10Hashar: mw-api-siteinfo: raise on http error [integration/jenkins] - 10https://gerrit.wikimedia.org/r/160616 (https://bugzilla.wikimedia.org/70695) [08:14:37] (03PS2) 10Hashar: mw-api-siteinfo: raise on http error [integration/jenkins] - 10https://gerrit.wikimedia.org/r/160616 (https://bugzilla.wikimedia.org/70695) [08:15:21] (03CR) 10Hashar: [C: 032] mw-api-siteinfo: raise on http error [integration/jenkins] - 10https://gerrit.wikimedia.org/r/160616 (https://bugzilla.wikimedia.org/70695) (owner: 10Hashar) [08:15:23] (03Merged) 10jenkins-bot: mw-api-siteinfo: raise on http error [integration/jenkins] - 10https://gerrit.wikimedia.org/r/160616 (https://bugzilla.wikimedia.org/70695) (owner: 10Hashar) [08:16:00] 3Wikimedia / 3Continuous integration: CI: mw-api-siteinfo.py should more clearly report site failure - 10https://bugzilla.wikimedia.org/70695#c8 (10Antoine "hashar" Musso) 5PATC>3RESO/FIX a:3spage Excellent finding S! I have copy pasted your one-liner fix and made you an Author: :] Thank you! [08:25:46] !sal [08:25:46] https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:25:07] (03CR) 10Zfilipin: "Let me know if you need help updating the commit." [ruby/api] - 10https://gerrit.wikimedia.org/r/159630 (https://bugzilla.wikimedia.org/70605) (owner: 10Damienkan) [09:31:45] 3Wikimedia Labs / 3deployment-prep (beta): wikidata beta (item pages, etc.) inaccessible with 503 errors - 10https://bugzilla.wikimedia.org/69708#c6 (10Aude) reproduced these issues in vagrant, which uses hhvm: Fatal error: Argument 1 passed to Wikibase\\ItemContent::__construct() must be an instance of Wik... [09:36:45] 3Wikimedia Labs / 3deployment-prep (beta): wikidata beta (item pages, etc.) inaccessible with 503 errors - 10https://bugzilla.wikimedia.org/69708#c7 (10Aude) i then restarted hhvm and the error is gone! before restarting, i did composer install of all of wikibase etc., which maybe exceeded some limit or such? [10:13:29] 3Wikimedia Labs / 3deployment-prep (beta): monitor unsigned salt keys - 10https://bugzilla.wikimedia.org/70862#c2 (10Yuvi Panda) Indeed, that seems ok to do. *Ideally* we would just do this in icinga instead of with diamond, but considering icinga status on labs I'd say go ahead with doing it in diamond. We... [10:13:41] (03PS1) 10Hashar: SemanticForms now pass jshint make it voting [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160623 (https://bugzilla.wikimedia.org/61630) [10:13:56] (03CR) 10Hashar: [C: 032] SemanticForms now pass jshint make it voting [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160623 (https://bugzilla.wikimedia.org/61630) (owner: 10Hashar) [10:14:04] (03Merged) 10jenkins-bot: SemanticForms now pass jshint make it voting [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160623 (https://bugzilla.wikimedia.org/61630) (owner: 10Hashar) [10:15:47] oh hashar your back,ish now? :P [10:16:29] 3Wikimedia Labs / 3deployment-prep (beta): monitor unsigned salt keys - 10https://bugzilla.wikimedia.org/70862#c3 (10Antoine "hashar" Musso) I already have too many things to complete which are long overdue. So I am unlikely to look at writing a diamond collector anytime soon. If you have some spare bandwid... [10:17:14] addshore: goood morning! [10:17:24] addshore: yeah had a 2 weeks parental leave. I came back yesterday [10:17:28] how is the wee one? :) [10:17:29] 3Wikimedia Labs / 3deployment-prep (beta): monitor unsigned salt keys - 10https://bugzilla.wikimedia.org/70862#c4 (10Yuvi Panda) Alright, I'll put it on my 'spare bandwidth TODO' list :) In the meantime, if anyone else wants to step in, please do! I'll be happy to help. [10:17:38] she sleeping just fine :] [10:17:42] and overall very quiet [10:17:44] 3Wikimedia Labs / 3deployment-prep (beta): monitor unsigned salt keys - 10https://bugzilla.wikimedia.org/70862 (10Yuvi Panda) [10:17:53] but I guess I am not going to work in the evening anymore [10:18:00] :D [10:22:35] (03PS8) 10Hashar: WIP kill jenkins slaves WIP [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/156455 [10:46:51] food time [11:17:24] (03PS9) 10Hashar: WIP kill jenkins slaves WIP [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/156455 [11:34:45] (03PS10) 10Hashar: Zuul cloner for MediaWiki extensions [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/156455 [11:39:19] (03PS1) 10Hashar: Update mw ext voting rule [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160631 [11:41:21] (03PS2) 10Hashar: Update mw ext voting rule [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160631 [11:43:07] (03CR) 10Hashar: [C: 032] "allll good" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160631 (owner: 10Hashar) [11:43:16] (03Merged) 10jenkins-bot: Update mw ext voting rule [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160631 (owner: 10Hashar) [12:00:14] (03PS1) 10KartikMistry: Add ContentTranslation qunit job [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160635 [12:00:23] (03CR) 10jenkins-bot: [V: 04-1] Add ContentTranslation qunit job [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160635 (owner: 10KartikMistry) [12:00:29] hashar: Help me in this ^ [12:00:31] :) [12:02:10] kart_: most probably the job is not deployed in Jenkins [12:02:40] hashar: yes. How to do that? [12:02:46] silly me. [12:03:31] aharoni: I am in the hangout [12:07:14] kart_: https://www.mediawiki.org/wiki/Continuous_integration/Tutorials/Adding_a_MediaWiki_extension that should cover it all [12:07:28] kart_: you need a change in integration/jenkins-job-builder-config [12:08:43] got it. [12:11:20] hashar: can you join aharoni and me in a pairing session? we need some jjb help [12:19:32] kart_: just add to add your extension to a long list of all the extensions, that should generate the needed jobs [12:19:34] zeljkof: busy writing doc :D [12:22:58] (03CR) 10Hashar: [C: 032] "Deployed. Will sort out the mess of REL / master / wmf branches with Zuul." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/156455 (owner: 10Hashar) [12:26:19] (03Merged) 10jenkins-bot: Zuul cloner for MediaWiki extensions [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/156455 (owner: 10Hashar) [12:29:37] (03PS1) 10KartikMistry: Add ContentTranslation Qunit test job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160638 [12:30:21] (03PS2) 10KartikMistry: Add ContentTranslation qunit job [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160635 [12:30:24] hashar_: ^ [12:30:37] (03CR) 10jenkins-bot: [V: 04-1] Add ContentTranslation qunit job [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160635 (owner: 10KartikMistry) [12:42:35] (03CR) 10Hashar: "All jobs created / refreshed \O/" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/156455 (owner: 10Hashar) [13:14:13] (03PS1) 10Hashar: Normalize some extensions layout [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160642 [13:14:42] (03CR) 10Hashar: [C: 032] Normalize some extensions layout [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160642 (owner: 10Hashar) [13:14:53] (03Merged) 10jenkins-bot: Normalize some extensions layout [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160642 (owner: 10Hashar) [13:55:58] (03PS2) 10KartikMistry: Add ContentTranslation Qunit test job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160638 [13:56:46] (03PS3) 10KartikMistry: Add ContentTranslation qunit job [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160635 [13:57:00] (03CR) 10jenkins-bot: [V: 04-1] Add ContentTranslation qunit job [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160635 (owner: 10KartikMistry) [14:27:34] (03PS1) 10Hashar: Test extensions with vendor and proper mw branch [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160656 [14:29:04] (03CR) 10Hashar: [C: 032] Test extensions with vendor and proper mw branch [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160656 (owner: 10Hashar) [14:29:31] (03Merged) 10jenkins-bot: Test extensions with vendor and proper mw branch [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160656 (owner: 10Hashar) [14:38:12] (03PS1) 10Hashar: Stop triggering mediawiki-core-lint for exts [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160658 [14:38:30] (03CR) 10Hashar: [C: 032] Stop triggering mediawiki-core-lint for exts [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160658 (owner: 10Hashar) [14:38:39] (03Merged) 10jenkins-bot: Stop triggering mediawiki-core-lint for exts [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160658 (owner: 10Hashar) [14:40:18] (03PS1) 10Hashar: fix deps expansions for zuul cloner [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160659 [14:40:30] (03CR) 10Hashar: [C: 032] fix deps expansions for zuul cloner [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160659 (owner: 10Hashar) [14:44:51] (03Merged) 10jenkins-bot: fix deps expansions for zuul cloner [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160659 (owner: 10Hashar) [14:47:14] manybubbles: around? [14:47:41] can we move pairing session tomorrow to 30 minutes earlier? [14:48:55] (03PS1) 10Hashar: Throttle extensions job: 1 max per node [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160662 [14:53:06] (03CR) 10Hashar: [C: 032] Throttle extensions job: 1 max per node [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160662 (owner: 10Hashar) [14:53:56] zeljkof: sure! [14:54:10] manybubbles: great, I have made the change in the calendar [14:54:13] thanks! [14:54:18] thank you! see you then! [14:56:35] (03Merged) 10jenkins-bot: Throttle extensions job: 1 max per node [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160662 (owner: 10Hashar) [15:03:07] (03PS1) 10Hashar: mwext prevent loading an empty extension [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160666 [15:03:44] (03CR) 10Hashar: [C: 032] mwext prevent loading an empty extension [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160666 (owner: 10Hashar) [15:05:47] (03PS3) 10KartikMistry: Add ContentTranslation Qunit test job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160638 [15:07:49] (03Merged) 10jenkins-bot: mwext prevent loading an empty extension [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160666 (owner: 10Hashar) [15:17:14] 3Wikimedia / 3Continuous integration: Have Jenkins run a voting PHPUnit job for GettingStarted - 10https://bugzilla.wikimedia.org/70631 (10Sam Smith) 5PATC>3RESO/FIX [15:39:55] hashar: you rechecked this and Jenkins still did not merge it? https://gerrit.wikimedia.org/r/#/c/160477/ [15:40:07] chrismcmahon: recheck does not cause a merge [15:40:10] it just run the test jobs [15:40:13] ah [15:40:13] ones need to revote +2 [15:41:40] hmm, I think I did that... [15:42:46] aha, I had to manually "Submit" the change. [15:44:02] Project beta-scap-eqiad build #21815: FAILURE in 0.58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21815/ [15:53:49] (03PS1) 10Hashar: Fix mw ext jobs throttling [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160676 [15:54:11] (03CR) 10Hashar: [C: 032] Fix mw ext jobs throttling [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160676 (owner: 10Hashar) [15:54:47] Yippee, build fixed! [15:54:47] Project beta-scap-eqiad build #21816: FIXED in 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21816/ [15:57:44] 3Wikimedia Labs / 3deployment-prep (beta): wikidata beta (item pages, etc.) inaccessible with 503 errors - 10https://bugzilla.wikimedia.org/69708#c8 (10Bryan Davis) I wonder if this is related to the hhvm cache issues that Ori has been looking into? I've only been following that via irc eavesdropping, but I... [15:57:55] (03Merged) 10jenkins-bot: Fix mw ext jobs throttling [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160676 (owner: 10Hashar) [16:00:33] coming coming [16:00:49] greg-g: joining in a minute or so sorry [16:01:12] cool [16:11:13] (03PS1) 10Yurik: Added Graph ext [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160680 [16:15:02] (03PS1) 10Dan-nl: Added extension dependencies for GWToolset testing [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160681 [16:15:51] (03CR) 10jenkins-bot: [V: 04-1] Added extension dependencies for GWToolset testing [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160681 (owner: 10Dan-nl) [16:17:56] (03PS2) 10Dan-nl: Added extension dependencies for GWToolset testing [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160681 [16:18:25] (03CR) 10Dan-nl: "* replaced tabs with spaces" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160681 (owner: 10Dan-nl) [16:18:30] (03CR) 10jenkins-bot: [V: 04-1] Added extension dependencies for GWToolset testing [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160681 (owner: 10Dan-nl) [16:20:46] (03PS3) 10Dan-nl: Added extension dependencies for GWToolset testing [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160681 [16:21:06] (03CR) 10Dan-nl: "* missed a space" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160681 (owner: 10Dan-nl) [16:34:44] 3Wikimedia Labs / 3deployment-prep (beta): Search is sometimes slow - 10https://bugzilla.wikimedia.org/70869#c5 (10Antoine "hashar" Musso) Chad / Nik are the best point to investigate ElasticSearch related issue. Maybe someone imported a bunch of articles on beta which caused a lot of indexing on ElasticSea... [16:36:16] !log Updated scap to 663f137 (Check php syntax with parallel `php -l`) [16:36:17] Logged the message, Master [16:37:43] bd808: why not have a logmsgbot here to !log automatically on git deploy? [16:37:59] 3Wikimedia Labs / 3deployment-prep (beta): Search is sometimes slow - 10https://bugzilla.wikimedia.org/70869#c6 (10Nik Everett) I can have a look at it soon - yeah. The Elasticsearch cluster in beta isn't designed for performance - just to be there and functional. [16:43:36] git deploy doesn't talk to logmsgbot :( [16:47:15] bd808|deploy: parallel php-l ? So we can drop php_parsekit I guess? [16:48:28] hashar: Yes! [16:48:43] bd808|deploy: awesome :] [16:50:34] https://integration.wikimedia.org/ci/job/mwext-Thanks-testextension/1/console <-- why is Echo/Flow not installed? those are specified as depedendencies... [16:52:22] legoktm: happened when I deployed the jobs overhaul [16:52:26] might need to be retriggered [16:52:28] ok [16:52:36] rechecking [16:52:42] else it is a bug [16:52:49] :-( [16:53:42] https://integration.wikimedia.org/ci/job/mwext-Thanks-testextension/2/console .. [16:55:45] legoktm: it fails with some fatal now :( [16:55:52] at least no more complains about Echo not being there [16:56:35] I am off for more IRL meeting [16:57:19] sweet, thanks :) [16:58:16] later hashar :) sleep well [16:58:38] greg-g: oh I am attending a board meeting tonight :D [16:58:48] rushing out! [16:58:52] see you tomorrow greg [16:58:54] :) [16:58:55] tah [16:58:56] hashar: CX-qunit changes are ok? [16:59:23] no hurry for it though! :) [16:59:59] * greg-g waves at kart_ [17:00:39] kart_: no clue. Please ping by email via bugzilla [17:00:53] kart_: I am off for the rest of the evening but can follow up tomorrow :D [17:01:11] hashar: via Gerrit :) [17:01:19] * kart_ waves back to greg-g [17:01:26] and off to bed :) [17:03:37] g'night [17:04:25] marxarelli: about that geolocation stuff, the bug in question was https://bugzilla.wikimedia.org/show_bug.cgi?id=57333 and you can see the sqlite profile we worked with still in the MF repo: https://git.wikimedia.org/tree/mediawiki%2Fextensions%2FMobileFrontend/1a4110629cfa0e9ce5e18b6fec4ff3d8f28a8d02/tests%2Fbrowser%2Ffeatures%2Fsupport [17:07:13] chrismcmahon: cool, thanks [17:07:26] chrismcmahon: do you know which feature it's used for? [17:11:18] marxarelli: test we tried to use it for was https://git.wikimedia.org/blob/mediawiki%2Fextensions%2FMobileFrontend/1a4110629cfa0e9ce5e18b6fec4ff3d8f28a8d02/tests%2Fbrowser%2Ffeatures%2Fnearby.feature [17:20:16] 3Wikimedia Labs / 3deployment-prep (beta): wikidata beta (item pages, etc.) inaccessible with 503 errors - 10https://bugzilla.wikimedia.org/69708#c9 (10Aude) @bryan I think quite certain it is related. this happens less often on test.wikidata, but I think because hhvm has higher limits in production. (JitAC... [18:01:33] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #709: SUCCESS in 31 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/709/ [18:05:18] Project browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs build #146: SUCCESS in 17 sec: https://integration.wikimedia.org/ci/job/browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs/146/ [18:16:53] Project browsertests-TwnMainPage-sandbox.translatewiki.net-linux-firefox-sauce build #130: SUCCESS in 12 min: https://integration.wikimedia.org/ci/job/browsertests-TwnMainPage-sandbox.translatewiki.net-linux-firefox-sauce/130/ [18:17:48] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #110: SUCCESS in 55 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/110/ [18:19:03] Project browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce build #47: STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce/47/ [18:23:23] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #166: SUCCESS in 4 min 16 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/166/ [18:25:14] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #11: SUCCESS in 7 min 26 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/11/ [18:27:17] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #11: STILL FAILING in 2 min 2 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/11/ [18:39:19] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #172: SUCCESS in 12 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/172/ [18:46:02] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce build #177: FAILURE in 6 min 42 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce/177/ [18:46:28] Project browsertests-Flow-test2.wikipedia.org-windows_8-internet_explorer-sauce build #152: STILL FAILING in 45 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-windows_8-internet_explorer-sauce/152/ [18:46:48] Project browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce build #83: SUCCESS in 47 sec: https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce/83/ [18:47:28] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #143: SUCCESS in 1 min 24 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/143/ [18:49:09] Project browsertests-UniversalLanguageSelector-sandbox.translatewiki.net-linux-firefox-sauce build #141: SUCCESS in 1 min 41 sec: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-sandbox.translatewiki.net-linux-firefox-sauce/141/ [19:04:03] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #224: STILL FAILING in 40 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/224/ [19:13:26] Yippee, build fixed! [19:13:27] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #63: FIXED in 9 min 22 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/63/ [19:31:59] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #9: STILL FAILING in 42 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/9/ [19:50:29] Project browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce build #189: SUCCESS in 1 hr 3 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce/189/ [19:50:38] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #202: FAILURE in 37 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/202/ [19:51:33] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #99: SUCCESS in 1 min 3 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/99/ [19:57:36] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #185: FAILURE in 6 min 56 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/185/ [20:00:50] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #143: SUCCESS in 3 min 8 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/143/ [20:02:00] * greg-g stares at https://bugzilla.wikimedia.org/show_bug.cgi?id=70778 ;) [20:10:16] Yippee, build fixed! [20:10:17] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #50: FIXED in 9 min 26 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/50/ [20:11:32] Project browsertests-Translate-meta.wikimedia.org-linux-firefox-sauce build #168: SUCCESS in 1 min 15 sec: https://integration.wikimedia.org/ci/job/browsertests-Translate-meta.wikimedia.org-linux-firefox-sauce/168/ [20:12:51] Yippee, build fixed! [20:12:51] Project browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce build #142: FIXED in 1 min 18 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce/142/ [20:13:43] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #110: SUCCESS in 52 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/110/ [20:34:08] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #277: SUCCESS in 1 hr 2 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/277/ [20:34:37] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #710: SUCCESS in 29 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/710/ [20:39:54] marxarelli|lunch: every time I've had a scrollIntoView problem in recent years the root cause has been a visibility or z-axis issue in the SUT, not in selenium. That said, Chrome and FF definitely treat the definition of "visible" differently [20:40:59] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #207: STILL FAILING in 49 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/207/ [20:42:02] chrismcmahon: so FF _does_ try to scroll but just not in this case for some reason? [20:42:33] marxarelli: FF would try to scroll if it believed that the element were visible *I think* [20:43:27] marxarelli: and this issue was reported fairly recently, it may be relevant https://code.google.com/p/selenium/issues/detail?id=7249 [20:43:52] I"m asking on #selenium right now, just fishing [20:45:35] marxarelli: Flow is pretty bizarre about what is and is not interactable at any given moment, could be a timing problem also. [20:46:36] chrismcmahon: huh, ok. could spagewmf try page.wait_until { element.visible? } you think? [21:23:33] Project browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce build #172: FAILURE in 48 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce/172/ [21:39:57] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #213: SUCCESS in 58 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/213/ [21:47:08] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #167: SUCCESS in 7 min 10 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/167/ [21:50:17] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #157: FAILURE in 41 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/157/ [21:54:16] Project browsertests-Flow-test2.wikipedia.org-linux-firefox-sauce build #154: STILL FAILING in 30 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-firefox-sauce/154/ [21:56:51] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #168: SUCCESS in 6 min 32 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/168/ [22:03:07] Project browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce build #46: STILL FAILING in 15 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce/46/ [22:04:38] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #10: SUCCESS in 7 min 47 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/10/ [22:05:09] Project UploadWizard-api-commons.wikimedia.org build #564: SUCCESS in 30 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.org/564/ [22:06:17] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #42: STILL FAILING in 1 min 7 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/42/ [22:09:56] Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #169: SUCCESS in 3 min 37 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/169/ [22:33:47] Project browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce build #155: STILL FAILING in 39 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce/155/ [22:36:39] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #168: SUCCESS in 2 min 51 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/168/ [22:58:05] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #164: SUCCESS in 21 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/164/ [22:59:21] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #111: SUCCESS in 1 min 15 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/111/ [23:05:32] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #21: STILL FAILING in 55 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/21/ [23:09:11] Project browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce build #186: SUCCESS in 1 hr 6 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce/186/