[00:01:22] werdna: watir api gives us a lot of stuff not available in pure selenium. the one I really rely on is specifying an element with two locators, e.g. "text: "Pick me", index: 5" or "class: generic_element, text: "I am special" [00:02:06] see y'all tomorrow... [01:26:21] 3Wikimedia / 3Continuous integration: [OPS] Jenkins: Package ruby-jsduck missing in Trusty - 10https://bugzilla.wikimedia.org/68257 (10jeremyb) [07:29:10] hello zeljkof [07:29:15] hi vikasyaligar [07:29:34] looks like beta labs are running good today :) [07:30:44] vikasyaligar: looks like there is still errors :( [07:30:44] https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox-sauce/LANGUAGE_SCREENSHOT_CODE=he,label=contintLabsSlave/72/consoleFull [07:32:05] zeljkof: yup ! looking into it :) [07:32:22] vikasyaligar: great [07:59:08] zeljkof: my firefox looks smaller[address bar, font and everything] when I run the browser tests, any idea why ? [07:59:26] zeljkof: something similar I found here => http://stackoverflow.com/questions/24101818/why-is-firefoxs-url-address-bar-so-small-when-firefox-is-opened-by-selenium [08:00:10] vikasyaligar: did you see the answer to that question? looks like it is selenium problem [08:00:47] zeljkof: It did not work on mine :( [08:01:07] vikasyaligar: what do you mean, "did not work"? [08:01:23] console output or it did not happen :) [08:02:05] I am still getting the small fonts and size. No change in firefox browser [08:05:14] you have to let me know what you did [08:05:22] you probably did not do everything needed [08:05:30] or there is a bug in the old version too [08:07:05] zeljkof: I tried out the solution => "pip install selenium==2.40.0" [08:07:24] vikasyaligar: oh noes :) [08:07:34] no no no [08:07:45] a little bit of history [08:07:45] zeljkof: is it for selenium in python :P [08:07:51] yes [08:07:58] ah ! I thought so [08:07:59] you can use selenium from several languages [08:08:03] we use it from ruby [08:08:12] so you have to change your gemfile [08:08:21] that is how we manage dependencies [08:08:38] vikasyaligar: see, as soon as you told me what you did, we have solved the problem :) [08:08:59] zeljkof: yup ! will be clear next time :) [08:09:27] that is a good practice to learn [08:09:42] if you know how to report the problem clearly, somebody will be able to help you [08:09:53] if you say "does not work", nobody can help you [08:11:48] zeljkof: yup lesson learnt :) [08:12:00] vikasyaligar: that is important :) [08:18:23] zeljkof: thank you :) [08:20:12] zeljkof: I changed the version of selenium-webdriver to 2.40 and now everything is working fine :) [08:21:10] vikasyaligar: great [08:30:38] vikasyaligar: we should switch language screenshots job from being run automatically twice a day to running only manually [08:30:45] do you know how to do that? [08:32:29] zeljkof: I think so ! let me try it out [08:32:39] vikasyaligar: great, let me know if you get stuck [08:59:31] zeljkof: the job is run twice because of triggers right ? [08:59:53] vikasyaligar: yes [09:01:29] what value should I overwrite it with ? i.e. => timed: false or timed: '' ? or there is any other way ? [09:03:08] zeljkof: any documentation available on it ? [09:03:17] vikasyaligar: http://ci.openstack.org/jenkins-job-builder/ [09:03:44] yup I am searching it in http://ci.openstack.org/jenkins-job-builder/triggers.html [09:08:44] vikasyaligar: you should probably just remove triggers section, that will leave manual only option [09:10:04] zeljkof: yup ! but that will affect other jobs also right ? [09:10:29] vikasyaligar: depending on how things are set up [09:12:03] zeljkof: I can see another job template using browsertests template => https://github.com/wikimedia/integration-jenkins-job-builder-config/blob/cloudbees/job_template.yaml#L7-L9 [09:12:24] won't that be affected ? [09:12:36] vikasyaligar: probably [09:12:42] do you know how to test it? [09:13:28] haha only thing I can think of it is to wait for a day :) [09:14:54] vikasyaligar: :) [09:15:00] there is a better way :) [09:15:15] ping hashar if he has the time to pair with you on jjb basics [09:15:46] or I can pair with you tomorrow or friday [09:16:03] going to deploy some stuff this morning [09:16:53] zeljkof: yup ! I am free tomorrow [Even today] :) [09:17:36] vikasyaligar: I have to finish something today [09:17:56] ping me tomorrow [09:18:05] zeljkof: ok sure thank you :) [11:11:45] (03PS1) 10Zfilipin: Fixed date for 0.1.4 release [ruby/api] - 10https://gerrit.wikimedia.org/r/148639 [12:48:12] (03CR) 10Hashar: "The job need to be manually refreshed though." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/148543 (owner: 10EBernhardson) [12:56:50] hashar: do you have a minute? [12:57:34] how come out of all open mw_api commits[1] only one[2] did not run ruby lint? [12:57:40] 1: https://gerrit.wikimedia.org/r/#/projects/mediawiki/ruby/api,dashboards/default:open [12:57:47] 2: https://gerrit.wikimedia.org/r/#/c/148639/ [12:57:59] because no ruby files were changed? [12:58:01] cause there was no .rb file edited? [12:58:10] the ruby lint jobs are only run when a .rb is changed [12:58:24] so you end up with a missing +1 from Jenkins hehe [12:58:26] hashar: thanks, makes sense, I did not know we perform a check [12:58:36] just force verify+2 and merge I guess [12:59:07] hashar: I am trying to add running unit tests for mw_api, so checking how it works now [12:59:49] you will need a job [12:59:54] maybe using rake ? [13:00:12] all that needs to happen is running "rspec" [13:00:22] we do not use rake [13:02:33] hashar: and how come gem build jobs do not run for mw_api and mw_selenium? [13:02:43] for example: https://gerrit.wikimedia.org/r/#/c/147527 [13:02:48] only ruby lint runs [13:03:02] I remember vaguely there was a gem build job [13:03:15] (03PS2) 10Zfilipin: WIP The first PdfHandler Selenium tests [selenium] - 10https://gerrit.wikimedia.org/r/144984 [13:03:23] (03PS3) 10Zfilipin: WIP better link to Sauce Labs job [selenium] - 10https://gerrit.wikimedia.org/r/137673 [13:03:29] (03PS2) 10Zfilipin: WIP The first GettingStarted test [selenium] - 10https://gerrit.wikimedia.org/r/144978 (https://bugzilla.wikimedia.org/52246) [13:06:51] zeljkof: have to look at zuul-config.git layout.yaml file:] [13:07:25] hashar: I am looking right now [13:07:46] I have to do some cleanup first, looks like I have a lot of repos cloned, have to sort them somehow [13:07:53] it is only on post merge [13:07:53] http://git.wikimedia.org/blob/integration%2Fzuul-config.git/6abffaf1608df4ca784de63be5ac9b808c771e86/layout.yaml#L1184 [13:08:24] so folks can apparently fetch it from https://integration.wikimedia.org/ci/job/mediawiki-selenium-gembuild/ for testing purposes hehe [13:08:39] we would need to change the pipelines to add gembuild [13:09:37] zeljkof: same goes for mediawiki/ruby/api I believe? [13:09:59] hashar: yes [13:10:24] having it after merge is good, but gerrit should scream if any patch set can not be built, right? [13:10:38] (03PS1) 10Hashar: Test gembuild for mw ruby gems [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/148651 [13:10:44] ^^ should do it [13:11:45] https://gerrit.wikimedia.org/r/#/c/148651/1/layout.yaml [13:12:03] check-voter runs jobs for anyone and vote +2 on success [13:12:12] check runs them for untrusted user and only vote +1 [13:12:28] test: only runs jobs for trusted people and vote +2 [13:13:05] (03CR) 10Hashar: [C: 032] Test gembuild for mw ruby gems [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/148651 (owner: 10Hashar) [13:13:10] (03Merged) 10jenkins-bot: Test gembuild for mw ruby gems [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/148651 (owner: 10Hashar) [13:19:56] (03CR) 10Hashar: "recheck" [selenium] - 10https://gerrit.wikimedia.org/r/147527 (owner: 10Dduvall) [13:20:28] zeljkof: ok https://gerrit.wikimedia.org/r/#/c/147527/ managed to trigger a run of gembuild [13:20:31] https://integration.wikimedia.org/ci/job/mediawiki-selenium-gembuild/124/console [13:20:32] :-D [13:21:08] hashar: great :) [13:21:25] do we have a ruby linter available, besides the basic syntax check? [13:22:05] non voting for now [13:22:20] but something to alert when coding conventions are broken [13:22:20] https://www.mediawiki.org/wiki/Manual:Coding_conventions/Ruby [13:27:17] if you find a ruby linter sure :-] [13:27:20] we can add it [13:29:21] hashar: will take a look [13:29:24] there has to be one [13:29:39] is there a reason that I have integration/jenkins repo cloned on my machine? [13:29:49] is there something that I could care about? [13:29:56] I do not remember ever working on it [13:30:06] maybe I have cloned it by mistake while looking for something else [13:31:56] zeljkof: integration/jenkins holds a bunch of shell scripts and utilities that are made available on all jenkins slave [13:32:22] hashar: hm, then maybe I have really done something with it [13:32:24] let me check [13:33:06] gotta get out to draw some architecture schema [13:33:32] hashar: have fun :) [13:46:09] pen & paper for the win [15:19:27] Beta breakage is me and _joe_ upgrading hhvm [15:27:48] hhvm upgrade went a bit sideways. _joe_ is working on a fix [15:28:01] Sorry for breaking the world first thing in my morning [15:45:50] Still working on the hhvm in beta. One server is reverted, the other not. _joe_ will fix one way or the other by 16:00Z [15:59:57] bd808: so how bad off is beta? e.g. should I expect editing to work there at all? [16:00:47] varnish seems to be mostly toast? [16:00:59] chrismcmahon: I think it's hit and miss depending on which backend varnish routes your request to. Let me check with _joe_ to see where he's at. [16:01:23] I just got the 503 but MF was working some just before that [16:02:15] chrismcmahon: _joe_ says he's almost got a new build to test to fix. [16:19:38] chrismcmahon: Should be all better now. [16:19:46] Latest greatest hhvm 3.1+20140723-1+wmf1 running [16:29:06] bd808: I think is better, thanks [16:31:42] Sorry for the breakage. I should have tested the upgrade on some other host first. I was just so excited for a better hhvm! [16:32:27] bd808: yeah, nbd. we've talked about creating more test envs, this is a good case for having e.g. a beta2 env. [16:40:50] 3Wikimedia / 3Continuous integration: Set up automatic builds for extensions ported to HHVM - 10https://bugzilla.wikimedia.org/63120#c6 (10Antoine "hashar" Musso) Also apt.wikimedia.org is missing the hhvm-dev which I believe provide hphpize. Should be added to modules/contint/manifests/packages.pp as well. [16:45:24] 3Wikimedia / 3Continuous integration: Set up automatic builds for extensions ported to HHVM - 10https://bugzilla.wikimedia.org/63120#c7 (10Antoine "hashar" Musso) (In reply to Antoine "hashar" Musso from comment #6) > Also apt.wikimedia.org is missing the hhvm-dev which I believe provide > hphpize. > > Shou... [16:59:33] never seen this before: [16:59:37] Due to heavy load on the server, connections may be temporarily [16:59:37] 16:56:05 blocked from locations that fetch an unusually high number of [16:59:37] 16:56:05 pages. If you've just been heavily browsing, go get a cup of coffee [16:59:39] 16:56:05 and come back and reload in a minute. [17:00:47] chrismcmahon: been popping up quite a bit recently on the beta cluster :/ [17:04:26] bd808: you know where that msg comes from? ^^ [17:07:45] JohnLewis: I've been the heaviest user of beta labs for over two years and I have literally never seen that message before. [17:08:40] I've only started seeing it fairly recently, 2/3 days? [17:09:02] chrismcmahon: Yes. it's from the apaches when hhvm dies. [17:09:41] bd808: aha. hhvm is dying then. [17:11:01] chrismcmahon: yes. on and off. You should poke ori for some info about what he's doing to debug and how we can streamline the bug reporting process. [17:53:42] chrismcmahon: I poked ori about the beta crashes. It looks like we are still having some issues with luasandbox -- https://bugzilla.wikimedia.org/show_bug.cgi?id=68413 [17:54:11] bd808: where is this logged, out of curiosity? [17:54:52] chrismcmahon: hhvm puts crash reports in /tmp on the deployment-mediawiki01 and deployment-mediawiki02 servers. [17:55:07] named like stacktrace.3498.log [17:56:22] yurgh, ok [18:22:05] 3Wikimedia / 3Continuous integration: Jenkins: label slaves with their ubuntu version - 10https://bugzilla.wikimedia.org/68340#c1 (10Antoine "hashar" Musso) I have added the label UbuntuPrecise on: gallium lanthanum integration-slave1001.eqiad.wmflabs integration-slave1002.eqiad.wmflabs integration-sla... [18:37:21] (03CR) 10Cmcmahon: [C: 032] Fixed date for 0.1.4 release [ruby/api] - 10https://gerrit.wikimedia.org/r/148639 (owner: 10Zfilipin) [18:37:56] (03CR) 10Cmcmahon: [C: 032] Added alumni section to readme file [selenium] - 10https://gerrit.wikimedia.org/r/148388 (owner: 10Zfilipin) [18:38:27] (03CR) 10Cmcmahon: [C: 032] json gem is not used in tests [ruby/api] - 10https://gerrit.wikimedia.org/r/148085 (owner: 10Zfilipin) [19:01:42] (03Merged) 10jenkins-bot: Fixed date for 0.1.4 release [ruby/api] - 10https://gerrit.wikimedia.org/r/148639 (owner: 10Zfilipin) [19:01:44] (03Merged) 10jenkins-bot: Added alumni section to readme file [selenium] - 10https://gerrit.wikimedia.org/r/148388 (owner: 10Zfilipin) [19:25:46] (03PS1) 10Ori.livneh: mod_proxy_balancer configs [operations/apache-config] (betacluster) - 10https://gerrit.wikimedia.org/r/148741 [19:30:23] 3Wikimedia / 3Quality Assurance: don't run screenshot tests in normal VE build - 10https://bugzilla.wikimedia.org/68467 (10Chris McMahon) 3NEW p:3Unprio s:3normal a:3None Screenshot tests have their own build, we do not want them running in the regular VE build: https://integration.wikimedia.org/ci/v... [19:39:43] (03PS1) 10Ori.livneh: Add retry=0 to ProxyPass directives [operations/apache-config] (betacluster) - 10https://gerrit.wikimedia.org/r/148743 [19:40:26] (03CR) 10Ori.livneh: [C: 032] Add retry=0 to ProxyPass directives [operations/apache-config] (betacluster) - 10https://gerrit.wikimedia.org/r/148743 (owner: 10Ori.livneh) [19:40:28] (03Merged) 10jenkins-bot: Add retry=0 to ProxyPass directives [operations/apache-config] (betacluster) - 10https://gerrit.wikimedia.org/r/148743 (owner: 10Ori.livneh) [20:33:36] chrismcmahon: Just waiting on jenkins for https://gerrit.wikimedia.org/r/#/c/148754/ to hit beta. It should help with the hhvm crashes. [20:33:56] It will have a side effect of making lua a bit slower. but slower is better than crashy [20:34:58] bd808: I'm getting weird failures from image uploads, from editing in Mobile, all with no diagnostics and no fatals. [20:35:19] That smells like hhvm. [20:36:51] chrismcmahon: As soon as this config change rolls out I'll archive the crash logs and then keep watch for a bit. [20:42:49] bd808: idle question, what software have we actually compiled to hhvm and what is still PHP being interpreted? obviously it's central stuff, but I can't imagine all the extensions on beta are all hhvm. [20:44:25] chrismcmahon: hhvm is an interpreter like zend php. So when we are using it we are running everything via hhvm [20:45:28] bd808: ah, OK, I was confused, I thought individual executables had to be compiled separately for HHVM [20:46:32] hhvm's predecessor hphp worked like that. But hhvm is much more like the php5 you are used to. [20:46:50] It can have a pre-compiled cache of byte code, but we aren't doing that yet [20:47:14] That's some of the stuff you've probably heard Greg ask me about in the releng meetings [20:47:52] Ok. config is updated and I archived the old crash logs. [20:47:57] No new ones yet [20:48:54] bd808: lemme kick off a couple of builds, see if I can make it crash some more [20:49:25] Sounds like a plan [20:50:14] bd808: got builds running now for Core, UW, VE, MF, and your update was in the middle of one for Flow [20:51:19] bd808: the lack of passing-status-up-to-the-ui (or even to the logs) in our features is awful. [20:52:32] chrismcmahon: logging is hard apparently. I'll keep trying to find ways to make it easier. [21:00:06] bd808: UW problem, been seeing this all day: https://saucelabs.com/tests/6f1d825d7ca84843b19d4427f8997de9 [21:01:14] stash failure. [21:01:33] There are a zillion open bugs (old bugs) about that. [21:02:07] I was supposed to figure out what caused it ... 11 months ago... but never did. [21:02:26] It's on the multimedia team's plate now. [21:03:04] Maybe gi11es or tgr wold want to poke around in beta and see if they can find clues. [21:04:49] bd808: only started happening yesterday: https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/ [21:05:07] bd808: or maybe today [21:05:25] chrismcmahon: The good news is that there are no new hhvm crash dumps in the last 20 minutes. [21:06:07] jdlrobson: "undefined method `new_topic_body_element'" never got merged to master? [21:06:13] oops ww [21:30:07] bd808: core tests back to passing: https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/ [21:30:17] w00t [21:30:38] 3Wikimedia / 3Continuous integration: Zuul: scale merge operations (tracking) - 10https://bugzilla.wikimedia.org/68480 (10Antoine "hashar" Musso) 3NEW p:3Unprio s:3normal a:3None Zuul merge operations are slowing down Zuul processing. git fetch is too slow when there are thousands and thousands of r... [21:31:06] 3Wikimedia / 3Continuous integration: Zuul: scale merge operations (tracking) - 10https://bugzilla.wikimedia.org/68480 (10Antoine "hashar" Musso) [21:31:08] 3Wikimedia / 3Continuous integration: Zuul repositories have too many refs causing slow updates - 10https://bugzilla.wikimedia.org/68481 (10Antoine "hashar" Musso) 3NEW p:3Unprio s:3normal a:3None Zuul merge operations are quite slow. The reason is that fetches from Gerrit are painfully slow for some... [21:31:24] bd808: I really want to stop this bullshit: Error loading data from server: readonly: The wiki is currently in read-only mode. [21:32:08] We could stop doing db updates.... other than that I'm not sure how to fix [21:32:31] Maybe complain to Sean Pringle that the beta db servers need some attention [21:32:35] 3Wikimedia / 3Continuous integration: Zuul: scale merge operations (tracking) - 10https://bugzilla.wikimedia.org/68480 (10Antoine "hashar" Musso) [21:32:39] 3Wikimedia / 3Continuous integration: Zuul: setup a second merger on lanthanum - 10https://bugzilla.wikimedia.org/68482 (10Antoine "hashar" Musso) 3NEW p:3Unprio s:3normal a:3Antoine "hashar" Musso Zuul merge operations are only done on gallium. We can scale it up a bit by having another server to r... [21:32:41] They really shouldn't lag as easily as they do [21:32:49] 3Wikimedia / 3Continuous integration: Zuul: setup a second merger on lanthanum - 10https://bugzilla.wikimedia.org/68482 (10Antoine "hashar" Musso) p:5Unprio>3Normal s:5normal>3enhanc [21:33:05] 3Wikimedia / 3Continuous integration: Zuul repositories have too many refs causing slow updates - 10https://bugzilla.wikimedia.org/68481 (10Antoine "hashar" Musso) p:5Unprio>3Normal s:5normal>3enhanc [21:34:08] bd808: again, this is a fairly recent development [21:35:22] chrismcmahon: https://github.com/wikimedia/operations-mediawiki-config/commit/38990c671fd3b8d15f31a7c819e7bdd52ecef3ef [21:35:26] 3 months ago [21:37:00] thanks bd808, I never found that one :-) [21:37:25] I have a strange memory for things [21:37:53] But by definition there can't be slave lag without slaves [21:38:07] So the problem is mostly that the salve lags too easily [21:38:09] *slave [21:38:25] Maybe a tuning problem on the master or the salve or both [21:39:07] bd808: this is my old bug for that, updated https://bugzilla.wikimedia.org/show_bug.cgi?id=65486 [22:01:35] 3Wikimedia / 3Continuous integration: Zuul repositories have too many refs causing slow updates - 10https://bugzilla.wikimedia.org/68481#c1 (10Antoine "hashar" Musso) Example of an operation that took 1m20s: 2014-07-23 21:59:22,755 DEBUG zuul.Repo: Resetting repository /srv/ssd/zuul/git/mediawiki/core 2014-... [22:05:07] 3Wikimedia / 3Continuous integration: Zuul repositories have too many refs causing slow updates - 10https://bugzilla.wikimedia.org/68481#c2 (10Antoine "hashar" Musso) 5NEW>3ASSI p:5Normal>3Highes a:3Antoine "hashar" Musso That cause a bunch of issues. Will get a script to clean up obsolete referenc...