[00:05:30] Project UploadWizard-commons.wikimedia.beta.wmflabs.org build #75: SUCCESS in 27 sec: https://wmf.ci.cloudbees.com/job/UploadWizard-commons.wikimedia.beta.wmflabs.org/75/ [00:22:28] Project UploadWizard-commons.wikimedia.org build #78: SUCCESS in 24 sec: https://wmf.ci.cloudbees.com/job/UploadWizard-commons.wikimedia.org/78/ [00:43:19] JeffHall: https://gerrit.wikimedia.org/r/#/c/114114/1/modules/ve-mw/test/browser/features/support/pages/visual_editor_page.rb <-- Oh dear, maybe I should have seen that coming. [00:44:27] James_F: Yeah, that is a weird one - hard to find an RSpec matcher that works for menu items! [00:49:17] JeffHall: Does that make the tests magically pass? It looks totally sane to me; tempted to +2 it… [00:53:15] James_F: It works locally on my MacBook, but CloudBees sometimes has it's own ideas! In any case, that change can't make the browser tests any more broken than they are right now! [00:55:50] JeffHall: You've convinced me. :-) [00:57:29] James_F: Thanks! I'll kick off a CloudBees job with that change shortly to see if it turns rad dots into green dots [00:57:50] * James_F nods. [00:57:56] Or even red dots. :-) [01:56:40] (03PS1) 10Jhall: Update VisualEditor jobs to sync with new env variable name. [selenium] - 10https://gerrit.wikimedia.org/r/114121 [02:09:26] James_F: Bummer - VisualEditor browser tests run via CloudBees are going to fail until we can get the shared CloudBees template updated, but I can't do that without breaking browser tests in other WMF repos [02:09:44] JeffHall: Great. :-( [02:10:40] James_F: I sent an e-mail to the QA mailing list - I'm not sure how to proceed without causing disruption, so I'll see if Chris and/or Zeljko can advise when they're online tomorrow morning [02:14:27] * James_F nods. [02:14:30] JeffHall: Oh well. [11:04:28] hashar: have a minute for jjb question? [11:05:47] I am reading this http://ci.openstack.org/jenkins-job-builder/configuration.html [11:06:26] and I am confused when to use a template vs macro [11:07:57] in short, I have two upload wizard config files [11:07:57] https://github.com/zeljkofilipin/jenkins-job-builder-config [11:08:19] and I am trying to refactor them to one file, since they are really similar, just a couple of lines are different [11:12:25] do you know how sacred food is in France? :-D [11:12:27] it is noon! [11:12:34] zeljkof: hello [11:12:45] hashar: oops :) [11:13:04] no rush, finish your snails and cheese :) [11:13:16] macro are snippets [11:13:33] they can be publishers, wrappers or builders [11:13:38] so it is merely like a shell script [11:13:56] job templates use several publishers/wrappers/builders (which can be macro) [11:14:10] and the job-templates takes parameters passed by the project or jobs groups [11:14:19] hashar: I think I am starting to understand, I will test my changes and push the code to github, so we can comment on my changes [11:14:51] I am looking at the two jobs [11:14:57] anyway, the change I make should result in the same generated xml files, so it is easy to test locally [11:15:08] do not tell me what to do, I have to figure out myself :) [11:15:18] hashar: I will ping you if I get stuck [11:16:34] so for example you have a publisher which send emails to some qa folks [11:16:37] you can make that a macro [11:16:48] something like: [11:17:26] publisher: [11:17:26] - name: 'mail-qa-folks' [11:17:26] - email: [11:17:26] recipients: foo bar blaz [11:17:31] then you can use that macro with: [11:17:42] publishers: [11:17:43] - mail-qa-folks [11:17:54] jjb would know that mail-qa-folks should be expanded [11:17:54] :-D [11:18:09] or you can use a job-template [11:18:24] and pass the targeted domain [11:18:42] - job-template: [11:18:42] name: UploadWizard-{targetdomain}' [11:18:59] then define a project that has different values for targetdomain [11:19:53] - project: [11:19:53] name: uploadwizard [11:19:53] targetdomain: [11:19:53] - commons.wikimedia.beta.wmflabs.org [11:19:53] - commons.wikimedia.org [11:19:54] jobs: [11:19:54] - UploadWizard-{targetdomain}' [11:20:00] and then you don't even need to use macro [11:20:15] hashar: how do I connect template and project? [11:20:20] example: http://etherpad.wikimedia.org/p/58040 [11:20:32] what to do instead of name: project-name [11:21:26] I think I got it [11:21:32] see the pad [11:21:35] testing locall [11:21:36] locally [11:22:17] hashar: no, fail [11:22:21] Failed to find suitable template named '{name}-commons.wikimedia.org' [11:22:49] -:] [11:22:59] commons.wikimedia.org is the part which is varying [11:23:23] you can name the template: '{project}-spoketest-{targetdomain}' [11:23:28] then: [11:23:47] - project: [11:23:47] name: UploadWizard [11:23:47] ... aka list the target domains and invoke the template '{project}-spoketest-{targetdomain}' [11:25:09] hashar: I think I got it [11:25:13] http://etherpad.wikimedia.org/p/58040 [11:25:15] testing... [11:26:06] yeah! :) [11:27:08] yeah [11:27:09] :D [11:28:04] you might want to add 'smoketest' in it [11:28:05] ok, now to figure out how do change things that are not the same [11:28:15] so [11:28:22] the project will be realiazed [11:28:34] and will realize the job-template by passing different values for 'site' [11:28:44] in the job-template you can expand site using: {site} [11:28:53] ex: [11:29:34] - shell: | [11:29:34] export API_URL="https://{site}/w/api.php" [11:30:33] excellent! :) [11:31:04] and you should replace 'site' by 'host' :-] [11:31:20] I found out site to be very ambiguous most of the time [11:34:27] ok [11:35:18] updated code and the pad [11:35:56] how can I have different value for MEDIAWIKI_PASSWORD_VARIABLE? [11:38:37] testing... reading docs... [11:38:52] updated pad with failed experiment... [11:38:59] how do you ship the password usually ? [11:39:10] it is in that env variable [11:39:17] ahh [11:39:26] and we have different env variables for each site [11:39:27] so the project now has two variables [11:39:33] yes [11:39:35] host and mediawiki_password_variable [11:39:40] yes [11:39:47] it invokes the templates with all possibilities [11:39:52] (a cartesian product) [11:39:58] aka there will be FOUR invocations [11:40:10] since the job template name is 'UploadWizard-{host}' [11:40:13] I just need 2 [11:40:17] the job name only vary based on host [11:40:22] so that will generate only two jobs [11:40:38] thohgheach job will be generated twice with different values of mediawiki_password_variable [11:40:50] in the end, both jobs will have for mediawiki_password_variable: the last value MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG :D [11:41:07] ok, not what I wanted :) [11:41:20] the only variance is the hostname [11:41:26] so based on the hostname given [11:41:41] you want to forge the target URL as well as the env variable [11:41:57] I would rename the env variable [11:42:11] i.e. if I am given domain : france.fr [11:42:24] I would expect the password to be in MEDIAWIKI_PASSWORD_FRANCE_FR [11:42:40] you can transform france.fr to FRANCE_FR using some shell hack :] [11:44:29] ok, not what I wanted [11:44:41] tr is your friend :] [11:45:47] echo 'france.fr' |tr '[:lower:].' '[:upper:]_' [11:45:47] FRANCE_FR [11:46:40] ENV_HOST_PREFIX= `echo '{host}' |tr '[:lower:].' '[:upper:]_'` [11:46:44] something like that [11:47:26] I am heading out for lunch [11:47:27] :-D [11:47:35] hashar: see you later [11:47:47] I will be here, crying and reading "man tr" :) [12:56:53] back [12:57:01] hashar: me too [12:57:08] I gave up on fixing the template [12:57:16] pairing with physikerwelt for texvc/math extension [12:57:17] hehe [12:57:33] paste the template in the ether pad http://etherpad.wikimedia.org/p/58040 [12:57:38] we can't get a solution somehow :-] [12:57:39] hashar: will get back to it later [12:59:37] hashar: this is what I have so far [13:03:17] http://etherpad.wikimedia.org/p/58040 [13:05:24] sounds good [13:05:37] but both are going to use MEDIAWIKI_PASSWORD_SELENIUM_USER_WIKIPEDIA_ORG [13:05:59] so based on the {host} variable, should get the password from different en variable [13:06:06] could be a case / switch [13:08:57] hashar: I do not have the time to play with that now :) [13:09:01] have to port other jobs [13:09:10] will play with templates once I have all jobs in jjb [13:09:32] well you are very close [13:09:42] simply have to read a different variable based on the {site} [13:12:21] hashar: no, it is actually a bit tricky [13:12:57] the way env variables are named, there is no easy way to translate from job name [13:12:57] I will have to think about it [13:12:57] possibly rename env variables [13:12:57] but later [13:12:57] here it is http://paste.debian.net/plain/82868 [13:12:57] easy :-] [13:14:20] hashar: thanks [13:14:23] but later :) [14:29:35] hashar: ok, looks like VE jobs are moved to JJB [14:29:45] checking for the first one to finish https://wmf.ci.cloudbees.com/job/VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox/339/console [14:29:52] but looking good [14:30:40] now trying to create a simple template for these two jobs [14:31:10] https://github.com/zeljkofilipin/jenkins-job-builder-config/blob/master/VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome.yaml [14:31:14] https://github.com/zeljkofilipin/jenkins-job-builder-config/blob/master/VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox.yaml [14:31:35] the only difference are these two lines [14:31:42] export BROWSER=firefox or export BROWSER=chrome [14:32:06] and export VERSION=32 or export VERSION=27 [14:59:57] (03CR) 10Cmcmahon: [C: 032] Wait for login process to complete [selenium] - 10https://gerrit.wikimedia.org/r/114096 (owner: 10Cmcmahon) [15:20:35] (03PS1) 10Zfilipin: Added links for ZeroRatedMobileAccess repository [selenium] - 10https://gerrit.wikimedia.org/r/114153 [15:39:22] Another day another channel [15:39:28] :-] [15:39:34] so yeah beta [15:39:59] the main work machine is deployment-bastion.pmtpa.wmflabs [15:40:03] that is more or less fenari / tin [15:40:15] Right. [15:40:20] the code is deployed by a jenkins job every 6 minutes or so (some details at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/How_code_is_updated ) [15:40:48] it is basically a git pull of mediawiki/extensions.git to a project shared directory under /data/project [15:41:40] I don't think I want to mess with the "real" code for beta. Especially since the shared storage there makes scap irrelevant. [15:41:50] to be more precise that is in /data/project/apache/common-local/php-master/extensions [15:41:56] the Apache document root is set to that directory [15:42:17] so whenever git pull has finished, code is used by the application servers and jobrunners [15:42:47] back in january 2013, Ryan attempted a sprint to have git-deployed used in production and he used beta to play with [15:44:18] so one would use git-deploy on deployment-bastion and that was pushing the files to different hosts in /srv/deployment/mediawiki/beta0 [15:44:23] that should still be around [15:44:59] using dsh, I am not sure how one can ssh from deployment-bastion to the other machines [15:45:05] might need a key generated locally on the instances [15:45:36] That would probably be safer than key forwarding in labs [15:46:33] What puppet master do the beta instances use? Or better said how can I mess with puppet config there? [15:46:52] Do you have a local puppet master for the project? [15:46:53] all instances use the regular puppet master (virt1000) [15:47:01] and hence the operations/puppet production branch [15:47:21] andrewboggot created a local puppet master [15:47:29] deployment-prep-master.pmtpa.wmflabs [15:47:37] and a client: deployment-prep-puppetclient.pmtpa.wmflabs [15:47:48] havent looked at them yet though :( [15:47:56] * bd808 nods [15:48:15] potentially you can hack stuff on the master [15:48:15] * bd808 is having ssh-agent woes [15:48:19] and see on the client what happens [15:48:28] I dont forward my keys to labs :-] [15:48:46] It's my local agent that is flipping out at the moment [15:50:34] bd808: the idea would be to test scappy isn't it ? [15:50:55] I am not sure puppet is up to date on deployment-prep-master , the agent probably got disabled [15:50:58] hashar: yes. Which is "scap" now :) [15:51:00] and operations/puppet might not be updated [15:55:06] Ok. What should I be careful not to break on the beta hosts? everything? :) [15:56:29] more or less :-] [15:56:38] the *cache* instances are varnishes [15:56:46] we have four of them (upload, bits, mobile, text) [15:56:57] they point to apache32 and apache33 which are hosting MediaWiki [15:57:08] we have jobrunner and video scaler [15:57:15] Something is seriously messed up with my labs access. I can ssh directly to bastion[123].pmtpa.wmflabs but proxying through to any internal labs instance is failing. [15:57:30] ah I am not alone :] [15:57:41] noticed I can't get access to deployment-bastion [15:57:50] I have pinged andrew about it in #wikimedia-labs [15:58:10] Oh. hmm.. I know that the nfs servers were having trouble yesterday. [15:58:30] the deployment-prep /home and /data/project are using nfs [15:58:32] or should [16:05:21] bd808: more random link for you: pinterest deployment http://engineering.pinterest.com/post/70621633046/deploying-software-at-pinterest [16:05:31] they do Jenkins [16:05:39] and Zookeeper to manage nodes [16:05:55] + looking toward bittorrent to push code hehe [16:05:57] Neat. I'll read up. [16:06:43] you can do crazy things with Jenkins : https://wiki.jenkins-ci.org/download/attachments/54723106/bpp1.png [16:07:22] I had some pretty complicated test pipelines at $DAYJOB-1. [16:07:37] I am sure we could do something neat [16:09:12] We had nightly build and validation jobs that did unit test + tarball build + deploy + integration test across a ~15 project stack [16:09:36] They worked "most" of the time. [16:09:53] About every 3rd jenkins upgrade would break it all somehow [16:13:03] ahah [16:13:18] we want to do tar ball -> tests -> release for mediawiki/core [16:13:26] havent worked much on it [16:13:47] openstack is a bit crazy, they run the tests from all their projects to make sure there is no regression [16:14:05] then maintain a git repository which tracks the commit of each projects that have been tested [16:14:28] so you can in theory continuously deploy projects using those versions [16:18:08] Setting up build and validation chain for mediawiki/core should be doable. I don't know what goes into the the official tarball step though. [16:19:05] We deployed to our integration test servers with mcollective which is roughly the same thing as salt [16:19:49] niceee [16:19:58] I had a script in a jenkins job that would send the commands out to tell the integration nodes to fetch the latest stable tarball from the appropriate project and and then run the install process [16:20:04] gotta work on setting up tests for the tar ball this quarter [16:20:17] will attempt to get all unit tests first (core + extensions shipped with the tarball) [16:20:22] then probably get browsertests as well [16:20:46] if all is fine, one can push the tar ball to http://releases.wikimedia.org/mediawiki [16:21:06] (03CR) 10KartikMistry: [C: 031] "LGTM." [selenium] - 10https://gerrit.wikimedia.org/r/114153 (owner: 10Zfilipin) [16:35:37] hashar: I'll look around in beta some more this afternoon and see if I can come up with a list of things I'd do to setup scap for testing. [16:35:57] bd808: feel free to create some more instances as a playground / sandbox [16:36:07] if you dont mess with /data/project/apache you should be fine [16:36:15] That might be the easiest thing to do (new instances) [16:37:27] deployment-scap + deployment-scap-slave[1…N] or something [16:38:11] This hideous patch makes testing in a vagrant image possible https://gist.github.com/bd808/9079567 [16:41:36] yeah scap1 scap2 scap3 .. :-] [16:42:09] I thought some configuration could be passed to scap [16:42:15] scap --conf ~/myscap.conf [16:42:20] which would gives all paths [16:42:31] defaulting to /usr/local/etc/deployment.sh or whatever the name is [16:42:49] that could be useful for integration tests, we would want all path to be in some venv [16:43:02] hashar: yes and no. The flag is there but it doesn't propagate settings to the downstream hosts yet [16:43:29] It's on my list of "stuff to add" but hasn't hit the top yet [16:43:30] you got to push the conf file as well ? [16:43:47] RIght now puppet makes the conf file on each node [16:44:04] That's the existing design that we haven't tried to change yet [16:44:58] So I can give a different config file to the host that is running `scap` but the nodes that run `scap-1` via ssh/dsh don't get the same settings… yet [16:58:56] (03CR) 10Krinkle: [C: 031] "Looks good, maybe omit "-php" from the macro name though." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/113958 (owner: 10Hashar) [17:00:57] (03PS2) 10Hashar: Add json-lint.php command to -jslint jobs [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/113958 [17:02:11] (03PS3) 10Hashar: Add json-lint.php command to -jslint jobs [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/113958 [17:04:56] (03CR) 10Krinkle: "fixme: Inconsistent coding style." (033 comments) [integration/jenkins] - 10https://gerrit.wikimedia.org/r/107163 (owner: 10Hashar) [17:05:16] (03CR) 10Krinkle: "(phpcs maybe :-), if it can catch that )" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/107163 (owner: 10Hashar) [17:08:08] (03PS1) 10Hashar: json-lint: fixing whitespaces [integration/jenkins] - 10https://gerrit.wikimedia.org/r/114161 [17:08:16] (03CR) 10Hashar: "Thanks, done with https://gerrit.wikimedia.org/r/114161" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/107163 (owner: 10Hashar) [17:10:18] (03PS2) 10Krinkle: json-lint: fixing whitespaces [integration/jenkins] - 10https://gerrit.wikimedia.org/r/114161 (owner: 10Hashar) [17:10:26] (03CR) 10Krinkle: [C: 031] "2 more spaces." [integration/jenkins] - 10https://gerrit.wikimedia.org/r/114161 (owner: 10Hashar) [17:47:20] see you tomorrow [19:09:15] (03PS4) 10MarkTraceur: Grep for jsduck warnings and exit nonzero for them [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/113033 [19:09:23] (03CR) 10MarkTraceur: [C: 032] "Effing fast forward commits" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/113033 (owner: 10MarkTraceur) [19:09:52] (03Merged) 10jenkins-bot: Grep for jsduck warnings and exit nonzero for them [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/113033 (owner: 10MarkTraceur) [20:10:49] (03CR) 10Jhall: [C: 032] "Works fine for me with PhantomJS version 1.9.7 (encountered a known bug with PhantomJS version 1.9.6)." [selenium] - 10https://gerrit.wikimedia.org/r/114153 (owner: 10Zfilipin) [20:10:53] (03Merged) 10jenkins-bot: Added links for ZeroRatedMobileAccess repository [selenium] - 10https://gerrit.wikimedia.org/r/114153 (owner: 10Zfilipin) [20:28:58] hello