[00:52:47] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Differential, 10Developer-Wishlist (2017), 07Jenkins: Add support for a wmf-ci.yaml type file for wikimedia jenkins - https://phabricator.wikimedia.org/T145669#3038697 (10mmodell) This might be helpful: > https://github.com/vadi... [01:38:59] CI is broken, I think? At least when it comes to this patch… https://gerrit.wikimedia.org/r/#/c/335548/ [01:43:13] andrewbogott: nope [01:43:36] andrewbogott: jenkins doesnt merge puppet, operations will merge and deploy at same time [01:43:52] I'm not worried about it not merging, I'm worried about it not testing [01:44:19] It could just need a retest [01:44:28] are you a whitelisted user? [01:46:04] Let me ask a different question: Have you seen CI run tests on other patches in the last hour or two? [01:46:14] andrewbogott: Yes, on other repos at least [01:46:18] ok [01:46:20] Yes [01:46:26] I'll just ignore that patch then :) [01:46:27] thanks [01:46:34] andrewbogott: sometimes the jenkins doesnt see stuff [01:46:38] It happens [02:09:37] andrewbogott hi, that repo dosent have a gate and submit pipeline. So it wont retest without the word recheck or you make changes to the patch. [06:34:52] Project selenium-Wikibase » chrome,test,Linux,contintLabsSlave && UbuntuTrusty build #275: 04FAILURE in 1 hr 54 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/275/ [06:53:59] Yippee, build fixed! [06:53:59] Project selenium-Wikibase » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #275: 09FIXED in 2 hr 13 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/275/ [12:45:07] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Upgrade Jenkins from 1.x to latest 2.x - https://phabricator.wikimedia.org/T144106#3039029 (10Paladox) @hashar i think upstream are discontinuing support for java 7 and instead going to support java 8. Or that's what it looks lik... [13:27:01] PROBLEM - Puppet run on deployment-mira is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:33:38] (03PS1) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [13:56:01] (03PS2) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [14:01:15] (03PS3) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [14:19:39] (03PS4) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [14:31:59] RECOVERY - Puppet run on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [14:53:32] (03PS5) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [15:04:10] (03PS2) 10Hashar: [ChangeAuthor] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/338505 (owner: 10Umherirrender) [15:04:24] (03CR) 10Hashar: [C: 032] [ChangeAuthor] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/338505 (owner: 10Umherirrender) [15:05:32] (03Merged) 10jenkins-bot: [ChangeAuthor] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/338505 (owner: 10Umherirrender) [15:06:23] (03CR) 10Hashar: "Deployed :]" [integration/config] - 10https://gerrit.wikimedia.org/r/338507 (owner: 10Krinkle) [15:07:57] (03PS3) 10Hashar: Make some tests non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/338506 (owner: 10Umherirrender) [15:08:03] (03CR) 10Hashar: [C: 032] Make some tests non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/338506 (owner: 10Umherirrender) [15:09:00] (03Merged) 10jenkins-bot: Make some tests non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/338506 (owner: 10Umherirrender) [15:27:24] 10Continuous-Integration-Config, 06Wikipedia-Android-App-Backlog, 07Technical-Debt: Figure out the right way to keep the Android SDK up to date in CI - https://phabricator.wikimedia.org/T158456#3039168 (10hashar) Our Jenkins instance has the [[ https://wiki.jenkins-ci.org/display/JENKINS/Android+Emulator+Plu... [15:43:34] (03PS6) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [15:47:09] (03PS7) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [15:52:36] (03PS8) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [15:54:56] (03PS9) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [15:57:10] (03PS10) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [16:03:20] (03PS11) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [16:07:47] (03PS12) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [16:09:09] (03PS13) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [16:14:17] (03PS14) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [16:17:05] (03PS15) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [16:21:28] (03PS16) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [16:24:06] (03CR) 10Paladox: "recheck" [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 (owner: 10Paladox) [16:30:43] (03PS17) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [16:33:38] (03PS18) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [16:34:52] (03Draft1) 10Paladox: Test [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338622 [16:35:17] (03Abandoned) 10Paladox: Test [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338622 (owner: 10Paladox) [16:53:16] (03PS1) 10Umherirrender: [DeepSea] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/338625 [16:53:31] (03PS2) 10Umherirrender: [DeepSea] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/338625 [16:57:34] (03PS19) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [17:00:20] Is it possible i could be added to the main test whitelist? [17:01:00] Zppix yes as long as you have some merges in gerrit.wikimedia.org. [17:01:08] I do [17:01:12] I think the min is 5 of your changes merged. [17:04:12] (03Draft1) 10Paladox: Whitelist Zppix [integration/config] - 10https://gerrit.wikimedia.org/r/338627 [17:04:14] (03PS2) 10Paladox: Whitelist Zppix [integration/config] - 10https://gerrit.wikimedia.org/r/338627 [17:04:27] Thanks! [17:04:55] Your welcome [17:18:36] (03PS20) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [17:22:16] (03CR) 10Zppix: [C: 031] Whitelist Zppix [integration/config] - 10https://gerrit.wikimedia.org/r/338627 (owner: 10Paladox) [19:35:10] (03CR) 10Hashar: [C: 032] [DeepSea] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/338625 (owner: 10Umherirrender) [19:35:59] (03Merged) 10jenkins-bot: [DeepSea] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/338625 (owner: 10Umherirrender) [19:40:36] (03PS3) 10Hashar: Whitelist Zppix [integration/config] - 10https://gerrit.wikimedia.org/r/338627 (owner: 10Paladox) [19:40:48] (03CR) 10Hashar: [C: 032] Whitelist Zppix [integration/config] - 10https://gerrit.wikimedia.org/r/338627 (owner: 10Paladox) [19:41:02] Ty hashar [19:41:19] :D [19:42:00] (03Merged) 10jenkins-bot: Whitelist Zppix [integration/config] - 10https://gerrit.wikimedia.org/r/338627 (owner: 10Paladox) [19:42:48] ^ is that autodeployed or is it deployed with rest of config changes? [19:42:55] manually deployed [19:43:06] so once merged, we run a script to do the deplloy [19:43:12] Ah [19:43:15] Cool [19:43:22] that is the fabfile.py at the root of integration/config.git [19:43:33] it uses http://www.fabfile.org/ [19:43:41] a python layer on top of ssh if you want [19:43:46] basically: ssh to the box [19:43:47] pull [19:43:50] reload zuul [19:43:51] done :] [19:44:04] Fancy git pull command basically? [19:44:22] yup [19:44:30] but done on the relmote server [19:44:38] so one needs access to the production server [19:44:43] Figured [19:49:27] hashar: how does jenkins choose what instance to run a job on [19:49:51] in Jenkins, jobs can be associated to a "label" [19:50:10] so we can tell that a job "phpunit-with-hhvm" is tied to a label named "jessie" [19:50:18] Zuul triggers the job [19:50:29] and Jenkins will then try to find an instance that has the label "jessie" [19:50:41] if one is available, jenkins runs the job on an instance [19:50:49] trying to reuse an instance that already build the job [19:51:19] on wikimedia setup we have instances that are discarded after each build. They have the labels: "ci-jessie-wikimedia" or "ci-trusty-wikimedia" [19:51:28] most everything runs on jessie [19:51:39] trusty is for Zend PHP 5.5 [19:51:41] And if it doesnt have any instances available at that time? [19:53:39] then [19:53:49] the jobs are triggered by zuul [19:53:56] and zuul will just keep the job in a queue [19:54:09] until a jenkins instance shows up with the associated label [19:55:31] So it cant create more instances temp? [19:57:54] yeah [19:58:00] there is a software named Nodepool [19:58:06] it listens for jobs pending in Zuul [19:58:17] and spawns instances on demand [19:58:34] once the build is complete, the instance is deleted automatically [20:00:30] the whole chain is roughly: [20:00:36] developer send a patch to Gerrit [20:01:01] Gerrit emits en event "patchset-created" which has a bunch of metadata such as the project, commit, git referene, author etc [20:01:09] Zuul listens for such events from Gerrit [20:01:42] based on a workflow ( zuul/layout.yaml in integration/config.git) zuul pass the event through different pipelines [20:01:51] each pipeline only react to some events [20:02:08] for example gate-and-submit pipeline only reacts when one votes Code-Review +2 [20:02:15] once the change enter a pipeline [20:02:28] a project had some jobs to trigger for that pipeline [20:02:39] zuul runs the jobs via a worker system named Gearman [20:02:59] Gearman basically hold a list of every jobs and which workers can run the job [20:03:35] Jenkins is connected to Gearman, for each instance attached to Jenkins, it will register in gearman a (job, instance) [20:03:53] then when Zuul triggers job in Gearman, Jenkins ends up running it [20:05:45] Cool [20:18:41] sleep() [21:40:46] (03PS21) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [21:41:29] (03PS22) 10Paladox: Test: Do not submit [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 [21:49:59] (03CR) 10Paladox: "I've built it here http://gerrit-jenkins.wmflabs.org/job/debian-glue-non-voting/14/ http://gerrit-test.wmflabs.org/gerrit/#/c/4/" [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/338620 (owner: 10Paladox) [21:50:36] ^ what are you doing exactly [21:56:27] Testing some zuul changes. [21:56:36] that patch is only for testing. [21:58:22] So i dont get it whats the point of gerrit jenkins.wmflabs [21:58:54] Thats testing jenkins changes [21:59:06] it also has gerrit-test.wmflabs.orf and gerrit-zuul.wmflabs.org [21:59:11] gerrit-test.wmflabs.org [22:02:35] I dont see how on jenkins test you can do anything [22:03:17] I can test gerrit changes. I can also test by creating tests in jenkins. [22:08:36] Hoe [22:08:37] Ehoe [22:08:39] How [22:08:51] geez i cant spell [22:21:47] paladox: how does one do that