[00:47:47] (03CR) 10Tim Landscheidt: "@JanZerebecki: Cf. I5d021706fe5642848941f3e9d7563d0f839104ed." [integration/config] - 10https://gerrit.wikimedia.org/r/282452 (https://phabricator.wikimedia.org/T114887) (owner: 10Tim Landscheidt) [04:36:26] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [04:39:41] PROBLEM - Puppet run on deployment-elastic07 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:06:32] RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [05:12:24] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:16:30] PROBLEM - Puppet run on deployment-db1 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [05:45:49] Project beta-scap-eqiad build #99789: 04FAILURE in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99789/ [05:51:31] RECOVERY - Puppet run on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [05:55:52] Project beta-scap-eqiad build #99790: 04STILL FAILING in 1 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99790/ [06:05:45] Project beta-scap-eqiad build #99791: 04STILL FAILING in 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99791/ [06:15:54] Project beta-scap-eqiad build #99792: 04STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99792/ [06:25:50] Project beta-scap-eqiad build #99793: 04STILL FAILING in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99793/ [06:35:50] Project beta-scap-eqiad build #99794: 04STILL FAILING in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99794/ [06:45:48] Project beta-scap-eqiad build #99795: 04STILL FAILING in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99795/ [06:55:47] Project beta-scap-eqiad build #99796: 04STILL FAILING in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99796/ [07:05:46] Project beta-scap-eqiad build #99797: 04STILL FAILING in 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99797/ [07:15:47] Project beta-scap-eqiad build #99798: 04STILL FAILING in 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99798/ [07:24:19] 07:15:43 07:15:43 ['/usr/bin/sync-master', 'deployment-tin.deployment-prep.eqiad.wmflabs'] on mira.deployment-prep.eqiad.wmflabs returned [255]: Permission denied (publickey,keyboard-interactive). [07:25:52] Project beta-scap-eqiad build #99799: 04STILL FAILING in 1 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99799/ [07:35:46] Project beta-scap-eqiad build #99800: 04STILL FAILING in 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99800/ [07:41:40] (03PS2) 10Hashar: dib: glue for Ubuntu Trusty imaging [integration/config] - 10https://gerrit.wikimedia.org/r/284900 (https://phabricator.wikimedia.org/T133203) [07:45:48] Project beta-scap-eqiad build #99801: 04STILL FAILING in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99801/ [07:55:51] Project beta-scap-eqiad build #99802: 04STILL FAILING in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99802/ [08:05:51] Project beta-scap-eqiad build #99803: 04STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99803/ [08:10:07] (03Abandoned) 10Zfilipin: WIP make JJB work for VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/284883 (owner: 10Zfilipin) [08:15:48] Project beta-scap-eqiad build #99804: 04STILL FAILING in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99804/ [08:25:46] Project beta-scap-eqiad build #99805: 04STILL FAILING in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99805/ [08:34:38] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Simplify creating Jenkins jobs for running browser tests daily - https://phabricator.wikimedia.org/T128190#2234305 (10zeljkofilipin) [08:35:51] Project beta-scap-eqiad build #99806: 04STILL FAILING in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99806/ [08:40:38] (03CR) 10Hashar: [C: 04-1] "Just need to add an example to README.md and then we can release 1.7.x" [selenium] - 10https://gerrit.wikimedia.org/r/282709 (https://phabricator.wikimedia.org/T128190) (owner: 10Zfilipin) [08:45:46] Project beta-scap-eqiad build #99807: 04STILL FAILING in 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99807/ [08:45:59] ^^^ will deal with that soonish ™ [08:47:39] !log mwdeploy@deployment-tin has lost ssh host keys file :( [08:47:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:50:36] Project beta-scap-eqiad build #99808: 04STILL FAILING in 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99808/ [08:50:39] :( [08:55:49] Project beta-scap-eqiad build #99809: 04STILL FAILING in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99809/ [09:05:48] Project beta-scap-eqiad build #99810: 04STILL FAILING in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99810/ [09:15:46] Project beta-scap-eqiad build #99811: 04STILL FAILING in 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99811/ [09:20:56] !log Keyholder / mwdeploy ssh keys have been messed up on beta cluster somehow :-( [09:20:59] I give up [09:20:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:24:32] 10Beta-Cluster-Infrastructure: Keyholder on beta cluster has lost credentials for mwdeploy user - https://phabricator.wikimedia.org/T133521#2234397 (10hashar) [09:24:51] !log beta / scap failure filled as T133521 [09:24:52] T133521: Keyholder on beta cluster has lost credentials for mwdeploy user - https://phabricator.wikimedia.org/T133521 [09:24:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:25:46] Project beta-scap-eqiad build #99812: 04STILL FAILING in 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99812/ [09:34:16] (03CR) 10Hashar: dib: glue for Ubuntu Trusty imaging (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/284900 (https://phabricator.wikimedia.org/T133203) (owner: 10Hashar) [09:34:29] (03PS3) 10Hashar: dib: glue for Ubuntu Trusty imaging [integration/config] - 10https://gerrit.wikimedia.org/r/284900 (https://phabricator.wikimedia.org/T133203) [09:35:49] Project beta-scap-eqiad build #99813: 04STILL FAILING in 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99813/ [09:45:44] Project beta-scap-eqiad build #99814: 04STILL FAILING in 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99814/ [09:51:15] (03PS4) 10Hashar: dib: glue for Ubuntu Trusty imaging [integration/config] - 10https://gerrit.wikimedia.org/r/284900 (https://phabricator.wikimedia.org/T133203) [09:55:42] Project beta-scap-eqiad build #99815: 04STILL FAILING in 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99815/ [10:01:15] (03PS5) 10Hashar: dib: glue for Ubuntu Trusty imaging [integration/config] - 10https://gerrit.wikimedia.org/r/284900 (https://phabricator.wikimedia.org/T133203) [10:01:35] (03CR) 10Hashar: "Forgot to export DIB_RELEASE" [integration/config] - 10https://gerrit.wikimedia.org/r/284900 (https://phabricator.wikimedia.org/T133203) (owner: 10Hashar) [10:05:42] Project beta-scap-eqiad build #99816: 04STILL FAILING in 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99816/ [10:11:31] RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [10:15:15] RECOVERY - Puppet run on integration-slave-trusty-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [10:15:41] Project beta-scap-eqiad build #99817: 04STILL FAILING in 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99817/ [10:16:38] (03PS3) 10Zfilipin: Make site-specific Cucumber tag optional [selenium] - 10https://gerrit.wikimedia.org/r/282709 (https://phabricator.wikimedia.org/T128190) [10:17:25] hashar: what do you think about the readme? https://gerrit.wikimedia.org/r/#/c/282709/3/README.md,cm [10:20:48] zeljkof: yeah that is goo [10:20:49] d [10:20:52] (03CR) 10Hashar: [C: 032] Make site-specific Cucumber tag optional [selenium] - 10https://gerrit.wikimedia.org/r/282709 (https://phabricator.wikimedia.org/T128190) (owner: 10Zfilipin) [10:21:06] zeljkof: lets release 1.7.0 :-} [10:21:19] hashar: ok, will release the gem [10:23:34] (03Merged) 10jenkins-bot: Make site-specific Cucumber tag optional [selenium] - 10https://gerrit.wikimedia.org/r/282709 (https://phabricator.wikimedia.org/T128190) (owner: 10Zfilipin) [10:25:43] Project beta-scap-eqiad build #99818: 04STILL FAILING in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99818/ [10:35:32] !log Refreshed Nodepool Jessie image ( image-jessie-20160425T100035Z ) [10:35:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:35:40] Project beta-scap-eqiad build #99819: 04STILL FAILING in 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99819/ [10:36:07] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Simplify creating Jenkins jobs for running browser tests daily - https://phabricator.wikimedia.org/T128190#2234559 (10zeljkofilipin) [10:38:10] !log Refreshing Nodepool Jessie snapshot based on new image [10:38:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:41:19] (03PS1) 10Zfilipin: Release minor version 1.7.0 [selenium] - 10https://gerrit.wikimedia.org/r/285159 (https://phabricator.wikimedia.org/T128190) [10:45:40] Project beta-scap-eqiad build #99820: 04STILL FAILING in 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99820/ [10:46:13] (03CR) 10Zfilipin: [C: 032] Release minor version 1.7.0 [selenium] - 10https://gerrit.wikimedia.org/r/285159 (https://phabricator.wikimedia.org/T128190) (owner: 10Zfilipin) [10:49:37] (03Merged) 10jenkins-bot: Release minor version 1.7.0 [selenium] - 10https://gerrit.wikimedia.org/r/285159 (https://phabricator.wikimedia.org/T128190) (owner: 10Zfilipin) [10:52:35] Project beta-scap-eqiad build #99821: 04STILL FAILING in 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99821/ [10:54:46] 05Continuous-Integration-Scaling, 03releng-201516-q4, 13Patch-For-Review, 07WorkType-NewFunctionality: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203#2234626 (10hashar) I have tried to build an image and `puppet apply` fails with `invalid option: --test` ``` Prep... [10:55:40] Project beta-scap-eqiad build #99822: 04STILL FAILING in 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99822/ [10:59:33] (03PS6) 10Hashar: dib: glue for Ubuntu Trusty imaging [integration/config] - 10https://gerrit.wikimedia.org/r/284900 (https://phabricator.wikimedia.org/T133203) [11:02:27] 05Continuous-Integration-Scaling, 03releng-201516-q4, 13Patch-For-Review, 07WorkType-NewFunctionality: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203#2234630 (10hashar) Patchset 6 expands `--test` to `--detailed-exitcodes --show_diff` https://gerrit.wikimedia.or... [11:03:10] (03PS7) 10Hashar: dib: glue for Ubuntu Trusty imaging [integration/config] - 10https://gerrit.wikimedia.org/r/284900 (https://phabricator.wikimedia.org/T133203) [11:03:47] RECOVERY - Puppet run on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [11:05:39] Project beta-scap-eqiad build #99823: 04STILL FAILING in 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99823/ [11:15:42] Project beta-scap-eqiad build #99824: 04STILL FAILING in 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99824/ [11:25:41] Project beta-scap-eqiad build #99825: 04STILL FAILING in 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99825/ [11:26:34] 05Continuous-Integration-Scaling, 03releng-201516-q4, 13Patch-For-Review, 07WorkType-NewFunctionality: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203#2234663 (10hashar) Later fails with: * base::service_unit attempting to start the Xvfb service. That can not wor... [11:34:47] (03PS8) 10Hashar: dib: glue for Ubuntu Trusty imaging [integration/config] - 10https://gerrit.wikimedia.org/r/284900 (https://phabricator.wikimedia.org/T133203) [11:35:38] Project beta-scap-eqiad build #99826: 04STILL FAILING in 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99826/ [11:40:06] 05Continuous-Integration-Scaling, 03releng-201516-q4, 13Patch-For-Review, 07WorkType-NewFunctionality: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203#2234671 (10hashar) Upstart/ missing packages solved in PS8 https://gerrit.wikimedia.org/r/#/c/284900/7..8/dib/pup... [11:45:40] Project beta-scap-eqiad build #99827: 04STILL FAILING in 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99827/ [11:53:48] 05Continuous-Integration-Scaling, 03releng-201516-q4, 13Patch-For-Review, 07WorkType-NewFunctionality: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203#2234703 (10hashar) [11:55:43] Project beta-scap-eqiad build #99828: 04STILL FAILING in 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99828/ [12:05:45] Project beta-scap-eqiad build #99829: 04STILL FAILING in 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99829/ [12:11:03] 06Release-Engineering-Team, 10Phabricator: Clean up tasks in archived #Staging Phabricator project - https://phabricator.wikimedia.org/T133529#2234728 (10Aklapper) [12:11:12] 06Release-Engineering-Team, 10Phabricator: Clean up tasks in archived #Staging Phabricator project - https://phabricator.wikimedia.org/T133529#2234740 (10Aklapper) p:05Triage>03Low [12:15:40] Project beta-scap-eqiad build #99830: 04STILL FAILING in 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99830/ [12:25:45] Project beta-scap-eqiad build #99831: 04STILL FAILING in 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99831/ [12:35:42] Project beta-scap-eqiad build #99832: 04STILL FAILING in 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99832/ [12:36:18] (03PS1) 10Hashar: dib: allow dist-upgrade to downgrade package [integration/config] - 10https://gerrit.wikimedia.org/r/285173 (https://phabricator.wikimedia.org/T133203) [12:36:35] 05Continuous-Integration-Scaling, 03releng-201516-q4, 13Patch-For-Review, 07WorkType-NewFunctionality: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203#2234794 (10hashar) Filled T133528 about libpcre3. Meanwhile I have hacked dist-upgrade to allow downgrade with `... [12:45:48] Project beta-scap-eqiad build #99833: 04STILL FAILING in 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99833/ [12:59:00] 05Continuous-Integration-Scaling, 03releng-201516-q4, 13Patch-For-Review, 07WorkType-NewFunctionality: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203#2234817 (10hashar) Forged `image-trusty-20160425T124552Z.qcow2`. Will boot that on labs. [12:59:31] (03PS2) 10Hashar: dib: allow dist-upgrade to downgrade package [integration/config] - 10https://gerrit.wikimedia.org/r/285173 (https://phabricator.wikimedia.org/T133203) [12:59:41] Project beta-scap-eqiad build #99834: 04STILL FAILING in 4 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99834/ [13:06:57] Project beta-scap-eqiad build #99835: 04STILL FAILING in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99835/ [13:07:05] hashar: will you be around in the next 10-20 minutes to help with migration of browsertests* to selenium* jobs? [13:07:11] all I need is a few +2s :) [13:07:30] I am doing some final testing, will be ready for the first jobs in 5-10 minutes [13:15:00] !log openstack image create --file /home/hashar/image-trusty-20160425T124552Z.qcow2 ci-trusty-wikimedia --disk-format qcow2 --property show=true # T133203 [13:15:01] T133203: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203 [13:15:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:15:17] zeljkof: self +2 them I guess ;-:} [13:15:30] I am busy building a Trusty image for Nodepool :D [13:15:34] hashar: will do, if needed [13:15:48] Project beta-scap-eqiad build #99836: 04STILL FAILING in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99836/ [13:15:52] just wanted to check if you are available for a quick review [13:16:06] but I know how context switching is killing productivity :) [13:16:18] (03PS1) 10Hashar: dib: mentions DIB_DEBOOTSTRAP_CACHE [integration/config] - 10https://gerrit.wikimedia.org/r/285176 [13:16:43] zeljkof: regarding the JJB job template, what you have shown me last week and this morning looks all fine to me [13:16:51] if something breaks / is missing that can be amended [13:17:10] zeljkof: "just" bump mediawiki_selenium and repositories you have tested, switch them to rake + yaml file [13:17:22] and get rid of the old browser tests in a couple week disabling them meanwhile [13:17:28] hashar: ok, will start moving the jobs as soon as am I happy with the final test [13:17:50] but definitely start migrating at least some repositories that are green right now ;-} [13:19:47] yes, will try to see how much green I can get, and what needs to happen to get moar green :) [13:25:49] Project beta-scap-eqiad build #99837: 04STILL FAILING in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99837/ [13:27:34] (03CR) 10Hashar: [C: 032] "Good enough for now" [integration/config] - 10https://gerrit.wikimedia.org/r/284900 (https://phabricator.wikimedia.org/T133203) (owner: 10Hashar) [13:28:02] (03CR) 10Hashar: [C: 032] "That is really a hack but unblock me till libpcre3 issue is sorted out on Trusty" [integration/config] - 10https://gerrit.wikimedia.org/r/285173 (https://phabricator.wikimedia.org/T133203) (owner: 10Hashar) [13:28:26] (03CR) 10Hashar: [C: 032] "Might be a regression of DIB 1.13.0" [integration/config] - 10https://gerrit.wikimedia.org/r/285176 (owner: 10Hashar) [13:28:39] (03Merged) 10jenkins-bot: dib: glue for Ubuntu Trusty imaging [integration/config] - 10https://gerrit.wikimedia.org/r/284900 (https://phabricator.wikimedia.org/T133203) (owner: 10Hashar) [13:28:53] zeljkof: ^^^those changes are for the Nodepool images. No impact on JJB conf :-) [13:28:58] so you can blindly rebase [13:29:29] (03Merged) 10jenkins-bot: dib: allow dist-upgrade to downgrade package [integration/config] - 10https://gerrit.wikimedia.org/r/285173 (https://phabricator.wikimedia.org/T133203) (owner: 10Hashar) [13:29:54] (03Merged) 10jenkins-bot: dib: mentions DIB_DEBOOTSTRAP_CACHE [integration/config] - 10https://gerrit.wikimedia.org/r/285176 (owner: 10Hashar) [13:31:12] hashar: thanks for letting me know [13:34:19] !log Nodepool is attempting to create a Trusty snapshot with name ci-trusty-wikimedia-1461591203 | T133203 [13:34:20] T133203: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203 [13:34:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:35:45] Project beta-scap-eqiad build #99838: 04STILL FAILING in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99838/ [13:36:19] Can someone take a look at beta-scap-eqiad? [13:45:45] Project beta-scap-eqiad build #99839: 04STILL FAILING in 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99839/ [13:48:06] Luke081515: I have filled a bug about it earlier this morning. Keyholder / ssh private keys are screwed up entirely [13:49:00] * twentyafterfour did it [13:49:08] hashar: Luke081515 I'll fix [13:50:24] ok, thanks [13:53:21] 05Continuous-Integration-Scaling, 03releng-201516-q4, 13Patch-For-Review, 07WorkType-NewFunctionality: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203#2235035 (10hashar) The image more or less boot on labs but fails to acquire network connection (ping does not wor... [13:55:06] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Migration of browsertests* Jenkins jobs to selenium* jobs - https://phabricator.wikimedia.org/T128190#2235054 (10zeljkofilipin) [13:55:48] Project beta-scap-eqiad build #99840: 04STILL FAILING in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99840/ [13:56:58] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Migration of browsertests* Jenkins jobs to selenium* jobs - https://phabricator.wikimedia.org/T128190#2066598 (10zeljkofilipin) [13:58:47] 05Continuous-Integration-Scaling, 03releng-201516-q4, 13Patch-For-Review, 07WorkType-NewFunctionality: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203#2235078 (10hashar) Some boot trouble I had with Jessie back in July 2015 is documented via T105152. The console l... [13:58:48] (03CR) 10JanZerebecki: "Yes I read all comments in I5d021706fe5642848941f3e9d7563d0f839104ed and found no disagreement." [integration/config] - 10https://gerrit.wikimedia.org/r/282452 (https://phabricator.wikimedia.org/T114887) (owner: 10Tim Landscheidt) [13:59:22] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: There should be a way to run custom Rake task in selenium* jobs - https://phabricator.wikimedia.org/T133542#2235080 (10zeljkofilipin) [14:03:27] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:05:48] Project beta-scap-eqiad build #99841: 04STILL FAILING in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99841/ [14:06:55] (03PS41) 10Zfilipin: WIP Migration of browsertests* Jenkins jobs to selenium* jobs [integration/config] - 10https://gerrit.wikimedia.org/r/274136 (https://phabricator.wikimedia.org/T128190) [14:08:54] Project selenium-CentralAuth » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 1 min 39 sec: https://integration.wikimedia.org/ci/job/selenium-CentralAuth/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [14:10:53] hashar: Maybe you or someone other can help me with https://www.mediawiki.org/wiki/Continuous_integration/Jenkins_job_builder#Install_JJB? the sudo pip install -e . fails, and I don't know what I made wrong :-/ [14:11:48] Luke081515: yeah sure [14:12:08] Luke081515: mind pasting the whole trace somewhere ? :} [14:12:22] sudo should not be needed [14:12:23] ideally [14:13:27] hashar: So my first question is, where I have to ran that command ;). I tried /var/lib/python [14:13:42] * Luke081515 is not a python expert :D [14:13:49] if I try it there, I get: [14:13:53] Directory '/var/lib/python' is not installable. File 'setup.py' not found. [14:13:56] Storing debug log for failure in /home/luke081515/.pip/pip.log [14:15:48] Project beta-scap-eqiad build #99842: 04STILL FAILING in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99842/ [14:15:48] yeah that does not make anysense [14:15:51] pip is a package manager for python [14:16:00] the same as npm for javascript or composer for php [14:16:25] we use a snapshot version of JJB which is in our Gerrit in integration/jenkins-job-builder.git [14:16:38] so at first you have to clone that repository on your local machine [14:16:41] then cd to it [14:17:40] once there, you can get pip to install it from that local copy (pip install .) but instead of copying files in /usr/whatever/python , just alias to the working copy (pip install -e . ) [14:17:46] so one can hack in his local copy [14:18:07] then there is a bunch of lameness because Debian pip usually attempt to install in /usr/ which requires root [14:18:26] so usually I recommend people to install in their user home instead with: pip install --user -e . [14:18:30] RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [14:19:00] that would install the software under ~/.local/ [14:19:12] and executables in ~/.local/bin/ which one would want to add to PATH [14:19:18] Luke081515: TLDR: that is tedious [14:19:30] ok [14:19:55] Luke081515: one sure thing /var/lib/python does not make sense. I have no idea what would be there, but that is definitely not JJB :-} [14:20:08] ok :D [14:25:50] Project beta-scap-eqiad build #99843: 04STILL FAILING in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99843/ [14:27:51] Project selenium-CentralNotice » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 37 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [14:27:51] dpkg-query: package 'openssh-server' is not installed and no information is available [14:27:54] that is never ending [14:27:56] Project selenium-CentralNotice » chrome,beta,Windows 7,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 42 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%207,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [14:27:59] Project selenium-CentralNotice » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 45 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [14:28:10] Project selenium-CentralNotice » firefox,beta,Windows 7,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 56 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%207,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [14:28:18] Project selenium-CentralNotice » chrome,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 1 min 4 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [14:28:50] (03PS1) 10Hashar: dib: enable network on Trusty image [integration/config] - 10https://gerrit.wikimedia.org/r/285184 (https://phabricator.wikimedia.org/T133203) [14:29:01] Project selenium-CentralNotice » firefox,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 1 min 47 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [14:31:23] (03CR) 10Hashar: [C: 032] "That got us network. Still failing due to openssh-server not being found but that is a different issue." [integration/config] - 10https://gerrit.wikimedia.org/r/285184 (https://phabricator.wikimedia.org/T133203) (owner: 10Hashar) [14:32:09] (03Merged) 10jenkins-bot: dib: enable network on Trusty image [integration/config] - 10https://gerrit.wikimedia.org/r/285184 (https://phabricator.wikimedia.org/T133203) (owner: 10Hashar) [14:32:36] 05Continuous-Integration-Scaling, 03releng-201516-q4, 13Patch-For-Review, 07WorkType-NewFunctionality: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203#2235184 (10hashar) Needed the `simple-init` element to be added. Now fails with: ``` lang=diff * Zuul Merger:... [14:35:48] Project beta-scap-eqiad build #99844: 04STILL FAILING in 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99844/ [14:48:49] Project beta-scap-eqiad build #99845: 04STILL FAILING in 4 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99845/ [14:52:04] is there something special that has to be done in wikitech for the mwdeploy user? I think I got the keys right in keyholder but still can't log in to deployment-mediawiki01 from deployment-tin as user=mwdeploy [14:54:09] (03PS1) 10Hashar: dib: install openssh-server for Trusty [integration/config] - 10https://gerrit.wikimedia.org/r/285186 (https://phabricator.wikimedia.org/T133203) [14:55:15] twentyafterfour: one sure thing mwdeploy has a fixed UID in LDAP so the uid is consistent across beta cluster [14:55:40] twentyafterfour: mwdeploy might be accessible via wikitech though I have no idea where is the password to edit its settings (maybe office wiki ) [14:56:02] not on wikitech [14:56:08] I guess it got added directly in ldap [14:56:26] and thus I would say the mwdeploy ssh authorized keys are shipped via puppet somehow [14:57:18] twentyafterfour: modules/mediawiki/manifests/users.pp  has a ssh::user key that takes the content of the key from a variable $mwdeploy_pub_key [14:57:24] yeah the keys are in puppet I fixed that up [14:57:42] it is probably on the puppetmaster in /var/lib/git/labs/private/ as a local commit [14:57:49] yeah ... [14:58:02] I think I got it to work now, just have to run puppet on all the scap targets [14:58:09] and there is also some hiera stuff [14:58:11] hieradata/labs/deployment-prep/common.yaml:"mediawiki::users::mwdeploy_pub_key": 'ssh-rsa AAAAB3NzaC..... [14:58:28] actually just getting "Host key verification failed." [14:58:38] (now that I got the user key right) [14:58:45] why would host keys fail? [14:59:13] I noticed this morning that on deployment-tin /home/mwdeploy/.ssh/known_hosts is not present [14:59:18] so maybe it disappeaered [14:59:30] eh, doesn't it run as jenkins-deploy? [14:59:31] maybe try to manually validate them ? [14:59:48] yeah the job definitely runs as jenkins-deploy [14:59:57] but I would expect the scap command to run with sudo -u mwdeploy [15:00:07] (maybe HOME is not reset properly) [15:00:22] yeah, but the jenkins-deploy user is the only one that needs to accept host-keys in that case. [15:00:49] magic [15:01:04] hmm [15:01:34] /mnt/home/jenkins-deploy/.ssh/known_hosts is there [15:01:54] 10Continuous-Integration-Infrastructure, 06Operations, 13Patch-For-Review: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#2235252 (10MoritzMuehlenhoff) [15:02:39] so I'm testing as myself not as jenkins-deploy [15:02:40] works: jenkins-deploy@deployment-tin:~$ SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l mwdeploy deployment-mediawiki01.deployment-prep.eqiad.wmflabs [15:02:59] (03CR) 10Hashar: [C: 032] "Works:" [integration/config] - 10https://gerrit.wikimedia.org/r/285186 (https://phabricator.wikimedia.org/T133203) (owner: 10Hashar) [15:03:05] just trying to sync-wikiversions as myself on deployment-tin .. [15:03:21] yeah, that might not work :) [15:03:36] 05Continuous-Integration-Scaling, 03releng-201516-q4, 13Patch-For-Review, 07WorkType-NewFunctionality: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203#2235273 (10hashar) Booted a Trusty image: ``` * Starting OpenSSH server[74G[ OK ] ssh start/running, process 6... [15:03:42] jenkins@ubuntu:~$ php [15:03:42] -bash: php: command not found [15:03:45] uhm, ok so the jenkins job should be fixed then? /me waits for wmf-insecte to say something [15:03:48] \L/ [15:03:52] lol [15:03:54] Project beta-scap-eqiad build #99846: 04STILL FAILING in 9 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99846/ [15:03:59] hmm still failing [15:04:19] but just deployment-jobrunner01 [15:04:23] (03Merged) 10jenkins-bot: dib: install openssh-server for Trusty [integration/config] - 10https://gerrit.wikimedia.org/r/285186 (https://phabricator.wikimedia.org/T133203) (owner: 10Hashar) [15:04:28] so: progress [15:04:29] yeah [15:04:41] so that's because I just ran puppet on deployment-jobrunner01 moments ago [15:04:48] next run should be success! :) [15:05:32] good luck :D [15:05:37] sweet. so that'll be all the private/public key autogen stuff working? [15:09:28] thcipriani: yes [15:09:37] Yippee, build fixed! [15:09:38] Project beta-scap-eqiad build #99847: 09FIXED in 3 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99847/ [15:10:06] There was a long standing local patch for scap on deployment-bastion that ignored host keys. That was due to the lack of host key resource collection in beta cluster puppet [15:10:49] I thought that was the case, but that seems to have been removed more recently. Just lost in the shuffle? [15:11:14] great, fixed [15:11:40] well it seems the host key error was a red-herring [15:11:44] thcipriani: here is the patch -- https://gerrit.wikimedia.org/r/#/c/148112/ [15:12:03] because jenkins isn't having trouble with host keys, now that I got the user keys right [15:12:27] 10Continuous-Integration-Infrastructure, 06Operations, 13Patch-For-Review: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#1177707 (10fgiunchedi) looks like all blocking subtasks are fixed now, @hashar how can we try this again? I tried accessing `... [15:20:38] Project selenium-CirrusSearch » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 52 sec: https://integration.wikimedia.org/ci/job/selenium-CirrusSearch/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [15:31:36] PROBLEM - Puppet run on deployment-memc03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:33:25] (03PS1) 10Hashar: dib: always install cloud-init [integration/config] - 10https://gerrit.wikimedia.org/r/285194 (https://phabricator.wikimedia.org/T133203) [15:34:03] PROBLEM - Puppet run on integration-slave-trusty-1014 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [15:36:12] Project browsertests-RelatedArticles-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #117: 04FAILURE in 11 sec: https://integration.wikimedia.org/ci/job/browsertests-RelatedArticles-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/117/ [15:36:29] Project selenium-Core » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 10 min: https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [15:36:39] PROBLEM - Puppet run on deployment-redis01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:37:23] PROBLEM - Puppet run on deployment-cache-upload04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:37:39] PROBLEM - Puppet run on integration-saltmaster is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:37:57] PROBLEM - Puppet run on deployment-eventlogging04 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:38:02] 10Continuous-Integration-Infrastructure, 06Operations, 13Patch-For-Review: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#2235498 (10hashar) `integration-slave-jessie-1001.integration.eqiad.wmflabs` but you are denied somehow: ``` pam_access(sshd:... [15:38:33] PROBLEM - Puppet run on deployment-jobrunner01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:39:03] PROBLEM - Puppet run on deployment-elastic05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:39:03] PROBLEM - Puppet run on deployment-ores-web is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:40:15] PROBLEM - Puppet run on integration-slave-trusty-1018 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:40:30] (03PS2) 10Hashar: dib: always install cloud-init [integration/config] - 10https://gerrit.wikimedia.org/r/285194 (https://phabricator.wikimedia.org/T133203) [15:40:41] PROBLEM - Parsoid on deployment-parsoid05 is CRITICAL: Connection refused [15:41:26] all those puppet errors are due to some labs DNS failure [15:41:56] PROBLEM - Puppet run on deployment-upload is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:42:40] PROBLEM - Puppet run on deployment-sentry2 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:42:40] PROBLEM - Puppet run on deployment-restbase01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:44:02] filled as https://phabricator.wikimedia.org/T133552 [15:45:34] PROBLEM - Puppet run on deployment-redis02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:45:42] RECOVERY - Parsoid on deployment-parsoid05 is OK: HTTP OK: HTTP/1.1 200 OK - 1514 bytes in 0.162 second response time [15:46:25] hashar: I have the situation: I want to create own unit tests, and want to create now a jenkins project for that, which type I have to choose? [15:48:45] Luke081515: freestyle project [15:48:49] I think [15:48:54] ok [15:50:57] Luke081515: and for a lot of use cases we already have prebuilt jobs :-D [15:51:03] such as running npm install && npm test :D [15:52:00] 10scap, 10Citoid, 06Services, 10VisualEditor, 10Scap3 (Scap3-Adoption-Phase1): Deploy Citoid with scap3 - https://phabricator.wikimedia.org/T116337#1746896 (10Jdforrester-WMF) [15:53:35] Project selenium-Flow-2016-04-25 » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 24 sec: https://integration.wikimedia.org/ci/job/selenium-Flow-2016-04-25/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [15:53:36] Project selenium-Flow-2016-04-25 » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 24 sec: https://integration.wikimedia.org/ci/job/selenium-Flow-2016-04-25/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [15:54:27] Project beta-scap-eqiad build #99855: 04FAILURE in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99855/ [15:58:07] Project beta-scap-eqiad build #99856: 04STILL FAILING in 1 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99856/ [15:59:15] PROBLEM - Puppet run on deployment-imagescaler01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:01:48] The Labs techops folks are looking at the dns flapping problem [16:02:23] It looks like one internal recursor may have decided not to work. it was just restarted [16:02:42] PROBLEM - Puppet run on integration-puppetmaster is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:02:42] PROBLEM - Puppet run on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:02:56] PROBLEM - Puppet run on deployment-ms-be01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:05:15] Project beta-scap-eqiad build #99857: 04STILL FAILING in 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99857/ [16:05:22] PROBLEM - Puppet run on integration-lightslave-jessie-1002 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [16:06:42] PROBLEM - Parsoid on deployment-parsoid05 is CRITICAL: Connection refused [16:07:10] (03CR) 10Hashar: [C: 04-2] "Not sure whether it is actually needed ..." [integration/config] - 10https://gerrit.wikimedia.org/r/285194 (https://phabricator.wikimedia.org/T133203) (owner: 10Hashar) [16:08:54] Project beta-scap-eqiad build #99858: 04STILL FAILING in 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99858/ [16:11:40] RECOVERY - Parsoid on deployment-parsoid05 is OK: HTTP OK: HTTP/1.1 200 OK - 1514 bytes in 0.128 second response time [16:11:51] PROBLEM - Puppet run on integration-slave-trusty-1011 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:11:51] PROBLEM - Puppet run on deployment-mediawiki01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:11:51] PROBLEM - Puppet run on integration-slave-trusty-1015 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:12:35] RECOVERY - Puppet run on integration-saltmaster is OK: OK: Less than 1.00% above the threshold [0.0] [16:14:09] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:16:12] Yippee, build fixed! [16:16:12] Project beta-scap-eqiad build #99859: 09FIXED in 1 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99859/ [16:16:33] Project selenium-Echo » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 3 min 9 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [16:16:40] Project selenium-Echo » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 3 min 17 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [16:22:21] Can somehelp me at this: I want to build a PHP-Unit test for my project (at my private jenkins), and I want to let this job run at a fresh wiki. How can I use that template? [16:26:58] 05Continuous-Integration-Scaling, 13Patch-For-Review, 07Upstream: diskimage cloud-init does not bring up network - https://phabricator.wikimedia.org/T105152#2235719 (10hashar) Fixed by upstream change https://review.openstack.org/#/c/200030/ which is in dib since `1.0.0`. [16:28:11] hasharAway: sry for asking again, but when you're back, can you help me with that at 18:22 CEST? Thanks :) [16:37:58] RECOVERY - Puppet run on deployment-ms-be01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:39:16] RECOVERY - Puppet run on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:41:42] RECOVERY - Puppet run on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:42:54] RECOVERY - Puppet run on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:44:06] RECOVERY - Puppet run on integration-slave-trusty-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [16:45:29] RECOVERY - Puppet run on integration-lightslave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [16:45:38] 05Continuous-Integration-Scaling, 03releng-201516-q4, 13Patch-For-Review, 07WorkType-NewFunctionality: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203#2235818 (10hashar) cloud-init ends up lacking the EC2 datasource and instead has None, None. Seems to be due to... [16:46:41] RECOVERY - Puppet run on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:46:53] RECOVERY - Puppet run on integration-slave-trusty-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [16:47:21] RECOVERY - Puppet run on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:47:35] RECOVERY - Puppet run on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:48:33] RECOVERY - Puppet run on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:51:33] Project selenium-Gather-2016-04-25 » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 9 min 2 sec: https://integration.wikimedia.org/ci/job/selenium-Gather-2016-04-25/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [16:51:53] RECOVERY - Puppet run on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [16:51:53] RECOVERY - Puppet run on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:51:53] RECOVERY - Puppet run on integration-slave-trusty-1015 is OK: OK: Less than 1.00% above the threshold [0.0] [16:53:11] Project selenium-GettingStarted » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 42 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:05:43] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 03Scap3, 06Operations, 13Patch-For-Review: Automate the generation deployment keys (keyholder-managed ssh keys) - https://phabricator.wikimedia.org/T133211#2235958 (10mmodell) >>! In T133211#2228392, @hashar wrote: > Hypothetically, can we rewind to d... [17:07:41] Project selenium-Math » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 29 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:07:56] Project selenium-Math » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 44 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:25:39] Project selenium-MultimediaViewer » firefox,mediawiki,Linux,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 24 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=mediawiki,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:26:02] Project selenium-PageTriage » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 54 sec: https://integration.wikimedia.org/ci/job/selenium-PageTriage/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:26:17] Project selenium-PageTriage » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 1 min 9 sec: https://integration.wikimedia.org/ci/job/selenium-PageTriage/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:30:39] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 03Scap3, 06Operations, 13Patch-For-Review: Automate the generation deployment keys (keyholder-managed ssh keys) - https://phabricator.wikimedia.org/T133211#2236031 (10mmodell) Todo: figure out how to get ssh-add to accept the password once so that we... [17:44:55] PROBLEM - Host deployment-mediawiki02 is DOWN: PING CRITICAL - Packet loss = 100% [17:45:48] 10Deployment-Systems, 03Scap3, 07WorkType-NewFunctionality: Grab git-rev from config - https://phabricator.wikimedia.org/T133572#2236137 (10thcipriani) [17:46:06] Project beta-scap-eqiad build #99869: 04FAILURE in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99869/ [17:46:13] RECOVERY - Host deployment-mediawiki02 is UP: PING OK - Packet loss = 0%, RTA = 0.57 ms [17:47:05] Project selenium-MultimediaViewer » internet_explorer 10.0,beta,Windows 8,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=internet_explorer%2010.0,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%208,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:47:09] Project selenium-MultimediaViewer » internet_explorer 11.0,beta,Windows 8.1,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=internet_explorer%2011.0,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%208.1,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:47:31] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:48:25] Project selenium-MultimediaViewer » internet_explorer 11.0,beta,Windows 7,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 23 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=internet_explorer%2011.0,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%207,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:51:34] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 26 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:52:24] Project selenium-MobileFrontend-2016-04-25 » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 25 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend-2016-04-25/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:52:37] Project selenium-PdfHandler » firefox,test,Linux,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 9 sec: https://integration.wikimedia.org/ci/job/selenium-PdfHandler/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:53:06] Project selenium-MultimediaViewer » chrome,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 27 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:53:15] Project selenium-RelatedArticles » chrome,beta-desktop,Linux,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 43 sec: https://integration.wikimedia.org/ci/job/selenium-RelatedArticles/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta-desktop,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:53:38] Project selenium-RelatedArticles » chrome,beta-mobile,Linux,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 1 min 7 sec: https://integration.wikimedia.org/ci/job/selenium-RelatedArticles/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta-mobile,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:53:49] PROBLEM - Puppet run on deployment-mediawiki02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:54:01] Project selenium-MultimediaViewer » internet_explorer 9.0,beta,Windows 7,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 28 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=internet_explorer%209.0,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%207,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:54:37] Project selenium-WikiLove » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 1 min 56 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:56:06] Yippee, build fixed! [17:56:07] Project beta-scap-eqiad build #99870: 09FIXED in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99870/ [17:57:43] twentyafterfour: This would https://github.com/phacility/phabricator/blob/c30fe65ee9c8a4cad3fdbd09032af926384f847f/src/applications/config/check/PhabricatorBinariesSetupCheck.php#L10 also need updating since where is only supported in cmd not in git for windows. [17:57:43] Project selenium-QuickSurveys » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 5 min 0 sec: https://integration.wikimedia.org/ci/job/selenium-QuickSurveys/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [17:57:56] Project selenium-MobileFrontend-2016-04-25 » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 31 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend-2016-04-25/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [18:02:30] 05Continuous-Integration-Scaling, 03releng-201516-q4, 13Patch-For-Review, 07WorkType-NewFunctionality: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203#2236177 (10hashar) Have to replace elements ubuntu-minimal, simple-init with simply ubuntu and: ``` Cloud-init v.... [18:06:01] (03PS1) 10Hashar: dib: fix cloud-init on Trusty to use Ec2 [integration/config] - 10https://gerrit.wikimedia.org/r/285218 (https://phabricator.wikimedia.org/T133203) [18:06:17] (03CR) 10Hashar: [C: 032] dib: fix cloud-init on Trusty to use Ec2 [integration/config] - 10https://gerrit.wikimedia.org/r/285218 (https://phabricator.wikimedia.org/T133203) (owner: 10Hashar) [18:07:00] (03Merged) 10jenkins-bot: dib: fix cloud-init on Trusty to use Ec2 [integration/config] - 10https://gerrit.wikimedia.org/r/285218 (https://phabricator.wikimedia.org/T133203) (owner: 10Hashar) [18:11:11] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Unplanned-Sprint-Work, 07FR-ActiveMQ, 03Fundraising Sprint Hermit Crab Husbandry: Run PHPUnit on PHP-Queue repo - https://phabricator.wikimedia.org/T133574#2236211 (10awight) [18:13:22] (03PS1) 10Awight: Run PHPUnit for PHP-Queue repo [integration/config] - 10https://gerrit.wikimedia.org/r/285219 (https://phabricator.wikimedia.org/T133574) [18:16:01] (03CR) 10Hashar: [C: 032] Run PHPUnit for PHP-Queue repo [integration/config] - 10https://gerrit.wikimedia.org/r/285219 (https://phabricator.wikimedia.org/T133574) (owner: 10Awight) [18:16:38] awight: speed merging :-} [18:16:48] (03Merged) 10jenkins-bot: Run PHPUnit for PHP-Queue repo [integration/config] - 10https://gerrit.wikimedia.org/r/285219 (https://phabricator.wikimedia.org/T133574) (owner: 10Awight) [18:16:56] O_O hashar: Thanks! [18:17:13] awight: and deployed! you can 'recheck' ! [18:18:26] wowza, here goes [18:19:24] https://gerrit.wikimedia.org/r/#/c/284987/ not detecting the love [18:19:41] nvm, the job just ran, wheee! [18:20:02] hmm [18:20:02] ah [18:20:46] 18:20:07 [62.0MB/36.26s] Package amazonwebservices/aws-sdk-for-php is abandoned, you should avoid using it. Use aws/aws-sdk-php instead. [18:21:01] hmm that is just a notice apparently [18:21:13] the real issue being the composer.json lacks a 'test' command [18:22:07] awight: https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#PHP has some boiler plate [18:22:29] Thanks! Pasting that in and cleaning up the require-dev fu [18:23:46] RECOVERY - Puppet run on integration-puppetmaster is OK: OK: Less than 1.00% above the threshold [0.0] [18:25:26] andrewbogott: good morning! Do you have some spare time for some Nodepool tweaking? :-} [18:25:41] hashar: not for a few days I fear [18:25:45] been working on adding a Trusty image to the pool so we can run hhvm / PHP 5.5 jobs on it https://gerrit.wikimedia.org/r/#/c/285178/ [18:26:03] it is a couple copy pasting and s/jessie/trusty/ ;-} [18:33:05] 05Continuous-Integration-Scaling, 03releng-201516-q4, 07Blocked-on-Operations, 13Patch-For-Review, 07WorkType-NewFunctionality: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203#2236277 (10hashar) **Summary** Provisionnent a Trusty image was more or less straight... [18:33:56] RECOVERY - Puppet run on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:39:00] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 03releng-201516-q4, 07WorkType-NewFunctionality: [keyresult] Migrate php composer (Zend and HHVM) CI jobs to Nodepool - https://phabricator.wikimedia.org/T119139#2236291 (10hashar) @joe provided HHVM on Jessie (T125821... [18:52:54] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UploadsLink, 10Wikimedia-Extension-setup, 13Patch-For-Review: Set up UploadsLink extension on the beta cluster - https://phabricator.wikimedia.org/T131844#2236324 (10Legoktm) 05Open>03Resolved Confirmed working on beta. [18:57:18] (03PS3) 10Legoktm: make-wmf-branch: Branch UploadsLink extension [tools/release] - 10https://gerrit.wikimedia.org/r/281644 (https://phabricator.wikimedia.org/T130018) (owner: 10Rillke) [18:57:29] (03CR) 10Legoktm: [C: 032] make-wmf-branch: Branch UploadsLink extension [tools/release] - 10https://gerrit.wikimedia.org/r/281644 (https://phabricator.wikimedia.org/T130018) (owner: 10Rillke) [18:58:09] ostriches, twentyafterfour: FYI https://gerrit.wikimedia.org/r/#/c/281644/ I'd like that extension to get into this weeks branch so when we turn it on next week I don't have to backport it just to scap [18:58:34] *looking* [18:58:35] (03Merged) 10jenkins-bot: make-wmf-branch: Branch UploadsLink extension [tools/release] - 10https://gerrit.wikimedia.org/r/281644 (https://phabricator.wikimedia.org/T130018) (owner: 10Rillke) [18:59:09] legoktm: All good, it'll be in tomorrow. [18:59:33] thanks :) [19:01:33] (03PS1) 10Hashar: archive integration/phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/285230 [19:03:53] once https://phabricator.wikimedia.org/rMSCA5cedf363eb4371a436a7342df5340726d3d4ea16 gets deployed, it will be easier to add extensions [19:05:48] (03CR) 10Hashar: [C: 032] archive integration/phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/285230 (owner: 10Hashar) [19:06:00] 10MediaWiki-Codesniffer, 10Fundraising-Backlog, 07FR-Smashpig: Write mutant code style config for SmashPig, or fully adopt MediaWiki style - https://phabricator.wikimedia.org/T133576#2236348 (10awight) [19:06:07] 10MediaWiki-Codesniffer, 10Fundraising-Backlog, 07FR-Smashpig: Write mutant code style config for SmashPig, or fully adopt MediaWiki style - https://phabricator.wikimedia.org/T133576#2236360 (10awight) p:05Triage>03Low [19:07:35] (03Abandoned) 10Hashar: dib: always install cloud-init [integration/config] - 10https://gerrit.wikimedia.org/r/285194 (https://phabricator.wikimedia.org/T133203) (owner: 10Hashar) [19:07:56] (03Merged) 10jenkins-bot: archive integration/phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/285230 (owner: 10Hashar) [19:25:49] (03PS1) 10Hashar: dib: provide composer [integration/config] - 10https://gerrit.wikimedia.org/r/285236 [19:26:06] (03PS2) 10Hashar: dib: provide composer [integration/config] - 10https://gerrit.wikimedia.org/r/285236 (https://phabricator.wikimedia.org/T128092) [19:27:41] 05Continuous-Integration-Scaling, 10OOjs-UI, 13Patch-For-Review, 07WorkType-NewFunctionality: Provide composer on the nodepool servers so OOjs UI can use it in the npm job - https://phabricator.wikimedia.org/T128092#2236531 (10hashar) a:03hashar Need some puppet refactoring then we will be able to provis... [19:31:14] PROBLEM - Host integration-trusty-1026 is DOWN: CRITICAL - Host Unreachable (10.68.17.98) [19:31:45] 05Continuous-Integration-Scaling, 10OOjs-UI, 07Blocked-on-Operations, 13Patch-For-Review, 07WorkType-NewFunctionality: Provide composer on the nodepool servers so OOjs UI can use it in the npm job - https://phabricator.wikimedia.org/T128092#2236550 (10hashar) Pending merge in puppet.git of the following... [19:42:26] I'll be converting a couple of trebuchet deploys to scap3 (T116340), and just encountered a need to have the version of a deployed jar file be conditional on a variable (that is canonical in puppet), what's the best way to do this? [19:42:27] T116340: Deploy Cassandra with scap3 - https://phabricator.wikimedia.org/T116340 [19:43:28] deploy them both, and have puppet create a link to the right version? (this is how i was planning to do it with given the status quo) [19:44:14] s/with given/given/ [19:45:46] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UploadsLink, 10Wikimedia-Extension-setup, 13Patch-For-Review: Set up UploadsLink extension on the beta cluster - https://phabricator.wikimedia.org/T131844#2236624 (10Rillke) >>! In T131844#2236324, @Legoktm wrote: > Confirmed working on beta. Thanks! [19:45:49] 05Continuous-Integration-Scaling, 03releng-201516-q4, 07Blocked-on-Operations, 13Patch-For-Review, 07WorkType-NewFunctionality: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203#2236625 (10hashar) 05Open>03Resolved Nodepool processed: ``` 2016-04-25 19:41 INFO... [19:45:51] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 03releng-201516-q4, 07WorkType-NewFunctionality: [keyresult] Migrate php composer (Zend and HHVM) CI jobs to Nodepool - https://phabricator.wikimedia.org/T119139#2236627 (10hashar) [19:46:19] !log Nodepool now has a couple trusty instances intended to experiment with Zend 5.5 / HHVM migration . https://phabricator.wikimedia.org/T133203#2236625 [19:46:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [19:54:35] Can someone help me at this: I want to build a PHP-Unit test for my project (at my private jenkins), and I want to let this job run at a fresh wiki. How can I use that template? [19:55:17] hashar: Fancy! [19:56:07] Is there docs somewhere about how to make jenkins run phpunit tests for the extension I just recently wrote? [20:04:31] 10Continuous-Integration-Infrastructure: Investigate installing php5.3 on trusty and/or debian instance - https://phabricator.wikimedia.org/T103786#2236708 (10hashar) p:05High>03Low Low priority. Will take care of Zend 5.5 / HHVM first. 5.3 can wait since it is barely used any more. [20:08:22] (03PS3) 10Hashar: dib: composer and Zend PHP for mw on Trusty [integration/config] - 10https://gerrit.wikimedia.org/r/285236 (https://phabricator.wikimedia.org/T119139) [20:09:12] bawolff: there is some at https://www.mediawiki.org/wiki/Continuous_integration/Entry_points [20:09:22] bawolff: usually people copy paste mediawiki/extensions/BoilerPlate [20:09:42] thanks. Something to copy/paste is basically what I'm looking for [20:09:44] bawolff: and integration/config.git zuul/layout.yaml is used to configure which jobs to trigger [20:09:49] so yeah BoilerPlate [20:10:10] the whole idea is to have tests run by simply having Jenkins run: npm install && npm test ; composer install && composer test [20:10:20] so you can define the utilities you want to run directly in your repo [20:10:46] then the CI conf is "just" about having the "npm" and "composer" jobs to trigger which usually is a single copy paste [20:11:52] urandom: I *think* you're referring to what we talked about at the deployment-cabal meeting this morning with mobrovac . Either way: you could have puppet handle the version linking, or you could use scap3 environments for this. [20:12:36] James_F: yeah more or less. The npm migration to Nodepool is pretty much finished so I am happily focusing on PHP this week ;-} [20:13:15] * James_F is so looking forward to it. [20:20:51] 10Continuous-Integration-Infrastructure, 07Blocked-on-Operations, 13Patch-For-Review: Disable HHVM fcgi server on CI slaves - https://phabricator.wikimedia.org/T126594#2236837 (10hashar) Pending review and merge of puppet patches by #operations https://gerrit.wikimedia.org/r/#/c/269946/ https://gerrit.wikim... [20:22:06] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 03releng-201516-q4, and 2 others: [keyresult] Migrate php composer (Zend and HHVM) CI jobs to Nodepool - https://phabricator.wikimedia.org/T119139#2236840 (10hashar) [20:22:08] 10Continuous-Integration-Infrastructure, 07Blocked-on-Operations, 13Patch-For-Review: Disable HHVM fcgi server on CI slaves - https://phabricator.wikimedia.org/T126594#2236841 (10hashar) [20:22:23] thcipriani: auh, yeah mobrovac has been involved in the discussion on this side, so probably so [20:22:47] (03PS1) 10Brian Wolff: Attempt to make LoginNotify run phpunit tests [integration/config] - 10https://gerrit.wikimedia.org/r/285252 [20:23:02] thcipriani: yeah, rt'd the fm, and environments was the only thing that looked close [20:24:00] thcipriani: but that would duplicate the How/What wouldn't it? One half of the equation would be in puppet, the other scap? [20:26:05] depends on what you want, certainly. Using scap probably gives more flexibility to the deployer at run-time [20:26:42] if that's a desirable outcome. If you don't want to think about which machines run in which environment, and that'll remain farily static, it may be better to use puppet to manage it. [20:27:08] thcipriani: well, it's a migration scenario, really, but one that might drag out over months [20:27:46] 06Release-Engineering-Team, 06Team-Practices, 10Developer-Relations (Jul-Sep-2016): Developer Summit 2017: Work with TPG and RelEng on solution to event documenting - https://phabricator.wikimedia.org/T132400#2236846 (10Rfarrand) p:05Low>03Lowest [20:27:47] thcipriani: we want to push out a newer version of Cassandra to a subset of all machines that run Cassandra, and need to do a number of (Puppet) things, conditionally [20:28:49] thcipriani: sounds like environments though would mean that for that subset, we'd need to deploy this dependency, conditionally, to match what happened in puppet [20:29:08] which sounds error prone [20:29:20] 10Continuous-Integration-Infrastructure, 07Blocked-on-Operations, 13Patch-For-Review, 07Puppet: mediawiki jobs fail intermittently with "mw-teardown-mysql.sh: Can't revoke all privileges" - https://phabricator.wikimedia.org/T126699#2236856 (10hashar) *summary* This task has a long history, the puppet patc... [20:29:30] which is fine if that's the case, i just wanted to make sure there wasn't something Better :) [20:29:59] :D [20:30:47] the onus for deploying to a subset would be on the deployer. I think a good way to set it up may be to not have a "default" environment. Rather just have an old or a new environment. [20:31:35] that way, a deployer would have to specify whether to run: scap deploy -E old OR scap deploy -E new to deploy explicitly to a specific subset of machines. [20:32:01] er...just `deploy -E old` for now [20:32:58] 10Continuous-Integration-Infrastructure: Investigate installing php5.3 on trusty and/or debian instance - https://phabricator.wikimedia.org/T103786#2236864 (10hashar) [20:33:00] 10Continuous-Integration-Infrastructure, 07Blocked-on-Operations, 13Patch-For-Review: Make /usr/bin/php a wrapper that picks the right PHP version on CI slaves - https://phabricator.wikimedia.org/T126211#2236861 (10hashar) 05Resolved>03Open We still need the puppet patch https://gerrit.wikimedia.org/r/#/... [20:33:38] 10Continuous-Integration-Infrastructure, 07Blocked-on-Operations, 13Patch-For-Review: Make /usr/bin/php a wrapper that picks the right PHP version on CI slaves - https://phabricator.wikimedia.org/T126211#2007557 (10hashar) a:05Legoktm>03None [20:33:54] 10Continuous-Integration-Infrastructure, 07Blocked-on-Operations, 13Patch-For-Review: Disable HHVM fcgi server on CI slaves - https://phabricator.wikimedia.org/T126594#2236867 (10hashar) a:05hashar>03None [20:40:04] hashar: is there a docu what I have to copy for a jenkins job, which should create a new wiki, and running the tests at that wiki? [20:40:41] Luke081515: we do that for qunit jobs [20:41:03] Luke081515: and mwext-selenium something [20:41:41] so in theory there is already a bunch of acres around but it is not straightforward :/// [20:42:14] thcipriani: scap deploy --to old [20:42:20] hashar: hm, ok. is there an example PHP unit test I can copy, and change so that it works for me? [20:42:20] hashar: Jenkins 2.0 was released. [20:42:42] thcipriani: bonus beers if you ship bash completion script ;-} [20:42:55] paladox: neat!!!! :)) [20:43:21] Yep, should we open a task about upgrading jenkins [20:43:28] since it is backwords compatible. [20:43:28] hashar: :) [20:44:03] paladox: na [20:44:06] Ok [20:44:18] paladox: I am willing to make the upgrade to Jenkins 2.x to be blocked until we kill gallium server [20:44:22] and move the current jenkins somewhere [20:44:32] Oh ok [20:44:36] from there we can setup yet another box with a standalone Jenkins 2.0 in parallel [20:44:41] and deploy jobs there [20:44:50] ideally we would have several jenkins in parallel [20:44:57] Ok [20:45:03] a use case would be to have all browser tests on a standalone jenkins [20:45:16] maybe a private jenkins for release stuff (like cutting mediawiki branches / releasing stuff etc) [20:45:16] Ok [20:45:33] and then a jenkins 1.x for the rest of CI which eventually one day will get to 2.x [20:45:36] (I just made that up) [20:45:43] Luke081515: jjb/mediawiki-extensions.yaml: name: 'mwext-mw-selenium' [20:46:15] Ok [20:46:18] ok, thx [20:46:44] Luke081515: that job clones mediawiki + vendor + Vector skin, install run a bunch of random script you might not need [20:47:07] Luke081515: and invokes mw-selenium which is a macro defined somewhere else. That ones run bundler install && bundle exec rake selenium or something like that [20:47:07] ok, thanks [20:47:40] Luke081515: with env variables to point the test suite to point to the locally available mediawiki. Something like http://localhost:9314/mwext-mw-selenium-123456/w/index.php [20:48:03] Luke081515: in the same file there is 'mwext-qunit' job which is similar [20:48:25] Luke081515: you might want to just copy that mwext-qunit one and replace the '- qunit-karma' with your custom command [20:48:27] something like: [20:48:33] - shell: | [20:48:38] ./boo.sh [20:49:06] Luke081515: you will want to first source: . /srv/deployment/integration/slave-scripts/bin/mw-set-env-localhost.sh [20:49:34] which really just does: [20:49:34] export MW_SERVER="http://localhost:9412" [20:49:35] export MW_SCRIPT_PATH="/$BUILD_TAG" [20:49:49] that is where the mediawiki is available where BUILD_TAG is provided by Jenkins itself [20:49:55] it is unique per job/build [20:51:39] Luke081515: if you figured out how to deploy a job in Jenkins, while you are iterating the development of a job I highly recommend you tie it to a specific slave for example: node: integration-slave-trusty-1012 [20:52:17] Luke081515: this way jenkins always execute the job on the same node, that will save you a bunch of time since when executing on a different slave the job would clone mediawiki + vendor etc [20:52:38] though Jenkins tries to be smart and attempt to reexecute a job on a slave the job has previously run on [20:53:03] rushing to bed for now. *wave* : -} [20:58:33] !log updated OCG to version 58a720508deb368abfb7652e6a8c7225f95402d2 [20:58:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:01:50] awight: your patch on https://gerrit.wikimedia.org/r/#/c/284987/2 can't be triggered because Zuul is waiting for Gerrit [21:01:54] ostriches: seems Gerrit task queue is full :( [21:02:03] Sonofa.... [21:02:05] One sec. [21:02:16] blames gerrit-patch-uploaded [21:02:25] That's weird... [21:02:36] spotted that one because of a CI change not processing, it is blocked on Zuul-merger trying to git clone wikimedia/fundraising/php-queue [21:02:45] I may have clogged the intertubes by jiggling the toilet handle too many times. [21:03:24] Specifically, I kicked jenkins-bot off of the reviewers list for my patch, then readded the group (?) manually and commented with "recheck" until I turned blue in the face [21:04:36] 165ba736 waiting .... 21:01:29.792 git-upload-pack '/wikimedia/fundraising/php-queue' (jenkins-bot) [21:04:43] awight: ^^^ that is Zuul blocked and waiting for Gerrit [21:04:55] it is blocked for whatever unrelated reason :-} [21:06:48] ostriches: and I thought "Sonofa" was some spanish slang .... [21:07:16] imma kill the stuck ones [21:07:22] +1 :-} [21:07:56] Now, do the waiting... ones unstuck? [21:08:06] not yet [21:09:26] seen one task processed [21:10:13] ostriches: that is just those upload pack of core for gerrit-patch-uploader [21:10:17] maybe it is slow as hell [21:11:05] twentyafterfour: What do you think about this picture for the phabricator jenkins account? :D https://integration.wikimedia.org/ci/static/f217739b/images/rage.png [21:11:11] Does gerrit-patch-uploader re-clone? [21:11:19] Or does it use a fresh base somewhere to not waste time? [21:11:27] no idea [21:11:51] thcipriani: hey, I have trouble connecting to beta instances via user deploy-service again. I checked keyholder says it's armed, I checked the fingerprints and they match but this weekend it was giving me "Agent admitted failure to sign using the key." and right now it asks password [21:12:06] oh [21:12:09] ostriches: queue completed [21:12:13] I have ran them via -v mode and I have logs of that [21:12:56] (hence "deploy" via scap3 fails) [21:13:36] awight: gerrit unlocked and here is jenkins +2 https://gerrit.wikimedia.org/r/#/c/284987/2 :-) [21:13:39] Amir1: there is a patch on beta that changes a bit of how keyholder runs—I may have to tag in twentyafterfour for this instance. [21:13:54] ostriches: thanks ! [21:13:58] heading to bed for really now [21:14:14] * thcipriani looks at beta keyholder [21:14:16] hashar: Thanks again! [21:14:21] zuul, y u no picking up my merge for 15 mins? [21:14:27] thanks [21:14:46] I saw some changes but I couldn't figure it out how I need fix this issue [21:15:21] ... [21:15:40] ah, ok, it finally picked it up. I should have complained sooner [21:16:14] Amir1: over the weekend it was broken because I had cherry-picked various stages of incomplete patch on beta. Now it should be working though [21:16:26] hmm...yeah, I definitely am prompted for pass as service-deploy on sca01 [21:16:28] twentyafterfour: thanks [21:16:30] Amir1: Can you tell me specifically what is failing [21:16:53] "SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l deploy-service deployment-sca01.deployment-prep.eqiad.wmflabs" [21:17:12] (you can try it with "deployment-ores-web" as well) [21:17:50] * twentyafterfour runs puppet to be sure it's updated [21:18:24] twentyafterfour: it is great to see progress on SSH, it's a huge burden. Is there anything I need to do for this scap3 config? https://gerrit.wikimedia.org/r/280403 [21:18:42] twentyafterfour: hi could you merge https://gerrit.wikimedia.org/r/#/c/285138/ please. [21:18:45] And please could you upload this upstream [21:18:55] uhm [21:18:55] hmm key doesn't match on the target in the case of sca01 [21:19:24] Could you also forward https://phabricator.wikimedia.org/rARC4e2619326961a9782d74a5cb271b2fae85d22986 to upstream. [21:19:25] thcipriani: puppet hasn't ran because it's using the old public_key_source [21:19:37] paladox: hang on dealing with scap right now [21:19:41] Ok [21:20:20] Amir1: I don't see puppet/modules/ores/manifests/scapdeploy.pp I take it this is cherry-picked on beta? [21:20:27] yup [21:20:41] Project browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #248: 04FAILURE in 4 min 39 sec: https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/248/ [21:20:41] Project beta-update-databases-eqiad build #8122: 04FAILURE in 40 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/8122/ [21:20:51] (the patch I just mentioned above) [21:21:04] ok it needs to use public_key_content => keyholder_key('key_name') instead of public_key_source [21:21:57] okay [21:22:04] I do it right now [21:22:10] and then cherry pick again [21:25:55] Amir1: actually, [21:25:56] replace public_key_source with this: [21:25:58] key_name => keyholder_pubkey('servicedeploy') [21:26:06] sorry I had it wrong ... [21:26:06] Great [21:26:22] I was actually asking what is the name [21:27:08] twentyafterfour: do you have any insight for https://phabricator.wikimedia.org/T129736 ? [21:27:40] the name is defined in hieradata/labs/deployment-prep/common.yaml [21:28:19] SMalyshev: sure I can make a bot account for you [21:28:42] twentyafterfour: the account is created, we just couldn't figure out how to use it [21:28:54] i.e. I have the API token but arc refuses it [21:30:29] SMalyshev: ah, well here's the thing - phabricator recently changed the format of api keys/tokens so any code that doesn't know how to use the new tokens won't work (e.g. most 3rd party api clients that haven't been updated in a year) [21:30:45] but arc should work, do you have a recent version of arcanist? [21:31:24] twentyafterfour: yes, I tried one both from git and from apt, same error - it says token should start with cli- [21:33:28] Valid API tokens should begin "cli-" and be 32 characters long. Make sure you visited the correct URI and copy/pasted the token correctly. [21:34:58] twentyafterfour: I'm trying to do it [21:34:58] https://gerrit.wikimedia.org/r/#/c/284418/22/modules/scap/manifests/target.pp [21:35:28] but per here, I think it should be key_name => 'servicedeploy', [21:35:34] Am I wrong? [21:35:57] (since it actually does the keyholder_pubkey($key_name, true), ) [21:37:07] SMalyshev: commented on the ticket [21:37:31] amir1 you're right [21:37:43] Amir1: sorry I was confused :-/ [21:38:02] no problem at all, I learned how to deal with this :) [21:38:09] Thanks for the tip [21:38:16] let me give it a try [21:40:46] twentyafterfour: how do I add token to .arcrc? [21:40:56] is there some designated property name? [21:48:45] SMalyshev: https://phabricator.wikimedia.org/P2956 [21:50:18] (03PS1) 10Addshore: Enable qunit and JSHint jobs on RevisionSlide [integration/config] - 10https://gerrit.wikimedia.org/r/285287 (https://phabricator.wikimedia.org/T133282) [21:55:18] (03CR) 10Paladox: [C: 04-1] "We are switching all jshint and jsonlint tests to npm." [integration/config] - 10https://gerrit.wikimedia.org/r/285287 (https://phabricator.wikimedia.org/T133282) (owner: 10Addshore) [21:55:27] twentyafterfour: it seems the public key in puppetmaster doesn't match with the armed key in the keyholder in tin [21:55:42] fingerprint of /var/lib/git/labs/private/files/ssh/tin/servicedeploy_rsa.pub [21:55:52] e6:d0:61:5e:e5:c7:5d:2d:3e:8e:c8:a5:eb:f3:c2:63 [21:56:00] (in puppetmaster) [21:56:03] and in tin [21:56:08] Amir1: that's because it no longer uses the file at that location [21:56:12] it generates the keys automatically [21:56:36] hmm [21:56:42] the one in keyholder is the one that should be returned by keyholder_pubkey(...) [21:56:57] but sca01 doesn't have the key in /etc/ssh/userkeys/deploy-serviice [21:56:58] this saves you from having to deal with the keys at all [21:57:13] scap::target should take care of that [21:57:30] if you pass key_name then it'll get installed [21:57:54] twentyafterfour: ok, looks like it's working now, thanks! [21:58:06] hmm [21:58:17] so let me run a puppet agent [21:59:21] SMalyshev: awesome! :) [21:59:31] Amir1: https://gerrit.wikimedia.org/r/#/c/284418/22/modules/scap/manifests/target.pp,cm ... yeah run puppet it should set up the userkey [21:59:35] twentyafterfour: "Error: Could not retrieve catalog from remote server: Error 400 on SERVER: keyholder_pubkey(): Wrong number of arguments given (2 for 1) at /etc/puppet/modules/scap/manifests/target.pp:90 on node deployment-sca01.deployment-prep.eqiad.wmflabs" [21:59:41] puppet agent returned this [22:00:32] it seems you need to remove "true" option [22:01:01] Amir1: yeah you're right, fixing... [22:03:38] Amir1: fixed [22:03:48] awesome [22:03:57] did you cherry-picked that? [22:04:01] yeah [22:04:30] I removed the old one and cherry-picked /23 [22:04:45] it's on top of your patch now so you'll have to rebase if you update yours [22:04:46] "Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter public_key_source at /etc/puppet/modules/service/manifests/deploy/scap.pp:33 on node deployment-sca01.deployment-prep.eqiad.wmflabs" [22:05:34] I'm not sure why it checks service module [22:05:43] but it seems this one needs fix [22:07:04] twentyafterfour: ^ [22:09:32] PROBLEM - Puppet run on deployment-ms-be02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:20:28] Yippee, build fixed! [22:20:29] Project beta-update-databases-eqiad build #8123: 09FIXED in 28 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/8123/ [22:24:56] I need to get some sleep, talk to you tomorrow :) [22:24:58] o/ [22:29:05] legoktm: Hi sorry i think i askesked this question but should we update to composer 1.0.2 instead of doing it to 1.0.0 and then 1.0.2 since 1.0.2 is a stable release and includes some fixes. [22:35:50] (03PS12) 10Paladox: Update composer to 1.0.2 stable [integration/composer] - 10https://gerrit.wikimedia.org/r/283852 (https://phabricator.wikimedia.org/T125343) [22:36:22] (03CR) 10jenkins-bot: [V: 04-1] Update composer to 1.0.2 stable [integration/composer] - 10https://gerrit.wikimedia.org/r/283852 (https://phabricator.wikimedia.org/T125343) (owner: 10Paladox) [22:36:39] (03PS2) 10Addshore: Enable JSHint check job on RevisionSlide [integration/config] - 10https://gerrit.wikimedia.org/r/285287 (https://phabricator.wikimedia.org/T133282) [22:37:09] (03PS1) 10Addshore: Enable qunit test template on RevisionSlider [integration/config] - 10https://gerrit.wikimedia.org/r/285300 [22:38:49] (03PS13) 10Paladox: Update composer to 1.0.2 stable [integration/composer] - 10https://gerrit.wikimedia.org/r/283852 (https://phabricator.wikimedia.org/T125343) [22:40:37] (03CR) 10Paladox: [C: 031] Enable qunit test template on RevisionSlider [integration/config] - 10https://gerrit.wikimedia.org/r/285300 (owner: 10Addshore) [22:40:59] (03CR) 10Paladox: [C: 031] "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/285287 (https://phabricator.wikimedia.org/T133282) (owner: 10Addshore) [22:42:37] it looks to me like service::deploy::scap is unnecessary, it just wraps scap::target [22:57:45] PROBLEM - Puppet run on deployment-sca02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [23:38:40] 10Deployment-Systems, 10scap, 06Discovery, 10Wikidata, and 2 others: Deploy wdqs with scap3 - https://phabricator.wikimedia.org/T129144#2237412 (10Smalyshev) Any guidance on this? [23:46:50] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 03Scap3, 06Operations, 13Patch-For-Review: Automate the generation deployment keys (keyholder-managed ssh keys) - https://phabricator.wikimedia.org/T133211#2237449 (10mmodell) Ok, I discovered that ssh-add will reuse the same passphrase on multiple key... [23:55:58] Project beta-scap-eqiad build #99909: 04FAILURE in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99909/