[00:00:07] hashar: Eh. I'm not sure that we want to discourage extensive unit tests. [00:00:21] SMalyshev: aloha yeah hmm. New repos in Gerrit do not have job set in CI. The CI part is done independently :-( [00:00:37] hashar: It seems to be stuck on wikimedia/fundraising/crm which has been merged and has been fully tested. [00:00:39] hashar: ah! that may explain it [00:00:42] https://integration.wikimedia.org/zuul/ [00:00:51] hashar: so how it should be done? [00:01:06] SMalyshev: so it really all depends what you want to run for wikidata/query/gui . You could get maven :D [00:01:10] hashar: does it need to be in layout.xml or something else? [00:01:15] yup [00:01:25] hashar: no, don't need maven, need npm [00:01:26] SMalyshev: You could either file a task or you can checkout integration/config and edit the layout.yaml file. [00:01:33] (03CR) 10Subramanya Sastry: [C: 04-1] "Let us actually wait to confirm this is the source." [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/269335 (owner: 10Subramanya Sastry) [00:01:39] paladox: ah, ok, will do [00:01:40] SMalyshev: so we have a bunch of basic entry points described at https://www.mediawiki.org/wiki/Continuous_integration/Entry_points [00:02:08] SMalyshev: but really for you, there is a jenkins job named 'npm' already that would grab your patch, run npm install && npm test and report back [00:02:27] paladox can assist ;-} he has a ton of experience with the zuul/layout.yaml file [00:03:14] James_F: oh I dont want to discourage more tests. The problem is that we blindly run them all. [00:03:35] James_F: say I propose a patch for Math ... the Scribunto LUA tests were being run which dont really make any sense [00:03:45] (03PS1) 10Smalyshev: Add wikidata/query/gui [integration/config] - 10https://gerrit.wikimedia.org/r/269338 [00:04:15] hashar: I think we should run all of the cross-dependency ones. [00:05:08] (03PS2) 10Smalyshev: Add wikidata/query/gui [integration/config] - 10https://gerrit.wikimedia.org/r/269338 [00:05:20] (But none of the others.) [00:05:27] I thought about splitting our tests in various tiers. true unit tests (dont even need mw core) mw integration (bring extensions as dependencies but dont run their tests) mw ext integrations (the mess we run) [00:05:31] hashar: something like this: https://gerrit.wikimedia.org/r/#/c/269338/1 ? [00:08:15] SMalyshev: yeah exactly [00:11:33] (03CR) 10Hashar: Add wikidata/query/gui (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/269338 (owner: 10Smalyshev) [00:11:46] oh I reviewed ps1 [00:12:08] (03CR) 10Hashar: [C: 032] Add wikidata/query/gui (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/269338 (owner: 10Smalyshev) [00:15:24] hashar: could you merge https://gerrit.wikimedia.org/r/269191 and https://gerrit.wikimedia.org/r/#/c/269188/ please [00:16:00] Also could you review https://gerrit.wikimedia.org/r/#/c/269310/ please [00:17:31] hashar https://gerrit.wikimedia.org/r/#/c/267548/ please [00:24:56] (03CR) 10Subramanya Sastry: "Looks like something else is going on .. but, this would be useful for when this is no longer in beta features." [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/269335 (owner: 10Subramanya Sastry) [00:31:00] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 7Documentation, 7Jenkins: Jenkins-mwext-sync needs documentation - https://phabricator.wikimedia.org/T62793#2010349 (10greg) [00:34:42] * James_F sighs at https://gerrit.wikimedia.org/r/#/c/269330/ holding up the queue despite being cancelled. [00:35:30] James_F: yeah found a sleep call in Zuul ... [00:36:04] I am killing zuul [00:36:19] !log killing zuul [00:36:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:37:12] !log live hacking Zuul code to have it stop sleeping() on force merge [00:37:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:37:26] hashar: will restarting zuul fix the lock in Jenkins that is keeping beta cluster from updating too? [00:37:28] cause I wake up in 5 hours, no point in wasting an hour trying to build a package :D [00:37:40] * James_F grins at hashar. [00:38:00] "status.json: Service Temporarily Unavailable". [00:38:00] Quite. [00:38:08] Beta cluster updates have been backed up all day which is not awesome heading into a branch cut [00:38:09] "status.json: hashar just killed me!". [00:39:13] !log gallium edited /usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/trigger/gerrit.py and modified: replication_timeout = 300 -> replication_timeout = 10 [00:39:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:41:10] 10Deployment-Systems, 10MediaWiki-Logging: Include dbname in fatal logs - https://phabricator.wikimedia.org/T62324#2010371 (10greg) [00:41:52] (03CR) 10Hashar: Add wikidata/query/gui [integration/config] - 10https://gerrit.wikimedia.org/r/269338 (owner: 10Smalyshev) [00:41:58] (03CR) 10Hashar: [C: 032] Add wikidata/query/gui [integration/config] - 10https://gerrit.wikimedia.org/r/269338 (owner: 10Smalyshev) [00:43:58] (03Merged) 10jenkins-bot: Add wikidata/query/gui [integration/config] - 10https://gerrit.wikimedia.org/r/269338 (owner: 10Smalyshev) [00:47:08] SMalyshev: sorry took a while but wikidata/query/gui can finally get npm to run in Jenkins [00:49:20] hashar: Ah, I see [00:49:56] SMalyshev: it is running at https://integration.wikimedia.org/ci/job/npm/51022/console [00:50:08] SMalyshev: the change adding a README change can no more be merged [00:50:14] because it has no npm entry point [00:50:24] so you will have to rebase it on top of the change adding the test files [00:50:30] and it should be all fine :-} [00:50:49] ah, ok [00:51:22] reviewing https://gerrit.wikimedia.org/r/#/c/269212/ :D [00:55:18] SMalyshev: your favorite frontend devs should be familiar with the npm / grunt and all the tasks in there [00:56:09] hashar: yes I think so [00:56:26] hashar: I didn't write these files, I just moved them from other repo [00:56:39] yeah that is what most folks ends up doing :-} [00:57:08] mediawiki/extensions/BoilerPlate is usually quite up-to-date [00:57:15] it is supposed to provide a skeleton to start an extension [00:58:15] jshint seems to be getting stuck [00:58:30] but not I see that old one actually didn't run it on any useful files [00:58:44] so I wonder if I should drop it [00:59:54] SMalyshev: i think jshint is attempting to process all the dependencies in node_modules [01:00:20] hashar: ahh, I forgot to add node_modules [01:00:27] yeah see my comments on ps1 [01:00:42] I think you need both node_modules/** in .jshintignore [01:00:54] and the !node_modules/** rule in the grunt file [01:01:11] not sure whether both are strictly needed though [01:01:27] hashar: we're back to a good place re merge gate timing, right? [01:01:56] yeah more or less [01:02:27] then go to bed [01:02:28] :) [01:02:31] will have to shuffle stuff around [01:02:38] yeah [01:02:38] tomorrow is going to be a long day [01:02:47] will nap in the morning, lunch and cut branch [01:02:50] * greg-g nods [01:02:56] nap definitely needed tomorrow :) [01:06:36] ok, seems to be fine now, I'll fine-tune it after [01:06:42] hashar: thanks for your help! [01:07:21] greg-g: yeah settled :( [01:07:30] hashar: g'night sir [01:07:38] SMalyshev: you are welcome. Happy hacking! [01:07:38] thanks for staying up late fighting this [01:07:56] bd808: sorry for the mess! Still have a ton of stuff to polish up [01:17:42] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: MediaWiki gate takes 20 minutes for extensions tests and 1.5 hour for at least a patch - https://phabricator.wikimedia.org/T126274#2010465 (10hashar) So in short: * 4 new 2 CPU Precise slaves have been added to help processing the php53 jobs * Scribu... [01:18:00] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 5Patch-For-Review: MediaWiki gate takes 20 minutes for extensions tests and 1.5 hour for at least a patch - https://phabricator.wikimedia.org/T126274#2010467 (10hashar) [01:31:36] "snowball effect of doom". [01:31:45] Nice. [01:41:48] something weird is going on with https://gerrit.wikimedia.org/r/#/c/269212/. It passed the verification, but the merge actually never happens [01:47:13] Yippee, build fixed! [01:47:14] Project beta-scap-eqiad build #89120: 09FIXED in 7 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/89120/ [01:49:09] SMalyshev: jenkins probably doesn't have merge rights in there? [01:49:35] legoktm: hmm... no idea. It's a standard repo set up like any other... any way to check it? [01:50:08] one sec [01:51:34] SMalyshev: fixed :) [01:51:36] legoktm: thanks, now seems to have worked! [01:52:14] legoktm: I wonder has something changed? I've never had to do that before with other repos and they worked fine [01:52:29] SMalyshev: for future reference, I went to https://gerrit.wikimedia.org/r/#/admin/projects/wikidata/query/gui,access and added Submit: JenkinsBot. [01:52:45] SMalyshev: most repositories are typically under mediawiki/* which automatically grants jenkins submit rights [01:52:47] legoktm: I see, thanks [01:53:06] legoktm: ahhh... interesting. That may be it, different namespace [01:53:31] yeah, wikidata/query/rdf has the same setting [01:54:08] so we could set that permission at the wikidata/query level or wikidata/ even and never have to worry about that again [01:54:13] but I'm not sure what's under those namespaces [01:57:11] 10Deployment-Systems, 3Scap3, 5Patch-For-Review: Make puppet provider for scap3 - https://phabricator.wikimedia.org/T113072#2010541 (10mmodell) Ok I updated the patch at https://gerrit.wikimedia.org/r/#/c/262742/18, works on labs with some minor hacks to get around environment-related peculiarities. [01:57:18] wikidata/ probably not a good idea [01:57:23] it's not all mine :) [01:57:37] wikidata/query maybe if I get more under that namespace [01:58:27] I just added you to the wikidata-query group, so you should have full ownership of the wikidata/query namespace and can adjust permissions as you wish :) [01:59:14] legoktm: thank you! [01:59:56] np [02:22:25] (03PS1) 10Legoktm: bin/php: Default to php5, because plain `php` would be itself [integration/jenkins] - 10https://gerrit.wikimedia.org/r/269355 [02:22:45] (03CR) 10Legoktm: [C: 032] bin/php: Default to php5, because plain `php` would be itself [integration/jenkins] - 10https://gerrit.wikimedia.org/r/269355 (owner: 10Legoktm) [02:23:32] (03Merged) 10jenkins-bot: bin/php: Default to php5, because plain `php` would be itself [integration/jenkins] - 10https://gerrit.wikimedia.org/r/269355 (owner: 10Legoktm) [02:26:36] Project beta-scap-eqiad build #89124: 04FAILURE in 7 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/89124/ [02:34:45] Yippee, build fixed! [02:34:46] Project beta-scap-eqiad build #89125: 09FIXED in 6 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/89125/ [02:42:33] (03PS2) 10Legoktm: Get integration/config out of the 'mediawiki' queue [integration/config] - 10https://gerrit.wikimedia.org/r/238988 [02:44:07] 10Continuous-Integration-Config: Get integration/config out of the mediawiki gate queue - https://phabricator.wikimedia.org/T126298#2010566 (10Legoktm) 3NEW a:3Legoktm [02:44:24] (03PS3) 10Legoktm: Get integration/config out of the 'mediawiki' queue [integration/config] - 10https://gerrit.wikimedia.org/r/238988 (https://phabricator.wikimedia.org/T126298) [02:49:21] (03CR) 10Legoktm: [C: 032] Get integration/config out of the 'mediawiki' queue [integration/config] - 10https://gerrit.wikimedia.org/r/238988 (https://phabricator.wikimedia.org/T126298) (owner: 10Legoktm) [02:52:23] (03Merged) 10jenkins-bot: Get integration/config out of the 'mediawiki' queue [integration/config] - 10https://gerrit.wikimedia.org/r/238988 (https://phabricator.wikimedia.org/T126298) (owner: 10Legoktm) [02:53:22] !log deploying https://gerrit.wikimedia.org/r/238988 [02:53:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [03:03:15] 10Continuous-Integration-Config, 5Patch-For-Review: Get integration/config out of the mediawiki gate queue - https://phabricator.wikimedia.org/T126298#2010595 (10Legoktm) 5Open>3Resolved [03:13:23] PROBLEM - Host deployment-mediawiki01 is DOWN: PING CRITICAL - Packet loss = 80%, RTA = 2709.90 ms [03:13:34] Project beta-scap-eqiad build #89129: 04FAILURE in 8 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/89129/ [03:15:45] RECOVERY - Host deployment-mediawiki01 is UP: PING OK - Packet loss = 0%, RTA = 1.54 ms [03:17:57] (03PS1) 10Legoktm: Set $PHP_BIN for hhvm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/269359 [03:18:14] (03CR) 10Legoktm: [C: 032] Set $PHP_BIN for hhvm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/269359 (owner: 10Legoktm) [03:19:16] (03Merged) 10jenkins-bot: Set $PHP_BIN for hhvm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/269359 (owner: 10Legoktm) [03:19:48] !log deploying https://gerrit.wikimedia.org/r/269359 [03:19:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [04:15:46] 3Scap3: Scap should touch symlinks when originals are touched - https://phabricator.wikimedia.org/T126306#2010701 (10Tgr) 3NEW [04:26:44] 3Scap3: Scap should touch symlinks when originals are touched - https://phabricator.wikimedia.org/T126306#2010715 (10bd808) Doing this for the general case would be computationally expensive. There is no "what links here" functionality in the POSIX filesystem standard. All files in /srv/mediawiki-staging would n... [04:36:47] Yippee, build fixed! [04:36:48] Project beta-scap-eqiad build #89133: 09FIXED in 7 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/89133/ [06:16:48] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 5Patch-For-Review: MediaWiki gate takes 20 minutes for extensions tests and 1.5 hour for at least a patch - https://phabricator.wikimedia.org/T126274#2010794 (10greg) Updated graph: {F3326090} [06:29:46] I know CI has a self-hosted puppetmaster, but is there a way I can test a puppet patch on just one slave? [07:13:15] I will let you know when I see hashar around here [07:13:15] @notify hashar [07:13:21] ty :) [09:35:51] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Flow, 10Wikidata, and 2 others: Wikidata QUnit broken on branch REL1_25 causing other extensions to fail - https://phabricator.wikimedia.org/T126073#2010908 (10JanZerebecki) Yes I'm fine with 5, too. [10:25:59] !log pooling in integration-slave-trusty-1018 [10:26:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:26:08] * legoktm waves at hashar [10:26:17] legoktm: morning [10:26:24] I am still half-asleep :D [10:27:12] hashar: I uploaded https://gerrit.wikimedia.org/r/269370, is there a good way I can test it on only one server? [10:27:45] yup [10:27:45] disable puppet on all slaves [10:27:51] cherry pick patch on puppet master [10:27:58] enable puppet on a single slave, see what happens [10:27:58] :( [10:28:11] integration-saltmaster instance would help [10:28:39] you can then do: sudo su - ; salt '*' cmd.run 'puppet agent -tv "testing https://gerrit.wikimedia.org/r/269370"' [10:28:45] that will prevent puppet agent from running [10:29:00] PROBLEM - Puppet failure on integration-slave-trusty-1017 is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [0.0] [10:29:10] no way around disabling puppet on all the slaves? :/ [10:29:56] PROBLEM - Puppet failure on integration-slave-trusty-1013 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [0.0] [10:30:15] not really [10:30:58] PROBLEM - Puppet failure on integration-slave-trusty-1018 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:36:01] RECOVERY - Puppet failure on integration-slave-trusty-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [10:36:05] RECOVERY - Puppet failure on integration-slave-precise-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [10:41:48] (03PS1) 10Hashar: bin/php: replace bash with lighter sh [integration/jenkins] - 10https://gerrit.wikimedia.org/r/269384 [10:41:50] (03PS1) 10Hashar: bin/php: use exec to replace shell with php command [integration/jenkins] - 10https://gerrit.wikimedia.org/r/269385 [11:04:02] RECOVERY - Puppet failure on integration-slave-trusty-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [11:09:14] (03CR) 10Hashar: "Thank you very much for this work!" [integration/config] - 10https://gerrit.wikimedia.org/r/268031 (https://phabricator.wikimedia.org/T125498) (owner: 10Legoktm) [11:09:31] 10Continuous-Integration-Config, 7WorkType-NewFunctionality: Zuul can only apply one parameter function. Prevent us from injecting both php55 and extension dependencies - https://phabricator.wikimedia.org/T125498#2011052 (10hashar) antoine-approve [11:11:41] (03CR) 10Hashar: "> Yes a cyclical dependency of components smells of bad architecture." [integration/config] - 10https://gerrit.wikimedia.org/r/268451 (owner: 10Dduvall) [11:12:07] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 5Patch-For-Review: MediaWiki gate takes 20 minutes for extensions tests and 1.5 hour for at least a patch - https://phabricator.wikimedia.org/T126274#2011054 (10JanZerebecki) >>! In T126274#2009859, @hashar wrote: > Later ideas: > * on... [11:13:37] !log disabling puppet on all *(trusty|precise)* slaves [11:13:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:15:13] hashar: is there documentation somewhere on how to cherry pick a patch onto the puppetmaster? [11:15:34] oh poor kunal :-D [11:15:41] legoktm: yeah let me find it [11:15:53] https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster [11:16:09] but really: [11:16:15] ssh integration-saltmaster.integration.eqiad.wmflabs [11:16:17] sudo su - [11:16:18] cd /var/lib/git/operations/puppet [11:16:30] git fetch && git cherry-pick FETCH_HEAD [11:17:16] the saltmaster is also the puppetmaster? [11:17:16] the puppet master is somehow looking for manifests in /var/lib/git/operations/puppet [11:17:21] aregh [11:17:23] stupid me [11:17:36] integration-puppetmaster.integration.eqiad.wmflabs [11:17:39] ok :) [11:17:52] I have split the roles on both integration and beta cluster [11:18:09] can't remember why but I guess a single instance could not handle both roles [11:20:42] !log enabling puppet just on integration-slave-trusty-1012 [11:20:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:20:53] !log cherry-picked https://gerrit.wikimedia.org/r/#/c/269370/ on integration-puppetmaster [11:20:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:25:22] :-) [11:26:15] $ update-alternatives --list php [11:26:23] /usr/bin/hhvm [11:26:23] /usr/bin/php5 [11:32:46] I'm reading that we have to use update-alternatives install [11:32:53] and now quickly writing puppet to do that... [11:33:27] oh [11:33:33] stupid update-alternatives [11:33:51] will try to get some european ops to review the patch as well [11:37:27] (03PS1) 10JanZerebecki: Bump Wikidata to wmf/1.27.0-wmf.13 [tools/release] - 10https://gerrit.wikimedia.org/r/269397 [11:39:53] hashar: uh, silly git question, how do I get rid of the cherry-pick now? [11:40:08] legoktm: look at the HEAD commit to confirm it is your ( git show HEAD ) [11:40:21] then rollback to commit before HEAD with : git reset --hard HEAD^ [11:40:40] (where HEAD^ means "commit before HEAD" ) [11:41:27] thanks [11:42:45] legoktm: you should get some sleep :-} [11:44:30] probably, but I'm so close :P [11:50:17] :-} [11:50:21] WOOHOO [11:50:24] (03CR) 10Hashar: [C: 032] Bump Wikidata to wmf/1.27.0-wmf.13 [tools/release] - 10https://gerrit.wikimedia.org/r/269397 (owner: 10JanZerebecki) [11:50:28] legoktm@integration-slave-trusty-1012:~$ php --version [11:50:28] PHP 5.5.9-1ubuntu4.14 (cli) (built: Oct 28 2015 01:34:46) [11:50:32] legoktm@integration-slave-trusty-1012:~$ PHP_BIN=hhvm php --version [11:50:32] HipHop VM 3.6.5 (rel) [11:50:40] legoktm@integration-slave-trusty-1012:~$ file /etc/alternatives/php [11:50:40] /etc/alternatives/php: symbolic link to `/srv/deployment/integration/slave-scripts/bin/php' [11:51:16] 6Release-Engineering-Team, 5WMF-deploy-2016-02-09_(1.27.0-wmf.13): MW 1.27.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T125596#2011124 (10hashar) Wikidata bumped to `wmf/1.27.0-wmf.13` by https://gerrit.wikimedia.org/r/#/c/269397/ [11:51:43] the only catch is if you set PHP_BIN=php it'll totally blow up, because it'll infinitely recurse [11:56:25] eek [11:56:54] legoktm: we might be able to catch that one somehow though [11:56:58] I think we can make our bin/php throw an error if $PHP_BIN is set to 'php' or '/usr/bin/php' [11:59:40] https://integration.wikimedia.org/ci/job/composer-package-php55/90/console used php55 and https://integration.wikimedia.org/ci/job/composer-package-hhvm/90/console used hhvm \o/ [11:59:45] yeah that is probably sufficient [11:59:46] (03Merged) 10jenkins-bot: Bump Wikidata to wmf/1.27.0-wmf.13 [tools/release] - 10https://gerrit.wikimedia.org/r/269397 (owner: 10JanZerebecki) [11:59:47] great! !!!! [12:00:49] ok, I'm going to re-enable puppet on all the slaves now [12:02:27] !log re-enabling puppet on all trusty/precise slaves [12:02:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:02:31] legoktm: kudos [12:02:45] if something goes wrong I will handle it (hopefully) :D [12:03:12] thanks, and good night :) [12:04:35] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Make /usr/bin/php a wrapper that picks the right PHP version on CI slaves - https://phabricator.wikimedia.org/T126211#2011129 (10Legoktm) p:5High>3Normal >>! In T126211#2010795, @gerritbot wrote: > Change 269370 had a related patch set uploaded (b... [12:07:50] 7Blocked-on-RelEng, 10Continuous-Integration-Infrastructure, 6Labs, 10Tool-Labs, 5Patch-For-Review: debian-glue tries to fetch obsolete package - https://phabricator.wikimedia.org/T125999#2011140 (10scfc) 5Open>3Resolved a:3scfc Now the package installation works (https://integration.wikimedia.org/... [12:08:03] 7Blocked-on-RelEng, 10Continuous-Integration-Infrastructure, 6Labs, 10Tool-Labs, 5Patch-For-Review: debian-glue tries to fetch obsolete package - https://phabricator.wikimedia.org/T125999#2011143 (10scfc) a:5scfc>3None [12:14:22] 7Blocked-on-RelEng, 10Continuous-Integration-Infrastructure, 6Labs, 10Tool-Labs, 5Patch-For-Review: debian-glue tries to fetch obsolete package - https://phabricator.wikimedia.org/T125999#2011150 (10hashar) 5Resolved>3Open I have manually updated the image somehow. Will have to ask around but we mos... [12:22:18] PROBLEM - SSH on integration-slave-precise-1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:14] PROBLEM - SSH on integration-slave-precise-1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:28] bah slaves dieing [12:33:00] !log all slaves dieing due to PHP looping [12:33:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:35:12] PROBLEM - SSH on integration-slave-trusty-1014 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:04] PROBLEM - SSH on integration-slave-precise-1012 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:18] PROBLEM - SSH on integration-slave-precise-1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:39:06] !log salt -v '*' cmd.run "bash -c 'cd /srv/deployment/integration/slave-scripts; git pull'" [12:39:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:39:56] PROBLEM - SSH on integration-slave-precise-1014 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:40:07] !log mass rebooting CI slaves from wikitech [12:40:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:40:18] PROBLEM - SSH on integration-slave-trusty-1017 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:42:45] PROBLEM - SSH on integration-slave-trusty-1011 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:43:21] PROBLEM - SSH on integration-slave-precise-1011 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:47] RECOVERY - SSH on integration-slave-precise-1014 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [12:45:03] RECOVERY - SSH on integration-slave-trusty-1014 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6 (protocol 2.0) [12:45:05] RECOVERY - SSH on integration-slave-precise-1003 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [12:45:11] RECOVERY - SSH on integration-slave-trusty-1017 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6 (protocol 2.0) [12:46:06] !log Mass testing php loop of death: salt -v '*slave*' cmd.run 'timeout 2s /srv/deployment/integration/slave-scripts/bin/php --version' [12:46:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:47:11] RECOVERY - SSH on integration-slave-precise-1002 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [12:47:33] RECOVERY - SSH on integration-slave-trusty-1011 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6 (protocol 2.0) [12:47:57] RECOVERY - SSH on integration-slave-precise-1012 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [12:48:11] RECOVERY - SSH on integration-slave-precise-1001 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [12:48:11] RECOVERY - SSH on integration-slave-precise-1011 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [12:50:55] PROBLEM - SSH on integration-slave-precise-1014 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:51:11] PROBLEM - SSH on integration-slave-trusty-1016 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:14:00] !log slave recurse infinitely doing /bin/bash -eu /srv/deployment/integration/slave-scripts/bin/mw-install-mysql.sh then loop over /bin/bash /usr/bin/php maintenance/install.php --confpath /mnt/jenkins-workspace/workspace/mediawiki-core-qunit/src --dbtype=mysql --dbserver=127.0.0.1:3306 --dbuser=jenkins_u2 --dbpass=pw_jenkins_u2 --dbname=jenkins_u2_mw --pass testpass TestWiki WikiAdmin https://phabricator.wikimedia.org/T126327 [13:14:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:14:10] 10Continuous-Integration-Infrastructure: php wrapper script cause death loop of doom - https://phabricator.wikimedia.org/T126327#2011268 (10hashar) [13:14:23] PROBLEM - Host integration-slave-trusty-1012 is DOWN: CRITICAL - Host Unreachable (10.68.18.2) [13:15:05] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Make /usr/bin/php a wrapper that picks the right PHP version on CI slaves - https://phabricator.wikimedia.org/T126211#2011274 (10hashar) Still cause a death loop of doom T126327 (should merge that ticket): ``` 8750 ? S 0:00 \_ /b... [13:15:30] !log removing https://gerrit.wikimedia.org/r/#/c/269370/ from CI puppet master [13:15:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:16:13] 10Continuous-Integration-Infrastructure: php wrapper script cause death loop of doom - https://phabricator.wikimedia.org/T126327#2011276 (10hashar) [13:16:15] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Make /usr/bin/php a wrapper that picks the right PHP version on CI slaves - https://phabricator.wikimedia.org/T126211#2011277 (10hashar) [13:17:59] !log salt -v --batch=3 '*slave*' cmd.run 'puppet agent -tv' [13:18:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:20:15] RECOVERY - Host integration-slave-trusty-1012 is UP: PING OK - Packet loss = 0%, RTA = 0.48 ms [13:25:46] RECOVERY - SSH on integration-slave-precise-1014 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [13:26:02] RECOVERY - SSH on integration-slave-trusty-1016 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6 (protocol 2.0) [13:27:50] PROBLEM - Puppet failure on integration-slave-trusty-1013 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [13:28:11] !log salt '*precise*' cmd.run 'update-alternatives --set php /usr/bin/php5' [13:28:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:28:45] !log salt '*trusty*' cmd.run 'update-alternatives --set php /usr/bin/hhvm' [13:28:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:37:49] RECOVERY - Puppet failure on integration-slave-trusty-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [13:58:10] hashar: are you restarting jenkins? [13:58:29] * aude sees "Jenkins is going to shut down" [13:58:34] yeah [13:58:45] aude: i noticed your patch it failed php53 lint for some reason [13:59:13] i can't see how [13:59:31] especially since it passed that already [13:59:56] oh for god sake git on precise sucks https://integration.wikimedia.org/ci/job/mediawiki-core-php53lint/1502/console [13:59:59] times out trying to clone [14:00:10] I am sure that is the reason [14:00:25] :/ [14:04:25] !log Manually git fetching mediawiki-core in /mnt/jenkins-workspace/workspace/mediawiki-core-php53lint of slaves precise 1001 to 1004 (git on Precise is remarkably too slow) [14:04:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:12:15] !log de pooling https://integration.wikimedia.org/ci/computer/integration-slave-precise-1012/ Mysql is gone somehow [14:12:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:13:02] !log pooling https://integration.wikimedia.org/ci/computer/integration-slave-precise-1012/ Mysql is back .. Blame puppet [14:13:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:28:11] Yippee, build fixed! [14:28:11] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #782: 09FIXED in 2 min 10 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/782/ [14:33:07] oh for god sake [14:33:18] that integration-make-wmf-branch doesn't work :D [14:33:43] 7Blocked-on-RelEng, 10Continuous-Integration-Infrastructure, 6Labs, 10Tool-Labs, 5Patch-For-Review: debian-glue tries to fetch obsolete package - https://phabricator.wikimedia.org/T125999#2011404 (10akosiaris) Taking a look at this I would say that it has nothing to do with the symlink mentioned above, b... [14:45:43] hashar: what do you mean? [14:45:55] 'git' 'clone' '-q' 'ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/AbuseFilter.git' 'AbuseFilter' [14:45:55] Permission denied (publickey). [14:45:56] :D [14:46:05] I created a local ssh key [14:46:14] ah, yeah, you have to keyforward to that machine [14:46:14] now trying to get sudo to recognize it somehow :-} [14:46:56] oh [14:49:18] ah [14:49:34] !log make-wmf-branch instance: created a local ssh key pair and set the config to use User: hashar [14:49:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:49:40] tis cloning [14:50:33] :D https://github.com/wikimedia/mediawiki-extensions-AbuseFilter/tree/wmf/1.27.0-wmf.13 [14:50:45] nice [14:50:51] !log pooling back integration-slave-precise1001 - 1004. Manually fetched git repos in workspace for mediawiki core php53 [14:50:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:51:09] !log ./make-wmf-branch -n 1.27.0-wmf.13 -o master [14:51:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:51:40] thcipriani|afk: any clue how long it takes usually ? :D [14:52:48] {"type":"ref-updated","submitter":{"name":"Hashar","email":"","username":"hashar"},"refUpdate":{"oldRev":"0000000000000000000000000000000000000000","newRev":"c0e49e1c21099a3e3d1bbe459d42b68f3a51e2cc","refName":"wmf/1.27.0-wmf.13","project":"mediawiki/extensions/MoodBar"}} [14:53:01] watching progress via: ssh -p 29418 hashar@gerrit.wikimedia.org 'gerrit stream-events' [14:53:13] hashar: I setup that machine after my rotation on deploy duty, so no idea :) guess: 15-20 minutes? [14:53:57] took me 1hr 18mins locally for branch cut before I setup that box. [14:54:15] so it's an improvement :P [14:57:04] remote: ERROR: In commit edd1d4ea6afa7ff9c91ab366273f0467503b0106 [14:57:04] remote: ERROR: committer email address root@integration-make-wmf-branch.integration.eqiad.wmflabs [14:57:04] remote: ERROR: does not match your user account. [14:57:08] that is going nowhere [14:57:37] that also mean I have probably sent all those commits with a crazy commiter name [14:57:56] https://github.com/wikimedia/mediawiki-extensions-UserMerge/commit/4e7d2240c80a275248dfd678ac0cdb596351df87 [14:58:16] ... [14:59:24] anyway, if it fails and you fix up what it's complaining about, you can use the --continue-from flag to pick up from the extension where you left off [15:00:14] will try --continue-from VisualEditor/VisualEditor [15:00:21] [ERROR] Could not find extension 'VisualEditor/VisualEditor' in any branched Extension list [15:00:22] eheh [15:00:38] just VisualEditor https://github.com/wikimedia/mediawiki-tools-release/blob/master/make-wmf-branch/config.json#L141 [15:01:33] * hashar tries [15:01:51] that works [15:01:57] --continue-from is nice [15:03:20] yeah :) the first time I tried to cut the branch it fell over somewhere in the middle, had to do a bunch of config.json local hacks. Would be nice to add --skip-extensions and --skip-skins at some point. [15:04:48] 10MediaWiki-Releasing, 6Release-Engineering-Team: make-wmf-branch doesn't ensure git has proper user.name and user.email - https://phabricator.wikimedia.org/T126334#2011449 (10hashar) 3NEW [15:05:05] I am not going to fix the wrong commit authorship [15:05:14] it is doing the git submodule add dance now [15:05:38] nice. [15:07:19] it is pushing! [15:07:29] To ssh://gerrit.wikimedia.org:29418/mediawiki/core.git [15:07:29] * [new branch] wmf/1.27.0-wmf.13 -> wmf/1.27.0-wmf.13 [15:07:34] not too bad [15:09:21] awesome. kudos on the branch cut! :) [15:10:29] 10MediaWiki-Releasing, 6Release-Engineering-Team: make-wmf-branch instance requires ssh auth forwarding - https://phabricator.wikimedia.org/T126335#2011463 (10hashar) 3NEW [15:11:21] !log mira: /srv/mediawiki-staging/multiversion/checkoutMediaWiki 1.27.0-wmf.13 php-1.27.0-wmf.13 [15:11:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:17:26] we should automatize the patching process as well :D [15:17:30] at least to check whether they apply [15:21:39] 6Release-Engineering-Team, 5Patch-For-Review, 5WMF-deploy-2016-02-09_(1.27.0-wmf.13): MW 1.27.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T125596#2011486 (10hashar) Branches cut and pushed. Have to verify special extensions have been properly set. [15:22:48] 6Release-Engineering-Team, 5WMF-deploy-2016-02-09_(1.27.0-wmf.13): checkoutMediaWiki fails with Fatal error: Class undefined: MWWikiversions in /srv/mediawiki-staging/multiversion/updateBranchPointers on line 26 - https://phabricator.wikimedia.org/T126336#2011492 (10hashar) 3NEW a:3hashar [15:23:05] poor branch pointers [15:23:16] Fatal error: Class undefined: MWWikiversions in /srv/mediawiki-staging/multiversion/updateBranchPointers on line 26 ||| https://phabricator.wikimedia.org/T126336 [15:30:18] 10Deployment-Systems, 10Salt, 5Patch-For-Review: Provide mechanism to add/remove minions from git-deploy - https://phabricator.wikimedia.org/T74319#2011517 (10ArielGlenn) a:3ArielGlenn [15:36:50] 6Release-Engineering-Team, 5WMF-deploy-2016-02-09_(1.27.0-wmf.13): checkoutMediaWiki fails with Fatal error: Class undefined: MWWikiversions in /srv/mediawiki-staging/multiversion/updateBranchPointers on line 26 - https://phabricator.wikimedia.org/T126336#2011546 (10Anomie) Looks like it was caused by {d052abd... [16:04:37] 10MediaWiki-Releasing, 5MW-1.26-release: Consider maybe backporting https://gerrit.wikimedia.org/r/#/c/249054/ to last stable - https://phabricator.wikimedia.org/T126344#2011667 (10Bawolff) 3NEW [16:06:25] !log Deleted corrupt integration-slave-precise-1003:/mnt/jenkins-workspace/workspace/mediawiki-core-php53lint/.git [16:06:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:32:11] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 5Patch-For-Review: MediaWiki gate takes 20 minutes for extensions tests and 1.5 hour for at least a patch - https://phabricator.wikimedia.org/T126274#2011736 (10Anomie) >>! In T126274#2010465, @hashar wrote: > * Scribunto has been remo... [16:51:17] 10Deployment-Systems, 6Release-Engineering-Team, 5WMF-deploy-2016-02-09_(1.27.0-wmf.13): checkoutMediaWiki fails with Fatal error: Class undefined: MWWikiversions in /srv/mediawiki-staging/multiversion/updateBranchPointers on line 26 - https://phabricator.wikimedia.org/T126336#2011793 (10greg) [16:56:40] 10MediaWiki-Releasing, 5MW-1.26-release: Consider maybe backporting https://gerrit.wikimedia.org/r/#/c/249054/ to last stable - https://phabricator.wikimedia.org/T126344#2011799 (10greg) @Krinkle: As author of the patch, do you agree it should be backported to 1.26? [16:58:10] 10Deployment-Systems: make-wmf-branch used master instead of the branch specified in special_extensions - https://phabricator.wikimedia.org/T125663#2011801 (10JanZerebecki) 5Open>3Resolved a:3JanZerebecki Worked fine when branching wmf/1.27.0-wmf.13. [17:03:54] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Flow, 10Wikidata, and 2 others: Wikidata QUnit broken on branch REL1_25 causing other extensions to fail - https://phabricator.wikimedia.org/T126073#2011818 (10JanZerebecki) https://gerrit.wikimedia.org/r/#/c/269310/ is an implemen... [17:31:42] marxarelli: did you forget to merge this? https://gerrit.wikimedia.org/r/#/c/268154/ [17:31:50] or was there a problem with it? [17:32:46] zeljkof: no problem, just forgot! [17:32:49] is Jenkins unwell? [17:33:28] marxarelli: no rush, but it is the last commit left to close T125532... ;) [17:33:33] https://gerrit.wikimedia.org/r/#/c/269440/ and https://gerrit.wikimedia.org/r/#/c/269439/ do not appear to be going through gate-and-submit [17:34:42] Krinkle, are you designated Jenkins-herder these days? [17:35:02] andrewbogott: i'll kick zuul [17:36:09] Someone mind having a look at https://gerrit.wikimedia.org/r/#/c/269444/? [17:36:15] thcipriani or marxarelli, specifically. [17:38:03] !log reloading zuul fails with "failed to kill 13660: Operation not permitted" [17:38:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:39:12] ostriches: that seems fine. Does it need to get removed from extension-list now, too? [17:39:23] !log restart of zuul fails as well. old process cannot be killed [17:39:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:39:30] thcipriani: Ah yes prolly. [17:39:32] Amending [17:40:17] !log killed old zull process manually and restarted service [17:40:19]