[00:03:19] ^ 23:55:50 23:55:50 ['/usr/bin/sync-common', '--no-update-l10n'] on deployment-tin.deployment-prep.eqiad.wmflabs returned [255]: Permission denied (publickey,keyboard-interactive). [00:05:23] It seems scap should be rearmed on labs. [00:05:45] * thcipriani looks [00:05:47] Project beta-scap-eqiad build #99910: 04STILL FAILING in 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99910/ [00:05:55] seems like scap is armed...at least it has keys in it... [00:06:22] https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99909/console [00:06:24] but yeah, not letting me get to the instances... [00:08:22] well, the key fingerprints don't match between keyholder and what's on disk for the targets, so that's a problem. [00:15:43] Project beta-scap-eqiad build #99911: 04STILL FAILING in 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99911/ [00:18:04] ah, the keyholder-auth.d fingerprints are wrong. [00:18:30] !log beta-scap-eqiad failure due to bad keyholder-auth.d fingerprints [00:18:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:25:55] Project beta-scap-eqiad build #99912: 04STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99912/ [00:35:51] Project beta-scap-eqiad build #99913: 04STILL FAILING in 1 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99913/ [00:44:01] 10Beta-Cluster-Infrastructure, 03Scap3: keyholder-auth.d broken on beta - https://phabricator.wikimedia.org/T133624#2237636 (10thcipriani) [00:46:04] Yippee, build fixed! [00:46:04] Project beta-scap-eqiad build #99914: 09FIXED in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99914/ [00:46:31] !log temporary keyholder fix in place in beta [00:46:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:52:03] RECOVERY - Puppet run on deployment-ores-web is OK: OK: Less than 1.00% above the threshold [0.0] [02:14:11] RECOVERY - Puppet run on integration-slave-trusty-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [02:33:26] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #923: 04FAILURE in 26 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/923/ [03:11:28] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #954: 04FAILURE in 21 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/954/ [03:21:39] PROBLEM - Host cache-rsync is DOWN: CRITICAL - Host Unreachable (10.68.23.165) [03:23:37] RECOVERY - Puppet run on deployment-ms-be02 is OK: OK: Less than 1.00% above the threshold [0.0] [03:31:58] RECOVERY - Puppet run on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [04:07:09] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #796: 04FAILURE in 8.4 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/796/ [04:18:17] Yippee, build fixed! [04:18:17] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #2: 09FIXED in 22 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/2/ [04:19:06] Yippee, build fixed! [04:19:06] Project selenium-MultimediaViewer » internet_explorer 10.0,beta,Windows 8,contintLabsSlave && UbuntuTrusty build #2: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=internet_explorer%2010.0,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%208,label=contintLabsSlave%20&&%20UbuntuTrusty/2/ [04:19:18] Yippee, build fixed! [04:19:19] Project selenium-MultimediaViewer » internet_explorer 11.0,beta,Windows 8.1,contintLabsSlave && UbuntuTrusty build #2: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=internet_explorer%2011.0,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%208.1,label=contintLabsSlave%20&&%20UbuntuTrusty/2/ [05:02:10] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #769: 04FAILURE in 10 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/769/ [06:42:51] PROBLEM - Puppet run on deployment-upload is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [06:56:08] PROBLEM - Puppet run on integration-slave-trusty-1013 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:09:06] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce build #392: 04FAILURE in 5.6 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce/392/ [07:36:10] RECOVERY - Puppet run on integration-slave-trusty-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [08:03:21] (03CR) 10Hashar: [C: 032] "Excellent :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/285252 (owner: 10Brian Wolff) [08:05:18] 10Beta-Cluster-Infrastructure: castor.integration.eqiad.wmflabs unreacheable deadlocking the whole CI - https://phabricator.wikimedia.org/T133652#2238376 (10hashar) [08:06:32] 10Beta-Cluster-Infrastructure: castor.integration.eqiad.wmflabs unreacheable deadlocking the whole CI - https://phabricator.wikimedia.org/T133652#2238388 (10hashar) p:05Triage>03Unbreak! a:03hashar [08:06:50] !log CI jobs deadlocked due to castor being unavailable | https://phabricator.wikimedia.org/T133652 [08:06:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:10:11] 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure: castor.integration.eqiad.wmflabs unreacheable deadlocking the whole CI - https://phabricator.wikimedia.org/T133652#2238394 (10hashar) Seems the /dev/vda disk is stalling somehow :( ``` castor login: [2863440.276096] INFO: task jbd2/vda3-8:113 bloc... [08:10:55] !log soft rebooting castor instance | T133652 [08:10:56] T133652: castor.integration.eqiad.wmflabs unreacheable deadlocking the whole CI - https://phabricator.wikimedia.org/T133652 [08:11:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:12:48] !log hard rebooting castor instance | T133652 [08:12:49] T133652: castor.integration.eqiad.wmflabs unreacheable deadlocking the whole CI - https://phabricator.wikimedia.org/T133652 [08:12:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:17:52] PROBLEM - Host castor is DOWN: CRITICAL - Host Unreachable (10.68.23.216) [08:20:22] !log shutoff instance castor, does not seem to be able to start again :( | T133652 [08:20:23] T133652: castor.integration.eqiad.wmflabs unreacheable deadlocking the whole CI - https://phabricator.wikimedia.org/T133652 [08:20:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:25:22] 10MediaWiki-Codesniffer, 03Google-Summer-of-Code-2016: [GSoC 2016 Proposal] Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T130574#2238408 (1001tonythomas) Welcome to #google-summer-of-code-2016 and to the Community Bonding period! Happy to have you here, and this should be c... [08:29:06] 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure: castor.integration.eqiad.wmflabs unreacheable deadlocking the whole CI - https://phabricator.wikimedia.org/T133652#2238451 (10hashar) It wont come back. I am going to create a new instance. [08:38:27] hashar: i was about to come in an complain but i see that you're fire fighting ;) [08:38:46] 10Continuous-Integration-Infrastructure: npm-node-4.3 test fails on a core patch - https://phabricator.wikimedia.org/T133655#2238486 (10Amire80) [08:44:40] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 0.58 ms [08:45:29] Most of CI is down / deadlocked due to wmflabs being unresponsive [08:45:36] filled as https://phabricator.wikimedia.org/T133654 [08:46:04] 10Continuous-Integration-Infrastructure: npm-node-4.3 test fails on a core patch - https://phabricator.wikimedia.org/T133655#2238516 (10hashar) [08:46:07] 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure: castor.integration.eqiad.wmflabs unreacheable deadlocking the whole CI - https://phabricator.wikimedia.org/T133652#2238517 (10hashar) [08:46:30] 10Continuous-Integration-Infrastructure: npm-node-4.3 test fails on a core patch - https://phabricator.wikimedia.org/T133655#2238486 (10hashar) Yup wmflabs is deadlocked somehow :( T133654 [08:52:52] RECOVERY - Host castor is UP: PING OK - Packet loss = 0%, RTA = 0.78 ms [08:54:10] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [08:56:41] 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure: castor.integration.eqiad.wmflabs unreacheable deadlocking the whole CI - https://phabricator.wikimedia.org/T133652#2238545 (10hashar) Labs process got restarted and castor instance managed to spawn. Jenkins refuses to add it back as a slave though :( [08:56:59] 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure: castor.integration.eqiad.wmflabs unreacheable deadlocking the whole CI - https://phabricator.wikimedia.org/T133652#2238550 (10yuvipanda) [08:59:47] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 10Monitoring, 06Operations: Have a paging check for Nova API accessible - https://phabricator.wikimedia.org/T133656#2238563 (10yuvipanda) [09:09:32] (03Merged) 10jenkins-bot: Attempt to make LoginNotify run phpunit tests [integration/config] - 10https://gerrit.wikimedia.org/r/285252 (owner: 10Brian Wolff) [09:10:24] 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure: castor.integration.eqiad.wmflabs unreacheable deadlocking the whole CI - https://phabricator.wikimedia.org/T133652#2238604 (10hashar) In Jenkins the castor slave thread seems to be blocked. ``` "Channel reader thread: castor" prio=5 BLOCKED hudso... [09:11:54] !log CI is back up! [09:11:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:12:10] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [09:13:31] 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure: castor.integration.eqiad.wmflabs unreacheable deadlocking the whole CI - https://phabricator.wikimedia.org/T133652#2238618 (10hashar) 05Open>03Resolved Root cause was Nova acting weirdly fixed by @yuvipanda which "restarted nova-conductor & sc... [09:13:55] RECOVERY - Host integration-trusty-1026 is UP: PING OK - Packet loss = 0%, RTA = 0.92 ms [09:15:27] (03CR) 10Hashar: "Did a recheck on https://gerrit.wikimedia.org/r/#/c/285251/ the 'composer test' command fails due to PHPCodeSniffer." [integration/config] - 10https://gerrit.wikimedia.org/r/285252 (owner: 10Brian Wolff) [09:16:07] PROBLEM - Host integration-trusty-1026 is DOWN: CRITICAL - Host Unreachable (10.68.17.98) [09:16:48] hashar: <3 thanks for fixing ci [09:17:09] 10Beta-Cluster-Infrastructure: Keyholder on beta cluster has lost credentials for mwdeploy user - https://phabricator.wikimedia.org/T133521#2238643 (10hashar) @mmodell @thcipriani did you get that one sorted out yesterday? [09:17:48] phuedx: you are welcome :-}  Though the actionable was really to restart some process on wmflabs which yuvipanda kindly accomplished :D [09:24:29] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 10Monitoring, 06Operations: Have a paging check for Nova API accessible - https://phabricator.wikimedia.org/T133656#2238670 (10hashar) [09:29:41] 06Release-Engineering-Team, 10Phabricator: Clean up tasks in archived #Staging Phabricator project - https://phabricator.wikimedia.org/T133529#2238678 (10hashar) Since #Staging project is abandoned, can we mass close its tasks ? https://phabricator.wikimedia.org/maniphest/query/12OMlv5TPW6b/#R [09:31:04] 10Staging, 13Patch-For-Review: Setup staging-tin as deployment host - https://phabricator.wikimedia.org/T88442#2238686 (10hashar) [09:51:05] Hello. We have a generalized failure for the operations-mw-config-phpunit job: https://integration.wikimedia.org/ci/job/operations-mw-config-phpunit/buildTimeTrend [09:51:27] 23:06:27 [xUnit] [ERROR] - No test reports found for the metric 'PHPUnit' with the resolved pattern 'log/junit-phpunit.xml'. Configuration error?. [09:52:37] Example of console output taken from https://integration.wikimedia.org/ci/job/operations-mw-config-phpunit/6010/console [10:13:34] hashar: are there docs for creating new ci jobs? [10:13:45] phuedx: not much [10:13:54] i'd like to experiment with running a static analysis tool (non voting) [10:14:08] it requires php7 with an extension installed [10:14:09] Dereckson: can you please copy paste that to a task ? :-} [10:14:42] phuedx: i think I have seen a task about a dependency on PHP 7. The thing is that we have no Zend 7 available anywhere :( [10:17:00] Dereckson: ah my bad [10:17:06] that is phpunit which I have dropped :( [10:29:44] hashar: I needz halp [10:29:58] !log restored integration/phpunit on CI slaves due to https://integration.wikimedia.org/ci/job/operations-mw-config-phpunit/ failling [10:30:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:30:17] a couple of repos have failing selenium tests because the commits need to be cherry picked to production branches [10:30:42] I know how to do that, but then the code probably has to be deployed, and I have no clue how to do that :| [10:31:20] Dereckson: Notice: /Stage[main]/Contint::Phpunit/Git::Clone[jenkins CI phpunit]/File[/srv/deployment/integration/phpunit]/ensure: created [10:31:28] Dereckson: it is back. Slaves will be updated as puppet run on them [10:32:00] zeljkof: what is up ? [10:32:34] hashar: can you help with reviewing/deploying a few simple patches to production branches? :) [10:32:36] Project selenium-VisualEditor-2016-04-26 » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 6 min 14 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor-2016-04-26/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [10:32:36] zeljkof: please cherry pick and add me as reviewer. I will +2 and take care of refresh on the deploy host [10:32:58] zeljkof: it just a few git submodule update to conduct and taking care of local patches that might exist [10:33:01] hashar: thanks! [10:35:03] hashar: okay, thanks for the quick fix [10:35:24] Dereckson: seems I broke it yesterday and nobody but you care about notifying the issue :( [10:35:41] I am force running puppet everywhere [10:44:43] https://integration.wikimedia.org/ci/job/operations-mw-config-phpunit/ works again :) [10:45:02] Yippee, build fixed! [10:45:03] Project selenium-VisualEditor-2016-04-26 » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #2: 09FIXED in 3 min 25 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor-2016-04-26/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/2/ [10:57:55] hashar: the first patch [10:57:56] https://gerrit.wikimedia.org/r/#/c/285370/ [10:58:03] working on the second one [11:04:14] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 4 min 40 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [11:09:21] hashar: the second patch https://gerrit.wikimedia.org/r/#/c/285372/ [11:09:51] Yippee, build fixed! [11:09:51] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #2: 09FIXED in 3 min 51 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/2/ [11:17:25] zeljkof: blindly +2ed :p [11:17:37] hashar: thanks, it should be fine (tm) [11:17:41] ;) [11:18:18] i will cut the wmf branches this aftenroon [11:25:15] Yippee, build fixed! [11:25:16] Project selenium-Echo-2016-04-25 » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #3: 09FIXED in 1 min 44 sec: https://integration.wikimedia.org/ci/job/selenium-Echo-2016-04-25/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/3/ [11:25:35] Yippee, build fixed! [11:25:35] Project selenium-Echo-2016-04-25 » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #3: 09FIXED in 2 min 3 sec: https://integration.wikimedia.org/ci/job/selenium-Echo-2016-04-25/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/3/ [11:27:55] hashar: deploying jobs https://integration.wikimedia.org/ci/view/selenium/ [11:28:16] jobs without trailing date are deployed and green [11:28:24] jobs with trailing date are still in testing [11:29:11] Project selenium-Echo » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 2 min 2 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [11:29:14] Project selenium-Echo » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 2 min 5 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [11:31:36] Yippee, build fixed! [11:31:36] Project selenium-MultimediaViewer-2016-04-26 » firefox,mediawiki,Linux,contintLabsSlave && UbuntuTrusty build #3: 09FIXED in 2 min 20 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer-2016-04-26/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=mediawiki,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/3/ [11:33:42] Project selenium-PdfHandler » firefox,test,Linux,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 36 sec: https://integration.wikimedia.org/ci/job/selenium-PdfHandler/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [11:49:31] zeljkof: neat :-) [11:49:50] timed out after 60 seconds (Watir::Wait::TimeoutError) [11:49:51] bah [11:50:03] hashar: yeah, still some problems :| [11:50:13] * zeljkof is out of lunch :9 [11:52:32] https://integration.wikimedia.org/ci/job/selenium-Echo/lastCompletedBuild/testReport/ that is neat [11:52:38] test results across browsers / envs [11:52:49] Project selenium-MultimediaViewer-2016-04-26 » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #3: 04FAILURE in 23 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer-2016-04-26/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/3/ [11:54:02] Yippee, build fixed! [11:54:03] Project selenium-MultimediaViewer-2016-04-26 » internet_explorer 11.0,beta,Windows 7,contintLabsSlave && UbuntuTrusty build #3: 09FIXED in 24 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer-2016-04-26/BROWSER=internet_explorer%2011.0,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%207,label=contintLabsSlave%20&&%20UbuntuTrusty/3/ [11:56:03] Yippee, build fixed! [11:56:04] Project selenium-MultimediaViewer-2016-04-26 » safari,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #3: 09FIXED in 26 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer-2016-04-26/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/3/ [12:04:18] \O/ [12:20:47] 07Browser-Tests, 10Continuous-Integration-Config, 10Wikidata, 03Wikidata-Sprint-2016-04-12, 03Wikidata-Sprint-2016-04-26: [Task] Move Wikidata browsertests into Wikibase repository - https://phabricator.wikimedia.org/T118727#2239128 (10Tobi_WMDE_SW) [12:20:52] 07Browser-Tests, 10Continuous-Integration-Config, 10Wikidata, 07Story, and 2 others: [Story] Run browsertests regularly on test.wikidata.org via Jenkins - https://phabricator.wikimedia.org/T101497#2239129 (10Tobi_WMDE_SW) [12:20:54] 10Continuous-Integration-Config, 10Wikidata, 03Wikidata-Sprint-2016-04-12, 03Wikidata-Sprint-2016-04-26: [Task] Setup a Jenkins job to run Wikidata browsertests on test.wikidata.org - https://phabricator.wikimedia.org/T101499#2239130 (10Tobi_WMDE_SW) [12:25:18] PROBLEM - Puppet staleness on deployment-tin is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [43200.0] [12:39:20] !log starting cut of 1.27.0-wmf.22 branch ( poke ostriches ) [12:39:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:55:29] 06Release-Engineering-Team, 05Release: MW-1.27.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T131556#2239214 (10hashar) Started branch cutting of 1.27.0-wmf.22 on `terbium`. Though it eventually failed: ``` ./make-wmf-branch -n 1.27.0-wmf.22 -o master .... 'git' 'submodule' 'add' '-f' '-b' '... [12:56:27] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #1031: 04FAILURE in 24 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/1031/ [13:45:29] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #3: 04FAILURE in 1 min 29 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/3/ [13:46:03] hashar: Morning. How goes that branching? [13:46:13] good [13:46:20] checked out on tin [13:46:45] I am finishing up a training with sara from talent & culture [13:46:51] then will look at applying the security patches [13:47:11] doc at https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys is great :-} [13:47:23] Yep, that's the Ultimate Guide to Follow :) [13:48:40] 10Continuous-Integration-Infrastructure, 06Operations, 13Patch-For-Review: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#2239441 (10fgiunchedi) promising indeed! we don't need php5-fss I think if we no longer need to run zend php, if we do need i... [13:52:05] 10Continuous-Integration-Infrastructure, 06Operations, 13Patch-For-Review: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#2239459 (10hashar) If php5-fss is solely Zend, yeah I guess we can phase it out from mediawiki::packages::php5 when running o... [14:02:33] aeoazeaurazrar [14:02:35] so many patches [14:14:13] (03CR) 10Brian Wolff: "I kind of expected that. I added phpcs as an afterthought and didnt test it locally. Ill fix the warnings sometime later today." [integration/config] - 10https://gerrit.wikimedia.org/r/285252 (owner: 10Brian Wolff) [14:16:01] 10Continuous-Integration-Config, 10Dumps-Generation, 06Operations, 13Patch-For-Review, 07WorkType-Maintenance: operations/dumps repo should pass flake8 - https://phabricator.wikimedia.org/T114249#2239536 (10ArielGlenn) 05Open>03Resolved Actually the hosts are configured and deployed now (except for m... [14:16:52] applying them [14:18:35] !log Applied security patches to 1.27.0-wmf.22 | T131556 [14:18:36] T131556: MW-1.27.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T131556 [14:18:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:18:53] Yippee, build fixed! [14:18:54] Project selenium-Echo-2016-04-26 » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #4: 09FIXED in 1 min 2 sec: https://integration.wikimedia.org/ci/job/selenium-Echo-2016-04-26/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/4/ [14:18:54] Yippee, build fixed! [14:18:55] Project selenium-Echo-2016-04-26 » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #4: 09FIXED in 1 min 2 sec: https://integration.wikimedia.org/ci/job/selenium-Echo-2016-04-26/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/4/ [14:24:59] Yippee, build fixed! [14:25:00] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #4: 09FIXED in 3 min 50 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/4/ [14:25:07] ostriches: I have bailed out on symlinking :D https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Restore_symlinks_on_deployment_server [14:25:16] I guess that is for the /php/ dir [14:25:45] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: There should be a way to run custom Rake task in selenium* jobs - https://phabricator.wikimedia.org/T133542#2239580 (10zeljkofilipin) [14:27:38] hashar: Honestly, I have no idea about those symlinks. They confuse me and make me sad :P [14:34:12] ostriches: nothing seems to create them apparently [14:34:18] or I screwed up / skipped some part [14:34:28] it is doing sync-masters now [14:35:10] hashar: Would the trusty nodepool image work with jessie by if we have php on trusty nodepool would jessie be able to use it. [14:35:32] hashar: The static ones aren't used anymore, nothing will create those. [14:35:47] php/ or w/e should be handled by updateWikiversions [14:38:20] 10Continuous-Integration-Infrastructure, 06Operations, 13Patch-For-Review: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#2239629 (10fgiunchedi) sounds good, thanks @hashar, see also {T131749} which seems to have some overlap with this (general th... [14:44:53] hashar: How can I setup a job with that mediawiki test, my instance has no zuul, so I guess jenkins would not read that yaml file. So how I can tell him to read the job, can you help me there? Thanks :) [14:47:33] 10Continuous-Integration-Infrastructure, 06Operations, 13Patch-For-Review: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#2239659 (10faidon) php5-fss shouldn't be needed even on Zend nowadays — Zend >= 5.5's native strtr() was made fast enough to... [14:56:36] zeljkof: Jan is not attending the weekly triage. I dont think there is much point in having the meeting? [14:57:06] Luke081515: in integration/config you have two namespaces: /zuul/ which is the workflow detailing which jobs to trigger for an event in Gerrit [14:57:25] Luke081515: and /jjb/ which are a bunch of YAML files to be used with jenkins-job-builder to generate a Jenkins job [14:57:45] hashar: I am fine with having it, or skipping it :) [14:57:49] which one do you prefer [14:58:00] I am all in moving browsertests* jobs to selenium* [14:58:05] ostriches: looks like I have filled the updateWikiversions somehow :-} [14:58:50] zeljkof: lets skip so [14:59:05] ostriches: oh man we still have to warm up HHVM jit cache [14:59:07] hashar: ok [14:59:08] I thought scap handled it for us somehow [14:59:17] no, it never has. [14:59:52] while true; do date; curl https://test.wikipedia.org/wiki/Special:Version; done; [15:00:06] meanwhile that URL shows 1.27.0-alpha (93f1116) [15:00:06] 13:13, 26 April 2016 [15:00:26] wrong version : [15:00:27] / [15:00:46] Let me looook [15:00:49] I'll fix it :) [15:01:09] wondering what I have screwed up [15:01:11] oh [15:01:18] or it is $wgVersion that needs to be set manually ? [15:01:52] No. [15:01:58] make-wmf-branch did that. [15:02:17] ahhh [15:02:25] 1.27.0-alpha? [15:02:26] What? [15:03:21] git show 6f03a10 -- includes/DefaultSettings.php shows nothing :\ [15:03:22] I must have screwed up makewmfbranch :( [15:03:41] How? It should Just Work. [15:03:48] it bailed out due to not being able to push [15:03:53] my git-config was wrong [15:04:00] RECOVERY - Puppet run on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [15:04:02] Yippee, build fixed! [15:04:02] Project selenium-Gather-2016-04-25 » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #3: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/selenium-Gather-2016-04-25/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/3/ [15:04:07] so I have replayed steps manually but have skipped that fixVersion() part :( [15:04:08] So...how did it get pushed? [15:04:19] copy pasted -} [15:04:35] Yeah I wouldn't do that. If it breaks I'd just stop and get it working. [15:06:05] 10Beta-Cluster-Infrastructure, 07Puppet, 07Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#2239716 (10Ladsgroup) [15:06:08] 10Beta-Cluster-Infrastructure, 10scap, 10Analytics, 06Services, and 3 others: Set up AQS in Beta - https://phabricator.wikimedia.org/T116206#2239717 (10Ladsgroup) [15:06:12] 10Beta-Cluster-Infrastructure, 03Scap3, 06Revision-Scoring-As-A-Service, 07Puppet: deployment-((sca|aqs)01|ores-web) puppet failures due to scap3 errors - https://phabricator.wikimedia.org/T132267#2239713 (10Ladsgroup) 05Open>03Resolved a:03Ladsgroup [15:07:59] 10Beta-Cluster-Infrastructure, 10scap, 10Analytics, 06Services, and 3 others: Set up AQS in Beta - https://phabricator.wikimedia.org/T116206#2239761 (10Ladsgroup) [15:08:03] 10Beta-Cluster-Infrastructure, 03Scap3, 07Puppet: deployment-((sca|aqs)01|ores-web) puppet failures due to scap3 errors - https://phabricator.wikimedia.org/T132267#2239758 (10Ladsgroup) 05Resolved>03Open [15:09:10] ostriches: https://gerrit.wikimedia.org/r/285407 :D [15:13:03] merged [15:13:09] 06Release-Engineering-Team, 13Patch-For-Review, 05Release: MW-1.27.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T131556#2239785 (10hashar) `hashar@tin Started scap: testwiki to php-1.27.0-wmf.22 and rebuild l10n cache` Been warming up HHVM byte code cache hitting test.wikipedia.org from... [15:16:04] twentyafterfour: Hi [15:16:30] I figured out a way to hide the changes branch for good whilst it still imports in the background. [15:17:17] But one thing is that it will stop importing other branchs if you want to ignore those too. [15:18:52] MediaWiki 1.27.0-wmf.22 (93f1116) [15:18:57] ostriches: looks better [15:29:12] hashar: Oh, on the JIT cache, don't worry about it. mw.org gets enough traffic that all the appservers get warmed in a few seconds after we do the deploy later today [15:29:42] RECOVERY - Puppet run on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:29:44] ostriches: yeah I can imagine. Did a while loop in the background until it got fast enough :-} [15:30:09] that Is all I have for this afternono [15:30:44] I'll wrap up the rest then. [15:30:55] mw.org/test2 deploy, purge old branches, etc. [15:34:54] 10Continuous-Integration-Config, 10Dumps-Generation, 06Operations, 13Patch-For-Review, 07WorkType-Maintenance: operations/dumps repo should pass flake8 - https://phabricator.wikimedia.org/T114249#2239865 (10hashar) Neat! Well done Ariel [15:35:14] Project selenium-Echo-2016-04-26 » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #5: 04FAILURE in 2 min 46 sec: https://integration.wikimedia.org/ci/job/selenium-Echo-2016-04-26/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/5/ [15:35:16] paladox: oh? how's that [15:36:04] By going into src/applications/repository/engine/PhabricatorRepositoryDiscoveryEngine.php [15:36:16] And commenting out if (!$repository->shouldTrackBranch($name)) { [15:37:31] Yippee, build fixed! [15:37:32] Project selenium-Echo-2016-04-26 » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #6: 09FIXED in 1 min 22 sec: https://integration.wikimedia.org/ci/job/selenium-Echo-2016-04-26/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/6/ [15:37:40] Im not sure how we can do regexp(/^(?!changes)/)) instead of for each repo. [15:38:31] Since some users may want to ignore a branch and not import it but we can do that if you want, that will also reduce the duplication in the code i uploaded to gerrit. [15:39:00] well there is already an option for putting a regex in the repo config for each repository [15:39:01] ostriches: I should be around just fine, the board meeting I was supposed to attend tonight has been cancelled [15:39:55] Yep but there are over 800+ repos which would be showing at least one refs/changes/ branch, i want to hide that branch but expose the refs so they get imported in the background [15:40:19] I got it to do that but by doing that it also causing other branches that are ignored to be imported. [15:40:40] Project selenium-Echo-2016-04-26 » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #7: 04FAILURE in 2 min 7 sec: https://integration.wikimedia.org/ci/job/selenium-Echo-2016-04-26/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/7/ [15:41:50] twentyafterfour: Oh yep sorry, maybe we should keep my duplicated code since you would need to do regexp(/^(?!changes)/) for each repo [15:42:05] But the branch will be hidden if you do that and will still imported in the background [15:42:33] Project selenium-Echo-2016-04-26 » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #8: 04FAILURE in 1 min 21 sec: https://integration.wikimedia.org/ci/job/selenium-Echo-2016-04-26/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/8/ [15:45:19] paladox: so how about if we make the default filter list start with that regexp already created, that way it doesn't have to be manually added to each repo [15:45:48] Oh yep [15:45:54] How would we do that [15:45:59] twentyafterfour ^^ [15:47:11] paladox: I'm looking into that now, probably in the constructor for PhabricatorRepository [15:47:20] Ok thanks [15:52:37] Yippee, build fixed! [15:52:38] Project selenium-Echo-2016-04-26 » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #9: 09FIXED in 47 sec: https://integration.wikimedia.org/ci/job/selenium-Echo-2016-04-26/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/9/ [15:53:03] Yippee, build fixed! [15:53:04] Project selenium-Echo-2016-04-26 » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #9: 09FIXED in 1 min 12 sec: https://integration.wikimedia.org/ci/job/selenium-Echo-2016-04-26/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/9/ [15:58:37] Project selenium-Gather » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 10 min: https://integration.wikimedia.org/ci/job/selenium-Gather/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [16:03:25] Project selenium-GettingStarted » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 1 min 1 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [16:04:56] selenium* jobs are getting nice and green ;) https://integration.wikimedia.org/ci/view/selenium/ [16:05:11] not there yet, but getting close [16:07:14] (03PS42) 10Zfilipin: WIP Migration of browsertests* Jenkins jobs to selenium* jobs [integration/config] - 10https://gerrit.wikimedia.org/r/274136 (https://phabricator.wikimedia.org/T128190) [16:10:48] 10Staging, 10RESTBase: Create staging-restbase - https://phabricator.wikimedia.org/T93869#2240037 (10demon) 05Open>03declined [16:10:51] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#2240038 (10demon) [16:10:53] 10Staging, 10releng-201415-Q3: Determine code update cycle/cadence for the staging cluster - https://phabricator.wikimedia.org/T91563#2240041 (10demon) 05Open>03declined [16:10:55] 06Release-Engineering-Team, 10Staging, 10releng-201415-Q3: [Quarterly Success Metric] Green nightly builds on the staging cluster (tracking) - https://phabricator.wikimedia.org/T88701#2240042 (10demon) [16:10:58] 10Staging: Create staging-pc* (parsercache) - https://phabricator.wikimedia.org/T93806#2240039 (10demon) 05Open>03declined [16:11:00] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#1018477 (10demon) [16:11:03] 10Staging: Create staging-eventlogging - https://phabricator.wikimedia.org/T91561#2240044 (10demon) 05Open>03declined [16:11:05] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#1018477 (10demon) [16:11:07] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#1018477 (10demon) [16:11:08] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#1018477 (10demon) [16:11:11] 10Staging, 13Patch-For-Review: Create staging-rcs* (RC stream) - https://phabricator.wikimedia.org/T91560#2240046 (10demon) 05Open>03declined [16:11:12] 10Staging: Create staging-logstash* - https://phabricator.wikimedia.org/T91559#2240047 (10demon) 05Open>03declined [16:11:14] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#1018477 (10demon) [16:11:16] 10Staging: Create staging-mw-imagescalers - https://phabricator.wikimedia.org/T91558#2240048 (10demon) 05Open>03declined [16:11:18] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#1018477 (10demon) [16:11:21] 10Staging: Create staging-mw-api* (MW Api servers) - https://phabricator.wikimedia.org/T91556#2240054 (10demon) 05Open>03declined [16:11:23] 10Staging: Create staging-tmh* (Video scalers) - https://phabricator.wikimedia.org/T91557#2240052 (10demon) 05Open>03declined [16:11:24] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#1018477 (10demon) [16:11:26] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#1018477 (10demon) [16:11:28] 10Staging, 13Patch-For-Review: Create staging-ocg* (OCG servers) - https://phabricator.wikimedia.org/T91555#2240055 (10demon) 05Open>03declined [16:11:30] 10Staging: Create staging-varnish**** - https://phabricator.wikimedia.org/T91551#2240062 (10demon) 05Open>03declined [16:11:32] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#1018477 (10demon) [16:11:34] 10Staging: Create staging-ms-fe* / staging-ms-be* (swift frontend/backend) - https://phabricator.wikimedia.org/T91553#2240060 (10demon) 05Open>03declined [16:11:36] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#1018477 (10demon) [16:11:40] 10Staging: Create staging-jobrunner (Job runners!) - https://phabricator.wikimedia.org/T91550#2240078 (10demon) 05Open>03declined [16:11:42] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#1018477 (10demon) [16:11:44] 10Staging, 13Patch-For-Review: Create staging-mw-app* (MW App servers) - https://phabricator.wikimedia.org/T91548#2240082 (10demon) 05Open>03declined [16:11:46] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#1018477 (10demon) [16:11:48] 10Staging, 13Patch-For-Review: Create staging-wtp* (Parsoid runners) - https://phabricator.wikimedia.org/T91549#2240080 (10demon) 05Open>03declined [16:11:50] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#1018477 (10demon) [16:11:52] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#1018477 (10demon) [16:11:54] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#1018477 (10demon) [16:11:57] 10Staging, 13Patch-For-Review: Create staging-db* (databases) - https://phabricator.wikimedia.org/T91545#2240085 (10demon) 05stalled>03declined [16:11:58] 10Staging: Create staging-terbium - https://phabricator.wikimedia.org/T91543#2240086 (10demon) 05Open>03declined [16:13:47] 06Release-Engineering-Team, 10Phabricator: Clean up tasks in archived #Staging Phabricator project - https://phabricator.wikimedia.org/T133529#2240115 (10demon) I declined most of them now. Last couple of them might still be useful or be repurposed. [16:15:47] 06Release-Engineering-Team, 10Staging, 10releng-201415-Q3: [Quarterly Success Metric] Green nightly builds on the staging cluster (tracking) - https://phabricator.wikimedia.org/T88701#2240126 (10demon) [16:15:49] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#2240125 (10demon) 05Open>03declined [16:26:06] Project beta-scap-eqiad build #100013: 04FAILURE in 1 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/100013/ [16:26:08] (03PS1) 10Daniel Kinzler: Add dependency for Wikibase on Echo. [integration/config] - 10https://gerrit.wikimedia.org/r/285423 [16:28:34] (03PS2) 10Daniel Kinzler: Add dependency for Wikibase on Echo. [integration/config] - 10https://gerrit.wikimedia.org/r/285423 [16:30:59] Project selenium-Echo » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 49 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [16:31:03] Project selenium-Echo » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #1: 09SUCCESS in 52 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [16:35:55] Project beta-scap-eqiad build #100014: 04STILL FAILING in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/100014/ [16:37:40] 18:26:03 16:26:03 sudo -u mwdeploy -n -- /usr/bin/rsync -l deployment-tin.eqiad.wmflabs::common/wikiversions*.{json,php} /srv/mediawiki on mira.deployment-prep.eqiad.wmflabs returned [255]: Permission denied (publickey,keyboard-interactive). [16:37:44] meh [16:38:44] Luke081515: yarp. looking at it now. er rather twentyafterfour is looking at it [16:40:12] thx [16:45:53] Project beta-scap-eqiad build #100015: 04STILL FAILING in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/100015/ [16:48:49] 05Continuous-Integration-Scaling, 13Patch-For-Review: Investigate using a cache store/restore system for package managers - https://phabricator.wikimedia.org/T116017#2240273 (10hashar) Had the left over puppet patch merged via Puppet SWAT. [16:49:10] (03PS4) 10Hashar: dib: composer and Zend PHP for mw on Trusty [integration/config] - 10https://gerrit.wikimedia.org/r/285236 (https://phabricator.wikimedia.org/T119139) [16:54:41] (03CR) 10Hashar: [C: 032] dib: composer and Zend PHP for mw on Trusty [integration/config] - 10https://gerrit.wikimedia.org/r/285236 (https://phabricator.wikimedia.org/T119139) (owner: 10Hashar) [16:55:01] yeah it'll be fixed as soon as puppet finishes [16:55:08] hashar: So if I use jenkins job builder for my jobs, how do I have to name my the yaml files, if my job at jenkins is named "core" for example? And can I trigger then via jenkins, for via time plan? [16:55:49] (03Merged) 10jenkins-bot: dib: composer and Zend PHP for mw on Trusty [integration/config] - 10https://gerrit.wikimedia.org/r/285236 (https://phabricator.wikimedia.org/T119139) (owner: 10Hashar) [16:56:08] Yippee, build fixed! [16:56:08] Project beta-scap-eqiad build #100016: 09FIXED in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/100016/ [16:57:07] should be good now :) [16:57:22] 10Beta-Cluster-Infrastructure, 03Scap3: keyholder-auth.d broken on beta - https://phabricator.wikimedia.org/T133624#2240314 (10mmodell) 05Open>03Resolved [16:58:35] (03CR) 10Brian Wolff: "I fixed the tests, so it all works now :)" [integration/config] - 10https://gerrit.wikimedia.org/r/285252 (owner: 10Brian Wolff) [17:01:19] twentyafterfour, is the sca01 puppet failure due to the keyholder changes? [17:01:58] Krenair: maybe? I'll look [17:02:25] Error: /Stage[main]/Citoid/Service::Node[citoid]/Scap::Target[citoid/deploy]/Ssh::Userkey[deploy-service/citoid/deploy]/File[/etc/ssh/userkeys/deploy-service.d/citoid/deploy]/ensure: change from absent to present failed: Could not set 'present' on ensure: Not a directory @ dir_s_rmdir - /etc/ssh/userkeys/deploy-service.d/citoid/deploy20160426-11959-1annrlp.lock at 70:/etc/puppet/modules/ssh/manifests/userkey.pp [17:03:38] hmm [17:03:59] 05Continuous-Integration-Scaling, 10OOjs-UI, 07Blocked-on-Operations, 13Patch-For-Review, 07WorkType-NewFunctionality: Provide composer on the nodepool servers so OOjs UI can use it in the npm job - https://phabricator.wikimedia.org/T128092#2240335 (10hashar) I have rebuild the Trusty image applying http... [17:04:01] Krenair: odd, seems like it could be related but indirectly? I'll work on it [17:04:18] ty [17:10:37] Yippee, build fixed! [17:10:37] Project selenium-Math-2016-04-25 » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #2: 09FIXED in 36 sec: https://integration.wikimedia.org/ci/job/selenium-Math-2016-04-25/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/2/ [17:10:38] Yippee, build fixed! [17:10:39] Project selenium-Math-2016-04-25 » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #2: 09FIXED in 37 sec: https://integration.wikimedia.org/ci/job/selenium-Math-2016-04-25/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/2/ [17:14:40] Krenair: fixed [17:15:11] Amir1: around? [17:40:52] 10Beta-Cluster-Infrastructure: Keyholder on beta cluster has lost credentials for mwdeploy user - https://phabricator.wikimedia.org/T133521#2240441 (10thcipriani) 05Open>03Resolved a:03mmodell Yup, @mmodell fixed it yesterday. Some tweaks were needed to key generation on puppet script he's been working on. [17:48:41] twentyafterfour: I'm around now [17:48:58] I couldn't connect, I tried several hours ago [17:52:29] 10Beta-Cluster-Infrastructure, 03Scap3, 07Puppet: deployment-((sca|aqs)01|ores-web) puppet failures due to scap3 errors - https://phabricator.wikimedia.org/T132267#2240491 (10mmodell) Puppet status: |status |host |detail| |{icon check color=green}|deployment-sca01 |Success| |{ico... [17:53:56] Amir1: try again? [17:54:35] anyone have a clue why "deployment repo missing from deployment-tin: /srv/deployment/analytics/aqs/deploy does not exist." [17:57:40] I've been meaning to ask otto...ottomatta about that. Seems like if you add it to scap::server::sources in hieradata/labs/deployment-prep/common.yaml that would be a solved thing. I'm just unsure why it isn't there in the first place. [17:57:55] seems like it maybe used to be there? [17:58:59] (03CR) 10JanZerebecki: [C: 031] Add dependency for Wikibase on Echo. [integration/config] - 10https://gerrit.wikimedia.org/r/285423 (owner: 10Daniel Kinzler) [18:02:14] sure [18:03:37] twentyafterfour: still failing [18:03:49] "SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l deploy-service deployment-sca01.deployment-prep.eqiad.wmflabs" asks password [18:03:55] (from tin) [18:08:28] I'm guesstimating that https://phabricator.wikimedia.org/T128092 is no longer blocked on Ops? [18:16:21] 05Continuous-Integration-Scaling, 10OOjs-UI, 07Blocked-on-Operations, 13Patch-For-Review, 07WorkType-NewFunctionality: Provide composer on the nodepool servers so OOjs UI can use it in the npm job - https://phabricator.wikimedia.org/T128092#2240615 (10hashar) ``` jenkins@ci-trusty-wikimedia-84003:~$ whic... [18:17:42] 05Continuous-Integration-Scaling, 10OOjs-UI, 07Blocked-on-Operations, 13Patch-For-Review, 07WorkType-NewFunctionality: Provide composer on the nodepool servers so OOjs UI can use it in the npm job - https://phabricator.wikimedia.org/T128092#2240621 (10Jdforrester-WMF) So… Resolved? [18:36:03] (03PS1) 10JanZerebecki: Revert "Add ContentTranslation as dependency to ArticlePlaceholder" [integration/config] - 10https://gerrit.wikimedia.org/r/285436 [18:45:25] error: AuthorizedKeysCommand /usr/sbin/ssh-key-ldap-lookup returned status 1 [18:45:39] why is it checking ldap for authorized keys? [18:46:30] :( [18:47:03] thcipriani: any clue what that's about? ^ [18:47:28] it seems the keys are correct (signatures match on both ends, etc..) but the target's sshd is refusing [18:54:24] (03PS2) 10JanZerebecki: Revert "Add ContentTranslation as dependency to ArticlePlaceholder" [integration/config] - 10https://gerrit.wikimedia.org/r/285436 [18:54:38] (03CR) 10JanZerebecki: [C: 032] Revert "Add ContentTranslation as dependency to ArticlePlaceholder" [integration/config] - 10https://gerrit.wikimedia.org/r/285436 (owner: 10JanZerebecki) [18:54:39] good morning [18:55:59] good evening :) [18:59:15] (03Merged) 10jenkins-bot: Revert "Add ContentTranslation as dependency to ArticlePlaceholder" [integration/config] - 10https://gerrit.wikimedia.org/r/285436 (owner: 10JanZerebecki) [18:59:32] ok so ... /etc/ssh/userkeys/${user}.d/* doesn't actually work [19:00:07] we have support for creating it but sshd doesn't actually read from there and I don't think any code compiles the user.d/* files into a combined file for sshd to read [19:00:15] (03PS3) 10JanZerebecki: Add dependency for Wikibase on Echo. [integration/config] - 10https://gerrit.wikimedia.org/r/285423 (owner: 10Daniel Kinzler) [19:00:59] (03PS1) 10JanZerebecki: add Wikidata dependency on ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/285444 [19:01:08] (03PS2) 10JanZerebecki: add Wikidata dependency on ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/285444 [19:01:18] (03CR) 10JanZerebecki: [C: 032] add Wikidata dependency on ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/285444 (owner: 10JanZerebecki) [19:03:10] hashar: So if I use jenkins job builder for my jobs, how do I have to name my the yaml files, if my job at jenkins is named "core" for example? And can I trigger then via jenkins, for via time plan? [19:04:01] (03CR) 10jenkins-bot: [V: 04-1] add Wikidata dependency on ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/285444 (owner: 10JanZerebecki) [19:04:56] (03CR) 10jenkins-bot: [V: 04-1] add Wikidata dependency on ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/285444 (owner: 10JanZerebecki) [19:08:21] (03PS3) 10JanZerebecki: add Wikidata dependency on ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/285444 [19:08:48] (03CR) 10JanZerebecki: [C: 032] add Wikidata dependency on ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/285444 (owner: 10JanZerebecki) [19:10:30] twentyafterfour: Hi did you manage to take a look to see how we can set regexp(/^(?!changes)/) for track only globally please. [19:12:45] (03Merged) 10jenkins-bot: add Wikidata dependency on ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/285444 (owner: 10JanZerebecki) [19:14:38] (03PS4) 10JanZerebecki: Add dependency for Wikibase on Echo. [integration/config] - 10https://gerrit.wikimedia.org/r/285423 (owner: 10Daniel Kinzler) [19:16:51] (03PS5) 10JanZerebecki: Add dependency for Wikibase on Echo. [integration/config] - 10https://gerrit.wikimedia.org/r/285423 (owner: 10Daniel Kinzler) [19:17:45] twentyafterfour, what is it that creates .git/DEPLOY_HEAD files? [19:19:26] (03CR) 10JanZerebecki: "Deployed to Jenkins: (['mwext-Wikibase-client-tests-mysql-hhvm', 'mwext-Wikibase-client-tests-mysql-php53', 'mwext-Wikibase-client-tests-m" [integration/config] - 10https://gerrit.wikimedia.org/r/285423 (owner: 10Daniel Kinzler) [19:19:49] (03CR) 10JanZerebecki: [C: 032] Add dependency for Wikibase on Echo. [integration/config] - 10https://gerrit.wikimedia.org/r/285423 (owner: 10Daniel Kinzler) [19:20:26] Luke081515: "core" can be different things depending on context :-}  pywikibot/core mediawiki/core etc [19:20:58] hashar: No, I named my own project like this, because it's my private jenkins, so I want to build other code than at the public jenkins ;) [19:21:00] Luke081515: if you dont know where to put your definition, just create a new file [19:21:05] oh [19:21:16] (03Merged) 10jenkins-bot: Add dependency for Wikibase on Echo. [integration/config] - 10https://gerrit.wikimedia.org/r/285423 (owner: 10Daniel Kinzler) [19:21:16] now I am lost ;-} [19:21:33] JJB read all the .yaml files and concatenate them in a single document [19:21:44] so the name of files or number of files are solely for human beings [19:21:59] ah, ok [19:22:19] now the trigger question... ;) [19:23:05] that is for your private jenkins isn't it ? [19:23:10] hashar: what do I have to setup, if I want jjb to start, when I manually start the job? what do I have to told jenkins? [19:23:28] Jenkins is really like a scheduler and supports triggering on time like crontab entries [19:24:27] and this starts the scripts from jjb then? [19:24:34] what do you want to do ? [19:24:57] JJB is basically an utility to define Jenkins jobs based on a YAML domain specific language [19:25:06] ah, ok [19:25:12] !log 4675213..eb480d8 [19:25:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [19:25:20] then I guess I know how to continue, thanks :) [19:25:22] !log reload zuul for 4675213..eb480d8 [19:25:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [19:25:28] then one runs JJB against those yaml files, it generates Jenkins configuration files (which are XLM) and push them over the Jenkins REST API to create/update/delete jobs [19:25:34] but it is Jenkins running them [19:26:15] ok, how can I push them over the REST API? Or can I copy them into the instance too? [19:26:44] http://www.mediawiki.org/wiki/CI/JJB ! [19:26:56] ok, thanks [19:27:02] you want to add a .ini file that has the URL of the jenkins installation + username + api token [19:27:02] https://www.mediawiki.org/wiki/Continuous_integration/Jenkins_job_builder#Authenticate_JJB [19:27:37] and usually you want to generate the Jenkins configuration files locally to verify them [19:27:41] with jenkins-jobs test config/ -o output/ [19:27:48] (assuming the yaml files are in /config/ dir ) [19:30:12] that will generate a file per job that contains the XML config [19:30:30] to update all jobs: jenkins-jobs --conf etc/jenkins_jobs.ini update config/ [19:30:37] or a single one: jenkins-jobs --conf etc/jenkins_jobs.ini update config/ name-of-my-job-here [19:36:56] paladox: if you are around . I have read https://jenkins.io/2.0/ and seems it is entirely back compatible with 1.x [19:37:09] hashar: Hi [19:37:17] :) [19:37:26] paladox: I guess they bumped the major version because of the pipeline plugin being integrated (nice thing) and the very nice UI rework [19:38:03] hashar: Yep, i woulden think it will break jenkins for us if we upgrade, but we will get the new ui which looks great and modern [19:38:09] so maybe we will get it installed :-} [19:38:52] Yep :) [19:39:05] Plus it will be easy to configure jobs since they also redesge [19:39:22] redesgned that so as you tell which parts are related [19:43:27] 10Continuous-Integration-Infrastructure, 07Jenkins, 07WorkType-Maintenance: Upgrade Jenkins from 1.642.3 to 1.651.1 - https://phabricator.wikimedia.org/T133737#2241069 (10hashar) [19:46:51] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins, 07WorkType-Maintenance: Upgrade Jenkins from 1.642.3 to 1.651.1 - https://phabricator.wikimedia.org/T133737#2241086 (10hashar) #releng > pick(random) [19:49:28] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins, 07WorkType-Maintenance: Upgrade Jenkins from 1.642.3 to 1.651.1 - https://phabricator.wikimedia.org/T133737#2241111 (10hashar) I went through the changelog , that seems harmless. [19:59:46] (03PS1) 10Hashar: dib: composer and Zend PHP for mw on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/285451 (https://phabricator.wikimedia.org/T128092) [20:00:53] meh, operations/software/cassandra-metrics-collector is shown at Unknown Project at zuul [20:02:09] (03PS1) 10JanZerebecki: Revert "Add dependency for Wikibase on Echo." [integration/config] - 10https://gerrit.wikimedia.org/r/285453 [20:02:19] looks like zuul stopped working [20:02:33] nothing happens there [20:04:35] (03CR) 10Hashar: [C: 04-2] "mediawiki::packages::php5 requires php5-fss which is not available on Jessie. Ongoing discussion on T95002" [integration/config] - 10https://gerrit.wikimedia.org/r/285451 (https://phabricator.wikimedia.org/T128092) (owner: 10Hashar) [20:04:41] hashar ^^ [20:04:47] Thats because of Unknown Project [20:04:55] hashar and Luke081515 [20:05:07] The problem is fixed upstream [20:05:24] https://gerrit.wikimedia.org/r/#/c/285445/ [20:06:41] oh my [20:08:31] hashar: Should we backport the fix for this [20:08:35] all changes pending in Zuul have been flushed ..... [20:08:35] into zuul [20:08:42] yeah that is T128569 [20:08:43] T128569: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569 [20:09:25] it definitely kills everything but I am not in the mood of context switching to repackaging Zuul [20:10:09] Ok [20:15:02] (03PS2) 10JanZerebecki: Remove Echo from Wikidata dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/285453 [20:17:48] (03CR) 10JanZerebecki: "Partially reverted in https://gerrit.wikimedia.org/r/#/c/285453/ ." [integration/config] - 10https://gerrit.wikimedia.org/r/285423 (owner: 10Daniel Kinzler) [20:18:35] (03CR) 10JanZerebecki: [C: 032] Remove Echo from Wikidata dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/285453 (owner: 10JanZerebecki) [20:18:56] (03CR) 10JanZerebecki: "https://phabricator.wikimedia.org/T110604#2241193" [integration/config] - 10https://gerrit.wikimedia.org/r/285453 (owner: 10JanZerebecki) [20:19:30] (03Merged) 10jenkins-bot: Remove Echo from Wikidata dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/285453 (owner: 10JanZerebecki) [20:23:03] !log reloading zuul for eb480d8..81a1f1a [20:23:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:34:57] 10Beta-Cluster-Infrastructure, 03Scap3, 07Puppet: deployment-((sca|aqs)01|ores-web) puppet failures due to scap3 errors - https://phabricator.wikimedia.org/T132267#2241296 (10Krenair) I tried setting up the AQS deploy repository to fix aqs01 but it's missing .git/DEPLOY_HEAD? [20:38:17] 05Continuous-Integration-Scaling, 10OOjs-UI, 07Blocked-on-Operations, 13Patch-For-Review, 07WorkType-NewFunctionality: Provide composer on the nodepool servers so OOjs UI can use it in the npm job - https://phabricator.wikimedia.org/T128092#2241317 (10hashar) @Jdforrester-WMF that is for Trusty but the N... [20:40:47] (03PS1) 10Hashar: dib: composer and HHVM on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/285514 (https://phabricator.wikimedia.org/T128092) [20:41:47] (03PS2) 10Hashar: dib: composer and HHVM on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/285514 (https://phabricator.wikimedia.org/T128092) [20:41:52] (03PS3) 10Hashar: dib: composer and HHVM on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/285514 (https://phabricator.wikimedia.org/T128092) [20:42:14] (03Abandoned) 10Hashar: dib: composer and Zend PHP for mw on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/285451 (https://phabricator.wikimedia.org/T128092) (owner: 10Hashar) [20:42:53] 10MediaWiki-Codesniffer: Relax MediaWiki.WhiteSpace.SpaceBeforeSingleLineComment.EmptyComment for multiline comments - https://phabricator.wikimedia.org/T133743#2241323 (10Tgr) [20:43:37] (03CR) 10Hashar: [C: 032] dib: composer and HHVM on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/285514 (https://phabricator.wikimedia.org/T128092) (owner: 10Hashar) [20:44:25] (03Merged) 10jenkins-bot: dib: composer and HHVM on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/285514 (https://phabricator.wikimedia.org/T128092) (owner: 10Hashar) [20:45:46] !log Regenerating Nodepool Jessie snapshot to include composer and HHVM | T128092 [20:45:46] T128092: Provide composer on the nodepool servers so OOjs UI can use it in the npm job - https://phabricator.wikimedia.org/T128092 [20:45:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:54:51] 10Beta-Cluster-Infrastructure, 03Scap3, 07Puppet: deployment-((sca|aqs)01|ores-web) puppet failures due to scap3 errors - https://phabricator.wikimedia.org/T132267#2241359 (10thcipriani) >>! In T132267#2241296, @Krenair wrote: > I tried setting up the AQS deploy repository to fix aqs01 but it's missing .git/... [20:56:02] (03PS1) 10Hashar: dib: hhvm puppet class requires 'cron' [integration/config] - 10https://gerrit.wikimedia.org/r/285516 (https://phabricator.wikimedia.org/T128092) [20:57:32] (03CR) 10Hashar: [C: 032] dib: hhvm puppet class requires 'cron' [integration/config] - 10https://gerrit.wikimedia.org/r/285516 (https://phabricator.wikimedia.org/T128092) (owner: 10Hashar) [20:58:48] (03Merged) 10jenkins-bot: dib: hhvm puppet class requires 'cron' [integration/config] - 10https://gerrit.wikimedia.org/r/285516 (https://phabricator.wikimedia.org/T128092) (owner: 10Hashar) [21:12:54] (03PS1) 10Hashar: dib: hhvm puppet class needs rsyslog and logrotate [integration/config] - 10https://gerrit.wikimedia.org/r/285520 [21:13:13] (03CR) 10Hashar: [C: 032] dib: hhvm puppet class needs rsyslog and logrotate [integration/config] - 10https://gerrit.wikimedia.org/r/285520 (owner: 10Hashar) [21:14:08] (03Merged) 10jenkins-bot: dib: hhvm puppet class needs rsyslog and logrotate [integration/config] - 10https://gerrit.wikimedia.org/r/285520 (owner: 10Hashar) [21:20:31] RECOVERY - Host cache-rsync is UP: PING OK - Packet loss = 0%, RTA = 0.93 ms [21:21:25] twentyafterfour, can you show me how to run this init? [21:21:28] or thcipriani [21:21:32] Yippee, build fixed! [21:21:33] Project browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #249: 09FIXED in 5 min 31 sec: https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/249/ [21:21:34] or anyone really [21:22:11] Krenair: deploy --init you mean? [21:22:18] it's literally just that? [21:22:36] you just have to run that inside the repo at: /srv/deployment/[repo] [21:22:38] that should be it [21:23:39] I just get a permission denied error though [21:23:57] but it won't run as root [21:24:23] needs to run as the service user? [21:24:27] for now we have 775 trebuchet:wikidev as the ownership of the repos on /srv/deployment iirc [21:24:44] you have to be able to create a file at /srv/deployment/[repo]/.git/ [21:24:53] only root can do that [21:24:58] I cloned it as root [21:25:24] chown -R [21:26:33] we don't allow scap to run as root. there really isn't a way to drop privileges in a sane way. [21:26:52] if it depends on the permission of the user that is running the command. [21:29:13] how do you set the correct hosts to deploy to? it thinks it's in production... [21:30:10] hosts are set via: ./scap/scap.cfg there's a line in that config: dsh_targets points to a file with a host list [21:30:18] usually it'll be in the scap directory [21:30:40] git_server: deployment-bastion.deployment-prep.eqiad.wmflabs? [21:30:55] this is committed to the repo.. [21:30:58] sigh [21:33:05] 05Continuous-Integration-Scaling, 10OOjs-UI, 07Blocked-on-Operations, 13Patch-For-Review, 07WorkType-NewFunctionality: Provide composer on the nodepool servers so OOjs UI can use it in the npm job - https://phabricator.wikimedia.org/T128092#2241414 (10hashar) Still a WIP :) [21:34:08] eh, you can override it for the purposes of generating a good DEPLOY_HEAD: deploy -D'git_server:deployment-tin.deployment-prep.eqiad.wmflabs' --init [21:34:51] I'm gonna go and commit my change [21:35:09] it didn't fix deployments because there's now 21:34:15 ['/usr/bin/deploy-local', '-v', '--repo', 'analytics/aqs/deploy', '-g', 'default', 'fetch'] on deployment-aqs01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. [21:35:34] (03PS1) 10Hashar: Revert "dib: hhvm puppet class needs rsyslog and logrotate" [integration/config] - 10https://gerrit.wikimedia.org/r/285524 [21:35:44] (03CR) 10Hashar: [C: 032] Revert "dib: hhvm puppet class needs rsyslog and logrotate" [integration/config] - 10https://gerrit.wikimedia.org/r/285524 (owner: 10Hashar) [21:35:58] Project beta-scap-eqiad build #100044: 04FAILURE in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/100044/ [21:36:06] blerg. key verification in beta. [21:36:22] also this [21:36:30] (03Merged) 10jenkins-bot: Revert "dib: hhvm puppet class needs rsyslog and logrotate" [integration/config] - 10https://gerrit.wikimedia.org/r/285524 (owner: 10Hashar) [21:36:51] https://phabricator.wikimedia.org/P2961 [21:38:36] ah, should be -D log_json:False [21:38:40] PROBLEM - Host cache-rsync is DOWN: CRITICAL - Host Unreachable (10.68.23.165) [21:40:45] my mistake [21:40:57] might be worth a nice error message though [21:41:19] yeah, definitely didn't know what was happening for a second in that paste. [21:41:35] I'll see if we can catch the problem. [21:43:52] thanks [21:46:03] Yippee, build fixed! [21:46:03] Project beta-scap-eqiad build #100045: 09FIXED in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/100045/ [21:47:24] 03Scap3, 10scap, 13Patch-For-Review: scap::target shouldn't allow users to redefine the user's key - https://phabricator.wikimedia.org/T132747#2241454 (10mmodell) Fix ssh::userkey support for multiple keys: https://gerrit.wikimedia.org/r/#/c/285519/ [21:47:36] Krenair: thank you for all the beta work. it's quite a sisyphean task. [21:47:43] I think I just made a mistake with gerrit [21:48:20] I pushed to master instead of HEAD:refs/for/master [21:48:26] or whatever it was [21:49:15] :D [21:49:45] thing is, it actually accepted it [21:50:01] that means you have push rights on that repo I guess? [21:50:20] apparently so [21:50:24] :| [21:51:45] ostriches, help? [21:52:18] ok I think I'm finally done breaking keyholder in beta... [21:52:30] I tried to reset --hard HEAD^ and force push, but it rejects that [21:57:44] Krenair: probably only ostriches and other gerrit admins will have force push. Can you just make a revert patch and push that? [21:58:07] twentyafterfour: Hi did you manage to take a look to see how we can set regexp(/^(?!changes)/) for track only globally please. [21:58:50] I am a gerrit admin (that's probably how I got it into this mess), I do have force push. But I get some git nonsense saying "Updates were rejected because the tip of your current branch is behind its remote counterpart. Integrate the remote changes (e.g. 'git pull ...') before pushing again." [21:59:17] I don't want a revert commit, I want it gone from gerrit's history so I can upload it and have it go via review properly [21:59:46] *nod* sounds like you need git ninja help from ostriches then [22:00:13] You did verify that it was pushed into master on the repo already? [22:00:18] yes [22:00:20] Krenair: can you force-push ? [22:00:29] just force push HEAD^ [22:00:31] I think git is correct about the state of the branches, but it's ignoring the --force [22:01:31] it shouldn't be rejecting this because of the state, it is a force push [22:04:42] hmm [22:04:46] debug1: Server accepts key: pkalg ssh-rsa blen 535 [22:04:48] Agent admitted failure to sign using the key. [22:04:57] RoanKattouw, help? [22:05:11] wtf .. finally got all the details right with keys (or so I thought) and now the agent doesn't wanna sign? [22:05:14] * twentyafterfour grumbles [22:07:19] Krenair: Which repo? [22:07:29] analytics/aqs/deploy [22:07:32] Looking [22:07:38] thanks [22:09:46] OK, I fixed [22:10:02] I edited the ACL, granted force push to the Administrators group, then did git reset --hard HEAD^ ; git push --force [22:10:19] Now we need to fix the ACL so you can't accidentally push into master [22:10:30] so wait [22:10:47] it ignored my --force because I didn't have the permission on the repo [22:10:53] instead of erroring loudly about that? [22:11:14] Yeah the error isn't clear [22:11:21] It pretends the repo does not allow force pushes I guess? [22:11:27] Or that you didn't use --force at all? [22:11:38] When I tried (before my ACL edit) I got: [22:11:38] ! [remote rejected] master -> master (non-fast forward) [22:12:35] yes, but what was the exact error? [22:12:43] paladox: I haven't gotten it figured out yet [22:12:45] error: failed to push some refs to 'ssh://gerrit.wikimedia.org:29418/analytics/aqs/deploy.git' [22:12:53] Those were the only errors I got [22:13:03] git push without --force will also give you a client-side error before contacting the server [22:13:07] s/before/without/ [22:13:18] i.e. git will refuse to contact the server in this scenario unless you set --force [22:13:30] twentyafterfour: Ok, im having a look in src/applications/diffusion/controller/DiffusionRepositoryEditBranchesController.php [22:13:42] Where track only is set. [22:14:11] Krenair: BTW I've edited the ACL so that pushing directly to master (with or without --force) is now completely disallowed, as is standard for Gerrit-managed repos [22:14:25] thank you for fixing it all RoanKattouw [22:14:38] have now pushed the commit for review properly [22:14:47] This seems like the sane thing to do, but I didn't even know this repo existed so I don't know if that would cause anyone to complain [22:15:03] (That said, if they do, the answer is probably "sorry you don't get direct push" anywya) [22:23:47] Figures this happens when I walk away from the computer for puppy stuff for 30 mins. [22:23:51] *sigh* [22:25:06] 10Beta-Cluster-Infrastructure, 07Puppet, 07Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#2241541 (10Krenair) [22:25:09] 10Beta-Cluster-Infrastructure, 10scap, 10Analytics, 06Services, and 3 others: Set up AQS in Beta - https://phabricator.wikimedia.org/T116206#2241542 (10Krenair) [22:25:14] 10Beta-Cluster-Infrastructure, 03Scap3, 13Patch-For-Review, 07Puppet: deployment-((sca|aqs)01|ores-web) puppet failures due to scap3 errors - https://phabricator.wikimedia.org/T132267#2241539 (10Krenair) 05Open>03Resolved AQS is now fine. [22:38:52] twentyafterfour: I managed to get the code shown in track only, in the php code but i doint know how to get it so it is processed [22:54:01] PROBLEM - Puppet run on integration-slave-trusty-1016 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:55:41] PROBLEM - Puppet run on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:56:54] PROBLEM - Puppet run on integration-slave-trusty-1012 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:57:08] PROBLEM - Puppet run on integration-slave-trusty-1013 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:14:19] Does anyone here know why mw-config was made fast-forward-only? [23:14:45] Having to rebase is a mild annoyance but it's amplified a lot by Zuul's bugginess in this area [23:15:05] Like, zuul will start gate-and-submit jobs for a patchset that was +2ed but cannot be submitted because it needs a rebase [23:15:21] And if you rebase an already-+2ed patchset, zuul does not rerun gate-and-submit, you have to remove your +2 and readd it [23:16:25] meh [23:16:47] sound like this causes lots of fun :D [23:22:07] catrope: the good thing is that, gate-and-submit doesn't need much time for this repo... for a repo with php5 jobs this will make everyone happy ;) [23:25:31] Yeah... [23:25:45] And thankfully it's in its own pipeline so it doesn't get blocked on other repos' phpunit jobs [23:30:39] RECOVERY - Puppet run on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [23:35:29] PROBLEM - Puppet run on integration-slave-trusty-1025 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:36:56] RECOVERY - Puppet run on integration-slave-trusty-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [23:37:10] RECOVERY - Puppet run on integration-slave-trusty-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [23:38:04] PROBLEM - Puppet run on integration-slave-trusty-1014 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]