[00:20:35] 10Release-Engineering-Team (Kanban), 10Phabricator (2017-07-05), 10Upstream: Unhandled Exception ("InvalidArgumentException") error when clicking on search link in "Activity Feed" panel - https://phabricator.wikimedia.org/T168108#3410090 (10mmodell) 05Open>03Resolved [00:25:08] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10MediaWiki-Database, 10MediaWiki-extensions-CentralAuth, 10Patch-For-Review: CentralAuth tests are broken - https://phabricator.wikimedia.org/T168802#3410129 (10MaxSem) 05Open>03Resolved a:03MaxSem Fixed for my repos, please... [00:27:36] (03CR) 10Thcipriani: "Patch mostly seems fine aside from the extra job that is created. It is noteworthy that we'll have to add phpflavor-php56 labels to some n" [integration/config] - 10https://gerrit.wikimedia.org/r/363493 (owner: 10Ejegg) [00:35:57] (03PS1) 10Mattflaschen: Forbid backtick operator [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/363512 [00:36:34] greg-g: Hey Greg, I realized I've been doing SWAT deploys since a while but never really asked for permission for it. I hope it's alright. Haven't broken anything yet. :) [00:37:22] Naughty Niharika [00:37:34] FWIW, I don't think there's any rules other than "know how to deploy" tbh [00:37:40] Shh, Reedy. [00:38:04] I've three docs full of notes about that now. :P [00:48:15] 10Release-Engineering-Team, 10Operations, 10Wikimedia-Site-requests: Run updateArticleCount.php on Wikimedia Commons - https://phabricator.wikimedia.org/T169822#3409698 (10Dcljr) Wow, that was fast. I thought it would take hours… [00:57:40] (03CR) 10Catrope: [C: 032] Forbid backtick operator [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/363512 (owner: 10Mattflaschen) [00:58:25] (03Merged) 10jenkins-bot: Forbid backtick operator [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/363512 (owner: 10Mattflaschen) [01:03:21] (03CR) 10Reedy: "Core can be fixed before ;)" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/363512 (owner: 10Mattflaschen) [01:20:39] (03CR) 10Legoktm: "Please don't merge patches without any tests" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/363512 (owner: 10Mattflaschen) [01:23:34] 10Release-Engineering-Team, 10Operations, 10Wikimedia-Site-requests: Run updateArticleCount.php on Wikimedia Commons - https://phabricator.wikimedia.org/T169822#3410222 (10demon) Weird, it was taking way longer for me earlier... Glad we got it done :) [01:24:41] (03PS5) 10Ejegg: Templatize phplint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/363493 [01:39:38] Niharika: awesome :) [01:40:23] Niharika: only officialness is https://wikitech.wikimedia.org/wiki/SWAT_deploys#SWAT_Team_members_roles.2C_responsibilities.2C_and_tips and me with final "no, sorry, not for you" :) [01:41:02] Niharika: if you're OK with it, I could add you to the list-of-people-to-ping-on-IRC, eg at https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170706T1800 [01:48:11] greg-g: Sure! :) [01:48:20] Thanks! [02:36:26] (03PS1) 10Legoktm: Add tests for forbidding use of backtick operator [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/363518 [02:36:30] (03PS1) 10Krinkle: Archive operations/debs/wikimedia-job-runner [integration/config] - 10https://gerrit.wikimedia.org/r/363519 [02:36:39] (03CR) 10Krinkle: [C: 032] Archive operations/debs/wikimedia-job-runner [integration/config] - 10https://gerrit.wikimedia.org/r/363519 (owner: 10Krinkle) [02:37:58] (03Merged) 10jenkins-bot: Archive operations/debs/wikimedia-job-runner [integration/config] - 10https://gerrit.wikimedia.org/r/363519 (owner: 10Krinkle) [02:38:43] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/363519 [02:38:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:15:37] Project beta-scap-eqiad build #162753: 04FAILURE in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/162753/ [03:26:07] Yippee, build fixed! [03:26:07] Project beta-scap-eqiad build #162754: 09FIXED in 2 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/162754/ [03:45:10] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<11.11%) [04:10:05] 10Release-Engineering-Team (Kanban), 10MediaWiki-Special-pages, 10MW-1.25-release, 10Patch-For-Review, and 3 others: Caching of Special:ActiveUsers is broken on small wikis - https://phabricator.wikimedia.org/T89027#3410363 (10Oznogon) [04:14:35] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [04:54:33] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [05:11:54] (03PS1) 10Legoktm: Add Squiz.Classes.SelfMemberReference to ruleset [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/363525 [05:13:50] (03PS2) 10Legoktm: Sniff that the short type form is used in @return tags [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/362593 (https://phabricator.wikimedia.org/T145162) [05:13:55] (03CR) 10Legoktm: Sniff that the short type form is used in @return tags (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/362593 (https://phabricator.wikimedia.org/T145162) (owner: 10Legoktm) [05:13:57] (03CR) 10jerkins-bot: [V: 04-1] Sniff that the short type form is used in @return tags [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/362593 (https://phabricator.wikimedia.org/T145162) (owner: 10Legoktm) [05:15:47] (03PS3) 10Legoktm: Sniff that the short type form is used in @return tags [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/362593 (https://phabricator.wikimedia.org/T145162) [06:50:02] Yippee, build fixed! [06:50:02] Project selenium-Wikibase » chrome,test,Linux,BrowserTests build #413: 09FIXED in 2 hr 10 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=BrowserTests/413/ [07:15:09] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:02:52] 10Release-Engineering-Team (Kanban), 10Diffusion, 10Phabricator (2017-07-05), 10Regression: Unable to browse refs/meta/config in Diffusion - https://phabricator.wikimedia.org/T137354#3410665 (10mmodell) 05Open>03Resolved [08:55:37] (03PS1) 10WMDE-Fisch: Enable phan checks for the FileImporter extension [integration/config] - 10https://gerrit.wikimedia.org/r/363551 [09:09:57] 10Release-Engineering-Team, 10Page-Previews, 10Reading-Web-Backlog, 10Reading-Web-Kanban-Board: Create bot that automatically rebases and rebuilds patches to master - https://phabricator.wikimedia.org/T167181#3410772 (10phuedx) I see that the topic was preserved on {d2a9d1845682710eaab30547393b95b3bcd063b7} 👌 [09:20:03] (03Abandoned) 10WMDE-Fisch: Enable phan checks for the FileImporter extension [integration/config] - 10https://gerrit.wikimedia.org/r/363551 (owner: 10WMDE-Fisch) [09:20:27] (03PS2) 10Addshore: Add extension-phan-generic to FileImporter [integration/config] - 10https://gerrit.wikimedia.org/r/363220 [09:20:33] (03CR) 10WMDE-Fisch: [C: 031] "This is ready to go now!" [integration/config] - 10https://gerrit.wikimedia.org/r/363220 (owner: 10Addshore) [09:29:14] 10Beta-Cluster-Infrastructure, 10Operations, 10Performance-Team, 10Thumbor: Beta thumbnails are broken - https://phabricator.wikimedia.org/T169114#3410797 (10fgiunchedi) thanks @Gilles for the debugging! I think it is due to me moving some swift settings from wikitech Hiera: page to horizon, I've put the l... [09:42:42] (03CR) 10WMDE-Fisch: [C: 031] Add extension-phan-generic to FileImporter [integration/config] - 10https://gerrit.wikimedia.org/r/363220 (owner: 10Addshore) [09:51:53] 10Release-Engineering-Team (Kanban), 10Math, 10Browser-Tests, 10JavaScript, and 2 others: WebdriverIO tests for Math - https://phabricator.wikimedia.org/T162455#3163692 (10hashar) The webdriver.io patch on https://gerrit.wikimedia.org/r/#/c/346982/ is a port of the ruby browser test. It basically: * edit a... [11:28:12] are there any docs for how to test/access services deployed to the beta cluster? [11:32:44] 10Deployment-Systems, 10Scap (Scap3-Adoption-Phase1), 10scap2, 10monitoring: Deploy statsv with scap3 - https://phabricator.wikimedia.org/T129139#3411138 (10fgiunchedi) a:03fgiunchedi I'll take a look at this [11:48:18] schana: hello; what are you willing to test? [11:48:38] I'm wanting to test the recommendation-api service [11:50:02] schana: I am not sure it is exposed [11:50:40] hashar: is there somewhere I could tunnel through that would have access to it? [11:51:51] schana: there is http://restbase-beta.wmflabs.org/ but it does not show anything useful [11:52:27] hashar: alternatively, is there any docs pertaining to the non-mediawiki aspects of the beta cluster? [11:52:42] I've been looking but haven't found anything too usefull [11:52:58] I dont think there is much :( or at least I am not aware about any such doc [11:53:08] maybe #wikimedia-services will know more [11:53:34] thanks, hashar [11:53:58] schana: ahhhh https://en.wikipedia.beta.wmflabs.org/api/rest_v1/ maybe it is there :D [11:54:22] but maybe the recommendation-api (whatever it can be) is not behind restbase [11:54:25] hashar: not there, unfortunately [11:55:21] schana: or maybe it is not set up on the beta cluster [11:56:17] hashar: I think it's set up on sca0[1-4] on port 9632 [11:56:19] https://meta.wikimedia.org/wiki/Recommendation_API states #wikimedia-research [11:56:38] hashar: I'm on the research team [11:56:45] hhhhooooooo :D [11:57:15] I'll ping services when they start coming online [11:59:16] schana: supposedly you get ssh access to deployment-xx hosts [11:59:25] so at least that is a start [11:59:59] hashar: I don't know what those hosts are or how to utilize them [12:03:05] PROBLEM - Puppet errors on integration-slave-docker-1000 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [12:05:04] !log deployment-prep created Web proxy for recommendation-api-beta.wmflabs.org -> http://10.68.20.183:9632 (deployment-sca01) for schana [12:05:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:06:34] bah that does not work :( [12:10:25] hashar: the info I have: T148129#3402480 [12:10:25] T148129: Productization of Recommendation API - https://phabricator.wikimedia.org/T148129 [12:11:08] ah [12:11:37] schana: at least you might be able to reach it using ssh port redirection: ssh -N -L 9632:127.0.0.1:9632 deployment-sca01.deployment-prep.eqiad.wmflabs [12:12:01] that will make it listen on 9632 on your local machine and forward it to the remote on the same port [12:12:15] then from your machine curl http://localhost:9632/ [12:14:58] hashar: that works, thanks! [12:15:14] hashar: well, maybe not [12:17:09] schana: that is not idea but I guess others will comment on it [12:17:53] thanks again, hashar [12:26:56] (03CR) 10Hashar: "> `R CMD check` runs scripts in tests/, which is how testthat.R gets triggered and then runs the unit tests (if testthat is installed)" [integration/config] - 10https://gerrit.wikimedia.org/r/362309 (https://phabricator.wikimedia.org/T153856) (owner: 10Hashar) [12:39:41] hi, I'm getting a failure for mediawiki-core-qunit-selenium-jessie for mw-core and the REL1_27 branch, but I don't understand what's wrong [12:39:58] c.https://gerrit.wikimedia.org/r/#/c/363571/ [12:43:01] RECOVERY - Puppet errors on integration-slave-docker-1000 is OK: OK: Less than 1.00% above the threshold [0.0] [12:43:56] castor failed. [12:48:37] paladox: is it something I need to fix in my patch? [12:48:44] nope [12:48:49] hashar ^^ [12:58:57] Project selenium-MinervaNeue-358083 » chrome,beta,Linux,BrowserTests build #3: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue-358083/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/3/ [13:04:24] Yippee, build fixed! [13:04:24] Project selenium-Math » chrome,beta,Linux,BrowserTests build #444: 09FIXED in 23 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/444/ [13:07:11] (03PS4) 10Hashar: (DO NOT SUBMIT) experimental R based job [integration/config] - 10https://gerrit.wikimedia.org/r/362309 (https://phabricator.wikimedia.org/T153856) [13:07:30] (03CR) 10Hashar: "I found a promising upstream bug https://github.com/hadley/devtools/issues/1343 . Apparently repos= is not being passed to transient depe" [integration/config] - 10https://gerrit.wikimedia.org/r/362309 (https://phabricator.wikimedia.org/T153856) (owner: 10Hashar) [13:11:09] hashar: ! could you turn phan on for FileImporter? [13:11:19] https://gerrit.wikimedia.org/r/#/c/363220/ [13:37:43] (03CR) 10Hashar: [C: 032] Add extension-phan-generic to FileImporter [integration/config] - 10https://gerrit.wikimedia.org/r/363220 (owner: 10Addshore) [13:37:46] addshore: yup [13:37:50] wooo [13:39:12] (03Merged) 10jenkins-bot: Add extension-phan-generic to FileImporter [integration/config] - 10https://gerrit.wikimedia.org/r/363220 (owner: 10Addshore) [13:44:55] addshore: done!!!! :] [13:46:49] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #451: 04FAILURE in 2 min 48 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/451/ [13:47:59] Thanks! [13:51:02] Yippee, build fixed! [13:51:03] Project selenium-MinervaNeue-358083 » chrome,beta,Linux,BrowserTests build #4: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue-358083/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/4/ [13:52:11] (03PS5) 10Hashar: R based job for wikimedia/discovery/ortiz [integration/config] - 10https://gerrit.wikimedia.org/r/362309 (https://phabricator.wikimedia.org/T153856) [13:56:37] (03CR) 10jerkins-bot: [V: 04-1] R based job for wikimedia/discovery/ortiz [integration/config] - 10https://gerrit.wikimedia.org/r/362309 (https://phabricator.wikimedia.org/T153856) (owner: 10Hashar) [13:58:07] hashar: how about https://gerrit.wikimedia.org/r/363222 ? thoughts? [14:00:29] Project selenium-MinervaNeue-358083 » firefox,beta,Linux,BrowserTests build #4: 04FAILURE in 33 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue-358083/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/4/ [14:01:57] hashar hi, it seems castor is causing mw 1.27 tests to fail [14:01:59] https://integration.wikimedia.org/ci/job/mediawiki-core-qunit-selenium-jessie/3475/console [14:05:09] (03PS6) 10Hashar: R based job for wikimedia/discovery/ortiz [integration/config] - 10https://gerrit.wikimedia.org/r/362309 (https://phabricator.wikimedia.org/T153856) [14:08:58] paladox: 00:02:32.605 ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? [14:09:07] let me check the job config [14:09:11] oh so it's not castor? [14:12:28] paladox: it is 100% my fault [14:12:32] oh [14:14:05] !log regenerating mediawiki-core-qunit-selenium-jessie jenkins job [14:14:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:15:14] paladox: should be good now. I had the job refreshed with some outdated version of jjb :/ [14:15:23] thanks :) [14:15:49] dcausse ^^ [14:15:53] should be fixed now :) [14:16:03] paladox, hashar thanks! [14:17:02] (03CR) 10Hashar: [C: 031] "So that is surely need a bit of further polishing up:" [integration/config] - 10https://gerrit.wikimedia.org/r/362309 (https://phabricator.wikimedia.org/T153856) (owner: 10Hashar) [14:17:34] dcausse: 100% my fault sorry about that and thank you to have reported it! [14:17:57] hashar: no worry! :) [14:35:10] https://integration.wikimedia.org/zuul/ nodepool looks slow [14:35:13] hashar ^^ [14:38:00] Yippee, build fixed! [14:38:00] Project selenium-MinervaNeue-358083 » firefox,beta,Linux,BrowserTests build #5: 09FIXED in 34 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue-358083/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/5/ [14:42:49] paladox: on that page look for a [Nodepool] button at the bottom [14:42:55] that brings you to https://grafana.wikimedia.org/dashboard/db/nodepool [14:42:59] which show the state of the pool [14:43:03] ah [14:43:03] https://grafana.wikimedia.org/dashboard/db/nodepool?orgId=1 [14:43:07] https://grafana.wikimedia.org/dashboard/db/nodepool?panelId=1&fullscreen&orgId=1&from=now-1h&to=now [14:43:21] bunch of nodes are being rebuild and others are waiting for deletion [14:43:26] yep [14:51:12] hullo all! [14:51:15] (03PS6) 10Hashar: Templatize phplint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/363493 (owner: 10Ejegg) [14:52:40] ah, looks like I missed a spot. Thanks for the parameter_functions fix hashar! [14:52:49] look good otherwise? [14:55:31] 10Gerrit, 10MediaWiki-Extensions-Pickle, 10Repository-Admins: Archive Pickle repo at Gerrit - https://phabricator.wikimedia.org/T167610#3411839 (10Florian) 05Resolved>03Open No, it's still active :) [14:55:40] 10Gerrit, 10MediaWiki-Extensions-Pickle, 10Repository-Admins: Archive Pickle repo at Gerrit - https://phabricator.wikimedia.org/T167610#3411841 (10Florian) a:03Florian [14:56:55] (03CR) 10Ejegg: "I added the phpflavor-php56 labels to integration-slave-jessie-1001 and -1002. We're only going to need php5.6 for the fundraising jobs to" [integration/config] - 10https://gerrit.wikimedia.org/r/363493 (owner: 10Ejegg) [14:57:20] 10Gerrit, 10MediaWiki-Extensions-Pickle, 10Repository-Admins: Archive Pickle repo at Gerrit - https://phabricator.wikimedia.org/T167610#3338669 (10Florian) [14:57:40] (03CR) 10Hashar: [C: 031] "Slightly amended to set PHP_BIN whenever the jobnames contains "php56". That environment variable is used by our 'php' shell script to sw" [integration/config] - 10https://gerrit.wikimedia.org/r/363493 (owner: 10Ejegg) [14:58:07] ejegg: I guess we can deploy it :] [14:58:27] woohoo! [14:58:47] 10Gerrit, 10MediaWiki-Extensions-Pickle, 10Repository-Admins: Archive Pickle repo at Gerrit - https://phabricator.wikimedia.org/T167610#3411856 (10Florian) [14:59:10] (03CR) 10Ejegg: [C: 031] "Approving Hashar's change" [integration/config] - 10https://gerrit.wikimedia.org/r/363493 (owner: 10Ejegg) [14:59:24] (03CR) 10Hashar: [C: 032] "Thank you both!" [integration/config] - 10https://gerrit.wikimedia.org/r/363493 (owner: 10Ejegg) [15:00:14] ejegg: you deserve a jjb barnstar :] [15:00:27] and kudos to thcipriani|afk for the phpflavor-php56 label [15:00:27] aww, thanks! [15:00:43] (03PS1) 10Florianschmidtwelzow: Archive the mediawiki/extensions/Pickle [integration/config] - 10https://gerrit.wikimedia.org/r/363614 (https://phabricator.wikimedia.org/T167610) [15:00:58] always fun to work with -releng and learn something new! [15:01:01] (03PS1) 10Mholloway: Android: Add -no-window command line option when booting emulator [integration/config] - 10https://gerrit.wikimedia.org/r/363615 [15:01:15] what is challenging is that we are supporting all teams [15:01:35] I have spent a few days learning R and how to install package in that language :d [15:01:45] luckily there are always people willing to assist / teach etc [15:01:46] oh wow... [15:02:15] (03CR) 10Hashar: [C: 031] Android: Add -no-window command line option when booting emulator [integration/config] - 10https://gerrit.wikimedia.org/r/363615 (owner: 10Mholloway) [15:02:15] you have to have a really wide range of knowledge [15:02:21] hmm [15:02:24] that is one could pretend [15:02:43] realisticly, I have no clue what I am doing [15:02:59] and yet things seem to work well most days! [15:03:04] I am merely copy pasting bash scripts provided by developers, mangle them with some obscure YAML syntax [15:03:11] and use jjb to pretend it is devops [15:03:11] :D [15:03:16] haha [15:03:47] (03Merged) 10jenkins-bot: Templatize phplint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/363493 (owner: 10Ejegg) [15:04:06] deployed [15:04:23] rockin! [15:04:39] (03PS6) 10Ejegg: Switch all fundraising tests to PHP5.6 [integration/config] - 10https://gerrit.wikimedia.org/r/363478 (https://phabricator.wikimedia.org/T169806) [15:05:00] ahah [15:05:06] I was going to ask about that patch :] [15:05:22] zend 5.5 => 5.6 should also make the job slightly faster [15:09:14] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:13:55] 10Release-Engineering-Team (Kanban), 10Math, 10VPS-Projects: Instances in math project show high system CPU usage - https://phabricator.wikimedia.org/T160824#3411953 (10hashar) 05Open>03Resolved a:03hashar That has been fixed around June 21th when all labvirt / instances have been rebooted. [15:20:11] (03CR) 10Hashar: [C: 032] Switch all fundraising tests to PHP5.6 [integration/config] - 10https://gerrit.wikimedia.org/r/363478 (https://phabricator.wikimedia.org/T169806) (owner: 10Ejegg) [15:20:18] ejegg: I am deployed the change and the jobs [15:20:40] awesome, thanks! [15:21:22] our civi-rockstar contractor has been writing some CiviCRM extensions that can actually be used outside of our crazy setup [15:21:54] but she likes to test with fancy new versions of phpunit that don't work in php5.5 [15:22:45] (03CR) 10Niedzielski: [V: 032 C: 032] Android: Add -no-window command line option when booting emulator (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/363615 (owner: 10Mholloway) [15:23:00] (03CR) 10jerkins-bot: [V: 04-1] Switch all fundraising tests to PHP5.6 [integration/config] - 10https://gerrit.wikimedia.org/r/363478 (https://phabricator.wikimedia.org/T169806) (owner: 10Ejegg) [15:23:04] ejegg: and I guess if your prod install is on jessie / zend 5.6 it makes sense this way [15:23:35] 00:00:52.129 FAILURE: Job wikimedia-fundraising-crm-composer-php56-jessie not defined [15:23:35] bah [15:24:22] (03CR) 10Hashar: [C: 032] "Failed because I forgot to deploy the job wikimedia-fundraising-crm-composer-php56-jessie" [integration/config] - 10https://gerrit.wikimedia.org/r/363478 (https://phabricator.wikimedia.org/T169806) (owner: 10Ejegg) [15:24:30] hmm [15:24:37] ahhh [15:25:06] that is the sole reason for that job that runs on CR+2 [15:25:13] yeah, being on the same version as production is also a very good reason [15:25:17] it compare the list of jobs in JJB against the actually deployed jobs [15:25:24] if one job is missing in Jenkins, it bails out :] [15:25:31] sensible [15:26:14] then I guess I will move the crm job to the Nodepool / disposable instances [15:26:51] oh, was thcipriani able to get those tagged with phpflavor-php56 ? [15:27:28] (03Merged) 10jenkins-bot: Switch all fundraising tests to PHP5.6 [integration/config] - 10https://gerrit.wikimedia.org/r/363478 (https://phabricator.wikimedia.org/T169806) (owner: 10Ejegg) [15:28:15] I didn't tag them, though they are tagged :) https://integration.wikimedia.org/ci/computer/integration-slave-jessie-1001/ https://integration.wikimedia.org/ci/computer/integration-slave-jessie-1002/ [15:28:44] ejegg did the magic :) [15:29:19] !log deployment-cache-upload04 manually ran apt-get upgrade to downgrade ldap-utils and libldap-2.4-2 (caused puppet failure) [15:29:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:29:23] I had a question I didn't know the answer to during review of this patch: can nodepool add a label? [15:29:48] yes [15:29:52] if you create a new class of nodes [15:30:08] but afaik there can only be one node per instance [15:30:35] ah, that was what I was having trouble figuring out. [15:30:54] many thanks to you both [15:31:01] thcipriani: https://github.com/wikimedia/puppet/blob/production/modules/nodepool/templates/nodepool.yaml.erb#L52-L74 [15:31:17] and each label is a split of the pool [15:31:53] but then, I guess we will start moving stuff to Docker soonish. Addshore had a request in that sense for phan iirc [15:32:13] :D [15:32:23] ejegg: I have deployed the zuul config [15:32:27] I saw that in scrollback. I need to think about a sane way to move other stuff. The operations/puppet job I built up was pretty ad-hoc [15:32:48] ejegg: addshore: and a note: I am not around next week [15:32:50] addshore: what jobs were you trying to move/create? [15:33:18] From where I am sat, the biggest hurdle I see is the wmf needs an image repo (that maybe auto builds images) that I could add too [15:33:27] thcipriani: I was going to move phan initially [15:33:41] nice, it's running fine so far: https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm/4530/console [15:33:53] mwext-php70-phan-jessie [15:34:07] wmf does have a docker repo, and base images are coming/here: https://phabricator.wikimedia.org/T162042 [15:34:32] the jessie-slim thing I started from was before those base images were made [15:34:37] so I should revisit [15:35:13] what repo are those images in? so if I wanted to build an image for phan for use in CI I would add it to the same repo as those base images? [15:35:14] I think maybe it makes sense to run a docker registry inside the integration project, currently I'm running one in ci-staging [15:35:15] k, looks like a bad ref in composer.lock, but the job itself is great [15:36:15] thcipriani: yeh that could make sense, and then a collection of images just for CI there. [15:36:26] addshore: so currently I'm just pulling from ci-staging: https://github.com/wikimedia/integration-config/blob/master/jjb/operations-puppet.yaml#L163 [15:36:29] thcipriani: well if staging get a namespaced are in the production docker registry, I guess CI could get a namespace of its own as well ? [15:36:44] aah okay [15:36:45] that is possible as well [15:37:27] I guess it just depends on how open we want it to be. [15:37:46] I would like to be able to pull the images to my local machine too ideally [15:38:12] ejegg: 00:02:39.370 - Installing wikimedia/omnimail-silverpop (dev-master e06e501): Cloning e06e50192a [15:38:12] 00:02:39.739 e06e50192ae17fe93f1045ee7562aa286272cbcc is gone (history was rewritten?) [15:38:16] that is horrible :]]]] [15:39:16] Yeh thcipriani hashar when I started looking into this I was expecting to find some stuff @ https://wikitech.wikimedia.org/wiki/Docker but its pretty empty [15:39:33] so just been digging around asking questions trying to find out where the infra is / whats happening [15:39:35] haha, it may just be something eileen hasn't put up on her github yet [15:39:48] addshore: blame me. I am responsible for that stub :] [15:39:52] so what I would like for these images to do is to fetch down new code inside the image as part of their run, and leave nothing on disk to clean. That's somewhat critical just because of the lack of k8s setup in CI right now. [15:40:40] this is what I tried to do with the puppet image: puppet reaches out to the zuul merger to grab code to test based on env vars passed into the container [15:40:54] RECOVERY - Puppet errors on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0] [15:41:10] --volume /srv/git:/srv/git:ro \ so it clones from there? [15:41:21] unfortunately that also means it only works within the integration project right now since the git daemon that the zuul merger runs is not public [15:41:36] addshore: that's just a cache so it can do: git clone --reverence [15:41:40] er reference :) [15:41:44] ack [15:42:03] "since the git daemon that the zuul merger runs is not public" not really sure about that stuff [15:42:37] yeah, I think to meet your use-case we'd need to vary the source of the code based on the environment...something like that [15:42:51] that would allow you to run the same tests as CI locally [15:42:58] hmm [15:43:56] so, say for phan, 1 env var would be a dir on the host machine that contains the git repo the job is being run for, another would then be the command to be run to get the change needed [15:44:25] it might make some sense to make a base docker image for CI that installs all the crap we use for CI and runs the git stuff. Then we can build actual tests on top. Does that make some sense? [15:45:03] so for me locally if I wanted to run phan on core I would pass in something like GIT=~/dev/git/gerrit/mediawiki GERRITFETCH=mediawiki/core refs/changes/46/276446/14 [15:45:40] in my head at least I could make an image with that which would work in CI and locally [15:45:51] Yippee, build fixed! [15:45:51] Project selenium-MobileFrontend » chrome,beta,Linux,BrowserTests build #480: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/480/ [15:46:48] seems reasonable for local. Could have whatever is running on the base image vary the way it fetches code based on whether it is passed a ZUUL_REF [15:47:13] we have this thing in CI called zuul-merger, it's why when you look at https://integration.wikimedia.org/zuul/ you see chains of patches [15:47:29] 15:37:52 [CHECKSTYLE] Searching for all files in /home/jenkins/workspace/mediawiki-core-php70-phan-jessie that match the pattern src/tests/phan/issues/latest [15:47:30] 15:37:52 [CHECKSTYLE] No files found. Configuration error? [15:47:30] each patch in the patch-set is the code as if the patch in from has already merged [15:47:54] has something changed today for mediawiki-core-php70-phan-jessie? [15:48:37] hrm, I that may have been part of stuff that landed earlier...checking [15:52:16] gilles: looks like my initial thought was incorrect, nothing should have changed with the actual job configuration today vs yesterday, still digging [15:52:57] https://integration.wikimedia.org/ci/job/mediawiki-core-php70-phan-jessie/jobConfigHistory/ [15:53:06] thcipriani: I mean, actually, locally, you could just spin up a whole CI infra in a container ;) maybe thats what we should just aim for ;) [15:53:32] like running your own jenkins? :) [15:53:46] thcipriani: sure :D [15:54:04] why stop at jenins ;) [15:54:11] *jenkins [15:54:45] gilles: what change is that on? O_o looks like phan failed to write the file that it should have [15:54:57] https://gerrit.wikimedia.org/r/#/c/362941/ [15:55:31] addshore: thcipriani: PLEASE DO YOu HAVE My BLeSSINg !!!!44!!! [15:55:42] hahahaaaaaa [15:56:44] gilles: oooh, pahn failed to install in the first place [15:56:53] bonus point if one manage to integrate https://github.com/travis-ci/travis-build :D [15:57:05] etsy/phan 0.7 requires ext-sqlite3 0.7-dev -> the requested PHP extension sqlite3 has the wrong version (7.0.21-1+0~20170706092253.2+jessie~1.gbp57c738) [15:57:42] some pinning is required? [15:57:59] argh [15:58:00] https://github.com/etsy/phan/tree/0.7 nothing has changed in phan [15:58:10] I bet that is a package installed from sury.org :^\ [15:58:13] 20170706 looks like that deb was built today [15:58:26] 7.0.21-1+0~20170706092253.2+jessie~1.gbp57c738 [15:58:43] we get php 7.0 from sury.org and automagically refresh the instances at 14:14UTC [15:59:16] 2017-07-06 14:19:53,156 INFO nodepool.image.build.wmflabs-eqiad.snapshot-ci-jessie: Debug: /Stage[main]/Contint::Packages::Php/Package[php7.0-sqlite3]/ensure: php7.0-sqlite3 "7.0.16-3+0~20170222101552.24+jessie~1.gbpb3eec3" is installed, latest is "7.0.21-1+0~20170706092253.2+jessie~1.gbp57c738" [15:59:21] 2017-07-06 14:19:53,689 INFO nodepool.image.build.wmflabs-eqiad.snapshot-ci-jessie: Notice: /Stage[main]/Contint::Packages::Php/Package[php7.0-sqlite3]/ensure: ensure changed '7.0.16-3+0~20170222101552.24+jessie~1.gbpb3eec3' to '7.0.21-1+0~20170706092253.2+jessie~1.gbp57c738' [15:59:40] ((from: ssh labnodepool1001.eqiad.wmnet grep sqlite3 /var/log/nodepool/image.log [16:07:32] looking at https://packages.sury.org/php/pool/main/p/php7.0/ [16:08:20] I think that is https://packages.sury.org/php/pool/main/p/php7.0/php7.0-sqlite3_7.0.21-1%2B0~20170706092253.2%2Bjessie~1.gbp57c738_amd64.deb [16:13:12] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:16:48] addshore: gilles: that looks like an issue with the php7.0 package we install from sury.org. Seems he realized a snapshot build [16:18:54] 10Continuous-Integration-Infrastructure, 10Jenkins: etsy/phan 0.7 requires ext-sqlite3 0.7-dev -> the requested PHP extension sqlite3 has the wrong version (7.0.21-1+0~20170706092253.2+jessie~1.gbp57c738) installed - https://phabricator.wikimedia.org/T169903#3412120 (10Florian) [16:19:20] 10Continuous-Integration-Infrastructure: mediawiki-core-php70-phan-jessie requested PHP extension sqlite3 has the wrong version - https://phabricator.wikimedia.org/T169904#3412134 (10hashar) [16:19:50] 10Continuous-Integration-Infrastructure: mediawiki-core-php70-phan-jessie requested PHP extension sqlite3 has the wrong version - https://phabricator.wikimedia.org/T169904#3412146 (10hashar) [16:20:32] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:20:57] !log Deleting Nodepool snapshot snapshot-ci-jessie-1499350442 - faulty php7.0-sqlite package that breaks phan jobs - T169904 [16:21:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:21:01] T169904: mediawiki-core-php70-phan-jessie requested PHP extension sqlite3 has the wrong version - https://phabricator.wikimedia.org/T169904 [16:21:12] it is broken somehow bah [16:27:27] 10Continuous-Integration-Infrastructure: mediawiki-core-php70-phan-jessie requested PHP extension sqlite3 has the wrong version - https://phabricator.wikimedia.org/T169904#3412157 (10hashar) That is probably a fault in the sury.org package :-( I have deleted the Nodepool reference image that got generated today... [16:27:33] I reverted the nodepool snapshot [16:27:36] so that might fix it [16:27:43] one might want to fill an issue to sury.org [16:27:50] I have to escape, got a board meeting in half an hour [16:35:25] PROBLEM - Puppet errors on deployment-mediawiki04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:50:25] RECOVERY - Puppet errors on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:57:43] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:58:52] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations, 10Patch-For-Review: Decide on /var/lib vs /home as locations of homedir for mwdeploy - https://phabricator.wikimedia.org/T86971#3412439 (10demon) 05Open>03Resolved a:03demon [16:58:58] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:00:08] That's probably due to our patch ^ [17:03:34] PROBLEM - Puppet errors on deployment-tmh01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:03:40] returned 6: usermod: user 'mwdeploy' does not exist in /etc/passwd [17:04:00] godog: I tried `nscd -i passwd` on the beta nodes, but it says the user DNE [17:06:47] RainbowSprinkles: odd, looks like puppet is trying to set the home to sth else [17:06:50] Error: Could not set home on user[mwdeploy]: Execution of '/usr/sbin/usermod -d /var/lib/scap mwdeploy' returned 6: usermod: user 'mwdeploy' does not exist in /etc/passwd [17:06:56] anyways, in a meeting [17:07:03] Hmmm [17:11:11] Error: /Stage[main]/Mediawiki::Users/User[mwdeploy]/home: change from /var/lib/mwdeploy to /home/mwdeploy failed: Could not set home on user[mwdeploy]: Execution of '/usr/sbin/usermod -d /home/mwdeploy mwdeploy' returned 6: usermod: user 'mwdeploy' does not exist in /etc/passwd [17:11:19] More interestingly: it's trying to change it *back* [17:11:23] To /home [17:11:29] Hmmmmmm [17:16:25] PROBLEM - Puppet errors on deployment-mediawiki04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:21:05] PROBLEM - Puppet errors on deployment-mediawiki05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:24:46] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:25:45] RainbowSprinkles: hrm, looks puppetmaster is out of sync with master, submodule has uncommitted changes so git-sync-upstream is bailing... [17:26:09] Ahhh ok [17:26:12] That 'splains it [17:26:22] * thcipriani commits submodule and runs git-sync-upstream [17:27:19] {{done}} [17:28:17] !log committed changes to modules/kafkatee on deployment-puppetmaster02 since having them uncommitted broke git-sync-upstream [17:28:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:32:44] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [17:38:33] RECOVERY - Puppet errors on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:41:50] thcipriani: Thanks [17:51:26] RECOVERY - Puppet errors on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:56:05] RECOVERY - Puppet errors on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [18:04:46] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:09:01] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [18:17:33] 10Continuous-Integration-Infrastructure, 10Jenkins: etsy/phan 0.7 requires ext-sqlite3 0.7-dev -> the requested PHP extension sqlite3 has the wrong version (7.0.21-1+0~20170706092253.2+jessie~1.gbp57c738) installed - https://phabricator.wikimedia.org/T169903#3412741 (10Florian) 05Open>03Invalid Seems to wo... [18:34:34] 10Release-Engineering-Team (Kanban), 10Phabricator: Form 33 should be modified to have the archiving of the git repository in the checklist - https://phabricator.wikimedia.org/T169673#3412784 (10SamanthaNguyen) @Mmodell Sorry to bother, but it seems like there's now 2 duplicate items on the checklist for marki... [18:48:18] 10Release-Engineering-Team (Kanban), 10Phabricator: Form 33 should be modified to have the archiving of the git repository in the checklist - https://phabricator.wikimedia.org/T169673#3412827 (10Aklapper) @SamanthaNguyen: I've removed the duplicated entry (though Phab thinks I also changed the Priority field d... [18:58:14] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Patch-For-Review: Upgrade jenkins to 2.60.1 (new lts release) - https://phabricator.wikimedia.org/T168644#3412850 (10greg) [18:58:49] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Operations: Reimage gerrit2001 as stretch - https://phabricator.wikimedia.org/T168562#3412853 (10greg) [18:59:03] 10Release-Engineering-Team (Next), 10Wikimedia-General-or-Unknown: Work out how to (mass) deploy trivial mediawiki-config changes - https://phabricator.wikimedia.org/T168326#3412855 (10greg) [18:59:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Patch-For-Review: Upgrade git package on zuul-merger instances contint1001 / contint2001 to benefit git-daemon - https://phabricator.wikimedia.org/T161086#3412858 (10greg) [19:00:45] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Zuul: Freshly provisionned zuul fails connecting to Gerrit due to ssh key host - https://phabricator.wikimedia.org/T157912#3412876 (10greg) [19:02:06] 10Continuous-Integration-Config, 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Puppet: Get rid of "import realm.pp" in manifests/site.pp - https://phabricator.wikimedia.org/T154915#3412889 (10greg) [19:02:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Patch-For-Review: Upgrade git package on zuul-merger instances contint1001 / contint2001 to benefit git-daemon - https://phabricator.wikimedia.org/T161086#3121040 (10demon) We have it experimental too, cf: {T140927} [19:04:38] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Zuul: Freshly provisionned zuul fails connecting to Gerrit due to ssh key host - https://phabricator.wikimedia.org/T157912#3020331 (10demon) I wonder if we could add gerrit's SSH key to the auto-ge... [19:08:44] 10Release-Engineering-Team (Watching / External), 10Puppet: Preload TestingAccessWrapper in production mwrepl - https://phabricator.wikimedia.org/T143607#3412958 (10greg) Adding @EBernhardson because I git blamed modules/mediawiki/manifests/mwrepl.pp and modules/mediawiki/manifests/init.pp :) Erik: Thoughts? [19:18:11] 10Release-Engineering-Team (Watching / External), 10Puppet: Preload TestingAccessWrapper in production mwrepl - https://phabricator.wikimedia.org/T143607#3413032 (10EBernhardson) mwrepl has a 'bypass access checks' option. Just type: set bac on [19:18:51] 10Release-Engineering-Team, 10Page-Previews, 10Reading-Web-Backlog, 10Epic: [EPIC] Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#3413033 (10Jdlrobson) I replied to Reading-web-team] Using a bundler in another of our extensions I suggest we revisit this in... [20:05:32] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:06:21] Project beta-scap-eqiad build #162867: 04FAILURE in 2 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/162867/ [20:07:00] Main page does load ^ [20:07:01] Lies [20:10:24] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 54641 bytes in 2.362 second response time [20:11:58] PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:11:59] 10Release-Engineering-Team (Watching / External), 10DBA: Missing / Dropped databases? - https://phabricator.wikimedia.org/T132838#3413255 (10greg) [20:13:59] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:16:33] Yippee, build fixed! [20:16:34] Project beta-scap-eqiad build #162868: 09FIXED in 2 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/162868/ [20:16:52] RECOVERY - App Server Main HTTP Response on deployment-mediawiki04 is OK: HTTP OK: HTTP/1.1 200 OK - 54172 bytes in 1.957 second response time [20:18:53] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 43070 bytes in 2.073 second response time [20:24:58] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:25:17] * RainbowSprinkles kicks beta [20:29:49] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 43080 bytes in 2.087 second response time [20:35:40] 10Release-Engineering-Team (Kanban), 10MediaWiki-General-or-Unknown, 10MW-1.29-release, 10Patch-For-Review: PHP Fatal error: Uncaught Error: Call to a member function canExist() on null - https://phabricator.wikimedia.org/T168337#3413311 (10demon) 05Open>03Resolved a:03demon [20:59:58] RainbowSprinkles oh i now get how your going to support scap with gerrit. Your going to do /srv/gerrit/gerrit/ but your going to create a redirect for the git folder to /var/lib/gerrit2/review_site/git? [21:00:55] A redirect for the git folder? That's already configured...it doesn't need symlinks [21:01:25] oh, even better then move everything under /srv/gerrit/gerrit :) [21:01:30] That's been at /srv/gerrit/git for ages [21:01:43] I'm going to move it all to srv [21:01:49] And out of var [21:01:53] ah thanks :) [21:02:22] * paladox can test now, knowing how we were going to do it. [21:03:09] I mean we could leave it in var and add symlinks but that overcomplicates it I think [21:03:25] yep, you gain more space in /srv/? [21:03:28] Can easily run from srv directly [21:03:35] or at least you do on labs. [21:03:50] Yes but that's not an issue. Gerrit itself is tiny [21:04:04] yep [21:04:36] Main thing is just simplicity in deploys [21:04:40] That's what I want [21:05:00] it will be quicker to deploy with scap instead of waiting for debs to be built. [21:10:32] * paladox will add code to do scap in gerrit class [21:10:44] will need to leave backwords compat code for now until the switch. [21:11:35] Well quicker but also just nicer. Means we can roll out plug-ins without redoing core [21:11:45] And vice versa [21:11:51]