[00:30:26] RECOVERY - Puppet staleness on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [3600.0] [00:34:19] 10MediaWiki-Codesniffer, 10Upstream: PHP_CodeSniffer 3.x breaks when prepend-autoloader: false is set (like it is in MediaWiki core) - https://phabricator.wikimedia.org/T167168#3423439 (10Legoktm) At this point if we don't see progress on my upstream PR, I'll create a temporary PHP_CodeSniffer fork with my pat... [00:43:23] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [00:45:48] (03CR) 10Jforrester: "Are the errors real?" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/362325 (https://phabricator.wikimedia.org/T169189) (owner: 10Legoktm) [01:12:25] PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._srv.byte_percentfree (No valid datapoints found)deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (<11.11%) [01:23:22] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:27:23] RECOVERY - Free space - all mounts on deployment-logstash2 is OK: OK: deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._srv.byte_percentfree (No valid datapoints found) [04:23:41] 10Deployment-Systems, 10Release-Engineering-Team (Watching / External), 10Operations, 10Performance-Team, and 2 others: Translation cache exhaustion caused by changes to PHP code in file scope - https://phabricator.wikimedia.org/T103886#3423647 (10Krinkle) >>! In T103886#3419367, @Joe wrote: > @Krinkle sur... [04:35:34] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Email, and 8 others: Get CI working on Omnimail - https://phabricator.wikimedia.org/T169593#3402888 (10Eileenmcnaughton) [04:35:36] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Email, and 8 others: Get CRM CI php version the same as live - https://phabricator.wikimedia.org/T169806#3423665 (10Eileenmcnaughton) 05Open>03Resolved [04:36:30] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Email, and 7 others: Excluded vendor directory from lint tests when it is in an included extension - https://phabricator.wikimedia.org/T168597#3369431 (10Eileenmcnaughton) 05Open>03Resolved [05:25:11] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%) [07:15:09] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:17:08] 10Deployment-Systems, 10Release-Engineering-Team (Watching / External), 10Operations, 10Performance-Team, and 2 others: Translation cache exhaustion caused by changes to PHP code in file scope - https://phabricator.wikimedia.org/T103886#3424026 (10MoritzMuehlenhoff) >>! In T103886#3423647, @Krinkle wrote:... [10:09:02] Project beta-scap-eqiad build #163536: 04FAILURE in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/163536/ [10:16:06] Yippee, build fixed! [10:16:07] Project beta-scap-eqiad build #163537: 09FIXED in 2 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/163537/ [10:58:07] Project beta-scap-eqiad build #163545: 04FAILURE in 1 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/163545/ [10:59:32] PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:00:21] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<100.00%) [11:06:08] Yippee, build fixed! [11:06:08] Project beta-scap-eqiad build #163546: 09FIXED in 2 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/163546/ [12:44:44] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:51:23] PROBLEM - Puppet staleness on deployment-kafka01 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [43200.0] [13:04:48] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:20:24] PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._srv.byte_percentfree (No valid datapoints found)deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (<33.33%) [14:07:57] hi. Is T150929 a security issue? It's marked as restricted but from here it seems like a non-staff user accessed it https://phabricator.wikimedia.org/T157660#3012350 [14:08:24] If it is security, could someone please take a look at it? (the other task mentioned above was marked as a duplicate) [14:25:24] PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._srv.byte_percentfree (No valid datapoints found)deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (<11.11%) [14:33:37] Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #451: 04FAILURE in 1 min 36 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/451/ [14:40:21] PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._srv.byte_percentfree (No valid datapoints found)deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (<22.22%) [14:48:42] PROBLEM - Puppet errors on deployment-sca04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:55:12] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:57:49] PROBLEM - Puppet errors on deployment-prometheus01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:00:41] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:10:17] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:12:41] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:13:21] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:13:37] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:15:48] PROBLEM - Puppet errors on deployment-sca03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:18:27] oh good, some dependency cycle for services on beta http://tyler.zone/changeprop-cycle.png [15:19:13] mobrovac: ^ did something merge recently? [15:19:27] puppet-wise for services I mean [15:19:59] euh not this week [15:20:13] will tkae a look after the meeting(s) thcipriani|afk [15:20:14] hrm... [15:20:35] okie doke [15:20:42] I may take a glance at cherry picks... [15:27:31] PROBLEM - Puppet errors on deployment-eventlogging03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:56:26] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Escape Blubber config values when compiling to Dockerfile - https://phabricator.wikimedia.org/T167999#3425538 (10thcipriani) 05Open>03Resolved [15:56:48] Yippee, build fixed! [15:56:49] Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #487: 09FIXED in 34 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/487/ [16:28:03] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): blubber: produces empy ENV statements - https://phabricator.wikimedia.org/T170285#3425719 (10dduvall) a:03dduvall [16:51:02] 10Gitblit-Deprecate, 10MW-1.30-release-notes (WMF-deploy-2017-05-23_(1.30.0-wmf.2)), 10Patch-For-Review: Fix references to git.wikimedia.org in all repos - https://phabricator.wikimedia.org/T139089#3425931 (10TerraCodes) [17:26:15] legoktm: just wondering, does https://github.com/legoktm/tools-ci run on labs, er, cloud/toolforge? [17:27:57] PROBLEM - Free space - all mounts on deployment-eventlogging03 is CRITICAL: CRITICAL: deployment-prep.deployment-eventlogging03.diskspace.root.byte_percentfree (<100.00%) [17:58:59] thcipriani or RainbowSprinkles hi, could you re review https://gerrit.wikimedia.org/r/#/c/363726/ please? :) [17:59:26] Not today [17:59:45] ok [18:19:24] greg-g: yes, but the new code is at https://phabricator.wikimedia.org/source/tool-ci/ [18:19:53] I re-wrote it to download files from phabricator instead of cloning all the git repos, it's significantly faster [19:07:42] legoktm: nice! [19:22:33] (03PS1) 10Umherirrender: Run extension-unittests-generic along with extension-gate [integration/config] - 10https://gerrit.wikimedia.org/r/364505 [19:27:12] (03CR) 10Umherirrender: "Mabye there is another way to have voting phpcs on Flow or TimedMediaHandler or the other repos" [integration/config] - 10https://gerrit.wikimedia.org/r/364505 (owner: 10Umherirrender) [19:31:55] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:58:31] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T167893#3348308 (10Jdforrester-WMF) >>! In T167893#3350385, @greg wrote: > Just to be clear, this is indeed `wmf.9` not `wmf.8`. From now on we are keeping a "each week... [20:00:34] 10MediaWiki-Codesniffer: Sniff MediaWiki.Commenting.FunctionComment using doc block not belongs to the function - https://phabricator.wikimedia.org/T170315#3427002 (10Umherirrender) [20:03:52] 10MediaWiki-Codesniffer: MediaWiki.Usage.ReferenceThis to strict - https://phabricator.wikimedia.org/T170316#3427037 (10Umherirrender) [20:06:25] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T167893#3427102 (10thcipriani) [20:10:44] 10Scap (Scap3-Adoption-Phase1), 10scap2, 10Cassandra, 10RESTBase-Cassandra: Deploy logstash logback encoder with scap3 - https://phabricator.wikimedia.org/T116340#3427143 (10GWicke) I am not aware of this being related to services. Dropping the tag for that reason. I suspect that this task is stale, and t... [20:18:36] PROBLEM - Free space - all mounts on deployment-kafka03 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka03.diskspace.root.byte_percentfree (<10.00%) [20:25:38] 10Deployment-Systems, 10Services: Use semantic versioning for services (for consistency with mediawiki core) - https://phabricator.wikimedia.org/T102550#1367472 (10GWicke) We generally use semver for node services and libraries. Release versions are tagged with vX.Y.Z. [20:26:16] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T167893#3427315 (10thcipriani) [20:26:38] 10Deployment-Systems, 10Services (watching): Use semantic versioning for services (for consistency with mediawiki core) - https://phabricator.wikimedia.org/T102550#3427316 (10GWicke) [20:34:04] 10MediaWiki-Codesniffer: Allow inline comments in mediawiki codesniffer - https://phabricator.wikimedia.org/T170025#3427350 (10Umherirrender) [20:34:07] 10MediaWiki-Codesniffer: MediaWiki.WhiteSpace.SpaceBeforeSingleLineComment.NewLineComment is too strict - https://phabricator.wikimedia.org/T168476#3427353 (10Umherirrender) [20:34:32] 10MediaWiki-Codesniffer: Add sniff to check if all returns of a function are void or non-void returns - https://phabricator.wikimedia.org/T168994#3427367 (10Umherirrender) See https://gerrit.wikimedia.org/r/#/c/170304/3/includes/specials/SpecialRandomInCategory.php for an example [20:37:41] bd808: Do you haz thoughts on T116340? [20:37:42] T116340: Deploy logstash logback encoder with scap3 - https://phabricator.wikimedia.org/T116340 [20:37:56] 10MediaWiki-Codesniffer: Using context variable when usable instead of globals - https://phabricator.wikimedia.org/T115433#1724214 (10Umherirrender) Happen by the MediaWiki.Usage.ExtendClassUsage.FunctionVarUsage sniff - https://gerrit.wikimedia.org/r/#/c/301335/ [20:38:05] 10MediaWiki-Codesniffer: Using context variable when usable instead of globals - https://phabricator.wikimedia.org/T115433#3427424 (10Umherirrender) 05Open>03Resolved p:05Triage>03Normal [20:38:40] 10MediaWiki-Codesniffer: Avoid assigment in return statements - https://phabricator.wikimedia.org/T170332#3427441 (10Umherirrender) [20:50:53] RainbowSprinkles: are we still good for doing the minerva deploy tomorrow? If so what time did you want to get started? [20:51:55] We said 10am, right? [20:53:38] RainbowSprinkles: yup just confirming :) not sure if you want to get started then or later [20:53:41] we have 10-12 window [20:53:46] Sounds good [20:53:52] sweeeett [20:54:11] RainbowSprinkles: gonna make MobileFrontend readonly later today [20:54:18] to make sure no changes get lost. [20:54:20] good idea or not? [21:01:47] Sounds fine [21:12:43] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T167893#3427743 (10greg) Expo facto and late message to wikitech-l/etc: https://lists.wikimedia.org/pipermail/wikitech-l/2017-July/088455.html [21:22:11] RainbowSprinkles what will the new gerrit group be called? [21:22:15] gerrit-deployment? [21:22:17] Don't need a new one [21:22:18] or gerrit-dev? [21:22:23] oh [21:22:23] I'm gonna use gerrit-root [21:22:28] ah ok [21:22:29] thanks [21:23:43] Considering you need to be a root to restart the service, we don't want admins deploying something they can't manage :) [21:23:59] lol :) [21:24:14] done, ive changed it to gerrit-admin now :). [21:24:23] also removed the symblink to the git dir [21:27:04] (03PS1) 10Legoktm: Wikidata extension depends upon CirrusSearch [integration/config] - 10https://gerrit.wikimedia.org/r/364601 [21:27:35] (03CR) 10Addshore: [C: 031] "Since around Change-Id: I831220363bbe3e94ee3888134d3eb3d92d06a264" [integration/config] - 10https://gerrit.wikimedia.org/r/364601 (owner: 10Legoktm) [21:28:37] (03CR) 10Legoktm: [C: 032] Wikidata extension depends upon CirrusSearch [integration/config] - 10https://gerrit.wikimedia.org/r/364601 (owner: 10Legoktm) [21:29:40] (03Merged) 10jenkins-bot: Wikidata extension depends upon CirrusSearch [integration/config] - 10https://gerrit.wikimedia.org/r/364601 (owner: 10Legoktm) [21:30:31] !log deploying https://gerrit.wikimedia.org/r/364601 [21:30:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:33:42] 10Release-Engineering-Team (Watching / External), 10Puppet: Preload TestingAccessWrapper in production mwrepl - https://phabricator.wikimedia.org/T143607#3427878 (10greg) @Mattflaschen-WMF ^^ is that sufficient for your use case? [21:49:01] 10Scap (Scap3-Adoption-Phase1), 10scap2, 10Cassandra, 10RESTBase-Cassandra: Deploy logstash logback encoder with scap3 - https://phabricator.wikimedia.org/T116340#3427971 (10greg) It's within the Cassandra dir on tin: ``` gjg@tin:/srv/deployment/cassandra$ ls -1 logstash-logback-encoder metrics-collector... [21:51:56] 10Continuous-Integration-Config, 10MinervaNeue, 10Patch-For-Review, 10Ruby, 10User-zeljkofilipin: Setup CI on Minerva repo - https://phabricator.wikimedia.org/T166750#3427976 (10Jdlrobson) @zeljkofilipin @hashar I'll be deploying this to the cluster tomorrow at 10am PST. I'd appreciate a merge of https:/... [21:51:58] Yippee, build fixed! [21:51:59] Project selenium-MobileFrontend » chrome,beta,Linux,BrowserTests build #488: 09FIXED in 25 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/488/ [21:52:59] (03CR) 10Jdlrobson: [C: 031] "This can now be merged. Depends-on is in master." [integration/config] - 10https://gerrit.wikimedia.org/r/361012 (https://phabricator.wikimedia.org/T166750) (owner: 10Jdlrobson) [22:07:38] (03PS3) 10Legoktm: Setup browser test job for Minerva skin [integration/config] - 10https://gerrit.wikimedia.org/r/361012 (https://phabricator.wikimedia.org/T166750) (owner: 10Jdlrobson) [22:08:50] (03CR) 10Legoktm: [C: 032] "INFO:jenkins_jobs.builder:Number of jobs generated: 1" [integration/config] - 10https://gerrit.wikimedia.org/r/361012 (https://phabricator.wikimedia.org/T166750) (owner: 10Jdlrobson) [22:10:48] (03Merged) 10jenkins-bot: Setup browser test job for Minerva skin [integration/config] - 10https://gerrit.wikimedia.org/r/361012 (https://phabricator.wikimedia.org/T166750) (owner: 10Jdlrobson) [22:18:33] thanks legoktm :) [22:19:06] yeah, bad day on the CI mergers front: Antoine is out this week and Zeljko has the day off today [22:19:10] thanks legoktm :) [22:19:17] on the RelEng CI mergers* [22:19:21] :) [22:20:28] 10Continuous-Integration-Config, 10MinervaNeue, 10Patch-For-Review, 10Ruby, 10User-zeljkofilipin: Setup CI on Minerva repo - https://phabricator.wikimedia.org/T166750#3428147 (10Jdlrobson) [22:21:07] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Services (watching), 10services-tooling: RFC: Streamline NodeJS testing+deployment - https://phabricator.wikimedia.org/T147581#3428151 (10GWicke) [22:22:44] Project selenium-CentralAuth » firefox,beta,Linux,BrowserTests build #454: 04FAILURE in 2 min 43 sec: https://integration.wikimedia.org/ci/job/selenium-CentralAuth/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/454/ [22:23:17] 10Scap, 10Services (later), 10User-mobrovac: Scap config management: Jinja2 fills templates with Pythonic values - https://phabricator.wikimedia.org/T145510#3428177 (10GWicke) [22:23:36] 10Beta-Cluster-Infrastructure, 10Graphoid, 10Services (watching): Automate Graphoid deployment to beta cluster (and auto-rebuild?) - https://phabricator.wikimedia.org/T146810#3428179 (10GWicke) [22:32:11] jdlrobson: let me know if there's anything else you need, I'm pretty excited to see this happen [22:32:27] legoktm: me too. it's been a drag to get here. [22:32:44] I've scheduled a build of https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-MinervaNeue/ to check it passes [22:34:09] thcipriani hi, do you know why i get this https://phabricator.wikimedia.org/P5721 error please? [22:34:19] it dosen't explain clearly what failed [22:34:20] legoktm: btw on the subject of push notifications - i was just requesting whatever storage we use is made generic. I'd hate to see something limited to only one push notification service per user (similar to how watchlist implementation limited us to one list) [22:34:27] only that it failed on the target host [22:34:46] but i agree you should start small - just dont limit the architecture :) [22:35:54] paladox: if you run scap deploy-log -v in that directory that may give you more info [22:36:02] ah thanks [22:36:22] that explains it [22:36:25] thanks alot [22:36:26] os.remove(fullname) [22:36:27] OSError: [Errno 13] Permission denied: '/srv/deployment/phabricator/deployment-cache/revs/6156a17594c2ced22d7cee34ec921344093d11f7/phabricator/scripts/repository/rebuild_summaries.php' [22:38:23] hmm strange [22:38:28] that dir did not exist [22:38:31] when i create it [22:38:39] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Improve Blubber unit test coverage - https://phabricator.wikimedia.org/T168001#3428214 (10dduvall) [22:38:47] and re do scap it still fails with permission errors. but the owner of the folder is me. [22:38:54] which it should as i am deploying as me [22:39:57] aha [22:39:59] found it [22:40:04] it was on target [22:40:11] somehow it manged to get phd to own it [22:42:07] fixed it now [22:42:10] rm -rf deployment-cache [22:42:39] oh puppet crowned them [22:45:11] 10MediaWiki-Releasing, 10MediaWiki-Containers, 10Services (doing), 10User-mobrovac, 10Wikimedia-Hackathon-2015: Ready-to-use Docker package for MediaWiki - https://phabricator.wikimedia.org/T92826#3428272 (10GWicke) [22:56:54] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #1: 09SUCCESS in 24 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/1/ [23:01:55] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): blubber: produces empy ENV statements - https://phabricator.wikimedia.org/T170285#3428348 (10dduvall) p:05Triage>03High [23:04:32] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #1: 09SUCCESS in 31 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/1/ [23:23:23] PROBLEM - Puppet errors on castor is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:30:27] (03CR) 10Legoktm: "Maybe? At least locally I see a pretty big slow down when using HHVM + parallelization due to an increased start up cost. With Hashar's li" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/362325 (https://phabricator.wikimedia.org/T169189) (owner: 10Legoktm)