[00:01:28] (03CR) 10Legoktm: [C: 032] Add phan for multiple extensions [integration/config] - 10https://gerrit.wikimedia.org/r/425446 (owner: 10Legoktm) [00:02:54] (03Merged) 10jenkins-bot: Add phan for multiple extensions [integration/config] - 10https://gerrit.wikimedia.org/r/425446 (owner: 10Legoktm) [00:03:40] !log deploying https://gerrit.wikimedia.org/r/425446 [00:03:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:05:48] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499#4122098 (10MBinder_WMF) >>! In T93499#4121780, @atgo wrote: > Thanks @mmodell. Bummer about the notifications, thanks for letting me know. > > If I might make a suggestion, this might be more intuitive, since I'd think a misse... [00:09:56] (03PS1) 10Legoktm: Add some Thanks dependencies for phan [integration/config] - 10https://gerrit.wikimedia.org/r/425447 [00:11:40] (03CR) 10Legoktm: [C: 032] Add some Thanks dependencies for phan [integration/config] - 10https://gerrit.wikimedia.org/r/425447 (owner: 10Legoktm) [00:12:51] (03Merged) 10jenkins-bot: Add some Thanks dependencies for phan [integration/config] - 10https://gerrit.wikimedia.org/r/425447 (owner: 10Legoktm) [00:16:22] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499#4122104 (10atgo) @mmodell that looks great and makes a lot of sense to me. Thank you! [00:19:01] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499#4122105 (10mmodell) >>! In T93499#4122098, @MBinder_WMF wrote: > I know that that "Bug" form stamps a task with [BUG], and that if a task has that and then the user changes their mind, it can't be removed. Would there be any suc... [00:21:10] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle AntiSpoof extension with MW 1.31 - https://phabricator.wikimedia.org/T191736#4122108 (10Legoktm) [00:21:20] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle CodeEditor extension with MW 1.31 - https://phabricator.wikimedia.org/T191742#4122109 (10Legoktm) [00:21:51] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle OATHAuth extension with MW 1.31 - https://phabricator.wikimedia.org/T191745#4122110 (10Legoktm) [00:29:19] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4122111 (10awight) Still finding strangeness... Reading the virtualenv so... [00:36:50] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations: mwscript rebuildLocalisationCache.php takes 40 minutes - https://phabricator.wikimedia.org/T191921#4122112 (10thcipriani) `fwrite` is definitely different in hhvm -- accounts for all the `lseek` in the hhvm output: https://gist.github.com/thcipriani/... [00:57:34] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<30.00%) [01:32:11] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle OATHAuth extension with MW 1.31 - https://phabricator.wikimedia.org/T191745#4115197 (10Legoktm) License review: OK, https://gerrit.wikimedia.org/r/425451 makes licensing clearer, and corrects license-name. [02:34:22] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4122183 (10Krinkle) [03:20:01] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle OATHAuth extension with MW 1.31 - https://phabricator.wikimedia.org/T191745#4122198 (10Legoktm) [03:20:19] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle OATHAuth extension with MW 1.31 - https://phabricator.wikimedia.org/T191745#4115197 (10Legoktm) [03:21:26] (03PS1) 10Legoktm: Bundle OATHAuth extension [tools/release] - 10https://gerrit.wikimedia.org/r/425455 (https://phabricator.wikimedia.org/T191745) [03:22:29] (03CR) 10Legoktm: "Do we need to indicate that this is only bundled in 1.31+?" [tools/release] - 10https://gerrit.wikimedia.org/r/425455 (https://phabricator.wikimedia.org/T191745) (owner: 10Legoktm) [03:25:36] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle CategoryTree extension with MW 1.31 - https://phabricator.wikimedia.org/T191735#4122203 (10Legoktm) [03:38:03] Project mediawiki-core-code-coverage-php7 build #199: 04STILL FAILING in 38 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/199/ [04:21:42] Project mediawiki-core-code-coverage build #3438: 04STILL FAILING in 1 hr 21 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3438/ [04:40:06] 10Phabricator: Showing of code bloock no work as expected - https://phabricator.wikimedia.org/T191951#4122282 (10Zoranzoki21) [04:44:27] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle CategoryTree extension with MW 1.31 - https://phabricator.wikimedia.org/T191735#4115103 (10Legoktm) This extension is a bit of a mess and clearly neglected :( I started to clean it up https://gerrit.wikimedia.org/r/#/q/project:mediawiki/extensions/CategoryTree+s... [05:18:48] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations: mwscript rebuildLocalisationCache.php takes 40 minutes - https://phabricator.wikimedia.org/T191921#4122327 (10Joe) What are the blockers for the use of PHP7? All I see on the ticket mentioned is the memcached issue, which ops are working on right no... [05:23:18] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<44.44%) [07:02:32] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:06:35] 10Phabricator (Upstream), 10Upstream: Keep the Phabricator toolbar on the top visible when scrolling down - https://phabricator.wikimedia.org/T191540#4122403 (10Johnywhy) >>! In T191540#4109440, @Aklapper wrote: > I don't think it is a common workflow / use case to open the notification menu that often while r... [07:50:40] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle AntiSpoof extension with MW 1.31 - https://phabricator.wikimedia.org/T191736#4122453 (10Legoktm) [08:28:26] (03PS1) 10Hashar: Make debian-glue voting on operations/debs/tidy-0.99 [integration/config] - 10https://gerrit.wikimedia.org/r/425489 (https://phabricator.wikimedia.org/T191771) [08:29:47] 10Phabricator: Showing of code bloock no work as expected - https://phabricator.wikimedia.org/T191951#4122519 (10Peachey88) 05Open>03declined That is correct, Single backticks (`) for single inline code block, or triple backticks for multi-line block. [08:31:39] 10Phabricator: Showing of code bloock no work as expected - https://phabricator.wikimedia.org/T191951#4122531 (10Peachey88) 05declined>03Invalid [08:38:11] Project mwext-phpunit-coverage-publish build #3234: 04FAILURE in 3.5 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3234/ [08:38:18] Project mwext-phpunit-coverage-publish build #3235: 04STILL FAILING in 3.4 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3235/ [08:38:24] Project mwext-phpunit-coverage-publish build #3236: 04STILL FAILING in 3.4 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3236/ [08:39:28] Yippee, build fixed! [08:39:29] Project mwext-phpunit-coverage-publish build #3237: 09FIXED in 1 min 0 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3237/ [09:03:16] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<11.11%) [09:17:09] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-Parser, 10Quibble, and 2 others: [REL1_30] Some parserTests fail on debian stretch using Tidy, because of a new version of libtidy - https://phabricator.wikimedia.org/T191771#4122586 (10hashar) [09:48:59] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-Parser, 10Quibble, and 2 others: [REL1_30] Some parserTests fail on debian stretch using Tidy, because of a new version of libtidy - https://phabricator.wikimedia.org/T191771#4122685 (10MoritzMuehlenhoff) >>! In T191771#4117753, @Lego... [10:01:16] 10Phabricator: Showing of code bloock no work as expected - https://phabricator.wikimedia.org/T191951#4122719 (10Aklapper) {nav icon=book, name=Help} above the text comment box links to https://secure.phabricator.com/book/phabricator/article/remarkup/ where this is explained. It is also linked from our Phabricat... [10:15:12] (03PS1) 10Thiemo Kreuz (WMDE): Optimize ShortCastSyntax sniff for performance [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425504 [10:28:17] (03PS1) 10Thiemo Kreuz (WMDE): Optimize PHPUnitClassUsage sniff for performance [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425505 [10:32:58] (03PS2) 10Thiemo Kreuz (WMDE): Replace strpos() with faster substr() comparisons [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425426 [10:35:50] (03CR) 10Thiemo Kreuz (WMDE): "I just realized the old code would have found ( $dbr )->query(), which the new code will not find any more. But this is a pretty awkward e" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425427 (owner: 10Thiemo Kreuz (WMDE)) [10:54:24] 10Continuous-Integration-Infrastructure, 10Lexicographical data, 10Wikidata, 10Browser-Tests, 10User-zeljkofilipin: MediaWiki core's node selenium tests flaky when run as part of mwext-mw-selenium-node-composer-jessie job - https://phabricator.wikimedia.org/T191537#4109191 (10zeljkofilipin) Apologies for... [10:55:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Lexicographical data, 10Wikidata, and 2 others: MediaWiki core's node selenium tests flaky when run as part of mwext-mw-selenium-node-composer-jessie job - https://phabricator.wikimedia.org/T191537#4122822 (10zeljkofilipin) a:... [10:59:37] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<10.00%) [10:59:46] 10Phabricator (Upstream), 10Upstream: Keep the Phabricator toolbar on the top visible when scrolling down - https://phabricator.wikimedia.org/T191540#4122835 (10Aklapper) > I access the toolbar all the time. Maybe you don't. Diffrent strokes. I'm happy to consider //common// use cases, it's just that I have n... [11:01:33] (03Abandoned) 10Thiemo Kreuz (WMDE): Use strrpos() to look for newlines [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425425 (owner: 10Thiemo Kreuz (WMDE)) [11:09:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Lexicographical data, 10Wikidata, and 2 others: MediaWiki core's node selenium tests flaky when run as part of mwext-mw-selenium-node-composer-jessie job - https://phabricator.wikimedia.org/T191537#4122842 (10zeljkofilipin) p:... [11:24:22] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)), 10Patch-For-Review, 10User-zeljkofilipin: WebdriverIO should run Chrome headlessly - https://phabricator.wikimedia.org/T167507#4122874 (10zeljkofilipin) [11:24:46] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10Epic, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), and 2 others: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#4122877 (10zeljkofilipin) [11:24:49] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)), 10Patch-For-Review, 10User-zeljkofilipin: WebdriverIO should run Chrome headlessly - https://phabricator.wikimedia.org/T167507#3335241 (10zeljkofilipin) 05Open>03Resolved Upda... [11:28:18] (03PS1) 10Thiemo Kreuz (WMDE): Make use of $phpcsFile->eolChar in two sniffs [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425513 [11:33:19] (03CR) 10jerkins-bot: [V: 04-1] Make use of $phpcsFile->eolChar in two sniffs [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425513 (owner: 10Thiemo Kreuz (WMDE)) [11:34:42] (03PS1) 10Thiemo Kreuz (WMDE): Fix IllegalSingleLineComment sniff fix for unclosed comments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425515 [11:47:05] (03PS2) 10Thiemo Kreuz (WMDE): Make use of $phpcsFile->eolChar in two sniffs [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425513 [12:08:18] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<44.44%) [12:38:41] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:43:02] 10Deployments, 10Release-Engineering-Team, 10Operations, 10Services (watching): Scap sync-file failing for 9 hosts - https://phabricator.wikimedia.org/T191972#4122995 (10mobrovac) [12:43:45] 10Deployments, 10Release-Engineering-Team, 10Operations, 10Services (watching): Scap sync-file failing for 9 hosts - https://phabricator.wikimedia.org/T191972#4123005 (10mobrovac) p:05Triage>03Unbreak! [12:45:45] 10Deployments, 10Release-Engineering-Team, 10Operations, 10Services (blocked): Scap sync-file failing for 9 hosts - https://phabricator.wikimedia.org/T191972#4123012 (10mobrovac) [12:47:58] 10Deployments, 10Release-Engineering-Team, 10Operations, 10Services (blocked): Scap sync-file failing for deploy1001.eqiad.wmnet - https://phabricator.wikimedia.org/T191972#4123014 (10jcrespo) [12:53:07] 10Deployments, 10Release-Engineering-Team, 10Operations, 10Services (blocked): Scap sync-file failing for deploy1001.eqiad.wmnet - https://phabricator.wikimedia.org/T191972#4123026 (10mobrovac) Apparently `deploy1001` has been recently reimaged. However, it doesn't seem like it has a role associated with i... [12:54:28] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:56:42] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#3589662 (10mobrovac) It seems that the reimage is now blocking deployments, cf. {T191972} [13:05:44] 10Deployments, 10Release-Engineering-Team, 10Operations, 10Services (blocked): Scap sync-file failing for deploy1001.eqiad.wmnet - https://phabricator.wikimedia.org/T191972#4123040 (10jcrespo) [13:06:44] 10Deployments, 10Release-Engineering-Team, 10Operations: Scap sync-file failing for deploy1001.eqiad.wmnet - https://phabricator.wikimedia.org/T191972#4123042 (10mobrovac) p:05Unbreak!>03High It's fialing only on `deploy1001`, so lowering the priority. [13:36:33] PROBLEM - Puppet errors on deployment-eventlog05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:15:10] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4123214 (10Halfak) This sounds surprising and strange. Please ping me on... [14:47:48] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Documentation, 10JavaScript, 10User-zeljkofilipin: Blog posts about new Selenium framework features - https://phabricator.wikimedia.org/T191982#4123337 (10zeljkofilipin) [14:47:55] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q4 Selenium framework improvements - https://phabricator.wikimedia.org/T190994#4123347 (10zeljkofilipin) [14:47:57] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Documentation, 10JavaScript, 10User-zeljkofilipin: Blog posts about new Selenium framework features - https://phabricator.wikimedia.org/T191982#4123346 (10zeljkofilipin) [14:48:06] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Documentation, 10JavaScript, 10User-zeljkofilipin: Blog posts about new Selenium framework features - https://phabricator.wikimedia.org/T191982#4123337 (10zeljkofilipin) p:05Triage>03Normal [14:50:41] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Documentation, 10JavaScript, 10User-zeljkofilipin: Blog posts about new Selenium framework features - https://phabricator.wikimedia.org/T191982#4123358 (10zeljkofilipin) [14:54:03] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Documentation, 10JavaScript, 10User-zeljkofilipin: Blog posts about new Selenium framework features - https://phabricator.wikimedia.org/T191982#4123383 (10zeljkofilipin) [15:16:59] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-04-03 (1.31.0-wmf.28)), 10Patch-For-Review, 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#4123495 (10zeljkofilipin) Updated documentation: [[ https://www.mediawiki... [15:20:07] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-04-03 (1.31.0-wmf.28)), 10Patch-For-Review, 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#4123505 (10zeljkofilipin) [15:27:44] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:41] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:38:47] Project mediawiki-core-code-coverage-php7 build #200: 04STILL FAILING in 38 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/200/ [15:44:40] Question: I have a couple of new repos, and am moving performance team apps out of puppet and into them so we can deploy with scap. How do I make Jenkins/zuul aware of these repos, so that merges happen? [15:48:02] marlier hi, you add the repos to layout.yaml in integration/config [15:48:09] for example see [15:48:33] https://gerrit.wikimedia.org/r/#/c/425446/1/zuul/layout.yaml [15:48:40] - name: mediawiki/extensions/AntiSpoof [15:48:44] that's the name of the repo [15:48:51] then - name: extension-unittests-generic [15:49:46] nifty, thanks [15:49:47] or something like [15:49:48] https://phabricator.wikimedia.org/P6983 [15:58:29] 10Release-Engineering-Team, 10MediaWiki-extensions-WikimediaIncubator, 10Epic, 10I18n: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585#4123710 (10Nikerabbit) [15:59:58] (03PS1) 10Imarlier: integration/config: add new perf repos to zuul layout [integration/config] - 10https://gerrit.wikimedia.org/r/425551 (https://phabricator.wikimedia.org/T191994) [16:01:12] (03CR) 10jerkins-bot: [V: 04-1] integration/config: add new perf repos to zuul layout [integration/config] - 10https://gerrit.wikimedia.org/r/425551 (https://phabricator.wikimedia.org/T191994) (owner: 10Imarlier) [16:02:53] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-04-03 (1.31.0-wmf.28)), 10Patch-For-Review, 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#4123738 (10zeljkofilipin) [16:03:46] PROBLEM - Puppet errors on integration-cumin is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:06:27] (03CR) 10Paladox: integration/config: add new perf repos to zuul layout (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/425551 (https://phabricator.wikimedia.org/T191994) (owner: 10Imarlier) [16:11:47] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-04-03 (1.31.0-wmf.28)), 10Patch-For-Review, 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#4123798 (10zeljkofilipin) [16:15:35] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-04-03 (1.31.0-wmf.28)), 10Patch-For-Review, 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#4123822 (10zeljkofilipin) [16:20:47] 10Deployments, 10Release-Engineering-Team, 10Operations: Scap sync-file failing for deploy1001.eqiad.wmnet - https://phabricator.wikimedia.org/T191972#4123848 (10Dzahn) a:03Dzahn [16:22:54] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-04-03 (1.31.0-wmf.28)), 10Patch-For-Review, 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#4123859 (10zeljkofilipin) [16:23:52] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-04-03 (1.31.0-wmf.28)), 10Patch-For-Review, 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#3716038 (10zeljkofilipin) The last video (User-should-be-able-to-change-p... [16:23:56] paladox: with regards your question about what tests we're planning, I don't really know what the options are...both navtiming and coal are python, both were written without much in the way of what I'd call proper structure... [16:24:03] I'm adding that, but it's not there yet. [16:24:15] ok [16:24:16] So I could use some advice about what the right answer is, at least for the time being. [16:24:44] i think the tox test could do (would need to be done in the repos in question). [16:26:10] marlier if your unsure about it now, and would like jenkins to at least vote on your changes you could use noop [16:26:29] though if you would like the tox test that's for python. [16:27:38] Project mediawiki-core-code-coverage build #3439: 04STILL FAILING in 1 hr 27 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3439/ [16:28:17] (03PS2) 10Imarlier: integration/config: add new perf repos to zuul layout [integration/config] - 10https://gerrit.wikimedia.org/r/425551 (https://phabricator.wikimedia.org/T191994) [16:30:34] (03CR) 10Imarlier: integration/config: add new perf repos to zuul layout (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/425551 (https://phabricator.wikimedia.org/T191994) (owner: 10Imarlier) [16:33:30] (03CR) 10Krinkle: [C: 032] integration/config: add new perf repos to zuul layout [integration/config] - 10https://gerrit.wikimedia.org/r/425551 (https://phabricator.wikimedia.org/T191994) (owner: 10Imarlier) [16:34:34] !log `fab deploy_zuul` for https://gerrit.wikimedia.org/r/425551 [16:34:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:34:44] (03Merged) 10jenkins-bot: integration/config: add new perf repos to zuul layout [integration/config] - 10https://gerrit.wikimedia.org/r/425551 (https://phabricator.wikimedia.org/T191994) (owner: 10Imarlier) [16:36:13] thcipriani: Do you know of a wiki being removed recently on beta? [16:36:33] logstash-beta has a bunch of errors from beta job runners trying to operate on 'sdwiki' which doesn't exist it seems [16:37:07] I haven't seen anything about removing a wiki from beta. [16:37:49] hrm, this is all phab comes up with: https://phabricator.wikimedia.org/T189493 [16:48:27] o/ [16:48:36] is logstash-beta broken? or changed? I can't login! :D [16:49:03] hm, me neither [16:49:44] addshore you need a username and password now [16:49:46] to login there [16:49:53] from where? [16:49:59] on logstash-beta [16:50:00] wait, no, I forgot – with those access credentials it works now :) [16:50:05] ssh deployment-tin.eqiad.wmflabs sudo cat /root/secrets.txt [16:50:20] (the window should tell you that but apparently some user agents hide the message) [16:53:17] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<33.33%) [17:14:04] 10Deployments, 10Release-Engineering-Team, 10Operations, 10Patch-For-Review: Scap sync-file failing for deploy1001.eqiad.wmnet - https://phabricator.wikimedia.org/T191972#4124068 (10Dzahn) deploy1001 has been removed from scap hosts and puppet ran on tin. This should have fixed the immediate scap issue. [17:21:05] 10Deployments, 10Release-Engineering-Team, 10Operations, 10Patch-For-Review: Scap sync-file failing for deploy1001.eqiad.wmnet - https://phabricator.wikimedia.org/T191972#4124089 (10Dzahn) 05Open>03Resolved [17:29:56] Would someone be able to check on the config of a couple of Gerrit repositories for me? performance/coal and performance/navtiming -- I think that they were initialized incorrectly, because I'm getting "refs/heads/master does not exist" when trying to merge changes. [17:43:44] marlier: The first commit will need to be merged without jenkins (see -perf). The repo doesn't have its initial commit yet, so Jenins can't clone the repo. [17:50:29] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10JavaScript, 10MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)), and 2 others: Run Selenium Cucumber tests in CI - https://phabricator.wikimedia.org/T179190#4124266 (10zeljkofilipin) Tests are now running from [[ https://gerrit.wiki... [17:55:02] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10JavaScript, 10MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)), and 2 others: Run Selenium Cucumber tests in CI - https://phabricator.wikimedia.org/T179190#4124291 (10zeljkofilipin) [[ https://gerrit.wikimedia.org/r/#/c/424609/10..... [17:55:17] Okay, cool, I got it worked out. [17:55:41] Krinkle: thanks, didn't understand what you meant originally, but managed to work my way to the right place. [18:02:29] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Host packaged helm charts at https://integration.wikimedia.org/charts - https://phabricator.wikimedia.org/T191821#4124312 (10demon) Do we really want this on integration.wm.o? Or is this only serving them from there for integration... [18:04:36] RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) [18:11:19] (03CR) 10Hashar: "Seems good to me. I have a minor nitpick about a substr in ValidGlobalNameSniff which could be made a little bit more inclusive." (032 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425426 (owner: 10Thiemo Kreuz (WMDE)) [18:15:09] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Host packaged helm charts at https://integration.wikimedia.org/charts - https://phabricator.wikimedia.org/T191821#4117802 (10hashar) That all depends how we are going to deploy services to Kubernetes in the future. If the deployment... [18:37:04] (03CR) 10Chad: [C: 032] "Honesty this list makes less sense than it used to since it gets added to branches as a submodule." [tools/release] - 10https://gerrit.wikimedia.org/r/425455 (https://phabricator.wikimedia.org/T191745) (owner: 10Legoktm) [18:37:36] (03Merged) 10jenkins-bot: Bundle OATHAuth extension [tools/release] - 10https://gerrit.wikimedia.org/r/425455 (https://phabricator.wikimedia.org/T191745) (owner: 10Legoktm) [18:39:19] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Host packaged helm charts at https://integration.wikimedia.org/charts - https://phabricator.wikimedia.org/T191821#4124452 (10demon) Well ideally developers can use the same sets of charts, that's the main reason I thought of it as o... [18:39:21] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:50] 10MediaWiki-Releasing, 10MW-1.31-release, 10Patch-For-Review: Bundle OATHAuth extension with MW 1.31 - https://phabricator.wikimedia.org/T191745#4124459 (10Jdforrester-WMF) 05Open>03Resolved a:03Legoktm [18:39:53] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4124461 (10Jdforrester-WMF) [18:44:04] (03CR) 10Hashar: [C: 032] Make debian-glue voting on operations/debs/tidy-0.99 [integration/config] - 10https://gerrit.wikimedia.org/r/425489 (https://phabricator.wikimedia.org/T191771) (owner: 10Hashar) [18:45:21] (03Merged) 10jenkins-bot: Make debian-glue voting on operations/debs/tidy-0.99 [integration/config] - 10https://gerrit.wikimedia.org/r/425489 (https://phabricator.wikimedia.org/T191771) (owner: 10Hashar) [18:51:13] no_justification how do we setup https://gerrit.wmfusercontent.org ? [18:51:38] dns which points to cobalt/etc [18:51:42] But I'm off today [18:51:44] * no_justification goes away [18:52:33] ok [18:56:37] PROBLEM - SSH on integration-slave-docker-1014 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:49] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle LoginNotify extension with MW 1.31 - https://phabricator.wikimedia.org/T191746#4115206 (10MaxSem) Does the installer support extension dependencies now? [18:59:28] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Host packaged helm charts at https://integration.wikimedia.org/charts - https://phabricator.wikimedia.org/T191821#4124560 (10mobrovac) IMHO, all deployment charts should reside in the same repository / on the same (sub)domain, regar... [19:01:28] RECOVERY - SSH on integration-slave-docker-1014 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [19:09:23] (03CR) 10Thiemo Kreuz (WMDE): "> does that actually saves any time in the processing?" (032 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/425426 (owner: 10Thiemo Kreuz (WMDE)) [19:28:57] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T183968#4124652 (10thcipriani) [19:52:19] (03PS1) 10Hashar: docker: hhvm/php55 quibble images on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/425600 [20:04:04] (03CR) 10Hashar: [C: 04-1] "No I have to try them both against mediawiki/core @master" [integration/config] - 10https://gerrit.wikimedia.org/r/425600 (owner: 10Hashar) [20:12:56] Project mwext-phpunit-coverage-publish build #3250: 04FAILURE in 3.5 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3250/ [20:13:00] Project mwext-phpunit-coverage-publish build #3251: 04STILL FAILING in 3.3 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3251/ [20:14:01] Yippee, build fixed! [20:14:02] Project mwext-phpunit-coverage-publish build #3252: 09FIXED in 58 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3252/ [20:23:19] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4124827 (10awight) [20:26:33] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4124865 (10awight) @akosiaris We're finally ready to deploy the puppet cha... [20:27:07] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#3851398 (10awight) p:05Triage>03High [20:27:10] twentyafterfour: Hi, do you have time to chat about git-lfs today? [20:35:27] awight: sure [20:37:34] twentyafterfour: Is there anything I can do, or is the next step to write a scap patch to not rewrite git-lfs submodule URLs? [20:42:18] FWIW we have a git_upstream_submodules config setting. That would mean that all submodules would be fetched from upstream. [20:42:45] with no url rewriting [20:42:55] * thcipriani missed all context [20:42:56] That might be rough during this transitional phase, since the other submodules are still huge… [20:43:28] gotcha, didn't know if it was just a single submodule or all [20:44:56] is there no way to override via lfsconfig? [20:44:58] thcipriani: If you have a curiosity attack, I documented the latest failure in https://phabricator.wikimedia.org/T180627#4117789 . My current approach is to introduce a tiny git-lfs submodule on a branch of our production repo, so we can iron out the LFS deployment glitches reasonably safely. [20:45:21] I am afflicted by curiosity attacks [20:45:23] I think I did try an lfsconfig thing… lemme look. [20:45:24] :) [20:46:09] Here was a .lfsconfig experiment, https://phabricator.wikimedia.org/T180627#4062932 [20:48:44] that's weird, I didn't realize git itself was aware of the .lfsconfig file maybe something weird inside git review could explain that [20:54:24] awight: thcipriani there is an override in lfs config [20:54:32] but it's annoyingly limited [20:56:45] I think turning on git_upstream_submodules might be the best workaround for now [20:57:24] That might be survivable—we’ve already limited fetch concurrency to 3 threads. [20:57:43] how many targets would you be deploying to? [20:57:48] 3 or 4 nodes? [20:57:50] So, this is a scap.cfg thing I can set myself right? [20:57:57] awight: yes it is [20:58:03] 18 nodes in production, 1 in beta [20:58:35] 18 nodes fetching huge files _might_ give gerrit a hard time, I'm not sure. I assume the lfs endpoint should be relatively efficient and bandwidth will likely be the bottlenexk [20:58:39] bottleneck [20:59:42] awight: what we never quite explained was why we got password prompts from git-lfs before... [21:00:10] if we don't run into that problem again then it should be workable with git_upstream_submodules [21:00:17] twentyafterfour: I think akosiaris limited concurrency because he was getting timeouts from gerrit. [21:00:21] err, from tin. [21:00:41] And maybe we could extend git_upstream_submodules to accept a list instead of a boolean [21:00:55] so you could mix and match [21:01:00] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Replace Text extension with MW 1.31 - https://phabricator.wikimedia.org/T191741#4124997 (10CCicalese_WMF) [21:01:31] twentyafterfour: is it possible for new Phabricator diffusion URI to be No I/O by default? It is annoying to change the behaviour of 6 URIS so I can set one URI to mirror (I have to do that because Diffusion errors when adding a mirror URI when others have I/O) [21:02:10] Hauskatze: The easiest way to deal with that would probably be a script via conduit [21:02:20] thcipriani: fyi, trying the workaround: https://gerrit.wikimedia.org/r/425710 [21:02:20] you can automate the process of setting up all the uris\ [21:02:28] script, conduit == chinese to me :) [21:07:35] Project mwext-phpunit-coverage-publish build #3258: 04FAILURE in 3.4 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3258/ [21:07:37] Project mwext-phpunit-coverage-publish build #3259: 04STILL FAILING in 2.6 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3259/ [21:08:47] Yippee, build fixed! [21:08:47] Project mwext-phpunit-coverage-publish build #3260: 09FIXED in 1 min 7 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3260/ [21:36:26] 10Release-Engineering-Team (Kanban), 10Scap, 10Scoring-platform-team, 10Patch-For-Review: [Blocked] Support git-lfs - https://phabricator.wikimedia.org/T180627#4125077 (10awight) Errors thrown by my experiment with `git_upstream_submodules: True`: ``` scap deploy-log -v -f scap/log/scap-sync-2018-04-11-000... [21:37:09] thcipriani: ^ errors when using the git_upstream_submodules config. Annoyingly, it errors on non-LFS submodules. [21:37:42] hrm, phab times out? [21:37:52] awight: how big is that submodule? [21:38:15] if it's big then it's only going to work over ssh. Phabricator can't do > 2gb https transactions [21:38:25] twentyafterfour: 93M .git/modules/submodules/draftquality/ [21:38:51] hmm that shouldn't be a problem [21:39:36] requires post_max_size to be set at 2gb or 3gb [21:39:44] currently set at 10mb [21:39:52] twentyafterfour ^^ [21:40:41] paladox: post_max_size only affects uploads not downloads [21:40:48] oh [21:41:04] currently cloning https://phabricator.wikimedia.org/source/draftquality.git locally... it's just sitting there taking forever to do (apparently) nothing [21:41:16] oh it finally worked [21:41:24] https://stackoverflow.com/questions/6527811/how-to-download-large-files-through-php-script [21:42:54] paladox: there is a more fundamental limitation in phabricator that prevents really huge transactions...but I don't think this is the problem since it worked for me just now [21:43:03] oh [21:44:58] awight: I'm not quite sure what could be causing the timeout. It does seem to be running somewhat slow at the moment. Maybe we can use the gerrit mirrors for now instead of the phabricator https endpoint? [21:45:56] Good idea, k I’ll point the other submodules there. or… heck, I can just remove them on this branch. [21:47:15] the problem might be that we need to use the https proxy in deployment-prep? I know there are some restrictions there but I thouhgt we already had git https proxy for phabricator configured correctly everywhere [21:49:42] 10Beta-Cluster-Infrastructure, 10Scap: Enable `scap log` on deployment-* servers - https://phabricator.wikimedia.org/T192032#4125106 (10MarcoAurelio) [21:50:34] twentyafterfour: So you’re suggesting that I try a submodule URL like https://gerrit.wikimedia.org/r/research/ores/wheels, right? [21:51:20] yep [21:51:25] :) [21:52:27] awight: yes [21:52:37] urgh, there is no such URL for https://phabricator.wikimedia.org/source/ores/manage/uris/ [21:53:01] I’ll… remove the submodule [21:53:13] awight, I'm not sure it should be there. [21:53:29] Don't you want source/ores-wheels? [21:53:33] it should have a mirror just for redundancy [21:53:42] Or maybe it's source/ores-deploy-wheels [21:53:44] if there isn't one in gerrit I can create it I think [21:53:54] https://phabricator.wikimedia.org/source/ores-deploy-wheels/ [21:54:21] halfak: I’m currently just savaging the repo in order to eliminate complexity in our experiment... [21:54:26] https://phabricator.wikimedia.org/source/ores-deploy-wheels/manage/uris/ [21:54:43] has a uri for gerrit: https://gerrit.wikimedia.org/r/research/ores/wheels [21:55:11] I was looking at another repo, but I’ve removed it now so no worries [21:56:57] 10Beta-Cluster-Infrastructure, 10Scap: Enable `scap log` on deployment-* servers - https://phabricator.wikimedia.org/T192032#4125128 (10MarcoAurelio) [22:01:23] twentyafterfour: \o/ That worked. [22:01:39] Feel free to poke at it on deployment-ores01 if you want. [22:02:09] The LFS file in my submodule was even hydrated. [22:02:29] I’ll squash and amend this work to make a single patch, and will add back our other submodules. [22:07:08] twentyafterfour: I think I do need help, if you can create the gerrit mirror for https://phabricator.wikimedia.org/source/ores/manage/uris/ or suggest how I should construct the gerrit URL if a mirror exists already? [22:08:32] twentyafterfour: actually, all of these github-hosted repos have the same problem [22:18:55] twentyafterfour: I’m seeing the same thing you were running into on deployment-ores01, none of the phabricator URLs can be cloned. [22:19:24] (03PS2) 10Hashar: docker: hhvm/php55 quibble images on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/425600 [22:20:00] I can clone from github, but that won’t port to production. [22:20:52] awight check if the enviromental variable HTTP_PROXY and HTTPS_PROXY are defined [22:20:57] they cannot be set in labs [22:21:13] echo $HTTP_PROXY [22:21:17] echo $HTTPS_PROXY [22:21:20] oh, you are active over here :) [22:21:27] heh [22:21:29] “env | grep PROX” gives me empty string [22:21:51] (same with echo) [22:22:00] oh [22:32:55] (03PS3) 10Hashar: docker: hhvm/php55 quibble images on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/425600 [22:34:20] twentyafterfour: How do you suggest I proceed? Is it even possible to mirror github -> phabricator -> gerrit? Would it make more sense to wait for scap support for mixed rewriting of submodules? [22:35:08] awight: yeah we can mirror like that I think [22:35:39] the problem on deployment_prep is probably the http proxy thing like paladox mentioned [22:35:50] yep [22:35:56] that carn't be set in labs [22:36:02] as it carn't reach prods internals [22:36:03] can't be set? [22:36:08] hmm [22:36:12] twentyafterfour the proxy variable [22:36:18] unless we have one setup for labs? [22:38:09] on prod we have this for the system-level gitconfig [22:38:11] https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/phabricator/templates/vcs/system.gitconfig.erb;9ddf42a8267af3f2a8d1dcf4962bbcc4e9a992dd [22:39:58] awight: should the gerrit mirror be https://gerrit.wikimedia.org/r/#/admin/projects/scoring/ores ? [22:41:57] twentyafterfour: sorry, not sure I understand the question. I have a choice over the path we use? [22:42:26] awight: well there are already a couple of ores repos and I don't know which one is correct or current [22:42:35] hehe tell me about it. [22:42:36] there is /research/ores and /scoring/ores [22:42:42] None of them. [22:43:02] The authoritative repo is https://github.com/wiki-ai/ores [22:43:04] I mean, I will set up mirroring to whichever is most appropriate path to use [22:43:23] sadly, I think the ideal path is /scoring/ores/ores [22:43:23] right, I was working on mirroring from phab to gerrit though it's not ideal it works [22:43:31] since it already pulls from github [22:45:09] should it be https://github/wiki-ai/ores -> https://phabricator.wikimedia.org/source/ores -> https://gerrit.wikimedia.org/r/p/research/ores/ ? [22:45:20] ok scoring/ores/ores it is [22:45:23] :) [22:45:46] ty, yeah the “research” path is something deprecated that we should remove one day [22:46:18] when I read ores, in my head i hear 'oars' [22:46:39] oars, oars, everywhere are oars [22:47:37] https://www.yelp.com/biz_photos/the-oar-block-island?select=Q-141WWeVmaM73g1cNK40Q [22:47:53] hahah yes that exactly [22:47:55] yes! it should be pronounced “oars” [22:48:15] you beat me to it I was busy imguring it: https://i.imgur.com/JH8yag5.jpg [22:48:50] The pronounciation actually became more urgent once we deployed JADE, cos anyone walking around saying “oh-rez” could easily fall into the habit of saying “jah-day” for JADE [22:49:00] the perils of minimalism... [22:49:47] hahah I like the visual image of oars everywhere. I'm tempted to make you a logo but not until git-lfs works ;P [22:50:49] that might be helpful! [22:50:59] Down with the mining metaphor, anyway :) [22:51:49] oh-rez [22:52:07] definitely that one for me :p [22:52:40] ARGH, tha pain ;-) [23:00:38] awight: this nice game comes to mind https://en.wikipedia.org/wiki/Rez_(video_game) [23:00:49] "musical rail shooter" [23:03:09] Now that you mention it, we could also be referring to reservations [23:11:20] hmm I'm having trouble pushing to gerrit but I can't see the whole error message, thanks phabricator for such useful log truncating, I love it when you do that [23:12:46] lol. Thank you for fighting the good fight. [23:15:25] hmm at least it's mirrored some of the refs, maybe gerrit just doesn't like all the /refs/pull/* from github [23:15:33] https://gerrit.wikimedia.org/r/plugins/gitiles/scoring/ores/ores/ [23:16:15] awight: does that look like it's up to date? [23:16:38] yes, the master branch is right [23:16:43] cool [23:16:57] well then I'm gonna assume it's just the pull requests that gerrit doesn't like [23:17:16] and we can safely ignore the error if that's the case [23:23:13] Fine by me, but it sounds like a self-flagellating cronjob [23:24:21] twentyafterfour gerrit should be able to support the pull requests [23:24:33] as there's a plugin that pulls pull requests from github [23:24:37] but also there's gerrithub [23:24:47] gerrit code review on github [23:25:00] /o\ [23:25:00] https://gerrithub.io/ [23:25:34] it's free and also is maintained by a person called luca who is also the maintainer of gerrit as of last year or ealier this year [23:25:52] paladox: we just want a basic git mirror right now [23:26:04] twentyafterfour yep [23:26:39] i think gerrit should support mirroing better like a ui solution. [23:38:17] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<44.44%)