[00:39:33] PROBLEM - Host integration-slave-docker-1058 is DOWN: CRITICAL - Host Unreachable (172.16.6.2) [00:41:13] PROBLEM - Host integration-slave-docker-1059 is DOWN: CRITICAL - Host Unreachable (172.16.6.20) [02:00:17] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%) [03:01:42] Project mediawiki-core-code-coverage-docker build #4291: 04FAILURE in 1 min 40 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/4291/ [04:20:11] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) >>! In T224857#5231820, @thcipriani wrote: > What affect have the `opcode_invalidate` calls for specific files via sy... [05:27:23] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10greg) >>! In T224857#5232237, @Joe wrote: > Just to be clear, I don't think the following things we do today are advisable... [05:44:37] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), 10Patch-For-Review, 10User-zeljkofilipin: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11 - https://phabricator.wikimedia.org/T210506 (10hashar) antoine-approve W... [06:18:13] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10mmodell) >>! In T224857#5229836, @Joe wrote: >>>! In T224857#5229810, @ArielGlenn wrote: > >> Before we start redefining... [06:21:46] (03PS1) 10Pwirth: Skip mediawiki-quibble-vendor-mysql-hhvm-docker for ExtendedSearch and Social [integration/config] - 10https://gerrit.wikimedia.org/r/514217 [06:25:27] (03PS2) 10Pwirth: Skip mediawiki-quibble-vendor-mysql-hhvm-docker for ExtendedSearch and Social [integration/config] - 10https://gerrit.wikimedia.org/r/514217 [06:29:35] (03PS3) 10Pwirth: Skip mediawiki-quibble-vendor-mysql-hhvm-docker for ExtendedSearch and Social [integration/config] - 10https://gerrit.wikimedia.org/r/514217 [06:39:03] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10jijiki) Even though I do agree with most of the things everyone has addressed here, our current problem remains the same:... [06:39:58] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) >>! In T224857#5232382, @mmodell wrote: >>>! In T224857#5229836, @Joe wrote: >>>>! In T224857#5229810, @ArielGlenn wr... [06:42:21] (03PS4) 10Pwirth: Skip mediawiki-quibble-vendor-mysql-hhvm-docker for ExtendedSearch and Social [integration/config] - 10https://gerrit.wikimedia.org/r/514217 [06:45:18] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [06:52:27] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-WikibaseRepository, and 3 others: Wikibase and WikibaseLexeme Selenium tests fail when UseCirrus is true in WikibaseCirrusSearch - https://phabricator.wikimedia.org/T224125 (10Smalyshev) p:05Triage→03Normal [06:57:51] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10greg) We're tangenting. :) [08:13:55] (03PS1) 10Urbanecm: Whitelist Ammarpad [integration/config] - 10https://gerrit.wikimedia.org/r/514233 [09:00:48] RECOVERY - Host integration-slave-docker-1058 is UP: PING OK - Packet loss = 0%, RTA = 1.48 ms [09:01:12] RECOVERY - Host integration-slave-docker-1059 is UP: PING OK - Packet loss = 0%, RTA = 2.59 ms [09:05:42] PROBLEM - Puppet staleness on integration-slave-docker-1058 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [43200.0] [09:06:55] PROBLEM - Puppet staleness on integration-slave-docker-1059 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [43200.0] [09:10:40] RECOVERY - Puppet staleness on integration-slave-docker-1058 is OK: OK: Less than 1.00% above the threshold [3600.0] [09:11:57] RECOVERY - Puppet staleness on integration-slave-docker-1059 is OK: OK: Less than 1.00% above the threshold [3600.0] [09:17:49] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are twice slower than new ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10hashar) So on `cloudvirt1012` the instances are shown in cpuinfo: `Intel Core Processor (Haswell, no TSX)`. Th... [09:26:52] 10Continuous-Integration-Config, 10Wikidata, 10wikiba.se website, 10Patch-For-Review: Move wikiba.se tests from php56 to php72 - https://phabricator.wikimedia.org/T224905 (10WMDE-leszek) Hi @hashar, indeed the intention has been to run on PHP 5.6 back then. Now we'd probably wish to go up to PHP 7.2 or so.... [09:33:21] 10Continuous-Integration-Config, 10Wikidata, 10wikiba.se website, 10Patch-For-Review: Move wikiba.se tests from php56 to php72 - https://phabricator.wikimedia.org/T224905 (10hashar) Regardless of who is hosting the website, I just would like to make sure that CI is aligned with whatever PHP version is bein... [09:36:11] 10Continuous-Integration-Config, 10Wikidata, 10wikiba.se website, 10Patch-For-Review: Move wikiba.se tests from php56 to php72 - https://phabricator.wikimedia.org/T224905 (10WMDE-leszek) Understood. Currently on the host 5.6 is used, but this is of course not optimal, so I'd like to have it bumped up to 7... [09:36:53] 10Gerrit: Gerrit upload-pack send ALL references causing massive network I/O on common operations - https://phabricator.wikimedia.org/T103990 (10Paladox) I’m wrong, it was removed from gerrit, but being re added inhttps://gerrit-review.googlesource.com/c/gerrit/+/226754 [09:42:13] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are twice slower than new ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10hashar) cloudvirt1012 is a HP ProLiant DL360 Gen9 which is referred on Wikitech install guide https://wikitech.... [09:47:14] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Quibble, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), and 2 others: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests - https://phabricator.wikimedia.org/T199116 (10awight) According t... [09:48:03] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Quibble, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), and 2 others: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests - https://phabricator.wikimedia.org/T199116 (10awight) [09:55:20] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] [WiP] Add FileImporter to gated extensions [integration/config] - 10https://gerrit.wikimedia.org/r/514065 (https://phabricator.wikimedia.org/T222802) (owner: 10Jkroll) [09:56:26] 10Scap: make-wmf-branch fails for /mediawiki/extensions/JADE - https://phabricator.wikimedia.org/T224972 (10zeljkofilipin) [09:56:44] 10Scap: make-wmf-branch fails for /mediawiki/extensions/JADE - https://phabricator.wikimedia.org/T224972 (10zeljkofilipin) p:05Triage→03Unbreak! [09:56:53] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10zeljkofilipin) [09:56:55] 10Scap: make-wmf-branch fails for /mediawiki/extensions/JADE - https://phabricator.wikimedia.org/T224972 (10zeljkofilipin) [09:58:01] 10Scap: make-wmf-branch fails for /mediawiki/extensions/JADE - https://phabricator.wikimedia.org/T224972 (10zeljkofilipin) [09:59:35] (03CR) 10Awight: "Comparing other patches that add gated extensions, this might be all that's needed!" [integration/config] - 10https://gerrit.wikimedia.org/r/514065 (https://phabricator.wikimedia.org/T222802) (owner: 10Jkroll) [10:01:24] 10Scap: make-wmf-branch fails for /mediawiki/extensions/JADE - https://phabricator.wikimedia.org/T224972 (10zeljkofilipin) Caused by one or more of the following tasks? - {T212182} - {T211046} - {T221437} [10:07:49] (03CR) 10Zfilipin: "Well, branch cut exploded _without_ this: T224972" [tools/release] - 10https://gerrit.wikimedia.org/r/508643 (https://phabricator.wikimedia.org/T212182) (owner: 10Ladsgroup) [10:09:21] 10Scap: make-wmf-branch fails for /mediawiki/extensions/JADE - https://phabricator.wikimedia.org/T224972 (10zeljkofilipin) Could it be fixed by @Ladsgroup's [[ https://gerrit.wikimedia.org/r/c/mediawiki/tools/release/+/508643 | 508643 ]]? [10:32:18] (03PS3) 10Hashar: wikiba.se, fr-civicrm: Add php72 tests alongside php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514053 (https://phabricator.wikimedia.org/T223348) (owner: 10Jforrester) [10:35:55] (03PS4) 10Hashar: wikiba.se, fr-civicrm: Add php72 tests alongside php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514053 (https://phabricator.wikimedia.org/T223348) (owner: 10Jforrester) [10:36:54] awight: hashar: James_F: Is something on-going right now that would explain CI failing for wmf.7 for npm stufff? [10:37:00] > gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable. [10:37:11] https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/514249/ etc. [10:38:50] (03Restored) 10Hashar: wikiba.se: Drop php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514054 (owner: 10Jforrester) [10:38:58] (03PS2) 10Hashar: wikiba.se: switch to php7.2 and drop php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514054 (owner: 10Jforrester) [10:39:39] (03CR) 10Hashar: [C: 04-1] "Restored and made to switch from 5.6 to 7.2 once the repository is ready. So blocked on T224905 :]" [integration/config] - 10https://gerrit.wikimedia.org/r/514054 (owner: 10Jforrester) [10:40:01] (03CR) 10Hashar: [C: 03+2] wikiba.se, fr-civicrm: Add php72 tests alongside php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514053 (https://phabricator.wikimedia.org/T223348) (owner: 10Jforrester) [10:40:40] (03Abandoned) 10Hashar: Wikiba.se: Drop php56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514056 (https://phabricator.wikimedia.org/T224905) (owner: 10Jforrester) [10:41:11] Krinkle: That's a surprising error. My naive guess is that it's related to the node 10 / npm 6 upgrade, which probably jiggles a bunch of node dependencies. [10:41:35] Yes it's coming from an attempt to install fibers [10:41:37] on node 10 [10:41:54] maybe some or all of the related commits that were merged in master need to be backported [10:42:22] From my limited understanding, rolling back might not be a good option at this point. Maybe we really do need to add python to the quibble image... [10:42:44] python has always been there afaik, node-gyp doesn't work without it. [10:42:46] it was already part of it [10:42:53] (03Merged) 10jenkins-bot: wikiba.se, fr-civicrm: Add php72 tests alongside php56 [integration/config] - 10https://gerrit.wikimedia.org/r/514053 (https://phabricator.wikimedia.org/T223348) (owner: 10Jforrester) [10:42:54] if it was removed today, that would explain the issue, thoguh. [10:43:06] Afaik it's just a side-effect of a incompatible fibers versions [10:43:12] Ah you're on a branch, yeah your backport suggestion seems like a good lead to follow. [10:43:13] not about actually python being missing. [10:43:31] Which patch that was merged today is needed to make things work? [10:43:57] I know there was a breaking change re. mocha, let me find an example patch [10:44:09] https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/514085/ [10:44:12] Krinkle: ^ [10:44:18] Will need porting in active wmf branches and all active REL branches. [10:44:21] 10Continuous-Integration-Config, 10Wikidata, 10wikiba.se website, 10Patch-For-Review: Move wikiba.se tests from php56 to php72 - https://phabricator.wikimedia.org/T224905 (10hashar) >>! In T224905#5232754, @WMDE-leszek wrote: > Understood. Currently on the host 5.6 is used, but this is of course not optima... [10:44:36] unless the CI image is configured such that the "new" stuff is only used for master and future wmf/REL branches [10:45:06] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10serviceops, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10mobrovac) >>! In T220401#5230212, @akosiaris wrote: > > [...] In fact, some numbers I 've heard (I have no actual pro... [10:45:37] awight: OK, I'm trying that one on wmf.7 right now. [10:45:41] Thank you! [10:46:05] SWAT starts in 15 minutes. If it doesn't work by then, please take over to make it work, revert, or otherwise delegate as needed. I'll have to leave by then. [10:46:27] Krinkle: sure thing. [10:46:52] Is this the branch you're trying to SWAT today? [10:48:08] * awight only sees config patches for deployment [10:48:12] wmf.7 is the current active production branch, it's tests are failing due to this. [10:48:24] Krinkle: yeah [10:48:25] Yeah, it's a regular deploy, not swat. Because stuff is broken. [10:48:30] yesterday we have switched to nodejs 10 [10:48:38] but wmf.7 still brings fibers 2.x [10:48:44] +1 gotcha. Is there a task yet or should I file one? [10:49:18] might be worth documenting on "the task" about this migration. [10:49:47] I will cherry pick stuff [10:49:59] hashar: Presumably REL branches as well. That needs to be fixed soon as well, and ideally planned ahead of time so that it isn't broken until the next person finds out :) [10:50:12] (03CR) 10Lucas Werkmeister (WMDE): [C: 03+1] "Well, if trying to cut a branch on JADE fails (T224972 – you can’t push new branches to it), then I guess keeping the old name doesn’t mak" [tools/release] - 10https://gerrit.wikimedia.org/r/508643 (https://phabricator.wikimedia.org/T212182) (owner: 10Ladsgroup) [10:50:16] But looks wmf.7 is passing now, so I'm unblocked now. [10:50:17] Thanks [10:51:18] hashar: to support your ci/yaml ideas, if we had those, then older branches would never break because they would retain the old docker images and job config the same as before, so that new stuff naturally spreads only to newer branches unless intentionally backported. That would be very nice. [10:51:47] Now we have the burdon of having the new quibble and needing it to be compatible with older branches and older tests, which is a lot of work I imagine. But we might get lucky this time if it's easy to migrate / backport. [10:53:56] awight: Krinkle https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/514257 build: upgrade wdio-mocha-framework for NodeJS 10 [10:53:58] that should do it [10:54:25] Krinkle: yeah that is partly why I want to have a zuul pipeline per branch [10:54:45] that would also simplify the configuration as you have found recently with the skip-if mess :\ [10:58:16] Yeah. Might as well have the docker image+version+command specified per-repo and naturally excluded/skipped as needed within the repo. [10:58:47] We could even do that now with a few hacks, if we're willing to let repo maintainers decide which image to execute (limited to wikimedia.org/releng:*) [10:58:57] only downside will be upgrades [10:59:16] but also upsides, it means we can't break them unless they pass CI. [10:59:50] and naturally will find them, and keep old version until it works, especially for less maintainers repos will be less stress, like with phpcs/eslint/etc. already [11:02:19] (03CR) 10Hashar: [C: 03+1] "And you also need to press the [Submit] button :]" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/510619 (https://phabricator.wikimedia.org/T223328) (owner: 10Brennen Bearnes) [11:03:06] 10Release-Engineering-Team (Kanban), 10dev-images, 10local-charts, 10Patch-For-Review: Create fabfile.py for dev-images - https://phabricator.wikimedia.org/T223328 (10hashar) 05Open→03Resolved [11:03:58] I like the sounds of it. That's also consistent with how Travis CI lets each repo configure its own base OS image, etc. [11:06:01] Krinkle: awight yes that is the spec for the next CI system we will use [11:20:46] what is happening with the branch cut today, and (hopefully) subsequent deploy, given the node change etc? [11:30:51] !log Upgrading Jenkins BlueOcean and all its dependencies [11:30:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:31:09] !log Upgrading Jenkins Warnings Next Generation Plugin # T224745 [11:31:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:32:44] !log Upgrading Jenkins Pipeline plugins [11:32:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:37:43] hashar: getting failures on a job that runs selenium tests against beta commons ... I don't know much about docker, zeljko said to ask you [11:37:47] see https://phabricator.wikimedia.org/T224978 [11:43:24] 10Continuous-Integration-Config, 10Multimedia, 10WikibaseMediaInfo: WikibaseMediaInfo selenium tests failing when run against beta commons - https://phabricator.wikimedia.org/T224978 (10hashar) Indeed the job uses the container `docker-registry.wikimedia.org/releng/npm-browser-test:0.4.1`. We have switched t... [11:43:55] 10Continuous-Integration-Config, 10Multimedia, 10WikibaseMediaInfo: WikibaseMediaInfo selenium tests failing when run against beta commons - https://phabricator.wikimedia.org/T224978 (10hashar) [11:43:57] cormacparle__: indeed it is broken! [11:43:58] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10JavaScript, 10Patch-For-Review: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10hashar) [11:44:15] ah yeah https://phabricator.wikimedia.org/T217545 [11:44:32] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10Patch-For-Review: Update selenium-daily-beta-* jobs to node10/npm6 - https://phabricator.wikimedia.org/T217545 (10hashar) [11:44:35] 10Continuous-Integration-Config, 10Multimedia, 10WikibaseMediaInfo: WikibaseMediaInfo selenium tests failing when run against beta commons - https://phabricator.wikimedia.org/T224978 (10hashar) [11:45:01] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10Patch-For-Review: Update selenium-daily-beta-* jobs to node10/npm6 - https://phabricator.wikimedia.org/T217545 (10hashar) That causes the WikibaseMediaInfo selenium daily job to fail T224978 [11:45:09] cormacparle__: I will look at upgrading those jobs [11:45:58] 10Release-Engineering-Team, 10Scap: Compare /common in mediawiki-staging and /usr directories at the start of scap - https://phabricator.wikimedia.org/T224980 (10aaron) [11:46:37] (03CR) 10Reedy: [C: 03+2] Rename JADE to Jade [tools/release] - 10https://gerrit.wikimedia.org/r/508643 (https://phabricator.wikimedia.org/T212182) (owner: 10Ladsgroup) [11:46:59] Thanks hashar [11:47:24] (03Merged) 10jenkins-bot: Rename JADE to Jade [tools/release] - 10https://gerrit.wikimedia.org/r/508643 (https://phabricator.wikimedia.org/T212182) (owner: 10Ladsgroup) [11:47:36] which brings me back to some very old ci change [11:48:04] 10Scap: make-wmf-branch fails for /mediawiki/extensions/JADE - https://phabricator.wikimedia.org/T224972 (10Reedy) >>! In T224972#5232822, @zeljkofilipin wrote: > Could it be fixed by @Ladsgroup's [[ https://gerrit.wikimedia.org/r/c/mediawiki/tools/release/+/508643 | 508643 ]]? Yup, also using JADE means "maste... [11:50:13] (03PS3) 10Hashar: docker: sync node10-test-browser with npm-browser-test [integration/config] - 10https://gerrit.wikimedia.org/r/494364 (https://phabricator.wikimedia.org/T217545) [12:10:23] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10Reedy) [12:10:25] 10Scap, 10User-Ladsgroup: make-wmf-branch fails for /mediawiki/extensions/JADE - https://phabricator.wikimedia.org/T224972 (10Reedy) 05Open→03Resolved a:03Ladsgroup [12:18:57] Is mediawiki-phpunit-coverage-patch-docker being broken known? [12:19:06] (03PS4) 10Hashar: docker: sync node10-test-browser with npm-browser-test [integration/config] - 10https://gerrit.wikimedia.org/r/494364 (https://phabricator.wikimedia.org/T217545) [12:19:28] (03CR) 10Hashar: "PS2 got deleted (spam)." [integration/config] - 10https://gerrit.wikimedia.org/r/494364 (https://phabricator.wikimedia.org/T217545) (owner: 10Hashar) [12:20:43] Reedy: if it is known, there is a task in phabricator. else one gotta be filled ! :] [12:21:15] Looks like more node fallout [12:21:24] ahh [12:21:45] if it does npm install, yeah surely [12:22:06] we haven't migrated all jobs [12:22:17] 10Continuous-Integration-Config: mediawiki-phpunit-coverage-patch-docker fails to install fibers@3.1.1 - https://phabricator.wikimedia.org/T224983 (10Reedy) [12:23:22] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10JavaScript, 10Patch-For-Review: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10hashar) [12:23:26] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10JavaScript, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), 10Patch-For-Review: Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 (10hashar) 05Resolved→03Open There are bunch of other quibb... [12:23:28] Reedy: thanks :) [12:23:31] 10Continuous-Integration-Config: mediawiki-phpunit-coverage-patch-docker fails to install fibers@3.1.1 - https://phabricator.wikimedia.org/T224983 (10hashar) [12:23:35] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10JavaScript, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), 10Patch-For-Review: Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 (10hashar) [12:47:05] (03CR) 10Hashar: [C: 03+2] "ENTRYPOINT kept as is (npm test)" [integration/config] - 10https://gerrit.wikimedia.org/r/494364 (https://phabricator.wikimedia.org/T217545) (owner: 10Hashar) [12:48:37] (03Merged) 10jenkins-bot: docker: sync node10-test-browser with npm-browser-test [integration/config] - 10https://gerrit.wikimedia.org/r/494364 (https://phabricator.wikimedia.org/T217545) (owner: 10Hashar) [12:55:24] 10Continuous-Integration-Infrastructure: fibers@3.1.1 installation failures - https://phabricator.wikimedia.org/T224986 (10zeljkofilipin) [12:55:31] 10Continuous-Integration-Infrastructure: fibers@3.1.1 installation failures - https://phabricator.wikimedia.org/T224986 (10zeljkofilipin) p:05Triage→03High [12:56:10] 10Continuous-Integration-Infrastructure: fibers@3.1.1 installation failures - https://phabricator.wikimedia.org/T224986 (10zeljkofilipin) [12:56:17] 10Continuous-Integration-Infrastructure: fibers@3.1.1 installation failures - https://phabricator.wikimedia.org/T224986 (10hashar) 05Open→03Invalid They are tracked as sub tasks of T211784 [12:57:19] zeljkof: that is https://phabricator.wikimedia.org/T217545 ;D [12:57:54] hashar: ah, I've created a duplicate? [12:58:23] anyway, I'll document failing jobs, so I can check later, you can resolve it [12:58:33] zeljkof: no need [12:58:35] they are all broken [12:58:52] ah, cool [12:58:53] anyway james got patches [12:58:58] train then [12:58:58] and I am going to update the jobs [13:01:24] (03PS1) 10Hashar: Update selenium-daily defaults [integration/config] - 10https://gerrit.wikimedia.org/r/514281 (https://phabricator.wikimedia.org/T217545) [13:01:51] !log Building docker-registry.discovery.wmnet/releng/node10-test-browser:0.6.0 # T217545 [13:01:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:01:54] T217545: Update selenium-daily-beta-* jobs to node10/npm6 - https://phabricator.wikimedia.org/T217545 [13:02:01] (03CR) 10Hashar: [C: 03+2] Update selenium-daily defaults [integration/config] - 10https://gerrit.wikimedia.org/r/514281 (https://phabricator.wikimedia.org/T217545) (owner: 10Hashar) [13:04:26] (03Merged) 10jenkins-bot: Update selenium-daily defaults [integration/config] - 10https://gerrit.wikimedia.org/r/514281 (https://phabricator.wikimedia.org/T217545) (owner: 10Hashar) [13:06:12] (03PS1) 10Hashar: selenium-daily-beta-MediaWiki to NodeJS 10 [integration/config] - 10https://gerrit.wikimedia.org/r/514282 (https://phabricator.wikimedia.org/T217545) [13:11:32] (03PS1) 10Hashar: Revert "Update selenium-daily defaults" [integration/config] - 10https://gerrit.wikimedia.org/r/514285 [13:11:38] (03CR) 10Hashar: [C: 03+2] Revert "Update selenium-daily defaults" [integration/config] - 10https://gerrit.wikimedia.org/r/514285 (owner: 10Hashar) [13:11:51] (03PS1) 10Reedy: Swap a / for a | in parameter type hints [tools/release] - 10https://gerrit.wikimedia.org/r/514286 [13:13:59] (03Merged) 10jenkins-bot: Revert "Update selenium-daily defaults" [integration/config] - 10https://gerrit.wikimedia.org/r/514285 (owner: 10Hashar) [13:15:07] (03PS2) 10Hashar: selenium-daily-beta-* jobs to NodeJS 10 [integration/config] - 10https://gerrit.wikimedia.org/r/514282 (https://phabricator.wikimedia.org/T217545) [13:18:29] (03CR) 10Hashar: [C: 03+2] selenium-daily-beta-* jobs to NodeJS 10 [integration/config] - 10https://gerrit.wikimedia.org/r/514282 (https://phabricator.wikimedia.org/T217545) (owner: 10Hashar) [13:19:07] (03CR) 10Reedy: [C: 03+2] Swap a / for a | in parameter type hints [tools/release] - 10https://gerrit.wikimedia.org/r/514286 (owner: 10Reedy) [13:19:43] (03Merged) 10jenkins-bot: Swap a / for a | in parameter type hints [tools/release] - 10https://gerrit.wikimedia.org/r/514286 (owner: 10Reedy) [13:20:51] (03Merged) 10jenkins-bot: selenium-daily-beta-* jobs to NodeJS 10 [integration/config] - 10https://gerrit.wikimedia.org/r/514282 (https://phabricator.wikimedia.org/T217545) (owner: 10Hashar) [13:21:15] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10Patch-For-Review: Update selenium-daily-beta-* jobs to node10/npm6 - https://phabricator.wikimedia.org/T217545 (10hashar) https://integration.wikimedia.org/ci/job/selenium-daily-beta-MediaWiki/292/console does manage to install npm dependencies now... [13:24:02] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10JavaScript, 10Patch-For-Review: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10hashar) [13:24:02] !log Update all selenium-daily* jobs to use NodeJS 10 instead of NodeJS 6. T217545 [13:24:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:24:05] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10Patch-For-Review: Update selenium-daily-beta-* jobs to node10/npm6 - https://phabricator.wikimedia.org/T217545 (10hashar) 05Open→03Resolved a:03hashar INFO:jenkins_jobs.builder:Number of jobs generated: 12 INFO:jenkins_jobs.builder:Reconfigu... [13:24:05] T217545: Update selenium-daily-beta-* jobs to node10/npm6 - https://phabricator.wikimedia.org/T217545 [13:25:19] zeljkof: ElectronPdfService works now https://integration.wikimedia.org/ci/job/selenium-daily-beta-ElectronPdfService/265/console [13:26:01] hashar: cool! [13:44:05] (03PS1) 10Thcipriani: make-wmf-branch: fix alreadyBranched [tools/release] - 10https://gerrit.wikimedia.org/r/514296 [13:49:07] (03CR) 10Zfilipin: [C: 03+2] "Thank you." [tools/release] - 10https://gerrit.wikimedia.org/r/514296 (owner: 10Thcipriani) [13:49:42] (03Merged) 10jenkins-bot: make-wmf-branch: fix alreadyBranched [tools/release] - 10https://gerrit.wikimedia.org/r/514296 (owner: 10Thcipriani) [13:50:50] (03PS1) 10Reedy: Update continue documentation [tools/release] - 10https://gerrit.wikimedia.org/r/514297 [14:23:39] (03PS1) 10Reedy: Some code style consistencies [tools/release] - 10https://gerrit.wikimedia.org/r/514302 [14:32:29] (03CR) 10Reedy: [C: 03+2] Some code style consistencies [tools/release] - 10https://gerrit.wikimedia.org/r/514302 (owner: 10Reedy) [14:33:04] (03Merged) 10jenkins-bot: Some code style consistencies [tools/release] - 10https://gerrit.wikimedia.org/r/514302 (owner: 10Reedy) [14:45:20] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10thcipriani) >>! In T224857#5232389, @jijiki wrote: > If there are not any better short-term solutions/ideas, depooling-dep... [14:51:49] (03CR) 10Jeena Huneidi: "whoopsie, thanks Hashar!" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/510619 (https://phabricator.wikimedia.org/T223328) (owner: 10Brennen Bearnes) [14:54:09] (03PS2) 10Jforrester: jjb: Migrate PHP code coverage jobs to node10 quibble, too [integration/config] - 10https://gerrit.wikimedia.org/r/514189 (https://phabricator.wikimedia.org/T224983) [14:56:59] 10Continuous-Integration-Config, 10Patch-For-Review: mediawiki-phpunit-coverage-patch-docker fails to install fibers@3.1.1 - https://phabricator.wikimedia.org/T224983 (10Jdforrester-WMF) I didn't want to deploy this last night without @hashar around to spot when I'd made things worse. ;-) [14:57:26] 10Diffusion, 10Gerrit, 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, and 2 others: Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10Halfak) [14:57:55] 10Diffusion, 10Gerrit, 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, and 2 others: Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10Halfak) I think that @mmodell helped us with a credential issue that manifested in a similar way back in April. [14:59:03] 10Diffusion, 10Gerrit, 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, and 2 others: Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10Halfak) See also {T203246} [14:59:58] Hey folks, let me know if you want to have a quick chat about this stuff ^ [15:02:32] Project mediawiki-core-code-coverage-docker build #4292: 04STILL FAILING in 2 min 31 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/4292/ [15:05:27] maintenance-disconnect-full-disks build 78353 integration-slave-docker-1021 (/: 95%): OFFLINE due to disk space [15:05:27] maintenance-disconnect-full-disks build 78353 integration-slave-docker-1021 (/: 95%): OFFLINE due to disk space [15:08:28] PROBLEM - Free space - all mounts on integration-slave-docker-1021 is CRITICAL: CRITICAL: integration.integration-slave-docker-1021.diskspace.root.byte_percentfree (<11.11%) [15:09:25] 10Continuous-Integration-Config, 10Patch-For-Review: mediawiki-phpunit-coverage-patch-docker fails to install fibers@3.1.1 - https://phabricator.wikimedia.org/T224983 (10Legoktm) Er, why does the PHPUnit job need npm dependencies installed? [15:09:35] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10JavaScript: Update MobileFrontend-npm-run-lint-modules-docker to run node10 - https://phabricator.wikimedia.org/T224997 (10Jdforrester-WMF) [15:10:36] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10JavaScript: Update MobileFrontend-npm-run-lint-modules-docker to run node10 - https://phabricator.wikimedia.org/T224997 (10Jdforrester-WMF) Or switch it off. [15:15:13] maintenance-disconnect-full-disks build 78355 integration-slave-docker-1021: OFFLINE due to disk space [15:16:49] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10JavaScript: Update MobileFrontend-npm-run-lint-modules-docker to run node10 - https://phabricator.wikimedia.org/T224997 (10Niedzielski) [15:19:41] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10zeljkofilipin) Group0 at 1.34.0-wmf.8. [15:20:07] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10JavaScript: Update MobileFrontend-npm-run-lint-modules-docker to run node10 - https://phabricator.wikimedia.org/T224997 (10Legoktm) Previous discussion: T196956#4277695 (I don't remember if there was a follow-up ticket) If it's valuable... [15:28:11] 10Continuous-Integration-Config, 10Wikidata, 10wikiba.se website, 10Patch-For-Review: Move wikiba.se tests from php56 to php72 - https://phabricator.wikimedia.org/T224905 (10Jdforrester-WMF) Repo is ready: https://gerrit.wikimedia.org/r/c/wikibase/wikiba.se/+/505774 wikiba.se-composer-php56-docker https:/... [15:32:33] (03CR) 10Hashar: "CI wise yes that is all that is needed. As for skipping slow tests or tests that are not relevant, there is no system for it yet :-( Cirru" [integration/config] - 10https://gerrit.wikimedia.org/r/514065 (https://phabricator.wikimedia.org/T222802) (owner: 10Jkroll) [15:34:33] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10serviceops, and 2 others: TEC3:O3:O3.1:Q4 Goal - Move cpjobqueue, Wikidata Termbox SSR (new service), Kask (session storage service) and ORES (partially) through the production CD Pipeline - https://phabricator.wikimedia.org/T220398 (10akosiari... [15:34:40] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10serviceops, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10akosiaris) 05Open→03Resolved a:03akosiaris And LVS done today. ` akosiaris@deploy1001:~$ curl -i https://sess... [15:37:43] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata, 10Patch-For-Review, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞): [Task] Add Scribunto to extension-gate in CI - https://phabricator.wikimedia.org/T125050 (10hashar) Similar is FileImporter which has some slow tests ( h... [15:38:43] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10serviceops, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10Eevans) >>! In T220401#5233899, @akosiaris wrote: > > And LVS done today. > > ` > akosiaris@deploy1001:~$ curl -i h... [15:40:14] maintenance-disconnect-full-disks build 78360 integration-slave-docker-1021: OFFLINE due to disk space [15:41:35] 10Diffusion: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10mmodell) The long pause seems like it should be a revealing clue but I'm not quite sure of what. [15:42:19] James_F: so for mediawiki/core @ REL1_27 https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/332029/ [15:42:34] fails due to tidy [15:42:35] .. [15:42:44] hashar: Oh well. [15:42:49] No one cares [15:42:52] I filed a bug about it [15:42:55] It's EOL this month [15:43:02] This week, if you get a move on. ;-) [15:43:07] which is probably due to the image that got migrated to Stretch which does not have tidy potentially [15:43:10] I'm waiting on other people [15:43:11] 10Diffusion: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10mmodell) [15:43:13] :P [15:43:15] Reedy: can we just EOL mediawiki this week ? [15:43:23] All of mediawiki? [15:43:25] Every version? [15:43:26] hashar: it needs the old tidy :| [15:43:26] sweet [15:43:27] na [15:43:30] i mean 1.27 [15:43:32] or do we really need a final release? [15:43:34] usemod, wasn't that the good version? [15:43:45] hashar: People get upset when we don't do one [15:43:47] hashar: what? of course we need a release... [15:43:55] Especially when patches are merged into the branch but not released [15:44:01] legoktm: hyeah I have eventually forward ported libtidy at some point to help with the tidy issue. Then there it says it can not find it so ... No clue [15:44:07] but yes, hashar, don't waste much time on 1.27 [15:44:15] ok ok :] [15:44:36] Reedy: Maybe don't wait any more? :-) [15:44:36] if we could cut the final release and finally dish out REL1_27 , that unlocks a lot of random things [15:44:42] such as dropping php5.5 at least [15:44:48] James_F: Well, we have to give notice [15:44:56] So, in theory, the earliest would be at least 24H from now [15:45:26] * James_F nods. [15:45:57] a day seems like really short notice... how does a week sound for middle ground? [15:46:12] 10Diffusion, 10Gerrit, 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, and 2 others: Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10Halfak) [15:46:12] we usually just give a day's notice [15:46:39] "Usually" and "no security releases for nearly a year" sit uncomfortably with each other. [15:46:40] Reedy, hashar: we can probably just disable parser tests on that branch, or make them skip if its the wrong tidy version [15:46:42] 1.27 has been slated to be EOL-d for months [15:47:25] Yeah, let's not work further on it. [15:47:30] or, even years [15:47:34] since it's release? or before? [15:47:35] I would say release a final release [15:47:43] hhvm is slow [15:47:43] and state that is the last ever [15:47:50] hashar: That is basically the plan for this week [15:47:54] then wait a few days just in case and dish out everything [15:48:06] +1 on the plan ;]]] [15:48:41] with the usual caveat... if we fsck up the release, we'll make a release to fix the broken release [15:48:58] Yeah. [15:49:00] (like when we put short arrays into REL1_23 which had a looow php support version) [15:49:03] * Reedy coughs loudly [15:49:03] 1.27.4-oops. [15:49:10] * James_F grins at Reedy. [15:49:27] gotta watch out for those ?? [15:49:53] We really need to drop php70 and php71 support from master soon. [15:49:55] yeah we can wait a few days after the final release before deleting the related infra [15:50:09] We're going to really struggle to support php70 in CI for years. :-( [15:50:45] well probably [15:50:49] we should just drop php 7.0 support [15:50:52] docker makes it really easy to support old runtimes [15:51:02] legoktm: Security issues, much? [15:51:10] and whenever we cut a new release for an old branhc, if the target php is EOL we just drop it as the supposedly supported version [15:51:11] php70 has been out of support for six months already. [15:51:32] so that eg right now, we could get get rid of php 7.0, then patch REL1_31 and REL1_32 to stop mentionning 7.0 [15:51:34] s/support/security support/ [15:51:47] hashar: That leads to issues like Reedy said, though. [15:51:49] and instead say they support 7.2 (and might 7.0 with no guarantee) [15:52:04] The problem is, of course, we don't run pre-release CI for security releases ;P [15:52:32] Retrospectively changing environment support for MW releases has happened, but we're reticent to do it. [15:53:55] James_F: what kind of issue? [15:55:05] hashar: "(like when we put short arrays into REL1_23 which had a looow php support version)" [15:55:23] If we'd been running php52 on REL1_23 we'd have noticed before release. [15:55:51] then at the time we did that, we might have bumped the minimul php version requirement [15:56:15] since what ever old php version we claimed to still support was long EOL anyway [15:56:49] MW has a long history of claiming to support security-vulnerable PHP versions. :-( [15:57:56] 10Diffusion: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10mmodell) `ssh -tv vcs@git-ssh.wikimedia.org` ` --- skipped verbose output from ssh session setup. Interesting part below: ---` ` debug1: SSH2_MSG_EXT_INFO received debug... [15:59:14] James_F: then probably we should break that habit :] [15:59:58] hard to make that decision by comittee, needs a BDFL :P [15:59:59] Yes, but that's a TechCom decision, I guess? Lead thy users not into temptation. [16:00:08] Maybe. [16:01:01] MW must run on PHP Master or gtfo? [16:01:26] "MW must run on a version that is supported upstream, or gtfo" [16:02:32] Reedy: Where's the work on PHP8 compatibility for MW? ;-) [16:02:46] srs biznes [16:02:47] (03PS2) 10Reedy: Update continue documentation [tools/release] - 10https://gerrit.wikimedia.org/r/514297 [16:02:57] lol [16:03:31] * hashar drops hhvm / php7.0 [16:03:36] ;D [16:03:48] can't wait to be off of hhvm (seriously) [16:03:48] (03CR) 10Jforrester: [C: 03+2] Update continue documentation [tools/release] - 10https://gerrit.wikimedia.org/r/514297 (owner: 10Reedy) [16:03:54] apergos: Ditto. [16:04:16] i already miss the integrated debugger :( no more setting breakpoints directly on mwdebug1002 :( [16:04:19] My current order of desires is killing php55 php56 hhvm jessie php70 php71. [16:04:23] (03Merged) 10jenkins-bot: Update continue documentation [tools/release] - 10https://gerrit.wikimedia.org/r/514297 (owner: 10Reedy) [16:04:38] hhvm is definitely the worst out of all of those [16:05:13] maintenance-disconnect-full-disks build 78365 integration-slave-docker-1021: OFFLINE due to disk space [16:05:54] Yes, but I've already written the patches for dropping 55 and 56. ;-) [16:06:27] Helping SRE move from HHVM in production is the most value-add thing people can do. Hint hint. ;-) [16:06:37] so just a reminder that this https://phabricator.wikimedia.org/T224857 needs agreement on the way forward in the short term [16:06:40] we still have php55 and 56 tests? [16:06:52] paladox: Sadly. [16:06:57] :( [16:07:05] i thought we got rid of them :P [16:07:11] paladox: In master. [16:07:17] ah [16:07:19] ok [16:07:19] 10Phabricator: Move various acl* projects as subprojects to Policy-Admins - https://phabricator.wikimedia.org/T224818 (10mmodell) @MarcoAurelio: correct, and only projects that control admins that can edit policies should be subprojects. Other access control projects do not belong under policy-admins. We could... [16:07:25] paladox: Which means CI needs to support them until their support timeline runs out. [16:07:26] 10Phabricator: Move various acl* projects as subprojects to Policy-Admins - https://phabricator.wikimedia.org/T224818 (10mmodell) 05Open→03Invalid [16:07:49] Ah i see, aim for the very latest php version in the next LTS :) [16:08:02] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Pipeline image build cleanup - https://phabricator.wikimedia.org/T177867 (10thcipriani) a:05dduvall→03thcipriani [16:10:37] !log Deleting integration-slave-docker-1021 and integration-slave-docker-1049 / too small disk (20G partition) and not enough ram (2G) # T221872 [16:10:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:10:40] T221872: composer-package-php73-docker seems to fail often on Parsoid builds - https://phabricator.wikimedia.org/T221872 [16:11:33] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Legoktm) Is whatever restart/depool solution that was proposed above smart enough to only do so when a *.php file is touch... [16:12:52] hm, has something changed with the nodejs-slim docker image recently?j [16:13:13] node-rdkafka doesn't seem to build properly anymore [16:13:17] https://integration.wikimedia.org/ci/blue/organizations/jenkins/service-pipeline-test-and-publish/detail/service-pipeline-test-and-publish/211/pipeline/ [16:13:21] this happens locally too [16:13:29] PROBLEM - Host integration-slave-docker-1021 is DOWN: CRITICAL - Host Unreachable (172.16.0.78) [16:13:51] integration-slave-docker-1021 < I have deleted it [16:14:04] PROBLEM - Host integration-slave-docker-1049 is DOWN: CRITICAL - Host Unreachable (172.16.5.177) [16:15:02] ottomata: look at a previously working build and try to spot difference? [16:15:20] ottomata: eg node version changed, some npm package that had a new release [16:15:46] surely node-rdkafka got released 3 days ago [16:15:54] that might be the reason [16:16:10] hmm, maybe, but we didn't update... [16:16:16] trying to see if it builds out of docker [16:18:24] ottomata: usually I download the build output of the last good and the first bad builds [16:18:28] then do a vimdiff against those [16:19:00] hm good idea... [16:19:16] 10Continuous-Integration-Config, 10Patch-For-Review: mediawiki-phpunit-coverage-patch-docker fails to install fibers@3.1.1 - https://phabricator.wikimedia.org/T224983 (10Jdforrester-WMF) >>! In T224983#5233809, @Legoktm wrote: > Er, why does the PHPUnit job need npm dependencies installed? Because it's just a... [16:19:24] ottomata: https://integration.wikimedia.org/ci/job/service-pipeline-test-and-publish/211/consoleText [16:19:42] gotta find the last successful build for that repo [16:20:01] yeah how do I do that? [16:20:06] trying the jenkins search.. [16:20:24] look in Gerrit for previous changes made to that repo? [16:20:30] one might have a recent build still available [16:20:38] ah i think i found [16:20:55] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Quibble, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), and 2 others: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests - https://phabricator.wikimedia.org/T199116 (10Jdforrester-WMF) Sh... [16:21:37] hmm [16:21:52] https://integration.wikimedia.org/ci/blue/organizations/jenkins/service-pipeline-test-and-publish/detail/service-pipeline-test-and-publish/145/pipeline [16:21:53] ? [16:21:54] how is that eventgate-ci.git barely has any change anyway? [16:22:10] ? [16:22:17] we mostly just use eventgate-ci for pushing tags [16:22:23] but we push all the changes to master there too [16:23:13] ... [16:25:21] i can repro outside of docker build [16:26:20] ah it is pulling a more recent version of node-rdkafka [16:28:00] hashar: do you know if there is a way to specify a specific dependency version in a blubber.yaml file? [16:28:12] the problem is that we use the wmf debian package for librdkafka [16:28:29] which is i guess not compatible with the version of node-rdkakfa that npm is choosing [16:28:36] yes I could set the version more explicitly in package.json [16:28:46] but, i'd rather not, since the reason for doing so is WMF specific [16:28:58] Pinning versions in package.json is strongly encouraged, generally. [16:29:59] ottomata: explicit pinning in package.json [16:30:20] and if using nodejs10, I guess a package-lock.json [16:33:58] 17:32:59 gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable. [16:34:25] https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/514329/ [16:34:50] ugh, node-gyp still needs python2 to be installed [16:35:32] shall I file a bug? :P [16:38:06] Reedy: well that is broken anyway [16:38:16] Reedy: that change tries to install fibers@2 which does not work with nodejs 10 [16:38:39] which I guess is because REL1_32 now needs to be fixed to use the proper fibers version (poke James_F ) [16:38:53] which would be bumping wdio-framework-mocha to 0.6.4 iirc [16:39:06] I'm guessing 1.31 (an LTS on our side) is going to be similarly broken? [16:39:27] and wmf branches [16:39:50] the basic patch I have send for 1.34.0-wmf.7 https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/514257/ [16:39:55] some folks were blocked [16:40:01] hmm [16:40:04] it is not merged bah [16:41:09] hashar: the thing is, eventgate will work with later versions of node-rdkafka [16:41:11] no problem [16:41:40] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10jijiki) >>! In T224857#5233723, @thcipriani wrote: >>>! In T224857#5232389, @jijiki wrote: >> If there are not any better... [16:41:50] its just that wmf needs a specific version, because we use a backported librdkafka debian package in apt.wm.org to build node-rdkafka [16:42:52] Reedy: so yeah we need to bump wdio-framework-mocha and that should work [16:42:53] and, beacuse the code repo is not WMF specific (it shouldnt' be), i don't want to force specific dependency versions just because we have special build requirements [16:43:34] ottomata: then your code is not compatible and gotta be fixed I guess ;D [16:43:57] code? [16:44:00] no the code is compatible [16:44:03] its wmf that is not compatible :p [16:44:11] if I told the code to build librdkafka itself [16:44:13] sorry [16:44:17] our build pipeline [16:44:23] rather than using the wmf debian librdkafka package [16:44:29] it would build and use the later version fine [16:44:36] yeah that is what I mean [16:44:43] your code is not compatible with the package we ship [16:44:45] ;D [16:45:30] potentially the code could check librdkafka version and early complain when the version is not the one expected [16:45:33] or something like that [16:45:43] anyway. have to head head out for dinner [16:45:52] the build pipeline shoudl do that I think, not the code [16:45:54] the code will work fine [16:46:04] with either node-rdkafka 2.4 or 2.7 [16:46:32] its node-rdkafka 2.7 that doesn't work with librdkafka 0.11.6 [16:48:19] yargh fine... [16:48:23] but i don't like it [16:57:33] (03PS1) 10Jforrester: mwext-MobileFrontend-npm-run-lint-modules-docker: Fix to run on master, wmf, and release branches [integration/config] - 10https://gerrit.wikimedia.org/r/514346 [16:57:35] (03PS1) 10Jforrester: mwext-MobileFrontend-npm-run-lint-modules-docker: Make non-voting for now [integration/config] - 10https://gerrit.wikimedia.org/r/514347 (https://phabricator.wikimedia.org/T224997) [16:57:51] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 53.33% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [16:59:16] Reedy: Yes, was going to fix branches once I'd fixed master. :-) [17:02:13] (03CR) 10jerkins-bot: [V: 04-1] mwext-MobileFrontend-npm-run-lint-modules-docker: Make non-voting for now [integration/config] - 10https://gerrit.wikimedia.org/r/514347 (https://phabricator.wikimedia.org/T224997) (owner: 10Jforrester) [17:03:55] (03PS2) 10Jforrester: mwext-MobileFrontend-npm-run-lint-modules-docker: Make non-voting for now [integration/config] - 10https://gerrit.wikimedia.org/r/514347 (https://phabricator.wikimedia.org/T224997) [17:04:31] (03PS1) 10Umherirrender: [ContentTranslation] Add GlobalPreferences as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/514354 [17:14:03] And CI is backed up because of MinervaNeue again [17:18:11] * James_F sighs. [17:18:24] Minerva is really the nadir of over-testing. [17:18:59] even better when the tests fail because of flakyness [17:19:27] Indeed. :-( [17:27:18] Happy to pare back the tests that are run against MFE/MN if they're not needed [17:27:19] (03CR) 10Jforrester: [C: 03+2] mwext-MobileFrontend-npm-run-lint-modules-docker: Fix to run on master, wmf, and release branches (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/514346 (owner: 10Jforrester) [17:27:28] (03CR) 10Jforrester: [C: 03+2] mwext-MobileFrontend-npm-run-lint-modules-docker: Make non-voting for now [integration/config] - 10https://gerrit.wikimedia.org/r/514347 (https://phabricator.wikimedia.org/T224997) (owner: 10Jforrester) [17:28:06] I may have misunderstood but I thought that the flakyness was coming from other extensions [17:28:41] (That may in part be what was meant by over-testing) [17:29:25] (03Merged) 10jenkins-bot: mwext-MobileFrontend-npm-run-lint-modules-docker: Fix to run on master, wmf, and release branches [integration/config] - 10https://gerrit.wikimedia.org/r/514346 (owner: 10Jforrester) [17:29:27] (03Merged) 10jenkins-bot: mwext-MobileFrontend-npm-run-lint-modules-docker: Make non-voting for now [integration/config] - 10https://gerrit.wikimedia.org/r/514347 (https://phabricator.wikimedia.org/T224997) (owner: 10Jforrester) [17:30:00] phuedx: It's not any one repo's fault (well, except Wikibase's), it's about how MF touches twenty things and so has twenty different sets of tests to run. :-( [17:30:26] phuedx: And yes, the flaky test is in core and has been "fixed" three times, to my recollection. Let's just skip the damn thing. [17:31:12] !log Reloading Zuul to make mwext-MobileFrontend-npm-run-lint-modules-docker non-voting T224997 [17:31:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:31:15] T224997: Update MobileFrontend-npm-run-lint-modules-docker to run node10 - https://phabricator.wikimedia.org/T224997 [17:31:29] Speaking of which, MF should now actually be able to merge code. [17:31:31] * James_F sighs. [17:32:45] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10MobileFrontend, 10JavaScript, 10Patch-For-Review: Update MobileFrontend-npm-run-lint-modules-docker to run node10 - https://phabricator.wikimedia.org/T224997 (10Jdforrester-WMF) p:05Triage→03High I've made it non-voting for now,... [17:32:45] Thanks, James_F <3 [17:32:57] phuedx: Thank me when it's properly fixed. :-) [17:33:08] I'll do that too :) [17:33:11] * James_F grins. [17:35:43] (03PS3) 10Jforrester: jjb: Migrate PHP code coverage jobs to node10 quibble, too [integration/config] - 10https://gerrit.wikimedia.org/r/514189 (https://phabricator.wikimedia.org/T224983) [17:39:33] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [17:39:46] 10Continuous-Integration-Config: The phpunit-coverage jobs shouldn't run npm install - https://phabricator.wikimedia.org/T225008 (10Jdforrester-WMF) [17:40:33] (03CR) 10Jforrester: [C: 03+2] jjb: Migrate PHP code coverage jobs to node10 quibble, too [integration/config] - 10https://gerrit.wikimedia.org/r/514189 (https://phabricator.wikimedia.org/T224983) (owner: 10Jforrester) [17:42:51] (03Merged) 10jenkins-bot: jjb: Migrate PHP code coverage jobs to node10 quibble, too [integration/config] - 10https://gerrit.wikimedia.org/r/514189 (https://phabricator.wikimedia.org/T224983) (owner: 10Jforrester) [17:44:56] !log Updating jjb PHP code coverage jobs to node10 quibble T224983 [17:44:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:44:59] T224983: mediawiki-phpunit-coverage-patch-docker fails to install fibers@3.1.1 - https://phabricator.wikimedia.org/T224983 [17:55:08] 10Continuous-Integration-Infrastructure, 10Front-end-Standards-Group, 10MediaWiki-extensions-General, 10Core Platform Team Backlog (Designing), 10Services (designing): Decide whether we want the package-lock.json to commit or ignore - https://phabricator.wikimedia.org/T179229 (10Jdforrester-WMF) 05Open... [18:00:40] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata, 10Patch-For-Review, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞): [Task] Add Scribunto to extension-gate in CI - https://phabricator.wikimedia.org/T125050 (10Anomie) >>! In T125050#5233908, @hashar wrote: > I guess we c... [18:06:25] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata, 10Patch-For-Review, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞): [Task] Add Scribunto to extension-gate in CI - https://phabricator.wikimedia.org/T125050 (10Jdforrester-WMF) If we want a integration test group whose ob... [18:06:46] o/ twentyafterfour. Could you spare a moment to look at https://phabricator.wikimedia.org/T203246 [18:06:56] I think you have solved this issue for us in the past so I'm hoping it is simple here. [18:09:10] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews: Code Stewardship Review: SpamBlacklist - https://phabricator.wikimedia.org/T224921 (10matmarex) [18:11:56] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10JavaScript, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), 10Patch-For-Review: Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 (10Jdforrester-WMF) [18:11:58] 10Continuous-Integration-Config: mediawiki-phpunit-coverage-patch-docker fails to install fibers@3.1.1 - https://phabricator.wikimedia.org/T224983 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF Seems to be working. He says. [18:22:41] 10Beta-Cluster-Infrastructure, 10Wikimedia-Hackathon-2019, 10Chinese-Sites, 10Patch-For-Review: Please create beta zhwikivoyage - https://phabricator.wikimedia.org/T223770 (10Shizhao) 05Open→03Resolved a:03Shizhao is ok [18:23:20] 10Beta-Cluster-Infrastructure, 10Wikimedia-Hackathon-2019, 10Chinese-Sites, 10Patch-For-Review: Please create beta zhwikivoyage - https://phabricator.wikimedia.org/T223770 (10Shizhao) a:05Shizhao→03None [18:24:35] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10JavaScript, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), 10Patch-For-Review: Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 (10Jdforrester-WMF) Remaining matches to `quibble-stretch(-bund... [18:29:51] James_F: any idea why the post merge builds are failing for this? https://integration.wikimedia.org/ci/job/trigger-service-pipeline-test-and-publish/211/console [18:29:59] and other recent changes [18:30:17] the logs just say unable to connect to localhost:8080 [18:34:09] Hmm [18:34:25] mvolz: No. [18:34:30] RUNNING: citoid-mediawiki-services-citoid-e3bx42ru-service-checker [18:34:31] FAILED: citoid-mediawiki-services-citoid-e3bx42ru-service-checker, run `kubectl logs citoid-mediawiki-services-citoid-e3bx42ru-service-checker --namespace ci` for more info [18:34:32] :) [18:34:36] Not very specific. [18:34:37] yeah [18:34:54] thought it might related to the new CI stuff but maybe not [18:35:01] I asked earlier in OPs but no bites. [18:35:02] No, shouldn't be. [18:35:09] k thanks [18:35:17] The new CI stuff is all in the legacy MW land, won't touch the pipeline stuff. [18:35:22] ah okay [18:35:39] ("Legacy" but also "current for the next 2 years". :-() [18:36:25] awwww [18:36:41] we'll get there someday! [18:52:35] 10Diffusion, 10Release-Engineering-Team (Kanban), 10Operations: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10greg) [19:02:35] 10Diffusion, 10Gerrit, 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, and 2 others: Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10Halfak) p:05Triage→03High [19:02:46] 10Diffusion, 10Gerrit, 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, and 2 others: Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10Halfak) This is blocking an ORES deployment. [19:09:38] ouch [19:09:47] 2h 5 minutes for jenkins to say no to https://gerrit.wikimedia.org/r/#/c/mediawiki/skins/MinervaNeue/+/512224/ [19:10:02] well, hhvm timed out [19:11:57] this is why I dream of 10 minute ci runs, so that if it's going to fail, it can be gone fast [19:12:04] and leave the queue for other changes [19:19:07] apergos: You'll be glad to know that 5–10 mins is roughly our target. [19:19:20] <3 [19:19:33] Target !== achieved. [19:19:44] But e.g. the splitting of unit and integration tests is very promising. [19:20:01] oh yeah that's smart [19:20:18] With https://gerrit.wikimedia.org/r/c/mediawiki/core/+/513106 I can run ~4000 MW unit tests in ~2 seconds. [19:20:25] holy crap [19:20:31] 😍 [19:20:49] CI won't be as fast, and there's still the 12000 other tests (most of which we'll have to say are integration tests). [19:20:52] But… [19:21:25] Also right now that patch breaks the code coverage tool, which I need to find out why and fix before merging. :-) [19:21:53] heh [19:22:03] still that's a giant leap forwards [19:24:52] Yes. [19:25:09] And enforcing unit test are actually unit tests, and not using globals or services, is a big step. [19:27:07] (03PS2) 10Jeena Huneidi: prereqs: Remove docker installation steps [releng/local-charts] - 10https://gerrit.wikimedia.org/r/514166 (https://phabricator.wikimedia.org/T223715) (owner: 10Brennen Bearnes) [19:34:56] PROBLEM - Host deployment-schema-1 is DOWN: CRITICAL - Host Unreachable (172.16.7.154) [19:42:24] PROBLEM - Host integration-slave-docker-1053 is DOWN: CRITICAL - Host Unreachable (172.16.7.104) [19:43:14] ^^^ I have deleted it [19:43:28] ah that's a good point! [19:46:44] * apergos is idling in here because they are still prepping for local profiling of dumps on today's new branch [19:47:05] these jobs are so sloooow. but at least they are slow enough that we can see where there are bottlenecks [20:04:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Update quibble jobs to use phpunit-patch-coverage 0.0.10 - https://phabricator.wikimedia.org/T220917 (10Jdforrester-WMF) a:03Jdforrester-WMF [20:10:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Update quibble jobs to use phpunit-patch-coverage 0.0.10 - https://phabricator.wikimedia.org/T220917 (10Jdforrester-WMF) [20:11:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Update quibble jobs to use phpunit-patch-coverage 0.0.10 - https://phabricator.wikimedia.org/T220917 (10Jdforrester-WMF) Though quibble-stretch used to pull this in (added in 46950e114b2a3a358d92f9d1a3eb49fb07d82e68), it was later removed and... [20:15:13] (03PS1) 10Jforrester: dockerfiles: [quibble-stretch-php70] Upgrade phpunit-patch-coverage from 0.0.9 to 0.0.10 [integration/config] - 10https://gerrit.wikimedia.org/r/514387 (https://phabricator.wikimedia.org/T220917) [20:21:48] (03PS1) 10Jforrester: jjb: Migrate PHP code coverage jobs to quibble-stretch-php70:0.0.31-5 [integration/config] - 10https://gerrit.wikimedia.org/r/514391 (https://phabricator.wikimedia.org/T220917) [20:26:03] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Update quibble jobs to use phpunit-patch-coverage 0.0.10 - https://phabricator.wikimedia.org/T220917 (10Jdforrester-WMF) [20:27:54] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [quibble-stretch-php70] Upgrade phpunit-patch-coverage from 0.0.9 to 0.0.10 [integration/config] - 10https://gerrit.wikimedia.org/r/514387 (https://phabricator.wikimedia.org/T220917) (owner: 10Jforrester) [20:30:01] (03Merged) 10jenkins-bot: dockerfiles: [quibble-stretch-php70] Upgrade phpunit-patch-coverage from 0.0.9 to 0.0.10 [integration/config] - 10https://gerrit.wikimedia.org/r/514387 (https://phabricator.wikimedia.org/T220917) (owner: 10Jforrester) [20:31:31] !log Updating docker-pkg files on contint1001 for quibble-stretch-php70 T220917 [20:31:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:31:33] T220917: Update quibble jobs to use phpunit-patch-coverage 0.0.10 - https://phabricator.wikimedia.org/T220917 [20:36:27] (03CR) 10Jforrester: [C: 03+2] jjb: Migrate PHP code coverage jobs to quibble-stretch-php70:0.0.31-5 [integration/config] - 10https://gerrit.wikimedia.org/r/514391 (https://phabricator.wikimedia.org/T220917) (owner: 10Jforrester) [20:39:23] (03Merged) 10jenkins-bot: jjb: Migrate PHP code coverage jobs to quibble-stretch-php70:0.0.31-5 [integration/config] - 10https://gerrit.wikimedia.org/r/514391 (https://phabricator.wikimedia.org/T220917) (owner: 10Jforrester) [20:39:30] !log Updating jjb PHP code coverage jobs to quibble-stretch-php70:0.0.31-5 T220917 [20:39:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:39:33] T220917: Update quibble jobs to use phpunit-patch-coverage 0.0.10 - https://phabricator.wikimedia.org/T220917 [20:46:06] 10Continuous-Integration-Config, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): mwext-MobileFrontend-npm-run-lint-modules-docker failing - node script running as php - https://phabricator.wikimedia.org/T196956 (10Jdforrester-WMF) >>! In T196956#4278622, @Jhernandez wrote: > We will revisit in the fut... [20:55:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Update quibble jobs to use phpunit-patch-coverage 0.0.10 - https://phabricator.wikimedia.org/T220917 (10Jdforrester-WMF) 05Open→03Resolved Success. [20:55:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Update quibble jobs to use phpunit-patch-coverage 0.0.10 - https://phabricator.wikimedia.org/T220917 (10Jdforrester-WMF) [20:56:22] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-VPS (Quota-requests): Request new Flavor for integration Cloud VPS project - https://phabricator.wikimedia.org/T225025 (10hashar) [20:56:37] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10serviceops, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) p:05Triage→03Normal [20:59:03] 10Release-Engineering-Team, 10Release Pipeline, 10Wikibugs: Hide successful PipelineBot messages from wikibugs's feed, the same way we do for jenkins-bot - https://phabricator.wikimedia.org/T218442 (10Jdforrester-WMF) [20:59:07] 10Release-Engineering-Team, 10MediaWiki-extensions-CodeReview, 10Wikimedia-Site-requests, 10Technical-Debt: Undeploy CodeReview - https://phabricator.wikimedia.org/T116948 (10Jdforrester-WMF) [20:59:09] 10Release-Engineering-Team, 10Wikimedia-Site-requests, 10Patch-For-Review: Remove Wikipedia Zero-related extensions from production - https://phabricator.wikimedia.org/T212865 (10Jdforrester-WMF) [20:59:13] 10Release-Engineering-Team, 10Wikimedia-Site-requests: Disable Wikipedia Zero-related config in production - https://phabricator.wikimedia.org/T212864 (10Jdforrester-WMF) [21:00:12] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban): mediawiki-phpunit-coverage-patch-docker fails to install fibers@3.1.1 - https://phabricator.wikimedia.org/T224983 (10Jdforrester-WMF) [21:01:08] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10MediaWiki-General-or-Unknown, 10MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), 10User-zeljkofilipin: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11 - https://phabricator.wikimedia.org/T210506 (10Jdforreste... [21:05:52] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10JavaScript, 10Patch-For-Review: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10Jdforrester-WMF) [21:05:55] 10Continuous-Integration-Config, 10Mobile-Content-Service, 10Reading-Infrastructure-Team-Backlog, 10JavaScript, 10Patch-For-Review: Upgrade the mobileapps CI job from npm3 to npm6 - https://phabricator.wikimedia.org/T211785 (10Jdforrester-WMF) 05Open→03Invalid We're going to jump straight to the pipe... [21:14:17] Ugh, looks like one stuck job is holding Zuul hostage for more than 2 hours [21:14:39] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Cloud-VPS (Quota-requests): Request new Flavor for integration Cloud VPS project - https://phabricator.wikimedia.org/T225025 (10greg) [21:15:06] 10Release-Engineering-Team (Backlog), 10MediaWiki-extensions-CodeReview, 10Wikimedia-Site-requests, 10Technical-Debt: Undeploy CodeReview - https://phabricator.wikimedia.org/T116948 (10greg) [21:15:24] 10Release-Engineering-Team (Backlog), 10Wikimedia-Site-requests, 10Patch-For-Review: Remove Wikipedia Zero-related extensions from production - https://phabricator.wikimedia.org/T212865 (10greg) [21:18:26] MaxSem: No. [21:19:25] The backlog has been processing roughly as normal. [21:19:29] It's just slow as normal. [21:20:13] 10Release-Engineering-Team, 10MediaWiki-extensions-Other, 10translatewiki.net, 10Technical-Debt, 10User-Zoranzoki21: Archive Extension:Google +1 - https://phabricator.wikimedia.org/T216675 (10MaxSem) Adding releng for remaining stuff. [21:23:04] (03PS1) 10Reedy: Update README to include upload-tar [tools/release] - 10https://gerrit.wikimedia.org/r/514398 [21:32:09] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Parsoid, 10phan-taint-check-plugin, and 2 others: composer-package-php73-docker seems to fail often on Parsoid builds - https://phabricator.wikimedia.org/T221872 (10hashar) 05Open→03Resolved Should be good now, I ended up d... [21:39:32] 10Project-Admins: Create project for Wiki Techstorm 2019 - https://phabricator.wikimedia.org/T222929 (10Ecritures) @Aklapper That is totally correct indeed :) I hope you will be able to do this for us. Then I will start putting new tasks on the Wiki-Techstorm-2019 board. Thanks for all your help. [21:40:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: On CI Jenkins, audit slave labels and remove unused ones - https://phabricator.wikimedia.org/T225031 (10hashar) [21:48:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: On CI Jenkins, audit worker labels and remove unused ones - https://phabricator.wikimedia.org/T225031 (10Jdforrester-WMF) [21:49:18] 10Release-Engineering-Team, 10Developer Productivity, 10local-charts: Create an interface for the local-charts ecosystem - https://phabricator.wikimedia.org/T224939 (10jeena) We had a meeting to discuss our next steps. We talked about whether we would like to use Go or Python to create a CLI with a web inte... [21:55:38] (03CR) 10Reedy: [C: 03+2] Update README to include upload-tar [tools/release] - 10https://gerrit.wikimedia.org/r/514398 (owner: 10Reedy) [21:56:13] (03Merged) 10jenkins-bot: Update README to include upload-tar [tools/release] - 10https://gerrit.wikimedia.org/r/514398 (owner: 10Reedy) [22:02:42] (03PS1) 10Reedy: Add a working example [tools/release] - 10https://gerrit.wikimedia.org/r/514404 [22:04:06] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-WikibaseRepository, and 3 others: Wikibase and WikibaseLexeme Selenium tests fail when UseCirrus is true in WikibaseCirrusSearch - https://phabricator.wikimedia.org/T224125 (10Smalyshev) > Ultimately, we shouldn't b... [22:05:16] (03CR) 10Reedy: [C: 03+2] Add a working example [tools/release] - 10https://gerrit.wikimedia.org/r/514404 (owner: 10Reedy) [22:05:47] (03Merged) 10jenkins-bot: Add a working example [tools/release] - 10https://gerrit.wikimedia.org/r/514404 (owner: 10Reedy) [22:26:18] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10local-charts: Move local-charts helm charts to a chart repository - https://phabricator.wikimedia.org/T224935 (10jeena) a:03jeena [22:29:44] (03CR) 10Brennen Bearnes: "This change is ready for review." [releng/local-charts] - 10https://gerrit.wikimedia.org/r/514166 (https://phabricator.wikimedia.org/T223715) (owner: 10Brennen Bearnes) [22:31:29] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10serviceops, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Ladsgroup) Here's my two cents: - I have done a similar thing with ores, in couple of months we ma... [22:36:01] (03PS3) 10Brennen Bearnes: prereqs: Remove docker installation steps [releng/local-charts] - 10https://gerrit.wikimedia.org/r/514166 (https://phabricator.wikimedia.org/T223715) [22:45:47] hello! I assume you all are aware of the slowness with the CI pipeline? This patch was +2'd and it took 22 hours before the gate-and-submit jobs completed, and it failed because of something unrelated :( https://gerrit.wikimedia.org/r/c/mediawiki/core/+/514105 [22:48:24] 22 hours sounds like a zuul gate getting wedged... none of the tests seem to have crazy runtimes. [22:49:00] lots of spikes on https://grafana.wikimedia.org/d/000000321/zuul?orgId=1 but I don't look at that dashboard often enough to know what it means [22:49:19] yeah everything seems really backed up. Jobs are taking 2-3 hours, so it's a long wait if you're at the end of the queue! [22:50:59] musikanimal: It's not immediately clear how it suddenly decided to start the gate-and-submit on that one. [22:51:15] Possibly my refresh of jobs caused zuul to re-examine queues? [22:51:23] Anyway, yeah, 2–3 hours isn't good. [22:51:35] there is a pretty deep queue for gate-and-submit at https://integration.wikimedia.org/zuul/ right now [22:51:49] it's been that way since sometime yesterday [22:51:50] Yes, well, there always is at this time of day in a work week. [22:52:08] But "10 patches waiting" shouldn't mean "three hours of waiting". [22:52:20] right [22:52:45] I wonder if this is due to hashar reducing executers yesturday (for jobs to run on)? [22:53:12] Possibly. But I think we were having the issues before the reduced executors? [22:53:29] there seem to be open docker slots on https://integration.wikimedia.org/ci/ but I have lost track of how the inter-job dependencies work [22:54:50] I know h.ashar has been trying to figure out why some of the vms are slower than others, I guess it could be related to that too. Like if a bunch of the docker runner nodes are on the cloudvirt hosts h.ashar has noticed slowness from [22:55:10] He killed a couple of slow WMCS hosts yesterday. [22:55:21] i doin't think it's the slowness of the tests. [22:55:24] I don't know the details, just that some of our hosts are faster than others. [22:55:26] that's the cause of this issue [22:55:28] we really should have bought dedicated hardware for jenkins runners about 6 years ago [22:55:33] rather im not seeing many jobs running https://integration.wikimedia.org/zuul/ [22:55:35] bd808: Indeed. :-( [22:55:49] It's not deadlocked. [22:56:09] If I had an "extra" cloudvirt I would offer it up [22:56:22] It's just almost all of the jobs are queued due to contention, because they're all in gate-and-submit. [22:56:34] and integration-docker-* has enough executors when looking. [22:56:35] gate does nearly all the mw jobs in one queue [22:56:37] bd808: Oh, yes, it's definitely not just a question of turning a tap. [22:56:58] But e.g. https://gerrit.wikimedia.org/r/c/VisualEditor/VisualEditor/+/446706 just merged in < 2 minutes from C+2. [22:57:08] (Which is in its own queue.) [22:57:36] At $DAYJOB-1 I "fixed" this problem by making every engineer's desktop workstation a jenkins runner node. Instant +50 node capacity :) [22:58:01] lol [22:58:04] I'd love to see that work here [22:58:12] we also had a much simpler codebase to test [22:58:13] Considering we can't get a dev env that works everywhere ;P [22:58:22] Reedy: We're working on it. [22:58:28] heh [22:58:41] Wewwy wewwy slowly. [22:58:44] But we're working on it. [22:59:18] I predict minikube is not going to work on more hosts than mw-vagrant. Other things will get better, but hardware/OS compat will get worse [23:00:00] bd808: Sure, but it'll be more prod-like. [23:00:08] And if necessary we can buy all the devs machines. [23:00:10] that's not a knock against the project though! [23:00:29] It's cheaper than endless engineer hours trying to make everything work everywhere. [23:02:25] 494 distinct gerrit patch authors in the last 90 days... [23:08:13] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews: Code Stewardship Review: SpamBlacklist - https://phabricator.wikimedia.org/T224921 (10Jrbranaa) Hello @marcella, this extension came up as a Code Stewardship review request, but I've noticed that it is marked as being stewarded by the Editing tea... [23:27:23] 10Phabricator, 10CommRel-Design, 10CommRel-Internals: Create Phabricator form for CommRel-Design and Comms requests and add a link to it in the "Star" dropdown - https://phabricator.wikimedia.org/T223102 (10hdothiduc) Hi @Aklapper thank you very much for all of this! You can actually remove me as a subscribe...