[00:00:06] (03CR) 10Jforrester: [C: 03+2] docker: Use php-ast 1.0.0 for PHP7.0 and unbreak everything [integration/config] - 10https://gerrit.wikimedia.org/r/520793 (https://phabricator.wikimedia.org/T218719) (owner: 10Daimona Eaytoy) [00:00:11] (03CR) 10Jforrester: [C: 03+2] dockerfiles: Upgrade composer-package* to use php-ast 1.0.1-1 image [integration/config] - 10https://gerrit.wikimedia.org/r/521356 (owner: 10Jforrester) [00:02:57] (03Merged) 10jenkins-bot: docker: Use php-ast 1.0.0 for PHP7.0 and unbreak everything [integration/config] - 10https://gerrit.wikimedia.org/r/520793 (https://phabricator.wikimedia.org/T218719) (owner: 10Daimona Eaytoy) [00:03:00] (03Merged) 10jenkins-bot: dockerfiles: Upgrade composer-package* to use php-ast 1.0.1-1 image [integration/config] - 10https://gerrit.wikimedia.org/r/521356 (owner: 10Jforrester) [00:04:37] !log Pushing php-ast:1.0.1-1 and related composer-package* docker image updates [00:04:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:09:32] 10Continuous-Integration-Config, 10phan-taint-check-plugin, 10Patch-For-Review: Upgrade php-ast to 1.0.1 in CI composer-package containers - https://phabricator.wikimedia.org/T218719 (10Jdforrester-WMF) Of course, the images that are working are the `composer-package*` images, but `mediawiki-phan` and `media... [00:35:05] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: Upgrade to Gerrit 2.16.10 - https://phabricator.wikimedia.org/T200739 (10Paladox) [01:48:04] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<22.22%) [05:04:18] 10Beta-Cluster-Infrastructure: beta-update-databases-eqiad is failing with a composer problem - https://phabricator.wikimedia.org/T210341 (10Legoktm) 05Open→03Resolved [05:08:29] (03CR) 10Legoktm: "I don't think this really does what quibble wants, because it's just pulling in hard dependencies, and not development dependencies. Also " [integration/quibble] - 10https://gerrit.wikimedia.org/r/502286 (https://phabricator.wikimedia.org/T193824) (owner: 10Hashar) [05:12:35] 10Release-Engineering-Team-TODO (201907), 10Release Pipeline, 10Core Platform Team (Extension Management (TEC13)), 10MW-1.34-notes (1.34.0-wmf.8; 2019-06-04), 10Patch-For-Review: Determine a standard way of installing MediaWiki lib/extension dependencies within c... - https://phabricator.wikimedia.org/T193824 [06:53:05] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:09:36] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Release, 10Train Deployments: 1.34.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T220738 (10Daimona) [07:17:10] (03PS1) 10Daimona Eaytoy: jjb: Bump composer-packate to latest [integration/config] - 10https://gerrit.wikimedia.org/r/521422 (https://phabricator.wikimedia.org/T218719) [07:19:09] 10Continuous-Integration-Config, 10phan-taint-check-plugin, 10Patch-For-Review: Upgrade php-ast to 1.0.1 in CI composer-package containers - https://phabricator.wikimedia.org/T218719 (10Daimona) @Jdforrester-WMF Cool, thanks! Yes, for those I created T227385, although until we keep installing ast 0.1.2, we c... [07:36:13] 10Deployments, 10Operations, 10Release: OSError: [Errno 1] Operation not permitted when running git fat pull - https://phabricator.wikimedia.org/T208259 (10Gehel) @thcipriani Thanks a lot for the detailed explanation! As this step is not really required on the deploy host, I'll change my notes to validate t... [08:11:58] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Operations, 10serviceops: Upload docker-ce 18.06.3 upstream package for Stretch - https://phabricator.wikimedia.org/T226236 (10hashar) >>! In T226236#5296968, @MoritzMuehlenhoff wrote: > Well, if there's a security update for Docker we'll want it... [08:17:28] 10Continuous-Integration-Config, 10Gerrit, 10Release-Engineering-Team-TODO, 10Patch-For-Review: Setup CI for operations/software/gerrit - https://phabricator.wikimedia.org/T189549 (10hashar) Indeed we now have a Docker image with Bazel 0.27.1 (`docker-registry.wikimedia.org/releng/bazel:0.1.0`), it lacks a... [08:20:45] PROBLEM - Disk space on contint1001 is CRITICAL: DISK CRITICAL - /mnt/docker/overlay2/f5724cda69fb24bcfe9776790633ebd77afef759c203c7530a560f548df7d96f/merged is not accessible: Permission denied https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space [08:21:21] 10Gerrit: Update Gerrit documentation on mediawiki.org before upgrading to Gerrit 2.16.x / PolyGerrit UI - https://phabricator.wikimedia.org/T227562 (10Aklapper) [08:21:31] 10Gerrit, 10Documentation: Update Gerrit documentation on mediawiki.org before upgrading to Gerrit 2.16.x / PolyGerrit UI - https://phabricator.wikimedia.org/T227562 (10Aklapper) [08:25:09] RECOVERY - Disk space on contint1001 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space [08:47:34] (03PS1) 10Daimona Eaytoy: Add a missing phan dependency for Flow [integration/config] - 10https://gerrit.wikimedia.org/r/521434 [08:55:59] (03PS1) 10Daimona Eaytoy: Enable phan for Flow [integration/config] - 10https://gerrit.wikimedia.org/r/521437 (https://phabricator.wikimedia.org/T224759) [08:59:41] (03PS2) 10Daimona Eaytoy: jjb: Bump composer-package to latest [integration/config] - 10https://gerrit.wikimedia.org/r/521422 (https://phabricator.wikimedia.org/T218719) [09:04:43] (03CR) 10Hashar: "Eeek I have missed the "dev-requires" section which is present since v1 of the schema. Though nobody is using it right now, I actually ne" [integration/quibble] - 10https://gerrit.wikimedia.org/r/502286 (https://phabricator.wikimedia.org/T193824) (owner: 10Hashar) [09:37:32] 10Continuous-Integration-Config, 10phan: Find a way to run phan PerfCheck plugin as non-voting in CI - https://phabricator.wikimedia.org/T227569 (10Daimona) [09:38:01] 10Continuous-Integration-Config, 10phan: Find a way to run phan PerfCheck plugin as non-voting in CI - https://phabricator.wikimedia.org/T227569 (10Daimona) [09:47:35] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10Operations, 10serviceops: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10hashar) 05Open→03Resolved Perfect thank you @thcipriani , those tests were exactly... [09:47:39] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Release Pipeline, 10Patch-For-Review: contint1001:/var/lib/docker growth - https://phabricator.wikimedia.org/T207702 (10hashar) [09:47:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10Operations, 10Release Pipeline: Switch CI Docker Storage Driver to its own partition and to use devicemapper - https://phabricator.wikimedia.org/T178663 (10hashar) [09:48:58] !log contint1001: removing /var/lib/docker , no more needed since we now use /mnt/docker # T207707 [09:49:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:49:01] T207707: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 [10:27:17] hallo [10:27:34] Does anyone have an idea why does https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Babel/+/521340/ fail? [10:31:46] zeljkof, Amir1 ^ [10:34:33] aharoni: I'm looking, but gerrit is so slow for me today, I'm having trouble opening any links :/ [10:35:47] aharoni: it's failure of phan on master [10:35:48] https://integration.wikimedia.org/ci/job/mwext-php72-phan-docker/2089/console [10:35:58] [10:36:23] Do you want me to fix it? [10:36:59] https://codesearch.wmflabs.org/search/?q=wfBCP47&i=nope&files=&repos= has been removed a gazillion years ago [10:38:38] 10Release-Engineering-Team (Deployment services), 10Operations, 10SRE-Access-Requests: Request access to deployment cluster for Jakob_WMDE - https://phabricator.wikimedia.org/T227193 (10MoritzMuehlenhoff) @Jakob_WMDE: Please generate a separate SSH key for the access to the Wikimedia production cluster (it n... [10:42:03] aharoni: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Babel/+/521464 [10:42:23] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Testing, 10Browser-Tests: Run browser tests in parallel - https://phabricator.wikimedia.org/T226869 (10zeljkofilipin) [10:45:09] (03PS1) 10Urbanecm: Whitelist Shreyas Minocha's alternate email address [integration/config] - 10https://gerrit.wikimedia.org/r/521465 [10:52:12] Amir1 - Thanks! Curious that this test failed, but others didn't. I've been writing much more emails and docs and docs recently, and I'm a bit disconnected from code and CI :/ [10:52:48] And it's a bit concerning that it's only being noticed now because of my trivial patch, and not automatically detected earlier. [10:54:31] aharoni: that's the issue with phan, we discussed this in the hackathon, it seems for some reason phan can't be updated gradually [10:54:37] so it breaks things quite often [10:55:49] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10User-zeljkofilipin: Finish incident documentation for 1.33.0-wmf.8 and 1.33.0-wmf.9 - https://phabricator.wikimedia.org/T214688 (10zeljkofilipin) 05Open→03Declined This is more than 6 months old. There's nothing... [10:56:06] (03PS2) 10Hashar: Whitelist Shreyas Minocha's alternate email address [integration/config] - 10https://gerrit.wikimedia.org/r/521465 (owner: 10Urbanecm) [10:56:30] (03CR) 10Hashar: [C: 03+2] "I have fixed the email, it lacked '7' :]" [integration/config] - 10https://gerrit.wikimedia.org/r/521465 (owner: 10Urbanecm) [10:58:26] (03Merged) 10jenkins-bot: Whitelist Shreyas Minocha's alternate email address [integration/config] - 10https://gerrit.wikimedia.org/r/521465 (owner: 10Urbanecm) [11:28:15] 10Release-Engineering-Team-TODO (201907), 10Release Pipeline, 10Core Platform Team (Extension Management (TEC13)), 10MW-1.34-notes (1.34.0-wmf.8; 2019-06-04), 10Patch-For-Review: Determine a standard way of installing MediaWiki lib/extension dependencies within c... - https://phabricator.wikimedia.org/T193824 [11:30:22] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Readers-Web-Backlog (Tracking), and 2 others: Provide generic wrapper inside wdio-mediawiki for API requests - https://phabricator.wikimedia.org/T220712 (10zeljkofilipin) p:05Triage→03N... [11:32:18] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Developer Productivity, 10MediaWiki-Core-Testing, and 3 others: TEC12:O1:O1.4 Goal – Support running Selenium browser tests in the docker local development environment - https://phabricator.wikimedia.org/T222234 (10zeljkofilipin) [11:33:50] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) a:03zeljkofilipin [11:34:05] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) [11:38:42] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Documentation, 10Ruby: Improve mediawiki_api documentation with inline yard - https://phabricator.wikimedia.org/T102726 (10zeljkofilipin) p:05Low→03Lowest mediawiki_api Ruby gem was used by Selenium Ruby framework. Since the framework depre... [11:40:32] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Testing-Initiative: Guides for initializing a test suite: unit testing & browser testing - https://phabricator.wikimedia.org/T108107 (10zeljkofilipin) Browser/Selenium testing documentation is good enough, I think: https://www.mediawiki.org/wiki/... [11:43:40] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Ruby, and 2 others: mediawiki_api gem recursion on log_in - https://phabricator.wikimedia.org/T111133 (10zeljkofilipin) p:05Low→03Lowest mediawiki_api Ruby gem was used by Selenium Ruby framework. Since the framework dep... [11:45:42] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Ruby: Ruby mediawiki_api client hides error details - https://phabricator.wikimedia.org/T127786 (10zeljkofilipin) p:05Low→03Lowest mediawiki_api Ruby gem was used by Selenium Ruby framework. Since the framework deprecated, I'm not sure if any... [11:47:14] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Ruby: mediawiki_api Ruby gem incorrectly assumes path to index.php - https://phabricator.wikimedia.org/T149169 (10zeljkofilipin) [11:47:42] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Ruby: mediawiki_api Ruby gem incorrectly assumes path to index.php - https://phabricator.wikimedia.org/T149169 (10zeljkofilipin) p:05Low→03Lowest mediawiki_api Ruby gem was used by Selenium Ruby framework. Since the framework deprecated, I'm... [11:53:56] 10Release-Engineering-Team-TODO (201907), 10MediaWiki-Uploading, 10Multimedia, 10Patch-For-Review: .odb file recognized as application/acad (first type entry in our list) - https://phabricator.wikimedia.org/T35515 (10hashar) I have rebased the patch. Pending review / +2 [11:54:07] 10Release-Engineering-Team-TODO (201907), 10MediaWiki-Uploading, 10Multimedia, 10Patch-For-Review: .odb file recognized as application/acad (first type entry in our list) - https://phabricator.wikimedia.org/T35515 (10hashar) 05Open→03Stalled [11:56:34] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Drop Ruby Selenium CI jobs; we don't support them any more - https://phabricator.wikimedia.org/T220035 (10zeljkofilipin) I'll check t... [12:01:29] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201907), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [12:09:12] (03Abandoned) 10Hashar: Replace tin.eqiad.wmnet with deploy1001.eqiad.wmnet [tools/scap] - 10https://gerrit.wikimedia.org/r/513238 (owner: 10Paladox) [12:15:05] 10Beta-Cluster-Infrastructure, 10RelEng-Archive-FY201718-Q1, 10Release-Engineering-Team-TODO (201907), 10media-storage, 10Patch-For-Review: deployment-ms-be03.deployment-prep and deployment-ms-be04.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990 (10hashar) All the... [12:33:54] (03CR) 10Hashar: "We still use php 5.6 for https://integration.wikimedia.org/ which is on contint1001 (Jessie). So blocked on upgrading the server: T224591" [integration/config] - 10https://gerrit.wikimedia.org/r/516570 (owner: 10Jforrester) [12:33:59] (03CR) 10Hashar: [C: 04-1] layout: [integration/docroot] Switch from php56 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/516570 (owner: 10Jforrester) [12:36:31] (03CR) 10Hashar: "I would like to dig into the origin of those 'composertest' jobs based on Quibble. Sounds to me we could just use 'composer install && com" [integration/config] - 10https://gerrit.wikimedia.org/r/519644 (owner: 10Kosta Harlan) [12:42:06] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Ruby: mediawiki_api Ruby gem improvements - https://phabricator.wikimedia.org/T227584 (10zeljkofilipin) [12:42:25] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Ruby: mediawiki_api Ruby gem improvements - https://phabricator.wikimedia.org/T227584 (10zeljkofilipin) p:05Triage→03Lowest [12:42:55] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Ruby, 10User-zeljkofilipin: mediawiki_api Ruby gem improvements - https://phabricator.wikimedia.org/T227584 (10zeljkofilipin) a:03zeljkofilipin [12:43:37] 10Release-Engineering-Team-TODO (201907), 10Ruby, 10User-zeljkofilipin: mediawiki_api Ruby gem improvements - https://phabricator.wikimedia.org/T227584 (10zeljkofilipin) [12:46:50] 10Release-Engineering-Team-TODO (201907), 10Ruby, 10User-zeljkofilipin: mediawiki_api Ruby gem improvements - https://phabricator.wikimedia.org/T227584 (10zeljkofilipin) [12:47:04] 10Release-Engineering-Team-TODO (201907), 10Ruby, 10User-zeljkofilipin: mediawiki_api Ruby gem improvements - https://phabricator.wikimedia.org/T227584 (10zeljkofilipin) [12:47:05] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Ruby: mediawiki_api Ruby gem incorrectly assumes path to index.php - https://phabricator.wikimedia.org/T149169 (10zeljkofilipin) [12:47:18] 10Release-Engineering-Team-TODO (201907), 10Ruby, 10User-zeljkofilipin: mediawiki_api Ruby gem improvements - https://phabricator.wikimedia.org/T227584 (10zeljkofilipin) [12:47:20] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Documentation, 10Ruby: Improve mediawiki_api documentation with inline yard - https://phabricator.wikimedia.org/T102726 (10zeljkofilipin) [12:47:32] 10Release-Engineering-Team-TODO (201907), 10Ruby, 10User-zeljkofilipin: mediawiki_api Ruby gem improvements - https://phabricator.wikimedia.org/T227584 (10zeljkofilipin) [12:47:34] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Ruby: Ruby mediawiki_api client hides error details - https://phabricator.wikimedia.org/T127786 (10zeljkofilipin) [12:47:45] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Ruby, and 2 others: mediawiki_api gem recursion on log_in - https://phabricator.wikimedia.org/T111133 (10zeljkofilipin) [12:47:47] 10Release-Engineering-Team-TODO (201907), 10Ruby, 10User-zeljkofilipin: mediawiki_api Ruby gem improvements - https://phabricator.wikimedia.org/T227584 (10zeljkofilipin) [12:51:23] zeljkof: shouldn't we just drop the mediawiki_api ruby gem ? :] [12:51:40] 10Release-Engineering-Team-TODO (201907), 10Ruby, 10User-zeljkofilipin: mediawiki_api Ruby gem improvements - https://phabricator.wikimedia.org/T227584 (10zeljkofilipin) [12:51:42] I mean just decline all the subtasks from https://phabricator.wikimedia.org/T227584 [12:51:54] hashar: my thoughts exactly :) https://phabricator.wikimedia.org/T227584 [12:52:07] oh [12:52:14] that is becoming meta meta [12:52:19] I'm not sure what the agreement on that is, feel free to decline all tasks [12:52:29] I wanted to bring it up during team meeting [12:53:09] we're tracking a few of those tasks, we should at least not track them (in releng team projects) and preferably close them [12:53:12] * zeljkof shrugs [12:54:11] replied [12:54:12] 10Release-Engineering-Team-TODO (201907), 10Ruby, 10User-zeljkofilipin: mediawiki_api Ruby gem improvements - https://phabricator.wikimedia.org/T227584 (10hashar) I have lost track of the status of the migration from Ruby to NodeJS based browser tests. One sure thing the effort started a couple years ago al... [12:57:44] (03CR) 10Urbanecm: "> Patch Set 2: Code-Review+2" [integration/config] - 10https://gerrit.wikimedia.org/r/521465 (owner: 10Urbanecm) [12:59:45] zeljkof: to rephrase from my reply: we should just DROP ruby based jobs entirely :] [12:59:55] hashar: agreed [12:59:59] there's a task for that [13:00:04] then decline everything related to it [13:00:16] shred ruby / mediawiki_selenium entirely from ci [13:00:22] and move along ;-] [13:00:45] +1 [13:23:02] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) [13:27:26] 10Release-Engineering-Team-TODO (201907), 10Reading-Infrastructure-Team-Backlog, 10Wikimedia-Site-requests, 10Epic, 10Patch-For-Review: Sunset Wikipedia Zero - https://phabricator.wikimedia.org/T187716 (10ema) [13:27:29] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) [13:27:33] hashar hi, i managed to get gerrit to build in the docker image by running "HOME=/opt/bazel bazel build --incompatible_string_join_requires_strings=false --disk_cache=/opt/bazel/disk-cache --repository_cache=/opt/bazel/repository-cache --extra_toolchains=@bazel_tools//tools/python:autodetecting_toolchain_nonstrict release" [13:31:23] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) [13:36:35] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) [13:40:49] paladox: yeah I noticed your message earlier :] [13:41:05] paladox: though all the command line settings aare in the gerrit repo in a .bazelrc file [13:41:10] so we should not need to pass them :] [13:41:23] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) 05Stalled→03Open [13:41:31] paladox: which container have you been using? releng/bazel? [13:41:47] hashar releng/gerrit [13:42:00] apart from we need to set HOME as gerrit uses ~ [13:42:07] yeah [13:43:04] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) [13:43:07] hashar: not seen this one before: ! [remote rejected] HEAD -> refs/for/master%topic=521268-patch7 (internal server error: Error inserting change/patchset) [13:43:10] paladox: I will try again :] [13:43:12] next attempt at `git review` worked OK tho [13:43:17] ok :) [13:43:25] point taken on the need of --extra_toolchains=@bazel_tools//tools/python:autodetecting_toolchain_nonstrict release" [13:43:41] yeh, we will need to switch it to python3 in that container by default [13:43:50] that flag will be removed sooner or later :) [13:43:51] note that in https://gerrit.wikimedia.org/r/#/c/integration/config/+/493328/26/dockerfiles/gerrit/run.sh I have set HOME as well [13:43:54] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) [13:44:09] ah [13:44:19] hashar that didn't work, as bazel need's to be in a WORKSPACE [13:44:28] and apparently we dont need python3 , but I should build it again [13:44:29] so we need to clone gerrit first then cd to the gerrit dir [13:44:37] (03CR) 10Hashar: Gerrit: Add CI for operations/software/gerrit (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/493328 (https://phabricator.wikimedia.org/T189549) (owner: 10Paladox) [13:44:51] well [13:45:10] the example-run.sh script does a clone of our gerrit.git to a directory ./src [13:45:21] then it does docker run -v "$(pwd)/src":/src [13:45:30] oh [13:45:33] and the entry point does a "cd /src" # https://gerrit.wikimedia.org/r/#/c/integration/config/+/493328/26/dockerfiles/gerrit/run.sh [13:45:35] ;] [13:45:56] with a few parameters passed to bazel to override options set in gerrit.git:.bazelrc [13:46:06] How do you get example-run.sh to run? [13:46:06] so that seems to work more or less, or at least it did yesterday [13:46:14] i only managed to get run.sh to run [13:46:34] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) [13:46:42] hmm [13:46:54] paladox: cd dockerfiles/gerrit && ./example-run.sh [13:46:55] ? [13:47:01] oh! [13:47:14] kostajh: no idea :-\ something went wrong on gerrit side [13:47:14] thought there was some magical docker command you had to run :) [13:47:28] kostajh: you "might" be able to find some trace in logstash [13:50:52] [2019-07-09 13:40:37,282] [ReceiveCommits-5] ERROR com.google.gerrit.server.git.receive.ReceiveCommits : [mediawiki/core-1562679632610-9ea74edf]Can't insert change/patch set for mediawiki/core [13:50:52] com.google.gerrit.extensions.restapi.RestApiException: Error inserting change/patchset [13:50:53] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) [13:51:08] com.google.gerrit.server.git.LockFailureException: Update aborted with one or more lock failures: PackedBatchRefUpdate[ [13:51:09] CREATE: 0000000000000000000000000000000000000000 b3ba2ea4e7e330277b1c96c532fd75ccba68cddd refs/changes/68/521268/10 (REJECTED_OTHER_REASON: transaction aborted) [13:51:09] UPDATE: d6f157dcfaac7f20fbe550c3318eb741ad78736e 5c3e91ee467b4e5c3dbbbcd924c47365d0b210a6 refs/changes/68/521268/meta (LOCK_FAILURE) [13:51:15] you hit a lock failure [13:51:16] kostajh: you got victim of some kind of lock failure :\ [13:51:21] (03PS3) 10Jforrester: jjb: Bump composer-package to latest [integration/config] - 10https://gerrit.wikimedia.org/r/521422 (https://phabricator.wikimedia.org/T218719) (owner: 10Daimona Eaytoy) [13:51:31] (03CR) 10Jforrester: [C: 03+2] jjb: Bump composer-package to latest [integration/config] - 10https://gerrit.wikimedia.org/r/521422 (https://phabricator.wikimedia.org/T218719) (owner: 10Daimona Eaytoy) [13:53:47] (03Merged) 10jenkins-bot: jjb: Bump composer-package to latest [integration/config] - 10https://gerrit.wikimedia.org/r/521422 (https://phabricator.wikimedia.org/T218719) (owner: 10Daimona Eaytoy) [13:54:50] 10Continuous-Integration-Config, 10phan-taint-check-plugin, 10Patch-For-Review: Upgrade php-ast to 1.0.1 in CI composer-package containers - https://phabricator.wikimedia.org/T218719 (10Jdforrester-WMF) 05Open→03Resolved [13:54:52] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201907), 10phan-taint-check-plugin, 10Patch-For-Review: Upgrade php-ast in Seccheck CI containers - https://phabricator.wikimedia.org/T227385 (10Jdforrester-WMF) [13:55:47] (03CR) 10Jforrester: "Can you make a follow-up patch moving this to PHP72 once we've eliminated 1.x seccheck?" [integration/config] - 10https://gerrit.wikimedia.org/r/521309 (https://phabricator.wikimedia.org/T227385) (owner: 10Daimona Eaytoy) [13:55:54] thcipriani: got a second? would you mind looking at https://gerrit.wikimedia.org/r/c/integration/config/+/520437 again and maybe merging it? I think according to your comment it will do what we want and I really need those images :) [13:56:13] (03PS3) 10Jforrester: docker: Install php-ast 1.0.0 for seccheck, back to PHP7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/521309 (https://phabricator.wikimedia.org/T227385) (owner: 10Daimona Eaytoy) [13:56:23] (03CR) 10Jforrester: [C: 03+2] docker: Install php-ast 1.0.0 for seccheck, back to PHP7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/521309 (https://phabricator.wikimedia.org/T227385) (owner: 10Daimona Eaytoy) [13:56:41] (03CR) 10Daimona Eaytoy: "> Can you make a follow-up patch moving this to PHP72 once we've" [integration/config] - 10https://gerrit.wikimedia.org/r/521309 (https://phabricator.wikimedia.org/T227385) (owner: 10Daimona Eaytoy) [13:57:42] 10Continuous-Integration-Config, 10phan-taint-check-plugin, 10Patch-For-Review: Upgrade php-ast to 1.0.1 in CI composer-package containers - https://phabricator.wikimedia.org/T218719 (10Daimona) Aye, thanks! https://integration.wikimedia.org/ci/job/composer-package-php70-docker/6421/console is our first work... [13:57:58] (03Merged) 10jenkins-bot: docker: Install php-ast 1.0.0 for seccheck, back to PHP7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/521309 (https://phabricator.wikimedia.org/T227385) (owner: 10Daimona Eaytoy) [13:59:27] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) [14:01:22] Gah, I'm an idiot. [14:05:54] (03PS1) 10Jforrester: Follow-up I8dae352d49: Add a changelog entry for mediawiki-phan-seccheck so it's picked up [integration/config] - 10https://gerrit.wikimedia.org/r/521502 [14:06:07] (03CR) 10Jforrester: [C: 03+2] Follow-up I8dae352d49: Add a changelog entry for mediawiki-phan-seccheck so it's picked up [integration/config] - 10https://gerrit.wikimedia.org/r/521502 (owner: 10Jforrester) [14:06:08] PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): File[/srv/deployment-charts/helmfile.d/services/staging/graphoid/.hfenv],File[/srv/deployment-charts/helmfile.d/services/eqiad/graphoid/.hfenv],File[/srv/deployment-charts/helmfile.d/services/codfw/graphoid/.hfenv] [14:07:48] 10Gerrit, 10Documentation: Update Gerrit documentation on mediawiki.org before upgrading to Gerrit 2.16.x / PolyGerrit UI - https://phabricator.wikimedia.org/T227562 (10Paladox) I've updated a few bit's see But overall it has images for PolyGerrit & GWTUI on the page. It even refers to switching back and four... [14:08:04] (03Merged) 10jenkins-bot: Follow-up I8dae352d49: Add a changelog entry for mediawiki-phan-seccheck so it's picked up [integration/config] - 10https://gerrit.wikimedia.org/r/521502 (owner: 10Jforrester) [14:08:43] !log Pushing mediawiki-phan-seccheck:0.3.6 docker image. [14:08:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:08:54] 10Release-Engineering-Team-TODO (201907), 10Release Pipeline, 10Core Platform Team (Extension Management (TEC13)), 10MW-1.34-notes (1.34.0-wmf.8; 2019-06-04), 10Patch-For-Review: Determine a standard way of installing MediaWiki lib/extension dependencies within c... - https://phabricator.wikimedia.org/T193824 [14:10:45] * James_F sighs. [14:23:24] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) [14:31:38] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): File[/srv/deployment-charts/helmfile.d/services/staging/graphoid/.hfenv],File[/srv/deployment-charts/helmfile.d/services/eqiad/graphoid/.hfenv],File[/srv/deployment-charts/helmfile.d/services/codfw/graphoid/.hfenv] [14:36:27] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) [14:36:43] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Gem::InstallError: watir requires Ruby version >= 2.3.0. - https://phabricator.wikimedia.org/T214382 (10zeljkofilipin) a:03zeljkofilipin [14:37:08] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201907), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Gem::InstallError: watir requires Ruby version >= 2.3.0. - https://phabricator.wikimedia.org/T214382 (10zeljkofilipin) [14:37:48] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Drop Ruby Selenium CI jobs; we don't support them any more - https://phabricator.wikimedia.org/T220035 (10zeljkofilipin) 05Stalled→... [14:38:10] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201907), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Drop Ruby Selenium CI jobs; we don't support them any more - https://phabricator.wikimedia.org/T220035 (10zeljkofilipin) [14:39:03] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201907), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Drop Ruby Selenium CI jobs; we don't support them any more - https://phabricator.wikimedia.org/T220035 (10zeljkofilipin) a:... [14:39:32] (03PS1) 10Kosta Harlan: Don't run phpunit-unit stage if the composer script doesn't exist [integration/quibble] - 10https://gerrit.wikimedia.org/r/521515 (https://phabricator.wikimedia.org/T87781) [14:40:01] (03CR) 10Kosta Harlan: "I could use some help with the unit test." [integration/quibble] - 10https://gerrit.wikimedia.org/r/521515 (https://phabricator.wikimedia.org/T87781) (owner: 10Kosta Harlan) [14:40:13] (03CR) 10jerkins-bot: [V: 04-1] Don't run phpunit-unit stage if the composer script doesn't exist [integration/quibble] - 10https://gerrit.wikimedia.org/r/521515 (https://phabricator.wikimedia.org/T87781) (owner: 10Kosta Harlan) [14:40:40] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) [14:41:30] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) [14:42:40] 10Release-Engineering-Team-TODO (201907), 10MediaViewer, 10Multimedia, 10Browser-Tests, and 3 others: MediaViewer selenium tests are failing with "Misconfigured -- Unsupported OS/browser/version/device combo" because our target MacOS has aged out - https://phabricator.wikimedia.org/T214389 (10zeljkofilipin... [14:43:41] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) [14:47:06] Pchelolo: yep, will merge [14:47:07] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10Jdlrobson) Is this ticket about getting rid of Ruby or porting all tes... [14:47:17] thank you thcipriani [14:48:48] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [14:52:34] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [14:53:20] (03PS5) 10Thcipriani: Build and publish RESTRouter image. [integration/config] - 10https://gerrit.wikimedia.org/r/520437 (https://phabricator.wikimedia.org/T226536) (owner: 10Ppchelko) [14:56:22] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10Readers-Web-Backlog, 10User-zeljkofilipin, 10Wikimedia-Incident: Run subset of MobileFrontend browser tests on merges in core - https://phabricator.wikimedia.org/T165940 (10zeljkofilipin) Random patch: https://ge... [14:57:05] (03CR) 10Thcipriani: [C: 03+2] Build and publish RESTRouter image. [integration/config] - 10https://gerrit.wikimedia.org/r/520437 (https://phabricator.wikimedia.org/T226536) (owner: 10Ppchelko) [14:57:31] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201907), 10Readers-Web-Backlog, 10User-zeljkofilipin, 10Wikimedia-Incident: Run subset of MobileFrontend browser tests on merges in core - https://phabricator.wikimedia.org/T165940 (10zeljkofilipin) [14:58:43] (03Merged) 10jenkins-bot: Build and publish RESTRouter image. [integration/config] - 10https://gerrit.wikimedia.org/r/520437 (https://phabricator.wikimedia.org/T226536) (owner: 10Ppchelko) [15:01:56] Oh, oops, sorry Pchelolo, meant to do that last week and forgot. [15:02:34] James_F: no problem :) I was fixing issues with it here and there, last week it probably would not have succeeded [15:02:48] Psh, details. :-) [15:03:36] !log reloading zuul to deploy https://gerrit.wikimedia.org/r/520437 [15:03:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:03:57] ^ Pchelolo should be live now [15:04:18] great! thank you thcipriani. Lemme try making the first image [15:04:45] 10Deployments, 10TechCom, 10I18n, 10Language-Team (Language-2019-July-September): Determine desired architecture to update localization strings for Wikimedia - https://phabricator.wikimedia.org/T206694 (10Pginer-WMF) [15:06:43] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201907), 10Readers-Web-Backlog, 10User-zeljkofilipin, 10Wikimedia-Incident: Run subset of MobileFrontend browser tests on merges in core - https://phabricator.wikimedia.org/T165940 (10zeljkofilipin) @Jdlrobson look... [15:08:10] thcipriani: can you help me a bit more? looks familiar: https://integration.wikimedia.org/ci/job/service-pipeline-test-and-publish/376/console ? [15:10:32] hrm, that's interesting [15:11:05] there's something about this error at T187897 but I have not idea how related it is [15:11:06] T187897: fatal: remote error: access denied or repository not exported: /mediawiki/extensions/ReadingLists - https://phabricator.wikimedia.org/T187897 [15:13:43] in this instance there is a /srv/zuul/git/mediawiki/services/restbase/deploy that appears to be a git repo, but the parent directory is not part of that repo afaict. [15:14:39] thcipriani: there's 2 repos, like with all the rest of the services [15:14:53] mediawiki/services/restbase - source, mediawiki/services/restbase/deploy - deploy repo [15:16:55] yeah, I expected to see some failure for zuul merger in the logs, but it looks normal [15:18:24] ah, no wait, I do see something: https://phabricator.wikimedia.org/P8731 [15:19:32] you'd think this would've been a problem for all the service repos [15:20:06] ye, how all the rest that were switched to pipeline worked? [15:20:35] https://phabricator.wikimedia.org/P8731#52499 [15:20:46] unless the deploy repo was created after ¯\_(ツ)_/¯ [15:22:14] so, here I'm all in your hands, I have no idea what's going on there.. [15:22:17] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201907), 10Readers-Web-Backlog, 10User-zeljkofilipin, 10Wikimedia-Incident: Run subset of MobileFrontend browser tests on merges in core - https://phabricator.wikimedia.org/T165940 (10Jdlrobson) 05Open→03Resolve... [15:23:35] Pchelolo: also new to me, but I'm fiddling [15:24:57] seems to be this one: https://phabricator.wikimedia.org/T157818 [15:26:45] thcipriani: oh exactly... would you mind doing the workaround? I don't have ssh access to those machines [15:27:01] yep, doing [15:28:15] 10Project-Admins: Create tag "wikibook" in phabricator - https://phabricator.wikimedia.org/T227158 (10martin.monperrus) The `wikibooks` tag will be used by users of the wikibooks platform who find bugs in it. [15:29:39] !log contint{1,2}001: rm -rf /srv/zuul/git/mediawiki/services/restbase due to T157818 [15:29:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:29:41] T157818: zuul-merger fails when repository names overlaps - https://phabricator.wikimedia.org/T157818 [15:29:44] Pchelolo: give it another shot [15:30:46] may have to push a -wmf.1 tag or something :\ [15:31:40] thcipriani: all right, it moved further :) [15:32:08] thank you. Let's hope... [15:32:45] :) [15:33:50] PROBLEM - Disk space on contint1001 is CRITICAL: DISK CRITICAL - /mnt/docker/overlay2/843610a04cd773214d6795a2696dcc669b539c8092219d1b436423a82698249e/merged is not accessible: Permission denied https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space [15:35:14] 10Release-Engineering-Team (Code Health), 10Developer-Advocacy, 10Code-Health: Launch the Wikimedia "Code Review Special Interest Group" - https://phabricator.wikimedia.org/T129842 (10Aklapper) For the records, this is kind of happening. See https://www.mediawiki.org/wiki/Code_Health_Group/projects/CodeReview [15:45:53] 10Continuous-Integration-Config, 10Release Pipeline, 10serviceops-radar, 10Core Platform Team (RESTBase Split (CDP2)), 10Services (done): Trigger RESTRouter image builds on push/tag - https://phabricator.wikimedia.org/T226536 (10Pchelolo) 05Open→03Resolved a:03Pchelolo This has been done and we've... [15:48:23] Pchelolo: magic! https://tools.wmflabs.org/dockerregistry/wikimedia/mediawiki-services-restbase/tags/ [15:49:04] up. Thank you thcipriani [15:49:10] one step closer to the goal :) [15:49:18] :) [15:50:12] (03PS1) 10Jforrester: dockerfiles: [php-ast] Add ast 0.1.2; [phan-seccheck] Inherit from php-ast [integration/config] - 10https://gerrit.wikimedia.org/r/521524 (https://phabricator.wikimedia.org/T227385) [15:50:26] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [php-ast] Add ast 0.1.2; [phan-seccheck] Inherit from php-ast [integration/config] - 10https://gerrit.wikimedia.org/r/521524 (https://phabricator.wikimedia.org/T227385) (owner: 10Jforrester) [15:51:00] 10Release-Engineering-Team (Deployment services), 10Operations, 10SRE-Access-Requests: Request access to deployment cluster for Jakob_WMDE - https://phabricator.wikimedia.org/T227193 (10Jakob_WMDE) @MoritzMuehlenhoff here is the public key: `ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAENSef8ugACShtUryIYbII3C0bwJ8D... [15:51:58] (03Merged) 10jenkins-bot: dockerfiles: [php-ast] Add ast 0.1.2; [phan-seccheck] Inherit from php-ast [integration/config] - 10https://gerrit.wikimedia.org/r/521524 (https://phabricator.wikimedia.org/T227385) (owner: 10Jforrester) [15:52:32] 10Gerrit: Enable sandbox branches in gerrit - https://phabricator.wikimedia.org/T227159 (10MarkAHershberger) >>! In T227159#5312322, @hashar wrote: > Ah indeed. So `All-Projects.git` has: > ` > [access "refs/heads/sandbox/${username}/*"] > create = group Registered Users > push = group Registered Users... [15:53:54] !log Pushing docker images php-ast:1.0.1-2 and mediawiki-phan-seccheck:0.3.6 [15:53:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:54:45] 10Gerrit: Enable sandbox branches in gerrit - https://phabricator.wikimedia.org/T227159 (10MarkAHershberger) >>! In T227159#5313336, @Paladox wrote: > @MarkAHershberger i had to create the account manually using createAndPromote.php, i've sent the details through email for logging in. Thanks. Trying now. I'll... [15:55:33] (03PS1) 10Daimona Eaytoy: Revert "layout: [SecurityCheckPlugin] Revert blocking jobs to php70; move php71+ to experimental" [integration/config] - 10https://gerrit.wikimedia.org/r/521526 [15:56:15] (03PS2) 10Kosta Harlan: [quibble-coverage] Fix code coverage generation for core codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/521190 [15:57:15] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10zeljkofilipin) >>! In T139740#5317419, @Jdlrobson wrote: > Is this tic... [15:57:41] (03CR) 10jerkins-bot: [V: 04-1] [quibble-coverage] Fix code coverage generation for core codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/521190 (owner: 10Kosta Harlan) [16:02:12] (03PS1) 10Jforrester: jjb: [mediawiki-core-php70-phan-seccheck-docker] Point to forwards-compatible docker image [integration/config] - 10https://gerrit.wikimedia.org/r/521529 [16:02:14] (03PS1) 10Jforrester: jjb: [mwext-php70-phan-seccheck-docker] Point to forwards-compatible docker image [integration/config] - 10https://gerrit.wikimedia.org/r/521530 [16:03:30] RECOVERY - Disk space on contint1001 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space [16:15:07] (03PS2) 10Jforrester: jjb: [mediawiki-core-php70-phan-seccheck-docker] Point to forwards-compatible docker image [integration/config] - 10https://gerrit.wikimedia.org/r/521529 [16:15:10] (03PS1) 10Jforrester: dockerfiles: [phan-seccheck] Pass $AST_ARG not AST_ARG [integration/config] - 10https://gerrit.wikimedia.org/r/521533 [16:15:11] (03PS1) 10Jforrester: jjb: [mwext-php70-phan-seccheck-docker] Point to forwards-compatible docker image [integration/config] - 10https://gerrit.wikimedia.org/r/521534 [16:15:28] (03Abandoned) 10Jforrester: jjb: [mwext-php70-phan-seccheck-docker] Point to forwards-compatible docker image [integration/config] - 10https://gerrit.wikimedia.org/r/521530 (owner: 10Jforrester) [16:15:38] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [phan-seccheck] Pass $AST_ARG not AST_ARG [integration/config] - 10https://gerrit.wikimedia.org/r/521533 (owner: 10Jforrester) [16:17:08] 10Release-Engineering-Team-TODO (201907), 10MediaWiki-Uploading, 10Multimedia, 10Patch-For-Review: .odb file recognized as application/acad (first type entry in our list) - https://phabricator.wikimedia.org/T35515 (10hashar) 05Stalled→03Resolved [16:17:18] (03Merged) 10jenkins-bot: dockerfiles: [phan-seccheck] Pass $AST_ARG not AST_ARG [integration/config] - 10https://gerrit.wikimedia.org/r/521533 (owner: 10Jforrester) [16:17:44] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review: Rebuild integration-slave-docker-* instances to use less RAM, new name and Stretch - https://phabricator.wikimedia.org/T226233 (10hash... [16:18:30] Shame. :-) [16:18:54] !log Push docker image +mediawiki-phan-seccheck:0.3.6-1 [16:18:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:29:37] 10Project-Admins: Create tag "wikibook" in phabricator - https://phabricator.wikimedia.org/T227158 (10Aklapper) @martin.monperrus: Please see my previous comment again: "It says: explaining **how** these tags will be used and by whom." Which specific "users" have expressed interest in this? Creating a place whe... [16:42:19] Why why why did I choose to run phan-seccheck locally for a test on VE, one of the largest extensions, and not SpamBlacklist or something simpler? [16:45:32] Eheh, good luck [16:45:57] Anywawy, with my latest fiddles I've got `docker run --volume /tmp:/src --volume /tmp:/cache --volume /tmp:/srv/git --env THING_SUBNAME="extensions/SpamBlacklist" --env SECCHECK_VERSION="1.5.0" --volume /Users/jdforrester/Documents/code/vagrant/mediawiki:/mediawiki --init --rm docker-registry.wikimedia.org/releng/mediawiki-phan-seccheck:0.3.6-2 -m checkstyle` working. [16:46:03] So… yay. [16:47:00] And does it work with SECCHECK_VERSION="2.0.0", too? [16:47:24] Yes. [16:47:37] Now testing without passing an explicit version at all. [16:48:20] Great, thanks [16:48:44] (My poor laptop sounds like it wants to take off.) [16:51:32] (03PS1) 10Jforrester: dockerfiles: [phan-seccheck] Pass $AST_ARG for the composer install step, too [integration/config] - 10https://gerrit.wikimedia.org/r/521542 [16:52:09] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: Upgrade to Gerrit 2.16.10 - https://phabricator.wikimedia.org/T200739 (10MarcoAurelio) Do we have any remaining blockers left to move forward? If not, any estimate time for when this will be do... [16:52:17] Daimona: BTW, setting to 2.0.0 massively reduces the amount of code studied (from 1502MB to 178MB) – not sure what's getting skipped? [16:52:35] Dang [16:52:45] What's the command used to invoke it? [16:52:49] Nothing obvious in the list. [16:53:01] Err. [16:53:31] `php -dextension=ast_1.0.0.so /opt/phan/vendor/phan/phan/phan -d . -k /opt/phan/vendor/mediawiki/phan-taint-check-plugin/scripts/mwext-fast-config.php --output php://stdout -m checkstyle` [16:53:41] I guess it's skipping vendor? [16:53:44] Thanks [16:53:46] Could be [16:53:56] (Which is correct, I think?) [16:54:05] Or maybe the old number was the amount of *parsed* code, and the new one is the amount of *analyzed* code [16:54:11] But hey, it passes and it unlocks 2.0.0. :-) [16:54:13] Maybe it was changed upstream [16:54:19] But yeah, that's great [16:54:25] Could you please run one last test? [16:54:31] Sure, what? [16:54:35] On CheckUser [16:54:39] I know what the output should be [16:55:03] https://phabricator.wikimedia.org/T227171#5306248 [16:55:09] Or anything else in the list [16:55:55] I'm only testing against ast 1.0.0, is that OK? [16:56:25] PROBLEM - Disk space on contint1001 is CRITICAL: DISK CRITICAL - /mnt/docker/overlay2/42082d324583d16691c4e0ddaf2572716f99740ea22f6ad23517419a3a3e3356/merged is not accessible: Permission denied https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space [16:58:04] Adding php7.2/ast1.0.1 to the image is tomorrow's problem. [16:59:19] RECOVERY - Disk space on contint1001 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space [16:59:31] Hmm. [16:59:43] Daimona: I get: `(Phan 1.3.2 crashed when parsing/analyzing 'includes/CheckUserLogPager.php')` [17:00:00] Eeeeeeek [17:00:14] Why's this so bad? [17:01:21] Sorry, to clarify, it works using ast 0.1.2 with CheckUser. [17:01:44] ast 0.1.2 + seccheck 1.5 is the current setup [17:01:56] So IIUC the php-ast container doesn't provide ast for php72? [17:02:12] And ast 1.0.0 + seccheck 2.0.0 on CheckUser crashes, right? [17:02:28] It does, but we're not inheriting it yet because you just butchered the image to remove php72. ;-) [17:03:22] Oh :/ [17:03:28] Yeah I'll submit a follow-up [17:03:34] So master of CU (ast 0.1.2 + `"phan-taint-check-plugin": "2.0.0"`) passes, but if I explcitly tell it to run SECCHECK_VERSION="2.0.0" it does. [17:03:43] Err. It dies. [17:03:57] Right now I'm screaming against https://gerrit.wikimedia.org/r/#/c/mediawiki/tools/phan/SecurityCheckPlugin/+/521500/ [17:04:09] Even with ast 1.0.0? [17:04:21] With 1.0.0, yes. [17:04:53] ... [17:05:05] Could you please try ast 1.0.1 with php>=7.1? [17:05:14] Umm. [17:05:32] I can try, but that will create an image that's not backwards-compatible, AIUI? [17:05:56] Aaaaaand composer-package-72 doesn't have ast https://integration.wikimedia.org/ci/job/composer-package-php72-docker/6914/console [17:06:01] Oh well, then no [17:06:19] Is there a stacktrace for the phan crash? [17:06:27] Yeah. [17:06:58] https://www.irccloud.com/pastebin/VxrhQhSi/ [17:07:00] I lost count of the new issues I just discovered [17:07:15] Got it [17:07:22] Apparently it's not parsing mw core [17:07:35] It complains about Database::select not existing [17:07:46] Oh, so is that something I'm doing wrong locally? [17:08:05] I don't think so [17:08:09] Let me try to run it locally [17:08:24] Oh actually [17:08:27] First thing [17:08:42] Is MW_INSTALL_PATH set up properly? [17:09:01] RecursiveDirectoryIterator::__construct(../..includes/): failed to open dir: No such file or directory [17:09:14] This means it expects MW to be at ../.., but cannot find it [17:09:24] Oh no [17:09:27] A missing slash I see [17:11:39] Yep [17:11:59] I'll include that in seccheck 2.0.1, which should then be the version to upgrade to [17:12:18] (And this also explains why the amount of analyzed code dropped so badly) [17:12:51] Ha. :-) [17:13:32] OK, so I can push the image and re-point the jobs, but we can't bulk-upgrade all repos until 2.0.1 is released? [17:13:39] Plan is: [17:13:54] * Get CI infrastructure able to cope with 2.0.x [17:14:01] * Migrate all repos to 2.0.x [17:14:18] Yes [17:14:24] * Move CI infrastructure to PHP 7.2 [17:14:28] Right? [17:14:54] Though I have to say everything for 2.0.1 is almost ready. We just need to get jenkins pass in the seccheck repo, and everything else is already merged [17:15:06] Yep, where x = 1 [17:15:15] OK, I'l do this task now and that task next. [17:15:22] x hopefully = 1. ;-) [17:16:42] Great, thanks [17:17:00] And of note, jenkins passing in our repo is 1 failing test away [17:17:04] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [phan-seccheck] Pass $AST_ARG for the composer install step, too [integration/config] - 10https://gerrit.wikimedia.org/r/521542 (owner: 10Jforrester) [17:17:38] And https://gerrit.wikimedia.org/r/#/c/integration/config/+/521526/ would be good too, once PHP72 gets ast correctly [17:19:13] (03Merged) 10jenkins-bot: dockerfiles: [phan-seccheck] Pass $AST_ARG for the composer install step, too [integration/config] - 10https://gerrit.wikimedia.org/r/521542 (owner: 10Jforrester) [17:20:49] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10CirrusSearch, and 4 others: Add a PHPUnit group to skip test on gated CI runs - https://phabricator.wikimedia.org/T225068 (10Gehel) [17:21:19] !log Push docker mediawiki-phan-seccheck:0.3.6-2 [17:21:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:26:04] Tahdah, it works: https://integration.wikimedia.org/ci/job/mwext-php70-phan-seccheck-docker/62611/console [17:26:49] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/521534 (owner: 10Jforrester) [17:26:54] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/521529 (owner: 10Jforrester) [17:29:43] (03Merged) 10jenkins-bot: jjb: [mediawiki-core-php70-phan-seccheck-docker] Point to forwards-compatible docker image [integration/config] - 10https://gerrit.wikimedia.org/r/521529 (owner: 10Jforrester) [17:29:49] (03Merged) 10jenkins-bot: jjb: [mwext-php70-phan-seccheck-docker] Point to forwards-compatible docker image [integration/config] - 10https://gerrit.wikimedia.org/r/521534 (owner: 10Jforrester) [17:30:52] OK, so. [17:31:47] Something working! Wow! [17:32:26] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201907), 10phan-taint-check-plugin: Upgrade php-ast in Seccheck CI containers - https://phabricator.wikimedia.org/T227385 (10Jdforrester-WMF) Status update: CI production is now ready to take phan-seccheck 2.x, and this no longer blocks T22717... [17:32:53] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201907), 10phan-taint-check-plugin: Upgrade php-ast in Seccheck CI containers - https://phabricator.wikimedia.org/T227385 (10Jdforrester-WMF) [17:32:58] 10Continuous-Integration-Config, 10phan-taint-check-plugin, 10Patch-For-Review: Upgrade php-ast to 1.0.1 in CI composer-package containers - https://phabricator.wikimedia.org/T218719 (10Jdforrester-WMF) [17:33:13] Do you have any clue why PHP72 isn't getting ast? [17:33:16] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201907), 10phan-taint-check-plugin: Upgrade php-ast in Seccheck CI containers - https://phabricator.wikimedia.org/T227385 (10Jdforrester-WMF) [17:33:17] 10Continuous-Integration-Config, 10LibUp, 10phan-taint-check-plugin: Upgrade taint-check to 2.0 in all repos - https://phabricator.wikimedia.org/T227172 (10Jdforrester-WMF) [17:33:36] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201907), 10phan-taint-check-plugin: Upgrade php-ast in Seccheck CI containers - https://phabricator.wikimedia.org/T227385 (10Jdforrester-WMF) [17:33:39] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review: Run phan secheck on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T226420 (10Jdforrester-WMF) [17:33:47] It seems to be the only version which does [17:34:02] And eventually it's the one we'll need for the seccheck72 job [17:35:09] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201907), 10phan-taint-check-plugin: Upgrade php-ast in Seccheck CI containers - https://phabricator.wikimedia.org/T227385 (10Jdforrester-WMF) [17:35:11] 10Continuous-Integration-Config, 10LibUp, 10phan-taint-check-plugin: Upgrade taint-check to 2.0 in all repos - https://phabricator.wikimedia.org/T227172 (10Jdforrester-WMF) [17:35:37] Daimona: The php70 and php72 images are "real" images, not based off sury builds. [17:35:51] So that difference is probably why they don't work. [17:36:10] Alright [17:36:22] I'll dig. [17:37:54] Hmm, the 70 image gets ast but not 72. [17:39:44] Heh, yeah [17:40:24] The image has both /usr/lib/php/20151012/ast.so and /usr/lib/php/20170718/ast.so [17:40:52] Could be some typo... [17:41:20] How does PHP determine which ones to run when there are two? [17:42:15] The php71 image has four ast.so files – in /usr/lib/php/20151012 /usr/lib/php/20160303 /usr/lib/php/20170718 and /usr/lib/php/20180731 [17:42:33] Same with the php73 image. [17:43:19] I think this is intended [17:43:29] The php-ast container just installs all of them [17:43:43] Does it have the right name in -dextension=..? [17:44:02] We only pull a single ast.so file from the php-ast container, we don't inherit from it. [17:44:12] I imagine these files are inherited from the sury image. [17:44:26] There's no explicit -dextension= line, that's why I'm curious. :-) [17:45:39] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO, 10Developer-Advocacy, 10Code-Health: Decision on Wikimedia Foundation service-level agreement on code review times - https://phabricator.wikimedia.org/T113707 (10greg) [17:46:30] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10Browser-Tests, 10Jenkins: When beta cluster is down Jenkins jobs should be aborted and not trigger e-mail notifications - https://phabricator.wikimedia.org/T101563 (10greg) [17:46:44] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, and 3 others: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955 (10greg) [17:46:56] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO, 10Developer Productivity: FY201819 TEC12:O1 Outcome – Local development is unified with testing and production - https://phabricator.wikimedia.org/T222237 (10greg) [17:47:10] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO, 10Developer Productivity, 10MediaWiki-Core-Testing, and 3 others: TEC12:O1:O1.4 Goal – Support running Selenium browser tests in the docker local development environment - https://phabricator.wikimedia.org/T222234 (10greg) [17:47:12] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO, 10Developer Productivity: Improve documentation for running local-charts and building docker images using the temporary Dockerfiles - https://phabricator.wikimedia.org/T220771 (10greg) [17:47:16] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10MediaWiki-extensions-LocalisationUpdate, 10Security: Reenable l10update in production - https://phabricator.wikimedia.org/T220481 (10greg) [17:47:21] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10observability, and 2 others: Increase "check_legal_html" coverage to group0 wikis - https://phabricator.wikimedia.org/T208284 (10greg) [17:47:26] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Keyholder, 10Operations: Keyholder phab repo duplicate work - https://phabricator.wikimedia.org/T203003 (10greg) [17:48:00] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Education-Program-Dashboard, 10MediaWiki-extensions-EducationProgram, 10Epic: Deprecate and remove the EducationProgram extension from Wikimedia servers after June 30, 2018 - https://phabricator.wikimedia.org/T125618 (10g... [17:48:05] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10scap2, 10Patch-For-Review: Eliminate symlinks in mediawiki-config (as much as possible) - https://phabricator.wikimedia.org/T126306 (10greg) [17:48:11] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201907), 10phan-taint-check-plugin: Upgrade php-ast in Seccheck CI containers - https://phabricator.wikimedia.org/T227385 (10Jdforrester-WMF) 05Open→03Stalled a:03Jdforrester-WMF [17:48:14] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review: Run phan secheck on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T226420 (10Jdforrester-WMF) [17:48:18] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10releng-201617-q2, 10Documentation: Document workflow and creation of CI jobs in Differential - https://phabricator.wikimedia.org/T130952 (10greg) 05Stalled→03Declined [17:48:37] Ah alright [17:48:43] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10observability: "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) - https://phabricator.wikimedia.org/T141520 (10greg) [17:48:45] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO: Add #Wikimedia-Incident to all open "actionables" in past incident reports - https://phabricator.wikimedia.org/T141493 (10greg) [17:48:48] I don't really know how it works :/ [17:48:51] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10Core Platform Team Backlog (Later), and 2 others: Review new service 'pre-deployment to production' checklist - https://phabricator.wikimedia.org/T141897 (10greg) [17:48:56] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Contributors-Team, 10Operations, and 2 others: High failure rate of account creation should trigger an alarm / page people - https://phabricator.wikimedia.org/T146090 (10greg) [17:49:02] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10observability, and 2 others: Tracking: Monitoring and alerts for "business" metrics - https://phabricator.wikimedia.org/T140942 (10greg) [17:49:07] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10Ruby: mediawiki_api Ruby gem incorrectly assumes path to index.php - https://phabricator.wikimedia.org/T149169 (10greg) [17:49:14] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Wikimedia-General-or-Unknown, 10Technical-Debt: mediawiki-config: Try and simplify/cleanup the pile of symlinks - https://phabricator.wikimedia.org/T156319 (10greg) [17:50:58] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests: Consider to switch frrwiki from group2 to group1 per Proofreadpage - https://phabricator.wikimedia.org/T166263 (10greg) >>! In T166263#3293610, @greg wrote: > Understood. So, a follow-on question wit... [17:51:25] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10User-Joe: [DRAFT][RfC] Deployment of python applications in production - https://phabricator.wikimedia.org/T180023 (10greg) [17:51:32] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10MediaWiki-Maintenance-scripts, 10Utilities-code-utils: Write some version of foreachwiki(indblist) that respects replag and/or has some --delay parameter between wikis - https://phabricator.wikimedia.org/T187852 (10greg) [17:51:48] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10User-Zppix: Jenkins is posting duplicate comments when a depended on change merge fails - https://phabricator.wikimedia.org/T215511 (10greg) [17:52:04] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10MediaWiki-General-or-Unknown, 10TechCom, and 2 others: Drop PHP 7.0 support from MediaWiki - https://phabricator.wikimedia.org/T216165 (10greg) [17:56:23] Oh, I see. [18:01:08] Daimona: ^^ Minor detail. :-) [18:01:19] Bah. [18:01:35] What was wrong? [18:01:44] (03PS1) 10Jforrester: dockerfiles: [composer-package-php72] Actually load the php-ast extension [integration/config] - 10https://gerrit.wikimedia.org/r/521554 [18:01:50] The "load every module in sight" code from sury isn't loaded. [18:02:04] So we didn't have the "actually load it" config. :-) [18:02:06] Duh. [18:02:13] 10MediaWiki-Releasing: Review 1.33.1 tarball - https://phabricator.wikimedia.org/T227599 (10Aklapper) [18:02:23] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [composer-package-php72] Actually load the php-ast extension [integration/config] - 10https://gerrit.wikimedia.org/r/521554 (owner: 10Jforrester) [18:02:47] Got it [18:02:52] At least it's done now [18:02:59] And CI is passing in the seccheck repo [18:04:22] (03Merged) 10jenkins-bot: dockerfiles: [composer-package-php72] Actually load the php-ast extension [integration/config] - 10https://gerrit.wikimedia.org/r/521554 (owner: 10Jforrester) [18:07:19] Could you please give a look at https://gerrit.wikimedia.org/r/#/c/integration/config/+/521526/, once ast is fixed with PHP72? [18:08:09] re. [18:08:19] Sure, even. [18:08:31] !log Push docker composer-package-php72:0.2.2-s2 [18:08:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:10:16] With that in place I can release 2.0.1 tonight [18:10:23] Well [18:10:31] Which means in the next hour [18:11:38] (03PS1) 10markahershberger: Tarball creation [tools/release] - 10https://gerrit.wikimedia.org/r/521559 (https://phabricator.wikimedia.org/T217960) [18:11:47] And after https://gerrit.wikimedia.org/r/#/c/mediawiki/tools/phan/SecurityCheckPlugin/+/521500/ is merged [18:12:20] (03CR) 10jerkins-bot: [V: 04-1] Tarball creation [tools/release] - 10https://gerrit.wikimedia.org/r/521559 (https://phabricator.wikimedia.org/T217960) (owner: 10markahershberger) [18:14:06] (03PS1) 10Jforrester: jjb: [composer-package-php72-docker] Actually load the ast extension [integration/config] - 10https://gerrit.wikimedia.org/r/521560 [18:14:08] (03PS1) 10Jforrester: jjb: [parsoidsvc-composer-package-php72-docker] Bump to latest docker image (no-op) [integration/config] - 10https://gerrit.wikimedia.org/r/521561 [18:14:24] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/521560 (owner: 10Jforrester) [18:14:36] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/521561 (owner: 10Jforrester) [18:14:48] (03CR) 10Jforrester: [C: 03+2] Revert "layout: [SecurityCheckPlugin] Revert blocking jobs to php70; move php71+ to experimental" [integration/config] - 10https://gerrit.wikimedia.org/r/521526 (owner: 10Daimona Eaytoy) [18:15:43] Daimona: Gosh, look, an actual CI pass: https://gerrit.wikimedia.org/r/c/mediawiki/tools/phan/SecurityCheckPlugin/+/521555 :-) [18:16:42] (03Merged) 10jenkins-bot: jjb: [composer-package-php72-docker] Actually load the ast extension [integration/config] - 10https://gerrit.wikimedia.org/r/521560 (owner: 10Jforrester) [18:17:47] (03Merged) 10jenkins-bot: jjb: [parsoidsvc-composer-package-php72-docker] Bump to latest docker image (no-op) [integration/config] - 10https://gerrit.wikimedia.org/r/521561 (owner: 10Jforrester) [18:17:50] (03Merged) 10jenkins-bot: Revert "layout: [SecurityCheckPlugin] Revert blocking jobs to php70; move php71+ to experimental" [integration/config] - 10https://gerrit.wikimedia.org/r/521526 (owner: 10Daimona Eaytoy) [18:18:42] !log Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Move back to php72 as primary [18:18:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:19:58] Daimona: Anything I can do to help with the release of 2.0.1? [18:23:42] Ahhh I've never been so happy for a +2 by jenkins [18:23:57] Let me check [18:23:58] 5 minutes [18:28:36] I rebased everything in the 2.x branch, let's see [18:30:41] Daimona: Are any of them critical for 2.0.1 and can't wait for 2.0.2 or whatever? [18:31:01] They can wait, I'd only like to include them because they're already +2ed [18:31:36] But V-1'ed, it seems. :-( [18:31:38] Although one is failing now [18:31:44] Yeah and it's not spurious [18:31:48] Well I guess we can delay them [18:32:26] So let's tag 2.0.1? [18:32:36] Go for it. [18:34:32] (03PS1) 10Paladox: Update its-base plugin submodule [software/gerrit] (wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/521564 [18:34:47] 10Release-Engineering-Team-TODO (201907), 10MediaWiki-Uploading, 10Multimedia, 10MW-1.34-notes (1.34.0-wmf.13; 2019-07-09): .odb file recognized as application/acad (first type entry in our list) - https://phabricator.wikimedia.org/T35515 (10Jdforrester-WMF) [18:35:18] (03PS4) 10Jforrester: Skip MediaWiki install for composer-test job [integration/config] - 10https://gerrit.wikimedia.org/r/519644 (owner: 10Kosta Harlan) [18:35:49] 10Continuous-Integration-Config, 10Release-Engineering-Team (Code Health), 10Code-Health: Create a CI container for mediawiki/tools/dependency-analysis - https://phabricator.wikimedia.org/T227603 (10hashar) [18:36:08] (03PS5) 10Jforrester: jjb: [mediawiki-quibble-composertest-*-docker] Skip MediaWiki install [integration/config] - 10https://gerrit.wikimedia.org/r/519644 (owner: 10Kosta Harlan) [18:36:19] 2.0.1 tagged - https://gerrit.wikimedia.org/g/mediawiki/tools/phan/SecurityCheckPlugin/+/refs/tags/2.0.1 [18:38:18] 10Continuous-Integration-Config, 10Release-Engineering-Team (Code Health), 10Code-Health: Create a CI container for mediawiki/tools/dependency-analysis - https://phabricator.wikimedia.org/T227603 (10hashar) [18:38:26] OK, now we just need legoktm to run LibUp? ;-) [18:38:26] (03CR) 10jerkins-bot: [V: 04-1] jjb: [mediawiki-quibble-composertest-*-docker] Skip MediaWiki install [integration/config] - 10https://gerrit.wikimedia.org/r/519644 (owner: 10Kosta Harlan) [18:38:34] 10Continuous-Integration-Config, 10LibUp, 10phan-taint-check-plugin: Upgrade taint-check to 2.0.1 in all repos - https://phabricator.wikimedia.org/T227172 (10Daimona) [18:38:39] Yep [18:38:54] ohai [18:38:59] And a huge thanks for your help :) [18:39:05] Always! [18:39:18] 10Continuous-Integration-Config, 10Release-Engineering-Team (Code Health), 10Code-Health: Create a CI container for mediawiki/tools/dependency-analysis - https://phabricator.wikimedia.org/T227603 (10hashar) @Daniel I am not sure why networkx has to be used with python 2. It has been supporting python 3 for a... [18:39:24] legoktm: You free? Batch run to bump phan-seccheck from 1.5.0 to 2.0.1 in all repos… [18:39:27] Daimona: \o/ [18:39:43] (sorry for disappearing, glad to see James_F was able to help you out) [18:39:51] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/519644 (owner: 10Kosta Harlan) [18:40:13] (03CR) 10Jforrester: [C: 03+2] "> Patch Set 3:" [integration/config] - 10https://gerrit.wikimedia.org/r/519644 (owner: 10Kosta Harlan) [18:40:35] Yeah, no prob :) [18:40:35] James_F: I think I'll have time tonight, I need to tweak some stuff still for the new libraryupgrader to work [18:41:30] (03CR) 10Thcipriani: [V: 03+2 C: 03+2] "Changes look good to me!" [software/gerrit] (wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/521564 (owner: 10Paladox) [18:42:00] 10Continuous-Integration-Infrastructure: contint1001 spurious disk space alarms - https://phabricator.wikimedia.org/T227605 (10hashar) [18:42:32] 10Continuous-Integration-Infrastructure: contint1001 spurious disk space alarms - https://phabricator.wikimedia.org/T227605 (10Dzahn) a:03Dzahn [18:42:52] (03Merged) 10jenkins-bot: jjb: [mediawiki-quibble-composertest-*-docker] Skip MediaWiki install [integration/config] - 10https://gerrit.wikimedia.org/r/519644 (owner: 10Kosta Harlan) [18:43:07] 10Continuous-Integration-Infrastructure: contint1001 spurious disk space alarms - https://phabricator.wikimedia.org/T227605 (10Dzahn) Need to exclude that from the disk space check.. has happened before. [18:43:47] (03PS1) 10Jforrester: layout: Swap mediawiki-quibble-composertest-php70-docker for mediawiki-quibble-composertest-php72-docker [integration/config] - 10https://gerrit.wikimedia.org/r/521568 [18:43:59] Kk. [18:44:09] (03CR) 10Jforrester: [C: 03+2] layout: Swap mediawiki-quibble-composertest-php70-docker for mediawiki-quibble-composertest-php72-docker [integration/config] - 10https://gerrit.wikimedia.org/r/521568 (owner: 10Jforrester) [18:44:21] James, would you mind looking again? https://gerrit.wikimedia.org/r/#/c/mediawiki/tools/phan/SecurityCheckPlugin/+/507854/ [18:44:23] 10Continuous-Integration-Infrastructure: contint1001 spurious disk space alarms - https://phabricator.wikimedia.org/T227605 (10hashar) I had a quick look at it during the day, we need to prevent both Diamond and the Icinga check_disk plugin from monitoring `/mnt/docker/overlay2/*` partitions. I found previous e... [18:44:24] Thanks! [18:44:34] Sure. [18:45:10] Maybe we can actually get even a 2.0.2 [18:45:28] Well, dropping php70/71 is a breaking change, so won't that be 3.0.0? [18:45:45] Or maybe not [18:45:46] Not that one [18:45:56] 2.0.2 would just be 4 cleanups [18:46:06] (03Merged) 10jenkins-bot: layout: Swap mediawiki-quibble-composertest-php70-docker for mediawiki-quibble-composertest-php72-docker [integration/config] - 10https://gerrit.wikimedia.org/r/521568 (owner: 10Jforrester) [18:46:14] Dropping php70 yes, either 2.1.0 or 3.0.0 [18:47:17] We're going to drop php71 from MW at the same time as PHP70, I imagine. [18:47:54] !log Zuul: Move mediawiki-quibble-composertest from php70 to php72 [18:47:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:47:59] That makes sense [18:48:14] Although I'd like to keep PHP71 for another bit [18:48:19] Sure. [18:48:26] Yet Another Breaking Change™. ;-) [18:54:50] Uhm well there are some phan issues to fix to make jenkins happy [18:55:04] I'll probably do that tomorrow, so 2.0.2 can wait [18:55:47] Time to go for me now, thanks again! [19:13:50] (03CR) 10Jforrester: "> Patch Set 8:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/502286 (https://phabricator.wikimedia.org/T193824) (owner: 10Hashar) [19:15:11] 10Continuous-Integration-Config, 10Release-Engineering-Team (Code Health), 10Code-Health: Create a CI container for mediawiki/tools/dependency-analysis - https://phabricator.wikimedia.org/T227603 (10daniel) >>! In T227603#5318565, @hashar wrote: > @Daniel I am not sure why networkx has to be used with python... [19:15:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Code Health), 10Test-Coverage: Migrate https://tools.wmflabs.org/coverage/mediawiki/ to CI infrastructure - https://phabricator.wikimedia.org/T182751 (10WDoranWMF) We're not positive if CPT should be owning this task. @Jrbranaa Could you... [19:16:47] 10Continuous-Integration-Config, 10Release-Engineering-Team (Code Health), 10Code-Health: Create a CI container for mediawiki/tools/dependency-analysis - https://phabricator.wikimedia.org/T227603 (10daniel) @hashar on a related note: {T227600} [19:17:30] 10Release-Engineering-Team (Code Health), 10MediaWiki-Core-Testing, 10Test-Coverage: Track test code coverage long term - https://phabricator.wikimedia.org/T182749 (10WDoranWMF) @Jrbranaa We don't think CPT owns this task, could you advise if Code Health is the correct place to direct it? [19:18:36] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Code Health), 10Test-Coverage: Migrate https://tools.wmflabs.org/coverage/mediawiki/ to CI infrastructure - https://phabricator.wikimedia.org/T182751 (10Jrbranaa) a:03Jrbranaa @WDoranWMF, I'll take ownership of this task for the time bein... [19:19:59] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: contint1001 spurious disk space alarms - https://phabricator.wikimedia.org/T227605 (10Dzahn) 05Open→03Resolved DISK space check on contint1001 is OK even before the merge but that's why you said "spurious". Now it should not happen anymore. [19:23:40] (03CR) 10Catrope: [C: 03+1] Add a missing phan dependency for Flow [integration/config] - 10https://gerrit.wikimedia.org/r/521434 (owner: 10Daimona Eaytoy) [19:31:51] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: contint1001 spurious disk space alarms - https://phabricator.wikimedia.org/T227605 (10hashar) 05Resolved→03Open There is a slight issue which is that `/mnt/docker` is now ignored but it is a partition. So we should ignore the images mount points... [19:33:14] (03CR) 10Jforrester: [C: 03+2] Add a missing phan dependency for Flow [integration/config] - 10https://gerrit.wikimedia.org/r/521434 (owner: 10Daimona Eaytoy) [19:34:54] (03Merged) 10jenkins-bot: Add a missing phan dependency for Flow [integration/config] - 10https://gerrit.wikimedia.org/r/521434 (owner: 10Daimona Eaytoy) [19:45:55] 10Release-Engineering-Team-TODO (201907), 10Release Pipeline, 10Core Platform Team (Extension Management (TEC13)), 10MW-1.34-notes (1.34.0-wmf.8; 2019-06-04), 10Patch-For-Review: Determine a standard way of installing MediaWiki lib/extension dependencies within c... - https://phabricator.wikimedia.org/T193824 [19:47:32] 10Release-Engineering-Team-TODO (201907), 10Release Pipeline, 10Core Platform Team (Extension Management (TEC13)), 10MW-1.34-notes (1.34.0-wmf.8; 2019-06-04), 10Patch-For-Review: Determine a standard way of installing MediaWiki lib/extension dependencies within c... - https://phabricator.wikimedia.org/T193824 [20:03:05] (03PS3) 10Kosta Harlan: [quibble-coverage] Fix code coverage generation for core codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/521190 [20:54:18] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Release, 10Train Deployments: Cannot save on 1.34.0-wmf.13 - https://phabricator.wikimedia.org/T227613 (10jeena) [21:02:14] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Ruby: Ruby mediawiki_api client hides error details - https://phabricator.wikimedia.org/T127786 (10Smalyshev) 05Open→03Declined [21:09:47] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO: deployment-tin has disk space issues - https://phabricator.wikimedia.org/T166492 (10Dzahn) 05Open→03Resolved a:03Dzahn deployment-tin does not exist anymore and deployment-deploy01 and 02 have 60GB disk space each. [21:10:11] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO: deployment-tin has disk space issues - https://phabricator.wikimedia.org/T166492 (10Dzahn) a:05Dzahn→03thcipriani [21:10:36] 10Release-Engineering-Team (Code Health), 10MediaWiki-Core-Testing, 10Code-Health, 10Test-Coverage: Track test code coverage long term - https://phabricator.wikimedia.org/T182749 (10Jrbranaa) @WDoranWMF, I do think this is a Code Health topic. I've added the Code-Health project tag to it so that it shows... [21:13:17] 10Phabricator, 10Regression: All Burnup Rate charts (burndown) are 404 - https://phabricator.wikimedia.org/T227247 (10mmodell) 05Open→03Resolved a:03mmodell The charts are now based on the 'facts' application and the application was uninstalled on our phabricator. I "installed" the `facts` application an... [21:13:56] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn) Next we need to make a decision whether we keep phab1003 as the prod host perman... [21:16:18] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [21:20:00] thcipriani/paladox: can we see what kind of requests are the new waiting ones? i see the top job is mwgate-node10-docker https://grafana.wikimedia.org/d/000000322/zuul-gearman?panelId=10&fullscreen&orgId=1&from=1562704807877&to=1562707078963 [21:20:32] there's alot of tests running at https://integration.wikimedia.org/zuul/ [21:20:37] (03PS27) 10Hashar: Gerrit: Add CI for operations/software/gerrit [integration/config] - 10https://gerrit.wikimedia.org/r/493328 (https://phabricator.wikimedia.org/T189549) (owner: 10Paladox) [21:20:58] paladox: i see.. so Mediawiki extensions [21:21:12] yeah, it seems like all the executors are running currently [21:21:18] yup [21:21:30] alright, thanks. so we just give them some time [21:23:44] hrm, the queue does seem abnormally large [21:23:55] but the wait times don't really [21:24:00] bunch of patches all at once? [21:24:33] in answer to that question https://gerrit.wikimedia.org/r/#/q/topic:bump-dev-deps+(status:open+OR+status:merged) [21:24:35] yes [21:25:42] (03CR) 10Hashar: "I have polished up the cache things :] Seems I manage to compile properly now." (035 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/493328 (https://phabricator.wikimedia.org/T189549) (owner: 10Paladox) [21:26:37] (03CR) 10Paladox: "Yay! Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/493328 (https://phabricator.wikimedia.org/T189549) (owner: 10Paladox) [21:27:49] 10Gerrit: Gerrit tools/download_file.py needs some enhancement - https://phabricator.wikimedia.org/T227617 (10hashar) [21:29:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Code Health), 10Code-Health-Group, 10Test-Coverage: Migrate https://tools.wmflabs.org/coverage/mediawiki/ to CI infrastructure - https://phabricator.wikimedia.org/T182751 (10Jrbranaa) [21:30:15] 10Gerrit: Gerrit tools/download_file.py needs some enhancement - https://phabricator.wikimedia.org/T227617 (10hashar) p:05Triage→03Low [21:33:24] 10Release-Engineering-Team (Code Health), 10Code-Health-Group, 10MediaWiki-Core-Testing, 10Code-Health, 10Test-Coverage: Track test code coverage long term - https://phabricator.wikimedia.org/T182749 (10Jrbranaa) [21:33:35] 10Project-Admins, 10Striker: Allow self-service creation of Maniphest projects for Tools - https://phabricator.wikimedia.org/T144111 (10bd808) [21:34:39] (03CR) 10Hashar: "We would need some branch filter in zuul/layout.yaml to only build for certain branches. I will complete it tomorrow and add it as a non v" [integration/config] - 10https://gerrit.wikimedia.org/r/493328 (https://phabricator.wikimedia.org/T189549) (owner: 10Paladox) [21:39:38] 10Phabricator, 10Regression: All Burnup Rate charts (burndown) are 404 (due to uninstalled "Facts" application) - https://phabricator.wikimedia.org/T227247 (10Aklapper) [22:45:05] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T220738 (10Ladsgroup) [22:48:11] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T220738 (10Ladsgroup) [22:50:41] (03PS1) 10Jdlrobson: Edit Project Config [extensions/MobileFrontend] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/521792 [22:52:54] (03Abandoned) 10Jdlrobson: Edit Project Config [extensions/MobileFrontend] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/521792 (owner: 10Jdlrobson) [22:55:10] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T220738 (10Krinkle) [22:55:24] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T220738 (10Krinkle) [22:55:46] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [22:57:49] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch/Buster - https://phabricator.wikimedia.org/T218729 (10Krenair) [23:24:15] (03PS1) 10Jforrester: jjb: Ensure all sec-check tasks are on mediawiki-phan-seccheck:0.3.6-2 [integration/config] - 10https://gerrit.wikimedia.org/r/521797 [23:24:37] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/521797 (owner: 10Jforrester) [23:26:50] (03Merged) 10jenkins-bot: jjb: Ensure all sec-check tasks are on mediawiki-phan-seccheck:0.3.6-2 [integration/config] - 10https://gerrit.wikimedia.org/r/521797 (owner: 10Jforrester) [23:37:14] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Release, 10Train Deployments: 1.34.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T220738 (10Jdforrester-WMF) [23:40:31] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Regression: All Burnup Rate charts (burndown) are 404 (due to uninstalled "Facts" application) - https://phabricator.wikimedia.org/T227247 (10greg) [23:46:39] 10Continuous-Integration-Config, 10LibUp, 10phan-taint-check-plugin: Upgrade taint-check to 2.0.1 in all repos - https://phabricator.wikimedia.org/T227172 (10Jdforrester-WMF) OK, I did a manual batch run for all Wikimedia production repos and pushed them into gerrit. Unfortunately, a number of these fail to...