[02:31:06] (03PS1) 10Yaron Koren: Edit Project Config [extensions/MintyDocs] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/526579 [02:36:08] (03Abandoned) 10Yaron Koren: Edit Project Config [extensions/MintyDocs] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/526579 (owner: 10Yaron Koren) [02:46:36] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<11.11%) [06:17:50] (03PS1) 10Legoktm: castor: Remove --compress from rsync [integration/config] - 10https://gerrit.wikimedia.org/r/526603 (https://phabricator.wikimedia.org/T188375) [06:24:17] (03PS2) 10Legoktm: castor: Remove --compress from rsync [integration/config] - 10https://gerrit.wikimedia.org/r/526603 (https://phabricator.wikimedia.org/T188375) [06:25:28] (03PS1) 10Legoktm: jjb: Use castor:0.2.3 image [integration/config] - 10https://gerrit.wikimedia.org/r/526604 [06:25:54] (03CR) 10Legoktm: [C: 03+2] castor: Remove --compress from rsync [integration/config] - 10https://gerrit.wikimedia.org/r/526603 (https://phabricator.wikimedia.org/T188375) (owner: 10Legoktm) [06:28:03] (03Merged) 10jenkins-bot: castor: Remove --compress from rsync [integration/config] - 10https://gerrit.wikimedia.org/r/526603 (https://phabricator.wikimedia.org/T188375) (owner: 10Legoktm) [06:56:37] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:08:42] (03CR) 10Legoktm: [C: 03+2] jjb: Use castor:0.2.3 image [integration/config] - 10https://gerrit.wikimedia.org/r/526604 (owner: 10Legoktm) [07:10:53] (03Merged) 10jenkins-bot: jjb: Use castor:0.2.3 image [integration/config] - 10https://gerrit.wikimedia.org/r/526604 (owner: 10Legoktm) [09:03:21] (03PS1) 10Kosta Harlan: Use plain phpunit command for mwcore-codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/526623 (https://phabricator.wikimedia.org/T224989) [09:03:37] (03PS2) 10Kosta Harlan: [quibble-coverage] Use plain phpunit command for mwcore-codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/526623 (https://phabricator.wikimedia.org/T224989) [09:34:34] (03PS1) 10Kosta Harlan: Use quibble-coverage:0.0.34-3 for mwcore-codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/526626 (https://phabricator.wikimedia.org/T224989) [09:46:53] (03PS2) 10Kosta Harlan: jjb: [mwcore-codehealth*] Use quibble-coverage:0.0.34-3 [integration/config] - 10https://gerrit.wikimedia.org/r/526626 (https://phabricator.wikimedia.org/T224989) [11:47:56] (03PS1) 10Holger Knust: Cassandra-table-properties: Increase build timeout in Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/526654 [12:59:14] thcipriani: I'm going to move some deployment-prep VMs in a minute. Should be over pretty quickly. [13:03:27] PROBLEM - Host deployment-elastic05 is DOWN: CRITICAL - Host Unreachable (172.16.5.136) [13:06:26] case in point [13:08:24] RECOVERY - Host deployment-elastic05 is UP: PING OK - Packet loss = 0%, RTA = 0.45 ms [13:14:59] all done [14:25:29] PROBLEM - Host deployment-sca02 is DOWN: CRITICAL - Host Unreachable (172.16.5.112) [14:30:32] RECOVERY - Host deployment-sca02 is UP: PING OK - Packet loss = 0%, RTA = 0.66 ms [14:31:31] PROBLEM - Host integration-slave-jessie-1004 is DOWN: CRITICAL - Host Unreachable (172.16.2.228) [14:31:53] PROBLEM - Host integration-slave-jessie-1001 is DOWN: CRITICAL - Host Unreachable (172.16.0.86) [14:36:12] PROBLEM - Host saucelabs-02 is DOWN: CRITICAL - Host Unreachable (172.16.3.20) [14:41:16] RECOVERY - Host saucelabs-02 is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms [14:42:16] (03PS1) 10Cicalese: Archived SemanticTitle extension [integration/config] - 10https://gerrit.wikimedia.org/r/526690 (https://phabricator.wikimedia.org/T229422) [14:46:09] RECOVERY - Host integration-slave-jessie-1004 is UP: PING OK - Packet loss = 0%, RTA = 1.21 ms [14:46:55] RECOVERY - Host integration-slave-jessie-1001 is UP: PING OK - Packet loss = 0%, RTA = 1.03 ms [15:38:52] (03PS3) 10Thcipriani: php7x: Add php_fpm module [tools/scap] - 10https://gerrit.wikimedia.org/r/525117 (https://phabricator.wikimedia.org/T224857) [15:39:13] (03CR) 10Jforrester: [C: 03+2] [quibble-coverage] Use plain phpunit command for mwcore-codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/526623 (https://phabricator.wikimedia.org/T224989) (owner: 10Kosta Harlan) [15:39:25] (03CR) 10Thcipriani: php7x: Add php_fpm module (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/525117 (https://phabricator.wikimedia.org/T224857) (owner: 10Thcipriani) [15:40:51] (03Merged) 10jenkins-bot: [quibble-coverage] Use plain phpunit command for mwcore-codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/526623 (https://phabricator.wikimedia.org/T224989) (owner: 10Kosta Harlan) [15:45:28] (03PS1) 10Giuseppe Lavagetto: operations-puppet: copy tox and bundle data over [integration/config] - 10https://gerrit.wikimedia.org/r/526711 [15:46:20] !log Docker: Pushing quibble-coverage:0.0.34-3 [15:46:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:51:04] <_joe_> James_F: I need to push a new version of operations-puppet [15:51:07] (03CR) 10Giuseppe Lavagetto: [C: 03+2] operations-puppet: copy tox and bundle data over [integration/config] - 10https://gerrit.wikimedia.org/r/526711 (owner: 10Giuseppe Lavagetto) [15:51:29] <_joe_> is there any particular caveat to pushing the images from contint1001? [15:51:38] <_joe_> I think I can manage to fiund out how to do it [15:52:13] <_joe_> but besides running docker-pkg and logging here [15:52:19] <_joe_> anything else I should do? [15:52:42] (03Merged) 10jenkins-bot: operations-puppet: copy tox and bundle data over [integration/config] - 10https://gerrit.wikimedia.org/r/526711 (owner: 10Giuseppe Lavagetto) [15:53:31] _joe_: Just run `fab deploy_docker` (or I can). [15:53:51] <_joe_> James_F: as what user? [15:54:01] That's what fab worries about. [15:54:22] <_joe_> I don't have "fab" as a command, I guess you somehow installed fabric? [15:54:35] Yes, it's the deploy system for CI. [15:54:41] I'll just do it for you. :-) [15:55:20] !log Docker: Pushing operations-puppet:0.5.4 [15:55:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:55:42] <_joe_> thanks [15:56:16] _joe_: https://www.mediawiki.org/wiki/Continuous_integration/Docker#Deploy_images for future reference. [16:00:34] _joe_: Built and published. [16:00:56] <_joe_> James_F: thanks! [16:04:25] (03PS3) 10Jforrester: jjb: [mwcore-codehealth*] Use quibble-coverage:0.0.34-3 [integration/config] - 10https://gerrit.wikimedia.org/r/526626 (https://phabricator.wikimedia.org/T224989) (owner: 10Kosta Harlan) [16:15:18] (03CR) 10Giuseppe Lavagetto: [C: 03+1] "LGTM!" [tools/scap] - 10https://gerrit.wikimedia.org/r/525117 (https://phabricator.wikimedia.org/T224857) (owner: 10Thcipriani) [16:19:13] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/526626 (https://phabricator.wikimedia.org/T224989) (owner: 10Kosta Harlan) [16:20:34] (03CR) 10Jforrester: [C: 03+2] Archived SemanticTitle extension [integration/config] - 10https://gerrit.wikimedia.org/r/526690 (https://phabricator.wikimedia.org/T229422) (owner: 10Cicalese) [16:26:01] (03Merged) 10jenkins-bot: jjb: [mwcore-codehealth*] Use quibble-coverage:0.0.34-3 [integration/config] - 10https://gerrit.wikimedia.org/r/526626 (https://phabricator.wikimedia.org/T224989) (owner: 10Kosta Harlan) [16:26:04] (03Merged) 10jenkins-bot: Archived SemanticTitle extension [integration/config] - 10https://gerrit.wikimedia.org/r/526690 (https://phabricator.wikimedia.org/T229422) (owner: 10Cicalese) [16:27:09] !log Zuul: Archive SemanticTitle T229422 [16:27:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:27:11] T229422: Remove extension SemanticTItle from CI - https://phabricator.wikimedia.org/T229422 [16:33:23] James_F: seems to be working... [16:33:30] Watching https://integration.wikimedia.org/ci/job/mwcore-codehealth-patch/1920/console [16:34:35] Fingers crossed. [16:54:23] yep! [16:54:49] Core will have 1.3% code coverage after the next patch is merged :) https://sonarcloud.io/dashboard?id=mediawiki-core&branch=526704&resolved=false [17:05:04] Rome wasn't built in a day ;) [17:05:46] ha [17:06:15] 28% when you include integration tests https://doc.wikimedia.org/cover/mediawiki-core/ [17:06:37] 1.3% seems too low for the proportion of tests that we've converted [17:07:09] (03PS4) 1020after4: Use mediawiki-dev chart from releases.wikimedia.org/charts/ [releng/local-charts] - 10https://gerrit.wikimedia.org/r/525888 (https://phabricator.wikimedia.org/T228913) (owner: 10Jeena Huneidi) [17:07:11] (03PS5) 1020after4: local-charts: Skeleton go-based cli [releng/local-charts] - 10https://gerrit.wikimedia.org/r/525227 (https://phabricator.wikimedia.org/T224939) [17:08:03] (03CR) 10Thcipriani: [V: 03+2 C: 03+2] "LGTM for a base." [releng/local-charts] - 10https://gerrit.wikimedia.org/r/525227 (https://phabricator.wikimedia.org/T224939) (owner: 1020after4) [17:09:44] (03PS6) 1020after4: local-charts: Skeleton go-based cli [releng/local-charts] - 10https://gerrit.wikimedia.org/r/525227 (https://phabricator.wikimedia.org/T224939) [17:10:31] (03CR) 1020after4: [C: 03+2] local-charts: Skeleton go-based cli [releng/local-charts] - 10https://gerrit.wikimedia.org/r/525227 (https://phabricator.wikimedia.org/T224939) (owner: 1020after4) [17:10:52] (03CR) 10Thcipriani: [V: 03+2 C: 03+2] local-charts: Skeleton go-based cli [releng/local-charts] - 10https://gerrit.wikimedia.org/r/525227 (https://phabricator.wikimedia.org/T224939) (owner: 1020after4) [17:14:55] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, 10Traffic: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10faidon) a:03JAufrecht [17:16:09] (03CR) 10Jforrester: "Is there a task for this? (Needs a follow-up patch in layout.yaml changing cassandra-table-properties's generic tox-docker template to thi" [integration/config] - 10https://gerrit.wikimedia.org/r/526654 (owner: 10Holger Knust) [17:17:52] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10Wikimedia-General-or-Unknown, 10Epic, 10phan: Enable mediawiki/mediawiki-phan-config on all Wikimedia-deployed repositories - https://phabricator.wikimedia.org/T224783 (10Daimona) [17:27:27] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, 10Traffic: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10greg) >>! In T226044#5380759, @BBlack wrote: > 2. Configure a new ph... [17:38:15] (03PS2) 10Jforrester: jjb: [cassandra-table-properties] Provide tox-docker with a longer timeout [integration/config] - 10https://gerrit.wikimedia.org/r/526654 (owner: 10Holger Knust) [17:38:17] (03PS1) 10Jforrester: layout: [cassandra-table-properties] Use cassandra-table-properties-tox-docker [integration/config] - 10https://gerrit.wikimedia.org/r/526733 [17:45:06] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/526654 (owner: 10Holger Knust) [17:45:09] (03CR) 10Jforrester: [C: 03+2] layout: [cassandra-table-properties] Use cassandra-table-properties-tox-docker [integration/config] - 10https://gerrit.wikimedia.org/r/526733 (owner: 10Jforrester) [17:47:48] 10MediaWiki-Codesniffer: Add a sniff to disallow @internal and @since on the same documented member - https://phabricator.wikimedia.org/T229454 (10Krinkle) [17:48:36] (03Merged) 10jenkins-bot: jjb: [cassandra-table-properties] Provide tox-docker with a longer timeout [integration/config] - 10https://gerrit.wikimedia.org/r/526654 (owner: 10Holger Knust) [17:48:39] (03Merged) 10jenkins-bot: layout: [cassandra-table-properties] Use cassandra-table-properties-tox-docker [integration/config] - 10https://gerrit.wikimedia.org/r/526733 (owner: 10Jforrester) [17:50:12] !log Zuul: Move cassandra-table-properties to use cassandra-table-properties-tox-docker [17:50:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:04:10] (03PS1) 10Jforrester: layout: Consistency of whitespace [integration/config] - 10https://gerrit.wikimedia.org/r/526739 [18:09:50] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, 10Traffic: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10JAufrecht) > Decide on a public pretty domainname for these more pub... [18:10:23] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10JAufrecht) a:05JAufrecht→03greg passing to Greg for step 2, con... [18:13:47] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10BBlack) >>! In T226044#5381138, @JAufrecht wrote: > I think it shou... [18:14:04] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10Krinkle) I tested the "Move Post" feature in Phame today - on the f... [18:18:12] (03PS1) 10Jforrester: jjb: [tox-docker] Increase default timeout to 20 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/526745 [18:18:14] (03PS1) 10Jforrester: layout: Move three repos to generic tox-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/526746 [18:18:16] (03PS1) 10Jforrester: jjb: Drop three custom tox-docker jobs, unused [integration/config] - 10https://gerrit.wikimedia.org/r/526747 [18:18:19] (03Abandoned) 10Jforrester: layout: Consistency of whitespace [integration/config] - 10https://gerrit.wikimedia.org/r/526739 (owner: 10Jforrester) [18:19:27] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10BBlack) Replying to myself earlier: apparently they're datestamped... [18:21:48] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/526745 (owner: 10Jforrester) [18:21:56] (03CR) 10Jforrester: [C: 03+2] layout: Move three repos to generic tox-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/526746 (owner: 10Jforrester) [18:24:03] (03Merged) 10jenkins-bot: jjb: [tox-docker] Increase default timeout to 20 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/526745 (owner: 10Jforrester) [18:24:05] (03Merged) 10jenkins-bot: layout: Move three repos to generic tox-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/526746 (owner: 10Jforrester) [18:27:14] !log Zuul: Move three repos to generic tox-docker job [18:27:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:27:16] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10Krinkle) >>! In T226044#5381193, @BBlack wrote: > [..] apparently t... [18:29:22] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/526747 (owner: 10Jforrester) [18:31:48] (03Merged) 10jenkins-bot: jjb: Drop three custom tox-docker jobs, unused [integration/config] - 10https://gerrit.wikimedia.org/r/526747 (owner: 10Jforrester) [18:37:55] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments: 1.34.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T220741 (10Jdlrobson) [18:45:28] (03PS1) 10Jforrester: layout: Migrate users of bespoke -tox-docker jobs to generic one [integration/config] - 10https://gerrit.wikimedia.org/r/526749 [18:45:30] (03PS1) 10Jforrester: jjb: Drop more tox-docker generics [integration/config] - 10https://gerrit.wikimedia.org/r/526750 [18:50:42] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10BBlack) Heh, apparently I can't even remember things I read and sai... [18:53:13] 10Phabricator, 10Developer-Advocacy, 10Patch-For-Review: Decrease Phab task cookie licking (assignee field set for years without progress) - https://phabricator.wikimedia.org/T228575 (10Dzahn) {P8842} [19:05:21] 10Phabricator, 10Developer-Advocacy: Decrease Phab task cookie licking (assignee field set for years without progress) - https://phabricator.wikimedia.org/T228575 (10Aklapper) Thanks for review and merge! Now that we have some data how big the problem is, let me sort out how to best go ahead here. :) [19:08:13] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10Aklapper) >>! In T226044#5381170, @Krinkle wrote: > I tested the "M... [19:20:14] !log ladsgroup@deployment-deploy01:/srv/mediawiki-staging/php-master$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki (T225056) [19:20:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:20:19] T225056: Run Item Terms Rebuild script - https://phabricator.wikimedia.org/T225056 [20:32:40] 10Phabricator (Upstream), 10Upstream: Phabricator boards should live update - https://phabricator.wikimedia.org/T129 (10Aklapper) [20:32:42] 10Phabricator: Migration of Mobile Apps team to Phabricator - https://phabricator.wikimedia.org/T851 (10Aklapper) [20:32:56] 10Phabricator (Upstream), 10Upstream: Phabricator boards should live update - https://phabricator.wikimedia.org/T129 (10Aklapper) [20:33:01] 10Phabricator, 10RelEng-Archive-FY201718-Q2, 10Patch-For-Review: Enable notification server (real-time pop-up notifications) in Phabricator - https://phabricator.wikimedia.org/T765 (10Aklapper) [20:33:10] 10Phabricator (Upstream), 10Upstream: Phabricator boards should live update - https://phabricator.wikimedia.org/T129 (10Aklapper) Upstream ticket https://secure.phabricator.com/T4900 has been closed as resolved. Further potential future improvements are listed in https://secure.phabricator.com/T13357 [20:43:10] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments: 1.34.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T220741 (10Jdlrobson) [20:57:44] (03CR) 10Legoktm: "The only value it gets is that its put in a different gate-and-submit queue (yesterday one of my libup patches got stuck behind an oojs/ui" [integration/config] - 10https://gerrit.wikimedia.org/r/526749 (owner: 10Jforrester) [21:05:35] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments: 1.34.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T220741 (10brennen) [21:06:59] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Parsoid-PHP, 10Patch-For-Review: Installing composer modules for deployment - https://phabricator.wikimedia.org/T213494 (10Legoktm) Are we still going to be deploying parsoid's /deploy repository? We can bundle the compose... [21:07:58] heh, thanks mdholloway with that project add for that logspam :) [21:08:41] greg-g: yeah, i think i know something about that one... [21:28:14] (03CR) 10Jforrester: "> Patch Set 1:" [integration/config] - 10https://gerrit.wikimedia.org/r/526749 (owner: 10Jforrester) [21:34:14] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Parsoid-PHP, 10Patch-For-Review: Installing composer modules for deployment - https://phabricator.wikimedia.org/T213494 (10cscott) Since Parsoid is currently PHP 7.2+, I think adding parsoid directly to core as a library w... [22:14:23] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Parsoid-PHP, 10Patch-For-Review: Installing composer modules for deployment - https://phabricator.wikimedia.org/T213494 (10ssastry) Just a point of process. There have been many suggestions here and on the gerrit patch. I... [22:54:51] (03PS5) 10Jeena Huneidi: Use mediawiki-dev chart from releases.wikimedia.org/charts/ [releng/local-charts] - 10https://gerrit.wikimedia.org/r/525888 (https://phabricator.wikimedia.org/T228913) [22:57:39] 10Phabricator: https://phabricator.wikimedia.org/tag/MediaWiki-extensions-LiquidThreads/ is a 404 but project exists - https://phabricator.wikimedia.org/T223540 (10Urbanecm) Same with https://phabricator.wikimedia.org/tag/growthexperiments/. [23:00:23] (03CR) 10Jeena Huneidi: [V: 03+2 C: 03+2] Use mediawiki-dev chart from releases.wikimedia.org/charts/ [releng/local-charts] - 10https://gerrit.wikimedia.org/r/525888 (https://phabricator.wikimedia.org/T228913) (owner: 10Jeena Huneidi) [23:02:41] Project beta-scap-eqiad build #260367: 04FAILURE in 0.82 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/260367/ [23:04:23] Project beta-scap-eqiad build #260368: 04STILL FAILING in 0.81 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/260368/ [23:12:00] Yippee, build fixed! [23:12:00] Project beta-scap-eqiad build #260369: 09FIXED in 2 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/260369/ [23:25:49] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO, 10Code-Stewardship-Reviews: Code Stewardship Review: OAuth extension - https://phabricator.wikimedia.org/T224919 (10bd808) Just a note in my role as concerned stakeholder, the OAuth functionality is used by a number of Wikimedia tools... [23:41:47] libup is theoretically running in auto mode now, except it's waiting for the queue in zuul to go down so I don't actually know yet :| [23:46:11] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO, 10Code-Stewardship-Reviews: Code Stewardship Review: SpamBlacklist - https://phabricator.wikimedia.org/T224921 (10Bawolff) The form seems a little underfilled out... There was basically one production error which arguably was intentio... [23:47:37] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<22.22%)