[04:01:21] (03CR) 10Ammarpad: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/631228 (https://phabricator.wikimedia.org/T235286) (owner: 10HitomiAkane) [06:30:15] (03PS1) 10JMeybohm: helm-linter: Update envoy to 1.15.1 [integration/config] - 10https://gerrit.wikimedia.org/r/631343 (https://phabricator.wikimedia.org/T264157) [07:47:03] hashar: can I get a quick review of https://gerrit.wikimedia.org/r/c/integration/config/+/631343/ please? [09:08:50] 10Beta-Cluster-Infrastructure: deployment-snapshot01.deployment-prep.eqiad.wmflabs keeps running out of space - https://phabricator.wikimedia.org/T245402 (10ArielGlenn) I needed to set profile::mediawiki::mcrouter_wancache::use_onhost_memcached: false in the deployment-deploy prefix hiera settings. Now fixing up... [09:53:55] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10Maps, 10Product-Infrastructure-Team-Backlog: [Bug] Puppet is failing for maps beta cluster - could not find group maps-admins - https://phabricator.wikimedia.org/T263726 (10MSantos) So, I haven't been able to understand this yet, so I'm going to ask (please cor... [10:22:56] 10Beta-Cluster-Infrastructure: deployment-snapshot01.deployment-prep.eqiad.wmflabs keeps running out of space - https://phabricator.wikimedia.org/T245402 (10ArielGlenn) Well the new template is not going around because /usr/local/bin/git-sync-upstream fails to rebase. In /var/spool/cron/crontabs/root we have: `... [10:46:02] 10Beta-Cluster-Infrastructure: deployment-snapshot01.deployment-prep.eqiad.wmflabs keeps running out of space - https://phabricator.wikimedia.org/T245402 (10ArielGlenn) That commit is swift-related, and there are a few of those in the 18 unmerged commits by @jbond from the production branch, tagging him here in... [10:46:06] 10Gerrit, 10Wikimedia-GitHub: Create new GitHub mirrored repository with core + all extensions + all skins - https://phabricator.wikimedia.org/T264166 (10kostajh) I made a proof of concept at https://github.com/kostajh/mediawiki-all-extensions-all-skins. It seems like GitHub still needs some time to index the... [11:18:26] (03CR) 10Lars Wirzenius: [C: 03+2] scap wikiversions-compile: Add --staging flag [tools/scap] - 10https://gerrit.wikimedia.org/r/631272 (owner: 10Ahmon Dancy) [11:19:44] (03Merged) 10jenkins-bot: scap wikiversions-compile: Add --staging flag [tools/scap] - 10https://gerrit.wikimedia.org/r/631272 (owner: 10Ahmon Dancy) [11:23:38] (03CR) 10Urbanecm: [C: 03+1] layout.yaml: Add HitomiAkane to Jenkins whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/631228 (https://phabricator.wikimedia.org/T235286) (owner: 10HitomiAkane) [11:25:49] (03CR) 10Lars Wirzenius: [C: 03+2] compile_wikiversions: Update comments to reflect the code [tools/scap] - 10https://gerrit.wikimedia.org/r/631271 (owner: 10Ahmon Dancy) [11:28:11] (03Merged) 10jenkins-bot: compile_wikiversions: Update comments to reflect the code [tools/scap] - 10https://gerrit.wikimedia.org/r/631271 (owner: 10Ahmon Dancy) [11:31:57] if anyone's around who knows this right quick: what's the canonical way to update a puppet cherry-pick on deployment-prep so that the sync from upstream will work? assume that the new cherry-pick will not work nicely with the current several-day-old version of puppet files lying around on the puppetmaster04 [11:32:27] assume also that I've added any new hiera settings on the appropriate instances [11:37:02] 10Beta-Cluster-Infrastructure: deployment-snapshot01.deployment-prep.eqiad.wmflabs keeps running out of space - https://phabricator.wikimedia.org/T245402 (10ArielGlenn) I've added the new hiera setting profile::swift::proxy::memcached_servers to both the prefix settings the the instance settings for deploy-ms-fe... [11:37:11] (03CR) 10Lars Wirzenius: [V: 03+2 C: 03+2] Add ansible subcommand to train-dev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/631290 (owner: 10Ahmon Dancy) [11:38:32] (03CR) 10Lars Wirzenius: [V: 03+2 C: 03+2] Move -i ansible/hosts into run_ansible() [tools/train-dev] - 10https://gerrit.wikimedia.org/r/631292 (owner: 10Ahmon Dancy) [11:40:28] 10Beta-Cluster-Infrastructure: deployment-snapshot01.deployment-prep.eqiad.wmflabs keeps running out of space - https://phabricator.wikimedia.org/T245402 (10ArielGlenn) Adding @dpifke since it's his work being affected. [11:52:26] 10Release-Engineering-Team-TODO, 10Release, 10Train Deployments: 1.36.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T263178 (10Ammarpad) [11:53:56] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Scap, 10Wikimedia-Site-requests: Move scap plugins from mediawiki-config repo to scap repo - https://phabricator.wikimedia.org/T248490 (10LarsWirzenius) ~~~sh liw@deploy1001:~$ scap version 3.15.0-1 liw@deploy1001:~$ dpkg -L scap | grep wikiv... [12:24:40] (03CR) 10Pwirth: "> Patch Set 3:" [integration/config] - 10https://gerrit.wikimedia.org/r/624632 (owner: 10Pwirth) [12:39:10] 10LibUp, 10UI-Standardization: Auto re-configure stylelint to lint vue files too - https://phabricator.wikimedia.org/T264252 (10Esanders) grunt-eslint has no such feature, we would have to change `*.{js,json}` to `*.{js,json,vue}` as well. Once grunt has handed a list of files to eslint/stylelint, then they c... [13:48:30] 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10GitLab-Test, 10User-brennen: Investigate whether issues, operations, wikis, etc. can be disabled globally on GitLab - https://phabricator.wikimedia.org/T264231 (10Aklapper) In my understanding it's currently not possible (but the upstream tic... [14:46:37] 10Phabricator, 10CSS: "Change Story Points (Estimate)" overflow - https://phabricator.wikimedia.org/T264253 (10Aklapper) [15:32:28] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Scap, 10MediaWiki Train Development Environment, and 3 others: Create a development & testing environment for scap + train deployments - https://phabricator.wikimedia.org/T255695 (10dancy) [15:32:42] (03PS1) 10Ahmon Dancy: deploy-promote: Add --yes option [tools/release] - 10https://gerrit.wikimedia.org/r/631489 [15:32:50] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Scap, 10MediaWiki Train Development Environment, and 3 others: Try "deploy-promote testwikis" in train-dev, document what seems to be missing - https://phabricator.wikimedia.org/T260376 (10da... [15:35:45] (03CR) 10Lars Wirzenius: [C: 03+2] deploy-promote: Add --yes option [tools/release] - 10https://gerrit.wikimedia.org/r/631489 (owner: 10Ahmon Dancy) [15:36:37] (03Merged) 10jenkins-bot: deploy-promote: Add --yes option [tools/release] - 10https://gerrit.wikimedia.org/r/631489 (owner: 10Ahmon Dancy) [15:46:08] 10LibUp: libup does not update eslint to the version of approved version - https://phabricator.wikimedia.org/T264326 (10Umherirrender) [15:51:24] 10LibUp: Update nvm for libup to match CI - https://phabricator.wikimedia.org/T264328 (10Umherirrender) [15:56:28] (03PS1) 10Ahmon Dancy: deploy-promote: Don't wait for merge in train-dev [tools/release] - 10https://gerrit.wikimedia.org/r/631498 [15:57:24] (03CR) 10DannyS712: deploy-promote: Don't wait for merge in train-dev (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/631498 (owner: 10Ahmon Dancy) [15:58:00] (03CR) 10Ahmon Dancy: [C: 04-1] deploy-promote: Don't wait for merge in train-dev (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/631498 (owner: 10Ahmon Dancy) [15:58:43] (03PS2) 10Ahmon Dancy: deploy-promote: Don't wait for merge in train-dev [tools/release] - 10https://gerrit.wikimedia.org/r/631498 [15:59:08] (03CR) 10Ahmon Dancy: deploy-promote: Don't wait for merge in train-dev (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/631498 (owner: 10Ahmon Dancy) [15:59:24] (03CR) 10Lars Wirzenius: [C: 03+1] deploy-promote: Don't wait for merge in train-dev (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/631498 (owner: 10Ahmon Dancy) [16:00:00] (03CR) 10Lars Wirzenius: [C: 03+2] deploy-promote: Don't wait for merge in train-dev [tools/release] - 10https://gerrit.wikimedia.org/r/631498 (owner: 10Ahmon Dancy) [16:00:48] (03Merged) 10jenkins-bot: deploy-promote: Don't wait for merge in train-dev [tools/release] - 10https://gerrit.wikimedia.org/r/631498 (owner: 10Ahmon Dancy) [16:08:03] 10LibUp: Update nvm for libup to match CI - https://phabricator.wikimedia.org/T264328 (10Jdforrester-WMF) I'm not sure we should encourage this; letting repos come up with random versions of node via `nvm` is pretty likely to result in much less good `npm audit` capabilities, which is the primary purpose of LibU... [16:10:48] (03PS1) 10Cwhite: Bump mtail to 3.0.0~rc35-3+wmf3 [integration/config] - 10https://gerrit.wikimedia.org/r/631502 (https://phabricator.wikimedia.org/T263728) [16:34:22] (03PS4) 10Ahmon Dancy: update-wikiversions: Use staging dir wikiversions to update php link [tools/scap] - 10https://gerrit.wikimedia.org/r/628170 [16:35:39] maintenance-disconnect-full-disks build 216893 integration-agent-docker-1013 (/: 17%, /srv: 97%, /var/lib/docker: 54%): OFFLINE due to disk space [16:43:10] Anyone around for me to pick their brain around the best way to run authenticated npm/git commands (e.g. npm publish or git fetch) through CI? [16:46:20] maintenance-disconnect-full-disks build 216895 integration-agent-docker-1013 (/: 17%, /srv: 7%, /var/lib/docker: 49%): RECOVERY disk space OK [16:47:06] nikkinikk: Jenkins does have a password vault [16:47:24] nikkinikk: See the use of composer-github-oauthtoken in integration/config for one example [16:47:57] awesome thanks :Reedy [16:47:58] (03PS1) 10Ahmon Dancy: Semi-working webserver setup [tools/train-dev] - 10https://gerrit.wikimedia.org/r/631509 [16:49:50] 10Release-Engineering-Team-TODO, 10User-DannyS712, 10ci-test-error: Scap plugins in operations/mediawiki-config don't pass tox, so CI is disabled - https://phabricator.wikimedia.org/T248040 (10Jdforrester-WMF) [16:49:54] 10LibUp, 10Page-Previews: Update nvm for libup to match CI - https://phabricator.wikimedia.org/T264328 (10Legoktm) libup doesn't use nvm, it installs nodejs using the same version as CI. I agree with James, this is pretty fragile. Isn't node6 pretty old anyways? Can we update them to node10? [16:50:05] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Scap, 10Wikimedia-Site-requests: Move scap plugins from mediawiki-config repo to scap repo - https://phabricator.wikimedia.org/T248490 (10Jdforrester-WMF) a:05Jdforrester-WMF→03LarsWirzenius [16:50:07] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Scap, 10Wikimedia-Site-requests: Move scap plugins from mediawiki-config repo to scap repo - https://phabricator.wikimedia.org/T248490 (10Jdforrester-WMF) 05Resolved→03Open Reverted as `scap` lost track of (some of?) the plugins. [16:56:49] 10LibUp, 10JSDoc WMF theme, 10MinusX: js docs generated files by libup are created with executable bits - https://phabricator.wikimedia.org/T264143 (10Umherirrender) The original files are under https://github.com/wikimedia/WikimediaUI-Style-Guide/tree/d405fc70a4a301b7f3bad572539f59796ff85e8b used in the jsd... [17:09:36] 10LibUp, 10JSDoc WMF theme, 10MinusX: js docs generated files by libup are created with executable bits - https://phabricator.wikimedia.org/T264143 (10Jdforrester-WMF) >>! In T264143#6509622, @Umherirrender wrote: > The original files are under https://github.com/wikimedia/WikimediaUI-Style-Guide/tree/d405fc... [18:06:25] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T263177 (10Pchelolo) https://gerrit.wikimedia.org/r/c/mediawiki/core/+/631240 has to me merged before next attempt to ro... [18:56:23] 10LibUp, 10Page-Previews: Update nvm for libup to match CI - https://phabricator.wikimedia.org/T264328 (10Jdforrester-WMF) I moved it to node 10 in {4fb6120fc9b7d48d646cfb2a82a3034541655ee9} to match the version in CI at the time (and still), v10.15.2. However, it looks like LibUp is running on v10.21.0 instead? [19:01:08] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T263177 (10mmodell) [19:21:59] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)): Investigate/trace rebuildLocalisationCache process for MW containerization - https://phabricator.wikimedia.org/T260827 (10dduvall) a:05dduvall→03dancy Thanks for taking a look, @tstarling. It looks like... [19:26:11] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T263177 (10mmodell) [19:31:35] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T263177 (10Pchelolo) [19:39:13] Reedy: To your knowledge, are you aware of any projects that do npm publishing via a pipeline job? I'm trying to see if its possible and wanted to know if any examples exist [19:50:03] nikkinikk: Not offhand, but that obviously doesn't mean there isn't anyone doing it... [19:50:41] Reedy: okie, i will poke around a little bit :) [19:51:48] It's possible it's being done on the GH side too [20:27:14] Hi, not sure who to direct this to - we're trying to test changes on beta for example this patch [20:27:14] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/OAuth/+/631487 [20:27:14] was merged a few hours ago but has not made it's way to beta. [20:27:14] Any thoughts on how to figure out if deployment is working? [20:30:34] wdoran: Probably CI broke [20:30:45] Yup [20:30:46] https://integration.wikimedia.org/ci/view/Beta/ [20:30:50] Jobs haven't run for 20 hours [20:30:52] * Reedy pokes [20:31:04] Is there a switch to flip? Or something to kick? [20:31:11] I'm just doing it [20:31:17] It happens a few times a week [20:31:24] thanks! [20:32:12] It'll take it a little while to catch up, maybe 20 mins or so [20:32:31] For future reference.. If you look at https://en.wikipedia.beta.wmflabs.org/wiki/Special:Version [20:32:39] MediaWiki 1.36.0-alpha (eeb494c) [20:32:39] 21:19, 30 September 2020 [20:32:50] Stuff like that usually means something possibly might not be quite right [20:32:59] nice, very helpful [20:33:36] And similarly, you can look for extension commits/dates [20:34:42] Project beta-scap-eqiad build #319479: 04FAILURE in 4.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319479/ [20:36:21] sync from production has been broken for a little while, see: https://phabricator.wikimedia.org/T245402 the last few comments [20:36:41] if dpifke is around he can probably move that forward [20:36:42] Project beta-scap-eqiad build #319480: 04STILL FAILING in 3.8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319480/ [20:37:00] otherwise jbond tomorrow [20:37:43] It's that causing the etcd failure? [20:37:50] I don't know about etcd [20:38:14] I just know that changes in puppet are not getting around [20:38:17] 21:36:42 20:36:42 stderr: Fatal error: Uncaught ConfigException: Failed to load configuration from etcd: (curl error: 7) Couldn't connect to server in /srv/mediawiki-staging/php-master/includes/config/EtcdConfig.php:202 [20:38:21] because of a cherry pick conflict [20:39:39] to know if that impacted anything else, someone would have to look at th production changes that we're missing I guess [20:43:57] Project beta-scap-eqiad build #319481: 04STILL FAILING in 3.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319481/ [20:43:59] 'etcd' => 'deployment-etcd-01.deployment-prep.eqiad.wmflabs:2379', [20:44:19] that host seems to be dead [20:45:44] is there a new instance? [20:45:59] nope [20:46:02] only one deployment-etcd [20:46:07] horizon says it's shut off [20:46:09] I'll ask in -cloud [20:46:56] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Sustainability (Incident Followup): Scap feature: restart php-fpm on deployment - https://phabricator.wikimedia.org/T264362 (10thcipriani) [20:47:16] ./hieradata/cloud/eqiad1/deployment-prep/hosts/deployment-etcd-01.yaml there's only that in production puppet [20:47:51] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Sustainability (Incident Followup): Scap feature: restart php-fpm on deployment - https://phabricator.wikimedia.org/T264362 (10thcipriani) p:05Triage→03Medium a:03thcipriani [20:54:06] Project beta-scap-eqiad build #319482: 04STILL FAILING in 6.7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319482/ [20:55:25] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Sustainability (Incident Followup), 10User-brennen: Scap feature: restart php-fpm on deployment - https://phabricator.wikimedia.org/T264362 (10brennen) [21:05:57] 10LibUp, 10Page-Previews: Update nvm for libup to match CI - https://phabricator.wikimedia.org/T264328 (10Legoktm) >>! In T264328#6510113, @Jdforrester-WMF wrote: > I moved it to node 10 in {4fb6120fc9b7d48d646cfb2a82a3034541655ee9} to match the version in CI at the time (and still), v10.15.2. However, it look... [21:08:45] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T263177 (10Krinkle) [21:17:01] Project beta-scap-eqiad build #319483: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319483/ [21:18:16] wdoran: It should be.. mostly deployed now [21:18:42] lol [21:19:05] Reedy: you're magic [21:25:36] Yippee, build fixed! [21:25:36] Project beta-scap-eqiad build #319484: 09FIXED in 7 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319484/ [21:44:07] You're a wizard h̷a̷r̷r̷y̷ reedy [22:19:06] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T263177 (10Urbanecm) [22:44:18] (03PS1) 10Ahmon Dancy: Make sync-world work when there are no proxies [tools/scap] - 10https://gerrit.wikimedia.org/r/631548 [22:44:25] (03PS1) 10Ahmon Dancy: Allow scap command path to be overriden via environment var [tools/scap] - 10https://gerrit.wikimedia.org/r/631549 [22:46:23] (03Abandoned) 10Ahmon Dancy: DNM: in progress scap hacks [tools/scap] - 10https://gerrit.wikimedia.org/r/630285 (owner: 10Ahmon Dancy) [22:50:59] (03PS2) 10Ahmon Dancy: Semi-working webserver setup [tools/train-dev] - 10https://gerrit.wikimedia.org/r/631509 [22:54:42] (03PS3) 10Ahmon Dancy: Semi-working webserver setup [tools/train-dev] - 10https://gerrit.wikimedia.org/r/631509 (https://phabricator.wikimedia.org/T260674) [23:47:39] (03CR) 10Jeena Huneidi: PipelineStage: fix default chart names (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/631301 (https://phabricator.wikimedia.org/T264236) (owner: 10Jeena Huneidi) [23:50:40] (03PS4) 10Ahmon Dancy: Semi-working webserver setup [tools/train-dev] - 10https://gerrit.wikimedia.org/r/631509 (https://phabricator.wikimedia.org/T260674)