[07:19:16] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Developer Productivity, 10Patch-For-Review, 10Regression: Quibble jobs for PHPUnit are missing log artefacts - https://phabricator.wikimedia.org/T291227 (10kostajh) 05Open→03Resolved >>! In T291227#7395741, @hashar wrote: > I have...
[07:21:52] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #28465: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/28465/
[08:08:05] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Developer Productivity, 10MW-1.38-notes (1.38.0-wmf.3; 2021-10-05), and 2 others: Quibble jobs for PHPUnit are missing log artefacts - https://phabricator.wikimedia.org/T291227 (10hashar) The patch work, though I don't really like how t...
[08:20:58] <wmf-insecte>	 Yippee, build fixed!
[08:20:58] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #28466: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/28466/
[09:03:56] <hashar>	 4g never ceases to amaze me: Receiving objects:   9% (245430/2726995), 72.25 MiB | 9.98 MiB/s
[09:20:46] <wikibugs>	 10Release-Engineering-Team (Seen), 10Release Pipeline, 10serviceops-radar: Pipeline: provide a way to rebuild all blubber images - https://phabricator.wikimedia.org/T214431 (10Joe) >>! In T214431#6879376, @thcipriani wrote: >>>! In T214431#6479172, @LarsWirzenius wrote: >> Meta comment: I have no strong opin...
[09:23:01] <hashar>	 I have cloned https://gitlab.com/gitlab-org/gitlab.git and they stopped adding a tag to their release since v11.3.0 3 years ago bah
[09:25:09] <_joe_>	 hashar: any idea where the image docker-registry.wikimedia.org/dev/stretch-php72-fpm-apache2 is built/defined?
[09:26:14] <_joe_>	 it's used in CI but it's under the /dev/ namespace and I have no idea where it's coming from
[09:27:48] <wikibugs>	 10Release-Engineering-Team (Next), 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review, 10User-brennen: GitLab minor release: 14.3.1 - https://phabricator.wikimedia.org/T292256 (10Jelto) I prepared `gitlab-ce` `14.3.2-ce.0` and `gitlab-runner` `14.3.2` on apt host
[09:32:13] <addshore>	 AntiComposite: thanks for that link!
[09:32:34] <_joe_>	 if not hashar, anyone else
[09:32:43] <_joe_>	 addshore: maybe you remember where that image comes from?
[09:32:46] <_joe_>	 it needs rebuilding
[09:32:53] <addshore>	 *reads up*
[09:33:02] <_joe_>	 /dev/stretch-php72-fpm-apache2 
[09:33:03] <addshore>	 aaah yes, that has moved to gitlab afaik
[09:33:08] <_joe_>	 no idea how that is built
[09:33:19] <addshore>	 probably https://gitlab.wikimedia.org/releng/dev-images ?
[09:33:26] <_joe_>	 but it needs to be rebuilt regularly on top of our base images
[09:34:19] <addshore>	 I don't know how and do not believe I am technically able to do things with that repo or those images
[09:34:28] <_joe_>	 yeah I do
[09:35:55] <hashar>	 I am in a phone call brb
[09:36:04] <_joe_>	 hashar: np we sorted this out
[09:42:08] <_joe_>	 addshore: any idea what is the correct ssh config to clone stuff from gitlab?
[09:42:24] <hashar>	 that is for LetsEncrypt update isn't it ?
[09:42:35] <addshore>	 You'll need to setup a key in the gitlab UI, but other than that i should be fairly normal *checks his*
[09:42:37] <_joe_>	 hashar: yes
[09:42:54] <_joe_>	 addshore: yeah I did set up the key but somehow I get permission denied
[09:43:00] <_joe_>	 I must've done something stupid
[09:43:28] <addshore>	 make sure you are not proxying?
[09:43:30] <addshore>	 ```Host *.wmnet *.wm *.wikimedia.org !gerrit.wikimedia.org !gitlab.wikimedia.org```
[09:43:44] <addshore>	 I think that might have been what caught me out before
[09:44:31] <addshore>	 With my ssh config, which I can only imagine is similar to others / the suggested, for gitlab I was trying to go through a bastion
[09:44:31] <hashar>	 I did reach out to Brennen about it, there should be no more Stretch based images since the dev images got migrated to Buster
[09:44:36] <hashar>	 hence why no images got rebuild
[09:45:54] <_joe_>	 addshore: found my issue, I registered the wrong ssh key
[09:46:15] <_joe_>	 hashar: well there are images that are based on the stretch base
[09:46:25] <_joe_>	 most notably: the one defined in mediawiki/code
[09:46:30] <_joe_>	 and the one for wwwportals
[09:46:52] <_joe_>	 ok sending patches for both
[09:55:08] <_joe_>	 actually, we don't have that image anymore
[09:55:17] <_joe_>	 so I have no idea what should I do, really
[09:57:48] <_joe_>	 hashar: so the problem is that both mediawiki/core and wikimedia/portals still reference the old stretch-based images
[09:58:06] <_joe_>	 and the ones they refer to have both apache2 and php-fpm in them (EWWWW)
[09:58:13] <_joe_>	 while the new buster images have nothing
[10:09:53] <hashar>	 _joe_: then I guess the completion from stretch to buster has not been completed
[10:10:22] <hashar>	 guess they can provide a buster image that has apache/fpm/phpX  which wuold make the migration easier
[10:10:29] <_joe_>	 hashar: I just created https://gerrit.wikimedia.org/r/c/wikimedia/portals/+/725765 but it's still very wrong
[10:11:02] <_joe_>	 hashar: I'm going to convert this to use a production image, because this is supposed to be published; but for mediawiki/core, I'll leave to y'all
[10:11:10] <hashar>	 else update the Blubber pipeline defs to use multiple containers, but I am not sure whether the Pipeline supports spawning multiple containers and linking them together (aka tells the apache container that fpm is at X:y)
[10:13:33] <_joe_>	 I also don't strictly get why we have a dev apache image that's absolutely equivalent to the production httpd one
[10:16:26] <hashar>	 _joe_: I imagine that got borrowed from the infancy of the dev images project which started like 3 years ago
[10:17:15] <hashar>	 might have been before we had a production images
[10:17:28] <hashar>	 or started with apache mod php rather than fpm
[10:17:51] <wikibugs>	 10Release-Engineering-Team (Next), 10wikimedia.biterg.io, 10Developer-Advocacy (Oct-Dec 2021), 10GitLab (Integrations): Bitergia gitlab read access for metrics - https://phabricator.wikimedia.org/T290247 (10Aklapper)
[10:19:56] <hashar>	 bah `git fetch --unshallow` does not grab tags obviously
[10:28:27] <wikibugs>	 10Release-Engineering-Team (Next), 10wikimedia.biterg.io, 10Developer-Advocacy (Oct-Dec 2021), 10GitLab (Integrations): Bitergia gitlab read access for metrics - https://phabricator.wikimedia.org/T290247 (10Aklapper)
[10:29:21] <wikibugs>	 10Release-Engineering-Team (Next), 10wikimedia.biterg.io, 10Developer-Advocacy (Oct-Dec 2021), 10GitLab (Integrations): Bitergia gitlab read access for metrics - https://phabricator.wikimedia.org/T290247 (10Aklapper) IIUC this should be actionable for WMF now, but there is a request from Bitergia in non-pu...
[10:31:44] <majavah>	 any phab admins around? https://phabricator.wikimedia.org/T271825#7397863 is yet another spam comment in the same task
[10:39:32] <Thiemo_WMDE>	 The spam bot is back, for the 3rd time on the same ticket. Is there a way to stop this? https://phabricator.wikimedia.org/p/Bernardloyd/
[10:40:25] <RhinosF1>	 I can disable the account
[10:41:09] <RhinosF1>	 Assuming it's the same bot, we could get a CU
[10:42:38] * RhinosF1 does some asking
[10:45:46] <RhinosF1>	 Thiemo_WMDE: underlying IP blocked
[10:46:07] <RhinosF1>	 majavah: ^
[10:54:08] <wikibugs>	 10Project-Admins: Create project tag for <safety-survey> - https://phabricator.wikimedia.org/T292081 (10Aklapper) @Madalina: Hi, is this a time-limited project, or will this be ongoing forever, basically? Is there any further information available, e.g. some wiki page?
[11:03:02] <wikibugs>	 10Release-Engineering-Team: mediawiki/core's pipeline references the discontinued docker-registry.wikimedia.org/dev/stretch-php72-fpm-apache2-blubber - https://phabricator.wikimedia.org/T292406 (10Joe)
[11:42:03] <hashar>	 majavah: RhinosF1: I have deleted the comment
[11:42:08] <hashar>	 +1 on having a checkuser performed
[11:42:54] <hashar>	  iguess it can be done against each of the three accounts on https://phabricator.wikimedia.org/T271825#7372753
[11:43:31] <hashar>	 I don't think I have the rights to do it
[12:13:04] <RhinosF1>	 hashar: it was done and IP blocked
[13:02:28] <wikibugs>	 10Project-Admins: Create project tag for <safety-survey> - https://phabricator.wikimedia.org/T292081 (10Madalina)
[13:07:22] <wikibugs>	 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10CAS-SSO, 10GitLab (Auth & Access): Attempting to login to gitlab.wikimedia.org sometimes results in CAS 500 Internal Server Error - https://phabricator.wikimedia.org/T291964 (10Michael) I also got this error just now when trying to log int...
[13:08:01] <wikibugs>	 10Project-Admins: Create project tag for <safety-survey> - https://phabricator.wikimedia.org/T292081 (10Madalina) @Aklapper  It is a time-limited project for the TST team for now, I don't know what will happen after we hand it off. The intent is to deploy a survey to measure safety perception and we're building...
[13:14:32] <hashar>	 jelto: hello, I have send a patch to add content-security-policy to gitlab, and somehow it seems the `::gitlab` class is not applied in the puppet compiler :\  https://gerrit.wikimedia.org/r/c/operations/puppet/+/725012/   
[13:17:01] <jelto>	 hashar: thats right, currently the gitlab modules are not used, only the "placeholder" profile. We are going to include the class soon (see https://gerrit.wikimedia.org/r/c/operations/puppet/+/724430). I would expect 1-2 weeks. If its urgent I would adopt this to the ansible code, if not I would suggest to wait until we are using the gitlab module
[13:17:22] <hashar>	 AHHHH
[13:18:39] <hashar>	 and I was getting confused cause the compiled catalog does show a resource for `apt::package_from_component{'gitlab-ce'`
[13:19:07] <hashar>	 which is defined in `gitlab::init` but turns out it comes from `::profile::gitlab` ;)
[13:19:57] <jelto>	 yes the current situation is not optimal. But we are about to migrate to the gitlab modules and remove ansible
[13:20:11] <hashar>	 I will rebase my puppet patch
[13:20:39] <hashar>	 where is the Ansible code hosted?  I would like to get the content-security-policy applied :]
[13:21:38] <jelto>	 hashar: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/gitlab-ansible, configfile is here: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/gitlab-ansible/+/refs/heads/master/roles/gitlab_server/templates/gitlab.rb.j2 Just tag or ping me if you need a review/help
[13:22:52] <hashar>	 amazing, going to copy paste stuff
[13:23:51] * hashar invokes erb2jinja2
[13:36:11] <wikibugs>	 10Release-Engineering-Team (Seen), 10local-charts: mediawiki/core's pipeline references the discontinued docker-registry.wikimedia.org/dev/stretch-php72-fpm-apache2-blubber - https://phabricator.wikimedia.org/T292406 (10brennen) I think the dev variant may be essentially abandonware at this point and not actua...
[13:52:33] <wikibugs>	 10Release-Engineering-Team, 10GitLab (Auth & Access): Gitlab 2fa password validation seems bugged - https://phabricator.wikimedia.org/T292431 (10brennen) I have a guess about what's going on here - users don't really //have// a GitLab password, as such, since they're authenticated through idp.wikimedia.org.  W...
[14:04:44] <hashar>	 jelto: turned out to be quite easy https://gerrit.wikimedia.org/r/c/operations/gitlab-ansible/+/725900  ;)
[14:08:20] <jelto>	 hashar: great! I will take a look after successful gitlab upgrade, one moment
[14:09:09] <hashar>	 I might have gotten it right
[14:09:24] <hashar>	 as long as it is report only, we can put any directive that would have no impact
[14:11:16] <jelto>	 https://grafana.wikimedia.org/dashboards/f/mtrpIBZ7z/gitlab
[14:12:28] <wikibugs>	 10Release-Engineering-Team, 10mwcli, 10GitLab (CI & Job Runners), 10User-brennen: Increase maximum artifacts size for gitlab CI - https://phabricator.wikimedia.org/T292372 (10brennen) From the list of settings: https://docs.gitlab.com/ee/api/settings.html#list-of-settings-that-can-be-accessed-via-api-calls...
[14:20:10] <wikibugs>	 10Release-Engineering-Team, 10mwcli, 10GitLab (CI & Job Runners), 10User-brennen: Increase maximum artifacts size for gitlab CI - https://phabricator.wikimedia.org/T292372 (10Addshore) >>! In T292372#7398731, @brennen wrote: > We've currently got: >  > ` > brennen@gitlab1001:~$ sudo du -hs /var/opt/gitlab/...
[14:21:01] <hashar>	 jelto: brennen proposed to test my CSP patch on gitlab-test . But obviously after the today gitlab upgrade ;)
[14:21:11] <hashar>	 I am off for a bit
[14:23:36] <wikibugs>	 10Release-Engineering-Team, 10mwcli, 10GitLab (CI & Job Runners), 10User-brennen: Increase maximum artifacts size for gitlab CI - https://phabricator.wikimedia.org/T292372 (10Addshore) `default_artifacts_expire_in` -> `The syntax of duration is described in artifacts:expire_in and the default value is 30 d...
[14:31:35] <brennen>	 !log gitlab1001: upgrading to 14.3.2 (T292256)
[14:31:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:31:39] <stashbot>	 T292256: GitLab minor release: 14.3.1 - https://phabricator.wikimedia.org/T292256
[14:32:23] <wikibugs>	 10Release-Engineering-Team (Next), 10serviceops, 10GitLab (Infrastructure), 10User-brennen: GitLab minor release: 14.3.1 - https://phabricator.wikimedia.org/T292256 (10Jelto) upgrade of `gitlab2001` to `14.3.2-ce.0` was successful.
[14:41:34] <wikibugs>	 10Release-Engineering-Team (Next), 10serviceops, 10GitLab (Infrastructure), 10User-brennen: GitLab minor release: 14.3.1 - https://phabricator.wikimedia.org/T292256 (10brennen) 05Open→03Resolved a:03brennen
[15:19:21] <wikibugs>	 10Release-Engineering-Team, 10mwcli, 10GitLab (CI & Job Runners), 10User-brennen: Increase maximum artifacts size for gitlab CI - https://phabricator.wikimedia.org/T292372 (10brennen) In the short term, I'll put together a MR on gitlab-settings here to bump artifact size and decrease retention a bit.  Long...
[15:21:55] <wikibugs>	 10Release-Engineering-Team, 10mwcli, 10GitLab (CI & Job Runners), 10User-brennen: Increase maximum artifacts size for gitlab CI - https://phabricator.wikimedia.org/T292372 (10Addshore) >>! In T292372#7399078, @brennen wrote: > In the short term, I'll put together a MR on gitlab-settings here to bump artifa...
[15:42:10] <wikibugs>	 (03PS1) 10Jforrester: Zuul: Add Lindsay Wardell personal e-mail to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/725954
[15:45:16] <wikibugs>	 10Project-Admins: Create dedicated project tag for User-dcaro - https://phabricator.wikimedia.org/T292450 (10dcaro)
[15:45:41] <wikibugs>	 10Project-Admins: Create dedicated project tag for User-dcaro - https://phabricator.wikimedia.org/T292450 (10dcaro) Note that I'll delete (or ask to delete) the old tag when I moved everything to the new.
[15:46:07] <wikibugs>	 (03CR) 10Inductiveload: Add examples extension to the CI gate (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/725304 (https://phabricator.wikimedia.org/T292288) (owner: 10Inductiveload)
[15:48:32] <wikibugs>	 (03PS4) 10Jforrester: jjb: Drop REL1_31 from and add REL1_36/1_37 support to meta-jobs [integration/config] - 10https://gerrit.wikimedia.org/r/683751
[15:50:07] <wikibugs>	 (03PS1) 10Ahmon Dancy: build-mv-image: Handle mid-week L10N rebuilds gracefully [tools/release] - 10https://gerrit.wikimedia.org/r/725959
[15:50:32] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/683751 (owner: 10Jforrester)
[15:50:51] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] jjb: [quibble] Update assert-no-errors to drop REL1_31 skip [integration/config] - 10https://gerrit.wikimedia.org/r/683752 (owner: 10Jforrester)
[15:52:07] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] Zuul: Add Lindsay Wardell personal e-mail to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/725954 (owner: 10Jforrester)
[15:52:15] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: Drop REL1_31 from and add REL1_36/1_37 support to meta-jobs [integration/config] - 10https://gerrit.wikimedia.org/r/683751 (owner: 10Jforrester)
[15:53:57] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: Add Lindsay Wardell personal e-mail to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/725954 (owner: 10Jforrester)
[15:54:24] <wikibugs>	 (03PS2) 10Ahmon Dancy: build-mv-image: Handle mid-week L10N rebuilds gracefully [tools/release] - 10https://gerrit.wikimedia.org/r/725959
[15:54:40] <James_F>	 !log Zuul: Add Lindsay Wardell personal e-mail to CI allow list
[15:54:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:55:23] <wikibugs>	 (03PS4) 10Jforrester: jjb: [quibble] Update assert-no-errors to drop REL1_31 skip [integration/config] - 10https://gerrit.wikimedia.org/r/683752
[15:55:26] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] "…" [integration/config] - 10https://gerrit.wikimedia.org/r/683752 (owner: 10Jforrester)
[15:57:24] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: [quibble] Update assert-no-errors to drop REL1_31 skip [integration/config] - 10https://gerrit.wikimedia.org/r/683752 (owner: 10Jforrester)
[16:03:06] <wikibugs>	 (03PS5) 10Jforrester: dockerfiles: [composer-scratch] Upgrade to 2.0.14 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/683030 (https://phabricator.wikimedia.org/T279857)
[16:15:04] <wikibugs>	 10Release-Engineering-Team, 10Infrastructure-Foundations, 10Puppet, 10User-brennen: logspam-watch: UTF-8 errors for some input - https://phabricator.wikimedia.org/T292246 (10hashar) That seems to be related to  `use open :encoding(UTF-8)` which causes perl to expect valid unicode coming from `cat exception...
[16:22:59] <wikibugs>	 10Release-Engineering-Team (Next), 10wikimedia.biterg.io, 10Developer-Advocacy (Oct-Dec 2021), 10GitLab (Integrations), 10User-brennen: Bitergia gitlab read access for metrics - https://phabricator.wikimedia.org/T290247 (10brennen)
[16:26:16] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Composer, 10Patch-For-Review: Re-build CI containers with Composer 2.x - https://phabricator.wikimedia.org/T279857 (10Jdforrester-WMF) 05Open→03In progress
[16:29:49] <wikibugs>	 (03CR) 10Jforrester: dockerfiles: [composer-scratch] Upgrade to 2.0.14 and cascade (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/683030 (https://phabricator.wikimedia.org/T279857) (owner: 10Jforrester)
[16:35:09] <wikibugs>	 (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/SportsTeams] Add BlogPage as phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/725477 (owner: 10Umherirrender)
[16:42:16] <wikibugs>	 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.38.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T281167 (10Pchelolo) T280806 will likely show up during a train deploy. We're removing old deprecated Action API parameters, and after many many many announcements there's...
[16:42:18] <wikibugs>	 (03PS1) 10Jforrester: Stop branching GettingStarted for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/725980 (https://phabricator.wikimedia.org/T235752)
[16:43:30] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/SportsTeams] Add BlogPage as phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/725477 (owner: 10Umherirrender)
[16:44:44] <wikibugs>	 (03CR) 10Jforrester: [C: 04-2] "@Krinkle, do you still need this?" [integration/config] - 10https://gerrit.wikimedia.org/r/720153 (owner: 10Jforrester)
[16:45:50] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/SportsTeams] Add BlogPage as phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/725477 (owner: 10Umherirrender)
[16:46:38] <James_F>	 !log Zuul: [mediawiki/extensions/SportsTeams] Add BlogPage as phan dependency
[16:46:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:55:42] <wikibugs>	 (03CR) 10Reedy: dockerfiles: [composer-scratch] Upgrade to 2.0.14 and cascade (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/683030 (https://phabricator.wikimedia.org/T279857) (owner: 10Jforrester)
[16:58:45] <wikibugs>	 (03PS6) 10Jforrester: dockerfiles: [composer-scratch] Upgrade to 2.1.8 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/683030 (https://phabricator.wikimedia.org/T279857)
[16:59:05] <wikibugs>	 (03CR) 10Jforrester: dockerfiles: [composer-scratch] Upgrade to 2.1.8 and cascade (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/683030 (https://phabricator.wikimedia.org/T279857) (owner: 10Jforrester)
[17:01:08] <James_F>	 Heya Reedy.
[17:02:12] <Reedy>	 Ohai
[17:11:36] <wikibugs>	 (03CR) 10Urbanecm: [C: 03+1] Stop branching GettingStarted for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/725980 (https://phabricator.wikimedia.org/T235752) (owner: 10Jforrester)
[17:17:22] <wikibugs>	 10Release-Engineering-Team (Doing), 10Scap, 10serviceops: Deploy Scap version 4.0.2 - https://phabricator.wikimedia.org/T291095 (10jijiki) >>! In T291095#7395126, @dancy wrote: >>>! In T291095#7390898, @jijiki wrote: >> @dancy it would be lovely if we can speed this up, right now we have `deploy1002` and `ma...
[17:49:47] <wikibugs>	 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.38.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T281167 (10Umherirrender)
[18:01:03] <wikibugs>	 (03CR) 10Reedy: [C: 03+1] dockerfiles: [composer-scratch] Upgrade to 2.1.8 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/683030 (https://phabricator.wikimedia.org/T279857) (owner: 10Jforrester)
[18:05:26] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] dockerfiles: [composer-scratch] Upgrade to 2.1.8 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/683030 (https://phabricator.wikimedia.org/T279857) (owner: 10Jforrester)
[18:07:09] <wikibugs>	 (03Merged) 10jenkins-bot: dockerfiles: [composer-scratch] Upgrade to 2.1.8 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/683030 (https://phabricator.wikimedia.org/T279857) (owner: 10Jforrester)
[18:12:52] <James_F>	 !log Docker: Publish composer 2.x-based images for T279857
[18:12:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[18:12:56] <stashbot>	 T279857: Re-build CI containers with Composer 2.x - https://phabricator.wikimedia.org/T279857
[18:17:00] <wikibugs>	 (03PS1) 10Jeena Huneidi: WIP: mw is deployable to k8s in traindev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993)
[18:17:25] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] WIP: mw is deployable to k8s in traindev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi)
[18:21:57] <wikibugs>	 (03PS2) 10Jeena Huneidi: WIP: mw is deployable to k8s in traindev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993)
[18:22:21] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] WIP: mw is deployable to k8s in traindev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi)
[18:24:23] <wikibugs>	 (03PS3) 10Jeena Huneidi: WIP: mw is deployable to k8s in traindev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993)
[18:26:23] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Composer: Re-build CI containers with Composer 2.x - https://phabricator.wikimedia.org/T279857 (10Jdforrester-WMF) https://gitlab.wikimedia.org/releng/dev-images/-/merge_requests/7 for upgrading the dev-images.
[18:34:29] <wikibugs>	 (03CR) 1020after4: [C: 03+2] "Looks like this is fully removed from prod via config so +2" [tools/release] - 10https://gerrit.wikimedia.org/r/725980 (https://phabricator.wikimedia.org/T235752) (owner: 10Jforrester)
[18:34:52] <wikibugs>	 (03CR) 10Jforrester: [C: 04-2] "Not yet." [tools/release] - 10https://gerrit.wikimedia.org/r/725980 (https://phabricator.wikimedia.org/T235752) (owner: 10Jforrester)
[18:35:19] <wikibugs>	 (03CR) 10Jforrester: [C: 04-2] "(We just agreed in CI that we'd let a train go by first just in case it needed to be re-enabled.)" [tools/release] - 10https://gerrit.wikimedia.org/r/725980 (https://phabricator.wikimedia.org/T235752) (owner: 10Jforrester)
[19:09:00] <brennen>	 dancy: i see disk is full on runner-1002 again; any objection to my running clear-docker-cache at the moment so i can upgrade the runner?
[19:35:31] <wikibugs>	 (03PS1) 10Jforrester: jjb: Switch everything to images using composer 2.x not 1.x [integration/config] - 10https://gerrit.wikimedia.org/r/726039 (https://phabricator.wikimedia.org/T279857)
[19:44:30] <hashar>	 brennen: hi, do we have any written process to migrate repositories from Gerrit to Gitlab? 
[19:45:09] <hashar>	 earlier today there was mention of some project using some obsolete images from the /dev/ namespace which are now on https://gitlab.wikimedia.org/releng/dev-images.git/
[19:45:16] <hashar>	 but the Gerrit repository hasn't been archived ;)
[19:45:38] <James_F>	 hashar: Oh, I did a search for uses when we migrated and fixed them.
[19:45:42] <James_F>	 hashar: Did I miss one? :-(
[19:46:37] <brennen>	 James_F: i commented on that ticket, there's a use of a very old php72 image in the .pipeline/blubber.yaml, but i don't _think_ that's actually being built anywhere.
[19:46:48] <brennen>	 (sorry, the mw/core/.pipline/...)
[19:47:08] <James_F>	 Oh, hmm, yeah I think core's blubber file is ignored now.
[19:47:09] <brennen>	 hashar: i don't think we really have a migration guide written yet.  it's something we should do.
[19:47:14] <hashar>	 James_F: in the backlog here Giuseppe mentioned a few uses from .pipeline/config.yaml files such as in wikimedia/portals and one in mediawiki/core docker-compose definition
[19:47:50] <James_F>	 Ah.
[19:47:56] <brennen>	 https://phabricator.wikimedia.org/T292406#7398539
[19:48:29] <brennen>	 they are not used in docker-compose, to be clear.
[19:48:37] <James_F>	 Yeah.
[19:48:48] <dancy>	 brennen: Go for it
[19:48:57] <hashar>	 and there was some question about us having a /dev/ stretch image that came with apache + php-fpm , while for Buster that is now two images ( buster-apache2 and buster-phpXX)
[19:49:16] <brennen>	 i did just find https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/libs/metrics-platform/+/7bdd3d8851c00c92cca47ba16496d0ae0dcf22c7/.pipeline/blubber/php.yaml#4 - which i think is also an image we don't update anymore
[19:50:09] <brennen>	 hashar: we no longer maintain such a /dev/ image, although i wonder if the /dev/stretch-* ones shouldn't be deleted from the registry at this point.
[19:50:49] <hashar>	 guess that will be a good way to find what breaks
[19:51:01] <hashar>	 (at the expense of loosing those images but maybe it is not a big deal)
[19:51:09] <brennen>	 yeah, i'm not entirely sure.
[19:51:28] <dancy>	 give them a new tag first, then untag the old name. 
[19:51:38] <brennen>	 the docs i found when jeena and i last looked at this basically said "things shouldn't be deleted from the registry" so i'm not clear.
[19:51:40] <hashar>	 as for moving repos, we definitely need a guide (or even better an automatic runbook !)
[19:51:42] <dancy>	 (if you want to retain them but make them inaccessible)
[19:52:29] <brennen>	 re: dev-images not being archived on gerrit, that's probably my mistake - i thought that had been done.
[19:52:47] <hashar>	 maybe we can announce on wikitech-l  / somewhere on slack  that the  /dev/stretch-* images are obsolete and should no more be used?  They have a broken lets encrypt so hopefully folks will find out rather quickly
[19:54:25] * brennen updates README and archives
[19:54:26] <hashar>	 brennen: no worries, also it is lacking `refs/changes/*` and `refs/notes/review` which we might want to carry over to gitlab even if it does not use them.  That carries the whole history and I guess we might want to keep those in git if we ever decommission gerrit entirly
[19:54:27] <hashar>	 ;)
[19:55:14] <hashar>	 for archiving https://phabricator.wikimedia.org/tag/projects-cleanup/ tasks are acted on by various volunteers (though really we should automatize a lot of that workflow).  There is a link on the left to easily file an archival request
[19:55:35] <hashar>	 maybe we can use a subproject for gitlab with a similar prefiled form
[19:55:49] <hashar>	 then people can ask in Phabricator to get their repos moved
[19:56:08] <hashar>	 and we could add an eligibility disclaimer at the top of the template task listing which repos we can move
[19:56:16] <hashar>	 oh joy too many ideas :\
[19:58:19] <dancy>	 skull and crossbones emoji
[19:59:02] <brennen>	 ...i think in fact the repo is already archived.
[19:59:55] <hashar>	 James_F: about nodeJS on CI, my understanding is we are 99% on nodeJs 12 (fresh being the last one cause of qemu)  + npm 7
[20:00:43] <hashar>	 but do we go for node 14? I  think that was the original intent until we missed a Debian package for it ;D
[20:03:16] <wikibugs>	 (03CR) 10Hashar: "This one is an interesting use case, I am following up on the task to discuss about it" [integration/config] - 10https://gerrit.wikimedia.org/r/725304 (https://phabricator.wikimedia.org/T292288) (owner: 10Inductiveload)
[20:03:48] <legoktm>	 I was expecting that at some point, we'd have Git redirects in place, so if you try pulling a migrated repository, it'll pull from GitLab instead
[20:07:25] <James_F>	 hashar: Yeah, I’m not sure. For now node12 suffices.
[20:10:15] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-extensions-Examples, 10Patch-For-Review: Add examples extension to the CI gate - https://phabricator.wikimedia.org/T292288 (10hashar) This one is an interesting use case. The extension-gate is intended to test together MediaWiki and extensions deployed to Wikimedi...
[20:10:23] <hashar>	 James_F: sold, thx ;)
[20:10:44] <James_F>	 :-)
[20:10:48] <hashar>	 oh
[20:11:02] <hashar>	 the context was https://phabricator.wikimedia.org/T267888  asked by timo
[20:11:09] <hashar>	 but maybe we can lower its prio
[20:12:34] <hashar>	 legoktm: I would rather break the links, leaving the gerrit repo almost empty with some ARCHIVED.md file pointing to the new repo (more or less what we do for extensions that got moved to github)
[20:12:57] <legoktm>	 that seems unnecessarily user hostile
[20:14:37] <legoktm>	 To this day I still find docs/git clones that have the old and now broken /r/p/ Gerrit URL syntax
[20:15:22] <hashar>	 that one has a redirect cause it is was rather simple to implement and has zero maintenance cost
[20:16:06] <hashar>	 but I guess that will be discussed in a to be written doc/process about moving repos from gerrit to gitlab
[20:16:39] <legoktm>	 um, there's no redirect for /r/p/
[20:16:47] <legoktm>	 or if there is supposed to be, it's not working
[20:17:26] <hashar>	 $ git ls-remote https://gerrit.wikimedia.org/r/p/test/gerrit-ping
[20:17:26] <hashar>	 warning: redirecting to https://gerrit.wikimedia.org/r/test/gerrit-ping/
[20:17:26] <hashar>	 57948442427a3edb7b8676a1efd9f04395816d44	HEAD
[20:17:38] <legoktm>	 anyways, not setting up redirects to GitHub makes sense because it's not under our control and sending people unknowingly to third-parties is bad. But GitLab is ours and it should also be a logical 1:1 mapping to maintain
[20:18:40] <legoktm>	 $ git clone https://gerrit.wikimedia.org/r/p/test/gerrit-ping
[20:18:40] <legoktm>	 Cloning into 'gerrit-ping'...
[20:18:40] <legoktm>	 fatal: project p/test/gerrit-ping not found
[20:18:40] <legoktm>	 fatal: Could not read from remote repository.
[20:18:43] <legoktm>	 :shrug:
[20:18:57] <hashar>	 nice
[20:20:55] <hashar>	 GIT_TRACE_CURL=1 git clone https://gerrit.wikimedia.org/r/p/test/gerrit-ping
[20:21:12] <hashar>	 would give a bunch of details, including the GET request ( GET /r/p/test/gerrit-ping/info/refs?service=git-upload-pack HTTP/1.1 )
[20:21:43] <hashar>	 which should bring back a received header without the /p/ : `Location: https://gerrit.wikimedia.org/r/test/gerrit-ping/info/refs?service=git-upload-pack`
[20:22:12] <legoktm>	 doesn't seem like it's having any effect
[20:22:32] <legoktm>	 oh, it might be trying to clone it over ssh
[20:22:50] <hashar>	 due to some insteadOf config maybe?
[20:22:54] * legoktm tries in a fresh container
[20:23:13] <legoktm>	 yeah :(
[20:23:23] <legoktm>	 so the redirect mostly works :)
[20:23:26] <legoktm>	 sorry about that
[20:23:30] <hashar>	 well
[20:24:07] <hashar>	 I rather spend 10 minutes figuring out what is the env variable to dump headers (it is `GIT_TRACE_CURL=1`) rather than you raging each time you encounter the issue! :-]
[20:24:24] <hashar>	 a happy stack requires an happy Kunal!
[20:24:30] <legoktm>	 <3
[20:25:01] <hashar>	 if you use some insteadOf you might have one which says that https.../r/p/(project)  should clone from ssh:.../r/(project)
[20:30:24] <hashar>	 brennen: do you need to validate the gitlab CSP config or can I just do it with jelto?
[20:30:52] <hashar>	 I am not sure whether whether the project uses a benevolent dictator model :D
[20:31:21] <hashar>	 (no pun intended!)
[20:33:50] <brennen>	 hashar: feel free to do with jelto, just make sure it's actually tested somewhere before throwing it at the production one. :)
[20:37:14] <hashar>	 brennen: I took note of your excellent suggestion to deploy first on the test gitlab :D
[20:37:24] <hashar>	 (or really I would just have git push to prod)
[20:38:05] <hashar>	 we shall see. I am escaping for the night
[21:02:12] <brennen>	 !log gitlab-runners: upgrading each shared runner to 14.3.2
[21:02:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:09:48] <brennen>	 !log finished upgrading shared runners (1002 - 1005) to gitlab-runner 14.3.2 (T292256)
[21:09:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:09:55] <stashbot>	 T292256: GitLab minor release: 14.3.1 - https://phabricator.wikimedia.org/T292256
[21:31:41] <wikibugs>	 10Release-Engineering-Team (Done by Wed 06 Oct), 10Security-Team, 10ContentSecurityPolicy, 10GitLab (Administration, Settings & Policy), and 4 others: Define a Content Security Policy for GitLab - https://phabricator.wikimedia.org/T285363 (10hashar) The update for today is * Gitlab does not come with any d...
[21:59:59] <wikibugs>	 (03CR) 10Niharika29: [C: 03+1] "True! I haven't deployed anything in a long while. Happy to be off the list." [tools/release] - 10https://gerrit.wikimedia.org/r/724810 (owner: 10Jdlrobson)
[22:03:12] <wikibugs>	 (03PS2) 10Thcipriani: Remove Niharika from available deployers [tools/release] - 10https://gerrit.wikimedia.org/r/724810 (owner: 10Jdlrobson)
[22:13:58] <wikibugs>	 (03CR) 10Thcipriani: [C: 03+2] Remove Niharika from available deployers [tools/release] - 10https://gerrit.wikimedia.org/r/724810 (owner: 10Jdlrobson)
[22:15:22] <wikibugs>	 (03Merged) 10jenkins-bot: Remove Niharika from available deployers [tools/release] - 10https://gerrit.wikimedia.org/r/724810 (owner: 10Jdlrobson)
[23:01:40] <wikibugs>	 10Release-Engineering-Team, 10serviceops: docker-reporter-releng-images => docker registry: status=3/NOTIMPLEMENTED - https://phabricator.wikimedia.org/T292485 (10Dzahn)
[23:06:38] <wikibugs>	 10Release-Engineering-Team, 10serviceops: docker-reporter-releng-images => docker registry: status=3/NOTIMPLEMENTED - https://phabricator.wikimedia.org/T292485 (10Dzahn) well.. just manually starting it fixed it for now:  [deneb:~] $ sudo systemctl start docker-reporter-releng-images   23:02 <+icinga-wm> RECOV...
[23:07:03] <wikibugs>	 10Release-Engineering-Team, 10serviceops: docker-reporter-releng-images => docker registry: status=3/NOTIMPLEMENTED - https://phabricator.wikimedia.org/T292485 (10Dzahn) p:05Triage→03Low
[23:10:10] <wikibugs>	 10Release-Engineering-Team, 10mwcli, 10GitLab (CI & Job Runners), 10User-brennen: Increase maximum artifacts size for gitlab CI - https://phabricator.wikimedia.org/T292372 (10brennen) Will test this tomorrow: https://gitlab.wikimedia.org/releng/gitlab-settings/-/merge_requests/8
[23:15:20] <wikibugs>	 10Release-Engineering-Team (Doing), 10mwcli, 10GitLab (CI & Job Runners), 10User-brennen: Increase maximum artifacts size for gitlab CI - https://phabricator.wikimedia.org/T292372 (10brennen)
[23:26:09] <wikibugs>	 10Release-Engineering-Team (Doing), 10Infrastructure-Foundations, 10Patch-For-Review, 10Puppet, 10User-brennen: logspam-watch: UTF-8 errors for some input - https://phabricator.wikimedia.org/T292246 (10brennen) a:03brennen Thanks for the reproduction case. I thought this class of error was already hand...