[00:01:34] <wikibugs>	 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) ` Error: Cannot create /etc/gitlab/config_backup/latest; parent directory /etc/gitlab/config_backup does not exist Error: /Stage[main]/Gitlab::Backup/File[/etc/gitlab/c...
[09:32:05] <claime>	 FYI, I reset the systemd failure for train-presync
[09:37:38] <claime>	 cgoubert@deploy1002:~$ wc -l /var/log/train-presync/syslog.log
[09:37:39] <claime>	 0 /var/log/train-presync/syslog.log
[09:37:50] <claime>	 Did y'all get the log on releng@ ?
[10:24:28] <wikibugs>	 (03CR) 10Lucas Werkmeister (WMDE): [C: 03+1] zuul: Link to report_url if available [integration/docroot] - 10https://gerrit.wikimedia.org/r/878860 (owner: 10Lucas Werkmeister (WMDE))
[10:31:00] <wikibugs>	 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: Align and refactor GitLab restore scripts - https://phabricator.wikimedia.org/T326315 (10Jelto)
[10:38:55] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Many thanks for the patch Lucas and for the extra review Timo :)" [integration/docroot] - 10https://gerrit.wikimedia.org/r/878860 (owner: 10Lucas Werkmeister (WMDE))
[10:39:09] <hashar>	 Lucas_WMDE: I am deploying your Zuul report_url patch ;)
[10:39:13] <Lucas_WMDE>	 \o/
[10:39:49] <wikibugs>	 (03Merged) 10jenkins-bot: zuul: Link to report_url if available [integration/docroot] - 10https://gerrit.wikimedia.org/r/878860 (owner: 10Lucas Werkmeister (WMDE))
[10:39:50] <hashar>	 claime: we got a mail when the security patches failed to apply on 10/01 at 04:00 UTC
[10:41:38] <claime>	 hashar: ok, I wanted to check it wasn't a kubernetes related failure, as well as clear up the WARN it generated in icinga. Thanks.
[10:42:00] <hashar>	 !log Deployed docroot change https://gerrit.wikimedia.org/r/878860 "zuul: Link to report_url if available"
[10:42:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:42:33] <hashar>	 Lucas_WMDE: it is live, but the code might be cached for up to an hour so you might need to force refresh / clear cache
[10:59:51] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] zuul: Link to report_url if available (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/878860 (owner: 10Lucas Werkmeister (WMDE))
[11:17:14] <Lucas_WMDE>	 thanks hashar and Krinkle <3
[11:17:19] <Lucas_WMDE>	 seems to work fine after a Ctrl+F5 \o/
[12:37:29] <wikibugs>	 (03PS2) 10Clément Goubert: build-image-incr.py: No new layer on empty rsync [tools/release] - 10https://gerrit.wikimedia.org/r/868398
[12:38:41] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] build-image-incr.py: No new layer on empty rsync [tools/release] - 10https://gerrit.wikimedia.org/r/868398 (owner: 10Clément Goubert)
[12:43:42] <wikibugs>	 (03PS3) 10Clément Goubert: build-image-incr.py: Retag image on empty rsync [tools/release] - 10https://gerrit.wikimedia.org/r/868398
[12:46:40] <wikibugs>	 (03CR) 10Clément Goubert: build-image-incr.py: Retag image on empty rsync (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/868398 (owner: 10Clément Goubert)
[13:45:51] <wikibugs>	 10Release-Engineering-Team, 10SRE, 10SRE-OnFire, 10serviceops-collab, 10Sustainability: Remove old scap repositories from deploy1002 - https://phabricator.wikimedia.org/T309162 (10hashar) The issue we had was to compare the state of the repositories between the two deployment servers. One of them had som...
[14:26:25] <wikibugs>	 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: Automate GitLab version upgrade process - https://phabricator.wikimedia.org/T323569 (10ops-monitoring-bot) GitLab instance gitlab2002.wikimedia.org upgraded to version 15.5.7-ce.0
[14:26:36] <wikibugs>	 10Continuous-Integration-Config: Upgrade tox from 3.x to 4.x in CI images - https://phabricator.wikimedia.org/T326810 (10Jdforrester-WMF)
[14:34:28] <wikibugs>	 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: Automate GitLab version upgrade process - https://phabricator.wikimedia.org/T323569 (10Jelto) I started the cookbook without `--dry-run` and the same GitLab version on the replica (`gitlab2002`). Execution was successful (see above).  I'll o...
[14:34:50] <wikibugs>	 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: Automate GitLab version upgrade process - https://phabricator.wikimedia.org/T323569 (10Jelto)
[14:40:48] <wikibugs>	 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review: Update quibble jobs to run maintenance scripts via maintenance/run.php - https://phabricator.wikimedia.org/T326333 (10hashar) a:03hashar
[14:43:57] <wikibugs>	 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Jelto) Thanks for looking at the test instance! We had the same issue with `gitlab-prod-1001` afair.  There is a [wiki page](https://wikitech.wikimedia.org/wiki/GitLab/Test_In...
[14:57:45] <wikibugs>	 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review: Update quibble jobs to run maintenance scripts via maintenance/run.php - https://phabricator.wikimedia.org/T326333 (10hashar)
[15:00:17] <wikibugs>	 10GitLab (Infrastructure), 10Release-Engineering-Team, 10serviceops-collab: GitLab minor version upgrade 15.7.3 - https://phabricator.wikimedia.org/T326815 (10Jelto)
[15:27:49] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Seen): Expand Gerrit Manager permissions - https://phabricator.wikimedia.org/T234474 (10LSobanski)
[16:16:41] <wikibugs>	 (03CR) 10Ahmon Dancy: [C: 04-1] build-image-incr.py: Retag image on empty rsync (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/868398 (owner: 10Clément Goubert)
[16:19:47] <claime>	 dancy: If at one point you have a few minutes to show me how to check train-dev, I'll take it :)
[16:21:19] <dancy>	 This particular change should be testable outside of train-dev by running 'make' in release/make-container-image twice back-to-back.   The second run should exercise your codepath. 
[16:21:41] <claime>	 a'ight
[16:22:43] <dancy>	 But if you want to have the full train-dev experience (which takes a lot of storage space), git clone https://gitlab.wikimedia.org/repos/releng/train-dev.git and run `./train-dev build <path-to-some-directory-with-plenty-of-disk>` then `./train-dev ssh deploy`
[16:23:00] <claime>	 Cool, thanks!
[16:23:02] <dancy>	 Inside the deploy container you can run ~/train to get a bunch of stuff set up.
[16:24:19] <dancy>	 To test your change I cherry-picked it into <train-dev-workdir/release> and ran this inside the deploy container `scap sync-world --stop-before-sync` back to back.
[16:26:12] * claime nods
[16:26:16] <claime>	 Thanks :)
[17:20:02] <wmf-insecte>	 Project beta-update-databases-eqiad build #64375: 04FAILURE in 0.98 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/64375/
[17:31:13] <Reedy>	 guessing that's difference in composer updates landing
[17:54:46] <Krinkle>	 dancy: regarding scap broken on webperf, what caused it to break now where it worked last Friday?
[17:55:13] <Krinkle>	 Is it an upgrade on the deploy host? Or something in scap adopting something new?
[17:55:29] <Krinkle>	 I thought our own OS bump was longer ago
[17:55:33] * Krinkle checks
[17:56:29] <Krinkle>	 Yeah it's been on bullseye since October at least
[18:08:14] <dancy>	 That is a very good question and I don't have an answer.  scap self-deployment hasn't changed.
[18:08:20] <wmf-insecte>	 Yippee, build fixed!
[18:08:20] <wmf-insecte>	 Project beta-update-databases-eqiad build #64376: 09FIXED in 9 min 12 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/64376/
[18:47:00] <wikibugs>	 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) >>! In T318521#8519718, @Jelto wrote: > We had the same issue with `gitlab-prod-1001` afair. >  > There is a [wiki page](https://wikitech.wikimedia.org/wiki/GitLab/Test...
[18:53:41] <wikibugs>	 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10taavi) How do you expect Let's Encrypt to perform the http-01 validation since gitlab-prod-1002 is not currently publicly accessible? Please be extra careful here that you don...
[18:54:55] <wikibugs>	 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) I don't expect anything, I just blindly follow docs to test if they work and report what happens.
[19:18:42] <wikibugs>	 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) Yes, floating IP / DNS records have to be setup and that makes sense.
[19:36:24] <wikibugs>	 10Continuous-Integration-Config, 10MW-1.39-release, 10PHP 8.1 support: Make Phan on PHP 8.1 voting on REL1_39 - https://phabricator.wikimedia.org/T326374 (10Reedy) Noting it's disabled due to {T231966} / {T226945}
[20:14:48] <wikibugs>	 10Phabricator: batch edit for Val (VPuffetMichel) - https://phabricator.wikimedia.org/T326858 (10VPuffetMichel)
[20:16:39] <wikibugs>	 10Phabricator: batch edit for Val (VPuffetMichel) - https://phabricator.wikimedia.org/T326858 (10RhinosF1) Hi,  Can you please give a reason?  Note: Phab account is linked to a WMF Staff SUL account
[20:19:33] <wikibugs>	 10Phabricator: batch edit for Val (VPuffetMichel) - https://phabricator.wikimedia.org/T326858 (10RhinosF1) One thing I will point out is that the batch edit tool on the UI is not silenced. You require shell access for that.
[20:26:42] <wikibugs>	 10Phabricator: batch edit for Val (VPuffetMichel) - https://phabricator.wikimedia.org/T326858 (10VPuffetMichel) Hi @rhinosF1! of course.  Here's what I would like and you can tell me how to go about it.  I would like to mass edit all children of a parent task. We have just created [[ https://phabricator.wikimedi...
[20:28:50] <wikibugs>	 10Phabricator: batch edit for Val (VPuffetMichel) - https://phabricator.wikimedia.org/T326858 (10RhinosF1) Is it likely to be something you only do once or do you see a need to do it in future?
[20:35:46] <wikibugs>	 10Phabricator: batch edit for Val (VPuffetMichel) - https://phabricator.wikimedia.org/T326858 (10VPuffetMichel) I've had the need in the past when we were triaging tasks for the editing team. It can be sometimes tedious to do this manually. While I can ask someone in the team to do this, I promised myself that t...
[20:37:08] <wikibugs>	 10Phabricator: batch edit for Val (VPuffetMichel) - https://phabricator.wikimedia.org/T326858 (10RhinosF1) Not my say but I very much doubt they’ll be an issue, just helping speed up the process by asking the questions that the Phabricator team will ask.
[20:37:37] <wikibugs>	 10Phabricator: batch edit for Val (VPuffetMichel) - https://phabricator.wikimedia.org/T326858 (10RhinosF1) p:05Triage→03Medium
[20:51:11] <Krinkle>	 Reedy: I'm trying to understand why Phan on REL branches is "hard". For master, we have a job that uses vendor and one that uses composer. I don't know which one the phan job uses off-hand, but afaik it doesn't matter so long as it uses composer for dev deps on top which we do in both variants.
[20:52:00] <Reedy>	 It's a while since I've looked at it, but IIRC (and me referencing the two tasks above was me looking at the config file) we ended up being caught be some weird edge cases
[20:53:19] <Krinkle>	 in master, we maintain mediawiki-vendor for wmf basically, not for tarballs
[20:53:56] <Krinkle>	 so in a sense, the CI jobs in master using vendor aren't testing the would-be tarball eithter way, that's solely in REL branches.
[20:53:58] <Krinkle>	 right?
[20:54:15] <Reedy>	 well, wmf production is a superset of tarballs
[20:54:24] <Krinkle>	 and in REL, we re-create vendor from scratch or overwrite or whatever dropping all wmf deps and making it purely for core+bundled, right?
[20:54:27] <Reedy>	 one extra extension in tarballs, but that has no vendor requirements
[20:54:42] <Reedy>	 and yeah, exactly that
[20:54:48] <Reedy>	 we tend to "trim" it
[20:54:56] <Krinkle>	 ok
[20:55:19] <Reedy>	 on the most simple level, it is odd that this "works fine" in master, but doesn't in REL1_XX
[20:55:21] <Krinkle>	 and it seems preferred to use vendor where possible, and so that's what the master phan job uses and what we'd like for REL.
[20:56:27] <Reedy>	 oh, maybe I'm confused
[20:56:28] <Reedy>	 >Phan in 1.31 and 1.32 won't run on mediawiki-core-php72-phan-docker because it has PHP 7.2, not 7.0.
[20:57:08] <Krinkle>	 I'm assuming also that, both for master and REL, this is presumably fine for core but sometimes breaks on ext repos. Naturally, for ext repos in master we can only use vendor for wmf-deployed repos and in REL we can only use it for bundled repos.
[20:57:09] <Reedy>	 but then.. Antoine's comment later suggests there is some vendor issue
[20:57:37] <Krinkle>	 so there's presumably gonna be a composer variant of phan either way for the non-bundled ext repos
[20:57:40] <Krinkle>	 (if we want phan there)
[20:59:17] <Krinkle>	 the original issue  with AbuseFilter doesn't make sense to me. Afaik that's bundled so Equivset should be in vendor@1.33. Git says it is indeed not in vendor@REL1_33, but it is in vendor@REL1_39
[20:59:26] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201908), 10AbuseFilter, and 2 others: mediawiki/vendor REL1_* no longer ship dependencies for wmf extensions that are not in the mediawiki tarball - https://phabricator.wikimedia.org/T189560...
[20:59:31] <Krinkle>	 so possibly a genuine mistake there and not specific to Phan
[20:59:39] <Krinkle>	 but either way fixed since
[21:00:03] <Reedy>	 Maybe it's easiest to just re-enable phan and see what breaks
[21:00:22] <Reedy>	 Other than a potential REL1_39 point release before the next security release, nothing really planned offhand
[21:01:07] <Reedy>	 And it's trivial to disable again if necessary
[21:19:32] <Krinkle>	 ack, we can put it under experimental if not already
[21:30:45] <Reedy>	 I don't think it is... commented out
[21:32:25] <wikibugs>	 10Release-Engineering-Team, 10Scap: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10Krinkle) I'm trying to determine what caused Scap to break this week, when deploying to the same webperf1003 server (running Bullseye for almost a year now) from deploy1002 worked fine last...
[21:32:34] <Krinkle>	 dancy: TLDR - no closer to a cause
[21:33:12] <dancy>	 nod. :-(
[21:36:28] <wikibugs>	 10Release-Engineering-Team, 10SRE, 10SRE-OnFire, 10serviceops-collab, 10Sustainability: Remove old scap repositories from deploy1002 - https://phabricator.wikimedia.org/T309162 (10Dzahn) >>! In T309162#8519453, @hashar wrote: >  That is what this task is about:  remove repos from the deployment servers w...
[21:38:25] <wikibugs>	 (03PS1) 10Reedy: Zuul: Add mwext-php81-phan-docker to extension-quibble-php81-or-later [integration/config] - 10https://gerrit.wikimedia.org/r/879649
[21:39:57] <Reedy>	 Krinkle: ^ dunno how many other places we want to add it
[21:40:25] <Reedy>	 considering that's just extensions
[21:44:03] <wikibugs>	 (03PS2) 10Reedy: Zuul: Add mwext-php81-phan-docker to extension-quibble-php81-or-later experimental [integration/config] - 10https://gerrit.wikimedia.org/r/879649
[21:44:28] <wikibugs>	 (03CR) 10Reedy: [C: 03+2] Zuul: Add mwext-php81-phan-docker to extension-quibble-php81-or-later experimental [integration/config] - 10https://gerrit.wikimedia.org/r/879649 (owner: 10Reedy)
[21:45:08] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Zuul: Add mwext-php81-phan-docker to extension-quibble-php81-or-later experimental [integration/config] - 10https://gerrit.wikimedia.org/r/879649 (owner: 10Reedy)
[21:46:35] <wikibugs>	 (03PS3) 10Reedy: Zuul: Add PHP81 Phan to experimental extension-quibble-php81-or-later [integration/config] - 10https://gerrit.wikimedia.org/r/879649
[21:46:40] <wikibugs>	 (03CR) 10Reedy: Zuul: Add PHP81 Phan to experimental extension-quibble-php81-or-later [integration/config] - 10https://gerrit.wikimedia.org/r/879649 (owner: 10Reedy)
[21:46:43] <wikibugs>	 (03CR) 10Reedy: [C: 03+2] Zuul: Add PHP81 Phan to experimental extension-quibble-php81-or-later [integration/config] - 10https://gerrit.wikimedia.org/r/879649 (owner: 10Reedy)
[21:48:26] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: Add PHP81 Phan to experimental extension-quibble-php81-or-later [integration/config] - 10https://gerrit.wikimedia.org/r/879649 (owner: 10Reedy)
[21:49:07] <Reedy>	 !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/879649
[21:49:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:50:37] <wikibugs>	 10Release-Engineering-Team, 10SRE, 10SRE-OnFire, 10serviceops-collab, 10Sustainability: Remove old scap repositories from deploy1002 - https://phabricator.wikimedia.org/T309162 (10Krinkle) >>! In T309162#7958771, @Dzahn wrote: > Top ten oldest repos by modifiation time, oldest first: >  > ` > May 30  201...
[22:01:16] <wikibugs>	 10Scap: Scap get stuck when trying to deploy a patch chain (from latest patch) - https://phabricator.wikimedia.org/T325074 (10jeena) 05Open→03Resolved a:03jeena
[22:27:07] <wikibugs>	 10Continuous-Integration-Config, 10phan-taint-check-plugin: Create a web demo for phan taint-check - https://phabricator.wikimedia.org/T257301 (10Daimona) 05Open→03Resolved
[22:58:37] <wikibugs>	 10Release-Engineering-Team, 10SRE, 10SRE-OnFire, 10serviceops-collab, 10Sustainability: Remove old scap repositories from deploy1002 - https://phabricator.wikimedia.org/T309162 (10Dzahn) I don't remember exactly but most likely find -mtime, yea. ACK!
[22:59:58] <wikibugs>	 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) 185.15.56.117 is now new floating IP associated with gitlab-prod-1002 (we got away with existing quota).  Existing floating IP is not mapped to instance "mapped fixed a...