[05:22:32] <wmcs-alerts>	 FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[05:27:32] <wmcs-alerts>	 RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[09:00:22] <wikibugs>	 (03open) 10santhosh: python: Add support for uv package management [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/176
[09:28:35] <wikibugs>	 (03CR) 10Hashar: [C:03+2] Add Chuiimuii_ofc to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1309637 (owner: 10VadymTS1)
[09:31:59] <wikibugs>	 (03Merged) 10jenkins-bot: Add Chuiimuii_ofc to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1309637 (owner: 10VadymTS1)
[09:39:00] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "I have deployed the CI configuration change. @harshsrivastavaishere@gmail.com your change will now be processed by CI and you can now also" [integration/config] - 10https://gerrit.wikimedia.org/r/1309637 (owner: 10VadymTS1)
[09:39:38] <wikibugs>	 (03CR) 10Hashar: [C:03+2] Zuul: Add User:1F616EMO to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1311455 (owner: 10Gerrit Patch Uploader)
[09:42:02] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: Add User:1F616EMO to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1311455 (owner: 10Gerrit Patch Uploader)
[09:57:05] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07Zuul, and 2 others: gerrit: Checks information failed to load. - https://phabricator.wikimedia.org/T432548#12136107 (10LSobanski)
[09:57:06] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "I have deployed the CI configuration change. @root@1f616emo.xyz your changes will now be processed by CI and you can now also comment `rec" [integration/config] - 10https://gerrit.wikimedia.org/r/1311455 (owner: 10Gerrit Patch Uploader)
[10:18:07] <wikibugs>	 (03update) 10santhosh: python: Add support for uv package management [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/176 (https://phabricator.wikimedia.org/T384584)
[11:17:39] <wikibugs>	 (03PS4) 10Hashar: dockerfiles: add rake-ruby3.3 image [integration/config] - 10https://gerrit.wikimedia.org/r/1310997 (https://phabricator.wikimedia.org/T424966) (owner: 10Abijeet Patro)
[11:24:15] <wikibugs>	 (03PS1) 10Hashar: jjb: add translatewiki-rake3.3-rake job [integration/config] - 10https://gerrit.wikimedia.org/r/1312472 (https://phabricator.wikimedia.org/T424966)
[11:24:17] <wikibugs>	 (03PS1) 10Hashar: zuul: add experimental ruby3.3 job to translatewiki [integration/config] - 10https://gerrit.wikimedia.org/r/1312473 (https://phabricator.wikimedia.org/T424966)
[11:24:20] <wikibugs>	 (03PS1) 10Hashar: zuul: switch translatewiki from ruby 2.7 to 3.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1312474 (https://phabricator.wikimedia.org/T424966)
[11:27:11] <wikibugs>	 (03CR) 10CI reject: [V:04-1] zuul: add experimental ruby3.3 job to translatewiki [integration/config] - 10https://gerrit.wikimedia.org/r/1312473 (https://phabricator.wikimedia.org/T424966) (owner: 10Hashar)
[11:27:13] <wikibugs>	 (03CR) 10CI reject: [V:04-1] jjb: add translatewiki-rake3.3-rake job [integration/config] - 10https://gerrit.wikimedia.org/r/1312472 (https://phabricator.wikimedia.org/T424966) (owner: 10Hashar)
[11:28:30] <wikibugs>	 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#12136359 (10GGalofre-WMF) Hi! As a PM, I'd need to become an admin to create new sprint boards and use options like "move tasks to a project" to move tasks in...
[11:30:12] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "The image builds properly locally and I used the `example-run.sh` script to run rake against the pending change Ib92fe1ecb8f793f0030647dd9" [integration/config] - 10https://gerrit.wikimedia.org/r/1310997 (https://phabricator.wikimedia.org/T424966) (owner: 10Abijeet Patro)
[11:34:14] <wikibugs>	 (03Merged) 10jenkins-bot: dockerfiles: add rake-ruby3.3 image [integration/config] - 10https://gerrit.wikimedia.org/r/1310997 (https://phabricator.wikimedia.org/T424966) (owner: 10Abijeet Patro)
[11:41:12] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "Image built: `docker-registry.wikimedia.org/releng/rake-ruby3.3:0.1.0`" [integration/config] - 10https://gerrit.wikimedia.org/r/1310997 (https://phabricator.wikimedia.org/T424966) (owner: 10Abijeet Patro)
[11:42:58] <wikibugs>	 (03PS2) 10Hashar: jjb: add translatewiki-rake3.3-rake job [integration/config] - 10https://gerrit.wikimedia.org/r/1312472 (https://phabricator.wikimedia.org/T424966)
[11:44:02] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "INFO:jenkins_jobs.builder:Number of jobs generated:  1" [integration/config] - 10https://gerrit.wikimedia.org/r/1312472 (https://phabricator.wikimedia.org/T424966) (owner: 10Hashar)
[11:44:18] <wikibugs>	 (03PS2) 10Hashar: zuul: add experimental ruby3.3 job to translatewiki [integration/config] - 10https://gerrit.wikimedia.org/r/1312473 (https://phabricator.wikimedia.org/T424966)
[11:44:50] <wikibugs>	 (03CR) 10Hashar: [C:03+2] zuul: add experimental ruby3.3 job to translatewiki [integration/config] - 10https://gerrit.wikimedia.org/r/1312473 (https://phabricator.wikimedia.org/T424966) (owner: 10Hashar)
[11:45:46] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: add translatewiki-rake3.3-rake job [integration/config] - 10https://gerrit.wikimedia.org/r/1312472 (https://phabricator.wikimedia.org/T424966) (owner: 10Hashar)
[11:46:41] <wikibugs>	 (03Merged) 10jenkins-bot: zuul: add experimental ruby3.3 job to translatewiki [integration/config] - 10https://gerrit.wikimedia.org/r/1312473 (https://phabricator.wikimedia.org/T424966) (owner: 10Hashar)
[11:48:00] <wikibugs>	 (03PS2) 10Hashar: zuul: switch translatewiki from ruby 2.7 to 3.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1312474 (https://phabricator.wikimedia.org/T424966)
[13:28:45] <wikibugs>	 (03PS1) 10Krinkle: jjb: Update notif channel for mobileapps hourly job [integration/config] - 10https://gerrit.wikimedia.org/r/1312508 (https://phabricator.wikimedia.org/T432593)
[13:29:06] <wikibugs>	 (03PS1) 10Krinkle: jjb: Remove mobileapps hourly job [integration/config] - 10https://gerrit.wikimedia.org/r/1312509 (https://phabricator.wikimedia.org/T432593)
[13:34:48] <User_Diegodlh>	 Hi! I need some help proposing some changes to Wikimedia's CommonSettings.php. I've read the docs but still have some doubts. Could someone help me with my questions here https://phabricator.wikimedia.org/T305571#11816864? Pinging thcipriani and lferreira-wmf as mentioned in the docs. Thanks!
[13:47:52] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07Zuul, and 2 others: gerrit: Checks information failed to load. - https://phabricator.wikimedia.org/T432548#12136748 (10Les4353) p:05Medium→03High UPDATE: Changed file list load failed. This is really critical, my god.
[13:49:04] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07Zuul, and 2 others: gerrit: Checks information failed to load. - https://phabricator.wikimedia.org/T432548#12136752 (10taavi) p:05High→03Triage ([[ https://www.mediawiki.org/wiki/Phabricator/Project_management#Setting_task...
[13:52:45] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07Zuul, and 2 others: gerrit: Checks information and changed files failed to load. - https://phabricator.wikimedia.org/T432548#12136762 (10Les4353)
[14:08:52] <wikibugs>	 (03PS3) 10Hashar: Switch translatewiki from ruby 2.7 to 3.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1312474 (https://phabricator.wikimedia.org/T424966)
[14:31:38] <wikibugs>	 (03CR) 10Phedenskog: [C:03+1] "Looks good for me." [integration/quibble] - 10https://gerrit.wikimedia.org/r/1299658 (https://phabricator.wikimedia.org/T427922) (owner: 10Mhurd)
[14:43:39] <hashar>	 User_Diegodlh: hi, I will reply on the task! :)
[14:44:01] <User_Diegodlh>	 Thank you hashar!!
[14:44:10] <hashar>	 User_Diegodlh: have you ever made a patch using Gerrit?
[14:44:31] <hashar>	 or do you know how to interact with gerrit?
[14:44:36] <User_Diegodlh>	 hashar: I haven't no
[14:45:27] <hashar>	 that is usually the most complicated part
[14:45:34] <User_Diegodlh>	 hashar  not sure how to interact either, but I guess that's something I can just read about. I'm most concerned about the right procedure to do so for the specific change I want to propose, as described in the task comment in phab
[14:45:47] <hashar>	 yeah I will answer on the task 
[14:45:59] <hashar>	 for Gerrit access you need a developer account ( https://www.mediawiki.org/wiki/Developer_account )
[14:46:37] <hashar>	 once you get one, you can follow the tutorial to setup git/gerrit on your machine  https://www.mediawiki.org/wiki/Gerrit/Tutorial
[14:47:01] <hashar>	 notably generate a ssh key and add it to your user account in Gerrit (that is covered by the tutorial)
[14:47:03] <User_Diegodlh>	 hashar awesome thanks! I do have a developer account already, sure. actually the change proposed is quite small. I am happy to learn how to do it myself, but if the change makes sense and it is easier that somebody else proposes it, i'm happy with that too
[14:47:38] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07Zuul, and 2 others: gerrit: Checks information and changed files failed to load. - https://phabricator.wikimedia.org/T432548#12137082 (10Les4353) REPRO INFO: Only can repro in Chrome / Edge 150
[14:47:39] <hashar>	 a quick alternative is to use the bot https://www.mediawiki.org/wiki/Gerrit_patch_uploader/en  to upload your files/diff
[14:48:21] <hashar>	 if you have made the change on your local machine, the tool has instructions to generate a patch/diff which you can upload via https://gerrit-patch-uploader.toolforge.org/
[14:48:29] <hashar>	 and the tool will create a change in Gerrit for you :]
[14:49:05] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07Zuul, and 3 others: gerrit: Checks information and changed files failed to load. - https://phabricator.wikimedia.org/T432548#12137088 (10Les4353)
[14:56:41] <wikibugs>	 (03PS2) 10Mhurd: Add --skip-npm-install to skip the standalone npm install [integration/quibble] - 10https://gerrit.wikimedia.org/r/1299658 (https://phabricator.wikimedia.org/T427922)
[15:00:16] <hashar>	 User_Diegodlh: https://phabricator.wikimedia.org/T305571#12137168
[15:02:18] <dancy>	 hashar: I have questions about deploying old zuul when you have some time.  In particular, I see deployment.eqiad.wmnet:/srv/deployment/zuul/deploy/src is at commit 55247e4b6c74cb4ac1eed7ab5202801f48e59e5d but there are 6 commits beyond that which exist in integration/zuul.  So my first question is... what's up with that.
[15:02:47] <User_Diegodlh>	 Amazing hashar! Thank you. I will check your reply as soon as I can this week and I will proceed with submitting the change proposal. Thank you so much for your help!
[15:08:42] <wikibugs>	 (03Abandoned) 10Ahmon Dancy: deploy: Repoint buster build at archive.debian.org [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1311933 (https://phabricator.wikimedia.org/T416707) (owner: 10Ahmon Dancy)
[15:13:40] <hashar>	 dancy: no idea?  That fork of Zuul has been stall for quite a while and I don't even remember how it is deployed :/
[15:13:58] <hashar>	 do you have changes you want to make to integration/zuul?
[15:14:08] <dancy>	 Yes: https://gerrit.wikimedia.org/r/c/integration/zuul/+/1311501
[15:24:12] <hashar>	 I am not sure we should spend time on that, then some code got written so I don't know 
[15:24:34] <hashar>	 I think the issue is the CampaignEvents chain of changes of doom and that one can be split
[15:24:40] <hashar>	 that does not fix the root cause though
[15:25:21] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07Zuul, and 2 others: gerrit: Checks information and changed files failed to load. - https://phabricator.wikimedia.org/T432548#12137339 (10LSobanski) Untagging #collaboration-services, please re-add us if needed.
[15:26:33] <hashar>	 for the deployment server I am not sure about its state
[15:28:44] <hashar>	 oh
[15:29:20] <hashar>	 dancy: so the integration/zuul/deploy.git should have the proper commit as a submodule under ./src/
[15:29:26] <hashar>	 and Gerrit is the reference
[15:29:38] <hashar>	 on one local machine, when fetching that repository and updating the submodule you get the proper state
[15:29:55] <hashar>	 the MakeFile then does a pip install inside a coontainer having the proper Debian OS
[15:30:09] <hashar>	 the resulting wheels are then added to a tarball which is added to the integration/zuul/deploy repo
[15:30:20] <hashar>	 the deployment only use that tarball, it does not rely whatsoever on the src submodule
[15:30:40] <hashar>	 on the deployment server my guess is the submodule got initialized by puppet when the repo was originally cloned
[15:31:08] <hashar>	 and the submodule there was left beind/never updated cause neither scap nor us have any neeed to update the src submodule.  The deployment artifacts are in the tarball
[15:31:22] <dancy>	 Where is this tarball that you speak of?
[15:31:34] <hashar>	 that explains the disprendancy of `./src` submodule state between the Gerrit repo and the clone on the deployment server
[15:32:25] <hashar>	 the tarballs for each target Debian versions are committed in integration/zuul/deploy.git  under `./artifacts`
[15:32:44] <dancy>	 ah, you're referring the wheels
[15:32:48] <hashar>	 yeah
[15:33:00] <hashar>	 so Zuul is deployed using wheels rather than the raw source files
[15:33:01] <dancy>	 Those are the dependencies, not the zuul software itself
[15:33:18] <dancy>	 Zuul appears to be installed via scap
[15:33:25] <hashar>	 and I think the venv has Zuul installed using the wheel that is inside the artifacts tarball
[15:34:14] <hashar>	 and scap has some scripts that invoke the Makefile to do the installation from the artifacts
[15:34:39] <hashar>	 so on the scap targets, the source code inside the src submodule reflects what is on the deployment server which is obsolete
[15:34:51] <wikibugs>	 06Release-Engineering-Team, 10Scap, 06collaboration-services: scap / puppet dependency issue on reimaged phabricator hosts - https://phabricator.wikimedia.org/T378769#12137412 (10LSobanski) a:03Arnoldokoth This process recently worked reasonably well when @Arnoldokoth upgraded Phabricator to Trixie, let's...
[15:34:56] <wikibugs>	 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#12137409 (10brennen) @GGalofre-WMF added! Happy Phorging.
[15:35:05] <hashar>	 but the installation does not use that because it uses the wheels in the artifacts tarball and I am pretty sure that includes the Zuul app as a wheel
[15:36:06] <dancy>	 https://gerrit.wikimedia.org/r/plugins/gitiles/integration/zuul/deploy/+/refs/heads/master/artifacts/bullseye/ doesn't show a zuul wheel
[15:36:22] <wikibugs>	 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Allow tools to use phabricator webhooks - https://phabricator.wikimedia.org/T321790#12137423 (10LSobanski) p:05Medium→03Low
[15:36:53] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Castor: Speedup mwext-codehealth-master-non-voting Castor job - https://phabricator.wikimedia.org/T427471#12137426 (10AMarkossyan-WMF)
[15:37:11] <wikibugs>	 10GitLab (Project Migration), 06collaboration-services: Migrate SRE repositories to GitLab - operations/debs - https://phabricator.wikimedia.org/T341991#12137429 (10LSobanski) 05Open→03Declined This is no longer actively tracked.
[15:37:20] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Castor: Speedup mwext-codehealth-master-non-voting Castor job - https://phabricator.wikimedia.org/T427471#12137433 (10Peter) @pwangai
[15:39:26] <wikibugs>	 10Diffusion, 10Cloud-Services, 06tools-platform-team: Create Conduit API method to lookup Phabricator accounts by MediaWiki user name - https://phabricator.wikimedia.org/T137001#12137464 (10valerio.bozzolan) (Thanks again) Surprisingly but understandably the API does not normalize underscores. So, "Valer...
[15:41:42] <wikibugs>	 10Diffusion, 10Cloud-Services, 06tools-platform-team: Create Conduit API method to lookup Phabricator accounts by MediaWiki user name - https://phabricator.wikimedia.org/T137001#12137483 (10Aklapper) >>! In T137001#12137464, @valerio.bozzolan wrote: > Is taking care of this normalization a welcomed sub-t...
[15:42:15] <wikibugs>	 10Diffusion, 10Cloud-Services: Create Conduit API method to lookup Phabricator accounts by MediaWiki user name - https://phabricator.wikimedia.org/T137001#12137502 (10taavi)
[15:43:37] <hashar>	 dancy: argh then I am wrong I guess :/
[15:46:32] <dancy>	 hashar: As it stands, it seems there are 6 commits that were merged into integration/zuul over the years but never deployed.  The most stable thing we could do to deploy https://gerrit.wikimedia.org/r/c/integration/zuul/+/1311501 is make a new branch starting at commit 55247e4b (the current production commit) and continue from there.
[15:49:53] <hashar>	 so Makefile.deploy does a pip install of the wheels
[15:50:01] <dancy>	 Nod.
[15:50:02] <hashar>	 AND a pip install of the what is in ./src submodule
[15:50:07] <dancy>	 Right
[15:50:15] <hashar>	 with some magic flags to prevent it from installing other dependencies
[15:50:20] <hashar>	 so we get a frozen/controlled set of dependencies
[15:50:53] <hashar>	 I think the original Makefile code had a tarball which adds dependencies AND the application as wheels but I must have refactored that at some point
[15:51:06] <dancy>	 nod.. by Jaime
[15:51:11] <hashar>	 cause having the app deployed frmo source and the dependencies as independent wheels make it easier to understand what will be deployed
[15:51:24] <hashar>	 ah yeah Jaime
[15:51:28] <hashar>	 damn my memory is so terrible this days
[15:51:28] <dancy>	 https://gerrit.wikimedia.org/r/c/integration/zuul/deploy/+/1009524
[15:51:39] <hashar>	 so it resolves that mystery
[15:51:59] <dancy>	 and https://gerrit.wikimedia.org/r/c/integration/zuul/deploy/+/998933/6
[15:52:17] <hashar>	 when I  look at the repo in Gerrit, the submodule src/   points at 61b71094 Add entry for ref-replication-scheduled event
[15:53:25] <hashar>	 and I can see the six commis
[15:53:31] <hashar>	 sudo -u deploy-zuul git -C /srv/deployment/zuul/deploy/src log --oneline --decorate origin/patch-queue/debian/jessie-wikimedia..HEAD
[15:53:49] <hashar>	 I mean, on the scap target contint1002 I can see the six commits got deployed
[15:53:58] <dancy>	 oh, interesting..
[15:54:21] <hashar>	 which should be the reference cause that is the current running/active state
[15:54:52] <wikibugs>	 10Continuous-Integration-Infrastructure, 10doc.wikimedia.org, 07Software-Licensing: integration.wikimedia.org should have a FOSS license - https://phabricator.wikimedia.org/T431663#12137571 (10Dzahn) @Tacsipacsi What do you think? Can we close this one as resolved?
[15:54:52] <hashar>	 the questions are:  what are the states of `src` on the deployment servers. Maybe we forgot to submodule update/scap does not do it/ the deployment server state got lost while switching over
[15:55:51] <hashar>	 at least the Gerrit repository integration/zuul/deploy has the submodule `src`  pointing at the same commit which is used on the scap target. The same 61b71094 Add entry for ref-replication-scheduled event
[15:55:57] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Doing 😎), 06collaboration-services: contint1003 is missing docker-buildx / auth to registry fails, causing blubber-based CI pipelines to fail - https://phabricator.wikimedia.org/T431582#12137579 (10Dzahn) Just waiting for confi...
[15:56:46] <dancy>	 hashar: Indeed I can see the change from commit 61b710948a4ec2e4b12ddc00287c8ad4ac6dc785 reflected in contint1002:/srv/deployment/zuul/venv/lib/python2.7/site-packages/zuul/connection/gerrit.py
[15:57:02] <hashar>	 \o/
[15:57:26] <dancy>	 OK.. that makes me slightly less worried.  So a git submodule update on the deployment server will make things feel less confusing perhaps
[15:57:45] <hashar>	 the source repo is in  integration/zuul and the branch used is patch-queue/debian/jessie-wikimedia  (that is a legacy branch name from when we had Zuul deployed using `git buildpackage` and pilling up patch to the sources in that branch
[15:58:11] <hashar>	 HEAD^ has the same commit 
[15:58:30] <hashar>	 and the commit on top is: e9041d9b Use upstream gear instead of our fork (4 years, 9 months ago)
[15:58:44] <hashar>	 which I guess had never been deployed
[16:00:02] <dancy>	 contint1002 has a /srv/deployment/zuul/venv/lib/python2.7/site-packages/gear-0.16.0.dist-info directory, which aligns with that change.
[16:00:28] <hashar>	 that was for your patch to randomize gear connections https://gerrit.wikimedia.org/r/c/integration/gear/+/616599/3/gear/__init__.py
[16:01:17] <hashar>	 so I think your patch got applied to the integration/gear fork, used in the list of requirements in order to have a deployable wheel that has your patch
[16:01:53] <hashar>	 and once your patch got included in a release (gear==0.16.0), that extra patch got merged to catchup with reality
[16:01:57] <hashar>	 albeit never deployed
[16:02:33] <hashar>	 which explains why the scap target are missing that commit (Use upstream gear instead of our fork)
[16:03:32] <dancy>	 I don't think the target is missing that commit.
[16:06:02] <dancy>	 Everything merged does seem to be deployed.  The only mystery is why `git submodule status` in deploy2003:/srv/deployment/zuul/deploy reports `+55247e4b6c74cb4ac1eed7ab5202801f48e59e5d src (2.5.1-25-g55247e4b)`.   But that doesn't matter anymore.    I just ran `git submodule update` and now it's up-to-date.
[16:06:09] <dancy>	 Thanks for the help researching!
[16:06:51] <dancy>	 (where up-to-date means it points at 61b710948a4ec2e4b12ddc00287c8ad4ac6dc785 now)
[16:06:52] <hashar>	 ahh
[16:07:18] <hashar>	 and the zuul/deploy repo has an artifacts/bullseye/gear-0.16.0-py2.py3-none-any.whl
[16:07:47] <hashar>	 as if the wheels got generated from the uptodate patchset-queue branch
[16:08:08] <hashar>	 so I guess the submodule never got bumped
[16:09:04] <dancy>	 My brain is tired now so I'm going to take a break for a bit. :-)
[16:09:21] <hashar>	 or the wheel has 0.16.0 but is missing the patch who knows :]
[16:14:47] <wikibugs>	 10Phabricator: Conduit API user.mediawikiquery: normalize underscores - https://phabricator.wikimedia.org/T432616 (10valerio.bozzolan) 03NEW
[16:15:50] <wikibugs>	 10Phabricator: Conduit API user.mediawikiquery: normalize underscores and first letter - https://phabricator.wikimedia.org/T432616#12137691 (10valerio.bozzolan)
[16:18:21] <wikibugs>	 10Phabricator: Conduit API user.mediawikiquery: normalize underscores and first letter - https://phabricator.wikimedia.org/T432616#12137695 (10valerio.bozzolan)
[16:19:08] <wikibugs>	 10Phabricator, 03Wikimania-Hackathon-2026: Conduit API user.mediawikiquery: normalize underscores and first letter - https://phabricator.wikimedia.org/T432616#12137696 (10valerio.bozzolan) Sounds something good for an hackathon :3
[16:24:45] <wikibugs>	 (03PS1) 10Hashar: Update Zuul src to catch up with our fork of Zuul [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1312557 (https://phabricator.wikimedia.org/T258630)
[16:25:09] <hashar>	 dancy: https://gerrit.wikimedia.org/r/c/integration/zuul/deploy/+/1312557
[16:25:17] <hashar>	 I wrote the rationale on the commit message
[16:25:43] <hashar>	 I am pretty sure I simply forgot to update the `src/` submodule to catch up with the change in integration/zuul which pinned gear=0.16.0
[16:25:48] <hashar>	 and that change is a noop
[16:26:11] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Update Zuul src to catch up with our fork of Zuul [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1312557 (https://phabricator.wikimedia.org/T258630) (owner: 10Hashar)
[16:26:17] <hashar>	 of course CI fails.. :b
[16:26:47] <dancy>	 Fun
[16:28:16] <hashar>	 looks like it is virtualenv installing a more recent version of pip
[16:31:49] <dancy>	 Ah yes. I was running into stuff like that locally.
[16:32:17] <hashar>	 that might be the pip version in docker-registry.wikimedia.org/python2-build-bullseye:latest
[16:42:50] <hashar>	 at least pip==20.3.4  internal/vcs/mercurial.py  does no have that `return [f"-r={rev}"]`
[16:42:56] <hashar>	 so something must be updating it somehow
[16:43:33] <hashar>	 and maybe that is virtualenv doing it
[16:43:43] <hashar>	 anyway that is all I have :\
[16:48:08] <dancy>	 Ill work on that problem.
[16:49:52] <hashar>	 it might be virtualenv installing a bundled vresion of pip that has erroneous dependencies
[16:51:12] <hashar>	 and iirc the job is to validate that zuul can be installed without external dependencies / the provided wheels fulfill the  list of requirements
[16:51:25] <hashar>	 so it is a failsafe to avoid missing a dep or having an erroneous wheel
[16:51:30] <hashar>	 I am off!
[17:34:06] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07Zuul, and 2 others: gerrit: Checks information and changed files failed to load. - https://phabricator.wikimedia.org/T432548#12138100 (10Aklapper) @Les4353 Please see https://www.mediawiki.org/wiki/How_to_report_a_bug and prov...
[17:58:00] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Traffic: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T432498#12138191 (10bd808) 05Open→03Invalid `lang=shell-session bd808@mbp03:~/projects/wmf$ ssh deployment-cache-text08.deployment-prep.eqiad1.wikimedia.cloud L...
[17:58:17] <wikibugs>	 10GitLab (Project Migration), 06collaboration-services, 10Observability-Alerting: Migrate SRE repositories to GitLab - operations/alerts - https://phabricator.wikimedia.org/T349626#12138208 (10Dzahn) Since T341991#12137429 was declined this might also be declined nowadays.
[17:59:02] <wikibugs>	 10GitLab (Project Migration), 06Traffic: Migrate DNS repository from Gerrit to Gitlab - https://phabricator.wikimedia.org/T355906#12138216 (10Dzahn) Since T341991#12137429 was declined this might also be declined nowadays.
[18:00:25] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07Zuul, and 2 others: gerrit: Checks information and changed files failed to load. - https://phabricator.wikimedia.org/T432548#12138222 (10Les4353)
[18:00:42] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07Zuul, and 2 others: gerrit: Checks information and changed files failed to load. - https://phabricator.wikimedia.org/T432548#12138224 (10Les4353) >>! In T432548#12138100, @Aklapper wrote: > @Les4353 Please see https://www.medi...
[18:01:19] <wikibugs>	 10Gerrit, 06Release-Engineering-Team, 07Browser-Support-Google-Chrome, 07Browser-Support-Microsoft-Edge: gerrit: Checks information and changed files failed to load. - https://phabricator.wikimedia.org/T432548#12138227 (10Les4353)
[18:17:50] <icinga-wm>	 RECOVERY - jenkins_service_running on contint1002 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins
[18:20:50] <icinga-wm>	 PROBLEM - jenkins_service_running on contint1002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins
[19:18:44] <wikibugs>	 06Release-Engineering-Team: pip install broken in Python2 virtualenvs created from python2-build-bullseye - https://phabricator.wikimedia.org/T432636 (10dancy) 03NEW
[19:36:33] <dancy>	 !log Restarting jenkins on contint1003 to clear out hours-long stuck jobs
[19:36:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:26:52] <wikibugs>	 (03PS1) 10Vaughn Walters: [DNM] jjb: [catalyst-daily-CentralNotice] Add CentralNotice job [integration/config] - 10https://gerrit.wikimedia.org/r/1312592 (https://phabricator.wikimedia.org/T427006)
[23:33:59] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Observability-Logging: scap on deployment-deploy04 failing to reach logging-logstash-04.logging.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T432254#12139355 (10colewhite) a:03colewhite I suspect the network unavailability as the cause because the victim of the...