[06:18:00] <wikibugs>	 (03CR) 10Hashar: [C:03+2] zuul: [mediawiki/extensions/VisualEditor] Add GlobalBlocking dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1292060 (https://phabricator.wikimedia.org/T427086) (owner: 10Harroyo-wmf)
[06:19:50] <wikibugs>	 (03Merged) 10jenkins-bot: zuul: [mediawiki/extensions/VisualEditor] Add GlobalBlocking dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1292060 (https://phabricator.wikimedia.org/T427086) (owner: 10Harroyo-wmf)
[06:24:07] <wikibugs>	 (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/WikimediaCustomizations] Add GP dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1292006 (https://phabricator.wikimedia.org/T426322) (owner: 10Stoyofuku-wmf)
[06:25:54] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add GP dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1292006 (https://phabricator.wikimedia.org/T426322) (owner: 10Stoyofuku-wmf)
[07:18:36] <codders>	 hey all - I'm looking to deploy a patch in the morning backport window and none of the deployers are around. I can deploy the patch myself, but it would be good to have another pair of eyes / hands around. Can anyone support?
[07:18:52] <RhinosF1>	 hashar: anyone from releng around for ^
[07:20:44] <hashar>	 codders: RhinosF1: yes I am here
[07:21:11] <codders>	 hashar: great! - are you okay for me to deploy https://gerrit.wikimedia.org/r/c/1291951/ ?
[07:22:15] <hashar>	 I am not familiar about thesettings code, don't you need to have a default => []?
[07:22:56] <codders>	 the same variable is already deployed to labs / beta and the code is running okay without a default
[07:22:57] <hashar>	 otherwise it is for testwikidata so it is fine yes
[07:23:16] <hashar>	 wmf-config/Wikibase.php:        if ( isset( $wmgWikibaseTmpWikiProjectsLinking ) ) {
[07:23:16] <hashar>	 wmf-config/Wikibase.php:                $wgWBRepoSettings[ 'tmpWikiProjectsLinking' ] = $wmgWikibaseTmpWikiProjectsLinking;
[07:23:16] <codders>	 great. then I'll give that a go
[07:23:19] <hashar>	 ah yeah isset()
[07:23:56] <hashar>	 +1 ed
[07:24:02] * hashar heads to -operations
[08:14:34] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T423913#11953911 (10ihurbain) ##### Risky Patch!  🚂🔥  * **Change**:  https://gerrit.wikimedia.org/r/c/mediawiki/core/...
[08:15:15] <ihurbain>	 i've added a note for a risky patch for this week's train (sorry i didn't do that earlier). it SHOULD be fine, but in case it's not, yell at me :P
[08:18:42] <wikibugs>	 06Release-Engineering-Team: registry.cloud.releng.team returs 500 responses when pushing layers - https://phabricator.wikimedia.org/T427250#11953937 (10brouberol)
[08:21:04] <hashar>	 ihurbain: thank you for the note!
[08:23:50] <wikibugs>	 10Phabricator (phabricator-next), 10Phabricator Antivandalism Extension, 06Release-Engineering-Team (Doing 😎): Make it more discoverable that a Phab user account triggered AVA countermeasures - https://phabricator.wikimedia.org/T426972#11953953 (10Aklapper)
[08:39:41] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T423913#11954013 (10mszwarc)
[08:42:07] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T423913#11954033 (10Dreamy_Jazz)
[08:44:59] <wikibugs>	 10Phabricator (phabricator-next), 10Phabricator Antivandalism Extension, 06Release-Engineering-Team (Doing 😎): Make it more discoverable that a Phab user account triggered AVA countermeasures - https://phabricator.wikimedia.org/T426972#11954040 (10A_smart_kitten) >>! In T426972#11953512, @Aklapper wrote: > `...
[08:52:26] <wikibugs>	 (03CR) 10Hashar: [C:03+1] Zuul: […/FlaggedRevs] Drop GoogleNewsSitemap dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1284724 (owner: 10Jforrester)
[08:53:28] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "Nice, thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/1284719 (owner: 10Hslater)
[08:56:48] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/PDFCreator] Reclassify PDFCreator dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1284719 (owner: 10Hslater)
[09:01:43] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "I have deployed the jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/1292620 (https://phabricator.wikimedia.org/T231966) (owner: 10Hashar)
[09:04:37] <wikibugs>	 (03Merged) 10jenkins-bot: Introduce Phan composer job [integration/config] - 10https://gerrit.wikimedia.org/r/1292620 (https://phabricator.wikimedia.org/T231966) (owner: 10Hashar)
[09:05:26] <hashar>	 !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1292620 (Introduce Phan composer job - T231966)
[09:05:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[09:05:28] <stashbot>	 T231966: Phan job should use composer instead of vendor for release branches ( undeclared class \Wikimedia\Equivset\Equivset ) - https://phabricator.wikimedia.org/T231966
[09:32:52] <wikibugs>	 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services: gitlab workers ulimit nofiles 1073741816 slows down fakeroot - https://phabricator.wikimedia.org/T426827#11954282 (10fgiunchedi) My `ulimit -n 4096` bandaid is deployed in cloud/cicd/gitlab-ci now; I don't know whether there's...
[10:53:19] <wikibugs>	 (03PS3) 10Kosta Harlan: zuul: [mediawiki/extensions/Wikibase] Add ConfirmEdit dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1289940 (https://phabricator.wikimedia.org/T426089)
[11:13:51] <wikibugs>	 06Release-Engineering-Team, 06ServiceOps new, 10Tool-schedule-deployment, 13Patch-For-Review: Extend functionality to support MediaWiki infrastructure Windows and related repos - https://phabricator.wikimedia.org/T385007#11954672 (10jijiki)
[11:47:44] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T423913#11954902 (10Lucas_Werkmeister_WMDE)
[12:57:57] <wikibugs>	 (03PS1) 10Hashar: Phase out quibble --commands [integration/config] - 10https://gerrit.wikimedia.org/r/1293711 (https://phabricator.wikimedia.org/T321617)
[13:01:04] <wikibugs>	 (03PS1) 10Hashar: Remove deprecated --commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/1293712 (https://phabricator.wikimedia.org/T321617)
[13:02:39] <wikibugs>	 10Continuous-Integration-Config, 10Quibble, 13Patch-For-Review: Fully remove deprecated --commands - https://phabricator.wikimedia.org/T321617#11955120 (10hashar)
[13:05:45] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Remove deprecated --commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/1293712 (https://phabricator.wikimedia.org/T321617) (owner: 10Hashar)
[13:14:51] <wikibugs>	 (03PS1) 10Hashar: Rename cmd to cli [integration/quibble] - 10https://gerrit.wikimedia.org/r/1293713 (https://phabricator.wikimedia.org/T300727)
[13:15:00] <Lucas_WMDE>	 !log ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-docs-publish # fix failure seen in mwext-node24-docs-publish 981, 985, 987
[13:15:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:15:32] <wikibugs>	 (03CR) 10Phuedx: "You're absolutely right. Let's wait until Friday?" [integration/config] - 10https://gerrit.wikimedia.org/r/1289154 (https://phabricator.wikimedia.org/T422358) (owner: 10Phuedx)
[13:19:22] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Rename cmd to cli [integration/quibble] - 10https://gerrit.wikimedia.org/r/1293713 (https://phabricator.wikimedia.org/T300727) (owner: 10Hashar)
[13:31:22] <wikibugs>	 (03PS1) 10Phedenskog: test: This is just a test commit to trigger CI [integration/quibble] - 10https://gerrit.wikimedia.org/r/1293718
[13:41:57] <wikibugs>	 (03PS1) 10Hashar: dockerfiles: quibble: stop spawning memcached [integration/config] - 10https://gerrit.wikimedia.org/r/1293720 (https://phabricator.wikimedia.org/T397810)
[13:44:31] <wikibugs>	 (03CR) 10Hashar: [C:03+2] dockerfiles: quibble: stop spawning memcached [integration/config] - 10https://gerrit.wikimedia.org/r/1293720 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar)
[13:46:24] <wikibugs>	 (03Merged) 10jenkins-bot: dockerfiles: quibble: stop spawning memcached [integration/config] - 10https://gerrit.wikimedia.org/r/1293720 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar)
[13:55:56] <wikibugs>	 (03PS1) 10Hashar: jjb: update Quibble jobs to stop spawning memcached [integration/config] - 10https://gerrit.wikimedia.org/r/1293723 (https://phabricator.wikimedia.org/T397810)
[13:57:00] <wikibugs>	 (03CR) 10Jforrester: "Neat!" [integration/config] - 10https://gerrit.wikimedia.org/r/1293723 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar)
[13:57:46] <hashar>	 James_F: I'll see whether I can drop Supervisord entirely and have everything managed by Quibble :)
[14:01:56] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Write lightweight OCI-image-based Puppet plans for beta cluster - https://phabricator.wikimedia.org/T425585#11955344 (10bking)
[14:07:04] <hashar>	 !log Updated integration-quibble-* jobs in order to validate running tests with Supervisord not managing memcached # T397810
[14:07:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:07:09] <stashbot>	 T397810: Quibble supervisord memcached confusion - https://phabricator.wikimedia.org/T397810
[14:07:49] <hashar>	 lets see how it goes
[14:08:09] <hashar>	 James_F: I think ultimately I want to get rid of Supervisord and teach Quibble how to spin up Apache+php-fpm and OpenSearch
[14:09:23] <wikibugs>	 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#11955399 (10ABran-WMF)
[14:10:37] <wikibugs>	 (03CR) 10Hashar: "I have updated the `integration-quibble-*` jobs and exercising them at I20b293a0573121b3dd611b0b4a48f0de99c4b028" [integration/config] - 10https://gerrit.wikimedia.org/r/1293723 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar)
[14:26:06] <wikibugs>	 (03PS1) 10Hashar: utils: fullrun-extensions: s/MetricsPlatform/TestKitchen/ [integration/quibble] - 10https://gerrit.wikimedia.org/r/1293735 (https://phabricator.wikimedia.org/T407905)
[14:29:04] <wikibugs>	 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#11955494 (10ABran-WMF)
[14:30:25] <wikibugs>	 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services: Deploy Phab/Phorge 2026-05-26 - https://phabricator.wikimedia.org/T427286 (10brennen) 03NEW
[14:31:13] <wikibugs>	 (03open) 10brennen: Update submodules for 2026-05-26 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/105 (https://phabricator.wikimedia.org/T427286)
[14:35:28] <wikibugs>	 (03CR) 10Hashar: [C:03+2] utils: fullrun-extensions: s/MetricsPlatform/TestKitchen/ [integration/quibble] - 10https://gerrit.wikimedia.org/r/1293735 (https://phabricator.wikimedia.org/T407905) (owner: 10Hashar)
[14:41:12] <wikibugs>	 10Phabricator (2026-05-26), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phab/Phorge 2026-05-26 - https://phabricator.wikimedia.org/T427286#11955572 (10Aklapper) This deployment includes * Translation updates: https://gerrit.wikimedia.org/r/plugins/gitiles/phabricator/tr...
[14:43:22] <wikibugs>	 (03Merged) 10jenkins-bot: utils: fullrun-extensions: s/MetricsPlatform/TestKitchen/ [integration/quibble] - 10https://gerrit.wikimedia.org/r/1293735 (https://phabricator.wikimedia.org/T407905) (owner: 10Hashar)
[14:46:15] <hashar>	 !log Updating Quibble Jenkins jobs to stop Supervisord from spawning Memcached # T397810
[14:46:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:46:17] <stashbot>	 T397810: Quibble supervisord memcached confusion - https://phabricator.wikimedia.org/T397810
[14:51:31] <wikibugs>	 (03CR) 10Hashar: "Verified to be working via I1b46538902de9b65fe70e47a58815a1d3df56c54" [integration/config] - 10https://gerrit.wikimedia.org/r/1293723 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar)
[14:51:40] <wikibugs>	 (03CR) 10Hashar: [C:03+2] jjb: update Quibble jobs to stop spawning memcached [integration/config] - 10https://gerrit.wikimedia.org/r/1293723 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar)
[14:52:00] <wikibugs>	 (03CR) 10Phuedx: "<3" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1293735 (https://phabricator.wikimedia.org/T407905) (owner: 10Hashar)
[14:55:07] <hashar>	 phuedx: thanks :-]
[14:55:27] <hashar>	 I considered adding you as a reviewer to that change and felt like it was not worth bothering you  :b
[14:55:31] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: update Quibble jobs to stop spawning memcached [integration/config] - 10https://gerrit.wikimedia.org/r/1293723 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar)
[15:01:26] <wikibugs>	 (03merge) 10brennen: Update submodules for 2026-05-26 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/105 (https://phabricator.wikimedia.org/T427286)
[15:02:06] <wikibugs>	 10Phabricator (2026-05-26), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phab/Phorge 2026-05-26 - https://phabricator.wikimedia.org/T427286#11955705 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=003a5dd7-ceba-4260-905e-d6aa41777e6c) set by jelto@cum...
[15:02:36] <wikibugs>	 10Phabricator (2026-05-26), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phab/Phorge 2026-05-26 - https://phabricator.wikimedia.org/T427286#11955707 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=9cc0e059-aa85-4bfc-885a-a8f5f59b9915) set by jelto@cum...
[15:09:38] <James_F>	 hashar: +1
[15:14:17] <wikibugs>	 10Phabricator (2026-05-26), 06Release-Engineering-Team (Doing 😎), 06collaboration-services: Deploy Phab/Phorge 2026-05-26 - https://phabricator.wikimedia.org/T427286#11955760 (10Aklapper) 05Open→03Resolved
[15:17:33] <wikibugs>	 (03PS1) 10Jforrester: Zuul: Add PHP 8.5 to a few missing PHP pipelines, oops [integration/config] - 10https://gerrit.wikimedia.org/r/1293747
[15:18:19] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [@hahmed] - https://phabricator.wikimedia.org/T427089#11955773 (10HAhmed-WMF) @Aklapper  Hi, yes that banner did display. I linked my LDAP account to my Phabricator and have "wmf" group access
[15:20:22] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] Zuul: Add PHP 8.5 to a few missing PHP pipelines, oops [integration/config] - 10https://gerrit.wikimedia.org/r/1293747 (owner: 10Jforrester)
[15:22:15] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: Add PHP 8.5 to a few missing PHP pipelines, oops [integration/config] - 10https://gerrit.wikimedia.org/r/1293747 (owner: 10Jforrester)
[15:26:00] <James_F>	 !log Zuul: Add PHP 8.5 to a few missing PHP pipelines, oops
[15:26:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:28:10] <hashar>	 :)
[15:28:25] <hashar>	 by the way, I got us Phan jobs using composer instead of vendor
[15:29:12] <hashar>	 I'll test it against all the repos using composer instead of vendor and see what happens
[15:29:29] <hashar>	 maybe they'll just work out of the box
[15:30:03] <hashar>	 (unless that requires the phan-config composer dependency, I should reread the doc
[15:30:52] <James_F>	 I saw.
[15:31:02] <James_F>	 That'd be especially great to add to REL1_* branches.
[15:31:10] <James_F>	 A five-year-old (?) TODO now TODONE.
[15:34:55] <hashar>	 hehe
[15:35:19] <hashar>	 which makes me consider whether we should namespace zuul/phan_dependencies.yaml per branch
[15:35:37] <hashar>	 or surely have a smarter way to clone those requirements
[15:36:02] <hashar>	 I think umherirender has commented somewhere that the requirements for phan could be defiined in some composer.json extra field
[15:36:11] <hashar>	 and it is not hard to imagine we could have quibble to process them from us
[15:36:45] <hashar>	 so Foobar owuld have  { "extras" : { "phan-requires": "Something" } } 
[15:37:05] <hashar>	 CI would run the phan job by only cloning Foobar then do the requires
[15:37:25] <hashar>	 and we could use that data to generate the .phan/config.php for directories to inspect
[15:37:36] <hashar>	 E_TOO_MANY_IDEAS
[15:53:46] <wikibugs>	 (03PS1) 10JavierMonton: images: maven-21 [integration/config] - 10https://gerrit.wikimedia.org/r/1293758 (https://phabricator.wikimedia.org/T412978)
[16:00:54] <wikibugs>	 06Release-Engineering-Team: registry.cloud.releng.team returs 500 responses when pushing layers - https://phabricator.wikimedia.org/T427250#11956020 (10CDanis) Can confirm this is also affecting https://gitlab.wikimedia.org/repos/sre/CIDERGRINDER
[16:07:19] <sukhe>	 hi folks. are issues on the gitlab CI known? 
[16:07:22] <sukhe>	 > 8.695 error: unexpected status from POST request to https://registry.cloud.releng.team/v2/repos/sre/cidergrinder/cache/gobuildtest/blobs/uploads/: 500 Internal Server Error
[16:07:27] <sukhe>	 cc cdanis ^
[17:01:16] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 10Catalyst (Luka Ijo Pimeja Jan), 05MW-1.46-release: Cannot select REL1_46 on PatchDemo - https://phabricator.wikimedia.org/T425165#11956404 (10Jdforrester-WMF)
[17:13:48] <brennen>	 sukhe: https://phabricator.wikimedia.org/T427250
[17:14:05] <sukhe>	 ah thanks
[17:14:41] <wikibugs>	 06Release-Engineering-Team: registry.cloud.releng.team returns 500 responses when pushing layers - https://phabricator.wikimedia.org/T427250#11956444 (10brennen)
[17:20:34] <wikibugs>	 10GitLab (CI & Job Runners), 06Release-Engineering-Team: registry.cloud.releng.team returns 500 responses when pushing layers - https://phabricator.wikimedia.org/T427250#11956500 (10brennen) I'm guessing [[ https://gitlab.wikimedia.org/repos/releng/reggie | reggie ]] just needs a restart here.
[17:35:07] <Raine>	 o/, if I wanted to increase the test timeout for a gerrit repo, how would I do that?
[17:35:56] <Raine>	 (yes I actually sorta have a good reason to need to do that, deployment-charts CI is sometimes quite tight and I just made it a minute or two worse)
[17:54:52] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 05FY25-26 Reading Lists (Roll out to all Wikis): ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#11956687 (10aude)
[18:03:15] <bd808>	 Raine: you would need to change the job's definition and get the update published to Jenkins. https://www.mediawiki.org/wiki/Continuous_integration/Jenkins_job_builder has some pointers. but it might be easier/faster to start with a phab task and nerd snipe someone into doing the needful for you.
[18:08:06] <bd808>	 the helm-lint timeout is 30 minutes right now. If the test is slower than that regularly there is maybe something else that needs to be done?
[18:10:30] <bd808>	 The run at https://integration.wikimedia.org/ci/job/helm-lint/32487/console looks like it twiddled it's thumbs for 17 minutes. Nothing obviously explaining that in the log
[18:12:36] <Raine>	 it's not slower regularly, only in special circumstances 
[18:12:50] <bd808>	 For https://integration.wikimedia.org/ci/job/helm-lint/32478/console the undocumented pause is 22 minutes. I guess that's the runtime of the "REPORTING ON DEPLOYMENTS" step whatever that is.
[18:12:52] <Raine>	 (when the actual CI code changes) 
[18:13:20] <Raine>	 yes, and reporting on deployments usually doesn't report on _all_ of them, only the changed ones
[18:13:29] <Raine>	 except when you change the CI code 
[18:13:53] <bd808>	 https://integration.wikimedia.org/ci/job/helm-lint/buildTimeTrend shows only one test hitting the 30 minute cap, and most builds in the 7-8 minute range
[18:14:35] <Raine>	 yes, but the 30min was the one that changed CI code, the short ones didn't 
[18:23:35] <Raine>	 anyways, thank you for the pointers bd808 <3
[19:32:44] <wikibugs>	 10GitLab (CI & Job Runners), 06Release-Engineering-Team: registry.cloud.releng.team returns 500 responses when pushing layers - https://phabricator.wikimedia.org/T427250#11956931 (10thcipriani) Looks like the pvc for reggie is out of space. looking at how to clean that now.
[19:44:34] <wikibugs>	 10Gerrit, 06Release-Engineering-Team, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Figure out how to maintain cv repo - given it is primarily committed to by non-WMF - https://phabricator.wikimedia.org/T423961#11956987 (10Lars) @Eileenmcnaughton I think this one is done now?
[20:03:04] <wikibugs>	 10GitLab (CI & Job Runners), 06Release-Engineering-Team: registry.cloud.releng.team returns 500 responses when pushing layers - https://phabricator.wikimedia.org/T427250#11957060 (10thcipriani) Opted to add more space to reggie for now:  ` ❯ ./scripts/resize-pvc.py reggie 128Gi Using kubectl context "do-nyc3-c...
[20:11:00] <wikibugs>	 10GitLab (CI & Job Runners), 06Release-Engineering-Team: registry.cloud.releng.team returns 500 responses when pushing layers - https://phabricator.wikimedia.org/T427250#11957078 (10thcipriani) Seeing layers get created now in the registry logs; e.g.:  ` "POST /v2/repos/abstract-wiki/wikifunctions/function-eva...
[20:13:37] <wikibugs>	 (03open) 10brennen: reggie: increase pvc size to 128Gi [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/609 (https://phabricator.wikimedia.org/T427250)
[20:15:15] <wikibugs>	 (03merge) 10thcipriani: reggie: increase pvc size to 128Gi [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/609 (https://phabricator.wikimedia.org/T427250) (owner: 10brennen)
[20:23:53] <wikibugs>	 10GitLab (Integrations), 06Release-Engineering-Team: @CodeReviewBot is sometimes only commenting when a GitLab MR is merged, and not also when it's opened - https://phabricator.wikimedia.org/T427324 (10A_smart_kitten) 03NEW
[20:42:56] <wikibugs>	 10GitLab (Integrations), 06Release-Engineering-Team: @CodeReviewBot is sometimes only commenting when a GitLab MR is merged, and not also when it's opened - https://phabricator.wikimedia.org/T427324#11957163 (10brennen) Thanks for noting. I was //just// looking at the comments on T427250 and wondering what had...
[20:52:37] <wikibugs>	 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#11957226 (10Dzahn) I switched the gitlab-lb IPs from reserved to active state in netbox and ran the netbox sync. Meaning now they actually exist in DNS....
[21:09:37] <wikibugs>	 10Fresh, 07Documentation, 13Patch-For-Review: Add --help docs to fresh-node commands - https://phabricator.wikimedia.org/T424867#11957269 (10LazyShrey) @Krinkle any reviews for new patch?
[21:32:52] <wikibugs>	 10Gerrit, 06Release-Engineering-Team, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Figure out how to maintain cv repo - given it is primarily committed to by non-WMF - https://phabricator.wikimedia.org/T423961#11957354 (10Eileenmcnaughton) yep
[22:02:30] <wikibugs>	 10GitLab (Integrations), 06Release-Engineering-Team: @CodeReviewBot is sometimes only commenting when a GitLab MR is merged, and not also when it's opened - https://phabricator.wikimedia.org/T427324#11957439 (10A_smart_kitten)
[22:07:46] <wikibugs>	 10GitLab (Integrations), 06Release-Engineering-Team: @CodeReviewBot is sometimes only commenting when a GitLab MR is merged, and not also when it's opened - https://phabricator.wikimedia.org/T427324#11957447 (10A_smart_kitten) >>! In T427324#11957163, @brennen wrote: > This triggers a faint itching somewhere i...
[22:17:32] <wikibugs>	 10GitLab (CI & Job Runners), 06Release-Engineering-Team: registry.cloud.releng.team returns 500 responses when pushing layers - https://phabricator.wikimedia.org/T427250#11957480 (10thcipriani) New size is now persisted in the statefulset.  BUT we've added 10G to the registry in the past hour:  {F84460301}  Mo...
[22:37:14] <wikibugs>	 10GitLab (Integrations), 06Release-Engineering-Team: @CodeReviewBot is sometimes only commenting when a GitLab MR is merged, and not also when it's opened - https://phabricator.wikimedia.org/T427324#11957553 (10brennen) p:05Triage→03Low I can confirm from logs that gitlab-webhooks got two `merge_request` e...