[00:16:10] <wikibugs>	 (03open) 10bd808: reggie: Allow POST from arbitrary subnets [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/494 (https://phabricator.wikimedia.org/T396924)
[08:45:28] <wikibugs>	 06Project-Admins: Request to create project: Wikidata Reference Validator - https://phabricator.wikimedia.org/T403556#11146839 (10JosefAnthony) >>! In T403556#11142155, @Bugreporter wrote: > Where does the tool host? If it is hosted in Toolforge, you don't need to file a task here. Please instead visit https://t...
[08:50:06] <wikibugs>	 (03CR) 10Hashar: "Nice +1 code wise, there a couple glitches in the commit message and I am not sure whether it is all correct :]" [integration/docroot] - 10https://gerrit.wikimedia.org/r/1183206 (https://phabricator.wikimedia.org/T402398) (owner: 10Krinkle)
[08:51:33] <wikibugs>	 (03CR) 10Hashar: [C:03+2] CoveragePage: Nicer page titles [integration/docroot] - 10https://gerrit.wikimedia.org/r/1183207 (https://phabricator.wikimedia.org/T402398) (owner: 10Krinkle)
[08:53:34] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T403616#11146882 (10SLyngshede-WMF) 05Open→03Resolved p:05Triage→03High a:03SLyngshede-WMF
[08:58:02] <wikibugs>	 (03CR) 10Hashar: [C:03+2] CoveragePage: Hide link to cover-skins (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/1183208 (https://phabricator.wikimedia.org/T402398) (owner: 10Krinkle)
[08:58:44] <wikibugs>	 (03CR) 10Hashar: [C:03+2] Zuul: Add Nicolasmichel to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1184088 (owner: 10Daimona Eaytoy)
[09:00:14] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: Add Nicolasmichel to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1184088 (owner: 10Daimona Eaytoy)
[09:06:14] <wmcs-alerts>	 FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[09:10:02] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 07Essential-Work: Upgrade Jenkins to 2.516.2 - https://phabricator.wikimedia.org/T403703 (10hashar) 03NEW
[09:24:24] <wikibugs>	 06Project-Admins: Request to create project: Wikidata Reference Validator - https://phabricator.wikimedia.org/T403556#11147007 (10Bugreporter) You will be able to create a project yourself once you have a Phabricator tool account (you can create one if you have toolforge access). For now, you can just use #Tools...
[09:33:28] <wmcs-alerts>	 FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-kafka-jumbo-9 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:38:28] <wmcs-alerts>	 FIRING: [5x] PuppetAgentNoResources: No Puppet resources found on instance deployment-kafka-jumbo-9 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:43:28] <wmcs-alerts>	 FIRING: [10x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief06 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:48:28] <wmcs-alerts>	 FIRING: [15x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief06 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:53:28] <wmcs-alerts>	 FIRING: [19x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:54:43] <wmcs-alerts>	 FIRING: [22x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:58:28] <wmcs-alerts>	 FIRING: [24x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:59:58] <wmcs-alerts>	 FIRING: [25x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:03:23] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 07Essential-Work: Upgrade Jenkins to 2.516.2 - https://phabricator.wikimedia.org/T403703#11147125 (10MoritzMuehlenhoff) 05Open→03Resolved p:05Triage→03Medium a:03MoritzMuehlenhoff 2.516.2 has been imported, enjoy :-)
[10:03:28] <wmcs-alerts>	 FIRING: [27x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:05:13] <wmcs-alerts>	 FIRING: [27x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:08:28] <wmcs-alerts>	 FIRING: [27x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:10:28] <wmcs-alerts>	 FIRING: [27x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:13:28] <wmcs-alerts>	 FIRING: [27x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:15:43] <wmcs-alerts>	 FIRING: [27x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:18:28] <wmcs-alerts>	 FIRING: [27x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:20:58] <wmcs-alerts>	 FIRING: [27x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:23:28] <wmcs-alerts>	 FIRING: [27x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:26:13] <wmcs-alerts>	 FIRING: [27x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:28:28] <wmcs-alerts>	 FIRING: [27x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:31:28] <wmcs-alerts>	 FIRING: [27x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:33:28] <wmcs-alerts>	 RESOLVED: [27x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:49:37] <wikibugs>	 (03PS2) 10Krinkle: CoveragePage: Refactor $crumbs by slug instead of page title [integration/docroot] - 10https://gerrit.wikimedia.org/r/1183206 (https://phabricator.wikimedia.org/T402398)
[10:49:58] <wikibugs>	 (03CR) 10Krinkle: CoveragePage: Refactor $crumbs by slug instead of page title (032 comments) [integration/docroot] - 10https://gerrit.wikimedia.org/r/1183206 (https://phabricator.wikimedia.org/T402398) (owner: 10Krinkle)
[10:51:14] <wmcs-alerts>	 FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[11:12:01] <wikibugs>	 (03open) 10jnuche: jenkins-rel: update plugins to address vulnerabilities [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/105 (https://phabricator.wikimedia.org/T403623)
[11:12:54] <wikibugs>	 10Phabricator (Upstream), 07Upstream: Setting two workboard columns in one transaction: "RuntimeException: Undefined variable: board_phid" - https://phabricator.wikimedia.org/T397573#11147329 (10Aklapper) 05Open→03Stalled p:05Triage→03Low I fail to reproduce. Plus https://phabricator.wikimedia.org/feed...
[11:13:56] <wikibugs>	 (03merge) 10jnuche: jenkins-rel: update plugins to address vulnerabilities [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/105 (https://phabricator.wikimedia.org/T403623)
[11:30:24] <wikibugs>	 (03open) 10jnuche: add DEBIAN_FRONTEND=noninteractive to package installation [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/106
[11:31:03] <wikibugs>	 (03merge) 10jnuche: add DEBIAN_FRONTEND=noninteractive to package installation [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/106
[11:38:04] <wikibugs>	 10Phabricator (Search): Various AphrontQueryTimeoutQueryException in global search when setting a Tag (does not happen without tag) - https://phabricator.wikimedia.org/T353738#11147372 (10Aklapper) 05Open→03Resolved Both https://phabricator.wikimedia.org/search/query/4kx89Dvs6pj1/ and https://phabricator...
[11:44:00] <wikibugs>	 10Phabricator (Upstream), 07Upstream: When querying tasks for "sort by date", allow ignoring unimportant changes (subscribers changes, token additions, etc.) - https://phabricator.wikimedia.org/T114211#11147391 (10Aklapper) 05Open→03Declined The `dateModified` timestamp value in the database stores the...
[11:50:16] <wikibugs>	 10Phabricator (Upstream), 07Upstream: Project tags can not be found via the fulltext search index - https://phabricator.wikimedia.org/T92630#11147420 (10Aklapper) Just copying relevant bits from old upstream https://web.archive.org/web/20201023085249/https://secure.phabricator.com/T7860 called `When users sear...
[11:50:27] <wikibugs>	 10Phabricator (Upstream), 07Upstream: In fulltext search, when users search for "#x y", treat "#x" as a project hashtag - https://phabricator.wikimedia.org/T92630#11147421 (10Aklapper)
[11:50:46] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 07Essential-Work: Upgrade Jenkins to 2.516.2 - https://phabricator.wikimedia.org/T403703#11147424 (10hashar) 05Resolved→03Open @MoritzMuehlenhoff we also need the Jenkins package for Bookworm ` releases2003$ apt-cache polic...
[12:29:32] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 07Essential-Work: Upgrade Jenkins to 2.516.2 - https://phabricator.wikimedia.org/T403703#11147551 (10MoritzMuehlenhoff) >>! In T403703#11147424, @hashar wrote: > @MoritzMuehlenhoff we also need the Jenkins package for Bookworm...
[12:55:26] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 07Essential-Work: Upgrade Jenkins to 2.516.2 - https://phabricator.wikimedia.org/T403703#11147690 (10hashar) Oops, I apologize I was expecting the version table to have two entries, looks like @jnuche had already upgraded it `\...
[13:04:59] <hashar>	 oh great
[13:05:06] <hashar>	 I got a job stuck somehow 
[13:05:06] <hashar>	 https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83/11439/console
[13:05:08] <hashar>	 hhmm
[13:11:49] <hashar>	 00:44:06.643 Time: 32:50.769, Memory: 2.85 GB
[13:16:10] <wmf-insecte>	 Yippee, build fixed!
[13:16:10] <wmf-insecte>	 Project beta-code-update-eqiad build #564134: 09FIXED in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/564134/
[13:16:15] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 07Essential-Work: Upgrade Jenkins to 2.516.2 - https://phabricator.wikimedia.org/T403703#11147791 (10hashar) 05Open→03Resolved Done per https://debmonitor.wikimedia.org/packages/jenkins
[14:10:34] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MW-1.45-notes (1.45.0-wmf.16; 2025-08-26), 13Patch-For-Review, 07Technical-Debt: Drop deprecated phpunit.php and suite.xml - https://phabricator.wikimedia.org/T395470#11148087 (10hashar) >>! In T395470#11071207, @gerritbot wrote: > Change #117672...
[14:26:30] <Krinkle>	 !log Cherry-pick https://gerrit.wikimedia.org/r/1184101/ to Beta Cluster puppetserver, to make PHP 8.3 available. ref T360995
[14:26:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:26:32] <stashbot>	 T360995: Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995
[14:28:08] <Krinkle>	 !log Install php8.3 on deployment-mwmaint in Beta Cluster.  https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/14f83cf52c91c64976b13a5e7664b572156163b0 ref T360995
[14:28:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:29:16] <Krinkle>	 !log Install php8.3 on deployment-jobrunner05 in Beta Cluster.  https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/35c3149ae0fa78cd97a1ab3c45227b8b36232af7 ref T360995
[14:29:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:31:08] <Krinkle>	 swfrench-wmf: I'll monitor a bit at https://beta-logs.wmcloud.org/goto/c62613a0d80f1f0426ec42be0b0110b1 to see how the beta jobrunner goes.
[14:31:22] <Krinkle>	 https://wikitech.wikimedia.org/wiki/OpenSearch_Dashboards#Beta_Cluster_Logstash
[14:36:44] <wikibugs>	 (03merge) 10dancy: reggie: Allow POST from arbitrary subnets [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/494 (https://phabricator.wikimedia.org/T396924) (owner: 10bd808)
[14:48:46] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade): Investigate how Zuul finger gateway works - https://phabricator.wikimedia.org/T403734 (10hashar) 03NEW
[14:56:31] <wikibugs>	 (03update) 10dancy: ci: Use wmcs runners and registry.cloud.releng.team [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/52 (owner: 10bd808)
[14:58:10] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Doing 😎), 06collaboration-services: Build zuul images for production - https://phabricator.wikimedia.org/T396245#11148382 (10dduvall)
[14:58:20] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Doing 😎), 06collaboration-services: Build zuul images for production - https://phabricator.wikimedia.org/T396245#11148383 (10dduvall) 05Open→03Resolved
[14:58:36] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Doing 😎), 06collaboration-services: Build zuul images for production - https://phabricator.wikimedia.org/T396245#11148384 (10dduvall)
[15:13:16] <swfrench-wmf>	 Krinkle: ack, thank you!
[15:13:41] <bd808>	 dancy: ugh. I don't know why that build is so flakey from WMCS hosting. :/ I will retry it again to try to get to the interesting bit of pushing the image
[15:14:07] <bd808>	 heh looks like you are ahead of me
[15:18:09] <bd808>	 dancy: boo. `error: failed to solve: failed to push registry.staging.cloud.releng.team/repos/releng/zuul/tofu-provisioning:job-605619: unexpected status from POST request to https://registry.staging.cloud.releng.team/v2/repos/releng/zuul/tofu-provisioning/blobs/uploads/: 403 Forbidden`
[15:18:28] <dancy>	 Hmm.  Looks like the HTTP request made it all the way through to Reggie.
[15:18:40] <dancy>	 (as opposed to before, where nginx blocked it)
[15:18:42] <bd808>	 oh, that's progress then
[15:26:24] <wikibugs>	 10GitLab (CI & Job Runners), 07IPv6: registry.cloud.releng.team should support IPv6 - https://phabricator.wikimedia.org/T403742 (10taavi) 03NEW
[15:35:06] <wikibugs>	 (03open) 10dancy: reggie-values.yaml.tftpl: Enable jwt.debug [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/495 (https://phabricator.wikimedia.org/T396924)
[15:35:09] <wikibugs>	 (03update) 10dancy: reggie-values.yaml.tftpl: Enable jwt.debug [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/495 (https://phabricator.wikimedia.org/T396924)
[15:36:09] <wikibugs>	 (03merge) 10dancy: reggie-values.yaml.tftpl: Enable jwt.debug [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/495 (https://phabricator.wikimedia.org/T396924)
[15:36:19] <wikibugs>	 (03CR) 10Jdlrobson: "What is the criteria for adding an extension to gate? Could you share a link?" [integration/config] - 10https://gerrit.wikimedia.org/r/1184176 (https://phabricator.wikimedia.org/T403560) (owner: 10Aude)
[15:40:54] <wikibugs>	 10GitLab (CI & Job Runners), 07IPv6, 07Upstream: registry.cloud.releng.team should support IPv6 - https://phabricator.wikimedia.org/T403742#11148633 (10bd808) That service is hosted in a Digital Ocean Kubernetes (DOKS) cluster. I am not yet able to find docs upstream about enabling IPv6 for a DOKS load balan...
[15:45:52] <wikibugs>	 10GitLab (CI & Job Runners), 07IPv6: Support IPv6 on WMCS hosted runners - https://phabricator.wikimedia.org/T403746 (10taavi) 03NEW
[15:49:35] <wikibugs>	 10GitLab (CI & Job Runners), 07IPv6, 07Upstream: Expose registry.cloud.releng.team Reggie registry via IPv6 - https://phabricator.wikimedia.org/T403742#11148667 (10bd808)
[16:05:17] <dancy>	 bd808: I'm enlisting dduvall for help.
[16:06:58] <wikibugs>	 (03CR) 10Jforrester: [C:04-1] "> What is the criteria for adding an extension to gate? Could you share a link?" [integration/config] - 10https://gerrit.wikimedia.org/r/1184176 (https://phabricator.wikimedia.org/T403560) (owner: 10Aude)
[16:11:35] <wikibugs>	 (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/Graph] Drop from gate [integration/config] - 10https://gerrit.wikimedia.org/r/1184837
[16:11:35] <wikibugs>	 (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/Graph] Drop from production listing [integration/config] - 10https://gerrit.wikimedia.org/r/1184838 (https://phabricator.wikimedia.org/T362317)
[16:13:52] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/Graph] Drop from gate [integration/config] - 10https://gerrit.wikimedia.org/r/1184837 (owner: 10Jforrester)
[16:15:20] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Graph] Drop from gate [integration/config] - 10https://gerrit.wikimedia.org/r/1184837 (owner: 10Jforrester)
[16:18:51] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T403747#11148791 (10hashar) TLDR: this is due to Jenkins being upgraded (T403703) and changing the HTML output for `/`.  The probe can be found at https://grafana.wikimedia.org/d/O0nHhd...
[16:27:43] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 07Essential-Work, 13Patch-For-Review: Upgrade Jenkins to 2.516.2 - https://phabricator.wikimedia.org/T403703#11148870 (10hashar)
[16:27:45] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services, 13Patch-For-Review: ProbeDown - https://phabricator.wikimedia.org/T403747#11148871 (10hashar)
[16:29:10] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 13Patch-For-Review: ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#11148882 (10bd808) >>! In T403560#11143944, @cscott wrote: > But I've had...
[16:31:02] <wikibugs>	 (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/Graph] Drop from production listing [integration/config] - 10https://gerrit.wikimedia.org/r/1184838 (https://phabricator.wikimedia.org/T362317)
[16:31:03] <wikibugs>	 (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/PageViewInfo] Drop Graph dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1184873 (https://phabricator.wikimedia.org/T403753)
[16:31:04] <wikibugs>	 (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/Graph] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/1184874 (https://phabricator.wikimedia.org/T362317)
[16:38:10] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Automoderator, 06Moderator-Tools-Team, 07Wikimedia-production-error: AutoModerator WikiPageConfig::getConfigData failed to load config from wiki: {error} - https://phabricator.wikimedia.org/T403756 (10Krinkle) 03NEW
[16:46:24] <Krinkle>	 !log Install php8.3 on deployment-mediawiki13 and deployment-mediawiki14 in Beta Cluster.  https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/ec8a68dc394279c574c10a78a4eff75cfcdcefbc%5E%21/#F0 ref T360995
[16:46:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:46:27] <stashbot>	 T360995: Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995
[16:52:10] <wikibugs>	 10Beta-Cluster-Infrastructure, 07Wikimedia-production-error: DBQueryError "The MariaDB server is running with the --read-only option" fails MainStash in Beta Cluster - https://phabricator.wikimedia.org/T401227#11148988 (10Krinkle) @bd808 Only writable primaries. (The MainStash feature uses no replicas in the t...
[16:56:44] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: ProbeDown - releases1003:8080 - https://phabricator.wikimedia.org/T403747#11149049 (10Dzahn)
[16:57:42] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: ProbeDown - releases1003:8080 - https://phabricator.wikimedia.org/T403747#11149056 (10Dzahn) 05Open→03Resolved Yes, it was all about the content string we are checking. So the word "DOWN" is a bit misleading for these cases...
[17:03:31] <wikibugs>	 (03CR) 10Reedy: [C:03+1] Zuul: [mediawiki/extensions/PageViewInfo] Drop Graph dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1184873 (https://phabricator.wikimedia.org/T403753) (owner: 10Jforrester)
[17:08:21] <Krinkle>	 !log Remove unused `profile::mediawiki::mcrouter_wancache::use_onhost_memcached` from deployment-deploy Hiera data in Horizon
[17:08:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:09:20] <Krinkle>	 !log Remove unused `profile::mediawiki::mcrouter_wancache::use_onhost_memcached` from deployment-snapshot Hiera data in Horizon
[17:09:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:16:23] <wikibugs>	 10Beta-Cluster-Infrastructure, 06SRE, 06Traffic: Make varnish-frontend-restart work on Beta Cluster - https://phabricator.wikimedia.org/T299054#11149191 (10Krinkle) I'm guessing the below has the same root cause, albeit on a deployment host, not a varnish host.  ` krinkle@deployment-deploy04:~$ sudo tail -n1...
[17:20:20] <Krinkle>	 !log Remove unused `mediawiki_php7` from deployment-snapshot Hiera data in Horizon
[17:20:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:22:06] <wmf-insecte>	 Project beta-update-databases-eqiad build #87425: 04FAILURE in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/87425/
[17:23:41] <Reedy>	 Krinkle: ^ That was probably you
[17:23:52] <Reedy>	 18:22:06 Warning: PHP Startup: Unable to load dynamic library 'xml.so' (tried: /usr/lib/php/20210902/xml.so (/usr/lib/php/20210902/xml.so: cannot open shared object file: No such file or directory), /usr/lib/php/20210902/xml.so.so (/usr/lib/php/20210902/xml.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
[17:23:52] <Reedy>	 18:22:06 Warning: PHP Startup: Unable to load dynamic library 'xmlreader.so' (tried: /usr/lib/php/20210902/xmlreader.so (/usr/lib/php/20210902/xmlreader.so: cannot open shared object file: No such file or directory), /usr/lib/php/20210902/xmlreader.so.so (/usr/lib/php/20210902/xmlreader.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
[17:23:52] <Reedy>	 18:22:06 Warning: PHP Startup: Unable to load dynamic library 'xmlwriter.so' (tried: /usr/lib/php/20210902/xmlwriter.so (/usr/lib/php/20210902/xmlwriter.so: cannot open shared object file: No such file or directory), /usr/lib/php/20210902/xmlwriter.so.so (/usr/lib/php/20210902/xmlwriter.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
[17:23:52] <Reedy>	 18:22:06 Warning: PHP Startup: Unable to load dynamic library 'xsl.so' (tried: /usr/lib/php/20210902/xsl.so (/usr/lib/php/20210902/xsl.so: cannot open shared object file: No such file or directory), /usr/lib/php/20210902/xsl.so.so (/usr/lib/php/20210902/xsl.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
[17:23:52] <Reedy>	 18:22:06 Error: Missing one or more required PHP extensions. Please see
[17:23:54] <Krinkle>	 Yeah, deploy05 is running puppet
[17:24:00] <Krinkle>	 it's mid-switch
[17:24:46] <wikibugs>	 (03CR) 10Jdlrobson: "Respectfully, without written criteria, it’s difficult for any of us to determine whether ReadingList belongs in that list." [integration/config] - 10https://gerrit.wikimedia.org/r/1184176 (https://phabricator.wikimedia.org/T403560) (owner: 10Aude)
[17:25:10] * Krinkle clicks rebuild
[17:29:16] <wmf-insecte>	 Yippee, build fixed!
[17:29:16] <wmf-insecte>	 Project beta-update-databases-eqiad build #87426: 09FIXED in 4 min 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/87426/
[17:29:25] <Krinkle>	 Yippee indeed, cc Reedy 
[17:29:37] <Reedy>	 just bad timing then ;)
[17:29:47] <Krinkle>	 puppet-agent doesn't look clean on deploy05, but it was already unclean. Somehow, despite printing a sea of red, it still ends green, which is odd.
[17:29:52] <Krinkle>	 > Sep  4 00:05:14 deployment-deploy04 puppet-agent[4133645]: Could not set 'present' on ensure: No such file or directory - A directory component in /srv/patches/.git/hooks/pre-commit20250904-4133645-18fjm7k.lock does not exist or is a dangling symbolic link (file: /srv/puppet_code/environments/production/modules/scap/manifests/master.pp, line: 57)
[17:29:58] <Krinkle>	 > Sep  4 00:05:14 deployment-deploy04 puppet-agent[4133645]: (/Stage[main]/Scap::Master/File[/srv/patches/.git/hooks/pre-commit]/ensure) change from 'absent' to 'present' failed: Could not set 'present' on ensure: No such file or directory - A directory component in /srv/patches/.git/hooks/pre-commit20250904-4133645-18fjm7k.lock does not exist or is a dangling symbolic link (file: 
[17:29:58] <Krinkle>	 /srv/puppet_code/environments/production/modules/scap/manifests/master.pp, line: 57)
[17:30:04] <Krinkle>	 deploy04* I mean
[17:30:15] <Krinkle>	 Doesn't seem to cause anything right now, but may be of interest to someonw
[17:30:57] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, 10Wikifunctions, and 3 others: ${{SCRIPT, template="wikimedia.template"}} is not sending a correct template - https://phabricator.wikimedia.org/T400286#11149274 (10vaughnwalters) 05In progress→03Resolved >>! In T40...
[17:31:05] <mutante>	 Krinkle: does /srv/patches/.git  exist?
[17:31:12] <dancy>	 Krinkle: I think I caused that problem.  
[17:31:26] <mutante>	 I merged that after checking it does.. in prod.
[17:31:29] <Krinkle>	 I'll leave you to it then :)
[17:32:23] <mutante>	 we should use "wmflib::dir::mkdir_p" to create that
[17:32:39] <mutante>	 because unlike standard puppet it takes care of all parts of a path
[17:32:57] <mutante>	 it's "mkdir -p"
[17:33:45] <dancy>	 And it should be followed by a git init if not done already.  Is there an existing pattern for that?
[17:37:16] <mutante>	 using git::clone should be enough
[17:37:21] <dancy>	 Ah yes, /etc/helmfile-defaults/mediawiki/release on the deploy server.  I'll follow that pattern.
[17:37:40] <dancy>	 mutante: It's a local repo.  Nothing to clone from.
[17:38:33] <mutante>	 ack!  there is another one here:
[17:38:33] <mutante>	 modules/profile/manifests/puppetserver/git.pp:            exec { "git init ${dir}":
[17:44:39] <wikibugs>	 (03CR) 10Reedy: [C:03+2] Zuul: [mediawiki/extensions/PageViewInfo] Drop Graph dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1184873 (https://phabricator.wikimedia.org/T403753) (owner: 10Jforrester)
[17:46:18] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/PageViewInfo] Drop Graph dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1184873 (https://phabricator.wikimedia.org/T403753) (owner: 10Jforrester)
[17:48:09] <Reedy>	 !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1184873 T403753
[17:48:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:48:11] <stashbot>	 T403753: Drop PageViewInfo's integration with the Graph extension on action=info, the extension is dead - https://phabricator.wikimedia.org/T403753
[17:58:09] <dancy>	 mutante: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1184891
[17:59:15] <wikibugs>	 (03open) 10dancy: lib/image.py: Add KOKKURI_DEBUG_AUTH support [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/141
[17:59:18] <wikibugs>	 (03update) 10dancy: lib/image.py: Add KOKKURI_DEBUG_AUTH support [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/141
[18:06:57] <wikibugs>	 (03update) 10dancy: lib/image.py: Add KOKKURI_DEBUG_AUTH support [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/141
[18:09:24] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T403616#11149413 (10bd808) 05Resolved→03Open `lang=shell-session, counterexample bd808@deployment-cache-text08.deployment-prep.eqiad1:/e...
[18:17:23] <wikibugs>	 (03update) 10dancy: lib/image.py: Add KOKKURI_DEBUG_AUTH support [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/141
[18:18:59] <wikibugs>	 (03update) 10dancy: lib/image.py: Add KOKKURI_DEBUG_AUTH support [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/141
[18:20:21] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Automoderator, 06Moderator-Tools-Team, 07Wikimedia-production-error: AutoModerator WikiPageConfig::getConfigData failed to load config from wiki: {error} - https://phabricator.wikimedia.org/T403756#11149470 (10Kgraessle) p:05Triage→03High
[18:20:28] <wikibugs>	 (03merge) 10dancy: lib/image.py: Add KOKKURI_DEBUG_AUTH support [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/141
[18:20:34] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Automoderator, 10Moderator-Tools-Team (Kanban), 07Wikimedia-production-error: AutoModerator WikiPageConfig::getConfigData failed to load config from wiki: {error} - https://phabricator.wikimedia.org/T403756#11149473 (10Kgraessle)
[18:40:21] <wikibugs>	 (03open) 10dancy: includes/kokkuri.yaml: Bump KOKKURI_IMAGE to 2.8.1 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/142
[18:40:23] <wikibugs>	 (03update) 10dancy: includes/kokkuri.yaml: Bump KOKKURI_IMAGE to 2.8.1 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/142
[18:42:26] <wikibugs>	 (03merge) 10dancy: includes/kokkuri.yaml: Bump KOKKURI_IMAGE to 2.8.1 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/142
[18:48:03] <dancy>	 Krinkle: The beta /srv/patches puppet problem is resolved.
[18:48:29] <Krinkle>	 awesome
[18:48:51] <mutante>	 confirmed noop on prod deploy
[18:53:23] <wikibugs>	 (03PS1) 10Subramanya Sastry: cloudvps-configs: Update README + minor tweak to setup_vm.sh script [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1184906
[18:59:53] <wikibugs>	 (03open) 10dancy: Remove .gitreview [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/998
[18:59:57] <wikibugs>	 (03update) 10dancy: Remove .gitreview [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/998
[19:00:34] <wikibugs>	 (03update) 10dancy: Remove .gitreview [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/998
[19:08:11] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.45.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T396378#11149677 (10dancy) 05Open→03Resolved Everything looks good on group2.  Closing.
[19:08:29] <wmcs-alerts>	 FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[19:46:35] <wikibugs>	 10Phabricator: display "policy" group memberships in phabricator user profiles - https://phabricator.wikimedia.org/T403778 (10Novem_Linguae) 03NEW
[20:21:43] <wikibugs>	 10Phabricator: display "policy" group memberships in phabricator user profiles - https://phabricator.wikimedia.org/T403778#11149931 (10Dzahn) I think filing it upstream was the correct place.
[20:24:45] <wikibugs>	 10Phabricator: display "policy" group memberships in phabricator user profiles - https://phabricator.wikimedia.org/T403778#11149954 (10Peachey88) This is caused by a downstream hack from memory, From memory it was introduced by the one that was designed not to show the acl projects at the start of the list.  Whe...
[20:37:37] <wikibugs>	 10GitLab (CI & Job Runners), 07IPv6, 07Upstream: Expose registry.cloud.releng.team Reggie registry via IPv6 - https://phabricator.wikimedia.org/T403742#11149992 (10bd808) I am seeing `ipv6:  Enable now` when I look at a droplet in a DOKS cluster, so maybe we just need to flip some feature flags?  https://clo...
[20:43:32] <wikibugs>	 10Phabricator: display "policy" group memberships in phabricator user profiles - https://phabricator.wikimedia.org/T403778#11150012 (10Novem_Linguae) Good info, thanks. I think another acceptable resolution to this ticket then could be to always display the "View All" button on user profiles.
[21:18:05] <wikibugs>	 10Beta-Cluster-Infrastructure: Configure etcd/confd/conftool in beta/deployment-prep like production - https://phabricator.wikimedia.org/T278007#11150195 (10bd808)
[21:18:07] <wikibugs>	 10Beta-Cluster-Infrastructure, 06SRE, 06Traffic: Make varnish-frontend-restart work on Beta Cluster - https://phabricator.wikimedia.org/T299054#11150196 (10bd808)
[21:18:52] <wikibugs>	 10Beta-Cluster-Infrastructure: Configure etcd/confd/conftool in beta/deployment-prep like production - https://phabricator.wikimedia.org/T278007#11150201 (10bd808)
[21:18:57] <wikibugs>	 10Beta-Cluster-Infrastructure, 06SRE, 06Traffic: Make varnish-frontend-restart work on Beta Cluster - https://phabricator.wikimedia.org/T299054#11150202 (10bd808)
[21:19:02] <wikibugs>	 10Beta-Cluster-Infrastructure: Configure etcd/confd/conftool in beta/deployment-prep like production - https://phabricator.wikimedia.org/T278007#11150203 (10bd808)
[21:19:06] <wikibugs>	 10Beta-Cluster-Infrastructure, 06SRE, 06Traffic: Make varnish-frontend-restart work on Beta Cluster - https://phabricator.wikimedia.org/T299054#11150204 (10bd808)
[21:35:49] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T403616#11150304 (10bd808) >>! In T403616#11149413, @bd808 wrote: > This is still happening because the guard condition added in https://ger...
[21:37:41] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 13Patch-For-Review: ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#11150311 (10thcipriani) Adding extensions to gated extensions is not the...
[21:50:51] <wikibugs>	 (03PS1) 10Subramanya Sastry: Update list of wikipedia prefixes known to visualdiff [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1184939
[21:53:59] <wikibugs>	 (03open) 10dancy: ingress-nginx.yaml.tftpl: Enable load balancer proxy protocol [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/496
[21:54:02] <wikibugs>	 (03update) 10dancy: ingress-nginx.yaml.tftpl: Enable load balancer proxy protocol [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/496
[21:55:13] <wikibugs>	 (03merge) 10dancy: ingress-nginx.yaml.tftpl: Enable load balancer proxy protocol [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/496
[22:03:21] <wikibugs>	 (03open) 10reedy: releases.json: Upgrade mediawiki/mediawiki-codesniffer to 48.0.0 in master [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/90 (https://phabricator.wikimedia.org/T403736)
[22:21:06] <wikibugs>	 (03open) 10dancy: ingress-nginx.yaml.tftpl: Enable proxy protocol [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/497 (https://phabricator.wikimedia.org/T396924)
[22:21:08] <wikibugs>	 (03update) 10dancy: ingress-nginx.yaml.tftpl: Enable proxy protocol [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/497 (https://phabricator.wikimedia.org/T396924)
[22:22:12] <wikibugs>	 (03update) 10dancy: ingress-nginx.yaml.tftpl: Enable proxy protocol [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/497 (https://phabricator.wikimedia.org/T396924)
[22:22:16] <wikibugs>	 (03update) 10dancy: ingress-nginx.yaml.tftpl: Enable proxy protocol [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/497 (https://phabricator.wikimedia.org/T396924)
[22:23:11] <wikibugs>	 (03merge) 10dancy: ingress-nginx.yaml.tftpl: Enable proxy protocol [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/497 (https://phabricator.wikimedia.org/T396924)
[22:24:03] <wikibugs>	 (03PS1) 10Jeena Huneidi: Always clean up catalyst environments [integration/config] - 10https://gerrit.wikimedia.org/r/1184944 (https://phabricator.wikimedia.org/T402591)
[22:51:55] <wikibugs>	 (03open) 10dancy: ingress-nginx.yaml.tftpl: Drop `use-forwarded-headers: "true"` [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/498 (https://phabricator.wikimedia.org/T396924)
[22:51:57] <wikibugs>	 (03update) 10dancy: ingress-nginx.yaml.tftpl: Drop `use-forwarded-headers: "true"` [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/498 (https://phabricator.wikimedia.org/T396924)
[22:53:14] <wikibugs>	 (03merge) 10dancy: ingress-nginx.yaml.tftpl: Drop `use-forwarded-headers: "true"` [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/498 (https://phabricator.wikimedia.org/T396924)
[23:07:18] <wikibugs>	 (03open) 10dancy: Revert "reggie-values.yaml.tftpl: Enable jwt.debug" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/499 (https://phabricator.wikimedia.org/T396924)
[23:07:20] <wikibugs>	 (03update) 10dancy: Revert "reggie-values.yaml.tftpl: Enable jwt.debug" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/499 (https://phabricator.wikimedia.org/T396924)
[23:18:45] <wikibugs>	 10GitLab (CI & Job Runners), 13Patch-For-Review: kokkuri cannot publish "public" images from WMCS runners due to a lack of a local registry - https://phabricator.wikimedia.org/T396924#11150566 (10dancy) @bd808 Pushes from WMCS to registry.cloud.releng.team and registry.staging.cloud.releng.team should be worki...
[23:21:25] <bd808>	 dancy: will I still need to set KOKKURI_REGISTRY_PUBLIC in the job to push to registry.cloud.releng.team?
[23:21:55] <dancy>	 Yes for the time being.  If it works out we can make it a default setting for wmcs runners
[23:22:03] <bd808>	 ack
[23:22:33] <wikibugs>	 (03update) 10bd808: ci: Use wmcs runners and registry.cloud.releng.team [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/52
[23:23:33] <dancy>	 hrmm. things may be broken
[23:23:58] <dancy>	 I'm going to revert and try again tomorrow
[23:24:40] <wikibugs>	 10GitLab (CI & Job Runners), 13Patch-For-Review: kokkuri cannot publish "public" images from WMCS runners due to a lack of a local registry - https://phabricator.wikimedia.org/T396924#11150568 (10dancy) Looks like the changes have broken registry.cloud.releng.team. I will revert and try again tomorrow.
[23:26:19] <wikibugs>	 (03open) 10dancy: ingress-nginx.yaml.tftpl: Disable PROXY protocol stuff [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/500 (https://phabricator.wikimedia.org/T396924)
[23:26:21] <wikibugs>	 (03update) 10dancy: ingress-nginx.yaml.tftpl: Disable PROXY protocol stuff [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/500 (https://phabricator.wikimedia.org/T396924)
[23:27:29] <dancy>	 hmm.. and working now..  Weird.
[23:27:37] * dancy blames eventual consistency
[23:29:39] <wikibugs>	 (03update) 10dancy: Revert "reggie-values.yaml.tftpl: Enable jwt.debug" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/499 (https://phabricator.wikimedia.org/T396924)
[23:30:52] <wikibugs>	 (03merge) 10dancy: Revert "reggie-values.yaml.tftpl: Enable jwt.debug" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/499 (https://phabricator.wikimedia.org/T396924)
[23:32:13] <wikibugs>	 10GitLab (CI & Job Runners), 13Patch-For-Review: kokkuri cannot publish "public" images from WMCS runners due to a lack of a local registry - https://phabricator.wikimedia.org/T396924#11150574 (10dancy) >>! In T396924#11150568, @dancy wrote: > Looks like the changes have broken registry.cloud.releng.team. I wi...
[23:32:34] <wikibugs>	 (03close) 10dancy: ingress-nginx.yaml.tftpl: Disable PROXY protocol stuff [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/500 (https://phabricator.wikimedia.org/T396924)
[23:33:18] <dancy>	 bd808: The failures with your build-deployer job seems to be unrelated to the registry.
[23:33:34] <bd808>	 yeah. it's the github rate limits :/
[23:34:35] <bd808>	 all of WMCS looks like the same IP to github which makes doing things like fetching a bunch of artifacts from there sketchy
[23:34:47] <dancy>	 Nod. Makes sense.
[23:34:49] <dancy>	 Alright I'm stepping out for a break.  I'll check back in a bit to make sure everything is stable. 
[23:45:37] <wikibugs>	 10GitLab (CI & Job Runners), 13Patch-For-Review: kokkuri cannot publish "public" images from WMCS runners due to a lack of a local registry - https://phabricator.wikimedia.org/T396924#11150607 (10thcipriani) >>! In T396924#11150574, @dancy wrote: >>>! In T396924#11150568, @dancy wrote: >> Looks like the change...