[00:11:06] 10Beta-Cluster-Infrastructure: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318#10886218 (10Krinkle) Ack. If we keep it all in the deployment-prep project, we can also keep it within the same servers rather than a new service. There is redirect infrastructure at all three... [00:22:25] !log Adding missing DNS entries under beta.wmcloud.org. There was already: *.wikipedia, *.m.wikimedia, *.wikivoyage, *.m.wikivoyage (for T355281). Adding: wikibooks, wikimedia, wikinews, wikiquote, wikisource, wikiversity, wiktionary, wikidata, upload (T289318). [00:22:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:22:28] T355281: Set up some beta cluster wikis with different registrable domain - https://phabricator.wikimedia.org/T355281 [00:22:29] T289318: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318 [00:32:49] !log Add `TXT *.wikimedia.beta.wmcloud.org. "v=spf1 -all"` to match beta.wmflabs.org DNS (ref T289318, changing email is out of scope for now, but might as well add the DNS records). [00:32:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:32:53] T289318: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318 [01:36:50] (03PS1) 10Mmta: Edit Repo Config [3d2png] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1153744 [02:19:19] 10Beta-Cluster-Infrastructure: irc.beta.wmflabs.org is unresponsive - https://phabricator.wikimedia.org/T396088 (10Krinkle) 03NEW [03:06:22] (03update) 10jhuneidi: artifacts: Support `exclude` patterns for copies/requirements [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/123 (owner: 10dduvall) [03:34:40] !log Update profile::acme_chief::certificates under deployment-acme-chief prefix in Horizon, to include remaining the wildcard and m-dot subdomains under beta.wmcloud.org for wikibooks, wikimedia, wikinews, wikiquote, wikisource, wikiversity, wiktionary (wikipedia and wikivoyage were already there) [03:34:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:34:45] !log Update profile::acme_chief::certificates under deployment-acme-chief prefix in Horizon, to include remaining the wildcard and m-dot subdomains under beta.wmcloud.org for wikibooks, wikimedia, wikinews, wikiquote, wikisource, wikiversity, wiktionary (wikipedia and wikivoyage were already there), ref T289318 [03:34:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:34:47] T289318: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318 [03:34:50] (03approved) 10jhuneidi: artifacts: Support `exclude` patterns for copies/requirements [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/123 (owner: 10dduvall) [03:58:20] !log Update profile::cache::haproxy::available_unified_certificates under deployment-cache in Horizon, to include remaining the wildcard and m-dot subdomains under beta.wmcloud.org for wikibooks, wikimedia, wikinews, wikiquote, wikisource, wikiversity, wiktionary. Remove `*.zero.wikipedia.beta.wmflabs.org` which wasn't responding/didn't work anymore. ref T289318 [03:58:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:58:23] T289318: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318 [04:45:59] !log gitpuppet@deployment-puppetserver-1:/srv/git/operations/puppet$ Cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/1153764, ref T289318 [04:46:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:46:01] T289318: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318 [04:46:48] The one thing I haven't done yet is change Hiera for deployment-mediawiki, which would provision local envoy to handle these domains [04:47:01] I'm holding off on that because it seems the edge side hasn't sorted itself out yet [04:47:21] it's been an hour since adding to acme-chief and cache-text but I'm not seeing any cert acquired yet [04:47:48] https://en.wiktionary.beta.wmcloud.org/ still returns an cert without any of these domains in it (I'd xpect it to error at the apache level, not at the edge like it does now) [04:47:51] https://crt.sh/?q=beta.wmcloud.org [04:48:02] it hasn't been asked for yet it seems [04:48:17] did run puppet on acme/puppserver/cache-text hosts [04:48:39] Maybe it happens on some other schedule, and there's no `notify` in puppet? [04:48:44] I'll just wait until tomorrow [04:48:45] o/ [05:07:14] (03update) 10dduvall: artifacts: Support `exclude` patterns for copies/requirements [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/123 [05:39:08] (03update) 10dduvall: artifacts: Support `exclude` patterns for copies/requirements [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/123 [05:39:12] (03update) 10dduvall: artifacts: Support `exclude` patterns for copies/requirements [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/123 [05:39:12] (03update) 10dduvall: artifacts: Support `exclude` patterns for copies/requirements [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/123 [05:45:02] (03merge) 10dduvall: artifacts: Support `exclude` patterns for copies/requirements [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/123 [05:54:02] (03open) 10dduvall: builder: Support `mounts` and `caches` [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/125 [05:54:11] (03update) 10dduvall: builder: Support `mounts` and `caches` [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/125 [06:01:42] (03update) 10dduvall: builder: Support `mounts` and `caches` [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/125 [06:04:33] (03update) 10dduvall: apt: Support `signed-by` field in apt configuration [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/124 (https://phabricator.wikimedia.org/T395645) [06:53:08] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog 📥): Plan for porting PipelineLib to Zuul Ansible - https://phabricator.wikimedia.org/T390119#10886524 (10hashar) [08:01:41] Krinkle: FWIW, acme-chief on deployment-acme-chief05 issued the certs on Jun 05 04:36:07 and enforced its staging time till 06:34 [08:02:01] https://www.irccloud.com/pastebin/rSGRY5vO/ [08:02:26] that's the current list of SANs that should be available for cp servers as well in deployment-prep [08:02:35] cheers [08:09:27] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10886737 (10Jelto) I created an environment to reproduce the error. I manually enabled object storage on `gitlab1003` (replica-b) and pointed `r... [08:35:56] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10886808 (10Jelto) I enabled object storage again for the production host. Upload of CI artifacts still works: https://gitlab.wikimedia.org/obli... [09:25:59] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 06Data-Persistence, 10Data-Persistence-Backup: Request mariadb database for Zuul - https://phabricator.wikimedia.org/T394844#10886890 (10Marostegui) @Dzahn What would happen if this database is unavailable for a few minutes?... [09:26:52] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 06Data-Persistence, 10Data-Persistence-Backup: Request mariadb database for Zuul - https://phabricator.wikimedia.org/T394844#10886893 (10Marostegui) [09:37:45] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10886937 (10Jelto) 05Open→03Resolved So far CI jobs look good, no artifact errors and no reported issues. Thanks again @MatthewVernon fo... [09:39:22] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10886950 (10Jelto) The artifact upload issues have been resolved (T396018). CI job logs and metric look... [09:55:20] Am I right that if a backport is canceled at the testserver phase, the revert needs to be merged AND backported to restore the previous state? [09:55:38] Because rn we have a pending image update on mw-on-k8s from a canceled deployment this morning [09:55:45] correct [09:55:53] ok I'll just do the scap backport [09:56:08] And I'll file a task for spiderpig to pop a warning that this needs to be done [09:56:14] because it doesn't seem clear to deployers [09:57:49] thanks taavi [10:03:05] 10Scap (SpiderPig 🕸️): spiderpig should give the revert procedure after a canceled deployment - https://phabricator.wikimedia.org/T396106 (10Clement_Goubert) 03NEW [10:06:36] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog 📥): Plan for porting PipelineLib to Zuul Ansible - https://phabricator.wikimedia.org/T390119#10887010 (10hashar) [10:29:29] (03open) 10jelto: set ci_delete_pipelines_in_seconds for all projects [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/70 (https://phabricator.wikimedia.org/T395014) [10:29:53] (03update) 10jelto: set ci_delete_pipelines_in_seconds for all projects [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/70 (https://phabricator.wikimedia.org/T395014) [10:37:36] 10Deployments, 10Release-Engineering-Team (Radar), 06serviceops, 07Wikimedia-production-error: httpb sometimes fails upon deployment with a HTTP 503 - https://phabricator.wikimedia.org/T380958#10887157 (10akosiaris) >>! In T380958#10885815, @cjming wrote: > encountered this deploying today - https://spider... [10:51:16] 10Deployments, 10Release-Engineering-Team (Radar), 06serviceops, 07Wikimedia-production-error: httpb sometimes fails upon deployment with a HTTP 503 - https://phabricator.wikimedia.org/T380958#10887212 (10akosiaris) I 'll be hammering mwdebug2001 with siege for the next 30 minutes in an effort to reproduce... [10:54:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-memc13 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:54:32] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-memc13 on project deployment-prep - https://phabricator.wikimedia.org/T396109 (10wmcs-alerts) 03NEW [10:59:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-memc11 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [11:05:00] 10GitLab, 06collaboration-services, 06serviceops, 07Kubernetes: Trusted gitlab runner containers need access to staging k8s cluster - https://phabricator.wikimedia.org/T325385#10887284 (10JMeybohm) 05Resolved→03Open We dismantled the CI access to k8s staging in {T288629}, so access from Gitlab runners... [11:14:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance deployment-memc11 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:35:08] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10887537 (10Jelto) All CI artifacts have been successfully migrated to object storage. The overall disk u... [13:31:57] (03merge) 10jelto: update artifact expiry to 6mo [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/69 (https://phabricator.wikimedia.org/T395014) [14:08:46] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10887784 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup Everything is merged, I removed the cherry-picks, this is... [14:12:54] (03CR) 10Hashar: [C:03+2] Add OOJSPlus dependency to UnifiedTaskOverview [integration/config] - 10https://gerrit.wikimedia.org/r/1152866 (owner: 10Hslater) [14:14:21] (03Merged) 10jenkins-bot: Add OOJSPlus dependency to UnifiedTaskOverview [integration/config] - 10https://gerrit.wikimedia.org/r/1152866 (owner: 10Hslater) [14:39:01] 10Deployments, 10Release-Engineering-Team (Radar), 06serviceops, 07Wikimedia-production-error: httpb sometimes fails upon deployment with a HTTP 503 - https://phabricator.wikimedia.org/T380958#10887906 (10akosiaris) Close to 100k requests later to `Main_Page` of `spcom.wikimedia.org`, 1 error. This is belo... [15:09:37] 10Continuous-Integration-Config, 10MediaWiki-extensions-Springboard: Commits to Springboard repo on Gerrit not getting handled - https://phabricator.wikimedia.org/T395981#10888083 (10Yaron_Koren) 05Open→03Resolved a:03Yaron_Koren @QChris - thanks for this fix! [16:03:20] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06SRE, 10SRE-Access-Requests: Requesting access to contint-roots for Corvus - https://phabricator.wikimedia.org/T395167#10888267 (10Dzahn) Hey @Corvus you have access now to hosts in production. Your SSH setup will be very similar to what you already... [16:03:29] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06SRE, 10SRE-Access-Requests: Requesting access to contint-roots for Corvus - https://phabricator.wikimedia.org/T395167#10888271 (10Dzahn) 05In progress→03Resolved a:03Dzahn [16:05:47] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 06Data-Persistence, 10Data-Persistence-Backup: Request mariadb database for Zuul - https://phabricator.wikimedia.org/T394844#10888277 (10Dzahn) @Corvus Would you have any comments on the last question above? [16:07:03] (03PS11) 10Jforrester: [DNM] Zuul: Drop all PHP 7.4 testing for MediaWiki things [integration/config] - 10https://gerrit.wikimedia.org/r/1127083 (https://phabricator.wikimedia.org/T328921) [16:07:14] (03PS12) 10Jforrester: Zuul: Drop all PHP 7.4 testing for MediaWiki things [integration/config] - 10https://gerrit.wikimedia.org/r/1127083 (https://phabricator.wikimedia.org/T328921) [16:07:35] (03CR) 10Jforrester: [C:03+2] "Per swfrench-wmf and claime." [integration/config] - 10https://gerrit.wikimedia.org/r/1127083 (https://phabricator.wikimedia.org/T328921) (owner: 10Jforrester) [16:09:01] (03Merged) 10jenkins-bot: Zuul: Drop all PHP 7.4 testing for MediaWiki things [integration/config] - 10https://gerrit.wikimedia.org/r/1127083 (https://phabricator.wikimedia.org/T328921) (owner: 10Jforrester) [16:09:32] !log Zuul: Drop all PHP 7.4 testing for MediaWiki things, for T328921 and T328922 [16:09:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:09:35] T328921: Drop PHP 7.4 and PHP 8.0 support from MediaWiki - https://phabricator.wikimedia.org/T328921 [16:09:36] T328922: Drop PHP 8.0 support from MediaWiki - https://phabricator.wikimedia.org/T328922 [16:10:06] It is done! [16:13:22] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 06Data-Persistence, 10Data-Persistence-Backup: Request mariadb database for Zuul - https://phabricator.wikimedia.org/T394844#10888302 (10Corvus) Zuul will reconnect automatically; no intervention should be required. If Zuul... [16:16:54] (03PS7) 10Jforrester: Zuul: [operations/mediawiki-config] Stop testing in PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127095 [16:19:41] \o/ [16:20:09] (03CR) 10Jforrester: [C:03+2] Zuul: [operations/mediawiki-config] Stop testing in PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127095 (owner: 10Jforrester) [16:21:08] NOICE [16:21:27] (03Merged) 10jenkins-bot: Zuul: [operations/mediawiki-config] Stop testing in PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127095 (owner: 10Jforrester) [16:21:44] !log Zuul: [operations/mediawiki-config] Stop testing in PHP 7.4 [16:21:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:21:54] (03PS6) 10Jforrester: Zuul: [mediawiki/services/parsoid] Stop testing in PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127094 [16:22:00] (03PS7) 10Jforrester: Zuul: [mediawiki/services/parsoid] Stop testing in PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127094 [16:22:04] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/services/parsoid] Stop testing in PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127094 (owner: 10Jforrester) [16:23:40] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/parsoid] Stop testing in PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127094 (owner: 10Jforrester) [16:23:52] !log Zuul: [mediawiki/services/parsoid] Stop testing in PHP 7.4 [16:23:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:24:02] (03PS10) 10Jforrester: Zuul: [integration/quibble] Stop testing in PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127093 (https://phabricator.wikimedia.org/T328921) [16:24:06] (03CR) 10Jforrester: [C:03+2] Zuul: [integration/quibble] Stop testing in PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127093 (https://phabricator.wikimedia.org/T328921) (owner: 10Jforrester) [16:24:07] JAMES_F: BYE PHP SEVEN FOUR. YOU STUCK TO US LIKE AN USED BUBBLE GUM. [16:24:12] 🎉 [16:24:12] * James_F grins. [16:24:22] I haven't felt this elated since we dumped HHVM. [16:24:30] You should have given a presentation at all staff and phase it out LIVE [16:24:36] ahah [16:24:38] I hear yeah [16:25:15] and we are lucky Tim got us to upgrade to php 7.4 or we would have been carrying 7.2 all along [16:25:23] https://imgflip.com/i/9wckwh etc. [16:25:36] Yes, massively. [16:25:53] (03Merged) 10jenkins-bot: Zuul: [integration/quibble] Stop testing in PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127093 (https://phabricator.wikimedia.org/T328921) (owner: 10Jforrester) [16:26:00] php7.4 image was introduced in October 2019 [16:26:12] so roughly 5 years [16:26:25] good meme [16:26:26] !log Zuul: [integration/quibble] Stop testing in PHP 7.4, for T328921 and T328922 [16:26:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:26:30] T328921: Drop PHP 7.4 and PHP 8.0 support from MediaWiki - https://phabricator.wikimedia.org/T328921 [16:26:31] T328922: Drop PHP 8.0 support from MediaWiki - https://phabricator.wikimedia.org/T328922 [16:26:45] James_F: can you announce it widely once done? :) [16:26:48] I shall. [16:26:56] You must! [16:27:05] (03PS5) 10Jforrester: Zuul: Stop testing PHP extensions with PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127097 [16:27:10] (03CR) 10Jforrester: [C:03+2] Zuul: Stop testing PHP extensions with PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127097 (owner: 10Jforrester) [16:28:37] (03Merged) 10jenkins-bot: Zuul: Stop testing PHP extensions with PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127097 (owner: 10Jforrester) [16:28:46] !log Zuul: Stop testing PHP extensions with PHP 7.4 [16:28:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:31:58] (03PS5) 10Jforrester: Zuul: Stop testing most libraries and tools in PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127098 [16:32:01] (03PS5) 10Jforrester: Zuul: [labs/tools/heritage] Raise PHP testing from 7.4 to 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1127099 [16:32:03] (03PS5) 10Jforrester: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Stop testing in PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127100 [16:32:06] (03PS5) 10Jforrester: Zuul: [mediawiki/tools/phan/PerfCheckPlugin] Use a template for CI [integration/config] - 10https://gerrit.wikimedia.org/r/1127101 [16:32:08] (03PS4) 10Jforrester: jjb: Drop PHP 7.4 jobs, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1127119 [16:32:51] (03CR) 10Jforrester: [C:03+2] Zuul: Stop testing most libraries and tools in PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127098 (owner: 10Jforrester) [16:34:18] (03Merged) 10jenkins-bot: Zuul: Stop testing most libraries and tools in PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127098 (owner: 10Jforrester) [16:34:24] !log Zuul: Stop testing most libraries and tools in PHP 7.4 [16:34:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:34:49] (03CR) 10Jforrester: [C:03+2] Zuul: [labs/tools/heritage] Raise PHP testing from 7.4 to 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1127099 (owner: 10Jforrester) [16:35:12] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Stop testing in PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127100 (owner: 10Jforrester) [16:36:43] (03Merged) 10jenkins-bot: Zuul: [labs/tools/heritage] Raise PHP testing from 7.4 to 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1127099 (owner: 10Jforrester) [16:36:48] !log Zuul: [labs/tools/heritage] Raise PHP testing from 7.4 to 8.1 [16:36:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:37:01] (03Merged) 10jenkins-bot: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Stop testing in PHP 7.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1127100 (owner: 10Jforrester) [16:37:20] !log Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Stop testing in PHP 7.4 [16:37:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:37:54] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/tools/phan/PerfCheckPlugin] Use a template for CI [integration/config] - 10https://gerrit.wikimedia.org/r/1127101 (owner: 10Jforrester) [16:38:33] (03PS5) 10Jforrester: jjb: Drop PHP 7.4 jobs, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1127119 [16:39:16] (03Merged) 10jenkins-bot: Zuul: [mediawiki/tools/phan/PerfCheckPlugin] Use a template for CI [integration/config] - 10https://gerrit.wikimedia.org/r/1127101 (owner: 10Jforrester) [16:39:18] !log Zuul: [mediawiki/tools/phan/PerfCheckPlugin] Use a template for CI [16:39:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:41:50] * bd808 runs "recheck" on a patch stuck since 2024-05-04 by PHP 7.4 [16:44:12] bd808: <3 [16:45:06] I knew it had been a while since SyntaxHighlight got caught by this, but I was a bit surprised that it was this long. [16:45:14] Ouch. [16:46:31] We kind of have a history of it getting trapped for various reasons at old versions. Now I need to remember how to update it all over again. I hope I wrote myself some notes somewhere. :) [16:47:05] bd808: Let's hope we succeed in moving to 8.3 a bit faster. [16:48:59] I think we will. We are almost over the hill of everything being in containers (modulo deployment-prep). Folks seem to have figured out reasonable ways to build and test newer base containers already. [16:50:21] Excellent. [16:50:32] not needing to reimage a fleet of physical servers each time we move to a new PHP will ultimately be a big time savings, but that part generally came towards the end of the big effort. [16:50:32] And of course PHP 8.5 will ship this Thanksgiving. [16:53:21] well... Bookworm ships 8.2 and that will be our base OS for probably 14-16 more months at least. [16:53:38] Trixie has 8.4, I think. [16:56:32] yeah. Trixie does ship 8.4. And it may be faster for us to adopt Trixie in containers than bare metal will catch up due to likely Puppet changes. [16:57:18] This all makes me more concerned about Beta Cluster being functionally "bare metal" still (vms, but treated like hardware) [16:59:14] It looks like Trixie hit hard freeze a couple of weeks ago. Time to start the bookworm migrations then too. [17:00:11] (03PS1) 10Máté Szabó: Add WikimediaMaintenance test dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1154074 (https://phabricator.wikimedia.org/T58074) [17:03:55] (03PS6) 10Jforrester: jjb: Drop PHP 7.4 jobs, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1127119 [17:03:55] (03PS1) 10Jforrester: Docker: [phpmetrics] Migrate parent image from php74 to php81 [integration/config] - 10https://gerrit.wikimedia.org/r/1154076 [17:03:55] (03PS1) 10Jforrester: jjb: [*-phpmetrics] Switch image to 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1154077 [17:03:56] (03PS1) 10Jforrester: Docker: [mediawiki-phan-testrun] Migrate parent image from php74 to php81 [integration/config] - 10https://gerrit.wikimedia.org/r/1154078 [17:03:56] (03PS1) 10Jforrester: jjb: [mw-tools-phan-mwcore-testrun] Switch image to 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1154079 [17:03:58] (03PS1) 10Jforrester: Docker: Drop all PHP 7.4 images [integration/config] - 10https://gerrit.wikimedia.org/r/1154080 [17:05:21] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/WikimediaMaintenance] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1154074 (https://phabricator.wikimedia.org/T58074) (owner: 10Máté Szabó) [17:05:32] (03CR) 10CI reject: [V:04-1] Docker: Drop all PHP 7.4 images [integration/config] - 10https://gerrit.wikimedia.org/r/1154080 (owner: 10Jforrester) [17:05:39] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/WikimediaMaintenance] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1154074 (https://phabricator.wikimedia.org/T58074) (owner: 10Máté Szabó) [17:06:53] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikimediaMaintenance] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1154074 (https://phabricator.wikimedia.org/T58074) (owner: 10Máté Szabó) [17:07:12] !log Zuul: [mediawiki/extensions/WikimediaMaintenance] Add dependencies, for T58074 [17:07:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:07:15] T58074: Echo: Generate periodic web notification to nudge users to confirm an unverified email address - https://phabricator.wikimedia.org/T58074 [17:07:35] (03CR) 10Jforrester: [C:03+2] Docker: [phpmetrics] Migrate parent image from php74 to php81 [integration/config] - 10https://gerrit.wikimedia.org/r/1154076 (owner: 10Jforrester) [17:08:12] (03PS2) 10Jforrester: Docker: [phpmetrics] Migrate parent image from php74 to php81 [integration/config] - 10https://gerrit.wikimedia.org/r/1154076 [17:08:12] (03PS2) 10Jforrester: jjb: [*-phpmetrics] Switch image to 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1154077 [17:08:12] (03PS2) 10Jforrester: Docker: [mediawiki-phan-testrun] Migrate parent image from php74 to php81 [integration/config] - 10https://gerrit.wikimedia.org/r/1154078 [17:08:12] (03PS2) 10Jforrester: jjb: [mw-tools-phan-mwcore-testrun] Switch image to 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1154079 [17:08:13] (03PS7) 10Jforrester: jjb: Drop PHP 7.4 jobs, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1127119 [17:08:15] (03PS2) 10Jforrester: Docker: Drop all PHP 7.4 images [integration/config] - 10https://gerrit.wikimedia.org/r/1154080 [17:08:23] (03CR) 10Jforrester: "…" [integration/config] - 10https://gerrit.wikimedia.org/r/1154076 (owner: 10Jforrester) [17:09:52] (03Merged) 10jenkins-bot: Docker: [phpmetrics] Migrate parent image from php74 to php81 [integration/config] - 10https://gerrit.wikimedia.org/r/1154076 (owner: 10Jforrester) [17:10:29] !log Where will Abstract Content go? [17:10:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:10:31] Err. [17:10:36] !log Docker: [phpmetrics] Migrate parent image from php74 to php81 [17:10:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:10:39] Thank you, clip board. [17:12:56] (03CR) 10Jforrester: [C:03+2] jjb: [*-phpmetrics] Switch image to 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1154077 (owner: 10Jforrester) [17:13:02] (03CR) 10Jforrester: [C:03+2] Docker: [mediawiki-phan-testrun] Migrate parent image from php74 to php81 [integration/config] - 10https://gerrit.wikimedia.org/r/1154078 (owner: 10Jforrester) [17:14:18] (03Merged) 10jenkins-bot: jjb: [*-phpmetrics] Switch image to 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1154077 (owner: 10Jforrester) [17:14:20] (03Merged) 10jenkins-bot: Docker: [mediawiki-phan-testrun] Migrate parent image from php74 to php81 [integration/config] - 10https://gerrit.wikimedia.org/r/1154078 (owner: 10Jforrester) [17:14:29] !log Docker: [mediawiki-phan-testrun] Migrate parent image from php74 to php81 [17:14:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:19:33] (03CR) 10Jforrester: [C:03+2] jjb: [mw-tools-phan-mwcore-testrun] Switch image to 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1154079 (owner: 10Jforrester) [17:20:43] (03Merged) 10jenkins-bot: jjb: [mw-tools-phan-mwcore-testrun] Switch image to 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1154079 (owner: 10Jforrester) [17:28:13] (03CR) 10Jforrester: [C:03+2] jjb: Drop PHP 7.4 jobs, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1127119 (owner: 10Jforrester) [17:28:31] (03CR) 10Jforrester: [C:03+2] Docker: Drop all PHP 7.4 images [integration/config] - 10https://gerrit.wikimedia.org/r/1154080 (owner: 10Jforrester) [17:29:23] (03Merged) 10jenkins-bot: jjb: Drop PHP 7.4 jobs, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1127119 (owner: 10Jforrester) [17:29:38] (03Merged) 10jenkins-bot: Docker: Drop all PHP 7.4 images [integration/config] - 10https://gerrit.wikimedia.org/r/1154080 (owner: 10Jforrester) [18:19:30] congrats James_F! (and everyone else who contributed) [18:23:24] apergos: Thanks! Yes, huge numbers of people helped get us to this point. [18:23:38] Now we get to unwind all the hacks we no longer need. [18:23:49] (03PS5) 10Jforrester: Docker: Re-build PHP images with php-uuid (and incidentally bump versions) [integration/config] - 10https://gerrit.wikimedia.org/r/1072320 (https://phabricator.wikimedia.org/T373752) [18:23:51] that has got to feel really goo, getting rid of all that cruft [18:23:54] (03CR) 10Jforrester: [C:03+2] Docker: Re-build PHP images with php-uuid (and incidentally bump versions) [integration/config] - 10https://gerrit.wikimedia.org/r/1072320 (https://phabricator.wikimedia.org/T373752) (owner: 10Jforrester) [18:24:00] *good [18:24:24] Yes, definitely. [18:25:06] (03Merged) 10jenkins-bot: Docker: Re-build PHP images with php-uuid (and incidentally bump versions) [integration/config] - 10https://gerrit.wikimedia.org/r/1072320 (https://phabricator.wikimedia.org/T373752) (owner: 10Jforrester) [18:26:24] !log Docker: Re-build PHP images with php-uuid (and incidentally bump versions), for T373752 [18:26:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:26:26] T373752: Build php-uuid package, and add to WMF production and CI - https://phabricator.wikimedia.org/T373752 [18:54:28] vgutierrez: Thanks, I see it working now. Looks like it was issued a couple hours after I signed off. https://crt.sh/?id=18817997322 [18:57:43] (03PS1) 10Jforrester: jjb: Switch jobs to newest PHP images with uuid [integration/config] - 10https://gerrit.wikimedia.org/r/1154102 (https://phabricator.wikimedia.org/T373752) [19:03:18] !log Update profile::tlsproxy::envoy::cfssl_options under deployment-mediawiki in Horizon, to include remaining the wildcard and m-dot subdomains under beta.wmcloud.org for wikibooks, wikimedia, wikinews, wikiquote, wikisource, wikiversity, wiktionary. ref T289318 [19:03:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:03:21] T289318: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318 [19:06:07] (03PS2) 10Jforrester: jjb: Switch jobs to newest PHP images with uuid [integration/config] - 10https://gerrit.wikimedia.org/r/1154102 (https://phabricator.wikimedia.org/T373752) [19:06:15] (03CR) 10Jforrester: [C:03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/1154102 (https://phabricator.wikimedia.org/T373752) (owner: 10Jforrester) [19:07:26] (03Merged) 10jenkins-bot: jjb: Switch jobs to newest PHP images with uuid [integration/config] - 10https://gerrit.wikimedia.org/r/1154102 (https://phabricator.wikimedia.org/T373752) (owner: 10Jforrester) [19:08:22] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318#10888799 (10Krinkle) >>! In T355281#9836179, @Tgr wrote: > Horizon hiera edits automatically create a commit in cloud/instance-puppet […]; I think it's good practice to l... [19:09:35] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10888800 (10LucasWerkmeister) >>! In T396012#10886096, @LucasWerkmeister wrote: >>>! In T396012#10886057, @taavi wrote: >> I set up a redire... [19:57:37] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318#10888891 (10bd808) https://meta.wikimedia.beta.wmcloud.org/wiki/Main_Page works. https://wikidata.beta.wmcloud.org/wiki/Wikidata:Main_Page gives an `Invalid host name (wi... [20:06:27] 10Beta-Cluster-Infrastructure: irc.beta.wmflabs.org is unresponsive - https://phabricator.wikimedia.org/T396088#10888902 (10bd808) I think this whole service needs to be rebuilt. When I did an audit of running services on 2025-02-06 I found that deployment-ircd was still running the software stack from https://w... [20:14:56] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-memc13 on project deployment-prep - https://phabricator.wikimedia.org/T396109#10888911 (10bd808) `lang=shell-session,counterexample bd808@deployment-memc13:~$ sudo -i puppet agent -tv Info: Using environment 'production' Info: Retrie... [20:18:18] (03PS1) 10VolkerE: Zuul: [wikimedia-ui-base] Sunset Wikimedia UI Base by removing [integration/config] - 10https://gerrit.wikimedia.org/r/1154110 (https://phabricator.wikimedia.org/T354310) [20:19:05] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-memc13 on project deployment-prep - https://phabricator.wikimedia.org/T396109#10888923 (10bd808) Caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1152274. That patch adds `profile::memcached::firewall_src_sets: ~` to hi... [20:19:37] (03PS2) 10VolkerE: Zuul: [wikimedia-ui-base] Sunset WikimediaUI Base by removing [integration/config] - 10https://gerrit.wikimedia.org/r/1154110 (https://phabricator.wikimedia.org/T354310) [20:20:33] !log Added `profile::memcached::firewall_src_sets: ~` to deployment-memc prefix puppet (T396109) [20:20:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:20:38] T396109: No Puppet resources found on instance deployment-memc13 on project deployment-prep - https://phabricator.wikimedia.org/T396109 [20:22:10] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-memc13 on project deployment-prep - https://phabricator.wikimedia.org/T396109#10888937 (10bd808) 05Open→03Resolved a:03bd808 https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/55cb264f4a0925e591e8f7e0854... [20:29:29] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance deployment-memc11 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:42:29] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance deployment-memc11 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:42:48] (03PS3) 10Jforrester: Zuul: [wikimedia-ui-base] Sunset WikimediaUI Base, archive repo's CI [integration/config] - 10https://gerrit.wikimedia.org/r/1154110 (https://phabricator.wikimedia.org/T354310) (owner: 10VolkerE) [20:43:13] (03CR) 10Jforrester: [C:03+2] Zuul: [wikimedia-ui-base] Sunset WikimediaUI Base, archive repo's CI [integration/config] - 10https://gerrit.wikimedia.org/r/1154110 (https://phabricator.wikimedia.org/T354310) (owner: 10VolkerE) [20:44:29] RESOLVED: [3x] PuppetAgentNoResources: No Puppet resources found on instance deployment-memc11 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:44:38] (03Merged) 10jenkins-bot: Zuul: [wikimedia-ui-base] Sunset WikimediaUI Base, archive repo's CI [integration/config] - 10https://gerrit.wikimedia.org/r/1154110 (https://phabricator.wikimedia.org/T354310) (owner: 10VolkerE) [20:44:52] !log Zuul: [wikimedia-ui-base] Sunset WikimediaUI Base, archive repo's CI, for T354310 [20:44:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:44:55] T354310: Sunset WikimediaUI Base - https://phabricator.wikimedia.org/T354310 [20:55:47] Project beta-scap-sync-world build #209655: 04FAILURE in 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/209655/ [21:04:34] (03CR) 10VolkerE: "#til" [integration/config] - 10https://gerrit.wikimedia.org/r/1154110 (https://phabricator.wikimedia.org/T354310) (owner: 10VolkerE) [21:05:32] Project beta-scap-sync-world build #209656: 04STILL FAILING in 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/209656/ [21:07:33] Krinkle: Are you/MW group wanting to announce the PHP 7.4 retirement? I don't want to steal your thunder. [21:07:39] 10Release-Engineering-Team (Radar), 10MediaWiki-extensions-VipsScaler, 10Structured Data Engineering, 06Structured-Data-Backlog, 10Wikimedia-Extension-setup: Undeploy VipsScaler from Wikimedia wikis - https://phabricator.wikimedia.org/T290759#10889091 (10ABreault-WMF) >>! In T290759#10613659, @Jdforreste... [21:08:51] 10Release-Engineering-Team (Radar), 10MediaWiki-extensions-VipsScaler, 10Structured Data Engineering, 06Structured-Data-Backlog, 10Wikimedia-Extension-setup: Undeploy VipsScaler from Wikimedia wikis - https://phabricator.wikimedia.org/T290759#10889114 (10ABreault-WMF) >>! In T290759#9886432, @Ladsgroup w... [21:15:47] Project beta-scap-sync-world build #209657: 04STILL FAILING in 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/209657/ [21:19:26] hrm, scap upset about php8.1 it seems [21:25:48] Project beta-scap-sync-world build #209658: 04STILL FAILING in 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/209658/ [21:35:47] Project beta-scap-sync-world build #209659: 04STILL FAILING in 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/209659/ [21:45:48] Project beta-scap-sync-world build #209660: 04STILL FAILING in 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/209660/ [21:46:11] 06Release-Engineering-Team, 10Scap, 06serviceops: Scap failing in beta with `Error: You are using an unsupported PHP version (PHP 7.4.33)` - https://phabricator.wikimedia.org/T396158 (10thcipriani) 03NEW [21:47:39] 06Release-Engineering-Team, 10Scap, 06serviceops: Scap failing in beta with `Error: You are using an unsupported PHP version (PHP 7.4.33)` - https://phabricator.wikimedia.org/T396158#10889270 (10thcipriani) I also wonder if this is related to {T380958} as `php_fpm` is the version passed to a script to gracef... [21:55:49] Project beta-scap-sync-world build #209661: 04STILL FAILING in 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/209661/ [22:05:12] 06Release-Engineering-Team, 10Scap, 06serviceops: Scap failing in beta with `Error: You are using an unsupported PHP version (PHP 7.4.33)` - https://phabricator.wikimedia.org/T396158#10889380 (10thcipriani) >>! In T396158#10889270, @thcipriani wrote: > I also wonder if this is related to {T380958} as `php_fp... [22:05:42] Project beta-scap-sync-world build #209662: 04STILL FAILING in 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/209662/ [22:14:47] James_F: feel free to. happy to review/co-sign if you like. [22:15:51] Project beta-scap-sync-world build #209663: 04STILL FAILING in 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/209663/ [22:16:10] (03open) 10dduvall: wmf: Only build for wmf- tags [repos/releng/zuul/zuul] (wmf/12.0.0) - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/8 [22:17:22] (03update) 10dduvall: wmf: Only build for wmf- tags [repos/releng/zuul/zuul] (wmf/12.0.0) - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/8 [22:20:31] 10Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T392174#10889436 (10thcipriani) 05Open→03Resolved Thank you @dduvall ! [22:25:57] Project beta-scap-sync-world build #209664: 04STILL FAILING in 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/209664/ [22:30:44] thcipriani: ugh. that T396158 would be because of https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1128015 landing and the hidden versioned php reference you found in scap.cfg. [22:30:45] T396158: Scap failing in beta with `Error: You are using an unsupported PHP version (PHP 7.4.33)` - https://phabricator.wikimedia.org/T396158 [22:31:04] yeah, trying to test my fix on beta now [22:31:19] and realizing this is outdated: https://wikitech.wikimedia.org/wiki/Release_Engineering/Runbook/Puppet_patches#Beta_Cluster_testing [22:31:35] I suppose this doesn't affect prod because prod runs that script while building a docker image rather than locally? [22:31:42] I don't know what the state of those php_fpm vars really are. That should have already been causing problems in the wmdebug hosts? [22:31:52] (that script = mergeMessageFileList.php) [22:31:55] Krinkle: that's my understanding, too [22:32:27] thcipriani: that shouldn't be out of date. I followed it last week [22:32:27] at least some of those fpm refs are used in prod, assuming scap is still the one restarting mwdebug hosts. [22:32:48] bd808: bash: cd: /var/lib/git/operations/puppet: No such file or directory [22:32:55] but those should be similar to prod in that they use appserver role and local php8.1-fpm [22:33:03] similr to beta* [22:33:10] the fpm stuff looks like it's passed to a script that ignores it and restarts all php-fpms on the host [22:33:19] I *think* we can ditch those [22:33:29] oh. you are in a releng page... https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/How_code_is_updated#Cherry-picking_a_patch_from_gerrit [22:33:32] but we'll need to do that in scap [22:34:00] * bd808 wonders why there was a second copy of beta puppet config instructions [22:34:01] then we can ditch the fpm vars in scap.cfg in puppet [22:34:19] * thcipriani hopes he didn't create it a month ago [22:35:50] Project beta-scap-sync-world build #209665: 04STILL FAILING in 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/209665/ [22:35:51] hrm I have the patch cherry picked, but puppet not picking it up [22:36:07] I think it has to do with git hooks [22:36:50] the puppetserver needs to be restarted. if you were su'ed to the gitpuppet user that should have happened magically [22:37:31] but yes, "magically" == git post-commit hook [22:38:03] * bd808 should have written the automation service to hide all of this already [22:38:17] yeah, manually re-ran the post-merge hook and that picked it up [22:39:09] unclear why no error and no worky post-cherry pick [22:39:42] I did scp the patch file rather than fetch the patch file from gerrit, otherwise process I followed seems like what's documented. [22:40:17] anyway, we'll see if this makes beta-scap-sync-world happy [22:45:50] Project beta-scap-sync-world build #209666: 04STILL FAILING in 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/209666/ [22:46:27] oh good. [22:49:22] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 10Scap, 06serviceops, 13Patch-For-Review: Scap failing in beta with `Error: You are using an unsupported PHP version (PHP 7.4.33)` - https://phabricator.wikimedia.org/T396158#10889479 (10bd808) [22:54:14] thcipriani: I wonder if scap is running mwscript in a container there like it does in prod, but with the wrong container? [22:54:31] that is now how I'm reading the code. [22:54:51] but I can't find the config where the wrong image is set [22:54:55] dduvall: does `scap mwscript` always use a container now? And if so where do we change the container used? [22:55:17] context is that we now need php8.1 to do anything [22:55:44] Project beta-scap-sync-world build #209667: 04STILL FAILING in 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/209667/ [22:56:10] I think the answers there are "yes" and the "mediawiki_runtime_image" config var, but I don't see that being overridden [22:56:22] `sudo -u mwbuilder -n -- /usr/bin/scap mwscript --no-local-config --directory /srv/mediawiki-staging --user www-data -- shell.php --wiki=enwiki` is a quick way to test if things are working [22:56:23] overwritten? whatevs. [22:57:13] it might be that prod will be broken too, it just isn't yet because the train won't carry that php8.1 update until next week [22:57:50] well. [22:58:00] the shell command yells at me for version, too [22:58:18] 10Release-Engineering-Team (Radar), 10MediaWiki-extensions-VipsScaler, 10Structured Data Engineering, 06Structured-Data-Backlog, and 2 others: Undeploy VipsScaler from Wikimedia wikis - https://phabricator.wikimedia.org/T290759#10889494 (10Ladsgroup) >>! In T290759#10889114, @ABreault-WMF wrote: >>>! In T2... [22:58:32] yeah, it should if things are using a container with php7.4 [22:58:36] oh, I think scap is out of date [23:00:50] in beta [23:00:55] a plain `mwscript shell.php enwiki` works because /usr/bin/php is 8.1.32 on the beta deploy server [23:01:23] 4.139.0 vs 4.171.0 in prod [23:01:51] and that's where we set the base image...somewhere between those two. [23:01:55] (03approved) 10jhuneidi: builder: Support `mounts` and `caches` [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/125 (owner: 10dduvall) [23:02:03] so options: update scap or add it to the beta scap config [23:02:20] updating scap seems like a good start [23:02:36] if that is still broken cool we have more to do [23:05:46] Project beta-scap-sync-world build #209668: 04STILL FAILING in 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/209668/ [23:06:08] following https://github.com/wikimedia/mediawiki-tools-scap/blob/master/RELEASE.md#beta-release [23:11:54] ok...maybe it's working [23:12:05] it's no longer erroring out early in any case [23:12:22] nevermind [23:14:39] well. it's waiting on a jenkins deploy to finish, so that's something. [23:16:19] bd808: what thcipriani said :) [23:16:23] heh, well it updated on the deploy host [23:16:28] so it'll probably just work now [23:16:39] but it'd be nice to update it everywhere [23:16:55] but it's waiting on; https://integration.wikimedia.org/ci/job/beta-scap-sync-world/209669/console [23:17:14] Yippee, build fixed! [23:17:14] Project beta-scap-sync-world build #209669: 09FIXED in 2 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/209669/ [23:17:45] a "I figured it out from first principles" barnstar for thcipriani :) [23:18:19] I never met a problem I couldn't make harder by not reading the docs or the code [23:19:27] I like to try the "just ping the person who might know" lazy route some days. Other days, yeah YOLO and hold the magnet just right to flip bits on the disk as they whiz by. [23:19:43] yeah, doesn't seem like `mediawiki_runtime_image` is configured explicitly in prod so it will use the default which is a php8 image [23:20:17] (maybe you already figured that out) [23:20:18] yep, and seems like scap was out-of-date on beta [23:20:30] so it was using either ... not that [23:20:33] or the wrong image [23:20:37] you know what that means? i'm going to stop working and EAT ICE CREAM [23:20:55] 23:20:41 Installation of scap version "4.171.0" completed for 11 hosts [23:21:05] excellent [23:21:10] !log update scap in beta to 4.171.0 to match prod [23:21:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:21:24] oooh. now I am making a note to eat ice cream at the baseball game later tonight. mint chocolate chip out of a tiny batting helmet! [23:22:39] (03update) 10dduvall: wmf: Only build for wmf- tags [repos/releng/zuul/zuul] (wmf/12.0.0) - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/8 [23:23:13] bd808: good on ya, sir! [23:24:08] (03merge) 10dduvall: builder: Support `mounts` and `caches` [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/125 [23:25:58] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 10Scap, 06serviceops, 13Patch-For-Review: Scap failing in beta with `Error: You are using an unsupported PHP version (PHP 7.4.33)` - https://phabricator.wikimedia.org/T396158#10889516 (10thcipriani) 05Open→03Resolved a:03thcipriani @Sco... [23:29:08] 10Scap: Are `php_fpm`/`php_version` inside `scap.cfg` used anymore? - https://phabricator.wikimedia.org/T396166 (10thcipriani) 03NEW [23:32:54] 10Scap: Investigate no error output from failed `scap mwscript` command during `scap sync-world` - https://phabricator.wikimedia.org/T396169 (10thcipriani) 03NEW [23:38:20] (03open) 10thcipriani: Update scap beta release scripts [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/827 [23:38:24] (03update) 10thcipriani: Update scap beta release scripts [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/827 [23:42:43] bd808: also `git rebase -i` in beta to remove my patch gave me `ERROR: Exiting rather than deploying something surprising` [23:43:28] hmmm... [23:43:29] sounds like half error message, half threat [23:43:48] re-ran the hook and all's well and patch is uncherry-picked [23:44:29] `ERROR: Exiting rather than doing something I'll regret tomorrow because you're not worth it` [23:44:53] I wonder why the hooks don't like you? were you a bad person in a past life? [23:45:03] and current life [23:45:15] untrue ;) [23:45:17] but never happened before [23:46:36] anyway: I've got tps reports to file, and all's well. I'm going to add a note about manually running the hook to the docs and stop filing tasks/patches/doc updates [23:48:40] I got that warning/error too with a rebase -- https://phabricator.wikimedia.org/P77192 [23:59:00] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 10Scap, 06serviceops, 07Essential-Work: Scap failing in beta with `Error: You are using an unsupported PHP version (PHP 7.4.33)` - https://phabricator.wikimedia.org/T396158#10889621 (10thcipriani) [23:59:40] 10Gerrit, 06collaboration-services, 10Wikimedia-GitHub, 07Essential-Work, 13Patch-For-Review: github mirror out of sync - https://phabricator.wikimedia.org/T395887#10889623 (10thcipriani)