[00:00:35] FIRING: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [00:05:35] RESOLVED: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [00:28:13] 10Beta-Cluster-Infrastructure, 13Patch-For-Review, 07Spike: Figure out how to install PHP 8.1 on bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752#10407832 (10bd808) [00:30:59] 10Beta-Cluster-Infrastructure, 13Patch-For-Review, 07Spike: Install PHP 8.1 on deplopyment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752#10407836 (10bd808) [00:32:39] 10Beta-Cluster-Infrastructure, 07OKR-Work, 13Patch-For-Review: Install PHP 8.1 on deplopyment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752#10407849 (10bd808) [08:29:41] 10GitLab, 10Projects-Cleanup: Delete GitLab repo toolforge-repos/povoconta and its Diffusion mirror (code hosted in GitHub instead) - https://phabricator.wikimedia.org/T348325#10408319 (10Aklapper) I deleted the out-of-sync Diffusion repo at https://phabricator.wikimedia.org/source/tool-povoconta/ ; untagging... [08:29:46] 10GitLab, 10Projects-Cleanup: Delete GitLab repo toolforge-repos/povoconta (code hosted in GitHub instead) - https://phabricator.wikimedia.org/T348325#10408323 (10Aklapper) [09:05:26] 10Release-Engineering-Team (Priority Backlog πŸ“₯): Delete wmf branches from Gerrit repositories - https://phabricator.wikimedia.org/T303828#10408529 (10hashar) The timer ran at 5:00 UTC and the received email had: ` Systemd timer ran the following command: /usr/bin/scap clean --delete-gerrit-branch auto Its... [09:16:59] 10Release-Engineering-Team (Priority Backlog πŸ“₯): Delete wmf branches from Gerrit repositories - https://phabricator.wikimedia.org/T303828#10408552 (10hashar) That comes from 899ca98e567cd4a9b4cca1aabc340561ff4e9f17 > scap clean auto: Tweak old version selection logic > > Clean._autoselect_versions_to_remove()... [09:33:41] 10Release-Engineering-Team (Doing 😎), 05Release, 05Train Deployments: 1.44.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T375667#10408573 (10hashar) The `scap stage-train` commands failed to deploy to some of the testservers cause that took more than ten minutes to deploy? ` 04:23:39 Starte... [09:41:28] 10Release-Engineering-Team (Doing 😎), 05Release, 05Train Deployments: 1.44.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T375667#10408585 (10hashar) The deployment failed once again: ` Error: Failed to render chart: exit status 1: WARNING: Kubernetes configuration file is group-readable.... [11:40:13] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 2 others: [REPO][CLIENT][SW] Re-start Wikibase test coverage reporting - https://phabricator.wikimedia.org/T288396#10408811 (10ItamarWMDE) [11:41:00] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 2 others: [REPO][CLIENT][SW] Re-start Wikibase test coverage reporting - https://phabricator.wikimedia.org/T288396#10408813 (10ItamarWMDE) **Prio Notes:** | Impact Area... [11:41:11] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 2 others: [REPO][CLIENT][SW] Re-start Wikibase test coverage reporting - https://phabricator.wikimedia.org/T288396#10408814 (10ItamarWMDE) p:05Triageβ†’03Medium [11:41:51] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#10408816 (10Aklapper) @NBaca-WMF, @HCoplin-WMF: Added all mentioned accounts to #acl_project-admins. [Recommended practices for project and workboard managemen... [12:01:53] (03update) 10hnowlan: kubernetes: add safeties around replica check [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/614 (https://phabricator.wikimedia.org/T371700) [12:04:02] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 2 others: [REPO][CLIENT][SW] Re-start Wikibase test coverage reporting - https://phabricator.wikimedia.org/T288396#10409004 (10ItamarWMDE) [12:05:33] 10Continuous-Integration-Config, 10Page-Previews, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: CI is broken for lots of MW repos due to extension/Popups explicitly requiring node 18 - https://phabricator.wikimedia.org/T382192#10409024 (10Lucas_Werkmeister_WMDE) I backported the cleanup... [12:58:15] 10Continuous-Integration-Infrastructure, 07Zuul: Jenkins randomly says that patches in mediawiki/tools/phan "failed to merge" - https://phabricator.wikimedia.org/T382211#10409195 (10hashar) 05Openβ†’03Resolved a:03hashar From contint2002 in /var/log/zuul/merger.log.2024-12-14: ` cmdline: git clone -v... [13:07:01] I'm miles out of the loop but is there any known fix for errors like https://integration.wikimedia.org/ci/job/mwext-php74-phan/63759/console `mkdir: cannot create directory β€˜log’: Permission denied` ?Some nudging to fix permissions on the runner or something required? [13:10:03] It's probably not the actual error [13:10:28] except maybe not in that case [13:15:25] meh, 3rd try lucky it got through :) [14:04:23] 06Release-Engineering-Team, 06collaboration-services, 10dev-images: Permission denied while running dev-images/fab deploy_devimages - https://phabricator.wikimedia.org/T382285#10409377 (10Daimona) I think when I sent [[https://gitlab.wikimedia.org/repos/releng/dev-images/-/commit/fc1525ca3f8712f33771cf993fb3... [14:23:27] 10Phabricator, 07PM: Reset status of tickets that languish in "in progress" - https://phabricator.wikimedia.org/T380300#10409430 (10Aklapper) p:05Triageβ†’03Low [14:23:43] 10Phabricator: Phabricator should use IDP for developer account logins - https://phabricator.wikimedia.org/T377061#10409431 (10Aklapper) p:05Triageβ†’03Low [14:25:07] 10Phabricator (Upstream), 07Upstream: Make it possible to autocomplete task references in comments and task descriptions - https://phabricator.wikimedia.org/T364072#10409442 (10Aklapper) p:05Triageβ†’03Low [14:25:11] 10Phabricator (Upstream), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 07Upstream: DomainException: Attempting to iterate an object which is not iterable - https://phabricator.wikimedia.org/T373316#10409443 (10Aklapper) 05Openβ†’03Stalled p:05Triageβ†’03Low [14:27:01] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 13Patch-For-Review: Unable to preview MP4 video in Phabricator task comments and descriptions - https://phabricator.wikimedia.org/T309222#10409451 (10Aklapper) [14:27:17] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 13Patch-For-Review: Unable to preview MP4 video in Phabricator task comments and descriptions - https://phabricator.wikimedia.org/T309222#10409452 (10Aklapper) a:03Aklapper [14:27:51] 10GitLab (Auth & Access), 10Phabricator, 06cloud-services-team, 10Toolforge: Look for ways to consolidate "we trust this human" access lists - https://phabricator.wikimedia.org/T364516#10409459 (10Aklapper) [14:28:32] 10Phabricator: Fix commit identity of Valerio Bozz. - https://phabricator.wikimedia.org/T381461#10409475 (10Aklapper) p:05Triageβ†’03Low [14:28:39] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Wikimedia-Phabricator-Extensions, 07Browser-Support-Google-Chrome, 13Patch-For-Review: Phab login via SUL works only on second time with Chrome (due to CSP and redirect) - https://phabricator.wikimedia.org/T376803#10409477 (10Aklapper) [14:29:18] 10Continuous-Integration-Config, 10Page-Previews, 10ci-test-error (WMF-deployed Build Failure): CI is broken for lots of MW repos due to extension/Popups explicitly requiring node 18 - https://phabricator.wikimedia.org/T382192#10409482 (10Jdforrester-WMF) 05Openβ†’03Resolved a:03Lucas_Werkmeister_WMDE... [14:29:40] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10DataΒ³: Phab feature request: Cycle time for a task entering a column to resolution, with support for wildcards - https://phabricator.wikimedia.org/T148805#10409489 (10Aklapper) p:05Mediumβ†’03Low [14:31:16] 10Phabricator, 06Project-Admins: Allow the use of team projects as representation of teams - https://phabricator.wikimedia.org/T126055#10409499 (10Aklapper) p:05Mediumβ†’03Low [14:31:34] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 07Performance Issue: Submitting actions taking ~10 seconds to load after making changes - https://phabricator.wikimedia.org/T360484#10409503 (10Aklapper) p:05Mediumβ†’03Low [14:48:49] 06Release-Engineering-Team, 06Infrastructure-Foundations, 07Puppet: Puppet git::clone should default mode to 0644 (read-only) instead of 0755 - https://phabricator.wikimedia.org/T371980#10409584 (10hashar) 05Openβ†’03Declined My intent was to remove the `umask` parameter (T338277) which was completed.... [14:56:15] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Aditya0545 - https://phabricator.wikimedia.org/T382134#10409596 (10Adityakumar0545) Could you please fix it [15:07:02] 06Release-Engineering-Team, 06collaboration-services, 10dev-images, 10docker-pkg: Permission denied while running dev-images/fab deploy_devimages - https://phabricator.wikimedia.org/T382285#10409643 (10brennen) [15:14:35] FIRING: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [15:19:08] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Wikimedia-Phabricator-Extensions, 07Browser-Support-Google-Chrome, 13Patch-For-Review: Phab login via SUL works only on second time with Chrome (due to CSP and redirect) - https://phabricator.wikimedia.org/T376803#10409699 (10Tgr) Upstream... [15:20:15] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Adityakumar0545] - https://phabricator.wikimedia.org/T377037#10409708 (10Aklapper) β†’14Duplicate dup:03T382134 [15:20:17] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Aditya0545 - https://phabricator.wikimedia.org/T382134#10409710 (10Aklapper) [15:20:55] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Aditya0545 - https://phabricator.wikimedia.org/T382134#10409712 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Sorry! I approved the account https://gitlab.wikimedia.org/Aditya0545 [15:24:35] RESOLVED: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [15:46:12] 10Phabricator (Upstream), 10Release-Engineering-Team (Seen), 07Upstream: Hovercards: (RuntimeException) Undefined index: objectPHID (due to RequestURI all lowercase) - https://phabricator.wikimedia.org/T346876#10409836 (10Aklapper) 05Openβ†’03Stalled [15:47:41] 06Release-Engineering-Team, 06collaboration-services, 10dev-images, 10docker-pkg: Permission denied while running dev-images/fab deploy_devimages - https://phabricator.wikimedia.org/T382285#10409839 (10dancy) Looks like the `dockerpkg-builder` user isn't in the `docker`group. [15:52:05] hashar: Thanks for the train fixups! [15:52:58] dancy: hi! looks like some testserver timed out while deploying the images [15:53:18] and from one our conversation last week I went to resume it with `scap sync-world` [15:53:21] which did the magic [15:54:00] the new scap release had a patch to have `scap clean` to remove the old branch [15:54:13] but wmf.5 from two weeks ago did not get cleaned cause last week we did not have a branch [15:54:22] so the old old branch is just the last old branch [15:54:27] and we keep it [15:54:29] :) [15:54:29] nod [15:54:34] so beside that, all good [15:55:00] and I will be able to tell whether scap clean properly delete the branch from Gerrit on the next train which is January 16th (I have added a reminder for that) [15:55:02] it is a long task :b [15:57:40] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services: Deploy Phabricator/Phorge 2024-12-17 - https://phabricator.wikimedia.org/T382346 (10brennen) 03NEW [15:59:22] (03open) 10brennen: update submodules for 2024-12-17 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/58 (https://phabricator.wikimedia.org/T382346) [15:59:29] (03merge) 10brennen: update submodules for 2024-12-17 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/58 (https://phabricator.wikimedia.org/T382346) [16:03:04] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phabricator/Phorge 2024-12-17 - https://phabricator.wikimedia.org/T382346#10409920 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=e3b8575e-9d9c-411b-9386-adbd97dd7beb) set... [16:09:20] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phabricator/Phorge 2024-12-17 - https://phabricator.wikimedia.org/T382346#10409936 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=c37ebbbd-03ed-405a-9842-c8694794d7c8) set... [16:12:46] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Priority Backlog πŸ“₯): Reports: Order Over-due Tasks by oldest date instead of task ID - https://phabricator.wikimedia.org/T380907#10409960 (10Aklapper) 05Openβ†’03Resolved Confirming this is fixed by today's deployment. [16:13:38] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Wikimedia-Phabricator-Extensions: Weird indentation on task list (due to empty rendering of deadline subtype tag) - https://phabricator.wikimedia.org/T379376#10409974 (10Aklapper) 05Openβ†’03Resolved Confirming this is... [16:15:13] 10Phabricator (2024-12-17), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phabricator/Phorge 2024-12-17 - https://phabricator.wikimedia.org/T382346#10409979 (10Aklapper) 05Openβ†’03Resolved Thanks as usual! [16:55:22] !log Doing in-place PHP 8.1 upgrade on deployment-mwmaint03.deployment-prep.eqiad1.wikimedia.cloud (T378752) [16:55:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:55:25] T378752: Install PHP 8.1 on deplopyment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752 [17:02:57] 10Beta-Cluster-Infrastructure, 07OKR-Work, 13Patch-For-Review: Install PHP 8.1 on deplopyment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752#10410145 (10bd808) Notes from deployment-mwmaint03.deployment-prep.eqiad1.wikimedia.cloud: * Stable state after 2 puppet runs now. The fi... [17:06:11] 06Release-Engineering-Team, 06collaboration-services, 10dev-images, 10docker-pkg: Permission denied while running dev-images/fab deploy_devimages - https://phabricator.wikimedia.org/T382285#10410147 (10brennen) Hmm, yeah. Doesn't seem like it was at any point either: User originally created here: https://... [17:06:12] 10Beta-Cluster-Infrastructure, 07OKR-Work, 13Patch-For-Review: Install PHP 8.1 on deplopyment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752#10410146 (10bd808) [17:09:27] !log Doing in-place PHP 8.1 upgrade on deployment-jobrunner05.deployment-prep.eqiad1.wikimedia.cloud (T378752) [17:09:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:09:30] T378752: Install PHP 8.1 on deplopyment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752 [17:18:08] 10Phabricator (Upstream), 07Upstream: Exception: "Call to phutil_nonempty_string() expected null or a string, got: int" in PhabricatorRepository.php (due to Importing) - https://phabricator.wikimedia.org/T347483#10410178 (10Aklapper) Deployed the debug code, posting some excerpts here for `"action":"browse"` w... [17:19:27] (03open) 10aklapper: Revert "Add temporary debug output for T347483" [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/82 [17:20:02] 10Beta-Cluster-Infrastructure, 07OKR-Work, 13Patch-For-Review: Install PHP 8.1 on deplopyment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752#10410179 (10bd808) [17:20:10] (03merge) 10aklapper: Revert "Add temporary debug output for T347483" [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/82 [17:20:47] !log Doing in-place PHP 8.1 upgrade on deployment-mediawiki14.deployment-prep.eqiad1.wikimedia.cloud (T378752) [17:20:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:20:50] T378752: Install PHP 8.1 on deplopyment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752 [17:24:25] bd808: <3 [17:26:09] 10Beta-Cluster-Infrastructure, 07OKR-Work, 13Patch-For-Review: Install PHP 8.1 on deplopyment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752#10410187 (10bd808) [17:26:44] 10Gerrit: [Gerrit] Contribution is not visible based on email - https://phabricator.wikimedia.org/T382028#10410194 (10hashar) If I do https://gerrit.wikimedia.org/r/q/owner:0freerunning%2540gmail.com that currently works for me. The problem is Gerrit has three different accounts associated to that email address:... [17:27:17] !log gerrit set-account 5579 --inactive # T382028 [17:27:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:27:19] T382028: [Gerrit] Contribution is not visible based on email - https://phabricator.wikimedia.org/T382028 [17:27:24] !log gerrit set-account 6691 --inactive # T382028 [17:27:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:27:41] !log Doing in-place PHP 8.1 upgrade on deployment-mediawiki13.deployment-prep.eqiad1.wikimedia.cloud (T378752) [17:27:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:27:45] T378752: Install PHP 8.1 on deplopyment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752 [17:28:51] James_F: It takes a village, or something. :) [17:29:04] * James_F supposes he's the village idiot, in that case. [17:29:23] Thank you for making Beta better, anyway. Much love. [17:30:31] It may seem altruistic, but mostly I am saving myself from more meetings to discuss how we will roll out PHP 8.1 in the beta cluster. [17:30:45] "Switch it off" not being an option. [17:31:25] * taavi really hopes that https://bash.toolforge.org/quip/cN0u54YB0IdT-khD1IoA will be proven wrong and I can stop renewing whyisbetabroken.com :-) [17:32:04] 10Gerrit: [Gerrit] Contribution is not visible based on email - https://phabricator.wikimedia.org/T382028#10410215 (10hashar) 05Openβ†’03Resolved a:03hashar I have disabled the two most recent accounts which will prevent Gerrit from associating `0freerunning@gmail.com` to them: | j lodhi | 6691 | jayprak... [17:36:43] 10Beta-Cluster-Infrastructure, 07OKR-Work, 13Patch-For-Review: Install PHP 8.1 on deplopyment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752#10410223 (10bd808) [17:42:57] !log Doing in-place PHP 8.1 upgrade on deployment-mediawiki13.deployment-prep.eqiad1.wikimedia.cloud (T378752) [17:42:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:43:00] T378752: Install PHP 8.1 on deplopyment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752 [17:43:48] ugh. wrong one [17:43:50] !log Doing in-place PHP 8.1 upgrade on deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud (T378752) [17:43:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:48:36] 10Beta-Cluster-Infrastructure, 07OKR-Work, 13Patch-For-Review: Install PHP 8.1 on deplopyment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752#10410234 (10bd808) [17:52:32] 10Beta-Cluster-Infrastructure, 07OKR-Work, 13Patch-For-Review: Install PHP 8.1 on deployment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752#10410245 (10Aklapper) [17:55:56] In many ways I think "deplopyment" is appropriate. [17:56:15] (03open) 10taavi: Remove monitoring stuff [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/65 (https://phabricator.wikimedia.org/T381647) [17:59:04] (03update) 10taavi: Remove monitoring stuff [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/65 (https://phabricator.wikimedia.org/T381647) [17:59:12] taavi: Sorry for complaining. :-) [17:59:46] taavi: Are they going to share a Phabricator user still? [18:00:01] (03update) 10jforrester: Remove monitoring stuff [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/65 (https://phabricator.wikimedia.org/T381647) (owner: 10taavi) [18:00:36] James_F: depends of if I get around creating a new bot for the new thing [18:00:40] Ack. [18:01:33] (03update) 10taavi: Remove monitoring stuff [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/65 (https://phabricator.wikimedia.org/T381647) [18:02:43] (03approved) 10jforrester: Remove monitoring stuff [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/65 (https://phabricator.wikimedia.org/T381647) (owner: 10taavi) [18:02:54] (03update) 10jforrester: Remove monitoring stuff [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/65 (https://phabricator.wikimedia.org/T381647) (owner: 10taavi) [18:03:54] (03merge) 10taavi: Remove monitoring stuff [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/65 (https://phabricator.wikimedia.org/T381647) [18:24:41] !log attempting another update of development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/69 [18:24:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:29:29] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Rahulkundu1209 - https://phabricator.wikimedia.org/T382355 (10Rahulkundu1209) 03NEW [18:30:11] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Rahulkundu1209 - https://phabricator.wikimedia.org/T382355#10410389 (10Rahulkundu1209) [18:33:41] πŸŽ‰https://meta.wikimedia.beta.wmflabs.org/wiki/Special:Version says "PHP: 8.1.31" [18:34:05] Nice! [18:34:11] well done [18:34:30] nice :) [18:34:34] \o/ [18:39:44] !log Setting PHP 8.1 hiera config at the global level via Horizon (T378752) [18:39:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:39:47] T378752: Install PHP 8.1 on deployment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752 [18:46:07] 06Release-Engineering-Team, 06collaboration-services, 10dev-images, 10docker-pkg: Permission denied while running dev-images/fab deploy_devimages - https://phabricator.wikimedia.org/T382285#10410440 (10brennen) > So I'm guessing this would work if just run as my user under _ssh_command_self, like it was be... [18:53:33] 10Beta-Cluster-Infrastructure, 10Wikimedia-Portals: Wikimedia.org portal broken in Beta Cluster (Domain unavailable) - https://phabricator.wikimedia.org/T173887#10410494 (10matmarex) 05Openβ†’03Resolved a:03matmarex Works for me now. (I worked on this because we wanted to use this portal as a landing p... [18:53:39] (03close) 10jforrester: Upgrade all buster images to composer 2.6.4 [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/52 (https://phabricator.wikimedia.org/T347360) [19:12:26] (03open) 10thcipriani: branch.py: ensure we're tracking skins/* changes [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/139 [19:12:29] (03update) 10thcipriani: branch.py: ensure we're tracking skins/* changes [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/139 [19:16:25] (03PS2) 10Jforrester: Zuul: Fold last two node18 templates into their users [integration/config] - 10https://gerrit.wikimedia.org/r/1104705 [19:17:43] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T382360#10410664 (10thcipriani) 05Openβ†’03Declined Pausing for holiday break [19:18:23] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T382361#10410669 (10thcipriani) 05Openβ†’03Declined [19:18:29] (03CR) 10Jforrester: [C:03+2] Zuul: Fold last two node18 templates into their users [integration/config] - 10https://gerrit.wikimedia.org/r/1104705 (owner: 10Jforrester) [19:19:48] (03Merged) 10jenkins-bot: Zuul: Fold last two node18 templates into their users [integration/config] - 10https://gerrit.wikimedia.org/r/1104705 (owner: 10Jforrester) [19:20:06] !log Zuul: Fold last two node18 templates into their users [19:20:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:21:24] (03PS2) 10Jforrester: jjb: Split out last two node18 jobs as 'legacy' for simplicity [integration/config] - 10https://gerrit.wikimedia.org/r/1104706 [19:21:30] (03CR) 10Jforrester: [C:03+2] jjb: Split out last two node18 jobs as 'legacy' for simplicity [integration/config] - 10https://gerrit.wikimedia.org/r/1104706 (owner: 10Jforrester) [19:21:44] (03PS2) 10Jforrester: Zuul: Switch from generic-node18 to legacy-node18 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1104707 [19:23:18] (03Merged) 10jenkins-bot: jjb: Split out last two node18 jobs as 'legacy' for simplicity [integration/config] - 10https://gerrit.wikimedia.org/r/1104706 (owner: 10Jforrester) [19:23:28] (03CR) 10Jforrester: [C:03+2] Zuul: Switch from generic-node18 to legacy-node18 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1104707 (owner: 10Jforrester) [19:24:56] (03Merged) 10jenkins-bot: Zuul: Switch from generic-node18 to legacy-node18 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1104707 (owner: 10Jforrester) [19:25:12] !log Zuul: Switch from generic-node18 to legacy-node18 jobs [19:25:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:25:21] (03PS2) 10Jforrester: jjb: Switch miscellaneous jobs from node 18 to node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1104708 [19:29:20] (03CR) 10Jforrester: [C:03+2] jjb: Switch miscellaneous jobs from node 18 to node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1104708 (owner: 10Jforrester) [19:29:57] (03PS2) 10Jforrester: Docker: [ajv, sonar-scanner] Re-platform to Node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1104710 [19:30:06] (03PS2) 10Jforrester: jjb: Switch codehealth jobs to sonar-scanner images with Node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1104711 [19:30:42] (03Merged) 10jenkins-bot: jjb: Switch miscellaneous jobs from node 18 to node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1104708 (owner: 10Jforrester) [19:30:52] (03PS2) 10Jforrester: jjb: Drop various now-unused node16 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1104709 [19:33:28] (03PS3) 10Jforrester: jjb: Drop various now-unused node16 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1104709 [19:33:33] (03CR) 10Jforrester: [C:03+2] jjb: Drop various now-unused node16 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1104709 (owner: 10Jforrester) [19:33:39] (03CR) 10Jforrester: [C:03+2] Docker: [ajv, sonar-scanner] Re-platform to Node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1104710 (owner: 10Jforrester) [19:34:57] (03Merged) 10jenkins-bot: jjb: Drop various now-unused node16 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1104709 (owner: 10Jforrester) [19:35:00] (03Merged) 10jenkins-bot: Docker: [ajv, sonar-scanner] Re-platform to Node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1104710 (owner: 10Jforrester) [19:37:58] !log Docker: [ajv, sonar-scanner] Re-platform to Node 20 [19:38:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:08:33] 10Continuous-Integration-Config, 06Web-Team: Upgrade all CI jobs for WMF-deployed projects from Node 18 to Node 20 - https://phabricator.wikimedia.org/T343827#10410759 (10Jdrewniak) [20:20:57] (03open) 10dancy: update-repos: Update helm charts [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/115 [20:20:59] (03update) 10dancy: update-repos: Update helm charts [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/115 [20:24:08] (03merge) 10dancy: update-repos: Update helm charts [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/115 [20:28:48] thcipriani: BTW I created https://wikitech.wikimedia.org/wiki/Deployments/Yearly_calendar/2025 but it's pretty much all ?s as I didn't want to assume which way you'd go. :-) [20:29:48] 10Continuous-Integration-Config, 10Page-Previews, 10ci-test-error (WMF-deployed Build Failure): CI is broken for lots of MW repos due to extension/Popups explicitly requiring node 18 - https://phabricator.wikimedia.org/T382192#10410913 (10Tacsipacsi) (wrong task) [20:51:40] !log Removed per-instance PHP 8.1 hiera config used during rollout via Horizon (T378752) [20:51:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:51:43] T378752: Install PHP 8.1 on deployment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752 [20:53:07] !log Deleted deployment-mediawiki81 testing instance (T378752) [20:53:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:07:00] 10Continuous-Integration-Config, 13Patch-For-Review: Upgrade all CI jobs for WMF-deployed projects from Node 18 to Node 20 - https://phabricator.wikimedia.org/T343827#10411024 (10Jdrewniak) [21:16:01] James_F: <3 I'll bash it towards reality [21:16:10] thcipriani: Thank you, good sir. [21:21:52] 10Beta-Cluster-Infrastructure, 07OKR-Work, 13Patch-For-Review: Install PHP 8.1 on deployment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752#10411080 (10bd808) Upgrade announced to wikitech-l: https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/WXG4... [21:40:02] maintenance-disconnect-full-disks build 659052 integration-agent-docker-1053 (/: 28%, /srv: 99%, /var/lib/docker: 37%): OFFLINE due to disk space [21:55:56] (03update) 10dancy: kubernetes: add safeties around replica check [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/614 (https://phabricator.wikimedia.org/T371700) (owner: 10hnowlan) [21:58:09] (03merge) 10dancy: kubernetes: add safeties around replica check [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/614 (https://phabricator.wikimedia.org/T371700) (owner: 10hnowlan) [21:58:43] (03open) 10dancy: kubernetes.py: Only call _get_total_expected_replicas when needed [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/620 [21:58:46] (03update) 10dancy: kubernetes.py: Only call _get_total_expected_replicas when needed [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/620 [21:59:10] (03update) 10dancy: kubernetes.py: Only call _get_total_expected_replicas when needed [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/620 [22:02:22] (03approved) 10swfrench: kubernetes.py: Only call _get_total_expected_replicas when needed [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/620 (owner: 10dancy) [22:02:31] (03merge) 10dancy: kubernetes.py: Only call _get_total_expected_replicas when needed [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/620 [22:03:15] (03open) 10dancy: Release 4.134.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/621 [22:03:37] 06Release-Engineering-Team, 10dev-images, 10MediaWiki-Docker, 10MW-1.44-notes (1.44.0-wmf.12; 2025-01-14): Use PHP 8.3 in MediaWiki-Docker - https://phabricator.wikimedia.org/T381728#10411211 (10Daimona) 05Openβ†’03Resolved I've tested this again, including my custom dockerfiles, and it's looking gre... [22:04:34] (03merge) 10dancy: Release 4.134.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/621 [22:07:22] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10dev-images, 10MediaWiki-Docker: APT errors when installing custom packages in MediaWiki-Docker - https://phabricator.wikimedia.org/T362702#10411241 (10Daimona) 05Openβ†’03Resolved "Resolved" by replacing the images in T381728 :) And made T382379 for tr... [22:19:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10411260 [22:21:12] > 21:40:32 npm warn tar TAR_ENTRY_ERROR ENOSPC: no space left on device, write [22:21:16] https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php81-selenium/5755/console [22:21:20] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Math/+/1105069 [22:21:30] > Building remotely on integration-agent-docker-1053 (pi [22:21:33] FYI [22:27:17] hrm, the maintenance job disconnected that one earlier [22:32:58] yeah I pruned it manually [22:33:11] the wmf-quibble selenium jobs seems to be consuming a bunch of disk space [22:33:15] maybe because of Cypress [22:33:37] I am pondering removing some of them [22:41:25] thcipriani: two years ago I wrote a blog post about the Gerrit H2 database files being too large https://phabricator.wikimedia.org/phame/post/view/300/shrinking_h2_database_files/ [22:41:46] guess who had their Gerrit exploding on startup almost exactly two years later? Opendev! :-\ [22:41:57] I should have written a java code to cover it [22:43:59] * hashar sleeps [23:27:09] (03open) 10dancy: Fix mirror-repos update flag file handling [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/116 [23:27:11] (03update) 10dancy: Fix mirror-repos update flag file handling [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/116 [23:33:17] (03update) 10dancy: Fix mirror-repos update flag file handling [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/116 [23:46:53] (03merge) 10dancy: Fix mirror-repos update flag file handling [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/116 [23:55:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10411367