[00:14:19] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-maps-master02 in project deployment-prep - https://phabricator.wikimedia.org/T405888#11248233 (10Dzahn) The chain that causes this is: - class profile::logoutd sets a base_dir "/etc/wikimedia/logout.d" and uses wmflib::dir::mkdi... [02:39:50] 06Project-Admins: Create project tag for Sinitic-Sites - https://phabricator.wikimedia.org/T406456#11248309 (10Winston_Sung) [02:39:57] 06Project-Admins: Create project tag for Sinitic-Sites - https://phabricator.wikimedia.org/T406456#11248310 (10Winston_Sung) [03:47:11] 10Phabricator (Upstream), 07Upstream: Record in task histories when subscribers are added during a task's creation - https://phabricator.wikimedia.org/T405245#11248339 (10Sdkb) I think it may be that if I'm subscribed to a parent task, I get automatically subscribed to any subtasks. There are certainly ple... [06:32:01] 10Phabricator (Upstream), 07Upstream: Record in task histories when subscribers are added during a task's creation - https://phabricator.wikimedia.org/T405245#11248590 (10A_smart_kitten) >>! In T405245#11203030, @Aklapper wrote: > The current behavior is consistent with setting other fields, e.g. assignee... [07:00:55] 10Continuous-Integration-Config, 10Librarization, 10MediaWiki-Vendor, 07Developer Productivity, 13Patch-For-Review: Add Jenkins job to mediawiki/vendor repo to verify output matches clean build - https://phabricator.wikimedia.org/T74952#11248643 (10hashar) > There's an old CI request to enforce that CI's... [08:08:39] (03Abandoned) 10TheDJ: Make AdvancedSearch a gated extension [integration/config] - 10https://gerrit.wikimedia.org/r/1192540 (https://phabricator.wikimedia.org/T224587) (owner: 10TheDJ) [08:42:08] PROBLEM - gerrit process on gerrit2003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /var/lib/gerrit/review_site/bin/gerrit.war daemon -d /var/lib/gerrit/review_site https://wikitech.wikimedia.org/wiki/Gerrit [09:04:02] !log gerrit: deleted invalid ref `refs/master`Β (funny refname) from mediawiki/services/ores/deploy | T275946 [09:04:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:04:04] T275946: Can't delete weird ref using git in Gerrit - https://phabricator.wikimedia.org/T275946 [09:04:08] RECOVERY - gerrit process on gerrit2003 is OK: PROCS OK: 1 process with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /var/lib/gerrit/review_site/bin/gerrit.war daemon -d /var/lib/gerrit/review_site https://wikitech.wikimedia.org/wiki/Gerrit [09:04:18] 10Gerrit: sync-with-gerrit causes issues when run on macos - https://phabricator.wikimedia.org/T406503#11248981 (10Peachey88) History for LDAPGroups {T200736} {T277010} [09:04:18] oh my god [09:04:20] I should mask [09:04:37] it is probably not a problem [09:06:09] 10Phabricator, 06Release-Engineering-Team (Doing 😎): In the Diffusion "Clone" dialog, show gerrit URI instead of gerrit-replica URI - https://phabricator.wikimedia.org/T406553 (10Aklapper) 03NEW p:05Triageβ†’03Low [09:10:49] 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎): In the Diffusion "Clone" dialog, show gerrit URI instead of gerrit-replica URI - https://phabricator.wikimedia.org/T406553#11249034 (10Aklapper) https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/commit/04820fc6471fe64ed724420f7e... [09:12:34] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-maps-master02 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [09:19:08] !log gerrit: deleted invalid refs `refs/for2.4.4` and `refs/for3.0-beta-9`Β (funny refname) from mediawiki/skins/Metrolook | T275946 [09:19:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:19:10] T275946: Can't delete weird ref using git in Gerrit - https://phabricator.wikimedia.org/T275946 [09:19:38] IIRC that is because people with Owner permission can push to anything under `refs/*` [09:47:03] !log gerrit: deleted invalid ref `refs/master` cb852fd9c6522d94127177a88f59da52bc2b93eb (funny refname) from wikidata/browsertests | T275946 [09:47:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:47:06] T275946: Can't delete weird ref using git in Gerrit - https://phabricator.wikimedia.org/T275946 [09:54:50] 10Diffusion, 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎): In the Diffusion "Clone" dialog, show gerrit URI instead of gerrit-replica URI - https://phabricator.wikimedia.org/T406553#11249263 (10A_smart_kitten) [09:55:22] !log gerrit: deleted invalid ref `refs/master`Β (funny refname) from operations/debs/kubeyaml | T275946 [09:55:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:55:24] T275946: Can't delete weird ref using git in Gerrit - https://phabricator.wikimedia.org/T275946 [09:56:26] !log gerrit: deleted invalid ref `refs/master`Β (funny refname) from labs/private | T275946 [09:56:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:59:17] 10Gerrit, 06Release-Engineering-Team (Development services), 14Release-Engineering-Team-TODO, 07Upstream: Can't delete weird ref using git in Gerrit - https://phabricator.wikimedia.org/T275946#11249276 (10hashar) [09:59:18] 10Gerrit, 06Release-Engineering-Team (Seen), 07Upstream: Investigate Gerrit git repositories having invalid references - https://phabricator.wikimedia.org/T337508#11249277 (10hashar) [10:10:13] 10Phabricator (Upstream), 07Upstream: Record in task histories when subscribers are added during a task's creation - https://phabricator.wikimedia.org/T405245#11249332 (10Aklapper) [10:16:57] 10Gerrit, 06Release-Engineering-Team (Seen), 13Patch-For-Review, 07Upstream: Investigate Gerrit git repositories having invalid references - https://phabricator.wikimedia.org/T337508#11249339 (10hashar) I went to propose https://gerrit.wikimedia.org/r/c/All-Users/+/1194149 which only blocks `refs/master`.... [10:18:23] 10GitLab, 06collaboration-services, 10VPS-project-devtools, 13Patch-For-Review: Puppet failure on gitlab-1002.devtools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T406234#11249346 (10Jelto) 05Openβ†’03Resolved The puppet issue is fixed, I'll resolve the task. [10:21:52] 10Gerrit, 06Release-Engineering-Team (Seen), 13Patch-For-Review, 07Upstream: Investigate Gerrit git repositories having invalid references - https://phabricator.wikimedia.org/T337508#11249363 (10hashar) On https://gerrit.wikimedia.org/r/admin/repos/test/gerrit-ping,access if I try to enter a ref `^refs/[^/... [11:09:31] 10Gerrit, 06collaboration-services: gerrit: config tweaks - https://phabricator.wikimedia.org/T406333#11249510 (10hashar) > [ ] transparent huge pages β†’ madvise on Debian + -XX:+UseTransparentHugePages on the JVM It is barely ever stalling on memory, I don't think it is worth optimizing. See [[ https://grafan... [12:43:05] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 13Patch-For-Review, 07Technical-Debt: Drop deprecated phpunit.php and suite.xml - https://phabricator.wikimedia.org/T395470#11249843 (10Daimona) [12:43:27] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 13Patch-For-Review, 07Technical-Debt: Drop deprecated phpunit.php and suite.xml - https://phabricator.wikimedia.org/T395470#11249844 (10Daimona) [13:13:31] 10Continuous-Integration-Config, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team: Block the merging of patches to MediaWiki core that would break CentralAuth tests - https://phabricator.wikimedia.org/T406582 (10Tgr) 03NEW [13:13:45] 10Continuous-Integration-Config, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team: Add CentralAuth to gated extensions - https://phabricator.wikimedia.org/T333541#11249939 (10Tgr) Right, the PHPUnit code overrides CentralAuth and restores the core authentication provider, but that does not help... [13:16:43] !log gerrit: update description of IPSet repository to "[ARCHIVED] wikimedia/ip-set library - Obsolete, has been fold in https://gerrit.wikimedia.org/g/mediawiki/libs/IPUtils/ - T401155" [13:16:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:16:46] T401155: Archive wikimedia/ip-set library - https://phabricator.wikimedia.org/T401155 [13:18:00] !log gerrit: removed ACLs for IPSet repo and reparented it with All-Archived-Projects to prevent replication to GitHub # T401155 [13:18:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:19:13] arnaudb: I have archived IPSet.git repo and it is now excluded from the GitHub replication [13:19:49] it is gone from `gerrit replication list --remote github [13:22:44] thanks! [13:33:16] hashar: Err. [13:33:22] hashar: We've not blanked the repo yet. [13:33:27] And now we can't. [13:34:54] (03PS1) 10Jforrester: Zuul: [IPSet] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1194195 (https://phabricator.wikimedia.org/T401155) [13:35:24] Also we can't publish emergency new versions for 1.39 that still uses it, if e.g. there's a security issue (though I doubt that's a big risk in the next two months). [13:38:43] 10Gerrit: PHP libraries as Gerrit top-level projects - https://phabricator.wikimedia.org/T125031#11250050 (10Jdforrester-WMF) [13:49:57] 10Gerrit, 06Release-Engineering-Team (Seen), 13Patch-For-Review, 07Upstream: Investigate Gerrit git repositories having invalid references - https://phabricator.wikimedia.org/T337508#11250114 (10hashar) The trailing `*` causes `InvalidNameException` somehow. I workaround it replacing it with a `+`Β to form:... [13:53:31] 06Release-Engineering-Team, 10ChangeProp, 10Citoid, 06Content-Transform-Team, and 19 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11250145 (10Jdforrester-WMF) [14:06:51] James_F: ah good point REL1_39 support [14:06:53] then based on https://phabricator.wikimedia.org/T401155 [14:07:10] If we archive it in CI we can still force-merge. [14:07:15] it seems the old branches should be updated to use the new wikimedia/ip-utils [14:07:27] which is 5.0.0 [14:07:34] But if you use your ultra-special powers to make the repo inaccessible, no-one but you can fix anything. [14:07:51] Yes, as you can see from the task the MW team decided not to do that, but just support it until December. [14:07:53] cause I did not realize we needed it for REL1_39 [14:07:55] therefore [14:07:58] we can't archive it yet [14:07:59] OR [14:08:42] we gotta upgrade the lib in REL1_39, and if I read the task I feel like Krinkle envisioned switching the repos from wikimedia/ip-set to wikimedia/ip-utils>=5.0.0 [14:08:52] No. [14:09:09] We did the switch. It landed in 1.41 or whatever. [14:09:17] [] Update older stable/LTS branches of MediaWiki. https://www.mediawiki.org/wiki/Version_lifecycle [14:09:17] [] MediaWiki 1.39 LTS (core+vendor) [14:09:26] Changing the old library for a new one isn't work the MW team is planning. [14:09:35] yeah that is fine to me [14:09:42] And they shouldn't do it, bluntly; that's a breaking change for downstream code. [14:09:48] Not appropriate for a security release. [14:09:49] so that means we need to keep wikimedia/ip-set project active in Gerrit [14:09:54] Indeed. [14:09:57] so we can release a new version of it for REL1_39 [14:09:58] and thus [14:10:02] un archiving it :] [14:12:05] 10Continuous-Integration-Config, 10MediaWiki-libs-IPUtils, 10Projects-Cleanup, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Archive wikimedia/ip-set library - https://phabricator.wikimedia.org/T401155#11250280 (10hashar) Per a discussion with @Jdforrester-WMF , vendor/core had ip-set removed wi... [14:12:20] I did that cause Gerrit failed to replicate to https://github.com/wikimedia/IPSet [14:12:24] because it hasbeen archived [14:12:31] Right, but prematurely. [14:12:43] oh [14:12:46] Pre-REL1_39, repo not emptied, mirrors not deleted. [14:12:48] I am just pressing buttons :b [14:12:53] Yes. :-P [14:13:07] !log unarchived https://github.com/wikimedia/IPSet/ # T401155 [14:13:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:13:21] T401155: Archive wikimedia/ip-set library - https://phabricator.wikimedia.org/T401155 [14:15:14] AH [14:15:24] I can't remember the magic push option [14:19:08] (03PS1) 10Hashar: Unarchive the repository, still needed for REL1_39 [IPSet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1194204 (https://phabricator.wikimedia.org/T401155) [14:19:15] James_F: ^:) [14:19:41] Looks reasonable. But merge conflicts? Also I don't have any rights for that repo now. :-) [14:19:52] inheritFrom: All-Archived-Projects needs to change. [14:19:57] yeah cause of the decription let me fix it [14:20:09] Not just that. [14:21:51] (03open) 10jforrester: releases: Bump covers-version to 1.7.0 and raise priority [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/94 (https://phabricator.wikimedia.org/T395743) [14:23:48] (03PS2) 10Hashar: Unarchive the repository, still needed for REL1_39 [IPSet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1194204 (https://phabricator.wikimedia.org/T401155) [14:24:09] cause I reverted the wrong commit :b [14:24:22] so basically [14:24:28] 10GitLab (CI & Job Runners), 06collaboration-services, 10Toolforge, 06cloud-services-team (FY2025/26-Q1), 13Patch-For-Review: tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742#11250330 (10Jelto) [14:24:34] I went to remove read-only state [14:24:39] fetched refs/meta/config [14:24:42] reverted the commimts [14:25:04] squashed, fixed conflict and then `git push origin HEAD:refs/for/refs/meta/config` [14:25:24] Ack. LGTM. [14:25:47] (03CR) 10Hashar: [V:03+2 C:03+2] Unarchive the repository, still needed for REL1_39 [IPSet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1194204 (https://phabricator.wikimedia.org/T401155) (owner: 10Hashar) [14:26:14] James_F: last minute call for you to Code-Review +1 the change with reason "Ack. LGTM." [14:26:15] :b [14:26:18] that is for your stats! [14:26:23] 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎), 06collaboration-services: Deploy Phabricator/Phorge 2025-10-07 - https://phabricator.wikimedia.org/T406597 (10brennen) 03NEW [14:26:47] (03open) 10brennen: update submodules for 2025-10-07 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/82 (https://phabricator.wikimedia.org/T406597) [14:27:03] (03CR) 10Jforrester: [C:03+1] Unarchive the repository, still needed for REL1_39 [IPSet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1194204 (https://phabricator.wikimedia.org/T401155) (owner: 10Hashar) [14:27:07] There you go. [14:27:18] 8) [14:27:40] thank you for having raised REL1_39 support! [14:27:47] 10Continuous-Integration-Config, 10MediaWiki-libs-IPUtils, 10Projects-Cleanup, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Archive wikimedia/ip-set library - https://phabricator.wikimedia.org/T401155#11250386 (10hashar) [14:27:56] I would dearly love not to! [14:27:58] now my question is where is the REL1_39 EOL task :) [14:28:03] REL1_39 is the bane of my support work. [14:28:09] no shit [14:28:12] I mean [14:28:48] surely maintaining software from 3 years ago is a pain [14:29:10] (03update) 10brennen: update submodules for 2025-10-07 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/82 (https://phabricator.wikimedia.org/T406597) [14:29:20] Yes, but a bunch of things came together in 1_39 to make it a little worse than normal, thanks to COVID. [14:29:22] on the other hands that probably saves a ton of times to downstream reusers so that they don't have to update every six months or so [14:30:07] and the LTS REL sounds like a somehow arbitrary cut [14:30:27] like, I am not sure we had migrated all classes out of the global namespace yet [14:30:35] We haven't done it today. [14:30:38] Let alone 3 years ago. [14:30:39] ah [14:30:50] * hashar loves decade long projects [14:31:01] it is tedious for sure :\ [14:31:02] See also: Parsoid. [14:31:42] do we have a task to EOL 1.39? [14:31:47] Yes. [14:32:02] It has a couple of things I've marked as it blocking. [14:32:11] T403199 -> T395470 etc. [14:32:12] T403199: Formally EOL MW 1.39 - https://phabricator.wikimedia.org/T403199 [14:32:12] T395470: Drop deprecated phpunit.php and suite.xml - https://phabricator.wikimedia.org/T395470 [14:32:21] Hmm, only one thing apparently? [14:32:26] I felt like the archival of IPSET (T401155) could have the EOL task as a subtask [14:32:27] T401155: Archive wikimedia/ip-set library - https://phabricator.wikimedia.org/T401155 [14:32:33] Yeah, that makes sense. [14:32:34] but I don't want to mess with the process [14:32:42] on the other hand {BE BOLD} [14:32:45] {{done}} [14:32:55] 10Continuous-Integration-Config, 10MediaWiki-libs-IPUtils, 10Projects-Cleanup, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Archive wikimedia/ip-set library - https://phabricator.wikimedia.org/T401155#11250435 (10Jdforrester-WMF) [14:33:02] \o/ [14:33:21] 10Continuous-Integration-Config, 10MediaWiki-libs-IPUtils, 10Projects-Cleanup, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Archive wikimedia/ip-set library - https://phabricator.wikimedia.org/T401155#11250443 (10Jdforrester-WMF) [14:34:22] 10Continuous-Integration-Config, 10MediaWiki-libs-IPUtils, 10Projects-Cleanup, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Archive wikimedia/ip-set library - https://phabricator.wikimedia.org/T401155#11250445 (10Jdforrester-WMF) [14:36:31] 10GitLab, 06collaboration-services, 10VPS-project-devtools: Puppet failure on gitlab-1002.devtools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T406234#11250458 (10dancy) Thanks @jelto! [14:36:56] (03update) 10daimona: releases: Bump covers-validator to 1.7.0 and raise priority [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/94 (https://phabricator.wikimedia.org/T395743) (owner: 10jforrester) [14:37:27] (03merge) 10daimona: releases: Bump covers-validator to 1.7.0 and raise priority [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/94 (https://phabricator.wikimedia.org/T395743) (owner: 10jforrester) [14:37:43] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎): "Trying to access array offset on value of type null" in server pull logs for Phabricator Diffusion repository - https://phabricator.wikimedia.org/T406528#11250461 (10Aklapper) [14:37:51] (03PS1) 10Techwizzie: Zuul: [mediawiki/extensions/Wanda] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1194216 [14:39:06] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/Wanda] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1194216 (owner: 10Techwizzie) [14:40:45] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Wanda] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1194216 (owner: 10Techwizzie) [14:55:53] (03PS1) 10Hashar: Disable motd banner: maintenance window has closed [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194220 (https://phabricator.wikimedia.org/T387833) [14:57:49] (03CR) 10Arnaudb: "lgtm!" [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194220 (https://phabricator.wikimedia.org/T387833) (owner: 10Hashar) [14:58:43] (03CR) 10Arnaudb: [C:03+1] Disable motd banner: maintenance window has closed [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194220 (https://phabricator.wikimedia.org/T387833) (owner: 10Hashar) [14:58:44] (03merge) 10brennen: update submodules for 2025-10-07 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/82 (https://phabricator.wikimedia.org/T406597) [15:00:00] (03CR) 10Hashar: [V:03+2 C:03+2] Disable motd banner: maintenance window has closed [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194220 (https://phabricator.wikimedia.org/T387833) (owner: 10Hashar) [15:00:20] 10GitLab (CI & Job Runners), 06serviceops: failed to push docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.10.5-py3.11-2025-10-03-192132-3003d4328df66a0086a350fdd2ba1dbd80a235c5: unknown: blob upload invalid - https://phabricator.wikimedia.org/T406392#11250573 (10dancy) >>! In T406... [15:01:20] (03Merged) 10jenkins-bot: Disable motd banner: maintenance window has closed [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194220 (https://phabricator.wikimedia.org/T387833) (owner: 10Hashar) [15:02:06] (03CR) 10Hashar: [C:03+2] "I have deployed the change and CI is now enabled on the repository. You might want to send a change now to verify lint/tests are passing." [integration/config] - 10https://gerrit.wikimedia.org/r/1194216 (owner: 10Techwizzie) [15:03:57] !log gerrit: disabled banner announcing the Oct 6th maintenance ( https://gerrit.wikimedia.org/r/1194216 ) [15:03:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:05:42] 10Diffusion, 10Phabricator (2025-10-07), 06Release-Engineering-Team (Doing 😎): In the Diffusion "Clone" dialog, show gerrit URI instead of gerrit-replica URI - https://phabricator.wikimedia.org/T406553#11250598 (10Aklapper) 05Openβ†’03Resolved Deployed today, verified it works. [15:05:49] 10Diffusion, 10Phabricator (2025-10-07), 06Release-Engineering-Team (Doing 😎): Add text instruction to use gerrit-replica when setting Diffusion repository URI - https://phabricator.wikimedia.org/T406408#11250600 (10Aklapper) 05Openβ†’03Resolved Deployed today, verified it works. [15:06:26] 10Phabricator (2025-10-07), 06Release-Engineering-Team (Doing 😎), 07Technical-Debt: Remove custom Wikimedia xhpast Linter - https://phabricator.wikimedia.org/T404760#11250604 (10Aklapper) 05Openβ†’03Resolved Part 2 out of 2 deployed today, resolving. [15:07:37] 10Phabricator (2025-10-07), 06Release-Engineering-Team (Doing 😎), 06collaboration-services: Deploy Phabricator/Phorge 2025-10-07 - https://phabricator.wikimedia.org/T406597#11250610 (10Aklapper) 05Openβ†’03Resolved a:03brennen Today's deployment included the following changes: * {T406408} in https://... [15:21:26] (03PS1) 10Hashar: Disable component rather than motd plugin [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194225 (https://phabricator.wikimedia.org/T387833) [15:22:43] (03CR) 10Hashar: [C:03+2] Disable component rather than motd plugin [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194225 (https://phabricator.wikimedia.org/T387833) (owner: 10Hashar) [15:23:28] (03Merged) 10jenkins-bot: Disable component rather than motd plugin [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194225 (https://phabricator.wikimedia.org/T387833) (owner: 10Hashar) [15:28:18] (03open) 10dancy: scap remove-patch: support removing multiple patches [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1012 (https://phabricator.wikimedia.org/T404593) [15:28:21] (03update) 10dancy: scap remove-patch: support removing multiple patches [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1012 (https://phabricator.wikimedia.org/T404593) [15:52:30] 06Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Improve garbage collection of unused MediaWiki images on deployment host - https://phabricator.wikimedia.org/T387927#11250902 (10dancy) 05In progressβ†’03Resolved We now have a systemd timer which runs `scap clean-images` weekly. And users in the... [15:53:45] 06Release-Engineering-Team (Radar), 06serviceops-radar, 06SRE: deployment server - low disk space on /srv - https://phabricator.wikimedia.org/T387796#11250909 (10dancy) 05Openβ†’03Resolved a:03dancy We now have a systemd timer which runs `scap clean-images` weekly. And users in the `deployment` grou... [15:55:54] (03update) 10dancy: scap remove-patch: support removing multiple patches [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1012 (https://phabricator.wikimedia.org/T404593) [15:56:30] (03update) 10dancy: scap remove-patch: support removing multiple patches [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1012 (https://phabricator.wikimedia.org/T404593) [15:56:35] (03update) 10dancy: scap remove-patch: support removing multiple patches [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1012 (https://phabricator.wikimedia.org/T404593) [16:02:09] (03update) 10dancy: scap remove-patch: support removing multiple patches [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1012 (https://phabricator.wikimedia.org/T404593) [16:03:17] 10GitLab (Upstream pit of despair πŸ•³οΈ), 06Release-Engineering-Team (Seen): Gitlab "Review changes" button not working - https://phabricator.wikimedia.org/T400091#11250992 (10dancy) 05Openβ†’03Resolved a:03dancy This appears to be resolved as of GitLab Community Edition v18.2.7 [16:05:53] (03update) 10dancy: scap remove-patch: support removing multiple patches [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1012 (https://phabricator.wikimedia.org/T404593) [16:14:28] 06Release-Engineering-Team (Doing 😎), 10wikimedia.biterg.io: Create account for Lauren de Lench on wikimedia.biterg.io - https://phabricator.wikimedia.org/T403836#11251060 (10Aklapper) 05Openβ†’03Resolved @ldelench_wmf No reply; assuming this is resolved. Please reopen if there is more to sort out - thanks! [16:31:39] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T405679#11251087 (10Urbanecm_WMF) ##### Risky Patch! πŸš‚πŸ”₯ * **Change**: https://gerrit.wikimedia.org/r/1192248, https://gerrit.wikimedia.... [16:37:14] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [16:37:20] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [17:08:58] 10GitLab (CI & Job Runners), 06collaboration-services, 10Toolforge, 06cloud-services-team (FY2025/26-Q1), 13Patch-For-Review: tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742#11251327 (10fnegri) a:05fnegriβ†’03None I merged the patch above, but `reprepro` is fai... [18:14:06] 10Beta-Cluster-Infrastructure: Remove need for manually applied MaxMind data hacks on Beta Cluster cache servers - https://phabricator.wikimedia.org/T403105#11251657 (10bd808) Something has changed making this an active problem again. While debugging an unblocking failure (T406091) I found: `lang=shell-session b... [18:20:24] 10Beta-Cluster-Infrastructure: Remove need for manually applied MaxMind data hacks on Beta Cluster cache servers - https://phabricator.wikimedia.org/T403105#11251706 (10bd808) `name=/etc/haproxy/lua/maxmind-lookup.lua local isp_dbpath = "/usr/share/GeoIP/GeoIP2-ISP.mmdb" local datacenter_dbpath = "/usr/share/Geo... [18:40:06] 06Release-Engineering-Team (Radar), 10Cloud-VPS (Quota-requests): Grant gitlab-runners-staging access to fast-iops volume type and a 4xiops instance flavor - https://phabricator.wikimedia.org/T406271#11251744 (10Andrew) ` root@cloudcontrol1011:~# openstack volume type show 2da5d224-a8e8-42ac-b5c6-3d234441d814... [18:59:20] 06Release-Engineering-Team (Radar), 10Cloud-VPS (Quota-requests): Grant gitlab-runners-staging access to fast-iops volume type and a 4xiops instance flavor - https://phabricator.wikimedia.org/T406271#11251793 (10Andrew) I added the flavor via tofu [19:34:05] thcipriani: you might remember about git throwing some "funny refname" that is Gerrit accepting a push to `refs/master`Β while cgit rejects it cause it requires at least two level of deepness `refs/xx/yyy` [19:34:33] all such refs I have found (a handful) are definitely typos/mistakes [19:35:12] so I think I have managed to craft a rule that BLOCK creating any references that is a single level aka block `refs/[^/]+` [19:35:14] https://gerrit.wikimedia.org/r/c/All-Users/+/1194149 :] [19:35:42] and from there a "gerrit replication" would not complains anymore about "some errors were found" [19:41:22] 10Beta-Cluster-Infrastructure: Remove need for manually applied MaxMind data hacks on Beta Cluster cache servers - https://phabricator.wikimedia.org/T403105#11251947 (10bd808) >>! In T403105#11251706, @bd808 wrote: > Hack achieved. :/ A similar hack has been applied on deployment-cache-upload08 as well. [19:47:18] 10Beta-Cluster-Infrastructure, 06Traffic, 07Documentation: Create a runbook for troubleshooting the CDN in deployment-prep - https://phabricator.wikimedia.org/T390213#11251967 (10bd808) https://cheatsheet.krishnaneupane.com/posts/haproxy has been helpful as more blocking has moved to haproxy. In our deployme... [19:57:26] !log Unblock 37.244.128.0/17 (T406565) [19:57:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:09:35] 10GitLab (Project Migration), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 10Fundraising analytics stack: Move wikimedia/fundraising/analytics from Gerrit to Gitlab - https://phabricator.wikimedia.org/T391404#11252075 (10AStein-WMF) @thcipriani re-answering your questions from... [20:17:15] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07ci-test-error, and 2 others: Various CI jobs failing after "mkdir: cannot create directory β€˜log’: Permission denied" - https://phabricator.wikimedia.org/T282893#11252165 (10cjming) just encountered this o... [20:36:51] 10GitLab (CI & Job Runners), 06serviceops: failed to push docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.10.5-py3.11-2025-10-03-192132-3003d4328df66a0086a350fdd2ba1dbd80a235c5: unknown: blob upload invalid - https://phabricator.wikimedia.org/T406392#11252254 (10amastilovic) >>! I... [20:42:47] 10Beta-Cluster-Infrastructure, 06Traffic: Copy the Traffic team on alerts for deployment-cache* hosts - https://phabricator.wikimedia.org/T406650 (10bd808) 03NEW [20:44:00] 10Beta-Cluster-Infrastructure, 06Traffic: Copy the Traffic team on alerts for deployment-cache* hosts - https://phabricator.wikimedia.org/T406650#11252285 (10bd808) [20:52:36] 10Beta-Cluster-Infrastructure, 06Traffic: Copy the Traffic team on alerts for deployment-cache* hosts - https://phabricator.wikimedia.org/T406650#11252325 (10bd808) @Aklapper what do you think about using Herald for this sort of thing? It would need to be a global rule something like: {F66736932,size=full} I... [21:02:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:02:39] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T406653 (10phaultfinder) 03NEW [21:07:41] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:12:46] (03CR) 10Tacsipacsi: Add a banner for a Gerrit switch over maintenance (031 comment) [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1193017 (https://phabricator.wikimedia.org/T387833) (owner: 10Hashar) [21:16:14] 10Beta-Cluster-Infrastructure, 06Traffic: Copy the Traffic team on alerts for deployment-cache* hosts - https://phabricator.wikimedia.org/T406650#11252371 (10ssingh) @bd808 and I discussed this today and decided to split this up in two parts: - Starting immediately and once we get alerts about broken Puppet r... [21:27:04] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10ConfirmEdit (CAPTCHA extension), 06Trust and Safety Product Team, and 2 others: ConfirmEdit hCaptcha: Get QUnit tests to run in CI - https://phabricator.wikimedia.org/T406231#11252420 (10thcipriani) To clarify, you all need to: - Add special s... [21:33:34] 10Beta-Cluster-Infrastructure: Email delivery for EmailAuth is extremely slow - https://phabricator.wikimedia.org/T406657 (10TheDJ) 03NEW [21:34:29] 10Beta-Cluster-Infrastructure: Email delivery for EmailAuth is extremely slow - https://phabricator.wikimedia.org/T406657#11252477 (10TheDJ) [21:39:51] 10Beta-Cluster-Infrastructure, 06Traffic: Copy the Traffic team on alerts for deployment-cache* hosts - https://phabricator.wikimedia.org/T406650#11252502 (10Aklapper) @bd808: I do not know how/where alertmanager/@wmcs-alerts currently sets the #Beta-Cluster-Infrastructure tag and task title for the Phab task... [21:39:55] 10Beta-Cluster-Infrastructure: Email delivery for EmailAuth is extremely slow - https://phabricator.wikimedia.org/T406657#11252504 (10TheDJ) 05Openβ†’03Resolved a:03TheDJ never mind, i actually found the cause, forgot that this account was on a secondary email domain that has lower delivery quality. [21:55:42] 10Beta-Cluster-Infrastructure: Email delivery for EmailAuth is extremely slow - https://phabricator.wikimedia.org/T406657#11252568 (10Tgr) You can check the code in Logstash (or the logfiles if Logstash is broken on beta). The MediaWiki job queue is not involved in sending the email, so maybe a mailer misco... [22:04:25] 10Beta-Cluster-Infrastructure, 06Traffic: Copy the Traffic team on alerts for deployment-cache* hosts - https://phabricator.wikimedia.org/T406650#11252612 (10bd808) >>! In T406650#11252502, @Aklapper wrote: > @bd808: I do not know how/where alertmanager/@wmcs-alerts currently sets the #Beta-Cluster-Infrastruct... [22:17:12] 06Project-Admins, 06Language and Product Localization, 10Projects-Cleanup, 10Wikipedia Subscriptions: Consider archiving Wikipedia Subscriptions - https://phabricator.wikimedia.org/T404331#11252652 (10Aklapper) @Nikerabbit Would you maybe know who can answer this? Thanks in advance! [23:08:27] 10Phabricator, 06Release-Engineering-Team (Radar), 06Traffic: Phabricator videos fail in Firefox ("Range" request gets 503 from Varnish) - https://phabricator.wikimedia.org/T397661#11252788 (10MusikAnimal) >>! In T397661#11244371, @Krinkle wrote: > Yes, it remains consistently broken in Firefox on Mac and Wi... [23:15:33] 10Beta-Cluster-Infrastructure, 06Traffic: Copy the Traffic team on alerts for deployment-cache* hosts - https://phabricator.wikimedia.org/T406650#11252792 (10bd808) >>! In T406650#11252612, @bd808 wrote: >>>! In T406650#11252502, @Aklapper wrote: >> I don't know the skillset of @Maintenance_bot which could be... [23:15:55] (03open) 10dduvall: jwt: Consider `project_path` and `job_project_path` claims [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/107 [23:16:01] (03update) 10dduvall: jwt: Consider `project_path` and `job_project_path` claims [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/107 [23:16:39] (03update) 10dduvall: jwt: Consider `project_path` and `job_project_path` claims [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/107 [23:17:41] (03update) 10dduvall: jwt: Consider `project_path` and `job_project_path` claims [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/107 [23:18:44] (03update) 10dduvall: jwt: Consider `project_path` and `job_project_path` claims [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/107 [23:22:50] (03update) 10dduvall: jwt: Consider `project_path` and `job_project_path` claims [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/107 [23:46:15] 10GitLab (CI & Job Runners), 06serviceops: failed to push docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.10.5-py3.11-2025-10-03-192132-3003d4328df66a0086a350fdd2ba1dbd80a235c5: unknown: blob upload invalid - https://phabricator.wikimedia.org/T406392#11252870 (10Scott_French) @ama... [23:48:15] (03update) 10thcipriani: CSS: ensure styles are attached to specific tags [repos/releng/logspam-finagler] - 10https://gitlab.wikimedia.org/repos/releng/logspam-finagler/-/merge_requests/1 [23:48:15] (03open) 10thcipriani: CSS: ensure styles are attached to specific tags [repos/releng/logspam-finagler] - 10https://gitlab.wikimedia.org/repos/releng/logspam-finagler/-/merge_requests/1 [23:48:15] (03open) 10thcipriani: Phab links in logstash filter context menu [repos/releng/logspam-finagler] - 10https://gitlab.wikimedia.org/repos/releng/logspam-finagler/-/merge_requests/2 [23:48:15] (03update) 10thcipriani: Phab links in logstash filter context menu [repos/releng/logspam-finagler] - 10https://gitlab.wikimedia.org/repos/releng/logspam-finagler/-/merge_requests/2 [23:48:22] (03update) 10thcipriani: CSS: ensure styles are attached to specific tags [repos/releng/logspam-finagler] - 10https://gitlab.wikimedia.org/repos/releng/logspam-finagler/-/merge_requests/1 [23:48:23] (03update) 10thcipriani: Phab links in logstash filter context menu [repos/releng/logspam-finagler] - 10https://gitlab.wikimedia.org/repos/releng/logspam-finagler/-/merge_requests/2 [23:48:59] (03update) 10thcipriani: CSS: ensure styles are attached to specific tags [repos/releng/logspam-finagler] - 10https://gitlab.wikimedia.org/repos/releng/logspam-finagler/-/merge_requests/1 [23:49:13] (03update) 10thcipriani: Phab links in logstash filter context menu [repos/releng/logspam-finagler] - 10https://gitlab.wikimedia.org/repos/releng/logspam-finagler/-/merge_requests/2