[00:02:20] !log Unblock 89.166.128.0/17 (T413702) [00:02:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:30:52] !log Unblock 85.134.128.0/17 (T413755) [00:30:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:40:21] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T412774#11494437 (10bd808) The instance shows 178 days of uptime. `lang=shell-session bd808@deployment-sessionstore06:~$ w 00:38:42 up 178 days, 9:48, 2 users, load average: 0... [00:46:51] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T412774#11494465 (10bd808) >>! In T412774#11494437, @bd808 wrote: > Is the alert flapping for some other reason like the instance being overloaded and that causing a scrape to fai... [00:50:03] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-mx03 in project deployment-prep - https://phabricator.wikimedia.org/T412949#11494485 (10bd808) →14Duplicate dup:03T412975 [00:50:06] 10Beta-Cluster-Infrastructure, 06collaboration-services, 13Patch-For-Review: Puppet failure: "Unable to locate package spamd" on deployment-mx03.deployment-prep - https://phabricator.wikimedia.org/T412975#11494482 (10bd808) [00:53:13] 10Beta-Cluster-Infrastructure, 06collaboration-services, 13Patch-For-Review: Puppet failure: "Unable to locate package spamd" on deployment-mx03.deployment-prep - https://phabricator.wikimedia.org/T412975#11494490 (10bd808) >>! In T412975#11478073, @thcipriani wrote: > Why not use the cloud MX? Dunno. I'm p... [00:54:00] 10Beta-Cluster-Infrastructure, 06collaboration-services, 13Patch-For-Review: Puppet failure: "Unable to locate package spamd" on deployment-mx03.deployment-prep - https://phabricator.wikimedia.org/T412975#11494491 (10bd808) [01:00:49] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T412774#11494495 (10Eevans) >>! In T412774#11494464, @bd808 wrote: >>>! In T412774#11494437, @bd808 wrote: >> Is the alert flapping for some other reason like the instance being o... [07:15:54] Thanks a lot, Thcipriani. [08:03:00] 10Gerrit, 10bacula, 06collaboration-services: Backup freshness: Stale bacula backup for gerrit1003 - https://phabricator.wikimedia.org/T413853 (10Jelto) 03NEW [08:06:24] (03CR) 10Hashar: [C:03+2] Disable banner for the 2025 developer survey [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1223167 (owner: 10Hashar) [08:07:05] (03Merged) 10jenkins-bot: Disable banner for the 2025 developer survey [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1223167 (owner: 10Hashar) [08:20:35] 10Gerrit, 10bacula, 06collaboration-services: Backup freshness: Stale bacula backup for gerrit1003 - https://phabricator.wikimedia.org/T413853#11494811 (10Dzahn) I ran the `check_bacula.py` and it shows: ` == jobs_with_stale_backups (6) == gerrit1003.wikimedia.org-Hourly-Tue-ReposEqiad-gerrit-repo-data ge... [08:21:13] 10Gerrit, 10bacula, 06collaboration-services: Backup freshness: Stale bacula backup for gerrit1003 - https://phabricator.wikimedia.org/T413853#11494812 (10Dzahn) I wouldn't be surprised if the "oldness check" fails somehow in relation to the year change. [08:25:25] 10Gerrit, 10bacula, 06collaboration-services: Backup freshness: Stale bacula backup for gerrit1003 - https://phabricator.wikimedia.org/T413853#11494822 (10Dzahn) manually started `job 68 (gerrit1003.wikimedia.org-Hourly-Tue-ReposEqiad-gerrit-repo-data)` via bconsole. [08:30:30] 10Gerrit, 10bacula, 06collaboration-services: Backup freshness: Stale bacula backup for gerrit1003 - https://phabricator.wikimedia.org/T413853#11494829 (10Dzahn) I tried to use `status` -> `client` and to connect from bconsole to the gerrit1003 client.. But that seems to just sit there: `Connecting to Cli... [08:39:12] 10Gerrit, 10bacula, 06collaboration-services: Backup freshness: Stale bacula backup for gerrit1003 - https://phabricator.wikimedia.org/T413853#11494836 (10Jelto) It looks like the disk is full on the backup hosts: https://alerts.wikimedia.org/?q=%40state%3Dactive&q=alertname%3DDiskSpace&q=cluster%3Dbackup ?... [08:45:43] 06Project-Admins: Create project tag for Tool-wikicontest - https://phabricator.wikimedia.org/T413628#11494841 (10Peachey88) Based on {T413623} I think they are working on the same codebase as the current tool account, they just don't have access to the tool account, to create the project [09:20:58] (03PS3) 10A smart kitten: Zuul: Add a few more experimental PHP 8.5 jobs to templates and repos [integration/config] - 10https://gerrit.wikimedia.org/r/1223189 (https://phabricator.wikimedia.org/T411814) (owner: 10Reedy) [09:21:51] (03CR) 10A smart kitten: Zuul: Add a few more experimental PHP 8.5 jobs to templates and repos (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1223189 (https://phabricator.wikimedia.org/T411814) (owner: 10Reedy) [09:24:38] 10Continuous-Integration-Config, 06Abstract Wikipedia team, 10WikiLambda, 03Abstract Wikipedia Fix-It tasks: Add the WikiLambda extension to the shared CI 'gate' - https://phabricator.wikimedia.org/T362085#11494886 (10A_smart_kitten) [09:24:43] 10Continuous-Integration-Config, 06Release-Engineering-Team (Seen), 10MW-on-K8s, 07Epic, 13Patch-For-Review: Have all Wikimedia production extensions and skins in the CI gate - https://phabricator.wikimedia.org/T249674#11494887 (10A_smart_kitten) [10:04:53] 06Project-Admins, 06Release-Engineering-Team (Doing 😎), 10CLDRPluralRuleParser, 06Language and Product Localization: Create tag for CLDRPluralRuleParser - https://phabricator.wikimedia.org/T411015#11494974 (10A_smart_kitten) [11:17:35] 10Phabricator: Silent batch-edit request for PHP 8.4/8.5 CI failure tasks currently tagged with the 'WMF-deployed Build Failure' project - https://phabricator.wikimedia.org/T413865 (10A_smart_kitten) 03NEW [11:42:37] 10GitLab (Infrastructure), 06collaboration-services: Make sure GitLab does not exceet apus object storage quotas - https://phabricator.wikimedia.org/T413871 (10Jelto) 03NEW [11:58:28] 10GitLab (Infrastructure), 06collaboration-services: Make sure GitLab does not exceed apus object storage quotas - https://phabricator.wikimedia.org/T413871#11495407 (10Aklapper) [12:10:03] maintenance-disconnect-full-disks build 769797 integration-agent-docker-1056 (/: 36%, /srv: 96%, /var/lib/docker: 35%): OFFLINE due to disk space [12:15:03] maintenance-disconnect-full-disks build 769798 integration-agent-docker-1056 (/: 36%, /srv: 51%, /var/lib/docker: 34%): RECOVERY disk space OK [13:16:35] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for ReaderExperiments-ImageBrowsing - https://phabricator.wikimedia.org/T413085#11495722 (10A_smart_kitten) H475 might want the extra condition `project tags incl. their milestones` `include none of` #reader_growth_team, in order to... [13:26:08] (03CR) 10Hashar: [C:03+2] quibble-coverage: allow coverage for Parsoid (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1223162 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [13:27:46] (03Merged) 10jenkins-bot: quibble-coverage: allow coverage for Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/1223162 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [13:32:21] (03merge) 10aklapper: Reduce bot comments and hide some bot actions [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/90 (https://phabricator.wikimedia.org/T393289) (owner: 10matmarex) [13:33:28] (03CR) 10Hashar: [C:03+2] jjb: update quibble-coverage for Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/1223163 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [13:34:55] (03Merged) 10jenkins-bot: jjb: update quibble-coverage for Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/1223163 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [13:40:59] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for ReaderExperiments-ImageBrowsing - https://phabricator.wikimedia.org/T413085#11495875 (10Aklapper) Oh very good point! Thanks so much for catching that! I should have checked if there are Sprint projects. :( I adjusted H475: * Wh... [13:53:13] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for ReaderExperiments-ImageBrowsing - https://phabricator.wikimedia.org/T413085#11495974 (10matthiasmullie) Thanks! [14:03:54] 10Continuous-Integration-Config, 10Parsoid, 10SonarQube Bot, 07ci-test-error, and 2 others: Parsoid patches are no longer being checked by SonarQube - https://phabricator.wikimedia.org/T413626#11496020 (10hashar) I have updated the jobs and sent a dummy change to Parsoid: https://gerrit.wikimedia.org/r/c/m... [14:05:34] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for ReaderExperiments-ImageBrowsing - https://phabricator.wikimedia.org/T413085#11496034 (10Aklapper) Sorry again for that (and the noise), I restored the Sprint tags on all recently edited tasks. [14:05:49] (03CR) 10A smart kitten: quibble-coverage: allow coverage for Parsoid (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1223162 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [14:07:05] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for ReaderExperiments-ImageBrowsing - https://phabricator.wikimedia.org/T413085#11496065 (10matthiasmullie) No worries! I think we could've lived without the sprint tags for all those already closed, but thanks for restoring them! :) [14:07:17] (03CR) 10Hashar: [C:03+2] utils: small script to show deps/reverse deps [integration/config] - 10https://gerrit.wikimedia.org/r/1215555 (owner: 10Hashar) [14:08:20] (03PS2) 10Hashar: Zuul: [mediawiki/extensions/VueTest] mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1223248 (https://phabricator.wikimedia.org/T357475) (owner: 10Kimberly Sarabia) [14:08:22] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/VueTest] mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1223248 (https://phabricator.wikimedia.org/T357475) (owner: 10Kimberly Sarabia) [14:08:53] (03Merged) 10jenkins-bot: utils: small script to show deps/reverse deps [integration/config] - 10https://gerrit.wikimedia.org/r/1215555 (owner: 10Hashar) [14:08:58] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/VueTest] mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1223248 (https://phabricator.wikimedia.org/T357475) (owner: 10Kimberly Sarabia) [14:09:00] (03CR) 10Hashar: [C:03+2] Remove archived repos from list of test exemptions [integration/config] - 10https://gerrit.wikimedia.org/r/1223249 (owner: 10Pppery) [14:09:53] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/VueTest] mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1223248 (https://phabricator.wikimedia.org/T357475) (owner: 10Kimberly Sarabia) [14:22:02] (03update) 10aklapper: PHPMailer: Remove use of class.phpmailer-lite.php [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/89 [14:23:07] (03CR) 10Hashar: [C:03+2] Remove archived repos from list of test exemptions [integration/config] - 10https://gerrit.wikimedia.org/r/1223249 (owner: 10Pppery) [14:23:57] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/FlaggedRevs] Add UserMerge phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1223277 (owner: 10Fomafix) [14:24:31] (03Merged) 10jenkins-bot: Remove archived repos from list of test exemptions [integration/config] - 10https://gerrit.wikimedia.org/r/1223249 (owner: 10Pppery) [14:25:28] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/FlaggedRevs] Add UserMerge phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1223277 (owner: 10Fomafix) [14:30:40] (03CR) 10Hashar: [C:03+2] "I have deleted the `operations-puppet-catalog-compiler-puppet5-test` job." [integration/config] - 10https://gerrit.wikimedia.org/r/1223192 (owner: 10Majavah) [14:32:05] (03PS2) 10Hashar: Reapply "jjb: Remove Puppet 5 PCC jobs" [integration/config] - 10https://gerrit.wikimedia.org/r/1223192 (owner: 10Majavah) [14:32:33] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-mx03 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [14:33:01] (03CR) 10Hashar: [C:03+2] "I ran `./utils/jjb-diff.sh` to generate the list of affected jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/1223192 (owner: 10Majavah) [14:34:59] (03Merged) 10jenkins-bot: Reapply "jjb: Remove Puppet 5 PCC jobs" [integration/config] - 10https://gerrit.wikimedia.org/r/1223192 (owner: 10Majavah) [14:40:01] 06Project-Admins: Create project tag for English WP:RSP Restructuring - https://phabricator.wikimedia.org/T413709#11496194 (10Audiodude) Great idea! Yes I'll migrate the lone issue once we have a phab tag (or I can do it now and open a task, but I don't want it to be confusing and mis-triaged if we don't have th... [15:17:38] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-mx03 in project deployment-prep - https://phabricator.wikimedia.org/T413885 (10wmcs-alerts) 03NEW [15:33:01] (03update) 10brennen: update submodules for 2026-01-06 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/92 (https://phabricator.wikimedia.org/T413832) [15:41:14] (03merge) 10brennen: update submodules for 2026-01-06 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/92 (https://phabricator.wikimedia.org/T413832) [15:43:25] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: Buildkit v0.26.3 released - https://phabricator.wikimedia.org/T412869#11496359 (10dancy) 05In progress→03Resolved [15:44:40] 10Gerrit, 10MediaWiki-Releasing, 10MediaWiki-General, 06MW-Interfaces-Team: Consider deleting the `list` Git tag from the mediawiki/core repository - https://phabricator.wikimedia.org/T413213#11496364 (10Atieno) p:05Triage→03Medium a:03Atieno [15:45:02] 10Gerrit, 10MediaWiki-Releasing, 10MediaWiki-General, 06MW-Interfaces-Team: Consider deleting the `list` Git tag from the mediawiki/core repository - https://phabricator.wikimedia.org/T413213#11496366 (10Atieno) Checking on this [15:45:31] 10Gerrit, 10MediaWiki-Releasing, 10MediaWiki-General, 06MW-Interfaces-Team (MWI-Sprint-24 (2025-12-02 to 2025-12-24)): Consider deleting the `list` Git tag from the mediawiki/core repository - https://phabricator.wikimedia.org/T413213#11496368 (10Atieno) [15:56:01] (03PS4) 10A smart kitten: Zuul: Add a few more experimental PHP 8.5 jobs to templates and repos [integration/config] - 10https://gerrit.wikimedia.org/r/1223189 (https://phabricator.wikimedia.org/T411814) (owner: 10Reedy) [15:56:06] (03CR) 10Reedy: [C:03+2] "ship it" [integration/config] - 10https://gerrit.wikimedia.org/r/1223189 (https://phabricator.wikimedia.org/T411814) (owner: 10Reedy) [15:57:30] (03Merged) 10jenkins-bot: Zuul: Add a few more experimental PHP 8.5 jobs to templates and repos [integration/config] - 10https://gerrit.wikimedia.org/r/1223189 (https://phabricator.wikimedia.org/T411814) (owner: 10Reedy) [16:04:40] 10Phabricator (phabricator-next): Deploy user style to reduce bot comments on Phabricator - https://phabricator.wikimedia.org/T393289#11496432 (10Aklapper) [16:29:15] (03approved) 10jforrester: releases: Bump mediawiki-phan-config to 0.18.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/104 (owner: 10matmarex) [16:29:50] (03merge) 10jforrester: releases: Bump mediawiki-phan-config to 0.18.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/104 (owner: 10matmarex) [16:38:45] 10Gerrit, 10MediaWiki-Releasing, 10MediaWiki-General, 06MW-Interfaces-Team (MWI-Sprint-24 (2025-12-02 to 2025-12-24)): Consider deleting the `list` Git tag from the mediawiki/core repository - https://phabricator.wikimedia.org/T413213#11496655 (10Atieno) Tagging @Krinkle @thcipriani @Arlolra @msantos I hav... [16:51:41] 10Continuous-Integration-Config, 10Parsoid, 10SonarQube Bot, 07ci-test-error, 07Regression: Parsoid patches are no longer being checked by SonarQube - https://phabricator.wikimedia.org/T413626#11496833 (10A_smart_kitten) [[https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/888015/4#message-240... [16:51:41] (03PS1) 10Reedy: Zuul: Add experimental PHP 8.5 jobs to skin templates [integration/config] - 10https://gerrit.wikimedia.org/r/1223690 (https://phabricator.wikimedia.org/T411814) [16:54:10] (03CR) 10Jforrester: [C:03+1] "Oops." [integration/config] - 10https://gerrit.wikimedia.org/r/1223690 (https://phabricator.wikimedia.org/T411814) (owner: 10Reedy) [16:54:15] (03CR) 10Reedy: [C:03+2] Zuul: Add experimental PHP 8.5 jobs to skin templates [integration/config] - 10https://gerrit.wikimedia.org/r/1223690 (https://phabricator.wikimedia.org/T411814) (owner: 10Reedy) [16:56:02] (03Merged) 10jenkins-bot: Zuul: Add experimental PHP 8.5 jobs to skin templates [integration/config] - 10https://gerrit.wikimedia.org/r/1223690 (https://phabricator.wikimedia.org/T411814) (owner: 10Reedy) [16:57:14] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1223690 T411814 [16:57:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:57:16] T411814: Make PHP 8.5 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) in CI - https://phabricator.wikimedia.org/T411814 [16:57:33] 10Continuous-Integration-Config, 10Parsoid, 10SonarQube Bot, 07ci-test-error, 07Regression: Parsoid patches are no longer being checked by SonarQube - https://phabricator.wikimedia.org/T413626#11496898 (10pwangai) @hashar It works, thank you. As for [[ https://phabricator.wikimedia.org/tag/sonarqube_bot... [16:59:02] 10Continuous-Integration-Config, 10Parsoid, 10SonarQube Bot, 06Test-Platform, and 2 others: Parsoid patches are no longer being checked by SonarQube - https://phabricator.wikimedia.org/T413626#11496919 (10Jdforrester-WMF) 05Open→03Resolved [17:06:15] is there an issue with gerrit? I'm trying to push, but it hangs for minutes. [17:11:20] Err, sorry - gitlab, not gerrit! [17:12:04] (03open) 10pppery: Base64 encode the expand icon in the custom timeline CSS [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/109 [17:12:36] It dies with "broken pipe" after about 5 minutes. Debug output here: https://phabricator.wikimedia.org/P86769 [17:13:18] duesen: It's working for me — are you pushing a very large patch or something unusual? [17:14:15] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Projects-Cleanup, 06Reader Growth Team: Remove ArticleSummaries from doc.wikimedia.org/cover-extensions - https://phabricator.wikimedia.org/T413232#11497052 (10Pppery) [17:20:05] James_F: no, small change [17:21:08] git fetch has the same problem [17:22:16] actually, trying to ssh into a deployment server also fails, with a mystery error. I had a similar issue before, which was resoled after a reboot. I was suspecting cosmic rays. But now it'S back, and a reboot did not fix it... [17:22:52] It feels like I'm being filtered/blocked by some heuristic. [17:23:45] (03update) 10pppery: Base64 encode the expand icon in the custom timeline CSS [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/109 [17:24:00] (03update) 10pppery: Base64 encode the expand icon in the custom timeline CSS [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/109 [17:27:44] going through a vpn doesn't fix it. [17:39:32] 10Phabricator (2026-01-06): Deploy user style to reduce bot comments on Phabricator - https://phabricator.wikimedia.org/T393289#11497255 (10Pppery) Follow-up at https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/109 [17:40:57] duesen: My only thought is maybe if your bastion is mis-configured? [17:44:41] (03PS6) 10Jdlrobson: Remove false positives of slightly different (correctly) sized images in Parsoid [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1214173 (https://phabricator.wikimedia.org/T412169) [17:47:30] James_F: but that doesn't explain why the problem is intermediate. It worked this morning. Then it was broken for a few hours. Now it's working again. [17:55:57] Most odd, yes. [17:59:22] clearly sunspots to blame [18:43:59] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T412774#11497496 (10bd808) 05Open→03Invalid Closing as invalid due to self-correcting nature. I will try to keep this incident in mind as I triage beta cluster things for... [19:06:34] 10Gerrit, 10bacula, 06collaboration-services: Backup freshness: Stale bacula backup for gerrit1003 - https://phabricator.wikimedia.org/T413853#11497523 (10Dzahn) Yes, they are at 99% usage. But the mounted object storage is so large that even that 1% left is still in the terabytes. I noticed when I do "stat... [19:12:38] 10Beta-Cluster-Infrastructure, 06collaboration-services, 13Patch-For-Review: Puppet failure: "Unable to locate package spamd" on deployment-mx03.deployment-prep - https://phabricator.wikimedia.org/T412975#11497557 (10bd808) a:03thcipriani Assigning to Tyler after checking to see that he would be ok with it. [19:15:07] 10Beta-Cluster-Infrastructure, 06collaboration-services, 13Patch-For-Review: Replace deployment-mx03 with a bullseye-based instance (was Puppet failure: "Unable to locate package spamd") - https://phabricator.wikimedia.org/T412975#11497582 (10bd808) [19:17:00] 10Beta-Cluster-Infrastructure, 06collaboration-services, 13Patch-For-Review: Replace deployment-mx03 with a bookworm-based instance (was Puppet failure: "Unable to locate package spamd") - https://phabricator.wikimedia.org/T412975#11497587 (10bd808) [19:17:21] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Priority Backlog 📥), 10Cloud-VPS (Debian Bullseye Deprecation), 07Epic: Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#11497588 (10bd808) [19:17:22] 10Beta-Cluster-Infrastructure, 06collaboration-services, 13Patch-For-Review: Replace deployment-mx03 with a bookworm-based instance (was Puppet failure: "Unable to locate package spamd") - https://phabricator.wikimedia.org/T412975#11497589 (10bd808) [19:32:20] ...now it'S broken agin. This is starting to really get in the way, wtf. [19:52:22] 06Project-Admins: Create project tag for Tool-wikicontest - https://phabricator.wikimedia.org/T413628#11497709 (10Agamyasamuel) >>! In T413628#11493640, @bd808 wrote: > The "tool-wikicontest" tag should go with the https://toolsadmin.wikimedia.org/tools/id/wikicontest tool account. This pseudo namespacing has be... [20:15:19] 10GitLab (CI & Job Runners), 06Release-Engineering-Team (Priority Backlog 📥), 06cloud-services-team: Recent incidents of buildkitd's storage volume filling up - https://phabricator.wikimedia.org/T395097#11497749 (10dancy) 05In progress→03Resolved @Andrew has since changed where he performing the buil... [20:51:18] duesen: You might learn more about what is going on if you set some envvars to enable git debugging. There are a number of different things available in the docs at https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables#Debugging [20:52:44] bd808: ah, right, thank you. It took me a while to realize that the issue sees be with git, not ssh. Though I was having an issue with sh as well, and I still think they are related. [20:53:14] I cleaned up my .ssh/config, and now things seem to be working. Could be a coincidence, though... [20:53:33] `GIT_SSH_COMMAND="ssh -vvv"` can be attempted for git over ssh debugging [21:14:43] bd808: yea, that's the one I found, but it didn't tell me what went wrong. The output is here: https://phabricator.wikimedia.org/P86769 [21:15:16] it just hangs at some point, and then times out. [21:17:03] That looks like the ssh protocol was probably fine and the thing that failed was the `debug1: Sending command: git-receive-pack 'repos/mediawiki/services/redioscope.git'` command to the git server. [22:44:31] 06Release-Engineering-Team: Disable Gerrit account for joelyrookewmde and remove from the Github org - https://phabricator.wikimedia.org/T412508#11498165 (10bd808) I would think that revoking Gerrit access could be as simple as locking the Developer account via https://idm.wikimedia.org/wikimedia/block/. That wo... [23:21:12] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work: Set up zuul web on zuul1001/zuul2001 - https://phabricator.wikimedia.org/T405119#11498203 (10thcipriani) >>! In T405119#11313695, @Dzahn wrote: > @dduvall Thanks for looking! Sorry, I should have made this mor... [23:40:22] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work: Set up zuul web on zuul1001/zuul2001 - https://phabricator.wikimedia.org/T405119#11498248 (10thcipriani) >>! In T405119#11498203, @thcipriani wrote: > ...Unsure I can investigate much further without knowing t...