[00:20:57] maintenance-disconnect-full-disks build 504725 integration-agent-docker-1026 (/: 29%, /srv: 12%, /var/lib/docker: 99%): still OFFLINE due to disk space [00:20:57] maintenance-disconnect-full-disks build 504725 integration-agent-docker-1028 (/: 29%, /srv: 19%, /var/lib/docker: 95%): still OFFLINE due to disk space [00:20:58] maintenance-disconnect-full-disks build 504725 integration-agent-docker-1029 (/: 29%, /srv: 16%, /var/lib/docker: 99%): still OFFLINE due to disk space [00:20:58] maintenance-disconnect-full-disks build 504725 integration-agent-docker-1030 (/: 30%, /srv: 33%, /var/lib/docker: 96%): still OFFLINE due to disk space [00:20:58] maintenance-disconnect-full-disks build 504725 integration-agent-docker-1031 (/: 29%, /srv: 19%, /var/lib/docker: 95%): still OFFLINE due to disk space [00:20:59] maintenance-disconnect-full-disks build 504725 integration-agent-docker-1033 (/: 29%, /srv: 13%, /var/lib/docker: 3%): RECOVERY disk space OK [00:20:59] maintenance-disconnect-full-disks build 504725 integration-agent-docker-1034 (/: 29%, /srv: 15%, /var/lib/docker: 96%): still OFFLINE due to disk space [00:21:00] maintenance-disconnect-full-disks build 504725 integration-agent-docker-1035 (/: 31%, /srv: 15%, /var/lib/docker: 100%): still OFFLINE due to disk space [00:21:00] maintenance-disconnect-full-disks build 504725 integration-agent-docker-1036 (/: 29%, /srv: 18%, /var/lib/docker: 99%): still OFFLINE due to disk space [00:21:01] maintenance-disconnect-full-disks build 504725 integration-agent-docker-1037 (/: 29%, /srv: 15%, /var/lib/docker: 97%): still OFFLINE due to disk space [00:21:01] maintenance-disconnect-full-disks build 504725 integration-agent-docker-1038 (/: 29%, /srv: 14%, /var/lib/docker: 95%): still OFFLINE due to disk space [00:21:05] bah [00:21:45] * thcipriani works through these [00:25:44] maintenance-disconnect-full-disks build 504726 integration-agent-docker-1026 (/: 29%, /srv: 12%, /var/lib/docker: 3%): RECOVERY disk space OK [00:25:44] maintenance-disconnect-full-disks build 504726 integration-agent-docker-1029 (/: 29%, /srv: 16%, /var/lib/docker: 71%): RECOVERY disk space OK [00:25:44] maintenance-disconnect-full-disks build 504726 integration-agent-docker-1030 (/: 30%, /srv: 33%, /var/lib/docker: 60%): RECOVERY disk space OK [00:25:44] maintenance-disconnect-full-disks build 504726 integration-agent-docker-1031 (/: 29%, /srv: 19%, /var/lib/docker: 49%): RECOVERY disk space OK [00:25:45] maintenance-disconnect-full-disks build 504726 integration-agent-docker-1034 (/: 29%, /srv: 15%, /var/lib/docker: 61%): RECOVERY disk space OK [00:25:45] maintenance-disconnect-full-disks build 504726 integration-agent-docker-1035 (/: 31%, /srv: 15%, /var/lib/docker: 67%): RECOVERY disk space OK [00:25:46] maintenance-disconnect-full-disks build 504726 integration-agent-docker-1036 (/: 29%, /srv: 18%, /var/lib/docker: 37%): RECOVERY disk space OK [00:25:46] maintenance-disconnect-full-disks build 504726 integration-agent-docker-1037 (/: 29%, /srv: 15%, /var/lib/docker: 31%): RECOVERY disk space OK [00:25:47] maintenance-disconnect-full-disks build 504726 integration-agent-docker-1038 (/: 29%, /srv: 14%, /var/lib/docker: 63%): RECOVERY disk space OK [00:26:02] !log integration: sudo cumin --force 'name:docker' 'docker buildx prune --force' [00:26:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:30:30] maintenance-disconnect-full-disks build 504727 integration-agent-docker-1028 (/: 29%, /srv: 19%, /var/lib/docker: 3%): RECOVERY disk space OK [03:58:28] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T340243 (10Peachey88) [03:59:48] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T340243 (10Peachey88) 05Resolved→03Open Looks to be a issue with file deletions, deleting more than expected: {T34... [04:52:21] 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟): Upgrade content license of Phabricator from 3.0 to CC BY-SA 4.0 - https://phabricator.wikimedia.org/T338440 (10EpicPupper) 05Open→03Resolved [05:33:27] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.41.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T340244 (10RhinosF1) Previous week train has been reopened [07:35:30] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/ReportIncident] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/934652 [07:35:32] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [extensions/ReportIncident] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/934652 (owner: 10QChris) [07:35:34] (03PS1) 10QChris: Import done. Revoke import grants [extensions/ReportIncident] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/934653 [07:35:36] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [extensions/ReportIncident] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/934653 (owner: 10QChris) [07:51:46] (03PS1) 10QChris: Zuul: Follow IncidentReporting -> ReportIncident extension rename [integration/config] - 10https://gerrit.wikimedia.org/r/934654 (https://phabricator.wikimedia.org/T340189) [08:01:54] thcipriani: I don't get how all those docker agent got disk filed :-\ [08:02:26] at least python `torch` is filing them T338317 [08:02:27] T338317: Python torch fills disk of CI Jenkins instances - https://phabricator.wikimedia.org/T338317 [08:02:50] but I cleaned them last night and I would not expect that build to have run an all instances [08:19:24] 10Phabricator, 10serviceops-collab, 10Developer-Advocacy (Apr-Jun 2023): Automate SQL queries for quarterly Phabricator statistics/metrics for Technical Community Newsletter - https://phabricator.wikimedia.org/T337387 (10Aklapper) Confirming that this worked as expected on 1st July (and [data was copied](htt... [08:20:34] 10GitLab, 10Release-Engineering-Team (Priority Backlog 📥): WMCS GitLab runners running frequently running out of disk space - https://phabricator.wikimedia.org/T340887 (10hashar) There is a similar issue on the `integration` Jenkins agents T338317. I tracked it down to https://gerrit.wikimedia.org/g/machinelea... [08:21:03] 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟): Remove "Prototype" suffix from "Reports" menu item on Project pages - https://phabricator.wikimedia.org/T337876 (10Aklapper) 05Open→03Resolved This is deployed now [08:27:37] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T340243 (10Novem_Linguae) [08:31:56] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T340243 (10hashar) 05Open→03Resolved Remarking as resolved after T340821 got fixed this morning. [08:39:05] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Temporarily replace the Phabricator logo for Pride Month - https://phabricator.wikimedia.org/T337964 (10Aklapper) Not sure how to switch back as it seems I need to upload (?) a "new" file with the previous logo from... somewhere (?) [08:48:35] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Temporarily replace the Phabricator logo for Pride Month - https://phabricator.wikimedia.org/T337964 (10taavi) I uploaded the old logo as the "new" one. Although now the `ui.logo` setting is set both in the database and in [[ https://gerrit.wikimedia... [08:49:51] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Temporarily replace the Phabricator logo for Pride Month - https://phabricator.wikimedia.org/T337964 (10TheresNoTime) [snrk] //I was hoping y'all would forget...// [08:51:16] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Temporarily replace the Phabricator logo for Pride Month - https://phabricator.wikimedia.org/T337964 (10stjn) Hopefully they forget at the rainbow-only logo version next year :-) [13:35:17] 10Beta-Cluster-Infrastructure, 10MediaWiki-File-management: Unable to upload files on Beta Commons - https://phabricator.wikimedia.org/T340908 (10LucasWerkmeister) [13:39:15] 10Beta-Cluster-Infrastructure, 10MediaWiki-File-management: Unable to upload files on Beta Commons - https://phabricator.wikimedia.org/T340908 (10LucasWerkmeister) If I read [LabsServices.php](https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/8a38488adb7a140238ae198c269847f3f94b5e8f/wmf-config/LabsS... [13:43:28] 10Beta-Cluster-Infrastructure, 10MediaWiki-File-management: Unable to upload files on Beta Commons - https://phabricator.wikimedia.org/T340908 (10LucasWerkmeister) Okay, on [beta-logs](https://wikitech.wikimedia.org/wiki/Logstash#Beta_Cluster_Logstash) I see a “Redis exception connecting to "deployment-memc09.... [13:47:29] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-File-management: Unable to upload files on Beta Commons - https://phabricator.wikimedia.org/T340908 (10LucasWerkmeister) But that bot seems to be able to reach memc09 just fine in a manual test: `lang=shell-session lucaswerkmeister@deployment-mediawiki11... [14:19:30] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-File-management: Unable to upload files on Beta Commons - https://phabricator.wikimedia.org/T340908 (10Uata1122) @Uata1122 [14:27:49] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-File-management: Unable to upload files on Beta Commons - https://phabricator.wikimedia.org/T340908 (10LucasWerkmeister) Hang on, I’ve been checking the wrong service. It’s a “**Redis** exception connecting”, coming from `RedisConnectionPool`. And while r... [14:34:36] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-File-management: Unable to upload files on Beta Commons - https://phabricator.wikimedia.org/T340908 (10LucasWerkmeister) If I’m reading the `lsof` output correctly, it should be listening on any host, and I can connect to it via a non-localhost IP, but on... [14:40:41] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-File-management: Unable to upload files on Beta Commons - https://phabricator.wikimedia.org/T340908 (10LucasWerkmeister) Hm, in Horizon I see “ALLOW IPv4 6379/tcp from 172.16.0.0/21” for memc09; mediawiki11 should be 172.16.3.203 according to `ip a`, whic... [14:56:12] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-File-management: Unable to upload files on Beta Commons - https://phabricator.wikimedia.org/T340908 (10LucasWerkmeister) I temporarily installed `mtr-tiny` on mwmaint02, but it didn’t really help. The connection between mwmaint02 and memc09 is direct, no... [15:39:02] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-File-management: Unable to upload files on Beta Commons - https://phabricator.wikimedia.org/T340908 (10LucasWerkmeister) As far as I can tell, all the other ports that are supposed to be allowed by the default security group are also not working: `lang=s... [16:25:44] 10Gerrit: Misspellings at cldr ms-arab - https://phabricator.wikimedia.org/T340912 (10Taufik)