[01:05:07] maintenance-disconnect-full-disks build 654773 integration-agent-docker-1046 (/: 27%, /srv: 98%, /var/lib/docker: 35%): OFFLINE due to disk space [01:10:03] maintenance-disconnect-full-disks build 654774 integration-agent-docker-1046 (/: 27%, /srv: 68%, /var/lib/docker: 35%): RECOVERY disk space OK [05:14:04] 06Project-Admins, 10MW-1.44-notes (1.44.0-wmf.12; 2025-01-14), 10Web-Team ( FY2024-25 Q2 Sprint 5 (Dec 2 2024)): Piko - https://phabricator.wikimedia.org/T381362 (10Matkospajdr) 03NEW [05:16:04] 06Project-Admins, 10MW-1.44-notes (1.44.0-wmf.12; 2025-01-14), 10Web-Team ( FY2024-25 Q2 Sprint 5 (Dec 2 2024)): Piko - https://phabricator.wikimedia.org/T381362#10374511 (10Matkospajdr) 05Open→03Invalid [07:35:27] 10Release-Engineering-Team (Priority Backlog 📥): Delete wmf branches from Gerrit repositories - https://phabricator.wikimedia.org/T303828#10374619 (10hashar) 05Resolved→03Open With 1.44.0-wmf.6, the timer ran `scap clean --delete-gerrit-branch auto` and that pruned the old branch: ` mediawiki/core$ git remot... [07:37:02] 10Release-Engineering-Team (Priority Backlog 📥): Delete wmf branches from Gerrit repositories - https://phabricator.wikimedia.org/T303828#10374624 (10hashar) The log is sent by email and shows lot of: `counterexample fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address ` `... [08:00:03] 10Release-Engineering-Team (Priority Backlog 📥): Delete wmf branches from Gerrit repositories - https://phabricator.wikimedia.org/T303828#10374660 (10hashar) The issue is for extensions and skins, the remote push url is the https one. git/curl would thus lookup for the username in a netrc which does not exist an... [08:06:00] 10Release-Engineering-Team (Priority Backlog 📥): Delete wmf branches from Gerrit repositories - https://phabricator.wikimedia.org/T303828#10374664 (10hashar) `scap prep` does set the push url for the repositories it is cloning (via `_clone_or_update_repo`). Extensions and skins are cloned using `git submodule u... [09:00:23] hashar: morn! should I, or do you want to? :) [09:44:34] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10374924 [09:44:38] 10GitLab (Pipeline Services Migration🐤), 06collaboration-services, 10Wikidata, 10Wikidata Query UI, and 3 others: move commons-query.wikimedia.org and query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10374926 (10ItamarWMDE) I'd be happy to join and observe, if possible. [10:07:15] (03merge) 10aklapper: Fix parsing of incoming mail with UTF-8 encoded headers [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/79 [10:08:33] (03update) 10aklapper: Reports: Order Over-due Tasks by oldest date [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/77 (https://phabricator.wikimedia.org/T380907) [10:09:25] (03merge) 10aklapper: Reports: Order Over-due Tasks by oldest date [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/77 (https://phabricator.wikimedia.org/T380907) [10:13:14] (03update) 10aklapper: Add temporary debug output for T347483 [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/78 (https://phabricator.wikimedia.org/T347483) [10:13:32] (03merge) 10aklapper: Task list view: Do not render tag for Deadline tasks without due date [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/42 (https://phabricator.wikimedia.org/T379376) [10:14:18] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Priority Backlog 📥): Reports: Order Over-due Tasks by oldest date instead of task ID - https://phabricator.wikimedia.org/T380907#10374999 (10Aklapper) [10:14:34] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Priority Backlog 📥), 10Wikimedia-Phabricator-Extensions: Weird indentation on task list (due to empty rendering of deadline subtype tag) - https://phabricator.wikimedia.org/T379376#10375000 (10Aklapper) [10:16:22] andre: sorry I have missed you ping, looks like Jeena is runing the train [10:16:34] hashar: uh, alright! :) [10:17:13] according to https://phabricator.wikimedia.org/T375665 [10:17:17] though I am added as a backup! [10:17:26] (03merge) 10aklapper: Add temporary debug output for T347483 [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/78 (https://phabricator.wikimedia.org/T347483) [10:56:19] 10Phabricator: Phabricator 2FA reset for Atieno Njira - https://phabricator.wikimedia.org/T381306#10375110 (10Aklapper) In general, see "I need access or permissions to..." on https://phabricator.wikimedia.org listing "reset my Multi-Factor Authentication (2FA)". :) Ideally this would be a video call with a thi... [11:27:46] (03update) 10jnuche: install-world: rework installer [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/601 (https://phabricator.wikimedia.org/T380772) [11:29:22] (03update) 10jnuche: install-world: rework installer [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/601 (https://phabricator.wikimedia.org/T380772) [11:32:51] (03update) 10jnuche: install-world: rework installer [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/601 (https://phabricator.wikimedia.org/T380772) [11:33:46] (03merge) 10jnuche: install-world: rework installer [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/601 (https://phabricator.wikimedia.org/T380772) [12:04:30] (03PS1) 10Xqt: jjb/zuul: use Python 3.8 instead of 3.7 for deeptest [integration/config] - 10https://gerrit.wikimedia.org/r/1100092 (https://phabricator.wikimedia.org/T378893) [12:55:45] (03open) 10hashar: Clear stdin before reading user input (take 2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/606 [13:00:03] maintenance-disconnect-full-disks build 654916 integration-agent-docker-1046 (/: 27%, /srv: 99%, /var/lib/docker: 36%): OFFLINE due to disk space [13:05:03] maintenance-disconnect-full-disks build 654917 integration-agent-docker-1046 (/: 27%, /srv: 93%, /var/lib/docker: 36%): RECOVERY disk space OK [13:11:04] 10Scap: scap installer: allow for non-deployment server installs - https://phabricator.wikimedia.org/T380772#10375582 (10jnuche) 05Open→03Resolved Changes deployed to prod as Scap `4.132.0` [13:29:11] (03update) 10jnuche: Clear stdin before reading user input (take 2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/606 (owner: 10hashar) [13:29:38] (03update) 10jnuche: Clear stdin before reading user input (take 2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/606 (owner: 10hashar) [13:33:00] arguably my most common issue with computing: `ModuleNotFoundError: No module named 'unitest'` [13:37:59] i'm having some failures on `integration-agent-docker-1046` with `no space left on device`, can someone take a look and free some? [13:40:13] most recent failure on 1046 is https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-selenium/57533/, which ran at :08, but still failed on lack of space [13:40:21] so the recovery from above doesn't seem to be full [14:21:27] 10Phabricator: Phabricator 2FA reset for Atieno Njira - https://phabricator.wikimedia.org/T381306#10375837 (10Aklapper) 05Open→03Resolved a:03Aklapper Sorted out and reset. :) [14:33:40] urbanecm: I was looking at it and should have !log [14:34:48] !log Cleared workspaces from integration-agent-docker-1046 [14:34:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:34:59] 6297 wmf-quibble-selenium-php81@2/src [14:35:02] so yeah too big [14:35:16] it is a build that was left behind since April 16th... [14:35:26] so I bet that host has been alarming a bit more frequently than the others [14:35:45] urbanecm: should be good now. Thanks for the ping about integration-agent-docker-1046 [14:35:49] 10Phabricator: Phabricator 2FA reset for Atieno Njira - https://phabricator.wikimedia.org/T381306#10375886 (10MBywater-WMF) Thanks @Aklapper! [14:58:37] of course sys.stdin.buffer is read-only ... [15:12:41] ty hashar! [15:23:04] 10GitLab (Pipeline Services Migration🐤), 06collaboration-services, 10Wikidata, 10Wikidata Query UI, and 2 others: move commons-query.wikimedia.org and query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10376055 (10Jelto) The migration of `query-scholarly.wikidata.org` to Wikikube... [15:23:40] 10GitLab (Pipeline Services Migration🐤), 06collaboration-services, 10Wikidata, 10Wikidata Query UI, and 2 others: move commons-query.wikimedia.org and query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10376057 (10Jelto) [15:28:33] (03update) 10hashar: Clear stdin before reading user input (take 2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/606 [15:46:58] 10Gerrit: Access issue for golson-wmf - https://phabricator.wikimedia.org/T378187#10376149 (10GOlson-WMF) {F57775370} I was able to add a second email, but when I go to delete the existing one, it pops up this error. [16:00:19] 10Gerrit: Add support for Vue language in Gerrit code editor - https://phabricator.wikimedia.org/T380668#10376329 (10hashar) [16:03:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10376336 [16:06:58] 10Gerrit (Gerrit 3.11): Add support for Vue language in Gerrit code editor - https://phabricator.wikimedia.org/T380668#10376361 (10hashar) Looks like @Paladox went to add support for Vue: * https://gerrit-review.googlesource.com/c/plugins/codemirror-editor/+/438682 //Added Vue language support// * https://gerri... [16:13:23] (03update) 10dancy: install-world: rework installer [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/601 (https://phabricator.wikimedia.org/T380772) (owner: 10jnuche) [16:14:31] 10MediaWiki-Releasing, 05MW-1.43-release, 13Patch-For-Review: Release MW 1.43.0-rc.0 - https://phabricator.wikimedia.org/T372320#10376402 (10MSantos) a:05ABreault-WMF→03Atieno [16:27:30] 10GitLab (CI & Job Runners), 07Documentation: Prominent docs on GitLab CI are missing information on runner options and best practices - https://phabricator.wikimedia.org/T381410 (10bd808) 03NEW [16:33:11] 10GitLab (CI & Job Runners), 07Documentation: Prominent docs on GitLab CI are missing information on runner options and best practices - https://phabricator.wikimedia.org/T381410#10376497 (10bd808) Some docs with useful information that ideally would be easier to find: * https://wikitech.wikimedia.org/wiki/Git... [16:33:41] (03update) 10hashar: Clear stdin before reading user input (take 2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/606 [17:14:20] (03merge) 10dancy: Clear stdin before reading user input (take 2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/606 (owner: 10hashar) [17:15:04] 10GitLab (CI & Job Runners), 06cloud-services-team, 10Cloud-VPS, 06QTE-TestingOverview: Future testing-infra growth on cloud-vps - https://phabricator.wikimedia.org/T381419 (10Andrew) 03NEW [17:18:33] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS: Future growth of deployment-prep? - https://phabricator.wikimedia.org/T381420 (10Andrew) 03NEW [17:20:40] (03open) 10hashar: scap prep: set pushInsteadOf in the submodules config [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/607 (https://phabricator.wikimedia.org/T303828 https://phabricator.wikimedia.org/T306425) [17:22:58] 10GitLab (CI & Job Runners), 06cloud-services-team, 10Cloud-VPS, 06QTE-TestingOverview: Future testing-infra growth on cloud-vps - https://phabricator.wikimedia.org/T381419#10376765 (10bd808) A couple of months ago I grabbed a larger quota for the Integration project with {T376847}. I have not yet built ou... [17:23:29] (03CR) 10Hashar: scap prep: set ssh push url on all repositories (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/805373 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [17:24:51] (03update) 10hashar: scap prep: set pushInsteadOf in the submodules config [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/607 (https://phabricator.wikimedia.org/T303828 https://phabricator.wikimedia.org/T306425) [17:28:01] 10Continuous-Integration-Infrastructure, 10GitLab (CI & Job Runners), 06cloud-services-team, 10Cloud-VPS, 06QTE-TestingOverview: Future testing-infra growth on cloud-vps - https://phabricator.wikimedia.org/T381419#10376792 (10hashar) [17:36:59] (03CR) 10Hashar: [C:03+2] "INFO:jenkins_jobs.builder:Creating jenkins job pywikibot-core-tox-deeptest-py38" [integration/config] - 10https://gerrit.wikimedia.org/r/1100092 (https://phabricator.wikimedia.org/T378893) (owner: 10Xqt) [17:38:08] 10Continuous-Integration-Infrastructure, 10GitLab (CI & Job Runners), 06cloud-services-team, 10Cloud-VPS, 06QTE-TestingOverview: Future testing-infra growth on cloud-vps - https://phabricator.wikimedia.org/T381419#10376813 (10hashar) Some time ago, @aborrero offered to have the Jenkins instances (`integr... [17:38:43] (03Merged) 10jenkins-bot: jjb/zuul: use Python 3.8 instead of 3.7 for deeptest [integration/config] - 10https://gerrit.wikimedia.org/r/1100092 (https://phabricator.wikimedia.org/T378893) (owner: 10Xqt) [17:46:28] 10GitLab (CI & Job Runners), 06collaboration-services, 06Security-Team, 07SecTeam-Processed, and 2 others: Findings in Security Readiness Reviews of Trusted GitLab Runners - https://phabricator.wikimedia.org/T317341#10376847 (10sbassett) [17:51:32] 10Continuous-Integration-Infrastructure, 10GitLab (CI & Job Runners), 06cloud-services-team, 10Cloud-VPS, 06QTE-TestingOverview: Future testing-infra growth on cloud-vps - https://phabricator.wikimedia.org/T381419#10376872 (10Andrew) >>! In T381419#10376812, @hashar wrote: > > We have 18 integration-age... [17:53:14] (03PS1) 10Thcipriani: Add a banner for the 2024 developer survey [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1100162 (https://phabricator.wikimedia.org/T351109) [17:53:45] (03CR) 10CI reject: [V:04-1] Add a banner for the 2024 developer survey [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1100162 (https://phabricator.wikimedia.org/T351109) (owner: 10Thcipriani) [17:54:14] 10Continuous-Integration-Infrastructure, 10GitLab (CI & Job Runners), 06cloud-services-team, 10Cloud-VPS, and 2 others: Future testing-infra growth on cloud-vps - https://phabricator.wikimedia.org/T381419#10376883 (10Dzahn) [17:54:52] (03PS2) 10Thcipriani: Add a banner for the 2024 developer survey [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1100162 (https://phabricator.wikimedia.org/T351109) [17:55:20] (03CR) 10CI reject: [V:04-1] Add a banner for the 2024 developer survey [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1100162 (https://phabricator.wikimedia.org/T351109) (owner: 10Thcipriani) [17:56:31] (03CR) 10Thcipriani: [C:04-1] "Not deployable until survey launch (2024-12-05)" [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1100162 (https://phabricator.wikimedia.org/T351109) (owner: 10Thcipriani) [17:58:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10376906 [17:58:33] (03PS1) 10Hashar: Reinstate the banner for the developer survey [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1100163 [18:55:51] 06Release-Engineering-Team, 06Security-Team: Grant Access to JLy (Jimmy Lee) to /repos/security under gitlab.wikimedia.org - https://phabricator.wikimedia.org/T381431 (10sbassett) 03NEW [18:56:30] 06Release-Engineering-Team, 06Security-Team: Grant Access to JLy (Jimmy Lee) to /repos/security under gitlab.wikimedia.org - https://phabricator.wikimedia.org/T381431#10377237 (10sbassett) [18:57:08] 06Release-Engineering-Team, 06Security-Team: Grant Access to /repos/security under gitlab.wikimedia.org for JLy (Jimmy Ly) - https://phabricator.wikimedia.org/T381431#10377254 (10sbassett) [18:58:24] 06Release-Engineering-Team, 06Security-Team: Grant Access to /repos/security under gitlab.wikimedia.org for JLy (Jimmy Ly) - https://phabricator.wikimedia.org/T381431#10377256 (10taavi) you should be able to self-serve this here: https://gitlab.wikimedia.org/groups/repos/security/-/group_members [19:00:53] 06Release-Engineering-Team, 06Security-Team: Grant Access to /repos/security under gitlab.wikimedia.org for JLy (Jimmy Ly) - https://phabricator.wikimedia.org/T381431#10377281 (10sbassett) >>! In T381431#10377256, @taavi wrote: > you should be able to self-serve this here: https://gitlab.wikimedia.org/grou... [19:02:04] 06Release-Engineering-Team, 06Security-Team: Grant Access to /repos/security under gitlab.wikimedia.org for JLy (Jimmy Ly) - https://phabricator.wikimedia.org/T381431#10377283 (10sbassett) 05Open→03Resolved p:05Triage→03Low a:03sbassett [19:10:02] 10MediaWiki-Releasing, 05MW-1.43-release, 13Patch-For-Review: Release MW 1.43.0-rc.0 - https://phabricator.wikimedia.org/T372320#10377319 (10Atieno) This change is based on a reverted change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1099034 . Reverted because Arlo hasn't signed the L3 agreement.... [19:12:03] (03update) 10hashar: scap prep: set pushInsteadOf in the submodules config [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/607 (https://phabricator.wikimedia.org/T303828 https://phabricator.wikimedia.org/T306425) [19:19:43] (03CR) 10Xqt: "thank you @hasharfor this quick submit" [integration/config] - 10https://gerrit.wikimedia.org/r/1100092 (https://phabricator.wikimedia.org/T378893) (owner: 10Xqt) [19:25:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10377366 [19:29:06] (03update) 10hashar: scap prep: set pushInsteadOf in the submodules config [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/607 (https://phabricator.wikimedia.org/T303828 https://phabricator.wikimedia.org/T306425) [19:37:41] (03CR) 10Hashar: [C:03+2] "You are welcome, and congratulations on bringing it in to Python 3.8+!" [integration/config] - 10https://gerrit.wikimedia.org/r/1100092 (https://phabricator.wikimedia.org/T378893) (owner: 10Xqt) [19:37:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10377432 [19:44:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10377455 [20:17:40] (03update) 10dancy: scap prep: set pushInsteadOf in the submodules config [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/607 (https://phabricator.wikimedia.org/T303828 https://phabricator.wikimedia.org/T306425) (owner: 10hashar) [20:18:07] (03merge) 10dancy: scap prep: set pushInsteadOf in the submodules config [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/607 (https://phabricator.wikimedia.org/T303828 https://phabricator.wikimedia.org/T306425) (owner: 10hashar) [20:33:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10377592 [20:49:34] dancy: jnuche: I could use some help troubleshooting scap issues I'm having on `wdqs1027`. Puppet run gave the following errors https://phabricator.wikimedia.org/P71504 which ultimately leads to a `Error: /Stage[main]/Query_service::Deploy::Scap/Scap::Target[wdqs/wdqs]/Package[wdqs/wdqs]: Provider scap3 is not functional on this host` [20:50:25] The wheel errors shown in that phab paste make me think there might be an issue with https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/601 [20:51:30] I'm not sure if it's the direct cause or just downstream of the real cause but on the broken host `wdqs1027` the wheels are in `/var/lib/scap/bullseye/4.132.0/` whereas comparing to a known-good host `wdqs2018` the files are in `/var/lib/scap/bullseye/` [20:52:12] known good host is on `4.123.0` compared to `4.132.0` on the broken host btw [21:03:03] * dancy reads [21:04:00] ryankemper: Taking a look [21:04:11] ty! [21:11:15] i haven't untangled things yet but I wonder if `ON_MASTER` is somehow evaluating to true because somehow we're getting the format of `$BASE_DIST_DIR/$DISTRO/$TAG` which matches what `get_dist_dir_path_master` is going to return as opposed to `get_dist_dir_path_target` which would return `$BASE_DIST_DIR/$DISTRO` [21:11:40] fwiw `wdqs1027` is not in production or anything so if you need to hack around on it to test stuff out feel free [21:11:53] Awesome. Thank you. [21:11:55] s/ON_MASTER/on_master [21:13:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10377717 [21:19:29] 10Scap: scap installer: allow for non-deployment server installs - https://phabricator.wikimedia.org/T380772#10377726 (10dancy) ryankemper reported a puppet bootstrapping problem on wdqs1027.eqiad.wmnet: P71504. Looks like `operations/puppet/modules/scap/files/bootstrap-scap-target.sh` needs to be updated to ma... [21:19:39] 10Scap: scap installer: allow for non-deployment server installs - https://phabricator.wikimedia.org/T380772#10377727 (10dancy) 05Resolved→03Open p:05Triage→03Medium [21:50:31] ryankemper: jnuche should be able to pick up this thread when he starts working again. [22:04:17] dancy: ack, thanks. is there any kind of simple hack we can do to unblock these hosts? We’re trying to do an lvs deploy in ~18 hours but need these hosts scap deployable for that [22:13:57] hmm. lemme see if I can think of something [22:20:25] I'll see if I can hack `/var/lib/scap/install_local_version.sh` and invoke it manually [22:22:20] I don't think that will satisfy Puppet. [22:23:32] Scap is properly installed in /var/lib/scap/bin/scap, but the /usr/bin/scap symlink hasn't been set up because bootstrap-scap-target.sh has not successfully run yet [22:36:09] Okay, hacking `/var/lib/scap/install_local_version.sh`like so let us manually invoke `sudo /var/lib/scap/install_local_version.sh -u scap`: [22:36:11] https://www.irccloud.com/pastebin/6xhYulMt/ [22:39:06] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1100204 is my proposal [22:42:55] Just got successful scap deploys to `wdqs102[6,7]` so we should be unblocked here now. thanks for your help! [22:43:09] Ok good! [23:13:38] (03open) 10lwatson: Draft: web: improve JobViewerPage UI [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/608 (https://phabricator.wikimedia.org/T379413)