[01:43:37] 10Scap, 13Patch-Needs-Improvement: scap deploy-local should make fewer assumptions about server/directories - https://phabricator.wikimedia.org/T146602#11373138 (10Pppery) [01:47:33] 06Release-Engineering-Team (Seen), 10Release Pipeline, 06serviceops, 06SRE: Kask functional testing with Cassandra via the Deployment Pipeline - https://phabricator.wikimedia.org/T224041#11373148 (10Pppery) [06:08:53] 06Release-Engineering-Team (Seen), 10Scap, 06MW-Interfaces-Team, 06serviceops, and 2 others: Add a jobrunner server to the Scap canary pool - https://phabricator.wikimedia.org/T172480#11373318 (10Krinkle) I assume so, yes. Scap (hopefully) still has the concept of promoting the next image to a canary pool... [06:33:47] 06Release-Engineering-Team, 06collaboration-services, 10mwcli, 06serviceops-radar: Set writeable HOME for nobody user in golang images instead of /nonexistent - https://phabricator.wikimedia.org/T331209#11373347 (10Krinkle) We've ran into a dozen jobs that fail for one reason or another due to the unusual... [06:48:51] 10Beta-Cluster-Infrastructure: Some production sites on the beta cluster were unable to be selected as import targets - https://phabricator.wikimedia.org/T410109 (10SunAfterRain) 03NEW [06:56:57] 10Beta-Cluster-Infrastructure: Some production sites on the beta cluster were unable to be selected as import targets - https://phabricator.wikimedia.org/T410109#11373370 (10Stang) Beta cluster is testing "[[https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/CommonSettings-labs.php#L... [07:32:10] 10Beta-Cluster-Infrastructure: Some production sites on the beta cluster were unable to be selected as import targets - https://phabricator.wikimedia.org/T410109#11373396 (10SunAfterRain) {F70207421} Additional note: Based on the current code, the import source should include all wikis available on the beta clu... [08:06:34] 10GitLab, 06collaboration-services: SystemdUnitFailed - gitlab2002 - backup-restore - https://phabricator.wikimedia.org/T410044#11373424 (10ABran-WMF) 05Openβ†’03Resolved a:03ABran-WMF [09:11:29] 06Release-Engineering-Team (Seen), 10Scap, 06MW-Interfaces-Team, 06serviceops, and 2 others: Add a jobrunner server to the Scap canary pool - https://phabricator.wikimedia.org/T172480#11373510 (10akosiaris) 05Openβ†’03Resolved a:03akosiaris >>! In T172480#11373318, @Krinkle wrote: > I assume Scap s... [09:29:53] 06Release-Engineering-Team (Doing 😎), 10Scap, 07Essential-Work: scap not complaining about dependencies only partially deployed with the train - https://phabricator.wikimedia.org/T397931#11373530 (10Lucas_Werkmeister_WMDE) Well, per the discussion in T362987 it sounds like I’ll continue running into the... [10:20:03] maintenance-disconnect-full-disks build 754511 integration-agent-docker-1050 (/: 36%, /srv: 98%, /var/lib/docker: 33%): OFFLINE due to disk space [10:20:35] (03open) 10taavi: Add make inside the runner container [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/85 [10:20:40] (03update) 10taavi: Add make inside the runner container [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/85 [10:22:50] (03merge) 10taavi: Add make inside the runner container [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/85 [10:25:03] maintenance-disconnect-full-disks build 754512 integration-agent-docker-1050 (/: 36%, /srv: 51%, /var/lib/docker: 33%): RECOVERY disk space OK [10:30:03] maintenance-disconnect-full-disks build 754513 integration-agent-docker-1055 (/: 36%, /srv: 98%, /var/lib/docker: 41%): OFFLINE due to disk space [10:35:03] maintenance-disconnect-full-disks build 754514 integration-agent-docker-1055 (/: 36%, /srv: 12%, /var/lib/docker: 40%): RECOVERY disk space OK [10:50:11] 10Phabricator, 10Phabricator maintenance bot, 10Pywikibot: Consider adding additional Pywikibot-related projects to H6 and/or Pywikibot's Maintenance_bot rule - https://phabricator.wikimedia.org/T410127 (10A_smart_kitten) 03NEW [11:11:06] 10Phabricator (Upstream), 07CSS, 07Upstream: When a workboard has a long sidebar, scrolling down that sidebar scrolls down the entire page (and scrolls away the workboard columns) - https://phabricator.wikimedia.org/T409705#11373908 (10A_smart_kitten) p:05Triageβ†’03Medium a:03A_smart_kitten Uhhh, so it... [11:25:03] maintenance-disconnect-full-disks build 754524 integration-agent-docker-1057 (/: 36%, /srv: 97%, /var/lib/docker: 38%): OFFLINE due to disk space [11:30:03] maintenance-disconnect-full-disks build 754525 integration-agent-docker-1057 (/: 36%, /srv: 31%, /var/lib/docker: 37%): RECOVERY disk space OK [14:33:25] (03update) 10jnuche: make Scap operations resilient to transient failures in GitLab [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1040 (https://phabricator.wikimedia.org/T358876) [14:38:29] (03update) 10jnuche: make Scap operations resilient to transient failures in GitLab [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1040 (https://phabricator.wikimedia.org/T358876) [14:43:04] (03update) 10jnuche: make Scap operations resilient to transient failures in GitLab [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1040 (https://phabricator.wikimedia.org/T358876) [14:49:57] why did I suddenly get logged out of phabricator o_O [15:09:19] (03update) 10jnuche: make Scap operations resilient to transient failures in GitLab [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1040 (https://phabricator.wikimedia.org/T358876) [15:52:35] (03open) 10jnuche: add logo to README [repos/releng/docpub] - 10https://gitlab.wikimedia.org/repos/releng/docpub/-/merge_requests/39 [15:53:20] (03merge) 10jnuche: add logo to README [repos/releng/docpub] - 10https://gitlab.wikimedia.org/repos/releng/docpub/-/merge_requests/39 [15:57:51] (03approved) 10dancy: make Scap operations resilient to transient failures in GitLab [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1040 (https://phabricator.wikimedia.org/T358876) (owner: 10jnuche) [16:20:09] (03merge) 10jnuche: make Scap operations resilient to transient failures in GitLab [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1040 (https://phabricator.wikimedia.org/T358876) [16:24:36] 06Release-Engineering-Team, 10Scap, 13Patch-For-Review: Remove hard dependencies on GitLab, Toolforge And Phorge/Phab - https://phabricator.wikimedia.org/T358876#11374830 (10jnuche) 05Openβ†’03Resolved a:03dancy After the latest MR it should be possible to perform train deployments even if there's an... [16:26:25] 10Scap: Initial group versions displayed by "scap train" can be wrong if previous scap attempt failed - https://phabricator.wikimedia.org/T410018#11374839 (10dancy) > https://versions.toolforge.org/ correctly showed still old `1.46.0-wmf.1` for group2 instead of `1.46.0-wmf.2`. Looks like https://versions.toolf... [16:27:17] 10Scap: Initial group versions displayed by "scap train" can be wrong if previous scap attempt failed - https://phabricator.wikimedia.org/T410018#11374843 (10dancy) > ` my guess is scap thinks that because it reads the local /srv/mediawiki-staging/wikiversions.json` That's right. [16:55:03] maintenance-disconnect-full-disks build 754590 integration-agent-docker-1043 (/: 35%, /srv: 96%, /var/lib/docker: 37%): OFFLINE due to disk space [17:00:03] maintenance-disconnect-full-disks build 754591 integration-agent-docker-1043 (/: 35%, /srv: 17%, /var/lib/docker: 37%): RECOVERY disk space OK [17:05:28] 06Release-Engineering-Team (Radar), 06collaboration-services, 06SRE, 06Traffic, 13Patch-For-Review: Deploy a TCP proxy across all DCs - https://phabricator.wikimedia.org/T408532#11374995 (10Dzahn) A proxy is running on 14 VMs, 2 in each of the 7 POPs. What is missing is the load-balancing part. [17:39:52] 10GitLab, 06collaboration-services, 07LDAP: sync-gitlab-group-with-ldap - failed to create user / duplicate email - https://phabricator.wikimedia.org/T410159 (10Dzahn) 03NEW [17:51:18] (03open) 10rzl: Revert "deployment-calendar: Moritz takes over for Reuven in the Puppet window" [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/217 [18:35:26] 10GitLab, 06collaboration-services, 07LDAP: sync-gitlab-group-with-ldap - failed to create user / duplicate email - https://phabricator.wikimedia.org/T410159#11375230 (10dancy) a:03dancy I'll take this one. [18:58:39] 06Release-Engineering-Team (Doing 😎), 10wikimedia.biterg.io: Automatically export and publish a list of WMF deployed code repositories in Bitergia's JSON format - https://phabricator.wikimedia.org/T385529#11375276 (10Dzahn) @Aklapper Wanna have a quick chat about this as part of T409903? Cheers [18:59:04] 10GitLab, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 10wikimedia.biterg.io: Bitergia crawler bot runs into errors collection GitLab data - https://phabricator.wikimedia.org/T409903#11375282 (10Dzahn) [18:59:08] 06Release-Engineering-Team (Doing 😎), 10wikimedia.biterg.io: Automatically export and publish a list of WMF deployed code repositories in Bitergia's JSON format - https://phabricator.wikimedia.org/T385529#11375283 (10Dzahn) [19:05:24] !log Delete Gitlab monica_wmde account (T410159) [19:05:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:05:27] T410159: sync-gitlab-group-with-ldap - failed to create user / duplicate email - https://phabricator.wikimedia.org/T410159 [19:09:00] 10GitLab, 06collaboration-services, 07LDAP: sync-gitlab-group-with-ldap - failed to create user / duplicate email - https://phabricator.wikimedia.org/T410159#11375291 (10dancy) ` Nov 14 19:06:33 gitlab1004 systemd[1]: Starting sync-gitlab-group-with-ldap.service - Sync various GitLab groups with their LDAP g... [19:11:07] 10GitLab, 06collaboration-services, 07LDAP: sync-gitlab-group-with-ldap - failed to create user / duplicate email - https://phabricator.wikimedia.org/T410159#11375296 (10Dzahn) Thank you! :) looks resolved! [19:11:19] 10GitLab, 06collaboration-services, 07LDAP: sync-gitlab-group-with-ldap - failed to create user / duplicate email - https://phabricator.wikimedia.org/T410159#11375297 (10Dzahn) 05Openβ†’03Resolved [19:23:03] 10GitLab, 06collaboration-services, 07LDAP: sync-gitlab-group-with-ldap - failed to create user / duplicate email - https://phabricator.wikimedia.org/T410159#11375322 (10dancy) Noting for the record that I had prior authorization from MΓ³nica to delete the monica_wmde account. [20:03:40] (03merge) 10jforrester: Revert "deployment-calendar: Moritz takes over for Reuven in the Puppet window" [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/217 (owner: 10rzl) [20:03:50] James_F: thanks! [20:03:50] (03update) 10jforrester: DeploymentCalendar: add arnaudb to sre-collab [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/215 (owner: 10arnaudb) [20:05:30] ^ +1:) [20:05:47] (03approved) 10jforrester: DeploymentCalendar: add arnaudb to sre-collab [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/215 (owner: 10arnaudb) [20:14:09] rzl: Welcome back. [20:18:34] thanks :) [20:21:49] Project beta-update-databases-eqiad build #89133: 04FAILURE in 1 min 48 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/89133/ [20:40:11] 20:21:49 Fatal error: Uncaught MediaWiki\Registration\ExtensionDependencyError: Capiunto requires Scribunto to be installed. [20:40:12] 20:21:49 in /srv/mediawiki-staging/php-master/includes/Registration/ExtensionRegistry.php:477 [20:40:38] Clearly https://github.com/wikimedia/mediawiki-extensions-Capiunto/commit/e5d2a6d2c40fb3ca1d4846a1379b88fc9ac50952 [20:41:08] Why does Loginwiki have Capiunto? [20:41:46] it's default false in production [20:41:49] but not on beta [20:43:11] 10Beta-Cluster-Infrastructure, 10Wikimedia-Site-requests: beta loginwiki has Capiunto but not Scribunto - https://phabricator.wikimedia.org/T410171 (10Reedy) 03NEW [20:43:27] 10Beta-Cluster-Infrastructure, 10Wikimedia-Site-requests: beta loginwiki has Capiunto but not Scribunto - https://phabricator.wikimedia.org/T410171#11375583 (10Reedy) p:05Triageβ†’03High [20:44:03] is it just me with no idea what Capiunto is [20:44:54] 10Beta-Cluster-Infrastructure, 10Wikimedia-Site-requests, 13Patch-For-Review: beta loginwiki has Capiunto but not Scribunto - https://phabricator.wikimedia.org/T410171#11375586 (10taavi) [20:46:16] it is not just you [20:46:49] infobox stuffs built ontop of scribunto [20:47:17] is that live on some production wiki? or just beta? [20:47:25] Isn't it also dead? Shouldn't WMDE have decommissioned it after their evaluation said to stop? [20:48:14] Friday extension undeploy [20:48:15] It's live in `wikidataclient-test` (i.e., prod test wikis). [20:48:16] What could go wrong [20:48:20] Let's not. [20:48:24] :( [20:50:12] Filed as T410172 [20:50:13] T410172: Consider dropping the decade-stalled Capiunto experimental extension from production - https://phabricator.wikimedia.org/T410172 [20:51:46] https://www.mediawiki.org/wiki/Developers/Maintainers (unsurprisingly) says Unassigned [20:51:56] hoo is the maintainer [20:51:58] insert jokes here [21:22:45] Yippee, build fixed! [21:22:46] Project beta-update-databases-eqiad build #89134: 09FIXED in 2 min 45 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/89134/ [22:30:37] 10Beta-Cluster-Infrastructure, 10Wikimedia-Site-requests: beta loginwiki has Capiunto but not Scribunto - https://phabricator.wikimedia.org/T410171#11375824 (10bd808) [22:33:10] 10Beta-Cluster-Infrastructure, 10Wikimedia-Site-requests: beta loginwiki has Capiunto but not Scribunto - https://phabricator.wikimedia.org/T410171#11375831 (10bd808) 05Openβ†’03Resolved a:03Reedy `lang=irc [21:22] Yippee, build fixed! [21:22] Project beta-update-databases... [23:57:22] dduvall: is it OK if I restart the loadbalancer in gitlab-runners-staging? It will probably break your k8s cluster for a minute or two. [23:57:39] andrewbogott: sure, no problem [23:57:43] If that's a pain then I'll look for a different labrat, I just want to see if restarting the amphora resets the mtu [23:58:10] not a pain at all [23:58:11] ok, here goes...