[01:17:39] 10Gerrit, 10Mobile: Cant type / in gerrit web editor - https://phabricator.wikimedia.org/T282387 (10Reedy) [05:00:59] 10Phabricator: Exception while accessing second page of notifications in phabricator - https://phabricator.wikimedia.org/T282400 (10santhosh) [05:32:44] 10Phabricator: Exception while accessing second page of notifications in phabricator - https://phabricator.wikimedia.org/T282400 (10DannyS712) [05:32:57] 10Phabricator (2021-03-24), 10Upstream: Unhandled exception clicking "Next" button on Notifications page: Rows must have unique IDs. An underlying query may be missing a GROUP BY. - https://phabricator.wikimedia.org/T276447 (10DannyS712) [05:33:48] 10Phabricator (2021-03-24), 10Upstream: Unhandled exception clicking "Next" button on Notifications page: Rows must have unique IDs. An underlying query may be missing a GROUP BY. - https://phabricator.wikimedia.org/T276447 (10DannyS712) @Aklapper you marked this as having been fixed upstream with a patch merg... [05:59:19] 10LibUp, 10User-DannyS712: "build: Updating y18n to 3.2.2" commits didn't all work properly - https://phabricator.wikimedia.org/T278857 (10Legoktm) 05Open→03Invalid Seems to have been fine? Please reopen if something is actually wrong. [06:10:16] 10LibUp, 10User-DannyS712: "build: Updating y18n to 3.2.2" commits didn't all work properly - https://phabricator.wikimedia.org/T278857 (10DannyS712) >>! In T278857#7073329, @Legoktm wrote: > Seems to have been fine? Please reopen if something is actually wrong. The bad commits linked in the task description... [06:14:27] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [06:14:56] 10LibUp, 10User-DannyS712: "build: Updating y18n to 3.2.2" commits didn't all work properly - https://phabricator.wikimedia.org/T278857 (10Legoktm) What is npm installing on those repos? y18n doesn't show up as vulnerable on https://libraryupgrader2.wmcloud.org/vulns/npm for those repos...? [07:17:59] 10Gerrit, 10Mobile: Can't type slash character ("/") in Gerrit web editor - https://phabricator.wikimedia.org/T282387 (10Aklapper) [07:41:45] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [07:46:42] 10Release-Engineering-Team (Doing), 10GerritBot, 10Developer Productivity, 10Regression: Gerritbot turns "+" into space, thus breaking most Gerrit URLs - https://phabricator.wikimedia.org/T280197 (10QChris) >>! In T280197#7067687, @hashar wrote: > I have send a tentative patch upstream to url encode POST m... [08:24:15] 10Deployments, 10Release-Engineering-Team (Radar), 10Language-Team: Spot check on-wiki messages against prior day's translatewiki updates - https://phabricator.wikimedia.org/T93308 (10Nikerabbit) Since l10n-update is currently disappeared, I guess this task is not relevant? Please reopen if you disagree. Fr... [08:48:06] (03PS6) 10Hashar: Add job to build our Gerrit plugins [integration/config] - 10https://gerrit.wikimedia.org/r/684417 [08:48:09] (03PS1) 10Hashar: dockerfiles: gerrit move GERRIT_CACHE_HOME in image [integration/config] - 10https://gerrit.wikimedia.org/r/688197 [08:48:51] (03CR) 10Hashar: [C: 03+2] dockerfiles: gerrit move GERRIT_CACHE_HOME in image [integration/config] - 10https://gerrit.wikimedia.org/r/688197 (owner: 10Hashar) [08:50:36] (03Merged) 10jenkins-bot: dockerfiles: gerrit move GERRIT_CACHE_HOME in image [integration/config] - 10https://gerrit.wikimedia.org/r/688197 (owner: 10Hashar) [08:51:50] (03PS7) 10Hashar: Add job to build our Gerrit plugins [integration/config] - 10https://gerrit.wikimedia.org/r/684417 [09:25:16] (03CR) 10Hashar: [C: 03+2] Merge 'upstream/stable-3.2' into wmf/stable-3.2 [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/687117 (owner: 10Hashar) [09:32:17] (03Merged) 10jenkins-bot: Merge 'upstream/stable-3.2' into wmf/stable-3.2 [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/687117 (owner: 10Hashar) [09:49:37] 10Release-Engineering-Team (Doing), 10SRE, 10GitLab (Initialization), 10User-brennen: Define auth strategy for GitLab - https://phabricator.wikimedia.org/T274461 (10jbond) below is a summary of the current issues copied from slack > * We would like to use the e-mail and ssh public key provided by CAS, but... [10:22:58] 10Scap, 10MediaWiki-extensions-LocalisationUpdate: Alert when l10update fails - https://phabricator.wikimedia.org/T171925 (10Nikerabbit) 05Open→03Invalid l10nupdate is not currently running [11:16:58] hmm, somehow beta cluster is no longer syncing from master. Is there a phab task about this? [11:18:48] likely just ci stuck, no need to open a task if someone lurking here kicks it [11:19:32] Zabe: https://phabricator.wikimedia.org/T282206 [11:19:59] thanks :) [11:29:08] 10Release-Engineering-Team (Doing), 10SRE, 10GitLab (Initialization), 10User-brennen: Define auth strategy for GitLab - https://phabricator.wikimedia.org/T274461 (10jbond) >> - Logout is still an issue > See above i dont think this is an issue I have just tested idp.wmfcloud.org configured with SingleSi... [11:35:28] 10Continuous-Integration-Config, 10Growth-Team (Current Sprint), 10Patch-For-Review: Recursive service instantiation: Circular dependency when creating service! MainWANObjectCache - https://phabricator.wikimedia.org/T282320 (10daniel) Hm, got a fully stack trace? I suspect this involves ObjectCache::newFrom... [11:36:47] 10Continuous-Integration-Config, 10Growth-Team (Current Sprint), 10Patch-For-Review: Recursive service instantiation: Circular dependency when creating service! MainWANObjectCache - https://phabricator.wikimedia.org/T282320 (10daniel) [11:43:01] 10Continuous-Integration-Config, 10Growth-Team (Current Sprint), 10Patch-For-Review: Recursive service instantiation: Circular dependency when creating service! MainWANObjectCache - https://phabricator.wikimedia.org/T282320 (10daniel) >>! In T282320#7074304, @daniel wrote: > Hm, got a fully stack trace? > >... [12:39:34] 10Release-Engineering-Team (Doing), 10SRE, 10GitLab (Initialization), 10User-brennen: Define auth strategy for GitLab - https://phabricator.wikimedia.org/T274461 (10MoritzMuehlenhoff) >>! In T274461#7073817, @jbond wrote: > >> * User dangle issue. If a user is deleted in Apereo and then another user gets... [13:17:58] 10Continuous-Integration-Config, 10Growth-Team (Current Sprint), 10Patch-For-Review: Recursive service instantiation: Circular dependency when creating service! MainWANObjectCache - https://phabricator.wikimedia.org/T282320 (10Tgr) For the time being, we could work around the issue by making UserNameUtils::i... [13:45:48] 10Continuous-Integration-Config, 10Growth-Team (Current Sprint), 10Patch-For-Review: Recursive service instantiation: Circular dependency when creating service! MainWANObjectCache - https://phabricator.wikimedia.org/T282320 (10daniel) So the trigger for this problem is probably >https://gerrit.wikimedia.org/... [14:08:32] 10LibUp, 10User-DannyS712: "build: Updating y18n to 3.2.2" commits didn't all work properly - https://phabricator.wikimedia.org/T278857 (10DannyS712) >>! In T278857#7073343, @Legoktm wrote: > What is npm installing on those repos? y18n doesn't show up as vulnerable on https://libraryupgrader2.wmcloud.org/vulns... [14:15:09] 10Continuous-Integration-Config, 10Growth-Team (Current Sprint), 10Patch-For-Review: Recursive service instantiation: Circular dependency when creating service! MainWANObjectCache - https://phabricator.wikimedia.org/T282320 (10Tgr) UserNameUtils is loaded in User::getID, which is called in the SessionBackend... [14:35:02] (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/VoteNY] Add SocialProfile as a phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/687420 (owner: 10Jack Phoenix) [14:35:07] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/VoteNY] Add SocialProfile as a phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/687420 (owner: 10Jack Phoenix) [14:37:36] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/VoteNY] Add SocialProfile as a phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/687420 (owner: 10Jack Phoenix) [14:38:31] !log Zuul: [mediawiki/extensions/VoteNY] Add SocialProfile as a phan dependency [14:38:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:39:07] (03CR) 10Jforrester: [C: 04-1] dockerfiles: [mediawiki-phan-taint-check-demo] Provide composer [integration/config] - 10https://gerrit.wikimedia.org/r/684019 (https://phabricator.wikimedia.org/T257301) (owner: 10Daimona Eaytoy) [14:50:55] (03CR) 10Gergő Tisza: [C: 03+1] "We had three CI breaks due to core changes in the last month, and community configuration ( https://www.mediawiki.org/wiki/Growth/Communit" [integration/config] - 10https://gerrit.wikimedia.org/r/687769 (https://phabricator.wikimedia.org/T247507) (owner: 10Kosta Harlan) [15:02:55] (03Abandoned) 10Jforrester: Stop branching the InterwikiSorting extension [tools/release] - 10https://gerrit.wikimedia.org/r/599069 (https://phabricator.wikimedia.org/T253764) (owner: 10Jforrester) [15:03:30] 10Release-Engineering-Team (Doing), 10MediaWiki-extensions-InterwikiSorting, 10Patch-For-Review, 10Technical-Debt: Undeploy the InterwikiSorting extension from Wikipedia production - https://phabricator.wikimedia.org/T253764 (10Jdforrester-WMF) 05Open→03Declined >>! In T253764#6921275, @Addshore wrote:... [15:05:17] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Quota-requests): Request increased quota for deployment-prep Cloud VPS project - https://phabricator.wikimedia.org/T281625 (10Andrew) 05Open→03Resolved a:03Andrew done! [15:39:35] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Doing), 10Zuul, and 2 others: Zuul can't stop jobs or set the build description - https://phabricator.wikimedia.org/T281737 (10hashar) [15:47:41] will try some Gerrit plugin upgrade tonight to exercise the doc I have wrote :] [16:33:29] (03PS1) 10Jgleeson: Update wikimedia-fundraising-civicrm-docker to pull civicrm-buildkit from upstream repo. [integration/config] - 10https://gerrit.wikimedia.org/r/688350 (https://phabricator.wikimedia.org/T277500) [16:35:59] (03CR) 10jerkins-bot: [V: 04-1] Update wikimedia-fundraising-civicrm-docker to pull civicrm-buildkit from upstream repo. [integration/config] - 10https://gerrit.wikimedia.org/r/688350 (https://phabricator.wikimedia.org/T277500) (owner: 10Jgleeson) [16:38:12] (03PS1) 10Lars Wirzenius: WIP: port to Python 3, drop support for Python 2 [tools/scap] - 10https://gerrit.wikimedia.org/r/688351 [16:39:41] (03PS2) 10Jgleeson: Pull civicrm-buildkit from upstream repo. [integration/config] - 10https://gerrit.wikimedia.org/r/688350 (https://phabricator.wikimedia.org/T277500) [16:41:43] (03PS3) 10Jgleeson: wikimedia-fundraising-civicrm-docker: pull civicrm-buildkit from upstream repo. [integration/config] - 10https://gerrit.wikimedia.org/r/688350 (https://phabricator.wikimedia.org/T277500) [16:43:04] (03CR) 10Lars Wirzenius: "This is a big change, but it can't really be usefully broken into smaller changes, alas." [tools/scap] - 10https://gerrit.wikimedia.org/r/688351 (owner: 10Lars Wirzenius) [16:45:09] (03CR) 10jerkins-bot: [V: 04-1] WIP: port to Python 3, drop support for Python 2 [tools/scap] - 10https://gerrit.wikimedia.org/r/688351 (owner: 10Lars Wirzenius) [16:54:24] 10Release-Engineering-Team (Doing), 10SRE, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10Sergey.Trofimovsky.SF) These were all addressed (please review). Please note, that some options are overridden in `host... [17:07:25] PROBLEM - SSH on contint2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [17:09:35] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T281146 (10Urbanecm) [17:09:37] 10Beta-Cluster-Infrastructure, 10Structured-Data-Backlog, 10WikibaseMediaInfo, 10Beta-Cluster-reproducible: scap sync fails on beta with Error: Class 'MediaWiki\Extension\MediaSearch\I18nUtils' not found - https://phabricator.wikimedia.org/T282206 (10Urbanecm) [17:17:40] 10Beta-Cluster-Infrastructure, 10Structured-Data-Backlog, 10WikibaseMediaInfo, 10Beta-Cluster-reproducible: scap sync fails on beta with Error: Class 'MediaWiki\Extension\MediaSearch\I18nUtils' not found - https://phabricator.wikimedia.org/T282206 (10Reedy) Yeah... That's not going to work. While the exten... [17:37:13] 10Beta-Cluster-Infrastructure, 10Structured-Data-Backlog, 10WikibaseMediaInfo, 10Beta-Cluster-reproducible: scap sync fails on beta with Error: Class 'MediaWiki\Extension\MediaSearch\I18nUtils' not found - https://phabricator.wikimedia.org/T282206 (10matthiasmullie) IIRC that file doesn’t require any other... [17:39:24] 10Beta-Cluster-Infrastructure, 10Structured-Data-Backlog, 10WikibaseMediaInfo, 10Beta-Cluster-reproducible: scap sync fails on beta with Error: Class 'MediaWiki\Extension\MediaSearch\I18nUtils' not found - https://phabricator.wikimedia.org/T282206 (10Reedy) I think so, yeah. Can't say 100% offhand, but sho... [17:46:10] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10Platform Engineering Roadmap: Convert static mediawiki configuration to form more suitable for containers - https://phabricator.wikimedia.org/T263166 (10AMooney) [18:18:28] 10Release-Engineering-Team (Doing), 10SRE, 10GitLab (Initialization), 10User-brennen: Define auth strategy for GitLab - https://phabricator.wikimedia.org/T274461 (10Sergey.Trofimovsky.SF) >>! In T274461#7073817, @jbond wrote: >> - SSH keys cannot be imported from SSO, regardless of SSO type, CAS or SA... [18:18:41] 10Release-Engineering-Team (Doing), 10SRE, 10GitLab (Initialization), 10User-brennen: Define auth strategy for GitLab - https://phabricator.wikimedia.org/T274461 (10Sergey.Trofimovsky.SF) Anyway, given that the decision to stick with SSO has been made, we're going with CAS, having following limitations in... [18:27:02] 10Beta-Cluster-Infrastructure, 10Structured-Data-Backlog, 10WikibaseMediaInfo, 10Beta-Cluster-reproducible, 10Patch-For-Review: scap sync fails on beta with Error: Class 'MediaWiki\Extension\MediaSearch\I18nUtils' not found - https://phabricator.wikimedia.org/T282206 (10dancy) Confirming that this is a t... [18:44:41] 10Release-Engineering-Team (Doing), 10SRE, 10GitLab (Initialization), 10User-brennen: Define auth strategy for GitLab - https://phabricator.wikimedia.org/T274461 (10Sergey.Trofimovsky.SF) Requesting some more information on the current LDAP schema, which attributes can and should be used for key mapping (h... [19:09:14] RECOVERY - SSH on contint2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [19:54:20] (03PS7) 10Ahmon Dancy: WIP: Convert train-dev to use docker instead of VMs [tools/train-dev] - 10https://gerrit.wikimedia.org/r/684040 [21:10:33] (03PS8) 10Ahmon Dancy: WIP: Convert train-dev to use docker instead of VMs [tools/train-dev] - 10https://gerrit.wikimedia.org/r/684040 [21:26:37] (03PS9) 10Ahmon Dancy: WIP: Convert train-dev to use docker instead of VMs [tools/train-dev] - 10https://gerrit.wikimedia.org/r/684040 [21:57:49] 10Release-Engineering-Team (Doing), 10SRE, 10GitLab (Initialization), 10User-brennen: Define auth strategy for GitLab - https://phabricator.wikimedia.org/T274461 (10Sergey.Trofimovsky.SF) More good news: inspired by this issue (https://gitlab.com/gitlab-org/gitlab/-/issues/24510), @Sfigor was able to make... [22:16:39] 10Release-Engineering-Team, 10GitLab, 10Software-Licensing: Add LICENSE/COPYING to operations/gitlab-ansible - https://phabricator.wikimedia.org/T282496 (10Reedy) [22:19:27] 10Release-Engineering-Team, 10GitLab (Initialization), 10Software-Licensing: Add LICENSE/COPYING to operations/gitlab-ansible - https://phabricator.wikimedia.org/T282496 (10greg) [22:55:27] 10Release-Engineering-Team (Doing), 10MediaWiki-extensions-InterwikiSorting, 10Technical-Debt: Undeploy the InterwikiSorting extension from Wikipedia production - https://phabricator.wikimedia.org/T253764 (10Bugreporter) In my opinion this seems something that should be merged to MediaWiki core. [23:10:53] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T281146 (10Jdforrester-WMF) [23:11:23] 10Beta-Cluster-Infrastructure, 10Structured-Data-Backlog, 10WikibaseMediaInfo, 10Beta-Cluster-reproducible, 10MW-1.37-notes (1.37.0-wmf.4; 2021-05-04): scap sync fails on beta with Error: Class 'MediaWiki\Extension\MediaSearch\I18nUtils' not found - https://phabricator.wikimedia.org/T282206 (10Jdforrester...