[00:00:35] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:05:35] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [03:57:14] من هیچگونه حسابی باز نکردم از حساب من سواستفاده کرده اند ونمیدونم کی هست [07:14:32] (03PS1) 10Kosta Harlan: zuul: Don't run PHPUnit tests in parallel for Extension:Translate [integration/config] - 10https://gerrit.wikimedia.org/r/1080450 (https://phabricator.wikimedia.org/T377197) [07:55:07] 10GitLab (Pipeline Services Migration🐤), 06Language and Product Localization: Migrate machinetranslation/cxserver/apertium to GitLab - https://phabricator.wikimedia.org/T344737#10232574 (10Nikerabbit) [07:56:07] (03PS2) 10Kosta Harlan: zuul: Don't run PHPUnit tests in parallel for Translate and Math [integration/config] - 10https://gerrit.wikimedia.org/r/1080450 (https://phabricator.wikimedia.org/T377197) [08:01:20] kostajh: the thing, as I get it, is that any extension depending on Math or Translate would end up surfacing the same issue wouldn't it? [08:02:54] hashar: potentially, yes. I'm not sure, though. [08:03:13] well the CentralAuth issue happened on Math as well :D [08:03:58] so you could exclude anything that depends on CentralAuth [08:04:00] hashar: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CampaignEvents/+/1079591 depends on Math, and had no issues [08:04:16] ^ depends on Translate, I mean [08:04:25] aka when `mediawiki/extensions/CentralAuth` is found in `params['EXT_DEPENDENCIES']` [08:04:36] :/ [08:05:10] cause it "randomly" fails based on the order [08:06:29] kostajh: an alternative is to disable testViewForExistingGlobalTemporaryAccount :D [08:11:55] yeah [08:12:00] I'll do that for now [08:15:01] ConvertibleTimestamp::setFakeTime( '20240505050505' ); [08:16:01] but if I read mediawiki/libs/Timetamp code, when given a string it parses it as a unix epoch (seconds since 1970) [08:16:40] that is Sunday, May 26, 2611 2:30:50.505 AM :) [08:17:15] I will send a patch [08:17:16] (03CR) 10Kosta Harlan: "Alternative is I47cf760384cc7971a6bee676c214e73a6afa86a2" [integration/config] - 10https://gerrit.wikimedia.org/r/1080450 (https://phabricator.wikimedia.org/T377197) (owner: 10Kosta Harlan) [08:30:09] ah no it is converted [08:30:23] behind the hood. That lib interface has some surprises :] [08:32:09] (03Abandoned) 10Hashar: zuul: Don't run PHPUnit tests in parallel for Translate and Math [integration/config] - 10https://gerrit.wikimedia.org/r/1080450 (https://phabricator.wikimedia.org/T377197) (owner: 10Kosta Harlan) [08:40:35] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: CI gitcache is not updated - https://phabricator.wikimedia.org/T376981#10232744 (10hashar) 05Open→03Resolved a:03hashar [09:01:52] 10Release-Engineering-Team (Seen), 10Scap: `scap clean` failure - https://phabricator.wikimedia.org/T218783#10232799 (10hashar) 05Invalid→03Open I am reopening this task cause we need the old branches to be cleaned (that is T303828, and ultimately T374716 about CI being slow). The original issue was scap... [09:31:08] 10Release-Engineering-Team (Seen), 10Scap, 13Patch-For-Review: `scap clean` failure: SSH Too many authentication failures: 7 - https://phabricator.wikimedia.org/T218783#10232979 (10hashar) [09:31:52] 10Phabricator, 06Trust-and-Safety, 10wikitech.wikimedia.org: Account recovery help needed for Developer account jgiannelos - https://phabricator.wikimedia.org/T377117#10232985 (10Jgiannelos) [09:33:00] 10GitLab (Auth & Access), 10Phabricator, 06Trust-and-Safety, 10wikitech.wikimedia.org: Account recovery help needed for Developer account jgiannelos - https://phabricator.wikimedia.org/T377117#10232988 (10Jgiannelos) [09:38:16] 06Project-Admins: umbrella project for Wikimedia Ukraine - https://phabricator.wikimedia.org/T374503#10233003 (10Aklapper) > So it seems that the old tasks are in the old project. The open ones are in the new one. Hmm, where to see old tasks in the old project (URI)? I admit I forgot about form 107. Was it... [09:55:21] 10GitLab (Auth & Access), 10Phabricator, 06Trust-and-Safety, 10wikitech.wikimedia.org: Account recovery help needed for Developer account jgiannelos - https://phabricator.wikimedia.org/T377117#10233048 (10Aklapper) > I still need 2fa reset for phabricator See the Phabricator frontpage linking to https://w... [10:20:19] 10GitLab (Auth & Access), 10Phabricator, 06Trust-and-Safety, 10wikitech.wikimedia.org: Account recovery help needed for Developer account jgiannelos - https://phabricator.wikimedia.org/T377117#10233098 (10Jgiannelos) >> gitlab > > You may have recovery codes? If not, https://docs.gitlab.com/ee/user/profil... [10:42:26] 10GitLab (Auth & Access), 10Phabricator, 06Trust-and-Safety, 10wikitech.wikimedia.org: Account recovery help needed for Developer account jgiannelos - https://phabricator.wikimedia.org/T377117#10233139 (10Jgiannelos) @Aklapper I found my old 2fa that is already registered in my phabricator (i have 2 regist... [11:52:38] 10GitLab (Auth & Access), 10Phabricator, 06Trust-and-Safety, 10wikitech.wikimedia.org: Account recovery help needed for Developer account jgiannelos - https://phabricator.wikimedia.org/T377117#10233259 (10Aklapper) @Jgiannelos: Not sure I can follow; you could check if https://phabricator.wikimedia.org/set... [12:50:02] maintenance-disconnect-full-disks build 641085 integration-agent-docker-1049 (/: 26%, /srv: 96%, /var/lib/docker: 41%): OFFLINE due to disk space [12:55:03] maintenance-disconnect-full-disks build 641086 integration-agent-docker-1049 (/: 26%, /srv: 16%, /var/lib/docker: 39%): RECOVERY disk space OK [12:59:37] hi releng folks, we're getting CI failures due to " There are no commands defined in the "phpunit:prepare-parallel" namespace. " on our mediawiki build steps. Can anyone help? https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/1080684 [13:00:43] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Jenkins: thirdparty/ci apt component is configured multiple times - https://phabricator.wikimedia.org/T375278#10233403 (10hashar) 05Open→03Resolved a:03hashar Done! ` hashar@contint1002:~$ sudo apt update Hit:1 http://mirrors.wikime... [13:10:41] jgleeson: probably worth filing a subtask of https://phabricator.wikimedia.org/T50217 [13:10:42] (per https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/XQZNOGXOJP62NSNHG24HIMOYWP5CG737/) [13:11:15] if DonationInterface has a custom CI job (with MediaWiki 1.39?) it probably needs to be excluded from parallel CI runs [13:18:26] I feel like it already was [13:18:31] But that doesn't mean it hasn't regressed [13:18:45] thanks Lucas_WMDE. Do you know when the changes were made. I can see passes on patches after Sept 24th e.g. https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/1078443 [13:19:15] hm, I think something was reenabled only a few days ago [13:19:22] maybe there wasn’t another wikitech-l email then [13:19:56] kostajh: ^ [13:19:59] codders: ^ [13:20:02] * Lucas_WMDE looks [13:20:19] the exclusion is made via zuul/parameters_functions.py [13:20:35] jgleeson: there was another change yesterday https://phabricator.wikimedia.org/T377176#10227885 [13:20:40] that’s probably the most recent change that broke it for you [13:21:00] ah [13:21:14] jgleeson: you can make a patch like https://gerrit.wikimedia.org/r/c/integration/config/+/1080450 but update it to specify DonationInterface [13:21:27] (or just restore that patch and edit it) [13:21:30] sure [13:21:47] kostajh: haven't I had flagged `phpunit:prepare-parallel` to be backported to all supported branches? [13:21:57] (03Restored) 10Kosta Harlan: zuul: Don't run PHPUnit tests in parallel for Translate and Math [integration/config] - 10https://gerrit.wikimedia.org/r/1080450 (https://phabricator.wikimedia.org/T377197) (owner: 10Kosta Harlan) [13:22:05] (03PS3) 10Kosta Harlan: zuul: Don't run PHPUnit tests in parallel for DonationInterface [integration/config] - 10https://gerrit.wikimedia.org/r/1080450 [13:22:29] hashar: we discussed not dealing with older release branches, and leaving PHPUnit testing in serial mode there [13:22:32] jgleeson: https://gerrit.wikimedia.org/r/c/integration/config/+/1080450 [13:22:36] ahh [13:22:46] jgleeson: if there's a task, please update https://gerrit.wikimedia.org/r/c/integration/config/+/1080450 and tag it with the task ID [13:22:55] so how is it handled when someone sends a patch to REL1_39? [13:23:24] and not params["ZUUL_BRANCH"].startswith("REL1") and not params["ZUUL_BRANCH"].startswith("fundraising") [13:23:24] :) [13:23:25] no task yet but I'll create one [13:25:58] I should really one day replace that custom https://integration.wikimedia.org/ci/job/quibble-donationinterface-REL1_39-php74/ job [13:26:17] and have DonationInterface switch from master to `fundraising/REL1_39` [13:26:21] that makes things easier [13:26:37] hashar: maybe the better fix instead of https://gerrit.wikimedia.org/r/c/integration/config/+/1080450 is to manually set `QUIBBLE_PHPUNIT_PARALLEL=0` for that job [13:26:39] alternative: move that wiki to the MediaWiki train :] [13:28:10] I'll keep the same logic and exclude DonationInterface specifying it oddly uses master when the code targets fundraising/REL1_39 [13:28:54] stepping away from the keyboard for a while. good luck [13:31:04] (03PS4) 10Jgleeson: zuul: Don't run PHPUnit tests in parallel for DonationInterface [integration/config] - 10https://gerrit.wikimedia.org/r/1080450 (https://phabricator.wikimedia.org/T377327) (owner: 10Kosta Harlan) [13:31:13] kostajh: ^ [13:31:51] * jgleeson sends pigeon to chase kostajh during walk [13:33:10] ah, RFC 1149 [13:33:42] :) [13:55:30] (03Abandoned) 10Hashar: zuul: Don't run PHPUnit tests in parallel for DonationInterface [integration/config] - 10https://gerrit.wikimedia.org/r/1080450 (https://phabricator.wikimedia.org/T377327) (owner: 10Kosta Harlan) [13:55:45] jgleeson|skrun:I will craft another change [13:56:06] I have abandoned that change in favor of an alternative change (disable a faulty test in CentralAuth) [13:59:47] hashar: isn’t it a different issue for DonationInterface? That the parallel run scripts aren’t available in 1.39? [13:59:56] https://phabricator.wikimedia.org/T328240#8639790 [13:59:56] :) [14:00:06] it uses a differentbranching model [14:00:09] so that master [14:00:18] is tested with mediawiki/core fundrasing/REL1_39 [14:00:21] I am crafting the change [14:02:34] (03PS1) 10Hashar: zuul: exclude DonationInterface from parallel PHPUnit [integration/config] - 10https://gerrit.wikimedia.org/r/1080719 (https://phabricator.wikimedia.org/T377327) [14:03:28] kostajh: ^ jgleeson|skrun ^ [14:05:12] (03CR) 10Hashar: [C:03+2] zuul: exclude DonationInterface from parallel PHPUnit [integration/config] - 10https://gerrit.wikimedia.org/r/1080719 (https://phabricator.wikimedia.org/T377327) (owner: 10Hashar) [14:06:28] (03Merged) 10jenkins-bot: zuul: exclude DonationInterface from parallel PHPUnit [integration/config] - 10https://gerrit.wikimedia.org/r/1080719 (https://phabricator.wikimedia.org/T377327) (owner: 10Hashar) [14:08:58] deployed [14:09:05] and rechecked the DonationInterface failing patch [14:09:11] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/1080684 [14:09:21] * hashar in meeting [14:22:52] James_F: thanks for the CR :) I owe you multiple $drinks. Would you (or someone else) be able to point me to documentation on how to add build artifacts in CI for T372887? [14:22:52] T372887: mw-tools-phan-demos-publish fails with: "configure: error: cannot run C compiled programs." - https://phabricator.wikimedia.org/T372887 [14:23:19] I'm sure there's a simple answer, but... [14:29:23] kostajh: jgleeson|skrun: DonationInterface is fixed :) [14:30:14] hashar: thank you! [14:48:34] Daimona: From a googling... it sounds like a version mismatch, shared library weirdness or similar [14:49:15] Yeah I'm sure it's something stupid like that. That's why I wanted to take a peek at the config.log file, assuming it'll point me right to whatever thing is broken this time. [14:49:42] Unfortunately the error is quite generic and my own googling attempt wasn't particularly helpful. [14:52:40] The old logs are gone now too [14:53:07] We have the new ones though, I saw the same symptoms for all postmerge jobs today [14:53:32] So at least we know that, it should be consistently reproducible [14:54:10] Yeah, I was more being lazy and hinting the references might want updating for anyone to look at it ;P [14:54:59] Of course I could also try and reproduce this locally, the problem being 1) I'm lazy as well, and 2) I don't really remember how the demo build process works in the first place :D [14:56:12] I do note that 7.4.16 is quite an old PHP version [14:56:26] WMF prod is 7.4.33 [14:56:34] Indeed, I was thinking about updating to PHP 8.1 straight away [14:56:43] Some image(s) that need updating/references changing? [14:56:49] Entirely possible that's enough to just fix it [14:57:33] That would need a change to the demo repo, which somehow is still in my personal github https://github.com/Daimona/demo [14:57:49] 10Continuous-Integration-Config, 10phan-taint-check-plugin: mw-tools-phan-demos-publish fails with: "configure: error: cannot run C compiled programs." - https://phabricator.wikimedia.org/T372887#10233958 (10Daimona) [14:58:04] But good point. For now I'll try updating to 7.4.33 and see what happens [14:58:55] looks like the image is relatively recently rebuilt by James [14:59:01] Arguably could be what is actually breaking :P [14:59:28] Based on what PHP version that is now including [14:59:39] And could be enough to explain the library/version weirdness [15:00:09] Well there's one way to find out and I'm about to try it :D [15:00:20] leeerrroooooyyyyyyyyy [15:05:12] That's not it, unfortunately https://integration.wikimedia.org/ci/job/mw-tools-phan-demos-publish/350/console [15:05:42] But I also realized that I don't need to export config.log as a build artifact when I can just `cat` it when compiling fails :P [15:17:48] Ohhhhhh but of course changes to node images could be relevant, emscripten requires node... [15:24:20] There's a notice in https://emscripten.org/docs/getting_started/downloads.html#emsdk-install-targets that might be relevant [15:25:13] * Reedy hands Daimona some turtles [15:29:05] https://commons.wikimedia.org/wiki/File:River_terrapin.jpg [15:29:32] 10Continuous-Integration-Config, 10phan-taint-check-plugin: mw-tools-phan-demos-publish fails with: "configure: error: cannot run C compiled programs." - https://phabricator.wikimedia.org/T372887#10234189 (10Daimona) I've updated the PHP version to 7.4.33 but that didn't help. I then made it output the whole c... [15:34:56] ty hashar & kostajh [15:44:11] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor, 07Upstream: castor-save-workspace-cache aborted during postbuild - https://phabricator.wikimedia.org/T352319#10234230 (10hashar) My patch got merged AND released by upstream: ` * f6b0b80 - (tag: 3.4.1-695.vf6b_0b_8053979) Upgrade version to 3.4.... [15:44:50] 10Phabricator, 13Patch-For-Review: Explore restricting editing task priority to Trusted-Contributors/staff/etc - https://phabricator.wikimedia.org/T363337#10234231 (10stjn) >>! In T363337#10124562, @Aklapper wrote: > To make this change in behavior less disruptive for PMs etc, should check first which non-deac... [15:45:59] 10Phabricator, 13Patch-For-Review: Explore restricting editing task priority to Trusted-Contributors/staff/etc - https://phabricator.wikimedia.org/T363337#10234235 (10Aklapper) > Was this done? Yes. > And is there any plan to add active members of Phabricator community to Trusted-Contributors more quickly aft... [15:47:01] 10Phabricator (phabricator-next): Explore restricting editing task priority to Trusted-Contributors/staff/etc - https://phabricator.wikimedia.org/T363337#10234236 (10Aklapper) [15:50:30] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T375658#10234243 (10cscott) Parsoid team is watching {T377287} but that's just logspam. We're going to backport a fix in the late backport window today. [15:53:06] 10Deployments, 10Release-Engineering-Team (Priority Backlog 📥), 07Kubernetes: Build and publish multiple MediaWiki production images for a given set of PHP versions - https://phabricator.wikimedia.org/T370934#10234276 (10Scott_French) [17:12:52] https://phabricator.wikimedia.org/T377372 -- I heard you list out teams, but they also flowed right out of my head. Should I try to tag in more folks than infra-foundations and observability? [17:14:30] I have a hunch that the current solution would look a lot like keeping a cookie jar active, POST of creds to https://idp.wikimedia.org/login, and then following redirects in other authn flows so that CAS tells services that the bot is authenticated. [17:15:34] ugh.. thcipriani ^^ those were @ you [17:18:41] bd808: those folks seem like the folks who might know offhand, thanks for filing <3 [17:20:01] Project beta-update-databases-eqiad build #79672: 04FAILURE in 0.47 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/79672/ [17:25:23] !log removed 2fa from JMando (confirmed in video call) [17:25:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:27:24] 10Continuous-Integration-Config, 10phan-taint-check-plugin: mw-tools-phan-demos-publish fails with: "configure: error: cannot run C compiled programs." - https://phabricator.wikimedia.org/T372887#10234991 (10Daimona) I have confirmed locally that updating emscripten to the latest version (3.1.69) fixes the iss... [18:32:18] Yippee, build fixed! [18:32:18] Project beta-update-databases-eqiad build #79673: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/79673/ [19:50:37] 10Gerrit, 06collaboration-services, 10Wikidata, 10Wikidata Analytics: Clean up old and unused WMDE Gerrit groups - https://phabricator.wikimedia.org/T376951#10235541 (10Dzahn) 05In progress→03Resolved I made all the changes as suggested over in T371581#10035058. - members removed - added "ARCHIVED... [20:10:29] 10GitLab: GitLab Private Repository Request for: wmfuniq (SDS2.3) - https://phabricator.wikimedia.org/T377388 (10BBlack) 03NEW [20:41:47] 10GitLab (CI & Job Runners), 06Growth-Team, 10MediaWiki-extensions-CommunityConfiguration, 07ci-test-error: CommunityConfiguration extension breaking CI for all patches to mediawiki-vendor - https://phabricator.wikimedia.org/T377391 (10cscott) 03NEW [20:42:50] 10GitLab (CI & Job Runners), 06Growth-Team, 10MediaWiki-extensions-CommunityConfiguration, 07ci-test-error: CommunityConfiguration extension breaking CI for all patches to mediawiki-vendor train branch - https://phabricator.wikimedia.org/T377391#10235792 (10cscott) [20:43:28] 10GitLab (CI & Job Runners), 06Growth-Team, 10MediaWiki-extensions-CommunityConfiguration, 07ci-test-error: CommunityConfiguration extension breaking CI for all patches to mediawiki-vendor train branch - https://phabricator.wikimedia.org/T377391#10235793 (10cscott) [20:43:29] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T375658#10235794 (10cscott) [21:38:55] 06Release-Engineering-Team, 10Scap: scap-o-scap: Bootstrapping a new host fails - https://phabricator.wikimedia.org/T310740#10235947 (10Dzahn) Today I ran into this issue again, after reimaging `phab2002`. The initial puppet run failed with some dependency problems because there had been no scap deploy ye... [21:39:12] 14Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap, 06Infrastructure-Foundations, 06serviceops: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559#10235945 (10Dzahn) Today I ran into this issue again, after reimaging `phab2002`. The initial puppet run faile... [22:28:40] 10GitLab (CI & Job Runners), 10MediaWiki-extensions-CommunityConfiguration, 07ci-test-error, 10Growth-Team (Current Sprint): CommunityConfiguration extension breaking CI for all patches to mediawiki-vendor train branch - https://phabricator.wikimedia.org/T377391#10236088 (10KStoller-WMF) [22:32:53] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#10236098 (10Milimetric) Hello, I am serving as interim Engineering Manager for the Data Products team. I need to create sprints and generally wield more power... [22:35:11] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#10236102 (10Ladsgroup) >>! In T706#10236098, @Milimetric wrote: > Hello, I am serving as interim Engineering Manager for the Data Products team. I need to cre... [22:41:50] Thanks Amir, but I'm tsk tsk-ing you for being up so late. tsk tsk [22:47:02] 10GitLab (CI & Job Runners), 10MediaWiki-extensions-CommunityConfiguration, 07ci-test-error, 10Growth-Team (Current Sprint), 13Patch-For-Review: CommunityConfiguration extension breaking CI for all patches to mediawiki-vendor train branch - https://phabricator.wikimedia.org/T377391#10236120 (10KStoller-WM... [23:14:47] 10Phabricator, 10Tool-ldap: https://ldap.toolforge.org/ integration assumes that `cn` and `uid` are equivalent - https://phabricator.wikimedia.org/T376769#10236212 (10Legoktm) Thanks, just deployed: https://gitlab.wikimedia.org/toolforge-repos/ldap/-/commit/7b5baa2b2f25f2ba2e88070bff8319fb20a062f4 So now: `... [23:21:32] 10GitLab (CI & Job Runners), 10MediaWiki-extensions-CommunityConfiguration, 07ci-test-error, 10Growth-Team (Current Sprint), 13Patch-For-Review: CommunityConfiguration extension breaking CI for all patches to mediawiki-vendor train branch - https://phabricator.wikimedia.org/T377391#10236213 (10KStoller-WM...