[08:22:13] (03PS1) 10Kosta Harlan: [WIP] Parallelize phpunit -databaseless, -standalone and -database [integration/quibble] - 10https://gerrit.wikimedia.org/r/758783 (https://phabricator.wikimedia.org/T235449) [08:24:55] (03PS2) 10Kosta Harlan: [WIP] Parallelize phpunit -databaseless, -standalone and -database [integration/quibble] - 10https://gerrit.wikimedia.org/r/758783 (https://phabricator.wikimedia.org/T235449) [08:27:59] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Parallelize phpunit -databaseless, -standalone and -database [integration/quibble] - 10https://gerrit.wikimedia.org/r/758783 (https://phabricator.wikimedia.org/T235449) (owner: 10Kosta Harlan) [08:46:37] (03PS3) 10Kosta Harlan: [WIP] Parallelize phpunit -databaseless, -standalone and -database [integration/quibble] - 10https://gerrit.wikimedia.org/r/758783 (https://phabricator.wikimedia.org/T235449) [08:49:33] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Parallelize phpunit -databaseless, -standalone and -database [integration/quibble] - 10https://gerrit.wikimedia.org/r/758783 (https://phabricator.wikimedia.org/T235449) (owner: 10Kosta Harlan) [08:58:14] awight: I don't think I've had enough coffee yet, but I don't know what I am missing in the test plan 001 yaml file, if you have some time to look [11:16:20] (03PS8) 10Awight: [WIP] ci-full-run: Add extensions to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 (owner: 10Kosta Harlan) [11:16:25] (03CR) 10Awight: [C: 03+2] [WIP] ci-full-run: Add extensions to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 (owner: 10Kosta Harlan) [11:17:04] (03PS9) 10Kosta Harlan: ci-full-run: Add extensions to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 [11:19:53] kostajh: I think there's a rebase issue, lemme see... [11:20:02] nope [11:20:56] oh, I think I see the problem [11:20:58] sorry [11:21:33] awight: I have no idea how that got into the patch, but that's I'm sure related https://gerrit.wikimedia.org/r/c/integration/quibble/+/758783/3/quibble/cmd.py#b336 [11:23:05] ✅ for sure! [11:26:08] kostajh: I'm still daydreaming about a dependency solver, but would prefer to start with an existing library. I know this has been done a thousand times, but maybe I'm not using the right search terms... [11:27:55] Want something like: * depend on another task (don't run before it), * some way of separating the build into phases, could be a placeholder step like "finished with linters", then the quibble command execution would just follow the tree and keep as many as max_workers processing going. [11:30:10] So we just need a tree structure and preferrably a declarative way of defining dependencies, imo [11:31:11] (03PS4) 10Kosta Harlan: [WIP] Parallelize phpunit -databaseless, -standalone and -database [integration/quibble] - 10https://gerrit.wikimedia.org/r/758783 (https://phabricator.wikimedia.org/T235449) [11:31:36] (03CR) 10jerkins-bot: [V: 04-1] ci-full-run: Add extensions to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 (owner: 10Kosta Harlan) [11:32:04] awight: yeah, that would be nice, though I still think we should map out what exactly could be offloaded to jenkins (or whatever CI future exists with GitLab) vs what should happen in a single container with quibble [11:34:58] kostajh: I'd love to understand how to make that decision. I think jenkins is good at spawning multiple jobs for environment variants (eg php 7/8), and has nice granular reporting, but otherwise it seems like an advantage to keep everything in one job, to keep the runner portable enough to be local, on jenkins, or GitLab. [11:37:23] awight: somewhat tangentially, have you set up a pipeline on circleci.com before? it's been a while since I have, but IIRC they have an intuitive UX for the type of thing I'm talking about in terms of splitting up a pipeline into various phases, where each step can build on previous step, and each step can have parallel tasks [11:45:12] (03PS5) 10Kosta Harlan: Parallelize phpunit -databaseless, -standalone and -database [integration/quibble] - 10https://gerrit.wikimedia.org/r/758783 (https://phabricator.wikimedia.org/T235449) [11:45:30] (03CR) 10Kosta Harlan: "This one might be riskier, and if merged should be tagged separately in case we need to rollback the change." [integration/quibble] - 10https://gerrit.wikimedia.org/r/758783 (https://phabricator.wikimedia.org/T235449) (owner: 10Kosta Harlan) [11:46:05] I haven't used circleci, maybe we can look at the pipeline description format and read it directly when running quibble locally. [11:46:15] (03CR) 10Kosta Harlan: "Follow up in I760768b6df5b6d583250ce88b8ad8d4527a148cb" [integration/quibble] - 10https://gerrit.wikimedia.org/r/587887 (https://phabricator.wikimedia.org/T235449) (owner: 10Awight) [11:47:41] legoktm[m]: welcome! We're just getting into some fun questions about where to define the CI pipeline: mostly inside of quibble, or externally... [11:47:47] d'oh [11:49:22] kostajh: I wonder if we can fuzz the tests to find single-threading assumptions faster--like running puppet with order=random. [11:49:48] phpunit seems to take a `--random-order-seed` argument O_O [11:50:34] I was thinking about that earlier today, in the context of T300461 [11:50:54] if we start using random ordering of tests, though, I think we'll just end up with a bunch of flaky builds and irritate everyone [11:51:00] and --random-order [11:51:06] oops ^ so wrong [11:51:14] `--order-by random` [11:51:30] kostajh: +1 maybe we can do an experimental thing, though... [11:52:16] I imagine that little glitches will be shaking out for months if we just fix the bugs one at a time. [12:11:24] kostajh: btw this is scary, possibly some parallelization we've already merged is responsible for the flapping? https://integration.wikimedia.org/ci/job/integration-quibble-fullrun/640/console [12:17:16] awight: no, that's from the fact that the tests are run all together which is not something we do in the existing CI config [12:17:31] see also T297078 [12:18:10] Basically, our PHPUnit tests are mostly reliable when they are run exactly as configured in CI. Once you start running them out of order, or running core tests in the same pass as extension tests (as in that build you referenced), things go haywire. [12:19:17] this became more apparent to me in https://gerrit.wikimedia.org/r/c/integration/quibble/+/742201/42 where chopping up the list of tests and running half in one thread and half in another already resulted in a bunch of failures. [12:32:46] blaargh [12:41:11] kostajh: cool! A small list of errors so far... [12:41:30] I suppose the point is that different things will fail each time :-/ [12:43:04] right [12:43:27] I think once we identify some patterns, though, it wouldn't be terrible to fix tests. identifying them in advance might be hard. But we have lists of flaky tests already [12:43:48] For example, some tests make assumptions about which namespaces are available, that should in theory be relatively easy to fix. [12:44:04] I wouldn't be surprised if we uncover application logic bugs though due to faulty assumptions in tests [12:44:44] +1 I also get lots of failures when running locally, there's probably a lot of overlap with tests that are sensitive to initial conditions. [12:52:34] awight: in happier news, https://gerrit.wikimedia.org/r/c/integration/quibble/+/758783 seems to have worked and reduced build time by another minute or so [12:57:22] heck yes. I wonder if we lose something by running phpunit after browser tests, though? [13:08:59] AFAIK, we don't actually use the order defined in cmd.py for the jobs defined in integration/config. [13:09:12] The browser tests are run separately, in their own jobs. So in practice I think reordering there is a no-op. [19:37:27] (03PS1) 10Kosta Harlan: release: Quibble 1.4.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/758952 (https://phabricator.wikimedia.org/T300340) [22:55:51] kostajh: we should pair that release in our morning [22:56:07] I am wondering whether you and awight have sufficient permissions in Gerrit [22:57:55] (03CR) 10Hashar: release: Quibble 1.4.0 (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/758952 (https://phabricator.wikimedia.org/T300340) (owner: 10Kosta Harlan)