[00:40:22] (03PS1) 10Jforrester: parameter_functions: Drop old comment about gate and re-sort [integration/config] - 10https://gerrit.wikimedia.org/r/587381 [00:46:30] (03CR) 10Jforrester: Split default_stages out into known_stages (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/581629 (owner: 10Jforrester) [00:48:34] 10phan, 10phan-taint-check-plugin, 10Patch-For-Review: phan and taint-check weird interactions: PhanUndeclaredThis, part 3 - https://phabricator.wikimedia.org/T249619 (10Daimona) 05Open→03Resolved [00:48:38] 10Continuous-Integration-Config, 10LibUp, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10phan, and 2 others: Upgrade phan to 0.10.2 and remove phan-taint-check-plugin - https://phabricator.wikimedia.org/T248630 (10Daimona) [00:55:29] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Epic: Have all Wikimedia production extensions and skins in the CI gate - https://phabricator.wikimedia.org/T249674 (10Jdforrester-WMF) [00:57:13] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Epic: Have all Wikimedia production extensions and skins in the CI gate - https://phabricator.wikimedia.org/T249674 (10Jdforrester-WMF) [00:57:15] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Epic: Have dependencies of gated extensions in the gate - https://phabricator.wikimedia.org/T204252 (10Jdforrester-WMF) [00:57:20] 10Continuous-Integration-Config, 10Wikidata, 10wikidata-tech-focus, 10Patch-For-Review, 10User-Addshore: Add more Wikibase extensions to gatedextensions - https://phabricator.wikimedia.org/T204153 (10Jdforrester-WMF) [00:57:26] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata, 10Wikidata-Campsite, 10Patch-For-Review: [Task] Add Scribunto to extension-gate in CI - https://phabricator.wikimedia.org/T125050 (10Jdforrester-WMF) [01:16:46] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Epic, 10Technical-Debt, 10ci-test-error: Figure out which production extensions have conflicting tests and fix them - https://phabricator.wikimedia.org/T238492 (10Jdforrester-WMF) [01:16:48] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Epic: Have all Wikimedia production extensions and skins in the CI gate - https://phabricator.wikimedia.org/T249674 (10Jdforrester-WMF) [01:18:08] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T247774 (10Krinkle) [02:20:34] Project beta-update-databases-eqiad build #41343: 04FAILURE in 33 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/41343/ [02:44:48] 10Beta-Cluster-Infrastructure, 10Security-Team, 10Wikimedia-Site-requests, 10ContentSecurityPolicy: Change Content Security Policy on betacommons to allow api.flickr.com - https://phabricator.wikimedia.org/T249486 (10sbassett) >>! In T249486#6032726, @Bawolff wrote: > Note, the flickr thing is part of the... [03:20:30] Project beta-update-databases-eqiad build #41344: 04STILL FAILING in 29 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/41344/ [04:20:30] Project beta-update-databases-eqiad build #41345: 04STILL FAILING in 29 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/41345/ [05:20:31] Project beta-update-databases-eqiad build #41346: 04STILL FAILING in 30 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/41346/ [05:31:26] 10Deployments, 10Release-Engineering-Team-TODO, 10Developer Productivity: Train changelog: Omit empty sections - https://phabricator.wikimedia.org/T249553 (10Majavah) How is that list initially created? [05:39:45] Yippee, build fixed! [05:39:45] Project mwcore-phpunit-coverage-master build #591: 09FIXED in 2 hr 39 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/591/ [05:48:28] (03CR) 10Awight: [C: 03+2] Split default_stages out into known_stages (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/581629 (owner: 10Jforrester) [05:48:35] (03CR) 10jerkins-bot: [V: 04-1] Split default_stages out into known_stages [integration/quibble] - 10https://gerrit.wikimedia.org/r/581629 (owner: 10Jforrester) [06:20:32] Project beta-update-databases-eqiad build #41347: 04STILL FAILING in 31 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/41347/ [07:20:31] Project beta-update-databases-eqiad build #41348: 04STILL FAILING in 30 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/41348/ [08:20:30] Project beta-update-databases-eqiad build #41349: 04STILL FAILING in 29 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/41349/ [09:08:09] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Quibble, 10Regression: Jenkins no longer fails build if mw-error.log or mw-dberror.log are non-empty - https://phabricator.wikimedia.org/T246358 (10daniel) [09:20:34] Project beta-update-databases-eqiad build #41350: 04STILL FAILING in 33 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/41350/ [09:34:54] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10Dzahn) >>! In T224591#6024260, @hashar wrote:... [10:20:28] Project beta-update-databases-eqiad build #41351: 04STILL FAILING in 28 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/41351/ [10:43:06] (03PS2) 10Awight: Split default_stages out into known_stages [integration/quibble] - 10https://gerrit.wikimedia.org/r/581629 (owner: 10Jforrester) [10:43:19] (03CR) 10Awight: "PS 2: manual rebase" [integration/quibble] - 10https://gerrit.wikimedia.org/r/581629 (owner: 10Jforrester) [10:45:22] (03CR) 10Awight: [C: 03+2] Split default_stages out into known_stages [integration/quibble] - 10https://gerrit.wikimedia.org/r/581629 (owner: 10Jforrester) [10:46:04] (03Merged) 10jenkins-bot: Split default_stages out into known_stages [integration/quibble] - 10https://gerrit.wikimedia.org/r/581629 (owner: 10Jforrester) [11:20:28] Project beta-update-databases-eqiad build #41352: 04STILL FAILING in 27 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/41352/ [12:09:25] (03CR) 10Alexandros Kosiaris: [C: 03+1] "Fine by me if this is going to unblock migrations and push the infrastructure forward. From what I gather it's the current state anyway. B" [integration/config] - 10https://gerrit.wikimedia.org/r/587304 (https://phabricator.wikimedia.org/T236675) (owner: 10Hashar) [12:19:32] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "Yea, thanks! There is much more to do in this class, and this patch is already a very welcome improvement. Thanks a lot for working with t" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/586430 (owner: 10Umherirrender) [12:20:28] Project beta-update-databases-eqiad build #41353: 04STILL FAILING in 27 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/41353/ [13:20:29] Project beta-update-databases-eqiad build #41354: 04STILL FAILING in 28 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/41354/ [13:46:39] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10hashar) [14:20:31] Project beta-update-databases-eqiad build #41355: 04STILL FAILING in 30 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/41355/ [14:27:47] dvpita, eljohnny: I've pinged you in a zulip chat with gsoc students [14:28:16] just fyi, no action is needed, but I wanted you to be aware of the discussion [14:28:53] I've also added you as reviewers on relevant patches, so you've aware of the action [14:32:21] zeljkof: don't see any notifications [14:32:37] gerrit? zulip? [14:33:26] dvpita: ^ [14:33:52] zeljkof: zulip [14:34:12] I'm pretty new to zulip, I don't really know how it works :) [14:34:19] I do get e-mail notification for pings [14:34:25] I signed up today so :D [14:34:26] and I see them in zulip [14:34:39] I did ping you in an edited post, maybe that's it [14:34:44] let me try again [14:34:45] no emails for me [14:35:39] ah, zulip said you're not subscribed to the topic, so I've subscribed you, I guess you should get a ping now :) [14:44:30] hashar: Hi, I wanted to air my doubts about the "command pattern" thing I introduced into Quibble. Whenever you have time. [14:45:55] awight: shout? ;] [14:46:03] :-) [14:46:05] I haven't looked much at Quibble to be honest [14:46:14] I am busy with other fun things [14:46:16] :\ [14:47:12] Me neither... But last year I introduced the "Command" changes, which made the pipeline a series of deferred actions. It builds up a playbook, then you can choose to print what steps would be taken, or run it. [14:47:32] I'm thinking that was a ridiculous level of abstraction, and maybe we should go back to simple immediate execution. [14:47:52] well I liked the idea [14:47:57] specially the --dry-run one [14:48:25] The place I got stuck was here, https://gerrit.wikimedia.org/r/#/c/integration/quibble/+/519776/ [14:48:38] and I like how building the plan and actual execution are decoupled [14:48:41] The issue is that some commands can't be entirely figured out ahead of time. [14:48:50] ah [14:48:57] I don't think I looked at that change [14:49:04] Yeah, it has some nice aspects... it's cool to be able to take inventory of your pipeline, validate stages and stuff, before actually running. [14:49:16] But it's also much more complicated. [14:49:49] so maybe it is not worth it after all? [14:50:12] I guess that's the question. [14:51:53] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10ops-monitoring-bot) Script wmf-auto-reimage wa... [14:52:36] Hmm [14:53:37] Lmk what you think of this generator approach in the patch I linked, when you feel like Quibble fun. It might be a way forward, which doubles down on the abstraction and finishes the job by getting the entire pipeline into these Commands. [14:53:41] !log contint2001 down for reimaging [14:53:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:54:01] Once that happens, it will be easy to swap out the whole pipeline. [14:55:04] But if this doesn't look like a good idea, then I can dismantle the Command abstraction, move to immediate execution, and I think all we lose is "--dry-run". [14:55:51] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Support redis as a cache store - https://phabricator.wikimedia.org/T152802 (10hashar) 05Open→03Declined If we go for HA, we will most certainly use the high availability plugin. Although the caches are not share... [14:56:19] awight: well I like the command abstraction [14:56:25] (03CR) 10Awight: "I'd like to find an incremental way to do this." [integration/quibble] - 10https://gerrit.wikimedia.org/r/519776 (owner: 10Awight) [14:56:35] awight: but I guess I don't really have brain cycles to jump into those changes yet :-\ [14:56:54] No problem! We can keep the command encapsulation, but without deferring execution. [14:58:18] I think I'll keep going down this Command path in that case, but will try to break up my patches to be more incremental. [14:58:43] Command objects are really nice once parallelism is in play. [14:59:20] hashar: contint2001 reimage is ongoing. sukhe reported an "ABORTED" jenkins-bot message on https://gerrit.wikimedia.org/r/c/operations/puppet/+/587531/ [14:59:25] bah [14:59:32] still waiting if it happens again after recheck [14:59:36] but have a meeting in 2 min [14:59:39] mutante: that is usually zuul cancelling a job [14:59:54] yea, but can it be related to zuul-merge on 2001 ? [14:59:58] merger [15:00:14] sorry, gotta go [15:00:34] will be back and watching the reimage progress [15:00:53] mutante: nop it is unrelated to contint2001 [15:01:01] dont worry :] [15:01:12] awight: I have a few other ideas as well [15:01:48] hashar: What would be a good place to exchange roadmappy ideas? [15:01:52] awight: one would be to have some kind of commands passed to quibble to execute a group of stage [15:01:59] similar to maven (if you are familiar with it) [15:02:03] +1 [15:02:26] so that quibble would have a linear chain of goals and one can instruct quibble to reach a specific goal or resume from a given goal [15:02:26] eg [15:02:48] Declarative pipelines would be very cool. [15:03:32] a chain such as: fetch-repo > install linters dependencies > run repo linters > fetch all repos > install dependencies > run tests [15:03:48] I think I see what you mean about goals, too. So the custom pipeline definition doesn't have to specify all the micro dependencies, but just high-level outcomes. [15:03:58] and potentially on CI we would have a parent job that just fetch the repo and run the linter [15:04:22] then if that passes, we archive the result, pass it to other sub jobs which resume from goal "fetch all repos" > "install dependencies" [15:04:31] and the result is then passed to multiple jobs each running a specific test [15:04:45] the idea is to stop having to git clone and composer/npm install for every single jobs [15:05:03] so yeah high level outcomes :D [15:05:16] but that ideas sounds pretty much like reinventing the wheel [15:05:37] I've been wondering about one detail: it there a big benefit in having the jobs split up at a Jenkins level, or would it be okay to run them all within Quibble? Yeah there is some redundancy there. [15:08:27] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): operations-puppet-tests-buster-docker times out after 5 minutes - https://phabricator.wikimedia.org/T249726 (10hashar) [15:10:28] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): operations-puppet-tests-buster-docker times out after 5 minutes - https://phabricator.wikimedia.org/T249726 (10hashar) p:05Triage→03Medium a:03hashar [15:10:50] (03PS1) 10Hashar: Slightly bump operations-puppet-tests timeout [integration/config] - 10https://gerrit.wikimedia.org/r/587539 (https://phabricator.wikimedia.org/T249726) [15:15:14] (03CR) 10Hashar: [C: 03+2] Slightly bump operations-puppet-tests timeout [integration/config] - 10https://gerrit.wikimedia.org/r/587539 (https://phabricator.wikimedia.org/T249726) (owner: 10Hashar) [15:15:48] (03CR) 10Muehlenhoff: [C: 03+1] "That's fine to unblock the migration to Buster and off Jessie, but please create a task to revisit this; we should be able to apply an inv" [integration/config] - 10https://gerrit.wikimedia.org/r/587304 (https://phabricator.wikimedia.org/T236675) (owner: 10Hashar) [15:16:06] (03Merged) 10jenkins-bot: Slightly bump operations-puppet-tests timeout [integration/config] - 10https://gerrit.wikimedia.org/r/587539 (https://phabricator.wikimedia.org/T249726) (owner: 10Hashar) [15:18:17] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review: operations-puppet-tests-buster-docker times out after 5 minutes - https://phabricator.wikimedia.org/T249726 (10hashar) I have raised the timeou... [15:20:37] Project beta-update-databases-eqiad build #41356: 04STILL FAILING in 36 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/41356/ [15:20:48] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are half the speed of newer ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10hashar) >>! In T223971#5916792, @Bstorm wrote: > This is likely to be fixed with the introduction of a number... [15:26:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10cloud-services-team (Kanban): Migrate integration-agent-puppet-docker-1001 to a different cloudvirt machine - https://phabricator.wikimedia.org/T249727 (10hashar) [15:26:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10cloud-services-team (Kanban): Migrate integration-agent-puppet-docker-1001 to a different cloudvirt machine - https://phabricator.wikimedia.org/T249727 (10hashar) [15:26:32] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review: operations-puppet-tests-buster-docker times out after 5 minutes - https://phabricator.wikimedia.org/T249726 (10hashar) [15:27:06] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review: operations-puppet-tests-buster-docker times out after 5 minutes - https://phabricator.wikimedia.org/T249726 (10hashar) 05Open→03Stalled The... [15:31:44] !log zuul-merger stopped on contint2001. It is being reimaged to Buster! # T224591 [15:31:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:31:47] T224591: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 [15:53:36] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services): Assess whether we should still disable seccomp in Docker for CI - https://phabricator.wikimedia.org/T249729 (10Jdforrester-WMF) [15:53:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Assess whether we should still disable seccomp in Docker for CI - https://phabricator.wikimedia.org/T249729 (10Jdforrester-WMF) [16:00:40] !log Updating wikipeg-npm-php-docker with seccomp over-ride for final T236675 testing [16:00:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:00:43] T236675: Investigate Docker slowness between 18.06.2 and 18.09.7 - https://phabricator.wikimedia.org/T236675 [16:01:24] !log Updating composer-package-php72-docker with seccomp over-ride for final T236675 testing [16:01:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:01:44] James_F: \o/ [16:01:56] which should be a noop [16:02:01] hashar: Yes, should be. [16:02:02] cause seccomp is not being used [16:02:14] Oh, BTW, there are 67 jjb jobs not touched by this. [16:02:27] https://www.irccloud.com/pastebin/PAV08D5z/ [16:02:49] The service pipeline stuff and to-migrate-to-docker ones I knew. [16:03:00] But I didn't realise the debian-glue jobs weren't in docker? [16:03:13] yeah [16:03:14] :( [16:03:23] Do we need to do that ahead of buster? [16:03:24] I havennt managed to run cowbuilder within a docker container [16:03:28] cause it mount /proc for example [16:03:30] Or just a TODO. [16:03:32] Ah, fun. [16:03:38] so the way we build Debian pakcages would have to be overhauled [16:03:48] for the pipeline lib, indeed calls to docker should be updated [16:04:27] !log Updating all docker JJB jobs with seccomp over-ride for T236675 [16:04:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:05:29] I am going outside a bit for some fresh air before the team call [16:05:35] thanks for the jobs update! [16:05:41] Of coursew. [16:05:51] Thanks for the impressive investigation. [16:10:41] (03PS2) 10Jforrester: Disable seccomp in Docker [integration/config] - 10https://gerrit.wikimedia.org/r/587304 (https://phabricator.wikimedia.org/T236675) (owner: 10Hashar) [16:11:13] (03CR) 10Jforrester: [C: 03+2] "Deployed. Let's follow-up in the future on T249729, as requested." [integration/config] - 10https://gerrit.wikimedia.org/r/587304 (https://phabricator.wikimedia.org/T236675) (owner: 10Hashar) [16:12:11] (03Merged) 10jenkins-bot: Disable seccomp in Docker [integration/config] - 10https://gerrit.wikimedia.org/r/587304 (https://phabricator.wikimedia.org/T236675) (owner: 10Hashar) [16:13:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Test-Coverage: mwcore-phpunit-coverage-master times out after 5 hours - https://phabricator.wikimedia.org/T232706 (10Jdforrester-WMF) [16:13:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review: Investigate Docker slowness between 18.06.2 and 18.09.7 - https://phabricator.wikimedia.org/T236675 (10Jdforrester-WMF) 05Open→03Res... [16:14:45] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Test-Coverage: mwcore-phpunit-coverage-master times out after 5 hours - https://phabricator.wikimedia.org/T232706 (10Jdforrester-WMF) 05Open→03Resolve... [16:16:50] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Growth-Team, 10MediaWiki-extensions-GettingStarted, 10User-notice: Deprecate/undeploy the GettingStarted extension - https://phabricator.wikimedia.org/T235752 (10Jdforrester-WMF) p:05Low→03Lowest [16:20:28] Project beta-update-databases-eqiad build #41357: 04STILL FAILING in 27 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/41357/ [16:26:34] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T247774 (10cscott) [16:27:47] 10Beta-Cluster-Infrastructure, 10Cognate, 10Core Platform Team, 10Wikidata, 10Wikidata-Campsite: Database Update broken on beta - https://phabricator.wikimedia.org/T249732 (10Reedy) [16:28:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations: Assess whether we should still disable seccomp in Docker for CI - https://phabricator.wikimedia.org/T249729 (10MoritzMuehlenhoff) [16:29:20] 10Beta-Cluster-Infrastructure, 10Cognate, 10Core Platform Team, 10Wikidata, 10Wikidata-Campsite: Database Update broken on beta - https://phabricator.wikimedia.org/T249732 (10Reedy) p:05Triage→03High [16:34:26] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Wikimedia-GitHub: Github Actions do not run correctly on Github mirrors of Gerrit-hosted repositories - https://phabricator.wikimedia.org/T245826 (10thcipriani) That address is unverified because it does not exist it see... [16:43:13] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T247774 (10Jdforrester-WMF) [16:53:45] 10Release-Engineering-Team-TODO, 10Performance-Team, 10Core Platform Team Workboards (Clinic Duty Team), 10PHP 7.2 support, 10Wikimedia-Incident: Performance regression from Apcu/ExtensionRegistry::loadFromQueue on PHP7 - https://phabricator.wikimedia.org/T187154 (10Krinkle) By far the biggest is `2231 w... [17:06:30] 10Gerrit, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Fundraising-Backlog: Reset fr-tech drush repo, create sub-repo for vendor - https://phabricator.wikimedia.org/T249547 (10Jdforrester-WMF) [17:09:28] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10Dzahn) contint2001 has been reimaged with buster using the wmf-auto-... [17:09:50] hashar: https://phabricator.wikimedia.org/T224591#6040713 [17:09:57] E: Package 'jenkins' has no installation candidate :) [17:10:01] Error: Could not find command '/usr/bin/helm' [17:10:04] these are next [17:10:11] but it's all progress [17:10:15] on buster now [17:11:16] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T247774 (10Jdlrobson) [17:11:33] ah cool thank you mutante [17:11:58] going to dinner, ttyl : [17:12:00] :) [17:12:07] i'll schedule downtime or something [17:13:15] [cumin1001:~] $ sudo cookbook sre.hosts.downtime -r reimage -t T224591 -H 24 contint2001.wikimedia.org [17:13:15] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10ops-monitoring-bot) Icinga downtime for 1 day, 0:00:00 set by dzahn@... [17:13:18] bbl [17:13:19] T224591: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 [17:15:42] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10hashar) `jenkins` ====== is managed via `reprepro` https://wikitech... [17:16:01] mutante: I replied on the task, we can continue tomorrow ;] [17:19:16] 10Deployments, 10Release-Engineering-Team-TODO, 10Developer Productivity: Train changelog: Omit empty sections - https://phabricator.wikimedia.org/T249553 (10thcipriani) p:05Triage→03Low >>! In T249553#6038671, @Majavah wrote: > How is that list initially created? Generated via https://gerrit.wikimedia.... [17:19:52] 10Release-Engineering-Team-TODO, 10Scap, 10serviceops: Deploy Scap version 3.14.0-1 - https://phabricator.wikimedia.org/T249250 (10thcipriani) p:05Triage→03Medium [17:19:59] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10MoritzMuehlenhoff) >>! In T224591#6040713, @Dzahn wrote: > contint20... [17:20:29] Project beta-update-databases-eqiad build #41358: 04STILL FAILING in 28 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/41358/ [17:21:48] 10Release-Engineering-Team-TODO, 10Scap, 10Wikimedia-Incident: Unit tests for wfLoadExtension @ operations/mediawiki-config - https://phabricator.wikimedia.org/T136782 (10thcipriani) p:05Triage→03Low [17:24:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10Zuul, 10Developer Productivity: Abort a Zuul pipeline when one job completed with failures (change zuul scheduler's failure check... - https://phabricator.wikimedia.org/T248531 [17:25:54] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Quibble jobs re-download npm packages every build (Castor not loading?) - https://phabricator.wikimedia.org/T234738 (10thcipriani) [17:27:36] next run should pass [17:29:41] 10phan: Phan complains that 'all' isn't a string - https://phabricator.wikimedia.org/T249738 (10kaldari) [17:31:37] 10phan: Phan complains that 'all' isn't a string - https://phabricator.wikimedia.org/T249738 (10Daimona) The issue message says: > (expected type to be non-nullable) it complains because $ns can be null, not because of 'all'. I haven't checked whether it can really be null though. [17:34:16] 10Phabricator (Search), 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10User-MModell, 10User-brennen: Make search context highlights work with the ferret search engine - https://phabricator.wikimedia.org/T230787 (10mmodell) 05Open→03Stall... [17:34:18] 10Phabricator (2020-01-23): No results for specific string in Global Phab Search, while Advanced Search lists expected results (due to ElasticSearch for Global Search vs Ferret for Maniphest Search)? - https://phabricator.wikimedia.org/T224082 (10mmodell) [17:34:21] 10Phabricator, 10Release-Engineering-Team (Development services), 10User-MModell: Make sure elasticsearch 6 is supported in phabricator - https://phabricator.wikimedia.org/T181393 (10mmodell) [17:37:13] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['contint2001... [17:47:47] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO, 10MediaWiki-extensions-CLDR, 10ParserFunctions, 10Wikimedia-Fundraising: Decide what to do about MW 1.31 / PHP 7.0 incompatible extensions in Wikimedia Fundraising production - https://phabricator.wikimedia.org/T249576 (10Jdforrester-WMF) [17:49:22] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10MediaWiki-Docker, 10Patch-For-Review: Set up CI for mediawiki/tools/cli - https://phabricator.wikimedia.org/T248412 (10Jdforrester-WMF) 05Open→03Resolved a:03kostajh [17:50:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations: Assess whether we should still disable seccomp in Docker for CI - https://phabricator.wikimedia.org/T249729 (10hashar) p:05Triage→03Medium `seccomp` has too much o... [17:51:26] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery, 10Discovery-Search, and 2 others: WikibaseCirrusSearch and WikibaseLexemeCirrusSearch tests should run with WIkibase in CI - https://phabricator.wikimedia.org/T244487 (10Jdforrester-WMF) WikibaseCirrusSearch tests are in gate but not in https://... [17:52:46] 10Phabricator Antivandalism Extension, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10User-brennen: PHAVA: Create a CLI and Test suite for phabricator antivandalism - https://phabricator.wikimedia.org/T249741 (10mmodell) [17:57:04] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Quality-and-Test-Engineering-Team (QTE), 10User-zeljkofilipin: Cypress: add folders to jenkins artifacts - https://phabricator.wikimedia.org/T248373 (10zeljkofilipin) [17:57:59] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10serviceops, and 3 others: Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10thcipriani) p:05Triage→03High a:03Jdforreste... [17:59:13] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap, 10Wikimedia-Site-requests: Move scap plugins from mediawiki-config repo to scap repo - https://phabricator.wikimedia.org/T248490 (10thcipriani) p:05Triage→03Medium a:03LarsWirzenius [17:59:28] 10Gerrit, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): check gerrit-replica healthcheck alerts - https://phabricator.wikimedia.org/T247186 (10thcipriani) a:03thcipriani [18:01:04] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Quibble jobs re-download npm packages every build (Castor not loading?) - https://phabricator.wikimedia.org/T234738 (10thcipriani) a:03Jdforrester-WMF tentatively assigning to @Jdforreste... [18:01:51] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/train-blockers] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/587568 [18:01:55] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [labs/tools/train-blockers] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/587568 (owner: 10QChris) [18:01:58] 10Continuous-Integration-Infrastructure (Slipway), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Release Pipeline, 10Wikimedia-Portals: Migrate wikimedia-portals-build to Docker container - https://phabricator.wikimedia.org/T213806 (10Jdrewniak) [18:02:02] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/train-blockers] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/587569 [18:02:05] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [labs/tools/train-blockers] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/587569 (owner: 10QChris) [18:16:06] (03PS2) 10Jforrester: deployment-calendar: Add train triage on Wednesday [tools/release] - 10https://gerrit.wikimedia.org/r/584640 (owner: 10Thcipriani) [18:18:34] Is there a list of jouncebot’s messages it uses to announce deploys [18:19:44] Yeah, it's in a git repo [18:21:07] Yippee, build fixed! [18:21:08] Project beta-update-databases-eqiad build #41359: 09FIXED in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/41359/ [18:21:48] Reedy: which one? [18:21:56] The repo for the bot :P [18:22:07] 10Scap, 10User-brennen: Investigate scap-cdb-rebuild idling until pressing ENTER repeatedly - https://phabricator.wikimedia.org/T223287 (10brennen) [18:22:30] 10phan: Phan complains that 'all' isn't a string - https://phabricator.wikimedia.org/T249738 (10kaldari) @Daimona - Why would `$ns !== 'all'` make it think that $ns can be null (removing that part of the code fixes the error). That doesn't make any sense. And even if it did, it's perfectly fine to pass a null va... [18:23:34] Reedy: and a link to that would be? [18:24:54] It's pretty easy to google for [18:24:54] https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/bots/jouncebot/+/master/etc/DefaultConfig.yaml [18:40:56] 10Beta-Cluster-Infrastructure, 10Cognate, 10Core Platform Team, 10Wikidata, and 2 others: Database Update broken on beta - https://phabricator.wikimedia.org/T249732 (10Reedy) 05Open→03Resolved a:03tstarling [18:41:59] 10Continuous-Integration-Infrastructure, 10Wikimedia-production-error (Shared Build Failure): Occasional Selenium build failures related to temporary directories / no space left - https://phabricator.wikimedia.org/T243223 (10Jdforrester-WMF) 05Open→03Resolved Optimistically declaring Resolved, then. [18:56:20] 10phan: Phan complains that 'all' isn't a string - https://phabricator.wikimedia.org/T249738 (10Daimona) >>! In T249738#6041077, @kaldari wrote: > @Daimona - Why would `$ns !== 'all'` make it think that $ns can be null? I can't answer this question, because it depends on the internal dynamics of phan. > And ev... [18:58:41] (03PS1) 10Jforrester: Release Quibble 0.0.41 [integration/quibble] - 10https://gerrit.wikimedia.org/r/587580 (https://phabricator.wikimedia.org/T225068) [18:58:44] (03PS1) 10Jforrester: changelog: Begin new 0.0.42 version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/587581 [18:59:26] (03CR) 10jerkins-bot: [V: 04-1] Release Quibble 0.0.41 [integration/quibble] - 10https://gerrit.wikimedia.org/r/587580 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester) [18:59:29] (03CR) 10jerkins-bot: [V: 04-1] changelog: Begin new 0.0.42 version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/587581 (owner: 10Jforrester) [19:00:26] 10phan: Phan complains that 'all' isn't a string - https://phabricator.wikimedia.org/T249738 (10kaldari) @Daimona - Thanks, I'll suppress it inline, but I hope there's a way to prevent this in the future, as it seems like a needless waste of time. [19:01:41] (03PS2) 10Jforrester: Release Quibble 0.0.41 [integration/quibble] - 10https://gerrit.wikimedia.org/r/587580 (https://phabricator.wikimedia.org/T225068) [19:01:43] (03PS2) 10Jforrester: changelog: Begin new 0.0.42 version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/587581 [19:03:25] (03CR) 10DannyS712: [C: 03+1] "LGTM" [integration/quibble] - 10https://gerrit.wikimedia.org/r/587580 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester) [19:03:42] (03CR) 10DannyS712: [C: 03+1] "LGTM" [integration/quibble] - 10https://gerrit.wikimedia.org/r/587581 (owner: 10Jforrester) [19:04:34] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Analytics, and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10Nuria... [19:04:45] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Analytics, and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10Nuria... [19:04:49] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Delete Jenkins label DebianJessie - https://phabricator.wikimedia.org/T239981 (10Nuria) [19:06:07] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Delete Jenkins label DebianJessie - https://phabricator.wikimedia.org/T239981 (10Jdforrester-WMF) [19:06:10] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Analytics, and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10Jdforr... [19:09:33] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Analytics, and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10Nuria... [19:13:58] 10Continuous-Integration-Infrastructure (Slipway), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Release Pipeline, 10Wikimedia-Portals: Migrate wikimedia-portals-build to Docker container - https://phabricator.wikimedia.org/T213806 (10Jdforrester-WMF) a:03jeena... [19:16:27] hashar: Can we delete https://integration.wikimedia.org/ci/job/analytics-refinery-release/ ? [19:23:57] hashar: Also, if you could bless https://gerrit.wikimedia.org/r/587580 I'll roll it. [19:31:52] 10phan: Phan complains that 'all' isn't a string - https://phabricator.wikimedia.org/T249738 (10kaldari) @Daimona - FYI, setting `null_casts_as_any_type` to false in the Phan config didn't work (https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/PageAssessments/+/587399/). How do you find out which Phan rul... [19:37:50] James_F: https://integration.wikimedia.org/ci/job/analytics-refinery-release/ << yes it can be deleted [19:40:29] for quibble [19:40:36] I am not sure I see much values in the patch for "Split default_stages out into known_stages" ;D [19:40:49] it duplicates the list ;] [19:41:11] but yeah that is harmless [19:41:30] (03CR) 10Hashar: [C: 03+2] "Lets go!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/587580 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester) [19:41:41] (03CR) 10Hashar: [C: 03+2] changelog: Begin new 0.0.42 version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/587581 (owner: 10Jforrester) [19:42:37] (03Merged) 10jenkins-bot: Release Quibble 0.0.41 [integration/quibble] - 10https://gerrit.wikimedia.org/r/587580 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester) [19:42:50] (03Merged) 10jenkins-bot: changelog: Begin new 0.0.42 version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/587581 (owner: 10Jforrester) [19:43:17] Thanks. [19:43:34] Hmm. [19:44:40] !log Tag Quibble 0.0.41 cc2b408 # T234738 T225068 [19:44:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:44:43] T234738: Quibble jobs re-download npm packages every build (Castor not loading?) - https://phabricator.wikimedia.org/T234738 [19:44:44] T225068: Add a PHPUnit group to skip test on gated CI runs - https://phabricator.wikimedia.org/T225068 [19:44:54] Oh, thanks. [19:45:06] Was just looking to see if we should rev ci-buster and so on whilst we're at it. [19:45:07] I have tagged it [19:45:09] 10phan: Phan complains that 'all' isn't a string - https://phabricator.wikimedia.org/T249738 (10kaldari) I fixed it by suppressing that particular error type: ` $cfg['suppress_issue_types'] = array_merge( $cfg['suppress_issue_types'], [ // This test seems to be buggy or overly strict (T249738) 'PhanTypeMismatc... [19:47:49] James_F: we still are based on Stretch aren't we? Given that is what is used in prod [19:47:57] For quibble jobs, yes. [19:48:17] But https://phabricator.wikimedia.org/T249218 has me worried that our dependencies might be wrong. [19:50:44] 10Release-Engineering-Team-TODO, 10Performance-Team, 10Core Platform Team Workboards (Clinic Duty Team), 10PHP 7.2 support, 10Wikimedia-Incident: Performance regression from Apcu/ExtensionRegistry::loadFromQueue on PHP7 - https://phabricator.wikimedia.org/T187154 (10Anomie) >>! In T187154#6040641, @Krink... [19:50:46] hashar: For quibble, why don't we hard-code the version in the Dockerfile.template, instead of fetching LATEST_TAG? [19:50:58] We have to manually re-build anyway. [19:51:15] `LATEST_TAG=$(git ls-remote --refs --tags https://gerrit.wikimedia.org/r/p/integration/quibble.git|cut -f2|cut -d/ -f3|sort --version-sort -r|head -n1)` is impressive but not necessary. [19:51:50] ah [19:52:03] yeah that is to save the troubles of having to remember to update the tag to fetch [19:52:14] We need to touch changelog. [19:52:16] so it is always using whatever is the latest version [19:52:18] It's not too hard. [19:52:22] so eventually [19:52:23] And avoids accidental upgrades. [19:52:44] I had a patch for docker-pkg to expose the version from the changelog as a variable that could be used in the Dockerfile.template [19:52:49] Hmm. [19:52:54] That'd be nice. [19:53:07] so theorically we could have done: git fetch {{ version }} && git checkout {{ version }} [19:53:09] ;] [19:53:19] and eventually I gave up at some point I guess [19:53:31] Fine, I'll leave it as-is for now. [19:53:44] :] [19:54:37] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are half the speed of newer ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10Andrew) We're in the process of replacing most of these hosts. Everything is slowed down by COVID but at lea... [19:56:29] I am going to brick my laptop [19:56:34] gotta update the firmware [19:56:35] :/ [20:00:04] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Wikimedia-GitHub: Github Actions do not run correctly on Github mirrors of Gerrit-hosted repositories - https://phabricator.wikimedia.org/T245826 (10thcipriani) a:05thcipriani→03WMDE-leszek > At WMDE we believe the i... [20:07:40] (03PS1) 10Jforrester: dockerfiles: [quibble-stretch] Install quibble 0.0.41 and cascade. [integration/config] - 10https://gerrit.wikimedia.org/r/587588 (https://phabricator.wikimedia.org/T225068) [20:07:43] (03PS1) 10Jforrester: jjb: [quibble*] Switch to quibble 0.0.41 images [integration/config] - 10https://gerrit.wikimedia.org/r/587589 (https://phabricator.wikimedia.org/T225068) [20:21:31] (03PS1) 10Jforrester: jjb: Provide jobs for running "standalone" phpunit tests only [integration/config] - 10https://gerrit.wikimedia.org/r/587592 (https://phabricator.wikimedia.org/T225068) [20:21:33] (03PS1) 10Jforrester: layout: [mediawiki/extensions/Scribunto] Run standalone path too [integration/config] - 10https://gerrit.wikimedia.org/r/587593 (https://phabricator.wikimedia.org/T225068) [20:22:26] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [quibble-stretch] Install quibble 0.0.41 and cascade. [integration/config] - 10https://gerrit.wikimedia.org/r/587588 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester) [20:23:59] (03Merged) 10jenkins-bot: dockerfiles: [quibble-stretch] Install quibble 0.0.41 and cascade. [integration/config] - 10https://gerrit.wikimedia.org/r/587588 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester) [20:24:32] (03CR) 10jerkins-bot: [V: 04-1] layout: [mediawiki/extensions/Scribunto] Run standalone path too [integration/config] - 10https://gerrit.wikimedia.org/r/587593 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester) [20:25:33] !log Docker: Publishing quibble 0.0.41 images [20:25:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:37:57] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:39:09] (03PS2) 10Jforrester: layout: [mediawiki/extensions/Scribunto] Run standalone path too [integration/config] - 10https://gerrit.wikimedia.org/r/587593 (https://phabricator.wikimedia.org/T225068) [20:39:58] (03CR) 10jerkins-bot: [V: 04-1] layout: [mediawiki/extensions/Scribunto] Run standalone path too [integration/config] - 10https://gerrit.wikimedia.org/r/587593 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester) [20:42:47] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 93355 bytes in 1.046 second response time [20:59:19] (03PS3) 10Jforrester: layout: [mediawiki/extensions/Scribunto] Run standalone path too [integration/config] - 10https://gerrit.wikimedia.org/r/587593 (https://phabricator.wikimedia.org/T225068) [21:00:10] (03CR) 10jerkins-bot: [V: 04-1] layout: [mediawiki/extensions/Scribunto] Run standalone path too [integration/config] - 10https://gerrit.wikimedia.org/r/587593 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester) [21:06:26] (03CR) 10Jforrester: [C: 03+2] jjb: [quibble*] Switch to quibble 0.0.41 images [integration/config] - 10https://gerrit.wikimedia.org/r/587589 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester) [21:06:29] (03CR) 10Jforrester: [C: 03+2] jjb: Provide jobs for running "standalone" phpunit tests only [integration/config] - 10https://gerrit.wikimedia.org/r/587592 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester) [21:07:20] (03Merged) 10jenkins-bot: jjb: [quibble*] Switch to quibble 0.0.41 images [integration/config] - 10https://gerrit.wikimedia.org/r/587589 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester) [21:07:22] (03Merged) 10jenkins-bot: jjb: Provide jobs for running "standalone" phpunit tests only [integration/config] - 10https://gerrit.wikimedia.org/r/587592 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester) [21:12:43] (03PS4) 10Jforrester: layout: [mediawiki/extensions/Scribunto] Run standalone path too [integration/config] - 10https://gerrit.wikimedia.org/r/587593 (https://phabricator.wikimedia.org/T225068) [21:14:40] (03PS5) 10Jforrester: layout: [mediawiki/extensions/Scribunto] Run standalone always [integration/config] - 10https://gerrit.wikimedia.org/r/587593 (https://phabricator.wikimedia.org/T225068) [21:14:42] (03PS1) 10Jforrester: layout: [mediawiki/extensions/Scribunto] Standalone, experimentally [integration/config] - 10https://gerrit.wikimedia.org/r/587599 (https://phabricator.wikimedia.org/T225068) [21:16:11] (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki/extensions/Scribunto] Standalone, experimentally [integration/config] - 10https://gerrit.wikimedia.org/r/587599 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester) [21:17:00] (03Merged) 10jenkins-bot: layout: [mediawiki/extensions/Scribunto] Standalone, experimentally [integration/config] - 10https://gerrit.wikimedia.org/r/587599 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester) [21:18:42] !log layout: [mediawiki/extensions/Scribunto] Standalone, experimentally T225068 [21:18:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:18:45] T225068: Add a PHPUnit group to skip test on gated CI runs - https://phabricator.wikimedia.org/T225068 [21:21:54] (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki/extensions/Scribunto] Run standalone always [integration/config] - 10https://gerrit.wikimedia.org/r/587593 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester) [21:22:43] (03Merged) 10jenkins-bot: layout: [mediawiki/extensions/Scribunto] Run standalone always [integration/config] - 10https://gerrit.wikimedia.org/r/587593 (https://phabricator.wikimedia.org/T225068) (owner: 10Jforrester) [21:28:37] !log layout: [mediawiki/extensions/Scribunto] Standalone, always T225068 [21:28:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:28:39] T225068: Add a PHPUnit group to skip test on gated CI runs - https://phabricator.wikimedia.org/T225068 [21:30:01] 10Continuous-Integration-Config, 10ContentTranslation, 10MediaWiki-extensions-Interwiki, 10MediaWiki-extensions-TranslationNotifications, and 2 others: Add i18n related extensions to mediawiki-extensions-(php55|php70|hhvm) - https://phabricator.wikimedia.org/T86930 (10Jdforrester-WMF) [21:30:03] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Epic: Have all Wikimedia production extensions and skins in the CI gate - https://phabricator.wikimedia.org/T249674 (10Jdforrester-WMF) [21:30:30] 10Continuous-Integration-Config, 10ContentTranslation, 10MediaWiki-extensions-Interwiki, 10MediaWiki-extensions-TranslationNotifications, and 2 others: Add i18n related extensions to the CI gate - https://phabricator.wikimedia.org/T86930 (10Jdforrester-WMF) [21:31:00] 10Continuous-Integration-Config, 10ContentTranslation, 10MediaWiki-extensions-Interwiki, 10MediaWiki-extensions-TranslationNotifications, and 2 others: Add i18n related extensions to the CI gate - https://phabricator.wikimedia.org/T86930 (10Jdforrester-WMF) [21:38:24] (03PS1) 10Jforrester: jjb: Skip "phpunit-standalone" everywhere except for the special job [integration/config] - 10https://gerrit.wikimedia.org/r/587601 [21:41:01] https://wikitech.wikimedia.org/w/index.php?title=Heterogeneous_deployment%2FTrain_deploys&type=revision&diff=1862747&oldid=1861133 [21:41:30] brennen: It's now official. [21:41:39] Before, we could just pretend this wasn't a bug. [21:41:42] Now we've documented it. [21:41:46] ;-) [21:41:49] it's not a bug, James_F [21:41:52] it's _process_ [21:41:56] Uh-huh. [21:42:06] https://xkcd.com/1172/ [21:42:19] maybe it's not fully process until somebody... yeah, that xkcd. [21:42:24] * James_F laughs. [21:43:23] (03CR) 10Jforrester: [C: 03+2] jjb: Skip "phpunit-standalone" everywhere except for the special job [integration/config] - 10https://gerrit.wikimedia.org/r/587601 (owner: 10Jforrester) [21:44:46] (03Merged) 10jenkins-bot: jjb: Skip "phpunit-standalone" everywhere except for the special job [integration/config] - 10https://gerrit.wikimedia.org/r/587601 (owner: 10Jforrester) [21:52:46] 10Scap, 10User-brennen: Investigate scap-cdb-rebuild idling until pressing ENTER repeatedly - https://phabricator.wikimedia.org/T223287 (10Krinkle) Might be related: {T246959} [21:56:03] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10CirrusSearch, and 4 others: Add a PHPUnit group to skip test on gated CI runs - https://phabricator.wikimedia.org/T225068 (10Jdforrester-WMF) 05Open→03Resolve... [21:56:08] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata, 10Wikidata-Campsite, 10Patch-For-Review: [Task] Add Scribunto to extension-gate in CI - https://phabricator.wikimedia.org/T125050 (10Jdforrester-WMF) [21:56:11] 10Continuous-Integration-Config, 10ContentTranslation, 10MediaWiki-extensions-Interwiki, 10MediaWiki-extensions-TranslationNotifications, and 2 others: Add i18n related extensions to the CI gate - https://phabricator.wikimedia.org/T86930 (10Jdforrester-WMF) [22:00:30] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review: Quibble jobs re-download npm packages every build (Castor not loading?) - https://phabricator.wikimedia.org/T234738 (10Jdforrester-WMF) This should now be better. Good... [22:47:11] (03PS1) 10Awight: Factor visitor out of executor [integration/quibble] - 10https://gerrit.wikimedia.org/r/587606 (https://phabricator.wikimedia.org/T249775) [22:48:15] (03CR) 10jerkins-bot: [V: 04-1] Factor visitor out of executor [integration/quibble] - 10https://gerrit.wikimedia.org/r/587606 (https://phabricator.wikimedia.org/T249775) (owner: 10Awight) [22:51:50] (03PS2) 10Awight: Factor visitor out of executor [integration/quibble] - 10https://gerrit.wikimedia.org/r/587606 (https://phabricator.wikimedia.org/T249775) [23:08:53] (03PS1) 10Awight: Support command recursion [integration/quibble] - 10https://gerrit.wikimedia.org/r/587608 (https://phabricator.wikimedia.org/T249775) [23:09:34] (03CR) 10jerkins-bot: [V: 04-1] Support command recursion [integration/quibble] - 10https://gerrit.wikimedia.org/r/587608 (https://phabricator.wikimedia.org/T249775) (owner: 10Awight) [23:10:52] (03PS2) 10Awight: Support command recursion [integration/quibble] - 10https://gerrit.wikimedia.org/r/587608 (https://phabricator.wikimedia.org/T249775) [23:32:45] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10serviceops, and 3 others: Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Krinkle) >>! In T236104#5925763, @gerritbot wrote... [23:59:15] (03PS1) 10Ppchelko: Enable code coverage for FlaggedRevs extension [integration/config] - 10https://gerrit.wikimedia.org/r/587610 [23:59:56] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Quibble, 10Regression: Jenkins no longer fails build if mw-error.log or mw-dberror.log are non-empty - https://phabricator.wikimedia.org/T246358 (10Krinkle) >>! In T246358#6014143, @Lucas_Werkmeister_WMDE wrote: >...