[15:58:15] check [15:58:43] o/ [16:00:36] * duesen wibbles [16:01:34] #startmeeting TechCom RFC discussion [16:01:47] * Reedy peers his head in [16:01:47] * duesen stares at meetbot [16:01:59] how is going to present the RFC, btw? [16:03:10] good question :D [16:04:10] ok, we have meetbot [16:04:14] Mark or Robert will present. Not sure what their nicks are. Mark is having IRC issues and will join shortly. [16:04:17] but do we have Mark or Robert? [16:04:35] Robert planned to be here. Not sure if he's on yet. [16:04:40] #startmeeting TechCom RFC discussion - RFC: Hybrid extension management - https://phabricator.wikimedia.org/T250406 [16:04:40] Meeting started Wed Jun 3 16:04:40 2020 UTC and is due to finish in 60 minutes. The chair is Krinkle. Information about MeetBot at http://wiki.debian.org/MeetBot. [16:04:40] Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. [16:04:40] The meeting name has been set to 'techcom_rfc_discussion___rfc__hybrid_extension_management___https___phabricator_wikimedia_org_t250406' [16:04:49] thanks Krinkle [16:04:53] is there an agenda for the meeting? I can't seem to find a link on https://www.mediawiki.org/wiki/Architecture_meetings (the URL on the calendar entry) [16:05:07] Krinkle: can you #chair me? Or you run the meeting :) [16:05:31] cscott: the agenda is https://phabricator.wikimedia.org/T250406 [16:05:36] basically. [16:05:38] * Krinkle will chair [16:05:40] #link https://phabricator.wikimedia.org/T250406 [16:06:07] #topic TechCom RFC discussion - RFC: Hybrid extension management [16:06:29] I think only chairs can set the topic... not sure though [16:06:35] btw, for restarting meetbot if needed: https://wikitech.wikimedia.org/wiki/Tool:Meetbot [16:06:48] afaik the bot listens to all? anyway, carry on [16:07:18] hexmode tells me he's on his way: "My irc proxy is there, but I'm having trouble connecting" [16:09:50] Krinkle: depends on the command, see http://meetbot.debian.net/Manual.html [16:09:59] The #topic command sais (Chairs only) [16:10:13] yeah, I'll have to do the end call. Although last time I was able to do that even when someone else started it, so mayve ours is different [16:10:37] Hello? [16:10:57] hexmode: welcome :) [16:11:37] is this the RFC channel ? Had trouble connecting so I'm confused [16:12:02] hexmode: yes. meetbot is listening :) [16:12:25] This is where IRC office hours are held, which once a week has a slot for RFC discussions, yes. [16:12:30] Ok, so I'm here (finally!) to talk about the composer RFC [16:13:47] hexmode: do you want to present it? [16:14:02] I have never presented an RFC and I thought Robert Vogel was doing this, but here I am. How do I start? [16:14:27] Tell is about the problem(s) the proposal is meant to solve. [16:14:57] ...and what the key questions are that you want to see discussed/answered today [16:15:15] Problem: previous RFC said "Don't use composer for extension" which was fine for the WMF. but... [16:15:38] #info Motivation and requirements are documented at https://phabricator.wikimedia.org/T250406 [16:15:56] composer, in the meantime, has become very popular and well supported in other PHP projects [16:16:46] So many 3rd party extension developers have used/abused it to handle extension installation [16:17:08] * hexmode waves to Osnard [16:17:26] (Feel free to jump in Osnard) [16:17:56] composer automates a ton of the installation [16:18:09] and this while MW itself does not [16:18:45] MW should not duplicate this functionality just to adhere to an out-modded RFC [16:19:58] This RFC is meant to standardize best practices and propose some (hopefully minor) changes to MW core so that we can begin to use composer properly. [16:21:29] (Sorry for being late. I was already connected as `rvogel` but didn't realize, that I can't send messages until i register an account with NickServ) [16:22:59] Osnard: what questions do we want to talk about? [16:23:16] I can think of the following: [16:23:45] 1) can we have a new RFC saying composer can be used to install extensions? [16:23:56] This is about the downloading of tarballs and/or git-cloning of extension repos, right? Does MW currently have logic for that? Are there significant transferrable skills and/or simplifications it would offer to users setting MW? E.g. would this mean new sysadmins run fewer commands or learn fewer things, or would they copy one command from a doc page instead of another? I suspect it would make no difference with only what is proposed, [16:23:56] but worth confirming here. This isn't to say it wouldn't be useful. As I understand it, this proposal would enable a separate extension to be created that would go a lot further and do a lot more. Just checking whether this is mainly to enable that or whether ther are other benefits for plain usage as well. [16:24:09] Osnard: welcome. I didn't realize that was the case in this channel either. We had a spam problem at some point, maybe that's why. [16:24:33] 2) can we get some modifications to core? [16:25:17] The way I'd describe the situation, there are two well-documented and agreed-upon install / site management methods: ExtensionDistributor and git. ED is good for beginners (you just need to FTP a bunch of files to the right folder on your shared host) but has all kinds of deficiencies. [16:25:28] git means you are your own for version and library management (submodules, your own vendor repo etc) - big sites will need that, but it's kind of an expert solution. [16:25:36] Krinkle: cli admins would have to just copy one line to install an extension instead of going through a multi-step install process [16:25:56] they still need to enable the extension in LocalSettings.php [16:26:03] So there's a big gap in the middle, which Composer fills, so it became the de facto third install method, but the documentation and shared agreement is lacking. [16:26:11] hexmode: the rfc says it will not install the extension and that it will not use composer to discover related/dependency extensions. [16:26:13] but composer eliminates a bunch of the download and update [16:26:43] Krinkle: I think we're confusing installing and enabling [16:26:45] hexmode: which means ther ewill still be editing of LocalSettings and running of a command per extension, correct? Which steps does it avoid or reduce? [16:27:24] installing I associate with install.php or mw-config GUI installer, which sets up tables etc. and installs MW. [16:27:29] the other is downloading [16:27:56] from my pov, the basic idea is to be able to create composer-local.json with extensions you want, and run composer install to download them all [16:28:04] Krinkle I believe it is mostly about downloading and updating. [16:28:10] #info composer has gotten popular and is well-supported in other PHP projects [16:28:11] (doesn't separating install and enable leave open the possibility of an Extension:ExtensionEnabler will gives you nice checkboxes and a GUI to enable extensions, with the set of enabled extensions automatically managed w/o manual editing of LocalSettings.php?) [16:28:11] finding the extension, finding the right version, downloading, copying files into place [16:28:21] #info 3rd party extension developers have used/abused it to handle extension installation [16:28:40] #info hexmode> cli admins would have to just copy one line to install an extension instead of going through a multi-step install process [16:28:41] cscott: yes [16:28:58] so maybe not even "copy one line to install an extension" [16:29:14] Krinkle: there is a gui for composer that I have not used [16:29:15] #info there are two well-documented and agreed-upon install / site management methods: ExtensionDistributor and git. ED is good for beginners … big sites will need git. [16:29:32] how does composer help to find the right version? as far as I see you still have to figure out the correct version yourself. [16:29:40] #info composer eliminates a bunch of the download and update steps [16:29:50] I think avoiding the update steps is especially significant. [16:29:57] using composer to install REL branches would seem to be the most failproof way? [16:30:29] Nikerabbit: probably overstated the right version bit. Was thinking of composer dependency resolution [16:30:47] tgr: using git does not require a vendor repo. we have composer.local.json for that. fetching lib dependencies with composer is already supported today and is what most dev environments do. [16:30:59] cscott: rel branches can be specified in composer with iirc dev-RELX_XX [16:31:11] composer merge plugin is another kettle of fish [16:31:13] you will get the library dependencies of your extensions as well, that is right, and a nice benefit over git clone [16:31:20] that is, if I were writing Extension:ExtensionManager, i'd query the MW version and use it to write a composer-local.json naming dev-REL1_XX, at least for all extensions managed by MW gerrit [16:31:23] Composer does not help with that. As Nikerabbit says, you'll manage a composer-local.json file. When you want to install a new extension, you add it with the exact version. When you want to update, you edit the versions. [16:31:47] it's still significantly simpler than fetching the files manually. [16:32:15] cscott: right [16:32:17] When extensions are using thrid-party libraries there are issues with the ExtensionDistributor-approach, as well as with the git-clone-approach. AFAIK ExtensionDistributor does not include any vendor-files at the moment. And when having multiple extensions that use the same library the git-clone-approach will leav you with multiple venrdo-subdirs [16:32:17] containing the same or even worse different code of the same library [16:32:18] well, when you update core, you just change REL1_XX to REL1_YY and use composer to update [16:32:46] hexmode: regarding "finding extension,version" - I assume that's the same for git-clone vs composer-require. one needs to copy the name or url, and specify a version, which generally would just be the same for all (REL1_x). [16:33:12] git clone does not check out vendor at all, you have to do it manually either per repo or using composer-local.json with extensions/* include. [16:33:17] Osnard: Yeah it does. bawolff fixed that in the last couple of weeks [16:33:17] it's not perfect, really i'd like to fetch the "latest version of extension X still compatible with MW 1.YY", but it's probably Good Enough for most folks who don't need to live on the bleeding edge [16:33:22] Krinkle: yes, but some devs are beginning to use semantic versioning and not just REL branches [16:33:24] Krinkle: you need a repo to store composer.local.json (presumably), and you'll have to piece together the right libraries by hand. It's a lot more effort/complexity than installing the extensions themselves via Composer. [16:33:52] and REL branches are really only useful for gerrit-hosted extensions [16:33:57] and as has been said, you could manually specify a version (not use the RELx) branch if you knew you wanted some new feature of (say) the Translate extension which maintains long inter-version compatibility [16:34:08] tgr: no, that is what WMF does because we store the result. there is no need to pick or pin anything. just wildcard incldue extensions/* in composer-local and composer require/update will handle it all. [16:34:29] "AFAIK ExtensionDistributor does not include any vendor-files at the moment" - that was fixed by Bawolff some days ago, actually [16:35:15] "just wildcard incldue extensions/* in composer-local and composer require/update will handle it all." <-- that's what I always do for local development [16:35:29] Regarding "composer merge plugin" and "composer-local.json" - is this proposal expected to reduce or remove the need for those? (not asking whether you want to remove them, but whether it means you can install MW + extensions without needing either of those to exist). [16:36:36] Krinkle: yeah, you are right, that works too. [16:37:43] * duesen remindes everyone to make liberal use of the #info tag. that makes it much easier to compose a summary later. [16:37:53] Krinkle No, this RFC is not aiming to remove "composer merge plugin" or "composer-local.json" [16:38:05] still, managing extensions via composer means one config file + one shell command to fetch / update everything that's needed. Significantly simpler than managing them separately. [16:38:07] It sounds like composer is described as a simpler alternative to downloading tarballs. This makes sense to me. I would prefer composer over downloading various tarballs (and making sure to replace directories etc.). However, do we think these audiences overlap? It seems to me that a sysadmin with CLI access and able to and having installed composer is likely also comfortable with Git, and would not consider using tarballs. [16:38:28] no need to use merge-plugin for extensions with the proposed method though [16:38:40] #info this RFC is not aiming to remove "composer merge plugin" or "composer-local.json" [16:38:59] Osnard: would it need composer merge/local in order to work, though? If not, where does it store the extension information? [16:39:08] you'd still need to merge composer-local.json with MW core's composer.json, I think? [16:39:20] #info tgr> "AFAIK ExtensionDistributor does not include any vendor-files at the moment" - that was fixed by Bawolff some days ago, actually [16:39:23] Krinkle: writing equivalent functionality from scratch (I've done it) is not easy at all [16:40:03] An advantage of expressing the list of extensions/skins and their versions in composer.json or composer.local.json rather than downloading them individually from git or ExtensionDistributor is having a config file that describes your working configuration. This is helpful when you are managing multiple servers (dev/staging/production) that you want to be sure you are keeping in sync or to quickly see what any differences [16:40:03] are. [16:40:07] I'd think of this as "enable the creation of better non-CLI tools" [16:40:28] folks writing UX should not also have to deal with all the complexities of network download and install [16:40:42] #info (paragraphse ) composer would allow one to edit one json file, update versions, and then run a composer command to update extensions in-place as well as any dependent extensions. [16:40:54] "blessing" composer for this use means that folks who want to write friendly configuration management tools can have them write a compoer-local.json and "be done" [16:41:12] (well, then invoke composer update programmatically) [16:42:09] i think formally separating "install" from "enable" is also a good architectural clarification [16:42:19] per the RFC, extensions are not allowed to depend on other extensions through their composer.json file [16:42:20] It may be useful to avoid to avoid the word "install", which different people use to mean: 1) just downloading, 2) downloading and enabling, 3) running the cli or web installer. [16:42:29] since as mentioned before it lets you create better tools for *enabling* various extensions, indepdent of the "install" step [16:42:53] `composer.local.json` would probably stay as it is. For `composer-merge-plugin`: I believe it is not really required for the RFC, but I can say that we (BlueSpice) are using it to organize our build files (we have dedicated `composer.json` files for various "packages" defined, that we include with the merge plugin. I can share a link, if you like) [16:43:15] composer.local.json is a non-standard construct powered by composer-merge [16:43:20] they are the same thing afaik [16:43:36] yes. that was my lame invention [16:43:37] Okay, makes sense [16:44:03] What does the `extra.installer-name` refer to in the proposal? What does it do? [16:44:44] This is used to determine the folder-name within the `extensions/` subdirectory [16:45:08] I saw for the first time the other day that you can have a "require" section as well as a "merge" section in composer.local.json. It is useful to keep the site specific composer requirements separate from those in the mediawiki/composer.json which would be overwritten on update. [16:45:20] #info composer.local.json is a non-standard construct powered by composer-merge [16:46:09] By default a pattern is used. E.g. if the packagename was `mediawiki/simple-s-a-m-l-php` it would install to `extensions/SimpleSAMLphp`. With the `extra.installer-name` you can have a package name like `mediawiki/simple-saml-php` [16:47:01] Composer-enabled extensions require composer/installers which interacts with the install process and directs Composer to download the extension to /extensions instead of /vendor. extra.installer-name is a configuration field used by that package. [16:47:04] Osnard: it sounds like it should be required to be present when needed, not always [16:47:10] CindyCicaleseWMF: that also makes more sense logically: the site depends on the extensions, mediawiki does not. [16:47:42] duesen: exactly [16:48:43] 10 minutes left. [16:48:48] ...the notion that mediawiki would be depending on extensions, or that extensions would require mediawiki as a library, was the reason I opposed composer for extension management in the past. It seems like this proposal avoids both. [16:48:53] Have I missed any unanswered questions? [16:50:59] AIUI the steps needed to standardize Composer are: codify a bunch of best practices already widely used (such as extensions having a composer.json which requires composer/installers); clarify some issues around extension.json / composer.json overlap (what should be duplicated? should we have a fake core package?); improve ExtensionRegistry behavior on requirement errors [16:51:04] There are a number of best practices that have evolved by those using composer to manage extensions. This RFC proposes, among other things, to document those best practices. It also differentiates between downloading extensions/skins (which can be done with composer) and enabling them (which should ONLY be done by MediaWiki extension registration with wfLoadExtension or wfLoadSkin). [16:51:30] Obviously if we do this, we need to make sure everything is composer installable presumably [16:51:53] Otherwise we continue with the problems we have currently - "I can install X but why can't I install Y with composer?" [16:52:08] Which we do get regularly enough in #mediawiki [16:52:34] Making it a *requirement* for extensions to be installable via composer is an entirely different proposal. [16:52:48] It currently is not a different proposal [16:52:49] Reedy: Maybe even some kind of automation for setting extensions up in packagist? [16:52:59] It's a dependent [16:53:00] I don't think it is. As Reedy says, this is currently a big support issue. [16:53:00] It says it will be required for WMF-developed extensions and thus a best practice [16:53:12] I believe the discussion on the RFC indicated that the "fake core package" would go away. [16:53:25] We can't document something as best practice, and then only have minority support [16:53:34] It's stupid [16:53:39] Agreed. [16:53:43] Have we also got anything that goes along with that to an install an extension.. You can't force users to use composer? [16:53:58] ie all extensions *should* support composer and normal git/tarball etc? [16:54:29] I know it will take time to ramp up composer support if changes are needed, and that's fine. But we document that it's Coming Soon(TM) etc [16:54:30] The other outstanding question on the RFC was what should happen when extension dependencies are not satisfied at runtime. Should you get a 500 or a user-friendly page indicating the cause of the failure. [16:54:46] Any proposal in here about who takes over ownership and maintenance of composer-merge-plugin? [16:55:07] Nikerabbit: It'd be nice, but it doesn't look like packagist's API supports it :( https://packagist.org/apidoc [16:55:08] The team I worked on when I wrote it was disbanded 5 years ago [16:55:15] It's mostly readonly [16:55:23] Maybe we can inspire upstream [16:55:30] IMO the next step for the RfC would be to write the draft Composer policy. It's easier to have a discussion based on that. Can we agree on this being a good direction overall (so people know they don't waste their time when writing it)? [16:56:09] tgr: +1 [16:56:18] Reedy: as an alternative, some process to do it, not unlike now where you don't even know where to request access [16:56:23] tgr: +1 [16:56:28] !bug 1 | Nikerabbit [16:56:28] Nikerabbit: https://bugzilla.wikimedia.org/show_bug?id=1 [16:57:07] ...we have a bot that generates bugzilla links? [16:57:21] Sadly. [16:57:25] * duesen checks the decade [16:57:35] It's still the '90s, don't worry. [16:57:40] CindyCicaleseWMF: ExtensionRegistry already throws an exception when dependencies are missing. There are also existing error page handlers. Using one here seems fine indeed and is easy to do right now. It cannot however be a skinned page given it can't init MW. It would be akin to the "No LocalSettings.php" custom error page we have. [16:58:06] Krinkle: agreed [16:58:20] Krinkle: We're just about to add "suggests" key alongside "requires" which could be dressed up nicely, potentially. [16:59:13] Any last words? [17:00:38] tgr: +1 [17:01:17] thak for chairing Krinkle! [17:01:21] #endmeeting [17:01:21] Meeting ended Wed Jun 3 17:01:21 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) [17:01:21] Minutes: https://tools.wmflabs.org/meetbot/wikimedia-office/2020/wikimedia-office.2020-06-03-16.04.html [17:01:21] Minutes (text): https://tools.wmflabs.org/meetbot/wikimedia-office/2020/wikimedia-office.2020-06-03-16.04.txt [17:01:21] Minutes (wiki): https://tools.wmflabs.org/meetbot/wikimedia-office/2020/wikimedia-office.2020-06-03-16.04.wiki [17:01:21] Log: https://tools.wmflabs.org/meetbot/wikimedia-office/2020/wikimedia-office.2020-06-03-16.04.log.html [17:01:23] LGTM [17:01:32] just missed getting that last word in [17:01:58] four words, compressed :D [17:02:41] i like to Be Bold about stretching the rules. ;)