[21:00:45] #startmeeting RFC meeting [21:00:46] Meeting started Tue Sep 11 21:00:45 2018 UTC and is due to finish in 60 minutes. The chair is kchapman. Information about MeetBot at http://wiki.debian.org/MeetBot. [21:00:46] Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. [21:00:46] The meeting name has been set to 'rfc_meeting' [21:00:53] o/ [21:00:54] whee! [21:00:59] #topic Create a proper command-line runner for MediaWiki maintenance tasks https://phabricator.wikimedia.org/T99268 [21:01:02] hey [21:01:16] #link https://phabricator.wikimedia.org/T99268 [21:01:33] anyone else here for this RFC meeting? [21:02:23] #link https://www.mediawiki.org/wiki/Requests_for_comment/Proper_command-line_runner_for_maintenance_tasks [21:02:47] just a reminder the #link, #info #action tags are your friends for things getting into the minutes [21:02:54] hello! [21:03:02] I think I addressed all the feedback on Phabricator, but not everything on-wiki yet [21:03:10] So...what do people think? [21:03:19] [21:04:02] i like it :D [21:04:28] my main question is about out of tree scripts, which happen during one-off maintenance, there was some talk in phab on that which i think looks ok [21:04:58] I just updated the wiki page with that, my idea was to have an --extra-include=~/foo.php [21:05:19] so the foo.php would include a registry entry and a class, or just a class? [21:05:21] and foo.php will be included before the autoloader loads the maintenance script in tree, so it'll mask it [21:05:25] just a class [21:05:28] nice [21:05:36] that sounds feasible [21:05:43] Is there a documented reason why don't we just build this on top of Symfony's Console stuff? [21:05:46] e.g. php maintenance.php --extra-include=~/myupdate.php update [21:06:01] #info --extra-include=~/foo.php proposed for helping set up one-off scripts, looks good for my concerns [21:06:31] ooh good question. i haven't looked at symfony bits... [21:06:37] so this would also allow running local scripts that are not in the main repo? [21:06:55] and overriding ones that are in the main repo with local versions? [21:06:57] Nikerabbit: regardless of whether we use symfony or not, I think we still need a registry [21:06:58] SMalyshev: right. sounds scary, but it's better than trying the same thing pasting into eval.php ;) [21:07:03] SMalyshev: yes, with the --extra-include feature [21:07:44] Nikerabbit: and once we have MaintenanceTask taken out of Maintenance, I don't think it would be too hard for someone else to have it be backed by symfony/console instead of our stuff [21:07:50] brion: not scary for me, I did such things on terbium (with some effort) to test/run some maintenance stuff. Making it easier (and less error prone) is good for me :) [21:07:57] yay \o/ [21:08:27] legoktm: ok, include sounds a bit like just addition, but if it's also working as override that's great [21:09:11] so where the registry would live? In main maintainer super-class? [21:09:22] e.g. maintenance.php or something? [21:09:43] legoktm: the part I'm thinking is that in Symfony defines an interface for "Commands" that are registered with "Application" (which would be the registry I guess?) [21:10:06] SMalyshev: probably maintenance.php, and extensions would be able to add stuff via extension.json [21:10:15] I'm not saying that we should use symfony, just the general observation that we like to build our own solutions (sometimes with good justifications) [21:10:24] +1 for easy extension.josn extending :) [21:11:12] #info (looking at Symfony's console application framework, may be relevant) [21:11:24] #info make sure the registry is extendable from extension.json easily [21:11:58] yeah i've spent enough of my life typing "mwscript extensions/TimedMediaHandler/maintenance/requeueTranscodes.php" :D [21:12:53] legoktm: quick question on the arg --include model -- if i'm adding a new script I guess I need to get in the registry? [21:13:12] we should have some discovery mechanisms built-in then (instead of autocompletion when running it from shell :) [21:13:17] Nikerabbit: I think in this case we're mostly mirroring the architectural setup that symfony uses (MaintenanceTask==Command, registry/maintenance.php==Application), which I think is putting us in a better position to eventually use it if we wanted to [21:13:21] ah class name matching... [21:13:24] that's workable :D [21:13:38] brion: I wrote "If --extra-include is provided, then the name of the script can be a class name (verified with instanceof MaintenanceTask) for one-off scripts that aren't in the registry already. " for one-off hacks that are never supposed to be in-tree [21:13:47] yep that'll do fine [21:14:08] #info --extra-includes model proposes using filename=classname to avoid needing a registry for out of tree one-offs, looks good [21:14:09] legoktm: fair enough [21:14:38] would nwscipt still be there and do the user-switching and wiki-argument and whatever it's doing? [21:14:45] (I'm also very much a huge fan of symfony/console :)) [21:15:16] brion: er, not filename. the "human name" [21:15:28] ah right [21:15:38] #info s/filename/human-readable command name/ [21:16:09] SMalyshev: user-switching? on the talk page I wrote that I think we'll be able to phase out mwscript once this is in place [21:16:30] doesn't it still have to run as www-data or whatever? [21:16:49] or have we removed enough filesystem access that that's not a problem? :D [21:17:28] hm, probably it does. [21:17:29] legoktm: I mean changing to user www-data and other stuff that's going on there [21:17:56] like handling --wiki [21:17:59] I'm pretty confident that we'll be able to remove/simplify the multiversion bits [21:18:02] but yeah i think it no longer has to route to a specific maintenance file yeah [21:18:10] confusingly --wiki= is also handled in core [21:18:14] oh yeah multiversion :D *shudder* [21:18:40] #info mwscript still needed for user-id switching and multiversion handling but will get simpler [21:18:45] is there a migration plan for this? Do we keep the existing script php files as simple wrappers for the real stuff for some time? [21:18:55] yeah I has been always confused which parts of --wiki is handled where so if that gets cleaned up it's great :) [21:19:22] Nikerabbit: "The maintenance/ file hierarchy will stay and users will continue to be able to execute maintenance scripts directly. (This guarantee should not extend to future maintenance tasks, as a way of gradually driving users to adopt the top-level entry-point.) And eventually the old entry points will be removed. " [21:19:44] "The classes that power maintenance scripts will be moved out of the entrypoints, probably into includes/Maintenance. This will probably happen gradually. The entry points will continue to function with some back-compat shim, but eventually will emit deprecation notices. " [21:19:45] we have a bunch of crons etc. using them and possibly also other services relying on current way, so I think we have to keep current way as BC wrapper [21:19:59] at least for a while [21:20:49] not sure if it might be of any help, but fwiw for the spicerack and cookbooks work recently done for the SRE automation we had similar problems to solve. I can share details if deemed useful (it's in Python) [21:21:06] legoktm: mhm right, sorry I missed that part. but it's good that we have time to update docs [21:22:17] I don't like breaking back-compat :) [21:24:30] i assume there'll be a suitable --help output listing all commands... [21:24:35] volans: I'll take a look at those when I get the chance :) [21:24:39] ...do we need to add descriptions to the registry? [21:25:08] don't we already have those in the current scripts? can just reuse those, no? [21:25:09] so maintenance scripts already have $this->addDescription(...) [21:25:33] and yes, my plan was to have no arguments or --help output a list of available commands [21:25:35] ah right :D [21:25:47] have to load them all and query their descriptions without launching them...? [21:25:49] though I want to split the commands into two groups, those for sysadmins to run, and those for developers to run [21:26:01] legoktm: feel free to ping me, I can give you the TL;DR, we basically allow to list the tree of available cookbooks, run a single one of them or open an interactive menu starting from any point in the tree [21:26:28] and by default we'd only show sysadmin scripts in the list [21:26:54] brion: maybe, or it could be moved to a public static property or something in the migration to MaintenanceTask [21:27:06] I wonder if we could hook up bash autocompletion to it eventually... [21:27:13] statics are handy, don't have to instantiate [21:27:42] #info maintenance.php will by default list all sysadmin commands that can be run, including a description of each [21:28:04] #info we might want to refactor how description is stored so the MaintenanceTask class doesn't have to be instantiated just for the description [21:28:34] legoktm: given there are 150 of them or so, do we need some kind of grouping to make it easier to find what one wants? [21:29:45] there are 238 *.php files in core's maintenance/ [21:30:01] I wonder if anyone has requested to have those descriptions and other help texts i18ned... quite a few software actually does that [21:30:17] yeah so if running maintenance.php produces a 238-line list, it can be a bit overwhelming [21:30:22] ...in case that part is getting refactored at some point [21:30:41] my feeling is that about half are for developers or update.php related [21:31:40] #info consider i18ning script descriptions and other help (per Nikerabbit) [21:32:12] i tend to see hierarchical commands in a lot of complex CLI tools like git, cargo, etc [21:32:26] might be wise to consider grouping things in multi levels that way [21:32:57] Nikerabbit suggested that on the talk page, I'm a bit skeptical of whether we'll need it [21:33:11] legoktm: sorry, I feel like I only have ideas that expand the scope of work to do :D [21:33:15] :D [21:33:32] I'd like to do the first split of sysadmin vs developer, and then see whether further subcommands would make sense [21:33:47] we could also have "php maintenance.php --grep whatever" to search the command descriptions ;) [21:34:09] honestly i tend to do "ls maintenance | grep -i dump" and such :P [21:34:11] that is mostly covered by php maintenance.php | grep whatever :) [21:34:14] lol [21:34:29] it's the unix way :D [21:35:48] we have some scripts like rebuildall.php that are just wrappers around other child scripts. instead I'd rather see one rebuild script with options like --table=recentchanges (from a CLI perspective, on the PHP side it logically might make sense to be multiple classes) [21:38:42] good ol' tech debt refactoring :D [21:38:45] I think we have an opportunity here to reconsider some of the scripts like that, but on the other hand we probably just want to convert everything quickly to the new system for consistency [21:39:54] I see no particular reason that those two things would have to happen together [21:40:40] 20 minutes left legoktm do you need anything else from this discussion? Others, do you have any additional questions or comments? [21:40:57] one of the advantages of this is that we're decoupling the PHP class structure from the CLI interface, so we can refactor as much as we want in PHP, and easily continue to support old CLI names [21:42:45] kchapman: I think we're on track to go to a last call after this meeting? brion? [21:42:55] +2 [21:43:03] or at least +1 :D looking good [21:44:06] I'm going to add the i18n part on the wiki page after the meeting is over, since the main inspiration of this was to make it easier/friendlier for sysadmins, and i18n should be straightforward and in service of that goal [21:46:09] legoktm okay great. TechCom will likely to discuss this RFC again tomorrow. (I say likely since usually there is a week lag since the IRC meetings are normally after the reg TechCom meeting) [21:46:23] * legoktm nods [21:46:38] legoktm did you want to wrap up early? [21:47:14] seems like we've run out of discussion topics :) [21:47:21] :D [21:47:35] +1 for sleep (for me :) [21:48:11] no reason to waste time then! [21:48:18] #endmeeting [21:48:18] Meeting ended Tue Sep 11 21:48:18 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) [21:48:18] Minutes: https://tools.wmflabs.org/meetbot/wikimedia-office/2018/wikimedia-office.2018-09-11-21.00.html [21:48:18] Minutes (text): https://tools.wmflabs.org/meetbot/wikimedia-office/2018/wikimedia-office.2018-09-11-21.00.txt [21:48:18] Minutes (wiki): https://tools.wmflabs.org/meetbot/wikimedia-office/2018/wikimedia-office.2018-09-11-21.00.wiki [21:48:18] Log: https://tools.wmflabs.org/meetbot/wikimedia-office/2018/wikimedia-office.2018-09-11-21.00.log.html [21:49:19] thanks everyone :) [21:49:19] \o/ [21:49:34] thank you for picking it up legoktm :D [21:50:10] and thanks kchapman !