[12:29:59] anomie: i'd appreciate another set of eyes on the patches on https://phabricator.wikimedia.org/T166631 (and on https://gerrit.wikimedia.org/r/356190). Krinkle is reviewing them, but we've both missed the problems before. [13:16:32] legoktm: https://gerrit.wikimedia.org/r/#/c/352191/ and https://gerrit.wikimedia.org/r/#/c/351926/ would be nice. https://gerrit.wikimedia.org/r/#/c/355825/ has a large number of extension patches waiting to be submitted on a quick "good idea/bad idea" response. [13:27:21] MatmaRex: I flagged https://gerrit.wikimedia.org/r/#/c/356369/ in code review on the original patch (PS19), I guess my comment got missed somewhere in the shuffle. You flagged https://gerrit.wikimedia.org/r/#/c/356371/ in PS19. [13:37:38] anomie: hm, yeah. thanks. [13:40:30] We all missed the 'ready' not being a state thing. [15:16:47] Krinkle: anomie: care to look at some extra patches? i found a few other less important / less broken issues. https://gerrit.wikimedia.org/r/#/q/status:open+project:oojs/ui+branch:master+topic:whoops [15:42:30] brion: Did we ever decide on whether we're going to put both the comment_id and actor_id references in the same auxiliary revision table versus using two separate auxiliary revision tables? [16:07:48] anomie: I think we said separate tables [16:07:53] ok [16:08:03] Then in future they go into the main table [17:17:34] We should drop CologneBlue and Modern from the tarball releases. Bundled stuff is supposed to be our best extensions to MW. Those two skins are ancient and barely maintained. Lots of new features are routinely broken in them because few developers use them. [17:21:13] bundle Timeless instead :o [17:22:00] :) [17:22:16] RainbowSprinkles: +1 to dropping Modern and CologneBlue from the tarball. [17:22:28] MatmaRex: I'd be down with that [17:22:50] Timeless is sick as fuck [17:25:00] Prolly don't want to sneak these in with 1.29.0, but in master I think we could :) [17:25:11] Er, make-release's master [17:26:24] RainbowSprinkles: Might be nice for it to pass a security review first. [17:26:52] That too [17:26:53] I would say "It's a skin! What could it possibly do to break security?" but then I've seen MediaWiki. [17:26:57] I meant for the dropping the other skins [17:27:13] Oh, right. Eh. 1.29.0-rc0 isn't out yet. ;-) [17:34:27] I'm doing an rc.0 today [17:34:38] I'm calling it: 1.29.0-rc.0: The passive aggressive release [17:38:20] James_F: RainbowSprinkles: it already had a security review [17:38:44] Ah good! [18:08:06] bd808: hey - can you tell me what the process is for hiding db tables from the labs replica? and how about individual columns? [18:17:45] DanielK_WMDE: For one, when you add new tables or columns the #schema-change Phab task is supposed to say whether the new columns/tables should be visible in Labs or not. IIRC the default is "not visible". [18:18:41] If it's something already there that needs hiding, I suppose you'd file a Phab task about that too. [18:24:58] Speaking of the Labs replica, is there a task for "Figure out if 'we' want to provide an `old_revision` view and the like, since MCR and related work will be changing many schemas"? [18:32:01] anomie: there was discussion about compat views. i don't think we have a separate ticket. probably should. [18:32:42] anomie: this is about hiding a table that is already there, and which we want to remove soon-ish. Before we kill it from the production cluster, we want to hide it from labs. [18:32:52] yea, some phab task somewhere... i was hoping for more details :) [18:40:38] DanielK_WMDE: My first guess based on vaguely similar tasks would be to tag it with #DBA, #Labs and/or #Labs-Infrastructure. [18:43:27] anomie: thanks [18:44:55] anomie: oh, btw - do you think the +2 for my ApiErrorReporter patch should come from the WIkidata team? Or are you looking for confirmation from other core devs? [18:44:57] DanielK_WMDE: its *magic* to me. I know there is a python script that gets manually run to setup part of the replication. I don't really know how the sanitizing parts work at all. I /think/ jcrespo takes care of most of that work these days. [18:45:39] bd808: hahaha! Another instance of "don't touch it, nobody remembers how it works"? [18:45:40] DanielK_WMDE: I was expecting Wikidata team reviews. [18:46:01] DanielK_WMDE: I don't think its that spooky :) [18:46:05] anomie: ok, thanks! I'll poke Thiemo about it, then. [18:47:02] bd808: what kind of latency should we expect for such a task? [18:48:23] If it's a DBA thing I have no idea really. If my team takes care of it we can probably get someone to look into it in a few days. Changing the schema should have a reasonable announce window for tool maintainers though. [18:50:11] sanitarium [18:50:17] We add tables and such to it [18:50:20] yeah. that [18:50:25] It's not black magic ;-) [18:50:37] bd808: yes, of course. we plan to drop a table. The idea is: announce this week, hide it from the labs replica of test.wikidata in four weeks, and hide it from the labs replica of wikidatawiki on six weeks. [18:51:01] then give it a few more weeks before we stop using it in production. [18:51:13] It's in puppet somewhere [18:51:14] then stop using it, and request dropping the table "whenever later". [18:51:22] https://github.com/wikimedia/puppet/blob/256751456a8a543e7d940443fa336e2dcb70dccb/modules/role/manifests/mariadb/sanitarium2.pp [18:52:27] filtered_tables.txt [18:52:28] :) [18:52:30] ok, thank you! [18:53:00] You filter tables and fields from tables there [18:53:26] Ah, have to list all fields, not whole tables. But yeah [18:53:28] All there :) [18:59:16] \o/ [19:04:50] anomie: are you around? I wanted to ask a couple of questions about mediawiki API, if you have a minute [19:05:18] SMalyshev: I have about 5 minutes now, then I'll be back 5 or 10 minutes after that. [19:06:15] anomie: ok, I'll just start then :) it's mostly about generators. First thing: for some generators, titles are given as "titles" parameter, e.g. links [19:06:39] for others, it's module's parameter, e.g. for categorymembers it's gcmtitle [19:06:58] so the question is what is the reason for this difference? [19:07:13] A prop module used as a generator uses the ApiPageSet, i.e. titles, pageids, or revids. A list module used as a generator uses whatever it normally uses. [19:07:31] anomie: ah, so it's prop vs list module, I see [19:08:03] second one then: if I use categorymembers as generator, cmprop seems to be ignored - i.e. I can not get back type [19:08:20] it works when used as list, but not as generator [19:08:38] is there a way to get this from a generator? [19:09:36] Generators mostly don't output anything except the generated pages. generator=search and generator=prefixsearch are exceptions, they give an 'index' property because people kept begging for it. [19:11:31] anomie: I've always gotten the impression that "generators are probably not what you want to use" ;-) [19:11:39] anomie: but you can do pageprops, right? so cmprop does not work because it is specific for categories? [19:12:39] RainbowSprinkles: well, in my case generator is useful because it creates an unified interface and common output format [19:12:46] :) [19:13:04] go generator for links and generator for categories look very similar both on input and output [19:13:38] except for limit parameter which everybody has separate name for even though it does exactly the same (annoying) [19:20:22] SMalyshev: (continuing) A danger with having generators return data as well is that you could easily wind up with collisions that make ApiResult throw a RuntimeException. Then also you have to consider what happens if ApiPageSet is told to resolve redirects, or if the generator is generating revids rather than pages. [19:20:55] RainbowSprinkles: I use generators a lot in my bot. It all depends on what you're doing. [19:21:16] Lemme rephrase: people seem often confused by them [19:21:34] SMalyshev: gcmprop doesn't work because generators don't. prop=pageprops works fine because it's running as a prop module, not as a generator. [19:23:06] anomie: wait, now I'm confused. So if I run generator=categorymembers I can apply prop=pageprops to it [19:23:18] but not cmprop [19:23:23] Yeah, each module has its own limit parameter, there's no global limit. That's occasionally useful (particularly to give a low limit for the generator and a high limit for all the prop modules), but it can be annoying to have to specify them all. [19:23:57] anomie: is there any chance for introducing global limit parameter (i.e., if I make a patch, would it be accepted?) [19:24:39] something along the lines "if there's limit=max treat it as cmlimit=max"? [19:24:59] SMalyshev: Yes, that's how generators work. generator=categorymembers will produce a bunch of titles which will be used by prop=pageprops to return page properties for each of those titles. Since the hidden category flag is a page property, that might even do exactly what you're looking for. [19:25:55] anomie: aha, thanks, I start to get it, so I can get out of generator only something accessible to prop module, but information specific to generating module does not pass through [19:26:46] SMalyshev: ... Maybe. If it makes the code complicated or if it only half works, then I'll probably -1 it. If it just automagically works without having to alter every module, I'd probably merge. Do consider what should happen if someone specifies both limit=20 and cmlimit=30. [19:27:12] anomie: module limit probably should win [19:27:25] That's what I'd do, yeah. [19:27:27] i.e. cmlimit in this case [19:27:52] ok, I'll try to make a patch then (not sure if it's easy, but worth trying) [19:28:20] anomie: as for props modules, the prop module just gets a title from generator, nothing else? [19:28:31] The main thing I'd be worried about is if say prop=coordinates didn't use your new parameter because no one happened to getting around to updating the GeoData extension. [19:29:32] anomie: well, it depends on how generalized this thing is. I am hoping not every extension does its own limits so maybe there's common code [19:29:44] SMalyshev: A prop module gets the list of titles to work on from an ApiPageSet. The ApiPageSet can be populated by specifying titles/pageids/revids, or by using a generator. [19:30:46] anomie: I see. so no way to pass any info except page identity. OK, makes sense (though makes my life a bit harder) [19:33:50] Well, it is possible for a module to generate revids instead of pageids or titles. Most prop modules just convert the revid into pageid though, but it can be usefully used for something like generator=recentchanges&grcgeneraterevisions=1&prop=revisions [19:37:29] so I guess there's no way to get to something like cmprop from generator (unless one makes new prop module)... [19:38:00] specifically, gcmprop=type, the rest is covered [19:39:59] I believe you can determine the type easily enough client side: If the page is in namespace 6 it's 'file', if it's in namespace 14 it's 'subcat', and everything else is 'page'. [19:41:08] https://phabricator.wikimedia.org/source/mediawiki/browse/master/includes/deferred/LinksUpdate.php;6c7d4481d71f7d4d1e6172ad2da68e61883fc0e5$580-586 [19:41:50] anomie: good point, ns is part of the standard result. numbers are a bit lowlevel but could work probably, thanks [21:02:04] RFC meeting starting now in #wikimedia-office: PostgreSQL schema change for consistency with MySQL