[00:00:03] <qgil>	 alright, thank you very much for this conversation!
[00:00:04] <spagewmf>	 No sleep 'til Hammersmith  :)
[00:00:18] <spagewmf>	 good luck and thanks!
[00:00:19] <quiddity>	 Is there anything else you want/need help with, in the meantime, from us?
[00:00:27] <qgil>	 and for all your feedback, testing, help...
[00:01:13] <qgil>	 quiddity, I will need help reviewing the blog post (that I haven't written yet)
[00:01:26] <qgil>	 I have been swimming in Phabricator so deep for so many months...
[00:01:28] <quiddity>	 okie
[00:01:44] <qgil>	 that I'm having difficulties to take two steps back and write a simple, informative thing
[00:01:49] <quiddity>	 I understand the feeling!
[00:01:50] <qgil>	 another perspective will be useful
[00:02:02] <qgil>	 thanks!
[00:02:06] <andre__>	 thanks everybody!
[15:46:09] <Elitre>	 hi everybody. the monthly VisualEditor office hour is going to take place here in 15 minutes or so.
[15:50:00] * James_F waves.
[15:59:56] <Elitre>	 so, welcome everybody to our monthly appointment with VisualEditor. As usual, the channel is logged, and logs are published shortly after the meeting ends on Meta. We're not just here to tell things, we're hear to listen! Please shoot your questions, I'll take note. While you type them, let's hear from our guest star, James.
[16:00:06] <James_F>	 Heya.
[16:00:37] * tommorris waves to James_F and listens.
[16:00:42] * marktraceur sits down with some popcorn
[16:00:45] <James_F>	 So, in the past month we've squashed quite a few bugs and made some general improvements, but I imagine the thing that most people care about is that we've got table structure editing now available.
[16:01:14] <Elitre>	 tommorris: we're glad to have you around! marktraceur: please share?
[16:01:21] <marktraceur>	 James_F: When will the public Parsoid API be available?
[16:01:31] * marktraceur hands Elitre the bag of popcorn
[16:01:36] <James_F>	 This means you can insert new tables, add and remove columns and rows, merge and unmerge cells, add, edit and remove table captions, and mark cells (and rows/columns) as "headers" or not.
[16:01:49] <Finnegan>	 oh my god tables. You just made my day.
[16:02:00] <James_F>	 This covers "most" of the functionality we want to deliver right now.
[16:02:07] <James_F>	 There are still some bits and pieces.
[16:02:11] <marktraceur>	 Finnegan: LOSE YOUR SHIT. EDIT A FRIGGIN TABLE.
[16:02:26] <James_F>	 For instance, drag-and-drop of table columns and rows would be really nice to have.
[16:02:36] <James_F>	 Also, making a table as having sortable columns or not.
[16:03:04] <James_F>	 The current interface is pretty janky and we're not happy with it - there are 5(!) places to do different kinds of control on a table, so we'll be addressing that.
[16:03:25] <James_F>	 Also, interacting with tables made up with templates as well as "real" table cells is a bit messy and confusing, and I want that to be nicer.
[16:03:28] <James_F>	 But it's a start. :-)
[16:03:45] <Elitre>	 Finnegan: you hadn't heard about tables yet? /me shamelessly plugs https://meta.wikimedia.org/wiki/VisualEditor/Newsletter ...
[16:03:52] * James_F grins at Elitre.
[16:04:28] <James_F>	 There are some table-related items that we're not currently planning to do soon – most visibly, adding or changing custom CSS on tables, rows and cells.
[16:05:03] * Finnegan signs up, wonders if she gets a keychain or something
[16:05:10] <James_F>	 I know that's used by a few very experienced users, but it's also widely abused to make hideous tables and I'm not sure how we would give people the tools without engendering a bunch of mess.
[16:05:20] <marktraceur>	 Finnegan: Your free toaster is in the mail
[16:05:23] <James_F>	 (Cf. the famous Jurassic Park quote in reverse.)
[16:06:09] <James_F>	 That's about it for me to spiel about.
[16:06:35] <James_F>	 As always, I'd recommend people who are keen to see what we're working out look at our roadmap ( https://www.mediawiki.org/wiki/VisualEditor/Roadmap ) and complain / praise as you see fit.
[16:07:06] <James_F>	 So, I guess it's time for questions.
[16:07:13] <James_F>	 marktraceur: The Parsoid API…
[16:08:04] <James_F>	 I'm not the expert on the Parsoid team's work – they are! – but you can see their detailed worklist for the quarter here: https://www.mediawiki.org/wiki/Parsoid/Roadmap/2014_15/Q2_tasks
[16:08:18] <marktraceur>	 Well, subbu's not here, and cscott_away looks away
[16:08:42] <James_F>	 Integrating Parsoid with "RESTbase" (the API system that the Services team has built) is currently underway.
[16:09:22] <James_F>	 I don't know exactly where they are with that; I know they have an ambition to complete it by the end of this quarter, but Q2 is always very disrupted by holidays and so aspirations and reality sometimes diverge.
[16:09:33] <marktraceur>	 Yeah
[16:09:37] <marktraceur>	 I'll bug -parsoid about it
[16:09:52] * James_F nods.
[16:10:10] <James_F>	 marktraceur: For what particular things are you looking?
[16:10:45] <marktraceur>	 James_F: Well, I'd like a way to go back and forth between wikitext and HTML for other clients. I know I could use the VE-specific API for now, but I've heard that's not to be trusted as stable
[16:11:09] <James_F>	 marktraceur: What's wrong with the existing Parsoid API?
[16:11:17] <marktraceur>	 e.g. my editDescriptions.js script on commons currently parses {{information}} templates on the frontend, and that's stupid
[16:11:22] <marktraceur>	 James_F: It's just not public AFAIK
[16:11:47] <James_F>	 marktraceur: It isn't? parsoid-lb.eqiad.wikimedia.org works fine?
[16:12:12] <YuviPanda>	 that's an internal URL :)
[16:12:12] <marktraceur>	 James_F: Ah, well, maybe we've had this conversation before, I should write it down this time
[16:12:21] <marktraceur>	 Oh, ty YuviPanda
[16:12:23] <YuviPanda>	 it's not 'public public' anywhere.
[16:12:28] <James_F>	 YuviPanda: Fair.
[16:12:39] <James_F>	 BTW, the Roadmap I linked above obviously is turning into reality right now through Bugzilla. As of tomorrow evening Bugzilla is getting shut down and all the content will be moved to Phabricator. This is going to be much easier for non-techies to see, interact and suggest changes.
[16:12:41] <marktraceur>	 So yeah, user JS on-wiki wouldn't be able to access that at all
[16:12:48] <YuviPanda>	 James_F: also, no https on that URL (since it's internal)
[16:12:51] <James_F>	 marktraceur: Err. Yes they can.
[16:12:55] <YuviPanda>	 it was made public as a hack, IIRC.
[16:13:04] <YuviPanda>	 marktraceur: it *can*, just shouldn't
[16:13:10] <marktraceur>	 Shouldn't, whatever
[16:13:43] <James_F>	 However, the switch-over to Phabricator will probably be a bit messy, so exact pointers to where to go to see that week's work, or what our current high-priority bugs are, will change (and haven't been worked out yet).
[16:14:26] <James_F>	 YuviPanda: It'd be nice for the "officially official" system to be in place, yes, but it's been working fine for nearly two years now.
[16:14:52] <YuviPanda>	 well, except no https :) and technically it can go away at any time when someone messes with LVS on the ops side.
[16:15:02] <James_F>	 marktraceur: https://www.mediawiki.org/wiki/Parsoid/API documents the current API.
[16:15:10] <James_F>	 YuviPanda: That's also the case when Ops break enwiki. :-D
[16:15:25] <YuviPanda>	 true, but when we break enwiki, we try to fix it as well :)
[16:15:44] <tommorris>	 reading the Parsoid roadmap, I see PhantomJS and print CSS for rendering. does this mean we might have Wikipedia having a clean, beautiful print stylesheet, and the export-as-PDF option on Wikipedia running on the same stylesheet? because currently, the printable version of Wikipedia is uuugly.
[16:15:44] <James_F>	 YuviPanda: Given the number of alarms on the Parsoid service, you'll hear about breaking it too. :-)
[16:16:04] <James_F>	 tommorris: Hey. Yes!
[16:16:15] <tommorris>	 given WebFonts exists, Wikipedia print outs could be non-ugly.
[16:16:26] <James_F>	 Well, we're not planning to do WebFonts just yet.
[16:16:38] <tommorris>	 rather than the poorly-laid-out Times New Roman monstrosities they currently are.
[16:16:58] <James_F>	 But e.g. https://en.wikipedia.org/w/index.php?title=Special:Book&bookcmd=rendering&return_to=Main+Page&collection_id=3446fb09b1da43719839b81e712e28c3cd1876b5&writer=rdf2latex&is_cached=1 is now created using Parsoid HTML via PhantomJS
[16:17:29] <James_F>	 Yeah, I think the current print stylesheet needs a lot of love to make it useful.
[16:17:40] <tommorris>	 is there any chance of getting custom print stylesheets on different WMF projects? with my wikinews hat on, it'd be great if we could have wikinews print-outs looking beautiful.
[16:17:53] <James_F>	 cscott_away is the expert on this.
[16:18:04] <James_F>	 I /think/ the print stylesheet is actually customiseable per-wiki.
[16:18:12] <tommorris>	 cool. will look into it.
[16:18:15] <James_F>	 But I've never fiddled with that side of things, so I'm not sure.
[16:18:33] <James_F>	 Improving from the technical side is now mostly done, I think.
[16:18:46] <James_F>	 So improvements on the side of making it actually nice to use is probably next. :-)
[16:19:37] <Elitre>	 (re: Phabricator, please create your accounts there ASAP if you haven't done it so far! Instructions at https://www.mediawiki.org/wiki/Phabricator/Help .) Who else has questions for James? Or maybe someone wants to tell us their experience with VE?
[16:19:50] <James_F>	 https://en.wikipedia.org/w/index.php?title=Special:Book&bookcmd=download&collection_id=c528a0a9498eff39be6322f2487f9259b6b1af23&writer=rdf2latex&return_to=Oregon+Trail+Memorial+half+dollar is a better example of the current PDF output.
[16:20:59] <James_F>	 Oh, yes, and during the switch-over to Phabricator there will be special arrangements for emergency bugs.
[16:21:20] <James_F>	 If you go to bugzilla.wikimedia.org there will be instructions.
[16:21:31] <James_F>	 If you're in IRC, I'll be around, as always. :-)
[16:22:12] <tommorris>	 with the phabricator changeover, what's the current process for filing bugs?
[16:22:32] <tommorris>	 the link that Elitre posted talks about "Creating a task". Tasks == bugs?
[16:22:44] <James_F>	 tommorris: Current process is to go to https://bugzilla.wikimedia.org/enter_bug.cgi?product=VisualEditor
[16:22:58] <James_F>	 tommorris: Yeah, in Phabricator they're called tasks (which is more accurate).
[16:23:14] <James_F>	 E.g. "switch VisualEditor on for Catalan Wiktionary" (which I did yesterday) wasn't really a "bug".
[16:23:40] <tommorris>	 (yay for catalan wiktionary)
[16:23:59] * tommorris is watching Phabricator rollout carefully.
[16:24:13] <James_F>	 With Phabricator, it'll be something like https://phabricator.wikimedia.org/maniphest/task/create/?projects=PHID-PROJ-2tvh6ublfqvqgkgnclqr
[16:24:41] <James_F>	 (NOT that actual URL; that is for the Editing Team at large.)
[16:25:14] <James_F>	 Right now the VisualEditor project hasn't been created in Phabricator (that will be done as part of the import), so… no useful link, sorry.
[16:26:32] <gwicke>	 tommorris: I did experiment with print CSS improvements at https://en.wikipedia.org/wiki/User:GWicke/common.css; from my POV the biggest issue for pure-CSS solutions is currently relatively poor support for break-before/break-after to control page breaks
[16:27:17] <tommorris>	 well, JavaScript could be used given that the PDF rendering is done in PhantomJS. ;)
[16:27:25] <gwicke>	 we could resurrect something like https://github.com/booktype/BookJS
[16:27:34] <gwicke>	 currently that's pretty abandoned
[16:27:47] <James_F>	 tommorris: "Could" or "should"? ;-)
[16:28:06] <tommorris>	 Could.
[16:28:07] <gwicke>	 also, it will only work with PhantomJS 2 once that's released, as Phantom 1 is based off some old Safari version without CSS region support
[16:29:40] * James_F nods.
[16:29:51] <James_F>	 gwicke: Is PhantomJS 2 really going to be a success?
[16:30:06] <gwicke>	 btw, VE folks: I'm really happy about our new table editing powers!
[16:30:15] <James_F>	 gwicke: We've moved our testing infrastructure over to headless Chrome and Firefox, because…
[16:30:20] <James_F>	 Ha. Thanks, gwicke. :-)
[16:31:59] <James_F>	 So, any more questions?
[16:32:07] <gwicke>	 James_F: anything more modern and reliable than phantomjs 1 with print-to-pdf support should work
[16:32:47] * James_F nods.
[16:33:22] <James_F>	 gwicke: In your copious free time (and his), Krinkle can talk about getting headless Chrome in the cluster to work.
[16:33:58] <James_F>	 In unrelated news (but lots of people in the office are talking about it right now), today is the day we'll switch the English Wikipedia over to the new search system, completing this marathon of work.
[16:34:35] <James_F>	 Which will have some benefits for using VisualEditor – when you are prompted for a link, a template, a category or a file it'll be based on the better, new search system.
[16:34:46] <James_F>	 Not to steal their thunder or anything. :-)
[16:36:59] * tommorris shall test table editing later.
[16:37:27] <James_F>	 tommorris: Excellent. :-)
[16:37:56] <Elitre>	 any other VE questions? otherwise we could close this conversations earlier, I guess.
[16:40:36] <James_F>	 Seemingly no.
[16:40:50] <Elitre>	 well, I guess this is a goodbye then :) Please remember there won't be any appointment in December, but we'll be here again early in January.
[16:41:38] <Elitre>	 January 7th , 10pm UTC: http://www.timeanddate.com/worldclock/fixedtime.html?hour=22&min=00&sec=0&day=07&month=01&year=2015 mark your calendars if you want (it's a Wednesday again).
[16:41:41] <James_F>	 See you all!
[16:41:52] <tommorris>	 Keep on being awesome, James_F, Elitre et al.
[16:42:04] <Elitre>	 thanks everybody for coming! talk to you on wikis and beyond!
[16:42:14] <James_F>	 tommorris: Thanks. :-)
[21:00:55] <TimStarling>	 #startmeeting RFC meeting
[21:00:55] <wm-labs-meetbot>	 Meeting started Wed Nov 19 21:00:55 2014 UTC and is due to finish in 60 minutes.  The chair is TimStarling. Information about MeetBot at http://wiki.debian.org/MeetBot.
[21:00:55] <wm-labs-meetbot>	 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
[21:00:55] <wm-labs-meetbot>	 The meeting name has been set to 'rfc_meeting'
[21:01:28] <TimStarling>	 #topic Text extraction | RFC meeting | PLEASE SCHEDULE YOUR OFFICE HOURS: https://meta.wikimedia.org/wiki/IRC_office_hours | Please note: Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE) | Logs: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/
[21:02:16] * MaxSem waves
[21:02:58] <TimStarling>	 hi
[21:03:06] <TimStarling>	 #link https://www.mediawiki.org/wiki/Requests_for_comment/Text_extraction
[21:03:23] <TimStarling>	 what sorts of clients use the extracts API at the moment?
[21:04:20] <MaxSem>	 Lots of third-party reusers, also Hovercards
[21:04:56] <TimStarling>	 do you want to give us a general update?
[21:05:00] <Marybelle>	 I feel like Elasticsearch might be using it or planning to?
[21:05:27] <TimStarling>	 manybubbles|away: ^ ?
[21:05:27] <MaxSem>	 TimStarling, update on what?
[21:05:50] <manybubbles>	 sorry, catching up
[21:05:51] <MaxSem>	 Marybelle, no - it uses just HtmlFormatter, not actually the extracts themselves
[21:06:02] <Marybelle>	 Ah, okay.
[21:06:20] <TimStarling>	 the RFC was created in November 2013
[21:06:49] <manybubbles>	 TimStarling: CirrusSearch uses HtmlFormatter internally
[21:07:00] <TimStarling>	 has there been any work on it recently? are you planning on working on it?
[21:07:38] <MaxSem>	 yup - last time Sean wanted to avoid storing all this crap in page_props so I revised the RFC to have a separate table, potentially movable to a different db cluster
[21:08:16] <TimStarling>	 right, it would be quite big for page_props
[21:08:52] <TimStarling>	 but as for moving to another DB cluster, well, we should probably think about what to do about links tables generally at some point
[21:09:10] <MaxSem>	 so, considering the recent allegations of performance problems and increasing usage, I'd like to move forward with storing it in DB
[21:09:24] <TimStarling>	 I see ExtractFormatter is also an HtmlFormatter
[21:09:52] <MaxSem>	 yup
[21:10:24] <TimStarling>	 so the idea of storing it in the DB on page save is to limit latency, right?
[21:10:33] <MaxSem>	 yes
[21:10:42] <MaxSem>	 and to allow for batch querying
[21:11:05] <DanielK_WMDE>	 how about making the plain-text rendering part of the ParserOutput, and put it into the parser cache?
[21:11:07] <TimStarling>	 so there wouldn't be memcached in front?
[21:11:21] <Marybelle>	 I'm a bit confused about what's living in Extension:TextExtracts and what's living in core and how that might change in the future.
[21:11:44] <MaxSem>	 DanielK_WMDE, I also don't want parser cache misses:)
[21:11:53] <MaxSem>	 TimStarling, yes
[21:11:54] <Marybelle>	 DanielK_WMDE: There's parser cache discussion at https://www.mediawiki.org/wiki/Talk:Requests_for_comment/Text_extraction#Some_notes_35442
[21:12:09] <DanielK_WMDE>	 what's the hit rate of the parser cache anyway?
[21:12:44] <MaxSem>	 this graph has been broken in gdash for a while:P
[21:12:49] <TimStarling>	 nobody knows
[21:12:59] <DanielK_WMDE>	 yay
[21:13:11] <TimStarling>	 for a few months there was http://noc.wikimedia.org/cgi-bin/pcache-hit-rate.py
[21:13:20] <TimStarling>	 but it is a 404 now, along with all profiling
[21:13:46] <MaxSem>	 even if it misses in 1% of cases, parses can be really slow
[21:13:52] <Marybelle>	 I don't see how extracts could only be stored on page save unless we want to encourage null editing.
[21:14:10] <MaxSem>	 that's why extracts currently limits batch queries so strictly
[21:14:21] <Marybelle>	 Which is basically what I said on the talk page. If the extracting logic changes or templates change, doesn't that require a re-run of the extractor?
[21:14:25] <DanielK_WMDE>	 MaxSem: right, but if it's good enough for sending html to browsers, it should be good enough for extracts too, no?
[21:14:47] <MaxSem>	 for 1 page, sure. but I want batch queries
[21:15:00] <TimStarling>	 presumably the refreshLinks jobs will update the extracts after a template change
[21:15:00] <DanielK_WMDE>	 MaxSem: what's the use case?
[21:15:02] <MaxSem>	 because that's what users want
[21:15:05] <Marybelle>	 Batch read queries or batch write queries?
[21:15:13] <MaxSem>	 read
[21:15:57] <DanielK_WMDE>	 is it a good idea to dump large numbers of big text blobs in the main database?
[21:15:59] <Marybelle>	 TimStarling: And a maintenance script would cover extractor logic changes?
[21:16:08] <DanielK_WMDE>	 would be much nicer to use external store - or rather, an abstracted blob store
[21:16:21] <TimStarling>	 how big are extracts?
[21:16:29] <MaxSem>	 DanielK_WMDE, that's why the RFC mentions an optional external DB storage;)
[21:17:03] <TimStarling>	 Marybelle: there would have to be a new maintenance script to populate it initially from the parser cache
[21:17:15] <DanielK_WMDE>	 MaxSem: yea, but that needs to be actually *done* ;)
[21:17:20] <TimStarling>	 so that maintenance script could be reused for updates afterwards
[21:17:30] <MaxSem>	 or dynamically populate it from memcached as people request it
[21:17:53] <DanielK_WMDE>	 at a guess, i'd imagine the size of an extract is about 1/2 of full page html
[21:17:54] <TimStarling>	 then you still have latency issues
[21:17:57] <DanielK_WMDE>	 is that about right?
[21:18:15] <Marybelle>	 I thought extracts were like a single paragraph of text.
[21:18:17] <TimStarling>	 oh, there is an exchars parameter
[21:18:31] <TimStarling>	 so the user chooses
[21:18:55] <MaxSem>	 TimStarling, plaintext is 89k for Obama
[21:19:27] <TimStarling>	 so ExtractFormatter converts the whole article to text, and then the API query takes a leading substring of that?
[21:19:30] <Marybelle>	 https://www.mediawiki.org/w/api.php?action=help&modules=query%2Bextracts
[21:19:50] <TimStarling>	 that does sound like a job for ES then
[21:20:03] <TimStarling>	 I was thinking it would be just a few hundred bytes per row, that is a job for a new table
[21:20:14] <MaxSem>	 nah
[21:20:25] <TimStarling>	 but ES is insert-only
[21:20:40] <MaxSem>	 lots of people request intro only or a couple of sentences, but lots want full page extracts too:)
[21:21:00] <Marybelle>	 A full-page extract is like lynxing the page?
[21:21:10] <MaxSem>	 essentially
[21:21:14] <Marybelle>	 I thought extracts were only... extracts.
[21:21:34] <MaxSem>	 TextExtracts are ... extracts of text
[21:21:49] <TimStarling>	 maybe not a job for ES, probably best to keep insert-only semantics there
[21:22:03] <Marybelle>	 Right, but extract typically means a small piece of something ("a short passage taken from a piece of writing...")
[21:22:09] <TimStarling>	 but certainly worth adding the option of separate DB configuration
[21:22:12] <MaxSem>	 why? there's going to be a crapton of old versions
[21:23:15] <Marybelle>	 If not ExternalStorage and not a separate table, then where?
[21:23:17] <gwicke>	 MaxSem: I'd be happy to chat about how restbase could help here
[21:23:43] <TimStarling>	 in the DB configuration we have
[21:23:48] <TimStarling>	 	# ExtensionStore shard1 - initially for AFTv5
[21:23:48] <TimStarling>	 	'extension1' => array(
[21:23:49] <TimStarling>	 		'10.64.16.18' => 10, # db1029
[21:23:49] <TimStarling>	 		'10.64.16.20' => 20, # db1031 snapshot host
[21:23:49] <TimStarling>	 	),
[21:23:53] <Marybelle>	 So a separate DB for extracts only?
[21:24:09] <TimStarling>	 I guess it just grabs an "external" database connection and then uses its own tables?
[21:24:16] <MaxSem>	 yup
[21:24:30] <MaxSem>	 also, echo uses it
[21:24:54] <TimStarling>	 yeah, we should have thought of some other name for that, it will be confusing
[21:25:08] <Marybelle>	 ExtensionStore v. ExternalStorage, you mean?
[21:25:14] <TimStarling>	 yeah
[21:25:26] <TimStarling>	 well, there is LBFactory::newExternalLB()
[21:25:33] <MaxSem>	 lawl
[21:25:45] <Marybelle>	 I'm not sure I understand how ExtensionStore splits between clusters and between wikis.
[21:26:11] <TimStarling>	 Marybelle: I assume it uses LBFactory::newExternalLB() which has facilities for that
[21:26:11] <DanielK_WMDE>	 how does this fit with https://www.mediawiki.org/wiki/Requests_for_comment/DataStore
[21:26:17] <MaxSem>	 each wiki has its own DB on the extension1 cluster
[21:27:14] <TimStarling>	 the doc comment for getExternalLB() says
[21:27:18] <Marybelle>	 So over 800 additional DBs?
[21:27:21] <TimStarling>	 "Get a cached (tracked) load balancer for external storage"
[21:27:34] <MaxSem>	 Marybelle, we already have them
[21:27:52] <TimStarling>	 maybe one or the other interface should be renamed?
[21:27:53] <Marybelle>	 Sure, but the "# for AFTv5" comment kind of scared me.
[21:28:09] <legoktm>	 flow and echo both use extension1
[21:28:14] <MaxSem>	 DanielK_WMDE, DS is not related anymore because the proposed table is going to be more functional than DS would allow
[21:28:25] <Marybelle>	 Kind of off-topic, but... is extension1 replicated to Labs?
[21:28:27] <Marybelle>	 And dumps.wm.o?
[21:28:35] <TimStarling>	 is ExtensionStore a class or just a pattern?
[21:28:37] <legoktm>	 and bouncehandler is using the "wikishared" db on extension1 (I think)
[21:30:13] <legoktm>	 I don't think it's replicated to labs.
[21:30:51] <Marybelle>	 Does extension1 have documentation? Is it storing plaintext? gzipped text? Something else?
[21:30:59] <TimStarling>	 there's nothing called ExtensionStore in AFTv5, it just calls getExternalLB() directly, so I guess it is just a pattern
[21:31:07] <MaxSem>	 extension1 is a bunch of DBs
[21:31:25] <TimStarling>	 		// connect to external, aft-specific, cluster
[21:31:25] <TimStarling>	 		if ( $wgArticleFeedbackv5Cluster ) {
[21:31:25] <TimStarling>	 			return wfGetLBFactory()->getExternalLB( $wgArticleFeedbackv5Cluster, $wiki );
[21:31:25] <TimStarling>	 		} else {
[21:31:26] <TimStarling>	 			return parent::getLB( $wiki );
[21:31:26] <TimStarling>	 		}
[21:31:35] <MaxSem>	 it's storing whatever is there in its tables, Marybelle
[21:32:30] <Marybelle>	 Right, but I guess I'm wondering high-level whether text extracts would be directly queryable from Labs via the sql command-line utility.
[21:33:08] <TimStarling>	 getExternalLB() doesn't do sharding
[21:33:23] <TimStarling>	 ExternalStore does sharding on top of getExternalLB
[21:33:39] <TimStarling>	 not that you really need sharding for storing a few GB
[21:35:18] <TimStarling>	 db1029, which is the extension1 master, has 52GB used out of 1.4 TB in /a
[21:36:43] <TimStarling>	 I tried pinging sean, guess I should have done that half an hour ago
[21:36:56] <Marybelle>	 52GB is sounds kind of high... I wonder what that is.
[21:37:03] <TimStarling>	 heh
[21:37:29] <Marybelle>	 I mean, if/when Flow gets used, I don't think we'll have trouble filling the space.
[21:37:57] <TimStarling>	 flow could use ExternalStore
[21:38:22] <TimStarling>	 or do its own sharding
[21:39:01] <Marybelle>	 MaxSem: So regardless of where the extracts get stored, would ?action=purge or a null edit update the extracts?
[21:39:20] <MaxSem>	 null edit should
[21:39:32] <MaxSem>	 it'll hook into LinksUpdate
[21:39:36] <TimStarling>	 anyway, I am mostly wondering whether db1029 has spinning rust for storage, and if it has, whether that would meet your latency goals once it is fully populated
[21:39:52] <TimStarling>	 it only has 64GB of memory
[21:41:19] <MaxSem>	 compared to hundreds of ms for extraction from a large article, it should:P
[21:41:20] <quiddity>	 Would the database of extracts help solve for #Section extracts, or is that a separate issue?   (i.e. the primary Wiktionary use-case, and the secondary Hovercards/Navpopups use-cases)
[21:41:32] <MaxSem>	 it's a separate issue
[21:42:22] <MaxSem>	 it will help the poppus with improved latency though
[21:42:37] <TimStarling>	 I think it's fair to assume that you're going to need extra hardware for this
[21:42:45] <TimStarling>	 say, an extra slave in the external1 cluster
[21:43:03] <MaxSem>	 hm, is it so overloaded already?
[21:43:49] <TimStarling>	 you are assuming text extract requests will have a negligible request rate compared to what it serves already?
[21:44:22] <MaxSem>	 depends on how gret is the load and how much it will grow
[21:44:29] <MaxSem>	 s/gret/geat/
[21:44:52] <TimStarling>	 it's not simple to tell how much load you will put on it
[21:45:15] <MaxSem>	 11M requests/day currently
[21:45:44] <MaxSem>	 grew like x3 last year
[21:45:50] <TimStarling>	 so ~130 per second
[21:45:59] <TimStarling>	 well, say double at peak
[21:46:25] <TimStarling>	 but it depends very sensitively on data set size
[21:46:27] <Marybelle>	 Time to serve request is directly correlated with article text length?
[21:46:50] <TimStarling>	 if most of the load is search engine crawlers, which hit every page randomly
[21:47:02] <quiddity>	 and keep in mind that Hovercards may eventually become much more widely used.
[21:47:03] <TimStarling>	 then that puts a lot of pressure on DB memory
[21:47:12] <MaxSem>	 search engines aren't using this api
[21:47:37] <Marybelle>	 I think I read somewhere that CirrusSearch wants to use extracts in results.
[21:47:58] <Marybelle>	 manybubbles: Is that right?
[21:48:27] <MaxSem>	 he's in a meeting. the same as me:P
[21:48:31] <quiddity>	 Is this the appropriate time to discuss #Section and cross-wiki extracts?   We'd love to be able to get a [[wikt:rust#Noun_3]] link within Wikipedia, to work with Hovercards/Navpopups... :)
[21:48:31] <TimStarling>	 db1031 is serving 3700 selects per second
[21:48:39] <Marybelle>	 Yes, an RFC meeting. We're all here. ;-)
[21:49:03] <TimStarling>	 so an extra 200 per second is not a big deal from that perspective
[21:49:12] <MaxSem>	 :)
[21:49:44] <TimStarling>	 as long as you don't use too much memory it will probably work
[21:49:48] <Marybelle>	 quiddity: Cross-wiki extracts would be covered by hitting a foreign API, I think.
[21:49:55] <Marybelle>	 Section extracts... sounds like work.
[21:50:24] <quiddity>	 navpopups currently handles it.  but hovercards doesn't.  :<
[21:50:57] <TimStarling>	 cross-wiki extracts should be easy enough, you could just do that on the client side, right?
[21:51:13] <TimStarling>	 what Marybelle said
[21:51:43] <Marybelle>	 quiddity raises a good point about grabbing an arbitrary piece of a full extract.
[21:51:46] <TimStarling>	 section extracts, well, the parser annotates section locations with HTML comments and later fills them in with section edit links
[21:52:01] <Marybelle>	 Store the full extract as a JSON blob? :-)
[21:52:08] <TimStarling>	 it would be easy enough to leave some sort of annotation in for HTMLFormatter to find
[21:52:14] <TimStarling>	 if there isn't enough information in there already
[21:53:06] <TimStarling>	 I think a list of offset and length would match the data best
[21:53:37] <Marybelle>	 Not sure what you mean.
[21:53:44] <Marybelle>	 Storing offsets to mark sections?
[21:53:50] <TimStarling>	 yeah
[21:54:21] <TimStarling>	 {'text': '....', 'sections': {0: [0,1000], 1: [1001, 300]}}
[21:54:25] <Marybelle>	 Would that work with subsections?
[21:54:30] <MaxSem>	 offsets wouldn't pass through DOM transformations
[21:54:57] <TimStarling>	 true
[21:55:10] <Marybelle>	 I was suggesting splitting 'text'.
[21:55:12] <aude>	 once again, miss most of the meeting :(
[21:55:15] <Marybelle>	 And concatenating for the full extract.
[21:55:25] <TimStarling>	 you would have the HTML annotation to go through the DOM transformation
[21:55:54] <TimStarling>	 and then store the offsets as you serialize to text
[21:56:22] <TimStarling>	 and yes, it trivially works with subsections
[21:56:34] <Marybelle>	 1.1, 1.2, etc.?
[21:56:52] <TimStarling>	 yes
[21:57:44] <TimStarling>	 well, those are just TOC headings, but yes anyway
[21:58:21] <TimStarling>	 range annotation is basically a superset of nesting
[21:59:05] <TimStarling>	 maybe you could do it with nesting, but Parser::extractSections thinks in terms of range annotation
[21:59:28] <DanielK_WMDE>	 i'd rather split the text and store it as a serialized array of sections
[21:59:35] <MaxSem>	 source range, these lose any meaning after parsing
[21:59:58] <TimStarling>	 no, destination range
[22:00:02] <DanielK_WMDE>	 possibly even store the most used bits (the intro?) separately
[22:00:38] <TimStarling>	 sections are not necessarily DOM subtrees
[22:00:40] <Marybelle>	 Why separately?
[22:01:25] <TimStarling>	 ideally they would be, so that you could preview section editing, but it's not enforced
[22:01:58] <TimStarling>	 anyway, that's a discussion for another day
[22:02:20] <TimStarling>	 #info extracts to use extension1 like AFTv5
[22:02:50] <TimStarling>	 #info TimStarling has some concerns about data set size overwhelming db1031's available memory but MaxSem doesn't think it is a problem. Probably TimStarling is just paranoid.
[22:03:35] <TimStarling>	 any objections to approving the RFC?
[22:04:56] <MaxSem>	 none, apparently...
[22:05:04] <TimStarling>	 #agreed RFC approved
[22:05:34] <TimStarling>	 thanks everyone
[22:05:37] <MaxSem>	 thanks, everyone:)
[22:05:57] <TimStarling>	 next week at the same time, we will have
[22:06:05] <TimStarling>	 https://www.mediawiki.org/wiki/Requests_for_comment/MediaWiki_HTTPS_policy
[22:06:11] <TimStarling>	 https://www.mediawiki.org/wiki/Requests_for_comment/Extensions_continuous_integration
[22:06:19] <TimStarling>	 #endmeeting
[22:06:19] <wm-labs-meetbot>	 Meeting ended Wed Nov 19 22:06:19 2014 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
[22:06:19] <wm-labs-meetbot>	 Minutes:        https://tools.wmflabs.org/meetbot/wikimedia-office/2014/wikimedia-office.2014-11-19-21.00.html
[22:06:19] <wm-labs-meetbot>	 Minutes (text): https://tools.wmflabs.org/meetbot/wikimedia-office/2014/wikimedia-office.2014-11-19-21.00.txt
[22:06:19] <wm-labs-meetbot>	 Minutes (wiki): https://tools.wmflabs.org/meetbot/wikimedia-office/2014/wikimedia-office.2014-11-19-21.00.wiki
[22:06:20] <wm-labs-meetbot>	 Log:            https://tools.wmflabs.org/meetbot/wikimedia-office/2014/wikimedia-office.2014-11-19-21.00.log.html