[18:01:02] #startmeeting IRC Office Hour on Phabricator [18:01:03] Meeting started Wed May 14 18:01:02 2014 UTC and is due to finish in 60 minutes. The chair is andre__. Information about MeetBot at http://wiki.debian.org/MeetBot. [18:01:03] Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. [18:01:03] The meeting name has been set to 'irc_office_hour_on_phabricator' [18:01:13] Hi & welcome everybody! [18:01:13] * guillom waves. [18:01:16] This is the IRC office hour about migrating our product/project management and software development infrastructure tools to Phabricator. [18:01:19] Who's here? [18:01:24] o/ [18:01:24] _o/ [18:01:37] I'm about [18:01:42] Hello [18:01:48] I'm kinda here [18:01:56] Cool! [18:02:00] o/ [18:02:27] so.... [18:02:30] welcome everybody! [18:02:42] I'll give you a brief (well... 10min?) intro on the latest news [18:02:50] and afterwards we can discuss questions? [18:02:54] * greg-g waves [18:03:18] alright! [18:03:23] #topic RFC closed [18:03:30] As you know, we ran an RFC about the proposal to move to Phabricator for three weeks at https://www.mediawiki.org/wiki/Requests_for_comment/Phabricator [18:03:46] We closed this RFC on May 6 by http://lists.wikimedia.org/pipermail/wikitech-l/2014-May/076332.html [18:03:53] #info The result of the RFC is that there seems to be general support for moving from our infrastructure tools (Bugzilla, RT, Trello, Mingle, Scrumbugz, Gerrit, Gitblit) to Phabricator. [18:04:28] #info This won't all happen at the same time though - we will start with replacing Bugzilla, RT, Trello, Mingle. [18:04:42] For the code review part however (Gerrit), more work in Phabricator's 'Differential' component is needed to fit the needs defined by our community, and we do not plan to switch off Gerrit on the day we start using Phabricator in production. [18:05:37] o? [18:05:41] o/ [18:05:43] and thanks to qgil we also have a nice comparison of Phab to Bugzilla here: for http://www.mediawiki.org/wiki/Requests_for_comment/Phabricator/versus_Bugzilla [18:05:43] So... Latest news: [18:05:44] #topic Zurich Hackathon news [18:05:44] #info Last weekend the Wikimedia Hackathon took place in Zurich. There were several Phabricator related sessions (which were recorded). [18:08:04] Umm. [18:08:27] I thought the bot was fixed to never ever ever remote the note? [18:08:35] * James_F sighs. [18:09:59] andre__: ? [18:10:22] I'm wondering if andre__ disappeared into the Abyss of Bad Wifi. [18:10:32] Right. [18:10:38] yep [18:10:40] he did [18:10:44] hm [18:10:47] Are there any questions yet about what andre has been saying so far? [18:10:59] <^d> Is there cake? I was told there would be cake. [18:11:08] I'm betting to see an "andre____________________________" in half hou or so :) [18:11:09] the cake is a lie [18:11:10] ^d: The cake is a lie. [18:11:12] ^d, yes, yes there is [18:11:14] can it be repeated? i did get it [18:11:16] sigh [18:11:18] sorry, got reconnected (24h) [18:11:18] * guillom high-fives twentyafterfour. [18:11:20] what was the last line printed here for y'all? :) [18:11:22] ^d, I got cake. Didn't you get cake? [18:11:34] #info Last weekend the Wikimedia Hackathon took place in Zurich. There were several Phabricator related sessions (which were recorded). [18:11:34] * wm-labs-meetbot has changed the topic to: Zurich Hackathon news (Meeting topic: IRC Office Hour on Phabricator) [18:11:34] andre____: several sessions recorded in Zurich [18:11:40] thanks [18:11:52] * andre____ starts again. Sorry. [18:11:54] 1) A general quick introduction to Phabricator by Shahyar: [18:11:58] #link https://www.youtube.com/watch?v=INoISTkEGlQ [18:12:04] 2) A discussion among ~10 people of what needs to be sorted out and done for Day 1 of Phabricator in production (being tracked at http://fab.wmflabs.org/project/board/31/ ) and the related stakeholders: [18:12:17] #link https://www.youtube.com/watch?v=9o6aQ46bcRE [18:12:25] 3) A session by Shahyar specifically explaining the code review concept and tool in Phabricator: [18:12:29] #link https://www.youtube.com/watch?v=OdJmEAliDAA [18:12:36] (remember: we do not plan to switch from Gerrit to Phabricator for code review soon, but in the long run) [18:12:46] #topic Current status / situation [18:13:14] #info We have the planning board for planning the "Wikimedia Phabricator Day 1 in production" at http://fab.wmflabs.org/project/board/31/ [18:13:25] You see several columns here: [18:13:42] 'Need discussion' needs more thoughts first until we know how to solve those tasks; 'Ready to Go' are tasks for which the way forward is clearly defined and somebody could start working on them (when no dependencies are left to solve); [18:13:55] 'Waiting for upstream' are tickets that we have reported to the upstream developers at https://secure.phabricator.com ; 'Doing' is what is currently actively being worked on [18:14:07] You get how this is supposed to work :) [18:14:49] For your info, we also keep upstream informed of what is important for Wikimedia's use of Phabricator - we have our own project board upstream at https://secure.phabricator.com/project/board/404/ [18:14:55] #link https://secure.phabricator.com/project/board/404/ [18:15:16] #info The team working on Wikimedia Phabricator and the stakeholders are "defined" in http://fab.wmflabs.org/T283 [18:15:33] I'll quickly try to introduce folks (feel free to correct my mistakes): [18:15:53] twentyafterfour will develop missing code. (This was a very simplified statement.) [18:16:10] like investigating the migration of Bugzilla and RT data to Phabricator; and code changes required to restrict access to certain projects by default [18:16:19] (for example Bugzilla's "Security" product whose tickets are not public but everybody can report tickets in that product) [18:16:46] twentyafterfour: Correct me if I'm wrong, because my internet connection yesterday was even worse :) [18:16:55] chasemp is the contact in the WMF Technical Operations team (setting up low-level infrastructure for Phabricator; puppetizing Phab; migrating from RT) [18:17:17] I'm also a gemini fyi :) [18:17:25] oh :) [18:17:42] jforrester is our proxy for the team of Product Managers [18:17:46] you know, context, sorry for interrupting :) [18:17:55] np :) [18:17:59] greg-g represents the Platform and Release Engineering/QA team [18:18:09] and I organize/prioritize these efforts and the progress and am to be blamed for things that go wrong. :) [18:18:27] Plus we have more stakeholders listed which will provide input to help us. [18:18:28] Right, I will be working on the upstream phabricator code to help make it suit our needs. At least that's where I am going to start focusing my attention initially. [18:18:32] Let's all blame andre____ for no specific reason. [18:18:45] I'm used to that, because I have Bugzilla! [18:18:49] also, twentyafterfour is 3 days on the job, welcome him ;) [18:18:58] * andre____ bows down [18:19:04] welcome! [18:19:18] ...so yeah, a big Thank you guys in advance for your help in getting this done! [18:19:18] I hadn't realized who twentyafterfour was. Welcome indeed! [18:19:37] and a huge huge thanks to qgil and guillom for all their help to actually get to the point where we are now! [18:19:41] also, rush <- chasemp, chasemp -> rush [18:19:42] twentyafterfour = mukunda modell = the f'in new guy [18:19:52] :) [18:19:53] in terms of phabricator and other systems currently [18:20:13] one nickname per system, please - another reason to move to one system only. ;) [18:20:19] #info Our general and central information page is at https://www.mediawiki.org/wiki/Phabricator [18:20:32] Now guillom and me are meeting in quiet places to remember the old days at the barricades... [18:20:48] Hey twentyafterfour. :-) [18:21:18] (James is the PM for VisualEditor) [18:21:31] ...and it seems that the team won't need sprints and reviews, but will meet regularly to discuss progress and potential roadblocks. [18:21:47] So yeah. That's all I have to say! [18:21:50] PdM, kthxbai. [18:21:54] Thanks, andre____ [18:22:16] * guillom congratulates andre____ on soon becoming the Phabmeister. [18:22:19] What did I miss? :D [18:22:35] andre____: how often is "regularly" for our checkins? [18:22:51] andre____: Tell people what they can do to help? [18:23:26] greg-g, I'm thinking of every two-three weeks? Probably two. [18:23:31] volunteers too, if needed? [18:23:31] agreed. [18:24:07] #info How to help: Take a look at the tasks on http://fab.wmflabs.org/project/board/31/ [18:24:11] matanya3: always welcome in this. In fact, I forget his name, but on person is goign to try to work with upstream to implement burn down charts (burn up charts are available already) [18:24:24] #tpyos [18:24:44] <^d> Question about applications. I think we should take the approach of "disable by default" and then we'd need to make an argument to turn things on. [18:24:55] greg-g, that might be SWidmann? [18:24:55] swidmann will work on burndown chart upstream, see/join https://secure.phabricator.com/T5051 [18:24:55] <^d> Keep life simpler. [18:24:57] andre____: yes, him! [18:25:02] <^d> (Not so much a question as a statement) [18:25:08] hey guys greg-g had poked me about emailing a version of the talking points on small migration patterns. like analytics and growth and a bit of ops to facility knowledge base growth [18:25:11] ^d: Agreed. [18:25:22] I'm thinking I would like to wait to send that email until I'm ready to actually facilitate that [18:25:23] So the board should list all the stuff we need to sort out before "going live". And these tasks welcome your comments and your work towards getting them resolved [18:25:30] otherwise it will just be a second email when it's a real thing [18:25:44] and second emails get disregarded much faster [18:25:56] may it be defining the way forward if the way forward is unclear, or working on "ready to go" tickets [18:25:59] chasemp: fair. I just wanted to make sure people were on the same page about it. [18:26:11] greg-g: I'm completely on board, got into it and thought [18:26:15] I guess as long as "this group here" is aware, then we're fine [18:26:16] hmmmm...this may be too early [18:26:23] ^d: Applications: http://fab.wmflabs.org/T330 ? [18:26:27] ^d ref apps installed by default, your feedback is welcome at http://fab.wmflabs.org/T330 [18:26:29] feel free to comment [18:26:51] <^d> qgil, andre____: Yeah, I've commented there already :) [18:26:56] heh, ok [18:27:14] greg-g: that wasn't meant to call you out, just to cement that we both thought it was a good idea in general [18:27:23] wiki for example [18:27:24] * greg-g nods [18:27:25] #topic Q&A [18:27:50] matanya3: you don't want yet another wiki? :) [18:27:55] <^d> Can we have a cake application? [18:28:08] ^d: Hack away, but try upstream first! [18:28:15] ^d: Make a cake meme. [18:28:16] the phabricator wiki is pretty lame, in my opinion [18:28:31] twentyafterfour, we've switched that off anyway :) [18:28:39] <^d> If only we knew some people who wrote wiki software. [18:28:42] :) [18:28:51] lol [18:29:56] also I wonder if we might be well served by making all the text fields support a syntax similar to mediawiki markup [18:30:14] twentyafterfour, that sounds related to http://fab.wmflabs.org/T100 [18:30:18] I'm not sure how similar it is currently [18:30:19] there's a bug about that... [18:30:22] cool [18:30:36] not much similarity [18:31:25] twentyafterfour: "Similar to" is possibly going to be a lot of effort, of course. [18:31:58] <^d> We could at least hack it up like we did with BZ. [18:32:04] <^d> So rudimentary [[foo]] links work [18:32:05] James_F: maybe not so much effort. phabricator has a pluggable markup system [18:32:19] call out to parsoid? :P [18:32:25] <^d> Even if we don't get all the magic, at least having that sort of basic syntax would be nice. [18:32:29] twentyafterfour: Hmm. In that case, could we… yeah, what greg-g said. [18:32:35] * robla fears getting too ornate with customizing Phab's markup [18:32:37] James_F: VE is a standalone plugin, right? :P [18:32:37] Ref chasemp plan to have an interim instance to host Analytics and other projects, I wonder whether there is consensus about this. [18:32:42] and it's php, surely there is a good wiki markup parser written in php...somewhere out there [18:32:44] greg-g: … yes. [18:32:53] yeah, I'd keep this idea on the low low low backburner [18:33:01] TrevorParscal: ^^^ We have a taker for first "third party" integration. ;-) [18:33:13] yeah? [18:33:28] (let's not get bogged down in parser discussion) [18:33:28] TrevorParscal: Phabricator. (Probably not seriously.) [18:33:37] "Third party integration" could also be connecting Gerrit to Phabricator though, which might be higher priority ;) [18:34:32] andre____: Integration of VisualEditor, I meant; we've been talking about it for years without actually doing anything. [18:34:42] Ahem, let me insist that chasemp 's plan to offer an instance for some teams to do real work needs discussion and agreement [18:34:50] James_F: ah, nice. Day2 stuff. :) [18:35:06] qgil: it seems like a good idea to me [18:35:15] qgil: what is there to agree on? I was thinking it was up to whoever wanted to participate [18:35:30] which could be just me in a room with a good book idk [18:35:37] andre____: Well, it is a decide-now-or-give-up-forever on the syntax used in our Phabricator instance, actually, so… [18:35:39] as long as chase has the time [18:35:59] Now I file a bug for Analytics in Bugzilla and they import it to their... Mingle? Trello? project. Would this mean that they would import to their Phabricator? And what happens with these Pabricator tasks when we open the Real Phabricator? [18:36:18] I also think it is a good idea in theory, but I'm not sure the teams and us have thought about the consequences [18:36:19] James_F: if that's the case, I'll be a pretty vocal proponent of "give up forever" [18:36:19] the get migrated [18:36:21] they* [18:36:23] errm, that instance for some teams would not be separate from the final "one" production instance? [18:36:42] qgil: those are good questions I think [18:36:46] robla: Picking your backend storage format isn't the kind of thing you get to change later (whereas the software that fiddles with it is). [18:36:52] migrating tasks/etc from one phab to another is easy, I thought we addressed that flow yesterday on call? [18:37:09] can I briefly lay out my informal plan that is somewhat oriented towards those with existing enthusiasm? [18:37:32] I'm just pointing out that Chase seems to have a plan in mind that I'm not sure it is known/ agreed by others, and we need to be on the same page. That's all. No strong opinion myself. [18:37:34] may help, tho I was goign to do this in an email for wider reading when we actually were ready to do something with it [18:37:36] James_F: migrating from Phabricator's markup to something else might be a bit lossy, but it's not intractably hard. It's not as though Phabricator implements templates or parser functions or anything crazy like that [18:37:48] robla: Eh. Lossy means pain. [18:38:08] if we had to migrate from an "Analytics Phabricator" (let's call it like that) to the general production Phab on day 1, I assume that ticket numbers etc might change again? [18:38:08] (or maybe I don't get the proposal/problem correctly?) [18:38:12] robla: But if we're OK with the pain, sure. [18:38:14] chasemp, you should document your plan either at http://fab.wmflabs.org/T294 or in a task related to it. [18:38:32] James_F: robla: we will have to figure out what to do with all of the comments imported into Phab from BZ that use BZ's fancy/special formatting, but... let's maybe move this on ticket somewhere? [18:38:41] greg-g: Sure! [18:38:44] qgil: I understand, my thought was email and copy to that ticket [18:38:50] as I don't think a lot of ppl are tracking it [18:38:54] greg-g, which fancy/special formatting? [18:39:07] andre____: the [[foo]] stuff [18:39:17] chasemp, whatever works for you as long as tyou print it out from your brain. :) [18:39:21] andre____: non-standard BZ stuff [18:39:41] greg-g: that's just the regex interpretation when rendering it, it's not stored in the comment text in the database [18:39:50] * greg-g nods [18:39:57] and our custom ones are in https://git.wikimedia.org/blob/wikimedia%2Fbugzilla%2Fmodifications.git/HEAD/extensions%2FWikimedia%2FExtension.pm [18:40:17] qgil: being so new it's mostly learning experience for me, since there doesn't seem to be a big hand forcefull hand here, in many ways early adoption is going to be based solely on enthusiasm and desire [18:40:47] I'm really just hoping to make it possible, and harness it to create a knowledge base within and figure smooth the bumps prior to wider adoption [18:40:58] 'plan' is a rather risky word for that kind of thing :) [18:41:06] but I will more clearly outline it [18:41:32] my typo rate is truly atrocious and I'm sorry [18:41:33] chasemp, yes, I agree, its just that you need to be aware that teams are supposed to work within a wider community, and radical changes in their workflow muust take this factor into account [18:41:57] qgil: good though, i was just counting on them shepard their own sheep I guess? [18:42:08] thought even [18:42:48] sheep in a good depend on my flock way [18:42:53] where have/should we document the requirement to reserve task and diff numbers for importing BZ/Gerrit/RT items? [18:43:13] chasemp, it makes sense that you assume this shepherding, and it makes also sense that I'm a bit skeptical about them doing so :) [18:43:20] e.g. BZ bug 12345 -> T12345 in Phabricator [18:43:32] anyway, moving on. Please document so we can discuss and agree. Overall I like your idea! [18:43:47] fwiw there is also the possibility to run multiple phabricator instances for multiple purposes rather than forcing everyone into a single global phab [18:44:09] I don't know if that's worth considering or not [18:44:10] twentyafterfour: I had the same thought, I think...that's taboo atm [18:44:14] twentyafterfour, in such context, can you move tasks from one instance to another? [18:44:16] twentyafterfour: that seems kinda painful [18:44:23] here at least, the idea of one ring to rule them all is what has driven this so far [18:44:32] yeah... moving from one instance to another wouldn't be easy [18:44:37] they would be islands [18:44:50] YOU SHALL CONFORM TO THE ONE TRUE TRACKER....JOIN US! [18:45:00] heheh [18:45:06] twentyafterfour, then yeah, the main point of this migration would be lost. [18:45:09] qgil has to throw gerrit into mordor [18:45:14] or am I missing a reference [18:45:32] my precious +2... [18:45:52] the only thing that might convince me that multiple instances might be necessary is my reading of https://secure.phabricator.com/T3820 [18:46:19] robla, these are different namespaces, which is still would run in one single instance. [18:46:21] ...which seems to imply that getting the security we need may be a many month process [18:46:38] yes that is true, without that ticket two instances is safest [18:46:44] qgil: yup, but that feature hasn't been implemented yet [18:46:52] <^d> Or we just stop having secret things :p [18:46:52] robla, specifically https://secure.phabricator.com/T3820#20 [18:46:55] I might be able to hack in a simple security feature that doesn't require large architecture changes in phabricator [18:46:57] robla: the action of "reserving" task (etc) numbers should probably become a separate task (blocking http://fab.wmflabs.org/T39 and/or directly related to http://fab.wmflabs.org/T294 ), I don't see one like that yet [18:47:16] I think twentyafterfour and I agree that if he starts knocking on doors and pushign things out [18:47:29] the response will be positive in this sense [18:47:29] <^d> +1 [18:47:30] and upstream will give 15% for every 10% we give [18:47:31] robla, twentyafterfour should help pushing those namespaces forward, since this is the foundation of "the Security usecase" [18:47:49] chasemp, +1 [18:48:23] twentyafterfour: do you have any sense of scope of getting T3820 implemented? days, weeks or months? [18:49:19] robla: the essential architecture that it'll be built on is already in place so I don't think it's many months, probably many weeks though [18:50:09] if I start hacking on it this should help move it forward though, and evan priestley is always quite helpful when someone takes initiative [18:50:44] not to mention that he's in san francisco so I plan to bribe him with drinks when I'm there at the end of the month [18:50:50] xlnt [18:50:53] :) [18:52:11] Just a pitch for those willing to help on the Code Review front while Day 1 focuses on Task/Bug management: I have started sorting out http://fab.wmflabs.org/project/board/29/ -- your opinions, tests, and help are welcome. [18:52:37] thanks qgil! [18:53:06] There is a lot that can be cleaned and decided before twentyafterfour chasemp etc are done with Day 1. [18:53:27] ... and for this task it is good to have people familiar with our Gerrit-based code review process [18:54:55] Listening to Shayar in Z�rich made me think that the former deviantArt guys will be more useful after the old Wikimedia dudes iron the discussion in our community [18:55:36] ** General note: We have ~5 minutes left. ** [18:57:33] Did I scare you out? [18:58:21] Any more comments / topics / questions here? [18:58:23] no, I think we're all pretty much aligned and have our tasks to go update :) [18:58:28] (Or is it silent because my wifi is down again? ;) [18:58:29] ah [18:58:50] Thanks andre_____________________________________ :) [18:59:46] Alright. Thanks a lot everybody who joined this session! Again, please take a look at http://fab.wmflabs.org/project/board/31/ for what has to be done, [18:59:50] thanks for kicking butt guys, I think this will be a medium to pretty well passable and affable success [19:00:02] ...or at http://fab.wmflabs.org/project/board/29/ if you're into code review [19:00:13] Thanks for your time & see you soon! [19:00:18] #endmeeting [19:00:51] er... [19:00:56] #finishmeeting [19:00:59] #finalizemeeting [19:01:01] whatever! [19:01:08] did you type startmeeting? [19:01:28] ah, no real meetbot [19:01:28] yes I did. but I reconnected. [19:01:45] interesting [19:01:45] hmm. [19:01:46] #endmeeting [19:01:46] Meeting ended Wed May 14 19:01:46 2014 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) [19:01:47] Minutes: https://tools.wmflabs.org/meetbot/wikimedia-office/2014/wikimedia-office.2014-05-14-18.01.html [19:01:47] Minutes (text): https://tools.wmflabs.org/meetbot/wikimedia-office/2014/wikimedia-office.2014-05-14-18.01.txt [19:01:47] Minutes (wiki): https://tools.wmflabs.org/meetbot/wikimedia-office/2014/wikimedia-office.2014-05-14-18.01.wiki [19:01:47] Log: https://tools.wmflabs.org/meetbot/wikimedia-office/2014/wikimedia-office.2014-05-14-18.01.log.html [19:01:49] ah [19:01:51] :) [19:02:11] we seem to have a slightly deaf meetbot [19:02:23] bye [19:02:35] thanks andre____ enjoy offline-ish EU! [19:02:50] qgil, heh :) thanks too [20:43:30] * greg-g pokes twentyafterfour to /join #mediawiki-core :) [20:49:51] greg-g: Need -staff assistance? [20:50:01] that too [20:50:04] * marktraceur does [20:50:36] ty [20:50:51] greg-g: Done [20:51:01] thanks. [20:51:14] twentyafterfour: you now have access to the staff channel as well. [20:51:38] btw, for people who want to understand why I'm writing https://www.mediawiki.org/wiki/Performance_guidelines , or bikeshed it :) , or talk about the security guidelines or architecture guidelines, it'll be here in 10 min [20:52:42] ohai [21:00:04] ok! Hi [21:00:09] #startmeeting Discussion of Performance guidelines | Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE). https://meta.wikimedia.org/wiki/IRC_office_hours [21:00:09] Meeting started Wed May 14 21:00:09 2014 UTC and is due to finish in 60 minutes. The chair is sumanah. Information about MeetBot at http://wiki.debian.org/MeetBot. [21:00:09] Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. [21:00:09] The meeting name has been set to 'discussion_of_performance_guidelines___channel_is_logged_and_publicly_posted__do_not_remove_this_note___https___meta_wikimedia_org_wiki_irc_office_hours' [21:00:17] #chair sumanah brion TimStarling paravoid ori [21:00:17] Warning: Nick not in channel: brion [21:00:17] Warning: Nick not in channel: TimStarling [21:00:17] Warning: Nick not in channel: paravoid [21:00:17] Warning: Nick not in channel: ori [21:00:17] Current chairs: TimStarling brion ori paravoid sumanah [21:00:32] hmmm [21:00:36] lol [21:00:52] It's the Zurich meeting all over again [21:00:56] ha! [21:01:04] it is ok since this is not a "please approve this" meeting [21:01:19] #link https://www.mediawiki.org/wiki/Architecture_meetings/Performance_guidelines_discussion_2014-05-14 [21:01:42] #topic Performance guidelines [21:01:42] Krinkle is here this time at least! [21:01:45] True! [21:01:49] #link https://www.mediawiki.org/wiki/Performance_guidelines [21:02:10] (As I said in my email to wikitech-l: [21:02:11] We discussed this some in Zurich but I'd love a [21:02:12] chance to ask some followup questions to firm everything up. I'd also [21:02:12] welcome the chance to explain the two similar documents I'm working on: [21:02:12] architecture and security guidelines. [21:02:13] ) [21:02:36] I am [21:02:37] #info We chatted about this during a discussion at the Zurich hackathon and it sounds like this document is about 80-90% there [21:02:51] #info I already have a big list of TODOs that I want to hit today and tomorrow https://www.mediawiki.org/wiki/Talk:Performance_guidelines#TODO_for_Sumana.2C_14_May_2014 before circulating this on #wikitech-l [21:03:07] #info The performance guidelines, along with the upcoming security guidelines https://www.mediawiki.org/wiki/Security_for_developers/Architecture , interrelate with https://www.mediawiki.org/wiki/Architecture_guidelines - the idea is for new developers to look at those 3 docs which describe how we do things [21:03:19] #info they are descriptive, not hypothetical/aspirational [21:04:35] If anyone has any questions for me - "what's the point", "why are these 2 principles separate", etc., I am happy to answer [21:05:06] I can also use this time to ask a few questions :-) for instance: should I list memcache as a persistence layer? maybe I should say, "maybe use memcache (and/or other caches) INSTEAD OF a persistence layer!" [21:05:57] Items in memcached can be removed randomly at the will of the server [21:06:07] So I'd recommend not listing it as persistence [21:06:19] Even if, in practice, WMF memcache servers tend not to delete anything unless they expire [21:06:24] redis would be the 'in-memory' persistence store [21:06:27] [citation needed] [21:07:02] So, in https://www.mediawiki.org/wiki/Performance_guidelines#Persistence_layer right now I say: "memcached cache - storage for things that persist between requests, and that you don't need to keep - you're fine with losing any one thing. Use memcached to store objects if the database COULD recreate them but it would be computationally expensive to do so." [21:07:03] Note, however, that Redis when used as $wgMemc should not be considered persistence [21:07:21] memcache also deletes in case of server reboot and doesn't gracefully handle outage of a node (in case you have a memcache cluster with a number of nodes) [21:07:41] sumanah: yeah that is good, but I'd recommend not mentioned memached specifically [21:07:45] Why not? [21:07:47] you can't shrink or grow memcache node count without pain [21:07:55] In reality, it's $wgMemc (or any BagOStuff object) that we should talk about [21:08:04] $wgMemc can be memcached [21:08:07] Or it can be another backend [21:08:12] It's abstracted [21:08:17] so you always have extra memcache hot spare not in use and then can substitute in for a dead host's slot [21:08:49] parent5446: I am willing to, in this document, often say something like "a cache (such as memcached)" or "a unique identifier (such as a primary key)". Some people learn better by thinking of general classes, and some through concrete examples [21:09:36] Yeah that's fine [21:09:41] Thanks [21:09:53] I mean, in the code we make the same mistake all the time anyway [21:09:56] i suppose my worry is more about using the words memcache and persistence anywhere close to eachother :) memcache should be treated as something more ephemeral than a persistence layer [21:09:57] parent5446: if you feel like taking a crack at writing "picking the right cache: a guide" then I would love that [21:10:22] ebernhardson: Roan said (quoting from notes): If something is very expensive to recompute ... use something closer to a store. (we use backend Varnish, which we call a cache, which is kind of closer to store on the cache-to-store spectrum.) [21:10:25] Yeah sure, put that an #action or whatever [21:11:11] sumanah: i imagine hes talking about the parser cache there? I suppose we then need to declare levels of 'computationally expensive' [21:11:11] #action parent5446 to write "picking the right cache: a guide for MW developers" [21:11:18] sumanah: right, but "store" or "persistence" can also be for canonical (permanent storage). memcache definitely isn't that [21:11:21] because what (i think) roan is talking about there is things that can take multiple seconds [21:11:51] in which case we dont just want them in an ephemeral cache, but also (in the case of parser) writen to a database [21:11:58] ebernhardson: jeremyb if there is an existing guide out there to our caching layers and which ones are for which purposes, that's better than https://www.mediawiki.org/wiki/Performance_guidelines#Caching_layers , please tell me so I can link to it [21:13:21] not aware of anything better [21:13:38] sumanah: For what it's worth, I think the section you have for "persistence" looks good. Taken in context you are saying the right things. [21:13:56] Taken out of context there is ambiguity in the wording [21:14:00] nod, nod [21:14:19] sumanah: I'm here [21:14:29] #action sumanah to improve the cache-to-store spectrum distinction, make sure no one thinks any of the caches are actually persistent stores! [21:14:34] Thank you RoanKattouw [21:14:36] #idea I want to check whether we think it's a first-class performance goal to make anonymous browsing and logged-in browsing the same speed. [21:14:48] I think it is [21:14:59] That'd be amazing if we could achieve that [21:15:02] RoanKattouw: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20140514.txt This meeting: Starts 2100. [21:15:05] Some people have been talking about this but there are no real efforts to do that yet [21:15:18] OK, so, would you put this goal into the aspirational bucket rather than the descriptive bucket? [21:15:19] So... make the app servers as fast as Varnish? Seems unlikely. [21:15:35] I think it means using more Varnish caching for logged-in users. [21:15:39] Yes, exactly [21:15:41] Caching as much as possible [21:15:59] bd808: more like turn off user specialization so everyone can get the same varnish output (with some ESI) [21:16:07] Which would also make things faster for logged-in users in, say, Europe [21:16:20] And exactly, we'd need techniques like ESI [21:16:27] * bd808 nods [21:16:33] Right now we don't use ESI at all but we would want serious ESI magic to pull this off [21:16:40] OK, so it sounds like this goes into the vision doc, agreed? [21:16:42] So it's probably a ways out to actually do this [21:16:56] sumanah: I assume the vision doc is for pie in the sky stuff? [21:17:02] ("vision doc" is what I am calling the "what we want to do in the future" thing that will probably be across security, perf, and architecture) [21:17:07] Or rather, is where pie in the sky stuff goes among other things [21:17:28] Mmmmm... mysterious sky pie [21:17:41] So, agreed? [21:17:44] Right [21:17:47] <^d> The other option is slowing down anonymous users to be more like logged in users. Might be less work than the other way-round. [21:17:49] <^d> [21:17:54] * RoanKattouw agrees [21:17:56] ouch :-) [21:17:59] ok [21:18:00] (with sumanah , not ^d ) [21:18:14] <^d> Aw, I was hoping you agreed with me there :p [21:18:26] #agreed that the goal of making anonymous browsing and logged-in browsing the same speed goes into the future architecture vision document [21:18:48] btw RoanKattouw, Tyler agreed to write a "so you need to pick the right cache for your app needs" doc :D [21:18:54] Sweet [21:19:27] So, another minitopic [21:19:29] I have in my notes: [21:19:29] "Maybe going too much into details: in context of frontend caching & Varnish - do not serve different content from the same URL. Diff content, diff URL. Must also be true for anon users! URLs must be deterministic - with proviso that not if you are logged in! Then wrinkles - cookies submitted ..... caching layers..... should strive to cache logged in requests..... longstanding problem." [21:19:53] This sounds like a "how to not break things in the face of caching" thing? [21:20:09] The fact that something here is called a "longstanding problem" makes my ears perk up .... what bit here is the problem? That is, what bit should be going in the vision doc? [21:20:27] I presume it's "we want to someday cache logged-in users". Am I right? [21:20:32] Yes I think so [21:20:39] Which is what we just agreed to put in the vision doc already [21:20:55] i could be off base, but i think the 'longstanding problem' is that we vary by query string instead of the main url. its not an end of the world type thing though [21:21:03] OK! Just wanted to confirm and make sure I'm understanding right [21:21:13] It's very easy to break the ability of Varnish to cache content. [21:21:20]