[21:01:10] #startmeeting RFC meeting [21:01:10] Meeting started Wed Sep 2 21:01:10 2015 UTC and is due to finish in 60 minutes. The chair is TimStarling. Information about MeetBot at http://wiki.debian.org/MeetBot. [21:01:10] Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. [21:01:10] The meeting name has been set to 'rfc_meeting' [21:01:31] #topic RFC: Master/slave datacenter strategy for MediaWiki | Please note: Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE) | Logs: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/ [21:01:35] \o [21:01:39] Hello [21:01:43] \o [21:01:48] hi [21:01:51] hi [21:01:59] hello [21:02:04] 'lo [21:02:07] hi [21:02:14] Hey [21:02:22] hey [21:02:28] hi [21:03:04] wuff [21:03:15] #link https://www.mediawiki.org/wiki/Requests_for_comment/Master_%26_slave_datacenter_strategy_for_MediaWiki [21:03:15] great meeting..short and sweet :D [21:03:19] heh [21:03:57] AaronSchulz: would it make sense to start by reminding everyone what is the problem you're trying to solve? [21:04:17] just to get everyone on the same page [21:04:19] +1 to ori [21:04:45] ori: I suppose, though it hasn't changed since I wrote the rfc intro yet :) [21:05:06] up to you; it's your show [21:05:07] AaronSchulz: feel free to cut and paste the appropriate part of the RFC :-) [21:05:31] https://www.mediawiki.org/wiki/Requests_for_comment/Master_%26_slave_datacenter_strategy_for_MediaWiki#Background [21:05:34] so, the idea is to have more traffic served by our new Dallas DC while also having the Ashburn one serve traffic [21:06:22] Well, I can put in some thoughts on how we're looking at the very basics at the traffic layer. Even if we didn't have true live master/slave at the applayer, the intent has been to have the two primary DCs have hot, in-use cache layers that do not rely on each other. [21:06:24] I heard that CDN is up in codfw (Dallas), which is cool. This will extend that to HTTP GET page requests for MediaWiki in general (logged out or in, doesn't matter) [21:07:11] The idea would be (if nothing else here came to fruition) that the non-primary directly reaches the applayer over the wan link, and we could configuration-switch both of them to backend to the opposite DC [21:07:38] So we will have appservers in codfw serving traffic, while eqiad remains the primary appserver DC? [21:07:48] yes [21:07:52] so, we can have user-facing traffic split already fairly easy. the key thing here is being able to truly use both at the same time at the applayer to keep various lower-level things warm and running [21:08:34] so it sounds like you are shooting for a phased approach, with CDN being possibly the first phase? [21:08:47] ideally it could lower latency in the common (non-switch) case as well, though keeping both DCs warm and at the ready is the main focus [21:08:51] AaronSchulz: what are the remaining problems and what do you need input/help on? [21:08:57] well I'm saying the CDN part I mentioned above isn't even a part of this. It's already being architected that way. It's a precursor to this :) [21:09:00] TimStarling: good question :) [21:09:37] bblack: I mean in terms of actual traffic switching, it would probably be the first step? [21:09:47] but having that traffic split/switch capability at the edge/cache layer does not give us hot/ready/tested/filled caches at lower layers, etc [21:09:58] I'll need to know the status of Swift replication (I've only heard bits and pieces) and I've been keeping an eye on Elastic replication (the search team has been doing some work on this) [21:10:48] gwicke: it's already happening in a limited way. Texas users are hitting codfw right now, but the config isn't in the final desirable state yet either. [21:11:03] I'll also need to recruit someone, maybe eevans/ori to work with me on getting cache relay purging working [21:11:07] What is the progress with adapting MW code and extensions to the New Ways of doing things (WANCache, idempotent GET, etc.), and is there a place where devs can educate themselves on these practices/changes? [21:11:12] I can speak to the swift part, ATM we are replicating originals in codfw from eqiad with 'swiftrepl' from mark [21:11:25] AaronSchulz: i'm still game [21:11:33] I tried last year per-container swift replication but didn't get very far [21:11:35] the model is there, but we need a real service for that (not just the toyish prototype) [21:11:35] bblack: oh, nice; didn't know that we were that far already [21:11:49] AaronSchulz: what do you mean by "cache relay purging" [21:12:08] paravoid: as in relaying memcached deletes and purges [21:12:17] to all DCs [21:12:34] there is an application layer (i.e. MW) option for swift isn't there? [21:12:44] TimStarling: for replication? [21:12:48] yeah [21:12:49] right, that's where I see the benefit of the work in this RFCs, beyond what we're doing at the edge: having memcached, db, redis, etc all working hot all the time, and ready for a fast easy switch to known-working stuff. [21:13:02] with the edge-layer work alone, we'd be looking at a very cold/questionable switch at the applayer [21:13:03] at least, I thought it was discussed [21:13:07] AaronSchulz: ah. the RFC hints at that ("memcached purger daemons") but it isn't really spelled out [21:13:09] I guess the FileJournal could be turned on again with SyncFileBackend in a service loop or something [21:13:42] it would need some slight bolstering (a --uselock mode flag, not much work) [21:13:54] right [21:13:58] I never got around to that since I wanted to see if a lower layer setup could work [21:14:12] paravoid: AaronSchulz has something working with https://github.com/AaronSchulz/python-memcached-relay but (AFAIK) is not sure it's the right solution; something off-the-shelf would be nicer. [21:14:13] (e.g. something in Swift...container sync, geo-clusters, ect) [21:14:52] we haven't tried native swift cluster replication no, but the thought of leaving the two unaware of each other was nice [21:15:46] is there anything we need to change in file storage to be friendlier to replication? just thinking of, eg, replacing large files on re-upload [21:16:22] content-addressable storage would reduce the number of updates [21:16:39] so swiftrepl is done and in production? is that enough? [21:16:53] there is a separate conversation in progress on that [21:17:12] TimStarling: I don't think that works by tailing updates though, but it would be useful for random things that still use swift outside of FileRepo (does math still do this?) [21:17:27] yeah, math and score [21:17:35] I would hardly call swiftrepl "done" [21:17:42] it has always been a hack coded in one afternoon or two [21:18:09] agreed, it does need some love if it stays [21:18:11] it sequentially goes through all files in all containers and copies them (or deletes them) across [21:18:15] TimStarling: I'd be worried it would take too long for some math/score files to show [21:18:21] that would need some though at the least [21:18:32] using swiftrepl/copyFileBackend [21:18:38] paravoid, godog: so looking ahead to the end of the meeting -- can you capture what remains to be done in a short one-liner TODO? [21:18:41] so yes, it takes hours to propagate changes [21:18:46] (re: swiftrepl) [21:19:03] using the syncFileBackend would require mariadb FileJournal updates just for math/score stuff...ugly though maybe not world ending [21:19:06] ori: sure [21:19:33] it could replicate quickly though, which is the advantage [21:19:37] I don't think swiftrepl is a solution that shuld be considered here at all, honestly [21:19:38] so one project that could potentially help with speeding up replication is the event queue we are going to work on next quarter [21:19:51] AaronSchulz: we don't do keep thumbs in the journal though, do we? [21:20:02] nope, same as math/score [21:20:03] it has been a long time since I cared for those things, forgive me if I'm totally mistaken :) [21:20:09] * AaronSchulz didn't want to spam the DB [21:20:18] gwicke: I don't think it would be responsible of us to bank on that. It sounds like a tricky thing to get right. [21:20:26] though thumbs at least can regenerate on request [21:20:45] sure, just saying that we'll likely have an event stream of things like file changes soon [21:20:55] possibly, yes; but then we'd have to handle propagating deletes somehow... [21:21:40] sure, that's why swift sucks for thumbnails isn't it? [21:21:48] among other reasons :) [21:21:50] *doesn't [21:22:21] it would be nice if the CDN and thumbnail regeneration didn't have to care about the other DCs [21:22:25] we also never garbage collect thumbs afaik [21:22:27] alas, one can dream... [21:22:59] direct FileBackend callers in deployed extensions are ConfirmEdit, Score, Math, GWToolset and timeline [21:23:42] I thought that MediaViewer needed thumbs generated up-front for performance reasons. I could be off or out of date, though. [21:24:10] i wouldn't mind just keeping thumbs in varnish though :) [21:24:18] matt_flaschen: yes standard thumb sizes are pregenerated now iirc [21:24:19] they can be precached via URL [21:24:25] yeah I saw that Aaron has a patch for that [21:24:37] https://gerrit.wikimedia.org/r/#/c/126210/2 "[WIP] Added support for CDN-only thumbnail storage" [21:24:39] brion: I think there's an RfC for that (thumbs in varnish) ;) [21:24:59] TimStarling: yeah, well the real work would be CDN for that, so that patch does very little ;) [21:25:03] "updated Apr 18, 2014" [21:25:15] that's a whole different can of worms, gilles resurrected that discussion and we had a meeting about it last week again [21:25:27] well they're in varnish anyways, the hot ones. the question is how much we want to trust that we never lose all of our varnish caching all at once globally, and fall back directly on regenerating from scratch. [21:25:36] swift saves us from the worst impact of that [21:25:40] brion: sure, if we could have variant purging and a decent (not nmap) persistence model...maybe Apache TrafficServer with some hacks [21:25:49] paravoid: right, easy to be derailed [21:25:58] (and yes, agreed, huge separate topic) [21:26:11] swift also acts as a third tier cache big/cold cache [21:26:23] right [21:26:46] yeah, and bounds the effort for huge multi-page tiffs by keeping base thumbs around [21:27:02] AaronSchulz: ATS eval vs varnish for all related things in our long-term plans regardless, but they'd serve a similar functional purpose and have roughly-similar issues wrt this [21:27:26] if we have to direct thumbnail regenerations to eqiad to keep the listings (for purge) in sync, we can start off that way. I'd like to actually use both DCs for scaling so that would be temporary until something better comes (either the thumbnail redo or just decent swift replication) [21:27:41] i'd also like to consider separating swift groups between original files and long-lived derivatives (such as large video transcodes)... they can't be created on demand as easily as thumbs but are still distinct i think [21:27:50] bblack: well I assume the disk persistence would be more trustable, no? [21:27:58] we have a few things on the table now, but it needs to be divided up into units of work and sequenced logically [21:28:01] bblack: but bucking would still need tricks, yes [21:28:06] s/it needs/they need/ [21:28:12] *bucketing [21:28:21] AaronSchulz: it's pretty trustable as it is. We've made a lot of improvements to it over the past couple of years to make it reliable. [21:28:55] bblack: good to hear; last time it came up in discussion it was very sketchy [21:28:59] but we could still face a C-level or VCL-level or hashing-level bug that causes to accidentally lose things there. falling back on swift to re-warm the caches is one thing. falling directly back on scalers is another. [21:29:18] * robla wonders just how much of this RFC should be incorporated into https://www.mediawiki.org/wiki/Architecture_guidelines [21:29:26] some kind of timeline with milestones & a testing plan could help to give others an idea of where we are at & what's ahead [21:29:50] not so much with dates, more the sequencing / dependencies as ori says [21:30:07] re: search the plan is to have the jobqueue at eqiad for now be responsible for updating indexes at both locations, this gets us parity for writes. We just now have hardware in for dallas and it's not yet even OS'd. Precursor refactor for jobqueue https://gerrit.wikimedia.org/r/#/c/235149/, there is a one off here that doesn't go through jobqueue that needs some thought: https://phabricator.wikimedia.org/T109126. A [21:30:07] nd no solution for redirecting read traffic has been sorted out to my knoweldge. Briefly we discussed having an etcd key that keeps the active "svc" url or some such and relying on its multi-dc replication if available. We we want the ability to hit search at either location from either location. But it's an open ended question. [21:30:13] (TimStarling: are '#info's fair game throughout the meeting, or should they be reserved for the end?) [21:30:16] that was longer than I expected [21:30:34] you can use #info and #action throughout [21:30:34] robla yes, and to RoanKattouw's question: https://www.mediawiki.org/wiki/Requests_for_comment/Master_%26_slave_datacenter_strategy_for_MediaWiki#Design_implications says "Some changes would be needed to MediaWiki development standards", I would love to meet with AaronSchulz and turn that list into wiki improvements, e.g. https://www.mediawiki.org/wiki/Performance_guidelines#Persistence_layer [21:31:06] +1 should add to guidelines! [21:31:10] chasemp: yeah, that was my understanding [21:31:12] #action spagewmf meet AaronSchulz and incorporate RFC guidelines into developer docs [21:31:27] *RFC's guidelines [21:31:43] brion: some has been done already (e.g. caching). I can't recall how much GET/POST stuff has been mentioned on mw.org yet... [21:32:07] #info Plan for search is to have the jobqueue at eqiad for now be responsible for updating indexes at both locations, this gets us parity for writes. We just now have hardware in for dallas and it's not yet even OS'd. Precursor refactor for jobqueue G235149, there is a one off here that doesn't go through jobqueue that needs some thought: T109126. [21:32:12] anyway, I still need to finish slogging through MW to get to more of the ops side stuff (e.g. the POST/GET vcl logic) [21:32:16] #info And no solution for redirecting read traffic has been sorted out to my knoweldge. Briefly we discussed having an etcd key that keeps the active "svc" url or some such and relying on its multi-dc replication if available. We we want the ability to hit search at either location from either location. But it's an open ended question. [21:32:29] relatedly, the stuff in the RfC about Sticky DC cookies: that's something we can implement wholly at the edge layer too. We'd nominally have a natural traffic split between the two regardless, but we can do the post-POST short cookie thing as well at that layer, so that even if the users falls into a corner case where they bounce between DCs at the edge layer, they still hit a consistent applayer [21:32:35] DC during their cookie window. [21:32:36] at this point, it looks like almost everything at least has a patch, so I hope that's close to wrapping up [21:33:04] I've been looking at DBPerformance logs in Kibana and tracking down/fixing usage patterns [21:33:25] tx ori [21:33:37] * AaronSchulz would also like to get slave lag down a bit, since it affects the WAN cache usage patterns if too high [21:33:46] #info direct FileBackend callers in deployed extensions are ConfirmEdit, Score, Math, GWToolset and timeline [21:33:51] AaronSchulz: is https://www.mediawiki.org/wiki/Requests_for_comment/Master_%26_slave_datacenter_strategy_for_MediaWiki#Deployment_steps still up to date? [21:33:53] row based replication, AaronSchulz [21:34:11] https://phabricator.wikimedia.org/T95501 [21:34:23] +SSDs will solve most of the issues [21:34:29] jynus: yeah, that will be awesome; I suspect there is lots we can do to MW in the mean time [21:35:05] it's not like we never do anything stupid in MW ;) [21:35:06] for mw, join the fight of "no transaction larger than 1 second" [21:35:11] #info if we have to direct thumbnail regenerations to eqiad to keep the listings (for purge) in sync, we can start off that way. I'd like to actually use both DCs for scaling so that would be temporary until something better comes (either the thumbnail redo or just decent swift replication) [21:35:18] :-) [21:35:18] how do we plan to address split-brain in all of this? if the world can still reach both DCs, but the DCs can't see each other, and that condition persists for a notable timeperiod. Do we allow writes to proceed on both sides? does everything related have the ability to handle that? [21:35:29] like I noticed/fixed stuff (e.g. a15cf051885b9) that randomly stumbled across [21:36:02] (jynus: I'd like to talk with you about that, by the way -- I have some ideas for better query performance monitoring. But that's off-topic.) [21:36:05] jynus: ori and I had a bit of talk about tooling for catching slave lag offenders [21:36:15] I miss the days of the ishmael tool [21:36:26] bblack: aren't writes (in the db sense) limited to one DC for now? [21:36:48] #info Slave lag is especially problematic for multi-DC b/c it affects the WAN cache usage patterns if too high. [21:36:49] yes i don't think we have full multi-master on the plan yet... [21:36:54] gwicke: that's one possible outcome, but some things in the RfC seem to sound like multi-DC writes, with some stickiness so that a single user doesn't face replication update woes [21:36:58] brion: baby steps... [21:37:01] :) [21:37:12] (the sticky cookie part) [21:37:24] #action AaronSchulz / jynus / ori to discuss improving observability for query performance [21:37:39] yea, what's the preferred_datacenter cookie thing about? [21:37:42] bblack: yeah, writes can happen across DC for some special cases, but it's heavily frowned upon [21:37:44] bblack: dc stickiness is nice for consistency i think, as long as it's paired with a master version check when doing a read after a write [21:37:46] lag affects read-write/read-only too [21:37:50] hence the whole GET/POST rule [21:37:51] my understanding is that writes to authoritative data at least will be limited to one DC for now [21:37:59] ok [21:38:07] you would configure MW so that the master is in eqiad and the slave is in codfw [21:38:13] though if you've got a version (chron protector) in the cookie ..... maybe don't need the stickiness [21:38:25] #info writes to authoritative data at least will be limited to one DC for now [21:38:27] so in a single-writeable-DC scenario, we still want sticky-cookie, but we want it to mean that post-POST, that user has to directly use the primary DC for reads for a short window as well? [21:38:28] so if MW decides to write to the DB from codfw, it would just be slow, not fatal [21:38:55] *nod* [21:38:55] brion: note that chronprot handles multiple LBs [21:39:14] I guess you could stuff positions in one/several cookies (maybe hmac it for sanity) [21:39:26] that could avoid stickiness for the DB itself, but not other stuff like swift [21:39:27] bblack: that would make sense, to make sure that people see the change they just make [21:39:31] *made [21:39:37] yes, I assumed that if we learn about writes too late, it is an option still [21:39:59] the sticky cookie bit needs to be implemented still, right? [21:40:08] at the edge layer, effectively codfw+eqiad will both be *capable* of backending requests to both codfw+eqiad MW/services. So we're saying GETs would flow to the local DC, POSTs to the write-DC even if it's not local, and POST sets a cookie for the user to keep their reads going there too for a short while. [21:40:30] #info at the edge layer, effectively codfw+eqiad will both be *capable* of backending requests to both codfw+eqiad MW/services. So we're saying GETs would flow to the local DC, POSTs to the write-DC even if it's not local, and POST sets a cookie for the user to keep their reads going there too for a short while. [21:40:36] ori: yeah not implemented. but as I was saying earlier, if it's truly just on POST/GET distinction, we can do that entirely in the traffic edge layer. [21:40:40] what is the schedule? it sounds like the schedule is constrained by ops setting up hardware in codfw? [21:40:43] bblack: nod [21:40:47] Schedule for what? [21:40:49] brion: so yeah, I'd stick with the sticky cookie initially at least [21:40:53] hardware for what? [21:41:02] for the schedule! [21:41:03] :D [21:41:05] search, I heard earlier [21:41:15] ah, yes, that's still pending [21:41:21] search is relatively late to the dallas ballgame. [21:41:23] as is a solution for swift, I guess. [21:41:43] bblack: yeah, just a matter of checking HTTP method as well as the sticky cookie if set [21:41:52] I'm guessing it's gonna take a while to get MW core and its extensions to be multi-DC-ready? In terms of correct WanCache usage, and GET idempotence, etc [21:42:02] right, I'm just saying, all of the cookie part can be out at the edge layer. we don't need to put MW code into it. [21:42:25] we need to match swift capacity in codfw by three machine as in eqiad, in the next swift hw order [21:42:27] the RB dallas hardware has an ETA of 9/15, so we might even have a small chance of making our goal of having replication set up by the end of this month [21:42:30] for swift it sounds like more development work in swift itself is needed [21:42:34] RoanKattouw: MW has come pretty far, and I'll probably finish that soon except a few bits. [21:42:35] and if we face split brain in that world, what will happen is half the world won't be able to do POST traffic at all, until we decide to DNS-move them off to the write-side of the split. [21:42:44] #action To do sticky-cookie request scheduling, all we need to know is the HTTP method, so this could be implemented entirely at the edge layer. This still needs to be done. [21:42:48] RoanKattouw: it could help to implement a check that would log a warning if a DB write is triggered during a GET request. easy enough via global state. [21:43:10] Those 'bits' being Flow & Echo [21:43:11] that should surface most problematic code [21:43:24] they have tasks...hopefully legoktm will get to them ;) [21:43:40] #action Flow and Echo still need to be made multi-DC-ready [21:43:48] :| [21:43:51] I guess the AJAX rollback thing is still unassigned (as of now) [21:43:52] bblack: for now, there would also be a lot of manual switching needed; my impression is that this is some ways out [21:44:06] maybe I can trick Krinkle into doing that [21:44:31] https://phabricator.wikimedia.org/T88044 [21:44:35] gwicke: I'm talking about a scenario where we're not really moving which DC is active for writes. we've just suddenly lost IP comms between the two DCs, but users can still hit the readonly DC [21:45:12] in that case, that half of the users become effectively readonly until ops notices the issue and flips full user load over to the write-DC at the DNS layer to work around it. [21:45:25] OK, so goals for next Q are due for WMF teams later this month, and to protect resourcing for this project (rather than doom it to the "things people do in their spare time because management doesn't understand them") we need some sort of timeline [21:46:21] Getting flow, echo, and ajax rollback ready is prerequisite, yes? and certainly doable in a quarter? [21:46:43] I was looking for a project management page since this seems quite complex from that perspective [21:46:43] as long as the right people are assigned, I don't see why not [21:46:53] I think I found it in the big list of blockers at https://phabricator.wikimedia.org/T88445 [21:47:17] #action Ori to create a workboard for multi-DC work [21:47:40] but maybe we need some prose or a gantt chart or something [21:47:45] ori: thanks for thinking through this stuff [21:47:46] Thanks Ori, I was about to ask something like "where can I find a list of tasks that are in my team's wheelhouse that block this project" [21:47:51] * AaronSchulz likes it when Tim says "prose" [21:47:55] i think patch for ajax rollback is half done, it just needs a little more lovin' to complete it. anybody more interested in it than i am? :) [21:48:03] heh [21:48:06] AaronSchulz: Relaying memcached deletes and purges - is that for the main cache, or for making wancache/stash/persisent stash use memcached. [21:48:17] ori: What is G235149? [21:48:24] there are a lot of bits to this, affecting a few teams, and ideally it would all come together at the same time [21:48:30] Krinkle: it means implementing the relay logic in WAN cache with the requisite services [21:48:52] so that we don't have e.g. hardware in codfw halfway through its warranty before it gets used [21:49:01] #action Q2 goal: ensure all PHP code ready for multi-DC: Flow, Echo, Ajax rollback and the direct FileBackend callers (ConfirmEdit, Score, Math, GWToolset, timeline) [21:49:02] Krinkle: see the EventRelayer part [21:49:03] I'm trying to figure out how to both help ori and AaronSchulz with protecting this work, and figuring out how to avoid a Gantt chart :-) [21:49:38] from the Traffic perspective, we're just pending some network link turnups for fuller capacity (soon) and we'll have user traffic splitting well. we already split a limit number of users to dallas. for getting to the independent tier1s + sticky-dc-cookie, we've got to fix the traffic sec issues in T108580 , and then implement the VCL changes for stickie-dc-cookie. [21:49:40] does EventLayer need a server? [21:49:49] bblack: is there any reason to assume container sync is still is crappy as it was? [21:49:58] my question on that stuff is, what kind of schedule am I on there? when does traffic have to be ready for the rest? [21:50:01] an updated version of https://www.mediawiki.org/wiki/Requests_for_comment/Master_%26_slave_datacenter_strategy_for_MediaWiki#Deployment_steps, perhaps slightly more milestone-y could already be helpful [21:50:06] AaronSchulz: swift container sync you mean? [21:50:16] (bblack was talking about varnish) [21:50:18] There are some tasks which apply to multiple extensions and it would it be easier to track and work on them if they had multiple subtasks (https://phabricator.wikimedia.org/T94480 applies to both PageTriage and Echo for example) [21:50:20] paravoid: yeah; how much does ops want to look into swift replication? [21:50:27] or should I assume it is a MW problem [21:50:33] bblack: I think you are in a position to dictate rather than be dictated to. What do you think is reasonable? [21:50:46] godog was experimenting with container sync some time ago, I'm not sure if he got anywhere [21:50:49] How much time do you need? Would it be sensible to make it a Q2 goal? [21:50:51] and if there was a #Multi-DC project, we could have a phab search for #XX-Team AND #Multi-DC [21:51:10] well reasonable would be not already having a year-long backlog of tasks. I can prioritize, but I don't want to rush through this next quarter if the applayer can't do it by then yet anyways (multi-dc reads, etc) [21:51:39] bblack: the few extensions that were brought up are a small minority; the bulk have already been migrated [21:51:44] paravoid AaronSchulz I didn't go very far, upstream hasn't been investing a lot in it last I looked, I can timebox some time to look into that with recent versions of swift [21:51:47] traffic sec issues in T108580 are an unknown, it's a significant work item that could consume a quarter goal [21:52:01] bblack: traffic security is slated for Q2 according to our roadmap [21:52:07] godog: they are pushing there geo-replication cluster stuff then I assume? [21:52:08] yeah, true [21:52:22] that would be risky to switch to probably, sigh [21:52:35] (not contesting, but trying to understand) how is traffic sec a blocker? is it simply that it is a priority and needs attention from the same set of people? [21:52:39] it has affinity features in theory (accept DELETE afaik), but not sure if trust that yet [21:52:49] if we get that fixed up in Q2, then we're just looking at some moderately-complex VCL work for the cookies. would not be a whole quarter's worth, just needs a couple weeks of dedication. [21:52:50] bblack: I want to +1 ori here on telling us what you need. you should be able to say "we need multiple quarters from multiple teams" if that's what its going to take [21:53:12] AaronSchulz: I'd propose postponing the swift decision (or even discussion) for after this meeting, as what gilles is doing in this space (and other performance team's goals) might be relevant [21:53:48] AaronSchulz: (and not split it just yet between "ops" and MW) [21:53:59] paravoid: yeah, if does content hashing (and some pixie dust for revdelete) then this would be a lot easier, for example [21:54:06] right [21:54:07] #action Swift to be discussed in follow-up meeting. [21:54:08] re traffic sec blocker: the problem is right now with a single tier-1 DC, we have traffic sec nailed down to some degree: we're not leaking user traffic. if we don't solve the really deep traffic sec issues in T108580 before promoting codfw to a tier1 site at the traffic layer, then we regress and start exposing user traffic again (on the codfw<->eqiad link) [21:54:23] I probably need a quarter to fix that whole issue [21:54:29] ok, we've got about 7 minutes left, please write only summaries and #info/#action now [21:54:38] paravoid: we can always keep image CDN misses going to ashburn as I mentioned earlier [21:54:40] time to wrap up [21:55:02] bblack, paravoid, godog: do we agree on postponing swift then? [21:55:18] #info (bblack) with a single tier-1 DC, we have traffic sec nailed down to some degree: we're not leaking user traffic. if we don't solve the really deep traffic sec issues in T108580 before promoting codfw to a tier1 site at the traffic layer, then we regress and start exposing user traffic again (on the codfw<->eqiad link). traffic sec issues in T108580 are an unknown, it's a significant work item that could consume the quarter. [21:55:36] AaronSchulz: ori already made it an action item :) [21:55:39] oops [21:55:43] that works [21:55:43] oh [21:55:51] haha, ok [21:56:02] #info there is a strong desire to get a better understanding of what is left & where help is needed, so that other teams can help [21:56:06] this has been extremely productive from my perspective. can we agree to do this regularly? [21:56:15] jynus: how much time do have to prod at T95501 ? [21:56:20] AaronSchulz, ori et al: let's sort it out in the coming weeks before the end of Q1 between our two teams [21:56:21] and is this the appropriate forum? (i think so, but checking.) [21:56:22] given this is a very large thing, I think it does need some regular syncups going forward [21:56:34] even if it's just file bugs about queries that suck (I don't mind fixing such things :) ) [21:56:36] bblack: yeah, agreed [21:56:58] as well as looking at the tooling aspects [21:57:09] most queries are identified, but needs work [21:57:10] you want another IRC meeting in a month or so? [21:57:12] #action Schedule regular sync-ups for multi-DC work [21:57:17] AaronSchulz: yep [21:57:23] or just a working group meeting? [21:57:28] RBR has a blocker which is labs replicas [21:57:30] TimStarling: yes, if it's cool with AaronSchulz and everyone else [21:57:36] I think this forum works well [21:57:38] ori: yeah, agreed this is productive for regular sync ups [21:57:45] ok [21:57:56] meetings are fine [21:57:58] new hardware- to be deployed thoughout the year [21:57:59] not sure on the medium [21:58:10] I wouldn't mind a conference call [21:58:11] jynus: is that #action or #info? [21:58:13] * AaronSchulz always things of IRC as a fallback [21:58:24] as in "tool X does not support this many people in a video conf" ;) [21:58:36] AaronSchulz, yeah, and keep the Collaboration team posted on what the priority blockers are for Flow and Echo (meeting might also help). Having legoktm do everything is not the only option. :) [21:58:43] ori, info resumed on T95501#1577186 already to fix lag issues [21:58:45] hangouts are great if we want to waste the first 40 minutes figuring out why it's not working [21:58:47] bluejeans would support that many people, wouldn't it? [21:58:52] yeah [21:58:52] https://meta.wikimedia.org/wiki/IRC_office_hours isn't just for RFC meetings [21:58:53] AaronSchulz: I'm willing to support whatever the main participants think is best [21:58:58] I think action items are generally short tasks that are mostly administrative [21:59:11] #summary new hardware- to be deployed thoughout the year [21:59:15] TimStarling: yeah [21:59:19] action items shouldn't be used for the actual work product [21:59:25] bluejeans worked well the last time we used it, and it supports up to 100 participants [21:59:38] bluejeans still hasn't launched WebRTC, has it? [21:59:39] i'm going to just lump that into #action Schedule regular sync-ups for multi-DC work [21:59:52] ok, I can't think of anything I need to say before we wrap up [21:59:55] paravoid: we used webrtc the last time, and didn't have issues [22:00:04] TimStarling is AaronSchulz 's https://www.mediawiki.org/wiki/Requests_for_comment/Master_%26_slave_datacenter_strategy_for_MediaWiki#Proposal approved ? [22:00:05] we just need to follow up on getting meetings setup [22:00:17] TimStarling: really? I guess that makes sense if "file x in phabricator" is the administrivia [22:00:19] ok, then I'd be willing to try it, although I'd honestly prefer IRC [22:00:24] especially for this size [22:00:25] Thanks SO MUCH to everyone who participated, I think that this has been extremely productive, and I feel a lot better about where we're at. [22:00:28] * ori prefers IRC too [22:00:38] * Krinkle also prefers IRC - asynchronous IO for the win. [22:00:52] Yeah IRC was great for this meeting [22:00:53] * robla thinks IRC is working well for this group, and likes it too [22:00:55] really agreed [22:00:59] I guess we can call it approved [22:01:01] heh [22:01:05] I know how they say in-person is higher bandwidth, but with a group this big I don't think that's true [22:01:10] Also easier with less accent variation, and able to re-read, and natural notes taken :) –plus urls rich content [22:01:15] we can't approve every last detail but it is broadly going to happen [22:01:23] yay [22:01:32] also, notetaking is easier on IRC :-) [22:01:34] #agreed RFC approved [22:01:35] there's a lot to be worked out still, I think [22:01:39] #endmeeting [22:01:41] Meeting ended Wed Sep 2 22:01:39 2015 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) [22:01:41] Minutes: https://tools.wmflabs.org/meetbot/wikimedia-office/2015/wikimedia-office.2015-09-02-21.01.html [22:01:41] Minutes (text): https://tools.wmflabs.org/meetbot/wikimedia-office/2015/wikimedia-office.2015-09-02-21.01.txt [22:01:41] Minutes (wiki): https://tools.wmflabs.org/meetbot/wikimedia-office/2015/wikimedia-office.2015-09-02-21.01.wiki [22:01:41] Log: https://tools.wmflabs.org/meetbot/wikimedia-office/2015/wikimedia-office.2015-09-02-21.01.log.html [22:01:42] \o/ [22:01:47] but there's very broad support for the direction [22:02:01] \o/ , is it worth flagging the accepted/still-contentious parts of the RFC #Proposal section ? [22:02:31] excellent work AaronSchulz! [22:02:44] +1 [22:02:44] thanks, everybody! [22:03:41] Yep, this is great stuff. [22:04:01] there's always been a lot of difficulty in marking a work in progress as "approved" [22:04:15] or approving anything, really [22:04:20] conceptual approval anyways :) [22:04:32] in practice only the shortest RFCs have been approved [22:04:34] right, it doesn't mean "we'll plough on even if we discover a fatal flaw" [22:04:37] so we've approved that we need to move in the direction of having more things to approve? :) [22:05:27] I don't know if we should put this in the approved column on the phab workboard since that will make it less visible for scheduling [22:05:57] but we've been talking about redoing the columns anyway [22:06:25] maybe we should have "in progress/check in" [22:12:58] TimStarling, robla: anything in "approved" is not "implemented" (separate column) thus is in progress, I agree it's hard to know which RFCs to check up on [22:14:03] if you click the anchor you see most RFCs don't have a priority, so ArchCom could use that to flag the RFCs to attend to [22:41:10] spagewmf: sorry I missed this part of the conversation. Yeah, in short, I agree that we need more clarity here [22:42:16] * robla contemplates which channel this conversation should move to, and tries to resist the temptation to create a new channel [22:42:43] robla: sure "Add a phab comment when RFC status changes" is good advice, but reviewing comments doesn't work to track RFCs [22:43:19] robla: there's #wikimedia-devtools for Phab, and maybe a Team Processes Group IRC channel ;-) [22:43:41] * robla thinks #wikimedia-tech may be the right home for "Architecture" conversations [22:46:32] robla: sure, I'm there. The #wikimedia-devtools and #wikimedia-teampractices are good to get advice on detailed Phab usage [22:50:04] * robla moves conversation over to #wikimedia-tech [23:21:34] AaronSchulz: I wanted to talk about your RFC in #wikimedia-tech, but you aren't there