[00:02:49] 10Traffic, 10Operations, 10Browser-Support-Apple-Safari, 10Upstream: Fix broken referer categorization for visits from Safari browsers - https://phabricator.wikimedia.org/T154702 (10Tbayer) I added an entry about this to the log at https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Pageview_h... [06:15:56] 10netops, 10Operations: pc2007.codfw.wmnet network blip? - https://phabricator.wikimedia.org/T211405 (10Marostegui) [06:19:30] 10netops, 10Operations: pc2007.codfw.wmnet network blip? - https://phabricator.wikimedia.org/T211405 (10MoritzMuehlenhoff) See IRC/internal channel, during the setup of logstash2001.codfw.wmnet it accidentally reused the 10.192.0.104 A record. [06:21:17] 10netops, 10Operations: pc2007.codfw.wmnet network blip? - https://phabricator.wikimedia.org/T211405 (10Marostegui) 05Open>03Resolved >>! In T211405#4805188, @MoritzMuehlenhoff wrote: > See IRC/internal channel, during the setup of logstash2001.codfw.wmnet it accidentally reused the 10.192.0.104 A record.... [09:23:35] 10netops, 10Operations: pc2007.codfw.wmnet network blip? - https://phabricator.wikimedia.org/T211405 (10Peachey88) [09:31:25] ERROR: Missing PTR '115.0.128.10.in-addr.arpa.' for name 'cp4015.ulsfo.wmnet.' and IP '10.128.0.115', PTRs are: ['5.1.1.0.0.0.0.0.8.2.1.0.0.1.0.0.1.0.1.0.3.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa.'] [09:31:29] ERROR: Missing PTR '116.0.128.10.in-addr.arpa.' for name 'cp4016.ulsfo.wmnet.' and IP '10.128.0.116', PTRs are: ['6.1.1.0.0.0.0.0.8.2.1.0.0.1.0.0.1.0.1.0.3.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa.'] [09:31:33] etc. [09:32:38] can someone fix those? [11:07:30] paravoid: those two hosts should be decom'ed (T176366) [11:07:31] T176366: Decom cp4005-8,13-16 (8 nodes) - https://phabricator.wikimedia.org/T176366 [11:08:34] it seems that https://github.com/wikimedia/operations-dns/commit/8a7cd08bdb88050564636bb148c535d31706d912 was incomplete, fixing [11:15:21] mmh no wait, that commit is fine [11:15:33] paravoid: where did you get those errors from? [11:20:26] oh, I see, 8a7cd08bdb88050564636bb148c535d31706d912 removed the mgmt entries instead of the prod ones [11:20:53] well, it's quite tricky all the errors there :) [11:21:04] it removed the forward mgmt dns, and the reverse IPv4 production dns [11:21:04] confusing, yes! [11:21:21] but left at least the reverse dns for prod and the forward for ipv4. I guess forward ipv6 was never there? [11:21:45] err, left the v6 reverse for prod and the forward to ipv4 for prod [13:15:45] bblack: should we resolve https://phabricator.wikimedia.org/T199675 ? [13:21:57] I should probably check first and see if any new EDAC errors appeared that went unnoticed [13:22:39] I'm kind of dubious of the "fix". Did the other BIOS rev have a known bug that caused false reporting of DIMM errors when the DIMM itself was fine? [13:23:27] or worse, does the new BIOS just suppress them? :) [13:24:51] 10Traffic, 10Operations, 10ops-eqsin: cp5001 unreachable since 2018-07-14 17:49:21 - https://phabricator.wikimedia.org/T199675 (10BBlack) 05Open>03Resolved No new EDAC errors reported since repooling, all we can do is assume it's ok for now I think. [13:25:23] the whole thing is kind of silly. they were scheduled to do hardware work and that to be scheduled for months, then it's a bios update and we're done. [13:25:32] s/that to be/that failed to be/ [13:28:25] in general I'm overdue for a #traffic phab cleanup [13:28:44] at least, merge/close obvious things, kill tickets obviated by the passage of time, etc [13:28:58] possibly re-org the board a better way than it is now [13:29:22] it'd be nice to push all the wishlisty things away from the real work queue (or vice-versa. I think we have far more wishlist than imminent work-queue) [13:35:36] really, phab doesn't fit the wishlist thing very well at all. I'm starting to think it's a bad idea to ever have "idea" tickets that describe something half-baked that we might like to do but lack the time [13:36:04] maybe some other forum for those things, like some wiki page that outlines known/probable future plans/ideas [13:38:56] I'm not sure I agree [13:39:18] but I'm not sure why you think it doesn't fit the wishlist thing very well yet, so hard to disagree either :) [13:39:33] well, they're not really tasks [13:39:54] I guess the meta-problem that makes me look at this, is that our dashboard only gets one dimension [13:40:14] right now we're using the dimension of sub-topics for our columns, because there's a lot of logical sub-groupings [13:40:29] yeah I've noticed that, I haven't found that useful myself [13:40:53] e.g. when #netops switched from kanban-like to grouping I got all confused [13:40:56] well, it's something we did a couple of years ago, I don't like it all that much anymore anyways [13:41:21] and I don't get any value from the topic grouping myself, maybe you or arzhel do [13:42:07] other teams do multiple dashboards to address the multiple dimensions issue [13:42:35] yeah that was my next question [13:42:38] see "workboards" on the left of https://phabricator.wikimedia.org/project/view/3634/ [13:42:49] I think you still need to assign multiple tags per task, though [13:42:50] not sure [13:43:51] yeah CPT has a whole interesting and complex layout using milestones and such too [13:44:09] I haven't explored any of that stuff, or how to create extra dashboards, etc [13:45:01] I'm not sure that I can, on my own [13:45:21] (as in, I don't think I have phab permissions to make major edits to how #traffic is organized) [13:46:01] but anyways, wishlist stuff could perhaps be pushed to some subproject tag (e.g. #traffic-wishlist) [13:46:12] but that also seems like a poor mental model, I don't know [13:46:36] tasks systems aren't well suited to recording half-baked future ideas nobody has committed to or integrated into some long-term work planning. [13:48:03] where do they go in kanban? We have like 357 open tasks at present. Maybe with cleanup we get that down to 200, but a realistic next 1-2Q's work queue for the resources we have, even with subtasks split out and goal meta-tasks and all, that would be well under ~30 tasks typically. [13:48:21] so almost by definition, ~80-90% of the tasks we have aren't imminent planned work in any real sense [13:49:16] and it's much more palatable to not break things up by our internal subjprojecting, if the total list is much smaller. [13:49:38] then a team level kanban layout doesn't have infinitely-long columns you can't visually track anything on anyways [13:50:42] you can keep a column that's "long-term ideas" [13:50:44] or epics, or whatever [13:51:07] and then columns for backlog -> up next -> ongoing, and then maybe one for externally blocked? [13:52:00] yeah but is that column even useful, or is it a bin of ~300 random tasks that becomes basically invisible because the only list of them is too long to parse [13:52:19] the one all the wishlist singular ideas fall into, I mean [13:52:46] I'm just saying, it's not useful now, and just leaving the existing tasks as they are and reorging to kanban isn't useful for doing something with those tickets either. [13:53:11] the topical splitting was mostly to make the total work in each column somewhat mentally tractable [13:54:07] (but even then, it fails at that. We have topical colums that have counts as high as 90 and 40, as examples, and those are important areas) [13:56:02] but the proposal is what? to move those 300 random tasks to a wiki? :) [13:56:14] that would make them even more invisible I think [13:56:26] and harder to reference, track as dependency etc. [13:56:43] the other idea would be to just not go through the trouble of filing them at all [13:57:02] sure [13:57:17] or in terms of what I can re-org on the existing set: delete them [13:57:35] it just seems a shame to delete them without capturing them in some other form, if they're useful for some kind of future planning [13:58:02] yeah [13:58:16] tbh, personally, if anything, I think we should be documenting /more/ those future ideas, not less [13:58:32] e.g. this anycast stuff and the bird deployment etc. we were talking the other day [13:59:03] I can see lots of sides to it, and the bird/anycast stuff is different kind of case since there's actual work being done in the background [13:59:14] I saw SAL items last night [13:59:18] with no task referenced [13:59:26] and still no task I could see of [13:59:29] ? [13:59:51] 23:47 XioNoX: troubleshoot bird bfd on dns2001/cr1-codfw [14:00:11] I suppose that's https://phabricator.wikimedia.org/T209989 [14:00:29] well that's a separate issue from how we re-org tasks and ideas [14:00:37] sort of! [14:00:41] eh [14:01:05] if there's a SAL entry, someone's doing something, so clearly there should be a task, and clearly good practice would be to reference it [14:01:15] https://phabricator.wikimedia.org/T98006 (Anycast DNS) is a good example of a "future idea" task though [14:01:19] it's not really a topic for how to org tasks better, it's just a declaration that someone did something wrong [14:01:44] something that doesn't exist yet, and probably won't get implemented in the immediate future [14:01:54] or am I misunderstanding the type of task you're talking abut? [14:02:18] well ideally that Anycast DNS ticket needs some refactoring. It evolved over time in content/meaning but not structure. [14:03:06] here's a better example: https://phabricator.wikimedia.org/T170567 [14:03:23] filed in July 2017 (1.5 years ago), virtually content-free except to name a desirable protocol, still not in immediate planning [14:03:39] all that ticket does is inflate ticket count :P [14:04:00] and recording a few interesting thoughts that happened along the way, I guess [14:04:06] well ok, the "content-free" thing I agree with, I don't like tasks that are "as subject says" much in general [14:04:14] even ones that are being actively worked on [14:04:19] well [14:04:22] and I see that often (not in traffic, just in general) [14:04:39] I think that gets to the core of the debate about the purpose of phab tasks [14:04:59] but yes, the commonality between this and anycast DNS is that maybe there is /some/ progress at some point for some reason and it's useful to capture that and how it's evolved [14:05:11] are they to record work that needs to be executed and its execution, or to have a broad/open design discussion and record the output of that before we implement, or somewhere between? [14:05:43] because it seems like for task-execution work, content-free tasks are ideal. this is like a meta-level of "don't wade into design discussion in a gerrit -1 review, go to the task" [14:05:46] I think (c), and possibly (b) for different types of tasks (e.g. TechComm's RFC tasks are like that) [14:06:15] "don't wade into design in a task for executing X, there are other forums where the design of X and the decision to do it were made" [14:06:38] like where? :) [14:06:46] I mean sure, in-person conversations and meetings and IRC and whatnot [14:06:51] but still, that should be captured somewhere [14:07:11] exactly, like where [14:07:16] rationale and decisions taken and actions taken [14:07:30] that's why I mentioned bird above [14:07:42] I think all over the org, SRE, and even within traffic, we have differing and overlapping views of what things do or don't belong in tasks at all, or where certain levels of work get done. [14:07:55] (and not to point any fingers or say a mistake was made, to be clear :) [14:08:43] I don't think it's an ideal tool for capturing a random idea and then slowly fleshing out the design of it over time and/or eventually getting it priority as a work item. [14:08:50] that statement (differing and overlapping views) is certainly true [14:09:05] but that's unsolvable :) [14:09:20] let's just find some common ground and do at least that, I'd say :) [14:10:11] well, let's find the common ground and document it as best practice. Everyone having different personal views is intractable, but without some central, documented, shared idea of what we're actually doing in practice, we're all left to our differing interpretations of reality. [14:10:39] ok [14:11:01] or let that happen per-team I guess is an ok answer too, but then it needs to happen :) [14:11:15] maybe everyone has different ideal ways of using a task system [14:11:37] I'm sure [14:11:50] so things categorically like the TLSv1.3 ticket [14:11:57] but the general principle I'd like to apply is... lean on overcommunicating, not undercommunicating in favor of other forums [14:12:20] we have a fairly useless task now that's sitting in an overloaded workboard column somewhere being ignored [14:12:22] i.e. a paragraph in the task description that describes the problem, and the surrounding discussions [14:12:59] and things like "we decided to try out bird for X, Y and Z reasons" and "we installed bird in system A as a testbed" be documented somewhere [14:13:06] sorry to mention bird and/or #netops all the time [14:13:09] there's use in having some record of "we need to do TLSv1.3, here's why, and here's what's blocking it (which may be just resourcing)" [14:13:20] I don't mean to suggest that this is the biggest issue or anything, just one that comes to mind [14:13:29] and #netops is more familiar to me than broader traffic [14:13:35] not personal or anything :) [14:13:46] yeah, agreed [14:14:04] but really even if we had well-formed tasks around all things like that TLSv1.3 ticket, they're still a jumble a kanban board won't sort out [14:14:42] no, my suggestion on that was that it would unclutter the views that matter for e.g. your weekly or monthly planning [14:14:45] the next step in organizing all such ideas is really to pull them into something like a loose Gantt chart of the team/org's long-term plan, if we think they fit into it at all (and if not, then why file it?) [14:15:21] and I don't think, as far as I'm aware, that phab is a good tool for that [14:15:40] why not? [14:16:06] is there a tool for that in phab somewhere? [14:16:10] to do what? [14:16:30] phab has reasonable notions of dependent tasks [14:16:35] to organizing the longer/bigger-picture view of this [14:16:59] yeah it does have dependencies, but it doesn't have a graph view or a good UI for sorting them out in a much broader picture [14:17:16] i mean, it does have a sort of tree view? https://phabricator.wikimedia.org/T205862 [14:17:22] I still don't understand the issue tbh :( [14:17:39] another task I just saw fly by: https://phabricator.wikimedia.org/T211254 [14:17:55] what kind of additional metadata would you need there? [14:18:38] it's not just a metadata problem, although sure additional fields might help with other aspects of this [14:18:55] oh ok [14:19:11] I mean the tooling is task-centric, not planning-centric [14:19:17] the way the UIs work [14:19:32] could you give an example of the type of thing you're thinking of? [14:19:59] I want something at least as good as a drag-n-drop gantt chart that can fit many many tasks and pull them around into dependencies and resource counts over time, etc [14:20:00] that would help me a lot I think [14:20:12] if I were to put all of this in phab instead of "some other forum" [14:21:27] gantt would necessitate predicting time a potentially very future task would need, well in advance [14:21:40] "Implement TLSv1.3" might have an estimated resource count field of 1.5 FTEQs, and 50 such tasks might be in our ~3yr view, and they have inter-dependencies that determine their order, and then resource totals in any given area of the depdendency windowing which we can match to available resources (which stretches or shrinks the notional abstract time of a simple dependency chart, into realtime s [14:21:43] s/time/time a task requires/ [14:21:46] lots) [14:22:04] we have to predict/guess time, yes, but that's a requirement of all planning. You iterate and edit as you go. [14:22:39] what I'm trying to say is that it sounds like a very waterfall-y view of planning [14:24:26] as opposed to the fast iteration cycles we're doing now that leave stale idea tasks laying around for 1.5 years with no tooling to plan for their eventual execution? [14:24:35] I don't think the waterfall sounds that bad at this point :) [14:24:50] yeah maybe that's at the core of the argument then :) [14:25:45] so the subcontext we're in right now, to be clear, is "how would we fix this where we keep it all in phab and it works well" [14:25:58] and I'm suggesting what tooling phab might need to accomplish that [14:26:14] ehm, so and so [14:26:31] the starting point is, yes, we could totally clean up our long-range tasks a bit, and flesh them out, and make sure their dependencies are sane [14:27:01] and let's say stuff them all under a meta-epic called "long range planning" so we can see the little dependency tree underneath it [14:27:02] I think we're talking about multiple problems simultaneously :P [14:27:08] or perceived problems [14:27:34] so now I've got these 50 or whatever bigger-picture future tasks organized in a tree [14:28:21] now my next problem is assigning resource estimates to them, and organizing them into a timeline estimate of real quarters, based on the dependency flow and how many resources we have to even work on them per quarter. [14:28:41] yeah, I personally wouldn't do that [14:28:45] but ymmv [14:28:48] and for bonus points, being able to adjust resourcing estimates and get different outputs [14:29:25] I wouldn't attempt to pre-compute resource estimates for a large number of tasks at the same time, and especially long-range/term tasks [14:29:27] e.g.: if we have the engineers we have now, we reach important goal "Deploy foo" 3 years from now. If we manage to hire a new effective resource 6 months from now, and another 6 months after that, it gets done in 2 years. [14:29:34] s/pre-compute/predict/ [14:29:42] how else do you guess at what to ask for? [14:30:03] this all comes down to being able to ask for resources to get the things done that seem important to do, and all the leading dependencies. [14:30:19] I perceive that whatever we're doing now, isn't very effective at that [14:31:10] I don't see how this conversation isn't going to become a "is waterfall a good practice" debate :P [14:31:52] I basically don't even believe that "if we have the engineers we have now, we reach important goal "Deploy foo" 3 years from now" is a thing I could reasonably say [14:32:44] but I can say that, about some things, just on my own tool-less rough planning and intuitions about the subject [14:32:54] I want tools to do that better, and more concretely, for more things [14:33:33] I know about how long it's going to take us to reach various subparts of the ATS work, and I know certain other important things depend on it. [14:33:44] Ditto the stages of certcentral, and the other TLS-related things that depend on that. [14:34:01] and how ATS (front and back, separately) interact with TLS-related issues in interesting dependency ways [14:34:38] and separately, how certain parts of these and other things need to be ready before we can effectively/efficiently do minipops, which tie into the long-range planning of edge sites [14:35:03] these can all be roughly estimated in quarters-ish timelines based on resourcing, to reach desirable end-states in a fairly waterfally way [14:35:23] but there's a lot to it, and I'm wishing for better tools [14:36:28] not just to lay out the dependencies themselves, but the estimates of the resourcing to complete them and how resource changes affect the timelines to reach various desirable end goals, so that I can argue for them. [14:36:40] yeah well, as I said above, I don't think we're even sharing a common view on how we can organize/plan work, so not finding common ground on the solution is just an artifact of that :) [14:36:49] and that's ok :) [14:37:47] what's the alternative view? [14:38:09] I mean, I know there are alternative views, but give me one you think would solve my root problems better :) [14:38:44] https://www.quora.com/Why-are-software-development-task-estimations-regularly-off-by-a-factor-of-2-3/answer/Michael-Wolfe :P [14:39:13] yes I've read that and similar before :) [14:40:15] but (a) I'm talking about larger and different scopes than development timelines for a software feature or whatever [14:40:48] different how? [14:41:00] and (b) You can't just say "planning sucks so don't bother planning". You still have to plan, and you still have to decide on sane resourcing asks to have good odds at making the plan work on some reasonable timeline estimate [14:42:06] I didn't say "don't bother planning" obviously [14:43:10] different how: you can take the pessimistic or optimistic approach to how that blog post scales to larger scopes. The pessimist view is "If I can't accurately plan how long it takes to do one feature request, and the total planning-horizon involves 50 feature requests, we're doomed because all these errors multiply" [14:44:00] The optimist view is "All my small-scale estimates will be wildly wrong, but if I'm even remotely good at getting them in the ballpark and being conservative, the long-run averages come out ok and you can still have a high level view of the long timeline that works" [14:46:05] and also, perhaps another clarifying point: I don't think we can accurately predict where we land in 3-5 years, at all. [14:46:21] I'm squarely on what you describe as the pessimistic view, and believe our longer-term planning should be very high level without such precise time/resource estimates, and then only get into the specifics (like who much time it would take us to implement X) once we are close to actually implementing an item [14:46:41] it's a tool, to make an editable guestimate you can plan the immediate work and immediate resourcing asks, based on guestimates of the longer-term fallout [14:46:50] I don't think 3-5 or 2030 planning makes a lot of sense, not unless it's something extremely vague [14:47:02] and every N (quarter? year?) you take a look at where you've gotten and what's changed, and you tweak all the values and regenerate it [14:47:26] so 5 years from now, I totally don't expect to land exactly where my 5 year timeline said today. [14:47:42] I just want the estimated 5 year view, so I can use it do something rational now and keep editing it as we go. [14:47:43] even annual planning is a bit too far off for my taste, but I cope by making a) high level plans and b) aspirational roadmaps that are not used for resourcing [14:48:09] yeah, I personally think it's a futile exercise to predict a 5y view [14:48:50] but we're talking about projects that sometimes span more than year, and have dependencies and resourcing to consider [14:49:06] if you can't have even a guestimate of how it works out over 3-5, you're flying blind and not getting resources [14:49:29] (or over-resourcing if money is being thrown at you like crazy in some other world, but that's not the case here!) [14:49:55] I don't agree, but ymmv :) [14:50:19] we've done this 1y roadmap ~6 months before the FY started, and we're about 6 months in [14:50:25] questions for you: [14:50:31] a) how is that going? [14:51:06] and b) if not well, is it because you hadn't done more accurate planning by making resource estimates for all the subtasks of each of those estimates and adding them up? [14:51:37] our 1 year map is pretty ok [14:51:51] I'm not worried about that scope [14:52:15] is it? [14:52:34] I see that list and I don't think even half is going to be done by June, although I'm not sure [14:52:39] I don't have a clear enough view probably [14:52:49] got the link? [14:52:51] (nor do I think it's a problem for that to not finish by then!) [14:52:59] https://docs.google.com/spreadsheets/d/1DmqGSMV-2pUM70lyCDRaNkvM2gIJPFD8DVfbujMqZ0Y/edit [14:53:25] I love how we never got past "unschedule" [14:53:30] yes :) [14:53:51] so yeah, my "1 year map is pretty ok", is not based on that list [14:54:05] heh [14:54:07] we're doing pretty horrible at that list. we never had a hope at it though, we didn't have the resources to tackle that list anyways [14:54:12] that's your list too :P [14:54:45] it was a collaborative wishlist. it's not what I made my quarterly plans from obviously, although there's overlap [14:55:26] on my lists... I think we're going to miss only 2-3 fwiw [14:55:35] not that this is an indication of anything [14:55:37] becuase you rejected unresourced ideas, or? [14:55:40] other than luck [14:56:15] well, that, plus trying to not make it too much of a wishlist (but still aspirational), plus... luck [14:56:26] yeah well not everyone on that list even came from me :) [14:56:31] err, everything [14:56:49] but yeah, there was an implicit step afterwards where we decided what we could actually do [14:57:17] e.g. page composition we stalled on ATS, which is a long-term project known to go beyond that 1yr plan. [14:57:43] "reliable purging" just didn't make the cut because it's not perceived to be a big enough trash fire to divert limited resourcing to it. [14:58:00] multi-dc: true progress is again stalled on ATS, so no real goal in this FY [14:58:08] etc... [14:58:30] that was known when that list was made, so maybe the hope was that ATS would had been ready earlier? [14:58:34] I don't know nor remember [14:58:40] mark: nice to see you again pushing code to pybal [14:58:42] no, we know ATS wouldn't be ready in time [14:59:15] ha [14:59:20] nice to see you back then ;-p [14:59:40] as far as 1-year horizons go, our rough plan for this whole FY is just to have it replace varnish for backend caches. Which we may or may not succeed at, but anything else that's ATS-dependent goes at least into the next FY (or further, if it also needs ATS frontends) [14:59:57] err I should've said "in the context of ATS" somewhere at the start of that [15:00:52] mark: still on vacations.. but I miss you guys [15:01:32] but this gets to the heart of how guestimate-y all this is [15:02:03] for all we know at this point, we may never get ATS frontends (or backends) into production. We may discover fairly late in the game that there are serious design issues with ATS that make it unsuitable in practice. [15:02:12] it's all just best estimates/guesses [15:03:28] in any case, the whole of the varnish->ATS conversion is a >1yr scope project [15:03:52] the whole of the perceivable scope of things we need to do in the TLS space is also a >1yr horizon, and the two interact [15:04:44] planning for 1-2+ edge DCs out into the future is also a multi-year effort. And all 3 of these and everything related are coming from a small finite resource pool (which is part of the timeline stretch factor for sure), and they all have complex inter-dependencies [15:05:14] you can cut the dependency cords if you're willing to re-do work as parts of things shift I guess, but that just means even more work. [15:05:30] so yeah, I need the ability to estimate and plan in >1yr chunks, I think. [15:06:58] I don't disagree in planning for a longer-term view (I most definitely do too), just not precisely at all, and without counting how much time X, Y and Z are going to take and adding that up [15:07:04] and be able to say things like "I guess we'll reach X state for TLS affairs important for censorship in mid-2020 and Y state for global latency in mid-2021 under a minimal plan of resource acquisition over the next FY or two, and those dates can at best shift up to these other earlier times if we go with this more aggressive resourcing plan" [15:07:41] it's hard, there's a lot to it, it's hard to do it right with "vi" or with our current phab workboard system [15:08:18] and this all ties back to "why are our wishlist tasks a giant unmanageable blob in phab, and should I move them somewhere else to record the long-term plans/ideas better, or find better tooling for planning, or whatever" [17:13:06] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10Andrew) *bump*