[17:52:42] hiya Silke_WMDE ! [17:53:06] hi addshore [17:53:08] :) [17:55:50] Silke_WMDE: I just sent an email to labs-l with a tentative migration timeline. The interesting dates are… beginning: 3/3; try to finish up by: 03-18 [17:56:05] Hm… interesting variety of date formats there [18:00:13] Hello everyone and welcome to the office hour! We are here to give status updates about the migration from Toolserver to Tool Labs. [18:01:15] Who is in the channel for this? andrewbogott? akosiaris? Kolossos? scfc_de? [18:01:20] hello [18:01:24] hello Silke_WMDE_ [18:01:28] Hey. [18:01:34] o/ [18:01:37] * Thiemo_WMDE is here to listen [18:01:43] Hi! [18:02:43] andrewbogott: Would you like to start by giving us an update about the migration of Labs to equid? [18:03:06] Sure. I just sent an email to labs-l with an update as well. [18:03:15] Time files when you're having "fun". Sorry about my arriving a bit late. [18:03:37] We've spent the last few weeks setting up a new labs cluster in eqiad. It's mostly working, and will be very similar to the current setup in tampa. [18:03:44] Hi Coren :) [18:04:11] The few exceptions: more space for instances, universal adoption of NFS for shared storage, a newer version of OpenStack (which should be invisible to users for the most part) [18:04:48] It's not ready for volunteers to fuss with, but Coren and I are hoping to do a few test migrations yet this week or weekend. [18:05:00] andrewbogott: Sounds good. Is your plan still to move Tool Labs as the first project? Do you have an idea about when you'll do this? [18:05:49] Coren will be working on that. He's not exactly going to 'move' anything -- rather, he's planning to set up a new toollabs cluster (better, stronger, etc.) and then migrate tools individually across. [18:06:23] o_O [18:06:28] The actual tools probably won't start to move until, say, the first week of March. [18:06:30] sounds like a lot of work [18:06:42] ok [18:06:45] Well, it's all puppetized so it should mostly just be a few checkboxes. [18:06:54] i see [18:06:57] Coren, is this accurate so far? [18:07:04] Indeed, I will be using pretty much the same scenario as the overall migration: the new tools infrastructure will be available for users to self-migrate their tools for a period of time before stragglers are moved. [18:07:28] Oh, ok -- maybe you should explain what 'self-migrating' a tool looks like [18:07:37] * Silke_WMDE nods [18:07:43] Self-moving has the advantage that the current things keep running without interruption while you make sure the new home works fine. [18:08:28] For /most/ tools, migrating from one infrastructure to the other should be as simple as copying the contents of your home(s) over, and starting your tool again from the new tools-login [18:09:08] Hm. So should I motivate further people to move their tools off the toolserver to the old Tool Labs right now? [18:09:20] So endusers can do so on their own schedule with no interruption. [18:09:41] I mean if they have to migrate twice that's not an incentive. [18:09:43] Silke_WMDE: The window is fairly small, and the work done on the tampa labs is not wasted; so I wouldn't recommend changing anything. [18:10:13] Silke_WMDE: And, at the end of the period, those tools that haven't been moved will be copied and restarted en masse. [18:10:22] ok. [18:10:34] What do the maintainers risk? [18:10:53] Willl they be notified to be online/available at the moment just in case? [18:11:42] I'll publish schedules for the major milestones, but once I start doing the mass copy there is nothing they can do until it's done. Self-migrating is the way to make sure their schedules are met. [18:12:08] ok [18:13:00] There are very few changes between the pmtpa and eqiad tools setup, and most will be invisible to maintainers. [18:13:11] Coren: I prioritized some BZ tickets for after the migration. Did that help? [18:13:35] Silke_WMDE: It will, once I'm done with the migration proper which is taking pretty much all of my time atm. [18:13:42] sure [18:14:31] Will there be for the interim period hostnames like tools-eqiad-login.wmflabs.org? [18:14:56] good question, thanks scfc_de! [18:16:15] indeed, and at what point would the tools.wmflabs.org hostname switch? [18:17:25] I know that moving the hostname to point at the new instance is easy, but I don't know quite when that should happen. At the very end, I'm guessing… Coren? [18:18:39] For public names, there will be -eqiad equivalents during the migration. [18:18:39] Coren Also: Will you post your schedule on labs-l or will you "hide it in a wiki"? ;) [18:19:12] After the migration, the "normal" names will be moved to eqiad. [18:19:26] The schedule, as soon as it is fixed, will go on-wiki and on labs-l. :-) [18:19:43] Thanks. [18:19:44] :) [18:19:47] tools's migration schedule depends on the main one; I should know the exact dates within the week. [18:20:29] scfc_de: Another point is for tools.wmflabs.org; the proxy will handle both sites. [18:20:51] scfc_de: So it will redirect according to where the tool currently lives. [18:21:06] wow, nice! [18:21:12] nice indeed :) [18:21:45] (How can it know? Especially if the old one is still running and the new one is, too...) [18:23:12] Silke_WMDE: The tools register at the proxy with the host they're running on, so it doesn't matter if the host sits in eqiad or pmtpa. [18:24:44] akosiaris_away: Are you available? [18:25:56] Or do you, Coren and andrewbogot, know anything about the status of the postgresql server? [18:26:22] Silke_WMDE: Last news I heard the server was up and running; what's missing is mostly the sync-with-osm bit. [18:27:04] Is that an easy part or a harder part? [18:27:29] My understanding is that this is relatively easy, but Alexandros is the only one who can tell you with certainty. [18:27:55] Silke_WMDE: i have a question for you about https://bugzilla.wikimedia.org/show_bug.cgi?id=57876 -- what has already done and what is scheduled / planned? [18:28:07] or maybe Kolossos - did you hear from akosiaris_away? [18:28:19] No, nothing. [18:28:23] I've had my head down working on the migration, so I haven't kept up to date in the details. [18:28:35] drdee: I'm looking it up, hang on... [18:28:42] thanks! [18:28:50] drdee: That's high priority and will be done soon after migration. [18:29:13] exactly [18:29:26] any more specifics? [18:29:30] Quite a lot of people are waiting for that one. [18:31:29] Coren: Can you give us an rough estimation of how much time this will take once the migration is done? [18:32:02] Coren: mark wants to know what has happened and what still needs to be done, maybe the ticket can be updated with that information? [18:32:32] I haven't looked at the overall picture of everything that's high priority; there isn't all that much work to do, it's mostly a question of writing the federated table definitions -- about a day's work. [18:32:59] k [18:33:19] The question hasn't so much been "too much work" as "not enough time". :-) [18:33:21] Coren: *I* just set a few tickets to high prio, not that many. [18:33:49] drdee: Realistically, it should be within a week or two of the tools migration's end. [18:34:01] drdee: More likely one than two. [18:34:12] that sounds good to me [18:34:30] A question not directly related to Toolserver/Tools: Projects will have two /data/projects? One in pmtpa, one in eqiad? [18:35:03] scfc_de: That's correct. We're only going to copy the contents when doing forcible moves. [18:35:10] Coren: I'll add "1 to 2 weeks after eqiad" to the BZ ticket drdee mentioned [18:35:22] thanks! [18:35:41] Coren: And they can be both mounted on one instance for migration? [18:36:19] scfc_de: Probably not; NFS is really crappy over high latency. But it will be possible to scp from one to the other, and I'll look into making rsync available too. [18:36:41] Coren: k [18:38:25] Do you know about anyone who could migrate the toolserver wiki? [18:39:22] Silke_WMDE: Honestly, I don't know. I know for sure that I won't have the time to do this for quite some time; but as long as we have a dump of it we can certainly import it. [18:40:02] Silke_WMDE: The actual migration is the easiest part. The tough one is getting WMF to set up one at their cluster. [18:40:21] Is it? Why? [18:41:20] https://gerrit.wikimedia.org/r/#/c/109458/: "This really hasn't been discussed, and it's also not a trivial task" [18:41:40] Silke_WMDE: Setting up a private wiki is always somewhat of an ordeal we (WMF) try to avoid in general. [18:41:58] It's not private. [18:42:16] (Private = not SUL, not normal production -- not necessarily "not visible to the public") [18:42:32] Coren: And does it make a difference to you if it is read-only? [18:42:51] an archive sort of thing no one edits any more? [18:43:04] Silke_WMDE: Not especially, though if it is read-only then there is no reason for it to not just be a /tool/. [18:43:12] Silke_WMDE: In which case, anyone could do it. [18:43:24] Coren: Avoiding = delaying is bad; either WMF can do it, or it cannot. If the latter, there are other options. [18:43:51] Making it "a tool" could be an option if someone wants to do it. [18:43:51] scfc_de: It's not "delay to avoid" it's "long, arduous dicussion to decide whether to allow". :-) [18:44:47] Silke_WMDE: If we make it a tool, we need to hire an admin for it. The reason for using the WMF cluster is that we don't have to think about any issues that may arise. [18:45:19] scfc_de: Or I ask the toolserver admins to create a tool for it [18:45:57] Silke_WMDE: It's not a one-time deal. MediaWiki releases security updates every other month :-). Someone has to keep that wiki up to date. [18:46:15] yes [18:46:33] i see [18:47:04] scfc_de: That's much less of an issue if the wiki is readonly with no registration, and lives outside of production. [18:47:13] scfc_de: Then it can be safely "frozen in time" [18:47:18] (for the most part) [18:48:17] If there are concerns about "private" wikis, they should be discussed either on the bug or the Gerrit change. [18:49:38] I'm sure we'll find a solution for that wiki. The - I would say - last ressort is WMDE maintaining it. [18:50:55] What other topics / questions do you have? [18:53:07] * Silke_WMDE is reading andrewbogott s mail about the timeline [18:56:45] Whats missing to get the OSM tile rendering running again? [18:57:10] Thiemo_WMDE: I think this depends on the postgresql server [18:57:21] we're waiting for it [18:57:43] Thiemo_WMDE: Mostly just the postgresql server and the osm DB replication. Neither of which depend on the migration, so they can arrive soon; Alexandros is the one who can give a better timeline on this. [18:58:37] Thiemo_WMDE: As Alexandros isn't here right now, I'll ask him for a status update via e-mail. [18:58:51] thanks. [19:00:06] thanks. [19:01:57] Any more questions anyone? [19:04:12] If not, I propose to declare the end of this office hour. [19:04:37] Seconded [19:04:39] * sumanah reads - gimme a sec [19:04:50] oh hi sumanah! [19:05:32] Hi Silke_WMDE :) [19:05:49] I have just read http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20140219.txt and have no further questions [19:05:54] * Coren waves at Sumana [19:06:06] hi Coren [19:06:59] ok, I 3rd the motion to close the office hour [19:07:03] Ok, then! Hope to see you again in about a month - with a ready-migrated Labs! [19:07:18] :) [19:07:20] * Coren knocks on wood. [19:07:33] Haha, you say that in English, too! [19:08:05] Good night everyone and thanks a lot for attending! [19:10:48] :) [22:07:04] Ironholds: Ello [22:07:14] SigmaWP, hey dude :) [22:51:53] hmm, http://f1000research.com/articles/3-62/v1 is an interesting set of lessons learned on training sessions [22:52:18] * rdwrer assumes sumanah means -staff? [22:52:52] no [22:53:26] Ah, 'kay.