[16:59:43] o7 [17:00:24] Hi all and welcome to the office hour about the famous migration. :) [17:00:31] hi nosy! [17:00:49] hi Silke_WMDE [17:01:05] * fsainsbu Hello from Tasmania 4 am local time [17:01:54] Oh hi fsainsbu!! Early good morning! Who else is there? Coren? akosiaris? scfc_de? Kolossos? [17:02:02] Hello together. [17:02:27] hello people [17:03:06] hi akosiaris! Thanks for coming! Would you like to start and give us a status update on the OSM-related stuff? [17:03:40] Sure [17:05:20] So. OSM database is up and running. 3 volunteers already have access to help me find out problems and to help in solving them. Planet.osm dump has been imported and the minutely sync from planet.osm using osmosis is to be enabled this week [17:05:58] I suppose after that, an email to toolserver-l and maps-l will be needed announcing the general availability of the DB and we are done [17:06:10] Cool. Is Kolossos among the 3? [17:06:43] yes [17:07:13] Yes, next step is to compare the performance. Kai is prepairing something for this. [17:08:03] That was the server a mysql database is run on at the same time, right? [17:08:54] In that front (performance), I also plan to write enable/write some ganglia plugins but this is not a blocker, but more of a nice to have nature [17:08:58] Silke_WMDE: yes [17:09:23] in fact, there are two machines that each are a primary for an SQL server. One mysql, one postgresql [17:09:47] for postgresql, asynchronous streaming replication is enabled between the two [17:09:59] Kai plan to render a set of tiles on Toolserver and on Labs to comparing caching behaviour, etc. [17:10:11] Good. [17:10:49] And then, after debugging, to use the db, we'd need postsql on Tool labs as stated here: https://bugzilla.wikimedia.org/show_bug.cgi?id=48896 [17:11:01] äh postgresql [17:11:08] please keep in mind that ts is already busy [17:11:25] * Coren is here, sorry. [17:11:31] hi Coren [17:11:33] Timezone confusion. :-) [17:11:42] Silke_WMDE: I think that ticket is pretty much ready to be closed [17:12:05] akosiaris: Nice, go ahead! :) [17:12:08] The moment we announce the GA of the OSM db, that ticket's goals are done. [17:12:18] i see [17:12:27] Are voluteers able to create own database on this server? [17:13:15] not right now. I suppose we could allow it. Do we need it ? [17:13:40] I suppose you are referring to entirely new postgresql databases Kolossos [17:13:55] Yes I need my database for WP geo coordinates and WIWOWM project. [17:14:20] sorry WIWOSM instead of WIWOWM. [17:14:36] so a completely new DB with a full reimport of planet.osm ? [17:14:47] like user databases on TS? [17:15:10] No only with some million coordinates (5 Mio. or so) [17:15:16] I am unaware how user databases on TS were implemented btw [17:15:16] lol [17:15:31] nosy: Can you answer that question? [17:16:45] each database clustershard (s1/s2 etc) has one server were also users are granted all rights to u_ [17:17:20] they can create their own databases starting with u_username and save their stuff there [17:17:37] nosy: For postgres as well? [17:17:56] i guess so...wait... [17:17:59] the postgresql/postgis parts are just about to get ready, including support for php module on web frontends? [17:18:13] no its only per user if one asks for access [17:19:05] so you manually create a db for users that ask for that called u_ and grant them full access to that db [17:19:08] Right; I expect that for postgres, user databases will be created only on request. [17:19:47] yes [17:20:11] akosiaris Coren Is that something you can implement? [17:20:23] On demand? Sure. [17:20:31] fsainsbu: just a moment [17:20:31] IMHO it is doable. bugzilla tickets should be enough I suppose [17:20:34] Manually requsts would be ok, we have on ptolemy only 5 user-databases. [17:21:39] Kolossos: Do you want to open a ticket in bugzilla? You are better at describing than I am. [17:22:09] Yes I will request a db. [17:22:20] akosiaris: what about fsainsbu's question about php in the web front end? [17:22:40] Than we have it, next step will be the render stack. This will be completely in hand of volunteers. Correct? [17:22:54] Kolossos: I meant a ticket to request the feature "user db" on postgresql server [17:23:27] Yes "user db", for me. [17:23:35] Silke_WMDE: fsainsbu: not following on that question. [17:24:02] web frontends? PHP module ? [17:24:02] fsainsbu: Can you be more precise? [17:24:33] I tried to run php with postgress as a test and it said module not found.. weeks ago so may be fixed? [17:26:02] ah. In labs I suppose. No not fixed but easy enough. We will fix it today by adding php5-pgsql module in installed packages [17:26:17] thanks [17:27:03] The rendering question: My last update was that you'll try to open a Labs project "tileserver", right? and render there? [17:27:14] We should have a documentation how to tunnel to the database to use PGAdmin as Frontend. [17:27:57] Kolossos: I suppose that is easy. Coren: wikitech ? or someplace else ? [17:28:42] akosiaris: We already have a document to do so w/ mysql on wikitech, so I expect that's the right place for it too. [17:28:45] In the labs.wiki we have an old OSM page we can clear it and start docu there. [17:29:09] who can write that documentation? [17:29:21] I 'll do it [17:29:39] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Configuring_MySQL_Workbench [17:29:39] cool, I'll create you a bz ticket [17:30:54] ^^ the render stack question [17:31:07] The renderstack: Have we storage for rendered tiles in Labs? (2-3 TB) [17:31:33] with mysql you can encrypt the conection, force ssl and open the port through the firewall. http://www.madirish.net/214 [17:32:06] Kolossos: Yes. [17:33:01] Fine. Rendering itself will be on VMs. So it's in the hand of volunteers. Correct? [17:33:47] Kolossos: yes [17:35:10] We have also Tiles that are not changing (Hillshading). We can directly start to copy this tiles from TS to Labs? That's perhaps a job for nosy. [17:35:13] Kolossos: Have you set up a Labs project already? [17:36:14] I don't set up a project. But dschwen move his wikiminiatlas. [17:36:43] and Kai can install a renderstack. [17:37:28] If you need help to get started with the Labs project for that poke us [17:37:41] Ok. [17:38:10] nosy: still there? ^^ [17:38:38] re wait [17:38:52] fsainsbu: so, you see, it's all just being "born" - quite a new infrastructure. :) [17:39:32] ok i could copy [17:39:59] Kolossos: Unless I am missing something, I see no reason why you should not be able to copy the tiles. [17:40:00] Sorry, I am impatient , the ssl idea means no setup for volunteers and once its working is easy to role out to new hosts.. [17:40:12] Kolossos: i remember marqs hillshading tiles what else should we syn? [17:41:20] As far as I remember it was only hillshading usefull, everthing else is old. [17:41:21] i guess we ll have a tiket for this in jira or bz [17:41:41] I believe we have it for maps at the moment. [17:41:46] and /tiles? [17:42:31] To copy old osm tiles makes not so much senses it's easier to create them new. [17:42:51] kolossos Do you have instructions for pulling data into postgis from wikipedia, or is that done as part of another process? [17:43:46] So once Kai has started the tileserver, him and/or Kolossos will poke you, nosy. [17:44:04] Filling Postgis with wp coords is done with a list of SQL commands, generated by PHP. [17:44:34] ok [17:44:37] Cause I was thinking that it could be done when page renders with coords to insert or update tables in postgis. [17:44:42] Coren: Can we talk about some ETAs for features? [17:44:53] Silke_WMDE: I'm all yours. [17:44:57] Copy hillshading tiles for TS is independent from tileserver. [17:45:08] Coren: https://bugzilla.wikimedia.org/show_bug.cgi?id=57876 [17:45:24] the "joining" folks need this [17:45:37] Silke_WMDE: Sometime this week. [17:45:57] Ok, I#ll put Sunday into the ticket. :) [17:46:17] * Coren works of feature requests this week now that migration is, essentially, done [17:46:20] on* [17:46:27] Coren what about the e-mail function that sounded almost done [17:47:02] Silke_WMDE: It's still almost done -- there's a bug right now for incoming email (from the outside) to tools which is on my todo for today; it's otherwise fully ready. [17:47:30] Do you think you'll make it this week, too? [17:47:37] It's going to be today even. [17:47:45] yay [17:47:49] It's #2 on my day's list. [17:48:07] And it's not a huge bug; it's a bit of a permission problem with exim. [17:48:08] Lol, Ive seen that fail [17:48:26] then we have some db tables feature requests: https://bugzilla.wikimedia.org/show_bug.cgi?id=59682 [17:48:32] https://bugzilla.wikimedia.org/show_bug.cgi?id=59683 [17:48:33] ie #2 never gets reached.. [17:48:58] Coren: what is the bug, i'm curious? [17:49:18] hi gifti [17:49:25] hi Silke_WMDE [17:51:16] Coren: I don't understand enough of the db tables right now to know if there are legal issues involved or if they just can ba done [17:52:03] 59682 sounds a bit as if exactly the relevant tables of wikidata weren't there. [17:53:34] Coren: ? [17:53:43] * Coren reading context. [17:53:50] ah ok [17:55:04] For the most part, those are just "we didn't know they'd be useful". Federated tables are a pain to maintain, so we wanted to keep the list to those which are actually used. [17:55:40] Ok, so, there is no problem [17:55:43] The shorter list should be okay to add. [17:56:30] the shorter list? [17:56:56] At the bottom of the bug reports, there's a short list of "important" views -- those should be okay to add without intervention from legal. [17:57:20] ah, I understand, thx [17:57:35] Adding /all/ of them is probably not going to happen; but once the support is there to add/remove tables from federation adding them on request is feasible. [17:58:17] "support to add/remove tables from federation" is that an existing ticket in bz? [17:58:39] Not explicitly, that I know of. [17:58:57] * Silke_WMDE will add one [17:59:35] Are there any other questions right now? [17:59:54] yes [18:00:03] where should i sync the tiles to? [18:00:11] i mean the static ones [18:00:27] they need to be accessible for web serving but not for rendering [18:00:36] gifti: exim is unable to drop privileges to execute some of the scripts needed to read the .forward information [18:00:46] nosy: I think, Kai will be able to tell you when the Labs project exists, correct, Kolossos? [18:00:59] nosy: I think that'll have to wait until the labs project exists. [18:01:23] ok [18:02:32] I'll put some of what we said into BZ, I hope I put in correct summaries. [18:03:16] Anything else? [18:03:44] If not, we'll chat in a few weeks and partly meet at the Hackthon. [18:03:59] Which of you will attend the Hackathon in Zürich? [18:04:30] I will, Kevin Leduc [18:04:39] Not me unless your sponsouring ;) [18:04:46] :) [18:04:53] I will, too [18:05:04] I will too [18:05:19] Coren andrewbogott Any of you coming to Zürich in May? [18:05:37] I will. [18:05:50] cool Kolossos [18:06:06] Silke_WMDE: I'm there for the whole hackaton, specifically to set up a 'Tool Labs' corner to assist people migrating. [18:06:44] very nice! [18:07:05] Well do enjoy spring, and have a great time, I am of to bed again... [18:07:20] btw I think that there are a bit more than 100 accounts active on TS that do not yet have a Tool Labs account [18:07:30] fsainsbu: good night and thanks for getting up to come :) [18:07:46] Your welcome, most informative [18:07:59] ^^though some accounts obviously have different names, like my own. [18:08:35] OK, everybody, see you soon! [18:08:56] Silke_WMDE: I'll be there. [18:09:10] * Coren waves. [18:09:18] yay andrewbogott [18:09:22] thanks for attending akosiaris Coren nosy [18:09:31] yw silke [18:09:39] thank you as well [18:09:40] and Kolossos and all! [22:11:45] cndiv: we have a ghost in r37 [22:11:50] someone keeps ringing us