[15:42:22] guillom: Good morning :) is Presidents' Day a holiday for WMF? [15:42:50] hi RoanKattouw , and yes [15:44:12] Blegh [15:46:44] Considering you're up and online already, I assume you are coming in? [15:51:07] *RoanKattouw swears he'll throw stuff at Nimish if he dares to take off the very day I fix the code he's been begging me to fix for weeks [15:53:42] RoanKattouw, no, I'm just used to getting up early [15:54:08] RoanKattouw, I don't think anyone was planning to work today [17:15:21] hey RoanKattouw [17:15:47] thank you for the late night deployment on friday [17:15:48] Hey Naoko [17:15:54] Oh sure [17:16:42] it's holiday here today [17:16:46] Yeah I heard [17:17:25] Is anyone working at all today? [17:17:32] adam was originally going to be in the office, but he decided to join his friends today [17:17:52] trevor and nimish are not working [17:17:58] I will be working [17:18:29] tsss tsss [17:19:18] hi guillom [17:19:28] hi :) [17:19:37] Yeah me too obviously, I didn't find out about Presidents' Day till like 3 hours ago [17:19:40] what are you up to? [17:20:10] *guillom is sorting photos & completing his list of upcoming events to attend with his camera [17:20:28] RoanKattouw: I'm glad you are online and available, as I want to make sure calcey has the updated build for testing tonight [17:20:39] I'm fixing the highlight code so Nimish's template folding code will work again [17:20:47] nice [17:21:02] Two weeks ago or so he told me how it was broken and he was writing 'hypothetical code' [17:21:11] right [17:21:30] He's been begging me to do this for weeks and now that I'm finally doing it he's not around :( :D [17:21:39] he will be excited to hear that it is working again [17:21:52] Oh right and I need to add TOC&dialogs to the optin stats for you [17:22:03] that'll be nice [17:22:13] Not sure if you heard but Trevor had me totally disable TOC and dialogs for the weekend on Friday night [17:22:18] so RoanKattouw, what is the current status of prototype/deployment? [17:22:27] No idea [17:22:37] yes, that was the discuss among erik, me and trevor [17:22:49] as the fixes could not go out on friday [17:23:12] OK cool [17:23:24] From my POV it was just Trevor telling me while you weren't on IRC :) [17:23:47] no, we were talking offline [17:24:20] thus we kept you late [17:25:11] so RoanKattouw, will you update all prototypes and deployments with the fixes available as of Friday? [17:25:27] especially the fix for line breaks and copy/paste? [17:25:51] Sure [17:26:05] i'll send out email to calcey about one more round of testing for tonight [17:26:44] did you take your driver's test btw? [17:27:02] No, it was delayed again [17:27:17] I got lucky though, the fourth attempt at having this test without snow is this Wednesday [17:27:21] As opposed to some time in March [17:27:37] only two days away [17:28:36] yeah [17:32:52] RoanKattouw: i like the updated prefstats [17:33:28] You mean the automatic updates? [17:35:22] yeah http://en.wikipedia.org/wiki/Special:PrefStats/ [17:35:28] hi Ryan_Lane [17:35:44] nkomura: hi :) [17:36:07] i'm off today and tomorrow, so will likely be around for most of both days [17:36:08] nkomura: Oh yeah of course, we have those too. Never gave that much thought [17:37:24] RoanKattouw: it will be still nice to get the daily for NTOC and CGD so that i can compare the trend [17:37:41] will you generate them in CSV format? [17:38:12] Ryan_Lane: nice, are you waiting on more info from RobH for tesla? [17:38:13] Will do [17:38:23] I'll basically just add them as columns to the existing optin stats [17:38:40] nkomura: yeah. need network info for the VMs [17:39:02] RoanKattouw: thanks [17:40:04] Ryan_Lane: gotcha. you guys cleared the disk configuration problem you were experiencing though right? [17:40:23] nkomura: yeah. [17:40:35] i'm glad to hear that [17:41:09] with raskin, what browsers are available for testing right now? [17:42:14] only safari. firefox and selenium have a bug right now [17:42:23] Ryan_Lane: I am checking about your IPs now [17:42:30] chrome isn't yet supported in selenium on os x [17:42:33] trying to nail down a good procedure for it [17:42:39] though i am technically notworking today [17:42:41] ;] [17:42:56] oh? you guys have off for presidents day too? [17:43:15] I did not realize it until I had already been working on the phone system for 3 hours but yep ^_^ [17:43:24] RobH: thanks for your help. sorry to make you work on your day off [17:43:27] lol [17:43:28] lol [17:43:29] no worries [17:43:37] imma nerd on my laptop anyhow ;P [17:43:39] RobH is a hard working man [17:43:43] heh [17:44:08] says naoko, who's working even more :) [17:44:18] ha [17:44:24] (Normally I can allocate whatever IP addresses that I deem needed, but these are in the sandbox, so I just emailed and asked Mark about how best to allocate these in the future.) [17:44:47] naoko is actually a cyborg, its the only way to explain her work ethic of never sleeping. [17:44:47] thanks RobH [17:45:17] Ryan_Lane: will you add IE7 and IE8 to raskin? [17:45:22] RobH: While you're not working, can you reboot srv127 remotely? [17:45:35] RobH: It doesn't look happy and doesn't respond to HTTP or SSH http://ganglia.wikimedia.org/pmtpa/?r=hour&c=Apaches+4+CPU&h=srv127.pmtpa.wmnet [17:46:17] nkomura: it's an OS X system. No IE availabel [17:46:17] *nkomura thinks it should read while RobH is STILL working :) [17:46:22] RoanKattouw: Nope, sure cannot, it doesnt have LOM. I am going to pull it out of LVS though manually. [17:46:39] and will dc task it for me to look at when in DC [17:47:07] Cool thanks [17:47:18] Ryan_Lane: I didn't know IEs are not supported on OS X [17:47:31] and its a memcached, thanks for pointing it out RoanKattouw ! [17:47:49] lol I think it's been down for like a week [17:47:55] nkomura: yeah, they dropped support with IE 5 [17:47:57] ssssssh [17:48:02] lets not point that out ;] [17:49:00] Ryan_Lane: re: FF bug with selenium, what was that bug about and whose court is the ball in? [17:49:54] nkomura: both selenium (http://jira.openqa.org/browse/SRC-743) and firefox (https://bugzilla.mozilla.org/show_bug.cgi?id=513747) [17:50:12] nkomura: the selenium people fixed it in their alpha, but didn't backport it [17:50:47] nkomura: also: https://bugzilla.mozilla.org/show_bug.cgi?id=519550 for firefox [17:52:57] Ryan_Lane: so the isssue is OS X specific [17:53:11] firefox now shows the bugs as fixed... can't tell which version it is targetting [17:53:14] nkomura: yep [17:53:32] nkomura: Prototypes and deployment are updated to trunk state as of now [17:53:59] thanks RoanKattouw [17:56:48] Ryan_Lane: I resent my email to mark, included you, and also gave you the other info you asked for [17:56:58] RobH: cool. thanks [17:57:17] no problem, as soon as he gets to it and replies back you will know. [17:57:19] =] [17:58:06] *RobH runs away before someone asks about shell requests! [17:59:12] run quick RobH! [17:59:47] hi everyone [18:00:06] nkomura: upgraded firefox on raskin to 3.6, seeing if it fixed the problem [18:00:58] nice [18:01:26] RoanKattouw: are u there? [18:01:58] Yeah [18:02:20] RobH: Hey you poked at srv127, I'm happy ;) [18:04:49] RoanKattouw: i created this page: http://usability.wikimedia.org/wiki/Toolbar_icons_Wishlist to start a discussion about which icons or icon-sets make sense and are used a lot - I d appreciate you thoughts on that [18:04:56] nkomura: still broken... looks like we'll have to wait for the newer selenium release [18:05:49] :-( [18:06:18] hannes-_-: Abf??hrrungsstriche is something dewiki can do locally (frwiki has quote marks that look like <> but slightly different, same there) [18:06:28] *An [18:07:08] hannes-_-: Locked/readonly looks like it inserts a template, which is wiki-specific, so that would have to be done locally as well [18:07:35] (of course we can still write example code for these buttons that communities can just tweak and copypaste into their Common.js) [18:08:47] this is all about changing the toolbar locally [18:09:53] my goal is to collect opinions about which icons are used a lot, so I could design those icons [18:10:18] there is no data about that available, right? [18:12:03] hi hannes-_- [18:12:26] Ryan_Lane: do you have some bandwidth today or tomorrow? [18:12:42] nkomura: yep. what's up? [18:13:09] i'd like you to start thinking about migrating prototypes from linode to tesla [18:13:23] hannes-_-: About what, frequent usage? Nimish should have some ClickTracking data on that [18:13:28] and also setting up a prototype system for multimedai project [18:13:30] He's not working today though, today's a US holiday [18:13:49] RoanKattouw: but thats only for the default icons [18:13:59] Nope, all icons for the old toolbar as well [18:14:06] yeah [18:14:08] Ryan_Lane, nkomura: What is tesla currently used for? [18:14:10] but not user specific [18:14:19] RoanKattouw: currently unused :) [18:14:28] RoanKattouw: will be used for selenium RC cluster [18:14:43] Right [18:14:58] Presumably that's lightweight enough that it can run a web server too? [18:15:20] RoanKattouw: the hardware should comfortably fit 10-12 VMs [18:15:43] and yeah, selenium should be fairly lightweight [18:16:15] *RoanKattouw doesn't want to be running LAMP inside a VM [18:16:23] eh? why not? [18:16:42] Well that'd be the most resource intensive part of the box [18:17:01] Where I work, we run all of our web stuff inside of VMs. It's actually one of the best cases for virtualization [18:17:02] If it makes sense to take anything outside the VMs, it's that [18:17:21] How so? [18:18:30] well, wikimedia really doesn't fit as well into that use case :) [18:18:49] but. usually web servers sit between 10-20% cpu usage [18:19:02] Yeah that's not really true for us :) [18:19:21] Mark recently split off API requests to their own Apache cluster and those do like 80% CPU at peak time [18:19:29] yeah [18:19:41] saw that in logs via twitter [18:19:42] And we'd be running MySQL on that box as well [18:20:11] (when we move prototype to tesla that is; the regular WMF Apaches don't run MySQL of course) [18:20:22] well, the system is dual quad core, with 32GB RAM, with a raid 10 hard drive configuration [18:20:37] Nice [18:20:49] yeah. it should be able to handle the load fairly well [18:21:13] and vmware is fairly good about not letting one VM kill others [18:21:35] we can limit CPU/memory and disk IO resources if needed [18:22:37] The beefiest box I've worked on is hume.wikimedia.org, which is octuple quad core (!) but has 'only' 4GB RAM and 7GB of local disk space [18:22:45] we can also add more network connections, if network IO is an issue [18:22:57] heh [18:23:50] that isn't a lot of RAM :). must be really CPU bound jobs running on that host [18:24:53] Yeah it's for heavy batch jobs [18:25:13] i'm used to systems around the class of tesla. i mostly scale horizontally at work. [18:26:10] *RoanKattouw wonders why the hell hume is running 41 Apache processes [18:39:16] nkomura: when we get the IP info back from Mark, I'll start working on moving prototypes to tesla. [18:39:45] thank you [18:39:55] Ryan_Lane: scp from prototype is really slow. Would you like me to start copying some stuff to NFS on the cluster so you'll have fast access to it when the time comes? [18:40:06] we also want to host commons prototype on tesla too [18:41:05] RoanKattouw: that would be great [18:41:09] as we do not want to import all media files for commons prototype, we need to think of a clean way to import a set of categories [18:41:38] Ryan_Lane: I'll just copy everything in /srv/org/wikimedia/prototype and /home , tell me if you need more [18:42:21] RoanKattouw: I think that's all we'll need. [18:42:29] Ryan_Lane: guillom sent you question about category import, if you can give some thoughts to it, that'll be great [18:43:21] nkomura: yeah. let me find that email again [18:44:38] nkomura: I was wondering. can we not use the foreign file repo for this? [18:45:08] and upload local content as needed for further testing? [18:46:04] will you explain what the foreign file repo mean or does? [18:46:36] I think the idea is they want a subset of assets they can manipulate? rather than the foreign file repo [18:46:44] foreign file repo pulls images and descriptions from other mediawiki installations. it is mostly used to pull content from commons [18:47:29] can we keep the category and template info intact [18:47:48] do they need to actually modify the files, or just the wiki content? [18:48:05] mdale is right about we do want a subset of assets [18:48:24] ok. foreign repo won't work then [18:48:33] well.. but we need to identify if we ~really~ need to modify those assets or not [18:49:18] it's a lot easier to export a subset of wiki pages then the content :) [18:49:19] importing assets is non trivial .. you also have to copy over all the templates used in the asset description ... assuming we want to keep the asset descriptions [18:49:31] Ryan_Lane: Hm I'm actually getting decent transfer speed from prototype to fenari, 5 MB/s [18:49:51] Ryan_Lane: are you working on usability css as well? [18:49:58] mdale: Of course what we'd do is use Special:Export to export the description pages, it handles those template dependencies [18:50:04] or was that Adam? [18:50:07] mdale: usability css? [18:50:07] Then we'd batch-import the actual files [18:50:12] (or in reverse order rather) [18:50:14] Yeah that's Adam [18:50:26] oh he is not here... [18:51:09] when he does stop by tell him that .ui-widget-content { border: medium none !important; } is not fun [18:51:34] should try to avoid global !important on core ui classes [18:51:39] Should be wikiEditor-namespaced? [18:51:42] yea [18:51:51] What file&line? [18:52:16] let me look... [18:53:15] Ryan_Lane: so back to media asset import, we want to preserve the functionality and features Commons support on the prototype [18:53:46] wikiEditor.dialogs.css line 97 [18:54:38] nkomura: yeah. reading through the commons-l discussion now [18:54:58] nkomura: one thing we'll have to pay special attention to is storage space. [18:55:02] Yeah that's Adam [18:55:17] Ryan_Lane: yes [18:55:18] nkomura: we originally had 2.2 TB of storage, but since we switched to raid 10, we have 1.3... [18:55:22] mdale: Why not drop him an e-mail @ amiller@wikimedia.org [18:55:28] oky [18:55:44] and that is for OS space, and storage space [18:57:00] how much space is estimated for selenium, and wikipedia prototypes and sandboxes? [18:57:16] the selenium servers should be fairly small [18:57:28] we can also take bottom up approach for Commons prototype [18:57:39] 5-10GB a piece. but they can grow to 15-25GB. [18:57:50] per VM [18:57:50] determine how many files we can support and select a category tree which fit in that size [18:58:05] nkomura: yeah. that seems like a good approach [18:58:16] will we be able to determine that? [18:59:15] not sure if we have a tool to do that currently [18:59:22] but i'm sure we can query it ;) [19:00:37] I'm estimating selenium + prototype storage space to run between 100-200GB [19:00:44] depending on growth [19:01:26] so let's say 550GB for media assets? [19:01:56] i want us to leave ample space so that the disk space does not prevent us from future growth [19:02:31] i meant to say 500GB [19:02:39] yeah. that's probably a good estimate. [19:03:12] that should be a good sample of commons [19:03:21] k, i'll ask around if we can determine a category tree to fit in that size [19:03:42] is partial import from Commons posssible though? [19:04:52] Ryan_Lane: i kicked you another email, reply back to that please (i am trying to cook myself somelunch so I am not exactly at keyboard =) [19:11:23] RobH: replied [19:12:01] nkomura: partial import should be doable. [19:12:55] nkomura: if we export the pages, and import them. when we import the individual files, everything should just work. [19:16:26] getting the individual files may be difficult. if we get a list together, we may be able to get someone to pull them off the image servers. [19:18:01] Ryan_Lane: good to hear that [19:21:52] Ryan_Lane: I can do that for you guys [19:22:27] cool. now we just need to figure out what to export :) [19:25:00] guillom: is there a page where I can see the category structure of commons? [19:25:27] nkomura, http://commons.wikimedia.org/wiki/Category:Categories [19:25:36] thanks [19:26:10] back in roughly 2 hours. [19:26:39] need anything before I go? [19:27:00] thanks Ryan_Lane|away [19:27:15] nothing at this moment [19:27:24] ok. back in a bit [19:34:27] RoanKattouw: i may need your help writing a query for total file size by Commons category :-) [19:34:47] Sure [19:38:20] i want to understand the category tree structure first, so lemme get a better grasp on that first and get back to you later this week [19:38:52] OK [19:39:02] Just drop me an e-mail saying what you want to know [19:39:10] will do [19:42:54] Ryan_Lane|away: I've backed up /srv/org/wikimedia/prototype and /home to two .tar.bz2 files in fenari.wikimedia.org:/home/catrope [19:45:21] RoanKattouw: how does deployment respond to you today? [19:45:28] Haven't tried [19:45:45] *RoanKattouw wonders whether his improved caching settings covered deployment [19:45:55] it is waiting response for a while now [19:46:02] prototype responded a little better [19:46:09] Then that's probably it [19:47:06] Hm no it should have an effect [19:47:07] *RoanKattouw pokes [19:47:43] the page is loaded [19:47:54] but it took over a minute [19:48:36] i see line breaks are not inserted upon paste in FF3 [19:48:47] Hm [19:48:53] but have you noticed this lag? [19:49:05] line breaks are inserted, and removed [19:49:10] ? [19:49:26] we may want to reduce the timing of that even more [19:49:29] I haven't tried [19:49:54] nkomura: Second load time should be a lot faster [19:50:01] Because client-side cache kicks in [19:50:09] First load was over a minute for me, second load took 17 s [19:50:24] k [19:50:36] *RoanKattouw experiments with gzipping everything rather than just JS/CSS, should save bandwidth [19:51:07] will you run sanity checks on IE? [19:51:14] i'll test chrome now [19:52:23] Hm that seems to work (gzipping that is) [19:54:11] nkomura: http://toolserver.org/~catrope/optin-20100215/all.total :) [19:54:26] *nkomura goes to look [19:54:57] nkomura: Columns are vector in - out - toolbar in - out - editwarning out - in (sorry) - toc in - out - dialogs in - out [19:55:08] EditWarning is the odd one out because it's enabled by default [19:55:15] So you can only turn it off and back on again [19:55:52] legend would be nice ;) [19:55:58] I know [19:56:11] I'll work on that [19:57:34] so for the history data you ran for me for the past 14 days [19:58:03] Yeah I'll regenerate Feb 8 and beyond with these new columns [19:58:13] Heck, why not everything ^^ [19:58:16] I'll add the legend first though [19:59:00] please keep Feb1 as the start date [19:59:12] so that i can see the trend before and after the release [19:59:29] (we only started tracking TOC and dialogs on Feb 8) [19:59:33] *Feb 8 [19:59:38] that's right [19:59:51] *nkomura forgot about that [20:00:05] I had to look up the date too :) [20:00:22] will you also create a CSV file for the history/trend data [20:00:34] yah [20:00:36] as i want to import into spreadsheet? [20:00:37] thanks [20:01:01] NTOC and Dialogues have pretty good retention rate [20:01:10] i'm happy to see that [20:03:19] RoanKattouw: i'm looking at deployment in chrome [20:04:11] i don't think the fix for overzealous collapsing is staged [20:04:26] it is still removing line breaks upon pasting [20:05:02] All code is staged AFAIK. I wasn't involved with that fix though, it was all Trevor and Nimish [20:05:55] okay [20:06:10] i'll add a note to the bug for that [20:06:51] what is the build number of the staged build on deployment? [20:07:26] is the version number in special page accurate presentation? [20:08:01] No [20:08:07] It's the same as the one on prototype ATM [20:09:18] so i have r62528 for prototype and r62529 for deployment [20:09:33] Sounds about right [20:10:14] will you help me to understand the difference? [20:11:27] Deployment runs off the wmf-deployment branch, prototype runs off trunk [20:11:50] Although they're separate trees of code, they share rev numbers, so r62000 can be a trunk rev and r62001 can be a wmf-deployment rev [20:12:15] Deployment reports the latest rev to wmf-deployment, which is (in terms of numbers) not at all related to revids in trunk [20:12:37] It so happens that someone committed something to wmf-deployment after I updated prototype to trunk, that's why its revid is higher [20:12:54] But generally the deployment revid is pretty meaningless, especially because I'm running it with all kinds of stuff merged in [20:13:56] how can we communicate which build to test on deployment with calcey? [20:16:55] We can tell them to look up the build number on prototype instead [20:17:02] I always update both at the same time [20:17:44] k [20:17:49] that's a good plan [21:06:51] nkomura: New and improved format: http://toolserver.org/~catrope/optin-20100215/all.csv (note the header row on top and the total row at the bottom) [21:07:08] nkomura: This also means I'm dropping the .total files, because the totals are at the bottom of the .csv files [21:07:58] nice! [21:08:54] *RoanKattouw starts regenerating all statistics to date in this format [21:08:55] *nkomura is saving a file on her desktop [21:09:16] *guillom is waiting for his furniture [21:10:17] wooo, nice [21:11:09] yup [21:11:13] i love it [21:14:49] *nkomura goes to make a cup of tea [21:21:11] i'm back [21:23:09] It's regenerating 10/31 now, the ones before that are done [21:24:11] thanks a lot, RoanKattouw. this will make the data manipulation a lot easier [21:24:42] Yeah having it all in CSV is nice [21:25:04] The aggregate file with all the dates' totals is also gonna be in CSV once I'm done regenerating [21:25:30] col [21:25:32] cool [21:42:06] nkomura: All done, optinlist updated as well [21:42:19] Oh I guess optinlist needs a header row, adding [21:44:20] i see it :-) [21:45:55] a qq [21:46:07] if i go to the page by date e.g. http://toolserver.org/~catrope/optin-20100214/ [21:46:36] Yeah [21:46:39] everthing excpet all.csv is for the specified date? [21:48:08] or should i just ignore the last modified date? [21:48:21] Ignore the last modified date [21:48:27] k [21:48:27] That's the date those stats were last regenerated [21:48:38] All stats in that dir, including all.csv , are for the specified date [21:48:44] this is great RoanKattouw [21:50:45] thank you [23:33:47] Ugh, mardi gras. I can't believe it took that long to get lunch [23:35:07] Mardi gras? It's Monday [23:35:47] it is the carnival in New Orleans [23:36:11] i wish to go and join some day [23:36:19] I know but "Mardi gras" means "fat/greasy Tuesday" [23:36:44] So I would assume it's on a Tuesday :) [23:36:50] it is :) [23:36:53] today is lundi gras [23:37:13] we tend to call the entire carnival season mardi gras [23:37:51] it's fun. locals tend to get jaded about it because it makes normal things kind of difficult to do [23:38:14] are you partying tomorrow, Ryan_Lane? [23:38:55] maybe at night. tuesday is the day I usually rest from carnival season [23:40:29] nkomura: you should definitely come one year. I could bring you some fun places. [23:40:39] i'd love that [23:41:56] do you know wikipedinas/wikimedians in New Orleans who take photos of mardi gras or write articles about it? [23:42:48] nope. I was thinking about starting to take photos of places in the city, and adding them to commons to learn photography though [23:43:04] I bet there are some in the city that do it already. [23:43:08] that'd be awesome! [23:43:38] there are some [23:43:39] http://commons.wikimedia.org/wiki/Category:Mardi_Gras/Carnival_costumes [23:43:46] but i bet every year is different [23:44:04] yeah. people come up with the craziest things to wear [23:44:36] it looks fun [23:44:52] I've never dressed up for it :) [23:44:53] OK, I'll try come in 2011 [23:45:57] my gig for next year -> http://commons.wikimedia.org/wiki/Category:Mardi_Gras/Carnival_costumes [23:46:46] that's a good idea :) [23:48:07] you'll want to come the week before mardi gras, leaving the day after, or the day after that. [23:48:17] they kind of kick everyone out mardi gras night [23:48:58] why? [23:49:18] RoanKattouw_away: copy/paste on IE on deployment is working good - FYI [23:49:26] that's when the party ends :) [23:50:00] They start cleaning up the french quarter then. The trash vehicles come to collect and weigh all of the trash [23:50:55] but the festival builds up to the day from the previous week? [23:51:08] yeah. the first parade is usually two weeks before [23:51:33] so there are multiple parades [23:52:07] yeah. this was the schedule for this year: http://www.mardigrasneworleans.com/schedule.html [23:52:49] around 60 parades. one is for dogs, which always makes me laugh [23:53:18] folks in new orleans know how to have fun :-) [23:53:36] you guys are serious party goers [23:53:42] heh. yeah. we had a special parade this year for our football team, who won the superbowl [23:54:20] congrats to that [23:54:30] it's exhausting. you'll have a lot of fun when you come. [23:54:40] thanks [23:55:02] i look forward to it ;)