[00:01:03] sweet. ok, catch you guys later! [00:02:45] peace out! [00:28:17] New patchset: Jdlrobson; "dynamically load pages to make mobile snappy" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28321 [00:28:17] New patchset: Jdlrobson; "jslint fixes" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28322 [00:41:18] New patchset: Jdlrobson; "ensure footer can be toggled open" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28323 [00:46:59] New patchset: Jdlrobson; "deal with cleanup templates inside cleanup templates" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28324 [01:23:47] im out, g'night everyone [04:12:18] [bugs] NEW [Bug 40987] Clicking jump back a section doesn't work on WP7 (priority: Unprioritized severity: normal) https://bugzilla.wikimedia.org/show_bug.cgi?id=40987 [04:12:19] [bugs] NEW [Bug 40986] WP7 requires two taps to begin typing in search box (priority: Unprioritized severity: enhancement) https://bugzilla.wikimedia.org/show_bug.cgi?id=40986 [14:44:27] what are those "Error 503 Service Unavailable [14:44:27] " errors ("Service Unavailable [14:44:27] , Guru Meditation: [14:44:27] XID: 1422779544) you get with de.m.wikipedia.org when navigating throughout articles and that are no longer here if you reload the page (sometimes twice)? [14:44:59] *between [14:45:04] *here > there [14:45:32] MaxSem: ^ [14:46:00] pill: the rest of the staff are on PST, so you might not get a response for a few hours. [14:47:47] all right; actually, i don't really need a reponse at all :). i'm at my desktop pc, i was just trying to check some pages in mobile view and got those 503 errors all the time. [14:48:26] just wanted to bringt this to your attention, just in case it's not already known :) [14:53:17] so what are the exact reproduction steps? [14:56:21] searching for an article and clicking on random wiki links :) [14:57:32] e.g. i got the error when accessing http://de.m.wikipedia.org/wiki/Existenzphilosophie and clicked on the link to "Existentialismus". [14:57:50] when i do it again, everything is fine. [15:03:05] apparently, just a Varnish glitch [17:15:05] MaxSem and/or brion: when you have a chance can you look at https://gerrit.wikimedia.org/r/#/q/status:open+project:mediawiki/extensions/MobileFrontend+branch:master+topic:login,n,z [17:15:20] lookin' [17:16:39] Change merged: Brion VIBBER; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28298 [17:17:45] Change merged: Brion VIBBER; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28299 [17:20:20] awjr: we could just drop forms from the strip list [17:20:33] there's an open bug about input forms being stripped: https://bugzilla.wikimedia.org/show_bug.cgi?id=38515 [17:20:45] but if we don't want to do that until we test them, then this is probably fine [17:21:11] brion: yeah i considered that and figured this would be a good intermediary step [17:21:20] excellent [17:22:57] my proposal is to stop filering forms on special pages [17:23:24] on content pages, forms via InputBox are still problematic [17:25:05] Change merged: Brion VIBBER; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28300 [17:27:20] Change merged: Brion VIBBER; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28301 [17:28:48] Change merged: Brion VIBBER; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28302 [17:32:23] New patchset: Brion VIBBER; "Fixes docblock for ExtMobileFrontend::renderLogin()" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28306 [17:32:32] * brion stans gerrit a few times [17:32:32] *stabs [17:33:30] Change merged: Brion VIBBER; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28306 [17:34:55] New patchset: Brion VIBBER; "Adds mobile login form as a template" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28303 [17:35:32] Change merged: Brion VIBBER; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28303 [17:36:34] brion: weird rebasing issues? [17:36:48] yeah i reviewed a couple of em out of order and it got confused [17:36:51] should all be done now :D [17:37:02] \o/ thanks dude [17:37:05] sho thing [17:40:24] Error 503 Service Unavailable [17:40:36] i'm seeing them all across the board [17:40:36] tfinc? [17:40:38] known issue ? [17:40:41] loading en.m [17:40:42] tfinc on what? [17:40:43] in the beta [17:40:45] nope [17:41:06] awjr: where do we keep our active graph of 503's ? [17:41:19] man do i love our new typography :D [17:41:33] ah there we go just got one [17:41:47] tfinc i don't know if we have graphs for 503s [17:41:47] (09:03:05 AM) MaxSem: apparently, just a Varnish glitch [17:42:07] 503s reported about 2.5 hours ago [17:42:22] awjr: we really need to get to a point were alarms go off after enough 503's have happened. this has happened one too many times [17:42:30] i'm adding a mingle story [17:43:26] tfinc would it be better to create an rt ticket for that? isn't that an operations issue/ [17:43:29] tfinc: yeah, i got one this morning on BART [17:43:45] i was trying to read about bat bombs! [17:44:18] its looks really bad when we just let these things slide [17:44:18] on the typography… i love the look but a lot of info boxes are slightly too wide for the margins… not sure if that's new or i'm just noticing it more now [17:44:53] brion: i think the table/navbox issues have been around for awhile, but the new larger spacing/font exacerbates them [17:44:58] yeah [17:45:07] chrismcmahon: is someone working on it? [17:45:21] awjr: too late i already added a story https://mingle.corp.wikimedia.org/projects/mobile/cards/229 … if rt is a better place for it then feel free to move it [17:45:50] but i don't like that we know its happening yet don't know what the current state of the problem, solution, etc [17:45:53] "as a reader, i would like to read about bat bombs" <- i hope that's the narrative in that card [17:45:58] :D [17:46:25] tfinc agreed, but i'm unclear whether this is something that is an operations issue or something that we need to engineer a solution for [17:47:00] awjr: and its that uncertainty that i want us to get better about. we need to get to a point where we can answer that quickly [17:47:27] awjr: if that means we need patrick and asher to jump then lets do that [17:47:35] I'd though MaxSem had dismissed the 503s, now I'm not sure [17:47:55] I didn't dismiss it [17:49:28] hey brion: at some point before/after the mobile design meeting today, can i grab you and get your expert opinion on the technical implementation (or impossibility thereof) on some watchlist stuff? [17:49:39] sure [17:50:02] cool beans [17:50:05] i'm remote today (incubating a cough i think i picked up at the conf) but available [17:50:44] hmm, that design meeting is at 1 right? [17:50:49] yep [17:50:52] didn't we reschedule today's standup for 1? d'oh [17:51:03] heh [17:51:57] all right i'm gonna charge the laptop then get back to resource loader guts [17:55:58] tfinc leslie is looking into the 503s [18:05:57] okay, our GeoData ticket is https://rt.wikimedia.org/Ticket/Display.html?id=3738 [18:13:24] jerith: ping [18:19:16] preilly: Pong. [18:19:34] jerith: what do you need done on silver? [18:19:59] preilly: The logs indicated the the redis server had fallen over. [18:20:13] Could you check if that's running and restart it if necessary? [18:20:14] ajerith: how do I best restart it? [18:21:24] The usual Ubuntu mechanism for restarting services. I usually call '/etc/init.d/ restart', but I think there's a new way that's better or something. [18:21:57] Then probably do the same with the supervisord that's running all the vumi stuff. [18:22:22] ajerith: okay [18:25:00] New patchset: Jdlrobson; "move ribbon out of #content wrapper" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28385 [18:25:00] New patchset: Jdlrobson; "make watch icon work on pages loaded via javascript" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28386 [18:25:24] the 503s are continuing to confound ops, but mark is now online and helping out [18:26:29] ajerith: Starting redis-server: [18:26:29] *** FATAL CONFIG FILE ERROR *** [18:26:29] Reading the configuration file, at line 306 [18:26:29] >>> 'vm-enabled no' [18:26:29] Bad directive or wrong number of arguments [18:26:43] Ugh. [18:26:57] That's new to me. [18:27:02] What version of redis is that? [18:27:02] * yuvipanda looks around [18:27:39] ajerith: checking now [18:28:13] ajerith: ii redis-server 2:2.6.0-rc7-wmf1 [18:28:59] Ah. That means we're probably still using the version from the PPA. [18:29:38] The redis version in the Precise repo is new enough, so maybe we should go back to that. [18:30:08] ajerith: can we not make this one work? [18:30:43] preilly: We probably could, but I think it's easier to switch to the version that isn't bleeding-edge. [18:31:03] ajerith: can I just comment out that line [18:31:13] ajerith: that appears to be the default [18:31:33] My operational experience with redis is "aptitude install redis" or "brew install redis". I've never even had to restart it. [18:32:01] ajerith: ha ha ha [18:32:30] Okay, not /entirely/ true. I've had to switch versions a few times for testing. [18:32:50] I've never even looked at its config. [18:50:41] [bugs] NEW [Bug 41124] Cannot open sections on various browsers (priority: Unprioritized severity: normal) https://bugzilla.wikimedia.org/show_bug.cgi?id=41124 [18:50:56] awjr: good. i'm eager to find out how to stop them :) [18:52:21] making cautious progress on ResourceLoader… seem to have implemented the server-side startup module filtering, now on to actually using it and seeing if it works [18:54:18] awjr: guessing we might need to consider a deployment later today to fix this bug [18:54:28] (https://bugzilla.wikimedia.org/show_bug.cgi?id=41124) [18:54:56] jdlrobson: do we have a fi? [18:54:56] er [18:54:56] fix? [18:55:04] soon we will [18:57:45] preilly: Any progress? [19:00:54] ajerith: redis is restarted [19:01:09] preilly: \o/ [19:01:23] Is the vumi stuff happy after a restart now? [19:04:27] ok sounds like the 503s are a result of other issues on the cluster - the only reason the desktop site is not being as severely affected as mobile is because there is more in the cache for desktop [19:05:48] awjr: one of these days we need to get going on the open source hiring module [19:06:40] awjr: got it [19:09:58] tfinc yeah… [19:10:12] awjr: i was just sitting with gayle and she reminded me of it [19:10:28] tfinc when are we on the hook for? i just thumbed through emails/calendar but couldnt find a date [19:10:40] awjr: no clue. i haven't done a thing for it yet [19:10:50] me neither, it fell off my radar kinda fast... [19:11:09] if its not in mingle or my email then i just won't do it [19:11:09] :) [19:11:16] heh exactly [19:12:12] ajerith: everything is restarted [19:12:19] New patchset: Jdlrobson; "window.onload fallback for browsers without DOMContentLoaded (bug 41124)" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28396 [19:12:24] awjr: > https://gerrit.wikimedia.org/r/28396 [19:12:36] jdlrobson: fix for the nokia issue? [19:12:36] preilly: No more errors in the USSD logs? [19:12:46] yup and blackberry etc [19:12:51] i'd say it's a pretty urgent fix [19:13:00] jenkins says patch failed [19:13:02] preilly: If there are "enquire_link OK" lines, everything's happy. [19:13:51] ajerith: which log? [19:14:00] oh but looks like the build failed due to a git issue [19:14:01] jdlrobson: ^ [19:14:14] jdlrobson: how can i verify that the patch works? [19:14:39] just trust me.. :) [19:14:39] preilly: /var/log/vumi/*ussd*.log -- there should be two of them, but either is fine. [19:14:47] faith based code review? [19:14:53] you can add a return at the top of DOMCOntentLoaded [19:14:59] that will simulate that never running [19:15:08] do that before the patch and after [19:15:24] you'll notice without patch it will fail to expand a section [19:15:35] but I'd say this is a pretty trivial fix [19:15:43] i'd rather spend time getting it deployed :) [19:16:05] yeah, it looks fine [19:16:09] jdlrobson: id ont know when we'll be able to a deployment [19:16:13] ajerith: 2012-10-17 19:14:31+0000 [Uninitialized] Connection failed. Reason: [Failure instance: Traceback (failure with no frames): : Connection was refused by other side: 111: Connection refused. [19:16:13] ] [19:16:14] there are problems on the cluster right now [19:16:15] 2012-10-17 19:14:31+0000 [Uninitialized] will retry in 47 seconds [19:16:19] 2012-10-17 19:14:31+0000 [Uninitialized] Stopping factory [19:16:51] preilly: That looks like the SMPP connection is timing out. [19:17:01] jdlrobson: is it not possible to do that in a resource loader module? [19:17:09] awjr: huh? [19:17:17] https://gerrit.wikimedia.org/r/#/c/28396/1/includes/skins/SkinMobile.php [19:17:17] ajerith: okay so what is the issue? [19:17:31] jdlrobson: does that have to be inline js? [19:17:38] Are there any errors in wikipedia_worker_0.log? [19:17:46] as opposed to js loaded by resourceloader? [19:18:13] theoretically no but I think it helps the code become more readible [19:18:27] If not, check the two SMS transport logs and see if they're also timing out on the SMPP connection. [19:18:33] when we move to jQuery only mode (after dynamic sections) in it should be one line [19:18:44] jdlrobson: yeah but we'll probably need to refresh the varnish cache for that fix [19:19:24] preilly: I suspect it's a problem on their side, but I want to rule out anything else on ours first. [19:19:24] awjr: make a story for it if you think it will make deployments easier [19:19:41] ajerith: it all looks good on this end [19:19:46] ajerith: from what I can tell [19:19:52] it used to be in mw-application.js but it moved because of the app [19:19:52] which isn't necessarily a big deal, but makes me even more nervous about deploying anytime soon with the cluster issues [19:20:21] preilly: In the SMS logs as well? [19:22:08] ajerith: tail -f tata_sms_outgoing_transport_1.log [19:22:12] 2012-10-17 19:17:30+0000 [EsmeTransceiver,client] enquire_link_resp OK [19:22:12] jdlrobson: where are your slides for the last meetup? [19:22:29] preilly: Okay, cool. That means everything's happy on our end. [19:22:41] jdlrobson: when possible it's best for us to avoid inline js and changes to html as they require cache purges - which isn't neessarily something that makes deployment more difficult, but it has impacts on performance [19:22:44] http://jonrobson.me.uk/jon-speaks.html < tfinc [19:22:55] and pisses off ops :p [19:23:38] awjr: sure - but we're going to need a cache purge regardless this time [19:23:58] awjr: we already have an issue with mwMobileFrontendConfig [19:24:06] preilly: I'll reply to the mail from Amit and tell them that they need to check the firewall on the USSD side. [19:24:15] we should write a story that we want all this packaged up in a javascript file and get it tackled [19:25:18] jdlrobson: for sure, i was just pointing out it's not just an issue around complicating deployments. [19:25:20] jdlrobson: what is the story? [19:26:55] awjr: as a deployer/ops guy I would like to remove the need for cache purges due to changes to javascript in the SkinMobile.php ... [19:31:29] hmm, from what's going on in #-ops, I think that purging Varnish now that the cluster is already overloaded is dangerous [19:31:32] not today [19:32:19] Yeah. You don't want to flush caches when your backend's already buckling under the pressure. [19:33:14] jdlrobson, ^^ [19:33:49] MaxSem: agreed, that's what i mentioned earlier [19:34:00] i've not been following but that makes sense to me.. [19:34:51] * ajerith heads out. [19:35:19] preilly: Drop me an email or something if there are further problems and I'll look into them tomorrow morning my time. [19:36:02] btw.. https://gerrit.wikimedia.org/r/#/q/status:open+project:mediawiki/extensions/MobileFrontend,n,z < *cough* [19:41:52] awjr: can you at least merge https://gerrit.wikimedia.org/r/#/c/28396/ ? [19:42:14] not this second [19:42:22] it's blocking my testing on #203 [19:42:28] brion: ^ ? [19:42:36] ? [19:42:39] looking [19:43:32] jdlrobson: there's a build fail [19:43:40] brion: i think that's something else [19:43:46] not related to my change [19:44:07] [exec] error: Could not fetch origin [19:44:07] mysterious [19:44:28] how can i trigger a rerun? [19:44:57] jdlrobson: log in on jenkins and a 'retrigger' should appear on the sidebar [19:45:00] at least if you have right perms [19:45:24] Change merged: Brion VIBBER; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28396 [19:45:30] \o/ [19:45:47] brion: don't think i have an account :( [19:45:56] but brion \o/ [19:45:57] should be same as your gerrit account [19:48:54] * jdlrobson tries [19:48:54] also brion could you check out https://gerrit.wikimedia.org/r/#/c/28170 [19:49:10] moment [19:51:57] New patchset: Brion VIBBER; "Initial stab at ResourceLoader integration on MobileFrontend" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28434 [19:54:41] New review: Brion VIBBER; "Looks good" [mediawiki/extensions/MobileFrontend] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/28170 [19:54:41] Change merged: Brion VIBBER; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28170 [19:57:52] is it the standup time? [20:01:06] MaxSem, brion aye [20:01:06] https://plus.google.com/hangouts/_/8bc98160cdf0396821e2ba381099b56fde42cb0c [20:11:06] oh - i added a note on https://mingle.corp.wikimedia.org/projects/mobile/cards/132 that it looks like the core part of this hasn't been deployed, so we won't see it running live in production just yet [20:11:20] mobile design meeting in r66 whenever y'all are ready - jdlrobson, brion (should we skype you in?) [20:11:33] Maryana: Skype me or hangout me, whichever you prefer [20:11:42] 'brionv' on skype [20:11:55] sorry, skype is my generic version of video chat. kinda like kleenex [20:12:24] hehe [20:14:35] brion it looks like the change was merged to the 1.21wmf2 branch, which means it will probably go out to all non-wikipedia sites today or tomorrow, and then to enwiki next monday [20:14:43] (depending on the cluster issues) [20:14:57] excellent [20:17:35] MaxSem: i jsut took a quick look at https://gerrit.wikimedia.org/r/#/c/27035/2 - the regex needs updating [20:17:49] take a look ata the 'mobile redirect domain list' in https://mingle.corp.wikimedia.org/projects/mobile/cards/194 [20:17:57] it's a little different than the regex that's currently in place [20:18:03] particularly for wikimedia.org sites [20:18:31] when everything's in puppet [20:29:18] awjr: mobile 503's should be gone now [20:29:24] Maryana: got disconnected again :( [20:33:44] awesome, thanks preilly [20:34:46] preilly are the other cluster issues resolved as well? [20:37:01] Maryana: it might be easier if you just finish the meeting without me and then check back in for the tech bits :) [20:38:05] yeah, sorry dude! it's the wifi in here, but there's no ethernet cable >:-[ [20:39:25] d'oh. no worries [20:56:23] Change merged: awjrichards; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28317 [20:58:55] New review: awjrichards; "-2'ing due to unmerged core dependency." [mediawiki/extensions/MobileFrontend] (master); V: 0 C: -2; - https://gerrit.wikimedia.org/r/28434 [21:04:45] brion: we're ready to talk watchlists nowish [21:05:10] yo [21:05:51] want to try google hangout, since skype seems to hate me? [21:06:28] Maryana: did hangout not work for you? [21:06:29] yeah let's try that [21:06:34] Maryana: make sure you on a wired connection [21:06:35] i'll start one and send the link [21:07:11] brion: thx! [21:07:15] Maryana: https://plus.google.com/hangouts/_/994467da332c2ea1cddf8fbcb61a6cf0b4e71edd?authuser=0&hl=en# [21:07:18] tfinc: i can't seem to find one in here... [21:12:28] http://www.mediawiki.org/wiki/Mobile_design/Watchlist_view [21:12:41] awjr: what was the doc with everyones travel ? [21:13:10] tfinc the one that laura's been using? [21:13:14] awjr: yes [21:13:18] https://docs.google.com/a/wikimedia.org/spreadsheet/ccc?key=0Ar8ziwjAnNjWdFVlRkJEOEE4WG1rZ0J6eGdVNGx6VlE#gid=3 [21:19:41] tfinc im not sure how reliable that doc is [21:19:55] my travel info in it is not correct, so i don't trust anyone else's [21:19:55] i have a meeting with erik in the afternoon about scheduling. its good enough. [21:19:59] k [21:20:04] awjr: so fix it [21:20:54] i've let laura know; im hesitant to mess with her document [21:25:13] whee [21:27:55] brion: wow.. looks like that popup gets http://en.wikipedia.org/w/index.php?title=The_Alchemist_(novel)&action=raw&ctype=text/css&oldid=518235589&maxage=0&smaxage=0 and http://en.wikipedia.org/w/index.php?title=The_Alchemist_(novel)&action=raw&ctype=text/css&oldid=515742375&maxage=0&smaxage=0 and then in javascript does the diff :) [21:28:20] hah [21:28:26] well if that's what we gotta do, we can do that [21:28:34] it would be better to do on the serber [21:28:34] *server [21:28:40] for the bad phones [21:28:42] yeah [21:28:52] * brion imagines large talk pages or Obama diffs [21:29:48] jdlrobson: i just checked in with asher about deploying that bug fix today; he said we should wait til tomorrow [21:29:58] k awjr [21:30:12] what i'd like to do is only deploy the bug fix tomorrow and wait to push out other changes til our next scheduled deployment window [21:30:45] brion: how did the iOS save page upgrade test go? [21:31:26] tfinc: assuming i tried it right, i didn't see my old version's saved pages… but i might have done something wrong and cleared data by mistake [21:31:47] brion: thats no good [21:31:55] we can't be losing pages like that [21:32:44] yeah lemme try fresh on my ipod [21:34:25] i think app store installations are hosed on this iPod… lemme reboot it [21:35:33] No drama... diff Obama. [21:36:16] reboot solves all woes :) [21:36:32] we should kill that alert popup after save, it's annoying [21:37:51] tfinc: ok problem appears to be…. that the previous version isn't fully saving somehow. [21:37:59] after killing and restarting the app, i have no saved pages in the list [21:38:35] so they're presumably not there when the new version gets installed either [21:39:03] lemme try the new version and make sure it doesn't do the same thing [21:40:38] ok, new version does *not* do that. [21:40:45] killing the app and restarting it shows my saved pages now [21:41:01] so i'm not sure the prev version was saving properly, it may have just claimed to save [21:42:56] New review: Jdlrobson; "A few comments.." [mediawiki/extensions/MobileFrontend] (master) C: 0; - https://gerrit.wikimedia.org/r/28434 [22:01:27] idle question, anyone know who maintain(s/ed) http://mobile-testing.wmflabs.org/ ? ISTR http://mobile-testing.wmflabs.org/w/ once did something useful. [22:04:12] New patchset: Brion VIBBER; "Initial stab at ResourceLoader integration on MobileFrontend" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28434 [22:04:14] New patchset: Jdlrobson; "update to latest e3 logger" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28453 [22:05:59] sorry, laptop snafu, if anyone replied to the mobile-testing labs instance q [22:06:27] New review: Brion VIBBER; "Patchset 2 updates:" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/28434 [22:14:19] chrismcmahon, that instance's DB was lost [22:15:23] I tried reimporting an enwiki dump, but Labs is so brutally slow these days that memcached connection timed out periodically during the import [22:22:57] thanks MaxSem [22:30:52] jdlrobson: does the js/css in https://gerrit.wikimedia.org/r/#/c/26492/5 mostly cover the design here http://www.mediawiki.org/wiki/Mobile_design/Sign_in? (aside from the overlay aspect) [22:32:04] awjr: should do.. [22:32:08] k thnx [22:39:22] MaxSem: eh? exactly how much are you trying to import [22:39:25] MaxSem: and in which way is it slow? [22:41:16] Ryan_Lane, the original setup by Patrick was to use both DB and MW checkout from project storage. on Monday, this resulted in cached page view times of ~30s [22:41:54] are you sure you have memcache configured correctly in mediawiki? [22:42:09] that is usually an indication that mediawiki is trying and failing to connect to mediawiki [22:42:09] memcached configuration wasn't changed [22:42:11] check the port [22:42:15] of the running service [22:42:18] vs the port configured [22:42:25] is memcache running? [22:42:40] nah, I simply copied the same config [22:42:45] .... [22:42:51] that's not what I'm askin [22:42:53] did you actually check? [22:43:19] no [22:43:26] so, check ;) [22:43:46] but tell me how slow `time ls` should take on Gluster [22:43:53] gluster is slow [22:44:01] are you putting the database on gluster? [22:44:29] gluster is slow, and it's running SATA disks [22:44:29] it used to be there, but MW file access was also mega slow [22:44:37] you can't put mysql on gluster [22:44:48] you should use /var/lib/mysql or /mnt [22:44:52] both of those use local disks [22:44:58] which are much, much faster [22:44:59] Ryan_Lane, how Memcached could be restored by moving MW from gluster to local FS? [22:45:12] memcache probably just needs to be restarted [22:45:17] and you need to make sure you're configured to use the right port [22:45:33] I know the symptoms of MW not seeing memcached [22:45:40] ;) [22:45:46] do you have apc installed? [22:46:11] there's no reason that cached pages should take 30 seconds [22:46:14] it's not a labs issue [22:46:36] once again: initial disposition: /var/www/w is a symlink to gluster [22:46:50] rm -rf /var/www/w [22:46:50] yeah yeah [22:47:12] cp gluster/w/ /var/www/w [22:47:14] again [22:47:21] 10 times speed improvement [22:47:21] that has nothing to do with your problem [22:47:41] do you have apc installed? [22:47:55] for the most part, php should never actually hit the files [22:48:06] apc caches the bytecode in memory [22:48:20] and yes, local storage is much, much faster than gluster [22:48:51] the gluster storage is SATA, across the network, and on a DFS [22:49:05] and it's raid 6 [22:49:20] yeah, but once everything's loaded, long-time running maintenance scripts should work at roughly the same speed with APC and without it [22:49:43] local storage is directly attached to the machine, is SAS and is in a raid-10 [22:49:56] MaxSem: depends [22:50:16] but if you don't have apc installed, install it [22:50:24] it takes two seconds and it makes everything faster [22:51:03] my concern is mysql performance on local FS, which isn't fast either [22:51:24] dump import is pretty much DB-bound [22:52:06] well, it's mysql in a virtual machine [22:52:42] you're writing to a raid-10 SAS [22:52:58] it's not going to get too much faster, unless it was SSD backed [22:53:26] maxsem@mobile-testing:~$ sudo apt-get install php5-apc [22:53:31] php-apc is already the newest version. [22:53:37] * Ryan_Lane nods [22:53:51] check memcache [22:53:56] netstat -tanvp | grep mem [22:54:00] which port is it on? [22:57:52] maxsem@mobile-testing:/var/www/w/maintenance$ php eval.php [22:57:52] > var_dump($wgMemc); [22:57:52] object(MemcachedPhpBagOStuff)#4 (2) { [22:58:06] > $t=microtime(true);$wgMemc->set('foo',$t);echo microtime(true)-$t;; [22:58:06] 0.0005030632019043 [22:58:29] and your pages are still taking 30s to generate? [22:58:38] cached ones? [22:58:53]