[00:00:12] tehe [00:02:24] awjr: I need to take off to pick up the youngin from practice [00:02:34] ok jcmish [00:02:45] shoot me an email if anything comes up [00:02:50] I'll have my ipad and cell with me [00:03:01] scap is still going; it would be good to have help with testing in prod when it's done [00:03:05] will you be back on soon? [00:06:27] 40 mins [00:07:05] HATE this connection [00:07:16] awjr: I'll be online in 40 mins [00:07:23] when will scap be done? [00:07:37] hopefully by then :p [00:07:48] im hoping in 15-20mins, jcmis [00:07:49] h [00:07:50] der [00:07:51] jcmish [00:10:36] MaxSem: question around https://bugzilla.wikimedia.org/show_bug.cgi?id=43843 - is there anyway we can speed up the indexing of page images? e.g. staggered batch jobs over the course of a few weeks? [00:11:39] not during the migration [00:12:15] I'm writing an email to wikitech-l about the appropriateness of this script for a couple of days now [00:12:24] wrote a couple words now [00:22:06] MaxSem: context is https://mingle.corp.wikimedia.org/projects/mobile/cards/376 - might need to remove images from the watchlist on stable if the number of pages without indexed images is sufficiently low [00:22:25] I know [00:22:49] but it can't be populated quickly, that's all [00:22:56] awjr: http://jonrobson.me.uk/isScapDoneYet.html ? [00:23:04] nooop [00:23:21] MaxSem, what do we use to generate thumbnails? imagemagick? [00:23:28] MaxSem: completely understand - was just keen to get an idea of how long we are likely to need to wait [00:24:03] jgonera, yes but irrelevant. it's slow not due to resizing [00:24:43] jdlrobson, at least a week [00:24:48] I see, I asked because some time ago I experimented with optimal parameters for fastest thumbnail generation [00:26:02] hehe [00:26:25] thumbnails aren\t generated until someone actually requests the image [00:27:11] we're waiting for image information to populate [00:27:14] can't we change that? just create some background job? [00:28:07] it will be interrupted by next week's migration [00:32:27] speaking of which... [00:32:39] awjr, can you please review https://gerrit.wikimedia.org/r/#/c/44556/ ? [00:33:49] quick look looks good; i'll take a closer look shortly [00:37:08] scaaaaaaaappppppp [00:37:11] never ends :( [00:39:25] this one might be a record breaker [00:39:42] 66 minutes so far [00:42:44] it's ok, everyone in the office wandered off to get cake [00:42:57] i miss out on all the fun :( [00:43:23] aw, sorry. there will be pictures of chip's bright red embarrassed face on commons, i'm sure [00:43:31] ! [00:43:32] ridiculous scap [00:43:43] that's no way to talk about chip, jdlrobson [00:44:02] haha [00:46:56] lulz [00:47:03] we scared chip half to death [00:47:10] and possible scarred [00:47:12] :D [00:48:38] photos or it didnt happen [00:48:47] ascii art is acceptable too [00:50:19] awjr: chip did an iPhone5 panorama [00:53:12] scap still running, it's been 80 minutes [00:53:39] this is so insane. [01:00:15] MaxSem: Change merged: awjrichards; [mediawiki/extensions/PageImages] (master) - https://gerrit.wikimedia.org/r/44556 [01:00:20] oh he's gone [01:00:40] did MaxSem finally go to sleep ? [01:00:45] man i hope so [01:00:50] he may be hibernating for a while if so [01:00:59] seeing as how i don't know when he actually sleeps [01:01:42] he was online around 2am local this morning... [01:03:08] awjr: http://jonrobson.me.uk/isScapDoneYet.html ? [01:03:13] lord of mercy no [01:03:19] i am about to explode [01:03:23] awjr: srsly? [01:03:25] it has now been exactly 90 minutes [01:03:26] yeah [01:03:30] this is truly epic [01:03:33] we must be doing it wrong [01:04:11] yeey broken css and js [01:04:42] i hope this doesn't embarrass us after winning the knight mobile award.. [01:05:27] >_< [01:05:45] 6:05 [01:05:46] TimStarling [01:05:46] 6:05 [01:05:46] awj's scap still has a long way to go, so there's no hurry [01:06:17] siggh [01:08:45] holy jeez best guess is we're about 64% of the way through syncing to the servers [01:08:53] :/ [01:09:59] there are various theories as to the slowness but no one in #wikimedia-operations seems particularly concerned [01:11:12] scaps a messss [01:11:35] im very roughly guessing another 20 minutes or so [01:12:35] awjr: play scatman on repeat until it is done [01:12:47] jdlrobson: do you want me to go more insane?! [01:12:50] (obviously in your head hear scatman as scapman) [01:13:00] awjr: for things to change there must be sacrafice [01:13:06] lol [01:13:10] scaprafice [01:13:15] zing! [01:13:22] good one [01:13:32] http://www.youtube.com/watch?v=FE0XcdM22Yo [01:13:36] * Maryana dies [01:13:41] you're the scapman! [01:15:03] jdlrobson, do you know when a character of code 27 can appear in page title? L35 in mf-photo.js: name = name.replace( String.fromCharCode( 27 ), '-' ); [01:15:42] character map says it's "escape" character, I'm not even sure what it is [01:15:43] the escape character? [01:16:06] when you click escape [01:16:11] some systems enter it [01:16:18] I think it was probably a case of being overly cautious [01:16:27] we originally wrote that function for the WikiLoves Monuments app [01:17:01] scap looks done for me - in that i'm seeing the new stuff [01:17:17] * jdlrobson does a dance [01:17:43] jdlrobson: keep dreaming [01:17:49] well it's done on my laptop [01:17:51] but no on my mobile [01:17:51] jdlrobson: and keep clikcing around im sure you'll find smoething broken eventually [01:18:38] got to go to engineering meet up in 15 mins :/ [01:19:51] jdlrobson: yeah, this is really bad timing [01:20:05] jdlrobson: will you be available to help test from the meetup? [01:22:44] awjr hopefully [01:22:50] jdlrobson, regarding dimensions of elements in the layout, I told Vibha that it's enough if she sends me an SVG file and I can get them myself from there [01:22:56] i already know we'll need a cache flush, so im going to touch and sync the files then request the flush as soon as scap is done; hopefully we wont need anything further [01:26:26] jgonera: true dat [01:26:36] awjr: sweet [01:26:50] * if scap ever finishes [01:27:09] heading up now to the meet up! [01:34:07] is scap done yet? [01:34:09] heading upstairs too, I'm getting my laptop and phone with me [01:34:36] 16 servers left.... [01:34:45] so close [01:36:57] ok, i think all the servers that matter are done [01:37:08] im gonna touch, sync and request a flush [01:37:19] that sounds like an unusual bathroom scenario [01:38:54] LOL [01:40:30] hmm i take it back, the ones that matter are not done quite yet [01:41:04] ok now they are [01:42:23] scap is done? [01:42:34] btw there's beer and mingling going on that i am missing right now [01:43:25] scap just finished [01:44:03] i'm very confused by http://fr.m.wikipedia.org/w/index.php?title=Sp%C3%A9cial:Liste_de_suivi&watchlistview=feed [01:44:17] no styling whatsoever [01:44:28] even with debug=truwe [01:44:45] cache is getting flushed [01:45:14] ok mobile cache is flushed [01:45:36] jdlrobson, jgonera ^ [01:45:41] please dbl check things in the wild [01:45:43] checking [01:46:00] ok there is a minor bug awjr but luckily doesn't effect stable [01:46:09] jdlrobson: what is it? [01:46:27] Special pages do not get css or javascript if they use aliases [01:46:32] e.g. https://de.m.wikipedia.org/wiki/Spezial:Mobile_Optionen [01:46:38] ^ that one does effect stable :-S [01:46:54] that is weird, why would that be the case? [01:47:19] it must be to do with the special page generation [01:47:45] wooo, we haz watchlist star tap logging! [01:47:46] why would that be the case? [01:47:59] and it looks like lots of anons are poking the star [01:48:06] wow! [01:48:07] (right after they touch, sync, and flush) [01:48:09] :) [01:48:14] hehehe [01:48:37] strtolower( $title->getText() ) ) < i assume awjr that $title->getText() is returning the translated name rather than the original page name? [01:48:39] 40 since scap finished! [01:49:09] jdlrobson: oh, that is possible; is that how the resource name is being generated? [01:49:11] so for Spezial:Mobile_Optionen it returns 'mobile_optionen' rather than mobileoptions [01:49:22] yup I should have checked that :/ [01:49:29] whoops [01:49:51] so it looks for a module called mobile.mobile_optionen.styles and fails to find it [01:49:53] so does that mean css/js is going to be broken on all special pages in non-english wikis in beta? [01:49:58] pages will still be functional [01:50:02] just ugo [01:50:06] just look odd and work as if no javascript [01:50:11] yeesh [01:50:13] not all awjr [01:50:17] only ones which have aliased the name [01:50:30] e.g. https://fr.m.wikipedia.org/wiki/Sp%C3%A9cial:MobileOptions is fine [01:50:42] oh i see [01:50:47] ugh [01:50:52] mm [01:50:56] this is annoying [01:51:07] and there is a deployment freeze next week [01:51:13] Russia is fine https://ru.m.wikipedia.org/wiki/%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F:MobileOptions [01:51:21] so… we fix now or we fix tomorrow and break no deploy on friday rule [01:51:23] So realistically this is only going to effect mobile options [01:51:30] in stable [01:51:33] everything else is in beta [01:51:51] Contact form has styling but it only effects contact pages which have forms [01:51:53] i dont think that's a big deal, mobile options looks/works fine [01:51:56] we don't so contact forms should render fine [01:52:20] so basically some mobile options pages of foreign wiki projects will look like that [01:52:20] https://de.m.wikipedia.org/wiki/Spezial:Mobile_Optionen [01:52:31] Maryana: what do you reckon? [01:52:34] i'm filing a bug now awjr [01:52:38] thanks jdlrobson [01:53:00] my inclination is to fix this tomorrow and break no-deploy rule [01:53:13] it would be good to avoid merging other code in the meantime, but we can always cherry pick a fix if we need to [01:53:15] argh [01:53:22] that's so weird [01:53:45] we probably shoulda noticed that would be a problem in code review :( [01:54:08] yeah, let's get this fixed [01:54:10] if we can [01:54:13] where does that happen jdlrobson? [01:54:28] oh, is that in skinMobile? [01:54:55] $id = explode( '/', strtolower( $title->getText() ) ); [01:55:15] yup [01:55:28] actually... [01:55:37] yeah, let's fix it tomorrow, shouldn't take too much time I guess [01:55:50] awjr: https://bugzilla.wikimedia.org/show_bug.cgi?id=44089 [01:56:27] awjr: shit happens [01:56:41] the english are always pissing off other nationalities so just blame me [01:57:01] can we turn of aliases easily awjr ? [01:57:04] that would be another option [01:57:10] as a short term fix [01:58:02] awjr: need to go soon - battery is low and i should mingle [01:58:14] but not thought works mingle [01:58:16] real life mingle [01:58:18] jdlrobson: go for it [01:58:20] have fun! [01:58:30] it's not a huge deal, we'll get it fixed [01:58:31] sorry about this latest screw up from me [01:58:38] glad it's not to public [01:58:40] no worries, it happens, it's no huge [01:58:43] *not [01:58:50] and it's not as bad as sticky cookies :p [03:23:18] Reedy saw your message. What's happening? [03:23:41] Probably easiest to join #wikimedia-operations [03:23:53] Tonnes of cache misses [03:23:58] Seems to be overloading the bits backend [03:24:00] causing site issues [03:24:44] I'll be home in five min [03:25:03] Will join then [03:25:12] Alright [03:25:18] Arthur is also online now [14:28:48] New patchset: Zfilipin; "Commited files that got lost in patch set 2 of Gerrit commit 42218" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/42755 [14:31:37] New review: Zfilipin; "Removed unnecesary code from BetaPage and fixed link in LoginSuccessPage." [mediawiki/extensions/MobileFrontend] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/42755 [17:12:46] awjr: I'll try to go on and join the hangout today [17:12:58] uhoh PissedPanda ? [17:13:05] but due to some issues, *both* my methods of internet access seem to be around 128kbps or something [17:13:10] jcmish: ^ explains the name [17:13:16] holy cow [17:13:21] that's just ridiculous [17:13:33] yeaaaah [17:14:23] awjr: hopefully not asking you to repeat much but did we ever find out the culprit? [17:14:43] I went to bed after your last exchange on the ops list [17:15:10] jcmish: not exactly, another im wrapping up another email on the subject [17:15:22] PissedPanda: ok, good luck :) [17:15:22] k I'll wait :D [17:22:26] awjr, it had some positive effect though [17:22:46] now that Tim rewrote scap, it takes just SIX MINUTES [17:22:53] LOL [17:25:12] between scap taking 130 minutes yesterday and the outage last night, i am not in a particularly good mood. but i am pleased that Tim rewrote scap just in time for it to be replaced by git deploy :p [17:25:43] im gonna go shower and finish waking up before standup, bbl [17:26:23] heh [17:26:27] looks like everyone's in a bad mood today [17:26:44] exactly (bites teh panda) [17:29:55] it's like the wikiflu, but virtual [17:31:09] and unlike wikiflu, rabid Max's bite is not curable [17:34:59] MaxSem: I've been subjected to random diseases from a billion other people. Pretty sure I can survive that :P [17:35:30] rabid Max is not a human [17:35:38] not entirely, at least [17:36:31] MaxSem: and going by recent events, neither are a large number of people in this country either [17:43:07] but we've got Depardieu! [17:46:17] http://www.speedtest.net/result/2445549820.png is slow death [17:50:35] http://www.speedtest.net/result/2445559323.png for me [17:52:45] hacks. [17:53:55] MaxSem: how is your up speed so fast?! [17:54:04] i am jealous. [17:54:18] this is what i get: http://www.speedtest.net/result/2445567255.png [17:54:58] cuz I get it via a 100M Ethernet [17:55:00] PissedPanda: at least your speeds are > than the majority of India... [17:55:13] awjr: usually maybe. [17:55:26] I get 24+ down on a wireless connection [17:55:27] apparently that's not saying much :p [17:55:30] and I could easily get full 100Mbps if I wanted [17:55:44] MaxSem: damn! where is that coming from? [17:55:58] we've got a town-wide network [17:55:59] awjr: i'm just on a bad spot today, since *both* my internet sources are slow [17:55:59] and why don't you want to? [17:56:09] I should clearly get another one [17:56:26] that's awesome MaxSem, people keep talking about laying fiber in towns in the US but… who knows when that will ever happen [17:57:18] awjr, are you on ADSL? [17:57:31] awjr: are you on the hangout? [17:57:56] okay, mine just died [17:57:58] yay [18:04:09] Are you guys in SF end of feb/start of march when platform and ops are? [18:04:15] I know we're apparently really short on desks.. [18:09:01] Reedy, you'll hav to bring you own then [18:09:12] the hammock's mine! [18:09:52] Haha [18:10:10] Are you going to teleconference into meetings in the office from the hammock? [18:11:40] You also didn't explicitly answer my question :p [18:16:38] hi, i heard you were wondering about me flushing varnish cache? that might indeed have happened yesterday but totally by accident when trying to purge it to fix wikidata, not meaning mobile [18:16:44] sorry if that caused any problems [18:18:22] mutante: well, mostly curious about what caused bits to die yesterday, certainly seems plausible that the flush was the culprit or at least related. the only effect on mobile was that it was initially assumed that our deployment yesterday was the culprit (which it also may have been at least related) [18:18:22] awjr: tfinc sent in standup status again, with more reliable packets. [18:18:28] thanks PissedPanda :) [18:18:30] PissedPanda: thanks [18:19:11] tfinc, Are you guys in SF end of feb/start of march when platform and ops are? I know we're apparently really short on desks.. [18:19:32] its only platform and ops [18:19:43] Reedy: i am expecting to be in SF at some point in march, not sure when exactly yet though [18:19:53] lol [18:19:56] Aha [18:20:05] Reedy: I am expecting to be in SF at some point in March too, but most probably 2014 :P [18:20:08] but, totally unrelated to platform/ops stuff [18:20:38] awjr: it did not die by itself, we restarted them [18:20:46] actually peter did it for me [18:21:06] i broke wikidata and we had to clean bits cache [18:21:07] mutante: what happens when they restart? do they restart colde? [18:21:09] *cold [18:21:15] I'm still not sure how bits and mobile would take down the cluster [18:21:18] mobile is fine on its own [18:21:39] bits should be fine on its own, as from past experience it gets back to a high cache rate in a few minutes [18:22:16] Reedy: the working theory i believe was that we were somehow making either uncached/uncachable requests to bits, or so many requests with unique URLs that we overloaded bits causing a huge spike in misses, resulting in al the 503s [18:22:43] awjr: hmmm.. everybody seemed to agree its ok to restart them. as long as you wait like 180s (or until its done) between restarting them.. so it wasnt all at once [18:23:00] basically you do one, wait until done, and then next [18:23:19] We've got the atypical WMF 5-6 week notice of an office visit... [18:23:25] like "restarting all bits varnish instances with a 180s stagger between each (applying shm_reclen change)"" [18:24:25] well, i am under the impression that there are either no or insufficient logs of the requests to bits when it started exploding - which would probably help us answer what exactly happened :p [18:25:05] lol [18:25:10] Why doesn't that suprise me [18:25:11] * Reedy grins [18:25:35] * awjr shakes his fist at Reedy [18:26:07] * tfinc started two new wikipedia articles last night on obscure japanese video games. has already had one wikipedian clean them up without me asking. if only i had received a notification about it rather then having to find out by hand [18:28:11] tfinc: Guess I need to sit down with a calendar and work out about doing Denmark too.. Yay, constraints [18:28:53] Reedy, well - Denmark can fall through. there should be some clarity on this next week [18:29:40] Aha. Right [18:29:45] I won't rush to book travel [18:31:26] Reedy: these days what's the best way to see may contributions across all projects ? [18:31:30] tfinc: echo, coming soon to a device near you :P [18:31:49] tfinc: Just the count? [18:31:56] YuviPanda, yeah - I'm already thinking of defences from it [18:32:00] YuviPanda: it will motivate me to edit more [18:32:02] MaxSem: *from* it? [18:32:14] once Echo is sortof stable we'll need to figure a way to make it do *push* [18:32:25] Reedy: split across edits/new articles/images/etc [18:32:25] yes - I got lots of useless spam when it got activated on mw.org [18:32:30] MaxSem: the cross linking ? [18:32:32] so that you can be nofied in 10ms without killing your battery [18:32:38] https://en.wikipedia.org/wiki/Special:CentralAuth/yourusernamegoeshere [18:32:43] That does basic count [18:32:44] yes [18:33:10] Reedy: does anything break it down by types of edits ? [18:33:26] There must be a toolserver project or 5 for something like that [18:33:34] MaxSem: the first was interesting but all others were useless to me [18:33:59] tfinc: toolserver.org/~tparis/pcount/index.php?name=Yuvipanda&lang=commons&wiki=wikimedia [18:33:59] jdlrobson: your code review image didn't come through forme. [18:34:01] for me. [18:34:04] it's a little bit better, I think [18:34:16] but it doesn't give you an overview across projects [18:34:19] (again, I think) [18:34:25] YuviPanda: oh . nice [18:34:45] hmmm, I should try to cross 1000 before end of week on commons [18:59:31] who is working on story 332? [18:59:49] awjr, jdlrobson, jgonera - I've come up with http://etherpad.wikimedia.org/RLRLRL - I'm still investigating the other possiblities of improvement, but would like your critique on what's there already [19:01:37] jorm I don't think anyone is yet. I'm just sitting with Vibha getting the story finalized [19:01:42] MaxSem: will take a look [19:01:58] some assets just got dumped up and i'm okay with them except for some of the language. [19:02:08] we *must* say "Donate" [19:02:33] "Upload" may be what we're doing, but the trigger language has to indicate that the user is releasing rights. [19:18:24] jorm sounds like a chat you should have with product to finalize language [19:24:39] New review: JGonera; "Uncaught TypeError: Cannot read property 'geosearch' of undefined nearby.js:41" [mediawiki/extensions/MobileFrontend] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/43594 [19:27:17] jdlrobson, have you worked on fixing https://bugzilla.wikimedia.org/show_bug.cgi?id=44089 or should I try? [19:29:55] jgonera: feel free! [19:34:43] jgonera: i'm just fixing up story 332 with vibha should be finished very soon [19:35:08] jdlrobson, ok, will you have half an hour to talk after that? [19:35:26] have the logging conversation at 12 [19:35:32] but straight after that i'm all yours [19:35:40] logging/tracking conversation [19:36:23] ok [19:37:57] jdlrobson: just to make sure, this is the convo with analytics? [19:37:59] tfinc: ^ [19:38:14] just a reminder about tracking for the apps (I raized this on the list, just thought I should remind you guys) [19:38:16] tfinc: ^ [19:38:24] tfinc: yup [19:39:25] say what now? [19:39:45] jgonera: 332 is ready to go now. Hopefully it's clear enough. [19:39:48] let me know if not [19:40:45] is there a special Dropbox login I should use to access https://www.dropbox.com/home/WMF%20Design/Mobile/Photo%20Uploads/Story%20332 ? [19:42:10] jgonera: you just need a dropbox account [19:42:13] do you have one? [19:42:29] it should be public [19:42:29] I do, I can't access this [19:42:47] ok vibha just fixing that [19:45:00] bear with us jgonera :) [19:45:12] ok, let me know when it works ;) [19:45:54] jgonera: try the new link now [19:46:17] hang on that doesn't work either [19:46:31] same thing [19:47:31] does link work now jgonera ? [19:48:43] jgonera: ? [19:48:44] yes, the decription says that the upload progress message should be shown as a toast but the design asset shows it differently, so how should it look like? [19:50:53] and what about the heart icon, do we keep it for now? [19:56:14] [aosp-morelang-ime] yuvipanda pushed 2 new commits to master: http://git.io/KgufIA [19:56:14] aosp-morelang-ime/master 4e321ac YuviPanda: Setup transliterated languages as proper subtypes [19:56:14] aosp-morelang-ime/master 59daba8 YuviPanda: Remove 'Secondary Language' preference [19:57:39] did we decide anything about killing the non-jQuery mode? [19:59:51] jdlrobson, ^ ? [20:02:24] jgonera its a toast in implementation but styled differently feel free to clarify story [20:02:33] sorry for late reply [20:02:42] ok [20:04:51] someone broke margins in watchlist... [20:05:03] jdlrobson, you changed margins lately? ;) [20:05:06] jgonera: ? how when? :-( [20:05:10] i haven't touched margins :/ [20:05:38] hm, I'm just looking at watchlist from master and it seems to have unnecessary margins (those that "text content" pages should have) [20:05:54] will check that [20:06:27] New review: Jdlrobson; "also install the geodata extension" [mediawiki/extensions/MobileFrontend] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/43594 [20:07:07] New patchset: JGonera; "(bug 44089) Fix js/css for non-English special pages" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/44699 [20:08:03] awjr: hey, tfinc just updated me regarding the bits outage yesterday, specifically re: the culprit likely not being MF. sorry for raising the alarm about the timestamps in URLs. [20:08:37]