[08:34:37] halfak: wheels! [17:55:06] http://labels.wmflabs.org/campaigns/trwiki/?campaigns=stats [17:55:09] somethıng ıs down [17:55:14] 502 Bad Gateway [17:55:24] @YuviPanda should I panic? [18:04:25] @halfak [18:04:49] ores seems to be online [18:05:34] Looks like I can't ssh into the web node. [18:05:44] "channel 0: open failed: administratively prohibited: open failed" [18:06:09] I am able to ssh to staging/test [18:06:10] halfak: are you sure the hostname is correct? I typically get that if it doesn't resolve [18:06:28] If someone changed it since the last deploy... [18:06:33] could be a labs dns issue as well, of course... [18:06:34] It's the host in our fabfile.py [18:06:42] which node is it? [18:06:53] labels-web.eqiad.wmflabs [18:07:15] Wait... web is up. [18:07:18] It's something else. [18:07:25] DB? [18:07:42] halfak: labels.wmflabs.org proxy points to http://wikilabels-01.wikilabels.eqiad.wmflabs:8080 [18:07:45] halfak: I get ping: unknown host labels-web.eqiad.wmflabs [18:07:52] halfak: madhuvishy and I switched it a month ago remember? [18:07:54] Yeah... Can't SSH to the node, but I get the default template at http://labels.wmflabs.org/ [18:08:26] YuviPanda, just checked, this is the host in the fabfile [18:08:45] So if we did a deploy with fabfile, this should be the web machine. [18:09:05] https://github.com/wiki-ai/wikilabels-wikimedia-config/blob/master/fabfile.py#L52 [18:09:07] halfak: I remember madhuvishy making a local commit to change the hostname [18:09:10] on the fabfile [18:09:12] when we did the move [18:09:15] maybe she didn't push it? [18:09:20] * halfak gumbles agressively [18:09:22] halfak: YuviPanda No, I din't change it [18:09:31] * halfak stifles grumbling [18:09:59] or make local commits. [18:10:18] oh... [18:10:20] we should change it [18:10:28] halfak: once lightning talks are over, i'll look into it and see that you have a staging setup [18:10:45] YuviPanda: should we switch staging to centralized puppet master too? [18:10:51] madhuvishy: yes, need to recreate it [18:10:59] halfak: I kicked it and it's back online now [18:11:15] YuviPanda, should I update the fabfile to point to wikilabels-01.wikilabels.eqiad.wmflabs? [18:11:20] Thanks YuviPanda [18:11:22] halfak: yup [18:11:36] halfak: I also need to add you to the labs project... (sorry!) [18:11:39] halfak: what's your wikitech username? [18:11:48] halfak [18:11:56] * halfak halfaks the halfak [18:12:21] interesting [18:12:25] 'failed to add halfak' [18:12:35] I'm already a member [18:12:37] if not an admin [18:12:59] that explains why it won't let me add you [18:13:03] Maybe you just need to grant me admin ;) [18:13:04] * YuviPanda goes to find his eyes [18:13:18] halfak: actually you are admin too [18:13:23] I... really need to go find new eyes [18:13:26] I tried it 3 times [18:13:30] and then tried to add you as an admin [18:13:39] from a link right next to where it listed members and admins [18:13:41] including you [18:13:45] YuviPanda: :P [18:13:49] :D [18:13:56] Got the log viewing command handy? [18:14:03] journalctrl or something [18:14:52] sudo journalctl -u uwsgi* [18:14:53] sudo journalctl -f -u uwsgi-* [18:15:04] halfak: and remove the -f if you want to see full log [18:15:45] Got it. No obvious errors. [18:16:33] YuviPanda: is the db configured? [18:16:58] madhuvishy, I can tell that it is working with the DB right now. [18:17:08] halfak: aah okay [18:18:09] madhuvishy: in wikilabels-01 yeah? [18:18:10] err [18:18:10] yeah [18:18:12] I suspect that my DB connector hand an event and died. [18:18:33] I wanted to confirm so that I could justify the time spent looking into how to make that more robust. [18:22:35] Soo. It looks like staging is in the same errored state, but I can't get the uwsgi log to show anything for the error. [18:22:48] Last event was 10 minutes ago. [18:22:54] And I've been recreating the error. [18:23:36] This is what I see: https://gist.github.com/halfak/04c88acd8b013ed89a8f [18:23:47] I created the 502 at 18:22 UTC [18:23:51] No log item [18:24:09] hmmm [18:24:30] I wonder if the uwsgi restart isn't happening properly [18:24:44] halfak: can you do 'ps auxf' to see the tree, see if there's more than one master uwsgi process? [18:24:50] halfak: or what host is this, I can probably look too [18:25:25] https://gist.github.com/halfak/5d36a1763e886a0198e1 [18:25:32] ^ This gist will answer both questions [18:37:34] hah [18:37:39] I seem to have a nack ın [18:37:47] I seem to have a nack in detecting our rare downtimes [18:53:47] halfak: it seems like it's all up now? [18:53:54] sorry was at lightning talks. [18:54:07] Yes. YuviPanda restarted something. [18:54:16] I did the fab deploy with the host as a command line param [18:54:18] I've been struggling to find out what was causing the error. [18:54:30] i made a mental note to push the right url and forgot [18:54:35] The repo now has the updated path. [18:54:39] sorry about that [18:54:42] No worries. [18:54:51] But still any idea how we can find out what that error was? [18:54:54] yup, saw that. I'll also set up staging today/tomorrow [18:54:56] uwsgi seems to not know. [18:55:17] Could it be that the error originates at nginx? [18:55:27] Before you do anything, check out staging (labels-test) [18:55:33] it is still in the errored state. [18:55:44] halfak: okay, checking [18:56:37] halfak: http://labels-test.wmflabs.org/campaigns/trwiki/?campaigns=stats is this not the expected response? [18:58:03] Darn. Must have gotten bumped somehow [18:58:20] Yeah, this URL was returning a 502 earlier: http://labels-test.wmflabs.org/campaigns/ [18:59:08] Hmmm, interesting [18:59:11] https://www.irccloud.com/pastebin/WKUpsjXb/ [18:59:28] See my gist above madhuvishy [18:59:38] https://gist.github.com/halfak/5d36a1763e886a0198e1 [18:59:58] Conversation with YuviPanda starting with "I suspect that my DB connector hand an event and died. " [19:00:42] I wonder if YuviPanda decided to restart the uwsgi [19:00:56] And how we can troubleshoot this without it staying in the errored state [19:05:49] hmmm halfak I see this pattern almost every day - not sure if it's normal [19:05:55] https://www.irccloud.com/pastebin/47U3VU4O/ [19:06:47] It looks like there's 30 minutes of downtime there. [19:07:18] yeah [19:07:25] and it keeps coming up in the logs [19:09:52] * halfak hates surprise maintenance issues [19:12:30] this is strange [19:14:39] YuviPanda: did you do a manual restart on wikilabels-test? [19:15:25] madhuvishy|lunch, if you have an idea, but not time to pursue it, please put it on my plate :) [19:15:28] madhuvishy: halfak yes sorry... [19:15:46] I thought I said it here but network lost it [19:16:21] oh! [19:16:29] Oh... wait. [19:16:40] I thought you meant you knew why labels was all derpy [19:16:52] Oh I restarted uwsgi [20:03:30] o/ YuviPanda [20:03:35] Postgres is different! [20:03:42] Because of issues a while back. [20:03:48] I need to help Shilad get back in. [20:04:05] I'm here now :) Thanks, Halfak! [20:04:30] YuviPanda & madhuvishy|lunch could potentially help. [20:05:38] Whoops. My bad. Was logging in with the wrong username :( (shiladsen instead of u_shiladsen). [20:05:42] It's still alive! [20:05:47] Woot! [20:06:03] nevermind any of this ^^^^^^^^^ [20:06:56] see I can solve problems by looking at screens [20:07:09] Deep opsen magic [20:07:33] Thanks :) [20:08:08] One related question: I have a 140GB mount on my VM, which is the largest I could find. Anybody know if it's possible to mount multiple disks, or larger ones? [20:08:45] I don't think that is possible, but you have some options still. [20:08:57] How would you use that additional storage? [20:09:05] More languages (hopefully all) [20:09:18] Big file writes? Random reads? [20:09:21] Append only? [20:09:32] Got it. Big read-only, random access files. [20:09:53] Random access of big files? [20:10:43] Yes. There are a bunch of matrices various algorithms need that are stored as memory mapped files. [20:11:03] Gotcha. How often do you expect to read one of those into memory? [20:12:04] And how big will each file be? [20:12:34] They are typically used as a kind of very fast version of memcache. So, truly random access. [20:12:50] Let me check size... [20:12:58] But they'll be in memory by that point, right? [20:13:14] I'm really only concerned about when you'll be reading from the disk. [20:13:48] Not really. The memory mapping allows the OS to decide what's kept in memory. [20:14:13] I see. So you want the OS to cache intelligently. [20:14:18] You tell the OS to memory map the file, it immediately says OK, then pages reads of the file in on request. Much faster than standard disk reads. [20:14:19] That's going to be harder. [20:14:20] Yeah. [20:14:33] We used to go to a database for it, but it was too slow. [20:14:53] Do you read parts of the file? e.g. with a seek offset? [20:15:51] * halfak thought you were pulling a while file into memory [20:16:38] Yes. By seek offset. Here's the file: https://github.com/shilad/wikibrain/blob/master/wikibrain-matrix/src/main/java/org/wikibrain/matrix/MemoryMappedMatrix.java [20:17:13] Reads in seek offsets for each matrix row, then grabs rows at random on request. But the OS handles the actual reading from disk lazily on demand. [20:17:20] (And caches entries as usual). [20:18:20] So, it looks like you *might* be able to use postgres' large object store and use seek() -- but I'm not sure if that will do intelligent caching. [20:18:23] http://www.postgresql.org/docs/current/static/largeobjects.html [20:19:18] YuviPanda, any recent news on larger disk capacity in labs? [20:19:39] I think the latency to PG would kill us, sadly. [20:20:02] yeah, mmap is goign to beat the pants off everything [20:20:20] how big? [20:20:22] we can make custom one offs [20:20:30] but only in extreme cases [20:22:46] Shilad, how do you feel about standing up additional instances for supporting more languages? [20:22:56] Would it work to partition that way? [20:23:16] I could definitely do that. [20:23:26] OK. That's much easier. [20:23:43] It might be tricky to do cross-lingual across partitions, but I suspect that's an uncommon use case. [20:23:53] You should be able to start up at least two x-large instances. [20:25:43] I think I probably need three to fit all languages, but two will get me most of the way there. [20:26:04] Any requests for specific languages on the first instance I bring up? [20:31:45] en, de, pt, es, fr, it, ja [20:31:50] in that order [20:32:04] My personal opinion [20:32:14] Oh! zh! [20:32:18] FOrgot about zh [20:33:46] Sounds good. I'll add no for Morten. [20:35:00] I haven't extensively tested CJK. This should be exciting :) [20:37:01] CJK for the confuse [20:37:05] :) [20:37:23] I just did some hacking with unicode so that I can tokenize CJK better. [20:37:28] Git times. [20:37:31] *Fun! [20:37:35] ALso git, I guess [20:38:12] I use Lucene's tokenizer. Hopefully it's git :) [21:11:10] shilad: halfak feel free to poke me if you need lifted quotas [21:16:13] shilad, BTW, quota == your CPU usage. [21:16:27] When YuviPanda lifts the quote, we could start up additional large machines. [21:16:31] Okay. Thanks! [21:16:42] I think it will be good to start with two if that doesn't sound like much more work [21:16:55] I agree. [21:33:54] halfak what is the brand and model of your battery btw? [21:34:36] See PM [21:40:42] Halfak: Do you have a pointer to documentation for your rev scoring API? Looking for a useful model. [21:41:08] shilad, the http API or the revscoring library? [21:41:26] HTTP end points: https://meta.wikimedia.org/wiki/Objective_Revision_Evaluation_Service [21:41:36] Python library: http://pythonhosted.org/revscoring/ [21:42:45] * Helder copies that link to https://phabricator.wikimedia.org/T108305#1572656 [21:45:11] http api, thanks! [21:48:27] "This format is extremely hard to parse." [21:48:29] WAT! [21:48:37] JSON is hard to parse? WUT [21:49:42] { "user_is_stupid": true } [21:50:14] WHAT? I CAN"T PARSE THAT! [21:50:54] TIL xml is friendly [21:52:06] I refuse to use WDQS because the output is XML and not JSON. So I'm using Magnus' hacky stuff. [21:55:09] halfak: didn't you know, json is very new? [21:55:19] http://www.gossamer-threads.com/lists/wiki/wikitech/547495 etc [21:55:55] http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs/20130728.txt has some fun stuff too [21:56:29] The early 2000s were within my lifetime. [21:56:36] I suppose that's new for a lot of technolgies. [21:57:29] also mysql doesn't scale, but that's another story. [21:58:58] Oh... well that one is true. [21:59:06] For some scales [21:59:07] I was 8 in 2000 [21:59:12] Which is true of all things ever [21:59:24] I was ... 17? [21:59:25] I was 9 [21:59:26] yeah [21:59:29] lol [21:59:31] Youngins [21:59:48] I'm too young to be an old, bearded internet person [22:00:30] halfak: I present, https://wikitech.wikimedia.org/wiki/XmlRcs [22:00:57] * halfak is catching up on this whole pile of ... stuff [22:01:01] wow [22:01:57] So, there's a python converter in the data flow for "Lightweight C/C++ program that doesn't use any 3rd party libs" [22:02:09] And it's running on another machine that can go down [22:02:36] But you don't have to run the 3rd party library that does the conversion yourself. [22:02:57] Is there a licensing issue that I'm missing? [22:03:10] no [22:03:19] But 3rd party libs are awesome [22:03:21] the XmlRcs app also constructed XML with string interpolation [22:03:30] It's like there's a wiki in my software and the template language doesn't suck. [22:03:42] heh [22:03:56] YuviPanda, but wait. I thought that headaches!? [22:04:01] For months or something. [22:04:04] For writing a parser [22:04:41] heh [22:04:52] PERL6 IS CLEARLY NEEDED FOR ITS SUPERIOR GRAMMAR SUPPORT [22:08:56] * halfak would check out perl6 if it checked any of the boxes I don't already have checked in python land. [22:09:04] And if it didn't uncheck a bunch. [22:09:25] Wait.. perl == longer beard and CS prof. cred. Those are some boxes. [22:09:40] halfak: I think it's the best language for doing any kind of complex parsing atm [22:11:30] Yeah. That's a good point. I did some work with ml-lex and ml-yak. [22:11:50] Only one option (that's meh) in python [22:12:09] Good utilities if you find you ever want to parse python *in* python. [22:12:48] I actually wrote my own tokenizer (which is largely based on perls regex syntax) library because python lacked a good/fast one. [22:13:05] * halfak *sighs* a deep breath [22:13:29] I've written enough libraries to know when to use someone's stuff. [22:15:41] Seriously though, I love the 3rd party library environment in python. [22:15:54] Except for the install pains, it's pretty awesome. [22:16:11] I just picked up mwapi [22:16:12] :) [22:18:05] halfak: +1 [22:18:12] halfak: libraries are great :D [22:18:25] halfak: I'm writing some for perl6. the grammars are basically the killer feature there, I think. [22:18:34] it's basically made regexes usable in a way I hadn't thought I could [22:18:39] I hope something like it makes it to python [22:22:44] :D One thing you rarely see outside of parser work is a well-structured lexicon. [22:23:08] -- a table of (regex, name) pairs [22:23:41] https://github.com/halfak/deltas/blob/master/deltas/tokenizers/wikitext_split.py#L64 [22:24:45] I decided not to implement the lexer. [22:25:11] ah [22:25:16] so I've never written a parser in my life [22:25:19] Not much use for this kind of work. Though, I would have liked having a lexer in this awful sight: https://github.com/halfak/deltas/blob/master/deltas/segmenters/paragraphs_sentences_and_whitespace.py [22:25:20] not smart enough :D [22:25:24] until the perl6 grammar thing [22:25:49] I see https://github.com/halfak/deltas/blob/master/deltas/segmenters/paragraphs_sentences_and_whitespace.py#L62 and my eyes roll over [22:25:55] I should try it some day.... [22:26:14] Don't look at that. [22:26:22] Never look at that! [22:26:24] lol [22:26:36] That would be 3 lines in a good lexer. [22:27:17] heh [22:27:23] I don't actually know what a lexer is either :P [22:27:33] It's the thing that groups together the tokens. [22:27:34] unlike madhuvishy I went to a terrible university :P [22:27:41] Me too. [22:27:46] I only got this stuff in grad school. [22:27:54] ah, heh [22:27:56] My undergrad was way better for my original major [22:27:59] I am not a CS student :P [22:28:00] Physical Therapy [22:28:05] halfak: oh wow, really? [22:28:11] Must get massages from you! [22:28:21] I started CS because I took a programming course and it was like amazing thought legos that work. [22:28:27] madhuvishy: IT at SSN :P [22:28:30] halfak: wow, nice! [22:28:41] still dont know how lexers work [22:28:49] I assumed you were one of those people woh've been coding since 10 [22:28:57] madhuvishy: liar :P [22:28:57] yeah me too [22:29:25] YuviPanda: I haven't written a fancy PERL6 parser either [22:30:19] I haven't wriwtten a perl6 parser, just a parser in perl6 :P [22:31:00] Na. I'm totally someone who went through college and some of grad school with an imposter syndrome around my lacking CS background. [22:31:52] I did play video games and go to LAN parties in highschool. I just thought that programming was boring work for boring people who like rules. [22:33:03] halfak: oh, so you got all the goodies in gradschool? [22:33:36] Yeah. That and on the job. [22:34:08] I did some software engineering work for 3M and Thomson West (now Thompson-Reuters). [22:34:20] That taught me the amazing and terrifying nature of Java. [22:34:29] Oh and VBScript. [22:34:34] And MSSQL Server [22:34:38] TSQL, man [22:34:55] aaah [22:34:57] vbscript [22:35:02] how I miss case insensitivity :D [22:35:05] halfak: sound like fond memories! [22:35:10] No. [22:35:12] Ahh [22:35:15] Nooooo [22:35:49] case discipline FTW [22:37:04] heh [22:38:53] YuviPanda, how do you feel about taking the License out of the main file in mwapi? [22:38:59] You already have a LICENSE file [22:39:13] I want to replace it with documentation. [22:39:15] you mean in the license file header? [22:39:17] sure [22:39:23] Cool. [22:39:28] I'm happy giving you merge rights [22:40:03] I have some potentially controversial suggestions that I'll submit as PRs either way. [22:40:11] ok! [22:40:20] :D [23:42:19] YuviPanda, https://github.com/yuvipanda/python-mwapi/pull/10 [23:45:12] halfak: merged [23:45:27] \o/ [23:45:41] No to the controversial stuff. [23:46:11] How do you feel about me removing methods that don't work or probably don't belong? [23:46:26] e.g. get_token() doesn't work anymore [23:46:46] halfak: +1! I'd like you to bump version up to at least 0.2 before starting doing any of that tho [23:47:29] YuviPanda, will do. I've usually been following the practice of bumping the version when I am ready to tag it or push it to pypi. [23:47:40] Would you like me to bump it in my PRs instead? [23:47:56] halfak: yeah, so that it's easy to see where the breaking changes start [23:48:03] Gotcha. Will do. [23:48:12] Time to go ride bikes now though. [23:48:13] :) [23:48:14] o? [23:48:18] *o/ [23:48:35] o? looks like he's scratching his head. How apt for a question mark. [23:48:37] halfak: \o/ [23:48:39] haha :D