[10:52:53] making tea [11:13:41] hey qchris [11:13:47] Hi average [11:14:28] waiting for this tea to cool down [14:39:57] hey ottomata [14:40:05] morning! [15:59:26] brb, [16:40:11] ping qchris [16:40:15] Hi nuria [16:42:47] ah ok, it is working, very well [16:42:54] :-) [16:44:52] ottomata: How often do we expect the pipelines that do the cache log line filtering (puppet: templates/udp2log/*) to restart? [16:46:29] hm [16:46:38] udp2log is sighupped daily for logrotate [16:46:43] that probably restarts each filter [16:46:46] (The broken log lines all seem to affect only oxygen, and aling nicely to full/half hour) [16:46:53] hm [16:46:58] puppet runs every 30 mins [16:47:06] We see them restart up to 15 times/day [16:47:23] Ja, puppet looked like the culprit to me as well [16:47:37] the service is not subscribed to the config file, so it wouldn't be weird changes in that [16:47:38] hm [16:48:11] hmm [16:48:17] plot of minutes of when the broken lines occur: http://i.imgur.com/gENZ6kL.png [16:48:20] but puppet has the service declared with [16:48:24] hasstatus => false, [16:48:24] hasstatus => false, [16:48:25] and [16:48:28] http://docs.puppetlabs.com/references/latest/type.html#service [16:48:29] says [16:48:38] If a service’s init script does not support any kind of status command, you should sethasstatus to false and either provide a specific command using the status attribute or expect that Puppet will look for the service name in the process table. [16:49:14] we don't set status [16:49:19] and hm the process name is [16:49:21] service { "udp2log-${name}": [16:49:31] but if I ps, the process name is of course just /usr/bin/udp2log [16:49:39] not sure how it looks in the process table [16:49:43] hmm, running puppet now, seeing what it does [16:49:46] Can it be it is restarted if some variable changes, on which the pipeline config depends? [16:50:13] in this case no, since the service does not subscribe to any config files [16:50:42] hm i'm not seeing puppet restart udp2log [16:50:57] Mhmm. [16:51:01] hm, it does too have a status command! [16:51:04] root@oxygen:~# service udp2log-oxygen status [16:51:04] * udp2log-oxygen is running [16:51:04] root@oxygen:~# echo $? [16:51:04] 0 [16:52:01] But all the log files of oxygen that I have access to basically have broken lines at the same date. [16:52:04] ok so every half an hour this happens [16:52:11] those are actual minutes in the hour? [16:52:15] Yes. [16:52:15] so, around 29 or 30? [16:52:20] Yes. [16:52:53] these are the times of recent puppet runs from syslog [16:52:54] on oxygen [16:52:55] https://gist.github.com/ottomata/7990254 [16:53:12] looks mighty suspicious, eh? [16:53:22] Yes. [16:53:32] hm, ok there is a run coming up then [16:53:39] i will watch syslog and udp2log [16:53:41] But that's ok. Because it seems there is no other hidden race conditions [16:53:46] It's just puppet. [16:54:05] At most 48 times/day throwing away something like ~1 second of traffic. [16:54:25] (Worst case was ~2.5 minutes) [16:54:44] But why is this only happening on oxygen and not on emery? [16:54:52] well, i'm wondering if I just set hasstatus => true if this will fix it [16:54:53] but not sure [16:54:55] yeah that is weird [16:54:59] should be same setup on both [16:55:11] cache_configuration = scope.lookupvar('::role::cache::configuration::active_nodes') [16:55:19] ^ is only in oxygen filters file. [16:55:28] Maybe that dependence triggers restart? [16:55:57] That would basically explain the numbers we're seeing. [16:56:43] right, that could be possible [16:56:50] but udp2log is not subscribed to its config file [16:56:53] Puppet might reinstantiate the filters template for oxygen upon running puppet. [16:56:54] so puppet shoudln't restart it even if the file changes [16:57:00] OH [16:57:02] man i am bad at reading sometimes [16:57:03] yes [16:57:05] it is subscribed [16:57:11] \o/ [16:57:17] i saw a require there and just thought, oh it is just requireing files [16:57:22] but it has a subscribe on a different line [16:57:22] subscribe => File["/etc/udp2log/${name}"], [16:57:25] hmmm [16:57:40] yeah there needs to be a sort in the template there [16:57:46] to make sure the regex string is always the same [16:57:59] ? [16:58:05] Ah! [16:58:05] well, its a hash right? [16:58:11] I think so. [16:58:13] the order is not guarunteed [16:58:24] when i do .values [16:58:30] i might get the values list in a different order [16:58:33] so i need to do values.sort [16:58:35] Yes. Totally! [16:58:48] Sounds like that might me the reason. [16:59:00] Oh wait ... Daily Scrum. [17:01:20] ottomata: Daily scrum. [17:01:30] ah [17:16:05] qchris: tried connecting throuhg an intermmediate node, no dice [17:16:40] average: :-( [17:17:36] I have to return the RAM I bought. I order a physical machine equipped with all the stuff. A friend recommended a configuration(the parts that I should get) so I should get that soon, I placed the order today [17:17:44] *ordered [17:19:38] so hopefuly I'll just install all the stuff I need on it when it arrives and I'll be able to use it. [17:33:48] milimetric_, ottomata, nuria: Do we have the Sprint Planning Meeting today? [17:34:20] iunnooooo [17:34:22] :) [17:34:26] qchris, i'm pretty sure this is the problem [17:34:33] i just watched another puppet run before I merged my sort change [17:34:34] File[/etc/udp2log/oxygen]/content: content changed [17:34:36] and it was restarted [17:34:40] That's good. [17:34:41] Service[udp2log-oxygen]: Triggered 'refresh' from 1 events [17:40:40] qchris: we're all in [17:40:49] sorry for the delay [17:41:20] ottomata: we'll make some high level card for your stuff and task it out soon [17:41:27] toby wanted to track it a little better [18:04:02] Ironholds: what do you mean there's no connection from 1002 and the analytics slaves? [18:04:06] what do you need other than a mysql client? [18:04:33] if there is a mysql client there, you should be able to connect with whatever your mysq l username / pw are [18:04:34] right? [18:04:42] ottomata: oh, cool. Yeah, I was just being on the safe side. [18:04:47] k [18:04:55] (that is, the "assume I don't know everything and there may be further hurdles" side) [18:09:57] Ironholds: [18:09:58] root@stat1002:~# which mysql [18:09:58] /usr/bin/mysql [18:11:32] aha; thanks :). Seems to be all connected up. [18:11:52] er. Any chance I can add in another RT ticket, this one to install RMySQL? I apologise for sucking your time into unexpected things :/ [18:17:11] Ironholds: gerrit change for operations/puppet repo ? [18:17:59] Ironholds: also, R packages can be installed locally http://csg.sph.umich.edu/docs/R/localpackages.html [18:18:13] Ironholds: so you can use that to bypass ops and just DIY [18:21:36] average: most of the time, but RMySQL is an exception (or, was the last time I tried it) [18:21:41] but I'll try again [18:22:47] Ironholds: come back with some errors please. they would help me figure out what's going on [18:23:24] shall do :) [18:27:45] "could not find the MySQL installation include and/or library [18:27:45] directories. Manually specify the location of the MySQL [18:27:45] libraries and the header files and re-run R CMD INSTALL." [18:27:48] I'll try that. [19:02:06] Ironholds: is there an rmysql package? [19:03:45] ottomata, there is, and it's now installed (danke to Stefan) [19:04:11] oh ok [19:04:11] cool [19:04:12] ! [19:18:46] Question about visualisation [19:19:16] I've a few timeseries that are pretty much ready to be generated, the script just needs to get to stat1 and run a few times [19:19:34] But I've also got a few things I'd like to see distribution on - load times, image sizes, image types [19:20:01] Does limn do that nicely or am I swimming against the tide there [19:20:34] milimetric_ , nuria ^^ [19:20:51] hey marktraceur, in a meeting in Elder [19:20:55] we'll come talk after? [19:21:00] (thanks average [19:21:01] ) [19:21:06] Sure [19:21:07] np [19:21:29] Woludn't mind having a concept of the difficulty before writing it :P [19:23:19] limn doesn't do much data analysis, so you'd have to compute that stuff yourself. Is that what you're asking? [19:46:46] I can compute it, I want to know if it can visualize a plot like that [19:46:53] Or what's the best way to do so [19:56:50] ok I'm here now, so please elaborate Mark [19:57:28] what I understand so far is: [19:58:06] 1) you have some timeseries and you want to plot them. in what format do you have the data ? [19:58:21] They'll be in CSV, but right now they're raw EventLogging stuff [19:58:26] I haven't run the script yet. [19:58:31] I see [19:58:46] 2) <-- this part is a bit fuzzy about the load times and image sizes and image types [19:58:51] do tell more [19:58:53] 'kay [20:00:17] average: There are a bunch of EL data in the logs that have information about image load times, and additionally list image filetypes (png, jpeg, etc.) and image sizes (in pixels and in bytes). I'd like to see distributions of the sizes and of the file types as well as the timeseries, but I'm not sure if I can do the distributions over time, or how best to do that, or what [20:07:11] marktraceur: I see. So what have you tried so far in terms of visualization [20:07:49] average: I haven't tried anything yet, really, I don't even know what the CSVs should look like yet [20:50:23] Ironholds: what sort of monster produced that json [20:50:59] and what will the cost to my eternal soul be in order to summon the dark forces necessary to turn it into something normal? [20:53:31] protonk: no idea. have fun! [20:53:34] the really fun thing, though. [20:54:05] there's like semi-colons and everything in the midst. [20:54:22] it's actually *R* escaping it. [20:54:24] if that's what the db provides, it needs a bug filed, IMO. [20:54:29] oh, don't give me what R gives you [20:54:34] returning with LIMIT 1 produces magical, magical normalised data. [20:54:38] give me what the db spits out [20:54:40] returning without produces escaping [20:54:41] aha [20:55:05] sent [20:58:33] that's still not valid json. e.g. s:5:"Media" couldn't anywhere be valid [20:58:41] nor should there be semicolons in there [20:59:27] * Ironholds waves hands [20:59:30] this is how it's in the DB [20:59:38] maybe PHP has some voodoo parser. It tends to for a lot of things :/. [20:59:52] I may just grab the list of wikis and query each one for the namespacenames site props. [21:12:49] milimetric_, any luck finding out what a CSV should look like? [21:13:07] yes, but we still don't have an example graph for you marktraceur [21:13:15] so the data is simple, it can look like this: [21:13:26] Label1,1234 [21:13:31] Label2,12345 [21:13:33] 'kay [21:13:44] so we're working on making that work atm [21:13:50] i'll def. ping you when it's done [21:13:54] I don't mind, as long as I can build up the CSVs [21:14:01] I haven't written the generation for it yet [21:14:03] cool, yeah, you can consider that a contract [21:14:20] brb, we're gonna do lunch [21:19:29] hey drdee [21:19:50] I'm trying the creds you gave me for analytics-l but they don't work [21:39:02] Ironholds, protonk: I missed the start of the conversation so I'm not sure what you're talking about but I think you're not looking at JSON but a serialized array in PHP stored as a blob in the MW db [21:39:22] DarTar: aha. that would explain it, although my followup is 'jesus why'/ [21:39:27] ah, that explains it [21:39:34] in any case, I'm resorting to halfak's 'query ALL the wikis!' idea. [21:39:58] the only thing more fun than development is development with only one functioning hand. [21:40:05] http://php.net/serialize [21:40:13] off to grab some food, bbl [21:40:29] Ironholds: just http://www.php.net/manual/en/function.unserialize.php ? [21:40:42] then I'd have to write PHP. Bah. [21:41:07] I' trying to minimise typing need [21:43:19] it's probably worth it. just write a script that unserializes it and then passes it to php's json encoder and returns that output [21:45:03] vs whatever craziness might occur if you try to use my heuristic, lol [23:35:25] ping DarTar [23:35:44] did you requested access for me to stat1.wikimedia.org? [23:37:13] nuria: he did, I can see the ticket. [23:37:16] * Ironholds checks again just to be sure. [23:38:26] argh. why is RT search so terrible [23:42:06] nuria: I lied! Can't see it :/. [23:42:10] I could swear...hmn.