[00:00:34] i happened to see erik in the hall, and ran the ideas for #92 by him [00:00:35] (and robla) [00:01:05] he asked a few questions i added to the notes, and slimmed down the optins [02:19:03] milimetric: http://test-reportcard.wmflabs.org/dashboards/donations [04:50:16] 03 20:07:11 < jeremyb_> pipe 10 is 10% and pipe 100 is 1%? [06:01:28] so, i think that's right. but still not sure [06:01:50] not sure where to look for docs either. i guess i'll try man page in labs tomorrow [06:02:17] can't tonight because last time i tried to ssh to labs i forkbombed myself (locally) and i'm too tired to figure out the problem now [06:02:46] if it is right then: [06:02:47] 04 05:57:43 <+gerrit-wm> New patchset: Jeremyb; "fix comment on udp2log sample rate" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57471 [06:09:56] jeremyb_: forkbombed ? [06:09:58] how ? [06:10:16] while(1){fork;} ? [06:10:20] average_drifter: i assume must be ProxyCommand infinite loop [06:10:26] ssh calls ssh calls ssh [06:10:31] but i'm too tired now [06:10:42] oh alright [06:10:58] jeremyb_: I can show you my proxycommand [06:11:08] nah, i'm very familiar with it :) [06:11:17] i wrote half the docs for it on [[help:access]] [06:11:25] i must have just left mine in a bad state [06:11:38] probably after a bastion outage or something [06:12:12] if you need any help running some commands on labs that would fix it please let me know [06:12:21] I mean if you can't get in, I can run them for you [06:12:33] nah :) [06:12:41] you can review my patch linked above :) [06:12:48] reading it now [06:14:22] jeremyb_: if you add me to the reviewers list [06:14:24] :) [06:14:30] just read it, looks legit [06:14:51] I'd give thumbs up on it [06:16:16] jeremyb_: reviewed [06:16:28] jeremyb_: +1-ed [06:16:32] :-) [06:17:34] I just saw an app on appstore, it's a text editor for ipad [06:17:56] I don't understand why someone would want to use it since you can't type on the ipad in the first place [06:18:55] hehe [12:41:41] hey guys... what's up with kraken? [12:41:47] when can people start using it? [12:42:23] i'm thinking e.g. if we find useful data in the NARA log then we can automate summarization / anonymization / etc. [12:42:37] and then use that as the input for new or existing tools [12:47:50] for reference, i think most GLAMs use these stats now: https://toolserver.org/~magnus/baglama.php [13:31:44] moooorning [13:32:45] hey jeremyb_ GLAM related pagevews are on the roadmap; see https://mingle.corp.wikimedia.org/projects/analytics/cards/150? [13:32:51] would love your input! [13:32:59] morning ottomata, milimetric [13:33:04] morning [13:33:55] ping average_drifter [13:34:30] heyayyyy no segfaults! [13:36:22] good! [13:36:29] * drdee is happy pumpkin [13:37:37] drdee: could you give me like ~20ish lines of that file NARA file as a sample? [13:37:50] (or whatever a convenient amount is) [13:37:52] hm, drdee, where did the card with the udp2log maintenance stuff go? gonna update about webstatscollector [13:37:58] drdee, can I give him anonymized lines? [13:37:58] s'ok? [13:38:59] did you start labs kraken yet? [13:39:21] no let's not give anonymized data right now, let me talk to multi chill first [13:39:44] about udp2log maintenance stuff, that should still be card 460 or not? [13:39:58] i lose things if they aren't in wip :) [13:40:00] thank you [13:40:08] it should be in wip [13:40:11] let me check [13:40:13] OH, the title changed [13:40:21] yes a little bit :) [13:40:28] to more accurately describe the feature [13:40:41] ping average_drifter, you wanna have a hangout with me and milimetric? [13:41:00] drdee, I'm handling a prod limn issue [13:41:02] unfortunately [13:41:06] k [13:41:24] drdee: pong [13:41:35] i guess so, ha [13:41:42] webstast collector is advanced monitoring :p[ [13:42:39] drdee: we could have a hangout but I'm not blocked right now [13:43:27] ottomata, can you merge https://gerrit.wikimedia.org/r/#/c/57471/ [13:43:46] average_drifter: what's the status of card 60 [13:43:54] do you need a brain bounce? [13:44:00] drdee: it's in progress, I'm not blocked [13:47:15] hm, drdee, i can't merge that, it says: [13:47:16] Need Rebase or Has Dependency [13:47:24] jeremyb_^ [13:47:47] fair warning [13:47:51] I am going to kill dschoon [13:47:54] uh oh [13:47:57] just saying [13:48:04] :) [13:48:12] you, know, so he expects it [13:49:21] ottomata: i wonder why. try again [13:49:29] ohhhh [13:49:38] it's now set to fast forward only [13:49:42] that looks happier, danke [13:49:42] i wonder since when [13:49:57] i just heard them talking about that in #ops, but they weren't talking like it was done [13:50:04] what does that mean though? you always have to rebase? [13:50:07] before you commit? [13:50:43] it means it doesn't do merge commits. if something is not a "fast forward" (git term of art) then it refuses [13:51:23] https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git;a=commitdiff;h=2e347cbabe583945e59591b672746a51aea465b5 [13:51:28] ottomata: ^ [13:52:04] anyway, can't think about that now :) [13:52:06] hm [13:52:22] k, welp, merged! [13:52:51] [travis-ci] develop/99e1347 (#121 by milimetric): The build passed. http://travis-ci.org/wikimedia/limn/builds/6050674 [13:53:31] [travis-ci] master/99e1347 (#122 by milimetric): The build passed. http://travis-ci.org/wikimedia/limn/builds/6050681 [13:54:25] drdee, average_drifter, I had deployment pain for breakfast so I'm gonna follow it with something more nutritious. I can chat in maybe 20 min [13:56:37] I don't have anything to showcase right now [13:56:43] unless you want to do live-coding [13:58:04] thx milimetric for being on the ball! [13:58:29] yeah, average_drifter, showcasing's not necessary [13:58:43] we just wanted to try out a way that we could help each other get unstuck [13:59:02] I think pair programming helps with that a lot [13:59:35] unless you think it would negatively impact your work, let's get into a hangout in a bit [13:59:48] drdee, how long should we wait for people to respond to the emery filter email before I start disabling stuff (I won't disable glam or nara) [14:01:00] milimetric: well I think it's ok if we have things to discuss, sure, it's very helpful to have another oppinion. last time for example, that was a big bug I didn't notice [14:02:10] I'll be in the hangout [14:02:40] yeah it can be a quick 10m meeting [14:05:36] milimetric: I'm in the hangout [14:05:45] milimetric: wanna join ? [14:07:09] milimetric ^^ [14:10:05] joining, i was making breakfast as stated above :) [14:33:17] drdee, any clue on how webstats dumps are rotated or deleted from locke? [14:34:26] that was the cronjob running on dataset1, right? [14:37:16] naw, there's a cron on snapshot1 [14:37:25] that copies from locke to dataset1(2?) [14:37:29] but it doesn't delete [14:37:36] locke only has logs about 12 days old [14:37:48] gadolinium isn't deleting them yet [14:37:52] it doesn't really matter [14:38:04] i'll just put a cron job in to delete them [14:38:24] ok, maybe apergos knows that [14:38:45] yeah, i pinged him in #ops, we'll see [14:38:51] its simple enough i'll just puppetize a cron [14:40:54] k [14:41:06] i rebranded the scrum meeting to "Anonymous Analytics " [15:22:33] erosen: hey [15:22:35] erosen: welcome back [15:22:50] average_drifter: hello again [15:22:52] thanks [17:01:41] erosen: scrum [17:01:46] aye [17:01:49] wasn't sure if we had it [17:02:45] drdee: can someone paste the linke [17:02:55] seems like no one is in the event link [17:03:18] dschoon: ^ [17:03:23] kraigparkinson: ^ [17:03:33] hm, we're here... [17:03:40] https://plus.google.com/hangouts/_/053b64aba37418dd6ee597f1630d15223077ff7a [17:03:42] that one? [17:03:54] https://plus.google.com/hangouts/_/2da993a9acec7936399e9d78d13bf7ec0c0afdbc [17:51:10] hooray. i arrive avec muffin [18:47:58] ottomata: wanna deploy some conf changes? [18:48:59] basically i want more ram :) [18:49:30] oo! sounds fun [18:49:33] gimme 7 mins? [18:49:56] sure [18:55:05] i have a meeting in 5 [18:55:09] so i'll commit a few changes [18:55:12] and you can go over them [18:55:21] ok cool [18:56:32] pushed [18:57:23] i have a ridiculous ipv6 job running atm for paravoid [18:57:25] just fyi [18:57:50] in kraken? [18:58:07] yes. [18:58:11] why? [18:58:36] he was curious, and i'm sitting in a meeting :) [18:59:31] apparently it has 18k maps and 1000 reduces. [19:00:45] ha, cool [19:00:54] yay fairscheduler! [19:01:00] exactly :D [19:01:03] it'll be done in 2h [19:06:46] drdee, kraigparkinson be there in a sec [19:06:48] coming downstairs [19:07:48] in diderot [19:20:41] kraigparkinson, drdee, milimetric: https://github.com/wikimedia/jquery.uls https://github.com/wikimedia/jquery.ime [19:31:56] https://www.mediawiki.org/wiki/Analytics/Requirements_Checklist [19:32:21] heya dschoon and milimetric, can I get some visualization advice? [19:32:22] http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&tab=v&vn=udp2log [19:32:31] sup [19:32:33] lookin [19:32:37] which of these would be better stacked that aren't? [19:32:42] I'm going to make InDatagrams stacked [19:32:49] do you think any others would be better? [19:33:13] (spikes 30 mins ago were caused by initialization of values ( i think)) [19:33:33] the graph shoudl scale down and be visible in another 30 mins [19:34:35] milimetric: my understanding about consensus-driven intentional communities mostly comes from occupy as well [19:34:41] i found this to be a good read: http://rortybomb.wordpress.com/2011/09/30/understanding-the-theory-behind-the-different-approach-of-the-occupy-wall-street-protests/ [19:34:52] the anarchists are interesting, but not really my bag [19:34:56] not pragmatic, imo [19:37:20] ottomata: my rule of thumb is that graphs which represent a quantity of events often look good stacked [19:37:28] graphs that are compound metrics -- averages, maxes, etc [19:37:30] do not [19:37:33] they become more confusing [19:37:46] stacking is a way of visually aggregating [19:37:55] aggregating aggregates is often dangerous [19:38:08] so yeah, InDatagrams, bufErrors [19:38:11] InErrors [19:38:19] drops [19:38:25] those might all work as stacked graphs [19:38:40] but the errors are weird, because they're not really quantities [19:39:17] these are all counters, ganglia converts them into rates [19:40:26] dschoon: made small changes to http://etherpad.wikimedia.org/92 (mostly removed some stuff) [19:40:40] ready to be emailed AFAIC [19:40:50] 1/2 are the same, imo [19:40:59] nobody cares how the data gets onto the cluster, right? [19:41:07] it's just a question of whether it goes there or not [19:41:40] otherwise, it looks great [19:42:59] gah [19:43:10] i just closed the shell where my ip6 job was running [19:43:15] i *hate* that about pig. [19:44:30] dschoon: nobody cares how the data gets onto the cluster, right? [19:44:51] right?? [19:44:59] at least, that's been my feel [19:45:16] i think ops does, if we show it's a temp cronjob that has a different connotation than setting up a real kafka consumer, effectively there is no difference of couse [19:45:43] oh they'd say that's a stupid waste of labor, when we already have a pipeline in place [19:46:05] *or [19:46:07] not oh [19:46:12] anyway, no harm in splitting them out [19:46:15] maybe [19:46:18] send? [19:46:20] sure. [19:46:36] mark, asher, faidon, tomasz, brion? [19:46:49] there's no labor [19:47:00] consuming 1/1000 stream with kafka atm is less labor than cronjob [19:47:08] its a line in a udp2log file, das all [19:47:24] the hadoop consumer is set up to consume topics based on regex [19:48:19] dschoon ^^ [19:48:28] that's what i said [19:48:36] that kafka is obviously the right answer [19:48:37] no, the recipeints [19:48:41] and doing it with cron is silly. [19:48:59] +Erik M, robla [19:49:49] does this conversation have to involve me now? [19:50:03] don't think so [19:50:11] let's try it first without robla and em [19:50:19] your call. [19:50:19] thanks [19:50:22] always loop them in later [19:50:24] ok [19:50:27] sending email [20:00:41] yeehaw, drdee, moved udp2log monitoring card to shipping [20:00:50] AWESOME! [20:01:07] more stuff here now [20:01:07] http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&tab=v&vn=udp2log [20:01:23] notice that oxygen's rx_queue is currently pretty full [20:01:24] http://ganglia.wikimedia.org/latest/graph_all_periods.php?hreg[]=locke%7Cemery%7Coxygen%7Cgadolinium&mreg[]=rx_queue&z=large>ype=line&title=rx_queue&aggregate=1&r=hour [20:01:55] which usually corresponds to udp2log drops [20:01:55] http://ganglia.wikimedia.org/latest/graph_all_periods.php?h=oxygen.wikimedia.org&c=Miscellaneous%20eqiad&m=drops&r=4hr [20:01:58] which do show up here [20:02:09] it looks like # drops are very low [20:02:15] which would keep packet loss average pretty low [20:02:21] yay for cool new stats! [20:02:37] with packet loss avg only, we'd never see this! [20:03:13] very nice! [20:03:30] you know, i think i'm going to put UDP OutDatagrams on that view too [20:03:41] only oxygen should show a large number for that [20:03:45] since it is multicast relay [20:03:52] but that will be a nice reminder of that fact [20:03:59] maybe SndBufErrors too [20:04:00] just in case :) [20:04:41] do we really need those? [20:05:04] they already are being collected [20:05:10] and might be relevant for oxygen [20:05:30] and, it'll be nice to see that stat if/when we move the relay anywhere else (e.g. gadolinum) [20:05:42] k [20:37:08] ping [20:38:40] does it make sense to put the sub-server level grouping (in gp.wmflabs.org/datafiles/gp/... the second 'gp') in the datasource path as well as the datafile path? [20:40:21] i'm a bit confused, because this url works: http://gp.wmflabs.org/data/datasources/gp/global_south.json?pretty=True [20:40:55] but the file is at /srv/gp.wmflabs.org/limn-data/datasources/global_south.json [20:49:14] yeah, i hate that part of the setup [20:49:35] it's really confusing, and it's an artifact of an ancient plan for multi-tenant support of different git repos [20:49:42] otherwise the git-root would bitch [20:51:27] k, so I shouldn't put the datasources under a special directory, just the datafiles, right? [20:51:56] btw, if my unaddressed ping was a bit mysterious, it's because i thought I was in a pm with milimetric, but this worked out [20:52:32] heh [20:52:38] i would've helped! [20:52:50] but i do not respond unless pinged [20:52:59] my ADHD doesn't let me [20:53:57] i also didn't really answer [20:54:02] because i didn't exactly understand :) [20:54:29] drdee: i added comments to your spreadsheet [20:54:42] ty [21:13:14] milimetric, updated card #92 (https://mingle.corp.wikimedia.org/projects/analytics/cards/92) the new version of the iOS mobile app has WikipediaMobile/3.3.1 prepended to the user agent string [21:13:38] WikipediaMobile/.*iPhone then [21:16:40] erosen: can you please have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/21 and tell me what the status is? [21:16:51] * erosen looking [21:18:11] drdee: sure [21:18:23] i'm still a little unclear about what should be in there [21:18:34] that makes two of us :) [21:18:35] that is, how did this card get created, and what is its intended scope? [21:18:47] those are exactly my questions :) [21:18:59] i can put in a brain dump of open and previously answered questions [21:19:05] hehe [21:19:08] i know you have been working on the maxmind geoip churn stuf [21:19:15] but i am not sure what's left to be done [21:19:20] maybe it's already finished [21:19:22] maybe not [21:19:24] i dunno :) [21:19:35] yeah, I suspect I should be involved in this card, I just don't know how it was created, or with what intention [21:20:11] so I'm guessing you or kraig created the card after having heard of ongoing work around maxmind [21:20:25] if it is just meant to be a place to collect thoughts around the general subject, that is fine [21:20:37] and I'll add some thoughts [21:21:05] k [21:21:05] basically i just want to make sure it isn't a really specific question in the guise of an ambiguous mingle title [21:21:27] feel free to sharpen the title [21:21:55] not saying it is a bad one, if it is really a general card [21:22:17] drdee: one last q: is there a standard mingle template you have been using [21:22:18] ? [21:24:08] with the new cards yes, not with the old cards like this one [21:24:13] i will do the formatting [21:24:16] no worries [21:28:02] k, thanks [21:29:22] drdee, kraigparkinson, is the absence of a hangout on the upcoming meeting on purpose? [21:29:25] if not, please fix [21:29:36] no, added it [21:29:44] just now [21:30:08] i thought we could practice our telepathy [22:33:44] ottomata, updated https://mingle.corp.wikimedia.org/projects/analytics/cards/510 please have a look regarding partman [22:47:07] drdee: could you add an item to that ticket for contacting me regarding the upgrade? i see you have "send email regarding pending planned down-time" there, but I might miss a generic e-mail, and I rely on Oxygen to relay EventLogging traffic from esams [22:48:10] 510 is regarding emery, but yes i will add you on 509 but that one is more tricky and requires more planning but thanks for letting me know [22:48:57] ori-l: done, https://mingle.corp.wikimedia.org/projects/analytics/cards/509 [22:49:19] drdee: thanks very much [22:49:25] you can subscribe to the card if you want [22:49:43] oh, neat, will do that too [22:57:04] milimetric: you about?