[00:02:41] spagewmf, we shut stat1001 down, if I recall [00:02:49] but if you have 1003 access you can get in through bast1001. [00:03:58] Ironholds! long time no see. If so can you update wikitech? https://wikitech.wikimedia.org/wiki/Datasets.wikimedia.org says "The site is hosted on stat1001" [00:04:29] ooh, nope, apparently it's still live just a new machine. my bad. [00:11:57] Ironholds: Thanks. I don't have access to either {stat1001,stat1003}.wikimedia.org through bast1001, doesn't like my deploy key or my gerrit/labs key. [00:13:32] then ask ottomata in the morning, is my advice. [00:16:03] thanks will do (now playing: "Touch ottomata in the morning, then just walk away" by Diana Ross) [00:21:55] milimetric, stat1002 has rsync to public-datasets, right? Or wrong? [00:22:14] because, I threw some files/dirs over >10 hours ago, and nada. [00:24:01] Ironholds: I think stat1003 is the machine [00:24:03] one sec [00:24:13] and it shouldn't take longer than 30 min. [00:24:21] well, stat1002 has a public-datasets folder, and I'm not sure why it should if there's no rsyncing [00:24:34] i think that one syncs too... hm.... [00:24:35] and if we /don't/ have syncing on 1002, that's a massive blocker on...a lot of work, and we should set it up. [00:24:47] look at the /readership/ folder, or /enwiki/test.txt [00:25:08] they're in stat1002:/a/public-datsets/ - they ain't in http://datasets.wikimedia.org/public-datasets/ [00:26:05] Ironholds: yeah, they're not on stat1003, and now I'm forgetting how C & A set this up [00:26:24] there was some long debate about it... grr. [00:26:30] i'll try to read puppet if it's urgent [00:28:13] milimetric, I'm blocked on all the urgent mobile tasks until it's done. So if you could? Although I hate the idea of you staying up late for this :/ [00:28:33] (PS1) Milimetric: [WIP] Transform projectcounts hourly files [analytics/refinery] - https://gerrit.wikimedia.org/r/169974 [00:28:51] no problem, I've built up early-leaving. I'm not sure I can help though I'll try [00:29:55] okay! [00:29:57] thanks :) [00:30:04] (in that case I will be back in 5, making a run to the convenience store) [00:30:51] i'll just type out what I find as I find it: [00:31:09] stat1002 is not set up to rsync in the same place stat1003 is (in puppet) [00:32:53] kk [00:35:04] * milimetric is about to go into hulk smash mode [00:37:22] Ironholds: it looks to me as if /a/aggregate-datasets is the place to put stuff in on stat1002 [00:37:38] aha. I'll test. Thanks! [00:37:40] and /a/public-datasets is the place to put stuff on stat1003 [00:37:43] I'm testing Ironholds [00:37:49] but /a/public-datasets is synced to stat1002 [00:37:51] this is profoundly silly [00:37:57] grr, Ops of Christmas Past! [00:38:04] okay! Lemme know what you find? [00:38:27] yeah, the comments are all out of date on the puppet definition [00:39:47] i *really* hate that something called aggregate datasets could be made public without any notice in there. Like - at least put a README there, goodness [00:41:52] Ironholds: http://datasets.wikimedia.org/aggregate-datasets/ should get a little test file soon, but I re-read the puppet and I'm fairly confident that this will work. I'd dump what you have on there and I'll leave myself a message here to ping the opsy folks tomorrow [00:42:17] milimetric, cool; thanks! [06:05:32] Analytics / Wikimetrics: Story: Wikimetrics uses some EL data - https://bugzilla.wikimedia.org/72735 (Kevin Leduc) NEW p:Unprio s:enhanc a:None Move some data in EL to LabsDBs or Data Warehouse so Wikimetrics can use it. The driving use case is generating target-site breakdowns for Vital Si... [06:06:57] Analytics / Wikimetrics: Story: Wikimetrics uses some EL data - https://bugzilla.wikimedia.org/72735#c1 (Kevin Leduc) p:Unprio>High Collaborative tasking on etherpad: http://etherpad.wikimedia.org/p/analytics-72735 [06:07:27] Analytics / Wikimetrics: Story: Wikimetrics uses some EL data - https://bugzilla.wikimedia.org/72735 (Kevin Leduc) [06:20:14] Analytics / Wikimetrics: Story: Wikimetrics compiles target-site breakdown on metrics based on MW tags - https://bugzilla.wikimedia.org/72736#c1 (Kevin Leduc) p:Unprio>High Implement breakdowns for existing Vital Signs metrics where we can use Mediawiki tags (mobile, mobile-app or other [desktop]). [06:20:27] Analytics / Wikimetrics: Story: Wikimetrics compiles target-site breakdown on metrics based on MW tags - https://bugzilla.wikimedia.org/72736#c2 (Kevin Leduc) collaborative etherpad at http://etherpad.wikimedia.org/p/analytics-72736 [06:24:29] Analytics / Wikimetrics: Story: Wikimetrics has connection to Data Warehouse - https://bugzilla.wikimedia.org/72737 (Kevin Leduc) NEW p:Unprio s:enhanc a:None - create new wikimetrics connections to warehouse - puppet changes, vagrant setup, localhost setup [06:25:42] Analytics / Wikimetrics: Story: Wikimetrics has connection to Data Warehouse - https://bugzilla.wikimedia.org/72737#c1 (Kevin Leduc) p:Unprio>High collaborative tasking on etherpad: http://etherpad.wikimedia.org/p/analytics-72737 [06:31:31] Analytics / Wikimetrics: Story: Wikimetrics compiles target-site breakdown for remaining metrics - https://bugzilla.wikimedia.org/72738 (Kevin Leduc) NEW p:Unprio s:enhanc a:None Daily report of target site breakdown for remaining existing metrics in Vital Signs. [06:32:59] Analytics / Wikimetrics: Story: Wikimetrics has connection to Data Warehouse - https://bugzilla.wikimedia.org/72737 (Kevin Leduc) [06:32:59] Analytics / Wikimetrics: Story: Wikimetrics compiles target-site breakdown for remaining metrics - https://bugzilla.wikimedia.org/72738#c1 (Kevin Leduc) p:Unprio>High Collaborative tasking on etherpad: http://etherpad.wikimedia.org/p/analytics-72738 [06:36:59] Analytics / Dashiki: Story: User selects breakdown in Vital Signs - https://bugzilla.wikimedia.org/72739 (Kevin Leduc) NEW p:Unprio s:enhanc a:None According to Pau's design: http://pauginer.github.io/prototypes/analytics-dashboard/index.html Implement a 'button' in the left nav bar to dis... [06:37:58] Analytics / Dashiki: Story: User selects breakdown in Vital Signs - https://bugzilla.wikimedia.org/72739 (Kevin Leduc) [06:38:41] Analytics / Dashiki: Story: User selects breakdown in Vital Signs - https://bugzilla.wikimedia.org/72739#c1 (Kevin Leduc) p:Unprio>High Collaborative tasking on etherpad: http://etherpad.wikimedia.org/p/analytics-72739 [06:44:45] Analytics / Dashiki: Story: Vital Signs User selects the Daily or Monthly Pageviews metrics - https://bugzilla.wikimedia.org/72740 (Kevin Leduc) NEW p:Unprio s:enhanc a:None Generate data on the cluster, move it on labsdbs for wikimetrics or make it directly available to Vital Signs There... [06:46:57] Analytics / Dashiki: Story: Vital Signs User selects the Daily or Monthly Pageviews metrics - https://bugzilla.wikimedia.org/72740#c1 (Kevin Leduc) p:Unprio>High Collaborative tasking on etherpad: http://etherpad.wikimedia.org/p/analytics-72740 [06:55:15] Analytics / EventLogging: List tables/schemas with data retention needs - https://bugzilla.wikimedia.org/72741 (Kevin Leduc) NEW p:Unprio s:enhanc a:None Do the ground work to identify - with product's help - tables/schemas, what they are used for and what sort of data they want to keep bey... [06:55:42] Analytics / EventLogging: List tables/schemas with data retention needs - https://bugzilla.wikimedia.org/72741 (Kevin Leduc) p:Unprio>Highes [06:55:43] Analytics / EventLogging: List tables/schemas with data retention needs - https://bugzilla.wikimedia.org/72741 (Kevin Leduc) a:Kevin Leduc [06:56:56] Analytics / EventLogging: List tables/schemas with data retention needs - https://bugzilla.wikimedia.org/72741#c1 (Kevin Leduc) - Aaron can help reviews schemas, he could show which tables contain sensitive data - Talk to individual teams to identify what they need [06:58:43] Analytics / EventLogging: Automate purge of raw logs older than 90 days - https://bugzilla.wikimedia.org/72742 (Kevin Leduc) NEW p:Unprio s:enhanc a:None - ops task - set up log polling mechanism with deletion (pupet change) [06:58:56] Analytics / EventLogging: Automate purge of raw logs older than 90 days - https://bugzilla.wikimedia.org/72742 (Kevin Leduc) p:Unprio>High [07:05:59] Analytics / EventLogging: Automate pruning of sampled logs after 90 days - https://bugzilla.wikimedia.org/72743 (Kevin Leduc) NEW p:Unprio s:enhanc a:None - similar to pruning raw logs after 90 days. Needs a cron running on stat1002, Oxygen and wherever else the data exists. [07:06:27] Analytics / EventLogging: Automate pruning of sampled logs after 90 days - https://bugzilla.wikimedia.org/72743#c1 (Kevin Leduc) Reposting a comment Christian made in a google doc: "It will probably be a puppet change too, but it might be a different one. Like cron vs. logrotate. I am not sure about a... [07:06:41] Analytics / EventLogging: Automate pruning of sampled logs after 90 days - https://bugzilla.wikimedia.org/72743 (Kevin Leduc) p:Unprio>High [07:11:14] Analytics / EventLogging: Automate purge of rows older than 90 days for select tables/schemas - https://bugzilla.wikimedia.org/72744 (Kevin Leduc) NEW p:Unprio s:enhanc a:None - don't turn it on, just make it available per schema. This is already done PER Table by sean, not per column - Thi... [07:11:28] Analytics / EventLogging: Automate purge of rows older than 90 days for select tables/schemas - https://bugzilla.wikimedia.org/72744 (Kevin Leduc) p:Unprio>High [07:14:12] Analytics / EventLogging: Story: User clicks on link to event capsule schema while viewing a schema - https://bugzilla.wikimedia.org/72745#c1 (Kevin Leduc) p:Unprio>Normal s:normal>enhanc On schema pages (for example) https://meta.wikimedia.org/wiki/Schema:NewEditorEdit Add a link to the ev... [07:14:56] Analytics / EventLogging: Story: User clicks on link to event capsule schema while viewing a schema - https://bugzilla.wikimedia.org/72745#c2 (Kevin Leduc) Collaborative tasking on etherpad: http://etherpad.wikimedia.org/p/analytics-72745 [07:37:14] Analytics / Wikimetrics: Story: WikimetricsUser tags a cohort using a pre-defined tag - https://bugzilla.wikimedia.org/72746 (Kevin Leduc) NEW p:Unprio s:enhanc a:None The story is here https://www.mediawiki.org/wiki/Analytics/Wikimetrics/Stories#Tag_your_own_cohort_with_existing_tags tran... [07:37:43] Analytics / Wikimetrics: Story: WikimetricsUser tags a cohort using a pre-defined tag - https://bugzilla.wikimedia.org/72746 (Kevin Leduc) p:Unprio>Highes [07:41:44] Analytics / Wikimetrics: Story: WikimetricsUser reads user names in a JSON report - https://bugzilla.wikimedia.org/72747 (Kevin Leduc) NEW p:Unprio s:enhanc a:None Story is here: https://www.mediawiki.org/wiki/Analytics/Wikimetrics/Stories#List_usernames_in_reports_that_include_individual_r... [07:42:13] Analytics / Wikimetrics: Story: WikimetricsUser reads user names in a JSON report - https://bugzilla.wikimedia.org/72747 (Kevin Leduc) p:Unprio>Highes [10:09:36] Analytics / Refinery: Raw webrequest upload partition for 2014-10-29T14/1H not marked successful - https://bugzilla.wikimedia.org/72756 (christian) NEW p:Unprio s:normal a:None The upload webrequest partition [1] for 2014-10-29T14/1H has not been marked successful. What happened? [1] ___... [10:09:53] Analytics / Refinery: Raw webrequest upload partition for 2014-10-29T14/1H not marked successful - https://bugzilla.wikimedia.org/72756#c1 (christian) NEW>RESO/FIX Commit 37228258e8680aa035206e8b89eaa9f57b28555f got merged, which updated the varnishkafka configuration for the upload caches. This... [10:10:06] Analytics / Refinery: Raw webrequest partitions that were not marked successful due to configuration updates - https://bugzilla.wikimedia.org/72300 (christian) [10:12:06] !log Marked raw upload webrequest partition for 2014-10-29T14/1H ok (See {{bug|72756}}) [13:58:27] Analytics / EventLogging: Add test flag to EventLogging - https://bugzilla.wikimedia.org/72365#c6 (christian) NEW>RESO/WON (In reply to nuria from comment #5) > [...] I see little value in adding a test flag > and I am of the opinion that we should not do it. Same here. (Hence, being bold and R... [14:02:22] haha [14:33:31] https://docs.google.com/a/wikimedia.org/spreadsheets/d/1UEnlRIRKKGBhQluyWUiCiwyn2pX9DSzK8kxfBw3kdDk/edit#gid=1717503637 [14:38:43] Analytics / Wikimetrics: report table performance, cleanup, and number of items - https://bugzilla.wikimedia.org/72635#c1 (nuria) There are several ways to go about this: #1. Purge from db anything older than 30 days that is not a recurrent reports. This can be done via a scheduler task #2 do not wri... [14:41:41] Analytics / Wikimetrics: Story: WikimetricsUser tags a cohort using a pre-defined tag - https://bugzilla.wikimedia.org/72746 (Kevin Leduc) [14:45:12] Analytics / EventLogging: Automate pruning of sampled logs after 90 days - https://bugzilla.wikimedia.org/72743#c2 (christian) (In reply to Kevin Leduc from comment #1) > Reposting a comment Christian made in a google doc: > [...] > * gadolinium (might be a on-time thing there), and s/on-time/one-time/ [14:47:11] Analytics / Wikimetrics: Story: WikimetricsUser reads user names in a JSON report - https://bugzilla.wikimedia.org/72747 (Kevin Leduc) [14:50:56] Analytics / Wikimetrics: report table performance, cleanup, and number of items - https://bugzilla.wikimedia.org/72635 (Kevin Leduc) [14:51:41] Analytics / Wikimetrics: report table performance, cleanup, and number of items - https://bugzilla.wikimedia.org/72635#c2 (nuria) We estimated #2, please have in mind recurrent reports need to be working as they are today. [15:22:21] nuria__: hey! we've statsd running on labmon1001.eqiad.wmnet and you can send metrics to that from your application using any statsd client library [15:26:00] YuviPanda: can you write a little 1 pager on how to get the statsd client on your lab instance and a dummy "hello world" example? When I send data to graphite in prod i found testing it was not that easy. [15:26:48] ah, hmm. usually you just use a library (like https://pypi.python.org/pypi/python-statsd) and specify the host [15:28:45] YuviPanda: but you can 1) do it directly (pip statsd) 2) have puppet install statsd that for you to be ready [15:29:18] nuria__: statsd client library, you mean? [15:29:59] YuviPanda: yes, also it will be nice to have the graphite endpoint configured on puppet ( asking here like it's x-mas...) [15:30:12] nuria__: I'm somewhat confused now... [15:30:20] nuria__: what do you mean by 'graphite endpoint configured on puppet'? [15:30:34] nuria__: if that is 'send to different hosts based on labs or prod' that is already there... [15:31:33] YuviPanda: maybe graphite endpoint is not a good description, rather the node endpoint that listens for your counters [15:32:01] YuviPanda: that came out not so clear either..ahem...let me see if i can find the example [15:32:08] in the puppet files in prod [15:34:24] YuviPanda: not sure if something like this exists in puppet for labs already [15:34:57] https://www.irccloud.com/pastebin/4EgtEcGM [15:35:31] "my=instance" should be myinstance [15:59:04] qchris: ok [15:59:07] so, to understand [15:59:19] in general, we miss kafkatee lines near the beginning of hours. [15:59:20] right? [15:59:23] right [15:59:27] first few seconds of each hour [15:59:36] and it doesn't matter the host or datacenter [15:59:38] I mean ... I did not look elsewhere [15:59:47] so we at least miss it there. [16:00:02] right, and, as far as we can tell those lines are in hive [16:00:05] so they are in kafka [16:00:10] Right, it doesn't matter the host or datacenter [16:00:15] which, would make sense if it happens to all hosts at the same time [16:00:23] sounds like either kafkatee or something on analytics1003 then [16:00:37] the only thing that I know that happens hourly on analytics1003 is webstastcollector [16:01:21] Does kafka not guarantee to produce to the consumer? [16:01:38] the consumer is responsible for consuming [16:01:52] if the messages are in kafka (which we are sure they are), then it has to be downstream from kafka [16:01:54] I'd count that as a kafkatee issue then. [16:01:56] yes [16:02:21] i somehow doubt that collector is the problem [16:02:24] but, shall I stop it anyway? [16:02:26] just in case? [16:02:33] Would be a nice way too test. [16:02:38] ok [16:03:04] It does not look implausible ... after all ... a few hundred MB are written in short time. [16:03:13] milimetric, nuria, kevinator: trying to reconnect [16:07:36] qchris: but it is not all hours that miss data during those first few seconds [16:07:37] right? [16:07:46] jsut, usually, if data is missing, that is where it is missing from [16:08:02] I did not check that. Instead, I checked if for some second [16:08:23] (where udp2log files have messages), there are no messages in the kafkatee files [16:08:33] i am reading the manpage for ts for the first time.....:o [16:08:44] And for those seconds, I checked if there are missing timestamps. [16:08:46] :-P [16:09:13] that inline function thingie is quite nice to fight DRY [16:09:30] ja, what is <( )??? [16:09:34] that is new to me too! [16:09:39] bash redirection. [16:09:43] from a subshell? [16:09:47] Yes. [16:09:54] The thing within <( ... ) [16:09:55] diff will take two streams of stdin? [16:10:10] i guess not stdin. [16:10:16] gets written to a tmp file, and the name of that tmp file gets substituted there. [16:10:22] does it open towo fds? [16:10:23] whoa. [16:10:27] yen. [16:10:35] s/yen/yes, 2 fds/ [16:10:42] your foo is great. [16:10:46] :-D [16:11:08] OH [16:11:10] bash grew a lot the last years ... so I never got around to learn zsh. [16:11:10] hahah [16:11:23] i somehow missed the ts() func declaration [16:11:23] Now I am a lamer, as I still use bash :-( [16:11:29] i was reading [16:11:30] man ts [16:11:33] and all like [16:11:38] what is going on here!? [16:11:43] ts - Time Stamping Authority tool (client/server) [16:11:49] whoa :-D [16:11:54] I did not know that existed. [16:12:03] me neither [16:12:27] During prototyping I just use "X" as function name ... but I figured that is less descriptive for timestamps. [16:12:51] aye [16:12:53] I thought you were kidding before when you said you're just reading "man ts" [16:14:28] ok, webstats not running on an03 anymore [16:14:42] Great! [16:16:54] ha, qchris, it would be nice if we had the kafka partition...and maybe even the kafkaoffset, of each message in the json [16:16:55] :) [16:17:22] hm, guess we can't know the offset...can we? dunno. [16:17:33] yes, that would help I guess. [16:18:09] partitoin would be nice, because then we could immediately see if there are problems with certain partitions [16:18:15] that would be easy to add, I think [16:18:17] maybe... [16:18:17] hm [16:18:24] we use a random partitioner, so, not sure [16:18:30] might have to be an rdkafka feature? [16:18:32] duno. [16:18:34] will ask snaps sometime. [16:18:52] It would certainly help debugging things. [16:19:16] But wouldn't that mean that kafka would have to mangle the message? [16:19:35] no, the partition is selected by the producer [16:19:38] Or would camus/kafkatee add that information on the fly? [16:19:40] Oh. [16:19:47] varnishkafka then? [16:20:12] Mhmmm. [16:21:32] Varnishkafka just passes it to librdkafka. [16:21:56] yes [16:21:59] So varnishkafka does not know what partition it sends to. [16:22:02] either varnishkafka or librdkakfa woudl do it [16:22:11] yeah, varnishkafka would have to know by librdkafka, i suppose [16:22:16] dunno, it probably has a way to select or know [16:22:17] who knows [16:22:18] but ja [16:22:32] camus probably could add that on the fly [16:22:37] if we worked hard enough to make it do so :) [16:22:43] Right. [16:23:17] Hahahaha ... but I guess there are bigger fish to fry :-( [16:28:06] so, hm, are there more things we can troubleshoot with this, or do we wait another 24 hours and then check again? [16:29:56] IIRC, you checked field #1 in the tsvs (hostname), which showed issues with field #3 (timestamps) [16:30:05] All the other fields are not yet vetted. [16:30:30] But those will just suffer the same missing lines. [16:31:04] yeah, if we have one issue we'll see the others [16:31:17] I guess it's fine to wait over the weekend. [16:31:26] Then there will be full, good tsvs. [16:31:42] well, this seems to happen enough times in 24 hours [16:31:44] I guess one could look at the sampled ones (if you have time) [16:31:45] according to your email [16:32:45] Yes, it would happen often enough. But the tsvs only get available at 6:30. [16:32:58] Especially the ones from the production udp2log instances. [16:33:07] aye, so tomorrow? [16:33:35] Tomorrow, there'll only be a file that is good after 16:00. [16:33:50] On Saturday, we'd have a full good file (hopefully). [16:33:58] that is nowish! [16:34:00] tomorrow [16:34:04] haha [16:34:09] tomorrow it is :-D [16:34:53] I mean ... we could look at the file on analytics1003 in one hour. If it shows the holes [16:35:19] for 17:00:10 ... then turning off webstatscollector did not help :-) [16:36:02] oh, true. ah, but if it does not show the holes then we have to wait [16:36:05] yeah, let's check [16:36:08] if we see holes, we keep thikning [16:36:50] k [17:10:41] Analytics / EventLogging: List tables/schemas with data retention needs - https://bugzilla.wikimedia.org/72741 (Dan Andreescu) [17:11:43] Analytics / Wikimetrics: report table performance, cleanup, and number of items - https://bugzilla.wikimedia.org/72635 (Dan Andreescu) [17:11:43] Analytics / Wikimetrics: Story: WikimetricsUser tags a cohort using a pre-defined tag - https://bugzilla.wikimedia.org/72746 (Dan Andreescu) [17:11:56] Analytics / EventLogging: database consumer could batch inserts (sometimes) - https://bugzilla.wikimedia.org/67450 (Dan Andreescu) [17:12:42] Analytics / EventLogging: Automate purge of raw logs older than 90 days - https://bugzilla.wikimedia.org/72742#c1 (nuria) NEW>RESO/DUP *** This bug has been marked as a duplicate of bug 72642 *** [17:12:42] Analytics / EventLogging: Story: Identify and direct the purging of Event logging raw logs older than 90 days in stat1002 - https://bugzilla.wikimedia.org/72642#c2 (nuria) *** Bug 72742 has been marked as a duplicate of this bug. *** [17:17:57] Analytics / EventLogging: Story: Identify and direct the purging of Event logging raw logs older than 90 days in stat1002 - https://bugzilla.wikimedia.org/72642 (Kevin Leduc) p:Unprio>High s:normal>enhanc [17:21:12] Analytics / Dashiki: Story: Vital Signs User selects the Daily or Monthly Pageviews metrics - https://bugzilla.wikimedia.org/72740 (Dan Andreescu) [17:23:59] Analytics / EventLogging: Create staging environment - https://bugzilla.wikimedia.org/72767 (Toby Negrin) NEW p:Unprio s:normal a:None Currently we don't have an environment we can test event logging backend changes. This is risky and non-optimal. [17:40:33] nuria, milimetric: hey guys, we are deploying in 20 mins, right? [17:40:46] or you already started? [17:43:49] YuviPanda: yt? [17:43:56] I had a dream [17:44:14] and you were in it, and quarry too [17:45:12] and the dream said we should have a private instance of quarry for internal use (with access to EventLogging data) [17:46:12] this could be used as a shared sandbox for people to play with the schemas they own, as a way of prototyping Vital Signs queries but also as an easy way to share queries within the org [17:48:55] DarTar, you have weird dreams [17:49:12] Ironholds: I know, I’m working on that [17:50:31] Ironholds: my brain is trying to tell me how to make myself (and my team) less and less involved in ad-hoc data requests [17:51:16] DarTar, now if we can just start having dreams about not giving me engineering projects ;p [17:51:45] I’ll add that to the dream queue for tonight [17:51:53] ta [17:53:10] mforns: I was about to deploy right now [17:53:30] I'm going to hop in the hangout to do it - cc: nuria__ [17:56:35] milimetric: be there [17:57:15] ok [18:00:19] DarTar: heh :) [18:00:30] DarTar: it's easy to set up, needs a machine (from toby) [18:00:38] hi toby [18:01:05] oops, he left his desk :-/ [18:01:29] heh [18:02:39] DarTar: yo [18:02:55] do we have a box for YuviPanda ? [18:03:13] we have a meeting [18:03:18] ah [18:03:20] right [18:03:22] coming [18:03:47] nuria__: sorry had gone away. I've no idea what https://www.irccloud.com/pastebin/4EgtEcGM means :| [18:04:13] Analytics / Refinery: Spike: Assess feasibility and effort to add fields to webrequest logs - https://bugzilla.wikimedia.org/72651#c3 (ewulczyn) Another thing that came up in our research group meeting today is to add the browser session cookie. I added this to the etherpad. [18:07:39] ottomata: can you merge https://gerrit.wikimedia.org/r/170103 [18:21:12] mforns / nuria__: he merged it, but now my burger's getting cold - let's reconvene in 20 min? [18:21:17] thanks ottomata ! [18:21:19] hehe [18:21:29] of course [18:33:02] yup! [18:33:45] milimetric, mforns: have a meeting in 10 mins [18:33:52] will join right after [18:33:55] ok [18:48:17] mforns: [18:48:24] let's do it now? [18:48:25] yep [18:48:32] im in the batcave [18:54:43] Analytics / Refinery: Raw webrequest bits partition for 2014-10-26T21/1H not marked successful - https://bugzilla.wikimedia.org/72548#c4 (christian) NEW>RESO/FIX ottomata had a look at the logs on cp3019 and said that there were produce errors about full buffers. So we're writing it off as tempor... [18:54:57] Analytics / Refinery: Raw webrequest partitions that were not marked successful due to network issues - https://bugzilla.wikimedia.org/72298 (christian) [18:54:58] Analytics / Refinery: Raw webrequest partitions that were not marked successful - https://bugzilla.wikimedia.org/70085 (christian) [19:18:20] milimetric: are you done deploying? [19:19:02] cc mforns [19:19:49] nuria__: yep [19:20:30] milimetric; did you deleted pages created and start fresh? [19:20:48] nuria__: so I was going to wait to hear from kevinator about that [19:20:52] but he's not on here [19:21:00] 'cause i want to sync up with his announcement [19:21:34] ok, if you did not sent an e-mail i can do that [19:21:40] to kevinator i mean [19:21:57] cause leaving teh other metrics with bots data i do not think is a big deal, [19:22:06] but the "pages created" changed in nature [19:23:25] qchris_away: yt? [19:23:26] no you arte away!@ [19:25:52] ottomata: i don't know if you've seen osquery yet, but I was just playing with it and I'm having fun doing opsy things... so that says a lot: https://github.com/facebook/osquery/wiki/using-osqueryi [19:26:35] nuria__: yea, i was just going to wait for him - he's probably just at lunch [19:27:02] whoa [19:27:30] that's awesome [19:39:02] nuria__, milimetric: I was about to send an email to kevinator, but if you want to tell him about pages created fresh start.. [19:41:11] no rush mforns / nuria__ we can wait 'till he's back [19:41:22] the recurrent stuff runs at 3 am, we got time [19:41:36] ok [19:48:14] milimetric, mforns,ottomata taking long lunch (spanish style) , will be back in couple hours [19:49:26] enjoy! [19:53:25] qchris_away: fyi, i see about 175 more lines in kafkatee for the 10-30T17 hour for the zero logs, [19:53:27] good sign [19:53:51] kafkatee has all logs that udp2log has for that hour for zero logs [19:59:49] Now ottomata is gone :-/ [20:00:33] nuria__: not sure if my email made it to your internal mailing list [20:10:23] tnegrin: not sure if my email made it to your internal mailing list [20:11:18] YuviPanda: /me checks email ... [20:12:20] YuviPanda: I received three recent emails from you via the internal list. [20:12:33] So it seems they made it to the internal list. [20:12:57] qchris: ah, cool :) [20:18:07] (CR) Milimetric: [WIP] Add schema for edit fact table (1 comment) [analytics/data-warehouse] - https://gerrit.wikimedia.org/r/167839 (owner: QChris) [20:19:32] (CR) Milimetric: [WIP] Add schema for edit fact table (1 comment) [analytics/data-warehouse] - https://gerrit.wikimedia.org/r/167839 (owner: QChris) [20:47:40] (PS1) Mforns: Avoid exception accessing unknown project database [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/170152 (https://bugzilla.wikimedia.org/72582) [20:53:57] Analytics / Wikimetrics: report table performance, cleanup, and number of items - https://bugzilla.wikimedia.org/72635 (Marcel Ruiz Forns) NEW>ASSI a:Marcel Ruiz Forns [20:56:11] kevinator: question for you [20:56:18] hi [20:56:25] we deployed [20:56:32] mforns was going to ping you [20:56:45] but then we realized we didn't fully talk through how we were going to re-generate reports [20:56:51] or maybe we did and I forgot the conclusion [20:57:02] so we have changes for pages / edits (namespace 0 is no longer the default) [20:57:07] kevinator: I wrote to you in private [20:57:17] and changes for the "rolling" metrics (bots are filtered out) [20:57:25] for both of those we could: [20:57:32] * delete all old reports and regenerate [20:57:41] * just let the system start using the new definitions as of today [20:57:55] when making that decision, note: we will regenerate everything most likely once we have the DW [20:58:54] Let’s keep data for the time being… and watch if we see a noticable step on the dashboard. [20:59:17] Once we get the DW we can re-generate everything [21:00:30] Should Vital Signs get a mailing list for these kind of announcements: new metric definition, changes in data… [21:01:18] If I hear people getting confused about the data, we can regenerate everything sooner. [21:03:04] ok [21:17:41] kevinator: that sounds like a good plan. cc: nuria__ ^ few lines above from kevin [21:17:49] basically: no regeneration right now [21:18:30] i mean, everyone will be confused if they try to look at the data [21:18:40] but as far as I know nobody's using this as their primary data source yet [21:31:57] Analytics / Wikimetrics: Apache's logs containing "client denied by server configuration: /srv/wikimetrics/wikimetrics/api.wsgi" - https://bugzilla.wikimedia.org/71606#c9 (Dan Andreescu) PATC>RESO/FIX deployment train picked this up choo choo! [21:33:30] :-) [23:02:58] kevinator: so we are on the same page we shall get a Jump on the dashboard for sure [23:03:19] kevinator: for bots, numbers will go down [23:03:34] kevinator: for pages created we are counting all pages thus they will go up [23:04:08] nuria__, milimetric: I was just thinking the ideal way to let people know why there’s a jump: annotations ;-) [23:04:41] kevinator: right, until then I think we probably need a public log [23:46:00] (CR) Nuria: Avoid exception accessing unknown project database (2 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/170152 (https://bugzilla.wikimedia.org/72582) (owner: Mforns)