[09:51:38] Analytics / Refinery: Duplicates/missing logs from esams bits for 2014-09-28T{18,19,20}:xx:xx - https://bugzilla.wikimedia.org/71435#c4 (christian) It happened again for 2014-10-16T17:xx:xx up to and including 2014-10-16T19:xx:xx [10:06:10] (CR) QChris: [C: -2] Add UAParserUDF from kraken (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/166142 (owner: Ottomata) [10:26:38] (CR) QChris: Add IpUtil, Geocode, and GeocodeCountryUDF classes (1 comment) [analytics/refinery/source] (otto-geo) - https://gerrit.wikimedia.org/r/164264 (owner: Ottomata) [10:31:53] Analytics / Refinery: Raw webrequest partitions that were not marked successful - https://bugzilla.wikimedia.org/70085 (christian) [10:31:54] Analytics / Refinery: Raw webrequest partitions for 2014-10-20T02:xx:xx not marked successful - https://bugzilla.wikimedia.org/72252 (christian) NEW p:Unprio s:normal a:None For the hour 2014-10-20T02:xx:xx, none [1] of the the four sources' bucket was marked successful. What happened?... [10:38:38] !log Ran kafka leader re-election. See {{bug|72252}}. [10:39:37] Analytics / Refinery: Raw webrequest partitions for 2014-10-20T02:xx:xx not marked successful - https://bugzilla.wikimedia.org/72252#c1 (christian) It seems that somewhere between 2014-10-20T02:05:00 and 2014-10-20T02:12:00 analytics1021 again got kicked out of its partition leader role. I now ran lea... [11:01:54] Analytics / Refinery: Raw webrequest partitions for 2014-10-20T02:xx:xx not marked successful - https://bugzilla.wikimedia.org/72252#c2 (christian) From the logs between 2014-10-20T02:05:08 2014-10-20T02:05:16, data worth <2 seconds got lost. It's noteworthy that we again did not see loss for the host... [11:48:42] Analytics / Refinery: Raw webrequest partitions that were not marked successful - https://bugzilla.wikimedia.org/70085 (christian) [11:48:42] Analytics / General/Unknown: Kafka broker analytics1021 not receiving messages every now and then - https://bugzilla.wikimedia.org/69667 (christian) [11:48:43] Analytics / Refinery: Raw webrequest partitions for 2014-10-20T02:xx:xx not marked successful - https://bugzilla.wikimedia.org/72252#c3 (christian) NEW>RESO/WON (In reply to christian from comment #2) > So I think we should move forward to roll out the > ACK experiment to more hosts, so we can ge... [11:51:08] Analytics / Refinery: Raw webrequest partitions for 2014-10-20T02:xx:xx not marked successful - https://bugzilla.wikimedia.org/72252#c4 (christian) s/big 69667/bug 69667/ [12:29:54] Analytics / Refinery: Several raw webrequest partitions now marked successful between 2014-10-13T13:xx:xx and 2014-10-13T22:xx:xx - https://bugzilla.wikimedia.org/72028#c3 (christian) (Since it also is about analytics1021 dropping out of it's leader role, also blocking on bug 69667) [12:30:09] Analytics / General/Unknown: Kafka broker analytics1021 not receiving messages every now and then - https://bugzilla.wikimedia.org/69667 (christian) [13:02:04] (PS1) QChris: Drop graphs from dashboard that were removed in 2013 [analytics/global-dev/dashboard-data] - https://gerrit.wikimedia.org/r/167565 (https://bugzilla.wikimedia.org/72254) [13:04:42] Analytics / Visualization: Remove non-public graphs from gp.wmflabs.org dashboard - https://bugzilla.wikimedia.org/72254 (christian) NEW p:Unprio s:normal a:None After taking some geowiki data files offline in 2013, some graphs no longer show data; on purpose. Since the dashboard at htt... [13:06:44] (CR) Milimetric: [C: 2 V: 2] Drop graphs from dashboard that were removed in 2013 [analytics/global-dev/dashboard-data] - https://gerrit.wikimedia.org/r/167565 (https://bugzilla.wikimedia.org/72254) (owner: QChris) [13:06:57] That was fast :-) [13:06:58] Thanks! [13:09:53] Analytics / Visualization: Remove non-public graphs from gp.wmflabs.org dashboard - https://bugzilla.wikimedia.org/72254#c3 (christian) PATC>RESO/FIX I deployed it, and now http://gp.wmflabs.org/ does no longer show empty graphs. [13:31:38] Analytics / Wikimetrics: Story: VSUser has bots filtered out of all metrics - https://bugzilla.wikimedia.org/72134#c3 (Dan Andreescu) The bot filter has been defined by research as: "exclude any user found in the user_groups table with a ug_group of 'bot'". Not all metrics will be filtered: No bot fi... [13:31:38] Analytics / Wikimetrics: Story: WikimetricsUser monitors health of Wikimetrics in graphite - https://bugzilla.wikimedia.org/72138#c3 (christian) For most services, WMF mostly uses Icinga to monitor them. To me, it seems Icinga would also be a good fit for Wikimetrics too, as wikimetrics has many parts... [13:33:39] Analytics / Wikimetrics: Story: WikimetricsUser monitors health of Wikimetrics in graphite - https://bugzilla.wikimedia.org/72138#c4 (Dan Andreescu) The reason for graphite in the title of the bug is because of the new monitoring that's been made available in labs. We assumed it was easy to add more m... [13:35:55] grr, search for a decent IRC client continues [14:02:50] milimetric... rejoining .. agian [14:02:53] *again [14:21:21] milimetric, mforns : I am going to use staging for testing oK? [14:21:29] ok [14:21:30] all good [14:21:40] it's clean as of Friday nuria, puppet is updated and clean [14:21:47] and wikimetrics is on master and clean [14:22:01] the recurrent reports might be in the db still though, i don't remember [14:22:09] milimetric: ok, great that way i do not have to do hardly anything... [14:22:43] oh nuria / mforns I forgot to mention - in our testing the pages / edits metrics last week we found them to be really slow for english wikipedia [14:23:01] it might have been a fluke so i'd like to re-run them today [14:23:05] milimetric: makes sense cause they are counting alot more pages [14:23:06] (server busy etc) [14:23:13] but yea - exactly [14:23:18] milimetric: did you write teh benchmarks? [14:23:32] no, we didn't look at time, we were just looking at correctness [14:23:58] but basically, what I think this means, is that we need to optimize these metrics too when we optimize the others [14:24:28] yes [14:26:34] lemme know when yall want to talk interview tings [14:29:50] ottomata: I'll be gone in half an hour. Wanna sync up around the interviewing right now? [14:30:05] ottomata: since milimetric has not interviewed yet let's just talk the three of us [14:30:24] ottomata: sounds ok? [14:30:32] I am in the batcave. [14:31:58] qchris: leemme restart and rejoin cause i had a bunch of trouble this morning [14:34:16] ok ah sorry [14:34:21] getting some coffee, be there in 2mins [14:34:25] Analytics / Wikimetrics: Story: WikimetricsUser monitors health of Wikimetrics in graphite - https://bugzilla.wikimedia.org/72138#c5 (christian) So “graphite” is just a placeholder for $SOME_MONITORING_SERVICE? Then just to have it written down ... in labs it seems we're getting (or already having?) S... [15:07:49] (PS6) Nuria: Add UAParserUDF from kraken [analytics/refinery/source] - https://gerrit.wikimedia.org/r/166142 (owner: Ottomata) [15:08:55] (CR) Nuria: "Reworded README." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/166142 (owner: Ottomata) [15:16:11] (PS2) Milimetric: Filter out bots from relevant metrics [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167064 (https://bugzilla.wikimedia.org/72134) [15:16:13] (PS1) Milimetric: Improve performance by excluding 0 results [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167583 [15:17:23] mforns / nuria: I added you to my patch that implements bot filtering. Please review, and then there's a dependent patch that cleans up a silly mistake: https://gerrit.wikimedia.org/r/#/c/167583/ [15:17:45] yep [15:35:23] also, milimetric: can you take a look at this one: https://gerrit.wikimedia.org/r/#/c/167134/2 [15:43:31] (CR) Milimetric: [C: -1] Adding index on cohort_wiki_user (2 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167134 (https://bugzilla.wikimedia.org/71255) (owner: Nuria) [15:44:07] Analytics / Wikimetrics: Story: WikimetricsUser downloads large CSV - https://bugzilla.wikimedia.org/71255#c6 (nuria) Bug was reproducible running "pages created" with per-user results on the cohort attached to the bug. [15:48:37] Analytics / Wikimetrics: Story: WikimetricsUser downloads large CSV - https://bugzilla.wikimedia.org/71255#c7 (nuria) Code changes fix issues with performance, now we have to do some refactor as to see whether we can fit similar changes on json report. [15:48:54] (CR) Nuria: Adding index on cohort_wiki_user (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167134 (https://bugzilla.wikimedia.org/71255) (owner: Nuria) [16:08:00] (PS3) Nuria: Adding index on wiki_user [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167134 (https://bugzilla.wikimedia.org/71255) [16:08:13] (PS3) Nuria: [WIP] Improving retrieval of user names on cvs report [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167356 (https://bugzilla.wikimedia.org/71255) [16:26:06] (CR) Mforns: [C: 1 V: 1] "LGTM" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167064 (https://bugzilla.wikimedia.org/72134) (owner: Milimetric) [16:52:38] Analytics / Dashiki: Update metric classification in Vital Signs dashboard - https://bugzilla.wikimedia.org/70871 (Dan Andreescu) PATC>RESO/FIX [16:52:43] (CR) Mforns: [C: 1 V: 1] "LGTM" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167583 (owner: Milimetric) [17:13:48] milimetric, mforns sorry guys forgot to tell you i was walking down to the coffeee shop [17:14:04] no problem! [17:14:09] np at all [17:30:58] ottomata, do we still have that weird query admin thing working internally behind the firewall? [17:31:04] the one hive points you to when something errors out [17:31:46] hue? [17:31:49] oh [17:31:51] ? [17:31:55] hive points you to? [17:32:01] the analytics1010.eqiad.wmnet urls? [17:32:10] yes to both [17:33:02] yup [17:33:03] tq [17:33:04] *ta [17:47:32] milimetric: the rework of the code to retrieve the user for cvs reports fixes teh issue [17:48:06] milimetric: now, that code is duplicated in two places is report.py we talked about refactoring that... correct? [17:49:38] nuria: it's up to you, if you think we've got room to breathe in the sprint and it's high priority enough [17:49:57] as a scrum master, I'd say let's get this issue done, mark it resolved, and move on to the next high priority thing [17:50:16] but I'd also say you have a bit more time to play with it because you haven't quite used up 8 points yet. [17:50:24] milimetric: duplicating code (rather triplicating) to retrieve users? [17:50:49] milimetric: tears come to my eyes... lemme see if i can do a fast refactoring today [17:51:01] milimetric: i see no need for streaming yet though [17:53:19] ok nuria, that's cool, good that you looked into streaming either way [18:21:00] kevinator: please look at my comments about goals [18:28:20] nuria: I saw your comments and will reply on that thread [18:41:26] (CR) Milimetric: Adding index on wiki_user (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167134 (https://bugzilla.wikimedia.org/71255) (owner: Nuria) [18:48:25] (PS4) Nuria: Adding index on wiki_user [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167134 (https://bugzilla.wikimedia.org/71255) [18:48:57] (CR) Nuria: Adding index on wiki_user (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167134 (https://bugzilla.wikimedia.org/71255) (owner: Nuria) [19:17:39] (PS4) Nuria: [WIP] Improving retrieval of user names on cvs report [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167356 (https://bugzilla.wikimedia.org/71255) [20:19:29] (CR) Ottomata: [C: 2 V: 2] [webstatscollector] Add 'CentralAutoLogin' condition [analytics/metrics] - https://gerrit.wikimedia.org/r/99079 (owner: QChris) [20:20:00] (CR) Ottomata: [C: 2 V: 2] [webstatscollector] Add condition to not count redirects [analytics/metrics] - https://gerrit.wikimedia.org/r/165748 (https://bugzilla.wikimedia.org/71790) (owner: QChris) [20:20:11] (CR) Ottomata: [C: 2 V: 2] [webstatscollector] Adjust 'wiki' path check [analytics/metrics] - https://gerrit.wikimedia.org/r/99078 (owner: QChris) [20:20:20] (CR) Ottomata: [C: 2 V: 2] [webstatscollector] Add '[uU]ndefined' condition [analytics/metrics] - https://gerrit.wikimedia.org/r/165395 (https://bugzilla.wikimedia.org/66352) (owner: QChris) [20:21:19] (CR) Ottomata: [C: 1] Add overview diagram for Oozie jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/167185 (https://bugzilla.wikimedia.org/71994) (owner: QChris) [20:27:49] (Abandoned) Ottomata: Removing kraken-dclass [analytics/kraken] - https://gerrit.wikimedia.org/r/132108 (owner: Ottomata) [21:02:35] (CR) QChris: "@Ottomata: Yay for the CR+1." [analytics/refinery] - https://gerrit.wikimedia.org/r/167185 (https://bugzilla.wikimedia.org/71994) (owner: QChris) [21:04:43] qchris: i just +1ed because I didn't feel like figuring out how to run it myself righ tnow [21:04:46] feel free to self merge! [21:07:40] Analytics / Wikimetrics: Make mechanism to deactivate recurrent reports in wikimetrics - https://bugzilla.wikimedia.org/72279 (nuria) NEW p:Unprio s:normal a:None Make mechanism to deactivate recurrent reports in wikimetrics. There are reasons why we might want to deactivate recurrent repo... [21:14:08] ottomata, hi, are we still on for tomorrow coworking? [21:14:47] yes sir! [21:17:57] (CR) Milimetric: Adding index on wiki_user (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167134 (https://bugzilla.wikimedia.org/71255) (owner: Nuria) [21:18:03] ottomata: No worries. Since I do not like self-merges ... I'll just let it sit in gerrit for now. I'll bring it up later again. [21:18:12] haha [21:18:14] i will merge it! [21:18:31] (CR) Ottomata: [C: 2 V: 2] Add overview diagram for Oozie jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/167185 (https://bugzilla.wikimedia.org/71994) (owner: QChris) [21:18:33] THERE! [21:18:35] Hahaha. It can really ... [21:18:38] Tooo late :-) [21:18:42] OMG!!!!! [21:18:53] :-D [21:20:32] (CR) Nuria: Adding index on wiki_user (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167134 (https://bugzilla.wikimedia.org/71255) (owner: Nuria) [21:20:52] qchris: late late eh? [21:21:06] :-D [21:34:56] (PS5) Nuria: Improving retrieval of user names on cvs report [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167356 (https://bugzilla.wikimedia.org/71255) [21:54:55] Analytics / Wikimetrics: labsdb issues forced wikimetrics-scheduler to be stopped - https://bugzilla.wikimedia.org/72281 (Dan Andreescu) NEW p:Unprio s:normal a:None Currently the wikimetrics-scheduler process is stopped on the production wikimetrics instance, which means no new data is get... [22:17:43] (PS5) Nuria: Adding index on wiki_user [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167134 (https://bugzilla.wikimedia.org/71255) [22:17:49] (CR) jenkins-bot: [V: -1] Adding index on wiki_user [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167134 (https://bugzilla.wikimedia.org/71255) (owner: Nuria) [22:19:31] (PS6) Nuria: Adding index on wiki_user [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167134 (https://bugzilla.wikimedia.org/71255) [22:19:50] (PS6) Nuria: Improving retrieval of user names on cvs report [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167356 (https://bugzilla.wikimedia.org/71255) [22:27:37] Analytics / Wikimetrics: Story: WikimetricsUser downloads large CSV - https://bugzilla.wikimedia.org/71255#c8 (nuria) Verified in staging that cvs report for 2000 users (with per-user results) gets created in couple seconds. Tested a variety of reports with the cohort attched to this bug and all thos... [22:40:53] (CR) Nuria: [C: 2] Improve performance by excluding 0 results [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167583 (owner: Milimetric) [22:44:04] (CR) Nuria: [V: 2] Improve performance by excluding 0 results [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167583 (owner: Milimetric) [22:45:38] !log Marked webrequest partitions for 2014-10-20T02/1H good {{bug|72252}} [22:47:22] (CR) Nuria: "Change looks good. I think we should run metrics in staging for past reports (say september) and look at the difference between that data " [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167064 (https://bugzilla.wikimedia.org/72134) (owner: Milimetric) [23:02:23] Analytics / General/Unknown: Thanks tool: How much does the confirmation button cost us? - https://bugzilla.wikimedia.org/69804 (Quiddity)