[00:05:56] Analytics-EventLogging, Hovercards: Large number of popup events not validating - https://phabricator.wikimedia.org/T91272#1123914 (Nuria) We have changed the schema in significant ways so these validation issues might no longer appear. [00:07:21] Analytics-EventLogging, Hovercards: Large number of popup events not validating - https://phabricator.wikimedia.org/T91272#1123916 (Nuria) New schema events have to be tested using teh dev server in vagrant:https://www.mediawiki.org/wiki/Extension:EventLogging/Guide#Installing_the_EventLogging_devserver [00:23:29] Analytics-EventLogging, Analytics-Kanban: Upgrade box for EventLogging (vanadium) - https://phabricator.wikimedia.org/T90363#1123956 (kevinator) [00:35:51] (PS2) Milimetric: [WIP] Add Sunburst Visualizer [analytics/dashiki] - https://gerrit.wikimedia.org/r/197234 [00:42:02] (PS3) Milimetric: [WIP] Add Sunburst Visualizer [analytics/dashiki] - https://gerrit.wikimedia.org/r/197234 [03:07:05] (PS4) Milimetric: [WIP] Add Sunburst Visualizer [analytics/dashiki] - https://gerrit.wikimedia.org/r/197234 [03:26:16] (PS5) Milimetric: [WIP] Add Sunburst Visualizer [analytics/dashiki] - https://gerrit.wikimedia.org/r/197234 [03:33:50] Analytics-Tech-community-metrics, Possible-Tech-Projects, ECT-March-2015, Epic, and 2 others: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1124140 (NiharikaKohli) [07:32:44] Analytics-Tech-community-metrics, Engineering-Community, WMF-Product-Strategy: Review Engineering Community analytics needs - https://phabricator.wikimedia.org/T92807#1124322 (Ironholds) p:Triage>Low Marking as low (not because it's not important! It is important. But it's been rated < pageviews... [09:50:45] (PS1) OliverKeyes: De-static-everything [analytics/refinery/source] - https://gerrit.wikimedia.org/r/197296 [10:51:40] Analytics-Tech-community-metrics, ECT-March-2015, Patch-For-Review: "Age of unreviewed changesets by affiliation" shows negative number of changesets - https://phabricator.wikimedia.org/T72600#1124747 (Dicortazar) Those are in February indeed. Do you mind double checking this? In addition, there's als... [10:53:22] Analytics-Tech-community-metrics, ECT-March-2015: KPI pages in korma need horizontal margins - https://phabricator.wikimedia.org/T88670#1124749 (Dicortazar) The horizontal scrollbar should have disappeared. [11:35:08] Analytics-Tech-community-metrics, ECT-March-2015, Patch-For-Review: "Age of unreviewed changesets by affiliation" shows negative number of changesets - https://phabricator.wikimedia.org/T72600#1124857 (Qgil) >>! In T72600#1124747, @Dicortazar wrote: > Those are in February indeed. In my browsers the g... [11:37:04] Analytics-Tech-community-metrics, ECT-March-2015: KPI pages in korma need horizontal margins - https://phabricator.wikimedia.org/T88670#1124859 (Qgil) Confirmed. Labels can be still cut. Actually, this morning I had to rely on Ctrl-- to reduce the page and see a data point that I needed in a discussion. [12:45:11] Analytics-Tech-community-metrics, ECT-March-2015: Migrate Korma functionality into upstream - https://phabricator.wikimedia.org/T92952#1124976 (Dicortazar) NEW a:Acs [12:50:23] Analytics-Tech-community-metrics: Migrate Korma identitites database to SortingHat - https://phabricator.wikimedia.org/T92953#1124994 (Dicortazar) NEW a:Acs [12:51:57] Analytics-Tech-community-metrics, ECT-March-2015: Migrate Korma identitites database to SortingHat - https://phabricator.wikimedia.org/T92953#1125011 (Dicortazar) [12:55:46] Analytics-Tech-community-metrics, ECT-March-2015: KPI pages in korma need horizontal margins - https://phabricator.wikimedia.org/T88670#1125027 (Dicortazar) Ok, you're right, we still have to check the labels cut, sorry about that. [12:58:19] Analytics-Tech-community-metrics, ECT-March-2015: Instructions to update user data in korma - https://phabricator.wikimedia.org/T88277#1125031 (Dicortazar) I'd like to close the {T92953} task before publishing this data. SortingHat already exports data in a specific format that we call can work on. That... [13:00:44] Analytics-Tech-community-metrics, ECT-March-2015: Migrate Korma functionality into upstream - https://phabricator.wikimedia.org/T92952#1125049 (Dicortazar) Just to mention that a 99% of the funcionality is already in master. The following step consists of keep working on the branch till {T92953} is ready... [13:02:23] Analytics-Tech-community-metrics, ECT-March-2015: Migrate Korma identitites database to SortingHat - https://phabricator.wikimedia.org/T92953#1125067 (Qgil) [13:02:24] Analytics-Tech-community-metrics, ECT-March-2015: Instructions to update user data in korma - https://phabricator.wikimedia.org/T88277#1125066 (Qgil) [13:29:33] (PS1) Mforns: Add config to run funnel_failure_rates_by_type [analytics/limn-edit-data] - https://gerrit.wikimedia.org/r/197318 (https://phabricator.wikimedia.org/T89251) [13:32:22] (CR) Mforns: "I could not manage to execute within stat1003 with my permits. So I created this changeset." (2 comments) [analytics/limn-edit-data] - https://gerrit.wikimedia.org/r/197318 (https://phabricator.wikimedia.org/T89251) (owner: Mforns) [13:43:12] (PS1) Mforns: Make row assignable [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/197319 (https://phabricator.wikimedia.org/T89251) [13:55:22] Analytics-Tech-community-metrics, Phabricator, Wikimedia-Hackathon-2015, ECT-March-2015: Metrics for Maniphest - https://phabricator.wikimedia.org/T28#1125190 (Qgil) >>! In T28#1081799, @Aklapper wrote: > Generally: Not even sure where and how to "publish" such metrics. Ok, http://korma.wmflabs.or... [13:56:43] mforns: did you try to execute that sql? [13:56:50] I think the -- style comments will break it [13:57:02] hey milimetric, yes I did [13:57:06] it works in my machine [13:57:08] qchris: oooo guard, sorries, will look at thta today [13:57:26] mforns: yeah, but not on stat1003, the comment policy is different (I know, that's insane) [13:57:39] but I tested it agains the real database [13:57:53] ok, I will remove the comment [13:58:06] if it is not inline, it is ok? [13:58:09] milimetric, ^ [13:58:39] mforns: no, it wants # comments like this [13:58:39] -- instead of like this [13:58:53] oh! [13:59:13] I really don't understand why, I was meaning to ask Sean about it. [13:59:45] milimetric, ok [14:00:24] (PS2) Mforns: Add config to run funnel_failure_rates_by_type [analytics/limn-edit-data] - https://gerrit.wikimedia.org/r/197318 (https://phabricator.wikimedia.org/T89251) [14:00:34] milimetric, changed it ^ [14:47:28] Analytics, operations: Fix Varnishkafka delivery error icinga warning - https://phabricator.wikimedia.org/T76342#1125299 (Dzahn) reopen for https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=all&type=detail&servicestatustypes=8&hoststatustypes=3&serviceprops=2097162&nostatusheader .. or new bug? [14:48:21] Analytics, operations: Fix Varnishkafka delivery error icinga warning - https://phabricator.wikimedia.org/T76342#1125300 (Dzahn) Resolved>Open [14:50:50] ottomata: Cool. Thanks! [15:08:49] Analytics, operations: Fix Varnishkafka delivery error icinga warning - https://phabricator.wikimedia.org/T76342#1125358 (Dzahn) Open>Resolved re-closing per: < ottomata> mutante: i think that is a new issue, that one was for check_ganglia. this one is from graphite, and the data is really not in... [15:17:06] Analytics-Engineering, operations, Graphite, Icinga: icinga UNKNOWN Varnishkafka Delivery Errors / varnishkafka data not in graphite - https://phabricator.wikimedia.org/T92965#1125408 (Dzahn) [15:29:28] morning [15:29:37] ottomata, I brought you a present [15:41:45] Analytics-Wikimetrics: Puppet is restarting queue every time it runs - https://phabricator.wikimedia.org/T74565#1125473 (Nuria) Open>Resolved [15:46:46] Analytics-Engineering, Analytics-Wikimetrics: Unable to add a custom cohort user - https://phabricator.wikimedia.org/T91751#1125495 (Nuria) Karen, If (once you are logged in) you go to "my cohorts" there is a button to "upload a new cohort" [16:00:07] Ironholds: oowhat?! [16:00:15] nuria: I disabled notifications of eventlogging validation alerts [16:00:27] it was texting me every 10 minutes or so sometimes [16:00:49] ottomata, remember how we kept talking about making things non-static and moving the oft-used generic utility functions into a dedicated class? [16:00:53] ottomata: ok, let's enable those back on once we deploy mforns_brb changes and VE team corrects their validation issues [16:00:53] patch in gerrit, yo [16:00:58] yes [16:01:09] ha, awesome [16:01:27] Ironholds: yes yes [16:01:42] nuria, hope it is helpful! :D [16:03:43] (unrelated: mozilla uses mercurial. Eurgh.) [16:15:01] Ironholds: once in your mind mercurial commands will NEVER Disappear [16:15:27] at the moment my mind is mostly preoccupied with "where the hell is nscore.h agh mozilla what is this" [16:15:28] ottomata: lemme know if you want to look at kafka +EL in beta labs again [16:15:36] but their universal encoding detector is too beautiful not to steal [16:15:48] k making lunch, will get to that after nuria [16:15:55] ottomata: k [16:16:13] qchris: also i want to talk with you about the guard, will ping you after lunch if you are still around [16:16:33] ottomata: Sure. I'll be around the whole evening ... [16:16:37] except for eating. [16:16:42] Mhmmm ... eating :-) [16:16:54] Ironholds: will look at your patch later on today/early tomorrow [16:17:02] coolio :) [16:20:07] Analytics-Cluster, Analytics-Kanban, Performance: Cluster report that looks at x-Analytics header and extracts the date to calculate uniques. - https://phabricator.wikimedia.org/T92977#1125610 (Nuria) NEW [16:21:00] Analytics-Engineering, operations, Graphite, Icinga, Patch-For-Review: icinga UNKNOWN Varnishkafka Delivery Errors / varnishkafka data not in graphite - https://phabricator.wikimedia.org/T92965#1125616 (fgiunchedi) fix merged, seems to have recovered, pending creation of related alarms [17:12:09] Analytics-Engineering, Analytics-EventLogging, Analytics-Kanban: Fix issue with database writes not keeping up with events - https://phabricator.wikimedia.org/T90029#1125752 (Nuria) While I do not think recent code patches changed anything core they definitely mitigated issues that affect db writes whe... [17:12:52] Analytics-Engineering, Analytics-Kanban: Backfilling EL events from 20150206 to 20150210 - https://phabricator.wikimedia.org/T89269#1125761 (Nuria) [17:12:53] Analytics-Engineering, Analytics-EventLogging, Analytics-Kanban: Fix issue with database writes not keeping up with events - https://phabricator.wikimedia.org/T90029#1125760 (Nuria) Open>Resolved [17:20:00] Analytics-Volunteering, Engineering-Community, Phabricator, Project-Creators, and 2 others: Analytics-Volunteering and Wikidata's Need-Volunteer tags; "New contributors" vs "volunteers" terms - https://phabricator.wikimedia.org/T88266#1125775 (Aklapper) a:Aklapper>kevinator I'm blocked here o i... [17:34:38] Analytics-Kanban: Onboard Madhu - https://phabricator.wikimedia.org/T92985#1125818 (kevinator) NEW [17:35:18] Analytics, Analytics-Kanban: Onboard Madhu - https://phabricator.wikimedia.org/T92985#1125829 (kevinator) [17:40:43] qchris_away: hiayaa [17:40:47] lemme know when you are back [17:41:01] Argh. Forgot to reset status. [17:41:04] np [17:41:05] ottomata: back :-) [17:41:43] so, i keep waffling about this comment i'm writing so asking you in chat is easier :) [17:41:45] so! [17:41:46] q [17:41:53] Sure. [17:41:54] do you think the guard should have its own refiner-source clone [17:41:59] or, should there be a generic one? [17:42:19] probably its own, right? [17:42:21] Either way is fine by me. [17:42:33] As long as the clone is on HEAD, [17:42:40] a shared clone is fine. [17:42:46] hm, [17:42:50] ya i am not sure. ok [17:42:58] You prefer a separate clone? [17:43:14] i think maybe? so that no one thinks it is for generic use? [17:43:18] and it can do what it needs to do? [17:43:19] dunno. [17:43:23] i can't decie [17:43:28] if we have a generic clone [17:43:36] then /a/refinery-source is fine, but we should make another class [17:43:44] refinery::source [17:43:46] that handles the clone [17:43:50] and ::guard should just include that [17:43:55] Ok. [17:44:01] as for /a [17:44:13] it would be good if we could infer that from $::role::statistics::working_path [17:44:24] Sure. But that [17:44:29] but i don't want to make refinery::guard depend on that class [17:44:36] would bring ... exactly. [17:44:38] we could parameterize it [17:44:43] and pass it in from the node ddf [17:44:44] def [17:44:48] since it is included there [17:44:56] class { 'role::analytics::refinery::source': [17:44:56] path => "${::role::statistics::working_path}/refinery-source" [17:44:56] } [17:45:02] role classes with parameters? [17:45:08] I thought that is frowned upon? [17:45:14] i know. but. apparently that is going to be encouraged more with hiera? [17:45:50] asking _joe_ in ops now... [17:46:01] But that still buys the dependency upon $::role::statistics::working_path [17:46:11] no [17:46:21] not if the parameter is passed in from the node def [17:46:32] the puppetization itself is path agnostic then [17:46:32] Oh. Right. From the node! [17:46:33] of guard [17:47:07] hm, qchris, ok. i think i should make the working_path a hiera var [17:47:32] bwerrrrrrrrr [17:47:34] k. [17:47:36] i don't want to do that right now though. [17:47:37] so. [17:47:43] Ha. [17:47:54] let's leave the variable hardcoded with a nice comment and TODO [17:47:56] actually. [17:48:01] Ok. [17:48:04] it is probably better to just introduce the dependency! [17:48:09] and add a nice coment and todo [17:48:17] Agreed. [17:48:44] ok so [17:48:46] new class [17:48:48] refinery::source [17:48:51] that just handles the clone [17:49:05] includes role::statistics [17:49:10] Yup. [17:49:23] Cool. Thanks. [17:49:36] then class reinfery::guard includes refinery::source, and does the other stuff :) [17:49:49] Exactly. [17:49:49] also, since the variable is being defined inside of the class [17:49:52] you can name the variable just $path [17:49:54] Will amend the change. [17:49:58] like is done for ::refinery class [17:50:03] that way your guard can access it as [17:50:08] $role::analytics::refinery::source::path [17:50:27] Yup. [17:50:37] cool. sorry i forgot about that! [17:50:58] No worries :-) [17:54:16] nuria: yt? [17:54:26] ottomata: yessir [17:56:55] wanna help me figure out why this doesn't validate? [17:56:56] :) [17:57:38] ottomata: yes of course, batcave? [17:57:41] k [19:04:19] ottomata: logster question [19:04:28] ottomata: where is puppet depot is teh source for: https://github.com/wikimedia/operations-debs-logster/blob/master/logster/parsers/LineCountLogster.py [19:04:50] puppet? [19:04:55] the puppetization of logster? [19:05:58] nuria: https://github.com/wikimedia/operations-puppet/tree/production/modules/logster [19:06:40] ottomata: but wait , is taht a different depot from "operations/puppet"? [19:07:43] cause if i git clone operations/puppet i do not have this logster/parser directory [19:08:26] no, same [19:08:33] are you asking about puppet [19:08:37] or about logster source? [19:09:01] nuria: https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/logster [19:09:02] puppet , looking into "/workplace/puppet/modules/logster" [19:09:15] there are 2 repositories [19:09:20] logster, and puppet :) [19:09:23] puppet has a logster module [19:09:25] ottomata: now i see, two repos [19:09:27] that uses the logster package [19:09:34] which is build from the logster repo [19:09:36] built* [19:11:03] nuria: do you think the varnishkafka eventlogging url should be /beacon/event.gif? [19:11:05] or something else? [19:11:13] is there a real reason to have .gif on the end? [19:11:46] ottomata: yes, so it is obviously a "beacon" image [19:12:04] and can be inserted in w/o issues [19:13:22] ottomata: I think browser (or most browsers) will not care [19:13:39] but that way it is easier to know that we need to answer back with the right mime type [19:14:39] ok [19:15:29] ottomata: let me take a look at vcl for a sec [19:17:44] ottomata: actually i do not think we are sending just 204 so it wouldn't matter but i think gif keeps things clean [19:18:37] i think it keeps things ugly! [19:18:41] it is not a gif! [19:18:43] there is no gif! [19:30:36] ottomata: You are going to have to complain to whoever started the internet [19:30:55] ottomata: as fake images are older than the times [19:32:05] ottomata: i also think that given VCL restrictions when it comes to code, having something as clear as "event.gif" to match on is not bad, now, this is not a hill to die on [19:32:58] indeed. :) [19:33:02] haha [19:33:09] * Ironholds grins [19:33:17] you should totally do that [19:33:24] "Postel, we need to talk about gifs!" [19:54:02] nuria: is there a doc about how to deploy eventlogging in prod? [19:54:25] OPH [19:54:26] oh [19:54:26] nm [19:54:27] found it [19:54:29] sorry, [19:54:40] it is very obvious, not sure why i didn't see it [20:15:11] ok [20:22:44] ja ok, nuria, this code works as is, with the python-kafka package I built [20:22:51] aham [20:23:08] i think we are ok to merge. if we do, i will deploy it betalabs and then puppetize the new stuff [20:23:37] ok [20:23:47] let me look at changes again [20:24:10] ottomata: let's do this one 1st right? https://gerrit.wikimedia.org/r/#/c/196637/ [20:24:53] yes, but we can do them at the same time. the second doesn't require any config changes. [20:25:15] ottomata: this one still has some fixes for flake8 [20:25:16] https://gerrit.wikimedia.org/r/#/c/196073/7 [20:25:40] yeah what is up?! [20:26:02] lines too long? [20:26:03] oh man [20:26:03] ok. [20:26:10] ottomata: ya [20:26:14] sheesh [20:26:28] so once both patches run flake8 clean we can merge both [20:26:32] k [20:26:33] on it [20:30:26] do I have to fix them all? it is telling me that my usage documentation is too long! [20:30:41] ahh wahtever i cna shorten [20:33:10] ottomata: tox -e flake8 will tell you all errors if you are using vagrant [20:33:29] ottomata: but ahem, yes ... you have to fix all those lovely line errors [20:33:42] ottomata: a joy everytime [20:37:59] nuria: [20:38:06] yesss [20:38:07] how would you separate this line? [20:38:08] return uri_append_query_items(uri, {'raw': True}) if 'raw=True' not in uri else uri [20:38:23] splitting the if [20:38:24] it is 88 chars [20:38:31] after the if? [20:38:38] return uri_append_query_items(uri, {'raw': True}) if [20:38:38] 'raw=True' not in uri else uri [20:38:39] if nanana: [20:38:42] blah [20:38:44] else : [20:38:45] BOOO [20:38:46] blah-blah [20:38:51] ottomata: jajaja [20:38:59] then i need a variable or 2 return statements [20:38:59] BOOO [20:39:36] jaja, you are going to have to come to terms with flake8 man [20:40:23] but that feels against pythonic! [20:41:18] bah [20:41:20] fine. [20:41:38] ottomata: i cannot help you there as my python looks like java and NOW that is not pythonic [20:41:51] it's real sad actually [20:45:23] ok nuria this one is happy [20:45:24] https://gerrit.wikimedia.org/r/#/c/196073 [20:47:04] this one too! [20:47:04] https://gerrit.wikimedia.org/r/#/c/196637/ [20:47:09] ottomata: k, merged, I am taking off for the next hour and a half but will see your other patch [20:47:13] woot! [20:47:14] ahhh, looking [20:47:59] ottomata: lemme run tests [20:49:08] ottomata: k, running clean [21:11:12] Analytics-Dashiki, Analytics-Kanban, Patch-For-Review: Pageviews not loading in Vital Signs - https://phabricator.wikimedia.org/T90742#1126568 (mforns) a:Nuria>mforns [21:23:07] Analytics, Analytics-Kanban: Turn off Zero - Limn dashboards & put up a "moved sign" - https://phabricator.wikimedia.org/T92920#1126606 (kevinator) [21:24:59] Analytics, Analytics-Kanban: Turn off Zero - Limn dashboards & put up a "moved sign" - https://phabricator.wikimedia.org/T92920#1123876 (kevinator) [21:30:18] ottomata, yt? [21:36:27] yup hey [21:37:51] hey ottomata, I'm in wikimetrics-staging1, how do I know if the local puppet master has all the latest changes? [21:39:57] cd /var/lib/git/operations/puppet [21:39:59] git log [21:40:03] see how up to date it is [21:46:56] ottomata, thanks! it actually has all the changes [21:47:06] I'm looking at: https://phabricator.wikimedia.org/T90742 [21:48:36] I've seen that in wikimetrics-staging the DailyPageViews files have 644 permits, whereas wikimetrics (production) the same files have 600 [21:51:13] but it doesn't seem to be a puppet problem [21:51:36] 600, hm, interesting, nuria was telling me about this wonce, but i don't remember what the outcome was [21:52:45] ottomata, oh, wikimetrics (production) puppet last change dates from Jan 30th [21:56:11] hm [23:15:21] Analytics-Kanban, Analytics-Wikimetrics: Utf-8 names on json reports appear as unicode code points: "\u0623\u0645\u064a\u0646" - https://phabricator.wikimedia.org/T93023#1126852 (Nuria) NEW [23:33:38] milimetric, yt? [23:50:45] Analytics-Dashiki, Analytics-Kanban, Patch-For-Review: Pageviews not loading in Vital Signs - https://phabricator.wikimedia.org/T90742#1126938 (mforns) DailyPageviews files have **600 permits on wikimetrics1** and the web application can not retrieve them. Whereas the same files in **wikimetrics-stagin...