[01:28:06] Analytics-Kanban, DBA, Editing-Analysis, Patch-For-Review: Edit schema needs purging, table is too big for queries to run (500G before conversion) {oryx} - https://phabricator.wikimedia.org/T124676#2065533 (Neil_P._Quinn_WMF) @jcrespo, yes, the data on the slave looks good. I think you can close this! [02:45:06] Analytics-Wikimetrics: Make wikimetrics role use hiera - https://phabricator.wikimedia.org/T86190#2065726 (yuvipanda) @madhuvishy this is no longer necessary, I suppose? [05:58:36] Analytics-Wikimetrics: Make wikimetrics role use hiera - https://phabricator.wikimedia.org/T86190#2065835 (madhuvishy) @yuvipanda I think so! We should be all good. [05:58:52] Analytics-Wikimetrics: Make wikimetrics role use hiera - https://phabricator.wikimedia.org/T86190#2065836 (madhuvishy) Open>Invalid [09:52:51] Analytics, ContentTranslation-Analytics, Operations, Ops-Access-Requests, Patch-For-Review: access for nikerabbit to researchers - https://phabricator.wikimedia.org/T127808#2065960 (ema) Open>Resolved @Nikerabbit can now access the db. Closing. [10:13:03] Analytics-Tech-community-metrics, DevRel-February-2016: Make GrimoireLib display *one* consistent name for one user - https://phabricator.wikimedia.org/T118169#2065992 (Aklapper) [10:25:31] Analytics-Tech-community-metrics, Developer-Relations, DevRel-March-2016, Gerrit-Migration: Make MetricsGrimoire/korma support gathering Code Review statistics from Phabricator's Differential - https://phabricator.wikimedia.org/T118753#2066001 (Aklapper) a:Aklapper [10:34:39] Analytics-Tech-community-metrics, DevRel-February-2016: Empty "subject" and "creator" fields for mailing list thread on mls.html - https://phabricator.wikimedia.org/T116284#2066023 (Aklapper) @Dicortazar was able to reproduce. [10:49:36] hey a-team :] [10:49:48] Analytics-Tech-community-metrics, DevRel-February-2016: demographics.html: "Tickets participants" has "184 attracted" data for 1yearNone As discussed in our meeting, unassigning... [10:50:54] mforns: o/ [10:51:11] \o mforns, elukey [10:51:20] hey guys [10:54:19] Analytics-Tech-community-metrics, DevRel-February-2016, DevRel-March-2016: Make GrimoireLib display *one* consistent name for one user - https://phabricator.wikimedia.org/T118169#2066057 (Aklapper) The issues found in T117871#2064483 also cover/affect this task (currently only the ITS data source is st... [10:54:43] Analytics-Tech-community-metrics, DevRel-February-2016: Empty "subject" and "creator" fields for mailing list thread on mls.html - https://phabricator.wikimedia.org/T116284#2066059 (Aklapper) p:Low>Lowest [10:58:34] Analytics-Tech-community-metrics, DevRel-February-2016, DevRel-March-2016: gerrit_review_queue.html: List of Repositories has not been updated recently - https://phabricator.wikimedia.org/T128170#2066064 (Aklapper) [10:58:54] Analytics-Tech-community-metrics, DevRel-February-2016, DevRel-March-2016: gerrit_review_queue.html: List of Repositories has not been updated recently - https://phabricator.wikimedia.org/T128170#2066077 (Aklapper) p:Triage>High [10:59:08] Analytics-Tech-community-metrics, DevRel-February-2016, DevRel-March-2016: gerrit_review_queue.html: List of Repositories has not been updated recently - https://phabricator.wikimedia.org/T128170#2066064 (Aklapper) [10:59:14] Analytics-Tech-community-metrics, Developer-Relations, DevRel-February-2016: Mark BayesianFilter repository as inactive - https://phabricator.wikimedia.org/T118460#2066080 (Aklapper) [10:59:16] Analytics-Tech-community-metrics, DevRel-February-2016, DevRel-March-2016: gerrit_review_queue.html: List of Repositories has not been updated recently - https://phabricator.wikimedia.org/T128170#2066064 (Aklapper) [10:59:20] Analytics-Tech-community-metrics, DevRel-February-2016, DevRel-March-2016: gerrit_review_queue.html: List of Repositories has not been updated recently - https://phabricator.wikimedia.org/T128170#2066064 (Aklapper) [10:59:52] Analytics-Tech-community-metrics, DevRel-February-2016, DevRel-March-2016: gerrit_review_queue.html: List of Repositories has not been updated recently - https://phabricator.wikimedia.org/T128170#2066064 (Aklapper) [11:00:30] Analytics-Tech-community-metrics, DevRel-March-2016: For some people, top-contributors.html displays three-digit numbers instead of names - https://phabricator.wikimedia.org/T128171#2066091 (Aklapper) [11:00:39] elukey, joal have you guys ever tested puppet code in mw-vagrant? [11:00:42] Analytics-Tech-community-metrics, DevRel-March-2016: For some people, top-contributors.html displays three-digit numbers instead of names - https://phabricator.wikimedia.org/T128171#2066104 (Aklapper) p:Triage>Normal [11:00:55] mforns: nope.. do you really need it? [11:01:12] Analytics-Tech-community-metrics, DevRel-February-2016, DevRel-March-2016: Many profiles on profile.html do not display identity's name though data is available - https://phabricator.wikimedia.org/T117871#2066105 (Aklapper) [11:01:21] elukey, not sure, Andrew said this was an option [11:02:08] elukey, yesterday I 'finished' the puppet code, now I'm going to test it, if you want we can batcave and try that [11:02:14] Analytics-Tech-community-metrics, DevRel-February-2016: What is contributors.html for, in contrast to who_contributes_code.html and sc[m,r]-contributors.html and top-contributors.html? - https://phabricator.wikimedia.org/T118522#2066106 (Aklapper) Note to myself: Upstream is frozen except for bugs, so I c... [11:02:39] Analytics-Tech-community-metrics, DevRel-March-2016: For some people, top-contributors.html displays three-digit numbers instead of names - https://phabricator.wikimedia.org/T128171#2066091 (Aklapper) [11:02:41] Analytics-Tech-community-metrics, DevRel-February-2016: Key performance indicator: Top contributors: Find good Ranking algorithm fix bugs on page - https://phabricator.wikimedia.org/T64221#2066107 (Aklapper) [11:03:17] Analytics-Tech-community-metrics, DevRel-February-2016: Key performance indicator: Top contributors: Find good Ranking algorithm fix bugs on page - https://phabricator.wikimedia.org/T64221#660476 (Aklapper) [11:03:34] mforns: would it be fine in ~one hour and a half? I am in the middle of a re-image for ops, then I'll grab something to eat quickly [11:03:43] Analytics-Tech-community-metrics, DevRel-February-2016, DevRel-March-2016: Key performance indicator: Top contributors: Find good Ranking algorithm fix bugs on page - https://phabricator.wikimedia.org/T64221#660476 (Aklapper) [11:04:45] elukey, of course, I'll be trying here [11:06:05] Analytics-Tech-community-metrics, DevRel-February-2016, DevRel-March-2016: Contributor pages which show user name but not any other data should include an explanation - https://phabricator.wikimedia.org/T58111#2066128 (Aklapper) [11:06:13] Analytics-Tech-community-metrics, DevRel-February-2016, DevRel-March-2016: Mismatch between six names and certain email address in mediawiki-identities data - https://phabricator.wikimedia.org/T123643#2066130 (Aklapper) p:Normal>Low [11:06:37] Analytics-Tech-community-metrics, DevRel-February-2016, DevRel-March-2016: Empty "subject" and "creator" fields for mailing list thread on mls.html - https://phabricator.wikimedia.org/T116284#2066133 (Aklapper) [11:08:11] mforns: one option is https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster#Set_up_a_developer.27s_machine_for_pushing_to_Labs_.28method_.232.29 [11:08:51] elukey, yes that's exactly what I was reading :] [11:08:54] thx! [11:10:28] so my undestanding is that you can spin up an instance that acts a puppet master [11:10:44] pull your commits from gerrit [11:10:54] run puppet and also check your code in there [11:11:21] I am not sure if you have a specific role for the puppet code that you have written [11:11:38] mforns --^ [11:11:48] mforns: I have not, no [11:12:02] np joal :] [11:12:22] elukey, aha [11:12:32] no, it's the role statistics [11:12:40] but it's a new module [11:13:46] I think you don't need to commit and pull from gerrit [11:14:00] you can configure you local machine to push to your labs instance [11:14:04] but I'm not sure [11:14:05] ah yes [11:14:09] * mforns reads [11:14:27] I thought you had it in gerrit, git allows that so you can [11:14:42] I'd start with a self hosted puppet master with the statistics role [11:15:05] elukey, makes sense [11:15:14] thx [11:50:08] mforns: how is it going? [11:58:26] elukey, following the tutorial [11:59:23] let me know if I can help, I am still finishing the re-image but I can take some breaks :) [12:00:11] ok [12:10:02] elukey, question: there's no statistics role in the labs instance configuration, do you know how I can configure it manually? [12:17:28] mforns: looking [12:21:47] I am seeing statitcs servers configured in heira, but also in site.pp [12:21:50] mmmm [12:22:53] what about adding a node in site.pp manually (in your puppet master) like node 'stat1001.eqiad.wmnet' { ? [12:26:11] I am trying to understand if it would be possible with https://wikitech.wikimedia.org/wiki/Hiera:Analytics [12:26:19] there must be an easier way [12:29:04] elukey, makes sense [12:31:48] Analytics-Tech-community-metrics, DevRel-March-2016: For some people, top-contributors.html displays three-digit numbers instead of names - https://phabricator.wikimedia.org/T128171#2066384 (Lcanasdiaz) Having a look at the file all-top.json I found out the bug is coming from the identifier we get from Me... [12:34:13] mforns: let's try to force the host into site.pp on the self hosted puppet master [12:34:16] brutally [12:34:18] :D [12:34:23] elukey, ok, am doing that [12:34:26] :] [12:56:40] elukey, I made some progress, but still stuck, am going to have lunch and will come back in an hour or so, maybe then we can hang out [12:59:06] I was about to say the same thing :) [13:05:42] :] [14:27:30] elukey, I'm back, I managed to have puppet agent execute in the labs instance and execute the config that I need, there are some errors I'm looking at right now [14:29:47] \o/ [14:38:03] elukey, can I do this? Class['...'] -> Define['...'] [14:38:19] set an ordering between a class and a define? [14:39:06] mforns: not that expert in puppet, we might be able to reason about it though.. [14:39:17] what are you trying to do? [14:39:30] elukey, there's an error when I execute the agent [14:39:49] I had: Class['::reportupdater'] -> Class['::reportupdater::job'] [14:40:06] the error says puppet can't find the class reportupdater::job [14:40:21] and it makes sense, because it is not a class, it is a define [14:40:32] so I was wondering if I can do Class['::reportupdater'] -> Define['::reportupdater::job'] [14:40:41] * mforns tests [14:42:06] no... [14:46:55] mforns: I am a bit ignorant about the thing that you are doing so I can't help much :( [14:49:41] https://docs.puppetlabs.com/puppet/latest/reference/lang_relationships.html#operands [14:50:21] so define seems not among the allowed operands [14:50:23] Hlllow [14:50:45] elukey: you trying to do dependencies with define? [14:50:53] if so, you can! [14:51:11] not me! Marcel! I was throwing random stuff from the internet to help [14:51:14] :D [14:51:18] Class['my::class'] -> My::Define::Name['titleofdefine'] [14:51:20] or [14:51:26] Class['my::class'] -> My::Define::Name[$title] [14:51:29] if in the define [14:51:53] note the capitalized first letters of the define [14:52:21] mforns: --^ don't listen to me [14:53:24] thanks elukey and ottomata, mmm [14:53:54] ottomata, like.. Reportupdater::Job[$title] ?? [14:54:12] trying [14:54:32] yup! [14:55:54] it vvorks! [14:56:07] but now I get another problem... :( [14:58:16] a-team, will be gone for about 1h30, back for standup ! [14:58:26] see ya joal [15:00:19] ottomata, I'm trying to test the puppet change on labs with the help of elukey. I launched a new instance and configured it as role::puppet::self. However there's no role 'statistics' that can be configured there, so we tried to set it up manually adding the node to site.pp, is that ok? [15:00:58] ottomata, now after some progress with code erros, I got no more code erros, but instead: Error: Failed to apply catalog: Could not find dependency File[/etc/ldap/ldap.conf] for Class[Puppet::Self::Config] at /etc/puppet/modules/puppet/manifests/self/master.pp:65 [15:02:22] hmmmmm that is a strange one [15:02:29] does that error persist mforns? [15:02:33] sometimes puppet in labs is not very smart [15:02:44] executing again [15:02:47] mforns: you are making a new module though, right? [15:02:51] do you need to apply role statistics? [15:02:52] also [15:02:52] yes [15:02:57] why? [15:03:15] good question [15:03:33] mforns: do you have access to this page? [15:03:33] https://wikitech.wikimedia.org/wiki/Special:NovaPuppetGroup [15:03:45] btw, especially for just puppet testing [15:03:49] editing site.pp is totally fine [15:03:51] ottomata, yes I have access [15:03:51] i do that sometimes [15:03:51] but [15:03:56] you can add new classes to that list there [15:04:02] aha [15:04:05] and then they will be available to check in the instanace configure page [15:04:11] but, sometimes, don't worry about it [15:04:15] doing like you do with site.pp is fine [15:04:20] for just one off testing [15:04:35] ok cool, thanks! [15:04:52] so ja, we talked about making this module not dependent on the statisics one, right? [15:05:02] anything that the statistics role would do would pass in parameters to the module [15:05:17] so in prod, likely we would use the module via the statistics role [15:05:21] ottomata, sure, I did it like that [15:05:27] just like the limn:: bla bla ones are used [15:05:27] but [15:05:31] if you are just ediitng site.pp for testing [15:05:42] you should be able to include your module classes directly in the node stanza [15:06:03] ah yes even more straightforward [15:06:12] node 'blblbla' { [15:06:12] class { 'reportupdate': ... } [15:06:12] reportupdater::job { 'testjob': ... } [15:06:12] } [15:06:19] I see..... [15:06:25] k [15:06:29] will do [15:06:31] cool :) [15:06:34] * mforns tries [15:13:45] ottomata, same ldap error, should I include something ldap-related in my node stanza? [15:14:05] Could not find dependency File[/etc/ldap/ldap.conf] for Class[Puppet::Self::Config] [15:15:21] mforns: you probably don't have /etc/ldap/ldap.conf on your server, and the puppet master needs it [15:16:02] I guess you can either comment it in the repo or touch a file in /etc/ldap/ldap.conf, but probably the latter won't work [15:16:04] elukey, sure, so probably there's another piece of puppet missing no? [15:16:32] elukey, the thing is, if I remove my changes, it works [15:16:38] ahhahaha [15:16:39] okok [15:16:48] hehehe [15:17:08] but it doesn't make any sense [15:18:00] puppet-lint doesn't give you any warning/errors? [15:18:14] it might be something messing up with site.pp [15:18:45] mforns: when you say that it does work without your changes, you mean without the site.pp config right? [15:18:54] elukey, yes [15:21:18] mforns: can you puppet-lint the new site.pp and check for errors? [15:21:31] elukey, sure [15:23:18] elukey, ottomata, it seems I had to: include standard [15:23:31] probably obvious [15:23:43] ah! [15:23:55] sorry, just read this [15:23:58] no wouldnt'h ave been obvious to me [15:24:00] nice find [15:24:10] guess that makes sense though [15:24:22] by defining a node stanza you are overriding any classes that are automatically applied to nodes in labs [15:28:49] ottomata: (if you have time) I am trying to run oozie with a custom coordinator.xml/properties and workflow.xml. These files are in hdfs under /tmp/elukey/pageviewdumps meanwhile the rest of the files are still in the refinery. The goal is to run oozie with some -D parameters to override [15:29:31] but my jobs are failing and some logs are indicating missing depts like /lib, etc.. so I am wondering if I should stop and use another method [15:29:55] I ran oozie like this [15:29:56] oozie job -config coordinator.properties -Duser=elukey -Dstart_time=2015-05-01T00:00Z -Dend_time=2015-05-01T01:00Z -Dworkflow_file=hdfs://analytics-hadoop/tmp/elukey/pageviewdumps/workflow.xml -Darchive_directory=hdfs://analytics-hadoop/tmp/elukey/pageviewdumps/archive -submit [15:30:42] and in coordinator.properties I specified that the coordinator.xml is under /tmp/elukey/pageviewdumps/ [15:31:16] coordinator_file = ${name_node}/tmp/elukey/pageviewdumps/coordinator.xml [15:46:55] elukey, ottomata, it seems to vvvork! [15:47:10] thanks guys :] [15:48:13] elukey: sorry, just saw these! [15:48:15] ah! looking [15:48:19] f yay! [15:48:21] mforns: yay! [15:49:10] elukey: where is your coordinator.properties? [15:49:21] in my home dir [15:49:51] joal: want to talk about uniques numbers? [15:50:05] but I managed to fire a job, I am checking logs [15:50:14] oozie is soo verbose [15:50:20] I'd like a oozie --color [15:50:21] :P [15:50:54] org.apache.oozie.command.CommandException: E0800: Action it is not running its in [OK] state [15:51:02] gooooood at least something is working :P [15:51:33] haha, ok [15:51:41] elukey: have you gotten just the workflow to work by itself? [15:52:40] I took the workflow.xml for pageviews hourly and removed some part with the help of Joseph and Dan (basically everything before transform) [15:53:08] I would like to manage to archive at least one hour of pageviews of May 2015 :) [15:54:18] aye, but, have you been able to run just a workflow [15:54:21] without a coordinator? [15:54:27] using your workflow.xml? [15:55:24] ah no I haven't tried, Joseph suggested to test a coordinator for one hour so it would have been ready for the whole processing once done [15:55:42] ja that woudl work too [15:55:51] but sometimes its nice to make sure workflow works first [15:55:55] I can try the workflow.xml though, now I kinda know what those files are :P [15:55:56] if it does, then you know that isn't your problem [15:56:00] hehe, ja [15:56:10] okok good suggestion, I'll try [15:56:12] lots of crazy abstractions in oozie to understand [16:00:57] /tmp/elukey/pageviewdumps/archive/pageview/legacy/hourly/2015/2015-05/pageviews-20150501-010000.gz [16:01:10] something did work even though the horrendous error [16:01:23] I believe it was in marking the dir as one [16:01:41] that I couldn't without being the hdfs user probably [16:01:53] ? [16:02:27] no sorry I am drunk reading oozie's logs [16:02:31] haha [16:02:32] mark_transformed_pageview_dataset_done [16:02:48] is the step giving errors but then thew workflow proceeded [16:02:59] elukey: "something did work" .. ay ..you are feeling the pain of working with oozie eh? [16:03:39] nuria: now I feel it, I guess I am starting to become a real part of the team :) [16:03:46] elukey: jajaja [16:05:48] http://www.confluent.io/blog/distributed-consensus-reloaded-apache-zookeeper-and-replication-in-kafka - very nice reading [16:07:57] elukey: you keep finding these ones i haven't read! i thought i've read all the confluent posts! :) [16:11:01] I think I found them in https://medium.com/this-is-not-a-monad-tutorial/interview-with-jay-kreps-about-apache-kafka-46fbfdb870ca#.p75fkfvhw or something similar :) [16:11:11] read that one for sure [16:14:25] not to bum anyone out, but our rolling new active editor numbers are down 40%!!! since last year: https://vital-signs.wmflabs.org/#projects=enwiki/metrics=RollingNewActiveEditor [16:14:32] I think we should have some sort of monitoring for that :) [16:14:52] how does nobody look at this to the point where Lila's allowed to leave and say she helped us stem the editor decline? [16:16:09] :O [16:16:23] ja, her saying that was the first i had ever heard of it [16:16:47] i think to stem editor decline, one just needs to make the world more interesting [16:16:53] then people will have more to write about [16:17:02] or maye [16:17:08] allow un notable people to have articles [16:17:12] ha! [16:17:22] everyone would be an active editor then! [16:17:58] "dear wikipedia article about me diary, today I found out my swellbow is not related to tennis elbow, and that I have to take a break from playing bike polo and I am very sad" [16:20:14] :( [16:20:31] is it a permanent break or just a long pause? [16:20:55] I guess that climbing is included in the stop [16:26:13] Analytics: Compile a request data set for caching research and tuning - https://phabricator.wikimedia.org/T128132#2066756 (Nuria) >Is this roughly correct and would this be a manageable dataset size? Actually I think what you are requesting is about 200 bytes gzipped per request (aprox) . But, to be honest,... [16:27:15] nuria: I'm back ! [16:27:28] nuria: uniques after standup ? [16:27:52] milimetric: well erik mueller made like 20 counter points to her arguments right? it's all on wikimedia-l [16:28:43] milimetric: mmm.. that trend does not add up with other wikis: https://vital-signs.wmflabs.org/#projects=enwiki,eswiki,frwiki/metrics=RollingNewActiveEditor [16:28:54] joal: we have the org wide meeting after standup [16:33:58] Analytics-Kanban: Productionize last access jobs for daily calculations {bear} [8 pts] - https://phabricator.wikimedia.org/T122514#2066781 (Nuria) Open>Resolved [16:34:53] * milimetric yeah it does, nuria, just gotta toggle off enwiki 'cause it's like a blinding sun: https://vital-signs.wmflabs.org/#projects=frwiki,eswiki/metrics=RollingNewActiveEditor [16:46:33] Analytics, Continuous-Integration-Infrastructure, Patch-For-Review: Add json linting test for schemas in mediawiki/event-schemas - https://phabricator.wikimedia.org/T124319#1952779 (Ottomata) Also related: T124438 [16:47:14] elukey: don't forget about this one! https://gerrit.wikimedia.org/r/#/c/271276/ :) [16:50:19] ottomata: didn't forget, I am probably doing it next week :) [16:50:35] k cool [16:50:40] I still need to figure out if I can deploy media-wiki or if I need to add it to the deployments [16:50:50] who should I talk with? Release team? [16:53:05] not sure elukey! they would know at the very least [16:53:07] so sure [16:53:16] Analytics, Continuous-Integration-Infrastructure, Patch-For-Review: Add json linting test for schemas in mediawiki/event-schemas - https://phabricator.wikimedia.org/T124319#2066845 (bd808) [16:53:19] Analytics, EventBus: Continuous Integration for mediawiki/event-schemas - https://phabricator.wikimedia.org/T124438#2066844 (bd808) [17:08:23] ottomata: https://gerrit.wikimedia.org/r/#/c/266510/17/wmf-config/InitialiseSettings.php [17:10:32] that goes into https://gerrit.wikimedia.org/r/#/c/266510/17/wmf-config/ProductionServices.php [17:10:39] so I'll send another patch [17:14:15] Analytics-Tech-community-metrics, DevRel-February-2016: Backlogs of open changesets by affiliation - https://phabricator.wikimedia.org/T113719#2066868 (Dicortazar) After the meeting we all agreed that this seemed to be a good approach. So, we're adding this into the project to daily update the dataset. P... [17:14:53] ah elukey so its already back? [17:15:13] oh, nope [17:15:14] https://gerrit.wikimedia.org/r/#/c/266510/17/wmf-config/ProductionServices.php [17:16:01] nope, I need to add it again in that file :P [17:18:48] ottomata: https://gerrit.wikimedia.org/r/#/c/273488/1 [17:21:00] Analytics-Kanban: Productionize last access jobs for monthly calculations {bear} [8 pts] - https://phabricator.wikimedia.org/T124678#2066877 (Nuria) Open>Resolved [17:21:14] Analytics-Kanban: Eventlogging should start with one bad kafka broker, retest that is the case {oryx} [5 pts] - https://phabricator.wikimedia.org/T125228#2066879 (Nuria) Open>Resolved [17:21:27] Analytics-EventLogging, Analytics-Kanban: Some blacklist matching schemas are being consumed by Eventlogging {oryx} [5 pts] - https://phabricator.wikimedia.org/T126410#2066881 (Nuria) Open>Resolved [17:21:48] Analytics-Kanban, Patch-For-Review: ady.wikipedia needs to be added to whitelist of pageview domains {hawk} [3 pts] - https://phabricator.wikimedia.org/T126754#2066882 (Nuria) Open>Resolved [17:22:04] Analytics-Kanban: Dedicated and/or automated Wikimedia pageviews API project/tag in Phabricator Maniphest [1 pts] - https://phabricator.wikimedia.org/T119151#2066885 (Nuria) Open>Resolved [17:22:24] Analytics-Kanban, Patch-For-Review: Improve the data format of the browser report {lama} - https://phabricator.wikimedia.org/T126282#2066886 (Nuria) Open>Resolved [17:22:39] Analytics-Kanban: Prepare for lightning talk on Last access uniques - https://phabricator.wikimedia.org/T126411#2066887 (Nuria) Open>Resolved [17:22:52] Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: Upgrade pykafka to v2.2 [5 pts] - https://phabricator.wikimedia.org/T126075#2066888 (Nuria) Open>Resolved [17:23:17] Analytics-Kanban: Correct bug in mobile_apps session metrics job {hawk} - https://phabricator.wikimedia.org/T125960#2066889 (Nuria) Open>Resolved [17:23:49] Analytics: Make beeline easier to use as a Hive client {hawk} - https://phabricator.wikimedia.org/T116123#2066892 (Nuria) [17:23:51] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Upgrade to CDH 5.5 {hawk} - https://phabricator.wikimedia.org/T119646#2066891 (Nuria) Open>Resolved [17:24:05] Analytics, Analytics-Kanban, Operations, Patch-For-Review: Increase HADOOP_HEAPSIZE (-Xmx) for hive-server2 - https://phabricator.wikimedia.org/T76343#2066893 (Nuria) Open>Resolved [17:26:35] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Upgrade analytics and beta project Analytics Clusters to CDH 5.5 [8 pts] - https://phabricator.wikimedia.org/T127115#2066908 (Nuria) Open>Resolved [17:26:37] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Upgrade to CDH 5.5 {hawk} - https://phabricator.wikimedia.org/T119646#2066910 (Nuria) [17:27:02] Analytics-Kanban, Patch-For-Review: Pageviews per country wikistats report. Feed data from hive {lama} [13 pts] - https://phabricator.wikimedia.org/T118323#2066911 (Nuria) Open>Resolved [17:50:08] ottomata: https://wikitech.wikimedia.org/wiki/Deployments#Tuesday.2C.C2.A0March.C2.A001 [17:54:38] going offline a-team, talk with you next week! [17:54:45] bye elukey ! [17:54:51] nice weekend [17:56:35] bye elukey :) [17:57:35] logging off as well, have a good weekend a-team, see you on monday ! [17:57:48] nice weekend joal ! [17:57:58] bye mforns ;) [18:00:56] madhuvishy: https://wikitech.wikimedia.org/wiki/Analytics/Data/Unique_Devices#Data_Quality_Analysis [18:01:04] data quality thresholds [18:01:33] nuria: thanks - will look after this meeting [18:08:32] madhuvishy: yaya of course [19:44:52] hmm elukey, it looks to me like mediawiki-vagrant is using varnish 4! [19:45:00] might be a good place to test [20:00:17] ahhh rats nuria, dunno about this varnish in mediawiki thing. [20:00:25] they've made mw-vagrant use varnish 4 [20:00:46] which is not compatible with varnishkafka (yet, elukey is kinda sorta working on it) [20:03:44] hmmm there is a varnish 4 compatible varnishncsa [20:03:46] could just write to file [20:03:48] oook [20:04:56] ori: yt? or is long meeting still happening? [20:05:46] ottomata: nuria I was talking ori yesterday about this client ip change (I was confused about the regexes so was asking some doubts) - and he was mentioning it's too complicated a change to make just for quick surveys - and we could get rid of client ips all together [20:06:23] if not for quick surveys - and leila says quick surveys is done in next couple weeks [20:06:31] so if we just wait [20:06:35] we can remove ips. [20:07:43] user agent is the only other PII - and that's easy because we can ask the clients to send it if they need it. ori's point was that introducing magic in the fields opens up unnecessary complexity - and we may get requests for other things to be appended on the fly. [20:08:00] indeed, if we can get away without IPs altogether [20:08:02] then let's do it [20:08:09] i thought that was a requirement though [20:08:22] ottomata: yup - only for quicksurveys as far as i know [20:08:46] i dont know any other usecase - and they are gonna be done in 2 weeks. so we might as well just wait [20:08:51] nuria: what do you think? [20:09:09] sure [20:09:22] i worked on it for a bit - but it was a good learning experience for me and i dont mind killing the patch. [20:59:12] Analytics-Tech-community-metrics, DevRel-February-2016: Backlogs of open changesets by affiliation - https://phabricator.wikimedia.org/T113719#2067479 (Aklapper) [20:59:26] Analytics-Tech-community-metrics, DevRel-February-2016: Backlogs of open changesets by affiliation - https://phabricator.wikimedia.org/T113719#1674183 (Aklapper) [20:59:34] Analytics-Tech-community-metrics, DevRel-February-2016: Backlogs of open changesets by affiliation - https://phabricator.wikimedia.org/T113719#1674183 (Aklapper) Open>Resolved That looks good to me, indeed. Thank you. Closing as resolved, if we want to tweak we can do that in a followup task. [21:16:14] madhuvishy: sorry -- did we have something scheduled? [21:16:18] I am here now [21:38:54] ori: really? [21:38:57] i had no idea [21:44:45] madhuvishy: No, I wasn't aware of anything being scheduled either; I misinterpreted ori: yt? or is long meeting still happening? [21:45:07] yeah I think he had something to ask you [21:45:29] may be about the varnish vagrant stuff he was talking about before that [21:45:53] oh ja i meant long meeting that started this morning in sf [21:46:02] i heard it was 3 hours, and 3 hours hadn't passed since hte start [21:46:15] ori, i'm getting there, am testing your change in mw-vagrant with eventlogging-devserver [21:47:14] think its working [21:47:31] ori, madhuvishy if we are making server side changes now [21:47:34] and we are going to remove IPs [21:47:44] HMMMM, nm [21:48:08] we are making it log to event.gif, so IP be sent anyway, we just make processor not include it [21:48:16] ok [21:48:18] nm [21:48:20] it'll be the app server IP [21:48:35] does server side use that anywa? [21:48:37] now [21:48:38] ? [21:48:42] I don't think so [21:49:45] i don't think so either - server side format string doesn't include ip [21:50:09] ok cool, nm then! [21:50:21] ottomata, madhuvishy: I meant to ask you something about the sendBeacon change, actually. Do you think it's feasible to simply migrate the implementation from UDP to sendBeacon in one fell swoop? Maybe there should be an intermediate stage in which the way the event is set depends on some flag parameter to logEvent, and we use that flag to log some dummy events (say have some "Schema:EventLoggingSendBeaconTest") [21:50:26] and then remove the older implementation [21:50:38] maybe it would be safer to do it like that, in two stages, rather than all at once? [21:51:43] hm, i dunno, i don't know how much folks value server side events...i guess PageEdit is from those [21:51:43] hm [21:52:04] ori, i think with enough testing its ok to do all at once [21:52:09] hmmm ori can this change not be tested in beta? if it can be well tested i'd risk it - if not - what you're saying makes sense [21:52:10] we'll deploy in beta before prod if we can [21:52:15] yeah [21:52:21] if it works in beta, i think it will work in prod [21:52:29] i wanted to set up the whole thing on mw-vagrant [21:52:34] buuut, mw-vagrant varnish is varnish 4:( [21:52:37] which means i can't use varnishkafka [21:52:53] ottomata: can we use docker? [21:53:04] oh ja, i mean i could custom set everything up [21:53:04] sure [21:53:17] may i can set up a dev environment [21:53:19] but, i was going to make it all work via a puppet role in mw-vagrant [21:53:34] and then commit it so we could all use it [21:53:40] can't do it in 2 hours, but can do it sometime soon - but beta might be fine? [21:53:44] and then we'd have the same flow we have in beta and prod in mw-vagrant [21:53:51] madhuvishy: i think beta is fine [21:54:14] i'm testing now and eventlogging-devserver basically does the same thing as varnishkafka + eventlogging-processor in one [21:54:21] hmmm - ottomata mw-vagrant seems like a pain to support :/ i want to just not worry about it [21:54:43] hehe, naw its good! i've already set it up for eventlogging-service-(eventbus) [21:54:49] with pip and virtualenv for EL [21:54:50] kidna nice [21:55:01] automatically updated when you run vagrant gitupdate [21:55:05] including dependencies! [21:55:05] :) [21:55:25] its good if we can support it in mw-vagrant, because then we can test it with actual MW extension [21:55:29] like i'm doing now [21:57:57] yeah makes sense [21:58:20] right [21:58:27] the extension [21:58:32] that makes sense [21:59:33] i'm doing it now, and am getting both NavigationTiming and PageEdit events via event.gif, so yay! [21:59:33] and they are validating :) [21:59:45] ottomata: nice! :) [22:00:07] or sorry [22:00:09] PageContentSaveComplete events [22:00:19] ottomata: what is the UA on the event? [22:00:21] and its got my user agent in it, so looks like no worries [22:00:24] sweeeeet [22:00:32] which, makes sense [22:00:32] but [22:00:38] that means [22:00:41]