[00:34:33] (PS1) Madhuvishy: [WIP] Add support to camus to consume schemaID-less avro [analytics/camus] - https://gerrit.wikimedia.org/r/243845 (https://phabricator.wikimedia.org/T113521) [00:38:30] kevinator, I modified the workflow slide a bit, if you want to have a look.. [00:42:19] omg mforns you're still online! [00:42:25] I'll have a look now [00:42:31] :] ok [00:45:48] yeah, it looks good. Did you change anything besides the percentages? [00:49:17] kevinator, yes, I unified adhoc requests and indicent resolution into the same category: unplanned [00:49:47] also added some more description to the third column [00:50:14] specially in the reactive rows, where I tried to explain a bit what we did [00:50:55] awesome! [00:51:05] ok [00:51:07] :] [00:51:50] going to bed! bye a-team, see you tomorrow [00:52:00] good night! [00:52:06] night! [01:26:23] Analytics-Kanban: Investigate sample cube pageview_count vs unsampled log pageview count [5 pts] {hawk} - https://phabricator.wikimedia.org/T108925#1704337 (kevinator) [02:54:13] Analytics-Backlog, MediaWiki-API, Reading-Infrastructure-Team, Research-and-Data, Patch-For-Review: Publish detailed Action API request information to Hadoop - https://phabricator.wikimedia.org/T108618#1704394 (bd808) [03:05:44] Analytics, MediaWiki-API: api.log does not indicate errors and exceptions - https://phabricator.wikimedia.org/T113672#1704407 (bd808) It looks like we will have to do some refactoring of ApiMain to expose the error code in a way that we can get it into the logging context. The best way to do this might b... [03:13:41] Analytics-Backlog, MediaWiki-API, Reading-Infrastructure-Team, Research-and-Data, Patch-For-Review: Publish detailed Action API request information to Hadoop - https://phabricator.wikimedia.org/T108618#1704421 (bd808) The desire for uniform `X-Request-Id` handing in the WMF stack is documented in... [03:13:47] Analytics, Varnish: Connect Hadoop records of the same request coming via different channels - https://phabricator.wikimedia.org/T113817#1704423 (bd808) Some discussion related to this topic starts at T108618#1698925 and continues in T108618#1702008, T108618#1702049, T108618#1702327 and T108618#1703767. [03:27:48] Analytics-Backlog, Discovery, Reading-Infrastructure-Team: Determine proper encoding for structured log data sent to Kafka by MediaWiki - https://phabricator.wikimedia.org/T114733#1704431 (bd808) NEW [03:28:00] Analytics-Backlog, Discovery, Reading-Infrastructure-Team: Determine proper encoding for structured log data sent to Kafka by MediaWiki - https://phabricator.wikimedia.org/T114733#1704445 (bd808) [03:28:03] Analytics-Backlog, MediaWiki-API, Reading-Infrastructure-Team, Research-and-Data, Patch-For-Review: Publish detailed Action API request information to Hadoop - https://phabricator.wikimedia.org/T108618#1525053 (bd808) [03:35:45] Analytics-Backlog, MediaWiki-API, Reading-Infrastructure-Team, Research-and-Data, Patch-For-Review: Publish detailed Action API request information to Hadoop - https://phabricator.wikimedia.org/T108618#1704457 (bd808) The structured data we collect and push into Hadoop should follow the naming co... [03:40:55] Analytics, MediaWiki-extensions-Gadgets, Possible-Tech-Projects: Gadget usage statistics - https://phabricator.wikimedia.org/T21288#1704467 (TasneemLo) As @yuvipanda mentioned, this task would be out of scope for a mentorship program like GSoC and Outreachy. So, removing the #Possible-Tech-Project pr... [03:41:21] Analytics, MediaWiki-extensions-Gadgets: Gadget usage statistics - https://phabricator.wikimedia.org/T21288#1704469 (TasneemLo) [03:51:32] Analytics-Backlog, MediaWiki-API, Reading-Infrastructure-Team, Research-and-Data, Patch-For-Review: Publish detailed Action API request information to Hadoop - https://phabricator.wikimedia.org/T108618#1704481 (bd808) [03:55:51] Analytics-Backlog, MediaWiki-API, Reading-Infrastructure-Team, Research-and-Data, Patch-For-Review: Publish detailed Action API request information to Hadoop - https://phabricator.wikimedia.org/T108618#1704482 (bd808) [03:57:58] Analytics-Backlog, Discovery, Reading-Infrastructure-Team: Determine proper encoding for structured log data sent to Kafka by MediaWiki - https://phabricator.wikimedia.org/T114733#1704483 (EBernhardson) @madhuvishy let me know earlier today on IRC the issues with binary encoding have been worked out. [06:26:20] Analytics-Tech-community-metrics: Port MediaWikiAnalysis to SQLAlchemy - https://phabricator.wikimedia.org/T114437#1704569 (Anmolkalia) Hi, I have a doubt. The dbms we will be using in the backend is till MySQL, right? So, I should continue using MySQL datatypes? Sibyl seems to be using mysql., [08:56:12] Analytics-Tech-community-metrics: Port MediaWikiAnalysis to SQLAlchemy - https://phabricator.wikimedia.org/T114437#1704738 (Anmolkalia) Hi, @jgbarah, this is the what the mapping file looks like. Let me know if this is fine. {F2661511} [09:14:38] Analytics, Services: restbase is not listening on port 7231 on aqs* - https://phabricator.wikimedia.org/T114742#1704767 (akosiaris) NEW [09:15:14] Analytics, Services: restbase is not listening on port 7231 on aqs* - https://phabricator.wikimedia.org/T114742#1704775 (akosiaris) [09:15:16] Analytics-Kanban, netops, operations, Patch-For-Review: Puppetize a server with a role that sets up Cassandra on Analytics machines [13 pts] {slug} - https://phabricator.wikimedia.org/T107056#1704774 (akosiaris) [10:08:08] Analytics-Tech-community-metrics, DevRel-October-2015: Checking code review metrics for a specific repository may take dozens of clicks - https://phabricator.wikimedia.org/T110520#1704844 (Dicortazar) @Aklapper, @Qgil, the issue here is how to have access to the information. Do you think that adding the... [10:13:31] Analytics-Kanban: Investigate sample cube pageview_count vs unsampled log pageview count [13 pts] {hawk} - https://phabricator.wikimedia.org/T108925#1704845 (JAllemandou) [10:15:28] Analytics-Kanban: Investigate sample cube pageview_count vs unsampled log pageview count [13 pts] {hawk} - https://phabricator.wikimedia.org/T108925#1534298 (JAllemandou) Confirmed: files don't contain days, so recombination among files is needed. This done, I got back on the same numbers as the one in DB for... [10:47:22] Analytics-Backlog, Analytics-Wikistats, DevRel-October-2015: Clean the code review queue of analytics/wikistats - https://phabricator.wikimedia.org/T113695#1704897 (Aklapper) @ezachte: Could you answer the last comment please? [11:01:12] Analytics-Tech-community-metrics, DevRel-October-2015: Checking code review metrics for a specific repository may take dozens of clicks - https://phabricator.wikimedia.org/T110520#1704947 (Aklapper) >>! In T110520#1704844, @Dicortazar wrote: > Do you think that adding the three charts info in the reposito... [11:07:05] Analytics, MediaWiki-extensions-Gadgets: Gadget usage statistics - https://phabricator.wikimedia.org/T21288#1704956 (Nemo_bis) Hm? It would be a legit and controversial special page in Gadgets extension, for instance. Probably too small a task but otherwise fine for OPW/GSOC. [11:09:42] Analytics-Tech-community-metrics, DevRel-October-2015: Checking code review metrics for a specific repository may take dozens of clicks - https://phabricator.wikimedia.org/T110520#1704961 (Dicortazar) Ok, let's go for this solution :). I'll keep you updated! [11:36:12] morning [11:38:35] Hi milimetric [11:38:49] hi joal [11:39:08] didn't see otto yesterday, but hopefully today [11:39:32] He was travelling yesterday, it was expected :) [11:39:51] Currently trying to pour some data [11:40:04] But getting issues with building jars :( [11:44:04] anything I can help with joal? [11:44:24] currently blocked by maven not willing to download my dependencies [11:44:38] I think I'll make the jat at home and pyush it [11:44:44] (how bad) [11:47:02] There is not really things I can ask you :) [11:49:39] If you want we can pair-test :) [11:50:17] i can try building on my machine, is that what you mean? [11:52:35] Yeah, did that already, and then scp on stat1002 [11:52:41] I just did that [11:53:08] I don't understand why it didn't want to built on stat1002 [11:53:10] anyway ... [11:54:38] wait are you using our archiva as the package repository? [11:55:30] I removed the restriction on central repo, but it has not worked [11:57:27] it's early morning so I may not be thinking in complete thoughts yet, but I think you have to get all the dependencies registered in our archiva, right? [11:57:36] or did you already do that like three months ago? :) [11:57:49] I didn't updated archiva [11:58:10] But if you remove the restriction on central repo, it works (at least locally) [11:58:27] right, but I don't think stat1002 can reach the internet [11:58:39] I have proxy setup [11:58:47] Mwarf ! [11:58:50] hm... I'm not sure we're supposed to get around it [11:58:56] :) [11:59:16] I'll ask otto when gets in, but problem solved, I scpied the jar [12:01:16] I'll now configure and run an oozie jar [12:25:47] Analytics-Backlog, Analytics-Wikistats, DevRel-October-2015: Clean the code review queue of analytics/wikistats - https://phabricator.wikimedia.org/T113695#1705220 (ezachte) Sorry. @Aklapper I missed your previous comment. And @JanZerebecki answering yours is still on my list. This week I will have t... [12:59:05] Analytics-Tech-community-metrics, DevRel-October-2015: Correct affiliation for code review contributors of the past 30 days - https://phabricator.wikimedia.org/T112527#1705342 (Dicortazar) Ok, for this, there's a bitbucket account with that info daily updated. That would be the JSON file to import and pla... [13:06:47] Analytics-Tech-community-metrics, DevRel-October-2015: Correct affiliation for code review contributors of the past 30 days - https://phabricator.wikimedia.org/T112527#1705359 (Qgil) Yes, because I cannot be the single point of failure. ;) [13:49:18] hi a-team! [13:50:31] Hi mforns :) [14:15:05] Analytics-Backlog, Analytics-Wikistats: Feed Wikistats traffic reports with aggregated hive data - https://phabricator.wikimedia.org/T114379#1705502 (ezachte) On prioritizing: Monthly PageView reports (MPV) is a small task compared to Traffic BreakDown reports (TBD). As MPV do higher-level (more aggreg... [14:24:18] Analytics, Discovery, EventBus, MediaWiki-General-or-Unknown, and 6 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#1705509 (Ottomata) @Joe, there are two parts to this MVP: - Centralized (and CI controlled) schema sharing - An easy way to get valid data into Kafka. With eventl... [14:29:31] Analytics, Discovery, EventBus, MediaWiki-General-or-Unknown, and 6 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#1705514 (Ottomata) @ori I just edited the [[ https://phabricator.wikimedia.org/project/sprint/profile/1474/ | EventBus project description ]] to include a version of... [14:31:43] joal: to the batcave! [14:31:52] milimetric: OMW ! [14:39:17] Analytics, Discovery, EventBus, MediaWiki-General-or-Unknown, and 6 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#1705528 (Joe) >>! In T114443#1703097, @Eevans wrote: >>>! In T114443#1701296, @Joe wrote: >> Apart from the concerns on a practical use case which I agree with, I ha... [14:42:44] sorry [14:45:57] Analytics, Discovery, EventBus, MediaWiki-General-or-Unknown, and 6 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#1705533 (Joe) >>! In T114443#1705509, @Ottomata wrote: > @Joe, there are two parts to this MVP: > > - Centralized (and CI controlled) schema sharing > - An easy wa... [14:50:50] Analytics-Backlog, Analytics-Wikistats: Feed Wikistats traffic reports with aggregated hive data - https://phabricator.wikimedia.org/T114379#1705542 (ezachte) Monthly PageView reports (MPV) on pagecounts-raw instead of pagecounts-all-sites: @Milimetric, no typo. but I can see how this is confusing. Here... [15:01:36] Analytics, Discovery, EventBus, MediaWiki-General-or-Unknown, and 6 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#1705568 (Joe) So, doing some research on the topic, there is already a kafka rest proxy builtin into Kafka: http://docs.confluent.io/1.0/kafka-rest/docs/intro.html... [15:02:30] Analytics, Discovery, EventBus, MediaWiki-General-or-Unknown, and 6 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#1705572 (Ottomata) @joe, yes, please see this ticket: https://phabricator.wikimedia.org/T88459 [15:03:05] Analytics-Backlog, Database: Set up bucketization of editCount fields {tick} - https://phabricator.wikimedia.org/T108856#1705574 (mforns) @jcrespo Here's a TSV with the data you ask for: {F2661997} ``` table_name edit_count_field MobileWebWatching_11761466 event_userEditCount GatherClicks_11639881 event_us... [15:13:12] ottomata: we fixed the binary->hdfs thing with camus [15:13:21] !!! AWESEOM! [15:13:35] i pushed an initial patch [15:13:42] for just the binary one [15:14:22] nuria: ottomata we can hangout and talk about that + schema repo if you want [15:14:37] * madhuvishy is up early [15:15:52] aah nuria is sick! Sorry feel better! [15:19:15] Analytics-Backlog, Database: Set up bucketization of editCount fields {tick} - https://phabricator.wikimedia.org/T108856#1705588 (jcrespo) @mforns Of course there is no reason to do a SUM of that, but can you assure that right now every user is checking not to do sums of values < 90 days? Because if not, c... [15:23:56] Analytics, Discovery, EventBus, MediaWiki-General-or-Unknown, and 6 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#1705598 (Joe) Speaking with @ottomata, it seems that, contrarily to what I understood, this service would proxy connection to kafka only from producers and not from... [15:25:48] Analytics-Backlog, Database: Set up bucketization of editCount fields {tick} - https://phabricator.wikimedia.org/T108856#1705607 (mforns) @jcrespo I understand. Breaking is not good :] But in this case, I would prefer that, because it makes the fact explicit, that we are purging the data after 90 days. We... [15:28:02] Analytics-Backlog, Database: Set up bucketization of editCount fields {tick} - https://phabricator.wikimedia.org/T108856#1705617 (jcrespo) Understood, now that it has been clarified. [15:29:32] Analytics-Kanban: Spike: understand wikistats enough to estimate replacing pageview data source {lama} [8 pts] - https://phabricator.wikimedia.org/T114660#1705622 (Milimetric) [15:33:53] (PS3) Joal: [WIP] Add cassandra load job for pageview API [analytics/refinery] - https://gerrit.wikimedia.org/r/236224 (https://phabricator.wikimedia.org/T108174) [15:41:54] Analytics-Kanban, Patch-For-Review: Enable use of Python 3 in Spark {hawk} [8 pts] - https://phabricator.wikimedia.org/T113419#1705671 (kevinator) Open>Resolved [15:42:05] Analytics-Kanban: Prepare data for Quarterly Review [13 pts] - https://phabricator.wikimedia.org/T113196#1705672 (kevinator) Open>Resolved [15:42:20] Analytics-Kanban: Report on zh wikipedia for Zhou - https://phabricator.wikimedia.org/T114190#1705673 (kevinator) Open>Resolved [15:43:01] Analytics, Privacy, Varnish: Connect Hadoop records of the same request coming via different channels - https://phabricator.wikimedia.org/T113817#1705675 (csteipp) I think this is a good idea for operational logs, but we should not make correlating the webrequest/pageviews dataset with mediawiki logs e... [15:50:10] Analytics-Backlog, Analytics-Wikistats, DevRel-October-2015: Clean the code review queue of analytics/wikistats - https://phabricator.wikimedia.org/T113695#1705688 (Aklapper) a:ezachte @ezachte: Thank you, great to hear this is on the radar! I'll assign this task to you for the time being. [15:52:13] Analytics, Services: restbase is not listening on port 7231 on aqs* - https://phabricator.wikimedia.org/T114742#1705693 (Ottomata) Yeah, it is busted. I was trying to deploy Friday but failed. Analytics and Services folks are going to have to look into it. There is some issue where restbase isn't star... [15:59:47] madhuvishy: https://plus.google.com/hangouts/_/wikimedia.org/camoos [16:15:30] joal: can you join a hangout with me and madhuvishy? [16:15:37] ottomata: sure [16:15:41] k ^^ [16:32:31] Analytics-Backlog, Analytics-Cluster: Remove refinery-hive.jar from hive-site.xml - https://phabricator.wikimedia.org/T114769#1705783 (Ottomata) NEW a:Ottomata [16:39:08] Analytics-EventLogging, Editing-Department, Improving access, Reading Web Planning, discovery-system: Add event_pageId and event_pageTitle to quicksurvey or all schema - https://phabricator.wikimedia.org/T114164#1705802 (atgo) Following up from email, I'm on board to add these columns to the sc... [16:55:49] Analytics, Discovery, EventBus, MediaWiki-General-or-Unknown, and 6 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#1705859 (Eevans) > A message queue is not a database, it's a router. ... Of course, but I drew the analogy because in both cases you have readers and writers of str... [16:58:02] Analytics-EventLogging, Editing-Department, Improving access, Reading Web Planning, discovery-system: Add event_pageId and event_pageTitle to quicksurvey or all schema - https://phabricator.wikimedia.org/T114164#1705870 (Jdlrobson) Currently I'm not clear whether we are proposing doing this for... [17:01:47] Analytics, Services, Patch-For-Review: restbase is not listening on port 7231 on aqs* - https://phabricator.wikimedia.org/T114742#1705893 (mobrovac) a:mobrovac [17:03:03] Analytics-Backlog, MediaWiki-API, Reading-Infrastructure-Team, Research-and-Data, Patch-For-Review: Publish detailed Action API request information to Hadoop - https://phabricator.wikimedia.org/T108618#1705894 (csteipp) > A "session identifier" identifies the same user-session across a series of... [17:05:50] Analytics-Backlog, Database: Set up bucketization of editCount fields {tick} - https://phabricator.wikimedia.org/T108856#1705918 (mforns) Cool! One note: This task can already start. But the auto-purging task T108850 needs to wait until we have notified all the schema owners. [17:07:16] Analytics-EventLogging, Editing-Department, Improving access, Reading Web Planning, discovery-system: Add event_pageId and event_pageTitle to quicksurvey or all schema - https://phabricator.wikimedia.org/T114164#1705943 (Tfinc) That would be useful data to have. @Deskana, are we already trackin... [17:10:59] Analytics-Backlog: Set up auto-purging after 90 days {tick} - https://phabricator.wikimedia.org/T108850#1705980 (mforns) Just wanted to add this note here, too. The activation in production of this auto-purging should wait until we notify all schema owners with the date this will happen. @jcrespo I know it's... [17:15:14] Analytics, Services, Patch-For-Review: restbase is not listening on port 7231 on aqs* - https://phabricator.wikimedia.org/T114742#1706006 (mobrovac) The patch fixes the issue - after running Puppet and restarting RESTBase on `aqs1001` the service is up and running. @Ottomata, please apply the change to... [17:19:39] Analytics, Services: restbase is not listening on port 7231 on aqs* - https://phabricator.wikimedia.org/T114742#1706017 (mobrovac) [17:37:18] Analytics-Backlog, Discovery, Reading-Infrastructure-Team: Determine proper encoding for structured log data sent to Kafka by MediaWiki - https://phabricator.wikimedia.org/T114733#1706046 (Ottomata) > The avro encoder in php doesn't look to directly support json encoding, but i don't think it needs to e... [18:03:32] (PS2) Madhuvishy: [WIP] Add support to camus to consume schemaID-less avro [analytics/camus] - https://gerrit.wikimedia.org/r/243845 (https://phabricator.wikimedia.org/T113521) [18:13:51] mforns: batcave wikistats? [18:13:56] sure [18:19:07] kevinator: batcave? [18:21:38] (PS3) Madhuvishy: [WIP] Add support to camus to consume schemaID-less avro [analytics/camus] - https://gerrit.wikimedia.org/r/243845 (https://phabricator.wikimedia.org/T113521) [18:26:35] joal: for starting a new camus module in refinery, will you move all the properties files elsewhere? [18:26:47] madhuvishy: no :) [18:26:57] leila: I'll be a few minutes late - in a meeting [18:27:25] madhuvishy: I'll add a folder refinery-camus with a pom, being a module of the refinery parent one [18:27:52] joal: ahh refinery-camus - we'll add this in refinery-source [18:27:54] the properties files are camus run-conf, not camus code-related :) [18:28:02] correct madhuvishy :) [18:28:13] i have been thinking refinery, which won't be the right place of course [18:28:18] cool thanks joal :) [18:28:24] np ! [18:30:58] madhuvishy: in refinery source [18:30:58] yeah [18:31:59] milimetric: i'm trying to figure out how to use Promises.all [18:32:11] i don't see any docs in native-promise-only [18:33:21] ottomata: it's just a shim so probably they're relying on the official ES 6 docs: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all [18:34:11] HMMM ok [18:34:13] i think i get it [18:44:13] milimetric: maybe got a sec for me in batcave? i think i understand but i'm doing something wrong... [18:44:27] ottomata: i'd love to but in 15 min. (meeting) [18:44:35] ok cool [18:51:23] Analytics-EventLogging, Editing-Department, Improving access, Reading Web Planning, discovery-system: Add event_pageId and event_pageTitle to quicksurvey or all schema - https://phabricator.wikimedia.org/T114164#1706361 (Deskana) There are two separate questions here. # "Should page ID and pag... [18:51:45] Analytics-Backlog, Database: Delete obsolete schemas {tick} - https://phabricator.wikimedia.org/T108857#1706364 (mforns) @jcrespo I've carefully reviewed this and confirmed with the schema owners that the deletion of those obsolete schemas can be started. So, yes, you can go ahead and delete those :] Tha... [18:51:48] joal: can you batcave for 2 minutes with me? I have a couple silly questions [18:51:57] madhuvishy: sure [18:57:19] ottomata: join me and joal in batcave for a sec? [18:57:29] ja [18:57:56] milimetric: so I guess we are good? :-) [18:58:11] I feel much more informed now, milimetric. [18:58:14] leila: yeah, we just need to implement our own instrumentation [18:58:24] :) [18:58:36] yup. :-) [18:58:38] yeah, interesting getting that perspective, it's not intuitive [18:59:00] right. I think we tend to be over conservative when we sit together. :-) [18:59:26] internet being weird, trying to join! [18:59:37] ottomata: you're in the hangout as far as we can tell [18:59:42] we hear background stuff [19:00:14] really? [19:00:14] huh [19:00:27] i'm going to switch to phone i [19:00:28] internet [19:15:27] joal, are you adding stuff to https://phabricator.wikimedia.org/T108174? [19:15:42] I mean, more changes? :] [19:16:04] no, no changes :) [19:16:14] Changes in the oozie part of it [19:16:33] --> I couldn't test oozie before having the cassandra cluster set yup [19:16:37] ok, because I thought this code was ready and I mentioned it in the quarterly review... [19:16:37] now I can :) [19:16:40] So I update [19:16:52] ok! I see, so, no problem with that! [19:17:10] no need to change the presentation [19:17:15] thx [19:17:19] np :) [19:27:21] hey, what is the server where you can query MySQL EventLogging tables? https://www.mediawiki.org/wiki/Extension:EventLogging/Guide doesn't say [19:28:20] spagewmf, I query from stat1003, but you must belong to the research group [19:28:47] yup, I just found it, https://wikitech.wikimedia.org/wiki/Analytics/EventLogging#Access_data_in_MySQL says stat1003 [19:29:12] spagewmf, let me know if you need help [19:30:06] (PS1) Madhuvishy: [WIP] Add refinery-camus module [analytics/refinery/source] - https://gerrit.wikimedia.org/r/243990 [19:30:50] joal: ^ made an initial patch, there are only class stubs at this point [19:31:01] madhuvishy: cool ! [19:31:10] Will change my patch [19:31:22] I also put in the scalalang dependencies for you [19:33:28] Analytics-Tech-community-metrics, Possible-Tech-Projects, Epic: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1706581 (Oakshweta11) Hi! I would like to this project for Outreachy December 2015-March 2016 I am familiar with data analy... [19:35:43] Analytics, MediaWiki-API: api.log does not indicate errors and exceptions - https://phabricator.wikimedia.org/T113672#1706584 (Anomie) One other thing to keep in mind is that I'm hoping to make it possible for there to be multiple errors as part of {T47843}. It may be that this bug is mistitled and erro... [19:45:19] Analytics-Cluster, Analytics-Kanban: Improve daily webrequest partition report {hawk} [5 pts] - https://phabricator.wikimedia.org/T113255#1706610 (mforns) a:mforns [19:57:46] ottomata: you might have to update camus jars before i can do this [19:57:48] https://github.com/linkedin/camus/blob/master/camus-api/src/main/java/com/linkedin/camus/coders/Message.java [19:57:51] is not in v4 [19:57:57] and is a dependency [19:58:09] joal: ^ [19:58:27] mwarf :( [19:58:28] ottomata: so the promises world took a religious turn, and basically you shouldn't try to force that json-refs code to be synchronous if it's async. You can do it, but shouldn't. The new ES6 stuff is available in Node 0.11.2+ [19:59:18] sorry madhuvishy :( [19:59:19] If you're just playing for now, then just get it to work and don't worry about it too much. But if you get further, let's talk again [20:00:02] joal: no problem! just saying we might have to do it sooner than later [20:00:07] ah hm i'm v0.10.33 [20:04:37] yeah, no cute delay ... await and generators for you then [20:04:43] (PS1) Ottomata: Setting up wmf branch to use archiva.wikimedia.org and maven release plugin [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/244043 [20:05:22] milimetric: i'll just try to do it all async, its fine [20:05:34] yeah, all async is the easiest [20:05:35] milimetric: js style q for you [20:05:37] or all sync [20:05:52] var f = function(var) { ... }; [20:05:53] or [20:06:00] function f(var) { ... } [20:06:07] ? [20:06:13] same thing? [20:06:15] or different? [20:06:18] not same [20:06:21] function f is preferred [20:06:26] Analytics-EventLogging, Editing-Department, Improving access, Reading Web Planning, discovery-system: Add event_pageId and event_pageTitle to quicksurvey or all schema - https://phabricator.wikimedia.org/T114164#1706669 (JKatzWMF) I think this field would be invaluable if it provides a reasonab... [20:06:28] it's hoisted to the top of the scope [20:06:28] ok [20:06:50] and there are other benefits [20:07:30] (basically parse time vs. run time) as usual, for questions like this, epic answer: http://stackoverflow.com/questions/336859/var-functionname-function-vs-function-functionname [20:09:06] (this is the fun one, sorry: http://stackoverflow.com/a/22173438/180664) [20:17:09] Analytics-EventLogging, Editing-Department, Improving access, Reading Web Planning, discovery-system: Add event_pageId and event_pageTitle to quicksurvey or all schema - https://phabricator.wikimedia.org/T114164#1706682 (atgo) Sounds like we have our answer for this specific and updating the ta... [20:17:22] Analytics-EventLogging, Editing-Department, Improving access, Reading Web Planning, discovery-system: Add event_pageId and event_pageTitle to quicksurvey [not all] schema - https://phabricator.wikimedia.org/T114164#1706683 (atgo) [20:18:32] oo milimetric cool! [20:18:32] var b = a.map(n => n * 2); [20:19:21] loooks like scala :p [20:19:33] maybe i'm going to like JS.........:o [20:20:43] es 6 even has "let" instead of "var" which is even cooler [20:20:52] but I'm surprised you're allowed the new lambda syntax [20:21:21] milimetric: You know ottomata has EVERY right ! [20:21:25] :) [20:21:43] good point, #opsenvy [20:24:11] Analytics-Backlog, Developer-Relations, MediaWiki-API, Reading-Infrastructure-Team, and 4 others: Metrics about the use of the Wikimedia web APIs - https://phabricator.wikimedia.org/T102079#1706703 (Spage) [20:24:40] haha, ehHHh? [20:24:46] let? [20:24:49] Analytics, MediaWiki-API: api.log does not indicate errors and exceptions - https://phabricator.wikimedia.org/T113672#1672882 (Spage) >>! In T113672#1691943, @Tgr wrote: > @Spage if you think error codes should be logged please update T102079 so there is a single canonical description of what needs to be... [20:36:50] milimetric, you still the Analytics team lead? [20:37:15] lead of being in meetings! 30 min. Ironholds :) [20:37:28] totally! :D [20:58:08] (PS2) Madhuvishy: [WIP] Add refinery-camus module [analytics/refinery/source] - https://gerrit.wikimedia.org/r/243990 [20:58:30] kevinator: there's an event on 5th floor lounge [20:58:54] yeah, I'm about to move out [20:59:24] okay, i'm on 3rd, we can chat on the couches here? [20:59:44] Ironholds: k [21:00:05] I'm not sure about this "lead" title, but what's up [21:00:43] madhuvishy: where are you? [21:02:28] madhuvishy: omw [21:03:01] (CR) Ottomata: "Mostly working, need to be on non-phone internet to upload, will try more tomorrow." [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/244043 (owner: Ottomata) [21:06:37] Hey HaeB [21:06:42] Before I leave [21:07:16] HAve you happy enough with the last comment I made on https://phabricator.wikimedia.org/T108925 ? [21:07:22] HaeB: --^ [21:08:18] hi joal [21:08:47] Please excuse my english HaeB, it's late in here :) [21:09:49] you mean "This done, I got back on the same numbers as the one in DB for the first few days." ? [21:10:29] (i've been following along and been impressed what has been unearthed by the investigation in recent days.. but not looked in detail) [21:10:39] Ironholds: my irc client died, what's up? [21:11:53] HaeB: I meant the things uncovered and the final diffs I have [21:13:39] well, :) I gotta run in a bit, so Ironholds: feel free to send an email and cc any internal / external list you think, or just catch me here later. [21:13:44] joal: impressive, yes.... could we compile the findings into a table listing all the differences? [21:13:52] i lost track a bit [21:14:24] HaeB: I can explain the various diffs I have found, but re-running the stuff over data is fairly costly [21:14:40] e.g. with two columns - pageviews05/R on the left, pageview_hourly/Java on the right [21:15:22] well for now i just mean having rows e.g. "action=edit counted? pageviews05: yes | pageview_hourly: no [21:15:45] a text table not an actual spreadsheet [21:15:52] just collect all the findings in one place [21:17:12] once we have that, we could also add the change dates you found [21:17:23] and start estimating the difference each causes [21:17:25] HaeB: not an issue: everything is in the tickets. wiki page, new comment with recap ? [21:17:33] Analytics-EventLogging, Performance-Team, Patch-For-Review: Support kafka in eventlogging client on terbium - https://phabricator.wikimedia.org/T112660#1706930 (Krinkle) ```lang=python import sys reload(sys) sys.path.append('/srv/deployment/eventlogging/EventLogging/server') sys.setdefaultencoding('utf... [21:17:51] hmm perhaps a wiki page is best, yes [21:17:57] i'm happy to contribute by editing it too [21:18:14] alternatively we could edit the task description [21:18:31] Ok I'll do that. As per estimating differences each cause, it's another day of work [21:19:16] how about a subpage of https://meta.wikimedia.org/wiki/Research:Page_view ? [21:19:48] understood.... we really need to know how reliable our historical comparisons are [21:20:01] it came up in the Reading quarterly review today, predictably [21:20:20] with terry and lila and geoff asking related questions [21:20:33] Analytics, Security: Unauthenticated JMX Connections Allow Remote Code Execution - https://phabricator.wikimedia.org/T114812#1706940 (dpatrick) NEW a:Ottomata [21:20:37] ..what we can say about seasonality [21:21:00] appreciate all the work you've put into this already, and it looks like it's bearing fruit [22:58:01] Analytics, RESTBase: configure RESTBase pageview proxy to Analytics' cluster - https://phabricator.wikimedia.org/T114830#1707333 (Eevans) [23:05:27] Analytics-Tech-community-metrics, Possible-Tech-Projects, Epic: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1707346 (jgbarah) >>! In T60585#1677452, @Simmimourya3107 wrote: > Is there any chance that this project gets proposed for... [23:08:07] Analytics-Tech-community-metrics, Possible-Tech-Projects, Epic: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1707347 (jgbarah) >>! In T60585#1706581, @Oakshweta11 wrote: > Hi! I would like to this project for Outreachy December 2015... [23:09:39] Analytics-Tech-community-metrics, Possible-Tech-Projects, Epic: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1707348 (jgbarah) [23:11:16] Analytics-Tech-community-metrics, Possible-Tech-Projects, Epic: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1077886 (jgbarah) Proposal for GSoC 2015, by Sarvesh Gupta, moved to this comment, to make the description more clear, whil... [23:28:20] (PS3) Madhuvishy: [WIP] Add refinery-camus module [analytics/refinery/source] - https://gerrit.wikimedia.org/r/243990 [23:35:05] Analytics, RESTBase: configure RESTBase pageview proxy to Analytics' cluster - https://phabricator.wikimedia.org/T114830#1707458 (mobrovac) These endpoints are the ones exposed by the AQS RESTBase instance. We yet have to bike-shed on the exact public layout. Yay bike-shedding :) [23:44:19] Analytics-Tech-community-metrics, Possible-Tech-Projects, Epic: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1707488 (jgbarah)