[03:35:18] Analytics-Kanban, Research-and-Data, Patch-For-Review: Remove Client IP from Eventlogging capsule {mole} - https://phabricator.wikimedia.org/T128407#2098121 (Nuria) I am for deploying, let's check at every step that things are proceeding as they should. [07:33:02] (PS1) Madhuvishy: Implement the Wikimedia User Agent policy in setting agent type [analytics/refinery/source] - https://gerrit.wikimedia.org/r/275760 (https://phabricator.wikimedia.org/T125731) [09:23:16] (CR) Joal: "Fully tested on cluster (daily and monthly), we indeed need a cron to export from hdfs to an outside-accessible place." [analytics/refinery] - https://gerrit.wikimedia.org/r/274187 (https://phabricator.wikimedia.org/T126767) (owner: Joal) [10:59:54] hi a-team! [11:00:45] Hi mforns :) [11:08:45] helloooo [11:08:50] Hi elukey :) [11:36:08] helo! [12:06:12] Varnish-kafka reading and parsing from varnish 4 shm seems working, I still need to port some tags but the bulk of the work seems done [12:06:20] fingers crossed [12:10:23] brb lunch! [12:11:38] cool :] [12:53:33] (CR) Mforns: [C: 1] "LGTM!" (2 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/275760 (https://phabricator.wikimedia.org/T125731) (owner: Madhuvishy) [13:25:04] (CR) Joal: [C: -1] "minor comments on test." (2 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/275760 (https://phabricator.wikimedia.org/T125731) (owner: Madhuvishy) [13:26:54] Analytics-Kanban: Communicate the WikimediaBot convention {hawk} - https://phabricator.wikimedia.org/T108599#2098665 (mforns) Thanks @bd808 I'll write the email to wikitech-l and let you and @Anomie know for the api-announce email and pinging high volume api users. [13:59:18] o/ joal and milimetric [13:59:26] Hi halfak|Mobile_ [13:59:50] I don't have anything for the live sync meeting this morning and I know schana is busy with other stuff. [13:59:56] I think we should cancel [14:00:11] halfak|Mobile_: only update I have is a working download script :) [14:00:29] Last pull request has been tested i [14:00:33] Woot. Will confirm with a test. Is there a PR ? [14:00:34] on frwiki [14:00:55] Cool. I'll kick off the next step with frwiki then. :) [14:01:13] And review the PR : ) [14:03:52] halfak|Mobile_: frwiki files are in /user/jallemandou/dwd/frwiki-20160111/xmlbz2 [14:27:47] GOOD MORNING [14:28:24] Hi ottomata :) [14:29:46] ottomata: playing with Spark, I realized out conf is not set to provide dynamic allocation ! [14:31:08] indeed! [14:31:20] ottomata: I thought it was ready :) [14:31:21] but, i thought it was just enabled by default now [14:31:23] do we have it disabled? [14:31:42] ottomata: I think it's enabled by default using Cloudera Management stuff to deploy [14:32:20] OH [14:32:20] hm [14:32:22] ok [14:32:30] no i hadn't changed configs as part of the upgrade [14:32:33] i changed as little as possible [14:32:41] so i didn't investigate that [14:32:42] makes sense :) [14:32:51] I thought it was done by default too [14:33:07] I looked at config changes needed --> :( [14:34:33] ok! [14:43:09] Analytics, Analytics-Cluster: Configure Spark YARN Dynamic Resource Allocation - https://phabricator.wikimedia.org/T101343#2098830 (Ottomata) [14:43:23] Analytics, Analytics-Cluster: Configure Spark YARN Dynamic Resource Allocation - https://phabricator.wikimedia.org/T101343#1336227 (Ottomata) Now that we've installed CDH 5.5, maybe this will be easier! We should try it! [14:53:01] Analytics, Analytics-Cluster, Deployment-Systems, scap, Scap3 (scap3-adoption): Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2098872 (Ottomata) Not sure if this blocks T109926; analytics/refinery is not a Services team owned thang. But, we should still do... [14:53:44] Analytics-Kanban, Research-and-Data, Patch-For-Review: Remove Client IP from Eventlogging capsule {mole} - https://phabricator.wikimedia.org/T128407#2098874 (Ottomata) Yeah! let’s do it! After standup? [15:05:42] Analytics, Analytics-Cluster: Configure Spark YARN Dynamic Resource Allocation - https://phabricator.wikimedia.org/T101343#2098946 (JAllemandou) Description of needed config is described [[ http://spark.apache.org/docs/1.5.0/job-scheduling.html#dynamic-resource-allocation | here ]] [15:16:59] ottomata: first draft of the vk parsing completed! [15:17:00] {"hostname":"cp1052.eqiad.wmnet","sequence":3456,"dt":"2016-03-08T15:05:39","D":"199","T":"0.000199","time_firstbyte":0.000044,"ip":"-","cache_status":"HIT","http_status":"200","response_size":11109,"http_method":"GET","uri_host":"localhost:6081","uri_path":"/","uri_query":"?test=123","content_type":"text/html","referer":"wikimedia.org","x_forwarded_for":"::1","user_agent":"curl/7.38.0","accept [15:17:07] _language":"-","x_analytics":"-","range":"-","x_cache":"-"} [15:17:11] OOo nice [15:17:31] elukey: you have code up somewhere? Should I look or wait? [15:17:56] I need to solve two TODOs then you'll be in the code review to start checking the new code [15:17:59] sounds good? [15:19:35] and whenever you have time I'd need to ask you something about https://github.com/wikimedia/varnishkafka/blob/master/varnishkafka.c#L2156 [15:23:11] ja [15:23:14] wassup? [15:23:55] elukey: ^ [15:26:26] so from my understanding the dispatch is called after each Varnish tag in the while, and rd_kafka_poll is called each time [15:26:51] VUT is a bit different and it handles this step by itself (the while I mean), you only need to set the callback function [15:27:19] very handy but in this way rd_kafka_poll is called only in the end of the output function called once the log is ready [15:27:32] not sure why it was put in the code multiple times [15:27:45] (with 0 is non blocking) [15:27:57] any thoughts? [15:30:24] (ref: https://github.com/varnish/Varnish-Cache/blob/4.1/lib/libvarnishtools/vut.c#L343) [15:31:28] elukey: i think that will be fine. i think rk (the rdkafka handle) is configured to do some things, like do the deliever report callback to print rdkafka stats [15:31:36] and mayyybe also handle any metadata changes? not sure about that [15:31:44] so, magnus is just calling it very frequently [15:31:48] since most of the time it will do nothing [15:32:20] does VSL_Dispatch block if there is nothign new in the VSL? [15:32:25] or, does VUT do that? [15:33:14] either way, it will probably be fine. I think calling rd_kafka_poll once per log line (transaction?) will be almost just as good as calling it for every tag [15:34:33] super [15:34:46] the last thing to sort out are the signal handles [15:34:50] *handlers [15:34:57] elukey: aye, i mean, i'm about 90% sure about what I just said [15:35:02] i might be wrong! [15:35:03] :p [15:35:09] you are never wrong :P [15:35:16] magnus is actually really responsive about this kinda stuff [15:35:23] so if we get confused on things, we can reach out to him [15:35:32] he's good about responding to questions on librdkafka github [15:35:39] yes I was planning to ask his opinion after the first round of reviewes [15:35:41] cool [15:35:43] yeah that sounds good [15:36:34] the only thing that I am still not sure is VUT, because I needed to import some other C files to make it work. Upstream will bundle the lib eventually as standalone but I might need to refactor the code to use the API directly [15:36:54] even if having VUT doing all the checks etc.. is super good [15:37:06] to use the API directly? [15:37:23] meaning, the official varnish API, and not VUT? [15:37:24] yes I meant the Varnish SHM/Logging lib [15:37:38] yes exactly, basically what VUT uses [15:38:23] aye, i guess by manually adding the code we are a little safer though, in case VUT interface changes. it uses V api and that prob won't change [15:38:31] at least not unexpectedly [15:39:00] yes same thought that I had.. [15:39:07] aye cool [15:45:10] Analytics-Tech-community-metrics, Developer-Relations, DevRel-March-2016, Gerrit-Migration: Make MetricsGrimoire/korma support gathering Code Review statistics from Phabricator's Differential - https://phabricator.wikimedia.org/T118753#2099080 (Aklapper) @Lcanasdiaz: Not yet. :( I'm waiting for an u... [15:46:45] Analytics-Tech-community-metrics, DevRel-March-2016: gerrit_review_queue.html: List of Repositories has not been updated recently - https://phabricator.wikimedia.org/T128170#2099085 (Aklapper) Open>Resolved @Lcanasdiaz: It is! Thank you! (Now hoping for a fix for top-contributors too, see T126971#20... [15:49:17] Analytics-Tech-community-metrics, Developer-Relations, DevRel-March-2016, Gerrit-Migration: Make MetricsGrimoire/korma support gathering Code Review statistics from Phabricator's Differential - https://phabricator.wikimedia.org/T118753#2099132 (Qgil) @Lcanasdiaz could you estimate the cost of this p... [16:03:37] ottomata: how about enabling ferm on eventlog1001 tomorrow? (when you get online/early Euro evening) [16:03:50] moritzm: that sounds fine [16:04:01] ok, great [16:04:03] will ping you in the morn when I get online....if I forget then PING ME! [16:04:09] sure :-) [16:21:13] mforns: hi [16:21:23] you wanna chat a bit before standup? [16:21:32] hi milimetric I'm in my 1x1.. [16:21:41] sorry, should've checked [16:22:26] np milimetric [16:30:08] milimetric, I'll be working until the end of staff meeting today. do you have a moment until there to discuss dashiki? [16:30:15] a-team: standduppp [16:33:29] Analytics-Kanban, Research-and-Data, Patch-For-Review: Remove Client IP from Eventlogging capsule {mole} - https://phabricator.wikimedia.org/T128407#2099287 (leila) [16:36:26] ottomata: hi! can I output test data to a kafka test topic? I'm playing with the python version of varnishkafka and would like to see evaluate performance a bit [16:36:44] Analytics-Kanban: Invalid page titles are appearing in the top_articles data - https://phabricator.wikimedia.org/T117346#2099292 (JAllemandou) a:JAllemandou [16:37:11] ema ja, there should be one called 'test', just use that [16:37:21] ottomata: awesome, thanks [16:37:47] ema: kafka is also set up in beta labs, and also is useable in mw-vagrant too [16:37:51] but, ja, in prod is fine [16:38:05] ottomata: right, I wanted to see what happens with prod data [16:38:10] k [16:44:13] ema: hmm, what python kafka client are you using, just curious? [16:44:27] ottomata: rd_kafka [16:44:36] https://bitbucket.org/yungchin/python-librdkafka [16:45:06] whoa, huh, didn't know that existed [16:45:15] pykafka has librdkafka support too [16:46:27] ottomata: the bug that I couldn't figure out was just that the stats user was denied access to run Hive queries [16:47:02] milimetric: that makes sense, stats does not have an account on analytics1001, and thus can't access (private) data in hdfs [16:47:14] yep, is it ok to add that? [16:47:17] Um [16:47:17] hm [16:47:20] need to think about it [16:47:23] k [16:47:26] does it have to run as stats user? [16:47:51] well, either that or the stats user has to have the right to pass "-n <>" to beeline [16:47:58] hive-stats? :) [16:48:15] i'm not sure that would work well either [16:48:27] it should be a hadoop account [16:48:29] hdfs not good? [16:48:40] can it just run in the hdfs user cron? [16:54:37] it can run as hdfs, but then all the queries and output TSVs would be owned by hdfs too, right? [16:55:57] Analytics-Cluster, Operations, hardware-requests: eqiad: New Hive / Oozie server node in eqiad Analytics VLAN - https://phabricator.wikimedia.org/T124945#2099401 (Ottomata) Hm, @robh, we could alternatively use WMF4541 for this, no? [16:56:00] (CR) Nuria: "I think there is nothing for us to announce to e-mail lists about this change, correct?" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/275760 (https://phabricator.wikimedia.org/T125731) (owner: Madhuvishy) [17:19:44] (PS2) Madhuvishy: Implement the Wikimedia User Agent policy in setting agent type [analytics/refinery/source] - https://gerrit.wikimedia.org/r/275760 (https://phabricator.wikimedia.org/T125731) [17:34:18] ottomata: can I keep on throwing stuff at the test topic as much as I please? :) [17:37:07] yup, i doubt you will hurt it :) [17:37:20] if you start doing more than 10K / sec, maybe we should talk [17:43:12] ottomata: nope, I think I'll stay below 5K / sec :) [17:46:56] mmmk [17:52:42] ema: don't break kafka please [17:52:44] :D [17:54:58] a-team, staff meeting is not in batcave ... Where do we go ? [17:59:48] just use batcave? [18:00:26] a-team: staff in batcave [18:01:55] ottomata: you are kaput in hangout [18:04:31] AH [18:04:33] didn't even realize [18:17:43] Analytics: Write a script to automatically run dependent jobs in cluster - https://phabricator.wikimedia.org/T115985#2099780 (Nuria) [18:19:30] Analytics: Make deployment process to the cluster easier, more streamlined - https://phabricator.wikimedia.org/T129253#2099799 (Nuria) [18:25:31] Analytics-Kanban, Patch-For-Review: Special projects not showing up on dashiki after Pageview API migration - https://phabricator.wikimedia.org/T129131#2099837 (Milimetric) [18:36:25] nuria: sorry, I was hoping to join the latter half of the analytics mtng [18:36:33] we are wrapping up [18:36:38] but the meeting about switchover plans for the job queue is running long [18:36:38] but we can talk 1 to 1? [18:36:51] ok, let's chat once we're both out of our respective meetings [18:37:05] k, let me know [18:38:00] milimetric, do you want to continue our chat for 5 mins? [18:43:18] joal: I updated my regex patch [18:43:47] I also switched to using \S for non-space characters instead of \w for word characters since it's a wider set [18:44:41] Freat Madhu ! [18:44:47] Great sorry [18:44:49] :S [18:45:55] madhuvishy: I merge that patch, will then deploy refinery-source then refinery tomorrow (ok ottmata?) [18:46:15] joal: okay cool [18:46:42] nuria: Shall we merge the last_access archiving stuff ? [18:47:04] joal: also I found this really cool tool - prestodb.io [18:47:24] madhuvishy: Have you tried it ? [18:47:24] it is an sql engine over distributed data stores [18:47:27] nooo [18:47:28] i want to [18:47:33] :) [18:47:36] it can query kafka, hive and mysql [18:47:54] and claims to be able to join across them [18:48:02] it will solve all our EL problems if it works! [18:48:09] :) [18:48:49] and it supports live querying data from kafka [18:49:16] if we add a statsd/graphite connector - so much awesomeness [18:49:25] madhuvishy: :) [18:49:40] madhuvishy: spark claims to do the same :) [18:49:49] joal: yes true [18:50:28] madhuvishy: presto is widely used, but I think it's difficult to have it working well under resource constreaints [18:50:57] madhuvishy: As for spark, those frameworks are thought in a world were hardware is cheap ;) [18:51:01] joal: aah yes I thought that could be the case - but specifically for the join EL data across mysql and hive [18:51:09] i'm super interested [18:51:11] right [18:51:23] madhuvishy: maybe try in spark first (already installed) [18:51:28] ;) [18:51:38] joal: but you have to write code for spark [18:51:49] Mostly SQL madhuvishy [18:51:50] this is more for analysts to be able to do their queries no? [18:52:10] right but spark is definitely more complicated [18:52:11] spark will need some config (code base I think), but after that, can run SQL [18:53:11] this has a much lower bar - and works like a sql interface (i'd use spark - but i wouldn't expect analysts to) [18:53:38] madhuvishy: We have not yet tried sparkSQL interface, maybe it's good enough? [18:53:57] joal: ah yeah I haven't tried either - will do sometime [18:54:07] a-team: talk with you tomorrow! byyeee [18:54:13] Bye elukey [18:54:59] madhuvishy: And by no mean I am against presto --> Widely used at facebook and other places, good support, sounds awesome (I tried it two years ago, when it started) [18:55:02] B [18:55:17] But I think the cost to enter before having it really usefull is not cheap [18:55:21] joal: yeah I get it [18:55:40] Maybe I'm wrong madhuvishy [18:55:51] Worth discussing :) [18:55:52] yup it might be worth trying [18:56:45] (CR) Joal: [C: 2] "LGTM ! Approved" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/275760 (https://phabricator.wikimedia.org/T125731) (owner: Madhuvishy) [18:59:34] Analytics-Kanban, Patch-For-Review: Clean up Client IP and hashing related code on Eventlogging {oryx} {mole} - https://phabricator.wikimedia.org/T128408#2100022 (madhuvishy) [18:59:36] Analytics-Kanban, Research-and-Data, Patch-For-Review: Remove Client IP from Eventlogging capsule {mole} - https://phabricator.wikimedia.org/T128407#2100021 (madhuvishy) [18:59:51] Analytics-Kanban, Research-and-Data, Patch-For-Review: Remove Client IP from Eventlogging capsule {mole} - https://phabricator.wikimedia.org/T128407#2073499 (madhuvishy) [18:59:53] Analytics-Kanban, Patch-For-Review: Clean up Client IP and hashing related code on Eventlogging {oryx} {mole} - https://phabricator.wikimedia.org/T128408#2073520 (madhuvishy) [19:09:38] Joal: yes, i can merge the last access stuff, should we open a different ticket for the cron work? [19:10:54] (CR) Nuria: [C: 2 V: 2] "Merging, remaining work is to add a cron that moves this files to a suitable location for them to be downloaded." [analytics/refinery] - https://gerrit.wikimedia.org/r/274187 (https://phabricator.wikimedia.org/T126767) (owner: Joal) [19:11:15] Analytics: Make deployment process to the cluster easier, more streamlined {hawk} - https://phabricator.wikimedia.org/T129253#2100077 (Nuria) [19:14:45] Analytics: Write a script to automatically run dependent jobs in cluster {hawk} - https://phabricator.wikimedia.org/T115985#2100082 (Nuria) [19:15:42] Analytics: Streamline backfilling of jobs in cluster {hawk} - https://phabricator.wikimedia.org/T115985#1737628 (Nuria) [19:18:15] hiii [19:19:01] nuria: are you around? [19:19:07] madhuvishy: eitherpad thing? [19:19:09] w plan? [19:19:10] yes [19:19:11] link please? [19:19:22] https://etherpad.wikimedia.org/p/el-clientips-drop [19:19:56] i'm on batcave [19:20:06] need to check EL server side is coming in Ok [19:20:10] haven't done that yet [19:20:36] omw to batcave give me a sec [19:20:39] cc madhuvishy [19:20:58] ori: fyi we're going ahead with dropping client ips [19:25:31] I know, I've been keeping up with the e-mails. So happy about that -- and grateful. Thanks for working on it. [19:35:11] madhuvishy: MediaViewer_10867062 [19:57:30] (PS2) Joal: Remove mobile/zero from dataset_dump script [analytics/refinery] - https://gerrit.wikimedia.org/r/275392 [20:00:09] (PS1) Joal: Correct Pageview whitelist check HQL [analytics/refinery] - https://gerrit.wikimedia.org/r/275897 [20:00:23] milimetric: here? [20:00:36] hi joal [20:00:40] Hi :) [20:00:55] I am wondering about the removal of the "-" pageview [20:01:07] Shall I do it in top only, or in pageview global ? [20:01:37] hmmm [20:01:43] I think in top only [20:01:57] Analytics-Cluster: Weird encoding problems in early Feb, too - https://phabricator.wikimedia.org/T128814#2086885 (JAllemandou) Hey Oliver, The table you are using in the example query is pagecounts_all_site - In this table the page titles are not converted :) [20:02:04] reason is, that's the only place where it's forced onto the user [20:02:07] Analytics-Cluster: Weird encoding problems in early Feb, too - https://phabricator.wikimedia.org/T128814#2100369 (JAllemandou) Open>declined [20:02:27] in the per-article endpoint, they have to request it, and I guess it's useful to some extent - the number of views to "unknown" [20:02:35] maybe someone could track that so we can get it down to 0 [20:02:36] Analytics-Cluster: Weird encoding problems in early Feb, too - https://phabricator.wikimedia.org/T128814#2100372 (Ironholds) *headdesks* As always, you save my bacon. Thanks ;). (What should I be using and when is that known to have issues?) [20:02:39] joal: ^ [20:02:44] makes sense, that's what I would have done, but I'd rather have your opinion before :) [20:02:48] ok, cool :) [20:03:01] Thanks ! [20:12:07] (PS1) Joal: Remove "-" page from top pageview in API. [analytics/refinery] - https://gerrit.wikimedia.org/r/275899 (https://phabricator.wikimedia.org/T117346) [20:12:23] (CR) Ottomata: [C: 2 V: 2] Remove mobile/zero from dataset_dump script [analytics/refinery] - https://gerrit.wikimedia.org/r/275392 (owner: Joal) [20:13:00] (CR) Ottomata: [C: 1] Correct Pageview whitelist check HQL [analytics/refinery] - https://gerrit.wikimedia.org/r/275897 (owner: Joal) [20:13:30] thanks for the reviews ottomata, I'll deploy tomorrow :) [20:13:54] (CR) Ottomata: [C: 1] "+1, but you should probably add a comment in the SQL as to what this is about, and perhaps link to T117346 there." [analytics/refinery] - https://gerrit.wikimedia.org/r/275899 (https://phabricator.wikimedia.org/T117346) (owner: Joal) [20:14:12] :) [20:17:02] (PS2) Joal: Remove "-" page from top pageview in API. [analytics/refinery] - https://gerrit.wikimedia.org/r/275899 (https://phabricator.wikimedia.org/T117346) [20:23:08] nuria: if you have a minute to merge those two, that'd be awesome :) If not, I'll self merge tomorrow before deploy (https://gerrit.wikimedia.org/r/#/c/275897/1, https://gerrit.wikimedia.org/r/#/c/275899/) [20:23:43] (PS3) Joal: Remove "-" page from top pageview in API. [analytics/refinery] - https://gerrit.wikimedia.org/r/275899 (https://phabricator.wikimedia.org/T117346) [20:24:28] (PS2) Joal: Correct Pageview whitelist check HQL [analytics/refinery] - https://gerrit.wikimedia.org/r/275897 [20:25:43] ottomata: in refinery-source there are changes in the submodule (git status tells me) [20:25:55] ottomata: should I add and commit them ? [20:26:08] ottomata: I don't know how to handle that :( [20:27:11] what submodule? [20:28:50] ottomata: modified: refinery-camus/src/main/resources/schema_repo (new commits) [20:31:19] ohhh of course hm [20:31:23] looking [20:31:44] oh, local joal? [20:31:49] after pulling [20:31:50] do [20:31:52] git submodule udpate [20:32:18] YAY ! [20:32:31] * joal bow to master gittomata [20:38:13] (PS1) Joal: Update changelog.md before deploying v0.0.27 [analytics/refinery/source] - https://gerrit.wikimedia.org/r/275907 [20:38:26] ottomata: last bother for tonight --^ [20:39:01] ottomata: v0.0.27 is a fatty one :) [20:39:37] joal: this also updates camus with our change? [20:40:01] ottomata: you did that, didn't you ? [20:40:13] i released a new camus jar in archiav [20:40:16] haven't updated refinery jars [20:40:28] I mean, I need to download refiniery jars + camus in refinery [20:40:32] doh [20:40:33] sorry [20:40:35] this is source [20:40:38] carry on! [20:40:39] np :) [20:40:45] (CR) Ottomata: [C: 2 V: 2] Update changelog.md before deploying v0.0.27 [analytics/refinery/source] - https://gerrit.wikimedia.org/r/275907 (owner: Joal) [20:41:00] (CR) Nuria: [C: 2 V: 2] "Nice, thanks for taking care of this one." [analytics/refinery] - https://gerrit.wikimedia.org/r/275897 (owner: Joal) [20:41:35] (CR) Ottomata: [C: 1] Remove "-" page from top pageview in API. [analytics/refinery] - https://gerrit.wikimedia.org/r/275899 (https://phabricator.wikimedia.org/T117346) (owner: Joal) [20:43:22] a-team, I'm off for tonight, deploying a big bunch of stuff tomorrow morning ! [20:43:27] Have a good evening all :) [20:43:33] (CR) Nuria: "Looks good, this should be documented somewhere on pageview API docs." [analytics/refinery] - https://gerrit.wikimedia.org/r/275899 (https://phabricator.wikimedia.org/T117346) (owner: Joal) [20:44:06] good evening Joal [20:46:44] (CR) Nuria: [C: 1 V: 1] Strip out www. in front of project names [analytics/aqs] - https://gerrit.wikimedia.org/r/275681 (https://phabricator.wikimedia.org/T127030) (owner: Milimetric) [20:48:47] (CR) Joal: "nuria: Done." [analytics/refinery] - https://gerrit.wikimedia.org/r/275899 (https://phabricator.wikimedia.org/T117346) (owner: Joal) [20:55:47] (CR) Nuria: [C: 2 V: 2] Remove "-" page from top pageview in API. [analytics/refinery] - https://gerrit.wikimedia.org/r/275899 (https://phabricator.wikimedia.org/T117346) (owner: Joal) [20:56:20] (CR) Nuria: "It is probably worth it to document this change here: https://wikitech.wikimedia.org/wiki/Analytics/PageviewAPI#Changelog" [analytics/aqs] - https://gerrit.wikimedia.org/r/275681 (https://phabricator.wikimedia.org/T127030) (owner: Milimetric) [21:02:26] Analytics-Kanban: Move vital signs to its own instance {crow} - https://phabricator.wikimedia.org/T123944#2100632 (Nuria) [21:02:28] Analytics-Kanban: Remove cron on wikimetrics instance that updates vital signs - https://phabricator.wikimedia.org/T125751#2100631 (Nuria) [21:02:30] Analytics-Kanban: Make Dashiki get pageview data from pageview API {melc} - https://phabricator.wikimedia.org/T124063#2100630 (Nuria) Open>Resolved [21:02:42] Analytics-Kanban, Patch-For-Review: Special projects not showing up on dashiki after Pageview API migration - https://phabricator.wikimedia.org/T129131#2100633 (Nuria) Open>Resolved [21:03:05] Analytics-Kanban: Remove cron on wikimetrics instance that updates vital signs - https://phabricator.wikimedia.org/T125751#1996333 (Nuria) Open>Resolved [21:06:38] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Set up Webrequest -> kafka flow in beta. - https://phabricator.wikimedia.org/T127369#2100653 (Nuria) Open>Resolved [21:06:50] Analytics-Kanban, Patch-For-Review: Corrext camus partition checker to not fail globally on one topic error - https://phabricator.wikimedia.org/T127909#2100654 (Nuria) Open>Resolved [21:07:16] Analytics, Analytics-Kanban, Operations, Patch-For-Review: Increase HADOOP_HEAPSIZE (-Xmx) for hive-server2 - https://phabricator.wikimedia.org/T76343#2100658 (Nuria) [21:07:18] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Create new Hive / Oozie server from old analytics Dell {hawk} - https://phabricator.wikimedia.org/T110090#2100657 (Nuria) Open>Resolved [21:13:16] Analytics-Kanban, Patch-For-Review: Update AQS yaml format to match new convention {melc} - https://phabricator.wikimedia.org/T127323#2100680 (Nuria) Open>Resolved [21:13:57] Analytics-Kanban, Patch-For-Review: Create reportupdater browser reports that query hive's browser_general table {lama} - https://phabricator.wikimedia.org/T127326#2100681 (Nuria) Open>Resolved [21:15:07] Analytics-Kanban, Patch-For-Review: Migrate limn-mobile-data/reportupdater reports to use standalone reportupdater - https://phabricator.wikimedia.org/T128375#2100683 (Nuria) Open>Resolved [21:16:00] Analytics, Beta-Cluster-Infrastructure, Services, scap, and 3 others: Set up AQS in Beta - https://phabricator.wikimedia.org/T116206#2100686 (Nuria) [21:16:03] Analytics, Analytics-Kanban, HyperSwitch, RESTBase, and 2 others: Separate AQS off of RESTBase - https://phabricator.wikimedia.org/T126294#2100685 (Nuria) Open>Resolved [21:32:37] (PS4) EBernhardson: Update camus to support reading avro schemas from an avro protocol [analytics/refinery/source] - https://gerrit.wikimedia.org/r/274307 (https://phabricator.wikimedia.org/T128530) [21:33:05] (CR) EBernhardson: Update camus to support reading avro schemas from an avro protocol (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/274307 (https://phabricator.wikimedia.org/T128530) (owner: EBernhardson) [21:35:08] (CR) Ottomata: "Oh awesome!" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/274307 (https://phabricator.wikimedia.org/T128530) (owner: EBernhardson) [21:37:46] (CR) Ottomata: "This doesn't look like it reads the idl format, does it? Usually idl files are suffixed with .idl, no?" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/274307 (https://phabricator.wikimedia.org/T128530) (owner: EBernhardson) [21:40:33] (CR) EBernhardson: "The actual idl has to be compiled into a json object, although perhaps it would be possible to make this read the idl directly. The thing " [analytics/refinery/source] - https://gerrit.wikimedia.org/r/274307 (https://phabricator.wikimedia.org/T128530) (owner: EBernhardson) [21:40:48] (PS2) Nuria: [WIP] Requests that come tagged with pageview=1 in x-analytics header are considered pageviews [analytics/refinery/source] - https://gerrit.wikimedia.org/r/274644 (https://phabricator.wikimedia.org/T128612) (owner: BearND) [21:41:10] (CR) EBernhardson: "err, that should read version.avpr :)" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/274307 (https://phabricator.wikimedia.org/T128530) (owner: EBernhardson) [22:15:17] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Ensure file.encoding is UTF-8 for all JVMs in the Analytics Cluster - https://phabricator.wikimedia.org/T128607#2100965 (Ottomata) [22:15:37] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Ensure file.encoding is UTF-8 for all JVMs in the Analytics Cluster - https://phabricator.wikimedia.org/T128607#2080328 (Ottomata) a:Ottomata [22:28:02] wikimedia/mediawiki-extensions-EventLogging#542 (wmf/1.27.0-wmf.16 - 4305169 : 20after4): The build has errored. [22:28:02] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/commit/4305169f0589 [22:28:02] Build details : https://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/114600093 [23:14:57] Analytics, Analytics-Cluster, Deployment-Systems, scap, Scap3 (scap3-adoption): Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2096617 (greg) [23:15:23] Analytics, Analytics-Cluster, Deployment-Systems, scap, Scap3 (scap3-adoption): Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2096617 (greg) @Ottomata yeah, that task has morphed into something it wasn't, fixed. [23:49:45] (PS6) BryanDavis: Add initial oozie job for ApiAction [analytics/refinery] - https://gerrit.wikimedia.org/r/273557 (https://phabricator.wikimedia.org/T108618) [23:54:34] (PS7) BryanDavis: Add initial oozie job for ApiAction [analytics/refinery] - https://gerrit.wikimedia.org/r/273557 (https://phabricator.wikimedia.org/T108618)