[02:45:54] <wikibugs>	 Analytics-Kanban, Editing-Analysis, Patch-For-Review: Queries for the edit analysis dashboard failing since December 2015 [5 pts] - https://phabricator.wikimedia.org/T124383#1985534 (Jdforrester-WMF) Sure.
[03:10:34] <wikibugs>	 Analytics-Kanban, hardware-requests, operations: 8 x 3 SSDs for AQS nodes. - https://phabricator.wikimedia.org/T124947#1985547 (GWicke) In the short term, a few tweaks could also help to reduce latency:  - Reduce replication for the per-article keyspace from 3 to 2. - Read with CL_ONE by default. - Pos...
[07:40:17] <grrrit-wm>	 (PS1) Madhuvishy: Make redirect uris configurable [analytics/wikimetrics-deploy] - https://gerrit.wikimedia.org/r/267639
[09:06:06] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations, DevRel-February-2016, DevRel-January-2016, developer-notice: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#1985697 (Aklapper) @dicortazar: Yes (we t...
[09:53:51] <wikibugs>	 Analytics-Kanban, hardware-requests, operations: 8 x 3 SSDs for AQS nodes. - https://phabricator.wikimedia.org/T124947#1985752 (JAllemandou) @Gwicke:  - The replication factor for the per-article table had been changed to 2 a few month ago. I think restbase config management for cassandra has changed i...
[12:38:33] <aharoni>	 AndyRussG: hi, around now?
[12:40:58] <AndyRussG>	 aharoni: hi! mmm not reaaaaaaaaaaaaaaaaally... Just getting ready to take my kids to school... I'll be "more around" in about 30-40 min ;)
[12:53:39] <AndyRussG>	 aharoni: ahhhrg was it something quick? These kids keep growing up and now it seems... they don't need me to get ready to go... *snif* (Though in 5 min I do have to walk the younger one...)
[12:58:09] <aharoni>	 AndyRussG: nothing too urgent
[12:58:25] <aharoni>	 AndyRussG: I was curious whether you use stat1003. I had some questions about it.
[12:58:50] <aharoni>	 (Anybody who uses stat1003 can answer ;) )
[12:59:38] <AndyRussG>	 aharoni: ah hmmm no the analytics cluster access I have is only stat1002. Not sure what the difference is...
[13:00:24] <aharoni>	 AndyRussG: well, _maybe_ it's the same... basically what I wanted to know is whether it's possible to run mysql there similarly to how it works on terbium.
[13:01:09] <aharoni>	 I have some scripts that I run on terbium regularly, and Nuria suggested configuring limn to run them on stat1003, but apparently mysql works considerably differently there.
[13:01:55] <AndyRussG>	 aharoni: yeah I'm definitely not the one to help you with that, really sorry!!!
[13:02:14] <AndyRussG>	 Good luck tho :)
[13:12:10] <elukey>	 aharoni: surely I won't be able to help since I am super new, but I'd like to know the differences
[13:12:55] <aharoni>	 elukey: well, a basic one is that terbium has an "sql" command for all users who log in, and it sets up the password and connects to the write database for the language that you need.
[13:13:00] <aharoni>	 stat1003 doesn't seem to have it.
[13:19:53] <elukey>	 good point, have no idea :) mforns, joal ?
[13:20:38] <joal>	 Hi elukey, aharoni :)
[13:20:49] <elukey>	 my understanding though is that terbium is a more generic sql bastion, stat1003 it should be limited to analytics
[13:21:02] <elukey>	 hellooooooooo
[13:21:18] <joal>	 aharoni: The 'sql' commands you are talking about is a default setting file (can't recall where it is stored though)
[13:21:43] <joal>	 aharoni: IIRC you need to be part of the research group in order to have access to that file
[13:22:51] <joal>	 aharoni: more precisely, if you are part of the research group, you'll have by default access to that file, allowing you to use the 'sql' command without al the npt-fancy parameters
[13:23:34] <aharoni>	 joal: and it will work on stat1003?
[13:24:13] <joal>	 aharoni: The **one** you should ask just joined :)
[13:24:20] <joal>	 halfak: Hi sir o/
[13:24:32] <halfak>	 o/
[13:24:37] <joal>	 aharoni is asking about no-parameter sql commands on stat1003
[13:24:40] <joal>	 halfak: --^
[13:24:58] <joal>	 halfak: you'll know better than I do I think :)
[13:24:58] <halfak>	 "no-parameter"
[13:25:12] <joal>	 preset for user /passwd
[13:25:17] <aharoni>	 good morning halfak
[13:25:24] <halfak>	 good morning aharoni :)
[13:27:24] <halfak>	 So, it might help if you could rephrase the Q
[13:27:36] <halfak>	 I'm not sure I understand from the context given.
[13:31:21] <aharoni>	 halfak: I have some scripts that I run on terbium regularly, and Nuria suggested configuring limn to run them on stat1003, but apparently mysql works considerably differently there. In particular, terbium has an "sql" command for all users who log in, and it sets up the password and connects to the write database for the language that you need, and stat1003 doesn't seem to have it.
[13:32:31] <halfak>	 aharoni, gotcha.  Let me share my workaround. :)
[13:33:06] <halfak>	 I set this in my bashrc:
[13:33:07] <halfak>	 alias dbstore='mysql --defaults-file=~/.my.research.cnf -h analytics-store.eqiad.wmnet -u research'
[13:33:22] <halfak>	 Then I can just run "dbstore enwiki" and expect to connect to English Wikipedia's DB.
[13:33:53] <halfak>	 You can actually treat this just like the "mysql <blah blah>" command, so I can pipe SQL to it and stuff too.
[13:34:16] <halfak>	 e.g. $ cat my_query.sql | dbstore hewiki > my_measurements_of_hewiki.tsv
[13:34:33] <halfak>	 Oh!  Wait one moment.
[13:34:38] <joal>	 halfak: I knew you'd be the one :)
[13:34:40] <halfak>	 My config line won't work out of the box for you.
[13:34:46] <halfak>	 Let me get a line that will.
[13:35:46] <halfak>	 aharoni, alias dbstore='mysql --defaults-file=/etc/mysql/conf.d/research-client.cnf -h analytics-store.eqiad.wmnet -u research'
[13:36:03] <halfak>	 Note that it directly references the /etc/ file to get your password and stuff.
[13:36:13] <halfak>	 I have a symlink in my homedir that you probably don't have.
[13:39:11] <joal>	 halfak: IIRC, you ned to be in the research group to access that file, no ?
[13:40:01] <halfak>	 joal, yeah.  That's right.  Not sure how else you can access the DB though.
[13:40:20] <joal>	 hm, best would ask for access through research group
[13:40:26] <halfak>	 AFAICT, we just put everyone in the "researchers" group
[13:40:30] <joal>	 Cause password in clear command text are BAD !
[13:40:38] <joal>	 :)
[13:41:02] <joal>	 halfak: one way or another, we all do bits of research ;) I use google many times a day !
[13:43:03] * joal apologizes to halfak for the bad joke
[13:45:05] <halfak>	 :P
[13:47:31] <elukey>	 can we add a link or a mention of this in https://wikitech.wikimedia.org/wiki/Stat1003 ?
[13:47:44] <elukey>	 without the passwords :P
[13:48:36] * joal is away for a moment
[13:48:45] * halfak looks to see if he pasted a password somewhere
[13:49:00] <halfak>	 Why do we keep talking about passwords in clear command text?
[13:53:51] <elukey>	 halfak: not sure I was kidding
[13:54:16] <elukey>	 I've read about password in clear text and I followed the joke
[13:57:00] <halfak>	 Ha.  Gotcha.
[13:57:34] <elukey>	 sorry :(
[14:45:24] <wikibugs>	 Analytics-Cluster, operations: Complete installation of analytics1017.eqiad.wmnet  - https://phabricator.wikimedia.org/T125055#1986160 (elukey) a:Ottomata
[14:45:41] <wikibugs>	 Analytics-Cluster, operations: Complete installation of analytics1017.eqiad.wmnet - https://phabricator.wikimedia.org/T125055#1973447 (elukey) Assigning to Andrew to have a reminder of this task in our queue.
[15:02:56] <wikibugs>	 Analytics-Kanban: Restore MobileWebSectionUsage_14321266 and MobileWebSectionUsage_15038458 - https://phabricator.wikimedia.org/T123595#1986218 (Ottomata) Ok!  ``` mysql:research@analytics-store.eqiad.wmnet [log]> SELECT table_name, (DATA_LENGTH + INDEX_LENGTH)/1024/1024/1024 as `TOTAL SIZE (GB)` FROM informa...
[15:03:14] <wikibugs>	 Analytics-Kanban: Restore MobileWebSectionUsage_14321266 and MobileWebSectionUsage_15038458 - https://phabricator.wikimedia.org/T123595#1986220 (Ottomata) a:Ottomata
[15:17:30] <grrrit-wm>	 (CR) MarkTraceur: [C: 1] "Looks like our numbers are all caught up, if someone could review/merge this and make sure it runs so we can catch up with our backlog tha" [analytics/limn-multimedia-data] - https://gerrit.wikimedia.org/r/266783 (owner: MarkTraceur)
[15:30:40] <wikibugs>	 Analytics, EventBus, Services: Ensure that EventBus extension gracefully handles service failures - https://phabricator.wikimedia.org/T125394#1986365 (Eevans) NEW a:Eevans
[15:53:39] <wikibugs>	 Analytics, EventBus, Services: Ensure that EventBus extension gracefully handles service failures - https://phabricator.wikimedia.org/T125394#1986451 (Eevans)
[15:55:24] <wikibugs>	 Analytics, EventBus, Services: Ensure that EventBus extension gracefully handles service failures - https://phabricator.wikimedia.org/T125394#1986365 (Eevans)
[16:05:02] <joal>	 Hey ottomata, just read the traffic thread: looks like it'll be easier to transition mobile out than planned !
[16:11:37] <joal>	 Today is not really easy for me (need to care Lino), would tomorrow be ok for you ottomata ?
[16:16:22] <wikibugs>	 Analytics-Cluster, Analytics-Kanban, hardware-requests, operations: Hadoop Node expansion for end of FY - https://phabricator.wikimedia.org/T124951#1986511 (mark) p:Normal>Low
[16:17:21] <grrrit-wm>	 (PS14) Joal: [WIP] Daily last_access uniques oozie job [analytics/refinery] - https://gerrit.wikimedia.org/r/216341 (https://phabricator.wikimedia.org/T92977) (owner: Madhuvishy)
[16:24:57] <grrrit-wm>	 (PS4) Milimetric: Implement a Tabular Layout [analytics/dashiki] - https://gerrit.wikimedia.org/r/267045 (https://phabricator.wikimedia.org/T118329)
[16:26:24] <grrrit-wm>	 (PS2) Milimetric: Clean up syntax errors and bad names [analytics/dashiki] - https://gerrit.wikimedia.org/r/267389
[16:30:16] <milimetric>	 MarkTraceur: I have no clue how this works, but are you sure you just want to check AUDIO, VIDEO, image with that exact casing in your queries?  You could to_lower and check that way if there are variations in the table
[16:30:19] <milimetric>	 otherwise I'll just merge this
[16:30:33] <MarkTraceur>	 milimetric: It's an enum field, not a string
[16:30:38] <MarkTraceur>	 So it's guaranteed to be that case
[16:30:52] <MarkTraceur>	 I believe
[16:31:21] <milimetric>	 sweet, merging
[16:31:32] <grrrit-wm>	 (CR) Milimetric: [C: 2 V: 2] Add per-media-type queries for uploads [analytics/limn-multimedia-data] - https://gerrit.wikimedia.org/r/266783 (owner: MarkTraceur)
[16:32:40] <milimetric>	 (I deleted the history file again so it can run.  We should file a bug for that... shouldn't have to do it, but it'll be tricky to detect config chagnes
[16:32:51] <MarkTraceur>	 Great
[16:33:27] <MarkTraceur>	 milimetric: Another thing, matmarex asked today whether we could fix the link encoding
[16:33:32] <MarkTraceur>	 In particular with spaces
[16:34:27] <milimetric>	 link encoding?
[16:34:49] <MarkTraceur>	 Yeah, like https://edit-analysis.wmflabs.org/multimedia-health/#projects=commonswiki/metrics=Cross-Wiki Uploads
[16:34:53] <MarkTraceur>	 The space breaks it
[16:35:27] <milimetric>	 oh, doh, fixing
[16:35:33] <MarkTraceur>	 Thanks :)
[16:36:54] <milimetric>	 MarkTraceur: that won't be deployed for a while 'cause I cleaned up some other stuff and we're pushing a new feature
[16:37:00] <milimetric>	 but sometime this week
[16:37:16] <MarkTraceur>	 OK, thanks!
[16:40:32] <milimetric>	 MarkTraceur: oh, but note that in the meantime it's not totally broken, if you copy the whole URL with the space it works ok
[16:56:15] <ottomata>	 joal:  yes, any time sounds fine, since bblack isn't in a hurry to turn off the monitoring events
[16:56:18] <ottomata>	 anytime this week is fine
[16:57:14] <wikibugs>	 Analytics, Discovery, EventBus, MediaWiki-General-or-Unknown, and 6 others: Reliable publish / subscribe event bus - https://phabricator.wikimedia.org/T84923#1986641 (Ottomata)
[16:58:07] <wikibugs>	 Analytics-Cluster, EventBus, Services, operations: Investigate proper set up for using Kafka MirrorMaker with new main Kafka clusters. - https://phabricator.wikimedia.org/T123954#1986647 (Ottomata) Open>Resolved
[17:01:58] <madhuvishy>	 AndyRussG: standup? :)
[17:02:00] <madhuvishy>	 uhhh
[17:02:02] <madhuvishy>	 sorry
[17:02:03] <madhuvishy>	 a-team
[17:02:10] <ottomata>	 ops meeting!
[17:02:15] <AndyRussG>	 madhuvishy: np :)
[17:02:28] <madhuvishy>	 waiting for mforns and  joal :)
[17:02:36] <ottomata>	 ahh i'll come to start of standup then go to ops meeting
[17:03:47] <madhuvishy>	 ah they are also out.
[17:06:25] <wikibugs>	 Analytics-Kanban, DBA, Patch-For-Review: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1986681 (Nuria) @jcrespo: Anything else here, is edit table still syncying?
[17:09:27] <yurik>	 ottomata, bblack depooled the last mobile, can't wait for the merge to be done :)
[17:09:52] <ottomata>	 hah, awesome :)
[17:10:00] <ottomata>	 what's your stake yurik?
[17:10:32] <yurik>	 ottomata, the 16 varnish servers for maps ;)
[17:11:45] <ottomata>	 ah, ha
[17:12:35] <yurik>	 ottomata, there is always a hidden motive ;)
[17:18:24] <wikibugs>	 Analytics: Dashiki: Add extension and category  (ala Eventlogging) for DashikiConfigs - https://phabricator.wikimedia.org/T125403#1986718 (Nuria) NEW
[17:34:19] <wikibugs>	 Analytics, WMDE-Analytics-Engineering: Remove http://datasets.wikimedia.org/aggregate-datasets/wikidata/ - https://phabricator.wikimedia.org/T125407#1986786 (Addshore) NEW
[17:34:24] <wikibugs>	 Analytics, WMDE-Analytics-Engineering: Remove http://datasets.wikimedia.org/aggregate-datasets/wikidata/ - https://phabricator.wikimedia.org/T125407#1986793 (Addshore) p:Triage>Low
[17:44:38] <wikibugs>	 Analytics: Upgrade Dashjiki to semantic-2 for all layouts - https://phabricator.wikimedia.org/T125409#1986826 (Nuria) NEW
[17:45:28] <addshore>	 Hi A-Team! :P
[17:57:16] <addshore>	 I'm looking at cache_status in webrequest and wondering what it means when that field contains a ' / empty rather than hit / pass / miss?
[18:02:17] <nuria_>	 ottomata: hardware meeting?
[18:11:39] <wikibugs>	 Analytics-EventLogging, MobileFrontend, Easy, Technical-Debt: Should be possible to override sampling in EventLogging schemas for development purpose - https://phabricator.wikimedia.org/T125122#1986921 (bd808) p:Triage>Low
[18:39:22] <nuria_>	 addshore: unless ottomata knows the answer to that .. it might be a question for ops channel
[18:39:35] <addshore>	 :D
[18:42:56] <wikibugs>	 Analytics: Debug wikimetrics docker dev setup failing on ubuntu 14.04 - https://phabricator.wikimedia.org/T125415#1987049 (madhuvishy) NEW
[18:44:28] <addshore>	 nuria_: any idea if using WHERE x_analytics_map['ns'] = '0' should be reliable? that should always be present for requests to ns 0 on a project right?
[18:45:35] <madhuvishy>	 addshore: i didn't even know ns was a thing
[18:45:42] <addshore>	 xD
[18:45:45] <madhuvishy>	 it isn't here https://wikitech.wikimedia.org/wiki/X-Analytics
[18:46:02] <addshore>	 oooh, not seen that page before....
[18:46:04] <madhuvishy>	 it's possible someone started setting it
[18:46:23] <madhuvishy>	 but either din't tell us - or they did and I have no idea
[18:46:39] <addshore>	 :D
[18:46:54] <addshore>	 Also, is there a way to determine requests from logged in users vs anons in webrequest?
[18:47:27] <madhuvishy>	 addshore: no :(
[18:48:50] <addshore>	 sad times:(
[18:49:59] <madhuvishy>	 addshore: fundraising wanted to know this too. I dont know exactly why, but i believe it's consistent with us not tracking any reader related information in this stream to keep the data sources as separate as possible
[18:50:20] <addshore>	 makes sense!#
[18:52:17] <madhuvishy>	 nuria_: I tried setting wikimetrics up on ubuntu14 with a VM - it still works
[18:52:52] <nuria_>	 addshore: nah, I doubt it, that has to be filled in client side
[18:53:29] <nuria_>	 addshore: and the fact that is not documented here: https://wikitech.wikimedia.org/wiki/X-Analytics
[18:53:50] <nuria_>	 addshore: makes me think it likely doesn't work as well
[18:54:08] <nuria_>	 addshore: there is no way to determine login vs anons
[18:54:29] <nuria_>	 madhuvishy: docker setup works correct?
[18:54:39] <madhuvishy>	 nuria_: yeah
[18:54:42] <madhuvishy>	 works great
[18:54:53] <nuria_>	 madhuvishy: ok, i am glad it was easy to test
[18:55:08] <madhuvishy>	 nuria_: so not sure why it didn't work for mforns
[18:55:08] <nuria_>	 madhuvishy: let's just reserve a bit of time so you can go over with marcel to see where difference lays
[18:55:24] <nuria_>	 madhuvishy: i think he will be off today but should be back arround tomorrow
[18:55:24] <madhuvishy>	 as far as i know we have the same versions
[18:55:28] <madhuvishy>	 yup yup
[18:55:35] <nuria_>	 madhuvishy: i bet is somnething completely unrelated
[18:55:48] <nuria_>	 madhuvishy: was it easy to map ports and such?
[18:56:14] <madhuvishy>	 nuria_: i dint need to - i used virtualbox - not vagrant
[18:56:20] <madhuvishy>	 so there was a full UI
[18:56:54] <nuria_>	 madhuvishy: ya, that is what i meant . I was using vagrant as an example, so good. I am glad it worked, let's keep it in progress until we re-test
[18:57:47] <madhuvishy>	 nuria_: ya alright.
[18:58:35] <nuria_>	 madhuvishy: if you are looking for a next item (not saying that you need one) there is this one: https://phabricator.wikimedia.org/T125228
[18:59:59] <wikibugs>	 Analytics-Kanban, DBA, Patch-For-Review: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1987113 (jcrespo) @Nuria, the Edit table finished syncing during the weekend, but sadly with errors, so I will have to for...
[19:05:23] <wikibugs>	 Analytics-Kanban, DBA, Patch-For-Review: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1987135 (Nuria) Excellent. Let us know when we can close this ticket.
[19:07:04] <MarkTraceur>	 halfak: Quick question for you, because I don't currently grok the dump script you helped me with - is there an easy way to make it update for only data it hasn't seen, like, data since the last dump?
[19:08:10] <addshore>	 nuria_: madhuvishy so that means the only way would be something client side?
[19:08:31] <addshore>	 I'm guessing passing a simple loggedin=1 in the x_analytics header is a no no?
[19:09:05] <madhuvishy>	 addshore: it is possible to - but we should review the security aspects in getting this data afaik
[19:09:05] <nuria_>	 addshore: i am not sure you can do that reliabily in all platforms even
[19:09:25] <addshore>	 hmmm
[19:09:33] <nuria_>	 addshore: so that is not really up to us, but think you are dealing with 4 platforms at least: desktop, mobile, ios and android
[19:10:00] <nuria_>	 addshore: a global feature like this one needs to be tagged globally
[19:10:20] <nuria_>	 addshore: and security review.. etc. but also, note that since webrequest is feed from varnish
[19:10:34] <nuria_>	 addshore: you are going to miss edit requests i believe
[19:11:33] <addshore>	 Basically I'm trying to work out how many page views to the main namespace occour on wikidata for logged in vs anon users ;)
[19:12:43] <grrrit-wm>	 (PS6) Madhuvishy: Development environment for wikimetrics using docker [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/267172 (https://phabricator.wikimedia.org/T123749)
[19:12:45] <nuria_>	 addshore: And , if you do not have the whole "edit" datastream
[19:12:52] <nuria_>	 addshore: i do not think is possible, correct?
[19:13:54] <addshore>	 well, wikidata doesnt really have an edit stream
[19:14:04] <addshore>	 once you are on the page everything else is done throguh JS and api calls
[19:14:54] <nuria_>	 addshore: i see, but any request that doesn't go through varnish (often the case for logged in) you do not have available, correct?
[19:15:04] <wikibugs>	 Analytics: Debug wikimetrics docker dev setup failing on ubuntu 14.04 - https://phabricator.wikimedia.org/T125415#1987192 (madhuvishy) @Yuvipanda tested it in his linux machine - Debian Stretch with kernel 4.2.0-1-amd64 and it seems to work fine there too.
[19:15:28] <nuria_>	 addshore: I might be ... ahem totally off here
[19:15:55] <addshore>	 well, they all still go through varnish :)
[19:16:09] <addshore>	 even api requests go through varnish, but they go straight through ;)
[19:16:41] <madhuvishy>	 milimetric: I made a task for the linux debugging here - https://phabricator.wikimedia.org/T125415. If you try it sometime, let me know how it went
[19:19:33] <madhuvishy>	 ottomata: I was looking at https://phabricator.wikimedia.org/T125228 - do you know how I can replicate a bad broker?
[19:22:43] <nuria_>	 addshore: ok, then , back to square 1: you have request but no way to tell them appart
[19:23:09] <addshore>	 currently looking for some asset (or something) that is only loaded by logged in users
[19:23:10] <wikibugs>	 Analytics, Wikipedia-Android-App: Database not updated for beta event logging and all-events.log reports 8x for each event - https://phabricator.wikimedia.org/T125423#1987230 (Niedzielski) NEW
[19:23:14] <addshore>	 ;)
[19:27:38] <addshore>	 bah, nothing different in the requests
[19:30:00] <wikibugs>	 Analytics, Wikipedia-Android-App: Database not updated for beta event logging and all-events.log reports 8x for each event - https://phabricator.wikimedia.org/T125423#1987283 (Nuria) @Niedzielski: if you need help with this you can always ping us on irc, i can help maybe in a few minutes.
[19:36:35] <nuria_>	 : addshore You want an idea?
[19:37:09] <addshore>	 Well, in JS I can tell if the user is logged in using wgUserId in mw.config
[19:37:30] <nuria_>	 addshore: why don't you send counters using EL?
[19:38:03] <addshore>	 Setting up EL might be a bit extreme for this case :/ we dont even really care about this over time afaik
[19:38:07] <nuria_>	 k
[19:38:39] <addshore>	 but in JS I guess if the user is logged in I could just load some other thing (which I could then track in the webrequest logs....
[19:42:09] <nuria_>	 addshore: not the best solution
[19:42:36] <nuria_>	 addshore: loading assets to "count" seems not so solid...
[19:42:51] <nuria_>	 addshore: what else could we do .....
[19:43:00] <addshore>	 yup, but it would only be in place for 24 hours ;)
[19:43:06] <nuria_>	 addshore: you could look (not sure if we have this data)
[19:43:15] <nuria_>	 addshore: at login page requests with wikidata referal
[19:43:46] <addshore>	 thats wouldnt give me page views of logged in users though :/
[19:43:47] <wikibugs>	 Analytics, Wikipedia-Android-App: Database not updated for beta event logging and all-events.log reports 8x for each event - https://phabricator.wikimedia.org/T125423#1987331 (Niedzielski) @Nuria, thanks! I just made this phab for tracking. This issue isn't blocking me today but I would like to know if it...
[19:44:19] <addshore>	 how easy is EL stuff to setup? and would EL be overkill simply for counting a single thing?
[19:44:53] <wikibugs>	 Analytics, Wikipedia-Android-App: Database not updated for beta event logging and all-events.log reports 8x for each event - https://phabricator.wikimedia.org/T125423#1987334 (Nuria)  I am going to reboot instance
[19:45:10] <ottomata>	 madhuvishy:  hi!
[19:45:17] <madhuvishy>	 ottomata: Hi!
[19:45:18] <ottomata>	 ja, use eventlogging somewhere in analytics labs project
[19:45:32] <ottomata>	 there are 3 kafka2xx brokers there
[19:45:35] <ottomata>	 in a cluster
[19:45:37] <ottomata>	 you can produce/consume from it
[19:46:03] <madhuvishy>	 ottomata: yeah okay -
[19:46:10] <madhuvishy>	 how do i make one of them "bad" though
[19:47:55] <ottomata>	 stop it!
[19:48:00] <ottomata>	 ssh to that instance
[19:48:03] <ottomata>	 sudo service kafka stop
[19:48:34] <madhuvishy>	 ottomata: alright then
[19:48:59] <ottomata>	 thanks for looking into that one madhuvishy!
[19:49:05] <ottomata>	 lemm eknow if i can help while you're poking at it
[19:49:13] <madhuvishy>	 ottomata: np. Sure!
[19:49:25] <wikibugs>	 Analytics-Kanban: Eventlogging should start with one bad kafka broker, retest that is the case - https://phabricator.wikimedia.org/T125228#1987346 (madhuvishy) p:Triage>Normal a:madhuvishy
[19:52:07] <nuria_>	 madhuvishy: i was just rebooting beta eventlogging
[19:53:19] <madhuvishy>	 nuria_: okayy?
[19:54:02] <ottomata>	 ja, easier to test our setup in analytics project though, because it has multiple brokers in a cluster
[19:54:05] <ottomata>	 the beta one only has a single broker
[19:54:15] <ottomata>	 so uh, yes, eventlogging will not be ok if that broker is bad
[19:55:42] <grrrit-wm>	 (PS1) MarkTraceur: Add illustration queries for enwiki [analytics/limn-multimedia-data] - https://gerrit.wikimedia.org/r/267722
[19:56:24] <nuria_>	 ottomata: ah i see
[19:57:16] <nuria_>	 ottomata: am i running puppet ok on eventlogging beta?
[19:57:19] <nuria_>	 https://www.irccloud.com/pastebin/0gbAXZ7r/
[19:58:02] <ottomata>	 sudo
[19:58:04] <ottomata>	 nuria
[19:59:09] <nuria_>	 ottomata: ok I am GRANDMA
[20:00:35] <ottomata>	 :)
[20:00:46] <nuria_>	 ottomata: but still, it doesn't run on beta right?
[20:00:48] <nuria_>	 https://www.irccloud.com/pastebin/4KEs4YCq/
[20:03:19] <ottomata>	 hmmm
[20:03:23] <ottomata>	 looking
[20:04:00] <nuria_>	 ottomata: el was old, i was going to deploy master & restart, just sync-ed
[20:04:12] <nuria_>	 ottomata: where is puppet depot deployed?
[20:04:14] <ottomata>	 nuria_:  i think this is a hiera problem
[20:04:25] <wikibugs>	 Analytics-General-or-Unknown, Beta-Cluster-reproducible: Regression: action=info pages broken - https://phabricator.wikimedia.org/T125432#1987378 (TheDJ) NEW a:Umherirrender
[20:04:31] <ottomata>	 i changed something but didnt' change hiera in beta
[20:05:12] <nuria_>	 ottomata: k
[20:05:15] <ottomata>	 hang on
[20:05:30] <nuria_>	 ottomata: k
[20:06:43] <ottomata>	 nuria_:  am going to wipe kafka in labs, just easier than fixing everything hackily
[20:06:50] <ottomata>	 in beta*
[20:07:04] <nuria_>	 ottomata: and reinstall from puppet?
[20:07:08] <wikibugs>	 Analytics-General-or-Unknown, Beta-Cluster-reproducible: Regression: action=info pages broken - https://phabricator.wikimedia.org/T125432#1987389 (Addshore) Reverting in https://gerrit.wikimedia.org/r/267724
[20:07:08] <ottomata>	 sorta ja
[20:07:16] <ottomata>	 it'll just re bootstrap under a new cluster name
[20:07:28] <ottomata>	 new zk chroot and new kafka data
[20:07:30] <ottomata>	 no reinstall necessary
[20:09:15] <nuria_>	 k
[20:16:44] <wikibugs>	 Analytics, Analytics-Wikistats, DevRel-February-2016, DevRel-January-2016: Clean the code review queue of analytics/wikistats - https://phabricator.wikimedia.org/T113695#1987418 (Aklapper)
[20:18:35] <ottomata>	 nuria_:  k, double checking a couple of things, should be good now
[20:21:27] <ottomata>	 hey milimetric, can I play with one of your trusty labs instances?
[20:21:29] <ottomata>	 dan-druid maybe?
[20:21:34] <ottomata>	 i'm testing some puppet stuff
[20:21:46] <ottomata>	 i might break it
[20:21:47] <ottomata>	 hmmm
[20:22:56] <ottomata>	 ahh, nm, i'm creating one
[20:36:00] <Alchimista>	 anyone can help me in some querys? How do i filter wikidata edits in the recentchanges table for a single wiki?
[20:40:13] <wikibugs>	 Analytics-Tech-community-metrics, DevRel-January-2016: Improve Key performance indicator: code contributors new / gone - https://phabricator.wikimedia.org/T63563#1987500 (Aklapper) Open>Resolved a:Aklapper Yeah. Resolving.  To potentially re-evalulate after fixing T118753.
[20:40:18] <Alchimista>	 dammit, find it
[20:42:33] <nuria_>	 Alchimista: probably research channel is best for those questions
[20:43:49] <nuria_>	 ottomata: looking
[20:43:56] <nuria_>	 ottomata: did you re-started EL too?
[20:44:03] <ottomata>	 yes
[20:44:04] <ottomata>	 nuria_:
[20:48:35] <nuria_>	 ottomata: looks like mysql is not installed locally anymore but on  labmon1001.eqiad.wmnet
[20:49:34] <nuria_>	 ottomata: but I cannot connect with mysql -h labmon1001.eqiad.wmnet
[20:52:10] <ottomata>	 ?
[20:52:13] <ottomata>	 thought it was local
[20:52:46] <ottomata>	 for beta el?
[20:52:48] <ottomata>	 nuria
[20:52:52] <ottomata>	 cat //etc/eventlogging.d/consumers/mysql-m4-master
[20:52:55] <ottomata>	 cat /etc/eventlogging.d/consumers/mysql-m4-master
[20:53:02] <ottomata>	 that will show you connect info for the EL mysql consumser
[20:53:51] <nuria_>	 ottomata: right, that is what i did. and that host (which i do not think has mysql) came out
[20:54:21] <nuria_>	 ah nooo waittt
[20:55:12] <nuria_>	 GRANDMA again
[20:57:33] <ottomata>	 heheh :)
[21:02:50] <milimetric>	 ottomata: phew, I missed your ping but I'm glad you didn't use dan-druid, I don't want it to break
[21:03:57] <ottomata>	 k
[21:31:52] <grrrit-wm>	 (CR) Bearloga: "Changes, fixes, documentation, performance estimations coming in next patchset." (5 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/254461 (https://phabricator.wikimedia.org/T118218) (owner: Bearloga)
[21:38:10] <joal>	 nuria_, madhuvishy: I have numbers for uniques + offsets --> The average offset number for the top 20 uniques host is 63% of uniques estimates without offset
[21:38:36] <joal>	 I makes feel that this number is too big, but I'd like your opinion
[21:39:18] <madhuvishy>	 joal: yeah definitely sounds big
[21:44:17] <grrrit-wm>	 (PS15) Joal: [WIP] Daily last_access uniques oozie job [analytics/refinery] - https://gerrit.wikimedia.org/r/216341 (https://phabricator.wikimedia.org/T92977) (owner: Madhuvishy)
[21:44:17] <joal>	 madhuvishy, nuria_ --^ Here is code I have run
[21:44:18] <madhuvishy>	 joal: will look at it in a little bit - at the engagement metrics meeting
[21:44:18] <joal>	 it's ready to be productionized (oozie job tested), except for this bizarre offset number
[21:44:18] <joal>	 np at all
[21:44:18] <joal>	 I'm going to bed :)
[21:44:18] <joal>	 We'll discuss that tomorrow :)
[21:44:18] <joal>	 Have a good end of day a-team !
[21:44:18] <milimetric>	 good night!
[21:44:40] <wikibugs>	 Analytics-Kanban, Editing-Analysis: Edit schema needs purging, table is too big  for queries to run (500G before conversion) {oryx} [8 pts] - https://phabricator.wikimedia.org/T124676#1987767 (Neil_P._Quinn_WMF) @Nuria, given that we have a lot more mobile events than we want or need, what if (rather than...
[21:48:53] <wikibugs>	 Analytics-Kanban: Investigate adding piwik to transparency report - https://phabricator.wikimedia.org/T125175#1987783 (Nuria) Tracking snippet:  <!-- Piwik --> <script type="text/javascript">   var _paq = _paq || [];   _paq.push(['trackPageView']);   _paq.push(['enableLinkTracking']);   (function() {     var...
[21:51:26] <wikibugs>	 Analytics-Kanban, DBA, Patch-For-Review: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1987796 (jcrespo) I have just confirmed it is in sync before 24 January (it is still a bit behind in replication, but that...
[21:59:12] <milimetric>	 Deskana: ok, so I'll try again without the split attention :)
[21:59:48] <milimetric>	 so the use case for global uniques is not just a finite set of questions
[22:00:15] <milimetric>	 the use case, broadly, is "this is valuable, we want to basically have full information so that we can do any of (long list) of things"
[22:00:48] <milimetric>	 so yes, we have lots of examples of useful analysis that relied on uniques.  Fundraising, search, apps, etc. all have these examples
[22:01:14] <milimetric>	 they approximate uniques or use the mobile app ones, as the case may be
[22:01:58] <milimetric>	 now, what I want for us to prove to ourselves is whether we can organize that data in our analytics infrastructure in a way that deals with the harder problems:
[22:02:18] <milimetric>	 * deleting it and dangerous derivative datasets automatically
[22:02:54] <milimetric>	 * cataloging it and overseeing who has access to it, also making sure they understand the privacy implications
[22:03:31] <Deskana>	 milimetric: Aha, right. That's clearer to me now.
[22:03:39] <milimetric>	 * joining this data to all, not just some, of the other pieces of data that we think might be useful
[22:04:11] <milimetric>	 yeah, so basically I think we should try to solve those problems, which I think are solvable and I've personally been pushing for for years
[22:04:45] <milimetric>	 once we're there, I think this conversation around unique tokens will be much easier to have in objective terms.
[22:04:59] <Deskana>	 milimetric: Right. Those questions sound less on the product-end than what I thought you were saying.
[22:05:05] <Deskana>	 milimetric: That seems reasonable enough to me.
[22:05:17] <milimetric>	 oh, I don't mess with the product end, that's above my pay grade :)
[22:05:25] <milimetric>	 that's all yall :)
[22:06:47] <Deskana>	 milimetric: Thanks for explaining.
[22:08:00] <milimetric>	 anytime.  We should keep talking about this though, and we should work together to see that we get the infrastructure to where it can handle either new data or removing old data
[22:08:05] <milimetric>	 basically, making it more agile
[22:27:58] <grrrit-wm>	 (PS1) Milimetric: Add all timezones to the timezone picker [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/267790 (https://phabricator.wikimedia.org/T121167)
[22:29:44] <grrrit-wm>	 (PS1) Ori.livneh: Exit when the main process crashes [analytics/statsv] - https://gerrit.wikimedia.org/r/267792 (https://phabricator.wikimedia.org/T117994)
[23:00:59] <grrrit-wm>	 (CR) Ori.livneh: [C: 2 V: 2] Exit when the main process crashes [analytics/statsv] - https://gerrit.wikimedia.org/r/267792 (https://phabricator.wikimedia.org/T117994) (owner: Ori.livneh)
[23:32:52] <madhuvishy>	 ottomata: I pulled eventlogging on a test instance
[23:33:01] <madhuvishy>	 and did sudo python setup.py install
[23:33:16] <madhuvishy>	 and it fails
[23:33:19] <madhuvishy>	 https://www.irccloud.com/pastebin/uWM9rolz/
[23:48:29] <wikibugs>	 Analytics-Kanban, Editing-Analysis: Edit schema needs purging, table is too big  for queries to run (500G before conversion) {oryx} [8 pts] - https://phabricator.wikimedia.org/T124676#1988159 (Nuria) @Neil_P._Quinn_WMF , if this was a flat file it will be real easy to remove data now    cc-ing @jcrespo for...
[23:49:11] <nuria_>	 who nuria
[23:52:07] <nuria_>	 a-team: we will sampling down the edit table to reduce its size: https://phabricator.wikimedia.org/T124676
[23:53:18] <milimetric>	 makes sense, nuria, if James doesn't know, you should poke him and/or Neil