[00:05:52] 10Analytics, 10Analytics-EventLogging, 10ArchCom-RfC, 06Discovery, and 10 others: RFC: Use YAML instead of JSON for structured on-wiki content - https://phabricator.wikimedia.org/T147158#2967355 (10MaxSem) 05Open>03declined Closing per above. [01:59:33] PROBLEM - YARN NodeManager Node-State on analytics1039 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:00:34] RECOVERY - YARN NodeManager Node-State on analytics1039 is OK: OK: YARN NodeManager analytics1039.eqiad.wmnet:8041 Node-State: RUNNING [02:23:30] 10Analytics, 10Analytics-EventLogging, 10MediaWiki-Vagrant, 13Patch-For-Review, 15User-bd808: Replace upstart with systemd unit in eventlogging::devserver and eventlogging::service - https://phabricator.wikimedia.org/T154265#2967670 (10bd808) 05Open>03Resolved a:03bd808 [02:35:41] 10Analytics, 10Analytics-EventLogging, 10MediaWiki-Vagrant, 13Patch-For-Review, 15User-bd808: Replace upstart with systemd unit in eventlogging::devserver and eventlogging::service - https://phabricator.wikimedia.org/T154265#2967688 (10bd808) 05Resolved>03Open Looks like I may have closed this one ea... [02:51:16] 10Analytics-Tech-community-metrics, 06Developer-Relations (Jan-Mar-2017): Merge detached Phab and mw.org identities in korma DB if Phab API shows that accounts are linked - https://phabricator.wikimedia.org/T156216#2967711 (10Aklapper) [02:51:41] 10Analytics-Tech-community-metrics, 06Developer-Relations (Jan-Mar-2017): Merge detached Phab and mw.org identities in korma DB if Phab API shows that accounts are linked - https://phabricator.wikimedia.org/T156216#2967711 (10Aklapper) p:05Triage>03Lowest [04:05:39] zareen: If you give us a little bit more info on what is what you are trying to do maybe we can help you find a better query. [04:40:32] 10Analytics, 10ChangeProp, 10Edit-Review-Improvements-ReviewStream, 10EventBus, and 4 others: Set up the foundation for the ReviewStream feed - https://phabricator.wikimedia.org/T143743#2967916 (10Mattflaschen-WMF) >>! In T143743#2966929, @Ottomata wrote: > === Proposal (as Otto understands it): > ReviewSt... [04:43:56] 10Analytics, 10ChangeProp, 10Edit-Review-Improvements-ReviewStream, 10EventBus, and 4 others: Set up the foundation for the ReviewStream feed - https://phabricator.wikimedia.org/T143743#2967917 (10Mattflaschen-WMF) >>! In T143743#2955780, @SBisson wrote: >>>! In T143743#2955161, @Mattflaschen-WMF wrote: >>... [07:25:58] ah snap hadoop-mapreduce-historyserver died because of OOM.. [07:25:58] sigh [07:57:35] aqs migrated to node6 [07:57:37] all good :) [07:58:29] 06Analytics-Kanban, 06Services (blocked), 15User-mobrovac: Upgrade AQS to node 6 - https://phabricator.wikimedia.org/T155642#2968081 (10elukey) Production nodes upgraded to node 6! [07:58:41] 06Analytics-Kanban, 06Services (blocked), 15User-mobrovac: Upgrade AQS to node 6 - https://phabricator.wikimedia.org/T155642#2968082 (10elukey) p:05Triage>03Normal [08:55:12] 06Analytics-Kanban, 15User-Elukey: Ongoing: Give me permissions in LDAP - https://phabricator.wikimedia.org/T150790#2968134 (10elukey) @MelodyKramer Hello! Just added you to the `wmf` group, you should be now able to access https://pivot.wikimedia.org. Please read [[ https://phabricator.wikimedia.org/T150790#... [09:12:47] 06Analytics-Kanban, 06Operations, 10ops-eqiad, 15User-Elukey: rack and set up aqs100[7-9] - https://phabricator.wikimedia.org/T155654#2949969 (10elukey) a:05Cmjohnson>03elukey [10:20:44] ah! I forgot about the IPs allocation for the cassandra instances [10:43:19] * elukey afk for a bit [11:06:30] (03PS1) 10Joal: Update sqoop script with labsdb specificity [analytics/refinery] - 10https://gerrit.wikimedia.org/r/334042 (https://phabricator.wikimedia.org/T155658) [11:49:11] back [13:16:00] 10Analytics, 10Datasets-Webstatscollector, 06Language-Team: Investigate anomalous views to pages with replacement characters - https://phabricator.wikimedia.org/T117945#2968701 (10Arthur2e5) @West.andrew.g's FFFD count seems to match up with UTF-8 byte count (verified with `'\u2013'` in `[[Obsessive���compul... [13:40:01] 10Analytics-Tech-community-metrics, 06Labs: http://korma.wmflabs.org/ is down: "502 Bad Gateway" - https://phabricator.wikimedia.org/T156253#2968737 (10Aklapper) [13:40:10] 10Analytics-Tech-community-metrics, 06Labs: http://korma.wmflabs.org/ is down: "502 Bad Gateway" - https://phabricator.wikimedia.org/T156253#2968737 (10Aklapper) p:05Triage>03High [13:40:33] hi nuria, i'm working on creating a retention metric which calculates the average number of days until a unique device returns to wp. right now, i'm generating this data over the available timespan in the webrequest table. i'll look into sampling the data to not use up so many resources. [14:16:58] Hi zareen [14:17:16] hi joal! [14:17:52] zareen: another approach to prevent using a lot of resources is to extract the fields that are needed from the table and possibly compress aggregating by wider time ranges [14:18:26] If you want to query last access data a lot of times over a long period, I'd suggest doing that :) [14:20:48] joal: hmm, do you know if there's any documentation or examples of such queries? [14:23:26] 06Analytics-Kanban, 15User-Elukey: Ongoing: Give me permissions in LDAP - https://phabricator.wikimedia.org/T150790#2968804 (10MelodyKramer) Thanks @elukey - appreciate it! I'll let you know if I run into any trouble! Have a great day! [14:23:27] zareen: Let's take the query you are currently running as an example [14:25:01] sure, i'm happy to modify my query to use less resources, but am not quite clear on how to do that :) [14:25:11] hi team :] [14:25:31] hey joal, do you have 10 mins to discuss FR in druid? [14:25:38] hi mforns [14:25:44] in meeting in 5 minutes [14:25:49] but I have 5 now :) [14:25:53] batcave? [14:26:04] zareen: idea would be to copy samller subset of data [14:26:08] mforns: sure [14:27:56] i can come too! [14:28:03] zareen: come ottomata :) [14:28:10] sorry zareen :) [14:29:31] joal: any chance you have a few minutes today to help me figure out how to do that? [14:30:01] zareen: sure ! [14:31:03] zareen: in 1/2 hour? [14:31:35] awesome, 1/2 hour works perfect. thanks, really appreciate it :) [14:43:58] 10Analytics-Tech-community-metrics, 06Labs: http://korma.wmflabs.org/ is down: "502 Bad Gateway" - https://phabricator.wikimedia.org/T156253#2968737 (10Paladox) @Aklapper korma is deprecated, it has been replaced by https://wikimedia.biterg.io/app/kibana#/dashboard/Overview See https://www.mediawiki.org/wiki/... [14:55:03] running home, bbiab [14:56:20] 10Analytics-Tech-community-metrics, 06Labs: http://korma.wmflabs.org/ is down: "502 Bad Gateway" - https://phabricator.wikimedia.org/T156253#2968863 (10Aklapper) @Paladox: No. "korma.wmflabs.org will be deprecated" is not "korma.wmflabs.org has been deprecated". Please read carefully. [15:08:14] ok now actually running home, bbiab :p [15:17:42] joal: hows it working out? [15:23:18] joal: ready whenever you are [15:28:38] zareen: I'm sorry I meesed up on time :( [15:29:00] zareen: I have 1h meeting now, will you be ready after? [15:29:07] chasemp: loading :) [15:29:26] joal: no worries, 1 hour from now also works [15:29:49] thanks zareen [15:30:08] chasemp: https://grafana-admin.wikimedia.org/dashboard/file/server-board.json?var-server=labsdb1009&var-network=eth0 - eth0 tells me something happens, but that's all ;) [15:30:23] heh [15:30:32] joal: how does it effect you when we add new views during the process? [15:30:47] just fyi https://phabricator.wikimedia.org/T59617#2968832 [15:31:27] chasemp: I have a list of views I try, so it shouldn't affect :) [15:31:31] kk [15:33:29] joal: out of curiosity where does the list of views you look for come from? [15:34:25] chasemp: a project list compiled from the existing projects (milimetric built it) [15:34:31] gotcha [15:34:42] chasemp: I had to remove a lot of them ;) [15:36:34] joal: just as an fyi this is how you could know if a table exists as a view or if it is translated to a view https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/role/templates/labs/db/views/maintain-views.yaml [15:36:45] in case you guys have to go digging for what is what [15:37:36] awesome, thanks chasemp [15:38:44] (03PS2) 10Joal: Update sqoop script with labsdb specificity [analytics/refinery] - 10https://gerrit.wikimedia.org/r/334042 (https://phabricator.wikimedia.org/T155658) [15:40:55] ottomata, I thought that maybe the 'region' field could be left blank by the streaming input and then populated by the daily job. This could be a good compromise, no? [15:41:21] joal, ^ [15:41:51] 06Analytics-Kanban, 13Patch-For-Review: Run a 1-off sqoop over the new labsdb servers - https://phabricator.wikimedia.org/T155658#2969128 (10JAllemandou) Follow-up: - sqoop-tool labs tool created for db credentials - sqoop launched on wiki groups 1 (enwiki) and 13 (lots of very small projects) Seems to be w... [15:42:17] mforns: hm, need to doucle how this is feasible, but sounds a good compromise (if they really need it ;) [15:42:33] joal we could also geocode in the streaming job [15:42:50] ottomata: we could, but I really prefer to keep that job simple [15:44:53] joal, if the streaming job sets region=NULL for all records, while keeping the field in its schema, then the daily job can populate that [15:53:57] 10Analytics, 10Analytics-Cluster: Add jmxtrans metrics from Hadoop yarn-mapreduce-historyserver - https://phabricator.wikimedia.org/T156272#2969142 (10Ottomata) [15:58:38] 06Analytics-Kanban, 13Patch-For-Review: Run a 1-off sqoop over the new labsdb servers - https://phabricator.wikimedia.org/T155658#2969159 (10jcrespo) I do not mind labsdb-web being open, but please do not use it, as it should be reserved for web-like requests. If needed, I think I could provide even load balan... [15:59:58] 06Analytics-Kanban, 13Patch-For-Review: Run a 1-off sqoop over the new labsdb servers - https://phabricator.wikimedia.org/T155658#2969164 (10JAllemandou) @jaime: I currently use labsdb-analytics, so I have no idea how and which machine got chosen. [16:03:18] 06Analytics-Kanban, 13Patch-For-Review: Run a 1-off sqoop over the new labsdb servers - https://phabricator.wikimedia.org/T155658#2969170 (10JAllemandou) for memory: https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/role/templates/labs/db/views/maintain-views.yaml [16:05:10] mforns: do I explicitely need to set the filed to null? Would druid accept no field set? [16:05:26] joal, no idea? [16:05:34] k, need some testing :) [16:05:41] we can try :] [16:12:05] 10Analytics, 10Analytics-EventLogging, 06Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs: EL unable to decode mobile events due to appinstallid - https://phabricator.wikimedia.org/T96940#2969192 (10Milimetric) You can check the schema for errors in logstash and/or Hadoop on the eventlogging_Error topic. Let me... [16:17:14] zareen: meetings over, ready when you want :) [16:17:56] joal: now works [16:18:03] ok cool :) [16:18:33] zareen: https://hangouts.google.com/hangouts/_/wikimedia.org/a-batcave-2 [16:21:26] joal: joining now [16:22:21] (03CR) 10Ottomata: [WIP] Add banner impressions jobs (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/331794 (https://phabricator.wikimedia.org/T155141) (owner: 10Mforns) [16:22:50] (03CR) 10Ottomata: [WIP] Add banner impressions jobs (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/331794 (https://phabricator.wikimedia.org/T155141) (owner: 10Mforns) [16:23:16] hehe np [16:24:30] joal: i think you have to accept my request to join the hangout [16:24:32] zareen - alternate place? [16:24:56] zareen: did not receive anthing :( [16:25:25] joal let me try again [16:25:28] (03CR) 10Ottomata: [WIP] Add banner impressions jobs (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/331794 (https://phabricator.wikimedia.org/T155141) (owner: 10Mforns) [16:26:13] Ah zareen, are you with a wikimedia email address? [16:26:14] zareen: let's talk about this, i do not think last access now has the data you need to calculate retention [16:26:26] joal: no, gmail [16:26:38] riiight zareen this is the reason :) [16:27:02] ottomata: did something changed on labs that you know of , there are no eventlogging processes running on deployment-eventlogging03 [16:27:10] ottomata: and rerunning puppet doesn't help [16:27:14] ottomata: any ideas? [16:27:18] looking/remembering [16:27:18] zareen: just sent you an invite on hangout [16:28:38] nuria: i wouldn't be surprised if that instance is busted, a lot of thigns changed when they switch puppet/hiera labs stuff to horizon instead of wikitech [16:28:48] i've had to delete and recreate other instances because of it [16:29:03] but nuria, i see el processes there.. [16:29:14] processor client-side-01 start/running 13795 [16:29:14] processor client-side-00 start/running 25488 [16:29:14] consumer mysql-m4-master-00 start/running 29986 [16:29:14] forwarder legacy-zmq start/running 6233 [16:29:14] consumer all-events-log start/running 6246 [16:29:14] consumer client-side-events-log start/running 6255 [16:29:24] zareen, joal: talking to fdans but can join in a bit maybe. please see: https://wikitech.wikimedia.org/wiki/Analytics/Data/UserRetention#Definition_3:_Average_frequency_of_the_returned_visitors [16:29:33] and i see events in all-events.log [16:30:02] (03CR) 10Mforns: [WIP] Add banner impressions jobs (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/331794 (https://phabricator.wikimedia.org/T155141) (owner: 10Mforns) [16:30:11] https://www.irccloud.com/pastebin/wCs0Tc6d/ [16:30:59] ottomata: ps doesn't show anything... [16:31:56] ottomata: am I seeing things? [16:32:15] 06Analytics-Kanban: Document the difference in aggregate data on wikistats and wikistats 2.0 - https://phabricator.wikimedia.org/T150963#2969270 (10Milimetric) Updated here: https://wikitech.wikimedia.org/wiki/Analytics/Wikistats2.0#New_Data_Source @Quiddity, can we ping you to look at this with us? I know you... [16:32:33] ps aux | grep eventlogging [16:32:35] shows me processes just fine [16:32:40] i think you are seeing things [16:32:48] ottomata: jaja [16:33:09] ottomata: did we change linux distro, that must be it [16:33:12] haha, nope [16:33:17] i think its your - in front of aux [16:33:23] not sure why that is different, but it doesn't something difffernet [16:33:36] ottomata:right [16:33:37] a Lift the BSD-style "only yourself" restriction, [16:33:41] -a Select all processes except both session leaders (see getsid(2)) and processes not associated with a terminal. [16:33:54] a and -a do different things [16:34:02] ottomata: well, something i just learned [16:34:20] ps is funky in many different ways, because i think there are 2 styles of using it [16:34:23] and they can be combined [16:34:27] BSD vs posix maybe? not sure [16:34:45] http://askubuntu.com/questions/484982/what-is-the-difference-between-standard-syntax-and-bsd-syntax [16:36:27] "Over time, the various strands of Unix extended ps with many more options. The BSD variant chose to retain the original syntax, with no leading -, and a and x still exist today. [16:36:27] " [16:36:37] "The System V variant chose to adopt the syntactic convention of - for options, and used different letters" [16:36:42] i do ps aux [16:36:45] but i think its a bad habit [16:36:51] ps -e... is probably nicer / more linuxy [16:42:43] 06Analytics-Kanban, 10MediaWiki-API, 06Research-and-Data: Copy cached API requests from raw webrequests table to ApiAction - https://phabricator.wikimedia.org/T155478#2944303 (10Milimetric) Sorry if I'm missing something obvious, why does this need research's attention? We could make a job to process out ap... [16:51:16] ottomata: did we have to build code to test on beta or did it read things directly from /srv/deployment? [16:57:52] if you want to actually deploy new code EL code to beta, you have to deploy it [16:58:04] so, either merge and deploy in beta, or cherry-pick on deployment-tin and deploy [16:58:13] but, you can also just copy your changes somewhere in your own home dir [16:58:15] and set PYTHONPATH [16:58:17] and then run whatever [17:01:20] 06Analytics-Kanban, 06Operations: Periodic 500s from piwik.wikimedia.org - https://phabricator.wikimedia.org/T154558#2915651 (10Milimetric) Yep, this is definitely still happening. I took a look and what's happening is every time someone visits a page instrumented with piwik, piwik makes a new **connection**... [17:02:05] 06Analytics-Kanban: Clean up datasets.wikimedia.org - https://phabricator.wikimedia.org/T125854#2969372 (10Ottomata) a:05Milimetric>03Ottomata [17:02:07] 06Analytics-Kanban, 06Research-and-Data: Coordinate with research to vet metrics calculated from edit data lake - https://phabricator.wikimedia.org/T153923#2969373 (10Milimetric) a:03Milimetric [17:07:59] nuria: thanks for the link about user retention, i hadn't seen that before. the metric i'm working on is a bit different. i'm asking "for a given date, what is the average number of days until next access". so for example to calculate this metric for Dec 1, 2016, i count how many returns there are on each Dec 2, 3, 4, etc (up to 31 days in the future, until cookies expire) with last-access date of Dec 1. then take the [17:07:59] average "days until the next access" over this timespan (so =1 day for returns on Dec 2, =2 days for returns on Dec 3, etc). unless i'm understanding this wrong, last-access-date should work for this calculation.. [17:08:03] 06Analytics-Kanban, 06Research-and-Data: Coordinate with research to vet metrics calculated from the data lake - https://phabricator.wikimedia.org/T153923#2969389 (10Milimetric) [17:08:34] zareen: on meeting can takl in a bit , i do not think it would work [17:08:59] 06Analytics-Kanban, 06Operations: Periodic 500s from piwik.wikimedia.org - https://phabricator.wikimedia.org/T154558#2969394 (10Milimetric) a:03Milimetric [17:09:49] nuria: sure, please ping me (and HaeB) when you have a few minutes to discuss [17:11:05] 06Analytics-Kanban, 06Operations: Periodic 500s from piwik.wikimedia.org - https://phabricator.wikimedia.org/T154558#2915651 (10jcrespo) Assuming this is mysql, this is something I can indeed help relatively easily without touching code. As a reminder, I do not only handle mysql for mediawiki and labs, I also... [17:12:25] 06Analytics-Kanban, 15User-Elukey: Ongoing: Give me permissions in LDAP - https://phabricator.wikimedia.org/T150790#2796693 (10Jseddon) Hey! Could I request access to pivot for the following account: Seddon (https://wikitech.wikimedia.org/wiki/User:Seddon) Confirming I am a WMF employee. [17:16:15] nuria: milimetric qq. has there been any thought about merging datasets.wm.org and analytics.wm.org? [17:16:45] we could do the same plan for datasets cleanup, but then also include hosting and redirecting them: analytics.wm.org/datasets/.... [17:16:45] ? [17:31:32] ottomata: I'm for it. I guess we could even provide remote redirects instead of local symlinks [17:31:36] ja [17:31:43] in meetings for a while, but we can talk later today? [17:31:52] 10Analytics-Tech-community-metrics, 06Labs: http://korma.wmflabs.org/ is down: "502 Bad Gateway" - https://phabricator.wikimedia.org/T156253#2969480 (10Aklapper) potentially due to T143349#2969456 [17:32:09] a-team, leaving for now, will be back online later (but not for long) [17:32:26] bye joal ! [17:34:14] 10Analytics-Tech-community-metrics, 06Developer-Relations (Jan-Mar-2017): Merge detached SCM and SCR identities in korma DB if they have the same email address but not the same uuid - https://phabricator.wikimedia.org/T156283#2969485 (10Aklapper) [17:37:06] 10Analytics-Tech-community-metrics, 06Developer-Relations (Jan-Mar-2017): Merge detached SCM and SCR identities in korma DB if they have the same email address but not the same uuid - https://phabricator.wikimedia.org/T156283#2969504 (10Aklapper) p:05Triage>03Low T156216 is similar but requires a script fo... [17:37:53] 06Analytics-Kanban, 15User-Elukey: Ongoing: Give me permissions in LDAP - https://phabricator.wikimedia.org/T150790#2969509 (10elukey) @Jseddon done! Please review [[ https://phabricator.wikimedia.org/T150790#2964466 | this ]] to make sure that you are aware of what the new permissions grant to your account :) [17:41:02] milimetric: ja for sure [17:43:08] 10Analytics-Tech-community-metrics, 06Labs: http://korma.wmflabs.org/ is down: "502 Bad Gateway" - https://phabricator.wikimedia.org/T156253#2968737 (10madhuvishy) @Aklapper Hi! Yes, unfortunately it looks like the instance that hosts korma.wmflabs.org got deleted as in T143349#2969456. Do let us know if we ca... [17:44:36] mforns: , yt? [17:44:47] ottomata, in meeting, will ping you [17:44:51] k [17:48:35] OH, mil imetric (it has been talked about https://phabricator.wikimedia.org/T132594) [17:59:29] 06Analytics-Kanban: Involve the community in Wikistats 2.0 design - https://phabricator.wikimedia.org/T156287#2969650 (10Milimetric) [18:20:28] 06Analytics-Kanban: Involve the community in Wikistats 2.0 design - https://phabricator.wikimedia.org/T156287#2969650 (10Nuria) If we want somewhat sensical metrics for wireframes (I am not sure they are needed) here are a few: We receive more than 500 million pageviews per day [1] and reach more than 700 mill... [18:20:37] * elukey afk! byeee o/ [18:21:37] 06Analytics-Kanban: Involve the community in Wikistats 2.0 design - https://phabricator.wikimedia.org/T156287#2969704 (10Nuria) [18:21:38] byyeee [18:21:39] 06Analytics-Kanban, 10Analytics-Wikistats: Visual Language for http://stats.wikimedia.org replacement - https://phabricator.wikimedia.org/T152033#2969703 (10Nuria) [18:22:52] zareen: yt? have time in 5 minutes [18:24:02] zareen: sorry forgot to cc HaeB [18:33:24] zareen: yt? [18:35:15] hey nuria, sorry had stepped away for lunch. i believe HaeB is in meetings for most of today and it makes most sense to hold off until he can join us (i'm working with him on this project) [18:35:45] 06Analytics-Kanban: Move datasets.wikimedia.org to analytics.wikimedia.org/datasets - https://phabricator.wikimedia.org/T132594#2969797 (10Ottomata) a:03Ottomata [18:36:05] zareen: Let's just talk and you can pass along info. [18:36:14] zareen: do you want to join batcave? [18:37:26] zareen: and you can discuss with HaeB later and we can touch base tomorrow maybe, this week is QR and i imagine that is why we are all on meetings most of it [18:38:48] ottomata, done with the meeting :] [18:39:14] zareen: https://hangouts.google.com/hangouts/_/wikimedia.org/a-batcave [18:39:37] zareen: or i can call you directly if you msg me privately your e-mail address [18:39:50] mforns: want to talk to you about https://phabricator.wikimedia.org/T125854 and structured vs unstructured stuff [18:40:02] looking [18:40:20] (if you know) [18:40:32] particularly [18:40:33] 1. move unstructured stuff from limn-public-data/* to common/legacy/limn-public-data/* [18:40:43] which is the unstructured stuff? [18:41:39] 06Analytics-Kanban: Involve the community in Wikistats 2.0 design - https://phabricator.wikimedia.org/T156287#2969832 (10Milimetric) first thoughts on the project, will morph into the consultation: https://www.mediawiki.org/wiki/Wikistats_2.0_Design_Project [18:42:56] ottomata, I guess unstructured stuff refers to the reports that are NOT formatted in a way Dashiki can understand. [18:43:22] mforns: which ones are those? [18:44:13] AFAIK all limn-public-data/ reports read by Dashiki are generated currently by reportupdater, both Dashiki and RU have a structuring and naming convention [18:44:20] mmm [18:44:46] don't know by heart [18:45:34] I'd have to look the reports that are being generated currently by RU and the Dashiki dashboards deployed currently and list them [18:46:21] ottomata, but the task mentions "unstructured" reports to be consumed by Dashiki as well [18:46:22] mforns / ottomata: I can chat about these but I have meetings for another hour and a half [18:46:30] ok, milimetric if you know more we will wait [18:46:35] ok [18:46:44] 06Analytics-Kanban: Document the difference in aggregate data on wikistats and wikistats 2.0 - https://phabricator.wikimedia.org/T150963#2969847 (10Quiddity) Possibly! Quick clarification questions: * **Where/who** - IIUC, that page is intended to be an announcement, and needs to summarize [various changes], to... [18:46:50] i definitely knew what I meant when I wrote that, i'll try to job my memory in the meantime [18:46:55] *jog [18:46:55] xD [18:52:48] hey mforns :) [18:52:53] hey joal ! [18:52:57] Here for 15 minutes [18:53:04] batcave on druid, or tomorrow? [18:53:22] joal, I have a meeting in 7 [18:53:30] is 7 enough? [18:53:40] yes ! to the baticueva ! [18:53:52] Arf, busy :) [18:54:04] ok [18:54:06] mforns: https://hangouts.google.com/hangouts/_/wikimedia.org/a-batcave-2 [18:58:58] nuria: thanks, also for sending the link! (i saw it back in june- as you may recall, it was prompted by a discussion on retention we had with jon k etc.) as zareen said, now is not a good time. we'll get back to you in due time [19:04:11] AHH eeting sorry! [19:10:04] 06Analytics-Kanban, 13Patch-For-Review: Move datasets.wikimedia.org to analytics.wikimedia.org/datasets - https://phabricator.wikimedia.org/T132594#2969931 (10Ottomata) analytics.wikimedia.org/datasets now serves the same stuff as datasets.wikimedia.org. We can decide to put a redirect at datasets.wikimedia.o... [19:23:58] 10Analytics, 10ChangeProp, 10EventBus, 06Reading-Web-Backlog, and 4 others: Subscription: Trending service should be able to subscribe to edits in real time - https://phabricator.wikimedia.org/T145553#2969966 (10Pchelolo) [19:44:07] 10Analytics-Tech-community-metrics, 06Labs: http://korma.wmflabs.org/ is down: "502 Bad Gateway" - https://phabricator.wikimedia.org/T156253#2970072 (10Paladox) @madhuvishy Hi, im not sure if this will help, but it points to the source code here https://github.com/Bitergia/mediawiki-dashboard per https://www.m... [20:15:04] 06Analytics-Kanban, 10MediaWiki-API: Copy cached API requests from raw webrequests table to ApiAction - https://phabricator.wikimedia.org/T155478#2970181 (10Nuria) [20:27:10] 10Analytics-Dashiki, 06Analytics-Kanban, 13Patch-For-Review: Add extension and category (ala Eventlogging) for DashikiConfigs - https://phabricator.wikimedia.org/T125403#2970262 (10Milimetric) I'm an idiot, it works perfectly fine https://deployment.wikimedia.beta.wmflabs.org/wiki/Config:Dashiki:TestConfig,... [20:28:24] HaeB: that's fine e-mail sent, i think hopefully the work arround I suggested makes sense [20:39:24] ottomata, milimetric, the problem with FR pivot data set was just the pivot bug in the legend (where it sorts by the first metric in the list by default), the data set is OK [20:42:08] ottomata: do yu happen to know ...what is holding the event streams deployment ? [20:44:14] cool, makes sense [20:44:24] I was looking for that but it was harder to see on screenshare [20:50:18] fdans: i will be using beta if you are not [20:50:29] fdans: let me know if you ahem are not zzz [20:50:54] nuria: please do! :D [20:54:34] 10Analytics-Dashiki, 06Analytics-Kanban, 13Patch-For-Review: Add extension and category (ala Eventlogging) for DashikiConfigs - https://phabricator.wikimedia.org/T125403#2970331 (10Nuria) NICEEE!!! [20:55:14] nuria: bblack i guess? he knows. I ping him every other day, but haven't heard anyting [20:55:16] will ping him now [21:13:08] ottomata (cc fdans) this is the way events will look on table with json UA: [21:13:12] https://www.irccloud.com/pastebin/yLxsdgSC/ [21:13:54] aye makes sense nuria [21:14:59] (03CR) 10Ottomata: "'banner_impressions' as name. Not sure! I'm not insisting. Only if we can think of a name that makes more sense." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/331794 (https://phabricator.wikimedia.org/T155141) (owner: 10Mforns) [21:15:11] milimetric: hiiii [21:15:18] got a sec to talk about datasets cleanup? [21:16:33] yes! In 5 minutes? [21:16:35] sure [21:21:35] ottomata: to the cave! [21:33:42] 10Analytics, 06Editing-Department, 10MediaWiki-Authentication-and-authorization, 07Browser-Support-Firefox, and 2 others: Frequent loss of session data (since around 2016-11-28) - https://phabricator.wikimedia.org/T151770#2970511 (10Tgr) Pinging #analytics as this probably means that the various cookie-bas... [21:35:28] 06Analytics-Kanban, 13Patch-For-Review: Clean up datasets.wikimedia.org - https://phabricator.wikimedia.org/T125854#2970519 (10Milimetric) Triaging https://datasets.wikimedia.org/limn-public-data/ : edit/ (old data, ping @Jdforrester-WMF if he still needs any of it, otherwise DELETE) ee/... [21:37:02] 10Analytics, 06Reading-analysis, 06Research-and-Data, 10Research-consulting: Propose metrics along with qualifiers for the press kit - https://phabricator.wikimedia.org/T144639#2970522 (10leila) draft 2-page proposal sent to @JKatzWMF, @Tbayer, @Neil_P._Quinn_WMF, @DarTar, and @Nuria for their final review... [22:04:08] 10Analytics, 10Pageviews-API: Invalid timestamp returned from monthly aggregate endpoint - https://phabricator.wikimedia.org/T156312#2970596 (10MusikAnimal) [22:04:36] 10Analytics, 10Pageviews-API: Monthly aggregate endpoint returns invalid timestamp - https://phabricator.wikimedia.org/T156312#2970610 (10MusikAnimal) [22:12:00] 10Analytics, 10Pageviews-API: Monthly aggregate endpoint returns invalid timestamp - https://phabricator.wikimedia.org/T156312#2970619 (10MusikAnimal) And actually, it looks like the aggregate monthly endpoint is also returning the wrong months. I used January 1st through March 1st to get data for January and... [22:12:32] 10Analytics, 10Pageviews-API: Monthly aggregate endpoint returns unexpected results and invalid timestamp - https://phabricator.wikimedia.org/T156312#2970623 (10MusikAnimal) [22:18:54] 06Analytics-Kanban, 13Patch-For-Review: Clean up datasets.wikimedia.org - https://phabricator.wikimedia.org/T125854#2970648 (10Ottomata) Just talked with Dan about the stat1002 aggregate-datasets vs stat1003 public-datasets problem. (These directory names have no real meaning, stay with me!) aggregate-datase... [23:00:22] 10Analytics, 10ArchCom-RfC, 10EventBus, 10Wikidata, and 3 others: RFC: Requirements for change propagation - https://phabricator.wikimedia.org/T102476#2970722 (10MZMcBride)