[00:37:47] 06Analytics-Kanban: Wikistats 2.0. - https://phabricator.wikimedia.org/T130256#3127295 (10Neil_P._Quinn_WMF) @Nuria @Milimetric: I apologize if this is a bad place for this feedback, but I couldn't think of a better one. I had a meeting with @HaeB, @Tnegrin, and @mpopov yesterday where we were discussed our me... [00:41:11] 10Analytics, 06Editing-Analysis: Pivot data quality query: Number of logged-in event-editors has three massive peaks in late November 2016 - https://phabricator.wikimedia.org/T161187#3127301 (10Jdforrester-WMF) 05Open>03Resolved a:03Jdforrester-WMF >>! In T161187#3124645, @JAllemandou wrote: > @Jdforrest... [04:08:11] 06Analytics-Kanban: Wikistats 2.0. - https://phabricator.wikimedia.org/T130256#3127408 (10Nuria) >Have you given any thought to supporting such annotations? Yes, Dashiki already supports annotations (it is been a while), see pageviews and look at bottom axis https://analytics.wikimedia.org/dashboards/vital-signs... [04:41:49] 06Analytics-Kanban: Wikistats 2.0. - https://phabricator.wikimedia.org/T130256#3127413 (10Nuria) Have in mind that wikistats has a strong community focus, its existance much predates the foundation and, really, its main goal is to motivate our editor community (cc @Erik_Zachte) I think for reporting data to th... [07:58:08] 10Analytics, 10Analytics-EventLogging, 10DBA, 10ImageMetrics, 13Patch-For-Review: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#3127562 (10Marostegui) Bad news, the table is there again @Nuria :-( ``` root@EVENTLOGGING m4[log... [09:17:51] hello team :] [10:56:00] 10Analytics, 06Editing-Analysis: Pivot data quality query: Number of logged-in event-editors has three massive peaks in late November 2016 - https://phabricator.wikimedia.org/T161187#3127928 (10JAllemandou) > > OK, so each of "Peer created event users", "System created event users" and "Self created event use... [11:26:04] 10Analytics-Tech-community-metrics, 06Developer-Relations (Jan-Mar-2017): Go through default Kibana widgets; decide which ones are not relevant for us and remove them - https://phabricator.wikimedia.org/T147001#3127981 (10Aklapper) This is better than I was afraid. :P (Realized that having the same "list of co... [11:26:18] 10Analytics-Tech-community-metrics, 06Developer-Relations (Jan-Mar-2017): Go through default Kibana widgets; decide which ones are not relevant for us and remove them - https://phabricator.wikimedia.org/T147001#3127983 (10Aklapper) [11:43:21] 10Analytics-Tech-community-metrics, 06Developer-Relations (Jan-Mar-2017): Deployment of Maniphest panel - https://phabricator.wikimedia.org/T138002#3128014 (10Aklapper) Note to myself: Once this is deployed, compare to https://rust-analytics.mozilla.community/app/kibana#/dashboard/GitHub-Issues-Timing and http... [11:45:27] 10Analytics-Tech-community-metrics: Clarify differences between similar widgets - https://phabricator.wikimedia.org/T160576#3128020 (10Aklapper) @Lcanasdiaz: I'd love to understand Bitergia's default choices and recommendations for the five items in the table above, and whether the "default" widgets per dashboar... [12:11:52] milimetric: When you have minute, I have funny stuff to show you :) [12:12:39] mforns: Would you, by any chance, know how to proxy from a labs instance onto the internetz? [12:12:49] nope [12:12:57] :/ sorry [12:13:04] np mforns :) [12:13:12] Asked just in case [12:25:14] (03PS12) 10Mforns: Add oozie workflow to load projectcounts to AQS [analytics/refinery] - 10https://gerrit.wikimedia.org/r/339421 (https://phabricator.wikimedia.org/T156388) [13:05:07] hey joal yt? :] [13:06:01] yes mforns [13:06:05] what's up? [13:06:25] :] I'm going to go ahead and deploy AQS to prod, with the double deploy we discussed [13:06:47] today I did all the tests and looks good. [13:07:09] I found a bug in the oozie load job, that wasn't outputing the all-sites in some cases [13:07:20] but fixed already and tested in prod test_keyspace [13:07:20] mforns: really ? not cool ! [13:07:25] yea [13:07:30] can you show me? [13:07:33] sure [13:07:35] batcave! [13:07:43] OMW ! [14:10:45] 10Analytics-Tech-community-metrics: Provide equivalent of "SCR: Code review users vs. Code review committers" in Kibana - https://phabricator.wikimedia.org/T151558#2821041 (10Aklapper) [14:10:47] 10Analytics-Tech-community-metrics: Provide equivalent of "SCR: People uploading patchsets vs. Reviewers per month" in Kibana - https://phabricator.wikimedia.org/T151559#2821054 (10Aklapper) [14:10:49] 10Analytics-Tech-community-metrics: Add remaining KPIs to Overview once available in kibana - https://phabricator.wikimedia.org/T116572#3128196 (10Aklapper) [14:11:08] 10Analytics-Tech-community-metrics: Provide equivalent of "SCR: People uploading patchsets vs. Reviewers per month" in Kibana - https://phabricator.wikimedia.org/T151559#2821054 (10Aklapper) There is a "Changeset Submitters" (`Change-submitters-per-month__gerrit_enrich`) widget covering "people uploading patchse... [14:16:22] 10Analytics-Tech-community-metrics: https://wikimedia.biterg.io shows 2017 contributors who are not listed in mediawiki-identities/wikimedia-affiliations.json - https://phabricator.wikimedia.org/T161235#3128203 (10Aklapper) [14:16:28] 10Analytics-Tech-community-metrics: https://wikimedia.biterg.io shows 2017 contributors who are not listed in mediawiki-identities/wikimedia-affiliations.json - https://phabricator.wikimedia.org/T161235#3125956 (10Aklapper) Another example: https://wikimedia.biterg.io:443/goto/0db583aa42df8122ded61bf9342b78d0 (`... [14:16:54] 10Analytics-Tech-community-metrics: https://wikimedia.biterg.io shows 2017 contributors who are not listed in mediawiki-identities/wikimedia-affiliations.json - https://phabricator.wikimedia.org/T161235#3128209 (10Aklapper) p:05Triage>03High [14:20:31] 10Analytics-Tech-community-metrics: Add remaining KPIs to Overview once available in kibana - https://phabricator.wikimedia.org/T116572#3128211 (10Aklapper) [14:28:01] 10Analytics-Tech-community-metrics, 07Regression: Only display organizations defined in Wikimedia's DB (disable assuming orgs via hostnames in email addresses) - https://phabricator.wikimedia.org/T161308#3128217 (10Aklapper) [14:34:08] 10Analytics-Tech-community-metrics: "Last Attracted Developers" lists established developers and developers without a First Commit Date - https://phabricator.wikimedia.org/T161309#3128236 (10Aklapper) [14:34:19] 10Analytics-Tech-community-metrics: "Last Attracted Developers" lists established developers and developers without a First Commit Date - https://phabricator.wikimedia.org/T161309#3128236 (10Aklapper) p:05Triage>03Normal [14:35:01] 10Analytics-Tech-community-metrics: Updated data in mediawiki-identities DB not deployed onto wikimedia.biterg.io? - https://phabricator.wikimedia.org/T157898#3128251 (10Aklapper) >>! In T157898#3124591, @Aklapper wrote: > * there are also some orgs that we don't have in our DB, such as "Debian GNU/Linux". Looks... [14:41:58] ottomata: Hellooooo ! [14:42:15] joal: hello! [14:42:17] ottomata: Do you know how I could access the internetz from a labs instance? [14:42:44] i think you should just be able to, no? [14:42:53] also, why are there 2 sqoop jobs runnign?! :o [14:42:56] https://yarn.wikimedia.org/cluster/scheduler [14:43:04] ottomata: normal ! [14:43:20] ottomata: 2 small wikis (or even up to k=3) in parallel [14:43:32] joal: funny stuff! [14:43:32] oh huh [14:43:37] they get the same job name? [14:44:13] ottomata: looks like so :( [14:44:47] ottomata: I've tried to clone some github repo (https), and didn't manage to (from cdh3-5) [14:45:13] hm [14:46:20] ottomata: it looks like the long-standing mapreduce bug which prevented stricter ferm rules in for mapreduce (https://phabricator.wikimedia.org/T111433) has finally been fixed in February: https://issues.apache.org/jira/browse/MAPREDUCE-6338 are there any plans to upgrade to >= 2.9.0 this year? [14:46:23] joal: i can curl to google.com just fine [14:46:38] ottomata: I couldn't :( [14:46:44] ah nice moritzm! unlikely [14:46:49] we just updated to the latest version of cdh [14:46:53] and it is still hadooop 2.6 [14:47:09] ah, ok [14:47:35] ottomata: disconnected - reconnected - now works [14:47:42] * joal cries in a corner [14:47:44] hm, ok! ;) [14:51:38] (03CR) 10Milimetric: [C: 04-1] Adding renamed tables to sql union statements (031 comment) [analytics/limn-flow-data] - 10https://gerrit.wikimedia.org/r/344055 (https://phabricator.wikimedia.org/T160454) (owner: 10Nuria) [14:51:56] 10Analytics, 10Analytics-EventLogging, 10DBA, 10ImageMetrics, 13Patch-For-Review: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#3128269 (10Nuria) Note that record is from 20170318 , a timestamp before the blacklisting changes... [14:52:48] 10Analytics, 10Analytics-EventLogging, 10DBA, 10ImageMetrics, 13Patch-For-Review: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#3128270 (10Marostegui) >>! In T141407#3128269, @Nuria wrote: > Note that record is from 20170318 ,... [15:00:48] ping joal : standduppp [15:00:54] ooops [15:01:05] 06Analytics-Kanban: Wikistats 2.0. - https://phabricator.wikimedia.org/T130256#3128285 (10Milimetric) Ooh, that prototype is still not ready to be shared, it's still very much early days. That said, I have thoughts on annotations. Dashiki primarily uses dygraphs for timeseries line graphs. Work on dygraphs ha... [15:10:07] 06Analytics-Kanban: Update DataLake History schema to only contain "objective" measures - https://phabricator.wikimedia.org/T157362#3128305 (10Nuria) [15:10:19] 06Analytics-Kanban: Update DataLake History schema to only contain "objective" measures - https://phabricator.wikimedia.org/T157362#3002613 (10Nuria) a:03fdans [15:39:54] joal: i was saying [15:40:03] if the total # of python deps we need to get is small enough l [15:40:05] like < 10 [15:40:12] i think we can try to build them [15:40:15] but i'm worried that the dep chain will get huge [15:40:30] ottomata1: deps for revscoring: https://github.com/wiki-ai/revscoring/blob/master/requirements.txt [15:40:44] yeah, but its the whole chain [15:40:46] Some of them have deb packages in jessie that will fit [15:40:53] I have found them [15:40:57] Some haven't [15:40:57] like, you say, scikit learn is too old, and the new one depends on things that are also too old [15:41:16] The only one I'm afriad of is scikit-learn - rest to manually build is easy [15:41:19] joal: what are the ones currently missing from jessie? [15:41:23] mforns_: the new data then is on projectcounts_raw? [15:41:24] oh that's theo nly one? [15:41:28] oh [15:41:33] nuria, yes [15:41:42] ottomata1: some others, but I think I can manage myself with eggs for those [15:41:46] 06Analytics-Kanban: Kill limn1 - https://phabricator.wikimedia.org/T146308#3128353 (10Milimetric) no worries either way, because the instance hasn't been updated in 6 months and only one person even noticed. [15:41:55] mabe not mmh3, but those (with sklearn) are the only ones [15:42:01] joal, BTW, how did you query projectcounts_raw to get mobile? [15:42:30] mforns_: anything different than desktop IIRC [15:42:57] in domain_abbrev_map access field? [15:43:45] mforns_: https://gist.github.com/jobar/b8d2c5faf126d43f4173606a3d5e695c [15:43:50] joal: wait, so, hm, get me a list of the things you want me to try to build [15:43:50] joal, thanks! [15:43:52] and i will try! [15:43:53] :) [15:45:57] ottomata1: https://gist.github.com/jobar/41e9a586c00b2b7f19d7697276a799e6 [15:46:08] ottomata1: currently playing with that on cdh3-5n [15:46:16] joal, nuria, the access method/site in domain_abbrev_map is not stored in the hostname (i.e. en.m.wikipedia.org) but in the access_site column (en.wikipedia.org, desktop|mobile|zero) [15:46:48] the hostname is always the regular one: en.wikipedia.org [15:47:03] mforns_: right, but projectcounts table dpoes not have access site [15:47:05] so, if you want to filter by access, you have to use that field [15:47:06] mforns_: [15:47:09] https://www.irccloud.com/pastebin/4v2WZRta/ [15:47:43] nuria, no, you have to join with domain_abbrev_map [15:48:01] like in joseph's example [15:48:14] https://gist.github.com/jobar/b8d2c5faf126d43f4173606a3d5e695c [15:49:35] mforns_: Doesn't look good [15:50:32] what do you mean joal? [15:50:40] joal: k, i'll see what i can do! [15:50:53] mforns_: trying again, will paste proper results in a minute [15:51:08] k [15:51:14] ottomata1: Currently trying to compile cython (needed for scikit-learn_ [15:51:26] 10Analytics, 10Analytics-EventLogging, 10DBA, 10ImageMetrics, 13Patch-For-Review: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#3128398 (10Marostegui) I dropped it on all the hosts ``` root@neodymium:/home/marostegui/databases... [15:51:29] 10Analytics, 10Analytics-Dashiki: Change default timeline for browser reports to be recent (not 2015) - https://phabricator.wikimedia.org/T160796#3128399 (10Milimetric) Ok, cool. Updating task description. I don't think there's a way to force it to always aggregate the same, because any number of days could... [15:51:31] querying projectcounts_raw as well [15:51:53] mforns: https://gist.github.com/jobar/aa503a6395e258a6acfee1d77462c54d [15:52:31] joal, ://////// [15:54:30] 10Analytics, 10Analytics-Dashiki: Change default timeline for browser reports to be recent (not 2015) - https://phabricator.wikimedia.org/T160796#3128419 (10Milimetric) [15:55:20] 10Analytics, 10Analytics-EventLogging, 10DBA, 10ImageMetrics, 13Patch-For-Review: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#3128422 (10Nuria) mmm.. master? man, bermuda triangle problem. I was expecting this came from the... [15:55:38] 10Analytics-EventLogging, 06Analytics-Kanban, 10DBA, 10ImageMetrics, 13Patch-For-Review: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#3128423 (10Nuria) [15:57:08] 10Analytics-EventLogging, 06Analytics-Kanban, 10DBA, 10ImageMetrics, 13Patch-For-Review: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#3128426 (10Marostegui) >>! In T141407#3128422, @Nuria wrote: > mmm.. master? man, bermuda t... [15:58:59] ottomata1: are you doing something with joseph? you got 1 sec? [16:06:27] nuria: sure [16:07:11] ottomata1: I think regex in https://gerrit.wikimedia.org/r/#/c/343809/4/hieradata/common.yaml is wrong, should not have '$' at end unless there is something special abbout puppet regexes [16:07:28] joal: are we having our meeting? :) [16:07:51] yes neilpquinn, excuse me I missed time [16:08:10] joal: no problem! [16:08:42] joal, well the reason is, pagecounts_raw do not have mobile nor zero views... [16:08:55] ok mforns :) [16:09:05] pagecounts_all_sites do, but the script that aggregates data, doesn't use pagecounts_all_sites [16:09:08] maybe it should [16:09:17] nuria: why? [16:09:37] that just makes it match the full string [16:09:47] so, the schema name must match one of those words exactly [16:09:48] ottomata1: because Blah_47476 would not be match by ^Blah$ correct? [16:09:50] with nothing before or after [16:09:53] that's correct [16:09:56] but its matching ont het schema name [16:10:00] not the revision [16:10:14] ottomata1: oohhh, it is matching all revisions [16:10:22] ottomata1: ahhh [16:10:25] 10Analytics, 10Analytics-Cluster, 06Operations, 10hardware-requests: EQIAD: stat1002 replacement - https://phabricator.wikimedia.org/T159838#3128458 (10RobH) a:05RobH>03Ottomata Oh, I failed to see the entire raid5 comment in the initial request until now. Raid 5 is horrible for write, and I'm pretty... [16:10:32] ottomata1: then nevermind [16:10:41] https://github.com/wikimedia/eventlogging/blob/master/eventlogging/handlers.py#L206-L215 [16:11:24] ottomata1: ah, yes, should have checked that before [16:11:28] 10Analytics, 10Analytics-Cluster, 06Operations, 10hardware-requests: EQIAD: stat1003 replacement - https://phabricator.wikimedia.org/T159839#3128462 (10RobH) a:03Ottomata @Ottomata: So we've discussed the raid level, but I realize now I never got the overall capacity requirement? Not the disk layout, bu... [16:11:30] ottomata1: thanks and sorry [16:11:37] ya np [16:37:48] 06Analytics-Kanban: Synchronise changes for productionisation of mediawiki history jobs - https://phabricator.wikimedia.org/T160154#3128519 (10JAllemandou) [16:38:22] 10Analytics, 10Analytics-Cluster, 06Operations, 10hardware-requests: EQIAD: stat1003 replacement - https://phabricator.wikimedia.org/T159839#3128520 (10Ottomata) [16:39:10] 10Analytics, 10Analytics-Cluster, 06Operations, 10hardware-requests: EQIAD: stat1003 replacement - https://phabricator.wikimedia.org/T159839#3080357 (10Ottomata) stat1003 is using about 3.2T space right now, and I don't expect it to grow much. If we can get something with at least 4T storage capacity, 6T... [16:39:21] 10Analytics, 10Analytics-Cluster, 06Operations, 10hardware-requests: EQIAD: stat1003 replacement - https://phabricator.wikimedia.org/T159839#3128523 (10Ottomata) a:05Ottomata>03RobH [16:44:00] 10Analytics, 10Analytics-Cluster, 06Operations, 10hardware-requests: EQIAD: stat1002 replacement - https://phabricator.wikimedia.org/T159838#3128531 (10Ottomata) @robh, see the section under **Disks** in the task description. The storage requirements aren't so strict, but as usual, more is better. stat10... [16:44:43] mforns: quick update about pagecounts_raw --> domain_abbrev is really not regular (capital or lower case) [16:44:48] 10Analytics, 10Analytics-Cluster, 06Operations, 10hardware-requests: EQIAD: stat1003 replacement - https://phabricator.wikimedia.org/T159839#3128538 (10RobH) Ok, we can do 4 * 4TB to hit 8TB in raid10. That comes out to more like 7.4TB usable. I'll also ask for quotes for 4 * 6TB to see what the price di... [16:45:14] joal, is that a question :] [16:45:16] ? [16:45:35] nope, I checked the values and they are very not coherent [16:45:48] where in the dumps? [16:45:55] in pagecounts_raw table [16:46:14] pagecounts_raw or projectcounts_raw? [16:46:28] projectcounts_raw my bad [16:46:31] k [16:46:43] ok, I can add a lowercase() in the scala job [16:47:08] mforns: Will you regenerate data anytime soon? [16:47:29] joal, not today, but on monday [16:47:36] Ah, didn't know :) [16:47:45] I thought scala was done already :) [16:48:15] joal, yes, you're right, I can add the lowercase now and leave the job running [16:48:32] actually, do you have 10 mins to batcave? [16:48:38] mforns: as you wisdh, I only thought it was not to be done ?) [16:48:43] mforns: I don't, in meeting [16:48:49] oh k [16:49:48] but if the domain_abbrevs are not lowercased in projectcounts_raw, then we need to regenerate the data correctly [16:50:54] I'm also writing an email with a possible solution to the mobile/zero stuff [17:06:32] joal, nuria, sent email about projectcounts problems [17:18:03] 06Analytics-Kanban, 13Patch-For-Review: Populate aqs with legacy page-counts - https://phabricator.wikimedia.org/T156388#3128613 (10Nuria) [17:21:20] mforns: Let's focus on getting desktop data in aqs, deploying new end point and having a UI that displays so we can have a 301 on the actual reportcard to some new UI that is meaningful. Even if its first version only has desktop data. [17:21:21] As long as end point admits (and I am sure that is already the case) queries for mobile|desktop as access methods it should be really not much work to set up another feeding pipeline with mobile data. [17:21:32] nuria, was reading that [17:22:04] yes, makes sense, but the only thing is the problem with the lowercase/upercase [17:22:19] I think we have to rerun the scala aggregation before loading to cassandra prod [17:23:00] as it takes ~10 hours, I can totally do that in the weekend, it's only waiting time [17:23:22] mforns: Because of the lowercase problem ah ok, ya, that is unrelated to mobile counts [17:24:20] mforns: i see [17:25:29] nuria, yes, np will do that [17:30:39] (03PS1) 10Mforns: Lowercase domain abbreviations in projectcounts aggregation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/344665 (https://phabricator.wikimedia.org/T156388) [17:32:16] (03CR) 10Mforns: "No rush to merge and deploy, I can run the job by checking out the code." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/344665 (https://phabricator.wikimedia.org/T156388) (owner: 10Mforns) [17:34:33] mforns: if you need a second pair of eyes while you do it you can count with my axe :) [17:34:45] fdans, :] [17:35:13] xD [17:35:58] fdans, I don't think there's anything super difficult in re-running the aggregation job now, but if you want to pair, we can batcave [17:36:30] or should I say ggloin? [17:37:03] mforns: you doing it now? [17:37:05] ggloin? [17:37:07] yes [17:37:30] xD it's gimli, son of gloin's IRC nickname [17:37:48] oh my god marcel [17:37:57] you started it :] [17:38:23] I'm in the elrond council (please let's stop with the lotr references, so lame) [17:39:06] ok, xD ping me if you want to pair later, I'll be doing this for some time [17:39:35] sorry, I meant I'm in the batcave now mforns [17:39:43] oh! [17:39:44] k [17:42:10] out for today a-team - Will see you on monday ! [17:42:17] bye joal! [18:06:22] (03CR) 10Nuria: [C: 032] Lowercase domain abbreviations in projectcounts aggregation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/344665 (https://phabricator.wikimedia.org/T156388) (owner: 10Mforns) [18:08:42] (03CR) 10Nuria: "The change needs to be merged with changes that went into master earlier on this week" [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/344114 (https://phabricator.wikimedia.org/T143906) (owner: 10Fdans) [18:09:17] 10Analytics, 10Analytics-Cluster, 06Operations, 10hardware-requests: EQIAD: stat1003 replacement - https://phabricator.wikimedia.org/T159839#3128793 (10Ottomata) ​Ok! [18:12:21] (03Merged) 10jenkins-bot: Lowercase domain abbreviations in projectcounts aggregation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/344665 (https://phabricator.wikimedia.org/T156388) (owner: 10Mforns) [18:12:57] fdans: you guys on batcave? [18:13:05] nuria: yep! [18:15:54] (03PS3) 10Nuria: Add legacy pageviews metric [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/344114 (https://phabricator.wikimedia.org/T143906) (owner: 10Fdans) [18:16:53] (03PS4) 10Nuria: Add legacy pageviews metric [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/344114 (https://phabricator.wikimedia.org/T143906) (owner: 10Fdans) [18:17:24] nuria: thanks for the rebase :) [18:17:41] (03CR) 10Nuria: Add legacy pageviews metric (031 comment) [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/344114 (https://phabricator.wikimedia.org/T143906) (owner: 10Fdans) [18:18:51] thanks a lot ggloin, sorry... fdans [18:19:21] bye team!! have a nice weekend! [18:19:44] http://24.media.tumblr.com/fd7dd1c81e42046d02fc83e40126d997/tumblr_mtrf688ApB1s3xs2ho1_500.gif [18:19:44] joal: success on mmh3! [18:19:47] 2.3.1 [18:19:48] ya? [18:19:59] oh, you need python3, right? [18:20:03] i think i got that... [18:21:06] (03PS1) 10Nuria: Updating Readme [analytics/analytics.wikimedia.org] - 10https://gerrit.wikimedia.org/r/344671 [18:21:19] oh ya, mmh3 good. [18:21:25] ok, going to try scikit learn... [18:22:40] oh joal, you can't use scikit learn 0.18? [18:22:42] there is a version in sid [18:22:45] i could just backport it for jessie [18:43:46] (03CR) 10Nuria: Adding renamed tables to sql union statements (031 comment) [analytics/limn-flow-data] - 10https://gerrit.wikimedia.org/r/344055 (https://phabricator.wikimedia.org/T160454) (owner: 10Nuria) [18:44:34] 06Analytics-Kanban: Collaborate with zero on asiacell report - https://phabricator.wikimedia.org/T161326#3128885 (10Nuria) [18:45:28] 06Analytics-Kanban: Collaborate with zero on asiacell report - https://phabricator.wikimedia.org/T161326#3128898 (10Nuria) [18:54:54] fdans: did you send pull request for pageviews.js? [18:55:23] just read irc backscroll... no comments on maximum geek out [18:55:50] fdans: no pulls present here: https://github.com/tomayac/pageviews.js/pulls [18:57:13] nuria: not yet, it's in a fork in my GitHub profile [18:58:06] fdans: did you maybe not send changes to github or am i looking in the wrong place? https://github.com/fdansv/pageviews.js [18:58:15] nuria: thought we were waiting to deploy to send the pr [18:58:51] nuria: https://github.com/fdansv/pageviews.js/tree/legacy-pageviews?files=1 [18:58:56] fdans: no, no need, right? tomas can look at code before we deploy [18:59:16] fdans: ah, sorry, isee [19:00:08] ok, I'll send it then [19:00:21] fdans: let's send pull request so we can get a CR, we can let him know we will send changes Monday [19:00:27] to AQS [19:01:16] cool, will open it now nuria [19:01:28] fdans: super thanks, will keep an eye for comments [19:17:58] nuria: here's the PR https://github.com/tomayac/pageviews.js/pull/8 [19:19:09] fdans: great, thank you [19:19:23] fdans: how is all access different from all-sites? [19:19:50] fdans: ah sorry agreggates mobile and desktop? [19:20:06] fdans: then it seems that all-sites shoudl be all-projects no? [19:20:36] that's the value coming from the dumps, right nuria ? [19:21:31] 06Analytics-Kanban: Wikistats 2.0. - https://phabricator.wikimedia.org/T130256#3128944 (10Neil_P._Quinn_WMF) >>! In T130256#3127413, @Nuria wrote: > Have in mind that wikistats has a strong community focus, its existance much predates the foundation and, really, its main goal is to motivate our editor community... [19:21:36] fdans: did you talk about this with joseph? cause they pageview api uses ""all-projects" for a total-sum-up-of-pageviews: https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/all-projects/all-access/all-agents/daily/2015100100/2015103100 [19:22:25] fdans: and all-access to add up 'mobile & desktop & apps' [19:23:00] fdans: maybe we need to clarify this monday? [19:23:18] fdans: [19:23:22] probably [19:23:22] will send e-mail [19:23:43] because there is a difference between the `access` and `access-site` parameter nuria [19:23:53] which is what joseph was mentioning the other day [19:24:34] https://github.com/fdansv/pageviews.js/blob/legacy-pageviews/pageviews.js#L50-L58 [19:26:23] nuria: yea will clarify with joseph and marcel monday [19:45:40] fdans: yt? [19:46:18] nuria: yeah! [19:46:29] fdans: question: did you try locally the tab layout for dashiki with reportcard? [19:46:40] yes [19:47:10] fdans: the config only has legacy pagecounts, https://meta.wikimedia.org/wiki/Config:Dashiki:ReportCard [19:48:13] (03PS5) 10Fdans: Add legacy pageviews metric [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/344114 (https://phabricator.wikimedia.org/T143906) [19:48:34] fdans: Take a look at task, https://phabricator.wikimedia.org/T130117 [19:49:20] fdans: I think we agreed that the reportcard needs to include "Pageviews overall, unique devices, Active Editors" [19:50:05] nuria: I was following https://phabricator.wikimedia.org/T143906 [19:50:10] for this change [19:51:01] fdans: ah ok, that is part of what needs to happen [19:51:21] the new endpoint is for pagecounts, not uniques or tops no? nuria [19:51:26] fdans: but we also need more data [19:51:43] fdans: yes, but the reportcard has more data than just legacy counts [19:52:08] nuria: ah yes, sorry, thought I misunderstood [19:52:29] fdans: ok, see old version: http://reportcard.wmflabs.org/ [19:53:17] fdans: we will add things as we can but the legacy counts are just one of the things we want to show [19:53:37] nuria: yes yes, but that would require two more endpoints right? [19:53:47] or would we pull data from dashiki from elsewhere? [19:53:51] fdans: no, that data alredy exists on aqs [19:53:59] fdans: we have pageviews and unique devices [19:54:14] fdans: remember we talked about stiching the data? [19:54:41] fdans: that was in the context of legacy pageviews + new pageviews, [19:54:47] fdans: makes sense? [19:55:55] it does yes, I'm sorry, my belief was that for every item in the reportcard we'd need extra legacy endpoints in aqs [19:56:01] nuria: ^ [19:56:19] ahhh [19:56:26] ya, [19:56:50] I misunderstood the scope a bit :/ [19:56:50] fdans: also some data on reportcard will not come from aqs, it will just be pulling cvs files (from now) [19:56:56] fdans: also some data on reportcard will not come from aqs, it will just be pulling cvs files (FOR now) [19:56:59] right [19:57:25] fdans: an example of cvs files config can be found on browser dashboards: https://meta.wikimedia.org/wiki/Config:Dashiki:SimpleRequestBreakdowns [19:58:25] fdans: this is how we would display monthly new editors, for example : https://analytics.wikimedia.org/dashboards/standard-metrics/#projects=eswiki,itwiki,enwiki,jawiki,dewiki,ruwiki,frwiki/metrics=(Beta)%20Monthly%20New%20Editors [19:59:45] 06Analytics-Kanban: Wikistats 2.0. - https://phabricator.wikimedia.org/T130256#3129099 (10Nuria) >I was not aware that a new incarnation of the report card is planned; perhaps you could give me some details? See: https://phabricator.wikimedia.org/T130117 For our first stab we are just moving it to dashiki and r... [19:59:47] nuria: I see, ok that makes sense [20:03:07] fdans: ok, we can work on this on monday, but notice that the reportcard can start existing now w/o the new endpoint so we can test configuration and such [20:05:00] (03CR) 10Nuria: [V: 032 C: 032] "Self merging changes to readme" [analytics/analytics.wikimedia.org] - 10https://gerrit.wikimedia.org/r/344671 (owner: 10Nuria) [20:06:30] nuria: ok, I'm going to push a new cs now because I overwrote the last one (of course) [20:06:50] fdans: ok, i am going to deploy a (mostly empty) version of reportcard [20:32:16] fdans: i think you might have tested the reportcard with this layout: https://analytics.wikimedia.org/dashboards/vital-signs/#projects=eswiki,itwiki,enwiki,jawiki,dewiki,ruwiki,frwiki/metrics=Pageviews [20:32:44] fdans: But i think we want the tabs one: https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-os [20:33:06] cc milimetric , right? we agreed on reportcard having tabs layout correct? [20:35:58] (03CR) 10Nuria: Add legacy pageviews metric (031 comment) [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/344114 (https://phabricator.wikimedia.org/T143906) (owner: 10Fdans) [20:36:57] fdans: so config of reportcard will look more like this: https://meta.wikimedia.org/wiki/Config:Dashiki:Sample/tabs [20:43:23] nuria: yeah you're right, I tested with that layout [20:43:28] makes sense... I'll redo this on Monday, testing with tabs [20:43:45] fdans: ok, see you monday [20:44:33] o/ [20:48:38] 06Analytics-Kanban: Investigate duplicate EventLogging rows - https://phabricator.wikimedia.org/T142667#3129182 (10Nuria) Continuing with work on this. - I dumped all NavigationTiming events from March 1st to March 15th. - Computer signatures like : md5(concat(event_domInteractive,timestamp,userAgent,webhost,wi... [21:03:44] 10Analytics, 10EventBus, 10Wikimedia-Stream, 13Patch-For-Review: Implement server side filtering (if we should) - https://phabricator.wikimedia.org/T152731#3129211 (10Ottomata) > I'm not sure, it seems like we open a huge attack surface if we're building AJV functions.. Agree. A quick look over https://gi... [21:15:03] 10Analytics, 10EventBus, 10Wikimedia-Stream, 13Patch-For-Review: Implement server side filtering (if we should) - https://phabricator.wikimedia.org/T152731#3129273 (10Pchelolo) Hm... I think event without code generation we still need to disallow some JSON schema features, for example regexes since they co... [21:20:15] 10Analytics-Tech-community-metrics, 10Gerrit: Gerrit patchset 99101 cannot be accessed: "500 Internal server error" - https://phabricator.wikimedia.org/T161206#3124611 (10Paladox) I will report this upstream. [21:22:32] 10Analytics-Tech-community-metrics, 10Gerrit: Gerrit patchset 99101 cannot be accessed: "500 Internal server error" - https://phabricator.wikimedia.org/T161206#3129319 (10Paladox) Reported it here https://bugs.chromium.org/p/gerrit/issues/detail?id=5866 [21:23:00] 10Analytics-Tech-community-metrics, 10Gerrit, 07Upstream: Gerrit patchset 99101 cannot be accessed: "500 Internal server error" - https://phabricator.wikimedia.org/T161206#3129320 (10Paladox) [21:27:49] 10Analytics-Tech-community-metrics, 10Gerrit: Numerous Gerrit patchsets cannot be accessed: "Cannot display change because it has no revisions." - https://phabricator.wikimedia.org/T161207#3124625 (10Paladox) We will be able to delete all drafts with gerrit 2.14. (admins can delete them too). [21:34:00] 10Analytics-Tech-community-metrics, 10Gerrit, 07Upstream: Gerrit patchset 99101 cannot be accessed: "500 Internal server error" - https://phabricator.wikimedia.org/T161206#3129359 (10Paladox) Ah, found the fix https://gerrit-review.googlesource.com/#/c/91583/