[07:57:14] Analytics-EventLogging, Analytics-Kanban: Flow events not validating on EL - https://phabricator.wikimedia.org/T95169#1193303 (Aklapper) [08:57:41] Analytics-EventLogging, Analytics-Kanban, operations: EventLogging query strings are truncated to 1014 bytes by ?(varnishncsa? or udp packet size?) - https://phabricator.wikimedia.org/T91347#1193451 (Tgr) [08:59:33] Analytics-EventLogging, Analytics-Kanban, operations: EventLogging query strings are truncated to 1014 bytes by ?(varnishncsa? or udp packet size?) - https://phabricator.wikimedia.org/T91347#1080600 (Tgr) What is the status of this bug? Is any further work planned, and if so, is there a rough schedule... [10:49:53] Analytics-Wikimetrics: Problem opening Alfagems cohort - https://phabricator.wikimedia.org/T95530#1193678 (Aklapper) [12:36:34] (CR) Joal: Add access_method, client_type and is_zero fields to refined webrequest table. (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/202914 (owner: Joal) [13:53:07] ottomata: do you know how to claim alarms in icinga? [13:53:16] that interface... holy moly [13:54:03] If i go to Alert History, I can see the EL problems from this morning [13:54:18] but where would they be if someone wanted to claim them? [14:00:22] claim them? [14:00:26] you can acknowledge them [14:03:02] Hi milimetric, thx for referal [14:03:17] hi renoirb_, ok, so what would you like to know? [14:03:36] are you at wmf? [14:04:10] We are wondering how we could get to know usage stats: how many edits per day, how many content pages, etc. [14:04:26] I’m not wmf, I work for the W3C. [14:05:08] so you're interested in general about statistics on wikimedia projects? [14:05:54] No, about usage statistics from a MediaWiki deployment, we run mediawiki wmf/1.25wmf branches [14:06:34] Analytics-EventLogging, Analytics-Kanban, operations: EventLogging query strings are truncated to 1014 bytes by ?(varnishncsa? or udp packet size?) - https://phabricator.wikimedia.org/T91347#1194117 (Nuria) No further work is planned. The limitation on the length of events is on varnish.Thus far, lengt... [14:07:17] Of course MW has maintenance/showSiteStats.php but except that, I wonder other ways to get usage reports [14:07:27] for that, renoirb_ oh! I see. so limn is just a graphing front end, you still have to get your data analyzed somehow [14:07:31] we have two tools that may be of interest [14:07:40] ok [14:08:02] one is "wikimetrics" which lets you run standard metrics on top mediawiki databases [14:08:15] there's nothing super wmf specific about it, but I don't know anyone else who's tried to run it [14:08:42] https://github.com/wikimedia/analytics-wikimetrics [14:09:42] the other tool is quarry. This lets you just ad-hoc query mediawiki databases, but it has some nice social features that help with learning how you should query, etc.: http://quarry.wmflabs.org/ [14:10:09] besides those tools, renoirb_, we just run ad-hoc cron analysis. [14:11:32] Analytics-EventLogging, Analytics-Kanban, Wikipedia-iOS-App, Mobile App Sprint 52 - iOS: Some events not validating for MobileWikiAppSavedPages schema - https://phabricator.wikimedia.org/T91290#1194136 (Nuria) Open>Resolved [14:11:38] once you have data, graphing it is easy, and limn may be one of the hardest ways to do it. But I'd suggest getting flat files first and then pinging me [14:13:40] Yesterday I tried with statmediawiki but it didnt complete. Most likely due to the fact that the queries to the database it does calls tables that changed since it was done [14:14:06] Let me see wikimetrics, i’ll run it in my sandbox [14:14:44] MediaWiki-Vagrant \o/ [14:15:51] I guess, milimetric, wikimetrics runs loops and gathers data on MW internals then, right? [14:16:30] renoirb_: you can schedule recurrent reports, which can use one of the standard metrics we have defined, configured the way that you need them [14:16:43] if you have a fixed set of users, you can use the UI to do everything [14:16:45] That’s great. [14:17:08] if you want to get metrics computed on your *whole* wiki, then you might need to use the "admin" script which has some WMF specific things [14:17:31] A few months ago, I couldnt run MediaWiki from wmf branch on MediaWiki-Vagrant. [14:17:37] Did this change? [14:17:48] but like I say, we haven't considered use cases like yours but I'd be happy to hack on it as a volunteer if you need something that's not available [14:18:31] renoirb_: I'm not super familiar with vagrant, I always have problems with puppet, virtualbox, etc. But I usually am ok running the standard branches on it [14:19:04] OK milimetric, that’s what I got too. [14:19:27] But I imagine that if I run a standard branch with a copy of the production database, I’ll be able to run and get the data anyway I guess. [14:19:35] but that shouldn't matter if you just want to play with wikimetrics, a fresh vagrant install out of the box should let you play [14:20:02] yep, let me know if you run into problems [14:20:53] So, just to rephrase and be sure I understand. A way to get more data is to import production database dump into a sandboxed MediaWiki install, and run scripts so it can crunch stats. Is this the idea? [14:23:29] wait... let me re-rephrase it [14:25:57] Just to re-phrase to be sure I get the idea milimetric; A way to get wiki activity reports and statistics, wikimetrics can be run against a sandboxed MediaWiki installation that we import a snapshot of production so it can crunch metrics. Is this the idea? [14:29:10] wow, milimetric, Quarry is cool! [14:49:52] Analytics, Analytics-Cluster, Analytics-Kanban: Install numpy on the cluster - https://phabricator.wikimedia.org/T95550#1194322 (ggellerman) NEW [14:57:54] haha [14:58:01] your kanban is too slow! mwoooahhaha [15:31:16] renoirb_: sorry - meetings all day :) your summary above sounds good [15:33:46] Analytics-Wikimetrics, Community-Wikimetrics: [BUG] Viewing ukwiki cohorts error - https://phabricator.wikimedia.org/T95320#1194547 (Capt_Swing) Hi @egalvezwmf, sorry to hear that. Did you see my comment above? If you can send the file that caused the error for you, we can try to debug this. But without... [15:36:47] Analytics-Kanban, Analytics-Volunteering, Analytics-Wikimetrics, Community-Wikimetrics, Easy: "Create Report" button does not appear when uploading a new cohort - https://phabricator.wikimedia.org/T95456#1194564 (Capt_Swing) p:Triage>Normal [15:43:53] YuviPanda: yo [15:43:55] around? [15:44:10] did you see my email about Quarry? [15:47:29] Analytics-Cluster, operations, ops-eqiad: analytics1020 hardware failure - https://phabricator.wikimedia.org/T95263#1194617 (Ottomata) Thanks Chris, how's it looking? [15:48:09] nuria: I guess you got your job to launch last night after I left? [15:48:43] Analytics, Analytics-Cluster, Analytics-Kanban: Install numpy on the cluster - https://phabricator.wikimedia.org/T95550#1194625 (Ottomata) Open>Resolved a:Ottomata Done! [15:49:38] Analytics-Cluster, operations, ops-eqiad: analytics1020 hardware failure - https://phabricator.wikimedia.org/T95263#1194633 (Cmjohnson) The basic things that would typically fix this did not work. I will have to try a firmware upgrade next but the only firmware update is for server2008 and in .exe fi... [15:50:14] Analytics-EventLogging, operations: Decommission vanadium - https://phabricator.wikimedia.org/T95566#1194634 (Ottomata) NEW a:Ottomata [15:58:33] milimetric do you have a place where you guys keep database queries for the datasources at http://reportcard.wmflabs.org/datasources? [15:58:42] I guess those could be run from Quarry [16:15:24] Analytics-EventLogging, operations, Patch-For-Review: Reclaim vanadium, move to spares - https://phabricator.wikimedia.org/T95566#1194744 (Ottomata) p:Triage>Normal a:Ottomata>Cmjohnson [16:15:58] Analytics-EventLogging, operations, Patch-For-Review: Reclaim vanadium, move to spares - https://phabricator.wikimedia.org/T95566#1194757 (Ottomata) I have decomissioned vanadium. It is powered off and read to be reclaimed. Please note that it currently has a failed disk: https://phabricator.wikimed... [16:16:12] Analytics-EventLogging, operations, ops-eqiad: vanadium failed disk /dev/sda - https://phabricator.wikimedia.org/T94926#1194760 (Ottomata) [16:16:15] Analytics-EventLogging, operations, Patch-For-Review: Reclaim vanadium, move to spares - https://phabricator.wikimedia.org/T95566#1194759 (Ottomata) [16:17:25] Analytics-EventLogging, operations, Patch-For-Review: Reclaim vanadium, move to spares - https://phabricator.wikimedia.org/T95566#1194766 (RobH) [16:19:13] Analytics-EventLogging, operations, Patch-For-Review: Reclaim vanadium, move to spares - https://phabricator.wikimedia.org/T95566#1194771 (Ottomata) [16:19:44] nuria: you back ? [16:27:52] whoa! did you guys know about the units cli tool? [16:27:53] type units [16:28:04] oh, well, at least on my mac it is installed [16:28:05] haha [16:31:49] aharoni: hey! still around? [16:31:55] yes! [16:34:01] Analytics-Cluster, Analytics-Kanban: Report monthly pageviews for the annual report - https://phabricator.wikimedia.org/T95573#1194832 (kevinator) NEW [16:46:11] renoirb_: we have all queries *except* for the ones used to do reportcard.wmflabs.org [16:46:23] ok [16:46:28] because reportcard is based on statistics computed by Erik Z on top of the dumps [16:46:32] so there's no SQL there [16:46:40] I cant finish to build wikimetrics in mediawiki-vagrant [16:46:43] it fails at permissions [16:46:53] when enabling the role? [16:46:55] that's weird [16:46:57] yes [16:47:08] I enabled the role, ran the git submodule update --init [16:47:15] did nothing at the output [16:47:21] ran `git submodule` [16:47:25] got a list of things, [16:47:34] then ran `vagrant reload --provision` [16:47:34] still errors. [16:47:46] * renoirb_ is trying to find the Puppet role [16:47:53] code [16:48:13] renoirb_: what's the error you're getting? [16:48:35] ==> default: Error: Failed to set owner to '1000': Operation not permitted - /vagrant/srv/wikimetrics [16:48:35] ==> default: Error: /Stage[main]/Wikimetrics/Git::Clone[analytics/wikimetrics]/File[/vagrant/srv/wikimetrics]/owner: change from vagrant_share to vagrant failed: Failed to set owner to '1000': Operation not permitted - /vagrant/srv/wikimetrics [16:48:38] that’s the first [16:49:01] MacOS X 10.10, current Vagrant/VirtualBox [16:50:19] I’ve played a bit with wmf roles, they are at https://github.com/wikimedia/mediawiki-vagrant/tree/master/puppet/modules/role/manifests [16:51:03] milimetric ^ [16:51:35] mforns: did you get this when you were provisioning vagrant with wikimetrics enabled? ^^ [16:51:50] I personally haven't seen that, but it might be a mac thing? I'm on Ubuntu [16:52:01] its not clear what’s missing. if in the clone I should have /vagrant/wikimetrics cloned or not [16:52:04] I could do that [16:52:09] milimetric, renoirb_, I haven't seen that either, yet [16:52:43] owner to "1000" that doesn't sound right :) [16:52:54] I am trying by deinit ing them [16:52:55] sounds like a bad username got created somewhere [16:52:55] git submodule deinit . [16:53:02] then re run [16:53:05] git submodule shouldn't be required here [16:53:13] puppet module will be in puppet/modules/wikimetrics/ [16:53:14] all the git commands should be handled by vagrant [16:53:33] i know but there are so many submodules that one broken reference, all the rest breaks. [16:53:37] all you should have to do is set up vagrant, enable the role, and then run vagrant provision [16:54:24] hm, if I were you I'd try it from a fresh install without doing manual commands, to see if you get the same problem [16:54:41] the four of us who have set this up haven't seen that before, and it looks weird [16:55:37] I know that about vagrant and MediaWiki Vagrant. That’s also how I develop WebPlatform.org wiki workspace using MediaWiki Vagrant :) [16:55:57] Just that I manually clone in mediawiki/ from a wmf branch instead of using roles. [16:56:28] But in the present workspace, I exclusively use commands from the docs and it fails. I´m "reverse engineering" where might be the problem. [16:56:57] ... and it **is** a fresh install. [16:57:15] renoirb_, have you enabled the centralauth role? [16:57:39] no. https://github.com/wikimedia/analytics-wikimetrics doesnt say so [16:57:46] the README I mean [16:57:57] and, if you disable wikimetrics role vagrant provision works fine? [16:58:15] Its a fresh git clone, ./setup.sh, vagrant up like https://github.com/wikimedia/mediawiki-vagrant says so. [16:59:02] if you disable wikimetrics role: vagrant roles disable wikimetrics, and then run vagrant provision, does it work fine? [16:59:25] let me try this [16:59:27] ok [17:02:00] yup milimetric, it works fine [17:02:47] should I enable specific roles, along with wikimetrics, milimetric ? [17:03:10] renoirb_: the centralauth role is good to have, but that shouldn't cause the problem you're seeing [17:03:22] it's more optional for one specific feature (expanding cohorts across all wikis) [17:03:42] renoirb_, milimetric, I think, after centralauth expansion, centralauth role is mandatory for wikimetrics [17:03:51] ok, trying with it [17:04:03] oiy, my bad - maybe just listen to mforns :) I'm gonna go grab lunch [17:04:05] I just enabled both, lets run git submodule and provision [17:04:17] what I've done in the last installations is enable centralauth and provision it [17:04:38] renoirb_, and after that, enable wikimetrics and provision again. [17:04:57] If that works, let me know and I'll update the docs [17:04:59] oh, ok. I’ll run provision with centralauth alone first then [17:05:13] yea [17:05:56] I gotta look at centralauth [17:06:33] I’ve worked recently on an IdP system providing OAuth2 tokens. Maybe I could make CentralAuth use it milimetric [17:06:38] oh, mforns [17:06:44] aha [17:07:25] Its not related to analytics though. But the mention of CentralAuth reminded me that I should look at this. [17:07:43] * renoirb_ is running provision with centralauth alone, so far so good. [17:07:51] renoirb_, good [17:08:13] ok, went without problems. Running with wikimetrics role now [17:08:17] ok [17:09:46] same error. [17:10:05] My feeling remains the same. Its a " Failed to set owner to '1000': Operation not permitted - /vagrant/srv/wikimetrics" [17:10:25] which makes me wonder why would puppet scripts do a chown on a folder that’s mounted from the host [17:11:02] renoirb_, can you have a look at the owner of your vagrant installation? [17:11:15] maybe some submodule got owner:root? [17:11:17] its my Mac OS local user [17:11:29] and wikimetrics folder? [17:12:08] its in /vagrant/srv/wikimetrics [17:12:12] ok [17:12:13] vagrant_share:dialout [17:12:16] withing the VM [17:12:53] but its been mounted through NFS, which means that its trying to chown from the client to change roles *inside* the host, which is where I think its the problem. [17:13:03] renoirb_, wait, I don't have any /srv/wikimetrics nor /vagrant/serve/wikimetrics folder... [17:13:21] renoirb_: right, neither do I [17:13:30] Inside the VM, /vagrant/mediawiki is where mediawiki is [17:13:48] If I look at the puppet errors (...) [17:14:09] renoirb_: and wikimetrics should be under /vagrant/wikimetrics [17:14:10] ==> default: Error: Failed to set owner to '1000': Operation not permitted - /vagrant/srv/wikimetrics [17:14:10] ==> default: Error: /Stage[main]/Wikimetrics/Git::Clone[analytics/wikimetrics]/File[/vagrant/srv/wikimetrics]/owner: change from vagrant_share to vagrant failed: Failed to set owner to '1000': Operation not permitted - /vagrant/srv/wikimetrics [17:14:10] ==> default: Notice: /Stage[main]/Wikimetrics/Git::Clone[analytics/wikimetrics]/Exec[git_clone_analytics/wikimetrics]: Dependency File[/vagrant/srv/wikimetrics] has failures: true [17:14:32] renoirb_: if you tried to check it out on your own problems might comes from that [17:14:36] cc mforns [17:14:47] ? [17:15:27] sorry, renoirb_ maybe i am just confusing you further, will let mforns handle it [17:15:33] joal: i am back [17:16:22] nuria, you mean if he has checked out the repos in several steps instead of git clone + git submodule init? [17:16:36] I see. [17:16:40] mforns: right [17:16:52] But that’s not exactly what I did. [17:16:55] Quarry: Quarry sorts by the first column by default despite and ORDER BY clause - https://phabricator.wikimedia.org/T95369#1195015 (Amire80) [17:17:15] I followed instructions from MediaWiki-Vagrant (i.e. no manual clone) [17:17:31] renoirb_, is your ~/vagrant/wikimetrics repo up to date? [17:17:42] on the host [17:17:48] wait, mis spoke. [17:18:19] I cloned MediaWiki-Vagrant, did first run. Then added wikimetrics role. It broke with the error I´m still getting. [17:18:31] I, then, cloned manually so it would be at /vagrant/wikimetrics [17:18:36] Then I discussed with you here [17:18:44] Quarry: Quarry sorts by the first column by default despite and ORDER BY clause - https://phabricator.wikimedia.org/T95369#1187983 (Amire80) [17:18:45] I disabled wikimetrics role [17:18:48] Quarry: Quarry does not respect ORDER BY sort order in result set - https://phabricator.wikimedia.org/T87829#1195024 (Amire80) [17:18:49] ran alone, it worked [17:18:56] ran with centralauth role, it worked [17:19:05] renoirb_, aha, did you run: git submodules init --recursive? [17:19:11] Of course!! :) [17:19:29] and ./setup.sh, yea you said you did... [17:19:44] YuviPanda: I'm looking at the Quarry project in Phab - is it reported? [17:19:49] that’s the way it works. Vagrant role is a ruby shell script helper that makes changes in submodules dependencies, we still have to run `git submodule` at every roles we change [17:20:02] aharoni: yes, in -research and -labs :) [17:20:11] same error [17:20:11] aharoni: oh, wait [17:20:16] aharoni: no, not reported yet. do file one [17:20:16] renoirb_, can you make sure your vagrant/wikimetrics repo is up to date? [17:20:21] ack [17:20:53] its **not** in /vagrant/wikimetrics [17:21:00] but in /vagrant/srv/wikimetrics [17:21:17] * renoirb_ is looking at the code change. Maybe my version is too recent compared to yours and it changed since then [17:21:30] https://github.com/wikimedia/mediawiki-vagrant/commit/ca1d3a25db4770c178b6226669eae17e65071a85 [17:21:50] maybe bd808 knows about that change [17:22:04] renoirb_, I know, I mean in your vagrant repo, the wikimetrics folder is a submodule, can you run: git pull? [17:22:22] oh, let me see [17:22:34] Quarry: it would be useful to run the same Quarry query conveniently in several database - https://phabricator.wikimedia.org/T95582#1195035 (Amire80) NEW [17:23:10] OK, I think I see. [17:23:37] Git clone is broken maybe because of a broken dependency and gitmodules dont do the full looop [17:23:57] (CR) Nuria: Add access_method, client_type and is_zero fields to refined webrequest table. (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/202914 (owner: Joal) [17:24:55] renoirb_, if this does not work, maybe you can try to checkout your vagrant/puppet/modules/wikimetrics repo to: 5f416bbce4188914f48a044a8df09f1c34b07183 [17:25:08] and try provisioning again? [17:26:49] that’s where I’m at. [17:26:55] But this makes me wonder. [17:27:23] Should I run `git submodule update --init` from **within** the VM, OR from **the host** OS? [17:27:33] the host os [17:28:42] The line at https://github.com/wikimedia/mediawiki-vagrant/commit/1f66605fdb811c5a2dbe272a10358b084a8985f8#diff-a56c489c1802d63b580e28bb52bfffd1R200 [17:28:56] is where it changed from /vagrant/wikimetrics to /vagrant/srv/wikimetrics [17:29:03] Its now using Puppet Hiera [17:29:11] %{hiera('services_dir')} [17:29:40] Maybe I should revert my MediaWiki-Vagrant version back a few revisions [17:30:09] renoirb_, you're right [17:30:23] Which MediaWiki-Vagrant commit hash are you using mforns milimetric ? [17:30:44] renoirb_, 29fdf3edf61f7f5cb21ff99c7199c06e3d5252c7 [17:31:03] it's quite recent [17:31:11] Set $wgDBservers to reveal some slag lag issues in testing (attempt #2) [17:31:13] not enough [17:31:18] ... compared to mine :) [17:31:58] https://github.com/wikimedia/mediawiki-vagrant/compare/29fdf3edf61f7f5cb21ff99c7199c06e3d5252c7...master [17:32:41] * renoirb_ sorry for not using Gerrit, but GitHub has so much more meaningful URLs than Gerrit [17:33:12] np, how's provisioning going? [17:34:13] It failed, I just checked out to your revision and trying again. [17:34:14] running [17:34:19] ok [17:39:20] (CR) Ottomata: Add access_method, client_type and is_zero fields to refined webrequest table. (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/202914 (owner: Joal) [17:39:53] It failed again mforns. [17:40:03] I just destroyed the VM [17:40:03] renoirb_, same error [17:40:04] ? [17:40:07] yup [17:40:23] I’m rebuilding the VM only with centralauth [17:40:27] at the same commit you had [17:40:32] hey milimetric [17:40:32] ok, makes sense [17:40:44] milimetric: heard you’re going to be in SF! Have you already booked your flights or is that still a WIP? [17:45:32] renoirb_, this will take some time I guess, please feel free to ping me when finished [17:45:54] ok [17:59:57] mforns, now I get "==> default: Error: Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class ::wikimetrics at /tmp/vagrant-puppet/modules-.../role/manifests/wikimetrics.pp:70 on node mediawiki-vagrant.dev [17:59:58] ==> default: Wrapped exception: [17:59:58] ==> default: Could not find declared class ::wikimetrics [17:59:59] ==> default: Error: Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class ::wikimetrics at /tmp/vagrant-puppet/modules-.../role/manifests/wikimetrics.pp:70 on node mediawiki-vagrant.dev [18:01:15] renoirb_, wow, also never saw this error [18:01:40] renoirb_, that is after enabling the role, right? [18:02:22] yup. [18:02:30] I’m deleting everything [18:02:37] trying again [18:06:30] renoirb_, if at some point you make it work, one thing I discovered last time I installed, is that for wikimetrics to work you must install watchdog: pip install watchdog [18:06:37] ottomata: are we meeting now? :-) [18:06:47] renoirb_, *inside the vm* [18:06:48] ja i'm in meeting [18:06:52] where you guys/? [18:06:54] has this happened before?! [18:06:57] aah! [18:06:57] leila: [18:07:05] in the video link in the calender invite [18:07:06] :D [18:07:20] ottomata, ^ [18:07:32] there are two meeting invites, tghmmmm [18:07:39] let me add you [18:07:48] foudn it [18:07:50] batcave! [18:14:26] We could make a changeset in the puppet module to have the pip module installed too [18:20:13] renoirb_, sure [18:23:40] (PS1) Milimetric: Add basic piwik usage tracking [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/203186 [18:34:40] Arrf ... too late [18:34:55] ottomata: news on the meeting ? [18:35:14] naw, no news, just chitchat :) [18:35:19] k :) [18:35:24] ottomata: about names [18:36:02] I don't like the user_agent bit because, even it is used for retrieving the value, there already are some user_agent specific values [18:36:10] ? [18:36:10] other ideas ? [18:36:16] user agent specific values? [18:36:28] well, user_agent and user_agent_map [18:36:44] with more precise data than the one that would be user_agent_type [18:36:57] Arff ... Names .... [18:37:23] Analytics-Engineering, Engineering-Community, Team-Practices, ECT-April-2015: Tech Talk: May 2015: Kanban - https://phabricator.wikimedia.org/T95202#1195532 (Rfarrand) No problem, lets sort out a June date in a few weeks if that is OK? Let me know when you have the title and summary too. [18:37:43] Analytics-Engineering, Engineering-Community, Team-Practices, ECT-April-2015: Tech Talk: June 2015: Kanban - https://phabricator.wikimedia.org/T95202#1195533 (Rfarrand) [18:41:26] ottomata: user_type [spider | automata | human] ? [18:42:29] i want to define what agent, user and client mean to us [18:42:34] ok, for this field [18:42:47] what is this field trying to convey? [18:42:49] yeah, you are right, that's the correct approach [18:43:07] agent or client seem to indicate the browser or app or something [18:43:16] yep [18:43:19] user_agent has a very specific meaning [18:43:25] yes [18:43:25] the user's browser/app/version [18:43:36] agent by itself makes sense [18:43:38] we are talking about unique clients already [18:43:39] agent_type [18:43:47] and have dropped the term users from uniques [18:44:03] client being an assumed human user [18:44:21] oh, because we filter out bots for that measurement? [18:44:38] Not for this one, but unique client, that's with bots filtered [18:44:45] aye [18:44:46] ok [18:44:56] agent_type might be good [18:45:03] Or so I think [18:45:09] agreed [18:45:50] It's a good mix between reused keyword from user_agent and clear meaning I think [18:45:57] I change and push [18:46:05] k [18:46:32] Shall I use user or human inside ? [18:46:39] We have spider, could be robot [18:47:01] and udf is IsCrawler [18:47:29] I keep user and spider [18:48:49] (PS2) Joal: Add access_method, client_type and is_zero fields to refined webrequest table. [analytics/refinery] - https://gerrit.wikimedia.org/r/202914 [18:49:07] ottomata: With yout permission, I'll deploy tomorrow [18:51:40] hmmm [18:51:46] what are the fields? [18:51:48] sorry [18:51:49] options? [18:52:09] user i guuuessssss [18:52:13] So, agent_type [user|spider] [18:52:23] and automata to be added [18:52:33] if we manage to have a vlear definition [18:53:01] user is weird, especially since it is user_agent [18:53:06] but, better than human [18:53:13] huhu [18:54:18] Gone to eat, will see you guys tomorrow ! [18:54:25] nick joal|night [18:54:34] better [19:02:25] Analytics, Scrum-of-Scrums, Wikipedia-Android-App, Wikipedia-iOS-App, and 4 others: Avoid cache fragmenting URLs for Share a Fact shares - https://phabricator.wikimedia.org/T90606#1195594 (DarTar) @dr0ptp4kt, @deskana: are there related plans for appending an oldid parameter and reviewing the cach... [19:04:03] Analytics, Scrum-of-Scrums, Wikipedia-Android-App, Wikipedia-iOS-App, and 4 others: Avoid cache fragmenting URLs for Share a Fact shares - https://phabricator.wikimedia.org/T90606#1195610 (Deskana) >>! In T90606#1195594, @DarTar wrote: > @dr0ptp4kt, @deskana: are there related plans for appending... [19:05:55] Analytics, Scrum-of-Scrums, Wikipedia-Android-App, Wikipedia-iOS-App, and 4 others: Avoid cache fragmenting URLs for Share a Fact shares - https://phabricator.wikimedia.org/T90606#1195618 (DarTar) @Deskana got it, I'd like to run past you some ideas about the backlink that I discussed with Moiz an... [19:07:01] Analytics, Scrum-of-Scrums, Wikipedia-Android-App, Wikipedia-iOS-App, and 4 others: Avoid cache fragmenting URLs for Share a Fact shares - https://phabricator.wikimedia.org/T90606#1195619 (Deskana) @DarTar That sounds great. Putting it in an email or a Phab task would be ideal, as I'm really overl... [19:09:39] renoirb_ don't know in what tz you are but i am on PST so I will still be here for a while, let me know if you need help [19:11:15] i just finished run mforns [19:11:18] i’m in EST nuria [19:11:29] we have about 6h offset [19:11:58] renoirb_: ok, you guys let me know cause you know mforns is in GMT [19:12:07] oh, ok. [19:14:07] renoirb_, how was it? [19:14:39] I just finished the initial, i’m now running 2/3 run. The one with centralath enabled [19:14:52] ok [19:15:04] Initial run fails if you dont create /vagrant/phab though [19:15:20] but i expect the changes between that particular commit and now has this adressed, or in the works. [19:15:36] maybe in a changeset in gerrit. [19:15:47] aha [19:17:43] (CR) Nuria: "The only thing i can see here is that this is going to count views in every wikimetrics instance, we just want to count views in the produ" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/203186 (owner: Milimetric) [19:19:42] (CR) Milimetric: "There's a piwik configuration that corresponds with this code, and right now it's set to only accept requests from metrics.wmflabs.org. T" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/203186 (owner: Milimetric) [19:31:33] (CR) Nuria: "Well, at worst, if the labs site gets down the js is just going to return a 404, which i do not think is a problem on wikimetrics site at " [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/203186 (owner: Milimetric) [19:51:37] Analytics-Cluster, Analytics-Kanban: Upgrade unique mobile jobs to harvest unique id from x-analytics field - https://phabricator.wikimedia.org/T95606#1195731 (Nuria) [20:15:31] Analytics-Cluster, Analytics-Kanban: Report monthly pageviews for the annual report - https://phabricator.wikimedia.org/T95573#1195838 (kevinator) [20:18:42] Analytics-Cluster, Analytics-Kanban: Report monthly pageviews for the annual report - https://phabricator.wikimedia.org/T95573#1195844 (Heather) Some breakdown would be really great for us. I don't know how much difficulty that adds. (Thank you so much for this!) [20:35:10] (CR) Ottomata: [C: 2] Add access_method, client_type and is_zero fields to refined webrequest table. [analytics/refinery] - https://gerrit.wikimedia.org/r/202914 (owner: Joal) [20:37:46] Analytics-Cluster, Analytics-Kanban: Report monthly pageviews for the annual report - https://phabricator.wikimedia.org/T95573#1195909 (leila) Kevin, the query I ran is: ``` select count(*) as count, month, day from wmf_raw.webrequest where webrequest_source='misc' and year=2015 and uri_path like '%2014/'... [20:41:29] Analytics-Cluster, Analytics-Kanban: Report monthly pageviews for the annual report - https://phabricator.wikimedia.org/T95573#1195920 (leila) @Heather, by breakdown, do you mean showing daily pageviews? that one we will do, you will get an output very similar to what I sent you few days ago, on a monthly... [21:34:11] I could make it run nuria [21:34:30] I had to remove ownership enforcement in modules/puppet/wikistats and now it completed [21:34:40] I see it locally at localhost:5000 [21:34:54] I imported my production database in it, ran maintenance/update.php [21:35:08] ... now I wonder, where do I create "projects" in wikimetrics? [21:35:11] Analytics-Cluster, Analytics-Kanban: Report monthly pageviews for the annual report - https://phabricator.wikimedia.org/T95573#1196118 (Nuria) Seems to me the cluster /oozie is overkill for this given the very few pageviews the report gets. We should probably report pageviews in this cases via a client s... [21:35:35] renoirb_: all righttttt [21:36:16] I see that it expects to run against wikipedia.org but that’s not big deal [21:36:27] renoirb_: so what you have is a testing setup, to test things you can use the "wiki" db that you have in vagrant, project is "wiki" [21:36:36] oh!! [21:36:39] that’s the mapping [21:36:44] renoirb_: this is all "testing" [21:36:51] of course! its local testing :) [21:37:12] renoirb_: so make sure you can see both wiki db and wikimetrics db [21:37:13] but the process will make me figure out what I have to change so i can use it on docs.webplatform.org [21:38:12] oh, the autocomplete works in the default project. I didnt think of writing the database name [21:38:15] renoirb_: right, you can even look arround our staging instance and that will be even closer [21:38:21] renoirb_: so 1st testing: http://localhost:5000/demo/create/fake-wiki-users/100 [21:38:27] this way you can create wiki users [21:38:56] renoirb_: look at db and now you have some fakes [21:39:24] No need, I imported the production database, I had a list of users, it’ll do. [21:39:37] renoirb_: ok, you are a pro, the instance in vagrant [21:39:47] renoirb_: only comes with "wiki" project [21:40:01] but you can from that one infer how everything is setup [21:40:06] in /etc/ [21:40:12] I deleted the (almost empty) wiki database, created an empty one, and imported mine. [21:40:15] there are the db configs to acess that "wiki" db [21:40:22] and the "wikimetrics" db [21:40:43] wikimetrics db is there as a workbench for wikimetrics to do its work. [21:40:45] "/etc/wikimetrics/db_config.yaml" [21:40:54] they are straight out from puppet [21:41:04] I didn’t think that a project, in the context of wikimetrics web app, would be a MediaWiki database. [21:41:19] now it makes sense [21:41:25] renoirb_: that is mw lingo [21:41:43] renoirb_: people equal projects with dbs in which your edits are stored [21:41:49] "enwiki" "dewiki" [21:42:12]