[02:07:11] Analytics-Visualization, Analytics-Kanban: Controls help you navigate between the Visual Editor sunburst visualizer and timeseries visualizer {lion} - https://phabricator.wikimedia.org/T89254#1048847 (kevinator) [02:07:41] Analytics-EventLogging, Analytics-Kanban: Reliable scheduler collects Visual Editor deployments {lion} - https://phabricator.wikimedia.org/T89253#1048849 (kevinator) [02:08:12] Analytics-EventLogging, Analytics-Kanban: Analyze failure types in Edit Schema VE events [13 pts] {lion} - https://phabricator.wikimedia.org/T89725#1048851 (kevinator) [02:08:38] Analytics-Visualization, Analytics-Kanban: New host/lab environment for Visual Editor visualizations in labs that can report usage metrics [13 pts] {lion} - https://phabricator.wikimedia.org/T89255#1048853 (kevinator) [02:08:52] Analytics-EventLogging, Analytics-Kanban: Reliable scheduler computes Visual Editor metrics [21 pts] {lion} - https://phabricator.wikimedia.org/T89251#1048855 (kevinator) [02:09:02] Analytics-Visualization, Analytics-Kanban: Build high level timeseries view of key metrics [8 pts] {lion} - https://phabricator.wikimedia.org/T88367#1048856 (kevinator) [02:09:32] Analytics-EventLogging, Analytics-Kanban: Script adds indices to the Edit schema on analytics-store [5 pts] {lion} - https://phabricator.wikimedia.org/T89256#1048859 (kevinator) [02:13:34] Analytics-Kanban: Analyze difference in Edit Schema "bounce rates" across wikis {lion} - https://phabricator.wikimedia.org/T89726#1048868 (kevinator) [02:13:47] Analytics-Kanban: Analyze device type and how it influences Edit Schema events {lion} - https://phabricator.wikimedia.org/T89728#1048870 (kevinator) p:Triage>Normal [02:13:55] Analytics-Kanban: Analyze difference in Edit Schema "bounce rates" across wikis {lion} - https://phabricator.wikimedia.org/T89726#1043805 (kevinator) p:Triage>Normal [02:14:39] Analytics-Kanban: Analyze different types of users in the context of Edit Schema events {lion} - https://phabricator.wikimedia.org/T89729#1048878 (kevinator) p:Triage>Normal [02:17:04] Analytics-Visualization, Analytics-EventLogging, Analytics-Kanban: Fully instrument editing experiences {epic} {lion} - https://phabricator.wikimedia.org/T89924#1048886 (kevinator) NEW [13:42:56] Wikimedia-Git-or-Gerrit, Analytics-Tech-community-metrics: Basic metrics about contributors exercising +2/-2 permissions in Gerrit - https://phabricator.wikimedia.org/T59038#1049662 (Qgil) Thank you, this is going in the right direction. About the graphic, I wonder why there is such spike on April 2013. The... [15:08:08] Analytics-Visualization, Analytics-Kanban: Build high level timeseries view of key metrics [8 pts] {lion} - https://phabricator.wikimedia.org/T88367#1049772 (kevinator) Open>Resolved code reviewed and merged. [15:19:12] Analytics-EventLogging, Analytics-Kanban: Script adds indices to the Edit schema on analytics-store [5 pts] {lion} - https://phabricator.wikimedia.org/T89256#1049806 (mforns) Open>Resolved [16:04:54] Wikidata, Analytics: active user statistics that have less lag than wikistats - https://phabricator.wikimedia.org/T88121#1049932 (Lydia_Pintscher) p:Triage>Normal Yes that is enough. What is usually interesting for me is the last 30 days. [17:02:54] Analytics-Kanban, Analytics-Visualization: New host/lab environment for Visual Editor visualizations in labs that can report usage metrics [13 pts] {lion} - https://phabricator.wikimedia.org/T89255#1050104 (Milimetric) a:Milimetric>Nuria [17:23:28] Analytics-Kanban: Analyze device class(mobile/desktop) and how it influences Edit Schema events {lion} - https://phabricator.wikimedia.org/T89728#1050123 (kevinator) [17:23:58] Analytics-Kanban: Analyze device class(mobile/desktop) and how it influences Edit Schema events {lion} - https://phabricator.wikimedia.org/T89728#1043833 (kevinator) a query of the data reveals 0 rows for phone, tablet, other blocked until we get more info from them (maybe it's not instrumented yet) [17:25:09] Wikidata, Analytics: active user statistics that have less lag than wikistats - https://phabricator.wikimedia.org/T88121#1050129 (JanZerebecki) Then this might be sufficient: $ date -d '-30days' --iso 2015-01-20 [wikidatawiki]> select count(*), change_user_id from wb_changes where change_time > 20150120000000... [17:30:11] Analytics-Kanban: Analyze different types of users in the context of Edit Schema events {lion} - https://phabricator.wikimedia.org/T89729#1050151 (kevinator) [17:30:33] Analytics-Kanban: Analyze different types of users in the context of Edit Schema events {lion} - https://phabricator.wikimedia.org/T89729#1043859 (kevinator) [17:35:59] Analytics-EventLogging, Analytics-Kanban: Tune Sampling rate of eventlogging navigation timing events - https://phabricator.wikimedia.org/T89848#1050167 (kevinator) p:Triage>Normal [17:43:16] Analytics-EventLogging, Analytics-Kanban: EL alarms should be included just in the tugsten host - https://phabricator.wikimedia.org/T89469#1050188 (kevinator) p:Triage>Normal [17:43:22] VisualEditor, VisualEditor-Performance, Analytics-Engineering, Analytics-Kanban: Report on the central tendency for length of pages which are edited for VisualEditor performance benchmarking - https://phabricator.wikimedia.org/T89788#1050190 (kevinator) p:Triage>Normal [17:45:53] Analytics-Kanban: Hand off of Christian's MaxMind geolocation databases repository - https://phabricator.wikimedia.org/T89453#1050197 (kevinator) p:Triage>Normal [17:49:49] Analytics-Kanban: Hand off of Christian's MaxMind geolocation databases repository - https://phabricator.wikimedia.org/T89453#1050215 (kevinator) p:Normal>Low [17:54:48] VisualEditor, VisualEditor-Performance, Analytics-Engineering, Analytics-Kanban: Report on the central tendency for length of pages which are edited for VisualEditor performance benchmarking - https://phabricator.wikimedia.org/T89788#1050231 (Milimetric) Thanks for the tag, Grace. James, how about a breakdow... [17:56:36] VisualEditor, VisualEditor-Performance, Analytics-Engineering, Analytics-Kanban: Report on the central tendency for length of pages which are edited for VisualEditor performance benchmarking - https://phabricator.wikimedia.org/T89788#1050237 (Jdforrester-WMF) >>! In T89788#1050231, @Milimetric wrote: > Thanks... [18:00:42] ottomata|afk: can we talk 5 min about Ashwin's access in batcave? [18:01:01] yup! i am no longer afk! [18:01:19] i thikn batcave is taken [18:01:22] leila: [18:01:33] am here: https://plus.google.com/hangouts/_/gs5dv2bj5cxj6ooo32u4uqcg44a [18:01:42] will be there in 2 min ottomata [18:01:44] k [18:03:46] VisualEditor, VisualEditor-Performance, Analytics-Engineering, Analytics-Kanban: Report on the central tendency for length of pages which are edited for VisualEditor performance benchmarking - https://phabricator.wikimedia.org/T89788#1050267 (Milimetric) > That'd be great. :-) The initial impetus of this requ... [18:05:15] milimetric, around? [18:05:34] milimetric, per your request - https://gerrit.wikimedia.org/r/#/c/191530/ )) [18:05:36] yes yurikR but in a meeting [19:03:49] wow. [19:03:56] there are IRS scammers! [19:03:58] http://www.irs.gov/uac/Newsroom/IRS-Warns-of-Pervasive-Telephone-Scam [19:04:06] i totally just had some weird phone conversations [19:04:23] it was suspcious at first, but they seemed pretty legit! [19:04:33] they started seeming weirder [19:04:36] then i googled, found this [19:04:45] and started being more resistant to their questions [19:05:03] then they told me they were going to fax something to my county and put a warrant out for my arrest, and hung up [19:09:28] hahaha, ottomata I got a phone call from someone at "Microsoft Corporation" who was calling me because my computer had sent a distress signal to him. [19:09:35] He wanted me to visit a website and install some software [19:09:44] haha [19:09:47] I followed along with him, it was crazy [19:10:34] yurik: cool, I'll review it! [19:18:30] leila: ssh key changed. [19:18:41] thanks ottomata. sorry about it. [19:18:45] np [20:07:25] (CR) Bmansurov: [C: 2] Integrated logging [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/180828 (https://phabricator.wikimedia.org/T84892) (owner: Rtnpro) [20:07:35] (Merged) jenkins-bot: Integrated logging [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/180828 (https://phabricator.wikimedia.org/T84892) (owner: Rtnpro) [20:19:09] hey nuria :). Now that I'm back I'm going to hit up the host extractor UDF tomorrow, if that makes sense? [20:23:27] (PS5) QChris: Add media file consumption reports [analytics/refinery] - https://gerrit.wikimedia.org/r/191118 [20:25:22] qchris: i will review for you tomorrow! [20:25:35] :-) [20:25:41] No time pressure. [20:38:49] (CR) QChris: Add media file consumption reports (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/191118 (owner: QChris) [20:50:49] milimetric, yt? [20:50:58] hi mforns [20:51:08] hi!, one comment about generator.py [20:51:38] yea [20:51:54] today, for graph_full executions (not timebound), the last point of data is recalculated every hour [20:52:08] is that required for the new scheduler? [20:54:03] hm [20:54:11] for example, here: http://mobile-reportcard.wmflabs.org/#apps-graphs-tab [20:54:15] i know what you mean [20:54:19] the last point is a daily measure [20:54:22] i'm thinking why that would ever be necessary [20:54:38] other than the scheduler isn't doing what we want it to do and we overcome that by just recomputing everything [20:54:38] and every hour the daily value is updated [20:54:40] I say no [20:54:54] oh wait [20:55:01] only the last point? [20:55:01] yes [20:55:03] yes [20:55:12] woo! [20:55:16] who wants some cool news? [20:55:22] well, in fact, the whole values are updated [20:55:22] :) [20:55:25] all of us do Ironholds [20:55:28] xD [20:55:37] mforns: right, so we do *not* want to recompute all the values [20:55:49] but we *do* want to recompute the "current" value (most recent one) [20:55:53] sampled logs + R implementation of new pageviews definition + 1 January 2015 == 632m pageviews [20:55:58] so if the desired output is daily, we want hourly recomputation [20:55:59] milimetric, ok [20:56:05] if it's monthly, we want daily computation [20:56:16] that part is good, in generate.py [20:56:16] hadoop + new pageviews definition as a UDF + 1 January 2015 == 638m [20:56:24] */the system works/* [20:56:38] :) that is awesome Oliver, gj [20:56:48] also - did you see the travel email [20:57:14] woohoo Ironholds [20:57:28] I did! I'm gonna set it up on Monday and get my own ticket if that's too late [20:57:32] I didn't realise it was so close! [20:57:44] milimetric, ok thanks! [21:32:49] milimetric, I remember you created a task in phab for wikimetrics auth through google not working? [21:32:55] I can not find it [21:33:19] laters all! [21:33:21] oh right! I totally forgot about that... the task is probably lost [21:33:28] just start a new one? :/ [21:33:31] milimetric, ok [21:41:12] ironholds: sorry i totally missed your ping [21:41:39] Ironholds: you mean this one right? https://gerrit.wikimedia.org/r/#/c/188588/ [21:41:50] nuria: how is the backfill going? [21:42:50] ori: well i used the code in mainline to backfill and it was dropping events, so i stop and took some time to add logging and such to research what was going on [21:42:57] what port does wikimetrics run on in vagrant? [21:43:21] kevinator: 5000 [21:43:40] thanks [21:43:54] ori: changes i did for backfilling are here: https://gerrit.wikimedia.org/r/#/c/190139/ [21:44:17] ori: some of them are backfilling specific, others I will move to mainline [21:44:19] nuria: ah, i hadn't reviewed because it had the [WIP] subject [21:44:40] ori: ya, i was just saying so you know what changed [21:45:43] ori: but issue can be summed up like: periodic thread cannot insert fast enough when it dies there are wayyyyy too many events for the main thread to clean up [21:46:21] yeah it doesn't make sense for the periodic thread to be used for backfilling [21:46:23] ori: sometimes 70.000 and that is what causing the big inserts we see in the logs, too big of a payload makes mysql "disconnect" [21:47:21] ori:the periodic thread has the same issues if it's used for backfilling than it has on regular execution, for backfilling I just throttled it: [21:47:39] https://gerrit.wikimedia.org/r/#/c/190139/7/server/eventlogging/handlers.py [21:48:22] Ori: but timewise, our vanadium box takes 7~8 seconds to insert a regular batch of 1000 events (for several schemas) [21:48:45] this is very useful; are you documenting this somewhere? [21:49:05] (i have a cold/fever so i am pretty stupid right now) [21:49:13] ori: I found that out today, testing batch sizes, will document once i have it fix [21:49:26] thank you so much [21:49:30] (seriously!) [21:50:08] Analytics-Wikimetrics, Analytics-Kanban: Wikimetrics authentication through Google accounts is broken - https://phabricator.wikimedia.org/T90009#1051418 (mforns) NEW [21:50:10] ori: i will keep you no more but it comes down to periodic thread not being able to sustain current throughput and 1 bug when signaling among threads [21:50:34] ori: thread communication is HARD no matter the language [21:51:50] Analytics-Visualization, Analytics-Kanban: Configure limn-mobile-data logging to output into console, file and/or logstash - https://phabricator.wikimedia.org/T84892#1051439 (mforns) Open>Resolved [21:51:51] Analytics-Engineering, Analytics-Visualization: [Volunteer] Improve Generate.py [13 pts for the Analytics Eng team] - https://phabricator.wikimedia.org/T76407#1051440 (mforns) [22:01:17] kevinator: Try `vagrant ssh` to get into the vm and then `sudo start wikimetrics-web` to see if you get a better error message [22:01:36] I haven't played with the wikimetrics role before [22:02:11] hmm, no error messages: wikimetrics-web start/running, process 3178 [22:02:41] nothing happens when i go to port 5000 though. [22:02:47] I get a page not available [22:02:53] Did it stay running? [22:03:04] * bd808 enables role to see what happens for him [22:03:15] how can I tell if it stayed running? [22:03:40] status wikimetrics-web ? [22:04:04] wikimetrics-web stop/waiting [22:04:14] ok so it died [22:04:48] try looking in /var/log/upstart for a wikimetrics-web log file [22:05:02] * bd808 waits for vagrant provision to run [22:05:34] I can confirm that vagrant provision just gave me the same error so it looks to be repeatable [22:05:36] nuria: wow, can't believe it takes so long to insert [22:06:09] that's a very simple explanation though [22:06:44] so doesn't this mean that we've been dropping events ever since the batch inserts started? [22:06:59] bd808: the last line in wikimetrics-web.log is ImportError: No module named _winreg [22:08:01] kevinator: Ok. sounds like a dependency may be missing then [22:08:14] I'm almost at a place to debug myself [22:08:30] :-) [22:09:20] nuria, yep :) [22:09:26] (sorry, typed it out, apparently missed the enter key) [22:22:05] kevinator: why is it trying to load a windows registry library? That's weird [22:22:56] This smells a bit like a werkzeug bug [22:22:57] * kevinator doesn’t know [22:23:31] This is the module that is failing to load -- https://docs.python.org/2/library/_winreg.html [22:28:40] MediaWiki-Vagrant, Analytics-Wikimetrics: Wikimetrics-web startup fails on Ubuntu 14.04 MediaWiki-Vagarnt due to _winreg load failure - https://phabricator.wikimedia.org/T90017#1051574 (bd808) NEW [22:29:40] milimetric: but rememeber this is backfilling so "distinct" inserts [22:29:45] *remember [22:29:58] milimetric: i have to get "real" numbers for batch inserts [22:30:33] milimetric: no, actually we have not dropped events prior to the 4th /5th which is still a mystery [22:31:37] bd808: there is no mention of that on wikimetrics code, are we sure it's not puppet apache on vagrant? [22:32:34] bd808: "werkzeug bug" ?? [22:33:01] It looks like it might be a bug in one of the python debs that we install. The only place I can see mention of _winreg is in urllib3 [22:33:23] nuria: that's still a probabilistic statement, that we have not dropped events. It just "looks" like we haven't, but we won't know for sure until we try to backfill or otherwise compare the log with the db [22:33:24] werkzeug is the stuff that flask is built on [22:33:56] bd808: ah I see , ah , ok , I though "werkzeug" was a like "quatum bug" or something ... ok [22:34:11] :) that's a weird name in fairness [22:34:12] milimetric: no we KNOW cause i already did that [22:34:20] oh ok [22:34:21] milimetric: if not i wouldn't say [22:34:21] good [22:34:42] milimetric: of course man, who do you think you are talking to [22:34:56] sorry, I did not mean any insult [22:35:08] I was just going off last time we talked when we were just talking about the relative number of events [22:35:16] milimetric: just talked to matt [22:35:20] that's too weird then, that it never happened before [22:35:33] ya, that .... mmmm... i cannot explain ... [22:35:37] which troubles me [22:36:05] cause as one of my coworkers said to me once " what you do not understand will break in ways that will always surprise you" [22:37:00] agreed, I always obsess about those things [22:37:09] and it usually pays off [22:39:08] bd808: I'm spinning it up on my previously working instance, and I will git pull first wikimetrics and then mediawiki-vagrant to try and help you guys debug [22:39:14] milimetric: i will take a look at the latest code again, see if something jumps out, matt was super patient to listen and i think is great than someone else becomes familiar with codebase [22:39:37] yeah, if we're to support this ourselves, we have to build a cross-team team [22:46:07] bd808 / kevinator: I have no problems with the latest mediawiki-vagrant and latest wikimetrics on Ubuntu 14.04 [22:46:20] anything you'd like me to try? [22:46:41] (I don't care if I hose my installation since I don't develop in vagrant anyway) [22:49:57] milimetric: weird. I'm not sure what to check if you aren't getting the same failure [22:50:12] maybe we could compare python package versions? [22:52:18] MediaWiki-Vagrant, Analytics-Wikimetrics: Wikimetrics-web startup fails on Ubuntu 14.04 MediaWiki-Vagarnt due to _winreg load failure - https://phabricator.wikimedia.org/T90017#1051640 (bd808) Installed python system packages: ``` $ dpkg -l|grep ^ii|awk '{printf "%-30s %s\n", $2, $3}'|grep python dh-python... [22:52:46] milimetric: want to run `dpkg -l|grep ^ii|awk '{printf "%-30s %s\n", $2, $3}'|grep python` and compare to the list I put in the bug? [22:57:49] MediaWiki-Vagrant, Analytics-Wikimetrics: Wikimetrics-web startup fails on Ubuntu 14.04 MediaWiki-Vagarnt due to _winreg load failure - https://phabricator.wikimedia.org/T90017#1051657 (Milimetric) I ran the same on my working version, and here's the diff: $> diff my.pythons bug.pythons 12,14c12,14... [23:00:20] bd808: looks like the urlib3 is indeed different, none of the other differences seem important (I pasted them into the bug there) [23:00:21] Analytics-EventLogging, Analytics-Engineering, Analytics-Kanban: Spike on requirements to prune EL data {oryx} - https://phabricator.wikimedia.org/T89293#1051681 (kevinator) [23:00:51] Yeah I was just looking at that [23:02:51] The only other place I see _winreg in python2.7/dist-packages is dateutil but we apparently have the same version of that [23:05:00] commenting the _winreg line out in urllib3 doesn't fix it [23:09:37] I can fix it by hacking /usr/lib/python2.7/dist-packages/six.py to catch and ignore the import failure [23:09:43] but that's not really a fix [23:11:06] sorry i gotta run - thanks for the investigation bd808 [23:11:25] o/ [23:16:56] MediaWiki-Vagrant, Analytics-Wikimetrics: Wikimetrics-web startup fails on Ubuntu 14.04 MediaWiki-Vagarnt due to _winreg load failure - https://phabricator.wikimedia.org/T90017#1051722 (bd808) ``` < python-urllib3 1.7.1-1build1 --- > python-urllib3 1.7.1-1ubuntu0.1 ``` ``` $ a... [23:17:20] kevinator: ^ there's horrible hack workaround [23:17:44] but the source of the bug is still a mystery and that hack may break other random things [23:17:56] ok, let me try it right now [23:23:41] bd808: how to I restart the wikimetrics-web service? [23:24:36] sudo start wikimetrics-web [23:25:57] ah, forgot to sudo [23:27:11] I just tried something else and completely hosed python in my VM. :) [23:27:11] thank goodness for it being a VM [23:27:47] :-) [23:28:45] I got it running :-) [23:29:32] bd808: in your hacked patch, I had to indent the __import__(name) for it to work [23:29:40] *nod* [23:30:28] bah I did a no-whitespace diff out of habit [23:32:18] I hand edited the diff. I think it wold still apply [23:32:42] Let's leave that bug open until someone can figure out the real problem [23:34:14] MediaWiki-Vagrant, Analytics-Wikimetrics: Wikimetrics-web startup fails on Ubuntu 14.04 MediaWiki-Vagarnt due to _winreg load failure - https://phabricator.wikimedia.org/T90017#1051771 (bd808) p:Triage>Normal I'm fairly confident that the error is related to that urllib3 version difference but I'm out o... [23:34:19] AnEng peeps: where does the vitalsigns source code live? [23:40:05] Ironholds: in wikimetrics [23:40:12] aha; ta :) [23:42:47] analytics/wikimetrics is the depot, now, there is no sql per metric as is alembic code [23:42:56] sorry sqlalchemy, argh [23:47:32] Ironholds: https://github.com/wikimedia/analytics-wikimetrics/tree/master/wikimetrics/metrics [23:47:33] nuria, yep, got it! :) [23:47:59] I was thinkin' of maybe picking up some interesting stuff there, because I like coding in my off hours and I like y'all having less to do, too :) [23:48:42] Ironholds: in wikimetrics? sure, it's a good environment to touch all layers db, web, ui [23:49:42] sweet! I'll check it out and cruise phab over the weekend [23:52:30] Analytics-Engineering, Analytics-Kanban: Fix issue with database writes not keeping up with events - https://phabricator.wikimedia.org/T90029#1051814 (Mattflaschen) NEW a:Nuria [23:53:16] Analytics-Engineering, Analytics-Kanban: Fix issue with database writes not keeping up with events - https://phabricator.wikimedia.org/T90029#1051814 (Mattflaschen) [23:53:26] Analytics-EventLogging, Analytics-Engineering, Analytics-Kanban: Fix issue with database writes not keeping up with events - https://phabricator.wikimedia.org/T90029#1051814 (Mattflaschen) [23:55:56] Analytics-Wikimetrics: Story: AnalyticsEng uses connection pooling on database URL - https://phabricator.wikimedia.org/T73140#1051827 (Nuria) This no longer applies I think, we are hitting 1 box in lab and we are not using a pool [23:57:54] Analytics-Wikimetrics: Story: Community has documentation on chosen dashboard architecture and alternatives - https://phabricator.wikimedia.org/T69125#1051840 (Nuria) Open>Resolved