[08:53:29] (CR) Aklapper: "@Ottomata: This patch has been sitting here for 13 months without a review. Is this still wanted? Can someone please decide (-1, +1, +2) a" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/225485 (owner: Ottomata) [10:20:36] Analytics-Tech-community-metrics, Developer-Relations (Jul-Sep-2016): Identify Wikimedia's most important/used info panels in korma.wmflabs.org - https://phabricator.wikimedia.org/T132421#2571208 (Aklapper) [10:24:07] Analytics-Tech-community-metrics, Developer-Relations (Jul-Sep-2016): Identify Wikimedia's most important/used info panels in korma.wmflabs.org - https://phabricator.wikimedia.org/T132421#2571213 (Aklapper) Updated the task description, to be further edited... To compare, go to https://dashboard.bitergia... [11:51:56] (PS1) Addshore: WikidataArticlePlaceholderMetrics also send search referral data [analytics/refinery/source] - https://gerrit.wikimedia.org/r/305989 (https://phabricator.wikimedia.org/T142955) [11:53:50] (CR) jenkins-bot: [V: -1] WikidataArticlePlaceholderMetrics also send search referral data [analytics/refinery/source] - https://gerrit.wikimedia.org/r/305989 (https://phabricator.wikimedia.org/T142955) (owner: Addshore) [12:02:01] (PS2) Addshore: WikidataArticlePlaceholderMetrics also send search referral data [analytics/refinery/source] - https://gerrit.wikimedia.org/r/305989 (https://phabricator.wikimedia.org/T142955) [12:26:08] (CR) Hoo man: WikidataArticlePlaceholderMetrics also send search referral data (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/305989 (https://phabricator.wikimedia.org/T142955) (owner: Addshore) [12:36:31] hi! I'd like to find out the User Agent of a certain IP. Tried to query webrequest on stat1004 but I don't have access to it not being a member of analytics-privatedata-users AFAIU [12:41:54] how do we proceed in these cases? Should I become a member of analytics-privatedata-users even though this is most likely a one-off query? [12:52:17] Analytics-Wikimetrics, Community-Wikimetrics: Scheduled reports 404 despite alleged success - https://phabricator.wikimedia.org/T143218#2571389 (Nemo_bis) [12:52:19] Analytics-Engineering, Analytics-Wikimetrics: Can't run reports on Wikimetrics - https://phabricator.wikimedia.org/T143399#2571387 (Nemo_bis) [12:52:41] Analytics-Engineering, Analytics-Wikimetrics, Community-Wikimetrics: Can't run reports on Wikimetrics - https://phabricator.wikimedia.org/T143399#2567093 (Nemo_bis) p:Triage>High [13:23:09] Analytics, Analytics-Wikimetrics: Can't run reports on Wikimetrics - https://phabricator.wikimedia.org/T143399#2571441 (mforns) [14:47:45] (CR) Ottomata: "Some comments inline. Here are some more general thoughts I just had." (8 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/303339 (https://phabricator.wikimedia.org/T141476) (owner: Milimetric) [14:52:11] ottomata: I mean, in my opinion we're doing it just because, the way I did it initially was without oozie [14:52:28] it seemed to be the team's opinion that we should do it in oozie [14:53:09] but I worked yesterday and today to try and make this datasets thing work and it's actually a real pain [14:53:19] so I'm fine with scrapping it [14:54:06] I can make my python scripts return proper exit codes in case of failure and we can make the cron email us, right? [14:54:23] milimetric: aye, i'm really sorry i didn' think about this before, we shoudl take a careful look. the dir moving thing you are doing might be nicer to do in oozie than all in script, dunno [14:54:40] no problem at all, I learned a lot of oozie! [14:54:54] which will come in very handy when we do the rest of the work on the scala oozie [14:55:16] nah, that's super easy in script, oozie's just calling hdfs dfs -mv anyway [15:30:27] Analytics-Kanban, Analytics-Wikimetrics: Can't run reports on Wikimetrics - https://phabricator.wikimedia.org/T143399#2571744 (Milimetric) a:mforns [15:33:06] Analytics-Kanban: Browser dashboard blogpost - https://phabricator.wikimedia.org/T141267#2571761 (Nuria) Open>Resolved [15:33:31] Analytics-Kanban: Productionize scala code for edit reconstruction - https://phabricator.wikimedia.org/T142552#2571763 (Nuria) [15:33:33] Analytics-Kanban: Page History: Add unit tests to PageHistoryDataExtractors and PageHistoryBuilder - https://phabricator.wikimedia.org/T142724#2571762 (Nuria) Open>Resolved [15:34:42] Analytics, Analytics-Cluster: Update Kafka MirrorMaker jmxtrans attributes and make MirrorMaker grafana dashboard - https://phabricator.wikimedia.org/T143320#2564526 (Milimetric) p:Triage>Normal [15:35:38] Analytics: Pre-generate mysql ORM code for sqoop - https://phabricator.wikimedia.org/T143119#2557510 (Milimetric) p:Triage>Normal [15:37:25] Analytics, Operations: Implement stats_print in kafkatee - https://phabricator.wikimedia.org/T76345#2571781 (Milimetric) Open>declined [15:38:58] Analytics, Security-Team: Establish a process to periodically review and approve access for hadoop/hue users - https://phabricator.wikimedia.org/T121136#1870462 (Milimetric) Let us know when you'll work on this, we can provide the groups that need to be checked. [15:41:13] Analytics, Analytics-Wikimetrics, Easy: Accented letters seem to be rejected in cohort names - https://phabricator.wikimedia.org/T111611#2571815 (Milimetric) p:Triage>Normal [15:47:49] Analytics, Analytics-Cluster: Investigate getting redirect_page_id as an x_analytics field using the X analytics extension. {pika} - https://phabricator.wikimedia.org/T89397#1035278 (Milimetric) Using the newly reconstructed page title history, we might be able to rebuild some of this data historically.... [15:50:32] Analytics, Analytics-Wikimetrics: Cannot remove invalid members from cohort - https://phabricator.wikimedia.org/T113454#2571871 (Milimetric) Open>Invalid It actually works in production, but it's really slow (this is due to the database having too many users, we'll try and clean that up and maybe... [15:51:06] Analytics-Kanban: Productionize edit history extraction for all wikis using Sqoop - https://phabricator.wikimedia.org/T141476#2571876 (Milimetric) [15:51:07] Analytics, Analytics-Cluster: Automate sqooping of page table into Hive - https://phabricator.wikimedia.org/T89394#2571878 (Milimetric) [15:51:35] Analytics, Analytics-Cluster, Epic: Epic: qchris transition - https://phabricator.wikimedia.org/T86135#2571883 (Milimetric) [15:51:37] Analytics, Analytics-EventLogging, Story: Story: Analytics Eng can monitor database replication lag - https://phabricator.wikimedia.org/T86136#2571880 (Milimetric) Open>Resolved a:Milimetric This is possible now [15:53:39] Analytics, Analytics-Wikimetrics: Once public a report cannot be made private - https://phabricator.wikimedia.org/T113452#2571890 (Milimetric) Open>Invalid This also works, it's just really slow - again database needs to be tuned up a bit. [15:54:23] Analytics-Cluster, Analytics-Engineering, Patch-For-Review: PageView reports by hive-webstatscollector should return undefined values when data is not available - https://phabricator.wikimedia.org/T76406#2571896 (Milimetric) [15:54:25] Analytics, Analytics-Dashiki: Pageview data files for mobile breakdowns: absence of data should not be represented as 'zero' - https://phabricator.wikimedia.org/T78025#2571894 (Milimetric) Open>Invalid we no longer use that data source, and the new source we use always has mobile breakdowns. [15:59:45] Analytics: Provide a ua-parser service using the One True Wikimedia UA-Parser™ - https://phabricator.wikimedia.org/T1336#2571938 (Milimetric) ping @Jdforrester-WMF: just trying to understand how to prioritize this, where/how/why would you need to call this service? [16:04:51] milimetric: that was a good article [16:04:52] the color one [16:19:39] :) yeah, I find stuff like that more useful than actual code sometimes [16:20:28] btw, the national park tour I was talking about: http://transitmap.net/image/28588650090 (Basically the U shape from Badlands to Arches is the sweet spot) [16:27:58] Analytics, Analytics-Wikimetrics: Cannot remove invalid members from cohort - https://phabricator.wikimedia.org/T113454#2572021 (Lokal_Profil) Invalid>Open I still get the "event is not defined" error in the js (running Firefox 48.0 on Ubuntu) [16:59:37] mforns: turns out i am not in the wikimetrics labs projects, could you add me? [17:10:18] nuria_, will try [17:12:20] nuria_, I can see you in the list [17:12:35] as a member [18:07:49] mforns: ah, ok, I was looking in horizon but you are right i can see wikimetrics on wikitech [18:09:34] Analytics: Provide a ua-parser service using the One True Wikimedia UA-Parser™ - https://phabricator.wikimedia.org/T1336#2572631 (Jdforrester-WMF) >>! In T1336#2571938, @Milimetric wrote: > ping @Jdforrester-WMF: just trying to understand how to prioritize this, where/how/why would you need to call this serv... [18:15:39] mforns: you were looking at wikimetrics-01.wikimetrics.eqiad.wmflabs? [18:15:48] mforns: cause i do not see any processes running [18:16:22] https://www.irccloud.com/pastebin/LmVdxUDn/ [18:22:43] mforns: ahem .. now it appears . ??? [18:22:44] ps -auxfw [18:24:17] mforns: ah no , wait if i'm root i do not see all processes [18:26:50] (Abandoned) Milimetric: Oozify sqoop import of mediawiki tables [analytics/refinery] - https://gerrit.wikimedia.org/r/303339 (https://phabricator.wikimedia.org/T141476) (owner: Milimetric) [18:27:09] ottomata: should I move the python scripts to refinery/bin and schedule them from there in puppet? [18:27:20] or put them in puppet? [18:27:35] ja keep them in refinery [18:27:37] hmm [18:27:38] bin... [18:27:40] yeah i guess bin. [18:27:40] hm [18:28:38] milimetric: , not sure it makes sense, but you could put core stuff in python/refinery, and then your bin script could be pretty thin [18:28:38] buuuut [18:28:38] i dunno, yeah bin is fine [18:30:15] ok, cool [18:35:52] nuria_, sorry I was having a snack [18:36:08] yes wikimetrics-01.wikimetrics.eqiad.wmflabs [18:36:38] mforns: np, will ping you once i figure out what is wrong if any, bottom line is that I think we should deactivate all vital signs reports [18:36:40] cc milimetric [18:36:53] aha [18:38:06] nuria_: sounds fine to me, they've been down for 7 months and nobody noticed, so nobody can really make an argument to keep them around [18:38:10] 7 months!!! [18:38:53] milimetric: ya, indeed, let me look at it some more but i will file a ticket to clean up those from dashiki (there are some files we are still pulling) [18:39:54] hehehe [19:09:08] mforns: we have no logs cause uwsgi module changed and I think we are running it without any log formal [19:09:12] *format [19:09:18] aaaaah [19:09:54] nuria_, this would make sense [20:02:25] Hey folks. Is there somewhere to get monthly user-pageview counts per article? [20:02:45] Or is the only option to do my own aggregation? [20:04:40] hi halfak! what do you mean with user-pageview counts? [20:05:24] you mean unique devices counts and pageview counts? [20:06:46] pageview counts from "user" (as in not-bot) [20:07:00] got it [20:07:10] mmm, the pageview api's per-article endpoint has only daily granularity [20:07:39] I think there's a task to add monthly granularity to that endpoint in our backlog, let me look for it [20:07:51] For a lot of work exploring article popularity, daily isn't enough. [20:07:55] Cool. Thanks [20:08:13] halfak: https://phabricator.wikimedia.org/T139934 [20:09:08] halfak, It's in the Q2 column, so next quarter [20:09:31] Thanks mforns [20:13:03] mforns: and also the celery logs are not there , I would assume they will be under upstart but no [20:13:22] nuria_, yes I didn't find any upstart logs... [20:13:33] mforns: uwsgi i think could be fixed to log somewhere but not having celery logs is a pain [20:13:43] yes [20:14:12] nuria_, maybe madhuvishy knows something about this? [20:14:27] mforns: I see how uwsci would break web logs but I think some prior changes broke celery logs [20:14:30] mforns: they are systemd services [20:14:38] use journalctl for logs [20:14:48] aha! [20:15:15] sudo journalctl -u wikimetrics* / wikimetrics-queue [20:15:45] madhuvishy, trying [20:16:23] mforns: ok, i restarted uwsgi and now we get logs here: /var/log/uwsgi/app# tail -f wikimetrics-web.log [20:18:31] nuria_, I think madhuvishy is right, the historical logs are accessible through journalctl [20:18:46] all logs should be in journalctl [20:20:02] madhuvishy: but wait, there are two things 1) when we changed from upstart to systemd logs went to use journalctl [20:20:25] madhuvishy: 2) uwsgi has not logging conf i can see [20:20:46] madhuvishy: I might be totally off here, do let me know if that is the case [20:21:05] nuria_: are you saying you cannot find web logs? they should be accessible through sudo journalctl -u wikimetrics-web [20:22:20] all the services are set up using systemd - web(uwsgi), scheduler and queue(celery) [20:22:33] madhuvishy, if I execute "sudo journalctl -u wikimetrics-web" I get: -- Logs begin at Sat 2016-08-13 03:00:06 UTC, end at Mon 2016-08-22 20:21:35 UTC. -- [20:23:08] mforns: aah - sorry it may be uwsgi-wikimetrics-web? [20:23:36] there you go [20:23:39] cool [20:23:54] i think the uwsgi puppet module automatically prepends uwsgi [20:25:11] madhuvishy: right, but those are not wikimetrics application logs [20:25:49] madhuvishy: ah, wait no, there is access log info there too, i take it back... [20:27:01] madhuvishy: it's http logging ok there, it's that format is a bit strange but it must be cause it is using [20:27:30] the default uwsgi fmt wahtever that is cause our module does not specify any , which is totally fine [20:27:52] madhuvishy: thank you, let me look a bit and see if celery logs have expected info [20:28:22] thanks madhuvishy :] [20:30:05] madhuvishy: thanks again! [20:31:41] mforns: and "/var/log/uwsgi/app" has logs for uwsgi itself [20:32:19] nuria_, I see [20:33:40] mforns: ok, will get back to this tomorrow to see if we can figure out errors for celery but will stop reports from being created any longer as they have been broken for a while [20:35:09] nuria_, 'a while' is soft hehe [20:35:24] yaya [20:35:53] ok [21:30:37] Analytics-Kanban, MW-1.28-release-notes, Patch-For-Review, WMF-deploy-2016-08-23_(1.28.0-wmf.16): Update mediwiki hooks to generate data for new event-bus schemas - https://phabricator.wikimedia.org/T137287#2573585 (Ottomata) We are ready to go! I want to wait until next week's deployment train,... [21:34:17] nuria_: mforns No problem!