[00:19:39] Analytics-Kanban, RESTBase, Services, RESTBase-API, User-mobrovac: Enable rate limiting on pageview api - https://phabricator.wikimedia.org/T135240#2347767 (GWicke) In addition to logging, there are also some metrics for 429 responses. Right now, those are only available globally (see the las... [00:51:44] Analytics-Wikistats, Browser-Support-Microsoft-Edge: Microsoft Edge user agent is not recognized - https://phabricator.wikimedia.org/T104531#2347822 (Danny_B) [03:04:15] Analytics-Kanban, RESTBase, Services, RESTBase-API, User-mobrovac: Enable rate limiting on pageview api - https://phabricator.wikimedia.org/T135240#2292830 (Antigng_) The rate limiting is breaking my bot. [03:42:23] Analytics-Kanban, RESTBase, Services, RESTBase-API, User-mobrovac: Enable rate limiting on pageview api - https://phabricator.wikimedia.org/T135240#2348029 (GWicke) @Antigng_, could you throttle your bot, so that it sends less than 10 requests per second? [04:39:21] Analytics, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown, Patch-For-Review: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#2348044 (kaldari) @Sadads: This is turned on on test.wiki now. Can you see if it's collecting... [08:42:20] Analytics, Wikipedia-Android-App-Backlog: Investigate recent decline in views and daily users - https://phabricator.wikimedia.org/T132965#2348368 (JAllemandou) Data backfilled and checked ! There is a decline, but very similar to previous periods in time.{F4102523} [09:05:12] Analytics, Pageviews-API, Services: Rate limiting breached should be logged also when throttling is enabled - https://phabricator.wikimedia.org/T136769#2348433 (mobrovac) Open>Resolved a:GWicke Merged and deployed. Excesses are now being logged too. [13:24:38] \o elukey! was going to review your patch, but I should wait? :) [13:28:11] Analytics, Revision-Slider, TCB-Team, WMDE-Analytics-Engineering, and 3 others: Data need: User Behaviour when comparing article revisions - https://phabricator.wikimedia.org/T134861#2349061 (Tobi_WMDE_SW) [13:28:57] Hi ottomata ! [13:29:17] ottomata: Loaded some smaller dataset in druid if you wanna play :) [13:30:20] hiii [13:30:22] oh cool1 [13:30:43] ottomata: really better response time for long time spans [13:31:28] pageviews-daily? [13:31:35] yup [13:45:34] milimetric: merged that thang [13:47:01] Analytics-Kanban, RESTBase, Services, RESTBase-API, User-mobrovac: Enable rate limiting on pageview api - https://phabricator.wikimedia.org/T135240#2349140 (Milimetric) @Antigng_ this limit will increase once we get our SSDs set up and make a few more improvements, but as it is people are get... [13:47:39] ottomata: thanks, do you have time to look at that MySQL_python upgrade? [13:52:45] ? [13:52:48] tell me more! [13:57:54] ottomata: sorry, here: https://phabricator.wikimedia.org/T134950#2346819 [13:58:19] basically we deployed this change that needs MySQL_python 1.2.5 but we only have 1.2.3 on stat1003 where reportupdater runs [13:58:38] we should switch to pymysql, that would be better in general [13:58:51] so if you want I can try to do that [13:59:08] otherwise I tested on my local that upgrading from 1.2.3 to 1.2.5 makes the problem go away [14:00:01] heh, milimetric ok, that will take a bit [14:00:02] gotta build a deb [14:00:16] if you want I can build a deb for pymysql [14:00:20] hmmm [14:00:38] ok, lemme talk it over with mforns_afk when he's back [14:00:43] thanks [14:03:31] ottomata: this is available, does that work? https://packages.debian.org/sid/python-pymysql [14:03:45] meaning, if we switch to pymysql can you just use that deb? [14:05:44] Analytics, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown, Patch-For-Review: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#2349161 (Milimetric) This edit should've triggered it, right? https://test.wikipedia.org/w/i... [14:07:21] oh, milimetric that is good! [14:07:23] possibly [14:07:36] ok, I'll see what it takes to migrate [14:07:38] shouldn't be bad [14:07:40] i bet it has binaries [14:07:44] so I might have to rebuild ilt [14:07:51] but, if it is in debian, it might not be so hard [14:07:51] dunno [14:07:56] pymysql is pure python [14:08:00] so probably no binaries [14:09:09] hm ok [14:09:14] worth i try [14:09:40] oh! [14:09:41] milimetric: https://packages.debian.org/jessie-backports/python-pymysql [14:09:47] it is in jessie backports! [14:09:51] oh but this is trusty [14:09:52] hmm [14:09:55] there's a backport [14:10:02] ah, you just wrote that :-) [14:10:51] ottomata: so wait, what version do you have that you can use [14:11:01] 0.6.2 or 0.7.4? [14:11:13] milimetric: Version: 0.6.2-2 is in trusty now [14:11:18] 0 work to install that [14:11:20] ok, I'll test with that [14:11:22] ottomata: the jessie backport is a simple rebuild for jessie, I'd be surprised if there are problems by doing the same for trusty [14:11:34] moritzm: aye, i just checked, it is already in trusty [14:11:36] no it's ok, no trouble yall, I'll test with 0.6.2, it probably works great [14:11:41] or at least, i find it via apt [14:11:44] k [14:16:52] Analytics-Kanban, RESTBase, Services, RESTBase-API, User-mobrovac: Enable rate limiting on pageview api - https://phabricator.wikimedia.org/T135240#2349190 (Antigng_) I could reduce the concurrency by lowering the number of threads in the pool. (Current is 50.) But what if another bot task ru... [14:45:33] (PS1) Milimetric: Switch to pymysql and use autocommit [analytics/reportupdater] - https://gerrit.wikimedia.org/r/292364 (https://phabricator.wikimedia.org/T134950) [14:46:09] ok, ottomata I migrated to pymysql ^ so when you install that I can self-merge or wait for marcel [14:46:31] (tested and it works great even with 0.6.2) [14:46:49] I figure if we need to upgrade it sounded easier to upgrade pymysql than mysql_python anyway [14:56:59] Analytics-Kanban: Gather user feedback from druid prototype for pageview data - https://phabricator.wikimedia.org/T136836#2349360 (Nuria) [15:06:19] milimetric: Hi ! [15:07:49] milimetric: Hi ! (as well :) [15:09:10] mforns: We're ALONEEEEE *sob* [15:09:27] hehehe [15:24:43] oh ok milimetric! [15:24:44] cool [15:24:52] makihng patch not [15:25:18] sorry missed that earlier ping [15:28:34] mforns: any chance with the fix point thing? [15:28:46] joal, working on that [15:29:24] yesterday I showed the code to Dan and he like it, and he spotted a couple bugs [15:29:28] *liked [15:29:38] mforns: cool :) [15:31:14] mforns: I didn't helped you today, but will have time tomorrow [15:31:59] joal, no worries, I think after the talk with you and dan, I have good ideas to implement, I just need a bit of time [15:32:09] milimetric: pymysql installed [15:32:16] mforns: no problemo :) Just Offering if needed :) [15:32:49] joal, sure let's talk tomorrow, I was just saying no problem if you can not today [15:33:04] ok cool :) Indeed I can't I'll gone soon [15:33:30] (in meetings yall) [15:33:31] sorry [15:33:32] :) [15:36:59] don't worry [15:43:50] Analytics, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown, Patch-For-Review: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#2349497 (kaldari) @Milimetric: Yes, EventLogging should be enabled on testwiki. [16:02:04] Analytics-Kanban: Gather user feedback from druid prototype for pageview data - https://phabricator.wikimedia.org/T136836#2349360 (JKatzWMF) - If you don't select user as a filter, then bots are automatically included. This might be a filter that you add in automatically as the likelihood that somebody wa... [16:28:32] madhuvishy: one thing that it would help is narrowing down content-type and is-pageview=false [16:28:34] right? [16:29:35] sure i can do that [16:29:45] madhuvishy: cause this query is going through all "files" including requests that are not pages [16:29:50] i.e css fiels [16:29:54] *files [16:30:03] or js or mov [16:30:09] yup [16:30:11] neither of which will have a page title [16:30:32] I know - which is why i have Null checks on the udf results [16:30:48] madhuvishy: but you are right the udf for get_pageview_info should handle that well [16:32:23] nuria_: is is_pageview=false a thing? or do we just do TRUE or NULL [16:32:51] i think !is_pageview works let me see [16:33:13] if you look at the row that it says processing failed [16:33:23] Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row [16:33:23] {"hostname":null,"sequence":null,"dt":null,"time_firstbyte":null,"ip":null,"cache_status":null,"http_status":null,"response_size":null,"http_method":null,"uri_host":"www.wikipedia.org","uri_path":"http://www.wikipedia.org","uri_query":"","content_type":null,"referer":null,"x_forwarded_for":null,"user_agent":null,"accept_language":null,"x_analytics":null,"ran [16:33:23] ge":null,"is_pageview":null,"record_version":null,"client_ip":null,"geocoded_data":null,"x_cache":null,"user_agent_map":null,"x_analytics_map":null,"ts":null,"access_method":null,"agent_type":"user","is_zero":null,"referer_class":null,"normalized_host":null,"pageview_info":null,"page_id":null,"webrequest_source":"text","year":2016,"month":5,"day":30,"hour":1 [16:33:23] 2} [16:33:30] is_pageview is NULL [16:33:40] !is_pageview would include this row? [16:33:50] nuria_: ^ [16:34:26] madhuvishy: that will get filter with request code=200 and text-html content type [16:34:48] boty .. that looks [16:34:51] *boy [16:35:08] like a bug in the fowrading layer [16:35:22] *fowarding [16:35:27] redirects are also counted as pageviews right? probably not all are 200 responses [16:35:51] 200 and 3xx , let me see i think we use 304 [16:36:09] but the bulk of filtering will be content type [16:36:18] it shouldn't matter though - I tried doing hostname!=null [16:36:21] still failed [16:36:40] as 1 page request has 40 request [16:36:56] and only one of those is text-html [16:37:04] so it woudl reduce 1 order of magnitude [16:37:26] *would [16:37:36] the data we are having to look at [16:38:26] yeah - but the query needs to work first [16:47:05] Analytics, Operations: Jmxtrans failures on Kafka hosts caused metric holes in grafana - https://phabricator.wikimedia.org/T136405#2349644 (Milimetric) [16:49:33] Analytics: Check if we can deprecate legacy TSVs production (same time as pagecounts?) - https://phabricator.wikimedia.org/T130729#2349655 (Milimetric) [16:50:31] Analytics-Kanban: lowercase project parameter - https://phabricator.wikimedia.org/T136016#2349658 (Milimetric) a:Milimetric>Nuria [16:52:46] Analytics-EventLogging, Performance-Team, Patch-For-Review: Support kafka in eventlogging client on terbium - https://phabricator.wikimedia.org/T112660#2349678 (Milimetric) [16:52:48] Analytics-Kanban: Operational improvements and maintenance in EventLogging in Q4 {oryx} - https://phabricator.wikimedia.org/T130247#2349677 (Milimetric) [16:52:50] Analytics, Analytics-EventLogging: Upgrade eventlogging servers to Jessie - https://phabricator.wikimedia.org/T114199#2349675 (Milimetric) Open>declined decided not to do this for now: systemd too complicated for event logging, didn't fit with the current setup [16:54:59] Analytics, Analytics-Cluster: Puppetize and deploy MirrorMaker using confluent packages - https://phabricator.wikimedia.org/T134184#2349683 (Milimetric) [16:55:17] Analytics-Kanban, WMDE-Analytics-Engineering: Remove http://datasets.wikimedia.org/aggregate-datasets/wikidata/ - https://phabricator.wikimedia.org/T125407#2349685 (Ottomata) Open>Resolved a:Ottomata [16:58:57] Analytics-Kanban, Datasets-Webstatscollector, RESTBase-Cassandra, Patch-For-Review: Better response times on AQS (Pageview API mostly) {melc} - https://phabricator.wikimedia.org/T124314#1952692 (Milimetric) [17:01:10] Analytics, Editing-Analysis: Move contents of ee-dashboards to edit-analysis.wmflabs.org - https://phabricator.wikimedia.org/T135174#2290688 (Milimetric) ping @Neil_P._Quinn_WMF we will let you drive this but we got your back, let us know if you need anything [17:01:48] Analytics: Unique devices endpoint Graphana Dashboard {bear} - https://phabricator.wikimedia.org/T132795#2210412 (Milimetric) p:High>Normal [17:03:25] Analytics, Analytics-Cluster: Audit kernel version on analytics worker nodes - https://phabricator.wikimedia.org/T109834#2349727 (Ottomata) p:Normal>Low @elukey you've been doing a bunch of reboots for kernel / java upgrades with Moritz, ja? We just want to make sure the kernel versions across H... [17:04:44] Analytics: pagecounts-ez files are missing data - https://phabricator.wikimedia.org/T132761#2349746 (Milimetric) Open>Resolved This seems to no longer be an issue, reopen otherwise [17:07:31] Analytics-Kanban, Patch-For-Review: Create repo analytics.wikimedia.org with index and build of browser reports for puppet to source and deploy to analytics.wikimedia.org - https://phabricator.wikimedia.org/T134506#2349761 (Milimetric) [17:07:33] Analytics: Deploy browsers reports to analytics.wikimedia.org using fab - https://phabricator.wikimedia.org/T134510#2349759 (Milimetric) Open>declined decided not to do it this way [17:08:23] Analytics: Making tests environment for pageview API deployments - https://phabricator.wikimedia.org/T131773#2178243 (Milimetric) p:High>Normal [17:11:05] Analytics: Investigate where records will al null fields are coming from - https://phabricator.wikimedia.org/T136844#2349786 (Krenair) [17:11:35] Analytics: Figure out a rollback strategy if the release job fails {hawk} - https://phabricator.wikimedia.org/T132179#2190755 (Milimetric) p:High>Normal [17:14:53] Analytics, Graph, Pageviews-API, Patch-For-Review: Unable to get pageviews for the title with ' in the name - https://phabricator.wikimedia.org/T129346#2349818 (Yurik) Open>Resolved a:Yurik [17:20:03] Analytics, Analytics-Cluster, Deployment-Systems, scap, Scap3 (Scap3-Adoption-Phase1): Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2096617 (Milimetric) @greg does this block you all for this quarter? We want to get it done but we can wait until next quart... [17:23:34] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Single Kafka partition replica periodically lags - https://phabricator.wikimedia.org/T121407#1877784 (Milimetric) [17:23:54] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Single Kafka partition replica periodically lags - https://phabricator.wikimedia.org/T121407#1877784 (Milimetric) a:elukey [17:24:28] Analytics-Kanban: Create separate Analytics-CI gerrit group and add maven-release-user - https://phabricator.wikimedia.org/T136221#2349896 (Ottomata) Ok! There was already a JenkinsBot group in gerrit. I added maven-release-user to that, and then gave JenkinsBot Push and Push Annotated Tags on both refiner... [17:25:01] Analytics-Kanban: Create separate Analytics-CI gerrit group and add maven-release-user - https://phabricator.wikimedia.org/T136221#2327290 (Ottomata) @madhuvishy can you check that this works? [17:27:31] Analytics: Stop generating pagecounts-raw and pagecounts-all-sites - https://phabricator.wikimedia.org/T130656#2349907 (Milimetric) 1. stop oozie jobs / bundles etc. 2. remove puppetized code that generates HTML (examples below) 3. announce that this has been done (wait until at least end of June for Kaldari... [17:27:33] Analytics-Kanban: Create separate Analytics-CI gerrit group and add maven-release-user - https://phabricator.wikimedia.org/T136221#2349908 (Ottomata) Also, @hashar, please approve what I did, in case you don't want us to reuse the JenkinsBot group for this. [17:28:45] Analytics-Kanban: Stop generating pagecounts-raw and pagecounts-all-sites - https://phabricator.wikimedia.org/T130656#2142207 (Milimetric) [17:31:27] madhuvishy: did those jenkins gerrit perms for ya, check if they work [17:31:39] ottomata: yup saw that - testing it now :) [17:31:42] thank youuu [17:33:55] (PS1) Milimetric: Improve directions for program metric [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/292390 [17:34:33] madhuvishy: ok if I self-merge that wikimetrics patch and deploy? ^ [17:35:54] (CR) Mforns: [C: 2 V: 2] "LGTM!" [analytics/reportupdater] - https://gerrit.wikimedia.org/r/292364 (https://phabricator.wikimedia.org/T134950) (owner: Milimetric) [17:36:23] Analytics, Analytics-Cluster, Deployment-Systems, scap, Scap3 (Scap3-Adoption-Phase1): Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2349979 (greg) Our quarterly goal was to get all services migrated this quarter; I know it's a tough thing that means work fo... [17:36:50] milimetric, I will check in a while after puppet deploys, that everything is going well [17:37:43] cool, thx [17:39:17] (CR) Milimetric: [C: 2 V: 2] Improve directions for program metric [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/292390 (owner: Milimetric) [17:41:19] omg so good madhuvishy :) [17:41:32] I was like really really fast at deploying wikimetrics before, and this is still nicer :) [17:41:42] :D [17:52:18] Analytics, Analytics-Cluster, Deployment-Systems, scap, Scap3 (Scap3-Adoption-Phase1): Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2350096 (thcipriani) FWIW, the migration process has become more streamlined than it was when the previous analytics service... [17:57:53] Analytics-Kanban: Gather user feedback from druid prototype for pageview data - https://phabricator.wikimedia.org/T136836#2350142 (Milimetric) Thanks for the feedback, @JKatzWMF. Most of the things you mention I agree with, but they're a bit out of scope for us because Pivot is a UI developed by Imply: http... [18:00:38] Analytics, Pageviews-API: Add support for outreachwiki to pageviews API - https://phabricator.wikimedia.org/T132313#2350149 (Nuria) To make this change backwards compatible: 1.create new udf that is_wikimedia_project_pageview() that really is wrapping the pageviewdefinition and excluding regexes 2. a... [18:08:04] Analytics, Analytics-Cluster: Beeline does not print full stack traces when a query fails {hawk} - https://phabricator.wikimedia.org/T136858#2350169 (madhuvishy) [18:13:45] nuria_: can you paste the query you ran? [18:13:51] it keeps failing for me [18:13:57] even with content_type [18:30:19] milimetric: [18:30:34] shouldn't the text be there in the regular cohort upload forms too? [18:30:47] there's some description below that may be we should amend? [18:31:06] there it says [18:31:07] User Names can even contain commas, but these need a project at the end,en [18:46:44] madhuvishy: here it is: https://etherpad.wikimedia.org/p/nuria [18:58:06] madhuvishy: yeah, we should improve all that but I'm thinking it'll be more worthwhile to interface with the wikiedu dashboard. Meanwhile we can just fix whatever's absolutely necessary in wikimetrics [18:58:24] milimetric: ah okay [18:58:51] yeah, so if someone reports something I'll throw up a quick fix [18:59:06] mforns: I'm back, will start hacking the code [18:59:18] milimetric, hi! batcave then? [18:59:31] yep, I'll be there [18:59:35] ok [19:53:29] !log stopping kafka broker and restarting kafka1014 [19:53:31] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [20:10:19] ooook i need a brain bounce! [20:10:24] madhuvishy: mayyyybe? its eventlogging stuff [20:14:03] Analytics-Kanban, Patch-For-Review: Event Logging doesn't handle kafka nodes restart cleanly - https://phabricator.wikimedia.org/T133779#2350753 (Ottomata) WELLL POOP SCOOPS. 1. kafka-python 1.2.0 doesn't really have a non-async mode anymore, and waiting for the produce response future to resolve causes... [20:14:45] ottomata: sure [20:15:11] k batcave looks used, batcave-2! [20:15:15] hang on 2 mins, lemme switch internet [20:15:48] ottomata: https://hangouts.google.com/hangouts/_/wikimedia.org/a-batcave-2 [20:15:50] sure [20:17:49] madhuvishy: https://hangouts.google.com/hangouts/_/wikimedia.org/a-batcave2 [20:18:02] ok joining [20:50:24] bye a-team! ttyt [20:50:36] nite [20:58:46] bye [21:57:23] milimetric: got some time for python brain bounce? [21:57:34] sure [23:52:55] milimetric: posted here: https://github.com/dpkp/kafka-python/issues/717