[01:27:29] 06Analytics-Kanban: Split unique devices data for Asiacell and non-Asiacell traffic in Iraq - https://phabricator.wikimedia.org/T158237#3035389 (10Nuria) {F5661953} {F5661955} [01:28:26] 06Analytics-Kanban: Split unique devices data for Asiacell and non-Asiacell traffic in Iraq - https://phabricator.wikimedia.org/T158237#3035390 (10Nuria) Pageviews tagged asiacell look quite small for overall pageviews (these are tagged pageviews, traffic might still be initiated from asiacell but we do not tagg... [01:30:25] 06Analytics-Kanban: Split unique devices data for Asiacell and non-Asiacell traffic in Iraq - https://phabricator.wikimedia.org/T158237#3035391 (10Nuria) {F5661990} {F5661991} [09:27:30] 06Analytics-Kanban: Check abnormal pageviews for XHamster - https://phabricator.wikimedia.org/T158071#3035845 (10Tbayer) >>! In T158071#3034178, @MusikAnimal wrote: >>>! In T158071#3032452, @Tbayer wrote: > ... >> What is the overall percentage of externally referred pageviews? Is it higher than normal? > > O... [10:03:40] 06Analytics-Kanban: Check abnormal pageviews for XHamster - https://phabricator.wikimedia.org/T158071#3035886 (10Tbayer) I ran a query [1] for the most frequent user agents on this page for yesterday, and 43 of the top 50 contained "Safari/534.30" and "Android". Per https://developer.chrome.com/multidevice/user... [11:40:20] 06Analytics-Kanban, 06Operations, 10Traffic, 15User-Elukey: Periodic 500s from piwik.wikimedia.org - https://phabricator.wikimedia.org/T154558#3036155 (10Milimetric) yes, definitely has been going on for a while because we never really looked at it. We assumed the numbers piwik reported made sense because... [11:40:35] oh man, that thing pings EVERYONE :) [11:40:37] sorry [11:40:58] fdans: I was curious what you miss from slack, I never really used it [11:41:13] except for the terrible hangouts experience over the last week, I didn't see much reason to try it [11:44:07] milimetric: I guess it's a UX thing [11:44:27] oh, what do you use for IRC btw? [11:44:34] for me as far as instant messaging goes it's a more human way of communicating [11:44:36] than irc [11:44:39] I use irccloud [11:44:58] huh, that seemed pretty similar to slack to me [11:45:03] I like this dark theme they have [11:45:23] oh maaaybe the dark theme will change my mind... :D [11:45:46] maybe it's a millennial thing, I miss a bit the stupid emojis and gifs [11:46:15] man, but luca's emojis are epic [11:46:38] hm, irccloud has some, never tried them [11:46:50] :eyes::eye: [11:48:11] :thunder_cloud_and_rain::flag-zm: [11:48:28] hm, as John Oliver says sarcastically: "cool" [11:56:49] elukey: if you don't mind, give https://gerrit.wikimedia.org/r/#/c/337672 a merge [11:57:14] oh sorry!!! vacation [11:57:15] aaaah [11:57:41] * milimetric flies around world really fast to turn back time [14:05:51] hellooo [14:12:09] 06Analytics-Kanban, 10Analytics-Wikistats: Simple json model of the site - https://phabricator.wikimedia.org/T157986#3036492 (10Milimetric) I started work on this but got a little lost daydreaming about the site. Here's the abstract structure I've got so far, I'll keep working on it: ``` root... [14:46:05] (03CR) 10Hashar: [C: 04-1] build: run karma test with just "npm test" (031 comment) [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/315659 (https://phabricator.wikimedia.org/T148019) (owner: 10Hashar) [14:47:07] (03PS5) 10Hashar: uild: run karma test with just "npm test" [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/315659 (https://phabricator.wikimedia.org/T148019) [14:51:12] (03CR) 10Hashar: "I rebased the patch. Gave it a try again with: npm install && npm test . That manages to runs some tests apparently but also fails with " [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/315659 (https://phabricator.wikimedia.org/T148019) (owner: 10Hashar) [15:33:40] 06Analytics-Kanban: Add hive table that maps wikiCode to projectName - https://phabricator.wikimedia.org/T158330#3036590 (10mforns) a:03mforns [15:57:01] ashgrigas: I will not be able to attend visuals meeting (cc mforns milimetric fdans) as we have a annual planning meeting that conflicts (scheduled yesterday, sorry). Will check with team after. [15:58:13] Ok I was going to email too - my daughter woke up sick and can't go to daycare so was going to ask if we can meet Monday [15:58:19] nuria: [15:58:27] Is that possible ? [15:58:35] ashgrigas: of course, rescheduling [15:59:03] ashgrigas: but monday is a us holiday, right? [16:00:00] ashgrigas / nuria: tuesday works a lot better for me, gotta deal with mold all weekend [16:59:53] ashgrigas: reschuling [16:59:56] sorry [17:00:00] rescheduling [17:14:44] 10Analytics, 10ChangeProp, 10EventBus, 10Revision-Scoring-As-A-Service-Backlog, and 2 others: Rewrite ORES precaching change propagation configuration as a code module - https://phabricator.wikimedia.org/T148714#3036905 (10Pchelolo) > E.g. send something like {"wiki": "enwiki", "revision": {"new": 123456}... [17:19:05] o/ joal [17:19:08] I have data for you. [17:19:15] It's still being rsync'd [17:19:19] But it'll appear here: https://datasets.wikimedia.org/public-datasets/all/wp10/20170101/ [17:19:31] In less than 30 mins [17:19:55] or you can find it on stat1003:/srv/public-datasets/all/wp10/20170101/ [17:33:04] 10Analytics, 10ChangeProp, 10EventBus, 10Revision-Scoring-As-A-Service-Backlog, and 2 others: Rewrite ORES precaching change propagation configuration as a code module - https://phabricator.wikimedia.org/T148714#3036925 (10Halfak) That endpoint does not exist. I'd been waiting on feedback about it being a... [17:39:31] 10Analytics, 10ChangeProp, 10EventBus, 10Revision-Scoring-As-A-Service-Backlog, and 2 others: Create generalized "precache" endpoint for ORES - https://phabricator.wikimedia.org/T148714#3036943 (10Halfak) [17:39:51] 10Analytics, 10ChangeProp, 10EventBus, 06Revision-Scoring-As-A-Service, and 2 others: Create generalized "precache" endpoint for ORES - https://phabricator.wikimedia.org/T148714#2730864 (10Halfak) [17:39:58] 10Analytics, 10ChangeProp, 10EventBus, 06Revision-Scoring-As-A-Service, and 2 others: Create generalized "precache" endpoint for ORES - https://phabricator.wikimedia.org/T148714#2730864 (10Halfak) a:03Halfak [17:43:10] 10Analytics, 10ChangeProp, 10EventBus, 06Revision-Scoring-As-A-Service, and 2 others: Create generalized "precache" endpoint for ORES - https://phabricator.wikimedia.org/T148714#3036955 (10Pchelolo) @Halfak One moment we need to clarify is what to return in case the wiki is not supported. In MCS we've st... [18:01:11] 10Analytics, 10ChangeProp, 10EventBus, 06Revision-Scoring-As-A-Service, and 2 others: Create generalized "precache" endpoint for ORES - https://phabricator.wikimedia.org/T148714#3036976 (10Halfak) That sounds fine to me. Was thinking that we could return 200 if the request is successfully handled whether... [18:02:20] 10Analytics, 10ChangeProp, 10EventBus, 06Revision-Scoring-As-A-Service, and 2 others: Create generalized "precache" endpoint for ORES - https://phabricator.wikimedia.org/T148714#2730864 (10Halfak) [18:31:07] nuria: how long has /srv/deployment/analytics/refinery/artifacts/refinery-hive.jar had the renamed UDFs? I see the patch was merged on Dec 22 but when did that jar get updated? because we have a bunch of scripts that import affected UDFs [18:32:18] nuria: also why were we not explicitly pinged about this? [18:35:19] 10Analytics, 10ChangeProp, 10EventBus, 06Revision-Scoring-As-A-Service, and 2 others: Create generalized "precache" endpoint for ORES - https://phabricator.wikimedia.org/T148714#3037020 (10Pchelolo) @Halfak Hm... So, in the context of #edit-review-improvements-reviewstream project we might want to reuse th... [18:35:20] nuria: not getting output and then manually trying to run queries and suddenly seeing "FAILED: Class org.wikimedia.analytics.refinery.hive.IsExternalSearchUDF not found // FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask" is not how I would like to find out about these things [18:41:44] nuria: "FAILED: Class org.wikimedia.analytics.refinery.hive.IdentifySearchEngineUDF not found // FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask" that one is the renamed one. not sure why "FAILED: Class org.wikimedia.analytics.refinery.hive.IsExternalSearchUDF" after adding [18:41:44] /srv/deployment/analytics/refinery/artifacts/refinery-hive.jar [18:44:58] nuria: weird, none of the UDFs can be found :\ [18:45:32] even though "Added resources: [/srv/deployment/analytics/refinery/artifacts/refinery-hive.jar]" [18:48:44] nuria: nevermind about earlier, I'm sorry about that. looks like things would be working OK when trying to add and use UDFs with the old names. [18:51:34] 06Analytics-Kanban: Check abnormal pageviews for XHamster - https://phabricator.wikimedia.org/T158071#3037075 (10MusikAnimal) >>! In T158071#3035845, @Tbayer wrote: >> not sure how you got that... >> Let me explain: more traffic means more links to your wikipedia page from any page where traffic is reported on o... [18:55:05] bearloga: ya, changes are backwards compatible (all changes in cluster have to be) [18:55:09] nuria: this is so weird, I can import UDFs from hdfs:///wmf/refinery/current/artifacts/refinery-hive.jar but not from /srv/deployment/analytics/refinery/artifacts/refinery-hive.jar :\ but looks like everything is working fine with the old names :) [18:56:02] (on stat1002) [18:56:26] bearloga: mmm, maybe that jar doesn't have what it should let me see [18:56:33] nuria: thank you for making it backwards compatible! super appreciate that [18:56:52] bearloga: changes will always be, otherwise every software change needs to stop teh world [18:56:54] *the [19:02:05] bearloga: what??? all jars in /srv/deployment/analytics/refinery/artifacts/ look to be corrupted [19:02:46] nuria: oh wow [19:02:52] bearloga: ??? [19:03:07] good luck! :) [19:03:08] bearloga: either that or the zip is defective [19:03:17] bearloga: but that is why they do not work [19:03:33] bearloga: cluster uses code deployed to hdfs;// [19:04:07] bearloga: not "all" but some are definitely funny [19:06:17] bearloga: waittt, maybe is a missmatch from 1.7 to 1.8 [19:07:21] nuria: 1.7 to 1.8? [19:08:12] bearloga: java vs [19:10:01] nuria: ah, for a moment there I thought y'all were using a way newer version of hive than what my account used (1.1.0) [19:10:55] nuria: speaking of, is there any talk within the team about upgrading hive? [19:12:11] bearloga: when we update to cdh 5.10 (i think) we will get some fixes for hive but i think that's it [19:13:59] 10Analytics, 10ChangeProp, 10EventBus, 06Revision-Scoring-As-A-Service, and 2 others: Create generalized "precache" endpoint for ORES - https://phabricator.wikimedia.org/T148714#3037162 (10Halfak) That makes a lot of sense and I'm happy to not have to define a new response type :) [19:16:08] 10Quarry: Users blocked from account creation on meta can not use Quarry - https://phabricator.wikimedia.org/T157342#3037165 (10Capt_Swing) Thank you @tgr and @bd808 for addressing this issue so promptly! [19:18:49] bearloga: ya cluster is all 1.7 and we build 1.7 so not sure what is going on with jar-tf not working for those files [19:23:44] 06Analytics-Kanban: Check abnormal pageviews for XHamster - https://phabricator.wikimedia.org/T158071#3037194 (10Nuria) >We would need to assume a SEO bot operator who is cunning enough to know about this internal Wikipedia page, No, sorry, you missunderstood. That was just an example. "Any" page from a domain... [19:24:27] nuria: wanna ping me when the corrupted jars are fixed? is there any benefit to switching our code that runs on stat1002 to importing jars from hdfs://? [19:25:30] bearloga: I am not going to work on that immedialy as you have a work arround so for now i would change links yes [19:30:59] bearloga: would talk to team about it tuesday [19:33:28] bearloga: i am a bit puzzled and do not want to do any change [19:35:22] nuria: got it, thanks! [19:50:07] 06Analytics-Kanban: Split unique devices data for Asiacell and non-Asiacell traffic in Iraq - https://phabricator.wikimedia.org/T158237#3037269 (10Nuria) {F5675987}{F5661990} {F5661991} [19:50:55] 10Analytics, 10ChangeProp, 10EventBus, 06Services (later), 15User-mobrovac: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#3037272 (10mobrovac) [20:18:14] 10Analytics, 10MediaWiki-API, 10PageViewInfo, 06Reading-Infrastructure-Team, 13Patch-For-Review: Add pageview stats to the action API - https://phabricator.wikimedia.org/T144865#3037301 (10Tgr) 05Open>03Resolved a:03Tgr [21:35:21] 10Analytics, 10EventBus, 10MediaWiki-Vagrant, 06Services (watching): Kafka logs are not pruned on vagrant - https://phabricator.wikimedia.org/T158451#3037453 (10Pchelolo)