[01:21:14] Analytics, Commons, Multimedia, Tabular-Data, and 4 others: Review shared data namespace (tabular data) implementation - https://phabricator.wikimedia.org/T134426#2727495 (Yurik) >>! In T134426#2384523, @MZMcBride wrote: > I'm a little lost here. Is the idea that only data that can be structured... [06:44:16] o/ [06:44:35] Oozie looks a bit better this morning, but the crawler issue is still ongoing [07:54:53] Analytics-Kanban, Operations, Traffic, Patch-For-Review: Varnishlog with Start timestamp but no Resp one causing data consistency check alarms - https://phabricator.wikimedia.org/T148412#2727782 (elukey) Found other occurrences of the same issue but with different URIs on other hosts: ``` - D... [11:14:31] I checked webrequest_stats and we are going much better with the new vk patch [11:15:30] now I am trying to get other occurrences of the remaining issue, hoping to get backend info [11:17:19] Awesome :) Thanks elukey ! [11:20:08] Analytics-Kanban: Migrate the simplest limn dashboards to dashiki tabular {frog} - https://phabricator.wikimedia.org/T126358#2012091 (mforns) Moved to paused until we know next steps in the editor engagement dashboards, the rest is done. [11:20:29] Analytics-Dashiki, Analytics-Kanban: Switch to fetch away from jquery - https://phabricator.wikimedia.org/T148053#2713455 (mforns) a:mforns [11:26:28] * elukey lunch! [11:41:34] Analytics-Dashiki, Analytics-Kanban: Switch to fetch away from jquery - https://phabricator.wikimedia.org/T148053#2728264 (mforns) What about other dependencies that use jquery? - URIjs - typeahead.js - semantic-2 - semantic-datepicker - semantic-ui When we gulp build, jquery is going to be pulled anyway... [12:44:02] joal: regarding https://gerrit.wikimedia.org/r/#/c/305989/6 I'm aware you told me some things, but I cant find them anywhere :/ [12:44:48] addshore: in the comments of the CR ? [12:45:08] joal: there are no comments from you :/ [12:45:13] your comment :) [12:45:46] oh, ahahhaaa! I was sure I had left them somewhere! I didn't check my own comments though! [12:45:51] :D [12:45:58] thanks! [12:46:00] np [13:04:12] elukey: o/ [13:04:17] hola! [13:04:56] ciao1 [13:05:05] ciao1 [13:05:10] wth? [13:05:17] hahah [13:05:28] :D :D :D [13:05:39] i tried to do a faux-regex to replace the 1 with a (bang), and the bang was interpolated [13:05:47] by irssi [13:05:50] ! [13:05:55] well damn [13:06:07] s/damn/damn!/ [13:06:20] dunno what happened now [13:06:29] anyways [13:07:16] elukey: you mentioned that aqs100[1-3] might have janky raid controllers, and that you were going to looking into/test? [13:08:33] yes but I didn't get the time to do it :( [13:08:41] i understand [13:09:57] joal: in scala then is the best way to make an empty mutable map val data = Map[String, String]() ? where map is scala.collection.mutable.Map ? [13:11:18] elukey: are these H310 controllers? [13:14:03] urandom: mmm super ignorant but we can take a look [13:14:43] elukey: thanks! [13:19:30] yep seems so! [13:19:36] damn. [13:22:30] mooorrrnin milimetric, lemme know if you have a few mins [13:28:06] hey ottomata [13:29:08] hiya [13:29:12] wassup [13:29:36] so, hm, uh, was going to write some usage docs for this kafka sse stuff, then thought about asking you for a preference [13:29:37] so [13:29:44] it is a class like Kasocki was [13:29:54] but, it can pretty easily be wrapped in single function call [13:29:56] Analytics-EventLogging, Operations: deploy eventlog2001 services - https://phabricator.wikimedia.org/T93220#2728649 (elukey) [13:29:59] Analytics-EventLogging, Icinga, Operations: eventlog2001 - CRITICAL status of defined EventLogging jobs - https://phabricator.wikimedia.org/T119930#2728646 (elukey) Open>Resolved a:elukey This is an old task, eventlog2001 is not running EL anymore (it is a spare host). [13:30:10] so, i could expose either the class, or some function wrapper in index.js [13:30:12] ok [13:30:15] not sure which is better [13:30:18] if ti was class, it'd be like [13:30:44] var k = new KafkaSSE(req, res, options) [13:30:44] k.connect(topics) [13:30:52] if it'd be a function, i'd just wrap that [13:30:53] so [13:31:03] var handlerFunc = require('kafka-sse'); [13:31:10] handlerFunc(req, res, assignments, options) [13:31:15] or [13:31:22] handlerFunc(req, res, topics, options) [13:31:27] (easier to think about it that way) [13:31:49] Kasocki is new KasockiClient(...) right? [13:32:00] KafkaSSE you mean? [13:32:05] no, kasocki [13:32:06] Kasocki was the socket.io one [13:32:09] yeah [13:32:21] oh [13:32:21] yeah [13:32:25] it worked by making a new class [13:32:26] Kasocki was class [13:32:32] (PS7) Addshore: WikidataArticlePlaceholderMetrics also send search referral data [analytics/refinery/source] - https://gerrit.wikimedia.org/r/305989 (https://phabricator.wikimedia.org/T142955) [13:32:43] there were more things you could do with it though [13:32:46] so, if you want to leave both as options, then I'd make them look kind of the same [13:32:47] not much more [13:32:48] hi team :] [13:33:04] if you want to just have one, I'd go with the function [13:33:15] i don't think i want to merge the code to handle both [13:33:20] but I think it'd be nice even if we don't start with multiple to grow into that, and the class seems more flexible [13:33:20] the service might be able to run both [13:33:30] but if we did that we'd just use routes and require eithe ror [13:33:57] it's just, if you decide to make changes, it's easier to stay backwards compatible with the class and calling methods in a specific way [13:34:17] if you have just one method, it can get ugly if you keep adding things there [13:34:27] (CR) jenkins-bot: [V: -1] WikidataArticlePlaceholderMetrics also send search referral data [analytics/refinery/source] - https://gerrit.wikimedia.org/r/305989 (https://phabricator.wikimedia.org/T142955) (owner: Addshore) [13:34:36] true, i mean,i don't mind changing this code in the future if we have to [13:34:49] either way it'd be abstracted by the service, so the routes would be the same [13:35:01] i'm just not sure which is cleaner [13:35:12] the single function call is kinda nice [13:35:13] right, but in case other people pick it up and use it, don't wanna mess with it too much [13:35:18] oh [13:35:31] mhm, naw i'm not worried about it, we can version if we have to i guess [13:35:38] yeah, true [13:36:45] I mean, you could also wrap kasocki stuff in a big function [13:36:55] and just pass everything to it from the start [13:37:08] I never actually understood why there's a difference here [13:37:16] seems to me you can do both both ways [13:39:06] milimetric, hi! regarding replacing jquery with fetch in Dashiki, what about the dependencies that use jquery? jquery will be packaged anyway when building, no? [13:39:44] mforns: yeah, but they might do the same [13:39:45] yeah true milimetric i guess you could... [13:40:22] mforns: but I donno, that one wasn't super urgent [13:42:42] mforns: did you start work on it? I'm thinking actually the switch to yarn would make a lot of sense before that [13:42:45] I can start on that now [13:47:32] Analytics-Kanban, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Spike: [Feed] Establish criteria for blacklisting likely bot-inflated most-read articles - https://phabricator.wikimedia.org/T143990#2728697 (mforns) @JAllemandou Wouldn't your pageviews/userAgent ratio help in identifying t... [13:48:05] milimetric, I just started looking into the fetch thing [13:49:42] mforns: what do you think? Does it make sense to switch to yarn first? [13:49:55] milimetric, I saw you had assigned the yarn task to you, were you planning to do it today? If so, OK, I'll place the fetch task for grabs and take another thing. Or else let me know if you want me to do the yarn one [13:49:59] mmm.. [13:50:26] milimetric, yes, maybe by using yarn, the changes needed for fetch are different [13:50:47] yep, let's batcave mforns [13:50:52] ok [14:01:36] Analytics-Kanban, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Spike: [Feed] Establish criteria for blacklisting likely bot-inflated most-read articles - https://phabricator.wikimedia.org/T143990#2728732 (JAllemandou) This is definitely the idea @mforns ! however last tests I did using... [14:02:21] milimetric: I'm running into a spark bug when running our algo on pages :( [14:02:40] mforns: --^ [14:03:00] joal, we're in the batcave [14:03:03] join? [14:03:09] yup [14:26:06] (PS8) Addshore: WikidataArticlePlaceholderMetrics also send search referral data [analytics/refinery/source] - https://gerrit.wikimedia.org/r/305989 (https://phabricator.wikimedia.org/T142955) [14:33:42] other interesting thing found today: clickhouse doesn't behave correctly in an example I have [14:36:43] oh? [14:37:03] ottomata: Wanna see? [14:37:17] hmm ok [14:37:20] at a cafe... [14:37:37] bc? [14:37:42] ottomata: Making a gist for you to repro [14:37:46] better? [14:37:49] naw, bc is fine [14:37:50] let's do it [14:37:51] ok [14:37:53] i think... [14:38:06] ottomata: milimetric and mforns are in there :) [14:38:11] let go to batcave-2 [14:38:27] k [15:04:47] ottomata: found a workaround: use agent_type IN ('user') [15:04:52] WEIRDOOOOH ! [15:04:54] weird [15:05:04] pretty dumb though [15:23:55] Analytics-Kanban, EventBus, Repository-Admins, Wikimedia-Stream, Services (watching): Create KafkaSSE diffusion repository - https://phabricator.wikimedia.org/T148651#2728996 (Ottomata) [15:36:14] Analytics-Kanban, EventBus, Repository-Admins, Wikimedia-Stream, Services (watching): Create KafkaSSE diffusion repository - https://phabricator.wikimedia.org/T148651#2729030 (Ottomata) [16:00:09] Analytics-Dashiki, Analytics-Kanban: Switch to fetch away from jquery - https://phabricator.wikimedia.org/T148053#2729063 (Nuria) Right, we no longer should have semantic-ui though. [17:10:26] Analytics: Edit analysis dashboard Failures by User Type chart does not update correctly - https://phabricator.wikimedia.org/T148656#2729192 (mforns) [17:11:39] * elukey afk! [17:12:18] Analytics: Edit analysis dashboard Failures by User Type chart does not update correctly - https://phabricator.wikimedia.org/T148656#2729206 (mforns) We might want to remove that chart from the dashboard instead of fixing it, if there are no people interested in the data it shows any more. [17:17:44] lucnhin! [17:18:26] Analytics-Kanban, Operations, Traffic, Patch-For-Review: Varnishlog with Start timestamp but no Resp one causing data consistency check alarms - https://phabricator.wikimedia.org/T148412#2729247 (elukey) Got some requests logged with the new query string but no trace of backend tags. Maybe this i... [17:38:11] milimetric, mforns : neil is taking off mon and tuesday next week, we will have ee dashboards meeting later in teh week [17:38:18] nuria, ok [17:39:24] nuria: k, np [17:42:49] Hi madhuvishy :) [17:43:02] hey joal :) [17:43:02] madhuvishy: you have two running jobs on the cluster, are they expecteD? [17:43:19] madhuvishy: I'm asking cause I've seen them for 2 days and wonder :) [17:43:23] joal: ahhh [17:43:34] they are my spark jupyter notebooks ;) [17:43:41] i will kill them [17:43:46] madhuvishy: I would have guessed ;) [17:43:57] but it isn't working fully yet [17:44:02] ok [17:44:03] gotta poke otto [17:44:22] madhuvishy: I must say resource releasing is an important for me :) [17:44:47] joal: yeah, need to see what's going on there [17:45:02] no prob :) Thanks for the cleaning madhuvishy ! [17:45:40] joal: np! thanks for the heads up [17:47:50] done [18:39:42] milimetric, back, yt? [18:41:13] to the batcave, mforns [18:41:40] omw! [18:50:11] quick check, Main_page on enwiki is almost always the highest viewed page? It turns out our popularity score recently has 670k pages with a higher popularity score than main page, which seems wrong. Going to investigate further but wanted to make sure my initial assumption is correct [18:54:12] Analytics-Dashiki, Analytics-Kanban: Migrate from bower to yarn and clean up folder hierarchy - https://phabricator.wikimedia.org/T147884#2706880 (Milimetric) p:Triage>Normal [18:55:04] ebernhardson: mmmm.. i do not think that is true, no. Main_page is what it gets shown if you request http://en.wikipedia.org and thus gets side traffic that way but my guess is that it shouldn't be the most popular page no. [18:57:13] (PS1) Milimetric: Clean up folder hierarchy [analytics/dashiki] - https://gerrit.wikimedia.org/r/316834 (https://phabricator.wikimedia.org/T147884) [18:59:00] hmm, ok. i was thinking it would be up there since it has 20-30M page views/day according to the pageviews app in tools, and the topviews tool in labs puts the top pages at >10M, but i'll poke around in the raw data on hive a bit more to make sure [18:59:08] err top pages at <10M [19:00:24] ebernhardson: the "Show only mainspace pages" option is on by default in the Topviews app [19:00:43] which just weens out the pages no one cares about, Main Page included, even though it technically is in the mainspace [19:02:34] milimetric: Can we finish the build change for dashiki: https://gerrit.wikimedia.org/r/#/c/314622? [19:02:43] uncheck that option and you'll see the Main Page and Special:Search, etc. [19:02:53] musikanimal: ahh, i see that now. thanks! [19:03:14] yup! hopefully not causing too much confusion for others [19:03:19] milimetric: on my opinion we should remove any references to piwik and performance from commit message as (for the reasons that i stated the other day) it does not have an effect on perf [19:03:46] musikanimal: makes sense, it looks like main page is 2 orders of magnitude above the rest, so probably worth filtering [19:04:20] yeah haha, exactly [19:13:10] (CR) Nuria: WikidataArticlePlaceholderMetrics also send search referral data (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/305989 (https://phabricator.wikimedia.org/T142955) (owner: Addshore) [19:19:32] (CR) Nuria: "This code must already be running, correct? Why the -1?" [analytics/refinery] - https://gerrit.wikimedia.org/r/315241 (https://phabricator.wikimedia.org/T147841) (owner: Joal) [19:22:20] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Bump up fd ulimit on hadoop workers - https://phabricator.wikimedia.org/T148206#2729739 (Nuria) Open>Resolved [19:22:32] Analytics-Kanban, Patch-For-Review: Roll up Sessions metric to weekly granularity - https://phabricator.wikimedia.org/T147492#2729740 (Nuria) Open>Resolved [19:22:44] Analytics-Dashiki, Continuous-Integration-Config, Patch-For-Review: Add CI job for Dashiki - https://phabricator.wikimedia.org/T148019#2729742 (Nuria) [19:22:46] Analytics-Kanban, Patch-For-Review: Remove version conflict in Dashiki bower install - https://phabricator.wikimedia.org/T148027#2729741 (Nuria) Open>Resolved [19:22:59] Analytics-Kanban, Patch-For-Review: Switch AQS to new cluster - https://phabricator.wikimedia.org/T144497#2729744 (Nuria) Open>Resolved [19:23:05] Analytics-Kanban, Datasets-Webstatscollector, RESTBase-Cassandra, Patch-For-Review, Services (watching): Better response times on AQS (Pageview API mostly) {melc} - https://phabricator.wikimedia.org/T124314#2729745 (Nuria) [19:23:24] Analytics-Kanban, Research-and-Data, Research-collaborations, Research-management, Patch-For-Review: Oozie job to extract data for WDQS research - https://phabricator.wikimedia.org/T146064#2729746 (Nuria) Open>Resolved [19:23:42] Analytics-Kanban: Varnishkafka should auto-reconnect to abandoned VSM - https://phabricator.wikimedia.org/T138747#2729748 (Nuria) Open>Resolved [19:23:53] Analytics-Kanban: Productionize Druid loader - https://phabricator.wikimedia.org/T138264#2729750 (Nuria) [19:23:55] Analytics-Kanban, Patch-For-Review: Productionize Pivot UI - https://phabricator.wikimedia.org/T138262#2729749 (Nuria) Open>Resolved [19:24:05] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents, Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), Patch-For-Review, WMF-deploy-2016-10-04_(1.28.0-wmf.21): EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2729751 (Nuria) Open>Resolved [19:24:20] Analytics-Kanban, EventBus, Wikimedia-Stream, Services (watching), User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2729753 (Nuria) [19:24:23] Analytics-Kanban, EventBus, Wikimedia-Stream, Services (watching): Kasocki Prototype - https://phabricator.wikimedia.org/T145095#2729752 (Nuria) Open>Resolved [19:28:51] (PS1) Nuria: Adding oureach wikipedia to Pageview whitelist [analytics/refinery] - https://gerrit.wikimedia.org/r/316838 (https://phabricator.wikimedia.org/T130249) [19:55:53] (PS2) Nuria: Adding several wikis to Pageview whitelist [analytics/refinery] - https://gerrit.wikimedia.org/r/316838 (https://phabricator.wikimedia.org/T130249) [20:21:34] (PS1) Mforns: Add version to package.json for Dashiki (uses npm now) [analytics/mediawiki-storage] - https://gerrit.wikimedia.org/r/316844 [20:22:16] (CR) Mforns: [C: 2 V: 2] Add version to package.json for Dashiki (uses npm now) [analytics/mediawiki-storage] - https://gerrit.wikimedia.org/r/316844 (owner: Mforns) [20:28:11] bye a-team! [20:28:36] byeaa! [20:34:45] (PS1) Nuria: Enhancing regex to support pageviews to non-knowledge wikis [analytics/refinery/source] - https://gerrit.wikimedia.org/r/316845 (https://phabricator.wikimedia.org/T130249) [20:36:55] Analytics-Kanban, EventBus, Wikimedia-Stream, Services (watching), User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2729942 (Ottomata) [21:51:15] Analytics-Kanban, EventBus, Wikimedia-Stream, Services (watching): Kafka SSE Prototype - https://phabricator.wikimedia.org/T148470#2730110 (Ottomata) [21:51:18] Analytics-Kanban, EventBus, Repository-Admins, Wikimedia-Stream, Services (watching): Create KafkaSSE diffusion repository - https://phabricator.wikimedia.org/T148651#2730106 (Ottomata) Open>Resolved a:Ottomata @qchris did this for me, thanks so much! I'll work with releng folks... [21:55:35] (PS1) Milimetric: [WIP] Migrate from bower to npm instead of yarn [analytics/dashiki] - https://gerrit.wikimedia.org/r/316904 (https://phabricator.wikimedia.org/T147884) [21:57:23] o/ nite yall [21:58:19] nighters milimetric