[00:00:16] Analytics-Wikistats, HTTPS: Fix the mixed content issue on Wikimedia Statistics - https://phabricator.wikimedia.org/T93702#1143679 (Chmarkine) [00:19:24] (PS9) Milimetric: Implement A/B comparison [analytics/dashiki] - https://gerrit.wikimedia.org/r/198169 [00:48:36] Analytics-EventLogging, MediaWiki-extensions-WikimediaEvents, VisualEditor: Make all Event Logging instrumentation target the same Schema:Edit revision - https://phabricator.wikimedia.org/T93586#1143773 (Krenair) @Krinkle has done some digging around on the ResourceLoader side of things. It appears t... [00:48:47] Analytics-EventLogging, MediaWiki-extensions-WikimediaEvents, VisualEditor: Make all Event Logging instrumentation target the same Schema:Edit revision - https://phabricator.wikimedia.org/T93586#1143776 (Krenair) a:Jdforrester-WMF>None [00:51:58] Analytics-Wikistats, HTTPS: Fix the mixed content issue on Wikimedia Statistics - https://phabricator.wikimedia.org/T93702#1143788 (Dzahn) is that index.html from puppet actually deployed on the prod server? [00:52:43] Analytics-EventLogging, MediaWiki-extensions-WikimediaEvents, VisualEditor: Make all Event Logging instrumentation target the same Schema:Edit revision - https://phabricator.wikimedia.org/T93586#1143791 (Krinkle) Fetching the startup module shows: ``` [ "schema.Edit", 138149343... [01:29:51] Analytics-EventLogging, MediaWiki-extensions-WikimediaEvents, VisualEditor, WikiEditor: Make all Event Logging instrumentation target the same Schema:Edit revision - https://phabricator.wikimedia.org/T93586#1143893 (Mattflaschen) [01:33:10] Analytics-Wikistats, HTTPS: Fix the mixed content issue on Wikimedia Statistics - https://phabricator.wikimedia.org/T93702#1143895 (Chmarkine) [09:16:10] Analytics-Engineering, operations, Graphite, Icinga, Patch-For-Review: icinga UNKNOWN Varnishkafka Delivery Errors / varnishkafka data not in graphite - https://phabricator.wikimedia.org/T92965#1144522 (fgiunchedi) Open>Resolved alarms created, resolving [10:13:27] hey folks, we just had a network switch outage in the rack that contains the hadoop namenode. the switch is rebooted, the namenode is in active state, and i have a meeting with ottomata in 5.5 hours so we'll do a thorough check-out together at that time. if you see oddness before then, that is the cause. [12:58:04] thanks for the reboot, jeff [14:04:40] nuria / mforns / kevinator / grace: you guys are all having trouble too right [14:04:43] it's not just me? [14:04:53] milimetric: now it's just you [14:04:53] now grace got it [14:04:56] ok [14:04:57] :{ [14:05:04] milimetric: re-start? [14:05:14] milimetric: we can wait 5 mins [14:05:27] k, brb [14:05:34] milimetric: video feed doesn't come [14:05:51] (PS10) Milimetric: Implement A/B comparison [analytics/dashiki] - https://gerrit.wikimedia.org/r/198169 [14:05:54] milimetric: want us to call your cell? [14:10:26] Analytics-EventLogging, Analytics-Kanban: Backfill client side data data for 2015-03-20 - https://phabricator.wikimedia.org/T93602#1144950 (kevinator) p:Triage>Normal [14:16:18] Analytics-EventLogging, Analytics-Kanban: Backfill client side data data for 2015-03-20 - https://phabricator.wikimedia.org/T93602#1144970 (ggellerman) a:Nuria [16:04:37] Analytics-Tech-community-metrics, Possible-Tech-Projects, ECT-March-2015, Epic, and 2 others: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1145312 (NiharikaKohli) Hello! The IRC meeting tomorrow has been shifted to #wikimedia-ect... [16:06:21] (PS11) Milimetric: Implement A/B comparison [analytics/dashiki] - https://gerrit.wikimedia.org/r/198169 [16:06:27] I'm super pumped about PS11 ^^ [16:06:29] It's ready [16:08:01] nuria / Marcel, btw: I forgot at standup: the VE folks found that somehow part of EL is not invalidating their old schema version, so it's still being used for the events [16:08:23] I don't know the details yet, I'll wait till I talk to Alex [16:08:30] milimetric, ok [16:18:27] milimetric: ahem... [16:18:46] milimetric: the events and version are sent "together" [16:19:17] milimetric: the event contains the version, this means that they have cached JS code longer than they expect [16:19:45] milimetric: EL just "reads" those values (see client-side-log) [16:19:52] milimetric: take a look at let me know [16:37:43] nuria: yeah, I think what's cached is the part of EL that retrieves the schema to use. But I only saw fragments of the conversation while we were in meetings [16:38:35] milimetric: the caching of js has two parts: js modules (and versions of those) and js inline specified inside the html/php [16:38:41] the 2nd is cached foreverrrrrr [16:38:55] 30 days, i think [16:39:04] milimetric: that is all mediawiki [16:39:43] yeah, that might? be it? [16:47:15] milimetric: how mediawiki works with caching system is kind of beyond me (seems very error prone) but the VE team knows that well [18:44:33] nuria: have a small question when you have time [18:44:42] YuviPanda: sure! [18:44:58] nuria: what’s the use case for the logster logs for wikimetrics? [18:45:26] YuviPanda: to know the pageviews wikimetrics, dashiki and other dashboards get [18:45:36] YuviPanda: does it seem like an ok use case? [18:45:53] nuria: ah, yeah. fair enough. [18:46:42] YuviPanda: so easiest way seems to be use logster, parse apache logs with a regex and report [18:46:55] nuria: yeah, fair enough. [18:47:16] YuviPanda: what i do not get at all are the units that are reported here: http://graphite.wmflabs.org/render/?width=586&height=308&_salt=1427222657.823&target=analytics.wikimetrics.pageviews.line_rate.value [18:48:08] (PS2) Ottomata: Add ua_map and x_analytics_map fields to refine table. [analytics/refinery] - https://gerrit.wikimedia.org/r/198789 (owner: Joal) [18:49:07] milimetric: rolling metrics are now in dashiki: https://metrics.wmflabs.org/static/public/dash/#projects=ruwiki,itwiki,dewiki,frwiki,enwiki,eswiki,jawiki/metrics=RollingNewActiveEditor [18:49:30] (CR) Ottomata: "We talked about Nuria's comment in IRC, and decided to defer replacing x_analytics and user_agent fields with these maps until later." (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/198789 (owner: Joal) [18:55:14] nuria: ha! weird [18:55:31] milimetric: good, VERY good actually [18:55:55] but is it a fluke or you think it's a sign of a healthier labsdb? [18:56:21] I mean, it's good either way, but I wouldn't put all my eggs in that basket again [19:00:18] milimetric: well, it will be worth to talk to YuviPanda and springle and see what has gotten to be so much better in labs that we have been able to backfill that data in no time [19:01:15] :) [19:02:16] YuviPanda: any words of wisdom? our db queries that were clogged up for months have run themselves all this week, backfilling even [19:02:38] what time period were these changes over? [19:02:43] YuviPanda: see enwiki: https://metrics.wmflabs.org/static/public/dash/#projects=ruwiki,itwiki,dewiki,frwiki,eswiki,jawiki,wikidatawiki,enwiki/metrics=RollingActiveEditor [19:02:54] I mean [19:02:55] * analytics-team pats self on back for resilient wikimetrics recurrent report structure [19:03:06] when was this terrible and when did it start being unterrible? [19:03:13] terrible: always [19:03:16] not terrible: last week [19:03:22] YuviPanda: the line tricks you cause it has few data points, but it did run as of recent [19:03:40] afaik nothing has changed last week [19:03:49] springle would have to confirm, however. [19:03:59] YuviPanda: ahhhh, that is not what i wanted to hear [19:04:03] :) [19:04:17] nuria: milimetric so it’s possible it’s a fluke :) [19:04:30] yeah, a welcome and nice fluke [19:04:42] but essentially the schema we have is not an analytics schema [19:04:46] and shouldn't be used for analytics [19:04:50] YuviPanda: exactly, i was looking for a db-ninja explanation of which i understand half the words [19:04:57] exhibit A: it takes two pages to write a common sense metric [19:05:04] hahaha [19:06:44] I think the quotiont of the integer casting in the write ahead journals were flushed recently, causing massive spikes in availability? [19:06:47] nuria: ^ does that work? [19:07:01] hi ggellerman. I’m moving to SF in 2 days. The office portals are mine. mwahahahaha :) [19:07:11] lol * 2 [19:07:12] YuviPanda: jahahah, you forgot "overflow" [19:07:31] YuviPanda: that is the ONE WORD that makes everyone scared [19:07:43] nuria: I thought ‘massive spikes’ did that? [19:07:57] YuviPanda: jaja .. right right [19:08:02] personally I think "OKR" is scarier :) [19:08:17] milimetric: it reminds me of http://en.wikipedia.org/wiki/Okra [19:08:38] mmm slimey [19:11:32] YuviPanda: i have the ifttt code duplicated in several locations now, and am about to clean things up. to confirm: uwsgi is loading ~/www/python/src/app.py, right? [19:11:42] ori: yup [19:11:53] milimetric: yeah, I hate that vegetable, as I hate most of them... [19:11:56] cool thanks! [19:17:43] ori: :D yw! do point out issues with toollabs as you see them… I’m super interested in hearing your opinions. [19:27:53] YuviPanda: Quick question about tool labs: If a tool owner is hit by a bus, you guys can take over the tool, right? [19:28:13] guillom: technically? yes. socially? we don’t have a process for it yet. [19:28:25] YuviPanda: OK :) Thanks. [19:28:32] yw [19:28:38] guillom: planning on driving a bus anytime soon? :) [19:29:20] YuviPanda: héhé; Updating my continuity page. [19:29:26] aaah [20:56:26] is the labs NFS backup likely to affect wikimetrics? https://lists.wikimedia.org/pipermail/labs-l/2015-March/003530.html [21:23:59] fhocutt: yeah, it will affect wikimetrics [21:24:13] at least it will affect the wikimetrics backup process which writes to NFS [21:24:22] but that recovers fairly well so far [21:24:41] is it likely to cause slow reports or downtime? I can give the list a heads-up if so [21:25:05] i don't think so [21:25:20] great. Thanks, milimetric. [21:25:37] i've thought about sending similar warnings in the past and i haven't regretted holding back so far [21:25:41] np [21:27:37] Analytics-Wikimetrics: Description of metrics includes link to on-wiki metrics documentation - https://phabricator.wikimedia.org/T93659#1146616 (Fhocutt) @egalvezwmf, @abit, this is an easy change to make. Do you have any suggested phrasings or resources beyond https://meta.wikimedia.org/wiki/Category:Global... [22:03:30] nuria, about the quantiles in spark, you said there was a java tool we could use? [22:05:46] I don't think java can hanldle spark RDDs [22:12:13] Analytics-Wikistats, HTTPS: Fix the mixed content issue on Wikimedia Statistics - https://phabricator.wikimedia.org/T93702#1146743 (QChris) Wikistats is not puppetized and the repo that is getting used in production is not in sync with the gerrit's wikistats repo. ------------------------ @ezachte: It s... [22:30:29] nuria, mforns: how can I create a test wikimetrics user with a unicode username? (on my machine, using vagrant) [22:31:13] fhocutt, maybe you want to upload a cohort with unicode chars? [22:31:31] yes, but also have those validated in my local DB [22:31:44] so I can successfully run reports on them [22:31:58] (or validated in some DB, the important thing is to be able to run reports) [22:32:19] so, you have vagrant running? [22:32:24] yes [22:32:33] can you access localhost:5000? [22:32:36] yes [22:32:48] so you already know this right? :] [22:33:13] let me understand [22:34:25] I tried to make a cohort with some accented names, and none of the usernames were valid, so I couldn't use those to run a report [22:34:38] fhocutt, I see [22:35:10] fhocutt, you can try to manually add some users (with unicode chars) to the wikmetrics database [22:35:27] and then upload a cohort that includes them [22:35:37] where is that db? I'm maybe looking in the wrong place. [22:35:42] in vagrant [22:35:48] let me see [22:36:02] just a sec [22:36:24] fhocutt: there are tests already in place that use users with utf-8 names for cohort upload [22:36:45] cc mforns , I think most problems with utf-8 you shall find on the flask layer , not below. [22:36:47] great! Are those the CSV tests, or something else? [22:38:29] fhocutt: some of the tests: ./test_api/test_centralauth_service.py ./test_controllers/test_cohort_upload.py ./test_utils/test_one_off_functions.py [22:38:46] thanks, nuria [22:39:20] fhocutt, you can grep for utf8 inside the test folder [22:40:21] thanks, mforns [22:40:33] fhocutt, anyway, anytime you need to test some behavior when hitting the db you can manually add stuff using 'mysql' inside vagrant [22:40:46] ah, inside vagrant [22:40:56] that helps! Thank you. [22:41:03] fhocutt, yes vagrant ssh [22:41:31] yes, I've used that for other things, didn't realize that was the way to the DB though [22:41:58] fhocutt, ok, good luck [22:48:18] nuria, about the quantiles in spark, you said there was a java tool we could use? [22:48:35] I don't think java can hanldle spark RDDs [22:48:41] mforns: quantiles? for session report? [22:48:45] yes [22:49:00] mforns: wait.. what are spark RDDS? [22:49:32] mforns: ah i see [22:49:37] a data structure that abstracts the lists that are to be computed in the cluster [22:50:21] mforns: when we talked originally about this i mentioned the hive udfs that do stat calculations but those do not help you here [22:50:23] from what I understand, to call a Java percentile method we'd have to load the RDDs in an in-memory array/list [22:50:34] aha [22:51:10] mforns: mmm...no, i do not think so. [22:51:28] mforns: can you show me the code you have thus far? [22:51:34] nuria, sure [22:51:37] batcave? [22:54:27] mforns: sure but it doesn't have to be today if it is too late... [22:54:40] np, it's ok [22:54:57] it will help me also [22:56:00] mforns: k [23:08:23] Analytics, Scrum-of-Scrums, Wikipedia-App-Android-App, Wikipedia-App-iOS-App, and 3 others: Avoid cache fragmenting URLs for Share a Fact shares - https://phabricator.wikimedia.org/T90606#1147030 (dr0ptp4kt) https://gerrit.wikimedia.org/r/#/c/198805/ submitted for review for Varnish. [23:17:43] mforns: the "mean" we have here: http://spark.apache.org/docs/1.2.1/api/scala/index.html#org.apache.spark.rdd.DoubleRDDFunctions [23:17:56] nuria, ok [23:18:11] nice [23:18:20] Analytics, MediaWiki-API-Team, MediaWiki-Authentication-and-authorization: Create dashboard to track key authentication metrics before, during and after AuthManager rollout - https://phabricator.wikimedia.org/T91701#1147077 (bd808) [23:25:26] Analytics-Wikimetrics: Description of metrics includes link to on-wiki metrics documentation - https://phabricator.wikimedia.org/T93659#1147103 (Abit) @Fhocutt, You can check out the Wikimetrics Learning Module [1] and the meta page linked from the support tab in wikimetrics. [2] If you want to include deta...