[03:17:21] bearloga / yurik: the 10k limit is more of a rough guideline, and it's specific to the size of our Druid cluster. In this case it sounds like you'll be able to get plenty of value within the limit, but in general if we have something that would provide amazing value, we could justify upgrading the cluster for that. [03:53:06] (03PS8) 10Milimetric: Optionally record data to Graphite [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/322365 (https://phabricator.wikimedia.org/T150187) (owner: 10MaxSem) [03:54:44] (03CR) 10jenkins-bot: [V: 04-1] Optionally record data to Graphite [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/322365 (https://phabricator.wikimedia.org/T150187) (owner: 10MaxSem) [03:55:55] 06Analytics-Kanban, 03Interactive-Sprint, 13Patch-For-Review: Enhance Report Updater to be able to send data to graphite - https://phabricator.wikimedia.org/T150187#2829677 (10Milimetric) @mforns: please review this when possible, I tested it and it seemed to do what was needed. See the config Max reference... [03:57:18] (03CR) 10Milimetric: "not sure what that error is, I don't have it locally, will check tomorrow." [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/322365 (https://phabricator.wikimedia.org/T150187) (owner: 10MaxSem) [09:38:00] (03PS1) 10Hjiang: patched up some queries, changed names, branched out a new folder, and added the daily edits by anon [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324038 [10:53:14] (03CR) 10MarcoAurelio: "@Nuria I think you forgot to submit the change? jenkins-bot does not operate in this repo and don't merge changes. Regards." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/323699 (https://phabricator.wikimedia.org/T151570) (owner: 10MarcoAurelio) [11:33:01] (03CR) 10Mforns: "Looks better :]" (0311 comments) [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324038 (owner: 10Hjiang) [13:47:40] (03CR) 10Mforns: "LGTM, but I think we can simplify the new_dates concept." (038 comments) [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/322365 (https://phabricator.wikimedia.org/T150187) (owner: 10MaxSem) [14:10:23] hey mforns, joal, wanna mob? [14:10:33] Heya [14:14:04] milimetric, mforns : id the cave [14:14:22] Arf, actually, not - errors [14:14:32] so... am I :) [14:14:33] oh [14:14:45] oh wow - I see you joining and dropping right away [14:14:55] :( [14:16:22] google so crazy [14:22:51] ottomata: Hellooooo :) [14:22:57] ottomata: I have a request for you [14:23:16] hiiii [14:23:18] ja? [14:24:33] I'd need to add a repo to archiva to download spark-related-jars that are not on central [14:24:36] ottomata: --^ [14:24:43] ottomata: Any idea how we do that? [14:25:57] For the moment I build on my own machine and then push the jar to stat [14:26:08] That's long and not a good usage of our bandwisth :) [14:26:11] ottomata: -^ [14:26:43] joal: so the jars are in some other maven repo out there, but not central? [14:26:49] correct ottomata [14:26:56] k, which one? [14:27:04] and, if we do that, can we make it a one time thing? just to get the jars into archiva? [14:27:53] ottomata: Not sure I understand the one-at-a-time :) [14:28:05] ottomata: You mean only one jar at a time in our poms? [14:28:14] as in, we add the remote repo as a mirror, do a mvn package to get archiva to cache all the stuff you need, and then remove the remote repo [14:28:18] ottomata: repo is spark-apckages one: https://dl.bintray.com/spark-packages/maven/ [14:28:33] ottomata: sounds good for now :) [14:28:44] ok [14:29:09] ottomata: Should I add the repo to our pom or will archive do the job? [14:29:38] archiva will do the job [14:29:40] ok done, try it [14:29:54] trying in a minute [14:30:46] (03PS5) 10Joal: [WIP] Update edit history to GraphFrames [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/322260 [14:30:48] (03PS46) 10Joal: [WIP] Refactor Mediawiki History scala code [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/301837 (https://phabricator.wikimedia.org/T141548) [14:30:50] (03PS40) 10Joal: [WIP] Join and denormalize all histories into one [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/307903 (owner: 10Milimetric) [14:31:44] Trying now ottomata [14:32:20] ottomata: o/ as curiosity, why only a one time pull rather than having it permanent? [14:32:47] I am super ignorant about archiva so I might say dumb things :D [14:33:10] ottomata: Correct build :) Thanks a lot it'll save me some time :) [14:33:21] elukey: adding it as a permanent is basically saying we trust this repo forever [14:33:38] which is kinda ok, but i think we err on not doing that [14:33:44] so, i'll leave it in our list of configured remote repos [14:33:50] but i will turn off the active proxy connector [14:36:04] okok [14:36:21] but we won't get updates, only if we explicitly request them right? [14:36:28] (re-adding it as repo temporarily) [14:37:32] elukey: correct, actually it's not 'updates' you wouldn't get, it's any new jar [14:37:43] (new version or different package) [14:38:02] joal: and what about updates of the imported jars? [14:38:19] elukey: updates = new jars (because new version numbeR) [14:38:20] we'd have to re-add the proxy and then build to get anything new we need [14:39:04] ahhh okok [14:39:16] thanks for the explanation [14:39:29] but I got the main picture [14:39:39] archiva is still a mistery to me, will need to work a bit on it [14:41:32] 06Analytics-Kanban: Replace stat1001 - https://phabricator.wikimedia.org/T149438#2752595 (10Ottomata) [14:41:53] 06Analytics-Kanban, 06Operations: setup/install thorium/wmf4726 as stat1001 replacement - https://phabricator.wikimedia.org/T151816#2830877 (10Ottomata) 05Open>03Resolved I will track this in the parent T149438 [14:42:03] stat1001 becomes thorium then :) [14:42:08] dun dun duunnnn [14:42:14] i like thorium, its nice and easy to type [14:42:25] unlike many other element names [14:42:41] i made an alias for the salt master cause what the heck how do you spell that [14:42:41] ? [14:43:11] hhaahahh yes [14:46:56] milimetric, joal, I was having lunch, are you in da cave? [14:47:13] I am :) take your time milimetric [14:54:08] ottomata: While I think of stuff to ask you: Would you mind pulling a last version of superset (previously named caravel - previsouly named panramix) and having it running for me ? [14:54:27] oh haha [14:54:32] it has a new name? :p [14:54:50] (03PS1) 10Gehel: discovery-stats: use the discovery-stats specific credential file [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/324194 (https://phabricator.wikimedia.org/T151063) [14:54:51] yeah it changed name [14:55:27] ottomata: I'm willing to try it again, with pivot discussion being around [14:55:42] ok joal will do [14:55:52] Thanks a lot ottomata :) [15:20:41] elukey: fyi, i just merged https://gerrit.wikimedia.org/r/#/c/324200/, this sets min.insync.replicas=1 on analytics brokers, and =2 on main brokers [15:20:48] default is 1 [15:20:53] so it won't change anything on analytics brokers [15:21:05] and on main, eventbus is not specifying acks, and default is 1, not all [15:21:16] so this won't have an effect there eyt [15:21:38] but it will allow us to chnage to producer acks=all, and have better reliability for messages if brokers go down [15:22:01] not 100% sure we should do this, but i will talk with services folks on monday about it [15:24:44] 06Analytics-Kanban, 06Operations, 05Security, 07audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/api - https://phabricator.wikimedia.org/T92338#2831034 (10Milimetric) a:05Ottomata>03Milimetric [15:26:16] ottomata: oh so with acks=all eventbus will be required to have one ISR before proceeding [15:26:42] with min.insync.replicas=2 [15:26:46] and acks=all [15:26:55] 06Analytics-Kanban, 06Operations, 05Security, 07audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/api - https://phabricator.wikimedia.org/T92338#2831045 (10Milimetric) [15:26:58] there must be at least 2 replicas in the ISR for a produce to succeed [15:27:11] okok [15:27:24] and I guess we have 1 (default) for varnishkafka -> analytics [15:28:21] but not sure if we set acks in there [15:29:00] kafka.topic.request.required.acks = 1 [15:29:07] yeah [15:29:25] and min insync replicas is still 1 there anyway [15:29:32] for analytics brokers [15:29:35] ok so the leader with acks=1 does not wait for ISR [15:30:07] only the leader acks the producer [15:30:09] yup [15:30:32] ok got it, thanks for letting me know :) [15:32:43] 06Analytics-Kanban, 06Operations, 05Security, 07audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/api - https://phabricator.wikimedia.org/T92338#2831069 (10Milimetric) email sent. Will delete all files at the end of this week, please look through if you need anything. [15:36:23] 06Analytics-Kanban, 06Operations, 05Security, 07audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/glam_nara - https://phabricator.wikimedia.org/T92340#2831095 (10Milimetric) a:05Ottomata>03Milimetric [15:39:01] 06Analytics-Kanban, 06Operations, 05Security, 07audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/glam_nara - https://phabricator.wikimedia.org/T92340#2831112 (10Milimetric) @leila, @Multichill, @JeanFred: We're going to remove all these old files at the end of this week. The files continue... [15:43:40] 06Analytics-Kanban, 06Operations, 06Zero, 07Mobile, and 2 others: Purge > 90 days stat1002:/a/squid/archive/mobile - https://phabricator.wikimedia.org/T92341#2831145 (10Milimetric) a:05Ottomata>03Milimetric [15:44:41] 06Analytics-Kanban, 06Operations, 06Zero, 07Mobile, and 2 others: Purge > 90 days stat1002:/a/squid/archive/mobile - https://phabricator.wikimedia.org/T92341#1106695 (10Milimetric) We will remove all these files at the end of this week. Please look through if you need them. [15:46:35] 06Analytics-Kanban, 06Operations, 06Zero, 05Security, 07audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/sampled - https://phabricator.wikimedia.org/T92342#2831170 (10Milimetric) a:05Ottomata>03Milimetric [15:47:45] 06Analytics-Kanban, 06Operations, 06Zero, 05Security, 07audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/sampled - https://phabricator.wikimedia.org/T92342#1106706 (10Milimetric) We will delete all these files at the end of this week. Please look through if you need them. [15:48:29] 06Analytics-Kanban, 06Operations, 06Zero, 05Security, 07audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/zero - https://phabricator.wikimedia.org/T92343#2831175 (10Milimetric) a:05Ottomata>03Milimetric [15:49:02] 06Analytics-Kanban, 06Operations, 06Zero, 05Security, 07audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/zero - https://phabricator.wikimedia.org/T92343#1106713 (10Milimetric) We will delete all of these files at the end of this week. Please look through if you need them. [15:56:54] joal: HMmmmmm this superset isn't as simple as the other thing [15:57:05] Mwarf :( [15:57:14] previously, i hacked it by installing it on a vm, and then copying the stuff over to druid1001, then runnin git [15:57:32] this takes a lot of python deps in a venv, and i've tried several different ways of building the venv to do it, no success [15:58:27] hm [16:00:05] ottomata: What would be a good way to do ? Should I try to install on a vm and let you know how I manageD? [16:00:10] If i do ... [16:01:17] hmmm, i dunno, i mean, i think i can install on a vm fine, its just in prod is a little harder since we don't really have pip there. i expected i could prep a virtualenv, and copy that virtualenv over, and then use it in prod, but i didn't work [16:01:40] elukey: standdduppp [16:01:47] cominggg [16:02:13] ottomata: Could be related to 'cryptography' lib (they say it can cause probs in doc) [16:04:09] yeah, but it builds fine [16:04:15] [@druid1001:/home/otto/superset] [venv] $ fabmanager create-admin --app superset [16:04:15] Could not find platform independent libraries [16:04:15] Could not find platform dependent libraries [16:04:15] Consider setting $PYTHONHOME to [:] [16:04:15] Fatal Python error: Py_Initialize: Unable to get the locale encoding [16:04:16] ImportError: No module named 'encodings' [16:04:16] Aborted [16:04:42] hmm, that was with a python3 venv, i guess i will try python2 real quick and see if it is any different [16:04:58] ottomata: docs says it should work with 2.3.4+ [16:05:02] yeah [16:05:07] with 3.4+ sorry [16:08:12] (03CR) 10Nuria: [C: 032] Adding self-identified bot to bot regex [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/323249 (https://phabricator.wikimedia.org/T150990) (owner: 10Nuria) [16:08:19] (03CR) 10Nuria: [V: 032] Adding self-identified bot to bot regex [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/323249 (https://phabricator.wikimedia.org/T150990) (owner: 10Nuria) [16:13:09] (03Merged) 10jenkins-bot: Adding self-identified bot to bot regex [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/323249 (https://phabricator.wikimedia.org/T150990) (owner: 10Nuria) [16:16:24] (ooo joal i might have it...) [16:16:33] YAY :) [16:23:42] mforns: batcave? [16:23:59] https://hangouts.google.com/hangouts/_/wikimedia.org/a-batcave-2 [16:28:15] milimetric, omw [16:31:02] 10Analytics, 06Discovery, 06Discovery-Analysis, 03Interactive-Sprint: Add Maps tile usage counts as a Data Cube in Pivot - https://phabricator.wikimedia.org/T151832#2831300 (10Nuria) How about we split the work here in two chunks: 1. let's get the data into pivot , play with it, make sure is a good fit.... [16:39:13] 06Analytics-Kanban, 10Wikimedia-Site-requests, 13Patch-For-Review, 05WMF-deploy-2016-11-29_(1.29.0-wmf.4): Create Wikivoyage Finnish - https://phabricator.wikimedia.org/T151570#2831362 (10Nuria) [16:48:39] (03PS9) 10Milimetric: Optionally record data to Graphite [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/322365 (https://phabricator.wikimedia.org/T150187) (owner: 10MaxSem) [16:50:47] (03CR) 10jenkins-bot: [V: 04-1] Optionally record data to Graphite [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/322365 (https://phabricator.wikimedia.org/T150187) (owner: 10MaxSem) [17:10:37] joal: ssh -N stat1002.eqiad.wmnet -L 8188:druid1001.eqiad.wmnet:8188 [17:10:40] http://localhost:8188 [17:10:43] admin /admin [17:10:46] admin / admin [17:11:25] * elukey doesn't see the new screen session fired up by ottomata and keeps working :P [17:12:01] Oh, elukey, look in that other direction, there are plenty other things to dicsuss :) [17:12:33] haha [17:12:41] 10Analytics, 06Operations: Install java 8 to stat1002 - https://phabricator.wikimedia.org/T151896#2831478 (10EBernhardson) [17:15:12] (03CR) 10Nuria: "Let's please make sure to add documentation about this addition to wikitech docs for reportupdater" [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/322365 (https://phabricator.wikimedia.org/T150187) (owner: 10MaxSem) [17:15:23] 06Analytics-Kanban: Replace stat1001 - https://phabricator.wikimedia.org/T149438#2831496 (10Ottomata) a:03Ottomata [17:17:25] ottomata: Thanks a lot ! Trying the beast [17:35:45] (03PS1) 10Ottomata: Add thorium to pivot scap targets [analytics/pivot/deploy] - 10https://gerrit.wikimedia.org/r/324236 (https://phabricator.wikimedia.org/T149438) [17:36:18] (03CR) 10Ottomata: [C: 032 V: 032] Add thorium to pivot scap targets [analytics/pivot/deploy] - 10https://gerrit.wikimedia.org/r/324236 (https://phabricator.wikimedia.org/T149438) (owner: 10Ottomata) [17:46:09] 06Analytics-Kanban: User limits for stat machines. Limit space on /home dir and possibly /tmp - https://phabricator.wikimedia.org/T151904#2831653 (10Nuria) [17:47:40] 10Analytics: Navigation timing spotty kafka metric - https://phabricator.wikimedia.org/T151907#2831696 (10Nuria) [17:47:58] 10Analytics: User limits for stat machines. Limit space on /home dir and possibly /tmp - https://phabricator.wikimedia.org/T151904#2831708 (10Nuria) [17:48:25] elukey: what was teh kafka metric that was spotty and sending us alarms? cc ottomata [17:51:16] nuria: https://graphite.wikimedia.org/S/Bz [17:51:40] so kafka.cluster.analytics-eqiad.kafka.*.kafka.server.BrokerTopicMetrics.MessagesInPerSec.eventlogging_NavigationTiming.OneMinuteRate [17:52:05] 10Analytics: Navigation timing spotty kafka metric - https://phabricator.wikimedia.org/T151907#2831723 (10Nuria) [17:52:16] elukey: thank youuu [17:54:28] milimetric: do you know how ezachte publishes to stats.wm.org? [17:58:35] ottomata: I believe he just copies files from 1002 to 1001 [17:58:51] after generating the html [18:00:00] ok [18:00:37] joal: stafff? on batcave [18:00:51] yes [18:55:52] if I want to use a custom hive UDF that's in my home dir on stat1002 but I'm working within Hue rather than CLI, does anyone know what path to use for ADD JAR? tried "ADD JAR mnt:///home/bearloga/Code/analytics-refinery-jars/refinery-hive-xcache.jar" and "ADD JAR hdfs:///home/bearloga/Code/analytics-refinery-jars/refinery-hive-xcache.jar" but probably doing [18:55:52] something very obvious very wrong [18:56:17] bearloga: you probably have to put in into hdfs, your homedir there would be fine [18:56:17] but [18:56:20] the homedir path is [18:56:24] hdfs:///user/bearloga [18:57:44] ottomata: thank you! [18:59:00] ottomata: let me know if you'll need me to follow up on thorium tomorrow :) [18:59:16] elukey: follow up with ezacthe? [18:59:28] i'm going to send an email today, its up and kicking. hm actually [18:59:33] it looks lke pivot is having troulbe talking to druid [18:59:34] dunno why [18:59:37] would you look into that elukey? [19:01:18] wikimedia/mediawiki-extensions-EventLogging#621 (wmf/1.29.0-wmf.4 - 7c7eb62 : Timo Tijhof): The build has errored. [19:01:18] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/compare/wmf/1.29.0-wmf.4 [19:01:18] Build details : https://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/179852822 [19:04:36] ottomata: sure! [19:04:53] I saw that you started the work but I can definitely help [19:05:22] cool danke! i'll work on the other part which is mostly ezachte and communication :) [19:17:45] logging off for today team, see ya tomorrow [19:19:20] (03CR) 10Milimetric: "These three are taken care of but the problem with the injection on the graphite send remains. I couldn't figure out how to send via pyth" (033 comments) [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/322365 (https://phabricator.wikimedia.org/T150187) (owner: 10MaxSem) [19:20:50] nite dude [19:25:49] (03CR) 10MaxSem: [C: 032 V: 032] discovery-stats: use the discovery-stats specific credential file [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/324194 (https://phabricator.wikimedia.org/T151063) (owner: 10Gehel) [19:26:02] (03PS1) 10MaxSem: discovery-stats: use the discovery-stats specific credential file [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/324247 (https://phabricator.wikimedia.org/T151063) [19:26:17] (03CR) 10MaxSem: [C: 032 V: 032] discovery-stats: use the discovery-stats specific credential file [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/324247 (https://phabricator.wikimedia.org/T151063) (owner: 10MaxSem) [19:38:22] anyone know the units of time_firstbyte in wmf.webrequest? seconds? minutes? [19:39:16] https://wikitech.wikimedia.org/wiki/Analytics/Data/Webrequest doesn't list the units [19:48:43] nuria milimetric joal ottomata: do yinz know? ^ I'm going to be doing some timestamp arithmetic with time_firstbyte and I really don't want to assume the wrong units :) [19:50:59] bearloga I don't know by heart but I'd just select time_firstbyte from webrequest where ... limit 100 and restrict to where time_firstbyte is not null [19:51:23] and to a small partition so it's quick [19:51:36] it's probably milliseconds if I had to guess [19:52:36] 06Analytics-Kanban, 06Wikipedia-iOS-App-Backlog, 10iOS-app-feature-Analytics: Drop in iOS app pageviews since version 5.2.0 - https://phabricator.wikimedia.org/T148663#2832311 (10Nuria) 05Open>03Resolved [19:52:52] 06Analytics-Kanban, 13Patch-For-Review: Fix dropdowns in metric selector in dashiki - https://phabricator.wikimedia.org/T150664#2832317 (10Nuria) 05Open>03Resolved [19:53:16] 06Analytics-Kanban, 10Graphite, 13Patch-For-Review: statsv improvements - https://phabricator.wikimedia.org/T150765#2832321 (10Nuria) 05Open>03Resolved [19:53:29] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 06Services (watching): Kafka SSE Prototype - https://phabricator.wikimedia.org/T148470#2832326 (10Nuria) [19:53:31] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 13Patch-For-Review, 06Services (watching): Prepare eventstreams (with KafkaSSE) for deployment - https://phabricator.wikimedia.org/T148779#2832325 (10Nuria) 05Open>03Resolved [19:54:04] 06Analytics-Kanban, 13Patch-For-Review: Change requests drop alarms to be more precise regarding data loss - https://phabricator.wikimedia.org/T148980#2832327 (10Nuria) 05Open>03Resolved [19:54:52] 06Analytics-Kanban: Missing raw pageview data for 7/29/2015 - 03:00 - https://phabricator.wikimedia.org/T147801#2832329 (10Nuria) Data is now re-generated. [19:55:45] milimetric: I'm planning to compute a "response timestamp" by adding time_firstbyte to ts, hence the need to know the specific units. does 0.020486ms make sense as a time to first byte? that seems...awfully fast. [19:55:48] 06Analytics-Kanban: Missing raw pageview data for 7/29/2015 - 03:00 - https://phabricator.wikimedia.org/T147801#2832336 (10Nuria) 05Open>03Resolved [19:56:03] 06Analytics-Kanban: Optimize Edit History denormalized table extraction for big wikis - https://phabricator.wikimedia.org/T146481#2832337 (10Nuria) 05Open>03Resolved [19:58:15] milimetric: maybe seconds, then? 0.000096ms seems especially fast too :) [19:59:13] haha, that seems very fast, yes. Seconds probably. Should be usually less than one second [20:04:21] bearloga; https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hive/QueryUsingUDF#Testing_a_UDF_you_just_wrote [20:05:03] bearloga: those are cache look uips [20:05:13] bearloga: no time to 1st byte as measured by client [20:05:26] bearloga: makes sense? [20:05:33] bearloga: that is why they are so fast [20:05:41] bearloga: most of the time is just in memory lookup [20:05:59] bearloga: let me know if that makes sense [20:06:50] bearloga: those numbers have little to do with perceived latency [20:07:22] ottomata: i am going to publish module to npm so Pchelolo can test, ok? [20:07:30] nuria: sure! [20:07:30] ottomata: will wait for your confirmation [20:07:32] reading this review now [20:07:38] i think we can merge before we publish if you like [20:07:45] then we can test and iterate further if we hafta [20:08:19] nuria: makes sense! although we should document that difference. [20:08:22] ottomata: ok, that sounds fine, that way code is on github [20:08:31] bearloga: wait , what difference? [20:09:00] nuria: I'll update the testing UDF section to include instructions for testing when using Hue because the instructions don't work for Hue [20:09:12] bearloga: indeed. thank you [20:10:30] nuria: that it's cache look up times, not time to 1st byte as measured by client. right now it's a little misleading, and several of us in discovery were operating under the assumption that that's what it was [20:10:37] bearloga: none of webrequest timings can tell you of perf because they all come from server, not client side, [20:10:43] bearloga: couldn't be any other way [20:11:17] bearloga: [20:11:18] ' [20:11:19] Varnish:time_firstbyte [20:11:19] Time from when the request processing starts until the first byte is sent to the client. For backend mode: Time from the request was sent to the backend to the entire header had been received. [20:11:41] got that from https://www.varnish-cache.org/docs/trunk/reference/varnishncsa.html [20:11:45] bearloga: think that webrequest data comes from varnish [20:11:57] i believe it is seconds [20:12:02] bearloga: it doesn't know about clients or browsers, it is mostly request data [20:12:07] not 100% on seconds though [20:16:34] bearloga: updated docs but think that all measures come from varnish and thus you can infer little about clients with webrequest dataset other than User agents [20:16:54] ottomata: shoudl i publish to npm from github? [20:16:59] nuria ottomata: thank you very much for the clarification! (and documentation update) and yeah, I think seconds makes sense. [20:17:14] ottomata: how do we mormally do that? cc Pchelolo [20:17:15] nuria: you can do that from your local, but it would be good to push a tag before you do that [20:17:25] anywya, this is good [20:17:27] i will merge, ok? [20:17:46] ottomata: k [20:18:00] merged, woohoo! [20:18:09] nuria: i think you don't need to publish for Pchelolo to test [20:18:21] if it goes to github, he can npm install from github [20:18:29] ottomata: if he wants to pull from npm ya, we should probably push [20:18:45] you can npm install from a github uri, / pu tit in package.json dependencies [20:21:19] ottomata: ok, Pchelolo you let us know what you prefer [20:28:29] ottomata: is there a process to create module in github? [20:28:34] ottomata: or is it automagical? [20:28:58] it should be automatic, but sometimes it doesn't happen [20:28:59] so that i dunno [20:29:12] maybe ping in releng [20:29:15] or -devtools [20:35:29] nuria ottomata: quick question :) varnish is set to client-mode, right? not backend mode, correct? [20:35:50] hmmmmmmmm there are multiple instances, but yes, i believe the requests we log are from client side [20:36:03] so it should be time to first byte sent to client making the request [20:36:12] so, if the req is cached on varnish front end [20:36:14] it'll be really fast [20:36:31] if on varnish backend, a little bit slower, but still really fast (also depends on how many levels it has to hop) [20:36:36] if it hits app servers, it'll be slower [20:36:48] 'client side' == frontend varnish [20:37:02] the varnish that actually handles the request interaction with the browser/http client [20:37:54] ottomata: ok, some manual magic was needed [20:37:56] https://github.com/wikimedia/node-rdkafka-statsd [20:39:31] great! oo looks like it needs a better README! maybe you can take some of that file doc and put it into readme [20:40:44] ottomata: k, will do and tag with version 0.1 right? [20:40:44] ottomata: thank you for clarifying! [20:41:10] nuria: let's get Pchelolo to test before we tag it [20:41:16] ottomata: k [20:41:19] if it works for him, then we'll do it [20:41:20] hmmM [20:41:37] ottomata: i think he might be out today cc Pchelolo https://github.com/wikimedia/node-rdkafka-statsd [20:42:20] ottomata: yes? [20:43:41] maybe i can test now... [20:43:49] with eventstreams...let's see [21:05:59] hmm, nuria we wanted to filter out the /bootstrap stuff too ,no? [21:06:23] oh [21:06:50] ottomata: i did not have that on my notes but sure [21:08:14] hmm, and nuria, are you sure string '-1' works? [21:08:32] oh yeah cause you are splitting the flattened key [21:09:20] ottomata: it is on unit tests [21:10:02] ottomata: so yes, if there is no filter fn nothing is filtered however, will add this to the better README patch [21:10:58] nuria: somehow its filtering a lot more than it should [21:11:40] ottomata: note that whitelist includes only [21:11:46] https://www.irccloud.com/pastebin/HokhrpUd/ [21:11:59] everything else will be filtered [21:12:23] OH right [21:12:23] hm [21:13:07] ahh so we are missing all of the other broker stats, hmmm [21:13:48] hm ok will submit patch for that [21:17:18] 10Analytics, 06Discovery, 06Discovery-Analysis, 03Interactive-Sprint: Add Maps tile usage counts as a Data Cube in Pivot - https://phabricator.wikimedia.org/T151832#2832798 (10debt) p:05Triage>03Normal [21:20:44] nuria: i'll submit patch to get rid of bootstrap stuff too [21:21:00] ottomata: i can do that , you do not have to [21:21:28] 06Analytics-Kanban, 06Operations, 05Security, 07audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/glam_nara - https://phabricator.wikimedia.org/T92340#2832815 (10Multichill) As discussed: The data prior to 2014-02-27 already got deleted. From 2015 we have https://dumps.wikimedia.org/other/me... [21:21:36] ottomata: just submitted one for README [21:21:38] https://gerrit.wikimedia.org/r/#/c/324277/ [21:23:26] hmmm, nuria let's leave bootstrap in, i'm looking at it some more, and its ok [21:23:33] ottomata: k [21:23:39] bootstrap looks like it is reporting stuff about the bootsrap startup of the client, i think its ok [21:23:52] but ja, about to expand whitelist, you might want to update readme after i submit this [21:26:22] ottomata: k, i made default wishlist with Pchelolo's picks [21:29:16] ottomata: k, almost done eh? [21:30:13] almost! [21:36:34] ottomata: tests pass now. i guess we can merge new list and i will update README chnageset [21:36:38] *changeset [21:37:19] ok [21:41:26] ottomata: k, merged and updated README [21:49:37] commented [22:07:04] 06Analytics-Kanban, 06Operations, 05Security, 07audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/glam_nara - https://phabricator.wikimedia.org/T92340#2833111 (10Milimetric) talking further, it turns out we can disable the glam_nara jobs alltogether, per @lzia and @Multichill, and purge all... [22:21:39] 10Analytics, 06Editing-Analysis: Move contents of ee-dashboards to edit-analysis.wmflabs.org - https://phabricator.wikimedia.org/T135174#2833234 (10Neil_P._Quinn_WMF) Gerrit changes [323861](https://gerrit.wikimedia.org/r/#/c/323861) and [324038](https://gerrit.wikimedia.org/r/#/c/324038) relate to this task.... [23:54:23] (03CR) 10Neil P. Quinn-WMF: [C: 031] "As I said in the Phab task, it would be good to move these queries into the analytics-limn-edit-data repo." (031 comment) [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324038 (owner: 10Hjiang)