[06:47:31] <elukey>	 hello team, I am going to be afk for a couple of hours, will be back online later on!
[08:58:37] <joal>	 Thanks for the nice feedback ori_, kudos to mforns and milimettric :)
[09:33:15] <legoktm>	 hi
[09:33:21] <legoktm>	 is https://wikitech.wikimedia.org/wiki/Analytics/Data_access#Analytics_slaves still accurate?
[09:33:32] <legoktm>	 legoktm@stat1003:~$ mysql --defaults-file=/etc/mysql/conf.d/research-client.cnf -h analytics-store.eqiad.wmnet -A
[09:33:32] <legoktm>	 Could not open required defaults file: /etc/mysql/conf.d/research-client.cnf
[09:33:32] <legoktm>	 Fatal error in defaults handling. Program aborted
[09:33:41] <legoktm>	 (I've never tried to access EventLogging data before)
[09:34:29] <legoktm>	 legoktm@stat1003:~$ groups
[09:34:29] <legoktm>	 wikidev statistics-privatedata-users
[09:38:18] <legoktm>	 James_F helped me, I needed to use stats1002
[09:39:36] <joal>	 Hi legoktm
[09:39:48] <legoktm>	 hi :)
[09:39:50] <joal>	 give me a minute, reading and trying to debug your connection
[09:39:56] <legoktm>	 joal: oh, I figured it out
[09:40:01] <joal>	 Arf, ok :)
[09:40:10] <joal>	 I'm too slow ;)
[09:41:56] <legoktm>	 hehe I just walked across the room to talk with james :P
[09:42:02] <joal>	 right
[09:50:58] <wikibugs>	 Analytics-Kanban: Consider SSTable bulk loading for AQS imports - https://phabricator.wikimedia.org/T126243#2398438 (JAllemandou) a:JAllemandou
[09:51:15] <wikibugs>	 Analytics-Kanban: Productionize Druid loader - https://phabricator.wikimedia.org/T138264#2398441 (JAllemandou)
[10:02:41] <urandom>	 joal: ^^^ sweet!
[10:02:54] <urandom>	 joal: i'm excited to see how that works out
[10:04:44] <joal>	 urandom: Hey !
[10:05:07] <joal>	 urandom: Having to reload the full cluster made me decide to try and investigate :)
[10:05:51] <joal>	 urandom: I have a questiom though: bulk loading prevents CPU load since no SSTable creation is needed, but the compaction cost is still to be paid, correct?
[11:17:18] * elukey lunch!
[12:38:21] <urandom>	 joal: not necessarily
[12:39:45] <urandom>	 joal: compaction happens, but you're streaming full files, so there is an economy of scale here
[12:40:09] <joal>	 makes sense
[12:40:29] <urandom>	 joal: the amount of compaction required should be less (probably much less)
[12:41:05] <joal>	 That'd be great :)
[12:41:11] <joal>	 Currently trying to test :)
[12:41:19] <joal>	 How is wikimania?
[12:42:02] <urandom>	 it's just the hackathon so far, but yeah, seems to be going great
[12:42:13] <urandom>	 the venue is really nice
[12:42:33] <urandom>	 and the organizers have done/are doing, a great a job
[12:43:00] <urandom>	 milimetric: you're here, aren't you?
[12:43:59] <urandom>	 come out, come out, wherever you are 𝅘𝅥𝅮
[12:45:11] <joal>	 urandom: milimetric should be there indeed :)
[13:51:14] <wikibugs>	 Analytics: Create ops dashboard with info like ipv6 traffic split - https://phabricator.wikimedia.org/T138396#2399103 (Nuria)
[13:58:35] <elukey>	 woa! --^
[14:20:11] <nuria_>	 elukey: crazy talk over here
[14:23:43] <elukey>	 ??
[14:27:58] <elukey>	 (brb from IRC)
[14:35:57] <elukey>	 back!
[14:53:55] <elukey>	 nuria_: I'd need to reboot the host in which piwik runs..
[14:54:07] <elukey>	 would it be feasible to have some downtime?
[14:54:34] <nuria_>	 elukey: yes, it is tier-22
[14:54:35] <elukey>	 not super urgent but we'd need to upgrade the kernel to 4.4
[14:54:38] <nuria_>	 dan's dixit
[14:54:43] <elukey>	 hahahah
[14:54:43] <nuria_>	 you can do it now
[14:54:50] <nuria_>	 there is no expectation of uptime
[14:54:58] <elukey>	 should I announce it to somebody?
[14:55:25] <nuria_>	 no, just to the team
[14:56:09] <elukey>	 all right
[14:57:49] <nuria_>	 elukey: i just updated thread i just sent to ios with this info
[14:58:03] <nuria_>	 elukey: let me know when it is done and i will check things are ok
[14:59:14] <elukey>	 oh boy apache with prefork and mod-php
[14:59:18] <elukey>	  /o\
[14:59:24] <joal>	 :)
[15:01:39] <elukey>	 !log rebooting bohrium.eqiad.wmnet (running piwik) for kernel upgrades
[15:01:46] <elukey>	 mmmmm no log again
[15:01:48] <elukey>	 grrr
[15:01:53] <elukey>	 nuria_: rebooting
[15:02:26] <joal>	 Actually elukey, it gets logged (but no feedback)
[15:02:36] <joal>	 elukey: see https://tools.wmflabs.org/sal/analytics
[15:03:15] <elukey>	 mmmmmm
[15:06:21] <elukey>	 nuria_: up and running again
[15:06:37] <nuria_>	 elukey: ok
[15:12:10] <elukey>	 btw, I don't like that apache settings :P
[15:12:23] <elukey>	 *setting
[15:12:37] <elukey>	 we should at least think about mpm-worker/event
[15:12:48] <elukey>	 and moreover something different than mod-php
[15:12:57] <elukey>	 that eats tons of memory
[15:13:03] <elukey>	 (maybe HHVM?)
[15:13:53] <wikibugs>	 Analytics-Wikistats: Determine total number of external links in all Wikipedias - https://phabricator.wikimedia.org/T137984#2399503 (ezachte) @PleaseStand kudoos for the in-depth analysis! This will be really useful also later on when the hadoop numbers for external links need to be vetted/understood.  For n...
[16:01:13] <nuria_>	 joal:yt?
[16:01:20] <joal>	 joining nuria_
[16:05:22] <wikibugs>	 Quarry: puppet disabled on quarry-main-01 - https://phabricator.wikimedia.org/T136315#2330367 (Krenair) Looks like it's been this way since the 20th of May at 17:21.
[16:06:38] <wikibugs>	 Analytics-Cluster, Analytics-Kanban, Operations, ops-eqiad: analytics1049.eqiad.wmnet disk failure - https://phabricator.wikimedia.org/T137273#2399672 (Cmjohnson) Disk has been ordered.
[16:11:06] <ori_>	 elukey: hhvm would be viable, but come on, the machine is at 8 gigs, we could just double it
[16:11:48] <ori_>	 it's a VM IIRC, so it's super easy
[16:13:26] <ori_>	 elukey: ssh ganeti01.svc.eqiad.wmnet, then: gnt-instance modify -B vcpus=8,memory=24g bohrium.eqiad.wmnet ; gnt-instance reboot bohrium.eqiad.wmnet
[16:23:29] <elukey>	 ori: ah it is on ganeti! Didn't check that..
[16:25:24] <elukey>	 also never jumped on ganeti01.svc.eqiad.wmnet
[16:25:30] * elukey adds it to the TODO list
[16:25:52] <elukey>	 ori: I am also going to follow up on memcached, I have some ideas but didn't get time to write them down
[16:26:22] <ori>	 great, I haven't had a chance to look either
[16:26:46] <ori>	 I can resize the VM if you like. The reason I had the exact commands so quickly is that they were in my .bash_history on ganeti1003
[16:27:12] <elukey>	 nono no need for now, I was just wondering how to improve the stack :)
[16:27:38] <wikibugs>	 Analytics: Create ops dashboard with info like ipv6 traffic split - https://phabricator.wikimedia.org/T138396#2399778 (Nuria) Time to 1st bit? cache info?
[16:28:15] <ori>	 cool
[16:28:27] <ori>	 no no, no no no no no, no no no no no, no no no there's no limit
[16:28:46] <elukey>	 :P
[16:30:45] <wikibugs>	 Analytics: Create ops dashboard with info like ipv6 traffic split - https://phabricator.wikimedia.org/T138396#2399813 (Nuria) @faidon: is this data of insterest per day? or per month?
[16:40:11] <wikibugs>	 Analytics: Evaluate alternatives to varnishkafka: varnishevents - https://phabricator.wikimedia.org/T138426#2399826 (Nuria)
[16:41:02] <elukey>	 https://gerrit.wikimedia.org/r/#/q/topic:traffic-sysctls,n,z - worth to read!
[16:50:45] <wikibugs>	 Analytics, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown, Patch-For-Review: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#2399872 (Legoktm) I found a few bugs preventing it from working (see patch)...but after that...
[16:50:48] <nuria_>	 joal: how do you add teh json serde to qury wmf_raw?
[16:51:10] <nuria_>	 *the
[16:57:28] <elukey>	 yessssssssssss I can repro the timeout!!
[16:57:31] <elukey>	 \o/
[16:59:55] <ema>	 \o/
[17:01:33] * elukey dances
[17:01:38] * elukey goes afk
[17:01:44] * elukey waves to joal and nuria_ 
[17:03:34] <nuria_>	 ciao elukey
[17:03:51] <nuria_>	 elukey: you tell me tomorrow how you were able to repro
[17:04:07] <nuria_>	 ema: i heard the talk went well, eh?
[17:08:23] <joal>	 nuria_: ADD JAR /usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core.jar ;
[17:12:42] <nuria_>	 joal: thank you. will document here: https://wikitech.wikimedia.org/wiki/Analytics/Data/Webrequest#wmf_raw.webrequest
[17:16:15] <joal>	 nuria_: allready a line here: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hive/Queries#JsonSerDe_Errors but can also be added to wmf_raw doc
[17:16:35] <joal>	 thanks nuria_ for keeping our docs up to date !
[17:18:08] <joal>	 Well done elukey :)
[23:02:22] <wikibugs>	 Analytics-Features: Page view tool does not display graphs - https://phabricator.wikimedia.org/T138448#2400667 (Tnegrin)