[02:09:13] <wikibugs>	 Analytics-Kanban, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Patch-For-Review, Spike: [Feed] Establish criteria for blacklisting likely bot-inflated most-read articles - https://phabricator.wikimedia.org/T143990#2733662 (Mholloway) @JMinor I took a stab at incorporating your heuri...
[05:52:57] <elukey>	 !log created 0001009-161020124223818-oozie-oozi-C to run webrequest-load-check_sequence_statistics-wf-upload-2016-10-21-3 (oozie errors)
[07:20:26] <elukey>	 FYI I am rebooting stat100[234]
[07:26:50] <elukey>	 all done, not sure if there are specific checks to do after a boot
[07:27:46] <joal>	 Hi elukey :)
[07:28:48] <elukey>	 o/
[07:30:36] <joal>	 What's up this morning?
[07:31:08] <elukey>	 nothing on fire for the moment :)
[07:31:16] <elukey>	 I rebooted all the stats, including 1001
[07:31:23] <elukey>	 (so a brief outage for our websites)
[07:31:44] <joal>	 okey, from your messaage I understood eveything went smotth?
[07:32:17] <joal>	 elukey: Do you know the kernel change we are deploying with this reboot?
[07:32:57] <elukey>	 yes it is to solve a vunerability bug (11 years old!) discovered in the kernel
[07:33:09] <joal>	 Maaaaaan .... 11 years old !
[07:34:13] <elukey>	 Riccardo gave me yesterday the link to the actual git commit: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=19be0eaffa3ac7d8eb6784ad9bdbc7d67ed8e619
[07:34:14] <joal>	 elukey: upgrading my debian: indeed, new linux image
[07:35:44] <elukey>	 joal: what it would be great to do today is to figure out if the kafka mess that happened yesterday caused data drop
[07:36:22] <joal>	 elukey: We restarted the jobs with higher threshold with mforns, but didn't check
[07:36:44] <joal>	 elukey: I guess looking for timestamp = '-' in webrequest upload is a good start
[07:36:48] <elukey>	 long story short: firewall connection tracking was saturated due to a race condition between puppet and the OS, and connections were dropped
[07:37:13] <elukey>	 I think that it could be worse: https://grafana.wikimedia.org/dashboard/db/varnishkafka
[07:37:13] <joal>	 elukey: And, since the problems only occurred on webrequest upload, I think there is no big deal
[07:37:25] <elukey>	 no no not only upload
[07:37:35] <elukey>	 ALL the brokers were at some point misbehaving
[07:37:36] <joal>	 yes, but oozie errors = only upload
[07:37:38] <elukey>	 at firewall level
[07:38:08] <joal>	 so, since no email from oozie, it means less than 1% data loss if any
[07:38:14] <joal>	 right, I understand
[07:38:44] <elukey>	 but what if we dropped half of the connections from vk, without causing holes? Is it possible?
[07:38:52] <elukey>	 I am thinking out loud
[07:39:18] <elukey>	 if you see https://grafana.wikimedia.org/dashboard/db/varnishkafka?panelId=20&fullscreen
[07:39:42] <elukey>	 from 15:00 to 17:30
[07:39:58] <elukey>	 these are librdkafka failed delivery reports
[07:40:03] <elukey>	 now we have retries
[07:40:32] <elukey>	 (s/now//)
[07:40:48] <elukey>	 buuuut I am paranoid you know :)
[07:40:59] <elukey>	 if feels weird that we didn't drop data with that mess
[07:42:50] <elukey>	 I am checking pivot atm
[07:45:08] <joal>	 elukey: sorry phone
[07:45:50] <joal>	 elukey: If there had been data loss, it would have happened between VK and kafka, not in vk - meaning, sequence numbers are not messed up
[07:46:49] <joal>	 And with our checks in load, would there have been (enough) problems with missing sequence numbers, we would have know it
[07:46:54] <joal>	 elukey: --^
[07:47:09] <joal>	 So I'm confident that if data has been lost, it's very small
[07:48:06] <joal>	 elukey: makes sense what I'm thinking?
[07:49:14] <elukey>	 sure sure
[07:49:20] <moritzm>	 ok to reboot bohrium now, the server hosting piwik?
[07:49:59] <elukey>	 I think so moritzm!
[07:50:55] <elukey>	 mobrovac: ciao! Whenever you want to do the kafka100[12] reboots let me know
[07:50:59] <elukey>	 even this afternoon
[07:55:05] <moritzm>	 it's back up, if anyone uses it, I'd welcome a quick functionality check. but at least it's showing me the mod_auth prompt :-)
[08:00:02] <elukey>	 we'll ask to nuria and milimetric, they are the masters of piwik :)
[08:00:19] <moritzm>	 ok :-)
[08:00:46] <elukey>	 but I can see data flowing in apache
[09:24:25] <moritzm>	 ok to reboot krypton? it runs burrow, so I wanted to doublecheck
[09:26:12] <mobrovac>	 elukey: i'm good to go
[09:26:28] <elukey>	 moritzm: go ahead, thanks!
[09:26:35] <elukey>	 mobrovac: all right let's do it
[09:27:00] <elukey>	 so what I am going to do is (and then you tell me what extra steps we'll need)
[09:27:11] <elukey>	 1) depool kafka1001
[09:27:49] <elukey>	 2) check kafka topics replication, and if good stopping kafka (extra check afterwards)
[09:27:56] <elukey>	 3) reboot kafka1001
[09:28:21] <elukey>	 4) wait to come up again, check topics, launch preferred replica election, check topics again
[09:28:24] <elukey>	 5) re-pool
[09:28:33] <elukey>	 and then same thing with kafka1002
[09:28:39] <mobrovac>	 yup
[09:28:51] <mobrovac>	 in step 4 we also need to check the proxy service is up before repooling
[09:31:47] <elukey>	 sure sure
[09:31:57] <elukey>	 all right proceeding with kafka1001
[09:35:51] <moritzm>	 krypton back up, let me know if anything is fishy
[09:36:14] <elukey>	 sure
[09:36:33] <elukey>	 rebooting kafka1001
[09:40:38] <elukey>	 all good, up and running
[09:40:42] <elukey>	 replicas are in sync
[09:40:48] <elukey>	 and mirror maker is up as well
[09:40:57] <elukey>	 proxy is up too
[09:41:03] <elukey>	 (checking traffic with httpry)
[09:41:55] <mobrovac>	 cool
[09:42:10] <elukey>	 repooling
[09:43:02] <elukey>	 ok I can see 201s flowing
[09:43:11] <elukey>	 let's wait a bit and then do 1002
[09:43:26] <elukey>	 (a bit == some datapoints in grafana)
[09:43:45] <mobrovac>	 i will restart cp now then
[09:43:47] <mobrovac>	 for sanity
[09:43:58] <mobrovac>	 and redo the same after kafak1002
[09:45:23] <mobrovac>	 ok, restarted, let's wait at least 60 seconds now to confirm messages are being handled by cp elukey
[09:46:28] <elukey>	 sure sure
[09:46:39] <elukey>	 mirror maker on 1002 didn't like this reboot
[09:46:45] <elukey>	 I am going to open a phab task
[09:50:46] <milimetric>	 piwik is working fine elukey
[09:53:53] <elukey>	 \o/
[09:54:38] <mobrovac>	 elukey: ok, we can proceed to kafka1002
[09:55:01] <elukey>	 super
[09:57:16] <dcausse>	 hi!
[09:57:38] <dcausse>	 joal: I took the liberty to add you as a reviewer of https://gerrit.wikimedia.org/r/#/c/316931 :)
[09:59:36] <dcausse>	 it involves some oozie and hql and I'm never very confident with this, so when you have some time it'd much appreciated :)
[10:03:31] <elukey>	 mobrovac: kafka1002 back serving traffic
[10:03:54] <mobrovac>	 ok, restarting cp again for sanity
[10:07:28] <elukey>	 joal: I am investigating 3776-21 Oct 2016 00:00:00, probably I'll fail but let's see if I manage to solve the issue (also the same thing for maps)
[10:09:03] <elukey>	 moritzm: analytics reboots should be completed
[10:09:19] <elukey>	 thanks mobrovac!
[10:14:32] <elukey>	 the jobs seems to have failed while waiting the _IMPORTED flag
[10:15:14] <elukey>	 trying to re-run them
[10:15:29] <elukey>	 going also to check the data on HDFS
[10:15:37] <elukey>	 (I should have done it before :P)
[10:16:35] <joal>	 dcausse: Sure, today I'm a bit sick, so probably on monday :)
[10:16:42] <elukey>	 yess data in /mnt/hdfs/wmf/data/raw/webrequest/webrequest_text/hourly/2016/10/21/00 with IMPORTED flag
[10:16:47] <dcausse>	 joal: sure, thanks for your help!
[10:25:34] <joal>	 elukey: Have you found the error with 2016-10-21T00:00 ?
[10:26:54] <elukey>	 joal: from the coordinator logs (hue goes in 500 not sure why) I saw only "waiting for _IMPORTED"
[10:27:02] <elukey>	 but they have it now so I re-run them
[10:27:11] <elukey>	 only maps has _SUCCESS though
[10:27:19] <elukey>	 that I don't remember when it is put in there
[10:29:18] <elukey>	 the maps coordinator seems to have finished
[10:29:22] <elukey>	 text is in progress
[10:30:09] <moritzm>	 elukey: awesome. what about the druid cluster? it's still on the wmf2 4.4 kernel
[10:30:19] <elukey>	 aaaarghhh
[10:30:24] <elukey>	 I forgot Druid
[10:30:41] <joal>	 elukey: 500 from hue is weird, no ?
[10:31:26] <elukey>	 only for the two failed, not sure why :(
[10:31:56] <elukey>	 joal: before you go, my understanding is that I can reboot safely one druid host at the time
[10:32:13] <elukey>	 since there is no special one doing coordination like in hadoop
[10:32:47] <elukey>	 in the meantime the text failed job is in refine
[10:32:53] <joal>	 elukey: Correct, except for clickhouse running on those machines as well
[10:33:18] <elukey>	 whattt
[10:33:23] <elukey>	 I didn't know it
[10:33:32] <joal>	 elukey: If the issue was on IMPORTED, then rerunning it with the files present is a good idea
[10:33:39] <joal>	 and should solve it
[10:33:40] <elukey>	 !log re-run webrequest-load-wf-text-2016-10-21-00 and webrequest-load-wf-maps-2016-10-21-00
[10:33:52] <joal>	 elukey: no need to rerun, just relauch right?
[10:34:20] <joal>	 elukey: test only
[10:34:24] <joal>	 (for clikchouse)
[10:34:26] <elukey>	 is there a diff? I just hit "re-run" a while a go on both in hue
[10:35:16] <joal>	 elukey: this is re-launch :)
[10:35:26] <joal>	 elukey: or re-run, as you wish
[10:35:38] <elukey>	 all right didn't do anything stupid then :)
[10:35:40] <joal>	 elukey: but no need to restart a coord as we do for upload for instance
[10:35:46] <joal>	 :)
[10:35:54] <elukey>	 ahhh yes yes ok
[10:35:59] <elukey>	 I got what you meant
[10:36:08] <joal>	 elukey: nothing to bother anyway, just multiple refine jobs on same data, no prob:)
[10:36:09] <elukey>	 so about clickhouse, all good if I reboot druid hosts?
[10:36:31] <joal>	 elukey: I don't think so, I think ottomata has them running in screens or something
[10:36:45] <elukey>	 bad ottomata is bad :D :D
[10:36:50] <joal>	 :D
[10:36:56] <joal>	 You know him, hmmmmmm ?
[10:37:13] <joal>	 So, I have a loading job currently running, should be done by end of afternoon
[10:37:33] <joal>	 Would you mind waiting for end-of-day to reboot them, maybe with ottomata to restart clickhouse after?
[10:37:50] <elukey>	 sure, no problem, I'll do them this evening
[10:38:07] <joal>	 Thanks a milion elukey, sorry for bothering :(
[10:38:28] <joal>	 elukey: I can't access hue anymore, don't know why
[10:39:01] <elukey>	 I can, what error do you see?
[10:39:19] <joal>	 elukey: nothing loads
[10:39:22] <joal>	 blank page
[10:39:38] <joal>	 elukey: might be related to chrome update
[10:39:46] <joal>	 elukey: will reboot later on today
[10:39:55] <elukey>	 works fine for me :(
[10:40:00] <joal>	 np
[10:40:07] <joal>	 elukey: lleaving for now, ok for you?
[10:40:17] <elukey>	 sure! get some rest :)
[10:40:23] <joal>	 Thanks mate :)
[10:40:28] <joal>	 Later a-team
[12:29:49] <elukey>	 !log created 0001387-161020124223818-oozie-oozi-C for webrequest-load-check_sequence_statistics-wf-upload-2016-10-21-11 (oozie errors)
[13:33:47] <elukey>	 mmmm
[13:34:14] <elukey>	 I suspect that the date/time that I have for the interview with ottomata is not rifght
[13:34:18] <elukey>	 *right
[13:55:53] <grrrit-wm>	 (PS2) Milimetric: [WIP] Migrate from bower to npm instead of yarn [analytics/dashiki] - https://gerrit.wikimedia.org/r/316904 (https://phabricator.wikimedia.org/T147884)
[14:02:14] <milimetric>	 elukey: in an interview just noticed https://stats.wikimedia.org/ has some problems with ssl (check out the console)
[14:03:38] <elukey>	 milimetric: I think it is only a matter or replacing references to 'http://stats.wikimedia.org/cgi-bin/search_portal.pl'
[14:03:58] <elukey>	 yeah
[14:03:58] <elukey>	 <form name='search' action='http://stats.wikimedia.org/cgi-bin/search_portal.pl' method='get'>
[14:04:55] <elukey>	 is there a git repo for this?
[14:05:24] <elukey>	 https://gerrit.wikimedia.org/r/#/admin/projects/analytics/wikistats
[14:06:12] <elukey>	 I think that a grep for "http://" would be enough, checking
[14:10:27] <elukey>	 well.. there are a lot of links to change :P
[14:30:50] <milimetric>	 elukey: did something change in the config for that server after the restart?
[14:31:13] <milimetric>	 (just got out of my interview)
[14:31:27] <elukey>	 nope
[14:34:49] <milimetric>	 hm... wonder why it's failing all of a sudden then
[14:51:00] <elukey>	 milimetric: are you using chrome?
[14:51:08] <elukey>	 might be a browser specific thing
[14:51:09] <milimetric>	 yep
[14:51:16] <milimetric>	 elukey: you don't see the problems?
[14:51:21] <elukey>	 yep I do
[14:51:26] <elukey>	 but I don't have firefox
[14:51:48] <milimetric>	 oh yeah, no problem in ff
[14:51:51] <milimetric>	 cool
[14:52:04] <elukey>	 IIRC joseph told me about a chrome update today
[14:52:12] <milimetric>	 mforns: back, wanna hang out in the cave for a bit before standup
[14:52:13] <elukey>	 so this might be the reason
[14:52:18] <mforns>	 milimetric, sure
[14:52:22] <milimetric>	 aha, ughhhh security is sux
[14:52:26] <elukey>	 but we'd need to update the repo asap milimetric :)
[14:52:28] <milimetric>	 maybe they fixed Dirty COW
[14:52:46] <milimetric>	 elukey: yeah, I emailed ezachte about it
[14:55:10] <elukey>	 super
[15:09:36] <wikibugs>	 Analytics-Kanban, ChangeProp, EventBus, Wikimedia-Stream, Services (watching): Write node-rdkafka event.stats callback that reports stats to statsd - https://phabricator.wikimedia.org/T145099#2734517 (Nuria) a:Ottomata>Nuria
[15:28:06] <wikibugs>	 Analytics: Put wikistats latest in gerrit - https://phabricator.wikimedia.org/T148842#2734545 (Nuria)
[15:28:32] <wikibugs>	 Analytics: Put wikistats latest in gerrit - https://phabricator.wikimedia.org/T148842#2734558 (Nuria)
[15:31:57] <wikibugs>	 Analytics, Research-and-Data, Research-management: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843#2734568 (DarTar)
[15:32:12] <elukey>	 mforns: I am going to restart the last failed oozie job :(
[15:32:30] <mforns>	 elukey, ok
[15:33:26] <elukey>	 !log created 0001564-161020124223818-oozie-oozi-C to re-run webrequest-load-check_sequence_statistics-wf-upload-2016-10-21-14 (oozie errors)
[16:01:33] <nuria>	 milimetric: can we get this changes merged before the karma npm refactor? https://gerrit.wikimedia.org/r/#/c/314622/
[16:02:04] <grrrit-wm>	 (Abandoned) Nuria: Making link to new browser reports more prominent [analytics/wikistats] - https://gerrit.wikimedia.org/r/279965 (https://phabricator.wikimedia.org/T129101) (owner: Nuria)
[16:03:23] <milimetric>	 nuria: actually I can abandon everything but the piwik config changes there
[16:03:40] <nuria>	 milimetric: but we agree piwik has no effect on perf right?
[16:03:44] <milimetric>	 the new way that htmlreplace means it makes sense to move the piwik script in the head the way you pointed out
[16:03:59] <milimetric>	 *the new way htmlreplace works (with the new version)
[16:04:14] <nuria>	 milimetric: ok, then should we abandon that patch?
[16:04:38] <milimetric>	 um... you can leave it and I can strip out the build process changes and leave the piwik in there.  That way I don't forget
[16:06:06] <milimetric>	 I gotta run to the doctor's be back later
[17:08:21] <wikibugs>	 Analytics-Kanban, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Patch-For-Review, Spike: [Feed] Establish criteria for blacklisting likely bot-inflated most-read articles - https://phabricator.wikimedia.org/T143990#2585510 (Niedzielski) ^ This is pretty neat {icon thumbs-up}
[17:32:17] <mforns>	 github is having problems
[17:32:21] <elukey>	 going afk team, have a good weekend!
[17:42:09] <mforns>	 bye elukey !
[18:53:39] <yurik>	 nuria, can we schedule an hour for the first meeting? I'm sure there will be plenty to discuss
[18:54:02] <nuria>	 yurik: can you send an agenda ? that way we can see whether we truly need an hour
[18:54:53] <yurik>	 nuria, hm, well, i was hoping for a demo of all the possible ways available from analytics - and judging by our chat here yesterday, there is plenty :)
[18:55:14] <nuria>	 yurik: That would take 3 hours yurik, not 1
[18:55:26] <yurik>	 nuria, hehe, i am hoping for a short one :)
[18:55:55] <nuria>	 yurik: that is why i was trying to see whta are you working on / what needs there are and then other more concrete meetings can be scheduled
[18:55:57] <yurik>	 we need to figure out how our existing tech and requirements can be solved with what your team has been working on, where we duplicate our efforts and try to eliminate that
[18:56:22] <yurik>	 i guess we could make it short at first and schedule an extra one later in the week
[18:56:23] <nuria>	 "all possible ways" of .. ahem ... doing what thing...
[18:56:47] <yurik>	 visualizing analitical data :)
[18:57:05] <yurik>	 which means - everything from gathering, to filtering/sorting/aggregating to visualizing
[18:57:19] <yurik>	 in other words - analitics data pipeline
[18:57:20] <nuria>	 ok, that is one thing that as i said has several solutions
[18:57:33] <nuria>	 depends on the usecase
[18:57:52] <yurik>	 right - and we should understand the difference of the solution, and present our usecases :)
[18:58:02] <nuria>	 it works the other way arround
[18:58:03] <yurik>	 anyway, i guess we can start with 30min, and move on from there
[18:58:10] <nuria>	 we learn about use cases 1st
[18:58:16] <yurik>	 oki
[18:58:30] <nuria>	 undersand what are we trying to do and then we work on tech
[18:58:38] <nuria>	 technology is not teh solution to lal problems
[18:58:40] <nuria>	 *all
[18:58:52] <nuria>	 ^ yurik
[18:59:12] <nuria>	 sorry
[18:59:22] <nuria>	 "technology is not the solution to all problems"
[19:00:02] <nuria>	 yurik: Example: This dataset : https://wikitech.wikimedia.org/wiki/Analytics/Data/Browser_general
[19:00:17] <yurik>	 sorry, prod issue :(
[19:01:06] <nuria>	 yurik: could have greatly helped with this project: https://github.com/wikimedia-research/Discovery-Portal-Adhoc-JavaScriptSupport
[19:09:49] <yurik>	 seems that would be perfectly aligned with what you do.  Sorry, seems like all cassandra servers just died on us, working on switching maps backend servers
[19:10:41] <nuria>	 yurik: k
[19:44:22] <wikibugs>	 Analytics-Kanban, ChangeProp, EventBus, Wikimedia-Stream, Services (watching): Write node-rdkafka event.stats callback that reports stats to statsd - https://phabricator.wikimedia.org/T145099#2735127 (Nuria) @ottomata: in what repo does this code go, I imagine it no longer goes into kasocki
[20:07:03] <mforns>	 bye a-team have a nice weekend!
[20:21:22] <milimetric>	 back