[07:45:19] <wikibugs>	 Analytics / Wikimetrics: Cannot edit or delete cohorts - https://bugzilla.wikimedia.org/67664 (Rahmanuddin Shaik) UNCO p:Unprio s:normal a:None Hi, I cannot edit a cohort, adding more usernames and removing irrelevant usernames (those found invalid, etc) is a good feature.  Also, I cannot d...
[08:50:19] <wikibugs>	 Analytics / General/Unknown: page view statistics for Wikinews seem to be wrong - https://bugzilla.wikimedia.org/67411#c9 (christian) NEW>RESO/FIX Ssl requests get fed into webstatscollector again.
[10:13:15] <grrrit-wm1>	 (CR) Nuria: [C: 2] "I have tested these changes on eswiki on dev populating about a year back of reports, things are working well and the throttle has worked " [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142007 (https://bugzilla.wikimedia.org/66841) (owner: Milimetric)
[10:13:25] <grrrit-wm1>	 (Merged) jenkins-bot: Remove limit on recurrent, add throttling [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142007 (https://bugzilla.wikimedia.org/66841) (owner: Milimetric)
[10:13:28] <milimetric>	 yay
[10:49:03] <wikibugs>	 Analytics / Refinery: Story: Admin has versioned and sync'ed files in HDFS - https://bugzilla.wikimedia.org/67129 (christian) a:christian
[11:35:03] <grrrit-wm1>	 (PS1) QChris: Add repository description [analytics/refinery] - https://gerrit.wikimedia.org/r/144676
[11:35:05] <grrrit-wm1>	 (PS1) QChris: Add basic deployment script [analytics/refinery] - https://gerrit.wikimedia.org/r/144677 (https://bugzilla.wikimedia.org/67129)
[11:49:34] <grrrit-wm1>	 (PS1) Yurik: Initial checkin - some data sanitizing code [analytics/zero-sms] - https://gerrit.wikimedia.org/r/144682
[12:27:28] <milimetric>	 nuria: http://test-reportcard.wmflabs.org/graphs/newly_registered
[12:27:29] <milimetric>	 :(
[12:27:50] <milimetric>	 it happens when I run more than one recurrent report at a time
[12:28:05] <milimetric>	 I'm thinking one run doesn't get a chance to finish before the next one starts, so I'm testing that theory now
[12:33:48] <milimetric>	 oh oops, pasted wrong thing
[12:33:54] <milimetric>	 http://pastebin.com/0fXJyq5D
[12:51:20] <grrrit-wm1>	 (CR) QChris: Add basic deployment script (4 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/144677 (https://bugzilla.wikimedia.org/67129) (owner: QChris)
[12:55:46] <milimetric>	 ok, figured it out
[12:55:48] <milimetric>	 2 problems
[12:56:15] <milimetric>	 1. pickle bug / limitation that causes the recursion limit to get hit for long chains of subtasks (this will happen if we try to backfill too much)
[12:57:13] <milimetric>	 2. overlapping runs cause index violations when they try to insert, harmless problem really
[12:57:39] <qchris>	 Harr. Why cannot things be easy?
[12:59:33] <milimetric>	 :)
[12:59:55] <milimetric>	 problem 2. is really a non-issue because we're running these things at a 2 minute increment just for testing / to generate data faster
[13:00:06] <milimetric>	 in real life it'll run nightly and the good news is that it finishes really fast
[13:00:23] <qchris>	 But pickle got in the way before. csalvia spent some time fixing issues around it :-(
[13:00:25] <milimetric>	 so we should never overlap.  Then, even if we do, that index actually protects us from any bad consequence
[13:00:35] <milimetric>	 yeah, the pickle issue is a bitch
[13:00:42] <milimetric>	 there's a work-around: sys.setrecursionlimit(10000)
[13:00:45] <milimetric>	 but... yeah :D
[13:00:47] <qchris>	 :-)
[13:00:56] <milimetric>	 feels *very* wrong
[13:01:05] <qchris>	 I guess if the problem occurs again .... sys.setrecursionlimit(20000)
[13:01:17] <milimetric>	 haha
[13:01:24] <milimetric>	 we could switch away from pickle too
[13:01:31] <milimetric>	 might not happen with other serializers
[13:01:44] <milimetric>	 and I think meanwhile we fixed the serialization problem that was preventing that switch (or we should anyway)
[13:01:51] <milimetric>	 really really time for that tech-debt sprint
[13:02:08] <qchris>	 Sure. :-)
[13:02:59] <milimetric>	 heh, amusingly, I just ran 5 projects at the same time, backfilling for 2 years
[13:03:05] <milimetric>	 it works but in a funny way
[13:03:24] <milimetric>	 1st run - creates pending reports for 1st project (doesn't finish yet)
[13:03:44] <milimetric>	 2nd run - integrity error creating pending reports for 1st project, goes onto second project (doesn't finish)
[13:03:57] <milimetric>	 1st run - gets to 2nd project, but 2nd run beat it there, so it goes to 3rd project
[13:04:13] <milimetric>	 so they stagger... sort of, and then 3rd run starts and it's ALL confusing at this point
[13:04:26] <milimetric>	 but basically, it all works out as long as recursion limit is high
[13:04:37] <qchris>	 Hahaha :-D
[13:06:38] <milimetric>	 resilience through ignorance!
[13:21:25] <milimetric>	 well, massive explosions and everything, wikimetrics just ran 2770 reports in about 20 minutes
[13:23:29] <qchris>	 \(^_^)/
[13:47:55] <nuria>	 milimetric, problem 2) we alredy knew about, we talked about it the other day
[13:49:11] <nuria>	 milimetric: part of teh problem that preventing the pickle switch is fixed, yes
[13:49:39] <nuria>	 it is a matter of not serializing to the queue the whole world
[13:56:43] <nuria>	 *problem 2) we already knew about,
[15:51:47] <nuria>	 kevinator: can you share the slide deck/
[15:51:55] <nuria>	 I would liek to add couple more slides
[15:52:01] <nuria>	 *I would like
[15:52:13] <kevinator>	 https://docs.google.com/a/wikimedia.org/presentation/d/1Y2uI_oOhXGpcn8y-EHBAAqzxS2Lp5OFEXIkIkd6J6A0/edit#slide=id.ge048dd32_142
[15:52:39] <kevinator>	 I created a copy with the same permissions.  I guess it didn’t notify you
[15:59:03] <kevinator>	 nuria: did you see the link I posted?
[15:59:19] <nuria>	 in a meeting now , will look
[16:07:27] <milimetric>	 whoops, I forgot to sign on
[16:07:32] <milimetric>	 whoops, I forgot to sign on
[16:07:34] <milimetric>	 kevinator: I'm around now if you pinged me
[16:12:04] <kevinator>	 I posted the link above… can you have a look at it?
[16:13:31] <kevinator>	 I wonder if I need to record somewhere how many points were closed this sprint (45).  If a story gets closed later does Scrumbugs know not to update the points remaining in an old sprint?
[16:14:12] <kevinator>	 milimetric: i was talking to you
[16:14:23] <milimetric>	 right
[16:14:28] <milimetric>	 I wasn't logged on so can you re-post?
[16:14:59] <milimetric>	 I'm not 100% sure but I think Scrumbugz updates old sprints.  However, if we don't close stories we should move them to the next sprint
[16:15:39] <kevinator>	 https://docs.google.com/presentation/d/1Y2uI_oOhXGpcn8y-EHBAAqzxS2Lp5OFEXIkIkd6J6A0/edit?usp=sharing
[16:16:22] <kevinator>	 IIRC we are only going to showcase schema counts in Graphite and Newly Registered User
[16:26:27] <milimetric>	 kevinator: the 34 points / developer / sprint is without interruptions
[16:26:43] <milimetric>	 so that's kind of the "ideal"
[16:27:22] <milimetric>	 but we're still working on calibrating, so not sure if it's worth it to mention points.  Just basically: "we've adjusted the points so that we have a broader range, comparing to previous sprints isn't too relevant"
[16:27:46] <milimetric>	 and yes, demo-wise, just those two
[16:28:03] <milimetric>	 if you clean up slide 5 (stray bullets), looks good
[16:28:23] <milimetric>	 a LOT easier since Scrumbugz, eh?!
[16:30:50] <kevinator>	 yes… a lot easier
[16:31:00] <kevinator>	 I removed teh reference to 34 points
[16:31:22] <kevinator>	 The stray bullets were to write the names of presenters.
[16:31:33] <kevinator>	 is nuria presenting schema counts in graphite?
[16:32:29] <nuria>	 kevinator: done with meeting, let me include couple slides
[16:32:53] <kevinator>	 thanks
[16:43:39] <nuria>	 kevinator: can you give edit permits to all users
[16:43:45] <nuria>	 please
[16:44:46] <kevinator>	 done
[16:47:44] <nuria>	 kevinator: i have added two slides, since regarding counts wht we have to show is very little you can share the deck while i talk briefly
[16:48:00] <kevinator>	 ok
[16:48:20] <nuria>	 Kevinator: just run through teh whole presentation fast in your laptop now so all slides and images are chached
[16:48:23] <nuria>	 *cached
[16:48:48] <kevinator>	 done… i got them
[16:49:58] <kevinator>	 I’m on my way to conference room now
[17:22:04] <danilo_>	 hi, I have made this wikipedia ranking https://tools.wmflabs.org/ptwikis/Wikirank , but I want to make it more precise by counting characters instead of bytes, is there any way to count characters in articles? a bytes per character coeficient for each language perhaps?
[17:25:43] <Ironholds>	 danilo_, without extracting the dumps or poking the API constantly?
[17:25:53] <Ironholds>	 hrm. Probably not :/.
[17:29:15] <danilo_>	 ok :/ thanks
[19:23:23] <milimetric>	 qchris / nuria: I'm going to try adding this to our /static/public directory configuration in Apache (in staging):
[19:23:24] <milimetric>	 Header set Access-Control-Allow-Origin "*"
[19:23:43] <milimetric>	 let me know if you think that's a crappy idea, I'm thinking it's fine
[19:40:07] <qchris>	 milimetric: I lack experience with this header. But reading about it does not make my alarm bells go off.
[19:40:19] <milimetric>	 yeah
[19:40:24] <qchris>	 Are we doing it to solve an issue people are having, or just to be proactive?
[19:40:36] <milimetric>	 agreed.  now the problem is how to set it and make it pass through the weird proxy thing we have set up
[19:40:49] <milimetric>	 :) thanks qchris!
[19:56:36] <nuria>	 milimetric
[19:56:40] <nuria>	 that is teh cors
[19:56:53] <nuria>	 so anyone can request those files in x-domain
[19:57:03] <milimetric>	 yeah, i know
[19:57:22] <milimetric>	 but it is teh broken through our proxy :(
[19:57:38] <nuria>	 are you adding it to flask?
[19:57:50] <nuria>	 I would add it to apache for that dir only , right?
[19:58:33] <milimetric>	 yeah, i'm trying to add that line to apache's <Directory> config for that line
[19:58:44] <milimetric>	 but apache serves through nginx
[19:58:56] <milimetric>	 and seems to not pass that along
[19:59:13] <nuria>	 mm...maybe hedaers are whitelisted on nginx side
[19:59:36] <nuria>	 BTw, we should also add cache headers while we are at it.
[19:59:44] <nuria>	 *headers
[20:01:28] <milimetric>	 yes, indeed
[20:03:22] <nuria>	 milimetric: in meeting , will check back in a bit
[20:03:27] <milimetric>	 no prob nuria, I know
[20:03:41] <milimetric>	 I'm just wondering out loud, I'll go bother labs people
[21:44:05] <wikibugs>	 Analytics / General/Unknown: Packetloss issues on oxygen (and analytics1003) - https://bugzilla.wikimedia.org/67694 (christian) NEW p:Unprio s:normal a:None It seems oxygen and analytics are having packetloss issues. Currently >10%.    From http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-...
[21:44:39] <qchris>	 milimetric: You said you wanted to grab production issues :-) ... ^
[21:45:05] <milimetric>	 yep, ok
[21:45:54] <qchris>	 Thanks :-)
[21:47:18] <wikibugs>	 Analytics / General/Unknown: Packetloss issues on oxygen (and analytics1003) - https://bugzilla.wikimedia.org/67694 (Dan Andreescu) a:Dan Andreescu
[21:48:02] <wikibugs>	 Analytics / General/Unknown: Packetloss issues on oxygen (and analytics1003) - https://bugzilla.wikimedia.org/67694 (Dan Andreescu)
[21:56:28] <tnegrin>	 thanks guys -- let me know if you need anything from me
[21:57:33] <wikibugs>	 Analytics / Refinery: Story: Admin has duplicate monitoring in Icinga - https://bugzilla.wikimedia.org/67128 (Dan Andreescu) a:Dan Andreescu>christian
[22:03:33] <wikibugs>	 Analytics / General/Unknown: Packetloss issues on oxygen (and analytics1003) - https://bugzilla.wikimedia.org/67694#c1 (Dan Andreescu) Will write here anything I find.  So it looks like packets into Analytics1003 increased by an unusual margin starting around 20:30 and going through 22:00 at the time o...
[22:07:38] <tnegrin>	 milimetric: check out this kafka graph
[22:07:38] <tnegrin>	 http://ganglia.wikimedia.org/latest/?r=week&cs=&ce=&tab=v&vn=kafka&hide-hf=false
[22:07:50] <tnegrin>	 looks like 1022 (upload server) is spiking?
[22:08:31] <wikibugs>	 Analytics / General/Unknown: Packetloss issues on oxygen (and analytics1003) - https://bugzilla.wikimedia.org/67694#c2 (Toby Negrin) Kafka graphs show a spike from 1022:  http://ganglia.wikimedia.org/latest/?r=week&cs=&ce=&tab=v&vn=kafka&hide-hf=false
[22:09:32] <wikibugs>	 Analytics / General/Unknown: Packetloss issues on oxygen (and analytics1003) - https://bugzilla.wikimedia.org/67694#c3 (Dan Andreescu) according to puppet, analytics1003 is set up with role::analytics::kafkatee::webrequest::mobile, but I'm not too familiar with the infrastructure.  My theory so far is...
[22:09:50] <milimetric>	 yeah, tnegrin, I was just commenting the same thing :)
[22:09:57] <milimetric>	 you think it's world cup semis too right?
[22:10:10] <Ironholds>	 huh. Okay, I hadn't thought of that ;p.
[22:10:11] <tnegrin>	 well -- yeah -- it is from europe
[22:10:25] <tnegrin>	 my bad -- VA
[22:10:39] <Ironholds>	 milimetric, so these are requests for images, nooot uploads?
[22:10:51] <tnegrin>	 anyone know how to dump the kafka stream?
[22:11:11] <milimetric>	 yeah, upload serves images that have been uploaded, I think
[22:11:22] <milimetric>	 and also takes upload requests of course
[22:11:30] <Ironholds>	 hmn.
[22:11:52] <milimetric>	 checking kafka graph closely now
[22:13:08] <Ironholds>	 I'm going to try and grab chad and see if there's any data we can get from the spike period itself
[22:13:12] <Ironholds>	 (if we act fast)
[22:13:43] <milimetric>	 yeah, the spike is clear  here: http://ganglia.wikimedia.org/latest/graph_all_periods.php?hreg[]=analytics1003.eqiad.wmnet&mreg[]=kafka.rdkafka.topics..%2B%5C.next_offset.per_second&z=large&gtype=stack&title=kafka.rdkafka.topics..%2B%5C.next_offset.per_second&aggregate=1&r=4hr
[22:14:09] <milimetric>	 it's just too coincidental that it started at the same exact time as the match
[22:15:25] <Ironholds>	 yup
[22:15:57] <tnegrin>	 totally
[22:17:01] <wikibugs>	 Analytics / General/Unknown: Packetloss issues on oxygen (and analytics1003) - https://bugzilla.wikimedia.org/67694#c4 (Dan Andreescu) problems seem fairly contained to esams, for example:  http://ganglia.wikimedia.org/latest/?hreg[]=%28amssq%7Ccp%29.%2B&mreg[]=kafka.varnishkafka%5C.kafka_drerr.per_sec...
[22:17:10] <milimetric>	 http://ganglia.wikimedia.org/latest/?hreg[]=%28amssq%7Ccp%29.%2B&mreg[]=kafka.varnishkafka%5C.kafka_drerr.per_second&z=large&gtype=line&title=kafka.varnishkafka%5C.kafka_drerr.per_second&aggregate=1&r=4hr&dg=1&tab=v
[22:17:13] <Ironholds>	 just to check, is this data being thrown on the floor or going somewhere? Only this seems like it'd make a fascinating writeup
[22:17:32] <milimetric>	 thrown on the floor from udp2log, but hopefully not from kafka, but maybe
[22:17:50] <milimetric>	 unfortunate timing on a team decision we *just* made to have me handle these kinds of problems
[22:17:58] <milimetric>	 obviously christian would have had an actual answer :)
[22:18:08] <Ironholds>	 yeah, but is kafka storing it anywhere? Like: is hive/hadoop down, or just inaccessible?
[22:18:18] <Ironholds>	 non-urgent question, I just think it could make an interesting writeup
[22:18:24] <milimetric>	 yeah, if it's not dropping it, it should be all written to hdfs
[22:18:34] <tnegrin>	 we can ask otto tomorrow where the kafka data is
[22:18:38] <Ironholds>	 "7, 1, 50k: what the world cup means for data"
[22:18:39] <tnegrin>	 I assume it's on the brokers
[22:18:42] <Ironholds>	 *nods*
[22:18:42] <milimetric>	 unless the cluster is still down, which I don't remember what andrew's last status on that was
[22:19:30] <milimetric>	 so, it's not coinciding with when the game started, but with when drdee said "holy shit"
[22:19:37] <milimetric>	 followed shortly by tnegrin saying "holy shit"
[22:19:44] <milimetric>	 so I'm assuming something crazy happened
[22:19:48] <milimetric>	 DON'T TELL ME
[22:19:54] <qchris>	 milimetric: udp2log drops. kafka should have them stored.
[22:19:59] <Ironholds>	 hehe
[22:20:01] <Ironholds>	 qchris, yay!
[22:20:02] <milimetric>	 :)
[22:20:03] <milimetric>	 right?
[22:20:11] <milimetric>	 #guardianangel
[22:20:12] <Ironholds>	 go analytics engineering :D
[22:20:16] <Ironholds>	 you know what this means, right?
[22:20:25] <Ironholds>	 Our analytics infrastructure is better at handling the world cup than Brazil
[22:20:28] <milimetric>	 lol
[22:20:41] <qchris>	 :-P
[22:20:42] <milimetric>	 Ironholds for director
[22:20:43] <tnegrin>	 DON"T TELL HIM
[22:20:46] <Ironholds>	 sorry!
[22:20:50] <milimetric>	 lol
[22:20:51] <Ironholds>	 milimetric, you should go watch the game.
[22:20:54] <tnegrin>	 HE'LL BE CROSS
[22:20:55] <milimetric>	 seriously
[22:21:02] <Ironholds>	 For best effect, play it double-speed with the benny hill theme tune in the background.
[22:21:04] <milimetric>	 ok guys, i'm out, doesn't look like anything urgent
[22:21:08] <terrrydactyl>	 you should watch it milimetric
[22:21:09] <milimetric>	 lol, got it
[22:21:09] <Ironholds>	 but that's my advice for all sporting events, so..
[22:21:14] <tnegrin>	 agreed
[22:21:27] <qchris>	 :-)
[22:21:28] <terrrydactyl>	 you can send you live commentary here. haha
[22:22:43] <Ironholds>	 kafka staying up, though? That's awesome. Talk about road-testing stuff!
[22:22:58] * Ironholds is tempted to send Otto an email with a big thumbs up
[22:23:01] <milimetric>	 let's see how it handles popes
[22:23:05] <milimetric>	 then I'll be impressed
[22:23:30] <Ironholds>	 whether it'll work, or pope the hell out, you mean?
[22:27:08] <Ironholds>	 (look, I make /myself/ laugh ;p)
[22:29:22] <marktraceur>	 Just gotta give kafka enough pope to hang itself with
[22:33:49] <milimetric>	 ok, signing off for good now
[22:38:55] <grrrit-wm1>	 (PS1) QChris: Adapt default auxpath for cluster setup [analytics/refinery] - https://gerrit.wikimedia.org/r/144842
[23:09:57] <marktraceur>	 Ironholds: How have you not groaned at me
[23:23:45] <Ironholds>	 marktraceur, for what?
[23:23:56] <marktraceur>	 2014-07-08 - 15:29:22  <marktraceur> Just gotta give kafka enough pope to hang itself with
[23:28:19] <Ironholds>	 uuugj
[23:37:16] <wikibugs>	 Analytics / General/Unknown: Packetloss issues on oxygen (and analytics1003) - https://bugzilla.wikimedia.org/67694#c5 (Bawolff (Brian Wolff)) We've had some limited reports that some deleted files are being purged, which could be caused by packetloss of the htcp packets. I wonder if these issues are r...
[23:56:03] <wikibugs>	 Analytics / Refinery: Story: Admin has duplicate monitoring in Icinga - https://bugzilla.wikimedia.org/67128#c6 (Kevin Leduc) Moving story to next sprint since it has not been completed this Sprint.
[23:56:16] <wikibugs>	 Analytics / Refinery: Story: Admin has versioned and sync'ed files in HDFS - https://bugzilla.wikimedia.org/67129#c5 (Kevin Leduc) Moving story to next sprint since it has not been completed this Sprint.