[04:20:32] <grrrit-wm>	 (PS3) Milimetric: Add pretty symlink for WikimetricsBot [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/143040 (https://bugzilla.wikimedia.org/66087)
[04:22:09] <grrrit-wm>	 (CR) Milimetric: Add pretty symlink for WikimetricsBot (2 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/143040 (https://bugzilla.wikimedia.org/66087) (owner: Milimetric)
[04:22:12] <grrrit-wm>	 (PS4) Milimetric: Add pretty symlink for WikimetricsBot [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/143040 (https://bugzilla.wikimedia.org/66087)
[04:32:49] <grrrit-wm>	 (CR) Milimetric: [C: -1] "Cool, I like in general. A couple of pointers and some nitpicks." (7 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142514 (owner: Milimetric)
[08:21:05] <grrrit-wm>	 (CR) QChris: [C: 2 V: 2] Ignore temporary vim files [analytics/refinery] - https://gerrit.wikimedia.org/r/143485 (owner: Milimetric)
[09:29:52] <hashar>	 qchris: Gerrit Code Review 2.8.1-4-ga1048ce ready !!  Thank you very much
[09:30:06] <qchris>	 Wooohooo! \o/
[09:31:02] <qchris>	 (And that war also sneaks in the fix for the stream-events that we did, but never got deployed)
[09:32:22] <hashar>	 that works on integration-dev.eqiad.wmflabs at least
[09:32:23] <hashar>	 \O/
[09:32:34] <hashar>	 OpenStack has been hit by that bug as well -:(
[09:32:47] <qchris>	 So glad to finally see a chance of closing that bug.
[09:33:59] <hashar>	 I have hunt for it a bit
[09:34:09] <hashar>	 I am very happy Krinkle found the bug report
[09:34:17] <qchris>	 Yup. Me too.
[09:34:57] <qchris>	 I tried replicating the bug for several days when it first got reported, but failed consistently (although trying several different jdks)
[09:35:06] <qchris>	 Now I know why I failed to reproduce :-)
[09:38:24] <hashar>	 my contact at openstack-infra (pelix) asked: any idea as to why the bug in mina-sshd wouldn't be ticked if using a host key generated with openssl?
[09:38:28] <hashar>	 who knows :-]
[09:38:34] <hashar>	 mina sshd patch fix it hehe
[09:39:40] <qchris>	 A OpenSSL generated key would not have fixed it.
[09:39:41] <qchris>	 But!
[09:40:07] <qchris>	 Extracting the (gerrit generated, or generated by whatever means) key and using BouncyCastle
[09:40:22] <qchris>	 from within gerrit to serve it to the user would have fixed it.
[09:40:36] <qchris>	 Because the error is in mina-sshd's DH implementation.
[09:40:54] <qchris>	 And using BouncyCastle from within gerrit, would not use that code.
[09:41:31] <qchris>	 And the error is only triggered for a certain kind of JDKs, because they changed how the represent keys.
[09:41:41] <qchris>	 (stripping vs. no stripping of leading zeros)
[09:41:51] <hashar>	 ohhh
[09:42:03] <hashar>	 sounds like a snowball of tiny bugs
[09:42:07] <qchris>	 Yup.
[09:42:11] <hashar>	 would you mind copy pasting the above explanation to https://bugzilla.wikimedia.org/show_bug.cgi?id=53895  ?
[09:42:20] <hashar>	 Openstack folks are lurking the bug, I feel to lazy to copy paste it :)]
[09:42:31] <qchris>	 Ok. Will do.
[09:42:33] <hashar>	 (afraid of adding non sense in between and thus forcing you to correct me later on hehe)
[09:42:36] <qchris>	 But first: lunch :-D
[09:42:50] <hashar>	 it is not even noon! :-D
[09:42:53] <hashar>	 have a good lunch
[09:43:06] <hashar>	 and thank you for your help with Gerrit related issues. It is much appreciated
[09:43:23] <hashar>	 socially since I have exactly no clue how it works and i am java illiterate
[09:43:27] <qchris>	 hashar: Noon or not ... my wife is starving since 11:00 ... and my wife is the boss :-D
[09:43:38] <hashar>	 definitely
[09:43:40] <hashar>	 same deal here
[09:43:51] <qchris_away>	 ;-)
[13:11:59] <grrrit-wm>	 (CR) Nuria: Fix wiki cohort display for report cohorts (4 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142514 (owner: Milimetric)
[13:13:23] <grrrit-wm>	 (CR) Milimetric: Fix wiki cohort display for report cohorts (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142514 (owner: Milimetric)
[13:13:34] * Ironholds awakes
[13:41:17] <grrrit-wm>	 (PS2) Milimetric: [WIP] Oozify sequence_stats hive script [analytics/refinery] - https://gerrit.wikimedia.org/r/143486 (https://bugzilla.wikimedia.org/67128)
[14:48:07] <milimetric>	 oh qchris_away: you know the hive trick you did to make it unnecessary to add the serde jar manually each time people ran hive?
[14:48:25] <milimetric>	 is that deep enough in the bowels of the machine that it means we don't have to add the jar from anywhere?
[14:48:40] <milimetric>	 as in: from coordinators, workflows, etc.
[14:56:19] <ottomata>	 milimetric: i think the only way to find out is to test it! :/ :)
[15:27:23] <qchris>	 milimetric: I totally do not know. I haven't tried.
[15:29:28] <milimetric>	 k qchris thanks
[15:34:05] <qchris>	 I just looked over Oozie's HiveMain (which is used by HiveAction to do the work), and it looks like it should work, if you provide the system's Hive configuration file.
[15:40:30] <hashar>	 qchris: thanks for your super long detailed description of gerrit hash mismatch issue ( https://bugzilla.wikimedia.org/show_bug.cgi?id=53895#c28 )
[15:40:34] <hashar>	 that is very nice
[15:40:55] <qchris>	 Ha. Not sure if anyone will gonna read it ... :-)
[15:41:09] <qchris>	 But it should give the OpenStack people some idea of how/why.
[15:41:34] <qchris>	 I hope they can solve it on their end in a similar way.
[15:42:02] <qchris>	 It certainly cost us much time to find a solution. Would be a pity if they'd have to spend as much time as we did.
[15:54:58] <milimetric>	 ottomata: sequence_stats needs webrequest_source right, as a parameter?
[15:58:22] <ottomata>	 yes, i believe so
[16:01:24] <hashar>	 qchris: openstack infra folks are tracking the bug :-]
[16:01:42] <hashar>	 qchris: they have the issue hitting them for some integration tests they are doing. That is what resurrected the bug on our side
[16:02:01] <hashar>	 qchris: guess they will "just" upgrade
[16:02:09] <qchris>	 :-D
[16:04:24] <Ironholds>	 oh, ottomata, while you're here, 2 things
[16:04:43] <ottomata>	 yup?
[16:04:51] <Ironholds>	 (1) it looks like RHive is now being actively supported by its parent company. woo! I'll be testing it when I get back and flinging bugs/patches upstream, and hopefully we'll have a connector solution there.
[16:05:25] <Ironholds>	 (2) when's the CDH5 conversion expected? I want to be sure I can adequately warn consumers that we will/won't have data in [period]
[16:09:15] <ottomata>	 1 cool!
[16:09:31] <ottomata>	 2: ahhhh, still not sure, i was hoping this week but the cdh5 labs testing is taking longer than expected
[16:11:32] <Ironholds>	 tis okay :)
[16:11:45] <Ironholds>	 hey qchris, you fancy writing a parser for puppet? ;p
[16:12:20] <Ironholds>	 trying to work out how to avoid manual tasks with updating the IP ranges for internal traffic in..whatever we call the pageviews solution. Unfortunately the only machine-readable, consistently updated list is in a puppet manifest.
[16:12:39] <Ironholds>	 This may be an area where I go "this sort of looks like a tremendous pain"
[16:12:46] <Ironholds>	 but, implementation details
[16:12:51] <qchris>	 Ironholds: Sad to see that your email to Ops has not seen official responses.
[16:13:00] <qchris>	 But the issue is even more involved :-/
[16:13:18] <qchris>	 Like X-Forwarded-For handling ...
[16:13:36] <qchris>	 And different parties in wmf trusting different IPs to set X-Forwarded-For correctly.
[16:13:45] <Ironholds>	 actually I got a great reply from manybubbles but it was reply, not reply-all
[16:13:46] <Ironholds>	 yep
[16:13:54] <Ironholds>	 I mean, we have the XFF extension, which has one set of rules
[16:14:11] <Ironholds>	 that's used by checkuser really effectively
[16:14:21] <Ironholds>	 but I'm skeptical that we don't have 3 or 4 competing standards there ;p
[16:14:49] <qchris>	 Like ... wp-zero trusting some special proxies :-)
[16:15:06] <Ironholds>	 yep :D
[16:15:18] <qchris>	 Aaaaand different legacy implementations using treating different parts as "internal"
[16:15:21] <Ironholds>	 the opera ranges are actually in the XFF setup for MW, which was interesting to discover, along with a lot of other ones
[16:15:26] <Ironholds>	 yep :(
[16:15:29] <qchris>	 Fun fun fun.
[16:15:49] <Ironholds>	 from my point of view, I think we may not need internal tracking/exclusion for the initial setup.
[16:15:58] <qchris>	 I'll respond to your Ops thread if nothing happens there. Just to get discussion started.
[16:16:00] <Ironholds>	 I mean, if we start talking about multimedia image hits and such, absolutely.
[16:16:27] <Ironholds>	 but I can't imagine there are text/html requests there, and the filters on API requests are positive, not negative (it looks for specified user agents and excludes anything without those user agents)
[16:16:34] <Ironholds>	 I will test this and see what I can see
[16:17:05] <qchris>	 Be sure to have a look at requests the ssl terminators make on behalf of users.
[16:17:22] <Ironholds>	 ohh, yes.
[16:17:27] <Ironholds>	 that's a good point. Oh dear.
[16:17:45] * Ironholds headscratches. Ugh.
[16:17:47] <qchris>	 I am sure Ops know more edge cases.
[16:17:52] <Ironholds>	 yep.
[16:18:02] <qchris>	 That's why I had hoped they chime in more to your question.
[16:18:18] <Ironholds>	 I am really pointing towards "we should not exclude this". I'll make the argument and see what people in ops/community/you and erikZ say when I propose it.
[16:20:12] <qchris>	 k
[16:24:23] <milimetric>	 doh, the script does it
[16:24:38] <milimetric>	 I wish I knew why my brain turned against me when I do this stuff...
[16:33:48] <grrrit-wm>	 (PS3) Milimetric: [WIP] Oozify sequence_stats hive script [analytics/refinery] - https://gerrit.wikimedia.org/r/143486 (https://bugzilla.wikimedia.org/67128)
[16:52:57] <grrrit-wm>	 (PS1) QChris: Use index when querying for editors in a date range [analytics/geowiki] - https://gerrit.wikimedia.org/r/143636
[16:54:21] <grrrit-wm>	 (CR) QChris: "Thanks go to springle for finding the issue, alerting" [analytics/geowiki] - https://gerrit.wikimedia.org/r/143636 (owner: QChris)
[17:26:31] <milimetric>	 qchris: if you're around, what bug number is the gerrit problem?
[17:26:41] <milimetric>	 never mind, found it
[17:26:42] <qchris>	 Let me dig it up.
[17:26:42] <milimetric>	 sorry
[17:26:45] <qchris>	 ok.
[17:51:18] <grrrit-wm>	 (PS4) Nuria: Fix wiki cohort display for report cohorts [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142514 (owner: Milimetric)
[17:53:01] <Ironholds>	 qchris, you're absolutely right
[17:53:06] <Ironholds>	 just conducted a test using the IP ranges we use
[17:53:11] <Ironholds>	 (as 'internal ranges')
[17:53:58] <Ironholds>	 definitely including a loot of requests through terminators; or, at least, the requests have the structure of normal requests and a lot of variation in UA
[17:54:34] <qchris>	 I'd prefer something like "qchris, I found it. It's simple, you overlooked $X which solves the issue completely"
[17:54:44] <qchris>	 Cannot we restart using that?
[17:55:27] <Ironholds>	 hahaha
[17:55:30] <Ironholds>	 if it would make you feel better ;p
[17:55:47] <Ironholds>	 but..hmn. This would suggest we may have been undercounting for quite a while.
[17:55:48] <qchris>	 Yes ... if you can provide a value for $X.
[17:56:12] <qchris>	 Under/Overcounting depends on where/what.
[17:56:14] <Ironholds>	 :D
[17:56:15] <Ironholds>	 yep
[17:56:33] <qchris>	 udp2log for example also has the requests arriving at the ssl terminators.
[17:56:44] <Ironholds>	 yep
[17:56:50] <qchris>	 That saves our neck for large parts of our infrastructure.
[17:56:54] <Ironholds>	 I will further filter against other filters and hope there are 0 left at the end
[17:57:06] <qchris>	 k.
[17:58:52] <Ironholds>	 hmn, okay. there are 6 left at the end. And they are all odd things.
[17:59:23] <Ironholds>	 5 embedded commons requests, and..an API request? Of type text/html?!
[17:59:58] <Ironholds>	 http://commons.m.wikimedia.org/w/api.php?useformat=mobile&r=[redacted]&origin=https://en.m.wikipedia.org
[18:02:41] <Ironholds>	 I should write this up.
[18:47:37] <grrrit-wm>	 (CR) Nuria: "Permits wise code works." (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/143040 (https://bugzilla.wikimedia.org/66087) (owner: Milimetric)
[19:09:01] <grrrit-wm>	 (PS5) Milimetric: Fix wiki cohort display for report cohorts [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142514
[20:28:20] <grrrit-wm>	 (PS6) Milimetric: Fix wiki cohort display for report cohorts [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142514
[20:33:02] <grrrit-wm>	 (CR) Milimetric: "Nuria, I factored out the mess of logic from the html template, sorry you had to deal with that in the first place. That flow is actually" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142514 (owner: Milimetric)
[20:48:14] <grrrit-wm>	 (CR) Milimetric: "I'm just starting out testing, but I think I see the problem. So, the links are ONLY created for WikimetricsBot. There was no Wikimetric" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/143040 (https://bugzilla.wikimedia.org/66087) (owner: Milimetric)
[20:58:46] <grrrit-wm>	 (CR) Milimetric: Add pretty symlink for WikimetricsBot (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/143040 (https://bugzilla.wikimedia.org/66087) (owner: Milimetric)