[00:00:29] <ejegg>	 ok, I'll upload another PS on that tomorrow morning.  Gonna head out for the night.  Thanks for all the feedback awight!
[00:01:41] <awight>	 me too
[00:01:44] <awight>	 see ya!
[00:01:47] <ejegg>	 so long
[16:57:05] <K4-713>	 Jeff_Green: I just realized that I told you "brb" at like... 4pm yesterday and then just evaporated.
[16:57:18] <K4-713>	 Silly me.
[16:59:08] <AndyRussG>	 Time machine!
[16:59:26] <K4-713>	 AndyRussG: Mine only seems to go forward.
[16:59:37] <K4-713>	 ...at a steady rate.
[16:59:56] <AndyRussG>	 no acceleration?
[17:00:06] <K4-713>	 I mean, I can't prove it.
[17:00:25] <K4-713>	 But sometimes I think it's slowing down, and then catching up to where it should have been later.
[17:00:55] <AndyRussG>	 Kinda stop-and-go traffic-like?
[17:01:01] <K4-713>	 Totally.
[17:01:09] <K4-713>	 It only seems to accelerate... while I'm asleep.
[17:01:18] <AndyRussG>	 Mmmm
[17:01:26] <AndyRussG>	 Mine is more like freefall with no air resistence
[17:01:51] <AndyRussG>	 (i.e. always going faster)
[17:02:02] <AndyRussG>	 but still fun :)
[17:02:25] <K4-713>	 Heh. I remember when I was a kid, it seemed like there were a couple years between all the holidays.
[17:02:50] <K4-713>	 And now I'm like "THE LAST ONE WAS LAST WEEK. The hell."
[17:03:14] <K4-713>	 So, you may be on to something.
[17:03:23] <K4-713>	 Maybe aging makes you less time-resistant.
[17:09:43] <K4-713>	 awight: You're going to the brownbag today, right?
[17:09:47] <K4-713>	 O_O
[17:10:03] <K4-713>	 @_@
[17:10:37] <XenoRyet>	 Is there going to be a way for remoties to get in on that?  Looked interesting.
[17:10:41] <K4-713>	 Yes!
[17:10:54] <XenoRyet>	 Cool, I'm totally 'going' then.
[17:11:02] <K4-713>	 There will be a link just before the thing starts.
[17:11:09] <XenoRyet>	 I mean, I'll just be at my desk still... but you know.
[17:11:27] <K4-713>	 And, I'll probably be hovering on some IRC channel to relay questions.
[17:11:37] <K4-713>	 In case the hangout doesn't work.
[17:11:40] <XenoRyet>	 Good times
[17:12:36] <awight>	 K4-713: I'm psyched, just read the Guardian article on Srđa and Otpor!
[17:13:05] <awight>	 "... you don't want to box Mike Tyson, you want to challenge him to chess."
[17:13:14] <K4-713>	 He seems to make all the sense.
[17:13:20] <K4-713>	 :D
[17:13:38] <K4-713>	 And yeah, bring questions.
[17:13:47] <K4-713>	 This is meant to be conversational.
[17:14:51] <K4-713>	 ...why the heck did I agree to review twelve people?
[17:14:52] <awight>	 And I totally appreciate the guerilla approach.  I'm not planning to sit in the back of a paddywagon if I could still be dancing, jump the fence for a second round, or whatever
[17:14:56] <awight>	 haha
[17:15:48] <K4-713>	 He said some stuff last night, about identifying things that are a) hilarious, and b) paint oppressors in to a corner.
[17:16:03] <K4-713>	 Do nothing, and they seem weak. Do anything, and they seem... crazy.
[17:16:14] <awight>	 >_>
[17:16:45] <awight>	 Welll I'm just gonna check up on my travel days... :p
[17:17:16] <K4-713>	 I mean, I really like this idea that being thoroughly ridiculous actually gets you somewhere.
[17:17:20] <K4-713>	 ...can't imagine why.
[17:17:28] <awight>	 :D
[17:23:17] <grrrit-wm>	 (CR) Awight: [C: 2] Fold parseResponseXX calls into processResponse (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209154 (https://phabricator.wikimedia.org/T98063) (owner: Ejegg)
[17:23:41] <grrrit-wm>	 (CR) Awight: "needs rebase." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209408 (https://phabricator.wikimedia.org/T98063) (owner: Ejegg)
[17:23:43] <grrrit-wm>	 (Merged) jenkins-bot: Fold parseResponseXX calls into processResponse [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209154 (https://phabricator.wikimedia.org/T98063) (owner: Ejegg)
[17:23:51] <grrrit-wm>	 (PS7) Awight: Fix comm status and error checking for PaymentStatus [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/206327 (https://phabricator.wikimedia.org/T90504) (owner: Ejegg)
[17:24:52] <awight>	 ejegg: You want https://gerrit.wikimedia.org/r/#/c/206327, or were you considering going back to the set( parse( ... thing?
[17:25:26] <awight>	 Yeah it is spooky to rely on hidden side effects.
[17:25:52] <ejegg>	 awight: in that patch, it's basically a couple of specialized versions of processResponse
[17:26:02] <awight>	 ah nvm that is not a side effect I was looking at
[17:26:07] <ejegg>	 maybe I should just rename them from inspect... to process...
[17:26:15] <awight>	 makes sense
[17:26:23] <ejegg>	 ok, renaming
[17:26:23] <awight>	 and--I like the idea
[17:27:33] <awight>	 K4-713: want to walk me through setting up the orphan slayer?  I've... haven't dont so yet.
[17:27:50] <K4-713>	 Ah, i can't really do that until after lunch.
[17:27:53] <awight>	 O_o /me inspect own grammar
[17:28:03] <awight>	 ok no problem
[17:28:12] <K4-713>	 Have a meeting / need to write the intro for the... brown bag. :p
[17:28:16] <grrrit-wm>	 (PS1) Ejegg: Clarify purpose of addResponseData [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209754
[17:28:18] <awight>	 oof, ok
[17:28:21] <K4-713>	 I'm so on top of everything.
[17:28:24] <awight>	 :)
[17:28:50] <awight>	 I just tried to vi something in this window :(
[17:28:55] <awight>	 Gonna be a long morning
[17:29:42] <K4-713>	 It's been a long series of mornings.
[17:29:47] <K4-713>	 Eep. Late.
[17:29:52] <awight>	 haha bye
[17:30:43] <grrrit-wm>	 (PS8) Ejegg: Fix comm status and error checking for PaymentStatus [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/206327 (https://phabricator.wikimedia.org/T90504)
[17:30:45] <grrrit-wm>	 (PS4) Ejegg: Add limbo queue message before redirect [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/207843 (https://phabricator.wikimedia.org/T90504)
[17:31:17] <grrrit-wm>	 (CR) Awight: [C: 2] "Thanks!" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209754 (owner: Ejegg)
[17:31:50] <grrrit-wm>	 (Merged) jenkins-bot: Clarify purpose of addResponseData [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209754 (owner: Ejegg)
[17:39:37] <grrrit-wm>	 (PS9) Awight: Fix comm status and error checking for PaymentStatus [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/206327 (https://phabricator.wikimedia.org/T90504) (owner: Ejegg)
[17:39:48] <grrrit-wm>	 (CR) Awight: "trifles" (5 comments) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/206327 (https://phabricator.wikimedia.org/T90504) (owner: Ejegg)
[17:39:56] <grrrit-wm>	 (CR) Awight: [C: 2] Fix comm status and error checking for PaymentStatus [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/206327 (https://phabricator.wikimedia.org/T90504) (owner: Ejegg)
[17:40:04] <grrrit-wm>	 (PS5) Awight: Add limbo queue message before redirect [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/207843 (https://phabricator.wikimedia.org/T90504) (owner: Ejegg)
[17:40:18] <grrrit-wm>	 (Merged) jenkins-bot: Fix comm status and error checking for PaymentStatus [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/206327 (https://phabricator.wikimedia.org/T90504) (owner: Ejegg)
[17:43:35] <grrrit-wm>	 (CR) Awight: "Could use a little error checking" (3 comments) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/207843 (https://phabricator.wikimedia.org/T90504) (owner: Ejegg)
[18:05:30] <grrrit-wm>	 (PS1) Ejegg: Use a better default when looking up error by code [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209770
[18:05:32] <grrrit-wm>	 (PS1) Ejegg: Add ResponseProcessingException error to transaction_response [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209771
[18:07:00] <grrrit-wm>	 (CR) Awight: [C: 2] Use a better default when looking up error by code [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209770 (owner: Ejegg)
[18:07:22] <grrrit-wm>	 (Merged) jenkins-bot: Use a better default when looking up error by code [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209770 (owner: Ejegg)
[18:07:42] <grrrit-wm>	 (CR) Awight: [C: 2] Add ResponseProcessingException error to transaction_response [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209771 (owner: Ejegg)
[18:09:06] <grrrit-wm>	 (Merged) jenkins-bot: Add ResponseProcessingException error to transaction_response [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209771 (owner: Ejegg)
[18:10:47] <grrrit-wm>	 (Abandoned) Awight: Clean up an orphan config [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/207739 (owner: Awight)
[18:18:08] <grrrit-wm>	 (PS6) Ejegg: Add limbo queue message before redirect [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/207843 (https://phabricator.wikimedia.org/T90504)
[18:18:40] <grrrit-wm>	 (PS2) Awight: Cleanup orphan rectifier [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/207738
[18:25:17] <grrrit-wm>	 (PS2) Ejegg: Replace setTransactionResult calls in Adyen adapter [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209408 (https://phabricator.wikimedia.org/T98063)
[18:27:07] <grrrit-wm>	 (PS1) Awight: use __DIR__ everywhere [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209778
[18:38:32] <grrrit-wm>	 (PS1) Ejegg: Get rid of more setTransactionResult calls [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209779
[18:40:04] <wikibugs>	 Fundraising Tech Backlog, MediaWiki-Vagrant: Multiwiki settings are not loaded correctly - https://phabricator.wikimedia.org/T98632#1273043 (awight) NEW
[18:41:27] <grrrit-wm>	 (CR) Awight: [C: 2] Add limbo queue message before redirect (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/207843 (https://phabricator.wikimedia.org/T90504) (owner: Ejegg)
[18:43:41] <grrrit-wm>	 (Merged) jenkins-bot: Add limbo queue message before redirect [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/207843 (https://phabricator.wikimedia.org/T90504) (owner: Ejegg)
[18:57:55] <grrrit-wm>	 (PS2) Ejegg: Get rid of more setTransactionResult calls [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209779
[18:57:57] <grrrit-wm>	 (PS3) Ejegg: Replace setTransactionResult calls in Adyen adapter [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209408 (https://phabricator.wikimedia.org/T98063)
[18:57:59] <grrrit-wm>	 (PS1) Ejegg: Finish eradicating setTransactionResult [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209785
[19:12:57] <wikibugs>	 Fundraising Tech Backlog, MediaWiki-Vagrant: Multiwiki settings are not loaded correctly - https://phabricator.wikimedia.org/T98632#1273152 (bd808) You need to use the mwscript wrapper script to get the MultiWiki configuration to load. Something like this should work:  `MW_INSTALL_PATH=/vagrant/mediawiki...
[19:16:42] <grrrit-wm>	 (PS1) Ejegg: Start chipping away at getTransactionAllResults [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209791
[19:23:42] <grrrit-wm>	 (PS2) Ejegg: use __DIR__ everywhere [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209778 (owner: Awight)
[19:24:02] <grrrit-wm>	 (CR) Ejegg: [C: 2] "optimized!" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209778 (owner: Awight)
[19:24:27] <grrrit-wm>	 (Merged) jenkins-bot: use __DIR__ everywhere [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209778 (owner: Awight)
[20:56:40] <K4-713>	 So... did you guys have standup?
[20:56:49] <K4-713>	 I was busy going over in the brownbag.
[20:57:11] <K4-713>	 AndyRussG, ejegg, XenoRyet. :)
[20:57:27] <XenoRyet>	 Yea, we did
[20:57:30] <AndyRussG>	 K4-713: yeah I imagined.... and yes :)
[20:57:36] <K4-713>	 Cool.
[20:57:40] <K4-713>	 Any blockers I should know about?
[20:58:01] <AndyRussG>	 K4-713: I haz none
[20:58:01] <ejegg>	 nope, awight's been showering my patches with attention
[20:58:30] <awight|fud|meet>	 hehe
[20:58:54] <awight>	 It's like chasing a pack of rhoombas
[20:59:58] <K4-713>	 awight: ...suddenly, I have more ideas for this weekend.
[21:00:06] * AndyRussG imagines 6 battling roombas each with a gerrit patch number on its back
[21:00:14] <K4-713>	 I wonder if they avoid eachother.
[21:00:22] <AndyRussG>	 Doubt it
[21:00:24] <AndyRussG>	 Maybe tho
[21:00:25] <AndyRussG>	 does anyone know if Ellery is on fr-tech? https://lists.wikimedia.org/mailman/listinfo/fr-tech doesn't say... anything...
[21:01:01] <K4-713>	 Oh yeah, it's not a "list".
[21:01:10] <K4-713>	 Ehrm, I think he was.
[21:01:13] <AndyRussG>	 a lost list?
[21:01:22] * K4-713 joins email lost
[21:01:26] <AndyRussG>	 Maybe I'll just cc him anyway so he notices
[21:01:37] <K4-713>	 Yeah, I think that's usually a good way to do things anyway.
[21:01:57] <K4-713>	 I, for instance, filter some lists unless somebody sends something to me directly.
[21:02:14] <AndyRussG>	 Yeah makes sense
[21:02:18] <AndyRussG>	 Mmm BTW Željko was telling my how in qa they have different mailing listts for bots and for people
[21:02:27] <K4-713>	 heh
[21:02:42] <AndyRussG>	 I was kinda imaginign that might be fun here too :) or has it been tried and discussed ad infinitum previously?
[21:03:13] <K4-713>	 There's one... that's more cronspammy.
[21:03:34] <K4-713>	 But I don't think everybody gets subjected to those anyway.
[21:04:10] <K4-713>	 It made sense for the alerts we keep on fr-tech to be there, because the same group of humans would have to react to them.
[21:04:36] <AndyRussG>	 Really! Didn't know that... Yeah even the uncronspammy one
[21:04:39] <K4-713>	 ...hopefully not just by trashing them all. :)
[21:04:49] <AndyRussG>	 is a bit spammful...
[21:05:14] <K4-713>	 Yeah, I can see how you might not get a lot of use out of... well, really, any of the alerts that come through there.
[21:05:20] <K4-713>	 It's all downstream from CN.
[21:08:03] <AndyRussG>	 Well, most... now with browser alerts some are CN-related
[21:09:09] <K4-713>	 Heh. That is true.
[21:09:15] <K4-713>	 True and cool.
[21:09:50] <AndyRussG>	 Yeah! Hopefully those will stabilize and we'll get more milage out of them
[21:10:20] <AndyRussG>	 Also hopeing to expand once we start to use more modern and thus less entrenched-reliable  browser stuff, like localstorage
[21:11:16] <AndyRussG>	 Yeah sometimes on fr-tech I have trouble seeing significant human messages because I haven't cleared out the botspam in a day or two.... not a complaint tho, really it's also fine as is :)
[21:11:41] <cwdent>	 at the sparkfun office there were several demented reprogrammed roombas roaming around
[21:11:52] <AndyRussG>	 I have some automatic filters for a lot of it, actually, but somehow they didn't work for everything, I should look into that
[21:12:05] <AndyRussG>	 cwdent: woo! sounds like fun
[21:12:27] <AndyRussG>	 Yeah we bought one about a month ago, I noticed it did come with an EUL
[21:12:29] <awight>	 Talk about a suck n cut!
[21:12:34] <AndyRussG>	 ?
[21:12:43] <K4-713>	 "It certainly does suck!"
[21:12:45] <cwdent>	 it was a scary place at times
[21:12:47] <AndyRussG>	 cwdent: Hi! btw :)
[21:12:52] <awight>	 https://www.youtube.com/watch?v=_b4Z5Kg08PY
[21:12:54] <cwdent>	 howdy!
[21:12:58] <K4-713>	 cwdent: I want... evil roombas.
[21:13:10] <cwdent>	 and yeah nice ww ref
[21:13:10] <K4-713>	 And some cats to ride them in to battle.
[21:13:17] <ejegg>	 with frikkin lasers?
[21:13:38] <K4-713>	 Did you just pivot on Mike Meyers?
[21:13:40] <K4-713>	 Yes.
[21:13:49] <K4-713>	 Woot friday.
[21:13:51] <cwdent>	 https://www.youtube.com/watch?v=QRXAHmMSfBQ
[21:13:59] <cwdent>	 ^ diff casey btw
[21:14:29] <cwdent>	 but i watched that thing tear 2" tool steel like paper
[21:15:07] <K4-713>	 :o
[21:17:12] <AndyRussG>	 hmmmmm
[21:18:09] <AndyRussG>	 damn!
[21:23:27] * AndyRussG finally understands what awight meant re: suck n cut
[21:23:39] <AndyRussG>	 Well the roomba sure does get a lot of cat hair...
[21:23:50] <awight>	 It was not really a fair analogy
[21:24:02] <AndyRussG>	 About infinity % more than I get by not vacuuming
[21:24:13] <AndyRussG>	 The cat is terrified of it, tho
[21:24:23] <AndyRussG>	 No videos of her riding it will be forthcoming, I'm afraid
[21:25:00] <awight>	 My parents' vacuum has a hilarious number of dog bite marks, the household tool they love to hate
[21:25:04] <K4-713>	 I mean, I assmume this is not news. https://www.youtube.com/watch?v=Of2HU3LGdbo
[21:25:55] * AndyRussG shares right away
[21:26:46] <K4-713>	 I can't stop wondering what the hell else was going on in that house that we don't know about, for a confluence of things like that to happen in one place.
[21:27:10] <cwdent>	 anyone read achewood? http://achewood.com/index.php?date=07192004
[21:27:15] <AndyRussG>	 The cat and the duck are surely the sanest members of the family
[21:28:16] <K4-713>	 cwdent: Not the frosty ones!
[21:28:45] <K4-713>	 A One that isn't cold, is hardly a One at all.
[21:28:47] <K4-713>	 ...anybody?
[21:29:01] <cwdent>	 strongbad!
[21:29:22] <K4-713>	 Yeeeeees.
[21:29:45] <cwdent>	 old school
[21:29:50] <cwdent>	 in terms of internet humor
[21:30:08] <cwdent>	 there are a ton of good achewood strips with the roomba http://achewood.com/index.php?date=05212004
[21:30:10] <AndyRussG>	 rrrrg I guess I'll have to start reading that now
[21:38:07] <AndyRussG>	 the bottleneck for querying logs on Hive is something inherent about Hive itself, right? I mean in addition to the fact that it takes them a long time to get ther...
[21:40:42] <awight>	 Hive is insanely slow to query
[21:41:21] <awight>	 and the current query we have for doing banner impression stuff performs like 10 redundant parse_urls on the same column, to extract params
[21:41:28] <cwdent>	 what is hive used for?
[21:41:56] <awight>	 cwdent: collecting our reverse proxy logs and doing funky analytics
[21:42:27] <awight>	 https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hive
[21:43:13] <wikibugs>	 Fundraising Tech Backlog: contribution_source triggers seem to be failing - https://phabricator.wikimedia.org/T98643#1273524 (awight) NEW
[21:43:50] <cwdent>	 gotcha, yeah sounds like the queries would be intense!
[21:43:56] <awight>	 days...
[21:44:11] <awight>	 with researchers pounding on your door telling you to turn your rock music down
[21:44:48] <cwdent>	 a bunch of disparate file formats it has to index?
[21:45:10] <awight>	 I donno about its data sources yet
[21:46:29] <AndyRussG>	 hmm
[21:46:46] <AndyRussG>	 what is the ideal future for real-time querying of unsampled banner logs? is there one?
[21:47:42] <awight>	 AndyRussG: I think we have to write that.
[21:48:01] <awight>	 https://phabricator.wikimedia.org/T90649
[21:49:42] <AndyRussG>	 hmm right I just hadn't thought much about the endgame of that data pipeline...
[21:50:19] <AndyRussG>	 cwdent: yeah fr analytics much enjoyes the full unsampled logs of what banners people have seen
[21:51:36] <AndyRussG>	 awight: just looking at your comment there... so kafkatee is indeed now able to cough up log entries just like what we get from udp2log
[21:51:58] <awight>	 K4-713: Remind me, can we kill the log parsing code in the orphan slayer?
[21:51:58] <AndyRussG>	 but now you're talking about including parsing and aggregation in the kafka process itself?
[21:52:47] <awight>	 AndyRussG: great!  Mmm, I don't know anything about kafka, I was imagining a cronjob or something.
[21:52:48] <AndyRussG>	 and that way the size of the data will be reasonable nuff to shovel into a RDBMS?
[21:52:55] <awight>	 I think that part's true.
[21:53:01] <awight>	 We'll have to do some aggregation.
[21:53:16] <AndyRussG>	 sure, but where?
[21:53:47] <awight>	 It depends on our goal.  IMO we should be aiming for public access to banner impression #s
[21:54:04] <awight>	 and we should be reusing as much existing analytics infrastructure as possible
[21:54:12] <awight>	 Ideally, there's nothing for us to maintain.
[21:54:49] <awight>	 I think my "public access" goal should be tempered by meganhernandez though, it's her job which will become more difficult if that happens
[21:54:51] <AndyRussG>	 and looking at it from Ellery's (cwdent: XenoRyet: he's FR analytics) perspective... so he'll want to be able to real-time query full unsampled? I guess we need to know what queries he'l do
[21:55:05] <awight>	 yep
[21:55:23] <awight>	 probably not exactly full unsampled, I think the per-minute aggregates are fine for him
[21:55:33] <awight>	 maybe even 5 min...
[21:55:42] <AndyRussG>	 hmmm
[21:55:57] <awight>	 We should have an estimate for how much data this is...
[21:56:07] <AndyRussG>	 I think he eventually wants to get his hands on unsampled, but I don't know if it's useful in realtime
[21:56:28] <awight>	 Also, it would be nice to archive stuff to another db at a year out or something.
[21:57:27] <AndyRussG>	 yeah I was thinking that... currently DjangoBannerStats (cwdent: XenoRyet: that's the python db system that slurps up 1:100 sampled banner impressions) is _never_ rotaed/cleaned out/ or something?
[21:57:39] * AndyRussG should probably know this
[21:58:07] <awight>	 AFAIK, true
[21:58:54] <awight>	 We should rename DjangoBannerStats "◎DjangoBannerStats"...  it's days are numbered.
[21:58:57] <awight>	 its
[21:59:13] <XenoRyet>	 AndyRussG: Thanks for the annotations.  Helpful for following the conversation.
[21:59:35] <mutante>	 hi, we got a notification from monitoring again
[21:59:40] <mutante>	 disk space on lutetium is low
[21:59:52] <mutante>	 ccogdil or pcoombe around?
[22:00:17] <AndyRussG>	 mmrmmmrrhrhrhg
[22:00:17] <AndyRussG>	 I guess that's considered part of the same Phab card you mentioned above?
[22:00:31] <awight>	 mutante: Thanks!  We were hoping you could give us a listing of home dir usage
[22:00:46] <awight>	 K4-713 was mentioning that this has happened before, and it's always a rogue home dir
[22:00:52] <mutante>	 1.2G	awight
[22:00:52] <mutante>	 5.1G	ccogdill
[22:00:52] <mutante>	 6.4G	pcoombe
[22:00:58] <awight>	 mutante: hold on!
[22:00:59] <mutante>	 the others under 1G
[22:01:01] <awight>	 thx
[22:01:15] <AndyRussG>	 XenoRyet: np! :)
[22:01:44] <mutante>	 awight: there are about 2.5G left on /  .. BUT > 350G on /srv
[22:02:13] <awight>	 mutante: ok, I wiped my homedir
[22:02:26] <mutante>	 awight: there is that /dev/mapper/tank-data
[22:02:32] <mutante>	 so unless it's a permission problem
[22:02:37] <mutante>	 you could just move stuff there
[22:02:47] <awight>	 ccogdill is on vacation, and pcoombe is nick'ed the-wub, but he's on England time
[22:02:56] <awight>	 mutante: cool, I'll pass that along
[22:03:17] <mutante>	 or maybe the entire /home could be moved there
[22:03:23] <mutante>	 but i would ping Jeff about that
[22:04:03] <mutante>	 awight: should i check if we can just zip some files in their home dirs while they are gone?
[22:04:20] <mutante>	 often just gzip will do a lot
[22:04:24] <awight>	 mutante: did the 1G freed help enough to get us through the weekend?
[22:04:42] <awight>	 They have some automated processes in there which email and upload stuff to people and services... things we depend on...
[22:04:50] <mutante>	 awight: i think so but i dont know much about what is running on lutetium
[22:05:10] <awight>	 It's generally our testing and scratch db box
[22:05:14] <mutante>	 awight: partly it's just the settings of monitoring too, there is a certain percentage of free space it wants
[22:05:50] <mutante>	 that could be changed to
[22:05:55] <awight>	 Well, I don't think their scripts will be able to catch up with my deleting, at least not for a few months.
[22:06:12] <awight>	 I'm emailing now...
[22:06:44] <mutante>	 i guess the best would be to move the entire /home out of the root filesystem
[22:06:50] <mutante>	 as a separate mount
[22:06:59] <mutante>	 then it can run out but not influence running services
[22:07:54] <awight>	 That works for me, we'll see what JGreen se
[22:07:56] <awight>	 sez
[22:11:59] <mutante>	 i'm gzipping some large files that were in directories called "old"
[22:12:06] <mutante>	 but not deleting anything
[22:12:09] <awight>	 sounds safe
[22:12:22] <awight>	 pls email with the commands run, if you don't mind
[22:12:55] <mutante>	 email to?
[22:15:19] <awight>	 (PM'd)
[22:16:46] <awight>	 mutante: Thanks for catching this before it became a huge weekend project!
[22:19:39] <awight>	 ejegg: Are you still able to use GlobalCollect locally?
[22:19:51] <awight>	 I just noticed the iframe is silently dying
[22:20:05] <ejegg>	 ooh, lemme see
[22:20:58] <mutante>	 15:20 < icinga-wm> RECOVERY - check_disk on lutetium is OK: DISK OK - free space: / 9191 MB
[22:20:59] <awight>	 Yep, I rolled back to 3dc72ba009707c53161251fa96f6e985198eec9d and it works fine
[22:21:01] <mutante>	 there you go
[22:21:03] <mutante>	 awight:
[22:21:08] <awight>	 whew!
[22:21:19] <mutante>	 i catch it because a bot tells us in the ops channel, so that worked:)_
[22:21:28] <awight>	 ;)
[22:21:38] <ejegg>	 oh boy, I broke it!
[22:22:12] <awight>	 sorry!
[22:22:30] <ejegg>	 heh, thanks for catching it now
[22:22:49] <awight>	 I'm sure deployment would have surfaced it as well
[22:23:26] <ejegg>	 explosively...
[22:23:35] <AndyRussG>	 ejegg: that's the best way! :)
[22:23:52] <AndyRussG>	 hmmm actually pretty straightforward: http://kafka.apache.org/documentation.html#introduction
[22:30:43] <grrrit-wm>	 (PS1) Awight: Remove cruft [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209816
[22:34:14] <grrrit-wm>	 (PS13) Awight: Use DonationQueue for limbo queuing [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/206312 (https://phabricator.wikimedia.org/T92922)
[22:34:25] <grrrit-wm>	 (CR) Awight: "PS 13: rebase" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/206312 (https://phabricator.wikimedia.org/T92922) (owner: Awight)
[22:35:16] * AndyRussG waves
[22:35:32] <AndyRussG>	 See everyone soon on the netz
[22:37:52] <ejegg>	 l8r AndyRussG
[22:40:01] <awight>	 see ya!
[22:52:28] <grrrit-wm>	 (PS1) Ejegg: Don't set error api result on empty array [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209819
[22:53:18] <ejegg>	 awight: whoops, returning empty error array!
[22:55:13] <awight>	 ejegg: aha!
[22:55:31] <awight>	 weird, I thought that was one of isset's conditions
[22:55:41] <awight>	 nope, just is not null...
[22:55:47] <ejegg>	 perzackly
[22:55:56] <grrrit-wm>	 (CR) Awight: [C: 2] Don't set error api result on empty array [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209819 (owner: Ejegg)
[22:55:58] <awight>	 Thanks!
[22:56:06] <ejegg>	 and empty includes isset
[22:56:15] <awight>	 I do wish we had a test for this...
[22:56:20] <grrrit-wm>	 (Merged) jenkins-bot: Don't set error api result on empty array [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209819 (owner: Ejegg)
[22:56:21] <awight>	 Maybe after Lyon!
[22:56:32] <ejegg>	 yeah, needs fakebank
[22:57:13] <ejegg>	 gee, somebody out there must have coded something to look like a banking website :)
[22:57:34] <ejegg>	 guessing the phishers don't mock the APIs
[23:10:26] <grrrit-wm>	 (PS1) Ejegg: Use RetryLoopCount in base do_transaction [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209822
[23:17:53] <grrrit-wm>	 (CR) Awight: [C: 2] "An improvement! Now I'm wondering what really happens in transactionConfirm_CreditCard, however. I think we're repeating the inner loop " [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209822 (owner: Ejegg)
[23:18:17] <grrrit-wm>	 (Merged) jenkins-bot: Use RetryLoopCount in base do_transaction [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/209822 (owner: Ejegg)
[23:25:22] <grrrit-wm>	 (PS14) Awight: Use DonationQueue for limbo queuing [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/206312 (https://phabricator.wikimedia.org/T92922)
[23:25:36] <mutante>	 hi, is cvirtue here?
[23:25:48] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] Use DonationQueue for limbo queuing [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/206312 (https://phabricator.wikimedia.org/T92922) (owner: Awight)
[23:26:00] <awight>	 mutante: Try gtalk
[23:31:47] <grrrit-wm>	 (CR) Awight: Use DonationQueue for limbo queuing (2 comments) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/206312 (https://phabricator.wikimedia.org/T92922) (owner: Awight)
[23:33:16] <grrrit-wm>	 (PS15) Awight: Use DonationQueue for limbo queuing [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/206312 (https://phabricator.wikimedia.org/T92922)
[23:34:18] <grrrit-wm>	 (CR) Awight: "Note: This does not completely mirror the orphan slayer implementation, we never pop messages out to rectify them." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/206312 (https://phabricator.wikimedia.org/T92922) (owner: Awight)
[23:35:20] <awight>	 ejegg: that one should be ready to examine again, ^^
[23:35:54] <ejegg>	 woohoo!
[23:36:01] <awight>	 I didn't manage to fully port the orphan slayer, I need to write a weasely popMultiple for Memcache.
[23:36:09] <awight>	 using getStats or something...
[23:36:29] <ejegg>	 heh, it is pretty annoying to see what it actually holds
[23:36:37] <awight>	 K4-713: this is scary,
[23:36:37] <awight>	 Found an orphan! globalcollect-5974128769
[23:36:37] <awight>	 False Orphan! globalcollect-5974128769
[23:36:44] <awight>	 yeah it is
[23:36:51] <K4-713>	 Why is that scary?
[23:37:00] <awight>	 how is it an orphan and not an orphan?
[23:37:01] <K4-713>	 Stuck message?
[23:37:07] <K4-713>	 Er...
[23:37:10] <awight>	 I retried the page a bunch
[23:37:19] <K4-713>	 Oh, that would do it too, I think.
[23:37:40] <K4-713>	 It's been a long time, though.
[23:37:44] * awight stares some more at the orphan slaying obelisk and beats the ground with a femur
[23:38:09] * K4-713 starts swaying lightly to the Blue Danube
[23:38:44] <K4-713>	 Just kidding. It's the KLF.
[23:51:29] <grrrit-wm>	 (CR) Ejegg: [C: 2] "Very nice! That memcache popMultiple is sure to be an interesting challenge" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/206312 (https://phabricator.wikimedia.org/T92922) (owner: Awight)
[23:51:53] <grrrit-wm>	 (Merged) jenkins-bot: Use DonationQueue for limbo queuing [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/206312 (https://phabricator.wikimedia.org/T92922) (owner: Awight)
[23:52:49] <awight>	 Crud, it's merged.  Now I have to lie in the mess I made.
[23:52:55] <ejegg>	 hah