[00:00:16] ccogdill: thank-yous are going out again [00:00:34] awight: thanks again so much for helping mark the contributions [00:00:35] okay cool, thanks awight [00:00:40] & ejegg! [00:00:53] ejegg: that was the easy part, you're the one digging the hole and filling it in again! [00:00:59] oof [00:01:14] next time, i'll try not to dig it in the first place [00:01:31] I just sit around and experiment with smileys ;# [00:03:17] ejegg: ouch. I think we have to keep the civimail switch off. [00:03:27] It's taking 1-2 seconds per TY sent [00:03:50] Normally about 6 per second [00:04:50] argh! And we just got hit by the same bug as a month ago, PHPMailer melts down, then we mark the contribution as TY failed. [00:06:00] oh man [00:06:35] so, PHPMailer melts down under too /little/ load? [00:06:46] I don't quite get that part. [00:07:25] Maybe my civimail stuff is devouring some resource? [00:08:26] Here are the things I found, last time: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/2145 [00:08:31] and https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/2146 [00:09:25] good observations [00:09:26] !log disabled thank-you activity records [00:09:30] Logged the message, Master [00:10:15] ejegg: I hope this isn't surfacing actual high-load fail... [00:10:43] Yeah like I said in the card, usually we only deliver 100 TYs [00:10:56] but if there's a backlog, we deliver many more, and then phpmailer dies [00:11:17] However, we need to survive sending the maximum... [00:12:58] ejegg: ok, I unset the 'failed' records... [00:13:58] ahhh, so backlog size has something to do with it [00:14:53] Well, I just think that the backlog causes the batch size to go up [00:15:07] Which scares me cos we'll have actual high load, soon enuf [00:15:11] oh right [00:15:54] has it done that without the civimail records being created? [00:16:14] It must have... [00:16:33] Yeah, cos turning TY off for a bit is all it takes to develop a backlog... [00:17:05] ahh, odd that it didn't come up last year then [00:17:06] ejegg: yeah, look at job 139192 for example [00:17:15] This doesn't sound... good. [00:17:16] that was 700 mails, and much faster [00:17:30] K4-713: well I'm less worried than I was about 20 seconds ago [00:17:41] Nice trend. [00:17:46] We should try to maintain that. [00:17:47] For a minute I thought there was a glass ceiling on how many TYs we could send [00:17:51] yow. [00:18:44] For the record, we average 4.9 ty [00:18:58] 4.9 ty sent per second, when churning through a batch of 700 [00:19:17] Did you look at the highest load from last year? [00:19:27] I think ty starts lagging behind. [00:19:53] Our highest donation rate is 2 donations / second... [00:20:14] Aggregated how big? [00:20:18] I bet TY does lag behind, though, cos of Jenkins inefficiency [00:20:23] over an hour [00:20:27] Ah. [00:20:35] this is just from our 10,000 donations in an hour record [00:20:52] I think there are some weirdly magical 15 minute periods you can see, too. [00:21:24] We should use duty cycle job management for TY... it's currently trying to overlap jobs. [00:21:24] But, yeah... we pause qc for any amount of time once we get going, and ty gets railed. [00:21:31] ...when we turn it back on. [00:21:35] exactly [00:21:48] also, if we pause TY. [00:21:51] Any idea when that got damaged? [00:22:03] wait, nothing's damaged currently [00:22:08] Using the queue as a buffer like that is kind of a design requirement. [00:23:01] Oh, also: Depending on how many job runners jenkins has, it may have to do a complicated dance with, say... a nightly audit. [00:23:31] But, we can't just add more runners, because everything bogs down. [00:23:39] Reminder to self: TY is not thread-safe, we can't add a runner [00:23:59] K4-713: back to "got damaged"--what u referring to there? [00:24:03] When you say it's trying to overlap jobs, though... [00:24:09] ah sorry [00:24:21] only that it's taking > 2 minutes, which its cron interval [00:24:27] so we should dial down the batch size [00:24:27] awight: I was probably misunderstanding the concern I was seeing in the backscroll here. [00:24:42] Ah, okay. Yeah, it won't do that on its own. [00:24:46] good call, back to 100? [00:24:56] it's not actually concurrently running--that would be really bad cos of the way it pulls the entire batch at the start of the run [00:25:02] We're probably going to have to spend day 1 tuning the jobs to some extent. [00:25:09] That's... completely normal. :) [00:25:25] * K4-713 crawls in to ganglia completely [00:25:54] I probably did that before and just didn't talk about it. [00:26:22] ejegg: at our current speeds, we could send 590 TYs in 2 minutes. So we should set the batch size to 75% of that, IMO [00:26:42] It's going to slow down across all the jobs when we turn this thing on. [00:26:46] ...for sure. [00:26:50] But going forward, I want to reuse our duty cycle logic, so the TY job just runs for 1'30", then quits regardless of count. [00:26:57] K4-713: yep [00:27:30] When we decide these duty cycles, we're going to have to take in to account that we want a gap. [00:27:35] !log set TY batch size=400 [00:27:36] ...a small gap, but a gap. [00:27:37] yes exactly [00:27:39] Logged the message, Master [00:27:53] I mean, I want to put a clock on everything. [00:27:55] It's unfortunate that we have two knobs for each job [00:28:11] ejegg: cool, thx [00:28:15] I'm gonna be distracted by dinner for the next little while [00:28:26] awight: LET ME TURN MY KNOBS. :) [00:28:30] ejegg: relax!!! [00:29:06] K4-713: nsfw 8D [00:30:10] aw. [00:30:13] * K4-713 goes home [00:30:41] * awight continues pretending to work [00:31:16] AndyRussG: have you asked jalexander or pbeaudette for Meta CN privs yet? [00:31:51] I thought I sent that email. [00:31:52] They will jump on that ridiculously quickly. [00:31:56] Did that not happen? [00:33:07] nope, https://meta.wikimedia.org/w/index.php?title=Special:ListUsers&group=centralnoticeadmin [00:33:23] Yeah AndyRussG needs that, yesterday! [00:33:23] Sent initially on November 18, actually. [00:33:27] Just bumped. [00:33:30] thx [00:33:32] awight: K4-713: it didn't happen, but I'm pretty sure I did see that e-mail fly by [00:34:04] AndyRussG: tell them him you need elevated privileges, to disable the fullscreen banner 8p [00:34:42] awight: wouldn't I need suprerprotect rights for that? [00:34:57] I'm telling. [00:35:19] hehe you should be good hitting Submit from the plane [00:35:44] * AndyRussG whistles pretending he han't seen wikimedia-l today [00:35:56] we're all going to need superprotect pretty soon [00:36:43] AndyRussG: Try meta again. :) [00:37:45] K4-713: cool! [00:37:49] * AndyRussG hugs Echo notifications [00:38:00] that's the kind of response time I would have expected [00:38:24] thanks! [00:38:25] I think jalexander has a Bureaucrat window open all day :) [00:38:26] Now, what was that banner that you wanted me to enable everywhere? [00:38:39] /m just send it to my paypal account [00:39:02] ^L :) [00:39:28] _now_ I'm a target worth hacking [00:39:35] yep! [00:41:04] your sneezes can be felt 'round the world, now [00:41:54] Hmmm actually that was two weeks ago when... I borked mobile... [00:42:44] Still it's nice to know that my sneezing strength continues to reach untold heights [00:42:57] * K4-713 hands AndyRussG a tissue [00:43:09] huh just one? :) [00:43:42] You no longer have to use the back door, to bork mobile :) [00:43:44] You can probably get more later. [00:45:13] Hmmmmmm :) [00:58:06] ejegg|dinnr: AndyRussG: small thing--you might want to add your days in SF to the Fundraising Schedule google calendar... [00:58:25] awight: Ah cool good idea :) [00:59:32] I should arrive in the office around 1:30 or 2 pm on Monday [00:59:41] wooooooot. [00:59:42] AndyRussG: great! take the day off :) [01:00:20] nmmrmrrrrmmmnnnnnn I dunno, prob'ly won't, shouldn't [01:00:35] <_< [01:00:36] >_> [01:00:47] * awight calls the union [01:00:53] It *is* kind of funny that you're not going to be online until later Monday. [01:01:10] Hummm [01:01:14] We start at 8 am? [01:01:18] We'll just be over here, overloading the donation pipeline as much as possible. [01:01:27] Ah, 6am our time. Probably. [01:01:29] That might move. [01:01:31] I'll invent some challenging new banners [01:01:50] CentralNotice deploy at 10 am? [01:01:50] I'm going to [wake up early|not sleep] in order to be up and monitoring. [01:02:00] ARGH. :p [01:02:03] I wonder if I can get wifi on the flight [01:02:26] Deploy from phone at possible. [01:02:47] yeah I did ask for the Monday direct flight instead of the Sunday all-day stop-over-ful ones [01:03:01] yuck! [01:03:24] Travel folks said that the direct flight on Sunday was very expensive, twice as expensive as the one on Monday [01:03:55] hehe there's nothing more comforting than an automated email from PAyPal saying "Postponed: Nov 21 Emergency Maintenance" [01:04:03] well ok then, nothing to see here [01:04:58] Stupid Thanksgiving being Stupid Late and Stupid Ruining Everything. [01:05:14] ...I'm done now. [01:05:27] Tantrum over. [01:08:05] awight: Did everything just suddenly stop again? [01:08:18] what. [01:08:23] And by "everything" I mean "all the bad things". [01:08:35] Um, I think we're running normally [01:08:39] wth [01:08:50] but I'm currently clearing all the failmail out of my inbox so I can see new ones [01:08:55] * K4-713 tunnels into jenkins to make sure [01:09:19] I didn't just... add an email filter in the night that dumps all the failmail in the trash, did I? [01:09:20] I know a few bad things that haven't stopped [01:09:27] Oh good. [01:09:31] I was worried there for a minute. [01:11:33] goodbye, trouble! [01:12:24] This calm doesn't line up with my worldview! [01:12:30] :) [01:12:54] I've got a few exciting things queued up, I'm just waiting for everyone to go home and think they're relaxing. [01:13:08] then i will snip the thread :) [01:15:25] I love it when the Red Alert noise comes out of my pocket super loud when I'm in public. [01:15:31] It's one of my favorite things. [01:15:48] shields loud! [01:16:25] My goal is to get a stranger to shout "SHUT OFF THAT DAMN NOISE" at me. [01:16:45] She can't take any more, cap'n! [01:16:50] One time, it went off in an airport. [01:16:53] In Italy. [01:17:01] I thought for sure I was about to be... detained. [01:17:02] We an't use the warp engines with the shields up! [01:17:05] yeah seriously [01:17:37] The funniest thing in the world, to me, are the signs that say "Rules...... 10. no joking about these rules" [01:18:28] awight: K4-713: ejegg|dinnr: confirmed Ellery's bug in impression data on lutetium [01:19:20] AndyRussG: would u mind explaining a thing for me... [01:19:33] awight: sure! [01:19:39] The original bug report looked odd, cos it seemed to be as designed. [01:19:54] ohhh sorry, I'm catching up. [01:20:05] Which is the original bug report? I saw Ellery's long e-mail with tables and graphs [01:20:06] The bannerimpressions table was not intended to be used to count hide results. [01:20:15] I was thinking it was. [01:21:41] Well we could re-porpoise it [01:21:41] AndyRussG: atgo created this card out of Ellery's email: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/2196 [01:22:14] AndyRussG: yeah, I think it makes sense to keep the data. We just have to adjust all the scripts that read from that table. [01:22:22] You're way ahead of me! [01:22:56] uhh vice versa, I was joking, but actually that's a good idea [01:23:44] tl;dr w/ the current script, this URL gets counted as an impression: http://meta.m.wikimedia.org/wiki/Special:RecordImpression?country=US&uselang=en&project=wikipedia&db=enwiki&bucket=0&anonymous=true&device=ipad&reason=empty&result=hide&campaign=foo&banner=foo [01:23:54] as if the banner had been shown [01:24:03] Mmm what shall I do? [01:24:56] Also I think have access so I could in theory fix and deploy while you folks are on holiday [01:24:59] donno, we either add a result column and fix ecom + statler, or we add a second table meh [01:25:46] AndyRussG: I'm with you. There's nothing to be thankful about, that some locals and the European immigrants allied against another local tribe... [01:26:34] what are the scripts that pull results out of there? yeah I know there's a gap in my knowledge of the data tube between these tables and Ellery's number crunching [01:26:46] I think just Ellery's tools, ecom, and statler. [01:27:05] And of course, none of these are in source control 8~l [01:27:06] awight: agreed! also it was Thanksgiving last month here... [01:27:31] eh? [01:27:39] yeah... [01:28:06] We've been careful to keep any really essential pieces of that infrastructure from being maintainable [01:28:13] hmmm [01:28:40] Maybe I could fix the script so it gives the expected resuts and as a separate patch count empties in a different table? [01:28:52] Since ecom is a bit of a ghost ship, I could go ahead and check that in unilaterally... [01:29:05] Then we'd get more data and still be able to keep the scripts? [01:29:05] I'm sure the-wub would be happy to check in statler... [01:29:28] There's that statler thing again. [01:29:30] AndyRussG: agreed. Although it means more monkeywrenching in the django scripts. [01:29:53] Ah I'm already there... [01:29:59] Looks easy enuf [01:30:17] I know nothing about re-processing data [01:30:23] tho [01:31:13] This would have been happening since 4c5b17f329b44ca26c95efc37f5c4e5eee91b4ac was deployed [01:31:56] dunno the exact date, the sample data I have from Oct. 23 was not yet affected [01:33:46] (Abandoned) Awight: Confirmed no more usage of toggleBanner [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/173661 (owner: Awight) [01:39:26] kbye! [01:42:33] K4-713: should I try to deploy a fix for this myself? I assume everyone else is taking a long weekend... [01:43:01] AndyRussG: Theoretically, we are going to have a long weekend... [01:43:40] yeah I hear... hmmm tho theoretically I think I don't get those... [01:43:58] ...but in reality, I'll hop back on at some point. [01:44:12] ah OK! [01:44:18] How about you shoot off an email to fr-online to ask if this is necessary. [01:44:49] OK [01:44:55] Almost fixed in fact [01:45:00] orly [01:45:01] * AndyRussG fiddle with python indentation [01:46:21] I'm definitely going to come back online later tonight to see if there's anything that needs review, at least. [01:50:07] (PS1) AndyRussG: Allow campaign and banner params on hide URLs [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/176214 [01:51:03] K4-713: ^ can never be too sure about this stuff, I know things are fragile in any case [01:51:48] thanks also! [01:51:49] back in a little! I'll get backscroll tho [02:06:39] actually ignore that silly patch [02:06:42] pls [02:08:45] AndyRussG: Ugh. My python eyeballs are not functioning correctly. [02:08:57] ...largely due to them being my regular eyeballs. [02:09:06] K4-713: Mine weren't either, silly wrongness in there [02:09:13] Oh. [02:09:26] * K4-713 squints in a determined way [02:10:53] I think maybe we should all just... stop. For a bit. [02:11:03] I am about to start doing silly things. [02:11:10] I should probably do those away from code. [02:11:33] (PS2) AndyRussG: Allow campaign and banner params on hide URLs [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/176214 [02:12:04] That should be better [02:12:26] You just... unnested. Right? [02:12:28] That's it? [02:12:58] grumble grumble whitespace python grumble [02:13:29] Maybe my python eyeballs were okay after all. [02:14:04] K4-713: this time, yes [02:14:12] It will probably be the only time. :) [02:14:24] git difftool HEAD^1 HEAD -t kdiff3 [02:14:28] (CR) Katie Horn: [C: 2] Allow campaign and banner params on hide URLs [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/176214 (owner: AndyRussG) [02:14:43] ...oh. No jenkins in there. [02:15:13] (CR) Katie Horn: [V: 2] Allow campaign and banner params on hide URLs [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/176214 (owner: AndyRussG) [02:15:21] Allrighty then. [02:15:34] K4-713: thanks! [02:15:49] Yep! [02:15:58] Should I try to deploy it or something one of these days? [02:16:08] AndyRussG: So, when this is out, we're... out from under all the major boulders, right? [02:16:15] That we know of? [02:16:57] I think so... I mean, I haven't followed much of the GlobalPay vs. WorldCollect saga [02:17:28] Thought I did see a few boulders being removed there... [02:17:40] AndyRussG: You just reminded me of a discussion I had a while ago, about... writing a tool that would generate payment processor names. [02:18:03] Maybe an hour for years of enjoyment. [02:18:35] Ah like UniversalCharge and WorldwideSendCash? [02:18:52] fun [02:18:58] GalaxyCoinPro [02:19:00] er [02:19:15] Hmmm [02:19:39] GlobalCashCollect [02:19:45] I do wonder what the "if "BannerController" in l:" line is about [02:20:12] Nice to know that "unnested" is a word [02:20:26] I mean, it is now. [02:20:37] When the pidgeons don't come home to unnest? [02:20:39] Legit because I didn't have to explain it. [02:20:46] :) [02:20:46] good point [02:21:13] Ah. [02:21:29] To answer your actual question, somebody will deploy this before Monday. [02:21:55] I'm not sure who that will be at the moment, but I think there's a 50/50 chance it's going to be me. [02:22:07] I'd like somebody looking at the impression data while we do it, though. [02:22:12] That's the blocker. [02:23:15] K4-713: got it, makes sense [02:23:46] Yeah this hasn't been tried on the real data that it's supposed to fix, I only have test data from before the change to CN that broke this was deployed [02:27:07] I can try to figure out what the other condition in there is about ("BannerController" in l) to see if maybe that needs changing [02:27:32] Yeah, sounds like a 2-person deploy to me. One to push the buttons, and one to spot. [02:27:52] Preferably the bug reporter, but I don't know what his availability is going to be this weekend. [02:28:04] hum [02:28:50] if his technique for spotting the issue is comparing hive and leutium results, I think there'll be some delay in fetching from hive, no? [02:29:07] I dunno how long or automatatized his other steps are [02:29:35] If you like I can reply on the thread, with a link to the patch, in any case... [02:39:42] AndyRussG: Yeah, that's probably best. [02:39:44] atgo: ! [02:39:49] HI! [02:39:53] HELLO. [02:39:55] oh man. the weather. [02:40:07] K4-713: K cool [02:40:13] atgo: miss the desert? [02:40:29] Can you tell me, how uncomfortable it's going to be, if we don't get the banner impression hide thingy changes in for Monday? [02:40:34] Is the answer "all the painful"? [02:40:42] Or, perhaps "12"? [02:41:38] atgo: The thing is, I think we need ellery to tell us if we've gotten it or not, and I have no idea what his availability is this weekend. [02:41:45] oh man. [02:41:45] And he should be eyeballing when we deploy. [02:41:54] hm [02:41:58] well that's a good meganhernandez question [02:41:58] Yep. [02:42:15] * K4-713 stares at meganhernandez [02:42:29] I literally was. [02:44:07] 0_________________0 [02:44:19] hahaha [02:44:31] I wish I were more certain of the patch. Such a mishmash of code at work there... What I'm also not sure of is whether it has the desired effect on reason=alterImpressionData cases, maybe looking more at Meta-based banner code is in order... [02:44:45] Too late. [02:45:02] If it was ever bandaid time, it's now. [02:46:17] Hmmrmrmrgg... [02:46:39] AndyRussG: I know. But this is absolutely a rabbit hole. [02:46:46] In the "old" and supposedly working impression logs, you have URLs like this: http://meta.m.wikimedia.org/wiki/Special:RecordImpression?country=US&uselang=en&project=wikipedia&db=enwiki&bucket=0&anonymous=true&device=ipad&reason=empty&result=hide [02:46:47] If there ever was one. [02:46:57] i.e. the standard hide [02:47:12] And URLs like this: [02:47:12] http://meta.wikimedia.org/wiki/Special:RecordImpression?country=US&uselang=en&project=wikipedia&db=enwiki&bucket=0&anonymous=true&device=desktop&banner=B14_1020_enUS_gg_y&campaign=C14_enUS_dsk_lw_FR&reason=alterImpressionData&result=hide [02:47:49] which were reason=alterImpressionData, result=hide, but unlike the first kind, had already campaign and banner params [02:48:44] The DjangoBanner script was doing different things with each kind [02:48:49] I think 8p [02:49:23] But that may also have been a bug [02:50:19] Now that CN started added campaign and banner params for result=hide URLs in general.... both those and alterImpressionData were getting counted as impressions... uhhh, I think 8p [02:50:27] Makes sense to me. [02:51:18] With the above patch, neither is counted as an impression, which may in fact be correct and may be fixing two bugs... (or not? not, if there was only one to begin with) [02:52:12] So I think the right thing to do is to go back to the JS that's creating this data and try to be clear about what is meant with each case, and that the desired info is being sent out to express the desired meaning [02:52:27] Okay, so... just talked to meganhernandez. Let's deploy it and be ready to roll it back if we need to. [02:52:45] heh [02:52:48] OK :) [02:52:53] We don't want to count them as impressions, but we do want to be able to pull the hides. [02:53:34] K4-713: Pull as in count? [02:53:37] hey ejegg|dinnr i'm marking LYBUNT as deployed [02:53:50] AndyRussG: Yeah, counting hides. [02:54:02] With the hide reason. [02:54:13] K4-713: K [02:54:28] IIRC that's not really counted properly in tables w/ this script [02:54:30] Just logs [02:54:45] But I may be remembering wrong [02:54:46] ly [02:54:47] hum [02:54:50] wrongly [02:55:40] Ever get the feeling that you have no idea what the heck is going on? [02:55:49] Quoting awight (^ today's backscroll): "The bannerimpressions table was not intended to be used to count hide results"... "I think it makes sense to keep the data. We just have to adjust all the scripts that read from that table." [02:56:50] My brain doesn't understand what's happening in the outside world... and the outside world doesn't understand my brain... [02:56:54] K4-713: often. [02:57:11] yeah that's why I wanted to go to banner Javascript [02:57:14] Wanna subscribe to my newsletter? [02:57:26] It'll just say "I'm sorry: What?" and that's it. [02:57:28] I think also awight made some doc, come to think of it [02:57:28] yes. sign me up 12 times [02:57:35] atgo: Done. [02:57:38] :p [02:57:51] K4-713: is it copyleft? [02:58:01] Sure. [02:58:27] K sounds good :) [02:59:10] AndyRussG: We should probably just email fr-tech with the current status of things and all our questions, and stop for today. [02:59:18] yep agreed [02:59:45] AndyRussG: Can you do that, and reference the card number? [02:59:53] Yep! [02:59:59] Thank you. [03:00:03] np :) [03:00:06] hey y'all [03:00:10] * K4-713 readjusts python eyeballs [03:00:16] Hey ejegg|dinnr. [03:00:43] How are you doing? [03:00:46] ejegg|dinnr: are you more ejegg, or more dinnr? or both equally? [03:00:47] So, that crazy stuff you were planning to do after work... might as well go ahead and do it. [03:00:59] hurf [03:00:59] 5 letters worth of each [03:01:06] that's what i feared [03:01:26] I've been patiently trying to back out of doing more things for several hours now. [03:01:31] I'm not being subtle, either. [03:01:32] proportional spacing can ruin a young mind [03:01:42] Uhm. [03:01:55] I'm still at the office, and I need to acquire some cheese for tomorrow. [03:01:56] OK, I'll see how fast I can do this [03:01:59] "ejegg" is much wider than "dinnr" in this font [03:02:00] This is a serious issue. [03:02:35] "this" is still "dash", right? or is there something left over from CRM? [03:02:46] how much cheese? worst case, your only option is theft [03:03:03] I also need brussel sprouts. [03:03:07] And an avocado. [03:03:24] I have a surplus of brussel sprouts, but no wormhole [03:03:27] I don't really want to mug three people, and I don't think I'm lucky enough to find an all-in-one. [03:03:49] Also, it's sort of hard to tell if somebody has an avocado on them. [03:03:49] * atgo reads backscroll [03:03:52] what....? [03:04:05] 0_______________0 [03:04:15] ^^my new answer to everything [03:05:06] K4-713: ejegg: K so I'll write the e-mail, also update the card, in about 1-2 hours, is that OK? [03:05:14] atgo: ^ [03:05:15] yes. [03:05:20] K cool [03:05:22] thanks [03:05:31] AndyRussG: yes, for the impressions? [03:05:57] yes, about the impressions, there's a patch but it may be a bit sketchy, not very sure [03:07:11] fine by me [03:07:20] Sounds good, thx :) [03:07:21] you guys are amazing, btw [03:07:31] thanks and likewise :) [03:07:42] K if I don't speak to all of you later and/or don't see you on IRC over the weekend, have a great time, and see you on Monday afternoon! :) [03:07:55] ejegg: I don't know how fast I'm going to be able to CR anything you do tonight, so rushing is probably not necessary. [03:08:34] I still have to handle this lack of dinner thing going on. [03:09:10] K4-713: nom things [03:09:15] Well, I'll get all the stuff ready for CR by the time I sign off, and will deploy as soon as somebody gets a chance to +2 [03:09:25] Works for me. [03:09:47] I'll kick that down the hallway as soon as I am capable of something resembling rational thought. [03:09:49] sorry for the spammy mess today [03:09:57] also, please see my most recent email to fr-online :P [03:11:48] brb... [03:14:26] AndyRussG|away: are we in a spot where we can mark the "bucket" cards as deployed? [03:25:33] (PS5) Ejegg: Big English header piece [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/175055 (owner: Ssmith) [03:26:48] (PS6) Ejegg: Big English header piece [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/175055 (owner: Ssmith) [03:27:23] (PS4) Ejegg: Add board stats piece [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/175161 (owner: Ssmith) [03:38:22] atgo: yes, steps 1-3, except for the one that was filling in the gaps in the tests (step 3) [03:38:45] so mingle is accurate? :) [03:44:00] atgo: lemme see :) [03:44:35] thanks! i'm going to send out the sprint summary tonight so just want to make sure i'm up to date [03:45:25] atgo: lemme update 2196... [03:45:43] danke [03:46:00] bitte [03:57:49] atgo: K done :) [04:00:13] thanks for all your hard work AndyRussG [04:00:18] i know this has been... a crunch [04:01:44] atgo: mmm np and likewise... [04:02:09] crunch time is... crunch time [04:02:42] word. [04:03:26] AndyRussG: is the idea that we need ellery to help with the impressions thing? [04:03:37] that's part of it [04:04:29] some better understanding of more of the Javascript that's in the banners, or at least better than the understanding currenty in my brain, would also be important [04:04:36] ok, sounds good [04:04:39] that's all i need :) [04:04:52] I think awight, ejegg and or the-wub might already have that understanding in their brains [04:05:07] but maybe I'm not sure [04:05:54] and I think can't hurt for someone who hasn't gone thru much of that code before (like me) to do so [04:06:21] yeah that makes sense [04:06:23] sounds good [04:06:32] does this sound accurate? [04:06:33] Impressions fixes for hide impressions is pending deployment (need help from Ellery and possible Adam, Elliott, or Peter) [04:06:44] minus the grammatical error... ;) [04:06:49] uuhhh [04:06:51] maybe [04:08:07] hokay [04:08:51] Maybe:, ...is pending deployment, but could do with a little more investigating to be quite sure about it...(need help from Ellery and possibly Adam, Elliott, or Peter, or alternately Andy can study banner code some more) [04:09:26] Think of it like this: we have users doing stuff, software doing stuff and sending results, then statistics programs crunching the results to summarize what users are doing [04:09:33] righto [04:09:58] So the first and currenty weak link at least in my understanding users doing stuff <-> software doing stuff [04:10:41] And Ellery needs to be able to make the round-trip in his mind, from users doing stuff to his statistical summaries [04:11:46] Gaining more understanding of how the software reports what (by studying it a bit more or asking others) seems important to a real solution [04:12:35] No sense in crunching numbers if you're not sure what the numbers actually mean in terms of human behaviour [04:12:39] yeah fair enough [04:13:17] :) [04:13:20] can never be too fair! [04:16:20] atgo: should I still write to fr-tech (as per backscroll) or was the above a draft of an e-mail you're drafting...? [04:16:32] i'm just doing the sprint summary [04:16:39] Ah OK got it [04:16:43] i think if you want to do something more detailed that would be not a bad idea [04:16:49] particularly if you need support from others [04:16:52] K will do :) [04:17:18] Also, should I add cards to Mingle for the emergency fixes we did this week? [04:18:23] yeah that would be good [04:18:26] thank you so much! [04:18:38] it's always good to have context for future reference... in case we do something that then causes other problems :P [04:24:23] Hmmm... that never happens, indeed [04:33:28] atgo: OK, done the mingle cards... [04:33:36] thank you :) [04:33:57] np :) [04:39:05] can i help you with anything AndyRussG? [04:39:32] all good here I think atgo, thanks! [04:39:40] ok! happy thanksgiving in canada? [04:40:26] Heh thanks, in fact Canadian Thanksgiving is in October :) [04:40:50] well enjoy the day, anyway [04:40:53] i'll be on email i'm sure :P [04:40:56] ciaoooo! thank you!! [04:41:00] thank you! [04:41:04] Happy thanksgiving there! :) [05:03:09] (PS2) Ejegg: Add totals earned chart widget [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/175742 (owner: Ssmith) [05:04:53] (PS3) Ejegg: Add totals earned chart widget [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/175742 (owner: Ssmith) [05:29:41] (PS2) Ejegg: Add distance-to-goal chart views to BE board [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/175801 (owner: Ssmith) [05:29:43] (PS4) Ejegg: Add totals earned chart widget [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/175742 (owner: Ssmith) [07:05:06] ejegg: Are you still actually there? [07:06:07] I probably logged in just now to make sure you weren't working. [14:22:45] (PS1) Awight: Remove unused banner variance parameters [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/176251 [15:03:59] morning awight and ejegg :) [15:06:36] AndyRussG: hi! [15:06:59] AndyRussG: do whatchu like with the above patch... ^^ [15:07:05] It was on my mind... [15:07:11] K u bet, thanks :) [15:07:20] * awight rubs hands [15:07:30] meanwhile, I'm adding salt to the mailing list :) [15:07:44] salt and spice? [15:07:45] My mind is more on the DjangoBannerStats patch... [15:08:03] Ah that mailing list! :) [15:10:56] Funny in the old log samples I had from the last time I worked on DjangoBanner, I had an interesting variety of combinations of URL params [15:11:23] I bet! [15:11:27] For example hideImpression ones look like this: Special:RecordImpression?country=US&uselang=en&project=wikipedia&db=enwiki&bucket=0&anonymous=true&device=desktop&banner=B14_1020_enUS_gg_y&campaign=C14_enUS_dsk_lw_FR&reason=alterImpressionData&result=hide [15:11:34] uh [15:11:37] urgh [15:11:48] Sorry I mean alterImpressionData ones [15:11:49] I think ejegg patched that reason thing [15:11:54] Yes indeed [15:12:35] And the normal empty ones looked like this: Special:RecordImpression?country=PH&uselang=en&project=wikipedia&db=enwiki&bucket=0&anonymous=true&device=desktop&reason=empty&result=hide [15:13:09] So the script, as it was, would count the first kind as an impression, and the second kind were not counted as such [15:13:25] Then we added campaign and banner params everywhere [15:13:34] And both kinds started to be counted as impressions [15:14:22] With the above patch I we remove the requirement of not having campaign and banner params for a URL to not count as an impressions [15:14:39] And both kinds should now _not_ be counted as impressions [15:15:40] I _think_ that's correct, but I feel I'm not familiar enough with the alterImpressionData stuff to really know [15:16:38] I guess across the board anything that says result=hide on it should not be counted as an impression, regardless of what else is in there, no? [15:17:13] (BTW the best viewer for the patch is kdiff3 I think, it shows you where there were only whitespace changes) [15:21:55] * AndyRussG stops pestering innocent co-workers who should be on holiday :) [15:22:25] brb! (I'll get backscroll tho) [15:22:26] backscrolling... [15:22:31] a ha [15:23:27] so, Ellery wants the hide results to be associated with the banner and campaign that *would* have been shown [15:23:30] for debugging [15:23:41] I know it seems weird at first sight [15:24:05] but yeah neither kind should be counted as an impression, by ecom et al. [15:24:28] I thought your second table suggestion was a good one. [15:25:29] Yeah agreed we should do a second table for the hide results [15:25:43] The banner and campaign params I don't think are problematic [15:28:11] The line that was causing trouble is "if banner == "" or campaign == "" or project is None:" [15:28:25] oh man that logic is junk [15:28:40] heheh I think so yes [15:28:48] But it was hard to know so [15:29:03] So I just unindented this block "if "result" in qs and qs["result"][0] == "hide":" [15:29:26] So that all the hide params wouldn't be caught up in that, since now _everything_ gets banner and campaign params [15:30:07] The line block that I did leave under hte no-banner no-campaign block is this one: 'if "BannerController" in l:' [15:30:10] This brings up another thing... not solvable at the moment, but https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/2194 [15:30:26] Just to not move it around, but really I don't understand what that line is for... [15:30:49] Me neither :( [15:30:58] Ah! iiinteresting [15:31:48] It's like code archeaology [15:32:34] Comment sez, "# we really don't care about these, so there is no need to log them as errors" [15:32:55] :) [15:32:58] okay then [15:34:04] perhaps there was a smallpox epidemic, and this culture lost the elders who were naughty enuf to send other junk through Special:RI [15:34:22] heheheh [15:34:42] time travelling banner impressions [15:35:00] Mmm I don't have the "bannercontroller" string anywhere in my sample logs [15:35:38] oh good. I like when code anticipates bugs [15:35:49] if (1 === 2)... [15:38:35] Heheh like in js ( true != 1 ) is false [15:40:59] Hmm maybe I found it...? There is a special page alias 'BannerController' => array( 'BannerController' ) in CentralNotice.alias.php [15:44:10] what is it? [15:44:47] there are some references in the README (we have a readme??) [15:45:06] looks like it was pre-ResourceLoader [15:45:58] hehehe wow [15:46:06] Let's rename it DONTREADME [15:49:57] word! [15:50:01] Well I guess we can safely stop worrying about "if "BannerController" in l:" [15:50:16] maybe that's what they meant with the old "don't tread on me" flag [15:50:26] "if you can read this, your're too close" :) [15:51:01] heh, if you can read this, you're about to fall into a spacetime wormhole to the pre-RL past [15:57:32] (PS1) Awight: remove outdated README [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/176284 [16:18:47] (PS1) Awight: Upgrade to Drupal 7.34 upstream [wikimedia/fundraising/crm/drupal] (contrib) - https://gerrit.wikimedia.org/r/176288 [16:41:25] ejegg: Here're some thoughts about DjangoBannerStats in case you're interested and have some thoughts, ahead of the e-mail that I think I might send: [16:41:52] In summary, the URL params of Special:RecordImpression constitute and API [16:42:23] And as such needs to be clear about what params mean and how they should be used, as in, there needs to be a contract [16:43:17] As it stands, bannerController.js cannot actually control what the JS in the banner does, it has to assume that code operates in good faith [16:43:27] Also a contract is needed there of course [16:44:00] So for the purpose of consistency in recording impressions I'd propose the following bits of contract: [16:44:31] If a banner returns false from alterImpressionData(), it _must_ hide the banner from the user [16:45:36] If alterImpressionData() returns false, bannerController _must_ set result=hide on the RI URL [16:46:16] If the banner is hidden for any reason whatsoever, bannerControler _must_ set result=hide on the RI URL [16:46:30] regardless of what other info it decides to send along on that URL [16:47:51] Those agreements already hold for bannerController.js as it's currently deployed. If we can agree to make all banner-contained JS hold to them, then we're good to go with the patch just merged to DjangoBannerStats [16:47:57] whaddya think? [22:31:37] (CR) Awight: "CR+1, just looking..." (3 comments) [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/176214 (owner: AndyRussG) [22:43:27] AndyRussG: whew, it's always rough getting into that obfuscation... [22:49:43] I have to go unravel the knot that caused :) [22:49:57] awight: heh, I noticed there was old stuff but I didn't realize it was so old until we talked about it earlier [22:50:14] I really want that codebase to go away, in January... [22:50:37] If I still have a job, that will make a fun project :) [22:50:55] I think your forray into documenting the params on Special:RI was great, if we continue on that route and treat these as real APIs with contracts, we'll surely be merrier... [22:51:16] huh? what do u mean? 8o [22:52:45] meh, it's been quite fun writing to wmfall, but I think my privileges will be revoked if I try that again ;) [22:53:11] nah! unthinkable [22:53:30] yeah, +1 on documentation. Not that my trembling stab at it was any good, but agreed that we need to be explicit, everywhere. [22:54:11] Also: your patch to the Django thing looks awesome with my Gerrit interface defaulting to ignore whitespace changes :) [22:54:25] I don't think anyone could construe your wmfall participation as anything but honest and thoughtful, which is great [22:55:26] <_< [22:55:26] >_> [22:55:48] * awight attempts to look shifty [22:56:04] ok well you can write me a nice recommendation :p [22:56:34] or nice to the 10th power [22:57:05] hehe integer overflow [22:57:07] ya your attempts at shifty appearance don't fool [22:58:36] Well, I think I'll deploy your impressions patch later tonight, will write if I do so. [22:58:39] and wmfall participations aside, your work is also fantastic, so no worries :) [22:59:03] hey now, this sounds like a farewall 8p [22:59:45] heh not at all [23:00:00] re: deploy, I think K4 was interested in getting Ellery involved [23:00:05] to monitor data... [23:00:09] Aha yeah that makes sense [23:00:32] cos I was sad imagining how I would confirm that hide data was not showing up! [23:00:50] She said she might drop by but I haven't seen her... maybe we should try to organize a deploy time by e-mail? [23:01:19] Yea at a 1:100 sample rate, it's hard to test except in the real world... [23:01:20] Sure, it sounds like we'll just need me and Ellery, and u can be optional? [23:01:35] sure [23:02:02] I can make time all week/weekend, so it's up to EW [23:04:56] whoa strange um undejavu. Did I not just ask you... [23:05:16] Was my email correct, that we will no longer have hide information in the db? [23:05:29] I see that it was never collected intentionally... [23:06:40] I had difficulty parsing Ellery's original request, I think he was saying he wants to continue getting the banner and campaign URL params for result=hide impressions, but he doesn't care that we won't have it in the pgehres schema? [23:07:21] Hehe, I can still tab-complete pgehres ;) [23:07:28] * awight waves [23:08:10] IIRC the hide info is just logged in like a Django logging thing, not in the DB [23:08:16] At least the hide details [23:08:34] ok that's what I see, too. [23:09:57] gotta run!