[00:16:13] (PS1) Katie Horn: Fixes the date formatting / normalizing. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/159638 [00:16:17] (CR) Awight: Bug fixes for the initial commit of the WP recon parser (3 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/159631 (owner: Katie Horn) [09:17:53] (PS4) Qgil: GatewayAdapter::isSupported [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/64872 (owner: Awight) [09:17:58] (CR) jenkins-bot: [V: -1] GatewayAdapter::isSupported [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/64872 (owner: Awight) [09:19:00] (CR) Qgil: [C: -1] "Depends on an outdated patch." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/64874 (owner: Awight) [09:19:36] (PS2) Qgil: Differentiate rapidhtml failure cases [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/95874 (owner: Awight) [09:23:10] (CR) Qgil: [C: -1] "Depends on an outdated patch." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/63823 (owner: Awight) [09:40:14] jessicarobell:hi, done [09:40:33] I generelly support tjanes translation [09:49:23] Hi Till_WMDE. Ok wonderful! Thanks so much for your help. I really appreciate it. [16:50:08] (PS1) Katie Horn: WIP: DO NOT MERGE - Generic audit logic and worldpay audit job [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/159761 [16:51:13] (CR) Katie Horn: [C: -1] "Don't." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/159761 (owner: Katie Horn) [16:56:18] Aw, rats. [16:56:27] awight is not here today. [16:56:38] atgomez: Hello. [16:56:55] hey K4-713! [16:57:18] Er, question for... somebody. Maybe not you, but whatever. [16:57:34] Did anybody do a chargeback with Worldpay? [16:57:38] Like, a real one. [16:57:48] that i do not know... maybe ccogdill would know from the DS perspective? [16:57:56] Also refunds. [16:58:37] a chargeback? not that I know of... [16:59:12] usually those are initiated by the gateway on behalf of the donor [16:59:24] but I’d think PPena would have gotten notification [16:59:36] hmmmmmm. [16:59:52] do you have a transaction ID or anything? [16:59:55] I didnt do a chargeback with WP [17:00:08] A transaction ID is exactly what I'm looking for. [17:00:12] someone from belgium [17:00:25] ...for both a chargeback and a refund. [17:00:41] I need to verify via actual data that they're showing up in the nightly files the way they're supposed to. [17:01:04] we have for refunds [17:01:09] not for chargeback [17:01:12] Dang. [17:01:18] is that something we want to start testing from now on? [17:01:33] Well, if we want that data to get into the system, I need to write that bit. :) [17:01:59] If I don't have an example to verify that the nightly recon script works, I can't say for sure that they will. [17:02:05] There are two recent test donations from me via WP that we can refund / chargeback [17:02:07] PPena I don’t have a record of any WP refunds [17:02:10] besides our test ones [17:02:26] Yeah, I need them to have been on our production account. [17:02:50] I think they were - one was at least, when I tested after deployment [17:02:53] it couldn’t have been one of our test donations? I thought we made some in production [17:03:06] Hang on: Let me start over. [17:03:33] I am currently in the process of working on the thing that uses the nightly reconciliation files to get data into civicrm. [17:03:46] That's both initial transactions, and refunds and chargebacks. [17:04:00] I have a large number of initial transactions to spot-check. [17:04:13] I do not have any information about actual refunds or chargebacks that I can identify. [17:04:35] ohh [17:04:51] hah well I was confused [17:05:07] Presumably, after we ran it in Belgium, somebody asked for a refund. Also reasonably likely that somebody charged something back. [17:05:27] usually the gateway would let us know [17:05:36] I was just wondering if a) that had happened, and b) what a couple of transaction IDs were if it had. [17:05:36] so I don’t think we’ve had any chargebacks yet [17:05:49] I can let you know when we get one! [17:06:20] Cool. How about the refunds? [17:06:41] Also, I don't actually know what the rules are about... using our own cards to test chargebacks. [17:07:02] Usually when you charge something back, you have to say "I DID NOT AUTHORIZE THIS" and I'm... sort of averse to lying to banks for some reason. [17:07:11] * K4-713 shrugs [17:07:25] yeah I don’t think we can do a test chargeback [17:07:50] we’ve certainly done test refunds. do you want me to find a transaction ID for one of those? [17:08:01] When you say "test" refunds... [17:08:08] ...do you mean on the production account? [17:08:30] I think "test" means something different to me. [17:11:08] ccogdill: If you did a refund of actual money on the production worldpay account, that would totally do it. I'd love a couple transaction IDs like that. [17:11:11] k4-713, I just mean a donation we made [17:11:15] to test it in production [17:11:19] heh [17:11:29] okay let me look through our reports and see if I can find one :) [17:11:34] Yeah, so, usually when we say "test" transactions down here, we mean on the testing/sandbox account. [17:11:49] ccogdill: Thank you! [17:12:44] yep! sorry for the confusion [17:13:34] No worries. I'm pretty confusing. :) [17:14:22] I really do think we need some kind of disambiguation list for what the word "test" means in fundraising. [17:14:55] Because, near as I can tell, it can mean anything from "we are only making transactions with fake test cards on the sandbox account" to "we are running an actual campaign right now." [17:15:28] ...I'm going to go test another cup of coffee. [17:17:18] hehe [17:17:49] K4-713, #8413204728 [20:45:45] ejegg: Do you know anything about wprldpay order IDs changing format? [20:48:43] no, did they change? [20:49:18] err, is this part of awight's big refactor? [20:49:55] wait, do you mean the IDs they generate, or the ones we create? [20:52:47] K4-713: what exactly changed? [20:53:12] Well, the latest contribution from you, in civi, is a 10-digit random number. [20:53:34] Before, they were all [contribution_tracking_id].[effort_id]. [20:54:53] can you tell when it changed? [20:57:21] nope [20:57:40] Well... wait. Sort of. [20:57:53] It was between that contribution, and the last one in civi before that. [20:58:00] :/ [20:58:12] Wait 1 while I figure out how to be more precise about that. [20:58:45] whoa, dang! So it must have been that deployment [20:58:55] let's see what I pulled in [20:59:03] Well, maybe not. [20:59:09] I think there was a month-long gap there. [20:59:26] Maybe longer. [21:00:53] ...two months, in fact. [21:01:22] 2014-06-27 07:20:17 was the last one with the format I'd expect. Yours was 2014-08-28 22:10:34. [21:01:36] ejegg: ^^ [21:02:25] The thing is, it looks like the thing WP uses to correlate these things didn't change. [21:02:50] So, I can't find you in the nightlies anymore. Nor, presumably, anybody else who uses it until we change it back. [21:04:50] so, mwalker committed code on 5/2 that changed order id generation - but it looks like it should have created the format you're expecting, not the random number [21:05:06] Yeah, 5/2 was before the last one that worked. [21:05:19] Something else must have changed. [21:05:27] let me see when the 5/2 code was deployed [21:05:56] I don't think we ran anything in WP until June, which would have taken it all for sure. [21:08:45] wait, were there any with either format between 6/27 and 8/28? [21:09:04] Nope. None at all. [21:09:08] dang [21:09:08] Which makes sense. [21:09:14] We stopped sending people there. [21:10:57] getTransactionSpecificValue is never used to generate the order id, is it? [21:11:15] ... [21:11:40] * K4-713 looks incredibly stupid [21:12:05] so the parent's generateOrderID function returns a random number [21:12:48] and I overrode getTransactionSpecificValue to call parent::getTransactionSpecificValue, then mangle the encoding [21:12:55] oooooooh. [21:13:24] ...no, wait. I'm still not sure how that would do that. [21:13:25] but I don't see where gTSV actually calls functions to get values [21:14:45] Yeah, I think this is a red herring. [21:15:19] (which is a phrase I need to use more) [21:15:54] Also, it looks like we haven't downloaded a worldpay recon file in months, either. [21:15:59] now i need to know what pickled herring tastes like [21:16:07] It's... [21:16:11] ...distinctive. [21:16:31] You're not like "Oh, I can't place the flavor". [21:16:59] nice. must stop by the specialty shop [21:17:12] Warn your friends and neighbors first. :) [21:17:19] heh [21:20:17] back in a sec, moving to less noisy spot [21:20:30] no worries. [21:20:44] ejegg: I don't think we're doing video standup today, anyway. [21:22:59] ... [21:23:40] ...the heck am I looking at. [21:24:13] If I didn't know better, I'd think that WP completely changed the naming convention on their nightly recon files about two months ago. [21:24:53] It's a little hard to tell with an alpha sorted list, though, because dates in their files go ddmmyy. [21:25:20] Am I being punk'd? [21:25:46] (does that even still happen) [21:26:13] there's a commit on 6/6 where you check to see if the order_id is decimal and if getOrderIdMeta has 'disallow_decimals' mark it as needing to be regenerated [21:26:46] That seems like a silly thing for me to do. [21:26:56] * K4-713 frowns [21:27:00] 'hack to make sure decimal order_ids that are leaking in from WorldPay are going to be thrown out and regenerated by GlobalCollect' [21:27:09] ohfer. [21:27:29] That commit... touches what files? [21:27:52] let me see - i was just reading history of gateway.adapter.php [21:27:56] hahaha [21:28:04] ...sleeping pills not working? [21:28:35] Yeah, that just feels like the right answer. [21:28:41] just trying to get insight into this puzzle [21:28:45] The... oid hack. Not sleeping pills. [21:29:16] that points to it being a problem, but doesn't seem like it would /cause/ the problem [21:29:55] just touches that file and the globalcollect adapter [21:30:12] the main adapter? [21:30:14] yep [21:30:16] yeah [21:30:17] That's got to be it. [21:30:31] so maybe the problem was introduced a little bit before that, and you solved it for GC [21:30:32] It's not letting the WP decimals through without a regen, sounds like. [21:30:50] huh? [21:31:12] but since we didn't add 'disallow_decimals' to the meta for WP, your fix only applied to GC [21:31:31] We want WP to have the other format with the decimal point. [21:31:35] It doesn't now. [21:31:43] ohhhh, right! [21:31:59] wait, what? [21:32:29] I mean, the commit message there sounds sensible. [21:32:47] ...if it actually worked that way. [21:32:53] right, but wouldn't it only apply to the GC adapter? [21:33:03] Unless I screwed something up. [21:33:10] Seems likely. [21:33:34] And, since we haven't actually used WP since then, there would be no change to notice the problem until now. [21:33:54] Chance. [21:33:57] urf [21:38:46] ejegg: Okay, well... rats. I'll make a card that describes the issue and let Anne know. [21:39:21] While I was trying to track that down, I also noticed that our nightly downloader for WP has either been misbehaving, or they have been sending us entirely empty files for months. [21:39:54] I can't find your transaction in there, because there's no there in there. [21:40:03] I guess no standup today? [21:40:07] >:[ [21:40:07] oh dang [21:40:46] AndyRussG: Yeah, I'm pretty much out of anything but IRC standup until I show up and say "Yay, the auditing works again!" [21:41:17] Got it... that sounds not so good... 8( [21:41:20] Can I help with anything? [21:41:47] Hm... not sure. Anne would probably know better than I do on that front. [21:41:47] I'll send updates by e-mail then [21:42:04] I have entered total tunnel vision mode. [21:42:18] Hmm like warp drive? [21:42:35] More like Ludicrous Speed. [21:42:44] In fact, it's exactly... haha [21:46:47] K good luck, K4-713 ejegg atgo-appt again please don't hesitate to ping if I can help with anything that's urgent... I'll be OOO from 4pm to around 7:30 pm Pacific time, but other than that I'm around [22:50:28] K4-713: where's the source for the downloader? [22:56:07] ejegg: Sorry, I had to go stare at the ceiling for a bit, apparently. [22:56:19] Now I've got some ideas, though, so... woo. [22:56:23] how's the ceiling looking? [22:56:25] The downloader is in fr-tools. [22:56:30] cool [22:56:37] Dude. The ceiling is *always* interesting. [22:56:48] I have discovered many answers up there. [22:57:06] See the word 'gullible' written anywhere? [22:57:15] WHERE. [22:57:17] :p [22:57:23] ...I can never seem to find it, but they swear it's up there [22:57:53] i think it may be a religious thing [22:58:07] I told somebody once that my SEO skills were so great, I could make them come up in the top 10 google results for "gullible". [22:58:13] nice [22:58:15] Oddly, I was not told to get lost. [22:58:33] is the downloader under 'sftp'? [22:58:39] ehrm... [22:58:50] oh wait, there's also 'audit' [22:58:54] Yeah, that one. [22:59:10] For the definitive answer, log in to Jenkins and see what it's running. :) [22:59:23] well that's a concise little script [22:59:32] There's more. [22:59:43] grumble grumble python grumble. [23:00:28] i just pulled in a php sftp lib for the silverpop downloads, if you want to use that for the rewrite [23:00:47] ...assuming you're rewriting inside the /crm repo [23:01:16] Ah, I'm not rewriting the download scripts right now. [23:01:23] cool. [23:01:43] I'm not sure if there's currently a problem with what we have, or if WP is pumping empty files into our directory. [23:01:46] oh, i see. most of the downloading logic is under the 'sftp' folder [23:01:49] It's definitely one of the two. [23:02:13] I think the card mentions the last one we pulled. [23:02:17] ...it used to work. [23:03:40] let's see what kind of error handling that paramiko library has for .get [23:08:13] huh, doesn't look like anything should be squelching exceptions [23:08:30] have you tried logging into the sftp server and looking at file sizes? [23:10:18] ejegg: I didn't try anything yet. [23:10:46] They may actually be empty. [23:11:04] ok. looks like it should raise an IOError if anything goes wrong during the transport. [23:20:54] so i'm getting 'access denied' when I try to ssh in to that server using the settings file the machine that runs the script. but that should be a totally different error in the logs. [23:21:31] blarg. sshing into wp's server using address and credentials from the config file on our box [23:21:54] their host key matches what we have in the credentials, so I know I'm getting to the right place [23:22:49] hurm... [23:23:10] Well, the jenkins job is at least seeing files on the other side. [23:24:47] whoa, odd. I tried with a variation on the password and got 'permission denied, please try again' [23:25:14] with the right p/w, it just closes the session with 'Access denied. See server log for details.' [23:25:26] * K4-713 squints [23:25:32] So maybe they're just terminating connections really fast? [23:25:32] * K4-713 squints more [23:26:00] i think their sftp server is wonky [23:26:59] Maybe we made it mad. [23:28:02] wow i'm dumb [23:28:25] i was typing ssh instead of sftp this whole time. [23:28:25] Not dumber than me. But go on. [23:28:30] HA [23:28:31] brain/finger disconnect [23:28:46] gee, they didn't give us shell access! [23:29:23] Weird. :) [23:29:51] well, they didn't freeze our account for those attempts [23:30:05] That's something, I guess. [23:30:14] wow, there are a buttload of files in our directory. Maybe we hit a quota? [23:30:40] That would be insane, because I think they kill old files for us. [23:30:58] So, the only ones I care about, are the ones that have RECON in them. [23:31:15] The rest are just sort of there, being functionally useless. [23:31:16] oh, it was just the first 'ls' that took forever, subsequent ones are faster [23:31:53] huh [23:32:18] Like... 16 seconds? [23:32:34] maybe 10 [23:32:49] Looks like it took Jenkins 16 seconds last time. [23:32:56] YAY LOGS. [23:32:59] just to list the files? [23:33:02] Logs are my favorite. [23:33:25] Er. It took 16 seconds to connect and skip the first file because we already had it. [23:33:47] well, definitely a bunch of zero size RECON files in there [23:33:48] the ls has to be in there somewhere. [23:33:53] wth. [23:34:03] as far back as april [23:34:04] Okay, we should yell at them, then. [23:34:12] Fantastic. [23:34:30] though there are some from apr-jun with content [23:34:35] Can you put that information in the card with instructions to poke them with sticks until our files have data in there? [23:34:42] ok, will do [23:35:11] At the very least, your one transaction on prod should have caused a file with something in it. [23:35:16] right [23:35:25] should there be 1 file per day? [23:35:33] Weekdays, yes. [23:35:47] Financial jobs peace out on the weekends for some archaic reason. [23:35:56] I mean, not ours. :) [23:36:08] But this is sort of normal for payments gateways. [23:36:36] I'd like to think that their servers need some time off to watch cartoons and eat cereal. [23:36:41] argh, ls does not respect the -t option in sftp [23:37:25] hah [23:37:39] ok, all the weekdays in august have a zero-length file [23:37:49] What about July? [23:37:57] And, isn't their date format the best ever? [23:38:10] omg so annoying!!@!@ [23:38:20] yeah, july is full of zeros too [23:38:35] but whatever august day I donated isn't missing, just incorrect [23:38:44] buh? [23:39:05] i mean, they did make a file for that day [23:39:11] it was just zero bytes [23:39:19] Ah. Yes, okay. [23:39:22] That's what I thought. [23:42:50] Jun 30th they had a 580 byte file, and Jul 1st was zero. bet they deployed some kinda code change [23:43:34] oh right, ticket already has that info [23:43:44] Happy Fiscal year! To celebrate, we have reset all your account preferences. [23:43:49] hah [23:44:04] * K4-713 pulls fire alarm to express joy [23:44:15] it's like jubilee [23:44:54] ...not the mutant [23:45:13] I was actually sort of thinking that. [23:45:56] did she ever face off against dazzler? [23:46:05] I... don't know. [23:46:18] That would have been great to put on a barge for somebody's birthday, though. [23:46:28] hah [23:46:47] looks like only in nerds' imagination [23:48:55] oh man, the comicvine battles forum looks like a dangerous timesink