[00:04:05] (CR) Awight: [C: 2] Fix state truncation warnings (1 comment) [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/289593 (owner: Ejegg) [00:09:18] (CR) Awight: [C: 2] "TIL insert ignore is evil, it seems! The business about silently coercing fields to a different type is real scary. Thanks for noticing!" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/289592 (owner: Ejegg) [00:09:43] (Merged) jenkins-bot: Clean up some warnings [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/289592 (owner: Ejegg) [00:09:45] (Merged) jenkins-bot: Fix state truncation warnings [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/289593 (owner: Ejegg) [00:10:05] be on later... [00:45:53] (CR) Eileen: "I did some performance tests on INSERT IGNORE & found it significantly slower than doing a left join + filter - on the civicrm_acl_contact" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/289592 (owner: Ejegg) [01:24:20] fundraising-tech-ops, Operations, ops-eqiad: Rack and setup Fundraising DB - https://phabricator.wikimedia.org/T136200#2328837 (Jgreen) [05:21:12] (CR) Awight: "Very nice! I'll omit "ignore" in the future." [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/289592 (owner: Ejegg) [10:54:36] (PS2) Gergő Tisza: Update mediawiki_api gem to 1.7.1 [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/290834 (https://phabricator.wikimedia.org/T135884) [11:54:36] (CR) JanZerebecki: [C: 2] Update mediawiki_api gem to 1.7.1 [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/290834 (https://phabricator.wikimedia.org/T135884) (owner: Gergő Tisza) [12:13:00] (Merged) jenkins-bot: Update mediawiki_api gem to 1.7.1 [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/290834 (https://phabricator.wikimedia.org/T135884) (owner: Gergő Tisza) [13:30:52] Fundraising Sprint Asbestos Removal 2016, Fundraising Sprint Bloodletting 2016, Fundraising Sprint X-Ray Spex, Fundraising Sprint Yo La Tengo, and 4 others: Do not show donation form error message: "No processors available". Fix UI and plug holes. - https://phabricator.wikimedia.org/T117872#2329936 (... [13:39:58] Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, Fundraising-Backlog: CentralNotice: test registration - https://phabricator.wikimedia.org/T134286#2329984 (Pcoombe) Thanks for the change! When should we expect this to be deployed? Just want to know when I can start working on T... [14:08:17] Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, and 4 others: donatewiki and paymentswiki: We should stop falling back to Russian when looking up unavailable Ukrani... - https://phabricator.wikimedia.org/T135254#2330120 [14:55:51] (PS3) Cdentinger: Use key as default queue name [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/287782 (owner: Ejegg) [14:59:05] (CR) Cdentinger: [C: 2] "Cleaner for sure" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/287782 (owner: Ejegg) [15:00:14] (Merged) jenkins-bot: Use key as default queue name [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/287782 (owner: Ejegg) [15:14:25] (CR) Cdentinger: "It seems to me like this piece shouldn't have to wonder if there's a damaged queue or not, the calling code might be a better place for th" (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/287872 (owner: Ejegg) [15:59:10] (CR) Cdentinger: "All this config stuff is very wat, with so much logic tied up in getters. It would be simpler to build a config tree from the various sou" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/288314 (owner: Ejegg) [16:26:08] Fundraising-Backlog, FR-Ingenico: Spike: Make an estimate of the proportion of people affected by order id collision - https://phabricator.wikimedia.org/T136328#2330785 (awight) [16:30:34] Fundraising Sprint Killing Time, Fundraising-Backlog, FR-Ingenico, Unplanned-Sprint-Work: Spike: Make an estimate of the proportion of people affected by order id collision - https://phabricator.wikimedia.org/T136328#2330816 (awight) [16:36:59] it's nice to get a fresh set of eyes on this code [16:37:42] remind us how wacky it is [16:39:38] heh, smash pig especially [16:48:08] (PS1) Ejegg: Don't cancel payments missing donor details [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/290980 (https://phabricator.wikimedia.org/T136038) [16:51:43] (PS2) Ejegg: Don't cancel payments missing donor details [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/290980 (https://phabricator.wikimedia.org/T136038) [16:51:47] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Spike: Configuring a banner sequence test - https://phabricator.wikimedia.org/T135398#2297497 (awight) It crossed my mind that we might be able to produce sample code for doing a sequence test, then hand off to the banner team to play with. Howeve... [16:53:42] I think that's all we need to change for now to stop cancelling payments on queue gliches: https://gerrit.wikimedia.org/r/290980 [17:00:11] (PS1) Ejegg: Rationalize accesibility of fns and methods for Extras [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/290982 [17:05:00] (PS2) Ejegg: Comment and DRY for functions filter [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/290830 [17:05:02] (PS2) Ejegg: Rationalize accesibility of fns and methods for Extras [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/290982 [17:05:04] (PS2) Ejegg: Rename hook functions [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/290829 [17:14:30] (PS1) Ejegg: Const and better name for initial ip velocity flag [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/290990 [17:17:44] (CR) jenkins-bot: [V: -1] Const and better name for initial ip velocity flag [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/290990 (owner: Ejegg) [17:45:18] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Spike: Configuring a banner sequence test - https://phabricator.wikimedia.org/T135398#2331241 (awight) Initial consensus in today's tech priorities meeting was that we should avoid anything involving buckets even for the trial implementation. [19:19:40] Fundraising Sprint Killing Time, Fundraising-Backlog, FR-Ingenico, Unplanned-Sprint-Work: Spike: Make an estimate of the proportion of people affected by order id collision - https://phabricator.wikimedia.org/T136328#2331592 (awight) ``` zgrep -c 'Order ID collision' payments.error-201605* payme... [19:44:43] Fundraising Sprint Killing Time, Fundraising-Backlog, FR-Ingenico, Unplanned-Sprint-Work: Spike: Make an estimate of the proportion of people affected by order id collision - https://phabricator.wikimedia.org/T136328#2331720 (awight) I've made two graphs that show the number of Order ID collision... [20:00:59] Fundraising Sprint Killing Time, Fundraising-Backlog, FR-Ingenico, Unplanned-Sprint-Work: Spike: Make an estimate of the proportion of people affected by order id collision - https://phabricator.wikimedia.org/T136328#2331743 (awight) The easy answer is, the probability of the next randomly chosen... [20:03:08] Fundraising Sprint Jabberwock Slaying, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, FR-Adyen, and 3 others: donatewiki and paymentswiki: We should stop falling back to Russian when looking up unavailable Ukranian messages - https://phabricator.wikimedia.org/T135254#2331750 (DStrine) [20:03:38] sorry, logging into Hangouts now... [20:08:30] Fundraising Sprint Ghostbusting , Fundraising Sprint Hermit Crab Husbandry, Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, and 3 others: Only subscribe primary emails, secondary addresses should be suppressed - https://phabricator.wikimedia.org/T131979#2331785 (CCogd... [20:12:38] Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, and 3 others: Hide "cardholder name" field from Adyen form - https://phabricator.wikimedia.org/T124467#2331817 (DStrine) [20:16:02] Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, Fundraising-Backlog, Security-Data-Mapping: Rough draft of data flow map - https://phabricator.wikimedia.org/T133810#2331828 (awight) [20:19:40] ejegg: Also, I got a surprising answer for https://phabricator.wikimedia.org/T136328 so that would be worth peeking at to review [20:22:18] (PS2) Ejegg: Const and better name for initial ip velocity flag [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/290990 [20:34:35] Fundraising-Backlog, Documentation, FR-Amazon, FR-Smashpig: Document Amazon transaction workflow; make settlement more robust - https://phabricator.wikimedia.org/T136116#2331917 (awight) [20:35:11] awight: oh hey, that looks like good news. I'll take a look at the math! [20:36:10] awight: ping. want to stand guard while I deploy that last .yaml config change? [20:36:50] Jeff_Green: great, thank you! [20:37:19] k I'm watching the logs etc [20:38:04] k [20:40:44] dstrine: I donno if you saw my changes to the Q1 goals scratchpad? [20:41:01] My biggest question to you was whether we should promote PayPal EC to a committed goal. [20:41:02] (PS3) Ejegg: Const and better name for initial ip velocity flag [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/290990 [20:41:18] awight: ok it's propagating [20:41:34] awight: I responded ... It's actually a q4 goal ;) [20:41:51] * awight will not say "it looks good so far." [20:42:02] ha [20:42:04] dstrine: But it wasn't completed... [20:42:47] Q1 begins on July 1st there is still pleanty of time this quarter [20:43:20] hehe thank you for the vote of confidence :) Okay that's fine for now, thanks for explaining time [20:43:50] adyen says "why don't you build a buffer for when we send notifications at the wrong time" [20:43:52] that's nice [20:44:14] (╯°□°)╯︵ ┻━┻ [20:44:18] awight the japan campaign start on July 12th so we have a few mone days to be totally campaign ready [20:44:32] * a few more days [20:45:55] dstrine: I should be more optimistic in general--once it's deployed as an internal test, we're unblocked on all the fine-tuning and audit / listener upgrades. [20:46:10] cwd grr I just saw that [20:46:49] I already told fr-online that we should move adyen by another week [20:48:40] We can be unblocked on that audit thing, but * it will take wasted workaround time and * fundraising should know that their numbers could be delayed by an additional day. [20:50:03] it's also enabling them to not fix things [20:50:13] and thus probably signing up for more of this long term [20:50:14] They don't need our help with that :p [20:50:16] in a small way [20:50:41] They also don't need to know whether we're unblocked ;) [20:51:39] that's true [20:51:55] but when the workaround spirals into a huge thing because of reasons [20:52:05] we'll be mad at ourselves instead of them [20:52:12] cwd and awight I'm going to ask them for a cleared ETA for their fix and manage this with fr-online. They should fix this. [20:52:34] Seriously though. I don't even want to think about some hacky subtract one day thing, dealing with days_in_month and crap [20:52:40] we'll move the schedule around accordingly [20:53:16] :) [20:53:18] ... just found one of my hens' secret egg lairs, I believe [20:53:33] cute [20:53:41] She owes us about ninety, it must be crazy in there [20:53:59] * awight thumbs through feed sack receipts [20:54:50] at scale sprouting wheat and breeding meal worms becomes worth it [20:54:58] Jeff_Green: Adyen jobs have run a few times, I think we're all good! Thanks again [20:55:06] sure [20:55:20] so we can close that ticket [20:55:25] yah [20:55:37] oh wait, what about the .php flavor, is it safe to remove that too now? [20:55:46] dstrine: nice one [20:55:50] Jeff_Green: yes! Hooray! [20:56:36] the tone of "multiple irrelevant notifications" [20:56:48] is that how you consider your notifications? [20:56:51] irrelevant? [20:57:07] we take them seriously [20:57:18] LOL thank you [20:57:30] (PS5) Ejegg: PayPal Express Checkout: support locale well enough to get ja_JP [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/287157 (https://phabricator.wikimedia.org/T131811) (owner: Awight) [20:58:00] In the case of beaters who are downloading and importing audit files by hand daily, they seem to be happy to row through the night as well. [21:00:13] apparently [21:00:39] (CR) Ejegg: [C: 2] "Looks good to me! I'll hop on that TODO and re-use this for Adyen." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/287157 (https://phabricator.wikimedia.org/T131811) (owner: Awight) [21:01:48] dstrine: gah, you beat me to that hovercards thread, too [21:02:44] ejegg: Does Adyen do the same evil magick, where they guess language based on country? PayPal actively ignores our language= hint if we send country :( [21:03:38] Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, Fundraising-Backlog: CentralNotice: test registration - https://phabricator.wikimedia.org/T134286#2332009 (awight) [21:03:40] Fundraising-Backlog, Hovercards, MediaWiki-extensions-CentralNotice, Reading-Web-Backlog, Reading-Web-Sprint-74-P: Measure impact of HoverCards on Central Notice interaction - https://phabricator.wikimedia.org/T131366#2332008 (awight) [21:15:02] (PS4) Cdentinger: Remove referrer from queue messages [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289888 (https://phabricator.wikimedia.org/T110564) (owner: Ejegg) [21:15:11] awight: no, they actually want locale, but they've been making do with just language [21:16:16] Fundraising Sprint Killing Time, Fundraising-Backlog, FR-Ingenico, Unplanned-Sprint-Work: Spike: Make an estimate of the proportion of people affected by order id collision - https://phabricator.wikimedia.org/T136328#2332048 (awight) @Ejegg Wow, I was completely upside-down. Yes, if the chance... [21:16:18] Fundraising Sprint Killing Time, Fundraising-Backlog, FR-Ingenico, Unplanned-Sprint-Work: Spike: Make an estimate of the proportion of people affected by order id collision - https://phabricator.wikimedia.org/T136328#2332049 (awight) [21:16:47] and... since grrit was away: https://gerrit.wikimedia.org/r/291105 Make locale staging generic [21:17:25] those css selectors to detect rtl should be fine with full locales too. in fact, that comparison operator is specifically for that case! [21:19:33] (CR) Awight: [C: 2] "Great, thank you! Too bad that PayPal assumes a national language, but I'm still comfortable with a first iteration that lets them stink " [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/291105 (owner: Ejegg) [21:20:14] I haven't seen these selectors yet, sounds funky! [21:20:38] Wow I was so far off about the collision math... [21:21:02] awight: ohh, I see your locale FIXME in the recurring patch. so, that's really annoying [21:21:02] 1 - answer [21:21:20] yah. Your patch doesn't make it any worse though [21:21:20] hehe, even less cause for alarm [21:21:53] adyen still delivers french for fr_US [21:22:10] That... hey that makes sense [21:22:16] they just measure the field heights in inches instead of cm [21:22:25] Did you see PayPal's "legacy" locale logic? Even more insane. [21:22:25] (Merged) jenkins-bot: PayPal Express Checkout: support locale well enough to get ja_JP [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/287157 (https://phabricator.wikimedia.org/T131811) (owner: Awight) [21:23:16] i remember averting my eyes from that [21:23:16] Sometimes a country is a language, but sometimes there is more than one language in a country. You have to already know this lookup table in order to send them the correct locale param. [21:23:43] fr_DE needs to be mapped to fr_FR, but en_DE will work. Something like that [21:23:49] blecch [21:24:25] At least the docs weren't overly clear about anything, so there was still a challenge involved [21:24:56] sorry, https://gerrit.wikimedia.org/r/#/c/287036/8 needs a manual rebase now... [21:25:01] Hey ahhh, anyone feel like babysitting a minor CN patch during SWAT in an hour and a half? [21:25:11] k I'll do that rebase [21:25:13] which one? [21:25:25] the a/b test api? [21:25:33] This one especially, yeah. https://phabricator.wikimedia.org/T134286 [21:25:41] Cos the-wub wants to be able to experiment with it [21:25:54] oh yeah, that looks pretty harmless. [21:26:30] I can prepare the mw-core patches and stuff, but need to watch some kids murder a numeric piñata early tonight [21:27:22] oh, thanks! [21:27:30] you sure? Thank you! [21:27:53] yeah, I think I remember how to fiddle with main cluster if need be [21:28:10] I'll just call up that 'how to deploy' url and refresh my memory [21:28:13] It should be pretty hands-off actually [21:28:50] SWAT deploys are more like, be present on #wikimedia-operations and be prepared to confirm that the feature isn't breaking anything [21:29:05] There will be person on the standing team doing the deploy [21:29:51] ejegg: I can prepare a banner that exercises the feature, too... [21:31:39] (CR) Ejegg: "Yeah, I shuddered to learn the wide variety of errors that 'IGNORE' lets slip. Thanks for squawking, python!" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/289592 (owner: Ejegg) [21:32:51] awight: cool, in the ops channel. if you wanna prep a banner too, that'd be awesome! [21:33:57] awight: Meetingtime? [21:34:07] I mean, you look busy. [21:35:31] ejegg: i'm confused about https://phabricator.wikimedia.org/T129935 -- the donor details don't get removed from the pending queue on capture? [21:36:19] cwd i forgot a step.. i think we mark the message as 'capture attempted' [21:36:25] and plop it back in pending [21:36:46] then when we get the 'capture succeeded' message, that's when we transfer from pending to verified [21:37:08] shouldn't the capture job be smart enough to ignore both those things? [21:38:14] cwd it does check the 'capture attempted' flag if it finds donor data [21:38:34] and that's a definite duplicate, so it still cancels in that case [21:39:05] I just changed the action for missing donor data and updated some comments [21:39:51] oh i see [21:40:05] the steps are split up like that so we can correctly record 'capture successful' messages that come from donor services manually settling at the console [21:40:07] but this is addressing the case where it does not find donor data in pending [21:40:35] yeah, it was a mistake to assume that's always a duplicate [21:40:50] oh hey, actually.... [21:40:52] could be a queue failure, or even excessive lag? [21:41:25] lag shouldn't be a factor... we send the pending message before we show the donor the CC iframe [21:41:44] http://arstechnica.com/tech-policy/2016/05/google-wins-trial-against-oracle-as-jury-finds-android-is-fair-use/ [21:41:49] but I was about to say, we could leave the pending message [21:41:57] cwd holy crap, that's awesome news! [21:42:01] yeah! [21:42:22] it's a sad competition when i'm rooting for google [21:42:24] but that was one for sure [21:42:30] * ejegg has faith in the american jury pool slightly renewed [21:44:16] it sounds like we are counting on that pending message being there [21:45:07] so i wonder if it's worth trying to salvage the situation where it's not? [21:45:14] or just show an error [21:46:40] cwd: I think we do log a warning [21:47:26] it seems unwise to send someone off to adyen if we know there's going to be a problem when they get back [21:48:01] cwd this smashpig logic is all after they've submitted their cc info [21:48:17] oh, you mean if sending the queue message fails... [21:48:29] yeah [21:48:42] right, I guess we made queue failure non-fatal a while back [21:49:05] because some people would come back from successful donations and get a fail page if the queue was down [21:49:13] then they'd try to donate again [21:49:23] yeah [21:49:31] that makes sense [21:49:39] but in this situation we can stop it before anything goes awry [21:49:53] sure, we could... [21:50:05] if we want to [21:50:29] but we'll still be able to process the donation from the audit and get donor data from logs [21:50:47] so I'd say let 'em donate, and it'll just take an extra day to thank them [21:51:26] yeah [21:51:33] i wonder if we could do it in a less ambiguous way [21:51:42] i.e. missing data could mean a or b [21:52:22] if there was something persistent we could flip that meant "couldn't schedule pending message" [21:52:35] (Merged) jenkins-bot: Make locale staging generic [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/291105 (owner: Ejegg) [21:52:38] cause that's a different situation than "double clicked donate" [21:52:54] cwd sounds like either a backup queue or db writes from the frontend [21:53:38] I think it's infrequent enough that we're not overburdening donor services [21:54:00] cwd i've got it... smashpig checks ganglia! [21:54:12] hehe [21:54:13] heheh [21:54:47] i'll counter that if it's sufficiently rare maybe we can just show them a please try again later? [21:55:02] if the queue is down it might be safest to hit the halt button [21:55:04] anyway, we can write some maintenance scripts to batch capture if it comes to that [21:55:28] cwd true.... lemme see what kind of complexity that adds in DI [21:55:58] my hope is that it would _remove_ complexity of dealing with an ambiguous situation [21:56:16] guessing there'll be an exception if the enqueue fails? [21:56:19] hmm, i'd still not want to reject ppl for half an hour in the heat of big english [21:57:21] yeah it would suck, but the busier the time the worse the cleanup will be too [21:57:57] if we spend our time replacing the queue with a better one we can kill all the birds :) [21:58:08] yeah [22:01:21] though that actually will introduce lag into the possibilities for missing donor data [22:02:00] how do you mean? [22:02:05] i.e., the job dumping pending queue to pending db is slower than the donor entering their card info [22:02:20] so we get an auth ipn and don't see donor data in the db [22:02:26] do we then check the queue too? [22:02:42] hmm, pending db? [22:02:42] no, we'd want to go in the opposite order for race conditions [22:03:11] cwd the idea is everything that reads from pending now should read from a db [22:03:28] like mysql? [22:03:35] err...exactly mysql? [22:03:59] frontend still dumps into a queue, but it's just consumed by something that populates the db [22:04:08] cwd pgsql? [22:04:28] hrm, seems like that introduces race conditions everywhere [22:04:30] dstrine: Sorry, trying to get a thing ready by 4pm but I'll jump into the test meeting when I'm done [22:04:49] what's the rationale for that? [22:05:06] uhhh... it made so much sense once upon a time [22:06:12] oh, 'cause we're looking for messages by id [22:06:21] treating the pending queue like a key/value store [22:06:43] and supporting that operation limits our choice of replacement queues [22:07:03] ah right [22:07:23] but that shoudl be fine in redis right? [22:07:29] yeah... [22:07:33] it's persistent and reasonably queryable [22:07:43] i guess we came up with this when kafka was still on the table [22:08:12] if we have utility in looking things up by ID i think we should use something that leaves it possible [22:08:34] the only reason i see for using something that has no concept of that is sheer speed [22:08:42] which we don't need [22:08:46] not that kind of speed [22:09:08] a lot of pure "queues" have no visibility into what's waiting [22:09:42] which is fine for a huge dump of atomic page view data for instance [22:09:59] but sucks for relational data, which as of now ours is [22:10:09] yeah. i still like the db [22:10:32] yeah [22:10:38] but i would be fine with redis too [22:11:08] the lack of relational features is probably a good safety belt to keep us from building it out :P [22:11:36] cwd the pending schema so far is just one table [22:13:06] does it use ct_id as the key? [22:14:11] cwd has a few keys - gateway, order_id, gateway_txn_id [22:16:21] ejegg: you can find your way back to CT from there though? or is it siloed off? [22:16:21] It's true that we don't need a pending queue if we can set up payments boxes to be write-only to the payments db, but that seems tricky. [22:16:46] Not tricky at a permissions level, but how do you avoid overwriting a record / handle duplicates, for instance. [22:17:31] https://github.com/wikimedia/wikimedia-fundraising-SmashPig/blob/c3c0100bf2db994479c0901fb4c7f4b7ffc4419a/Schema/001_CreatePendingTable.sql [22:18:26] cwd i think ct_id is part of the message, which is in a json text field [22:19:07] hmm, lappy needs a charge. back soon [22:19:26] isn't ct_id something that will be unique across all gateways? [22:20:52] Damn, every time... I need to hack GeoIP so I can run CentralNotice locally... [22:21:07] i don't understand why we want to write from a pending queue to a pending db [22:21:52] why not just have a pending db on the front end? [22:21:54] cwd: The pending queue is only for the frontend to be decoupled from anything else [22:22:17] but something still has to read from unpriv and write to priv in either case [22:22:18] The frontend will never (is not allowed to by PCI) to read from it [22:22:57] It doesn't have to be a "queue", just a pipeline that causes new pending things to vacuum tube out of the frontend and into our database. It could be synchronous I suppose, but no reason to do that. [22:23:23] yeah [22:23:33] (PS1) Awight: ext.centralNotice.display: API for registering tests [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/291118 (https://phabricator.wikimedia.org/T134286) [22:23:37] isn't that what we have now? [22:23:48] just need something that falls over less than activemq [22:24:02] (CR) Awight: [C: 2] "Cherry-picked over a lot of changes, but this still seems to work." [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/291118 (https://phabricator.wikimedia.org/T134286) (owner: Awight) [22:24:36] cwd: No, we have 10 or so various pending things, none of which is a database [22:25:21] Thank you for asking these questions btw! All the queue ideas need lots of review, cos this is the bed we're going to be shitting in for the next few years... [22:25:54] hehe, my pleasure [22:27:53] i feel like this should definitely be doable without a job copying a front end queue to a back end db [22:28:48] but in any case i would vote for considering a missing pending messages to be bad data [22:28:57] We have one of those for everything else, though. What's the issue, the race condition? [22:29:09] that is what ejegg mentioned [22:29:19] Also: how else can could we implement this? [22:29:43] AFAICT, it's either a cronjob or an event listener [22:30:03] hmm, i think we are talking about a few different things [22:30:45] ejegg|afk: Okay we're on the schedule. https://wikitech.wikimedia.org/wiki/Deployments#Week_of_May_23th [22:30:54] I'll comment on the task with the campaign and stuff [22:31:01] the point i'm trying to make is https://phabricator.wikimedia.org/T129935 might make more ambiguity [22:31:21] i think we shoudl focus on making the queue highly available rather than making the app cope with it being broken [22:31:34] and just display error page if the queue is gone [22:31:40] because it shouldn't be [22:32:33] (Merged) jenkins-bot: ext.centralNotice.display: API for registering tests [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/291118 (https://phabricator.wikimedia.org/T134286) (owner: Awight) [22:32:43] Fundraising Sprint Hermit Crab Husbandry, Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, and 4 others: SmashPig should read config from /etc - https://phabricator.wikimedia.org/T133601#2332308 (Jgreen) [22:32:45] Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, Fundraising-Backlog, FR-Smashpig, Technical-Debt: Remove redundant SmashPig config from /etc - https://phabricator.wikimedia.org/T136121#2332306 (Jgreen) Open>Resolved latest .yaml config successfully deployed, de... [22:33:38] awight: cool, lappy is charging [22:34:27] Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, Fundraising-Backlog, Patch-For-Review: CentralNotice: test registration - https://phabricator.wikimedia.org/T134286#2332319 (awight) Example campaign configuration: https://meta.wikimedia.org/w/index.php?title=Special:Central... [22:34:42] cwd I still don't like the queue needing to support key/value too [22:35:51] why use a queue at all? why not just a k/v store? [22:36:38] hmm [22:37:25] cwd: That sounds nice, I don't see how it's related to this bug though [22:37:49] agreed [22:38:19] i'm saying i think we are signing up for pain if we try to support donating while the queue is down [22:38:43] heh, thanks for shining a beacon back to task at hand awight [22:39:18] Well. One reason to not use k/v store or mysql directly is that we don't want our frontend to have access to data records for any other donors. It's not even allowed to by PCI. So the k/v store you're talking about is just a place where we point pending messages to disappear. [22:39:25] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops: Remove Engage Civi users and revoke SSL keys - https://phabricator.wikimedia.org/T114797#2332352 (Jgreen) a:Jgreen>awight Adam I'm not sure what to do, can you clarify whether we're removing some or all Engage users? [22:39:47] At that point, we can make it a function that soes something slightly risky like a write-only user can splat rows into a database, but there are issues [22:40:05] * if the order ID already exists and is a primary key, we don't want to blindly overwrite rows. [22:40:49] * For a possibly small price (slight delay before pending messages hit our db) we can eliminate a coupled component which would bring down the frontend if we need to do maintenance [22:41:03] s/possibly/arguably/ [22:41:11] awight: going through phabricator tasks...is T129706 still a thing, or did we bypass that question with the yaml thing? [22:41:11] T129706: Add SmashPig config to settings repo - https://phabricator.wikimedia.org/T129706 [22:41:34] stashbot++ [22:42:03] ejegg: I don't know why, but the test banner is not displaying yet: https://aa.wikibooks.org/wiki/Main_Page [22:42:16] * awight high-fives bd808 for stashbot [22:42:35] Jeff_Green: I'd say that's still a wish [22:43:02] The files are currently split up like * defaults vs * semi-sensitive WMF overrides [22:43:28] I'd rather make it look like * defaults * fr-dev overrides like fraud scores * fr-tech-ops overrides like connection credentials [22:43:33] ok. maybe instead of ticketing specifics, we should have a serious breakout session on overall configuration [22:43:35] yeah [22:43:41] Any time! [22:44:00] also: this is a good time of year to review that stuff, thank you [22:44:06] are you going to wikimania? [22:44:13] yup! [22:44:38] how about we take a break at some point with as many people who are interested and there, and map it out? [22:45:05] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops: Remove Engage Civi users and revoke SSL keys - https://phabricator.wikimedia.org/T114797#2332369 (awight) Oops! This is blocked on changing the way we do imports. Please don't take any action yet. When the time comes, it wil... [22:45:33] Jeff_Green: sounds good, it's probably an hour or two [22:45:44] yawp. cool [22:47:10] ejegg: what about the duplicate merchant ref? would that be something to key off to signify invalid? [22:48:20] ejegg: Okay, the banner looks good. If you load https://aa.wikibooks.org/wiki/Main_Page you should get a console error about registerTest. [22:48:52] When the SWAT deployment goes out, that error will go away and you should see that the S:RI request has a testIdentifiers param with some value. [22:49:23] ejegg: & you got my note about SWAT right? You don't have any deployment steps, they just want an owner around to watch for smoke [22:49:59] Fundraising Sprint Ghostbusting , Fundraising Sprint Hermit Crab Husbandry, Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, and 3 others: Only subscribe primary emails, secondary addresses should be suppressed - https://phabricator.wikimedia.org/T131979#2332389 (Ejegg... [22:50:18] awight: yep, thnks! [22:52:22] cwd: right, this is about how we detect duplicate merchant refs [22:53:18] the consumer can't ask the db if one has already been captured? [22:54:13] cwd another race condition, since the thing that captures doesn't insert directly into civi [22:54:34] oh, but with the pending db, yes, that's what we would do [22:54:49] actually, we do that with the pending queue now [22:55:12] we could leave things in the pending queue after we send 'em to completed [22:55:21] but I'm wary of adding to queue bloat [22:55:33] Off-topic, just thinking about the race condition--I wouldn't want to count on the pending db to protect us from duplicate charges. That should happen at an earlier step, like session-stored fraud scores. Don't take any more steps to charge things once a transaction has been finalized on this session. [22:55:54] session? [22:56:09] This is to keep people from making duplicate charges from the frontend [22:56:35] ah--i guess the issue is that Adyen isn't debounced, right [22:56:39] this is probably pretty particular to the adyen case also [22:56:41] yeah [22:56:41] so my idea won't work [22:56:56] we should try to get them to not do that [22:57:01] sigh [22:57:07] awight: yeah, i think somehow they let ppl auth multiple times against one iframe post [22:57:16] That's sucky [22:57:35] so uh... yeah we can use the pending db to debounce [22:57:58] we're reading and writing directly to the db from the listener that finishes or cancels the transaction [22:58:21] Only the frontend might use a queue to buffer pending [22:58:56] ah, ok [22:59:00] that seems sensible [22:59:15] long as the front end never looks at the queue [22:59:38] well...there's that race condition though [22:59:51] which? [23:00:06] if the IPN message came in before the message had been popped to the db [23:00:26] Those don't seem like a race [23:00:26] unless the listener also scheduled a message [23:00:43] instead of trying to capture itself [23:01:14] the listener will be atomic by doing { write to pending db that I'm authorizing this shit; send auth API call to processor ; record result in pending db } [23:01:44] the frontend can't do anything to hurt those steps, I believe [23:03:01] if the capture is in the same FIFO as the authorize then it works [23:03:03] is that the plan? [23:03:13] cwd the listener does schedule a job [23:03:42] so the capture job will *always* get scheduled after the auth? [23:03:57] so we can play with that delay if we see a lot of lag in the pending queue->db job [23:04:02] cwd yeah [23:04:19] ok right on, i understand the utility of FIFO on the front end in that case [23:05:50] so in this universe, a missing pending message would mean breakage? [23:06:17] cwd just means no automatic capture [23:06:38] but what would be the legitimate situation where there isn't one? [23:07:19] not sure, but we can't anticipate everything [23:07:35] just trying not to be too drastic in ambiguous situations [23:08:00] well, it's only ambiguous if we code it that way [23:08:37] Gotta run, sorry to leave this conversation mid-stream! [23:08:46] i mean, maybe someone misconfigured the expiry job and deleted too many pending messages [23:09:06] tons of random mishaps possible [23:09:44] if you except those things as hidden inputs to your code then there are infinite possibilities [23:10:04] hehe [23:10:12] right, so we don't want to assume we know what missing donor info means [23:10:35] well, it's our duty to decide what it means [23:12:45] i think this is the sort of thing functional programming is trying to describe, but gets all mixed up with math and misinterpreted [23:13:11] but at a minimum if your functions are dealing with things that are not explicit inputs and outputs...you get roaches [23:14:39] so we can decide what missing info means and execute that procedure, or we can say we don't know what it means and start covering cases [23:15:09] and each one is a hidden input to the function that can go off in a crazy direction [23:15:24] i'm using the word function loosely here [23:16:07] hey does anyone have the link to that Job description that was shared to us - I want to forward it to someone & can;t find it [23:16:13] (PS1) Ejegg: Keep Civi users off suppression list [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/291136 (https://phabricator.wikimedia.org/T122411) [23:16:46] cwd or we could say we're not sure and leave it for a human to review [23:17:40] the patch in CR now is just acknowledging our uncertainty [23:18:38] yeah, i'm off in the weeds [23:19:29] (CR) Cdentinger: [C: 2] Don't cancel payments missing donor details [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/290980 (https://phabricator.wikimedia.org/T136038) (owner: Ejegg) [23:19:32] gotta go pack! [23:19:41] good luck! [23:27:10] (Merged) jenkins-bot: Don't cancel payments missing donor details [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/290980 (https://phabricator.wikimedia.org/T136038) (owner: Ejegg) [23:37:09] (CR) Legoktm: [C: 2] Minor clean up in CNChoiceDataResourceLoaderModule [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/290824 (owner: Krinkle) [23:39:18] (Merged) jenkins-bot: Minor clean up in CNChoiceDataResourceLoaderModule [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/290824 (owner: Krinkle) [23:40:47] (PS1) Ejegg: Check in php-queue lib, not submodule [wikimedia/fundraising/SmashPig/vendor] - https://gerrit.wikimedia.org/r/291139 [23:41:25] (CR) Ejegg: [C: 2 V: 2] "self-merging lib fix" [wikimedia/fundraising/SmashPig/vendor] - https://gerrit.wikimedia.org/r/291139 (owner: Ejegg) [23:49:08] (PS1) Ejegg: Get fix in vendor submodule [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/291141 [23:49:35] (CR) Ejegg: [C: 2] "self-merging lib fix" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/291141 (owner: Ejegg) [23:54:22] !log updated SmashPig from f0bf4385afac65a27f99c5f657c3d0931c991fa8 to 90757321a3bfa1045202e06e3dd1960a0043493a [23:54:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master