[00:03:45] awight: evaporate cooling is one of my favorite things about the southwest [00:05:37] swampfest [00:06:02] nothing but a pump and a fan [00:06:30] might cost $100 if you ran it for a solid year [00:08:06] Too bad it doesn't work as well in already humid environments [00:09:35] (PS3) Eileen: Add 'Endowment Gift' to Financial types [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299861 (https://phabricator.wikimedia.org/T139629) [00:11:28] yeah they don't do shit unless it's pretty dry [00:11:32] but it's always pretty dry here [00:11:53] humidity 21% [00:13:51] ouch! [00:15:50] Don't run out of gas [00:20:23] awight: feeling pretty good about the redis stuff, how about you? [00:20:56] It's a bold start [00:21:08] I think we're in a great place with the other debt, though [00:22:58] yes [00:23:06] great strides [00:23:46] What in particular makes you optimistic about the queue stuff? [00:24:23] ah i guess i just understand the situation better [00:24:29] doesn't seem as daunting [00:25:30] hehe excellent [00:25:36] I'm barely holding onto the threads [00:26:49] (PS7) Awight: Pending database supports delete and fetch [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/299691 (https://phabricator.wikimedia.org/T131275) [00:27:20] (CR) Awight: "@ejegg: Thanks for noticing the order_id-centricity!" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/299691 (https://phabricator.wikimedia.org/T131275) (owner: Awight) [00:29:33] cwd: Are you working on this trip? [00:29:56] It'd be great to bounce idea during tomorrow's tech time, if so [00:30:00] *ideas [00:30:23] yep i'm taking off friday and monday but will be around besides that [00:30:44] cool. [00:30:46] friday is the conference and monday we are babysitting for dani's sister [00:30:51] ah, I should write a note about some halfdayness, too... [00:31:07] (CR) XenoRyet: [C: 2] Pending database supports delete and fetch [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/299691 (https://phabricator.wikimedia.org/T131275) (owner: Awight) [00:31:57] XenoRyet: thanks! [00:32:05] No worries [00:32:15] (Merged) jenkins-bot: Pending database supports delete and fetch [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/299691 (https://phabricator.wikimedia.org/T131275) (owner: Awight) [00:32:22] I like writing tests when they make the patch more palatable... [00:32:39] & it's hilarious how many bugs I found while going from WIP to tested... [00:32:49] maybe 1 per 5 lines of code [00:32:54] Nice [00:33:54] That seems like about the ratio I usually get on early tries. [00:35:01] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, fundraising-tech-ops: Migrate paymentswiki database to 1.27 schema - https://phabricator.wikimedia.org/T140855#2478218 (awight) [00:36:36] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, fundraising-tech-ops: Change paymentswiki to deploy from `fundraising/REL1_27` - https://phabricator.wikimedia.org/T140856#2478233 (awight) [00:37:33] I swear I'm sometimes emitting more bugs than code [00:39:05] isn't it crazy to imagine writing code for the mars rover? [00:39:08] one opportunity [00:39:17] lots of simulation... [00:39:46] I don't really believe that unit testing finds many of the bugs, it seems to only do regressions [00:41:24] yeah, that's useful though [00:41:35] totally! [00:41:38] even though i am the worst at writing tests [00:50:07] (PS1) Awight: Depend on SmashPig; use looser caret versioning [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299913 [00:50:34] cwd: ^ donno if that helps you with the donationqueue module [00:52:02] awight: what does the caret do? [00:52:50] that's just a semantic versioning thing where "^1.1.2" is the same as >= 1.1.2 <= 1.999.999 [00:53:11] that part won't help ;) [00:53:14] ah gotcha [00:53:38] oh, example of DI depending on another of our libs? [00:54:05] or even SP itself, if you decide the new classes can live there for now. [00:54:15] we already have a bunch of machinery for this inside of PS [00:54:16] SP [00:54:24] sure, works for me [00:54:41] depends how ambitious you're feeling... [00:54:43] i will try to see where DQ can fit in to SP? [00:54:53] sounds perfect [00:55:41] (PS1) Awight: Create pending_db_orphan_rectifier branch for forked deployment [extensions/DonationInterface] (pending_db_orphan_rectifier) - https://gerrit.wikimedia.org/r/299914 [00:56:05] cool [00:56:30] well i'm gonna head to the airport and set up shop there for awhile [00:56:38] fly safe! [00:56:43] thanks! [00:56:49] hi five on the 1.27 oc [00:56:56] o/ [01:01:46] (PS1) Awight: [WIP] Convert orphan rectifier to use the PendingDatabase [extensions/DonationInterface] (pending_db_orphan_rectifier) - https://gerrit.wikimedia.org/r/299915 (https://phabricator.wikimedia.org/T131275) [01:03:27] (CR) jenkins-bot: [V: -1] [WIP] Convert orphan rectifier to use the PendingDatabase [extensions/DonationInterface] (pending_db_orphan_rectifier) - https://gerrit.wikimedia.org/r/299915 (https://phabricator.wikimedia.org/T131275) (owner: Awight) [02:38:19] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, fundraising-tech-ops, Schema-change: Migrate paymentswiki database to 1.27 schema - https://phabricator.wikimedia.org/T140855#2478550 (Danny_B) [02:39:38] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Epic: [Epic] Move payments to mediawiki 1.27 - https://phabricator.wikimedia.org/T139531#2478552 (Danny_B) What is the difference between this task and {T138016}, please? [02:39:54] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Patch-For-Review: Move payments to MediaWiki 1.27 - https://phabricator.wikimedia.org/T138016#2386861 (Danny_B) What is the difference between this task and {T139531}, please? [03:43:02] (PS1) Ejegg: Update SmashPig and config handling [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299936 (https://phabricator.wikimedia.org/T140814) [03:46:40] (PS2) Ejegg: Compare pending DB row with queue message (SEE NOTES) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299816 (https://phabricator.wikimedia.org/T140484) [03:47:04] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Patch-For-Review: Move payments to MediaWiki 1.27 - https://phabricator.wikimedia.org/T138016#2478594 (cwdent) [03:47:06] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Epic: [Epic] Move payments to mediawiki 1.27 - https://phabricator.wikimedia.org/T139531#2478596 (cwdent) [03:47:21] (CR) Ejegg: "PS2: rebase onto SmashPig update to get PendingDatabase class" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299816 (https://phabricator.wikimedia.org/T140484) (owner: Ejegg) [03:47:24] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Patch-For-Review: Move payments to MediaWiki 1.27 - https://phabricator.wikimedia.org/T138016#2386861 (cwdent) @Danny_B nothing, thanks! [03:57:38] (PS3) Ejegg: Compare pending DB row with queue message (SEE NOTES) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299816 (https://phabricator.wikimedia.org/T140484) [04:11:33] Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, Fundraising Sprint Licking Cookies, and 5 others: Move queue2civicrm_generate_message to test support - https://phabricator.wikimedia.org/T133712#2478600 (Ejegg) Open>Resolved [04:23:54] (PS4) Ejegg: Compare pending DB row with queue message (SEE NOTES) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299816 (https://phabricator.wikimedia.org/T140484) [04:24:29] (PS2) Ejegg: Update SmashPig and config handling [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299936 (https://phabricator.wikimedia.org/T140814) [04:28:25] (PS1) Ejegg: DO NOT MERGE: import info from pending DB, not AMQ [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299938 (https://phabricator.wikimedia.org/T122641) [04:43:19] (PS1) Ejegg: Remove pre-1.27 compatibility shims [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299939 (https://phabricator.wikimedia.org/T138016) [04:44:59] (CR) jenkins-bot: [V: -1] Remove pre-1.27 compatibility shims [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299939 (https://phabricator.wikimedia.org/T138016) (owner: Ejegg) [04:52:15] (CR) Ejegg: "recheck me after I73f01f87b06 gets rid of 1_25 tests" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299939 (https://phabricator.wikimedia.org/T138016) (owner: Ejegg) [04:57:15] hey, are we not using ContributionReporting anymore? [10:27:23] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Epic: [Epic] Move payments to mediawiki 1.27 - https://phabricator.wikimedia.org/T139531#2479367 (Danny_B) [10:27:25] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Patch-For-Review: Move payments to MediaWiki 1.27 - https://phabricator.wikimedia.org/T138016#2479368 (Danny_B) [10:27:27] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Patch-For-Review: Make RapidHTML work with mediawiki 1.26+ - https://phabricator.wikimedia.org/T139551#2479366 (Danny_B) [10:27:39] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, fundraising-tech-ops, Schema-change: Migrate paymentswiki database to 1.27 schema - https://phabricator.wikimedia.org/T140855#2479369 (Danny_B) [10:27:41] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Epic: [Epic] Move payments to mediawiki 1.27 - https://phabricator.wikimedia.org/T139531#2435061 (Danny_B) [10:27:43] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Patch-For-Review: Move payments to MediaWiki 1.27 - https://phabricator.wikimedia.org/T138016#2386861 (Danny_B) [10:27:53] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Epic: [Epic] Move payments to mediawiki 1.27 - https://phabricator.wikimedia.org/T139531#2435061 (Danny_B) [10:27:55] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, fundraising-tech-ops: Change paymentswiki to deploy from `fundraising/REL1_27` - https://phabricator.wikimedia.org/T140856#2479372 (Danny_B) [10:27:57] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Patch-For-Review: Move payments to MediaWiki 1.27 - https://phabricator.wikimedia.org/T138016#2386861 (Danny_B) [11:17:17] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Spike: Spike: Create an alert for suspicious dips in banner impressions - https://phabricator.wikimedia.org/T116047#1739102 (Pcoombe) I now have a cron job which emails me if there were no entries in pgehres.bannerimpressions in the last hour (... [14:12:36] aaarg aaarg arrg!!!! [14:23:22] aarrg arg? [14:23:46] * AndyRussG 's transformation is nearly complete! [14:41:56] Fundraising-Backlog, fundraising-tech-ops, Operations, Patch-For-Review: Allow Fundraising to A/B test wikipedia.org as send domain - https://phabricator.wikimedia.org/T135410#2480166 (Jgreen) DNS changes are deployed, let's leave this ticket open until the fundraising mailing tests are complete... [14:42:14] Fundraising-Backlog, fundraising-tech-ops, Operations, Patch-For-Review: Allow Fundraising to A/B test wikipedia.org as send domain - https://phabricator.wikimedia.org/T135410#2480168 (Jgreen) p:High>Low [14:48:50] Anyone been stuck on this one at all? Could not find gem 'mediawiki-vagrant (= 0.14.0)' in any of the gem sources listed in your Gemfile or available on this machine. [15:01:22] AndyRussG: uuugh yeah it's a bug in vagrant [15:01:37] well, incompatibility between vagrant and bundler. try to upgrade vagrant? [15:01:56] cwd: hmmm k! Yeah I just now got a response in #wikimedia-dev [15:06:02] cwd: hmmm I'm already on vagrant 1.8.4+dfsg-1 [15:07:11] crap, that's what ejegg said too. upgrading fixed it for me. let me find the thread... [15:07:24] AndyRussG: https://github.com/mitchellh/vagrant/issues/7493 [15:07:42] the gem is local, in lib/ [15:08:11] AndyRussG cwd I think I was getting another error [15:08:15] lemme try again [15:10:57] cwd I'm pretty inexperienced with ruby - lib/mediawiki-vagrant is a dir full of .rb files and other dirs. That's a gem? [15:12:25] ejegg: i am too :P but i learned from a friend that the whole repo is a gem [15:12:31] oh hey, just found it in .vagrant.d/gems/cache/mediawiki-vagrant-0.14.0.gem [15:13:49] so... Could not find gem 'mediawiki-vagrant (= 0.14.0)' in any of the gem sources listed in your Gemfile or available on this machine [15:13:57] any idea where I should copy it to? [15:14:32] looking up docs on gem sources... i see source 'https://rubygems.org' at the top of the Gemfile [15:14:56] so i don't think it's an issue of where is it but that bundler can't include a local gem [15:15:11] (the ruby community seems to be centralized around rubygems.org) [15:15:28] blarg! [15:15:31] The Vagrant from debian sid needs this patch -- https://patch-diff.githubusercontent.com/raw/mitchellh/vagrant/pull/7505.patch -- I've heard that stretch already has it [15:15:37] this bug does report it with 1.8.4: https://github.com/mitchellh/vagrant/issues/7493 [15:15:52] bd808: can confirm that latest stretch works [15:16:05] I think it depends on what platform you are on. 1.8.4 works fine on OSX [15:17:57] it must because i have the exact same version as AndyRussG [15:18:00] hmm, going to snag https://github.com/mitchellh/vagrant/pull/7505/ and see if that fixes it for me [15:19:14] ah yeah, i manually applied that [15:19:19] and then upgraded [15:19:24] i assumed it was overwritten but maybe not [15:19:31] would not apply automatically though [15:19:50] cwd: ejegg: bd808: hmm debian experimental doesn't have a newer version, but there's a 1.8.5 .deb available for download from the vagrant site [15:20:29] oh hey, I'll try that first [15:20:53] cwd: ejegg: bd808: yeah so far that one's working! [15:23:03] sweet, gem installed! [15:23:09] thanks AndyRussG ! [15:23:23] :) [15:24:26] k, gonna destroy and re-up with fresh git checkouts [15:25:26] hope you're not on mobile data [15:25:36] heh, not right now [15:32:32] i wonder why mine works [15:35:03] probably shouldn't ask [15:36:19] cwd: what OS are u on? [15:37:00] debian stretch [15:48:45] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Patch-For-Review: Move payments to MediaWiki 1.27 - https://phabricator.wikimedia.org/T138016#2480529 (Jgreen) [15:48:47] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, fundraising-tech-ops: Change paymentswiki to deploy from `fundraising/REL1_27` - https://phabricator.wikimedia.org/T140856#2480527 (Jgreen) Open>Resolved The deployment tool config change is done for projects payments-wiki, payments-wiki... [15:49:41] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, fundraising-tech-ops, Schema-change: Migrate paymentswiki database to 1.27 schema - https://phabricator.wikimedia.org/T140855#2480530 (Jgreen) p:Triage>Normal [15:50:35] well vagrant up appears to still be doing stuff... [15:51:07] the initial git clone of mediawiki is not a speedy thing [15:51:42] yurp! [15:52:29] Definitely not "stuck", no apparent error messages [15:54:57] i wish i could choose verbose mode mid process [15:58:13] heh good point! [15:58:21] Actually I spoke too soooon: [15:58:23] ==> default: Error: /usr/bin/git clone --recurse-submodules https://secure.phabricator.com/diffusion/PHU/libphutil.git /vagrant/srv/libphutil returned 128 instead of one of [0] [15:58:25] ==> default: Error: /Stage[main]/Arcanist/Git::Clone[libphutil]/Exec[git_clone_libphutil]/returns: change from notrun to 0 failed: /usr/bin/git clone --recurse-submodules https://secure.phabricator.com/diffusion/PHU/libphutil.git /vagrant/srv/libphutil returned 128 instead of one of [0] [15:58:34] Still seems to be chugging along nonetheless [15:58:59] Doesn't seem fatal [15:58:59] yeah i got a bunch of different errors [15:59:06] hummm [15:59:12] some of them went away when i tried again [15:59:24] it took a good 3 days to get it back working [15:59:25] yeah usually my first tactic is just to keep retrying vagrant up to see if it finishes up [15:59:35] there are still a few things that are broken [15:59:36] Ooof [15:59:38] I don't think that anything depends on libphutil directly so that probably won't break your build. [15:59:48] K yeah [15:59:59] failed git clones generally need to be fixed manually [16:00:15] fyi same for arcanist.git [16:01:37] weee! reached unread e-mails 0 on my Phab spam mailbox! [16:16:50] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Gift Source and Financial Type for Endowment Gifts - https://phabricator.wikimedia.org/T139629#2438173 (RLewis) @Eileenmcnaughton just to clarify, can Leanne and I add the gift source and financial type on staging... [16:22:01] (PS1) Awight: Use pop when we don't need transactionality [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300053 [16:23:05] (PS1) Awight: Reword docs; stop doing popAtomic(function () {}) [wikimedia/fundraising/php-queue] - https://gerrit.wikimedia.org/r/300054 [16:24:11] fr-tech: cwd: I'm seriously considering using SmashPig config from DonationInterface. Anyone care to throw their body into the gears? [16:24:45] In particular, I'm trying to write tests for the orphan rectifier, so I need some way of mocking the Ingenico API. SP's dependency injection should make that easy... [16:25:23] hrm, u all |away. I'll try again in a minute [16:27:00] Seddon: hi! Just wondering why WMFR is interested in tomorrow's payments outage... [16:28:16] I see that we're doing a FR pre-test today... [16:29:01] Awight: I've got the two mixed up. Apologies [16:29:35] k no worries at all, sometimes there are interesting skeletons behind these mysteries [16:29:50] awight: if you think the smashpig config format is good then i say use it everywhere! [16:30:50] cwd|afk: it's slightly >> $wg hell [16:31:25] I might separate dependency injection from the config... but not totally certain that makes sense. [16:31:42] The idea, I guess, is that we might want to use the injection but another config source [16:31:58] i'm still confused about the dep injection [16:32:14] it's a pretty trivial implementation, lemme drop a link here [16:32:46] https://github.com/wikimedia/wikimedia-fundraising-SmashPig/blob/master/Core/Configuration.php#L250-L274 [16:35:32] ack. nvm it doesn't decouple nicely from the config. [16:35:41] Mostly because of the "persistent" flag [16:35:49] awight: yeah, makes sense to me to keep 'em together [16:35:50] it's funny that in the extensive scaffolding there is no query builder [16:36:06] That's for the better, IMO [16:36:11] We just use PDO [16:36:16] yep [16:36:46] & not in a way that we would want to extend AFAICT [16:36:59] ack, i considered wrapping it for about 30 seconds [16:37:07] wrapping which? [16:37:10] PDO [16:37:26] D: I think i may have just done that in PendingDatabase [16:37:46] heh, that's a very specific adapter [16:38:02] yah. [16:38:15] but we sure as hell don't need a generic PdoDataStore [16:38:46] oh hey, Q: I realized that the return signature in the PEndingDb class is probably wrong. We should return json_decode(row.message) rather than the db row, eh? [16:39:04] awight: aren't you returning the set of rows even? [16:39:04] ejegg: ah right, point taken [16:39:10] ejegg: yep [16:39:20] also, we do want the row ID if we're going to need that for delete [16:39:25] dang. [16:39:32] good call [16:39:36] awkward [16:39:49] should we be able to delete by gateway and order id? [16:39:55] Actually, I like that better [16:40:15] really? I was pretty happy about being able to delete just one message related to a transaction [16:40:25] hmm, lemme think [16:40:27] The thought was that we'll eventually have several messages from different sources [16:40:39] which might be handled by different consumers [16:40:48] there /should/ only be one pending message per gateway/order id [16:41:28] with the queue, we've been using the first one that comes along and deleting all the matching ones when we're done [16:41:57] I guess we need to be careful about defining pending messages. So pending is just data about an incomplete transaction that we don't expect to get back from the PSP? [16:42:09] yep, exactly [16:42:38] things that are not "pending": incoming PSP status changes through the listener, refunds, audit... [16:42:41] k [16:43:00] then I'm on board with gw+order id being the unique key [16:43:04] i'd rather try to keep the pending db de-duped than make all the consumers deal with possible duplicates [16:43:51] my spidey-sense is that it's good to be robust to that [16:44:11] k, lemme rewrite this junk [16:44:32] ejegg: here's a fun one: https://gerrit.wikimedia.org/r/300053 [16:44:36] rewrite it to what now? [16:44:49] you mean the consumers should all have logic to decide which message to use in case of multiples? [16:45:14] cwd: db::fetch will return a single row, and db::delete will take the entire message as an argument, and will delete where gateway== and order_id== [16:45:23] ejegg: or at least not blow up [16:45:48] can probably make the producer not insert dupes [16:45:50] awight: return the decoded message? [16:46:03] ejegg: your clarification about "pending"ness was helpful though, maybe I can let go of the dupes boogeyman [16:46:07] ejegg: yah [16:46:12] cwd right, i was thinking merge the data if we get another pending message off the queue [16:46:21] on duplicate key update.... [16:46:54] ah, sure [16:47:00] i'm sure it's possible with redis also [16:47:04] ejegg: cool, that's a great place to do any possible monkey biz [16:47:16] cwd yeah, but things won't stay in redis more than a couple minutes [16:47:19] cwd: I think we're merging into mysql, in this case [16:47:34] popAtomic from redis and upsert into mysql [16:47:55] * awight warshes mouth with a gallon of jargon soap [16:49:02] (PS2) Awight: Use pop when we don't need transactionality [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300053 [16:59:05] awight / cwd any opinion what we call the redis pending queue? [16:59:34] i've got 'pending-atomic' in SmashPig, but that's pretty specific to how we're consuming it [16:59:36] "pending-new" IMO [16:59:54] also, "completed-new" "refund-new" etc [16:59:59] k [17:00:00] cos it's gonna get confusing... [17:02:59] few updates for payments vagrant: https://gerrit.wikimedia.org/r/300061 [17:03:11] rad [17:04:27] ejegg: 00:00:49.780 puppet/modules/payments/manifests/donation_interface.pp:60 WARNING: indentation of => is not properly aligned [17:04:34] oops, fixing [17:04:36] strict style lint = :( [17:05:37] k, indentation corrected [17:06:02] ejegg: yeah i used "pending-new" in the DI producer [17:06:09] but it's just config at this point [17:06:28] cool, I'll change the default in smashpig too, for consistency [17:06:55] err, maybe after awight's current thing gets merged [17:07:05] we want to move DonationQueue to SP? [17:07:14] so DI will call the set et al from there? [17:07:16] ejegg: don't worry about the conflict, I"m not in there [17:07:25] ah cool [17:07:26] that will require refactor++ [17:07:44] cwd: I would invest in that project :) [17:07:54] cwd yeah, hmm, does that give us anything in SmashPig right now? [17:08:28] ejegg: with the super simple version i made it all happens in DI [17:08:59] should be easy to port [17:09:08] (PS1) Awight: Change PendingDatabase signatures [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300063 (https://phabricator.wikimedia.org/T131275) [17:09:17] but i think it's worth talking about what smash pig provides for this problem [17:09:19] Sanity check would be appreciated. I didn't have to get all stdClass on it... [17:09:29] we don't seem to be using any of the stuff from the same namespace [17:09:37] What I like about stdClass is that we get compiler validation of keys [17:09:52] cwd: good question [17:10:14] awight: hmm, easier to merge arrays than stdClass though [17:10:55] the dep injection (if i understand it) seems to be more an artifact of the SP convention [17:10:55] it's crazy. lemme take that out. [17:11:03] but could be accomplished any number of ways [17:11:53] (CR) Ejegg: [C: 2] "ty for the sanity" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300053 (owner: Awight) [17:12:57] the smash pig style feels overcomplicated to me, but if you think it will scale better than a simpler approach, i won't argue [17:13:03] (Merged) jenkins-bot: Use pop when we don't need transactionality [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300053 (owner: Awight) [17:13:38] (PS2) Awight: Change PendingDatabase signatures [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300063 (https://phabricator.wikimedia.org/T131275) [17:14:20] cwd: Please do argue... & if you see opportunities for introducing simpler interfaces go for it [17:14:37] The SP config thing causes me to pull my hair out [17:15:13] It's really cool, in a way that only a proprietary configuration framework can be [17:15:43] right [17:15:53] (PS1) Ejegg: 'pending-new' instead of 'pending-atomic' [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300064 [17:15:55] smacks of "because we could" more than "because we should" to me [17:16:26] it gives us * declarative config that can include comments. I like that the most. [17:16:50] * ability to override config based on "view" thing. It's useful but I wish we didn't have it. [17:17:18] * dependency injection with fairly simple syntax. a win to have this at all, but sad that it's proprietary [17:17:40] * reference to other nodes. Horrible. I think I killed that recently by doing the same thing in yaml. [17:18:11] the dependency injection is mostly useful for tests, where it's extremely useful. [17:18:37] ah, like injecting test class rather than the real one? [17:18:44] we don't go around switching backends in production, like ever. but cool to have to keep this possibility in mind while writing features. [17:18:49] cwd: exactly. [17:19:06] https://github.com/wikimedia/wikimedia-fundraising-SmashPig/blob/master/Tests/data/config_queue.yaml [17:19:16] https://github.com/wikimedia/wikimedia-fundraising-SmashPig/blob/master/Tests/data/config_pending_db.yaml [17:19:26] here's a really good one... [17:19:35] https://github.com/wikimedia/wikimedia-fundraising-SmashPig/blob/master/PaymentProviders/Amazon/Tests/config_test.yaml [17:19:39] That's what I'm talkin' about [17:20:11] ^ that one is why I would use the SP DI from DonationInterface, so I can mock out the Ingenico adapter. [17:20:32] The orphan rectifier is overdue for tests. [17:22:38] since the pending queue doesn't implement any of the datastore classes [17:22:43] does it belong there? [17:22:55] good Q [17:23:02] or, pendingdb u mean [17:23:02] cwd oh yeah, take a look at how I use it in CRM [17:23:10] mm? [17:23:13] I wanna see [17:23:51] err yeah, but i don't think queue will either? [17:23:56] cwd https://gerrit.wikimedia.org/r/#/c/299936/2/sites/all/modules/wmf_audit/tests/AmazonAuditTest.php [17:24:23] dude. yuck [17:24:36] nice to reuse it, though [17:24:51] yeah, you can override with whatever path you want [17:25:12] hehe, I like that anything is fair game in Drupalland [17:25:15] might be nice to have a shortcut method on Context [17:25:30] good idea [17:25:33] Well, anything that's using SmashPig needs to initialize that stuff [17:26:03] real code will just say Context::init with defaults tho [17:26:26] hmm, still needs the 'view' for a lot of things [17:26:40] is that even useful? [17:27:07] view? [17:27:08] ... [17:27:11] oh hey, scrum of scrums coming up. any news from fr-tech besides our wins [17:27:12] jury's out. [17:27:14] ? [17:27:16] hehe [17:27:57] awight: you know, telling it which node to override defaults with, so you can redefine anything datastore or logging crap under the 'adyen' node for example [17:28:12] let's see how often we actually do that... [17:28:19] We might want to get ottomata interested in the banner history rewrite, but I don't think AndyRussG's on that this week (?) [17:28:42] banner history, not banner sequence? [17:28:48] Is ottomata interested in doing something on that? [17:28:54] ejegg: The one place I believe "config views" are helpful is when multiple SPs live on a single box. [17:29:19] ejegg: hmm maybe mention that we stopped loading CN RL modules for special pages and when there's an action=edit param [17:29:21] AndyRussG: he's the invaluable guy in the Kafkaesque castle [17:29:32] that's the override file, i'm talking about the overrides within a single file [17:29:33] Yeah! [17:30:22] ejegg: oh? I thought we were doing the single-file overrides, cos we can only deploy one SP.yaml file [17:30:46] awight: so, there's the default: key at the top of SmashPig.yaml that defines things like logging [17:30:59] yes. and then gateways override stuff [17:31:01] and then under the other top-level keys like adyen: or amazon: [17:31:09] those are the "config views" [17:31:16] right, but how often do we actually need that? [17:31:24] I think we currently use it everywhere. [17:31:31] lemme check what we're really overriding... [17:31:50] hmm, I see loggging/root-context and endpoints/listener [17:31:52] DonationInterface ::getGlobal btw [17:32:01] inspired all that nefariousness [17:32:03] nefarity [17:32:12] bah, also 'actions' [17:32:30] though by the time we get to 'actions' i'm pretty sure we're in processor-specific code [17:32:44] I can see we would want to override logging per gateway. but yeah, actions and endpoints seem unjustified. [17:32:52] what are the practical advantages of having globals but not using the global scope? [17:32:59] it seems like the same thing to me [17:33:02] but with a bunch of code [17:33:21] cwd: IMO it's that the shared config can be addressed as an object [17:33:25] ejegg: shouldn't be any problems with the CN RL changes ^ but in case anyone has unsolved mysteries, it might be a thread to follow... [17:33:44] cwd: so with this whole "view" thing, you can get two slightly different config objects if you wish [17:34:00] I don't think of it as globals, but as reference data... [17:34:58] ejegg|scrummy: https://phabricator.wikimedia.org/T139439 [17:35:00] cwd: Even if it does nothing for production code, it makes testing a lot nicer cos we haven't polluted the global namespace with hundreds of config variables that might be cross-contaminating test cases... [17:35:41] cwd: like how in DonationInterface, we're simulating OOP config with this ::getGlobal nonsense [17:35:49] Fundraising Sprint Licking Cookies, Fundraising Sprint Muggle Baiting, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Spike: Monitor deployment of: Suppress CentralNotice ResourceLoader modules on Special and action=edit pages - https://phabricator.wikimedia.org/T139439#2481114 (AndyRuss... [17:36:35] it makes it a lot nicer when testing, to ask a gateway for its configuration [17:37:36] cwd: To your point about Core/DataStores, I think the key-value store is going to go away entirely. [17:41:05] so for all the pending stuff, we basically need 2 backend configs (redis and mysql) and a test version for both. is that about right? [17:41:21] nice [17:41:48] Most components will only use one or the other of queue or db, of course [17:41:55] yeah [17:42:07] but if we were to package it up as a lib, would that be the extent of the necessary config? [17:42:09] I'd be really happy consistently get the backend via some SP thing... [17:42:24] sorry, or any library, not necessarily SP [17:42:57] donno. * backend configs * logging [17:43:24] logging seems like we could do the standard thing [17:43:25] I wish SP had PSR-4 logging, so we weren't coupling to a prioritary thing [17:43:30] what's the standard? [17:43:35] psr4 yeah [17:43:49] DI does psr4 but not SP IIRC [17:44:29] if DonationQueue was a lib it could do psr4 [17:44:36] I like [17:44:57] It wouldn't be hard to make SP psr4 [17:45:02] it just seems really easy to decouple from DI and SP at this point [17:45:16] and if the config is actually dead simple, maybe we don't need to overthink it [17:45:34] where do you keep the file? what format? is it separate from all the other component config? [17:46:44] i think it's separate? unless it's coupled in a way i don't know about [17:46:56] i like the symfony yaml thing because it's simple and standard [17:47:20] The symfony::configuration component? [17:47:37] well just what we used for gateway config in DI [17:47:49] ah, so a hand-rolled config class [17:48:08] heh doesn't sound so appealing when you say it like that [17:48:11] hehe [17:48:24] i want to emphasize that what i want to do is *not* make another framework [17:48:32] It also needs to do dependency injection, and pull from /etc/fundraising/*.yaml [17:48:37] that's what I'm saying... [17:48:40] but more see if these basic parts need a framework [17:48:55] or if they can just be a collection of standard tools in vendor and a little bit of calling code [17:49:15] want. [17:49:33] Fundraising-Backlog, fundraising-tech-ops, Operations, Patch-For-Review: Allow Fundraising to A/B test wikipedia.org as send domain - https://phabricator.wikimedia.org/T135410#2481184 (CCogdill_WMF) Thanks @Jgreen, everything looks good on our end. We're still pushing IBM for security specs. We... [17:49:52] What I'd really want is to reuse what*ever* config, logging, and dependency injection stuff we have, as long as it doesn't impose too many assumptions or glue [17:50:14] yeah agreed [17:50:14] Last thing I want to do is discourage your investigation of a better way, though [17:50:56] It sounds perfect if you want to pursue a nice way to librarize our stuff [17:51:14] this might be a good opportunity to try that anyway [17:51:23] I just have to question the assumption that not reusing existing working machinery is helpful [17:51:54] wee vagrant is working! [17:51:57] hi everyone, France test is ready to go in 10 minutes [17:51:59] woohoo! [17:52:03] double woohoo ;) [17:52:14] the-wub: cool! [17:52:18] let's say DQ is in a different repo and is just the files you put in SP, plus the redis adapter [17:52:51] it has a yaml config file that contains the datastores and a test version for each [17:52:55] https://etherpad.wikimedia.org/p/Fundraising_queue_modules [17:53:01] if you wish [17:53:47] results sheet and banner links are here, if anyone's curious https://docs.google.com/spreadsheets/d/17arGLTPupnrCmo7O0He38js-gmOoF3OY53sOwiAbAp0/edit#gid=966649769 [17:53:49] cwd: Does it have default config? Or just an example config and then production config in /etc? [17:54:19] donno! do you have an opinion? [17:55:29] I think we have to customize the config for any real deployment. e.g. redis password [17:56:11] oh, yeah definitely [17:56:28] ah so mock test queue...that's interesting [17:57:08] yeah anything that uses DonationQueue should also be able to use the mock queue in its *own* tests, i guess [17:59:30] Donno if that etherpad is helpful. [18:00:49] MBeat PPena awight dstrine : France test is up :) [18:00:59] the-wub whoooo [18:01:04] awight: definitely [18:01:06] oops! I'll start monitoring late [18:01:13] the-wub: cool [18:02:17] awight: cwd ejegg|adk fr-online was wondering if they could test japan at 4:30 PDT. would someone be able to watch that as well? [18:02:29] dstrine: sure thing, I can. [18:02:34] ok thanks [18:02:48] the-wub: ^^^^ looks like 4:30 is cool for japan too [18:02:53] thanks awight dstrine! [18:03:10] awight: mock queue seems easy, just a new class in newBackend [18:03:51] cwd: ah, so it would be a normal donationqueue but with a sqlite::memory backend? That works [18:04:21] Might be nice to pass the configuration module an array to avoid the useless config file [18:04:24] meh [18:04:27] awight: does it even need that? for the purpose of a test does it need to persist after php is done? [18:04:41] sqlite i mean [18:04:42] cwd: here's the thing that really trips me up, though... dependencies. (notating in the etherpad) [18:05:48] cwd: sqlite::memory is cleared after the test case is done [18:05:58] it does need to persist for the duration of the test, tho [18:06:05] so can't just be null... [18:06:22] cwd / awight : hijacked one of awight's vagrant patches: https://gerrit.wikimedia.org/r/293058 [18:06:26] the-wub: some successful donations are coming through [18:06:33] sure yeah, seems like a class that stashes an object would suffice [18:07:37] or even just returns true on backend operations. depends what the tests look like [18:08:09] I suppose a real mock queue would provide more instrumentation than just DonationQueue(sqlite::memory [18:09:01] i think this might be a good situation to write the test first [18:09:24] hehe faux real [18:10:05] and roll up the simplest fake backend that will work [18:10:30] done. [18:11:21] https://github.com/wikimedia/mediawiki-extensions-DonationInterface/blob/master/tests/includes/TestingQueue.php [18:11:38] awight: cwd: ejegg: is there a vagrant branch or anything where queue stuff is going, or not yet? [18:11:45] I don't like it though, cos it avoids all of the php-queue integration [18:11:59] AndyRussG: IMO master is a good place for that [18:12:28] AndyRussG: There are some patches going through CR already, too, e.g. https://gerrit.wikimedia.org/r/293058 [18:13:08] awight: https://github.com/CoderKungfu/php-queue/tree/master/test/PHPQueue/Backend [18:13:44] aww nuts, no sqlite backend [18:13:57] awight: PDO backend can do sqlite [18:14:03] yes! thx [18:14:18] I thought I remembered us using that somewhere [18:14:54] https://github.com/CoderKungfu/php-queue/blob/master/test/PHPQueue/Backend/PDOTest.php#L17 [18:14:57] huh? [18:15:18] That's unfortunate. [18:15:38] i was hoping i was missing something [18:15:47] hmm, "gas" "prom" and "snow" "den". I blame myself for this file. [18:16:41] that's the best part [18:16:55] hehe I actually did have a gas prom once [18:17:32] got too excited and drove away from the filling station with the whole assembly... [18:17:41] oh i thought it was a command [18:17:44] So I threw it in my trunk and continued to the Jr. prom [18:17:49] (CR) Ejegg: [C: 2] Add 'Endowment Gift' to Financial types [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299861 (https://phabricator.wikimedia.org/T139629) (owner: Eileen) [18:17:52] that car was reeking for months [18:18:03] oh man [18:18:07] gnarly [18:18:51] I think the situation with that PDOTest is that only mysql was supported when we got our hands on this library. [18:19:22] (PS6) Ejegg: Add the predis library [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/293249 (owner: Awight) [18:19:25] the weird thing to me is that it wants to actually connect to a database [18:19:31] We should rewrite the test to use sqlite::memory, and even better have test include which does all that transparently [18:19:34] right? [18:19:38] All those tests are like that. [18:20:07] (Merged) jenkins-bot: Add 'Endowment Gift' to Financial types [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299861 (https://phabricator.wikimedia.org/T139629) (owner: Eileen) [18:20:11] Believe it or not, this seems to be the best and only queue abstraction out there [18:20:19] (CR) Ejegg: [C: 2] Add the predis library [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/293249 (owner: Awight) [18:21:32] (Merged) jenkins-bot: Add the predis library [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/293249 (owner: Awight) [18:21:53] (PS3) Ejegg: Change PendingDatabase signatures [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300063 (https://phabricator.wikimedia.org/T131275) (owner: Awight) [18:22:46] (PS1) Awight: Implement fetch oldest [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300073 (https://phabricator.wikimedia.org/T131275) [18:23:01] (CR) Ejegg: [C: 2] "Thanks! Will definitely be more convenient to work with" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300063 (https://phabricator.wikimedia.org/T131275) (owner: Awight) [18:24:15] (Merged) jenkins-bot: Change PendingDatabase signatures [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300063 (https://phabricator.wikimedia.org/T131275) (owner: Awight) [18:24:16] *love* how fast the SmashPig test suite is. 32 tests in 0.5s [18:24:31] it should run every time I save a source file... [18:26:26] France looking good so far. about 21 ingenico donations, and 10 adyen (although we know those are delayed to show up) [18:26:48] and 111 paypal :) [18:27:14] awight: weren't you planning to use popAtomic for the orphan processor? [18:28:02] derp, never mind [18:28:24] that's just for the thing that moves redis->mysql yeah? [18:28:53] and the rectifier will work from mysql? [18:28:58] yeah, it is now [18:29:21] i guess popAtomic will still be good for the completed donations queue [18:30:00] I think it's the only valid way to pop [18:30:21] mirroring that to redis as well, i guess [18:33:18] (PS2) Ejegg: Implement fetch oldest [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300073 (https://phabricator.wikimedia.org/T131275) (owner: Awight) [18:33:28] (CR) Ejegg: [C: 2] Implement fetch oldest [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300073 (https://phabricator.wikimedia.org/T131275) (owner: Awight) [18:34:43] (Merged) jenkins-bot: Implement fetch oldest [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300073 (https://phabricator.wikimedia.org/T131275) (owner: Awight) [18:35:54] cwd: do this and its parent look good to you? https://gerrit.wikimedia.org/r/293059 [18:36:38] looking [18:37:05] awight: are you thinking of resurrecting the 'antimessage' for the frontend to tell the processor that something in its pending queue is all taken care of? [18:37:15] or would the orphan processor look directly in Civi? [18:37:33] Civi seems like the simplest [18:37:35] ejegg: definitely ask Civi [18:37:38] cool [18:38:26] we're already doing mysql queries and have Civi + wmf_civicrm modules yeah [18:39:12] ... I'm trying to get code coverage reports on scrutinizer [18:39:38] wait, smashpig has schema? [18:39:51] It does now ;) [18:40:05] ... another frameworky thing that we're only half-supporting [18:40:11] There's no automated installer, for example. [18:42:00] seems like that changes what smash pig is pretty significantly [18:42:14] +1 [18:42:32] so is pendingdatabase its own module, or part of donationqueue? [18:43:01] and, are we making donation-ey assumptions, or keeping it clean and generalized? [18:43:24] One of PHP-Queue's downfalls is coupling to a "job queue" / job running concept. [18:43:34] i think we should wrap it all up into one thing that is as donation-centric is as useful for us [18:43:36] is DonationQueue a plugin to PHP-Queue? [18:43:43] k [18:43:51] because really this is a bunch of standard operations, the only thing specific is the chain of events [18:44:02] So if it's donation-centric, will we ever use it outside of the SmashPig context? [18:44:21] fair point [18:44:23] (PS3) Ejegg: Update SmashPig and config handling [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299936 (https://phabricator.wikimedia.org/T140814) [18:44:41] Maybe we should start by encapsulating each module as a directory under SP. [18:45:14] I agree that "KeyedOpaqueDataStore" has no place being next to a PendingDatabase. [18:45:35] i do worry that it will get ants siting in that directory [18:45:50] (PS5) Ejegg: Compare pending DB row with queue message (SEE NOTES) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299816 (https://phabricator.wikimedia.org/T140484) [18:46:10] (CR) Ejegg: [C: -1] "Needs update for PendingDatabase signature change" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299816 (https://phabricator.wikimedia.org/T140484) (owner: Ejegg) [18:50:04] awight: cwd ejegg XenoRyet hey... any of you have production access to varnish text caches? I want to run a command like what was run here: https://phabricator.wikimedia.org/T132374#2229057 [18:50:15] (CR) Awight: [C: 2] Update SmashPig and config handling (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299936 (https://phabricator.wikimedia.org/T140814) (owner: Ejegg) [18:50:25] However our wikitech prod access policy sez access is not granted for single-use [18:50:27] AndyRussG: no, I don't [18:50:33] me neither [18:50:37] nor do I [18:50:39] hmmm K thx.... [18:50:40] awww [18:50:54] Also not me. [18:50:57] awwwww [18:50:59] K [18:51:08] Too bad it's not in the NCSA banner logs [18:51:49] National Strength and Conditioning Association? [18:52:17] Should I ask for access? I guess it might be useful to monitor our cookie evaporation over time, see if we ever need to do further cookieburning passes [18:52:22] hehe thanks, I'll remember that as the abbreviation I never knew [18:52:27] totally. [18:52:41] (Merged) jenkins-bot: Update SmashPig and config handling [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299936 (https://phabricator.wikimedia.org/T140814) (owner: Ejegg) [18:52:41] K will do :) [18:53:16] Yeah I guess a 1-time access-granting is less burdensome than 4 or 5 times asking other folks with access to run commands [18:53:52] It's expressly for the purpose of you drastically reducing future cookie volume, seems like a pretty appropriate reason to let you sit the tiger [18:55:02] (PS6) Ejegg: Compare pending DB row with queue message (SEE NOTES) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299816 (https://phabricator.wikimedia.org/T140484) [18:56:06] (CR) Ejegg: Update SmashPig and config handling (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299936 (https://phabricator.wikimedia.org/T140814) (owner: Ejegg) [18:56:40] select count(*) from fredge.payments_initial where date > '2016-07-20T17:00:00'; [18:56:43] > 1176 [18:56:44] the-wub: ^ [18:57:12] select count(*) from fredge.payments_initial where date > '2016-07-20T17:00:00' and country = 'FR'; [18:57:15] > 742 [18:59:36] awight: what is that number? attempted payments? [19:02:26] (PS2) Ejegg: DO NOT MERGE: import info from pending DB, not AMQ [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/299938 (https://phabricator.wikimedia.org/T122641) [19:04:43] the-wub: sorry--That was lacking a "payments_final_status" clause, gimme a minute... [19:06:49] Hmmm provisioning the fundraising role on vagrant is giving me ==> default: Error: /Stage[main]/Mediawiki/Exec[update_all_databases]: Failed to call refresh: /usr/local/bin/foreachwiki update.php --quick --doshared returned 1 instead of one of [0] [19:06:51] ==> default: Error: /Stage[main]/Mediawiki/Exec[update_all_databases]: /usr/local/bin/foreachwiki update.php --quick --doshared returned 1 instead of one of [0] [19:07:11] Also ==> default: Error: /usr/local/bin/drush exchange-rates-update returned 1 instead of one of [0] [19:07:24] the-wub: Maybe I don't know how to do this, but here's what I came up with: [19:07:27] select count(*) from fredge.payments_initial where date > '2016-07-20T17:00:00' and country = 'FR' and ((gateway='adyen' [19:07:30] and payments_final_status = 'pending') or (payments_final_status = 'complete')) and validation_action='process'; [19:07:33] 847 [19:07:57] If that's the right query, then the number is of initiated payments that we reasonably believe are successful. [19:08:12] Gotta drive, back soon! [19:08:22] AndyRussG: I don't know what that is, offhand! [19:08:33] awight: hmmm K thx [19:08:42] Yeah tried re-provisioning, still error [19:09:04] awight: that doesn't seem right. I only see 46 adyen donations in civi, in fact only 119 clicks (from contribution_tracking)! [19:09:09] ejegg: Whaddyu think about where DonationQueue should live? I think the options on the table are * as a module in SmashPig, or * as its own library. [19:09:26] the-wub: ooh, maybe we're quadruple-counting. Checking... [19:10:33] I don't see anything obviously repeating. [19:11:58] my only argument against smash pig is it doesn't seem to need any of the tooling [19:12:15] and the sub argument is that adding schema to smash pig feels muddy [19:14:20] the-wub: I'm seeing 547 donations from FR, in Civi. utm_* looks correct. Can you paste one of your queries so I can understand what I'm doing wrong? [19:16:46] cwd: I think our lives will be better if you can split up SmashPig. How about rewriting the frameworkey stuff to be minimal and standards-based? [19:17:00] Maybe just one piece at a time cos scope [19:17:02] yes please [19:17:20] Then, we can go ahead and rely on whatever is in SP/new lib [19:17:20] is this one a good place to start? [19:17:29] DQ? [19:17:31] yeah [19:18:01] i don't really care if they are in the same repo or whatever [19:18:08] awight: https://www.irccloud.com/pastebin/7JuHClCk/ [19:18:33] B1617_0720_frFR_dsk_p2_sm_pym_adyen being the only banner with an adyen credit card button [19:18:41] Good question--the frameworkey stuff is your problem right? [19:18:41] This is higher-level [19:18:41] The problem with DQ is that it's blocked on being a useful standalone library by the ponderous bulk of SP [19:18:41] *of the SP dependency [19:18:41] cwd: right [19:19:23] awight: yeah the frameworky stuff is what i think we'd live to regret [19:19:31] erk, why is vagrant DI looking for contribution_tracking in the wiki db now? [19:19:52] ejegg: ah i've had that happen too...i think it's an install time option [19:19:53] the-wub: Okay I get 144 contributions, if I apply the utm_source REGEXP 'B1617_0720_frFR_dsk_p2_sm_pym_adyen' [19:20:00] huh, ok [19:20:11] i defintely ended up with the CT table in the wiki db [19:20:30] i thought that's where it lived for about a year [19:20:46] oh hey, there are a bunch of globals we should set [19:20:51] the-wub: https://phabricator.wikimedia.org/P3521 [19:21:31] I think the right hand side of the "between" clause should use "now()" btw [19:23:18] cwd: yeah it's a silly mw extension that installs the table. It's a total jumble between the extension and wmf_civicrm/tracking.inc [19:23:50] Making a ContributionTracking library is gonna be fun... [19:24:02] heh [19:24:05] one foot in front of the other [19:24:13] Talk about schema. [19:24:24] awight: cool that seems right. and yeah that query is output from a python script that fills in the times or just defaults to the current time for end. [19:24:28] tightly coupled schema at that [19:24:38] the-wub: aha o.c. [19:25:20] /srv/br/methods.py if you were curious [19:25:23] cwd: ejegg: It should be workable to have the contribution_tracking table in the wiki db [19:25:53] i never had a problem with that setup [19:25:56] except for confusion [19:26:01] the-wub: Thanks, I'm very curious. I don't think I knew about that module, who wrote it? [19:26:04] but it seemed to work [19:26:04] cwd even with the crm bits? [19:26:16] the-wub: /me git statuses and gets sad [19:26:19] hmmm, well not that i noticed anyway :P [19:26:30] I'm inclined to leave it in drupal to match prod [19:26:32] awight: me :) It's one I need to get into fundraising-tools eventually [19:26:46] ejegg: yeah i'd agree with that. did you find where it's configured? [19:26:48] ejegg: yeah, the c_t database is configured as a drupal global I think [19:26:52] and puppet DTRT [19:27:08] the-wub: please do! Can I help with that in any way? [19:27:12] I'd love to jettison fr.py [19:27:16] It's easy to recreate [19:27:27] cwd there's nothing setting the ContributionTracking extension globals, so I'm trying to make a payments::contribution_tracking class [19:27:48] instead of just jamming them in with the DonationInterface settings [19:27:57] I'm so happy to see all these areas of the code get some tweaking [19:28:07] awight: if you could magic up some more hours in my day... [19:29:40] the-wub: What does "br" stand for? [19:30:34] awight: no idea. that folder was there and full of tools before I started [19:31:18] the-wub: Mind if I call the new module simply "stats"? [19:32:40] awight: I don't mind [19:33:22] (PS1) Awight: Snapshot of pcoombe's campaign statistics tools [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/300090 [19:33:55] I only grabbed the essentials to run methods.py. Are there others you use regularly? [19:34:26] the-wub: Also, in your home dir on the staging box, do: git clone https://gerrit.wikimedia.org/r/wikimedia/fundraising/tools [19:36:00] awight: amountchoices.py, amountstats, statler, timeline.py are all mine [19:37:06] we use ecom/fr.py still too [19:37:53] d'oh. okay [19:38:01] better to have it in source control than not [19:41:00] awight: have you successfully stuck a json payload into that table through PDO? i was wondering if it might get escaped weird [19:41:33] it works, happily [19:41:47] nice [19:42:14] postgres has a binary json column type that you can query and index into [19:42:17] (PS2) Awight: Snapshot of pcoombe and legacy campaign statistics tools [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/300090 [19:43:37] fr-tech: Fun background thing we should all do: take a peek at https://gerrit.wikimedia.org/r/#/c/300090/ -- The first time the /srv/br tools have been saved to git. [19:43:55] oh huh [19:44:54] someone got quints today [19:49:06] who wrote all this stuff? [19:49:18] ellery? [19:49:20] It's an amalgam [19:50:08] I mixed up some nicer new stuff from the-wub with legacy modules that need an exorcism [19:52:14] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Gift Source and Financial Type for Endowment Gifts - https://phabricator.wikimedia.org/T139629#2481748 (Eileenmcnaughton) Actually I'll deploy it onto live this morning for you (my morning) - I added it to the inst... [19:52:38] (PS1) Awight: Disable Adyen JCB [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/300096 (https://phabricator.wikimedia.org/T140691) [19:53:16] fr-tech: ^ rush config tweak fyi [19:54:44] (CR) Ejegg: [C: 2] Disable Adyen JCB [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/300096 (https://phabricator.wikimedia.org/T140691) (owner: Awight) [19:54:51] ty [19:55:03] awight: still gott cherry-pick [19:55:06] *gotta [19:55:19] yep thanks for the notice [19:55:21] I'll get that 'stop processing' patch out after tests [19:55:32] ejegg: You want to do it during the window tomorrow? [19:55:39] thanks both! I also wasn't sure what to do with that phab task now [19:55:42] what kind of tests? [19:55:47] Fundraising-Backlog: Strine's line - https://phabricator.wikimedia.org/T140944#2481757 (DStrine) [19:55:54] after this japan test [19:56:13] Fundraising-Backlog: Strine's spacer task - https://phabricator.wikimedia.org/T140944#2481770 (DStrine) [19:56:45] lol ^^ [19:56:49] Fundraising-Backlog: Strine's spacer task ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - https://phabricator.wikimedia.org/T140944#2481757 (DStrine) [19:57:15] wait, is lol a wikibugs trigger? [19:58:03] ejegg: are you laughing at my task ? [19:58:28] I can explain it's purpose later [19:58:39] phab layout mashing? [19:58:42] *its [19:58:48] (PS1) Awight: Disable Adyen JCB [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/300098 (https://phabricator.wikimedia.org/T140691) [19:58:49] yes [19:59:08] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/300099 [19:59:21] ejegg: ugh, warning that git cherry is currently reporting "+" for maybe five patches, but four of them are already merged, causing cherry-pick hell [19:59:29] (Merged) jenkins-bot: Disable Adyen JCB [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/300096 (https://phabricator.wikimedia.org/T140691) (owner: Awight) [19:59:52] dstrine: That's amaaazing [19:59:57] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Gift Source and Financial Type for Endowment Gifts - https://phabricator.wikimedia.org/T139629#2481779 (RLewis) @Eileenmcnaughton awesome, thank you. [20:00:14] ejegg: happy for this to be deployed? - https://gerrit.wikimedia.org/r/#/c/300099/ There is something going on with smash pig in there [20:00:20] I hope you become their boss one day, or otherwise find a way to accomplish revenge [20:01:28] (CR) Awight: [C: 2] Disable Adyen JCB [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/300098 (https://phabricator.wikimedia.org/T140691) (owner: Awight) [20:02:04] eileen: sure, let's go for it! I'll keep an eye out for audit flops [20:02:19] eileen: it will need a vendor update - I'll prep that if you want [20:02:37] wat. My mw-core checkout no longer recognizes submodules [20:02:58] k they're back. That was scary [20:03:11] ejegg: cool - let me know if you want me to do something or if you are just going to deploy it [20:03:33] eileen: I'll take over that patch & deploy [20:04:05] thanks ejegg - it does need a drush updb [20:04:13] k, will do [20:04:14] dstrine: that task is the phabricator equivalent of the crow that drops nuts in the street so cars run them over and break the shell [20:04:50] (PS1) Awight: update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/300102 [20:05:00] (CR) Awight: [C: 2] update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/300102 (owner: Awight) [20:05:54] cwd: um.. sure I will explain its role at sprint planning. [20:06:43] (PS1) Ejegg: Update smash-pig and other libs [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/300103 [20:07:47] (CR) Ejegg: [C: 2 V: 2] Update smash-pig and other libs [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/300103 (owner: Ejegg) [20:08:18] https://integration.wikimedia.org/ci/job/php53lint/4678/console [20:08:30] (PS2) Ejegg: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/300099 (owner: Eileen) [20:08:44] I don't think we even want to lint that dir? [20:09:37] (CR) Ejegg: [C: 2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/300099 (owner: Eileen) [20:12:31] (Merged) jenkins-bot: Disable Adyen JCB [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/300098 (https://phabricator.wikimedia.org/T140691) (owner: Awight) [20:14:17] (CR) Awight: "Isn't that the bug where we have to use an older version of Composer?" [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/300103 (owner: Ejegg) [20:15:01] (Merged) jenkins-bot: update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/300102 (owner: Awight) [20:15:15] (CR) Ejegg: "ah, yep, that's the one" [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/300103 (owner: Ejegg) [20:20:00] !log update paymentswiki from 7c6fb5a3b90fffdf2229cc903fb546e0e1e47998 to f23f15656eb488f5008b45b940077abbaa779004 [20:20:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:20:47] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/300099 (owner: Eileen) [20:23:33] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: DonationInterface gateway/config/*.yaml should be overrideable by deployed configuration - https://phabricator.wikimedia.org/T140947#2481834 (awight) [20:25:24] Devious thought: using SmashPig config to load gateway config in DonationInterface would give us a way to implement ^ for free. [20:25:35] cwd: ^ that seems relevant [20:25:36] !log updated civicrm from a386eb5a76ec97b3b01c46a49309dfa39bbc58b0 to 2f4805fa2d2a7c57881408be2b3a017d26d8f43e [20:25:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:29:38] eileen: deployed, your updates look good [20:29:56] ejegg: great! thanks [20:31:33] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Gift Source and Financial Type for Endowment Gifts - https://phabricator.wikimedia.org/T139629#2438173 (Eileenmcnaughton) @RLewis can you confirm if these are there now [20:31:38] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: DonationInterface gateway/config/*.yaml should be overrideable by deployed configuration - https://phabricator.wikimedia.org/T140947#2481866 (awight) We could get this for free with SmashPig::Configuration [20:32:01] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Adding Gift Source option for Engage users - https://phabricator.wikimedia.org/T137569#2481867 (Eileenmcnaughton) a:Eileenmcnaughton [20:32:19] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Adding Gift Source option for Engage users - https://phabricator.wikimedia.org/T137569#2372119 (Eileenmcnaughton) @LeanneS is this all good now? [20:33:45] (PS2) Awight: 'pending-new' instead of 'pending-atomic' [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300064 (owner: Ejegg) [20:33:52] (CR) Awight: [C: 2] 'pending-new' instead of 'pending-atomic' [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300064 (owner: Ejegg) [20:38:09] (Merged) jenkins-bot: 'pending-new' instead of 'pending-atomic' [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300064 (owner: Ejegg) [20:38:18] awight: it is worth noting that DonationQueue is doing the "new $classname" thing from config as well [20:38:35] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, FR-Astropay, Unplanned-Sprint-Work: Investigate and clean up Astropay 'MISSING PREDECESSOR' messages - https://phabricator.wikimedia.org/T139982#2448861 (Ejegg) Open>Resolved The safety net worked! The audit processor picked up 117... [20:38:50] i don't see anything in there we couldn't move to a module pretty easily. though i think the message format specific stuff would live better in DI, or a config file [20:39:01] or a DI config file [20:41:06] (CR) Awight: Snapshot of pcoombe and legacy campaign statistics tools (6 comments) [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/300090 (owner: Awight) [20:41:16] cwd: oh god [20:41:32] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-ActiveMQ, and 2 others: Read 'completion messages' from pending DB and ActiveMQ, scream if no match - https://phabricator.wikimedia.org/T140484#2481897 (Ejegg) a:Ejegg [20:42:00] so we're not even benefitting directly from the dependency injection [20:42:00] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Amazon, FR-Astropay, and 2 others: Quit looking in pending queue for completion message information - https://phabricator.wikimedia.org/T122641#2481899 (Ejegg) a:Ejegg [20:42:39] awight: if we moved that to SP we could [20:42:51] +1 [20:42:51] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Amazon, and 3 others: Quit looking in pending queue for completion message information - https://phabricator.wikimedia.org/T122641#1909593 (Ejegg) [20:43:37] but is the way SP does it better? it's certainly more complicated [20:44:10] cwd: I don't see it being better at all, but the part I like is that we use a single function every time we do that [20:44:21] ... so it becomes possible to fix at a stroke, later [20:44:24] yeah [20:44:25] or never :) [20:44:30] less fragmented is better [20:44:45] never tends to be the way things go [20:44:50] :) [20:45:07] it's only natural [20:46:41] awight: are all the classes DQ might instantiate shipped with phpqueue? [20:46:50] yeah [20:47:01] $className = $serverConfig['type']; [20:47:05] cool [20:47:11] so we just migrate that dep from DI to SP [20:47:21] NBD [20:47:31] cwd: I hope I checked is_interface there... [20:47:47] !class_exists [20:47:49] close enough [20:48:21] why did I put newBackend so far from the ctor... sigh [20:48:57] i guess we can just kill that function and replace with config&object [20:49:03] I don't even know why I would check class_exists, I'm sure that throws an exception w/o my help [20:49:07] yes! [20:49:23] a function called object makes me wince but oh well [20:49:25] note my array_merge on the configs too [20:49:27] +1 [20:49:40] fire up an IDE and change that everywhere, before we have to use it again [20:50:29] cwd: n.b. the one catch in that horrible newBackend function is that we assign the queue: key on the new object [20:50:57] hmmm [20:51:02] i wonder if we can represent that in config [20:51:45] I don't rememeber what it's for. I can't imagine php-queue needs that [20:51:52] but something that already bothers me about the interface is you are only able to pass things into the constructor [20:52:48] what should you do instead? [20:52:53] ahem in addition [20:53:54] i dunno, whatever you need to in a situation. not be boxed in by a previous decision [20:54:04] for example tho [20:54:36] I feel like I was constrained by that once, but can't imagine why [20:54:38] well for instance if this is a thing we want to be able to set on the object [20:54:45] queue: key [20:55:03] that should be a ctor param though--or are you suggesting a setter? [20:55:11] we are firing up classes we didn't write so assuming access to the constructor is all you need seems brittle [20:55:22] It's not good to tweak member vars from another class, methinks [20:56:08] cwd: Good point. It gets safer as we build on top of our own classes, i.e. php-queue backends [20:56:59] It does seem valid to require the caller to wrap any really funky module in a shim that can do everything through the constructor [20:58:27] it still feels like premature optimization when we don't really know what the needs are [20:58:56] i don't think there's a substitute for the simplest possible working implementation [20:59:07] when it comes to deciding what optimizations will be helpful in the future [21:00:02] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Gift Source and Financial Type for Endowment Gifts - https://phabricator.wikimedia.org/T139629#2481956 (RLewis) @Eileenmcnaughton yes, they're both on production [21:00:06] I'm all for something simpler--suggest one in pseudocode or something... [21:00:25] * awight eats redundant word [21:00:36] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Gift Source and Financial Type for Endowment Gifts - https://phabricator.wikimedia.org/T139629#2481961 (Eileenmcnaughton) Open>Resolved a:Eileenmcnaughton [21:01:02] every time. What's that thing where phpunit doesn't find any tests. [21:01:08] i definitely spend a lot more time bitching about bad interfaces than i save by writing good ones [21:01:12] awight: master branch? [21:01:27] cwd: that's fine but now I'm excited to see the goods [21:02:24] awight: tbh i think the way DQ does it in DI right now is more to the point. dead simple [21:02:44] if we want to shove that into something fancy down the road, the barrier is very low [21:02:52] where as in SP there is more proprietary buy in [21:02:53] wfm [21:03:09] I just don't want any more config and logging frameworks unleashed on the world [21:03:27] ya, 100% on board [21:18:06] (PS1) Ejegg: Add missing contribution_tracking columns [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/300169 [21:19:49] (CR) jenkins-bot: [V: -1] Add missing contribution_tracking columns [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/300169 (owner: Ejegg) [21:21:51] (PS2) Ejegg: Add missing contribution_tracking columns [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/300169 [21:24:52] A few more fr-vagrant fixes: [21:25:03] class ContributionTrackingProcessor { static function [21:25:06] ack [21:25:16] https://gerrit.wikimedia.org/r/#/c/300164/ [21:29:16] (PS1) Awight: Method to override configuration with data [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300171 [21:30:01] (CR) jenkins-bot: [V: -1] Method to override configuration with data [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300171 (owner: Awight) [21:33:26] really simple thing for review, fr-tech: [21:33:55] https://gerrit.wikimedia.org/r/300169 [21:34:12] will help get vagrant in shape [21:34:52] Looking [21:35:01] (CR) Awight: [C: 2] "Thanks!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/300169 (owner: Ejegg) [21:35:34] Heh, beat me by like a second. [21:35:35] (PS2) Awight: Method to override configuration with data [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300171 [21:35:55] oh man, mw update's composer check is braindead: monolog/monolog: 1.18.2 installed, ~1.18.2 required. [21:36:06] d'oh [21:36:07] X| [21:36:26] i said approximately not exactly! [21:36:53] that's stopping the update to 1_27 schema in vagrant [21:37:19] I can see where to fix it in the upgrade script, but I don't want to reimplement the composer version logic [21:37:26] (Merged) jenkins-bot: Add missing contribution_tracking columns [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/300169 (owner: Ejegg) [21:37:34] Ehh, I'll file a bug [21:42:01] (PS2) Awight: [WIP] Convert orphan rectifier to use the PendingDatabase [extensions/DonationInterface] (pending_db_orphan_rectifier) - https://gerrit.wikimedia.org/r/299915 (https://phabricator.wikimedia.org/T131275) [21:42:20] wait. I don't have to fork that. [21:46:08] (PS1) Awight: [WIP] Convert orphan rectifier to use the PendingDatabase [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/300173 (https://phabricator.wikimedia.org/T131275) [21:46:28] (Abandoned) Awight: Create pending_db_orphan_rectifier branch for forked deployment [extensions/DonationInterface] (pending_db_orphan_rectifier) - https://gerrit.wikimedia.org/r/299914 (owner: Awight) [21:46:33] (PS1) Ejegg: Work around a stupid core upgrade check [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/300174 [21:46:54] i should relocate for prawning [21:48:41] (CR) Awight: [C: 2] "Sad sauce." [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/300174 (owner: Ejegg) [21:48:54] ejegg|afk: wow. I thought that was the whole point [21:52:41] (Merged) jenkins-bot: Work around a stupid core upgrade check [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/300174 (owner: Ejegg) [21:58:20] fundraising-tech-ops: recycle db1008 for frack virtual test cluster protoyping - https://phabricator.wikimedia.org/T140953#2482169 (Jgreen) [22:00:29] ejegg|afk: Here's an interesting one: the orphan rectifier reads from oldest to newest, but sometimes cannot process a transaction. It seems like we need a database table for information about the processing of transactions, so we can join or test for "where not (last_rectified_date > $job_start_time)" [22:02:07] awight: what about just updating the pending message [22:02:07] ? [22:03:22] what would we update though? Stick some orphan-specific stuff into the json blob? [22:11:11] awight: yeah, that's what I was thinking. The Ingenico stuff is only in there for orphan processing [22:14:50] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Patch-For-Review, and 2 others: Log IP addresses when outgoing requests fail - https://phabricator.wikimedia.org/T139698#2482245 (Ejegg) Open>Resolved [22:15:02] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Patch-For-Review: Move payments to MediaWiki 1.27 - https://phabricator.wikimedia.org/T138016#2482246 (Ejegg) Open>Resolved [22:15:46] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Patch-For-Review, and 3 others: Convert GlobalCollect credit card forms from RapidHTML to Mustache - https://phabricator.wikimedia.org/T123002#2482248 (Ejegg) Open>Resolved [22:15:49] Fundraising Sprint House of Pain, Fundraising Sprint Indigo Girls, Fundraising Tech Backlog, MediaWiki-extensions-DonationInterface, and 3 others: [Epic] Make the forms come from one template - https://phabricator.wikimedia.org/T86248#2482249 (Ejegg) [22:15:52] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Patch-For-Review: Move payments to MediaWiki 1.27 - https://phabricator.wikimedia.org/T138016#2386861 (Ejegg) [22:15:54] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Patch-For-Review: Make RapidHTML work with mediawiki 1.26+ - https://phabricator.wikimedia.org/T139551#2482250 (Ejegg) Open>Resolved [22:16:01] Fundraising Sprint Hermit Crab Husbandry, Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, and 8 others: Don't make donors guess minimum donation amount - https://phabricator.wikimedia.org/T105618#2482252 (Ejegg) Open>Resolved [22:16:17] Fundraising Sprint Licking Cookies, Fundraising Sprint Muggle Baiting, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, and 3 others: Kill staged_vars - https://phabricator.wikimedia.org/T137387#2482254 (Ejegg) [22:16:19] Fundraising Sprint Licking Cookies, Fundraising Sprint Muggle Baiting, Fundraising-Backlog, FR-Adyen, and 4 others: Adyen staging functions should be rewritten as helpers - https://phabricator.wikimedia.org/T137157#2482253 (Ejegg) Open>Resolved [22:16:47] Fundraising Sprint Hermit Crab Husbandry, Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, and 10 others: [Epic] Consolidate queue abstractions - https://phabricator.wikimedia.org/T131271#2482256 (Ejegg) [22:16:50] Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, Fundraising Sprint Licking Cookies, and 8 others: Write high-level SmashPig DonationQueue - https://phabricator.wikimedia.org/T133965#2482255 (Ejegg) Open>Resolved [22:17:00] ejegg: It's not a searchable key though, so I'm not sure how to getOldest message and step over the unrectified one [22:17:04] Fundraising Sprint Licking Cookies, Fundraising Sprint Muggle Baiting, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, and 2 others: Add autocomplete hints to donation form - https://phabricator.wikimedia.org/T137716#2482257 (Ejegg) Open>Resolved [22:17:47] Fundraising Sprint Licking Cookies, Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Patch-For-Review, and 2 others: Ingenico donation form: email address autosuggestion is confusing donors - https://phabricator.wikimedia.org/T138957#2482258 (Ejegg) Open>Resolved [22:18:27] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, FR-Ingenico, MediaWiki-extensions-DonationInterface, Unplanned-Sprint-Work: Ingenico iDEAL donation form Continue button unresponsive for all banks - https://phabricator.wikimedia.org/T140028#2482259 (Ejegg) Open>Resolved [22:18:54] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Pipeline Report - https://phabricator.wikimedia.org/T128818#2482262 (Ejegg) [22:18:56] Fundraising Sprint Licking Cookies, Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Create new prospecting fields - https://phabricator.wikimedia.org/T138908#2482261 (Ejegg) Open>Resolved [22:21:20] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Unplanned-Sprint-Work: *Existing* campaigns are causing messages to be rejected: '\'C15_Email1\' is not a valid option - https://phabricator.wikimedia.org/T139969#2448458 (Eileenmcnaughton) Sorry I've caused co... [22:43:40] Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, Fundraising Sprint Licking Cookies, Fundraising Sprint Muggle Baiting, and 2 others: CentralNotice: finalize list of wild useless cookies for slurp, and push config change - https://phabricator.wikimedia.org/T135090#2482324 (A... [22:45:02] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, fundraising-tech-ops, FR-ActiveMQ, Epic: Provision Redis cluster for Fundraising - https://phabricator.wikimedia.org/T130283#2482328 (awight) a:Jgreen [22:45:53] Fundraising-Backlog: SPRINT N (7/20 - 8/3) Queue work and bugs - https://phabricator.wikimedia.org/T136251#2328180 (DStrine) [22:49:33] Fundraising-Backlog, FR-Ingenico: 3502596148 mysteriously in 600 - https://phabricator.wikimedia.org/T139000#2482353 (DStrine) Open>declined [22:52:04] Fundraising Sprint Muggle Baiting, Fundraising Sprint N 2016, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Update CRM SmashPig usage for new config format - https://phabricator.wikimedia.org/T140814#2482374 (DStrine) [22:52:06] Fundraising Sprint Muggle Baiting, Fundraising Sprint N 2016, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, and 3 others: Donations message with empty body! - https://phabricator.wikimedia.org/T140515#2482375 (DStrine) [22:52:11] Fundraising Sprint Muggle Baiting, Fundraising Sprint N 2016, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Unplanned-Sprint-Work: *Existing* campaigns are causing messages to be rejected: '\'C15_Email1\' is not a valid option - https://phabricator.wikimedia.org/T139969#2482379 (DStrine) [22:52:13] Fundraising Sprint Muggle Baiting, Fundraising Sprint N 2016, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: CiviCRM dedupe causes runaway query - https://phabricator.wikimedia.org/T139964#2482380 (DStrine) [22:52:15] Fundraising Sprint Licking Cookies, Fundraising Sprint Muggle Baiting, Fundraising Sprint N 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Spike: Monitor deployment of: Suppress CentralNotice ResourceLoader modules on Special and... - https://phabricator.wikimedia.org/T139439#2482382 [22:52:18] Fundraising Sprint N 2016, Fundraising Tech Backlog, Fundraising-Backlog: Major Donors on email suppression list who should be opted in - https://phabricator.wikimedia.org/T139102#2482383 (DStrine) [22:52:20] Fundraising Sprint Licking Cookies, Fundraising Sprint Muggle Baiting, Fundraising Sprint N 2016, Fundraising-Backlog, and 2 others: Handle refund mismatch more correctly - https://phabricator.wikimedia.org/T138542#2482384 (DStrine) [22:52:22] Fundraising Sprint Licking Cookies, Fundraising Sprint Muggle Baiting, Fundraising Sprint N 2016, Fundraising-Backlog, and 4 others: Spike: Monitor deployment rolling back our "googleoff" tag - https://phabricator.wikimedia.org/T137761#2482387 (DStrine) [22:52:25] Fundraising Sprint Licking Cookies, Fundraising Sprint Muggle Baiting, Fundraising Sprint N 2016, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Investigate Civi Load Time issue - https://phabricator.wikimedia.org/T138334#2482385 (DStrine) [22:52:27] Fundraising Sprint Killing Time, Fundraising Sprint Muggle Baiting, Fundraising Sprint N 2016, Fundraising-Backlog, and 4 others: IL donation form: CVV field only allows three digits, blocks AmEx donors - https://phabricator.wikimedia.org/T137811#2482386 (DStrine) [22:52:29] Fundraising Sprint Killing Time, Fundraising Sprint Licking Cookies, Fundraising Sprint Muggle Baiting, Fundraising Sprint N 2016, and 3 others: Special:LoadBanners should scream if translation fails - https://phabricator.wikimedia.org/T137247#2482390 (DStrine) [22:52:32] Fundraising Sprint N 2016, Fundraising-Backlog: SPRINT N (7/20 - 8/3) Queue work and bugs - https://phabricator.wikimedia.org/T136251#2482391 (DStrine) [22:52:34] Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, Fundraising Sprint Licking Cookies, Fundraising Sprint Muggle Baiting, and 3 others: CentralNotice: finalize list of wild useless cookies for slurp, and push config change - https://phabricator.wikimedia.org/T135090#2482392 (D... [22:52:37] Fundraising Sprint Licking Cookies, Fundraising Sprint Muggle Baiting, Fundraising Sprint N 2016, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Agree acceptance criteria for deciding the script is 'ready' - https://phabricator.wikimedia.org/T135062#2482393 (DStrine) [22:52:39] Fundraising Sprint Hermit Crab Husbandry, Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, and 12 others: [Epic] All pending producers should write to a single pending queue - https://phabricator.wikimedia.org/T133195#2482394 (DStrine) [22:52:42] Fundraising Sprint Hermit Crab Husbandry, Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, and 6 others: CentralNotice: Slurp up existing wild cookies and copy them to LocalStorage or del... - https://phabricator.wikimedia.org/T132640#2482395 [22:52:45] Fundraising Sprint Freshmaking, Fundraising Sprint Ghostbusting , Fundraising Sprint Hermit Crab Husbandry, Fundraising Sprint Internet Exploring, and 8 others: TY mails failing silently for recurring donations - https://phabricator.wikimedia.org/T131200#2482396 (DStrine) [22:52:47] Fundraising Sprint Muggle Baiting, Fundraising Sprint N 2016, Fundraising-Backlog, fundraising-tech-ops, and 2 others: Provision Redis cluster for Fundraising - https://phabricator.wikimedia.org/T130283#2482397 (DStrine) [22:52:49] Fundraising Sprint Licking Cookies, Fundraising Sprint Muggle Baiting, Fundraising Sprint N 2016, Fundraising-Backlog, and 3 others: Convert CentralNotice to use extension registration - https://phabricator.wikimedia.org/T108721#2482398 (DStrine) [22:52:52] Fundraising-Backlog, FR-ActiveMQ, FR-Smashpig: SmashPig tasks which read from pending queue should compare to pending db - https://phabricator.wikimedia.org/T140959#2482399 (Ejegg) [23:12:14] I'm still really sketched out over "nit-picking", we might have to do a recall vote on this one... [23:12:17] https://books.google.com/books?id=kSr4fO2zYrIC&pg=PA269&lpg=PA269&dq=nit-picking+1956&source=bl&ots=s1r1zaZNtV&sig=tOZ6Bd0YS_qlpCPZqmh396hz0xU&hl=en&sa=X&ved=0ahUKEwicqf6DlYPOAhUX0mMKHe1VBLAQ6AEIJTAB#v=onepage&q=nit-picking%201956&f=false [23:13:54] Fundraising Sprint Muggle Baiting, Fundraising Sprint Nitpicking, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 3 others: Read 'completion messages' from pending DB and ActiveMQ, scream if no match - https://phabricator.wikimedia.org/T140484#2482481 (Ejegg) [23:14:13] Fundraising Sprint Muggle Baiting, Fundraising Sprint Nitpicking, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 3 others: Quit looking in pending queue for completion message information - https://phabricator.wikimedia.org/T122641#1909593 (Ejegg) [23:15:31] awight: i'll gladly shift my votes to "not programming" :) [23:16:06] hehe, me too. [23:16:14] Howdy all, just a reminder that our Japan pre-test will start in 15 minutes [23:16:19] I think we should have a "block" option as part of our voting [23:16:34] spatton: thx! I'm monitoring the logs, I'll be around for at least the first hour. [23:16:46] AndyRussG|absent, ejegg - EST is cray, meetings till after y [23:16:51] and by y i mean 7 [23:17:37] MST is easy, right in the middle of everyone (except NZ of course) [23:20:08] yeah, I'm about out of think at this point [23:21:10] uh huh [23:21:55] See you late tomorrow... I'm parenting until the afternoon. [23:22:29] Did want to chat more about the fate of DonationQueue, and all the frameworkey stuff [23:30:31] Thanks awight! We are liiive from the matrix [23:32:06] Nothing yet... [23:32:51] awight: yes! [23:33:24] spatton: 3 people have opened the form, no bites yet though [23:33:32] if moving to SP is consensus i won't complain, but i'd be happy to try packing it up as a standalone [23:34:19] cwd: I'm sure you'll lead that goat of a problem to fresh water [23:35:26] I like the way you're thinking: If we don't use any libraries, the framework stuff almost disappears [23:38:22] awight: I'm seeing 5 Ingenico donations, no Adyen data yet [23:39:24] spatton: 5 adyen donations are in pending:process [23:42:44] select count(*) from fredge.payments_initial i join drupal.contribution_tracking t on i.contribution_tracking_id=t.id join civicrm.wmf_contribution_extra e on e.entity_id = t.contribution_id where date > '2016-07-20T23:30:00' and i.country = 'JP' and ((i.gateway='adyen' and payments_final_status = 'pending') or (payments_final_status = 'complete')) and validation_action='process'; [23:42:50] > 9 [23:48:32] ok well it's dinner time here but i will dig into that tomorrow! good night and good luck