[00:00:35] I'm heading out for the day. Seeya! [00:00:51] byeee [00:03:20] Fundraising Sprint Freshmaking, Fundraising Sprint Ghostbusting , Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: create protocols and documentation for various civicrm database backup/restore/failov... - https://phabricator.wikimedia.org/T133617#2237501 [00:06:21] eileen: thinking about dump/restore/failover suddenly having the log tables in the civicrm database looks more attractive [00:07:14] Jeff_Green: in the same database? [00:07:59] it would simplify the sync with staging in that the triggers would not point to different DBs [00:08:13] yeah, at least there's a serious entry for the pro column [00:08:33] ? what does that mean? [00:08:39] oh I get it [00:08:42] the pros and cons [00:08:48] I thought pro column was tech thing... [00:08:53] ha [00:09:27] awight: do you have any opinion on it? [00:09:45] Is there anything to gain in terms of not dying if the logging database exceeds some usage quota? [00:10:06] I don't think so [00:10:15] Jeff_Green: I feel like 'whatever works best from backup, restore & replication' is the right choice [00:10:26] +1 [00:10:50] Sorry that my initial 2-database opinion was based on a bad understanding... [00:10:52] eileen: ok, we can evaluate as we go through that process [00:11:47] ok - cool - you are going to dig into that against that Phab aren't you? [00:12:25] Thanks for coming in late! [00:12:52] yeah I'll try to give it some thought tomorrow [00:13:21] i don't really know what to expect in terms of growth of that table [00:14:30] The log tables will grow quite a lot... [00:15:07] but, perhaps not markedly more than existing growth has been because we have been inserting new contacts rather than updating in general [00:15:17] so, that's the same sort of growth pattern - only add [00:16:04] That makes sense--probably a small constant * current growth, like 2-3x faster but linearly [00:16:33] is there any point where we can roll off old log data? [00:17:30] Tied to our TBD data destruction policy, somehow... could be that we want to keep many years of data... [00:17:35] that might be a bit of an ongoing conversation... [00:17:54] some tables yes - the setting table, for example, we have fairly low interest in [00:18:39] (PS2) Awight: [WIP] Simplify config overrides read YAML [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/285317 (https://phabricator.wikimedia.org/T133601) [00:19:14] ok, lots to think about. I'm going to head out for the night unless there's anything else that needs doing [00:19:31] (CR) jenkins-bot: [V: -1] [WIP] Simplify config overrides read YAML [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/285317 (https://phabricator.wikimedia.org/T133601) (owner: Awight) [00:19:35] nothing here, thanks! [00:20:07] ok have a good rest of today, see you tomorrow [00:27:09] awight: quick question... [00:27:38] in the dedupe code in batch mode it aborts when it hits a 'conflict' [00:28:13] for our purposes differences in the calculated fields e.g 'is_2006_donor' should not be seen as a conflict [00:28:27] eileen: argh. ok [00:28:31] so I have to define 'not a conflict; [00:28:37] & I have 2 possible definitions [00:28:38] I guess we need a whitelist for that [00:28:40] k [00:28:53] 1) is a field in the group WMF Donor [00:28:56] or [00:29:42] 2) is a contact custom field that is set to calculated field [00:29:46] (PS3) Awight: [WIP] Simplify config overrides read YAML [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/285317 (https://phabricator.wikimedia.org/T133601) [00:30:08] I think the column name is actually 'is_view' which is confusing [00:30:28] (CR) jenkins-bot: [V: -1] [WIP] Simplify config overrides read YAML [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/285317 (https://phabricator.wikimedia.org/T133601) (owner: Awight) [00:30:53] is_view isn't quite good enuf, because it isn't necessarily a value calculated from other fields, right? [00:31:05] no…. [00:31:16] it's just a field that has had that checkbox ticked [00:31:43] Can we be sure that every field in WMF Donor group will be calculated? [00:31:54] eileen: argh, no. [00:31:59] do_not_solicit for example [00:32:11] hmm [00:32:15] I'd be happy if there were a per-field whitelist [00:32:22] yeah sounds like it [00:32:30] not sure how that fits into the GUI, though [00:32:31] the field names should be consistent even if the ids aren't [00:32:51] * awight grits teeth at custom field IDs [00:32:56] we could whitelist is_xxxx_donor [00:33:09] I think they're defined through the future already [00:33:13] ... the whole future ;) [00:33:14] yeah! [00:33:18] :-) [00:33:26] OK - I'll just write in all the fields [00:33:31] ./wmf_common/wmf_dates.php:define('WMF_MAX_ROLLUP_YEAR', 2025); [00:33:44] it's less serious for a merge to be blocked than allowed in error [00:44:26] cwd|afk: btw, it's a mistake that SmashPig/vendor is a submodule on master... that should only exist on the deployment branch. [00:47:54] won't that make CI depend on composer install? [00:49:05] maybe that doesn't matter... [00:49:41] CI already does [00:49:57] I might have been wrong about it being a submodule, actually-- [00:50:12] I was getting submodule update warnings, but they went away when I rm'ed the directory [00:58:01] Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Write a hook to skip the calculated tables from conflict consideration - https://phabricator.wikimedia.org/T133625#2237657 (Eileenmcnaughton) [01:02:11] (PS1) Eileen: Fix merge hook to remove calculated fields from merge_conflicts [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/285322 (https://phabricator.wikimedia.org/T133625) [01:03:24] until tomorrow! [01:03:28] (PS4) Awight: [WIP] Simplify config overrides read YAML [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/285317 (https://phabricator.wikimedia.org/T133601) [01:04:14] (CR) jenkins-bot: [V: -1] [WIP] Simplify config overrides read YAML [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/285317 (https://phabricator.wikimedia.org/T133601) (owner: Awight) [01:26:14] Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Write a hook to skip the calculated tables from conflict consideration - https://phabricator.wikimedia.org/T133625#2237717 (Eileenmcnaughton) After applying this & running on staging I... [01:28:25] (PS1) Eileen: Backport markConflicts code [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/285325 (https://phabricator.wikimedia.org/T132396) [05:46:37] (PS1) AndyRussG: [WIP] Mixed storage for impression diet and large banner limit [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/285336 (https://phabricator.wikimedia.org/T132639) [05:47:56] (Abandoned) AndyRussG: [Draft] Use LocalStorage for impression diet [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/284934 (https://phabricator.wikimedia.org/T132639) (owner: AndyRussG) [05:48:46] (PS2) AndyRussG: [WIP] Mixed storage for impression diet and large banner limit [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/285336 (https://phabricator.wikimedia.org/T132639) [08:18:49] (CR) jenkins-bot: [V: -1] [WIP] Mixed storage for impression diet and large banner limit [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/285336 (https://phabricator.wikimedia.org/T132639) (owner: AndyRussG) [09:03:37] (CR) Awight: "Another thing to look into, in this and the SmashPig FifoDataStore patch, is we might only want to commit() when used with pop(). Plus, s" [wikimedia/fundraising/php-queue] - https://gerrit.wikimedia.org/r/284977 (https://phabricator.wikimedia.org/T131271) (owner: Awight) [15:21:34] Fundraising-Backlog, Epic, FR-Adyen: [epic] Adyen campaign ready - https://phabricator.wikimedia.org/T118202#2239833 (DStrine) Open>Resolved [15:21:36] Fundraising-Backlog, Epic, FR-Adyen: [epic] Get Adyen back up - https://phabricator.wikimedia.org/T90748#2239835 (DStrine) [15:22:00] Fundraising-Backlog: [epic] Accept credit cards in France via Enhanced Silent Order Post - https://phabricator.wikimedia.org/T110111#2239836 (DStrine) Open>Resolved [15:22:18] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, MediaWiki-extensions-DonationInterface, Epic, and 3 others: [Epic] Clean up "pending" queue usages - https://phabricator.wikimedia.org/T130897#2149972 (DStrine) [15:23:01] Fundraising-Backlog, Epic, FR-Adyen: [epic] Get Adyen back up - https://phabricator.wikimedia.org/T90748#1066487 (DStrine) Open>Resolved [15:46:03] (PS1) Ejegg: Merge master into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/285413 [15:46:37] (CR) Ejegg: [C: 2] Merge master into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/285413 (owner: Ejegg) [15:47:01] (CR) Ejegg: [V: 2] Merge master into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/285413 (owner: Ejegg) [15:48:22] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/285415 [15:48:45] (CR) Ejegg: [C: 2] Update DonationInterface submodule [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/285415 (owner: Ejegg) [16:02:52] cwd: letsencrypt is baller. [16:03:19] The -auto packaging is a bit aggressive for my taste, but it got the job done, even using the --manual method. [16:03:50] awight: about to deploy https://gerrit.wikimedia.org/r/285413/ [16:04:13] ejegg: Thrilling, thanks! I'll check on iDeal things [16:05:59] awight: yeah it's ridiculous how easy it is [16:07:25] My only disappointment is pretty unreasonable--I had imagined for some reason that they even did the green EV bar of ignorance [16:07:35] !log updated payments from [16:07:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:07:47] !log updated payments from f09297028acace67588c2de845b754e2ace75c97 to 3ac3a0d7bb6e2d8443a87fb62bd5412a95f75aa2 [16:07:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:07:56] awight: you should get green bar [16:08:09] now trusted by some overwhelming percentage of clients [16:08:35] I get the green lock which is rad, but not the EV [16:08:57] ooh, what you pay extra for if you have an online store? [16:09:28] what a racket [16:09:56] i know... [16:10:39] Fundraising-Backlog, fundraising-tech-ops, Unplanned-Sprint-Work: Cron failure on payments1001 - https://phabricator.wikimedia.org/T133703#2240025 (awight) [16:12:32] ejegg: iDEAL looking good [16:12:36] cool [16:12:45] red errors are looking pretty good too [16:13:10] though it looks like we don't do country-specific overrides on client-side error messages [16:13:38] back button click fix seems good too [16:14:09] awight: did you have an old ideal test link? [16:14:23] seeing a ffname error in the log [16:14:26] "payment_method":"rtbt.rtbt_ideal" [16:14:27] Back button from AstroPay gives an error, not sure that's what we want, but not a critical bug at least [16:14:34] with the dotted method [16:14:42] oh, what error are you getting? [16:14:46] and what payment method? [16:15:29] I tried MX/CC and it held down the back button to get to the payments form (skipping over their redirect page) [16:15:33] ejegg: I don't think it was me [16:15:48] My IP is 173... [16:15:56] will checkout, one sec [16:16:10] ejegg: ah--I hit the "Volver" link to get back [16:17:03] awight: oh right, they now give us the 'failed at bank' code on that link [16:17:19] slightly better than the exact same code as successful payments [16:17:25] but still not optimal [16:17:57] ejegg: There's only one rtbt.rtbt from today, so it must have been me or one of us, not something in the wild. [16:18:06] huh, that ideal ffname error has a customerip of 8.XX [16:18:16] and name/email asdf [16:18:27] I suspect techsen ;) [16:18:35] gotta be [16:18:59] jessicarobell: deploy looks good! [16:19:29] great! thanks ejegg! [16:22:52] Fundraising-Backlog: Silverpop export not capturing language changes in Civi (not sure if this is language-only or not) - https://phabricator.wikimedia.org/T96410#2240149 (MBeat33) p:Normal>High This continues to be an issue, and it's donor-facing in that the longer it's unresolved the more donors wi... [16:25:21] Fundraising-Backlog, MediaWiki-extensions-DonationInterface, FR-Astropay: Client-side validation messages are not country specific - https://phabricator.wikimedia.org/T133706#2240152 (Ejegg) [16:26:21] Fundraising-Backlog, fundraising-tech-ops, Unplanned-Sprint-Work: Cron failure on payments1001 - https://phabricator.wikimedia.org/T133703#2240179 (Jgreen) Open>Resolved a:Jgreen FIxed. I added a job on friday running as user www-data (same as apache) to mail modsec alerts on 5 minute int... [16:28:28] Fundraising-Backlog, fundraising-tech-ops, Unplanned-Sprint-Work: Cron failure on payments1001 - https://phabricator.wikimedia.org/T133703#2240184 (awight) Thanks! [16:29:10] Fundraising Sprint Hermit Crab Husbandry, Fundraising Tech Backlog, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, and 3 others: Don't make donors guess minimum donation amount - https://phabricator.wikimedia.org/T105618#2240186 (Ejegg) Note that we use an informative minimum amount... [16:30:56] so, guess what version of PHP ships with Ubuntu 16.04? [16:31:33] and guess what version of drupal doesn't work under PHP7.0? [16:31:49] I'd been meaning to switch to HHVM locally anyway... [16:32:13] omg [16:32:40] but the good news is Mediawiki and DonationInterface seem happy under 7.0 [16:33:00] It doesn't look too bad, https://www.drupal.org/node/2454439 [16:34:28] ah, might be partly our fault... ParseError: Invalid numeric literal in drupal_load() (line 397 of /home/elliott/src/php/fundraising/crm/sites/all/modules/queue2civicrm/queue2civicrm.module). [16:35:14] "02801" hehe [16:35:16] oookay [16:35:27] I wonder if that's been interpreted as octal under PHP 5? [16:35:41] who knows... [16:35:50] * awight glares at test code masquerading as production [16:36:58] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Move queue2civicrm_generate_message to test support - https://phabricator.wikimedia.org/T133712#2240254 (awight) [16:37:23] huh, it's only ever used in that test file too [16:37:51] cool, drupal loads now! [16:37:59] ...but civi still doesn't [16:38:34] ParseError: syntax error, unexpected 'clone' (T_CLONE), expecting identifier (T_STRING) or '(' in require_once() (line 1 of /home/elliott/src/php/fundraising/crm/civicrm/packages/DB/DataObject.php(207) : eval()'d code). [16:40:01] ooh that's nastier [16:40:10] eileen probably has the back story [16:42:26] cwd: I've been having regrets about commit()... lmk if you want to talk it out [16:43:06] awight: you're using that instead of peek, right? [16:43:22] yeah i'd love to hear what you're thinking [16:43:37] just don't want to do anything long-running before committing or rolling back... [16:43:51] like, say, payment processor web service calls [16:44:13] ejegg: yeah, pop+commit = peek + ack [16:44:57] I like commit better than ack, for sure [16:45:28] The thing that's bothering me has to do with ejegg's points on https://gerrit.wikimedia.org/r/#/c/284977/ -- I don't actually want to support push + commit. [16:47:00] commit that only works for pop sure sounds like 'ack' to me [16:48:25] i think i'm not understanding [16:48:46] why would you ever want to not commit a transaction, unless you roll back? [16:50:20] I'm pretty sure we always want to auto-commit and flush push(), like immediately. [16:51:01] ejegg|brb: The difference between commit and ack as I'm imagining them is that ack() requires that we feed it something about the message we popped, commit can be called without that extra, internal data flow [16:51:40] oh, you mean specifically acking a particular message [16:51:43] cwd: If the app crashes, we don't want to consume the message we were processing... that's the main one as far as I care [16:51:53] ejegg: yeah, I find that tedious [16:52:17] that makes sense [16:52:24] yeah [16:52:58] also, I'm secretly trying to optimize for the Kafka integration. [16:53:28] it seems like that would pretty radically change the desired behavior [16:53:38] i keep thinking in sql terms [16:55:48] Sorry--what and how would change? [16:55:54] like ack vs commit? [16:56:00] (PS3) Ejegg: WIP Send failmail and set no_thank_you on TY errors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/281842 (https://phabricator.wikimedia.org/T131200) [16:57:21] (CR) Cdentinger: [WIP] Support auto_commit=false; pop deletes (1 comment) [wikimedia/fundraising/php-queue] - https://gerrit.wikimedia.org/r/284977 (https://phabricator.wikimedia.org/T131271) (owner: Awight) [16:57:53] awight: would pop behavior in sql actually involve deleting a row? [16:58:33] LATAM campaign starting in 2 minutes.... [17:01:49] cwd: I think that can be opaque to the client [17:02:16] All that pop() guarantees is that, after implicit or explicit commit, you'll never see that message again. [17:02:33] We're free to move to an archive table, file, or toggle a flag on the row... [17:02:36] or delete [17:02:52] awight: yeah i'm just thinking from PDO's perspective doesn't that make a difference? [17:02:59] how we want to implement pop [17:03:05] I'd probably leave it deleting for now--PDO is just a nice development backend as I see [17:03:05] (PS3) AndyRussG: [WIP] Mixed storage for impression diet and large banner limit [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/285336 (https://phabricator.wikimedia.org/T132639) [17:03:27] Caminando por el campo, una vaca me encontré [17:03:33] cool [17:03:53] cwd: True--commit would suggests things we're not making guarantees about, in that case [17:04:48] AndyRussG: They're such dinosaurs! [17:05:36] cwd: well, I've been toying with just taking the Kafka API verbatim, and calling our function commitOffset() [17:05:50] Feathers suggest an evolutionary link to birds [17:06:02] hehe [17:06:26] citation needed. [17:08:52] (PS4) AndyRussG: [WIP] Mixed storage for impression diet and large banner limit [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/285336 (https://phabricator.wikimedia.org/T132639) [17:11:08] awight: i still don't understand which part exactly is bothering you, can you elaborate? [17:12:26] Sure! It's that providing commit() creates expectations that all the other PHP-Queue operations are transactional, which is actually difficult or impossible I believe [17:12:38] and I'm only interested in pop+commit [17:13:27] when do you foresee committing? will the job be shelled out to a new php? [17:13:33] we then run into the batching issue again [17:13:52] even if that's all we want, the current PS makes pushes transactional if there was previously a pop [17:14:05] need two different connections, i think [17:14:35] oh, because it sets that variable? [17:14:53] because it starts a txn on the shared connection [17:15:18] i see an explicit begin() in push... [17:16:14] but anyway stuff should rollback() if it's not going to commit, i don't think it's good practice to leave a dangling transaction [17:17:04] What I'd really like might be foolish--it's the ability to say FifoStore::getProducer, and that bugger *only* supports push() [17:17:32] i mean we could certainly computer that [17:18:04] i don't understand what it means for a push to be transactional. what would it need isolation from? [17:18:22] I'm not interested in transactional push... [17:18:48] but i see begin() in push() [17:18:52] it's more, that it's an obstacle to sanity yet there's an expectation to support it if I call my third API function commit() [17:19:17] cwd: That was just me being stupid [17:20:06] ah so are we losing stuff in translation between what commit means to sql and what it means in the context of popping from a queue? [17:20:29] yes, I think that's it [17:20:47] every sql query is a transaction of some type [17:21:04] yah statements are atomic [17:21:26] it seems to me like the only thing that needs isolation is the queue consumer [17:21:33] so more than one consumer can't grab the same item [17:22:21] am i tracking at all? [17:23:14] nothing pushing to the queue should need any context [17:23:31] +1 [17:24:20] But pop() is going to be atomic no matter what, so I see the need as more like atomicity of an entire queue consume iteration [17:24:34] yeah [17:24:38] ... a message is either consumed and processed, or not consumed [17:25:09] in what case do we want it to not consume? only if the app crashes? [17:25:45] since it's an async operation it seems like there will need to be a judgment call at some point as to when it is considered consumed [17:26:09] async? [17:26:47] isn't the idea for the queue consumer to shell out long running jobs? [17:27:27] I think of them as batch jobs [17:27:33] not long-running cos PHP [17:28:29] oh right so we're still doing this through the run every 2 minutes jenkins thing [17:28:34] i keep forgetting that [17:28:40] just for now... [17:29:24] in the ideal future is it an async operation? [17:30:02] I'm still not grokking which part is async [17:30:03] we could build the pipeline we want all the way out to the php worker and contain all the nasties there [17:30:53] i'm imagining a queue consumer doling out jobs to php [17:31:17] and not waiting for one to finish before handing out another [17:31:48] That would be real nice, and I feel like we can have that iff. we figure out the right solution to this commit thing [17:32:39] you want the api to have a method called commit() that actually removes the job from the queue? [17:32:57] and pop() selects it but does not delete? [17:33:03] It doesn't seem to change the problem much, though--we either have a native PHP Fifo library with a pop+commit API, or we have a separate consumer which feeds PHP somehow, but PHP will still need to commit as it eats, right? [17:33:35] ah, i see [17:33:58] cwd: I ~think so, the library will support auto-commit for upstream compatibility, but we'll always use as non-autocommitt [17:34:11] * awight commits self [17:34:11] so the job wouldn't get removed until the php worker is done? [17:34:15] hehe [17:34:30] exactly [17:34:48] well from an abstract point of view i'd want to do that differently [17:35:03] We could *maybe* have the atomicity around the entire batch instead of individual records, but I fear that forces a bunch more complexity on us [17:35:06] i don't know how divergent this is from the current scheme [17:35:08] tell me more! [17:35:42] well i'd want it to be like, the message describes a job that needs doing, consumer pops it off and hands it to php, message is no longer on the stack [17:35:55] php could then throw it into an "in progress" table to track that part [17:36:20] having the worker tightly coupled to the queue seems broken doesn't it? [17:36:22] there's a race condition there... [17:36:32] if it dies between pop and push to in progress [17:36:59] +1 I'd prefer to decouple, as long as we actually don't need the native, coupled features [17:37:35] those features being loading the app up right? [17:37:36] anyway, the in progress table seems to be simulating atomic behavior [17:38:01] uh, like commit() would be a feature that so far looks like it needs to be native [17:38:24] So hard... I really don't see a way to work around this yet [17:38:26] what if php scheduled another job to say that it was finished? [17:38:26] surprisingly [17:38:38] wat [17:38:49] instead of changing the queue [17:40:09] you mean like when the capture completed message comes in? [17:40:13] i see what you mean about that race condition...i guess i was calling that 'async' :) [17:41:03] i was thinking when it finishes whatever the message described [17:41:23] maybe this will be simpler if we imagine pure file streams... [17:41:25] i guess i'm having trouble with the idea that the thing that does all the work is the same thing that is watching the queue [17:41:54] We have record-based data coming in, and a consumer job has processed up to a given offset. [17:42:37] I totally feel you on that last thing. I don't care if the queue consumer is PHP or whatever, but decoupling from our processing code is a huge win. [17:43:02] The queue thing should be an extension with no dependencies pointing at it from any other modules. [17:43:58] We have a crude version of that so far, I'm sure you've mucked around in wmf_common/Queue::dequeue_loop ? [17:44:08] dstrine: I'm trying to figure out what strings we'll need to get manually translated for Adyen. Do you have a list of payment methods we'll want to use for Israel? [17:44:12] i don't think i have [17:44:22] All a consumer has to provide is a callback that takes one record. [17:44:30] but the blocker for that workflow seems to be loading up the app for each job no? [17:44:37] instead of batching [17:45:08] Its interface is just, throw an exception if we can't do stuff. The WmfException type determines whether we will reject or not consume the message. We almost always consume it, though. [17:45:39] That workflow pretty much works [17:46:00] does the job itself describe a batch of operations? [17:46:07] no, just one [17:46:15] but--we can call using any harness. [17:46:25] or are we loading up a carb with jobs and feeding them through? [17:46:31] We can call the callback from a dequeue_loop, or from a json_batch_import [17:47:36] XenoRyet: thanks. PPena might know this. is there any documentation for payment methods for Israel? I see "cc, pp" in multi linigual master doc. is that authoritative? [17:48:03] awight: so does that mean that batching of jobs will happen worker-side? [17:48:12] so the queue and consumer don't have to care [17:48:18] dstrine we dont have anything special to offer to IL, so its just CC and Pats Pena [17:48:30] both in local currency [17:49:08] So just the usual four credit cards then? [17:49:10] PPena: "pats pena" is a new processor? ;) [17:49:57] hehe, chat macro? [17:50:52] PPena: just to confirm we're talking about Credit Card and PayPal for IL? [17:51:05] dstrine yep [17:51:25] my autocorrect is killing me! [17:51:26] ;) [17:51:33] cwd|brb: the interface to a job processor is currently just function($msg) throws Exception -- check this out: https://github.com/wikimedia/wikimedia-fundraising-crm/blob/master/sites/all/modules/wmf_common/Queue.php#L105-L110 [17:51:40] We can probably stick with that. [17:51:46] I write PP and freaking autocorrects- I must love myself! [17:52:26] it's cool... I'm a terrible typer :P [17:52:30] XenoRyet: ^ [17:52:54] Yep, got it. That's what I needed to know. [17:58:47] XenoRyet: Could you shoot me the documentation for custom Adyen iframe text? I'd like to understand what kind of devilish pact we're making ;) [17:59:25] awight: Sure, one sec to find the link. [18:01:08] It's like hearing about a college where everyone creates their own major or something. I have to be actually in their food co-op eating an eggplant sandwich before I can believe that something so wonderful can exist. [18:01:53] awight: i think i may be chasing a ghost; this queue consumer provides callbacks but the whole thing is still synchronous and serial? [18:02:01] and that's the desired operation? [18:02:06] awight: Actually seems kind of poorly documented. This link says where to do it: https://docs.adyen.com/developers/hpp-manual#extraoptions [18:03:10] XenoRyet: "Edit language files" ? [18:03:21] whoa [18:03:31] This is so futuristic [18:03:39] Yea, if you click that, it gives you a form to provide your own translations. [18:04:31] It's kind of clunky though, and there's aproxamately 14.8 gazillion strings to choose from. [18:05:08] cwd: The only bottlenecked consumer is the main donations one... and it's still fast enough that we don't need to parallelize any time soon [18:05:16] including, as Egegg discovered, the "Pay using 7-11" string. [18:05:24] Awesome! [18:05:28] Pay using slurpee [18:06:00] awight: ok i understand now [18:06:52] cwd: XenoRyet huh, so we can customize existing languages, and add multiple additional languages? [18:07:22] so the callback would commit? [18:07:28] awight: Yea, looks that way. If we can get a handle on this thing, we can do any language we want. [18:07:32] yeah the adyen iframe is fancy [18:08:54] cwd: I expect to keep commit() in the dequeue_loop layer, it would fire here, https://github.com/wikimedia/wikimedia-fundraising-crm/blob/master/sites/all/modules/wmf_common/Queue.php#L110 [18:09:41] XenoRyet: hrm! It sort of looks like we could download your resources_he.properties work, and start code-generating those files from our i18n armory [18:09:50] awight: that seems good [18:10:25] But I'm still hung up on what this interface should look like. commitOffset() is the top contender so far [18:10:41] That way nobody will expect it to work with push. ? [18:12:33] i don't understand what the intended effect with push would be? [18:12:49] exactly... Let's slam that door closed [18:13:17] yeah...i mean pdo will probably throw an exception if you call commit from not inside a txn [18:13:29] so that will pavlov the programmer away [18:13:44] if it doesn't, we can [18:14:02] i gotta run for a bit! [18:14:06] k thanks! [18:34:11] (PS5) AndyRussG: [WIP] Mixed storage for impression diet and large banner limit [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/285336 (https://phabricator.wikimedia.org/T132639) [18:38:40] (PS6) AndyRussG: [WIP] Mixed storage for impression diet and large banner limit [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/285336 (https://phabricator.wikimedia.org/T132639) [18:44:24] (CR) Ejegg: "I'm messing around with ways to fix https://phabricator.wikimedia.org/T105618 on the backend, and I'm starting to really like this approac" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/281084 (owner: Awight) [18:56:20] awight: mind if I pick that one up and try to use it for the min amount message? ^^ [19:04:43] Jeff_Green: are you about [19:04:48] yes [19:05:17] we have jumped on that google call [19:05:23] in the meetimg [19:05:35] ? [19:06:00] somehow I thought that was in an hour [19:06:08] enfiring the googles [19:50:50] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: [Epic] Keep track of new custom field creation - https://phabricator.wikimedia.org/T133739#2241114 (awight) [19:50:58] eileen: fwiw ^ [20:01:30] Jeff_Green: awight ejegg do we need to do a separate backup before we kick off next week? [20:02:09] we already do a nightly backup [20:02:27] so ... if that's recent enough than I think it's probably fine not to do an additional one [20:03:24] you know what we can do--let's stop replication to one of the slaves until things look good [20:03:38] that way we can use that as a basis for restore if necessary [20:04:42] cool [20:05:54] dstrine: we're BSsing if u wish to join [20:07:54] google chucked me off the call while it re-sends me that code to verify me... [20:11:35] Fundraising-Backlog: Silverpop export not capturing language changes in Civi (not sure if this is language-only or not) - https://phabricator.wikimedia.org/T96410#1216446 (Ejegg) We might even want to do a full export from Silverpop some time and see how it compares to civi data [20:15:48] Fundraising-Backlog: Silverpop export not capturing language changes in Civi (not sure if this is language-only or not) - https://phabricator.wikimedia.org/T96410#2241188 (CCogdill_WMF) Happy to help with that @Ejegg. Le mardi 26 avril 2016, Ejegg a écrit : > Ejegg adde... [20:19:37] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: [Tracking] CentralNotice: reducing initial JS load? - https://phabricator.wikimedia.org/T133741#2241197 (AndyRussG) [20:19:48] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Frontend maintenance flag could be gentler - https://phabricator.wikimedia.org/T133131#2221444 (DStrine) p:Triage>Normal [20:20:01] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: [Tracking] CentralNotice: reducing initial JS load? - https://phabricator.wikimedia.org/T133741#2241214 (AndyRussG) [20:20:03] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: CentralNotice: Use only one cookie library - https://phabricator.wikimedia.org/T133434#2241215 (AndyRussG) [20:20:38] Fundraising-Backlog, MediaWiki-extensions-DonationInterface, FR-Astropay: Client-side validation messages are not country specific - https://phabricator.wikimedia.org/T133706#2241218 (Ejegg) Crappy quick fix for AstroPay: add all the country-specific translations for fiscal_number to the messages mod... [20:21:05] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Get QA attention on DonationInterface - https://phabricator.wikimedia.org/T132526#2241234 (DStrine) p:Triage>Normal [20:21:38] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Frontend maintenance flag could be gentler - https://phabricator.wikimedia.org/T133131#2241237 (awight) One idea--the maintenance variable can hold a timestamp, and the frontend stops people from entering the pipeline 5 minutes prior, then hard... [20:23:46] Fundraising Tech Backlog, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, FR-Paypal, Patch-For-Review: Assisted currency conversion for PayPal is broken again - https://phabricator.wikimedia.org/T98447#1268061 (DStrine) [20:24:27] Fundraising Tech Backlog, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, FR-Amazon, and 3 others: Assisted currency conversion for PayPal is broken again - https://phabricator.wikimedia.org/T98447#1268061 (DStrine) [20:24:51] (PS2) Awight: [WIP] Change meaning of validation variables [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/280141 (https://phabricator.wikimedia.org/T98447) [20:28:25] awight: ejegg Jeff_Green I added this https://collab.wikimedia.org/wiki/Checklists_for_maintenance_%26_Recovery [20:28:43] thanks eileen ! [20:28:48] there is a link from https://collab.wikimedia.org/wiki/Fundraising/Engineering [20:29:31] cool. I'll put a link in motd on the servers [20:30:18] good idea [20:44:38] Fundraising Tech Backlog, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, FR-Amazon, and 3 others: Assisted currency conversion for PayPal is broken again - https://phabricator.wikimedia.org/T98447#1268061 (Ejegg) Looks like just settings: ``` $wgPaypalGatewayFallbackCurrency = 'USD... [20:58:33] Fundraising Sprint Asbestos Removal 2016, Fundraising Sprint Bloodletting 2016, Fundraising Sprint Cat Herding, Fundraising Sprint Dirt Farming, and 5 others: Some Ingenico donations not in Civi - https://phabricator.wikimedia.org/T122730#1912265 (DStrine) [21:00:45] ejegg: What were you gonna say earlier? [21:02:15] awight: did you by any chance bring crm/deployment up to date recently? [21:02:19] or eileen [21:02:23] I have not [21:02:37] awight oh, just wanted to take over your validation encapsulation patch [21:02:53] to use for the minimum amount error [21:03:05] ejegg: This one? https://gerrit.wikimedia.org/r/#/c/280141/ [21:03:35] Whatever it is, feel free! I've unhanded all that [21:04:29] nope, this one: https://gerrit.wikimedia.org/r/281084 [21:04:59] Humn. Yes please do [21:05:05] I'm curious to see where you can take it [21:05:21] Rockin! Basically just going to follow the lead [21:05:26] I like that method sig [21:05:56] I think validation and normalization both fit into this model-centric encapsulation paradigm [21:06:05] easy to set different error messages for a single field [21:06:07] normalization is more troublesome cos of reasons we discussed already... [21:06:23] and to have all of the variables + existing messages as input [21:06:23] cool, that sounds like a great use case [21:06:39] It's lazy but has been working handsomely for *staging [21:07:15] And I'm guessing anywhere the order matters, staging and validation have the same dependencies [21:07:27] That sounds right [21:07:39] rockin [21:08:23] (CR) Awight: [WIP] Explore encapsulating validation along with transformations (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/281084 (owner: Awight) [21:14:20] (PS1) Ejegg: Allow 7 digit fiscal numbers for AR [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/285522 [21:14:37] most trivial patch evar ^^^ [21:14:52] MBeat just relayed a donor complaint [21:15:44] heh, just found something that says they can be 10 digits, PS2 coming [21:15:58] trivial shmivial we’ll take it, ty ejegg [21:16:05] (CR) jenkins-bot: [V: -1] Allow 7 digit fiscal numbers for AR [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/285522 (owner: Ejegg) [21:16:52] oh wow, i even had a 7 digit number test that was supposed to fail! [21:17:32] Too bad about the error message, [21:17:34] 00:01:11.451 Failed asserting that true matches expected false. [21:17:47] At least it says, "expected". [21:18:11] hehe [21:18:13] Validation functions give me the willies when they return booleans [21:18:18] which boolean? [21:19:19] I don't think there's a correct boolean answer. [21:21:36] (PS2) Ejegg: Allow 7-10 digit fiscal numbers for AR [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/285522 [21:22:20] ^^ also improves error messages for the test that will be obsolete soon [21:26:46] (CR) Cdentinger: [C: 2] Allow 7-10 digit fiscal numbers for AR [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/285522 (owner: Ejegg) [21:26:52] cwd: What about this--the failure of any one record while importing a batch should not in any way (including a fatal error) affect the processing of otherwise successful records. [21:27:17] Now, how do we translate that into a batch workflow? [21:27:33] heh, well if it's a "catchable fatal error"... [21:27:42] nah even uncatchable [21:27:53] We should be able to restart the batch at the most recent successful offset [21:27:57] yeah i'm just playin [21:28:12] such a baroque architecture of exceptions. [21:28:56] how many are there for "this array behavior sucks"? [21:29:00] i don't think we can stop the app from shitting the bed on fatal errors right? [21:29:06] list and map are. almost the same thing. [21:29:23] so this implies something else shelling out php [21:29:27] that can recover [21:29:37] I don't think that's necessary [21:29:43] but a good idea nonetheless [21:30:13] When a fatal error kills the process, the next one that is spawned should begin exactly at or after the rejected message. [21:30:26] Maybe it retries three times. [21:30:34] where does it mark the tries? [21:31:03] same database as the client group offset is stored in :) [21:31:20] ahh, so that's interesting [21:31:23] and if they are in the same db, then we don't need two phase commit [21:31:28] that's sort of like a row status, no? [21:31:45] Let's scrap that for now [21:31:54] what about this: [21:32:53] consumer finds message, pops, schedules another message: "in process" before giving the job out. when finished, job runner schedules another message: "completed" [21:33:02] (PS4) Ejegg: [WIP] Explore encapsulating validation along with transformations [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/281084 (owner: Awight) [21:33:22] then all the parts are decoupled but without a race condition [21:33:30] well, move the "in process" message before the pop [21:33:58] The in process message can't be in a queue, this has to be random access [21:34:42] (CR) Ejegg: "PS4: manual rebase" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/281084 (owner: Awight) [21:34:49] Okay, I'm starting to get it. The pending db allows us to track row status for just the in-process things. [21:35:19] the worker could zap that after scheduling the "done" message [21:35:27] so that interstitial step is just for failure recovery [21:35:42] but you're right that it's not a queue [21:35:49] {row} + {time_became_pending, type, last_status, retry_count} [21:36:04] I agree with your "done" message. Each processor can be a pump between two queues [21:36:21] yeah [21:36:48] and i think everything is recoverable at any point [21:37:00] But in some cases, the message actually _is_ "done", right? After we import into the CRM, for example? [21:37:31] yeah i think that's when it pushes the done message? [21:38:03] Huh--since we get transactionality from the pending db, maybe we can omit commit() from the API? [21:38:38] Who consumes the done queue? [21:38:50] (Merged) jenkins-bot: Allow 7-10 digit fiscal numbers for AR [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/285522 (owner: Ejegg) [21:39:00] shrug [21:39:08] could be a new job that did what ever thing [21:39:11] send some emails or something? [21:40:02] I wonder if it's okay to have a large archive of unconsumed messages. I guess a daily cleanup script can shuffle those to files. [21:40:33] I think a new "done" queue would be a very natural starting point for analytics consumers. [21:40:39] what would those be? the failures? [21:40:48] oh hey--that's currently payments-initial [21:41:00] failures can be... payments-errors [21:41:05] Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Update trigger mysql in git to reflect latest change from upstream (connection_id) - https://phabricator.wikimedia.org/T133745#2241440 (Eileenmcnaughton) [21:42:42] 4 queues: initial/pending/error/complete ? [21:42:45] cwd sorry I didn't see that before - I cherry-picked the civicrm update to avoid your one - the only other one at the time [21:43:24] eileen: np! i saw that the sha1 was different between master and deployment but then i noticed it was because of the cherry pick [21:44:58] cool [21:45:21] eileen: did you deploy deployment? last i checked it looked like that branch was old and not used [21:46:13] yeah ... [21:46:25] how do you mean? [21:47:38] i could have sworn when i looked yesterday the last commit to crm/deployment was like 2 years ago [21:47:44] and i assumed we just deployed master [21:47:50] maybe civicrm/deployment? [21:48:17] aaah that's right! [21:48:20] you're right [21:48:22] sorry [21:48:37] can we delete the branch if it's hugely dated and confusing? [21:49:37] probably a good idea, unless awight or ejegg can think of a reason not to [21:50:24] wait, crm/deployment is not used? [21:50:28] lemme see [21:50:36] civicrm/deployment [21:50:42] ohhh [21:50:50] right, that can probably go [21:52:01] (PS5) Ejegg: Explore encapsulating validation along with transformations [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/281084 (owner: Awight) [21:54:32] gateway_common's getting pretty crowded [21:55:01] any suggestions for a dir name to hold the validation and staging helpers? [21:55:13] fair chance of crashing your computer with syntax highlighting [21:55:37] oh wait you mean the dir [21:55:44] heh, yeah [21:56:00] well, I gotta relocate - back in a few! [21:56:01] my first thought would be like ... helpers/validation [21:56:20] but this is the hardest problem in computer science [22:38:59] going to deploy that fiscal # length fix [22:39:17] Cool! [22:39:37] (PS1) Ejegg: Merge master into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/285540 [22:39:49] (CR) Ejegg: [C: 2 V: 2] Merge master into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/285540 (owner: Ejegg) [22:39:50] ejegg: "transformers" [22:40:18] We should use that for the class names too, and never the generic "helper" [22:40:23] err *interface names [22:40:37] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/285541 [22:40:50] (CR) Ejegg: [C: 2] Update DonationInterface submodule [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/285541 (owner: Ejegg) [22:41:27] transformersAndValidators :P [22:41:43] ayii [22:42:01] I had a fanciful name back way when--"membrane" [22:42:31] heh [22:42:43] But neologisms are sadly almost never the right thing to do [22:43:28] FieldFormat...Things? [22:46:04] or maybe validation and staging helpers that operate on the same field shouldn't be mashed together. [22:49:25] I could be talked out of it, but I really like the mashing together. They could also be both split and grouped, like DonorAddress\StagingTransformer, but that seems silly. [22:50:04] Or the old way, mashed and grouped by being staging functions, on whatever data. But I think we can rule that out as being a good idea. [22:50:38] i do like the topicality of mashed transformers [22:51:07] ping btw, https://gerrit.wikimedia.org/r/#/c/285222/ just cos I'd like to not rebase around that [22:51:46] I'm facing the same dilemma in PHP-Queue, I like having all of a backend's functions mashed together, but want to address various slices of the API separately. [22:52:21] For example, FifoWrapper::getProducer() returns a FifoDataStore [22:52:31] hrm, yeah. and those classes store a fair amount of state in local vars [22:52:46] and the backend class implements several different interfaces [22:53:23] Yeah. Backends could be done with helpers, the formal and javaesque way, but talk about pagination hell... that's :n hell [22:54:07] !log updated payments-wiki from 3ac3a0d7bb6e2d8443a87fb62bd5412a95f75aa2 to 16ed5af8c8544ea1c8d837ae16585eba4cbbfd4e [22:54:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:04:15] Fundraising-Backlog, FR-ActiveMQ: Upstream whatever we can to PHP-Queue - https://phabricator.wikimedia.org/T133754#2241639 (awight) [23:08:34] phooey, languageSpecificFallback is still sorta broken [23:09:01] awight: ah, looking at the style fix now [23:21:07] (CR) Ejegg: "I found Waldo in PaymentProviders/Worldpay/Audit/TransactionReconciliationFile.php . Now Brian Kernighan sends me a dollar, right?" (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/285222 (owner: Awight) [23:23:25] (CR) Awight: "The smoking gun and the glove! That way so cheesy of me... we even have this lovely builtin logging facility." [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/285222 (owner: Awight) [23:25:31] (CR) Ejegg: [C: 2] Make MediaWiki-compatible style changes [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/285222 (owner: Awight) [23:26:36] (Merged) jenkins-bot: Make MediaWiki-compatible style changes [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/285222 (owner: Awight) [23:49:31] (PS1) Cdentinger: frig smashpig -- updated composer.lock [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/285555 [23:50:47] (Abandoned) Cdentinger: frig smashpig -- updated composer.lock [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/285555 (owner: Cdentinger) [23:53:49] (PS7) Awight: [WIP] New interface for transactional reads [wikimedia/fundraising/php-queue] - https://gerrit.wikimedia.org/r/284977 (https://phabricator.wikimedia.org/T131271) [23:53:51] (PS1) Awight: Check success in push and create table if necessary [wikimedia/fundraising/php-queue] - https://gerrit.wikimedia.org/r/285556 [23:56:47] (PS8) Awight: New interface for transactional reads [wikimedia/fundraising/php-queue] - https://gerrit.wikimedia.org/r/284977 (https://phabricator.wikimedia.org/T131271)