[00:18:39] oh yeah. i forgot to say that i probably will miss standup tomorrow as i have to do kid drop and then drop the car at the shop. chances of me being somewhere with coverage right the time is pretty slim. [00:19:36] right at the time, that is. [02:33:10] (03PS1) 10Eileen: Add year column to wmf_donor_receipt [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/744904 (https://phabricator.wikimedia.org/T297159) [02:45:27] (03CR) 10jerkins-bot: [V: 04-1] Add year column to wmf_donor_receipt [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/744904 (https://phabricator.wikimedia.org/T297159) (owner: 10Eileen) [04:42:49] 10Fundraising Sprint Vagranty McVagrantface, 10Fundraising Sprint We all meet again, 10Fundraising-Backlog, 10SRE, and 6 others: Deal with donatewiki Thank You page launching in apps - https://phabricator.wikimedia.org/T259312 (10Tsevener) This fix is in new release candidate Testflight 6.8.2 (1868). [05:53:35] (03CR) 10jerkins-bot: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_37) - 10https://gerrit.wikimedia.org/r/744975 (owner: 10L10n-bot) [09:04:01] (03CR) 10Raimond Spekking: [V: 03+2 C: 03+2] "false positive" [extensions/DonationInterface] (REL1_37) - 10https://gerrit.wikimedia.org/r/744975 (owner: 10L10n-bot) [12:18:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: recurring is 7872 7500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 31 days 11 hours - memory use is 7.10M (peak 8.24M, 0.14% of max, fragmentation 1.69%), connected_slaves is 3, donations is 72, jobs is 0, jobs-adyen is 8, jobs-paypal is 120, payments-antifraud is 31, payments-init is 114, pending is 10, refund is 0, unsubscribe is 19 https://icinga.w [12:18:18] org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [12:23:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: recurring is 7536 7500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 31 days 11 hours - memory use is 7.22M (peak 8.24M, 0.14% of max, fragmentation 1.66%), connected_slaves is 3, donations is 428, jobs is 0, jobs-adyen is 11, jobs-paypal is 146, payments-antifraud is 24, payments-init is 6, pending is 7, refund is 0, unsubscribe is 25 https://icinga.wi [12:23:17] rg/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [12:28:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: recurring is 8004 7500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 31 days 11 hours - memory use is 7.38M (peak 8.28M, 0.15% of max, fragmentation 1.63%), connected_slaves is 3, donations is 378, jobs is 0, jobs-adyen is 16, jobs-paypal is 118, payments-antifraud is 28, payments-init is 140, pending is 15, refund is 1, unsubscribe is 35 https://icinga [12:28:17] a.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [12:33:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: recurring is 8470 7500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 31 days 11 hours - memory use is 7.44M (peak 8.69M, 0.15% of max, fragmentation 1.65%), connected_slaves is 3, donations is 131, jobs is 0, jobs-adyen is 17, jobs-paypal is 114, payments-antifraud is 28, payments-init is 7, pending is 13, refund is 0, unsubscribe is 40 https://icinga.w [12:33:17] org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [12:38:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: recurring is 8896 7500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 31 days 11 hours - memory use is 7.86M (peak 8.87M, 0.15% of max, fragmentation 1.58%), connected_slaves is 3, donations is 332, jobs is 0, jobs-adyen is 0, jobs-paypal is 118, payments-antifraud is 21, payments-init is 129, pending is 12, refund is 0, unsubscribe is 47 https://icinga. [12:38:18] .org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [12:43:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: recurring is 8530 7500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 31 days 11 hours - memory use is 7.47M (peak 9.13M, 0.15% of max, fragmentation 1.65%), connected_slaves is 3, donations is 262, jobs is 0, jobs-adyen is 36, jobs-paypal is 86, payments-antifraud is 43, payments-init is 11, pending is 17, refund is 0, unsubscribe is 54 https://icinga.w [12:43:17] org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [12:48:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: recurring is 8669 7500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 31 days 11 hours - memory use is 7.34M (peak 9.13M, 0.15% of max, fragmentation 1.67%), connected_slaves is 3, donations is 53, jobs is 0, jobs-adyen is 0, jobs-paypal is 37, payments-antifraud is 24, payments-init is 114, pending is 18, refund is 0, unsubscribe is 64 https://icinga.wi [12:48:17] rg/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [12:53:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: recurring is 8692 7500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 31 days 11 hours - memory use is 7.47M (peak 9.13M, 0.15% of max, fragmentation 1.65%), connected_slaves is 3, donations is 296, jobs is 0, jobs-adyen is 0, jobs-paypal is 29, payments-antifraud is 21, payments-init is 10, pending is 19, refund is 0, unsubscribe is 69 https://icinga.wi [12:53:17] rg/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [12:58:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: recurring is 8716 7500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 31 days 11 hours - memory use is 7.68M (peak 9.13M, 0.15% of max, fragmentation 1.65%), connected_slaves is 3, donations is 276, jobs is 0, jobs-adyen is 27, jobs-paypal is 38, payments-antifraud is 23, payments-init is 133, pending is 9, refund is 0, unsubscribe is 74 https://icinga.w [12:58:17] org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [13:03:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: recurring is 8092 7500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 31 days 11 hours - memory use is 6.99M (peak 9.13M, 0.15% of max, fragmentation 1.71%), connected_slaves is 3, donations is 95, jobs is 0, jobs-adyen is 2, jobs-paypal is 28, payments-antifraud is 34, payments-init is 13, pending is 16, refund is 0, unsubscribe is 4 https://icinga.wiki [13:03:17] /cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [13:08:18] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: recurring is 8118 7500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 31 days 12 hours - memory use is 7.50M (peak 9.13M, 0.15% of max, fragmentation 1.62%), connected_slaves is 3, donations is 272, jobs is 0, jobs-adyen is 0, jobs-paypal is 29, payments-antifraud is 27, payments-init is 122, pending is 16, refund is 0, unsubscribe is 12 https://icinga.w [13:08:18] org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [13:13:18] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: recurring is 8132 7500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 31 days 12 hours - memory use is 7.20M (peak 9.13M, 0.15% of max, fragmentation 1.68%), connected_slaves is 3, donations is 278, jobs is 0, jobs-adyen is 15, jobs-paypal is 33, payments-antifraud is 26, payments-init is 15, pending is 20, refund is 1, unsubscribe is 21 https://icinga.w [13:13:18] org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [13:18:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: recurring is 8156 7500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 31 days 12 hours - memory use is 7.37M (peak 9.13M, 0.14% of max, fragmentation 1.60%), connected_slaves is 3, donations is 59, jobs is 0, jobs-adyen is 3, jobs-paypal is 46, payments-antifraud is 47, payments-init is 134, pending is 13, refund is 0, unsubscribe is 28 https://icinga.wi [13:18:17] rg/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [14:03:18] RECOVERY - check_redis on frqueue1003 is OK: OK: REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 31 days 12 hours - memory use is 6.08M (peak 9.13M, 0.13% of max, fragmentation 1.85%), connected_slaves is 3, donations is 78, jobs is 0, jobs-adyen is 10, jobs-paypal is 54, payments-antifraud is 7, payments-init is 7, pending is 12, recurring is 6198, refund is 0, unsubscribe is 3 https://icinga.wikimedia.org/cgi-bin/ [14:03:18] tinfo.cgi?type=2&host=frqueue1003&service=check_redis [14:15:58] Jeff_Green: damilare is getting 'Service access denied due to missing privileges.' when trying to login to icinga [14:16:11] we're looking at this alert https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis&scroll=25 [14:16:45] also it looks like the icinga-wm bot is cutting off the wikimedia.org from the urls [14:16:48] Ok, is this new, has he been able to log in there before? [14:17:07] No I haven't attempted a log in before [14:17:09] ha I'm not surprised about the bot :-P [14:17:32] ok, I think you'll have to do an access request to SRE [14:17:53] that's just a phabricator task, I think there's an SRE-Access-Requests tag or similar [14:18:54] Ok let me check that [14:20:01] looking at the URLs above, I think the overall message is just too long for the bot to handle in a single message, our best bet there is probably to shorten the rest of the report to leave room for the url, which I think is probably not desireable [14:39:24] (03CR) 10Jgleeson: [C: 03+2] "We tested this and it worked as expected. Nice work!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/741733 (https://phabricator.wikimedia.org/T291223) (owner: 10Ejegg) [14:41:32] (03Merged) 10jenkins-bot: Option to return breakdown rows for payments_fraud [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/741733 (https://phabricator.wikimedia.org/T291223) (owner: 10Ejegg) [14:46:00] (03CR) 10Jgleeson: [C: 03+2] "We tested the code that uses these changes and it worked as expected! nice work" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/741738 (owner: 10Ejegg) [14:47:59] (03Merged) 10jenkins-bot: Create multiple tables for one DataStore class [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/741738 (owner: 10Ejegg) [15:32:50] (03CR) 10Damilare Adedoyin: "Hi Elliot, nice work on the updates to the smashpigDatabase and paymentFraud class. Jack and I have been reviewing the build up patches th" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/741740 (owner: 10Ejegg) [15:40:31] Error: DATABASE_CONTENTION Contribution not saved due to database load [15:40:33] :O [15:55:32] fr-tech not really sure if we can do anything about those deadlock emails [15:56:42] I think it only affects those transactions mentioned in the email [15:57:00] and they end up in the damage table [16:01:31] hmm we got another one [16:02:11] looks like the last one was the smashpig recurring job [16:02:38] I guess that's hitting the same tables as the donation queue consumer jobs? [16:03:09] hmmm [16:03:19] I'm gonna see if I can trace that through [16:19:02] hmm so that error is happening in the internals of the wmf_civicrm_contribution_message_import [16:19:19] I can't see anything specifically in smashpig that calls that code. i can see that the recurring queue consumer calls it [16:20:07] so the source name is Smashpig, so I think these are messages dropped on the recurring queue by the smashpig listeners [16:20:13] and the host is civi1001 [16:20:26] which makes me think this is the recurring queue consumer trying to save the recurring donation [16:20:47] which makes sense as paypal sent us a wave of recurring messages today I think [16:23:10] so I THINK the issue is that we've got competing queue consumers both processing donations, the donations_queue_consumer and the recurring_queue_consumer and since the volume is high from both directions, we're getting deadlocks when each of them tries to write a transaction to wmf_contribution_extra at the same time [16:25:59] which isn't too bad as the loser in the race gets their message dumped in the damaged db tbl [16:29:55] hmm they're still trickling through fr-tech [16:30:15] looks like we have 56 messages on the donations queue and 2000+ on the recurring queue [16:31:01] I guess we could switch off the queue consumers for either of those queues temporarily to allow one to finish [16:31:40] ah the donations queue count just shot up to 300 [16:33:01] any thoughts fr-tech? [16:33:42] jgleeson we could pause the donations queue consumer for a bit [16:34:40] for the longer term, theres a ticket to send all payment messages to the same queue, qnd leave the recurring queue just for subscr start / end / failure messages [16:35:02] sounds good ejegg|away. I'll push up the process-control job update! [16:36:27] ahh, we should do a bit of cleanup on that damaged table too - mass requeue what has a chance of working and deleting some obvious junk (like most of the paypal/pending messages in there) [16:36:40] i can do that cleanup in a half hr or so [16:38:19] cool [16:39:44] fr-tech there's an update to the process-control project repo on frpm1001 which comments out the donations queue consumer job schedule which effectively switches off the scheduled task/cron. [16:39:57] if someone would like to review that I can push it out [16:40:26] I'm leaving in 5 minutes so if it doesn't get reviewed and go out I'll check back in on it when I get back from taking Oscar to boxing [16:40:47] thanks btw ejegg|away [17:01:11] 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog, 10fundraising sprint Wireless Zipline: Amazon donor name shenanigans - https://phabricator.wikimedia.org/T297234 (10DStrine) p:05Triage→03High [17:01:19] 10Fundraising Sprint Visual C Saw, 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog, 10fundraising sprint Universal Cereal Bus, and 4 others: Switch eoy email over fully to the new method - https://phabricator.wikimedia.org/T297159 (10DStrine) p:05Triage→03High [17:01:30] 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog: Enable South Africa through Dlocal - https://phabricator.wikimedia.org/T293508 (10DStrine) p:05Triage→03Medium [17:01:36] 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog: Adyen forms for Euro Bundle - https://phabricator.wikimedia.org/T293292 (10DStrine) p:05Triage→03High [17:01:46] 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog, 10FR-LATAM: Adding PIX as a payment type for Brazil - https://phabricator.wikimedia.org/T280303 (10DStrine) p:05Triage→03Medium [17:01:54] 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog, 10FR-India, 10MW-1.36-notes (1.36.0-wmf.2; 2020-07-28): re-enable AMEX cards in India - https://phabricator.wikimedia.org/T258509 (10DStrine) p:05Triage→03Medium [17:02:04] 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog, 10FR-PayPal-ExpressCheckout, 10MediaWiki-extensions-DonationInterface: Payments-wiki should send payments-init message on failed PayPal EC - https://phabricator.wikimedia.org/T249276 (10DStrine) p:05Triage→03Medium [17:02:13] 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog: Paypal Orphan Slayer Missing PayerID - https://phabricator.wikimedia.org/T205566 (10DStrine) p:05Triage→03Medium [17:09:31] 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog: Adyen forms for Euro Bundle - https://phabricator.wikimedia.org/T293292 (10DStrine) [17:10:07] 10Fundraising Sprint Vestigial tails shoot from the hip, 10Fundraising Sprint Window dressing is mostly olive oil, 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog, and 4 others: Switch PE and UY to dlocal for processing - https://phabricator.wikimedia.org/T208963 (10DStrine) [17:10:15] 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog: Adyen forms for Euro Bundle - https://phabricator.wikimedia.org/T293292 (10DStrine) [17:12:24] 10fundraising-tech-ops, 10observability: check_mysql on fr* is extremely spammy - https://phabricator.wikimedia.org/T296811 (10Jgreen) 05Open→03Resolved a:03Jgreen As far as I can tell Dallas's monitoring adjustments have reigned in the noise as much as is feasible for the moment. Software fixes are in t... [17:12:29] 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog: Enable South Africa through Dlocal - https://phabricator.wikimedia.org/T293508 (10DStrine) [17:30:57] 10Fundraising Sprint Visual C Saw, 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog, 10fundraising sprint Universal Cereal Bus, and 3 others: set up catch-all mail handling on fundraising staging server - https://phabricator.wikimedia.org/T297304 (10Jgreen) [17:48:52] 10Fundraising Sprint Visual C Saw, 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog, 10fundraising sprint Universal Cereal Bus, and 3 others: set up catch-all mail handling on fundraising staging server - https://phabricator.wikimedia.org/T297304 (10Jgreen) Two changes were made to the sys... [17:50:32] 10Fundraising-Backlog, 10fundraising sprint Wireless Zipline: Amazon donor name shenanigans - https://phabricator.wikimedia.org/T297234 (10DStrine) p:05High→03Medium [18:02:07] ooh fr-tech there are a few adyen things in the damaged queue with a new error: CIVI_REQ_FIELD Required Field 'gateway_txn_id' not found in message. [18:02:10] looking into em [18:04:07] hmm, they look like they're from the recurring processor [18:04:32] hmm I thought that was something we ended up fixing [18:05:23] ohhhh [18:05:28] right, i see the date now [18:05:33] August [18:05:39] sorry for the false alarm [18:05:55] I guess I'll delete 'em [18:06:30] thanks cstone [18:07:25] I just looked up a random one and it's been charging happily [18:07:48] ooh is this the less than 1.00 thing [18:08:01] yep [18:08:03] yep yep [18:08:27] dates from 6/11 to 8/25 [18:08:32] so definitely not happening any more [18:08:37] yeah where is the task [18:09:14] https://phabricator.wikimedia.org/T285396 ? [18:10:25] ahh okay yes [18:13:24] k, i reverted that p-c patch since we don't need to turn the comsumer off after all [18:42:29] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [18:45:13] looks like amazon may be having issues again. the minFraud messages ^^ are from the codfw hosts. keeping an eye on it for now. [18:45:42] thanks dwisehaupt [18:47:03] dwisehaupt: I just tried to ack that alert and got frav1002 check_log_messages [18:47:05] Not Authorized [18:47:16] actually, it may not be amazon, but still the alerts are only in codfw. [18:47:29] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [18:54:26] 10Fundraising-Backlog: Storing many copies of the same file in civicrm (Planned_Giving_Guide, etc) - https://phabricator.wikimedia.org/T297308 (10Dwisehaupt) [19:13:21] fr-tech: doing a grafana upgrade on frmon for a security update. should be minimal impact. [19:14:06] upgrade done. [19:14:12] let me know if you see anything odd. [19:21:50] thanks! [19:28:20] 10Fundraising-Backlog: Paypal donations from IPN mistakenly mapped as recurring - https://phabricator.wikimedia.org/T297311 (10Damilare) [19:30:41] 10Fundraising-Backlog, 10fundraising sprint Wireless Zipline: Transactions at status 600 - https://phabricator.wikimedia.org/T296956 (10EMartin) 05Resolved→03Open Hi, for transactions of the 7th of December, after the fix went into place, we are seeing a couple of new scenarios: - while we see 320 transact... [19:33:19] 10Fundraising Sprint Visual C Saw, 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog, 10fundraising sprint Universal Cereal Bus, and 3 others: set up catch-all mail handling on fundraising staging server - https://phabricator.wikimedia.org/T297304 (10DStrine) p:05Triage→03Medium [19:50:39] 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog: Enable South Africa through Dlocal - https://phabricator.wikimedia.org/T293508 (10EMartin) ZA ID: The ID in the ZA is mandatory for all our merchants processing in ZA, and since it was implemented Dlocal has not seen any impact on conversion r... [19:57:11] 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog, 10fundraising sprint Wireless Zipline: Transactions at status 600 - https://phabricator.wikimedia.org/T296956 (10DStrine) [20:06:42] (03PS2) 10Eileen: Add year column to wmf_donor_receipt [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/744904 (https://phabricator.wikimedia.org/T297159) [20:18:14] (03CR) 10jerkins-bot: [V: 04-1] Add year column to wmf_donor_receipt [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/744904 (https://phabricator.wikimedia.org/T297159) (owner: 10Eileen) [20:22:41] damilare: I think the WIP: antifraud tests is still gonna break even after we merge ejegg|food's patch [20:22:56] it looks like there's a couple of references to PaymentsFraudBreakdownDatabase [20:23:06] do you wanna take those out [20:23:15] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/743467 [20:23:40] I just noticed it when running tests locally after pulling in the latest unmerged smashpig code. the tests were still failing [20:23:58] (03PS14) 10Jgleeson: Resolve pending transactions w/o DonationInterface [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/722703 (https://phabricator.wikimedia.org/T291223) (owner: 10Ejegg) [20:24:00] (03PS20) 10Jgleeson: Add tests for Pending Transaction Consumer and Resolver [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/739354 (https://phabricator.wikimedia.org/T291223) [20:24:31] (03Abandoned) 10Jgleeson: testResolveCreatesValidPaymentsInitMessage [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/742530 (https://phabricator.wikimedia.org/T291223) (owner: 10Jgleeson) [20:26:15] I'm just gonna rebase your WIP patch over the latest consolidated test & implementation patches. I just added the payments-init stuff to those two [20:26:33] (03PS9) 10Jgleeson: WIP: testAntiFraudQueueMessageCreatedAfterHostedStatusCall [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/743467 (https://phabricator.wikimedia.org/T291223) (owner: 10Damilare Adedoyin) [20:26:58] that's now a less noisy patch chain ^ [20:28:28] hmm let's make those patch names a bit more specific [20:30:07] (03PS15) 10Jgleeson: Pending Transaction Resolver: Implementation [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/722703 (https://phabricator.wikimedia.org/T291223) (owner: 10Ejegg) [20:30:09] (03PS21) 10Jgleeson: Pending Transaction Resolver: Tests [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/739354 (https://phabricator.wikimedia.org/T291223) [20:30:11] (03PS10) 10Jgleeson: WIP: testAntiFraudQueueMessageCreatedAfterHostedStatusCall [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/743467 (https://phabricator.wikimedia.org/T291223) (owner: 10Damilare Adedoyin) [20:34:37] calling it a day. have a good one fr-tech, bye for now! [21:07:29] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [21:12:29] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [21:31:14] (03PS1) 10Eileen: Fix error - use custom fields not core fields for original currency [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/745330 (https://phabricator.wikimedia.org/T297159) [21:31:57] I just found (through updating tests) a serious bug in the end of year summary code ^^ - would be good to get it out there quick-ish [21:32:45] hmm there may be a test battle ahead though [21:43:13] (03CR) 10jerkins-bot: [V: 04-1] Fix error - use custom fields not core fields for original currency [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/745330 (https://phabricator.wikimedia.org/T297159) (owner: 10Eileen) [21:52:13] (03PS2) 10Eileen: Fix error - use custom fields not core fields for original currency [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/745330 (https://phabricator.wikimedia.org/T297159) [22:03:09] (03CR) 10jerkins-bot: [V: 04-1] Fix error - use custom fields not core fields for original currency [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/745330 (https://phabricator.wikimedia.org/T297159) (owner: 10Eileen) [22:16:16] (03PS3) 10Eileen: Fix error - use custom fields not core fields for original currency [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/745330 (https://phabricator.wikimedia.org/T297159) [22:27:48] (03CR) 10jerkins-bot: [V: 04-1] Fix error - use custom fields not core fields for original currency [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/745330 (https://phabricator.wikimedia.org/T297159) (owner: 10Eileen) [22:59:10] (03PS4) 10Eileen: Fix error - use custom fields not core fields for original currency [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/745330 (https://phabricator.wikimedia.org/T297159) [23:04:09] (03PS1) 10Eileen: Set original currency details in test [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/745336 (https://phabricator.wikimedia.org/T297159) [23:05:14] (03PS1) 10Eileen: Test readability improvement [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/745337 (https://phabricator.wikimedia.org/T297159) [23:06:04] It would be great to get a +2 on https://gerrit.wikimedia.org/r/745336 - since there is some test dependency between the main repo & civi repo & juggling [23:09:45] (03CR) 10jerkins-bot: [V: 04-1] Fix error - use custom fields not core fields for original currency [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/745330 (https://phabricator.wikimedia.org/T297159) (owner: 10Eileen) [23:50:29] (03CR) 10Cstone: [C: 03+2] "yay tests" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/745336 (https://phabricator.wikimedia.org/T297159) (owner: 10Eileen) [23:51:41] thanks cstone - I'm having a wee fight locally but at least that will help me see if CI will accept the next patch with just that [23:52:02] im going to be making dinner but ill check back periodically [23:55:08] (03PS1) 10Eileen: Fix field name [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/745340 (https://phabricator.wikimedia.org/T297159) [23:55:27] cstone: argh - my local probs were in that patch you just approved see ^^ [23:55:49] ahh okay [23:56:11] do you need that whole chain?