[01:03:14] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 95.86, 70.67, 34.80 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [01:05:16] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 88.03, 80.48, 42.73 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [01:08:12] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 29.66, 56.90, 39.88 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [01:10:16] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 65.91, 82.12, 54.90 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [01:13:12] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 3.36, 26.01, 31.58 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [01:15:16] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 22.70, 50.47, 49.44 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [01:20:16] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 2.03, 19.80, 36.37 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [01:38:12] RECOVERY - check_load on frban1002 is OK: LOAD OK - total load average: 3.14, 3.60, 9.03 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [01:50:16] RECOVERY - check_load on frban2002 is OK: LOAD OK - total load average: 1.80, 3.58, 8.19 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [08:27:30] (03PS1) 10Eileen: Improve guzzle timeout support [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1164949 (https://phabricator.wikimedia.org/T379878) [10:23:52] 10fundraising-tech-ops: Q1:rack/setup/install frand200[12] - https://phabricator.wikimedia.org/T367804#10957902 (10Clement_Goubert) [12:24:47] 10fundraising-tech-ops, 10decommission-hardware: Decommission frack hosts: frnetmon1001 - https://phabricator.wikimedia.org/T398079#10958339 (10Jgreen) a:03Jgreen [12:29:24] 10fundraising-tech-ops, 10decommission-hardware: Decommission frack hosts: frnetmon1001 - https://phabricator.wikimedia.org/T398079#10958390 (10Jgreen) [12:56:18] 10fundraising-tech-ops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad: Decommission frack hosts: frnetmon1001 - https://phabricator.wikimedia.org/T398079#10958475 (10Jgreen) a:05Jgreen→03None [13:27:26] (03CR) 10Jgleeson: [C:03+2] "LGTM! I also took a peek at https://github.com/mrmarkfrench/silverpop-php-connector/commit/ac02533a7f15f375a41fec57309d97d5c9515a75" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1164949 (https://phabricator.wikimedia.org/T379878) (owner: 10Eileen) [13:45:16] (03Merged) 10jenkins-bot: Improve guzzle timeout support [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1164949 (https://phabricator.wikimedia.org/T379878) (owner: 10Eileen) [14:25:27] hi fr-tech! [14:26:27] It looks like we've fixes most of the problems that were failing validation on recurring donations before the API call [14:26:41] mostly seeing bad Venmo tokens in those failmails now [14:27:11] So how about we tone down the log level on those validation errors when there is a RawResponse, i.e. we have made an API call? [14:28:57] The api errors are returned as a PaymentProviderResponse with the error property filled [14:34:28] ahh so it's thrown in SmashPig payment processor in Civi [14:34:52] yep yep, we just added that to make sure we didn't miss any more errors like we've been having with Gravy [14:35:22] where our validator code rejects the payment before we even make the API call [14:35:49] So I'm guessing those Venmo ones are normal failures where we don' [14:36:06] ... where the donor cancels in the app? [14:36:41] I guess I could check to make sure we have a healthy amount of recurring Venmos before I silence those failmails. [14:56:14] right, looks pretty healthy I think [14:56:35] ones in status 5 have an avg of 10 installments each [15:04:55] (03PS1) 10Ejegg: Stop recurring failmail when making API calls [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165040 [15:05:13] ok, ^^^ will quiet down the failmails we don't need to hear about [15:08:47] (03PS1) 10Ejegg: Update to Gravy 0.27 [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165041 [15:18:38] (03PS1) 10Ejegg: Pass through user IP for gravy auths [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165047 (https://phabricator.wikimedia.org/T397763) [15:19:32] (03CR) 10CI reject: [V:04-1] Pass through user IP for gravy auths [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165047 (https://phabricator.wikimedia.org/T397763) (owner: 10Ejegg) [15:25:26] (03PS2) 10Ejegg: Pass through user IP for gravy auths [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165047 (https://phabricator.wikimedia.org/T397763) [15:27:58] (03CR) 10Jgleeson: [C:03+2] "LGTM!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165040 (owner: 10Ejegg) [15:28:03] makes sense to me ejegg [15:28:39] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10959219 (10Jgreen) [15:29:54] (03CR) 10Wfan: [V:03+2 C:03+2] Synchronize standalone using wmf [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1160979 (owner: 10Damilare Adedoyin) [15:49:05] (03Merged) 10jenkins-bot: Stop recurring failmail when making API calls [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165040 (owner: 10Ejegg) [15:56:12] thanks wfan [16:02:51] (03CR) 10Ejegg: [C:04-1] Pass through user IP for gravy auths [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165047 (https://phabricator.wikimedia.org/T397763) (owner: 10Ejegg) [16:03:02] (03CR) 10Jgleeson: [C:03+2] "LGTM!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165047 (https://phabricator.wikimedia.org/T397763) (owner: 10Ejegg) [16:03:37] ejegg: something wrong with that? [16:05:07] (03CR) 10Ejegg: [C:04-1] ""Extra inputs are not permitted" - oops, need to remove this from the params array" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165047 (https://phabricator.wikimedia.org/T397763) (owner: 10Ejegg) [16:05:25] (03CR) 10Ejegg: [C:04-2] Pass through user IP for gravy auths [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165047 (https://phabricator.wikimedia.org/T397763) (owner: 10Ejegg) [16:05:52] thanks jgleeson ! [16:05:52] (03CR) 10Jgleeson: Pass through user IP for gravy auths [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165047 (https://phabricator.wikimedia.org/T397763) (owner: 10Ejegg) [16:06:06] It's actually an interesting twist for our 'mapper' model [16:06:20] since we now might want to mapper to output two arrays :P [16:06:22] ejegg: I took a look at https://github.com/gr4vy/gr4vy-php/commit/5047f9519d7bb7990441ae04c35c5b3cd352f3cb [16:06:33] your param looked good to me [16:06:37] does it not work in the wild? [16:06:47] yeah, in the wild it gets an error from the Gravy API [16:06:59] ah [16:07:01] if we have it both in the header and in the posted parameters [16:07:09] so I need to strip it out of the posted parameters [16:10:14] ok, adding an 'unset' line seems to make it work [16:10:24] just gonna look at the sandbox to see if the IP shows up [16:17:25] hmm, not sure where exactly it would be in the UI [16:17:47] Oddly there are two x-forwarded-for headers on the auth api call in the API logs [16:18:04] lemme debug through it and make sure the code is doing what I expect [16:21:49] ohhh, it needs the headers in a different format too [16:27:23] ok, that added my internal IP address to the x-forwarded-for list on the API log [16:27:36] but i'm still not seeing it anywhere else in the UI [16:27:40] maybe it's a search param? [16:28:11] yep, and it worked! [16:28:16] ok, pushing up amended patch [16:28:19] (03PS3) 10Ejegg: Pass through user IP for gravy auths [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165047 (https://phabricator.wikimedia.org/T397763) [16:28:29] ok jgleeson, it seems to work that way ^^^ [16:28:58] I did the gravy SDK update in a parent patch, but i'd be happy to squash that in if it's better that way [16:29:10] lemme remove my C-2 [16:29:37] 10fundraising-tech-ops, 10FR-Tech-Analytics: Migrate Fundraising analytics process-control jobs to Dagster - https://phabricator.wikimedia.org/T397947#10959710 (10Jgreen) [16:29:37] (03CR) 10Ejegg: "OK, this version seems to work in smoke tests." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165047 (https://phabricator.wikimedia.org/T397763) (owner: 10Ejegg) [16:31:28] 10fundraising-tech-ops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: Decommission frack hosts: civi2001 - https://phabricator.wikimedia.org/T397380#10959726 (10Jgreen) a:05Dwisehaupt→03None [16:33:03] ah yeah jgleeson I initially tried to upgrade to 1.0.2 but it seems to require a pretty big change [16:35:12] gravy sdk 1.x also has a few more dependencies [16:35:48] I guess we're still getting enough out of it that we wouldn't want to craft our own HTTP requests? [16:39:51] 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 06FR-donorrelations, 13Patch-For-Review, 07payments-orchestration: pass IP info to Gravy to enable search - https://phabricator.wikimedia.org/T397763#10959776 (10Ejegg) [16:41:59] (03PS1) 10Ejegg: Upgrade phpseclib and some silverpop libs [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1165061 [16:42:01] (03CR) 10Ejegg: [C:03+2] Upgrade phpseclib and some silverpop libs [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1165061 (owner: 10Ejegg) [16:42:02] ejegg: maybeeee [16:42:46] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1165062 [16:43:34] jgleeson: I guess even if we did that we'd want to use that same jwt library they're depending on [16:44:25] will deploy Civi after lunch [16:45:29] I'd prefer not to rely on the SDK, but as they seem to be adding features to it frequently, integrating directly would put us on the hook to incorporate any necessary updates as they occur, and we'd have to keep a close eye on the API to be aware of changes. [17:00:14] (03Merged) 10jenkins-bot: Upgrade phpseclib and some silverpop libs [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1165061 (owner: 10Ejegg) [17:11:55] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Gravy Card Payment AVS check triggering false positives. - https://phabricator.wikimedia.org/T396355#10959893 (10jgleeson) Gravy has provided a report on the affected transactions [[ https://wikimedia.slack.com/archives/C070F1DVBRN/p... [17:25:30] (03PS1) 10Wfan: chmod for scripts [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1165066 [17:25:58] (03PS2) 10Wfan: instead of update primary email via EPC add a email confirm step [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1164497 (https://phabricator.wikimedia.org/T396523) [17:38:50] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Multi-use payment method ID is invalid or could not be found - venmo - https://phabricator.wikimedia.org/T397995#10960047 (10Ejegg) I think maybe this is just what happens when a Venmo user deactivates a recurring payment from their end, but I'm not sure.... [17:43:11] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 05WMF-NDA: Multi-use payment method ID is invalid or could not be found - venmo - https://phabricator.wikimedia.org/T397995#10960075 (10Ejegg) [17:44:34] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 05WMF-NDA: Multi-use payment method ID is invalid or could not be found - venmo - https://phabricator.wikimedia.org/T397995#10960081 (10AnnWF) [17:45:20] (03CR) 10CI reject: [V:04-1] instead of update primary email via EPC add a email confirm step [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1164497 (https://phabricator.wikimedia.org/T396523) (owner: 10Wfan) [17:46:21] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 05WMF-NDA: Multi-use payment method ID is invalid or could not be found - venmo - https://phabricator.wikimedia.org/T397995#10960084 (10AnnWF) [17:52:53] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1165062 (owner: 10Ejegg) [17:53:43] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1165062 (owner: 10Ejegg) [18:13:11] !log fundraising civicrm upgraded from e35d3778 to 2a8460ed [18:13:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:17:09] ack fr-tech, what's all this payments gravy failmail? [18:17:32] invalid payment method passed in: bancomer [18:17:39] ok, where do we need to add that? [18:23:31] (03PS1) 10Ejegg: Add bancomer [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165093 [18:23:47] fr-tech ^^^ might stop the active failmail [18:27:05] (03CR) 10Damilare Adedoyin: [C:03+2] Add bancomer [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165093 (owner: 10Ejegg) [18:27:10] thanks damilare ! [18:27:17] snp [18:27:22] np! [18:27:43] (03Merged) 10jenkins-bot: Add bancomer [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165093 (owner: 10Ejegg) [18:31:47] (03PS1) 10Ejegg: Update SmashPig [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1165095 [18:31:49] (03CR) 10Ejegg: [C:03+2] Update SmashPig [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1165095 (owner: 10Ejegg) [18:32:46] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1165105 [18:32:48] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1165105 (owner: 10Ejegg) [18:35:42] hi ejegg, the fix might be incomplete. I'd push a patch up shortly [18:35:51] oh ok [18:38:36] ah does it need to go in the CAPTURE_ONLY_PAYMENT_METHOD too? [18:41:14] (03PS1) 10Damilare Adedoyin: Add bancomer to reference data BT list [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165108 [18:41:40] ejegg it should be listed in the BT submethods list in reference data [18:41:53] ah ok [18:42:10] well, I wonder how we could make it easier to add a new one with changes in just one place [18:43:00] (03CR) 10Ejegg: [C:03+2] Add bancomer to reference data BT list [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165108 (owner: 10Damilare Adedoyin) [18:43:06] yea that would be really useful [18:43:13] (03PS2) 10Ejegg: Update to Gravy 0.27 [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165041 [18:43:17] (03PS4) 10Ejegg: Pass through user IP for gravy auths [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165047 (https://phabricator.wikimedia.org/T397763) [18:43:32] (03Merged) 10jenkins-bot: Add bancomer to reference data BT list [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165108 (owner: 10Damilare Adedoyin) [18:44:05] damilare or jgleeson since we've got to tag a new version anyway, should we try to get those updates in to pass the user IP ? [18:44:16] sure [18:44:22] does it work now ejegg ? [18:45:01] yep jgleeson ! [18:45:11] Even lets you search with ip= in the console [18:45:23] I guess it would be prudent just to update composer locally and put through a test! [18:45:33] after +2ing it last time when it was broke [18:46:10] I can do that shortly if that helps. I'm just testing the paypal cancellation flow for the demo on today's call [18:46:39] yay for demo-able tickets! [18:46:54] I guess I'll demo this one too even if it doesn't get deployed [18:48:17] (03CR) 10CI reject: [V:04-1] Update SmashPig [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1165095 (owner: 10Ejegg) [18:49:00] ooh we need to loosen up that psr/container requirement [18:51:08] (03PS1) 10Ejegg: Loosen up psr/container requirement [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165109 [18:51:27] fr-tech we'll need ^^^ to actually be able to update SmashPig in DI [18:54:39] (03CR) 10CI reject: [V:04-1] Loosen up psr/container requirement [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165109 (owner: 10Ejegg) [18:55:31] oh needs a lock file update [18:56:15] ah and that'll conflict with the patch to update gravy sdk [18:56:20] will put that on top of it then [18:56:46] (03PS2) 10Ejegg: Loosen up psr/container requirement [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165109 [18:58:41] (03CR) 10Damilare Adedoyin: [C:03+2] Update to Gravy 0.27 [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165041 (owner: 10Ejegg) [18:59:09] (03CR) 10Damilare Adedoyin: [C:04-1] Update to Gravy 0.27 [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165041 (owner: 10Ejegg) [18:59:19] (03PS3) 10Ejegg: Update to Gravy 0.27 [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165041 [18:59:19] (03PS5) 10Ejegg: Pass through user IP for gravy auths [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165047 (https://phabricator.wikimedia.org/T397763) [18:59:37] sorry ejegg +2'd the wrong patch [18:59:50] ah ok, i was rebasing that one anyway [19:00:03] (03CR) 10Damilare Adedoyin: [C:03+2] Loosen up psr/container requirement [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165109 (owner: 10Ejegg) [19:00:12] thanks! [19:00:33] (03Merged) 10jenkins-bot: Loosen up psr/container requirement [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165109 (owner: 10Ejegg) [19:00:41] np is the gravy version bumbp ready to go out? [19:00:54] damilare: it's a very small version bump [19:01:07] just needed for the extra header we add in the follow-on patch [19:01:15] okk [19:01:50] (03CR) 10Damilare Adedoyin: [C:03+2] Update to Gravy 0.27 [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165041 (owner: 10Ejegg) [19:02:55] (03Merged) 10jenkins-bot: Update to Gravy 0.27 [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165041 (owner: 10Ejegg) [19:03:55] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Donations Set Up in May Failing Second Instalment (Japan/Gravy) - https://phabricator.wikimedia.org/T397002#10960390 (10SHust) @Ejegg I'll be filing a Phab ticket for the email send, and wanted to check, has the list you shared been reviewed to remove any... [19:03:55] well, I can tag that version without the IP addresses for now, just to stop losing all the bancomer donors [19:23:48] 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 07fr-current-sprint, 10MW-1.45-notes (1.45.0-wmf.8; 2025-07-01): Gravy dLocal handle fiscal numbers with waivers - https://phabricator.wikimedia.org/T395361#10960482 (10XenoRyet) 05Open→03Resolved [19:23:53] (03Merged) 10jenkins-bot: Pass through user IP for gravy auths [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1165047 (https://phabricator.wikimedia.org/T397763) (owner: 10Ejegg) [19:24:03] 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 07fr-current-sprint, 10MW-1.45-notes (1.45.0-wmf.7; 2025-06-24): Enable Bank transfer in Peru - https://phabricator.wikimedia.org/T393987#10960493 (10XenoRyet) 05Open→03Resolved [19:24:06] 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 07fr-current-sprint, 07payments-orchestration: Gravy dLocal Internal Test Plan - https://phabricator.wikimedia.org/T389547#10960496 (10XenoRyet) 05Open→03Resolved [19:24:16] 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donor-portal, and 2 others: Implement recurring donation pause - https://phabricator.wikimedia.org/T388751#10960499 (10XenoRyet) 05Open→03Resolved [19:26:08] (03PS2) 10Ejegg: Update SmashPig [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1165095 [19:26:11] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Donations Set Up in May Failing Second Instalment (Japan/Gravy) - https://phabricator.wikimedia.org/T397002#10960526 (10XenoRyet) 05Open→03Resolved [19:26:34] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 10Fr-drupal-upgrade-2025: Snooze field needs fixing - https://phabricator.wikimedia.org/T397251#10960531 (10XenoRyet) 05Open→03Resolved [19:27:19] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Paypal, 10FR-Smashpig, 07payments-orchestration: Ignore Gravy-related refunds in the PayPal IPN listener - https://phabricator.wikimedia.org/T397670#10960538 (10XenoRyet) 05Open→03Resolved [19:27:49] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Disable Fiscal Number checks on recurring Google Pay & Apple Pay charges - https://phabricator.wikimedia.org/T397322#10960542 (10XenoRyet) 05Open→03Resolved [19:28:10] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MW-1.45-notes (1.45.0-wmf.7; 2025-06-24): Add optional UI for gravy form - https://phabricator.wikimedia.org/T397095#10960545 (10XenoRyet) 05Open→03Resolved [19:28:28] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MW-1.45-notes (1.45.0-wmf.7; 2025-06-24): donate_interface-redirect-explanation shown on dLocal forms but not on gr4vy equivalent - https://phabricator.wikimedia.org/T397199#10960547 (10XenoRyet) 05Open→03Resolved [19:28:38] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: make sure 3rd party returned email type is billing in civi - https://phabricator.wikimedia.org/T396172#10960549 (10XenoRyet) 05Open→03Resolved [19:29:25] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: 3DS failures in CAD June 25-26 - https://phabricator.wikimedia.org/T397943#10960555 (10AKanji-WMF) p:05Triage→03High Discussed in standup June 30: * As per above, notable increase in 3DS authentication failures in Canada and Au... [19:29:29] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Fr-drupal-upgrade-2025, 10MW-1.45-notes (1.45.0-wmf.7; 2025-06-24): Comms Pref Centre - checksum not working - https://phabricator.wikimedia.org/T397048#10960559 (10XenoRyet) 05Open→03Resolved [19:29:45] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-email-preferences, 10MW-1.45-notes (1.45.0-wmf.8; 2025-07-01): Deal with expired checksum in 'fallback' email prefrences messages - https://phabricator.wikimedia.org/T397657#10960567 (10XenoRyet) 05Open→03Resolved [19:30:36] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Mass Card Declines Following Japan Email 1 - https://phabricator.wikimedia.org/T394007#10960577 (10XenoRyet) 05Open→03Resolved [19:31:20] 03Fundraising Sprint: Left-Handed Churro , 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 07fr-current-sprint: Import templates missing / not saving new - https://phabricator.wikimedia.org/T394929#10960585 (10XenoRyet) 05Open→03Resolved [19:31:20] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Update SmashPig Gravy listener to use the new transaction payload - https://phabricator.wikimedia.org/T393958#10960587 (10Damilare) p:05Medium→03Lowest [19:31:22] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Update SmashPig Gravy listener to use the new transaction payload - https://phabricator.wikimedia.org/T393958#10960588 (10Damilare) p:05Lowest→03Medium [19:31:37] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Update SmashPig Gravy listener to use the new transaction payload - https://phabricator.wikimedia.org/T393958#10960589 (10XenoRyet) 05Open→03Resolved [19:32:39] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Update SmashPig Gravy listener to use the new transaction payload - https://phabricator.wikimedia.org/T393958#10960592 (10XenoRyet) [19:33:55] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Donors included wrongly in email send due to "country". - https://phabricator.wikimedia.org/T381437#10960597 (10Ejegg) 05Open→03Resolved a:03Ejegg [19:34:06] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Donors included wrongly in email send due to "country". - https://phabricator.wikimedia.org/T381437#10960599 (10XenoRyet) [19:34:55] (03CR) 10Ejegg: [C:03+2] Update SmashPig [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1165095 (owner: 10Ejegg) [19:39:34] (03PS2) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1165105 [19:42:26] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1165105 (owner: 10Ejegg) [19:46:05] 10fundraising-tech-ops: Q1:rack/setup/install frand200[12] - https://phabricator.wikimedia.org/T367804#10960636 (10Jgreen) a:03Jgreen [19:46:14] 06Fundraising-Backlog, 10FR-BigBundle: Enabling Offline BT for Poland (and Other countries) - https://phabricator.wikimedia.org/T127715#10960639 (10AKanji-WMF) 05Open→03Declined Superseded by payment orchestration work. [19:53:06] 10fundraising-tech-ops: Q1:rack/setup/install franio100[1-3] - https://phabricator.wikimedia.org/T367820#10960673 (10Jgreen) a:03Jgreen [20:03:03] (03Merged) 10jenkins-bot: Update SmashPig [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1165095 (owner: 10Ejegg) [20:03:04] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1165105 (owner: 10Ejegg) [20:12:49] 06Fundraising-Backlog, 07payments-orchestration: Add VIPPS in Norway - https://phabricator.wikimedia.org/T390870#10960788 (10AKanji-WMF) [20:15:14] 06Fundraising-Backlog: RML status in Civi - https://phabricator.wikimedia.org/T397944#10960806 (10AKanji-WMF) [20:32:07] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: donor name transposed from Gravy to Civi: issue? - https://phabricator.wikimedia.org/T397860#10960899 (10AKanji-WMF) p:05Triage→03Medium [20:35:08] !log payments-wiki upgraded from 5066fdea to 52f6940f [20:35:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:36:05] hi Jeff_Green, there's a new feature for process-control merged into the codebase: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/process-control/+/1162944 [20:36:23] Would you be able to package that for production use? [20:36:31] And can I help with that build? [20:36:55] It's been a while but I can refresh my memory on that packaging bit [20:37:34] I haven't tried building that package since redoing our package-building system, hopefully it "just works" -- I can test in vbox [20:37:51] k, let me know [20:39:50] ok fr-tech we should stop getting those bancomer failures at least [20:43:10] (03PS1) 10Ejegg: Update composer.lock [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1165158 [20:43:12] (03CR) 10Ejegg: [C:03+2] Update composer.lock [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1165158 (owner: 10Ejegg) [20:45:58] hmm, those silverpop errors are more than transient at this point [20:46:01] account problem? [20:46:03] maintenance? [20:46:09] let's look at those logs [20:48:11] hmm, connection timed out [20:48:30] Jeff_Green: we might have a firewall problem connecting to acoustic - just checking if anything is getting through [20:51:16] ejegg: the policies are based on ipsets so that should handle IP changes automatically, but they're restricted by local user [20:51:27] so if you're testing, it would have to be jenkins or www-data [20:52:13] ah, i'm just looking at logs [20:52:53] looks like everything has been 'cURL error 28: Connection timed out after 10001 milliseconds' since 18:00 GMT [20:54:10] both the jobs run via process-control and those run via the coworker [20:56:51] hmm, it doesn't respond to a ping from me [20:57:05] did anything change recently on our end? [20:57:09] but traceroute api.pilot.silverpop.com doesn't seem to show it failing [20:58:06] I'm getting "Destination Net Unreachable" from home [20:58:26] (03Merged) 10jenkins-bot: Update composer.lock [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1165158 (owner: 10Ejegg) [20:58:30] maybe a cloud or CDN outage? [20:59:11] could be. I guess I could ask Trilogy [21:01:18] fwiw the error I get is weird, it doesn't look like a simple dead host but rather an unroutable network [21:01:40] makes me think something went wrong in BGP land [21:05:37] so not necessarily something they could fix either Jeff_Green ? [21:06:44] dunno offhand [21:07:15] their status page thinks the API is up [21:07:59] it could be up for some places and not others [21:08:57] I get an error from traceroute too [21:11:05] hmm, their chatbot says we should be connecting to a different url [21:11:15] We understand how frustrating API timeouts can be. Our documentation shows that api.pilot.silverpop.com transitioned to api-campaign-pilot.goacoustic.com, with continued availability. However, there is no information about specific timeout issues in June 2025. Please ensure you are using the updated URL for best results. [21:11:31] but i don't actually see the original URL in the linked article: https://help.goacoustic.com/hc/en-us/articles/360048880034-Standardized-Acoustic-URLs [21:11:59] let me see where that setting is [21:12:42] would it be possible to add that new URL to the list of allowed connections Jeff_Green ? [21:13:18] it is possible but it will take some time to sort it out [21:13:29] how urgent is it? [21:13:34] huh we closed the 'migrate URLs' task back in 2020: https://phabricator.wikimedia.org/T255692 [21:13:43] ha [21:13:52] was it a different migration maybe? [21:13:57] Jeff_Green: well, for now we are not syncing anything from Civi to Acoustic or vice versa [21:14:23] not sure if that will persist till the evening's full data upload or not [21:14:27] ok, I can probably turn it around within in an hour [21:15:25] thanks! I see there are also a couple of pending email snoozes that keep failing too [21:19:25] that URL doesn't seem right does it? "pilot" ? [21:19:35] right, makes it seem like a beta [21:19:41] but it's all over their URL list [21:22:38] that list has "apipilot.silverpop.com" (note missing . after api) replaced by "api-campaign-pilot.goacoustic.com" - that seems like it could be it [21:22:51] right, I see that too [21:23:02] ok, I'll get an ipset set up [21:23:12] odd that it would hang on so long after the stated deadline [21:25:19] yeah [21:33:27] hmm, not finding a setting for the base URL in the Civi UI [21:33:50] I'm not finding an ipset for the old hostname :-P [21:33:59] the world is out of alignment! [21:34:25] oh man [21:37:11] ok I think we must be missing something re. the silverpop hostname, we have three hostnames allowed: [21:37:16] campaign-us-4.goacoustic.com [21:37:21] api-campaign-us-4.goacoustic.com [21:37:25] transfer-campaign-us-4.goacoustic.com [21:37:40] weird [21:37:54] ohhh did something change in the library update??? [21:37:58] looking [21:38:11] is that maybe a test URI? [21:38:35] like we're accidentally running on test config that has a deprecated hostname? [21:39:04] ah hmm, we actually did just update the silverpop-php-connector from 1.6.6 to 1.6.8 [21:39:09] I bet that was it [21:39:58] let me see if eileen had some notes on that [21:40:00] hmm, nope [21:40:19] ok, I think we need to roll those libraries back for now [21:43:34] that hostname shows up in a couple files under vendor/ [21:43:42] yep yep [21:44:19] fr-tech I'm rolling back that last CiviCRM deployment [21:44:36] !log civicrm rolled back from 2a8460ed to e35d3778 [21:44:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:45:00] till we can make sure the right base URL is being passed through for the API calls [21:45:37] but that means we'll have another day of failmails on normal Venmo recurring failures [21:46:38] OK, so it was this vendor update that coincided with the errors: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/vendor/+/1165061 [21:47:25] those two one-line changes in silverpop-php-connector don't look like they could be it [21:54:07] oooh i bet it's the removal of setBaseUrl on line 64 here [21:54:10] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/vendor/+/1165061/1/wikimedia/omnimail-silverpop/src/Connector/SilverpopGuzzleConnector.php#62 [22:00:57] err, could it be? [22:01:59] huh, doesn't seem like it [22:02:10] changes in the calling code then? [22:05:55] ok, I have to step through this [22:15:05] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Add Payment Descriptor to Gravy transactions - https://phabricator.wikimedia.org/T397965#10961170 (10AnnWF) [22:21:23] ooooffff [22:23:34] so yeah, in BaseRequest->construct, we get a new SilverpopXmlConnector with SilverpopGuzzleXmlConnector::getInstance() [22:23:47] and that ends up with the default endpoint [22:24:30] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 10FR-email-preferences: Comms preference centre - scope adding to Acoustic email footers - https://phabricator.wikimedia.org/T392742#10961197 (10ppenloglou) Hey all! Been trying to get this to work but I'm noticing the foll... [22:25:03] heck, how did that change??? [22:26:11] the good news is that API calls are indeed getting through again after the rollback [22:48:46] ejegg: process-control is deployed [22:51:32] nice! thanks Jeff_Green [22:51:42] np [23:03:01] ok, it's something in this commit: b0651b56c0676dd4297a20254aedc6bab817418c [23:03:25] really doesn't look like it could be in the calling code [23:03:43] so it's just that silverpop-php-connector change from 1.6.7 to 1.6.8 [23:04:24] oh or omnimail-silverpop 1.19->1.20 [23:19:48] so bizarre [23:19:53] it's got to be that constructor [23:20:02] I'm just not sure how that's happening [23:20:18] will try changing just that to see if it works [23:49:55] 06Fundraising-Backlog: Sprint M - Priorities 2025 - https://phabricator.wikimedia.org/T398261 (10AKanji-WMF) 03NEW [23:51:01] 06Fundraising-Backlog: Sprint M - Priorities 2025 - https://phabricator.wikimedia.org/T398261#10961407 (10AKanji-WMF) [23:51:27] 06Fundraising-Backlog: Sprint M - Priorities 2025 - https://phabricator.wikimedia.org/T398261#10961411 (10AKanji-WMF) [23:51:56] 06Fundraising-Backlog: Sprint M - Priorities 2025 - https://phabricator.wikimedia.org/T398261#10961412 (10AKanji-WMF) [23:52:35] 06Fundraising-Backlog: Sprint M - Priorities 2025 - https://phabricator.wikimedia.org/T398261#10961413 (10AKanji-WMF) [23:56:31] 03Fundraising Sprint M - 2025, 06Fundraising-Backlog: Sprint M - Priorities 2025 - https://phabricator.wikimedia.org/T398261#10961446 (10AKanji-WMF) [23:58:55] ok, I could do a tiny patch to fix backwards, or a bigger fix to go forwards