[03:00:10] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on payments1012.frack.eqiad.wmnet:9100 - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [03:51:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [04:06:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [07:00:10] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on payments1012.frack.eqiad.wmnet:9100 - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [08:04:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [12.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [08:47:37] FIRING: CiviCRMDupInvoiceID: CiviCRM duplicate invoice ID: 5 messages in the past minute. - https://alerts.wikimedia.org/?q=alertname%3DCiviCRMDupInvoiceID [08:52:37] RESOLVED: CiviCRMDupInvoiceID: CiviCRM duplicate invoice ID: 5 messages in the past minute. - https://alerts.wikimedia.org/?q=alertname%3DCiviCRMDupInvoiceID [08:54:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.7] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [09:36:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [5.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [09:36:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [5.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [09:36:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [5.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [09:41:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [5.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [09:41:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [5.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [09:41:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [5.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [11:00:10] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on payments1012.frack.eqiad.wmnet:9100 - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [11:04:53] (03CR) 10Jgleeson: [C:03+2] "LGTM! thanks for this" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1288000 (owner: 10Umherirrender) [11:06:55] (03Merged) 10jenkins-bot: Avoid null array offset in GatewayAdapter::getMonthlyConvertAmounts [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1288000 (owner: 10Umherirrender) [11:24:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb2003.frack.codfw.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [11:29:45] !log payments-wiki upgraded from 1a056dc0 to 40a24102 [11:29:50] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [11:54:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb2003.frack.codfw.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [12:25:00] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1288637 (owner: 10L10n-bot) [13:05:10] RESOLVED: PuppetConstantChange: Puppet performing a change on every puppet run on payments1012.frack.eqiad.wmnet:9100 - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [13:17:22] (03CR) 10EMcFarland: Move ext.centralNotice.startUp from onBeforePageDisplay to onSiteNoticeAfter (031 comment) [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1285504 (https://phabricator.wikimedia.org/T422082) (owner: 10Krinkle) [13:21:32] (03PS1) 10Ejegg: json_encode array before logging [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1288854 [13:28:28] (03CR) 10Jgleeson: [C:03+2] "Nice fix!" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1288854 (owner: 10Ejegg) [13:30:31] (03Merged) 10jenkins-bot: json_encode array before logging [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1288854 (owner: 10Ejegg) [13:34:18] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: When Civi import shows two contacts exist that match imported data, provide a link to merge them - https://phabricator.wikimedia.org/T426620 (10Lars) 03NEW [13:34:50] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1288861 [13:34:58] ooh, merge-on-import [13:35:15] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1288861 (owner: 10Ejegg) [13:37:43] (03CR) 10CI reject: [V:04-1] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1288861 (owner: 10Ejegg) [13:40:42] huh, I see no test fail there [13:41:34] (03CR) 10Ejegg: [V:03+2 C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1288861 (owner: 10Ejegg) [13:48:09] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add CiviCRM docs on formbuilder search forms - https://phabricator.wikimedia.org/T420980#11931287 (10Damilare) The documentation has now been updated with instructions for Search Forms: https://docs.civicrm.... [13:48:45] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add CiviCRM docs on formbuilder search forms - https://phabricator.wikimedia.org/T420980#11931289 (10Damilare) a:03Damilare [13:56:58] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add CiviCRM docs on formbuilder search forms - https://phabricator.wikimedia.org/T420980#11931342 (10Lars) Thanks Dami, this is great. I'm going to pop this back into Doing here because there are a couple de... [14:25:50] (03PS1) 10Ejegg: List some dynamically-generated messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1288876 (https://phabricator.wikimedia.org/T379197) [14:26:24] !log payments-wiki upgraded from 40a24102 to e31ac2a9 [14:26:24] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [14:27:15] (03CR) 10Damilare Adedoyin: [C:03+2] "Thanks for adding this, LGTM!" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1288876 (https://phabricator.wikimedia.org/T379197) (owner: 10Ejegg) [14:30:14] thanks damilare [14:30:25] np! [14:30:42] i hope you got some sleep on the rest of your trip! [14:37:40] yes I did and we added one more trip to visit family in Chichester too [14:37:51] nice [14:41:04] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 14Fundraising-Backlog-Old, and 4 others: Some messages in the Donation extensions are outdated and should be removed - https://phabricator.wikimedia.org/T111677#11931568 (10Ejegg) I clean... [14:46:03] (03PS1) 10Ejegg: Remove translated currency names [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1288880 (https://phabricator.wikimedia.org/T111677) [14:48:38] Jeff_Green: do we want something like this for our servers? https://github.com/jnuyens/modulejail/tree/master [14:48:51] FIRING: InterfaceOpticLowPower: Too low optic power on - pfw1-codfw:xe-7/2/0 (Core: cr2-codfw:xe-0/0/1:0 {#122503}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#InterfaceOpticLowPower - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceOpticLowPower [14:51:02] (03PS13) 10Krinkle: impressionDiet: Implement new "dailyLimit" parameter [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1273814 (https://phabricator.wikimedia.org/T421662) [14:51:04] (03CR) 10Ejegg: [C:03+2] impressionDiet: Implement new "dailyLimit" parameter [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1273814 (https://phabricator.wikimedia.org/T421662) (owner: 10Krinkle) [14:54:31] (03Merged) 10jenkins-bot: List some dynamically-generated messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1288876 (https://phabricator.wikimedia.org/T379197) (owner: 10Ejegg) [14:57:30] ejegg: interesting! [15:05:31] fr-tech how do we set up webhooks on paypal sandbox? I'm not seeing messages for paypal recurring donations locally and cant seem to log into the sandbox merchant developer account [15:09:55] let me see if I can get in damilare [15:10:05] i was having problems before too [15:10:12] thanks! [15:10:22] (03Merged) 10jenkins-bot: impressionDiet: Implement new "dailyLimit" parameter [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1273814 (https://phabricator.wikimedia.org/T421662) (owner: 10Krinkle) [15:11:47] huh, right, the links from the sandbox merchant console seem to go to the production developer site [15:12:09] yea exactly [15:13:33] I was also having recurring issues locally [15:14:05] were you able to fix it cstone ? [15:14:28] I didn't dig into it more [15:56:02] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Recurring-Donations: Some PayPal recurrings recorded with bad conversion rate - https://phabricator.wikimedia.org/T426098#11932071 (10Cstone) a:03Cstone [15:57:11] FIRING: NodeDown: Node civi2002 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=civi2002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [16:01:57] FIRING: HostMemoryUtilizationWarn: Host high memory utilization (instance fransw1001.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DHostMemoryUtilizationWarn [16:02:11] FIRING: NodeDown: Node civi2002 has been down for long. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=civi2002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [16:02:36] FIRING: StrongswanFailure: Strongswan is not fully established on frlog2002.frack.codfw.wmnet:9100 for peerhost civi2002 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DStrongswanFailure [16:05:45] (03PS1) 10Cstone: WIP: Don't normalize paypal recurrings twice [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1288910 (https://phabricator.wikimedia.org/T426098) [16:06:41] ^^^ those were me doing a reboot and failing to get a proper silence in for alertmanager. [16:11:57] RESOLVED: HostMemoryUtilizationWarn: Host high memory utilization (instance fransw1001.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DHostMemoryUtilizationWarn [16:25:31] (03CR) 10Raimond Spekking: [C:03+2] "false positive" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1288637 (owner: 10L10n-bot) [16:32:17] (03CR) 10CI reject: [V:04-1] WIP: Don't normalize paypal recurrings twice [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1288910 (https://phabricator.wikimedia.org/T426098) (owner: 10Cstone) [16:36:13] ohhhh cstone was that the currency issue? [16:36:26] currency rate squared? [16:36:28] Yeah [16:36:42] hah, wow [16:39:48] ohhh it's because they're being pushed to the donations queue now [16:42:30] hmm, could we instead fix the normalization to actually be idempotent? [16:46:59] (03CR) 10Ejegg: "One edge case is inimportSubscriptionPayment on line 141 when it sends the message to importSubscriptionSignup and would now not be normal" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1288910 (https://phabricator.wikimedia.org/T426098) (owner: 10Cstone) [16:48:05] I feel like we should just be sending those PayPal recurrings straight to donations [16:48:56] But yeah it also is converting USD to USD happily so I have that as another thing to look into [17:08:11] FIRING: NodeDown: Node frav1003 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frav1003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [17:08:47] ^^known, reboot. need to figure out how to keep the 'warning' messages from showing up in irc since they aren't really an issue. [17:14:59] (03PS3) 10Ejegg: Stop outputting invalid legend messages [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1237305 (https://phabricator.wikimedia.org/T59109) [17:16:19] (03CR) 10Ejegg: "OK Timo, I've updated it to continue outputting the legends where the messages DOES exist." [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1237305 (https://phabricator.wikimedia.org/T59109) (owner: 10Ejegg) [17:21:05] (03PS1) 10Ejegg: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1288926 [17:21:11] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1288926 (owner: 10Ejegg) [17:24:11] RESOLVED: NodeDown: Node frav1003 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frav1003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [17:31:21] (03PS2) 10Ejegg: Add test link for Yape (and mention CO bt submethods) [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1277126 (https://phabricator.wikimedia.org/T423232) [17:31:23] (03CR) 10Ejegg: [C:03+2] Add test link for Yape (and mention CO bt submethods) [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1277126 (https://phabricator.wikimedia.org/T423232) (owner: 10Ejegg) [17:32:03] (03CR) 10Ejegg: [V:03+2 C:03+2] Add test link for Yape (and mention CO bt submethods) [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1277126 (https://phabricator.wikimedia.org/T423232) (owner: 10Ejegg) [17:32:49] (03PS1) 10Ejegg: Update civiproxy to trixie / PHP8.4 [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1288927 [17:34:11] (03PS1) 10Ejegg: Update SmashPig container to trixie [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1288928 (https://phabricator.wikimedia.org/T424774) [17:35:46] (03Merged) 10jenkins-bot: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1288926 (owner: 10Ejegg) [17:35:58] 06Fundraising-Backlog, 07fr-current-sprint: Update the recurring failure/cancellation logic - https://phabricator.wikimedia.org/T413905#11932744 (10CHudson-WMF) Thanks for the background on that. [17:36:23] (03CR) 10Ejegg: [C:03+2] Restore description_of_stock field values incorrectly deleted [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287871 (https://phabricator.wikimedia.org/T426417) (owner: 10Lars SG) [17:55:18] 10Fundraising analytics stack: Upgrade Metabase to on-premise Enterprise - https://phabricator.wikimedia.org/T403723#11932796 (10IAckerman-WMF) Deprecating for now [18:00:55] (03Merged) 10jenkins-bot: Restore description_of_stock field values incorrectly deleted [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1287871 (https://phabricator.wikimedia.org/T426417) (owner: 10Lars SG) [18:33:46] (03CR) 10Wfan: Overwrite name when both existing and incoming email location types are low confidence (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281812 (https://phabricator.wikimedia.org/T425196) (owner: 10Lars SG) [18:34:25] 10fundraising-tech-ops: Fundraising systems Debian Trixie support - https://phabricator.wikimedia.org/T401847#11932887 (10Jgreen) [18:37:29] (03CR) 10Wfan: [C:03+2] Don't merge identical email addresses with different location types [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282929 (https://phabricator.wikimedia.org/T425195) (owner: 10Lars SG) [18:37:43] (03CR) 10Wfan: [C:03+2] Setting to exclude location types from Email::Clean [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282926 (https://phabricator.wikimedia.org/T425195) (owner: 10Lars SG) [18:56:55] 10fundraising-tech-ops: Fundraising systems Debian Trixie support - https://phabricator.wikimedia.org/T401847#11932965 (10Jgreen) [18:57:33] 10fundraising-tech-ops: Fundraising systems Debian Trixie support - https://phabricator.wikimedia.org/T401847#11932966 (10Jgreen) [19:04:53] (03Merged) 10jenkins-bot: Setting to exclude location types from Email::Clean [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282926 (https://phabricator.wikimedia.org/T425195) (owner: 10Lars SG) [19:04:56] (03Merged) 10jenkins-bot: Don't merge identical email addresses with different location types [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282929 (https://phabricator.wikimedia.org/T425195) (owner: 10Lars SG) [19:13:48] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Make minfraud risk score to Fraud Scan SK - https://phabricator.wikimedia.org/T426255#11933052 (10XenoRyet) 05Open→03Resolved [19:14:10] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add link in Zendesk to contact in CiviCRM & unsubscribe form by email - https://phabricator.wikimedia.org/T415378#11933071 (10XenoRyet)... [19:14:30] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog: Make SFTP transfer atomic - https://phabricator.wikimedia.org/T423600#11933080 (10XenoRyet) 05Open→03Resolved [19:14:50] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add custom fields to Direct Mail activities for segment and ask - https://phabricator.wikimedia.org/T422728#11933085 (10XenoRyet) 05Open→03Resolved [19:15:13] 03Fundraising Sprint Hutch Ado About Nothing, 13Patch-For-Review: Switch from OpenSSL to age - https://phabricator.wikimedia.org/T423897#11933087 (10XenoRyet) 05Open→03Resolved [19:15:37] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 07payments-orchestration: Blik Payment Method- Poland - https://phabricator.wikimedia.org/T405606#11933092 (10XenoRyet) 05Open→03Resolved [19:15:53] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07FR-Fraud: Allow DR to refund transactions from Civi and flag as fraud - https://phabricator.wikimedia.org/T421277#11933095 (10XenoRyet) 05Open→... [19:16:43] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog: FR-Tech AI Safe Usage Guide - https://phabricator.wikimedia.org/T421487#11933101 (10XenoRyet) 05Open→03Resolved [19:16:59] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Feature Request: Ability to Edit / Change recurring cancellation reason in Civi dropdown... - https://phabricator.wikimedia.org/T389197#11933105 [19:17:23] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Recurring-Donations: Avoid duplicate db queries in Message::getContributionRecurID - https://phabricator.wikimedia.org/T423736#11933108 (10XenoRyet) 05Open→03Resolved [19:17:36] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add email-once-per-year and dm-once-per-year tags to Civi & Acoustic - https://phabricator.wikimedia.org/T423454#11933111 (10XenoRyet) 05Open→03Res... [19:17:50] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10fundraising-tech-ops, 10Wikimedia-Fundraising-CiviCRM: Add both_funds_usd_total_fy2627 to Acoustic export - https://phabricator.wikimedia.org/T415684#11933114 (10XenoRyet) 05Open→03Resolved [19:18:21] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Adyen discrepancy - batch 1154 (caused by Gr4vy bugs) - https://phabricator.wikimedia.org/T420527#11933117 (10XenoRyet) 05Open→03... [19:18:38] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10dev-images, 13Patch-For-Review: Create trixie/php8.4 fundraising dev images - https://phabricator.wikimedia.org/T424774#11933120 (10XenoRyet) 05Open→03Resolved [19:19:08] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: New unhandled dlocal payment method (audit) - https://phabricator.wikimedia.org/T423983#11933122 (10XenoRyet) 05Open→03Resolved [19:19:19] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Civi: add message to Forget Me button re: recurring donations - https://phabricator.wikimedia.org/T392320#11933123 (10XenoRyet)... [19:19:21] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Fix batch refund under coworker - https://phabricator.wikimedia.org/T426679 (10Ejegg) 03NEW [19:19:55] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 05MW-1.47-notes (1.47.0-wmf.2; 2026-05-12): gravy recurring like to have recurring for descriptor - https://phabricator.wikimedia.org/T423718#11933140 (10XenoRyet) 05Open→03Resolved [19:20:05] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Delete disabled fields from wmf_contribution_extra - https://phabricator.wikimedia.org/T420301#11933142 (10XenoRyet) 05Open→03Resolved [19:20:44] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Paypal, 10Recurring-Donations, and 2 others: Check on handling of payment-method.deleted IPNs from Gravy - https://phabricator.wikimedia.org/T423638#11933145 (10XenoRyet) 05Open→03Resolved [19:20:54] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-MGFE: Updating the Contribution History on the Contact Summary to reflect the current FY totals - https://phabricator.wikimedia.org/T424937#11933147 (10XenoRyet) 05Open→03Resolved [19:20:58] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog: FR-Tech AI Safe Usage Guide - https://phabricator.wikimedia.org/T421487#11933149 (10jgleeson) We also built this AI safe-usage utility as part of this task https://gitlab.wikimedia.org/... [19:21:46] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Handle new Gravy Monitoring Incident IPNs - https://phabricator.wikimedia.org/T426680 (10jgleeson) 03NEW [19:22:00] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Gravy Recurring: Validation error during recurring charge, in field: processor_contact_id. Message: Input should be a valid UUID - https://phabricator.wikimedia.org/T424213#11933166 (10XenoRyet) 05Open→03Resolved [19:22:04] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Handle new Gravy Monitoring Incident IPNs - https://phabricator.wikimedia.org/T426680#11933179 (10jgleeson) Patch here https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/SmashPig/+/1287437 [19:22:05] 06Fundraising-Backlog, 10FR-Smashpig: Implement Blik via Adyen direct - https://phabricator.wikimedia.org/T426681 (10Ejegg) 03NEW [19:22:20] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Handle new Gravy Monitoring Incident IPNs - https://phabricator.wikimedia.org/T426680#11933183 (10jgleeson) p:05Triage→03Medium [19:22:42] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: payments/index.php not found or unable to stat - https://phabricator.wikimedia.org/T423539#11933185 (10XenoRyet) 05Open→03Resolved [19:23:21] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Manually captured gravy transaction pending messages already set to resolved before the capture IPN comes in - https://phabricator.wikimedia.org/T421195#11933190 (10XenoRyet) 05Open→03Resolved [19:23:47] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Pending transactions log that they are resolved when they aren't actually resolved - https://phabricator.wikimedia.org/T421253#11933196 (10XenoRyet) 05Open→03Resolved [19:24:20] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Pix Recurring bankstatement show dlocal instead of wikimedia Foundation - https://phabricator.wikimedia.org/T423715#11933198 (10XenoRyet) 05Open→03Declined [19:24:50] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Adyen incorrect backend_processor_trxn_id - another variant? - https://phabricator.wikimedia.org/T423982#11933200 (10XenoRyet) 05Open→03Resolved [19:26:16] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: recurring dlocal gravy charge failed due to cvv issue - https://phabricator.wikimedia.org/T422773#11933206 (10XenoRyet) 05Open→03Resolved [19:27:02] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Handle new Gravy Monitoring Incident IPNs - https://phabricator.wikimedia.org/T426680#11933208 (10XenoRyet) 05Open→03Resolved [19:27:25] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: omnimail_omnigroupmembers_load throwing exception when file not found - https://phabricator.wikimedia.org/T426018#11933210 (10XenoRyet) 05Open→03Resolved [19:28:27] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Paypal settlement anomaly to handle - https://phabricator.wikimedia.org/T417347#11933213 (10XenoRyet) 05Open→03Resolved [19:42:05] 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2526Q4 maintenance window - https://phabricator.wikimedia.org/T364186#11933281 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt Maintenance week complete. Accomplished all tasks except for the moving to new payments hosts. Other tasks will close... [19:47:14] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add link in Zendesk to contact in CiviCRM & unsubscribe form by email - https://phabricator.wikimedia.org/T415378#11933301 (10KHansen) >... [19:48:38] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Donors who give monthly via cheque should be recurring in Civi calculated fields & Acoustic - https://phabricator.wikimedia.org/T426685 (10Lars) 03NEW [20:20:49] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Donors who give monthly via cheque should be recurring in Civi calculated fields & Acoustic - https://phabricator.wikimedia.org/T426685#11933384 (10Lars) [20:24:02] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Remove un-needed separate endowment/foundation calculated fields from Civi & Acoustic - https://phabricator.wikimedia.org/T418885#11933391 (10Lars) Note for the next step on this: Send a list of fields to be deleted to Cassie / Analytics a week before del... [20:31:38] 06Fundraising-Backlog, 06FR-donorrelations: Duplicate email types in Civi? - https://phabricator.wikimedia.org/T425705#11933424 (10Lars) →14Duplicate dup:03T425190 [20:31:41] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: Clean up emails where contact has two of the same type and address after backfilling location types - https://phabricator.wikimedia.org/T425190#11933421 (10Lars) [20:31:41] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1288981 [20:45:02] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1288981 (owner: 10Eileen) [20:45:45] !log civicrm upgraded from 150d484e to bfa13f7f [20:45:46] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [21:06:48] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: When Civi import shows two contacts exist that match imported data, provide a link to merge them - https://phabricator.wikimedia.org/T426620#11933484 (10Lars) [[ https://github.com/civicrm/civicrm-core/pull/35686 | Upstream PR ]] [21:31:46] 10fundraising-tech-ops: Fundraising access request for Liv Burke - https://phabricator.wikimedia.org/T425990#11933558 (10AStein-WMF) waiting for liv to install the ssl cert [21:31:53] 10fundraising-tech-ops: Fundraising access request for Liv Burke - https://phabricator.wikimedia.org/T425990#11933559 (10AStein-WMF) 05In progress→03Stalled [22:14:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frav1003.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [22:24:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frav1003.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [22:27:11] FIRING: NodeDown: Node frpm2002 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frpm2002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [22:32:10] FIRING: WidespreadPuppetFailure: Puppet has failed on over 5% of hosts - TODO - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [22:37:58] ^ expected from the frpm2002 reboot [22:38:02] they will clear. [23:23:46] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: gzip not (no longer?) working in audit files - https://phabricator.wikimedia.org/T426691 (10Eileenmcnaughton) 03NEW [23:31:04] (03PS1) 10Eileen: Fix move file to use full path [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1289002 (https://phabricator.wikimedia.org/T426691) [23:46:56] (03PS1) 10Eileen: Fix generate finance batch to email us when nothing found [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1289004 (https://phabricator.wikimedia.org/T426691)