[01:59:24] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [34.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [01:59:24] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [34.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [01:59:24] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [34.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [02:04:24] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [24.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [02:04:24] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [24.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [02:04:24] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [24.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [05:53:01] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb1007.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [05:58:01] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb1007.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [07:36:51] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1356717 (owner: 10L10n-bot) [07:52:33] (03CR) 10Raimond Spekking: [C:03+2] "false positive" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1356717 (owner: 10L10n-bot) [09:17:50] (03CR) 10Laurabarluzzi: [C:03+1] "I did a first pass, and overall it looks good although I didn't focus my review on how this function will be used since I see there is the" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1355069 (https://phabricator.wikimedia.org/T439286) (owner: 10Jgleeson) [09:54:41] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 07fr-combowiki, 13Patch-For-Review: Combowiki: Handle amount selection from the url (amount and preSelect) - https://phabricator.wikimedia.org/T440257#12447715 (10Laurabarluzzi) @Cstone do you want me to take this over? If you're too busy I'm happy... [10:12:40] (03CR) 10Laurabarluzzi: [C:03+1] "I did a first pass, and overall it looks good. I didn't approve since this is chained to the other one I had a question for." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1355867 (https://phabricator.wikimedia.org/T439286) (owner: 10Jgleeson) [10:44:48] thanks for the review larssandergreen [10:44:53] oops laurabar * [12:48:28] 10Fundraising Payments token migration 2026, 06Fundraising-Backlog: Check on adyen ideal Contract not found recurring failures - https://phabricator.wikimedia.org/T440419#12448531 (10Damilare) Hi @MBeat33, thanks for reporting both issues. Yes, that was a stray error from the revert. I had assumed all the recu... [13:31:12] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10fundraising-tech-ops, 03Civi RoadMap FY 26-27: FR-Tech FY2627Q2 maintenance window - https://phabricator.wikimedia.org/T419556#12448768 (10Jgreen) [13:50:27] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10fundraising-tech-ops, 03Civi RoadMap FY 26-27: FR-Tech FY2627Q2 maintenance window - https://phabricator.wikimedia.org/T419556#12449000 (10Jgreen) [13:53:06] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10fundraising-tech-ops, 03Civi RoadMap FY 26-27: FR-Tech FY2627Q2 maintenance window - https://phabricator.wikimedia.org/T419556#12449040 (10Jgreen) [13:56:06] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10fundraising-tech-ops, 03Civi RoadMap FY 26-27: FR-Tech FY2627Q2 maintenance window - https://phabricator.wikimedia.org/T419556#12449051 (10Jgreen) [14:01:22] morning fr-tech [14:15:17] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint, 07fr-payments-ACH: We retry failed recurring SEPA donations forever if the reason is retryable - https://phabricator.wikimedia.org/T420775#12449200 (10Ejegg) kicking this out of sprint. We brought it in a little while back when it... [14:25:40] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: Adding check number field to DAFpay payment method - https://phabricator.wikimedia.org/T429835#12449263 (10MDemosWMF) @Eileenmcnaughton Any updates on the merge? [14:59:43] (03PS4) 10Lars SG: Change Gift_Data.Donor_Specified from CheckBox to Radio [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1355199 (https://phabricator.wikimedia.org/T386031) [15:23:26] 10Fundraising Payments token migration 2026, 06Fundraising-Backlog: Check on adyen ideal Contract not found recurring failures - https://phabricator.wikimedia.org/T440419#12449669 (10Damilare) Hi @MBeat33, I've been able to identify 59 other cases that had created a new recurring donation before we could recov... [15:37:32] FIRING: HAProxyBackendDown: HAProxy backend minio_backend on fransw2001.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendDown [15:38:09] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Validation error during recurring charge, in field: first_name - https://phabricator.wikimedia.org/T441706 (10AnnWF) 03NEW [15:39:27] ^^^ that's expected from the mass reboots for kernel updates in the off colo. [15:41:13] (03PS1) 10Wfan: Avoid send invalid name for gravy recurring [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1356920 (https://phabricator.wikimedia.org/T441706) [15:41:43] (03CR) 10CI reject: [V:04-1] Avoid send invalid name for gravy recurring [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1356920 (https://phabricator.wikimedia.org/T441706) (owner: 10Wfan) [15:41:46] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Gravy Google Pay: no name validation at signup lets "." through, recurring charges then fail on buyer/billing_details/first_name - https://phabricator.wikimedia.org/T440452#12449880 (10Ejegg) Proposed general fix for wallet signups with bad info: {T441162... [15:43:58] (03PS2) 10Wfan: Avoid send invalid name for gravy recurring [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1356920 (https://phabricator.wikimedia.org/T441706) [15:45:01] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10fundraising-tech-ops, 03Civi RoadMap FY 26-27: FR-Tech FY2627Q2 maintenance window - https://phabricator.wikimedia.org/T419556#12449946 (10Dwisehaupt) [15:49:57] (03PS2) 10Lars SG: Fix core bug that doesn't fill in default contact when a contact entity ref is multi-value [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1355984 (https://phabricator.wikimedia.org/T386031) [15:52:14] 06Fundraising-Backlog, 07fr-combowiki: Combowiki: Build? the paypal braintree component - https://phabricator.wikimedia.org/T441711 (10Cstone) 03NEW [16:05:11] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10fundraising-tech-ops, 03Civi RoadMap FY 26-27: FR-Tech FY2627Q2 maintenance window - https://phabricator.wikimedia.org/T419556#12450125 (10Dwisehaupt) [16:11:22] FIRING: GnmiInterfaceCountersDrop: ... [16:11:23] fasw2-e16a-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=fasw2-e16a-eqiad:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [16:35:08] (03PS1) 10Wfan: only EUR for sepa [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1356943 [16:36:36] (03CR) 10CI reject: [V:04-1] only EUR for sepa [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1356943 (owner: 10Wfan) [16:39:55] (03PS2) 10Wfan: only EUR for sepa [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1356943 [16:40:24] FIRING: SystemdService: Systemd service coworker.service stopped on civi1002.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DSystemdService [17:04:58] (03CR) 10Dmiranda: [C:03+2] "Thanks for adding a link to the doc! made it helpful to review, everything looks good +2" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1356943 (owner: 10Wfan) [17:05:24] FIRING: SystemdService: Systemd service coworker.service stopped on civi1002.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DSystemdService [17:06:47] (03Merged) 10jenkins-bot: only EUR for sepa [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1356943 (owner: 10Wfan) [17:19:54] RESOLVED: SystemdService: Systemd service coworker.service stopped on civi1002.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DSystemdService [17:40:07] 10Fundraising Payments token migration 2026, 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 06App Growth, and 4 others: Create API for Apps fundraising donation processing through Gravy - https://phabricator.wikimedia.org/T438879#12450730 (10ABendall-WMF) Please note App Growth team - We have be... [17:43:10] 10Fundraising Payments token migration 2026, 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 06App Growth, and 3 others: Create API for Apps fundraising donation processing through Gravy - https://phabricator.wikimedia.org/T438879#12450740 (10ABendall-WMF) [18:16:07] (03PS1) 10Ejegg: Adyen: Don't send encrypted_ params to ML service [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1356969 [18:17:37] (03CR) 10CI reject: [V:04-1] Adyen: Don't send encrypted_ params to ML service [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1356969 (owner: 10Ejegg) [18:18:57] (03PS2) 10Ejegg: Adyen: Don't send encrypted_ params to ML service [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1356969 [18:20:38] (03CR) 10CI reject: [V:04-1] Adyen: Don't send encrypted_ params to ML service [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1356969 (owner: 10Ejegg) [18:20:45] (03CR) 10Dmiranda: [C:03+2] "Thanks @wfan@wikimedia.org! LGTM" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1356920 (https://phabricator.wikimedia.org/T441706) (owner: 10Wfan) [18:21:12] (03Merged) 10jenkins-bot: Avoid send invalid name for gravy recurring [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1356920 (https://phabricator.wikimedia.org/T441706) (owner: 10Wfan) [18:28:27] (03PS3) 10Ejegg: Adyen: Don't send encrypted_ params to ML service [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1356969 [18:55:35] (03PS1) 10Jgleeson: Add tests for percentage gateway routing [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1356974 (https://phabricator.wikimedia.org/T441723) [18:57:03] (03CR) 10CI reject: [V:04-1] Add tests for percentage gateway routing [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1356974 (https://phabricator.wikimedia.org/T441723) (owner: 10Jgleeson) [18:59:11] (03PS2) 10Jgleeson: Add tests for percentage gateway routing [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1356974 (https://phabricator.wikimedia.org/T441723) [19:00:07] (03PS1) 10Ejegg: Blank out a list of fields in log-outbound [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1356975 (https://phabricator.wikimedia.org/T165198) [19:00:34] damilare: there's an attempt at it ^^^ [19:00:35] (03CR) 10CI reject: [V:04-1] Blank out a list of fields in log-outbound [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1356975 (https://phabricator.wikimedia.org/T165198) (owner: 10Ejegg) [19:02:30] (03PS2) 10Ejegg: Blank out a list of fields in log-outbound [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1356975 (https://phabricator.wikimedia.org/T165198) [19:11:26] FIRING: [2x] NodeDown: Node frqueue1006 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [19:13:32] FIRING: HAProxyExporterDown: HAProxy on pay-lb1002.frack.eqiad.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [19:13:59] now sure why these two are firing. there are silences in place via pas [19:14:54] ah, i had a typo so it only partially applied. fixed up. [19:25:20] 10Fundraising Payments token migration 2026, 06Fundraising-Backlog: Check on adyen ideal Contract not found recurring failures - https://phabricator.wikimedia.org/T440419#12451066 (10MBeat33) Thank you @Damilare for the followup with the donors from SK 9949. I will contact the batch of 59 to confirm what the... [19:29:14] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint, 13Patch-For-Review: Show iframe with donor details in Zendesk - https://phabricator.wikimedia.org/T433866#12451094 (10Lars) 05Open→03Resolved [19:29:16] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 07fr-combowiki: combowiki: show the privacy policy - https://phabricator.wikimedia.org/T434081#12451098 (10Cstone) [19:29:28] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Test Kitchen: Record a hide reason to beacon/impression when banner hidden by TestKitchen - https://phabricator.wikimedia.org/T438077#12451101 (10Ejegg) [19:29:41] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Test Kitchen: Record a hide reason to beacon/impression when banner hidden by TestKitchen - https://phabricator.wikimedia.org/T438077#12451102 (10Ejegg) 05In progress→03Resolved [19:30:04] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 07fr-combowiki, 05MW-1.47-notes (1.47.0-wmf.24; 2026-10-13): Combowiki: display endowment with utm_medium=endowment - https://phabricator.wikimedia.org/T434233#12451105 (10dmiranda) 05Open→03Resolved [19:30:16] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 07fr-combowiki: combowiki: show the privacy policy - https://phabricator.wikimedia.org/T434081#12451112 (10Cstone) 05Open→03Resolved a:03Cstone [19:30:23] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 07fr-combowiki, 07fr-current-sprint: Combowiki: When you click around on the page there's a JS error in the console - https://phabricator.wikimedia.org/T440261#12451118 (10Laurabarluzzi) 05Open→03Resolved p:05Triage→03Low [19:30:25] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint, 13Patch-For-Review: Show iframe with donor details in Zendesk - https://phabricator.wikimedia.org/T433866#12451117 (10Lars) [19:30:30] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Still allow previewing banners despite CentralNoticeProhibitedExperiments - https://phabricator.wikimedia.org/T438057#12451120 (10Ejegg) [19:31:00] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Donation receipt defaults to include email address that received receipt - https://phabricator.wikimedia.org/T440189#12451123 (10Lars) 05Open→03Resolved [19:31:44] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 07fr-combowiki, 05MW-1.47-notes (1.47.0-wmf.24; 2026-10-13): Combowiki: Add SEPA - https://phabricator.wikimedia.org/T439597#12451132 (10Laurabarluzzi) 05Open→03Resolved [19:31:48] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Prefer individual over org when 2+ contacts share and email on Acoustic export - https://phabricator.wikimedia.org/T437967#12451136 (10Lars) 05Open→03Resolved [19:31:49] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 07fr-combowiki, 13Patch-For-Review: Combowiki: Implement 3DS redirect, it's currently going to payments - https://phabricator.wikimedia.org/T439623#12451129 (10jgleeson) 05Open→03Resolved [19:31:53] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Still allow previewing banners despite CentralNoticeProhibitedExperiments - https://phabricator.wikimedia.org/T438057#12451138 (10Ejegg) 05Open→03Resolved [19:32:09] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add AF_recurring_latest_cancel_date to Acoustic export - https://phabricator.wikimedia.org/T441023#12451145 (10Lars) 05Open→03Resolved [19:32:16] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 07fr-combowiki, 05MW-1.47-notes (1.47.0-wmf.24; 2026-10-13): Combowiki: Better UX for the country drop down - https://phabricator.wikimedia.org/T440262#12451148 (10AnnWF) 05Open→03Resolved [19:32:34] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 07fr-combowiki: Combowiki: Handle paypal cancel flow error message - https://phabricator.wikimedia.org/T440411#12451154 (10AnnWF) 05Open→03Resolved [19:32:53] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Improve new SK-based message template admin view - https://phabricator.wikimedia.org/T440398#12451159 (10Lars) 05Open→03Resolved [19:33:06] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Gravy paypal sandbox api auth missing - https://phabricator.wikimedia.org/T441198#12451162 (10AnnWF) 05Open→03Resolved [19:33:22] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: New calendar year sum amount field - https://phabricator.wikimedia.org/T438035#12451164 (10Lars) 05Open→03Resolved [19:33:30] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: paypal job Job message needs 'class' and 'payload' - https://phabricator.wikimedia.org/T440679#12451166 (10AnnWF) 05Open→03Resolved [19:33:47] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Logdog should include syslog on civi1002 - https://phabricator.wikimedia.org/T434029#12451169 (10AnnWF) 05Open→03Resolved [19:33:56] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Smashpig: Failmail should truncate the error context to at most 30 lines before the error - https://phabricator.wikimedia.org/T438355#12451171 (10AnnWF) 05Open→03Resolved [19:34:00] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 07fr-combowiki: Combowiki: Add in country + payment methods specific fields - https://phabricator.wikimedia.org/T439938#12451173 (10Laurabarluzzi) 05Open→03Resolved [19:34:08] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-Banners, 10MediaWiki-extensions-CentralNotice, 05MW-1.47-notes (1.47.0-wmf.24; 2026-10-13): Did TK / banner suppression code worsen our 'banner bump'? - https://phabricator.wikimedia.org/T438401#12451178 (10Ejegg) a:03Ejegg [19:34:08] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 07fr-combowiki, 05MW-1.47-notes (1.47.0-wmf.24; 2026-10-13): reset country to use geolocate after remove from the country param from url - https://phabricator.wikimedia.org/T440412#12451180 (10AnnWF) 05Open→03Resolved [19:34:32] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-Banners, 10MediaWiki-extensions-CentralNotice, 05MW-1.47-notes (1.47.0-wmf.24; 2026-10-13): Did TK / banner suppression code worsen our 'banner bump'? - https://phabricator.wikimedia.org/T438401#12451182 (10Ejegg) 05Open... [19:36:16] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations, and 2 others: Epic: Civi much slower than usual! - https://phabricator.wikimedia.org/T402837#12451184 (10Lars) a:05Lars→03None [19:36:44] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Monthly convert thank you email doesn't include frequency - https://phabricator.wikimedia.org/T439890#12451188 (10Lars) 05Open→03Resolved [19:37:01] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations, and 2 others: Epic: Civi much slower than usual! - https://phabricator.wikimedia.org/T402837#12451194 (10Lars) 05Open→03Resolved [19:37:32] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: PhoneConsent.RemoteUpdate failed to send Mail - https://phabricator.wikimedia.org/T440674#12451195 (10Lars) 05Open→03Resolved [19:38:02] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: checkout audit Money addition issue total_amount 3907.34should equal -106.64 plus 4013.92 - https://phabricator.wikimedia.org/T440245#12451197 (10AnnWF) a:03dmiranda [19:39:17] 10Fundraising Payments token migration 2026, 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 07payments-orchestration: Add new ACH Connector to Gravy - https://phabricator.wikimedia.org/T438723#12451209 (10Damilare) 05Open→03Resolved [19:39:39] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 07fr-combowiki, 05MW-1.47-notes (1.47.0-wmf.24; 2026-10-13): Render second-stage form in Combowiki for banner donations - https://phabricator.wikimedia.org/T439621#12451211 (10Damilare) 05Open→03Res... [19:39:49] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 07fr-combowiki, 05MW-1.47-notes (1.47.0-wmf.24; 2026-10-13): Combowiki GPay payment method bug - https://phabricator.wikimedia.org/T440226#12451217 (10Damilare) 05Open→03Resolved [19:40:54] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Deploy opps fall out - https://phabricator.wikimedia.org/T440559#12451220 (10Lars) a:03Eileenmcnaughton [19:42:22] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10Wikimedia-Fundraising-Banners, 05MW-1.47-notes (1.47.0-wmf.21; 2026-09-22): Can we remove 'duration' parameter from Special:HideBanners requests - https://phabricator.wikimedia.org/T438199#12451228 (10Ejegg) [19:42:36] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10Wikimedia-Fundraising-Banners, 05MW-1.47-notes (1.47.0-wmf.21; 2026-09-22): Can we remove 'duration' parameter from Special:HideBanners requests - https://phabricator.wikimedia.org/T438199#12451230 (10Ejegg) 05Open→03Resolved [19:43:12] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: checkout audit Money addition issue total_amount 3907.34should equal -106.64 plus 4013.92 - https://phabricator.wikimedia.org/T440245#12451239 (10dmiranda) 05Open→03Resolved [19:43:27] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Deploy opps fall out - https://phabricator.wikimedia.org/T440559#12451242 (10Lars) 05Open→03Resolved [19:43:33] 03Fundraising Sprint: Shelf Aware, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Adyen audit ignores PaymentCost rows - https://phabricator.wikimedia.org/T440672#12451243 (10dmiranda) 05Open→03Resolved [19:44:10] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 07payments-orchestration: SEPA payments migrated to Gr4vy failing en masse - https://phabricator.wikimedia.org/T438431#12451250 (10Damilare) 05Open→03Resolved [19:45:26] FIRING: NodeDown: Node civi1002 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=civi1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [19:47:08] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10fundraising-tech-ops, 03Civi RoadMap FY 26-27, 13Patch-For-Review: FR-Tech FY2627Q2 maintenance window - https://phabricator.wikimedia.org/T419556#12451272 (10Dwisehaupt) [19:50:26] RESOLVED: [2x] NodeDown: Node civi1002 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:11:23] FIRING: GnmiInterfaceCountersDrop: ... [20:11:23] fasw2-e16a-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=fasw2-e16a-eqiad:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [20:26:35] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Fix 'unsupported field' warnings in queue consumers - https://phabricator.wikimedia.org/T441334#12451423 (10AnnWF) [20:37:56] FIRING: NodeDown: Node fransw1003 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fransw1003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:38:07] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 07fr-combowiki, 13Patch-For-Review: Combowiki: variant? where paypal, venmo, apple pay, and google pay buttons are loaded to start - https://phabricator.wikimedia.org/T437335#12451466 (10Cstone) a:05Cstone→03Laurabarluzzi [20:39:32] FIRING: HAProxyExporterDown: HAProxy on fransw1003.frack.eqiad.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [20:39:54] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-MGFE, 07fr-current-sprint: Add custom field for event location type (online, hybrid, in person) - https://phabricator.wikimedia.org/T441359#12451478 (10JSimmons) Hi @Lars - thanks for flagging this. I think fo... [20:40:54] 03Fundraising Sprint: Shelf Aware: Sprint S Priorities - https://phabricator.wikimedia.org/T440374#12451483 (10Damilare) Chaos crew completed tasks - {T441093} - DonationInterface: Reflected XSS on Special:EmailPreferences - {T438431} - SEPA payments migrated to Gr4vy failing en masse - {T440672} - Adyen audit i... [20:42:27] 03Fundraising Sprint: Shelf Aware: Sprint S Priorities - https://phabricator.wikimedia.org/T440374#12451524 (10Damilare) Feature team completed tasks - {T440672} - Adyen audit ignores PaymentCost rows - {T438199} - Can we remove 'duration' parameter from Special:HideBanners requests - {T441097} - SmashPig: Adyen... [20:46:47] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-MGFE, 07fr-current-sprint: Add custom field for event location type (online, hybrid, in person) - https://phabricator.wikimedia.org/T441359#12451571 (10Lars) @JSimmons Does it make sense to add a custom field... [20:58:38] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-MGFE, 07fr-current-sprint: Add custom field for event location type (online, hybrid, in person) - https://phabricator.wikimedia.org/T441359#12451607 (10JSimmons) @Lars You're right - it does seem like a lot of... [21:00:49] 06Fundraising-Backlog, 07fr-combowiki: Combowiki: Use forbidden country config - https://phabricator.wikimedia.org/T441736 (10Cstone) 03NEW [21:01:02] RESOLVED: HAProxyExporterDown: HAProxy on fransw1002.frack.eqiad.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [21:02:26] RESOLVED: NodeDown: Node fransw1002 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fransw1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [21:02:48] 06Fundraising-Backlog, 07fr-combowiki: Combowiki: Use forbidden country config - https://phabricator.wikimedia.org/T441736#12451651 (10AnnWF) a:03AnnWF [21:12:13] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add "Send paper receipts" custom field & handle it - https://phabricator.wikimedia.org/T441737 (10Lars) 03NEW [21:13:02] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Add "Send paper receipts" custom field & handle it - https://phabricator.wikimedia.org/T441737#12451690 (10Lars) [21:20:27] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-MGFE, 07fr-current-sprint: Add custom field for event location type (online, hybrid, in person) - https://phabricator.wikimedia.org/T441359#12451708 (10Lars) @JSimmons Sounds good, I'll close this up, but feel... [21:20:36] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-MGFE, 07fr-current-sprint: Add custom field for event location type (online, hybrid, in person) - https://phabricator.wikimedia.org/T441359#12451709 (10Lars) 05Open→03Invalid [21:23:02] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Add "Send paper receipts" custom field & handle it - https://phabricator.wikimedia.org/T441737#12451717 (10Lars) Or consider combining with "Exclude from paper receipt", so we have Send paper receipt: always / never (with unset bein... [21:26:41] 10Fundraising Payments token migration 2026, 06Fundraising-Backlog: Check on adyen ideal Contract not found recurring failures - https://phabricator.wikimedia.org/T440419#12451725 (10MBeat33) 20 out of the 59 had donation creation dates before the range of October 3-12, the rest I have contacted and will follo... [21:26:51] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10fundraising-tech-ops, 03Civi RoadMap FY 26-27, 13Patch-For-Review: FR-Tech FY2627Q2 maintenance window - https://phabricator.wikimedia.org/T419556#12451726 (10Dwisehaupt) [21:55:25] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10fundraising-tech-ops, 03Civi RoadMap FY 26-27: FR-Tech FY2627Q2 maintenance window - https://phabricator.wikimedia.org/T419556#12451865 (10Jgreen) [22:01:48] 03Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10FR-email-preferences: Email Preference center - handle 500 errors better - https://phabricator.wikimedia.org/T440660#12451892 (10Lars) [23:19:46] (03PS1) 10Eileen: Re-add our patches, since merged to 6.21 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1357007 [23:19:46] (03PS1) 10Eileen: Dedupe fix merged to 6.21 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1357008 [23:23:54] (03PS1) 10Eileen: Port rollback fix from 6.21 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1357010 [23:24:04] (03CR) 10CI reject: [V:04-1] Dedupe fix merged to 6.21 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1357008 (owner: 10Eileen) [23:24:17] (03CR) 10CI reject: [V:04-1] Re-add our patches, since merged to 6.21 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1357007 (owner: 10Eileen) [23:27:49] (03CR) 10CI reject: [V:04-1] Port rollback fix from 6.21 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1357010 (owner: 10Eileen) [23:30:16] (03PS4) 10Eileen: CiviCRM 6.20 rc bfea9a1 - stock [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1345248 [23:58:50] (03PS1) 10Eileen: Fix some ts calls [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1357014