[00:25:20] (03PS2) 10Cstone: WIP: Add second recurring failure email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1199940 (https://phabricator.wikimedia.org/T365488) [01:17:02] (03PS6) 10Ejegg: Send activity tracking for recurring pause + cancel [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200169 (https://phabricator.wikimedia.org/T407644) [01:17:56] (03PS7) 10Ejegg: Send activity tracking for recurring pause + cancel [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200169 (https://phabricator.wikimedia.org/T407644) [01:20:10] (03CR) 10CI reject: [V:04-1] Send activity tracking for recurring pause + cancel [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200169 (https://phabricator.wikimedia.org/T407644) (owner: 10Ejegg) [01:21:58] (03PS8) 10Ejegg: Send activity tracking for recurring pause + cancel [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200169 (https://phabricator.wikimedia.org/T407644) [01:23:50] (03CR) 10CI reject: [V:04-1] Send activity tracking for recurring pause + cancel [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200169 (https://phabricator.wikimedia.org/T407644) (owner: 10Ejegg) [01:24:44] still??? [01:34:13] (03PS9) 10Ejegg: Send activity tracking for recurring pause + cancel [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200169 (https://phabricator.wikimedia.org/T407644) [01:42:29] finally passing! ^^ [02:04:24] (03PS1) 10Ejegg: Reformat JS tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200198 [02:13:36] (03PS1) 10Ejegg: Fix 'missing required prop' warnings in tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200199 [02:15:46] (03CR) 10CI reject: [V:04-1] Fix 'missing required prop' warnings in tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200199 (owner: 10Ejegg) [02:20:49] (03PS2) 10Ejegg: Fix 'missing required prop' warnings in tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200199 [02:25:46] ejegg: when that silverpop export mysteriously failed last night (maybe from recurring being charged at same time) you just reran it and it worked the 2nd time right? [02:25:54] er last time not last night [02:26:38] looks like it failed at the same spot [02:26:59] ooh you had it in the response ok ill rerun it [02:32:59] hmmmm the id is changing each time [02:36:40] (03PS1) 10Ejegg: Add test for activity tracking parameter [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200200 (https://phabricator.wikimedia.org/T407644) [02:39:14] oh let me see [02:39:35] are those really recent ones? [02:40:09] huh, we had some logic once upon a time to cut off a bit before the very latest CID added, I think maybe to prevent this kind of error [02:47:44] do you remember what those ids are ejegg is it to the email address? [02:47:52] it doesnt look like they are contact ids [02:49:15] yeah, the email [02:49:36] the one from the first crash is for cid 54284285 [02:49:45] no duplicate relationships on that one [02:52:29] i mean, it will probably run if we turn the donations queue off till it gets past that query [02:52:37] but we can't do that every night [02:56:09] yeah i wonder why tonight, more people donating from big email send? [02:56:19] I guess so? [02:56:39] that first one did have a recurring from today [02:56:43] I'm looking at that insert and there's nothing obvious that seems like it could be returning multiple rows [02:56:59] yeah, that first one had a recurring that went in right at the same time they were being exported [02:57:14] did th eothers not? [02:57:18] so does it briefly create duplicate addresses with is_primary or something? [02:57:35] ill make a phab for this since its happened twice [02:57:59] haven't looked at the others, but the same thing happened the other night - it was a crash on a duplicate for one that had a donation importing at exactly the moment of email export [02:58:07] it's still not running, right? [02:59:13] yeah i tried twice more and it broke on different ids [02:59:35] k, i'm going to turn the donations queue off briefly to get it past that query [03:01:21] !log temporarily disabled donations queue consumer to get Acoustic export to work [03:01:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:01:28] on the first one that broke, i dont see any recent activity on their civi account [03:01:34] i wonder why they were in the export [03:01:58] because of the recurring that came in just then, right? [03:02:15] doesnt the export only grab people modified in the last whatever days [03:02:19] or is that query doing everyone [03:02:32] modified includes having a new recurring donation! [03:02:35] oh i see what you mean [03:02:37] hah [03:02:40] how has it not broken before?? [03:02:53] I think there must be a new change to that select [03:02:59] yeah [03:03:09] like maybe the all_funds_latest_donation_date [03:03:12] seems a bit odd [03:03:20] but I don't see how it could make a duplicate [03:03:41] 06Fundraising-Backlog: Silverpop build export files failing during smashpig recurring job - https://phabricator.wikimedia.org/T408887 (10Cstone) 03NEW [03:05:38] no, that donation date stuff is old [03:07:28] cstone: donations queue is off, i'm going to try to run the export again [03:07:39] okie thanks ejegg [03:08:54] boooo [03:11:50] huh [03:12:07] what the heck updated the modified date on cid 41906070 [03:12:08] ? [03:12:39] she would have had a recurring today [03:12:46] maybe setting it to processing? [03:12:51] ohh the recurring charge job picked up the record [03:12:54] yep yep exactly [03:15:57] (03PS1) 10Cstone: WIP: Add activity types for recurring failure email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1200201 (https://phabricator.wikimedia.org/T365488) [03:16:56] ugh, let's just make it ignore ppl changed in the last 3 min [03:18:02] I'm going to turn the qc back on [03:18:35] okie [03:18:49] theres no email sends tomo so [03:19:04] can fix it tomorrow [03:19:19] (03PS1) 10Ejegg: Acoustic export:ignore changes from the last 3 minutes [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1200202 [03:19:27] want to just try that ^^^ [03:19:28] ? [03:19:33] yeah sure ejegg I just know its late there [03:20:43] argh, it really is [03:21:00] and I need to get kiddo to school early tomorrow for halloween parade [03:21:24] !log restarted donations queue consumer [03:21:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:21:44] well, if you feel like deploying and testing that change on production, please go ahead [03:21:56] otherwise I guess we'll see about it tomorrow [03:21:59] thank you! [03:22:05] thanks ejegg [14:48:26] 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2526Q2 maintenance window - https://phabricator.wikimedia.org/T364184#11331446 (10AStein-WMF) [16:08:33] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Snooze status flapping - https://phabricator.wikimedia.org/T402464#11331662 (10SHust) French CID 35879501 was snoozed until 10/31/25 and received the RML on 10/31/25, as shown under the mailing events. However, they donated on 10/17... [16:50:29] 03Fundraising Sprint: UTM_key lime pie, 06Fundraising-Backlog, 10FR-payments-general, 05MW-1.46-notes (1.46.0-wmf.1; 2025-11-05): PaymentsWiki W25 design test - https://phabricator.wikimedia.org/T407241#11331728 (10spatton) Hi @Ejegg sorry for the wrinkle; I just checked with @ehughes and it is not feasibl... [17:20:03] 03Fundraising Sprint: UTM_key lime pie, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing, and 2 others: Batch data entry update - Contact edit pop-out - https://phabricator.wikimedia.org/T408258#11331856 (10MDemosWMF) @Lars Not super urgent but in the edit form for Organizat... [17:41:03] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Non-donors have donor_status_id 100 (should be 1000) since Oct 9 - https://phabricator.wikimedia.org/T408769#11331885 (10Lars) The problem was that the database default was 100, even though the default shown in the UI wa... [18:12:03] (03CR) 10Cstone: [C:03+2] Acoustic export:ignore changes from the last 3 minutes [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1200202 (owner: 10Ejegg) [18:14:35] (03Merged) 10jenkins-bot: Acoustic export:ignore changes from the last 3 minutes [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1200202 (owner: 10Ejegg) [18:42:54] thanks cstone ! [18:43:03] I'll try deploying + running that now [18:45:14] (03PS1) 10Ejegg: Acoustic export:ignore changes from the last 3 minutes [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1200402 [18:45:14] (03PS1) 10Ejegg: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1200403 [18:45:21] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1200403 (owner: 10Ejegg) [18:53:27] hmm, no gate+submit on that one [18:53:40] (03PS1) 10Lars SG: Fix incorrect donor_status_id and default [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1200405 (https://phabricator.wikimedia.org/T408769) [18:53:53] (03CR) 10CI reject: [V:04-1] Fix incorrect donor_status_id and default [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1200405 (https://phabricator.wikimedia.org/T408769) (owner: 10Lars SG) [18:54:09] wat, didn't it say it just merged? [18:54:23] oh i had something on the wrong branch? [18:54:38] lemme fix that [18:55:30] (03PS2) 10Ejegg: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1200403 [18:55:32] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1200403 (owner: 10Ejegg) [18:56:03] (03Merged) 10jenkins-bot: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1200403 (owner: 10Ejegg) [19:00:30] (03PS2) 10Lars SG: Fix incorrect donor_status_id and default [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1200405 (https://phabricator.wikimedia.org/T408769) [19:00:43] (03CR) 10CI reject: [V:04-1] Fix incorrect donor_status_id and default [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1200405 (https://phabricator.wikimedia.org/T408769) (owner: 10Lars SG) [19:01:38] I just re-sent a bunch of damaged donations to the queue - they might error again [19:02:30] (03PS3) 10Lars SG: Fix incorrect donor_status_id and default [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1200405 (https://phabricator.wikimedia.org/T408769) [19:04:52] ok, re-running just the 'build files' export job [19:08:02] got past the step that it was failing on before! [19:08:17] woo nice ejegg [20:09:17] (03PS5) 10Wfan: Only check if checksum exist when email updated [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196546 (https://phabricator.wikimedia.org/T406983) [20:09:18] (03CR) 10Ejegg: [C:03+2] Only check if checksum exist when email updated [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196546 (https://phabricator.wikimedia.org/T406983) (owner: 10Wfan) [20:10:33] (03PS3) 10Cstone: Log the unstaged data when rejected [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1199535 [20:10:35] (03CR) 10Ejegg: [C:03+2] Log the unstaged data when rejected [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1199535 (owner: 10Cstone) [20:19:32] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: Non-donors have donor_status_id 100 (should be 1000) since Oct 9 - https://phabricator.wikimedia.org/T408769#11332352 (10Lars) And one [[ https://github.com/civicrm/civicrm-core/pull/33981 | more up... [20:19:32] (03PS8) 10Wfan: Handle queue differently only send email_checksum if email update [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1195203 (https://phabricator.wikimedia.org/T406983) [20:21:17] (03CR) 10CI reject: [V:04-1] Handle queue differently only send email_checksum if email update [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1195203 (https://phabricator.wikimedia.org/T406983) (owner: 10Wfan) [20:26:05] (03Merged) 10jenkins-bot: Only check if checksum exist when email updated [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196546 (https://phabricator.wikimedia.org/T406983) (owner: 10Wfan) [20:28:34] (03Merged) 10jenkins-bot: Log the unstaged data when rejected [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1199535 (owner: 10Cstone) [20:29:11] (03PS9) 10Wfan: Handle queue differently only send email_checksum if email update [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1195203 (https://phabricator.wikimedia.org/T406983) [20:31:32] (03CR) 10CI reject: [V:04-1] Handle queue differently only send email_checksum if email update [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1195203 (https://phabricator.wikimedia.org/T406983) (owner: 10Wfan) [20:45:22] (03PS3) 10Eileen: Upgrade damaged to entityType 2 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1198360 [20:47:35] (03CR) 10Ejegg: [C:03+2] "It still works for me locally, too. Though I still get 'BAO for User is not available. This could be a load-order issue' the issue trying " [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1198360 (owner: 10Eileen) [20:50:23] (03PS1) 10Ejegg: Allow array in ThankYouPage indexed by appeal [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200426 (https://phabricator.wikimedia.org/T407241) [20:52:57] (03CR) 10CI reject: [V:04-1] Allow array in ThankYouPage indexed by appeal [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200426 (https://phabricator.wikimedia.org/T407241) (owner: 10Ejegg) [21:03:02] (03Merged) 10jenkins-bot: Upgrade damaged to entityType 2 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1198360 (owner: 10Eileen) [21:47:14] (03PS10) 10Wfan: Handle queue differently only send email_checksum if email update [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1195203 (https://phabricator.wikimedia.org/T406983) [21:48:34] (03CR) 10CI reject: [V:04-1] Handle queue differently only send email_checksum if email update [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1195203 (https://phabricator.wikimedia.org/T406983) (owner: 10Wfan) [21:53:04] (03PS11) 10Wfan: Handle queue differently only send email_checksum if email update [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1195203 (https://phabricator.wikimedia.org/T406983) [21:54:49] (03CR) 10CI reject: [V:04-1] Handle queue differently only send email_checksum if email update [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1195203 (https://phabricator.wikimedia.org/T406983) (owner: 10Wfan) [22:58:51] 03Fundraising Sprint: UTM_key lime pie, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing, and 2 others: Batch data entry update - Contact edit pop-out - https://phabricator.wikimedia.org/T408258#11332657 (10Lars) @MDemosWMF I've seen that a few times as well. It is supposed... [23:16:15] 03Fundraising Sprint: UTM_key lime pie, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing, and 2 others: Batch data entry update - Contact edit pop-out - https://phabricator.wikimedia.org/T408258#11332688 (10Lars) [[ https://lab.civicrm.org/dev/core/-/issues/6187 | Upstream i... [23:33:12] PROBLEM - check_ipsec on frban1002 is CRITICAL: Strongswan CRITICAL - ok: 1 not-conn: frdev1002_v4 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_ipsec [23:38:10] PROBLEM - check_ipsec on frban1002 is CRITICAL: Strongswan CRITICAL - ok: 1 not-conn: frdev1002_v4 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_ipsec [23:40:52] ^^ addressing that. it's related to the frdev reboot [23:42:21] ok. should recover on the next check. [23:43:05] 10fundraising-tech-ops: Schedule fr-tech reboots for microcode updates - https://phabricator.wikimedia.org/T408742#11332716 (10Dwisehaupt) [23:43:10] RECOVERY - check_ipsec on frban1002 is OK: Strongswan OK - 2 ESP OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_ipsec [23:46:07] (03PS12) 10Wfan: Handle queue differently only send email_checksum if email update [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1195203 (https://phabricator.wikimedia.org/T406983)