[00:18:47] fr-tech this patch seems to fix it for me: https://github.com/ejegg/omnimail-silverpop/commit/61143f56bddac79fabf847b570b78029d86ed487 [00:19:07] we could point omnimail-silverpop to my fork temporarily [00:32:55] (03PS1) 10Ejegg: Temporarily point omnimail-silverpop at fixed fork [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165193 [00:32:59] ok, this seems to do it ^^^ [00:33:18] wfan: do you have a moment to review that? [00:34:02] The issue that required the rollback is described here: https://github.com/eileenmcnaughton/omnimail-silverpop/issues/13 [00:42:16] hmm, i'm feeling like it's less bad to self-merge than to leave that annoying failmail overnight [00:43:33] (03CR) 10Ejegg: [C:03+2] Temporarily point omnimail-silverpop at fixed fork [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165193 (owner: 10Ejegg) [00:46:05] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1165194 [00:46:37] (03CR) 10CI reject: [V:04-1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1165194 (owner: 10Ejegg) [00:47:33] (03PS1) 10Ejegg: Update omnimail-silverpop [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1165196 [00:47:35] (03CR) 10Ejegg: [C:03+2] Update omnimail-silverpop [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1165196 (owner: 10Ejegg) [00:48:58] (03PS2) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1165194 [00:49:00] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1165194 (owner: 10Ejegg) [00:52:58] (03CR) 10CI reject: [V:04-1] Temporarily point omnimail-silverpop at fixed fork [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165193 (owner: 10Ejegg) [00:53:21] argh, all the tests [00:53:37] maybe just a search+replace [00:55:37] ok, removing my C+2 [00:58:26] (03CR) 10CI reject: [V:04-1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1165194 (owner: 10Ejegg) [01:02:35] (03PS2) 10Ejegg: Temporarily point omnimail-silverpop at fixed fork [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165193 [01:13:23] (03PS1) 10Ejegg: Merge branch 'REL1_43' into fundraising/REL1_43 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1165200 [01:13:27] (03CR) 10Ejegg: [C:03+2] Merge branch 'REL1_43' into fundraising/REL1_43 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1165200 (owner: 10Ejegg) [01:13:49] (03CR) 10Ejegg: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165193 (owner: 10Ejegg) [01:14:04] oops, that was already testing [01:17:00] (03CR) 10Ejegg: [C:03+2] Temporarily point omnimail-silverpop at fixed fork [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165193 (owner: 10Ejegg) [01:17:56] (03Merged) 10jenkins-bot: Update omnimail-silverpop [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1165196 (owner: 10Ejegg) [01:18:37] (03PS3) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1165194 [01:22:24] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1165194 (owner: 10Ejegg) [01:23:44] (03CR) 10CI reject: [V:04-1] Merge branch 'REL1_43' into fundraising/REL1_43 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1165200 (owner: 10Ejegg) [01:23:50] oh hmm [01:24:07] ah lock file error [01:25:15] (03PS2) 10Ejegg: Merge branch 'REL1_43' into fundraising/REL1_43 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1165200 [01:33:04] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1165194 (owner: 10Ejegg) [01:37:20] (03CR) 10CI reject: [V:04-1] Merge branch 'REL1_43' into fundraising/REL1_43 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1165200 (owner: 10Ejegg) [01:42:24] uh, weird, failures in core tests [01:46:20] !log fundraising civicrm upgraded from e35d3778 to 5ae93148 [01:46:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:56:20] (03PS1) 10Ejegg: Update SmashPig [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1165203 [01:57:31] (03CR) 10Ejegg: [C:03+2] Update SmashPig [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1165203 (owner: 10Ejegg) [01:57:47] (03PS1) 10Wfan: add wmf to urls for setup script [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1165204 [01:58:13] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1165205 [01:58:20] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1165205 (owner: 10Ejegg) [01:59:43] (03Merged) 10jenkins-bot: Update SmashPig [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1165203 (owner: 10Ejegg) [01:59:44] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1165205 (owner: 10Ejegg) [02:08:53] (03PS1) 10Ejegg: Update composer.lock [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1165207 [02:09:08] (03CR) 10Ejegg: [C:03+2] Update composer.lock [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1165207 (owner: 10Ejegg) [02:11:17] well, I can get that IP patch out anyway [02:13:30] !log payments-wiki upgraded from 52f6940f to a92f03c3 [02:13:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:22:21] (03Merged) 10jenkins-bot: Update composer.lock [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1165207 (owner: 10Ejegg) [02:25:52] 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 06FR-donorrelations, 07payments-orchestration: pass IP info to Gravy to enable search - https://phabricator.wikimedia.org/T397763#10961747 (10Ejegg) 05Open→03Resolved a:03Ejegg OK @MBeat33, this is deployed and IP addresses are now... [02:32:03] (03PS3) 10Ejegg: Merge branch 'REL1_43' into fundraising/REL1_43 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1165200 [02:44:46] (03CR) 10CI reject: [V:04-1] Merge branch 'REL1_43' into fundraising/REL1_43 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1165200 (owner: 10Ejegg) [02:54:37] hmph, looks like an error in the backport [03:18:43] (03PS1) 10Eileen: Revert "Temporarily point omnimail-silverpop at fixed fork" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165210 [03:33:39] (03CR) 10CI reject: [V:04-1] Revert "Temporarily point omnimail-silverpop at fixed fork" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165210 (owner: 10Eileen) [03:52:04] (03PS2) 10Eileen: Revert "Temporarily point omnimail-silverpop at fixed fork" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165210 [06:18:20] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10961884 (10ayounsi) @Jgreen please sync up with us on IRC for the Netops part. [12:34:37] 10fundraising-tech-ops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: Decommission frack hosts: frnetmon1001 - https://phabricator.wikimedia.org/T398079#10963058 (10Jclark-ctr) [12:34:45] 10fundraising-tech-ops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: Decommission frack hosts: frnetmon1001 - https://phabricator.wikimedia.org/T398079#10963059 (10Jclark-ctr) 05Open→03Resolved a:03Jclark-ctr [13:01:24] 10fundraising-tech-ops, 10FR-Tech-Analytics: Configure Fundraising eqiad analytics cluster - https://phabricator.wikimedia.org/T398317 (10Jgreen) 03NEW [13:08:38] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10963400 (10ayounsi) [13:13:33] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10963414 (10Jgreen) [13:26:23] 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 06FR-donorrelations, 07payments-orchestration: pass IP info to Gravy to enable search - https://phabricator.wikimedia.org/T397763#10963471 (10MBeat33) many thanks, @Ejegg. I've added an item for Gravy to make that info more readily visible. [13:27:49] happy new fiscal year fr-tech! [13:28:00] :) [13:28:28] I had the idea in my head that it was tonight at midnight GMT that we needed to do the recalculation but no, the optimal time would have been last night. [13:28:45] so I guess we can go ahead with that whenever it's OK to put Civi in maintenance mode [13:30:26] (03CR) 10Ejegg: [C:03+2] Revert "Temporarily point omnimail-silverpop at fixed fork" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165210 (owner: 10Eileen) [13:31:14] (03PS2) 10Jgleeson: Switch to Logger::debug() for Maintenance scripts [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1164213 [13:31:14] (03PS2) 10Jgleeson: Log and return early if AVS & CVV are empty in RiskScorer. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1164214 (https://phabricator.wikimedia.org/T396355) [13:31:15] (03PS2) 10Jgleeson: Refactor: Replace Trustly string with PaymentMethod enum value [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1164215 [13:31:15] (03PS2) 10Jgleeson: Refactor: remove unused Gravy mapper methods [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1164216 [13:31:15] (03PS2) 10Jgleeson: Refactor: smol visibility update [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1164217 [13:48:40] (03Merged) 10jenkins-bot: Revert "Temporarily point omnimail-silverpop at fixed fork" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165210 (owner: 10Eileen) [13:52:25] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10963580 (10Jgreen) 05Open→03Resolved a:03Jgreen [13:53:03] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10963585 (10Jgreen) p:05Triage→03Medium [13:56:13] 10fundraising-tech-ops: Replace fundraising LVS service with BIRD/HAProxy service - https://phabricator.wikimedia.org/T398321 (10Jgreen) 03NEW [13:56:26] 10fundraising-tech-ops: Replace fundraising LVS service with BIRD/HAProxy service - https://phabricator.wikimedia.org/T398321#10963606 (10Jgreen) [13:56:31] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10963607 (10Jgreen) [13:57:25] 10fundraising-tech-ops: Replace fundraising LVS service with BIRD/HAProxy service - https://phabricator.wikimedia.org/T398321#10963608 (10Jgreen) [13:57:29] 10fundraising-tech-ops: Replace fundraising LVS service with BIRD/HAProxy service - https://phabricator.wikimedia.org/T398321#10963610 (10Jgreen) [13:57:34] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: Test prototype fundraising pybal replacement based on haproxy + anycast-healthchecker. - https://phabricator.wikimedia.org/T373942#10963611 (10Jgreen) [13:57:56] 10fundraising-tech-ops: Replace fundraising LVS service with BIRD/HAProxy service - https://phabricator.wikimedia.org/T398321#10963613 (10Jgreen) [13:57:58] 10fundraising-tech-ops, 06SRE: Q1:rack/setup/install frqueue2003, pay-lb2001, pay-lb2002 - https://phabricator.wikimedia.org/T369566#10963614 (10Jgreen) [14:02:21] (03PS1) 10Ejegg: Merge branch 'REL1_43' into fundraising/REL1_43 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1165535 [14:02:23] (03CR) 10Ejegg: [C:03+2] Merge branch 'REL1_43' into fundraising/REL1_43 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1165535 (owner: 10Ejegg) [14:03:40] k, looks like they fixed the bug with the backported tests - i think that should merge this time ^^^ [14:05:44] (03Abandoned) 10Ejegg: Merge branch 'REL1_43' into fundraising/REL1_43 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1165200 (owner: 10Ejegg) [14:06:29] (03CR) 10Ejegg: [C:03+2] Switch to Logger::debug() for Maintenance scripts [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1164213 (owner: 10Jgleeson) [14:07:08] (03Merged) 10jenkins-bot: Switch to Logger::debug() for Maintenance scripts [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1164213 (owner: 10Jgleeson) [14:23:57] (03PS2) 10Ejegg: Maintenance script to get Gravy refund details [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1163912 [14:28:59] (03CR) 10Ejegg: [C:03+2] Maintenance script to get Gravy refund details [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1163912 (owner: 10Ejegg) [14:29:29] (03Merged) 10jenkins-bot: Maintenance script to get Gravy refund details [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1163912 (owner: 10Ejegg) [14:47:59] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1165547 [14:49:46] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1165547 (owner: 10Ejegg) [14:50:10] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1165547 (owner: 10Ejegg) [15:08:44] !log standalone SmashPig upgraded from ad4baa32 to bdc59e01 [15:08:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:12:57] oh derp, I didn't merge my own maintenance script [15:15:04] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1165552 [15:16:10] PROBLEM - check_listener_ipn on frpig1002 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 241 bytes in 0.017 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1002&service=check_listener_ipn [15:18:09] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1165552 (owner: 10Ejegg) [15:18:33] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1165552 (owner: 10Ejegg) [15:21:12] PROBLEM - check_listener_ipn on frpig1002 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 241 bytes in 0.016 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1002&service=check_listener_ipn [15:21:51] !log SmashPig upgraded from bdc59e01 to 8486f9fb [15:21:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:21:59] ooh, what's up with those failmails? [15:22:44] argh, missing vendor update [15:24:07] (03PS1) 10Ejegg: Add psr/container [wikimedia/fundraising/SmashPig/vendor] - 10https://gerrit.wikimedia.org/r/1165559 [15:24:09] (03CR) 10Ejegg: [C:03+2] Add psr/container [wikimedia/fundraising/SmashPig/vendor] - 10https://gerrit.wikimedia.org/r/1165559 (owner: 10Ejegg) [15:24:36] (03CR) 10Ejegg: [V:03+2 C:03+2] Add psr/container [wikimedia/fundraising/SmashPig/vendor] - 10https://gerrit.wikimedia.org/r/1165559 (owner: 10Ejegg) [15:24:45] (03PS1) 10Ejegg: Missing vendor update [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1165560 [15:24:55] (03CR) 10Ejegg: [V:03+2 C:03+2] Missing vendor update [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1165560 (owner: 10Ejegg) [15:25:38] !log SmashPig upgraded from 8486f9fb to 52397453 [15:25:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:25:45] sorry for the failstorm fr-tech [15:26:14] RECOVERY - check_listener_ipn on frpig1002 is OK: HTTP OK: HTTP/1.1 200 OK - 482 bytes in 0.028 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1002&service=check_listener_ipn [15:47:04] oh weird, resource not found when I try to look up the refund [16:14:30] fr-tech it would be nice to log either the full incoming gravy IPN or the full job message [16:14:36] let's see if that's just upping the log level [16:15:04] ohhh wait for my current purposes i think i can just look at the damaged queue [16:16:04] annd it's the same ID as the parent txn... [16:16:32] so we're calling getRefundDetails [16:17:35] and not getting anything, from what I can see in the maintenance script [16:33:44] 10fundraising-tech-ops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: Decommission frack hosts: civi2001 - https://phabricator.wikimedia.org/T397380#10964583 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [16:34:15] (03Merged) 10jenkins-bot: Merge branch 'REL1_43' into fundraising/REL1_43 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1165535 (owner: 10Ejegg) [16:36:40] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: handle ach recurring with no email issue - https://phabricator.wikimedia.org/T397855#10964600 (10AnnWF) We have couple orders that is missing emails for recurring: steps to stop them: search kit for email is not exist https://civicrm.wikimedia.org/civicrm... [17:00:20] ejegg: I added a comment on that when I reviewed the patch [17:02:41] looks like it got self-merged https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/SmashPig/+/1163912 [17:02:44] (03PS1) 10Wfan: add default value for template view when proxy not returing contribution data [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1165576 [17:12:36] ack, sorry jgleeson [17:12:41] I should revert that [17:12:55] I was just really curious to see what was coming back in the code for the RefundAction [17:13:04] so yeah, maybe let's just change the RefundAction? [17:13:15] or maybe it actually works with some of the other Gravy ones? [17:14:03] Yeah I can't recall whether I checked the SDK docs, that might help [17:15:41] so let's see if the refund queue logs have any gravy ones that made it in [17:15:44] (03PS1) 10Wfan: Add space in between links for epc and donor potal [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165580 [17:16:46] (03PS2) 10Wfan: Add space in between links for epc and donor potal [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165580 [17:18:09] 06Fundraising-Backlog, 06FR-donorrelations, 10Fr-drupal-upgrade-2025: Request to Revert field back to what it was - https://phabricator.wikimedia.org/T397342#10964797 (10AnnWF) [17:19:25] oho, looks like we previously got the backend_processor populated [17:21:24] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Add space in between links for epc, donor portal and recurring upgrade - https://phabricator.wikimedia.org/T398355 (10AnnWF) 03NEW [17:21:47] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Add space in between links for epc, donor portal and recurring upgrade - https://phabricator.wikimedia.org/T398355#10964832 (10AnnWF) p:05Triage→03Low a:03AnnWF [17:22:34] (03PS3) 10Wfan: Add space in between links of epc, donor portal and recurring upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165580 (https://phabricator.wikimedia.org/T398355) [17:38:52] (03CR) 10Ejegg: [C:03+2] Add space in between links of epc, donor portal and recurring upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165580 (https://phabricator.wikimedia.org/T398355) (owner: 10Wfan) [17:39:47] (03PS2) 10Eileen: Initial annual WMF Donor rollover [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1163916 (https://phabricator.wikimedia.org/T383195) [17:39:48] (03CR) 10Ejegg: [C:03+2] Initial annual WMF Donor rollover [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1163916 (https://phabricator.wikimedia.org/T383195) (owner: 10Eileen) [17:41:34] Jeff_Green: will you be free in about one hour to put civi in maintenance mode and update triggers? [17:41:43] 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 07fr-current-sprint, 07payments-orchestration: How will we expose payment processor errors/issues with gravy - https://phabricator.wikimedia.org/T387411#10964898 (10jgleeson) [17:41:51] hmm, or maybe we should give a little more notice [17:41:57] maybe right after sprint planning? [17:42:13] yup, I'm around either way [17:42:36] k, let's do it at 4pm EST [17:43:25] oh yeah, I'm going to try to sit in on sprint planning while dallas is out, but I don't think I'm on the last for those meetings [17:55:10] (03Merged) 10jenkins-bot: Add space in between links of epc, donor portal and recurring upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165580 (https://phabricator.wikimedia.org/T398355) (owner: 10Wfan) [17:55:11] (03Merged) 10jenkins-bot: Initial annual WMF Donor rollover [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1163916 (https://phabricator.wikimedia.org/T383195) (owner: 10Eileen) [17:58:27] (03PS5) 10Damilare Adedoyin: WIP [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1163874 [17:58:39] (03CR) 10CI reject: [V:04-1] WIP [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1163874 (owner: 10Damilare Adedoyin) [18:02:08] PROBLEM - check_memory on fransw2001 is CRITICAL: CRIT Memory 95% used. Largest process: trino-server-co (20695) = 93.6% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory [18:07:10] PROBLEM - check_memory on fransw2001 is CRITICAL: CRIT Memory 95% used. Largest process: trino-server-co (20695) = 93.6% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory [18:07:11] argh I copypasted HTML into an etherpad and now it's broken [18:07:15] https://etherpad.wikimedia.org/p/Fr-tech-update-segments [18:07:40] even broke the timeslider??? [18:08:04] ah ok, fixed on reload [18:12:10] PROBLEM - check_memory on fransw2001 is CRITICAL: CRIT Memory 96% used. Largest process: trino-server-co (20695) = 94.4% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory [18:17:10] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap: Support for MOTO transactions for direct mail - https://phabricator.wikimedia.org/T396511#10964961 (10EMartin) This may be on hold for now. Engage is not PCI compliant presently for MOTO to Gr4vy. [18:17:10] PROBLEM - check_memory on fransw2001 is CRITICAL: CRIT Memory 96% used. Largest process: trino-server-co (20695) = 94.4% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory [18:24:11] ok, got a few steps outlined in that etherpad for the segment updates [18:28:30] hmm, maybe we want to actually set @civicrm_disable_logging=1 when we do that wmfdonor update [18:29:30] oh i see, the trigger on wmf_donor just sets modified_date on civicrm_contact [18:30:25] which is good, so we get the contacts added to the silverpop export [18:30:35] hmm, though we could check to see if there were any differences [18:32:10] RECOVERY - check_memory on fransw2001 is OK: OK Memory 70% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory [18:52:20] (03PS11) 10Eileen: Update geocoder to latest remote [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1160374 (https://phabricator.wikimedia.org/T379684) [18:52:25] (03CR) 10Ejegg: [C:03+2] Update geocoder to latest remote [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1160374 (https://phabricator.wikimedia.org/T379684) (owner: 10Eileen) [19:07:49] (03Merged) 10jenkins-bot: Update geocoder to latest remote [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1160374 (https://phabricator.wikimedia.org/T379684) (owner: 10Eileen) [19:15:06] 03Fundraising Sprint M - 2025, 06Fundraising-Backlog: Map consolidated Gr4vy file to Civi field and table locations - https://phabricator.wikimedia.org/T397662#10965099 (10XenoRyet) [19:15:07] 03Fundraising Sprint M - 2025, 06Fundraising-Backlog, 07fr-current-sprint: Benevity - moving possible duplicate errors to group for dedupe - https://phabricator.wikimedia.org/T396535#10965100 (10XenoRyet) [19:15:09] 03Fundraising Sprint M - 2025, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations, and 2 others: Civi recurring donations display question - https://phabricator.wikimedia.org/T393227#10965101 (10XenoRyet) [19:15:11] 03Fundraising Sprint M - 2025, 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint: Major Gifts + segmentation framework - do softcredits show up OK - https://phabricator.wikimedia.org/T392034#10965102 (10XenoRyet) [19:15:12] 03Fundraising Sprint M - 2025, 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 07Fundraising-FRUP, 07fr-current-sprint: Payment method migration: Credit cards - Import tokens to Civi - https://phabricator.wikimedia.org/T391991#10965103 (10XenoRyet) [19:15:14] 03Fundraising Sprint M - 2025, 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 07Epic, 07fr-current-sprint: Setting up consolidated settlement reporting for all PSPs via Gr4vy for new recon process - https://phabricator.wikimedia.org/T391496#10965104 (10XenoRyet) [19:15:16] 03Fundraising Sprint M - 2025, 06Fundraising-Backlog, 07payments-orchestration: Add VIPPS in Norway - https://phabricator.wikimedia.org/T390870#10965105 (10XenoRyet) [19:15:22] 03Fundraising Sprint M - 2025, 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 10FR-Donor-portal, and 3 others: Create donor portal page for recurring donation pause/cancel process - https://phabricator.wikimedia.org/T388749#10965106 (10XenoRyet) [19:15:28] 03Fundraising Sprint M - 2025, 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 10FR-Donor-portal, and 3 others: Allow donors to request a donor portal login link by email - https://phabricator.wikimedia.org/T388739#10965107 (10XenoRyet) [19:15:32] 03Fundraising Sprint M - 2025, 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 07fr-current-sprint, 07payments-orchestration: How will we expose payment processor errors/issues with gravy - https://phabricator.wikimedia.org/T387411#10965109 (10XenoRyet) [19:15:34] 03Fundraising Sprint M - 2025, 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 10FR-email-preferences, 07fr-current-sprint: Comms preference centre - email duplication check - https://phabricator.wikimedia.org/T385911#10965110 (10XenoRyet) [19:15:38] 03Fundraising Sprint M - 2025, 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 07Epic, and 2 others: Gravy Dlocal - https://phabricator.wikimedia.org/T384967#10965111 (10XenoRyet) [19:15:42] 03Fundraising Sprint M - 2025, 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint: Come July 1 we need to roll over the donor segment & status values - https://phabricator.wikimedia.org/T383195#10965112 (10XenoRyet) [19:15:46] 03Fundraising Sprint M - 2025, 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Fix dedupe merge wrapper script to track modified date - https://phabricator.wikimedia.org/T383194#10965113 (10XenoRyet) [19:15:50] 03Fundraising Sprint M - 2025, 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 07fr-current-sprint, 07payments-orchestration: Set up recurring charge for venmo - https://phabricator.wikimedia.org/T376122#10965114 (10XenoRyet) [19:15:54] 03Fundraising Sprint M - 2025, 06Fundraising-Backlog, 07fr-current-sprint, 13Patch-For-Review: 2nd Recurring Failure Email in Civi - https://phabricator.wikimedia.org/T365488#10965115 (10XenoRyet) [19:16:02] 03Fundraising Sprint M - 2025, 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 06FR-donorrelations, 07fr-current-sprint: Gravy & the PayPal recurring cancellation process - https://phabricator.wikimedia.org/T393226#10965117 (10XenoRyet) [19:16:08] 03Fundraising Sprint M - 2025, 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Fix resource url pointer in wmf_civicrm - https://phabricator.wikimedia.org/T396174#10965118 (10XenoRyet) [19:16:14] 03Fundraising Sprint M - 2025, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Fr-drupal-upgrade-2025, 07fr-current-sprint: Standalone dev builds - enable display_errors - https://phabricator.wikimedia.org/T396171#10965119 (10XenoRyet) [19:35:34] 06Fundraising-Backlog, 06FR-donorrelations: Schedule email send to gravy failed recurring donors - https://phabricator.wikimedia.org/T398362 (10SHust) 03NEW [19:48:50] wow, it took Paypal nearly 24 hours to cancel that recurring payment! [20:02:47] jgleeson: r u around for that segment meeting [20:02:54] coming! [20:02:55] !log disabled queue consumers for segment updates [20:02:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:03:20] Jeff_Green: would you mind putting the civi UI in maintenance mode please? [20:03:32] yup [20:06:04] done [20:06:16] thanks! [20:10:16] (03PS1) 10Ejegg: Update triggers for new FY and standalone [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165595 [20:10:48] 06Fundraising-Backlog, 06FR-donorrelations: Feature request assessment: Zendesk macro updater - https://phabricator.wikimedia.org/T387503#10965254 (10MBeat33) Prioritized after Gravy and donor portal, and after EN6C [20:11:05] (03PS2) 10Ejegg: Update triggers for new FY and standalone [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165595 (https://phabricator.wikimedia.org/T383195) [20:14:09] (03CR) 10Wfan: [C:03+2] Update triggers for new FY and standalone [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1165595 (https://phabricator.wikimedia.org/T383195) (owner: 10Ejegg) [20:14:30] (03PS1) 10Ejegg: Update omnimail-silverpop repo pointer [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1165596 [20:14:32] (03CR) 10Ejegg: [C:03+2] Update omnimail-silverpop repo pointer [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1165596 (owner: 10Ejegg) [20:15:10] PROBLEM - check_coworker on civi1002 is CRITICAL: PROCS CRITICAL: 1 process with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_coworker [20:15:13] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1165597 [20:16:39] 03Fundraising Sprint M - 2025, 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Come July 1 we need to roll over the donor segment & status values - https://phabricator.wikimedia.org/T383195#10965258 (10AnnWF) a:03Ejegg [20:19:38] (03CR) 10Ejegg: [V:03+2 C:03+2] Update omnimail-silverpop repo pointer [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1165596 (owner: 10Ejegg) [20:20:10] PROBLEM - check_coworker on civi1002 is CRITICAL: PROCS CRITICAL: 1 process with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_coworker [20:25:09] PROBLEM - check_coworker on civi1002 is CRITICAL: PROCS CRITICAL: 1 process with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_coworker [20:27:32] (03CR) 10Wfan: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1165597 (owner: 10Ejegg) [20:30:09] PROBLEM - check_coworker on civi1002 is CRITICAL: PROCS CRITICAL: 1 process with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_coworker [20:35:09] PROBLEM - check_coworker on civi1002 is CRITICAL: PROCS CRITICAL: 1 process with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_coworker [20:38:35] (03CR) 10Ejegg: [V:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1165597 (owner: 10Ejegg) [20:39:54] !log fundraising civicrm upgraded from 5ae93148 to 521d0dbe [20:39:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:40:09] PROBLEM - check_coworker on civi1002 is CRITICAL: PROCS CRITICAL: 1 process with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_coworker [20:40:42] Jeff_Green: OK, that's the new trigger SQL file deployed to production [20:41:00] ok [20:41:26] it's checked in at /ext/wmf-civicrm/sql/triggers.mysql [20:41:39] Could you please run that if it looks good? [20:41:51] this was the diff: https://gerrit.wikimedia.org/r/1165595 [20:42:02] yes, going through the guide here https://wikitech.wikimedia.org/wiki/Fundraising/techops/procedures/services-civicrm_database_trigger_updates [20:42:59] ah, cool, I guess we need to stop replication first? [20:43:12] ahh, and I can update that one path for where it is on standalone [20:43:42] oh I hadn't thought of that [20:44:03] the symlink is still there, but i've updated it with the shorter path [20:45:11] PROBLEM - check_coworker on civi1002 is CRITICAL: PROCS CRITICAL: 1 process with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_coworker [20:50:09] PROBLEM - check_coworker on civi1002 is CRITICAL: PROCS CRITICAL: 1 process with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_coworker [20:52:28] Ok, the triggers are applied. There are changes, they look plausible :-| [20:52:57] ejegg: if I put the before/after triggers somewhere you can see them, can you double check that the changes are as you expect? [20:53:13] PROBLEM - check_mysql on frdb1004 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [20:53:52] you can ignore all the replica status alerts, that's b/c I stopped replication on a bunch of hosts, I'll ack the alerts in a sec [20:54:38] ok, sure Jeff_Green I can take a look [20:55:03] ejegg: they're at frpm1002:/tmp/triggers.* [20:55:09] PROBLEM - check_coworker on civi1002 is CRITICAL: PROCS CRITICAL: 1 process with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_coworker [20:56:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [20:56:32] ACKNOWLEDGEMENT - check_mysql on frdb1004 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) Jeff_Green this is intentional https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [20:56:32] ACKNOWLEDGEMENT - check_mysql on frdb2003 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) Jeff_Green this is intentional https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [20:56:32] ACKNOWLEDGEMENT - check_mysql on frdb2005 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) Jeff_Green this is intentional https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [20:58:08] Jeff_Green: yep, those look like the expected changes. Proceed! [20:58:21] ok thx! [20:59:12] is there anything to do other than take civicrm out of maint mode and restart replication? [20:59:40] after you restart replication I'll run cv updb to queue up some updates [20:59:47] and then we can take it out of maint mode [21:00:00] and i'll try a few queue consumers with small batches [21:00:05] I don't think I need to restart replication to do that [21:00:10] PROBLEM - check_coworker on civi1002 is CRITICAL: PROCS CRITICAL: 1 process with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_coworker [21:00:12] oh ok [21:00:17] I can run it now then? [21:00:32] yep [21:01:06] ah dang, Failed to initialize upgrade data folder: /var/www/.cv/upgrade [21:01:13] I think there's a ticket for that [21:01:31] I guess I'll have to do that through the UI then [21:02:36] T395977 [21:02:37] T395977: Problem upgrading CiviCRM with cv - https://phabricator.wikimedia.org/T395977 [21:03:08] Jeff_Green: want to just re-enable replication and take it out of maintenance mode? [21:03:19] sure [21:05:10] RECOVERY - check_coworker on civi1002 is OK: PROCS OK: 1 process with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_coworker [21:05:20] done! [21:06:05] ugh, i'm getting an Access denied loop [21:06:12] RECOVERY - check_mysql on frdb2005 is OK: Uptime: 2865568 Threads: 4 Questions: 215973770 Slow queries: 1417 Opens: 6668 Open tables: 1044 Queries per second avg: 75.368 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [21:06:13] will try clearing cookies [21:08:12] RECOVERY - check_mysql on frdb1004 is OK: Uptime: 2303386 Threads: 5 Questions: 115971896 Slow queries: 1218 Opens: 5636 Open tables: 1043 Queries per second avg: 50.348 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [21:16:17] ejegg: how's it looking? [21:17:06] Jeff_Green: I was able to run the upgrade from the commandline after all, using a different (deprecated) cv command to just run the extension updates [21:17:16] so the calculations should be queued up [21:17:20] but now I can't seem to log in [21:17:26] ohhh that might have to do with replication? [21:17:40] I don't think so [21:17:43] could it be querying a replica and not finding something [21:17:54] afaik we aren't currently pushing queries to the replica [21:18:03] Jeff_Green: can you log in? [21:18:16] I don't think I have a functional civi account [21:18:56] Jeff_Green: when you disable replication do you somehow disable the rpow extension that routes queries to the r/o db? [21:19:42] no, double checking but I'm almost positive we ended up pointing that also to the origin server because of race conditions [21:20:32] yeah, it's pointed at frdb1005 [21:21:06] they're configured as fundraisingdb-read.wmnet and currently that is a cname for frdb1005.frack.eqiad.wmnet [21:21:30] oh ok, huh, i have no idea why i'm unable to log in then [21:21:36] will try another browser [21:22:13] no dice [21:22:19] will try a password reset [21:23:11] anything in the logs? [21:27:39] waa - wfan is looking in the user admin and I have no account suddenly [21:28:06] looking in the db logs [21:28:18] wat [21:29:04] could it be from using the deprecated cv command? [21:29:38] oh, i hope not [21:29:47] what else could it be? [21:29:50] that would be bizarre [21:30:42] ugh, the sql:cli cv command isn't getting me into the db either [21:30:54] Failed opening required 'ext/rpow/autoload.php' [21:31:14] is replication going again? [21:31:35] yes [21:32:01] but all the traffic is surely hitting frdb1005 [21:32:13] ok, will look at the log on frdev [21:33:32] deleted at 2025-07-01 [21:33:46] how??? [21:33:50] this is from binlogs? [21:33:53] oh there's a uid I think [21:34:02] this is in the log_civicrm_uf_match [21:34:27] ok so users must be in civicrm_uf_match [21:55:28] 06Fundraising-Backlog: Fast track Phab for Amazon Pay - https://phabricator.wikimedia.org/T398377 (10EMartin) 03NEW [22:01:30] 06Fundraising-Backlog: make wmf-cv sql:cli works - https://phabricator.wikimedia.org/T398378 (10AnnWF) 03NEW [22:06:03] !log fundraising scheduled jobs restarted [22:06:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:08:07] 03Fundraising Sprint M - 2025, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations, and 2 others: Civi recurring donations display question - https://phabricator.wikimedia.org/T393227#10965582 (10MBeat33) We're testing the WIP view. Thank you @AMJohnson for noticing that the payme... [22:10:51] ok, back in tech-talk [22:14:20] is coworker running? [22:14:48] That queue task only has run_count 1 [22:15:41] oh wait, the contact ID params are incremented [22:15:47] so it seems to be working [22:19:39] 03Fundraising Sprint M - 2025, 03Fundraising Sprint: Left-Handed Churro , 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint: Come July 1 we need to roll over the donor segment & status values - https://phabricator.wikimedia.org/T383195#10965590 (10Ejegg) Triggers are updated and... [22:20:46] ok, gotta deploy the payments-wiki upgrade [22:22:38] !log payments-wiki upgraded from a92f03c3 to 9c7f3a73 [22:22:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:23:53] ok, that still seems to work [22:52:17] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Epc consumer with invalid data - https://phabricator.wikimedia.org/T398381 (10AnnWF) 03NEW [22:52:51] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Epc consumer with invalid data - https://phabricator.wikimedia.org/T398381#10965673 (10Ejegg) Seems to be an expired checksum