[00:39:09] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Civi: multiple issues with recurring donations - https://phabricator.wikimedia.org/T399339#11026443 (10Ejegg) @MBeat33 I deployed a fix that I hope will solve most of the 'request failed validation' issues before tonight's batch of... [01:10:14] PROBLEM - check_load on fran2001 is CRITICAL: LOAD CRITICAL - total load average: 34.35, 25.38, 13.52 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_load [01:35:10] RECOVERY - check_load on fran2001 is OK: LOAD OK - total load average: 0.29, 0.79, 4.48 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_load [06:02:22] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Civi: multiple issues with recurring donations - https://phabricator.wikimedia.org/T399339#11026604 (10MSuijkerbuijk_WMF) @Ejegg Will this fix +21k failed validation recurring from June and July? That is an important amount so tryin... [06:11:14] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 95% used. Largest process: trino-server-co (2229367) = 94.1% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [06:16:14] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 95% used. Largest process: trino-server-co (2229367) = 94.1% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [06:21:16] RECOVERY - check_memory on fransw2003 is OK: OK Memory 75% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [07:30:41] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1171937 (owner: 10L10n-bot) [08:23:49] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Adyen daily audit file check - https://phabricator.wikimedia.org/T399034#11026782 (10RLewis) @MDemosWMF let me know if you need anything else. CID - 241037 Financial type - Total: 50K Date:... [08:31:43] (03CR) 10Reedy: Convert to using virtual domains (031 comment) [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1129227 (https://phabricator.wikimedia.org/T389348) (owner: 10Reedy) [10:55:29] (03CR) 10Raimond Spekking: [C:03+2] "i18n review" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1171937 (owner: 10L10n-bot) [13:27:48] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Civi: multiple issues with recurring donations - https://phabricator.wikimedia.org/T399339#11027785 (10Ejegg) @Mbeat33 The fix seems to have solved the 'bad_request' problems related to Google Pay and Apple Pay for last night's run.... [14:00:46] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Civi: multiple issues with recurring donations - https://phabricator.wikimedia.org/T399339#11027923 (10MSuijkerbuijk_WMF) @Ejegg Is FR Tech to manually retry the 21k that failed in June and July? It is important to go back and retry... [14:04:59] (03PS1) 10Ejegg: setup.sh --civicrm installs our standalone version [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1172041 [14:04:59] (03PS1) 10Ejegg: Point civi scripts at wmf not old wmff [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1172042 [14:23:46] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Civi: multiple issues with recurring donations - https://phabricator.wikimedia.org/T399339#11028059 (10krobinson) @MSuijkerbuijk_WMF before we retry all of them, let's do a donor experience check. These haven't cancelled, so the n... [14:34:29] 10BDC-Implementation: install and automate dbt-metabase package - https://phabricator.wikimedia.org/T389972#11028122 (10IAckerman-WMF) We've promised to turn on self-serve access in Metabase by July 31st. : ) Do you think that timeline will work? [14:41:45] 06Fundraising-Backlog, 10FR-Smashpig, 07payments-orchestration: Gravy IPN listener needs to handle more refund statuses - https://phabricator.wikimedia.org/T400273 (10Ejegg) 03NEW [15:06:13] PROBLEM - check_load on fransw2003 is CRITICAL: LOAD CRITICAL - total load average: 70.61, 52.17, 36.90 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_load [15:10:04] 10fundraising-tech-ops, 06DC-Ops, 10ops-codfw: Q1:rack/setup/install frdata2002 & frmx2002 - https://phabricator.wikimedia.org/T400275 (10RobH) 03NEW [15:10:28] 10fundraising-tech-ops, 06DC-Ops, 10ops-codfw: Q1:rack/setup/install frdata2002 & frmx2002 - https://phabricator.wikimedia.org/T400275#11028250 (10RobH) [15:11:13] RECOVERY - check_load on fransw2003 is OK: LOAD OK - total load average: 23.75, 40.26, 36.81 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_load [15:11:42] 10fundraising-tech-ops, 06DC-Ops, 10ops-codfw: Q1:rack/setup/install frdata2002 & frmx2002 - https://phabricator.wikimedia.org/T400275#11028268 (10RobH) a:03Jgreen @Jgreen, As discussed in IRC, I'm assigning this over to you to double-check the assumed hostnames and update the racking details as you see f... [15:33:13] 10fundraising-tech-ops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: Decommission frack hosts: frpig2001 pay-lvs2001 pay-lvs2002 - https://phabricator.wikimedia.org/T397868#11028369 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [16:15:41] (03PS2) 10Ejegg: setup.sh --civicrm installs our standalone version [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1172041 [16:15:41] (03PS2) 10Ejegg: Point civi scripts at wmf not old wmff [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1172042 [16:17:34] (03PS4) 10Jgleeson: Add support for Gravy 'partial_name_match' AVS result. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1170214 (https://phabricator.wikimedia.org/T399775) [16:17:37] (03CR) 10Ejegg: [C:03+2] Add support for Gravy 'partial_name_match' AVS result. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1170214 (https://phabricator.wikimedia.org/T399775) (owner: 10Jgleeson) [16:38:00] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Civi: multiple issues with recurring donations - https://phabricator.wikimedia.org/T399339#11028684 (10MSuijkerbuijk_WMF) @krobinson @Ejegg @AKanji-WMF Adding all here to review and reach an agreement. The risk here is that donors... [16:45:18] (03PS2) 10Jgleeson: Extract out Gravy response status normalizing to own class [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1167601 (https://phabricator.wikimedia.org/T387411) [16:45:23] (03CR) 10Ejegg: [C:03+2] Extract out Gravy response status normalizing to own class [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1167601 (https://phabricator.wikimedia.org/T387411) (owner: 10Jgleeson) [16:45:59] (03Merged) 10jenkins-bot: Extract out Gravy response status normalizing to own class [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1167601 (https://phabricator.wikimedia.org/T387411) (owner: 10Jgleeson) [16:56:37] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 07fr-acoustic: Don't failmail on first acoustic job error - https://phabricator.wikimedia.org/T397630#11028732 (10Ejegg) We've configured this to wait till 3 failures for the following jobs: omnimail_activity_load omnimail_groupmember_load omnimail_mai... [17:20:49] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 07fr-acoustic: Populating both_funds_latest_donation_source field - https://phabricator.wikimedia.org/T379700#11028877 (10NNgu-WMF) Hi yall, thank you for the work and brainstorming around this so far! I was wondering what... [18:20:13] (03CR) 10Ejegg: [C:03+2] "Boom!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1171717 (owner: 10Eileen) [18:21:37] (03PS3) 10Eileen: Remove drush/drush from composer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1171726 [18:21:45] (03CR) 10Ejegg: [C:03+2] "This is great!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1171726 (owner: 10Eileen) [18:35:13] (03Merged) 10jenkins-bot: Remove drupal directory [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1171717 (owner: 10Eileen) [18:40:03] (03Merged) 10jenkins-bot: Remove drush/drush from composer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1171726 (owner: 10Eileen) [19:03:32] eileen: did you want to discuss the stuff from yesterday in tech talk today? [19:04:35] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Why are donors seeing this error? - https://phabricator.wikimedia.org/T396542#11029207 (10Ejegg) 05Open→03Resolved a:03Ejegg [19:07:47] (03PS7) 10Jgleeson: Remove ACH-specific status mapping from bank payment mapper [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169779 (https://phabricator.wikimedia.org/T399577) [19:07:50] (03CR) 10Ejegg: [C:03+2] Remove ACH-specific status mapping from bank payment mapper [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169779 (https://phabricator.wikimedia.org/T399577) (owner: 10Jgleeson) [19:08:24] (03Merged) 10jenkins-bot: Remove ACH-specific status mapping from bank payment mapper [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169779 (https://phabricator.wikimedia.org/T399577) (owner: 10Jgleeson) [19:09:06] (03PS4) 10Eileen: Standardise / fix getFields function [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170222 (https://phabricator.wikimedia.org/T396071) [19:27:27] ejegg: yea can jump on tech talk [19:35:45] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1172097 [19:35:59] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1172097 (owner: 10Eileen) [19:36:50] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1172097 (owner: 10Eileen) [19:39:34] (03open) 10ejegg: Update CSV import extension [repos/fundraising-tech/community-civicrm] - 10https://gitlab.wikimedia.org/repos/fundraising-tech/community-civicrm/-/merge_requests/19 [19:40:52] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Adyen daily audit file check - https://phabricator.wikimedia.org/T399034#11029341 (10MDemosWMF) @RLewis Thanks! I've entered this on CID 241037. Let me know if there is anything missing! [19:43:21] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170222 (https://phabricator.wikimedia.org/T396071) (owner: 10Eileen) [20:14:54] 10fundraising-tech-ops: Fundraising access request for Hawa Ahmed - https://phabricator.wikimedia.org/T399009#11029411 (10HAhmed-WMF) 05Resolved→03Open Hi team, I'm setting up my Wikimedia laptop and the Yubikey generated a new certificate. I'm no longer able to access Metabase and the codebase. Here's the n... [20:36:42] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Adyen daily audit file check - https://phabricator.wikimedia.org/T399034#11029458 (10Ejegg) @EMartin if we could come up with a special URL parameter to allow for larger donations on payments-w... [20:38:34] !log * civicrm upgraded from 3c23a5c0 to fccd9ef9 [20:38:34] * config revision changed from 9e8116e2 to cf92f15c [20:38:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:38:38] 10fundraising-tech-ops: Replace fundraising LVS service with BIRD/HAProxy service - https://phabricator.wikimedia.org/T398321#11029463 (10Jgreen) We ended up rolling this back because switching to HAProxy breaks multiple things that depend on nginx seeing the real Client IP. Since HAProxy handles the tcp connect... [20:39:46] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Adyen daily audit file check - https://phabricator.wikimedia.org/T399034#11029464 (10EMartin) @Ejegg Sure! Can we share a link to this URL parameter with donors? The 12k block we have on our... [20:41:50] (03PS1) 10Eileen: Remove drupal dir within core [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172109 [20:42:28] ejegg: so the last big in crm (still more in fundraising dev) ^^ [20:46:41] oh yeah, I started deleting wmff in a couple patches in dev: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/dev/+/1172042 [20:47:31] (03CR) 10Ejegg: [C:03+2] Remove drupal dir within core [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172109 (owner: 10Eileen) [20:58:52] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1172112 [20:59:18] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1172112 (owner: 10Eileen) [21:04:27] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1172112 (owner: 10Eileen) [21:17:21] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07Unplanned-Sprint-Work: Activity import issues - https://phabricator.wikimedia.org/T399230#11029500 (10AKanji-WMF) @Eileenmcnaughton sounds like we're waiting on the update to enable the imports Jayme is looking to do? [21:26:26] 10Fundraising analytics stack: [MinIO] Determine the method we want to use for uploading csvs to MinIO to query as tables - https://phabricator.wikimedia.org/T393095#11029520 (10AStein-WMF) @IAckerman-WMF doing some board cleanup. can i get some more details on this? specifically: - are we talking about the abi... [21:28:32] 10Fundraising analytics stack: Vet Acoustic Connect for an audience-building solution for online fundraising team - https://phabricator.wikimedia.org/T387997#11029529 (10AStein-WMF) Closing this for now since it seems like acoustic connect wont be ready any time soon [21:28:36] 10Fundraising analytics stack: Vet Acoustic Connect for an audience-building solution for online fundraising team - https://phabricator.wikimedia.org/T387997#11029530 (10AStein-WMF) 05Open→03Resolved [21:29:23] 10Fundraising analytics stack, 06Data-Engineering-Radar, 06Data-Platform-SRE, 07Epic: [Trino] Connect to Google Sheets - https://phabricator.wikimedia.org/T385748#11029531 (10AStein-WMF) 05Open→03Resolved [21:29:32] 10Fundraising analytics stack, 06Data-Engineering-Radar, 06Data-Platform-SRE, 07Epic: [Trino] Connect to Google Sheets - https://phabricator.wikimedia.org/T385748#11029533 (10AStein-WMF) we decided to use dlt instead of the trino connector [21:53:26] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Infrastructure Security, 06Infrastructure-Foundations, and 2 others: Re-opening our DMarcian Trial - https://phabricator.wikimedia.org/T394788#11029549 (10nisrael) Great thank you Jesse! Just want to confirm, am I safe toinstruct our rep at DMarcian to resta... [21:55:04] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Infrastructure Security, 06Infrastructure-Foundations, and 2 others: Re-opening our DMarcian Trial - https://phabricator.wikimedia.org/T394788#11029551 (10jhathaway) >>! In T394788#11029549, @nisrael wrote: > Great thank you Jesse! Just want to confirm, am I... [23:36:36] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising-Backlog, 07fr-current-sprint: Handle Settlement from Audit process - https://phabricator.wikimedia.org/T399782#11029680 (10Eileenmcnaughton) Just thinking a bit more about it - lets say go with settlement_date as being the date it settles to the payment pro... [23:39:35] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising-Backlog, 07fr-current-sprint: Update Pending code to KEEP pending records - smashpig.pending - https://phabricator.wikimedia.org/T400304 (10Eileenmcnaughton) 03NEW