[00:01:50] (03CR) 10Ejegg: "Will test locally soon, just a couple of questions" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1150259 (owner: 10Eileen) [00:02:44] (03CR) 10Eileen: Adjust bootstrap to run with standalone (032 comments) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1150259 (owner: 10Eileen) [00:35:37] 03Fundraising Sprint: Jollof Rice, 06Fundraising-Backlog, 06FR-donorrelations: Gravy & the PayPal recurring cancellation process - https://phabricator.wikimedia.org/T393226#10858001 (10Cstone) Just to update that civi UI is not making the cancel paypal api call for gravy [08:59:10] ejegg: I just caught up with the updates over the weekend. Nice! [09:13:22] (03PS1) 10Jgleeson: Move standalone scripts under scripts/standalone [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1151137 (https://phabricator.wikimedia.org/T372005) [09:34:23] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Transactions settled at Adyen but stuck at Capturing at Gr4vy? - https://phabricator.wikimedia.org/T393112#10858617 (10jgleeson) Hi @SHust To rebuild and import those two, as well as all the other stuck recurring transactions, into... [10:03:39] 03Fundraising Sprint: Jollof Rice, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Create WMF standalone development build - https://phabricator.wikimedia.org/T372005#10858754 (10jgleeson) After catching up with all the standalone progress during the C... [10:17:48] (03PS1) 10Jgleeson: Update README to include new standalone builds [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1151154 (https://phabricator.wikimedia.org/T372005) [12:50:09] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap: Migrating recurring donors off of Fundraise UP - https://phabricator.wikimedia.org/T390331#10859227 (10Damilare) Status update: Gravy has imported the card payment method from Stripe, awaiting the payment method tokens to import into CiviCRM from Gravy. [13:38:46] 10fundraising-tech-ops, 10FR-Tech-Analytics: Migrate FR DMARC parser and database to FR analytics - https://phabricator.wikimedia.org/T394724#10859397 (10Jgreen) [13:40:58] 10fundraising-tech-ops, 10FR-Tech-Analytics: Migrate FR DMARC parser and database to FR analytics - https://phabricator.wikimedia.org/T394724#10859400 (10Jgreen) [13:57:11] PROBLEM - check_load on fransw2002 is CRITICAL: LOAD CRITICAL - total load average: 5.56, 14.97, 16.00 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [14:07:11] RECOVERY - check_load on fransw2002 is OK: LOAD OK - total load average: 0.07, 2.91, 9.23 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [14:47:31] (03CR) 10Damilare Adedoyin: [C:03+2] Move standalone scripts under scripts/standalone [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1151137 (https://phabricator.wikimedia.org/T372005) (owner: 10Jgleeson) [14:47:36] (03CR) 10Damilare Adedoyin: [V:03+2 C:03+2] Move standalone scripts under scripts/standalone [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1151137 (https://phabricator.wikimedia.org/T372005) (owner: 10Jgleeson) [14:47:39] (03CR) 10Damilare Adedoyin: [C:03+2] Update README to include new standalone builds [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1151154 (https://phabricator.wikimedia.org/T372005) (owner: 10Jgleeson) [14:47:41] (03CR) 10Damilare Adedoyin: [V:03+2 C:03+2] Update README to include new standalone builds [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1151154 (https://phabricator.wikimedia.org/T372005) (owner: 10Jgleeson) [14:47:56] thanks damilare ! [14:50:06] 03Fundraising Sprint: Jollof Rice, 06Fundraising-Backlog, 10fundraising-tech-ops, 06Fundraising-Tech-Roadmap: FR-Tech FY2425Q4 maintenance window - https://phabricator.wikimedia.org/T337585#10859714 (10Jgreen) [14:51:09] np [14:51:15] 03Fundraising Sprint: Jollof Rice, 06Fundraising-Backlog, 10fundraising-tech-ops, 06Fundraising-Tech-Roadmap: FR-Tech FY2425Q4 maintenance window - https://phabricator.wikimedia.org/T337585#10859715 (10Jgreen) [14:52:16] 03Fundraising Sprint: Jollof Rice, 06Fundraising-Backlog, 10fundraising-tech-ops, 06Fundraising-Tech-Roadmap: FR-Tech FY2425Q4 maintenance window - https://phabricator.wikimedia.org/T337585#10859725 (10Jgreen) [14:54:19] 03Fundraising Sprint: Jollof Rice, 06Fundraising-Backlog, 10fundraising-tech-ops, 06Fundraising-Tech-Roadmap: FR-Tech FY2425Q4 maintenance window - https://phabricator.wikimedia.org/T337585#10859742 (10AStein-WMF) [15:03:17] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Trouble voiding transaction at Gr4vy - https://phabricator.wikimedia.org/T394090#10859773 (10MBeat33) 05Open→03Resolved a:03MBeat33 I tested voiding 221fceaa-dab0-4276-b932-b28d8c7085af last week, and it remains at Void st... [15:07:52] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Transactions settled at Adyen but stuck at Capturing at Gr4vy? - https://phabricator.wikimedia.org/T393112#10859790 (10SHust) Sounds great, @jgleeson, thanks for letting me know! [15:09:04] 03Fundraising Sprint: Jollof Rice, 06Fundraising-Backlog, 10fundraising-tech-ops, 06Fundraising-Tech-Roadmap: FR-Tech FY2425Q4 maintenance window - https://phabricator.wikimedia.org/T337585#10859804 (10AStein-WMF) [15:24:21] 03Fundraising Sprint: Jollof Rice, 06Fundraising-Backlog, 10fundraising-tech-ops, 06Fundraising-Tech-Roadmap: FR-Tech FY2425Q4 maintenance window - https://phabricator.wikimedia.org/T337585#10859846 (10Jgreen) [15:28:28] 03Fundraising Sprint: Jollof Rice, 06Fundraising-Backlog, 10fundraising-tech-ops, 06Fundraising-Tech-Roadmap: FR-Tech FY2425Q4 maintenance window - https://phabricator.wikimedia.org/T337585#10859861 (10Jgreen) [15:29:06] 03Fundraising Sprint: Jollof Rice, 06Fundraising-Backlog, 10fundraising-tech-ops, 06Fundraising-Tech-Roadmap: FR-Tech FY2425Q4 maintenance window - https://phabricator.wikimedia.org/T337585#10859863 (10Jgreen) [15:33:24] 03Fundraising Sprint: Jollof Rice, 06Fundraising-Backlog, 10fundraising-tech-ops, 06Fundraising-Tech-Roadmap: FR-Tech FY2425Q4 maintenance window - https://phabricator.wikimedia.org/T337585#10859883 (10Jgreen) [15:50:18] 06Fundraising-Backlog, 06FR-donorrelations: Settled Adyen Donation Not Appearing in Civi - https://phabricator.wikimedia.org/T395340 (10SHust) 03NEW [16:01:49] 03Fundraising Sprint: Jollof Rice, 06Fundraising-Backlog, 10fundraising-tech-ops, 06Fundraising-Tech-Roadmap: FR-Tech FY2425Q4 maintenance window - https://phabricator.wikimedia.org/T337585#10860069 (10Jgreen) [16:03:46] 03Fundraising Sprint: Jollof Rice, 06Fundraising-Backlog, 10fundraising-tech-ops, 06Fundraising-Tech-Roadmap: FR-Tech FY2425Q4 maintenance window - https://phabricator.wikimedia.org/T337585#10860082 (10Jgreen) [16:07:02] (03PS1) 10Jgleeson: Update Gravy snagged transaction importer for recurring [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1151253 (https://phabricator.wikimedia.org/T393112) [16:15:09] !log payments into maintenance mode for kernel update/reboot of frqueue1003 [16:15:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:20:22] !log payments back out of maintenance mode after update/reboot [16:20:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:43:32] !log stopping process-control and coworker on civi1002 and frdev1002 for updates and reboots. [16:43:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:50:10] PROBLEM - check_coworker on civi1002 is CRITICAL: PROCS CRITICAL: 1 process with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_coworker [16:53:07] ^^ known and downtime added. [17:00:11] RECOVERY - check_coworker on civi1002 is OK: PROCS OK: 0 processes with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_coworker [17:01:46] 03Fundraising Sprint: Jollof Rice, 06Fundraising-Backlog, 10fundraising-tech-ops, 06Fundraising-Tech-Roadmap: FR-Tech FY2425Q4 maintenance window - https://phabricator.wikimedia.org/T337585#10860477 (10Dwisehaupt) [17:02:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [17:03:15] PROBLEM - check_mysql on frdb1004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [17:04:45] ^^ this is related to the frdb1005 reboot. hosts are reconnected/ing but alerts are lagging. [17:07:11] RECOVERY - check_mysql on frdb2003 is OK: Uptime: 5331676 Threads: 113 Questions: 7691222057 Slow queries: 50720 Opens: 75198 Open tables: 2699 Queries per second avg: 1442.552 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [17:08:15] RECOVERY - check_mysql on frdb1004 is OK: Uptime: 2397845 Threads: 6 Questions: 161744182 Slow queries: 1184 Opens: 8058 Open tables: 1308 Queries per second avg: 67.453 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [17:43:24] 10fundraising-tech-ops, 10FR-Tech-Analytics, 10FR-tech-data-integrity, 10Data-Engineering (Q4 2025 April 1st - June 30th): Low volume in new webrequest feed - https://phabricator.wikimedia.org/T395089#10860771 (10Ahoelzl) [17:43:32] 10fundraising-tech-ops, 10FR-Tech-Analytics, 10FR-tech-data-integrity, 10Data-Engineering (Q4 2025 April 1st - June 30th): Low volume in new webrequest feed - https://phabricator.wikimedia.org/T395089#10860772 (10Ahoelzl) p:05Triage→03High [17:46:13] 06Fundraising-Backlog: Gravy dLocal placeholder fiscal numbers - https://phabricator.wikimedia.org/T395361 (10jgleeson) 03NEW [17:48:38] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Verify which countries need a 'fiscal_number' to process a payment on Gravy - https://phabricator.wikimedia.org/T394051#10860809 (10jgleeson) Cristiano from Gravy sent this list on Slack (contains duplicates): ` case "AR": return "2... [17:51:06] (03CR) 10Jgleeson: [C:03+2] "self-merging maintenance script" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1151253 (https://phabricator.wikimedia.org/T393112) (owner: 10Jgleeson) [17:53:55] (03Merged) 10jenkins-bot: Update Gravy snagged transaction importer for recurring [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1151253 (https://phabricator.wikimedia.org/T393112) (owner: 10Jgleeson) [17:57:29] PROBLEM - Host frban2002 is DOWN: PING CRITICAL - Packet loss = 100% [18:00:15] RECOVERY - Host frban2002 is UP: PING OK - Packet loss = 0%, RTA = 30.40 ms [18:00:42] (03PS1) 10Jgleeson: Add backend processor fields to Gravy donation message script. [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1151282 (https://phabricator.wikimedia.org/T381012) [18:00:43] (03PS1) 10Jgleeson: Update Gravy snagged transaction importer for recurring [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1151283 (https://phabricator.wikimedia.org/T393112) [18:01:14] (03CR) 10Jgleeson: [C:03+2] Add backend processor fields to Gravy donation message script. [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1151282 (https://phabricator.wikimedia.org/T381012) (owner: 10Jgleeson) [18:01:18] (03CR) 10Jgleeson: [C:03+2] Update Gravy snagged transaction importer for recurring [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1151283 (https://phabricator.wikimedia.org/T393112) (owner: 10Jgleeson) [18:01:36] (03Merged) 10jenkins-bot: Add backend processor fields to Gravy donation message script. [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1151282 (https://phabricator.wikimedia.org/T381012) (owner: 10Jgleeson) [18:01:49] (03Merged) 10jenkins-bot: Update Gravy snagged transaction importer for recurring [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1151283 (https://phabricator.wikimedia.org/T393112) (owner: 10Jgleeson) [18:01:53] fr-tech there's a conflict on smashpig deployment, looks like the ach valdiation is fighting some other validation updates so I've cherry picked around it for now [18:02:09] PROBLEM - check_mysql on frdb1003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1003&service=check_mysql [18:07:09] RECOVERY - check_mysql on frdb1003 is OK: Uptime: 2316962 Threads: 7 Questions: 1130111148 Slow queries: 486 Opens: 29782 Open tables: 2654 Queries per second avg: 487.755 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1003&service=check_mysql [18:08:37] !log SmashPig upgraded from f96b898e to 546d17e5 [18:08:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:16:05] 03Fundraising Sprint: Jollof Rice, 06Fundraising-Backlog, 06FR-donorrelations: Gravy & the PayPal recurring cancellation process - https://phabricator.wikimedia.org/T393226#10860964 (10MBeat33) From today's Fr-Tech checkin, if we could make Civi's API call to cancel recurring donations **also** cancel the au... [18:35:30] 03Fundraising Sprint: Jollof Rice, 06Fundraising-Backlog, 10fundraising-tech-ops, 06Fundraising-Tech-Roadmap: FR-Tech FY2425Q4 maintenance window - https://phabricator.wikimedia.org/T337585#10861027 (10Jgreen) [18:38:16] (03PS3) 10Wfan: Add a placeholder phone number for gravy india card payment [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1139592 (https://phabricator.wikimedia.org/T391970) [18:43:12] (03CR) 10Jgleeson: [C:03+2] "Tested this on tech-talk. LGTM!" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1139592 (https://phabricator.wikimedia.org/T391970) (owner: 10Wfan) [18:43:43] 03Fundraising Sprint: Jollof Rice, 06Fundraising-Backlog, 06FR-donorrelations: Gravy & the PayPal recurring cancellation process - https://phabricator.wikimedia.org/T393226#10861054 (10MBeat33) Also, @krobinson @SHust @AMJohnson because of the new integration Civi-PayPal-Gravy, if we want PayPal donors to au... [18:53:04] (03Merged) 10jenkins-bot: Add a placeholder phone number for gravy india card payment [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1139592 (https://phabricator.wikimedia.org/T391970) (owner: 10Wfan) [18:57:47] 14Fundraising Sprint Impossible shepherds pie: End of Sprint Notes - https://phabricator.wikimedia.org/T394773#10861115 (10Aklapper) @Damilare, @XenoRyet: This task has `open` status but has no non-archived project tags so this open task does not show up on any active workboard. Please either correct the task st... [19:26:26] 03Fundraising Sprint: Jollof Rice, 06Fundraising-Backlog, 10fundraising-tech-ops, 06Fundraising-Tech-Roadmap: FR-Tech FY2425Q4 maintenance window - https://phabricator.wikimedia.org/T337585#10861192 (10Dwisehaupt) [19:27:22] 14Fundraising Sprint Impossible shepherds pie: End of Sprint Notes - https://phabricator.wikimedia.org/T394773#10861195 (10XenoRyet) 05Open→03Resolved [19:38:44] 06Fundraising-Backlog: Gravy dLocal placeholder fiscal numbers - https://phabricator.wikimedia.org/T395361#10861205 (10AnnWF) a:03AnnWF [19:40:21] (03PS3) 10Wfan: Use state code for state if state null [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1149684 (https://phabricator.wikimedia.org/T395129) [19:47:35] 10fundraising-tech-ops, 10FR-Tech-Analytics, 10FR-tech-data-integrity, 10Data-Engineering (Q4 2025 April 1st - June 30th): Low volume in new webrequest feed - https://phabricator.wikimedia.org/T395089#10861217 (10Jgreen) Rerunning this using the full broker list, so we get all the partitions. [19:55:40] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Settled Adyen Donation Not Appearing in Civi - https://phabricator.wikimedia.org/T395340#10861252 (10AKanji-WMF) p:05Triage→03Medium [19:56:51] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Docker: When setup.sh fails on a port already being used, the containers after that aren't set up - https://phabricator.wikimedia.org/T395173#10861256 (10AKanji-WMF) p:05Triage→03Medium [19:59:15] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Deduped contacts - still receiving emails at old address, Email 3 not showing up in Activity - https://phabricator.wikimedia.org/T395150#10861261 (10AKanji-WMF) p:05Triage→03Medium @Eileenmcnaughton I'm moving this t... [19:59:54] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: gravy state null but state_code not null - https://phabricator.wikimedia.org/T395129#10861265 (10AKanji-WMF) p:05Triage→03Medium [20:00:25] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: JO donation form: other amount issue - https://phabricator.wikimedia.org/T395093#10861267 (10AKanji-WMF) p:05Triage→03Low [20:22:10] 06Fundraising-Backlog, 06FR-donorrelations: some PayPal contribs missing PII from Gravy - https://phabricator.wikimedia.org/T395375 (10MBeat33) 03NEW [20:29:00] 06Fundraising-Backlog, 06FR-donorrelations: some PayPal contribs missing PII from Gravy - https://phabricator.wikimedia.org/T395375#10861349 (10jgleeson) This looks related to {T394298} [20:33:09] 06Fundraising-Backlog, 06FR-donorrelations: some PayPal contribs missing PII from Gravy - https://phabricator.wikimedia.org/T395375#10861360 (10EMartin) @jgleeson This is pertinent to Fundraise Up transactions only, correct? [20:41:33] 06Fundraising-Backlog, 06FR-donorrelations: some PayPal contribs missing PII from Gravy - https://phabricator.wikimedia.org/T395375#10861369 (10jgleeson) @EMartin these ones are gravy transactions based on the transaction IDs [20:45:23] 06Fundraising-Backlog, 06FR-donorrelations: some PayPal contribs missing PII from Gravy - https://phabricator.wikimedia.org/T395375#10861391 (10jgleeson) We can follow up with Gravy Tomorrow regarding these items to check for updates on the earlier example we provided. Our understanding is that donor details s... [20:46:13] 06Fundraising-Backlog, 06FR-donorrelations: some PayPal contribs missing PII from Gravy - https://phabricator.wikimedia.org/T395375#10861393 (10MBeat33) thank you @jgleeson, and also for catching that the last one above is Trustly [21:17:10] PROBLEM - check_coworker on frdev1002 is CRITICAL: PROCS CRITICAL: 1 process with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_coworker [21:22:10] RECOVERY - check_coworker on frdev1002 is OK: PROCS OK: 0 processes with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_coworker [21:41:34] PROBLEM - Host payments2005 is DOWN: PING CRITICAL - Packet loss = 100% [21:42:34] ^^ heh. downtime expired mid reboot. it'll be right back up. [21:44:24] PROBLEM - Host payments2004 is DOWN: PING CRITICAL - Packet loss = 100% [21:45:16] RECOVERY - Host payments2005 is UP: PING OK - Packet loss = 0%, RTA = 30.35 ms [21:47:12] RECOVERY - Host payments2004 is UP: PING OK - Packet loss = 0%, RTA = 30.46 ms [23:08:14] PROBLEM - Host frdb1006 is DOWN: PING CRITICAL - Packet loss = 100% [23:11:12] RECOVERY - Host frdb1006 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms [23:59:42] (03PS1) 10Wfan: JOD is three decimal currency [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1151388 (https://phabricator.wikimedia.org/T395093)