[00:01:51] !log payments-wiki upgraded from eeff4fda to b942983f [00:01:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:03:28] (03Abandoned) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1092279 (owner: 10Ejegg) [00:04:35] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Bank Transfer Transactions: Status Mismatch (Gr4vy/dLocal vs. CiviCRM) - https://phabricator.wikimedia.org/T399577#11033390 (10Ejegg) This is deployed [00:04:57] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising-Backlog: Add partial_name_match AVS response code to Gravy fraud filters - https://phabricator.wikimedia.org/T399775#11033393 (10Ejegg) This is deployed [01:10:13] PROBLEM - check_load on fran2001 is CRITICAL: LOAD CRITICAL - total load average: 35.73, 26.68, 14.14 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_load [01:35:11] RECOVERY - check_load on fran2001 is OK: LOAD OK - total load average: 0.21, 0.74, 4.62 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_load [02:29:58] 06Fundraising-Backlog, 10FR-Donor-portal: Donor Portal - Define activity types for Analytics - https://phabricator.wikimedia.org/T400433 (10AKanji-WMF) 03NEW [03:09:03] 06Fundraising-Backlog: Imports are stalled - need to manually push through - https://phabricator.wikimedia.org/T400419#11033485 (10Eileenmcnaughton) I thought it might have been the last drupal code removal but rolling that didn't help - I'm looking to try to figure out where ` 'api/Exception.php'` should be .... [03:27:21] (03PS1) 10Eileen: Temp hack to try to get coworker running again [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172505 [03:28:15] (03CR) 10Eileen: [C:03+2] "Self +2 as I'm trying to get coworker out of a broken state" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172505 (owner: 10Eileen) [03:57:36] (03CR) 10CI reject: [V:04-1] Temp hack to try to get coworker running again [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172505 (owner: 10Eileen) [03:58:50] (03CR) 10Eileen: [C:03+2] "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172505 (owner: 10Eileen) [03:59:33] 06Fundraising-Backlog: Imports are stalled - need to manually push through - https://phabricator.wikimedia.org/T400419#11033524 (10Eileenmcnaughton) p:05Triage→03High [04:26:16] (03PS2) 10Eileen: Temp hack to try to get coworker running again [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172505 [04:26:26] (03CR) 10Eileen: [C:03+2] Temp hack to try to get coworker running again [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172505 (owner: 10Eileen) [04:41:34] (03Merged) 10jenkins-bot: Temp hack to try to get coworker running again [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172505 (owner: 10Eileen) [04:49:20] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1172508 [04:49:57] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1172508 (owner: 10Eileen) [04:52:02] !log civicrm upgraded from 3c23a5c0 to 8d57b366 [04:52:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:05:07] (03PS2) 10Eileen: Set timeout on sftp connection as well as guzzle [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172169 [05:05:07] (03PS1) 10Eileen: Attempt to add links for files included in Kernel require once [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172509 (https://phabricator.wikimedia.org/T400419) [05:07:20] (03CR) 10Eileen: [C:03+2] Attempt to add links for files included in Kernel require once [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172509 (https://phabricator.wikimedia.org/T400419) (owner: 10Eileen) [05:19:34] (03PS2) 10Eileen: Attempt to add links for files included in Kernel require once [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172509 (https://phabricator.wikimedia.org/T400419) [05:19:40] (03CR) 10Eileen: [V:03+2] Attempt to add links for files included in Kernel require once [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172509 (https://phabricator.wikimedia.org/T400419) (owner: 10Eileen) [05:20:28] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1172510 [05:21:12] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1172510 (owner: 10Eileen) [05:22:01] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1172510 (owner: 10Eileen) [05:46:20] 06Fundraising-Backlog: Imports are stalled - need to manually push through - https://phabricator.wikimedia.org/T400419#11033551 (10Eileenmcnaughton) It seems coworker is not booting correctly - I probably need to talk with @Jgreen & @ejegg to figure how to fix it - it stopped on 22 July & almost certainly relat... [07:09:11] (03PS1) 10Eileen: Hush snooze complaints [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172516 [07:26:04] (03CR) 10CI reject: [V:04-1] Hush snooze complaints [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172516 (owner: 10Eileen) [08:03:52] !log config revision changed from 1e605150 to 1081636a [08:03:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:29:46] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Civi: multiple issues with recurring donations - https://phabricator.wikimedia.org/T399339#11033882 (10krobinson) Approved [[ https://docs.google.com/document/d/11ML0Aeiulj9bI2mD7B0zUIY1XcHUFBW79Bo67mhGjA8/edit?tab=t.0 | draft of Ci... [10:03:49] 10Wikimedia-Fundraising-CiviCRM: CiviCRM Mailing: Monthly Convert - Dynamic Content not rendering - https://phabricator.wikimedia.org/T400454 (10ppenloglou) 03NEW [10:17:44] 06Fundraising-Backlog: Civi donor segment doesn't match with Acoustic donor segment - https://phabricator.wikimedia.org/T400416#11034193 (10Peachey88) [12:48:13] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: ESI test string is still shipped by CentralNotice - https://phabricator.wikimedia.org/T400472 (10R4356th) 03NEW [12:49:16] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 06Traffic: ESI test string is still shipped by CentralNotice - https://phabricator.wikimedia.org/T400472#11034545 (10R4356th) Can SRE please confirm if it can be removed? [14:08:21] 06Fundraising-Backlog, 06FR-donorrelations: Source of Truth Unclear: Boleto Donation Mismatch Between Gravy, dLocal, and Civi - https://phabricator.wikimedia.org/T400479 (10SHust) 03NEW [14:14:34] (03PS1) 10Ejegg: Handle missing resultCode in Adyen recurring [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1172649 (https://phabricator.wikimedia.org/T399021) [14:16:49] 14Fundraising Sprint Impossible shepherds pie, 14Fundraising Sprint: Hot Pot!, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Create CiviCRM Api call to retrieve information for donor portal main page - https://phabricator.wikimedia.org/T388744#11034808 (10AKanji-WMF) @Ejegg Anal... [14:18:35] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 06Traffic: ESI test string is still shipped by CentralNotice - https://phabricator.wikimedia.org/T400472#11034815 (10ssingh) Thanks for reporting. At least on the Traffic/CDN side, the changes were removed in T308799#9752983 where we set it to fals... [14:32:14] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CiviCRM Mailing: Monthly Convert - Dynamic Content not rendering - https://phabricator.wikimedia.org/T400454#11034844 (10greg) [14:43:14] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 84.55, 70.76, 35.62 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:45:20] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 97.75, 85.88, 46.84 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:47:46] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: WMFDonor updates in WMF Thank you are unexpected - https://phabricator.wikimedia.org/T400309#11034884 (10Ejegg) The TY job updates the thankyou_date on civicrm_contribution after each send - I'd imagine this is what's triggering the... [14:48:10] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 55.50, 79.39, 50.86 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:50:18] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 44.29, 70.51, 52.15 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:53:12] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 9.37, 42.10, 43.26 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:54:34] ACKNOWLEDGEMENT - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 9.37, 42.10, 43.26 Jeff_Green Yes, we know. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:54:34] ACKNOWLEDGEMENT - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 44.29, 70.51, 52.15 Jeff_Green Yes, we know. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [15:03:46] que es? ^ [15:13:00] (03PS2) 10Eileen: Use apiv4 in updateWMFDonorLastDonation [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172134 (https://phabricator.wikimedia.org/T400309) [15:13:11] (03CR) 10Ejegg: [C:03+2] "This all looks correct!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172134 (https://phabricator.wikimedia.org/T400309) (owner: 10Eileen) [15:17:08] (03PS2) 10Eileen: Update Contribution.get api call to v4 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172135 (https://phabricator.wikimedia.org/T400309) [15:17:14] (03CR) 10Ejegg: [C:03+2] "Looks good. Thanks for the cleanup!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172135 (https://phabricator.wikimedia.org/T400309) (owner: 10Eileen) [15:23:12] RECOVERY - check_load on frban1002 is OK: LOAD OK - total load average: 4.19, 4.26, 9.89 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [15:25:16] RECOVERY - check_load on frban2002 is OK: LOAD OK - total load average: 3.25, 3.33, 8.92 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [15:29:08] (03Merged) 10jenkins-bot: Use apiv4 in updateWMFDonorLastDonation [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172134 (https://phabricator.wikimedia.org/T400309) (owner: 10Eileen) [15:35:42] (03Merged) 10jenkins-bot: Update Contribution.get api call to v4 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1172135 (https://phabricator.wikimedia.org/T400309) (owner: 10Eileen) [15:41:22] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Civi: multiple issues with recurring donations - https://phabricator.wikimedia.org/T399339#11035137 (10MBeat33) adding Kristie's question from Slack: Did these 6k donors already get the donation failed email? Are they in the recurr... [17:07:13] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 95% used. Largest process: trino-server-co (3093746) = 94.0% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [17:12:13] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 96% used. Largest process: trino-server-co (3093746) = 95.0% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [17:17:15] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 97% used. Largest process: trino-server-co (3093746) = 95.9% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [17:27:13] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 95% used. Largest process: trino-server-co (3093746) = 93.7% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [17:32:13] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 98% used. Largest process: trino-server-co (3093746) = 97.4% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [17:37:11] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 98% used. Largest process: trino-server-co (3093746) = 97.0% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [17:42:13] RECOVERY - check_memory on fransw2002 is OK: OK Memory 51% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [17:49:34] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 06FR-donorrelations: Feature request assessment: Zendesk macro updater - https://phabricator.wikimedia.org/T387503#11035535 (10MBeat33) Breadcrumb from Naa, our Zendesk rep: Zendesk’s Update Many Macros endpoint allows batch updates, and you can specify exa... [18:05:52] (03PS1) 10Ejegg: Rename Adyen3DSecure to Frictionless3DSecure [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1172689 (https://phabricator.wikimedia.org/T398845) [18:07:50] (03CR) 10CI reject: [V:04-1] Rename Adyen3DSecure to Frictionless3DSecure [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1172689 (https://phabricator.wikimedia.org/T398845) (owner: 10Ejegg) [19:10:17] (03PS1) 10Ejegg: Pass through browser_info array to Gravy [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1172702 (https://phabricator.wikimedia.org/T398845) [19:21:49] 10Fundraising analytics stack: Vet Meiro for an audience-building solution for online fundraising team - https://phabricator.wikimedia.org/T387998#11035851 (10AStein-WMF) @IAckerman-WMF i want to close this for now as the team evaluates meiro. lmk if you want to keep it open or reassign until we know if we'll m... [19:24:26] 10Fundraising analytics stack: Select approach for Trino to Sage integration - https://phabricator.wikimedia.org/T388711#11035863 (10AStein-WMF) 05Open→03Stalled [19:24:28] 10Fundraising analytics stack: Select approach for Trino to Sage integration - https://phabricator.wikimedia.org/T388711#11035864 (10AStein-WMF) Seems like there isn't much to do here until some processes are finalized by finance and the civi team. [19:25:07] 10Fundraising analytics stack: [Dagster] Orchestrate scripts - https://phabricator.wikimedia.org/T377354#11035866 (10AStein-WMF) @IAckerman-WMF i'm going to close this for now as specific scripts will have their own phabs [19:25:18] 10Fundraising analytics stack: [Dagster] Orchestrate scripts - https://phabricator.wikimedia.org/T377354#11035868 (10AStein-WMF) 05Open→03Resolved [19:25:50] 10fundraising-tech-ops: Superset and Metabase access for JTanner - https://phabricator.wikimedia.org/T399490#11035870 (10AStein-WMF) we're running into some ssh issues. we'll meet on monday with jospeh to mirror his config [19:25:56] 10fundraising-tech-ops: Superset and Metabase access for JTanner - https://phabricator.wikimedia.org/T399490#11035871 (10AStein-WMF) p:05Triage→03High [20:35:10] 10Fundraising analytics stack: Vet Meiro for an audience-building solution for online fundraising team - https://phabricator.wikimedia.org/T387998#11035999 (10IAckerman-WMF) Sounds good, if there are no infra / hardware aspects to vet.