[00:37:06] (PS3) Eileen: Fix cleanup on opt-in test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/566377 [00:37:08] (PS3) Eileen: Add basic thank-you extension, exposing our thank you logic as a civi api [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/566413 (https://phabricator.wikimedia.org/T227903) [00:38:38] (CR) Eileen: "I fixed the casing - I think the index really is just an index in terms of 0 vs an id - when we need to refer to it again a known index i" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/566377 (owner: Eileen) [00:43:03] (CR) jerkins-bot: [V: -1] Fix cleanup on opt-in test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/566377 (owner: Eileen) [00:43:13] (CR) jerkins-bot: [V: -1] Add basic thank-you extension, exposing our thank you logic as a civi api [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/566413 (https://phabricator.wikimedia.org/T227903) (owner: Eileen) [00:50:43] Fundraising-Backlog, fundraising-tech-ops, Epic: Epic: civi crm db approaching data partition capacity - https://phabricator.wikimedia.org/T241083 (DStrine) [00:56:30] Fundraising-Backlog, fundraising-tech-ops: Decide what to do about the increasing civi crm db size - https://phabricator.wikimedia.org/T243870 (DStrine) [00:56:44] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops: Decide what to do about the increasing civi crm db size - https://phabricator.wikimedia.org/T243870 (DStrine) [00:56:54] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops, Fr-planning-ahead: Decide what to do about the increasing civi crm db size - https://phabricator.wikimedia.org/T243870 (DStrine) [01:00:44] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops, Fr-planning-ahead: Decide what to do about the increasing civi crm db size - https://phabricator.wikimedia.org/T243870 (DStrine) [02:02:57] Fundraising-Backlog, FR-Ingenico: January spike in duplicate Ingenico donations? - https://phabricator.wikimedia.org/T243873 (MBeat33) [02:05:43] Fundraising-Backlog, FR-Ingenico: January spike in duplicate Ingenico donations? - https://phabricator.wikimedia.org/T243873 (MBeat33) [05:03:28] Fundraising Sprint Snoop (Dogg|Lion), Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice, and 3 others: Central Notice message groups are slow to index - https://phabricator.wikimedia.org/T111189 (abi_) Some logs from my local after merging Niklas' patch,... [06:09:05] Fundraising-Backlog, FR-Ingenico: January spike in duplicate Ingenico donations? - https://phabricator.wikimedia.org/T243873 (MBeat33) Evidence this may be a may be a donor facing bug: **Zendesk #726556** "I tried to make a donation to Wikimedia, but realized that I was incorrectly logging my Mastercard... [06:09:09] Fundraising-Backlog, FR-Ingenico: January spike in duplicate Ingenico donations? - https://phabricator.wikimedia.org/T243873 (MBeat33) [06:37:11] PROBLEM - check_puppetrun on frav1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[mariadb-client] [06:42:11] PROBLEM - check_puppetrun on frav1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[mariadb-client] [06:47:11] PROBLEM - check_puppetrun on frav1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Package[mariadb-client] [06:52:11] PROBLEM - check_puppetrun on frav1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[mariadb-client] [06:57:11] PROBLEM - check_puppetrun on frav1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[mariadb-client] [06:59:01] ACKNOWLEDGEMENT - check_puppetrun on frav1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[mariadb-client] Dwisehaupt Looks like there may be a conflict with mariadb client versions not meeting requirements. acking for now. [07:13:12] Fundraising-Backlog, FR-Ingenico: January spike in duplicate Ingenico donations? - https://phabricator.wikimedia.org/T243873 (MBeat33) PayPal seems to be affected too: **ZD 726346** "Hello,I tried to contribute$50 via paypal & was under impression it didn't go thru. So then I found another way to give sa... [07:13:24] Fundraising-Backlog, FR-Ingenico: January spike in duplicate donations? - https://phabricator.wikimedia.org/T243873 (MBeat33) [08:34:13] Fundraising-Backlog, FR-Ingenico: January spike in duplicate donations? - https://phabricator.wikimedia.org/T243873 (Cstone) I looked into this a bit. Since it affected both ingenico and paypal as well as different card types maybe a more site wide issue? Some of them happened seconds apart while others... [15:52:17] Fundraising-Backlog, fundraising-tech-ops: Get new CiviCRM cert - https://phabricator.wikimedia.org/T243909 (Dwisehaupt) [16:33:53] Fundraising-Backlog, fundraising-tech-ops: Get new CiviCRM cert - https://phabricator.wikimedia.org/T243909 (Dwisehaupt) Your certificate has been regenerated and sent via email. I have sent the password via SMS. Please let me know if you need any help. [19:33:51] Fundraising-Backlog, fundraising-tech-ops: Get new CiviCRM cert - https://phabricator.wikimedia.org/T243909 (kaythaney) Thanks! All set! [20:08:18] if anyone is about at some point I'd like to know how people feel about me doing the civi upgrade tomorrow - it's not the sort of way we normally organise an outage but it gets it done when it seems to work for people (ie. Rosie not working, everyone else at all hands) but due to all hands probably not many people read my emails. My Monday, your Sunday is also an option [20:08:37] the slowness of it makes it trickier than normal [21:27:33] Fundraising-Backlog, fundraising-tech-ops: Get new CiviCRM cert - kthaney - https://phabricator.wikimedia.org/T243909 (Dwisehaupt) [21:28:01] Fundraising-Backlog, fundraising-tech-ops: Get new CiviCRM cert - kthaney - https://phabricator.wikimedia.org/T243909 (Dwisehaupt) Open→Resolved p:Triage→Normal [21:41:08] Fundraising-Backlog, fundraising-tech-ops: Issue new SSL Client Certificate for rkhumar - https://phabricator.wikimedia.org/T243039 (Dwisehaupt) a:Dwisehaupt Certificate regenerated and sent via email. Password sent via SMS. [21:47:08] Fundraising Sprint Byzantine Empire Strikes Back, Fundraising-Backlog, FR-Ingenico: January spike in duplicate donations? - https://phabricator.wikimedia.org/T243873 (DStrine) p:Triage→High [22:07:14] Fundraising Sprint Byzantine Empire Strikes Back, Fundraising-Backlog, FR-Ingenico: January spike in duplicate donations? - https://phabricator.wikimedia.org/T243873 (Ejegg) There are timeouts in the logs for both the PayPal and the Ingenico donations mentioned here. Looking at email addresses with m... [22:18:34] Fundraising Sprint Byzantine Empire Strikes Back, Fundraising-Backlog, Recurring-Donations: Civi: January 2020 Ingenico recurrings processing in duplicate? - https://phabricator.wikimedia.org/T243356 (AndyRussG) Here are the queries we used for this: - P10261 Find donors who got charged too soon. -... [22:52:15] RECOVERY - check_puppetrun on frav1002 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:26:21] Fundraising Sprint Byzantine Empire Strikes Back, Fundraising-Backlog, FR-Ingenico: January spike in duplicate donations? - https://phabricator.wikimedia.org/T243873 (MBeat33) Thank you @Ejegg and @Cstone