[00:13:03] (PS3) Ejegg: IDE reformat [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/621127 (owner: Eileen) [00:13:10] (CR) Ejegg: [C: +2] IDE reformat [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/621127 (owner: Eileen) [00:20:30] (Merged) jenkins-bot: IDE reformat [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/621127 (owner: Eileen) [04:03:09] kristi [09:11:51] Wikimedia-Fundraising-Banners: Remove hover text from banners - https://phabricator.wikimedia.org/T258517 (Pcoombe) Not as hover text, since in most cases there is no hover state on mobile. However the aria-labels would be read by screenreaders, so best to remove them where they are not needed. [13:19:54] Fundraising-Backlog, MediaWiki-extensions-DonationInterface, I18n: Donation form should use the appropriate name order - https://phabricator.wikimedia.org/T259945 (CDenes_WMF) Hi All, Adding onto this that we had a Donor Services request to change the last name/first name order for Japanese as well... [13:29:58] tzag fr-tech [13:30:31] I'm gonna check out the paypal email sent over reporting "steep drop off rates" :| [14:27:16] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: GlobalCollect_endpoint_critical 9 [=1] [14:27:48] hmm fr-tech so I looked into the paypal suspected drop off thingy and from what I can see it looks like we don't use the 'DoDirectPayment' call mentioned in Matt's email. We do however use 'DoExpressCheckoutPayment' and we've successfully processed 4978 of those so far today [14:29:16] morning [14:30:29] looks like ingenico connection timeouts or possibly a dns issue. [14:30:55] morning. oh rly dwisehaupt [14:31:05] gotta relocate brb [14:32:02] yeah. stream of failmail. [14:32:16] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: GlobalCollect_endpoint_critical 9 [=1] [14:34:18] i think it's on ingenico's side. i'm seeing the same dns answer as reported in the failmail and seeing my browser spin when trying to hit the url [14:37:16] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: GlobalCollect_endpoint_critical 9 [=1] [14:38:10] dwisehaupt: should we take down the multilingual bundle banners? [14:38:24] that would probably be prudent. [14:38:44] this would be causing issues with people trying to make payments through ingenico [14:39:23] Ok, thanks! Just disabled bundle campaigns [14:40:26] thanks. [14:42:16] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: GlobalCollect_endpoint_critical 9 [=1] [14:44:10] ACKNOWLEDGEMENT - check_log_messages on frav1002 is CRITICAL: CRITICAL: GlobalCollect_endpoint_critical 9 [=1] Dwisehaupt known ingenico issue [14:46:26] Wikimedia-Fundraising-Banners: Remove hover text from banners - https://phabricator.wikimedia.org/T258517 (jbolorinos-ctr) Ok yeah that's what I was wondering, well thank you for confirming Peter! Glad to hear this will also be helpful for screen readers on mobile! [14:57:15] jgleeson: that "steep drop off rates" may be a red herring [14:57:26] i'm not sure we ever DO make DoDirectPayment calls! [14:58:53] yeah looks that way ejegg [15:16:38] Fundraising-Backlog, fundraising-tech-ops, fr-tech-ops-okr: OKR 2019-2020 Q3: Support fundraising through maintaining infrastructure for payments and donor data - https://phabricator.wikimedia.org/T243019 (Jgreen) Open→Resolved [15:24:19] fundraising-tech-ops: 2019-2020 Q3 fundraising hardware refresh and capex - https://phabricator.wikimedia.org/T242257 (Jgreen) [15:24:30] fundraising-tech-ops, Patch-For-Review: rack/setup/install civi2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T242270 (Jgreen) Open→Resolved [15:44:42] Wikimedia-Fundraising-Banners: Remove hover text from banners - https://phabricator.wikimedia.org/T258517 (Pcoombe) Open→Resolved [15:51:44] ejegg: re:paypal thing. Should we go back to matt explaining we don't use that call and the on we do it working fine? [15:51:54] one* [15:52:47] is there a tech contact at ingenico we should reach out to and verify that we are having issues? [15:55:11] dwisehaupt: I think Evelyn is contacting them based on her last reply to the email thread [15:55:42] awesome. [15:59:32] jgleeson|baksoon: oops, just saw that - yeah, I'll reply to matt@paypal [16:26:10] fundraising-tech-ops: 2019-2020 Q3 fundraising hardware refresh and capex - https://phabricator.wikimedia.org/T242257 (Jgreen) Stalled→Resolved [16:26:12] Fundraising-Backlog, fundraising-tech-ops, fr-tech-ops-okr: OKR 2019-2020 Q3: Support fundraising through maintaining infrastructure for payments and donor data - https://phabricator.wikimedia.org/T243019 (Jgreen) [16:27:35] fundraising-tech-ops: 2020-2021 fundraising hardware refresh and capex - https://phabricator.wikimedia.org/T257244 (Jgreen) [16:35:53] fundraising-tech-ops: set up backup cycle for fran1001 - https://phabricator.wikimedia.org/T261003 (Jgreen) [17:03:49] fundraising-tech-ops: set up backup cycle for fran1001 - https://phabricator.wikimedia.org/T261003 (Jgreen) iptables done [17:12:30] (PS1) Jgleeson: Updated OANDA endpoint to www.oanda.com to fix weak DH key problem. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/621774 (https://phabricator.wikimedia.org/T260092) [19:57:10] RECOVERY - check_log_messages on frav1002 is OK: OK