[00:30:03] this one has been reviewed upstream https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/civicrm/+/456050/ [00:32:36] (PS1) Eileen: Extended reports update. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/456055 (https://phabricator.wikimedia.org/T151271) [04:30:49] Fundraising-Backlog: Activities tab- multi select option for exclude filter - https://phabricator.wikimedia.org/T203057 (NNichols) [04:53:31] (PS2) Eileen: Extended reports update. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/456055 (https://phabricator.wikimedia.org/T151271) [08:04:18] Fundraising-Backlog: Pre form help option - https://phabricator.wikimedia.org/T203056 (Aklapper) [11:00:23] Fundraising-Backlog, Fr-CentralNotice-translations, MediaWiki-extensions-CentralNotice, Trust-and-Safety, and 3 others: CentralNotice provides a means for non interface-admins to bypass new CSS/JS restrictions - https://phabricator.wikimedia.org/T202244 (Tgr) [12:01:58] Fundraising Sprint Queue is pronounced GJif, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: FY17-18 total not reflected in WMF Donor section - https://phabricator.wikimedia.org/T202102 (Eileenmcnaughton) I looked into this & you have found 4 of a total of 1511 contacts that matched this pattern.... [12:53:43] Fundraising-Backlog: test and deploy payments-listener Debian Stretch / PHP 7 - https://phabricator.wikimedia.org/T197887 (Jgreen) >>! In T197887#4535239, @Ejegg wrote: > What is "test fundraising.wikimedia.org functionality" ? It's a site we support mainly for legacy Paypal recurring payments, at this poin... [13:03:21] Fundraising Sprint Queue is pronounced GJif, Fundraising-Backlog, Fr-Ingenico-integration_2017-18: Implement Ingenico Connect API call to create bank transfer payment - https://phabricator.wikimedia.org/T171311 (jgleeson) Is this one of the ones we moved out in the last sprint because we thought it w... [14:09:07] mepps, I could hear :) [14:09:17] oh ha, you were totally gone for me [14:09:23] ah [14:20:21] (PS4) Jgleeson: WIP: Backend Stats Comparison Script (bannerimpressions part) [wikimedia/fundraising/FRUEC] - https://gerrit.wikimedia.org/r/456016 (https://phabricator.wikimedia.org/T198752) [14:47:24] (Restored) Hashar: Jenkins job validation (DO NOT SUBMIT) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/324066 (owner: Hashar) [14:47:35] (CR) Hashar: "check experimental" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/324066 (owner: Hashar) [15:14:01] fr-tech: any news for scrum of scrums? [15:15:46] fr-tech: also, hi! [15:21:17] fr-tech: being maybe a bit overcautious bc pcoombe is out and I've got a call in 10, just mentioning that no impression data yet for en6C tests that have been up 21 minutes; all seem normal to y'all? [15:21:47] hmmm [15:21:59] spatton_: I'll check the jobs to make sure everything's still running [15:23:04] Thanks ejegg, and they just started showing up! Tends to happen right after I freak out :) [15:23:06] ty [15:27:01] heh, right on [15:31:08] (CR) Hashar: "recheck" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/324066 (owner: Hashar) [15:47:47] seeing lots of Ingenico failmail & error reports from donors in ZD queue [15:49:20] woah fr-tech [15:49:35] failmail attack in relation to MBeat's post [15:49:36] erm yeah [15:49:54] looks like ingenico might be down [15:50:05] gonna see if I can hit the endpoint from payments [15:51:43] holy crap [15:51:58] yeah, lots of timeouts [15:52:25] emailing merchantservices [15:52:46] fr-tech should we stop the test? [15:52:54] yeah, let's [15:52:59] this is very donor facing [15:53:04] hmmm [15:53:09] spatton_: let's take the test down [15:53:23] doing that now. [15:53:42] donors are sending us error ref #s that we can’t see at Ingenico console [15:53:49] ty spatton_ , ejegg [15:54:19] hmm so the iframe is still working... [15:54:55] https://grafana.wikimedia.org/dashboard/db/frtechmail-dashboard?orgId=1&from=now-1h&to=now [15:55:01] ok, 4 campaigns brought down; two for en2C, two for en4C. Yeah, donation counts for, for ex, en4C dsk lg look pretty normal fwiw [15:55:22] mepps, some safari donors are reporting that the card # fields are not opening when they select card type, if that’s iframe-related [15:56:21] yeah looks like ingenico is reachable but their hostedcheckouts service is throwing back weird errors [15:56:32] interesting MBeat, huh gleeson_ [15:56:36] jgleeson_ [15:56:56] we've seen this error before, 'CONNECT_PLATFORM_ERROR' [15:57:26] MBeat: that sounds like a case when our API call to get the iframe URL made it, but when the donor's browser tried to load that URL, Ingenico's web server timed out [15:57:38] So failures from all sides [15:57:57] ty ejegg [15:58:01] we had the same problem on mass on the 25th July [15:58:30] jgleeson_: I think that was when we upped our API call timeout from 7 to 12 seconds, right? [15:58:44] I think so [16:00:47] en2C was set to run another couple hours; no problem to hold or reenable another day, but let me know if things rectify themselves [16:03:49] ejegg, looks like the first request times out and then the follow on gets the failure [16:05:03] jgleeson_: interesting [16:05:58] actually ejegg ignore that, the first request IS NOT timing out, it was only timing out before you upped the timeout from 7 seconds [16:06:02] in July [16:06:30] in this case, the first request is just getting a straight 500 response containing the error [16:06:35] fr-tech, mbeat, spatton_: next time this happens, should our response be to swap CC processors and keep the banners up? [16:07:26] jgleeson_: ejegg spatton_ could we switch to older ingenico so fr-online can continue their banner tests? [16:07:41] or is this affecting all global collect/ingenico? [16:07:59] hmm, interesting question. If this were a continuous campaign - say, JP next week! - keeping banners up (and if emails have gone out) is definitely a priority. For one of these weekly's, I'm not sure there's that level of urgency. [16:08:23] if testing the banners w/other processors is helpful, that’s fine [16:08:24] I'm not familiar with the levers we have to switch processor, others in fr-tech would have to answer that [16:08:28] dstrine, ^ [16:09:43] well, we're specifying the ingenico gateway in the actual banner files, so if we remove that, we'll default to GlobalCollect (I believe) [16:10:58] woah cwd, that's a big jump [16:13:32] interesting that payments1001 has so many fewer than 1002 and 1003 [16:13:41] is it getting less traffic? [16:13:51] ah, you just reminded me that I can't access payments1001 [16:13:59] or just more persuasive at getting a response from the connect API? [16:15:05] ejegg, I can get to payments1004 but not 1001-3, is that normal? [16:15:16] s/get to/ssh in to/ [16:17:25] jgleeson: yah [16:17:33] ah cool [16:18:38] Fundraising-Backlog, Fr-Ingenico-integration_2017-18: Implement Ingenico Connect API call to create bank transfer payment - https://phabricator.wikimedia.org/T171311 (DStrine) [16:18:43] Fundraising-Backlog, Fr-Ingenico-integration_2017-18: Implement Ingenico Connect API call to create bank transfer payment - https://phabricator.wikimedia.org/T171311 (DStrine) I removed it. not sure how that got back in :P [16:30:47] hey fr-tech, we were also planning to run a Japan (ingenic) pre-test later today, at 21 UTC or so. Should we just postpone that until further notice? [16:32:10] spatton_: I think we're OK for a Japan test [16:32:32] thanks ejegg|brb, so - would we be ok to reenable GB & IE too, you think? No pressure just wondering. [16:32:50] spatton_: sure, I think the issue is just a matter of volume. [16:33:07] so without the US on I'm guessing we don't see the failmails [16:35:55] thanks ejegg, note to all fr-tech that I've just reenabled enGB & enIE - running for two hours, 16:35 - 18:35 UTC. LMK if y'all see issues! [16:36:04] 10-4 [16:37:13] in the pipe, 5 by 5 [16:41:11] ejegg: is it ok to ask the platform_error affected donors to retry now? [16:46:49] MBeat: yes, definitely [16:47:35] great, thank you ejegg [17:20:45] fr-tech hmm, looks like the WX audit parser failed the last couple of days too. Taking a look at that [17:28:01] weird, predis connection refused? [17:28:48] hmph, also, those log search lines look fishy [17:45:13] like it's searching for new-API (ingenico) transactions in the old-API [17:45:22] payments logs [17:45:41] gonna get some lunch [19:19:40] fr-tech well FRUEC is now writing LandingPage events to the DB! [19:19:51] wooo AndyRussG! [19:19:56] yeeeeeeeee :) [19:19:57] I'm a bit confused about the lack of utm_source on some events [19:20:19] the table definition sets a "not null" condition [19:22:12] however if it's missing, the script sets it to an empty string [19:22:24] so it's kinda like it's using the empty string instead of null [19:23:02] Same for utm_campaign, utm_medium and utm_key [19:24:16] Seems like we get quite a few without those [19:24:30] Try these queries: [19:24:32] select * from landingpageimpression_raw where utm_source = '' order by timestamp desc limit 20; [19:24:59] select * from landingpageimpression_raw where utm_source = '' and landingpage not like '%default%' order by timestamp desc limit 20; [19:25:23] I guess we can just make the new script do the same as the old, just seems kiiiiiiiiiiinda hacky [19:28:37] anyway, I'll make a note of it in the etherpad [19:34:21] (PS4) AndyRussG: WIP Add landingpage event processing [wikimedia/fundraising/FRUEC] - https://gerrit.wikimedia.org/r/455869 [19:35:15] fr-tech ^ so here's the apparently working FRUEC that loads up landingpages into the DB as well! [19:35:28] still left it as WIP since I just want to double-check stuff first [19:36:12] The issue I mentioned above ^ is now in the etherpood, any insight would be great, thx in advance!!! [19:40:40] K kid pickup time.... [19:49:07] (PS6) Mepps: Rename iface & base class: Unsubscribe -> Subscription [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/452008 (https://phabricator.wikimedia.org/T200576) (owner: Ejegg) [19:49:16] (CR) Mepps: [C: 2] "I definitely like this naming better!" [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/452008 (https://phabricator.wikimedia.org/T200576) (owner: Ejegg) [19:57:03] (Merged) jenkins-bot: Rename iface & base class: Unsubscribe -> Subscription [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/452008 (https://phabricator.wikimedia.org/T200576) (owner: Ejegg) [20:50:17] (CR) jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/456231 (owner: L10n-bot) [20:59:11] (PS6) Mepps: Consolidate flags [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/445534 (https://phabricator.wikimedia.org/T194517) [20:59:13] (PS18) Mepps: Move to problem array, split out functions [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/445535 (https://phabricator.wikimedia.org/T194517) [21:03:56] (CR) Mepps: "ejegg can you be a little more specific both in your instructions but also in the documentation within this patch on how the settings chan" [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/452015 (https://phabricator.wikimedia.org/T200576) (owner: Ejegg) [21:07:40] (PS19) Mepps: Move to problem array, split out functions [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/445535 (https://phabricator.wikimedia.org/T194517) [21:14:37] (Abandoned) Eileen: Drop autoincrement ID and FKs on group_contact_cache [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374588 (https://phabricator.wikimedia.org/T174404) (owner: Ejegg) [21:19:43] (PS3) Eileen: Extended reports update. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/456055 (https://phabricator.wikimedia.org/T151271) [22:02:16] fr-tech anyone around for the tech talk? [22:02:48] ejegg: I’m here [22:02:59] I'm kind of half-here right now. Keeping an eye on Marek. [22:03:09] I could tune in and listen probably, but not much more than that. [22:06:19] cwd is Redis running on staging? [22:06:20] cwd: we're talking about civi caching in tech talk right now [22:12:09] Hey fr-tech, I'm double-checking we're ok for a Japan pre-test at 23 UTC for 2 hours? [22:12:31] tskaff: sounds good on our side! [22:13:04] Thanks ejegg! cc MBeat too [22:13:25] sure thing tskaff [22:14:32] The current form switcher settings are to send JP/JPY users to the old ingenico forms ('globalcollect' gateway) [22:20:44] ty ejegg [22:21:46] Yes thanks ejegg; double-checking I don't need to adjust anything in the banners that are pointing to Special:IngenicoGateway [22:33:40] tskaff: ah, OK, those banners will send people direct to the new Ingenico API [22:33:43] should be fine too [22:33:57] Oh ok phew [22:34:35] you'll just want to look for 'ingenico' as the gateway in Civi's wmf_contribution_extra table, or as the start of the trxn_id [22:34:48] rather than the old 'globalcollect' [22:35:40] Ok got it. [22:39:32] eileen: no it would be a new redis instance [22:40:15] spatton was asking if it's better to take the gateway:'ingenico' specification out of the banners? [22:40:35] https://phabricator.wikimedia.org/T202936 [22:40:54] cwd - perhaps you could add an instance against that Phab? [22:41:17] The civicrm changes to use it are some settings in civicrm.settings.php [22:41:22] Fundraising-Backlog, fundraising-tech-ops: Test Redis caching on Civi staging - https://phabricator.wikimedia.org/T202936 (Ejegg) a:cwdent [22:42:34] Fundraising-Backlog, fundraising-tech-ops: Test Redis caching on Civi staging - https://phabricator.wikimedia.org/T202936 (Eileenmcnaughton) To switch Civi over we need to edit sites/default/civicrm.settings.php specifying password & port & altering the Cache engine in the relevant define in there. [22:42:48] eileen: sure sounds good [22:45:05] hey ejegg! Sorry, I'm 'that guy' getting involved very last minute with JP :) I have caught up just a tad, but can you repeat for my simple ears - which is the better gateway, GlobalCollect or Ingenico? Ingenico ended up being specified as the gateway in these banners, and sounds like that's 'new' Ingenico ... [22:45:16] if you have a preference, it's not too late to cut the specification out so it'll default to whatever you've set for JP [22:45:18] Is that better? [22:47:06] spatton yeah, let's keep 'em on the new API [22:47:24] so, no change to the banners [22:47:52] okydoky, so actually - realized that this Ingenico gateway is only specified in the dsk lg banners. So you'd actually like 'em all pointing to the new API? [22:47:53] The more diverse traffic we can get to the new API, the more I can trust our integration [22:48:01] I can update the others too [22:48:24] spatton: sure, if it's not too much trouble [22:48:27] not at all! [22:48:41] that might be easier for donor services and stats-lookers [22:51:16] Good point. TY ejegg [22:53:45] ejegg: I tried the Redis POC page & it seemed not to work [22:54:11] nice, guess they fixed it [23:16:37] ^ back online a bit later! [23:32:19] ejegg Just an FYI things are looking ok on my end and I'll be running out to pick up my son in a bit. If there's any need to contact me just use email and I can take down campaigns quickly. Many thanks! [23:37:18] (PS1) Eileen: Drupal security updates - to 7.59 [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/456304 [23:38:56] I just put the security update into the review 1 [23:38:58] https://gerrit.wikimedia.org/r/#/dashboard/self [23:39:01] ie. drupal [23:42:51] (CR) jerkins-bot: [V: -1] Drupal security updates - to 7.59 [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/456304 (owner: Eileen)