[01:52:06] (CR) Aaron Schulz: Convert extractMessageFields() caching to the WAN cache (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/493354 (owner: Aaron Schulz) [01:53:39] (PS1) Aaron Schulz: Fix backwards parameters in cache set() call [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/494150 [01:53:58] (PS2) Aaron Schulz: Fix backwards parameters in cache set() call [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/494150 [01:55:42] (CR) jerkins-bot: [V: -1] Fix backwards parameters in cache set() call [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/494150 (owner: Aaron Schulz) [01:56:37] (CR) jerkins-bot: [V: -1] Fix backwards parameters in cache set() call [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/494150 (owner: Aaron Schulz) [02:09:37] (PS3) Aaron Schulz: Fix backwards parameters in cache set() call [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/494150 [03:05:02] (CR) Krinkle: [C: +2] Fix backwards parameters in cache set() call [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/494150 (owner: Aaron Schulz) [03:08:32] (Merged) jenkins-bot: Fix backwards parameters in cache set() call [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/494150 (owner: Aaron Schulz) [03:15:21] (CR) jenkins-bot: Fix backwards parameters in cache set() call [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/494150 (owner: Aaron Schulz) [10:04:18] fundraising-tech-ops: Backup strategy for Grafana - https://phabricator.wikimedia.org/T217353 (fgiunchedi) @cwdent rsync'ing or otherwise copying the database sounds good to me, in fact in production copying into bacula is what we do for backups (cc @CDanis for visibility) [13:38:30] Fundraising-Backlog, Wikimedia-CentralNotice-Administration: Banner and JS review request for 'stewvote' banner (desktop & mobile) - https://phabricator.wikimedia.org/T214773 (MarcoAurelio) Open→Resolved [14:25:28] (CR) Jforrester: [C: +1] build: Switch from jshint/jscs to eslint [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/493229 (owner: Esanders) [14:26:29] PROBLEM - check_puppetrun on frbackup2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 9 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libssl1.0.0] [14:29:04] fr-tech, I've been digging a little into our ingenico failmail relating to the failing getHostedCheckouts call. The debug output contains the heads and this stood out to me 'Content-Length: 0' so I'm stepping through the code to see how we'd get to the point of submitting the request WITHOUT adding any post params. [14:29:44] the headers*** [14:30:16] "bring me the heads of the errors!!!" [14:31:23] RECOVERY - check_puppetrun on frbackup2001 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [14:31:36] I guess that's how debugging was done in Victorian times. [14:38:16] jgleeson: thanks! I added a bit of debugging a while back to try to catch empty calls, but it hasn't fired on those [14:38:30] I have an idea where it might be failing [14:38:45] but yet to prove it [14:39:10] I'm trying to figure out a way to break json_encode [14:40:36] ejegg, we encode right before the request here but I don't think we check to see if that assignment has a value https://github.com/wikimedia/wikimedia-fundraising-SmashPig/blob/master/PaymentProviders/Ingenico/Api.php#L63 [14:40:51] https://gerrit.wikimedia.org/r/469526 [14:40:54] so if somehow it's failing (due to nonstandard chars or some other means) it might explain why it's empty [14:41:09] hmm, that could be it [14:42:12] there's little in the php.net docs on what actually causes that method to return false [14:43:36] if we can confirm a failure happens, we can get some info via json_last_error() post the failure [14:44:16] http://php.net/manual/en/function.json-last-error.php [14:44:36] the return values for that method indicate what can go wrong [14:50:38] ejegg, if I set that return value to false manually. I get back the dreaded Bad Request 400 error [14:51:01] it also sets the content length to 0 as we're seeeing [14:51:05] -e [14:51:57] maybe if I pulled one of the real failures and see if they're anything unusual about the name of the donor or something else that json_encode doesn't like [14:52:15] there's* [15:52:43] (PS7) AndyRussG: Create CampaignChange hook [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/493459 (https://phabricator.wikimedia.org/T208511) [15:53:15] (CR) jerkins-bot: [V: -1] Create CampaignChange hook [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/493459 (https://phabricator.wikimedia.org/T208511) (owner: AndyRussG) [15:53:39] (PS8) AndyRussG: Create CampaignChange hook [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/493459 (https://phabricator.wikimedia.org/T208511) [15:55:56] (CR) jerkins-bot: [V: -1] Create CampaignChange hook [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/493459 (https://phabricator.wikimedia.org/T208511) (owner: AndyRussG) [16:02:19] (PS9) AndyRussG: Create CampaignChange hook [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/493459 (https://phabricator.wikimedia.org/T217565) [16:05:45] (CR) jerkins-bot: [V: -1] Create CampaignChange hook [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/493459 (https://phabricator.wikimedia.org/T217565) (owner: AndyRussG) [16:09:29] (CR) jerkins-bot: [V: -1] Create CampaignChange hook [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/493459 (https://phabricator.wikimedia.org/T217565) (owner: AndyRussG) [16:10:12] ^ yeah I should run those tests locally first... [16:23:55] jgleeson: yeah, that sounds great (looking at donor info) [16:24:12] unfortunately, I think we json_encode that for logging and for sending to the queues [16:24:19] Have you been able to find anything? [16:25:25] sadly not yet ejegg. I've tried inputting non ut8 characters in but looks like they're being encoded correctly [16:25:52] I can't get much more than the email address of the donor from the logs [17:14:12] Fundraising-Backlog, Fr-Ingenico-integration_2017-18: Ingenico: audit lagging on manual settled transactions? - https://phabricator.wikimedia.org/T217582 (MBeat33) [17:25:23] Fundraising-Backlog: Thank You email: revise how we specify local currency - https://phabricator.wikimedia.org/T207674 (MBeat33) We're still seeing questions from donors about what currency they donated in, after they receive the TY emails. It tends to be from countries whose currencies share the $ symbol, l... [17:31:01] fr-tech has anybody tried to change skin settings at Adyen lately? [17:31:18] They seem to have eliminated that part from the merchant console [17:32:12] I think last time I tried was last summer maybe? It's been a while. [17:36:36] Fundraising-Backlog: Thank You email: revise how we specify local currency - https://phabricator.wikimedia.org/T207674 (CCogdill_WMF) Let's remove the amount from the TY email subject line while we're at it. I think the currency code will look funky there. [19:09:55] fr-tech any tips on getting a bit of output to the command line for debugging when running mw phpunit tests? [19:10:09] or maybe I should just hook up the debugger? [19:10:59] None of these methods seem to work so far: https://stackoverflow.com/questions/7493102/how-to-output-in-cli-during-execution-of-php-unit-tests [19:11:58] hmm [19:12:36] maybe just use error_log to file if it's quick thing? [19:13:25] hmmm where will that go on cli? [19:14:29] I was thinking to file but I wonder if you can direct it to STDOUT [19:16:14] I still it might actually out to STDOUT if you just call error_log("error") [19:16:34] don't see it [19:16:39] ah no AndyRussG it will default to stderr [19:19:05] jgleeson: hmmm no dice so far... php tests/phpunit/phpunit.php --group CentralNotice 2> test.txt [19:19:23] trying to write stuff with error_log() [19:19:34] http://php.net/manual/en/function.error-log.php [19:19:53] the message_type of 4... gives me hope that you might be able to do it [19:20:08] although I'm not sure where that SAPI logging handler is set [19:20:31] I guess in error_log in php.ini? [19:20:42] jgleeson: k gonna try! thx! [19:20:44] what do you see for php -i | grep error [19:21:21] error_log => no value => no value [19:23:24] AndyRussG: is the wfDebugLog function useful here? [19:23:38] ejegg: ah hmm also a thought [19:23:41] Or does it have to be on the command line? [19:24:31] though actually I might be putting the test code in the wrong place, just realized 8p [19:25:53] AndyRussG, try error_log("error!", 4); [19:26:25] jgleeson: yep, tried it [19:26:33] :( [19:27:15] jgleeson: ^ however, I might be doing something else wrong.... [19:55:07] (PS1) Jgleeson: Add additional, additional logging to capture json_encode failures on Ingenico API calls [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/494298 (https://phabricator.wikimedia.org/T202417) [19:56:13] ejegg, I've added a small patch to try and pick up any json_encode failures if they arise & [19:56:17] ^ [19:58:26] AndyRussG, did you work it out by the way> [19:58:27] ? [19:58:51] jgleeson: just resorted to re-reading the code to see if I can figure out the error that way [20:02:04] jgleeson: thanks, I'll take a look! [20:03:23] hmm I just tried running my own local mw tests and now I'm getting some redis connection library error :| [20:03:37] Hmm [20:03:48] jgleeson: that's in vagrant? [20:03:53] yup [20:04:33] I feel like I've seen this before [20:04:36] so the fundraising role relies on redis for the queue, and the redis role sets up redis as mediawiki's session storage and I think object cache [20:04:37] in a previous ife [20:04:39] in a previous life [20:04:55] is the redis server running in the vm? [20:05:07] yup [20:07:54] hmm hmm hmm... is there a password? [20:08:00] nope [20:08:03] really strange [20:08:53] hmmm should I have a redis.ini [20:09:00] I don't see one [20:09:40] jgleeson: that extra logging looks good [20:16:14] hmmm looks like php7.2 on vagrant doesn't install redis [20:28:40] ejegg: I see the process unsubscribes job got disabled again - did your fix resolve the reason it was disabled? [20:29:01] eileen: oh, there's this other patch: https://gerrit.wikimedia.org/r/493759 [20:29:11] turns out some have 'Prospect' in the contact_id [20:29:15] * ejegg shrugs [20:29:34] oh weird! [20:30:48] (CR) Eileen: [C: +2] "Let's give this a whirl but if there are tonnes of them we might need a better fix as we could load 5000 records to process & 5000 of them" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/493759 (owner: Ejegg) [20:31:30] fr-tech, be there in 2 minutes! [20:35:56] (Merged) jenkins-bot: Filter out non-numeric contact_id rows [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/493759 (owner: Ejegg) [20:40:37] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Upstream our remaining Civi patches - https://phabricator.wikimedia.org/T217600 (Eileenmcnaughton) [20:57:58] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Create job to set the field civicrm_email.on_hold to 1 for hard bounce & reply mail block - https://phabricator.wikimedia.org/T217603 (Eileenmcnaughton) [21:00:46] Fundraising Sprint Owls, Fundraising Sprint Pluto is still a planet, Fundraising Sprint Queue is pronounced GJif, Fundraising Sprint Raw data can give you salmonella, and 7 others: slow anonymous save - https://phabricator.wikimedia.org/T199753 (Eileenmcnaughton) [21:00:51] Fundraising Sprint A series of unfortunate event handlers, Fundraising Sprint Bert and Ernie's Excellent Adventure, Fundraising Sprint Casino Royale With Cheese, Fundraising Sprint Da Vinci Coder, and 5 others: slow loading of contribution tab when many con... - https://phabricator.wikimedia.org/T209415 [21:17:36] Fundraising Sprint Casino Royale With Cheese, Fundraising Sprint Da Vinci Coder, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Changes to Benevity import - https://phabricator.wikimedia.org/T215196 (Eileenmcnaughton) Open→Resolved @LeanneS I'm going to close th... [21:24:08] (PS2) Eileen: Add helper for deleting payment processor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/493324 [21:27:49] (CR) jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/494313 (owner: L10n-bot) [21:28:03] ejegg: I also put a patch in frpm git - but just to disabled omnirecipient_repair job [21:40:30] ok cool, that'll go out with the patch then [21:40:52] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/494361 [21:41:05] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/494361 (owner: Ejegg) [21:41:45] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/494361 (owner: Ejegg) [21:46:00] !log updated Fundraising CiviCRM from 616c58cebe to 316e038a69 [21:46:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:49:30] !log re-enabled Omnimail unsubscribe processing, disabled recipient repair job [21:49:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:53:42] (CR) Eileen: Add helper for deleting payment processor (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/493324 (owner: Eileen) [21:54:25] Fundraising Sprint Owls, Fundraising Sprint Pluto is still a planet, Fundraising Sprint Queue is pronounced GJif, Fundraising Sprint Raw data can give you salmonella, and 3 others: Mailing data double counting in CiviCRM? - https://phabricator.wikimedia.org/T200240 (Eileenmcnaughton) [21:54:27] Fundraising Sprint Da Vinci Coder, Fundraising-Backlog, Patch-For-Review: Re-run omnirecipient repair to catch the few missed ones - https://phabricator.wikimedia.org/T215865 (Eileenmcnaughton) Open→Resolved [22:00:27] fr-tech the Adyen skin controls only exist on ca-test, not ca-live [22:01:23] That seems different from last time I was in there, but I guess it was long enough ago that I can't be sure. [22:02:46] (CR) Ejegg: [C: +2] "Thanks! If this doesn't catch it, I wonder if the problem might be in the OutboundRequest or CurlWrapper" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/494298 (https://phabricator.wikimedia.org/T202417) (owner: Jgleeson) [22:03:47] (Merged) jenkins-bot: Add additional, additional logging to capture json_encode failures on Ingenico API calls [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/494298 (https://phabricator.wikimedia.org/T202417) (owner: Jgleeson) [22:03:57] fr-tech anyone free to review these DonationInterface patches? https://gerrit.wikimedia.org/r/493334 https://gerrit.wikimedia.org/r/493618 [22:18:25] Fundraising-Backlog, fundraising-tech-ops: Deploy inconsistency between civi1001 and mintaka - https://phabricator.wikimedia.org/T217342 (Ejegg) That directory is created by civicrm to compile templates to. I guess if nobody's ever looked at Civi on mintaka it wouldn't exist yet? [22:19:23] cstone is that unit test ready to send up to gerrit yet? Or at least a WIP version? [22:23:03] oh yeah! should I tie it to the original change or make it its own thing? [22:23:31] same change please! [22:24:11] ok cool im just finishing eating and then Ill put it up [22:29:24] Fundraising Sprint Da Vinci Coder, Fundraising Sprint They Live, Fundraising Sprint USB stands for underhanded socket bureaucracy, Fundraising Sprint Vestigial tails shoot from the hip, and 3 others: Extend deletion to multiple silverpop databases - https://phabricator.wikimedia.org/T205332 (Eilee... [22:47:42] Fundraising Sprint Owls, Fundraising Sprint Pluto is still a planet, Fundraising Sprint Queue is pronounced GJif, Fundraising Sprint Raw data can give you salmonella, and 3 others: Mailing data double counting in CiviCRM? - https://phabricator.wikimedia.org/T200240 (Eileenmcnaughton) [22:47:46] Fundraising Sprint Da Vinci Coder, Fundraising-Backlog, Patch-For-Review: Re-run omnirecipient repair to catch the few missed ones - https://phabricator.wikimedia.org/T215865 (Eileenmcnaughton) Resolved→Open Dang re-opening this - I realise I filtered on 'Sent' - perhaps for priority reasons... [22:49:50] (PS4) Cstone: Add in check to set no_thank_you field for recurring payments after the first recurring payment. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/490666 (https://phabricator.wikimedia.org/T213209) [22:54:30] ejegg the test is up now [22:54:36] (PS1) Eileen: Update repair omnirecipient to do all event types [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/494371 (https://phabricator.wikimedia.org/T215865) [22:55:35] (Abandoned) Eileen: Remove join on civicrm_financial_type table when rendering search results. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/491398 (owner: Eileen) [22:56:00] dang ejegg looks like we are not yet done with omnirecipient repairs - https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/494371/ [22:59:45] oh fun! [22:59:54] cstone awesome, I'll take a look! [23:00:21] cstone if you have some time for code review, there are a couple of DonationInterface patches in the review column [23:00:34] one of them a bit bigger than the other, but the smaller one depends on it [23:01:00] (CR) Ejegg: [C: +2] Update repair omnirecipient to do all event types [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/494371 (https://phabricator.wikimedia.org/T215865) (owner: Eileen) [23:02:31] has anyone else noticed that `git review -d ` no longer works? [23:05:18] huh it worked for me on Friday but yeah its erroring for me now [23:05:37] (Merged) jenkins-bot: Update repair omnirecipient to do all event types [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/494371 (https://phabricator.wikimedia.org/T215865) (owner: Eileen) [23:06:35] ejegg: I just reverted turning off the repair job - you probably don’t need to check that one huh? [23:06:58] also - hopefully this one is all good now https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/493324/ [23:07:33] also - running a slow query on staging if anyone sees any fallout [23:08:09] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/494373 [23:08:17] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/494373 (owner: Eileen) [23:08:32] yeah cstone it's the case for me now that it rarely works :( [23:09:03] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/494373 (owner: Eileen) [23:09:43] !log civicrm revision changed from 316e038a69 to 196493f372, config revision is 8ca90b4c7b [23:09:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:14:14] (CR) Jgleeson: [C: +2] "Code looks good to me and works as described when 'opt_in=' passed with 1/0/ / in querystring. Tests passing." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/493334 (https://phabricator.wikimedia.org/T217254) (owner: Ejegg) [23:20:36] thanks! [23:20:39] (CR) Jgleeson: [C: +2] "Looks good to me." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/493618 (https://phabricator.wikimedia.org/T215939) (owner: Ejegg) [23:22:24] (Merged) jenkins-bot: Hide opt_in buttons when given on querystring [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/493334 (https://phabricator.wikimedia.org/T217254) (owner: Ejegg) [23:22:26] (Merged) jenkins-bot: Only show emailExplain when opted out [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/493618 (https://phabricator.wikimedia.org/T215939) (owner: Ejegg) [23:25:21] Fundraising Sprint Da Vinci Coder, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic, and 2 others: Reflect all undeliverables via Silverpop in CiviCRM - https://phabricator.wikimedia.org/T161761 (Eileenmcnaughton) OK this is running although it has some catch up to do - `select count(*)... [23:37:43] Fundraising Sprint Da Vinci Coder, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic, and 2 others: Reflect all undeliverables via Silverpop in CiviCRM - https://phabricator.wikimedia.org/T161761 (Eileenmcnaughton) OK - it is dropping `` select count(*) FROM civicrm_mailing_provider_dat... [23:38:04] ejegg: quick q - I think I should resolve the process_unsubscribes job but I feel it would be good to check back on it in a few weeks [23:38:25] we don’t have a way to track things we should ‘check back on’ except maybe a follow on ticket I gues [23:43:01] eileen: right, let's resolve it [23:43:07] maybe a calendar entry? [23:43:34] tickets get lost in the wilderness to the right so easily [23:43:48] lol I was about to say the same for calendar entries :-) [23:43:50] jgleeson: thanks again! [23:44:18] hah, well, let's make a fake little meeting and invite both of us, so there's a chance one doesn't ignore it! [23:44:29] ejegg: ok :-) [23:44:39] 2 weeks away ? [23:44:44] sure! [23:44:59] so it's on 5,000 per run, 2 runs per hour? [23:45:21] 240,000 per day-ish [23:46:20] oh shoot, but that 'numeric' test is dumb, I see. it'll just keep pulling the same 'Prospect' ids and skipping them at the start of the batch. [23:46:45] eileen does the civicrm_api3 params syntax allow a 'BETWEEN' ? [23:46:57] Yes, it does - good thinking [23:47:00] like 'BETWEEN' => [0, 99999999999999999999999] [23:47:05] exactly [23:47:10] thanks, one sec [23:47:47] heh, better make that within the bounds of int, though [23:48:03] :-) [23:51:00] (PS1) Ejegg: Better way to skip non-numeric contact_id [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/494378 [23:53:25] Fundraising Sprint Da Vinci Coder, Fundraising-Backlog, Patch-For-Review: Re-run omnirecipient repair to catch the few missed ones - https://phabricator.wikimedia.org/T215865 (Eileenmcnaughton) OK so here is current for that search (it takes a long time to run - longer with out some filtering like `... [23:56:18] (CR) jerkins-bot: [V: -1] Better way to skip non-numeric contact_id [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/494378 (owner: Ejegg)