[09:55:24] Fundraising Sprint Unbreaking Now, Fundraising Sprint Value Subtracting, Fundraising-Backlog, FR-2016-17-Q2-Campaign-Support, Patch-For-Review: Add field to Silverpop export: highest_native_amount - https://phabricator.wikimedia.org/T135407#2791945 (CCogdill_WMF) @Ejegg yes, this is so wonder... [16:29:05] hi cwd [16:29:51] mornin! [16:30:36] cwd you have something going for the paypal failmail? [16:31:04] well it's a tough one [16:31:19] i emailed a few times over the weekend but i think they were lost in the noise [16:31:44] this should shut it up: https://gerrit.wikimedia.org/r/#/c/321016/ [16:31:46] shoot, sorry [16:32:27] ah no worries, it is an awkward situation where those failures seem to be pretty intermittent [16:32:28] cwd we could just disable the listener for now [16:32:39] oh huh, that stinks [16:32:56] yeah, just a small % of donations so i didn't want to disable [16:33:14] but enough to be a pretty aggravating amount of spam [16:33:37] i also emailed PP asking what's up, they said a ticket has been filed [16:33:46] right, we should hound them on that [16:33:50] they are returning "fatal failure" when we try to verify the message [16:34:01] like literally an html page with that as the body text [16:34:01] yeah, I see that in the failmail [16:34:08] well, nice to have that cURL logging [16:34:14] ya [16:34:25] i was also talking to Jeff_Green about a more general way to shut up fail mails [16:35:58] cwd for smashpig, we can override the failmail logging stream per-gateway [16:36:21] ah ha [16:36:33] unfortunately that's the settings file we need Jeff_Green to help deploy [16:36:39] nuts [16:38:13] ejegg couldn't alert settings be pulled out to a file that's part of the main codebase? [16:47:31] or even broadcast using localsettings? [17:40:38] (CR) Ejegg: "Ugh... https://www.paypal-knowledge.com/infocenter/index?page=content&id=FAQ1336&actp=LIST" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321016 (owner: Cdentinger) [18:00:38] fr-tech: One big pile is better than two little piles. [18:00:38] -- Arlo Guthrie [18:00:38] -- discuss. [18:00:56] Hmmm it depends on what [18:01:15] Thumbtacks? [18:01:37] Piranha? [18:01:58] Not wishing to knock Arlo G., tho [18:02:29] nuclear fuel rods? [18:05:27] so critical [18:07:30] heheh [18:08:04] Superconductors at near-zero degrees K? [19:30:53] (PS1) Cdentinger: Throw ListenerSecurityException on broken response. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321474 [19:31:38] ejegg|afk: ^ there's that. i have a better fix incoming that requires a little refactoring but this should shut the mail up and cause them to resend later [19:32:35] (CR) Ejegg: "Also need to catch this in Listener.php and set the response status, right? For other gateways that happens in ListenerBase, which you've " [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321474 (owner: Cdentinger) [19:34:22] ah thanks, i see now [19:39:01] (CR) Cdentinger: "Do you mean here? https://github.com/wikimedia/wikimedia-fundraising-SmashPig/blob/master/Core/Listeners/ListenerBase.php#L130" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321474 (owner: Cdentinger) [19:48:01] (PS2) Cdentinger: Throw ListenerSecurityException on broken response. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321474 [20:19:55] (PS3) Cdentinger: Throw ListenerSecurityException on broken response. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321474 [20:21:40] (CR) jenkins-bot: [V: -1] Throw ListenerSecurityException on broken response. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321474 (owner: Cdentinger) [20:22:07] (PS4) Cdentinger: Return 403 when PP sends us a broken message. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321474 [20:23:49] (CR) jenkins-bot: [V: -1] Return 403 when PP sends us a broken message. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321474 (owner: Cdentinger) [20:27:01] (PS5) Cdentinger: Return 403 when PP sends us a broken message. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321474 [21:47:51] ejegg: hiya! [21:48:19] i buried my own question under sedimentary patchsets [21:49:31] the last one should pretty much address the issue, however it does repeat some stuff from RestListener [21:59:54] ok, cool, taking a look now [22:00:18] just done with my own K4 1:1 [22:00:24] (PS1) Eileen: Allow selection of primary when merging. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/321533 (https://phabricator.wikimedia.org/T146941) [22:03:50] (CR) Ejegg: [C: 2] "'Notice' or 'warning' might be more appropriate than 'debug', but this'll do what we need right now. Thanks!" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321474 (owner: Cdentinger) [22:04:17] ty ejegg ! [22:04:29] i'll deploy that once it merges [22:04:44] cwd oops, I guess I meant RestListener did the catching [22:04:56] specifically catching the securityException [22:05:00] ejegg: about logging the ID, which i think we should, it's hard because it's stored in different places for different docs [22:05:20] so i was going to generalize the stuff in the job class that parses it [22:05:28] i didn't want to just store the whole thing cause massive PII [22:05:29] (Merged) jenkins-bot: Return 403 when PP sends us a broken message. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321474 (owner: Cdentinger) [22:05:35] Ah, true. We could log the raw values immediately on receipt, like with the Amazon listener [22:05:54] rather than waiting till we drop the message in the job queue [22:06:42] that's not a bad idea [22:06:47] somewhere they won't end up in emails [22:10:32] (PS1) Cdentinger: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321545 [22:11:32] (CR) jenkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321545 (owner: Cdentinger) [22:11:44] * cwd grumble [22:12:55] Error Output: PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/20-xhprof.ini on line 2 in Unknown on line 0 [22:14:56] oho [22:16:01] i don't see any of those... [22:17:23] oh jeez, not even a file name? [22:17:50] unless... the # is in xhprof.ini? [22:18:03] ya...i'll check [22:20:11] oh hey, I've got to re-run a paypal audit file, which reminds me I've got a fix in for detecting which refunds are already in Civi: https://gerrit.wikimedia.org/r/320938 [22:22:20] no idea how to get to the new trusty CI hosts [22:22:33] this failmail sure is one of Those Problems [22:22:48] sharks.xkcd [22:27:18] ejegg: well there are massive amounts of # comments in smashpig [22:27:28] grr [22:27:56] so we should fix that, but for now, I guess force it? [22:28:25] I'll see if my IDE can do a mass fix [22:28:44] a lot of them are in not php files, maybe there aren't too many [22:28:54] i don't get why this is just an issue now [22:31:48] somebody musta just updated the CI test [22:32:39] oh yeah, I see exactly 5 [22:32:43] patch incoming [22:33:52] nice [22:34:07] yeah almost nothing outside of tests [22:34:12] just my sarcastic comment [22:34:52] is that PP message class even used? [22:35:16] nm ignore me [22:35:19] (PS1) Ejegg: Get rid of # comments [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321552 [22:35:19] it is used [22:36:46] (PS1) Cdentinger: Couple more # comments [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321553 [22:36:55] ejegg: plopped that on top of yours ^ [22:37:19] oops, forgot about things not at the start of lines [22:38:00] (CR) Ejegg: [C: 2] Couple more # comments [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321553 (owner: Cdentinger) [22:38:25] (CR) Cdentinger: [C: 2] Get rid of # comments [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321552 (owner: Ejegg) [22:38:43] oh, I wonder if we'll have to squash those to get our V+2 [22:38:55] turns out # is a pretty rare character in php outside of comments and urls [22:39:30] (Merged) jenkins-bot: Get rid of # comments [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321552 (owner: Ejegg) [22:39:38] nice [22:39:49] ejegg: i think it will be ok, the puking test only runs on deployment i guess [22:39:54] huh [22:39:56] so i'll just rebase [22:40:29] ah yeah it's a 5.5 test [22:40:41] might be we should tweak the CI setup [22:40:46] (Merged) jenkins-bot: Couple more # comments [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/321553 (owner: Cdentinger) [22:42:50] (PS1) Cdentinger: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321554 [22:43:14] oh yeah, might have to squash those deployment commits i guess [22:43:34] one big merge ought to do it [22:44:05] good call [22:45:17] (PS1) Cdentinger: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321555 [22:45:37] (Abandoned) Cdentinger: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321545 (owner: Cdentinger) [22:45:41] (Abandoned) Cdentinger: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321554 (owner: Cdentinger) [22:46:51] (CR) jenkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321555 (owner: Cdentinger) [22:46:55] grr [22:47:47] ok, force-merge time [22:48:31] ugh [22:48:34] same shit [22:49:14] asking in -releng [22:50:28] sorry but I got a dumb question. I know I miss-typed something in a sql query. How do I stop the query? [22:51:43] dstrine: control+c [22:52:07] ah cool, Thanks!! [22:52:50] (PS2) Cdentinger: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321555 [22:53:53] (CR) jenkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321555 (owner: Cdentinger) [23:01:10] ejegg: doesn't it seem weird that no other paypal users are complaining about this? [23:02:52] cwd maybe nobody verifies IPNs? [23:03:03] or they've all got retry loops [23:03:06] that's possible [23:03:28] (PS1) Paladox: Remove "phpunit" from script section in composer.json [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321559 [23:04:33] (CR) Ejegg: "Can you take out the composer.lock change?" [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321559 (owner: Paladox) [23:05:01] (CR) Paladox: "> Can you take out the composer.lock change?" [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321559 (owner: Paladox) [23:05:54] (PS2) Paladox: Remove "phpunit" from script section in composer.json [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321559 [23:07:30] (CR) Ejegg: [C: 2] Remove "phpunit" from script section in composer.json [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321559 (owner: Paladox) [23:09:22] back in a bit [23:14:49] (Merged) jenkins-bot: Remove "phpunit" from script section in composer.json [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321559 (owner: Paladox) [23:19:15] (CR) Paladox: "recheck" [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321555 (owner: Cdentinger) [23:22:15] (CR) Cdentinger: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321555 (owner: Cdentinger) [23:23:18] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/321555 (owner: Cdentinger) [23:28:03] !log updated smashpig payment listeners from 142e60bfe09982f9f86b7839cbf35d9e0e96b8d7 to a0ae95edd001d51d0cff5c080d0372c14efcf9a9 [23:28:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:30:44] stashbot: y u no longer call me master? [23:31:21] See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help. [23:32:12] * cwd ^f master [23:47:30] seems to have stopped the fail mail [23:49:24] cwd: sweet!!! [23:49:54] the problem is still happening but at least we're not hearing about it :S [23:59:45] plenty of evidence still available in the logs. gotta run!