[09:32:57] fundraising-tech-ops, Operations, ops-eqiad: rack/setup beryllium replacement frauth1001 - https://phabricator.wikimedia.org/T143902#2647745 (faidon) [09:33:00] fundraising-tech-ops, Operations, ops-eqiad: Rack/Setup pay-lvs1003 and pay-lvs1004 - https://phabricator.wikimedia.org/T143900#2647746 (faidon) [15:15:13] (CR) AndyRussG: "Fantastic work!! Nice methodology too!!! :)" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/310457 (https://phabricator.wikimedia.org/T145591) (owner: Ejegg) [15:56:55] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Receipt/Thank you letters from Civi - Fix template? - https://phabricator.wikimedia.org/T137496#2648745 (MBeat33) related: {T108088} & {T115662} [16:25:16] ejegg|afk: considering the adyen bugs from thursday, can fr-online run any other tests? [16:29:33] Fundraising-Backlog, fundraising-tech-ops: Give CPS members access to civi - https://phabricator.wikimedia.org/T146051#2648955 (DStrine) [16:46:46] Fundraising-Backlog: Estimate the proportion of people who send us complaints vs are affected by an issue - https://phabricator.wikimedia.org/T120722#2649016 (MBeat33) For {T145776} of the 261 affected donors, only 2 (.76%) have so far contacted us via Zendesk to confirm their donations. [16:55:38] dstrine: I'm pretty sure they can now. Looking at the pending queues / dbs [16:56:22] ok was it just related to adyen? [16:56:28] is adyen ok now? [16:57:40] dstrine: It delayed Astropay and Amazon donations getting to Civi, but Adyen is the only one that needs the pending donor data to actually clear the transaction [16:58:25] ejegg: so would it be ok to schedule another adyen france test? [16:58:29] yep! [16:58:41] errr.... wait [16:59:02] sorry, those pending db entries are still pretty stale :( [16:59:09] what the heck... [16:59:32] I may have misunderstood awight's mail, let's see what version of the code is live [17:02:02] ejegg: thanks for looking into this! [17:02:54] AndyRussG: are you free for a meeting in 30min? [17:03:09] aha, I guess he rolled back the thing that will get Adyen happy again, but I'm pretty sure it just needs a matching settings update. [17:03:25] Give me 10 minutes and I'll know [17:04:17] ok [17:08:45] dstrine: yea sure! [17:08:52] ok thanks [17:09:40] likewise thx for the headsup :) [17:15:24] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/311464 [17:19:19] fr-tech and MBeat - I'm about to deploy a DonationInterface patch that triggered a couple of error message last Thursday, but I think I've got the right settings change to match this time. So it /shouldn't/ break again, hopefully [17:19:35] ty for headsup ejegg [17:23:09] (CR) Ejegg: [C: 2] Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/311464 (owner: Ejegg) [17:27:12] (CR) Daniel Kinzler: [C: 1] "Code looks good. Did not test." [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/310940 (https://phabricator.wikimedia.org/T145738) (owner: Reedy) [17:30:07] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/311464 (owner: Ejegg) [17:39:04] !log updated payments-wiki from 392d67520d14998d61823755b22df50ab45afb35 to 4cd18776ca67fe0db04b4173cb47f39dda59f43b [17:39:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:39:30] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, Unplanned-Sprint-Work: Ingenico audit is broken by queue trickery - https://phabricator.wikimedia.org/T145852#2649339 (awight) Looks like @Ejegg fixed this already... [17:40:28] Fundraising Sprint Qwerty Thwacking, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: CentralNotice banner sequence: implement feature for MVP - https://phabricator.wikimedia.org/T144453#2649342 (DStrine) [17:40:56] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2649344 (DStrine) [17:41:12] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, Browser-Tests-Infrastructure, MediaWiki-extensions-CentralNotice, User-zeljkofilipin: CentralNotice: Intermittent unexplained browser test failures - https://phabricator.wikimedia.org/T145718#2649345 (DStrine) [17:41:32] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Impressions slow to update on jaJP dsk campaign - https://phabricator.wikimedia.org/T145446#2649347 (DStrine) [17:41:54] (PS1) Awight: One more Configuration call needing staticization [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/311469 [17:42:10] (CR) Awight: [C: 2] One more Configuration call needing staticization [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/311469 (owner: Awight) [17:42:17] (Merged) jenkins-bot: One more Configuration call needing staticization [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/311469 (owner: Awight) [17:44:12] dstrine: Looks like the deploy was OK this time, checking to see that Adyen pending entries made it to db [17:45:19] yep! Checking the adyen capture jobs [17:45:24] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, MediaWiki-Logging: Some wmf_civicrm exceptions should be logged at info and not error - https://phabricator.wikimedia.org/T146063#2649382 (awight) [17:45:44] oh, it's disabled - enabling [17:46:22] fr-tech: there may be some failmails from this adyen job, from payments captured before we restored the pending queue [17:46:40] !log update civicrm from 1df25962834885121bd7a9ba856c0c69c2b9cfda to c9157ba6aac3ad3c2fd61c7bb0475851fb8ec421 [17:46:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:47:16] thanks for catching that awight [17:47:32] !log reenable banner history queue consumer [17:47:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:48:13] oh boy, new kernel! rebooting... [17:48:24] ejegg|brb: whew, I found a clutch of bugs in the new orphan rectifier too... maybe take a peek and call out if you think anything is UBN [17:50:14] UBN? [17:50:18] oh, unbreak now [17:50:28] k, will do shortly [17:51:35] (PS1) Awight: Remove APC Configuration caching [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/311472 [17:52:14] (PS4) Awight: FIXME [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/305580 (owner: Ejegg) [17:52:23] (CR) Awight: FIXME [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/305580 (owner: Ejegg) [17:52:28] (CR) Awight: [C: 2] FIXME [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/305580 (owner: Ejegg) [17:53:44] awight: T145848 looks pretty high priority [17:53:46] T145848: Orphan rectifier not sending to payments-initial nor payments-fraud queues - https://phabricator.wikimedia.org/T145848 [17:55:11] Fundraising Tech Backlog, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Technical-Debt: Simplify the orphan slayer - https://phabricator.wikimedia.org/T95523#1193069 (Ejegg) Simplified to your taste now @awight? [17:56:16] ejegg: True, but I think that's always been the case. zgrep payments1004 payments-initial*201609* | wc -l -> 0 [17:56:21] (Merged) jenkins-bot: FIXME [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/305580 (owner: Ejegg) [17:56:37] ooh [17:56:58] Fundraising Tech Backlog, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Technical-Debt: Simplify the orphan slayer - https://phabricator.wikimedia.org/T95523#2649479 (awight) Open>Resolved a:awight :D Now, for Round 2... [17:57:19] Leaving at "High"... [17:57:48] awight: I merged your CVV equality test if you want to knock out T145846 [17:57:49] T145846: New orphan rectifier balks on CVV="0" - https://phabricator.wikimedia.org/T145846 [17:58:03] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, FR-ActiveMQ, MediaWiki-extensions-DonationInterface: Orphan rectifier not sending to payments-initial nor payments-fraud queues - https://phabricator.wikimedia.org/T145848#2649488 (awight) For context, the legacy rectifier didn't send t... [17:59:01] MBeat: We just noticed that the orphan rectifier has never written to payments-initial nor payments-fraud, which might account for some of the missing records you've been seeing. [17:59:27] great, thank you awight [18:03:36] (CR) Awight: [C: 2] Mirror messages to php-queue backend (2 comments) [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311070 (https://phabricator.wikimedia.org/T145419) (owner: Ejegg) [18:05:42] hmm, we ran the capture, but still no 'record success' job for that adyen test donation. [18:05:54] lessee what the IPN logs say [18:06:47] (CR) Awight: [C: 2] "Deployment: requires $wgFundraisingEmailUnsubscribeQueueParameters" (1 comment) [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311165 (https://phabricator.wikimedia.org/T145419) (owner: Ejegg) [18:07:02] (Merged) jenkins-bot: Mirror messages to php-queue backend [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311070 (https://phabricator.wikimedia.org/T145419) (owner: Ejegg) [18:07:11] k, looks like not a problem on our side at least - we just haven't gotten the capture success IPN notification from Adyen yet [18:07:51] (Merged) jenkins-bot: Only send unsubscribe message via PHP-Queue [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311165 (https://phabricator.wikimedia.org/T145419) (owner: Ejegg) [18:09:02] thanks for the CR awight ! [18:10:36] of curse! [18:11:18] (CR) Awight: [C: 2] "Fantastic!" (3 comments) [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311174 (https://phabricator.wikimedia.org/T145419) (owner: Ejegg) [18:12:00] Fundraising Sprint Pretending This Isn't Happening, Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, and 5 others: Migrate donations to new queue - https://phabricator.wikimedia.org/T131277#2649570 (awight) Looks like this is pending deployment now? [18:13:14] Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, Patch-For-Review, Spike: Spike: Zendesk migration from Web Portal to Helpdesk - https://phabricator.wikimedia.org/T143142#2649578 (awight) @MBeat33 Were you planning to throw the switch today? [18:13:48] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, fundraising-tech-ops, FR-ActiveMQ, Unplanned-Sprint-Work: fundraising activemq queue/memory bloat - https://phabricator.wikimedia.org/T145634#2636545 (awight) Open>Resolved [18:15:59] (PS1) Awight: Merge master into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/311477 [18:16:08] (CR) Awight: [C: 2] Merge master into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/311477 (owner: Awight) [18:17:11] (PS1) Awight: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/311478 [18:17:20] (CR) Awight: [C: 2] Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/311478 (owner: Awight) [18:17:36] (Merged) jenkins-bot: Add tests for thank you unsubscribe [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311174 (https://phabricator.wikimedia.org/T145419) (owner: Ejegg) [18:17:41] (Merged) jenkins-bot: Merge master into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/311477 (owner: Awight) [18:17:53] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Not able to save activity edits - https://phabricator.wikimedia.org/T137740#2649627 (CaitVirtue) For further context, a bunch of reminders are effectively broken at this time. Some of these reminders are for things like "submit grant report by X date"... [18:25:34] fr-tech: Heads-up, I need to run to a meeting but left a small fix merging on the paymentswiki deploy branch. (https://gerrit.wikimedia.org/r/#/c/311477/ and https://gerrit.wikimedia.org/r/311478 ) [18:25:56] It's not urgent, but I wanted to advertise that the deployment branch is ahead of production. [18:27:38] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/311478 (owner: Awight) [18:35:09] dstrine: so, we're not seeing the same kind of errors with Adyen as before, but my test txn is taking an awful long time to get from authorized to captured. The logs definitely show us making the capture API call (we didn't even get that far with payments on Thursday) and getting a successful status and a capture reference back, but the payment is still at 'Authorized' in the console and we haven't gotten an IPN message about the capture succee [18:35:48] I'm going to wait a little bit longer to see if that clears up before emailing Adyen [18:36:26] oh ok so it could be and adyen thing that just coincided with queue bugs? [18:37:15] i dunno. the queue bug was enough on its own to cause all the stuff last week. [18:37:40] But now that its fixed, I can't see anything with our code that would be causing this delay [18:41:37] ok thanks ejegg ! ppena_ and the-wub we may need to report some bugs to adyen. We don't have any news just yet on when we can test france again. [19:02:10] migrating... [19:34:16] (PS2) Eileen: Only store the full country code if it is on our valid list. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310732 (https://phabricator.wikimedia.org/T96410) [19:35:06] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Issues manually merging records - https://phabricator.wikimedia.org/T144440#2599873 (Eileenmcnaughton) a:Eileenmcnaughton [19:36:18] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Receipt/Thank you letters from Civi - Fix template? - https://phabricator.wikimedia.org/T137496#2369861 (Eileenmcnaughton) a:Eileenmcnaughton [19:40:44] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, fundraising-tech-ops, FR-ActiveMQ, Unplanned-Sprint-Work: fundraising activemq queue/memory bloat - https://phabricator.wikimedia.org/T145634#2649994 (Jgreen) @awight can we dump the queued messages now? activemq is still clinging t... [19:42:51] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, fundraising-tech-ops, FR-ActiveMQ, Unplanned-Sprint-Work: fundraising activemq queue/memory bloat - https://phabricator.wikimedia.org/T145634#2650008 (Ejegg) We can dump pending, banner history, antifraud, and payment initial. [19:50:27] (PS1) Eileen: Reply on CiviCRM address rather than contribution tracking language. [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/311491 (https://phabricator.wikimedia.org/T96410) [19:51:44] Fundraising Sprint Octopus Untangling, Fundraising Sprint Pretending This Isn't Happening, Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, and 3 others: Silverpop export not capturing language changes in Civi (not sure if... - https://phabricator.wikimedia.org/T96410#2650065 [20:42:35] (PS1) Eileen: Update English languages to only keep a small number of recognised variants [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/311503 (https://phabricator.wikimedia.org/T96410) [20:43:25] (PS2) Eileen: Rely on CiviCRM address rather than contribution tracking language. [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/311491 (https://phabricator.wikimedia.org/T96410) [21:23:18] (PS1) Eileen: Further fix on address merging to support form override [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/311597 (https://phabricator.wikimedia.org/T144440) [21:25:04] (CR) jenkins-bot: [V: -1] Further fix on address merging to support form override [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/311597 (https://phabricator.wikimedia.org/T144440) (owner: Eileen) [21:26:40] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/311597 (https://phabricator.wikimedia.org/T144440) (owner: Eileen) [21:29:57] (CR) Ejegg: "Good call on the truncation! I can do that in a follow-on patch if you want to merge this for CI's sake." [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/310457 (https://phabricator.wikimedia.org/T145591) (owner: Ejegg) [21:37:35] omg. [21:37:41] I missed the meeting [21:50:54] back in a couple hours... [21:51:25] this bit of pending cleanup is ready now that the orphan slayer is switched: https://gerrit.wikimedia.org/r/301630 [21:52:12] (PS15) Awight: Quit deleting from pending queue, stop saying limbo [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/301630 (https://phabricator.wikimedia.org/T133433) (owner: Ejegg) [21:52:20] ejegg|afk: great, I'll take a look [21:55:29] Fundraising-Backlog: Customized Communications Preference field is not working correctly in Civi - https://phabricator.wikimedia.org/T146096#2650601 (RLewis) [21:59:44] (CR) Awight: [C: 2] "Good thing to clean up, thanks!" (5 comments) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/301630 (https://phabricator.wikimedia.org/T133433) (owner: Ejegg) [22:00:31] (CR) Awight: Quit deleting from pending queue, stop saying limbo (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/301630 (https://phabricator.wikimedia.org/T133433) (owner: Ejegg) [22:02:08] !log update paymentswiki from 4cd18776ca67fe0db04b4173cb47f39dda59f43b to d572ee9e3ef0a97c044f02e9866469a8f3fa5858 [22:02:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:04:04] (PS10) Awight: Move refund queue consumer off ActiveMQ [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/309603 (https://phabricator.wikimedia.org/T145234) (owner: XenoRyet) [22:06:06] (Merged) jenkins-bot: Quit deleting from pending queue, stop saying limbo [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/301630 (https://phabricator.wikimedia.org/T133433) (owner: Ejegg) [22:12:11] (CR) Awight: [C: -1] "Great! The only blocker is that the required fields are not actually being tested. A few small cleanup suggestions as well, but it's mer" (7 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/309603 (https://phabricator.wikimedia.org/T145234) (owner: XenoRyet) [22:12:30] (PS3) Awight: Move recurring queue consumer off ActiveMQ [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/309618 (https://phabricator.wikimedia.org/T145233) (owner: Ejegg) [22:13:16] MBeat: Are the status 600s looking well-cared for? [22:13:37] The new orphan rectifier ran over the weekend, and the old rectifier has been turned off now. [22:14:24] (CR) Eileen: "nope - not good enough - need to figure out context" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/311597 (https://phabricator.wikimedia.org/T144440) (owner: Eileen) [22:14:30] (CR) Eileen: [C: -1] Further fix on address merging to support form override [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/311597 (https://phabricator.wikimedia.org/T144440) (owner: Eileen) [22:21:34] (PS2) Eileen: Update English languages to only keep a small number of recognised variants [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/311503 (https://phabricator.wikimedia.org/T96410) [22:27:21] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Long page load approx once per day, maybe more - https://phabricator.wikimedia.org/T118341#1797797 (CaitVirtue) Adding data here bc it makes me feel like I'm providing good info while waiting for the page to load: 9/19 3:26pm PDT Load time: 36 seconds... [22:40:08] (CR) Awight: [C: 2] "Needs a lot of cleanup, but it looks equivalent to the brokenness we already experience at home." (28 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/309618 (https://phabricator.wikimedia.org/T145233) (owner: Ejegg) [22:40:52] (PS2) Awight: Use default test filename for DO_PAYMENT success [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/306802 (owner: Ejegg) [22:41:45] (CR) Awight: [C: -1] "Should be abandoned? This was fixed higher up in test usage, and we really do want to identify specific files for the 3 API responses inv" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/306802 (owner: Ejegg) [22:44:38] fr-tech: Can I review for anyone? [22:58:50] awight: you could re-review the one ejegg|afk looked at but wanted more eyes on [22:59:10] which is this https://gerrit.wikimedia.org/r/#/c/310219/ [22:59:59] eileen: sure, thx! [23:01:55] !log update orphan rectifier config for T145848 [23:01:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:02:09] T145848: Orphan rectifier not sending to payments-initial nor payments-fraud queues - https://phabricator.wikimedia.org/T145848 [23:07:09] (PS2) Eileen: Further fix on address merging to support form override [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/311597 (https://phabricator.wikimedia.org/T144440) [23:20:47] (CR) Awight: "Just questions, and maybe a small typo." (4 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310219 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [23:22:00] gotta run! [23:24:49] Fundraising-Backlog, fundraising-tech-ops: Give CPS members access to civi - https://phabricator.wikimedia.org/T146051#2650920 (DStrine)