[01:38:00] Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, Fundraising Sprint Testing on Production, Fundraising-Backlog, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2715208 (AndyRussG) Earlier today @Ejegg and @XenoRyet deployed the M... [01:56:29] (PS1) Cdentinger: Add test for retry exception [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315897 [02:12:27] (CR) Cdentinger: [C: 1] "Using an exception gives me the willies but I can't think of a better idea. It is halting a process (not a literal process because dequeue" (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315858 (https://phabricator.wikimedia.org/T147378) (owner: Ejegg) [02:31:16] Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, Fundraising Sprint Testing on Production, Fundraising-Backlog, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2715263 (RobLa-WMF) In order to understand if rMWaa5be0c1e03d is a sa... [04:26:40] Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, Fundraising Sprint Testing on Production, Fundraising-Backlog, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2715343 (AndyRussG) Still not able to reproduce on the beta cluster a... [07:59:51] (CR) Ejegg: Retry capture later when donor details are missing (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315858 (https://phabricator.wikimedia.org/T147378) (owner: Ejegg) [13:04:14] (Abandoned) Hashar: Add "composer test" entry point [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/195375 (owner: Awight) [16:03:38] Fundraising Sprint Stirring The Pot, Fundraising Sprint Testing on Production, Fundraising-Backlog, FR-PayPal-ExpressCheckout, FR-Paypal: Renew PayPal API certificate. Due date: Aug 1st - https://phabricator.wikimedia.org/T136742#2346343 (cwdent) Open>Resolved New cert is in place [16:23:34] (CR) Ejegg: [C: -1] "Thanks! I think the @expectedException notation would be the way to go here" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315897 (owner: Cdentinger) [16:34:20] (PS2) Cdentinger: Add test for retry exception [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315897 [17:00:56] fr-tech: No man is an Iland, intire of it selfe; every man is a peece of the [17:00:56] Continent, a part of the maine; if a Clod bee washed away by the Sea, [17:00:56] Europe is the lesse, as well as if a Promontorie were, as well as if [17:00:56] a Mannor of thy friends or of thine owne were; any mans death diminishes [17:00:57] me, because I am involved in Mankinde; And therefore never send to know [17:00:57] for whom the bell tolls; It tolls for thee. [17:00:57] -- John Donne, "No Man is an Iland" [17:00:58] -- discuss. [17:21:12] (CR) Cdentinger: [C: 2] Retry capture later when donor details are missing [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315858 (https://phabricator.wikimedia.org/T147378) (owner: Ejegg) [17:22:06] ejegg: if you wanna push that test through we can try this thing [17:22:21] (Merged) jenkins-bot: Retry capture later when donor details are missing [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315858 (https://phabricator.wikimedia.org/T147378) (owner: Ejegg) [17:22:40] oh nm i had the dependencies backwards [17:24:49] cool, I'll deploy in a sec [17:27:10] oh i don't mind doing it [17:27:21] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/315976 [17:27:31] this is just for standalone job runner right? [17:27:32] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/315976 (owner: Ejegg) [17:31:46] (CR) Ejegg: [V: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/315976 (owner: Ejegg) [17:34:08] !log updated SmashPig from 3c3d115d8388515eaf1f88b4263a56985454f3a1 to 2306010284ae2452a6a81f61fa44fa35fcc6a42f [17:34:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:34:28] cwd yeah, only the runner is affected [17:40:10] well, normal donation flow works. I'll try pausing the pending consumer and see if that does the right thing [17:40:34] ah good idea [17:42:10] !log disabled pending queue consumer [17:42:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:43:13] i sure put my yubikey in the sd card slot a lot [17:43:57] Heh, this Asus solves that by not having any SD slots [17:44:25] Of course, if I want my mouse and some wired internet at the same time... [17:46:07] ok, looks like it got send to damaged with a retry date [17:46:15] !log enabled pending queue consumer [17:46:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:47:19] i saw that, didn't see it on the db replica but maybe there's some lag [17:47:24] in the damaged table i mean [17:49:38] let's see [17:51:35] ah, it got re-processed already [17:52:33] sweet [17:53:18] weird, I don't see anything in the requeue job log about jobs-adyen [17:55:16] oh wait, it's no re-processed yet [17:55:32] sorry, got mixed up with my previous test donation [17:58:22] cwd dang, ok, now I have no idea where that message went [17:58:46] I see the log message saying it's going to file it for re-queue [17:59:42] aha, now I see it getting re-queued [18:00:42] and... captured [18:04:35] it just never made it to the damaged replica? [18:16:09] cwd I'm really not sure about that [18:30:00] Fundraising-Backlog, FR-Adyen, FR-Smashpig: Adyen audits not downloading - https://phabricator.wikimedia.org/T148216#2717587 (Ejegg) [18:30:27] relocating, back shortly [19:10:08] Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, Fundraising Sprint Testing on Production, Fundraising-Backlog, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2717891 (AndyRussG) Is this also a bug? In `MessageCache::replace()`,... [19:19:53] Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, Fundraising Sprint Testing on Production, Fundraising-Backlog, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2717936 (AndyRussG) Here is a general theory: - There is a bug in `M... [19:48:06] Fundraising Sprint Testing on Production, Fundraising-Backlog, FR-Adyen, FR-Smashpig, Unplanned-Sprint-Work: Adyen audits not downloading - https://phabricator.wikimedia.org/T148216#2717981 (Ejegg) [20:12:04] (CR) Ejegg: [C: 2] Add test for retry exception [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315897 (owner: Cdentinger) [20:13:07] ejegg: ^ pretty neat feature with the @expectedException, though i kind of hate that they use comments for that [20:13:10] (Merged) jenkins-bot: Add test for retry exception [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315897 (owner: Cdentinger) [20:13:43] cwd yeah, kind of odd. consistent with their dataprovider stuff tho [20:13:58] yeah. must be from the java world. [20:14:17] seems like a cardinal sin of programming to have comments be parsed [20:14:38] yarp [20:16:41] MBeat: we got the adyen fix out. spatton is in a meeting but will be free in about 45 minutes. I'd like to start the test as soon as he is free [20:16:53] cool! [20:37:23] dstrine, how long a test are we aiming for? [20:37:47] I was thinking the usual hours [20:37:50] hour [20:37:55] singular [20:38:17] but it's pretty late in france. I'm not sure how many donations we'll get [20:38:20] great. if we get super low traffic, I’m fine w/extending if need be [20:38:57] ok thanks. I know it's a bit late for you. let's just keep talking through it here [20:42:58] MBeat: if we don't see any red flags right way maybe you don't need to stay on super late? [20:43:16] cool, ty [20:43:56] happily ejegg’s two tests today have been sent for settle already [20:44:31] good stuff [20:48:52] (PS1) Ejegg: Fix Adyen audit downloads [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315999 (https://phabricator.wikimedia.org/T148216) [20:49:34] cwd: CR? ^^ [20:52:09] sure [20:52:51] Fundraising Sprint Testing on Production, Fundraising-Backlog, FR-Amazon, FR-Smashpig: Amazon donation order ids are being set incorrectly by the listener - https://phabricator.wikimedia.org/T147973#2710746 (Ejegg) p:Triage>High a:Ejegg [20:53:11] thanks! [21:10:10] (CR) Cdentinger: [C: 2] Fix Adyen audit downloads [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315999 (https://phabricator.wikimedia.org/T148216) (owner: Ejegg) [21:11:10] thanks cwd. Any objection to another deploy, since it's again pretty limited scope? [21:11:15] (Merged) jenkins-bot: Fix Adyen audit downloads [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315999 (https://phabricator.wikimedia.org/T148216) (owner: Ejegg) [21:14:18] no objection here [21:14:32] can't get any more broken than it is! [21:14:40] well i shouldn't say things like that [21:16:18] * dstrine glares at cwd [21:16:26] :X [21:32:14] dstrine: hey! Here now. Getting the campaign configured [21:32:36] MBeat: ejegg XenoRyet cwd ^^^ [21:32:39] cool spatton [21:32:44] 10-4 [21:32:48] got it [21:32:49] thanks for helping!! [21:33:55] cool, ty for the heads up [21:43:20] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/316009 [21:43:48] (PS2) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/316009 [21:44:01] (CR) Ejegg: [C: 2 V: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/316009 (owner: Ejegg) [21:44:12] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/316009 (owner: Ejegg) [21:44:52] spatton: have you started the test? [21:46:44] Have not; enabling it to go live at 2:50pm pst / 21:50 UTC, in 4 minutes [21:46:55] dstrine: check above ^ [21:47:24] oh ok thanks [21:47:26] !log updated SmashPig from 2306010284ae2452a6a81f61fa44fa35fcc6a42f to 961fc4c14f94181a0e364615a0a05dd8f4646912 [21:47:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [21:47:40] MBeat: how you doing on time? I know it's a tad late [21:48:01] no worries, I can monitor console dstrine [21:48:05] ok cool [21:48:28] MBeat: thanks a ton! I bet the crickets have cranked up by now [21:48:44] hah, there are still some diehards singing [21:48:53] these days [21:48:56] yay Friday EOD test... feels like the holidays... [21:49:22] Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, Fundraising Sprint Testing on Production, Fundraising-Backlog, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2718353 (AndyRussG) Just a note about the underlying `ObjectCache` im... [21:51:13] Test is live! [21:52:01] ty spatton [21:54:33] thanks spatton ! [21:59:40] I've got 26 clicks so far, no donations that I can see yet. [22:00:11] yeah it's super late [22:02:55] Fundraising Sprint Testing on Production, Fundraising-Backlog, FR-Amazon, FR-Smashpig: Amazon donation order ids are being set incorrectly by the listener - https://phabricator.wikimedia.org/T147973#2718377 (Ejegg) Everything looks OK from our side, sent an email to our contacts there. [22:08:06] Have the first handful in dstrine and MBeat, check the email thread for the link to the results sheet if it's useful [22:08:34] cool, thanks spatton [22:18:02] Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, Fundraising Sprint Testing on Production, Fundraising-Backlog, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2615311 (GWicke) @aaron knows that code quite well. It might be worth... [22:27:19] MBeat: do you see any issues on your end? [22:27:49] no, things look good - they’re settling nicely and nothing’s gotten stopped at Auth yet [22:29:55] Nice [22:30:13] do we think a 1-hr test is sufficient? It'd conclude in 20 minutes, if so. [22:31:38] MBeat: do you know a rough percentage of users affected by that bug? [22:32:01] from 10/4? I think it was 3.5% [22:32:19] dstrine: [22:33:23] spatton: and MBeat if we pass 100 donations in cc and we don't see anything weird that might be a good enough test right? [22:34:35] what I’m hoping is that a few stay at Auth status so I can check to see that their scores merit that [22:36:41] ok [22:38:08] I wouldn't mind leaving the test running a while longer to get the total donation count more comparable to recent tests. But the time of day will factor in to behavior, anyway. [22:38:30] We'll be right around 100 donations in the next 10 mins [22:38:35] cc donations, I mean. [22:45:41] we're at 106. Mbeat do you have any transactions worth watching? [22:46:21] none stoped at auth, but that just might be how it rolls out today/tonight > I’m fine with not staying live until one shows up dstrine [22:47:54] spatton: and MBeat I'm inclined to stop at 1 hour. we can always do another test on monday if people want [22:47:54] I'm not seeing anything that would have had the problem before - There's a 'retry later' step that would be logged when we hit the problem situation. [22:48:06] ooh ^^ [22:48:07] ok [22:48:16] But that log showed up right when I artificially introduced the problem earlier today [22:48:16] cool [22:48:22] Nice [22:48:47] and I got my ty email, just a bit later than I would have without the problem [22:48:59] Aight dstrine and mbeat and ejegg, I will let this expire then? Runs out in 2 mins [22:49:09] seems fine to me spatton [22:49:13] I think so that's cool [22:49:17] thank you! [22:49:49] Thanks for working late MBeat! [22:49:59] np [22:51:07] Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, Fundraising Sprint Testing on Production, Fundraising-Backlog, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2615311 (Ejegg) a:AndyRussG [22:55:05] spatton: would you like the honors of updating the thread or should I? [22:55:15] Please do :) [22:55:25] ok [23:05:20] Fundraising Sprint Testing on Production, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-2016-17-Q2-Bugs: Generic Individuals Import Error - https://phabricator.wikimedia.org/T147521#2718511 (Ejegg) @LeanneS, sorry I took so long to look at this! @awight changed the required field name fo... [23:13:51] (PS1) Ejegg: More complex import validation [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/316021 (https://phabricator.wikimedia.org/T147521) [23:14:23] (CR) jenkins-bot: [V: -1] More complex import validation [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/316021 (https://phabricator.wikimedia.org/T147521) (owner: Ejegg) [23:15:42] (PS1) Ejegg: Fix composer.lock mismatch with deployment [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/316022 [23:16:36] (CR) Ejegg: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/316021 (https://phabricator.wikimedia.org/T147521) (owner: Ejegg) [23:19:46] ejegg: Is that damaged UI bug still up for grabs? I have a few minutes... [23:20:24] all yours if you want it awight [23:20:38] ejegg: IMO this is the next step, 315044 master Revert "Pager glue for the damaged queue" [23:20:52] I'll prepare something to follow after that [23:22:25] ah, ok [23:22:48] I like having the pager in prod more than bulk ops, but we can wait for the replacement to deploy [23:22:57] Repeal and Replace! [23:23:39] ejegg: oh huh--I was making assumptions about what would be more useful to MBeat.. lemme just ask [23:24:09] well, also, the bulk re-queue is still not implemented [23:24:51] ehh, we can always re-pick the patch. I'll merge that revert [23:25:10] (CR) Ejegg: [C: 2] Revert "Pager glue for the damaged queue" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/315044 (https://phabricator.wikimedia.org/T142058) (owner: Awight) [23:25:15] ok cool, I won't bother MBeat after hours then [23:26:04] Fundraising Sprint Testing on Production, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-2016-17-Q2-Bugs, Patch-For-Review: Generic Individuals Import Error - https://phabricator.wikimedia.org/T147521#2718537 (Ejegg) a:Ejegg [23:26:17] I'm going to run. Have a good weekend everyone! [23:26:52] see you dstrine [23:27:06] (PS1) Awight: Remove remaining pager support, until we can implement correctly [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/316027 (https://phabricator.wikimedia.org/T142058) [23:27:06] dstrine: bye! [23:27:11] ejegg: ^ e-z [23:27:38] * awight lets the chainsaw idle back down again [23:29:13] (CR) Ejegg: [C: 2] Remove remaining pager support, until we can implement correctly [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/316027 (https://phabricator.wikimedia.org/T142058) (owner: Awight) [23:30:19] (Merged) jenkins-bot: Revert "Pager glue for the damaged queue" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/315044 (https://phabricator.wikimedia.org/T142058) (owner: Awight) [23:30:40] Well, I'm going to flee for now. Have a good weekend awight [23:31:43] (Merged) jenkins-bot: Remove remaining pager support, until we can implement correctly [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/316027 (https://phabricator.wikimedia.org/T142058) (owner: Awight) [23:38:45] good idea! [23:43:08] (PS5) Awight: Regression test for T143903 [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314509 (https://phabricator.wikimedia.org/T143903) [23:43:27] (PS8) Awight: Tests for the bulk export [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315638 (https://phabricator.wikimedia.org/T131770) [23:47:13] (PS4) Awight: WIP fix py tests [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315741 (https://phabricator.wikimedia.org/T145012) [23:54:27] Bye dstrine and ejegg