[02:09:53] !log updated payments-wiki from 55d0808853 to c81e25f8d3 [02:09:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:21:10] (CR) Jgleeson: "I've not been able to successfully test this locally after running through the test instructions." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/426068 (https://phabricator.wikimedia.org/T1888678) (owner: Ejegg) [15:41:19] hey dstrine, what time do you want to test new Ingenico? Should we wait until ejegg|away is around? [15:42:29] pcoombe: he is off today but others are around. fr-tech would you all be around for a 1 hour test? [15:46:02] Fundraising-Backlog: New Ingenico iframe is slightly cut off at top - https://phabricator.wikimedia.org/T194513#4200581 (Pcoombe) [15:46:08] dstrine, happy to support where I can but slightly reluctant to take responsibility for the full end-to-end process if anything goes wrong. mepps XenoRyet AndyRussG think we can support it between us in ejegg|away absence? [15:46:54] AndyRussG, is also away this afternoon [15:47:30] jgleeson: understood. I think others have this experience. pcoombe: we have standup in 15 minutes. I will discuss with the team [15:53:09] Fundraising-Backlog: New Ingenico form has incorrect CVV instructions for Amex - https://phabricator.wikimedia.org/T194514#4200585 (Pcoombe) [15:58:42] are we testing today? yeah we're around [15:59:42] ejegg|away is off for a while so we definitely shoulnd't wait for him [16:02:02] pcoombe: we're down to monitor MBeat do you have any concerns about a quick test? [16:02:27] this is an early test so we'll shut down at the sign of anything wonky [16:04:57] no concerns, dstrine [16:05:10] having wifi issues but will monitor GC [16:05:17] thanks MBeat pcoombe we are ok to go whenever you feel [16:05:28] (PS8) Gopavasanth: Remove 'UnitTestList' hook [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/428059 (https://phabricator.wikimedia.org/T142120) [16:06:31] Thanks, just checking with Thea. I may not be around for the whole hour. [16:11:53] Okay dstrine MBeat I've set the banners to go up at 1615 UTC (in 4 minutes) [16:12:08] pcoombe Thanks! [16:13:29] (PS9) Gopavasanth: Remove 'UnitTestList' hook [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/428059 (https://phabricator.wikimedia.org/T142120) [16:13:48] The campaign is C1718_enUS_dsk_FR in case anyone needs to take it down [16:14:00] and our monitoring sheet is here: https://docs.google.com/spreadsheets/d/1ggDfFg70UtQjHxy-ddNW_mkRIUX1JGI5zEEB_y-9ngU/edit#gid=1275334304 [16:15:10] 10-4 [16:15:22] I've got the logs open and watching for weirdness. [16:17:20] Cool, we should be up now. (It helps if I remember to actually tick the "enabled" box) [16:21:25] thanks pcoombe !!! [16:26:12] First cc donations have shown up. But I also saw a couple of failmails [16:26:51] I see one error in the log so far, but most seem to be going through ok. [16:29:13] fr-tech I just got a wave of failmail [16:29:21] me too, looking through them [16:29:22] ^ [16:29:29] I think they are ok errors [16:29:36] s/ok/expected [16:29:39] Not authorised. [16:29:44] well they're obviously british [16:29:54] :) [16:30:01] Logs just got a wave of not authorized/card expired type messages. [16:30:49] hmmm I think this might just be the code over-logging the expected errors [16:31:00] expected rejections [16:31:57] Having them in the log is fine I think, but we definitely don't want failmail about that. [16:32:35] I think the failmail is being triggered due to the exception being thrown [16:35:46] I'm seeing another error about order status. Nothing to stop the test over, but we should look into it after. [16:35:55] Other than the failmail, things look okay from my end. Seeing plenty of donations in civi. [16:38:04] Someone just bounced off session velocity [16:41:03] Sidenote, is Pivot for banner activity (https://pivot.wikimedia.org/#banner_activity_minutely/) broken for everyone else or just me? [16:41:46] not loading for me [16:41:48] I can only get "Query error io.druid.java.util.common.ISE: JavaScript is disabled". Other datasets in Pivot seem to work fine though [16:41:50] Getting some errors that are reporting as client side errors now. That'll need looking into. [16:42:11] pcoombe, I see the same [16:42:35] XenoRyet what are you seeing? [16:43:31] Just in the error log, getting lines that are reporting 'Client side error' with an array of what looks like outgoing data. [16:44:18] It's not high enough occurance to cancel the test though, and looks like we have good logging on it. [16:44:36] Just one of the bugs we'll have found. [16:45:12] The other one I'm curious about is saying 'We don't have an order status after doing a GET_ORDERSTATUS.' [16:45:21] I'll make tickets for these in a little bit here. [16:45:57] XenoRyet: Is there a project or tracking task to file bugs against for this new form? I put a couple of minor ones in Phab earlier. [16:47:22] pcoombe: fr-ingenico-integration_2017-18 I think. That's where we've been tracking overall progress anyway. [16:47:38] fr-tech, this is the line which is logging the response error codes and looks like it's triggered the failmail https://github.com/wikimedia/wikimedia-fundraising-SmashPig/blob/master/PaymentProviders/Ingenico/PaymentProvider.php#L155 [16:47:58] should we be using a different logger method for that? [16:48:01] Cool, thanks [16:48:26] I'm not 100% on how we categorise these type of errors [16:49:32] We should look how the other intigrations are doing it. We definitely want it logged I think, but maybe not at the error level. [16:50:19] I was soooo close to Inbox Zero before this test started. Not anymore! :) [16:50:28] ;-) [16:52:08] pcoombe: let the failmail flow through you [16:52:26] makes one feel alive [16:53:23] Failmail is the mind-killer. failmail is the little-death that brings total obliteration. I will face my failmail. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path. Where the failmail has gone there will be nothing. Only I will remain. [16:53:42] :) [16:54:04] XenoRyet: well done spacing guild member [16:54:26] It fit too well [16:54:31] Okay I need to log off shortly. tskaff are you okay to hang around and monitor until 17:15? [16:54:51] XenoRyet, the recurring queue consumer messages are one that need looking into also [16:54:56] that technically shouldn't happen [16:55:00] pcoombe I'm around! My day is just starting :) [16:55:06] Thanks! [16:55:51] jgleeson: Go ahead and make a ticket about it. We've got enough that we'll need tickets for all this stuff. [16:56:03] does anyone know, is the gateway='globalcollect' in fredge for today’s test? or is there a new gateway name? [16:56:04] will do [16:56:13] One thing I did notice before leaving, we have no recurring donations in civi yet despite ~70 attempts. Is that expected, are they on a delay? [16:56:23] MBeat: should be Ingenico, I believe [16:56:28] ty XenoRyet [16:56:54] pcoombe, looks like we've seen a few errors related to recurrings [16:56:59] gonna create a ticket for it [16:57:00] or little i ingenico anyway [16:57:23] I count 5 so far pcoombe [16:57:27] Fundraising-Backlog, Fr-Ingenico-integration_2017-18: Ingenico We don't have an order status after doing a GET_ORDERSTATUS. - https://phabricator.wikimedia.org/T194517#4200693 (XenoRyet) [16:58:03] Fundraising-Backlog, Fr-Ingenico-integration_2017-18: New Ingenico form has incorrect CVV instructions for Amex - https://phabricator.wikimedia.org/T194514#4200703 (Pcoombe) [16:58:12] Fundraising-Backlog, Fr-Ingenico-integration_2017-18: New Ingenico iframe is slightly cut off at top - https://phabricator.wikimedia.org/T194513#4200704 (Pcoombe) [16:58:46] Thanks everyone, catch you later. [16:58:50] Fundraising-Backlog, Fr-Ingenico-integration_2017-18: Ingenico client side error - https://phabricator.wikimedia.org/T194518#4200705 (XenoRyet) [16:59:08] Thanks to you too pcoombe. See you later [16:59:12] Bye Peter! [16:59:18] thanks pcoombe !!! [17:00:46] Fundraising-Backlog, Fr-Ingenico-integration_2017-18: Ingenico Recurring: "Recurring donation but no subscription ID or recurring payment token found" error being thrown during import - https://phabricator.wikimedia.org/T194519#4200719 (jgleeson) [17:01:58] cheers pcoombe [17:02:26] Ingenico console is not searching by donor email address; I pinged Sal fyi [17:03:14] Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising Sprint Junebugs prefer July, and 4 others: ... - https://phabricator.wikimedia.org/T190871#4200734 [17:03:25] so it looks like the correct logger method should be Logger::warning() [17:03:36] I'll push up a patch to try and quiet down the failmail [17:03:47] Sounds good [17:05:38] XenoRyet: 2 donors so far were scared off by “Wells Fargo-Verified by Visa “requirement pop up screens that ask for more PII; are we expecting WF to show up as part of the new integration? [17:06:09] Heh, I ran into that myself when trying to test yesterday. [17:06:39] That said, I'm not sure what the plan is there. It's definitely too clunky and bad UI in its current form. [17:07:15] We should probably make a ticket about that too, just in case it isn't on anyone's radar yet. [17:07:16] so, put it in Phab? [17:07:20] got it! [17:10:05] most fail mails i've seen in a while [17:10:26] Yep, it's a boatload [17:10:39] luckily it's just wrong logger level, so just noise. [17:11:25] is it possible to turn off or fix whatever is hollering? [17:11:32] in-progress cwd [17:11:52] (PS1) Jgleeson: Update expected ingenico failure logging level to prevent excessive failmail [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/432611 [17:11:56] ^^ [17:12:04] my inbox thanks you :) [17:12:20] Test is almost over as well. Should quiet down in a few more minutes [17:12:48] XenoRyet, if you wanna check out https://gerrit.wikimedia.org/r/432611 we can push that out [17:12:54] looking [17:13:10] (CR) XenoRyet: [C: 2] Update expected ingenico failure logging level to prevent excessive failmail [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/432611 (owner: Jgleeson) [17:13:33] (Merged) jenkins-bot: Update expected ingenico failure logging level to prevent excessive failmail [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/432611 (owner: Jgleeson) [17:13:49] Huh, we got a server time-out from Ingenico [17:14:18] Just the one, but that'll be worth looking for next test. [17:16:39] Confirming the test is down [17:17:20] Cool. Good test, lots of good data. [17:17:30] Awesome, thanks all! [17:17:36] You too [17:18:20] great time for my internet to drop off [17:18:32] It's like it knows... [17:19:06] thanks everyone!!!! [17:21:46] I'm stepping out for a second. brb [17:22:35] (PS1) Jgleeson: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/432612 [17:25:55] fr-tech, when we're working with DI and update smashpig, do we pull in changes via composer or do we do something else? I forget [17:26:25] i think composer jgleeson_ [17:26:49] hmm..i know we can also update smashpig directly [17:27:02] yeah my memory is hazy on this [17:27:36] I'm gonna look through my log history to see if I can work it out [17:28:06] composer would mean we'd need to publish the changes to github I think, so packagist could pull them in [17:28:22] unless we have hooks setup in gerrit? anyone remember? [17:31:17] yes for ocmposer you have to add a new version tag jgleeson_ [17:35:08] fr-tech, anyone recently done a release to smashpig? I could use some help reminding me how we deploy it [17:35:58] so far merged into deployment but now looking at packagist, I'm wondering whether I should create a new tag from deployment and push that to be pulled in via composer here https://packagist.org/packages/wikimedia/smash-pig [17:36:40] It's been a while for me, but I don't remember doing those steps last time I did it. [17:37:25] ah [17:37:29] Just f_c_u and rsync_blaster after the deploy branch had merged. [17:37:53] actually [17:38:06] now I'm remembering something about updating the vendor submodule [17:38:21] and we composer update locally prior [17:38:30] That sounds right. [17:42:36] jgleeson_2 you do need to create a new tag [17:42:41] we changed the workflow somewhat recently [17:43:10] ok thanks, from deployment? [17:47:19] mepps, so doin it this way, I'd need to push the tag and update and redeploy payments wiki I think [17:47:28] to pull in the new version of smashpig [17:47:46] with the 1 hour test our the way, is this something I should hold off on until Monday? [17:48:31] I don't think we'll see anymore new ingenico failmail for now [17:51:35] yeah jgleeson_2 this can wait [17:51:43] let's not deploy on a friday anyways [17:51:57] cool. it looks like the tag is of master [17:52:05] which is unusual for us as it includes all the tests [17:52:13] maybe it's so we can actual more like a normal package [17:52:39] +1 not deploying on fridays [17:54:22] Yea, definitely not worth a friday deploy. Though sounds like things have changed since I last did it, so if I'm around when you do deploy, I'd like to watch. [17:58:28] yeah XenoRyet, I'll be going extra slow to make sure I get that right since its the first time using this workflow [17:58:46] so far, I've pushed the tag to gerrit https://gerrit.wikimedia.org/r/#/admin/projects/wikimedia/fundraising/SmashPig,tags [17:59:10] now just waiting for it to show up in packagist, so composer can see it. https://packagist.org/packages/wikimedia/smash-pig [17:59:14] Is the new flow documented anywhere? [18:00:27] it looks like we don't have a repo hook setup in packagist to pull in new tags so I'm guessing ejegg|away manually checks for updates for that packagist smash-pig project using his account? [18:01:02] which might be a blocker mepps if he is away [18:01:16] if we can't login to link up the latest tag, that is [18:02:42] scratch that fr-tech, I appear to have access [18:03:04] just sync up to bring in the latest tag of smashpig, 0.5.6.3 [18:03:17] sync'd * [18:05:36] i can check for updates jgleeson_2 [18:05:54] mepps, I've updated the packagist project [18:06:02] great jgleeson_2 [18:07:40] I'm now just trying to confirm that I need to deploy payments wiki [18:08:12] as the version in DI is ^0.5, which should pull in the newest tag [18:08:21] version of smash-pig [18:10:44] you still have to update composer.lock jgleeson_2 [18:11:24] ahhh good point [18:13:16] mepps, in this instance do we deploy the final changes via payments wiki or DonationInterface (as it's technically DI which will be updated) but that is being pulled in via payments wiki [18:14:52] cwd, Jeff_Green when we run f_c_u and rsync blaster, does that update ALL instances of DI on our servers? [18:15:09] f_c_u -p DonationInteface * [18:15:21] +r [18:15:37] jgleeson: it does unless you tell it otherwise [18:15:55] you can specify where to blast with rsync_blaster args [18:16:14] if you do "rsync_blaster -i" it will blat details on what it syncs, where [18:16:43] i think it's -i anyway? I'm more confident -h is help if I'm wrong about the former [18:20:02] ok, I'm a little confused how we maintain Independence between an rsync blast of payments-wiki and one of DonationInteface. Would the former not overwrite the more granular lattter, with it being a subfolder of the payments wiki [18:21:23] mepps, can you remember whether we rsyncblast only DI or the entire payments wiki project when updating DI within payments wiki? [18:21:42] jgleeson DI doesn't have it's own project as far as i know [18:21:56] to update DI you have to update payments-wiki and then rsync_blast it [18:21:56] it's rsync blaster'able [18:22:01] according to the help output [18:22:16] i would do the whole project [18:22:41] donation interface is deployed a couple random places iirc [18:24:03] yep cwd, I'm a little confused about how hierarchy works with our f_c_u and rsync blaster [18:24:21] do we have an rsync setup from the server to all instances of DI ? [18:24:34] which is maintained somehow when a parent directory is also rsync'd [18:25:05] looking [18:25:09] thanks [18:25:24] fr-tech, I wanna make 100% sure of this before touching payments wiki :) [18:26:21] jgleeson: i think /etc/FrDeploy.conf will clear it up [18:29:06] deploy is of course complicated [18:29:31] it's mostly git except for a few files that need su [18:29:46] which adds a lot of confusion [18:31:04] from that file, it looks like payments-wiki is what should be deployed [18:31:20] although how we test that isn't broke, post-deploy, I then wonder [18:31:42] if you want to deploy payments yep payments-wiki is the magic [18:32:36] looks like a I need a drink this weekend to prepare for the release on Monday then.. [18:33:01] :) [18:33:03] is ejegg out? [18:33:20] yup, pretty much the whole week [18:33:29] next week [18:33:33] ah ha [18:33:39] @_@ [18:33:45] are you trying to deploy a fix? [18:34:04] yeah it's a patch to the ingenico stuff that was causing all that failmail [18:34:42] it's been a while but i can push that out [18:34:42] the change is minor but the release chain is a pain [18:34:48] yeah it is [18:35:42] if all you need is the git/rsync steps it's no biggie [18:35:47] git on frack i mean [18:36:03] yeah I think that's all it is [18:36:52] dstrine, when are we next likely to test the new ingenico stuff? [18:36:54] you should just have to go: [18:36:57] fundraising_code_update -p payments-wiki [18:36:58] rsync_blaster ALL:payments-wiki [18:37:08] once everything is merged [18:37:11] that's the miserable part [18:37:16] the merging [18:37:34] cwd, happy to do that but the scary thing for me is confirming nothing is broken [18:37:46] I guess I could just revert if we get any unlikely failmail swarms [18:38:02] I know the f_c_u and rsync stuff has that built in [18:38:24] yep, revert yep! [18:38:31] you can just do: [18:38:51] fundraising_code_update -p payments-wiki:rollback [18:39:30] jgleeson: https://gerrit.wikimedia.org/r/#/c/432611/ ? [18:39:33] that what's going up? [18:39:46] yep [18:40:13] it looks like we're using tag for pulling smashpig in [18:40:14] jgleeson: usually we look at the bugs that shake out of a test and assess how many should be fixed before another test. Subsequent test should just be to verify that we are actually fixing the critical parts [18:40:40] or we are successfully building new parts correctly [18:41:06] I think there are a few new things that need to be completed before we consider ourselves "campaign ready" [18:41:09] jgleeson after rolling back i say we let this wait until monday (apologies if you've already made that call--i didn't see it in backscroll) [18:41:11] well, the mails seem to have stopped [18:41:19] cwd we stopped the test [18:41:24] yeah [18:41:33] so i vote for waiting too [18:41:35] mepps, yeah this is just prep for monday [18:41:40] definitely not gonna go today [18:41:44] also it's late in liverpool [18:41:47] there was already a rollback? [18:41:57] no [18:42:19] I think mepps means scrollback? [18:42:27] oh heh i get it [18:43:44] it's probably a storm in a teacup as they say over here and should hopefully go without a hiccup. ejegg|away, won't speak to me at the hackathon if I take down payments-wiki during his vacation! [18:44:24] fr-tech no need to stay on past your usual work day or anything. No need to address the bugs we found either. We'll process all this in backlog meeting monday [18:44:59] Yep [18:45:19] jokes aside, I think I have it mapped out so will reconvene Monday when everyone is around and do the release if we're all confident it makes sense and we have a backup in the event of an unlikely issue. [18:45:32] so jgleeson, nothing has been deployed? [18:46:02] if so, go enjoy your weeknd! [18:46:03] no, not yet. So far the fix has been pushed to the repo with the new tag but nothing is using it [18:46:37] I'm going the gym now, so no rush [18:46:45] it's a bit like the dentist... [18:46:54] any excuse [18:47:04] anyway, have a good weekend all. Catch you monday [18:47:13] thanks for all the info and help fr-tech! [18:47:26] have a good weekend [19:42:14] Fundraising-Backlog, fundraising-tech-ops, DC-Ops: Decom tellurium - https://phabricator.wikimedia.org/T194408#4201344 (cwdent) [19:47:24] fundraising-tech-ops: unattended-upgrades on frack - https://phabricator.wikimedia.org/T191907#4201377 (cwdent) Working pretty good except on stretch where it does not work yet [19:47:26] fundraising-tech-ops: unattended-upgrades on frack - https://phabricator.wikimedia.org/T191907#4201379 (cwdent) Working pretty good except on stretch where it does not work yet [19:47:28] fundraising-tech-ops: unattended-upgrades on frack - https://phabricator.wikimedia.org/T191907#4201378 (cwdent) Working pretty good except on stretch where it does not work yet [19:49:41] nice [19:49:45] that is phabricator being broken [20:38:37] fr-tech we got one bug from DS from the test today: https://phabricator.wikimedia.org/T194543 [20:39:28] I can take a look real quick [20:39:33] whoops [20:40:45] XenoRyet: MBeat tells me we can push these through at times. is there any time limit on that? [20:41:26] He'd know better than me. [20:41:31] I don't want to spin up new work on friday afternoon if we don't have to [20:41:33] ok [20:41:56] I can at least spot check the query. If it's more complicated than that we can talk about it on Monday [20:42:29] I see he didn't mark it UBN or anything [20:45:33] dstrine: there’s really no time limit on settling them; once we mistakenly settled something over a year old & really confused a donor [20:45:46] Heh, I bet it did. [20:45:58] the informal guidelin is that we try to settle within two days if they’re legit [20:46:45] gah, got to migrate out of closing library - later mates [20:47:43] ok thanks MBeat and XenoRyet just wanted to make sure we were cool to wait on that