[00:01:01] Fundraising-Backlog, Recurring-Donations: Account Updater via WorldPay - https://phabricator.wikimedia.org/T91860#1416137 (atgo) a:atgo>None [00:03:06] Fundraising Tech Backlog, Fundraising-Backlog: Create Alipay-specific TY page - https://phabricator.wikimedia.org/T88700#1416148 (atgo) Open>Invalid Invalid since we're not using Alipay [00:05:13] Fundraising-Backlog: Fix WP mobile forms - https://phabricator.wikimedia.org/T85583#950225 (atgo) [00:07:22] Fundraising Dash, Fundraising-Backlog: New Dash widget request: Rejections widget - https://phabricator.wikimedia.org/T89528#1416174 (atgo) a:atgo>None [00:07:34] Fundraising-Backlog: Make automated refunds possible in Worldpay - https://phabricator.wikimedia.org/T89046#1026176 (atgo) [00:09:25] Fundraising-Backlog, Epic: [epic] Get Adyen back up - https://phabricator.wikimedia.org/T90748#1416189 (atgo) [00:10:28] Fundraising-Backlog: Parse Amazon last name data differently (or: review the way we receive Amazon name data) - https://phabricator.wikimedia.org/T86720#1416194 (atgo) [00:10:31] Fundraising-Backlog, Epic: [epic] Amazon upgrade - https://phabricator.wikimedia.org/T87625#1416193 (atgo) [00:14:07] Fundraising-Backlog: Leave blank values in Silverpop file blank, not NULL - https://phabricator.wikimedia.org/T91692#1416200 (atgo) a:atgo>None [00:14:46] Fundraising-Backlog: Worldpay: donors on Firefox receiving weird SSL error - https://phabricator.wikimedia.org/T91694#1416202 (atgo) [02:28:17] (PS1) Cdentinger: handle square refund with existing contribution [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/222051 [02:28:56] (Abandoned) Cdentinger: handle square refund with existing contribution [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/222051 (owner: Cdentinger) [02:32:02] (PS1) Cdentinger: WIP square audit file import - updates for new audit file - fix test - correctness adjustments - fix test again - handle refund of existing contribution [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/222052 (https://phabricator.wikimedia.org/T92582) [02:33:07] (PS10) Cdentinger: WIP square audit file import - updates for new audit file - fix test - correctness adjustments - fix test again - handle refund of existing contribution [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/218579 (https://phabricator.wikimedia.org/T92582) [02:33:54] (Abandoned) Cdentinger: WIP square audit file import - updates for new audit file - fix test - correctness adjustments - fix test again - handle refund of existing contribution [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/222052 (https://phabricator.wikimedia.org/T92582) (owner: Cdentinger) [02:45:23] (CR) Cdentinger: WIP square audit file import - updates for new audit file - fix test - correctness adjustments - fix test again - handle refund of (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/218579 (https://phabricator.wikimedia.org/T92582) (owner: Cdentinger) [02:58:14] (PS9) AndyRussG: WIP - Pls. Don't Merge - Refactor banner display, RL modules and client-side API [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) [02:59:04] (CR) jenkins-bot: [V: -1] WIP - Pls. Don't Merge - Refactor banner display, RL modules and client-side API [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) (owner: AndyRussG) [03:37:40] (PS10) AndyRussG: WIP - Pls. Don't Merge - Refactor banner display, RL modules and client-side API [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) [03:38:24] (CR) AndyRussG: "Mixins and kvStore now re-wired and ready for review :)" [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) (owner: AndyRussG) [03:38:26] (CR) jenkins-bot: [V: -1] WIP - Pls. Don't Merge - Refactor banner display, RL modules and client-side API [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) (owner: AndyRussG) [04:08:36] Fundraising-Backlog: [epic] 3D Secure for Europe - https://phabricator.wikimedia.org/T104391#1416439 (Reedy) But 3D Secure et al are just pure evil :( [04:55:06] (PS11) AndyRussG: WIP - Pls. Don't Merge - Refactor banner display, RL modules and client-side API [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) [04:55:53] (CR) jenkins-bot: [V: -1] WIP - Pls. Don't Merge - Refactor banner display, RL modules and client-side API [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) (owner: AndyRussG) [04:56:19] (PS12) AndyRussG: WIP - Pls. Don't Merge - Refactor banner display, RL modules and client-side API [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) [04:57:04] (CR) jenkins-bot: [V: -1] WIP - Pls. Don't Merge - Refactor banner display, RL modules and client-side API [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) (owner: AndyRussG) [05:18:04] (PS13) AndyRussG: WIP - Pls. Don't Merge - Refactor banner display, RL modules and client-side API [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) [05:18:49] (CR) jenkins-bot: [V: -1] WIP - Pls. Don't Merge - Refactor banner display, RL modules and client-side API [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) (owner: AndyRussG) [05:20:30] (PS14) AndyRussG: WIP - Pls. Don't Merge - Refactor banner display, RL modules and client-side API [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) [05:21:14] (CR) jenkins-bot: [V: -1] WIP - Pls. Don't Merge - Refactor banner display, RL modules and client-side API [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) (owner: AndyRussG) [15:48:55] morning ejegg [15:49:14] hi cwdent|afk [15:49:28] I'm just about to update drupal [15:49:38] exciting [15:49:48] https://gerrit.wikimedia.org/r/#/c/218579/10/sites/all/modules/offline2civicrm/ChecksFile.php [15:49:58] regarding that problem from yesterday [15:50:14] cwdent: oh, do you mind CRing this? https://gerrit.wikimedia.org/r/#/c/221894/ [15:50:27] sure thing [15:50:51] (CR) Cdentinger: [C: 2] Update Drupal submodule to 7.38 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221894 (https://phabricator.wikimedia.org/T103006) (owner: Ejegg) [15:51:54] thanks! [15:52:23] (Merged) jenkins-bot: Update Drupal submodule to 7.38 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221894 (https://phabricator.wikimedia.org/T103006) (owner: Ejegg) [15:56:19] (PS1) Ejegg: Update Drupal submodule to 7.38 [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/222136 (https://phabricator.wikimedia.org/T103006) [15:56:48] (CR) Ejegg: [C: 2 V: 2] Update Drupal submodule to 7.38 [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/222136 (https://phabricator.wikimedia.org/T103006) (owner: Ejegg) [15:56:50] (Merged) jenkins-bot: Update Drupal submodule to 7.38 [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/222136 (https://phabricator.wikimedia.org/T103006) (owner: Ejegg) [15:58:59] !log disabled queue consumers [15:59:03] Logged the message, Master [16:02:10] !updated civicrm from from e923225e423948bd70440e2d1131460b10cefac1 to 4fe0648ea9f36282731bf651a59ca1a617db6c08 [16:06:35] !log enabled queue consumers [16:06:39] Logged the message, Master [16:37:50] (CR) Ejegg: [C: -1] "refundLastTransaction gums up the works a bit with this solution" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/218579 (https://phabricator.wikimedia.org/T92582) (owner: Cdentinger) [16:38:29] (PS15) AndyRussG: WIP - Pls. Don't Merge - Refactor banner display, RL modules and client-side API [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) [16:41:47] * AndyRussG gives spaghetti "the look" [16:43:49] Fundraising Sprint N*E*R*D, Fundraising-Backlog, Astropay Integration, Unplanned-Sprint-Work, Patch-For-Review: Donate by Bank Transfer in Brazil via Astropay - https://phabricator.wikimedia.org/T98216#1417956 (Ejegg) I hope it was as simple as it looked! Astropay's API is really consistent ac... [16:47:45] Fundraising Sprint N*E*R*D, Fundraising-Backlog, Astropay Integration, Unplanned-Sprint-Work, Patch-For-Review: Donate by Bank Transfer in Brazil via Astropay - https://phabricator.wikimedia.org/T98216#1417969 (atgo) AWESOME. [16:50:44] hi awight [16:51:17] hope your tot is doing well! [16:51:38] Hi! Thanks for holding down the fire hose! [16:52:21] She was awesome yesterday, even if there was a brief moment where I considered abandoning boppa day [16:52:49] Kid has some impressive attitude [16:53:18] Not surprising ;) [16:53:25] How is your clotting factor? [16:53:27] AndyRussG: :p [16:53:49] ehh, my toes still look like a dead man's [16:53:58] but the ankle itself is feeling decent [16:54:03] ejegg: awww :/ [16:54:13] Is that (the toes view) supposed to happen? [16:55:02] doc said it was normal [16:55:13] ah cool! [16:55:15] lots of swelling i'd guess [16:55:19] crikey [16:55:34] yeah, basically a big bruise. [16:55:36] zombie toes slowly absorbing the zombie-antidote [16:55:37] Eat a ton of sausages and eggs so you don't have to see your feet. [16:55:40] hehe [16:55:47] hey AndyRussG could i grab you for like 10-15 min at some point today? have a couple of questions for how things work [16:56:15] atgo: sure! anytime on my calendar is cool! [16:56:21] cool :) [16:56:22] oh hey, does anyone have anything to ask of other teams? [16:56:28] * awight spins around Katie's chair to prove she is not K4-713 [16:56:29] or warn other teams about? [16:56:45] awight: ... what? [16:57:10] cwdent: XenoRyet, anything you want me to bring to scrum of scrums? [16:59:12] all good here [17:00:32] Jeff_Green: lmk when you'll have time to talk about my crackpot theories [17:00:33] ejegg: thanks! Maybe just that the CN bannerController API refactor is ending up as a very big change, and includes a huge amount of cruft cleanout and reorganization of code... Since it runs everywhere and impacts performance, we probably should get input from other teams [17:02:32] AndyRussG: OK, I'll call that out [17:02:40] ejegg: fantastic, thanks! [17:02:52] * AndyRussG is not scared.... err, not much! [17:02:54] Any team in particular you'd like to hear from? [17:04:53] ejegg: Well I've gotten quite a bit of input from Performance and Mobile, and continuing that would be fantastic... Maybe reading? Not sure who else, but I expect others may be interested... [17:04:58] ejegg: Na, I'm good [17:05:56] cool [17:06:18] awight: gimme about 20 min to finish pfw work with faidon [17:07:42] Jeff_Green: great, thx! [17:08:41] awight: ejegg: cwdent: XenoRyet: pls don't forget to forgive me for my horrible planning and take a peek at the patch of my brainhurt if you have a sec (np if not).... ;p [17:25:11] awight ok, i've got some time, might be a little fragged as we're finishing up one last bit of pfw config [17:27:00] No worries, I ship that way by default. [17:27:20] You have time to read my missive yet? [17:27:50] i did [17:28:03] Rubber meets the road at: can we use jessie-backports? [17:28:07] i saw redis cluster, i had it confused with sentinel [17:28:25] yeah it took me three times through the docs before the version implications sank in [17:28:33] short answer is: I don't know [17:28:36] yeah [17:28:36] k [17:29:04] i don't think there's a problem policy-wise, but technically I'm not sure how much hell will break loose, so it will require study [17:29:13] The authors claim production-readiness, but I'm trying to find some case studies of real people using redis cluster [17:29:22] sure, that actually sounds promising then. [17:29:54] It's the case that WMF ops is already supporting some Deb nodes, I guess? [17:30:15] I don't know if they're used in production yet [17:30:20] ack. [17:30:33] Fortunately, this is sort of the right time of year, if any, to make a dramatic payments cutover [17:30:44] dude [17:30:50] 8D [17:30:55] before you get too excited [17:31:13] I expect fundraising puppet to completely explode on debian [17:31:21] wheee [17:31:22] do not expect that to be a quick thing [17:31:34] paths... [17:31:39] everything [17:31:59] right. Any way to use jessie-backports on ubuntu? [17:32:01] paths, configuration style, package release incompatibilities, etc etc [17:32:08] yeuuchky [17:32:09] i don't know, never tried [17:32:18] and there's a pretty good chance that will get messy too [17:32:36] so what I can say is if by some miracle everything were perfect, we're still talking about a fairly long timeframe [17:32:42] I'd be happy to make an ubuntu deb for Redis 3, if that helps anything [17:33:21] I recently waded into the acid pool with deb packaging, c.f. http://mentors.debian.net/package/photo-booth [17:34:13] deb packaging is awful [17:34:25] anyway, there's no way this is a trivial project [17:34:32] Nope [17:34:46] If you see any alternative, holler [17:35:15] if it were up to me i would go back to the code and the logic [17:35:44] because you're talking about a project that's going to blow weeks to months of time [17:35:46] And do what? Rewrite it to use memcache? [17:35:56] or redis without clustering [17:36:11] Well I'm fine w/o clustering, but that's not HA [17:36:46] can you use isolated redis nodes in a way that doesn't faceplant when one disappears for 10 minutes? [17:37:07] honestly that seems like the easiest approach, and the best use of hardware [17:37:08] Sounds like, implementing failover in the application logic [17:37:26] ya [17:37:40] There are already two official things for that... [17:38:20] ? [17:38:36] redis-sentinel, which still requires the client to figure out all the failover stuff, and is an experimental package anyway, plus has already been deprecated [17:38:39] and then redis-cluster [17:39:35] right, and twemproxy probably, I haven't had a chance to thoroughly study that other than to determine that there's not a package available [17:39:52] What u think about the hand-rolled redis 3 package for Trusty? [17:40:04] twemproxy is also deprecated by redis-cluster... [17:41:17] dunno about redis 3 [17:41:28] aka redis-with-cluster [17:41:58] kneejerk reaction: relatively bleeding edge, not available as a maintained package, what could possibly go wrong? [17:42:12] Well it is available, just not in crappy Ubuntu [17:42:13] I can't answer the question without digging in [17:42:35] --[ Redis 3.0.0 ] Release date: 1 Apr 2015 [17:42:57] yep, I think they're on 3.0.4, which is still... zeroey [17:43:02] i don't think that's really something you can blame on ubuntu :-P [17:43:09] hehe true [17:43:19] we don't know anyone who is using it [17:43:25] just giving credit to Debian devs for being perverse [17:44:02] the other thing that bugs me about going redis-ha [17:44:18] I am not familiar with redis, so I have to get up to speed on administering it. [17:44:40] everything I've heard from people who have is "omg run" [17:44:40] On that note, I'm creeped out that we haven't needed this for the main cluster. [17:44:47] we have [17:44:49] i mean [17:45:04] the redis rig has taken down production :-) [17:45:07] hahaha [17:45:13] Redis 2, smoke machine [17:45:14] but the standards are lower [17:45:22] good god y'all [17:45:39] seriously--we can tolerate a little downtime in production, it's not like it's the 14 days of good fundraising time [17:45:51] more like, 3 days [17:45:57] ya [17:45:59] although I guess that improved last year [17:46:18] how long does data need to persist in this thing? [17:46:23] it's essentially a cache right? [17:48:40] I gotta go AFK for about 15 min [17:50:34] Fundraising-Backlog: Rearrange order of CC logos on GC form for Japan - https://phabricator.wikimedia.org/T102496#1418180 (atgo) a:Ppena>None [17:50:52] Fundraising-Backlog: Rearrange order of CC logos on GC form for Japan - https://phabricator.wikimedia.org/T102496#1366056 (atgo) [17:51:19] Jeff_Green: It doesn't need to persist more than an hour. [17:52:08] Here's Twitter's experience pre-Redis 3, grisly reading. http://highscalability.com/blog/2014/9/8/how-twitter-uses-redis-to-scale-105tb-ram-39mm-qps-10000-ins.html [17:57:41] if i want to push a new commit to an existing change in gerrit without smushing it into the last one, can i just amend the commit to have the same change-id? [17:58:50] cwdent: That will cause the new commit to replace the other one [17:59:01] ah, that's what i was wondering [17:59:02] What're you trying to accomplish? [17:59:17] Sounds like you just want a second commit? [17:59:20] i just want this change to be visually separate [17:59:27] yeah, should i just make a 2nd change? [17:59:41] Totally, unless there's a special effect you're going for [17:59:56] cool yeah, i dunno why i was thinking that was bad [18:00:09] probably cos the idea of branches in gerrit is totally evil [18:00:13] back [18:00:29] (PS1) Cdentinger: protected function handleDuplicate [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/222148 [18:00:34] Jeff_Green: hey, nothing much happened other than I answered the persistence thing above. [18:00:41] yup saw [18:00:53] I asked about production redis 3 stories in #redis, not much of a response yet [18:01:56] what exacly will be written/read from it in your new scheme? [18:02:33] This will be all the session data from incomplete donations. [18:02:54] when is it read vs written? [18:03:00] If it's completed within 20 minutes, this data gets sucked back in by the payments box completing the transaction and sent to the main queue [18:03:35] If it looks abandoned, the orphan slayer comes around after that 20 minutes and makes API calls to GlobalCollect to get the transaction status. [18:03:49] what if each payment box wrote its own local instance, and upon completion of the transaction it checked all the others for any data they might have for the same transaction? [18:04:02] Writes happen from payments1-3 as we redirect the donor off to GC [18:04:13] Reads happen on successful return from GC, or from the slayer. [18:04:40] so write local, and cycle through all the others at read time? [18:04:49] Jeff_Green: That sounds the same as a sharding approach, but I don't see how that improves availability [18:05:15] say payments1001 disappears for 5 minutes [18:05:34] a transaction completion request gets shunted over to payments1002 b/c lvs sees 1001 is down [18:05:53] What happens to the person making a donation on 1001? They're screwed, right? [18:05:57] payments1002 times out on the redis request to 1001, but otherwise doesn't die [18:06:12] i dunno, this is why I'm asking [18:06:23] what would have to happen for them not to die? [18:06:33] can it only be achieved by a triple write? [18:06:42] (CR) Ejegg: "I like this way of doing it! Needs a 'return true' at the end of SquareFile's handleDuplicate. I'd lean towards squashing this into the " [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/222148 (owner: Cdentinger) [18:06:49] I see. Well, failing to read on completion can be made survivable. We'll just leave the thing for the orphan slayer to clean up. [18:07:03] ok [18:07:16] and say payments1001 comes back online with the data it had before it went down [18:07:44] could the orphan slayer make use of that? [18:08:06] Sure, the slayer can be taught to retry every 5 minutes if the source machine is down. [18:08:13] Hrm, actually there's a second thing which might not be ok. [18:08:30] We also have a FIFO queue of transactions waiting to be resolved. [18:09:03] In your scheme, we'd have to split that up so that each box tracks its own queue... [18:09:04] and in the current model that's all in one redis bucket? [18:09:06] yeah [18:09:13] which is not too cool anyway [18:09:56] So if each box has its own FIFO dequeue, the slayer just has to know to read round-robin and take a chill pill if it can't access a node [18:09:59] not too bad [18:10:38] would it be a problem if it saw the same transaction twice, say if a downed host comes back but meanwhile another host has added it to its own FIFO? [18:10:48] No, that's fine [18:11:12] does all of this make your eyeballs explode or is it sounding kind of sane? [18:11:47] Good question [18:12:06] I have to take a nap and figure that out. [18:12:06] ha [18:12:49] It kind of blows up the abstraction layer I was working on, but that's not the end of the world. [18:13:37] it makes the redis pain orders of magnitude less painful though, and seemingly pretty resilient [18:13:43] It causes the application to be coupled to Redis specifically, and with transparent clustering we can swap out the backend with a simple 6-month twist of the thumbscrews [18:14:26] well redis just acts as a dumb store [18:14:32] you could use anything really [18:14:42] right? [18:14:55] (CR) Ejegg: "Or maybe reorder the commits to first add ChecksFile::handleDuplicate, then use it in the Square commit. Or not, nbd!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/222148 (owner: Cdentinger) [18:16:10] Jeff_Green: ah great point [18:17:10] The data structure dependency for the FIFO behavior already locks us into either activemq or redis anyway, so it's not like we're losing memcache with this decision. [18:18:26] maybe we can work long term on a proper redis cluster, in payments, to replace activemq [18:18:32] jfyi, this FIFO thing is what drove me off the Redis cliff in the first place. I'd originally planned to stay in Memcache, but emulating FIFO was going to require a batshit number of edge cases. Memcache can only hold 1MB in each key, there are no commands to get all keys by age or range, etc. [18:18:41] Yes please, but yeah next year [18:18:46] ya [18:19:02] once we see what color smoke emits from http://redis.io/topics/whos-using-redis ideally [18:19:15] totally [18:19:28] Trust that I'm not relishing anything about bleeding edginess [18:19:35] another idea ... [18:19:53] if we added a slave on some other predictable payments host as a backup [18:20:05] we could not use persistence [18:20:13] wat [18:20:24] so i.e. if a host faceplants, its slave has the data it had just before it died [18:20:37] and when it comes back it's empty [18:20:51] i dunno mostly just thinking out loud [18:20:53] but you need to manage promotion to master and client failover [18:21:05] right [18:21:20] otherwise you have double data or split brain issues [18:22:34] Definitely, implementing redis cluster in PHP for < 3.x is dead last on my list of fun things to do this summer [18:22:49] ya [18:25:05] http://download.redis.io/redis-stable/00-RELEASENOTES [18:25:12] I was wrong, they're only up to 3.0.2 [18:25:27] No horrific bugs fixed in the Cluster code yet, it seems [18:25:47] otoh, I can't find any sign that anyone is using it in the field. [18:26:33] i found some article talking about how it's finally not vaporware [18:26:45] maybe we'll try it and it will be great! [18:26:45] Written, I think, when it still was [18:26:48] lol [18:26:57] yeah a year ago [18:29:36] https://github.com/antirez/redis/labels/cluster [18:33:08] Okay, sorry. I'm off the cluster witch hunt for the moment. I'll try to take your app-level sharding thing and run with it. [18:33:27] ok [18:34:01] http://code.flickr.net/2014/07/31/redis-sentinel-at-flickr/ fyi [18:34:18] ya i almost sent that to you last night [18:34:24] it looks sorta promising [18:34:39] Jeff_Green: So the failure mode you're expecting is, one payments box goes down for a bit and then recovers? [18:34:45] Is this a thing that has happened? [18:35:15] that makes it possible for us to reboot for kernel updates, something we can do now [18:35:40] but I would also plan for the case where it doesn't come back for a few hours and when it does there's no data in redis [18:35:55] (CR) Ejegg: [C: -1] "Don't merge yet, I think I'm going to go with specifying logo in the submethod metadata" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/221890 (https://phabricator.wikimedia.org/T98216) (owner: Ejegg) [18:36:01] cool. [18:37:06] I haven't looked at the stock redis package closely to see if it restarts itself on update, that's also possible [18:37:28] one would hope not but I've seen that with mysql so I trust nothing [18:38:15] In the real world, losing up to all of the in-flight redis data means between a handful and a few thousand donations land in our database with no donor contact information. [18:38:47] Correction--probably not more than a few dozen. [18:39:19] you mean if it doesn't come back when a host reboots? [18:39:23] These would be the people who didn't come back to us for some reason, which are not numerous. [18:39:26] Yeah. [18:39:54] what if it comes back hours later with the data? [18:39:56] Redis is supposed to recover nicely from the file backing, but you're describing a situation where we have to rebuild the machine? [18:40:12] yeah [18:40:19] That's a new case we should handle, currently there's a maximum age that will reject stale data. [18:40:28] payments hosts have two drives in a RAID1 but you never know [18:40:33] yep [18:40:42] taser-armed lawnmowers. [18:47:42] (PS1) Ejegg: Add more_info_links partial for mustache forms [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/222160 (https://phabricator.wikimedia.org/T101234) [18:54:32] awight|floor: http://artpad.art.com/gallery/?mz7ld66ftr8 [19:05:57] Jeff_Green: running to get lunch, but I'm currently thinking your scheme will work great. One small change to suggest, that we have pay1-3 replicate to pay4, so as long as the orphan slayer is running, it has access to as much complete data as possible. [19:06:33] awight|fud: that sounds do-able [19:06:56] Wondering about how to disable expiry if pay4 goes down for a long time. There's a LocalSettings.php global that controls expiry time, I guess we could turn that knob manually? [19:07:21] Rats, actually I'm relying on expiry built into Redis. [19:07:29] I'll try to find a way around that... [19:07:44] really fudding now. [20:28:01] (PS1) Ejegg: Add logo filename to submethod meta, add big logos [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/222180 (https://phabricator.wikimedia.org/T101234) [20:32:08] (PS11) Cdentinger: WIP square audit file import - updates for new audit file - fix test - correctness adjustments - fix test again - handleDuplicate [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/218579 (https://phabricator.wikimedia.org/T92582) [20:38:03] (PS2) Ejegg: Add logo filename to submethod meta, add big logos [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/222180 (https://phabricator.wikimedia.org/T101234) [20:39:06] (PS2) Ejegg: Hide 'your card is safe' message for non-card methods [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/221892 (https://phabricator.wikimedia.org/T98216) [20:39:08] (PS2) Ejegg: Add Brazil banks and logos for Astropay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/221888 (https://phabricator.wikimedia.org/T98216) [20:39:30] (CR) jenkins-bot: [V: -1] Hide 'your card is safe' message for non-card methods [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/221892 (https://phabricator.wikimedia.org/T98216) (owner: Ejegg) [20:39:49] (Abandoned) Ejegg: Use bank- prefix for BT logos [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/221890 (https://phabricator.wikimedia.org/T98216) (owner: Ejegg) [20:40:47] Fundraising Sprint Lou Reed, Fundraising Sprint Miles Davis, Fundraising Sprint N*E*R*D, Fundraising Tech Backlog, and 2 others: Spike: How to make our new limbo implementation high-availability? - https://phabricator.wikimedia.org/T103206#1418661 (awight) [20:41:08] ejegg: good call on that late return, and i merged the last commit into the previous ones here https://gerrit.wikimedia.org/r/#/c/218579 [20:41:28] nice! I'll check that out [20:42:58] (PS3) Ejegg: Hide 'your card is safe' message for non-card methods [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/221892 (https://phabricator.wikimedia.org/T98216) [20:45:07] Jeff_Green: How certain can we be that the load balancer sends a donor back to the same box for each request? [20:45:27] That won't actually affect the design, but I'm curious. [20:45:49] Sorry I ask this every year and forget the answer. [20:45:54] are our sessions stored on the machines, or are they cluster-wide? [20:46:14] They're supposed to be clustered via memcache [20:46:19] ah, cool [20:46:26] I feel like we realized that wasn't working at some point, though. [20:47:01] you can query memcache from payments1004 if anyone gets overly curious... [20:47:32] awight: sorry, missed the irc client bing-bong [20:47:48] Seemed pretty timely to me [20:47:59] it's just a matter of whether or not LVS decides a host is unavailable [20:48:09] great [20:48:23] i think the way we're tuned it's very stable until you do something like restart apache [20:51:08] Jeff_Green: you know the outcome of the memcache clustering thing we were talking about ^^ ? [20:51:30] We think it's working correctly, and all boxes have access to every session? [20:51:56] the existing clustering? [20:52:00] yeah [20:52:30] devil is in the details, but I was under the impression 1001-1003 know about eachother, and 1004 stands alone [20:52:48] ok good to know [20:52:59] am I right? :-$ [20:54:38] Fundraising Sprint Lou Reed, Fundraising Sprint Miles Davis, Fundraising Sprint N*E*R*D, Fundraising Tech Backlog, and 2 others: Spike: How to make our new limbo implementation high-availability? - https://phabricator.wikimedia.org/T103206#1418692 (awight) [20:59:52] fundraising-tech-ops: Get Anna Stillwell civi certificate - https://phabricator.wikimedia.org/T104495#1418703 (atgo) NEW a:Jgreen [21:00:44] Fundraising Sprint Lou Reed, Fundraising Sprint Miles Davis, Fundraising Sprint N*E*R*D, Fundraising Tech Backlog, and 2 others: Spike: How to make our new limbo implementation high-availability? - https://phabricator.wikimedia.org/T103206#1418714 (awight) [21:01:27] Fundraising Sprint Lou Reed, Fundraising Sprint Miles Davis, Fundraising Sprint N*E*R*D, Fundraising Tech Backlog, and 2 others: Spike: How to make our new limbo implementation high-availability? - https://phabricator.wikimedia.org/T103206#1384509 (awight) [21:02:56] Fundraising-Backlog: Create a civi account for Anna Stillwell - https://phabricator.wikimedia.org/T104496#1418723 (atgo) NEW [21:04:07] (PS12) Ejegg: Square audit file import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/218579 (https://phabricator.wikimedia.org/T92582) (owner: Cdentinger) [21:04:22] (PS13) Ejegg: Square audit file import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/218579 (https://phabricator.wikimedia.org/T92582) (owner: Cdentinger) [21:05:27] (CR) Ejegg: [C: 2] "This oughtta do it! We should investigate whether CoinbaseFile needs the same refund fix." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/218579 (https://phabricator.wikimedia.org/T92582) (owner: Cdentinger) [21:06:12] heading out for a bit, but I'll be back for shrimp chicken [21:21:07] Fundraising Tech Backlog, MediaWiki-extensions-DonationInterface, Epic: Modify DonationInterface limbo code for high availability deployment - https://phabricator.wikimedia.org/T104499#1418827 (awight) NEW a:awight [21:23:06] Fundraising Tech Backlog, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Epic: Modify DonationInterface limbo code for high availability deployment - https://phabricator.wikimedia.org/T104499#1418847 (atgo) [21:24:29] (CR) Awight: [C: 2] Add more_info_links partial for mustache forms [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/222160 (https://phabricator.wikimedia.org/T101234) (owner: Ejegg) [21:25:31] (Merged) jenkins-bot: Add more_info_links partial for mustache forms [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/222160 (https://phabricator.wikimedia.org/T101234) (owner: Ejegg) [21:38:43] (PS16) AndyRussG: WIP - Pls. Don't Merge - Refactor banner display, RL modules and client-side API [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) [21:39:22] (CR) AndyRussG: "New API!!!!" [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) (owner: AndyRussG) [21:39:39] (CR) jenkins-bot: [V: -1] WIP - Pls. Don't Merge - Refactor banner display, RL modules and client-side API [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) (owner: AndyRussG) [21:39:58] atgo: one sec :) [21:43:13] Fundraising Sprint Miles Davis, Fundraising Sprint N*E*R*D, Fundraising-Backlog, Astropay Integration, Patch-For-Review: Some style issues with Astropay form - https://phabricator.wikimedia.org/T101234#1418901 (awight) Bug report: The "Please correct errors in your donation amount" messages sho... [22:13:27] Fundraising Sprint N*E*R*D, Fundraising Tech Backlog, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 3 others: Make recurring next scheduled calculation HHVM-compatible - https://phabricator.wikimedia.org/T91898#1098378 (awight) [22:13:44] Fundraising Sprint N*E*R*D, Fundraising Tech Backlog, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Unplanned-Sprint-Work: Deprecate "language" parameter - https://phabricator.wikimedia.org/T96621#1418987 (awight) [22:19:19] Fundraising Sprint N*E*R*D, Fundraising Tech Backlog, Fundraising-Backlog, Astropay Integration, Patch-For-Review: Make audit processing testable - https://phabricator.wikimedia.org/T104507#1419018 (awight) NEW a:awight [22:22:19] Fundraising Sprint N*E*R*D, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Deploy schema change - https://phabricator.wikimedia.org/T104508#1419026 (awight) NEW a:awight [22:27:19] Fundraising Sprint N*E*R*D, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Deploy CentralNotice schema change - https://phabricator.wikimedia.org/T104508#1419055 (awight) [23:03:13] argh, the robots got me! [23:05:08] ejegg|mtg: i have to catch a ride home pretty soon, would you mind helping me deploy that square file in the morning? i don't want to break civi and go afk [23:05:20] cwdent: sure thing! [23:05:43] ty! [23:08:26] (PS3) Ejegg: Port cal_days_in_month to hhvm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221776 (https://phabricator.wikimedia.org/T91898) (owner: Awight) [23:10:28] (CR) Ejegg: [C: 2] "Come on down to the PHP slots parlor - loosest typing this side of Vegas!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221776 (https://phabricator.wikimedia.org/T91898) (owner: Awight) [23:11:20] (Merged) jenkins-bot: Port cal_days_in_month to hhvm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221776 (https://phabricator.wikimedia.org/T91898) (owner: Awight) [23:18:45] (CR) Awight: "Hehe, yeah it can be hard to tell in php when you're loosening a screw and when you're tightening" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221776 (https://phabricator.wikimedia.org/T91898) (owner: Awight)