[00:27:52] (CR) Ejegg: "Seems to work, just curious about the check on child activities." (1 comment) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/282095 (owner: Eileen) [00:32:09] (CR) Eileen: CRM-18214 Add link to detail report for contact merge activities, with oid if applicable. (1 comment) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/282095 (owner: Eileen) [03:16:11] (PS1) Eileen: CRM-18409 fix inability to view activity if source contact id is deleted. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/284126 [03:16:40] (PS2) Eileen: CRM-18409 fix inability to view activity if source contact id is deleted. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/284126 (https://phabricator.wikimedia.org/T122519) [03:22:11] Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Permissions issue? "You do not have permission to access this page" - https://phabricator.wikimedia.org/T122519#2216725 (Eileenmcnaughton) I have 2 theories as to this The first is that the source contact I... [04:12:56] (PS1) Eileen: CRM-17983, CRM-18401 sanitise post params on activity tab [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/284127 [10:11:26] morning jessicarobell! The LV/RO/SK test just went up (a few minutes late because I forgot to enable) [10:12:02] Ah ok, thank you the-wub! [10:13:29] results sheet is here: https://docs.google.com/spreadsheets/d/17arGLTPupnrCmo7O0He38js-gmOoF3OY53sOwiAbAp0/edit#gid=1950059998 [10:15:07] Great. thanks! Saw that :) [10:16:42] Thank you for setting that up the-wub! [10:35:20] ./statler B1516_0419_esES_dsk_p2_sm_dsn_ -s 2016-04-19 [10:41:26] looking pretty low traffic so far. conversion seems good, except for cc in SK [10:44:32] Fundraising-Backlog: Astropay MX form: minimum amount error - https://phabricator.wikimedia.org/T132959#2217476 (Pcoombe) @Ejegg @awight @Ppena Can you clarify what the minimum amounts for the Astropay forms are / should be? For MXN all I could find in DonationInterface code is the minimum being 15.82, and... [12:04:55] hi, what's the status of fundraising this year? I'm getting a lot of messages that I usually associate with the end of year fundraiser. I'm a bit out of things, but I can't find anything like a fundraiser planning or timeline on meta or foundation wiki. https://meta.wikimedia.org/wiki/Fundraising#Latest_Updates didn't help me much either [12:19:45] hi MartyH, what country are you in? we spread out our fundraising in different countries throughout the year now. At the moment we are fundraising in the Netherlands and Spain [12:20:01] and doing a brief test in Latvia, Romania and Slovakia today [12:23:03] the-wub: sorry, freenode webchat went down, I'm in the Netherlands [12:23:14] Wikimedia-Fundraising: Way for not logged-in users to hide donation request banners for period of time - https://phabricator.wikimedia.org/T132243#2217725 (Pcoombe) Open>Resolved a:Pcoombe Hi @Walter, we already automatically set cookies for users on the [[ https://wikimediafoundation.org/wiki/Sp... [12:27:51] MartyH: okay, that explains it. we've got one more week of fundraising planned there. [12:32:12] ah, ok. It would be nice if the status of fundraising was tracked somewhere on meta. Maybe it is, but I can't find it. A week-by-week or month-by-month this week/month we plan to run these campaigns in these regions [12:33:02] MartyH: That's a good idea [12:33:04] I understand that may feel like a lot of bookkeeping for bookkeepings sake, but at least for me it would be really nice [12:34:38] MartyH: does https://meta.wikimedia.org/wiki/CentralNotice/Calendar work for you? [12:35:27] by the looks of things that's perfect. I'll see if I can link that from meta/fundraising [12:53:41] Fundraising-Backlog: Ingenico: iDEAL form error with Abn-Amro bank - https://phabricator.wikimedia.org/T132680#2217806 (MBeat33) Open>Resolved a:MBeat33 Abn-Amro continues to function, so the error the donor got is likely maintenance on the bank's end. We'll monitor for any more of this but for n... [13:26:48] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Banner device settings should be set at campaign level - https://phabricator.wikimedia.org/T127011#2217884 (Pcoombe) Relatedly it could also make sense to move the Anonymous/Logged In setting to the campaign level at the same time. [13:54:27] FYI I just disabled fundraising banners in advance of the datacenter switch. sent an email to fr-online [14:06:27] (CR) AndyRussG: "Great!! Looking beautiful so far!!!" (2 comments) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/254326 (https://phabricator.wikimedia.org/T111456) (owner: Krinkle) [15:15:03] back in a bit, need to find a power socket [16:08:05] okay, I'm back and so are the fundraising banners :) [16:15:10] (PS1) Aaron Schulz: Use GUI read-only errors instead of DB class errors [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/284217 [17:24:07] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Add payment method to thank you page URL - https://phabricator.wikimedia.org/T133072#2218910 (Pcoombe) [17:27:31] Fundraising-Backlog: Astropay MX form: minimum amount error - https://phabricator.wikimedia.org/T132959#2218934 (awight) p:Triage>High Increasing priority, cos that sounds like a bad thing for the MX campaign. [17:27:44] Fundraising-Backlog: Astropay MX form: minimum amount error - https://phabricator.wikimedia.org/T132959#2218942 (awight) [17:28:21] Fundraising Sprint Hermit Crab Husbandry, FR-Astropay: Donate by Cash in Mexico - https://phabricator.wikimedia.org/T98219#2218955 (awight) [17:28:58] Fundraising-Backlog, FR-Astropay, Patch-For-Review: [epic] Processing via Astropay for Spanish-speaking LATAM countries - https://phabricator.wikimedia.org/T102143#2218974 (awight) [17:29:03] Fundraising Sprint Hermit Crab Husbandry, FR-Astropay: Donate by credit/debit card in Mexico - https://phabricator.wikimedia.org/T98218#2218977 (awight) [17:33:38] oh good [17:34:12] okeedoekee [17:34:23] For context, if anyone else is listening: https://phabricator.wikimedia.org/T130283#2216191 [17:34:32] ok, so having viewed that.... [17:34:58] you recommend concrete shoes? [17:35:08] I'm not terribly concerned about which version we would use [17:35:25] although I guess I would probably lean toward the same as we're opting for for prod [17:35:33] Me neither, other than ZooKeeper probably causes hellish extra complications. [17:35:52] right, so I would say we go fully stock vs confluent packages [17:35:54] 0.9 gets rid of that, in favor of some self-brokering thing I do not understand yet. [17:36:27] actually that raises a good q, I didn't ask why we haven't opted for the stock packages in the past [17:36:29] 0.9 doesn't need zookeeper any more? [17:36:30] There isn't *any* stock package for debian [17:36:48] ah well there is that, ok! [17:36:53] Ubuntu neither, it seems [17:36:55] http://packages.ubuntu.com/search?keywords=kafka [17:37:12] ejegg: exactly [17:37:51] ah, here's the big leap: committing offsets in Kafka rather than through ZK [17:38:10] https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka%28ratherthanZookeeper%29in0.8.2? [17:38:17] https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoesKafkadependonZookeeper? [17:38:29] yeah looking at one of the many tutorials out there, it looks like one way is to use a stock zookeeper and jre packages and a kafka binary [17:38:46] hrm. it looks like the brokers still rely on ZK, so that's not simplified. It's only the client dependency which goes away. [17:38:55] Jeff_Green: ok--where does the binary come from? [17:39:13] looks like an apache project mirror but I'm not positive [17:39:23] I'm looking at this: https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-on-ubuntu-14-04 [17:40:02] Another alternative is that we could fork the Confluence packaging files and build ourselves. [17:40:25] well...except that they don't distribute the source packages [17:40:30] wat. ok [17:40:49] There's a good reason to avoid them. [17:40:52] maybe their package just bundles the binary, that would be amusing [17:41:46] we do have another context for using packages straight from a vendor--we do that with RAID controllers since there's really no alternative [17:42:34] also afaik we would only need the confluent package for the server side, not the client side [17:42:49] * awight rereads ottomata's letter explaining why the decision to use the Confluent package [17:43:00] Jeff_Green: yes. client side is just librdkafka1, right [17:43:05] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Add payment method to thank you page URL - https://phabricator.wikimedia.org/T133072#2218910 (Ejegg) Maybe the solution is to use a Mustache template for the urls and render it with all the donation data we have in session. Then it's just a se... [17:43:37] gasp--which is not stable in Ubuntu [17:44:03] it's in jessie, but that's a whole project to migrate frack nodes. [17:44:08] awight: and apparently the .8 client will work with the .9 server, but I think we would need .9 on the client side too for encryption [17:44:13] yep [17:44:19] I'm already working on jessie [17:44:30] however, it didn't look like my PHP client does any of that good stuff. [17:44:35] trying to get to a point where we can install jessie as we get new hardware [17:44:43] i love it [17:44:58] one less dependency on coked-out Afrikaaners... [17:45:02] ha [17:45:36] so what I have been leaning toward, and granted I'm buried in tasks so leaning is sort of a relative term... [17:46:19] we build a new jessie box or two and install whichever flavor of queue we decided to try first, and we can test with it using lutetium [17:46:39] we could do this using aluminium if we aren't able to get new hardware in place in time for testing [17:46:52] Either works for me. [17:47:05] k, I'll assume jessie for the server nodes. [17:47:09] ya [17:47:26] is the kafka install getting complicated enough to look at redis again? [17:47:41] cwd: From my perspective, yessir. [17:47:47] cwd: https://phabricator.wikimedia.org/T130283#2216191 [17:47:54] cwd: I think we should think of it as looking at both at this point [17:47:59] It's not even the install I'm scared of, but the bleeding edginess [17:48:07] although I'm not the one who would have to write PHP [17:48:21] hehe--nor the HHVM php extension and packaging [17:48:29] * awight coughs up bile [17:48:34] the thing that appeals to me most about redis is the lack of glue needed [17:48:51] no special brokers or consumers are needed, there are redis bindings for everything [17:49:26] the sticky point with redis is HA [17:49:34] it's very malleable which is an appealing property of something getting dropped in in place of something else [17:49:46] true [17:49:58] HA as in proxy? [17:50:05] oh high availability [17:50:06] cwd: fwiw, the place I arrived at last night was that the big non-risk step we can take is to refactor all of our clients on top of php-queue. Then, we simulate whatever cool stuff Kafka gives us, like multiple consumers and stored offsets, in the other data store adapters. Then it doesn't matter wtf we provision as long as it's high availability. [17:50:12] yeah [17:50:33] minimum HA config is 3 machines [17:51:13] redis slaves sound pretty easy... [17:51:17] cwd: for Redis, the crazy seems to be limited to, https://packages.debian.org/jessie-backports/redis-server [17:51:23] we would be using a backport of Redis 3 [17:51:31] And we would be the first team at WMF to do so [17:51:53] cwd: agreed re. slaves, just means no seamless failover [17:51:54] is that to get features? [17:52:01] cwd: it's to get transparent clustering [17:52:18] Jeff_Green: what about the other HA as in proxy? [17:52:20] http://redis.io/topics/cluster-spec [17:52:40] could we do failover at another level? [17:53:02] cwd: We could, but after doing it at application level, I'm daunted by the complexity. [17:53:25] cwd: possibly, I don't really know what that would look like but can explore it [17:53:53] in any case it's not going to be any worse than what we have now :) [17:54:25] ha, is that the threshold for success? [17:54:35] That's a given--but if we don't eliminate the SPOFiness, it's lost labor [17:55:23] and HA would qualify as eliminating teh spofness? [17:55:32] IMO yes [17:55:36] well even simple replication would be better than what we have now [17:55:41] yeah [17:55:49] we could do something along the lines of failing open if the master croaks [17:56:04] If the frontend can 99.99% always write its data to something persistent, we're good. [17:56:10] well, my comment was nonsense ^^^ but failing noncatestrophically [17:56:17] yeah, it's all relative [17:56:32] the service itself is still a spof but what can you do about that [17:56:40] yep. [17:58:00] awight: how hard do you think it would be to get from where we with payments redis clients, to something that can smartly work with two servers? [17:58:15] Jeff_Green: That's pretty easy. [17:58:21] i.e. if we had two redis machines cross-replicating, and one falls over [17:58:41] That sort of makes me happy. [17:59:00] if we had redis slaves coupled with an haproxy load balancer that pinged them and could depool, that'd be 2 lines of defense [17:59:02] That's the application-level failover I wrote for the last batch of Redis support. [17:59:10] (PS3) Ejegg: CRM-18409 fix inability to view activity if source contact is deleted [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/284126 (https://phabricator.wikimedia.org/T122519) (owner: Eileen) [17:59:29] +application level failover [17:59:40] hmm "+"? [17:59:43] (CR) Ejegg: "PS3: linebreaks for commit msg" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/284126 (https://phabricator.wikimedia.org/T122519) (owner: Eileen) [17:59:47] It's not that awesome [18:00:03] what is the nature of it? [18:00:17] It causes us to make assumptions about our topology, in code [18:00:45] I mean, I tried to make it general enough to work on a single node, but it still has some crufty code paths for tracking which servers are down, etc. [18:01:50] http://redis.io/topics/sentinel [18:01:56] (CR) Ejegg: [C: 2] "Looks scary at first glance, but following logic in the BAO's checkPermission logic keeps it from opening up any further than it ought to." [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/284126 (https://phabricator.wikimedia.org/T122519) (owner: Eileen) [18:01:59] this sounds liek what we want right? [18:02:16] cwd: Sentinel is the last generation of clustering, we don't want to go there [18:02:24] it's deprecated in favor of Redis 3 [18:02:31] oh gotcha, it's built in? [18:02:35] in 3, yes [18:02:45] Sentinel is basically an extension for Redis 2 [18:03:02] RedisConf 2016 is near! Join us in San Francisco, May 10 - 11 2016, Mission Bay Conference Center. [18:03:10] har har [18:04:38] don't forget your polo [18:05:10] blargh [18:05:25] ok so...where do we go from here? [18:05:36] other than redisconf [18:05:40] hmm. [18:05:41] hee [18:05:54] we currently read from the queue in serial right? [18:06:00] I can keep evaluating stuff, but I think I have a pretty good picture of the landscape. [18:06:21] (Merged) jenkins-bot: CRM-18409 fix inability to view activity if source contact is deleted [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/284126 (https://phabricator.wikimedia.org/T122519) (owner: Eileen) [18:06:29] cwd: yeah, most of our queues are currently FIFO only [18:06:49] cwd also, so far we haven't run two consumers against the same queue anywhere [18:07:03] ok. I could probably get redis up pretty quickly on lutetium, but getting far with kafka is going to be a while [18:07:10] that makes failover sort of a non issue until we actually parallelize the code doesn't it? [18:07:15] Jeff_Green: If you aren't too scared by the Kafka obstacles, I can do some more risky packaging of the PHP extension and stuff [18:08:03] Jeff_Green: I'm happy with spending maybe a month just on the code changes to support php-queue (our generalized shim layer) [18:08:21] We wouldn't need any production or staging gear until the end of that window. [18:08:28] ok [18:08:47] Those changes I'm planning should create a layer that we can apply to either Redis or Kafka. [18:09:02] right, which is time well spent either way [18:09:06] yep [18:09:45] There's isn't a huge business motivation behind my decision there, other than * reduce future risk * make our stuff delicious, flexible open-source [18:09:54] yeah [18:09:57] so it's questionable [18:10:19] and I would rather not double andrew's efforts in the meantime re. kafka .9 [18:10:53] +2 [18:11:23] awight: want to break out some new tickets at grooming? [18:11:32] I'm not sold on the 0.9 features, fwiw [18:11:46] just cos I don't see us benefitting via PHP client [18:12:11] I would only go with 0.9 in order to stay in sync with WMF [18:12:30] (PS4) Ejegg: CRM-18214 Add link to detail report for contact merge activities, with oid if applicable. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/282095 (owner: Eileen) [18:12:34] right, but it's not as though there's a big win for sticking with 0.8 is there? [18:12:40] nope. just packaging [18:13:00] which apparently is not a thing anyway. sorry [18:13:11] it's a half-thing [18:13:16] hehe [18:13:39] Punting this for a bit is okay, but any ideas about how to keep forward momentum? [18:13:47] next steps... [18:14:07] * keep an eye on ottomata's efforts [18:14:21] * Read about Redis 3 adoption [18:14:23] donno... [18:14:46] personally i'd opt for the stock jessie package unless there's a problem with it [18:14:55] jessie-backports? wfm [18:15:07] looking.... [18:15:29] https://packages.debian.org/jessie-backports/redis-server [18:16:00] numbering scheme madness: 2:3.0.6-2 [18:16:12] * awight tightens blinders [18:16:28] 2^293.%39PIx11.04-3 [18:16:40] I'd still like to avoid application-level failover, unless anyone would like to challenge that? [18:16:44] cwd: you said "+"? [18:17:16] oh, no i think avoiding it is best unless there's a good reason [18:17:17] I would just like to say, I don't know how it's going to look to avoid it completely [18:17:23] it's probably the cheesiest way right? [18:17:54] Jeff_Green: good point--I don't think there's any discoverability thing, if all the Redis nodes go down and have to be respawned with new IPs [18:18:05] blind faith in an IP:port to always behave perfectly is the cheesiest way imo :-) [18:18:21] hehe that's true [18:18:28] * awight adds an "if 0" [18:18:32] hmm yeah [18:18:54] speaking of cheese, $2 slices as rosalee's on tues [18:18:57] i think we should expect to have to do some work from both sides [18:18:57] gonna get me some [18:19:22] What I'm imagining is that we have a list of IPs that should work, and Redis can manage all the multi-master sync BS [18:19:46] (CR) Ejegg: [C: 2] "Works! Was afraid there might be a case where we add links twice due to action being reset to getEntityAction, but it seems there's only a" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/282095 (owner: Eileen) [18:20:18] awight: i would feel better about that if redis had a longer history on one clustering approach [18:20:24] +1 [18:20:48] but I will look into clustering when I get time [18:20:59] Jeff_Green: Well, I'm fine with your manual failover proposition if you are. [18:21:02] for now [18:21:19] It allows us to do hot maintenance and stuff, at least. [18:21:28] i don't know how it would work though [18:21:53] I think you'd have a master and slave, and promote the slave manually in case stuff [18:22:03] but I'm green [18:22:20] (Merged) jenkins-bot: CRM-18214 Add link to detail report for contact merge activities, with oid if applicable. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/282095 (owner: Eileen) [18:22:38] (PS2) Ejegg: CRM-17983, CRM-18401 sanitise post params on activity tab [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/284127 (owner: Eileen) [18:23:38] awight ya. we would have to develop & test a protocol from the ops side [18:24:13] Probably a good chunk of downtime even with that in place [18:25:13] I can work on testing that anyway [18:25:54] Feel free to check in next week or something, if you have more thoughts. [18:26:11] I'll focus on the shim work for now. [18:26:17] ok [18:27:10] I'll try to add this chat to Phabricator, it looks like I should reopen https://phabricator.wikimedia.org/T130304 [18:28:21] yup [18:28:42] if I think of things I'll comment on the tickets [18:28:55] thanks for the time! [18:28:57] but I'm two large tasks deep at the moment, so...progress will be slow [18:28:58] (CR) Ejegg: [C: -1] "Can't just export unsubscribes from silverpop_excluded - it's got non-primary and deleted emails, but no primary emails for existing conta" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/282303 (owner: Awight) [18:28:59] sure! [18:29:06] Jeff_Green: no rush [18:30:05] (CR) Awight: "Ah, good point. I'll squash with the next patch." [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/282303 (owner: Awight) [18:30:46] (CR) Ejegg: [C: 2] "Big improvement! getWhiteListedParametersFromPost looks like it could be useful elsewhere too." [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/284127 (owner: Eileen) [18:39:37] Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog, fundraising-tech-ops, FR-ActiveMQ: [Epic] Provision high-availability Kafka cluster for Fundraising - https://phabricator.wikimedia.org/T130283#2219388 (awight) [18:39:39] Fundraising Sprint Freshmaking, Fundraising-Backlog, fundraising-tech-ops, FR-ActiveMQ, Patch-For-Review: Spike: Investigate suitability of Kafka instead of Redis - https://phabricator.wikimedia.org/T130304#2219383 (awight) Resolved>Open Some discouraging developments are documented i... [18:39:45] Fundraising Sprint Freshmaking, Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog, fundraising-tech-ops, and 2 others: Spike: Investigate suitability of Kafka instead of Redis - https://phabricator.wikimedia.org/T130304#2219392 (awight) [18:40:16] Fundraising-Backlog, fundraising-tech-ops, FR-ActiveMQ: [Epic] Provision high-availability Kafka cluster for Fundraising - https://phabricator.wikimedia.org/T130283#2219393 (awight) [18:40:39] (CR) Ejegg: "good numbers to have, sorry I let this languish till a path conflict prevents rebase. Also, I agree with the first punctuation change but" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/282108 (owner: Awight) [18:41:55] (CR) Awight: "hehe, I think I agree" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/282108 (owner: Awight) [18:42:40] Does this look like a good solution to failmail overload on TY letter problems? https://gerrit.wikimedia.org/r/281842 [18:43:20] (PS2) Awight: Comment about query performance [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/282108 [18:44:16] (Merged) jenkins-bot: CRM-17983, CRM-18401 sanitise post params on activity tab [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/284127 (owner: Eileen) [18:44:34] ejegg: I think that, generally, we should be doing something better about multiple failures. In the offline2civicrm import, I just implemented a consecutive failure threshold, maybe we should do that here as well? [18:45:41] Although, I appreciate that your patch just gets stuff into Civi, where we can do the cleanup work asynchronously [18:45:47] awight would we want that for the ty job? [18:46:12] Seems a better fit for mthings run manually [18:46:31] It makes sense--if the TY job's sole responsibility is to thank people and mark their records, it should probably stop if it's unable to do that [18:47:00] Sure, if it's a problem like the mailing infrastructure [18:47:22] I don't understand the bug you ran into, I guess [18:47:24] but if it hits a run of screwy looking records, should it bail on the good ones? [18:47:40] Why would we get contributions in the query result, which later don't exist? [18:48:42] Also, if Jenkins is going to run the job again 2 minutes later, does it help much to bail out? [18:49:06] no... We don't have a good way to deal with that [18:49:13] Wanna design that? :) [18:49:29] Geometric backoff might be nice. [18:49:33] let's see, the original issue was something with retrieving recurring donations [18:50:44] Fundraising-Backlog, fundraising-tech-ops, FR-ActiveMQ: [Epic] Provision high-availability Kafka cluster for Fundraising - https://phabricator.wikimedia.org/T130283#2131633 (Ottomata) Happy to chat too if you all are having trouble. Here's a crazy idea...don't use PHP for backend job processing stuf... [18:51:29] awight we could have it flip its own off switch (set thank_you_batch=0) like that hand in a box [18:52:23] That might make more sense. From memory, fatal errors in jobs never resolve themselves. [18:55:56] awight aha, there were recurring donations attached to deleted contacts: https://phabricator.wikimedia.org/T131200 [18:56:02] hrm [18:56:10] shouldn't we support that? [18:56:38] the contact retrieval fail , i think cos api call [18:57:19] That doesn't sound right, at least it should be possible to retrieve deleted contacts. [19:00:32] huh, doesn't look like either of our follow-on checks should have been failing it [19:01:31] in any case, we do check things that are specific to particular contributions, and not send email if the checks fail [19:01:50] I feel like in that case, we really should mark the contribution and send failmail [19:02:34] for the generic mailer failure, something like consecutive failures disabling the job seems right [19:02:42] brb! [19:15:42] (PS1) Ejegg: Add comments, underscore in key [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/284266 [19:15:53] hmm, I should get some food... [19:39:17] (CR) Krinkle: kvStoreMaintenance: Refactor to use requestIdleCallback (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/254326 (https://phabricator.wikimedia.org/T111456) (owner: Krinkle) [19:40:33] (CR) Krinkle: kvStoreMaintenance: Refactor to use requestIdleCallback (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/254326 (https://phabricator.wikimedia.org/T111456) (owner: Krinkle) [19:44:52] (CR) AndyRussG: [C: 2] "\o/" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/254326 (https://phabricator.wikimedia.org/T111456) (owner: Krinkle) [19:46:28] (Merged) jenkins-bot: kvStoreMaintenance: Refactor to use requestIdleCallback [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/254326 (https://phabricator.wikimedia.org/T111456) (owner: Krinkle) [19:54:09] is the issue with HA that redis does not support dual master mode? [19:54:54] (CR) Ejegg: [C: 2 V: 2] Comment about query performance [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/282108 (owner: Awight) [19:55:39] cwd: Pretty much--and if the master goes down, it's all over [19:56:01] Redis 3 does support automatic failover though [19:56:12] this must be a stupid question but, why not use postgres? [19:56:18] eh [19:56:27] I hadn't considered it [19:56:37] But the php-queue layer should be able to handle that [19:57:39] i doubt writing a queue like interface would be very complicated or time consuming [19:57:43] even if it didn't [19:57:49] there's already a MySQL adapter [19:57:54] +1 easy [19:58:07] yep [19:58:36] i have to vouch for postgres as maybe the best experience i've had with a service [19:58:44] every time i thought it was wrong, i was wrong [19:58:44] Also, I'm seduced by the Kafka premise that we don't actually want to pop anything [19:59:05] I'm totally impressed by the small amount of postgres coding I've done [19:59:12] you mean like mark deleted vs actually remove? [19:59:21] Pretty much the only problem was that I became too spoiled to go back to any other rdb [19:59:28] not even mark deleted [19:59:28] you got that right [19:59:34] there's just a per-consumer offset [19:59:41] ah sure [19:59:53] k, relocating to the conference room, but please ping later if you want to elaborate [20:02:06] AndyRussG: backlog grooming? [20:02:15] ejegg|meet: coming! [20:07:53] Fundraising-Backlog, MediaWiki-extensions-DonationInterface, FR-Astropay: Consider finalizing AstroPay cash methods before redirect - https://phabricator.wikimedia.org/T132956#2219920 (awight) [20:11:27] Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog, Unplanned-Sprint-Work: Astropay MX form: minimum amount error - https://phabricator.wikimedia.org/T132959#2215283 (Ejegg) [20:12:08] Fundraising-Backlog, MediaWiki-extensions-DonationInterface, FR-Astropay: Spike: Consider finalizing AstroPay cash methods before redirect - https://phabricator.wikimedia.org/T132956#2219952 (awight) [20:12:27] Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, FR-Astropay: Spike: Consider finalizing AstroPay cash methods before redirect - https://phabricator.wikimedia.org/T132956#2215169 (awight) [20:14:05] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, MediaWiki-Vagrant: assert statements in latest Civi break wmf_civicrm install in vagrant - https://phabricator.wikimedia.org/T132810#2219969 (awight) Noting that mediawiki-vagrant uses HHVM as its PHP engine, probably related. [20:15:28] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, MediaWiki-Vagrant: assert statements in latest Civi break wmf_civicrm install in vagrant - https://phabricator.wikimedia.org/T132810#2219979 (awight) We should probably commit the workaround @ejegg suggests, for now. [20:15:40] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, MediaWiki-Vagrant: assert statements in latest Civi break wmf_civicrm install in vagrant - https://phabricator.wikimedia.org/T132810#2219980 (Ejegg) Maybe HHVM incompat? [20:22:16] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, I18n: Allow CentralNotice banner variables to be marked as non translated - https://phabricator.wikimedia.org/T53470#2220017 (awight) To clarify why I set T116235 as a dependency, if we managed to migrate banners to translatable pages, then thi... [20:23:00] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Easy: Add CentralNotice keyboard shortcuts - https://phabricator.wikimedia.org/T132731#2220023 (Ejegg) @Jseddon Any opinion on this and how it relates to the rest of the hoped-for UI refresh? [20:27:31] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Easy: Make CentralNotice banner completely believe it to be in a certain country - https://phabricator.wikimedia.org/T111071#2220046 (awight) [20:37:05] Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog: SPRINT H (4/13-4/27) GOAL: Adyen for new countries (Israel, Ukraine and more!) - https://phabricator.wikimedia.org/T131161#2220079 (awight) [20:46:49] Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog, FR-Adyen: Adyen form for French (France), Japanese (Japan), Ukrainian (Ukraine), & Hebrew (Israel) - https://phabricator.wikimedia.org/T128812#2220153 (XenoRyet) a:XenoRyet [20:51:20] sorry awight|sequester your last comment popped up just as I was closing the session [20:51:33] & I didn't manage to read it [20:52:25] eileen: I was saying that fr-all is protective of their keying vendor, who is on Eastern time [20:52:43] If we do an outage during Eastern business day, we should give like a week's notice. [20:53:07] ok - so Jeff_Green said he was flexible about timing if he has notice [20:53:22] so you are suggesting we aim for like 5.30 or 6 over there? [20:53:27] cool. Yeah I think our Civi users are very flexible as well [20:53:35] eileen: If that works for Jeff, totally. [20:53:38] oh I do have one blackout time which is 8:30-11PM EST on thursday nights [20:53:52] I feel like it's better if I send an email suggesting a time [20:53:55] than if I ask [20:54:45] Jeff_Green: anything after your work hours is good for me due to tz [20:54:56] so - do you want to suggest a time & I'll email it out [20:55:00] Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog, Unplanned-Sprint-Work: Astropay MX form: minimum amount error - https://phabricator.wikimedia.org/T132959#2215283 (Ejegg) a:Ejegg [20:55:07] awight: who is the group to email? [20:55:39] eileen: how about mon-weds between 7-11PM EST? [20:56:00] next week or the week after is fine for me [20:56:46] eileen: fr-all [20:57:13] Jeff_Green: http://www.timeanddate.com/worldclock/meetingdetails.html?year=2016&month=4&day=26&hour=23&min=0&sec=0&p1=2099&p2=179&p3=224 [20:57:27] and then if all looks good the long outage for the same time a week later? [20:57:59] eileen: oh I don't mean to say we need an outage of that duration, it really seems like a 1 hr max doesn't it? [20:58:19] Jeff_Green: so next weeks one is only to do one table [20:58:31] oh i see now [20:58:35] & yeah an hour seems very conservative [20:58:41] cwd: Have you used some of these strategies with postgres? [20:58:43] but, the 'full' one [20:58:52] right [20:58:56] will take a long time - ie. about 2.5 hours on staging [20:59:41] awight: yeah we had master/slave replication [20:59:45] and manual promotion [21:00:05] k. That's the leading candidate for our Redis "HA" as well. [21:00:09] yeah [21:00:17] it looks like you can do all kinds of things though [21:00:46] async multimaster might be worth looking into [21:02:11] shared disk sounds fun, as long as it's not the disk that failed [21:03:13] block device replication might be something to look at too [21:03:49] Fundraising Sprint Freshmaking, Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog, fundraising-tech-ops, and 2 others: Spike: Investigate suitability of Kafka instead of Redis - https://phabricator.wikimedia.org/T130304#2220334 (awight) @cwdent is throwing Postgres into the ring... w... [21:05:14] awight: the downside of postgres is probably that it has a lot of features we don't really need [21:05:17] cwd: You do diagrams? I think I'll review the interfaces we need, with the Kafka paradigm in mind, and how to emulate on various backends [21:05:22] being an rdb [21:05:28] cwd: The other down side is that we would be a snowflake at WMF [21:05:45] yeah [21:05:54] that's sort of surprising [21:06:23] Actually... someone's using it: https://github.com/wikimedia/operations-puppet/tree/production/modules/postgresql [21:06:52] nice [21:07:08] * Jeff_Green heading out, have a good day/morning/evening folks [21:07:42] Looks like it's a dependency of lots of maps components. [21:07:47] Makes sense: needs real db [21:07:51] So that's good news. [21:07:58] also has native GIS data types [21:08:04] fancy [21:08:08] so fancy [21:08:38] so what key/val big data stores have over it is huge write speed right? [21:08:57] Speed is not an issue for us [21:09:04] that's kinda what i was thinking [21:09:10] i doubt pgsql would ever be the bottle neck [21:09:24] Mostly, I was interested in the great data structure support, which makes our emulation layer really thin [21:09:44] postgres definitely has that, as well as a wonderful sql [21:09:50] But objectively, I think stability and availability are the things we care about most. [21:10:29] i can't see a good reason to give up all the things a good sql delivers unless you can't afford them because of speed [21:11:21] if i was querying the db instead of fucking aroudn with the activemq console i'd be a more effective person [21:11:54] I guess another thing that was on my mind, is that SQL feels more prone to attacks [21:12:17] whereas I'm not aware of any type of injected query for Redis [21:12:21] or Kafka [21:12:30] I don't know if that's a real thing, though. [21:13:03] PDO prepared statements are pretty damn good [21:13:19] +1 [21:13:24] it has to be one of the least shitty things about php [21:13:38] but it just takes one weak-link client [21:14:18] i wonder if sql is actual more vulnerable or just more ubiquitous [21:14:25] *ly [21:14:40] dang Jeff_green just left - I wanted him to check my proposed sched email was what he thought he was agreeing to! [21:14:58] Maybe schedule it and ask for forgiveness ;) [21:15:04] :-) [21:15:19] well - I sent it to him & you so you can check it looks like what he agreed to :-) [21:16:18] hehehe [21:16:51] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-ActiveMQ, FR-Smashpig: [Epic] Rewrite all queue clients to use a single shim library, improve library - https://phabricator.wikimedia.org/T133108#2220402 (awight) [21:18:48] awight: i'm not suggesting we use this but i wrote a stupid simple queue in postgres that was ridiculously fast: https://github.com/caseydentinger/glerk [21:19:33] it has listen and notify commands so you can push updates about new data [21:19:33] Fundraising-Backlog, FR-ActiveMQ: fundraising-tools queue operations should use a queue abstraction - https://phabricator.wikimedia.org/T130308#2220429 (awight) [21:19:40] built right into postgres [21:19:45] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, MediaWiki-extensions-DonationInterface, FR-ActiveMQ: Migrate contribution tracking to new queue - https://phabricator.wikimedia.org/T131279#2220448 (awight) [21:19:47] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops, MediaWiki-extensions-DonationInterface, and 2 others: Migrate pending to new queue and finish cleanup - https://phabricator.wikimedia.org/T131274#2220451 (awight) [21:19:50] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, MediaWiki-extensions-DonationInterface, FR-ActiveMQ, FR-Smashpig: Migrate donations to new queue - https://phabricator.wikimedia.org/T131277#2220450 (awight) [21:19:54] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-ActiveMQ: Migrate fredge to new queue - https://phabricator.wikimedia.org/T131273#2220452 (awight) [21:19:56] Fundraising-Backlog, FR-ActiveMQ: Migrate banner impressions to new queue and rewrite - https://phabricator.wikimedia.org/T131278#2220449 (awight) [21:19:58] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-ActiveMQ: CRM consumers should use the PHPQueue library - https://phabricator.wikimedia.org/T112832#2220456 (awight) [21:19:59] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, MediaWiki-extensions-DonationInterface, FR-ActiveMQ, FR-Smashpig: Consolidate queue abstractions - https://phabricator.wikimedia.org/T131271#2220453 (awight) [21:20:03] Fundraising-Backlog, FR-ActiveMQ: fundraising-tools queue operations should use a queue abstraction - https://phabricator.wikimedia.org/T130308#2220455 (awight) [21:20:05] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-ActiveMQ, FR-Smashpig: [Epic] Rewrite all queue clients to use a single shim library, improve library - https://phabricator.wikimedia.org/T133108#2220447 (awight) [21:20:06] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, MediaWiki-extensions-DonationInterface, Epic, and 3 others: [Epic] Clean up "pending" queue usages - https://phabricator.wikimedia.org/T130897#2220454 (awight) [21:20:10] Fundraising-Backlog, FR-ActiveMQ, FR-Smashpig: Create SmashPig PhpQueueDataStore compatible with StompDataStore - https://phabricator.wikimedia.org/T129386#2220458 (awight) [21:20:12] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-ActiveMQ, FR-Smashpig: [Epic] Rewrite all queue clients to use a single shim library, improve library - https://phabricator.wikimedia.org/T133108#2220402 (awight) [21:21:03] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, MediaWiki-extensions-DonationInterface, FR-ActiveMQ: Migrate contribution tracking to new queue - https://phabricator.wikimedia.org/T131279#2161951 (awight) [21:21:05] Fundraising-Backlog, FR-ActiveMQ: Migrate banner impressions to new queue and rewrite - https://phabricator.wikimedia.org/T131278#2161935 (awight) [21:21:06] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, MediaWiki-extensions-DonationInterface, FR-ActiveMQ, FR-Smashpig: Migrate donations to new queue - https://phabricator.wikimedia.org/T131277#2161921 (awight) [21:21:09] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops, MediaWiki-extensions-DonationInterface, and 2 others: Migrate pending to new queue and finish cleanup - https://phabricator.wikimedia.org/T131274#2161875 (awight) [21:21:12] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, MediaWiki-extensions-DonationInterface, Epic, and 3 others: [Epic] Clean up "pending" queue usages - https://phabricator.wikimedia.org/T130897#2149972 (awight) [21:21:15] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, MediaWiki-extensions-DonationInterface, FR-ActiveMQ, FR-Smashpig: Consolidate queue abstractions - https://phabricator.wikimedia.org/T131271#2161797 (awight) [21:21:19] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-ActiveMQ: Migrate fredge to new queue - https://phabricator.wikimedia.org/T131273#2161859 (awight) [21:21:21] Fundraising-Backlog, FR-ActiveMQ, FR-Smashpig: Create SmashPig PhpQueueDataStore compatible with StompDataStore - https://phabricator.wikimedia.org/T129386#2104049 (awight) [21:21:23] Fundraising-Backlog, FR-ActiveMQ: fundraising-tools queue operations should use a queue abstraction - https://phabricator.wikimedia.org/T130308#2132120 (awight) [21:21:25] Back in a little while! [21:21:25] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-ActiveMQ: CRM consumers should use the PHPQueue library - https://phabricator.wikimedia.org/T112832#1647403 (awight) [21:21:26] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops, MediaWiki-extensions-DonationInterface, and 2 others: [Epic] SPOF: Replace ActiveMQ donation queues with a more robust software stack - https://phabricator.wikimedia.org/T108229#2220459 (awight) [21:21:30] Fundraising-Backlog, FR-ActiveMQ: Write specialized delay queue and handlers - https://phabricator.wikimedia.org/T131282#2220472 (awight) [21:21:32] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-ActiveMQ, FR-Smashpig: [Epic] Rewrite all queue clients to use a single shim library, improve library - https://phabricator.wikimedia.org/T133108#2220402 (awight) [21:21:34] Fundraising-Backlog, FR-ActiveMQ: Write specialized delay queue and handlers - https://phabricator.wikimedia.org/T131282#2162012 (awight) [21:22:18] cwd: That's nice! 135 whopping lines of code ;) [21:22:53] yeah like i say stupid simple "queue" layer [21:23:12] I guess that's the next real step--figure out exactly what our shim layer should look like. [21:23:24] I'm not happy with the jumbled interfaces I added to PHP-Queue [21:23:41] you could rewrite this in php, conceptually [21:23:44] using listen/notify [21:23:51] if go isn't your thing [21:23:53] https://github.com/adamwight/php-queue/blob/master/src/PHPQueue/Interfaces/FifoQueueStore.php https://github.com/adamwight/php-queue/blob/master/src/PHPQueue/Interfaces/IndexedFifoQueueStore.php https://github.com/adamwight/php-queue/blob/master/src/PHPQueue/Interfaces/KeyValueStore.php [21:23:54] and by you i mean who ever [21:24:15] so, what's the listen/notify? Filesystem? [21:24:35] it's a postgres built in [21:24:50] client calls listen and it leaves teh socket open [21:25:03] whoa [21:25:07] when postgres calls "notify " the client can do something [21:25:27] so you could have the app code call notify when it adds a row [21:25:33] ...or a table trigger [21:25:36] * cwd ducks [21:26:13] I think the FIFO is really simple--it's just a table with an autoincrement column, and consumers each have their own "group" offset for reading in sequence [21:26:25] Probably doesn't require any triggers or code [21:26:43] indexed fifo is an abomination, I think we can get rid of it [21:27:14] yep it's probably not a real limitation but you can get rid of polling [21:27:14] Those should just be a table with two indexes, on timestamp and on transaction id [21:27:28] Ah I see, you mean in the consumers [21:27:30] Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog, Unplanned-Sprint-Work: Astropay MX form: minimum amount error - https://phabricator.wikimedia.org/T132959#2220481 (Ejegg) Yep, the backend has different values :( We've got a $1.50 minimum in place for AstroPay, and we're not accounting... [21:27:41] I've been wondering about that [21:27:44] yep, they can just wait for notify [21:28:17] I'm not sure what the consumers would look like in an ideal world [21:28:18] and fwiw i am not the first person who has done this :) [21:29:11] btw, https://github.com/adamwight/php-queue/blob/master/src/PHPQueue/Backend/PDO.php [21:29:28] nice [21:31:44] I like the idea of consumers which don't even need to batch. They're just online and listening to events. [21:33:04] I also like that we can turn them off [21:33:23] awight: another question, do we actually need multiple consumers per queue? [21:33:30] if speed is not our concern [21:33:43] and in a better world, our jobs are idempotent and we can re-run on the day's messages [21:33:47] I donno... [21:33:54] I started coming up with nice things that gets us [21:34:04] but the offset paradigm is also nice for transactionality. [21:34:20] parallel workers add so much potential complexity [21:34:23] If there's no such thing as pop(), we're free to restart a fatally crashed job [21:34:28] yeah [21:34:35] I don't think it adds complexity, if it's transparent. [21:34:51] There's no "read N times" rule, cos each consumer manages its own incoming queue [21:34:51] yeah, if they never crash into each other [21:34:54] yep [21:35:20] so here was the argument i had about this: [21:35:23] k [21:35:42] https://github.com/caseydentinger/glerk/blob/master/glerk.go#L58 [21:35:59] one of the admins absolutely insisted we run parallel binaries of this [21:36:11] even though you really don't get anything out of it [21:36:24] go "threads" internally [21:36:31] and shells out these phps that won't block each other [21:36:44] well. we don't need that, but if we did it would certainly add another level of locking [21:37:04] I'm thinking more, consumers with orthogonal functionality [21:37:12] oh, sure [21:37:20] running on different queues [21:37:21] so one consumer is pushing things into Civi, and another is extracting statistical stuff [21:37:25] totally [21:37:26] or on the same one, even [21:37:45] if it has different job types or somethign? [21:37:53] could be the same records, even! [21:38:05] and it shells out several phps for instance? [21:38:55] For example, contribution_tracking would become a feed of all of our interactions with a donor and with their transaction. One job maintains a table in drupal.ct which gives us the current status of a transaction, another job creates rollup statistics about interaction timing and stuff [21:39:11] and a third monitors payment gateway health [21:39:32] ah ha, so each of those consumers would at some point look at the same record [21:39:37] luckily--I think the offset/persistence thing gives us that ability, as a side effect of just legit persistence [21:39:42] yeah [21:40:31] that NOTIFY keyword will notify everyone listening to a thing, so as long as none of them change the row in such a way that the other consumers would ignore it, they'd all get their chance [21:41:19] notify seems fragile--what if the consumer is offline. [21:42:51] awight: https://github.com/caseydentinger/glerk/blob/master/glerk.go#L135 [21:43:05] when it comes on line it clears out everything before listening [21:43:22] thanks awight [21:43:36] Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog, Unplanned-Sprint-Work, WMF FR: Astropay MX form: minimum amount error - https://phabricator.wikimedia.org/T132959#2220558 (Ppena) [21:43:36] cwd: race condition :p [21:43:50] how come? [21:43:52] eileen: If that's helpful! [21:43:53] Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog, Unplanned-Sprint-Work, WMF FR: Astropay MX form: minimum amount error - https://phabricator.wikimedia.org/T132959#2215283 (Ppena) Thanks @Ejegg for confirming we can pick a minimum depending on the PSP. I didn't know that @jrobell wo... [21:44:15] cwd: if something is added between work() and listen(), we miss it [21:44:18] nbd [21:44:45] hmm, i think it should still get found? [21:44:48] during the work loop? [21:45:37] cwd: not if the record is added after we call "begin" on an otherwise empty iteration [21:45:51] or if it comes in after the "commit" [21:46:18] etc. my point though is that this scheme is complicated by having two distinct mechanisms [21:46:19] well it won't write to the table in the middle of an exclusive lock like that [21:46:51] no, the writing process will wait, then insert exactly in the race interval [21:48:03] um. anyway, I don't have much of a point. listen is great and something like that might be really nice [21:48:08] it could happen [21:48:27] it'd pick them both up on the next run or 90s later, which ever is shorter [21:49:04] I like the idea that we could get almost zero latency for all of our pipelines. [21:49:28] and reduce total kilowatt hours [21:49:31] Optimizing batch processing for maximum load isn't great for everyday stuff [21:49:38] Fundraising Sprint Elevator Maintenance 2016, Fundraising Sprint Freshmaking, Fundraising Sprint Ghostbusting , Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog: Civi: batch import of refunds, problem transaction: currency switched - https://phabricator.wikimedia.org/T127979#2220594... [21:50:22] cwd: So I made a first pass on splitting out the first phase of cleanup work, https://phabricator.wikimedia.org/T133108 [21:50:30] Fundraising Sprint Elevator Maintenance 2016, Fundraising Sprint Freshmaking, Fundraising Sprint Ghostbusting , Fundraising Sprint Hermit Crab Husbandry, Fundraising-Backlog: Civi: batch import of refunds, problem transaction: currency switched - https://phabricator.wikimedia.org/T127979#2220599... [21:51:02] awight: python interacts with the queue? [21:51:06] :( [21:51:20] Once upon a time, I thought I would be able to make that our workhorse [21:51:36] but yeah. c.f. https://github.com/wikimedia/wikimedia-fundraising-tools/tree/master/queue [21:51:38] a noble effort [21:52:11] Fundraising-Backlog, FR-ActiveMQ: Write PHP-Queue module for Kafka - https://phabricator.wikimedia.org/T131269#2220604 (awight) [21:52:34] Fundraising-Backlog, FR-ActiveMQ: Pick a PHP Kafka client library - https://phabricator.wikimedia.org/T131268#2220606 (awight) [21:53:03] cwd: I moved those two things out of the sprint, so there's breathing room to pull in some of this cleanup. Feel free to pick a fun one! [21:53:24] (or none :) [21:54:05] awight: i'm just going to play devil's advocate for a second [21:54:05] I'm considering an early commute, the office is boring. Let's chat more tomorrow... [21:54:10] please do :D [21:54:46] what if we move everything to a new queue wrapper, then actually move the first queue to a new back end, and the queue wrapper is not helpful [21:55:02] I did already move DI to the new wrapper [21:55:06] another way to say that is [21:55:16] and it made it possible to write to both activemq, memcache, and redis [21:55:24] what if the 2nd step, actually moving the queue, would inform our decisions about the queue wrapper [21:55:46] In my imagination, actually moving the queue will be a no-op [21:55:57] although I plan to do it often during development. [21:55:58] that's definitely the ideal reality [21:56:22] I think, we write solid test suites for each module in php-queue [21:56:23] like what if we built it to support those and then wanted to do sql and thought dang i would have done the wrapper differently [21:56:37] yeah, that will happen [21:56:52] I think you had a helpful aphorism about wrappers around data stores... [21:57:31] which was that? i talk a lot [21:57:36] I'm trying to remember. [21:57:39] hehe [21:57:58] "it's only as good as the shittiest back end" [21:58:08] Something like, a generalized library for any data stores is a way to get rid of any of the useful features. [21:58:11] yeah. [21:58:12] more elegant. [21:58:25] that's how i've begun to feel about a lot of things [21:58:31] like code review systems that support svn [21:58:47] I had that feeling many times when doing the PHP-Queue work in the first place [21:58:50] however. [21:59:02] The silver lining is that we actually *want* to use the minimum possible features [21:59:16] yeah agreed [21:59:34] cos we will eventually hate our queue backend and want to decouple. plus, open sourceness requires not coupling to some ridiculous entire environment [21:59:35] then if down the road we want to expand into the internals of whatever store [21:59:38] we can do that then [21:59:49] It's a huge PITA, unfortunately. [22:00:08] hehe yeah, like running mw on psql [22:00:10] I'll try to take some notes about the interfaces we need and regroup with you tomorrow... [22:00:13] gah [22:00:21] Fundraising-Backlog, Hovercards, Reading-Web-Sprint-72-N: Avoid z-index conflict with HoverCards & Central Notice and friends - https://phabricator.wikimedia.org/T131364#2220697 (dr0ptp4kt) [22:00:29] cool yeah, safe travels [22:00:37] i'll be caulking [22:00:40] hehe be back online soon. Thanks! [22:00:45] I said ha ha [22:01:13] I've grown to love that stuff. After discovering that there is caulk you can use to make gaskets for an engine [22:01:24] 100% silicone? [22:01:46] permatex (ultra black) [22:01:48] all good. [22:01:59] Apparently it's recommended over the original gaskets. [22:02:18] awesome, and one size fits all [22:02:23] yes! [22:02:52] well good luck hosing off after your work party. [22:03:00] :) [22:03:07] talk to you soon [22:05:18] hmm I love the way I forgot to update the subject line of my email [22:08:36] (PS1) Ejegg: Account for gateway-specific min/max client-side [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/284366 (https://phabricator.wikimedia.org/T132959) [22:10:18] (PS1) Ejegg: Turn on client-side amount validation for new forms [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/284367 (https://phabricator.wikimedia.org/T132959) [22:11:52] (CR) jenkins-bot: [V: -1] Turn on client-side amount validation for new forms [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/284367 (https://phabricator.wikimedia.org/T132959) (owner: Ejegg) [22:16:59] heading to the local civi meetup [22:17:14] haven't been there for months [22:17:23] ejegg|afk: cool! [22:17:34] who goes to it? [22:17:50] it's hosted at the FSF [22:18:19] ah right - those guys pop up on IRC from time to time [22:18:26] and the participants are mostly nonprofit civi admins [22:18:47] nice [22:18:51] with a couple of ppl that do xontract work setting up civi [22:19:29] k, hopping on my bike! [22:19:34] :-) [22:25:38] oh awesome, say hi to rms