[01:23:08] crud, what's with that failmail? [01:28:52] fr-tech anyone else around? [01:31:17] !log disabled fundraising queue consumer jobs [01:31:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:01:49] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Rash of errors consuming donations queue - https://phabricator.wikimedia.org/T192977#4156099 (Ejegg) [02:20:32] yowza, the refund queue has 83,500 items? [02:22:25] ok... that queue consumer needs to start running overtime [02:23:14] and 83k should definitely trigger an alert [02:52:07] eggre: I'm here-ish [02:52:43] !log turned fundraising queue consumers back on [02:52:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:53:36] AndyRussG: thanks! The first problem seems to be transitory - the donations make it back in if you requeue them and reimport [02:54:05] the other one might be related to the recurring donation refund / audit bug that mepps just fixed [02:54:14] but we haven't deployed that fix yet [02:54:39] anyway, I'mma let those queue consumers burn through what's queued, then requeue the ones that didn't make it [02:54:44] gonna eat for now [03:26:43] !log disabled ingenico recurring donation charge job [03:26:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:28:43] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Rash of errors consuming donations queue - https://phabricator.wikimedia.org/T192977#4156119 (Ejegg) p:Triage>Unbreak! Dang, this is also affecting the Ingenico recurring charge jobs. Those might be more serious - I'm afraid that if we fail inse... [03:31:47] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Rash of errors consuming donations queue - https://phabricator.wikimedia.org/T192977#4156122 (Ejegg) Weird, those two users with anime avatars subscribe to all the Unbreak Now tasks? [03:41:01] Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: Ingenico audit wobble? March 13 re... - https://phabricator.wikimedia.org/T190098#4062989 [03:46:29] Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: Ingenico audit wobble? March 13 re... - https://phabricator.wikimedia.org/T190098#4156128 [04:34:05] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Ingenico, Regression: Possible Globalcollect audit parsing regression - https://phabricator.wikimedia.org/T127111#4156134 (Ejegg) Open>declined Related to old audit parser, now using WX parser [05:01:27] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Provision PSP maintenance scripts to the CRM box - https://phabricator.wikimedia.org/T142659#4156144 (Ejegg) Open>Resolved a:Ejegg We seem to have settled on doing this on payments1004 [06:10:13] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 5 keys, up 71 days 5 hours - memory use is 1.86M (peak 83.16M, 0.07% of max, fragmentation 3.04%), connected_slaves is 2, donations is 1, jobs is 0, jobs-adyen is 1, jobs-paypal is 3, payments-antifraud is 0, payments-init is 0, pending is 0, recurring is 0, refund is 33, unsubscribe is 2 [09:37:52] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Rash of errors consuming donations queue - https://phabricator.wikimedia.org/T192977#4156099 (Eileenmcnaughton) This seems to relate to deadlocks - it's possible there was a notable db event occuring - but it's also possible the triggers on wmf_contribu... [09:40:10] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Rash of errors consuming donations queue - https://phabricator.wikimedia.org/T192977#4156693 (Eileenmcnaughton) Also - I use the practice in general in CiviCRM of creating a pending contribution before attempting to make a payment & then updating it whe... [13:53:50] fundraising-tech-ops, Operations, ops-eqiad: rack frbast1001 - https://phabricator.wikimedia.org/T187363#4157629 (Jgreen) No display output after the host started pxeboot sequence, turns out it needed "Redirection After Boot" enabled in BIOS. [14:01:19] Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Dinosaur Cookies co-existed with Gingerbread People, Fundraising Sprint Elevators were never intended to go down, Fundraising Sprint Fhabricator is spelled with an "F", and... - https://phabricator.wikimedia.org/T185933#4157651 [14:32:23] jgleeson and mepps: I think the new triggers eileen added are the cause of the initial rash of failmail [14:32:44] then there's the refund stuff, which I think is an unrelated problem [14:32:54] mepps, want to update SmashPig in CRM and deploy that? [14:36:42] thanks ejegg and mepps [14:37:48] jgleeson and mepps: so, when we have a campaign in NL, most of the donations come in via bank transfer (the iDEAL system) [14:37:50] cwd: Jeff_Green: fr-tech what happens to the files for Landing Page/sampled banner impressions once they're processed? Do they crash and burn or land in some long-term storage vault? [14:38:00] AndyRussG: long term storage [14:38:16] ejegg: okok thx [14:38:34] ejegg: so maintaining that is also a requirement for the new system, right? [14:39:04] jgleeson and mepps so, that bank transfer is an async donation, where we don't get a notification of payment up front [14:39:25] Even though it'll be the same data, analytics may want to doube-check legal and privacy implications [14:39:31] AndyRussG: i don't think we've ever really had the discussion about why and how long to store those files [14:39:47] AndyRussG: It would be nice to have, but if we can get the data from Kafka for a good long time after, that might be good enough [14:40:06] afaik kafka doesn't store long term [14:40:11] Jeff_Green: maybe someone had it once? we should conduct some oral history interviews [14:40:21] we just want to be able to go back and fix stuff from a few months ago if we realize we had a problem [14:40:23] Yeah yesterday I asked on analytics, for EL topics it's 8 days [14:40:36] Elsewhere it's stored 90 days [14:40:38] AndyRussG: the fact that we store them at all is probably my fault, I saw data and thought it was bad to drop it on the floor [14:40:46] :0 [14:40:48] :) [14:40:58] Well, broken data is a pain to sweep up [14:41:04] true [14:41:28] jgleeson and mepps: so what that async-ness means is that ALL the iDEAL donations come in via the audit files [14:41:48] put it in the closed is usually fine unless it piles up in front of the holiday decorations and can't get at them in 8 months [14:41:52] closet [14:42:04] jgleeson and mepps: and since we had the jobs off during the normal audit timeslot the night of the CRM custom fields deploy [14:42:40] jgleeson and mepps: last night was the first audit processed with the new custom fields and triggers, and the first real stress test [14:43:51] stress test, as in, the donations queue got up to 18,300 or so [14:45:01] so, there's som mysterious horrible thing going on with how Civi works with transactions [14:45:44] AndyRussG: I'm all for having us clearly define what's useful/needed to store as part of the revamp. [14:45:52] yeah [14:46:12] ejegg: Jeff_Green: I think possible backfill and long-term storage are requirements we need to define before fully designing the new system then [14:46:42] jgleeson and mepps: seems like somewhere something is getting rolled back after inserting the contact, but WITHOUT throwing an exception that stops us trying to insert the next stuff [14:46:42] is there talk of changing the db schema? [14:46:50] Jeff_Green: not yet [14:46:53] k [14:47:00] for now it has to be 100% backwards compatible [14:47:10] We will need a parallel DB for testing [14:47:17] cool [14:47:31] maybe we can give it a modern name and keep the new one [14:47:45] kafka-python consumer to directly ingress into the db as the kafka messages arrive looks nice, and is probably more stable than the current system, would benefit from being a system that others use and so we have that upstream maintenance support [14:48:05] however if we need infinite backfill ability that may make it less desireable [14:48:33] Just about to summarize the issue here: T192839 [14:48:33] T192839: [Spike] Plan ingress system to use with new Kafka topic for landing page and impression data - https://phabricator.wikimedia.org/T192839 [14:48:37] we could probably make the kafkatee host do both things [14:49:35] i.e. have kafka-python write files at the same time [14:49:41] Jeff_Green: right.. As in, keep writing the files, but normally just ingress directly from kafka via python-kafka (https://github.com/dpkp/kafka-python) [14:49:48] sorry kafka-python [14:49:49] ya [14:49:53] kafka-python-python [14:49:58] python-kafka-ka [14:50:11] but is that what we really want? [14:50:39] to me, direct db injection doesn't sound like the most reliable option so I don't think it's a ~bad~ idea [14:51:12] hmm right... I mean, we'd have to wait for a certain number of message to arrive, and then insert [14:51:17] but it does make for some potential snarls to figure out, i.e. what happens if the db goes away for a minute [14:51:27] yup [14:51:27] can u be more specific about what might be bad about it? [14:51:47] kafka lets you start reading from any point [14:52:34] still, I mean, if we're gonna keep writing the files anyway, is there a good argument for not just always slurping from then, instead of having two possible ingress routes (direct from kafka for normal processing, and file slurping for backfill)? [14:52:48] I guess it depends how we architect it. My first thought is that fetching kafka on one host and inserting to a db on another host raises the possibility of the latter part falling over for some reason. [14:53:23] Jeff_Green: yeah... I mean, I think that's what Kafka is for, so there must be a standard setup for that [14:53:34] But on the other hand, we already do have a failsafe for that [14:53:46] in the way of the files and the script that can do backfills easily [14:53:56] that's great [14:54:44] and kafka stores long enought that we should have time to do db repairs before we start missing the opportunity to recover lost db data [14:55:03] (as long as we notice in time) [14:55:19] isnt stuff stored long term in hive? [14:55:26] or is that aggregated? [14:55:56] it's much longer term yeah, not sure how far back it goes [14:57:40] is there a web front end or something for that, like a graphing program? [14:58:07] analytics has some tools, I'm not at all familiar with them, what I've used is sql-ish [14:58:42] same here [14:58:53] "pivot" ? [15:03:22] ugh netsplit [15:04:20] The kafka messages themselves stick around 8 days... so if we ingress directly from kafka, at least as afar as I understand, that'd be our easy backfill window [15:04:31] Jeff_Green: cwd: ^ [15:05:14] cwd: there is a graphical tool (pivot) but not exactly for the same data in the case of banner impressions, and there's nothing for landing page impressions [15:05:35] right, which is a little slim imo, I can easily imagine a scenario where we don't notice a subtle issue for weeks or more [15:05:36] gotcha [15:05:43] yep [15:06:19] are the blockers to getting pivot useful to FR terrible? [15:07:25] sure would be nice to use the same tool [15:09:39] Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: [Spike] Plan ingress system to use with new Kafka topic for landing page and impression dat... - https://phabricator.wikimedia.org/T192839#4157884 [15:10:56] Fundraising-Backlog, FR-PayPal-ExpressCheckout: Paypal EC orphan rectifier fails with validation failure - https://phabricator.wikimedia.org/T191654#4157894 (mepps) a:mepps [15:15:38] https://community.toadworld.com/platforms/oracle/w/wiki/11647.streaming-kafka-messages-to-mysql-database [15:15:47] cwd we would have to put all the civi data in pivot [15:16:08] cwd: also fr-online needs backwards compatibility with the current workflow [15:16:15] the whole point of pulling the banner data onto frack is to correlate and jonin [15:16:38] one idea for the future is to start to store more data in Druid which is the back-end that supports Pivot [15:17:06] and then maybe we could create parallel tools with the same interface, but that pull data from Druid instead of MySQL [15:18:00] and then eventually start to provide some graphical Pivot-y tools to see if fr-online might prefer those to the current front-end (command line tools infact) [15:18:10] Druid is fantastic btw [15:18:33] but Pivot has license issues, i.e. the authors are moving to closed-source for future versions, IIRC [15:19:02] Druid can be queried in a bunch of ways, we do so currently from python notebooks [15:19:31] and it optimizes for aggreating queries (that you have to plan for) [15:20:19] it's basically a store for partial pre-processed aggregate queries on Hive data now [15:20:49] https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [15:24:27] (PS1) Mepps: Log error and move to damaged on ValidationException in orphan rectifier, but continue Bug: T191654 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/428940 (https://phabricator.wikimedia.org/T191654) [15:26:30] (CR) jerkins-bot: [V: -1] Log error and move to damaged on ValidationException in orphan rectifier, but continue Bug: T191654 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/428940 (https://phabricator.wikimedia.org/T191654) (owner: Mepps) [15:34:45] Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: [Spike] Plan ingress system to use with new Kafka topic for landing page and impression dat... - https://phabricator.wikimedia.org/T192839#4158091 [15:35:32] Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog, and 3 others: Option to enforce CS... - https://phabricator.wikimedia.org/T190100#4158094 [15:47:34] Fundraising Sprint Fhabricator is spelled with an "F", Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, ... - https://phabricator.wikimedia.org/T189613#4158189 [15:47:38] (PS1) Mepps: Update smashpig, and other composer dependencies [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/428943 [15:47:40] Fundraising-Backlog: Adapt Druid banenr_activity jobs to EventLogging-based impression recording - https://phabricator.wikimedia.org/T186048#4158190 (AndyRussG) [15:47:43] Fundraising-Backlog: centralnotice_analytics: adapt ImpressionsQuery for EventLogging-based impressions recording - https://phabricator.wikimedia.org/T186047#4158191 (AndyRussG) [15:47:49] Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Dinosaur Cookies co-existed with Gingerbread People, Fundraising Sprint Elevators were never intended to go down, Fundraising Sprint Fhabricator is spelled with an "F", and... - https://phabricator.wikimedia.org/T185933#4158192 [15:47:50] Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Dinosaur Cookies co-existed with Gingerbread People, Fundraising Sprint Elevators were never intended to go down, Fundraising Sprint Fhabricator is spelled with an "F", and... - https://phabricator.wikimedia.org/T185932#4158193 [15:47:52] Fundraising Sprint Asymmetrical Earth Theory, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising-Backlog, Epic: [Epic] Kafkatee changes - https://phabricator.wikimedia.org/T183978#4158187 (AndyRussG) [15:50:53] Fundraising Sprint Fhabricator is spelled with an "F", Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, ... - https://phabricator.wikimedia.org/T189613#4158217 [15:52:14] Fundraising Sprint Fhabricator is spelled with an "F", Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, ... - https://phabricator.wikimedia.org/T189613#4158227 [15:52:52] Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: [Spike] Plan ingress system to use with new Kafka topic for landing page and impression dat... - https://phabricator.wikimedia.org/T192839#4158239 [15:56:54] (PS1) Jgleeson: WIP - Added manual test cli script to test out creating an ingenico payment against an existing recurring token [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/428945 [15:57:35] (CR) jerkins-bot: [V: -1] WIP - Added manual test cli script to test out creating an ingenico payment against an existing recurring token [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/428945 (owner: Jgleeson) [15:57:57] (CR) Ejegg: [C: 2] Update smashpig, and other composer dependencies [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/428943 (owner: Mepps) [15:58:07] (PS2) Jgleeson: WIP - Added manual test cli script to test out creating an ingenico payment against an existing recurring token [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/428945 [15:58:37] (CR) jerkins-bot: [V: -1] WIP - Added manual test cli script to test out creating an ingenico payment against an existing recurring token [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/428945 (owner: Jgleeson) [15:59:38] (PS3) Jgleeson: WIP - Added manual test cli script to test out creating an ingenico payment against an existing recurring token [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/428945 [16:01:21] Fundraising Sprint Fhabricator is spelled with an "F", Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, ... - https://phabricator.wikimedia.org/T189613#4158323 [16:02:26] (Merged) jenkins-bot: Update smashpig, and other composer dependencies [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/428943 (owner: Mepps) [16:12:42] (PS1) Mepps: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 [16:13:03] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 (owner: Mepps) [16:13:40] (CR) Mepps: "recheck" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 (owner: Mepps) [16:14:00] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 (owner: Mepps) [16:29:09] mepps oh, we need to actually update vendor ourselves [16:29:32] c+2 that, and add the submodule update to the deploy merge [16:34:27] (CR) Mepps: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 (owner: Mepps) [16:34:48] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 (owner: Mepps) [16:35:26] (PS2) Mepps: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 [16:35:39] (CR) Mepps: [C: 2] "recheck" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 (owner: Mepps) [16:35:49] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 (owner: Mepps) [16:36:00] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 (owner: Mepps) [16:36:02] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 (owner: Mepps) [16:41:04] fr-tech sorry I got cut off at the end of standup. my phone died [16:42:11] dstrine: no worries [16:42:36] erg ejegg i i updated the vendor directory but still no luck, feeling at a loss [16:44:59] mepps I don't see anything for the crm/vendor repo [16:45:18] (PS3) Mepps: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 [16:45:20] that commit needs to be +2ed and merge [16:45:35] then you can do a submodule update as part of the commit in the crm repo [16:45:48] the failing bulid does? [16:45:52] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 (owner: Mepps) [16:45:59] no, we need a whole separate commit [16:46:02] in the crm/vendor repo [16:46:09] ohh [16:46:13] i forgot it was separate [16:51:09] (PS1) Mepps: Update with latest [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/428958 [16:51:25] (CR) Mepps: [C: 2] Update with latest [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/428958 (owner: Mepps) [16:53:45] :) [16:54:15] mepps I think some of our vendor dirs get merged by zuul, but some of them need a manual V+2 / submit [16:54:22] I forget which is the case with CRM [16:54:54] it looks like jenkins bot is doing something [16:56:25] (CR) jerkins-bot: [V: -1] Update with latest [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/428958 (owner: Mepps) [16:56:29] (CR) jerkins-bot: [V: -1] Update with latest [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/428958 (owner: Mepps) [16:57:14] k, cool [16:58:02] grr, or not... [16:58:08] :( :( [16:58:15] nothing helpful at least [16:58:29] oh damn, you don't want to add all the require-dev packages [16:59:37] (CR) Ejegg: [C: -1] "composer install needs --no-dev" [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/428958 (owner: Mepps) [17:00:57] cwd had a bunch of scripts to automate all this deploy drudgery [17:02:45] (PS2) Mepps: Update with latest [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/428958 [17:02:48] Fundraising-Backlog: 2 fundraising tech grafana dashboards broken - https://phabricator.wikimedia.org/T193056#4158645 (Ejegg) [17:03:07] heh [17:03:21] it's brutally unscriptable [17:04:26] (CR) Ejegg: [C: 2] Update with latest [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/428958 (owner: Mepps) [17:05:09] heh, i guess that was kinda a forest of edge cases, huh? [17:05:15] hard to product-ize [17:05:54] i've had to be satisfied with little aliases like this: [17:06:05] alias cplus2='ssh -p 29418 gerrit.wikimedia.org gerrit review --code-review +2' [17:09:02] phew, finally merged ejegg [17:09:08] yays! [17:09:10] (Merged) jenkins-bot: Update with latest [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/428958 (owner: Mepps) [17:15:59] (CR) Mepps: [C: 2] "recheck" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 (owner: Mepps) [17:16:17] mepps you'll need to amend that commit with the submodule update [17:16:19] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 (owner: Mepps) [17:16:25] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 (owner: Mepps) [17:16:32] yeah i think if you want to script git you have to use a "porcelain" [17:16:37] ooh, double whammy [17:16:42] CLI is way too interactive [17:16:53] grr arg ^^ i thought it was merged but wrong commit (too many going on) [17:17:15] * mepps bangs head against wall [17:17:32] that always helps :) [17:18:09] * ejegg starts thinking about lunch [17:19:26] Fundraising-Backlog: Cancel & refund the remaining unintended recurring donations from Big EN - https://phabricator.wikimedia.org/T192958#4158719 (MBeat33) Also, RECURRING GLOBALCOLLECT 8974545132-5 from December 3rd, 2017 came from a banner on the Initial List tab of [[ https://docs.google.com/spreadsheets... [17:19:45] that sounds more productive ejegg :) [17:22:19] (PS4) Mepps: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 [17:26:44] okay build succeeded... [17:29:32] ejegg now how do i get that to merge ^^ [17:29:44] (CR) Mepps: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 (owner: Mepps) [17:30:02] ha i hadn't +2ed and it had directed me to a weird place where i couldn't, nevermind [17:30:56] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428948 (owner: Mepps) [17:31:02] Nice! [17:31:10] fr-tech any other news for Scrum of Scrums ? [17:33:39] !log update civicrm from 6ddeb167ec to 219798b2c5 [17:33:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:44:18] Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: [Spike] Plan ingress system to use with new Kafka topic for landing page and impression dat... - https://phabricator.wikimedia.org/T192839#4158762 [17:57:47] (PS1) Ejegg: Fix SA-CORE-2018-004 [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/428964 [17:58:01] (CR) Ejegg: [C: 2] Fix SA-CORE-2018-004 [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/428964 (owner: Ejegg) [17:58:24] (PS1) Ejegg: Update drupal for security fix [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/428965 [17:58:30] (CR) Ejegg: [C: 2] Update drupal for security fix [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/428965 (owner: Ejegg) [18:02:50] (CR) jerkins-bot: [V: -1] Update drupal for security fix [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/428965 (owner: Ejegg) [18:03:03] (Merged) jenkins-bot: Fix SA-CORE-2018-004 [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/428964 (owner: Ejegg) [18:03:05] (CR) jerkins-bot: [V: -1] Update drupal for security fix [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/428965 (owner: Ejegg) [18:04:03] (CR) Ejegg: [C: 2] Update drupal for security fix [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/428965 (owner: Ejegg) [18:04:12] (CR) Ejegg: [V: 2 C: 2] Update drupal for security fix [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/428965 (owner: Ejegg) [18:04:56] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428966 [18:05:10] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428966 (owner: Ejegg) [18:09:18] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428966 (owner: Ejegg) [18:15:12] huh, there goes the power... [18:16:17] !log updated CiviCRM from 219798b2c5 to 47197006d5 [18:16:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:30:59] fr-tech power is out at my house, I'mma head to a cafe [19:41:37] Hi XenoRyet [19:41:51] I think mepps could use a hand with the refund issue [19:42:03] so, all the GC recurring refunds are marked wrong in Civi [19:42:04] Was just reading backscroll [19:42:17] the FIRST one is the one we DIDN'T refund [19:42:25] but it's the only one that's actually marked refunded [19:42:31] XenoRyet, i just need the contribution ids for the first payment for the recurring payments [19:42:47] Are we talking first batch, second, or both? [19:43:16] all of them [19:43:52] Ok, I'm on it. I'll let you know when I get the list. [20:26:56] mepps: Do you need contribution tracking IDs or just the ID from the civicrm_contribution table? [20:27:35] just civi ids xenoryet, though still trying to figure out the best place to write this script [20:27:49] where did you write the globalcollect script--was it through DI? [20:28:15] The script to run the GC refunds was pre-existing, but yea it's in DI [20:28:42] hmm--what about the cancellation script--that was thorugh wmf_civicrm? [20:28:58] maybe what we should do is just combine the queries in a wmf_civicrm query [20:28:59] Yep, that's where that one went. [20:31:20] So what do we think happened anyway? [20:34:15] we know what happened [20:34:32] * mepps here's the fix for it: https://gerrit.wikimedia.org/r/#/c/427946/ [20:34:52] oops XenoRyet here's the fix for it: https://gerrit.wikimedia.org/r/#/c/427946/ [20:35:44] Ah, I see [20:42:19] (PS1) Mepps: Start query for updating contributions mistakenly marked refunded Bug: T190098 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429097 (https://phabricator.wikimedia.org/T190098) [20:42:36] XenoRyet, you can add your query here ^^ [20:42:54] we could also select the ids first... [20:43:24] (PS2) Mepps: WIP:Start query for updating contributions mistakenly marked refunded Needs query from XenoRyet Bug: T190098 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429097 (https://phabricator.wikimedia.org/T190098) [20:43:56] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: String of numbers in Civimail email From showing to recipients - https://phabricator.wikimedia.org/T193082#4159347 (LeanneS) [20:44:06] Yea, I'll put the query in when I get it worked out right [20:44:32] (CR) jerkins-bot: [V: -1] WIP:Start query for updating contributions mistakenly marked refunded Needs query from XenoRyet Bug: T190098 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429097 (https://phabricator.wikimedia.org/T190098) (owner: Mepps) [20:48:19] (PS1) Eileen: Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 [20:53:20] (CR) Ejegg: Use post hook rather than triggers for original amount / currency (2 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 (owner: Eileen) [20:53:48] (CR) jerkins-bot: [V: -1] Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 (owner: Eileen) [20:54:22] (CR) Eileen: "ejegg just checked & the ones we deliberately exclude are" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/428544 (owner: Eileen) [20:57:08] (PS2) Eileen: Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 [20:57:24] mepps: Balpark, how many rows are we expecting here? [20:57:39] 17k is too many, right? [20:57:46] XenoRyet: same as the total # of emails in the GC lists [20:57:57] should be 1 row per contact_id [20:58:50] So we should prob do this in batches [20:58:54] ok, yea that's too many then, must be something wrong with the query [21:00:25] argh, how are there only 3 contribution_recur records marked 'in process' when there are 13 distinct IDs that seem to have failed? [21:01:43] We should be getting 4k and change rows. Let me figure out what I forgot to exclude [21:02:19] (CR) jerkins-bot: [V: -1] Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 (owner: Eileen) [21:19:35] (PS1) Eileen: Remove quiet rollback [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/429104 [21:36:28] (PS3) Eileen: Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 [21:40:33] (CR) jerkins-bot: [V: -1] Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 (owner: Eileen) [21:48:39] (PS4) Eileen: Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 [21:49:58] (CR) jerkins-bot: [V: -1] Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 (owner: Eileen) [21:50:26] (PS5) Eileen: Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 [21:53:57] (CR) jerkins-bot: [V: -1] Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 (owner: Eileen) [22:12:39] ok, got all the missing recurring contribs in [22:13:07] i'm going to try running a few more then [22:13:20] (PS3) XenoRyet: Start query for updating contributions mistakenly marked refunded Query from XenoRyet added Bug: T190098 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429097 (https://phabricator.wikimedia.org/T190098) (owner: Mepps) [22:14:09] well huh, did we really get them all in that one run? [22:14:21] guess we'll have to wait till next morning [22:14:45] (CR) jerkins-bot: [V: -1] Start query for updating contributions mistakenly marked refunded Query from XenoRyet added Bug: T190098 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429097 (https://phabricator.wikimedia.org/T190098) (owner: Mepps) [22:17:13] (PS4) XenoRyet: Start query for updating contributions mistakenly marked refunded Query from XenoRyet added Bug: T190098 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429097 (https://phabricator.wikimedia.org/T190098) (owner: Mepps) [22:18:00] (PS1) Ejegg: Ensure we don't empty a queue into the void [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/429116 [22:18:06] fr-tech little safety feature ^^^ [22:18:45] Fortunately when I found that I already had a file with the queue messages, I just wanted to get them out of the wrong queue I'd put 'em in [22:19:04] (CR) jerkins-bot: [V: -1] Start query for updating contributions mistakenly marked refunded Query from XenoRyet added Bug: T190098 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429097 (https://phabricator.wikimedia.org/T190098) (owner: Mepps) [22:19:08] (PS6) Eileen: Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 [22:22:32] (CR) jerkins-bot: [V: -1] Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 (owner: Eileen) [22:24:03] (PS5) XenoRyet: Start query for updating contributions mistakenly marked refunded Query from XenoRyet added Bug: T190098 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429097 (https://phabricator.wikimedia.org/T190098) (owner: Mepps) [22:24:52] (PS1) Ejegg: Reformat rgc.module to drupal standard [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429117 [22:28:31] (PS1) Ejegg: Recurring GC job sends queue message [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429118 [22:30:09] (CR) XenoRyet: [C: 2] Ensure we don't empty a queue into the void [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/429116 (owner: Ejegg) [22:30:15] thanks [22:30:23] No worries [22:30:50] (Merged) jenkins-bot: Ensure we don't empty a queue into the void [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/429116 (owner: Ejegg) [22:32:21] mepps does that query really do the trick? I thought we'd have to do something via API to get all the financial rows right [22:33:07] (CR) jerkins-bot: [V: -1] Recurring GC job sends queue message [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429118 (owner: Ejegg) [22:37:37] (PS7) Eileen: Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 [22:42:51] (CR) jerkins-bot: [V: -1] Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 (owner: Eileen) [22:58:08] (PS8) Eileen: Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 [23:02:55] (CR) jerkins-bot: [V: -1] Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 (owner: Eileen) [23:27:08] Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Dinosaur Cookies co-existed with Gingerbread People, Fundraising Sprint Elevators were never intended to go down, Fundraising Sprint Fhabricator is spelled with an "F", an... - https://phabricator.wikimedia.org/T186883#4159765 [23:36:07] (PS9) Eileen: Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 [23:41:21] (CR) jerkins-bot: [V: -1] Use post hook rather than triggers for original amount / currency [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429098 (owner: Eileen)