[00:09:28] Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising Sprint Junebugs prefer July , and 2 others:... - https://phabricator.wikimedia.org/T190098#4192866 [00:39:15] cwd: any idea why the queue graphs are gone now? [00:39:16] https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=11&fullscreen&from=now-30m&to=now [00:48:38] phew, refund queue up to 15000 [00:48:55] do we still have that running overtime? [00:50:16] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: refund is 14899 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 1 keys, up 85 days 37 minutes - memory use is 7.95M (peak 83.16M, 0.13% of max, fragmentation 1.38%), connected_slaves is 2, donations is 0, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 0, payments-init is 0, pending is 0, recurring is 0, unsubscribe is 0 [00:50:29] oh hey, let's ack that one [00:50:56] ejegg: i haven't looked into it yet https://phabricator.wikimedia.org/T193456 [00:52:11] ACKNOWLEDGEMENT - check_redis on frqueue1001 is CRITICAL: CRITICAL: refund is 14899 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 1 keys, up 85 days 37 minutes - memory use is 7.95M (peak 83.16M, 0.13% of max, fragmentation 1.38%), connected_slaves is 2, donations is 0, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 0, payments-init is 0, pending is 0, recurring is 0, unsubscribe is 0 El [00:52:11] unning large batch of refunds - The acknowledgement expires at: 2018-05-10 00:51:39. [00:52:40] cwd ah, cool [00:54:13] !log running refund queue consumer overtime [00:54:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:11:59] (PS15) Ejegg: WIP: SmashPig payment processor extension [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/426068 (https://phabricator.wikimedia.org/T1888678) [02:08:56] ah heck [02:09:05] that's just refunding the initial contributions again [02:15:12] (PS1) Ejegg: Fix gateway_parent_id for recurring refunds [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/432032 (https://phabricator.wikimedia.org/T190098) [02:16:24] Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising Sprint Junebugs prefer July, and 2 others:... - https://phabricator.wikimedia.org/T190098#4192985 [02:20:24] (PS1) Ejegg: Once again reset initial contrib status [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/432033 (https://phabricator.wikimedia.org/T190098) [02:23:31] (PS1) Eileen: Move hack to block search without criteria from core-hack to extension. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/432034 (https://phabricator.wikimedia.org/T194210) [02:25:28] (Abandoned) Eileen: CiviCRM 5.0 stock (7713e22ddbe8) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/420930 (owner: Eileen) [02:26:01] (PS1) Eileen: Revert "wmf: CRM-17158 block empty searches unless deliberate" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/432035 (https://phabricator.wikimedia.org/T194210) [02:26:26] (CR) Eileen: "In conjunction with https://gerrit.wikimedia.org/r/#/c/432035/" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/432034 (https://phabricator.wikimedia.org/T194210) (owner: Eileen) [02:38:12] (PS1) Ejegg: Add conditions for queue emptying [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/432037 [02:38:30] (CR) jerkins-bot: [V: -1] Add conditions for queue emptying [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/432037 (owner: Ejegg) [03:42:00] (PS2) Ejegg: Add conditions for queue emptying [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/432037 [10:45:43] Fundraising-Backlog, Wikimedia-Fundraising-Banners: CSP would break "Remind me later" functionality - https://phabricator.wikimedia.org/T194019#4193632 (Pcoombe) @AndyRussG All our banners with the option e.g. https://en.wikipedia.org/wiki/NASA?banner=B1718_0101_mlWW_dsk_p1_lg_template&force=1&country=US... [14:27:29] !log disabled refund queue consumer [14:27:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:32:08] (CR) Mepps: [C: 2] Fix gateway_parent_id for recurring refunds [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/432032 (https://phabricator.wikimedia.org/T190098) (owner: Ejegg) [14:32:33] (Merged) jenkins-bot: Fix gateway_parent_id for recurring refunds [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/432032 (https://phabricator.wikimedia.org/T190098) (owner: Ejegg) [14:44:54] to discuss: better naming conventions for gateway IDs in refund messages [14:47:14] jgleeson: so, we also have a refund queue that's currently full of bogus messages [14:47:23] :O [14:47:30] but the thing is, it might have SOME good messages in it [14:47:42] we just want to clear out everything where gateway=globalcollect [14:47:54] here's my first idea: https://gerrit.wikimedia.org/r/432037 [14:51:32] looks good! [14:52:48] I'm trying to see how conditions is passed into the maintenance script [14:53:07] ah is it just if gateway is sety [14:53:09] jgleeson: yeah, that could be better [14:53:10] set* [14:53:30] I didn't have a good plan for arbitrary arrays [14:57:06] I'm wondering if there's a nice way in redis to unpop a message [14:57:15] I'm sure you can do that in rabbitmq [14:57:36] do you can effectively sent it back if it doesn't match some condition, once consumed [14:57:49] so you can* [14:58:08] jgleeson: yeah, we could backend-agnostically just send it back to the same queue [14:58:21] but then we have to figure out where we start seeing the same messages again [14:59:08] you mean due to the order being changed? [14:59:09] so I decided to lean on the damaged / retry table [14:59:28] no, i mean if we're trying to empty the queue but leave some subset of message on it [14:59:41] and we do that by putting the ones we want to keep back at the end of the queue [14:59:55] we have to determine where we start hitting the ones we've already re-queued [15:00:07] so we don't just infinitely re-process them [15:00:27] we could record a hash of the first one we re-queue [15:00:36] but then we'd want to be sure it's unique [15:00:45] or we'd stop early when it had a twin [15:04:12] hmmm ok [15:04:17] feels messy [15:04:26] without understanding it more [15:04:44] yeah... code-wise, the damaged / retry table was definitely the simplest [15:05:13] yup. I was just reading up on how redis handles MQ like behaviour and ack/rej [15:05:15] and with the retry job running normally, those messages will get dumped back on within 20 min [15:05:26] got it [15:05:44] ok will quickly pull it down to run tests and then +" [15:05:46] +2 [15:06:07] thanks! [15:06:21] even though CI runs them [15:06:26] :) [15:06:33] always worth verifying [15:07:49] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi: edit Contribution Status is hanging 5/9 - https://phabricator.wikimedia.org/T194281#4194320 (MBeat33) [15:07:57] all passed with flying colours, as they say [15:08:07] (CR) Jgleeson: [C: 2] Add conditions for queue emptying [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/432037 (owner: Ejegg) [15:08:35] (Merged) jenkins-bot: Add conditions for queue emptying [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/432037 (owner: Ejegg) [15:09:19] it also looks like XenoRyet's ct_id patch has fixed the issue, but I'm now trying to understand how it wasn't working yesterday when I tried with the same checkout @_@ [15:09:28] ooh, weird [15:09:29] brb coffee [15:09:35] yeah, that one could use a test too [15:20:50] (PS1) Ejegg: Update SmashPig lib [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/432099 [15:20:53] (CR) Ejegg: [C: 2] Update SmashPig lib [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/432099 (owner: Ejegg) [15:22:01] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432100 [15:22:17] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432100 (owner: Ejegg) [15:22:19] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432100 (owner: Ejegg) [15:25:35] (Merged) jenkins-bot: Update SmashPig lib [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/432099 (owner: Ejegg) [15:25:37] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432100 (owner: Ejegg) [15:26:04] (CR) Ejegg: [C: 2] "recheck" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432100 (owner: Ejegg) [15:26:23] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432100 (owner: Ejegg) [15:26:26] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432100 (owner: Ejegg) [15:26:31] hmnm [15:26:44] oright [15:29:49] (PS1) Ejegg: Update SmashPig [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/432101 [15:30:01] (CR) Ejegg: [C: 2] Update SmashPig [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/432101 (owner: Ejegg) [15:30:24] (PS2) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432100 [15:30:44] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432100 (owner: Ejegg) [15:39:30] (Merged) jenkins-bot: Update SmashPig [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/432101 (owner: Ejegg) [15:39:56] what did I miss ejegg ? [15:51:42] jgleeson: just updating the lib in crm [15:52:22] (PS1) Ejegg: Update composer [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/432112 [15:52:39] (CR) Ejegg: [C: 2] Update composer [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/432112 (owner: Ejegg) [15:58:39] (Merged) jenkins-bot: Update composer [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/432112 (owner: Ejegg) [16:36:11] mepps did you have time to drop in on the tech talk? [17:34:27] fr-tech any Scrum of Scrums news? [17:34:35] None here [17:34:40] sorry, last-minute request [17:36:45] ejegg: none here, thx :) [17:48:47] (PS3) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432100 [17:49:02] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432100 (owner: Ejegg) [17:50:13] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432100 (owner: Ejegg) [17:59:05] (CR) Mepps: Add conditions for queue emptying (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/432037 (owner: Ejegg) [18:35:18] Fundraising-Backlog, FR-Ingenico: Ingenico audit wobble? iDEAL donations not in Civi May 1-3 - https://phabricator.wikimedia.org/T194296#4194799 (MBeat33) [18:40:51] (CR) Ejegg: Add conditions for queue emptying (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/432037 (owner: Ejegg) [19:41:04] ok fr-tech, I'mma try dumping the globalcollect stuff out of the refund queue to re-do those audit things [19:47:59] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/432159 [19:48:02] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/432159 (owner: Ejegg) [19:48:23] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/432159 (owner: Ejegg) [19:49:27] (PS2) Ejegg: Once again reset initial contrib status [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/432033 (https://phabricator.wikimedia.org/T190098) [19:49:34] fr-tech oops, one more thing left: ^^^^ [19:49:46] anybody mind nudging that one over the line? [19:50:16] ejegg how does that while true loop work out? [19:50:27] there's a break condition in there [19:50:41] when the fetched count is zero [19:50:54] (CR) Mepps: [C: 2] Once again reset initial contrib status [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/432033 (https://phabricator.wikimedia.org/T190098) (owner: Ejegg) [19:50:59] kk [19:51:00] thanks! [19:51:19] !log updated SmashPig standalone from 99585c8084 to 2b4186715e [19:51:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:55:42] (Merged) jenkins-bot: Once again reset initial contrib status [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/432033 (https://phabricator.wikimedia.org/T190098) (owner: Ejegg) [19:57:28] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432161 [19:57:53] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432161 (owner: Ejegg) [19:58:44] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432161 (owner: Ejegg) [20:00:16] !log updated CiviCRM from 4ff35ad4 to ca8acdccf6 [20:00:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:13:39] sweet, looks like that dump conditions patch worked [20:15:14] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 7 keys, up 85 days 20 hours - memory use is 1.85M (peak 83.16M, 0.06% of max, fragmentation 2.78%), connected_slaves is 2, donations is 1, jobs is 0, jobs-adyen is 0, jobs-paypal is 1, payments-antifraud is 1, payments-init is 2, pending is 0, recurring is 1, refund is 44, unsubscribe is 1 [20:16:24] !log restarted refund queue consumer [20:16:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:42:46] ejegg looks like the refund queue timed out agian [20:43:57] mepps man, refunds in civi are ridiculously slow! [20:44:07] apparently! [20:45:22] wtf, there was one that took 13 minutes! [20:46:03] and I guess the queue connection died in the interim [20:47:23] now it's trying the next one, and it's been 7 min on it so far [20:47:38] We might need to make this a priority [20:49:06] yipe [20:49:55] let's see what kind of horrible stuff it's doing to the db [20:53:28] hmm, something with the creditnote_id [20:53:43] I remember that being a performance-killer we hacked out at some point [20:59:27] !log disabled refund queue consumer [20:59:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:04:15] (PS1) Eileen: Save our server [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/432285 [21:05:22] (CR) Ejegg: [C: 2] Save our server [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/432285 (owner: Eileen) [21:10:00] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: upstream creditnote_id fix - https://phabricator.wikimedia.org/T194313#4195372 (Ejegg) [21:12:44] (Merged) jenkins-bot: Save our server [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/432285 (owner: Eileen) [21:17:05] (PS1) Ejegg: Update Civi submodule for creditnote hack [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/432288 [21:18:46] (CR) Ejegg: [C: 2] Update Civi submodule for creditnote hack [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/432288 (owner: Ejegg) [21:19:25] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432289 [21:19:43] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432289 (owner: Ejegg) [21:23:24] (Merged) jenkins-bot: Update Civi submodule for creditnote hack [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/432288 (owner: Ejegg) [21:23:26] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/432289 (owner: Ejegg) [21:25:24] !log updated CiviCRM from ca8acdccf6 to 4c6a9c9c1c [21:25:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:25:43] ok, let's see if the refund qc works now [21:27:25] !log re-started refund queue consumer [21:27:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:27:58] so fast! [21:28:02] thanks eileen [21:28:20] ok, reprocessing those audit files again [21:39:22] ejegg: lol [21:48:18] Fundraising-Backlog, FR-Amazon, FR-Smashpig: Support Amazon payments initiated off paymentswiki - https://phabricator.wikimedia.org/T194317#4195498 (Ejegg) [22:02:49] hmm, that mostly looks like it's doing the right thing this time [22:03:09] mepps: we're definitely marking a bunch of non-initial things as refunded! [22:03:40] but this record still looks odd: /civicrm/contact/view?reset=1&cid=22341035 [22:03:59] all installments except the LAST are marked refunded [22:04:17] I'll dig around in the audit files to see if that matches realit [22:04:35] oh, the last one might be refunded soon - that qc is still running [22:05:11] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: refund is 13847 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 3 keys, up 85 days 21 hours - memory use is 7.54M (peak 83.16M, 0.14% of max, fragmentation 1.48%), connected_slaves is 2, donations is 0, jobs is 0, jobs-adyen is 0, jobs-paypal is 2, payments-antifraud is 0, payments-init is 0, pending is 0, recurring is 4, unsubscribe is 0 [22:06:14] ejegg: think it's going to overflow for a while? [22:06:36] cwd I'll ack it for an hour or so [22:07:00] with eileen's refund perf hack we went for 13 min for one refund to 1,617 in 6 min [22:07:00] ok [22:07:07] haha [22:07:12] nice [22:08:00] we should throttle it or raise the limit if we consistently go over [22:08:16] this is definitely atypical [22:09:00] basically re-running a couple months worth of refunds of the inadvertent recurring donations [22:09:24] 'cause we realized the audit parser was associating all of them with the first donation in the chain rather than the right installment [22:10:11] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: refund is 13011 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 4 keys, up 85 days 21 hours - memory use is 7.24M (peak 83.16M, 0.14% of max, fragmentation 1.54%), connected_slaves is 2, donations is 0, jobs is 0, jobs-adyen is 0, jobs-paypal is 2, payments-antifraud is 0, payments-init is 2, pending is 0, recurring is 6, unsubscribe is 0 [22:10:55] ACKNOWLEDGEMENT - check_redis on frqueue1001 is CRITICAL: CRITICAL: refund is 13011 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 4 keys, up 85 days 21 hours - memory use is 7.24M (peak 83.16M, 0.14% of max, fragmentation 1.54%), connected_slaves is 2, donations is 0, jobs is 0, jobs-adyen is 0, jobs-paypal is 2, payments-antifraud is 0, payments-init is 2, pending is 0, recurring is 6, unsubscribe is 0 Elli [22:10:56] ning big refund batch - The acknowledgement expires at: 2018-05-09 23:50:09. [22:10:57] gotcha [22:10:59] sounds good [22:11:07] cwd: just out of curiousity, what does the 'start obsessing over' action do in icinga? [22:11:17] haha no idea [22:12:14] huh, lots of warnings about ipmi sensors [22:13:47] ejegg: where do you see that? [22:14:07] it's a known issue with some of these servers [22:14:08] just clicked on the full 'critical' list out of curiousity [22:14:16] don't think any of them were on our cluster [22:14:36] there is a kernel module that doesn't work with a sensor [22:14:54] blacklisting the module stops it [22:14:58] but i don't think it matters [22:15:20] seems like a lot of alertspam [22:15:48] https://unix.stackexchange.com/questions/422101/smbus-ipmi-genericserialbus-write-requires-buffer-of-length-66-found-length-32 [22:16:08] it's on my list to get that in puppet [22:21:04] hmm, sloppy bios data? [22:44:37] ejegg: blacklist acpi_power_meter fixes it [23:10:04] (PS1) Eileen: Stock CiviCRM + core updates (5.2) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/432320 [23:10:06] (PS1) Eileen: Re-apply WMF patches [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/432321 [23:10:08] (PS1) Eileen: Revert STFU changes since I think we finally nailed them... [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/432322 [23:13:22] (CR) jerkins-bot: [V: -1] Stock CiviCRM + core updates (5.2) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/432320 (owner: Eileen) [23:15:18] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 3 keys, up 85 days 23 hours - memory use is 2.20M (peak 83.16M, 0.07% of max, fragmentation 2.44%), connected_slaves is 2, donations is 0, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 0, payments-init is 0, pending is 0, recurring is 0, refund is 933, unsubscribe is 1 [23:45:12] (PS4) XenoRyet: WIP: Ingenico ct_id fix. [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/431676 (https://phabricator.wikimedia.org/T190871) [23:47:18] (CR) jerkins-bot: [V: -1] WIP: Ingenico ct_id fix. [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/431676 (https://phabricator.wikimedia.org/T190871) (owner: XenoRyet) [23:53:13] (Abandoned) Eileen: Reapply WMF patches [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/420931 (owner: Eileen)