[00:00:10] oh no worries :) [00:00:20] but it's a good question because I was expecting to see slow queries with the word Benevity in them - but didn't [00:00:43] if you DO see any slow queries them check for the user id in the query [00:00:49] Just wondering what the other signs might have been... I guess this was a DB slowdown? Maybe instead of a slow query it was a lot of queries? Are there any Grafana graphs that show this activity coinciding with the queue accumulation? [00:01:02] (back in a bit, but I'll see backscrool) [00:01:09] one thing i did notice while looking to see if it were a civi action at a distance thing is that we generate a lot more in logs - https://frmon.frdev.wikimedia.org/d/000000377/host-overview?refresh=5m&panelId=12&fullscreen&orgId=1&from=1575244856665&to=1575331256666&var-server=civi1001.frack.eqiad.wmnet&var-datasource=Prometheus&var-cluster= [00:01:29] AndyRussG: I'm not sure because we were trying to determine if the Benevity import WAS causing the slow down & struggled to [00:01:45] eileen: would that UID match up to a civi UID? [00:02:03] as in, the UID of a slow query [00:02:20] so generally the best query is SELECY * FROM civicrm_uf_match where uf_id = x; [00:02:29] SELECT.... [00:02:51] OR in the drupal DB SELECT * FROM users WHERE uid = x; [00:02:58] cool. adding that to my notes [00:03:12] civicrm_uf_match has a user id (uf_id) & a contact_id - which maps to civicrm_contact.id [00:03:29] (if the info in the uf_match table isn't enough ) [00:03:41] uf is 'user framework' - aka cms [00:04:59] thanks! [00:21:02] i'm wondering if the naming of the global_connect log files has some meaning. [00:21:34] for instance, there is a file named 20170501_globalcollect.working that was written to yesterday. [00:35:38] (back) [00:37:46] eileen: dwisehaupt: hmm queue size kinda stopped dropping: https://frmon.frdev.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=1575323727920&to=1575333396127 [00:38:03] yeah. just saw that. [00:38:31] Processing rate has dropped, but not by as much as before. [00:40:15] not seeing the db contention that we did earlier. [00:40:55] but i need to run and drop $daughter at ballet. i think it will be fine to keep an eye on until i return in ~30. [00:41:14] i'll have my phone and laptop with so if it blows up don't hesitate to call/txt [00:41:21] Will do [00:41:32] yeah q is levelling [00:41:57] back in 30 [00:42:32] amazon appeared in the q [00:51:20] Gotta go pick up my kids too [00:51:53] eileen: where do you see the Amazon thing? You mean there haven't been much Amazon transactions so far? [00:52:16] in donations queued to civicrm [00:52:36] - does that mean donations imported from a queue (rather than waiting in a q) [00:52:55] Oh bell! [00:54:04] eileen: sorry for the ignorance, how do you actually monitor queue content? [00:54:57] yeah - I get confused but I think the donation queue is what is waiting to go in & 'donations queued ' is what has gone in? [00:56:06] Hmmm I don't know... Where can I see either of those? [00:59:32] eileen: are these just log files in frlog1001? [01:05:45] no grafana - https://frmon.frdev.wikimedia.org/d/Pq1YNMviz/fundraising-overview?refresh=1m&orgId=1&panelId=9&from=now-6h&to=now [01:17:31] eileen: ah ok right... yeah I see, Amazon started to go up, though I guess everything else spiked at the same time, too, no? [01:18:07] yeah - I only commented because I didn't see it at all before thatt [01:20:21] GlobalCollect seems to have entered the queue like 10 minutes ago [01:27:21] it was feeling unloved [01:27:26] Stuff just keeps going up in the queue delay graph [01:27:52] also payments memcache current items [01:30:29] yeah & speed is down :-( [01:30:54] ccccccknfuvhtvccefclktjgetnthhuugvrfkllrifhl [01:31:12] I think the thank you emails are just generally not fast enough [01:45:21] K I'll be afk again for a while, back online in like 1 hr or a bit less [01:52:20] eileen: we've recently started to include data from more tables in the TY mail - maybe that's slowed it down? [01:53:33] ejegg|away: it runs for 70 seconds including a sleep(45) so it looks like it's tuned down really low [01:57:55] eileen the idea is for it to not collide too badly with the queue consumer [01:58:05] since it queries the same tables that the queue consumer writes to [01:58:18] ah ok [02:03:59] another possibility would be to make it a queue consumer: https://gerrit.wikimedia.org/r/389721 [02:04:26] so the donation qc inserts a message into the ty queue, since it has all that data ready [02:05:21] ok, really |away now [02:20:56] (PS1) Eileen: Minor caching addition. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/554211 [02:21:14] that might help on those file missed ... [02:23:37] (CR) jerkins-bot: [V: -1] Minor caching addition. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/554211 (owner: Eileen) [02:26:36] (PS2) Eileen: Minor caching addition. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/554211 [03:07:09] (CR) Ejegg: [C: +2] Minor caching addition. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/554211 (owner: Eileen) [03:12:26] (Merged) jenkins-bot: Minor caching addition. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/554211 (owner: Eileen) [03:24:09] (PS1) Eileen: CiviCRM submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554220 [03:25:26] (CR) Eileen: [C: +2] CiviCRM submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554220 (owner: Eileen) [03:35:40] Fundraising-Backlog, FR-Amazon: Amazon TY emails in Civi being sent more slowly? - https://phabricator.wikimedia.org/T239677 (MBeat33) [03:38:29] Fundraising-Backlog, FR-Amazon: Amazon TY emails in Civi being sent more slowly? - https://phabricator.wikimedia.org/T239677 (Eileenmcnaughton) mbeat33 the thank you queue is pretty behind. We don't think we can do more than wait [03:39:40] Fundraising-Backlog, FR-Amazon: Amazon TY emails in Civi being sent more slowly? - https://phabricator.wikimedia.org/T239677 (MBeat33) Open→Resolved a:MBeat33 Got it, thanks, @Eileenmcnaughton [04:01:11] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Big english queue speeds - https://phabricator.wikimedia.org/T239678 (Eileenmcnaughton) [04:03:11] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Big english queue speeds - https://phabricator.wikimedia.org/T239678 (Eileenmcnaughton) [04:07:16] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Exception recovery - https://phabricator.wikimedia.org/T239679 (Eileenmcnaughton) [04:22:57] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Big english queue speeds - https://phabricator.wikimedia.org/T239678 (Eileenmcnaughton) [04:37:17] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Big english queue speeds - https://phabricator.wikimedia.org/T239678 (Eileenmcnaughton) I can see we are starting to catch up - 11 minutes behind & under 5k donations behind. I think this is because the incoming donations as slowing as night falls in t... [05:03:06] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Big english queue speeds - https://phabricator.wikimedia.org/T239678 (Eileenmcnaughton) 8 minutes behind & 3k - we won't turn the corner on thank yous until the donations are caught up though [05:41:08] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Big english queue speeds - https://phabricator.wikimedia.org/T239678 (Eileenmcnaughton) Donations all caught up now - time for thank yous to up their game [07:30:15] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Big english queue speeds - https://phabricator.wikimedia.org/T239678 (Eileenmcnaughton) Thank yous still drifting further & further behind - 4 hours now - am wondering about tuning the job [07:44:08] ccccccknfuvhtndidigvgntkvudilihjjnnneuueettb [11:12:14] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: recurring is 2014 2000 - REDIS 3.2.6 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 137 days 9 hours - memory use is 4.12M (peak 20.40M, 0.09% of max, fragmentation 1.86%), connected_slaves is 2, donations is 617, jobs is 0, jobs-adyen is 0, jobs-paypal is 1, payments-antifraud is 563, payments-init is 151, pending is 1, refund is 0, unsubscribe is 23 [11:17:14] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: recurring is 2208 2000 - REDIS 3.2.6 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 137 days 9 hours - memory use is 4.57M (peak 20.40M, 0.09% of max, fragmentation 1.68%), connected_slaves is 2, donations is 574, jobs is 0, jobs-adyen is 0, jobs-paypal is 60, payments-antifraud is 407, payments-init is 53, pending is 2, refund is 0, unsubscribe is 30 [11:22:14] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: recurring is 2252 2000 - REDIS 3.2.6 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 137 days 9 hours - memory use is 4.03M (peak 20.40M, 0.09% of max, fragmentation 1.82%), connected_slaves is 2, donations is 358, jobs is 0, jobs-adyen is 0, jobs-paypal is 4, payments-antifraud is 361, payments-init is 138, pending is 9, refund is 0, unsubscribe is 37 [11:22:16] ACKNOWLEDGEMENT - check_redis on frqueue1001 is CRITICAL: CRITICAL: recurring is 2252 2000 - REDIS 3.2.6 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 137 days 9 hours - memory use is 4.03M (peak 20.40M, 0.09% of max, fragmentation 1.82%), connected_slaves is 2, donations is 358, jobs is 0, jobs-adyen is 0, jobs-paypal is 4, payments-antifraud is 361, payments-init is 138, pending is 9, refund is 0, unsubscribe is 37 [13:42:14] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 3.2.6 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 137 days 12 hours - memory use is 3.77M (peak 20.40M, 0.08% of max, fragmentation 1.82%), connected_slaves is 2, donations is 538, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 573, payments-init is 224, pending is 3, recurring is 902, refund is 0, unsubscribe is 158 [15:13:35] (PS1) Ejegg: Cache API call result in TY send inner loop [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554304 [15:14:09] fr-tech ^^^ might help us catch up on the late TY mails [15:15:26] (CR) Mepps: [C: +2] Cache API call result in TY send inner loop [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554304 (owner: Ejegg) [15:15:36] thanks mepps! [15:15:42] woohoo [15:16:14] how 'bout that recurring queue backup - did we really get that many recurring signups overnight? [15:16:25] is AU all about monthly giving this year? [15:17:53] sure seems like it eh [15:17:57] either that or NZ [15:20:04] pcoombe / spatton / Seddon : do the recurring signups seem reasonable, or might there be something leading to people signing up without intending to? [15:20:18] like 7200 since midnight UTC [15:20:46] (Merged) jenkins-bot: Cache API call result in TY send inner loop [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554304 (owner: Ejegg) [15:20:53] let's see how many of those are monthly conversions [15:21:28] 3640 are conversions [15:21:44] ok, I'mma see if that API caching speeds up the TY mails much [15:27:53] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/554306 [15:27:58] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/554306 (owner: Ejegg) [15:33:18] looks good on staging [15:33:30] ty mail test goes from 34 sec for 1000 to 25 sec [15:33:37] ok, pushing to prod [15:33:55] heh, when that merge takes [15:35:37] oh right, I guess we still have to submit manually on deploy. boo [15:38:21] !log updated fundraising CiviCRM from 5cf2d2713f to 4f3341455f [15:38:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:40:41] dang, that doesn't seem to have done much [15:42:55] Fundraising-Backlog: small format fix for recurring TY email - https://phabricator.wikimedia.org/T239719 (MBeat33) [15:45:59] hey ejegg, recurring signups from banners look reasonable ... just looked back at our last cluster of tests and don't see any with an abnormally high recurring rate. [15:46:52] Caitlin Cogdill did confirm that they are now pointing all their email donors to form flows w/ the monthly convert at the end. She said they sent emails to NZ in the last 6 hours or so [15:48:30] spatton: thanks...! The spike in the recurring queue is in a NZ awake-ish time [16:01:30] ok, thanks! [16:02:16] !log reduced donations queue consumer 10 sec per cycle and increased TY mail sender 10 sec per cycle [16:02:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:07:21] ooh, silly me, I did that timing change on frdev [16:08:24] ok, now applied on prod [16:11:15] (PS1) Ejegg: Set cache namespace for gift table name [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554320 [16:11:41] jeez, ty mail sending is getting slower and slower [16:14:37] and it seems to be in the actual handoff to phpmailer / the mail sending daemon [16:14:46] rather than in the CiviMail record creation [16:15:42] whelp, today looks like it'll be our best ever [16:17:42] Jeff_Green: any idea why it looks like the email daemon is taking up to 1.5 sec sometimes to accept a mail from phpmailer? [16:18:12] ejegg: my guess is that with all the logthrashing going on on that box, it's io bound [16:19:29] https://frmon.frdev.wikimedia.org/d/000000377/host-overview?refresh=5m&orgId=1 [16:19:32] ooh, logging is doing it? [16:19:40] take a look at disk utilization and saturation [16:20:08] ah dang [16:20:41] ok, so we should look at what kind of log lines we can cut? [16:20:53] we wrote almost 9GB of logs to /var/log/syslog yesterday [16:21:00] yesplz [16:22:07] in the past few minutes we logged 300-600 messages per second [16:27:22] (PS1) Ejegg: Stop logging full antifraud messages [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554324 [16:27:54] OK, that's not syslog, but it's the biggest offender in /var/log/process-control ^^ [16:28:21] actually, might be mirrored to syslog now that I think of it [16:28:34] fr-tech what do you think of that one? ^^^ [16:28:43] looks like we urgently need to stop log spam [16:29:00] in order to let the mailer have some disk I/O to get the TY mails out [16:29:29] ejegg: by volume it's both yeah, I wasn't counting the p-c logs in the 600/sec [16:36:52] Hmmm interesting [16:42:15] ooh, all the qcs are logging the full message [16:42:20] howdy [16:42:35] but not in a useful way [16:43:17] well, readable, but nothing we can use to recover a bunch of dropped messages [17:00:15] (CR) XenoRyet: [C: +2] Stop logging full antifraud messages [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554324 (owner: Ejegg) [17:00:26] ejegg: ^ [17:00:30] looks good [17:06:27] (Merged) jenkins-bot: Stop logging full antifraud messages [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554324 (owner: Ejegg) [17:27:52] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/554342 [17:28:04] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/554342 (owner: Ejegg) [17:41:20] Jeff_Green: any ideas on how to find the spammiest bits in the logs / the most impactful to remove? [17:43:55] !log updated fundraising CiviCRM from 4f3341455f to 26b788378e [17:43:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:45:11] ejegg: the vast majority of logs, by line count, are from these tags: [17:45:17] 3432766 drupal [17:45:21] 2467593 php [17:45:28] 523476 SmashPig-QueueJobRunner [17:46:56] at the volume the logs are generated, and the extreme length of the log lines, I think it's almost easier to look in code for where things are logging and evaluate whether it's unnecessary or redundant [17:47:19] ok [17:48:16] !log adjusted timing of thank you mailer and donations QC to give 5 more sec / cycle to TY mails [17:48:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:50:26] oh, we're still logging the paypal orphan rectifier at debug level [17:50:34] will disable that [17:51:04] I was looking for the drupal knob, maybe we're logging debug there too? [17:51:39] hmm, let's see [17:52:07] !log disabled PayPal orphan rectifier debug logging [17:52:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:52:15] fwiw SmashPig.yaml has log-level: 7 [17:55:23] well, let's hope for the best and dial that down to 6 [17:55:50] it's set in a couple places, I have no idea if what it logs is useful at that level? [17:56:20] I'll bump it down to 6 in provider-defaults.yaml [17:58:21] looking at all the /etc/*yaml config files, it seems like debug is set in most of them [18:05:09] Jeff_Green: oh hey, we can delete at least astropay-audit.yaml and globalcollect-audit.yaml [18:05:31] those are the old config files for when those ran under python tools [18:06:13] ok [18:06:17] /etc/SmashPig.yaml is also now completely obsolete [18:09:11] darn, TY mail send was up to 650 / batch but now it's back to 400ish [18:10:19] fundraising-tech-ops: remove/unpuppetize deprecated fundraising audit config files - https://phabricator.wikimedia.org/T239735 (Jgreen) [18:12:16] I'm going to try turning off CiviMail records for a batch or two [18:19:20] Well, with those off it does get through the whole batch [18:21:54] ok, going to bump up the batch limit and see how many we can get in one cycle [18:23:34] fr-tech I used drush variable-set to get a value larger than we make available in the dropdown [18:23:44] so the settings page may be wonky till we set it back [18:24:01] 10-4 [18:45:23] hey fr-tech i'm looking in backscroll but didn't see if we'd already discussed the failmail we're getting? [18:45:55] mepps no discussion, but there's not much we can do [18:46:02] okay ejegg [18:46:11] CONNECT_PLATFORM_ERROR is pretty opaque [18:46:23] It's weird that it's happening 20 after the hour [19:02:37] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: GlobalCollect_endpoint_critical 1 [=1] [19:04:35] Fundraising-Backlog, FR-Ingenico, Recurring-Donations: Ingenico recurring cancel process question - https://phabricator.wikimedia.org/T239741 (MBeat33) [19:07:31] RECOVERY - check_log_messages on frav1002 is OK: OK [19:22:47] Fundraising-Backlog, FR-Ingenico, Recurring-Donations: Ingenico recurring cancel process question - https://phabricator.wikimedia.org/T239741 (MBeat33) [19:27:29] Wikimedia-Fundraising-CiviCRM: Thank you letter not showing in the Activity field for Endowment Donation - https://phabricator.wikimedia.org/T239745 (RLewis) [19:29:22] umm fr-tech, is anyone else seeing the numbers in dash? [19:29:44] Yea, crazy good day. [19:29:54] BDE right? [19:30:06] By more than a million dollars [19:30:27] I was checking the SQL thinking that can't be right [19:30:42] On a less fun note, there is a bunch of failmail coming through just now. [19:31:34] it looks like ingenico is timing out [19:31:41] XenoRyet i was asking ejegg about that [19:31:49] he said there wasn't much we could do :/ [19:32:02] These look a little different from what we were seeing earlier [19:32:10] yeah these aren't the same [19:32:13] these are timeouts I think [19:32:16] yeah, the unknown SSL protocol thing is new [19:32:34] let's send something to merchant services [19:33:01] oh actually yeah they're not timeouts [19:33:14] interesting [19:33:15] the endpoint monitoring shows 5-8s times for globalconnect from the frpig hosts [19:34:53] the latest failmail is timeout [19:35:01] looks like ingenico are having some problems [19:36:29] yeah. testing ssl connections and getting mostly failures with the odd success. [19:39:05] Logs seem to be showing most ingenico donations still getting through [19:40:12] maybe we're throwing unprecedented traffic which would correlate with the numbers we're seeing in dash? [19:41:53] it's unprecedented traffic for us, sure, but you'd think a global card processor would be able to handle it [19:42:01] if they can handle i.e. black friday sales [19:42:47] yeah I guess so [19:48:50] (CR) Eileen: [C: +2] Set cache namespace for gift table name [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554320 (owner: Ejegg) [19:53:46] !log shifted 20 more sec / cycle from donations QC to thank you mailer [19:53:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:54:33] (Merged) jenkins-bot: Set cache namespace for gift table name [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554320 (owner: Ejegg) [20:12:47] wow our q is cranking! [20:14:13] looks good on grafana eileen [20:14:25] or do you mean the thank you emails [20:14:33] I mean the top line [20:14:41] donations received today [20:14:52] ah yeah crazy! [20:16:12] although the calc seems fishy in some way - it increased so much since I last looked but the hourly totals are hours behind so I can't check [20:43:28] (PS1) Eileen: Reduce logging on contact creation [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554363 [20:45:31] Fundraising-Backlog, fundraising-tech-ops: Postfix virtual mapping is capturing thank you messages destined for donors that use postmaster@domain for their address - https://phabricator.wikimedia.org/T239751 (Dwisehaupt) [20:52:02] (PS2) Eileen: CiviCRM submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554220 [20:52:04] (PS2) Eileen: Reduce logging on contact creation [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554363 [20:56:35] Fundraising-Backlog, fundraising-tech-ops: Postfix virtual mapping is capturing thank you messages destined for donors that use postmaster@domain for their address - https://phabricator.wikimedia.org/T239751 (Dwisehaupt) We may want to look at dmarc@ and bounce also as they are unbounded. No hits on thos... [21:25:42] I rebased so this will merge on a + 2 now https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/554363/ [21:29:23] (CR) Ejegg: [C: +2] Reduce logging on contact creation [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554363 (owner: Eileen) [21:30:03] Fundraising-Backlog: Investigate why orphan-slayer job is running for hours - https://phabricator.wikimedia.org/T239756 (jgleeson) [21:35:04] (Merged) jenkins-bot: Reduce logging on contact creation [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554363 (owner: Eileen) [21:35:53] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/554369 [21:36:22] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/554369 (owner: Eileen) [21:40:21] !log civicrm revision changed from 26b788378e to 0f51030071, config revision is 17b6730a72 - includes 3 possible performance improvements - logging reduction, cache a query result & cache file existence [21:40:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:40:38] Fundraising-Backlog, fundraising-tech-ops: Postfix virtual mapping is capturing thank you messages destined for donors that use postmaster@domain for their address - https://phabricator.wikimedia.org/T239751 (Dwisehaupt) p:Triage→High a:Dwisehaupt Change pushed and puppet run on civi1001 to p... [22:01:22] Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog, FR-PayPal-ExpressCheckout: Duplicate donations caused by PayPal's “Final Approval” messaging - https://phabricator.wikimedia.org/T235220 (MBeat33) New case today #669121: "donated today through Paypal, and got a notice that I needed to "c... [22:17:07] Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog, FR-PayPal-ExpressCheckout: Duplicate donations caused by PayPal's “Final Approval” messaging - https://phabricator.wikimedia.org/T235220 (EMartin) I submitted to PP and PP Spoof department for review. They advised that they directly reac... [23:14:48] Fundraising-Backlog, fundraising-tech-ops: Issue new SSL Client Certificate for cstone - https://phabricator.wikimedia.org/T239767 (Dwisehaupt) [23:15:52] Fundraising-Backlog, fundraising-tech-ops: Issue new SSL Client Certificate for cstone - https://phabricator.wikimedia.org/T239767 (Dwisehaupt) Certificate renewed and passed on to cstone. [23:17:23] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Big english queue speeds - https://phabricator.wikimedia.org/T239678 (Eileenmcnaughton) Thank you lag now 30 mins (that is the best it has been & catching up) [23:21:52] Fundraising-Backlog, fundraising-tech-ops: Issue new SSL Client Certificate for cstone - https://phabricator.wikimedia.org/T239767 (Dwisehaupt) Open→Resolved update has been pushed out and cstone verified that the new cert works. [frack::puppet::private] c5f4fb8 Reissuing of cstone client ssl cert [23:28:35] Fundraising-Backlog, fundraising-tech-ops: Issue new SSL Client Certificate for jsamra - https://phabricator.wikimedia.org/T238763 (Dwisehaupt) Certificate has expired. I have gone ahead and revoked and pushed out the updated CRL as the cert has stopped working for now. Will reissue when jsamra confirms...