[00:34:17] (CR) XenoRyet: [C: 2] Allow global fns in CustomFilters_Functions [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/476148 (owner: Ejegg) [00:35:22] (Merged) jenkins-bot: Allow global fns in CustomFilters_Functions [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/476148 (owner: Ejegg) [00:42:22] (PS1) Ejegg: Revert "Force CA donations to old GlobalCollect form" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/476183 [00:42:27] (CR) Ejegg: [C: 2] Revert "Force CA donations to old GlobalCollect form" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/476183 (owner: Ejegg) [00:42:55] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/476184 [00:43:00] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/476184 (owner: Ejegg) [00:43:41] (Merged) jenkins-bot: Revert "Force CA donations to old GlobalCollect form" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/476183 (owner: Ejegg) [00:43:59] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/476184 (owner: Ejegg) [00:47:32] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/476185 [00:47:42] (CR) Ejegg: [C: 2] Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/476185 (owner: Ejegg) [00:52:21] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/476185 (owner: Ejegg) [01:02:18] !log updated payments-wiki config to 51abff4b66 [01:02:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:47:53] Am I reading grafana correctly? It looks like our queue processing is keeping up [03:49:57] eileen_: so it seems! [03:50:21] i don't know the technical details but i would bet you have something to do with that :) [03:50:54] cwd I was just thinking about it & I feel like my first big English it took a day or so to catch up & Katie said that was well below the margin of concern [03:51:05] last year I feel like there was some lag catching up [03:51:47] we’ve done a bunch of things including code improvements & server changes but I must say it would be pretty amazing if we are really keeping up with peak flow [03:52:26] I wonder if we can see any speed bump from that trigger tweak last week [03:53:14] ya i have definitely seen the queue a day or more behind [03:53:15] no I can’t [03:53:32] how did you check? [03:53:45] well https://grafana.wikimedia.org/dashboard/db/fundraising-overview?orgId=1&from=now-14d&to=now&refresh=1m [03:53:53] processing rate shows no noticeable change [03:54:11] (last weeks tweak wasn’t expected to speed up average donors - only big donors [03:54:43] so for a donor with 100 donations I wouldn’t expect it to matter - it was the donors with thousands of donations it mattered on [03:54:44] eileen_: heh well i was actually looking at that very board earlier and https://grafana.wikimedia.org/dashboard/db/fundraising-overview?orgId=1&from=now-1y&to=now&refresh=1m&panelId=22&fullscreen [03:55:10] not really noticeable change since the stat was created [03:55:25] which leads me to think i don't know what the stat represents [03:55:46] actually there is a story I can see there [03:56:28] we started collecting per year data in around March & the peaks are under 4 per second from then [03:56:56] under ? at some point I made some improvements to the queries we were using [03:57:01] yeah there is some change with banners but not obvious like total requests: https://grafana.wikimedia.org/dashboard/db/fundraising-overview?orgId=1&from=now-14d&to=now&refresh=1m&panelId=6&fullscreen [03:57:12] probably september [03:57:29] at which point it starts to go up to 5,6 & 7 [03:57:34] & we became happy again [03:58:45] I am trying to remember which change it was - I think we started assuming that new donations with the current date were the most recent & stopped calculating most recent donation details for them [03:59:11] which reminds me - there is a risk that we see queue speed drop if we get a situation where we are processing yesterday’s donations today [04:00:43] ah yeah i sorta remember that change [04:01:03] I’m currently experiencing some performance issues [04:01:16] it started about 5 minutes after the kids got home from school [04:01:26] when the internet slowed down notably.... [04:02:10] cwd I did some stats on switching to Redis - I think it’s gonna give about 200ms per page load for logged in users [04:03:39] see https://github.com/civicrm/civicrm-core/pull/12665#issuecomment-442289297 [04:05:04] heheh [04:06:26] that looks a lot like gnuplot [04:08:49] cwd yep it was not very much fun…. [04:12:19] gnuplot builds character! [04:12:31] like camping in the winter or ditch digging [04:13:31] lol [05:17:09] (PS1) Eileen: WIP thank you updates [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/476199 (https://phabricator.wikimedia.org/T209895) [05:34:40] Fundraising Sprint Window dressing is mostly olive oil, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: hand update civi thank you email with current text - https://phabricator.wikimedia.org/T209895 (Eileenmcnaughton) I made some changes to this in testing today I hit a cou... [07:05:43] * Seddon takes up watch [13:51:47] Fundraising Sprint Window dressing is mostly olive oil, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: hand update civi thank you email with current text - https://phabricator.wikimedia.org/T209895 (DStrine) I think this is all good for now. I'd be cool if you could figure... [17:23:13] Fundraising-Backlog, fundraising-tech-ops: Need updated CiviCRM cert for new computer - https://phabricator.wikimedia.org/T210021 (CCogdill_WMF) Thanks @cwdent! I have the cert installed and on my Chrome browser (which I quit completely), however went I go to the dash URL and was prompted with the cert,... [17:27:09] ... so when your on the https role the paymentswiki urls still contain 8080 and fail [17:27:13] you're* [17:29:25] I'll push the patch shortly for that, XenoRyet I think that might be of interest for you too as I vaguely remember we were both fighting with TLS/SSL locally on vagrant [17:29:44] Yea, I don't remember where I left off on that, but it was a thing [17:46:49] (PS1) Ejegg: Revert "Handle invalid GC SET_PAYMENT response and fix some" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/476318 [17:47:02] (CR) Ejegg: [C: 2] Revert "Handle invalid GC SET_PAYMENT response and fix some" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/476318 (owner: Ejegg) [17:51:14] Fundraising-Backlog, fundraising-tech-ops: Need updated CiviCRM cert for new computer - https://phabricator.wikimedia.org/T210021 (cwdent) @CCogdill_WMF hrm, I think this is a keychain problem based on: https://www.nicksherlock.com/2017/09/fixing-err_ssl_client_auth_signature_failed-on-macos/ @spatton j... [17:58:51] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/476319 [17:59:09] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/476319 (owner: Ejegg) [18:01:35] Fundraising-Backlog, fundraising-tech-ops: Need updated CiviCRM cert for new computer - https://phabricator.wikimedia.org/T210021 (CCogdill_WMF) Ugh, unfortunately my Access Control is already set to "confirm before allowing access". I don't even get the option to hit allow. When I navigate to dash, I ge... [18:23:48] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/476324 [18:26:03] (CR) jerkins-bot: [V: -1] Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/476324 (owner: Ejegg) [18:27:22] grr, zuul is super slow [18:31:50] Fundraising-Backlog, fundraising-tech-ops: Need updated CiviCRM cert for new computer - https://phabricator.wikimedia.org/T210021 (cwdent) Nasty. Can you try removing/re-adding the cert? Here is the annoying distinction: the cert has an associated password to unencrypt it, which is what I send via SMS.... [18:31:53] (Merged) jenkins-bot: Revert "Handle invalid GC SET_PAYMENT response and fix some" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/476318 (owner: Ejegg) [18:31:58] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/476319 (owner: Ejegg) [18:55:25] g'night fr-tech! [18:59:42] (CR) Ejegg: [C: 2] Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/476324 (owner: Ejegg) [19:04:07] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/476324 (owner: Ejegg) [19:05:30] !log updated payments-wiki from 3e0c97a969 to 6552acdc0f [19:05:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:21:06] (PS1) Ejegg: Force CA donations to old GlobalCollect form [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/476358 [19:21:26] (CR) Ejegg: [C: 2] Force CA donations to old GlobalCollect form [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/476358 (owner: Ejegg) [19:21:50] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/476361 [19:21:56] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/476361 (owner: Ejegg) [19:31:16] (Merged) jenkins-bot: Force CA donations to old GlobalCollect form [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/476358 (owner: Ejegg) [19:31:18] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/476361 (owner: Ejegg) [19:56:21] Fundraising-Backlog, donate.wikimedia.org: Typo in "Ways to give" on donate.wikimedia.org - https://phabricator.wikimedia.org/T210650 (Deskana) [20:11:21] do we know how much we got on our first full day last year? [20:25:03] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/476379 [20:25:05] (CR) Ejegg: [C: 2] Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/476379 (owner: Ejegg) [20:28:08] Fundraising-Backlog, donate.wikimedia.org: Typo in "Ways to give" on donate.wikimedia.org - https://phabricator.wikimedia.org/T210650 (Pcoombe) Open>Resolved a:Pcoombe [Fixed](https://donate.wikimedia.org/w/index.php?title=Ways_to_Give&diff=30925&oldid=30836). Thanks Dan! [21:48:50] (PS2) Eileen: WIP thank you updates [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/476199 (https://phabricator.wikimedia.org/T209895) [22:32:59] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/476379 (owner: Ejegg) [22:33:18] finally! [22:33:22] ok, deploying that [22:34:25] Leanne is doing an engage import at the moment [22:34:43] it might be triggering deadlocks - since I just saw 2 failmails [22:35:00] which I think were the same transaction [22:35:27] import is 25% complete - if there looks to be more issues will try to kill [22:36:14] ejegg: we never turned off the dedupe scripts for the kick off did we? [22:36:52] eileen: nope, we're depending on the built-in throttle [22:37:02] i've only seen them run a couple times a day [22:37:06] hmm - I haven’t seen any sign of a queue build up [22:37:33] right, it's querying the count of recent donations for the throttle, I think [22:37:45] We don't have a good way of knowing the queue length in code [22:37:53] (without breaking our abstraction) [22:38:15] and the # of recent donations has been consistently high [22:38:28] so I’m just looking here https://grafana.wikimedia.org/dashboard/db/fundraising-overview?orgId=1&from=now-7d&to=now&refresh=1m&panelId=9&fullscreen [22:39:03] ah that’s perhaps not the graph I want - that’s just a count of what is coming in [22:39:07] not what is backed up [22:39:47] this is where I see if our queues are keeping up I guess https://grafana.wikimedia.org/dashboard/db/fundraising-overview?orgId=1&from=now-7d&to=now&refresh=1m&panelId=15&fullscreen [22:40:49] so am I right in understanding we’ve not had more than 500 donations backed up since we turned on banners [22:41:07] (from memory 10k would have been acceptable in my first BE) [22:42:14] ah maybe this is the graph we’ve watched in previous years to see q delay https://grafana.wikimedia.org/dashboard/db/fundraising-overview?orgId=1&from=now-7d&to=now&refresh=1m&panelId=21&fullscreen [22:42:59] no evidence of any spike in delays with Leanne doing the benevity import.... [22:47:55] (PS3) Eileen: Thank you updates [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/476199 (https://phabricator.wikimedia.org/T209895) [22:51:19] (PS1) Ejegg: Allow overriding form settings in LocalSettings.php [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/476418 [22:51:27] (PS4) Eileen: Thank you updates [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/476199 (https://phabricator.wikimedia.org/T209895) [22:52:04] fr-tech there's a little patch to deal with the fact we can't override FormSettings in LocalSettings currently: https://gerrit.wikimedia.org/r/476418 [22:58:16] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Slowish query being triggered by one of our jobs - https://phabricator.wikimedia.org/T210676 (Eileenmcnaughton) [23:01:27] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Slowish query being triggered by one of our jobs - https://phabricator.wikimedia.org/T210676 (Eileenmcnaughton) Found where it's being triggered ` try { if (array_key_exists('invoice_id', $contribution)) { watchdog('wmf_civicrm', 'Che... [23:47:59] Fundraising Sprint Kraftwerk, MediaWiki-extensions-CentralNotice: [bug] Banners adding themselves to some buckets - https://phabricator.wikimedia.org/T99915 (jrbs) I'm guessing this is still an issue, but I'm not sure it belongs with #trust-and-safety