[00:06:46] cstone: if you can +1 this when you get back I'll retest the number of queries too https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/civicrm/+/629230 (haven't deployed yet [00:57:45] I ran the first full-q test - Successfully processed 983 contribution(s). [00:58:08] it was slower than shorter runs (which could be a memory leak) [01:13:34] hey im back [01:13:41] nice eileen [01:14:04] I'll still waiting for enough to do the second test [01:14:17] with the changes deployed [01:25:34] (CR) Cstone: [C: +2] dev/core#2032 Add potential to vary query log files per process [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/629230 (owner: Eileen) [01:28:22] thanks - q at 600 [01:32:30] (Merged) jenkins-bot: dev/core#2032 Add potential to vary query log files per process [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/629230 (owner: Eileen) [01:35:48] (PS1) Eileen: Submodule CiviCRM update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/630977 [01:36:00] (CR) Eileen: [C: +2] Submodule CiviCRM update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/630977 (owner: Eileen) [01:42:33] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6837 5000, payments-init is 1764 1500, pending is 2041 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 36 days 11 hours - memory use is 12.83M (peak 46.50M, 0.22% of max, fragmentation 1.42%), connected_slaves is 2, donations is 711, jobs is 0, jobs-adyen is 3, jobs-paypal is 710, recurring is 4, refund is 1, unsubscribe [01:42:46] (Merged) jenkins-bot: Submodule CiviCRM update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/630977 (owner: Eileen) [01:45:04] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/630978 [01:45:13] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/630978 (owner: Eileen) [01:47:29] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 7062 5000, payments-init is 1824 1500, pending is 2100 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 36 days 11 hours - memory use is 13.21M (peak 46.50M, 0.22% of max, fragmentation 1.40%), connected_slaves is 2, donations is 770, jobs is 0, jobs-adyen is 3, jobs-paypal is 741, recurring is 4, refund is 1, unsubscribe [01:52:33] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 7238 5000, payments-init is 1869 1500, pending is 2145 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 36 days 11 hours - memory use is 13.54M (peak 46.50M, 0.23% of max, fragmentation 1.39%), connected_slaves is 2, donations is 814, jobs is 0, jobs-adyen is 3, jobs-paypal is 762, recurring is 4, refund is 1, unsubscribe [01:57:33] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 7340 5000, payments-init is 1894 1500, pending is 2181 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 36 days 11 hours - memory use is 13.69M (peak 46.50M, 0.23% of max, fragmentation 1.38%), connected_slaves is 2, donations is 839, jobs is 0, jobs-adyen is 3, jobs-paypal is 772, recurring is 4, refund is 1, unsubscribe [02:01:12] 852 queued up - can redis relate to the paused queues? [02:01:44] first test got through just shy of 1k [02:02:10] howdy [02:02:33] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 7491 5000, payments-init is 1934 1500, pending is 2221 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 36 days 11 hours - memory use is 13.94M (peak 46.50M, 0.23% of max, fragmentation 1.38%), connected_slaves is 2, donations is 878, jobs is 0, jobs-adyen is 3, jobs-paypal is 786, recurring is 4, refund is 1, unsubscribe [02:02:40] hi dwisehaupt we are testing [02:03:12] ah. cool. [02:03:17] i'll ack the alerts then. [02:05:08] ACKNOWLEDGEMENT - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 7491 5000, payments-init is 1934 1500, pending is 2221 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 36 days 11 hours - memory use is 13.94M (peak 46.50M, 0.23% of max, fragmentation 1.38%), connected_slaves is 2, donations is 878, jobs is 0, jobs-adyen is 3, jobs-paypal is 786, recurring is 4, refund is 1, uns [02:05:08] ehaupt Queue testing in progress [02:08:31] i'll hang out in case you need me. just sitting in on a school parent meeting. [02:17:59] dwisehaupt: what causes tem [02:18:12] the queues are off so we can backlog up some donations [02:18:42] I was wondering how that could cause redis issues [02:18:55] yeah. what we are seeing is a backlog in the queues. mostly the antifraud queue right now. [02:19:20] basically row three of the fundraising overview dashboard [02:19:39] this is expected with the queues stopped [02:20:02] ok - I can run the antifraud [02:20:37] so it will be happy with that run ? [02:21:21] honestly it's fine. nothing to worry about since it's a known test. [02:22:44] The donations q is about 1 k now - so that's probably enough but I'll wait a bit longer [02:27:16] ok - I've set the second test going [02:30:15] !log process-control config revision is 646817a2c0 [02:30:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:30:24] queues enabled again [02:32:22] 988 in 90 sec vs 983 [02:32:31] very marginal! [02:36:20] eileen: were you logged in when you tried to annotate on that dashlet? [02:36:30] hmm good question! [02:37:06] no - not sure I know how to but might leave that for tomorrow [02:38:31] you have an account. from the login page there is a link to reset your password. that should work. [02:40:02] ok I'll try in a bit - server should be back to normal now [02:40:13] also, if you want to wait you can always add the annotation later by going back in time on the graph. [02:40:17] cool. [02:40:30] cool [02:51:21] gotta get these kids to bed. [04:12:33] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 7 keys, up 36 days 13 hours - memory use is 3.44M (peak 46.50M, 0.11% of max, fragmentation 2.57%), connected_slaves is 2, donations is 470, jobs is 0, jobs-adyen is 0, jobs-paypal is 680, payments-antifraud is 19, payments-init is 8, pending is 0, recurring is 18, refund is 0, unsubscribe is 7 [05:03:46] (PS1) Eileen: Fix mistake that snuck in to query parser [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/630985 [06:52:47] fundraising-tech-ops, Operations, netops: Automate diff and commit of frack ACL - https://phabricator.wikimedia.org/T260655 (ayounsi) We can use Juniper permissions system to restrict which part of the config a user can change: https://www.juniper.net/documentation/en_US/junos/topics/topic-map/junos-... [09:20:27] Wikimedia-Fundraising-Banners, donate.wikimedia.org: Add Carte Bancaire logo and enable card type logos in France - https://phabricator.wikimedia.org/T204545 (Pcoombe) Resolved→Open Re-opening as Carte Bancaire is back in again. I don't know how important adding this logo to banners is considered... [14:23:28] Fundraising-Backlog: As a developer, I'd like the end of the silverpop export logs to be clear that it succeeded - https://phabricator.wikimedia.org/T264191 (mepps) [14:38:02] Wikimedia-Fundraising-Banners: France Campaign Banners (for launch) - https://phabricator.wikimedia.org/T263812 (jbolorinos-ctr) Open→Resolved All issues found have now been reported and addressed. Closing this task now as Resolved, these banners are now READY TO TEST! [14:38:30] Wikimedia-Fundraising-Banners: QA Dsk Lg - https://phabricator.wikimedia.org/T263882 (jbolorinos-ctr) All bugs found have been submitted and addressed. Closing task as Resolved, these banners are now READY TO TEST! [14:39:04] Fundraising-Backlog: As a developer, I'd like not to see errors at the end of the silverpop export log - https://phabricator.wikimedia.org/T264191 (mepps) [14:39:14] Wikimedia-Fundraising-Banners: France Campaign Banners (for launch) - https://phabricator.wikimedia.org/T263812 (jbolorinos-ctr) [14:39:17] Wikimedia-Fundraising-Banners: QA Dsk Lg - https://phabricator.wikimedia.org/T263882 (jbolorinos-ctr) Open→Resolved [14:39:38] Fundraising-Backlog: As a developer, I'd like not to see errors at the end of the silverpop export log - https://phabricator.wikimedia.org/T264191 (mepps) Even though there are success messages above, I missed them because the "ERROR" seemed troubling. [14:39:52] Wikimedia-Fundraising-Banners: France Campaign Banners (for launch) - https://phabricator.wikimedia.org/T263812 (jbolorinos-ctr) [14:39:57] Wikimedia-Fundraising-Banners: QA M Lg - https://phabricator.wikimedia.org/T263891 (jbolorinos-ctr) Open→Resolved All bugs found have been submitted and addressed. Closing this task now as Resolved, since these banners are now READY TO TEST! [14:40:23] Wikimedia-Fundraising-Banners: QA Dsk Sm - https://phabricator.wikimedia.org/T263894 (jbolorinos-ctr) All bugs found have been reported and addressed. Closing task now as Resolved, these banners are now READY TO TEST! [14:40:28] Wikimedia-Fundraising-Banners: QA Dsk Sm - https://phabricator.wikimedia.org/T263894 (jbolorinos-ctr) Open→Resolved [14:40:30] Wikimedia-Fundraising-Banners: France Campaign Banners (for launch) - https://phabricator.wikimedia.org/T263812 (jbolorinos-ctr) [14:41:02] Wikimedia-Fundraising-Banners: QA M Sm - https://phabricator.wikimedia.org/T263895 (jbolorinos-ctr) All bugs found have been submitted and addressed. Closing this task now as Resolved, these banners are now READY TO TEST! [14:41:08] Wikimedia-Fundraising-Banners: QA M Sm - https://phabricator.wikimedia.org/T263895 (jbolorinos-ctr) Open→Resolved [14:41:10] Wikimedia-Fundraising-Banners: France Campaign Banners (for launch) - https://phabricator.wikimedia.org/T263812 (jbolorinos-ctr) [14:49:46] Wikimedia-Fundraising-Banners: BETA banners Sep 22 - https://phabricator.wikimedia.org/T263602 (jbolorinos-ctr) [14:49:48] Wikimedia-Fundraising-Banners: [Enhancement][Banner icon variant] Move error messages above banner icons - https://phabricator.wikimedia.org/T263708 (jbolorinos-ctr) Open→Resolved Nice, thanks @scervantes! Verified this has now been updated {F32368909} [15:14:54] Wikimedia-Fundraising-Banners: QA for en6C pre-test on September 30 - https://phabricator.wikimedia.org/T263890 (jbolorinos-ctr) [15:14:56] Wikimedia-Fundraising-Banners: 2020-09-30 en6C pre-test: desktop large batch - https://phabricator.wikimedia.org/T263921 (jbolorinos-ctr) Open→Resolved a:jbolorinos-ctr All issues found have been reported and addressed. Closing this now as Resolved, these banners are READY TO TEST! [15:15:36] Wikimedia-Fundraising-Banners: 2020-09-30 en6C pre-test: mobile large batch - https://phabricator.wikimedia.org/T263892 (jbolorinos-ctr) Open→Resolved a:jbolorinos-ctr All bugs found have been reported and addressed. Closing task as Resolved, since these banners are now READY TO TEST! [15:15:38] Wikimedia-Fundraising-Banners: QA for en6C pre-test on September 30 - https://phabricator.wikimedia.org/T263890 (jbolorinos-ctr) [15:15:52] Wikimedia-Fundraising-Banners: 2020-09-30 en6C pre-test: mobile small batch - https://phabricator.wikimedia.org/T263893 (jbolorinos-ctr) Open→Resolved a:jbolorinos-ctr All bugs found have been reported and addressed. Closing task as Resolved, since these banners are now READY TO TEST! [15:16:06] Wikimedia-Fundraising-Banners: 2020-09-30 en6C pre-test: desktop small batch - https://phabricator.wikimedia.org/T263966 (jbolorinos-ctr) All bugs found have been reported and addressed. Closing task as Resolved, since these banners are now READY TO TEST! [15:16:17] Wikimedia-Fundraising-Banners: 2020-09-30 en6C pre-test: desktop small batch - https://phabricator.wikimedia.org/T263966 (jbolorinos-ctr) Open→Resolved a:jbolorinos-ctr [15:16:19] Wikimedia-Fundraising-Banners: QA for en6C pre-test on September 30 - https://phabricator.wikimedia.org/T263890 (jbolorinos-ctr) [15:16:28] Wikimedia-Fundraising-Banners: QA for en6C pre-test on September 30 - https://phabricator.wikimedia.org/T263890 (jbolorinos-ctr) Open→Resolved a:jbolorinos-ctr All bugs found have been reported and addressed. Closing task as Resolved, since these banners are now READY TO TEST! [17:12:08] (PS1) Ejegg: Fix maintenance script output [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/631217 [17:19:47] (CR) Jgleeson: [C: +2] Fix maintenance script output [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/631217 (owner: Ejegg) [17:20:15] (Merged) jenkins-bot: Fix maintenance script output [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/631217 (owner: Ejegg) [17:21:12] thanks jgleeson [17:21:28] (PS1) Ejegg: Break out recurring overall timing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/631218 (https://phabricator.wikimedia.org/T261689) [17:21:57] there's the other one jgleeson ^^^ [17:25:09] np, looking now [17:25:51] (CR) Jgleeson: [C: +2] "tested on call, looks good to me!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/631218 (https://phabricator.wikimedia.org/T261689) (owner: Ejegg) [17:25:54] fr-tech don't forget to vote on docker repo names at the bottom of https://etherpad.wikimedia.org/p/fr-tech-docker [17:26:37] ejegg: ^ thx! also dunno if there are any additional suggestions :) [17:26:55] I mean, if u have any, pls write 'em in [17:33:35] (Merged) jenkins-bot: Break out recurring overall timing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/631218 (https://phabricator.wikimedia.org/T261689) (owner: Ejegg) [17:37:46] Wikimedia-Fundraising-Banners: [Candidate] FR Banners due Aug 28 - https://phabricator.wikimedia.org/T259833 (mepps) [17:37:50] Fundraising Sprint Raw data never hurt anyone, Fundraising Sprint 🐍 is not a valid zipcode, Fundraising-Backlog, fundraising sprint Theme songs for programming languages, MW-1.36-notes (1.36.0-wmf.10; 2020-09-22): Missing spacing between : and text on p... - https://phabricator.wikimedia.org/T261439 [17:58:05] Fundraising-Backlog: Contributions in Civi currently not saving - https://phabricator.wikimedia.org/T264222 (LeanneS) [19:00:08] Fundraising-Backlog: Contributions in Civi currently not saving - https://phabricator.wikimedia.org/T264222 (Eileenmcnaughton) @mepps maybe I missed a beat on that soft-credit query patch - that's my first guess [19:06:25] hmm, db contention failmail [19:06:30] let's see [19:07:29] errr, I can't drush civicrm-sql-cli any more? [19:18:40] https://phabricator.wikimedia.org/T264229 [20:39:23] (PS1) Ejegg: Remove accidentally committed old civi drush file [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/631280 [20:40:05] eileen we figured it out ^^!! [20:40:20] (CR) Mepps: [C: +2] Remove accidentally committed old civi drush file [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/631280 (owner: Ejegg) [20:41:43] Fundraising-Backlog, fundraising sprint Theme songs for programming languages, FR-Q2-FY2020-21-cleanup-list: Make easy on/off switch for Monthly covert - https://phabricator.wikimedia.org/T250918 (mepps) a:mepps I might take a look at this tomorrow. [20:47:41] (Merged) jenkins-bot: Remove accidentally committed old civi drush file [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/631280 (owner: Ejegg) [20:55:59] ok, I'll update that submodule pointer and deploy [20:56:45] oh wow that looks silly! [20:57:00] (PS1) Ejegg: Update drupal submodule [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/631285 [20:57:15] (CR) Ejegg: [C: +2] Update drupal submodule [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/631285 (owner: Ejegg) [20:57:25] it makes sense why i couldn't get my debugging code the other drush/civicrm.drush.inc to run! [20:58:01] :-) [20:58:20] was it when I did the civi upgrade I just included an extra file? [20:58:44] I can't quite figure it out - how that happened [21:02:12] (PS1) Ejegg: Move csv upload button under format options [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/631286 [21:02:44] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/631287 [21:03:17] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/631287 (owner: Ejegg) [21:04:35] eileen: no, I added an extra file in the drupal upgrade [21:04:54] (Merged) jenkins-bot: Update drupal submodule [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/631285 (owner: Ejegg) [21:04:55] I had a local 'drush' subdir there [21:04:59] ejegg: ah OK- that explains why I'm not sure how I did it :-) [21:05:03] haha [21:05:09] yeah, my fault this time [21:05:17] gotta share it around :-) [21:13:42] mepps: just thinking about https://phabricator.wikimedia.org/T264222 - my money is on that soft credit query patch being involved & I'm thinking about reverting it & deploying to see - because that would give a quick fix if it works [21:14:25] Fundraising-Backlog: Contributions in Civi currently not saving - https://phabricator.wikimedia.org/T264222 (Eileenmcnaughton) @LeanneS @RLewis this is only with soft credits? [21:17:10] Fundraising-Backlog: Contributions in Civi currently not saving - https://phabricator.wikimedia.org/T264222 (LeanneS) @Eileenmcnaughton I think it's happening with all contributions. [21:18:38] eileen: I forgot to mention - I had to roll back your code to get rid of the db_switcher call [21:18:50] I did see that [21:19:03] it was still looking in the default db [21:19:04] nnext roud [21:19:09] not doing any prefixing [21:19:12] next round [21:19:49] !log updated fundraising CiviCRM from 6e843649ac to 5a53bfe6ed [21:19:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:21:39] mepps yep, now working on live [21:21:59] thanks again for the debugging [21:23:41] (PS1) Eileen: Fix processPCP error [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/631289 (https://phabricator.wikimedia.org/T264222) [21:24:07] ejegg: mepps ok - this new bug is ^^ - we should get that out asap [21:24:31] oh yeah, so that can be null, right? [21:25:00] ooh, does this relate to the benevity issue? [21:25:13] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages: Queue speed - the new frontier - https://phabricator.wikimedia.org/T264061 (Eileenmcnaughton) a:Eileenmcnaughton [21:25:31] I think benevity pre-dates - about to look [21:28:05] ejegg: that error affects adding contributions via the form [21:30:27] I'm going to make that one unbreak now [21:31:00] (CR) Ejegg: [V: +2 C: +2] "Yep, actual return value is void" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/631289 (https://phabricator.wikimedia.org/T264222) (owner: Eileen) [21:31:12] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages, Patch-For-Review: Contributions in Civi currently not saving - https://phabricator.wikimedia.org/T264222 (Eileenmcnaughton) p:Triage→Unbreak! [21:31:24] oops, overzealous with the V+2 [21:32:55] (PS1) Eileen: Submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/631294 [21:33:03] (CR) Eileen: [C: +2] Submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/631294 (owner: Eileen) [21:33:25] hmm, recurring charge job failing too [21:33:25] ejegg: well it has to pass another round of tests now anyway [21:33:30] is that the same thing? [21:33:35] probably not [21:33:55] it is the forms that were affected because they have a soft credit field [21:34:47] oh weird [21:34:51] Command cvapi needs the following extension(s) enabled to run: [error] [21:34:54] 2020-09-30 21:21:02,302 ERROR civicrm. [21:35:09] oh right, that should work next turn [21:35:29] I guess it ran just after I updated and before I drush cc all'ed [21:41:29] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/631297 [21:41:40] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/631297 (owner: Eileen) [21:41:58] :) [21:44:13] ejegg: also - on that query parsing - I found the change I had & lost locally that resulting in it failing when demo-ing https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/630985 [21:45:08] !log civicrm revision changed from 5a53bfe6ed to 256adda03c, config revision is 646817a2c0 [21:45:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:45:46] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages, Patch-For-Review: Contributions in Civi currently not saving - https://phabricator.wikimedia.org/T264222 (Eileenmcnaughton) OK - I just deployed a fix [21:46:15] pulling it down to look [21:46:38] it was something lost in git fu [21:47:31] (PS2) Ejegg: Fix mistake that snuck in to query parser [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/630985 (owner: Eileen) [21:47:38] (CR) Ejegg: [C: +2] Fix mistake that snuck in to query parser [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/630985 (owner: Eileen) [21:50:13] eileen: did you deploy those patches to save some DB queries on contact create? [21:51:23] ejegg: yep [21:51:42] hmm, still had a pretty big queue backlog in today's test [21:51:43] it did save some - quite a few - but impact on q speed was ho-hum [21:52:23] I have tracked down a memory leak https://github.com/civicrm/civicrm-core/pull/18641 but haven't put up a backport yet [21:52:58] ooh, nice investigation there [21:53:26] lol@branch name [21:53:34] beyond that we could do some php profiling - which might help - or we are at concurrent queues & better deadlock handling [21:53:54] :-) I'm glad you spotted it - sometimes I think I'm just amusing myself [21:54:27] I found a few things that I thought were leaks but were only excessive memoy usage in phpunit [21:54:49] turns out that phpunit can be a memory hog without it strictly being a leak [21:55:40] it doesn't deconstruct the test classes - so if they have properties that are objects they don't get released [22:04:55] (Merged) jenkins-bot: Fix mistake that snuck in to query parser [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/630985 (owner: Eileen) [23:17:05] dang, there are a LOT of paypal recurring payments in the damaged table [23:17:13] where we don't have the subscription record [23:17:59] (PS1) Eileen: Fix benevity import to currency convert fee amount [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/631314 (https://phabricator.wikimedia.org/T263744) [23:19:26] eileen: oho! we were fishing for a red herring with the thousands separator back in tech talk [23:19:29] nice find [23:19:58] ejegg: yeah -but wondering what data might be wrong - as fee_amount is wrong on ? old exports [23:22:37] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages, Patch-For-Review: Error message when importing a Benevity file - https://phabricator.wikimedia.org/T263744 (Eileenmcnaughton) @rlewis I tracked it down - but it's a bit problematic. Basic... [23:22:48] per ^^ comment [23:24:54] (CR) jerkins-bot: [V: -1] Fix benevity import to currency convert fee amount [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/631314 (https://phabricator.wikimedia.org/T263744) (owner: Eileen) [23:26:37] oh shoot, but I guess we can reconstruct that [23:27:18] so it would be the fee for every non-USD benevity donation [23:27:23] I guess we can find that [23:29:44] we store gateway as 'benevity' i think? [23:30:08] ug test fail - it's passing locally [23:30:24] need to set some exchange rates? [23:30:24] yep [23:30:33] maybe! [23:38:19] ok eileen, so there are 21 JPY / Benevity donations in that I can see [23:40:33] fees seem to be pretty small, so any of the currencies that are between 0.70 and 1.30 USD wouldn't be off by more than a few cents [23:40:59] ILS needs correcting, SEK [23:41:09] PLN [23:43:46] I guess we [23:44:05] 'd have to keep track of a 'fixed up till this date' variable [23:44:23] hmm yeah [23:44:24] since it won't be obvious from the data which are fixed and which aren't [23:44:53] I guess we could do a query on which ones are out by more then x [23:46:56] or even re-fixing would be OK - we can get the rate by original_currency / total_amount I suppose [23:47:24] ah, do we have an original_fee? [23:47:37] no, I think we only have one column for the fee [23:47:43] so we have to overwrite it [23:48:01] (PS2) Eileen: Fix benevity import to currency convert fee amount [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/631314 (https://phabricator.wikimedia.org/T263744) [23:48:09] and if we batch the fix we have to keep track of which are already done [23:48:26] total number affected is about 2.5k, so we might not need to batch [23:48:37] oh, or we could add a contribution tag [23:49:21] yeah 2.5 is probably batch-optional [23:49:53] ok, easier then [23:50:30] wrap it in a txn and stop the queues while running [23:54:46] (CR) jerkins-bot: [V: -1] Fix benevity import to currency convert fee amount [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/631314 (https://phabricator.wikimedia.org/T263744) (owner: Eileen) [23:55:19] yeah - sounds good - are you working on that? (I'm just trying to sort the test at the moment) [23:55:39] Wikimedia-Fundraising-Banners: Share links disappear after clicking back [Otherwaystogive variant] - https://phabricator.wikimedia.org/T264256 (jbolorinos-ctr) [23:56:11] Wikimedia-Fundraising-Banners: Share links disappear after clicking back [Otherwaystogive variant] - https://phabricator.wikimedia.org/T264256 (jbolorinos-ctr) [23:57:07] hadn't started, but i can! [23:57:31] (PS3) Eileen: Fix benevity import to currency convert fee amount [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/631314 (https://phabricator.wikimedia.org/T263744) [23:57:35] up to you [23:58:07] looking at wmf_civicrm_recalculate_contributions [23:59:20] hmm, I'mma have to sign off soonish