[00:36:22] Fundraising Sprint Raw data can give you salmonella, Fundraising Sprint Sasquatches can't find us either, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: RichText fields munged - https://phabricator.wikimedia.org/T204179 (Eileenmcnaughton) @NNichols looks like this has b... [00:57:10] (PS3) Ejegg: Ensure we're not matching on deleted contacts [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462044 (https://phabricator.wikimedia.org/T205134) [00:57:41] (CR) Ejegg: "Thanks Eileen, silly copypasta" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462044 (https://phabricator.wikimedia.org/T205134) (owner: Ejegg) [01:11:11] (CR) Eileen: [C: 2] Ensure we're not matching on deleted contacts [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462044 (https://phabricator.wikimedia.org/T205134) (owner: Ejegg) [01:16:59] (Merged) jenkins-bot: Ensure we're not matching on deleted contacts [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462044 (https://phabricator.wikimedia.org/T205134) (owner: Ejegg) [01:20:58] thanks eileen ! [01:21:09] no worries [01:21:13] I'll deploy that (along with the data update I was about to deploy earlier) [01:22:00] so they can track down via searching in trash all the old contribs that got glued to deleted contacts? [01:23:09] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/462614 [01:32:04] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/462614 (owner: Ejegg) [01:33:58] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/462614 (owner: Ejegg) [01:54:14] !log updated fundraising CiviCRM from b0223f56b0 to ca49aed673 [01:54:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:19:58] After tһe аcquisіtіഠᥒ by Private Ιntᥱrnеt Aⅽϲeѕѕ‚ Frᥱenode is now bеіᥒg uѕeԁ to рᥙsh ΙCⲞ sⅽаms һttⲣs：/᜵wwᴡ．coindesk．ϲoⅿ/hɑndshake－re∨ealеd-vcs-bɑck╴рlan-tо-ɡive－awаy-100˗ⅿiⅼlⅰon-iᥒ╴crỿptο/ [02:19:58] "Αⅼl tоld， Handsһаke ɑims to giᴠе $ᒿ50 wοrth of ⅰts tokеnѕ tο *eɑϲһ* user of the wеbsiteѕ tһe cഠmpɑny һɑs раrtᥒershipѕ with – GitΗub， thе PᒿΡ Fоundɑtiоᥒ and *FRΕEΝΟᎠE*‚ a сhat ϲhɑᥒᥒel for peеr-tο-pеer рrⲟϳᥱctѕ․ Aѕ ѕuсh, ԁеⅴеⅼoperѕ ᴡhο hɑve eⅹⅰstіᥒɡ acсounts οn еacһ could rᥱc [02:19:58] worth of Ηanⅾshаke tokеns．" [02:19:58] Hаndsһɑke crуptоcᥙrrency sϲam іs орerated bу Аᥒⅾrew Ⅼee (ᒿ76-88－053Ꮾ), tһе frɑ∪ⅾѕter in сhⅰеf at Рriⅴatᥱ Intеrᥒᥱt Accesѕ wһich ᥒow owᥒs ᖴreenode [02:40:00] ejegg: so what I think I’m looking at doing with benevity etc import is switching to using drupal batch api https://hardcoredev.com/blog/using-batch-api-in-drupal-7-tutorial-in-simple-code/ so it has a progress bar & won’t time out [02:40:37] I think I might need to switch to using this csv package https://csv.thephpleague.com/ & so some amount of refactoring [03:11:10] eileen: oops, sorry, went afk without saying [03:11:25] no worries at all - [03:11:43] ooh, nifty - never heard of the batch API [03:12:27] sounds like just the thing [03:13:53] (PS1) Eileen: Switch variables to being class properties [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462617 [03:14:12] ejegg: I put up a patch but I just had a thought - in the loop we do [03:14:13] set_time_limit(10); [03:14:33] but from my testing the time to save a contact with 18k donations is around 13 seconds [03:14:52] eileen: ooh, let's increase that then! [03:15:01] - we only have perhaps 3 contacts with that many - but they are regularly touched by Benevity import [03:15:12] shall I go to 30 sec? [03:15:16] or just 20? [03:15:22] Sure, no harm in 30 that I can see [03:15:47] (TBH I”m not 100% sure about where the timeout occurs in that I *think* sql time doesn’t count in php time - but no harm trying) [03:17:33] (PS2) Eileen: Switch variables to being class properties [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462617 [03:17:35] (PS1) Eileen: Increase timeout per row on import (some contacts with many rows take > 10 sec to recalc wmf_donor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462618 [03:23:10] (CR) jerkins-bot: [V: -1] Increase timeout per row on import (some contacts with many rows take > 10 sec to recalc wmf_donor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462618 (owner: Eileen) [03:24:09] (CR) jerkins-bot: [V: -1] Switch variables to being class properties [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462617 (owner: Eileen) [03:26:00] (PS2) Eileen: Increase timeout per row on import (some contacts with many rows take > 10 sec to recalc wmf_donor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462618 [03:26:02] (PS3) Eileen: Switch variables to being class properties [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462617 [03:26:04] (PS1) Eileen: Use class properties rather than passing vars [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462619 [03:32:35] (PS4) Eileen: Switch variables to being class properties [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462617 [03:32:38] (CR) jerkins-bot: [V: -1] Switch variables to being class properties [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462617 (owner: Eileen) [03:33:26] (CR) jerkins-bot: [V: -1] Use class properties rather than passing vars [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462619 (owner: Eileen) [03:33:28] Looks like there'll be plenty to review tomorrow morning! [03:33:54] signing off for the evening. have a good night eileen and AndyRussG ! [03:34:06] ejegg: see ya [03:38:33] (CR) Eileen: WIP Endowment-specific financial type and no_ty (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462575 (https://phabricator.wikimedia.org/T203603) (owner: Ejegg) [03:41:00] (PS2) Eileen: Use class properties rather than passing vars [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462619 [03:46:44] (PS1) Eileen: Extract importRow function. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462620 [03:52:27] (CR) jerkins-bot: [V: -1] Extract importRow function. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462620 (owner: Eileen) [03:59:03] (CR) AndyRussG: "Hey... Quick question here... Could someone explain why almost all the assert.expect() calls were removed with this patch? Thanks much!!!" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/456422 (owner: Esanders) [04:10:31] (PS2) Eileen: Extract importRow function. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462620 [04:16:32] (CR) jerkins-bot: [V: -1] Extract importRow function. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462620 (owner: Eileen) [04:20:09] (PS3) Eileen: Extract importRow function. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462620 [04:27:04] Fundraising Sprint Raw data can give you salmonella, Fundraising Sprint Sasquatches can't find us either, Fundraising-Backlog, Epic: Update EventLogging schema for CentralNotice - https://phabricator.wikimedia.org/T203499 (AndyRussG) Ooops! I was mistaken... This is not necessary, since we curren... [04:27:21] Fundraising Sprint Asymmetrical Earth Theory, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising-Backlog, Epic: [Epic] Fundraising kafkatee changes - https://phabricator.wikimedia.org/T183978 (AndyRus... [04:27:23] Fundraising Sprint Raw data can give you salmonella, Fundraising Sprint Sasquatches can't find us either, Fundraising-Backlog, Epic: Update EventLogging schema for CentralNotice - https://phabricator.wikimedia.org/T203499 (AndyRussG) Open>Invalid [05:20:11] Аftеr tһе acqᥙіsition by Pri∨atᥱ Internet Αccess, Freᥱᥒoԁe іs noᴡ bеіnɡ ᥙseԁ to рush IϹO scɑms https:／/www．cοindеsk.cοm／hɑᥒԁѕhɑkе╴re⋁eаⅼеd﹣ᴠcѕ-bɑck╴plan－to˗give-away-100⎼mіlⅼioᥒ-iᥒ-crỿptⲟ／ [09:23:41] "Aⅼl toⅼd, Ηanԁshakе aіmѕ tο ɡivе ＄250 ᴡortһ ⲟf its tοkenѕ to ＊each* ᥙser of the webѕⅰteѕ the cഠⅿрanỿ hɑѕ partᥒerѕhiрѕ witһ – GⅰtHub, ... [10:08:18] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Patch-For-Review: Port CentralNotice Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180223 (zeljkofilipin) [11:35:30] Fundraising Sprint Sasquatches can't find us either, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Support Tracking endowment contributions in civi - https://phabricator.wikimedia.org/T203601 (Pcoombe) @Ejegg I think utm_medium probably makes the most sense. [12:13:41] ＂Aⅼl tοldᛧ Ηandѕһake ɑimѕ tο give ﹩ᒿ50 wഠrth οf іtѕ tokens to *еacһ＊ ᥙsеr of the ᴡеbsites thе comⲣanу һas рartnershіps with – ᏀitᎻ∪b， thᥱ P2Р Fouᥒdatⅰon and *FRΕENОDЕ*, a chat channᥱⅼ for pеᥱr-to-peᥱr prഠjeϲtѕ． As suсh, ... [13:14:40] Fundraising-Backlog, fundraising-tech-ops, Operations, Traffic: SSL cert for links.email.wikimedia.org - https://phabricator.wikimedia.org/T188561 (Jgreen) @CCogdill_WMF @BBlack there's no change as far as I can see in Trilogy's SSL rating according to Qualys, still a B with the main issues being... [13:27:01] Fundraising-Backlog, fundraising-tech-ops, Operations, Traffic: SSL cert for links.email.wikimedia.org - https://phabricator.wikimedia.org/T188561 (BBlack) I would choose not to proceed with a vendor who cares so little about security. The "Weak DH" issue, in particular, made security headlines... [14:51:10] "Αⅼⅼ toⅼd, Ⲏɑnԁshake ɑimѕ to give ﹩ᒿ50 worth of its tⲟkens to ﹡each﹡ uѕеr of thе webѕіtеs the cοⅿpaᥒу һas partᥒershiⲣs ᴡith – GitΗub, the PᒿP ᖴounԁatіഠn аᥒd *ᖴᏒEᎬNOⅮE⋆, ... [15:06:43] Fundraising-Backlog, FR-Smashpig: Undefined property: SmashPig\Core\Logging\LogEvent::$exception @LogEvent.php:102 - https://phabricator.wikimedia.org/T205446 (Ejegg) [15:15:52] Fundraising Sprint Sasquatches can't find us either, Fundraising-Backlog, Patch-For-Review: Contacts and Imported gifts going straight to trash after import - https://phabricator.wikimedia.org/T205134 (Ejegg) Ok @NNichols and @LeanneS , there's a fix up for both Benevity and Engage imports so they sh... [15:22:16] PROBLEM - check_puppetrun on bismuth is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 44 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ssh] [15:27:16] PROBLEM - check_puppetrun on bismuth is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 49 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ssh] [15:32:16] RECOVERY - check_puppetrun on bismuth is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [15:51:26] fundraising-tech-ops: fix aiderator bug around initializing a new host - https://phabricator.wikimedia.org/T170342 (Jgreen) Open>Resolved fixed [15:51:56] fundraising-tech-ops, Operations, ops-codfw, Patch-For-Review: move/setup/install frauth2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T204079 (Jgreen) a:Jgreen>cwdent [15:52:38] fundraising-tech-ops: EPIC: migrate fundraising hosts to Debian Stretch - https://phabricator.wikimedia.org/T185013 (Jgreen) [16:04:45] Wikimedia-Fundraising, MediaWiki-extensions-NavigationTiming, Performance-Team: Track when a CentralNotice banner was displayed to the user in NavTiming - https://phabricator.wikimedia.org/T195840 (AndyRussG) @Peter, @Gilles, looks great!!! :) [16:15:43] Fundraising-Backlog, MediaWiki-Cache, MediaWiki-extensions-CentralNotice, Performance-Team (Radar): Mediawiki namespace pages, including CentralNotice banners, are slow to save - https://phabricator.wikimedia.org/T158084 (AndyRussG) This may well be a less severe symptom as the same problem that'... [17:01:11] Fundraising Sprint Raw data can give you salmonella, Fundraising Sprint Sasquatches can't find us either, Fundraising-Backlog, Language-Team, and 4 others: Save times for changes to translation variable text in centralnotice paralysingly slow - https://phabricator.wikimedia.org/T203925 (Imarlier) [17:12:37] XenoRyet: tech talk? [17:12:48] hoping to get folks set up to test Amazon donations [17:12:50] oh, sorry, lost track of the time. [17:12:56] brt [17:29:02] hello [17:54:35] MBeat and dstrine We'll be putting up France for the pre-test in the next 1/2 hour, for 2 hours. Let me know if you have any concerns, thanks! [17:54:54] ty tskaff [17:57:36] tskaff: none. break a leg! [18:20:27] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Extend deletion to multiple silverpop databases - https://phabricator.wikimedia.org/T205332 (CCogdill_WMF) To add more context, we found that IBM only runs the Forget Me job on a per database level. If a record also exists in another database, i.e. Majo... [18:53:45] hello [19:06:46] The one who wants to speak with me [19:19:06] Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice: Document CentralNotice-Translate integration - https://phabricator.wikimedia.org/T205477 (AndyRussG) [19:25:58] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Patch-For-Review: Port CentralNotice Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180223 (AndyRussG) Hi @zeljkofilipin... Many apologies for not responding here, and thanks for the sample test!!!!! Can you say how ur... [19:29:41] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Validate CentralNotice translatable message names/fix errors on messages with spaces - https://phabricator.wikimedia.org/T205478 (AndyRussG) [19:44:40] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Fix CentralNotice rebuilding of Translate message cache - https://phabricator.wikimedia.org/T205483 (AndyRussG) [19:52:28] Fundraising Sprint Sasquatches can't find us either, Fundraising-Backlog, Patch-For-Review: Contacts and Imported gifts going straight to trash after import - https://phabricator.wikimedia.org/T205134 (NNichols) I was able to restore all of the gifts that came from engage but if I run the same search... [20:12:10] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, and 5 others: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591 (AndyRussG) [20:12:16] Fundraising Sprint Raw data can give you salmonella, Fundraising Sprint Sasquatches can't find us either, Fundraising-Backlog, Language-Team, and 4 others: Save times for changes to translation variable text in centralnotice paralysingly slow - https://phabricator.wikimedia.org/T203925 (AndyRussG) [20:30:30] Fundraising Sprint Raw data can give you salmonella, Fundraising Sprint Sasquatches can't find us either, Fundraising-Backlog, Language-Team, and 4 others: Save times for changes to translation variable text in centralnotice paralysingly slow - https://phabricator.wikimedia.org/T203925 (Imarlier)... [20:42:11] hello [21:04:12] Fundraising-Backlog, MediaWiki-Cache, MediaWiki-extensions-CentralNotice, Performance-Team (Radar): Mediawiki namespace pages, including CentralNotice banners, are slow to save - https://phabricator.wikimedia.org/T158084 (Imarlier) Analysis at https://phabricator.wikimedia.org/T203925#4616856 sug... [21:05:21] Fundraising Sprint Raw data can give you salmonella, Fundraising Sprint Sasquatches can't find us either, Fundraising-Backlog, Language-Team, and 4 others: Save times for changes to translation variable text in centralnotice paralysingly slow - https://phabricator.wikimedia.org/T203925 (Imarlier)... [21:12:54] dstrine fr-tech Ian from Performance has dug into the Translate bug and seems to have found the cause in core [21:12:58] ^ [21:13:06] AndyRussG: yay! [21:13:10] fetching all of the things, one by one! [21:13:20] every single one of the things [21:13:23] we want them all!!!! [21:13:33] but not in one bite [21:18:03] small mouthfulls [21:18:06] chew carefully [21:34:40] Fundraising Sprint Raw data can give you salmonella, Fundraising Sprint Sasquatches can't find us either, Fundraising-Backlog, Core-Platform-Team, and 5 others: Save times for changes to translation variable text in centralnotice paralysingly slow - https://phabricator.wikimedia.org/T203925 (Fjala... [21:46:40] hello [21:59:31] (CR) Ejegg: [C: 2] Increase timeout per row on import (some contacts with many rows take > 10 sec to recalc wmf_donor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462618 (owner: Eileen) [21:59:41] ���+���-&ŰF� [21:59:43] ��w==kV߻�_�w$N�#?���&��Zl�4N�5�x뤃 �8W� ]s [21:59:46] � [21:59:49] o/#d ��f/ [21:59:51] ����6 %�JT3���<�L�}��u�2�� �C���Kɹ�uQ�(������ [21:59:52] T3: Upgrade logstash elasticsearch instance to latest version - https://phabricator.wikimedia.org/T3 [22:00:44] (CR) Ejegg: [C: 2] Switch variables to being class properties [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462617 (owner: Eileen) [22:05:53] (Merged) jenkins-bot: Increase timeout per row on import (some contacts with many rows take > 10 sec to recalc wmf_donor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462618 (owner: Eileen) [22:07:01] ejegg: I’ve kinda dithered about that deadlock patch because I think it will be a dog to upstream but it looks like we have a weekly jump in deadlocks https://grafana.wikimedia.org/dashboard/db/fundraising-database?panelId=29&fullscreen&edit&orgId=1&from=now-90d&to=now&tab=general - so I guess everytime we run a one hour test we get a fistful & so it must be worth trying [22:07:18] (Merged) jenkins-bot: Switch variables to being class properties [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462617 (owner: Eileen) [22:07:22] yeah? [22:07:28] Let's go for it! [22:08:52] we have a test tomorrow so we can look to see if we still get a jump [22:09:31] (PS2) Ejegg: Remove On Duplicate UPDATE for new inserts to improve performance & reduce deadlocks [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/459667 (owner: Eileen) [22:10:09] (CR) Ejegg: [C: 2] Remove On Duplicate UPDATE for new inserts to improve performance & reduce deadlocks [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/459667 (owner: Eileen) [22:10:34] eileen: yeah, I feel like if that patch is going to cause failures they'll be pretty obvious [22:10:38] Also looking at the Endowment triggers - I think I handled ‘INSERT” and ‘UPDATE’ but not ‘merge’ & ‘delete’ - I knew I hadn’t attempted to do it fullly & I think if I do a test for delete & merge then that will cover it off [22:11:14] ejegg: what worries me about the patch WRT upstreaming is that we actually had to make a change to accomodate it - albeit in code that was fairly weird [22:11:48] still upstreaming will be my challenge at the sprint - to see if I can get more of our stuf upstreamed [22:11:55] there are a couple of PRs in the queu [22:12:41] I’ll put this one on staging - https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/461507/ & we can see if that problem you hit is visible there [22:14:58] ah, thanks! [22:18:14] ejegg: I ust put it on staging but have to sort out my browser cert before I can load staging (I have the password now - just getting chrome to use the new cert) [22:18:37] looking now [22:19:08] (Merged) jenkins-bot: Remove On Duplicate UPDATE for new inserts to improve performance & reduce deadlocks [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/459667 (owner: Eileen) [22:22:40] I am so ready to kill that ‘Quick Autoreviewed’ tag [22:25:23] eileen: looks like the same problem on staging - the groups control shows the one I've added myself to in display mode [22:25:40] but clicking edit shows just the save and cancel buttons [22:26:04] (PS1) Eileen: CiviCRM submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462824 [22:28:33] ejegg: I agree the groups is not looking right [22:28:55] I’ll try to report upstream - but we don’t actually need them to work for MG so I think we should push out the update [22:29:05] since it does fix one thing (tag colours) [22:29:25] right, the groups are still working fine on their dedicated tab [22:30:33] yeah - the fixes on them was kinda tangental. I’ll try to figure out enough to log but I think we can ignore [22:31:04] (CR) Eileen: [C: 2] CiviCRM submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462824 (owner: Eileen) [22:36:58] (Merged) jenkins-bot: CiviCRM submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462824 (owner: Eileen) [22:43:45] ejegg: oddly enough when I switch to my ‘master’ install of civi with all latest that groups thing still works - perhaps there is something in core or apiv4 not yet ported back [22:44:15] (but I guess that will work it’s way through to us by the time we really care) [22:58:29] oho [22:59:39] (PS3) Ejegg: Use class properties rather than passing vars [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462619 (owner: Eileen) [22:59:44] (CR) Ejegg: [C: 2] Use class properties rather than passing vars [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462619 (owner: Eileen) [23:02:27] (PS4) Ejegg: Extract importRow function. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462620 (owner: Eileen) [23:02:57] (CR) Ejegg: [C: 2] Extract importRow function. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462620 (owner: Eileen) [23:04:57] (PS3) Ejegg: Further upstream fixes for contact layout editor. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/461507 (https://phabricator.wikimedia.org/T203266) (owner: Eileen) [23:06:20] (Merged) jenkins-bot: Use class properties rather than passing vars [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462619 (owner: Eileen) [23:06:34] (CR) Ejegg: [C: 2] "Groups control is broken with existing civicrm submodule, but seems to be working with Civi master. Tags are fixed, and those are what mat" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/461507 (https://phabricator.wikimedia.org/T203266) (owner: Eileen) [23:07:49] eileen: want to un-WIP this one? https://gerrit.wikimedia.org/r/461248 [23:08:30] ejegg: yeah OK - my only reservation is upstreaming pain potential but let’s roll it out & I’ll worry about that later [23:08:51] ah, yeah [23:09:05] (Merged) jenkins-bot: Extract importRow function. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462620 (owner: Eileen) [23:09:58] (PS2) Eileen: Fix for cache miss [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/461248 (https://phabricator.wikimedia.org/T204731) [23:10:02] ejegg: ^^ [23:10:15] cool [23:12:23] (CR) Ejegg: [C: 2] Fix for cache miss [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/461248 (https://phabricator.wikimedia.org/T204731) (owner: Eileen) [23:12:39] (PS2) Ejegg: Remove another cache bypass [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/461257 (owner: Eileen) [23:13:26] ejegg: I think those bypasses mostly matter if we switch to Redis - & perhaps are temporary depending how TIm goes after are last conversation [23:13:35] but it might still be worth switching to Redis - hmm [23:13:41] yeah! [23:14:00] cwd: is redis available on prod Civi? [23:14:09] (local redis, for caching?) [23:14:38] (Merged) jenkins-bot: Further upstream fixes for contact layout editor. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/461507 (https://phabricator.wikimedia.org/T203266) (owner: Eileen) [23:16:26] ejegg: no [23:16:49] is there enough evidence that it's faster to do it on prod yet? [23:18:32] cwd the queue won’t be faster - but UI can be faster [23:18:51] from testing the best outcome is queue is same speed [23:19:21] (we should test that again on staging before enabling Redis on live as we have made changes) [23:19:56] (Merged) jenkins-bot: Fix for cache miss [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/461248 (https://phabricator.wikimedia.org/T204731) (owner: Eileen) [23:22:41] (Merged) jenkins-bot: Remove another cache bypass [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/461257 (owner: Eileen) [23:25:18] ejegg: thanks - I’ll merge to deploy now - is there anything else that should go out? [23:26:13] (PS1) Eileen: CiviCRM submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462830 [23:26:22] (CR) Eileen: [C: 2] CiviCRM submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462830 (owner: Eileen) [23:32:06] (PS1) Eileen: WIP endowment trigger fix [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462831 (https://phabricator.wikimedia.org/T203601) [23:32:11] eileen: nope, that's all! [23:32:26] ejegg: ok - deploying now [23:33:56] so anything but best outcome is the queue gets slower? [23:34:21] maybe slightly, due to looking things up in redis vs looking things up in static arrays [23:34:43] but its a heck of a lot less contention for the db between UI and queue [23:35:09] and should mean fewer deadlocks & failmails [23:35:33] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/462832 [23:36:01] (CR) Eileen: [C: 2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/462832 (owner: Eileen) [23:36:58] cwd ejegg I think we have put static caches in to prevent extra Redis lookups & we have a plan (ie. make Tim do it) for that to be maintainable going forwards [23:37:00] ^����zꂗb��Q�T�����9� � [23:37:03] 2?x�xi����BCB�B �;+��sQ���s�{�x�:�4�L' [23:37:06] ��u;�p����N�^o�?�6h\��l'� �(;#�A��W� ׍�� [23:37:31] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/462832 (owner: Eileen) [23:38:58] i thought i had heard queue speed was a concern at the moment as far as keeping up w/ big english? [23:39:21] We got it speeded up a fair bit [23:39:46] and there are 3 more patches in the thing eileen's putting out now that should help even more [23:40:25] ejegg: I think those patches are mostly about prevening Redis slowing it down from memory [23:40:30] perhaps not the deadlocks one [23:40:37] ah right [23:40:48] cwd we did have a big dip in queue speed a while back & it’s back on track now [23:40:59] cool [23:41:13] although part of ‘back on track’ is we assume the contribution is the latest if it is ‘today' [23:41:45] selecting over a smaller range? [23:41:49] if the queue backs up during BE we might need to tweak that to assume the day actually importing is the latest [23:42:02] so - the issue is the ‘last_donation_date’ field [23:42:15] when we import a donation do we assume it’s the most recent or check [23:42:28] as of now we only assume if the date is ‘today' [23:42:48] aah, the big select is too expensive? [23:42:54] yeah [23:43:08] was there some threshold where that changed? [23:43:25] well to go back earlier - we used to always assume it was the latest [23:43:28] just curious, doubt i'll think of anything you have not [23:43:33] (with some inaccuracy resulting) [23:43:38] (CR) jerkins-bot: [V: -1] WIP endowment trigger fix [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/462831 (https://phabricator.wikimedia.org/T203601) (owner: Eileen) [23:45:26] My thinking is ‘today’ is an ok assumption but if we found ourselves with older items backed up the queue would slow down & we should consider altering it to ‘in the last 3 days’ at that point [23:47:02] wouldn't that actually slow it down by widening the search? [23:47:26] or is the slow part updating the field? [23:48:25] cwd no so what we do is we update the field to be the date of the contribution if it is today [23:48:40] if it’s not today then we do a calc to figure out the latest contribution date [23:48:52] ah ok i get it [23:49:05] so if the queue were backed up by 2 days [23:49:24] what about a set_latest_contribution_date queue? [23:49:25] it might make sense as an intervention to mess with the ‘should I calculate’ cut off [23:49:46] cwd you mean go through as a separate job? I worry people use the field in real time [23:49:56] at this stage queue speed looks good [23:50:18] when we added the ‘don’t check for today’ the queue backlog didn’t come back [23:50:27] yeah [23:50:30] works for me [23:50:47] if you could make a task i can get redis on civi tomorrow [23:50:53] got some painting to do tonight [23:50:54] I’m kinda being detailed about it so we all know we have that option - not because I think we need it [23:51:08] yeah i appreciate it [23:51:11] cwd - so if you put Redis on it would be there but not in use [23:51:28] until you flip a switch? [23:52:15] yeah - or add some config I guess [23:53:02] ��q��ZN���Xq�S����q�5Y��C�+���2�|���>�/I��� [23:53:05] ����!,N�¡�4�Σ��g���S]b���S��b�����ća/�II [23:53:05] !log civicrm revision changed from ca49aed673 to 018ff6325f, config revision is e71b8fdbdf [23:53:09] �sC���-=�i>ڴK,��RC �4 [23:53:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:53:12] >�A##H�\���F�tp#�udR;z;)�ޘ����h�o�mZ,m�Gk'�F�� [23:53:55] �G��mFs9'��(g�p�]j�1��d<�ŭ�PE�s���4�ٷ�RV�8��֓ [23:53:58] ���D~�W�[ �,�^�6�ځ�~U���6XK�'��� /���`"�6 [23:54:01] �8�+{�f6�����;���6�����v�]A�vJ����Ä���w\Q� [23:54:04] �r�A��K!9ە׵�� ��?�c� �cG����{C���c����NY [23:54:06] �=3~�A��v [23:55:10] eileen: cool well i won't hurry but it's not a lot of work [23:55:28] cwd - OK - I’ll add a phab anyway [23:55:47] thanks [23:56:32] Fundraising Sprint Sasquatches can't find us either, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Benevity import errors - https://phabricator.wikimedia.org/T204008 (Eileenmcnaughton) Just deployed the deadlock fix - state of deadlocks as of deploy {F26203116} https://grafana.wikimedia.org/da... [23:58:03] eileen: did you see the grafana deadlock dashboard? useful? [23:58:31] cwd yep looks like it’s cumulative https://grafana.wikimedia.org/dashboard/db/fundraising-database?panelId=29&fullscreen&edit&orgId=1&from=now-90d&to=now&tab=general [23:59:06] yep [23:59:07] but I’m hoping that if we look during tomorrows 1 hour test we won’t see that jump we’ve seen other weeks [23:59:18] they are infrequent enough that it's hard to graph [23:59:46] yeah - my take on it from that graph is that every time we run a one hour test there is a big jump