[00:08:13] Fundraising Sprint RadioActivewear, Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Message with blank payment_method and amount = 0 sent to payments-init - https://phabricator.wikimedia.org/T173347#3633847 (Ejegg) Open>Resolved [00:08:41] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising-Backlog, FR-Ingenico, and 3 others: spike: investigate creating an ingenico form with no city and state - https://phabricator.wikimedia.org/T151769#3633861 (Ejegg) Open>Resolved [00:08:56] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising-Backlog, MW-1.30-release-notes (WMF-deploy-2017-09-19 (1.30.0-wmf.19)), and 2 others: how to get back to A/B testing? - https://phabricator.wikimedia.org/T173869#3633865 (Ejegg) Open>Resolved [00:21:47] Fundraising Sprint Hermit Crab Husbandry, Fundraising Sprint Internet Exploring, Fundraising-Backlog, MW-1.28-release (WMF-deploy-2016-05-10_(1.28.0-wmf.1)), Patch-For-Review: Style fields more cleanly for when donor enters form without currency... - https://phabricator.wikimedia.org/T113306#3633901 [00:42:18] (PS4) Eileen: The big group clean up. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379463 (https://phabricator.wikimedia.org/T174407) [00:42:20] (PS1) Eileen: Changes to group clean up based on feedback from Caitlin. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/380667 (https://phabricator.wikimedia.org/T174407) [00:42:42] (Abandoned) Eileen: Group follow up. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379465 (https://phabricator.wikimedia.org/T174407) (owner: Eileen) [00:43:50] (PS2) Eileen: The big group clean up. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/380667 (https://phabricator.wikimedia.org/T174407) [00:44:22] (Abandoned) Eileen: The big group clean up. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379463 (https://phabricator.wikimedia.org/T174407) (owner: Eileen) [00:49:24] Fundraising-Analysis, Fundraising-Backlog, Wikimedia-Fundraising: Review 'Do not email ever' group in CiviCRM - https://phabricator.wikimedia.org/T176690#3633929 (Eileenmcnaughton) [00:54:25] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Look at creating job to clear tmp dir in Civi - https://phabricator.wikimedia.org/T176691#3633945 (Eileenmcnaughton) [01:04:13] (PS1) Eileen: Fix test to meet php standard. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/380668 [01:18:07] (PS2) Eileen: When updating an email return early on email match & set on_hold otherwise. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379467 (https://phabricator.wikimedia.org/T170350) [01:34:25] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Prevent showing CentralNotice on Special pages and History/Diff pages to partially resolve banner-bump issues - https://phabricator.wikimedia.org/T176656#3634045 (Quiddity) a:Jseddon>None [01:44:08] (PS3) Eileen: When updating an email return early on email match, set on_hold to 0 if email is not on hold in DB already. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379467 (https://phabricator.wikimedia.org/T170350) [01:45:14] Fundraising Sprint RadioActivewear, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email, Patch-For-Review: Reset on_hold in wmf_civicrm_message_email_update - https://phabricator.wikimedia.org/T170350#3634075 (Eileenmcnaughton) OK - I've pushed up a version that a) checks existing em... [02:08:22] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Fr-CiviCRM-dedupe-FY2017/18: Improve dedupe handling of Country only addresses - https://phabricator.wikimedia.org/T176699#3634083 (Eileenmcnaughton) [02:11:26] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Fr-CiviCRM-dedupe-FY2017/18: Cleanup of upstream merge code by extensionising - https://phabricator.wikimedia.org/T176700#3634099 (Eileenmcnaughton) [02:12:54] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Make CRM version stamp thing robust on dev boxes - https://phabricator.wikimedia.org/T88376#3634115 (Eileenmcnaughton) @DStrine @Ejegg - I think this can be closed out. I'm sure Adam knew what he meant.... [02:24:53] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Changing contact records from individual records to organization records - https://phabricator.wikimedia.org/T156193#3634118 (Eileenmcnaughton) [02:24:55] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Ensuring converted organization records are consistent - https://phabricator.wikimedia.org/T156194#3634117 (Eileenmcnaughton) [02:25:28] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Fr-CiviCRM-dedupe-FY2017/18, Epic: Epic: Dedupe V2: resolve top conflicts - https://phabricator.wikimedia.org/T143057#3634120 (Eileenmcnaughton) [02:25:30] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Changing contact records from individual records to organization records - https://phabricator.wikimedia.org/T156193#2966913 (Eileenmcnaughton) [02:26:37] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic: [EPIC] dedupe exact matches - https://phabricator.wikimedia.org/T133807#2244813 (Eileenmcnaughton) @DStrine we should close this one shouldn't we - seems like the last epic [02:27:25] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Fr-CiviCRM-dedupe-FY2017/18, Epic: Epic: Dedupe V2: resolve top conflicts - https://phabricator.wikimedia.org/T143057#2555439 (Eileenmcnaughton) [02:28:07] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Fr-CiviCRM-dedupe-FY2017/18: Investigate weird email conflicts - https://phabricator.wikimedia.org/T176701#3634126 (Eileenmcnaughton) [02:31:22] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Fr-CiviCRM-dedupe-FY2017/18: Add functionality to be able to do mesh dedupes on selected search results - https://phabricator.wikimedia.org/T176702#3634139 (Eileenmcnaughton) [02:48:00] Fundraising Sprint RadioActivewear, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Update CiviCRM ganglia bits to point to Prometheus - https://phabricator.wikimedia.org/T171524#3634154 (Jgreen) >>! In T171524#3633641, @Ejegg wrote: > @jgreen or @cwdent : Where should we be... [02:49:08] Fundraising-Backlog, fundraising-tech-ops: start testing mariadb 10.1.23 for fundraising - https://phabricator.wikimedia.org/T176489#3634155 (Jgreen) frdb1003 is up and running stretch with stock mariadb 10.1.23, we'll see how it does [02:57:45] (PS1) Eileen: CRM-21006 Ensure that Buttons are correctly displayed [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380670 [02:57:47] (PS1) Eileen: CRM-21006 - Escape all title and alt attributes [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380671 [03:10:23] (PS1) Eileen: CRM-21059 Upgrade Smarty to 2.6.30 [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380672 [03:15:44] (PS1) Eileen: CRM-21014 Ensure that preimum name is EOU [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380676 [03:16:57] (PS1) Eileen: CRM-20907 Ensure that contact_type is valid in deduperules [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380677 [03:29:03] (PS1) Eileen: CRM-21123 Ensure that the selectedChild on message templates is one of allowed types [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380679 [03:29:05] (PS1) Eileen: CRM-20895 - Encode display of search criteria [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380680 [03:29:07] (PS1) Eileen: CRM-20906 Validate the extension key is of general standard [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380681 [03:33:03] (PS1) Eileen: CRM-21022 - Parameterize variables in SQL query [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380682 [13:20:04] Fundraising-Backlog, fundraising-tech-ops: fundraising database replication lag master thread - https://phabricator.wikimedia.org/T173472#3635214 (Jgreen) Here's an anecdote Eilieen found with a similar case where lag seemed to correlate to upgrading the master to 10.0.23, and went away after rolling bac... [15:04:20] Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, Fundraising Sprint Quill Pencil, and 4 others: Are we losing transactions witih repeated ct_id? - https://phabricator.wikimedia.org/T171349#3635730 (mepps) @Ejegg I'm currently thinking I like... [15:29:47] Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, Fundraising Sprint Quill Pencil, and 4 others: Are we losing transactions witih repeated ct_id? - https://phabricator.wikimedia.org/T171349#3635869 (Ejegg) OK, we can try that one and see what... [15:48:33] (PS5) Mepps: ApprovePayment tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379292 [15:51:50] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T176765#3635945 (Ejegg) [16:24:04] (PS6) Ejegg: ApprovePayment tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379292 (owner: Mepps) [16:24:17] (CR) Ejegg: [C: 2] "Nice consolidation!" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379292 (owner: Mepps) [16:24:38] hey ejegg! [16:24:46] hi mepps! [16:24:49] i'm looking at that dedupe failmail--let me know if it's something expected [16:24:54] whew, looks like eileen's been busy [16:25:33] mepps oh, are the machines back up to investigate? [16:26:00] oh it seems like--i'm in the logs [16:26:12] i hadn't thought about the outage--should i not be in there? [16:26:44] no worries, if they plugged in again, we're fine to be on them [16:27:00] (Merged) jenkins-bot: ApprovePayment tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379292 (owner: Mepps) [16:31:31] mepps: I see DB Error: unknown error in those dedupe logs [16:31:38] which usually means a lock timeout [16:32:05] maybe there was another thing doing big ugly queries at the same time? [16:32:07] i see this: 2017-09-26 14:50:10,040 ERROR JOIN civicrm_email t2 USING (email) WHERE t1.contact_id < [16:32:07] 2017-09-26 14:50:10,040 ERROR t2.contact_id AND t1.email IS NOT NULL AND t1.email '' AND [16:32:07] 2017-09-26 14:50:10,040 ERROR t1.contact_id IN [16:32:20] could it be the missing "=" after email? [16:32:27] ugh, those 'error' lines are misleading [16:32:44] nah, USING (email) is perfectly cromulent mysql-ese [16:32:57] ohhh. /that/ missing = [16:34:06] huh, that seems like something that would blow up always [16:34:39] is it just overzealous logging sanitization? [16:34:58] the query would be something like t.email <> '' [16:35:09] so maybe the <> got interpreted as HTML? [16:35:11] ehhh [16:36:47] hmm [16:37:35] relocating before I end up stuck in the apt all day, back soon! [16:37:41] sounds good! [17:14:46] mepps feel like working through some of the latest core patches that eileen put up? [17:15:35] ejegg do you mean reviewing the open ones or looking for the source of dedupe error? i'm currenlty looking at that still [17:16:14] oh, did you find anything likely in the logs ? [17:17:14] i mean i'm still following up on the missing symbol after email [17:17:50] i was just looking at this patch: https://gerrit.wikimedia.org/r/#/c/373176/ [17:18:05] but i can also review the recent open changes [17:18:25] mepps I'm looking at a log which didn't error out, and it's got the same missing symbol [17:18:32] dedupe_civicrm_contacts-20170926-000001.log [17:18:38] okay ejegg, i'll ignore then [17:19:10] (CR) Ejegg: [C: 2] CRM-21006 Ensure that Buttons are correctly displayed [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380670 (owner: Eileen) [17:19:51] ejegg looks like you're doing some CR, are there specific patches you want me to look at? [17:20:21] want to work from the top of the list and I'll work from the bottom? [17:20:40] ie you start from CRM-21022 - Parameterize... [17:23:14] sure ejegg [17:23:24] thanks! [17:24:05] (CR) Mepps: [C: 2] CRM-21022 - Parameterize variables in SQL query [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380682 (owner: Eileen) [17:24:39] (CR) Ejegg: [C: 2] "Looks fine. Explicit :'html' removed in some places, but that's the default for 'escape'" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380671 (owner: Eileen) [17:25:13] (Merged) jenkins-bot: CRM-21006 Ensure that Buttons are correctly displayed [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380670 (owner: Eileen) [17:26:08] (CR) Ejegg: [C: 2] "Better math, removes function-scoped statics. +2!" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380672 (owner: Eileen) [17:26:28] (CR) Mepps: [C: -1] "Typo in function name." (2 comments) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380681 (owner: Eileen) [17:26:43] (CR) Mepps: [C: 2] CRM-20895 - Encode display of search criteria [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380680 (owner: Eileen) [17:26:45] (CR) Ejegg: [C: 2] CRM-21014 Ensure that preimum name is EOU [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380676 (owner: Eileen) [17:28:23] (CR) Mepps: [C: 2] CRM-21123 Ensure that the selectedChild on message templates is one of allowed types [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380679 (owner: Eileen) [17:29:05] (CR) Mepps: [C: 2] "Okay I see the typos are from upstream." [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380681 (owner: Eileen) [17:29:20] looks like upstream needs a fix? [17:29:36] (CR) Mepps: [C: 2] CRM-20907 Ensure that contact_type is valid in deduperules [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380677 (owner: Eileen) [17:29:49] ah, I was just trying to figure that one out [17:30:02] it doesn't change $this->contact_type from string to int? [17:30:11] (CR) Mepps: [V: 2] CRM-21014 Ensure that preimum name is EOU [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380676 (owner: Eileen) [17:30:25] oops we overlapped [17:30:33] heh, no woorries [17:30:37] or worries [17:31:00] woorries are Chewbakka's concerns [17:31:03] (Merged) jenkins-bot: CRM-21006 - Escape all title and alt attributes [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380671 (owner: Eileen) [17:31:05] (Merged) jenkins-bot: CRM-21059 Upgrade Smarty to 2.6.30 [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380672 (owner: Eileen) [17:31:07] (Merged) jenkins-bot: CRM-21014 Ensure that preimum name is EOU [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380676 (owner: Eileen) [17:31:22] (CR) XenoRyet: [C: 1] "My gut reaction is that this is a tolerable level of ugliness to avoid a performance hit. And as ugliness goes, this is pretty readable an" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379945 (https://phabricator.wikimedia.org/T171349) (owner: Ejegg) [17:31:23] ejegg: also woollies. [17:31:33] i don't think ejegg, where are you seeing the change? [17:32:11] hmm i think i see what you mean [17:32:20] ever notice that ewok is suspiciously close to an anagram of wookie? [17:33:50] (Merged) jenkins-bot: CRM-20907 Ensure that contact_type is valid in deduperules [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380677 (owner: Eileen) [17:34:15] ejegg: In fact... [17:34:29] there's a story here, isn't there [17:34:35] ejegg: Originally, RotJ was meant to end on the Wookiee homeworld. [17:34:48] mepps ok, it doesn't actually change the type [17:34:59] cool ejegg [17:35:09] but that $contactTypes['values'][$contactType] seems a bit unnecessary, then! [17:35:34] But, by that time, Wookiees had been established as more sophisticated than they originally had in mind, and they needed a virtually uncontacted society, so... [17:35:59] K4-713: ahh, i thought it was the marketers deciding they needed teddie bears [17:36:14] That was an accident that they ran with. [17:36:34] In related news, I am worried about Porgs. [17:36:44] ??? [17:37:08] Oh jeez [17:37:11] ...might be a tad spoileriffic if you care about that kind og thing. [17:37:20] this season's must-have toy [17:37:33] You could just take the skin off a furby and... [17:37:39] ...that got dark. [17:39:32] (Merged) jenkins-bot: CRM-21123 Ensure that the selectedChild on message templates is one of allowed types [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380679 (owner: Eileen) [17:39:38] (Merged) jenkins-bot: CRM-20895 - Encode display of search criteria [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380680 (owner: Eileen) [17:39:40] (Merged) jenkins-bot: CRM-20906 Validate the extension key is of general standard [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380681 (owner: Eileen) [17:41:07] well, once the last couple settle, let's do a CRM deploy [17:41:19] XenoRyet: feel like doing a CRM deploy? [17:41:26] needs a civicrm submodule bump too [17:42:01] just one thing left for jenkins to merge [17:42:07] I'm actually in a little bit of a different headspace if you don't mind taking this one. [17:42:31] Though if you're busy I can shift over. [17:42:33] k [17:42:45] mepps, do you feel like doing the deploy? [17:43:11] (Merged) jenkins-bot: CRM-21022 - Parameterize variables in SQL query [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/380682 (owner: Eileen) [17:43:13] (Abandoned) Ejegg: Donations QC: don't call ganglia module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/378765 (https://phabricator.wikimedia.org/T176167) (owner: Ejegg) [17:43:27] ejegg, sure it's been a little while [17:43:33] mepps I'll disable that ganglia module [17:46:52] OK, disabled on prod and and staging [17:47:19] ejegg there's no deployment branch of civicrm, is there? [17:47:39] not the submode, but there is of the crm repo [17:47:43] right [17:47:45] cool [17:47:56] ooh, just noticed one thing [17:48:09] jgreen wants the prometheus files to have extension .prom [17:48:17] lemme whip that up right quick [17:48:29] (PS1) Mepps: Latest updates to civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/380797 [17:48:36] cool, ejegg [17:52:16] (PS1) Ejegg: Prometheus: add extension, update default dir [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/380799 (https://phabricator.wikimedia.org/T171524) [17:52:30] ok mepps, ^^^ should do it [17:52:55] cool ejegg, just waiting for jenkins now [17:53:01] (CR) Mepps: [C: 2] Latest updates to civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/380797 (owner: Mepps) [17:54:04] Fundraising-Backlog, Analytics, MediaWiki-extensions-CentralNotice: Make banner impression counts available somewhere public - https://phabricator.wikimedia.org/T115042#1713344 (ksmith) @Nuria : Have you read the [[ https://docs.google.com/document/d/1R3G04PCe3xZAR2azWPzdWVO4vQQKraZUrlOMQidfh0w/edit... [17:57:23] (Merged) jenkins-bot: Latest updates to civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/380797 (owner: Mepps) [18:01:48] (CR) Mepps: [C: 2] Prometheus: add extension, update default dir [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/380799 (https://phabricator.wikimedia.org/T171524) (owner: Ejegg) [18:02:45] thanks mepps! [18:03:50] Fundraising Sprint RadioActivewear, Fundraising-Backlog, MW-1.31-release-notes (WMF-deploy-2017-09-26 (1.31.0-wmf.1)), Patch-For-Review: Fix LandingCheck language fallback - https://phabricator.wikimedia.org/T174693#3636280 (Ejegg) Looks like this'll go out to foundationwiki tomorrow (September 27) [18:05:32] (Merged) jenkins-bot: Prometheus: add extension, update default dir [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/380799 (https://phabricator.wikimedia.org/T171524) (owner: Ejegg) [18:10:16] (PS1) Mepps: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/380803 [18:10:37] ejegg can you look at this: i just want to make sure i chose the right files to delete/keep [18:10:47] sure thing mepps [18:12:02] yep, the files look fine! We usually list the merged commits in the message [18:15:53] (PS2) Mepps: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/380803 [18:16:06] (CR) Mepps: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/380803 (owner: Mepps) [18:16:14] cool [18:17:11] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/380803 (owner: Mepps) [18:25:27] !log Updated crm from 89b800719e73b8d76941c48dcf1b22111d7e31a0 to 3f48cb03b4a216a34a61f9744a297de4fb99d8a6 [18:25:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:43:02] mepps ok, I enabled metrics_reporting [18:43:31] Great ejegg [18:43:37] going to get some lunch in me [18:51:59] PROBLEM - Host payments1004 is DOWN: PING CRITICAL - Packet loss = 100% [18:52:08] ugh [19:15:37] fr-tech, for a SWAT deploy of CentralNotice bugfixies, how tomorrow at 11 am Pacific Time sound? thx!!! [19:16:05] sounds good to me AndyRussG [19:20:19] RECOVERY - Host payments1004 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [20:03:07] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Make CRM version stamp thing robust on dev boxes - https://phabricator.wikimedia.org/T88376#3636828 (DStrine) Open>declined [20:05:01] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T176765#3636850 (DStrine) [20:06:44] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Look at creating job to clear tmp dir in Civi - https://phabricator.wikimedia.org/T176691#3636863 (DStrine) [20:08:21] Fundraising Sprint RadioActivewear, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Add placeholding & limit to omnimail recipients, like groupmember - https://phabricator.wikimedia.org/T176255#3619036 (Eileenmcnaughton) [20:08:46] Fundraising Sprint RadioActivewear, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Manage groups page seems to be refreshing groups when it shouldn't - https://phabricator.wikimedia.org/T176258#3636872 (Eileenmcnaughton) Open>Resolved [20:08:54] Fundraising Sprint RadioActivewear, Fundraising-Backlog, FR-PayPal-ExpressCheckout: Investigate PayPal-EC related FailMail - https://phabricator.wikimedia.org/T176339#3636873 (XenoRyet) Open>Resolved [20:09:12] Fundraising Sprint RadioActivewear, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email, Patch-For-Review: Omnimail recipient load (silently) broken - https://phabricator.wikimedia.org/T175394#3636874 (Eileenmcnaughton) Open>Resolved [20:09:32] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising-Backlog, Patch-For-Review: Establish methodology for creating load to replicate replag - https://phabricator.wikimedia.org/T175665#3636891 (Eileenmcnaughton) Open>Resolved [20:10:58] Fundraising Sprint RadioActivewear, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, MW-1.31-release-notes (WMF-deploy-2017-09-26 (1.31.0-wmf.1)), Patch-For-Review: Fix warnings in Amount staging helper - https://phabricator.wikimedia.org/T160962#3636894 (Ejegg) Open>Resolv... [20:11:09] Fundraising Sprint RadioActivewear, Fundraising-Backlog: Deploy French thank you email - https://phabricator.wikimedia.org/T176585#3636896 (Ejegg) Open>Resolved [20:14:48] Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, Fundraising Sprint Quill Pencil, and 4 others: Populate country column when creating c_t rows during offline import - https://phabricator.wikimedia.org/T171658#3636921 (Ejegg) Still waiting for... [20:16:49] Fundraising Sprint Murphy's Lawyer, Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, and 8 others: Implement Ingenico Connect API calls to get payment status - https://phabricator.wikimedia.org/T163948#3215844 (mepps) Open>Resolved [20:45:59] Fundraising Sprint S 2017, Fundraising-Backlog, MediaWiki-extensions-DonationInterface: email address validation not clearing error on blur - https://phabricator.wikimedia.org/T176501#3637133 (DStrine) [20:46:01] Fundraising Sprint RadioActivewear, Fundraising Sprint S 2017, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, and 2 others: Are we overwriting good country data with bad? - https://phabricator.wikimedia.org/T176450#3637134 (DStrine) [20:46:04] Fundraising Sprint S 2017, Fundraising-Backlog, MediaWiki-extensions-DonationInterface: New payments-wiki errors: We don't have an order status after doing a GET_ORDERSTATUS - https://phabricator.wikimedia.org/T176355#3637135 (DStrine) [20:46:07] Fundraising Sprint S 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: UI batch merge dropping limit at the end - https://phabricator.wikimedia.org/T176256#3637137 (DStrine) [20:46:09] Fundraising Sprint S 2017, Fundraising-Backlog, FR-Paypal: PayPal IPN processing jobs: undefined index subscr_id - https://phabricator.wikimedia.org/T176152#3637139 (DStrine) [20:46:13] Fundraising Sprint S 2017, Fundraising-Backlog, FR-PayPal-ExpressCheckout: Handle additional reason codes for Paypay-EC refunds - https://phabricator.wikimedia.org/T176225#3637140 (DStrine) [20:46:15] Fundraising Sprint S 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Fr-CiviCRM-dedupe-FY2017/18: Implement merge rule to allow resolution on conflicts on punctuation to prefer more punctuation - https://phabricator.wikimedia.org/T175748#3637143 (DStrine) [20:46:17] Fundraising Sprint S 2017, Fundraising-Backlog: Clean up Damaged queue - https://phabricator.wikimedia.org/T175862#3637142 (DStrine) [20:46:19] Fundraising Sprint S 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Fr-CiviCRM-dedupe-FY2017/18: Update wmf_civicrm import normalisation to replace htmlampersand with & - https://phabricator.wikimedia.org/T175744#3637146 (DStrine) [20:46:21] Fundraising Sprint S 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Fr-CiviCRM-dedupe-FY2017/18: Implement merge rule on whitespace to prefer ones with more whitespace - https://phabricator.wikimedia.org/T175746#3637145 (DStrine) [20:46:23] Fundraising Sprint S 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Fr-CiviCRM-dedupe-FY2017/18: Implement merge rule to allow overwriting any name fields that are wholly numeric in the case of a conflict - https://phabricator.wikimedia.org/T175747#3637144 (DStrine) [20:46:25] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising Sprint S 2017, Fundraising-Backlog, and 3 others: Investigate email: BH data storage/transfer issue for iPad donations - https://phabricator.wikimedia.org/T174719#3637149 (DStrine) [20:46:28] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising Sprint S 2017, Fundraising-Backlog, and 4 others: CN Campaign Suppression prior to scheduled start time - https://phabricator.wikimedia.org/T175358#3637148 (DStrine) [20:46:31] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising Sprint S 2017, Fundraising-Backlog, and 3 others: Find source of unlimited dedupe queries, prevent them - https://phabricator.wikimedia.org/T175382#3637147 (DStrine) [20:46:33] Fundraising Sprint RadioActivewear, Fundraising Sprint S 2017, Fundraising-Backlog, MW-1.31-release-notes (WMF-deploy-2017-09-26 (1.31.0-wmf.1)), Patch-For-Review: Fix LandingCheck language fallback - https://phabricator.wikimedia.org/T174693#3637150 (DStrine) [20:46:35] Fundraising Sprint RadioActivewear, Fundraising Sprint S 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: CiviCRM dedupe jobs should gracefully time out - https://phabricator.wikimedia.org/T172303#3637153 (DStrine) [20:46:37] Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, Fundraising Sprint Quill Pencil, and 5 others: Create orphan rectifier for PayPal Express Checkout - https://phabricator.wikimedia.org/T172202#3637154 (DStrine) [20:46:40] Fundraising Sprint Prank Seatbelt, Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising Sprint S 2017, and 3 others: Drop IDs on cache tables - https://phabricator.wikimedia.org/T174404#3637151 (DStrine) [20:46:42] Fundraising Sprint RadioActivewear, Fundraising Sprint S 2017, Fundraising-Backlog, Patch-For-Review: process-control should make slow-starting jobs easier - https://phabricator.wikimedia.org/T171873#3637155 (DStrine) [20:46:44] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising Sprint S 2017, Fundraising-Backlog, and 2 others: Create lists of CiviCRM groups to allow MG & DS to review with a view to tidy up - https://phabricator.wikimedia.org/T174407#3637152 (DStrine) [20:46:46] Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, Fundraising Sprint Quill Pencil, and 5 others: Populate country column when creating c_t rows during offline import - https://phabricator.wikimedia.org/T171658#3637157 (DStrine) [20:46:50] Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, Fundraising Sprint Quill Pencil, and 5 others: Are we losing transactions witih repeated ct_id? - https://phabricator.wikimedia.org/T171349#3637158 (DStrine) [20:46:52] Fundraising Sprint RadioActivewear, Fundraising Sprint S 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Update CiviCRM ganglia bits to point to Prometheus - https://phabricator.wikimedia.org/T171524#3637156 (DStrine) [20:46:54] Fundraising Sprint RadioActivewear, Fundraising Sprint S 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Reset on_hold in wmf_civicrm_message_email_update - https://phabricator.wikimedia.org/T170350#3637160 (DStrine) [20:46:56] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising Sprint S 2017, Fundraising-Backlog, and 4 others: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591#3637159 (DStrine) [20:48:06] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Search for email only works if primary email address - https://phabricator.wikimedia.org/T152048#3637169 (Ejegg) This is fixed and deployed, right @Eileenmcnaughton and @LeanneS ? [20:48:56] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising Sprint S 2017, Fundraising-Backlog, and 3 others: Find source of unlimited dedupe queries, prevent them - https://phabricator.wikimedia.org/T175382#3592103 (Eileenmcnaughton) a:Ejegg>Eileenmcnaughton Stealing... [20:49:03] Fundraising Sprint S 2017, Fundraising-Backlog, FR-Email: Add field to Silverpop export: first donation date - https://phabricator.wikimedia.org/T150467#2786781 (Ejegg) [20:49:40] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising Sprint S 2017, Fundraising-Backlog, and 3 others: Find source of unlimited dedupe queries, prevent them - https://phabricator.wikimedia.org/T175382#3637190 (Ejegg) Go for it, and best of luck! [21:01:39] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Search for email only works if primary email address - https://phabricator.wikimedia.org/T152048#3637205 (Eileenmcnaughton) That feature is deployed but it needs to be turned on and ... I don't trust it. We need to test that it doesn't cause any weird o... [21:15:21] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, FR-PayPal-ExpressCheckout: Handle additional reason codes for Paypay-EC refunds - https://phabricator.wikimedia.org/T176225#3617986 (XenoRyet) a:XenoRyet [21:16:18] mepps: have fun at the Civi sprint! [22:04:51] /nick XenoRyet [22:17:07] fr-tech I'd love to get T171349 out before the email barrage. [22:17:08] T171349: Are we losing transactions witih repeated ct_id? - https://phabricator.wikimedia.org/T171349 [22:17:33] mepps came down on the side of the simpler solution that might slow down donation queue consumption [22:17:51] what do you think eileen and XenoRyet ? [22:18:39] ejegg: on the 171349? [22:18:50] yeah [22:19:15] it's either search for the invoice ID before each contribution insert [22:19:25] I liked the one that doesn't have us taking a performance hit, but I'm not strongly convicted about it. [22:19:39] or make a special case for requeueing messages that alters a couple fields [22:19:47] (PS1) Pcoombe: Don't load CentralNotice on history or diff pages [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/380891 (https://phabricator.wikimedia.org/T176656) [22:19:55] so re performance [22:20:07] currently our bottleneck is in write [22:20:21] the idea of doing an extra look-up doesn't seem so concerning [22:20:37] reducing lookups did a lot for speeding up thank you emails [22:21:19] but that was a specific query wasn't it? [22:21:33] & it moved the bottleneck to replication.... [22:21:47] replaced a few api calls with one sql query [22:22:08] but yeah, it's probably fine to do one extra lookup, we should just measure! [22:22:25] right - I think the getcount is probably better… [22:22:35] because it will impose a limit on 'return' [22:23:04] when does a contribution have/not have an invoice_id? [22:23:42] eileen almost everything online does have one [22:23:49] it's the [22:23:54] 'merchant reference' [22:24:04] OK - so currently it will fail if there is an invoice_id duplicate [22:24:15] usually the contribution tracking ID plus a suffix number [22:24:27] eileen: yep, and worse, pretend that it succeeded [22:24:33] and totally discard the message [22:24:40] :-( [22:24:58] cause we start a Civi txn when we start processing the message [22:25:12] the insert failure marks it for rollback, but doesn't roll it back immediately [22:25:24] so the current logic, which catches that failure, alters, and reinserts [22:25:30] is all useless [22:25:38] since that stuff is part of the same txn [22:25:42] ok - note it IS possible to call create with 'is_transactional' => 0 [22:25:50] oho! [22:25:59] but, I feel like us rethrowing & requeueing is probably the 'correct' approach [22:26:29] yeah, that's what we do in the general case [22:26:47] why does it seem ugly to you in this case? [22:27:01] because we should be 'making up' a new invoice_id seamlessly? [22:27:29] just because the requeueing logic gets less generic [22:27:35] ah right [22:27:50] all the other exceptions are handled with properties like ->isRequeue() [22:28:14] and this one has to be recognized individually and get its own code block [22:28:20] hmm [22:28:25] because it needs changing [22:28:52] right, and in a specific way [22:29:08] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Patch-For-Review: Prevent showing CentralNotice on Special pages and History/Diff pages to partially resolve banner-bump issues - https://phabricator.wikimedia.org/T176656#3637537 (Pcoombe) a:Pcoombe [22:29:13] so I'm not 100% sure but it might be that the second create could be done with is_transactional = 0 [22:29:17] there's no way to do the alteration at the point we throw the DUPLICATE_INVOICE [22:29:18] & it WOULD stick [22:30:19] eileen what about the contact insert that we did before the failing contribution insert? [22:30:26] that's currently getting rolled back too [22:31:25] yeah that's what I wasn't quite sure about [22:31:39] I think we need to keep the transactionality [22:32:05] I think we also need to check before committing the related db transactions whether the native Civi txn has been marked for rollback [22:32:19] because we're NOT rolling back the contribution_tracking table update [22:32:35] and it ends up pointing at a nonexistent civi contributrion [22:34:40] I thought we wrapped stuff in a transaction - in benevity at least [22:35:45] can't find it tho [22:36:00] eileen: we do in the queue consumer [22:36:04] one sec, lemme find it [22:37:32] https://github.com/wikimedia/wikimedia-fundraising-crm/blob/master/sites/all/modules/wmf_common/WmfDatabase.php [22:38:16] My instinct is that the reason it looks ugly having special handling in WMFQueueConsumer is because that module doesn't 'own' the modifyDuplicateInvoice handling & that it would stop looking ugly if we used a hook/listener approach. [22:38:41] (which doesn't mean we need to do that now - but if it were a good approach for the future we could signpost it) [22:39:58] That sounds about right [22:40:04] lemme add a TODO [22:40:45] Ah I see - we use transactionCall from import checksFile [22:41:12] & from processMessageWithErrorHandlin [22:42:19] why am I not seeing more files call processMessageWithErrorHandling() [22:45:03] eileen: it's just part of BaseQueueConsumer [22:45:30] so all of the QueueConsumers that inherit from TransactionWmfQueueConsumer get that behavior [22:46:34] ok [22:47:24] gotta pack up at the co-working space, but I'll be back on from the apartment [22:48:05] ok [22:49:00] (PS3) Ejegg: Duplicate invoice special handling in message requeue [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379945 (https://phabricator.wikimedia.org/T171349) [22:49:49] fundraising-tech-ops: rsyslog hung after firewall switch - https://phabricator.wikimedia.org/T176821#3637573 (cwdent) [22:56:31] (CR) Eileen: [C: 2] "I think this is OK with the comment changes. I don't think it would be ugly with a listener framework & this signposts the way to the list" (2 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379945 (https://phabricator.wikimedia.org/T171349) (owner: Ejegg) [22:58:10] (CR) Eileen: "I gave the other one a + 2 because I think catching & requeing makes sense. The missing piece is that the handling for duplicates should b" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379941 (https://phabricator.wikimedia.org/T171349) (owner: Ejegg) [23:31:36] thanks eileen ! [23:31:55] for some reason it was failing until I added the parent commit [23:32:14] deleting an extra contribution in the CiviFixtures [23:39:21] ejegg: looks like the last fix on dedupe caused limit to be dropped :-( [23:39:30] oh no! [23:39:41] so it was a new bug, not a new prevalence of the old one [23:41:30] which commit was that? [23:42:01] one of mine... [23:42:21] I tidied up the calls to retrieve duplicates [23:42:27] & it got lost in there [23:42:43] won't affect other sites even though it is in core because limit is kinda hidden [23:42:45] oops, i think i reviewed that one [23:42:57] you are not the only person who did! [23:43:04] it got reviewed upstream too [23:43:31] you should enter the underhanded code contest! [23:44:50] :-) [23:45:05] I think that might be a backhanded compliment :-) [23:45:23] hehe [23:46:14] if you wanted to hide functionality, i bet you could pull it off [23:47:40] well some codebases are easier to hide things in than others.... [23:48:20] the thing with limit is that it's a bit half-baked because of the confusion between how many matches to dedupe vs how many contacts to find matches for [23:48:39] from a UI only the former makes sense but for performance the latter matters [23:48:43] right... there oughtta be two settings [23:49:34] yeah so there is a task to make that logical from a UI point of view - & to make it clear you can't really combine them exactly [23:50:07] ie. if you limit to search 50 contacts & 50 matches you might get 30 matches [23:50:16] & think that was it [23:50:30] not realising that there are 30 matches in the next 50 contacts too [23:50:34] also -…. offset [23:50:52] ah yeah, totally invisible now [23:51:11] I went through it with the guy from Oxford Uni & we agreed that to really expose all that it needed to be done with a bit more thought [23:51:23] so we snuck in our limit as an …. interim… [23:51:44] sorry John is Cambridge not Oxford [23:51:56] they have alumni or someting [23:52:18] ahhh, finally spotted the limit breakage [23:52:23] :-) [23:52:54] well this time I'm gonna kill this while I'm at it.... [23:52:55] https://github.com/eileenmcnaughton/civicrm-core/blob/4c4602082a427ed12efb2250febc089a20d2962b/CRM/Dedupe/Merger.php#L666 [23:53:10] $where .= " LIMIT {$batchLimit}" [23:53:12] !! [23:58:09] heh, nice [23:58:29] I see you already had a "// WTF" comment