[01:20:13] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3079 [01:25:13] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2872 [01:30:13] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 1035014 Threads: 1 Questions: 19902231 Slow queries: 5789 Opens: 11099 Flush tables: 1 Open tables: 600 Queries per second avg: 19.228 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1 [06:17:29] (PS4) Eileen: Add ability to find duplicates for selected contacts. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/373155 (https://phabricator.wikimedia.org/T151270) [09:45:43] (Restored) Hashar: Jenkins job validation (DO NOT SUBMIT) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/151840 (owner: Hashar) [09:45:46] (PS5) Hashar: Jenkins job validation (DO NOT SUBMIT) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/151840 [09:45:53] (CR) Hashar: "check experimental" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/151840 (owner: Hashar) [13:35:35] (PS2) Mepps: Add test for Templating [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/373974 (owner: Ejegg) [13:35:49] (CR) Mepps: [C: 2] Add test for Templating [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/373974 (owner: Ejegg) [13:41:27] (Merged) jenkins-bot: Add test for Templating [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/373974 (owner: Ejegg) [13:45:34] (PS7) Mepps: Cache result of template loads, including language fallback lookups [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/364790 (https://phabricator.wikimedia.org/T170435) (owner: Ejegg) [13:45:39] (CR) Mepps: [C: 2] Cache result of template loads, including language fallback lookups [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/364790 (https://phabricator.wikimedia.org/T170435) (owner: Ejegg) [13:46:48] (PS5) Mepps: Use SmashPig version of Currency classes [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/363744 (https://phabricator.wikimedia.org/T163868) (owner: Ejegg) [13:46:52] (CR) Mepps: [C: 2] Use SmashPig version of Currency classes [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/363744 (https://phabricator.wikimedia.org/T163868) (owner: Ejegg) [13:47:57] (CR) Mepps: "ejegg is this something you want merged or just to be here for reference?" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/364531 (owner: Ejegg) [13:51:16] (Merged) jenkins-bot: Cache result of template loads, including language fallback lookups [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/364790 (https://phabricator.wikimedia.org/T170435) (owner: Ejegg) [13:51:48] (Merged) jenkins-bot: Use SmashPig version of Currency classes [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/363744 (https://phabricator.wikimedia.org/T163868) (owner: Ejegg) [14:07:58] (PS4) Mepps: Add Spanish translations for missing fiscal numbers [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/363386 (owner: Ejegg) [14:08:03] (CR) Mepps: [C: 2] Add Spanish translations for missing fiscal numbers [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/363386 (owner: Ejegg) [14:09:53] (Merged) jenkins-bot: Add Spanish translations for missing fiscal numbers [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/363386 (owner: Ejegg) [14:55:20] wow, CR bonanza! [14:55:28] thanks so much, mepps! [15:03:44] ok, going to test out those recurring donations [16:00:08] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Ingenico, Recurring-Donations: Don't retry recurring donations after expired card or fraud flag errors - https://phabricator.wikimedia.org/T174450#3562413 (Ejegg) [16:00:18] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Ingenico, Recurring-Donations: Don't retry recurring donations after expired card or fraud flag errors - https://phabricator.wikimedia.org/T174450#3562427 (Ejegg) p:Triage>High [16:04:10] !log restarted ingenico recurring charge job [16:04:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:05:19] (PS1) Ejegg: WIP break up giant audit function [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374559 [16:14:23] got a little bit of replag last night, presumably from turning dedupe back on [16:14:27] and it must have had a lot of work to do [16:14:44] but it was quite a while later [16:14:49] i will compile a timeline [16:29:10] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/374562 [16:29:13] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/374562 (owner: Ejegg) [16:30:01] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/374562 (owner: Ejegg) [16:32:37] !log turned off donation queue consumer and thank you mailer [16:32:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:33:32] !log updated civicrm from 7dfea0dd7677d1389975d9b3cf369981b7bdb38b to f00611d9aa3ae4a1ffbff70dcca90c7ae07b6c68 [16:33:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:48:03] (PS1) Ejegg: Fix third param of watchdog() calls [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374568 [17:19:53] (PS1) Ejegg: Actually cache fallback language templates [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374583 (https://phabricator.wikimedia.org/T170435) [17:23:26] !log restarted donation queue consumer and thank you mailer [17:23:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:28:42] (PS1) Ejegg: Drop autoincrement ID on group_contact_cache [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374588 (https://phabricator.wikimedia.org/T174404) [17:30:23] (PS2) Ejegg: Drop autoincrement ID on group_contact_cache [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374588 (https://phabricator.wikimedia.org/T174404) [17:33:59] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Drop IDs on cache tables - https://phabricator.wikimedia.org/T174404#3562900 (Ejegg) @jcrespo has suggested a new schema for group_contact_cache in P5937 - I just had one question about the UNIQUE_KEY constraint [17:35:28] Fundraising Sprint Prank Seatbelt, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Drop IDs on cache tables - https://phabricator.wikimedia.org/T174404#3562913 (Ejegg) p:Triage>Normal a:Ejegg [17:42:22] Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, Fundraising-Backlog, and 2 others: Populate country column when creating c_t rows during offline import - https://phabricator.wikimedia.org/T171658#3472423 (Ejegg) Huh, recent Benevity imports... [17:54:59] (CR) Mepps: [C: 2] "Looks great locally!" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/372157 (https://phabricator.wikimedia.org/T171346) (owner: Ejegg) [17:57:34] Thanks mepps! [17:58:02] Yea, I was just wrapping up looking at that one myself. Looks good here too. [17:58:18] Didn't really come up with any new ideas for that scroll problem you were talking about yesterday. [17:58:19] Like I said to XenoRyet yesterday, there's still one annoyance - if you're filling out fields in the iframe and you tab past the pay button, all of the fields can scroll out of view [17:58:47] We can work on that in another patch though. [18:00:23] ejegg: I was gonna start prepping that SmashPig deploy. You said there was something in there you wanted to keep an eye on? [18:01:16] XenoRyet: yep, it's the one where we switch PayPal message verification to using the CurlWrapper class instead of calling cURL directly [18:01:44] Cool [18:02:28] I'll get things prepped and let you know when I'm ready to push the button. [18:03:03] (Merged) jenkins-bot: Position hosted checkout iframe [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/372157 (https://phabricator.wikimedia.org/T171346) (owner: Ejegg) [18:05:23] ejegg saw the tabbing issue you mentioned, want me to look at it? [18:09:41] mepps if you feel like it... it's kind of a picky little browser thing [18:10:13] Are you waiting on CR for the orphan module? [18:11:12] ejegg actually not sure what next steps are to resolve the globals issue--i know it needs to be set in settings.php [18:11:20] in localsettings for prod [18:11:33] so i guess we need to edit the CI server settings if those are different? [18:11:34] oh right, for production it'll be fine [18:11:48] but yeah, we need to find where we're setting all that stuff in CI [18:12:19] (PS1) Ejegg: WIP fix c_t.country population, add test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374597 (https://phabricator.wikimedia.org/T171658) [18:12:28] getting lunch, back in a bit! [18:17:36] (CR) jerkins-bot: [V: -1] WIP fix c_t.country population, add test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374597 (https://phabricator.wikimedia.org/T171658) (owner: Ejegg) [18:27:54] (PS1) XenoRyet: Merge branch 'master' into HEAD [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/374603 [18:31:34] (CR) XenoRyet: [C: 2] Merge branch 'master' into HEAD [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/374603 (owner: XenoRyet) [18:31:42] (Merged) jenkins-bot: Merge branch 'master' into HEAD [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/374603 (owner: XenoRyet) [18:44:33] ejegg when you're back arou,d let's talk about what i should be prioritizing before leaving on vacation [18:52:36] ejegg|food: FYI I'm firing off that deploy now. [19:01:11] XenoRyet: k, hopping on the logging box [19:01:27] looking good so far [19:01:32] speaking of logging [19:01:49] !log updated Smashpig from 98c55161deffed8364043a27089e7d47cddc3059 to 8f4388d407f313c1083cf96ed106efcc5d5a29f7 [19:01:52] almost forgot [19:02:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:02:25] mepps: one thought, can we set the globals in the test setup for CI? [19:03:04] There's nothing that should need to be in the settings file, is there? [19:03:40] ejegg hmm [19:05:35] anyway, the top two things on my mind are getting the orphan stuff done, and maybe pushing ahead a bit more with ingenico connect [19:05:43] but I'm probably forgetting something on the Civi front [19:05:57] cool ejegg, i'm looking at this scrolling thing [19:06:17] i think i have a fix but i'm not sure it's reading the js file correctly [19:06:44] cool! lemme take a look at the latest orphan module thing and see if I have any ci suggestions [19:10:13] mepps, want to update the donation-interface package over at packagist ? [19:10:59] Then we can update composer.lock and use the real function [19:11:05] yeah! dev-deployment? [19:11:10] dev-master [19:15:56] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/374608 [19:16:02] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/374608 (owner: Ejegg) [19:16:55] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/374608 (owner: Ejegg) [19:18:42] (PS1) Ejegg: Update DonationInterface [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/374611 [19:18:46] (CR) Ejegg: [C: 2] Update DonationInterface [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/374611 (owner: Ejegg) [19:25:13] (Merged) jenkins-bot: Update DonationInterface [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/374611 (owner: Ejegg) [19:37:01] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Review reducing smartGroupCache clearing per https://issues.civicrm.org/jira/browse/CRM-21109 - https://phabricator.wikimedia.org/T174408#3563618 (DStrine) p:Triage>Normal [19:37:55] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Create lists of CiviCRM groups to allow MG & DS to review with a view to tidy up - https://phabricator.wikimedia.org/T174407#3560690 (DStrine) p:Triage>Normal [19:42:46] Fundraising Sprint Outie Inverter, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Wysiwig not working in CiviCRM - https://phabricator.wikimedia.org/T172346#3563642 (DStrine) Open>Resolved [19:43:28] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Create lists of CiviCRM groups to allow MG & DS to review with a view to tidy up - https://phabricator.wikimedia.org/T174407#3563646 (DStrine) [19:45:04] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Review reducing smartGroupCache clearing per https://issues.civicrm.org/jira/browse/CRM-21109 - https://phabricator.wikimedia.org/T174408#3563648 (DStrine) [19:48:09] Fundraising Sprint Outie Inverter, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review, Unplanned-Sprint-Work: Civi imports currently aren't working. - https://phabricator.wikimedia.org/T172918#3513186 (DStrine) [19:49:37] Fundraising Sprint Murphy's Lawyer, Fundraising Sprint Navel Warfare, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: "Thank-you Sent" field auto populating date - https://phabricator.wikimedia.org/T170440#3563659 (DStrine) [19:50:58] Fundraising Sprint Kickstopper, Fundraising Sprint Loose Lego Carpeting, Fundraising Sprint Murphy's Lawyer, Fundraising Sprint Navel Warfare, and 4 others: Reflect all unsubscribes via Silverpop in CiviCRM - https://phabricator.wikimedia.org/T161760#3563665 (DStrine) [19:53:16] Fundraising Sprint Murphy's Lawyer, Fundraising-Backlog, WMF-NDA: Amazon (aaaaaargh!) audit report downloads failing, something about an IP address - https://phabricator.wikimedia.org/T169805#3563675 (DStrine) [19:53:50] Fundraising Sprint Murphy's Lawyer, Fundraising-Backlog: SQL challenge: Mailing and donation data - https://phabricator.wikimedia.org/T169591#3563679 (DStrine) [19:54:35] Fundraising Sprint Kickstopper, Fundraising Sprint Loose Lego Carpeting, Fundraising Sprint Murphy's Lawyer, Fundraising-Backlog, and 4 others: Add index 'name' to civicrm_campaign table - https://phabricator.wikimedia.org/T168499#3563680 (Eileenmcnaughton) [19:56:19] (PS2) Ejegg: Fix c_t.country population, add test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374597 (https://phabricator.wikimedia.org/T171658) [19:59:36] Fundraising Sprint Loose Lego Carpeting, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Restore all of the Saved Field Mappings list to Civi - https://phabricator.wikimedia.org/T169886#3563704 (DStrine) [20:06:44] Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, Fundraising-Backlog, and 5 others: Merge CRM and DonationInterface queue wrappers - https://phabricator.wikimedia.org/T95647#3563727 (Ejegg) Open>Resolved a:Ejegg [20:07:44] Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, Fundraising-Backlog, and 5 others: Merge CRM and DonationInterface queue wrappers - https://phabricator.wikimedia.org/T95647#1196764 (Ejegg) [20:09:45] Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, Fundraising-Backlog, and 3 others: Are PayPal refunds for recurring donations incorrectly being tagged as EC or vice versa? - https://phabricator.wikimedia.org/T171351#3563768 (Ejegg) @XenoRyet... [20:42:56] OK, deploying payments-wiki [20:49:23] the short replag last night was like 2 hours after dedupe started [20:49:38] does dedupe do something with a long tail, like filling up a queue? [20:50:44] not as far as I know - I think it makes the changes as it goes along [20:52:03] that's what i thought [20:52:21] i guess it could be a case of replication being slower than the master [20:52:34] Fundraising Sprint Prank Seatbelt, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-WMF-Audit, Patch-For-Review: WX audit files: deal with sparse refund records - https://phabricator.wikimedia.org/T173875#3563961 (Ejegg) Open>Resolved Seems to be working! [20:52:36] and it queues up all those cache table writes [20:53:14] last wednesday turning off dedupe did not seem to affect the lag but it could be because they already had huge backlogs by that point [20:53:32] Fundraising Sprint Prank Seatbelt, Fundraising-Backlog, Patch-For-Review, Unplanned-Sprint-Work: Deploy pt-br and ja thank you letter - https://phabricator.wikimedia.org/T173809#3563977 (Ejegg) Open>Resolved @jrobell, please re-open this ticket if there are any problems with the email. [20:53:48] i think i would like to turn off dedupe well before the banners go up tomorrow [20:53:55] and see if we still see the lag [20:55:07] seems like the best way to find out if it is a red herring [20:56:03] !log updated payments-wiki from 58d33b837167f14dc9388c8d3acc0281b012932f to bf97cd236bdec421082db8a8a782965de9c98358 [20:56:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:57:13] well, we've got one patch in review to get rid of a useless autoinc on the cache table, and eileen's idea to cut way down on the churn in that table just got prioritized for next sprint [20:58:35] nice [20:58:37] what is the idea? [21:00:13] there's something recalculating that table on i think every contact changed during merge [21:00:42] and she's got a plan to expose an api option to not do that [21:01:05] so only do it once per batch instead of every time? [21:01:16] i think so [21:01:25] sounds reasonable [21:01:46] i haven't burrowed in that part of the code, just going by her Jira ticket [21:01:59] !log updated DjangoBannerStats from 5963e7c116f083ada28476c1c41ff1b46232d772 to ce95eab3514a2cd989f090598e01686efba72386 [21:02:09] ^^^ hopefully won't blow up! [21:02:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:02:32] pretty sure that should all still work on django 1.7 [21:04:45] Fundraising Sprint Deferential Equations, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: disable/remove accounts for Brent Cohn from CPS data - https://phabricator.wikimedia.org/T158051#3564036 (DStrine) [21:07:13] Fundraising Sprint Ivory Tower Defense Games, Fundraising Sprint Judgement Suspenders, Fundraising Sprint Kickstopper, Fundraising Sprint Loose Lego Carpeting, and 10 others: Move already-encapsulated useful classes into SmashPig - https://phabricator.wikimedia.org/T163868#3212798 (Ejegg) fr-tech... [21:12:07] Fundraising Sprint Prank Seatbelt, Fundraising-Backlog, Patch-For-Review: Replace deprecated transaction.commit_manually in DjangoBannerLoader - https://phabricator.wikimedia.org/T162504#3564062 (Ejegg) Open>Resolved Working fine, with one fewer warning in the logs! [21:25:14] (CR) Ejegg: "recheck" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/370520 (https://phabricator.wikimedia.org/T172723) (owner: Ejegg) [21:25:42] (CR) jerkins-bot: [V: -1] PayPal audit: check for existing recurring donations [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/370520 (https://phabricator.wikimedia.org/T172723) (owner: Ejegg) [21:25:54] ok then [21:25:58] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising-Backlog, Patch-For-Review: Upgrade to drupal 7.5 - https://phabricator.wikimedia.org/T143268#3564134 (Eileenmcnaughton) [21:26:09] must not be mocking everything [21:26:35] Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, Fundraising-Backlog, and 2 others: We still have contacts whose merge is blocked on mis-rounded geocodes - https://phabricator.wikimedia.org/T158271#3564138 (Eileenmcnaughton) [21:30:24] Fundraising Sprint Far Beer, Fundraising Sprint Gondwanaland Reunification Engine, Fundraising Sprint Homebrew Hadron Collider, Fundraising-Backlog, Patch-For-Review: Benevity import not updating emails for existing contacts - https://phabricator.wikimedia.org/T161666#3564157 (Eileenmcnaughton) [21:30:44] Fundraising Sprint Far Beer, Fundraising Sprint Gondwanaland Reunification Engine, Fundraising Sprint Homebrew Hadron Collider, Fundraising-Backlog, and 2 others: Spike: investigate getting silverpop data into civi - https://phabricator.wikimedia.org/T159767#3564158 (Eileenmcnaughton) [21:35:10] PROBLEM - check_puppetrun on frdb2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[mariadb-client] [21:36:13] grumble [21:36:40] grumble indeed [21:36:47] oops sorry :) [21:36:50] i was just grumbling [21:37:12] hah [21:40:18] RECOVERY - check_puppetrun on frdb2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [21:40:58] so I scrolled up to find the cause of grumbling & found mocking.... [21:41:21] or not mocking [21:42:08] oh just puppet [21:42:16] or more precisely apt [21:43:41] will sometimes remove a package before failing to install the new version [21:43:54] specifically mariadb [21:43:58] glad i didn't start with 1001 [21:46:39] (PS2) Ejegg: PayPal audit: check for existing recurring donations [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/370520 (https://phabricator.wikimedia.org/T172723) [21:47:04] (CR) jerkins-bot: [V: -1] PayPal audit: check for existing recurring donations [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/370520 (https://phabricator.wikimedia.org/T172723) (owner: Ejegg) [21:47:09] eileen2: did you see the backscroll about the 2 hour delayed replag? [21:47:57] wonder what you think about me turning it off before the banners tomorrow [21:48:00] Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, Fundraising Sprint Gondwanaland Reunification Engine, Fundraising-Backlog, and 3 others: Enable PayPal EC everywhere it's available - https://phabricator.wikimedia.org/T159755#3564288 (awight) [21:48:17] Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, Fundraising Sprint Gondwanaland Reunification Engine, and 3 others: PHP-Queue: remove IndexedFifo and KeyValue interfaces - https://phabricator.wikimedia.org/T159175#3564290 (awight) [21:48:27] (CR) Hashar: PayPal audit: check for existing recurring donations (1 comment) [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/370520 (https://phabricator.wikimedia.org/T172723) (owner: Ejegg) [21:48:58] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising-Backlog, Release-Engineering-Team, MW-1.29-release (WMF-deploy-2017-04-04_(1.29.0-wmf.19)), Patch-For-Review: Fix raw HTML tags breaking 1.29.0-wmf19+ - https://phabricator.wikimedia.org/T162716#3564291 (awight) [21:50:13] (PS3) Ejegg: PayPal audit: check for existing recurring donations [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/370520 (https://phabricator.wikimedia.org/T172723) [21:50:38] (CR) jerkins-bot: [V: -1] PayPal audit: check for existing recurring donations [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/370520 (https://phabricator.wikimedia.org/T172723) (owner: Ejegg) [21:51:21] (CR) Hashar: [C: 1] "\O/" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/370520 (https://phabricator.wikimedia.org/T172723) (owner: Ejegg) [21:52:12] (PS4) Ejegg: PayPal audit: check for existing recurring donations [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/370520 (https://phabricator.wikimedia.org/T172723) [21:53:42] cwd - arg no [21:54:00] do you know what 'caused' this replag [21:54:17] i do not [21:54:18] (what was causing load to show as replag I guess) [21:54:32] all i know is it was like 2 hours after i turned dedupe on [21:55:09] my theory is that the limit we are hitting is single threaded replication [21:55:14] just can't keep up with the master [21:55:37] but i have had about 8000 theories about this so far [21:58:36] (CR) Ejegg: "Take a look at the RecurringGlobalCollectTest for an example of setting up DonationInterface settings in tests. We could also add common s" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/370225 (owner: Mepps) [21:59:01] XenoRyet: did you see the note about the damaged messages? [21:59:50] No, must've missed that one. [22:00:11] so that is one way in which the faster hardware could make this problem worse [22:00:22] if it increases the differential in what the master and the replicas can do [22:00:59] (PS12) Ejegg: Enable PHPCS, but only with rules phpcbf could fix [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/285228 (https://phabricator.wikimedia.org/T133576) (owner: Awight) [22:01:13] anyway i feel like turning off dedupe *before* the banners would be a good test [22:01:41] since they might become clogged with bin logs fairly quickly but take awhile to report a problem [22:10:06] i'm going to take off for a bit but will check back later [22:28:18] XenoRyet|brb: it's a comment on the mistagged paypal refund ticket. we should make sure all the old ones get cleared up too! [22:31:13] ejegg: Yea, good call. [22:41:55] Remind me where that table lives and how best to get at it? [22:45:58] cwd turning off dedupe is a good thing to test. We can also tinker with the limit on the dedupe batch script [22:46:56] actually that limit might be higher now than it was during BE - shame we didn't have git on it back then [22:47:14] but I suspect I set it low during BE & increased it as load went down [22:50:19] ejegg|away: I was looking to review this but am a bif confused https://gerrit.wikimedia.org/r/#/c/374583/1 [22:54:48] (CR) Eileen: "I'm not 100% following this yet." (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374583 (https://phabricator.wikimedia.org/T170435) (owner: Ejegg) [22:59:28] (CR) Eileen: [C: 1] "So I think we should try this, but I think we should plan an outage / test period & also try on staging first. Note this appears to just b" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374588 (https://phabricator.wikimedia.org/T174404) (owner: Ejegg) [23:06:01] Fundraising Sprint Prank Seatbelt, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Drop IDs on cache tables - https://phabricator.wikimedia.org/T174404#3560617 (Eileenmcnaughton) Just a note that AUG said they got improved performance by converting one of these tables to MEM... [23:08:33] (CR) Eileen: [C: 1] "What say we see if drush queue_performance helps us at all? ie. run drush queue_performance on staging, then run this update, then re-run" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374588 (https://phabricator.wikimedia.org/T174404) (owner: Ejegg) [23:13:55] (CR) Eileen: [C: 2] Fix c_t.country population, add test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374597 (https://phabricator.wikimedia.org/T171658) (owner: Ejegg) [23:19:05] eileen1: sounds good, I will turn it off before bed tonight, unless you want to? I believe banners start at 9am pst [23:19:40] (Merged) jenkins-bot: Fix c_t.country population, add test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374597 (https://phabricator.wikimedia.org/T171658) (owner: Ejegg) [23:22:34] cwd we should just turn it off now [23:22:40] rather than try to remember timing [23:39:38] Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, Fundraising-Backlog, and 3 others: Are PayPal refunds for recurring donations incorrectly being tagged as EC or vice versa? - https://phabricator.wikimedia.org/T171351#3564563 (XenoRyet) @Ejegg... [23:46:39] eileen1: cool, doing [23:56:22] pre-test in dsk jaJP, starting in 3! @MBeat. How's your evening?