[00:00:11] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 1330229 Threads: 1 Questions: 84984050 Slow queries: 8010 Opens: 11620 Flush tables: 1 Open tables: 610 Queries per second avg: 63.886 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [00:34:29] (PS2) Mepps: Fix duplicate check logic [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/367682 (https://phabricator.wikimedia.org/T171349) [00:40:38] (CR) jerkins-bot: [V: -1] Fix duplicate check logic [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/367682 (https://phabricator.wikimedia.org/T171349) (owner: Mepps) [00:43:01] (PS3) Mepps: Fix duplicate check logic [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/367682 (https://phabricator.wikimedia.org/T171349) [00:44:50] (PS4) Mepps: Fix duplicate check logic [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/367682 (https://phabricator.wikimedia.org/T171349) [00:55:11] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2315 [01:00:11] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2313 [01:05:11] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2318 [01:25:11] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 1333970 Threads: 1 Questions: 87549463 Slow queries: 7270 Opens: 11780 Flush tables: 1 Open tables: 608 Queries per second avg: 65.630 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [01:30:11] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1242 [01:35:11] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1413 [01:40:11] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 1336229 Threads: 1 Questions: 87915829 Slow queries: 8014 Opens: 11839 Flush tables: 1 Open tables: 610 Queries per second avg: 65.793 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 139 [03:09:00] (PS1) Eileen: Limit Silverpop group import to Opt In by default [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/368125 [03:29:20] !log disabled recurring Ingenico charges [03:29:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:42:23] (PS5) Ejegg: Fix duplicate check logic [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/367682 (https://phabricator.wikimedia.org/T171349) (owner: Mepps) [03:45:17] (CR) Ejegg: [C: 2] "That check works! I just realized the getSingle call is a bit overkill to just see if a contribution exists, though. It looks like it join" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/367682 (https://phabricator.wikimedia.org/T171349) (owner: Mepps) [03:53:37] (Merged) jenkins-bot: Fix duplicate check logic [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/367682 (https://phabricator.wikimedia.org/T171349) (owner: Mepps) [04:37:18] Fundraising Sprint Navel Warfare, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email, Patch-For-Review: Import email-only contacts from 'remind me later' links into CiviCRM - https://phabricator.wikimedia.org/T160949#3477017 (Eileenmcnaughton) [04:39:10] Fundraising Sprint Navel Warfare, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email, Patch-For-Review: Import email-only contacts from 'remind me later' links into CiviCRM - https://phabricator.wikimedia.org/T160949#3115981 (Eileenmcnaughton) @CCogdill_WMF I've imported 16 of these... [06:02:43] (PS1) Eileen: Add throttling to Omnrecipients.load function [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/368127 (https://phabricator.wikimedia.org/T161762) [06:36:06] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising Sprint Homebrew Hadron Collider, Fundraising Sprint Ivory Tower Defense Games, Fundraising Sprint Judgement Suspenders, and 8 others: retrieve the text/ html and statistics data for m... - https://phabricator.wikimedia.org/T161758#3477076 [06:36:43] Fundraising Sprint Murphy's Lawyer, Fundraising Sprint Navel Warfare, Wikimedia-Fundraising-CiviCRM, Patch-For-Review, Unplanned-Sprint-Work: increase email limit from 50 to 700 in civi - https://phabricator.wikimedia.org/T170900#3477078 (Eileenmcnaughton) @DKaufman did this work for you? Is... [06:52:25] Fundraising Sprint Judgement Suspenders, Fundraising Sprint Kickstopper, Fundraising Sprint Navel Warfare, Fundraising-Backlog, and 4 others: retrieve lists of contacts who received a particular mailing - https://phabricator.wikimedia.org/T161762#3477093 (Eileenmcnaughton) I just added a patch to... [08:27:34] (CR) Hashar: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/151840 (owner: Hashar) [08:29:38] (CR) jerkins-bot: [V: -1] Jenkins job validation (DO NOT SUBMIT) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/151840 (owner: Hashar) [08:30:02] (CR) Hashar: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/151840 (owner: Hashar) [08:31:21] (CR) Hashar: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/363141 (https://phabricator.wikimedia.org/T161724) (owner: Hashar) [08:32:06] (Abandoned) Hashar: Jenkins job validation (DO NOT SUBMIT) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/151840 (owner: Hashar) [08:33:27] (CR) jerkins-bot: [V: -1] CI: install CiviCRM with a fake sendmail [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/363141 (https://phabricator.wikimedia.org/T161724) (owner: Hashar) [08:48:02] (CR) Hashar: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/363141 (https://phabricator.wikimedia.org/T161724) (owner: Hashar) [08:49:19] (CR) Hashar: [C: 1] "The previous failure was a job running on Nodepool disposable instance. It goes pass Drupal complaining it can not send an email due to la" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/363141 (https://phabricator.wikimedia.org/T161724) (owner: Hashar) [08:56:01] :( [08:56:43] (CR) Hashar: "Or maybe setting sendmail = /usr/bin/true causes omnimail to fail somehow :-(" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/363141 (https://phabricator.wikimedia.org/T161724) (owner: Hashar) [14:19:28] morning ejegg [14:52:19] hi mepps! [14:52:56] how's your morning going ejegg? [14:54:26] jut getting started, a bit late [14:54:32] and yours? [15:09:43] not too bad, finally found the test i need to work on next in DI [15:22:04] (PS10) AndyRussG: Controls to purge banner content from front-end cache for a language [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/364910 (https://phabricator.wikimedia.org/T168673) [15:22:19] ^ re-smoke-tested :) [15:26:34] mepps cool! [15:27:01] currently looking at how defineTransactions and addCodeRange works [15:27:23] mepps oh hey, do you know if there's an API call that's even lower impact than getSingle, for just testing if something exists? [15:27:33] or the right params for getSingle? [15:27:49] cool AndyRussG, taking a look! [15:29:00] ejegg hmm maybe just getcount? [15:29:18] cool [15:31:01] oh right, I need to fix the ingenico recurring charge job [15:32:17] (PS11) Ejegg: Controls to purge banner content from front-end cache for a language [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/364910 (https://phabricator.wikimedia.org/T168673) (owner: AndyRussG) [15:32:27] (CR) Ejegg: [C: 2] "Looks great!" (2 comments) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/364910 (https://phabricator.wikimedia.org/T168673) (owner: AndyRussG) [15:33:04] Jeff_Green: ejegg how bad is this civi situation? Eileen is not on today or tomorrow. Does anything need to be shut off? [15:33:17] also is this related: https://phabricator.wikimedia.org/T171858 [15:34:19] dstrine: that one was my fault, something in the recurring charge apparatus was dying when it tried to send the antifraud queue message [15:34:30] I'm trying to fix that right now [15:35:02] " recurring charge apparatus " = civi or T171858 or both? [15:35:08] also MBeat fyi^ [15:35:30] interesting, ty dstrine [15:37:32] dstrine: T171858 [15:40:16] (CR) jerkins-bot: [V: -1] Controls to purge banner content from front-end cache for a language [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/364910 (https://phabricator.wikimedia.org/T168673) (owner: AndyRussG) [15:40:40] (PS1) Mepps: Simplify duplicate check call [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/368191 [15:40:50] ejegg see above [15:43:03] (PS1) Ejegg: Initialize SmashPig context for recurring Ingenico charges [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/368193 (https://phabricator.wikimedia.org/T171858) [15:43:29] (CR) Ejegg: [C: 2] "recheck" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/364910 (https://phabricator.wikimedia.org/T168673) (owner: AndyRussG) [15:43:32] thanks mepps! [15:43:47] mepps, mind taking a look at that 'Initialize SmashPig context' patch? [15:44:00] one-liner, oughtta fix the recurring Ingenico bug from last night [15:44:31] If you look at the process-control logs, you see a 'called getSourceName on null' error [15:44:39] which means there's no active SmashPig context [15:45:38] sure! [15:46:08] (CR) Mepps: [C: 2] Initialize SmashPig context for recurring Ingenico charges [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/368193 (https://phabricator.wikimedia.org/T171858) (owner: Ejegg) [15:47:59] (CR) Ejegg: [C: 1] "Thanks!" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/368191 (owner: Mepps) [15:48:09] Jeff_Green: are you actively concerned about the civi lag? [15:49:00] dstrine: it's been a problem when the email history import job is running, but right now it isn't running [15:49:51] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/368195 [15:49:57] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/368195 (owner: Ejegg) [15:50:06] Jeff_Green: ok thanks. do you think we need to do anything before Eileen is on again next week? [15:51:08] as long as it is disabled then it can wait [15:52:58] ok [15:58:21] (Merged) jenkins-bot: Controls to purge banner content from front-end cache for a language [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/364910 (https://phabricator.wikimedia.org/T168673) (owner: AndyRussG) [16:00:57] (Merged) jenkins-bot: Initialize SmashPig context for recurring Ingenico charges [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/368193 (https://phabricator.wikimedia.org/T171858) (owner: Ejegg) [16:00:59] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/368195 (owner: Ejegg) [16:01:47] !log updated CiviCRM from e83c012581305012145eae45495e7e8ea6f4e249 to ceff739dcfeb5e5bcf40d880146cfb44eaf462ea [16:01:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:06:10] ejegg: thanks! Hmmm I think I've seen that qunit test flap before... I guess we should keep an eye on it... [16:06:34] AndyRussG: yeah, it's always the localStorage expiry one [16:06:44] I think it's intentionally randomized or soemthing? [16:13:14] mmm dunno [16:13:32] I'll check.... [16:13:46] shouldnt be anything random in tests [16:18:39] (PS2) Ejegg: Add throttling to Omnrecipients.load function [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/368127 (https://phabricator.wikimedia.org/T161762) (owner: Eileen) [16:19:13] (CR) Ejegg: [C: 2] "Cool, configurable rows/sec seems like just the thing to keep ops happy! Would be really cool if we could generalize this." (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/368127 (https://phabricator.wikimedia.org/T161762) (owner: Eileen) [16:23:58] AndyRussG: I meant I think the expiration is intentionally randomized, to avoid browser freezeups on excessive background processing [16:31:06] (Merged) jenkins-bot: Add throttling to Omnrecipients.load function [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/368127 (https://phabricator.wikimedia.org/T161762) (owner: Eileen) [16:47:36] MBeat: did you already push through those 4 stranded txns from overnight? [16:47:50] yah, i settled ‘em ejegg [16:48:07] OK, cool, I'm about to fix the db records so they go through right next month [16:48:15] great, ty! [16:51:55] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Ingenico: Deal with recurring donations stuck in 'In Progress' status - https://phabricator.wikimedia.org/T171868#3478795 (Ejegg) [16:58:38] ejegg: hmmm I'll check! Anyway test-wise needs fixin' [16:59:15] yarp [17:00:39] hrrm, looks like the recurring job will refuse to run if I just tweak the recurring record without the latest contributions in there. I've set a reminder to fix those recurring records on Monday, when the contributions should exist thanks to the audit file [17:00:49] ok... what's next up? [17:01:12] mepps: do you mind if I just tweak that patch to be $duplicate > 0 ? [17:01:37] I know it's the same result, just feeling a little ocd about it for some reason [17:01:54] err, sorry to anyone with actual ocd [17:05:03] !log restarted recurring Ingenico charge job [17:05:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:09:34] Sure ejegg [17:09:40] thanks! [17:24:29] (PS1) AndyRussG: Comments and minor no-op cleanup [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/368216 [17:25:11] dstrine: hey, could you please make a new component tag for 'process-control' [17:26:29] (CR) jerkins-bot: [V: -1] Comments and minor no-op cleanup [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/368216 (owner: AndyRussG) [17:27:56] Fundraising-Backlog: process-control should make slow-starting jobs easier - https://phabricator.wikimedia.org/T171873#3478936 (Ejegg) [17:28:12] (PS1) Ejegg: WIP slow-start jobs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/368217 (https://phabricator.wikimedia.org/T171873) [17:28:42] (CR) jerkins-bot: [V: -1] WIP slow-start jobs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/368217 (https://phabricator.wikimedia.org/T171873) (owner: Ejegg) [17:31:52] (PS2) AndyRussG: Comments and minor no-op cleanup [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/368216 [17:32:26] (PS2) Ejegg: WIP slow-start jobs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/368217 (https://phabricator.wikimedia.org/T171873) [17:33:34] (PS2) Ejegg: Simplify duplicate check call [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/368191 (owner: Mepps) [17:34:06] (PS3) Ejegg: Simplify duplicate check call [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/368191 (owner: Mepps) [17:34:21] (CR) jerkins-bot: [V: -1] Comments and minor no-op cleanup [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/368216 (owner: AndyRussG) [17:34:30] (CR) Ejegg: [C: 2] "Thanks, mepps!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/368191 (owner: Mepps) [17:37:14] aaarg silly brian [17:37:17] brain [17:39:52] ejegg what are these lower and upper parameters in addCodeRange in DI? [17:40:19] mepps ah, man, that's pretty globalcollect-specific [17:40:26] err, ingenico [17:40:49] so, ingenico has numeric codes to define the status of a payment [17:41:09] and there can be a bunch in a range that mean the same thing to us [17:41:33] ahh i see [17:41:44] so addCodeRange lets us define a whole set of codes as meaning success, failure, etc, without having to list them individually [17:43:29] AAAAAARGH! still seeing dead session errors in yesterday's paypal EC test [17:43:34] what. the. hell. [17:45:52] ok, some of these people are coming back after 7 hours [17:46:30] ah ejegg that would make sense [17:47:25] but most of them are much more timely [17:47:34] (Merged) jenkins-bot: Simplify duplicate check call [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/368191 (owner: Mepps) [17:48:36] darn [17:48:39] AAARGH, more paypal audit failmail [17:50:20] ok, I'mma eat lunch before I deal with that stuff [17:50:45] (PS3) AndyRussG: Comments and minor no-op cleanup [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/368216 [17:51:12] anyway those are nice little CI tests for when one's brain isn't working as advertised [17:52:08] oh fnord, are we not ensuring a session on the first page-render? [17:52:44] and does that mean we're inserting twice as many ct_ids as necessary? [17:52:47] ugggggh [17:56:54] ejegg|food: bon appétit, don't let the "ugggggh" spoil ur arepas :) [18:03:04] ejegg|food let me know if you want to go over this when you get back [18:06:11] happy anniversary MBeat [18:06:29] ty ejegg|food “letting the days go by…” [18:06:43] and a happy 6 year mark to Jeff_Green too! [18:07:08] ha! thanks [18:07:22] +1 go Jeff_Green [18:22:42] mepps sure, I'd love some help figuring this out [18:24:08] Jeff_Green: could you get us access to yesterday's web logs for the payments cluster (on the logging box, if possible?) We're just looking at 2017-07-26 13:00:00 - 15:00:00 [18:24:21] sure [18:24:28] do you want nginx or apache logs? [18:24:52] nginx please! [18:24:58] ok [18:27:09] thank you [18:30:54] ejegg see frlog1001:/tmp/logs [18:33:44] thank you! [18:33:50] np [18:41:11] Jeff_Green: sorry, I think we need the next day's logs - those cut off at 2017-07-26 06:25 [18:42:42] oh crud, sorry I thought about that and apparently went the wrong direction grabbing the extra day :-( [18:43:49] no worries [18:47:41] ejegg: ok they're there now [18:47:51] thanks again Jeff_Green [18:50:38] yw [19:36:20] Fundraising Sprint Kickstopper, Fundraising Sprint Loose Lego Carpeting, Fundraising Sprint Murphy's Lawyer, Fundraising Sprint Navel Warfare, and 4 others: PayPal EC dead session error - https://phabricator.wikimedia.org/T167923#3349710 (mepps) One short term fix is to add different error messag... [19:36:31] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Compound order IDs should never end in ".0" - https://phabricator.wikimedia.org/T171891#3479438 (Ejegg) [19:43:19] Fundraising Sprint Kickstopper, Fundraising Sprint Loose Lego Carpeting, Fundraising Sprint Murphy's Lawyer, Fundraising Sprint Navel Warfare, and 4 others: PayPal EC dead session error - https://phabricator.wikimedia.org/T167923#3479491 (Ejegg) [19:43:21] Fundraising Sprint Kickstopper, Fundraising Sprint Loose Lego Carpeting, Fundraising Sprint Murphy's Lawyer, Fundraising Sprint Navel Warfare, and 4 others: Resultswitchers: send straight to ty page on reload - https://phabricator.wikimedia.org/T167990#3479488 (Ejegg) Resolved>Open Dang,... [20:12:30] dstrine MBeat we are going to create a new error page for the dead session error, would love feedback on content for that: https://phabricator.wikimedia.org/T167923 [20:12:49] got it, thanks mepps [20:29:19] (CR) Umherirrender: "See also T155182" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/368108 (owner: Ejegg) [20:30:14] PROBLEM - check_puppetrun on alnitak is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 5 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[rsyslog-gnutls],Package[rsyslog] [20:31:34] (CR) Raimond Spekking: [C: 2] Fix blank i18n message added by TranslateWiki [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/368108 (owner: Ejegg) [20:35:14] RECOVERY - check_puppetrun on alnitak is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:40:00] (Merged) jenkins-bot: Fix blank i18n message added by TranslateWiki [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/368108 (owner: Ejegg) [20:55:07] (PS1) Ejegg: WIP order sequence numbers start at 1, not 0 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/368266 (https://phabricator.wikimedia.org/T171891) [21:06:35] (CR) jerkins-bot: [V: -1] WIP order sequence numbers start at 1, not 0 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/368266 (https://phabricator.wikimedia.org/T171891) (owner: Ejegg) [21:28:12] (PS1) Ejegg: Store processed payments in main cache, not session [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/368307 (https://phabricator.wikimedia.org/T167990) [21:36:29] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/368311 [21:36:37] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/368311 (owner: Ejegg) [21:37:26] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/368311 (owner: Ejegg) [21:46:12] !log updated CiviCRM from ceff739dcfeb5e5bcf40d880146cfb44eaf462ea to 23f2bbf73557a7a88e783f68459112cf4bba1c79 [21:46:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:53:33] ejegg: I see you deployed that throttling - is it sill off? [21:53:48] eileen1: yep, I was just about to try a run [21:54:07] cwd: do you have an idea of how many inserts per second cause replag? [21:54:22] not really - that's why I thought Jeff's offer to watch was good [21:54:38] can you add start_date=2016-12-15 to the first run [21:54:54] I'm not sure if it got cut off last run so putting it back a step will cover that [21:55:13] (although might be less good for testing throttling - maybe throttle test first - hmm [21:55:36] ejegg: i think it depends [21:55:51] eileen1: well, let me try it with that and the default throttling (100,000 rows in 5 minutes) [21:55:54] what kind of load the servers are under otherwise [21:55:59] the nature of the inserts [21:57:27] I also don't know how INSERT IGNORE commands affect replication - ie. if the IGNORE part happens is there anything to replicate [21:57:30] ok, that seemed to finish pretty quick [21:57:46] hmm - might be no emails [21:57:59] or just kicking off a download [21:58:45] ok - I am relocating (have a meeting about Hunderwasser project now but will go online from in there) [22:02:12] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, Performance-Team, WMDE-Fundraising-CN: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591#3436052 (ksmith) Is this in the hands of the Translate extension team a... [22:08:07] Fundraising Sprint Navel Warfare, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email, Patch-For-Review: Import email-only contacts from 'remind me later' links into CiviCRM - https://phabricator.wikimedia.org/T160949#3479906 (Eileenmcnaughton) a:Eileenmcnaughton [22:08:15] back on [22:10:27] looks like replication can keep up with 55k in 3:30 [22:10:46] just based on running select now(), count(*) from civicrm_mailing_provider_data; on the slave db [22:11:30] I ran another day with the default throttle settings, and the subscriber copy is still catching up after 2 min [22:12:14] so once that settles I'll run another throttling to like 15k every 60 seconds [22:21:06] ejegg: ok cool [22:21:12] thanks for testing that [22:25:07] Fundraising Sprint Navel Warfare, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email, Patch-For-Review: Import email-only contacts from 'remind me later' links into CiviCRM - https://phabricator.wikimedia.org/T160949#3479917 (Ejegg) @Eileenmcnaughton I think we only export records t... [22:25:41] Fundraising Sprint Navel Warfare, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email, Patch-For-Review: Import email-only contacts from 'remind me later' links into CiviCRM - https://phabricator.wikimedia.org/T160949#3479920 (Eileenmcnaughton) ah! [22:26:36] guess it's too much to expect progress indicators during the course of an API call [22:32:36] d'oh, it would help if I changed the start and end dates [22:35:55] huh, definitely some lag in the subscriber even before the inserts get heavy [22:37:08] ok, it seems to be pausing [22:37:38] at approximately 15k rows [22:38:06] and there it goes again [22:38:14] throttling seems to work eileen! [22:38:38] I'll add those settings to the scheduled job and turn it back on [22:46:52] !log enabled omnimail recipient load job, throttling inserts to 15,000 every 60 sec [22:47:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:47:23] ok, I'm going to hit the road [23:02:49] i gotta roll too, i am out tomorrow, but i'll check in in the morning [23:05:14] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3677 [23:05:25] Fundraising Sprint Navel Warfare, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email, Patch-For-Review: Import email-only contacts from 'remind me later' links into CiviCRM - https://phabricator.wikimedia.org/T160949#3480088 (CCogdill_WMF) @Eileenmcnaughton @Ejegg it looks like that... [23:10:04] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3709 [23:15:14] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2991 [23:16:33] !log disabled Omnimail recipient load job [23:16:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:19:55] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising Sprint Homebrew Hadron Collider, Fundraising Sprint Ivory Tower Defense Games, Fundraising Sprint Judgement Suspenders, and 8 others: retrieve the text/ html and statistics data for m... - https://phabricator.wikimedia.org/T161758#3480119 [23:20:14] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2766 [23:25:14] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2763 [23:25:17] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising Sprint Homebrew Hadron Collider, Fundraising Sprint Ivory Tower Defense Games, Fundraising Sprint Judgement Suspenders, and 8 others: retrieve the text/ html and statistics data for m... - https://phabricator.wikimedia.org/T161758#3480121 [23:25:34] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising Sprint Homebrew Hadron Collider, Fundraising Sprint Ivory Tower Defense Games, Fundraising Sprint Judgement Suspenders, and 8 others: retrieve the text/ html and statistics data for m... - https://phabricator.wikimedia.org/T161758#3480122 [23:30:14] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 1413470 Threads: 1 Questions: 92335253 Slow queries: 7759 Opens: 14715 Flush tables: 1 Open tables: 600 Queries per second avg: 65.325 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [23:33:28] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising Sprint Homebrew Hadron Collider, Fundraising Sprint Ivory Tower Defense Games, Fundraising Sprint Judgement Suspenders, and 8 others: retrieve the text/ html and statistics data for m... - https://phabricator.wikimedia.org/T161758#3480140 [23:36:40] !log update process-control to 2c1c8a3bcb0186 - new frequency on receipient load [23:36:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:39:27] !log update process-control to 24c7bbe699a6bb685 (renable omnirecipient) [23:39:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log