[04:58:50] Fundraising-Backlog, fundraising-tech-ops: Monitor and investigate possible event dropping by Kafkatee - https://phabricator.wikimedia.org/T239564 (AndyRussG) [06:57:50] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint X-rays, Fundraising-Backlog: FRUEC: Detailed comparison of events in old and new log files for banner impression pipeline - https://phabricator.wikimedia.org/T236834 (AndyRussG) == Detailed sta... [07:00:10] Fundraising-Backlog, fundraising-tech-ops: Monitor and investigate possible event dropping by Kafkatee - https://phabricator.wikimedia.org/T239564 (AndyRussG) [07:28:01] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint X-rays, Fundraising-Backlog: FRUEC: Debug minor discrepancy in landing page data between old and new pipelines - https://phabricator.wikimedia.org/T236835 (AndyRussG) >>! In T236835#5682487, @I... [07:44:04] Fundraising-Backlog: Investigate options for dropped CN EventLogging events for new pipeline - https://phabricator.wikimedia.org/T239570 (AndyRussG) [07:44:34] Fundraising-Backlog: Investigate options for dropped CN EventLogging events for new pipeline - https://phabricator.wikimedia.org/T239570 (AndyRussG) [07:44:51] Fundraising-Backlog: Investigate options for dropped CN EventLogging events for new pipeline - https://phabricator.wikimedia.org/T239570 (AndyRussG) [07:44:53] Fundraising Sprint Asymmetrical Earth Theory, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising-Backlog, Epic: [Epic] Fundraising kafkatee changes - https://phabricator.wikimedia.org/T183978 (AndyRus... [12:11:28] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint X-rays, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (KHaggard) @Ejegg One thing I would like to ask about: even though the jobs are running smoothly again, the to... [13:24:22] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint X-rays, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (Ejegg) @KHaggard thanks for including the unsubscribes in that screenshot. It looks like a drop of 401,614 do... [13:26:07] (CR) Ejegg: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/553419 (owner: Eileen) [13:56:41] Fundraising-Backlog: Investigate options for dropped CN EventLogging events for new pipeline - https://phabricator.wikimedia.org/T239570 (Ejegg) The old pipeline is parsing logs of hits to /beacon/impression, sent via sendBeacon with a fallback of creating an img with that src if navigator.sendBeacon is fals... [14:03:44] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint X-rays, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (CCogdill_WMF) @Ejegg nope, that sounds really high. We have a total suppression list of about 2M -- 426k woul... [14:06:15] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint X-rays, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (Ejegg) @CCogdill_WMF those import screenshots show it was 3.27M on Nov 21st and 3.69M on Dec 1st. I'm sure it... [14:09:53] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint X-rays, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (CCogdill_WMF) Ah, there must be duplicate rows on that import or something. We show 3,065,196 in IBM now. Hig... [14:14:39] Fundraising-Backlog: Investigate options for dropped CN EventLogging events for new pipeline - https://phabricator.wikimedia.org/T239570 (Ejegg) OK, I see that EventLogging uses the same img.src fallback as the old pipeline beacon-sender, but that EventLogging also will skip sending if navigator.doNotTrack o... [14:18:16] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint X-rays, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (Ejegg) OK, so the new issue is not missing donors, but an unexpected number of donors marked as unsubscribed... [15:00:02] (CR) Ejegg: [C: +1] "Looks like it fixes the issue! One possible simplification noted inline." (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/553420 (https://phabricator.wikimedia.org/T236855) (owner: Eileen) [15:24:59] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint X-rays, Fundraising-Backlog: Ingenico iframe says Donate/Cancel Donation in US, Pay/Cancel elsewhere - https://phabricator.wikimedia.org/T238366 (Ejegg) Open→Resolved [15:25:14] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint X-rays, Fundraising-Backlog, FR-Ingenico, MediaWiki-extensions-DonationInterface: Sending over-long first name field to Ingenico Connect - https://phabricator.wikimedia.org/T228094 (Ejegg) Open→Resolved [15:48:25] crud, adding new widgets to a board is broken :( [15:51:22] and... debugging dash in prod is basically useless with the minified js changing all the names [15:52:41] ok, it's broken un-minified too [15:52:44] which is a relief [15:53:26] debugging the minifier would be really hard [16:00:01] Fundraising-Backlog, Recurring-Donations: Recurring monthly conversion - processing date question - https://phabricator.wikimedia.org/T239627 (MBeat33) [16:12:24] banner team (seddon / spatton etc) heads up we're seeing some people being steered to the old globalcollect gateway which is disabled for all but ideal [16:12:34] looking up which banners it would be [16:14:39] Thanks ejegg, cc pcoombe too ^ [16:19:36] (PS1) Ejegg: Fix missing return val from displayPage [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/554100 [16:20:17] (PS2) Ejegg: Fix missing return val from displayPage [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/554100 [16:23:33] fr-tech anyone want to review that ^^^ ? [16:23:51] It's a one-liner, and fixes not being able to add new widgets to a board in dash [16:32:39] (CR) Cstone: [C: +2] "Looks good, was able to add new widgets." [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/554100 (owner: Ejegg) [16:32:49] thanks cstone! [16:33:03] crossing my fingers hoping I can still minify :P [16:33:10] haha i should have realized it was broken sooner I thought you just couldnt add more widgets haha [16:33:12] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint X-rays, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (KHaggard) I can open a new phab task - just confirming that: the contacts in the MSL should not be on the _al... [16:33:15] (Merged) jenkins-bot: Fix missing return val from displayPage [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/554100 (owner: Ejegg) [16:33:28] good luck with the minify [16:35:43] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/554102 [16:37:26] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/554102 (owner: Ejegg) [16:38:00] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/554102 (owner: Ejegg) [16:43:14] !log updated fundraising internal dashboard from 8fc2726736 to 3a93d2aba4 [16:43:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:55:32] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint X-rays, Fundraising-Backlog, FR-Adyen: Adyen audit question: Nov refunds reaching Civi - https://phabricator.wikimedia.org/T238428 (Ejegg) This looks like it's related to the small volume. I guess Adyen only sends us an audit file afte... [17:00:27] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint X-rays, Fundraising-Backlog, FR-Adyen: Adyen audit question: Nov refunds reaching Civi - https://phabricator.wikimedia.org/T238428 (Ejegg) Oh, even in October we were only getting new files on Fridays. It looks like we would need to su... [17:00:58] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint X-rays, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (Ejegg) Yep, we only export a given email address to one list or the other, never both. [17:09:05] !log disabled fundraising job omnimail_groupmember_load [17:09:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:44:40] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint X-rays, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (KHaggard) Thanks! New ticket is here: [17:45:31] Fundraising-Backlog: Investigation: Tracking the drop in _all_wikimedia contact source job total rows - https://phabricator.wikimedia.org/T239635 (KHaggard) [17:46:14] Fundraising-Backlog: Investigation: Tracking the drop in _all_wikimedia contact source job total rows - https://phabricator.wikimedia.org/T239635 (KHaggard) [18:05:55] Fundraising-Backlog: Investigate options for dropped CN EventLogging events for new pipeline - https://phabricator.wikimedia.org/T239570 (AndyRussG) I think it's blocking on the URL path `/beacon/event`. See https://easylist.to/easylist/easyprivacy.txt and T220627#5638168. I kinda hope the first option woul... [19:01:30] Fundraising-Backlog, fundraising-tech-ops: Issue new SSL Client Certificate for ccogdill - https://phabricator.wikimedia.org/T238757 (Dwisehaupt) Open→Resolved p:Triage→Normal renewed and updated CRL pushed. [frack::puppet::private] bf09bab Reissuing of ccogdill client ssl cert [19:02:29] Fundraising-Backlog, fundraising-tech-ops: Issue new SSL Client Certificate for jseddon - https://phabricator.wikimedia.org/T238762 (Dwisehaupt) Open→Resolved p:Triage→Normal renewed and updated CRL pushed [frack::puppet::private] 2f35cab Reissuing of jseddon client ssl cert [19:06:31] Wikimedia-Fundraising-Banners: RML Nag button does not wrap text correctly in IE11 - https://phabricator.wikimedia.org/T239649 (jbolorinos-ctr) [19:16:23] Wikimedia-Fundraising-Banners: RML Nag button does not wrap text correctly in IE11 - https://phabricator.wikimedia.org/T239649 (spatton) Thanks @jbolorinos-ctr, I can reproduce this in CrossBrowserTesting for IE11. I'm tagging @EWilfong_WMF in, too - Eric, can someone from Trilogy check this out? @Pcoombe, j... [19:49:42] (PS1) Ejegg: Allow rate limiting client-side error API [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/554133 (https://phabricator.wikimedia.org/T181748) [19:50:10] fr-tech looks like it's really easy to get rate-limiting in Mediawiki ^^^ [19:50:22] just going to test that one out locally [19:58:11] Seems quiet here - am I actually connected [19:58:40] hi eileen! [19:59:00] yeah, not a lot of hubbub in the channel [19:59:00] hey - just trying to catch up & see if all is well - maybe a deadlock just turned up? [19:59:36] ooh look, there IS a bit of failmail [19:59:44] let's see what kind of queries are happening [19:59:52] killing a query [20:00:43] what was it? [20:01:53] it was the one Dallas killed the other week by Rosie [20:02:21] she didn't know what she did to cause it & I hadn't asked more [20:02:23] but it's contact group 493 [20:02:54] ohh, with a distance filter [20:04:38] actually it was Pittsburgh Proximity Search (Smaller) [20:05:06] but I think Rosie might be doing something that refreshes group counts because then it tried another group [20:10:38] ok, so should we just delete that smart group [20:10:40] ? [20:10:51] seems unlikely to be used very often [20:19:49] ejegg: apparently Nora does use it - I think the issue is Rosie does 'something' that causes it to refresh a bunch of groups [20:20:00] https://usercontent.irccloud-cdn.com/file/gC3ATj5E/Screen%20Shot%202019-12-03%20at%209.15.08%20AM.png [20:20:26] expanding this would do it - we can actually disable that - now that Nora has created group craziness [20:20:28] And is Nora able to get useful results out of it? Would it be better to use a set of zip codes ? [20:21:02] I'm not sure that one was hung tbh - I think it was cycling through the groups - it just looked like lastt ttime [20:21:07] The SQL generated does some trig with the lat and long in the filter, and I'm pretty sure we don't have any indexes for that [20:23:18] yeah - it's ok for them to search an indexed field sometimes - the thing is it seems like Rosie didn't knowingly cause that group to refresh [20:56:04] Jeff_Green: are you using thinkfan ? [20:56:25] no, it's just that the fan itself is failing [20:56:48] I'm waiting for OIT to tell me what to do about it [20:57:05] ahh, dang [20:57:20] it's gotten really bad over the past day or two [21:08:20] Fundraising Sprint X-rays, Fundraising-Backlog: Investigation: Tracking the drop in _all_wikimedia contact source job total rows - https://phabricator.wikimedia.org/T239635 (XenoRyet) [21:08:43] Fundraising Sprint X-rays, Fundraising-Backlog, Recurring-Donations: Recurring monthly conversion - processing date question - https://phabricator.wikimedia.org/T239627 (XenoRyet) [21:10:54] Fundraising Sprint X-rays, Fundraising-Backlog: Investigation: Tracking the drop in _all_wikimedia contact source job total rows - https://phabricator.wikimedia.org/T239635 (Ejegg) Related discussion - as we import events from Silverpop, we may be turning temporary suppression into permanent opt-outs in... [21:12:43] Fundraising Sprint Usual Subscripts, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Civi-Dedupe: New dedupe blocking issue - on hold - https://phabricator.wikimedia.org/T221914 (Eileenmcnaughton) We unsubscribe based on the following silverpop... [21:28:00] Fundraising Sprint X-rays, Fundraising-Backlog: Investigation: Tracking the drop in _all_wikimedia contact source job total rows - https://phabricator.wikimedia.org/T239635 (Eileenmcnaughton) @KHaggard this is consistent with the number of contacts unsubscribed based on us having grabbed the following ev... [21:31:04] (CR) XenoRyet: [C: +2] Allow rate limiting client-side error API [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/554133 (https://phabricator.wikimedia.org/T181748) (owner: Ejegg) [21:32:59] (Merged) jenkins-bot: Allow rate limiting client-side error API [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/554133 (https://phabricator.wikimedia.org/T181748) (owner: Ejegg) [21:33:00] mepps: are we catching up? [21:33:11] yes sorry eileen [21:43:42] XenoRyet: I added myself to on call with you on 2 days because I can take over when I come online since those are Mondays [21:43:44] for me [21:43:58] also cstone if you want me to do xmas that is boxing day for me [21:44:06] Cool, sounds good. [21:45:31] eileen: I dont mind I am not doing anything for xmas [22:08:15] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/554170 [22:08:44] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/554170 (owner: Ejegg) [22:14:07] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/554171 [22:16:09] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/554170 (owner: Ejegg) [22:17:19] (CR) Ejegg: [C: +2] Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/554171 (owner: Ejegg) [22:18:22] (CR) jerkins-bot: [V: -1] Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/554171 (owner: Ejegg) [22:20:14] (CR) Ejegg: [C: +2] Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/554171 (owner: Ejegg) [22:21:45] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/554171 (owner: Ejegg) [22:46:19] !log updated payments-wiki from 06a8c3cdff to f61c9f0692 [22:46:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:35:57] fr-tech our queue consumption has drastically slowed down [23:36:08] and the queue length is getting really long [23:36:23] anyone else around to help figure out why? [23:36:41] see https://frmon.frdev.wikimedia.org/d/Pq1YNMviz/fundraising-overview?refresh=1m&orgId=1&panelId=22&fullscreen&from=now-3h&to=now [23:36:45] Yea, I'm around [23:36:54] ejegg: i was noticing that right as i was wrapping up in the waiting room. [23:38:44] Seems to be going up again just now [23:39:09] Looks like there was some failmail right around when it happened, but no details. [23:41:14] looking at the frdb1002, it looks like there may have been db contention [23:41:39] https://frmon.frdev.wikimedia.org/d/000000273/mysql?orgId=1&var-dc=Prometheus&var-server=frdb1002.frack.eqiad.wmnet&var-port=9004&from=1575319281974&to=1575330081974 [23:41:47] doing some more digging. [23:47:04] i don't see any stinkers in the slow log, but i do see that it started taking ~10-12 sec to do an insert for a contribution. [23:49:47] Queue size is going down again [23:51:10] Yea, seems to be back up to speed, working through the backlog. [23:51:16] not seeing anything glaring on the host metrics that indicate any change around that time. [23:55:30] ok, I'mma head out for now and take a peek at the metrics later [23:58:12] (I also have to be afk for a bit, gotta walky wog, back in like 25 min) [23:58:12] hi all - the overlap between the slow down & us experimenting on MG call with Benevity import is too close to ignore [23:58:35] note that the Benevity import does get a bit slow on some specific contacts [23:58:43] queue is catching up OK now [23:58:50] hmmmm [23:58:52] XenoRyet: ^ [23:58:52] & MG won't try the import again this week [23:58:56] eileen: that's interesting. [23:59:05] and good to note. [23:59:26] eileen: how would we have been able to diagnose this if we hadn't had your info? [23:59:56] Yeah - sorry we were actually all discussing on the MG call & not looking on IRC - we noticed the slow down & stayed in the call