[00:01:10] AndyRussG or cstone: would one of you be able to attend the scrum of scrums meeting tomorrow? I have a conflicting meeting [00:01:56] it's at 11:30 AM US eastern time [00:12:02] Fundraising Sprint Rocky Horror Presentation Layer, Fundraising Sprint Sysadmin Kane, Fundraising-Backlog: 500 error when testing Endowment thank you emails in Civi - https://phabricator.wikimedia.org/T232504 (Ejegg) p:Triage→Normal a:Ejegg [01:02:24] Fundraising Sprint Never Ending Query, Fundraising Sprint Office  , Fundraising Sprint Rocky Horror Presentation Layer, Fundraising Sprint Sysadmin Kane, and 2 others: One-time import of target smart data into IBM -- how to do this? - https://phabricator.wikimedia.org/T231538 (Eileenmcnaughto... [01:02:55] Fundraising Sprint Rocky Horror Presentation Layer, Fundraising Sprint Sysadmin Kane, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Gender for original TargetSmart list - https://phabricator.wikimedia.org/T231362 (Eileenmcnaughton) @LeanneS OK to close? [01:03:26] hello [01:34:02] hi eileen ! [01:34:10] heya [01:34:31] how's Malaysia? [01:35:06] so I was looking at making that silverpop timeout configurable [01:35:30] I submitted a couple of little patches to mrmarkfrench and he merged them like within the hour [01:35:47] think it's OK to stop pointing at the fork repo? [01:36:04] and maybe get rid of the *Guzzle* wrappers in ominmail-silverpop? [01:36:22] not that the task is in sprint... [01:36:50] also, thanks for that scraper off switch [01:37:20] those IP address confirmation emails make it look like we need to enter that code from a browser with the same cookies as the one that made the login attempt [01:37:39] which would be tricky - especially if it wants the same IP address [01:38:20] did you try entering the confirmation code from 'any' browser? [01:38:38] err, no... I guess I should do that :P [01:38:43] Re upstream repo - I think I HAD stopped at one point then he merged all my PRs so I started again [01:38:54] I assume the port is still closed at the moment though? [01:39:11] eileen: oh yeah, we'd need to ask ops to open it again [01:43:19] ejegg: yeah which won't be until tomorrow I guess [01:44:09] is there something I can review on the time out - I saw you did some work on it? I guess it's tangental huh? [01:46:43] eileen: nope, the stuff I added to upstream silverpop-php-connector was pretty basic [01:47:16] next step would be to add it to omnimail-silverpop, which is why I was asking about tearing out those Guzzle-wrapper classes [01:47:23] which now appear to be unnecessary [01:48:00] anyway, it's late here. I'll take another look at the IP confirmation code tomorrow [01:48:36] have a good one! [01:55:43] ok cool [09:15:16] Fundraising Sprint Never Ending Query, Fundraising Sprint Office  , Fundraising Sprint Rocky Horror Presentation Layer, Fundraising Sprint Sysadmin Kane, and 2 others: One-time import of target smart data into IBM -- how to do this? - https://phabricator.wikimedia.org/T231538 (CCogdill_WMF)... [10:05:20] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 5010 (threshold c = 5000) [10:10:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 5140 (threshold c = 5000) [10:15:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 5301 (threshold c = 5000) [10:20:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 5431 (threshold c = 5000) [10:25:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 5523 (threshold c = 5000) [10:30:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 5638 (threshold c = 5000) [10:35:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 5761 (threshold c = 5000) [10:40:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 5806 (threshold c = 5000) [10:45:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 5934 (threshold c = 5000) [10:50:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 6005 (threshold c = 5000) [10:55:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 6127 (threshold c = 5000) [11:00:20] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 6180 (threshold c = 5000) [11:05:20] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 6315 (threshold c = 5000) [11:10:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 6357 (threshold c = 5000) [11:15:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 6461 (threshold c = 5000) [11:18:05] fr-tech, looks like a civi1001 mailq is backing up [11:18:43] I can see a few timeouts when connecting to gmail and TLS handskake failures connecting to hotmail [11:19:10] same with yahoo [12:02:15] Fundraising-Backlog: Update email copy for the endowment Thank you email - https://phabricator.wikimedia.org/T228843 (jrobell) @Dstrine - Just checking in on the timeline. Based on our convo on tech priorities, I thought this task would be looked at this sprint, however I can see it being scheduled for end o... [12:20:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 7529 (threshold c = 7500) [12:25:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 7666 (threshold c = 7500) [12:30:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 7714 (threshold c = 7500) [12:35:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 7820 (threshold c = 7500) [12:40:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 7816 (threshold c = 7500) [12:45:14] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 7896 (threshold c = 7500) [12:49:23] Fundraising-Backlog: Update email copy for the endowment Thank you email - https://phabricator.wikimedia.org/T228843 (Pcoombe) [12:49:27] Fundraising-Backlog, Patch-For-Review: Update Endowment automatic thank you email - https://phabricator.wikimedia.org/T232121 (Pcoombe) [12:50:15] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 7886 (threshold c = 7500) [12:51:19] Fundraising-Backlog, Patch-For-Review: Update Endowment automatic thank you email - https://phabricator.wikimedia.org/T232121 (Pcoombe) An update on this, it's in review but testing is blocked by a bug we found ({T232504}). I believe Elliott is working on fixing that this sprint. [12:55:15] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 8053 (threshold c = 7500) [13:00:15] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 8063 (threshold c = 7500) [13:05:19] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 8130 (threshold c = 7500) [13:08:54] ACKNOWLEDGEMENT - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 8130 (threshold c = 7500) Jeff_Green known, gmail throttling us [13:25:16] RECOVERY - check_mailq on civi1001 is OK: OK: postfix mailq (3567) is below threshold (5000/7500) [13:32:23] ok wow, the queue dropped 4k in 5 minutes [13:32:42] Jeff_Green: was that your work? [13:32:52] yes [13:32:55] writing it up now [13:54:31] Thanks Jeff! [13:58:59] is the campaign still up? [14:00:17] I just took the rest of the domains out of the 'fragile' list to reevaluate how they behave [14:08:34] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi: enable 'known fraudster' checkbox to also unsubscribe - https://phabricator.wikimedia.org/T232856 (MBeat33) @Jgreen in email thread //civi1001 mailq backup// 9/18: "The more we can do to stop attempting mail deliveries to bad addresses, the bett... [14:17:53] Jeff_Green: looks like it was an email sent to French donors [14:18:41] ejegg: you mean an email-only campaign? [14:20:08] let's see, schedule says France emails end this week, then France banners are up all October [14:20:25] There are currently banners up in Japan [14:21:46] Oh, there are emails going out to Israel too that end this week [14:22:31] ok [14:23:31] Jeff_Green: the Asana page they've started using in place of the Multi-Lingual Master google sheet is quite readable, if you can stand signing up for another account: https://app.asana.com/0/1125967519298615/timeline [14:24:41] ejegg: oh, nice! [14:35:48] ejegg: wow, you're right, this is totally readable [14:36:23] Jeff_Green: handy how you can toggle between the different views at the top there [14:36:56] yeah, this is great [14:43:26] (PS1) Ejegg: Thank you test fixes [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/537653 (https://phabricator.wikimedia.org/T232504) [14:45:57] Jeff_Green: I've got a meeting in 15 min with pcoombe and some other folks on the fr-creative team - the ones I've been asking about staging git access for. I can add you to the invite if you think it would be helpful to talk with them about options for how to test the TY mail templates. [14:49:25] fr-tech can anyone attend scrum of scrums today (in 40 min) ? [14:51:49] ejegg: can't really today... Just dealing with a difficult situation for my kids (a sudden death in their mom's family). E-mail forthcoming [14:52:05] oh shoot, sorry to hear that AndyRussG [14:52:46] ejegg: I've got too many things going on at once to tackle that on right now. [14:52:54] err grammar ^^^ [14:53:04] fr-tech I'll fill out the etherpad for the SoS meeting in any case - anything you'd like to bring to the attention of other teams? [14:53:11] ok Jeff_Green [14:53:23] ejegg jgleeson|away responded to your end of day about it, I can step in though if he ends up not being able to [14:53:31] ejegg: Thanks much. It is pretty tragic, though I think everyone's dealing with it in the best way. I'll tell folks about more later also [14:54:07] cstone oh shoot, I totally missed that. Thanks for pointing it out! [14:54:27] ejegg: If you fill out the SoS etherpad I might be able to just go briefly and say stuff and field questions with vagueries... [14:54:45] AndyRussG: see cstone's comment, I guess jgleeson|away already volunteered [14:55:00] and I'm just bad at email [14:55:31] ah, I was reading eileen's response to a different enderday and thought it was yesterday's [14:56:01] fundraising-tech-ops, Operations, ops-codfw: rack/setup/install frqueue2001 - https://phabricator.wikimedia.org/T232630 (Papaul) ` papaul@fasw-c-codfw# run show interfaces ge-[0-1]/0/18 descriptions Interface Admin Link Description ge-0/0/18 up up frqueue2001:eth0 ge-1/0/18 up... [14:56:32] fundraising-tech-ops, Operations, ops-codfw: rack/setup/install frqueue2001 - https://phabricator.wikimedia.org/T232630 (Papaul) [14:58:13] ejegg: cstone: ah ok great! Sorry I missed the reference, so I'm worse at e-mail... Thanks! [14:58:23] hey hey fr-tech! [15:00:40] hi jgleeson ! [15:00:41] hey jgleeson ! [15:04:29] jgleeson: hi! I'll try to respond to your CR of CNCM on the Gerrit task for now, hope that's OK! (See above and e-mail I just sent to fr-tech.) I'll also send up some patches [15:04:37] Thanks again for working on that! [15:06:19] AndyRussG: fully understandable. family first and sorry again for your bad news. [15:07:01] AndyRussG: the review stuff can wait seriously. [15:07:26] I think if mepps were here also she'd be saying the same thing [15:08:24] jgleeson: thanks much!! [15:10:42] cstone: also thanks for the heads-up ^ about the e-mail about SoS, and also willingness to step in... [15:16:17] (PS1) Ejegg: Disambiguate currency in TY email footer [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/537660 [15:17:52] well I wish I discovered that sooner. To update civirm via the cli you can issue `drush cvupdb` [15:18:05] civicrm* [15:18:28] no problem AndyRussG and I agree what jgleeson said above. Sorry to hear about the bad news [15:22:56] fr-tech, any news for scrum of scrums? [15:23:24] Fundraising-Backlog, Patch-For-Review: Update Endowment automatic thank you email - https://phabricator.wikimedia.org/T232121 (DStrine) @jrobell this bug is blocking @Pcoombe from previewing edits: T232504 Someone is working on it. We hope to have this unblocked soon. [15:23:59] dstrine: I'm about to review the patch blocking that ticket [15:24:13] it's a small change so will hopefully be a quick one to + [15:24:15] 2 [15:24:46] jgleeson: ok I was just making sure Jessica was watching the right task. She had been commenting on the duplicate [15:25:32] sure [15:27:04] dstrine: is this our final sprint before we down tools? [15:30:02] jgleeson: beyond this point you all should talk as a team about what you want to deploy. After October 1st we usually don't want to mess too much with ingencio or any common payment flows. [15:30:10] We are usually more comfortable with messing with other payment systems (like we could mess with dlocal a bit more) and there are side projects that could happen in civi that are not critical to the main donation pipeline. Also CentralNotice is a bit more open until November 1st [15:30:15] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 8058 (threshold c = 7500) [15:31:50] thanks for the clarification dstrine [15:33:16] pcoombe: OK, your endowment email update is on staging again [15:33:52] Great, thanks ejegg! Will test it shortly [15:34:54] pcoombe / ejegg where do you go to test email sending in civi? I was clicking around message templates but I can't find it [15:35:18] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 9242 (threshold c = 7500) [15:35:41] jgleeson: https://civicrm.frdev.wikimedia.org/admin/config/thank_you/test on staging, https://civicrm.wikimedia.org/admin/config/thank_you/test on prod [15:36:19] thanks pcoombe, I'll see if that loads on my local civicrm instance [15:37:26] Oh yeah, I could load that on my local but couldn't figure out how to actually send emails from it. Probably needs some extra setup [15:37:43] jgleeson: is there a way to make vagrant actually send email? [15:37:56] or do you just look at the files in the outgoing mail spool? [15:39:13] Fundraising Sprint Rocky Horror Presentation Layer, Fundraising Sprint Sysadmin Kane, Fundraising-Backlog, Patch-For-Review: 500 error when testing Endowment thank you emails in Civi - https://phabricator.wikimedia.org/T232504 (Ejegg) Patch is up on staging, and we are able to test the endowment... [15:40:16] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 10552 (threshold c = 7500) [15:45:16] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 11760 (threshold c = 7500) [15:50:16] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 13218 (threshold c = 7500) [15:50:25] ejegg: yeah I just use mail spool although I did spend some time trying to get https://mailcatcher.me/ working locally so they're easier to work with. Do you mean sending email out to the real world from a vagrant box? [15:54:42] (PS1) Ejegg: Update totals-earned chart for 2019 [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/537677 (https://phabricator.wikimedia.org/T232886) [15:55:16] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 14282 (threshold c = 7500) [15:55:43] jgleeson or at least to a local mailbox account on the host [15:55:54] those encoding are pretty gnarly [15:55:58] *encodings [15:56:13] also, thanks for taking scrum of scrums for me! [16:00:16] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 15722 (threshold c = 7500) [16:05:16] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 16741 (threshold c = 7500) [16:10:16] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 17369 (threshold c = 7500) [16:15:16] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 18580 (threshold c = 7500) [16:18:25] oh boy, there's the banner test donations [16:18:55] yeah, mail is being generated for sure. :) [16:20:16] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 20107 (threshold c = 7500) [16:25:16] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 21488 (threshold c = 7500) [16:26:34] Fundraising-Backlog, Patch-For-Review: Update Endowment automatic thank you email - https://phabricator.wikimedia.org/T232121 (Pcoombe) @kaythaney I just sent you a couple of test emails, one with a named donor and one anonymous. Please can you check these and confirm if they look correct? [16:30:06] ACKNOWLEDGEMENT - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 21488 (threshold c = 7500) Jeff_Green gmail is rate limiting our mail [16:30:11] Fundraising-Backlog: For the new data pipeline, purge existing log files - https://phabricator.wikimedia.org/T233237 (AndyRussG) [16:36:05] Fundraising Sprint Rocky Horror Presentation Layer, Fundraising Sprint Sysadmin Kane, Fundraising-Backlog: Turn off LandingPage events from testwiki - https://phabricator.wikimedia.org/T203020 (Urbanecm) Open→Resolved a:AndyRussG [16:36:07] Fundraising Sprint Asymmetrical Earth Theory, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising-Backlog, Epic: [Epic] Fundraising kafkatee changes - https://phabricator.wikimedia.org/T183978 (Urbanec... [16:46:06] (Abandoned) Ejegg: Disambiguate currency in TY email footer [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/537660 (owner: Ejegg) [16:54:23] Fundraising Sprint Sysadmin Kane, Fundraising-Backlog, FR-Paypal, FR-Q2-FY2019-20-cleanup-list: Paypal refunds missing predecessor - https://phabricator.wikimedia.org/T211847 (jgleeson) a:jgleeson [17:01:32] Fundraising-Backlog: For the new data pipeline, purge existing log files - https://phabricator.wikimedia.org/T233237 (AndyRussG) New data should be coming in fine, now, so we're ready for this any time. Thanks much!!! [17:35:51] Fundraising-Backlog, fundraising-tech-ops: frdev server and mysql and Civi access for jkumalah - https://phabricator.wikimedia.org/T232543 (DStrine) [17:36:15] Fundraising-Backlog, fundraising-tech-ops: frdev server and mysql and Civi access for jkumalah - https://phabricator.wikimedia.org/T232543 (DStrine) [17:36:46] Fundraising-Backlog, fundraising-tech-ops: frdev server and mysql and Civi access for jkumalah - https://phabricator.wikimedia.org/T232543 (DStrine) I just updated this task to include civi access. [17:39:04] (CR) Jgleeson: "recheck" [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/512354 (owner: Jgleeson) [17:39:37] (CR) jerkins-bot: [V: -1] WIP migration from bower to yarn for package management [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/512354 (owner: Jgleeson) [17:57:15] (PS2) Jgleeson: WIP migration from bower to yarn for package management [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/512354 [17:58:14] (CR) jerkins-bot: [V: -1] WIP migration from bower to yarn for package management [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/512354 (owner: Jgleeson) [18:49:34] fr-tech, I managed to get fundraising-dash running on yarn but it looks like a few things are failing when trying to read in stats [18:52:51] I'll pause this for now and see if I can finish review on the thank you test fixes [18:53:11] the rebased patch loaded for me and installed everything as expected... I think [18:56:21] argh, pcoombe and ejegg I can't see how I set the Financial Type using the https://civicrm.frdev.wikimedia.org/admin/config/thank_you/test page. Any ideas? [18:56:51] jgleeson we changed that yesterday you can change the template to endowment now [18:57:50] thanks cstone [18:59:22] hmmm looks like ejegg is right, I don't see the error locally that pcoombe reported when selecting the endowment_thank_you templte [19:00:08] Fundraising-Backlog, FR-Adyen, FR-Q2-FY2019-20-cleanup-list: Process Adyen IPN messages in parallel - https://phabricator.wikimedia.org/T233251 (Ejegg) [19:00:19] I'm gonna jump off now but I'll check irc later [19:21:49] Fundraising-Backlog: test - https://phabricator.wikimedia.org/T233255 (DStrine) [19:22:13] Fundraising-Backlog: test - https://phabricator.wikimedia.org/T233255 (DStrine) Open→Resolved [21:00:19] PROBLEM - check_puppetrun on civi1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:04:43] Fundraising-Backlog, Patch-For-Review: Update Endowment automatic thank you email - https://phabricator.wikimedia.org/T232121 (kaythaney) @Pcoombe confirmed! look good! [21:05:21] RECOVERY - check_puppetrun on civi1001 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [21:17:22] pcoombe: do you have +2 in the civi repo? If not I'm happy to +2 the email update as long as you're happy with it [21:17:43] ah, you're the submitter, so I guess it's good form to have someone else +2 in any case. [21:38:21] Fundraising Sprint Sysadmin Kane, Fundraising-Backlog, FR-Q2-FY2019-20-cleanup-list, Recurring-Donations, FR-Email: Annual totals receipt thank you email for recurring donors - https://phabricator.wikimedia.org/T195907 (Ejegg) I just reviewed the [[https://github.com/jake-mw/CDNTaxReceipts/bl... [22:08:59] fr-tech is anyone else having trouble connecting to github via ssh? [22:10:24] ah, it was using a 'git://' url [22:10:36] an ssh://git@github.com url actually works fine [23:41:48] (PS1) XenoRyet: WIP No Card Selector Ingenico Variant [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/537775 [23:46:04] (CR) jerkins-bot: [V: -1] WIP No Card Selector Ingenico Variant [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/537775 (owner: XenoRyet)