[00:09:14] (PS1) Ejegg: WIP send opt-in message on donation failure [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/495154 (https://phabricator.wikimedia.org/T216293) [00:13:27] (CR) jerkins-bot: [V: -1] WIP send opt-in message on donation failure [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/495154 (https://phabricator.wikimedia.org/T216293) (owner: Ejegg) [00:22:06] Fundraising Sprint Da Vinci Coder, Fundraising Sprint Ewoks Take Manhattan, Fundraising-Backlog, Fr-Ingenico-integration_2017-18: Ingenico: audit lagging on manual settled transactions? - https://phabricator.wikimedia.org/T217582 (MBeat33) Thanks, @Ejegg I can just factor in a bit more of a delay... [00:32:34] Fundraising Sprint Ewoks Take Manhattan, Fundraising-Backlog, FR-Smashpig, Fr-Ingenico-integration_2017-18, Patch-For-Review: Error: POSTing empty data to hostedcheckouts endpoint - https://phabricator.wikimedia.org/T202417 (Ejegg) p:Triage→Normal a:Ejegg [00:33:32] (PS1) Ejegg: Stop mangling donor data on truncation [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/495155 (https://phabricator.wikimedia.org/T202417) [01:20:38] (CR) Ejegg: [C: -1] "Thanks! For the class_exists tests, we need to keep the argument a string, so we don't get errors when the corresponding php extensions ar" (3 comments) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/493965 (owner: Umherirrender) [01:25:16] (CR) Ejegg: [C: +1] "Thanks for all the cleanup! Looks good at first glance, but I should test a bit before +2ing" [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/493229 (owner: Esanders) [01:26:49] (CR) Ejegg: [C: +2] Apply drupal formatting to offline2civicrm.install [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/494840 (owner: Eileen) [01:31:05] (Merged) jenkins-bot: Apply drupal formatting to offline2civicrm.install [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/494840 (owner: Eileen) [01:49:13] (CR) Reedy: [C: +1] "No they don't, ::class has magical error suppressing properties" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/493965 (owner: Umherirrender) [02:14:58] (CR) MaxSem: [C: +1] Use ::class for class name resolution (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/493965 (owner: Umherirrender) [02:53:38] (CR) Ejegg: "Oh cool, thanks for the info MaxSem! downvote retracted." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/493965 (owner: Umherirrender) [13:48:24] PROBLEM - check_puppetrun on mintaka is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:51:28] PROBLEM - check_puppetrun on alnilam is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 5 minutes ago with 9 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_impression_logs],File[/usr/lib/nagios/plugins/check_strongswan],File[/usr/lib/nagios/plugins/check_timedatectl],File[/usr/lib/nagios/plugins/check_frtech_mail] [13:51:28] PROBLEM - check_puppetrun on frbackup2001 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 5 minutes ago with 9 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_impression_logs],File[/usr/lib/nagios/plugins/check_strongswan],File[/usr/lib/nagios/plugins/check_timedatectl],File[/usr/lib/nagios/plugins/check_frtech_mail] [13:51:28] PROBLEM - check_puppetrun on payments1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:51:29] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 5 minutes ago with 8 failures. Failed resources (up to 3 shown): File[/usr/local/bin/aide_new_db],File[/etc/apparmor.d/local/usr.bin.freshclam],File[/usr/lib/nagios/plugins/check_impression_logs],File[/usr/lib/nagios/plugins/check_strongswan] [13:52:34] PROBLEM - check_puppetrun on frdb1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:53:24] RECOVERY - check_puppetrun on mintaka is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [13:53:34] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 7 minutes ago with 15 failures. Failed resources (up to 3 shown): Service[mcollective],File[/home/jgreen/.vimrc],File[/home/cwdent/.vimrc],File[/etc/vim/vimrc.local] [13:55:28] PROBLEM - check_puppetrun on alnitak is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:55:36] PROBLEM - check_puppetrun on payments2002 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 9 minutes ago with 11 failures. Failed resources (up to 3 shown): File[/usr/local/bin/aide_new_db],File[/etc/apparmor.d/local/usr.bin.freshclam],File[/usr/lib/nagios/plugins/check_impression_logs],File[/usr/lib/nagios/plugins/check_strongswan] [13:55:36] PROBLEM - check_puppetrun on payments2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:56:24] RECOVERY - check_puppetrun on frbackup2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [13:56:26] RECOVERY - check_puppetrun on alnilam is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [13:56:26] RECOVERY - check_puppetrun on frpig2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:56:28] RECOVERY - check_puppetrun on payments1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:57:34] RECOVERY - check_puppetrun on frdb1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [13:58:34] RECOVERY - check_puppetrun on americium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:00:28] PROBLEM - check_puppetrun on alnitak is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [14:00:38] RECOVERY - check_puppetrun on payments2002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [14:00:44] RECOVERY - check_puppetrun on payments2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:05:30] PROBLEM - check_puppetrun on alnitak is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [14:10:28] RECOVERY - check_puppetrun on alnitak is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [15:07:24] PROBLEM - check_puppetrun on frdb2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:12:24] PROBLEM - check_puppetrun on frdb2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:17:24] PROBLEM - check_puppetrun on frdb2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:22:24] PROBLEM - check_puppetrun on frdb2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:27:22] cwd / Jeff_Green can you copy the emails in the civi inbox to a temp folder please? [15:27:35] the bounce processor seems to be repeatedly tripping over something [15:27:43] sure [15:27:58] thanks! [15:28:04] That'll be a great help in diagnosis [15:37:07] this mailbox is enormous [15:37:25] does civi have a delete-processed-after-some-timeframe option? [15:39:08] Jeff_Green: ohh, I thought it was deleting them as it processed them! [15:39:24] Maybe it's enormous because the processing job keeps crashing? [15:39:26] it seems to be moving them to a 'processed' maildir [15:39:34] ah, go tit [15:39:37] *got it [15:39:44] so, probably not [15:39:56] but I can't imagine any reason to keep the processed ones [15:40:01] I'm not positive, i'm just seeing the rsync output which is going on forever [15:40:05] lemme take a peep at the code [15:40:24] i could see keeping them for 30 days or something, in case we find an issue with the parser? [15:43:58] yeah, I don't see anything in the Civi code to clear that dir [15:44:22] could it be the reason it's blowing up? [15:44:36] no, it looks like it's having trouble parsing a specific mail [15:44:45] ok [15:44:50] it doesn't seem to read the processed folder ever [15:47:18] RECOVERY - check_puppetrun on frdb2001 is OK: OK: this test is currently disabled [15:57:32] Jeff_Green: with an mbox folder, can you get away with just deleting files in 'processed' using something like tmpreaper? [15:57:51] or does something need to update metadata to keep things consistent? [15:58:22] I'm not sure, I haven't dug that deep into maildir before [15:58:25] lemme see... [16:05:13] it sounds as though dovecot will tolearate that, but it would probably be safer/cleaner to use imap commands to do it from php [16:07:06] hmm, ok [16:07:18] so that would be a new mailing job [16:07:33] a process_control thing? [16:07:43] looks like the imap library they use has delete functions [16:07:58] Jeff_Green: yeah, I think we would build it as a standard CiviCRM job [16:08:02] sounds good [16:08:12] can you purge the 'ignored' folder at the same time? [16:08:18] so that people could run it through the UI, or through a drush command (that can be called by process-control) [16:08:25] sounds good to me [16:08:25] Jeff_Green: sure, both folders [16:09:10] just gonna take a good look through settings and docs to make sure there's no existing thing I missed trawling code [16:09:48] cool [16:10:13] oh I forgot to mention--there's a copy of the maildir in your homedir for debugging [16:11:05] thanks! [16:12:16] Jeff_Green: oops, getting permission denied when I try to cd into it [16:12:25] oh crud, i forgot to chown it [16:12:26] sec [16:12:54] fixed [16:13:28] oh yeah, that processed folder is huge [16:13:41] well, I guess I don't need the processed folder for this investigation [16:13:46] deleting my copy [16:14:30] oh hmm [16:14:54] so... now, tmp, and cur are all empty [16:15:29] ok, what exactly is the EmailProcessor doing with the latest email when it crashes? [16:58:40] Jeff_Green: So, maybe the error isn't with a specific message at all! [16:59:02] what's your theory? [16:59:12] I just noticed that these error messages have this line: Could not read from the stream [16:59:33] huh [16:59:41] which seems to be thrown only when the connection with the imap server drops or returns something funky [17:00:00] I check if dovecot is logging anything interesting [17:00:15] thanks! [17:00:35] there might be something at 15:10:03 or 15:10:08 [17:06:14] there's a connection at 15:10 but no indication that it was abnormal [17:07:06] Jeff_Green: ok, interesting [17:07:21] the job started at 15:10:10 [17:07:37] could we be hitting a limit within the php imap library? [17:07:42] and recorded one error at 15:10:03 'Trying to get property of non-object EmailProcessor.php:276' [17:08:05] then recorded the 'An error occured while sending or receiving mail. Could not read from the stream. It was probably terminated by the host.' at 15:10:08 [17:08:13] so it didn't take very much time for that to happen [17:08:25] dovecot isn't logging a ton, it's just showing connect/disconnect [17:08:38] I saw a couple of connection failures, but not near that time [17:09:11] I guess we could try emptying the processed / ignored mailboxes and see if that speeds the connection up? [17:09:46] yeah [17:09:51] can we do that manually via some imap client in the absence of a civi job? [17:10:01] just to see if that stops the bounce failmail [17:10:10] Fundraising-Backlog, Fr-CiviCRM-dedupe-FY2017/18: Civi dedupe: merge screen upgrades - https://phabricator.wikimedia.org/T217903 (MBeat33) [17:10:16] if it does, that'd be a good reason to prioritize creating that job [17:10:29] sec, lemme see if I can get alpine talking to dovecot there [17:13:59] there are 176K messages in that maildir :-P [17:14:27] wheee [17:14:31] can I delete everything before say 2/1? [17:14:37] be my guest! [17:14:59] i guess if dovecot has to read them all on connection, that could lead to a timeout [17:15:05] go alpine go [17:15:24] it could, although I've used dovecot/imap at this kind of volume successfully [17:16:05] maybe the php library has a (probably-reasonable) timeout [17:21:54] Fundraising Sprint Ewoks Take Manhattan, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Update opt-in queue consumer to be able to create new donor records - https://phabricator.wikimedia.org/T217710 (Cstone) a:Cstone [17:23:11] go pine go [17:30:04] ejegg: ok that leaves 7K messages in 'processed' [17:30:19] thanks! [17:30:24] hope that does the trick [17:30:39] I did the same for 'ignored' [17:31:21] fwiw alpine took forever to tag and purge all those, so I think if it's a timeout it would be on the php side [17:36:08] Fundraising-Backlog, Fr-CiviCRM-dedupe-FY2017/18: Civi dedupe: URL generator for dedupe searches - https://phabricator.wikimedia.org/T217909 (MBeat33) [18:45:28] (PS3) Umherirrender: Use ::class for class name resolution [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/493965 [18:45:38] (PS4) Umherirrender: Use ::class for class name resolution [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/493965 [18:45:49] (CR) Umherirrender: Use ::class for class name resolution (6 comments) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/493965 (owner: Umherirrender) [18:53:56] (CR) D3r1ck01: [C: +2] Use ::class for class name resolution [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/493965 (owner: Umherirrender) [18:55:37] (Merged) jenkins-bot: Use ::class for class name resolution [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/493965 (owner: Umherirrender) [19:02:55] (PS2) Umherirrender: Avoid local var in CampaignLog [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/492513 [20:09:48] cstone and XenoRyet have you had a chance to pair up? Anything I can help with on that ticket? [20:10:16] not yet going to eat luch now but I can when I get back [21:04:53] (PS1) Umherirrender: Remove unused use statements [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/495317 [21:54:07] (CR) jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/495345 (owner: L10n-bot) [21:58:11] Fundraising Sprint Ewoks Take Manhattan, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: New Citibank import - https://phabricator.wikimedia.org/T217390 (Ejegg) Hi @LeanneS - it looks like we should be using the Bank Reference to detect duplicates. Can you check to see if th... [22:34:19] (CR) D3r1ck01: Remove unused use statements (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/495317 (owner: Umherirrender) [22:38:56] Fundraising Sprint Ewoks Take Manhattan, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: New Citibank import - https://phabricator.wikimedia.org/T217390 (Ejegg) @LeanneS, I see the problem with September's sheet now - there's a column with no header, duplicating the amounts,... [22:39:13] (CR) Ejegg: [C: +2] Add Citibank import for individuals [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/494851 (https://phabricator.wikimedia.org/T217390) (owner: Eileen) [22:39:34] (PS2) Umherirrender: Remove unused use statements [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/495317 [22:39:38] (CR) Umherirrender: Remove unused use statements (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/495317 (owner: Umherirrender) [22:41:41] (CR) D3r1ck01: Remove unused use statements (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/495317 (owner: Umherirrender) [22:44:31] (Merged) jenkins-bot: Add Citibank import for individuals [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/494851 (https://phabricator.wikimedia.org/T217390) (owner: Eileen) [23:17:03] Fundraising Sprint Ewoks Take Manhattan, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: New Citibank import - https://phabricator.wikimedia.org/T217390 (LeanneS) @ejegg Oh interesting! It Looks like Finance added that in by accident. I'll test again. [23:44:01] Fundraising Sprint Ewoks Take Manhattan, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: New Citibank import - https://phabricator.wikimedia.org/T217390 (LeanneS) Okay the latest test was a success!