[01:26:12] (Abandoned) Eileen: WIP ptp reeports [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/517916 (owner: Eileen) [01:32:18] Fundraising Sprint Office  , Fundraising Sprint Princess Mongodb, Fundraising Sprint Quick and the Deadlocked, Fundraising-Backlog, and 2 others: Prepare next civi upgrade - https://phabricator.wikimedia.org/T228826 (Eileenmcnaughton) I've resolved test fails & recut it - still to do 1) cre... [01:34:09] Fundraising Sprint Never Ending Query, Fundraising Sprint Quick and the Deadlocked, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Nurse love & iterate on our targetsmart batch job - https://phabricator.wikimedia.org/T228715 (Eileenmcnaughton) We have been hitting rows that require manual in... [01:34:33] PROBLEM - check_procs on civi1001 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [01:48:12] I think the targetsmart import is creating a lot of smarty cruft in templates_c - those fail mails were me clearing caches [01:48:19] otoh it's a temporary issue [01:50:15] RECOVERY - check_procs on civi1001 is OK: PROCS OK: 424 processes [02:01:26] Fundraising Sprint Never Ending Query, Fundraising Sprint Quick and the Deadlocked, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Nurse love & iterate on our targetsmart batch job - https://phabricator.wikimedia.org/T228715 (Eileenmcnaughton) Here is a list of the files we are currentl... [02:02:30] Fundraising Sprint Never Ending Query, Fundraising Sprint Quick and the Deadlocked, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Nurse love & iterate on our targetsmart batch job - https://phabricator.wikimedia.org/T228715 (Eileenmcnaughton) Also note I'm seeing smarty template cruft clog... [02:29:53] looks like timeouts / deadlocks? [02:30:43] oh, or clearing the caches did that? [03:28:53] (PS1) Eileen: Update afform [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/532842 (https://phabricator.wikimedia.org/T228826) [03:28:55] (PS1) Eileen: Update contactlayouteditor in tandem [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/532843 (https://phabricator.wikimedia.org/T228826) [03:28:57] (PS1) Eileen: Update deduper for 5.17 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/532844 [03:31:03] (CR) jerkins-bot: [V: -1] Update deduper for 5.17 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/532844 (owner: Eileen) [03:41:11] (PS2) Eileen: Update deduper for 5.17 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/532844 (https://phabricator.wikimedia.org/T228826) [03:41:13] (PS1) Eileen: Add extension to manage our query modifications [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/532845 (https://phabricator.wikimedia.org/T228826) [03:42:36] (CR) jerkins-bot: [V: -1] Add extension to manage our query modifications [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/532845 (https://phabricator.wikimedia.org/T228826) (owner: Eileen) [03:43:24] (CR) jerkins-bot: [V: -1] Update deduper for 5.17 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/532844 (https://phabricator.wikimedia.org/T228826) (owner: Eileen) [03:45:32] (PS2) Eileen: Add extension to manage our query modifications [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/532845 (https://phabricator.wikimedia.org/T228826) [04:43:13] (PS3) Eileen: Add extension to manage our query modifications [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/532845 (https://phabricator.wikimedia.org/T228826) [05:00:04] fundraising-tech-ops: puppetize and configure fruec banner log parser/db injector script - https://phabricator.wikimedia.org/T231249 (AndyRussG) > frdeploy configuration is puppetized and tested, needs to be adjusted to 'deployment' branch once that branch exists Cool! K, I created that branch! :) > Does t... [05:52:18] (PS4) Eileen: Add extension to manage our query modifications [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/532845 (https://phabricator.wikimedia.org/T228826) [05:52:20] (PS3) Eileen: Update deduper for 5.17 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/532844 (https://phabricator.wikimedia.org/T228826) [06:04:25] Wikimedia-Fundraising-Banners, User-DannyS712: Special:HideBanners renders as a blank page - https://phabricator.wikimedia.org/T230571 (AndyRussG) Open→Invalid [06:08:28] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Epic: Client-side event impression event logging is broken - https://phabricator.wikimedia.org/T231404 (AndyRussG) [06:10:13] Wikimedia-Fundraising-Banners, User-DannyS712: Special:HideBanners renders as a blank page - https://phabricator.wikimedia.org/T230571 (DannyS712) >>! In T230571#5442842, @Ejegg wrote: > Oh yeah, just going off on a tangent. This bug report describes the intended behavior of the special page. I'm sorry,... [06:14:38] Wikimedia-Fundraising-Banners, User-DannyS712: Special:HideBanners renders as a blank page - https://phabricator.wikimedia.org/T230571 (AndyRussG) >>! In T230571#5444537, @DannyS712 wrote: >>>! In T230571#5442842, @Ejegg wrote: >> Oh yeah, just going off on a tangent. This bug report describes the intend... [06:20:17] Fundraising-Backlog, MediaWiki-extensions-FundraiserLandingPage, Epic: Landing page event logging events have -1 as schema revision number. - https://phabricator.wikimedia.org/T231406 (AndyRussG) [06:30:11] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Performance-Team (Radar), Spike: Spike: Investigate alternatives to Special:HideBanners cookie storm for cross-domain banner close-button - https://phabricator.wikimedia.org/T117433 (DannyS712) [06:33:01] (PS1) AndyRussG: Fix location of eventlogging schema revision in extension.json [extensions/FundraiserLandingPage] - https://gerrit.wikimedia.org/r/532858 (https://phabricator.wikimedia.org/T231406) [06:34:15] Fundraising Sprint Quick and the Deadlocked, Fundraising-Backlog, MediaWiki-extensions-FundraiserLandingPage, Epic, Patch-For-Review: Landing page event logging events have -1 as schema revision number. - https://phabricator.wikimedia.org/T231406 (AndyRussG) [06:34:34] Fundraising Sprint Quick and the Deadlocked, Fundraising-Backlog, MediaWiki-extensions-FundraiserLandingPage, Epic, Patch-For-Review: Landing page event logging events have -1 as schema revision number. - https://phabricator.wikimedia.org/T231406 (AndyRussG) a:AndyRussG [06:34:49] Fundraising Sprint Quick and the Deadlocked, Fundraising-Backlog, MediaWiki-extensions-FundraiserLandingPage, Epic, Patch-For-Review: Landing page event logging events have -1 as schema revision number. - https://phabricator.wikimedia.org/T231406 (AndyRussG) [09:13:40] (CR) jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/532905 (owner: L10n-bot) [13:17:28] (CR) Ejegg: [C: +2] "You got your v2 attributes in my v1 json! Nice catch, AndyRussG" [extensions/FundraiserLandingPage] - https://gerrit.wikimedia.org/r/532858 (https://phabricator.wikimedia.org/T231406) (owner: AndyRussG) [13:39:46] Fundraising-Backlog, FR-dlocal: dLocal: on donation form, show Netbanking / UPI choice sooner in flow? - https://phabricator.wikimedia.org/T231452 (MBeat33) [13:43:29] Fundraising-Backlog, FR-dlocal: dLocal: on donation form, show Netbanking / UPI choice sooner in flow? - https://phabricator.wikimedia.org/T231452 (MBeat33) [13:44:17] Fundraising-Backlog, FR-dlocal: dLocal: on donation form, show Netbanking / UPI choice sooner in flow? - https://phabricator.wikimedia.org/T231452 (EMartin) Thanks [13:48:50] (Merged) jenkins-bot: Fix location of eventlogging schema revision in extension.json [extensions/FundraiserLandingPage] - https://gerrit.wikimedia.org/r/532858 (https://phabricator.wikimedia.org/T231406) (owner: AndyRussG) [14:08:59] ejegg, I ran into an issue testing out the python3 patch. I couldn't get the paypal audit download_nightly script to see the process module. I tried adding it to my PYTHONPATH before running the script but that never worked. Did it work for you without tinkering your python module sys path? [14:10:53] jgleeson I always run the scripts from the root of the tools dir with e.g. PYTHONPATH=`pwd` python3 FundraiserStatisticsGen/methodgen.py -c FundraiserStatisticsGen/fundstatgen.cfg temp [14:11:27] lemme check on that download_nightly then... [14:12:42] yeah, this command works for me: PYTHONPATH=`pwd` python3 audit/paypal/download_nightly [14:13:24] ok thanks I'll try that [14:19:06] ejegg, so that worked! although it's now fallen over looking for a logging config value [14:19:14] should that be a path to a file? [14:19:17] logging.config.dictConfig(config.logging) [14:19:19] ^ [14:19:27] oh right, lemme see [14:19:48] in my yaml config I don't have a key for logging [14:20:10] ahh, and it's missing from the example [14:20:43] yeah, I've got a big chunk in there, inherited from ??? [14:21:32] hmm maybe a `diff -y your-config.yaml example.yaml` out would be good for me to see? [14:21:48] then we could update the example to have all the things it might need [14:22:24] https://phabricator.wikimedia.org/P8994 [14:22:35] here's the logging chunk, anyway ^^^ [14:23:00] thanks [14:23:13] it's like prod, but minus syslog [14:23:38] I'm getting deja vu here [14:23:44] feel like we've run into this before [14:30:33] ok confirmed the download_nightly script is working [14:32:03] I tried to be clever using the vagrant vm as my sftp server using vagrants insecure keys as the fundraising-tools sftp client module looked like it could handle private keys but I was getting 'Oops, unhandled type 3 ('unimplemented')' when using it [14:32:34] in the end i just enabled passwordauth on sshd on the same box and gave it the vagrant/vagrant credentials and it worked [14:32:35] Wikimedia-Fundraising-Banners: Check UX of desktop fundraising banners on iPad - https://phabricator.wikimedia.org/T231325 (EWilfong_WMF) 44 x 44px is the recommended target size for inputs according to the [[ https://www.w3.org/WAI/WCAG21/Understanding/target-size | W3 WCAG ]]. It's going to be tough to ge... [14:32:48] great! [14:33:18] I'll just try out the parse and then I think I can +2 [14:35:28] :) :) [14:44:20] ejegg, the parse_nightly script appears to run but the output is mostly 'Could not parse filename: file-name*.csv [14:45:00] oh hmm, is it just pointing at an empty directory? [14:45:03] I wasn't expecting it to do anything as it's the test fixtures being compared to my local civicrm [14:45:26] no it looks like it can see the right files [14:45:41] (that I moved to a dir to be read via the download_nightly) [14:46:09] archive_path: /home/vagrant/processed [14:46:16] I see the files downloaded in here [14:46:17] ^ [14:46:21] in there* [14:46:50] Archiving /home/vagrant/incoming/classic_subscr_signup.csv to /home/vagrant/processed/classic_subscr_signup.csv [14:48:21] and it's not parsing those? [14:48:22] ah it looks like there is a filename pattern match [14:48:28] within parse_nightly [14:48:29] ah right [14:49:35] ah I could do with a real filename to see lol [14:50:02] looks like type-data.sequence.version.CSV [14:50:15] s/data/date/ [14:50:36] right, that's the filename pattern as provided by paypal [14:50:54] I guess I'm confused why the unit tests are passing [14:51:01] if these files don't meet that convention [14:51:24] ah maybe the tests don't cover/use parse [14:52:36] yeah it looks like the tests skip that level [14:52:42] and use the type handlers directly [14:52:46] fr-tech: any news for scrum of scrums? [14:52:52] jgleeson: ah right [14:59:03] ejegg: hey thanks, nothing here! [15:02:01] it looks like the test files can't be read by the unicode_csv_reader package as they're not utf16 encoded ?!?! [15:04:24] hmmm [15:05:01] it looks like the files are utf8 and for some reason the script is trying to parse then using UTF16 [15:07:21] I wonder if a paypal TRR files are actually using utf16 and our test files are utf8 [15:08:22] read_encoded(path, version, callback, column_headers, encoding='utf-16') [15:08:26] looks like that is the case [15:09:16] wonder if I can use iconv to convert the test file [15:12:55] Fundraising-Backlog, Wikimedia-Fundraising-Banners, FR-dlocal: dLocal: on donation form, show Netbanking / UPI choice sooner in flow? - https://phabricator.wikimedia.org/T231452 (Pcoombe) We could add the buttons to banners quite easily. However it would need fr-tech work if we want to then remove th... [15:13:37] still doesn't like it [15:16:52] Fundraising-Backlog, Wikimedia-Fundraising-Banners, FR-dlocal: dLocal: on donation form, show Netbanking / UPI choice sooner in flow? - https://phabricator.wikimedia.org/T231452 (spatton) Thanks for sharing this feedback from Rakhi, @MBeat33! I just want to endorse the idea ... we have generally seen... [15:20:07] ejegg, I think there are some encoding differences between python2 and 3 breaking your changes on the paypal audit [15:20:10] or at least for me [15:20:35] running parse_nightly on python2 works for me but on python3 it fails due to 'iterator should return strings, not bytes (did you open the file in text mode?) [15:20:35] ' [15:21:09] the code which handles this looks like it uses some of the python2 hacks for dealing with different character encodings [15:21:49] specifically this https://github.com/wikimedia/wikimedia-fundraising-tools/blob/master/audit/paypal/unicode_csv_reader.py [15:22:34] I'll do the usual thing and add feedback on the patch [15:22:58] I've got to head out now though so I'll do it when I get back just before standup [15:30:40] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Client-side event impression event logging is broken - https://phabricator.wikimedia.org/T231404 (AndyRussG) [15:31:03] Fundraising Sprint Quick and the Deadlocked, Fundraising-Backlog, MediaWiki-extensions-FundraiserLandingPage, MW-1.34-notes (1.34.0-wmf.21; 2019-09-03): Landing page event logging events have -1 as schema revision number. - https://phabricator.wikimedia.org/T231406 (AndyRussG) [15:32:09] (CR) Krinkle: "Oops, sorry :)" [extensions/FundraiserLandingPage] - https://gerrit.wikimedia.org/r/532858 (https://phabricator.wikimedia.org/T231406) (owner: AndyRussG) [15:34:48] jgleeson|away: thanks! I'll do more testing locally [15:39:58] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Minimal update Allocations UI for campaign fallback - https://phabricator.wikimedia.org/T231469 (AndyRussG) [16:08:57] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Move DonationInterface messages used in banners to WikimediaMessages - https://phabricator.wikimedia.org/T231474 (Ejegg) [16:26:57] (CR) Jgleeson: [C: -1] "download_nightly.py worked once I added in the logging: block to the config." [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520350 (https://phabricator.wikimedia.org/T197805) (owner: Ejegg) [16:31:01] (PS1) Jgleeson: Add logging: block to yaml.example config files [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/533063 [16:32:50] (CR) Ejegg: [C: +2] "Thanks!" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/533063 (owner: Jgleeson) [16:45:01] (Merged) jenkins-bot: Add logging: block to yaml.example config files [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/533063 (owner: Jgleeson) [17:28:30] argh my PoE isn't working, brb! [17:34:15] sorry fr-tech, hangouts isn't loading for me and I think it's due to my internet issues [17:34:25] gmail is but hangouts just spinning :/ [18:02:03] (PS1) Jgleeson: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/533068 [18:05:13] (CR) Jgleeson: [V: +2 C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/533068 (owner: Jgleeson) [18:35:24] (PS1) Jgleeson: Remove test Unsubscribe consumer files that sneaked through. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/533073 [18:36:54] (CR) Jgleeson: [V: +2 C: +2] Remove test Unsubscribe consumer files that sneaked through. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/533073 (owner: Jgleeson) [20:05:27] (PS1) Jgleeson: Added in batch size drupal variables for antifraud and paymentsinit queue comnsumers [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/533086 [20:09:16] ejegg, are we +2'ing all civi patches or just those on deployment? [20:10:26] woah [20:10:32] looks like there was a preexisting bug [20:12:55] (PS2) Jgleeson: Added in batch size drupal variables for antifraud and paymentsinit queue comnsumers [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/533086 [20:13:28] just the deploy branch [20:15:01] it looks like there is a huge queue on zuul [20:15:10] am I reading that correctly? https://integration.wikimedia.org/zuul/ [20:21:40] fr-tech, looks like zuul is down [20:22:01] oh man [20:22:15] let's see if they're on the case in -releng [20:22:26] ejegg, I asked a little earlier [20:22:37] looks like they were unaware [20:22:52] ah, yeah, just reading backscroll. fun [20:22:59] so.... lemme take a look at that patch [20:24:18] also, it looks like the redis failmail is related to us switching off the consumers [20:30:03] (CR) Ejegg: [C: +2] Added in batch size drupal variables for antifraud and paymentsinit queue comnsumers [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/533086 (owner: Jgleeson) [20:30:13] jgleeson: i'd be happy to V+2 that one too [20:30:28] since any errors will be pretty obvious [20:30:38] (CR) Ejegg: [V: +2 C: +2] Added in batch size drupal variables for antifraud and paymentsinit queue comnsumers [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/533086 (owner: Jgleeson) [20:31:11] ok cool [20:31:16] I'll deploy those changes [20:31:33] sorry, I mean merge to deployment first [20:31:35] and then deploy [20:31:36] :) [20:32:36] (PS1) Jgleeson: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/533091 [20:32:57] (CR) Jgleeson: [V: +2 C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/533091 (owner: Jgleeson) [20:34:44] ok that's deployed [20:35:23] great jgleeson! [20:35:45] oh sorry mepps, the patch to make testing the main patch, is deployed :) [20:35:57] easier* [20:36:12] still great jgleeson ;), i was just cheering for a deployment in general [20:36:19] ha [20:39:45] ejegg, so we stil can't see the optin module admin page in drupal but there are no messages in that queue at current so even if we could, it would be not use to us for testing [20:40:02] maybe we could leave the opt-in queue job off for now until we get a message to then drip feed test it [20:40:16] (once drush en works) [20:40:52] we current have antifraud, paymentsinit and unsubscribe consumers set to a batch size of 1 [20:41:14] jgleeson: we can send a test message through outselves! [20:41:32] hmm, looking into why that config page is missing though [20:41:55] yeah I guess I could just use the the test fixtures [20:43:40] oh right, it's that 'access arguments' line [20:43:47] ahhh [20:43:50] I forgot about that [20:43:53] I'll fix and repush [20:44:00] k [20:44:06] https://etherpad.wikimedia.org/p/fr-tech-queue-consumers-release [20:44:10] although I did note it down [20:44:15] I just didnt read my notes... [20:44:26] story of my life [20:45:38] heh [20:47:24] (PS1) Jgleeson: Fixed access arguments value for optin module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/533097 [20:48:33] fr-tech, could someone +2 that patch? zuul is blocked up at current so patches need manually pushing through [20:49:04] please * [20:50:28] (CR) Mepps: [C: +2] Fixed access arguments value for optin module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/533097 (owner: Jgleeson) [20:50:36] thanks mepps! [20:51:30] (CR) Ejegg: [V: +2] Fixed access arguments value for optin module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/533097 (owner: Jgleeson) [20:56:49] (PS1) Jgleeson: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/533098 [20:57:06] (CR) Jgleeson: [V: +2 C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/533098 (owner: Jgleeson) [20:58:45] jgleeson: oh shoot, why are we still getting multiqueue consumer failmail? [20:59:00] I thought you'd turned that off [20:59:47] I wasn't aware the jobs we turned off were related to those failmails [21:00:02] we switched off the queue_consumer jobs? [21:00:04] Fundraising Sprint Never Ending Query, Fundraising Sprint Quick and the Deadlocked, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Nurse love & iterate on our targetsmart batch job - https://phabricator.wikimedia.org/T228715 (Eileenmcnaughton) Update on where the various jobs are at... [21:01:43] ahh, you created new jobs for the p-i and antifraud queues [21:01:51] yes [21:01:53] but you left the existing one running. deleting the existing one [21:02:47] ok, that should fix it [21:03:03] !log deleted fredge_multiqueue_consumer process-control job [21:03:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:03:17] oh I see! [21:03:23] I left fredge alive and well [21:05:39] ok civi deployed, clearing cache [21:06:59] ok I see the optin configuration menu link! [21:07:13] batch size set to 1, we're good to go [21:09:14] great [21:22:01] tested antifraud and paymentsinit both working as expected [21:23:08] (PS1) Ejegg: Update yaml lib [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/533103 [21:23:15] (CR) Ejegg: [C: +2] Update yaml lib [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/533103 (owner: Ejegg) [21:23:21] woohoo jgleeson ! [21:23:36] +1 to the yaml update ejegg, that might help me pull in rector [21:24:18] (Merged) jenkins-bot: Update yaml lib [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/533103 (owner: Ejegg) [21:29:42] hmm opt-in is a little tricker to confirm [21:29:53] unsubscibed worked! [21:30:01] looks like we have a custom field for opt-in [21:30:13] yes [21:30:26] want to send a test message for your own contact? [21:30:28] so I can drop a message on the queue to opt myself back in [21:30:35] after I unsubscribe [21:30:48] https://payments.wikimedia.org/index.php?title=Special:EmailPreferences/optin&variant=wle_001 [21:31:37] great, so we can turn the batch size back to unlimited for the ones you've tested [21:32:30] they all work! [21:32:35] phew [21:32:41] that was longwinded! [21:32:47] wonderful. Thanks for sticking around so late! [21:32:58] I also need to turn targetsmart back on [21:33:06] I'll do them separately [21:33:11] ok, cool [21:33:14] consumers/targetsmart [21:33:26] targetsmart2 is failing mysteriously [21:33:46] so you can maybe leave that one off [21:33:51] sure [21:33:53] brb! [21:34:38] jgleeson: if you need to head out I can turn stuff back on [21:35:59] ohh shoot, actually I gotta run too [21:38:43] no it's cool [21:44:34] eileen, just a heads up, the targetsmart jobs have been temporarily switched off [21:44:59] we've been deploying some changes to the queue consumers and the load on civi1001 was high enough to slow things down [21:45:01] jgleeson: ah ok [21:45:16] I'll put them back on in a minute or two [21:45:20] great [21:47:13] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 2014 2000 - REDIS 3.2.6 on 127.0.0.1:6379 has 1 databases (db0) with 6 keys, up 40 days 20 hours - memory use is 3.34M (peak 8.09M, 0.08% of max, fragmentation 1.94%), connected_slaves is 2, donations is 0, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 638, pending is 0, recurring is 0, refund is 0, unsubscribe is 6 [21:53:58] fr-tech, those icinga alerts should go away now. The queues have all been processed [21:54:17] eileen, ejegg|afk menionted that targetsmart2 job was failing [21:54:23] and suggesting leaving it off [21:54:33] is that a problem for you [21:54:35] jgleeson: if you turn it on I'll fix it [21:54:41] ah ok [21:54:42] it will just be some bad data [21:54:47] so I can switch them all back on [21:54:53] I'm keen to get them finished today so will keep fixing [21:54:58] sure [21:56:35] eileen, they're back on [21:56:43] great [21:57:13] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 3.2.6 on 127.0.0.1:6379 has 1 databases (db0) with 6 keys, up 40 days 20 hours - memory use is 1.83M (peak 8.09M, 0.06% of max, fragmentation 2.73%), connected_slaves is 2, donations is 0, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 13, payments-init is 2, pending is 0, recurring is 0, refund is 0, unsubscribe is 6 [21:57:38] ah target smart 2 has a weird line - I'll track it down though [22:10:28] ok it's back on track [22:10:49] nice! [22:18:55] Team - I just want to commend whoever came up with Quick and the deadlocked - am loving it [22:19:10] lol [22:19:16] AndyRussG, I think [22:19:17] ? [22:20:19] I have a suspicion Andy is over-represented in our sprint names :-) I think Andy Dylan and perhaps David have a high hit rate [22:32:03] jgleeson: no it wasn't me! Maybe ejegg? [22:32:25] I had a nice one but I thought of it too late... QuickTime and Furious [22:39:56] :-)