[00:28:10] (PS1) Ejegg: WIP implement length for Predis and PDO backends [wikimedia/fundraising/php-queue] - https://gerrit.wikimedia.org/r/314209 (https://phabricator.wikimedia.org/T147226) [00:30:12] (PS2) Ejegg: WIP implement length for Predis and PDO backends [wikimedia/fundraising/php-queue] - https://gerrit.wikimedia.org/r/314209 (https://phabricator.wikimedia.org/T147226) [04:08:01] Fundraising Sprint Stirring The Pot, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Fatal exception when cloning/saving banners with translatable messages - https://phabricator.wikimedia.org/T146880#2691633 (AndyRussG) Here is my theory so far: - When you add a new translatable message, a... [04:10:22] Fundraising Sprint Stirring The Pot, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Fatal exception when cloning/saving banners with translatable messages - https://phabricator.wikimedia.org/T146880#2691635 (AndyRussG) Adding @Krinkle and @aaron because Revision.php... :) Thx in advance 4 a... [16:26:38] Fundraising-Backlog, FR-Adyen: Adyen GB form not loading correctly - https://phabricator.wikimedia.org/T147475#2693687 (DStrine) [16:38:56] Fundraising Sprint Stirring The Pot, Fundraising-Backlog, fundraising-tech-ops, FR-Smashpig, Unplanned-Sprint-Work: SmashPig fredge database credentials are incorrect - https://phabricator.wikimedia.org/T146950#2675781 (Ejegg) Open>Resolved [16:39:10] Fundraising Sprint Pretending This Isn't Happening, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-ActiveMQ, and 2 others: [Epic] Rewrite all queue clients to use a single shim library, improve library - https://phabricator.wikimedia.org/T133108#2693748 (Ejegg) [16:39:13] Fundraising Sprint Pretending This Isn't Happening, Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, and 6 others: Migrate donations to new queue - https://phabricator.wikimedia.org/T131277#2693747 (Ejegg) Open>Resolved [16:39:21] Fundraising-Backlog, FR-PayPal-ExpressCheckout, FR-Paypal, FR-Smashpig: Make PayPal listener understand Express Checkout - https://phabricator.wikimedia.org/T130851#2693751 (Ejegg) [16:39:23] Fundraising Sprint Octopus Untangling, Fundraising Sprint Pretending This Isn't Happening, Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, and 7 others: Move legacy PayPal listener to SmashPig - https://phabricator.wikimedia.org/T141654#2693750 (Ejegg) Open>Res... [16:39:30] Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, Fundraising-Backlog, and 2 others: Migrate refunds to new queue - https://phabricator.wikimedia.org/T145234#2693752 (Ejegg) Open>Resolved [16:44:01] so missing predecessor is not a bad error? [16:44:29] cwd we definitely get some of those from time to time, but I want to investigate [16:44:45] since we just deployed the new recurring consumer [16:44:57] Who knows something about ContributionTracking config? :) [16:45:35] * cwd touches nose [16:45:49] contribution-tracking-setup.php is a weird anti-pattern in the wmf-config directory. Is there a reason that the 3 not-so-private entries can go in CommonSettings and the 1 definitely private one go to PrivateSettings? :) [16:46:26] *can't go [16:47:10] ostriches: hmm, we may be able to just nuke it entirely [16:47:28] Even better! [16:47:33] I think it used to start generating the donation tracking IDs on wikis off the fundraising cluster [16:47:52] but no longer [16:49:05] It's enabled on donatewiki, foundationwiki and testwiki [16:49:07] Right now [16:51:23] Fundraising-Backlog, MediaWiki-extensions-ContributionTracking: Clean up Contribution Tracking settings in main wmf config repo - https://phabricator.wikimedia.org/T147479#2693840 (Ejegg) [16:52:00] let's kill it! [16:52:03] ostriches: thanks for pointing that out - got a bunch of stuff in progress right now, but we'll clean that up as soon as we can! [16:52:48] Awesome thx. I'm kinda on a mission to clean up weird untracked files in wmf-config, and this one had me stumped :) [16:54:21] Bahahahaha, the config in InitialiseSettings for this hasn't changed since Roan's initial commit of the files from *fenari* in Feb 2012 :D [17:00:24] fr-tech: I went over to my friend, he was eatin' a pickle. [17:00:24] I said "Hi, what's happenin'?" [17:00:24] He said "Nothin'." [17:00:24] Try to sing this song with that kind of enthusiasm; [17:00:24] As if you just squashed a cop. [17:00:24] -- Arlo Guthrie, "Motorcycle Song" [17:00:24] -- discuss. [17:00:50] (PS1) Ejegg: Check payments_init for 'failed' in pending QC [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/314302 (https://phabricator.wikimedia.org/T143945) [17:01:04] Fundraising-Backlog, MediaWiki-extensions-ContributionTracking, Scap3 (Scap3-MediaWiki-MVP): Clean up Contribution Tracking settings in main wmf config repo - https://phabricator.wikimedia.org/T147479#2693901 (demon) [17:02:09] ostriches: hah, I'm not surprised! The fr-cluster has been too well isolated for that kind of interaction since before my time here [17:18:44] WikiPage::doUpdateRestrictions() [17:19:17] WikiPage::insertProtectNullRevision() [17:20:20] Revision::insertOn() [17:20:39] Revision::checkContentModel() [17:20:56] Revision::getContent() [17:44:21] Fundraising Sprint Stirring The Pot, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Paypal, and 2 others: New failmail from recurring queue consumer - https://phabricator.wikimedia.org/T147487#2694067 (awight) [17:49:17] (PS3) Awight: Add inline help to the campaign editor [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/314196 (https://phabricator.wikimedia.org/T134958) [17:49:48] fr-tech: https://www.mediawiki.org/wiki/Manual:$wgNamespaceProtection [17:51:39] niiice [17:51:51] ejegg: ^ fyi [17:52:19] yay techtalk! [17:52:23] :) [17:56:25] oh hey, so we /don't/ need to apply those protections after all? [17:56:38] That's the dream, at least [17:56:42] cool! [18:04:03] Fundraising-Backlog: MARKER for reviewing sprint 3 - https://phabricator.wikimedia.org/T147489#2694137 (DStrine) [18:07:22] Fundraising Sprint Stirring The Pot, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Paypal, and 2 others: New failmail from recurring queue consumer - https://phabricator.wikimedia.org/T147487#2694165 (awight) Just grabbing a few at random: For S-98R21416J3634250J, S-1VT05926SF672601J, S... [18:11:41] Fundraising Sprint Stirring The Pot, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Paypal, and 2 others: New failmail from recurring queue consumer - https://phabricator.wikimedia.org/T147487#2694067 (Ejegg) OK, so we should have silently set these up for delayed processing rather than s... [18:12:46] ejegg: Want to review that paging patch, and I'll deploy the damaged UI? [18:13:37] awight: sure! [18:13:51] * awight rubs palms [18:14:01] actually, we should be able to deploy the unsubscribe consumer again and finally get off the cherry-picker [18:14:16] Yea, I was just going to ask about that [18:15:18] XenoRyet: yeah, we just got the settings in for that [18:15:21] Cool [18:15:26] fire at will! [18:15:38] What's the extra bit when the last deploy was rolled back? [18:15:41] Fundraising Sprint Stirring The Pot, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Paypal, and 2 others: New failmail from recurring queue consumer - https://phabricator.wikimedia.org/T147487#2694187 (awight) Yep--We need to do something special to let SmashPig know not to failmail for W... [18:15:56] XenoRyet: it's like f_C_u -p civicrm=HEAD [18:16:03] cool [18:16:15] I'll get that out then. [18:18:29] !log updated civicrm from 412d999b671e13982d718437ab3be56efaa25baf to 5f53ef867cf3c2bb2f919c2feb9323eededfb7e7 [18:18:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:19:01] Alright, done [18:20:01] XenoRyet: I'm drush cc'ing for the damaged patch... [18:20:17] cool [18:20:21] huh--ok I guess it's not part of whatever got deployed, nbd [18:20:37] still needed for the new qc [18:20:54] ok I'll cc registry then [18:21:15] & all. done. [18:21:26] Ok, looks like there's still one thing in the unsub queue. Imma press that button and hope for blue dot. [18:23:38] Yep, blue dot. Looks like the message was consumed as well. [18:23:51] woohoo! [18:23:56] congratulations [18:24:03] tis a good day [18:24:34] Fundraising Sprint Stirring The Pot, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Smashpig, Unplanned-Sprint-Work: Missing "gross" and "currency" failmails from donations queue consumer - https://phabricator.wikimedia.org/T147491#2694221 (awight) [18:24:39] yey [18:24:54] I didn't even have to use my A.K., verily [18:27:43] ejegg: just looking at the adyen processcapturerequest job [18:27:57] and the donationinterfacemessage just returns an array [18:28:01] Fundraising Sprint Stirring The Pot, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Smashpig, Unplanned-Sprint-Work: Missing "gross" and "currency"; positive amount; failmails from donations queue consumer - https://phabricator.wikimedia.org/T147491#2694235 (awight) [18:28:12] oh you mean instead of an object? [18:28:25] (for the paypal job) [18:28:53] yeah [18:29:42] gotcha thanks. i was thinking an array of arrays for some reason [18:29:58] oh so we don't have to use that Message class at all [18:30:00] wicked [18:33:07] Fundraising Sprint Stirring The Pot, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Smashpig, Unplanned-Sprint-Work: Missing "gross" and "currency"; positive amount; failmails from donations queue consumer - https://phabricator.wikimedia.org/T147491#2694279 (awight) [18:33:43] cwd: ^ interesting fails caused by PP IPN messages. [18:33:57] I'm not sure why "new_case" is getting into the pipeline [18:34:23] We might want "adjustment" though, not sure if the old code handled those / what we want to do with that. [18:35:18] if they are recurring the request gets copied wholesale [18:35:20] unedited [18:35:36] aha--cos those are hitting the donations queue [18:35:55] uhm i think so [18:35:59] the subscr_payment one [18:36:04] (CR) Ejegg: [C: 2] "Thanks! Will definitely want to do this with a loop when we add more params, but this works for now." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314054 (https://phabricator.wikimedia.org/T142058) (owner: Awight) [18:36:22] cwd: nah these are txn_type=new_case and adjustment [18:36:37] oh weird... [18:36:44] well i am right in that code now [18:36:53] fixing so we can stop mirroring [18:37:40] cwd: OK I see that SmashPig.yaml has paypay.messages.txn_types entries which cause those to route to donations [18:37:49] oops-- messages.verified.txn_types [18:38:48] (Merged) jenkins-bot: Pager glue for the damaged queue [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314054 (https://phabricator.wikimedia.org/T142058) (owner: Awight) [18:39:31] ejegg: ty, I'll deploy that now [18:39:41] ooh, but they are failmailing? [18:40:37] cwd: yah--Not even sure what we want to do with that stuff. "adjustement" seems important [18:40:39] new_case perhaps not [18:41:43] cwd: Maybe we want to drop the messages for now, and write a TODO task for parsing them one day [18:42:47] are they getting all the way to civi? [18:43:23] ejegg: SourceFields sets with object access which blows up on an array [18:43:30] should we convert them all to arrays? [18:43:36] (PS1) Ejegg: Quick fix to turn off Stomp mirroring [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/314319 [18:43:51] (PS1) Awight: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/314320 [18:43:54] cwd yeah, I think that's part of the cleanup work [18:44:16] (PS2) Awight: Merge master into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/314320 [18:44:42] (CR) Awight: [C: 2] Merge master into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/314320 (owner: Awight) [18:44:53] awight: might need smashpig updated in vendor too [18:45:00] ejegg: cool ty [18:45:07] (Merged) jenkins-bot: Merge master into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/314320 (owner: Awight) [18:45:19] cwd: Not quite, they're hitting the donations queue consumer, which doesn't know what to do [18:45:33] IMO we shouldn't put them in the queue [18:45:44] Can we just remove from the txn_types array? [18:45:50] oh shoot, I need to head over to my credit union [18:46:31] awight: i'm pretty sure that will work [18:46:43] they should just get quietly ignored [18:47:30] should being the operative word [18:47:40] (PS1) Awight: Update SmashPig lib [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/314321 [18:47:53] (PS1) Awight: Update composer libs [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/314323 [18:48:04] (CR) Awight: [C: 2] Update SmashPig lib [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/314321 (owner: Awight) [18:48:21] (CR) Awight: [C: 2] Update composer libs [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/314323 (owner: Awight) [18:48:53] (Merged) jenkins-bot: Update composer libs [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/314323 (owner: Awight) [18:50:03] (CR) Awight: [V: 2] Update SmashPig lib [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/314321 (owner: Awight) [18:51:16] !log update fundraising CRM from 5f53ef867cf3c2bb2f919c2feb9323eededfb7e7 to d52c04db4de38d3961661d719b462d27f44c831c [18:51:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:57:55] (PS1) Cdentinger: stop mirroring verified [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/314326 [18:58:05] lmk if this is dumb ^ [18:58:16] maybe we can just do all of them at once [19:01:34] (CR) Awight: [C: 2] stop mirroring verified [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/314326 (owner: Cdentinger) [19:07:46] (Merged) jenkins-bot: stop mirroring verified [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/314326 (owner: Cdentinger) [19:20:32] (PS1) Awight: Display retry_date in damaged message item view [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314329 [19:23:38] awight: I think we want to filter out anything with a retry_date [19:23:47] (PS1) Awight: Blank deprecated repo [wikimedia/fundraising/PaymentsListeners] - https://gerrit.wikimedia.org/r/314330 [19:23:53] or mbeat wanted that, in any case [19:24:29] ejegg: aha [19:25:07] (Abandoned) Awight: Display retry_date in damaged message item view [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314329 (owner: Awight) [19:25:38] I think I see what to do - fix sendToDamagedStore in baseQueueConsumer [19:25:41] gotcha ->condition( 'retry_date', null ); [19:25:50] so it doesn't log an error if there's a retry date [19:26:44] ejegg: That sounds like a nice hack. I'll unhand it, then [19:27:25] but... it still looks like those things aren't getting a retry date [19:27:44] oof [19:30:21] Fundraising-Backlog, fundraising-tech-ops: Use Redis 3 when it's available - https://phabricator.wikimedia.org/T113399#2694584 (awight) Open>declined This will happen naturally. The priority is much lower now that we don't need high-availability logic in frontend code. [19:30:30] (PS1) Ejegg: Don't log an error when delaying a message [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/314332 [19:31:20] (CR) jenkins-bot: [V: -1] Don't log an error when delaying a message [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/314332 (owner: Ejegg) [19:32:54] (PS2) Ejegg: Don't log an error when delaying a message [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/314332 [19:33:42] (CR) Awight: [C: 2] "Great first step!" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/314332 (owner: Ejegg) [19:34:56] awight I'm guessing this has a problem: [19:34:58] if ( $message['date'] + $ageLimit < time() ) { [19:35:56] in WmfQueueConsumer [19:36:23] hmm, or is it? [19:38:18] yep, I see the 'removing failed message' lines in the logs. [19:39:06] Fundraising-Backlog, fundraising-tech-ops, FR-Smashpig: Figure out a sane way to deploy SmashPig.yaml configuration - https://phabricator.wikimedia.org/T147503#2694610 (awight) [19:39:37] aha, no 'date' set at all in the recurring messages [19:40:05] (Merged) jenkins-bot: Don't log an error when delaying a message [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/314332 (owner: Ejegg) [19:41:05] ah yes that is true! [19:41:12] at least for PayPal [19:41:23] and i *think* it was always that way [19:41:27] ok, let's hack that in, I guess [19:41:45] can the new consumer take a normalized message? [19:41:49] err, dang, actually the payment date and the retry timeout are different things [19:42:21] let's see, does source_enqueued_time get updated? [19:43:10] Fundraising-Backlog, fundraising-tech-ops, FR-Smashpig: Figure out a sane way to deploy SmashPig.yaml configuration - https://phabricator.wikimedia.org/T147503#2694650 (awight) [19:43:23] hehe. Jeff_Green ^ when you have a chance to review [19:43:27] (CR) Ejegg: "Dang, I just realized this removed the checkboxes for batch ops" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314054 (https://phabricator.wikimedia.org/T142058) (owner: Awight) [19:44:35] aargh [19:49:08] ejegg: meeting? [19:57:24] (PS1) Ejegg: Fix retry date calculation logic [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314337 [20:02:31] Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Monitoring, FR-2016-17-Q2-Bugs, Spike: Spike: Identify most offensive issues in the DonationInterface logs - https://phabricator.wikimedia.org/T139423#2694752 (DStrine) [20:03:08] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-2016-17-Q2-Campaign-Support: Not all online donations getting tagged to Restrictions and Gift Source Fields - https://phabricator.wikimedia.org/T138361#2694753 (DStrine) [20:03:29] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-2016-17-Q2-Bugs: Don't reject messages on unknown database error - https://phabricator.wikimedia.org/T140694#2694754 (DStrine) [20:03:49] Fundraising-Backlog, FR-Adyen, FR-WMF-Audit, FR-2016-17-Q2-Bugs: Adyen refunds: look at reconciliation / auditing to make sure they get to Civi - https://phabricator.wikimedia.org/T135641#2694755 (DStrine) [20:04:08] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-PayPal-ExpressCheckout: Damaged queue messages missing from Drupal interface - https://phabricator.wikimedia.org/T137735#2694756 (awight) Open>declined [20:04:35] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, FR-2016-17-Q2-Campaign-Support, Spike: Spike: Prioritized checklist of pre-December CentralNotice and related essentials - https://phabricator.wikimedia.org/T141918#2694761 (DStrine) [20:07:15] Fundraising-Backlog, FR-Amazon, FR-PayPal-ExpressCheckout, FR-Paypal, and 3 others: Assisted currency conversion for PayPal is broken again - https://phabricator.wikimedia.org/T98447#2694785 (DStrine) [20:08:37] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, FR-2016-17-Q2-Campaign-Support: Study more details of the impressions-to-pageviews correlations - https://phabricator.wikimedia.org/T122590#2694821 (DStrine) [20:08:59] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-2016-17-Q2-Campaign-Support: CiviCRM bounce import job is failing - https://phabricator.wikimedia.org/T122265#2694822 (DStrine) [20:09:30] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-2016-17-Q2-Campaign-Support: CiviCRM bounce import job is failing - https://phabricator.wikimedia.org/T122265#2694838 (awight) [20:09:32] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Fix performance problems recording thank you emails to CiviMail records - https://phabricator.wikimedia.org/T86863#2694839 (awight) [20:14:26] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, MediaWiki-Vagrant: assert statements in latest Civi break wmf_civicrm install in vagrant - https://phabricator.wikimedia.org/T132810#2694868 (Ejegg) Open>Resolved a:Ejegg Our fix got merged upstream! [20:15:12] fr-tech: I think this will fix the MISSING_PREDECESSOR messages: https://gerrit.wikimedia.org/r/314337 [20:15:41] well, new ones. I'll give the ones sitting in damaged now a retry date [20:16:03] ejegg: cwd: fyi there does seem to be a date in recurring messages, payment_date [20:16:24] yeah, I realized we don't actually want to look at payment date [20:16:35] huh. right, cos it could be any time [20:16:37] source_enqueued_time makes more sense for retry timeout [20:16:37] ok all the IPNListener_Standalone crap I could fine is purged [20:16:46] nice to hear, ty! [20:16:53] woohoo, cleanup time! [20:17:32] omg I almost pasted donor PII into this chat [20:17:43] eek [20:18:19] Jeff_Green: note for you buried among backscroll above, T147503 includes a nasty but possibly workable solution [20:18:19] T147503: Figure out a sane way to deploy SmashPig.yaml configuration - https://phabricator.wikimedia.org/T147503 [20:18:39] awight: yup, saw that [20:18:49] ejegg: If it had ended in a linefeed, I'd be fired right now [20:18:54] >-< [20:19:16] highlight/middle button? [20:19:50] hmm, i wonder if I can disable that for one terminal profile [20:20:29] nope... [20:20:40] it was actually in my pasta buffer [20:20:57] fugu pasta [20:25:27] ejegg: are we deprecating Messages entirely? there are a boat load of them [20:25:38] but they should all just be arrays to push to redis? [20:26:23] Fundraising-Backlog, fundraising-tech-ops, Spike: Spike: decide what to do with shoot_banners script - https://phabricator.wikimedia.org/T142064#2694955 (AndyRussG) [20:26:25] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Epic: [Epic] Analytics, graphs and monitoring for banners and CentralNotice - https://phabricator.wikimedia.org/T124132#2694956 (AndyRussG) [20:26:29] Well, anything that ends up on the queue should be [20:26:31] Message was a terrible thing [20:26:46] but mass refactor w/o a futuristic plan is scary [20:26:53] Some of those are just our representation of what the processor sent us [20:27:04] what about e.g. AdyenMessage? [20:27:05] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, FR-2016-17-Q2-Campaign-Support, Spike: Spike: Prioritized checklist of pre-December CentralNotice and related essentials - https://phabricator.wikimedia.org/T141918#2694970 (AndyRussG) Related: T124132 [20:27:18] there is a grip of Stuff there [20:27:39] cwd yeah, that's one of the ones we just use to hold incoming info [20:27:43] a Stuff-grip? [20:27:54] It sounds like you and ejegg already talked about it--but refactoring all that sounds like a lowish priority [20:28:04] plus, we don't have a plan going forward, or do we? [20:28:19] Yeah, I'm happy waiting for January to fix all that [20:28:33] For example, the getDestinationQueue smartrecord nonsense needs to be rethunk [20:29:30] cwd I've about got myself convinced that we /do/ need another wrapper around PHPQueue, to have redis queue settings in one place instead of yaml-magic, and to add the source fields uniformly [20:29:44] it could also array-ify objects [20:29:59] It is kind of nice to have properties instead of keys [20:30:20] what is better about it? [20:30:29] since it's going into redis anyway [20:30:45] easier to notice misspellings etc [20:31:06] well, except for where we just dynamically declare it all [20:31:59] i am wondering if flipping back and forth between array and object could be lossy [20:32:02] or error-y [20:33:26] ejegg: use of SourceFields appears limited enough that we could change the 3 instances to arrays pretty easy like [20:33:31] the json flip-flop we're using works ok for now - it's the same thing that happens at either end of the queue [20:34:03] cwd yeah, everything that hits a queue should get those [20:34:10] see: wrapper [20:34:35] I think it would be healthy to have a known schema for our messages, however that's accomplished. [20:34:57] right, that's what I like about the classes [20:35:03] Incoming stuff from the processor doesn't really belong in that regime, though--cos they might change their schema at any point, so we'll probably want those fields to be more dynamic [20:35:14] (prince of robustness et al) [20:35:26] my feeling is that the consumer should handle any schema validation [20:35:28] but we'll need to nail down the serialization before we totally get rid of keyedOpaqueEtc [20:35:34] otherwise we're duplicating the effort [20:36:10] cwd: agreed that the CRM consumer should validate, but the flip side of robustness is that we're strict about our output from the listener for example [20:36:45] e.g. we really don't want to set "address1" or something, and since that's an optional field in the consumer, our bad field is simply ignored. [20:36:52] what's the difference between the output of the listener and the input of the consumer? [20:37:08] Instead, we should get a hard fail in the listener when we try to assign to a nonschematic field [20:37:18] nothing... why? [20:37:23] (PS3) Ejegg: Remove obsolete and broken wmf_unsubscribe module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312441 (https://phabricator.wikimedia.org/T145419) [20:38:09] well i'm saying being strict in the input of the consumer is the same as being strict in the output of the listener, unless we are concerned with queueing messages we don't need to [20:38:20] but where do you divide the validation in that case? [20:38:58] cwd properties just help us humans code it [20:39:22] (especially us ide-dependent humans) [20:39:30] most of the Message classes i've opened have been empty [20:39:36] seems like a sign that they can be rethought [20:39:38] The listener should not be able to set $msg->address1 when creating an output message, cos we will have written a schema for the type of message it's trying to send. [20:39:39] yeah, those are some bs [20:40:07] The consumer should be OK with whatever comes in--maybe a legacy audit processor is glitchy or something [20:40:35] cwd: 110% agreed that Messages and ... ExpatriatedMessages have got to go [20:40:41] so you are saying validate schema in the listener, and validate everything else in the consumer? [20:40:45] um [20:40:54] I'm not exactly suggesting we validate [20:41:15] more like, build something in a organized fashion rather than jam random array keys into a lump in various places and hope for the best :p [20:41:32] It's a problem we face everywhere in our components [20:41:36] it seems to me that's what the consumer is doing [20:41:53] (PS2) Ejegg: Fix retry date calculation logic [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314337 (https://phabricator.wikimedia.org/T147487) [20:41:54] then puts it in donations, recurring, whatever, or bombs [20:42:05] which consumer now? [20:42:29] the job consumer in the paypal case [20:42:43] oh. I'm not counting that as a consumer, but gotcha now [20:43:09] yeah I don't know the best way to solve this issue. [20:43:18] (PS4) Ejegg: Stop mirroring to stomp from audit processors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312442 [20:43:29] Even a stupid struct or something won't save us from omitting the date field, for example. [20:43:31] fr-tech ^^^ is ready to go now [20:43:41] reading... [20:43:53] well there's the var_map thing i stuck in config, it's a bit silly but could be the basis for a schema check [20:44:22] (a lot of it is just verbiage) [20:44:26] this should probably be figured out RFC style... [20:45:45] awight oops, extra phpdoc inserted [20:46:28] ejegg: ooh, what does the IDE do differently with properties? [20:46:59] I'd like functions to include type hints for a DonationsMessage parameter. Message handling would be consolidated in a library, probably part of SmashPig. Maybe we build stuff using a factory that can do basic completeness checking at the point of instantiation. [20:47:46] schema versioning would let 3rd parties use different revisions of our code on each end of the pipeline, whatever is fun [20:49:01] awight: what is an example of message processing that is more complicated than msg&execute() ? [20:49:03] (PS5) Ejegg: Stop mirroring to stomp from audit processors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312442 [20:49:25] cwd IDE'll give you autocomplete and misspelling detection for properties [20:49:28] cwd: maybe wmf_civicrm_contribution_message_import [20:50:34] i sometimes find keys to be more robust, for things like $obj->{'my-weird-property'} [20:50:49] just by virtue of being quotes [20:50:54] *d [20:51:01] like ejegg was saying though, how do we protect against empty( $msg['wrongprop'] )? [20:51:34] maybe there should even be a getter which will return null or something [20:51:39] gah [20:51:45] (PS3) Ejegg: Fix retry date calculation logic [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314337 (https://phabricator.wikimedia.org/T147487) [20:52:03] Anyway, I'd suggest we take that particular refactor slowly [20:52:24] cwd yeah, those headers with hyphens were invented to specifically never be properties... untill they had to be [20:52:45] they were only set in the stomp wrapper [20:53:11] (CR) Awight: Stop mirroring to stomp from audit processors (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312442 (owner: Ejegg) [20:54:05] it's weird that the parser shits on - but not _ [20:54:16] i guess - is a word delimiter in pcre etc [20:55:50] also an arithmatic operator! [20:56:23] ah indeed [20:57:10] i would personally dig something with a setter [20:57:35] the worst thing about those classes imo is mingling the metadata with the regular data [20:57:44] the worst practical implication [20:58:06] yeah, the headers vs data fields were actually kind of useful [20:58:47] (CR) Awight: [C: 2] "woohoo! net negative. Just one question inline about the exception catching." (3 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312442 (owner: Ejegg) [20:59:04] _1 [20:59:05] +1 [20:59:49] so a base class with a setter for a payload [21:00:01] could also have a schema check on rehydrate [21:00:22] store the schema in a flat file [21:00:40] step 3: prophet [21:01:54] (Merged) jenkins-bot: Stop mirroring to stomp from audit processors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312442 (owner: Ejegg) [21:02:04] * awight raises arms to give thanks [21:02:27] * cwd says sooth [21:02:33] ejegg: Are you already unhooking the python audit activemq, or can I? [21:05:48] ejegg: & what bulk operation checkboxes were you mentioning earlier? I don't see anything like that in /damaged or /queue/damaged [21:05:55] awight: I haven't yet [21:06:02] ejegg: cool, doing that now. [21:06:36] awight: the patch with the broken paging had a column of checkboxes that you could use to select multiple messages to delete [21:07:04] I think XenoRyet is mucking with that same code atm [21:07:10] to add the source fields [21:07:27] I haven't actually started yet. I'm still puttering around with Adyen. [21:08:00] (PS1) Awight: Stop mirroring to ActiveMQ [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314416 [21:09:02] (PS2) Awight: Stop mirroring to ActiveMQ [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314416 (https://phabricator.wikimedia.org/T145660) [21:09:30] ejegg: Really? I saw nothing at commit 1540aefd117698747cc625a22b445ac835c0c0d5^ [21:09:59] Cool, deferring to XenoRyet [21:10:21] (PS1) Ejegg: Update FundraisingEmailUnsubscribe [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/314417 [21:10:41] awight: oh weird, I definitely had that at some point. Maybe my rebase killed it [21:10:46] awight: I haven't put any code to paper yet. You're not stepping on my toes in there. [21:12:18] Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, Fundraising-Backlog, and 2 others: Move recurring queue consumer off ActiveMQ - https://phabricator.wikimedia.org/T145233#2624193 (awight) Moving to "done", looks like it's just clean... [21:16:12] (PS1) Ejegg: More audit cleanup [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314421 [21:16:26] awight just caught a bug where I got rid of a return value ^^^ [21:16:34] and made a couple of your suggested changes [21:20:24] far as i can tell every damaged message is from the paypal listener [21:20:30] * cwd feels shame [21:20:54] ejegg|brb: I'm tweaking that table_pager btw [21:21:00] cwd nah, it's that retry date calc thing that's broken [21:21:10] baahaha you popped your cherry bomb [21:21:28] * awight smears self with mud [21:21:38] cwd this might be the fix: https://gerrit.wikimedia.org/r/314337 [21:23:20] ejegg: don't recurring messages have neither of those? [21:23:30] date types i mean [21:23:36] cwd they seem to get the source_ one! [21:23:40] at least from the listener [21:23:47] ooh, SourceFields [21:23:58] yah [21:24:40] (CR) Cdentinger: [C: 2] Fix retry date calculation logic [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314337 (https://phabricator.wikimedia.org/T147487) (owner: Ejegg) [21:24:45] sounds pretty good to my ear [21:24:47] thanks! [21:25:05] oh, I'll want to pick up that SmashPig thing awight just merged too... [21:26:01] i've got one waiting to go in SP [21:27:22] (Merged) jenkins-bot: Fix retry date calculation logic [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314337 (https://phabricator.wikimedia.org/T147487) (owner: Ejegg) [21:27:57] cwd need CR? [21:28:20] nope it's merged, just the stop mirroring one [21:28:33] it's fugly and i have been trying not to unfugly [21:28:38] but the urge is strong [21:28:54] risky unfugly? [21:29:37] (PS1) Ejegg: Update SmashPig to fix failmail [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314424 [21:30:05] i think it pretty much bleeds into the Message refactor [21:30:48] ahh [21:31:03] (PS2) Ejegg: Fix audit refund bug from parent patch, clean up [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314421 [21:31:08] cwd: Gross, I just ran into the same data vs metadata issue in Drupal, they used a prefix. How's this for rotten tomato award: https://api.drupal.org/api/drupal/includes!form.inc/function/theme_tableselect/7.x [21:31:08] (CR) Ejegg: [C: 2] "self-merging lib update" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314424 (owner: Ejegg) [21:31:43] rotten tomato is not even stinky enough... [21:31:47] weird man [21:31:51] just plain weird [21:31:51] oh man, those # properties [21:31:57] #surprise! [21:32:18] i try not to think too hard when i have to do drupal ui stuff [21:32:28] copy. paste. [21:33:30] I'd like to deploy CRM, but I realized I broke refunds in wmf_audit: https://gerrit.wikimedia.org/r/314421 [21:33:55] programming is like housecleaning, but with a much wider margin for screwing off and taking a dump on the couch [21:34:05] k looking [21:34:20] * awight gets scoldy [21:34:29] nah nvm [21:36:30] (Merged) jenkins-bot: Update SmashPig to fix failmail [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314424 (owner: Ejegg) [21:36:38] (CR) Awight: [C: 2] "bubble away! yeah, a queue exception should bring us to a halt." (2 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314421 (owner: Ejegg) [21:37:27] (CR) Ejegg: Fix audit refund bug from parent patch, clean up (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314421 (owner: Ejegg) [21:38:31] (CR) Awight: Fix audit refund bug from parent patch, clean up (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314421 (owner: Ejegg) [21:39:45] (Merged) jenkins-bot: Fix audit refund bug from parent patch, clean up [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314421 (owner: Ejegg) [21:41:00] thanks for all the CR awight and cwd! [21:41:17] thanks for the code! [21:41:40] hehe, that's the fun part [21:42:15] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/314432 [21:43:21] WFMU is on a roll [21:44:00] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Generic Individuals Import Error - https://phabricator.wikimedia.org/T147521#2695203 (LeanneS) [21:45:30] speaking of scrobbles, simple lastfm scrobbler for android can now send to your own gnufm server [21:45:44] you do so? [21:45:53] (url pls) [21:46:28] I'm curious about that scene, wondering how people can follow each other. Seems overwhelming [21:47:35] awight: yeah, I don't know how to federate [21:47:35] (PS1) Ejegg: Update smashpig [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/314434 [21:47:36] OMG. theme_tableselect takes ['element']['#field'] and converts it to theme_table('field'... [21:48:26] If the stations were being archived in a federable way, then you could listen to people's playlists after the fact [21:48:37] also, i want album art. trying to get rid of non-free dependencies for that silly location tracker / song map overlay I made a couple years ago [21:49:07] eh? send *that* along [21:49:09] well, not that the album art is free, just that lastfm's api included it in lookups [21:49:17] I don't think I've seen it [21:49:31] awight: https://github.com/ejegg/Tunemapper [21:49:59] (CR) jenkins-bot: [V: -1] Update smashpig [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/314434 (owner: Ejegg) [21:50:16] not in demoable shape just atm [21:50:54] grr polyfill breaking php55lint. [21:51:19] (CR) Ejegg: [C: 2 V: 2] Update smashpig [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/314434 (owner: Ejegg) [21:52:13] (PS2) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/314432 [21:55:35] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/314432 (owner: Ejegg) [21:56:44] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/314432 (owner: Ejegg) [21:56:48] cwd why not push direct to 'data-store/donations'? verified always sounded odd, I wanted to deprecate it [21:57:13] also, there's other stuff pushing to verifed that might need array-ifying [21:57:58] ejegg: fine with me! i think it was just for backcompat? [21:59:16] cool, let's see... [21:59:58] !log updated civicrm from d52c04db4de38d3961661d719b462d27f44c831c to 8bc490854c397cca1a0fa75f313e2ce5f068d3a1 [22:00:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:00:14] ok, with any luck all that recurring failmail should stop [22:00:25] i'll add retry_date to the existing ones [22:02:31] oh, only 25, not so bad [22:06:03] dang, did they just disappear? [22:08:18] nope, that's not it, there are 114 in the queue [22:08:33] why is the consumer not seeing them now? [22:09:00] drat... [22:09:29] that sounds fun [22:09:36] needs moar logging [22:10:55] code looks right, SmashPig.yaml definition for data-store/recurring-new looks right [22:12:30] other queue consumers are getting messages [22:14:45] last got any messages 4 hrs and 35 min ago. hmm [22:16:50] batch size and time limit are sane [22:17:23] * cwd ponders the number of worldwide locality abbreviations that are NA [22:17:50] Heh, good point [22:18:48] speaking of packing your data with metadata [22:24:38] awight: oh phooey, I think that wmf_common_init think might be to blame [22:24:51] let's initialize that on demand [22:25:05] ejegg: btw https://gerrit.wikimedia.org/r/#/c/314416/ [22:25:20] Yeah, that's better anyway [22:25:34] too bad about the duplication, though [22:25:53] (CR) Ejegg: [C: 2] "Looks good!" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314416 (https://phabricator.wikimedia.org/T145660) (owner: Awight) [22:25:59] (Merged) jenkins-bot: Stop mirroring to ActiveMQ [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314416 (https://phabricator.wikimedia.org/T145660) (owner: Awight) [22:27:22] (PS1) Awight: Merge master into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/314445 [22:27:33] (CR) Awight: [C: 2] Merge master into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/314445 (owner: Awight) [22:27:39] (Merged) jenkins-bot: Merge master into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/314445 (owner: Awight) [22:28:23] !log update fundraising-tools from 5427a601ed64ecfe01ef84eda13f28c0dd08a3af to bde60d5ebbdb89c8ecdf0653e92d29f6a5939ec7 [22:28:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:31:26] (PS1) Ejegg: Initialize smashpig context on-demand [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314446 [22:32:24] ugh, local civi won't load now: CiviCRM_API3_Exception: "DB Constraint Violation - possibly dashboard_id should possibly be marked as mandatory for this API. If so, please raise a bug report." [22:32:28] ????? [22:32:37] :| [22:32:48] awight: mind CRing https://gerrit.wikimedia.org/r/314446 [22:32:59] hoping that's all we need [22:33:15] can't think what else would have borked it [22:33:58] or cwd ^^ [22:34:04] just avoiding some drupal magic [22:35:22] whew, i'll have to take your word for it :) [22:35:28] (CR) Cdentinger: [C: 2] Initialize smashpig context on-demand [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314446 (owner: Ejegg) [22:36:08] at worst that patch will break the damaged message ui [22:36:34] but without it anything smash-piggy is suspect [22:36:42] thanks! [22:36:43] yeah i'm just a dummy when it comes to crm [22:37:13] Drupal magic function names are no fun [22:37:34] is drupal the reason for the function name namespacing? [22:37:46] I mean, it was acting as intended, I just shouldn't have applied it that broadly [22:37:51] cwd yeah [22:37:52] (Merged) jenkins-bot: Initialize smashpig context on-demand [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/314446 (owner: Ejegg) [22:38:02] k, let's see if that fixes the recurring consumer [22:39:07] not sure /how/ that _init thing would have broken the one consumer [22:39:13] but here's hoping [22:39:27] otherwise I'm really stumped [22:40:19] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/314447 [22:40:40] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/314447 (owner: Ejegg) [22:40:47] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/314447 (owner: Ejegg) [22:40:53] ejegg: I think this is a typo, https://github.com/drupal/drupal/blob/7.x/includes/form.inc#L3469 [22:41:16] I uh... yeah still working on pager + tableselect [22:41:48] Definitely a rabbit-shaped hole [22:42:16] should be element_children($element['#options']) ? [22:42:46] * cwd squints [22:42:51] Yah that's what I'd imagine [22:43:14] This theme function has an extra "element" level above everything [22:43:25] which may have thrown off the copypasta maker [22:43:33] !log updated civicrm from 8bc490854c397cca1a0fa75f313e2ce5f068d3a1 to 17fab4ded647bad51b30ce65157c88a87e1f7e40 [22:43:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:43:45] maybe I'll patch that in our fork and see what happens [22:44:08] * ejegg crosses fingers and runs recurring qc [22:44:54] hehe, the day started with such promise [22:45:01] arrrrggghh still nothing [22:45:24] and I need to sign off :( [22:46:12] ejegg: should i roll anything back? [22:47:16] good call. Reading the deployment branch and backscroll now to try to understand [22:49:28] I don't know where to roll back to... [22:50:19] hrm the one before last had a lot of stuff [22:50:23] Maybe just graft the recurring fixes from the last two deploys onto the recurring QC deploy from yesterday? [22:50:53] Can you dump us the current issues as you see them, or does cwd have the context? [22:51:13] my context is limited [22:51:18] recurring consumer has issues is all i know [22:51:49] current issue is just that recurring messages exist in redis but the qc keeps saying 0 processed [22:52:04] ejegg: they don't get popped? [22:52:14] subquestion: have they ever, under redis? [22:52:21] I see, build #62752 is the last time we saw messages [22:52:38] since yep [22:53:04] cwd yep, they were popping (and some failing) till 5:40 pm utc today [22:53:59] so... around the time the whole batch of stuff up to the unsubscribe QC got deployed [22:54:18] I don't see any server admin log entry around then [22:54:33] Only 18:18 XenoRyet: updated civicrm from 412d999b671e13982d718437ab3be56efaa25baf to 5f53ef867cf3c2bb2f919c2feb9323eededfb7e7 [22:54:39] 40 minutes later [22:54:49] recurring qc only runs 2x/ hour [22:55:17] That was the unsubscribe consumer, nothing else went out with that deploy. [22:55:21] The job at 18:01 was already empty [22:55:35] maybe legitimately empty? [22:55:45] I doubt it, never happens [22:55:58] * awight checks that statement [22:56:16] ejegg: ok n e way we can take it from here! [22:56:57] sorry guys! [22:56:59] and thanks! [22:57:22] np, you're working too late either way :p [22:57:23] I'll have the lappy with me if needed [22:58:55] Weird, I checked the build-misc logs and there really were no deployments until 18:17 UTC today [22:59:28] Guess I'll check the job configuration and stuff [23:00:06] settings? Jeff_green pushed out two changes today. One was before the emptyness, and the other just turned off sending verified to stomp, afaik [23:00:14] ok, really out now [23:00:18] see ya [23:02:28] uh, oh. JGreen removed the PaymentsListeners stuff around 16:00, are we still getting incoming messages? [23:02:37] ... [23:03:09] plenty of PP messages [23:03:19] yep, cool. [23:03:33] donations queue looks good [23:04:20] There's stuff in Redis recurring, 117 rows [23:05:20] hmmmmmg [23:06:13] bredcrumb: The rqc is trying to use the "recurring-new" SmashPig queue [23:06:53] ok, that's defined on the job runner node [23:07:22] recurring_disable: 0 [23:07:24] awight: refund also has stuff [23:07:29] but not processing [23:07:32] ooh [23:07:39] That does point to SmashPig config badness [23:07:43] yeah [23:08:29] recurring_batch: "150" [23:08:30] recurring_batch_time: "90" [23:08:32] So that's all good. [23:10:11] ouch, I'm... 4 levels of inheritance down into queue consumers :) [23:10:20] in smash pig or crm? [23:10:29] both [23:10:53] RecurringQueueConsumer < TransactionalWmfQueueConsumer < WmfQueueConsumer < BaseQueueConsumer! [23:11:01] It all made sense at the time of the crime... [23:11:07] uhm, ok my first thought it there are -new config nodes for both of these [23:11:15] as well as not -new ones [23:11:35] I did see default.data-store.recurring-new on the production server [23:12:13] but that's something scary to clean up [23:12:17] it looks to me like that's the right one for redis [23:12:20] down with the -new! [23:12:29] ... in good time. [23:12:44] except the queues i see in redis do not have -new [23:12:54] OH [23:13:03] ah but they specify the queues in config [23:13:06] the "queue" key will override that [23:13:07] yah [23:13:20] unless that is what regressed? [23:13:44] this would be just for the consumers [23:13:49] do the crm consumers know to do that? [23:13:55] hmm [23:14:02] ...are these consumers under crm? [23:14:07] the new refund/recurring ones [23:14:21] CRM code that inherits from the SmashPig BaseQueueConsumer [23:14:32] hrm, ok [23:14:51] am i crazy or is that a little crazy? [23:15:05] i guess it doesn't matter, it's just an interface [23:15:11] but it is tightly coupled [23:16:03] BaseQueueConsumer::getQueue in theory has the 'queue' override logic to prefer the configured value [23:16:22] However, that code starts with: // Get a reference to the config node so we can mess with it [23:16:24] ;) [23:17:44] * awight punches lockers about how we haven't figured out a way to deploy SmashPig.yaml [23:18:19] http://koam.images.worldnow.com/images/23951029_BG1.jpg [23:19:02] * awight attempts "drush eval" to debug [23:33:30] omg, that sort of works. [23:34:12] you did what now? [23:34:41] fr-tech any opinion on removing the CentralNotice $wgNoticeProtectGroup global config variable? I think we can just make editing CNBanner namespace pages limited to people with the 'centralnotice-admin' right [23:35:02] * AndyRussG burps [23:35:10] Perfect! [23:35:17] :) [23:35:58] That's great that the namespace protection might work for us [23:36:13] Mmm hope it does! [23:36:34] Can't think of any cases when you'd want to individually open up editing rights for CNBanner:* [23:36:41] cwd: long story short, if you run that script you'll see the actual, configured queue object and happily it has 'queue_name' => 'recurring', [23:37:34] well that's good [23:37:41] although it would have also been a nice solution [23:37:47] yah [23:39:39] sweet, we have access to queue->peek() [23:40:23] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Generic Individuals Import Error - https://phabricator.wikimedia.org/T147521#2695527 (DStrine) p:Triage>High [23:40:58] That's positive. I don't see anything when peeking from that object