[00:04:55] (CR) Ejegg: "One suggestion" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310218 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [00:09:51] (CR) Eileen: Clean up legacy junk data from preferred_language field. (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310218 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [00:16:44] (PS9) Eileen: Clean up legacy junk data from preferred_language field. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310218 (https://phabricator.wikimedia.org/T96410) [00:17:38] ejegg: I just fixed that change [00:17:49] cool [00:19:17] (CR) Ejegg: Clean up legacy junk data from preferred_language field. (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310218 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [00:20:43] eileen oops, i clarified what i meant there ^ [00:22:54] ejegg: pretty sure I didn't see anything like that [00:23:39] ok, no worries then [00:23:52] (PS10) Ejegg: Clean up legacy junk data from preferred_language field. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310218 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [00:24:10] (CR) Ejegg: [C: 2] Clean up legacy junk data from preferred_language field. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310218 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [00:24:19] (CR) Eileen: "Running" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310218 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [00:24:59] (PS6) Ejegg: Remove handling for bad data (language = '_' as we are cleaning that up [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310724 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [00:25:08] (CR) Ejegg: [C: 2] Remove handling for bad data (language = '_' as we are cleaning that up [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310724 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [00:25:27] ejegg: I also put this up to ask questions about - https://gerrit.wikimedia.org/r/#/c/313329/ - specifically at what point I need to trigger an error in order to abort with a fail mail [00:27:11] (Merged) jenkins-bot: Clean up legacy junk data from preferred_language field. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310218 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [00:27:38] (Merged) jenkins-bot: Remove handling for bad data (language = '_' as we are cleaning that up [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310724 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [00:34:31] eileen: hmm, looking [00:36:47] (PS1) Ejegg: Don't escape unicode while hoop-jumping [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313341 [00:36:52] cwd ^^^ [00:37:39] (PS4) Eileen: Update backoffice message template to allow receipt resends. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312952 (https://phabricator.wikimedia.org/T137496) [00:38:16] (CR) Cdentinger: [C: 2] Don't escape unicode while hoop-jumping [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313341 (owner: Ejegg) [00:38:50] ejegg: ty! still trying to imagine how segfault is an appropriate response to encoding problems [00:39:01] (Merged) jenkins-bot: Don't escape unicode while hoop-jumping [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313341 (owner: Ejegg) [00:39:12] cwd right? [00:39:19] I guess better than silent data corruption [00:39:35] but exception seems appropriate [00:39:59] cwd you want to deploy that one? [00:40:35] you bet [00:43:08] (CR) Ejegg: "Looks like that missing setting is configurable at admin/config/recurring_globalcollect/configure" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/313329 (owner: Eileen) [00:44:13] eileen: argh drupal magic! [00:44:19] that's the worst kind of magic [00:46:56] ejegg: any idea how I trigger fail_mail - ie. does the drush validate exit & failmail? [00:46:57] ccccccebfefrrgceiulflbrftrthukdfvfuginthkevu [00:47:03] I think maybe not? [00:47:37] eileen: I have no idea... gotta look into it [00:47:48] (PS1) Cdentinger: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/313342 [00:47:58] eileen: is that welsh? [00:48:07] cwd hah! [00:48:39] https://www.youtube.com/watch?v=fHxO0UdpoxM [00:49:51] (CR) Cdentinger: [C: 2 V: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/313342 (owner: Cdentinger) [00:52:27] hmm, I need to take off [00:52:47] i hate to leave when the donations queue looks this close to a switch [00:52:51] cwd welsh ish [00:53:05] (actually yubikey language) [00:53:49] ejegg: If I want to deploy those couple of patches should I cherry-pick or are you on the verge of a merge [00:55:07] i always forget the v+2 & submit on that repo [00:55:24] eileen: ha yeah i do it all the time [00:55:27] those things have a hair trigger [00:56:13] eileen: I guess I don't quite have time to deploy tonight [00:56:22] so cherry-pick it is [00:56:53] !log updated SmashPig from 3458f93599084815da46a2540e9ed762c8b120ce to 3d0a76b5918f280602e5dabeeb0373c1a118590d [00:56:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:57:32] have a good evening, folks [00:57:48] see ya! [01:00:35] nighyt [01:00:50] nuts, JSON_UNESCAPED_UNICODE not available till 5.4 [01:01:45] well those are all in the damaged store [01:02:59] i better roll that back actually [01:04:15] !log rolled smashpig back because JSON_UNESCAPED_UNICODE is unavailable in php5.3 and don't want consumers to explode [01:04:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:00:29] (PS1) Eileen: Clean up legacy junk data from preferred_language field. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/313344 (https://phabricator.wikimedia.org/T96410) [02:00:31] (PS1) Eileen: Remove handling for bad data (language = '_' as we are cleaning that up [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/313345 (https://phabricator.wikimedia.org/T96410) [02:03:39] (CR) Eileen: [C: 2] "merge to deploy, self-approving" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/313344 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [02:03:46] (Merged) jenkins-bot: Clean up legacy junk data from preferred_language field. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/313344 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [02:03:59] (CR) Eileen: [C: 2] "merge to deploy - self approving" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/313345 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [02:04:06] (Merged) jenkins-bot: Remove handling for bad data (language = '_' as we are cleaning that up [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/313345 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [02:14:56] !log civicrm upgraded from 86371233a9218a526f85cd76d7008f966042fde7 to 6768613907598c52b0af1ffae35faa9078f15f63 [02:15:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:36:37] (CR) Krinkle: [C: -1] "Afaik there's no need for the back-compat. Extensions are branched along core and are not supposed to work with older core versions - exce" [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/313334 (owner: Aaron Schulz) [02:38:44] Fundraising Sprint Stirring The Pot 2016, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Spike: SPIKE: investigate level of completeness of civi deduped DB - https://phabricator.wikimedia.org/T146384#2676152 (Eileenmcnaughton) I ran this query to find the number of email addresses that are m... [03:19:31] (PS2) Eileen: WIP add check for stable data before running GC charges. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/313329 [06:00:43] Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, and 4 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2676251 (Nikerabbit) I saw this on SoS but nothing seems to point towa... [13:20:23] fr-tech: http://xkcd.com/1739/ [13:28:58] !log disabled Adyen job runner [13:29:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [13:29:14] AndyRussG: you got that right [13:34:26] cwd: yeah I was just like heh life :) [13:45:43] didja see the mouseover alt text? [13:48:09] too true [14:00:13] Fundraising-Backlog: Job runners are segfaulting - https://phabricator.wikimedia.org/T146987#2677061 (cwdent) [14:01:12] Fundraising Sprint Stirring The Pot 2016, Fundraising-Backlog: Job runners are segfaulting - https://phabricator.wikimedia.org/T146987#2677073 (cwdent) [14:09:22] (CR) AndyRussG: [C: -1 V: -1] "Thanks!!! Ahh and... weeeee a bug! In the admin UI, in both the list of campaigns and in the campaign detail, it's showing campaigns as no" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/310457 (https://phabricator.wikimedia.org/T145591) (owner: Ejegg) [14:54:45] Fundraising-Backlog, FR-Paypal: Recent Paypal recurring donation not in Civi - https://phabricator.wikimedia.org/T146993#2677209 (MBeat33) [14:56:36] Fundraising Sprint Stirring The Pot 2016, Fundraising-Backlog: Job runners are segfaulting - https://phabricator.wikimedia.org/T146987#2677226 (cwdent) One thing I notice is the QueueFactory is the only place outside of tests that calls config->override() which does a bunch of recursive late static bindi... [15:24:50] Fundraising-Analysis, Fundraising-Backlog, Analytics, MediaWiki-extensions-CentralNotice: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#2677273 (Nuria) Let's look at this one as another use case for streaming cc @Milimetric and @Ottomata [15:31:54] (CR) Ejegg: "Thanks for the CR, and the puzzle to solve!" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/310457 (https://phabricator.wikimedia.org/T145591) (owner: Ejegg) [15:33:49] ejegg: yw! [15:34:12] fr-tech anyone interested, I'm IRCing w/ ori about the CN bug over in #wikimedia-perf [15:35:21] cwd any ideas on the segfaults? [15:37:36] ejegg: https://phabricator.wikimedia.org/T146987 [15:38:02] best i've got, still digging, however changing vagrant to use php instead of hhvm and running tests returns some stuff that might be useful [15:38:30] Fundraising Sprint Stirring The Pot 2016, Fundraising-Backlog: Job runners are segfaulting - https://phabricator.wikimedia.org/T146987#2677061 (Ejegg) Oho, quite possibly. How about I try going back to the code we used in BaseQueueConsumer? [15:39:06] yeah.... definitely looks like the culprit, now that I see where the Adyen runner dies [15:39:50] when i switch to php5.5 i get a bunch of "Only variables should be passed by reference" [15:39:57] (no 5.3 available) [15:40:18] could be a lead, but yeah avoiding that function is probably simpler :) [15:41:21] (PS1) Ejegg: Go back to old queue defaults code [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313419 [15:41:46] (PS2) Ejegg: Go back to old queue defaults code [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313419 (https://phabricator.wikimedia.org/T146987) [15:42:30] cwd oh yeah, 'only variables should be passed by ref'? What line was that? [15:43:31] QueueFactory::getQueue( 'verified' )->push( $queueMessage ); [15:43:33] ah ha [15:43:55] i still can't explain the segfault [15:43:59] but that's gotta be it [15:44:29] oh, so not even the override thing? Just calling push directly? [15:45:19] because... config->object was an & function? [15:45:36] awight: ping! I've been talking to ori in #wikimedia-perf about everyone's favourite cache mystery [15:45:52] also.. should I just be able to ssh into flourine to see logs? I'm getting denied... [15:47:03] AndyRussG: ooh that's exciting--yeah you should have a login but the spelling is fluorine, could that be it? [15:47:21] hmmmm [15:48:25] ejegg: cwd: That sounds incredibly suspect. [15:48:26] cwd well, we can revert the QueueFactory thing for now, and just get the mirroring going [15:48:31] ahahaha could be I was using the Montreal spelling yep [15:48:44] flóurine... [15:49:10] * AndyRussG pushes cat away from keyboard when production shell is open [15:49:20] haha it was a cruel choice to name a machine after a common spelling error [15:49:37] I never get it right the first time--need to create an "fl" alias some day [15:49:46] ejegg: well that's the line in the call stack, but i'm not sure how granular that is [15:50:13] they should change the name so fl is that unicode char that merges the "F" and the "L"... The kerning would be so much better [15:50:25] cause nothing in there is passing by reference afaict [15:50:42] hmph [15:51:46] (PS1) Ejegg: Revert "Get all queues with QueueFactory::getQueue" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313420 [15:51:51] AndyRussG: baahahaha [15:52:22] cwd: r u definitely on the #deployment branch for that? [15:52:26] cwd let's set that aside and get the mirroring going [15:52:40] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi dedupe: enable Find & Merge Duplicate Contacts to use search results - https://phabricator.wikimedia.org/T146997#2677320 (MBeat33) [15:53:01] awight: hmm? i'm testing on master... [15:53:09] even better, all shell commands on production should be required to be valid LaTeX... That'd catch a bunch of typoes! [15:55:26] ugh, I guess I really need to set up a vm with php5.3 [15:55:52] cwd: oh k, thought it might have been a production stack trace [15:56:05] awight: heh no, prod is segfault [15:56:51] gotta run until 10:30 Pac, and I haven't managed to set up "at 10am fortune -o | irc... [15:57:16] AndyRussG: break a leg! Looking forward to listening in to Metrics [15:57:42] awight: heh thanks I'm at the hair salon right now infact [15:57:46] jk [15:58:14] actually if anyone want to help me test my setup/screen sharing in a bit that'd be fun! [15:58:20] cwd think we can just revert and figure out the segfaults after we kill more stomp queues? [15:58:36] ejegg: sounds good to me [15:59:21] one sec... [16:01:13] ejegg: any interest in trying this first? https://gerrit.wikimedia.org/r/#/c/313419/ [16:01:37] i feel like it does the same thing as the revert effectively but keeps QueueFactory [16:01:44] and i still suspect that treeUpdate because the pass by ref error [16:02:48] although in my perfect world we wouldn't need QueueFactory and we'd just fix up the config interface to do it right [16:02:57] but that is of course more work [16:03:03] cwd I guess we could, but doesn't the loadConfigFromPaths use override with arrays read in from yaml? [16:03:10] that seems to be working fine [16:03:36] I'm more suspicious of the function &object vs function getQueue [16:04:56] well... let me try some livehacks in a local copy up in prod [16:05:23] hehe, tell me more [16:06:18] just going to try to find a maint script that uses QueueFactory and can be called in a non-destructive way [16:06:33] oh right i forgot this was on that box [16:06:44] that makes it easier [16:10:52] cwd waitaminute, the pending queue consumer has been running fine [16:11:04] and that's been using QueueFactory [16:11:17] hmmm [16:11:31] but it's assigning the result, then using it [16:11:54] that treeMerge will only recurse based on config right? [16:11:58] could it be a config difference? [16:12:06] ooh, yeah, let's see if there's already a key there [16:13:16] nope, no 'queue' constructor arg for pending in defaults or in the override file [16:13:24] so it's doing the override with that array [16:13:57] if only i could reproduce the darn thing [16:14:24] (Abandoned) Ejegg: Go back to old queue defaults code [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313419 (https://phabricator.wikimedia.org/T146987) (owner: Ejegg) [16:16:35] AndyRussG: did you see the email I added you to yesterday? It's regarding metrics persentation [16:18:00] dstrine: yeah! In about 30 min wanna help me test my setup? [16:18:05] (PS1) Ejegg: Put an ampersand on it, maybe it'll work [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313423 (https://phabricator.wikimedia.org/T146987) [16:18:32] cwd if it's any more complicated than ^^^ I'd like to postpone the autopsy [16:18:50] AndyRussG: I'd like to but I don't know how we'll get a true test since we'll be using bluejeans for the real presentation. [16:19:16] We can make sure you have a solid internet connection and that everything works on hangouts [16:19:42] ok sounds good [16:20:02] AndyRussG: I jsut realized they didn't add you to the 15 mintues prep before metrics [16:20:08] I just added you to that too [16:20:42] ah K thx! [16:22:03] (CR) Cdentinger: [C: 2] Put an ampersand on it, maybe it'll work [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313423 (https://phabricator.wikimedia.org/T146987) (owner: Ejegg) [16:22:58] (Merged) jenkins-bot: Put an ampersand on it, maybe it'll work [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313423 (https://phabricator.wikimedia.org/T146987) (owner: Ejegg) [16:25:08] ejegg: want me to deploy and try that out? [16:25:20] sure! [16:45:13] (PS1) Cdentinger: Merge branch 'master' into 'deployment' [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/313427 [16:46:27] (CR) Cdentinger: [C: 2 V: 2] Merge branch 'master' into 'deployment' [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/313427 (owner: Cdentinger) [16:48:40] Fundraising Sprint Stirring The Pot 2016, Fundraising-Backlog, Patch-For-Review, Unplanned-Sprint-Work: Job runners are segfaulting - https://phabricator.wikimedia.org/T146987#2677458 (DStrine) [16:48:53] !log updated smashpig from 3458f93599084815da46a2540e9ed762c8b120ce to 40c4a7c664dc53f16943aa0b83f30ab1ce435c15 [16:48:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:49:01] * ejegg crosses fingers [16:49:30] ejegg: will adyen fire on its own soon? [16:50:03] i notice dedupe is blowing up right now too [16:50:41] cwd I'll re-enable the adyen job [16:51:14] !log enabled adyen job runner [16:51:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:51:20] oh right thanks [16:51:53] rats [16:52:03] oh well [16:52:11] shall we just revert back a ways? [16:52:30] just the QueueFactory should be enough... lemme rebase that revert [16:52:37] cool [16:52:51] or revert the ampersand patch too, whatever [16:54:17] (PS2) Ejegg: Revert "Get all queues with QueueFactory::getQueue" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313420 [16:54:19] (PS1) Ejegg: Revert "Put an ampersand on it, maybe it'll work" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313430 [16:54:36] cwd want to merge & deploy that bit? [16:55:32] eh, i'll self-merge the reverts [16:55:41] (CR) Ejegg: [C: 2] Revert "Put an ampersand on it, maybe it'll work" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313430 (owner: Ejegg) [16:55:51] (CR) Ejegg: [C: 2] Revert "Get all queues with QueueFactory::getQueue" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313420 (owner: Ejegg) [16:55:58] kk, will deploy as soon as they merge [16:56:23] (Merged) jenkins-bot: Revert "Put an ampersand on it, maybe it'll work" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313430 (owner: Ejegg) [16:56:37] (Merged) jenkins-bot: Revert "Get all queues with QueueFactory::getQueue" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313420 (owner: Ejegg) [16:56:37] cool. I reverted on master, lemme merge to deploy too [16:57:30] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/313431 [16:57:45] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/313431 (owner: Ejegg) [16:57:58] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/313431 (owner: Ejegg) [16:58:09] cwd ok, all set for deploy [16:58:22] ejegg: how did you get that to merge with just c:2? [16:58:35] i have been doing c+v+sumbit [16:58:38] sometimes zuul actually works there ... [16:58:41] no idea [16:58:44] huh [16:58:51] ok thanks! deploying [16:59:24] gotta afk for a few, back soon! [16:59:43] !log updated smashpig from 40c4a7c664dc53f16943aa0b83f30ab1ce435c15 to 0d88feaf8ecab0286e36a91303bb234c68fd6384 [16:59:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:01:49] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Using translatable messages in CentralNotice banners lead to an MWException - https://phabricator.wikimedia.org/T147002#2677522 (AndyRussG) [17:02:09] fr-tech maybe I could test my setup during the tech talk? I may not really be there much because metrics prep though.... Hopefully some, thx 4 understanding!!! [17:03:42] i'll be there soon, fixin the queues... [17:05:55] !log updated PaymentListeners from b4d77a991e100f97d98fcd72eaf03940a4e1845d to 21647c8f4b781b74ae2dc4377334410b4eed7e3c [17:06:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:17:58] (PS1) Cdentinger: REVERT fb4b4427cdc67a656408b8f947bbbefb7396fe25 [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313435 [17:22:36] fr-tech: if anyone is around to press that ^ ...don't feel comfortable self merging a revert of someone else's code [17:23:38] cwd oh dang, good catch [17:24:13] cwd what's that vendor thing doing in there tho? [17:24:35] yikes, wtf [17:24:59] (Abandoned) Cdentinger: REVERT fb4b4427cdc67a656408b8f947bbbefb7396fe25 [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313435 (owner: Cdentinger) [17:25:09] one sec, i need to roll back listeners [17:25:38] !log rolled back PaymentListeners [17:25:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:25:50] just keeping stuff out of the damaged queue i guess [17:27:07] (PS1) Ejegg: Revert "Don't escape unicode while hoop-jumping" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313437 [17:27:38] (CR) Ejegg: [C: 2] "Like cwd said, this option's newer than php5.3" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313437 (owner: Ejegg) [17:27:52] ok every time i reset to master, vendor is modified (new commits) [17:27:59] and i have to update [17:28:08] i know i've done this before [17:28:12] but i can't remember what's up [17:28:23] (Merged) jenkins-bot: Revert "Don't escape unicode while hoop-jumping" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313437 (owner: Ejegg) [17:31:54] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/313438 [17:32:46] XenoRyet, ejegg: AndyRussG is showing off his metrics stuff in tech talk if you wanna join [17:32:54] Cool, brt [17:51:16] (PS1) Cdentinger: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/313439 [17:51:43] (CR) Cdentinger: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/313439 (owner: Cdentinger) [17:53:08] (CR) Cdentinger: [V: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/313439 (owner: Cdentinger) [17:54:28] !log updated smashpig from 0d88feaf8ecab0286e36a91303bb234c68fd6384 to 077ffcc3c4485f62a2f9c80eb4843ef8c72d0c4f [17:54:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:56:26] Processed 27 (27 successful) jobs in 23 seconds. [17:56:33] at last [17:59:00] WORD [17:59:34] if they look good in civi i'll redeploy the redirect [18:00:29] K4-713: o/ [18:02:25] cwd: Oh hey. [18:06:45] AndyRussG: fyi, I'm pretty sure https://phabricator.wikimedia.org/T147002 and https://phabricator.wikimedia.org/T146880 are the same. will let you decide which way round to merge them [18:06:58] we might need to bump the issue to UBN :( [18:20:57] the-wub: thx!! yeah [18:21:21] !log rolled forward PaymentListeners again [18:21:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:21:32] * cwd waits for fail mail [18:25:30] !log disabled CiviCRM dedupe jobs [18:25:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:36:23] * awight gives colleagues standing ovation, thank you for showing off our work! [18:39:22] yes, good stuff! [18:58:55] awight: Yeah, you beat me to it. That was excellent. [18:59:20] Also, special thanks for dstrine for detangling the logistical hairball. [18:59:26] long relocation, see yous in an hour [19:06:31] yeah sooooooooooo smooth, polished and clear, everyone (except me!) [19:06:43] also demonstration of excellent work! [19:38:05] yeah thanks to AndyRussG for the demo!! [19:40:33] fr-tech we need a logo or something. I couldn't find an image that represented what we do. The lucky cat was about the best I could do. It's a technical thing that beckons in the money. [19:41:12] heh [19:41:12] I liked the lucky cat, but yea we probably could find something better. [19:41:37] XenoRyet: we could always draw one of those cats as the logo [19:44:18] (PS1) Ejegg: Deal with refunds having no txn_type [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313455 [19:45:52] dstrine: the cat was nice! [19:47:34] ejegg: meeting? [19:56:08] (CR) Cdentinger: [C: -1] "The one that failed recently actually had payment_status: Refunded (not Reversed)" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313455 (owner: Ejegg) [20:18:58] (PS2) Ejegg: Deal with refunds having no txn_type [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313455 [20:19:09] AndyRussG|bassoo: robla was just asking about https://phabricator.wikimedia.org/T144952 he wanted to check in and maybe see if he could help. Just FYI [20:26:03] (PS2) Aaron Schulz: Use DatabaseBase::factory() over Database::factory() [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/313334 [20:26:40] relocating... [20:26:50] (CR) Cdentinger: [C: 2] Deal with refunds having no txn_type (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313455 (owner: Ejegg) [20:27:38] (Merged) jenkins-bot: Deal with refunds having no txn_type [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313455 (owner: Ejegg) [20:27:45] ejegg|afk: small nitpick in line there, would be nice to put those values in config [20:27:54] *or* remove them from config [20:29:16] but as of now they are not isolated to either place [20:31:01] anyway i have to run to this meeting, back later [20:59:51] dstrine: ah thanks! yeah I saw that he signed up on the bug :) [21:01:33] dstrine: did you have him on IRC or IRL? [21:08:25] If all goes according to plan, slander will ping us some time during 14:09 Pacific [21:08:30] slander: help [21:08:30] If I only had a brain: https://github.com/adamwight/slander -- Commands: help config kill last [21:09:08] fr-tech: What use is magic if it can't save a unicorn? [21:09:08] -- Peter S. Beagle, "The Last Unicorn" [21:09:08] -- discuss. [21:09:17] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/313504 [21:09:28] hehe, nice! [21:09:37] -o is actually too offensive, I think [21:09:46] Next timer is 2016-09-30T14:09:00 [21:09:48] ka-ching [21:10:59] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/313504 (owner: Ejegg) [21:11:11] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/313504 (owner: Ejegg) [21:12:03] slander: config [21:12:03] Configuration: [{"jobs": "fortune: {'at_time': '14:09', 'target': 'fr-tech', 'class': 'fortune'}", "irc": {"maxlen": 10000, "host": "irc.freenode.net", "realname": "fr-tech-talk notification bot", "nick": "slander", "port": 6667, "channel": "#wikimedia-fundraising"}, "poll_interval": 60, "source_url": "https://github.com/adamwight/slander"}] [21:13:04] cool [21:13:31] I'm just commenting that out now... seems unnecessarily open-sourcey [21:14:15] !log updated SmashPig from 077ffcc3c4485f62a2f9c80eb4843ef8c72d0c4f to 2169b71016e2deb1114655d0256d3286ff057943 [21:14:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [21:15:55] (PS3) Eileen: Add check for stable data before running GC charges. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/313329 [21:16:29] awight: ejegg I have some concern that the refund bug could happen when we change month [21:16:45] I added a check to the drush script in this pr [21:16:47] https://gerrit.wikimedia.org/r/#/c/313329/ [21:16:53] sorry - double charge bug [21:17:04] eileen: Thanks for remembering! [21:17:12] to be sure that if it does kick in the charges would not run [21:17:22] ah right, I was trying to figure out if that would failmail. but at least it'll stop double charges [21:17:26] I'm not sure it's 100% ok because the failmail doesn't run locally [21:17:34] but as ejegg says... [21:18:01] I think if the failmail fails we'd still get mail because it is failing on not substantiating a class [21:18:05] eileen: actually, you can explicitly send a failmail if you want [21:18:37] Any failure at that level would be a 'job failed in jenkins' type thing [21:19:06] oh, we just need to know the drush exit code when validation fails [21:19:46] ooh, reverse dns [21:20:29] fr-tech: A soft drink turneth away company. [21:20:29] -- discuss. [21:20:42] heh, was that scheduled? [21:20:48] yah [21:20:56] hehe it might be funny if it goes haywire [21:21:54] be a nice distraction from everything else going haywire [21:22:00] ejegg: I did add an explicit failmail [21:22:08] It just didn't work locally [21:22:13] oh, cool! sorry, didn't see PS3 [21:23:02] k it should be set up for 10am [21:23:24] (CR) Krinkle: [C: 2] Use DatabaseBase::factory() over Database::factory() [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/313334 (owner: Aaron Schulz) [21:24:11] (Merged) jenkins-bot: Use DatabaseBase::factory() over Database::factory() [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/313334 (owner: Aaron Schulz) [21:24:54] eileen: is there some reason you're excluding coinbase instead of limiting to globalcollect? There's paypal recurring in contribution_recur too [21:24:57] fr-tech: throw me scraps! I'm done with the bot... [21:25:38] why would you need a company for soft drinks that turneth away? [21:25:46] awight: any idea if I'm building the WIP damaged message UI in a sane manner? There seem to be a lot of ways to skin cats in our drupal modules [21:25:59] AndyRussG|sortof: yeah that's an odd one--a weak drink maybe [21:26:14] ejegg: yeah I was originally 'all-in' but then found coinbase to be dodgey so I excluded [21:26:23] I guess GC only makes more sense [21:26:39] I was just less sure if there was only one possible string to catch [21:26:54] they should all be RECURRING GLOBALCOLLECT [21:26:57] ejegg: cool, I'll look right after triple-opining this RGC safety stop [21:27:03] OK I'll change to that [21:27:14] I think that's in the module where it finds things to charge [21:27:26] ejegg: eileen: That's questionable, we changed the civicrm_contribution_recur.trxn_id format at some point [21:27:36] oh really? [21:27:44] some might be ^GLOBALCOLLECT, others ^RECURRING GLOBALCOLLECT [21:27:49] yah worth double-checking at least. [21:28:24] (PS4) Eileen: Add check for stable data before running GC charges. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/313329 (https://phabricator.wikimedia.org/T144557) [21:29:29] awight: looks like all have the recurring, we just added a timestamp after the order id [21:29:39] ejegg: cool, ty for looking [21:29:49] that makes the query cheaper [21:32:43] ejegg: awight|meet shall we still to the 'not' for now [21:32:54] since we know it's catching them all [21:33:50] eileen If you look in wmf_civicrm_get_next_sched_contribution you can see that one limits to like RECURRING GLOBALCOLLECT% [21:34:07] I'd rather any paypal screwiness not make us thing globalcollect is broken [21:34:14] *think [21:35:27] ejegg: is it right now? [21:36:23] yep, that limit looks good! [21:36:32] Just parsing the rest of the sql... [21:37:52] There is still something I haven't figured out in the whole bug - but having this in place as we roll over to the first will give me some peace of mind [21:37:58] AndyRussG: sorry for the delay... was in a meeting. my conversation with him was IRL [21:41:06] dstrine: np, thx! [21:42:19] eileen: maybe 3 days either way, for folks who give on the 31st but were charged on Feb 28th ? [21:44:38] ejegg: ok - can do that [21:45:00] sorry for the annoying trickle-style CR. That's the only remaining thing I can see to change [21:45:22] (PS5) Eileen: Add check for stable data before running GC charges. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/313329 (https://phabricator.wikimedia.org/T144557) [21:45:31] ejegg: no that's fine [21:45:52] I think they way you think about it changes when you look at it from CR [21:46:06] ie. I started with a query & added criteria until it 'came clean' [21:46:22] whereas you are looking at the logic of the final query [21:46:38] yeah, definitely a flaw-seeking mentality sets in [21:47:02] nitpick mode on [21:47:55] good! [21:49:04] (CR) Ejegg: [C: 2] "Very reassuring to have this air raid siren installed" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/313329 (https://phabricator.wikimedia.org/T144557) (owner: Eileen) [21:49:38] ok, I'm actually going to try deploying the new donation queue consumer today [21:49:53] O_O [21:50:04] just going to drain the existing mirrored stuff from redis and start mirroring the queue from payments [21:50:30] ejegg: do you want me to cherry-pick that to deploy - or leave it to get scooped up in your deploy? [21:51:03] eileen: Let me deploy that one thing first? If you're off by the time I get it sorted I'll make sure it gets up too [21:51:55] (Merged) jenkins-bot: Add check for stable data before running GC charges. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/313329 (https://phabricator.wikimedia.org/T144557) (owner: Eileen) [21:53:03] ejegg: ok cool - I'll leave it to you - as long as I know it's going out before your 1 Oct I'm happy [21:55:55] Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Longterm fix + regression test for T144489 - https://phabricator.wikimedia.org/T144557#2678486 (Eileenmcnaughton) We are deploying an ambulance at the top of the cliff... [21:57:02] eileen: I don't understand the NOT BETWEEN--if a next_sched_date is bad, won't it have e.g. date_diff=1 [21:58:19] awight: she adds a month [21:59:04] also, one of us should write a patch to add source_ fields to the redis messages from smashpig [21:59:11] okay awesome [21:59:15] Yeah the date_diff is the difference between the scheduled date and last receive date + 1 month [21:59:27] eileen: ejegg: thanks for explaining! [21:59:30] so if it is outside of 3 days either side we worry [21:59:43] (CR) Awight: "Nice query!" (3 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/313329 (https://phabricator.wikimedia.org/T144557) (owner: Eileen) [22:01:19] The query is quite fast - my original query took over a minute so that was a bit too long! [22:01:31] ooh, i missed that group by desc bit. what does the desc do? [22:02:19] i'm writing that source fields patch... [22:04:00] ooh, huh. does that mean another wrapper around php-queue? [22:04:39] hmm, I'll just set 'em on the donations for now, I guess [22:05:26] ejegg: the DESC is probably not required [22:05:41] it orders the results - which mattered when building up the query [22:05:41] k [22:05:43] but not now [22:06:06] because I added AND next_sched_contribution_date > '2016-01-01' [22:06:56] hmm not running that fast actually - maybe that was caching that made it fast [22:07:20] if that's a problem I did wonder if it could be a separate jenkins job with a dependency [22:07:48] or that set a variable if it fails [22:08:43] Fundraising Sprint Stirring The Pot, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Spike: SPIKE: investigate level of completeness of civi deduped DB - https://phabricator.wikimedia.org/T146384#2659343 (Eileenmcnaughton) a:Eileenmcnaughton [22:12:14] eileen: Out of scope for this project, but I've been thinking about the ability for jobs to turn themselves off, e.g. in case they're emitting massive failmail [22:12:23] AndyRussG, I have a question about the site notice when you have a chance: https://www.mediawiki.org/wiki/Topic:T53wjhp29sgu6jms . [22:12:26] It would be nice to turn other jobs off, too [22:13:06] yes - I thought I saw jenkins allowing job dependencies [22:13:36] I think those are still in series, though? [22:13:46] The query doesn [22:14:10] ... doesn't have to be very fast IMO, the recurring Ingenico charge job only runs hourly, and most of the runs are gratuitous. [22:14:18] ah good [22:14:26] Only the first 3 or so per day actually process donations, the rest are placebos [22:14:49] well my main concern is to have something in place as we cross the 'month-line' in case i was wrong about it only affecting longer months… [22:14:51] We could codify that in the cronjob... and write another alarm to tell us if we missed any charges due to lack of job runs [22:15:39] I like the idea of running some jobs or similar to expose any data anomalies [22:15:59] Once the query is deployed, we can decide how to triage, I guess. If it's really slow like 30 min, then we can set the recurring charge job to run every 2hr or so [22:16:39] no - slow like (1 min 31.79 sec) [22:17:30] oh hah. nbd [22:18:37] (switching to vpn to help debug it) [22:24:48] ^ Or anyone else who's able to look at that.matt_flaschen [22:25:45] (PS1) Ejegg: WIP set source fields on messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313516 [22:25:45] Whoops, but if someone else has a chance to look at my site notice question it would be appreciated. [22:26:37] (CR) jenkins-bot: [V: -1] WIP set source fields on messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313516 (owner: Ejegg) [22:29:12] matt_flaschen: Good question, I vaguely remember a few inconclusive conversations about SiteNotice vs CentralNotice. Wikitech is not targetable in CentralNotice, however. Not sure whether there's a bug about that yet. [22:29:38] SiteNotice is a good choice, but lacking all of CN's user-administrable features. [22:29:59] Lemme read the code to confirm whether site-notice divs take precedence [22:30:53] Thanks, awight [22:32:27] (PS1) Ejegg: Add source fields to messages (smaller version) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313519 [22:32:42] awight: what do you think of ^^^ [22:33:07] we only need the source fields for donations/refunds/recurring, yes? [22:33:32] matt_flaschen: sitenotice is set in the skin, includes/skins/SkinTemplate.php line c. 460 for example. That means it's included in the cached, rendered page until invalidated, I believe? That would be a down side I wasn't aware of. [22:34:40] ejegg: Everything should have a trace history of itself, eventually, but yeah those queues especially because that's where source_* metadata is imported to the db. [22:35:01] awight, only for logged-out users, right? [22:35:53] matt_flaschen: yes, good point. Can you serve site notices to anons? [22:37:21] The docs say that anonnotice is displayed only to anons, but imply sitenotice is displayed to everyone. [22:38:17] awight, could we use CentralNotice for a dismissable text notice shown only to logged in users? It's probably okay to drop wikitech.wikimedia.org. I don't think there are many people who go there who don't at least occasionally go to MediaWiki.org. [22:38:27] https://www.mediawiki.org/wiki/Manual:Interface/Sitenotice [22:39:50] CentralNotice allows arbitrary HTML I think, so the text part should be fine, and I think dismissable is also there, so I guess remaining one is logged-in-only. [22:39:51] (CR) Awight: [C: 2] "Like a stiletto in the night--" (2 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313519 (owner: Ejegg) [22:40:39] (Merged) jenkins-bot: Add source fields to messages (smaller version) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313519 (owner: Ejegg) [22:40:55] ftw: https://www.mediawiki.org/wiki/Extension:DismissableSiteNotice [22:40:58] matt_flaschen: hi! sorry just saw the pings (my IRC client sometimes doesn't make noise at exactly the wrong time)... Yes CN allows arbitrary HTML. Can it do logged-in only? Yeah... K reading the original question now :) [22:41:10] I guess we might as well extend CentralNotice to target wikitech, then? [22:41:32] Probably it's kept somewhat isolated for reasons [22:41:36] awight, yeah, in theory, but that limitation is probably because it's a separate cluster? [22:41:41] So maybe not feasible/desirable. [22:43:04] yeah also not linked to CentralAuth, but usually the same login as for some dev tools... [22:43:31] It's not hard to set up CN and it could be configured to have its own separate banner/campaign administration controls [22:44:12] Basically in CN there are two roles for wikis: "subscribing" (get the banners) and "infrastructure" (control the banners) [22:44:25] Any subscribing wiki can point to only one infrastructure wiki [22:44:30] But they can also be the same wiki [22:44:43] AndyRussG, and the dismiss button is built-in, right? [22:44:45] (for example, meta) [22:45:11] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: New ability to target wikitech and smaller projects with CentralNotice - https://phabricator.wikimedia.org/T147036#2678639 (awight) [22:45:27] (PS1) Ejegg: Fix hostname under CRM [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/313526 (https://phabricator.wikimedia.org/T144034) [22:45:28] matt_flaschen: well that makes it sound better than it is. It's just a convenience button for adding the HTML to the banner... [22:45:55] awight: found your 'localhost' ^^ [22:46:11] matt_flaschen: ...well actually there's a bit more. There's built-in code to prevent dismissed banners from reappearing within a certain period [22:47:00] AndyRussG, so I have to choose something to get a Dismiss, or it's there by default? [22:47:06] also build-in stuff for limiting how many times a banner is shown to a user, and targeting languages [22:47:33] you click a button when you add or edit a banner and it inserts the code... Mmm the icon is also included [22:47:51] AndyRussG, got it, thank you. [22:49:25] this is what it adds:
Close
 [22:49:37] matt_flaschen: yw! [22:51:34] See where it says "Insert: Close button" just above the banner content text area: https://meta.wikimedia.beta.wmflabs.org/wiki/Special:CentralNoticeBanners/edit/browser_test_b1 [22:53:29] (CR) Awight: [C: 2] "/me jumps out of armchair in astonishment" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/313526 (https://phabricator.wikimedia.org/T144034) (owner: Ejegg) [22:54:40] Maybe the close button should be a mixin, enabled by default? [22:54:54] err, s/mixin/widget feature/ [22:55:12] (Merged) jenkins-bot: Fix hostname under CRM [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/313526 (https://phabricator.wikimedia.org/T144034) (owner: Ejegg) [22:58:21] awight: hmm I dunno, I've almost always wanted to tweak it even for simple banners [22:58:23] ejegg: Diving into your damaged UI now :) [22:58:27] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/313527 [22:59:05] AndyRussG: maybe the campaign feature provides an icon media uploader, css toggles... [22:59:31] ejegg: hi! looks like you deployed that refund fix. thanks! i don't see any more fail mail. [22:59:57] cwd yep! We should re-queue any refunds that got dumped into the damaged table [23:00:10] about to deploy another patch, to add the source_ fields to redis messages [23:00:37] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/313527 (owner: Ejegg) [23:00:38] ah, so we know which endpoint stuff came from? [23:00:41] AndyRussG, awight, thanks again. [23:00:50] (CR) Ejegg: [V: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/313527 (owner: Ejegg) [23:00:59] np :) [23:02:55] robla: hey! yt? dstrine mentioned that you might be able to help with the CentralNotice banner/message bug? many thx in advance!!!! [23:03:44] hi AndyRussG , yup, with big emphasis on the "might be" part :-) [23:03:49] !log updated SmashPig from 2169b71016e2deb1114655d0256d3286ff057943 to 3811f0f1c4bed1bd0b02264b5865ae36021cb275 [23:03:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:05:08] AndyRussG: I'm happy to use the highest bandwidth, most convenient option for you to help get me up to speed on the issue [23:05:20] * robla pulls up the task [23:05:54] * robla reads https://phabricator.wikimedia.org/T144952 a little more closely [23:06:00] !log enabled mirroring completed donations queue from payments-wiki [23:06:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:08:06] woot, getting source_ fields in redis, including some from DonationInterface [23:08:28] robla: cool thx! I think here is good! possibly other fr-tech folks might like to weigh in. awight did a bunch of the detailed investigating [23:08:42] also whatever option is good for you is great [23:09:22] cwd ok, let's see, now that we have mirroring from all the places, what's our order of operations? [23:09:28] ->turn off old QC [23:09:33] ->deploy new code [23:09:39] ->turn on new QC [23:09:39] ? [23:09:41] in essence it's pretty simple. https://github.com/wikimedia/mediawiki-extensions-CentralNotice/blob/731d248f7d7c9a265193500d41fced0fa691de39/includes/BannerRenderer.php#L135-L141 [23:09:46] Am I forgetting anything [23:09:59] ...is the CN code that's getting empty messages when it shouldn't [23:10:02] well, let me re-do that non-trivial merge first [23:10:49] robla: pretty sure it's something to do with MessageCache keys not being invalidated correctly across data centers [23:11:25] Earlier I was starting to debug into the core code to see where the MessageCache keys are actually written, and with what options [23:11:42] AndyRussG: yeah....ok, I think this seems like fair game to head over into #mediawiki-core and take the conversation there [23:11:56] the data center guess is because there were times that some users got the empty messages while some didn't [23:11:58] K! [23:13:18] (PS1) Ejegg: Revert "Revert "Mirror queue messages from wmf_audit"" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/313532 [23:13:20] (PS1) Ejegg: Revert "Revert "Mirror messages to new queues in GC audit"" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/313533 [23:14:12] (CR) Ejegg: [C: 2] Revert "Revert "Mirror queue messages from wmf_audit"" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/313532 (owner: Ejegg) [23:14:19] (Merged) jenkins-bot: Revert "Revert "Mirror queue messages from wmf_audit"" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/313532 (owner: Ejegg) [23:14:32] (CR) Ejegg: [C: 2] Revert "Revert "Mirror messages to new queues in GC audit"" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/313533 (owner: Ejegg) [23:14:57] (Merged) jenkins-bot: Revert "Revert "Mirror messages to new queues in GC audit"" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/313533 (owner: Ejegg) [23:18:21] ejegg: hehe sorry missed that [23:18:50] ejegg: yeah that sounds right to me. feeling confident enough to deploy the new consumer? [23:19:52] ejegg: cwd: what are the rollback steps? [23:20:12] Is it just configuration, or are there config + code steps? [23:20:41] can we deploy it first and just switch back and forth? [23:21:01] yah hopefully config and code can come in almost any order? [23:21:03] since we'll still be mirroring? [23:21:08] eso [23:21:20] awight: I believe rollback is: stop the QC, roll back code, start the QC [23:21:40] ok, that wfm [23:21:54] not everything can be configurable :) [23:22:09] even roll it back w/o stopping, why not... [23:22:34] but it sounds like that causes no issues but taunting duplicates [23:23:31] yeah... [23:23:41] ok, I'mma stop the QC to deploy, in any case [23:23:49] godspeed! [23:24:20] (PS1) Ejegg: Merge commit 'ae5a926512fb07c3a74ed5163e43f83dd93bc917' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/313535 [23:24:28] oh hey, anyone want to verify that one? [23:24:40] sure [23:24:42] I diffed it with origin/deployment, and it looks kosher. [23:24:50] just the QC and the audit mirroring [23:25:31] !log disabled donations queue consumer [23:25:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:25:41] ejegg: is deployment tied to a different vendor commit than master? [23:25:53] doubly revert is klassy :D [23:25:54] i keep getting weird git status and think that's it... [23:26:12] cwd crm shouldn't have a vendor submodule on master [23:26:32] ... and we deploy crm#deployment with crm/vendor#master [23:26:34] oooh, speaking of which, let me make sure that's in order [23:26:48] ejegg: sorry i meant smashpig [23:27:10] same deal there, SP#deployment with SP/vendor#master afaik [23:28:39] ejegg: The merge looks great. What filled you with doubt? [23:29:28] wait, smash pig master shouldn't have a vendor submodule? i've definitely got one [23:30:04] awight: nothing, just earlier attempts at the merge seemed harder [23:30:09] here goes... [23:30:29] (CR) Ejegg: [C: 2] Merge commit 'ae5a926512fb07c3a74ed5163e43f83dd93bc917' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/313535 (owner: Ejegg) [23:30:39] ejegg: cool. It read smoothly. [23:30:49] ack [23:30:52] almost... suspiciously so :) [23:30:57] composer.lock is messed up [23:32:13] hum, I see the markers. Yeah you did the right thing and chose the newer stamps. [23:32:16] oh derp, no, that's the conflict resolution [23:32:37] cool, moving on [23:34:38] (Abandoned) Ejegg: Merge commit 'ae5a926512fb07c3a74ed5163e43f83dd93bc917' into HEAD [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/312332 (owner: Ejegg) [23:34:44] (Abandoned) Ejegg: WIP set source fields on messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/313516 (owner: Ejegg) [23:36:00] (Merged) jenkins-bot: Merge commit 'ae5a926512fb07c3a74ed5163e43f83dd93bc917' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/313535 (owner: Ejegg) [23:37:52] sorry but... really no vendor submodule on SP/master? [23:38:22] !log updated CiviCRM from 6768613907598c52b0af1ffae35faa9078f15f63 to 637659ee8257562492405385d3fadaee53db998b [23:38:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:38:35] auspicious sign: rsync conf code 7crm [23:39:01] ok... let's dial that batch size down to 1 and try it out [23:39:20] ejegg: owow! [23:39:44] nice [23:39:51] red light :( [23:40:03] oh hah [23:40:10] segfault? [23:40:13] :P [23:40:13] drupal class cache, just gotta cc all [23:41:18] ok, cool, duplicate contribution as expected [23:41:21] let's try 5 more [23:42:09] whee! [23:44:33] still working through duplicates... [23:47:28] u could truncate the queue [23:47:41] ooh, paypal messages need order_id [23:47:48] or... you'd want to watch for not overshooting [23:47:49] hmm, should that be ct_id? [23:48:17] oh phooey, var_map already has custom: [23:49:57] ok, we're into fresh territory [23:54:44] cwd yeah, i'd like order_id to be set to ct_id for paypal [23:55:05] but i'm not sure how to map a thing to two destinations with var_map [23:55:21] yeah... [23:55:41] it would probably "just work" with duplicate keys [23:55:44] My head is in quick hack mode right now - you want to think of a proper way? [23:55:48] cwd you think? [23:56:02] if it's just iterating over them i don't see why not [23:56:12] but it might be less ugly to hack it into the job class [23:56:18] i'll look into it [23:56:36] err, yeah, it's using foreach [23:56:37] sure [23:59:08] cwd bummer, guess the yaml parser discards the duplicate key [23:59:30] ah that sucks [23:59:35] i'll hack it into the job [23:59:48] order_id => ct_id?