[00:05:33] <grrrit-wm>	 (PS6) Ejegg: Remove all traces of stomp pending queue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310665 (https://phabricator.wikimedia.org/T131274)
[00:06:00] <grrrit-wm>	 (CR) Ejegg: Remove all traces of stomp pending queue (2 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310665 (https://phabricator.wikimedia.org/T131274) (owner: Ejegg)
[00:09:39] <eileen>	 so awight az_CYRL is not one we would 'conclude' from our imports
[00:09:44] <grrrit-wm>	 (PS1) Ejegg: Add 'IF NOT EXISTS' to table creation [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310707
[00:09:52] <eileen>	 because it is not country based
[00:10:46] <eileen>	 ejegg: did you come to any conclusions on https://gerrit.wikimedia.org/r/#/c/310219/
[00:11:42] <ejegg>	 eileen: I haven't found anything stuffing contribution_tracking with invented language data
[00:11:57] <ejegg>	 but I'm still a little wary
[00:12:00] <eileen>	 :-)
[00:12:29] <eileen>	 I think you are right - we should not clean up civi data until that is merged
[00:12:32] <ejegg>	 I'd like to find a set of donors that came from e.g. an event and make sure they don't have conflicting language data
[00:12:59] <ejegg>	 or major gifts. actually, that's probably the easiest to check
[00:13:09] <ejegg>	 and the most likely to be updated by hand
[00:13:33] <eileen>	 cool
[00:18:58] <grrrit-wm>	 (CR) Awight: [C: 2] Remove all traces of stomp pending queue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310665 (https://phabricator.wikimedia.org/T131274) (owner: Ejegg)
[00:19:23] <awight>	 ejegg: want to squash the hair-raising hack?
[00:19:36] <ejegg>	 awight: oh sure, will do
[00:19:40] <grrrit-wm>	 (Merged) jenkins-bot: Remove all traces of stomp pending queue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310665 (https://phabricator.wikimedia.org/T131274) (owner: Ejegg)
[00:20:30] <awight>	 oof, I think that's it for today.  See you in 16hr (sounds so luxurious when you count it like that)
[00:21:19] <awight>	 ejegg: so would you prefer to leave that vagrant schema migration without a "unless" clause?
[00:21:48] <grrrit-wm>	 (PS5) Ejegg: Re-enable recurring GC tests, add more [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/306806 (https://phabricator.wikimedia.org/T141487)
[00:22:13] <grrrit-wm>	 (CR) Awight: [C: 2] Add 'IF NOT EXISTS' to table creation [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310707 (owner: Ejegg)
[00:22:41] <awight>	 ejegg: I was thinking we could provide a schema management framework outside of MediaWiki... did we chat about that already?
[00:22:42] <ejegg>	 awight: yeah, I think it would be tedious to update vagrant for every new script
[00:22:44] <awight>	 Some other day...
[00:22:47] <awight>	 yep
[00:22:52] <grrrit-wm>	 (Merged) jenkins-bot: Add 'IF NOT EXISTS' to table creation [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310707 (owner: Ejegg)
[00:23:01] <ejegg>	 i'm sure there's something simple out there to use
[00:23:20] <ejegg>	 or at least something we don't have to invent
[00:23:36] <ejegg>	 have a good 16 hrs!
[00:24:26] <awight>	 ok that's really it
[00:24:29] <awight>	 bye noe
[00:24:32] <awight>	 w
[00:32:39] <grrrit-wm>	 (PS3) Ejegg: Clean up antifraud queue usage [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310670 (https://phabricator.wikimedia.org/T131273)
[00:43:57] <grrrit-wm>	 (Abandoned) Ejegg: Include contribution_tracking_id in test message; set minimal $wgDonationInterface globals [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310677 (owner: Awight)
[00:44:07] <grrrit-wm>	 (Abandoned) Ejegg: Hair-raising hack to pass dummy gateway response code to DonationInterface [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310694 (owner: Awight)
[00:58:39] <wikibugs>	 Fundraising Sprint Octopus Untangling, Fundraising Sprint Pretending This Isn't Happening, Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, and 3 others: Silverpop export not capturing language changes in Civi (not sure if... - https://phabricator.wikimedia.org/T96410#2639052
[01:25:27] <wikibugs>	 Fundraising Sprint Octopus Untangling, Fundraising Sprint Pretending This Isn't Happening, Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, and 3 others: Silverpop export not capturing language changes in Civi (not sure if... - https://phabricator.wikimedia.org/T96410#2639075
[01:28:03] <grrrit-wm>	 (CR) Eileen: "I thought about your question and I think long term the answer may be that it is not required. However, I think it might still make sense " [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310192 (https://phabricator.wikimedia.org/T96410) (owner: Eileen)
[01:34:24] <grrrit-wm>	 (PS4) Eileen: Add Index to contact.preferred_language [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310192 (https://phabricator.wikimedia.org/T96410)
[01:34:26] <grrrit-wm>	 (PS3) Eileen: Update preferred language based on contribution_tracking. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310219 (https://phabricator.wikimedia.org/T96410)
[02:07:14] <wikibugs>	 Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, and 2 others: Investigate the discrepancy between pageview counts and banner impressions for WLM 2016 - https://phabricator.wikimedia.org/T145565#2639092 (AndyRussG...
[02:35:39] <grrrit-wm>	 (PS3) Eileen: Clean up legacy junk data from preferred_language field. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310218 (https://phabricator.wikimedia.org/T96410)
[02:35:41] <grrrit-wm>	 (PS1) Eileen: Remove handling for bad data (language = '_' as we are cleaning that up [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310724 (https://phabricator.wikimedia.org/T96410)
[02:36:29] <grrrit-wm>	 (PS2) Eileen: Remove handling for bad data (language = '_' as we are cleaning that up [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310724 (https://phabricator.wikimedia.org/T96410)
[02:36:31] <grrrit-wm>	 (PS4) Eileen: Clean up legacy junk data from preferred_language field. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310218 (https://phabricator.wikimedia.org/T96410)
[02:39:57] <grrrit-wm>	 (PS5) Eileen: Clean up legacy junk data from preferred_language field. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310218 (https://phabricator.wikimedia.org/T96410)
[02:40:54] <grrrit-wm>	 (PS6) Eileen: Clean up legacy junk data from preferred_language field. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310218 (https://phabricator.wikimedia.org/T96410)
[02:45:22] <wikibugs>	 Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, and 2 others: Investigate the discrepancy between pageview counts and banner impressions for WLM 2016 - https://phabricator.wikimedia.org/T145565#2639097 (LilyOfThe...
[02:47:33] <wikibugs>	 Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, and 2 others: Investigate the discrepancy between pageview counts and banner impressions for WLM 2016 - https://phabricator.wikimedia.org/T145565#2639098 (LilyOfThe...
[03:00:56] <wikibugs>	 Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Browser-Tests: CentralNotice: Intermittent unexplained browser test failures - https://phabricator.wikimedia.org/T145718#2639099 (AndyRussG)
[03:01:48] <wikibugs>	 Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Browser-Tests: CentralNotice: Intermittent unexplained browser test failures - https://phabricator.wikimedia.org/T145718#2639112 (AndyRussG)
[03:05:11] <wikibugs>	 Fundraising-Backlog: [Tracking - https://phabricator.wikimedia.org/T145719#2639113 (AndyRussG)
[03:07:16] <wikibugs>	 Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Tracking: [Tracking] CentralNotice: Unexplained banner display failures - https://phabricator.wikimedia.org/T145719#2639126 (AndyRussG)
[03:16:16] <wikibugs>	 Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Browser-Tests: CentralNotice: Intermittent unexplained browser test failures - https://phabricator.wikimedia.org/T145718#2639131 (AndyRussG)
[03:16:19] <wikibugs>	 Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, and 2 others: Investigate the discrepancy between pageview counts and banner impressions for WLM 2016 - https://phabricator.wikimedia.org/T145565#2639132 (AndyRussG)
[03:16:22] <wikibugs>	 Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Impressions slow to update on jaJP dsk campaign - https://phabricator.wikimedia.org/T145446#2639133 (AndyRussG)
[03:16:25] <wikibugs>	 Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2639134 (AndyRussG)
[03:16:28] <wikibugs>	 Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Tracking: [Tracking] CentralNotice: Unexplained banner display failures - https://phabricator.wikimedia.org/T145719#2639130 (AndyRussG)
[03:16:31] <wikibugs>	 Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, and 2 others: Banners not allocated correctly - https://phabricator.wikimedia.org/T144393#2639135 (AndyRussG)
[04:01:25] <grrrit-wm>	 (PS6) Cdentinger: [WIP] Payments API for PayPal [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/306601 (owner: Awight)
[04:01:35] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] [WIP] Payments API for PayPal [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/306601 (owner: Awight)
[04:17:19] <grrrit-wm>	 (PS7) Cdentinger: [WIP] Payments API for PayPal [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/306601 (owner: Awight)
[04:17:21] <grrrit-wm>	 (PS28) Cdentinger: WIP smashpig paypal listener [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/305761 (https://phabricator.wikimedia.org/T141654)
[05:15:59] <grrrit-wm>	 (PS1) Eileen: Only store the full country code if it is on our valid list. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310732 (https://phabricator.wikimedia.org/T96410)
[05:20:18] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] Only store the full country code if it is on our valid list. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310732 (https://phabricator.wikimedia.org/T96410) (owner: Eileen)
[06:21:36] <wikibugs>	 Fundraising Sprint Octopus Untangling, Fundraising Sprint Pretending This Isn't Happening, Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, and 3 others: Silverpop export not capturing language changes in Civi (not sure if... - https://phabricator.wikimedia.org/T96410#2639218
[06:22:26] <wikibugs>	 Fundraising Sprint Octopus Untangling, Fundraising Sprint Pretending This Isn't Happening, Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, and 3 others: Silverpop export not capturing language changes in Civi (not sure if... - https://phabricator.wikimedia.org/T96410#2639219
[12:53:24] <the-wub>	 morning all. France and Italy tests going up in about 10 minutes
[13:30:14] <the-wub>	 MBeat ppena_ : I'm pausing the France test, since I don't see any Adyen donations in civi yet and thought they should be there by now
[13:30:24] <the-wub>	 anything from your end?
[13:30:39] <MBeat>	 ty for the heads up, checking the console now for activity
[13:31:27] <MBeat>	 they should be getting through to Civi, the-wub , there’s about 70 transactions at the console
[13:33:26] <MBeat>	 the first one is still at authorized, from 2016-09-15 06:05:58 PDT and it looks like all the valid one are still at auth status
[13:34:36] <MBeat>	 good catch the-wub I will ask Adyen if there’s any reason these are not being sent for settle
[13:35:22] <the-wub>	 thanks MBeat. I'll leave the France test down. Italy is still going, everything looks fine there
[13:35:30] <MBeat>	 cool
[13:40:14] <MBeat>	 i pinged Adyen, do you think this is worth a Phab task, the-wub ?
[13:43:00] <the-wub>	 MBeat: probably, it would be a blocker for France if it doesn't get fixed
[13:43:57] <MBeat>	 ty, will make one. do you know what the usual time for an Adyen contrib to reach civi should be? like less than an hour, right?
[13:46:09] <the-wub>	 I think it was about 20 minutes in the previous tests
[13:57:00] <MBeat>	 ty
[14:09:39] <MBeat>	 it looks like we were set to have to manually capture all transactions, the-wub
[14:10:11] <MBeat>	 updated T146776 to see about switching that back
[14:10:15] <cwd>	 MBeat: is that from adyen?
[14:10:24] <cwd>	 like we were set up that way on their end?
[14:19:34] <MBeat>	 that’s from adyen, cwd - not sure what or where that setting changed
[14:20:50] <cwd>	 weird.  the adyen workflow could definitely be affected by the queue work which is why i ask, but not (to my knowledge) changing account settings
[14:27:20] <MBeat>	 cwd, I think we’re going to start manually pushing these through at the console, there are about 400 - unless there’s a way to submit a batch settle?
[14:28:35] <cwd>	 yikes, i'm guessing there's a way but i don't know :(
[14:28:41] <cwd>	 elliott would probably be the authority
[14:29:33] <MBeat>	 ty, I’ll ping him
[15:14:06] <wikibugs>	 Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Technical-Debt: CentralNotice should not call WikiPage::doEdit() - https://phabricator.wikimedia.org/T145738#2640824 (DStrine)
[15:28:46] <AndyRussG>	 morning dstrine! how's the special eventz going?
[15:29:49] <AndyRussG>	 I guess u saw this one? https://phabricator.wikimedia.org/T145719
[15:30:37] <AndyRussG>	 I wasn't sure if I should actually call it a "Tracking" bug because I might want to put it in a "Doing" column while I investigate all of those things at once....
[15:30:56] <AndyRussG>	 Maybe it's a "Pre-Possible-Task-Merge-Agglomerator" task
[16:01:06] <wikibugs>	 Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Browser-Tests: CentralNotice: Intermittent unexplained browser test failures - https://phabricator.wikimedia.org/T145718#2640974 (zeljkofilipin) a:zeljkofilipin I will take a look.
[16:01:39] <wikibugs>	 Fundraising-Backlog, Browser-Tests-Infrastructure, MediaWiki-extensions-CentralNotice, User-zeljkofilipin: CentralNotice: Intermittent unexplained browser test failures - https://phabricator.wikimedia.org/T145718#2640992 (zeljkofilipin)
[16:07:02] <wikibugs>	 Fundraising-Backlog, Browser-Tests-Infrastructure, MediaWiki-extensions-CentralNotice, User-zeljkofilipin: CentralNotice: Intermittent unexplained browser test failures - https://phabricator.wikimedia.org/T145718#2641048 (zeljkofilipin) p:Triage>Normal
[16:23:37] <wikibugs>	 Fundraising-Backlog, Browser-Tests-Infrastructure, MediaWiki-extensions-CentralNotice, User-zeljkofilipin: CentralNotice: Intermittent unexplained browser test failures - https://phabricator.wikimedia.org/T145718#2641165 (AndyRussG)
[16:24:07] <wikibugs>	 Fundraising-Backlog, Browser-Tests-Infrastructure, MediaWiki-extensions-CentralNotice, User-zeljkofilipin: CentralNotice: Intermittent unexplained browser test failures - https://phabricator.wikimedia.org/T145718#2639099 (AndyRussG) @zeljkofilipin Thanks!! :)
[17:03:28] <awight>	 ejegg: Looks bad!  Anything I can do to help?
[17:04:13] <awight>	 fr-tech: I'm in the hangout if there's any organizing to do...
[17:04:21] <ejegg>	 thanks awight, not sure yet
[17:04:28] <awight>	 kk
[17:04:43] <ejegg>	 we didn't deploy smashpig yesterday, did we?
[17:04:52] <cwd|brb>	 i am on the phone with verizon
[17:04:53] <awight>	 uh... /me checks server admin log.  I didn't
[17:04:55] <cwd|brb>	 be there asap
[17:05:03] <awight>	 cwd: lol, you should invite them
[17:05:19] <cwd>	 they are so infuriating
[17:05:29] <awight>	 ejegg: Last logged update was Aug 11:  20:25 ejegg: updated SmashPig from 5c180de6e424be10f9d61052c2c8dca7e0e825af to 4ba14a10518f80fe7e20fadc493d78c796c15921
[17:05:41] <ejegg>	 that long ago? huh
[17:05:46] * awight verifies deployment commit
[17:07:26] <awight>	 ejegg: Uh, oh.  Last deployment was actually Sep 8, my doing: e11af5793df9a0e2dbcf9ac138a8f2a3fa1bf574
[17:08:55] <ejegg>	 hmm, looks more like a pending db failure
[17:08:58] <ejegg>	 let's see
[17:09:20] <ejegg>	 dang, we should log those ids as we consume them
[17:09:24] <awight>	 I sure was messing with that.  I deployed the CRM change to delete from pending during fredge consumption
[17:10:01] <ejegg>	 oh phooey, are we sending 'complete' messages from the frontend?
[17:10:29] <awight>	 ejegg: logging: https://gerrit.wikimedia.org/r/#/c/310439/
[17:10:43] <awight>	 ah jesus, V-1
[17:10:52] <grrrit-wm>	 (PS2) Awight: Fix status label [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945)
[17:11:16] <ejegg>	 awight: test setup needs fixin
[17:12:12] <ejegg>	 that /shouldn't/ be it... we're sending 'pending:process' messages
[17:12:44] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] Fix status label [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945) (owner: Awight)
[17:13:26] <awight>	 ejegg: hehe, and I don't even have the right label for deleting "complete" messages
[17:13:49] <awight>	 ejegg: But to answer the earlier question, we're definitely sending "complete"s from DI
[17:14:00] <ejegg>	 awight: not for Adyen
[17:14:12] <ejegg>	 or at least, that's not what we're logging
[17:14:47] <awight>	 right
[17:17:53] <ejegg>	 let me check the pending db, see if anything new is getting in there
[17:18:26] <ejegg>	 hmm, consumer job is reporting 10-15 pulled from the queue per minute
[17:22:48] <ejegg>	 oh snap, bet those queue mirror settings are busted
[17:23:06] <ejegg>	 nothing but globalcollect in the pending db since 9/13
[17:24:05] <ejegg>	 ...which means we've probably got some amazon and astropay donations on infinite loop in the queue consumer
[17:24:08] <awight|ttalk>	 I did mess with mirroring yesterday
[17:25:04] <ejegg>	 lessee now, we all looked and thought it was ok
[17:25:45] <awight|ttalk>	 on second thought... I checked that activemq was not being filled.  I didn't check that Redis was getting data
[17:27:15] <ejegg>	 shoot, settings look fine
[17:27:36] <awight|ttalk>	 that's weird that Ingenico is working.  what are the source_*?
[17:28:38] <ejegg>	 let's see
[17:29:03] <ejegg>	 or maybe something's just deleting all the others...
[17:30:26] <ejegg>	 awight source is all DonationInterface version 996ca30076946c6148d9688a905c82e2e346e165, on the three main payments servers
[17:32:48] <ejegg>	 awight oh crap, it's my stupid mirroring code
[17:33:06] <ejegg>	 the 'main' queue gets ->set()
[17:33:18] <ejegg>	 and the 'mirrored' queue gets ->pushed
[17:33:34] <ejegg>	 hack coming up
[17:34:23] <ejegg>	 just clever enough to get into real trouble...
[17:35:29] <awight|ttalk>	 aaah nice find
[17:36:55] <grrrit-wm>	 (PS1) Ejegg: Push limbo message, don't set [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/310884 (https://phabricator.wikimedia.org/T145776)
[17:37:05] <ejegg>	 ^^^ oughtta do it
[17:38:15] <grrrit-wm>	 (CR) Awight: [C: 2] "Looks like we're all done with activemq pending and limbo..." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/310884 (https://phabricator.wikimedia.org/T145776) (owner: Ejegg)
[17:41:42] <grrrit-wm>	 (Merged) jenkins-bot: Push limbo message, don't set [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/310884 (https://phabricator.wikimedia.org/T145776) (owner: Ejegg)
[17:48:46] <ejegg>	 deploying
[17:49:02] <awight|ttalk>	 cool
[17:49:49] <grrrit-wm>	 (PS1) Ejegg: Push limbo message, don't set [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/310888 (https://phabricator.wikimedia.org/T145776)
[17:50:08] <grrrit-wm>	 (CR) Ejegg: [C: 2] Push limbo message, don't set [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/310888 (https://phabricator.wikimedia.org/T145776) (owner: Ejegg)
[17:52:18] <ejegg>	 awight: wait, that'll break the current orphan rectifier!
[17:52:32] <awight>	 No, I think it's okay
[17:52:37] <awight>	 gc-cc-limbo is Redis
[17:52:50] <ejegg>	 right, but set and push store differently
[17:52:55] <awight>	 fack
[17:53:39] <awight>	 That's horrible.
[17:53:45] <ejegg>	 yep...
[17:54:12] <awight>	 Well.  Shall I see if I can deploy the new rectifier?
[17:54:18] <ejegg>	 sure!
[17:54:21] <awight>	 ... and then we had a hard cutover ;)
[17:54:32] <ejegg>	 heh, we tried!
[17:55:19] <grrrit-wm>	 (Merged) jenkins-bot: Push limbo message, don't set [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/310888 (https://phabricator.wikimedia.org/T145776) (owner: Ejegg)
[17:56:32] <awight>	 !log Purging GC messages from pending with timestamp < '2016-09-15 13:44:55'
[17:56:37] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[17:58:09] <awight>	 ejegg: ah, we need that status label patch
[17:59:34] <ejegg>	 oh right
[17:59:54] <awight>	 meh I'm hacking it
[18:00:46] <ejegg>	 awight: still needs test setup fix
[18:01:41] <grrrit-wm>	 (PS3) Awight: Fix status label [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945)
[18:01:45] <awight>	 ejegg: yes, thanks for identifying that
[18:01:52] <awight>	 I have high hopes for ^
[18:04:58] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] Fix status label [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945) (owner: Awight)
[18:05:57] <grrrit-wm>	 (PS4) Awight: Fix status label [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945)
[18:07:49] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] Fix status label [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945) (owner: Awight)
[18:20:46] * awight facepalms
[18:22:21] <grrrit-wm>	 (PS5) Awight: Fix status label [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945)
[18:24:02] <grrrit-wm>	 (PS1) Awight: Merge master into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/310901
[18:24:15] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] Fix status label [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945) (owner: Awight)
[18:24:38] <ejegg>	 awight: that SmashPig deploy changes the name of pending-new, jfyi
[18:25:07] <awight>	 oy
[18:25:09] <ejegg>	 I'll update the smashpig settings on our deploy host
[18:25:15] <awight>	 mm can you?
[18:25:23] <ejegg>	 Jeff_Green: are you around to push a smashpig settings update soon?
[18:25:27] <awight>	 that.
[18:27:37] <grrrit-wm>	 (PS1) Awight: Update libs [wikimedia/fundraising/SmashPig/vendor] - https://gerrit.wikimedia.org/r/310903
[18:27:44] <grrrit-wm>	 (CR) Awight: [C: 2] Update libs [wikimedia/fundraising/SmashPig/vendor] - https://gerrit.wikimedia.org/r/310903 (owner: Awight)
[18:28:02] <grrrit-wm>	 (PS1) Awight: Update composer libs [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/310904
[18:28:13] <grrrit-wm>	 (CR) Awight: [C: 2] Merge master into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/310901 (owner: Awight)
[18:28:21] <grrrit-wm>	 (CR) Awight: [C: 2] Update composer libs [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/310904 (owner: Awight)
[18:28:33] <grrrit-wm>	 (Merged) jenkins-bot: Merge master into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/310901 (owner: Awight)
[18:30:41] <grrrit-wm>	 (PS1) Awight: Merge master into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/310905
[18:31:51] <Jeff_Green>	 ejegg: yep
[18:32:07] <grrrit-wm>	 (CR) Awight: [V: 2] Update libs [wikimedia/fundraising/SmashPig/vendor] - https://gerrit.wikimedia.org/r/310903 (owner: Awight)
[18:33:29] <ejegg>	 Jeff_Green: whenever you're ready, please go ahead and push the version that's in my homedir on the deploy host
[18:34:52] <grrrit-wm>	 (PS6) Awight: Fix status label [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945)
[18:35:10] <grrrit-wm>	 (CR) Awight: [C: 2] Merge master into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/310905 (owner: Awight)
[18:35:53] <Jeff_Green>	 ejegg: looking
[18:36:14] <grrrit-wm>	 (Merged) jenkins-bot: Merge master into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/310905 (owner: Awight)
[18:36:26] <grrrit-wm>	 (PS8) Cdentinger: [WIP] Payments API for PayPal [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/306601 (owner: Awight)
[18:36:28] <grrrit-wm>	 (PS1) Awight: update libs [extensions/DonationInterface/vendor] - https://gerrit.wikimedia.org/r/310906
[18:36:43] <grrrit-wm>	 (PS1) Awight: Update composer libs [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/310907
[18:36:45] <grrrit-wm>	 (CR) Awight: [C: 2] update libs [extensions/DonationInterface/vendor] - https://gerrit.wikimedia.org/r/310906 (owner: Awight)
[18:37:04] <grrrit-wm>	 (CR) Awight: [C: 2] Update composer libs [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/310907 (owner: Awight)
[18:37:23] <grrrit-wm>	 (CR) Awight: [V: 2] update libs [extensions/DonationInterface/vendor] - https://gerrit.wikimedia.org/r/310906 (owner: Awight)
[18:37:30] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] Fix status label [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945) (owner: Awight)
[18:38:40] <Jeff_Green>	 ejegg: just the rename of the pending queue, right?
[18:38:51] <ejegg>	 Jeff_Green: yep
[18:39:05] <ejegg>	 awight: I'mma inline-edit that to add .php to the filename in .info
[18:39:31] <grrrit-wm>	 (PS7) Awight: Fix status label [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945)
[18:39:54] <Jeff_Green>	 ejegg: deploying...
[18:39:58] <ejegg>	 thanks!
[18:40:02] <Jeff_Green>	 np
[18:40:13] <Jeff_Green>	 did you get to the bottom of the smashpig explosion earlier?
[18:40:20] <ejegg>	 There may be one failmail from the queue run right after it takes effect
[18:40:24] <Jeff_Green>	 ok
[18:40:52] <ejegg>	 Jeff_Green: yeah, we turned off writing to activemq pending, and that inadvertently changed how we write to redis pending
[18:41:05] <Jeff_Green>	 ok
[18:41:11] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] Fix status label [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945) (owner: Awight)
[18:41:23] <Jeff_Green>	 i did a round of package updates this AM, wanted to make sure I didn't kill something
[18:42:08] <ejegg>	 definitely not your fault!
[18:43:01] <grrrit-wm>	 (PS8) Awight: Fix status label [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945)
[18:43:26] <grrrit-wm>	 (Merged) jenkins-bot: Update composer libs [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/310907 (owner: Awight)
[18:43:39] <ejegg>	 !log updated smashpig consume pending job with new queue name
[18:43:45] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[18:44:04] <awight>	 ejegg: Thanks for all the help!
[18:44:27] <ejegg>	 awight: looks like you've got a lot of things in the air - can I do anything?
[18:44:49] <awight>	 I think I'm just going to deploy this new code one piece at a time, for starters
[18:44:55] <awight>	 SP, then DI, then CRM
[18:45:02] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] Fix status label [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945) (owner: Awight)
[18:45:28] <awight>	 ejegg: I suppose SP could go out now, if you're looking for stuff
[18:45:35] <awight>	 DI too, for that matter.
[18:45:51] <awight>	 I've still got my head in the crocodile's mouth on this CRM fixup
[18:45:53] <ejegg>	 k, will do
[18:46:08] <ejegg>	 yeah, let me take the other two deploys
[18:48:52] <awight>	 Ooh
[18:49:08] <ejegg>	 ah, typo in damaged job queue name is why those never showed up in the civi damaged message ui
[18:49:20] <ejegg>	 they're all in 'jobs-requests-damaged'
[18:49:30] <ejegg>	 instead of 'job-requests-damaged'
[18:50:06] <awight>	 what is this, Rails? :)
[18:50:20] <ejegg>	 Ooh what?
[18:50:35] <awight>	 just a light failmail shower
[18:50:59] * ejegg preps Rails patch to use CLDR plural rules
[18:53:05] * ejegg is leeching wifi in the PDX corporate business district, still the only person without tattoos
[18:53:24] <awight>	 bahaha
[18:53:36] <awight>	 It's okay, they assume you've got a big one under the shirt
[18:53:48] <ejegg>	 heh, yakuza all the way, baby
[18:55:04] <ejegg>	 ooh, SmashPig deploy needs more settings
[18:55:28] <awight>	 In case you need it, https://www.theguardian.com/world/2016/apr/18/woman-makes-fake-fingers-yakuza-japan-reformed-gangsters
[18:58:42] <ejegg>	 sweet
[19:06:01] <awight>	 ejegg: Finally, ready for CR: https://gerrit.wikimedia.org/r/#/c/310915/  https://gerrit.wikimedia.org/r/#/c/310439/
[19:06:58] <awight>	 ejegg: You want me to turn the jobs off?
[19:07:52] <ejegg>	 awight: looking at patch
[19:08:06] <awight>	 !log disabled fredge, donations, and banner-history consumers
[19:08:08] <ejegg>	 just adding more db settings for SP
[19:08:11] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[19:10:31] <grrrit-wm>	 (CR) Ejegg: "Oh shoot, this is a lot of the same stuff as I8a482363442c2" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945) (owner: Awight)
[19:11:16] <awight>	 ejegg: See PS9
[19:12:03] <awight>	 I'll rebase onto your patch
[19:12:15] <ejegg>	 oh thanks!
[19:12:21] <grrrit-wm>	 (CR) Ejegg: "Duplicates a lot of I8a482363442c2" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310915 (owner: Awight)
[19:12:41] <grrrit-wm>	 (CR) Awight: [C: 2] Update SmashPig & DonationInterface libraries [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310377 (owner: Ejegg)
[19:17:07] <grrrit-wm>	 (Merged) jenkins-bot: Update SmashPig & DonationInterface libraries [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310377 (owner: Ejegg)
[19:17:09] <grrrit-wm>	 (PS2) Awight: New testing configuration helper for queue and database [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310915
[19:17:11] <grrrit-wm>	 (PS10) Awight: Fix status label [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945)
[19:17:13] <grrrit-wm>	 (Merged) jenkins-bot: Move donation queue consumer off ActiveMQ [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/308800 (https://phabricator.wikimedia.org/T131277) (owner: Ejegg)
[19:17:15] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] Clean up DonationQueueConsumer [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310415 (https://phabricator.wikimedia.org/T131277) (owner: Ejegg)
[19:17:30] <ejegg>	 reviewing
[19:17:54] <awight>	 thx
[19:19:33] <grrrit-wm>	 (CR) Ejegg: "Looks useful, just got a duplicate entry in yaml" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310915 (owner: Awight)
[19:20:20] * ejegg witnesses a low-speed Jaguar-bollard collision
[19:21:01] <grrrit-wm>	 (PS3) Awight: New testing configuration helper for queue and database [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310915
[19:21:02] <ejegg>	 dude was remarkably calm about the fender damage
[19:21:13] <awight>	 Don't mess with immovable objects
[19:21:43] <grrrit-wm>	 (CR) Ejegg: [C: 2] New testing configuration helper for queue and database [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310915 (owner: Awight)
[19:22:55] <grrrit-wm>	 (PS11) Awight: Fix status label [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945)
[19:25:17] <awight>	 ejegg: Sorry to leave you my messes to clean up...
[19:25:29] <grrrit-wm>	 (Merged) jenkins-bot: New testing configuration helper for queue and database [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310915 (owner: Awight)
[19:25:40] <grrrit-wm>	 (PS1) Awight: Update libs [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/310920
[19:25:59] <ejegg>	 awight: oh, plenty of this mess is mine
[19:26:32] * awight nudges https://gerrit.wikimedia.org/r/#/c/310439/ onto the table
[19:28:01] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] Fix status label [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945) (owner: Awight)
[19:28:30] <ejegg>	 hrm?
[19:28:53] <awight>	 aack
[19:29:20] <grrrit-wm>	 (CR) Awight: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945) (owner: Awight)
[19:29:46] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] Update libs [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/310920 (owner: Awight)
[19:32:17] <ejegg>	 I thought Lucy wouldn't pull away the football that time
[19:32:23] <grrrit-wm>	 (PS1) Awight: Satisfy lint [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/310922
[19:32:44] <awight>	 Bart: 0, hamster: 1
[19:33:17] <grrrit-wm>	 (CR) Ejegg: [C: 2] "oh PHP..." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/310922 (owner: Awight)
[19:34:34] <grrrit-wm>	 (PS6) Awight: Re-enable recurring GC tests, add more [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/306806 (https://phabricator.wikimedia.org/T141487) (owner: Ejegg)
[19:34:36] <grrrit-wm>	 (Merged) jenkins-bot: Satisfy lint [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/310922 (owner: Awight)
[19:34:42] <grrrit-wm>	 (CR) Awight: [C: 2] Re-enable recurring GC tests, add more [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/306806 (https://phabricator.wikimedia.org/T141487) (owner: Ejegg)
[19:36:57] <awight>	 the-wub: MBeat: Just a warning that the donations queue consumer has been disabled for about an hour.  We expect to have it running again soon...
[19:37:08] <MBeat>	 ty awight
[19:38:18] <grrrit-wm>	 (Merged) jenkins-bot: Re-enable recurring GC tests, add more [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/306806 (https://phabricator.wikimedia.org/T141487) (owner: Ejegg)
[19:39:10] <ejegg>	 awight: going to cherry-pick around the donation queue consumer switchover?
[19:39:30] <ejegg>	 Seems to have cascaded into master as soon as that lib update merged
[19:40:00] <grrrit-wm>	 (PS3) Awight: FIXME [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/305580 (owner: Ejegg)
[19:40:06] <grrrit-wm>	 (CR) Awight: [C: 2] FIXME [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/305580 (owner: Ejegg)
[19:40:29] <grrrit-wm>	 (Abandoned) Awight: Update composer libs [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/309506 (owner: Awight)
[19:40:41] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] FIXME [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/305580 (owner: Ejegg)
[19:40:49] <awight>	 ejegg: thanks for the heads-up!
[19:40:49] <ejegg>	 wat
[19:40:53] <awight>	 hahaha
[19:40:58] <awight>	 fix *this*
[19:41:19] <awight>	 Seems to be happening half the time, CI issues
[19:41:30] <awight>	 stderr: 'warning: failed to remove sites/default/files/
[19:41:42] <awight>	 ah, actually ERROR 1269 (HY000) at line 1: Can't revoke all privileges for one or more of the requested users
[19:41:56] <awight>	 huh, or both failures.
[19:41:57] <awight>	 anyway
[19:42:04] <grrrit-wm>	 (CR) Awight: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/305580 (owner: Ejegg)
[19:42:11] <ejegg>	 yeah, it's pretty wild that we're setting up and tearing down a whole CMS and CRM on every test
[19:42:18] <awight>	 i love it
[19:44:49] <AndyRussG|mob_>	 fr-tech late 4 standup rrrrg traffic sorry!!
[19:46:41] <grrrit-wm>	 (PS1) Awight: Reject isRequeue messages after expiry [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310926 (https://phabricator.wikimedia.org/T142028)
[19:46:43] <grrrit-wm>	 (PS1) Awight: Refactor common duplicate-checking code [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310927
[19:46:45] <grrrit-wm>	 (PS1) Awight: Update .gitignore [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310928
[19:46:47] <grrrit-wm>	 (PS1) Awight: Remove wmf_zendesk_reports [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310929 (https://phabricator.wikimedia.org/T143142)
[19:46:50] <grrrit-wm>	 (PS1) Awight: Clean up legacy junk data from contribution_tracking.language field. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310930 (https://phabricator.wikimedia.org/T96410)
[19:46:51] <grrrit-wm>	 (PS1) Awight: Update SmashPig & DonationInterface libraries [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310931
[19:46:53] <grrrit-wm>	 (PS1) Awight: Fix status label [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310932 (https://phabricator.wikimedia.org/T143945)
[19:46:55] <grrrit-wm>	 (PS1) Awight: Re-enable recurring GC tests, add more [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310933 (https://phabricator.wikimedia.org/T141487)
[19:47:39] <grrrit-wm>	 (CR) Awight: [C: 2] Reject isRequeue messages after expiry [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310926 (https://phabricator.wikimedia.org/T142028) (owner: Awight)
[19:47:41] <grrrit-wm>	 (CR) Awight: [C: 2] Refactor common duplicate-checking code [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310927 (owner: Awight)
[19:47:43] <grrrit-wm>	 (CR) Awight: [C: 2] Update .gitignore [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310928 (owner: Awight)
[19:47:45] <grrrit-wm>	 (CR) Awight: [C: 2] Remove wmf_zendesk_reports [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310929 (https://phabricator.wikimedia.org/T143142) (owner: Awight)
[19:47:51] <grrrit-wm>	 (CR) Awight: [C: 2] Clean up legacy junk data from contribution_tracking.language field. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310930 (https://phabricator.wikimedia.org/T96410) (owner: Awight)
[19:47:56] <grrrit-wm>	 (CR) Awight: [C: 2] Update SmashPig & DonationInterface libraries [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310931 (owner: Awight)
[19:48:06] <grrrit-wm>	 (CR) Awight: [C: 2] Fix status label [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310932 (https://phabricator.wikimedia.org/T143945) (owner: Awight)
[19:48:10] <grrrit-wm>	 (CR) Awight: [C: 2] Re-enable recurring GC tests, add more [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310933 (https://phabricator.wikimedia.org/T141487) (owner: Awight)
[19:50:36] <grrrit-wm>	 (Merged) jenkins-bot: Reject isRequeue messages after expiry [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310926 (https://phabricator.wikimedia.org/T142028) (owner: Awight)
[19:50:38] <grrrit-wm>	 (Merged) jenkins-bot: Refactor common duplicate-checking code [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310927 (owner: Awight)
[19:50:40] <grrrit-wm>	 (Merged) jenkins-bot: Update .gitignore [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310928 (owner: Awight)
[19:50:42] <grrrit-wm>	 (Merged) jenkins-bot: Remove wmf_zendesk_reports [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310929 (https://phabricator.wikimedia.org/T143142) (owner: Awight)
[19:50:44] <grrrit-wm>	 (Merged) jenkins-bot: Clean up legacy junk data from contribution_tracking.language field. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310930 (https://phabricator.wikimedia.org/T96410) (owner: Awight)
[19:50:46] <grrrit-wm>	 (Merged) jenkins-bot: Update SmashPig & DonationInterface libraries [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310931 (owner: Awight)
[19:50:48] <grrrit-wm>	 (Merged) jenkins-bot: Fix status label [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310932 (https://phabricator.wikimedia.org/T143945) (owner: Awight)
[19:50:50] <grrrit-wm>	 (Merged) jenkins-bot: Re-enable recurring GC tests, add more [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310933 (https://phabricator.wikimedia.org/T141487) (owner: Awight)
[19:57:29] <grrrit-wm>	 (PS1) Awight: update libs [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/310939
[19:57:38] <grrrit-wm>	 (CR) Awight: [C: 2] update libs [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/310939 (owner: Awight)
[19:57:52] <grrrit-wm>	 (PS1) Reedy: Remove old calls to WikiPage::doEdit() [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/310940 (https://phabricator.wikimedia.org/T145738)
[19:57:59] <grrrit-wm>	 (CR) Awight: [V: 2] update libs [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/310939 (owner: Awight)
[19:58:20] <grrrit-wm>	 (PS1) Awight: Update composer libs [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310941
[19:59:22] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] Remove old calls to WikiPage::doEdit() [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/310940 (https://phabricator.wikimedia.org/T145738) (owner: Reedy)
[19:59:53] <grrrit-wm>	 (PS1) Cdentinger: Test failure path [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310942
[20:00:38] <grrrit-wm>	 (PS1) Awight: update more libs [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/310943
[20:00:47] <grrrit-wm>	 (CR) Awight: [C: 2 V: 2] update more libs [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/310943 (owner: Awight)
[20:01:03] <grrrit-wm>	 (PS2) Awight: Update composer libs [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310941
[20:01:14] <grrrit-wm>	 (CR) Awight: [C: 2] Update composer libs [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310941 (owner: Awight)
[20:04:37] <grrrit-wm>	 (CR) Awight: [V: 2] Update composer libs [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/310941 (owner: Awight)
[20:04:46] <grrrit-wm>	 (PS2) Cdentinger: WIP Test failure path [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310942
[20:10:02] <ejegg>	 ok, deploying smashpig
[20:11:49] <ejegg>	 !log updated SmashPig from e11af5793df9a0e2dbcf9ac138a8f2a3fa1bf574 to 12a7b78cf280f96ff3d2314abb2a9290506ae454
[20:11:57] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[20:15:58] <ejegg>	 so far so good
[20:19:46] <ejegg>	 !rolled back SmashPig to e11af5793df9a0e2dbcf9ac138a8f2a3fa1bf574
[20:19:52] <ejegg>	 think it's just a settings thing
[20:27:23] <ejegg>	 phooey, i don't get it
[20:41:46] <ejegg>	 I don't see data-store/verified overridden in the config, and I can totally create the MultiQueueWriter locally with the defaults
[20:45:50] <ejegg>	 ugh, some php5.3 reflection class thing maybe?
[20:47:19] <ejegg>	 no, that's been the same code all along
[20:47:41] <ejegg>	 so nodeExists is acting funny?
[20:55:13] <ejegg>	 awight: do you have any idea why we can't find the args for MultipQueueWriter constructor for the verified thing?
[20:56:03] <ejegg>	 I'm just not seeing anything that makes sense
[21:03:37] <awight>	 sorry
[21:03:50] <awight>	 was stealing leftovers
[21:04:34] <ejegg>	 mmm, leftovers
[21:04:45] <grrrit-wm>	 (CR) Reedy: "recheck" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/310940 (https://phabricator.wikimedia.org/T145738) (owner: Reedy)
[21:04:45] <awight>	 ejegg: Which error you taling about?   WD php: SmashPig\Core\SmashPigException: Dissimilar types cannot be   [31;40m [1m[error] [0m
[21:04:48] <awight>	 merged at configuration node data-store/pending/inst-args/0. in
[21:04:51] <awight>	 ?
[21:05:32] <ejegg>	 nope, Last chance error handler fired. (2) Missing argument 1 for SmashPig\Core\DataStores\MultiQueueWriter::__construct()
[21:05:59] <awight>	 gotcha
[21:06:02] <ejegg>	 it was an amazon capture completed, so it should have been looking for the verified data-store
[21:06:23] <ejegg>	 which is only specified in defaults, not in the override
[21:06:37] <ejegg>	 and changed from a straight-up StompDataStore to a MultiQueueWriter
[21:07:19] <ejegg>	 I added a minimal test locally that just mocks verified-stomp and creates the data-store/verified object, and there's no issue there
[21:07:40] <ejegg>	 attempting to try on the CRM box with a checkout in my homedir
[21:08:43] <awight>	 Creepy that it's been running the defaults.
[21:09:09] <awight>	 How were we not overriding the parameters to mirror to Redis?
[21:09:10] <ejegg>	 nah, defaults are fine for that one - no connection info
[21:09:32] <ejegg>	 we haven't turned on redis mirroring for donations yet
[21:09:48] <awight>	 Ah gotcha
[21:12:31] <ejegg>	 uhhh odd
[21:12:46] <awight>	 ejegg: Did you see this one?  Undefined variable: backends
[21:13:19] <ejegg>	 I just edited out all the sending/receiving from the TestDatastore and tried running it just to create the verified object
[21:13:28] <ejegg>	 the first time I ran it - no output
[21:13:30] <awight>	 {"config":{}}
[21:13:39] <ejegg>	 awight Just now?
[21:13:58] <awight>	 No this around the time of your original error, Sep 15 20:16:31
[21:14:01] <ejegg>	 Right, that's the same issue - missing argument 1 is $backends
[21:14:06] <awight>	 k
[21:14:20] <ejegg>	 anyway, I ran php Maintenance/TestDatastore.php verified
[21:14:22] <ejegg>	 no output
[21:14:34] <ejegg>	 I ran php Maintenance/TestDatastore.php verified --config-node amazon
[21:14:37] <ejegg>	 no output
[21:14:46] <ejegg>	 I ran php Maintenance/TestDatastore.php verifiedx --config-node amazon
[21:15:01] <ejegg>	 and got an exception that verifiedx/class does not exist
[21:15:19] <ejegg>	 then I ran php Maintenance/TestDatastore.php verified --config-node amazon again
[21:15:37] <ejegg>	 and got  Attempting connection to STOMP server 'tcp://silicon.frack.eqiad.wmnet:61613'
[21:15:43] <ejegg>	 STOMP server connection success.
[21:15:46] <ejegg>	 Done
[21:15:51] <awight>	 fancy ;)
[21:15:52] <ejegg>	 So... caching issue?
[21:16:04] <awight>	 I don't think we have any caching
[21:16:18] <ejegg>	 configuration is cached in apc, right?
[21:16:24] <awight>	 Do we have APC?
[21:16:35] <ejegg>	 iono...
[21:16:42] <awight>	 aha right Configuration
[21:16:45] <awight>	 lemme check...
[21:17:15] <ejegg>	 looking like disabled?
[21:17:31] <awight>	 php -r 'var_export(extension_loaded( 'apc' ));'
[21:17:32] <awight>	 true
[21:17:44] <ejegg>	 ok
[21:18:53] <awight>	 php -r 'var_export(apc_fetch("smashpig-settings-amazon"));'
[21:18:54] <awight>	 false
[21:18:58] <awight>	 donno what that's about
[21:19:22] <ejegg>	 yeah, phpinfo gives 'APC Support => disabled'
[21:19:33] <awight>	 wat.  Why is extension_loaded then...
[21:19:41] <awight>	 Maybe PHP config
[21:20:39] <ejegg>	 huh, that makes it look like it IS enabled. apc.enabled=1 in apc.ini
[21:20:55] <awight>	 yep for both CLI and apache
[21:20:59] <ejegg>	 can't think why else I'd get a nondeterministic result from that test
[21:21:29] <awight>	 you could hack in a Logger statement to see the config
[21:22:18] <cwd>	 are we talking about smashpig caching config?
[21:22:19] <ejegg>	 awight owait, I'm testing on the wrong box. need listener, not crm
[21:22:22] <awight>	 cwd: yeah
[21:22:45] <cwd>	 i saw that and wondered, is it necessary for performance?  sounds like it could be buggy
[21:22:49] <awight>	 ejegg: That explains it.  and fwiw yeah APC is disabled on the CRM box
[21:23:10] <awight>	 cwd: It's a totally silly microoptimization that adds terrifying, nondeterministic complexity :)
[21:23:16] <awight>	 we need more of that
[21:23:43] <awight>	 brb
[21:23:51] <ejegg>	 awight then I'm totally confused as to why that test gave me no output the first time, then reported success the second time
[21:23:53] <cwd>	 hehe ok that explains it
[21:28:12] <awight>	 ejegg: no output usually means PHP swallowed a fatal error...
[21:28:48] <ejegg>	 awight yeah I figured it was using some part of old config the first time and died like the listener did in the logs
[21:29:17] <ejegg>	 then something made it use the right config
[21:29:53] <awight>	 Any ideas why the config wasn't actually cached?
[21:30:19] <awight>	 pretty sure apc_fetch("smashpig-settings-amazon") should have pulled something
[21:31:05] <awight>	 We're not storing with a TTL, so it should be there
[21:31:20] <awight>	 ah--maybe php vs cli PHP?
[21:31:23] <awight>	 err apache vs cli
[21:31:28] <ejegg>	 err, but I was initially testing on the CRM box
[21:32:05] <ejegg>	 so I have NO idea how it could have changed between runs if APC is off
[21:33:03] <awight>	 but the cache should have been populated on the listener by normal jobs
[21:33:21] <ejegg>	 hmm - for 5 minutes, right?
[21:33:35] <ejegg>	 hasn't been a lot of Amazon IPN traffic today
[21:33:39] <awight>	 No TTL I discovered
[21:33:44] <awight>	 http://php.net/apc_store
[21:33:47] <ejegg>	 ooh
[21:34:08] <awight>	 We have our *own* cache invalidation mechanism cos S+M
[21:34:31] <ejegg>	 :P
[21:35:11] <ejegg>	 uh, smashpig-settings-default is also giving false
[21:36:03] <awight>	 yep
[21:36:14] <awight>	 This is why it's unfortunate that we can't log from Configuration
[21:36:32] <awight>	 Logger should have a fallback mode.
[21:36:45] <awight>	 anyway.
[21:37:54] <grrrit-wm>	 (CR) Krinkle: choiceData always checks for mw.centralNotice object (2 comments) [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/285674 (https://phabricator.wikimedia.org/T133765) (owner: AndyRussG)
[21:38:04] <ejegg>	 ok, i have a fresh copy of sp like we just tried to deploy in my homedir on the listener box, and have edited TestDatastore to only connect
[21:38:07] <ejegg>	 let's see
[21:38:23] <awight>	 So, I don't understand how MultiQueueWriter can deal with the methods used in TestDataStore.
[21:38:49] <ejegg>	 I commented out everything except the ->object call and the Done log line
[21:38:54] <awight>	 k
[21:39:16] <ejegg>	 well huh, worked like a charm first time
[21:39:44] <ejegg>	 still not seeing anything cached in apc
[21:39:58] <ejegg>	 but I kindof want to just deploy again and try an amazon donation
[21:40:14] <ejegg>	 I mean, I hate this uncertainty
[21:40:15] <awight>	 ejegg: http://stackoverflow.com/questions/3722595/how-to-share-apc-cache-between-apache-and-php-cli
[21:40:28] <awight>	 (only answers the Why empty?)
[21:40:40] <awight>	 +1 turn on the live fail
[21:40:41] <ejegg>	 ah
[21:41:02] <awight>	 Just cos the symptoms might be more interesting ;)
[21:41:21] <grrrit-wm>	 (CR) Krinkle: [C: 2] jQuery usage tweaks (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/291917 (owner: Ejegg)
[21:41:28] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] jQuery usage tweaks [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/291917 (owner: Ejegg)
[21:41:36] <grrrit-wm>	 (PS3) Krinkle: jQuery usage tweaks [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/291917 (owner: Ejegg)
[21:41:51] <ejegg>	 !log updated SmashPig from e11af5793df9a0e2dbcf9ac138a8f2a3fa1bf574 to 12a7b78cf280f96ff3d2314abb2a9290506ae454
[21:42:10] <grrrit-wm>	 (PS5) Krinkle: Remove 'UnitTestList' hook [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/303261 (https://phabricator.wikimedia.org/T142120) (owner: Reedy)
[21:42:24] <awight>	 ejegg: I'm ready to try the new CRM... shall we?
[21:42:30] <awight>	 or you want to iron out the SP fail first
[21:42:40] <ejegg>	 let me just see if this works
[21:43:00] <awight>	 Take your time.  I'm a fan of isolating one moving piece at a time, too
[21:43:10] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] jQuery usage tweaks [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/291917 (owner: Ejegg)
[21:43:12] <awight>	 Just feeling reckless today :p
[21:43:43] <ejegg>	 bah, same error
[21:43:54] <awight>	 with the production job?
[21:43:59] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] Remove 'UnitTestList' hook [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/303261 (https://phabricator.wikimedia.org/T142120) (owner: Reedy)
[21:43:59] <awight>	 ouch
[21:44:06] <ejegg>	 it's right in the listener, not a job
[21:44:12] <ejegg>	 but even caching makes no sense
[21:44:30] <ejegg>	 it's trying to create a MultiQueueWriter, which is new
[21:44:55] <ejegg>	 it's just not sending the constructor params
[21:45:12] <ejegg>	 aaaauuuugh
[21:46:09] <awight>	 Logging...
[21:47:20] * awight curses APC
[21:48:16] <ejegg>	 grumble... still busted on second test donation
[21:48:36] <eileen>	 ppena - A kiwi is person who comes from NZ - or a native bird - all-staff emails about eating kiwis for breakfast are scary :-)
[21:49:12] <eileen>	 (we call the fruit kiwifruit)
[21:49:13] <grrrit-wm>	 (CR) Krinkle: [C: 1] "recheck" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/310940 (https://phabricator.wikimedia.org/T145738) (owner: Reedy)
[21:49:37] <awight>	 and kiwiberries are another species entirely ;) https://en.wikipedia.org/wiki/Kiwifruit#/media/File:Actinidia_fruits.jpg
[21:50:39] <awight>	 ejegg: If you think it's caches, let's try "touch"ing the config files?
[21:50:55] <ejegg>	 awight: how could this even be apc though?
[21:51:07] <ejegg>	 thinking it can't be, if we're getting the new class name
[21:51:09] <awight>	 I don't think it is
[21:51:15] <awight>	 k
[21:53:13] <ejegg>	 going to try an adyen donation to see if the jobs queue works from the frontend
[21:53:44] <awight>	 Configs and code look good to me
[21:54:50] <ejegg>	 awight: it's dying on the StompDataStore too
[21:55:11] <ejegg>	 something is f'ed with the inst-args / constructor-parameters switchover
[21:55:26] <awight>	 yep
[21:56:32] <awight>	 I really want to dump the apc contents for fun...
[21:56:33] <ejegg>	 argh too many changes at once :P
[21:56:47] <awight>	 CAnnot figure out how to get into the apache-php mem space though
[21:57:14] <PPena>	 eileen LOL! So Sorry! Love you guys too much to cannibalize.
[21:57:20] <awight>	 Most interested in the "configTimes"
[21:57:27] <awight>	 <_<
[21:58:06] <ejegg>	 could apc be caching the actual code?
[21:58:23] <ejegg>	 like, it's still looking for inst-args when we've moved on?
[21:58:27] <awight>	 no
[21:58:31] <awight>	 well--maybe it could be
[21:58:35] <awight>	 but that's pretty well tested
[21:59:26] <eileen>	 PPena: phew
[22:03:04] <grrrit-wm>	 (PS1) Awight: Live hack to dump configuration [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/311036
[22:03:07] <awight>	 ejegg: Shall we? ^
[22:03:39] <ejegg>	 eek
[22:04:05] <awight>	 hmm, or that will dump an infinite number of passwords.  nvm
[22:05:32] <grrrit-wm>	 (PS2) Awight: Live hack to dump configuration [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/311036
[22:05:54] <grrrit-wm>	 (PS1) Ejegg: Live hack: try restoring inst-args for one thing [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/311039
[22:06:04] <ejegg>	 awight: or that one ^^
[22:06:46] <ejegg>	 ok, let's do yours first
[22:07:08] <grrrit-wm>	 (CR) Ejegg: [C: 2] "worth a shot..." [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/311036 (owner: Awight)
[22:07:41] <ejegg>	 oh wait, that's on top of an unmerged lib update
[22:07:47] <awight>	 ejegg: How about a hack to Configuration to read from constructor-parameters if it's available, then check inst-args?
[22:08:15] <grrrit-wm>	 (PS3) Awight: Live hack to dump configuration [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/311036
[22:08:59] <ejegg>	 awight: it's getting the new value for class... how would it be not getting constructor-parameters?
[22:09:13] <ejegg>	 my hunch is still somehow old code
[22:09:22] <grrrit-wm>	 (PS1) Awight: Backwards compatibility with old config key [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/311040
[22:09:31] <awight>	 ejegg: yeah agreed about that
[22:09:37] <awight>	 however, I can't suspect old code
[22:09:43] <awight>	 I don't think APC has that bug
[22:09:50] <cwd>	 i hate to bail while you guys are battling this but i said we'd try to get an early start
[22:09:56] <awight>	 have a blast!
[22:10:34] <ejegg>	 !log updated SmashPig from 12a7b78cf280f96ff3d2314abb2a9290506ae454 to af19422065c08669269179019fea1e7c208d8a7e
[22:10:37] <cwd>	 thanks!  good luck with this one and feel free to hack on the paypal listener if it gets in the way.  hopefully it's close
[22:10:41] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[22:10:48] <cwd>	 have a good weekend!
[22:10:50] <awight>	 ejegg: I hear 'apache2ctl graceful' will clear the opcode cache though
[22:11:32] <ejegg>	 oh yeah?
[22:12:06] <ejegg>	 wtf.... inst-args persist
[22:12:23] <awight>	 https://bugs.php.net/bug.php?id=56947
[22:12:27] <ejegg>	 ok, let's do your back-compat patch
[22:12:35] <awight>	 you getting logs now?
[22:12:39] <awight>	 looking
[22:12:43] <ejegg>	 yeah...
[22:12:50] <awight>	 WAT
[22:12:55] <ejegg>	 it's a mix of old and new!
[22:12:59] <awight>	 wat
[22:13:11] <awight>	 it's gotta be coming from an override?
[22:13:14] <ejegg>	 srsly, this makes zero sense
[22:13:32] <awight>	 apc clear would knock out the config cache, too
[22:13:56] <ejegg>	 but how is the override deleting constructor-parameters?
[22:14:14] <ejegg>	 Shouldn't it just add another key?
[22:14:51] <ejegg>	 verified is nowhere in the override
[22:15:13] <ejegg>	 this is totally bonkers
[22:16:07] <awight>	 If Jeff_Green's gone for the night, we need to clear that cache ourselves
[22:16:15] <awight>	 aka not use it.
[22:16:56] <ejegg>	 How on earth could it even break that way?
[22:16:58] <grrrit-wm>	 (PS1) Awight: Livehack to not use APC [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/311043
[22:17:25] <awight>	 Maybe... a listener read from the config file during rsync?
[22:17:45] <ejegg>	 seems highly unlikely to have happened twice...
[22:17:48] <awight>	 This is turning into quite the choose your own adventure
[22:17:56] <grrrit-wm>	 (CR) Ejegg: [C: 2] Livehack to not use APC [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/311043 (owner: Awight)
[22:18:32] <grrrit-wm>	 (Merged) jenkins-bot: Livehack to not use APC [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/311043 (owner: Awight)
[22:19:19] <ejegg>	 !log updated SmashPig from af19422065c08669269179019fea1e7c208d8a7e to db68be988194c960aebca691d0fd8e6a6d24246a
[22:19:25] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[22:19:35] <awight>	 I'm buckled in
[22:20:29] <awight>	 SPCID-0785269755 | {"class":"SmashPig\\Core\\DataStores\\MultiQueueWriter","constructor-parameters":[["verified-stomp"]]} |  |
[22:20:32] <awight>	 wth man
[22:20:35] <ejegg>	 yeah....
[22:20:35] <awight>	 ok well.
[22:20:40] <awight>	 to the moon!
[22:20:54] <ejegg>	 new apc feature: partial cache updates
[22:20:57] <awight>	 hehe
[22:21:26] <grrrit-wm>	 (PS1) Awight: Revert "Live hack to dump configuration" [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/311045
[22:21:40] <ejegg>	 ok, so... crm next, then DI?
[22:21:47] <awight>	 My rsync theory doesn't hold water...
[22:22:01] <awight>	 maybe DI then crm cos of dependencies?
[22:22:12] <awight>	 lemme create the paymentswiki patch
[22:23:12] <ejegg>	 phew, I need lunch
[22:23:12] <grrrit-wm>	 (PS1) Awight: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/311046
[22:23:21] <grrrit-wm>	 (CR) Awight: [C: 2] Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/311046 (owner: Awight)
[22:23:49] <awight>	 ejegg: sorry to hear you skipped lunch while I was walking around obliviously!
[22:23:55] <ejegg>	 awight: i'm monitoring IRC, back soon
[22:24:00] <awight>	 I can botch things from here
[22:28:02] <awight>	 !log update paymentswiki from 996ca30076946c6148d9688a905c82e2e346e165 to 609c1e5f03ddcd9d72f562d7911cf77cc459371b
[22:28:09] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[22:29:58] <awight>	 !log rollback paymentswiki from 609c1e5f03ddcd9d72f562d7911cf77cc459371b to 996ca30076946c6148d9688a905c82e2e346e165
[22:30:07] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[22:32:51] <AndyRussG>	 Oooh CN Gerrit storm! Thx Krinkle, Reedy, ejegg|food :)
[22:33:32] <Krinkle>	 AndyRussG: Just going through some 'bookmarked changes to remember'. Was hoping for a cors/json patch :)
[22:50:07] <grrrit-wm>	 (PS1) Awight: Revert "Update DonationInterface submodule" [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/311054
[22:50:16] <grrrit-wm>	 (CR) Awight: [C: 2] Revert "Update DonationInterface submodule" [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/311054 (owner: Awight)
[22:50:49] <awight>	 !log update fundraising CRM from f381bd1c15e30e4d47fe372e128143baee6a7c7a to 5ba6976f2552564b51085abff0afd5f76195229b
[22:50:57] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[22:54:11] <grrrit-wm>	 (PS1) Awight: One more Configuration call needing staticization [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/311058
[22:54:22] <awight>	 fr-tech: ^ if you can
[22:55:25] <awight>	 !log Reenabled donations and fredge consumers
[22:55:31] <awight>	 MBeat: the-wub: FYI ^
[22:55:42] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[22:55:46] <MBeat>	 yay, thanks awight
[22:56:05] <awight>	 I should never hold my breath during outages...
[22:56:20] <MBeat>	 smell the flowers, blow out the candle
[22:56:35] <awight>	 totally zen!
[22:59:50] <grrrit-wm>	 (Merged) jenkins-bot: Revert "Update DonationInterface submodule" [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/311054 (owner: Awight)
[23:00:45] <awight>	 !log update paymentswiki to from 996ca30076946c6148d9688a905c82e2e346e165 to 392d67520d14998d61823755b22df50ab45afb35 (reverted DI submodule update)
[23:00:51] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[23:22:28] <awight>	 !log updating paymentswiki to possible broken 609c1e5f03ddcd9d72f562d7911cf77cc459371b
[23:22:37] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[23:23:01] <awight>	 MBeat: I might be breaking the Ingenico frontend for a few minutes...
[23:23:34] <MBeat>	 ok, ty for alert. will donors get any particular kind of error?
[23:24:40] <awight>	 MBeat: done, I've rolled back to the working version
[23:24:49] <awight>	 !log rolled back paymentswiki to 392d67520d14998d61823755b22df50ab45afb35
[23:24:56] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[23:25:19] <awight>	 MBeat: Donors would have gotten a "no payment processors available" exception when clicking on a card type
[23:25:34] <MBeat>	 cool, good to know - thanks
[23:28:35] <AndyRussG>	 Krinkle: aaarg pls remind me about what we can do with cors for CN? IIRC it seemed the hide-cookie storm wouldn't be helped much by it
[23:28:43] * AndyRussG should have made a task
[23:29:25] <Krinkle>	 AndyRussG: The free geo ip client uses jsonp which generates a random callback query parameter and is not cacheable by either the third-party service nor the client's browser, it's also somewhat less efficient to append a <script src> compared to XHR.
[23:29:48] <Krinkle>	 basically, set dataType: $.support.cors ? 'json' : 'jsonp'
[23:35:33] <AndyRussG>	 Krinkle: ah yea right... thx!!
[23:36:03] <ejegg>	 woohoo!  Think I got a place to live next month
[23:36:10] <AndyRussG>	 ejegg: cool!
[23:36:16] <AndyRussG>	 In which city?
[23:36:21] <ejegg>	 New Orleans
[23:36:49] <AndyRussG>	 Wow! That's far!
[23:36:55] <AndyRussG>	 Zyedeco!
[23:37:09] <grrrit-wm>	 (CR) Ejegg: [C: 2] One more Configuration call needing staticization [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/311058 (owner: Awight)
[23:37:27] <ejegg>	 yeah, and funk!
[23:38:11] <awight>	 MBeat: Another scary thing I'm doing: running the new orphan rectifier.  That's what caused us to charge fraudy status 600s last time, but hopefully that hole is patched.
[23:39:08] <MBeat>	 I’ll holler if I see anything unwholesome :)
[23:40:34] <awight>	 Looks like we're in the clear.
[23:40:53] <grrrit-wm>	 (Merged) jenkins-bot: One more Configuration call needing staticization [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/311058 (owner: Awight)
[23:41:53] <ejegg>	 awight: was ^^^ the issue?
[23:42:01] <ejegg>	 or something more?
[23:42:03] <awight>	 That was the banner-history job
[23:42:12] <awight>	 I just sent an email about the rest of the fun times
[23:42:35] <ejegg>	 ah thanks, lemme read that
[23:43:00] <awight>	 BTW it does look like the new rectifier can be turned on--I'm just waiting 20 min for stale-enough messages to come in.
[23:43:55] <ejegg>	 awight: ooh, I see, the config for the old rectifier queue still has that key in it
[23:44:24] <awight>	 As it should, I believe...
[23:44:36] <ejegg>	 right
[23:45:17] <ejegg>	 I could have done a workaround that looked at config in DonationQueue::set and decided whether to call set or push
[23:45:22] <awight>	 I might go staff mix for a few minutes, but be back shortly to fire up the new rectifier
[23:45:26] <awight>	 yeeeah
[23:45:27] <ejegg>	 might have saved us a bunch of time.
[23:45:27] <awight>	 ...
[23:45:46] <awight>	 I could have saved us a month by just rewriting the rectifier in-place.
[23:45:58] <ejegg>	 enjoy the staff mixer!
[23:46:09] <awight>	 On the flip side, we really did need to be able to update DI for CRM again
[23:46:15] <awight>	 brb...
[23:58:42] <AndyRussG>	 cool!
[23:59:16] <AndyRussG>	 (that comment ^ was meant for like 20 minutes ago... Dunno why my client sent it just now!)
[23:59:32] <AndyRussG>	 (not to say that updating DI for CRM isn't cool, too, though...)