[00:01:02] <wikibugs>	 Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, Fundraising Sprint Testing on Production, Fundraising-Backlog, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2711496 (AndyRussG) @K4-713 Thanks! I'm pretty sure it's somehow rela...
[00:46:19] <grrrit-wm>	 (PS2) Awight: Shuffle config usages around to be less of a jerk [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314486
[00:47:05] <grrrit-wm>	 (CR) Awight: Shuffle config usages around to be less of a jerk (2 comments) [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314486 (owner: Awight)
[00:51:01] <grrrit-wm>	 (PS2) Awight: Enable flake8 tests [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314487
[00:51:18] <grrrit-wm>	 (PS3) Awight: Regression test for T143903 [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314509 (https://phabricator.wikimedia.org/T143903)
[00:55:33] <grrrit-wm>	 (PS3) Awight: Enable flake8 tests [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314487
[00:55:35] <grrrit-wm>	 (PS3) Awight: Shuffle config usages around to be less of a jerk [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314486
[00:55:37] <grrrit-wm>	 (PS4) Awight: Regression test for T143903 [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314509 (https://phabricator.wikimedia.org/T143903)
[00:56:09] <awight>	 cwd: ^ almost in shape to enable tox on tools btw!
[00:58:21] <grrrit-wm>	 (PS1) Cdentinger: Fix subscr_cancel error [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315613
[00:59:04] <cwd>	 excellent
[01:00:32] <awight>	 I'm wondering how to test the silverpop export tho
[01:00:50] <awight>	 It would be great if I could get away with a sqlite::memory
[01:01:08] <awight>	 But i don't think it's happening
[01:04:48] <cwd>	 i hear the words silverpop export a lot but i don't actually know anything about it
[01:04:53] <cwd>	 is it a csv ?
[01:08:11] <awight>	 check it out... https://github.com/wikimedia/wikimedia-fundraising-tools/blob/master/silverpop_export/update_table.sql
[01:08:25] <awight>	 Not actually "silverpop" anymore as they were absorbed by IBM or something
[01:08:39] <awight>	 it's a nightly export of our everything.
[01:09:12] <cwd>	 yikes
[01:09:25] <cwd>	 best not to misplace those
[01:09:58] <cwd>	 i don't see why it wouldn't work with sqlite as long as you populate all the source data
[01:10:14] <awight>	 might be a big pain
[01:10:26] <cwd>	 yeah looks like it
[01:10:27] <awight>	 the key and collation stuff has slightly different syntax
[01:10:41] <cwd>	 rgh
[01:10:57] <cwd>	 shoudln't pdo handle that?
[01:11:12] <cwd>	 well not if they are raw queries
[01:11:28] <cwd>	 dag
[01:11:31] <awight>	 there's no PDO layer or nothing yeah
[01:11:42] <awight>	 ACtually there's one layer, but not something to write home about
[01:12:01] <awight>	 We replace database names from python and run each statement in its own python thing
[01:12:05] <awight>	 it's gnasty
[01:13:47] <cwd>	 yeah i can't think of a way besides firing up a test mysql
[01:14:46] <cwd>	 nothing besides mysql is probably going to understand that query
[01:14:48] <awight>	 not the worst thing to do
[01:14:50] <awight>	 SET storage_engine=MEMORY
[01:15:42] <cwd>	 but how does that work in CI?
[01:16:29] <awight>	 I'm looking forward to finding out ;)
[01:16:57] <awight>	 it should be a *lot* faster than disk-backed tho
[01:17:57] <awight>	 gtg dine, see you tomorrow!
[01:18:45] <cwd>	 later!
[03:55:34] <grrrit-wm>	 (PS2) Cdentinger: Fix subscr_cancel error [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315613
[04:50:02] <wikibugs>	 Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, Fundraising Sprint Testing on Production, Fundraising-Backlog, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2711798 (AndyRussG) Summary of what seem to be the most important dat...
[06:20:13] <grrrit-wm>	 (CR) Awight: [C: 2] "Great, but with the @dataProvider reservations I mentioned inline." (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315613 (owner: Cdentinger)
[06:21:19] <grrrit-wm>	 (Merged) jenkins-bot: Fix subscr_cancel error [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315613 (owner: Cdentinger)
[08:09:53] <grrrit-wm>	 (PS1) Awight: WIP Tests for the bulk export [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315638 (https://phabricator.wikimedia.org/T131770)
[13:00:13] <dstrine>	 ejegg|away:  are you still on east coast time? do you see the fail mail  over the last 4 hours?
[14:11:12] <grrrit-wm1>	 (PS1) Cdentinger: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/315678
[14:14:17] <grrrit-wm1>	 (CR) Cdentinger: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/315678 (owner: Cdentinger)
[14:14:29] <grrrit-wm1>	 (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/315678 (owner: Cdentinger)
[14:18:17] <cwd|afk>	 !log updated smashpig from 28ba033be2f50fc7837c788dc452753ff82284e1 to e89f1b5316955b5d6576d4c71f8ab840db9b4de2
[14:18:22] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[15:21:40] <grrrit-wm>	 (CR) Awight: "check experimental" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315638 (https://phabricator.wikimedia.org/T131770) (owner: Awight)
[15:39:07] <grrrit-wm>	 (PS2) Awight: WIP Tests for the bulk export [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315638 (https://phabricator.wikimedia.org/T131770)
[15:39:38] <grrrit-wm>	 (CR) Awight: "check experimental" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315638 (https://phabricator.wikimedia.org/T131770) (owner: Awight)
[15:48:56] <grrrit-wm>	 (PS3) Awight: WIP Tests for the bulk export [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315638 (https://phabricator.wikimedia.org/T131770)
[15:49:53] <grrrit-wm>	 (CR) Awight: "check experimental" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315638 (https://phabricator.wikimedia.org/T131770) (owner: Awight)
[15:58:24] <grrrit-wm>	 (PS4) Awight: WIP Tests for the bulk export [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315638 (https://phabricator.wikimedia.org/T131770)
[15:58:54] <grrrit-wm>	 (CR) Awight: "check experimental" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315638 (https://phabricator.wikimedia.org/T131770) (owner: Awight)
[16:05:21] <ejegg>	 dstrine: sorry... on central time, but taking a look at the failmail now
[16:08:01] <cwd>	 ejegg: was my bad, should be fixed now
[16:08:58] <ejegg>	 ah yeah, just saw what it was
[16:09:00] <ejegg>	 thanks!
[16:11:53] <cwd>	 didn't know it was gonna get so noisy
[16:15:22] <ejegg>	 So, I'm looking back over the stomp queues. something seems to still be sending refunds to stomp. trying to figure that one out
[16:16:17] <cwd>	 hmmm
[16:16:38] <cwd>	 wouldn't be the orphan rectifier since it just handles from end stuff huh?
[16:17:33] <ejegg>	 right
[16:17:41] <ejegg>	 either listener or audit
[16:18:08] <ejegg>	 but the settings on production don't seem like they should be sending anything to stomp
[16:18:33] <cwd>	 hmm
[16:18:35] <cwd>	 something in tools?
[16:18:52] <ejegg>	 let's see...
[16:22:14] <ejegg>	 oh wait, these are from the listener, but they stopped yesterday.
[16:23:07] <cwd>	 adyen listener?
[16:23:09] <ejegg>	 I definitely asked Jeff_Green to deploy an update to SmashPig.yaml yesterday, but I thought we had turned off the recurring and refund mirroring last week
[16:23:12] <ejegg>	 amazon listener
[16:24:49] <cwd>	 gotta take the pooch for a jog, back soon
[16:24:53] <Jeff_Green>	 ejegg: i don't remember an update yesterday
[16:25:19] <Jeff_Green>	 last change was the 11th I think
[16:26:35] <ejegg>	 Jeff_Green: didn't we just update it to add the jobs-adyen queue?
[16:26:49] <ejegg>	 oh right, I guess that was the 11th
[16:26:56] <Jeff_Green>	 yeah that's right
[16:27:13] <Jeff_Green>	 speaking of listeners, what do you think the chances are that the listener can run cross-site
[16:27:33] <ejegg>	 you mean in a different data center?
[16:27:41] <Jeff_Green>	 yeah, using the codfw listener for a while?
[16:28:06] <Jeff_Green>	 i want to upgrade thulium to jessie we won't have a replacement box until next FY
[16:40:25] <ejegg>	 oho, we had configured refund and recurring mirroring in the defaults, not in the override file
[16:40:36] <ejegg>	 so the code deploy yesterday was what turned it off
[17:00:02] <wikibugs>	 Fundraising Sprint Octopus Untangling, Fundraising Sprint Pretending This Isn't Happening, Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, and 4 others: Decimal dates are breaking our queue consumer - https://phabricator.wikimedia.org/T140667#2713565 (DStrine)
[17:00:56] <slander>	 fr-tech: You will be surrounded by luxury.
[17:00:56] <slander>	  -- discuss.
[17:01:56] <wikibugs>	 Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Zero at the beginning of zip code not being reflected upon import - https://phabricator.wikimedia.org/T148056#2713567 (LeanneS)
[17:03:27] <cwd>	 makin coffee, be there soon
[17:07:19] <Jeff_Green>	 ooh coffee. now I'm making coffee too!
[17:11:52] <AndyRussG|mostly>	 aaaarg!
[17:13:09] <Jeff_Green>	 aaargh?
[17:15:17] <AndyRussG|mostly>	 oh yeah!
[17:17:24] <AndyRussG|mostly>	 Really just an existential aaarg, not a specific something broke kinda aaaarg
[18:10:26] <awight>	 dang--CI jobs don't report the machine name any more?
[18:11:07] <awight>	 huh.  Or they're respawned per job now?
[18:19:03] <wikibugs>	 Fundraising-Backlog: Remove refunds from Silverpop donation history - https://phabricator.wikimedia.org/T148062#2713848 (Krenair)
[18:24:07] <wikibugs>	 Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, Fundraising Sprint Testing on Production, Fundraising-Backlog, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2713885 (AndyRussG) Following up on the patch ^ by @Ejegg, addressing...
[18:24:38] <AndyRussG|mostly>	 ejegg: ^ some more thoughts, dug a wee bit more... also thx!!!!
[18:26:00] <ejegg>	 thank you!
[18:30:30] <AndyRussG|mostly>	 ejegg: mebbe we can deploy ur patch to the beta cluster (before it gets merged) to see if the error still happens
[18:30:56] <ejegg>	 AndyRussG|mostly: oh cool, can you show me how to do that?
[18:31:46] * ejegg looks for docs
[18:34:26] <ejegg>	 aha, https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/How_code_is_updated#Cherry-picking_a_patch_from_gerrit
[18:35:05] <ejegg>	 XenoRyet: want to try figuring this out with me? ^^
[18:35:24] <XenoRyet>	 Sure
[18:35:33] <ejegg>	 I think we might need to get access to the beta cluster labs project first
[18:35:36] <ejegg>	 let's see
[18:35:47] <awight>	 ejegg: I haven't worked out mysql password fu yet, but wanted you to know: https://gerrit.wikimedia.org/r/#/c/315638/4
[18:35:50] <awight>	 I'm psyched!
[18:36:06] <ejegg>	 owow!
[18:37:39] <ejegg>	 XenoRyet: https://wikitech.wikimedia.org/wiki/Help:Access
[18:38:20] <ejegg>	 XenoRyet: have you logged into any labs stuff yet? You'd use your wikitech/gerrit password
[18:38:27] <XenoRyet>	 No, this will be the first time.
[18:38:59] <ejegg>	 you have a wikitech account though, right?
[18:39:27] <ejegg>	 oh yah, of course you do, cause gerrit
[18:39:37] <XenoRyet>	 Yea
[18:40:42] <ejegg>	 OK, looks like we're both at this step: https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Projects
[18:41:21] <ejegg>	 so this is the project in question: https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep
[18:41:59] <ejegg>	 aha, and AndyRussG is listed as an admin
[18:42:08] <AndyRussG|mostly>	 ejegg: XenoRyet I have access
[18:42:15] <ejegg>	 AndyRussG|mostly: could you please add us to the project?
[18:42:19] <XenoRyet>	 cool
[18:44:39] <AndyRussG|mostly>	 ejegg: XenoRyet: yeah! what are ur wikitech usernames?
[18:44:47] <AndyRussG|mostly>	 Just Ejegg and XenoRyet ?
[18:44:56] <XenoRyet>	 Yep
[18:45:08] <ejegg>	 yep
[18:46:23] <grrrit-wm>	 (PS5) Awight: WIP Tests for the bulk export [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315638 (https://phabricator.wikimedia.org/T131770)
[18:46:37] <grrrit-wm>	 (CR) Awight: "check experimental" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315638 (https://phabricator.wikimedia.org/T131770) (owner: Awight)
[18:47:33] <AndyRussG|mostly>	 ejegg: XenoRyet: K I just added u both!
[18:47:50] <AndyRussG|mostly>	 You saw the extra .ssh config you need to add, right?
[18:48:05] <XenoRyet>	 Cool, thanks.
[18:48:18] <XenoRyet>	 Getting to that part, I think.
[18:48:20] <ejegg>	 rockin, looking at the config thing
[18:50:04] <AndyRussG|mostly>	 Running the Hive query in the notes again adding hostname
[18:55:39] <ejegg>	 ok, here I am on deployment-puppetmaster
[18:55:46] <AndyRussG|mostly>	 weee!
[18:55:56] <ejegg>	 XenoRyet: are you able to get there?
[18:56:00] <XenoRyet>	 Almost
[18:58:45] <XenoRyet>	 Ok, yea, in now.
[18:59:03] <ejegg>	 cool, I'm looking at the cherry-pick instructions
[18:59:26] <ejegg>	 https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/How_code_is_updated#Cherry-picking_a_patch_from_gerrit
[18:59:37] <XenoRyet>	 Yea, looking there too.
[19:00:03] <ejegg>	 hmm, wait, is that puppet-specific?
[19:00:55] <grrrit-wm>	 (PS6) Awight: Tests for the bulk export [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315638 (https://phabricator.wikimedia.org/T131770)
[19:01:13] <grrrit-wm>	 (CR) Awight: "check experimental" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315638 (https://phabricator.wikimedia.org/T131770) (owner: Awight)
[19:03:54] <XenoRyet>	 Yea, this looks pretty puppetty
[19:04:28] <ejegg>	 OK, let's look for something more like scap
[19:06:38] <ejegg>	 AndyRussG|mostly: where are you changing the deployed core code on beta? That doesn't mess with the beta-code-update-eqiad job ?
[19:07:18] <AndyRussG|mostly>	 ejegg: never done it actually :)
[19:07:32] <ejegg>	 oh, I thought you did some livehacking earlier
[19:08:24] <ejegg>	 will ask in #wikimedia-labs
[19:10:13] <AndyRussG|mostly>	 ejegg: yeah that had been an idea but instead we took advantage of instrumentation already in the MessageCache class, and pushed a config change to turn it on
[19:10:39] <ejegg>	 ah cool. asking in -releng per yuvi's suggestion
[19:10:48] <AndyRussG|mostly>	 ah yes I was gonna say :)
[19:11:49] <AndyRussG|mostly>	 ejegg: XenoRyet: here's the logstash query for MessageCache stuff on beta
[19:12:06] <grrrit-wm>	 (PS7) Awight: Tests for the bulk export [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315638 (https://phabricator.wikimedia.org/T131770)
[19:12:36] <grrrit-wm>	 (CR) Awight: "check experimental" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315638 (https://phabricator.wikimedia.org/T131770) (owner: Awight)
[19:16:25] <ejegg>	 XenoRyet: AndyRussG|mostly - getting answers from Krenair in -labs
[19:16:43] <AndyRussG|mostly>	 cool!
[19:19:42] <awight>	 ejegg: silverpop_export tests pass on CI now, thanks to paladox!
[19:20:08] <ejegg>	 awight: freaking cool!
[19:20:28] <ejegg>	 oh, i see you reshuffled that config stuff too. will re-examine shortly
[19:20:32] <awight>	 It was hairy, but a pretty lightweight amount of python glue
[19:20:45] <awight>	 the SQL is... open to debate
[19:21:26] <awight>	 I went minimal.  lots of missing columns if we don't need them, I'm inserting contacts like "insert (id) values (2)"
[19:21:29] <awight>	 :)
[19:21:47] <awight>	 nice to be able to focus on the actual code being tested though
[19:21:59] <awight>	 I love programming languages built for lazy, bad programmers :)
[19:24:37] <ejegg>	 heh
[19:26:53] <ejegg>	 AndyRussG|mostly: I need to eat and plug in the lappy, but I can test that core change in 1/2 hr or so
[19:28:51] <AndyRussG|mostly>	 ejegg|food: also fin!
[19:28:52] <AndyRussG|mostly>	 fine
[19:34:10] <grrrit-wm>	 (PS1) Awight: WIP fix py tests [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315741 (https://phabricator.wikimedia.org/T145012)
[19:34:29] <grrrit-wm>	 (CR) Awight: "check experimental" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315741 (https://phabricator.wikimedia.org/T145012) (owner: Awight)
[19:38:54] <grrrit-wm>	 (PS2) Awight: WIP fix py tests [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315741 (https://phabricator.wikimedia.org/T145012)
[19:39:17] <ejegg|food>	 plz start standup without me fr-tech. i'll be 10 min late or so
[19:41:16] <grrrit-wm>	 (CR) Awight: "check experimental" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315741 (https://phabricator.wikimedia.org/T145012) (owner: Awight)
[19:42:53] <grrrit-wm>	 (PS3) Awight: WIP fix py tests [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315741 (https://phabricator.wikimedia.org/T145012)
[19:43:09] <grrrit-wm>	 (CR) Awight: "check experimental" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/315741 (https://phabricator.wikimedia.org/T145012) (owner: Awight)
[19:57:53] <AndyRussG|mostly>	 ejegg|food: I should run at 1:05 or 1:10 Pacific Time and then should back by 1:45
[20:05:27] * robla just read the backlog of T144952, and I'm happy to help if/how I can
[20:17:48] <ejegg>	 Thanks robla! I just added you as a reviewer for a possibly-related patch: https://gerrit.wikimedia.org/r/315712
[20:20:34] * robla looks at the patch on the off chance that he might be able to say something constructive about it
[20:33:00] <grrrit-wm>	 (PS1) Ejegg: Update PopulateQueueFromDump for PHPQueue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315754 (https://phabricator.wikimedia.org/T147982)
[20:33:31] <ejegg>	 fr-tech I need to use that to move the 65 Stomp messages into Civi ^^
[20:38:04] <grrrit-wm>	 (PS4) Ejegg: Shuffle config usages around to be less of a jerk [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314486 (owner: Awight)
[20:38:10] <grrrit-wm>	 (CR) Ejegg: [C: 2] Shuffle config usages around to be less of a jerk (1 comment) [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314486 (owner: Awight)
[20:38:17] <grrrit-wm>	 (Merged) jenkins-bot: Shuffle config usages around to be less of a jerk [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314486 (owner: Awight)
[20:40:39] <grrrit-wm>	 (PS4) Ejegg: Enable flake8 tests [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314487 (owner: Awight)
[20:44:10] <ejegg>	 oh hey, Yakkety Yak is official
[20:44:17] <ejegg>	 aka ubuntu 16.10
[20:45:10] <cwd|afk>	 haha
[20:48:37] <ejegg>	 cwd got a minute to review a maintenance script? https://gerrit.wikimedia.org/r/315754
[20:50:42] <cwd>	 i will look at it shortly, pulling on some strings with this adyen fraud thing
[20:52:10] <ejegg>	 ah cool
[20:52:16] <ejegg>	 let me know if I can help with that at all
[20:55:36] <AndyRussG>	 ejegg: back, test-ready netime :)
[20:55:42] <ejegg>	 AndyRussG: cool, let me get my bearings
[20:56:03] <AndyRussG>	 K :)
[20:56:28] <AndyRussG>	 I'm all here for all the rest of the day so no rush :)
[20:56:45] <AndyRussG>	 (though if it works it'd be fun to deploy......!)
[21:01:01] <grrrit-wm>	 (CR) Ejegg: [C: 2 V: 2] Enable flake8 tests [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314487 (owner: Awight)
[21:01:57] <ejegg>	 AndyRussG: ok, I'm all set to do that, let me just time it right after an automatic update
[21:02:23] <ejegg>	 AndyRussG: looks like one will happen in a minute
[21:02:39] <AndyRussG>	 aaaah they're on a schedule, cool!
[21:02:57] <ejegg>	 yah, every 10 minutes on the 3s
[21:05:30] <grrrit-wm>	 (Merged) jenkins-bot: Enable flake8 tests [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/314487 (owner: Awight)
[21:05:43] <ejegg>	 ok, doing the cherry-pick
[21:08:17] <ejegg>	 ok, going to sync
[21:08:45] <AndyRussG>	 weeeeeeeeeeeeee :)
[21:09:38] <ejegg>	 AndyRussG: OK, all synced
[21:10:07] <AndyRussG>	 :D
[21:10:16] <ejegg>	 now we have like 3 minutes to test
[21:10:29] <ejegg>	 sorry, didn't realize how long the real updater would take
[21:12:45] <ejegg>	 AndyRussG: eek, about to be another auto-update. did you have time to do anything?
[21:13:24] <AndyRussG>	 ejegg: yeah haven't seen the error yet!
[21:13:40] <ejegg>	 hmm
[21:13:52] <ejegg>	 that's not conclusive though, right?
[21:14:11] <AndyRussG>	 ejegg: correct, it didn't always appen
[21:15:06] <ejegg>	 well, the patch will be live for another 2 minutes
[21:15:24] <AndyRussG>	 ejegg: ah k testing smore
[21:15:53] <ejegg>	 (I un-picked in /srv/mediawiki-staging, but the scap jobs rebuilds the i18n cache before syncing files)
[21:17:30] <ejegg>	 AndyRussG: oop, overwritten
[21:17:49] <ejegg>	 I can keep doing that dance tho
[21:18:48] <ejegg>	 AndyRussG: ok, re-applied :)
[21:18:57] <AndyRussG>	 ahhh cool!
[21:20:10] <AndyRussG>	 ejegg: no errors so far...!
[21:20:12] <AndyRussG>	 https://logstash-beta.wmflabs.org/app/kibana#/dashboard/default?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-1h,mode:quick,to:now))&_a=(filters:!(),options:(darkTheme:!f),panels:!((col:1,id:Dashboards,panelIndex:1,row:1,size_x:12,size_y:1,type:visualization),(col:1,id:Events-Over-Time,panelIndex:2,row:2,size_x:12,size_y:2,type:visualization),(col:1,id:Event-Types,panelIndex
[21:20:14] <AndyRussG>	 :3,row:4,size_x:9,size_y:2,type:visualization),(col:10,id:Event-Level,panelIndex:4,row:4,size_x:3,size_y:2,type:visualization),(col:1,columns:!(type,level,wiki,host,message),id:Default-Events-List,panelIndex:5,row:6,size_x:12,size_y:25,sort:!(%27@timestamp%27,desc),type:search)),query:(query_string:(analyze_wildcard:!t,query:%27centralnotice%20AND%20message%20AND%20key%27)),title:default,uiStat
[21:20:16] <AndyRussG>	 e:(P-2:(vis:(legendOpen:!f)),P-3:(vis:(legendOpen:!f)),P-4:(vis:(legendOpen:!f))))
[21:20:28] <ejegg>	 hehe
[21:20:35] <ejegg>	 I'll take your word for it
[21:20:56] <AndyRussG>	 https://goo.gl/x0CpO4
[21:21:28] <AndyRussG>	 Feel free also to click on the new banner previews!
[21:21:30] <AndyRussG>	 https://meta.wikimedia.beta.wmflabs.org/wiki/Special:CentralNoticeBanners
[21:21:43] <ejegg>	 k, looks good!
[21:22:31] <ejegg>	 hmm, i may have just gotten a blank
[21:23:32] <AndyRussG>	 Ooof
[21:23:37] <AndyRussG>	 Ah no, they're all blank
[21:23:39] <ejegg>	 oh wait, that's just the default text
[21:23:42] <ejegg>	 "bannerHtml":"\u003C!-- Empty banner --\u003E"
[21:23:49] <AndyRussG>	 Yeah that's all good :)
[21:24:01] <AndyRussG>	 Check network tab, BannerLoader response
[21:24:04] <ejegg>	 ok, beta-scap's about to run us over again
[21:27:19] <AndyRussG>	 ejegg: still looking great!
[21:27:39] <AndyRussG>	 lmk when it's overwritten and back to master? if the error reproduces easily then, that'd also something!
[21:27:55] <ejegg>	 oops, I just re-overwrote master
[21:28:04] <AndyRussG>	 hehe all good :)
[21:28:12] <AndyRussG>	 K I'll keep trying to get it!
[21:30:42] <AndyRussG>	 on a less positive note, the meatloaf I made for today's midday meal tastes awful
[21:30:54] <AndyRussG>	 we'll see if people eat it smothered in bbq sauce
[21:31:25] * AndyRussG listens for gagging sounds from desk
[21:32:43] <AndyRussG>	 still no errors!
[21:33:39] <AndyRussG>	 nor gagging
[21:33:50] <AndyRussG>	 I don't hear any "mmmmm ymmm" noises from the dining room either, though
[21:37:22] <AndyRussG>	 ejegg: when's the next update?
[21:37:44] <ejegg>	 AndyRussG: oop, let me see
[21:38:09] <ejegg>	 AndyRussG: I think it just finished
[21:38:20] <AndyRussG>	 K let's try now
[21:38:21] <ejegg>	 yeah, looks like the patch is wiped out across beta
[21:41:59] <AndyRussG>	 ejegg: hmmm still no errors happening also... the watched kettle
[21:42:33] <ejegg>	 huh. you've been adding new banners?
[21:44:20] <AndyRussG>	 ejegg: yea a million
[21:44:26] <AndyRussG>	 are you sure he patch is wiped out?
[21:44:35] <ejegg>	 hmm, I thought it was
[21:44:51] <ejegg>	 it's definitely not in the /srv/mediawiki-staging tree any more
[21:45:06] <ejegg>	 but I can do a sync-file to make sure it's across the cluster
[21:45:15] <AndyRussG>	 ah K pls yeah that'd be phab
[21:45:30] <ejegg>	 err, after the normal scap finishes and releases its lock
[21:45:42] <AndyRussG>	 hmm
[21:45:50] <AndyRussG>	 So maybe it wasn't synced on all hosts?
[21:46:56] <AndyRussG>	 if this fixes it we can definitely ask for quick review and maybe deploy from core-ish folks
[21:48:16] <ejegg>	 I was pretty sure the last scap wiped it out - i think that does a full rsync
[21:48:23] <ejegg>	 but I'm re-syncing just the one file
[21:48:30] <AndyRussG>	 K lmk
[21:49:07] <ejegg>	 ok, definitely synced now
[21:55:55] <AndyRussG>	 ejegg: nope no errors also
[21:56:24] <ejegg>	 AndyRussG: hrm...
[21:56:35] <ejegg>	 let's see what else has been happening in code lately
[21:57:25] <AndyRussG>	 yeah exactly
[22:03:12] <ejegg>	 definitely some cachey stuff in the recent logs
[22:12:08] <AndyRussG>	 ejegg: I don't see anything that looks like it might impact, so far... Last error logged was Oct. 11 3 hrs UTC
[22:12:25] <AndyRussG>	 ejegg:  ah no
[22:12:34] <AndyRussG>	 2016-10-12T17:12:40	
[22:14:00] <grrrit-wm>	 (CR) XenoRyet: [C: 2] Update PopulateQueueFromDump for PHPQueue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315754 (https://phabricator.wikimedia.org/T147982) (owner: Ejegg)
[22:14:25] <ejegg>	 ty XenoRyet
[22:14:32] <XenoRyet>	 No worries
[22:14:47] <ejegg>	 relocating for outlet...
[22:15:21] <grrrit-wm>	 (Merged) jenkins-bot: Update PopulateQueueFromDump for PHPQueue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315754 (https://phabricator.wikimedia.org/T147982) (owner: Ejegg)
[22:16:52] <AndyRussG>	 maybe it's rather dependent on traffic volume somehow?
[22:34:20] <grrrit-wm>	 (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/315847
[22:34:30] <grrrit-wm>	 (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/315847 (owner: Ejegg)
[22:40:28] <grrrit-wm>	 (CR) Ejegg: [V: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/315847 (owner: Ejegg)
[22:41:18] <ejegg>	 !log updated SmashPig from e89f1b5316955b5d6576d4c71f8ab840db9b4de2 to 3c3d115d8388515eaf1f88b4263a56985454f3a1
[22:41:23] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[22:42:10] <ejegg>	 OK, I'm going to re-queue the 65 missing messages. Wouldn't be surprised if some of them fail!
[22:43:24] <ejegg>	 ok, re-queued, let's see what the consumer does
[23:06:02] <grrrit-wm>	 (PS1) Ejegg: Retry capture later when donor details are missing [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315858 (https://phabricator.wikimedia.org/T147378)
[23:06:32] <ejegg>	 cwd super-lazy version: ^^^
[23:07:03] <ejegg>	 not even adding the source_enqeueued_date field till we try to requeue the first time
[23:07:12] <ejegg>	 but it should make sure we don't infinitely retry things
[23:08:23] <ejegg>	 oh ha, that requeue delayed messages job should run more often
[23:11:34] <ejegg>	 well, only 19 of those orphaned stomp messaged left. Guess I can fix the json dumps by hand
[23:13:11] <cwd>	 yikes
[23:13:21] <cwd>	 wont' the audit find em?
[23:13:38] <cwd>	 or wait, this was the audit?
[23:20:27] <ejegg>	 cwd some of them were, i think
[23:20:36] <ejegg>	 a bunch are from 9/29...
[23:24:49] <ejegg>	 ok, re-queued the edited batch, let's hope these make it through
[23:28:27] <ejegg>	 all but 2. huh
[23:29:08] <ejegg>	 both amazon messages, from the listener
[23:29:24] <ejegg>	 let's see if they got some janky order ids in the pending db
[23:31:37] <ejegg>	 nope, not there at all
[23:35:50] <wikibugs>	 Fundraising-Backlog: Access for new contractor, Allen - https://phabricator.wikimedia.org/T148105#2715060 (LeanneS)
[23:45:11] <grrrit-wm>	 (PS2) Ejegg: Retry capture later when donor details are missing [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/315858 (https://phabricator.wikimedia.org/T147378)
[23:45:29] <ejegg>	 fr-tech this is a potential fix for the Adyen thing dstrine said would be worth a friday deploy ^^^
[23:49:34] <cwd>	 i've been thinking about it
[23:49:44] <XenoRyet>	 What did the problem end up being?
[23:50:04] <cwd>	 my gut feeling is we shouldn't use exceptions for this but i'm not sure i have a better alternative
[23:51:52] <ejegg>	 XenoRyet: the donor info wasn't in pending db yet when the auth came in
[23:52:18] <ejegg>	 somebody entered cc details fast or our pending qc was lagging
[23:52:33] <ejegg>	 anyway, we should be able to retry later when that happens
[23:54:46] <XenoRyet>	 Ok, yea.  I think I remember how that works.
[23:55:03] <XenoRyet>	 But the important bit is that donor data will show up later, so we can try again.
[23:56:22] <XenoRyet>	 Bah, didn't see the clock.  I've got to go pick up the boy.  I can look at it first thing in the morning if cwd doesn't get to it tonight.
[23:56:31] <ejegg>	 k, thanks!
[23:56:57] <ejegg>	 yeah, getting dark here, I suppose I'll pick this up in the morning
[23:58:10] <cwd>	 i also have to run and let out a friend's dog but i will CR this later