[00:20:15] (CR) Ejegg: "Looks functional! Some style nitpicks and TODOS for later inline." (6 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312084 (https://phabricator.wikimedia.org/T145419) (owner: XenoRyet) [01:04:14] (PS1) Ejegg: Remove obsolete and broken wmf_unsubscribe module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312441 (https://phabricator.wikimedia.org/T145419) [01:05:45] (CR) Awight: [C: 2] Remove obsolete and broken wmf_unsubscribe module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312441 (https://phabricator.wikimedia.org/T145419) (owner: Ejegg) [01:06:06] heh, that wasn't too hard to sell [01:06:32] * awight|broom turns eye of Sauron [01:08:12] ah man, battling grub & luks for personal stuff. It might as well be 1998 and the source code passed by hardcopy [01:09:12] Why is grub fragile? It should do absolutely everything imaginable to boot a working system, not need every setting placed in like filigree. [01:10:45] Swap missing? Boot. new blkid? Boot. Restaurant burned down? Boot. [01:18:11] (PS1) Ejegg: Stop mirroring to stomp from audit processors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312442 [01:29:06] (PS2) Ejegg: Stop mirroring to stomp from audit processors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312442 [01:36:18] (PS1) Ejegg: Eradicate Stomp from crm repo [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312443 [03:17:13] (PS1) Ejegg: Add placeholder class for NotificationOfFraud messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312460 [03:18:43] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, FR-Adyen, Unplanned-Sprint-Work: Adyen failmails on new message type - https://phabricator.wikimedia.org/T146447#2661308 (Ejegg) [03:19:02] (PS2) Ejegg: Add placeholder class for NotificationOfFraud messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312460 (https://phabricator.wikimedia.org/T146447) [04:08:00] (CR) Ejegg: "recheck" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/311640 (https://phabricator.wikimedia.org/T145447) (owner: AndyRussG) [05:45:17] (CR) Awight: [C: 2] Add placeholder class for NotificationOfFraud messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312460 (https://phabricator.wikimedia.org/T146447) (owner: Ejegg) [06:13:44] (Merged) jenkins-bot: Add placeholder class for NotificationOfFraud messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312460 (https://phabricator.wikimedia.org/T146447) (owner: Ejegg) [14:22:15] fundraising-tech-ops: upgrade all frack servers to debian/jessie - https://phabricator.wikimedia.org/T146479#2662370 (Jgreen) [14:24:03] fundraising-tech-ops: frack jessie integration cleanup - https://phabricator.wikimedia.org/T138222#2662385 (Jgreen) [14:24:05] fundraising-tech-ops: frack eqiad hardware refresh - https://phabricator.wikimedia.org/T133524#2662386 (Jgreen) [14:24:07] fundraising-tech-ops: upgrade all frack servers to debian/jessie - https://phabricator.wikimedia.org/T146479#2662383 (Jgreen) [14:24:32] fundraising-tech-ops: frack eqiad hardware refresh - https://phabricator.wikimedia.org/T133524#2234514 (Jgreen) [14:24:35] fundraising-tech-ops, Patch-For-Review: replace beryllium with new hardware running jessie - https://phabricator.wikimedia.org/T145101#2662388 (Jgreen) Open>Resolved [14:24:45] fundraising-tech-ops, Patch-For-Review: replace beryllium with new hardware running jessie - https://phabricator.wikimedia.org/T145101#2620111 (Jgreen) [14:24:47] fundraising-tech-ops, Operations, ops-eqiad: rack/setup beryllium replacement frauth1001 - https://phabricator.wikimedia.org/T143902#2662390 (Jgreen) Open>Resolved [14:26:37] fundraising-tech-ops: upgrade all frack servers to debian/jessie - https://phabricator.wikimedia.org/T146479#2662394 (Jgreen) p:Triage>Normal [14:26:53] fundraising-tech-ops: encrypt fundraising kafka collector traffic - https://phabricator.wikimedia.org/T142993#2662395 (Jgreen) p:Triage>Normal [14:27:07] fundraising-tech-ops: configure TLS for fundraising syslog collection - https://phabricator.wikimedia.org/T142994#2662396 (Jgreen) p:Triage>Normal [17:11:23] cwd: AndyRussG XenoRyet ejegg|away Jeff_Green I still don't understand Adam's email (and I know none of you wrote it so sorry for dumb questions). How is everything? is anything being wonky to the point were we need to inform fr-online or MBeat ? [17:11:54] dstrine: no we are just trying to push some queue work forward [17:12:06] but it might be a bad day to do that [17:12:45] reading it now... [17:14:10] cwd: thanks. His message is ambiguous. Are TY emails currently broken? And are things not hitting civi? [17:15:01] no nothing is broken that i know of [17:15:21] that would be the danger if the deploy went bad over the weekend [17:15:49] ok cool. Yeah I'm not a fan of friday deploys for these reasons in general [17:18:09] I used to work on a team that had a rule for friday deploys. The person proposing them had to get up in front of everyone and say "I sincerely apologize but I would like to kill a kitten" And then they would describe what they wanted to do. It was a very effective in making people slow down and think through things. [17:18:10] dstrine: the code is something we can roll back easily, and it'll be pretty clear if it's broken [17:18:20] (PS5) XenoRyet: Move Unsubscribe Queue Consumer off ActiveMQ [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312084 (https://phabricator.wikimedia.org/T145419) [17:19:02] It's up to you guys. [17:19:09] or maybe just that one ^^^ - way lower volume, and we'll still get the satisfaction of having pruned another stomp queue this week [17:20:10] Any chance any of it is going to go bad silently or with a delayed reaction? [17:20:11] ejegg: let's just make sure to over communicate to MBeat if there is a deploy [17:21:00] +1 dstrine [17:21:23] XenoRyet: let's stop the normal unsub queue consumer for a bit so we actually get something new for the new code [17:21:36] Sounds like a plan [17:22:13] ah... but doing that one first means more cherry-pick and merge hell... [17:22:30] That might be a compelling reason to hold off on a Friday deploy [17:22:44] oh well [17:23:06] ejegg: isn't some of the deploy stuff jeff has to do? [17:23:09] anyway, plenty of cleanup stuff to work on [17:23:27] cwd ooh yeah. actually, those settings should be fine to deploy with or without the rest of the change [17:24:21] Jeff_Green: there's an updated SmashPig.yaml in my deploy-server homedir that we'd like to get out before ops are gone [17:24:27] ejegg: don't we have to switch the config to point to the queue, as well as the apache rewrite? [17:24:47] seems like that part might be hard to roll back... [17:25:14] cwd the SmashPig config change is the part we need Jeff_Green for, and that's the stuff that turns on mirroring [17:25:54] Pointing the consumer at the new queue / rolling that back is under our control [17:26:00] but there's also a change to point the listener url at smashpig [17:26:10] instead of the listeners repo [17:26:35] right... I think we can do that in the paypal console if we need to - just point it at the new endpoint we've been testing [17:27:24] ah yeah that's true [17:28:21] * ejegg curses gerrit-new's unusability in windows narrower than full screen [17:28:57] totally broken table / css mess [17:30:55] hey. [17:31:01] Hi Jeff_Green [17:31:15] looking... [17:31:16] So, we'd like to turn on mirroring for the last few queues [17:31:41] also, we don't need that inst-args backwards compatibility any more [17:34:07] ok, deploying [17:34:11] thanks! [17:34:14] np [17:34:23] * ejegg hustles over to logger host [17:40:03] ejegg: decent amount of stuff waiting to go out for smashpig [17:40:20] cwd yeah? looking [17:40:47] i notice that fundraising-misc has some spam now [17:41:03] not sure if that's old [17:42:22] hmm, seeing banner history and paypal audit parser stuff in there, which shouldn't be new [17:44:49] cwd actually not seeing too much in the sp deploy queue - just the undefined date fix and that new dummy class for adyen fraud notifications, plus some tests [17:45:24] and moving jobs-paypal key, which works now that the config is updated [17:45:31] dstrine: which e-mail? [17:45:59] AndyRussG: the one about deploying the PayPal listener on a friday [17:46:08] cwd and I got impatient about queue stuff... [17:47:01] well i am ambivalent [17:47:22] cwd I'mma deploy smashpig [17:47:24] i don't see a good reason to not deploy smashpig anyway [17:47:26] cool yeah [17:49:17] yeah AndyRussG people already answered but thanks! [17:50:42] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/312539 [17:51:07] Ahh right K [17:52:26] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/312539 (owner: Ejegg) [17:52:38] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/312539 (owner: Ejegg) [17:53:45] !log updated SmashPig from 8ac116037440746eaf64b9e99e1ee962d5d33475 to 372cd4008fee3fd02ad2eae9163cf7b28d2ef7c8 [17:53:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:54:07] rsync_blaster was surprised at the friday deploy. confirmation code odam [17:56:26] haha [17:57:19] one of these days the randomstringifier will scare or seriously offend us, and we'll have to come up with a new scheme [17:57:53] hehe [17:58:09] handy that it can be 4 letters [19:30:36] (PS1) Ejegg: Fix all db accessors sharing a single instance [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312550 (https://phabricator.wikimedia.org/T141477) [19:46:32] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Operations, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2663252 (AndyRussG) @awight @Pcoombe Just to note, Special:BannerLoader calls always go to metawiki,... [19:52:26] Fundraising Sprint Pretending This Isn't Happening, Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, and 3 others: Donation import should merge and delete pending db info - https://phabricator.wikimedia.org/T141477#2663259 (Ejegg) Found a late stat... [20:15:38] (PS1) Ejegg: WIP use ct_id to find completed, avoid race [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312563 (https://phabricator.wikimedia.org/T141477) [20:30:47] (PS1) Ejegg: WIP maint script to create all db tables [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312577 [20:53:57] Fundraising-Backlog, Project-Admins: Please create a #signal-to-noise tag - https://phabricator.wikimedia.org/T140531#2663359 (DStrine) Open>declined [23:30:35] Fundraising-Backlog: Estimate requirements for DS ticketing system transition - https://phabricator.wikimedia.org/T146511#2663705 (MBeat33)