[03:39:56] !log disabled civicrm dedupe contacts job [03:40:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [03:46:33] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Dedupe job failing - https://phabricator.wikimedia.org/T146607#2666057 (cwdent) [04:09:40] (CR) Cdentinger: "Call me old fashioned but I really like wrapping at 80 chars, esp with split diff in gerrit" (3 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312084 (https://phabricator.wikimedia.org/T145419) (owner: XenoRyet) [15:23:45] (PS2) Reedy: Remove old calls to WikiPage::doEdit() [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/310940 (https://phabricator.wikimedia.org/T145738) [16:00:19] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Epic: [Epic] Analytics, graphs and monitoring for banners and CentralNotice - https://phabricator.wikimedia.org/T124132#2667597 (Nuria) [16:00:23] Fundraising-Analysis, Analytics: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#2667594 (Nuria) Open>Resolved a:Nuria I think issues on ticket have solved, let us know otherwise. [16:00:28] Fundraising-Backlog, Wikimedia-Fundraising, MediaWiki-extensions-CentralNotice, Operations, and 4 others: Redo /beacon/impression system (formerly Special:RecordImpression) to remove extra round trips on all FR impressions (title was: S:RI should pyr... - https://phabricator.wikimedia.org/T45250#2667598 [16:38:50] cwd XenoRyet feel ready to deploy some queue stuff? [16:39:18] I guess I'll empty out the redis donations queue since it's got a pretty big backlog [16:39:34] and it wouldn't be as exciting to have it all be duplicates [16:39:43] AndyRussG: I'm going into a meeting shortly. I was wondering how this CN bugs going? https://phabricator.wikimedia.org/T144952 [16:40:18] yeah ready any time [16:40:29] Yea, lets do it. [16:40:35] woot [16:40:39] ejegg: does the db late static binding thing need to go? [16:40:45] AndyRussG: also we were holding off on tests last week. Should we continue to do so? [16:40:56] cwd not right away [16:40:56] cwd: ejegg XenoRyet WOOT!! [16:41:15] groovy [16:41:18] cwd the lsb bug is just related to the orphan rectifier [16:41:21] ...don't we still need to paypal creds? [16:41:29] *the [16:41:32] cwd oh yeah.... [16:41:58] or Jeff_Green-powers, which we don't have either. [16:42:00] hmm [16:43:16] ah yeah, no real ops presence this week [16:43:24] * ejegg 's plans screech to a halt [16:43:37] k, I'll keep pushing for those paypal creds [16:43:50] oh, but awight can help there [16:44:02] yeah [16:44:06] i think so... [17:01:11] dstrine: Hi, I'm sick in bed, tried working on it this morning, didn't get anywhere.......... :( [17:02:45] oh noes ok [17:02:47] feel better!! [17:03:55] dstrine: aaaaarg yeah very ohnoes [17:04:37] oof, sounds rough AndyRussG ... hang in there! [17:04:40] fr-tech ^ can anyone grab the CN bug, I'm sick, feverish, and worse, sorry and thanks [17:04:58] AndyRussG: thanks!!!! Yeah not a good time to get sick. Dunno if it's a stomach flu or something I ate.... [17:05:19] Probably something from a restaurant yesterday, since my older daughter is sick too [17:05:29] oh man [17:05:39] * AndyRussG tries to stop eliciting pity, fails 8/ [17:05:44] rrrrrrrgggg [17:06:05] AndyRussG: I was just writing--yeah I'd love to poke at the bug for a bit [17:06:11] --as long as it's not too contagious [17:06:48] awight we were also hoping to ask for your help swapping the paypal listener URL in the processor console [17:07:24] I've requested an account, but for now I think you're our Obi Wan [17:08:42] Fundraising-Backlog, Wikimedia-Fundraising, MediaWiki-extensions-CentralNotice, Operations, and 4 others: Redo /beacon/impression system (formerly Special:RecordImpression) to remove extra round trips on all FR impressions (title was: S:RI should pyr... - https://phabricator.wikimedia.org/T45250#2667997 [17:08:45] Fundraising-Analysis, Analytics: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#2667994 (awight) Resolved>Open Unfortunately, I don't know of any progress towards this goal. We still need to run the expensive Hive query, repeatedly parsing... [17:09:17] Fundraising-Analysis, Fundraising-Backlog, Analytics, MediaWiki-extensions-CentralNotice: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#2667998 (awight) [17:09:54] Fundraising-Analysis, Fundraising-Backlog, Analytics, MediaWiki-extensions-CentralNotice: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#1064105 (awight) (see also T115042) [17:10:30] ejegg: ooh yeah let's do that. But did you all switch the nginx config for /paypal already? [17:10:44] I was thinking that's the safest way to provisionally deploy. [17:11:11] awight: no, we didn't switch anything nginx [17:11:33] do we need to? Or can we just point the site at the URL we've been using for testing? [17:11:47] awight: wouldn't rolling back on the pp console be easier? [17:11:51] esp since jgreen is out [17:11:54] It's best to keep the PP url stable, don't you think? [17:12:19] We have no control over how long that's cached for, and historical things are very possibly hardcoded to the old URL [17:12:29] you think so? [17:12:44] So at the least, we'll have to point apache config to redirect from paypal to the new code, anyway. [17:12:45] Are we actually sending it anywhere, or is it just set in the console? [17:12:47] Yeah I'm sure of it [17:12:55] For example, look at the weblog for fundraising.wmo [17:13:13] awight: are you able to deploy anything nginx or apache related? [17:13:16] That's a *very* legacy endpoint, that we can't disable for the same reason [17:13:19] no [17:13:20] We need JG [17:13:24] well crap [17:13:26] * awight reads about his vacation [17:13:32] ops offsite [17:13:44] Ugh [17:14:09] WElll. We can hack the code to do a redirect [17:14:23] * cwd shudders [17:14:25] A 302 temporary [17:14:34] * awight spits in the gutter [17:14:42] eww [17:14:44] awight: if the old listener is still in place does it matter if it's cachey? [17:15:01] cwd it does if the old listener can only write to stomp [17:15:08] cwd: uh the cache would be just of URL not response [17:15:35] awight: i'm not sure what you mean, caching the url on pp's side? [17:16:27] yeah exactly, it's permanently set in some large number of historical subscriptions [17:16:45] awight: wat [17:16:51] no... [17:16:53] So whatever happens, we aren't migrated until the /paypal endpoint is doing the new thing [17:17:05] you start a subscription and you can never change your IPN? [17:17:21] ejegg: depending on the parameters we sent, and on the version of the PP API we were hitting [17:17:38] The easiest proof is to read CRM host weblogs for *IPN [17:17:41] *IPN* [17:17:52] Those are all legacy subscriptions [17:17:57] blecch [17:18:37] yeah. And those are redirecting to the /paypal endpoint, so we'll need JG or a code workaround at that endpoint no matter what. [17:18:53] rats [17:19:05] That's why I'm thinking the best deployment is for /paypal to just do the new thing. It's also an instant rollback, just like swapping out a webroot [17:19:45] right [17:20:05] so, I guess the 302 is our only option right now [17:20:33] do we know if the PP api will do the right thing with 302? [17:21:30] yeah it does cos legacy endpoints are already doing so [17:22:10] (PS1) Awight: Redirect to new listener implementation [wikimedia/fundraising/PaymentsListeners] - https://gerrit.wikimedia.org/r/312853 [17:22:40] hehe, I do wonder how many redirects it can tolerate [17:22:58] X| [17:26:21] (CR) Ejegg: [C: 2] "Follow the white rabbit!" [wikimedia/fundraising/PaymentsListeners] - https://gerrit.wikimedia.org/r/312853 (owner: Awight) [17:27:18] Fundraising-Backlog, FR-Ingenico: Ingenico orphan rectifier failing with no logs - https://phabricator.wikimedia.org/T146677#2668075 (awight) [17:27:27] Shall I enfire? [17:28:16] lemme open some more windows [17:28:24] awight: I can take it from here if you want to examine the CN thing [17:29:00] ejegg: sure, thanks! [17:29:25] so we'll still be consuming paypal from activemq [17:29:25] oh ha, of course no gate-and-submit on that repo [17:29:36] ...right? [17:29:44] (CR) Ejegg: [V: 2] Redirect to new listener implementation [wikimedia/fundraising/PaymentsListeners] - https://gerrit.wikimedia.org/r/312853 (owner: Awight) [17:30:01] cwd for starters, yep [17:30:30] cool [17:30:34] then when that looks good, we turn on the donations mirroring from the frontend and try swapping the consumer [17:30:36] that job is "PayPal Queue Consume" ? [17:30:47] MBeat: heads-up that we're tampering with PayPal IPN today... the risks are of getting corrupted data in Civi, or missing transactions, especially for recurring donors. Missing transactions should still come in through the nightly audit processor, but corrupted ones would not be corrected. This won't affect how donors are being charged. [17:30:50] err.. we have one of those? [17:31:11] sorry no such job [17:31:17] read pending as paypal [17:31:38] what is the consumer for the legacy queue? [17:32:24] cwd just 'donations queue consume' [17:32:26] awight, ejegg - not sure if you saw that email but i turned dedupe off last night caseu fail mail [17:33:01] and made a ticket [17:33:04] cwd ah yeah, that was getting torrential [17:33:04] anyway it's still off [17:33:35] just don't wanna forget again :P [17:35:11] cwd: Was it all the jobs btw, or just one of them? [17:35:50] awight: i'm not sure, "unknown error" [17:36:31] Dang, we used to have debugging that would tell us the native MariaDB errorno [17:36:51] Anyway, it's some timeout-related thing I'm sure [17:37:00] yea [17:37:05] (hmm, timestamps would tell us) [17:37:16] Anyway, thanks for shutting it off before it broke our email [17:39:10] !log updated standalone paypal IPN listener from b4d77a991e100f97d98fcd72eaf03940a4e1845d to 21647c8f4b781b74ae2dc4377334410b4eed7e3c [17:39:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:39:27] ok, let's see if that works [17:39:29] the current paypal listener has no "step 1" right? [17:39:37] cwd: right [17:39:39] it writes right to the donation/recur queue [17:39:46] yeah (+refund) [17:39:53] ah yeah, thx [17:42:49] i see a couple in jobs-paypal [17:43:02] cwd hmm, missing_predecessor stuff in the 'error' log, but I don't think it's from stuff to the new listener [17:44:35] heh, we should rename the WIP job runner [17:44:47] oh, yeah we'll get lots of missing_predecessor won't we [17:44:49] should we? [17:45:01] what is missing predecessor? [17:45:34] it's usually a subscription charge coming in before the subscr_start message [17:45:39] awight: wait, why will we get a lot of those? [17:46:05] nuts [17:46:13] PayPal message verification failed [17:46:27] should rollback ASAP [17:46:27] did someone fire that job? [17:46:29] ah crud [17:46:32] rolling back [17:47:21] re: missing predecessor, I was imagining that people would make recurring donations through the frontend, and we'd get the donation message but not the subscr start--but that's not true. Both come through the IPN. [17:47:35] !log rolled back paypal IPN listener to b4d77a991e100f97d98fcd72eaf03940a4e1845d [17:47:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:47:51] awight: aren't we still queueing them in the same order? [17:48:40] ejegg: yeah I think I was wrong--also they won't be generated at all without a listener [17:48:41] cwd yeah I fired the job [17:48:44] ah ha, so there were 4 blank messages in redis [17:49:02] i don't know why but those were probably the 4 errors there [17:49:09] 5th one seems to have worked [17:49:13] oh crud, so it's NOT doing the right thing with the 302 redirect? [17:50:14] ejegg: well there was a good message in there too [17:50:18] after the 4 blanks [17:50:51] oh hey, I just got my TY email for that test donation [17:50:57] so... what were the blanks? [17:51:05] were they there already? [17:51:23] no idea [17:51:37] crap, have we tested more than 1 kind of message against paypal2? [17:51:42] they could have come from anywhere [17:51:57] ejegg: yeah we tested recurring [17:53:46] were the job queue messages totally empty, or just the payload was empty? [17:56:46] payload was 'p=listener/smashpig' [17:57:03] or, yknow, the valid json equivalent [17:57:12] Yeah totally clicks [17:57:54] so... roll forward again? [17:58:32] doing it [17:59:27] !log updated standalone paypal IPN listener from b4d77a991e100f97d98fcd72eaf03940a4e1845d to 21647c8f4b781b74ae2dc4377334410b4eed7e3c [17:59:38] * awight munches peanuts [17:59:38] we'll get a couple more now [17:59:56] lemme look at the jobs queue [18:09:32] !log rolled back paypal IPN listener to b4d77a991e100f97d98fcd72eaf03940a4e1845d [18:43:10] (PS2) Ejegg: Refactor common database code [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312614 [19:07:05] (CR) Awight: [C: 2] Fix all db accessors sharing a single instance [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312550 (https://phabricator.wikimedia.org/T141477) (owner: Ejegg) [19:07:52] (Merged) jenkins-bot: Fix all db accessors sharing a single instance [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312550 (https://phabricator.wikimedia.org/T141477) (owner: Ejegg) [19:14:57] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Operations, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2668421 (awight) I'm digging a rabbithole through the cache vaults, and found this interesting resul... [19:29:25] (PS1) Ejegg: Always throw exception on execute failure [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312862 [19:32:19] (CR) Ejegg: [C: -1] "oops, binding everything as param_str... dang" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312614 (owner: Ejegg) [19:37:52] !log enabling awight_test5 banner at 1% of nlwiki [19:37:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:39:14] (PS3) Ejegg: Refactor common database code [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312614 [19:41:51] nlwiki? whoa... [19:42:11] "Hello Amsterdam!" [19:42:26] right, guess you need traffic... [19:48:10] ejegg: awight AndyRussG XenoRyet meeting? [19:48:19] Whoops, time warp. [19:48:23] brt [19:49:39] eileen: are you trying to call in? [19:51:40] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Operations, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2668518 (awight) I'm using something closer to the original command for probing the message cache, a... [20:13:11] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Operations, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2668547 (spatton) @awight our typical practice is to disable a given campaign before we swap new ban... [20:17:06] ejegg: yah I went for 1% of 1% of the overall traffic... [20:17:48] ah, nice [20:25:07] awight: "While output buffering is active no output is sent from the script (other than headers)" [20:25:09] sigh [20:26:11] yeeeah, that writing was on the wall [20:26:49] PHP: recursive backronym for untestable punctuation clusterphuk [20:27:46] writing tests in php is a healthy outlet for the clever hacking impulse [20:28:10] baaahaha seriously. [20:28:25] It's music to the soul, to wiggle that coathanger [20:29:08] hold on just gonna nest a few ternaries [20:31:00] If anyone has time this patch should reduce the conflicts we are hitting https://gerrit.wikimedia.org/r/#/c/312182/ on preferred language - it may need to be broadened but just focusses on different versions of the 'same' language [20:32:01] eileen: ooh, did you happen to notice how i disabled dedupe yesterday? it was fail mailing [20:34:49] cwd yeah - I should turn that back on. You don't know if it was clashing with silverpop do you? [20:35:19] no idea, just "unknown error" messages from the db [20:35:52] ok that can happen when something else locks the DB [20:35:54] guess we could see if the export was running around 8:00pst? [20:36:14] yeah - it has a window of 3 hours when it doesn't run dedupe to avoid it [20:42:31] Just turned it back on & trying to run it [20:45:58] relocating... [20:48:21] (PS1) Awight: Cleanup: simplify in-memory cache invalidation [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/312933 [20:54:12] bd808: Can you tell who owns this repo? Or know how to reset the maintainer? https://scrutinizer-ci.com/g/wikimedia/mediawiki-core/ [20:55:25] * awight stares down EditPage [21:07:56] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Operations, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2668865 (awight) @spatton I've changed some things about the backend in order to help diagnose this... [21:11:00] awight: All I can tell you is that the owner is not me. :) [21:12:25] hehe that's a good start [21:12:58] * awight holds the bowl of tapas just out of reach O_o [21:14:01] wait, you must not be on European time :) [21:14:21] we don't really acknowledge devops here, at least when it comes to feeding staff small fish [21:14:23] awight: no, UTC-6 [21:19:19] (PS2) Ejegg: Always throw exception on execute failure [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312862 [21:19:21] (PS1) Ejegg: Error-check the rest of the db calls [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312938 [21:19:36] eh... those should be squashed [21:20:52] (PS3) Ejegg: Always throw exception on execute failure [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312862 [21:24:52] (PS2) Ejegg: Use ct_id to find completed, avoid race [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312563 (https://phabricator.wikimedia.org/T141477) [21:26:14] fr-tech ^^^ are non-trivial but pretty straightforward, could use CR to help the orphan rectifier do the right thing and help us not lose pending messages [21:28:24] Cool, I need a break from the l10n cache [21:28:35] sounds pretty gnarly [21:28:55] I'm not finding much... [21:34:58] (PS1) Ejegg: Payments-init consumer only deletes pending on fail [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312939 [21:39:01] (CR) Awight: [C: 2] "Good direction to go in!" (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312614 (owner: Ejegg) [21:39:54] (Merged) jenkins-bot: Refactor common database code [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312614 (owner: Ejegg) [21:42:11] (PS1) Cdentinger: Just respond 200 to GET requests [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312941 [21:45:12] KALX freaky in the good way. [21:46:02] that is some janky shit right there [21:46:26] my programming not the radio [21:46:29] i love the radio [21:46:31] hehe [21:46:38] downtempo Tuvan throat metal [21:47:35] heading to the homework help club, back in a couple hrs... [21:47:47] have fun! [21:48:57] rural new mexico has some excellent eclectic radio [21:49:08] send it along! [21:49:25] or line-of-sight only? [21:49:48] some of them probably stream [21:50:13] being on metered internet is actually nice for a lot of reasons [21:50:22] disincentive to dick around on youtube [21:50:30] and i listen to the radio pretty much exclusively [21:51:14] the only times it sucks are like...vagrant provision [21:52:16] I barely get the college station from this apartment. I live by the river... [21:52:33] Well, it's been undergrounded. but still it's a valley [21:52:37] but since most of my computer life is pushing text around, and since i've been calling in to most hangouts, i only use 100-200mb per day [21:53:08] and that still involves reading the news and stuff [21:53:13] You know your mom brings you cookies in that Thoreau's cabin. [21:53:23] i bet i could get by on dial up [21:53:41] aaargh, the latency, the everything... [21:53:52] mosh [21:53:56] ppp deflate [21:54:02] I'm basically on my knees with this "alternative" DSL provider though. [21:54:13] I get buffer underflows all day long on 128kbps radio [21:54:54] One block from the fiber, but there's no splice point for miles [21:55:00] * awight bluffs most jargon [21:55:01] it's hilarious how much better the verizon hotspot is than any wired internet around here [21:55:09] hehe [21:55:18] Same here. And I thought those were pine trees... [21:55:57] if it doesn't turn out to be giving us all cancer i think radio wireless is gonna be the independent revolution [21:56:56] consumer internet is sufficiently terrible to summon it [21:57:06] That and Wardenclyffe electrical resonance [21:59:02] eileen: are you free to call into this meeting? [21:59:14] (CR) Awight: [C: 2] "Everything is more readable!" (2 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312862 (owner: Ejegg) [21:59:58] (Merged) jenkins-bot: Always throw exception on execute failure [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312862 (owner: Ejegg) [22:05:34] bd808: I'm not sure why, but there it is... https://scrutinizer-ci.com/g/wikimedia/mediawiki/ [22:12:26] awight: if you have time to look https://gerrit.wikimedia.org/r/#/c/312941/ [22:12:45] it's a little gross, to the end of having the test actually test something [22:12:53] (CR) Awight: "Neat! Probably a fun candidate for https://etherpad.wikimedia.org/p/fr-tech-talk" (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312563 (https://phabricator.wikimedia.org/T141477) (owner: Ejegg) [22:14:21] imo it might be a little cleaner to have the consumer shuck them off [22:21:35] (CR) Awight: [C: -1] "I think we need to abort before inserting into the job queue." (3 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312941 (owner: Cdentinger) [22:24:10] whoops, good call [22:24:35] awight: i don't know about the GET assumption [22:25:22] it needs to at least hold for the other listners [22:25:50] they shouldn't touch any of this code [22:26:16] am i missing something? [22:26:41] ooh d'oh [22:26:59] Foiled by namespace :) [22:27:24] ah yeah, i figure if we could configure the messages we could shut the app down at a higher level [22:27:31] Technically, this is the only thing we can literally call an "IpnListener", if you want to get granular [22:27:44] ... since "IPN" is a PP trade name [22:27:45] cause ipn is a paypal trademark or what? [22:27:47] haha [22:28:13] It might increase the hilarity of future confoundings [22:29:04] but i am not even sure paypal will never GET [22:29:08] i have no idea [22:29:27] i feel like making the consumer throw them away might be cleaner [22:30:13] i have to go pick up a shower stall from home depot before the debates [22:30:17] back in a bit [22:32:14] I was thinking, let's send an unusual parameter for monitoring: plo.wmo/paypal?is_monitor=1 [23:00:23] Awight yes! Can we control that? [23:02:01] Definitely. But not for a week ;) [23:02:16] rebooting for pkgs... [23:02:44] (PS1) Awight: Cleanup: Use helper functions [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/312949 [23:23:22] (PS1) Eileen: Fix previous update query. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312951 (https://phabricator.wikimedia.org/T96410) [23:23:24] (PS1) Eileen: Update backoffice message template to allow receipt resends. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312952 (https://phabricator.wikimedia.org/T137496) [23:33:28] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Operations, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2669351 (awight) p:Unbreak!>High Reducing the priority because we're not actively losing ban...