[00:40:20] is anyone available to take banners down? [00:47:17] oh I see [00:47:31] I need to figure out how though [00:49:00] hey, sorry stepped afk [00:59:53] Fundraising Tech Backlog, Fundraising-Backlog, Patch-For-Review: another round of activemq queue decrufting needed - https://phabricator.wikimedia.org/T134916#2307614 (Jgreen) p:Triage>Unbreak! [01:03:10] Fundraising Tech Backlog, Fundraising-Backlog, Patch-For-Review: another round of activemq queue decrufting needed - https://phabricator.wikimedia.org/T134916#2307618 (Jgreen) Looks like the offending queue is banner-history which has been growing steadily since the weekend and is now up to 150K mess... [01:13:20] Fundraising Tech Backlog, Fundraising-Backlog, Patch-For-Review: another round of activemq queue decrufting needed - https://phabricator.wikimedia.org/T134916#2307622 (Jgreen) >>! In T134916#2307618, @Jgreen wrote: > Looks like the offending queue is banner-history which has been growing steadily sin... [01:23:29] Jeff_Green: hi, do banners still need to come down? [01:24:01] we got them down [01:24:22] so what we know: banner history consumer job got disabled on the 15th, and since then it's ramped up to 150K messages [01:24:36] awight says we can dump that entire queue [01:25:28] oof, yeah i think i may have done that? [01:25:40] i turned off a queue cause it was screaming recently [01:25:45] and promptly forgot about it [01:26:32] cwd: it has come back to haunt you [01:26:42] how are you at activemq admin? [01:27:09] i haven't been there in forever, lemme see... [01:27:31] yes i have creds, one moment [01:35:20] Wheee! [01:35:36] aloha! [01:35:41] Hehe [01:35:44] purging banner history now unless you scream [01:35:45] It's not like that [01:35:50] Please do [01:36:14] done [01:36:28] cwd: once the queue is purged I'd like to do a precautionary mq restart before we start banners [01:36:49] have we re-enabled the consumer? [01:36:52] probably unnecessary, but I don't have too much faith in its stability once its overheated. it's like a yugo [01:36:57] nope [01:36:59] hehe [01:37:02] unsafe at any speed [01:37:08] want me to turn it back on? [01:37:26] Cwd: the problem is our recent long messages will jam the consumer until we fix it [01:37:43] So it wouldn't consume anyway. Needs ex-lax [01:37:46] why are the messages long? did we change something? [01:41:28] restarting activemq [01:42:19] ok, we're back [01:42:50] cwd: Jeff_Green: thanks for jumping on this grenade! [01:43:00] heh [01:43:17] well, my fault to begin with [01:43:34] it's sort of fun to have something different to look at for once [01:43:58] cwd: Hardly! [01:44:05] what, instead of megabytes of pcre? [01:44:13] you don't like that? [01:44:23] I knew a lifetime of sketchy coding would lead to this, but still, I blame the Internet. [01:44:26] and a defective everything other than pcre [01:44:47] awight: so it's all Al Gore's fault? [01:45:14] I still want a "thanks for the internet, Al" shirt to wear double ironically [01:45:25] I taught him everything he knows about taking credit for things he had nothing to do with [01:48:11] awight: there are a couple patches for that issue [01:48:27] https://gerrit.wikimedia.org/r/289571 https://gerrit.wikimedia.org/r/289566 [01:48:39] first just turns on banner history module for test rig [01:48:52] second validates messages before trying to insert 'em [01:49:27] if you wanna CR I can push 'em out and turn the thing back on [01:49:32] or cwd ^^^ [01:51:06] ejegg: yeah looks sane to me [01:51:09] and tested even! [01:51:31] (CR) Cdentinger: [C: 2] Add banner history module to wmff setup [wikimedia/fundraising/civicrm-buildkit] - https://gerrit.wikimedia.org/r/289571 (owner: Ejegg) [01:51:34] hello fundraisers! can someone tell me how to force a current fundraising banner to appear? [01:51:50] +1 data validation woot [01:52:03] awjr: hi! [01:52:05] awjr sure! there's a page where you can find out what's current [01:52:09] * awjr waves [01:52:18] awjr: try the preview link on the banner editor page [01:52:19] and a couple of querystring params that will force it to show up [01:52:24] awight: linky? [01:52:27] (CR) Cdentinger: [C: 2] Validate banner history messages before insert [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/289566 (https://phabricator.wikimedia.org/T134916) (owner: Ejegg) [01:52:45] awjr: https://meta.wikimedia.org/wiki/Special:CentralNotice is a good starting point [01:52:59] ejegg: you submitted these this afternoon, did you have a premonition? [01:53:23] cwd Jeff filed a ticket, and I remembered the queue jams from earlier [01:53:33] awesome [01:53:37] ty awight and ejegg [01:54:03] (CR) jenkins-bot: [V: -1] Validate banner history messages before insert [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/289566 (https://phabricator.wikimedia.org/T134916) (owner: Ejegg) [01:54:22] X( [01:54:42] The 'add module' one that enables bh testing might need manual submit [01:55:17] (CR) Ejegg: [V: 2] Add banner history module to wmff setup [wikimedia/fundraising/civicrm-buildkit] - https://gerrit.wikimedia.org/r/289571 (owner: Ejegg) [01:55:18] ehh, I think it's not too taboo [01:55:30] hehe [01:55:35] did the 2nd depend on the 1st? [01:55:45] cwd yep, in a cross-repo way [01:55:50] goootcha [01:55:57] (CR) Ejegg: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/289566 (https://phabricator.wikimedia.org/T134916) (owner: Ejegg) [01:59:45] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/289584 [02:00:11] awjr: holler if you have trouble getting that banner preview, u may have noticed that our structure fire is only smoldering now ;) [02:00:15] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/289584 (owner: Ejegg) [02:00:33] awight: ;) i figured it out! thank you [02:00:45] Great! [02:02:20] !log updated civicrm from 7952ba43a012cb6a2e8d16af19bb13ed520bd56f to b7b46740d701942507dca0a98a75f3f87b6b31b1 [02:02:32] OK, I'm going to restart the consumer [02:02:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:03:33] the 'BANNER_HISTORY' wmfExceptions are of the sort that emails, so it'll be noisy if someone's still dumping garbage down that chute [02:04:23] should have a matching patch on the frontend I guess - these were cause I thought we'd chew through the queue one at a time [02:06:00] !log enabled banner history queue consumer [02:06:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:06:38] whew, that's some verbose logging [02:07:52] Jenkins build is back to normal :) [02:08:30] ejegg: on point fixes [02:09:05] apology for turning this off and forgetting about it [02:09:26] heh, no worries! [02:09:51] http://i.kinja-img.com/gawker-media/image/upload/s--YkJK8LUP--/lmhlajtdc8bi2l1spw5q.jpg [02:11:47] hehe [02:12:40] ok, i'm going back to my regularly scheduled loafing... see ya mañana [02:13:04] samesies, g'night! [02:41:25] Fundraising Sprint Ghostbusting , Fundraising Sprint Hermit Crab Husbandry, Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, and 2 others: Only subscribe primary emails, secondary addresses should be suppressed - https://phabricator.wikimedia.org/T131979#2307700 (Ejegg... [02:43:42] Fundraising Sprint Ghostbusting , Fundraising Sprint Hermit Crab Husbandry, Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, and 2 others: Only subscribe primary emails, secondary addresses should be suppressed - https://phabricator.wikimedia.org/T131979#2307701 (CCogd... [02:45:32] (PS1) Ejegg: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/289589 [02:45:47] (CR) Ejegg: [C: 2 V: 2] Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/289589 (owner: Ejegg) [02:47:51] !log updated misc fundraising tools from e2978024e6f6b6881d087ac5d07e4c40f7374709 to 220afdeaa36bc3feaaff1f781e7761d7878c4ee8 [02:48:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [03:07:59] (PS1) Ejegg: Fix improper merge [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/289591 [03:09:10] (CR) Ejegg: [C: 2 V: 2] Fix improper merge [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/289591 (owner: Ejegg) [03:10:29] !log updated fundraising tools from 220afdeaa36bc3feaaff1f781e7761d7878c4ee8 to b2425aef2154d6b689900f4848cca02880321230 [03:10:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [04:48:08] (PS1) Ejegg: Clean up some warnings [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/289592 [04:50:33] adyen download is 404, disabling job momentarily [05:49:13] (PS1) Ejegg: Fix state truncation warnings [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/289593 [12:22:13] fundraising-tech-ops: test jessie for payments servers, upgrade servers to jessie - https://phabricator.wikimedia.org/T135562#2308503 (Jgreen) Open>Resolved p:Triage>Normal [15:09:47] Fundraising-Backlog, FR-Fraud: Ingenico: massive surge in fraud attempts 5/19 - https://phabricator.wikimedia.org/T135745#2309361 (MBeat33) [15:13:04] (CR) Anomie: "Note that, under SessionManager, the session is always "set up". The call being moved is telling the session that it should last beyond th" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/282479 (https://phabricator.wikimedia.org/T132251) (owner: Florianschmidtwelzow) [15:18:46] Fundraising-Backlog, FR-Fraud: Ingenico: massive surge in fraud attempts 5/19 - https://phabricator.wikimedia.org/T135745#2309420 (MBeat33) [15:21:07] Fundraising-Backlog, FR-Fraud: Ingenico: massive surge in fraud attempts 5/19 - https://phabricator.wikimedia.org/T135745#2309424 (MBeat33) [15:23:38] Fundraising-Backlog, FR-Fraud: Ingenico: massive surge in fraud attempts 5/19 - https://phabricator.wikimedia.org/T135745#2309433 (MBeat33) [15:23:50] Fundraising-Backlog, FR-Fraud, FR-Ingenico: Massive surge in fraud attempts 5/19 - https://phabricator.wikimedia.org/T135745#2309434 (Danny_B) [15:36:21] T135745 big Ingenico fraud attempt spike today, if anyone has suggestions for countering that [15:36:21] T135745: Massive surge in fraud attempts 5/19 - https://phabricator.wikimedia.org/T135745 [15:59:35] (PS5) Florianschmidtwelzow: Replace usage of wfSetupSession() [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/282479 (https://phabricator.wikimedia.org/T132251) [16:33:34] hi cwd! If you're up for some more CR, these are all related to getting Adyen out in Japan and Ukraine https://gerrit.wikimedia.org/r/#/q/owner:Ejegg+status:open+project:mediawiki/extensions/DonationInterface++-age:3d,n,z [16:35:10] Fundraising-Backlog, FR-Fraud, FR-Ingenico: Massive surge in fraud attempts 5/19 - https://phabricator.wikimedia.org/T135745#2309361 (Ejegg) yikes, that's a heckofa spike. Let's take this ticket private so we can discuss specific countermeasures. [16:35:13] sure thing [16:35:21] i just ping you about something in privchan btw [16:35:23] also adyen [16:46:19] Fundraising-Backlog: Adyen audit download reporting 404 - https://phabricator.wikimedia.org/T135753#2309735 (cwdent) [16:56:17] (PS3) Cdentinger: Localize donation amount [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289066 (https://phabricator.wikimedia.org/T93196) (owner: Ejegg) [16:58:57] (CR) Cdentinger: [C: 2] "Looks great!" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289066 (https://phabricator.wikimedia.org/T93196) (owner: Ejegg) [17:00:19] Fundraising-Backlog: Adyen audit download reporting 404 - https://phabricator.wikimedia.org/T135753#2309785 (cwdent) Open>Invalid Eeeeeh I turned it back on and it worked. Whatever... [17:00:40] (Merged) jenkins-bot: Localize donation amount [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289066 (https://phabricator.wikimedia.org/T93196) (owner: Ejegg) [17:01:18] (PS2) Cdentinger: Pull out some inline styles [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289083 (https://phabricator.wikimedia.org/T135416) (owner: Ejegg) [17:01:32] Fundraising Sprint Jabberwock Slaying, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Mailing list export script chokes on 8-bit email addresses - https://phabricator.wikimedia.org/T135204#2291624 (Ejegg) Open>Resolved [17:01:34] Fundraising Sprint Ghostbusting , Fundraising Sprint Hermit Crab Husbandry, Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, and 3 others: Changing email address in Civi should be exported to Silverpop - https://phabricator.wikimedia.org/T122411#2309790 (Ejegg) [17:06:13] (CR) Cdentinger: [C: 2] Pull out some inline styles [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289083 (https://phabricator.wikimedia.org/T135416) (owner: Ejegg) [17:07:37] (PS6) Ejegg: Replace usage of wfSetupSession() [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/282479 (https://phabricator.wikimedia.org/T132251) (owner: Florianschmidtwelzow) [17:07:39] (Merged) jenkins-bot: Pull out some inline styles [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289083 (https://phabricator.wikimedia.org/T135416) (owner: Ejegg) [17:10:56] (PS2) Cdentinger: Configure Adyen for JP and UA [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289100 (https://phabricator.wikimedia.org/T135418) (owner: Ejegg) [17:13:52] (CR) Ejegg: [C: 2] "Anomie yeah, it was just a matter of 'looks weird'. Anyway, looks good now and works well locally." [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/282479 (https://phabricator.wikimedia.org/T132251) (owner: Florianschmidtwelzow) [17:15:32] (Merged) jenkins-bot: Replace usage of wfSetupSession() [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/282479 (https://phabricator.wikimedia.org/T132251) (owner: Florianschmidtwelzow) [17:17:18] (CR) Cdentinger: [C: 2] Configure Adyen for JP and UA [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289100 (https://phabricator.wikimedia.org/T135418) (owner: Ejegg) [17:18:39] (Merged) jenkins-bot: Configure Adyen for JP and UA [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289100 (https://phabricator.wikimedia.org/T135418) (owner: Ejegg) [17:40:24] (PS4) Cdentinger: Adyen iframe CSS for RTL languages [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289002 (https://phabricator.wikimedia.org/T135416) (owner: Ejegg) [17:43:24] (CR) Cdentinger: "Cool!" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289002 (https://phabricator.wikimedia.org/T135416) (owner: Ejegg) [17:43:31] (CR) Cdentinger: [C: 2] Adyen iframe CSS for RTL languages [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289002 (https://phabricator.wikimedia.org/T135416) (owner: Ejegg) [17:46:24] (Merged) jenkins-bot: Adyen iframe CSS for RTL languages [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289002 (https://phabricator.wikimedia.org/T135416) (owner: Ejegg) [18:10:41] (PS1) Ejegg: Filter on selection_weight=0 earlier [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289708 (https://phabricator.wikimedia.org/T135520) [18:11:01] thanks for all the CR cwd ! [18:11:16] sure thing! all looks good [18:32:55] cwd: that last one with the message language override ought to make sure we don't show any Russian messages in Ukraine [18:34:41] will check it out shortly [18:41:57] Fundraising Sprint Ghostbusting , Fundraising Sprint Hermit Crab Husbandry, Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, and 2 others: Only subscribe primary emails, secondary addresses should be suppressed - https://phabricator.wikimedia.org/T131979#2310142 (CCogd... [18:43:16] (PS1) Ejegg: Log number of rows exported [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/289714 [18:51:14] Fundraising Sprint Ghostbusting , Fundraising Sprint Hermit Crab Husbandry, Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, and 2 others: Only subscribe primary emails, secondary addresses should be suppressed - https://phabricator.wikimedia.org/T131979#2310167 (Ejegg... [19:00:03] Fundraising Sprint Ghostbusting , Fundraising Sprint Hermit Crab Husbandry, Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, and 2 others: Only subscribe primary emails, secondary addresses should be suppressed - https://phabricator.wikimedia.org/T131979#2310188 (Ejegg... [19:12:50] (PS1) Ejegg: Remove old email edit hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/289716 (https://phabricator.wikimedia.org/T122411) [19:14:30] Fundraising Sprint Ghostbusting , Fundraising Sprint Hermit Crab Husbandry, Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, and 2 others: Only subscribe primary emails, secondary addresses should be suppressed - https://phabricator.wikimedia.org/T131979#2310221 (Ejegg... [19:18:30] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Easy: Spike: investigate negative scroll offset for cancelling "banner bump" - https://phabricator.wikimedia.org/T107967#2310229 (TheDJ) Another CN campaign is running now, and there are lots of people complaining about jumping pages again. In... [19:30:07] (CR) Cdentinger: "Wouldn't it be simpler to just do:" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289708 (https://phabricator.wikimedia.org/T135520) (owner: Ejegg) [19:34:07] (CR) Ejegg: "The 'heaviest' logic actually already excludes selection_weight = 0, since it sets min_weight to 0 and only chooses if the form's is > min" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289708 (https://phabricator.wikimedia.org/T135520) (owner: Ejegg) [19:59:11] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Easy: Spike: investigate negative scroll offset for cancelling "banner bump" - https://phabricator.wikimedia.org/T107967#1508646 (AndyRussG) Hi @TheDJ... Can you tell me which campaign you're talking about? Maybe link to some specific complaint... [20:38:30] (CR) Jdlrobson: [C: -1] "needs rebase. still relevant?" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/78172 (owner: Mwalker) [20:43:26] (PS1) Ejegg: Disallow setting contribution tracking ID from request [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289731 (https://phabricator.wikimedia.org/T135745) [20:43:55] AndyRussG|ahweig: cwd mind trying that one out locally for a bit ^^^ ? [20:44:09] I'll bang on it too, look for unintended consequences [20:44:24] sure [20:45:35] (CR) jenkins-bot: [V: -1] Disallow setting contribution tracking ID from request [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289731 (https://phabricator.wikimedia.org/T135745) (owner: Ejegg) [20:47:29] ejegg: I'll look, but I don't have a local setup again yet :( [20:47:51] oh phooey, that's right [20:48:18] probably I should try to get a "persistent" live usb system... [21:27:55] ugh, some bizarre thing is happening locally where my special pages are all redirecting to themselves [21:28:01] at least it's just once... [21:28:28] weird... [21:33:47] nvm, AstropayGateway-> AstroPayGateway [21:33:55] gotta update my links [21:34:01] and probably some code on donatewiki [21:34:12] Wasn't that taken care of with the old alias thing? [21:43:25] i just fixed the case for that one recently [21:43:33] (PS2) Ejegg: Disallow setting contribution tracking ID from request [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289731 (https://phabricator.wikimedia.org/T135745) [21:43:42] ok cwd AndyRussG I think that one should be OK ^^^ [21:44:00] ugh, I hate to mess with session continuity... [21:44:33] Hmmm [21:45:04] should i be seeing a ct row just from loading the GC form for instance? [21:45:15] (CR) jenkins-bot: [V: -1] Disallow setting contribution tracking ID from request [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289731 (https://phabricator.wikimedia.org/T135745) (owner: Ejegg) [21:45:57] i have nothing in that table on vagrant [21:46:35] cwd yes, you should [21:46:44] hrmph [21:46:54] hrmph indeed... trying vagrant [21:52:06] ah dammit, ct_id is unstaged for some adapters... WTH? [21:54:19] this change does feel pretty complicated [21:54:34] good chance of unintended consequences [22:50:37] (PS1) Ejegg: WIP run some fraud filters on gateway ready [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289787 (https://phabricator.wikimedia.org/T135745) [22:52:38] (CR) jenkins-bot: [V: -1] WIP run some fraud filters on gateway ready [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289787 (https://phabricator.wikimedia.org/T135745) (owner: Ejegg) [23:10:54] (PS2) Ejegg: WIP run some fraud filters on gateway ready [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289787 (https://phabricator.wikimedia.org/T135745) [23:12:35] (CR) jenkins-bot: [V: -1] WIP run some fraud filters on gateway ready [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289787 (https://phabricator.wikimedia.org/T135745) (owner: Ejegg) [23:22:21] (PS3) Ejegg: WIP run some fraud filters on gateway ready [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289787 (https://phabricator.wikimedia.org/T135745) [23:23:41] (CR) jenkins-bot: [V: -1] WIP run some fraud filters on gateway ready [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289787 (https://phabricator.wikimedia.org/T135745) (owner: Ejegg) [23:32:50] (PS4) Ejegg: Run some fraud filters on gateway ready [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289787 (https://phabricator.wikimedia.org/T135745)