[01:09:37] (PS1) Ejegg: Add GatewayInitialFilter hook [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289801 [01:11:05] (CR) jenkins-bot: [V: -1] Add GatewayInitialFilter hook [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289801 (owner: Ejegg) [01:13:28] (PS2) Ejegg: Add GatewayInitialFilter hook [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289801 [01:14:46] (CR) jenkins-bot: [V: -1] Add GatewayInitialFilter hook [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289801 (owner: Ejegg) [01:22:24] (PS3) Ejegg: Add GatewayInitialFilter hook [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289801 [01:23:42] (CR) jenkins-bot: [V: -1] Add GatewayInitialFilter hook [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289801 (owner: Ejegg) [01:36:25] (PS4) Ejegg: Add GatewayInitialFilter hook [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289801 [01:38:23] cwd|afk: when you're back, a smaller version of that patch is passing tests [01:41:26] (PS5) Ejegg: Add GatewayInitialFilter hook [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289801 [01:44:41] (PS1) Ejegg: Persist risk score array in session [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289804 [02:09:01] (PS1) Ejegg: Fire IP velocity filter early [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289809 [02:10:48] (PS2) Ejegg: Fire IP velocity filter early [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289809 [02:20:50] (PS6) Ejegg: Add GatewayInitialFilter hook [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289801 [02:27:35] (PS2) Ejegg: Persist risk score array in session [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289804 [02:30:14] (PS3) Ejegg: Persist risk score array in session [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289804 [02:31:29] (PS3) Ejegg: Fire IP velocity filter early [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289809 [02:33:16] (Abandoned) Ejegg: Run some fraud filters on gateway ready [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289787 (https://phabricator.wikimedia.org/T135745) (owner: Ejegg) [05:14:29] Fundraising-Backlog: Adyen smashpig job runner failing - https://phabricator.wikimedia.org/T135803#2311575 (cwdent) [08:45:18] !log Update payments fraud config [08:45:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [09:22:56] (CR) Awight: "How will we deal with shared IP addresses, like universities?" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289809 (owner: Ejegg) [09:24:21] (CR) Awight: "Does this relatively benefit attackers who purge their session cookie?" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289804 (owner: Ejegg) [11:44:37] Fundraising-Backlog, Hovercards, Reading-Web-Backlog: Avoid z-index conflict with HoverCards & Central Notice and friends - https://phabricator.wikimedia.org/T131364#2312236 (dr0ptp4kt) Open>Invalid [15:02:55] (CR) Ejegg: "No more than the current setup - the GatewayReady hook should fire for each reqest, so we can still run all the filters on the request tha" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289804 (owner: Ejegg) [15:07:35] (CR) Ejegg: "same as now, I guess, it'll just be slightly easier for one jerk to penalize the whole bunch." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289809 (owner: Ejegg) [15:52:43] Fundraising Sprint Ghostbusting , Fundraising Sprint Hermit Crab Husbandry, Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, and 2 others: Only subscribe primary emails, secondary addresses should be suppressed - https://phabricator.wikimedia.org/T131979#2313005 (Ejegg... [15:54:37] (PS4) Ejegg: WIP popAtomic for Stomp [wikimedia/fundraising/php-queue] - https://gerrit.wikimedia.org/r/288037 [16:03:20] ejegg: ty for emailing adyen [16:03:55] cwd I wonder if there's a bigger problem though - so many missing pending messages too [16:04:11] Maybe we're not dropping them in the queue for some reason? [16:04:37] how did you notice missing pending messages? [16:04:44] i was not aware of that part [16:05:51] some of the failmails [16:06:24] the log says they look for a pending message, don't find it, and try to cancel [16:06:50] (the cancel code needs a fix deployed - there's quite a deploy backlog there!) [16:08:20] i don't think i see these messages, all the fail mails i'm finding are complaining about the report 404 [16:11:29] they say it's not a known issue, maybe we're looking in the wrong place or something? [16:12:09] cwd this one hsa your address on the to list - FAILMAIL -- ALERT -- (adyen) SmashPig-Adyen-StompJobRunner::SPCID-092754XXXX::corr_id-adyen-36353723.1 [16:13:05] cwd huh, I'll try moving the job back to the queue and see if it works now. [16:13:39] nice speedy reply [16:14:35] yesterday it came back to life right away [16:15:43] cwd you re-queued the report message? [16:16:53] ah no i just turned it back on and it said back to normal [16:17:02] i get it [16:17:13] so then when we got the message again and still couldn't get the same report [16:17:44] oh huh, i don't see any jobs in the damaged message queue [16:17:48] does that mean this same queue is responsible for processing orders and downloading the report? [16:18:09] yeah, all the adyen jobs are in the jobs queue [16:18:46] ohh, i think it didn't get sent to the damaged queue because of that exception after the cancel call [16:19:08] yeah, looks like it bombed first [16:19:44] dang, bunch of astropay donations in the damaged queue with 'missing predecessor', probably related to the queue timeout. Should be possible to recreate those from logs... [16:22:44] Fundraising Sprint Jabberwock Slaying, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Smashpig, Unplanned-Sprint-Work: Re-create pending queue messages for AstroPay donations - https://phabricator.wikimedia.org/T135856#2313052 (Ejegg) [16:23:13] man everything is on fire this week [16:23:31] some of it is a cascade... [16:23:58] what was at the top? [16:24:43] some jerk dumping invalid data into the banner history queue, i think [16:24:59] ooh yeah, that camel cased sql [16:26:07] ok, I think I can snag that AstroPay data [16:26:26] if you have any time, I'd love more CR on stuff [16:26:41] got that fraud patch broken into 3 and improved a bunch [16:26:59] also the language fallback override is probably high priority [16:27:40] sure, i will find some time [16:28:30] thansk! [16:51:37] Fun times! [16:52:02] Aaarg! [16:52:06] stimulating, that's for sure [16:52:12] could be worse [16:52:15] though I wouldn't want it to keep going [16:52:30] back in a little while aaaaaaarg arrrrrgggg arrrggh [17:28:47] Fundraising Tech Backlog, Fundraising-Backlog, Design-Research-Backlog, MediaWiki-extensions-DonationInterface, Design Research and product team collaboration: How can we create the best mobile credit card experience? - https://phabricator.wikimedia.org/T96962#2313271 (ggellerman) Removing WM... [17:29:00] Fundraising Tech Backlog, Fundraising-Backlog, MediaWiki-extensions-DonationInterface: How can we create the best mobile credit card experience? - https://phabricator.wikimedia.org/T96962#2313274 (ggellerman) [17:29:54] (PS1) Ejegg: Not using referrer from messages [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/289887 [17:33:16] (CR) jenkins-bot: [V: -1] Not using referrer from messages [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/289887 (owner: Ejegg) [17:34:11] (PS1) Ejegg: Remove referrer from queue messages [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289888 [17:37:16] (CR) jenkins-bot: [V: -1] Remove referrer from queue messages [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289888 (owner: Ejegg) [17:38:41] (PS2) Ejegg: Not using referrer from messages [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/289887 [17:41:42] Fundraising Sprint Jabberwock Slaying, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Astropay, and 2 others: Re-create pending queue messages for AstroPay donations - https://phabricator.wikimedia.org/T135856#2313320 (Danny_B) [17:44:54] (PS2) Ejegg: Remove referrer from queue messages [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289888 [18:38:56] (PS1) Ejegg: Custom IPVelocity settings per netblock [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289893 [18:41:17] (PS2) Ejegg: Custom IPVelocity settings per netblock [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289893 [18:46:08] gotta get me some eats... [19:04:18] (CR) AndyRussG: Fire IP velocity filter early (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289809 (owner: Ejegg) [19:07:19] (CR) Ejegg: Fire IP velocity filter early (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289809 (owner: Ejegg) [19:08:36] (CR) AndyRussG: Fire IP velocity filter early (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289809 (owner: Ejegg) [19:09:37] ejegg|food: hey... Which patch is curently the most urgent? Also, would you like a rubber duck to talk this over at? LMK if that'd be useful for you, also would help me understand more details... [19:11:29] AndyRussG: actually, there's a message language override patch id like to get through [19:11:43] its th last one for the adyen batch [19:13:22] https://gerrit.wikimedia.org/r/289117 [19:13:53] but of the security ones,just the firat 2 would be nice [19:15:15] that's https://gerrit.wikimedia.org/r/289801 [19:15:34] and https://gerrit.wikimedia.org/r/289804 [19:19:47] K thx [19:29:07] Ok, back from food [19:29:56] AndyRussG: talking through this stuff sounds good - can I hangout-call you? [19:31:46] fundraising-tech-ops: Frack (boron and bismuth) access for Darian Patrick - https://phabricator.wikimedia.org/T135165#2313653 (dpatrick) ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVTBGOHReZ/qPMpLnUCr99Xu+lgMBF7Z1cnzPiChhesVe9p+2/ieIRvn7ZHCl+7URoVaSRuW8khetQDRnr1YBlFM4PcBolD8/lBst/90Y6Mjdy7cEOKRXpt6RxT5iDqpJRgFAR... [19:54:38] Fundraising Sprint Jabberwock Slaying, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Astropay, and 2 others: Re-create pending queue messages for AstroPay donations - https://phabricator.wikimedia.org/T135856#2313722 (Ejegg) It was possible, but a bit tedious. Need to script out much of... [19:57:35] ejegg: yeah! ooopse though, I just saw your ping (irc pings not configured to make a noise on this live usb system rrrrg!) [19:58:00] Maybe a little after standup? [19:58:01] ah, no worries, maybe stay on the line after standup [19:58:05] totally [19:58:30] I may skip BS time to go pick up my daughter (just around the corner) [19:58:36] ah, cool [19:58:56] K thx :) [20:01:36] Fundraising Sprint Jabberwock Slaying, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Astropay, and 2 others: Re-create pending queue messages for AstroPay donations - https://phabricator.wikimedia.org/T135856#2313726 (Ejegg) Oh hey, the audit parser should pick these up... [20:28:29] (PS4) Ejegg: Fix time and message limits [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/288449 (https://phabricator.wikimedia.org/T133965) [20:31:58] ejegg: cwd: back! [20:32:14] Are u still in the standup call? [20:32:59] joinin yonder hangout now [20:33:02] AndyRussG: there's a new one [20:33:06] in yr emails [21:23:59] (CR) jenkins-bot: [V: -1] Fix time and message limits [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/288449 (https://phabricator.wikimedia.org/T133965) (owner: Ejegg) [21:39:16] (PS2) Ejegg: Filter on selection_weight=0 earlier [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289708 (https://phabricator.wikimedia.org/T135520) [21:40:10] ok AndyRussG, that's slightly simplified, with some fixmes ^^^ [21:40:51] relocating... [21:51:30] ejegg: the more i think about it, fail2ban seems like the perfect thing for this. if we had it watching for instance some payments logs, we could feed it data from the app, which can make a much better analysis of traffic than a firewall, but when it decides to punt stuff it is done early, with iptables [21:54:18] (PS5) Ejegg: Fix time and message limits [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/288449 (https://phabricator.wikimedia.org/T133965) [21:56:08] https://integration.wikimedia.org/zuul/ [21:56:12] sux to be zuul today [21:59:01] (PS1) Ejegg: Remove referrer from DI messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/289966 [22:00:49] (PS3) Ejegg: Remove referrer from queue messages [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/289888 (https://phabricator.wikimedia.org/T110564) [22:01:09] (PS3) Ejegg: Not using referrer from messages [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/289887 (https://phabricator.wikimedia.org/T110564) [22:01:21] (PS2) Ejegg: Remove referrer from DI messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/289966 (https://phabricator.wikimedia.org/T110564) [22:10:22] (PS1) Ejegg: Merge remote-tracking branch 'origin/REL1_25' into fundraising/REL1_25 [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/289967 [22:42:55] (CR) Ejegg: [C: 2] Merge remote-tracking branch 'origin/REL1_25' into fundraising/REL1_25 [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/289967 (owner: Ejegg) [22:43:56] (PS1) Ejegg: Update vendor and composer.lock [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/289971 [22:50:14] (CR) jenkins-bot: [V: -1] Merge remote-tracking branch 'origin/REL1_25' into fundraising/REL1_25 [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/289967 (owner: Ejegg) [22:52:49] yowza, what is that about [22:53:07] didn't run the phpunit tests the first time for some reason [22:53:54] certainly does look like some breaking changes [22:56:10] breaking their tests anyway... [22:56:45] yeah...confusing [22:57:19] grr, not any real diff between us and REL1_25 [22:59:08] damn, what the heck is the difference here? [22:59:51] i was just thinking, why do we bother pinning to a different branch? [23:00:11] i mean i can imagine why, but it might be easier to do rolling updates [23:01:09] cwd yeah, just pull each week or something? [23:01:51] yeah something like that [23:02:24] waitaminit - the other branch must have been force merged https://gerrit.wikimedia.org/r/#/c/289896/ [23:02:24] these commits go back to october so there aren't very many [23:02:32] we might just be able to pull when we deploy [23:02:39] yep, forced by MaxSem [23:02:46] ugg [23:03:27] but it's not those patches breaking our tests... [23:03:47] argh, i just wanna get this out [23:04:30] the tests take so agonizingly long to run [23:04:31] oh hey, test for our core branch are running under php55 [23:06:08] indeed [23:06:25] was able to get it wired of for DI, probably wouldn't be that much work... [23:06:54] I'm running some test groups locally to see what's up [23:07:04] you wanna ask releng for an assist? [23:08:14] and... only a couple failures locally, but different [23:08:25] dang [23:08:30] sure i'll ping tyler [23:12:21] bombs for me immediately: MWException from line 886 of /srv/vagrant/mediawiki-fr/includes/db/Database.php: DatabaseBase::factory no viable database extension found for type 'mysql' [23:38:07] (PS1) Ejegg: Merge 1.25.6 updates [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/289974 [23:38:29] (CR) Ejegg: [C: 2] Merge 1.25.6 updates [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/289974 (owner: Ejegg) [23:42:42] (CR) jenkins-bot: [V: -1] Merge 1.25.6 updates [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/289974 (owner: Ejegg) [23:54:33] fundraising-tech-ops: Frack (boron and bismuth) access for Darian Patrick - https://phabricator.wikimedia.org/T135165#2314474 (dpatrick) Open>Resolved I have access now, so we can call this done.