[01:11:15] anyone looking at the failmail storm? [01:11:41] oh - looks like a hacking attempt [01:11:43] [01:13:51] mepps: not sure if you are about but is there something we can / should do if someone is trying to hack [01:14:56] ejegg|away: ^^ cstone dstrine anyone about? [01:15:03] Yikes Eileen! Contact security? [01:15:16] yeah I dunno - I mean I don't think they are getting in [01:15:18] Also Jeff_Green [01:15:19] but they are trying [01:15:21] Oh ok [01:15:29] If you look at all that failmail [01:15:37] AndyRussG: [01:15:47] they are entering js into the payment_method box [01:16:01] I assume it isn't doing anything - they just keep trying & we could block their IP? [01:16:43] I can’t see it on my phone [01:17:12] Yeah blocking their ip is a good idea [01:17:23] mepps: do you know how to? [01:17:34] I do not Eileen [01:17:43] It must be in local settings [01:19:15] ok - do you want to text Jeff? I can but it's international [01:19:18] There’s documentation on wiki of how to shut stuff down if we need to [01:19:36] Yeah I’m trying to remember where the contact sheet is [01:19:43] https://office.wikimedia.org/wiki/Contact_list [01:21:20] ThNks! It’s tough on my phone [01:22:51] they might have just stopped - I was . on the verge of turning off all the queues [01:23:04] maybe I did turn them off [01:23:10] will clear emails & turn on & see [01:23:23] I texted him [01:24:01] grateful for having you in NZ right now nonetheless [01:25:14] lol [01:25:39] well queues are off now - I'll see if he responds before trying to turn them back on [01:25:47] Cool thanks [01:26:12] dstrine are you around by any chance? [01:26:34] Or MBeat? [01:26:44] hi mepps [01:27:00] big fraud attack! [01:27:14] Yeah [01:27:31] Just wanted to make sure you saw queues are off [01:27:32] https://collab.wikimedia.org/wiki/Fundraising/Engineering/Shutting_the_pipeline_down#Choosing_how_to_shut_down [01:27:38] thank you [01:28:03] I looked for all the activity on the IP in fredge, doesn’t look like anything got through as fraud [01:28:04] Hmm Eileen do we know what processor this was? [01:28:10] Oh good [01:28:13] seems to be ingenico [01:28:41] Okay if it doesn’t get through as fraud are we okay to leave ingenico up as far as you’re concerned MBeat? [01:29:15] yes! we just sent out a batch of Japan emails, so I’m happy to monitor & not block the JP donors - thank you for asking [01:29:27] Eileen I might text dstrine too, seems like something he’d want to know [01:29:49] ok - I'm just creating a phab [01:30:55] not much detail yet but https://phabricator.wikimedia.org/T228822 [01:32:05] Great, thanks for being on this [01:32:33] It’s hard from my phone! I can get my computer if needed though [01:32:38] I'm not too sure whether they stopped or whether my stopping the . queues happened first [01:33:29] SO I guess as long as they are all from one IP then we just have someone hammering away - or scripting away at our form [01:33:40] I haven’t heard from Jeff [01:33:53] Yes that makes sense [01:34:27] I think if we block that ip, we could turn the queues on and see what happens [01:35:55] yeah - I guess I'll go trawling for how to block an ip [01:37:14] I was just searching collab [01:37:37] hmm the amount of donations queued via Ingenico seems static at 20 https://frmon.frdev.wikimedia.org/d/Pq1YNMviz/fundraising-overview?refresh=1m&orgId=1&panelId=15&fullscreen [01:37:44] it might have stopped on it's own [01:38:49] Do you want to turn them on? I can join on watching the failmail [01:40:10] yeah I'll do it now [01:41:47] ah ejegg|away was talking to us on the other channel & I missed it [01:41:58] anyway we are back on [01:42:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 2152 2000 - REDIS 3.2.6 on 127.0.0.1:6379 has 1 databases (db0) with 7 keys, up 5 days 6 minutes - memory use is 3.40M (peak 3.63M, 0.08% of max, fragmentation 1.93%), connected_slaves is 2, donations is 224, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 181, pending is 0, recurring is 2, refund is 0, unsubscribe is 2 [01:42:23] ty eileen [01:42:58] Oh ha I missed it too [01:44:18] My biggest learning from this is that the wiki office contact list is not super phone accessible and that we need better documentation of how to block ips [01:47:14] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 3.2.6 on 127.0.0.1:6379 has 1 databases (db0) with 7 keys, up 5 days 11 minutes - memory use is 1.90M (peak 3.63M, 0.06% of max, fragmentation 2.55%), connected_slaves is 2, donations is 12, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 73, payments-init is 12, pending is 0, recurring is 3, refund is 0, unsubscribe is 2 [01:50:48] mepps: yeah - we should probably just all add each other's numbers to our phones [01:51:11] also I am ASSUMING that the hack attempts were futile & nothing useful could be achieved by it [01:51:18] Agreed. It turns out I had Jeff’s number but forgot I’d added it [01:51:43] I guess how it COULD have worked is IF the page we used to view them didn't escape them we could have run malicious script [02:01:03] !log payments-wiki revision changed from 224c6b2d7b to a28ad541ed, config revision is 8dcb77cf22 [02:01:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:37:58] Fundraising Sprint Men In Slack, Fundraising Sprint Never Ending Query, Fundraising Sprint Office  , Fundraising-Backlog, and 4 others: New total fields needed for MG prospecting - https://phabricator.wikimedia.org/T222958 (Eileenmcnaughton) Open→Resolved [08:38:03] Fundraising Sprint King Kong vs. Mozilla, Fundraising Sprint Land before Timeouts, Fundraising Sprint Men In Slack, Fundraising Sprint Never Ending Query, and 3 others: Add calendar year totals fields to CiviCRM & export ? some? of them to silverpop - https://phabricator.wikimedia.org/T170972 (Eil... [09:23:43] Fundraising-Backlog: Update email copy for the endowment Thank you email - https://phabricator.wikimedia.org/T228843 (jrobell) [15:11:07] fr-tech any news for scrum of scrums? [15:14:38] non from me [15:45:31] (PS2) Ejegg: WIP deal with gateway_txn_id in a standard way [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/524535 [15:47:27] (CR) jerkins-bot: [V: -1] WIP deal with gateway_txn_id in a standard way [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/524535 (owner: Ejegg) [15:48:41] almost! [16:04:06] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/525314 [16:04:43] cstone: did you see the failmail about the audit file failing? Might have something to do with the new India payment methods [16:04:52] I haven't taken a look yet [16:06:16] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/525314 (owner: Ejegg) [16:07:32] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/525314 (owner: Ejegg) [16:08:34] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/525315 [16:08:43] (CR) Ejegg: [C: +2] Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/525315 (owner: Ejegg) [16:10:36] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/525315 (owner: Ejegg) [16:13:32] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), Performance-Team (Radar): Remove hacky EventLogging-duplicating code and use new lightweight EL facilities - https://phabricator.wikimedia.org/T214709 (Krinkle) [16:13:42] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), Performance-Team (Radar): Remove hacky EventLogging-duplicating code and use new lightweight EL facilities - https://phabricator.wikimedia.org/T214709 (Krinkle) p:Triage→Normal [16:17:30] (CR) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/525315 (owner: Ejegg) [16:21:11] (PS3) Ejegg: Banner history logger: remove loading of schema module [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/523853 (https://phabricator.wikimedia.org/T227711) (owner: AndyRussG) [16:21:20] (CR) Ejegg: [C: +2] "Looks good!" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/523853 (https://phabricator.wikimedia.org/T227711) (owner: AndyRussG) [16:25:25] (Merged) jenkins-bot: Banner history logger: remove loading of schema module [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/523853 (https://phabricator.wikimedia.org/T227711) (owner: AndyRussG) [16:31:20] (CR) jenkins-bot: Banner history logger: remove loading of schema module [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/523853 (https://phabricator.wikimedia.org/T227711) (owner: AndyRussG) [16:40:48] (CR) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/525315 (owner: Ejegg) [16:58:27] (CR) Cstone: WIP: Add recurring token support to creating a subscription. (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/524909 (https://phabricator.wikimedia.org/T227048) (owner: Cstone) [17:05:13] Fundraising-Backlog: Portuguese TY email: matching gift link error - https://phabricator.wikimedia.org/T209506 (TSkaff) I just sent myself test emails in pt and pt-br and the format looks ok and the link functions. But I wonder whether: [$matchingUrl programa de doações deste género] should be $matchingU... [17:06:47] (PS3) Ejegg: Add gateway_txn_id and subscr_id to subscr_start message [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/524538 (https://phabricator.wikimedia.org/T216560) [17:06:58] cstone ^^^ now adds the subscr_id [18:15:16] !log updated payments-wiki from a28ad541ed to 70b432d309 [18:15:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:22:17] ok, looks like new-API card donations still go through [18:23:14] and old-API ones don't get a hosted payment page [18:23:16] cool [18:33:34] ah ejegg one last thing for the installments, do you want to send any of that in the msg or should I just set it to 0 in the insert? [18:46:02] cstone: you can set it to 0 in the insert [18:46:10] ok thanks! [18:46:35] hmm, looks like XenoRyet's crm/vendor submodule update didn't actually include the SmashPig update [18:46:49] that's why the last dlocal audit parse run failed [18:46:58] lemme try to set that straight [18:48:43] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi: TY email not sent for recurring donation 7/22 - https://phabricator.wikimedia.org/T228922 (MBeat33) [18:49:56] hmm, what's the matter here [18:49:58] ? [18:52:32] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi: TY email not sent for recurring donation 7/22 - https://phabricator.wikimedia.org/T228922 (MBeat33) p:Triage→High Hoo boy, I see hundreds of recurring donations from the 22nd without TY email send time stamps. [18:52:48] the packagist page shows sha 3dbd [18:53:07] the repo shows tag v0.5.10 as commit 3dbd [18:53:27] but composer install keeps cloning 5a056c7, which is v0.5.9 [18:53:32] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi: TY emails not sent for recurring donation 7/22 - https://phabricator.wikimedia.org/T228922 (MBeat33) [18:54:36] ahh, composer.lock issue? [18:55:48] yep, somehow the sha change didn't get checked in [18:57:43] (PS1) Ejegg: Fix SmashPig version [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/525342 [19:04:04] ejegg, were previous recurring donations always the same amount as the initial contribution? [19:04:20] jgleeson: oh yeah, that too! [19:04:40] cool, I wanted to make sure before adding comments [19:10:30] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi: TY emails not sent for recurring donation 7/22 - https://phabricator.wikimedia.org/T228922 (MBeat33) Open→Resolved a:MBeat33 Cleared up after checking w/fr-tech, this is from the rollout of the new recurring TY process. Sorry for the a... [19:16:34] ejegg, are we using the effort_id as a counter for installments also? [19:16:53] in the queue context [19:17:15] jgleeson: yeah, that's kind of a holdover from the old globalcollect [19:17:23] got it [19:24:41] (CR) Ejegg: [C: +2] Fix SmashPig version [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/525342 (owner: Ejegg) [19:25:14] (PS1) Ejegg: Fix SmashPig update [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/525345 [19:25:27] (PS4) Jgleeson: Update SmashPig recurring processor to handle making first payments. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/525098 (https://phabricator.wikimedia.org/T227051) [19:25:37] (CR) jerkins-bot: [V: -1] Fix SmashPig update [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/525345 (owner: Ejegg) [19:29:39] (CR) Cstone: [C: +2] "Sending through the new parameters nicely!" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/524538 (https://phabricator.wikimedia.org/T216560) (owner: Ejegg) [19:29:52] (Merged) jenkins-bot: Fix SmashPig version [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/525342 (owner: Ejegg) [19:30:24] (PS2) Cstone: WIP: Add recurring token support to creating a subscription. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/524909 (https://phabricator.wikimedia.org/T227048) [19:30:42] (Merged) jenkins-bot: Add gateway_txn_id and subscr_id to subscr_start message [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/524538 (https://phabricator.wikimedia.org/T216560) (owner: Ejegg) [19:30:59] (PS2) Ejegg: Fix SmashPig update [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/525345 [19:34:38] (CR) Ejegg: [C: +2] Fix SmashPig update [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/525345 (owner: Ejegg) [19:35:18] (Merged) jenkins-bot: Fix SmashPig update [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/525345 (owner: Ejegg) [19:37:13] (PS1) Ejegg: Update sha for smash-pig in composer.lock [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/525350 [19:37:23] (CR) Ejegg: [C: +2] Update sha for smash-pig in composer.lock [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/525350 (owner: Ejegg) [19:42:29] (Merged) jenkins-bot: Update sha for smash-pig in composer.lock [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/525350 (owner: Ejegg) [19:43:08] !log updated fundraising CiviCRM from 875ab97742 to 121feb5d53 [19:43:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:45:55] OK, that should make the dlocal audit process correctly today [19:48:07] fr-tech I've got to go catch a bus before they close the roads! back online in a few hours [20:46:43] (CR) jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/525378 (owner: L10n-bot) [23:31:28] Fundraising Sprint Never Ending Query, Fundraising Sprint Office  , Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Finalise & test field mapping - https://phabricator.wikimedia.org/T228712 (LeanneS) This is great, thanks @Eileenmcnaughton! Yeah they do have some extra/unneeded columns i... [23:32:55] Fundraising Sprint Never Ending Query, Fundraising Sprint Office  , Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Finalise & test field mapping - https://phabricator.wikimedia.org/T228712 (Eileenmcnaughton) @LeanneS the 'full_address' is just street_address+supplemental_address+ supple... [23:36:20] Fundraising Sprint Never Ending Query, Fundraising Sprint Office  , Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Finalise & test field mapping - https://phabricator.wikimedia.org/T228712 (LeanneS) @Eileenmcnaughton No, for some reason, the street address is called that in their file....