[00:20:54] (Merged) jenkins-bot: Add phan-taint-check-plugin [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/451291 (owner: Umherirrender) [13:23:58] Fundraising Sprint Pluto is still a planet, Fundraising-Backlog, Fr-Ingenico-integration_2017-18: implement IDEAL in new ingenico integration - https://phabricator.wikimedia.org/T201626 (jgleeson) a:jgleeson>None [13:25:28] Fundraising Sprint Owls, Fundraising Sprint Pluto is still a planet, Fundraising-Backlog: Queries and maybe scripts to verify equivalence of data in new-Kafka-pipeline-testing and pgehres production databases - https://phabricator.wikimedia.org/T198752 (jgleeson) a:jgleeson [13:33:18] Fundraising-Backlog, fundraising-tech-ops: Katie Horn fundraising offboarding - https://phabricator.wikimedia.org/T201830 (Jgreen) [13:33:41] Fundraising-Backlog, fundraising-tech-ops: Katie Horn fundraising offboarding - https://phabricator.wikimedia.org/T201830 (Jgreen) FR cluster shell account: done [13:34:01] Fundraising-Backlog, fundraising-tech-ops: Katie Horn fundraising offboarding - https://phabricator.wikimedia.org/T201830 (Jgreen) Fundraising root@ mail: done [13:35:02] Fundraising-Backlog, fundraising-tech-ops: Katie Horn fundraising offboarding - https://phabricator.wikimedia.org/T201830 (Jgreen) fundraising database: done [13:35:55] Fundraising-Backlog, fundraising-tech-ops: Katie Horn fundraising offboarding - https://phabricator.wikimedia.org/T201830 (Jgreen) dev database: done [13:58:09] (PS1) Jgleeson: WIP: Updated Ingenico CANCELLED_BY_CONSUMER mapping to now map to FAILED [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/452366 [14:27:38] Fundraising-Backlog, fundraising-tech-ops: Katie Horn fundraising offboarding - https://phabricator.wikimedia.org/T201830 (Jgreen) remove bastion user-level iptables policy: done remove yubikey: done remove ssh public key: done remove ssl client cert: done [14:29:28] Fundraising Sprint Pluto is still a planet, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Banner previews occasionally fail to load - https://phabricator.wikimedia.org/T200853 (Pcoombe) @AndyRussG This just happened again for me with https://en.wikipedia.org/wiki/Bricolage?banner=spatton_d... [15:19:09] Fundraising Sprint Pluto is still a planet, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Banner previews occasionally fail to load - https://phabricator.wikimedia.org/T200853 (AndyRussG) Fantastic, thanks @Pcoombe!!! [15:26:56] Fundraising-Backlog, fundraising-tech-ops: Katie Horn fundraising offboarding - https://phabricator.wikimedia.org/T201830 (Jgreen) @Eileenmcnaughton, @Ejegg - could you disable Katie's civicrm user? [15:31:39] Fundraising-Backlog, fundraising-tech-ops: Katie Horn fundraising offboarding - https://phabricator.wikimedia.org/T201830 (Ejegg) Civi account blocked, and removed from admin and fr-tech groups. :( [15:31:54] ^^ so sad! [15:34:08] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Don't use freegeoip service by default anymore - https://phabricator.wikimedia.org/T201753 (DStrine) [15:50:58] (CR) Ejegg: [C: -1] "Let's do this a little differently. It's nice for 'canceled' to have its own status code for diagnostic purposes. We could change the mapp" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/452366 (owner: Jgleeson) [15:52:20] (CR) Ejegg: [C: -1] "Also, try setting $wgDonationInterfaceRapidFail = true in your localsettings and see if that makes it render a failure form locally." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/452366 (owner: Jgleeson) [16:27:34] hi awight! [16:54:23] saurabhbatra: Congratulations! I saw over the weekend that you’d put an API together, but haven’t had time to dig in yet. [16:54:54] Impressive timing ;-) [16:56:24] I’m not sure what the most useful next steps will be, I consider the project to be complete, but if you want to tie up loose ends I’m happy to help if possible. [16:57:09] e.g. maybe reviewing the documentation on how to gather training data and create the model, maybe documenting the API parameters... [16:57:46] It would probably be a good idea if I tried to reproduce the model pipeline as a way of verifying the steps [17:00:54] awight: exactly what i was gonna ask [17:01:25] re - https://developers.google.com/open-source/gsoc/help/work-product [17:02:01] where do you think this documentation belongs? [17:02:26] and since we're on the topic, where do you think known issues related to the model belong? [17:03:05] Good question, I think a wiki page under meta.wikimedia.org/wiki/Research would make sense [17:03:25] (known issues which might be sensitive, i.e. their knowledge might help scammers) [17:04:05] This isn’t a format that you could use at this point, but as a nice example of documentation under that Research namespace, see the “work log” here https://meta.wikimedia.org/wiki/Research_talk:Automated_classification_of_edit_quality [17:04:46] U could also write as an ipython notebook included in your github repo, whichever you prefer [17:05:08] For known issues, I personally like to keep those close to the work itself, maybe under a “future directions” heading [17:05:17] or “known issues” depending on perspective ;-) [17:05:27] ah, in a kaggle-ish sort of fashion [17:06:25] I haven’t seen much of Kaggle, so not sure [17:06:47] But I do like the idea of keeping things together! [17:07:10] Maybe GH repo makes the most sense for that reason—remind me what we’re going to do about the private data, though... [17:07:32] It would be impossible for anyone to replicate the work without it, but also impossible to share. [17:07:55] I guess that’s just the nature of the domain. [17:08:22] so how i've structured my api, all models/data mappings go in a private folder [17:08:27] which is not included in the repo [17:08:31] https://github.com/saurabhbatra96/wmf-fd-api/blob/master/private/README.txt [17:09:06] cool. IMO the pickled models are fine to commit [17:09:37] right, then i'll put them in as well [17:09:38] Maybe decompile with pickletools just to be 100% sure they don’t include training data, but otherwise it’s irreversible [17:09:49] they don't [17:10:06] Great [17:10:10] but technically it would be possible for people to compute scores on their own [17:10:30] I think that’s good, unless I’m missing something? [17:10:53] if i'm a scammer, that's a gold mine worth of info [17:11:06] even if our model isn't incorporated [17:11:26] saurabhbatra: hmm, that’s right we did have some hesitation about this exact thing at the beginning of the project [17:11:48] people can use our model to double-check that their fraud passes as normal [17:12:21] Maybe best to ignore my suggestion above, and leave the model private for now as you say. [17:12:53] In the future, adversarial fraud designed to evade detection will be useful for further refinement of our models, but we’re not ready yet. [17:12:55] plus re- reusability, i think that just won't work out the way my code is written rn [17:12:59] for ex. https://github.com/saurabhbatra96/wmf-fd-api/blob/master/utils/preproc.py#L81 [17:13:01] kk [17:13:09] pre-processing is pretty much hard-coded in [17:13:11] ah hehe [17:13:20] which can be fixed with some effort though [17:13:40] How can I be most helpful? Code review, doc review, replicating the pipeline… lmk [17:14:24] i think i'm currently focussed on the gsoc aspect as the deadline is tomorrow [17:15:02] i'll put some documentation into the repo itself as you suggested [17:15:55] maybe we can plan out our other steps with eileen in the loop whenever she comes on today [17:16:01] +1 I’m available to review whenever you want [17:16:38] i'll do a quick write-up and hit you up in an hour or so [17:16:40] :-) [17:17:24] right on! [17:36:28] ejegg, do you have a minute minute to talk about the feedback on https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/DonationInterface/+/452366/ [17:37:01] just about to grab some food. is 20-25 min from now good for you? [17:37:42] I'm breaking shortly too, sure I'll catch you later on [18:45:36] awight: currently i came up with this - https://github.com/saurabhbatra96/wmf-fraud-pipeline [18:46:29] i'll add the research and findings section later, just need to copy stuff off of the mail thread minus the sensitive info [18:47:24] you might not be able to easily replicate using the steps i've mentioned because of pip packages in frdev [18:47:48] however, i can share the private models with you in /tmp in case you want to check out the api [18:48:57] Perfect, I’ll replicate on frdev, please do copy data and modela to a new directory in /tmp [18:56:33] will ping you in a while [18:58:18] saurabhbatra: u know that mysql outputs TSV natively? [18:58:32] yup [18:58:46] i changed that to csv with sed [18:58:50] kk I was just guessing what the sed was about, I’ll look more closely [18:59:07] TSV is better anyway :p [18:59:11] oh it just makes it a csv and encloses the values with quotations [18:59:49] ok nbd [18:59:50] i'll note in down as an issue [19:00:07] this repo might be a good place to keep track of them [19:00:09] Not a priority for today though [19:00:15] +1 [19:03:39] awight: checkout /tmp/private [19:04:01] just git clone the api on your system and replace the private folder with this [19:04:09] https://github.com/saurabhbatra96/wmf-fd-api [19:04:36] schema for the request is given here - https://github.com/saurabhbatra96/wmf-fd-api#request [19:06:00] you might wanna get real data to try the api out on [19:06:08] try this - https://github.com/saurabhbatra96/wmf-fraud-pipeline/blob/master/data-extraction/fraud-query.sql [19:06:22] with receive_date>='2018-08-01' [19:06:34] saurabhbatra: I was planning to do this all on frdev, unless there’s a reason not to? [19:06:50] your user might not have the pip packages installed [19:06:56] I can also replicate locally but thought it frdev was more relevant to handing off to fr-tech [19:07:28] plus some requirements might not be there as packages [19:07:29] https://github.com/saurabhbatra96/wmf-fd-api/blob/master/requirements.txt#L2 [19:08:25] and you'll have to curl instead of using a gui based api tester... [19:09:34] aha got it [19:09:37] okay, local it is then [19:09:41] that reminds me, cwd - did you get the time to work on the "pip package installation error thingy" on frdev? [19:10:12] brb 2 mins, making some ramen :-) [19:11:54] saurabhbatra: remind me what that is? as i recall you were going to install stuff in your home dir [19:12:01] ramen, good idea [19:13:36] just make sure you don’t accidentally boil or microwave the pip packaging [19:18:29] cwd: that was kinda problematic, pip packages don't like being copy-pasted [19:18:49] plus my version of python differs from the one on frdev so wheels wasn't an option [19:19:09] so we ended up hand-writing needed library functions [19:21:25] cwd: Note for the future, it might be cool to have a frdev labs node for this kinda thing, to verify puppet etc. [19:22:09] awight: yeah definitely, at this point we don't have anything made for development in frack [19:22:54] It’s a scary thought. [19:23:33] We’ll hook up fraudsters with a complete target environment, including a service to tell them whether we’ll detect their attacks :-) [19:24:49] lol :-) [19:25:37] awight: any hiccups with the api installation? [19:25:53] heh yeah [19:25:58] saurabhbatra: I’ve only installed the pipeline so far, the API is a separate repo right? [19:26:01] at the same time i don't know of an alternative [19:26:34] oh hey, we'll need a copy of the repo in gerrit for deployment to frack, right? [19:26:40] cwd: We either reduce the barrier to entry for others, raise it for ourselves, or abandon FLOSS :-) [19:26:58] so you can't pip install --user on your laptop and rsync the dir to frdev? [19:27:19] ejegg: The repo for saurabhbatra’s antifraud service, or the puppet poo? [19:27:30] the antifraud service [19:28:06] we could do that, or package it [19:28:22] so what we're doing right now is [19:28:26] I guess it depends how much change we expect? [19:28:29] cwd: ah no, wheels would work (from a VM with the same version of python) but the binary virtualenv is not portable at all. [19:28:41] there is this repo https://github.com/saurabhbatra96/wmf-fd-api [19:29:10] and it's missing some files in a private directory - https://github.com/saurabhbatra96/wmf-fd-api/blob/master/private/README.txt [19:29:45] which are there in frdev + my local [19:30:06] tehy don't have any PII but contain the model as a binary which we don't want to expose [19:30:21] *they [19:30:24] right [19:30:54] I guess we could put the model in the same cluster-only repo we use to deploy settings files [19:32:09] ejegg: that is something that needs to be figured out [19:32:34] awight: you can ignore the pipeline code and try out the api directly [19:32:39] with /tmp/private [19:34:03] Fundraising Sprint Pluto is still a planet, Fundraising-Backlog, Fr-Ingenico-integration_2017-18, MediaWiki-extensions-DonationInterface: Deal with Ingenico Connect code 300620: MerchantReference XXXX already exists - https://phabricator.wikimedia.org/T201773 (DStrine) [19:34:18] be back in 10 [19:35:50] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Don't use freegeoip service by default anymore - https://phabricator.wikimedia.org/T201753 (Ejegg) Looks like https://ipstack.com/ has a free tier [19:43:31] saurabhbatra: Sorry about the hour… I had another conversation which was waiting to happen for months, so was fully distracted. I can get through these steps, now. [19:44:52] ejegg: While we’re on the subject, be warned that the models are pickled so only run if the code has non-breaking module changes & stuff. [19:45:02] Unfortunately, that’s the state of the art for scikit-learn. [19:46:38] eh what is the hostname [19:46:41] frdev1001.eqiad.wmnet [19:46:43] frdev. [19:46:44] not [19:47:44] O_o [19:48:18] “frdev1001” with no network [19:49:51] back [19:49:56] awight: you got the link? [19:50:06] *server address [19:50:19] I got in, ty [19:51:25] eileen_: currently i'm going with this - https://github.com/saurabhbatra96/wmf-fraud-pipeline#wmf-fraud-pipeline [19:51:33] re- the blog post [19:51:56] just got a mail from google saying there are only 14 hours left for the submission [19:52:40] GRR timezones [19:52:48] saurabhbatra: ok we can cross post later [19:53:08] it'll give wmf some time to check up the draft also [19:53:50] eileen_: btw adam and i wanted to have a chat regarding api integration [19:54:56] yep ejegg was just discussing that [19:54:58] the "how should we go about it" [19:55:47] currently adam's checking out the api [19:55:50] saurabhbatra: ooh I like fraud_query [19:56:07] :) actually I was going through the whole thing, cos stubborn. [19:56:29] Unless you wanted me to focus on the API? Sorry to waste your early morning [19:56:54] awight: lol, no problem at all [19:57:05] i just thought it might be inconvenient for you that's all [19:57:17] ah, no! [19:57:34] I’m free now, and can give this attention until the deadline [19:57:40] i'm up for at least another hour so take your time, this (cross-checking the entire pipeline) is better than only checking out the api [19:57:50] kk [19:58:38] so integration is a post-deadline thing isn’t it [19:58:57] eileen_ : yup [19:59:27] ok cool we should plan a discussion with ejegg but not distract you now [19:59:35] lol I feel like I just saw the cover put back on the nuclear button [19:59:59] awight: LOL [20:00:15] we need emojis in IRC so bad... [20:00:43] eileen_: I'll drop a mail once we're done with this? [20:05:52] 🔴 [20:07:22] saurabhbatra: It’s so sweet that we have a concrete database condition that we can use to find historical fraud. Can you confirm that MBeat had spot-checked the output? [20:08:06] i don't think i asked him to tbh [20:08:31] although i did confirm that our condition was correct [20:08:38] kk, I’m thinking more about biases caused by ommission, but that would be hard to spot-check anyway. [20:08:44] (asked ejegg i think) [20:08:47] Cool! [20:09:26] The genuine query is probably more expensive than it needs to be, but that’s an optimization for later. [20:10:00] i could not figure out a way to get random data and that was the only one that worked in minutes [20:10:07] randomly sampled* [20:11:01] Totally, SQL doesn’t make it easy [20:11:19] Yeah we’re pulling all 4.4M rows [20:11:24] i tried order by rand() and woke up the next day to a query still executing :-) [20:12:22] Fun M$ thing we don’t have, https://msdn.microsoft.com/en-us/library/cc441928.aspx [20:12:22] saurabhbatra: have you ever tried using modulo division on the ID? [20:12:36] like 'WHERE id % 50 = 1' [20:12:41] +1 ^ [20:12:47] Also nice that it gives a stable result set [20:13:19] saurabhbatra: k thanks for the warning! I’ll copy your data for that step [20:13:20] ejegg: ah yes, that would work too [20:13:57] saurabhbatra: Would you mind copying your data/ into /tmp/private? [20:14:06] awight: yup copying it rn [20:14:09] ty! [20:15:18] saurabhbatra: And after all my talk, the query actually completed :-) [20:15:46] saurabhbatra: I’d suggest moving all code out of “data” dirs [20:16:11] yup, i'll move it into model-training maybe [20:16:19] it did feel out of place there [20:16:25] also, just a convention I’ve seen, data could be in “datasets” to be more specific, and models could be in a “models” dir [20:16:28] moved my csvs to /tmp/fd-data [20:16:42] great [20:16:44] make sure you delete these once you're done though [20:16:52] contains pii, so... [20:17:19] :100%: [20:18:51] saurabhbatra: Inconsequent thing, the README has me build genuine_data with 1 more row [20:19:02] cos wc -l is actually num_rows+1 [20:19:33] hehe maybe that’s why tail -n+2 ? [20:19:40] weird though because your CSVs are the same length, somehow [20:19:43] i thought i'd mention it, but it just adds an unecssarycomplications [20:19:55] complication* [20:19:57] +1 :-D good call [20:20:12] that stuff will all be automated later, anyway. [20:20:26] tail indexes from 1 rather 0 so that's why -n+2 [20:21:08] In fr-tech we account for every micro-ruble is the only reason I would mention this at all ;-) [20:22:12] lol, noted :-) [20:22:37] i'm adding issues to the projects from what you point out/what i remember [20:22:45] Something’s wrong with this line, [20:22:46] cat fraud-data.csv < (tail -n+2 genuine-data.csv) > orig-data.csv [20:22:51] I’m using bash [20:23:01] don’t think you can redirect input like that [20:23:23] ah, i think dropping the < might work [20:23:24] Maybe it’s actually: [20:24:37] tail -n+2 -q fraud-data.csv genuine-data.csv > orig-data.csv [20:24:53] All rows except headers, from both files [20:25:07] aah you’re keeping the header on just one file [20:25:16] actually we wanna keep the headers [20:25:22] (for pandas) [20:25:29] head -n 1 fraud-data.csv > orig-data.csv [20:25:30] which makes pre-processing easier [20:25:35] tail -n+2 -q fraud-data.csv genuine-data.csv >> orig-data.csv [20:25:43] +1 to keeping headers [20:31:15] dang that preprocessing is fast [20:31:17] fixed [20:31:46] everything is pretty fast tbh, we're only working with 16k rows [20:31:47] looks good. You can do all the tailing in one command but nbd [20:32:05] well 16k rows takes an hour to extract in ORES [20:32:10] but that’s cos we make API requests. [20:32:49] here everything can fit in the cache :-) [20:33:10] yah the cpu cache [20:33:47] saurabhbatra: [20:33:48] File "model-train-gb.py", line 13, in [20:33:48] clf.fit(X_train, y_train) [20:33:49] NameError: name 'X_train' is not defined [20:34:00] darn [20:34:07] change that to X and y [20:34:07] I think “X, y" [20:34:10] yup [20:34:12] jinx [20:34:13] fixing [20:34:15] :-) [20:34:25] works now [20:35:39] kk the pipeline is reviewed and makes sense [20:35:47] One thing I wanted to check… are we coalescing nulls... [20:36:06] it depends on the field [20:36:24] for ex. if it's categorical [20:36:36] NULL is considered a separate category of its own [20:37:06] With the expression max(if(pfb.filter_name='getAVSResult', pfb.risk_score, NULL)) [20:37:09] if it's a float value, we have x_isnan boolean column for it and we're imputing it as 0 [20:37:19] isn’t it going to give null if *any* getAVSResult value is null? [20:37:43] https://github.com/saurabhbatra96/wmf-fraud-pipeline/blob/master/data/feature-eng.py#L46 [20:37:45] i.e. I worry about max(null, 1) [20:37:48] * awight tries it [20:38:34] should not, since we're grouping by payments_fraud_id [20:42:41] OK cool I proved to myself that max([1, null]) == 1 [20:42:59] usually SQL is conservative about that, e.g. 5 + NULL = NULL [20:43:13] looking at the API now [20:44:31] this should be smoother... [20:45:41] which part should be smoother? [20:45:58] the api part [20:46:02] https://github.com/saurabhbatra96/wmf-fd-api#installation-and-usage [20:46:17] aha [20:46:28] you'll have to do it on local though [20:46:43] You did it in a weekend, we’re happy with what’s here! [20:47:01] cause missing pip packages [20:47:31] We would need to all talk about a proper integration anyway, so I agree it was kinda pointless to refine any more than this [20:48:46] yup exactly [20:49:06] we can re-iterate after we have a version in staging [20:49:15] Maybe spell out explicitly, “cp wmf-fraud-pipeline/private/gb-clf.pkl wmf-fd-api/private/" [20:49:38] you'll need to copy all of the stuff [20:49:45] including data-mappings [20:50:08] data-mappings map categorical features to numbers and vice-versa [20:50:34] so are important to convert api requests into feature vectors [20:50:36] 2 *.pkl files; anything else? [20:50:40] cool! [20:51:02] gib_model.pki too [20:51:12] which contains our name gibberish scorer [20:52:01] ok [20:52:28] although i could put it in there as is now that i think about it... [20:52:31] When we do integrate, I’d recommend use use setup.py console_scripts for the “python *.py” entry points [20:52:41] then developers can use pip install -e . [20:52:54] +1 [20:52:57] and a wheel or debian package would magically create [20:53:38] i agree, it's far from shippable rn [20:54:16] It’s a great proof of concept! [20:54:21] Ship it! :p [20:54:30] if we want to re-iterate often, we'll want a smooth "release engineering" experience [20:54:55] I think fr-tech will have fun playing in python-land for a minute [20:56:02] that reminds me, we'll probably have to write some php to integrate too [20:56:14] The API readme should have steps for curl-POSTing an example JSON blob [20:56:35] ah yes, agreed [20:56:59] Do you have example values u can PM me? [20:57:05] otherwise I can just make it up [20:57:36] ACTUALLY [20:57:40] oops [20:57:44] sorry for the caps [20:57:45] I don’t like this ad-hoc schema documentation btw, it’s confusing [20:58:24] you might wanna run the fraud-query [20:58:27] IMHO it might be better to include sample data and response in that readme, and/or a more table-like view of the parameters [20:58:34] (CR) Jack Phoenix: [C: 2] Add an extra message to allow users to add custom navbar [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/446698 (https://phabricator.wikimedia.org/T138284) (owner: Brian Wolff) [20:58:38] with receive_date>='2018-08-01- [20:58:49] >='2018-08-01'* [20:58:58] and try it out on "real" data [20:59:28] turning those columns into json sounds annoying :-) [20:59:47] i used postmaster for it, just copy pasted values [20:59:59] postman, ugh [21:00:16] https://www.getpostman.com/ [21:00:28] * saurabhbatra blinks away the sleep [21:02:22] it's going to take some time and copy-pasting but you might like the results [21:03:17] saurabhbatra: OK I’m totally thrilled by what I’ve seen, and convinced by the argument that there’s not much to gain by improving the code in this phase. My suggestion for the final eval is to just spend time with the top-level documentation, it sounds like you’ve been working on that with eileen_ [21:04:22] Install docs are fine in my opinion also, I’d say further improvements should be for your own aesthetic reasons and not due to any inadequacy [21:04:42] yup, i'm gonna make changes tomm [21:05:00] submitting the links today to google cause i don't wanna delay [21:05:20] Sounds good, I’m sure they don’t start reading until > deadline [21:05:36] Thanks for being patient while I fumble around! [21:05:39] i think it's just for you guys to see [21:05:51] they don't look around afaik [21:06:11] oh of course, that must be right. [21:06:39] cause student evaluations start tomm and you'll get a mail with the links that i submit [21:06:40] The results will be public so it’s prudent to groom, but that’s not deadline-related [21:06:44] yes [21:07:10] deadline or not, i'll finish up by tomm [21:07:25] You’ve been impressively in sync with the GSoC process this whole time! My reminders from you usually come a day before the official emails ;-) [21:07:45] (Merged) jenkins-bot: Add an extra message to allow users to add custom navbar [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/446698 (https://phabricator.wikimedia.org/T138284) (owner: Brian Wolff) [21:07:54] i guess they email the students before they email the mentors then :-) [21:08:28] signing off now [21:08:40] see you tomm, same time! :-) [21:09:07] o/ [21:09:28] That actually makes a lot of sense, so the mentors aren’t constantly nagging students [21:15:08] fr-tech how does this bit look? https://gerrit.wikimedia.org/r/451200 [21:15:37] Think it should delete the pending row, or do some extra cross-checking of amounts and such? [21:18:39] what type of data is present in the pending message? [21:19:06] table* [21:19:30] jgleeson: it's all the stuff we gather on our side [21:19:47] for CC processors it's got name and address [21:19:56] but for paypal just the stuff that comes in on the querystring [21:20:04] like amount and utm_XXX [21:20:15] contribution_tracking contains most of the utm_ fields [21:20:31] but not the new opt_in field [21:20:50] I don't think I've done much with paypal messages, are these the IPN callbacks? [21:21:12] jgleeson: yeah, this job is to normalize and sort IPN messages [21:21:40] those sometimes come in before the 'donations' message from the front-end [21:24:32] i'm a little confused why we'd have pending data locally on recurring IPN messages from paypal [21:24:39] ejegg, [21:25:29] jgleeson: ah, for recurring, we would only have something in pending for the initial payment [21:25:43] not any of the subsequent ones [21:26:17] paypal sends us a 'subscription started' message [21:26:29] that we use to populate the contribution_recur table [21:26:36] ah I see [21:26:38] before we process the 'first payment' message [21:27:09] so if that came from a payment where the donor hit our site with opt_in on the querystring [21:27:38] we want to pass it through with the 'subscription started' message [21:27:54] (which also populates the civicrm_contact table, I should have said!) [21:30:12] that makes sense [21:38:48] (PS2) Jgleeson: Updated behaviour for FinalStatus::CANCELLED to match FinalStatus::FAILED [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/452366 [21:42:16] jgleeson: cool! I think we also want to update finalizeInternalStatus though [21:42:30] the issue is that when a person returns to our resultswitcher [21:42:58] we mark the donation as 'processed' so we don't mistakenly double-charge them if they reload the page [21:43:36] so in that finalizeInternalStatus, we make sure they start a new attempt for some statuses [21:43:47] hmm, actually, are we incrementing the order ID in any case? [21:44:16] would be cool if we didn't NEED to erase their form data if they just hit cancel to choose a different card [21:44:58] trying out your patch to see [21:47:35] ejegg, so we want to force the reset here? (guessing based on what you said) https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/DonationInterface/+/451786/3/gateway_common/gateway.adapter.php@2064 [21:47:50] jgleeson: that was my initial idea [21:47:59] but if it's not necessary, all the better [21:48:10] jgleeson: sorry for the delay in finding where Kafka-fed logs are... it's alnitak:/srv7banner_logs/2018/ ... Don't say 'ls' there, since there's a lot of stuff [21:48:16] might still be nice to explicitly list CANCELLED in the non-force cases [21:48:34] ls /srv/banner_logs/2018/ | grep -v beacon | tail -n 10 [21:48:41] should get you the most recent ones I think [21:48:55] thank you AndyRussG ;) [21:50:47] jgleeson: thanks likewise! [21:51:28] jgleeson: oh huh, I guess we're not actually using that 'mark completed' logic in the globalcollect / ingenico gateways any more. That seems like an oversight :( [21:52:11] jgleeson: so... for now, want to just put 'CANCELLED' along with the non-force cases in the finalizeInternalStatus switch? [21:53:27] so I added it in the original patch (before the latest updates) to the $force=false group, [21:53:56] ah, there it is! [21:54:02] sorry, I was looking at older code [21:54:24] ah got it! [21:54:26] I was confused [21:54:31] (PS3) Ejegg: Updated behaviour for FinalStatus::CANCELLED to match FinalStatus::FAILED [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/452366 (owner: Jgleeson) [21:54:36] Fundraising Sprint Owls, Fundraising Sprint Pluto is still a planet, Fundraising-Backlog: Queries and maybe scripts to verify equivalence of data in new-Kafka-pipeline-testing and pgehres production databases - https://phabricator.wikimedia.org/T198752 (AndyRussG) Location of the log files to be cons... [21:54:44]