[00:15:40] atgo: If you have a minute, please make me a private task... [00:15:47] i'm here! yeah 1 sec [00:16:02] No rush [00:16:39] Sorry about not having my own creds yet, it's on my bucket list [00:18:11] ...okay, I'm out for the day. Not that I appeared to be terribly far in. [00:18:24] But I have some weeds to whack. [00:18:41] RaaaaaAAAaaarrr. kbye [00:29:14] Wikimedia-Fundraising, Community-Liaison, Performance-Team, Patch-For-Review, Privacy: parsing legacy GeoIP cookies fails (no regex match), enwiki geonotice broken for users with those legacy cookies - https://phabricator.wikimedia.org/T103720#1403266 (AndyRussG) So far, it seems this doesn't aff... [00:33:39] awight: does my comment on the above phab task seem correct? ^ [00:34:05] AndyRussG: oh! Why is that? [00:34:36] And CN shouldn't run on special pages at all [00:34:37] No idea. I guess we need to understand it. But on article pages, window.Geo does get populated correctly. [00:34:40] Exactly [00:34:48] correctly, even if your cookie is wrong? [00:34:52] Yep [00:35:05] It's forced to regenerate, and that works?... [00:35:23] The cookie stays wrong, say for .wikipedia.org, but you get a new en.wikipedia.org cookie that's all good [00:35:31] Yeah there's some complicated voodoo happening there [00:35:47] Hmm lemme just investigate that the corrected value does get through to CN [00:37:45] * awight looks very embarrassed to have said "confirmed" when what I should have said was, I did not reproduce the error but the bug as reported would be bad. [00:38:38] awight: ah no you were right. It seems the corrected cookie doesn't get through to CN! [00:38:45] aaarrg [00:39:15] I'm just reading the code, sorry I was too lazy to actually run the damn thing [00:39:51] The code for the patch looks decent [00:41:07] Reading the deployed code again, the country == 'XX' fallback code path looks like it should work, as u say [00:41:32] OK running a debugger now [00:41:42] awight: https://phabricator.wikimedia.org/T103938?workflow=create [00:42:37] Wikimedia-Fundraising, Community-Liaison, Performance-Team, Patch-For-Review, Privacy: parsing legacy GeoIP cookies fails (no regex match), enwiki geonotice broken for users with those legacy cookies - https://phabricator.wikimedia.org/T103720#1403283 (AndyRussG) Rrrrg, quick update! It looks lik... [00:43:01] awight: Yeah it gets the fallback XX value [00:43:55] atgo: working on the GeoIP thing now. I was pretty sure we weren't affected, but now it looks like we are! [00:44:01] aww [00:44:07] well good thing it's like the lowest campaign season! [00:44:29] heh low campaign season, high end-of-the-quarter-code-like-a-maniac season [00:45:41] Fundraising Sprint House of Pain, Fundraising Sprint Indigo Girls, Fundraising Sprint James Brown, Fundraising Sprint Kraftwerk, and 6 others: Change errors on GC forms from popup to red text - https://phabricator.wikimedia.org/T86214#1403298 (CCogdill_WMF) Confirming that the message text all lo... [00:55:53] awight: arg no I was right the first time, it doesn't affect CN. However it does trigger an additional GeoIP lookup [00:57:54] AndyRussG: I can't explain it, but in my debugger I'm falling through to bannerController.js line 404, which doesn't do the new geoiplookup [00:59:18] awight: really! That's bizarre, 'cause I'm not. [00:59:34] AndyRussG: omg, I go through the line 388 side of the condition 4 times, then the line 404 side once [00:59:34] Are you not getting 'XX' as your country value initially when you have a bad cookie [00:59:45] ? [01:00:11] I'm getting 'XX' for sure, there must be some epic fail between me and the FF builtin debugger [01:00:21] awight: maybe it's debuggerfail? Sometimes they do weird things [01:00:28] I'm using FF native debugger [01:00:43] A watch expresion on mw.centralNotice.data.country === 'XX' is true, and I'm in the other side of that condition [01:00:46] wheeee [01:01:35] I'd trust the interpreter to correctly bifurcate on country === 'XX' sooner than I'd trust it to show current execution line correctly. Sometimes I've seen "wrong" execution lines when exiting a code block [01:01:50] sooner than I'd trust the debugger, I meant [01:02:34] Agreed. Well, I think you're right that it's not an issue [01:02:48] awight: I'm definitely going through line 393. Set a breakpt there [01:02:49] Calling it a night, either way :) [01:03:23] My breakpoint at 393 never triggered, I chalked it up to async fail [01:03:25] Sounds good! Ah yeah I forgot you're an hour later than you used to be (relative to me) [01:03:36] Which debugger? [01:03:51] FF native, also [01:04:13] Huh! Running on production or local? [01:04:36] on dewiki [01:04:42] (PS2) Awight: WIP Tests for audit processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 [01:05:03] Achso! [01:05:19] (CR) jenkins-bot: [V: -1] WIP Tests for audit processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 (owner: Awight) [01:07:59] AndyRussG: If I set $.cookie('GeoIP', 'US:Foo'); in the console, I get an empty window.Geo, and $.cookie('GeoIP') is not repaired [01:08:21] Don't know how to explain that... It's definitely not worth staying up late, though! [01:08:33] Have fun, see you soon! [01:08:39] cya! thx :) [02:19:52] (CR) Cdentinger: [C: 2] "i can neither confirm nor deny the usefulness of these fields but the code does what it says" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/219437 (owner: Awight) [02:30:48] (CR) AndyRussG: [C: 2 V: 2] "Thanks! LGTM! :)" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/220559 (https://phabricator.wikimedia.org/T103720) (owner: Gilles) [02:31:35] (Merged) jenkins-bot: Parse older format of Geo cookies [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/220559 (https://phabricator.wikimedia.org/T103720) (owner: Gilles) [02:42:13] (CR) Cdentinger: [C: 2] "what even is this compressed binary file here have a +2" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/219438 (owner: Awight) [02:52:12] (CR) Cdentinger: [C: 2] "this update is now error free when the contribution type id is stuck in an array, but i believe the previous patch was fine too and it's a" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/219436 (owner: Awight) [02:52:50] (Merged) jenkins-bot: Add and fix migrations for legacy custom fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/219436 (owner: Awight) [02:53:26] (Merged) jenkins-bot: Remove fictitious letter_code field [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/219437 (owner: Awight) [02:54:13] (Merged) jenkins-bot: Iterate offline keying template [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/219438 (owner: Awight) [03:05:22] (CR) Cdentinger: "i get 2 failures running this test, both say: Exception: field ID needs to be of type Integer for index" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/218567 (owner: Awight) [03:24:37] Wikimedia-Fundraising, Community-Liaison, Performance-Team, Patch-For-Review, Privacy: parsing legacy GeoIP cookies fails (no regex match), enwiki geonotice broken for users with those legacy cookies - https://phabricator.wikimedia.org/T103720#1403595 (AndyRussG) OK, my previous comment was also... [04:34:10] (PS1) AndyRussG: Merge branch 'master' into campaign_mixins [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221048 [06:08:48] Wikimedia-Fundraising, Community-Liaison, Performance-Team, Patch-For-Review, Privacy: parsing legacy GeoIP cookies fails (no regex match), enwiki geonotice broken for users with those legacy cookies - https://phabricator.wikimedia.org/T103720#1403682 (Gilles) a:Gilles>None Can someone take... [06:13:48] (PS1) Ori.livneh: Parse older format of Geo cookies [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/221057 (https://phabricator.wikimedia.org/T103720) [06:13:58] (CR) Ori.livneh: [C: 2] Parse older format of Geo cookies [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/221057 (https://phabricator.wikimedia.org/T103720) (owner: Ori.livneh) [06:15:15] Wikimedia-Fundraising, Community-Liaison, Performance-Team, Patch-For-Review, Privacy: parsing legacy GeoIP cookies fails (no regex match), enwiki geonotice broken for users with those legacy cookies - https://phabricator.wikimedia.org/T103720#1403689 (ori) a:ori [15:17:32] Fundraising-Backlog: New password+CIVI certificate for PPENA - https://phabricator.wikimedia.org/T103997#1405043 (Ppena) NEW [16:19:35] Fundraising-Backlog, fundraising-tech-ops: New password+CIVI certificate for PPENA - https://phabricator.wikimedia.org/T103997#1405284 (atgo) p:Triage>High a:Jgreen [16:20:04] (PS1) AndyRussG: Refactor BannerChoiceData => ChoiceData [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221141 (https://phabricator.wikimedia.org/T100686) [16:21:48] Fundraising-Backlog: Unique WP donations sent twice for settlement in France - https://phabricator.wikimedia.org/T95936#1405296 (atgo) a:atgo>None [16:23:49] hey AndyRussG! am i reading correctly that the geoip issue is definitely not effecting banner displays? [16:24:07] atgo: correct, that was the final and I believe correct assessment [16:24:09] also good morning :D [16:24:13] atgo: hi! :) [16:24:32] oh great! you already replied on the thread [16:24:34] I just hadn't dug deep enough the first two times... silly me [16:24:35] Yep! [16:28:43] Fundraising-Backlog: Sprint P GOAL: - https://phabricator.wikimedia.org/T102198#1405311 (atgo) [16:31:45] awight: boo! Some flavour for ur morning coffee: https://gerrit.wikimedia.org/r/#/c/221048/1 https://gerrit.wikimedia.org/r/#/c/221141/ [16:32:18] I wish I drank coffee... I save it for like bimonthly freak-outs [16:33:26] (CR) Awight: [C: 2] Merge branch 'master' into campaign_mixins [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221048 (owner: AndyRussG) [16:33:55] Hmmm... morning cheerios sugar pops? [16:34:03] awight, atgo: just replied to ayo again, looks like they do ding us 1.5% on transactions [16:34:13] (Merged) jenkins-bot: Merge branch 'master' into campaign_mixins [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221048 (owner: AndyRussG) [16:34:21] but not all of them, at least not according to the file i got [16:34:46] so i'm like whatever about that, we have net and total columns so it can be accurately recorded [16:35:08] but i need more clarification about what these fields even are [16:35:38] i should gently suggest they make a spec [16:37:40] cwdent: Cool, keeping track of the fees is optional for us, cos WMF does that accounting something like a monthly aggregate level [16:38:04] But IMO it's nice to have, and I'd be interested in whether the variations in fee happen in production data. [16:38:32] cwdent: Here's our documentation for normalized donation messages, https://wikitech.wikimedia.org/wiki/Fundraising/Normalized_donation_messages [16:39:04] The consumer is liberal, so you can specify net, fee, or both, and the same stuff will be recorded in Civi [16:39:22] It just does net = gross - fee, etc. [16:40:07] cool, sounds good [16:40:15] awight: i think this _is_ production data [16:40:35] hehe, well we can be nice guys and let them know they forgot to charge us fees, I guess [16:40:54] yeah [16:41:16] hell i'll write the spec if i can get a good explanation of the data [16:41:26] although i wonder if it's not under active development [16:49:56] (CR) Awight: [C: 2] Refactor BannerChoiceData => ChoiceData (1 comment) [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221141 (https://phabricator.wikimedia.org/T100686) (owner: AndyRussG) [16:50:59] (Merged) jenkins-bot: Refactor BannerChoiceData => ChoiceData [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221141 (https://phabricator.wikimedia.org/T100686) (owner: AndyRussG) [16:52:29] awight: thanks! [16:55:04] awight: anyone else interested: there are two minorish pieces of the banner history pie that need some details filled in... one is the EventLogging->Kafka setup for the actual logging, and the other is linking banner histories to donation data [16:55:56] Pls LMK if sometime u have a sec to chat about either of those, especially the second one :) thx again!! [16:57:44] sure--now is good? [17:00:02] awight: yeah...almost! one sec :) [17:03:49] awight: yeah all set, now (or any other time that works 4 u) [18:10:59] hey awight did i remember that you're going on vacation aroudn wikimania? [18:11:59] if so please add it to the calendar [18:17:16] sorry, testing my highlighting. [18:17:19] aa awight [18:17:21] a awight: [18:17:22] awight: a [18:17:24] wtf [18:17:34] atgo: yep, thanks for the poke! [18:17:44] haha [18:18:05] * atgo pictures awight saying all of that outloud [18:18:40] it's broken [18:18:47] as of 10:04 [18:18:48] anyway [18:20:46] hahaha [19:25:44] Fundraising Sprint Kraftwerk, Fundraising Sprint Lou Reed, Fundraising Sprint Miles Davis, Fundraising Tech Backlog, and 3 others: Provide access to limbo messages without knowing keys - https://phabricator.wikimedia.org/T99152#1405850 (awight) [19:26:17] Fundraising Sprint Kraftwerk, Fundraising Sprint Lou Reed, Fundraising Sprint Miles Davis, Fundraising Tech Backlog, and 3 others: Orphan slayer reads from frack Redis rather than ActiveMQ - https://phabricator.wikimedia.org/T99017#1405853 (awight) [19:27:05] Fundraising Dash, Fundraising Sprint Miles Davis, Fundraising Sprint N*E*R*D, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Generate fake data for Dash/CiviCRM - https://phabricator.wikimedia.org/T101904#1405857 (awight) Open>Resolved [19:27:08] Fundraising Dash, operations: Create sandbox site for Dash - https://phabricator.wikimedia.org/T87809#1405858 (awight) [19:28:11] awight: K4-713: can anyone give me a ballpark figure of the highest donations/sec rate we should expect during end-of-year campaigns? I'm talking to ottomata (Andrew Otto) on #wikimedia-analytics about more dtails of history logging) [19:28:38] AndyRussG: ...yes. [19:28:45] But we're both in a meeting at the moment. [19:29:06] What are you using this for? [19:29:34] Fundraising Tech Backlog, Wikimedia-Fundraising-CiviCRM, Epic: [epic] Port our CiviCRM customizations to 4.4 upstream, and to extensions - https://phabricator.wikimedia.org/T99838#1405866 (awight) [19:29:43] Banner history, seems we may log it directly for all donors in addition to a sample of all users in a campaign [19:29:56] K4-713: ^ [19:29:59] hm [19:30:06] Logging... banner history. [19:30:11] For donors. [19:30:15] Yes [19:30:36] Does that mean "save all the banner history on a user's conversion to a donor?" [19:30:42] *"?. [19:30:55] Mmm more or less [19:30:59] okay [19:31:22] Basically whenever anyone's in a campaign, a history of the banners they see (or that get hidden from them for some reason) will accumulate on their computer, in localStorage [19:31:34] Yeah, I get all that. [19:31:40] I'm just wondering at what point you save it all. [19:31:44] Of those, only a sample, say 1/100, will get sent back to us [19:31:49] You know, on our systems. [19:32:09] Yeah non-donors a smallish random sample [19:32:12] As opposed to letting it accumulate and decide things on their client. [19:32:24] Ah yes that also happens [19:32:30] But it's also in large part for analytics [19:32:34] yep [19:32:46] It doesn't have any personally identifying data [19:32:55] So... when they hit the payments cluster, or when they finish off a payment? [19:33:11] Fundraising Tech Backlog, Wikimedia-Fundraising-CiviCRM, Epic: [epic] Port our CiviCRM customizations to 4.4 upstream, and to extensions - https://phabricator.wikimedia.org/T99838#1405874 (awight) [19:33:16] Mmmm unknown at this point [19:33:20] k [19:33:30] Because the numbers I'm going to give you, are the final ones only. [19:33:32] But yeah Ellery & co would like to have the histories for all donors, too [19:33:46] Any place other than the end of the process, will be larger. [19:33:47] That's fine, it's just so Andrew O. can have an idea of what volumes we'll be looking at [19:34:09] We should get figures on abandonment rates at the different steps too, then. [19:34:31] meganhernandez should have piles of that. :) [19:34:33] K4-713: The history will be passed from the wiki they see the banner on to payments wiki, and payments wiki will do the EventLogging call [19:34:54] hmm [19:34:56] Exactly where and when is probably flexible, I think... Let's overshoot and go with the higher figures then, I guess [19:35:05] Why are we eventlogging from payments wiki? [19:36:03] That could get sticky. [19:36:54] K4-713: Because the other option is to send the logging request right when a person clicks on "donate" and then wait for that request to come back (to make sure it gets thru) before sending them on to payments, and that could degrade their donating experience [19:39:44] The issue here I'm trying to think through, is that having more systems communicate with user's clients when they're on the payments cluster, can easily make PCI more complicated. [19:40:33] K4-713: eventlogging can be done server-side [19:40:40] That's possibly worse. [19:40:51] Unless it's contained on those machines. [19:41:13] Ah no definitely not :/ [19:41:14] This is a write-only link to eventlogging, and no cardholder data is transmitted. I think it's kosher. [19:42:08] mmm [19:42:11] At the point we log the banner history message over server-to-server eventlogging, the user hasn't even begun filling out the payments form, so it's hard to imagine a leak. [19:42:35] Well, unless the answer to my other question was "on completion". [19:42:44] Hmmm [19:42:49] naw we don't need to add anything there. [19:42:52] okay [19:42:57] The full banner history would be put in the parameter of the POST request when we go to payments wiki [19:43:06] ...will it fit? [19:43:18] We'll already have a link between contribution_tracking id and the banner history, and have other ways of counting payments form -> attempted payment conversion [19:43:41] yah, it's a biggish post but still c. 2kB [19:44:17] hm [19:44:55] Just for ottomata's part in this (still talking on #analytics) what is our highest donation/sec rate? Also if you know an approximate banner impression/donation for such circumstances, that'd be fab [19:44:57] phab [19:45:09] Still in the meeting until 1pm. :p [19:45:19] K4-713: ah k sorry...thx! [19:49:02] AndyRussG: fyi, https://collab.wikimedia.org/wiki/Fundraising/Engineering/Fun_SQL_Queries [19:49:25] You can do those from lutetium [19:51:52] AndyRussG: Our highest donation rate, averaged over the hour, was 3.8 donations per second, last Dec 2nd. [19:52:51] awight: cool thanks! [19:53:04] That doesn't seem like a lot! [19:53:09] Added the query here, https://collab.wikimedia.org/wiki/Fundraising/Engineering/Fun_SQL_Queries#Highest_donation_rate [19:53:25] No, it's nice and slow! Luckily, the bottom line is the integral... [19:54:50] awight: do you know what percentage of banner impressions that might be? Maybe around 1/100? [19:56:11] AndyRussG: sure, lemme try [19:58:11] Sorry I meant 1/1000 [19:58:11] select sum(count) from pgehres.bannerimpressions where timestamp between '2014-12-02 22:00:00' and '2014-12-02 23:00:00' and campaign like '%_FR'; [19:58:18] still calculating... [19:58:37] total fundraising banner impressions over the hour was 3920500 [19:59:33] 1/280 of the banner impressions resulted in a donation [20:00:15] Hmm, I realized these aren't the numbers you want, though. You actually want the number of impressions against the number of people entering the paymentswiki funnel [20:00:53] ppena might have abandonment numbers. [20:01:06] In fact, she totally does. [20:01:17] awight: K4-713 well really I just wanted impression numbers, now that I think about it a bit moar better [20:02:00] select count(*) from drupal.contribution_tracking where ts between '20141202220000' and '20141202230000'; [20:02:08] So for that hour, we got an average of 1089 impression / second [20:02:09] * K4-713 yawns hugely [20:02:11] That was long. [20:02:14] 21858 people landed on paymentswiki during that hour [20:02:38] I need some serious caffeine or something incredibly surprising. [20:02:39] So 1/180 of the impressions converted into a person on paymentswiki [20:02:48] AndyRussG: ^ I think that's what you wanted [20:02:57] awight: cool thanks! [20:03:10] Cos that's the criteria for who sends their banner history AFAI understand [20:05:57] buh, standup? [20:06:39] ...Bueller? [20:06:45] Bueller? [20:25:35] (PS3) Awight: WIP Tests for audit processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 [20:25:37] (PS1) Awight: Audit cleanup and fixes [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221274 [20:25:39] (PS1) Awight: Correct code to run the audit processor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221275 [20:25:41] (PS1) Awight: Robustness fixes [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221276 [20:26:29] (CR) jenkins-bot: [V: -1] WIP Tests for audit processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 (owner: Awight) [20:33:25] K4-713: awight: thanks 4 your help w/ numbers for bh logging [20:33:25] K4-713: It looks like you made a decision in the audit code, to reject records without contribution_tracking. That can't be right? [20:33:36] any time! [20:33:39] ack! [20:33:41] heh [20:33:52] ack ack ack? [20:34:01] awight: What specific audit code? [20:34:34] K4-713: It's possible that that was a bug, cos if you set the fakedb flag it will stuff some audit lines into the utm_ variables [20:34:43] which module [20:34:46] Seems like we should do the same thing, if there's nothing in the db? [20:34:49] wmf_Audit [20:34:57] wmf_audit_get_contribution_tracking_data [20:35:23] let me see if I have anything like that checked out here. [20:37:18] awight: Okay. Where are you seeing the alleged rejection? [20:37:52] That function returns a falseish value if you don't have the fakedb flag set, or it finds actual CT data [20:38:29] Oh, wait. It's the DATA_INCOMPLETE. [20:38:31] That does it. [20:38:35] The false return value returns an error code, yeah [20:38:45] And it's supposed to get sent when that error is handled? [20:38:55] Oh wait. I take it back. That is explicitly nonfatal. [20:39:12] At least, the thing I was looking at, is. [20:39:18] hm. [20:39:22] OK I think this is a bug, then, it doesn't sound intentional [20:39:35] Well... maybe, maybe not. [20:39:40] hehe [20:40:09] I'm still looking at it to see what's... taking umbrage. [20:40:37] dict. ah, thanks for the word [20:41:12] lol [20:41:17] Ah, it's on the outside. [20:41:23] MISSING_MANDATORY_DATA. Yeah? [20:41:31] I'd say, the audit processor's duty is to get all the stragglers in to our CRM database, and disregard quite a bit of damage. [20:41:31] I mean, that seems pretty intentional to me. [20:42:02] I think, the idea there was that we would notice the problem due to failmail, and run it in makemissing mode. [20:42:17] interesting [20:42:42] In that way, we would not... automatically create a giant pile of stuff to clean up, if there was a problem upstream that needed to be delicately handled. [20:42:59] Oddly, it has... never come up. [20:43:08] But that's what I seem to remember having in mind. [20:43:28] Catching problems close to their inception instead of noticing a year later that contribution tracking was all f'd. [20:43:51] I mean... I keep thinking about what that would do to the A/B tests. [20:44:03] I'm sold, I'll preserve that behavior and write a test that we failmailed. [20:44:04] If, say, one gateway was not working with contribution_tracking correctly. [20:45:02] awight: For the record, I think it is deeply cool that you're doing this. [20:45:17] Thanks. [20:45:36] aww, shucks 8D [20:45:44] https://twitter.com/ieure/status/614130731770617856 [20:46:02] Sorry to be a dick about not treading lightly [20:47:02] cwdent: haaa [20:47:03] Tests are the main thing I'm going for, though [20:47:16] awight: Yeah, that's important. [20:47:31] I'm just happy somebody else is getting in there. [20:47:45] ...in a way that will let other people get in there. [20:49:06] totally. I learned that there's a type of diving where you never take a buddy--cave mapping. Cos your buddy can kill you by getting their butt stuck on an upstream stalagmite. Our repos are not like that. [20:49:42] * K4-713 chokes, nearly spits red bull all over computer [20:50:14] While I have never done that myself, the description does sound hauntingly familiar, now that you mention it. [20:50:36] Not that many people live to tell that story [21:08:22] awight, K4-713 I was just about to ask some relevant audit processor questions [21:08:43] like, should we do anything with a notification of a canceled or expired payment? [21:09:23] Yes. [21:09:29] i.e. make sure any 'pending' queue entry is deleted? [21:09:40] But... what does "canceled" mean, in this context? [21:09:43] Yell and scream if we already imported it as cleared? [21:09:49] Is that like a refund, or a chargeback? [21:10:11] It might just map to one of those. [21:10:18] no, i think it's when the user doesn't fill out their payment info [21:10:26] er. [21:10:32] or hits the 'cancel' button at the credit card form [21:10:44] Ah. We're talking abandonment, then? [21:10:55] yeah, I think so. [21:11:06] In that case, I don't think we want it to get to the CRM. [21:11:10] Or like they never show up at the shop with the boleto printout and the cash [21:11:16] So, just go ahead and eat it. [21:11:20] k [21:11:28] The "pending" queues... [21:11:38] ...well, we cheat a bit with those, and just prune them after a while. [21:11:56] All of them. [21:12:05] ooh, I maybe need to specify an expiration on those... [21:12:17] That's happening in a weird place. [21:12:27] There's a jenkins job... [21:12:33] or is there a process that just kills all pending queue entries over the age of 25? [21:12:47] Heh. [21:12:55] Maybe I should rename that job "carousel". [21:13:00] hehe [21:13:23] You can specify a... number of days, for each queue it prunes, I think. [21:13:31] But the params are all *in* jenkins. [21:13:44] oh, cool [21:14:08] Which reminds me: We should probably add that queue to the list it works with. :D [21:14:20] ah, i was thinking of Logan's Run, and it was age 30... [21:14:24] yep [21:14:47] There were a bunch of people chanting "RENEW" at me when I turned 30. [21:14:53] oh jeez! [21:15:13] did you have a bunch of leds taped to your hand? [21:15:17] I'm like "SHUT UP, I'm trying to eat cake." [21:15:23] They tried. [21:15:33] Or, they thought about trying. [21:15:44] man, my friends aren't nearly nerdy enough... [21:15:48] And then they thought they didn't want to accidentally electrocute me for my birthday. [21:15:53] hehe [21:16:00] Thoughtful. :) [21:17:09] Listening to "Open letter to NYC" for the first time. How is that possible... [21:19:58] awight: does this look like sufficient pre-redirect logging for the astropay audit queue consumer? https://gerrit.wikimedia.org/r/221005 [21:23:05] ejegg: Great approach. Might be overkill, actually. We only need the fields which don't come through the audit file. [21:23:43] But on the other hand, I'm in favor of the simplicity [21:24:02] heh, it was certainly the easiest way [21:24:38] Totally! [21:25:24] (CR) Awight: [C: 2] Before Astropay redirect, log details for audit processor [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/221005 (https://phabricator.wikimedia.org/T90507) (owner: Ejegg) [21:25:38] thanks! [21:26:02] (Merged) jenkins-bot: Before Astropay redirect, log details for audit processor [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/221005 (https://phabricator.wikimedia.org/T90507) (owner: Ejegg) [21:26:15] (PS4) Awight: Tests for audit processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 [21:26:36] (PS5) Awight: Tests for audit processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 [21:26:38] (PS2) Awight: WIP AstroPay audit module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/220944 [21:26:40] (PS2) Awight: Audit cleanup and fixes [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221274 [21:26:42] (PS2) Awight: Correct code to run the audit processor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221275 [21:26:44] (PS2) Awight: Robustness fixes [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221276 [21:27:26] (CR) jenkins-bot: [V: -1] Tests for audit processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 (owner: Awight) [21:28:00] (CR) jenkins-bot: [V: -1] WIP AstroPay audit module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/220944 (owner: Awight) [21:33:30] (PS3) Awight: Move shared functions to the base class [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/220835 [21:33:32] (PS6) Awight: Minimum split of BaseAuditProcessor from functions that don't require context [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/220714 [21:33:34] (PS2) Awight: Move the Drush command hook to the base module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/220940 [21:33:36] (PS6) Awight: WIP Tests for audit processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 [21:33:38] (PS3) Awight: WIP AstroPay audit module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/220944 [21:33:40] (PS3) Awight: Audit cleanup and fixes [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221274 [21:33:42] (PS3) Awight: Correct code to run the audit processor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221275 [21:33:44] (PS3) Awight: Robustness fixes [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221276 [21:36:21] (CR) jenkins-bot: [V: -1] WIP Tests for audit processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 (owner: Awight) [21:36:59] (CR) jenkins-bot: [V: -1] WIP AstroPay audit module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/220944 (owner: Awight) [21:42:35] (PS7) Awight: WIP Tests for audit processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 [21:43:21] (CR) jenkins-bot: [V: -1] WIP Tests for audit processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 (owner: Awight) [21:52:03] (PS8) Awight: WIP Tests for audit processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 [21:52:53] (CR) jenkins-bot: [V: -1] WIP Tests for audit processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 (owner: Awight) [21:58:26] Looks like composer "install" doesn't lock correctly for our dev-master libraries [21:58:39] looks like a phab upgrade happened? [21:59:00] Anything noticeable? [21:59:21] the header bar is definitely bluer than before [22:00:06] (CR) Cdentinger: [C: 2] "my bad, i hadn't run 7032 so the civi api was failing to find the custom group in a non obvious fashion" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/218567 (owner: Awight) [22:00:52] (Merged) jenkins-bot: Test more fields used by WmfImportFile format [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/218567 (owner: Awight) [22:17:44] (CR) Cdentinger: [C: 2] "tried spamming contribution type and just got generic import, looks good" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/219492 (https://phabricator.wikimedia.org/T88836) (owner: Awight) [22:18:19] Fundraising Tech Backlog, Wikimedia-Fundraising-CiviCRM, Continuous-Integration-Infrastructure: civicrm-buildkit doesn't composer update our wikimedia projects correctly - https://phabricator.wikimedia.org/T104031#1406224 (awight) NEW [22:19:10] (CR) jenkins-bot: [V: -1] Passthrough more payment types [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/219492 (https://phabricator.wikimedia.org/T88836) (owner: Awight) [22:20:17] cwdent: that failure ^ was due to me livehacking the jenkins job :( [22:21:12] whoops! [22:21:15] what for? [22:22:02] mmm. https://phabricator.wikimedia.org/T104031 it looks like we're getting chronically outdated vendor/ libs [22:22:23] (CR) Awight: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/219492 (https://phabricator.wikimedia.org/T88836) (owner: Awight) [22:23:09] (CR) Awight: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 (owner: Awight) [22:23:18] hrm, isn't .lock in the repo? [22:23:28] seems pretty bad if it doesn't get checked for changes [22:24:51] Yeah, looks like composer fail [22:25:28] youhadonejob.jpg [22:29:57] awight: should i hold off on merging while you're fiddling? [22:30:27] I think it's merged now [22:31:20] Getting progressively more crept out at composer behavior though. I just removed the whole workspace, and it rebuilt with the wrong revision of my library [22:31:52] to get it to pick up the new one? [22:32:54] Yeah but it insisted on the old version [22:32:55] (CR) Cdentinger: [C: 2] Drop "Amount (USD)" column [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/219493 (https://phabricator.wikimedia.org/T88836) (owner: Awight) [22:32:55] gack [22:33:53] (Merged) jenkins-bot: Drop "Amount (USD)" column [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/219493 (https://phabricator.wikimedia.org/T88836) (owner: Awight) [22:36:13] (CR) Cdentinger: [C: 2] Update Civi to pick up fix [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/219498 (owner: Awight) [22:37:21] (CR) Cdentinger: [C: 2] Migration for Restrictions and Gift Source options [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/219480 (owner: Awight) [22:37:38] (Merged) jenkins-bot: Update Civi to pick up fix [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/219498 (owner: Awight) [22:38:00] (Merged) jenkins-bot: Migration for Restrictions and Gift Source options [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/219480 (owner: Awight) [22:42:34] (CR) Cdentinger: [C: 2] Provide Generic Org import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/220001 (https://phabricator.wikimedia.org/T88836) (owner: Awight) [22:43:14] (Merged) jenkins-bot: Provide Generic Org import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/220001 (https://phabricator.wikimedia.org/T88836) (owner: Awight) [22:43:17] (PS2) Ejegg: WIP Astropay audit file processing [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/220998 (https://phabricator.wikimedia.org/T90507) [22:53:49] (PS9) Awight: WIP Tests for audit processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 [22:54:31] (CR) jenkins-bot: [V: -1] WIP Tests for audit processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 (owner: Awight) [22:56:08] ejegg: The dev-master wikimedia libs e.g. donation-interface seem to be reverting back to 5c4e45c177923934b41b63a0ab2bb746ad092031 [22:56:27] I'm getting suspicious, cos that was when you added us to packagist? [22:56:54] awight: let me check... they said they'd do daily fetches [22:57:02] Unfortunately, I can't reproduce this locally [22:58:21] awight: nope, on the site is says rev 8b478fa35b6392b57e2676b6497dab8de920643b - which is like a week old [22:58:38] awight: this is in jenkins? [22:58:48] It's getting weird, composer.lock has that revision in the patchset under test, but composer install is still pulling 5c4e45c177923934b41b63a0ab2bb746ad092031 [22:59:02] I'm seeing inconsistencies in what revs are checked out too, just looking at the drupal merge again [22:59:06] yeah, for example integration-slave-trusty-1012:/mnt/jenkins-workspace/workspace/wikimedia-fundraising-civicrm/src/wikimedia/fundraising/crm/vendor/wikimedia/donation-interface [22:59:35] try 'git log' in vendor [22:59:40] These are my notes so far https://phabricator.wikimedia.org/T104031 [22:59:41] it's from march [22:59:47] git log in vendor shouldn't mean anything [22:59:58] right, vendor/w*/d* [23:00:00] yeah [23:00:02] wtf [23:00:16] rm -rf doesn't fix... [23:00:20] the jenkins log says zuul checked out master there [23:00:37] well, it's master of crm but vendor is created by composer install, I think [23:00:43] oooh [23:00:46] that's the problem [23:00:46] thx [23:00:52] fuck. [23:01:13] also, zuul-cloner claims it's doing shady things with fallback - [23:01:44] Wow, you just saved me a grey hair [23:01:45] it looks in each repo checked out it something like the ref it wants exists, when that ref only exists in the triggering repo [23:02:00] I'm gonna stop the zuul-clone of vendor and let composer handle it [23:02:02] wouln't be too bad, but it accepts anything that looks like it: [23:02:15] oooof [23:02:30] are you sure that's not just the stale vendor/ repo? [23:02:33] check this: Project wikimedia/fundraising/crm/civicrm in Zuul does not have ref refs/zuul/contrib/Z5a3f6766408f4e1b9ad3eadb9d488249 [23:02:38] yes that makes sense [23:02:39] :zuul.Cloner:Falling back to branch contrib [23:02:45] Prepared wikimedia/fundraising/crm/civicrm repo with branch contrib [23:02:45] that's the ref of the crm changeset being tested [23:02:47] oh, contrib! [23:03:22] I don't see anything about contrib in the integration-config repo [23:03:27] that's the log anyway.. but looking at the dir, it's master. so confused. [23:03:48] I think it's just because the repo has a branch with the same name [23:03:54] No, I believe it. This is the vendor/ repo that we use for deployment now [23:04:12] yeah, i think zuul cloner has some real problem [23:04:13] sorry, right you're talking about the contrib branch thing [23:04:39] omg, check git log in the drupal dir [23:04:59] it's not even checking out the ref it's supposed to be testing! [23:05:15] so, log doesn't correspond to filesystem [23:05:27] Well that's getting exciting, https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm/470/consoleFull [23:05:29] and claimed log behavior has some problems [23:05:42] you guys are on the CI server? [23:05:43] It's possible there has already been another build on that server [23:05:48] cwdent: yeah [23:05:54] jumping over to 1015 [23:05:59] You want the magic underwear for that? [23:06:49] think i'm sufficiently magicked up - less you've got a superpower I don't know about? [23:07:12] cwdent: Added you to the labs integration project [23:07:21] * awight eats a superdot [23:07:33] oh, sorry, you were talking to cwdent! [23:07:37] cool, thanks! [23:08:27] /mnt/jenkins-workspace/workspace is machine-specific, right? [23:09:36] good question [23:10:02] asking in -releng [23:10:08] Looks like it, /dev/mapper/vd-second--local--disk on /mnt type ext4 (rw) [23:10:48] I think it would have to be, since workspaces don't have a buildid. Hmm, that sketches me out. What's the no concurrency setting for, then... [23:11:15] oh, cool [23:12:28] awight: i'm going to trigger another check of that drupal patch, unless that's likely to upset your forensics [23:13:18] no problem [23:13:28] (CR) Ejegg: "recheck" [wikimedia/fundraising/crm/drupal] (contrib) - https://gerrit.wikimedia.org/r/219443 (https://phabricator.wikimedia.org/T103006) (owner: Ejegg) [23:13:29] hehe, I think I found a second issue--composer not running in the right directory [23:13:35] ooh [23:14:05] Which account for about half the hair I pulled out [23:15:04] (PS10) Awight: WIP Tests for audit processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 [23:15:40] (CR) jenkins-bot: [V: -1] WIP Tests for audit processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/221003 (owner: Awight) [23:15:44] ejegg: Same 7014 fail! [23:16:12] aw man [23:16:16] I wonder if this is always the outcome of setting up CI, that you debug the tests rather than coding [23:16:36] yeah, we may be at the tipping point of test usefulness. [23:17:23] But we've only just begun!! [23:17:38] I need inception dates, model numbers [23:18:13] hum, maybe that folder /is/ shared [23:18:50] I just rechecked the drupal patch on 1012, and the head in crm is your WIP tests patch [23:19:32] That's worth writing a bug to the CI elves [23:19:45] yah, will do [23:20:33] but maybe you are spending this time debugging the tests instead of debugging broken wikipedia [23:20:42] at least that's what i convinced myself about testing [23:21:27] This fragility is really dragging us down, though [23:22:38] yeah it seems really complicated [23:24:11] maybe the right idea is how every framework enforces tests from project inception now. but that doesn't really do anything for all the code that was written before 2013 [23:24:49] awight: is that composer change just for CI or will it make it out to live? [23:25:24] The CI team was making recent noises about this promising approach of keeping everything needed to test a repo internal to that repo, like .travisrc does [23:25:33] cwdent: the zuul/layout thing is just for CI [23:28:36] (PS1) Awight: Update dependencies [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/221312 [23:28:58] ejegg: I think it's safe to do this update, it should be compatible with the deployed crm code. ^^ [23:29:15] (CR) jenkins-bot: [V: -1] Update dependencies [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/221312 (owner: Awight) [23:29:19] hehehe [23:29:46] yeah, i don't know if all the stuff zuul's checking out is compatible with each other [23:30:01] i bet a comments-only change would fail now [23:31:13] awight: also, you need to delete the trait files for old-fashioned phplint :( [23:31:25] gah [23:31:35] awight: in psr/log someplace [23:31:45] thx. I also removed the .git directories in wikimedia/* but I guess that wasn't necessary [23:31:52] psr/log/Psr/Log/LoggerAwareTrait.php and psr/log/Psr/Log/LoggerTrait.php [23:32:50] (PS2) Awight: Update dependencies [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/221312 [23:33:05] I see there are a lot more references to traits... [23:33:28] (CR) jenkins-bot: [V: -1] Update dependencies [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/221312 (owner: Awight) [23:33:42] 18 more files [23:33:47] aw jeez [23:34:31] if we want to play with all these shiny toys we're gonna need to up our minimum php version [23:35:49] For some reason, the phplint build isn't being triggered. Maybe that's cos the civicrm job is failing. [23:36:26] https://integration.wikimedia.org/ci/job/phplint/19024/console [23:37:01] ends with a few hundred egrep: writing output: Broken pipe... [23:37:18] lol [23:37:19] but lists your trait files [23:37:50] and setDisallowChangesToGlobalState turns out to be a phpunit incompatibility, somehow [23:38:09] yeah, me too!