[15:38:20] (CR) Ejegg: [C: 2] Merge REL1_25 into fundraising/REL1_25 [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/226088 (https://phabricator.wikimedia.org/T94528) (owner: Awight) [15:39:36] (PS3) Ejegg: update DonationInterface to current deployment head [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/226089 (https://phabricator.wikimedia.org/T94528) (owner: Awight) [15:39:59] (CR) Ejegg: [C: 2 V: 2] update DonationInterface to current deployment head [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/226089 (https://phabricator.wikimedia.org/T94528) (owner: Awight) [15:44:06] (CR) Ejegg: [C: -1] "The submodule commit this points to, I5d403f888d3fd8c749730df1c4d0af63904568ee adds require-dev packages, though it looks like they were a" [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/226093 (https://phabricator.wikimedia.org/T94528) (owner: Awight) [15:56:29] * AndyRussG waves [15:56:52] hey AndyRussG [15:56:59] back in french canadia? [15:57:12] cwdent: hi! Yes indeeeeed!! [15:57:22] AndyRussG: are you able to go to the team health check meting tomorrow? [15:57:23] You're also back home, I think, right? [15:57:35] dstrine: yes, u bet! [15:57:44] thanks! can you accept the invite? [15:58:07] also does everyone see that we moved sprint planning to 12 PST? [15:58:07] dstrine: sure..... sorry for not doing so soooner 8p [15:58:54] oops yeah i never remember to reply to invites [15:59:15] yea AndyRussG i got back the same day everybody left for wikimania [16:00:00] Hi ejeg! You're missing a 'g'! Quickquestion: you're going to Scrum of Scrums, right? I saw that I got added as a guest to SoS, not sure why... and if you're planning to go... I mean, I don't think there's much need for more than one of us there or anything [16:00:19] cwdent: ah right! Man that was an intense week... How was the wedding, BTW? [16:01:10] it was good! thanks. i gave a speech in front of 150 people that i was dreading for the last year so i am very relieved [16:02:27] then i went to minneapolis and met my new niece who is adorable [16:02:47] cwdent: woohoo! congrats! Yeah sounds like a lot of fun :) [16:03:24] Sounds like a lot of key moments :) [16:03:40] AndyRussG: yep, going to attend SoS today [16:04:06] ejegg: Cool thanks! Yeah I don't really have anything to bring up, also [16:04:23] Was the missing 'g' on account of only being "partially here"? [16:04:43] Grace just re-did the invite to take off a lot of ppl who don't go any more, and she must have added you since you subbed for me a few times lately [16:05:21] heh, nope, i just hit up to edit last command "/nick ejegg|away" and backspaced a little too far! [16:06:26] cwdent, do you want anything from / plan to interfere with other teams? [16:06:43] if I ever work a half-day I'm totally going to truncate my IRC nickname to indicate it: dstr [16:06:53] hehe, nice [16:07:18] you would be eje? [16:07:36] sure, still identifiably me [16:09:43] ejegg: nothing i can think of, thanks [16:09:54] ok, cool [16:10:21] i imagine it'll be a quiet one, with everybody recuperating from Mexico [16:15:58] does anybody know where they post videos of wikimania sessions? I see one on commons and one on youtube, but not much else [16:19:48] (CR) Ejegg: [C: -1] "Oops, I neglected this when you submitted, then made a duplicate patch exactly a month later that you were kind enough to merge." [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/209846 (owner: Awight) [16:23:58] (CR) Ejegg: "Yes, we should definitely do this! Guess this is where we'll need to decouple PaymentMethod from the gateway object, and maybe make it or" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/222699 (owner: Awight) [16:26:02] (CR) Ejegg: "Newer DI is a go! This should be mergeable now." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/225261 (https://phabricator.wikimedia.org/T105848) (owner: Ejegg) [16:40:26] hi awight ! [16:40:41] Mornoon! [16:41:00] and a loverly mornoon it is [16:41:50] i merged the first 2 of those 1.25 changes, but noticed some unwanted require-dev additions to the vendor submodule [16:42:40] oof, thanks for catching that! I noticed too but ineffectually shrugged [16:42:48] (new Ayn Rand book title) [16:43:21] slackjectivism? [16:43:45] hehe. "and then Atlas realized he was too much of a egotistic dick to have any friends" [16:47:46] So glad I haven't read that [16:48:15] heh, ditto. [16:48:41] page count + personality of recommenders was enough to deter me [16:50:25] awight, anything you want me to bring to scrum of scrums? [16:50:59] ejegg: not that I can think of! [16:51:08] k [16:52:26] http://www.newyorker.com/magazine/2013/07/29/ask-ayn [16:54:16] (PS3) Awight: update vendor submodule [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/226093 (https://phabricator.wikimedia.org/T94528) [16:56:55] cwdent: awesome, thanks. "I am joking, because I am not joyless." [16:58:02] "... which will be made into a movie in the year 1982, and the market will reward it with success." [17:31:54] oh, so civi actually does a generated create table for these custom groups [17:31:58] the plot thickens [17:34:50] Yeah... For the legacy ones, at least. For the newer fields, we're able to build up our own table schema as we see fit. [17:35:04] Remind me what you're trying to do? [17:35:20] Get the development schema to match production? [17:35:29] just trying to find if those engage batch ids are actually making it in [17:35:37] Aaah [17:35:52] i think they are...possible they just aren't exposed in the UI? [17:36:00] They should be visible [17:36:01] per https://phabricator.wikimedia.org/T100853 [17:37:47] cwdent: Here's an example where this import was working, select * from civicrm_contribution c join civicrm_value_1_check_number_9 cu on c.id=cu.entity_id join wmf_contribution_extra e on e.entity_id=c.id where e.gateway='engage' limit 10; [17:38:29] Oh, looks like it *is* working in recent imports, select * from civicrm_contribution c join civicrm_value_1_check_number_9 cu on c.id=cu.entity_id join wmf_contribution_extra e on e.entity_id=c.id where e.gateway='engage' order by c.id desc limit 10; [17:40:14] yeah... perhaps it was a problem with the files? she's definitely right about contribution type, i have a fix for that [17:41:23] The one she mentioned does have a batch number: select * from civicrm_contribution c join civicrm_value_1_check_number_9 cu on c.id=cu.entity_id join wmf_contribution_extra e on e.entity_id=c.id where c.trxn_id = 'ENGAGE C7A2E36D9D4FEE010C3B8CB2306143B9'\G [17:43:09] huh, interesting, could it just be non obvious in the UI? [17:44:06] Wow, search by trxn_id sucks. [17:44:58] heh, search is hard [17:45:16] esp text search on mysql tables with 400M rows [17:45:39] That's silly, it should be an exact search [17:46:20] Anyway, contribution_id=11956857, you can type that into the quick search in the upper left corner [17:47:29] cwdent: aaah, I think the contribution type thing is breaking display of the "check_number" custom group [17:47:39] cos it doesn't apply to the Cash type. [17:48:07] ah, so many the type fix will get both? [17:48:22] One can hope [17:48:31] well i'll push that up for CR quick [17:48:37] d'argh [17:49:04] I just tried to change the type in the GUI, and got blocked by the thousands separator in the amount. [17:49:08] How did that get in there... [17:49:42] why would that affect contribution type? [17:49:54] It prevents the form from validating [17:50:31] doh. so i regex'd out commas for square. should that go in checksfile? [17:50:44] ouch, and we can't set it to Engage somehow, because there's a validation for batch number! [17:50:51] Yeah, please push that into the base class. [17:51:15] We want to eventually train everyone out of using punctuation, but as long as we lock out the Europeans, this should be a good fix for now. [17:51:39] what about... [^0-9.] [17:52:43] there is probably a money parse lib for php that would be a little smarter [17:52:55] It's so scary... [17:53:15] Cos: european uses "," to separate the pennies and suddenly we've multiplied by 100 [17:53:24] uuugh right [17:53:37] I sort of prefer to have validation block on commas... [17:54:12] With square, we couldn't necessarily change them, but for our internal format I think we're good using a strict validation [17:54:37] yeah, and it looks like engage is all standard? [17:55:56] Yep, they're a contractor [17:56:18] so all the amount types are decimal(20,2)... mysql lets a comma in there?? [17:56:37] I... it... wat [17:56:49] you are right [17:57:02] uuhm yeah...wtf...could it just be display? [17:57:28] https://civicrm.wikimedia.org/civicrm/contact/view/contribution?reset=1&action=update&id=11956857&cid=492797&context=contribution [17:57:33] try to save [17:57:50] total_amount: 1000.00 [17:57:53] in the db [17:57:57] I'm broken [17:58:08] it must be some front end form validate? [17:58:12] yeah [17:58:16] displays a comma, doesn't want a comma [17:58:53] i only see USD in the dropdown, is that all that ever shows up here? [17:59:17] if so we could just stop showing the comma or strip it before save [17:59:21] Yeah, total amount is always converted to USD [17:59:30] I can't believe this has been this broken all along, though [17:59:51] yeah...has no one ever saved a >$1k contribution? sounds fishy [18:01:09] I need a burrito to go any further [18:01:16] (PS4) Awight: update vendor submodule [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/226093 (https://phabricator.wikimedia.org/T94528) [18:06:44] yeah good idea [18:27:24] ejegg: Yeah, I don't think monolog is actually used by core in 1.25 [18:27:48] ouch, mvn [18:27:49] nvm [18:27:56] grep -r Monolog indeed [18:28:42] yeah, why is that only 'suggests'? [18:29:04] Well, $wgMWLoggerDefaultSpi defaults to array( 'class' => 'MWLoggerLegacySpi' ) [18:29:09] So I think we're good [18:30:08] ah, ok then [18:34:03] (PS1) Cdentinger: corrects engage contribution type [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/226325 [18:34:19] Fundraising Tech Backlog, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Unfathomable comma issue makes it annoying to edit contributions >= $1000 - https://phabricator.wikimedia.org/T106568#1471908 (awight) NEW [18:34:58] awight: ^ that seems to work for new ones, but what's the policy about correcting the legacy data? [18:35:38] cwdent: You can do it with raw SQL, or write a drush command if it might happen again. [18:35:52] (CR) jenkins-bot: [V: -1] corrects engage contribution type [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/226325 (owner: Cdentinger) [18:35:58] It's recommended to paste the SQL you use into the bug [18:35:58] dang [18:36:12] If you have any doubts, other people would be happy to review the SQL before you run it [18:36:27] cool, yeah. am i crazy or should everything that says engage should have that contribution type? [18:36:54] sounds right to me. The tests need a little updating, but that's all [18:37:46] yeah, i should make that commit hook to run the tests [18:37:54] hehe [18:42:52] - 'gross' => 51.229999999999997 [18:42:54] + 'gross' => '51.23' [18:42:56] https://pbs.twimg.com/profile_images/2727748211/c3d0981ae770f926eedf4eda7505b006.jpeg [18:43:10] fixed itself when i fixed the expected contribution type... [18:43:19] baaa [18:44:12] i blame the leap second [18:44:25] and obama [18:44:50] that is real creepy, cos when I looked, the expected value was 51.23, float [18:45:00] yeah. floats. can't trust that [18:45:43] juggling types between mysql and php [18:45:46] priceless [18:45:49] literally [18:46:05] ssssh [18:46:07] the amount of money that is lost to that globally must be staggering [18:46:09] this is where we take our big cut [18:46:21] hehe [18:46:28] .0000000000000003 x all the people does add up to almost something [18:46:29] office spacing wmf [18:46:45] We're just a/b testing various hustles [18:47:06] maybe we should publish some results [18:47:17] (PS2) Cdentinger: corrects engage contribution type [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/226325 [18:52:35] (PS5) Ejegg: update vendor submodule [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/226093 (https://phabricator.wikimedia.org/T94528) (owner: Awight) [18:53:05] (CR) Ejegg: [C: 2] "looks good!" [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/226093 (https://phabricator.wikimedia.org/T94528) (owner: Awight) [18:55:17] (CR) Awight: [C: 2] "Might be a case issue, we'll see." (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/226325 (owner: Cdentinger) [18:56:47] (PS1) Ejegg: Use RSA host key in AstroPay example config [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/226330 (https://phabricator.wikimedia.org/T94088) [18:57:09] (Merged) jenkins-bot: corrects engage contribution type [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/226325 (owner: Cdentinger) [18:58:03] i just threw this in .git/hooks/pre-commit: /srv/crm/vendor/bin/phpunit || exit 1 [18:58:21] (CR) Awight: [C: 2] "Slightly sketchy that we're storing these in the example config, but I'm all in favor of making it easy to deploy..." [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/226330 (https://phabricator.wikimedia.org/T94088) (owner: Ejegg) [18:58:23] (Merged) jenkins-bot: Use RSA host key in AstroPay example config [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/226330 (https://phabricator.wikimedia.org/T94088) (owner: Ejegg) [18:58:46] cwdent: That's probably a good idea. Note that those hooks can't be commited, however >.< [18:59:15] works good, but on that note, ejegg there is that one DI test that fails strict mode cause of $_SESSION on cli...i have a stupid fix for it, but maybe you have a non stupid one? [18:59:17] (CR) Umherirrender: "recheck" [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/226064 (owner: Paladox) [19:00:19] cwdent: heh, nope, I think my fix was blunt force too: https://gerrit.wikimedia.org/r/217760 [19:00:45] (CR) Umherirrender: [C: 2] Remove i18n shim [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/226064 (owner: Paladox) [19:01:00] (Merged) jenkins-bot: Remove i18n shim [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/226064 (owner: Paladox) [19:01:47] oh heh, i must need to update DI [19:12:07] Fundraising Sprint ODB, Fundraising-Backlog: Sprint O - Wikimania! GOAL: Finish Banner History & Astropay. Investigate Amazon. Dance like nobody is watching. - https://phabricator.wikimedia.org/T102197#1472091 (DStrine) Open>Resolved [19:31:30] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Spike: Determine how to load CentralNotice RL modules and when to execute campaign and banner selection logic - https://phabricator.wikimedia.org/T106577#1472190 (DStrine) [19:36:33] Fundraising Sprint ODB, Fundraising Sprint P, Fundraising-Backlog, Astropay Integration, Patch-For-Review: Update Brazil fiscal number to CPF - https://phabricator.wikimedia.org/T106137#1472208 (Ejegg) a:Ejegg [19:36:54] Fundraising Sprint Lou Reed, Fundraising Sprint Miles Davis, Fundraising Sprint N*E*R*D, Fundraising Sprint ODB, and 4 others: Create an Civi import function for hand keyed (including DAF) donations - https://phabricator.wikimedia.org/T88836#1472211 (awight) @LeanneS: When you're happy with the im... [19:37:50] Fundraising Sprint P, Fundraising-Backlog, Astropay Integration: AstroPay gateway not incrementing order ID in some circumstances - https://phabricator.wikimedia.org/T106039#1472214 (Ejegg) a:Ejegg [19:38:21] Fundraising Sprint P, Fundraising-Backlog, Astropay Integration: Parse AstroPay error codes / description and display appropriate messages to donor - https://phabricator.wikimedia.org/T106053#1472219 (Ejegg) [19:43:40] Fundraising Sprint ODB, Fundraising Sprint P, Fundraising-Backlog, Astropay Integration, and 2 others: Validate CPF for AstroPay - https://phabricator.wikimedia.org/T106171#1472259 (Ejegg) a:Ejegg [19:44:32] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Spike: Determine how to load CentralNotice RL modules and when to execute campaign and banner selection logic - https://phabricator.wikimedia.org/T106577#1472272 (DStrine) [19:46:20] Fundraising Sprint N*E*R*D, Fundraising Sprint ODB, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, and 3 others: Publishing translations for central notice banners fails - https://phabricator.wikimedia.org/T104774#1472288 (awight) [19:49:08] Fundraising-Backlog: Set up import for Major Gifts events payment/invitation tool - https://phabricator.wikimedia.org/T101191#1472303 (awight) [20:02:08] Fundraising Sprint P, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Spike: Determine how to load CentralNotice RL modules and when to execute campaign and banner selection logic - https://phabricator.wikimedia.org/T106577#1472406 (DStrine) [20:02:10] Fundraising Sprint P, Fundraising-Backlog: Recent recurring GC donations not recurring or displaying as such at GC - https://phabricator.wikimedia.org/T105041#1472407 (DStrine) [20:02:13] Fundraising Sprint N*E*R*D, Fundraising Sprint ODB, Fundraising Sprint P, Fundraising-Backlog, and 4 others: Publishing translations for central notice banners fails - https://phabricator.wikimedia.org/T104774#1472408 (DStrine) [20:02:16] Fundraising Sprint Lou Reed, Fundraising Sprint Miles Davis, Fundraising Sprint P, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Recent Engage Import - Contribution Type = Cash when it should be Engage - https://phabricator.wikimedia.org/T100853#1472411 (DStrine) [20:02:26] Fundraising Sprint ODB, Fundraising Sprint P, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Banner history mixins and data - https://phabricator.wikimedia.org/T90918#1472416 (DStrine) [20:02:29] Fundraising Sprint Lou Reed, Fundraising Sprint Miles Davis, Fundraising Sprint N*E*R*D, Fundraising Sprint ODB, and 5 others: Create an Civi import function for hand keyed (including DAF) donations - https://phabricator.wikimedia.org/T88836#1472418 (DStrine) [20:02:31] Fundraising Sprint Flaming Lips, Fundraising Sprint Grandmaster Flash, Fundraising Sprint House of Pain, Fundraising Sprint Indigo Girls, and 10 others: Implement campaign associated mixins per findings from T90913 - https://phabricator.wikimedia.org/T90915#1472417 (DStrine) [20:42:37] hey ejegg, after composer update my DI in vendor is at master but says Your branch is ahead of 'origin/master' by 2 commits. [20:43:15] regarding the fix for that test [20:43:31] cwdent: I think that's because vendor is only a real submodule in deployment [20:43:46] Or... 2 commits? [20:44:12] yeah, i'm confused [20:44:24] what does git log look like? [20:44:27] what's the procedure for keeping that up to date? [20:44:40] cwdent: bit of a pita [20:44:40] last thing i see is Stop exceptioning in drupal formatMessage stub [20:44:41] If you have vendor cloned as a real repo, note that it won't get updated as a submodule, from DI:master [20:45:27] For development, you should remove the directory and just populate it with composer update. [20:45:32] awight: it's just from composer, not git clone [20:45:33] on master we just check in updated composer.lock and run composer install locally [20:45:34] yeah that's what i did [20:45:51] ls -ld vendor/.git u sure? [20:46:45] if you've ever checked out deployment and submodule updated, it doesn't remove vendor and .git when you switch back to master [20:46:49] so such file or directory [20:46:53] huh [20:46:54] *no [20:46:59] that's good. [20:47:32] "Stop exceptioning" is hella old. [20:47:45] current DI origin/master is d3d31c59cc59f6b91dd9ebcca8396c6c6758c3f2 [20:48:51] so git pull inside DI got me there [20:48:59] but why doesn't composer update? [20:49:11] composer update is unrelated to git [20:49:23] it just reads composer.json and composer.lock, and downloads that stuff to vendir/ [20:49:26] or [20:49:28] vendor [20:49:53] oh alright, i was thinking composer.lock would contain the specified versions [20:50:21] i mean i do see a sha in there [20:50:43] should that get updated? am i thinking about this wrong? [20:51:51] ah, also, the dev-master listed on packagist doesn't follow our repo [20:51:53] Donno how the shas get computed. Sometimes that's a git ID but maybe not always [20:52:05] i think i don't have it set up to crawl right [20:52:21] but anything evil that composer does with git doesn't show up in our file hierarchy [20:52:35] it's just wget -r; md5tree [20:53:26] after git pull in the DI dir, composer update does _not_ kick the version back to what's specified in .lock [20:53:52] composer update always looks for newer stuff; composer install should respect .lock [20:54:08] ooh word [20:54:33] and if you don't use --no-dev, you might have phpunit conflicts [20:54:50] or maybe i'm just misconfigured [20:55:17] so when i use --no-dev it removes phpunit [20:55:37] oh, i dunno if it removes an installed phpunit [20:55:44] but i need to use the composer one cause for whatever reason the apt one breaks [20:55:47] i do a lot of rm -rf vendor/ :P [20:55:57] hehe lemme try that [20:56:41] and you do composer install or update? [20:57:02] then composer install --no-dev [20:58:12] ok, do you then go git pull in the wmf dirs? [20:58:52] crm master has been updated with that test fix right? [20:59:12] its DI version i mean...maybe not? [21:04:20] cwdent: sorry, was there fix for those warnings that actually got merged? [21:07:10] oh geez haha i thought yours did [21:07:13] chasing a ghost [21:09:12] ejegg: so the one i did just does global $_SESSION; $_SESSION = array(); inside the test. do these things run CLI in real life and are throwing warnings right now? [21:09:44] cwdent: they do run in real life (the orphan slayer runs as a mw maintenance script via cli PHP) [21:09:56] Lemme just take a look at those logs [21:10:23] ok yeah so fixing it in the business files is probably best [21:10:40] also, i haven't looked at all the phpunit bootstrapping that happens for mw / drupal tests, which might be partially setting up a session [21:12:51] oh right, it's also running under CLI via drush for the recurring job, and we're definitely logging those warnings there [21:15:38] cool, i'll test out this patch then [21:15:56] ejegg: what phpunit do you use? composer or apt or other? [21:16:37] apt [21:17:10] 4.2.6 [21:18:49] huh yeah it works...i was having all kinds of problems before [21:18:51] oh well! [21:42:31] Fundraising Sprint Kraftwerk, Fundraising Sprint Lou Reed, Fundraising Sprint Miles Davis, Fundraising Sprint N*E*R*D, and 6 others: Upgrade payments mw-core to REL1_25 - https://phabricator.wikimedia.org/T94528#1472913 (awight) [21:56:56] Wikimedia-Fundraising, MediaWiki-extensions-CentralNotice, Epic, Patch-For-Review, and 2 others: Special:RecordImpression should die in a fire - https://phabricator.wikimedia.org/T45250#1473036 (ori) What's the status of this? Are we any closer to the complete shutdown of S:RI? [22:10:45] Fundraising Sprint Lou Reed, Fundraising Sprint Miles Davis, Fundraising Sprint The Pogues, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Recent Engage Import - Contribution Type = Cash when it should be Engage - https://phabricator.wikimedia.org/T100853#1473107 (cwdent) The batch number... [22:20:25] https://puri.sm/ [22:20:34] damn it, now i want one [22:22:52] Wikimedia-Fundraising, MediaWiki-extensions-CentralNotice, Epic, Performance, Varnish: Eliminate PHP backend call for Special:RecordImpression - https://phabricator.wikimedia.org/T106624#1473154 (awight) NEW [22:23:06] AndyRussG|bassoo: ^ [22:23:33] awight: ah hrmm thx! [22:24:16] AndyRussG: I'm happy to take that one on, if you want to give me some guidance. [22:25:02] awight: thx u bet, if ur interested in impression logging! I also haz another one that needs some love and may be more urgent [22:25:08] One sec, I'll pull it up [22:25:30] http://meta.wikmedia.org/beacon/RecordImpression?params, I guess [22:25:43] https://phabricator.wikimedia.org/T97676 [22:25:48] https [22:25:56] awight: oh yeah sounds awesome [22:26:35] WRT the phab task ^ I asked ottomata to document the details of how kafkatee is working vis-a-vis udp2log [22:26:51] Basically the details of the pipeline will affect how you process the stats [22:26:54] Or should [22:27:15] As you can see on the task, no further response was had from ellery or ottomata [22:27:53] But I think that by looking at the code we could figure it out, it's probably not a lot of code, and then pass the doc to Ellery to get a final go-ahead, and then let them deprecate udp2log [22:28:31] The issue is as follows: what step you do the sampling at makes a world of difference [22:28:45] Fundraising Sprint Kraftwerk, Fundraising Sprint Lou Reed, Fundraising Tech Backlog, Analytics-Cluster, operations: Verify kafkatee use for fundraising logs on erbium - https://phabricator.wikimedia.org/T97676#1473174 (awight) @ellery: Ping, we're waiting to see if you think we should care abo... [22:28:45] (I think! I guess I may be wrong) [22:29:20] I've heard ewulczyn use pretty much those exact words. I think he does care about this issue. [22:29:41] Let's say you get 1 000 000 requests, and you have to log calls to Special:A, Special:B and Special:C and want to sample [22:30:54] So let's say you filter first to get a set of all the calls to A, B and C [22:31:06] Wikimedia-Fundraising, MediaWiki-extensions-CentralNotice, Epic, Performance, Varnish: Eliminate PHP backend call for Special:RecordImpression - https://phabricator.wikimedia.org/T106624#1473176 (awight) [22:31:07] Then you randomly take 1 of ever 100 of that set [22:31:31] And then you dole out your logs to loggerA, loggerB and loggerC [22:32:18] In that case you can't say that you're sampling 1 of every 100 for logger A because you're also randomzing the proportion of A, B and C in your sample [22:32:40] So you couldn't rightly use the data as if it were a 1:100 sample of A [22:33:41] I guess they may well already know this, but I was gonna take a peek at the Kafkatee code to understand the pipeline and make sre [22:34:20] cwdent: query looks good. You know how to run write queries on barium? [22:35:51] eek, should we try on staging first? [22:35:52] AndyRussG: I see the issue. My gut instinct is that it can't be any worse than udp2log, which is "sampling" every 100th record with no randomness, AFAIK [22:35:57] cwdent: ya sure [22:36:19] awight: I dunno worse or better, just so long as we know exactly what it's doing [22:37:32] In the above case, even if it's sampling every 100th record exactly, that may be OK, but where does the sampling happen? Are things filtered before or after the sample? And is the new kafkatee different? [22:37:37] Ah, and I believe kafkatee samples randomly, so it will almost never be 1/100 the number of actual impressions. [22:39:40] Right, but is it { all requests -> sample 1/100 -> filter out only S:RI } , or { filter a set of requests of interest -> sample 1/100 -> filter out only S:RI }, or {filter out only S:RI -> sample 1/100 } ? [22:40:04] Sorry, I'm remaining silent on that cos I have no clue [22:40:28] IMHO only the last one is really correct, but in truth it's not my department and I may be missing something (like maybe it's OK 'cause it averages out? dunno), but... [22:41:13] what I think we need to close the task on our end is to make a small document that compares the old and new pipelines at this level of detail side by side, and ask Ellery to look at the doc and sign off that the new thing is OK with him [22:41:27] ok, sure I can do that. [22:41:37] awight: fantasmagorical! thanks a ton! [22:41:52] Fundraising Sprint Kraftwerk, Fundraising Sprint Lou Reed, Fundraising Tech Backlog, Analytics-Cluster, operations: Verify kafkatee use for fundraising logs on erbium - https://phabricator.wikimedia.org/T97676#1473197 (awight) a:AndyRussG>awight [22:41:58] Yeah this'll also feed into the new S:BL logging pipeline [23:04:15] Fundraising Tech Backlog, MediaWiki-extensions-DonationInterface, Recurring-Donations: Spike: Errors while charging recurring GlobalCollect might be handled wrong - https://phabricator.wikimedia.org/T105857#1473224 (awight) Interesting! I'm not sure we're having granular fail like that, but it sound... [23:18:02] Fundraising Sprint Lou Reed, Fundraising Sprint Miles Davis, Fundraising Sprint N*E*R*D, Fundraising Sprint ODB, and 5 others: Create an Civi import function for hand keyed (including DAF) donations - https://phabricator.wikimedia.org/T88836#1473289 (LeanneS) Thanks, Adam! Both the individual and... [23:18:17] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic: Bulk import for existing and future payment methods into CiviCRM - https://phabricator.wikimedia.org/T90242#1473294 (LeanneS) [23:18:19] Fundraising Sprint Lou Reed, Fundraising Sprint Miles Davis, Fundraising Sprint N*E*R*D, Fundraising Sprint ODB, and 5 others: Create an Civi import function for hand keyed (including DAF) donations - https://phabricator.wikimedia.org/T88836#1473292 (LeanneS) Open>Resolved [23:19:04] Fundraising Sprint Lou Reed, Fundraising Sprint Miles Davis, Fundraising Sprint N*E*R*D, Fundraising Sprint ODB, and 5 others: Create an Civi import function for hand keyed (including DAF) donations - https://phabricator.wikimedia.org/T88836#1473302 (awight) Woohoo! /me drinks a big glass of wate... [23:22:17] (PS1) Cdentinger: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/226449 [23:29:23] Fundraising Sprint Kraftwerk, Fundraising Sprint Lou Reed, Fundraising Tech Backlog, Analytics, and 2 others: Verify kafkatee use for fundraising logs on erbium - https://phabricator.wikimedia.org/T97676#1473365 (awight) [23:30:37] Fundraising Sprint The Pogues, Wikimedia-Fundraising, MediaWiki-extensions-CentralNotice, Epic, and 3 others: Special:RecordImpression should die in a fire - https://phabricator.wikimedia.org/T45250#1473381 (awight) [23:48:26] (CR) Cdentinger: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/226449 (owner: Cdentinger) [23:48:30] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/226449 (owner: Cdentinger) [23:53:57] Wikimedia-Fundraising, MediaWiki-extensions-CentralNotice, Epic, Patch-For-Review, and 2 others: Special:RecordImpression should die in a fire - https://phabricator.wikimedia.org/T45250#1473596 (awight) [23:54:05] Fundraising Sprint The Pogues, Wikimedia-Fundraising, MediaWiki-extensions-CentralNotice, Epic, and 2 others: Eliminate PHP backend call for Special:RecordImpression - https://phabricator.wikimedia.org/T106624#1473598 (awight) [23:56:55] !log updated civicrm from 292ad137f6b3ffc818a3bd617ca4f335931091f3 to 83cacfa1e0852ffaf47d2f02e7d843cf6f3bcda4 [23:57:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master