[00:00:18] awight: yeeeeah, just doing the merge patches now :) [00:01:02] AndyRussG: oh sorry! UTC fail. I see you now. [00:01:08] Done selling my soul for a Toyota hatchback [00:01:14] Tercel?? [00:01:16] awight: it's OK! I do want to smoke test [00:01:19] Matrix [00:01:41] oh good, that's worth a soul [00:01:42] It's a Corolla with a hatchback [00:01:57] nice form factor [00:02:10] If my credit were better it'd have cost me only 20% of a soul [00:03:00] But yeah even with the school so close and my working from home, there are times that not having a car is really inconvenient [00:03:31] Buses here are decent but there's only so much to be said about hauling tired kids on the bus and subway in the winter [00:03:36] what have you been doing? Extending the igloo's antechamber? [00:03:57] No, taking the bus :) [00:04:05] And not going out [00:04:13] Mmm and a few times, renting a car for a day [00:04:19] ouch! [00:05:02] yeah... well, I mean, if we would only use the car on the weekends, renting every weekend would be not that much more expensive [00:05:10] But you have to go and get it and bring it back [00:06:04] I wanted a plug-in Prius but that'll have to wait until after we get hats with solar panels [00:11:15] that's really weird!! gitg doesn't show my wmf_deploy branch correctly. It puts it at a differnt sha than gitk [00:12:44] Huh when I run gitk from git cola it does the same as gitg [00:13:05] I'm just gonna pretend that didn't happen [00:15:26] oooh that's why, I was looking at the wrong repo [00:15:32] * AndyRussG git-facepalms [00:15:52] almost deployed CentralObscure instead of CentralNotice [00:16:00] I mean, CentralOverlook [00:17:09] O_o [00:19:05] (PS1) AndyRussG: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/194002 [00:20:11] atgo: awight: K4-713: ejegg: pizzzacat: anyone know if and how we should alert the community consultation banner owners that there's a CN deploy coming up? [00:20:22] Ah. [00:20:24] Just so they shout out if there's an issuze [00:20:36] is that jamesofur? [00:20:37] I think jamesofur is holding that one up. [00:20:40] Yes. [00:20:56] yeah he's not in IRC [00:21:01] nope [00:21:01] i did just see him IRL though [00:21:03] K4-713: atgo: I'll cc him on the e-mail to fr-all? [00:21:09] yeah that makes sense [00:21:13] :) thx! [00:21:16] * K4-713 nods [00:21:28] i'm going to ping him in gchat too just in case [00:21:51] AndyRussG: that's courteous, but we haven't given him any impression numbers yet, so there's probably nothing to change from his perspective :) [00:22:19] yeah awight is there anything we can/will do about that..? or are we just sort of up a creek? [00:23:37] awight: huh I thought that had happened [00:24:15] (CR) AndyRussG: [C: 2] Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/194002 (owner: AndyRussG) [00:24:47] atgo: the deploy starts at 17:00 PST [00:26:47] cool, thanks! yeah cc him on the email and i also gave him a heads up [00:27:48] atgo: AndyRussG: I've been working on it in the background for days... I was told at one point to kill my query, cos it was overloading the anal cluster!! [00:28:01] hence, https://phabricator.wikimedia.org/T90649 [00:28:46] oh yeah I remember seeing that [00:29:03] are u doing something very different from what ellery does? [00:29:17] I don't think so, it just takes *hours* [00:29:42] udp2log isn't good enuf for then? [00:30:05] (Merged) jenkins-bot: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/194002 (owner: AndyRussG) [00:30:08] Jenkins! [00:31:28] AndyRussG: well, the udp2log results in our db don't include hide numbers [00:31:50] AndyRussG: I could run a custom grep, I spose [00:31:57] aaaah yeah I rememberu even asked me about it hmmfpffb [00:32:01] is that what you guys do?? [00:32:27] must have some faulty brainbits [00:32:33] * awight acts like I've never heard of this toolchain before :) [00:32:46] amnesia? [00:33:00] Sorry you are inheriting one of the larger messes I've inherited :) [00:33:47] Wikimedia-Fundraising, Analytics: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#1080580 (atgo) p:Triage>Normal [00:33:51] Heh it's OK [00:34:19] I wash my hands every few hours, when I think about how I want to murder that code... [00:36:04] o_O [00:36:09] out out damn code-spot? [00:36:27] I can still smell the breakpoint. [00:37:35] the burnt sulpher smell of a still-smoulduring breakpoint and the pitter patter of a fleeing xde-bug [00:37:48] So many little feet. [00:38:19] moral is: don't make assumptions cos a program not born of woman could easily end your career [00:38:22] or maybe I read that wrong [00:38:43] the moral is: carry a bough whenever you want to deploy code. [00:39:49] !log disabled queue consumers [00:39:52] Logged the message, Master [00:40:18] AndyRussG: ping [00:40:28] ori: hi! [00:40:46] How's it going? [00:41:15] OK :) I'm looking at your patch for sampling S:RI [00:41:31] you have if ( Math.random() > sampleRate ) { return; } [00:41:33] ori: which, the just-merged one, or the WIP one? [00:41:43] yeah.... [00:41:44] and the sample rate is set to 1/100 [00:41:51] doesn't that make it log 99 out of 100 times? [00:42:32] nah, that should be correct [00:42:57] !log updated crm from 3c002f32e04652ae56a4fe791bc6158ab981ed8d to f8fb0f61531431348f3a8a3ee107056a864d537b [00:43:00] Logged the message, Master [00:43:16] ori: Yeah I think that's OK [00:43:28] ori: basically "return" there means, don't call S:RI [00:43:59] for a second I was worried since I've only been smoke testing with relatively high values like 1/2 and 1/3 8p [00:44:16] wgCentralNoticeSampleRate = 1 / 100; [00:44:20] trials = 1000; [00:44:24] hits = 0; [00:44:29] while (trials--) { if (Math.random() > wgCentralNoticeSampleRate) hits++; } [00:44:38] that would be the opposite logic [00:45:24] oh yeah, that's correct. [00:45:28] I misread the code. My bad. [00:45:38] ori: np! really appreciate ur taking a look :) [00:46:14] ori: we're justa bout to deploy (not the WIP patch but the two before) if u have any other last-minute comments or concerns ;) [00:46:54] https://gerrit.wikimedia.org/r/#/c/194002/ [00:47:40] i have a patch [00:47:41] just a sec [00:48:07] (PS1) Ori.livneh: Send S:RI request asyncronously [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/194009 [00:48:15] ^ AndyRussG, awight [00:48:43] (CR) jenkins-bot: [V: -1] Send S:RI request asyncronously [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/194009 (owner: Ori.livneh) [00:49:15] re: sendBeacon -- see [00:50:02] (PS2) Ori.livneh: Send S:RI request asyncronously [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/194009 [00:50:19] ori: fancy! we did talk about something like that but I was gonna wait... [00:51:58] !log enabled queue consumers [00:52:00] really nice to see there's dedicated work on it... [00:52:02] Logged the message, Master [00:53:41] ejegg: nice one! [00:53:55] * ejegg dusts off hands [00:54:36] I ran a batch of one recurring gc, and our latest DI code seemed to handle it as usual [00:56:18] (CR) Awight: Send S:RI request asyncronously (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/194009 (owner: Ori.livneh) [00:57:27] ori: http://engineeredweb.com/blog/09/12/performance-comparison-documentcreateelementimg-vs-new-image/ [00:57:32] awight: fine by me; that would be even better [00:59:03] AndyRussG: great! People suffering under the yoke of IE8 will appreciate the 20% improvement :) [01:00:15] yeah and those using 6 and 7 will recant sooner [01:00:29] Aren't we breaking .js for them, anyway? [01:00:34] Yeah in fact I think so [01:00:49] uh [01:00:52] yep, https://www.mediawiki.org/wiki/Compatibility#Browser_support_matrix [01:01:00] the difference is something like an eight of a microsecond [01:01:16] http://jsperf.com/new-image-vs-createelement-img/ [01:02:40] (CR) AndyRussG: Send S:RI request asyncronously (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/194009 (owner: Ori.livneh) [01:03:06] AndyRussG: are you hoping to deploy ori's patch too, or just the stuff you merged half an hour ago? [01:03:15] i'm hoping it rides along [01:03:22] ok [01:03:50] ejegg: We have an hour and a half slot, so I have time to smoke test it, though not on a variety of platforms or anything... [01:04:02] ok, cool [01:04:18] We could merge it to master and kick the cross-browser tests, tho... Not sure how long master takes to get to the beta cluster [01:04:28] ori: awight: ejegg: ^ ? [01:04:44] < 5 minutes [01:04:49] K [01:04:55] I think deploying it would be fun :) [01:05:02] totes! [01:05:11] K lemme just smoke test locally [01:05:18] * AndyRussG readies fire extinguisher [01:07:22] * awight teleports coffee northwards [01:10:53] (Abandoned) Ejegg: Update DonationInterface and Smashpig [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/190393 (owner: Ejegg) [01:11:17] (Abandoned) Ejegg: Step 1: try to kill the submodules [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/190399 (owner: Ejegg) [01:12:30] ori: awight: ejegg: funny, this way I don't see the call to S:RI in my network tab in Firefox (Iceweasel) dev tools [01:12:34] Lemme try chrome [01:12:41] I do see the hits on the server tho [01:12:48] AndyRussG: maybe you're getting sampled? [01:13:06] either sampled or not, testing both... [01:13:44] AndyRussG: aww, we're into your deployment window already :( sorry! [01:14:00] awight: np! this is fun :) [01:14:23] hehe he's an adrenaline junkie [01:14:31] ^ may be a deficiency of FF dev tools [01:15:27] hmmm adrenaline times are actually when I tend to make my biggest gaffs [01:15:34] * AndyRussG takes a deep breath [01:18:53] yeah on chrome dev tools it does show up [01:21:35] awight: ejegg: K4-713: ori: with ori's very advanced patch, I'm getting POSTs to S:RI rather than GET requests. Is that gonna gunk up our ancient udp2log or any of the Hive queries that Ellery pulls? [01:22:07] I don't know [01:22:27] me either [01:22:28] AndyRussG: even for FF? I don't think I'm getting any requests to S:RI either. [01:22:29] Does it post to the full URL with all QS params? [01:22:32] it shouldn't matter [01:22:35] yes [01:22:39] (re: full URL) [01:22:50] yeah I'm getting the params fine [01:23:05] awight: clear your cache and check your apache log [01:23:17] k, I did find it in the log [01:23:23] Interesting. [01:23:25] very strange that it doesn't appear in the net panel [01:23:47] maybe you're filtering for image requests? it might show up as a different type of request [01:23:49] yeah! stealth background requests, that's an interesting bug [01:24:02] for someone that has a use for it [01:24:15] From my limited understanding of the udp2log and hive, I'd expect the POST requests to show up in the same place. [01:24:46] Last time I looked at udp2log, param order mattered. [01:24:52] But, that was like... 2012. [01:24:57] * awight sobs [01:24:59] param order isn't changed [01:24:59] K4-713: I think we fixed that [01:25:12] I thought I remembered something about that not being an issue anymore. [01:25:15] also yeah [01:25:21] and also I don't see param order changing [01:25:55] Great. [01:26:07] awight: ori: ejegg: K4-713: my only worry is the POST and all [01:26:11] Well... of course, just when things get interesting in here, I have to run off. [01:26:11] I know there are campaigns up [01:26:21] K4-713: but... its your call! [01:26:34] if you see a sharp decline, revert [01:26:57] I was just about to say that with slightly different words. [01:26:58] heh also a strategy [01:27:24] And, you know... email at least fr-tech when you're all done. [01:27:41] K if there are no objections I'll merge and we can prepare the patches while we watch cucumber tests, yea? [01:27:44] I don't see a reason to hold off, though. [01:27:46] K4-713: also yeah [01:27:51] K :) [01:28:16] * AndyRussG pulls trigger [01:28:28] (CR) AndyRussG: [C: 2] Send S:RI request asyncronously [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/194009 (owner: Ori.livneh) [01:28:41] ori: thanks a ton, eh? [01:28:58] thank you [01:29:05] (Merged) jenkins-bot: Send S:RI request asyncronously [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/194009 (owner: Ori.livneh) [01:29:48] AndyRussG: Good luck. Let me know how it goes. [01:29:53] AndyRussG: deploying to all wmf branches? [01:30:03] ejegg: yep! [01:31:08] (PS1) AndyRussG: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/194015 [01:31:20] wait don't! [01:31:23] just kidding [01:32:13] LOL [01:32:48] I'm not supposed to laugh too much while I work, my kids will come in and scold me [01:34:50] (CR) AndyRussG: [C: 2] Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/194015 (owner: AndyRussG) [01:35:57] (Merged) jenkins-bot: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/194015 (owner: AndyRussG) [01:38:46] K the new code is up on the beta cluster [01:40:10] BTW the background request is also not showing up in FF with Firebug [01:41:33] (CR) Awight: [C: 2] "This is great, thank you! There are a few comments inline, please address in follow-on patches if you'd like..." (16 comments) [wikimedia/fundraising/crm/civicrm] (giantrabbit-v4.4) - https://gerrit.wikimedia.org/r/192872 (owner: Anemirovsky) [01:42:28] (Merged) jenkins-bot: This commit contains new features for CiviCRM's report interface. [wikimedia/fundraising/crm/civicrm] (giantrabbit-v4.4) - https://gerrit.wikimedia.org/r/192872 (owner: Anemirovsky) [01:43:52] AndyRussG: ejegg: okay, I guess this is the perfect time for me to step out :D lmk how it goes! [01:44:15] awight: will do, thanks! [01:44:24] just kicked the browser tests [01:44:32] break a chair leg :p [01:44:35] (the ones that are passing normally, there's a bit of a WIP mess in there) [01:44:58] gnight adam [01:45:03] too bad the browser tests have such a long queue wait [01:45:38] time is out of joint... [01:45:39] k [01:46:53] actually they're going relatively fast [01:53:43] ejegg: here's wmf19: https://gerrit.wikimedia.org/r/#/c/194020 [01:53:48] (I don't have +2) [01:54:29] ok, one sec! [01:55:48] heh now it'll definitely merge! :) [01:56:15] as long as the butler doesn't kill it! [01:58:10] https://gerrit.wikimedia.org/r/194023 [01:58:49] we think he's the butler but in fact he's running the show [01:59:53] ejegg: ori: https://gerrit.wikimedia.org/r/#/c/194020/ jenkins fails? [01:59:54] aargh, that dumb gallery tag failure [02:00:05] happened to a trivial patch of mine this morning too [02:00:11] is that a bogus failure? [02:00:13] I can override it if so [02:01:05] weird, it's even more failures than I was getting this morning [02:01:07] https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm/3510/console [02:01:11] but none of them look related [02:01:14] to CN [02:02:17] ori: I'm pretty sure those are all bogus [02:02:40] well searching the full logs for "centralnotice" turns up nothing... [02:06:25] http://etherpad.wikimedia.org/p/LogsOfAnotherNoviceDeployer [02:06:52] ejegg: ^ ori (if you're interested) [02:06:59] heh [02:07:19] I'm on tin... [02:09:57] ejegg: just rebased wmf19 [02:10:18] whee, patches! [02:13:49] K no CN security updates... [02:14:14] cool [02:14:42] did submodule update [02:15:11] good stuff [02:15:23] ready to sync-dir, then? [02:15:38] I think so, right? [02:15:46] yep [02:16:05] same command as the sync-file, no? [02:16:20] yep [02:18:04] sync-dir php-1.25wmf19/extensions/CentralNotice/ 'CenralNotice update' ? [02:18:21] ejegg: ^ ? [02:18:22] that looks right [02:18:31] K... [02:18:32] here goes [02:19:45] Success, apparently :) [02:20:21] nice! [02:21:55] now 18 [02:23:16] yup [02:24:58] K diff looks fine (see etherpad) [02:26:21] No CN security patches... [02:26:32] Doing the submodule [02:26:33] update [02:27:12] ejegg: look ready for the sync-dir again? [02:27:19] ori: ? [02:28:01] yes [02:28:12] cool! thx, here goes [02:28:21] yep [02:28:46] Command is sync-dir php-1.25wmf18/extensions/CentralNotice/ 'CenralNotice update' [02:29:04] K here goes [02:29:06] looks good [02:31:56] yup! [02:32:04] ori: ejegg: try this in Chrome: http://en.wikipedia.org/wiki/Boris_Nemtsov?debug=true&random=0.1 [02:32:30] The random forces the browser to choose the allocation segment that does have the community banner, you can see S:RI going out there via POST [02:33:04] And like this, you can see it's not going out, which is the other significant change: http://en.wikipedia.org/wiki/Boris_Nemtsov?debug=true&random=0.5 [02:33:45] (execpt 1 in 100 times) [02:34:59] yep, I see the request [02:35:51] The mobile site is not broken or anything... [02:38:50] Iinteresting, the strategy consultation doesn't go out on mobile, only desktop [02:42:40] ejegg: ori: also banners showing up correctly on mobile [02:42:45] http://en.m.wikipedia.org/wiki/Boris_Nemtsov?debug=true&country=SE [02:42:59] (you have to do it with an android or iphone mobile UA, though) [02:43:00] yep, all looks good to me so far [02:43:14] hide seems to be working fine [02:43:22] yup! [02:43:34] ejegg: ori: thanks for staying around and for ur help :) [02:43:41] yw! [02:44:03] ejegg: I'll send an e-mail to FR tech [02:44:21] ejegg: ori: ...and I'll comment on the Phab task [02:44:41] good stuff [02:44:42] ejegg: do you want to try monitoring upd2log? [02:44:49] or tell me how briefly :) [02:44:55] or shall we leave it 'till morning? [02:44:57] i'll take a look on the cluster [02:45:16] fantastic! thanks much :) [02:47:12] it's a little late, but here's your cat of the day: http://36.media.tumblr.com/5871d516d73704523ec3c511f8ef61dc/tumblr_nk4jkcuGSE1rl1jado1_1280.jpg [02:47:37] ty pokercat^W pizzzacat [02:48:17] :) [02:49:40] pizzzacat: my! [02:49:42] thanks [02:54:03] AndyRussG: S:RI numbers from the udp logs look a lot like last monday at this hour [02:54:18] ejegg: fantastic! [02:54:34] Are we sure we're past all caching thresholds? [02:55:27] no idea! [02:56:07] I can see on enwiki that even without debug=true we're getting the new JS that makes the post instead of the get [02:56:30] ejegg: I think as long as we confirm that the POSTs are getting through udp2log, we're good [02:56:31] sounds good to me... [02:56:51] how long from actual request to server to udp2log? [02:56:58] ori: ^ any idea? [02:57:27] oops, I was looking at numbers from before the deploy [02:57:30] one moment please [02:58:19] ahh, hmm [02:58:29] numbers may be down significantly... [02:59:37] crud [02:59:57] ejegg: oh noes! [03:00:07] ejegg: wait, no, that's OK... [03:00:12] ejegg: that's what this was for, in fact [03:00:14] is it? [03:00:27] oh, the hide ones [03:00:41] let me try just counting result=show [03:00:50] well, yeah! We made S:RI sampled instead of unsampled for the empty allocation block when there are throttled campaigns [03:01:50] ok, much less discrepancy [03:03:12] grepping for result=show, I get 6429 for the last 15 min and 6487 for the chunk before that [03:03:27] and one week ago, those numbers are 7177 and 7263 [03:03:37] ejegg: fantastic, yeah I think that's good :) [03:03:38] which could just reflect campaigns in play [03:03:52] ejegg: what if you grep for "empty" for the same time slots? it should be much less [03:04:16] and that'll show that the time slots you're checking are carrying data created from the newly deployed [03:04:38] the dearly deployed! [03:04:53] yep, 15k a week ago vs 359 in the last 15 min [03:05:04] ejegg: wohoo! [03:05:15] That's also worthy of note in the Phab task, in fact :) [03:05:18] and only 125 of those are POSTs [03:05:34] the 'die in a fire' task? [03:05:49] heheh the respectfully cremated task yeah [03:06:00] will note [03:06:07] ah cool! [03:06:09] thanks [03:06:42] yes we do expect still some empties from POSTS [03:06:44] and also hides [03:07:00] The fact that you're seeing the posts in there means that they're getting thru [03:08:41] well! I'm happy we didn't break the world's foremost beacon of free knowledge...today... [03:10:33] https://code.google.com/p/fbug/issues/detail?id=7728 [03:10:59] yeah, I'll star that bug [03:11:53] Wikimedia-Fundraising, MediaWiki-extensions-CentralNotice, Epic, HTTPS-by-default, and 3 others: Special:RecordImpression should die in a fire - https://phabricator.wikimedia.org/T45250#1080917 (Ejegg) After tonight's deployment, RecordImpression numbers are down by at least 50% even before the new... [03:13:05] ok, signing off for the night! [03:13:18] ejegg|away: bye, thanks again! [03:30:36] Wikimedia-Fundraising, MediaWiki-extensions-CentralNotice, Epic, HTTPS-by-default, and 3 others: Special:RecordImpression should die in a fire - https://phabricator.wikimedia.org/T45250#1080937 (AndyRussG) Hi... here are the patches deployed for this: - https://gerrit.wikimedia.org/r/193767 sample... [05:19:09] (PS2) Ssmith: Chart now renders [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/193535 [05:19:55] (CR) jenkins-bot: [V: -1] Chart now renders [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/193535 (owner: Ssmith) [17:27:51] § Fundraising Sprint Devo, § Fundraising Sprint the Cure: Add contribution tracking ID to credit card statement on Worldpay - https://phabricator.wikimedia.org/T88060#1082374 (Jdforrester-WMF) [17:39:24] * K4-713 burbles lightly [17:39:53] Jeff_Green: Good Aftermath. How are we doing? [17:40:04] oh fine [17:40:12] Grand. [17:40:34] I... am becoming vaguely overwhelmed with the number of things to get done. [17:40:51] But, just vaguely. And we have lists. [17:41:33] (I guess it's kinda weird it took this long) [17:42:36] Jeff_Green: Er, I think I may have to go through some chatlogs for last week, and make double-sure I have everything on my list. [17:42:53] How late are you available today for a phone call with Tom? [17:43:37] if we can be done by 5:30 it's fine [17:44:02] Yours or mine? [17:44:04] :) [17:44:13] I'm guessing yours. [17:44:19] UTC! too late! [17:44:22] mine yeah [17:44:26] haha [17:44:41] what's on the agenda for the call? [17:44:58] I think Tom just wanted to check in about all the things up in the air. [17:45:12] And make sure there wasn't anything on our list that he didn't have. [17:45:29] I'll probably just .pdf the collab page and send it to him sometime before that. [17:45:49] alright [17:46:09] I think he's got a better list of all the documents he was waiting for us to cough up, vs what he actually has. [17:46:14] That got way out of my control. [17:46:35] yeah [17:46:44] Good news in email this morning, though. [17:46:56] it would be great to have even a crummy template for each doc we need [17:47:08] i mean even if it's some janky boilerplate, just as a placeholder [17:47:13] Oh, I wasn't talking about the ones we need to *write*. [17:47:14] what news? [17:47:18] oh ha [17:47:26] I was more referring to all the things we probably had already. [17:47:29] oic [17:47:47] Once you started sending things too, I totally lost track of what went where. [17:47:55] ha [17:47:59] So, he's going to have to tell us where we are with that. [17:48:12] maybe we should start ticketing everything [17:48:17] Yep, that was my plan. [17:48:27] "THANK YOU FOR YOUR REQUEST TOM, WE'LL GET SOMEONE RIGHT ON THAT." [17:48:33] RIGHT AWAY. [17:48:47] Despite my intense impulse to dive right in and do the things I know I can do... there are other people I need to start pushing around. :) [17:49:20] * Jeff_Green upvotes new "wallop" powers for K4-713 [17:49:32] MUAHAHAHAha. *cough* [17:50:59] Oh, and the good news was the news about the contribution tracking database being not in scope without an uphill battle. [17:51:14] That one... worried me. [17:52:31] Implementing UUIDs throughout the pipeline (and in fr-analytics, and all the accounting crm stuff) in under two weeks is not my idea of a teambuilding party. [17:52:59] And, that was the only way out I could see. [17:55:35] yeah [18:01:07] bummer, another BAD_AUDIT_LINE. I had really hoped Paypal's fix last week would be the end of them [18:02:24] ejegg: As long as there are paypal payments, they'll find a way to emit bad audit lines. [18:03:13] heh [18:04:44] * K4-713 ponders trying to create a paypal account with the ascii beep character in the first name [18:05:15] that would be awesome [18:05:39] I bet they have something that doesn't... handle that properly. [18:05:56]