[16:35:15] ejegg AndyRussG K4-713... heads up on this one: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1963 [16:35:23] banner impressions are down :( [16:35:27] the-wub reported it this morning. [16:35:31] Oh dear. [16:35:42] yuuup [16:35:42] ooh, yikes [16:35:44] Oh deary deary deary dear [16:35:56] so he's saying that it looks like it happened before the CN deploy [16:36:04] hmmmmm. [16:36:09] Maybe? [16:36:20] Definitely a strong candidate. [16:36:24] oh wait, i think this could be the result of that cache problem [16:36:58] ejegg: Can you say more about that? [16:36:59] Are the numbers updating again, or is it still stalled? [16:37:21] The cache problem served the old bannerController with calls to $.toJSON / etc [16:37:37] and the new dependencies, which didn't include jquery.json [16:38:10] Thi the-wub [16:38:12] Hi [16:38:20] But the script was trying to use it in insertBanner, before it hits the impression recorder [16:38:23] hi AndyRussG [16:38:44] Resulting in a JS error and no impression recorded [16:39:30] I didn't think that was affecting 100% of users, though. I had to try a couple machines on crossbrowsertesting before I could replicate it [16:40:28] I'm actually not sure it does have anything to do with the js change though. I would expect a more gradual tailing off in that case, from people seeing old banners in that case [16:40:44] whereas in the database, they seem to stop quite abruptly [16:41:18] I just looked at the job. [16:41:26] they didn't start again around 20:30 UTC, did they? [16:41:35] Starting at Sep 16, 2014 5:38:05 PM (UTC), there wasn't anything to import. :) [16:41:39] Bam. [16:41:53] ejegg: What does that tell you? [16:42:28] That I need to read up on what is doing the importing from where :) [16:42:43] I've only played with the js side of this so far... [16:42:47] ...and it should be noted that the banner import job runs every five minutes. [16:42:50] * AndyRussG also frantically reads code... [16:42:53] So, it's not that big of a window. [16:46:32] the-wub: ejegg: K4-713 impressions are just recorded as hits on Special:RecordImpression, imported from a server log? [16:46:39] this is the DjangoBannerStats job? [16:46:55] It's called "Banner impressions loader" in jenkins. [16:48:14] we're definitely hitting Special:RecordImpression from the browser [16:49:52] Ah, if you tunnel in to jenkins, the logs from builds from 3465 and before should be helpful. [16:50:29] Hmmm I don't think I'm so privileged as to be able to do such a thing [16:52:02] i think we have a missing logs folder [16:53:23] Hmm if there's log rotation at specific times that might make sense [16:54:50] Jeff_Green: Did you do anything weird yesterday? [16:54:55] :) [17:02:30] K4-713: yes? [17:02:30] probably? [17:02:32] i did package updates on the payments hosts. did something break? [17:02:47] on barium too actually [17:02:54] OH [17:03:08] Yeah, there may be either a permissions issue or something gone funny with logrotate for banner impressions. [17:03:25] i unmounted barium:/archive/udplogs [17:03:34] and it's not remounting.. why..... [17:03:52] can I reboot barium? [17:04:10] oh nm. i see the issue [17:04:21] \o/ [17:04:44] something changed re. iptables vs the netapp [17:08:27] grrrrr. [17:08:40] K4-713: can I reboot barium? [17:09:17] Jeff_Green: Do it. [17:09:25] i hate nfs [17:09:28] rebooting [17:18:34] atgo, awight, pizzzacat, K4-713, AndyRussG: Scrum of Scrums is soon. Anything you need from other teams or need to warn other teams about? [17:19:43] ejegg: nah not really, thanks for asking & for attending :) [17:21:41] * Jeff_Green is annoyed with nfs [17:23:15] ejegg not that I can think of! thanks! [17:23:33] ok, cool! [17:31:20] K4-713: alright, the netapp is mounted again. do you know--will the stats self-heal? or do we need to manually run to backfill [17:31:40] Jeff_Green: I don't know off-hand, lemme check the code... [17:32:30] Jeff_Green: ugh, yeah looks like we have to run w/o the --recent flag [17:32:36] Which I fear parses *every*thing [17:35:40] hrm [17:35:49] when you say everything... [17:36:18] should I move files aside or something? do you think it will die a horrible smoldering death trying? [17:44:02] (PS5) Ejegg: Add OANDA and ECB importers, update with fallthrough [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/159347 [17:59:55] ejegg: I'd like to deploy CRM code, anything in the pending column that you know needs babysitting? [18:00:11] let me check... [18:01:17] awight: don't think so. The new OANDA config is in there, but not the actual job change [18:01:42] OK rad [18:02:05] yeah, my hook change could cause some serious issues... I'll slow-start the queue consumer... [18:02:26] definitely sounds worth watching [18:07:25] (PS1) Awight: Merge remote-tracking branch 'origin/master' into HEAD [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/160999 [18:07:38] (CR) Awight: [C: 2] Merge remote-tracking branch 'origin/master' into HEAD [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/160999 (owner: Awight) [18:23:35] ejegg: CRM stuff has been deployed and restarted. [18:24:13] A moment of wild appreciation for CR and automated testing! [18:29:18] woo! [18:29:35] hey awight... if you're doing a bunch of deployments, will the state field in Australia go out today? https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1799 [18:31:13] ccogdill: awight want to go for lunch before the brown bag? maybe at quarter 'til? [18:31:40] sure! I don’t have anything today [18:31:45] where were you thinking atgo? [18:31:58] oh now you're asking me to think!? [18:32:02] goodness. [18:32:14] ermm.. i haven't been to working girl in ages? [18:32:41] hehe [18:32:45] working girl is good for me [18:36:11] ok. see you downstairs in 10 [18:45:28] heading to the ground floor, atgo. see you down there! [18:52:33] Hey awight ejegg, anyone got a sec for some CN questions? [18:58:55] AndyRussG: I've got some time and might even have an answer or two [19:00:30] ejegg: fantastic, thanks... so I'm looking at how buckets work... This is partly to give some answers that Ellery wanted, and partly also because I should just know (p [19:00:32] 8p [19:01:37] So if you peek at Special:GlobalAllocations on Meta [19:01:53] (oops it's one of those days... Our servers are currently experiencing a technical problem. This is probably temporary and should be fixed soon. Please try again in a few minutes.) [19:03:05] is that the memory issue you were just fixing? [19:03:19] Yes indeedy doo [19:03:38] And I don't say that very often, BTW [19:03:47] heh [19:03:53] ccogdill: https://collab.wikimedia.org/wiki/Fundraising/Engineering/Fun_SQL_Queries [19:04:20] Uuhh well if you were to look at that page and it were to work, you know under each allocation there's an explanation of which campaigns and banners get sent where [19:04:39] So it's a table with columns: project, country, bucket [19:05:18] And then the funny thing is that within each _bucket_ is where you get informaiton about how different campaigns interact: two or more campaigns competing for attention within a bucket [19:05:53] Which seems odd from the point of view of A/B testing, since there buckets are conceptually independent for each test you undertake [19:06:00] oh, okay... Shoot, I may not be much help here [19:06:22] But let me look at some code and try to understand better. [19:06:22] So is that really how it works? You get assigned a bucket before it's decided what campaign you go to? [19:06:30] Yeah that's what i was doing :) [19:07:17] So in specials/SpecialBannerLoader L39, it's getting a bucket as a parameter [19:07:50] I also see a getBucket in bannerController.js [19:07:53] I thought this was something I'd figured out, but that was 'slots', not 'buckets' [19:08:09] Hmmm what are slots? what's the difference? [19:08:31] slots are how to choose a random banner within a campaign, I think [19:09:00] In BannerChooser.php [19:11:30] Hmmm [19:11:31] OK [19:11:55] Well the fact that that has a different name would indeed seem to indicate that it comes after buckets [19:14:36] So the CN user doc also has some perplexificious bucket bits: https://meta.wikimedia.org/wiki/Help:CentralNotice#Targeting_with_Buckets [19:14:54] So you can add js code to a banner itself to change what bucket it goes into? [19:15:37] oh, interesting. Hard to think where that would help... [19:16:54] ejegg: yeah... Hmmm before I bother you to delve to deep lemme ask you quickly some stuff you might know offhand... [19:17:52] For example for AB tests involving banners, my undestanding from Ellery was that he doesn't get the full data set about impressions, just data on one in every hundred impressions? Is that correct? And I guess that comes off the same logs that borked this afternoon? [19:18:16] yeah, exactly [19:18:37] OK cool [19:18:57] I guess we have a custom apache UDP logger that just fires off crazy numbers of packets [19:19:20] and a bunch of filters that sample packets at different rates and decide whether to log them [19:19:29] Hmmm [19:19:41] I don't know anything about that architecture, I should learn 8p [19:19:52] Do you think it's practical to think about eventually feeding analyitics whole datasets of banner impressions? [19:20:38] The other question I have is: Ellery seemed to have the impression that there was a 7-day limit on cookies that made it impossible to run tests longer than 7 days [19:21:13] shoot, I guess I only have interacted with the hide cookies so far [19:21:22] Ah hmm OK [19:21:49] (I'm spewing questions like there's no tomorrow 'cause I have to rush off to get my kids from school, but I'll be back soon, and will leave the computer on, so I should get full backscroll) [19:22:17] I guess the real way to do it is to just debug thru the whole process a few times on both client and server [19:22:21] Hopefully someone with more knowledge than me will chip in and make reading the backscroll worthwhile [19:23:01] been helpful so far!! thanks much, gotta catapult, brb [20:29:04] Uhhhh [20:29:11] Why did I just get a Mingle email from atgo [20:29:20] ...huh? [20:29:24] You're in fundraising now. [20:29:26] Surprise! [20:29:33] i mean, you're in this channel so... [20:29:41] marktraceur: forward? [20:29:53] Maybe this is a card I'm on. Sec. [20:30:11] i may have done the @team thing [20:30:14] which... maybe not the best idea [20:30:24] i meant @tech [20:30:26] Ah [20:30:35] do you...want to be in the mingle project? [20:30:35] I don't know why I'd be on either. :P [20:30:38] i could also just take you out [20:30:41] That'd be super [20:31:06] ok.. i'll try to figure that out [20:31:18] aha! success [20:38:59] Jeff_Green: ... We have a branch of paymentswiki that's been upgraded to MW-1.23, and I'd like to test it... Is this something we could already do with payments1004? [20:39:49] (CR) Ejegg: [C: 2] Tests for WorldPay order_id generation [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/160089 (owner: Awight) [20:42:03] you should be able to, yeah [20:42:08] atgo: fyi, https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1947 needs more feedback from Megan [20:42:30] Jeff_Green: hrm, ok I'll give it a whirl. Is there a route to the webserver there? [20:43:01] awight: why not comment there and ask megan? :) [20:43:03] see how it goes [20:43:21] awight: you have to use ssh tunnels [20:43:27] ok cool [20:43:47] (CR) Ejegg: [C: 2] Clean up testhack [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/160072 (owner: Awight) [20:44:05] atgo: great thx, yeah I was mostly wondering who pulled the list of recurring contributions that "should" be in the ecom results [20:44:10] sounds like megan then [20:46:59] (Merged) jenkins-bot: Clean up testhack [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/160072 (owner: Awight) [20:47:01] (Merged) jenkins-bot: Tests for WorldPay order_id generation [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/160089 (owner: Awight) [20:48:03] awight: i'm going to delete this one, right? https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1928 [20:48:03] we have this info afaik [20:48:55] atgo: yep thx [20:49:48] ty atgo [20:52:29] awight: ejegg K4-713 I'm getting some strange CentralNotice admin breakage on my local install, after updating core. No tabs, just the "Special Page" tab and that's it. Has anyone else ever experienced that? [20:53:17] Not me. Then again, I haven't had a working local CN instance since 2011. [20:53:45] Hmm, yeah if it's a new regression I'll assume it's more recent than that [20:54:00] I think most of CN is currently more recent than that. :) [20:54:05] AndyRussG: maybe you aren't in infrastructure mode? [20:54:19] $wgNoticeInfrastructure = true; [20:54:21] is the default [20:54:33] but if u set to false, I'd imagine seeing the effect you're describing. [20:55:10] awight: No it says "$wgNoticeInfrastructure = true;" [20:55:22] I did muck about a bit with the config of this wiki tho [20:56:10] Installed cldr and EducationProgram, since the Vagrant install I was using for EP stuff crossed me [20:56:41] AndyRussG: what did you update mw-core to? [20:57:01] Jeff_Green: bug report... fr code update to a treeish that does not exist says: error: fatal: bad object ba7114586256ef0d3815ea7d42d342606758c772 [20:57:16] which is good--however, then it goes ahead and updates the code to unknown version. [20:58:02] 1.24wmf21 [20:58:05] awight: fugly [20:58:16] Lemme move to master.. [20:59:49] Jeff_Green: rats--is it possible to use this tool to deploy something that's not on the deployment branch? [20:59:56] I'm trying to load this: https://gerrit.wikimedia.org/r/#/c/160027/ [21:00:10] it's on branch fundraising/REL1_23 [21:00:54] AndyRussG: I haven't seen anything like that, I might be able to help debug, but the first thing to try is a fresh LocalSettings.php, just add the require_once ...Centralnotice.php and that's all [21:01:38] awight iirc it is always set to a specific branch [21:01:52] for testing you want to update the staging instance anyway though [21:01:58] which iirc is a different project [21:02:01] I am [21:02:07] thx for setting that up btw! [21:02:07] awight: Thanks! Yeah I was just wondering if it was a known previous issue. Looking into it now... :) [21:02:53] AndyRussG: I'm on 1.24wmf20 and it looks good [21:02:54] awight: yeah that's hardcoded to fundraising/REL1_22 [21:03:07] awight: K! thanks [21:03:08] Jeff_Green: even with the -pPROJ=REVLOCK syntax? [21:03:19] branch is branch [21:03:30] No mortuary today, only seafood, right? [21:03:32] Jeff_Green: hrm. Can you point staging to REL1_23 pls? [21:03:33] revlock is commit within that branch [21:03:43] AndyRussG: whoa [21:03:51] looney bin, then the brewery, aye? [21:04:25] what if we were to have staging vs deploy branches, rather than i.e. fundraising/REL1_22? [21:04:26] I mean no standup (it's called "autopsy" on the calendar), only sprint review("Shrimp Prawning") ? [21:04:43] And, regarding your question, yes [21:04:46] Jeff_Green: That sounds better to me [21:04:59] Jeff_Green: unfortunately, I hear that we're using the staging box for a production job (orphan slaying) [21:05:07] so... we'd have to fix that first [21:05:30] awight: I'm not sure what's going to happen when you run this after changing the branch [21:05:43] yah it will be a rocky ascent [21:05:44] should be amusing [21:06:02] rocky with angry goats [21:06:35] rocky angry goats in the seafood mortuary? [21:07:23] rocky racoon's angry goad? [21:07:44] hey... that guy is doing *my* job :p [21:08:10] * AndyRussG tries to wiggle out of the straightjacket [21:19:58] K4|meetings: awight MQL makes so much more sense now! [21:20:09] atgomez: omg that is not a good sign [21:20:14] hahahahah because sql! [21:24:45] awight: I think we need to add the Gift_Data custom group to one of the crm module install scripts [21:25:43] awight: ejegg: Not a CentralNotice or core regression. Just a silly interference between CentralNotice and EducationProgram. The latter returns false after processing SkinTemplateNavigation::SpecialPage because... wtf. [21:27:02] oh wow [21:27:12] ejegg: yeah... I thought I... [21:27:39] awight: oops, i probably need to run an update script [21:28:37] ejegg: I really think I did that, but don't see it in the code :-/ [21:28:43] ejegg: yeah, "drush updatedb" [21:29:10] thanks, will try that [21:29:43] yeah, I don't see 'Gift_Data' added any place in the code either [21:31:42] ejegg: I may have thrown out my changes... sad sauce. [21:32:02] git reflog? [22:02:00] awight: ejegg having some isssues [22:02:08] Should be there in a sec [22:05:06] AndyRussG: you joining? [22:05:23] atgo: yes having problems here with my hangout [22:05:34] yeah we are too [22:06:57] clear browsing data and try again! worked for us [22:07:18] Anyone wanna try Jitsi? its loads better