[00:02:20] Hmmmmmm. [00:03:21] HMMMM. [00:03:32] I think I know why orphan slaying stopped talking to anything. [00:08:52] doesn't sound like a very sociable profession [16:25:28] * K4-713 waves [16:25:33] Good morning, ejegg. [16:25:33] Hi K4-713 1 [16:25:35] ! [16:25:38] Good morning, AndyRussG. [16:26:06] Good morning, everybody! Ah, before I get too far in to anything... [16:26:25] There are a couple hot items that I want to make sure are being kicked along. [16:27:41] The biggest and baddest one is the loss of banner impressions. [16:28:00] AndyRussG, I think you were looking at this? [16:28:20] Ah hi K4-713 [16:28:33] Sorry I was getting other pings and didn't see this your first one 8p [16:28:39] No worries. [16:28:40] Yeah didja see the card? [16:28:55] I saw *a* card. [16:28:57] * K4-713 looks [16:29:17] 2066? [16:29:43] Yep! [16:30:25] Okay. You should move it over to "In Development". :) [16:30:30] I don't have a way to test or benchmark that proposal [16:30:31] Ah OK [16:31:08] Hmm. Is this still log filters written in C? [16:31:17] Yes [16:32:02] I assume that's for speed [16:32:04] Last time I had to touch this stuff, somebody was nice enough to hand over some raw logs to parse. And, with a little jiggling, it would be made to work locally. [16:32:11] *could [16:32:14] Right [16:32:57] I did install compile seek out dependencies etc but so I don't know the exact format of the real logs [16:32:58] I guess I should ask if you actually know C. Heh. [16:33:12] Uhh well enough to read the code and get an idea of what's going on [16:33:23] Aha, okay. [16:33:40] Yeah, sounds like the next step is getting some raw logs to test against. [16:33:44] Not well enough to really write much of anything, it's been a really long time... [16:34:11] I seem to remember writing a small wrapper locally, that output both the final counts that got through the filter, and the time it took. [16:34:21] ...now that you mention it. [16:34:31] I don't think it got into a repo or, you know, anything useful. [16:34:48] Ah hmm! I was thinking of something like that [16:35:02] But it was a pretty quick thing to setup. Then again, I learned how to code with C / C++. [16:35:47] Hurm. I'm in meetings until... 2pm or something, and I'd like to make sure you have everything you need before I disappear. [16:35:57] If someone could pass me some redacted logs, even a small snippet, I could try some more [16:36:21] I cannot for the life of me remember who got them to me the last time. [16:36:42] ...but I'd probably start with somebody in analytics. I mean, that seems... reasonable. [16:36:48] To me. [16:37:50] At the very least, they should have a better idea of who we should really be asking. [16:38:16] Ah hmmm [16:38:19] OK [16:39:04] You know what would probably be the best way to do this? [16:39:24] What? (eyes glued to IRC window...) [16:39:28] Hi. I asked this question on Sunday as well but no one was around. Isn't the hideBanner cookie supposed to work globally? [16:39:45] Glaisher: Yes. Why do you ask? [16:39:50] When one clicks on the close button at meta for a banner, it should hide the banner at other wikis as well. [16:39:55] It's not working. [16:40:33] Glaisher: Uh hmmm... [16:41:02] AndyRussG: ...When you're testing the filter, write the output of each run to another file, of just the impressions we think we care about. Run it once unaltered to get the output as we would have it now. Then you can diff the two outputs to see if we picked up things we want, or lost things we want, or what. When the counts are different. Because they probably will be. [16:41:08] Make sense? [16:41:33] Glaisher: OK thanks, I can look into taht too... Before you go, do you have more details about which browser/version/OS and what wiki other than meta? Thanks again :) [16:42:15] K4-713: OK, yes [16:42:50] AndyRussG: Let's see whether you can reproduce it. For example, you go to https://en.wiktionary.org and then go to https://de.wikiquote.org/wiki/Hauptseite [16:42:52] OK I'll ask on the analytics channel for redacted logs then... [16:43:05] after closing the banner at enwikt [16:43:31] or some other random wiki [16:44:42] Glaisher: well, I'm not getting any banners on enwiki, but that's probably because of geolocation [16:44:46] Glaisher: Are you getting fundraising banners right now? And if so, where are you? [16:44:48] ...ha. [16:44:53] FDC [16:44:59] not fundraising [16:45:03] OH. [16:45:23] That... is more than a bit different. [16:45:33] I'm not sure if they have the omni-hide code in place. [16:46:09] FDC? [16:46:22] You haven't seen it? [16:46:25] It's everywhere. [16:46:31] http://meta.wikimedia.org/wiki/Special:CentralNoticeBanners/edit/FDCpropreviewOct2014 [16:46:58] I haven't. And I'm not getting it when I go anywhere, either. [16:47:19] Oh, wait, this is probably running for logged-in or something. [16:47:25] That's weird. The campaing is enabled on all projects. [16:47:32] Yeah. [16:47:34] Logged in only [16:47:37] Yes logged in [16:48:07] OK lemme try [16:49:28] Glaisher: I got the banner logged in on English Wikipedia, closed it, then went to German Wikipedia, and didn't get it [16:49:46] Could you try some other project? [16:49:50] not Wikipedia [16:52:38] Glaisher: sure [16:53:30] Well, if this turns out to be an FDC banner problem, we should probably poke whoever set up their stuff. [16:53:57] It's using the default thing. [16:54:07] [16:54:28] the-wub: ^^ : Apparently not-fundraising banners are not hiding across all projects when closed. Can you check ours? [16:55:47] K4-713: on it [16:55:53] Thanks! [16:56:06] And, now I go. Good luck, everybody. :/ [16:56:57] Most probably it doesn't work on different projects. It doesn't appear on other language wikis once you close it on the same project. [16:59:07] (PS1) Ejegg: Log uncaught exceptions to syslog [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/168105 [17:00:29] fundraising ones are working as intended. I think it's always been the case that the hide cookies don't work over mulitiple projects, since we need to specify a domain for them [17:00:36] Glaisher: the-wub: it worked for me, going to Meta logged-in, closing the banner, then going to Enwiki logged in [17:02:49] the cookie is set for *.wikipedia.org (or whatever domain you happened to be on when you closed it) [17:03:05] so it will work across languages, but not across projects [17:03:22] the-wub: Glaisher I think it may stop work if your browser is configured to accept third-party cookies [17:04:02] the-wub: Yeah, that's probably it. [17:06:00] the-wub: Glaisher Just going to meta sets the third-party cookie on wikipedia.org in my setup [17:59:26] hey tech! i'll be at standup in 1 minute :) [17:59:30] it's been a while [18:00:30] atgo: OK one sec! thx :) [18:38:25] Anyone with +2 in integration/config want to look at this patch to turn of jslint voting in dash so we can get some stuff reviewed before we go back and do all the style fixes? https://gerrit.wikimedia.org/r/168127 [18:48:19] (PS1) Ejegg: Import 'url' module correctly, move exception logger [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/168131 [18:59:11] (CR) Ssmith: [C: 2] Import 'url' module correctly, move exception logger [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/168131 (owner: Ejegg) [18:59:16] (Merged) jenkins-bot: Import 'url' module correctly, move exception logger [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/168131 (owner: Ejegg) [19:03:36] (PS1) Ejegg: Merge remote-tracking branch 'gerrit/master' into deployment [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/168135 [19:03:56] (CR) Ejegg: [C: 2] "+2 for deploy" [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/168135 (owner: Ejegg) [20:12:18] AndyRussG|brb: hey, if you're around? [20:12:44] This would be a great time to talk about impressions, and CN cookie fu [20:13:56] ejegg: anything u want to chat about? :) [20:14:14] hi awight ! [20:14:21] hey [20:14:27] AndyRussG|brb seemed to be running with that ball [20:14:28] I have a kid, and conference room [20:14:47] aww, quit braggin! [20:14:54] hehe [20:14:59] awight: heheh [20:15:10] I have 2 kids an 0 conferences rooms [20:15:21] i have... um, newfound nginx knowledge? [20:15:28] lol I have no hands [20:15:31] awight: did you see https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/2066 ? [20:15:59] mind video chat? [20:16:19] I'm unclear on a few things... [20:16:43] Like, what are bogus impressions? [20:18:03] awight: sure one sec [20:20:11] awight: Hangoutable NEtime [20:21:20] Hmm no video [20:23:05] I heard it ring... [20:23:06] awight: https://plus.google.com/hangouts/_/g3f2sinckz2o65dqal6jwns454a?authuser=0&hl=en [20:23:07] one more try? [20:23:08] ok [21:26:27] * AndyRussG waves at K4-713 [21:26:45] Hey AndyRussG. How's it going over there? [21:27:00] not bad... how was your trek across meetingdom? [21:27:54] Not too bad. [21:27:58] Cool [21:28:44] How's ImpressionQuest going? [21:28:50] ejegg and I talked to Adam about the udp-filter and impressions. So we're just going to open up the filter to send all impresson logs to the DjangoBannerStats [21:29:03] and then hack anything that needs hacking in DjangoBannerStats [21:29:48] Adam's concern was not that the script would choke on bogus hits on Special:RecordImpression, but that it would choke on the new information being sent in [21:30:53] I think that I don't meet the requirements for working with unredacted logs but ejegg will send some redacted snippets once we start getting decent data through udp-filter [21:31:41] Sadly, I didn't approach Jeff_Green about the udp filter update till a minute before he had to take off [21:32:05] and he didn't want to rush the change through [21:33:21] Ah hmmm [21:34:24] I'll sending the change thru Gerrit NEway, make sense? [21:34:40] Hmmm that's a big repository [21:34:45] Oh, I already put it up for review [21:35:13] https://gerrit.wikimedia.org/r/168186 [21:37:49] Ah cool... I was just gonna comment on the deep architectural implications and then +2, but I don't even have +2 there... [21:37:56] thanks btw! [21:38:42] yw [21:39:24] Sorry, this day is being really distracting. [21:39:39] Also: If anybody said anything about needing a "fundraising NDA", that's not a thing anymore. [21:41:10] K4-713: Ah OK! I actually checked my contract and it does have a section about confidentail information bla bla bla [21:42:19] I was getting ready to send an e-mail to you and some other folks asking about that [21:42:55] Yeah, it's been a long time since we didn't just get that from everybody. [21:43:05] So do you know I can receive unredacted snippets for testing stuff...? [21:48:55] AndyRussG: Ah, in what sense? [21:49:22] Like, the actual file transmission, or who has them to begin with, or...? [21:49:31] Well, it'd be nice to get some data to test any changes to BannerDjangoStats on [21:49:50] ejegg: K4-713 I'll ask at #wm-operations if anyone else can push the filter change out? [21:50:19] AndyRussG: Jeff was worried about the storage implications [21:50:50] ejegg: Ah I see [21:51:40] How does that file get deleted or rotated or whatever, anyway? [21:51:58] My entire self just shrugged. [21:52:25] As far as I know, it's all magical server elves. [21:52:45] lol spoken like a true programmer [21:52:48] Some make cookies, some fix shoes, some rotate your logs. [21:53:14] wait, there's progammers that make cookies??? [21:53:24] * K4-713 looks innocent [21:53:25] no, the programmers only write the recipe [21:53:36] haha [21:53:57] and argue about the size of the chocolate chips [21:55:11] So i guess we'll heed Jeff's concerns and not ask anyone else to push anything out or anything [21:56:01] no? [21:56:14] AndyRussG: that's what I was thinking [21:56:43] K got it [22:07:03] AndyRussG, ejegg: ...dang. This place is maximum distracting today. [22:07:16] So, can I get the summary of what happens next? [22:07:30] I'll, ah... attempt to stick around for the answer this time. :/ [22:09:07] Tomorrow morning, we swap the filter to get all the RecordImpression results [22:09:10] K4-713: what happens next is: we wait for jeff's return and wisdom to deploy the change to the udp-filter [22:09:19] ^ what ejegg said [22:09:38] ejegg: you stopped the Jenkins job that runs DjangoBannerStats, right? [22:09:43] Ah, okay. I'm with you now. [22:09:43] Once a couple of files are there, I can scrub IPs, referrers, useragents, and send to AndyRussG [22:09:48] yeah, I stopped the job [22:09:55] K cooool [22:10:14] ejegg: You actually shouldn't have to scrub. The NDA these days is supposed to be the same for everybody. [22:10:24] ah, cool [22:10:36] encryption though? [22:10:54] That is definitely a thing. [22:11:01] Hmmm [22:11:02] I mean, is it cool to send those to sit on a non-encrypted drive? [22:11:07] Though, it's primarily my thing, I think. [22:11:12] that's why I was going to scrub anyway [22:11:19] In that case: scrub away. [22:11:25] :) thx [22:27:58] Oh hey: AndyRussG. What happened with the banner hide thing this morning? [22:28:02] Did that get sorted out? [22:30:54] K4-713: you mean the bucket thing? [22:31:22] No, the thing where clicking "hide" on the FDC banners wasn't hiding across all projects. [22:31:44] It was unclear when I left if that was a CN thing, or if it was particular yo their banner, or what. [22:31:50] *to their banner [22:32:36] Ah yeah it was confirmed that it's not a bug [22:32:47] :) [22:32:50] That's what I thought. [22:33:06] The person reporting the issue was disabling third party cookies [22:33:23] Oh. Ha. [22:33:52] So meta.mediawiki.org couldn't set a cookie for wikipedia.org [22:33:57] yep [22:34:04] Couldn't get the cookie swarm. [22:34:18] And quite a swarm it is! [22:34:22] Seriously. [22:34:52] I'm pretty sure I had third-parties disabled, when I enabled them it was like chocolate chip overload [22:40:24] AndyRussG: hey, I'm around for an hour or two, if I can help with anything? [22:40:43] Hey awight cool thx [22:41:08] I could start picking apart the impressions loader? Or are u already in there? [22:41:43] Just getting to it after a swarm of non-work distrctions 8p [22:46:41] ejegg: anything I can do to help with dash deployment? CR? [22:48:37] feel free to jump in on the impressions loader too of course [22:50:43] awight: thanks! Not sure if there is anything now. [22:51:09] Just trying to figure out how nginx is mutating the oauth token request on the way from node to drupal [22:51:34] I feel all brainscattered today... there's also bucket stuff.. [22:51:37] once it hits drupal, something is different enough that the signature doesn't match [22:51:46] ejegg: awww man [22:51:48] Just researching exactly what is being signed [22:52:04] ejegg: I remember that, now that you mention it... 513-character query or something. [22:52:17] could reproduce locally, and it was just the host header that I needed to fix in my nginx config [22:52:44] but Jeff got that header sorted and there's still something off [22:52:45] AndyRussG: let me know if you can carve a small piece for me. Shall I chase down udp-filter stuff, or that's done already? [22:53:11] awight: got a filter patch in review, Jeff will check it out in the morning [22:53:20] awight: patch is here if you have +2 ... https://gerrit.wikimedia.org/r/168186 [22:53:51] I'd say leave it for Jeff - he wanted to be around for its debut to watch the storage [22:53:58] sure! [22:54:33] Oh, we might as well zap that hideBanner filter too. Don't need that now that everybody's got the new cookie JS [22:54:36] Ah OK sorry ejegg -- I didn't know merging it would actually deploy it! [22:54:56] AndyRussG: I dunno either, but I figured the elves might do just that [22:55:21] ppfff darn elve [22:55:23] elves [22:55:26] yep, sure seems to be on the "production" branch [22:55:57] Oh right! [22:56:52] ejegg: http://oauth.net/core/1.0/#rfc.section.9.1 [22:57:14] sounds like the entire request is used as hash input [22:57:47] "Parameters are sorted by name" fwiw [22:57:59] yeah, sorting should be cool [22:58:19] u have example queries? This is all GET, hopefully? [22:58:28] huh, only header that should matter would be Host [22:58:31] nope, all POSTs [22:58:39] oof. [22:59:06] and the apache access log on the civi server isn't logging host header [22:59:25] So, hoping for a packet capture or something when Jeff is available again [22:59:38] ejegg: u read this? http://stackoverflow.com/questions/9709836/nginx-loadbalancer-and-oauth [23:00:13] this is promising, The problem was in the 'host' header. For OAuth to work correctly, it needs to include the port also. This did the trick: proxy_set_header Host $host:$server_port; – ddario Mar 14 '12 at 21:02 [23:00:32] ooh you said that already. [23:00:42] yeah, we're using the $http_host var [23:00:49] which has both server and port [23:01:11] Also setting proxy_http_version to 1.1 [23:01:13] and the consumer signs using the proxied port? [23:01:26] yeah [23:01:44] so the host header drupal sees won't actually match the port it's serving on [23:01:56] https://www.drupal.org/node/1264390 [23:02:59] https://www.drupal.org/node/1264390#comment-5794372 [23:03:26] hmm, wonder if prod has a different version of the OAuth library? [23:03:55] It's not having that problem on my setup, with nginx forwarding from 9898 to 80 [23:03:59] also, the Drupal ?q= rewrite might be an obstacle [23:04:06] grr [23:04:27] barf: https://www.drupal.org/files/oauth-q_query_string_ignore-1264390-11-D6.patch [23:04:30] got the same rewrite locally and it's fine [23:04:53] ejegg: are you setting $base_url locally? [23:05:02] hmm, let me check [23:05:14] "As far as I can tell, the code in the OAuth library is wrong, it will break on a reverse proxy or if your servers hostname isn't set to the public hostname. See:" [23:06:12] We are using the public hostname though [23:06:32] just doing the node equivalent of adding an /etc/hosts file entry to redirect it to a different IP [23:06:32] Ok [23:07:20] "The described issue seems to be fixed in 7.x-3.x-dev as pointed out in comments #20, #22, #24 [23:07:30] of services_oauth [23:07:48] yah, and i'm running the same modules locally as in prod [23:07:50] can you log the input to the signing routines? [23:08:19] Yeah, guess I could mess with that lib a bit just for diagnostics [23:12:50] ejegg: awight: how often did you say the banner loader is run? [23:13:25] AndyRussG: every 15 min [23:13:27] my charge is up... [23:13:50] thanks! [23:33:10] ejegg: You still around? [23:33:18] K4-713: yep [23:33:23] what's up? [23:33:46] Have you had any more time to poke the dash stuff today? [23:33:57] I think everything happened at once. [23:34:05] ...a little hard to sort out. [23:34:42] Yeah, it's all set and deployed, but the token request from node to drupal is being denied for an invalid signature [23:35:02] Did you find where the logs were going? [23:35:03] We almost fixed it with nginx config [23:35:27] Oh yeah, the logs are going to /var/log/syslog [23:35:34] Okay, good. :) [23:35:45] now that I moved the exception logging up past the first place it was crashing [23:36:01] I think I just found the other problem with the signature [23:36:07] haha [23:36:10] Oh really? [23:36:11] and why I couldn't reproduce locally [23:36:13] Exciting. [23:36:56] Yeah, it's the php oauth lib assembling an http url since nginx is doing the ssl [23:37:16] and checking the sig against that, when node was signing something including an https url [23:37:54] so if we can set an x_forwarded_proto header in nginx, a little tweak to the oauth lib will let us get that right [23:39:52] Aaah. [23:40:52] i just can't test https locally cause node's oauth complains about my self-signed cert [23:41:20] That's annoying. But, we can work with it. [23:49:43] If the banner loader runs every 15 minutes, why does it look at files from the past 2 hours (apparently)? https://git.wikimedia.org/blob/wikimedia%2Ffundraising%2Ftools%2FDjangoBannerStats/cf5a875d49f4c4cf229d7f864a73d4c2f588ebf9/fundraiser%2Fanalytics%2Fmanagement%2Fcommands%2FLoadBannerImpressions2Aggregate.py#L84 [23:50:34] That question implies that it was built using logic. [23:50:50] I'm not sure that's accurate. [23:50:54] ;) [23:50:57] I guess in case the jobs are shut down for a little while [23:51:15] Yeah, or maybe if there was a long hiccup in the file rotation? [23:51:24] It keeps track of which files it's already processed, so it doesn't duplicate stuff [23:51:37] Yeah. I mean, it's clearly not overcounting. [23:52:16] Everything we have should know how to handle an apparent re-process.