[04:55:53] Fundraising Sprint X-Ray Spex, Wikimedia-Fundraising-CiviCRM, Unplanned-Sprint-Work: Saving a donation is very slow for Engage - https://phabricator.wikimedia.org/T118415#1800297 (awight) Agreed that this is urgent to fix. Thanks for filing the bug! [05:40:43] Fundraising Sprint X-Ray Spex, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Unplanned-Sprint-Work, FR-GlobalCollect: Determine what was breaking GlobalCollect in paymentswiki 3730cff20 - https://phabricator.wikimedia.org/T118349#1800331 (awight) Just noting that the offending ch... [08:28:33] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Improve date range search fields - https://phabricator.wikimedia.org/T116443#1800482 (awight) p:Normal>Low Lowering the priority, unless anyone objects? [13:54:36] Fundraising Sprint X-Ray Spex, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Unplanned-Sprint-Work, FR-GlobalCollect: Determine what was breaking GlobalCollect in paymentswiki 3730cff20 - https://phabricator.wikimedia.org/T118349#1800889 (MBeat33) It looks like all the affected d... [16:24:15] boo! [16:36:01] * awight leaps through the ceiling [16:36:04] hi! [16:37:58] http://www.cs.yale.edu/homes/perlis-alan/quotes.html [16:39:22] O_o [16:48:54] Heh [16:56:40] (PS1) Awight: WIP some things to READ [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/252709 [16:58:39] Fundraising-Backlog: Recent iframe errors from up-to-date browsers - https://phabricator.wikimedia.org/T112181#1801229 (MBeat33) The donor using Chrome in Zendesk # 166118 has confirmed that LastPass prevents the selection of credit card type. They were able to proceed after disabling LastPass, and we will a... [17:05:04] AndyRussG: and XenoRyet I just sent you a meeting invite [18:50:36] * awight is beating head against DjangoBannerStats installation [18:53:00] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Deadlock should result in requeueing the message - https://phabricator.wikimedia.org/T118487#1801561 (awight) NEW [18:57:03] awight: cwd: XenoRyet: when you have a second: https://gerrit.wikimedia.org/r/#/c/252118/ :) thx! [18:57:22] ^ I was thinking of booking a slot on this evening's SWAT deploy... [19:01:40] (CR) Awight: [C: 2] "That'll work! Please update the hide code enum as well." (3 comments) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/252118 (owner: AndyRussG) [19:02:14] the-wub: Jeff_Green: I'm ready to start the kafkatee deployment... [19:02:22] hey awight [19:02:24] the-wub: Mind if I take campaigns down? [19:02:25] hi! [19:02:54] (PS2) Awight: WIP some things to READ [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/252709 [19:03:05] lemme do one last pre-fliparoo rsync, should take just a couple minutes [19:03:06] awight: sure, be my guest! [19:03:08] Jeff_Green: I won't lie, I was unable to run the banner impressions loader locally [19:03:11] ok cool [19:03:22] awight whaddya mean? the change is untested? [19:03:42] (Merged) jenkins-bot: impressionDiet: don't restart cycle when restartCycleDelay is 0 [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/252118 (owner: AndyRussG) [19:04:03] mmm [19:04:07] yeah [19:04:46] awight: can you take the WMDE campaigns down at the same time please [19:04:51] awight: which didn't work? the funky python DB named after someone? [19:04:54] Assuming the file format is actually exactly the same, only one thing actually changed, and that's the sample rate code. [19:04:57] ;) [19:04:58] the-wub: sure! [19:04:59] I confirmed via email with kai that it's okay [19:05:18] awight: I did test the files before the sample rate change [19:06:21] Jeff_Green: we're not deploying any new code--the code path I'm concerned about is already deployed and is being executed. [19:06:32] awight ok [19:06:42] The thing I wanted to test but couldn't was that changing the sample rate would be reflected correctly in the database. [19:07:05] Should be immediately obvious, so I'm personally mostly okay with deploying it live [19:07:14] right [19:07:16] err s/live/haphazardly/ [19:11:00] !log Disabled WMDE and WMF Fundraising campaigns to do banner impression maintenance [19:11:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:11:21] awight let me know when it's consumed all the logs in the new location [19:12:05] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Patch-For-Review: Summary of campaign settings changes is not logged in CentralNotice logs when modified using Special:CentralNotice main page table - https://phabricator.wikimedia.org/T110315#1801602 (awight) Resolved>Open Reopening, I ju... [19:12:11] Jeff_Green: will do [19:13:00] * awight grimaces at the fact that we're still using a "2014" directory [19:13:12] we shouldn't be? [19:13:33] hehe I think our strategy is still on a 2010 wiki [19:13:42] heh, much of donate wiki is still in "FR2011" templates [19:13:48] totally [19:13:49] oh we're not talking about logs [19:13:54] Jeff_Green: I was [19:13:59] then got giddy [19:14:23] just to confirm, the parser is aimed at /archive/banner_logs/2015/ now yeah? [19:14:34] wat. Okay the config file I'm looking at says /archive/udplogs/2014/ but that must not be the case [19:14:39] SUX [19:14:46] I can't wait for this tool to die. [19:14:58] incoming logs should be going to 2015/ [19:15:08] Jeff_Green: yeah, confirmed that the script is somehow using the correct path [19:15:22] how. [19:15:29] maybe there's a s/2014/$currrent_year/g somewhere ;-P [19:15:47] warning: I have to fetch my kids in about 30 min, it will take me about 15 min, then back [19:16:24] Jeff_Green: wait, correction. the path we use is /archive/udplogs/2015 not banner_logs [19:16:31] Still not seeing the config, though [19:16:51] yeah, that makes sense for the old path [19:16:56] ok [19:17:11] * awight starts flipping tables, looking for the real config [19:18:04] I can't find it. cannot find it. [19:18:06] aaauuugh [19:18:32] is it stored in db? [19:19:04] * awight wtf's all over the place [19:19:10] perhaps [19:19:34] or a jenkins job param? [19:20:01] looked there [19:20:19] AndyRussG: did you run into this gotcha when you were hakking on the DjangoBannerStats mess? [19:20:50] AndyRussG: It turns out, /etc/fundraising/django_settings.py UDP_LOG_PATH is silently ignored [19:20:54] * AndyRussG reads backscroll [19:21:05] something else is providing the correct path and I'll need to change that [19:21:38] omg, found it [19:21:41] evile [19:22:33] awight: I don't recall any issues around that [19:23:47] Jeff_Green: okay, we've consumed the last files, up to /archive/udplogs/2015/beaconImpressions-sampled100.tsv-20151112-191502.log.gz [19:23:52] I'll disable the job now. [19:24:01] awight: hold a sec [19:24:02] !log banner impressions job disabled [19:24:03] k [19:24:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:24:31] actually, go ahead and disable the job [19:24:47] done and done :D [19:26:49] * awight waves at meganhernandez [19:26:53] awight is there a filename glob used for matching input files? [19:26:58] FYI, we took campaigns down at 11:11 [19:27:04] Jeff_Green: there is [19:27:11] lmk if we need to change that. [19:27:13] i mean...i want to drop empty marker files right at the end of the udp2log collection [19:27:34] can I put a file matching the first part of the name but i.e. with some irrelevant extension [19:27:39] mmm that sounds fine [19:27:55] i will name them .marker.biz :-) [19:28:06] Jeff_Green: here's the regex, I shat you not: [19:28:07] ./fundraiser/analytics/management/commands/LoadBannerImpressions2Aggregate.py: now = "beaconImpressions-sampled*.tsv-%s*" % time_now.strftime("%Y%m%d-%H") [19:28:14] s/regex/glob/ [19:28:32] buh. there goes that idea. ok nm. [19:28:48] right--ends in a stupid wildcard [19:29:02] beaconImpressions-MARK-sampled would be fine [19:29:05] (PS1) AndyRussG: impressionDiet: comment and hide reason code for when cycle doesn't restart [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/252757 [19:29:26] i wanted to include the date [19:29:33] no big whoop [19:29:43] (CR) Awight: [C: 2] "Thanks!" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/252757 (owner: AndyRussG) [19:30:03] beaconImpressions-MARK-sampled.tsv-crazydate.log.gz [19:30:12] brb [19:31:11] (PS3) Awight: WIP some things to READ [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/252709 [19:31:14] (Merged) jenkins-bot: impressionDiet: comment and hide reason code for when cycle doesn't restart [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/252757 (owner: AndyRussG) [19:31:43] (CR) AndyRussG: impressionDiet: don't restart cycle when restartCycleDelay is 0 (3 comments) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/252118 (owner: AndyRussG) [19:31:53] (PS1) Awight: Don't override log dir config [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/252761 (https://phabricator.wikimedia.org/T97676) [19:32:02] Urgent review if possible ^ [19:33:22] AndyRussG: ^ [19:33:47] Jeff_Green: Remind me how we're changing the /etc/fundraising file? [19:33:55] I prepare a patch for you? [19:34:17] sure, or I can do the change. whichever [19:34:26] k [19:34:30] ok I've got our udp2log endpoints: [19:34:46] awight: I can +2 but if you want any serious review it'd take longer, since my brain is not at all in that codespace :( LMK! [19:34:48] beaconImpressions-sampled100.tsv-20151112-191502.log.gz 2015-11-12T19:15:01 [19:34:55] AndyRussG: nah I just need a rubber stamp ;) [19:35:00] landingpages.tsv-20151112-191502.log.gz 2015-11-12T19:15:01 [19:35:33] (CR) AndyRussG: [C: 2] "Inspected #57!" [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/252761 (https://phabricator.wikimedia.org/T97676) (owner: Awight) [19:35:45] Jeff_Green: confirmed that those have been processed. [19:35:47] AndyRussG: hehehe [19:36:08] (CR) Awight: [V: 2] Don't override log dir config [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/252761 (https://phabricator.wikimedia.org/T97676) (owner: Awight) [19:37:28] !log Update DjangoBannerStats from 12e819a04a40ee6fab5dd55fcaf072661df31106 to 9cc2acba7f6b46e243ada0fac6c48c9a49460a07 [19:37:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:43:29] Fundraising Sprint William Shatner, Fundraising Sprint X-Ray Spex, Fundraising Tech Backlog, Fundraising-Backlog, and 4 others: Slow searches: Transaction by ID # using % and also Contribs by email address - https://phabricator.wikimedia.org/T117492#1801692 (MBeat33) Searches are still slow enough... [19:49:57] awight: ok please see barium:/archive/banner_logs/2015/20151112-udp2log-kafkatee-cutover.README [19:50:04] that documents the splice itself [19:50:39] now one thing I'm puzzled about is the log file size, the new files are about the same size as the old ones, and I don't understand why that would be assuming the 100->10 sampling rate change [19:51:08] scareee [19:51:42] Fundraising Sprint William Shatner, Fundraising Sprint X-Ray Spex, Fundraising Tech Backlog, Fundraising-Backlog, and 4 others: Slow searches: Transaction by ID # using % and also Contribs by email address - https://phabricator.wikimedia.org/T117492#1801706 (Eileenmcnaughton) Thanks Michael - that... [19:51:52] Jeff_Green: what directory contains the new things? [19:52:00] they're in the same dir [19:52:19] in theory you should be able to start the processor and we're done, but we need to figure out the filesize thing and make sure it's all sane [19:52:53] i need to get my kids, back in 10-15 [19:52:58] (PS1) AndyRussG: ext.centralNotice.display: Fix class of CN div [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/252768 (https://phabricator.wikimedia.org/T118348) [19:53:06] awight: don't start the processor until I get back pls [19:53:31] Jeff_Green: Agreed that the sample rate doesn't look right [19:53:40] I see the same number of entries per second. [19:54:50] I'm gonna eat a thing, brb. [20:07:05] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Patch-For-Review, Regression: CentralNotice setting class=cn-undefined - https://phabricator.wikimedia.org/T118348#1801760 (AndyRussG) Thanks @Krinkle and @awight! Yea fortunately it's nothing grave ;) [20:09:06] awight back, looking into the sampling rate [20:09:09] ok awesome [20:09:22] so.... [20:09:24] whut. [20:09:55] lemme ping Mr. Analytics. [20:21:59] meganhernandez: the-wub: Looks like we might overrun our maintenance window :( [20:22:41] awight meganhernandez I'm talking with ottomata now, we're looking at what's up with the sampling rate [20:23:00] Yes, no rush! I'm just making sure people know we'll need more time. [20:23:20] yup yup [20:23:21] My fault for scheduling it so tight... [20:24:26] (CR) Awight: [C: 2] ext.centralNotice.display: Fix class of CN div [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/252768 (https://phabricator.wikimedia.org/T118348) (owner: AndyRussG) [20:24:48] awight: thx once more! [20:25:46] (Merged) jenkins-bot: ext.centralNotice.display: Fix class of CN div [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/252768 (https://phabricator.wikimedia.org/T118348) (owner: AndyRussG) [20:27:34] Fundraising-Backlog: Add field data back into Silverpop export - https://phabricator.wikimedia.org/T105254#1801849 (CCogdill_WMF) [20:31:07] awight: we're both confused and it's going to take some time to figure it out, if we're holding people up should we reschedule? [20:31:40] awight or...how long is it ok to hold people up? [20:33:40] the-wub: meganhernandez: ^? [20:33:51] Jeff_Green: how about we deploy with 1:100 for today [20:34:03] Then, at least we've gotten over the kafkatee hurdle [20:34:53] awight I would say yes except I'm not 100% confident what we're looking at is 1:100. it looks right, but that contradicts the configured rate and there's no explanation why yet [20:35:18] ouch [20:36:02] the-wub: meganhernandez: If the options are to drag this maintenance out for a couple more hours, or do it again next week, which would be better for you (and Kai)? [20:36:47] hi awight i have not been following irc so i’m not sure where things are at, but i’d prefer to get through it today if we can. [20:37:00] Jeff_Green: does that work for you? [20:37:22] awight yep [20:37:52] meganhernandez: Okay, we'll go ahead with it today, then! Thanks for the sanity check... [20:37:54] awight: hmm... not sure about WMDE. I had said we should get it done in less than 2 hours, but now is probably not a bad time since it's night in Germany [20:38:10] the-wub: good point. okay I feel less guilty about that. [20:38:39] or well, do you need me or peter for anything? it’s late so not sure how much longer we’ll be on tonight [20:39:09] meganhernandez: nope, I think we can do this ourselves. I'll put the campaigns back up and send an email when we're finished. [20:39:18] the-wub: wanna send me Kai's email? [20:39:33] sounds good to me awight [20:39:56] i agree, it’s prob better for germany to just wrap it up today. they’re not doing much fundraising this late at night anyway [20:40:21] that works for us [20:40:46] awight: I'll add you in on the thread with them [20:41:10] thanks! [20:57:12] Fundraising Sprint X-Ray Spex, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Unplanned-Sprint-Work: Saving a donation is very slow for Engage - https://phabricator.wikimedia.org/T118415#1801938 (DStrine) [21:01:39] Fundraising Sprint William Shatner, Fundraising-Backlog, Patch-For-Review: Dedupe data in Silverpop file - https://phabricator.wikimedia.org/T107045#1801959 (atgo) a:Ejegg>None [21:27:58] Jeff_Green: I just had a terrible thought--banner impressions would have gone down dramatically cos I took campaigns down. [21:28:01] Checking on that now. [21:28:13] I'll look at numbers only for a campaign I didn't touch. [21:28:49] awight: i wondered about that [21:31:09] Jeff_Green: OMG. the sample rate is correct. [21:31:13] So sorry about that. [21:31:33] I just did "zgrep -c WLAfrica" on both log streams, it's definitely 10x higher now. [21:31:38] it's correct now? [21:31:40] yep [21:31:55] Actually, please do "zgrep -c WLAfrica beaconImpressions-sampled10.tsv-2015111 [21:31:59] 2-*.log.gz" [21:32:15] lmk if anything was adjusted before those last two files were written [21:32:27] There's a steep drop at 20151112-211501 [21:32:57] Looks suspicious like the config was jiggled back to 1:100 [21:33:05] *ly [21:34:59] awight can we put up a tiny campaign of some kind? [21:35:22] something that will get us some banner traffic that we don't care about too much? [21:35:25] uuh [21:35:36] There are plenty of campaigns still up, for example *WLAfrica* [21:35:44] oh, never mind then! [21:35:47] (sorry, meant to be wildcards, not bold) [21:36:55] AndyRussG: fwiw, there is an awful lot of traffic going to campaigns I've disabled... [21:37:04] hi all! [21:37:41] ottomata: hi, thanks for the help! [21:37:52] I was just confessing to Jeff_Green, I think I started a goose chase here. [21:38:11] I thought I started it :-P [21:38:25] We were counting loglines per second and stuff, and failed to account for the fact that I had turned off most of the CentralNotice campaigns at the same time as the cutover. [21:38:29] hehe [21:38:39] what is a synonym for "d'oh"? [21:38:43] I'm going to rsync more logs back into the dir from the udp2log pipeline [21:38:52] great [21:39:01] (then we delete later?) [21:39:04] i removed all the kafkatee logs to now [21:40:43] I don't get the plan, I'll just follow your lead [21:41:34] hey, not totally sure what is happening, but, i am comparing a few things [21:41:54] running beaconImpressinos filter with kafkatee on stat1002 [21:41:58] both sample 10 and sample 100 [21:42:06] awight: I'm catching barium:/srv/banner_logs/2015 up with udp2log collected stuff up to now [21:42:08] everything looks good there, bout 10x difference in file size [21:42:28] Here's a data point: [21:42:29] zgrep -c WLAfrica beaconImpressions-sampled100.tsv-20151112-213001.log.gz [21:42:32] 32 [21:43:01] Jeff_Green: s/srv/archive/, right? [21:43:16] barium? [21:43:27] that's where I'm looking [21:43:38] do those come from americium [21:43:43] oh man so manh places to look! [21:43:55] oh sorry, barium:/archive/banner_logs -- that's an nfs mount of americium:/srv/banner_logs [21:46:49] Jeff_Green: grr, just noticed another "feature". The impression loader can either process the last hour of files, or everything. I don't see any nice way to restrict it to e.g. the last 4 hours [21:47:17] when we're done it shouldn't matter [21:47:46] you'll be able to do "all" -- just don't do it until I get the splicing done [21:47:48] k [21:48:07] so here's where we are--udp2logs are in the dir, kafkatee is switched back to 1:10 collection [21:48:17] ok lemme spot-check [21:48:35] kafkatee is empty so far [21:49:00] yep, next rotation is at :00 [21:49:07] k [21:49:44] Jeff_Green: that is /srv/kafkatee/beaconImpressions-sampled10.tsv.log on americium, ja? [21:49:52] I'm going to look at the incoming log rate on americium (kafkatee) vs the rate on erbium (udp2log) [21:49:55] ottomata: yeah [21:50:21] i dunno if i trust my msgs_per_sec script, but it is reporting the same I see for sampeled 10 on my stat1002 kafkatee [21:50:23] so thats good I think [21:50:50] Jeff_Green: any of the sampled-10s in /srv/banner_logs are kafkatee, right? [21:51:04] ottomata: try this: [21:51:10] awk '{print $3}' beaconImpressions-sampled10.tsv.log |awk -F':' '{print $1 $2}'|sort|uniq -c [21:51:28] per min? [21:51:30] kafkatee is collecting about 180-200 hits/min [21:52:05] Jeff_Green: I'm gonna run an "all" banner impression loader job now, to catch up with new udp2log files, ok? [21:52:18] cool, when i look at that on 10 vs 100 on stat1002 via kafkatee [21:52:20] awight: no, hold pls [21:52:21] it looks right [21:52:22] ok [21:52:58] i get 13-27/min from udp2log which is 1:100 sampling, I guess that makes sense right? [21:53:24] awight: which ones? Does it look wrong? [21:53:31] Fundraising Sprint UB40, Fundraising Sprint Vengaboys, Fundraising Sprint William Shatner, Fundraising Sprint X-Ray Spex, Fundraising-Backlog: Recharge minFraud account - https://phabricator.wikimedia.org/T113105#1802150 (K4-713) [21:53:55] yes [21:54:09] and Jeff_Greenwhen i compare on erbium sampled 100 kafkatee vs udp2log [21:54:10] it is the same [21:54:14] Jeff_Green: ~awight/etc-fundraising-django_settings.patch [21:54:27] AndyRussG: Shouldn't those stop within 5 minutes? [21:54:29] Fundraising Sprint UB40, Fundraising Sprint Vengaboys, Fundraising Sprint William Shatner, Fundraising Sprint X-Ray Spex, Fundraising-Backlog: Recharge minFraud account - https://phabricator.wikimedia.org/T113105#1655014 (K4-713) Good news, everyone! The queries have landed. 3,512,665 remain a... [21:54:36] ottomata: ok, so I think all this is pointing to it being fine! [21:55:18] awight: which ones? Sorry I'm lost... I think it takes a bit longer [21:55:43] awight at :00 when the kafkatee logs rotate, I'm going to move them out of the dir to prepare for splicing them [21:55:49] Jeff_Green: keep an eye on it though, its possible there is a real problem that only happens after it has been running for a while [21:55:57] Fundraising Sprint UB40, Fundraising Sprint Vengaboys, Fundraising Sprint William Shatner, Fundraising Sprint X-Ray Spex, Fundraising-Backlog: Recharge minFraud account - https://phabricator.wikimedia.org/T113105#1802167 (atgo) YAY!!! [21:55:58] ottomata: ok [21:58:08] AndyRussG: I'm running stuff like zgrep -c _FR beaconImpressions-sampled100.tsv-20151112-* [21:58:55] ah, mostly mobile campaigns [21:59:35] here's a desktop one: C1516_mlWW_dsk_lw_FR [21:59:40] beaconImpressions-sampled100.tsv-20151112-191502.log.gz:516 [21:59:40] beaconImpressions-sampled100.tsv-20151112-193001.log.gz:4 [22:00:11] 400 impressions every 15 minutes, and it doesn't seem to be tapering off still, 2 hours later [22:01:03] I don't see any browser patterns [22:01:11] awight: hmmm K lemme see [22:01:35] If you want, I can make a task for investigating this... [22:01:56] ok the new 1:10 kafkatee rotated beaconImpressions is 170-200 hits/min [22:02:28] they're at barium:/srv/banner_logs/*gz if you want to spot check [22:03:29] Jeff_Green: those look good. [22:03:47] great. redoing the splicework now [22:04:41] hey K4-713 [22:04:50] https://www.youtube.com/watch?v=bX_kKA6gXCg [22:05:45] Jeff_Green: fyi, it looks like the script can handle a renamed directory, if you want to break with "udplogs"? [22:08:39] awight: I don't see that one in the choice data or anything... Are you still getting impressions for it? [22:08:51] Maybe caching has changed? [22:09:03] I think it's cache-related [22:09:12] just creepy that we go from 100 to 1% [22:09:32] Huh? [22:09:46] seems like a lot of mobile stuff, which hints that it's someone else's proxy [22:10:04] Yeah that wouldn't surprise me [22:10:05] um, that the number of impressions seem to go down to about 1% after you disable a campaign, but stay level there for a long time [22:10:36] awight: Ah right! gotcha [22:10:55] Unless it seems like something awful, let's make a card and investigate it properly then ;) [22:17:20] Fundraising Sprint William Shatner, Fundraising Sprint X-Ray Spex, Fundraising-Backlog, Unplanned-Sprint-Work, Patch-For-Review: Dedupe data in Silverpop file - https://phabricator.wikimedia.org/T107045#1802241 (awight) (I think @atgo's intention was to put this back in the sprint?) [22:18:01] awight: we've definitely had complaints about people on mobile seeing old banners before [22:18:58] AndyRussG: the-wub: totally... This is a huge improvement over the previous CentralNotice code, but if we're directing millions of people to banners and have to shut them off because of e.g. broken payment methods, having tens of thousands of people still getting banners will be an issue [22:19:07] I wonder if there's anything we can do... [22:21:38] Fundraising-Backlog: Worldpay: donors receive error messages despite successful contribs - https://phabricator.wikimedia.org/T117239#1802252 (Ppena) Is this something I should follow up with WP, or is this internal? [22:22:30] Fundraising Sprint William Shatner, Fundraising Sprint X-Ray Spex, Fundraising-Backlog, Unplanned-Sprint-Work, Patch-For-Review: Dedupe data in Silverpop file - https://phabricator.wikimedia.org/T107045#1802255 (CCogdill_WMF) I'm not performing the merges, @awight, DS sends these to me after th... [22:24:19] nvm me, I'm going to put my head back in the sand... [22:24:22] the-wub: awight: K that makes it a priority [22:24:43] :p [22:24:45] (what u and the-wub said before ^) [22:25:08] well. I'll make a card and we can triage later [22:25:17] maybe some cache header changed... the-wub did this start notoriously at any particular time? [22:26:04] AndyRussG: oh no, it's been a longstanding thing that we hear about occasionally. not sure if there is a task for it somewhere [22:27:10] awight ok the splice is done [22:27:22] Jeff_Green: great, looking in a second [22:28:35] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Spike: Is there a way to prevent cached banner views? - https://phabricator.wikimedia.org/T118516#1802288 (awight) NEW [22:28:54] Jeff_Green: what's the directory, now? [22:29:01] still /archive/udplogs/2015 ? [22:29:21] Jeff_Green: were you interested in renaming the dir? [22:30:09] new dir name is: barium:/archive/banner_logs/2015 [22:30:21] okay [22:30:32] the old name refers to an nfs mount that's going away [22:30:45] pls make that config change when you can [22:30:59] less urgent: bannerImpressions* can go away [22:31:03] ok finishing notes first [22:31:07] we only parse beaconImpressions [22:31:08] sure [22:39:14] awight both changes made and propagate [22:39:17] ...d [22:46:12] here i go... [22:46:22] crossing fingers [22:46:27] hiding under desk [22:46:45] what's the recovery plan? restore from db dump? [22:47:12] well [22:47:19] i guess that depends how peter designed the db [22:47:25] <_< [22:47:26] >_> [22:47:40] i vaguely remember talking about being able to remove data by parsed file [22:48:08] I could theoretically surge it, but the failure scenario that scares me the most is that old files are double-processed [22:48:33] is there a backup, in case we need it? [22:48:36] well we'd better confirm the backup then. yeah [22:48:38] such a big db... [22:48:39] k [22:48:40] checking [22:49:24] 7.0G pgehres.20151111-094735.db1008.sql.gz [22:49:27] looks promising [22:49:30] good enuf [22:49:33] ok, proceeding. [22:49:44] Fundraising Sprint William Shatner, Fundraising Sprint X-Ray Spex, Fundraising-Backlog, Unplanned-Sprint-Work, Patch-For-Review: Dedupe data in Silverpop file - https://phabricator.wikimedia.org/T107045#1802402 (MBeat33) @awight cid=7943301 and cid=12047028 are the same donor, from Zendesk #166766 [22:50:00] one would hope it stores filename by just filename and maybe md5? [22:50:09] hah [22:50:10] rather than full path [22:50:19] that last part is true. no md5 tho [22:50:36] failed to run [22:50:43] * awight kicks things [22:50:48] what error? [22:50:49] config buffoonery [22:51:40] should I not have taken your diff literally? i noticed you removed the /yyyy/ part [22:52:55] oh it's a sad situation [22:52:58] it's worse than that [22:53:16] * Jeff_Green contemplates how that can be [22:53:19] the script is hardcoded to get the path from a source file rather than config file [22:53:28] !! [22:53:30] !!!! [22:53:54] It was hard to see, cos the... config file name is a constant... don't ask [22:54:04] let's make a cron job to drop a symlink just before it runs :-P [22:54:12] yes! [22:54:28] luckily, it failed by just not seeing the config parameter [22:55:20] (PS1) AndyRussG: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/252861 [22:55:27] okay, taking the most direct path :( [22:56:03] meanwhile, I have a meeting I can't miss in 4 minutes [22:56:23] Not a good way to ease into the Big English season... [22:56:38] nahhh, this is piddly stuff [22:56:51] (PS1) Awight: Reinstate insane hardcoded config [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/252862 (https://phabricator.wikimedia.org/T97676) [22:56:59] AndyRussG: one more wax seal, sir ^ [22:57:09] Jeff_Green: you gonna be around in 30 minutes? [22:57:38] hehe: https://www.google.com/googlecalendar/unavailable.html [22:57:48] awight: cool! Way to go [22:58:05] * awight cries out loud [22:58:13] gtg [22:58:35] awight--sorta. I gotta eat, will check in after. SMS on your return if I'm not here and you need things [23:35:29] (CR) AndyRussG: [C: 2] Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/252861 (owner: AndyRussG) [23:37:49] https://gerrit.wikimedia.org/r/#/c/252862/ [23:37:56] Anyone available to pretend that is a good idea? ^ [23:38:09] AndyRussG: ^ [23:38:13] (Merged) jenkins-bot: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/252861 (owner: AndyRussG) [23:38:19] i have no idea if it's a good idea [23:38:24] but if you tell me to merge it i will [23:38:28] :) please [23:38:47] sorry... things don't normally go down like this, I swear [23:38:49] (CR) Cdentinger: [C: 2] Reinstate insane hardcoded config [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/252862 (https://phabricator.wikimedia.org/T97676) (owner: Awight) [23:39:03] hehe [23:39:10] does that one need to kick verified too? [23:40:01] i got it [23:40:08] (CR) Awight: [V: 2] Reinstate insane hardcoded config [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/252862 (https://phabricator.wikimedia.org/T97676) (owner: Awight) [23:41:01] !log update DjangoBannerStats from 9cc2acba7f6b46e243ada0fac6c48c9a49460a07 to 08ffecf412a6a9ff3e173dded6814092834c7d1e [23:41:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:41:57] wat [23:48:17] this code is _*SS_ [23:53:28] Jeff_Green: crap, I just noticed that landingpages files don't include a sample rate? [23:53:52] Luckily, I think that's unused at the moment. [23:56:32] Fundraising-Backlog, Wikimedia-Fundraising, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-DonationInterface: Looks like we've been randomly skipping landingpages impression logfiles - https://phabricator.wikimedia.org/T118526#1802608 (awight) NEW [23:58:49] awight they're 1:1 [23:59:20] for whatever reason that's the naming scheme we've used historically [23:59:29] ooh thx