[00:02:34] OK, I'll check in on the export in 45 min or so to see if it got the Civi-marked bounces [00:03:57] ejegg|bike: quoted true?... [00:05:12] K4-713: my screen. it's covered in cute [00:06:51] ejegg|bike: oic :) [00:07:05] (CR) Awight: [C: 2] Fix on/off switch for TY adding CiviMail records [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/169967 (owner: Ejegg) [13:09:05] meganhernandez: hi, exactly the person i was looking for! [13:10:20] I thought it might be a good idea to have blog post and a bunty of the kind that is decribed here: http://kottke.org/14/10/you-should-consider-subscribing-to-wikipedia [13:10:38] probably should be talked with Karen from comm team [13:12:47] hi matanya [13:13:10] yeah, that article is great. i’ve actually been talking with comms about it (katherine mayer, i think is who you’re talking about) [13:13:54] yeah, my memory leaks names [13:14:26] meganhernandez: Maher* [13:14:32] matanya: So does Megan's I guess :) [13:14:53] oops yep, [13:16:07] if you need any help i'm here :) [13:18:22] thanks! i’ll let katherine know [13:18:25] btw meganhernandez is it by design i never get donation requests ? [13:18:36] what’s the best way to get a hold of you? [13:18:47] here, talkpage, mail [13:19:12] i'm quite available, and kinda known among the wikis :D [15:25:04] Morning ejegg|away! Feel like doing a quick +2 FTW? https://gerrit.wikimedia.org/r/#/c/169867/ [15:25:07] ? [15:34:13] (CR) AndyRussG: [C: 2] "Trivial and there's a deploy scheduled in little while. Many apologies to the rules. :)" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/169867 (owner: AndyRussG) [15:34:28] (Merged) jenkins-bot: Fix links in campaign pager on banner pages [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/169867 (owner: AndyRussG) [15:48:16] AndyRussG: shall I prepare the mw-core patches for CN, or did you do that already? [15:49:05] awight: Hi! I think I'm doing it... Just got a local copy of wmf_deploy and was about to say "git merge master" [15:49:18] yepo that will do it [15:49:26] Though I see previous cases are mergining of "remotes/origin/master".. [15:49:27] OK [15:49:37] we've been doing this optional thing, where u "git cherry -v origin/wmf_deploy origin/master", then put that into the merge commit msg [15:49:48] awight: thanks for your message regarding CN-admin-list [15:50:20] matanya: oh, sure! Weren't you the one who mentioned that meta admins could use CN? Sorry if my memory is failing :) [15:50:50] don't think so, but maybe, OOM killer killed mine [15:50:51] Hmmm... Ok did the merge, I guess now it's git review? [15:51:17] AndyRussG: yep, and you can self-merge [15:51:26] AndyRussG: btw, have you seen this? http://www.mediawiki.org/wiki/MediaWiki_1.25/Roadmap [15:52:10] matanya: hehe store fewer full movies... [15:52:10] Hadn't looked yet... [15:52:27] AndyRussG: that's the chart which explains which branches of mw-core we need to deploy [15:52:29] :) [15:52:35] awight: right [15:52:58] awight: git review says I'm about to submit a zillion commits, that's correct, yea? [15:53:17] oof [15:53:23] not really... but probably harmless [15:53:33] you might need to "git fetch --all" to make that warning go away [15:53:45] it means your local "gerrit" remote ( :) ) is behind [15:53:52] not confusing at all. [15:53:59] I did a git fetch... [15:54:01] all [15:54:05] oh hrm [15:54:13] this is still in CN? [15:54:15] It's listing all the changes I'm merging into wmf_deploy [15:54:26] (PS1) AndyRussG: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/170057 [15:54:27] that sounds a bit fishy [15:54:51] Ordinarily, I merge "origin/master" cos I can be more certain that I'm not pushing locally forked patchsets [15:55:07] Ah hmm OK lemme try that [15:56:08] (Abandoned) AndyRussG: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/170057 (owner: AndyRussG) [15:56:41] AndyRussG: oh hrm that looked fine [15:57:36] (PS1) AndyRussG: Merge remote-tracking branch 'origin/master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/170059 [15:57:58] Just did the same w/ origin/master, still showed a list of load of commits in there [15:58:16] yeah donno why that is... [15:58:34] but the one you pushed for review was fine, so I'm sure this one is also. [15:58:45] * AndyRussG represses silly jokes because time is short :) [15:59:09] Last deploy ejegg|away let me follow along by doing a Hangout screenshare [15:59:11] hehe, 1 hr is fine for the deploy, it just gets tight if we need to rollback after testing [15:59:31] yeah the hangouts copiloting seemed to work well [15:59:35] a bit boring :) [16:00:38] Don't let me take the fun part, if you want to do the whole deploy, btw. I thought I might help by leaving your hands free for testing. [16:01:03] awight: I can't, no rights 4 that [16:01:07] OH [16:01:08] grr [16:01:48] u should start that ball rolling, or hit it with a broom or something! [16:02:09] heh seems it's unsure if that specific ball (deploy rights) is rollable [16:02:13] might be square [16:03:05] Wanna +2 the merge in Gerrit? ( https://gerrit.wikimedia.org/r/#/c/170059 ) [16:03:21] on it... [16:03:51] (CR) Awight: [C: 2] Merge remote-tracking branch 'origin/master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/170059 (owner: AndyRussG) [16:04:05] AndyRussG: you have +2 in CN, though? [16:04:11] awight: yes [16:04:31] ok. You're fine self-merging deployment branches... [16:04:32] I just self-merged another one-line patch to get it in with today's deploy in fact [16:04:36] Ah OK [16:04:56] oh, self-merging code not cool--can you send me the link and I will toss some unholy water on it? [16:04:57] Yea just deploy newbie jitters [16:05:20] https://gerrit.wikimedia.org/r/169867 [16:05:28] we try not to self-merge actual changes, even in emergencies... [16:06:07] (CR) Awight: "Good change! retroactive +2" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/169867 (owner: AndyRussG) [16:06:19] Ah hmm, OK didn't know that in this sort of 1-liner case it wasn't doable (since I have seen it done...) [16:06:21] thx for the heads-up! [16:06:36] AndyRussG: there are... many reasons to avoid that [16:06:37] heh np thanks for the time-machine approval [16:06:45] hehe [16:06:55] K yeah that makes sense [16:07:40] Should i be looking for mw-core patchsets? [16:08:04] Not made yet... [16:08:15] looks like the active branches are 1.25wmf5 and 1.25wmf6... [16:08:45] Why don't I let you push those buttons still, would that be cool? I'd love to follow on an hangout tho [16:09:39] sure! [16:10:23] :) [16:28:48] AndyRussG: CN deploy in 30? [16:28:58] Did I read that right this morning? [16:29:09] K4-713: no happening right now :) [16:29:24] * K4-713 blinks [16:29:53] Oh, it's *over* in 30. [16:30:02] I knew 1700 UTC was in there somewhere. [16:34:10] awight: hang on I have to restart my silly VM, back in the hangout in a sec [16:35:39] AndyRussG: no worries. [16:35:45] AndyRussG: Looks like your new code, here [16:35:49] here: http://bits.wikimedia.org/static-1.25wmf5/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js [16:35:52] true? [16:35:54] Jeff_Green: Is something up with Jenkins? [16:37:05] I'm trying to get in to look at why a nightly failed last night, and the web interface won't even do the initial load of the login screen. [16:37:10] Spinning... [16:37:13] orlly [16:37:20] Yeah, I've never seen this before. [16:37:29] i did firewall rule cleanup, probably missed something [16:37:34] how are you trying to access it? [16:37:40] K4-713: if this is the GC audit, it timed out... [16:37:51] awight: I know. [16:37:55] k then [16:38:15] awight: But it timed out because it was trying to look at too many things. [16:38:22] K4-713: login page shows up for me [16:38:23] Anyway. [16:38:25] K4-713: how are you accessing jenkins [16:38:32] Jeff_Green: Let me look at my .bashrc [16:38:39] there appeared to be several possible ways, I tried not to break any of them but I probably did [16:39:00] oh, that was happening to me the other day when I had a dead ssh session hanging on to the forwarded port [16:39:29] K4-713: actually, just try to access it now and I'll watch firewall logs [16:40:03] okay, I can do that too. Just pasted the command elsewhere. [16:40:24] ...and hit it again. [16:43:22] AndyRussG: Sorry I wasn't here for that CR earlier. can I help test the deploy? [16:43:47] ejegg: Sure! testing on enwiki right now :) [16:44:19] (CR) Awight: "This patch is deployed, but does not work. And please exercise self-merge self-restraint!" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/157484 (https://bugzilla.wikimedia.org/54180) (owner: Ori.livneh) [16:44:58] That looks serious. [16:45:36] kinda stinks. He's great, but I'm going to apply trout-slapping in -dev [16:54:45] awight: So, what does this mean for today? [16:55:18] K4-713: all our stuff has been tested and verified working [16:55:42] Bucket fixes? [16:55:45] There was one more rider, which was a favor from ori, but it didn't work. And I got my cup in a bunch cos he had self-merged it and ignored AndyRussG's comments... [16:55:53] bucket stickiness is confirmed working. [16:56:00] woooooooooooooooo [16:57:12] enwiki still working, I can still read about the illegal fried pencil shaving trade [16:57:16] yeah, definitely a case of hi-fives all around! [16:57:23] :) [16:57:29] congrats! Looking good [16:58:09] https://meta.wikimedia.org/w/index.php?title=Special:CentralNotice&method=listNoticeDetail¬ice=C14_mlWW_dsk_lw_FRe [16:58:49] ori's patch seems to work here... [16:59:10] "illegal fried pencil shaving trade" [16:59:19] ...I know what all those individual words mean. [16:59:20] AndyRussG: https://meta.wikimedia.org/wiki/Special:CentralNoticeBanners/edit/B14_103_enWW_dsk_hg_txt_cntl [17:01:22] ooops awight correct, it doesn't work [17:01:35] i was looking at campaigns, not banners, bllrrrrg [17:01:44] really it's all the same [17:04:39] * AndyRussG runs from the music [17:04:58] lol seeeriously [17:05:08] Isn't that illegal or something? [17:05:14] I mean, Dolly Parton is a freaking genius, but... [17:05:16] Oh noes. [17:05:29] darn West coast lax music laws [17:05:33] K4-713: ? [17:05:33] here's my dollar i gotta go [17:05:34] What the hell happened to the fundraising source in ganglia? [17:05:45] awight: thanks! [17:06:20] yah see ya in a few [17:06:34] Jeff_Green: You know anything about the frack being removed from ganglia? [17:06:58] I'd really like to think that somebody would at least run that one by us. [17:07:26] >_< [17:07:34] ejegg: AndyRussG: I was hoping to chat about #2062 in a bit... so I can write the canonical Oh Noes [17:09:01] Last hour queuesize graph is also... empty. [17:09:21] awight: sure... that would go down the treatment phases road [17:10:07] K4-713: sorry, I meant "shavingS". The illegal fried pencil shavings trade [17:10:21] * AndyRussG quickly creates a page on ENwiki [17:10:46] You know what you get when you sharpen a pencil [17:10:54] Then fry the shavings [17:11:04] and trade them for eraser dust [17:11:05] Yeeeee...sss... [17:11:06] illegally [17:11:52] Huh. [17:12:09] Is that what the kids are doing now? [17:12:29] And, just to be clear: This is *not* a euphemism? Or... [17:12:34] no [17:12:43] uuuhh I wouldn't know, what do you think they tell me stuff? [17:12:51] Oh. Yes. [17:12:53] Yes, I see. [17:12:57] It's all very mysterious. [17:13:37] * awight blinks [17:13:43] kbye [17:13:51] smrt [17:14:45] Well, this is interesting. [17:15:40] There are... 500 or so GC records that we don't have in the DB and can't seem to reconcile. [17:15:57] ooh [17:16:09] Which is why it was timing out. [17:16:29] all around the same time? [17:16:36] Can't tell from right here. [17:16:43] Hopefully, the job won't time out again. [17:16:50] ...and then I'll be able to see that breakdown. [17:17:08] Or, I could just finish the WX upgrade and run that instead forever. [17:17:12] :D [17:17:44] I like that better anyway. [17:18:14] forward thinking [17:18:23] Nah, just really lazy. [17:19:18] I suppose it might actually find them all, too. [17:20:30] Actually... this very strange. [17:20:51] Looks like these are all cc transactions. [17:21:00] Should have gotten these in realtime. [17:21:28] Something is Wrong. [17:24:46] Yeah, okay. Confirmed, something is Very Wrong. [17:25:05] I'm only seeing amazon messages in the donations queue. [17:26:26] erk [17:34:39] Hey Jeff_Green: Let me know when you get back. There's been an Incident. [17:35:05] Seems that payments can't talk to the message queue. [17:38:11] Are we logging that someplace? logs seem disturbingly normal for that kind of problem [17:38:20] Yes they do. [17:41:57] Okay... weird. Now I'm definitely seeing gc messages in the queue. [17:44:31] I'm.... just going to go ahead and write all of today's weirdness off to the firewall rule changes. [17:45:52] would some sort of heartbeat monitor be useful? Like each payment server dropping an "i'm alive" message on the queue once a minute? [17:47:16] or just often enough that each run of the QC should expect to find at least one of those from each server? [17:49:48] This is something we're going to have to talk to Jeff about. [17:50:09] Because, whatever is dropping the messages, something is going to have to look at them and do the actual alert. [17:50:32] And usually when we don't get notifications, that's the thing that's broken. [17:50:40] The queue consumer? [17:50:43] no [17:50:48] The alert thingy. [17:50:52] ahh [17:51:54] And we can't rely on threshholds for regular activity, because there's no way to tell from that single point if it's because we've taken everything down, or if something melted. [17:52:08] QC could send failmail if the heartbeat queue is missing any server [17:52:13] on each run [17:52:40] then we won't be alerted if we take down both qc and payments cluster [17:53:25] I think there are (theoretically) more straightforward ways to do this. [17:53:27] we'd have to do it as a background job though, so that might not fail the same way as actual queue inserts from web hits [17:53:35] Like, things we already have. [17:54:08] So, I don't want to think too hard about it until Jeff_Green can comment. [17:55:08] It's... sort of the difference between integration tests and unit tests, if that makes sense. [17:55:44] If there are too many moving parts in the test, we're not sure which one is either causing the screaming, or not screaming when we expect it to. [17:57:55] right [18:00:59] AndyRussg ejegg is hangouts freaking out for you, too? [18:01:41] we're in [18:01:55] there in a sec [18:09:35] k4-7ie back [18:15:45] atgomez: lemme send you the pointer to the proposal for the next CN step if u like [18:15:53] so you have it on hand for your check-in? [18:16:35] again, https://docs.google.com/a/wikimedia.org/document/d/150lrUt8b4aiPK19XCcoE6Yy3TwN_wo529UGQjk0eTbo/edit [18:16:57] *only* "Immediate Solutions: Option 3, Detail" starting on page 5 [18:17:00] not the whole document [18:25:02] hey k4-713 - megan says the donations seem to be coming through, maybe it's a bit low but not hugely [18:25:14] Okay. We definitely had some out time last night. [18:25:27] It'll rebuild once the logs rotate, so no loss here. [18:25:32] maybe not during one of their active tests [18:25:32] cool [18:25:41] But, we're missing at least 500+ transactions from the last 24 hours. [18:25:53] Looks like all credit card. [18:25:55] ah... yea just figured i'd let you know she was seeing relatively normal numbers [18:26:00] ok i'll let her know [18:26:02] cool. [18:27:31] atgo: I just sent you an e-mail with a pointer to the CN proposal, just in case it's useful right now :) [18:27:40] hehe thanks [18:27:45] the short term fix is out, yeah [18:27:46] ? [18:27:49] AndyRussG [18:27:52] yep! [18:28:01] Deployed and smoke tested on production [18:29:01] AndyRussG: ejegg: anyone care enough chat about #2062, or should I just make something up... [18:29:49] I'd be fine with doing it as described in the ticket, but just substituting localStorage for cookie as you suggest [18:29:55] ejegg: Checkin? [18:30:18] awight: anytime... [18:30:21] K4-713: sure, if you don't need to urgently converse with Jeff [18:30:29] AndyRussG: ok! [18:30:53] ejegg: you would be ok with transmitting 1/20 of the time, or even at a lower sampled rate...? [18:31:11] right, that makes sense too [18:31:13] ok [18:35:43] AndyRussG: http://caniuse.com/#feat=namevalue-storage [18:36:52] awight: you've froze! [18:45:18] http://www.mediawiki.org/wiki/Analytics/Logging_infrastructure [18:48:44] reconnecting... [18:49:41] can't hear u [18:49:43] new FR banner: donate to fix up office wifi [18:49:43] nor see [18:49:50] lol [18:49:53] and hangouts [18:49:59] heh [18:50:01] I can hear me [18:50:06] :D [18:50:23] I can't hear or see you [18:50:36] err, all I was gonna say was, yeah as long as we're not sending Referrer [18:51:01] I also feel strongly about making the localstorage contents really transparent, with self-explanatory field names and contents [18:51:10] no mysterious IDs [18:51:11] hmmm is referrer an especially bad piece of data? [18:51:15] yeah [18:51:21] that's the article you've been reading [18:51:32] ah right got it [18:51:34] we do capture that for debugging [18:51:43] but we have so far avoided ever using it in analytics [18:52:02] k, hangouts is dead [18:52:21] If you have anything more I should be aware of, lemme know, otherwise I think I can write the email [18:52:55] yeah hangouts hung [18:53:02] hehe [18:53:11] hangups [18:53:14] nah I think whatever you suggest will be great [18:53:23] :p ok pizzas all around [18:53:41] regarding transparency of localstorage data, is obfuscation any better? [18:53:49] eh? [18:53:54] why would it be better? [18:54:00] right, it wouldn't [18:54:06] ok [18:54:12] ah ok I misunderstood [18:54:25] I thought you were saying the opposite ^ above [18:54:28] the perceived advantage in the state ID thing, AFAICT, is that it makes the data smaller [18:54:31] noo [18:54:44] which state ID? [18:55:22] yeah, that's all i was thinking it would solve [18:55:38] In terms of the actual data in there, maybe try to coordinate/use/adapt the reason field now in the hide cookie? [18:56:50] by state ID, you mean the unique ID? [18:59:09] AndyRussG: I had a different proposal in a murmur on the card [18:59:48] where each possible (simplified) history would have a row in a db with an ID [19:00:09] ejegg: oh interesting [19:00:16] yeah silly murmurs, not always so visible [19:00:21] thanks lemme check that out :) [19:01:13] It would shrink down the data storage, but make it opaque client side and require recursive rollups server side [19:01:24] so, probably not the solution we want [19:02:37] ah hmm [19:03:56] hey jessicarobell meganhernandez, sisolak is going to send a test of the c’est le début French email [19:04:05] it’s definitely winning for now, are you happy to use that SL? [19:04:28] wow that’s awesome [19:05:47] yeah! control isn’t far behind, but the other two SLs are losing for sure [19:05:56] so it’ll be nice to test it somewhere [19:06:05] else [19:06:09] That's interesting ccogdill. I really didn't think it would work that well. Do we have any English records for France? Ie could we do two tests one in English and one in French? [19:06:28] we have a really small number of English emails - I think less than 2k [19:06:35] aha. [19:07:27] but we will send to them! [19:08:09] AndyRussG: sorry, RL happened. reading backscroll. [19:08:27] np! [19:08:45] ok. yep, writing email then [19:11:07] ccogdill and meganhernandez: I am just wondering if the results will be comparable between France and the English speaking countries we sent to today. The "we're starting" might work in those countries because they are already aware of our fundraiser. They have already donated to us last year and maybe also received emails in the past. In France, the fundraiser is very new and the records we have are [19:11:08] from people who donated years ago, right? I am just afraid the "c'est le début" might be confusing. But hey, I might be totally wrong. [19:12:07] i think it is kinda confusing even in english, which is probably why it’s getting clicks [19:12:34] here i’ll usertest real quick! [19:13:15] yeah I agree jessicarobell, it’s certainly not my favorite SL. but Megan has a point that sometimes vague is better [19:13:53] yeah sure, it might just get people curious enough to open [19:14:19] also, we always have email 2 :) we could try a less vague email 2 SL with France to counteract this one? [19:14:59] yeah, that's also an option [19:15:06] Sisolak just sent the test [19:15:31] ok usertested on 2 french guys, they gave it a thumbs up [19:16:26] awesome [19:16:28] so let’s do it? [19:17:22] I have to run out for a couple minutes - will follow up on email if you guys are offline when I get back! thanks for checking in on this [19:20:14] i’m ok with it is jessicarobell is [19:20:42] ok great. Thanks for user testing it meganhernandez. sure, let's try it. [19:21:10] by “usertesting” i mean sent it to the roommates and then went outside and asked them what they think [19:21:20] so not a huge sample, by any means :) but they were into it [21:08:01] K4-713: can u help me with fredge? [21:08:44] Er. [21:08:46] Maybe? [21:08:48] What do you need? [21:08:54] I just... cannot successfully donate. [21:09:02] ? [21:09:08] yeah. [21:09:10] What do you mean? [21:09:13] So i want to look at fredge [21:09:17] Oh. [21:09:20] I tried my CC through GC and WP. [21:09:21] no dice [21:09:21] So, "use fredge" [21:09:31] um right ok I think I can take it from here :D [21:09:36] bahaha [21:09:39] That was easy. [21:10:18] I was about to say "describe payments_fraud" but I'm not sure if I called the table payments_fraud or payments_antifraud. [21:10:31] Anyway. [21:11:03] If you have your contribution tracking id, you can look yourself up. [21:18:11] atgo: You have a minute? [21:18:20] yah [21:18:24] i'm couch-ing at the moment [21:18:27] want me to come there? [21:18:29] I need to explain wtf just happened for the last five hours. [21:18:33] I'll go over there. [21:18:44] yes, come to couch-landia [21:18:46] it's nice here. [21:18:48] the water's fine [21:22:58] K4|overthere: we aren't getting source_type. from anything. [21:26:01] K4|overthere: ejegg: shields up! [21:26:03] +---------------------+-------------+ [21:26:03] | max(c.receive_date) | source_type | [21:26:03] +---------------------+-------------+ [21:26:03] | 2104-08-25 00:00:00 | NULL | [21:26:05] | 2014-09-16 20:11:22 | audit | [21:26:07] | 2014-10-29 21:16:20 | direct | [21:26:10] | 2014-09-17 18:07:58 | listener | [21:26:12] | 2014-09-17 18:08:02 | payments | [21:26:15] +---------------------+-------------+ [21:26:56] 2104, huh? [21:27:42] ;) that happens... ? [21:27:52] oh man, since 9/17? [21:34:03] Not seeing any deployments around then to blame [21:35:25] hrm. I was imagining it was the Civi hook work. [21:36:05] nope, you're right. [21:36:30] oh, maybe it was the hooks. hooks work went into master a week or two prior [21:38:15] well, I don't see anything in the server admin log, but I think I was on strike from calling ! log [21:39:13] deploy branch log shows my klugy merge on the 23rd [21:39:34] but no obvious deploy merges before that. [21:41:00] sheesh [21:41:05] huh, I may not be looking for the right stuff [21:41:30] this data is still coming into the queue consumer, and exists in the normalized contribution before it's saved... [21:41:39] I gotta think it's the hooks. [21:42:08] oh, right, and I'm barking up the wrong tree in DI [21:42:13] oh! [21:42:19]