[00:09:49] ejegg|biab: This second reporting patch looks cool, but I'm a little wary of merging a patch that starts with "WIP"... [00:11:20] that doesn't stop some people... [00:12:14] (CR) Katie Horn: "WIP, huh? What are we waiting on?" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/174313 (owner: Ejegg) [00:18:31] AndyRussG: Ha, I know! Which is why sometimes I -1 my own patches, too... [00:19:32] There really should be an official "not yet" button. [00:22:52] Well there's the draft patch feature, but I think it has other disadvantages [00:26:00] There was definitely a reason we all independently decided not to use that, yes. [00:26:10] Can't actually remember what it was, though. [00:26:57] Heh was thinking the same thing [00:27:14] I think part of it is that it doesn't show up on normal lists of stuff, no? [00:32:07] testing testing.. is this thing on? [00:32:09] new IRC client [00:33:12] atgo: welcome to alternate universe IRC [00:33:22] the other one always kicks m eout [00:33:35] mmm maybe you're originally from here then [00:33:51] the yellow pill [00:33:56] or was it green? [00:34:23] haha [00:52:59] K4-713: the WIP was basically 'need to test in a real replicated db' [00:55:00] Ah, okay. Do we have to merge it before we can test it, then? [01:14:59] K4-713: I'll try to test with Jeff_Green tomorrow morning on a smaller scale before applying it to the whole export [01:24:40] K4-713: ejegg: quick question: centralnotice-admin rights... ejegg you have them? should I ask for them too? might be fun for testing campaings and fiddling with many different criteria (like in esperanto like ejegg did or on aa.wikibooks) though if not doable I can also ask someone else to help me test... [01:25:18] I'm in favor of all of fr-tech having those rights, but you in particular should have them. [01:25:38] If only to, you know... shut off campaigns that are crashing everything. [01:26:31] I'm... surprised we didn't do this yet. [01:26:34] :/ [01:26:46] Fixing. :) [01:31:34] K4-713: ah ok thanks! I was gonna file a ticket but thought I should ask if it was a good idea first... but thanks! [01:31:56] I have to provide a statement of approval anyway. :) [02:05:05] K4-713: for travel, I'm requesting to travel more or less the afternoon of Sunday, November 30th and depart SF the morning of Saturday, December 6th. Does that sound right? [02:12:08] Also--everything that week will be at the office, right? [02:40:56] AndyRussG: Bah, forgot to log off. [02:41:08] But, yes: That looks right. [02:54:33] K4-713: Ok thanks! :) [02:55:06] Yep! Sorry it took so long. [02:57:04] Ah np! sorry to bother you when you had planned to be logged off... [04:49:07] (PS1) Ejegg: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/174367 [04:50:56] (CR) Ejegg: [C: 2] Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/174367 (owner: Ejegg) [04:56:59] !log updated tools from 419fb7aa32c6d0776056968378e358ee01985565 to fe9b463379fac35ad5e71a57fbbb95ae39e2356e [04:57:04] Logged the message, Master [07:03:15] Hey meganhernandez. We're melting. [07:03:21] How are you? [07:05:08] hi i took them down [07:05:08] https://meta.wikimedia.org/wiki/Special:CentralNotice [07:05:12] looks good? [07:06:45] meganhernandez: Ha. I... can't actually parse what that page is trying to tell me without a lot of time. [07:07:06] our campaigns are down [07:07:08] Not really used to deciphering the campaign naming conventions. [07:09:23] meganhernandez: So, in case it wasn't apparent from the emails, I'm not sure what it's going to take right now to get things working again. [07:09:55] Looks like we're going to have to deploy something, and even if I manage to work it out myself tonight, I still need somebody to review the patch. [07:10:04] nah go to bed [07:10:10] we’ll figure it out tomorow [07:10:31] Eh. I'm... probably going to at least determine what to start messing with tomorrow. [07:10:39] I hate to go to sleep with mysteries. [07:11:19] just writing this to the thread: Fundraising campaigns came down at 7:03UTC: https://meta.wikimedia.org/wiki/Special:CentralNoticeLogs [07:11:20] I'm taking this as a no for the US test at 6am Pacific / 9am Eastern and a no for putting up Israel banners. Let's check in tomorrow on what we need to do to get campaigns back up. Thanks for flagging the problem, [07:11:31] ooh, what's melting? [07:11:44] sound good K4-713 ? [07:11:48] * ejegg checks email [07:12:01] ejegg: ! [07:12:04] You're up late. [07:12:21] meganhernandez: Yeah, sounds good. [07:13:16] ejegg: We forgot to alter the curl options for GC to... work. [07:13:32] So, we're not using TLS? [07:13:36] oopsie [07:13:37] Nope. [07:13:54] Also, they didn't switch over when they said they were going to. [07:14:08] yeah, I thought this was supposed to happen a couple days ago [07:14:12] yesterday [07:14:27] This should not surprise me. [07:14:49] how big is the curl update? [07:14:52] good luck guys... i'd stay up if i was going to be useful and also not just get myself sick(er) [07:15:08] seriously though, if it makes more sense to deal with it in the morning that's definitely just fine [07:15:14] ejegg: Don't know. I've not looked at this exact thing before, and I don't think my hope IP block is whiteliseted. [07:15:19] rest well, atgo [07:15:20] sleep fuundraisers [07:15:32] see you cats in the morning [07:15:40] I can't! It's still slightly mysterious! ;) [07:15:49] But, you guys should go. [07:15:53] Huh. [07:16:08] It looks like maybe they're not prepared to flip the switch. [07:16:08] k. looking forward to an update [07:16:11] good luck [07:16:26] Based on the fact that it seems to be coming back, despite the part where we're still on SSL. [07:16:54] * K4-713 silently places bets on why [07:16:57] Oh, they did say they would turn it off for an hour first just to test. [07:17:05] Maybe that part of the plan got pushed back too [07:17:18] Or, maybe they did it and all the alarms started going off. [07:17:31] revenue dropped to zero [07:18:01] They could spin it as the best ever antifraud measure. [07:18:09] hehe [07:18:17] Good news, everyone! No more chargebacks... [07:19:36] Okay. I'm probably going to regret saying this, but it *looks* like an easy fix. [07:19:41] >_> [07:19:43] <_< [07:19:49] 'course, I can't test from here. [07:19:57] I really should just shut up. [07:20:03] curl_setopt( $handle, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1) [07:20:04] ? [07:21:48] well, if they're letting us back in now, might as well wait till morning to try it [07:23:48] Argh, I just found that exact thing. [07:24:14] But yes. There's an overrideable getCurlBaseOpts function in the parent class... [07:27:04] It's not immediately clear to me, though, what we want to set CURLOPT_SSLVERSION to. [07:27:38] GC seems to want TLSv1.0, specifically. [07:27:50] So... 4? [07:29:45] the const CURL_SSLVERSION_TLSv1 seems to be defined as 1 [07:33:16] I think that's version 1.x. [07:33:22] 4 is 1.0 specifically. [07:34:06] Or, CURL_SSLVERSION_TLSv1_0 if you prefer. [15:28:25] (CR) Pcoombe: [C: 2] "Thanks Elliott!" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/174319 (owner: Ejegg) [17:30:17] * K4-713 waves [17:30:50] meganhernandez: Hey, good morning. [17:31:23] hi K4-713 [17:31:39] How are we doing? [17:33:05] all banners are down [17:33:33] I saw that you did a short test this morning. What happened there? [17:33:37] just put them up for 15 mins with jeff earlier. [17:33:54] jefff wanted to check something, took it down [17:34:00] hm. Okay. [17:34:07] i’ve been on the phone since then so he probably has more info on what’s going on from his side [17:34:12] Okay. [17:34:24] Well, from what GC was saying, it wasn't anything on our end anyway. [17:34:57] Total rookie mistake on my part. [17:35:26] I'm not entirely sure what their status is right now, though. [17:35:44] meganhernandez: Do you know what we're waiting on? [17:36:28] Jeff_Green: And a very Good Aftermath to you, sir. :) What's it look like? [17:36:41] it looks like Wednesday! [17:36:48] Compliments of the Season [17:36:53] heh [17:36:55] this is not a scam but a real winning of gifts! [17:37:23] It's actually raining at my house. [17:37:32] RAINING HERE TOOOOO!! [17:37:41] i'm excited. and wearing mah rain boots [17:37:49] Oh good. I thought I might be imagining things. [17:38:45] Like I blew some kind of fuse and my brain just started lying to me to increase happiness. [17:39:43] K4-713: not necessarily a good sign [17:40:06] AndyRussG: How would I know the difference? :) [17:40:12] Good point [17:40:29] Mmm in that case, I guess just carry on... [17:40:39] Yep! [17:40:43] haha [17:41:17] * AndyRussG turns off philosophical skepticism switch in brain [17:41:33] AndyRussG: So, one of the weird side effects of GC being strange, is that it seems you have a few more hours to iron out CN changes. [17:41:56] How's that going? [17:42:15] OK! should have patches up this morning [17:42:38] no blockers or anything [17:42:39] Heh. [17:42:59] only not having drank coffee early enough in the morning [17:43:14] That's a big deal. [17:43:38] I keep trying to reduce my consumption but it's all in vain [17:43:54] I have too much blood in my caffeine stream. [17:45:14] Jeff_Green: Where did we leave the GC thing? And did you learn anything good from the test this morning? [17:45:36] this AM's test was very contained and short [17:45:55] The last round of emails I read had to do with wondering whether or not they tried to switch data centers on us. [17:46:00] afaik GC has been cut over to their Amsterdam since 30 minutes after the outage last night, is otherwise functioning normally [17:46:07] okay [17:46:22] they didn't try, they 'did' :-) [17:46:31] I can tell you right now we didn't write anything to prevent that on our end. [17:46:42] But, in the past, our account was an exception. [17:47:35] what does that mean? [17:47:44] should we need to write anything? [17:48:26] Not that was ever suggested before last night. [17:48:41] But, we don't want to process payments on servers that are not in the US. [17:49:21] Previously, they did something on their end to make sure our account was an exception to their failover switch. [17:49:45] Sounds like we're not sure if that's still the case. [17:51:48] PPena: Hey, good morning. [17:52:05] I'm trying to figure out where we are right now with the GC situation. [17:54:30] K4-713: could you use geoip to check the location of their API servers? [17:54:49] if it's really important we never cut over I can remove their non US DC from the list [17:54:56] in firewall rules I mean [17:55:02] See, I'm not sure it is anymore. [17:55:12] We had to do some new things for WP. [17:55:37] Which resulted in new legal text... [17:56:01] So, I don't want to do anything weird if we don't have to worry about it anymore. [17:57:05] In fact... let me just write that email now. [17:57:11] I thought this was a newly solved problem. [17:57:49] And if we don't have to turn everything off if they fail us over, then we... shouldn't be off right now. [17:58:45] atgo (or atgo_): You have line of sight on pizzzacat? We're supposed to have a checkin in two minutes. :) [17:59:42] Or, wait. That's awight. [17:59:49] I can totally calendar. [17:59:52] :/ [18:04:34] K4-713 Jeff_Green - email's already out to legal :) [18:04:43] orly. [18:04:46] What does it say? [18:04:53] I just wrote 80% of a new one. [18:05:04] ok [18:05:57] And, can we *please* stop being selective about who gets those? It's a lot easier not to do a 45-minute wheelspin if we can all see what's going on. [18:12:36] atgo__: ^^ [18:13:31] Jeff_Green: I rewrote the queries for silverpop export. Think now's a good time to test locking on the cluster, since we have nothing coming in? [18:14:26] * pizzzacat looks around the office and sees tumbleweeds [18:14:43] Bah. [18:14:44] hk [18:14:49] * pizzzacat hears The Good, The Bad, and The Ugly soundtrack [18:16:50] ejegg: have you tested how it behaves on lutetium already? [18:17:09] pizzzacat: How's it going? [18:17:37] No, just locally. The only table locks are IS, which I gather are shared intents to lock individual rows [18:17:47] K4-713 hi! it's going great. Big English widgets are looking good [18:17:49] But I figure the real test will be in a replicated table [18:17:56] how's everything over there? [18:18:06] ejegg: let's run it on lutetium first and then squint at slow query logs [18:18:16] OK, will deploy there [18:18:19] k [18:18:24] lemme make sure logging is enabled [18:18:58] K4-713: PPena sent out an email to legal [18:19:21] ejegg: slow query logging is enabled [18:19:24] pizzzacat: i'll be there! [18:19:27] i'm just upstairs :) [18:19:43] * pizzzacat wipes away tears [18:19:45] ok atgo! [18:19:52] I hope you wore your boots today. [18:20:09] SURE DID!! [18:20:29] yay! me too! [18:20:37] booots partyyyy [18:20:49] <:o) [18:27:11] About to tune into the scrum of scrums. Any news for them besides CN, dash, and GC fun? [18:29:48] Aw, boots party. [18:30:16] K4-713: we miss your boots :P [18:30:44] haha [18:32:23] how're you feeling today? [18:33:38] Better? [18:33:56] It's a little hard to know right now. Just got off the phone with a detective... [18:34:59] Actually... brb. [18:40:54] arg K4|afk :/ [18:47:27] so K4-713 Jeff_Green PPena GC is still using amsterdam as far as we know, right? [18:47:49] Yeah, I think so. I mean, I didn't see anything after their "We'll flip it back when we know we can" email. [18:47:56] atgo: checking [18:47:59] cool that's what i was running on, too [18:48:05] thanks Jeff_Green [18:48:19] They're usually pretty good about telling people when they take that kind of action. [18:48:31] yeah netherlands [18:48:31] I mean, they want to send that all clear. [18:48:37] also K4-713... megan was wondering when might be good on friday to do a GC vs. WP test.. i assume you want to keep an eye on things? [18:48:45] ahhh ok thanks Jeff_Green K4-713 [18:49:11] atgo: Yeah, I'd like to keep an eye on things. [18:49:27] ok! she was asking me to help coordinate a time [18:49:34] (assuming GC is back up) [18:49:59] But, I need to... write something on my hand to the effect of "STOP ASSUMING IT'S OUR PROBLEM." [18:50:36] huh? [18:50:46] oooh [18:50:47] hand. yes [18:50:48] hehe [18:51:53] Eventually I'll have to cross out "OUR" and write "THEIR". [18:52:14] * K4-713 shrugs [18:52:28] just don't get it tattooed and this is doable [18:52:53] I dunno. Across the forehead backwards might be useful too. [18:53:15] I was going to go with "Wabbit Season" across the forehead, but... [18:53:21] :p [19:02:17] K4|meeting: atgo: ejegg: There is an edge case issue with the legacy algorithm for choosing banners, which was copied in the current code. It is: if you set a campaign to have two buckets, but only assign banners to one of those buckets, and the user is in the bucket with no banner, and that campaign overlaps with another one, and the overlapping campaign does have a banner for the user, then the overlapping [19:02:17] campaign will get too much allocation. [19:02:25] Also: boo! :) [19:02:59] To work, the per-campaign buckets must change the procedure slightly and will not have this problem. [19:03:04] AndyRussG: ooh, tricky [19:03:24] The only problem is that allocation will then be a little bit different in the old and new code... [19:04:01] Yeah! I had this sense there was something blargey hiding in the legacy/still-current algorithm [19:04:13] Now I've found it 8p [19:05:06] ejegg: Do you see where I'm coming from? First we're filtering out banners that the user shouldn't get based on several criteria, including buckets [19:05:18] pulling up that code... [19:06:10] If the user is in bucket B and the campaign has two buckets, the fact that the campaign has two buckets will make the code filter out banners for bucket A, thus filtering out the campaign entirely [19:06:41] If there's another campaign competing it'll fill up all the missing space in the allocation pie [19:07:17] that makes sense AndyRussG [19:07:20] So in that case the user in bucket B will never get no banner, as he should sometimes [19:07:27] ah, I see. So we could just use the number of populated buckets [19:08:26] ejegg: we _could_ do that... [19:08:30] It depends on what the goal is [19:08:55] Right, they might want to intentionally have no banners for bucket b [19:08:56] If the goal is to make allocation stay exactly the same, errors and confusing edge cases and all, that might be the way to go [19:09:14] It'd be weird but the UI gives you that option [19:10:13] I would kinda tend towards make it work the way you'd expect (allocate some space to no banners for that bucket, in that case) sooner than make it work exactly the same in such weird cases [19:10:49] so add a null banner to each empty bucket [19:10:52] by "sooner than make it work exactly the same" I mean, "sooner than make it work exactly the same as it always has" [19:11:46] AndyRussG ejegg what if we raise this point to the production team and see what they prefer? [19:11:48] ejegg: yeah that'd be reaonable [19:12:03] maybe they do want the behavior as is. probably not, but why guess? :) [19:12:39] ejegg: other option would be just to make the JS detect such cases and act appropriately, maybe warn somewhere [19:12:56] and AndyRussG ejegg at least that way they'd be aware. we need to stay focused on getting out the new features/changes this week so CN can be stable before Dec 1 [19:13:44] atgo: maayybe.... we could just phrase it like this: do you ever set a campaign to have more buckets than you actually use, that is, set a campaign to have 2 buckets and only assign banners to 1? [19:13:59] And if not, do you care what happens in those circumstances? [19:14:29] yeah! sounds good... are you feeling like this changes _needs_ to go in with the rest ofo this work? [19:15:14] atgo: yes, something has to be decided on this point to make per-campaign buckets work [19:15:19] I think [19:15:40] * AndyRussG tries to make sure there are no loose marbles in his brain [19:16:06] ok.. meganhernandez is heading to the airport right now but should have wifi on her flight. let's throw the question to her, peter, and jessica and see what they think [19:16:30] OK, cool! :) thanks [19:16:41] thanks :D [19:17:04] For now we could just work on the assumption that they're gonna say, "Meh, we never do that, don't really care," does that make sense? [19:17:15] yep! [19:17:20] K cool! [19:18:17] ejegg: what technical approach would you favor? Making banner choice data provider spit out null place-holder banners for those cases, or just keep it all on the JS side? [19:18:19] PPena: did you see Jeff_Green's question in email? [19:18:37] atgo yes. let's leave it up for now. [19:18:53] ok! [19:19:01] ok great [19:19:09] atgo I will walk to Legal's desk in the pm if they have not replies [19:19:13] replied* [19:19:39] ha sounds good. i thought you said "i will walk and PM them" and i was like "ppena, get offline :P" [19:22:16] AndyRussG: I'd do it in the js [19:22:19] hey guys - reminder no standup today because prawning [19:22:30] ah, right, weekly now! [19:22:32] atgo-away: oh yeah right thx [19:22:48] please update mingle to reflect real life if it doesn't :) [19:23:08] atgo-away: K will do :) what time do the shrimp fly again? [19:23:27] 3PST [19:23:32] K thanks! [19:27:56] ejegg: K makes sense :) [19:28:30] man that old algorithm, it just keeps going and going [19:28:40] really needs to be put out of its misery [19:28:59] thanks! [19:29:33] * AndyRussG makes a list of naughty vs. nice algorithms [19:32:11] (PS1) Ejegg: Switch back to official passport-drupal [wikimedia/fundraising/dash/node_modules] - https://gerrit.wikimedia.org/r/174479 [19:33:38] (PS1) Ejegg: Switch back to official passport-drupal [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/174481 [19:33:44] (CR) jenkins-bot: [V: -1] Switch back to official passport-drupal [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/174481 (owner: Ejegg) [19:35:02] (CR) Ejegg: [C: 2] "Self-merging change to library origin" [wikimedia/fundraising/dash/node_modules] - https://gerrit.wikimedia.org/r/174479 (owner: Ejegg) [19:35:19] (CR) Ejegg: [V: 2] "Self-merging change to library origin" [wikimedia/fundraising/dash/node_modules] - https://gerrit.wikimedia.org/r/174479 (owner: Ejegg) [19:35:59] (CR) Ejegg: [C: 2] "Changing lib origin" [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/174481 (owner: Ejegg) [19:36:08] (Merged) jenkins-bot: Switch back to official passport-drupal [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/174481 (owner: Ejegg) [19:57:52] ejegg: just thought of a way to avoid the issue.. [19:58:46] Silly me, I was thinking that we had to choose a campaign before filtering for buckets so we could know which per-campaign bucket to choose/retrieve... [19:58:56] ok [19:59:44] So I had been (wrongly) thinking that we had to do some additional allocation only on the campaign level and then make sure the same random number would always land on a given campaign and on a banner in that campign [20:00:15] However, the solution is much simpler: when the client receives bannerChoiceData, just retrieve and choose buckets for _all_ the campaigns there right away [20:00:19] * AndyRussG facepalms [20:00:32] Then we can filter banners and process exactly as we're doing now [20:01:37] So, the issue becomes a non-issue, just a quirk in our current algorithm that'll remain constant on both client and server [20:01:44] Wait, what's the change you're proposing? [20:01:58] retrieve and choose buckets for all the campaigns right away, huh? [20:02:05] buckets stop being global and become per-campaign [20:02:22] So yeah, we get bannerChoiceData and there are a few campaigns in there [20:02:32] So choose or retrieve buckets for all of 'em [20:03:06] Then the initial filter pass to make the flat list of banners doesn't filter on global bucket, it filters on whatever bucket we have for each campaign [20:03:49] ok, but if a campaign-specific bucket is empty it would still filter all banners in that campaign out, right? [20:04:15] exactly, and that's what happens currently and what happened before any changes [20:04:39] So it'll be like a quirk, I think, but it doesn't need any resolution now... [20:04:52] oh, or are you saying we'd never assign a person to that bucket? [20:05:44] Mmm no, they'd be assigned a bucket for that campaign, and it might be a bucket with no banner, but if that happens the banner would be filtered out and whatever other campaigns are in there would eat up the vacated allocation, as happens now [20:06:15] oh, so you're just saying this issue is not a blocker to the per-campaign bucketing! [20:06:18] gothca [20:06:21] gotcha [20:06:23] exactly, yeah [20:06:52] I thought it had to be resolved somehow but now I think I was wrong... [20:08:48] fewer blockers is good! [20:33:23] yeah! [20:40:53] ejegg: heads up, at 11 PST all wikipedias are moving to our new banner code (with the feature still turned off)! [20:43:12] AndyRussG: isn't it 12:45 pst now? [20:44:50] oh, it was the 11-1 window. so they might already have it! [20:44:52] checking [20:44:55] no [20:45:24] Ah you're right bout the time [20:46:44] I checked and it wasn't there... [20:47:10] maybe it takes a while to filter thru [20:47:21] I think we have a cache or something [20:48:53] what's it called, paint thinner? [20:49:36] heehee [20:49:44] was it turpentine? [20:49:50] shellac? [20:50:43] ooh, I see it under debug [20:51:47] or at least in the resourceloader manifest [20:58:32] ejegg: hmm I'm not getting it yet on enwiki [21:02:23] Oh wait, never mind. We already had the .lib and ChoiceData listed in the manifest from the prev deploy [21:02:41] ejegg: which manifest are you talking about? [21:02:46] This one should add them to the dependencies for bannerController [21:03:07] The big JS array with dependencies and versions for each RL module [21:03:09] I don't see the lib loaded as a js source [21:03:12] Are hangouts broken right now, or is it just me? [21:03:18] right, it's not there [21:03:31] K4|fooding: I think we're skipping standup today [21:03:41] since we're doing a weekly prawning [21:03:42] also [21:04:00] Yeah, but I've got another thing. [21:04:05] ejegg: for my edification, where is this manifest of which you speak, please? [21:04:11] And then we have prawning. [21:04:44] AndyRussG: when I load with debug = true, it's the script that comes from this request: https://bits.wikimedia.org/en.wikipedia.org/load.php?debug=true&lang=en&modules=startup&only=scripts&skin=vector&* [21:05:23] in there, mw.loader.register is called with a huge array [21:05:48] lines 200-7277 [21:06:42] K4-713: I was just able to join fr-tech-daily, not that there's anyone else there to test with [21:07:10] I think it solved itself for me, too. Thanks for looking. [21:07:56] ejegg:right, I see! Hmm... thanks. How did you come upon that, BTW? [21:08:23] just searching all sources for ChoiceData using firebug [21:08:42] ah right [21:08:56] do you like firebug better than the built-in FF dev tools? [21:09:18] should I ask on ops what happened to 1.25wmf8? [21:09:18] (CR) Ejegg: "This CSS file is just loaded on the payment gateway pages, so we shouldn't need to restrict the selector. Our email fields have a consist" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/173435 (owner: Ejegg) [21:10:37] worth checking ops channel logs at least [21:16:34] Hmmmm [21:29:49] ejegg: _now_ I'm getting wmf8 on enwiki, together with bannercontroller.lib and the empty choiceData as expected. Mobile works great, banners are showing! [21:46:23] AndyRussG: great! [21:46:28] Yeah! [21:46:54] One small step for a deployer, a giant leap for bannerkind [21:49:28] K4-713: guess the temp table trick didn't work, and now we have 4 contribution messages on the damaged queue because inserts into civicrm_email timed out. I should resend those 4 messages via the civi interface, right? [21:51:05] ejegg: Yeah, that should work. [21:51:07] Also: Darn. [21:51:22] What went wrong? [21:52:12] K4-713: not sure, trying to correlate failmail timing with which query would have been running [21:52:40] Ooo. Fun. [21:52:53] hrmph, can't browse damaged message with a decimal point in them [21:53:00] wat [21:53:14] Have you tried looking at jenkins logs? [21:53:21] For the debugging, I mean. [21:53:36] unfortunately, i was running from the command line [21:53:42] d'oh! [21:54:06] going by timestamps on the running commentary in irc [21:54:08] heh [21:54:09] I'm a little surprised you could do that, actually. [21:55:39] Run from the command line, I mean. [21:56:16] yeah, ditto. Pretty convenient way to test alternate settings / code though [21:56:29] ejegg: In the future, it may be helpful to, ah... set up an unscheduled jenkins job to run things in verbose mode. [21:56:38] Yep, definitely [21:56:44] Then we get all the logs, and the permissions don't go funny if the process moves files or anything. [21:56:53] ah, that too! [21:57:10] ...says the main abuser of the command line resulting in funny permissions. [21:57:14] <-- [21:57:16] pretty sure I found the culprit [21:57:57]