[00:00:04] we have submodules to avoid checking the same code in multiple repos, but under vendor we're already duplicating tons of 3rd party repos [00:00:07] well, I was sort of okay with the situation in which composer managed everything correctly, except for the submodules that we wanted to manage manually. [00:00:26] The 3rd-party stuff seems fine [00:00:50] submodules weren't to avoid bloat, IMO, they were to make it simple to upgrade our libs [00:00:53] mmm [00:00:58] I think I'm starting to agree with you, though [00:01:03] actually, yeah. [00:01:08] please do! [00:01:12] but to update DonationInterface, I have to move the submodule out of the way, composer install the new version (to update autoload and .lock), then move the submodule back into place and update it to the new revision [00:01:30] heh, you convinced yrself before i could type my argument [00:02:00] I was imagining we wanted to work in the vendor/wikimedia/* dirs but of course not. We do the merge square dance elsewhere, and pull down the final product. [00:02:37] that totally works. [00:02:52] d'oh, I think I have to push one more change to DI's composer.json to make the WMFFramework happen [00:03:09] I thought that was already done? [00:03:25] ehh, no, I'll require the alias switcher file where we need it in crm [00:13:57] AndyRussG: why do the fixtures have to be removed while u refactor? [00:14:20] Also, do you have a use case in mind for the mock globals? [00:15:34] awight: well they were changing a lot, so I wanted to look at a clean slate for a while, but they'll all go back before this patch is out of WIP [00:15:42] The mock globals' use case is in there already [00:16:10] number of buckets? It's the same for all tests tho [00:16:16] It's cos we have to test all the available buckets and provide expected results for all [00:16:47] I'm fine with setting the globals, I only ask cos I'm gatekeeping a bit wrt complexity that might not get used [00:16:49] Shouldn't depend on the actual config value, easier if it's the same for all fixtures [00:17:05] No it's already used there... grep around and ull find it [00:17:07] right. But why not just set directly, outside the fixture data? [00:17:30] because it's tied to the fixture data. That's the value needed by this specific fixture data [00:17:37] Change it and you have to change the fixture data [00:17:55] do you plan to add fixture data that requires different settings? [00:18:15] Well, not yet :) [00:18:21] But we might! [00:18:36] And it's not much complexity added for a gain in clarity of what goes with what [00:18:47] * AndyRussG is sure it's all very very clear [00:18:51] jk [00:18:52] hehehe [00:19:08] ok no worries, if it might get used then it's totally rad [00:19:32] yeah grep and you'll see it, in both phpunit and qunt [00:19:44] qunit [00:20:47] O_O [00:21:03] I'll stop bothering u now :p [00:21:40] I do see it being read and used to set globals--my doubt was only whether we would be actually changing this per fixture [00:21:57] what tipped me off was that the key is at the top level of the file, rather than under the tests... [00:28:27] arrrggghh. vendor gitdir royally snafu'ed. Let's see if I can at least make it work the wrong way [00:32:04] (CR) Awight: "Great idea!" (10 comments) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188687 (owner: AndyRussG) [00:39:11] (PS1) Ejegg: Update DonationInterface and Smashpig [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/190393 [00:39:32] aww [00:40:14] no freaking clue how to actually fix the submodules [00:40:43] I mean, we can all rm -rf and clone fresh if it comes to that [00:41:00] hmm [00:41:07] well, let me try one more time [00:42:45] also: word on the street is that Linus hates submodules, hence the awful and broken implementation in git. [00:43:34] well, the linux kernel probably isn't checking in a whole lot of third party libs [00:44:08] I'm remembering a terrible, vague snippet--you need to use a weirdly bleeding-edge version of git in order to remove submoduleas [00:44:48] the version I've got /does/ manipulate .gitmodules for you when you git rm the path [00:44:58] what version? [00:45:08] I have 1.7.10.4 [00:45:11] 2.1.0 [00:45:18] oh. u got it then [00:45:23] weirdo ;) [00:45:41] the 3-d version [00:45:45] but there was still stuff in .git/modules [00:45:53] that was screwing everything up [00:48:57] I rsync'd too late... [00:48:59] sigh [00:50:31] (PS2) AndyRussG: Sample banner impressions client-side if no choiceData [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188394 (https://phabricator.wikimedia.org/T45250) (owner: Ejegg) [00:50:34] (CR) jenkins-bot: [V: -1] Sample banner impressions client-side if no choiceData [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188394 (https://phabricator.wikimedia.org/T45250) (owner: Ejegg) [00:51:37] well, it worked but I couldn't say what the steps were. I'll see if I can update in another dir [00:53:56] (PS1) Awight: WIP convert wikimedia stuff from submodules to vanilla dirs [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/190398 [00:54:27] Hrm? [00:54:59] I hacked off all the .git/modules cruft... just to see if we can do the merge on our dev boxen without The Pain. [00:55:20] yeah, but look at the diff for the wikimedia subdirs [00:55:23] -Subproject commit a1944f2c21dcc24d05b0846ff8e5c8176aeffda1 [00:55:24] +Subproject commit 34dd2fd6a22014fd4ff9a33ef8535da16677e2e9 [00:55:48] that's as far as I've gotten [00:55:53] yeah that's not good [00:55:56] it might take two commits [00:56:05] I also killed the lines in .git/config [00:56:05] and... I can't do the submodule rm with my stable git. [00:56:19] can you try committing just the submodule rm? [00:56:38] oooh. another clue: we have to run "git submodule sync" when doing the checkout [00:56:54] yep, I did that as 1 commit, now am trying to add and it only adds the top-level dir [00:57:06] thinking it's cause there's a .git file in each [00:57:17] can I see the rm? [00:57:20] so i've added */**/.git to .gitignore [00:57:26] yipes! [00:57:33] I don't think it's like that... [00:57:44] (PS1) Ejegg: Step 1: try to kill the submodules [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/190399 [00:57:46] I think this is fallout from the submodules [00:58:00] well, that looks fine [00:58:50] didn't work for me, either. [00:58:57] § Fundraising Sprint Devo, § Fundraising Dash: BUG: Fraud widget numbers don't seem right - https://phabricator.wikimedia.org/T87810#1035810 (CCogdill_WMF) @Ejegg thanks for the explanation! That's not at all how my query results were originally explained to me, so @PPena - the data I pulled for France, Israe... [00:59:41] I did: rm -rf vendor && git checkout vendor && cd vendor && git submodule sync (no submodules) && git review -d 190399 && cd .. && git submodule update -i --recursive [00:59:45] * awight gasps [01:00:02] I think they're being stored in the crm/.git dir [01:00:20] this is interesting: Submodule 'DonationInterface' (https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm/vendor) registered for path 'wikimedia/DonationInterface' [01:00:23] Submodule 'wikimedia/SmashPig' (https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm/vendor) registered for path 'wikimedia/SmashPig' [01:00:34] the "wikimedia/" prefix is missing from the DI submodule [01:00:46] oh and the URLs are hosed. [01:00:48] weird [01:01:23] oooh [01:01:27] you mean in crm/.git/modules/vendor? [01:01:36] it was my submodule update recursive that caused them to be recreated [01:01:39] ejegg: yes [01:02:11] I think your commit works [01:02:16] yeah, I tried submodule deinit vendor, rm -rf vendor, then a non-recursive update [01:02:24] hmm? [01:02:32] but did you git review -d 190399? [01:03:29] (PS3) AndyRussG: Sample banner impressions client-side if no choiceData [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188394 (https://phabricator.wikimedia.org/T45250) (owner: Ejegg) [01:03:31] (CR) jenkins-bot: [V: -1] Sample banner impressions client-side if no choiceData [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188394 (https://phabricator.wikimedia.org/T45250) (owner: Ejegg) [01:04:43] (PS1) Awight: WIP testing whether top-level submodule purge helps [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/190401 [01:07:27] sigh. [01:07:28] git submodule update -i --recursive [01:07:32] Submodule 'civicrm' () registered for path 'civicrm' [01:07:34] Submodule 'drupal' () registered for path 'drupal' [01:07:37] Submodule 'vendor' () registered for path 'vendor' [01:07:39] warning: unable to rmdir wikimedia/DonationInterface: Directory not empty [01:07:42] warning: unable to rmdir wikimedia/SmashPig: Directory not empty [01:07:45] Submodule path 'vendor': checked out '40df8782f1a4567b40d43208c7c9ee12d4898c27 [01:07:47] Hmm [01:11:30] yeah, the submodule hangover is passed, but I still can't add DonationInterface and Smashpig as normal subdirs unless I move their .git folders [01:14:37] (PS4) AndyRussG: Sample banner impressions client-side if no choiceData [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188394 (https://phabricator.wikimedia.org/T45250) (owner: Ejegg) [01:15:14] (CR) AndyRussG: "Rebased" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188394 (https://phabricator.wikimedia.org/T45250) (owner: Ejegg) [01:15:16] (CR) jenkins-bot: [V: -1] Sample banner impressions client-side if no choiceData [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188394 (https://phabricator.wikimedia.org/T45250) (owner: Ejegg) [01:16:52] (PS5) AndyRussG: Sample banner impressions client-side if no choiceData [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188394 (https://phabricator.wikimedia.org/T45250) (owner: Ejegg) [01:17:14] (CR) AndyRussG: "Fixed silly rebase error" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188394 (https://phabricator.wikimedia.org/T45250) (owner: Ejegg) [01:22:05] MediaWiki-extensions-CentralNotice, Wikimedia-Fundraising: Special:RecordImpression should die in a fire - https://phabricator.wikimedia.org/T45250#1035861 (AndyRussG) As per discussions with FR, I just amended Elliot's [[ https://gerrit.wikimedia.org/r/#/c/188394/ | CN patch for review ]] to only downsample... [01:22:15] Got 2 run! [01:34:39] Fundraising-Backlog, § Fundraising Sprint E: Change errors on credit card forms from popup to red text - https://phabricator.wikimedia.org/T86214#1035891 (CCogdill_WMF) [03:10:17] § Fundraising Sprint Devo, § Fundraising Dash: BUG: Fraud widget numbers don't seem right - https://phabricator.wikimedia.org/T87810#1036004 (atgo) So @ppena does this work for you? Should we update the widget to display the same results as Caitlin's query? [03:24:50] § Fundraising Sprint Devo, § Fundraising Dash: BUG: Fraud widget numbers don't seem right - https://phabricator.wikimedia.org/T87810#1036008 (CCogdill_WMF) @atgo I don't think we should replace the fraud widget; that is correct as it is. It would just be nice to have a separate rejection widget :) Le jeudi 1... [03:35:10] MediaWiki-extensions-CentralNotice, Wikimedia-Fundraising: Special:RecordImpression should die in a fire - https://phabricator.wikimedia.org/T45250#1036011 (AndyRussG) I should add the caveat that it won't be an improvement at times like now, when there are campaigns targeting all projects and all languages.... [09:14:48] Wikimedia-Fundraising, Wikimedia-Fundraising-CiviCRM: Allow mismatched refunds - https://phabricator.wikimedia.org/T89437#1036231 (awight) NEW [09:49:44] hi jessicarobell, which desktop banner do you want to test first in Sweden: remind me later or the open amount field? [09:51:08] I think the open amount field is a better first test, to try for enSE. Thank you the-wub! [09:51:22] ok thanks [16:19:42] AndyRussG: where is that dependency injection extension? :( [16:20:23] It's already been the thing to use in a dozen places! yet unattainable... [16:21:07] awight: from my perspective, I guess it's lost in backlog! [16:22:05] right below a bunch of other things like an EP RFC, Enums (which I'd like to improve a bit before rolling out Editor Campaigns) and the data mapper :( [16:22:52] Ho hmm so much technical debt and so little income [16:22:59] But you're right, the interest rates are kiling us [16:24:00] BTW Andrew Otto was having a look at our hhvm log mystery and posted on Phab [16:25:18] Yeah that was good to see. [16:25:27] What do you think? [16:25:52] Yeah that was nice [16:26:01] I think it's ops's problem now, basically [16:26:32] If nobody can say for certain why the error seems to be repeated, they should certainly investigate IMO. [16:26:46] Yeah... I mean, I don't want the issue to be sidelined because no one cares and then the site to explode because no one cares [16:26:54] I agree that if they aren't curious, I'll just stand here and skullscratch a bit. [16:27:28] I can understand they have bigger fish to fry, but it's out of our hands because the mystery and potential consequences are on their turf [16:27:57] Unless, of course, it turns out to be our code that is magically calling SpecialBannerRandom [16:28:31] But assuming it's not, then they're the ones who can investigate and would take responsibility for any ill consequences of not doing so [16:29:15] So I'm like, either wait until they investigate and tell us what's up, or until they say they won't investigate and tell us to go ahead and kill that class [16:29:41] That way at least we won't get 503s because somebody else's problem [16:29:55] Does that make sense? [16:30:36] (back in a few minutes, then I'm out on a short errand and back again, I'll get backscroll tho!) [16:30:46] (P.S. Hope Mari feels better!) [16:31:29] that totally sounds right. donno if you want to explain as much in the task [16:31:49] awight: I did say as much to Andrew [16:32:03] :) ok sorry, I missed that comment. Thanks! [16:33:09] np, thanks also! :) [16:33:22] have fun bassooning! [17:33:45] Hehe, I think the octupus broke grrrit-wm [17:33:55] awight: Nice opener. [17:34:01] * awight tips hat [17:34:41] * awight shepherds octopus back into its tank with a jagged broom handle [17:35:09] is grrrit-wm who broadcasts our patches as we post for review? [17:35:19] cuz I just pushed one and don't see an announcement [17:35:32] o_O [17:35:42] it quit the channel a second ago, maybe our furious onslaught of patches damaged its feelings [17:36:05] "Ow, my feeling! ...That was my only copy." [17:36:09] octopi are pretty sensitive. [17:36:19] haha [17:36:24] * awight inadvertently changes color [17:36:54] would an octopus wear eight pants, or four? [17:37:45] * K4-713 squints [17:38:03] * awight one more question about the tech lead scheme you drunkenly shared last night. [17:38:06] aargh [17:38:23] Maybe you just call octopus pants a "bunch" instead of a "pair". [17:39:48] Also, I wasn't that drunk. [17:39:50] :p [17:39:52] Baahaha [17:39:56] FOR THE RECORD [17:40:18] I'm just saying: It's a wide spectrum. [17:40:19] K4-713: hey, do you have time to CR a bit? [17:40:24] * K4-713 sighs [17:40:27] Nope. [17:40:30] ok np [17:40:43] * awight eyes various other busy people [17:40:49] Seems indicative of... something. [17:42:00] K4-713: well, don't worry about a full CR but does this make sense IYO? https://gerrit.wikimedia.org/r/#/c/190484/ [17:42:17] * K4-713 takes large swig of coffee,;clicks [17:42:28] Oh, haha [17:42:35] cos: [17:42:49] So... the escaping bit. [17:42:54] it causes slight but permanent brain damage to anyone we might onboard :) [17:42:57] yes [17:43:10] I was thinking, escaping should be done at the point of use. But that's a religious thing [17:43:16] I have mild urges to travel back in time and smack myself for doing that. [17:43:21] Yes, exactly. [17:43:25] ok [17:43:28] Exactly that thing. [17:43:30] I'll work towards that [17:43:42] You can call it "normalized" if you change where the escaping happens. [17:43:43] For now, I think we can ignore "escaped" cos it might be a lie [17:43:49] Nah, it's in DD. [17:43:56] OK well that's reassuring at least [17:44:07] I wouldn't lie about something like that. [17:44:10] Probably. [17:44:18] But if DD::$normalized escaped, let's just call that "normalized" for now [17:44:21] That assumes, of course, that it's still there. [17:44:55] if there's a place in the code where we actually NEED escaped data, I'll add a big red comment that says "this has already been escaped to tell your friend to chill" [17:45:08] Hurm. [17:45:25] I just don't like it. Any of it. At all. [17:45:42] But, I could like it. [17:45:51] I could like it again. [17:46:30] Also, the old name doesn't really say what exactly it's been escaped *for*. [17:46:46] The answer, is basically "forms". [17:47:30] ah. I think that will be done transparently by any modern template renderer [17:48:03] So, we probably need to change that anyway. [17:48:08] You normally have to say "{{ varname | noescape_cos_i_like_to_die }} [17:48:21] Unless you like double-escaping. [17:48:24] right [17:48:36] I love it. [17:49:00] kthx [17:49:10] Call me in a week and you can do the CR, still :p [17:49:18] * K4-713 nods gravely [17:50:47] Aw, nuts. [17:50:51] I have run out of time. [17:51:04] * K4-713 vanishes [18:31:48] § Fundraising Tech Backlog, Wikimedia-Fundraising, MediaWiki-extensions-DonationInterface: Expiration date error message is wrong when date < now - https://phabricator.wikimedia.org/T89388#1037184 (CCogdill_WMF) This isn't a part of her doc; I actually didn't have her go through the Worldpay flow. We do have... [18:49:01] § Fundraising Tech Backlog, Wikimedia-Fundraising, MediaWiki-extensions-DonationInterface: Expiration date error message is wrong when date < now - https://phabricator.wikimedia.org/T89388#1037216 (awight) I'd love to see this doc! Just a small suggestion for the UX discussion, it's not technically difficult... [18:55:38] Wikimedia-Fundraising-CiviCRM, § Fundraising Sprint Devo: BUG: Undo a recent merge of two contact records in Civi - https://phabricator.wikimedia.org/T89408#1037249 (awight) I think we have to recover the anonymous contribution IDs from a backup, then do an update in our database to: a) Undelete the anonymous... [18:57:36] Wikimedia-Fundraising-CiviCRM, § Fundraising Sprint Devo: BUG: Undo a recent merge of two contact records in Civi - https://phabricator.wikimedia.org/T89408#1037262 (awight) [19:03:15] awight: thanks for the CR of the samplepatch! As well as improving following your comments, I'm gonna re-submit with the more agressive version, which will turn on sampling if there are no campaigns after client-side filtering, too (following my afterthoughts on the Phab task, https://phabricator.wikimedia.org/T45250) [19:10:27] AndyRussG: interesting, sounds great to me! [19:11:20] awight: K thanks! [19:12:49] K4-713: hi, how's it going? If u have a bit of time today, there are a couple issues floating about that I'd be quite happy to have ur input on :) thx in advance! [19:13:16] Time is currently the one thing I am having a serious problem with. [19:13:21] But, what's up? [19:17:36] K4-713: briefly... :) (1) It'd be nice to decide ASAP what to do as a temporary measure for https://phabricator.wikimedia.org/T45250 [19:17:52] This is turning down RecordImpressions whenever possible... [19:17:55] Hum. [19:18:09] I was reading the thread, and it looked like you had a temp patch all ready to go. [19:18:11] Is that not the case? [19:18:12] I'm about to submit a more agressive version of the patch, as mentioned in my last comment on the thread [19:18:15] Yeah [19:18:21] Just wanted to check in that that's a go [19:18:29] Yeah, I don't see any reason not to. [19:18:35] Cool! :) thanks [19:18:59] But, I maintain that we need to get everybody in the same place so we can figure out where the misfire is. [19:19:20] Which misfire? [19:19:21] It seems like all the things that you guys wanted to do to address the problem, were also unacceptable. [19:19:37] Ah right [19:19:44] This tells me that we don't have a clear understanding about what's ultimately needed. [19:19:56] Yeah in the latest patch and the one I'm about to send up Ellery's and Megan's concerns are addressed [19:20:19] Yeah, I was about to say, that seems like a decent start. [19:20:19] This shouldn't affect any of the data they're actually using, since it just turns things down when there are no campaigns [19:20:42] I'm kind of... astonished that wasn't the way it worked before. [19:20:43] Ops has provided a few moments of clarity, I think--mostly, they want us to stop loading the clients. Server load is not a do-or-die thing. [19:20:53] We will still have to find something better medium-term, and yeah that does require a bit more detailed study of what's needed [19:21:00] err, wrong idiom but u know what I mean... [19:21:24] If the extra request is slowing down page render, it's totally unacceptable. [19:21:44] It's surely making it less responsive especially on some platforms [19:21:52] We're not sure that's the case--anecdotally it looked okay to me, but Ori thinks this request is in the critical path, and delays onLoad. [19:21:55] Also, from my perspective, this is perfect timing to open the door to talk about Ellery's plans going forward, and why they may be too... brutal? [19:22:06] (conveniently the opinion I had to start with) [19:22:09] AndyRussG: that sounds right! [19:22:11] K4-713: lol [19:22:22] I'll also ask someone for some immediate varnish tuning, BTW (that was shunned earlier in the thread along the lines of "let's get to the root cause instead", but I think it's a fine temporarymeasure) [19:22:41] There's no reason for PHP to be handling this another second [19:23:00] Yep. [19:23:04] AndyRussG: That sounds great, but again it's the client side that really seems to be bringing on the bile [19:23:25] awight: yeah! And also, agreed, it's good they're worried about that :) [19:23:32] So, just for my own... no, sanity is the wrong word. [19:23:48] Mmm? [19:23:52] Just for my own [approximation of sanity] - when do you want to deploy that patch? [19:23:59] Monday [19:24:02] okay [19:24:08] And they know it's Monday? [19:24:09] Oh wait, is that a holiday? [19:24:19] Tuesday [19:24:21] Office holiday, yes. [19:24:25] Okay. [19:24:36] Would you make sure everybody knows that's the plan? [19:24:44] OK u bet! [19:24:49] Thanks. [19:24:50] thanks :) [19:24:55] Mmmm issue #2.... [19:25:01] There's more? [19:25:06] Okay, let 'er rip. [19:25:08] Yeah that was number one [19:25:17] https://phabricator.wikimedia.org/T89258 [19:25:28] So, long story slightly less long: [19:25:37] Wikimedia-Fundraising, MediaWiki-extensions-DonationInterface, § Fundraising Sprint Devo: Tech Review of LATAM processor - https://phabricator.wikimedia.org/T87046#1037402 (awight) stalled>Resolved [19:25:39] * K4-713 gets popcorn [19:25:55] The deploy last Tuesday kinda broke some dead code that we didn't even care about breaking 'cause it's... dead [19:26:10] But then it started causing errors in the hhvm.log... like, a lot! [19:26:18] Can we roll it into the nearest ditch? [19:26:27] Well, yes, but... [19:26:35] The errors are not fully explained [19:26:51] They are partly explained by 30k/day incoming requests that trigger the dead code [19:27:01] Right, I get that. [19:27:03] * awight lets albatross fester [19:27:07] But, that might not even be us. [19:27:32] Yeah, it's surely some eternally cached JS... But are in fact a million more errors in the logs than can be accounted for by the 30k/day requests [19:27:52] So, I've been bugging Ops and others to figure out why that is [19:27:58] Okay, so why *wouldn't* we roll it in to the nearest ditch? [19:28:14] Yeah also in my last comment on that thread... [19:28:30] "What if whatever is calling this broken code then breaks worse and breaks other stuff? Or what if this is a symptom of some unexpected behavior that needs to be uncovered, and removing the code for Special:BannerRandom just masks it? I confess I'd feel happier and cozier getting to the bottom of this before deploying such a change." [19:28:48] So basically I'd like to add one more comment on that thread [19:29:09] Saying that we'll go ahead and remove the code, but if shit breaks because of it, it's on Ops's shoulders [19:29:20] (in slightly more diplomatic wording) [19:29:38] I mean, if it were /my/ cluster and that were happening, I'd definitely want to know why [19:29:41] Hurm. [19:29:50] But as FR we don't have the means. And it's clearly and ops issue [19:29:58] What if you change the dead endpoint to do nothing but log something mildly informative about wth is going on? [19:30:26] K4-713: well yes, in theory [19:30:59] But it's not just Web entry point... It's that I don't have a backtrace that tells me what is calling it [19:31:10] So it'd somewhat hypothetical is all [19:31:30] I mean, I have _one_ backtrace, that points to a legit request that triggers this code [19:31:31] we have one backtrace... that says it's a normal web request from the ghostly legacy code [19:31:34] hehe [19:31:40] * awight goes back to popcorn [19:32:01] But what I don't have is a sample of backtraces to know that all the calls to that code are due to requests [19:32:19] Which in fact would be weird, because if that's the case they're not making it into the request logs [19:32:44] A theory would be that there are _way_ too many retries being made by varnish when it gets a 503 [19:32:59] hum [19:33:08] Another theory is that some other code in core or another extension is somehow getting there [19:33:33] Yet another theory, I suppose the most likely, is that I'm stupidly missing something totally obvious [19:33:33] Well, normally, I'd say let's just roll it into a ditch in a way that's trivial to revert. [19:33:50] Mmmm [19:33:58] With informative logging that would tell you if something deeply alarming was still going on. [19:34:14] OK, I think we can try that :) [19:34:20] I mean, I don't object [19:34:28] I'm not sure what we'd be hoping to learn from holding off. [19:34:53] Yeah... but I just wouldn't like to see a broken site and everyone pointing fingers and saying, it wasn't my problem [19:34:56] And, if we're clogging the error logs... learning is down all over. [19:35:03] Yeah also true [19:36:21] So I guess we could try that, again I just wanted to add to the thread that the decision is ops's, but I wouldn't want to go saying that without checking in :) [19:37:17] I dunno, maybe I'm taking the wrong approach/attitude [19:38:01] § Fundraising Dash, § Fundraising Sprint Devo: BUG: Fraud widget numbers don't seem right - https://phabricator.wikimedia.org/T87810#1037430 (CCogdill_WMF) I just checked in with Pats about this. We want to make sure we're understanding everything correctly, and if so, make a few requests. As I understand it... [19:38:15] AndyRussG: It seems to me that they already made their preferences known. [19:38:51] K4-713: OK.... sounds good! [19:38:55] Ha, more related emails reinforcing this premise. [19:39:32] hmm? [19:39:50] Went to Engineering. [19:40:02] Hum [19:40:25] greg-g wants us to stop emitting errors. :) [19:40:58] K4-713: OK we can do a small and simple patch for just that [19:41:10] and I think even deploy if that's OK [19:41:34] AndyRussG: Great. Just make sure you can unwind it just as simply in case it's more insidious than we thought. [19:42:24] OK [19:42:37] K4-713: So you're OK w/ a Friday deploy in this case, I guess [19:42:50] K4-713: mmm. point of interest, I've already written a ridiculous patch which destroys all of the legacy behavior. [19:43:06] K4-713: :) [19:43:07] It's not what you describe, I don't suggest unwinding it. [19:43:11] awight: I can't stop looking at the word "ridiculous" in that statement. [19:43:38] WEll... https://gerrit.wikimedia.org/r/#/c/190239/ [19:43:40] I got carried away. [19:44:13] It's stuff that has to be done eventually, so rather than pull on single threads which each might cause minor issues, I took the whole sweater off. [19:44:29] I see. [19:44:44] I mean, it can be rolled back if deployment fails, but I don't suggest doing so after patches go on top of it. [19:44:53] * K4-713 raises eyebrow [19:45:16] IMO, the only thing we have to lose is that potentially, 30k/day of people getting old banner allocations now get nothing. [19:45:32] awight: If you can convince AndyRussG to deploy it... [19:45:37] AndyRussG: please temper with your more sane approach? [19:45:41] :D [19:45:52] awight: no [19:45:56] :p [19:45:58] oh dear [19:46:04] then we're lost ;) [19:46:10] awight: K4-713: greg-g: I'm making a patch to just bow out on Special:BannerRandom::execute() [19:46:26] It's simple, definitely won't break anything else, etc etc [19:46:28] Sound good? [19:46:43] of course! [19:46:47] K :) thanks! [19:46:48] just bow out? [19:46:49] Woah. [19:46:53] We agree on a thing. [19:47:04] greg-g: "return;" i believe [19:47:08] :) [19:47:11] What's the fastest way to bow out in PHP/MW? [19:47:27] Get a job at Uber. [19:47:29] Just that? Not send some code like doesn't exist or something [19:47:31] baaahahaha [19:47:39] :( [19:47:55] IMO we don't want to spam 404s either? [19:48:07] +95/-745, wow [19:48:10] We could add a hard deprecation phase in later if we wish [19:48:21] * awight glares [19:48:27] can you test on beta cluster? [19:48:45] greg-g: I think AndyRussG is suggesting a one-liner to obviate my patchturd [19:48:58] http://en.wikipedia.org/wiki/List_of_HTTP_status_codes how about 204 deleted? [19:49:07] awight: it's not a patchturd! [19:49:07] awight: :) :) [19:49:41] Well, I can hardly label it surgically precise :p considering that I caused this snafu in the first place! [19:50:56] what's the mediawikiway of sending a 204 response? [19:51:08] I think my feelings of guilt drove me to lay the baroque patchturd. [19:51:52] lol, guilt -> indigestion? [19:52:09] usually [19:52:13] if you're a good catholic, at least [19:52:48] Doesn't Phabricator have like a place for quotes like bugzilla did? [19:53:02] haha, someone is working on it (of course) [19:53:09] * greg-g goe sto get lunch [19:53:20] Oh hey yeah. [19:53:23] Lunch. [19:53:38] AndyRussG: if you must :p https://office.wikimedia.org/wiki/Bash [19:53:55] Wouldn't be the first time. [19:54:08] awight: I just noticed, though, that yours have a sort of theme. [19:54:09] aww man that's a good one from Feb [19:54:26] K4-713: :( one day I might progress beyond the scatophilia [19:57:06] Wikimedia-Fundraising-CiviCRM, § Fundraising Sprint Devo: BUG: Gateway Reconciliation Report not pulling all donations - https://phabricator.wikimedia.org/T89288#1037467 (awight) @RLewis, can you give more details about the report you run? I assume the discrepancy is something related to soft credits, but I'... [19:58:36] AndyRussG: looks like, $this->getOutput()->setStatusCode ; but I'd be curious to hear from greg-g or other opsen which status codes will not be annoying to them. [19:58:52] ask ops [19:59:12] K [19:59:49] awight: looks good! [20:00:31] hey K4-713 i'm ravenous... want to go for noms? [20:05:53] Ahrg silly me, I misread what 204 meant [20:06:38] yeah 410 seems appropriate [20:06:39] § Fundraising Sprint E, Fundraising-Backlog: Error-specific messaging on "transaction could not be completed" page - https://phabricator.wikimedia.org/T89371#1037478 (CCogdill_WMF) [20:06:51] glad to hear it won't cause new, different spam issues! [20:12:02] Wikimedia-Fundraising-CiviCRM, § Fundraising Sprint Devo: BUG: Gateway Reconciliation Report not pulling all donations - https://phabricator.wikimedia.org/T89288#1037520 (RLewis) @awight I don't actually run a specific report. I use the find contributions search fun in Civi itself. See the attached screensho... [20:14:15] (Abandoned) Awight: WIP fix and continuation of previous patch [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/185603 (owner: Awight) [20:15:07] Wikimedia-Fundraising-CiviCRM, § Fundraising Sprint Devo: BUG: Gateway Reconciliation Report not pulling all donations - https://phabricator.wikimedia.org/T89288#1037528 (awight) @RLewis: ok, thanks that gives me enough to work with! [20:16:18] Wikimedia-Fundraising-CiviCRM, § Fundraising Sprint Devo: BUG: Gateway Reconciliation Report not pulling all donations - https://phabricator.wikimedia.org/T89288#1032444 (awight) [20:22:47] awight: hrm it still sends all the skin stuff [20:30:59] (PS1) AndyRussG: Deprecate Special:BannerRandom and return a 410 [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/190506 (https://phabricator.wikimedia.org/T89258) [20:31:14] awight: ^ [20:31:21] Jeff_Green: ^ [20:33:44] awight: wanna book a deploy today? :) If you have time... I trust your deploy-fu here about a million times more than mine... [20:34:32] awight: and so I guess we might as well deploy S:RI too... [20:34:54] AndyRussG: Mmm, tough call. Making it an exciting Friday vs spamming the logs for the weekend... [20:35:05] Monday is off the table? [20:35:41] Well, since K4|away got an e-mail escalating the issue... [20:36:12] * awight backscrolls [20:37:30] AndyRussG: what happened to setStatusCode btw? [20:38:06] awight: I was still getting all the skin, i.e. sidebar, bla bla bla [20:38:24] I copied thumb.php [20:38:31] AndyRussG: greg-g's email is a pretty mild escalation, but since your patch is much safer, we might as well give it the old Friday boogie! [20:38:56] awight: If u want we can wait 4 a K4|away stamp-o-approval [20:39:31] I'm doing another patch to log any other execution of this heinous codepath [20:40:21] How about we book 1 hour from now, and try to get S:RI in too? It's quite simple too [20:40:32] AndyRussG: for that, see if you can dig into the X-cache headers to identify the varnish server tormenting us [20:40:37] AndyRussG: sure [20:42:05] awight: OK! [20:43:03] or... I don't know if there's any way to identify the original web request? [20:43:09] to confirm the retry theory? [20:45:22] Hmm [20:46:34] AndyRussG: creepy, I see what you mean about setStatusCode. You have to execute OutputPage::output() and the skin is barfed out, etc. That looks buglike. [20:50:23] (CR) Awight: [C: 2] "Works as intended!" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/190506 (https://phabricator.wikimedia.org/T89258) (owner: AndyRussG) [20:51:28] AndyRussG: S:RI seems sketchier to Fri-deploy, cos it has potential side effects that will mess with ewulczyn and meganhernandez's analytics. [20:51:46] (Merged) jenkins-bot: Deprecate Special:BannerRandom and return a 410 [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/190506 (https://phabricator.wikimedia.org/T89258) (owner: AndyRussG) [20:51:56] AndyRussG: I'll note a medium-grade issue on that patch, too... [20:52:25] awight: K. If you're being cautious... [20:52:47] awight: issue on which patch, S:RI or S:BR? [20:53:25] (CR) Awight: "another thing, not strictly necessary before deployment, but shouldn't this patch be coordinated with an analytics patch which expects and" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188394 (https://phabricator.wikimedia.org/T45250) (owner: Ejegg) [20:53:31] that one:) [20:54:13] AndyRussG: it's also fine to just tell ewulczyn that numbers will be off until he fixes the analytics scripts [20:54:18] I think... [20:55:40] awight: supposedly it won't touch the numbers he uses... [20:56:55] wat. [20:56:59] okay, that's fair [20:57:14] They aren't making big decisions based on hide:empty :) [20:57:28] Did you want to tweak that patch before merge? [20:59:45] I'm fine with merging as-is, btw, my js suggestion is just a "nice to have". [21:00:53] awight: yeah I was halfway thru tweaking, I'll finish that [21:01:42] Right now I'm on what to do with wfLogWarning( $this->getRequest()->getAllHeaders() ), should I try to output a warning with everything in there as a string? [21:02:31] awight: are we doing standup? I imagine not... [21:03:48] looks like we don't have a quorum [21:04:22] AndyRussG: whoa. for instrumenting S:BR ? [21:04:36] Yeah [21:04:46] I'm sure there will still be requests on Monday... [21:04:59] awight: This is on top of the previous pach [21:05:02] patch [21:05:30] It'll stop filling up the hhmv log, unless that goes to the same place... [21:05:32] Totally up to you, if you want to go for it and can do both patches. [21:06:00] ergh, I guess there is no debug logging on production. [21:06:03] that's awkward. [21:06:20] awight: I'm thinking it could be a quick deploy and undeploy [21:06:22] Maybe ask ops if that's any less spammy. [21:09:10] AndyRussG: I don't see our deployment window on https://wikitech.wikimedia.org/wiki/Deployments ... [21:09:22] Luckily, there isn't much competition :) [21:10:46] awight: right, one sec... [21:12:40] awight: 2pm PST? [21:14:38] cool [21:15:05] don't stress yourself out :) IMO the S:BR logging is much lower priority than the other stuff. [21:21:21] K4-713: we're deploing at 2pm ur time. I'm thinking, might as well throw in Special:BannerRandom, but awight (!) calls for caution [21:22:08] Well, if awight is covering the break... :p [21:22:27] Yeah, I would prefer separating the issues, too. [21:22:32] I mean, generally. [21:22:43] But particularly when there are mysteries. [21:24:40] AndyRussG: wait, fixing S:BR the main point of our deploy, I thought. But slowing down RecordImpression seems like a acceptable rider, if I'm watching for total borkage anyway. [21:25:20] Extra debug logging should be more of a Mon or even Tue thing, perhaps. [21:25:36] AH yes sorry I got it backwards [21:25:41] Aha [21:25:45] awight: K4-713: yeah the rider would be S:RI [21:25:53] Apparently we will have a steady supply of these evilly multiplying requests available for all of time, now. [21:26:04] Since... we can do it, and we're dploying anyway [21:26:07] awight: which ones? [21:26:10] Yeah, well, okay. General policy on riders: "You break it, you bought it." [21:26:23] the Special:BannerRandom hits [21:26:39] K4-713: K that's also getting quoted on office? [21:26:47] Please do. [21:26:48] :) [21:26:50] they have to be entombed in concrete or something, and even then we'll still see leakage in a few years. [21:27:01] awight: that's so nuts [21:27:13] yeah I'm sad at what I referenced. [21:27:26] If buying it on a Friday isn't a deterrent, please have at it. [21:27:28] :D [21:28:09] It... sort of is. The rumor is that greg-g knows people who will want us trout-slapped if this persists over the weekend [21:28:20] mostly me [21:28:22] :P [21:28:25] s/[.]/. But,/ [21:28:28] That sounds like the opposite of what I was saying. [21:28:41] I forgot... a not. [21:28:56] * awight rubs hands for the deploy :p [21:28:57] First part? [21:29:02] Let me try again [21:29:12] Friday is as deterrent as always [21:29:14] This isn't inspiring confidence. :p [21:29:22] however, what we are doing is particularly nasty [21:29:29] and it's causing kittens to secretly cry [21:29:31] So [21:29:46] * K4-713 looks at business cards to see if there's room for "what we are doing is particularly nasty" [21:29:53] I'm totally onboard with helping AndyRussG deploy the small patch to turn off Special:BannerRandom [21:30:00] I will buy the farm if it goes under [21:30:06] I'll cosign. [21:30:12] PRobably also tangent. [21:30:15] RAR [21:30:19] (/me sees he has accidentally deployed the mixed metaphor trigger) [21:30:28] finally, there are two riders. [21:30:34] I'm okay with the RecordImpression slowdown [21:30:43] Yeah, and as long as we're in there. [21:30:49] Okay, do it. [21:31:07] but trying to excersize a veto over any more screwing with Special:BannerRandom to do debug logging, unless of course we can deploy my magnum opus and just spend the rest of the day wallowing in regret [21:31:17] ok [21:31:18] :) [21:31:34] You misspelled "holiday weekend". [21:31:38] <_< [21:31:40] fuck yeah. [21:31:50] * AndyRussG is ...........lost..... [21:32:01] but laughing anyway [21:32:05] Caus the president [21:32:11] doesn't know a single resident / in my hood [21:32:27] I forgot joke comprehensibility takes a nosedive on Friday afternoons [21:32:35] I have this theory that working in FR-tech causes you to speak a language that doesn't exist anywhere else. It's sort of like that Darmok episode of Star Trek. [21:33:20] K4-713: if there's not room on the business card, you've got those JDs :P [21:33:28] http://genius.com/373762/Goodie-mob-inshallah/In-a-country-run-by-a-president-that-doesnt-know-a-single-resident-in-my-hood [21:34:26] That whole album is worth listening to [21:34:31] :) [21:35:01] (PS1) AndyRussG: Warn and die at unexplained SpecialBannerRandom execution [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/190559 (https://phabricator.wikimedia.org/T89258) [21:36:50] AndyRussG: do you still want ^ ? [21:37:05] awight: I think so, just smoke testing :) [21:37:20] If we do it, we should have the revert already prepared [21:37:35] (CR) Awight: [C: 1] "I like how this is done. But still discussing on IRC." [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/190559 (https://phabricator.wikimedia.org/T89258) (owner: AndyRussG) [21:37:48] awight: Hmm, lemme reorder tho, we should put the S:RI first [21:38:04] AndyRussG: smoke testing will not tell you if we're going to cause chronic mild distress to Ops over the weekend... [21:38:27] Please refrain from killing ops. [21:38:40] Even if you do it mildly. [21:38:47] Just... ulcers? [21:39:36] awight: K4-713: yeah that patch is just for deploying and reverting right away -- _if_ there are still hhvm.log messages after the previous patch to turn off S:BR [21:40:34] okay, I'm happy to livehack [21:40:40] § Fundraising Dash, § Fundraising Sprint Devo: BUG: Fraud widget numbers don't seem right - https://phabricator.wikimedia.org/T87810#1037779 (atgo) Hey @ccogdill_wmf - thanks for the detailed feedback. I'm going to leave tech (@ejegg?) to answer your first question and the question about the numbers being off... [21:40:42] (via merge and revert) [21:44:07] AndyRussG: btw, thank you for your persistence in digging to the bottom of the somewhat rotten foundations around here!! [21:45:38] awight: heh it may be the end of me, I suppose [21:45:46] hope not [21:46:15] AndyRussG: I guess ^d just nailed it as usual. [21:46:56] We're already pooling cash to get him a bottle of something. [21:52:23] (PS2) AndyRussG: Warn and die at unexplained SpecialBannerRandom execution [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/190559 (https://phabricator.wikimedia.org/T89258) [21:58:17] AndyRussG: I'll prepare the $wgDebugLogGroups config [21:58:20] (PS6) AndyRussG: Sample banner impressions client-side if no choiceData [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188394 (https://phabricator.wikimedia.org/T45250) (owner: Ejegg) [21:58:52] awight: I guess I didn't understand that bit [21:59:04] awight: if it's complicated and it's only one header we could just put it in the xception message [21:59:10] I think that's easier [21:59:26] What I hears was that we could even leave this instrumentation in for the weekend, if we're just dumping to our own file. [21:59:55] The multiplied requests thing was not happening every time, AFAIR? [22:00:15] and I would want to clean up any more error logging before our deploy window closes... we might not catch one? [22:00:19] awight: it was happening all day every day [22:00:21] is [22:00:41] every request? I thought some only caused one backend execution [22:00:52] some <-> only [22:01:01] eugh I can't even read that sentence [22:01:01] There are a million more hhmv.log errors than apparent requests to S:BR [22:01:19] OK [22:01:33] So yea let's not leave it up :) [22:01:59] can we still go to a file? [22:02:11] it's just, wfDebugLog( 'groupname', msg), I believe [22:03:25] Wikimedia-Fundraising-CiviCRM, § Fundraising Sprint Devo: BUG: Gateway Reconciliation Report not pulling all donations - https://phabricator.wikimedia.org/T89288#1037877 (awight) This turns out to be very simple, the gateway theory in the task description is correct: hand-entered contributions have "Contribut... [22:07:50] AndyRussG: oh good, thanks for the 2-hour window [22:10:07] awight: the S:RI patch is smoke tested [22:10:37] awight: Let's merge that before merging the debug patch, so that that goes last :) [22:12:03] awight: if we can just send stuff 2 a different file no one cares about, we might as well not even throw the exception? [22:12:30] (CR) Awight: "FWIW, as written I don't think this will be useful for debugging the multiplied back-end requests, in many imaginable scenarios. It might" (2 comments) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/190559 (https://phabricator.wikimedia.org/T89258) (owner: AndyRussG) [22:13:07] AndyRussG: hey, you read my mind! [22:13:18] k lemme do that... [22:13:49] awight: wanna check out S:RI? I made a public property instead of passing the var around like a dinosaur flu [22:13:50] http://php.net/manual/en/function.debug-backtrace.php I guess [22:14:03] AndyRussG: yes, where is the PS? [22:15:21] awight: https://gerrit.wikimedia.org/r/#/c/188394/ [22:16:06] thx, that snuck right under my nose! [22:17:19] (PS7) Awight: Sample banner impressions client-side if no choiceData [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188394 (https://phabricator.wikimedia.org/T45250) (owner: Ejegg) [22:17:32] (rebase) [22:18:17] do we need ejegg|away to remove his -2 in order to merge dat? [22:19:47] (CR) Awight: [C: 2] "Sorry I'm being persnickety :)" (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188394 (https://phabricator.wikimedia.org/T45250) (owner: Ejegg) [22:20:08] hehe, I clicked the close box next to him. [22:20:16] hopefully there are no voodoo side effects ;) [22:20:39] bwahahaha [22:23:16] hey so uh [22:23:19] I'll start the deploy [22:23:36] if you want, you can add some more wax to the S:BR logging patch and prepare a revert, too, if you don't mind [22:23:44] MediaWiki-extensions-General-or-Unknown, § Fundraising Tech Backlog: Convert FundraisingTranslateWorkflow to use extension registration - https://phabricator.wikimedia.org/T88050#1037912 (atgo) [22:24:14] (CR) AndyRussG: Sample banner impressions client-side if no choiceData (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188394 (https://phabricator.wikimedia.org/T45250) (owner: Ejegg) [22:24:45] § Fundraising Sprint E, Fundraising-Backlog, § Fundraising Tech Backlog: BUG: Donation link geolocation not working consistently - https://phabricator.wikimedia.org/T87677#1037915 (atgo) @ccogdill_wmf @pcoombe have you heard anything else about this? if not, I'm going to reduce priority since this seems to be... [22:25:05] awight: OK u bet! [22:26:00] (CR) Awight: Sample banner impressions client-side if no choiceData (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188394 (https://phabricator.wikimedia.org/T45250) (owner: Ejegg) [22:26:59] § Fundraising Sprint E, Fundraising-Backlog, § Fundraising Tech Backlog: Create Alipay-specific TY page - https://phabricator.wikimedia.org/T88700#1037931 (atgo) p:Triage>Low [22:27:12] Aaugh, Jenkins isn't merging for us [22:27:14] suddenly [22:27:48] awight: I'm confused about ur last comment in the S:RI patch [22:27:56] Fundraising-Backlog: Spike: Let's take a look at recurring - https://phabricator.wikimedia.org/T86087#1037940 (atgo) @meganhernandez_wmf did you see if Ellery would be able to look into this? Thanks! [22:27:58] Lemme ask to make sure I'm not missing something [22:28:19] Err, the funky smell is that you have two conditionals that *look* identical [22:28:24] but the variable has changed, meanwhile [22:28:55] You could make the side effect and conditionals much cleaner by assigning filteredChoiceData = filter(choiceData) inside of the controller lib. [22:29:00] awight: ahahah right completely correct [22:29:13] Took me a minute of skullscratch to grok that [22:29:24] pff OK sorry about that, yeah you're totally right [22:29:36] My fault... moreso :) [22:30:15] (CR) AndyRussG: Sample banner impressions client-side if no choiceData (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188394 (https://phabricator.wikimedia.org/T45250) (owner: Ejegg) [22:30:40] no on the contrary, it's me not realizing some things are not obvious [22:30:53] atgo: we're about to deploy a fix to Special:RecordImpressions and also the snarly hhvm.log error flood [22:30:59] oooh exciting [22:31:12] atgo: hahah Yea Friday evening is the best [22:31:26] danger zone :P [22:31:27] time for cutting-edge deploys [22:31:29] it's going to go great! [22:31:34] Meanwhile, I am obstructing things right and left :) [22:31:40] awight: no on the contrary! [22:32:02] * awight looks mildly disappointed [22:33:57] heh [22:34:09] awight: I don't see where to find the backend server header you want [22:34:14] https://www.mediawiki.org/wiki/Manual:Varnish_caching ? [22:34:16] AndyRussG: it's not X-Cache? [22:34:29] Mmm we can just try that sure [22:34:30] that's what it looks like from the browser, at least. [22:34:43] (I bet opsen could tell us...) [22:36:35] awight: I see [22:36:39] (PS1) Awight: Merge master into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/190577 [22:36:53] (CR) Awight: [C: 2] Merge master into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/190577 (owner: Awight) [22:37:32] (Merged) jenkins-bot: Merge master into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/190577 (owner: Awight) [22:42:20] (PS3) AndyRussG: Warn in a special log on unexpected code execution [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/190559 (https://phabricator.wikimedia.org/T89258) [22:42:55] (PS4) Awight: Warn in a special log on unexpected code execution [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/190559 (https://phabricator.wikimedia.org/T89258) (owner: AndyRussG) [22:44:43] (CR) Awight: Warn in a special log on unexpected code execution (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/190559 (https://phabricator.wikimedia.org/T89258) (owner: AndyRussG) [22:46:48] (PS5) AndyRussG: Warn in a special log on unexpected code execution [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/190559 (https://phabricator.wikimedia.org/T89258) [22:47:58] AndyRussG: fyi I'm deploying the base patches now [22:48:07] gonna get to the logging in a minute... [22:48:25] The endpoint one and the S:RI, right? [22:48:57] yep [22:58:38] § Fundraising Sprint E, Fundraising-Backlog, § Fundraising Tech Backlog: BUG: Donation link geolocation not working consistently - https://phabricator.wikimedia.org/T87677#1038025 (CCogdill_WMF) @atgo nothing lately, I think it's fine to reduce the priority unless someone else has reports of it. [22:59:13] (PS6) Awight: Warn in a special log on unexpected code execution [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/190559 (https://phabricator.wikimedia.org/T89258) (owner: AndyRussG) [22:59:43] * K4-713 stares helpfully [22:59:58] K4-713: Special:RecordImpressions sucessfully turned down... [23:00:06] (CR) Awight: [C: 2] "Awesome. For the record, I'm assuming we can leave this one live as long as needed, cos this should not happen." [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/190559 (https://phabricator.wikimedia.org/T89258) (owner: AndyRussG) [23:00:10] Nice. [23:00:25] hi! [23:00:27] (Merged) jenkins-bot: Warn in a special log on unexpected code execution [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/190559 (https://phabricator.wikimedia.org/T89258) (owner: AndyRussG) [23:00:34] awight: You still in there, then? [23:01:01] K4-713: awight: I don't see no more hhvm.log errors! [23:01:14] AndyRussG: oh whew [23:01:22] ...and the site is still up? [23:01:25] :p [23:01:55] I just don't like to assume anything. [23:02:09] K4-713: still up and serving piping hot FR banners, yep [23:02:16] Eeeexcellent. [23:02:35] (PS1) Awight: Merge master into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/190584 [23:02:49] (CR) Awight: [C: 2] Merge master into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/190584 (owner: Awight) [23:04:13] (Merged) jenkins-bot: Merge master into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/190584 (owner: Awight) [23:15:32] awight: I'm getting some PHP borkiness on that last version of the logger [23:15:49] Nothing that should break, tho [23:16:04] oooh [23:16:05] Just not log the info we want :( [23:16:09] rats [23:16:25] I just deployed wmf17, looking at the files now... [23:16:43] Probably nothing coming in the log in any case [23:16:55] what is the error string? [23:16:58] 0 [23:17:07] Unless it isn't [23:17:21] Some BS about concatenation and addition using the same operater I think [23:17:32] haha [23:17:35] ARGH [23:17:41] o_O [23:17:43] Are you seeing anything in the log file? [23:17:51] Which is where, BTW? [23:17:57] ooh [23:18:00] K4-713: don't worry site's still up [23:18:12] right string concat is "." in this stupid corporate Perl port [23:18:13] heh [23:18:26] Bahahahahah [23:18:43] * AndyRussG double trout-slaps self [23:18:44] well, legoktm is catching up with us... [23:18:47] quick fix? [23:18:52] coming [23:19:00] his thing sounds much more important though so I'm about to step aside [23:19:06] we can follow up afterwards [23:19:27] LOL there is no php warning, it seems. Of course you meant to add two strings and get zero. [23:19:31] and print it :) [23:19:59] I'll deploy the wmf16 thing just so legoktm isn't left with undeployed merges... [23:25:21] AndyRussG: done. Well so ah, we just wait for "0" to show up in the logfile, now :) [23:25:33] * K4-713 squints [23:25:45] awight: arg! [23:26:28] hee [23:26:31] it's sort of hilarious IMO, like the bug infected us with a virus which causes us to be unable to capture it [23:26:35] awight: one sec... yes [23:26:59] I mean, I'm fixing it, just wanted to smoke test again, since the above idiocy of mine [23:27:08] I looked right at it, too... [23:27:20] In my defense, I wrote some Python last night? [23:27:57] Yeah and I was all Javascript today [23:28:03] Plus PHP sucks! [23:28:07] I have to send it as a new change [23:28:32] yeeeah [23:28:59] uuhhh one sec... [23:29:11] AndyRussG: what do you think about me setting off that code by hand? There's this eval.php tool... [23:29:20] Nope [23:29:56] no what? [23:30:29] don't like change stuff by hand, I think [23:31:23] oh [23:31:29] no, there's this tool you can call from tin [23:31:38] it will run a single function if you want [23:31:45] I can use that to see if the logging works in production [23:32:36] awight: Ah I see OK [23:33:18] (PS1) AndyRussG: Fix detail in warn in a special log on unexpected code execution [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/190596 (https://phabricator.wikimedia.org/T89258) [23:33:28] awight: ^ there it is... [23:35:20] (CR) Awight: [C: 2] Fix detail in warn in a special log on unexpected code execution [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/190596 (https://phabricator.wikimedia.org/T89258) (owner: AndyRussG) [23:35:49] (Merged) jenkins-bot: Fix detail in warn in a special log on unexpected code execution [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/190596 (https://phabricator.wikimedia.org/T89258) (owner: AndyRussG) [23:35:53] awight: OK that should do it, sorry about that [23:36:02] no worries! [23:37:59] (PS1) Awight: Merge master into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/190597 [23:38:10] (CR) Awight: [C: 2] Merge master into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/190597 (owner: Awight) [23:38:41] (Merged) jenkins-bot: Merge master into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/190597 (owner: Awight) [23:43:28] AndyRussG: for the record, I still think it's worthwhile to do that thing I mentioned in my comment, also log fromt the Special:BannerRandom endpoint, so we learn something about why the requests are multiplied. [23:46:17] Wikimedia-Fundraising, Wikimedia-Fundraising-CiviCRM: Allow mismatched refunds - https://phabricator.wikimedia.org/T89437#1038145 (K4-713) This definitely needs to happen. Side note: All the ones I spot-checked, were globalcollect chargebacks. This makes me even more interested in getting them in the db. [23:46:40] awight: ah hmmm OK now I understand.... [23:49:05] Wikimedia-Fundraising-CiviCRM, § Fundraising Sprint Devo: BUG: Gateway Reconciliation Report not pulling all donations - https://phabricator.wikimedia.org/T89288#1032444 (awight) @PPena: please let us know if you can use Rosie's numbers to reconcile this gateway? Thanks! [23:49:19] awight: any results on the log file? [23:49:26] nope [23:49:40] I'm pretty sure there will never be! [23:49:53] ... "never will be" [23:49:54] awight: cool! :) [23:50:10] mmm [23:50:12] but that's sad [23:50:25] I wanted to know how the cartoon ends, too [23:56:23] awight: yeah... I guess I didn't understand ur theory, but now I do 8p [23:56:43] We could leave this instrumentation and then move it, or move it right away, after legoktm [23:56:54] hm? [23:57:03] I'm done deploying [23:57:08] AndyRussG: I think we should just leave it [23:57:25] it's either a no-op, or will catch our poltergeist in the act... [23:57:40] sorry [23:57:47] let me change my vote to "move it" [23:57:56] that seems more useful [23:58:11] than photographs of nothing :) [23:58:39] our window is rapidly closing, however... [23:58:46] want to re-up? [23:59:36] AndyRussG: I've deployed the "."s, now :)