[16:31:31] bd808: On about 5th thought, I decided to just drop that part of it all together. [16:51:55] *nod* even better [17:21:10] ori: have you guys thought of setting up a service so you can send stats to statsd from the browser? [17:21:20] I randomly googled and saw there are such things: https://github.com/godmodelabs/statsc [17:23:12] milimetric: we have that [17:23:15] ours is called statsv ;) [17:23:17] for varnish [17:25:05] milimetric: https://github.com/wikimedia/analytics-statsv [17:25:23] js code is in wikimediaevents for now [17:25:32] oh duh, right, I always forget about statsv [17:26:53] hm... maybe it'd be fun to play around with grafana and see if I can make it more useful now that we have much more stuff in statsd [17:39:07] anomie: I outed you on wmfall and wikitech-l as being a Sr Software Engineer now. :) [17:39:12] anomie: Congratulations! [17:39:23] Thanks [17:39:33] Herp derp: https://phabricator.wikimedia.org/T105373 [17:39:54] congrats! [17:44:16] ostriches: it's not obvious to me how that would ever be passed a non-array to iterate [17:44:26] corrupt cache? [18:04:24] bd808: Mebbe? Coincided with the general freakout a bit ago... [19:56:11] anomie, tgr: I think that https://gerrit.wikimedia.org/r/#/c/219446/7 may break all/most oauth clients (now live on enwiki) [19:56:45] See https://phabricator.wikimedia.org/T105387 and this comment in Brad's test app "Note that /wiki/Special:OAuth fails when checking the signature, while index.php?title=Special:OAuth works fine." [19:56:59] * anomie looks [19:58:34] ragesoss is saying that his app is seeing an unexpected 301 now which I'm guessing is https://www.mediawiki.org/w/index.php?title=Special:OAuth redirecting to https://www.mediawiki.org/wiki/Special:OAuth [19:58:37] bd808: Is it live on mediawiki.org and testwiki? [19:58:44] all wikis [19:58:59] oauth-hello-world doesn't seem to be broken: https://test.wikipedia.org/w/index.php?title=User_talk:Anomie&diff=233467&oldid=223494 [19:59:09] But https://tools.wmflabs.org/oauth-hello-world/index.php?action=identify seems to be broken [19:59:24] Although the identify bit doesn't work. Oh, you noticed that too. [19:59:41] Not sure why though; oddly, curl isn't reporting any error. [19:59:46] But yeah I made some posts too that did seem to work [19:59:53] * anomie investigates [20:00:09] < ragesoss> bd808: logging in gives no error visible to the user, but the identify and other features then show errors on that hello world app [20:00:49] identify was broken before this was merged [20:01:20] I assumed it was related to the HTTPS redirects [20:02:11] the other hello-world features work correctly [20:03:11] bd808: Adding "&dummy=1" to the URL for the identify fixes it, so yeah, that broke OAuth for any GET request that doesn't have additional parameters. [20:03:29] Or something did anyway [20:03:49] bd808: does the redirect thing apply to special pages? [20:04:00] that seems horribly broken on a conceptual level [20:04:12] tgr: https://gerrit.wikimedia.org/r/#/c/219446/7 looks like it does [20:05:14] I'm not very familiar with the internals but I don't think it is safe to assume anything about the semantics of the action parameter in Special:Foo?action=bar [20:08:17] bd808: that patch was merged yesterday [20:08:27] no it was commented on yesterday [20:08:32] so it'll only embark on the deploy train next tuesday I think? [20:08:34] merged june 30th [20:08:57] uhh [20:09:10] gerrit shows the last comment date under "updated"? [20:09:14] yes [20:09:16] that's just sad [20:09:31] use the "included in" thing to see what branches it is in [20:09:39] it makes sense :P [20:11:38] what's the included in thing? [20:11:57] Forrestbot? [20:12:57] no, http://i.imgur.com/miQvklA.png [20:13:06] it's in the top right if you use hte new change screen [20:16:05] there is a new change screen? :) [20:16:17] apparently I need a gerrit beginner's course [20:16:20] tgr: Don't use it! It's horrible. [20:16:26] The old change screen has this too. [20:16:35] As per legoktm's screenshot. :-) [20:16:52] tgr: James_F is wrong, the new change screen is amazing [20:16:57] tgr: Compare https://gerrit.wikimedia.org/r/#/c2/223205/ vs. https://gerrit.wikimedia.org/r/#/c/223205/ [20:17:25] We're looking into fixes via the OAuth gem we use. [20:17:39] new change screen adds a little indicator when the patch has been updated, has a giant CR+2 button, etc. [20:17:42] bd808, tgr: Untested, but should work to suppress the redirect: https://gerrit.wikimedia.org/r/223952 [20:17:58] I can't believe how they dropped the shortcuts to navigate between files in the new change screen [20:18:23] Nikerabbit: [ and ] work for me? [20:19:09] also, [20:19:14] legoktm: not for me in the new change screen [20:19:25] is there no index on img_user? [20:20:09] Nikerabbit: weird...I'm using Firefox. [20:20:39] chrome here [20:21:07] also a related issue, the [] keys don't work on the old change screen on windows [20:21:30] works in whatever version of chromium I have installed :/ [20:21:37] legoktm: only on img_user_text I think [20:22:04] yeah I see CREATE INDEX /*i*/img_usertext_timestamp ON /*_*/image (img_user_text,img_timestamp); [20:22:26] anomie, bd808 thanks! [20:22:30] UserMerge is doing selects with WHERE img_user = ... [20:23:50] special:listfiles uses user_text [20:23:51] ugh [20:24:48] no aaron :( [20:25:45] * legoktm skips commonswiki for now [20:32:04] doesn't look like flaggedrevs has a fr_user index either... [20:41:30] anomie: do we have a version of https://tools.wmflabs.org/oauth-hello-world/index.php that is pointed at beta? [20:41:50] bd808: If not, I can make one. Let me check. [20:42:07] I'd like to (1) reproduce the problem there, (2) push your patch, (3) backport if it fixes things [20:42:52] There's https://tools.wmflabs.org/oauth-hello-world/beta.php, but it's apparently suffering a little from bitrot. [20:48:17] bd808: you can test on https://gerrit.wikimedia.org/r/#/c/210036/ [20:48:54] why haven't we merged that? [20:48:59] * bd808 can't remember [20:49:11] I was wondering if I should ask that :) [20:50:43] Probably because authorize didn't work for me in PS3? [20:50:58] at some point, yes [20:51:07] * bd808 tests it out again [20:51:25] I hardcoded something that was randomly generated by the vagrant installer [20:51:41] ah yeah I remember anomie point that out now [20:53:44] bd808: https://tools.wmflabs.org/oauth-hello-world/beta.php seems to function now [20:53:47] bd808: anyway I can verify anomie's patch works [20:54:04] tgr: want to do the +2 honors then? [20:54:13] and start the backport? [20:54:19] on it [20:59:06] SWAT is in two hours; we should probably deploy this before that, right? [20:59:19] not sure how much breakage it is causing [20:59:48] I think it's small enough and a big enough deal to deploy now [21:00:26] we should just check that tin is open before we merge the backport [21:31:54] ragesoss: can you verify the fix? [21:33:52] tgr: did you sync the change? [21:34:02] yes [21:34:39] hello-world works now, but never hurts to test on the original report [21:35:19] yup. I apparently don't get a ping from sync-file. I thought I did [21:35:26] https://tools.wmflabs.org/oauth-hello-world/index.php?action=identify does work for me now [21:36:42] tgr: still getting errors [21:38:58] Same 301 redirect that breaks it [21:39:27] Do the 301's get cached by Varnish? [21:39:42] ragesoss: is the app public? [21:40:14] tgr: dashboard-testing.wikiedu.org [21:40:54] tool labs requests would also go through varnish and that did not seem to be cached [21:40:59] for long, anyway [21:41:21] ragesoss: what is the exact redirect? can you add that to the bug? [21:41:40] "500 — internal server error. :-(" needs more info for sure to debug from here [21:41:50] :) [21:47:38] ragesoss: do you have your client configured to hit an https://.... URL on the wiki? [21:47:54] (although that should have broken a while ago) [21:52:24] https://phabricator.wikimedia.org/T105387#1443192 [21:52:39] so yeah, it's cached somewhere [21:52:58] maybe we can get bblack to purge it? [21:53:06] I wonder why it is nginx and not apache that responds [21:53:25] yeah, I just asked [21:54:01] those headers are all showing a cache miss though... [21:54:13] X-Cache: cp1054 miss (0), cp4017 miss (0), cp4017 frontend miss (0) [21:54:33] Yes, it is configured to hit https, bd808 [21:54:49] Just broke today with the deployment [21:55:14] ragesoss: *nod* we are reproducing still with the url that tgr dug up [21:56:11] hmmm... [21:58:04] does not seem to be caching [21:58:12] I get the same redirect for /w/index.php?title=Special:OAuth/identify&&&&&&&& [21:58:29] but &foo=1 doesn't redir [21:58:38] which was the original behavior [21:59:03] because I'm not sending OAuth headers, duh [21:59:10] code is deployed... [21:59:19] oh [21:59:23] let me try that again with a valid request [21:59:44] that said, probably varnish needs to be split on that header [22:01:06] does a 301 with cache-control:private get cached? [22:01:35] it should not AFAIK [22:03:59] hmmm... it actually might not care about that header [22:04:04] https://www.varnish-cache.org/trac/ticket/1124 [22:04:21] I can confirm that adding &foo=1 to our OAuth gem on the /identify URL makes it work now [22:05:49] tgr: confirmed, we have vcl to respect private, no-cache and no-store [22:06:36] ragesoss: ok. you are definitely still being bitten by the change in https://gerrit.wikimedia.org/r/#/c/219446/ [22:06:48] now we wonder why [22:08:58] This vcl should keep it from caching in Varnish I think -- https://github.com/wikimedia/operations-puppet/blob/production/modules/varnish/templates/vcl/wikimedia.vcl.erb#L423-L426 [22:10:39] in practical terms a ban would solve this since no client requests /identify without an OAuth header, so in the future Varnish should not see 301s [22:14:17] So for ragesoss we need a ban on https://en.wikipedia.org/w/index.php?title=Special:OAuth/identify correct? assuming that his client interactions are cached [22:15:18] that's request from the client side is a GET with some headers holding the authorization to verify? [22:19:07] bd808 working for my app now! [22:19:59] ragesoss: wat? [22:21:05] * ragesoss shrugs [22:22:02] works for me now too [22:52:07] bd808: Get a bunch of the key badge thingies, put them on your waist. So people can use the usb stick but have to be within 3' of you or so :p [22:59:15] heh [22:59:34] Dan is the one on the hook for keeping them around :)