[00:00:17] don't mind me, apparently it does call isConfiguredProxy [00:02:12] maybe there should be an export/import format for CDB instead of building a search tree on every page request [00:03:48] You'll make ori tick adding a new cdb ;) [00:04:07] on a random request, building the IPSet tree is 2ms [00:04:09] yeah, import/export as a PHP static array [00:04:21] so might be overoptimization [00:06:51] 2ms is a lot [00:07:20] Paul Irish has a good presentation called "Delivering the goods in under 1000ms", in which he argues for a 1000ms target for page loads [00:07:41] he claims that backend response time should be 200ms or less if you want to have any hope of hitting that [00:08:05] so 2ms is 1% [00:08:09] of that allowance [00:08:18] link, btw: https://docs.google.com/presentation/d/1MtDBNTH1g7CZzhwlJ1raEJagA8qM3uoV7ta6i66bO2M/present?slide=id.g3eb97ca8f_1860 [00:09:32] tgr: Regarding sessions. I'm still confused a bit. If we always start a session now when MW starts in index.php, what's stopping regular anons from being thrown in a session anytime they cache miss? [00:09:51] or is creating not the same as persisting and/or emitting a cookie? [00:10:31] following after reading https://gerrit.wikimedia.org/r/273313 - which looks a commit that shouldn't be [00:11:20] Krinkle: we always set up a session, we don't always persist it [00:11:31] where persist = save to redis + emit cookies [00:11:51] starting a session is pretty much just creating a bunch of php objects [00:12:59] tgr: So when does persist happen? [00:13:12] under what circumstances that is [00:13:27] called by application code [00:13:32] login and such [00:14:04] tgr: There didn't used to be a doPerist() method, so what was it migrated from? [00:14:04] and the session self-persists in some situations where it was already persisted but nonstandard [00:14:17] e.g. token cookie present but session cookie missing [00:14:30] session_start() probably [00:14:38] or rather wfSessionStart [00:14:42] Right [00:15:19] we typically use that when either a user logs in, or when an anon shows an edit page (from which point they can recieve talk page messages which only render if you bypass html cache for page views - but don't involve a login session) [00:15:29] as for the commit, that just prevents vary headers in a situation where there shouldn't be any cookies [00:15:58] yes, those are the two situations I remember [00:16:14] except you don't need to log in, just visit Special:Userlogin [00:16:22] It looks weird. getVaryHeaders() returns naturally empty if there was no persistence invoked. And doing that would fail if disabled anyway. [00:16:32] It looks out of place for that code to be checking no-session, again. [00:16:43] Right? [00:16:56] * Krinkle is probably asking the wrong question and is misunderstanding [00:17:10] no, it asks the session provider for a list of cookies [00:17:27] the provider is the concrete implementation of the session, such as centralauth [00:17:47] and it typically just returns a static list IIRC [00:18:31] since vary headers should be emitted on every URL where content changes when you have a session, even on requests that don't have a session [00:18:44] that is, vary should always be the same for the same url [00:18:55] that's how I understood bblack, anyway [00:20:18] that was the same pre-sessionmanager, the vary code just used to be in outputpage or some other hidden place [00:20:39] we never looked at the session to decide what the vary header should be [00:22:50] Krinkle: https://github.com/wikimedia/mediawiki-extensions-CentralAuth/blob/master/includes/session/CentralAuthSessionProvider.php#L431 and https://github.com/wikimedia/mediawiki/blob/master/includes/session/CookieSessionProvider.php#L301 is the exact code [00:35:24] tgr: Not to see what it should be but to know whether to emit one at all [00:36:02] If the session provider has a fixed list, and anons just browsing around cache misses, there won't be vary headers [00:36:12] eventhough MW_NO_SESSION is not set [00:36:22] because getVaryHeaders() is never called in that case [00:36:22] right? [00:37:04] tgr: So getVaryHeaders() is only called when we want it. Which is presumably based on whether the session persists or not. [00:37:14] In which case, why is there a need for a MW_NO_SESSION check? [00:37:24] I don't see how in a stable implementation that is not paranoia [00:42:22] I don't mean to question the code, just using it as an example to enter the realm of sessionmanager and understand it. It touches so many layers that I'll get a better sense of it. But can't help not making sense of how it fits in yet. [00:45:03] Krinkle: getVaryHeaders should be called on every OutputPage request [00:45:06] https://github.com/wikimedia/mediawiki/blob/master/includes/OutputPage.php#L2010 [00:45:50] that gets the list of providers, which only depends on configuration, and each provider returns the things it can vary on, which again only depends on configuration [00:46:10] nothing wrong with questioning the code :) [00:47:04] so unless you set MW_NO_SESSION you will always get all the things any request with a session could conceivably vary on [00:47:46] which is not a big deal since anything that does not use sessions at all probably does not use OutputPage either, but still more correct this way [00:48:22] tgr: "request with a session" [00:49:04] If getVaryHeaders() is a fixed list from providers, and centralauth is a provider, and it includes a very on session cookie, and it's called on all OutputPages... that would.. [00:49:09] oh wait, vary is not the header itself [00:49:12] *facepalm* [00:49:15] Krinkle: the point is, if $URL might return different content depending on your session, the server must send a vary header [00:49:25] saying it varies doesn't make it vary in the way my mind thought. I knew this but misread [00:49:25] this is true even if the request has no session [00:49:38] Yeah yeah [00:49:42] otherwise it could be cached and reused for another request which does [00:49:52] wiki/Foo should vary by cookie since those with the session cookie shouldn't reuse the cache [00:50:06] but even the anon cache will "vary by session cookie" to make it not .. eyah, I get it [00:50:36] I wrongly misread the emitting Vary as being the thing that also emits the session cookie header [00:50:40] yes, and SessionManager has no way of knowing whether a given request is for a varying URL, without external input [00:51:15] that's slightly problematic currently because there are a bunch of URLs which don't need a session but we can't tell at setup time [00:51:20] like action=raw [00:51:31] so we are splitting the cache unnecessarily [00:51:44] but again, that behavior existed pre-sessionmanager as well [00:52:12] tgr: Hm.. private wikis? [00:52:15] read is a permission too [00:52:40] sure, but that part is known at setup time [00:53:13] Right. Itd be nice to be able ot re-use Varnish cache from anons for liu for action=raw on public wikis [00:53:13] can just check User::isEveryoneAllowed [00:53:16] like for load.php and static urls [00:53:22] Yeah [00:53:29] That logic shouldn't be in Setup.php though [00:53:33] separation of concerns [00:53:59] But maybe the Action class can (in)directly communicate up to the session manager that it shouldn't vary [00:54:03] via OutputPage possibly [00:54:07] It can just disable it [00:54:24] the problem is, you only learn that is a RawAction when MediaWiki does its routing/redirect check logic, which is well after session setup [00:54:57] tgr: Yeah, but session setup should be harmless [00:54:57] which means it is entirely possible by that point that session cookies have been emitted [00:55:03] like you say, just some php objects [00:55:13] session setup fixes weird cookies [00:55:13] wait what? [00:55:23] It emits the cookies at setup? [00:55:38] if you have a valid but weird session, it's fixed at setup [00:55:53] Is that the common case? [00:56:04] Loginpage changes session half-way, right? [00:56:05] basically when you have enough cookies that your session can be verified but something is missing [00:56:13] it controls that [00:56:23] can it go in the opposite direction as well? [00:56:39] what do you mean by opposite direction? [00:57:10] a GET to the login page persists the session [00:57:12] Login page shown to anon, at setup time is not yet special, it's just another page, could be an anon page view. Nothing there should be emitting anything. [00:57:21] Then login page handling makes teh session persistent [00:57:24] by doing something [00:57:28] a successful login changes the session id and authenticates the session [00:57:59] if there is no valid session in the request, session setup is a noop [00:58:13] is there is a valid session, but it is not normal in some way, it is normalized [00:58:30] what does it mean for there to be a "session in the request" you mean, existing cookies from the client? [00:58:35] typically that means that some cookies expired but enough remained to make sure it is the right user [00:58:40] yes [00:58:42] ok [00:58:53] page views by anon have no session, and don't emit cookies. [00:58:59] with centralauth there are something like 5 session cookies [00:59:01] login view does. [00:59:12] if some of those are missing they are recreated [00:59:16] sure [00:59:30] (I'm answering "what do you mean by opposite direction?" by the way) [00:59:38] that's the only thing that happens on setup, but it is enough to make not varying unsafe [00:59:39] but first establishing the first direction [01:00:18] well, atypical session handlers can do weirder things [01:00:28] e.g. OAuth can be configured to use cookies [01:00:58] the idea being that you can have multiple parallel machines emit OAuth requests without their sessions getting mixed up [01:01:15] in that case, the session cookie will always be set if its missing [01:01:38] since a valid OAuth request is considered as an authenticated session by itself, with or without cookies [01:02:23] basically what I'm saying is, ActionRaw can call something in OutputPage that makes it not emit getVaryHeader [01:03:12] yeah but then you get a request with partial cookies -> Setup.php emits missing cookies -> ActionRaw disables Vary -> you have a problem [01:33:05] tgr: I don't fully grasp the idea of partial cookies. Is that new? Garbage in, garbage out, right? Intuitively I'd ignore partial cookies instead of fixing it up [01:33:11] Effectively logged out [01:34:13] Seems like a recipe for disaster to mismatch with the wrong session. sessionId+name is stable and somewhat foolproof (not secure proof, but accident proof). Whereas just session id, might cause a problem and tie with an existing name. Under what valid use cases would an "optional" cookie be missing? [01:35:02] Yeah, I don't get it either. [01:35:04] I think it is not entirely new but the checks became more rigorous [01:35:06] basically when you have enough cookies that your session can be verified but something is missing [01:35:10] not entirely sure though [01:35:16] The only case where a response might have new cookies + disable vary is if a page does both intentionally, in which case it could throw (e.g. if a page both starts a session and says it won't vary, it's lieing and we can throw), e.g. if the Login page claims to not vary [01:35:16] to me, if your session can be verified, then nothing is missing [01:35:19] anomie knows more about it [01:35:28] and conversely: if something is missing, your session can't be verified. [01:36:05] ori: we have two ways of authentication, a session cookie which identifies a redis object which has short expiration, and a token cookie which refers to a DB record [01:36:27] the second one is optional, there is a "remember me" checkbox at login [01:37:05] and with centralauth there is a separate central session cookie and local session cookie, as there are separate redis objects [01:37:34] if some of that goes missing the rest might still be enough to authenticate the user [01:37:49] so you're thinking of cases where the ephemeral session cookie is present but the token cookie is absent? [01:38:01] or the other way around [01:38:35] that should not happen, though, right? if it does happen, it should be treated as highly suspicious until proven otherwise [01:38:58] rather than a "these sorts of things just happen" [01:39:18] it's especially dangerous because it could be masking a serious bug [01:39:59] it can happen in a number of fringe ways [01:40:28] e.g. logging in without "remember me" then going to the login page again and logging in with "remember me" [01:40:41] I think; not quite sure about that [01:41:09] I think the logic for this came from anomie going through the OWASP session handling requirementes [01:41:16] if the centralauth cookie expired but the local didn't it means you'll stop getting new auto-login sessions on wikis you're not already logged-in in which is harmless. it shouldn't extend itself indefinitely. We only add CA cookies on-login typically. Not sure if that's what's being added. That wasn't the case previously, right? [01:41:29] and one of them says "web application must verify all cookies (and enforce relationships between them)" [01:42:01] Krinkle: there is a central centralauth cookie and a local centralauth cookie [01:42:07] centralauth is a scary thing [01:42:12] But we have unexpected behavior occurring in the wild right now, with the session cross-over bug. So surely you see the danger in silently accommodating conditions which are not specified in the code. [01:42:30] The users getting logged in as other users should be conceptually impossible [01:42:36] I don't know why that is even remotely possible. [01:42:53] ori: yes, there is a fair chance that the wrong logins are related to that in some way [01:42:56] If session object username doesn't match the prefix_Username cookie, don't login. It doesn't protect against malice, but that's not what's happening right now [01:43:07] even though there should be at least two additional layers of defense [01:43:33] all input is suspicious [01:43:36] s/don't login/don't activate the session [01:43:42] you expect the user to have two security tokens, the user has one [01:43:48] don't just give the user the other one! [01:43:58] revoke the one that the user has [01:44:03] and scream bloody murder [01:45:02] Krinkle: re user logged in as other user, see the description of https://phabricator.wikimedia.org/T125283 for how it might happen [01:45:20] where by "might" I mean "theoretically cannot" [01:45:30] but still the closest to an idea we have so far [01:45:54] I have a theory [01:46:20] ori: I think the token cookie has a longer expiry time than the session cookie, so it's normal for the session to be missing [01:46:28] would have to check the code though [01:46:41] I don't think we set the session cookie when it is the other way around [01:46:46] maybe the Set-cookie header on logout gets clobbered somehow [01:46:49] ...the token cookie... [01:46:56] so that instead of unsetting all four(?) cookies we unset only some [01:47:36] actually, no, that's not convincing, because this didn't happen on shared computers [01:47:41] It seems a fair amount of code is (admitedly poorly) relying on sessions not being created unless wgUser was interacted with and we didn't have bugs with that situation. Now we're forcing sessions everywhere in an envirnoment where that is not (yet) expected and supported. [01:48:04] What has SessionManager fixed? Can we undo the session everywhere aspect of it until things account for that? [01:48:06] ori: theoretically, varnish is supposed to uncache anything with a set-cookie header [01:48:21] see the task description, I tried to collect the relevant facts there [01:48:50] I'm not worried about cache pollution; I was worried about a user only getting partially logged-out, and then the accommodating code "fixing" things into a full session [01:49:11] well, that would still be the correct session [01:49:24] the logout would fail, but that would be a different bug [01:49:35] Yeah, I agree; it's not convincing. [01:49:47] we had that bug for a short while, although it was CA-related and much more complex [01:50:41] Krinkle: I don't think anything relied on sessions not being created [01:50:58] a bunch of code relied on wgUser (and thus the session) always being there [01:51:15] that assumption is broken by MW_NO_SESSION [01:51:38] that's an availability problem, not a security problem [01:52:08] "Set-Cookie header gets cached" "SessionManager emits Set-Cookie in some cases old code did not" " they should set Cache-control: private if the session is persisted... RawAction did not do that.., load.php and similar secondary endpoints still don't " [01:52:18] that sounds like it was working (albeit not in a nice way) [01:52:20] we have bugs that we didn't have before, I think it's reasonable to ask which bugs did we have before that we now don't [01:52:24] but wasn't anymore because of the change [01:52:40] yep, hence I asked what it fixed. [01:53:00] For future reference though, it seems refactoring and behaviour change should really be separated. [01:53:07] well if by "relied on it" you mean "it happened to work" then yes [01:53:12] I like the separation now (though fewer classes wouldn't have hurt) [01:53:20] tgr: Well, that is most of MediaWiki. [01:54:00] Changing X on the false condition that things work with X in a perfect way when we know it doesn't... [01:54:07] anomie is the better persion to ask for a full account of the things sessionmanager does, there are quite a few [01:54:30] - it's trying to make the order of method calls more deterministic [01:55:09] with the old logic, unstubbing wgUser set up the session, setting up the session called LoadUserFromSession, which called CentralAuth autocreate [01:55:26] which called the user creation hooks and whatever else [01:56:02] Yeah, the old code was a spaghetti [01:56:06] and not very OOP [01:56:07] which meant a number of hooks could be callied in lots of different places in the request lifecycle, depending on what and when unstubbed wgUser [01:56:40] SessionManager moves autocreation to a deterministic time [01:56:49] In general though when changing the behaviour like that, existing code needs to change before, not after. [01:56:57] which is a lot of work, but we're doing it either way [01:57:06] - it unifies a bunch of different things as session handling, mainly cookies and OAuth [01:57:26] Gotta go, will check out the RL patches tomorrow [01:57:27] o/ [01:57:27] OAuth used some nasty hacks before to replace the PHP session with its own session [01:58:19] - it adds some security features that we haven't taken advantage of yet, basically hooks to add extra data to the session and validate it based on that [01:58:31] think suspicious geolocation changes [01:59:14] - it detaches session handling from PHP (which does its own cookie handling, its own warped serialization format etc) [02:01:59] - it adds the ability to have multiple parallel OAuth session via an Authorization header + cookie mix, which would have been super painful to handle in the old code [02:02:39] - it makes authentication / session code more localized in the codebase [02:03:03] those are the things I can think of [02:03:35] plus nicer logging and unit test coverage and whatnot [02:04:46] ori: basically we have three new bugs, right? the mystery login thing, the extra redis calls and the slower page save which might or might not be related [02:05:43] the second of which should be fixable, it's just some kind of caching failure [05:43:53] if the session is now set up in a particular place in the request-processing flow, why introduce a User::isSafeToLoad()? [14:25:40] tgr|away: I'm impressed with your patience with the grilling yesterday. I'd have quickly gotten fed up and said "Go actually read the code before coming in here and telling us to do things that we're already doing." [15:30:22] ori: https://gerrit.wikimedia.org/r/273462 [15:34:54] ori: User::isSafeToLoad() exists because the User object now refuses to try to load itself from the session until after the auto-creation has been done near the end of Setup.php, if necessary. Things that could be called either before or after that time, such as MessageCache's ParserOptions setup, need a way both to avoid the associated warning from User and to avoid in-process caching the results of whatever it is they do if it was done using [15:34:54] the anonymous user rather than the logged-in user. [15:36:13] (previously, the auto-creation would happen whenever a User object tried to load itself from session, no matter how early in the request that might occur, which caused all sorts of strange behavior to the point that T43201 was filed) [15:36:13] T43201: UserLoadFromSession considered evil - https://phabricator.wikimedia.org/T43201 [17:28:12] anomie: thanks, that's helpful [17:28:24] anomie: could you review https://gerrit.wikimedia.org/r/#/c/273413/ ? [17:29:25] ori: I'll have a look after the meeting [17:29:31] anomie: perfect, thanks [19:15:08] ori: thanks for following up on the reject parser cache hook, it had slipped from my mind [19:16:09] np, thanks [19:18:37] ori: Found some stuff in https://gerrit.wikimedia.org/r/#/c/273413/ that would have gotten a -1 had it not been merged already. [19:18:50] anomie: Thanks, I'll look and fix-up. [19:20:48] bd808: Re T124832, I have no ideas. [19:20:49] T124832: Setup testing server for AuthManager in Labs - https://phabricator.wikimedia.org/T124832 [19:21:33] anomie: *nod* local topic branches will probably work. They will just be a bit of a pain to make sure we keep up to date [19:50:38] bd808: initially, that's like 5 core patches and single patches in a handful of extensions, so no big deal [19:52:24] by the time we actually deploy stuff, it would be great to have a single "make authmanager disappear" flag that provides full rollback without filtering dozens of commits [19:53:25] *poof* [19:53:43] bd808: in unrealated news, you are the most privacy-conscious person around me so I would like your opinion to stake out the range of reasonable opinions in this matter :) [19:54:08] do you see any problems with sending a public talk page message to people who only have private Gather lists? [19:54:40] "Gather is discontinued, thank you for using blah blah, you can request access to your lists [here]" [19:54:54] tgr: I don't see any problem with it as long as it's just a generic "go look at this info" ping [19:55:13] cool [19:55:20] that saves me a bunch of work [19:55:44] (I still think we should build an Echo channel for MassMessage though) [19:57:07] https://phabricator.wikimedia.org/T58362 [19:57:52] yeah, something like that [19:58:08] Echo is a bit underutilized [20:06:15] bd808: I found a bug in MetadataMergeException ;) [20:07:22] anomie: {{SOFIXIT}} ;) [20:07:35] bd808: Already did, https://gerrit.wikimedia.org/r/#/c/273508/1/includes/session/MetadataMergeException.php [20:07:52] doh! [20:08:10] Is there not a \Session\Exception class? [20:08:34] Use IDEs, Luke! [20:09:01] anomie: I see what I did there [20:35:23] bd808: hello. Regarding our private exchange about jenkins/logstash, may I copy paste your writings to a public phabricator task ? [20:36:57] hashar: yes you can. no problem [20:37:13] will try to get a nice summary [20:37:41] I forgot you were subscribed to that list, I sent my mail to private to avoid having you to sink in it and feel obliged to help ;-D [20:50:52] hashar: :) that's nice of you. i'll try not to actually do the work [22:18:55] bd808: one small wart of the logging changes in core. Suppose I want to debug memcached. I look at MemcachedPeclBagOStuff.php and see that it is calling $this->debugLog() with what looks like useful debugging data. I want to get that data. How do I know where it is going and how to configure it? Previously, the log channel was right there. Now I have to hunt down the constructor, which isn't always easy. [22:19:20] grepping for 'new MemcachedPeclBagOStuff' won't find it [22:20:57] In a magical future full of unicorns and rainbows you'd just look a the the DI container config [22:21:16] but today you may be right that it is random [22:22:03] I would personally ditech our weird habit of arbitrary log channel names and instead make them all match the class name except for "exceptional" log channels [22:22:07] *ditch [22:22:28] that's actually a great idea [22:22:32] but I learned how to do logging in the land of java where that's a common pattern [22:22:41] why don't we do that? [22:23:12] * bd808 mumbles something about raisins [22:23:34] I've been loathe to change all the logging in the stack [22:23:36] /r/nocontext [22:24:48] I *think* all the bagostuff logs are in the same channel but I might be wrong about that [22:25:58] you are :) [22:26:14] (not that I knew this either, I just spent five minutes figuring it out) [22:27:10] CACHE_MEMCACHED => [ 'class' => 'MemcachedPhpBagOStuff', 'loggroup' => 'memcached' ], [22:27:20] yeah [22:27:47] which ObjectCache::newFromParams() uses to construct a logger [22:29:24] filed as https://phabricator.wikimedia.org/T128224 [22:29:50] Fatal error: no version entry for `extensions/MassMessage/maintenance/sendMessages.php`. in /srv/mediawiki/multiversion/MWMultiVersion.php on line 365 [22:30:03] am I using the wrong working directory or what? [22:30:32] missing a wiki parameter? [22:30:37] yeah, looks like [22:30:53] or a busted dblist? [22:30:55] uh [22:31:02] yeah, it was at the wrong position [22:31:15] if you do --wiki=foobarwiki it'll work anywhere [22:31:17] Or sohuld [22:33:47] legoktm: how long until a massmessage job is scheduled? [22:33:54] I sent a test message to a single page [22:38:44] what is the purpose of the forceHTTPS cookie? [22:40:30] ori: in mixed HTTP/HTTPS MediaWiki installs it decides where you get redirected [22:41:54] the details are complicated and I never fully understood them but a decision is made based on your user preferences, site settings like wgSecureLogin and the current protocol whether you should be forced to use HTTPS [22:42:03] and the cookie does that [22:42:20] on Wikimedia sites it is fairly pointless [22:43:30] it wasn't so pointless until the all HTTPS switchover [22:43:41] sure [22:43:58] but it may cause more problems than it helps [22:44:30] I was helping mooeypoo with her dev vm the other day when it got set somehow [22:44:32] yeah, it can sometimes screw up vagrant for example even if you never enabled the https role [22:44:56] I think that has something to do with the way that "keep me logged in" is handled [22:45:09] it happens occasionally in beta cluster too [22:45:33] given that: http://motherboard.vice.com/read/google-will-soon-shame-all-websites-that-are-unencrypted-chrome-https [22:45:51] "Google wants everything on the web to be travelling over a secure channel. That’s why in the future your Chrome browser will flag unencrypted websites as insecure, displaying a red “x” over a padlock in the URL bar. With this upcoming change in Chrome, Google makes it clear that the web of the future should all be encrypted, and all sites should be served over HTTPS, which is essentially a secure layer on top of the usua [22:45:51] l HTTP web protocol. " [22:45:57] we made a similar statement by going https-only [22:46:07] so I wonder if there should be an option in core, wgAllowInsecure or something [22:46:30] unless it is true we just assume that we're only available via https [22:47:09] Chrome is going to yell about 80% of the internet, forever [22:47:12] Wonder how a lot of places are going to cope with the change... [22:47:25] MediaWiki mostly detects capabilites from the schema of wgServer [22:47:38] "Google declares shared hosting dead" [22:47:48] heh [22:47:49] if it is http: that's a HTTP-only site, https: likewise, // means both are supported [22:48:01] so that flag kind of exists already [22:48:38] bd808, "Google declares all people who run websites should be technically competent" [22:48:59] I think you're underestimating how quickly the tide can turn on the web. It's becoming easier, and it will become the norm very rapidly. [22:49:17] well "Google declares that the 'open web' should conform to their self-chosen specs" [22:49:33] well, like I said, we made the same statement ourselves, right? [22:50:15] Us going https only is different than changing MW to putt a "this site owner is a dubmass" banner on all pages not served over HTTPS [22:50:29] oh, sorry, I wasn't suggesting that at all [22:50:30] (Friday typing) [22:50:39] the default can even be to allow http [22:50:42] I'm saying that's what GOOG is doing [22:51:10] logstash exhausts my browser any time I am doing something nontrivial [22:51:18] all I'm saying is that it makes sense to frame it in a language that makes it clear encryption is important, not a neat perk [22:51:25] is it somehow doing the search in frontend logic or what? [22:51:31] tgr: yeah. its been getting slow lately :/ [22:51:54] tgr: yeah Kibana is just a static site and a lot of js [22:52:10] and the version we are running it really really old [22:52:26] testing and upgrading is on my "someday" list [22:52:49] tgr: usually near instant? you can check with showJobs.php --group [22:52:54] the latest versions have worse date handling issues than the one we have now though (or did the last time I checked) [22:53:45] legoktm: "Unable to connect to redis server rdb1007.eqiad.wmnet:6380" [22:54:08] those resid errors have been happening for weeks :/ [22:54:11] *redis [22:55:01] * bd808 thought he opened a bug about hat [22:55:12] * bd808 gives up on typing [22:56:02] well it's happening on 100% of my massmessaging attempts [22:56:27] ori: ^ ideas about redis connection failures? [22:56:33] tgr: uhhh, that sounds like a job queue issue [22:56:50] even the server-side system just dumps jobs into the queue [22:57:22] yeah, it happens on the job scheduling attempt [22:57:35] do you normally run it from terbium? [22:57:54] they were/are getting reimaged [22:57:59] or mw1152 or whatever the current terbium replacement is [22:58:07] i'll take a look [22:58:11] yes, terbium [22:58:15] but I haven't run it since SULF [23:01:18] tgr: what exactly did you run? [23:01:30] mwscript extensions/MassMessage/maintenance/sendMessages.php --wiki=enwiki --pagelist /home/tgr/T127784_users_test.txt --subject /home/tgr/T127784_subject.txt --message /home/tgr/T127784_body.txt [23:02:02] getting food, will look in half an hour [23:02:07] thx [23:03:40] logstash has lots of this -- https://logstash.wikimedia.org/#dashboard/temp/AVMf0nU5O3D718AOfhAN [23:05:27] now has a dashboard -- https://logstash.wikimedia.org/#/dashboard/elasticsearch/redis-connect-fail [23:06:00] I don't see any failed jobs [23:06:36] legoktm: I don't even get the "Queued $count jobs" message so I assume scheduling them fails? [23:06:47] but then I don't get any error message either [23:08:01] tgr: are you running this on terbium? [23:08:11] legoktm@terbium:~$ cat /home/tgr/T127784_users_test.txt [23:08:11] cat: /home/tgr/T127784_users_test.txt: No such file or directory [23:10:53] legoktm: on mw1152 [23:11:11] some time ago it was said on operations-l that should be used instead [23:15:10] wait, that can't be right [23:15:31] output from even before scheduling the job is missing [23:24:48] legoktm: how do you normally run that script? [23:24:58] it doesn't have require_once RUN_MAINTENANCE_IF_MAIN; [23:25:04] at the end [23:25:07] uhhhh [23:26:13] wow, major facepalm [23:26:23] gj legoktm [23:26:50] tgr: I wrote a script that directly interfaced with the MM classes for SUL stuff since it had some weird requirements [23:27:11] extensions/CentralAuth/maintenance/postRenameNotification.php [23:29:03] theoretically, I can just replace the script name with eval.php, require it, and then add the last two lines manually, right? [23:29:49] I typically copy the scripts into my homedir, and then run mwscript ../../../../../../home/legoktm/foo.php [23:30:19] bd808: "Google declares everyone hosting a website should have to pay money for an SSL certificate." [23:30:30] yeah I guess that's simpler [23:30:53] anomie: don't we have letsencrypt now? [23:31:20] for some use cases [23:31:35] they don't do wildcards or SNI yet as far as I know [23:32:01] and are still in beta [23:32:18] Is their root included in browsers? [23:32:41] I think it is, yes [23:32:49] or at least it's cross signed [23:33:04] Yeah, it's cross signed by quite a few [23:33:14] "DST Root CA X3" on my chrome [23:33:25] and the reason for this touching care of users' privacy: so that no asshole ISP rewrites page HTML to replace Google ads with something else [23:33:27] LE being an intermediatary [23:33:52] anomie: you can check with https://isreedyintheuk.com/ :) [23:34:18] * anomie learned about letsencrypt.org today. Cool! [23:34:49] MaxSem: yeah :/ Oh and we are now promoting results that use our proprietary js system to serve "fast" pages we can cache [23:36:12] MaxSem: but since I have an asshole isp that inspects and injects js payload in pages I actually like the care you mentioned [23:37:52] :( [23:38:21] my 6in4 tunnel helps [23:39:04] but yeah CableOne sucks for doing content injection to provide "alerts" [23:39:25] like "your bill is 5 minutes late! pay with a credit card here!" [23:39:59] their billing cycle is also completely fubar. Bills cut on the 27th and due on the 10th [23:40:46] "well you can sing up for direct withdrawal of funds"