[07:45:01] 10Traffic, 10Analytics, 10Operations, 10Research, and 6 others: Referrer policy for browsers which only support the old spec - https://phabricator.wikimedia.org/T180921#4224098 (10TheDJ) @gh87 yeah, so that is correct. We indicate as referer policy: "origin, origin-when-crossorigin, origin-when-cross-origi... [07:54:21] mmh there seem to be a few fields listed in vk's format string which don't have a matching webrequest column (cache_status, accept_language, x_analytics, ...) [07:54:34] elukey: we might want to get rid of those? [07:58:38] ema: seems weird, will check later on [07:58:54] elukey: thanks! [07:59:23] maybe they are sent to kafka and then we discard them for some reason in the final webrequest format, but they may be used while refining data etc.. [07:59:56] just to say that there are a lot of steps before the final webrequest that we all see [08:00:20] right [08:00:42] for example, I am pretty sure that the data is available in the raw format [08:01:28] hive -> use wmf_raw; -> show tables [08:02:44] yep all there [09:34:07] mark: hey I found this blast from the past T53700 [09:34:07] T53700: https://login.wikimedia.beta.wmflabs.org/ trapped in an infinite self-redirect - https://phabricator.wikimedia.org/T53700 [09:34:56] I guess we can get rid of the code adding x-f-p to vary at the backend layer https://gerrit.wikimedia.org/r/#/c/434644/? [09:37:40] er [09:37:41] not sure? [09:37:57] i mean, links generated by mediawiki which may not all be protocol relative may contain http vs https and thus need the vary? [09:42:58] requests go through the frontend first though, and those w/ X-Forwarded-Proto != "https" result in either a synth 301 or synth 403 [09:44:55] I guess that's true [09:45:36] mark: hmmm I think you made me discover a bug in treq documentation [09:45:36] IOW, at the backend layer X-F-P is always "https" so adding X-F-P to Vary seems pointless [09:47:35] ema: if that's true then I agree [09:48:42] vgutierrez: you mean about Agent? [09:48:50] indeed [09:48:55] i didn't look hard [09:49:11] so.. on the doc: https://treq.readthedocs.io/en/release-17.8.0/howto.html#agent-customization [09:49:24] and that's what I do in the code [09:49:27] BUT [09:49:38] https://github.com/twisted/treq/blob/master/src/treq/client.py#L101-L104 [09:50:00] it looks like you need to get your own instance of treq.HTTPClient to be able to use your own reactor [09:50:19] why the reactor arg then? [09:50:29] mark: think is in treq.request the reactor passed in **kwargs is only used for the whole timeout callback [09:50:35] s/think/thing/ [09:50:36] right [09:50:40] that feels like a bug to me then [09:50:47] i mean, if you have it as an arg, use it for everything [09:50:52] but I also want to be able to control the reactor used to perform the request as well [09:51:20] let me refactor my code.. (and my tests) [09:51:24] to test if I'm right [10:01:33] yup... [10:02:10] mmh there's the exception of Host headers not matching any of our wildcard certs (eg: wikipedia.com), for those X-F-P is actually unset as they don't match the regex in https_recv_redirect [10:02:44] that's a reasoning we did go through already vgutierrez, didn't we? :) [10:02:47] hmmm usually wikipedia.com traffic will be http and not https [10:03:40] take into account that the user will get a nasty error if $user tries https://*.wikipedia.com [10:09:03] right, so there's no ambiguity really [10:10:06] of course curl -k and friends should be considered :) [10:10:11] if they try https://*.wikipedia.com, cert error, if they try http://*.wikipedia.com X-F-P will be unset and mediawiki will do its redirect magic [10:15:06] vgutierrez: of course, yeah, grr :) [10:24:24] ema: here's my take on the A-L normalisation we talked about yday - https://gerrit.wikimedia.org/r/#/c/434558/ when you have a minute [10:24:49] hm actually, i should also probably check that A-L is in the format we suppose it to be and unset it otherwise [10:25:02] lemme change that [10:28:47] {{done}} [10:51:50] 10Traffic, 10Operations: Identify bots using AES128-SHA maintainers running on toolforge - https://phabricator.wikimedia.org/T194380#4224611 (10MaxBioHazard) I re-run this bot with this option. [11:35:47] 10netops, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack/cable/configure asw2-c-eqiad switch stack - https://phabricator.wikimedia.org/T187962#4224809 (10jcrespo) m1 master is on C, so the following services may go down: * bacula * etherpadlite * librenms * puppet * racktables * rt [11:36:14] 10Traffic, 10netops, 10Operations: cp intermitent IPsec MTU issue - https://phabricator.wikimedia.org/T195365#4224812 (10ayounsi) p:05Triage>03Low [11:36:32] 10Traffic, 10netops, 10Operations: cp intermittent IPsec MTU issue - https://phabricator.wikimedia.org/T195365#4224827 (10ayounsi) [12:28:08] 10netops, 10Cloud-Services, 10Operations: Allocate public v4 IPs for Neutron setup in eqiad - https://phabricator.wikimedia.org/T193496#4224987 (10faidon) OK, so it looks 185.15.56.0/24 is proposed to be used immediately in eqiad, to replace 208.80.155.128/25 in the next ~6 months. Additionally, 185.15.57.0/... [12:40:28] hahaha https://twitter.com/LadyNetwkr/status/999065787385204736 [12:48:54] `/win 12 [12:48:56] hi :) [12:51:57] hi :) [12:52:43] XioNoX: rofl [12:53:27] so yeah I think that incidentally dropping X-F-P from vary at the backend layer would work also for Host header values not matching the https_recv_redirect regex [12:54:09] (A) Host: en.wikipedia.com, X-F-P: https (curl -k for instance) -> 301 https://en.wikipedia.org [12:54:37] (B) Host: en.wikipedia.com, X-F-P unset -> 301 http://en.wikipedia.org [12:55:28] depending on which request comes in first we'd end up caching a different 301 response but without big functionality issues [12:55:39] which sucks a bit though :) [12:55:42] that's potentially-scary, but yeah in theory all the +secure redirects are not cache entries anyways, they're direct "int" responses, so there is no real cache variance. [12:56:48] all secure redirects except for host headers not matching the https_recv_redirect regex, those reach the applayer and do get cached [12:56:55] it's probably safest to stall on moving all the non-canonical redirects over to a separate service though (at which point we have no plain HTTP traffic into varnish. nginx does the universal http->https redirect without any special matching regex) [12:57:25] then we can start ignoring and not varying on XFP with basically-zero risk [12:58:05] cool, yes [13:14:09] bblack: any specific reason for calling text_normalize_path at the backend layer? Wouldn't it be enough to do that stuff once at the frontend? [13:25:35] FYI https://gerrit.wikimedia.org/r/#/c/434032/ is pending a final review (I've improved it a bit today). I'd also like to know when could be a good time to merge it and/or if you have any suggestion for further testing. The compiler is useless for facts changes ;) [13:26:25] for context is a request from XioNoX to have this additional data. [13:30:44] ema: in the backend case, it's only being called for PURGE traffic, since that comes into both layers directly and needs normalization for PURGE reqs to match stored paths. [13:31:45] ema: in general, I was at some relatively-recent time trying to clean up the text+upload path-normalization further... and I got part of the new improvements deployed and then stepped back when I realized I still failed to understand some things about the diffs between MW and RB normalization rules... [13:31:59] ema: so some of that's kind of in a fragile state at this point, need to followup/cleanup further. [13:32:20] bblack: gotcha, so things like https://github.com/wikimedia/puppet/blob/production/modules/varnish/templates/text-common.inc.vcl.erb#L145 could be moved to text-frontend I guess [13:34:18] ema: maybe? I guess so, since I don't think we even try to set varied headers for PURGE, and I think varnish PURGEs all variants together regardless. [13:35:02] ema: You could leave the layer-common "normalize_path" stuff for things that affect cache hashing and must also affect be PURGE, and some other "request normalization" for things like inbound header manglined pre-vary that's fe-only. [13:35:15] s/manglined/mangling/ [13:38:53] ema: in general re T195327 , are they already emitting the Vary header? we should make sure that's working sanely (that the Vary is consistently emitted by the applayer for a given wiki or not, as opposed to some runtime variance of the vary based on whether a particular article yet has language variants existing) [13:38:54] T195327: Normalise the Accept-Language header for REST API requests - https://phabricator.wikimedia.org/T195327 [13:41:26] (which is just a basic general rule really, but it's hard for people to "get" it unless they've stared at the problems it causes sometimes. Any given URI should always emit consistent "Vary: X" headers from the applayer, regardless of other things that might change over time (e.g. the other non-cache-varying aspects of the request, or something that changes over time behind the URI, like whether [13:41:32] a second variant of language has been added to the application's data yet) [13:41:42] ) [13:42:37] that a wiki might have the feature turned off at the wiki level, and not emit the "Vary: A-L" at all, then later start emitting it universally when the feature is turned on for the wiki, is ok. [13:43:05] but having a given URI flip/flop between emitting the Vary or not is problematic. [13:50:34] bblack: no, restbase is not sending 'Vary: A-L' yet I don't think [13:50:41] mobrovac: can you confirm? [13:51:26] 10Traffic, 10Operations: Identify bots using AES128-SHA maintainers running on toolforge - https://phabricator.wikimedia.org/T194380#4225244 (10Vgutierrez) >>! In T194380#4224611, @MaxBioHazard wrote: > I re-run this bot with this option. Thanks! I've already seen it behaving properly :) [13:52:23] a few wikis are though, notably zh [13:57:21] ema: I wonder if wiki behavior is similar (that they only truly vary on first entry of A-L, not the whole string, and thus this normalization should be universal to at least MW+RB) [14:16:01] 10Traffic, 10netops, 10Operations: cp intermittent IPsec MTU issue - https://phabricator.wikimedia.org/T195365#4225342 (10ayounsi) [14:21:17] 10Traffic, 10netops, 10Operations: cp intermittent IPsec MTU issue - https://phabricator.wikimedia.org/T195365#4225379 (10BBlack) I don't have complete thoughts, but keep in mind in general it's complicated to go changing our actual host interface MTUs to anything larger than 1500 ("jumbo frames"), for a few... [14:46:56] bblack: no, that's not the case: https://github.com/wikimedia/mediawiki/blob/master/tests/phpunit/languages/LanguageConverterTest.php#L78 [14:47:27] (forgive the pun) [14:59:31] 10Traffic, 10Operations, 10Wikimedia-Hackathon-2018: Create and deploy a centralized letsencrypt service - https://phabricator.wikimedia.org/T194962#4225611 (10BBlack) >>! In T194962#4221028, @Krenair wrote: >>>! In T194962#4217296, @BBlack wrote: >> * The reference earlier to emulating the puppet fileserver... [15:05:44] mobrovac: ema: does there need to be some rationalization of how MW looks at more than the first entry but RB doesn't? Maybe non-issue as they're different endpoints with different API [15:09:56] (it kind of reminds me of the split on pathname encoding normalization. it seems like it's best on things like this, if at all possible, to keep all our APIs in sync on these basic issues so the world isn't so confusing) [15:33:37] bblack, ema I'll be out for tomorrow's meeting as well :( [15:34:02] (flight @ 17.20) [15:36:02] I'll send my breefing tomorrow [15:42:13] ok [15:42:57] mail sent to kafka-dev BTW [15:48:56] \o/ [15:49:47] vgutierrez: did you attach some cookies to the email to get better results? :-P [15:55:44] actually I'm chatting with Magnus right now, apparently our approach of influencing the TLS behavior on the JVM side through system & security properties could be enough [15:55:52] let's see how it goes [16:55:40] 10Traffic, 10Analytics, 10Operations, 10Research, and 6 others: Referrer policy for browsers which only support the old spec - https://phabricator.wikimedia.org/T180921#4226106 (10Nuria) Latest data about this show internal referrers for safari creeping up from Late march: {F18492888} [19:14:44] ema: bblack: afaik, mw doesn't look at A-L at all [19:14:56] hence for now we only want to activate it for RB [19:15:05] and yeah, we don't send the vary header yet [20:00:21] 10Traffic, 10Operations, 10Wikimedia-Hackathon-2018: Create and deploy a centralized letsencrypt service - https://phabricator.wikimedia.org/T194962#4226649 (10Krenair) Ah so the trick is where we normally `puppet:///` in a file source, between the second and third slashes can be a host name of the machine t... [22:03:28] 10netops, 10Operations: update switch port label from naos.codfw.wmnet to deploy2001.codfw.wmnet - https://phabricator.wikimedia.org/T195422#4227061 (10Dzahn) [22:03:52] 10netops, 10Operations: update switch port label from naos.codfw.wmnet to deploy2001.codfw.wmnet - https://phabricator.wikimedia.org/T195422#4227063 (10Dzahn)