[08:53:31] 10Traffic, 06Operations, 10Phabricator: phab.wmfusercontent.org "homepage" yields a 500 - https://phabricator.wikimedia.org/T166120#3291219 (10fgiunchedi) I saw there's indeed a phabricator redirect map (in php), would that be the place @mmodell or some other place like apache? [08:56:54] 10Traffic, 10MediaWiki-General-or-Unknown, 06Operations, 06Security-Team: Mediawiki replies with 500 on wrongly formatted CSP report - https://phabricator.wikimedia.org/T166229#3291220 (10fgiunchedi) doh :( I'm for changing that to 400, we log those anyways on our side, thoughts @Anomie @Bawolff ? [10:31:00] ema: o/ good morning [10:45:04] joal: FWIW he's out today/tomorrow, in case you are waiting ! [10:47:54] and tomorrow too AFAIK [10:55:03] Ah, right [10:55:07] Thanks godog [12:53:38] 10Traffic, 10MediaWiki-General-or-Unknown, 06Operations, 06Security-Team: Mediawiki replies with 500 on wrongly formatted CSP report - https://phabricator.wikimedia.org/T166229#3291687 (10Anomie) Yes, 400 would seem to be more semantically appropriate than 500. I have no idea whether a 400 response also sh... [13:04:48] 10Traffic, 10Analytics, 06Operations, 13Patch-For-Review: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557#3291750 (10Ottomata) Just nocied this ticket again! Can/should we move forward with this? [13:10:58] 10Traffic, 10Analytics, 06Operations, 13Patch-For-Review: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557#3291758 (10BBlack) From my perspective, where we last stalled out is waiting for Analytics to say it's ok to merge https://gerrit.wikimedia.org/r/#/c/2... [13:24:16] 10Traffic, 10netops, 06Operations, 10Pybal: Frequent RST returned by appservers to LVS hosts - https://phabricator.wikimedia.org/T163674#3291785 (10elukey) Sorry for the late response! >>! In T163674#3274227, @BBlack wrote: > So from the above, apache really has 3 different modes of operation: > > 1) def... [14:42:51] 10Traffic, 10MediaWiki-Cache, 10MediaWiki-JobQueue, 06Operations, and 2 others: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan - https://phabricator.wikimedia.org/T124418#3292114 (10Krinkle) >>! In T124418#3276610, @BBlack wrote: > Not resolved, as the purge graphs can attest! Which graph... [14:46:49] bblack: I'll try that (sudo service varnish-frontend restart) - regarding volatile, do you have other suggestions? Perhaps it's possible to subt the html.erb template inside the incl.vcl.erb template somehow? [14:46:53] Does puppet or erb have a fileread feature? [14:47:06] or should we version the file? [14:48:03] bblack: restart worked for varnish-frontend. [14:48:17] new version is now showing up [14:50:52] E.g. turning the inc.vcl into an .erb template where instead of synth(regsub(std.fileread(), "%error", error)) we'd do synth(regsub("<%= @errorpagehtml %>", "%error", error)) [14:50:54] or something like that [14:51:04] not sure if multi-line strings are a problem [14:51:13] Meh, it'd also need escaping [14:51:16] there's lots of ways to tackle the problem, none are pretty [14:51:32] the easiest is just to live with duplication and inline the html in the pages [14:51:39] (in the VCL I guess I mean) [14:52:05] bblack: Well, we're already inlining the templates in a way, but generating different html files for each case at puppet time [14:52:15] So none of that happens at run time [14:52:17] like the browsersec one does [14:52:24] Yeah, I've got a patch for that one. [14:52:31] varnish has an "extended quote format" or whatever that uses {""} to avoid escaping issues [14:52:55] ... a patch that would make it hard to change it without restarting varnishes, and it's one we expect to change frequently during cipher removal process [14:53:01] Maintaining them in two different repos in 5 differnet programming languages, and escaping and file formats is just not maintainable. [14:53:29] Yeah, we need to find a solution that just works with vcl reload [14:54:00] probably some generator to use pre-commit makes the most sense [14:54:09] sort like we do for apache redirect files [14:54:36] bblack: Perhaps we can use $title as unique portion of file names, and pass that through to the vcl.erb fileread() call as well. errorpage is now its own resource type in my patch. [14:54:46] Then we'd put it in a directory managed by puppet so old ones are cleared. [14:55:18] although having it somehow embedded in the vcl files dynamically would be preferable. [14:55:22] just have to figure a way [14:55:37] to substitue the html.erb template and then in turn embed that inside vcl.erb [14:59:24] I'll try to circle back to this in a bit (and the PURGE ticket mess), conf call starting now [15:02:09] bblack: Thanks. I've got 15+ use cases for error pages, most of which I intent to puppetise with relative ease. Only 2 (vcl/common/errorpage and browsersec) are used from within vcl, so it's a relatively small problem. All the others are fine to read them from an html file and not subject to restart needs and such. [15:33:16] Krinkle: where are the other "use cases for error pages" centralized at for the basic template (the structural html + css) [15:36:14] 10Traffic, 06Operations, 13Patch-For-Review: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557#3292319 (10Nuria) [15:36:32] 10Traffic, 10Analytics, 06Operations, 13Patch-For-Review: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557#1803407 (10Nuria) 05duplicate>03Open [16:00:06] 10Traffic, 06Operations, 06Performance-Team, 13Patch-For-Review: Evaluate/Deploy TCP BBR when available (kernel 4.9+) - https://phabricator.wikimedia.org/T147569#3292408 (10BBlack) So I've stared at NavTiming graphs, and honestly it's hard to read any notable difference in the tea leaves. I'm still invest... [16:22:48] 10Traffic, 06Operations, 10Phabricator: phab.wmfusercontent.org "homepage" yields a 500 - https://phabricator.wikimedia.org/T166120#3292533 (10mmodell) Well apache would probably be a better choice but we could do it in php land for sure. [16:43:50] bblack: The ones currently stashed in mediawiki-config.git/errorpages (that are referenced in puppet manifests and apache files and just assume that repo to exixst in a certain place). [16:44:09] As well as the erropage for dynamicproxy/toollabs proxy in puppet [16:44:16] T113114 [16:44:16] T113114: Make all wiki-facing error pages consistent - https://phabricator.wikimedia.org/T113114 [16:45:00] Took about 1-2 years to make them look consistent, but design has various ideas for this, and not having to do this all again would be very preferable of course. [16:52:29] ok [18:52:58] bblack: ping regarding errorpage.inc [18:55:13] https://gerrit.wikimedia.org/r/#/c/350966/ - also see rest of topic:errorpage but this is the first patch that starts changing it [18:57:16] Apparently there's a way to call one erb template from another - https://docs.puppet.com/puppet/4.10/lang_template_erb.html#calling-puppet-functions-from-templates [19:09:08] I assume templates don't need to be ERB syntax necessarily, so I suppose we could use call_function('template', [' ...']) but that requires the file to be in puppet, probably can't reference the already-expanded /etc/varnish/errorpage.html file. But given its Ruby, File.read may be available? [19:09:25] And then some Ruby string wrangling to escape it neatly for valid inclusion as a VCL string [20:42:37] bblack: Okay, I think I've got it. Using template() directly in puppet to assign content to a variable, and then use {" and ERB in VCL to embed the string. [20:42:53] https://gerrit.wikimedia.org/r/#/q/topic:errorpage+(status:open+OR+status:merged) [21:55:22] 10netops, 06Operations, 10ops-eqiad: ripe-atlas-eqiad is down - https://phabricator.wikimedia.org/T163243#3293469 (10Cmjohnson) 05Open>03Resolved Faidon got this back up and running today.