[02:34:28] 10Traffic, 10netops, 10Operations: IPv6 ~20ms higher ping than IPv4 to gerrit - https://phabricator.wikimedia.org/T211079 (10ayounsi) >>! In T211079#4812380, @faidon wrote: > - It's been a while, but I believe an import statement in the neighbor block overrides the parent one in its entirety, and does not su... [09:01:40] bblack: mmh, can we actually use X-Next-Is-Cache? It seems that we're unsetting it very early in vcl_recv https://github.com/wikimedia/puppet/blob/production/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb#L104 [09:05:00] I'd say that all our current if(!X-Next-Is-Cache) guards after cluster_be_vcl_switch are always true [09:07:26] aha, no! We re-set it explicitly in set_backend__ https://github.com/wikimedia/puppet/blob/production/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb#L17 [09:07:31] ignore me [09:12:35] bblack: ok, I think https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/478680/ is ready to go! [09:15:11] 10Traffic, 10Analytics, 10Operations, 10Performance-Team: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10Gilles) @Anomie very good point, I think it will be very hard for someone to find out about such a whitelist. Things will work for them on... [09:17:20] 10Traffic, 10Analytics, 10Operations, 10Performance-Team: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10Gilles) >>! In T210484#4779199, @TheDJ wrote: > what about ?debug=true ? We already vary on that right ? might as well vary which set of... [09:17:50] 10Traffic, 10Operations, 10Continuous-Integration-Infrastructure (Slipway), 10Patch-For-Review, 10User-ArielGlenn: CI jobs for authdns linting need to run on Stretch - https://phabricator.wikimedia.org/T205439 (10hashar) CI runs two jobs for `operations/dns`: operations-dns-tabs ==== Does a git shallow... [09:19:01] 10Traffic, 10Analytics, 10Operations, 10Performance-Team: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10Gilles) >>! In T210484#4794749, @fdans wrote: > Analytics needs x-analytics in every request, not only in debugging ones but we don't need... [09:27:03] 10Traffic, 10Operations, 10Performance-Team, 10media-storage: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Gilles) [09:27:10] 10Traffic, 10Operations, 10Performance-Team, 10media-storage: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Gilles) p:05Triage>03Normal [10:11:09] 10Traffic, 10Operations, 10Continuous-Integration-Infrastructure (Slipway), 10Patch-For-Review, 10User-ArielGlenn: CI jobs for authdns linting need to run on Stretch - https://phabricator.wikimedia.org/T205439 (10BBlack) @hashar - I'm re-working the tools for the linting checks on operations/dns in the c... [14:04:18] 10Traffic, 10Operations, 10Patch-For-Review: Migrate most standard public TLS certificates to CertCentral issuance - https://phabricator.wikimedia.org/T207050 (10Vgutierrez) [14:25:26] ema: https://github.com/grafana/grafana/pull/14449 [14:27:40] cdanis: that was fast :) [14:32:17] not a very complicated change, just a matter of knowing where to put the X, i think :) [14:48:44] cool! [14:51:51] I feel like spending hours chasing down knowing where to put the X is half of what I do sometimes :) [14:54:48] XioNoX: when you get a chance, can you delete the authdns IPv6 routing from the routers? It's these (removed from DNS already too): https://gerrit.wikimedia.org/r/c/operations/puppet/+/478939/2/modules/role/manifests/authdns/data.pp [14:55:05] just for consistency so we don't have something that's unconfigured/documented elsewhere still in router config [15:02:38] 10Traffic, 10Operations, 10Continuous-Integration-Infrastructure (Slipway), 10Patch-For-Review, 10User-ArielGlenn: CI jobs for authdns linting need to run on Stretch - https://phabricator.wikimedia.org/T205439 (10BBlack) @hashar - So where we're at now is that we just need our CI switched to a Docker wit... [15:08:55] 10Traffic, 10Operations, 10Patch-For-Review: Migrate most standard public TLS certificates to CertCentral issuance - https://phabricator.wikimedia.org/T207050 (10Vgutierrez) [15:09:11] so.. all the HTTPS services listed in T207050 are already using certcentral managed TLS certificates \o/ [15:09:12] T207050: Migrate most standard public TLS certificates to CertCentral issuance - https://phabricator.wikimedia.org/T207050 [20:21:13] 10Traffic, 10netops, 10Operations: IPv6 ~20ms higher ping than IPv4 to gerrit - https://phabricator.wikimedia.org/T211079 (10ayounsi) The issue is not present in eqdfw, eqiad, esams, as HE is not sending those routes through the RS. Pushing the "avoid HE prefixes from the RS" change to those sites to ensure... [20:26:03] 10Traffic, 10netops, 10Operations: Free up 185.15.59.0/24 - https://phabricator.wikimedia.org/T211254 (10ayounsi) Talked a bit over IRC, tldr, the rationale has been added to the beginning of the task's description. Triggering conversation was about removing WMCS 185.15.56.0/23 from prod ACLs. [20:28:01] brief sanity check if anyone has a sec, gerrit.wikimedia.org (cobalt) is now behind varnish? and since misc is no longer a thing(?) it would possibly be picked up via regular collection for webrequests and land in hadoop as text source? [20:28:23] no one can find the logs :D [20:31:43] 10Traffic, 10netops, 10Operations, 10IPv6: Fix IPv6 autoconf issues once and for all, across the fleet. - https://phabricator.wikimedia.org/T102099 (10herron) On a personal level I firmly believe interface config belongs in the OS install phase (as described in option 1) and ideally never modified by Puppe... [20:31:45] 10Traffic, 10netops, 10Operations: IPv6 ~20ms higher ping than IPv4 to gerrit - https://phabricator.wikimedia.org/T211079 (10ayounsi) 05Open>03stalled All done, marking the task as stalled until T204281 [20:32:01] 10netops, 10Operations, 10Performance-Team (Radar): Stop prioritizing peering over transit - https://phabricator.wikimedia.org/T204281 (10ayounsi) [20:32:05] 10Traffic, 10netops, 10Operations: IPv6 ~20ms higher ping than IPv4 to gerrit - https://phabricator.wikimedia.org/T211079 (10ayounsi) [20:49:24] 10Traffic, 10Analytics, 10Operations, 10Performance-Team: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10Milimetric) >>! In T210484#4812997, @Gilles wrote: >>>! In T210484#4794749, @fdans wrote: >> Analytics needs x-analytics in every request,... [21:49:20] 10netops, 10Operations: Outbound BGP graceful shutdown - https://phabricator.wikimedia.org/T211728 (10ayounsi) p:05Triage>03Normal [21:57:29] fwiw mutante clued me in that only wmfusercontent.org portion is behidn varnish there [22:00:38] and that part got stalled for other reasons. so no avatars on gerrit yet [22:00:56] 10netops, 10Operations: Replace accepted-prefix-limit with prefix-limit - https://phabricator.wikimedia.org/T211730 (10ayounsi) p:05Triage>03Low