[00:29:27] 10Traffic, 10Analytics, 10Operations, 10Browser-Support-Apple-Safari, and 3 others: Referrer policy for browsers which only support the old spec - https://phabricator.wikimedia.org/T180921 (10leila) [06:00:06] morning! not sure if already reported by on cp4029 varnishmtail says [06:00:07] cp4029 varnishmtail-backend[33406]: flag provided but not defined: -logfds [08:02:16] elukey: yeah, reported by mutante already. Thanks! [08:16:58] godog: it looks like -logfds is gone from the latest mtail version [08:23:07] https://github.com/google/mtail/commit/8cd80a91576f1fba57832049bd6ae39d08ffe3fd [08:25:48] so yeah that breaks stdin support https://github.com/google/mtail/issues/3 [08:30:22] womp womp [08:31:18] that's unfortunate to say the least [08:34:10] in the immediate I guess we could remove rc24 from stretch-wikimedia [08:39:55] godog: yeah. rc19 also wasn't a lucky release for us (T225604) so maybe we should stick with rc5 from stretch-backports? [08:39:56] T225604: log spam from mtail 3.0.0~rc19 on wezen - https://phabricator.wikimedia.org/T225604 [08:41:33] sounds good to me, for sure until we find a solution [08:47:15] I'm going to upload rc5 from stretch-backports into stretch-wikimedia btw [08:56:17] I haven't done any tests but my hunch is that a named pipe might work, provided the write behaves when the pipe is full [08:56:24] the writer even [09:01:22] reading from stdin seems like a good thing to support though, who knows why upstream says logfds can't possibly work in 8cd80a91576f1fba57832049bd6ae39d08ffe3fd [09:09:09] indeed, I'm not sure, I can ask upstream to clarify though [09:45:50] 10Traffic, 10Operations, 10Goal, 10Patch-For-Review, 10User-fgiunchedi: Deprecate python varnish cachestats - https://phabricator.wikimedia.org/T184942 (10fgiunchedi) >>! In T184942#5320841, @Gehel wrote: >>>! In T184942#5306856, @fgiunchedi wrote: >> I don't know offhand, although I'd be interested to k... [09:53:04] godog: thanks that would be great! [09:53:26] ok will do [10:02:31] 10Traffic, 10Operations, 10Goal, 10Patch-For-Review, 10User-fgiunchedi: Deprecate python varnish cachestats - https://phabricator.wikimedia.org/T184942 (10Gehel) >>! In T184942#5323915, @fgiunchedi wrote: > Since the maps performance dashboard with graphite metrics is broken anyways ATM I think it makes... [10:29:28] in different news, we can now go ahead with https://gerrit.wikimedia.org/r/c/operations/puppet/+/520187 [10:30:17] nice! [10:32:49] indeed, feels good! let me know what you think [10:33:56] looking [10:41:59] godog: tried the patch on traffic-text-stretch.traffic.eqiad.wmflabs, interestingly the first puppet run works as advertised, but a second one fails [10:42:09] Error: Could not enable varnishstatsd-default: [10:42:09] Error: /Stage[main]/Varnish::Logging::Backend/Varnish::Logging::Statsd[default]/Systemd::Service[varnishstatsd-default]/Service[varnishstatsd-default]/enable: change from false to true failed: Could not enable varnishstatsd-default: [10:46:40] also passing ensure = absent to base::service_auto_restart does not remove the service from /etc/debdeploy-autorestarts.conf [10:52:50] oh, interesting, thanks for the test ema! [10:52:59] going to lunch now, will take a look later [12:27:29] ema: review updated to fix the systemd service, for base::service_auto_restart on that host I can't find statsd on /etc/debdeploy-client/autorestarts.conf [12:28:35] godog: ty! [12:28:38] godog: it's under /etc/debdeploy-autorestarts.conf [12:28:48] etc/debdeploy-client/autorestarts.conf that can be ignored, if no entries get dropped from there it's a (harmless) bug in the debdeploy puppet support code, no need to handle in our code [12:29:02] "in your code" I meant [12:30:23] ema: I can't find that file managed by puppet though [12:31:45] ah interesting [12:31:56] ok we can ignore that then [12:32:03] actually, it should get handled correcttly already, it's done in modules/base/manifests/service_auto_restart.pp [12:32:17] if absented, the file_line also gets removed [12:32:52] let's see if the latest PS I got the => alignment right [12:32:55] moritzm: the file in question right now is /etc/debdeploy-autorestarts.conf, which does not seem to be handled by puppet at all [12:34:54] godog: updated patch tested successfully, +1! [12:34:55] that file is completely unused and probably only present on hosts which have been installed a while ago, the first iteration of that code used /etc/debdeploy-autorestarts.conf, but was eventually moved to /etc/debdeploy-client/autorestarts.conf [12:35:31] ema: neat, thank you! [12:36:54] moritzm: ack, we might want to remove it from the fleet to avoid confusion [12:37:27] sure [12:49:06] moritzm: done [12:50:24] thanks! [13:25:44] godog: while we're in mtail-land: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/522081/ [14:01:39] bblack: this seems sane enough for you? https://gerrit.wikimedia.org/r/c/operations/dns/+/521414 it's the CR adding the ncredir-lb DNS records [14:12:31] vgutierrez: checking, will take a few, I'm digging through the history on the IPs in that range [14:12:48] np :) [14:13:05] (some were decommed years ago, but for stuff that was hardcoded freaking everywhere, trying to see if I can find a "cleanest" one so our initial stats are more pristine) [14:13:20] oh ok, that makes sense :) [14:16:11] vgutierrez: 232 is probably the best one. It was decommed from a relatively-low-traffic use back in 2014 and apparently never re-used in all the interim things since :) [14:16:42] (.232 on the IPv4 I mean. IPv6 I didn't look at, but probably doesn't matter as much there!) [14:16:52] for both eqiad & codfw, right? [14:17:11] yeah [14:17:19] ack, let me update the CR then [14:17:22] just s/225/232/ on your IPv4 stuff [14:18:03] 225, 226, and some others had known bigger uses (e.g. old mobile or bits or login IPs, before those were all folded to text and/or decommed) [14:18:53] of course this is just pedantry, it would've worked fine [14:19:15] we still get a fair amount of DNS lookups for those older LB hostnames, but hopefully not traffic for the old IPs, since the DNS lookups have been dead a long time [14:19:18] (years) [14:19:29] IPv6 should be :9 to use the same "slot" than 232 [14:19:34] ok sure :) [14:22:24] updated [14:26:33] vgutierrez: the geo-resources file is still like PS1 [14:26:42] arg, right [14:27:50] hmmm [14:28:00] also, give me a sec to think about the mapping stuff [14:28:15] (re: only 2 Dcs defined) [14:28:40] I think with undefinied_datacenters_ok it's ok to have fewer DCs in the mapfile than in the resource definition, but maybe not the other way around [14:28:40] yeah, I thought it was ok cause we set "undefined_datacenters_ok = true" [14:29:13] I think that only works the other direction [14:29:31] (i.e. the country map entries don't all have to contain "bh", which is why that config line was added) [14:29:46] so yeah [14:29:52] so we need to duplicate the whole map with 2 DCs instead of the whole 5? [14:30:07] probably the simplest solution is just to define the rest of the datacenter names, using their nearest core [14:30:23] oh ok [14:30:26] ulsfo => { addrs_v4 => 208.80.153.225, addrs_v6 => 2620:0:860:ed1a::2 } # codfw [14:30:28] etc [14:30:33] gotcha [14:32:32] that's not ideal either, but I think we can live for ncredir [14:33:24] the unideal part is later if we choose to depool a core edge in the admin_state file, e.g. "geoip/generic-map/codfw => DOWN", it will fail to depool the ulsfo_DC->codfw_IP hack above [14:33:55] but we depool core sites relatively-rarely, and ncredir doesn't get much traffic, etc [14:34:36] we could push a feature patch to make things work more-properly with a new config key that allows undefinedness in the other direction, too. But my Q1 task is to replace all that code with something new anyways, so it seems like a waste to work on the old version much now. [14:37:46] relatedly, I had been meaning to get rid of the discovery-geo-maps file as well, now that undefined_datacenters_ok is available and set [14:38:00] err I guess the file is "discovery-map" [14:38:24] but it still suffers the same basic problems I think [14:39:07] vgutierrez: actually hold on a sec before you merge it up [14:39:19] let me really re-read the code and documentation, maybe I have this all wrong heh :) [14:40:47] from https://github.com/gdnsd/gdnsd/wiki/GdnsdPluginGeoip#undefined_datacenters_ok--false I interpreted that my approach was ok, specifically: "a map M might define 3 datacenters named A, B, and C, but a resource using map M might only define result addresses for datacenters B and C in its dcmap" [14:41:07] but of course you're the one that knows about gdnsd :) [14:41:40] hmmm yeah I've become lost in the history of this stuff I think [14:41:52] the kind of undefinedness that goes the other way, is something that became default-ok a long time ago [14:42:08] undefined_datacenters_ok is this direction you're trying to use now heh [14:42:20] so yeah [14:42:24] remove the fake DC lines :) [14:42:37] and the setting does work for discovery-map, I should clean that up too [14:44:26] ack [14:45:19] I live to confuse! :) [14:45:30] PS5 uploaded [14:47:58] no problem, better safe than sorry :) [14:51:38] so.. from an eqsin preferred country (TH), I'm getting codfw: ncredir-lb.wikimedia.org has address 208.80.153.232 [14:52:42] sounds right! [14:53:37] yep [14:56:02] 10HTTPS, 10Traffic, 10Operations, 10Goal, 10Patch-For-Review: Create a secure redirect service for large count of non-canonical / junk domains - https://phabricator.wikimedia.org/T133548 (10Vgutierrez) [15:00:27] 10Domains, 10Traffic, 10Operations, 10WMF-Legal, 10Patch-For-Review: Move wikimedia.ee under WM-EE - https://phabricator.wikimedia.org/T204056 (10tramm) Another two weeks passed... How can we help to bring some action? [15:06:20] btw traffic folks, I'm rolling out a new version of python3-conftool, no behavioral changes expected (and only minimal code changes). so far live on mw-canary and cp-canary hosts [15:08:14] thanks cdanis [15:12:00] vgutierrez: keep in mind on the back burner: once we have ganeti at all the edges, we can expand ncredir everywhere too and cut some latency out for those few reqs that use non-canonical links [15:12:24] yeah [15:12:49] that will be a quick win once we get ganeti deployed everywhere [15:57:48] btw traffic folks I'm going to depool cp4022 for just a few minutes to test the conftool change [16:44:02] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Development services): Set up a subdomain for Phame to enable caching - https://phabricator.wikimedia.org/T226044 (10greg) [17:23:21] bblack: https://www.cambus.net/fuzzing-dns-zone-parsers/ would be cool if he also tackles gdnsd :) Maybe worth leaving a comment on the HN thread? https://news.ycombinator.com/item?id=20410685 [17:40:13] I did on lobsters dupe thread: https://lobste.rs/s/nn92ty/fuzzing_dns_zone_parsers [17:49:26] cool :) [17:55:09] 10Traffic, 10Operations: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293 (10Cmjohnson) I am removing the ops-eqiad tag on this task, if you need additional dc ops work please add the tag back. [18:27:51] 10netops, 10Operations: Standardize cross confederation BGP policies - https://phabricator.wikimedia.org/T227808 (10ayounsi) p:05Triage→03Low [18:28:02] (conftool upgrade is now everywhere) [18:29:12] 10netops, 10Operations: Cleanup confed BGP peerings and policies - https://phabricator.wikimedia.org/T167841 (10ayounsi) [18:29:17] 10netops, 10Operations: Standardize cross confederation BGP policies - https://phabricator.wikimedia.org/T227808 (10ayounsi) [21:57:01] 10Traffic, 10Operations: Wikipedia is unavailable on Symbian phone's btowsers - https://phabricator.wikimedia.org/T227828 (10Krinkle) [21:57:07] 10Traffic, 10Operations: Wikipedia is unavailable on Symbian phone's browsers - https://phabricator.wikimedia.org/T227828 (10Krinkle) [22:01:19] 10Traffic, 10Operations: Wikipedia is unavailable on Symbian phone's browsers - https://phabricator.wikimedia.org/T227828 (10Aklapper) Thanks! Do you have any way to find out which exact Symbian version you are using? And/or would you share in public which exact phone this is about?