[03:33:23] 10Traffic, 10Operations: Renew unified certificates 2017 - https://phabricator.wikimedia.org/T178173#3692711 (10BBlack) [03:33:26] 10Traffic, 10Operations, 10Wikimedia-Planet, 10procurement: *.planet.wikimedia.org SSL cert expires 2017-11-22 - https://phabricator.wikimedia.org/T178444#3692714 (10BBlack) [03:33:30] 10HTTPS, 10Traffic, 10Operations, 10Phabricator, 10procurement: wmfusercontent.org SSL cert expires 2017-11-22 - https://phabricator.wikimedia.org/T178443#3692715 (10BBlack) [09:49:18] we now have cumin aliases for cache types / DCs [09:49:27] \o/ thanks ema! [09:49:29] eg: sudo cumin A:cp-misc_eqiad 'hostname' [09:49:37] volans: ty! [09:49:59] but also: sudo cumin A:cp-upload 'hostname' [09:50:27] cumin really is pretty cool [09:51:09] except the colour scheme ofc! [09:51:25] :) [09:51:27] indeed, I don't know what the developer was thinking... [09:52:44] volans: is there any way to ask cumin to print the expanded list of hostnames that match the query? [09:52:54] eg: A:cp-misc_eqiad -> [09:52:58] cp1045.eqiad.wmnet [09:53:01] cp1051.eqiad.wmnet [09:53:02] [...] [09:54:32] I can of course run something like `sudo cumin --force A:cp-misc_eqiad 'hostname' | awk '/wmnet/ {print $2}'` [09:54:46] but that means running hostname on all machines which is not really necessary :) [09:56:46] ema: so, not super easily, you can open a feature request ofc, in the meanwhile you have 2 possible workarounds [09:57:02] but both manual, you wanted something scriptable? [09:57:09] ideally, yes [09:58:41] * volans start laughing at himself [09:58:46] sudo cumin 'A:cp-misc_esams' 2>&1 | tail -n2 | head -n1 | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" | nodeset -e -S '\n' [09:58:53] * volans continues to laugh at himself.... [10:01:14] ema: ^^^ [10:01:30] alternatively you create a 6-line python script and run it with sudo: [10:01:31] import cumin [10:01:32] from cumin.query import Query [10:01:32] config = cumin.Config() [10:01:32] hosts = Query(config).execute('A:cp-misc_eqiad') [10:01:34] for host in hosts: [10:01:36] print(host) [10:01:54] oh yeah perfect! [10:01:56] thanks :) [10:02:04] (with the cp-misc_eqiad from sys.argv[1] I guess) :D [10:04:46] volans: I guess it is not really necessary to be root in order to execute a query? [10:05:05] no, but to read the config file yes as of now [10:06:13] (in the sense that we can easily change that in puppet if we want to) [10:06:26] oh but I can probably call cumin.Config('/home/ema/cumin.yaml') [10:07:01] mmmh yes, but that would generally be a bad idea [10:07:05] how you keep it in sync? [10:07:17] also the aliases are read from the same directory [10:07:21] ah [10:07:24] where the config is [10:09:21] there's lots of value in the host querying capabilities of cumin I think, perhaps we should try to decouple it a bit from the remote execution part? The former being something that users have access to without special permissions [10:09:56] volans: added a note for rebooting, BTW: https://wikitech.wikimedia.org/w/index.php?title=Cumin&diff=1773106&oldid=1772061 [10:10:32] ema: I agree, in the TODO there is a generic item "generalize backends to allow to return other data too, not only the host certnames." that is related [10:11:50] moritzm: thanks! I might change the graphic to make it more evident, now the eye goes to the echo reboot part, at least mine ;) [10:12:43] ema: as for the permissions, the config file is root only because it might contain passwords to connect to backends [10:12:53] ah right, I was just about to ask [10:13:02] backends = ? [10:13:18] like in the openstack case for WMCS, but that password is public [10:13:41] mmh I see [10:13:43] and the access to those hosts is root only, so we can change the permission of the file in puppet [10:13:52] I don't see a big problem for it, moritzm ? [10:15:10] which file is that? [10:15:18] moritzm: /etc/cumin/* [10:15:20] /etc/cumin/* the directory is roon only [10:15:27] s/roon/root/ [10:15:43] in prod we don't have any secret there [10:16:03] yeah, data in there seems all harmless on sarin [10:16:21] in WMCS we have the openstack password that is already public [10:16:29] and also is a different puppetization [10:16:31] ack, seems fine to change [10:16:38] so independent from the one on sarin/neo [10:24:34] ema: unrelated, cumin has a (very simple) man page finally :) [10:24:44] volans: yeah I've seen it! volans++ [10:27:02] thank you for asking for it and opening the task, sorry it took a while ;) [10:27:07] really, that's great [10:27:48] the last build had a clean lintian :) [10:28:19] what are you talking about, nothing is lintian clean [10:28:44] try it! :D [10:29:14] volans: so here's why I needed to query cumin: https://phabricator.wikimedia.org/P6148 [10:29:16] lintian on stretch or lintian on sid? [10:29:26] on boron, so stretch [10:29:52] with that script you get a clusters file suitable for clusterssh, with both cache and lvs host groups [10:29:59] ema: but why do you need clusterssh if yoy have cumin in the first place? [10:30:20] (I knew it was for that ;) ) [10:30:36] volans: dstat, varnishncsa, ... [10:33:49] so now we just need native cumin+clusterssh integration! :) [10:35:15] ema: remind me how those things uses clusterssh... [10:36:35] volans: so you run eg `clusterssh lvs-eqiad`, and that opens a separate $TERM per host. Then you type a command (eg: dstat 10) and it's executed on all terms at the same time [10:37:30] ema: in 2017, having to use clusterssh on a cluster is a no-no for me :D you shouldn't need it at all IMHO [10:38:08] volans: ok, but I'm the user and you're the developer, so I'm right to begin with [10:38:39] :D [10:42:44] speaking as a user, I also want nodejs bindings [10:43:17] lol [10:44:21] ema: what I'm trying to say is that should be part of monitoring mostly [10:50:58] volans: I've got to go out for lunch, we'll resume the "there's no devops in this world" conversation later [10:51:16] rotfl, have a good lunch [13:31:04] I'll start rebooting cache nodes for kernel upgrades (4.9.25 -> 4.9.30) and to pick up the systemd unit sec thing [13:35:43] or not :) [13:35:57] moritzm mentioned that 4.9.51 will be available soon [13:37:22] it's uploaded, but stuck in NEW, when it's out (hopefully no longer than a day or so), I'll import [13:37:41] sounds great [13:37:47] brings a solid four months of 4.9.x bugfixes compared to 4.9.30 [13:40:41] ema: FYI I'm on cp4028 right now, experimenting with the 3DES stuff [13:41:01] bblack: ack [13:41:47] since turning on the 100% blockage yesterday, our rate of 3DES requests has gone up considerably. Basically, a lot of these supposed UAs are retrying their requests over and over [13:41:57] doh [13:42:03] was true with a 418 response code, still true with 403 [13:42:32] I've tried on cp4028 a temp hack to have it use a 302 redirect to a /sec-warning URL, which returns the sec warning page with a 200 status [13:43:02] no matter what I do, the requests are spammy. They also look somewhat illegitimate (in that many never even try to fetch the logo image on the page) [13:43:34] also, a lot of them are from ChinaNet. So I kind of suspect this is some kind of (caching?) proxy for GFW that's re-spamming 3DES-encrypted requests over and over [13:44:03] it's not really a "problem", it's just an annoying TLS stats spike, and I wish there was a simple way to make them stop, but I doubt it. [13:44:21] could be some kind of disfunctional Baidu crawler too I guess (which hides its UA and pretends to be IE) [13:45:06] anyways, very few IPs, lots of requests, same titles over and over seconds apart :P [13:45:58] probably also explains why stats didn't drop off very linearly as we increased the blockage% over time [13:46:14] whatever these "UAs" are, they just retry harder the more they get blocked, resulting in more requests :P [13:47:46] do they keep on retrying if we return the sec warning with 200 too? [13:48:15] well, I've tried having the original URLs return a 302 -> /sec-warning, and *that* returning a 200 [13:48:36] right, do they stop at that point? [13:48:59] I've been avoiding the obvious "just return sec-warning text + 200 for all URLs over 3DES" so far, because if this is some kind of crawler it's going to cache that and think it's our actual content for all those URLs [13:49:34] no, they don't stop. they follow the redirect, then they try the same URL from the same IP again a few seconds later (the original URL) [13:50:28] so at least 418,403,400, and 302->200 don't stop it [13:51:29] if it's a shitty bot as it seems, 404 would perhaps work but bleah [13:51:37] the repetition of the URLs is crazy, too [13:51:52] e.g. /wiki/List_of_England_Test_cricketers is one I know I've seen here and there in the logs over and over for the past 30 minutes at least [13:53:19] but it's hard to say what the deal really is. the IPs are clearly some kind of bots or proxies of some kind. [13:53:43] but maybe the fetches aren't true retries, just internally-popular URLs [13:54:35] (e.g. there's some site inside china that's popular, and has links to "wikipedia", but when followed from there they hit some internal maybe-caching proxy. So the test cricketers URL is popular in some article there today and we keep seeing proxied requests for it) [13:56:21] that all the IPs I investigate turn up as CHINANET or China Unicom with some province name after it makes me think of those kinds of theories [13:58:20] I think I'm gonna deploy a patch with the 302->200 method. initially it didn't look much different, but it may be making a long-term impact in these rates that's hard to see without the global stats graphing [13:58:53] +1 [14:16:17] ema: can you doublecheck for sanity? https://gerrit.wikimedia.org/r/#/c/384982/ [14:17:53] bblack: sure [14:21:32] bblack: we might want to add if (X-C-P ~ "des" on line 277? Just in case we serve any actual content under /sec-warning somewhere? [14:22:12] I think it's fair to assume we don't :) but also, that's how everyone who's not on IE8/XP views /test-sec-warning to validate/test the output looks sane [14:22:26] ok! [14:25:09] bblack: the vcl seems fine, give me a few minutes to test it [14:32:27] bblack: works as advertised! [15:30:05] 302->200 did kill off the reqrate [15:30:14] (with cacheable 200, that was a last-minute idea I hadn't tried on 4028) [15:41:22] 10netops, 10Operations, 10ops-ulsfo: connect new office link to asw-ulsfo - https://phabricator.wikimedia.org/T176350#3694265 (10RobH) 05stalled>03Resolved I neglected to resolve this, but it was handled within a day or so of our onsite work being completed. [16:19:05] 10Traffic, 10Operations, 10ops-ulsfo: rack/setup/install cp40(29|3[012]).ulsfo.wmnet - https://phabricator.wikimedia.org/T178423#3694375 (10RobH) [17:07:56] bblack: using a tunnel from ulsfo to eqiad over NTT would reduce the latency from 78ms to 64ms [17:08:03] just a random data point [17:08:44] vs our existing transports from ulsfo->eqiad? [17:09:03] bblack: yep, vs ulsfo->eqord->eqiad [17:09:21] we have ulsfo->codfw->eqiad as a path too, I guess eqord is better? [17:09:53] bblack: yeah, prefered/shorter is via eqord [17:10:11] anyways, interesting. I guess NTT has some slightly-shorter path between ulsfo<->eqiad on their network, probably direct even. [17:10:49] that would also allow us to have backup tunnels going the same path at our transport Waves [17:11:41] yeah, it might be worth configuring them in some cases, at least as a last-resort option at the ready [17:11:46] anyway, that would open up a whole lot of possibilities [17:13:36] yeah, and it's nice that we can get the better latency too (from their more-robust connectivity map in the US than ours I'm guessing!). But it still feels like a bad idea to make it our primary option. [17:14:18] (even in the US, I'd still rather have a pair of real transports hooked up to ulsfo, I mean, instead of 1 real one and the tunnel, or worse just the tunnel) [17:14:38] but I guess I can't really articulate well why. just seems dicey to rely on it as anything but a backup option [17:15:18] yeah, I agree, unless we *really* need those 14ms, which I don't think we do [17:15:55] but hey, if we get an NTT transit into sg before our transport is ready, we could experiment with bringing up real traffic on tunneled transport too for the first few countries :) [17:16:17] bblack, paravoid: "I can assist you with these changes. I can change the MTU for all your ports globally. Do you have a date/tim in mind to make these changes? What mtu do you want?" [17:16:37] 9092? [17:16:53] I forget what the absolute ethernet max is [17:17:25] ask them what they support or recommend [17:17:25] He says "globaly" because I asked if it was US only, or worldwide (eg. Singapore). So I guess we have our answer [17:17:32] awesome :) [17:17:46] from http://www.us.ntt.net/newsletter/0510_didyouknow.html they say 4550, but I'll ask, yeah [17:17:48] 9000 is the "standard" definition of ethernet jumbo [17:18:03] but really even if they only offer 4K or whatever, it's plenty to tunnel with [17:18:28] timeing wise, anytime is fine I guess? [17:18:42] wasn't ~4500 the MTU for FDDI? makes me wonder heh [17:20:09] XioNoX: yeah, just avoid any overlap with e.g. esams depooling work, or potential Telia/Zayo scheduled maint [17:20:13] but should be fine [17:24:37] I'd stay conservative rather than go as high as possible [17:24:54] avoiding pmtu would be best [17:43:19] speaking of pmtu [17:43:36] that cloudflare link about mtu issues reference in a commit review earlier, from like 2015 [17:44:05] was talking about how they (and apparently others?) are just fixing their ipv6 mtu at 1280 to avoid various ipv6 pmtu issues that exist due to various tunneling still employed for users [17:44:46] https://blog.cloudflare.com/path-mtu-discovery-in-practice/ -> summary at bottom "Like many other companies we've reduced the MTU on IPv6 to safe value of 1,280. We'll consider bumping it to improve performance once we get more confident that our other fixes are working as intended." [17:45:39] the only ipv6-relevant "other fix" was their PMTU daemon, which was to work around ECMP issues with ICMP Packet Too Big routing, which we shouldn't face since we don't use ECMP inside our DCs to reach the caches [17:46:28] I do wonder, though, how robust that ipvs schedule_icmp stuff really is [17:46:57] does it look deeper in the ICMP and route it to the right cache for the actual connection, even when it's an intermediate router that sent the Too Big from a different IP than the user? [17:47:41] bblack: I can investigate [17:48:15] paravoid, bblack, "9014 is the max. this would be 9018 on Cisco XR or junos. This is fairly common for customers. I need to step out for about an hour. If you're in the US want to do this around 3pm CT? You can call me if you want or we can sync up over email" [17:49:20] 10Traffic, 10Operations, 10ops-ulsfo: rack/setup/install cp40(29|3[012]).ulsfo.wmnet - https://phabricator.wikimedia.org/T178423#3694640 (10RobH) [17:51:13] nice! [17:53:29] paravoid, sooo, 9000? [17:54:06] XioNoX: to be clear what I'm asking, since my wording above is poor: the lvsNNNN have the setting net.ipv4.vs.schedule_icmp=1, which is supposed to make sure that at least if the TCP peer of an LVS connection sends us some kind of related ICMP message, it also gets forwarded to that same cache as the rest of the TCP connection. [17:54:51] all our transport links are 9192, GTT MPLS 9000, Frack 8900 (SRX limitation) [17:54:55] XioNoX: but what I'm not sure about is if an intermediate router sends an ICMP-too-big for PMTU for that connection, does ipvs schedule it correctly, or it goes to some other cache because it's just looking at the router's source address on the icmp packet, not the client IP of the connection (which I think is contained inside?) [17:54:58] 10Traffic, 10Operations, 10ops-ulsfo: rack/setup/install cp40(29|3[012]).ulsfo.wmnet - https://phabricator.wikimedia.org/T178423#3694663 (10RobH) cp4030 was ready to install, but now it won't let me connect to com2. I was connected, but my session timed out while I was on via com2, and now it won't let me ba... [17:55:21] bblack: IPv6 tunneling is dying or probably already dead [17:55:30] used to be popular but is very quickly dying [17:55:59] paravoid: AT&T (which is a big provider in the US) still has issues. I use them and see the same thing, but someone else reported about it over the weekend. [17:56:25] 10Traffic, 10Operations, 10ops-ulsfo: rack/setup/install cp40(29|3[012]).ulsfo.wmnet - https://phabricator.wikimedia.org/T178423#3694668 (10RobH) [17:56:34] (basically, with AT&T standard 6rd-based ipv6 turned on, they couldn't connect to our sites. disabling ipv6 fixed it, and I asked them to try manually reducing their ethernet mtu on the client box, and that also fixed it over v6 to us) [17:56:58] oh 6rd, that's kinda different [17:57:03] but weird that it failed? [17:57:17] XioNoX: dunno... go for what they recommend I'd say [17:57:31] hard to really make a decision without knowing [17:57:33] my read on that AT&T issue is the router firmware in the routers they place in people's homes sucks. their actual 6rd connection has an MTU of 1472 due to tunneling, but the router doesn't tell the user's laptop about that, so it tries to use 1500 and fails. [17:57:35] also please cc: noc@ :) [17:58:15] (or at least, fails sometimes to some endpoints. the user was reporting success to other sites I think, just failure with us) [17:59:25] so I tend to think these mtu-broken AT&T users mostly-work most places because v6 pmtud works. but with us, maybe it doesn't.... [17:59:43] (hence wondering about ipv6 and schedule_icmp from intermediate routers sending icmp-too-big) [18:00:22] 10Traffic, 10Operations, 10ops-ulsfo: rack/setup/install cp40(29|3[012]).ulsfo.wmnet - https://phabricator.wikimedia.org/T178423#3694688 (10RobH) [18:00:24] maybe some at&t device sent us a toobig to try to get us to back down to 1472, and the cache never got it due to schedule_icmp not being as robust as we hope [18:02:51] (and if ipv6 pmtud from intermediates is broken for us in a way that's not easy to fix, clamping our side down to e.g. 1280 mtu might make everything better for such users) [18:05:19] paravoid: yeah, if 9014 is common with their customers, I'd guess that's fine. I suggesed 9000 to be on the safe side [18:22:51] I guess I can ask zayo as well [18:41:11] 10Traffic, 10Operations, 10ops-ulsfo: rack/setup/install cp40(29|3[012]).ulsfo.wmnet - https://phabricator.wikimedia.org/T178423#3694773 (10RobH) [20:19:17] 10Traffic, 10Operations, 10ops-ulsfo: rack/setup/install lvs400[567].ulsfo.wmnet - https://phabricator.wikimedia.org/T178436#3695031 (10RobH) [20:56:26] 10Traffic, 10Operations, 10ops-ulsfo: rack/setup/install lvs400[567].ulsfo.wmnet - https://phabricator.wikimedia.org/T178436#3695131 (10RobH) [21:03:34] 10Traffic, 10Operations: rack/setup/install lvs400[567].ulsfo.wmnet - https://phabricator.wikimedia.org/T178436#3695158 (10RobH) a:05RobH>03None [21:04:01] 10Traffic, 10Operations: rack/setup/install lvs400[567].ulsfo.wmnet - https://phabricator.wikimedia.org/T178436#3692125 (10RobH) This is now ready for someone in #traffic to take over for service implementation/replacement of the existing lvs400[1-4]. [21:09:32] 10Traffic, 10Operations, 10ops-ulsfo: decommission lvs400[1-4].ulsfo.wmnet - https://phabricator.wikimedia.org/T178535#3695169 (10RobH) [21:09:36] 10Traffic, 10Operations: rack/setup/install lvs400[567].ulsfo.wmnet - https://phabricator.wikimedia.org/T178436#3692125 (10RobH) [21:09:39] 10Traffic, 10Operations, 10ops-ulsfo: decommission lvs400[1-4].ulsfo.wmnet - https://phabricator.wikimedia.org/T178535#3695169 (10RobH) [23:53:13] 10Traffic, 10Operations, 10ops-ulsfo, 10Patch-For-Review: replace ulsfo aging servers - https://phabricator.wikimedia.org/T164327#3695537 (10RobH)