[06:52:48] 10netops, 10Operations, 10fundraising-tech-ops: Automate diff and commit of frack ACL - https://phabricator.wikimedia.org/T260655 (10ayounsi) We can use Juniper permissions system to restrict which part of the config a user can change: https://www.juniper.net/documentation/en_US/junos/topics/topic-map/junos-... [08:58:02] 10Traffic, 10Operations, 10Patch-For-Review: Discarded VCL files stuck in auto/busy state cause high number of backend probe requests - https://phabricator.wikimedia.org/T236754 (10ema) 05Open→03Resolved a:03ema Confirmed on a busy text@esams node (cp3050): issuing various VCL reloads, all old VCLs ge... [14:36:13] nice to see T236754 finally closed <3 [14:36:14] T236754: Discarded VCL files stuck in auto/busy state cause high number of backend probe requests - https://phabricator.wikimedia.org/T236754 [14:36:35] our CDN is going green ;P [14:41:18] 10Traffic, 10Operations: External Monitoring alerting on 400 Bad Request errors - https://phabricator.wikimedia.org/T264111 (10colewhite) p:05Triage→03Medium a:03ema [14:43:39] ema: thanks for digging into the watchmouse alerts! I'll see about re-enabling alerting to noc@ again [14:46:43] shdubsh: thanks! [14:46:55] 10Traffic, 10Operations: External Monitoring alerting on 400 Bad Request errors - https://phabricator.wikimedia.org/T264111 (10ema) In the checks definition, under "Advanced" -> "Custom HTTP request headers" we used to specify a header with quotes, which was passed verbatim by CA App Synthetic Monitor. Quotes... [14:50:11] 10Traffic, 10Operations: External Monitoring alerting on 400 Bad Request errors - https://phabricator.wikimedia.org/T264111 (10colewhite) 05Open→03Resolved Notifications re-enabled. [15:18:36] new cloudflare toy https://radar.cloudflare.com/domain/wikipedia.org [15:19:11] "This ranking is calculated using a range of data that Cloudflare has about global Internet traffic patterns." [15:19:31] country wise looks more interesting: https://radar.cloudflare.com/az [15:19:39] like the DDoS stuff [15:20:07] sukhe: might be relevant for censorship too ^ [15:23:12] I am really curious how they calculate popularity for domains they don't front [15:23:47] cdanis: DNS [15:24:35] oh, duh [15:25:47] XioNoX: thanks for sharing. very interesting, this is like an improved version of the Google internet monitor and Oracle intelligence thing [15:30:25] 10Traffic, 10Operations, 10Platform Team Initiatives (API Gateway), 10Story: Client Developer has a cookie-free API call - https://phabricator.wikimedia.org/T258748 (10WDoranWMF) [15:37:50] 10Traffic, 10Operations, 10Platform Team Initiatives (API Gateway), 10Story: Client Developer has a cookie-free API call - https://phabricator.wikimedia.org/T258748 (10Joe) Those cookies are harmless cookies we set at the edge cache for all requests. We could add an exception for the api gateway, but it w... [15:53:40] TIL https://speed.cloudflare.com/ [15:55:10] I don't like that it geolocalise me in the right town, instead of Paris or where my ISP NAT is [15:56:14] looks like my v6 is is maxmind, they probably sync up their IP allocation? [15:56:16] ahah [15:56:32] I only have v4 with my ISP (who otherwise seems quite good) [15:56:34] cdanis: if I'm in a hurry I just browse fast.com [15:56:52] haha I didn't know that one either [15:56:53] not bad [16:04:42] on the topic of getting popular domains through DNS, a lot of people in India have switched to 1.1.1.1. probably because of https://www.cloudflare.com/network/. honestly surprised by some cities they cover, that's an exhaustive list [16:05:56] cdanis: the speed thing is pretty nice, almost reminds of the nmap screens in movies [16:07:20] haha yes exactly [16:12:10] speed.cf sends me to a server in ashburn :P [16:12:39] and rates me at 38ms to there, and 363 down + 197 up. [16:13:06] the latency sounds about right, the speed should be better (I know other speed tests with closer fast endpoints can hit ~800-900 both ways) [16:20:47] did you compute the bandwidth-latency product? :) [16:21:18] no, I didn't :P [16:21:28] either way I'm sure it's just queueing inefficiency crap [16:21:35] aye [16:21:57] which is reasonable in today's internet at these speeds, honestly [16:22:11] if you're testing over 100+100, you don't really have a problem and everything else is just bragging rights [16:22:48] haha yes [16:22:53] even compressed netflix streams for "4k" content only use something like 25 mbps [16:23:11] I have somewhere between 150+15 and 200+50 depending on the time of day [17:23:26] 10Traffic, 10Analytics, 10Operations: varnishkafka 1.1.0 CPU usage increase - https://phabricator.wikimedia.org/T264074 (10herron) p:05Triage→03High [18:32:54] 10Traffic, 10netops, 10Operations, 10ops-eqiad: lvs1016 enp5s0f0 interface errors - https://phabricator.wikimedia.org/T264227 (10BBlack) p:05Triage→03High [18:36:58] 10Traffic, 10netops, 10Operations, 10ops-eqiad: lvs1016 enp5s0f0 interface errors - https://phabricator.wikimedia.org/T264227 (10BBlack)