[07:56:14] 10netops, 10Operations: Allow labnet/labnodepool/labvirt to connect to debmonitor hosts/443 - https://phabricator.wikimedia.org/T198375 (10MoritzMuehlenhoff) Thanks, confirmed working fine. All missing hosts were able to ingest their package data and servermon and debdeploy are now tracking the same number of... [07:57:00] elukey: morning sir, let's move forward with cache::text as well? [08:05:45] damn... today it's a bank holiday in Italy [08:06:05] it is in Rome, I am working from Bologna so we can restart :) [08:06:42] hahahha [08:06:50] \o/ [08:18:48] 10Traffic, 10Analytics, 10Operations: Size of headers processed by varnish? - https://phabricator.wikimedia.org/T198152 (10ema) p:05Triage>03Normal [08:51:08] 10Traffic, 10Analytics, 10Operations: Size of headers processed by varnish? - https://phabricator.wikimedia.org/T198152 (10ema) Both [[ https://varnish-cache.org/docs/5.1/reference/varnishd.html#http-req-hdr-len | varnish ]] and [[http://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_bu... [09:00:29] ema: IMHO it could make sense to have varnish configured to allow a slightly higher amount of headers than nginx [09:00:51] ema: mainly because nginx adds additional headers that are passed to varnish [09:02:06] vgutierrez: refactored the vk dashboard this morning (I was invistigating another issue) https://grafana.wikimedia.org/dashboard/db/varnishkafka [09:02:38] it should be easier now to drill down a single caching host [09:02:54] elukey: https://grafana.wikimedia.org/dashboard/db/varnishkafka?panelId=34&fullscreen&orgId=1 [09:03:00] elukey: the units here are wrong, right? [09:03:03] (still need to figure out a way to group the cp hosts by segment) [09:04:12] 136 gigabytes per second... that's almost 1tbps /o\ [09:04:53] I was reviewing it as well, in theory the metric is called "txbytes" and I've put bytes/second, so probably this needs to be fixed [09:04:57] lemme see [09:05:18] seemed to much to me as well :D [09:06:39] so tx bytes is 'Total number of bytes sent' [09:06:47] right [09:06:56] I guess you can benefi from graphite perSecond() function there [09:06:59] needs a perSecond() and it should ok after that [09:07:00] exactly [09:07:52] done! It was in all other ones, the tx traffic was not right [09:07:53] thanks :) [09:08:28] np :) [09:08:37] let me know if you see anything else weird [09:09:21] we are used to see mbps instead of MBps, but we can live with that I guess :) [09:12:51] I got the unit in grafana, can check if there is a better one [09:12:56] vgutierrez: also, https://phabricator.wikimedia.org/T198256#4323723 [09:13:09] re: you qs about prot buffer yesterday [09:13:20] (if you want to follow up in there feel free!) [09:29:21] vgutierrez: that was about the maximum length of a single request header field, not the cumulative length of all headers [09:30:46] ema: hmm right then :D [09:37:59] so, now cp3037 (upload) has been rescued by remote hands [09:38:23] we might want to repool it and see how it behaves, but maybe not right before the weekend starts :) [09:38:45] the management interface is back online as well [09:39:27] awesome [09:39:37] yey.. Monday seems a saner option [09:57:09] elukey: cache::text done, let's go home! [10:01:35] niceeee [10:05:24] ema: we are very close to remove ipsec to jumbo :) [10:05:44] oh yes [10:05:53] nice! [14:13:56] 10Traffic, 10Operations, 10Security-Team, 10Wikimedia-General-or-Unknown: Add restrictive CSP to upload.wikimedia.org - https://phabricator.wikimedia.org/T117618 (10Aklapper) [16:58:25] 10Traffic, 10netops, 10Operations: Can't reach eqiad or esams from Comcast in Portland, Oregon - https://phabricator.wikimedia.org/T198502 (10brion) [17:00:03] 10Traffic, 10netops, 10Operations: Can't reach eqiad or esams from Comcast in Portland, Oregon - https://phabricator.wikimedia.org/T198502 (10ayounsi) a:03ayounsi [17:04:14] 10Traffic, 10netops, 10Operations: Can't reach eqiad or esams from Comcast in Portland, Oregon - https://phabricator.wikimedia.org/T198502 (10ayounsi) 1st look seem to indicate an issue between Telia and Comcast or within Comcast. ``` ayounsi@bast1002:~$ mtr 73.37.60.183 -z --report-wide Start: Fri Jun 29... [17:09:15] 10Traffic, 10netops, 10Operations: Can't reach eqiad or esams from Comcast in Portland, Oregon - https://phabricator.wikimedia.org/T198502 (10brion) ``` $ sudo mtr bast1002.wikimedia.org -z --report-wide Password: Start: 2018-06-29T10:07:32-0700 HOST: Orac.local L... [17:11:59] 10Traffic, 10netops, 10Operations: Can't reach eqiad or esams from Comcast in Portland, Oregon - https://phabricator.wikimedia.org/T198502 (10brion) and in ipv4: ``` $ sudo mtr bast1002.wikimedia.org -z --report-wide -4 Start: 2018-06-29T10:10:31-0700 HOST: Orac.local... [17:17:56] 10Traffic, 10netops, 10Operations: Can't reach eqiad or esams from Comcast in Portland, Oregon - https://phabricator.wikimedia.org/T198502 (10ayounsi) Telia's NOC contacted. [17:23:49] 10Traffic, 10netops, 10Operations: Can't reach eqiad or esams from Comcast in Portland, Oregon - https://phabricator.wikimedia.org/T198502 (10brion) Now getting ``` $ sudo mtr bast1002.wikimedia.org -z --report-wide Password: Start: 2018-06-29T10:22:50-0700 HOST: Orac.local... [17:32:08] 10Traffic, 10netops, 10Operations: Can't reach eqiad or esams from Comcast in Portland, Oregon - https://phabricator.wikimedia.org/T198502 (10bearND) I'm affected, too. Comcast in CO. [17:32:13] 10Traffic, 10netops, 10Operations: Can't reach eqiad or esams from Comcast in Portland, Oregon - https://phabricator.wikimedia.org/T198502 (10ayounsi) Traffic takes another path, GTT to us, HE back, but still no luck, so the issue seems to be within Comcast. Looking at some Netops IRC channels, there seem t... [18:05:02] 10Traffic, 10netops, 10Operations: Can't reach eqiad or esams from Comcast in Portland, Oregon - https://phabricator.wikimedia.org/T198502 (10bearND) translatewiki.net was also affected for me. But both TWN and Gerrit are back for me now. [18:10:31] 10Traffic, 10netops, 10Operations: Can't reach eqiad or esams from Comcast in Portland, Oregon - https://phabricator.wikimedia.org/T198502 (10brion) 05Open>03Resolved Seems to have cleared up for me too now. Marking resolved. \o/ [18:11:27] 10Traffic, 10Analytics, 10Operations: Size of headers processed by varnish? - https://phabricator.wikimedia.org/T198152 (10Nuria) Ya, 8k seems quite a bit, not sure why would we need more than that in either end. [21:09:21] 10netops, 10Operations, 10fundraising-tech-ops: NAT and DNS for fundraising monitor host - https://phabricator.wikimedia.org/T198516 (10cwdent) [22:40:49] 10netops, 10Operations, 10fundraising-tech-ops: NAT and DNS for fundraising monitor host - https://phabricator.wikimedia.org/T198516 (10ayounsi) I don't neither. Note that 8.155.80.208.in-addr.arpa domain name pointer frbast1001.wikimedia.org. [22:58:43] 10netops, 10Operations, 10fundraising-tech-ops: NAT and DNS for fundraising monitor host - https://phabricator.wikimedia.org/T198516 (10cwdent) @ayounsi ah yes thanks, I forgot to update the documentation for that, but just did