[07:22:31] 10netops, 10Operations, 10cloud-services-team (Kanban): Remove 185.15.56.0/24 from network::external - https://phabricator.wikimedia.org/T265864 (10ayounsi) p:05Triage→03Medium [07:51:14] 10netops, 10Operations, 10observability, 10Security, 10User-jbond: ulog: filter out diffscan from ulog - https://phabricator.wikimedia.org/T265590 (10ayounsi) Shameless plug T238414. [07:53:00] 10Traffic, 10Operations, 10Performance-Team (Radar): Consider collecting more timestamp milestones from ATS-TLS - https://phabricator.wikimedia.org/T265869 (10Gilles) [08:17:08] 10Domains, 10Traffic, 10Education-Program-Dashboard, 10Operations: Create short link for outreachdashboard.wmflabs.org - https://phabricator.wikimedia.org/T146332 (10Ladsgroup) @bd808 Hey, I don't have access to [[https://wikitech.wikimedia.org/wiki/Nova_Resource:Redirects|redirects project]] in cloudVPS a... [08:28:03] 10Domains, 10Traffic, 10Operations, 10Patch-For-Review: wikiknihy.cz - transfer to Wikimedia Czech Republic? - https://phabricator.wikimedia.org/T127573 (10Ladsgroup) 05Open→03Declined So currently, https://wikiknihy.cz/ redirects to cs.wikibooks.org without any issues. I call this declined as it seems... [08:30:00] 10Traffic, 10Operations, 10Sustainability (Incident Followup): upload.wikimedia.org returns HTTP status code 503 for truncated urls, not 404 - https://phabricator.wikimedia.org/T106517 (10Joe) While those urls weren't my original report (which was about truncated URLs), it seems the behaviour has in the mean... [08:47:06] 10HTTPS, 10Traffic, 10Operations, 10Wikispore: Make Wikispore HTTPS-only - https://phabricator.wikimedia.org/T260701 (10Ladsgroup) [09:05:05] 10Traffic, 10Operations, 10Sustainability (Incident Followup): upload.wikimedia.org returns HTTP status code 503 for truncated urls, not 404 - https://phabricator.wikimedia.org/T106517 (10Tgr) Yeah, as mentioned in T106517#1569938, that's intentional. A 400 means the URL couldn't be resolved into a valid fil... [09:08:50] 10netops, 10Operations: Spike of multicast traffic - https://phabricator.wikimedia.org/T212273 (10ayounsi) 05Open→03Declined No mo' multicast. [09:10:38] 10Traffic, 10netops, 10Analytics, 10Operations: Turnilo: per-second rates for wmf_netflow bytes + packets - https://phabricator.wikimedia.org/T263290 (10elukey) @CDanis I added the following config to turnilo on an-tool1005 (staging instance): ` measures: - name: bytes title: Bytes... [09:27:40] 10Traffic, 10netops, 10Analytics, 10Operations: Turnilo: per-second rates for wmf_netflow bytes + packets - https://phabricator.wikimedia.org/T263290 (10faidon) Yay, that's awesome! You can't imagine how much time this would save! I changed the config a little bit. Specifically: * Bits per second is more... [10:11:47] 10Traffic, 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Turnilo: per-second rates for wmf_netflow bytes + packets - https://phabricator.wikimedia.org/T263290 (10elukey) a:03elukey [10:57:10] 10Traffic, 10Operations, 10observability: prometheus-varnish-exporter@frontend.service: Unit entered failed state - invalid character 'C' - https://phabricator.wikimedia.org/T203191 (10Marostegui) [11:54:04] 10netops, 10Operations: Upgrade Junos on asw2-esams - https://phabricator.wikimedia.org/T252631 (10ayounsi) Also https://kb.juniper.net/InfoCenter/index?page=content&id=JSA11086 [14:37:43] 10Traffic, 10Operations: Remove SLAAC IPs from Ganeti hosts - https://phabricator.wikimedia.org/T265904 (10Reedy) [14:44:52] 10Traffic, 10Operations: ATS trying to set socket options SO_MARK / IP_TOS - https://phabricator.wikimedia.org/T265911 (10ema) [15:07:48] 10Traffic, 10Operations: ATS trying to set socket options SO_MARK / IP_TOS - https://phabricator.wikimedia.org/T265911 (10Marostegui) p:05Triage→03Medium [15:27:47] 10netops, 10Operations, 10ops-codfw: patch in FB peering into cr2-eqdfw - https://phabricator.wikimedia.org/T265412 (10Papaul) On site visit set for the 21St [15:35:32] 10netops, 10Operations, 10ops-codfw: patch in FB peering into cr2-eqdfw - https://phabricator.wikimedia.org/T265412 (10RobH) [15:38:44] 10Domains, 10Traffic, 10Education-Program-Dashboard, 10Operations: Create short link for outreachdashboard.wmflabs.org - https://phabricator.wikimedia.org/T146332 (10bd808) >>! In T146332#6559215, @Ladsgroup wrote: > @bd808 Hey, I don't have access to [[https://wikitech.wikimedia.org/wiki/Nova_Resource:Red... [15:38:48] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad: patch in FB peering into cr2-eqdfw - https://phabricator.wikimedia.org/T265916 (10RobH) [16:01:44] 10netops, 10Operations, 10cloud-services-team (Kanban): Enable L3 routing on cloudsw nodes - https://phabricator.wikimedia.org/T265288 (10RLazarus) Note October 27 is also scheduled for the MediaWiki datacenter switchback -- please let's not have both events going at the same time. :) The switchback is sched... [16:25:27] 10Traffic, 10DNS, 10Operations, 10Release-Engineering-Team, 10Patch-For-Review: Create redirect to integration.wikimedia.org/zuul - https://phabricator.wikimedia.org/T207008 (10Jdforrester-WMF) [16:25:42] 10Traffic, 10DNS, 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (CI & Testing services): Create redirect to integration.wikimedia.org/zuul - https://phabricator.wikimedia.org/T207008 (10Jdforrester-WMF) [16:45:54] just curious, why didn't ATS frontend work out after all? what were the problems? [17:02:09] 10netops, 10Operations, 10observability, 10Security, 10User-jbond: ulog: filter out diffscan from ulog - https://phabricator.wikimedia.org/T265590 (10herron) What are the downsides to using iptables rules for this? Also, what are thoughts about creating a generalized iptables "skiplog" chain, and adding... [17:08:55] 10netops, 10Operations, 10observability, 10Security, 10User-jbond: ulog: filter out diffscan from ulog - https://phabricator.wikimedia.org/T265590 (10ayounsi) Downside of having a special iptables rule for diffscan's IP is that it will not be treated anymore as a "random external host" and could cause fa... [17:12:41] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad: patch in FB peering into cr1-eqiad:xe-3/2/1 - https://phabricator.wikimedia.org/T265916 (10RobH) [17:26:14] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad: patch in FB peering into cr1-eqiad:xe-3/2/1 - https://phabricator.wikimedia.org/T265916 (10RobH) [17:26:29] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad: patch in FB peering into cr1-eqiad:xe-3/2/1 - https://phabricator.wikimedia.org/T265916 (10RobH) [17:29:10] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad: patch in FB peering into cr1-eqiad:xe-3/2/1 - https://phabricator.wikimedia.org/T265916 (10RobH) Please note that FB provided a circuitID for the other peering connection but not this one, so its entry for the circuit is N/A. I noticed that we have other circ... [17:30:24] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad: patch in FB peering into cr1-eqiad:xe-3/2/1 - https://phabricator.wikimedia.org/T265916 (10RobH) [17:32:19] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad: patch in FB peering into cr1-eqiad:xe-3/2/1 - https://phabricator.wikimedia.org/T265916 (10RobH) [17:33:17] 10netops, 10Operations, 10ops-codfw: patch in FB peering into cr2-eqdfw - https://phabricator.wikimedia.org/T265412 (10RobH) [17:34:30] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad: patch in FB peering into cr1-eqiad:xe-3/2/1 - https://phabricator.wikimedia.org/T265916 (10RobH) [17:34:38] 10netops, 10Operations, 10ops-codfw: patch in FB peering into cr2-eqdfw - https://phabricator.wikimedia.org/T265412 (10RobH) [17:35:02] 10Domains, 10Traffic, 10Education-Program-Dashboard, 10Operations: Create short link for outreachdashboard.wmflabs.org - https://phabricator.wikimedia.org/T146332 (10Ladsgroup) @Ragesoss requested this (look at the above comments) and he's CTO of wiki edu foundation (the org maintaining the dashboard)