[09:54:08] 10netops, 10Operations, 10fundraising-tech-ops, 10observability: Add alert[12]001 to network ACLs - https://phabricator.wikimedia.org/T260533 (10ayounsi) @herron anything left to do? [10:00:00] 10Traffic, 10netops, 10Operations: Anycast: consistent ICMP packet too big routing - https://phabricator.wikimedia.org/T253732 (10ayounsi) {F32414504} I made a diagram. [12:32:29] 10Traffic, 10Operations: ATS trying to set socket options SO_MARK / IP_TOS - https://phabricator.wikimedia.org/T265911 (10ema) >>! In T265911#6577824, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operations), href=https://sal.toolforge.org/log/BzaaZHUBhxWNv8gI1jEo} [2020-10-26T11:11:10Z... [12:52:10] 10netops, 10Operations, 10Patch-For-Review: fastnetmon misreports attack type and protocol - https://phabricator.wikimedia.org/T241374 (10CDanis) 05Stalled→03Resolved a:03CDanis [12:54:58] 10netops, 10Operations: fastnetmon misreports attack type and protocol - https://phabricator.wikimedia.org/T241374 (10Nintendofan885) [13:33:12] 10netops, 10Operations, 10fundraising-tech-ops, 10observability: Add alert[12]001 to network ACLs - https://phabricator.wikimedia.org/T260533 (10herron) 05Open→03Resolved a:03herron Nope! I think we're good here [14:03:53] 10Traffic, 10Operations, 10Patch-For-Review: Large text objects are randomized to cache backends - https://phabricator.wikimedia.org/T266040 (10BBlack) Notes on the large increase in large_objects_cutoff from late last week: * Graph link: https://grafana.wikimedia.org/d/000000500/varnish-caching?viewPanel=1... [14:29:13] 10HTTPS, 10Traffic, 10Wikidata, 10wikiba.se website: Set HSTS on wikiba.se (force HTTPS) - https://phabricator.wikimedia.org/T232246 (10Nintendofan885) [16:57:46] Hi traffic team, I have a small change that affects the traffic servers. We're switching tls terminators on the node that powers stats.wikimedia.org: nginx, the old one, runs on port 443, and the new one, envoy, is running on 8443. Looking for a +1 from this team for https://gerrit.wikimedia.org/r/c/operations/puppet/+/634669 [17:00:28] 10netops, 10Operations, 10ops-eqiad, 10Sustainability (Incident Followup): eqiad row D switch fabric recabling - https://phabricator.wikimedia.org/T256112 (10wiki_willy) [17:03:14] bblack: --^ (if you have time) [17:10:46] 10netops, 10Operations, 10ops-eqiad, 10User-Kormat, 10User-jijiki: Upgrade eqiad rack D4 to 10G switch - https://phabricator.wikimedia.org/T196487 (10wiki_willy) [17:24:28] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad: patch in FB peering into cr1-eqiad:xe-3/2/1 - https://phabricator.wikimedia.org/T265916 (10Cmjohnson) @robh the circuit at 17/18 with ID 21557287 is connected to cr1 xe-3/2/1 with fiber number 2648 [17:24:31] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad: patch in FB peering into cr1-eqiad:xe-3/2/1 - https://phabricator.wikimedia.org/T265916 (10Cmjohnson) forgot to add I do not have a link light [17:30:20] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad: patch in FB peering into cr1-eqiad:xe-3/2/1 - https://phabricator.wikimedia.org/T265916 (10RobH) [17:31:28] 10netops, 10DC-Ops, 10Operations: patch in FB peering into cr1-eqiad:xe-3/2/1 - https://phabricator.wikimedia.org/T265916 (10RobH) >>! In T265916#6579218, @Cmjohnson wrote: > forgot to add I do not have a link light I show good RX light for the connection. Laser receiver power :... [17:31:38] 10netops, 10DC-Ops, 10Operations: patch in FB peering into cr1-eqiad:xe-3/2/1 - https://phabricator.wikimedia.org/T265916 (10RobH) a:05Cmjohnson→03ayounsi [17:32:47] 10netops, 10DC-Ops, 10Operations: patch in FB peering into cr1-eqiad:xe-3/2/1 - https://phabricator.wikimedia.org/T265916 (10RobH) I've updated the circuit (with its circuit id) and updated the cable (with its cable id and set to status connected) [17:33:17] razzi, elukey: +1! [17:34:02] 10netops, 10DC-Ops, 10Operations: patch in FB peering into cr1-eqiad:xe-3/2/1 - https://phabricator.wikimedia.org/T265916 (10RobH) [17:34:12] 10netops, 10DC-Ops, 10Operations: patch in FB peering into cr1-eqiad:xe-3/2/1 - https://phabricator.wikimedia.org/T265916 (10RobH) [18:18:23] 10Traffic, 10Operations, 10Patch-For-Review: Deprecate TLSv1.2 weak ciphersuites - https://phabricator.wikimedia.org/T258405 (10AntiCompositeNumber) We're getting a few OTRS tickets about this, a note in Tech News or on wikitech-l would have been appreciated. [18:20:03] ema: how can/should we test razzi's change? or should we just merge and cross our fingers? [18:20:23] is this all in puppet? would this just be disabling puppet on ATS nodes, then running on one, and doing a request to make sure it work? [18:20:30] ottomata: easiest thing to do is to 1) disable puppet on all cps 2) run puppet by hand on one 3) test locally with curl --resolve and 4) reenable [18:20:40] ok wasn't sure if there was somehting fancier [18:20:42] ok! [18:20:45] like curl https://stats.wikimedia.org --resolve stats.wikimedia.org:443:127.0.0.1 [18:20:49] yeah, there's not :) [18:20:52] ok [18:20:57] 10Traffic, 10Operations, 10Patch-For-Review, 10User-notice: Deprecate TLSv1.2 weak ciphersuites - https://phabricator.wikimedia.org/T258405 (10Urbanecm) #user-notice is definitely warranted [18:22:47] cdanis: all cp* or just cache::tetxt [18:22:48] text [18:22:52] ? [18:23:04] assuming cache::text pretty much does everything, including stats.wm.org [18:26:42] yup upload is just upload.wm.o and maps [18:33:49] there is no more "misc" [18:42:21] 10Traffic, 10Operations, 10Performance-Team, 10SRE-swift-storage, 10Patch-For-Review: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10dpifke) The Swift object-expirer is running in beta if we want to start testing this there. There are some loose ends bef... [18:55:54] 10Traffic, 10Operations, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs, 10iOS-app-v6.8-Manta-Ray-On-A-Riding-Mower: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC - https://phabricator.wikimedia.org/T264881 (10JMinor) [19:19:00] cdanis: razzi and I stumped ourselves with --resolve (not the first time i've stumped myself with this either) [19:19:27] we are trying to verify that https://stats.wm.org on cp1075 is fowarded to thorium.eqiad.wmnet 8443 [19:19:34] on thorium we are doing [19:19:42] sudo tcpdump port 8443 [19:19:48] and on cp1075 [19:20:09] curl --head https://stats.wikimedia.org/test404_$RANDOM --resolve stats.wikimedia.org:443:127.0.0.1 [19:21:51] but we can't seem to verify that OUR curl sends traffic to thorium 8443 [19:22:19] there is def traffic coming in on 8443 now (assuming natural traffic from load balacners, since cp1075 is pooled) [20:33:19] 10netops, 10DC-Ops, 10Operations: patch in FB peering into cr1-eqiad:xe-3/2/1 - https://phabricator.wikimedia.org/T265916 (10ayounsi) 05Open→03Resolved Interface up. Thanks!