[08:54:53] 10netops, 06Infrastructure-Foundations, 06SRE: Should we channelize unused QSFP28 ports on QFX5120s to provide 'buffer' for 10G upgrades? - https://phabricator.wikimedia.org/T367408#9897254 (10ayounsi) Can we move the cables instead of moving the servers ? For example Port 44 to 47 can be used right away at... [09:46:48] 06Traffic, 06Data-Platform-SRE: Migrate Cloudelastic load balancing to IPIP encapsulation (LVS) - https://phabricator.wikimedia.org/T367511#9897612 (10Vgutierrez) [09:46:49] 06Traffic, 10CirrusSearch, 06Discovery-Search, 10Elasticsearch, 06Infrastructure-Foundations: Migrate services behind high-traffic2 LVS to IPIP encapsulation - https://phabricator.wikimedia.org/T367312#9897611 (10Vgutierrez) [09:47:50] 06Traffic, 10CirrusSearch, 06Discovery-Search, 10Elasticsearch, 06Infrastructure-Foundations: Migrate services behind high-traffic2 LVS to IPIP encapsulation - https://phabricator.wikimedia.org/T367312#9897613 (10Vgutierrez) [09:51:49] 10netops, 06Infrastructure-Foundations, 06SRE: Should we channelize unused QSFP28 ports on QFX5120s to provide 'buffer' for 10G upgrades? - https://phabricator.wikimedia.org/T367408#9897652 (10cmooney) @ayounsi perhaps I was a little quick to conclude all the blocks were assigned, you are correct. The advan... [11:05:55] 10netops, 06Traffic, 06Infrastructure-Foundations: drmrs/esams/magru LVS : remove cross-rack links - https://phabricator.wikimedia.org/T367731 (10ayounsi) 03NEW [11:13:11] 10netops, 06Traffic, 06Infrastructure-Foundations: POPs LVS : remove public vlan trunking - https://phabricator.wikimedia.org/T367732 (10ayounsi) 03NEW [12:36:39] 10netops, 06Traffic, 06Infrastructure-Foundations: drmrs/esams/magru LVS : remove cross-rack links - https://phabricator.wikimedia.org/T367731#9898286 (10Vgutierrez) Don't be to aggressive with this one, we could need to rollback at some point, let's wait a few weeks at the very least [12:39:55] vgutierrez: did you have a recent-ish patch about switching to hashing based on URL for the upload bwlimits? [12:40:01] I swear I saw it but I can't find it [12:40:18] cdanis: I haven't got the time to push that one yet [12:40:27] right but is it pending somewhere? [12:40:31] nope [12:40:33] ah ok [13:17:25] 06Traffic, 06Data-Platform-SRE, 10Elasticsearch, 06Infrastructure-Foundations: Migrate services behind high-traffic2 LVS to IPIP encapsulation - https://phabricator.wikimedia.org/T367312#9898524 (10Gehel) [13:29:09] 06Traffic, 06Data-Engineering, 10Observability-Logging: Upgrade ulsfo hosts to haproxy 2.8.10 - https://phabricator.wikimedia.org/T367756 (10Fabfur) 03NEW [13:49:13] 10netops, 06Traffic, 06Infrastructure-Foundations: drmrs/esams/magru LVS : remove cross-rack links - https://phabricator.wikimedia.org/T367731#9898761 (10ayounsi) Of course ! not planning on doing it today :) The task is there to not forget. [15:04:53] 10netops, 06Traffic, 06Infrastructure-Foundations: POPs LVS : remove public vlan trunking - https://phabricator.wikimedia.org/T367732#9899192 (10ayounsi) p:05Triage→03Low [15:05:14] 10netops, 06Traffic, 06Infrastructure-Foundations: drmrs/esams/magru LVS : remove cross-rack links - https://phabricator.wikimedia.org/T367731#9899194 (10ayounsi) p:05Triage→03Low [15:05:51] 10netops, 06Infrastructure-Foundations: Capirca setup for routed Ganeti VMs - https://phabricator.wikimedia.org/T367265#9899196 (10ayounsi) p:05Triage→03Medium a:03ayounsi [15:08:40] 06Traffic, 10Data-Platform-SRE (2024.06.17 - 2024.07.07), 13Patch-For-Review: Consider migrating Search Platform-owned clusters to IPIP encapsulation (LVS) - https://phabricator.wikimedia.org/T365616#9899264 (10Gehel) [15:19:21] 06Traffic, 06SRE, 10SRE-swift-storage: Rise in ms-fe2* TCP retransmits since 11:40 UTC today - https://phabricator.wikimedia.org/T367056#9899352 (10ayounsi) We had a quick look at the network side and couldn't find any smoking gun. In the future if you could run a packet capture on both sides `tcpdump -i in... [15:19:38] 10netops, 06Traffic, 06Infrastructure-Foundations: POPs LVS : remove public vlan trunking - https://phabricator.wikimedia.org/T367732#9899368 (10cmooney) > Slight drawback here is that in ulsfo/eqsin LVS traffic towards public hosts (if any) will hair-pin through the routers and back to the switches. I thin... [16:24:03] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 22.2R3 - https://phabricator.wikimedia.org/T364092#9899763 (10cmooney) [16:25:02] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 22.2R3 - https://phabricator.wikimedia.org/T364092#9899765 (10cmooney) >>! In T364092#9766653, @ayounsi wrote: > Both Junos 22.2R3-Sx and Junos 22.4R3 are latest recommended. fyi, I went with 22.4R3 in magru. Doh, I went with 22.2R... [16:56:39] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 22.4R3 - https://phabricator.wikimedia.org/T364092#9899880 (10cmooney) [18:45:28] Hi, I changed the footer icons in bottom of every page which is 2KB larger but since it gets heavily cached on client side + it's desktop only, it shouldn't cause any issues but heads up anyway: https://phabricator.wikimedia.org/T256190#9900464 [18:48:22] <3 [18:57:00] 06Traffic, 06DC-Ops, 10ops-ulsfo, 06SRE, 13Patch-For-Review: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9900583 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host cp4042.ulsfo.wmnet with OS b... [19:15:55] 06Traffic, 06DC-Ops, 10ops-ulsfo, 06SRE, 13Patch-For-Review: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9900601 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host cp4042.ulsfo.wmnet with OS bulls... [19:16:08] 06Traffic, 06DC-Ops, 10ops-ulsfo, 06SRE, 13Patch-For-Review: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9900602 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host cp4042.ulsfo.wmnet with OS b... [20:05:09] 06Traffic, 06DC-Ops, 10ops-ulsfo, 06SRE: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9900818 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host cp4042.ulsfo.wmnet with OS bullseye completed: - cp404... [20:54:37] 06Traffic, 06DC-Ops, 10ops-ulsfo: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9901190 (10BCornwall) [21:41:28] 06Traffic, 06DC-Ops, 10ops-ulsfo, 13Patch-For-Review: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9901323 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1002 for host cp4043.ulsfo.wmnet with OS bullseye [21:44:54] 06Traffic: Deprecate TLS 1.2 - https://phabricator.wikimedia.org/T367821 (10BCornwall) 03NEW [21:45:35] 06Traffic: Deprecate TLS 1.2 - https://phabricator.wikimedia.org/T367821#9901342 (10BCornwall) p:05Triage→03Low [22:26:34] 06Traffic, 06DC-Ops, 10ops-ulsfo, 13Patch-For-Review: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9901512 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1002 for host cp4043.ulsfo.wmnet with OS bullseye ex... [22:28:59] 06Traffic, 06DC-Ops, 10ops-ulsfo, 13Patch-For-Review: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9901515 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1002 for host cp4043.ulsfo.wmnet with OS bullseye [23:08:07] FIRING: [8x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.96:443 @ cp4043 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_text - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS [23:13:07] RESOLVED: [8x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.96:443 @ cp4043 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_text - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS [23:14:07] FIRING: [8x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.96:443 @ cp4043 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_text - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS [23:15:01] 06Traffic, 06DC-Ops, 10ops-ulsfo: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9901639 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1002 for host cp4043.ulsfo.wmnet with OS bullseye completed: - cp4043 (**P... [23:19:07] RESOLVED: [8x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.96:443 @ cp4043 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_text - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS [23:25:47] 06Traffic, 06DC-Ops, 10ops-ulsfo: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9901678 (10CDobbins)