[11:12:46] 10netops, 06Infrastructure-Foundations, 06SRE: Consider blocking IPv4 Options and certain IPv6 extension headers on network edge - https://phabricator.wikimedia.org/T443214 (10cmooney) 03NEW p:05Triage→03Low [11:13:18] 10netops, 06Infrastructure-Foundations, 06SRE: Consider blocking IPv4 Options and certain IPv6 extension headers on network edge - https://phabricator.wikimedia.org/T443214#12501010 (10cmooney) [11:42:28] Morning all! I'm planning on merging this small stack today, if that's OK. It should be a no-op other than idp_test. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1352484 [12:06:41] 10netops, 06Infrastructure-Foundations, 06SRE: Consider blocking IPv4 Options and certain IPv6 extension headers on network edge - https://phabricator.wikimedia.org/T443214#12501246 (10cmooney) [12:53:28] FIRING: KeyholderUnarmed: 1 unarmed Keyholder key(s) on netmon2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [12:58:28] RESOLVED: KeyholderUnarmed: 1 unarmed Keyholder key(s) on netmon2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [15:00:05] 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143#12502510 (10hnowlan) Looks like we've seen a definite drop over the last 6 months but we've stabilised just above 2k. Are these legit messages or is there more tuning needed?... [15:05:22] 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143#12502580 (10ayounsi) We still need to upgrade more switches. Do you have an updated list, so we can compare to our upgrade plans and make sure recently upgraded devices don't... [15:31:43] 10netops, 06Infrastructure-Foundations: Investigate consistent/resilient ECMP - https://phabricator.wikimedia.org/T442996#12502878 (10cmooney) It still could be useful. I've looked at this feature before and how it works, I can't think of any specific downside to using it. SR-Linux implementation / config I... [15:36:38] 10netops, 06Infrastructure-Foundations: Investigate enabling multipath for Pybal BGP group - https://phabricator.wikimedia.org/T443003#12502915 (10cmooney) +1 from me for this. For the active/active future we'll definitely need it. And right now it won't cause an issue as we use different attributes with the... [15:58:14] 10netops, 06Infrastructure-Foundations, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705#12503129 (10cmooney) p:05Triage→03Low [16:01:33] 10netops, 06Infrastructure-Foundations, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705#12503169 (10cmooney) >>! In T442705#12489392, @MLechvien-WMF wrote: > Could we attach a [[ https://wikitech.wikimedia.org/wiki/Incident_response/Incident_sev... [21:33:11] 07Puppet, 10Beta-Cluster-Infrastructure: Cannot rollout Trixie hosts in deployment-prep without rm -rf /var/lib/puppet/ssl - https://phabricator.wikimedia.org/T443373 (10Southparkfan) 03NEW [21:33:51] 07Puppet, 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 06tools-infrastructure-team: Cannot rollout Trixie hosts in deployment-prep without rm -rf /var/lib/puppet/ssl - https://phabricator.wikimedia.org/T443373#12506006 (10Southparkfan) [22:02:40] 07Puppet, 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 06tools-infrastructure-team: Cannot rollout Trixie hosts in deployment-prep without rm -rf /var/lib/puppet/ssl - https://phabricator.wikimedia.org/T443373#12506104 (10dancy) Thanks for the report @Southparkfan. I'll look into this. [22:42:53] 10Packaging, 10SRE-tools, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade prometheus-atlas-exporter - https://phabricator.wikimedia.org/T429672#12506289 (10Lizimedia) ` root@10801bc77cf6:/# curl -O https://apt-staging.wikimedia.org/wikimedia-staging/pool/main/p/prometheus-atlas-exporter/prometheu...