[03:32:30] FIRING: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [03:33:15] FIRING: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [03:34:12] FIRING: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [03:37:30] FIRING: [2x] SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [04:02:30] FIRING: [2x] SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [04:07:30] FIRING: [2x] SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [04:17:30] FIRING: [2x] SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [04:18:15] RESOLVED: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [04:20:33] FIRING: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [04:27:30] FIRING: [2x] SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [04:37:30] FIRING: [2x] SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [04:40:33] RESOLVED: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [04:42:30] FIRING: [2x] SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [04:52:30] RESOLVED: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [04:54:12] RESOLVED: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [08:34:08] 06Traffic: liberica triggers one DNS query per HTTPCheck execution/realserver - https://phabricator.wikimedia.org/T372731 (10Vgutierrez) 03NEW [08:34:46] 06Traffic: liberica triggers one DNS query per HTTPCheck execution/realserver - https://phabricator.wikimedia.org/T372731#10072175 (10Vgutierrez) p:05Triage→03Medium [09:27:52] XioNoX, topranks https://phabricator.wikimedia.org/T372731#10072517 --> it looks like you can trigger IPIP encapsulation loops with IPVS /o\ [09:28:23] 06Traffic: liberica triggers one DNS query per HTTPCheck execution/realserver - https://phabricator.wikimedia.org/T372731#10072517 (10Vgutierrez) While working on this task this I found another issue, if we target the service VIP from the load balancer we will trigger double IPIP encapsulation as soon as we have... [09:51:55] 06Traffic: liberica triggers one DNS query per HTTPCheck execution/realserver - https://phabricator.wikimedia.org/T372731#10072572 (10Vgutierrez) A valid workaround for this issue is targeting the realserver IP rather than the service VIP: ` 09:46:24.231961 IP 172.16.14.64 > 10.64.32.89: IP 10.64.130.16.35920 >... [10:18:12] vgutierrez: not sure I understand why it's getting encapsulated twice in that usecase [10:18:42] XioNoX: realservers are configured in IPIP mode on IPVS [10:18:58] so even if the traffic originates in the load balancer as soon as the target address is the VIP, traffic gets balanced [10:19:26] ah ok, I see [10:36:03] is that an issue though? [10:37:07] topranks: healthchecks won't work [10:37:46] also you don't want to depend on the forwarding plane to be able to healthcheck realservers to determine if those should be on the forwarding plane or not [10:38:05] so I'd say is far from ideal :) [10:38:17] ok, I’ll need to dig in properly tomorrow to understand I think (off today but I got curious!) [10:38:26] oh sorry about the ping :) [10:38:36] just mentioning it as a curiosity [10:38:36] Oh no problem at all :) [10:38:47] yeah definitely interesting one [13:58:27] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Request additional mgmt IP range for frack servers - https://phabricator.wikimedia.org/T370164#10073333 (10Papaul) @Dwisehaupt all working thank you. [14:21:48] 10netops, 06Infrastructure-Foundations: Apply egress Source Address Validation on the Wikimedia core routers - https://phabricator.wikimedia.org/T372158#10073453 (10joanna_borun) p:05Triage→03Low [14:22:02] 10netops, 06Infrastructure-Foundations: Apply egress Source Address Validation on the Wikimedia core routers - https://phabricator.wikimedia.org/T372158#10073459 (10joanna_borun) a:03Southparkfan [14:22:30] 10netops, 06Infrastructure-Foundations, 07sre-alert-triage: Alert in need of triage: BGP status (instance cr1-esams) - https://phabricator.wikimedia.org/T372248#10073461 (10ayounsi) p:05Triage→03Low [14:23:25] 10netops, 06Infrastructure-Foundations: Publish, and maintain ASPA records for valid AS14907 upstreams - https://phabricator.wikimedia.org/T372161#10073465 (10joanna_borun) p:05Triage→03Low a:03Southparkfan [14:31:02] ryankemper, inflatador: https://gerrit.wikimedia.org/r/c/operations/puppet/+/991427 got my attention after seeing some errors in trafficserver logs [14:32:06] both wdqs1023 and wdqs1024 are missing some SNI on their TLS material https://www.irccloud.com/pastebin/0B6RIcJP/ [14:32:30] mainly the -experimental ones [14:33:06] query-full-experimental.wikidata.org && query-main-experimental.wikidata.org && query-scholarly-experimental.wikidata.org [14:33:27] could you remove the offending rules or fix the TLS material on the backend servers please? [14:33:44] vgutierrez ACK, taking a look now...I would've thought the envoy proxy settings would've taken care of that [14:34:13] I guess we missed those FQDNs...anyway, will fix by today [14:34:27] thx [14:38:10] created T372779 for the work [14:38:11] T372779: Add missing FQDNs to graph split certs - https://phabricator.wikimedia.org/T372779 [14:40:52] 06Traffic, 06Data-Platform-SRE: Add missing FQDNs to graph split certs - https://phabricator.wikimedia.org/T372779#10073633 (10Vgutierrez) [14:43:24] 10netops, 06Infrastructure-Foundations, 07sre-alert-triage: Alert in need of triage: BGP status (instance cr1-esams) - https://phabricator.wikimedia.org/T372248#10073647 (10ayounsi) 05Open→03Resolved Peer removed. [14:47:22] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: cr1-eqiad: disk failure - https://phabricator.wikimedia.org/T372781 (10ayounsi) 03NEW p:05Triage→03High [15:35:56] 06Traffic, 06Commons, 10MediaWiki-File-management: c:File:K1 1000 M Jeux Olympiques Paris 2024 Podium.jpg serving an outdated original from cache - https://phabricator.wikimedia.org/T372662#10073950 (10BCornwall) Thanks for reporting this, @AntiCompositeNumber. While it's great that it resolved, it's now... [16:05:05] vgutierrez: inflatador: yes these rules just need to be torn down. my bad, I thought I'd already merged a patch. work will be tracked in https://phabricator.wikimedia.org/T371833 [16:06:58] inflatador: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1063849 [16:13:02] 10Wikimedia-Apache-configuration, 06collaboration-services, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), and 3 others: Apache 2.4.61 throws a 403 Forbidden for links containing %3F - https://phabricator.wikimedia.org/T370110#10074161 (10Mstyles) @Aklapper are you okay to resolve this tic... [16:25:53] 06Traffic, 06WMF-Legal, 13Patch-For-Review, 07Privacy: Add no-transform to Cache-Control header - https://phabricator.wikimedia.org/T218618#10074289 (10BCornwall) 05Stalled→03In progress [16:27:06] 06Traffic, 06Data-Platform-SRE: Add missing FQDNs to graph split certs - https://phabricator.wikimedia.org/T372779#10074318 (10RKemper) →14Duplicate dup:03T371833 [16:28:30] 06Traffic, 06Data-Platform-SRE: Add missing FQDNs to graph split certs - https://phabricator.wikimedia.org/T372779#10074339 (10RKemper) The backend rules for the experimental endpoints should have already been torn down, thus the errors. There's no cert changes needed (that we know of) for the *production*... [16:52:02] vgutierrez: which host are you looking at trafficserver logs on? I want to verify removing the backend.yaml rules fixed it [16:52:59] it's fixed, now those result on a redirect to https://www.wikidata.org/wiki/Wikidata:Main_Page [16:53:13] before, they were triggering a 502 [16:53:18] and an SNI error on trafficserver logs [16:53:29] thx :D [19:59:43] 10Wikimedia-Apache-configuration, 06collaboration-services, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), and 3 others: Apache 2.4.61 throws a 403 Forbidden for links containing %3F - https://phabricator.wikimedia.org/T370110#10075380 (10Aklapper) See the previous three comments [22:37:38] 06Traffic, 13Patch-For-Review: Clean up Varnish VCL - https://phabricator.wikimedia.org/T370200#10075826 (10BCornwall) 05Open→03In progress p:05Medium→03Low