[11:12:46] 10netops, 06Infrastructure-Foundations, 06SRE: Consider blocking IPv4 Options and certain IPv6 extension headers on network edge - https://phabricator.wikimedia.org/T443214 (10cmooney) 03NEW p:05Triage→03Low [11:13:18] 10netops, 06Infrastructure-Foundations, 06SRE: Consider blocking IPv4 Options and certain IPv6 extension headers on network edge - https://phabricator.wikimedia.org/T443214#12501010 (10cmooney) [12:06:43] 10netops, 06Infrastructure-Foundations, 06SRE: Consider blocking IPv4 Options and certain IPv6 extension headers on network edge - https://phabricator.wikimedia.org/T443214#12501246 (10cmooney) [14:22:46] 06Traffic, 10Gerrit, 07Browser-Support-Google-Chrome, 07Wikimedia-production-error: Error 403: Wikimedia Error when viewing Gerrit using Google Chrome - https://phabricator.wikimedia.org/T439314#12502236 (10Meno25) 05Open→03Resolved a:03ssingh >>! In T439314#12491623, @ssingh wrote: > Hi everyone... [15:00:05] 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143#12502510 (10hnowlan) Looks like we've seen a definite drop over the last 6 months but we've stabilised just above 2k. Are these legit messages or is there more tuning needed?... [15:02:18] hi! I'd like some help with increasing HTTP timeouts for the service behind textlb. [15:02:50] atsukoito: rough topic.. we like decreasing timeouts [15:03:14] Is it eating the resources? [15:03:27] potentially :) [15:03:37] so, what service are you targeting? [15:03:50] it is for a single domain only, `airflow-dumps.wikimedia.org`, and for a specific set of URLs [15:04:33] is it only hieradata/common/profile/trafficserver/backend.yaml, or should it be done in hieradata/role/common/cache/text.yaml too [15:04:52] the former [15:04:54] just backend.yaml [15:05:23] 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143#12502580 (10ayounsi) We still need to upgrade more switches. Do you have an updated list, so we can compare to our upgrade plans and make sure recently upgraded devices don't... [15:07:28] if it is just a long response from server, then it is `@pparam=proxy.config.http.transaction_no_activity_timeout_out=300`, right? or also `@pparam=proxy.config.http.transaction_active_timeout_out=300`? [15:08:29] atsukoito: no_activity would help if TTFB is awfully high [15:08:42] yes [15:08:52] _active_timeout helps if need to support sending huge payloads to slow clients [15:09:07] i see, thanks.. [15:12:32] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1361474 does this look okay? [15:17:36] can I please ask anyone from traffic to chip-in? cc vgutierrez [15:18:21] atsukoito: so.. that's a 4 minutes TTFB? [15:19:10] yeah.. we do have one EP that is super-slow. when the current dumps finishes, we will be re-doing it so it won't take lots of time [15:19:39] this EP is only used by DPE SRE around few days a month [15:20:03] 3m45 TTFB ;_; [15:22:20] atsukoito: so can you put that endpoint behind idp? [15:26:19] it is not a public EP, airflow-dumps is behind idp (it manages authentification by itself) [15:26:24] oh ok [15:27:21] https://phabricator.wikimedia.org/P97071 [15:29:08] vgutierrez: thanks [15:29:20] np [15:31:43] 10netops, 06Infrastructure-Foundations: Investigate consistent/resilient ECMP - https://phabricator.wikimedia.org/T442996#12502878 (10cmooney) It still could be useful. I've looked at this feature before and how it works, I can't think of any specific downside to using it. SR-Linux implementation / config I... [15:36:38] 10netops, 06Infrastructure-Foundations: Investigate enabling multipath for Pybal BGP group - https://phabricator.wikimedia.org/T443003#12502915 (10cmooney) +1 from me for this. For the active/active future we'll definitely need it. And right now it won't cause an issue as we use different attributes with the... [15:58:14] 10netops, 06Infrastructure-Foundations, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705#12503129 (10cmooney) p:05Triage→03Low [16:01:33] 10netops, 06Infrastructure-Foundations, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705#12503169 (10cmooney) >>! In T442705#12489392, @MLechvien-WMF wrote: > Could we attach a [[ https://wikitech.wikimedia.org/wiki/Incident_response/Incident_sev... [16:16:06] Krinkle: I'm going to merge in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1341936 [16:21:05] notably, I'm not going to merge in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1341937/ as that needs more discussion [16:21:43] okay [16:26:54] vgutierrez: it doesn't go thru, `Request served via cp3069 cp3069, Varnish XID 206311702`, I'll go look for logs [16:27:09] meeting right now [16:27:21] no worries, just added a note here [16:48:08] 20261102.15h51m37s CONNECT: attempt fail [INACTIVE_TIMEOUT] to 10.2.2.91:30443 for host='airflow-dumps.wikimedia.org' connection_result=Connection timed out [110] error=Connection timed out [110] attempts=3 url='https://airflow-dumps.discovery.wmnet:30443/ui/grid/structure/mediawiki_dumps_sql_xml_large_a_to_z_full?include_downstream=false&include_upstream=false&limit=1&order_by=-run_after' [16:49:10] 06Traffic, 06DBA, 10MediaWiki-Action-API: internal_api_error_DBConnectionError on some en/ja/pl/zh projects - https://phabricator.wikimedia.org/T443274 (10AlexisJazz) 03NEW [16:49:39] 06Traffic, 06DBA, 10MediaWiki-Action-API, 06SRE: internal_api_error_DBConnectionError on some en/ja/pl/zh projects - https://phabricator.wikimedia.org/T443274#12503620 (10AlexisJazz) [16:53:29] 06Traffic, 06DBA, 10MediaWiki-Action-API, 06SRE: internal_api_error_DBConnectionError on some en/ja/pl/zh projects - https://phabricator.wikimedia.org/T443274#12503655 (10DamianZaremba) Cluebot III on enwiki has also been seeing this intermittently. First instance was (during a page edit call); ` 2026-11-... [16:57:46] 06Traffic, 06DBA, 10MediaWiki-Action-API, 06SRE, 13Patch-For-Review: internal_api_error_DBConnectionError on some en/ja/pl/zh projects - https://phabricator.wikimedia.org/T443274#12503700 (10Ladsgroup) We are on one dc and one of the three replicas for the ES has gone down due to hardware issues leaving... [18:01:38] 06Traffic, 06DBA, 10MediaWiki-Action-API, 06MediaWiki-API-Platform-Team, and 3 others: internal_api_error_DBConnectionError on some en/ja/pl/zh projects - https://phabricator.wikimedia.org/T443274#12504161 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup [18:02:31] 06Traffic, 06Commons: HTTP 429 error on image requests on Commons (non-WMF native apps treated as bot-traffic) - https://phabricator.wikimedia.org/T413570#12504180 (10ssingh) >>! In T413570#12491859, @TheDJ wrote: >>>! In T413570#12491590, @ssingh wrote: >> Thanks for sharing the error message string 0c640b1....