[00:38:07] 10Traffic, 10Operations, 10observability, 10Wikimedia-Incident: cp1083: ats-tls and varnish-fe crashed due to insufficient memory - https://phabricator.wikimedia.org/T241593 (10Krinkle) [00:38:39] 10Traffic, 10Operations, 10observability, 10Wikimedia-Incident: cp1083: ats-tls and varnish-fe crashed due to insufficient memory - https://phabricator.wikimedia.org/T241593 (10Krinkle) Looks like follow-up from T241573. [06:08:42] Krinkle: ack, it looks like a sync benchmark like that (fetching 1k times the favicon) is not representative at all [06:48:43] 10Traffic, 10Operations: Session resumption seems to be broken in ATS for TLSv1.3 - https://phabricator.wikimedia.org/T245419 (10Vgutierrez) This time looks like `openssl s_client -reconnect` is at fault here. It doesn't seem to be working as expected with TLSv1.3: ` vgutierrez@traffic-cache-atstext-buster:~$... [06:49:09] 10Traffic, 10Operations: Session resumption seems to be broken in ATS for TLSv1.3 - https://phabricator.wikimedia.org/T245419 (10Vgutierrez) 05Open→03Declined [06:49:15] 10Traffic, 10Operations, 10Goal, 10Patch-For-Review, 10Performance-Team (Radar): Support TLSv1.3 - https://phabricator.wikimedia.org/T170567 (10Vgutierrez) [12:50:47] 10Traffic, 10Operations: ATS session cache efficiency reduced in TLSv1.3 - https://phabricator.wikimedia.org/T245502 (10Vgutierrez) [12:51:53] 10Traffic, 10Operations: ATS TLS session cache efficiency reduced in TLSv1.3 - https://phabricator.wikimedia.org/T245502 (10Vgutierrez) p:05Triage→03Medium [12:55:33] 10Traffic, 10Operations: ATS TLS session cache efficiency reduced in TLSv1.3 - https://phabricator.wikimedia.org/T245502 (10Vgutierrez) [13:40:25] 10Traffic, 10Operations, 10Wikimedia-Logstash, 10observability, 10User-fgiunchedi: Port varnishlog consumers to log to syslog / logging infra - https://phabricator.wikimedia.org/T227108 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This is now complete! All varnish logging goes through the loggin... [14:21:30] 10netops, 10Operations: Upgrade cr2-esams to JTAC recommended - https://phabricator.wikimedia.org/T237027 (10ayounsi) [14:21:33] 10netops, 10Operations: Upgrade routers - https://phabricator.wikimedia.org/T243080 (10ayounsi) [14:53:23] 10netops, 10Operations, 10ops-esams: 2*10G optics down on cr2-esams - https://phabricator.wikimedia.org/T245520 (10ayounsi) p:05Triage→03High [14:58:12] 10netops, 10Operations, 10ops-esams: 2*10G optics down on cr2-esams - https://phabricator.wikimedia.org/T245520 (10ayounsi) Can probably be combined with T242097. [14:59:24] 10netops, 10Operations, 10ops-esams: 2*10G optics down on cr2-esams - https://phabricator.wikimedia.org/T245520 (10mark) There are multiple 10G LR optics on-site for sure. Longer distance ones, less so. [15:04:24] 10netops, 10Operations, 10Wikimedia-Incident: Juniper HA audit - https://phabricator.wikimedia.org/T191667 (10ayounsi) [17:05:04] 10netops, 10Operations: cr3-knams:xe-0/1/3 down - https://phabricator.wikimedia.org/T244497 (10RobH) [17:27:28] 10netops, 10Operations, 10ops-esams: 2*10G optics down on cr2-esams - https://phabricator.wikimedia.org/T245520 (10RobH) >>! In T245520#5893147, @mark wrote: > There are multiple 10G LR optics on-site for sure. Longer distance ones, less so. @Mark, Can you advise where those spare optics are being stored f... [19:14:54] 10Traffic, 10Android-app-Bugs, 10Operations, 10Wikipedia-Android-App-Backlog, and 4 others: When fetching siteinfo from the MediaWiki API, set the `maxage` and `smaxage` parameters. - https://phabricator.wikimedia.org/T245033 (10LGoto) [19:24:34] 10Traffic, 10Android-app-Bugs, 10Operations, 10Wikipedia-Android-App-Backlog, and 4 others: When fetching siteinfo from the MediaWiki API, set the `maxage` and `smaxage` parameters. - https://phabricator.wikimedia.org/T245033 (10JoeWalsh) @Joe could MediaWiki use a default value other than 0 if a client do... [20:12:10] 10Traffic, 10Android-app-Bugs, 10Operations, 10Wikipedia-Android-App-Backlog, and 4 others: When fetching siteinfo from the MediaWiki API, set the `maxage` and `smaxage` parameters. - https://phabricator.wikimedia.org/T245033 (10Krinkle) @JoeWalsh There's a separate task about the MW default where I just a... [20:17:29] 10netops, 10Operations, 10Wikimedia-Incident: Investigate Juniper storm control - https://phabricator.wikimedia.org/T245192 (10ayounsi) a:05ayounsi→03None [20:21:01] 10Traffic, 10MediaWiki-Debug-Logger, 10Operations: noc.wikimedia.org doesn't route to the docroot when WikimediaDebug browser extension is live - https://phabricator.wikimedia.org/T245552 (10Jdforrester-WMF) [20:24:16] 10netops, 10Operations, 10Wikimedia-Incident: Investigate Juniper storm control - https://phabricator.wikimedia.org/T245192 (10Papaul) a:03Papaul [23:11:06] 10Traffic, 10MediaWiki-Debug-Logger, 10Operations: noc.wikimedia.org doesn't route to the docroot when WikimediaDebug browser extension is live - https://phabricator.wikimedia.org/T245552 (10Krinkle) This has regressed last month as well and was fixed shortly after. Presumably somehing went wrong in the ATS...