[07:52:55] 10Traffic, 10Operations: puppet restarts nginx instead of reloading it on ncredir servers - https://phabricator.wikimedia.org/T233518 (10Vgutierrez) p:05Triage→03Normal [09:12:48] 10Traffic, 10Operations, 10Patch-For-Review: puppet restarts nginx instead of reloading it on ncredir servers - https://phabricator.wikimedia.org/T233518 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez [14:44:29] I am checking benchmarking restoration on codfw, please ping if that creates network saturation, CC XioNoX [14:45:58] thx for the heads up! [14:46:10] cross dc or only codfw? [14:46:18] always within DC [14:46:41] 668 MBYtes/s [14:46:59] I will let you see if your monitoring is good enough to spot it :-D [14:58:19] remember when I said "1Gbit/s is going to be enough for databases"? [14:58:32] we may have to review that in the next years [14:59:41] if our disaster recovery objective has to go down but our monolythic dataset keeps growing [15:02:58] 10netops, 10Operations, 10observability, 10Patch-For-Review: Deploy ripe-atlas-tools for ad-hoc network tests - https://phabricator.wikimedia.org/T232711 (10fgiunchedi) [15:04:56] 10netops, 10Operations, 10observability, 10Patch-For-Review: Deploy ripe-atlas-tools for ad-hoc network tests - https://phabricator.wikimedia.org/T232711 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Completed! I've updated the ripe atlas documentation at https://wikitech.wikimedia.org/wiki/RIPE_At... [15:30:26] 10Traffic, 10Analytics, 10Operations: Images served with text/html content type - https://phabricator.wikimedia.org/T232679 (10Ottomata) 05Open→03Declined Nuria I think we can decline this yes? Doing so, feel free to reopen if I am wrong. [15:31:25] 10netops, 10Icinga, 10Operations, 10observability: scs monitoring missing in Icinga - https://phabricator.wikimedia.org/T233318 (10fgiunchedi) Sounds great! Adding ssh + ping for starters should be quite easy in puppet [15:38:23] 10Traffic, 10Analytics, 10Operations: Cookies and misc services caching - https://phabricator.wikimedia.org/T232453 (10fdans) cc @Aklapper gasserandreas seems to be moving stuff around our board, could you take a look at it? Seems malicious. [18:07:46] 10netops, 10Operations: asw2-d2-eqiad crash - https://phabricator.wikimedia.org/T233645 (10ayounsi) p:05Triage→03High [18:10:59] 10netops, 10Operations: asw2-d2-eqiad crash - https://phabricator.wikimedia.org/T233645 (10ayounsi) [18:22:08] 10Traffic, 10MobileFrontend, 10Operations, 10Readers-Web-Backlog (Tracking): Sections on some mobile pages are not collabsable - https://phabricator.wikimedia.org/T233373 (10Jdlrobson) This is likely a caching issue. We recently moved some code around and had reports that this might not have gone as smooth... [18:38:19] 10Traffic, 10MobileFrontend, 10Operations, 10Readers-Web-Backlog (Tracking): Sections on some mobile pages are not collabsable - https://phabricator.wikimedia.org/T233373 (10AntiCompositeNumber) It only occurred while logged-out. I'm not logged-out and on the mobile site often, so I haven't noticed it aga... [18:43:09] 10Traffic, 10MobileFrontend, 10Operations, 10Readers-Web-Backlog (Tracking): Sections on some mobile pages are not collabsable - https://phabricator.wikimedia.org/T233373 (10Jdlrobson) I'd expect no reports after the end of this week. If so I think we can safely assume caching and close these tickets. Than... [18:45:53] 10netops, 10Operations, 10Wikimedia-Incident: asw2-d2-eqiad crash - https://phabricator.wikimedia.org/T233645 (10ayounsi) [18:56:05] 10Traffic, 10Core Platform Team, 10Operations, 10Performance-Team, and 6 others: Serve Main Page of WMF wikis from a consistent URL - https://phabricator.wikimedia.org/T120085 (10Izno) This one will probably require a user notice before WMF rollout and maybe even a "do you guys want us to do this" question... [19:29:19] 10Traffic, 10Core Platform Team, 10Operations, 10Performance-Team, and 6 others: Serve Main Page of WMF wikis from a consistent URL - https://phabricator.wikimedia.org/T120085 (10Krinkle) This is still an open RFC. Consultation with the community will be part of this RFC, including asking for input and fee... [19:56:21] 10Traffic, 10Anti-Harassment, 10CheckUser, 10MediaWiki-User-management, 10Operations: Users editing from 127.0.0.1 - https://phabricator.wikimedia.org/T233657 (10Anomie) I'm going to poke this at #Traffic, since it seems unlikely that 127.0.0.1 is supposed to be showing up in XFF there. Is 10.128.0.127 s... [19:58:56] 10Traffic, 10Anti-Harassment, 10CheckUser, 10MediaWiki-User-management, 10Operations: Users editing from 127.0.0.1 - https://phabricator.wikimedia.org/T233657 (10CDanis) 10.128.0.127 is cp4027 which @Vgutierrez was using to experiment with ATS terminating TLS (see also T231627) I've depooled it for now,... [20:04:55] 10Traffic, 10Anti-Harassment, 10CheckUser, 10MediaWiki-User-management, 10Operations: Users editing from 127.0.0.1 - https://phabricator.wikimedia.org/T233657 (10Anomie) >>! In T233657#5517441, @CDanis wrote: > I've depooled it for now, which should stop this. I confirm that new entries with 127.0.0.1 h... [20:09:05] 10Traffic, 10Operations, 10Patch-For-Review: Move cache text cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231627 (10CDanis) I depooled cp4027 today when {T233657} surfaced. Gonna guess that the ATS TLS termination is missing a special client IP header that nginx knows to insert? [20:09:20] 10Traffic, 10Anti-Harassment, 10CheckUser, 10MediaWiki-User-management, 10Operations: Users editing from 127.0.0.1 - https://phabricator.wikimedia.org/T233657 (10CDanis) 05Open→03Resolved a:03CDanis [20:16:17] 10Traffic, 10Analytics, 10Operations: Publish tls related info to webrequest via varnish - https://phabricator.wikimedia.org/T233661 (10Nuria) [20:32:14] 10Traffic, 10Anti-Harassment, 10CheckUser, 10MediaWiki-User-management, 10Operations: Users editing from 127.0.0.1 (due to experimenting with ATS terminating TLS) - https://phabricator.wikimedia.org/T233657 (10Aklapper) [20:53:20] Hi, perf has a VCL change drafted to improve SVG compression for MW. – https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/537974/ [20:53:40] This is currently blocking a change that enables more use of SVGs (right now these requests are rare). [20:54:26] 10Traffic, 10Operations, 10Performance-Team, 10Patch-For-Review: Apache configuration: SVGs served by MediaWiki aren't gzipped - https://phabricator.wikimedia.org/T232615 (10Krinkle) [20:54:53] 10Traffic, 10Operations, 10Performance-Team, 10Patch-For-Review: Enable gzip compression for interface icon SVGs served by MediaWiki - https://phabricator.wikimedia.org/T232615 (10Krinkle) [21:41:45] 10Traffic, 10Operations, 10Patch-For-Review: Move cache text cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231627 (10Vgutierrez) Thanks for the depool @CDanis, so it looks like a combination of things: nginx/ATS set `X-Client-IP` and `X-Forwarded-For`. The behavior for `X-Client-IP` is t... [21:58:41] 10Traffic, 10Operations: varnish-fe is handling X-Forwarded-For differently when ats is in front of it - https://phabricator.wikimedia.org/T233667 (10Vgutierrez) [21:58:54] 10Traffic, 10Operations: varnish-fe is handling X-Forwarded-For differently when ats is in front of it - https://phabricator.wikimedia.org/T233667 (10Vgutierrez) p:05Triage→03High [21:59:32] 10Traffic, 10Operations: varnish-fe is handling X-Forwarded-For differently when ats is in front of it - https://phabricator.wikimedia.org/T233667 (10Vgutierrez) [21:59:34] 10Traffic, 10Anti-Harassment, 10CheckUser, 10MediaWiki-User-management, 10Operations: Users editing from 127.0.0.1 (due to experimenting with ATS terminating TLS) - https://phabricator.wikimedia.org/T233657 (10Vgutierrez) [22:52:00] 10Traffic, 10Core Platform Team, 10Operations, 10Performance-Team, and 6 others: Serve Main Page of WMF wikis from a consistent URL - https://phabricator.wikimedia.org/T120085 (10Izno) >>! In T120085#5517353, @Krinkle wrote: > This is still an open RFC. [snip] Totally missed this was in the RFCs bucket. (...