[00:05:19] 10Traffic, 10Operations, 10Performance-Team (Radar): Collect client network errors, deprecation, intervention and crash reports - https://phabricator.wikimedia.org/T207860 (10CDanis) 05Resolved→03Open a:05Gilles→03None SRE is interested in collecting at least network error reports. [00:17:32] 10Traffic, 10InternetArchiveBot, 10Operations: Support TLSv1.3 in IABot - https://phabricator.wikimedia.org/T251414 (10Krinkle) [00:43:57] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad, 10cloud-services-team (Hardware): (Need By: TBD) rack/setup/install WMCS 10G switches - https://phabricator.wikimedia.org/T251632 (10bd808) [00:47:54] 10Traffic, 10Operations, 10Goal, 10Performance-Team (Radar), 10Wikimedia-Incident: Support TLSv1.3 - https://phabricator.wikimedia.org/T170567 (10Krinkle) (See also [Navigation Timing metrics spec](https://www.w3.org/TR/navigation-timing-2/#processing-model).) | {icon arrow-circle-up color=red} request... [00:59:12] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad, 10cloud-services-team (Hardware): (Need By: TBD) rack/setup/install WMCS 10G switches - https://phabricator.wikimedia.org/T251632 (10bd808) [01:02:50] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad, 10cloud-services-team (Hardware): (Need By: TBD) rack/setup/install WMCS 10G switches - https://phabricator.wikimedia.org/T251632 (10bd808) [02:20:21] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, 10Thumbor: Thumbnail rendering of complex SVG file leads to Error 500 or Error 429 instead of Error 408 - https://phabricator.wikimedia.org/T226318 (10AntiCompositeNumber) As for the issue mentioned in the task title, I don't believe 408 wo... [08:17:25] 10Traffic, 10Operations, 10Privacy Engineering, 10Research, 10Privacy: wikiworkshop.org has Facebook button, external statcounter, https to http redirect - https://phabricator.wikimedia.org/T251732 (10Dzahn) Is there a specific thing we are waiting for? [08:36:20] 10Traffic, 10Operations: Stale nic firmware files on some hosts - https://phabricator.wikimedia.org/T253374 (10fgiunchedi) [09:50:02] 10Traffic, 10Operations, 10ops-eqsin: cp5012 memory errors - https://phabricator.wikimedia.org/T251219 (10Vgutierrez) 05Stalled→03Resolved cp5012 seems stable, I'll reopen this task if I see any sign of memory issues. Thanks @RobH [11:18:50] 10Traffic, 10Operations: varnish warnings: Invalid conf pair: lg_dirty_mult/lg_chunk - https://phabricator.wikimedia.org/T253379 (10ema) [11:18:58] 10Traffic, 10Operations: varnish warnings: Invalid conf pair: lg_dirty_mult/lg_chunk - https://phabricator.wikimedia.org/T253379 (10ema) p:05Triage→03Medium [11:20:36] 10Traffic, 10Operations, 10conftool, 10Patch-For-Review, and 2 others: Figure out a security model for etcd - https://phabricator.wikimedia.org/T97972 (10Volans) +1 as the above schema of auth reflects mostly my old comment/proposal. What's the deploy strategy? [12:07:54] 10Traffic, 10Operations: Stale nic firmware files on some hosts - https://phabricator.wikimedia.org/T253374 (10CDanis) Thanks for the report, I'll have to dig through Puppet logs I guess. Here's all eqiad hosts that have an mtime more than 10 minutes in the past: https://w.wiki/RYU In codfw it's just the tha... [12:51:38] 10Traffic, 10Operations: Stale nic firmware files on some hosts - https://phabricator.wikimedia.org/T253374 (10CDanis) P11283 contains syslog from one of the hosts where it didn't start properly. We see the service unit being installed, a systemctl daemon-reload, the timer unit being installed, then three sys... [13:48:02] 10Traffic, 10Beta-Cluster-Infrastructure, 10Operations, 10User-DannyS712: 502 error on beta commons - https://phabricator.wikimedia.org/T250103 (10Aklapper) [14:59:05] 10Traffic, 10Operations, 10Privacy Engineering, 10Research, 10Privacy: wikiworkshop.org has Facebook button, external statcounter, https to http redirect - https://phabricator.wikimedia.org/T251732 (10JFishback_WMF) The last things I see are: 1) Remove as much of the YUI and Bootstrap CSS and JS as poss... [15:22:51] 10Traffic, 10Operations, 10Privacy Engineering, 10Research, 10Privacy: wikiworkshop.org has Facebook button, external statcounter, https to http redirect - https://phabricator.wikimedia.org/T251732 (10leila) @bmansurov please check JFishback_WMF's comment above and make the changes requested. [15:36:43] 10Traffic, 10Operations, 10Patch-For-Review: Stale nic firmware files on some hosts - https://phabricator.wikimedia.org/T253374 (10CDanis) 05Open→03Resolved Fixed, and shouldn't happen again for this or other similar systemd::timer::jobs [15:36:45] 10Traffic, 10Operations: track NIC firmware version numbers across the fleet - https://phabricator.wikimedia.org/T236744 (10CDanis) [17:20:29] 10Traffic, 10Core Platform Team, 10Operations, 10serviceops, and 2 others: Reduce rate of purges emitted by MediaWiki - https://phabricator.wikimedia.org/T250205 (10Krinkle) >>! In T250205#6154883, @aaron wrote: > I'm not fond of the idea of not sending purges for indirect edits Agreed. The proposal to st... [18:01:33] 10Traffic, 10Operations, 10Patch-For-Review: check_http and SNI support - https://phabricator.wikimedia.org/T253292 (10RLazarus) p:05Triage→03Medium [18:23:13] 10Traffic, 10Core Platform Team, 10Operations: Whitelist x-wikimedia-debug header field (currently not allowed by Access-Control-Allow-Headers in preflight response) - https://phabricator.wikimedia.org/T252826 (10RLazarus) Hi CPT, adding you back -- is it possible this is due to this section of [[ https://ge... [19:03:09] 10Traffic, 10Core Platform Team, 10Operations, 10Developer Productivity: Whitelist x-wikimedia-debug header field (currently not allowed by Access-Control-Allow-Headers in preflight response) - https://phabricator.wikimedia.org/T252826 (10Krinkle)