[07:29:53] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: eqiad row D switch upgrade - https://phabricator.wikimedia.org/T172459 (10jcrespo) [08:08:09] 10Traffic, 10Android-app-Bugs, 10Operations, 10Page Content Service, and 4 others: Incorrect language variant returned for PCS endpoints - https://phabricator.wikimedia.org/T249284 (10ema) >>! In T249284#6091252, @Pchelolo wrote: > Tagging #traffic - seems like Varnish/ATS layer no longer unsets 'cache-con... [08:23:36] 10Traffic, 10Core Platform Team, 10Operations: Move all purge traffic to kafka - https://phabricator.wikimedia.org/T250781 (10ema) >>! In T250781#6076720, @Pchelolo wrote: > AFAIK cdnPurgeJob is only involved if the delayed purge is required if reboundDelay option is set. For every rebound purge job there's... [08:35:52] 10Traffic, 10Operations: Puppet cleanup after purged transition - https://phabricator.wikimedia.org/T251374 (10ema) [08:36:01] 10Traffic, 10Operations: Puppet cleanup after purged transition - https://phabricator.wikimedia.org/T251374 (10ema) p:05Triage→03Medium [08:36:47] 10Traffic, 10Operations, 10Patch-For-Review: Create vhtcpd replacement - https://phabricator.wikimedia.org/T249583 (10ema) 05Open→03Resolved purged deployed fleet-wide. [10:29:21] 10Traffic, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar), 10Sustainability (MediaWiki-MultiDC): Make CDN purges reliable - https://phabricator.wikimedia.org/T133821 (10Joe) Since [[ https://github.com/wikimedia/operations-software-purged | purged ]] is now in production, and that we have s... [10:29:40] 10Traffic, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar), 10Sustainability (MediaWiki-MultiDC): Make CDN purges reliable - https://phabricator.wikimedia.org/T133821 (10Joe) At a later time, we could think of changing the logic, and make purges avoid race conditions, removing the need for t... [14:19:41] 10Traffic, 10InternetArchiveBot, 10Operations: Support TLSv1.3 - https://phabricator.wikimedia.org/T251414 (10Vgutierrez) [15:48:29] 10netops, 10Operations, 10Wikimedia-Incident: Investigate Juniper storm control - https://phabricator.wikimedia.org/T245192 (10Papaul) - Documentation in place - Add action-shutdown [16:11:38] 10Traffic, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar), 10Sustainability (MediaWiki-MultiDC): Make CDN purges reliable - https://phabricator.wikimedia.org/T133821 (10BBlack) >>! In T133821#6092865, @Joe wrote: > - Define a schema for a "url purge message". If I can throw in another $0.0... [16:12:28] 10Traffic, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar), 10Sustainability (MediaWiki-MultiDC): Make CDN purges reliable - https://phabricator.wikimedia.org/T133821 (10Pchelolo) I volunteer RESTBase stack to be a guinea pig for this. Currently we already are processing all the purges via k... [16:23:27] 10netops, 10Operations, 10Wikimedia-Incident: Investigate Juniper storm control - https://phabricator.wikimedia.org/T245192 (10Papaul) @ayounsi for the restore process when the interface is shutdown do you want for us to setup a recovery timeout or manually restore the interface? if we are going to use the... [17:18:06] 10Traffic, 10Android-app-Bugs, 10Operations, 10Page Content Service, and 4 others: Incorrect language variant returned for PCS endpoints - https://phabricator.wikimedia.org/T249284 (10Pchelolo) You've just challenged my core belief - I was 100% sure Varnish layer was not passing through the 'cache-control'... [17:19:08] 10Traffic, 10Operations, 10Parsoid, 10RESTBase, 10Core Platform Team Workboards (Clinic Duty Team): HTTP 400 Error when trying to save an edit on English Wikipedia: Error contacting the Parsoid/RESTBase server - https://phabricator.wikimedia.org/T250815 (10Pchelolo) #traffic This seems like a borderline... [17:29:09] 10Traffic, 10Operations, 10Parsoid, 10RESTBase, 10Core Platform Team Workboards (Clinic Duty Team): HTTP 400 Error when trying to save an edit on English Wikipedia: Error contacting the Parsoid/RESTBase server - https://phabricator.wikimedia.org/T250815 (10BBlack) Yeah, the linked Lua code is, I think, t... [18:55:34] 10Traffic, 10Operations: ats-tls ran out of FDs on cp1089 - https://phabricator.wikimedia.org/T248736 (10Ottomata) Is there a more permanent fix? Any idea why ATS was leaking the socket FDs? [19:47:37] 10Traffic, 10Operations, 10Parsoid, 10RESTBase, 10Core Platform Team Workboards (Clinic Duty Team): HTTP 400 Error when trying to save an edit on English Wikipedia: Error contacting the Parsoid/RESTBase server - https://phabricator.wikimedia.org/T250815 (10matmarex) >>! In T250815#6091098, @Pchelolo wrot... [19:54:21] 10Traffic, 10Operations, 10Parsoid, 10RESTBase, 10Core Platform Team Workboards (Clinic Duty Team): HTTP 400 Error when trying to save an edit on English Wikipedia: Error contacting the Parsoid/RESTBase server - https://phabricator.wikimedia.org/T250815 (10Pchelolo) [20:07:16] 10Traffic, 10Android-app-Bugs, 10Operations, 10Page Content Service, and 4 others: Incorrect language variant returned for PCS endpoints - https://phabricator.wikimedia.org/T249284 (10bearND) > I am inclined to say, that maybe unsetting it in RESTBase actually makes more sense - after all RESTBase depends... [23:56:59] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, 10Thumbor: Ghostscript outputs errors to stdout despite -q, preventing Thumbor from generating some thumbnails properly - https://phabricator.wikimedia.org/T236240 (10AntiCompositeNumber)