[01:39:46] effie: ^^ re cp1087 you manually powercycled the server, right? [02:17:13] 10Traffic, 10Operations: cp1087 reboot - https://phabricator.wikimedia.org/T239449 (10Vgutierrez) 05Open→03Resolved p:05Triage→03Normal a:03Vgutierrez Yep, this is most likely another occurrence of T238305 [02:17:29] 10Traffic, 10Operations: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305 (10Vgutierrez) [02:17:51] 10Traffic, 10Operations: cp1087 reboot - https://phabricator.wikimedia.org/T239449 (10Vgutierrez) [02:17:53] 10Traffic, 10Operations: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305 (10Vgutierrez) [06:59:35] vgutierrez: yes [06:59:45] yup, I've seen the SAL, thanks :) [06:59:51] :) [10:44:36] FYI there is a 503 report for a commons image in -tech [10:45:38] ema, vgutierrez ^^^ [11:19:31] they said to reply to en:User:Waddie96 [11:19:55] ema: my understanding is that they are passing through the only varnish backend [12:44:06] volans: ty [12:48:50] so the report is about https://commons.wikimedia.org/wiki/File:Terremoto_in_Albania_(49131201913).jpg returning 503 when fetched via cp3064 (which indeed is the only varnish-be in esams) [12:48:59] I see that the file is cached now though [12:49:53] and I get a 200 also forcing a pass, so it looks very much like a transient issue [14:23:51] 10Traffic, 10Operations, 10Performance-Team (Radar): ATS doesn't support X-Wikimedia-Debug - https://phabricator.wikimedia.org/T237687 (10ema) 05Open→03Resolved >>! In T237687#5679746, @Krinkle wrote: >> ` if ts.client_request.get_url_host() == 'appservers-rw.svc.wmnet' then` > > Looks like condition... [14:23:53] 10Traffic, 10Operations, 10Patch-For-Review: Replace Varnish backends with ATS on cache text nodes - https://phabricator.wikimedia.org/T227432 (10ema) [14:44:37] 10Traffic, 10Operations, 10Pybal, 10SRE-tools, 10serviceops: Applications and scripts need to be able to understand the pooled status of servers in our load balancers. - https://phabricator.wikimedia.org/T239392 (10akosiaris) `need to be able to understand the pooled status` I have to question this. Why... [15:01:44] 10Traffic, 10Operations, 10Performance-Team, 10Patch-For-Review: 200ms / 50% response start regression starting around 2019-11-11 - https://phabricator.wikimedia.org/T238494 (10ema) >>! In T238494#5698785, @Krinkle wrote: > Latency remains elevated. Do we have a status update or better idea about the root... [15:18:12] 10Traffic, 10Operations: 404 loading images from Virgin Media - https://phabricator.wikimedia.org/T161360 (10Aklapper) p:05High→03Triage a:05Timothycrice→03None @Timothy.davis18: Hi, is this still a problem now, two and a half years later? Or has this problem solved itself? Thanks! [15:45:31] 10Traffic, 10Operations, 10fixcopyright.wikimedia.org, 10Core Platform Team Workboards (Clinic Duty Team), and 3 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10ema) p:05Triage→03Normal [15:46:02] 10Traffic, 10Operations: 404 loading images from Virgin Media - https://phabricator.wikimedia.org/T161360 (10ema) p:05Triage→03Normal [16:29:31] 10Traffic, 10Operations: 404 loading images from Virgin Media - https://phabricator.wikimedia.org/T161360 (10Aklapper) @ema: I don't understand how a task about an issue which happened 30 months ago and we're unsure if there is still a problem can have a "Medium" priority... [21:31:28] 10Traffic, 10Operations, 10Performance-Team, 10Patch-For-Review: 200ms / 50% response start regression starting around 2019-11-11 - https://phabricator.wikimedia.org/T238494 (10Krinkle) If we expect to fix it reasonably soon I suppose it's not worth reverting over indeed. I do have a gut-feeling though tha... [22:12:15] 10Traffic, 10Operations, 10Patch-For-Review: Traffic Server packaging and initial puppetization - https://phabricator.wikimedia.org/T200178 (10hashar) >>! In T200178#4444797, @ema wrote: > CI tests [[https://integration.wikimedia.org/ci/job/debian-glue/1232/console | were failing ]] due to CI slaves being j...