[02:56:30] Found https://blog.apnic.net/2017/05/23/anycast-latency-many-sites-enough/ while looking for something else [02:56:34] TL;DR; 12 sites are enough if in well chosen locations [09:54:24] 10Traffic, 10Operations, 10serviceops: Increased latency in appservers - 22 Nov 2019 - https://phabricator.wikimedia.org/T238939 (10fgiunchedi) Found this task only now, but see also {T238973} [12:03:38] 10Traffic, 10Operations, 10serviceops: Increased latency in appservers - 22 Nov 2019 - https://phabricator.wikimedia.org/T238939 (10Mathis_Benguigui) Hi, shouldn't this task be in //unbreak now!// priority? [12:05:38] 10Traffic, 10Operations, 10serviceops: Increased latency in appservers - 22 Nov 2019 - https://phabricator.wikimedia.org/T238939 (10Joe) >>! In T238939#5686415, @Mathis_Benguigui wrote: > Hi, shouldn't this task be in //Unbreak now!// priority? Probably, given I'm investigating on Saturday. But I think ther... [12:08:32] 10Traffic, 10Operations, 10serviceops: Increased latency in appservers - 22 Nov 2019 - https://phabricator.wikimedia.org/T238939 (10Mathis_Benguigui) p:05Triage→03Unbreak! [12:11:59] 10Traffic, 10Operations, 10serviceops: Increased latency in appservers - 22 Nov 2019 - https://phabricator.wikimedia.org/T238939 (10Joe) Just to clarify - the situation got worrisome only this morning, when latencies skyrocketed and the issue became user-visible. I'm not sure the two issue are the same, but... [12:24:28] 10Traffic, 10Operations, 10serviceops: Increased latency in appservers - 22 Nov 2019 - https://phabricator.wikimedia.org/T238939 (10Joe) 05Open→03Resolved a:03Joe Restarting php-fpm on the affected servers did solve the issue. I decided against doing deeper debugging before restarting the fleet because... [13:59:59] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10Joseagush) Well, now all templates (Mal) said "Galat script: no module." What just happen? Nevertheless, It wor... [20:57:10] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10Urbanecm) >>! In T238285#5686501, @Joseagush wrote: > Well, now all templates (Mal) said "Galat script: no modu...