[16:37:03] 10Traffic, 10Operations: Text eqiad varnish 503 spikes - https://phabricator.wikimedia.org/T175803#3603561 (10Urbanecm) Dupe of T176047 ? [16:49:29] 10Traffic, 10Operations: Text eqiad varnish 503 spikes - https://phabricator.wikimedia.org/T175803#3612145 (10Samtar) [16:50:52] 10Traffic, 10Operations: Text eqiad varnish 503 spikes - https://phabricator.wikimedia.org/T175803#3612150 (10Urbanecm) p:05High>03Unbreak! Breaking a lot of things. [17:00:30] 10Traffic, 10Operations: Text eqiad varnish 503 spikes - https://phabricator.wikimedia.org/T175803#3612156 (10Paladox) p:05Unbreak!>03High Changing to high as things are stable now. But when things break again we can set it to unbreak now. [18:15:51] 10Traffic, 10Operations: Text eqiad varnish 503 spikes - https://phabricator.wikimedia.org/T175803#3603561 (10Yann) Request from 88.182.181.224 via cp1052 cp1052, Varnish XID 966459488 Error: 503, Backend fetch failed at Sat, 16 Sep 2017 18:15:18 GMT [18:30:43] 10Traffic, 10Operations: Text eqiad varnish 503 spikes - https://phabricator.wikimedia.org/T175803#3612223 (10Paladox) Hmm, not stable now. [19:18:20] <+icinga-wm> PROBLEM - Ulsfo HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [1000.0] [19:18:29] <+icing... [18:32:49] 10Traffic, 10Operations: Text eqiad varnish 503 spikes - https://phabricator.wikimedia.org/T175803#3612226 (10Samtar) It looks like cp1052 had a spike, but has since recovered {F9585689} `RECOVERY - Esams HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]` [19:47:19] 10Traffic, 10Operations: Text eqiad varnish 503 spikes - https://phabricator.wikimedia.org/T175803#3612304 (10Yann) Request from 88.182.181.224 via cp1052 cp1052, Varnish XID 34013240 Error: 503, Backend fetch failed at Sat, 16 Sep 2017 19:46:47 GMT