[17:32:27] <tgr>	 anomie: around for the meeting?
[21:52:26] <Krinkle>	 ori: AaronSchulz: https://grafana.wikimedia.org/dashboard/file/varnish-http-errors.json?from=now-60d&to=now
[21:52:38] <Krinkle>	 Has someone looked into the peaked 500 errors?
[21:52:41] <Krinkle>	 It's consistently up it seems
[21:53:03] <Krinkle>	 From a steady ~100/min to a steady ~700/min
[21:53:13] <Krinkle>	 300-1200/min
[21:53:50] <Krinkle>	 https://grafana.wikimedia.org/dashboard/file/varnish-http-errors.json?from=1474494435487&to=1474842172912
[21:54:42] <Krinkle>	 Possible log entries correlating  on the 23rd:
[21:54:44] <Krinkle>	 - 10:07 moritzm: upgrading hhvm on codfw mediawiki servers
[21:54:58] <Krinkle>	 - 12:43 hashar: on tin dropping stall versions /srv/mediawiki-staging/php-1.28.0-wmf.{8,9,10} T141551
[21:54:58] <stashbot>	 T141551: MW-1.28.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T141551
[21:57:38] <AaronSchulz>	 i haven't
[22:02:44] <greg-g>	 Krinkle: I think that's 8/23, not 9/23?
[22:03:11] <greg-g>	 9/23 is a Friday
[22:03:20] <greg-g>	 was*, whatever
[22:03:34] <Krinkle>	 greg-g: the regression was 9/23
[22:03:39] <Krinkle>	 Ah yeah, I copied SAL from 8/23
[22:03:46] <greg-g>	 :)
[22:03:56] <Krinkle>	 https://wikitech.wikimedia.org/wiki/Server_Admin_Log#2016-09-23 04h to 17h roughly
[22:04:12] <Krinkle>	 04:03 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.20/includes/deferred: 5af1b93db1bb3d14844c55e4e3ed17fe963de551 (duration: 00m 48s)
[22:04:12] <Krinkle>	 04:02 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.20/includes/libs/rdbms: 5af1b93db1bb3d14844c55e4e3ed17fe963de551 (duration: 00m 51s)
[22:05:21] <greg-g>	 we've a HUGE increase in redis errors lately
[22:05:24] <greg-g>	 for a while
[22:05:44] <greg-g>	 https://phabricator.wikimedia.org/T125735
[22:06:01] <greg-g>	 probably unrelated, but it's the other thing that's been bugging me long term