[08:29:56] <wikibugs>	 10Traffic, 06Operations, 10media-storage, 13Patch-For-Review, 15User-Urbanecm: Some PNG thumbnails and JPEG originals delivered as [text/html] content-type and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3165796 (10Urbanecm) https://upload.wikimedia.org/wikipedia/commons/thu...
[11:15:19] <wikibugs__>	 10netops, 06Operations, 10ops-esams: cr2-esams FPC 0 is dead - https://phabricator.wikimedia.org/T162239#3165924 (10elukey) We just saw a icinga recovery for cr2-esams, and now mtr from netmon1001 to cp3008 for example goes through cr2-eqiad -> cr2-esams.  Some 503s were registered while probably VRRP shifte...
[11:51:40] <wikibugs__>	 10netops, 06Operations, 10ops-esams: cr2-esams FPC 0 is dead - https://phabricator.wikimedia.org/T162239#3165961 (10elukey) from cr2-esams show log messages:  ``` Apr  8 10:44:00  re0.cr2-esams fpc0 CLKSYNC: Transitioned to centralized mode Apr  8 10:44:03  re0.cr2-esams fpc0 I2C Failed device: group 0x52 ad...
[11:55:37] <bblack>	 !log banning obj.http.Content-Type ~ text/html on cache_upload
[11:55:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:48:22] <wikibugs>	 10Traffic, 06Operations, 10media-storage, 13Patch-For-Review, 15User-Urbanecm: Some PNG thumbnails and JPEG originals delivered as [text/html] content-type and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3165972 (10BBlack)
[12:53:24] <wikibugs__>	 10Traffic, 06Operations, 10media-storage, 13Patch-For-Review, 15User-Urbanecm: Some PNG thumbnails and JPEG originals delivered as [text/html] content-type and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3150362 (10BBlack) The continued reports above were expected, as detail...
[13:09:59] <wikibugs>	 10netops, 06Operations, 10ops-esams: cr2-esams FPC 0 is dead - https://phabricator.wikimedia.org/T162239#3156449 (10BBlack) As best as I can tell from looking at a longer section of the cr2-esams logs, it really does look like esams remote hands already swapped in the replacement part and things came up norm...
[13:27:03] <bblack>	 !log banning obj.http.X-Orig-Content-Type !~ . on cache_upload in eqiad
[13:27:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:51:08] <bblack>	 ema: something to maybe poke at next week.  While monitoring for the T162035 issues, I've observed the behavior shown in (open) bug https://github.com/varnishcache/varnish-cache/issues/1799
[13:51:09] <stashbot>	 T162035: Some PNG thumbnails and JPEG originals delivered as [text/html] content-type and hence not rendered in browser - https://phabricator.wikimedia.org/T162035
[13:51:40] <bblack>	 ema: (which is that due to stupid varnish bugs, someones what should be a hit (or at worst a miss) converts into a pass.  There's some VCL magic that can be used to workaround)
[13:54:58] <bblack>	 !log banning obj.http.X-Orig-Content-Type !~ . on cache_upload in codfw
[13:55:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:56:23] <bblack>	 !log banning obj.http.X-Orig-Content-Type !~ . on cache_upload in esams
[14:56:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:46:01] <bblack>	 !log banning obj.http.X-Orig-Content-Type !~ . on cache_upload in ulsfo
[15:46:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:00:04] <bblack>	 !log banning obj.http.Content-Type ~ text/html on cache_upload
[16:00:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:03:37] <wikibugs>	 10Traffic, 06Operations, 10media-storage, 13Patch-For-Review, 15User-Urbanecm: Some PNG thumbnails and JPEG originals delivered as [text/html] content-type and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3166190 (10BBlack) The last round of bans mentioned above is complete n...
[18:35:41] <wikibugs__>	 10netops, 06Operations, 10ops-esams: cr2-esams FPC 0 is dead - https://phabricator.wikimedia.org/T162239#3166306 (10BBlack) Update: others noticed the serial number didn't change.  So, the new part is not yet installed, and we're not sure whether the old part recovered spontaneously, or due to some local act...