[06:53:05] <wikibugs>	 10Traffic, 10Operations: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305 (10Marostegui) >>! In T238305#5774514, @wiki_willy wrote: > In going through all the affected systems in this task, I'd like to treat db2125 and backup2001 separately, since they seem like one-offs an...
[08:22:46] <wikibugs>	 10netops, 10Operations: Upgrade LibreNMS to 1.59 - https://phabricator.wikimedia.org/T241962 (10ayounsi) 05Open→03Resolved Went smoothly.
[10:11:50] <wikibugs>	 10Domains, 10Traffic, 10DNS, 10Operations: Donate wikiźródła.pl and wikisłownik.pl to the Foundation - https://phabricator.wikimedia.org/T240446 (10tomasz) p:05Normal→03Low
[10:13:53] <wikibugs>	 10Domains, 10Traffic, 10DNS, 10Operations: Donate wikiźródła.pl and wikisłownik.pl to the Foundation - https://phabricator.wikimedia.org/T240446 (10tomasz) I passed the AuthInfo codes to Doneva on Saturday 28 December 2019, however she wass out-of-office until 2 January 2020, so I'm currently still awaitin...
[10:34:03] <wikibugs>	 10Traffic, 10Analytics, 10Operations: Create replacement for Varnishkafka - https://phabricator.wikimedia.org/T237993 (10ema)
[11:18:03] <wikibugs>	 10Traffic, 10Operations: Upgrade cache cluster to debian buster - https://phabricator.wikimedia.org/T242093 (10Vgutierrez)
[11:22:58] <wikibugs>	 10netops, 10Operations: mr1-esams RMA (2020 edition) - https://phabricator.wikimedia.org/T242097 (10ayounsi) a:03ayounsi
[11:26:30] <wikibugs>	 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache cluster to debian buster - https://phabricator.wikimedia.org/T242093 (10Vgutierrez) p:05Triage→03Normal
[13:18:24] <wikibugs>	 10Traffic, 10Operations, 10Performance-Team (Radar): Some HTTP requests for MW failing due to "ERR_SPDY_PROTOCOL_ERROR 200" - https://phabricator.wikimedia.org/T220022 (10Krinkle) 05Open→03Declined Closing this together with several other TLS/HTTP2 related issues as we've switched from Nginx to ATS for t...
[13:40:44] <wikibugs>	 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10geraki) The issue exists also in elwiki: https://el.wikipedia.org/wiki/%CE%91%CE%B8%CF%8E%CE%B1_%CE%AE_%CE%AD%C...
[14:56:12] <wikibugs>	 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10geraki) I was wrong in the last comment. It happens when the semicolon is last character in a **short url**....
[16:06:19] <wikibugs>	 10Traffic, 10Elasticsearch, 10Operations, 10Discovery-Search (Current work), and 2 others: Sustained periods (2-4h) of bad latency on production-search eqiad - https://phabricator.wikimedia.org/T241421 (10Gehel) >>! In T241421#5773665, @ema wrote: > Clearly, however, this can easily turn in a cat and mouse...
[20:08:54] <wikibugs>	 10Traffic, 10Operations, 10Performance-Team (Radar): Add profiling for Varnish and VCL - https://phabricator.wikimedia.org/T175710 (10Krinkle)
[20:33:49] <wikibugs>	 10Domains, 10Traffic, 10Design-Research, 10Operations: Register a domain and redirect URL - https://phabricator.wikimedia.org/T241944 (10CRoslof) I've registered wikipersonas.org. If it turns out we aren't actually going to use it, though, or if we are going to use a different domain name instead, please l...
[20:51:06] <wikibugs>	 10Traffic, 10Operations, 10Performance-Team, 10observability: Ensure graphs used by Performance account for Varnish-to-ATS migration - https://phabricator.wikimedia.org/T233474 (10Krinkle)
[22:46:02] <wikibugs>	 10Traffic, 10Operations: cp3055 crashed - https://phabricator.wikimedia.org/T240425 (10Krenair) Down again, 2020-01-07, 22:44:19ish based on icinga IRC message
[23:09:35] <wikibugs>	 10Traffic, 10Operations: cp3055 crashed - https://phabricator.wikimedia.org/T240425 (10CDanis) Nothing in `racadm getsel` or `racadm lclog view` (latter just has me logging in over SSH).
[23:50:39] <wikibugs>	 10Traffic, 10Operations: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305 (10Papaul) backup2001 is at 1.3.7 for BIOS version and the last time we did only the IDRAC upgrade since sometimes when the IDRAC version is not up to date we might not see and log at system crash. so...