[00:52:40] <wikibugs>	 10DBA, 10MediaWiki-Parser, 10MediaWiki-Platform-Team, 10Patch-For-Review: WMF ParserCache disk space exhaustion - https://phabricator.wikimedia.org/T167784#3473378 (10tstarling) I added disk free space, and the derivative of disk free space, to the parser cache dashboard in Grafana: https://grafana.wikimed...
[02:05:57] <wikibugs>	 10DBA, 10MediaWiki-Parser, 10MediaWiki-Platform-Team, 10Patch-For-Review: WMF ParserCache disk space exhaustion - https://phabricator.wikimedia.org/T167784#3473448 (10tstarling) I had a look at cache-fragmenting parser options on pc1004 parsercache.pc001.   | Only canonical options | 40% | | Rows with wrap...
[05:14:42] <wikibugs>	 10DBA, 10Wikimedia-General-or-Unknown, 10Performance: Clean up skin properties - https://phabricator.wikimedia.org/T171643#3473683 (10demon) >>! In T171643#3472259, @matmarex wrote: >> The **only** valid values here are monobook, vector, cologneblue, modern and the various minervas (which should be merged, s...
[05:15:11] <wikibugs>	 10DBA, 10MediaWiki-Parser, 10MediaWiki-Platform-Team, 10Patch-For-Review: WMF ParserCache disk space exhaustion - https://phabricator.wikimedia.org/T167784#3473685 (10tstarling) The cache miss spike I mentioned earlier was apparently due to wikidata: 75% of the cache entries written with the relevant expir...
[06:20:45] <wikibugs>	 10DBA, 10MediaWiki-Parser, 10MediaWiki-Platform-Team, 10Patch-For-Review: WMF ParserCache disk space exhaustion - https://phabricator.wikimedia.org/T167784#3473725 (10tstarling) Note that the whole wikidata request rate spike only reduced disk space usage from 11% to 9% -- so by deleting the relevant rows,...
[07:42:16] <wikibugs>	 10DBA, 10MediaWiki-Parser, 10MediaWiki-Platform-Team, 10Patch-For-Review: WMF ParserCache disk space exhaustion - https://phabricator.wikimedia.org/T167784#3473813 (10jcrespo)
[07:49:01] <wikibugs>	 10DBA, 10MediaWiki-Parser, 10MediaWiki-Platform-Team, 10Patch-For-Review: WMF ParserCache disk space exhaustion - https://phabricator.wikimedia.org/T167784#3473815 (10jcrespo) @tstarling Do not worry about the current state- we are taking care of that with purging and defragmenting- my worry is for the lon...
[08:04:27] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1066 - https://phabricator.wikimedia.org/T169448#3473825 (10jcrespo) This is taking a long time to be rebuilt :-/ - It is still doing it.
[08:29:10] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Degraded RAID on db1066 - https://phabricator.wikimedia.org/T169448#3473901 (10jcrespo) I depool it and not it finishes :-(
[11:10:21] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1068 - https://phabricator.wikimedia.org/T171723#3474269 (10Volans) p:05Triage>03High This is s4 master.
[13:55:45] <wikibugs>	 10DBA, 10MediaWiki-Parser, 10MediaWiki-Platform-Team, 10Patch-For-Review: WMF ParserCache disk space exhaustion - https://phabricator.wikimedia.org/T167784#3344144 (10Anomie) >>! In T167784#3473448, @tstarling wrote: > Removing wrapclass from the cache key seems like low-hanging fruit.  Recall that it was...
[14:16:25] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Degraded RAID on db1066 - https://phabricator.wikimedia.org/T169448#3474822 (10jcrespo) 05Open>03Resolved a:03Cmjohnson
[17:31:36] <wikibugs>	 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: pdu phase inbalances: ps1-a3-codfw, ps1-c6-codfw, & ps1-d6-codfw - https://phabricator.wikimedia.org/T163339#3475474 (10Papaul) p:05Normal>03Low
[19:39:30] <wikibugs>	 10DBA, 10MediaWiki-Parser, 10MediaWiki-Platform-Team, 10Patch-For-Review: WMF ParserCache disk space exhaustion - https://phabricator.wikimedia.org/T167784#3344144 (10Krinkle) >>! In T167784#3474779, @Anomie wrote: >>>! In T167784#3473448, @tstarling wrote: >> Removing wrapclass from the cache key seems li...