[14:55:23] PROBLEM - Disk space on Hadoop worker on an-worker1094 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/c 15 GB (0% inode=99%): https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [15:46:13] PROBLEM - Disk space on Hadoop worker on analytics1077 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/e 16 GB (0% inode=99%): https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [16:32:41] PROBLEM - Disk space on Hadoop worker on an-worker1078 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/e 16 GB (0% inode=99%): https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [19:14:13] PROBLEM - Disk space on Hadoop worker on an-worker1095 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/f 16 GB (0% inode=99%): /var/lib/hadoop/data/l 28 GB (0% inode=99%): https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [19:25:48] !log deleted 1.2 TB from hdfs://analytics-hadoop/user/analytics/.Trash/200808000000 [19:25:50] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:26:08] !log deleted 300G from hdfs://analytics-hadoop/user/analytics/.Trash/200814000000 [19:26:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:28:03] PROBLEM - Disk space on Hadoop worker on an-worker1095 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/f 17 GB (0% inode=99%): /var/lib/hadoop/data/l 16 GB (0% inode=99%): https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [19:31:51] !log deleted 1.2 TB from hdfs://analytics-hadoop/user/nuria/.Trash/* [19:31:53] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:32:01] RECOVERY - Disk space on Hadoop worker on an-worker1095 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [19:34:15] !log deleted 1.2 TB from hdfs://analytics-hadoop/user/analytics/.Trash/200811000000 [19:34:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:41:21] PROBLEM - Disk space on Hadoop worker on an-worker1094 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/f 14 GB (0% inode=99%): /var/lib/hadoop/data/j 25 GB (0% inode=99%): /var/lib/hadoop/data/i 22 GB (0% inode=99%): https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [19:45:48] PROBLEM - Disk space on Hadoop worker on an-worker1095 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/f 31 GB (0% inode=99%): /var/lib/hadoop/data/l 15 GB (0% inode=99%): https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [20:00:33] PROBLEM - Disk space on Hadoop worker on analytics1077 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/l 16 GB (0% inode=99%): https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [20:17:51] PROBLEM - Disk space on Hadoop worker on analytics1070 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/j 15 GB (0% inode=99%): https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [20:21:21] PROBLEM - Disk space on Hadoop worker on an-worker1078 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/e 20 GB (0% inode=99%): /var/lib/hadoop/data/g 16 GB (0% inode=99%): https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [20:27:45] PROBLEM - Disk space on Hadoop worker on an-worker1083 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/k 15 GB (0% inode=99%): https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [20:40:59] PROBLEM - Disk space on Hadoop worker on analytics1070 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/j 15 GB (0% inode=99%): https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [21:01:21] PROBLEM - Disk space on Hadoop worker on an-worker1081 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/m 15 GB (0% inode=99%): https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [21:13:19] PROBLEM - Disk space on Hadoop worker on an-worker1084 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/d 12 GB (0% inode=99%): https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [22:30:07] RECOVERY - Disk space on Hadoop worker on analytics1077 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration