[07:11:22] !log re-set SET GLOBAL max_connections=300 on bohrium's mysql (got lost after the restart) [07:11:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:15:17] some 500s were registered in misc, and the mysql max-conns weren't enough [07:15:20] :/ [11:39:15] ?quit [22:17:35] !log restarted webrequest-load Maps 12 Mar 2017 14:00:00 [22:17:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:21:56] I think that the issue is an1028, in which we have a broken disk and yarn is running.. (it tries to store /var/lib/hadoop/data/yarn/etc.. on the broken disk) [22:24:06] !log stopped yarn nodemanager on an1028 [22:24:07] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:24:56] !log restarted webrequest-load-text 12 Mar 2017 16:00:00 and 17:00:00 [22:24:57] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:26:11] ok so only the journalnode is working on an1028 (that should not access the /dev/sdaX disks used by the datanode/nodemanagers) [22:28:18] * elukey afk again :)