[04:44:43] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 31.07 ms [04:48:56] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.13 ms [05:33:55] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [05:35:58] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.23 ms [05:50:39] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 31.15 ms [05:52:42] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.09 ms [08:02:46] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49847MiB (10% inode=99%); [08:55:44] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:55:49] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:55:51] PROBLEM - mw164 HTTPS on mw164 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [08:55:52] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:55:53] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [08:55:56] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:55:59] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [08:56:00] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [08:56:01] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:01] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [08:56:01] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 470.04, 165.29, 60.72 [08:56:04] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:14] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:15] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [08:56:24] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 mw184 mediawiki [08:56:25] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [08:56:26] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [08:56:30] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 6692 bytes in 2.081 second response time [08:56:32] PROBLEM - mw174 MediaWiki Rendering on mw174 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4661 bytes in 8.220 second response time [08:56:34] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:35] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4661 bytes in 9.230 second response time [08:56:36] PROBLEM - mw164 MediaWiki Rendering on mw164 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4661 bytes in 6.282 second response time [08:56:36] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4661 bytes in 1.367 second response time [08:56:37] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4661 bytes in 0.287 second response time [08:56:53] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4661 bytes in 0.096 second response time [08:56:57] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4661 bytes in 0.092 second response time [08:57:02] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4661 bytes in 0.090 second response time [08:57:05] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 6692 bytes in 0.117 second response time [08:57:12] PROBLEM - mw184 MediaWiki Rendering on mw184 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4661 bytes in 0.527 second response time [08:57:21] PROBLEM - mw154 MediaWiki Rendering on mw154 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4661 bytes in 0.143 second response time [08:57:23] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4661 bytes in 0.091 second response time [08:57:46] PROBLEM - db171 MariaDB Connections on db171 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Too many connections in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne [08:57:46] on line 66Fatal error: Uncaught mysqli_sql_exception: Too many connections in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [08:57:48] RECOVERY - mw164 HTTPS on mw164 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.053 second response time [08:57:49] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.058 second response time [08:57:49] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.058 second response time [08:57:54] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.054 second response time [08:57:55] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.052 second response time [08:57:59] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4108 bytes in 0.049 second response time [08:58:10] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.052 second response time [08:58:21] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.051 second response time [08:58:24] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [08:58:26] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4107 bytes in 0.050 second response time [08:59:01] PROBLEM - db171 APT on db171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:59:11] PROBLEM - db171 Puppet on db171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:59:54] PROBLEM - db171 MariaDB on db171 is CRITICAL: Received error packet before completion of TLS handshake. The authenticity of the following error cannot be verified: 1040 - Too many connections [09:02:24] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 11 backends are down. mw162 mw171 mw172 mw181 mw182 mw154 mw163 mw164 mw173 mw174 mw183 [09:04:24] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [09:05:54] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [09:10:05] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.15 ms [09:14:22] I get a message that my wiki is not availabe. Where can I find more info? Here is the url: https://r2.miraheze.org/wiki/Main_Page [09:15:23] there are server issues right now [09:20:59] PROBLEM - changeprop151 Current Load on changeprop151 is WARNING: LOAD WARNING - total load average: 10.90, 15.66, 8.30 [09:22:59] RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 1.63, 10.56, 7.33 [09:39:55] RECOVERY - db171 APT on db171 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [09:40:29] RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [09:40:33] RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.414 second response time [09:40:39] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.182 second response time [09:40:39] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.201 second response time [09:40:48] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.192 second response time [09:40:50] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.382 second response time [09:40:57] RECOVERY - mw184 MediaWiki Rendering on mw184 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.212 second response time [09:41:03] RECOVERY - mw154 MediaWiki Rendering on mw154 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.191 second response time [09:41:07] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.199 second response time [09:41:24] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.188 second response time [09:41:28] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.192 second response time [09:41:33] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.184 second response time [09:41:37] RECOVERY - db171 MariaDB Connections on db171 is OK: OK connection usage: 1.7%Current connections: 17 [09:41:37] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.324 second response time [09:41:38] RECOVERY - db171 MariaDB on db171 is OK: Uptime: 94 Threads: 11 Questions: 69712 Slow queries: 0 Opens: 2756 Open tables: 2747 Queries per second avg: 741.617 [09:41:49] !log [paladox@mwtask181] starting deploy of {'config': True, 'force': True} to all [09:41:49] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.198 second response time [09:41:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:42:09] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.329 second response time [09:42:10] !log [paladox@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 21s [09:43:45] !log [paladox@mwtask181] starting deploy of {'config': True, 'force': True} to all [09:44:02] !log [paladox@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 16s [09:44:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:44:10] RECOVERY - mw174 MediaWiki Rendering on mw174 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.211 second response time [09:44:12] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.221 second response time [09:44:16] RECOVERY - mw164 MediaWiki Rendering on mw164 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.203 second response time [09:44:19] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.162 second response time [09:44:22] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.204 second response time [09:44:22] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.200 second response time [10:37:58] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 31.10 ms [10:42:11] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.15 ms [10:44:19] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 1.24, 1.25, 11.63 [10:48:16] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.37, 1.29, 9.27 [11:29:05] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 80%, RTA = 31.14 ms [11:31:09] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.12 ms