[00:05:28] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 19204MiB (21% inode=98%); [00:18:30] PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to matomo151.wikitide.net port 443 after 0 ms: Couldn't connect to server [00:19:21] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 55% [00:25:21] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 2% [00:26:31] RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.337 second response time [00:30:49] [02puppet] 07The-Voidwalker created 03patch-nginxlogrotate (+1 new commit) 13https://github.com/miraheze/puppet/commit/11c7d30275c6 [00:30:49] 02puppet/03patch-nginxlogrotate 07The-Voidwalker 0311c7d30 add maxsize to nginx logrotate… [00:31:28] [02puppet] 07The-Voidwalker opened pull request #4191: add maxsize to nginx logrotate (03master...03patch-nginxlogrotate) 13https://github.com/miraheze/puppet/pull/4191 [00:31:36] [02puppet] 07coderabbitai[bot] commented on pull request #4191: --- […] 13https://github.com/miraheze/puppet/pull/4191#issuecomment-2676468938 [00:35:15] [02puppet] 07The-Voidwalker pushed 2 new commits to 03patch-nginxlogrotate 13https://github.com/miraheze/puppet/compare/26805c621e37...acdfd0413498 [00:35:15] 02puppet/03patch-nginxlogrotate 07The-Voidwalker 0370ec547 revert oops [00:35:15] 02puppet/03patch-nginxlogrotate 07The-Voidwalker 03acdfd04 Merge branch 'patch-nginxlogrotate' of https://github.com/miraheze/puppet into patch-nginxlogrotate [00:38:38] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.62, 7.73, 6.16 [00:39:05] [02puppet] 07The-Voidwalker 04force-pushed 03patch-nginxlogrotate from 03acdfd04 to 033e43b85 13https://github.com/miraheze/puppet/commits/patch-nginxlogrotate [00:39:06] 02puppet/03patch-nginxlogrotate 07The-Voidwalker 033e43b85 add max size to nginx logrotate config… [00:40:36] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.09, 7.30, 6.18 [00:41:09] PROBLEM - wiki.ooer.ooo - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [00:42:39] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.84, 15.83, 9.37 [00:44:32] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.37, 7.92, 6.68 [00:46:30] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.13, 7.75, 6.78 [00:50:39] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 8.78, 16.79, 13.24 [01:00:17] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.19, 7.60, 7.20 [01:04:13] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.20, 7.20, 7.15 [01:14:04] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.07, 7.40, 7.19 [01:16:04] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.24, 7.35, 7.20 [01:22:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:24:04] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.72, 7.61, 7.27 [01:26:04] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.62, 7.66, 7.34 [01:28:04] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.59, 7.75, 7.39 [01:30:04] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.59, 7.71, 7.42 [01:34:04] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.73, 7.78, 7.47 [01:36:04] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.89, 7.90, 7.55 [01:44:04] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.78, 7.36, 7.33 [01:46:04] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.64, 7.36, 7.33 [01:55:55] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49699MiB (10% inode=99%); [02:00:04] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.07, 7.13, 7.20 [02:02:04] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.56, 6.94, 7.13 [02:06:04] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.23, 7.47, 7.29 [02:12:04] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.52, 7.79, 7.54 [02:22:04] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.73, 8.04, 7.68 [02:24:04] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.81, 7.56, 7.55 [02:32:04] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.17, 5.97, 6.77 [04:09:29] [02puppet] 07The-Voidwalker merged pull request #4191: add maxsize to nginx logrotate (03master...03patch-nginxlogrotate) 13https://github.com/miraheze/puppet/pull/4191 [04:09:29] [02puppet] 07The-Voidwalker pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/e66494a0e0d8e198fd631e3e4c58f82bfd5df7e1 [04:09:29] 02puppet/03master 07The-Voidwalker 03e66494a add maxsize to nginx logrotate (#4191)… [04:09:30] [02puppet] 07The-Voidwalker 04deleted 03patch-nginxlogrotate at 033e43b85 13https://github.com/miraheze/puppet/commit/3e43b85 [04:19:59] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 22.72, 15.76, 8.78 [04:21:59] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 18.15, 16.14, 9.77 [04:28:23] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [04:29:16] PROBLEM - cp37 Current Load on cp37 is CRITICAL: LOAD CRITICAL - total load average: 20.04, 13.29, 9.29 [04:31:14] RECOVERY - cp37 Current Load on cp37 is OK: LOAD OK - total load average: 10.58, 12.16, 9.37 [04:31:48] PROBLEM - swiftproxy171 HTTPS on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [04:32:15] PROBLEM - swiftproxy171 Current Load on swiftproxy171 is CRITICAL: LOAD CRITICAL - total load average: 9.89, 9.15, 4.67 [04:32:20] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4136 bytes in 0.071 second response time [04:33:47] RECOVERY - swiftproxy171 HTTPS on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.031 second response time [04:36:15] RECOVERY - swiftproxy171 Current Load on swiftproxy171 is OK: LOAD OK - total load average: 3.19, 6.27, 4.57 [05:19:21] PROBLEM - wiki.beergeeks.co.il - Cloudflare on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [07:54:39] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.34, 19.36, 11.05 [07:56:39] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 11.03, 17.31, 11.40 [08:21:41] PROBLEM - hastursnotebook.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hastursnotebook.org' expires in 14 day(s) (Mon 10 Mar 2025 07:54:11 AM GMT +0000). [08:21:54] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/29ac9b3f3bb2dcc80f3c8659dfc1b9aba955d769 [08:21:54] 02ssl/03master 07WikiTideSSLBot 0329ac9b3 Bot: Update SSL cert for hastursnotebook.org [08:34:34] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.75, 7.22, 6.44 [08:42:26] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.81, 7.46, 6.78 [08:44:24] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.14, 7.25, 6.79 [08:46:22] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 9.03, 7.82, 7.05 [08:48:20] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.86, 7.73, 7.10 [08:50:22] RECOVERY - hastursnotebook.org - LetsEncrypt on sslhost is OK: OK - Certificate 'hastursnotebook.org' will expire on Sat 24 May 2025 07:23:18 AM GMT +0000. [09:10:04] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.20, 7.38, 7.17 [09:14:04] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 5.80, 7.03, 7.13 [09:20:04] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.09, 7.35, 7.19 [09:21:55] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27199MiB (5% inode=99%); [09:22:04] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.58, 7.03, 7.10 [09:24:04] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.30, 7.24, 7.14 [09:28:04] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.02, 7.71, 7.39 [09:40:04] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.47, 7.89, 7.55 [09:42:04] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.86, 7.43, 7.42 [10:06:04] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.21, 6.07, 6.63 [10:14:04] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.11, 6.56, 6.57 [10:16:04] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 6.31, 6.52, 6.56 [10:22:04] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.86, 7.28, 6.86 [10:30:04] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.81, 6.65, 6.76 [10:34:04] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.32, 7.34, 6.99 [10:36:04] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.10, 6.54, 6.75 [12:10:25] PROBLEM - mw184 Current Load on mw184 is WARNING: LOAD WARNING - total load average: 21.84, 15.81, 8.93 [12:10:43] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 11.30, 8.94, 4.95 [12:11:58] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.88, 17.34, 9.85 [12:12:43] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 12.78, 10.25, 5.91 [12:12:56] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.36, 17.20, 10.15 [12:13:56] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.35, 18.26, 10.94 [12:14:25] RECOVERY - mw184 Current Load on mw184 is OK: LOAD OK - total load average: 15.91, 17.07, 11.08 [12:15:27] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.28, 19.08, 11.93 [12:17:25] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 19.46, 18.83, 12.69 [12:17:45] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.63, 18.47, 12.70 [12:19:53] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.76, 20.90, 14.26 [12:20:21] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 21.61, 19.05, 13.10 [12:20:56] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 19.33, 19.50, 14.37 [12:21:34] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.15, 21.03, 15.02 [12:22:21] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 20.30, 19.42, 13.96 [12:22:22] PROBLEM - mw174 Current Load on mw174 is WARNING: LOAD WARNING - total load average: 23.37, 20.10, 14.68 [12:23:08] PROBLEM - mw184 Current Load on mw184 is WARNING: LOAD WARNING - total load average: 22.17, 20.86, 15.47 [12:24:54] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 20.92, 19.95, 14.70 [12:24:56] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.07, 21.30, 16.27 [12:25:03] RECOVERY - mw184 Current Load on mw184 is OK: LOAD OK - total load average: 14.79, 19.56, 15.68 [12:25:16] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.41, 20.91, 15.98 [12:25:23] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.76, 19.67, 15.91 [12:25:34] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.27, 19.99, 14.71 [12:25:53] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 14.37, 19.42, 15.94 [12:25:56] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw172 [12:25:58] PROBLEM - mw164 Current Load on mw164 is WARNING: LOAD WARNING - total load average: 22.72, 19.74, 14.45 [12:25:58] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.84, 20.32, 16.71 [12:26:22] RECOVERY - mw174 Current Load on mw174 is OK: LOAD OK - total load average: 14.88, 19.32, 15.78 [12:27:03] PROBLEM - mw154 Current Load on mw154 is WARNING: LOAD WARNING - total load average: 20.43, 18.46, 13.79 [12:27:34] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 18.04, 19.57, 15.23 [12:27:52] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [12:27:58] RECOVERY - mw164 Current Load on mw164 is OK: LOAD OK - total load average: 17.79, 19.34, 14.98 [12:28:43] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 7.17, 11.55, 10.80 [12:29:03] RECOVERY - mw154 Current Load on mw154 is OK: LOAD OK - total load average: 18.47, 18.17, 14.23 [12:29:58] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.41, 21.40, 18.02 [12:30:22] PROBLEM - mw174 Current Load on mw174 is WARNING: LOAD WARNING - total load average: 19.55, 20.41, 17.12 [12:30:32] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 22.09, 19.95, 16.25 [12:31:53] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.95, 20.25, 17.30 [12:31:58] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.21, 19.72, 17.81 [12:32:22] RECOVERY - mw174 Current Load on mw174 is OK: LOAD OK - total load average: 11.10, 17.17, 16.35 [12:32:28] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 18.55, 19.59, 16.59 [12:34:21] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.32, 21.02, 17.57 [12:34:43] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 15.59, 12.59, 11.33 [12:35:07] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.66, 20.25, 18.34 [12:35:53] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.04, 18.82, 17.47 [12:36:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:36:54] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 16.25, 19.10, 17.45 [12:38:21] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 19.60, 20.38, 18.11 [12:39:13] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.16, 20.03, 17.87 [12:41:13] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 20.13, 19.91, 18.09 [12:41:34] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.60, 20.55, 18.21 [12:42:58] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.53, 19.76, 18.47 [12:43:34] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 19.73, 20.26, 18.40 [12:44:54] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 22.31, 20.02, 18.30 [12:44:57] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.35, 18.20, 18.05 [12:46:54] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 18.20, 19.09, 18.17 [12:46:56] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.52, 19.69, 19.43 [12:48:43] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 3.94, 8.38, 11.00 [12:53:13] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.37, 20.17, 18.88 [12:56:43] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.28, 11.19, 11.00 [12:57:13] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.65, 19.17, 18.80 [12:58:43] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 4.26, 9.19, 10.36 [13:00:43] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.21, 6.44, 9.22 [13:01:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:13:55] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 55371MiB (12% inode=99%); [15:17:22] RECOVERY - wiki.beergeeks.co.il - Cloudflare on sslhost is OK: OK - Certificate 'wiki.beergeeks.co.il' will expire on Fri 28 Mar 2025 01:41:54 AM GMT +0000. [16:22:05] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.14, 6.75, 5.80 [16:22:51] RECOVERY - rarewarewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'rarewarewiki.com' will expire on Sat 24 May 2025 03:08:20 PM GMT +0000. [16:25:35] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 74510MiB (33% inode=99%); [16:26:32] RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 82659MiB (37% inode=99%); [16:29:56] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 6.12, 6.77, 6.28 [16:36:31] RECOVERY - crashspyrowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'crashspyrowiki.com' will expire on Sat 24 May 2025 03:08:41 PM GMT +0000. [16:41:41] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.97, 3.19, 2.00 [16:43:40] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.58, 3.79, 2.37 [16:45:00] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/a44a3700a6b97b5cc2b378582565a6bf86ae3bb8 [16:45:01] 02puppet/03master 07paladox 03a44a370 graylog: upgrade to 6.1.7 [16:47:38] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.54, 3.90, 2.77 [16:49:38] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 5.55, 4.44, 3.10 [16:55:35] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.25, 3.85, 3.30 [16:59:33] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 3.19, 3.38, 3.23 [17:03:30] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 5.12, 4.47, 3.70 [17:05:29] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.50, 4.00, 3.62 [17:11:26] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.16, 3.77, 3.62 [17:12:01] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49811MiB (10% inode=99%); [17:13:25] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.27, 3.54, 3.55 [17:19:01] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9709MiB (10% inode=98%); [17:19:06] [02puppet] 07Universal-Omega created 03cloud-ssh (+1 new commit) 13https://github.com/miraheze/puppet/commit/b6bc422b79e4 [17:19:06] 02puppet/03cloud-ssh 07CosmicAlpha 03b6bc422 Temp allow cloud ssh without bastion [17:19:29] [02puppet] 07Universal-Omega opened pull request #4192: Temp allow cloud ssh without bastion (03master...03cloud-ssh) 13https://github.com/miraheze/puppet/pull/4192 [17:19:40] [02puppet] 07coderabbitai[bot] commented on pull request #4192: --- […] 13https://github.com/miraheze/puppet/pull/4192#issuecomment-2676995969 [17:23:20] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 3.15, 3.24, 3.40 [17:29:17] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.21, 3.57, 3.50 [17:35:17] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.40, 3.69, 3.54 [17:37:17] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.41, 3.58, 3.52 [17:49:17] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 3.02, 3.22, 3.36 [18:03:50] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.96, 6.39, 6.17 [18:05:03] [02puppet] 07Universal-Omega pushed 1 new commit to 03upgrade-packages 13https://github.com/miraheze/puppet/commit/c14a7fc4ca253c81a1043ea18d9f5ec883e32561 [18:05:04] 02puppet/03upgrade-packages 07CosmicAlpha 03c14a7fc Add arguments [18:05:47] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 6.11, 6.41, 6.22 [18:17:33] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.03, 14.43, 6.93 [18:23:33] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.47, 20.76, 12.22 [18:25:33] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.73, 21.31, 13.46 [18:29:33] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.25, 22.69, 15.76 [18:42:29] PROBLEM - ssbuniverses.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ssbuniverses.com' expires in 14 day(s) (Mon 10 Mar 2025 06:31:05 PM GMT +0000). [18:42:45] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/6c375e9bbe45bcf9059a934a4ac370a599b436c5 [18:42:45] 02ssl/03master 07WikiTideSSLBot 036c375e9 Bot: Update SSL cert for ssbuniverses.com [18:49:33] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 8.18, 20.26, 21.39 [18:51:33] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 3.28, 14.27, 19.05 [19:11:13] RECOVERY - ssbuniverses.com - LetsEncrypt on sslhost is OK: OK - Certificate 'ssbuniverses.com' will expire on Sat 24 May 2025 05:44:10 PM GMT +0000. [21:03:00] [02puppet] 07Universal-Omega pushed 1 new commit to 03upgrade-packages 13https://github.com/miraheze/puppet/commit/2959a1d371819d532dd7fa6519567d2ffeed813e [21:03:00] 02puppet/03upgrade-packages 07CosmicAlpha 032959a1d Add include all and help options and improve validation [21:05:01] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 18308MiB (20% inode=98%); [21:11:24] [02puppet] 07Universal-Omega pushed 1 new commit to 03upgrade-packages 13https://github.com/miraheze/puppet/commit/1537f8ed8e9a38f1154b1f74bbf3d49f970a8fcb [21:11:24] 02puppet/03upgrade-packages 07CosmicAlpha 031537f8e Fix [21:12:56] [02puppet] 07Universal-Omega pushed 1 new commit to 03upgrade-packages 13https://github.com/miraheze/puppet/commit/6f125e16894116f3a4e3056dc2fa16a2c1568b51 [21:12:56] 02puppet/03upgrade-packages 07CosmicAlpha 036f125e1 Wording [21:18:34] [02puppet] 07Universal-Omega pushed 1 new commit to 03upgrade-packages 13https://github.com/miraheze/puppet/commit/752cacd0b969d795a33b3a4d7c37440fc7649413 [21:18:34] 02puppet/03upgrade-packages 07CosmicAlpha 03752cacd Consistent wording [22:55:46] PROBLEM - mwtask181 jobrunner-high.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10002 milliseconds [22:55:53] PROBLEM - mwtask181 SSH on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:08] PROBLEM - mwtask181 videoscaler.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10003 milliseconds [22:57:14] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [22:57:25] PROBLEM - mwtask181 jobrunner.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10004 milliseconds [22:57:27] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:45] PROBLEM - mwtask181 PowerDNS Recursor on mwtask181 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:58:56] PROBLEM - mwtask181 php-fpm on mwtask181 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [22:59:30] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:59:50] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:59:59] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 90.37, 71.89, 34.32 [23:00:32] PROBLEM - mwtask181 ferm_active on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:01:30] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 22.60, 12.95, 6.02 [23:02:00] RECOVERY - mwtask181 SSH on mwtask181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u5 (protocol 2.0) [23:02:00] RECOVERY - mwtask181 PowerDNS Recursor on mwtask181 is OK: DNS OK: 0.149 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205 [23:02:02] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:02:51] PROBLEM - mwtask181 mathoid on mwtask181 is CRITICAL: connect to address 10.0.18.106 and port 10044: Connection refused [23:03:30] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 27.35, 18.27, 8.83 [23:05:01] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 32 minutes ago with 0 failures [23:05:17] RECOVERY - mwtask181 php-fpm on mwtask181 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [23:05:20] RECOVERY - mwtask181 ferm_active on mwtask181 is OK: OK ferm input default policy is set [23:05:30] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 19.16, 19.68, 10.58 [23:05:33] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3955 bytes in 2.089 second response time [23:05:35] RECOVERY - mwtask181 videoscaler.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 1.917 second response time [23:05:49] RECOVERY - mwtask181 jobrunner.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 1.119 second response time [23:05:53] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.853 second response time [23:06:24] RECOVERY - mwtask181 jobrunner-high.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 1.500 second response time [23:06:51] RECOVERY - mwtask181 mathoid on mwtask181 is OK: TCP OK - 0.000 second response time on 10.0.18.106 port 10044