[00:00:05] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.90, 4.11, 3.47
[00:02:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.39, 22.59, 22.40
[00:02:04] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.22, 3.54, 3.33
[00:03:36] <botifico>	 [02python-functions] 07dependabot[bot] created branch 03dependabot/pip/dot-github/setuptools-73.0.1 - 13https://github.com/miraheze/python-functions
[00:03:38] <botifico>	 [02python-functions] 07dependabot[bot] pushed 031 commit to 03dependabot/pip/dot-github/setuptools-73.0.1 [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/d5c03cbc202b
[00:03:40] <botifico>	 [02python-functions] 07dependabot[bot] 03d5c03cb - Bump setuptools from 71.0.1 to 73.0.1 in /.github
[00:03:41] <botifico>	 [02python-functions] 07dependabot[bot] labeled pull request 03#56: Bump setuptools from 71.0.1 to 73.0.1 in /.github - 13https://github.com/miraheze/python-functions/pull/56
[00:03:42] <botifico>	 [02python-functions] 07dependabot[bot] opened pull request 03#56: Bump setuptools from 71.0.1 to 73.0.1 in /.github - 13https://github.com/miraheze/python-functions/pull/56
[00:03:45] <botifico>	 [02python-functions] 07dependabot[bot] labeled pull request 03#56: Bump setuptools from 71.0.1 to 73.0.1 in /.github - 13https://github.com/miraheze/python-functions/pull/56
[00:03:46] <botifico>	 [02python-functions] 07dependabot[bot] closed pull request 03#55: Bump setuptools from 71.0.1 to 73.0.0 in /.github - 13https://github.com/miraheze/python-functions/pull/55
[00:03:48] <botifico>	 [02python-functions] 07dependabot[bot] commented on pull request 03#55: Bump setuptools from 71.0.1 to 73.0.0 in /.github - 13https://github.com/miraheze/python-functions/pull/55#issuecomment-2299962552
[00:03:50] <botifico>	 [02python-functions] 07dependabot[bot] deleted branch 03dependabot/pip/dot-github/setuptools-73.0.0
[00:03:51] <botifico>	 [02python-functions] 07dependabot[bot] deleted branch 03dependabot/pip/dot-github/setuptools-73.0.0 - 13https://github.com/miraheze/python-functions
[00:03:53] <botifico>	 [02python-functions] 07coderabbitai[bot] commented on pull request 03#56: Bump setuptools from 71.0.1 to 73.0.1 in /.github - 13https://github.com/miraheze/python-functions/pull/56#issuecomment-2299962698
[00:04:02] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.80, 4.52, 3.73
[00:04:45] <botifico>	 [02python-functions] 07coderabbitai[bot] edited pull request 03#56: Bump setuptools from 71.0.1 to 73.0.1 in /.github - 13https://github.com/miraheze/python-functions/pull/56
[00:06:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.18, 23.50, 22.87
[00:06:21] <botifico>	 [02python-functions] 07coderabbitai[bot] edited a comment on pull request 03#56: Bump setuptools from 71.0.1 to 73.0.1 in /.github - 13https://github.com/miraheze/python-functions/pull/56#issuecomment-2299962698
[00:09:31] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[00:09:34] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:11:33] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 6.467 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[00:11:34] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[00:12:25] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:16:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:17:56] <miraheze-github>	 miraheze/python-functions - dependabot[bot] the build passed.
[00:18:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.70, 22.83, 22.33
[00:22:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.54, 23.80, 22.91
[00:23:52] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.29, 3.20, 3.99
[00:24:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.95, 24.50, 23.27
[00:25:52] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.11, 4.15, 4.22
[00:39:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.40, 3.29, 3.96
[00:41:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.65, 4.24, 4.23
[00:47:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.59, 3.12, 3.77
[00:49:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.04, 4.29, 4.12
[00:50:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.52, 20.78, 23.31
[00:50:11] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[00:51:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:51:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.44, 3.83, 4.00
[00:52:07] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.089 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[00:56:30] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[00:58:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.27, 22.79, 23.14
[00:59:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.42, 3.50, 3.66
[01:00:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.56, 20.61, 17.73
[01:00:31] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.078 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[01:02:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.90, 20.46, 18.04
[01:02:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.84, 20.51, 18.09
[01:04:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 13.96, 18.08, 17.47
[01:04:41] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.11, 19.56, 18.02
[01:05:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.60, 3.65, 3.76
[01:06:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.25, 23.57, 23.70
[01:06:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:07:46] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.28, 2.46, 3.31
[01:10:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.23, 24.69, 23.96
[01:12:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.03, 23.36, 23.56
[01:18:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.49, 24.23, 23.73
[01:20:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.61, 21.74, 22.89
[01:26:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.40, 22.34, 22.49
[01:32:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.51, 23.81, 23.36
[01:41:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:42:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.52, 23.82, 23.11
[01:43:14] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.15, 20.56, 17.53
[01:43:25] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.87, 21.65, 17.90
[01:43:31] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.22, 22.07, 18.63
[01:44:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.49, 23.74, 23.23
[01:45:13] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.25, 18.80, 17.24
[01:45:19] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.36, 19.02, 17.38
[01:45:26] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.99, 18.92, 17.90
[02:11:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:18:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.78, 19.09, 20.29
[02:28:53] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.06, 21.35, 20.62
[02:30:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.61, 22.35, 21.07
[02:32:47] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.20, 20.85, 20.70
[02:34:43] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.64, 19.56, 20.25
[02:46:18] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.72, 24.34, 21.83
[02:46:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.33, 20.82, 17.15
[02:47:26] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 3 backends are down. mw151 mw161 mw182
[02:48:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.89, 22.19, 18.01
[02:48:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.81, 20.75, 17.50
[02:48:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.49, 22.22, 18.15
[02:49:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.57, 21.55, 18.52
[02:49:25] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[02:50:11] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.34, 23.93, 22.41
[02:50:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 16.33, 20.63, 18.01
[02:50:41] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.59, 19.77, 17.56
[02:50:44] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.16, 19.93, 17.82
[02:52:08] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.35, 25.10, 22.99
[02:52:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.76, 22.31, 18.90
[02:53:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.79, 23.65, 19.85
[02:54:05] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.61, 23.46, 22.70
[02:54:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.81, 22.08, 19.29
[02:55:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.08, 22.23, 19.84
[02:56:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.84, 23.77, 22.87
[02:56:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 13.59, 19.57, 18.75
[02:57:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.29, 19.76, 19.22
[03:00:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.46, 23.01, 22.78
[03:01:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.11, 20.32, 19.54
[03:02:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.05, 25.42, 23.70
[03:03:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.44, 18.80, 19.05
[03:04:07] <icinga-miraheze>	 PROBLEM - mon181 Backups Grafana on mon181 is WARNING: FILE_AGE WARNING: /var/log/grafana-backup.log is 864225 seconds old and 92 bytes
[03:05:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.66, 3.48, 1.52
[03:06:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.30, 23.75, 23.40
[03:07:47] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.99, 2.93, 1.54
[03:16:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.03, 22.11, 22.45
[03:24:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.42, 20.66, 18.72
[03:28:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 18.02, 19.84, 18.88
[03:30:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.85, 22.79, 23.82
[03:36:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.79, 23.22, 23.57
[03:36:35] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.33, 4.76, 3.33
[03:38:34] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.67, 3.89, 3.19
[03:38:40] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.83, 21.20, 18.86
[03:40:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.71, 21.25, 19.30
[03:40:32] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.51, 3.33, 3.09
[03:41:28] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.01, 21.02, 18.93
[03:42:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.47, 23.61, 23.85
[03:42:28] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.70, 19.81, 18.96
[03:42:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.30, 19.74, 19.01
[03:43:24] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.00, 19.02, 18.45
[03:43:53] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[03:44:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.08, 23.97, 23.96
[03:44:35] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.99, 5.78, 4.10
[03:45:49] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.063 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[03:46:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.37, 23.26, 23.70
[03:48:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.69, 25.22, 24.35
[03:48:32] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.92, 3.93, 3.75
[03:49:13] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.87, 22.65, 19.95
[03:50:05] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.60, 21.08, 19.59
[03:51:08] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.30, 21.53, 19.92
[03:51:59] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 19.24, 20.12, 19.39
[03:54:27] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.09, 3.43, 3.56
[03:55:00] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 17.28, 19.52, 19.48
[03:56:26] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.91, 2.93, 3.35
[04:00:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.32, 21.70, 23.43
[04:03:20] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.34, 3.40, 3.38
[04:04:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.75, 23.78, 23.83
[04:05:53] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:06:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.29, 22.71, 23.46
[04:11:30] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:16:29] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[04:16:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:19:13] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.01, 3.63, 3.94
[04:21:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.69, 4.80, 4.34
[04:24:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.81, 17.99, 19.84
[04:29:07] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.37, 3.49, 3.99
[04:31:06] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.14, 4.18, 4.18
[04:36:39] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[04:38:10] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:38:34] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.235 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[04:42:15] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[04:43:34] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.51, 22.60, 20.57
[04:45:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.45, 20.58, 17.00
[04:45:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 28.82, 19.76, 15.34
[04:46:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:47:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.45, 20.85, 17.61
[04:47:27] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.23, 22.92, 21.22
[04:47:32] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 15.63, 17.91, 15.22
[04:48:57] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.66, 3.53, 3.95
[04:49:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 12.80, 17.66, 16.82
[04:50:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.63, 4.55, 4.29
[04:51:21] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.50, 23.54, 21.68
[05:01:05] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.29, 22.92, 22.79
[05:01:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:06:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:11:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:14:41] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.87, 3.16, 3.99
[05:16:39] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.77, 22.75, 21.94
[05:16:39] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 3.66, 3.71, 4.12
[05:18:36] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.01, 21.83, 21.70
[05:18:38] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.59, 3.53, 3.99
[05:20:37] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.95, 3.89, 4.05
[05:26:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.68, 3.52, 3.88
[05:28:19] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.49, 18.62, 20.05
[05:34:31] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.95, 3.86, 3.73
[05:36:30] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.80, 3.27, 3.52
[05:38:29] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.35, 4.08, 3.79
[05:41:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:42:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.86, 22.98, 20.68
[05:42:26] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.37, 2.98, 3.38
[05:42:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.06, 18.97, 16.06
[05:44:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.69, 21.95, 20.60
[05:44:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.30, 17.78, 15.99
[05:46:22] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.90, 3.61, 3.60
[05:50:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.93, 23.54, 21.56
[05:50:20] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.25, 3.58, 3.57
[05:52:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.52, 21.31, 20.96
[05:53:37] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[05:54:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.72, 23.68, 21.88
[05:56:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.81, 20.78, 21.02
[05:56:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:57:38] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.068 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[05:58:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.78, 22.49, 21.60
[06:00:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.42, 21.38, 21.29
[06:06:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.06, 17.43, 19.69
[06:08:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.12, 3.53, 3.94
[06:12:09] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.03, 3.65, 3.88
[06:14:08] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.43, 3.26, 3.69
[06:18:07] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.09, 4.17, 3.93
[06:20:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.54, 22.64, 20.65
[06:22:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.16, 22.32, 20.76
[06:26:02] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.85, 3.83, 3.99
[06:26:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.06, 22.63, 21.21
[06:28:01] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.93, 4.22, 4.11
[06:30:00] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.29, 3.73, 3.94
[06:30:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.28, 23.92, 22.12
[06:33:57] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.03, 3.21, 3.62
[06:37:35] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[06:39:00] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[06:41:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:41:38] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 1.323 second response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[06:41:49] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates).
[06:44:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.38, 17.84, 19.94
[06:46:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:47:30] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:49:10] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[06:54:00] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:55:17] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.075 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[06:56:04] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[06:56:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.61, 20.44, 16.78
[06:56:47] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.87, 22.94, 20.69
[06:57:04] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.91, 19.92, 15.78
[06:57:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.91, 21.70, 17.47
[06:57:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 30.05, 21.45, 16.72
[06:58:14] <icinga-miraheze>	 PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 2 backends are down. mw171 mw181
[06:58:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.88, 22.87, 17.61
[06:58:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.30, 20.80, 17.40
[06:59:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.09, 22.50, 17.74
[07:00:14] <icinga-miraheze>	 RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy
[07:00:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.45, 22.14, 18.04
[07:00:44] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.76, 19.14, 17.19
[07:01:01] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.91, 20.26, 17.01
[07:01:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 14.90, 19.83, 17.87
[07:01:32] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.52, 20.17, 17.46
[07:02:26] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:02:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 14.10, 19.67, 17.67
[07:04:32] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001920163631 secs
[07:04:34] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.08, 23.82, 22.41
[07:13:56] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[07:14:18] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.54, 18.08, 20.09
[07:19:38] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[07:21:24] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[07:21:33] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[07:23:01] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:23:20] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates).
[07:23:34] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 7 minutes ago with 0 failures
[07:23:39] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.070 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[07:29:13] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[07:41:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:41:57] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[07:43:57] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001411587 secs
[07:47:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.83, 20.27, 16.37
[07:47:16] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.40, 20.32, 18.46
[07:47:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.86, 18.95, 15.32
[07:47:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.05, 2.84, 3.77
[07:49:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.74, 18.47, 16.19
[07:49:13] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.34, 20.90, 18.94
[07:49:32] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 14.67, 17.24, 15.15
[07:49:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.97, 4.00, 4.08
[07:50:18] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[07:50:25] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:52:29] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[07:52:38] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[07:53:06] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.85, 22.98, 20.07
[07:54:02] <icinga-miraheze>	 PROBLEM - wiki.triplescripts.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.triplescripts.org All nameservers failed to answer the query.
[07:55:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.09, 21.65, 19.92
[07:57:31] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates).
[07:57:39] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures
[07:58:31] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.075 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[07:58:44] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[08:00:53] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.19, 18.97, 19.47
[08:03:32] <icinga-miraheze>	 PROBLEM - wiki.18t.rip - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.18t.rip All nameservers failed to answer the query.
[08:04:22] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[08:04:36] <icinga-miraheze>	 PROBLEM - zh.tardis.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - zh.tardis.wiki All nameservers failed to answer the query.
[08:04:38] <icinga-miraheze>	 PROBLEM - wiki.sadboyzpod.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.sadboyzpod.com All nameservers failed to answer the query.
[08:05:55] <icinga-miraheze>	 PROBLEM - wiki.chevrine.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.chevrine.com All nameservers failed to answer the query.
[08:06:23] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.000666230917 secs
[08:08:03] <icinga-miraheze>	 PROBLEM - www.samafia.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.samafia.org All nameservers failed to answer the query.
[08:09:44] <icinga-miraheze>	 PROBLEM - swests.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - swests.org All nameservers failed to answer the query.
[08:09:50] <icinga-miraheze>	 PROBLEM - wiki.wubbygame.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.wubbygame.com All nameservers failed to answer the query.
[08:10:39] <icinga-miraheze>	 PROBLEM - www.dovearchives.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.dovearchives.wiki All nameservers failed to answer the query.
[08:10:54] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[08:13:04] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:13:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.80, 2.81, 3.84
[08:14:22] <icinga-miraheze>	 PROBLEM - www.istpcomputing.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.istpcomputing.com All nameservers failed to answer the query.
[08:14:50] <icinga-miraheze>	 PROBLEM - wiki.gamergeeked.us - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gamergeeked.us All nameservers failed to answer the query.
[08:15:05] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.000479131937 secs
[08:15:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.92, 3.85, 4.06
[08:20:54] <icinga-miraheze>	 PROBLEM - wiki.scvo.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.scvo.org All nameservers failed to answer the query.
[08:26:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[08:31:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[08:31:37] <icinga-miraheze>	 PROBLEM - wiki.kirbygang.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.kirbygang.com All nameservers failed to answer the query.
[08:31:51] <icinga-miraheze>	 PROBLEM - correypedia.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - correypedia.org All nameservers failed to answer the query.
[08:35:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.76, 3.05, 3.86
[08:39:20] <icinga-miraheze>	 PROBLEM - wiki.ventistudio.fr - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.ventistudio.fr All nameservers failed to answer the query.
[08:39:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.09, 3.52, 3.83
[08:41:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.22, 3.72, 3.89
[08:43:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.42, 4.04, 3.96
[08:43:56] <icinga-miraheze>	 PROBLEM - wiki.sukotuze.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.sukotuze.com All nameservers failed to answer the query.
[08:46:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.87, 22.07, 19.58
[08:46:14] <icinga-miraheze>	 PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw181
[08:46:17] <icinga-miraheze>	 PROBLEM - wiki.veloren.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.veloren.net All nameservers failed to answer the query.
[08:46:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.87, 19.05, 15.75
[08:46:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.71, 19.54, 15.98
[08:47:59] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/sysctl.d]
[08:48:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.73, 21.28, 19.63
[08:48:14] <icinga-miraheze>	 RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy
[08:48:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.91, 18.64, 16.02
[08:48:38] <icinga-miraheze>	 PROBLEM - guia.esporo.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - guia.esporo.net All nameservers failed to answer the query.
[08:48:41] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.25, 18.28, 15.93
[08:49:05] <icinga-miraheze>	 PROBLEM - wiki.rosestulipsandliberty.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.rosestulipsandliberty.com All nameservers failed to answer the query.
[08:49:11] <icinga-miraheze>	 PROBLEM - dlfm-wiki.top - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dlfm-wiki.top All nameservers failed to answer the query.
[08:49:28] <icinga-miraheze>	 PROBLEM - cp41 APT on cp41 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (1 critical updates).
[08:49:37] <icinga-miraheze>	 PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (1 critical updates).
[08:49:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.74, 3.71, 3.88
[08:50:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.68, 19.66, 19.24
[08:50:13] <icinga-miraheze>	 PROBLEM - small.polandballwiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - small.polandballwiki.com All nameservers failed to answer the query.
[08:50:21] <icinga-miraheze>	 PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 65 packages available for upgrade (1 critical updates).
[08:50:24] <icinga-miraheze>	 PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 65 packages available for upgrade (1 critical updates).
[08:50:39] <icinga-miraheze>	 PROBLEM - cloud16 APT on cloud16 is CRITICAL: APT CRITICAL: 102 packages available for upgrade (1 critical updates).
[08:50:41] <icinga-miraheze>	 PROBLEM - os161 APT on os161 is CRITICAL: APT CRITICAL: 53 packages available for upgrade (1 critical updates).
[08:52:06] <icinga-miraheze>	 PROBLEM - cp36 APT on cp36 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (1 critical updates).
[08:52:21] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:52:27] <icinga-miraheze>	 PROBLEM - cp26 APT on cp26 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (1 critical updates).
[08:52:51] <icinga-miraheze>	 PROBLEM - cp27 APT on cp27 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (1 critical updates).
[08:52:53] <icinga-miraheze>	 PROBLEM - reports171 APT on reports171 is CRITICAL: APT CRITICAL: 64 packages available for upgrade (1 critical updates).
[08:53:13] <icinga-miraheze>	 PROBLEM - wiki.neonai.dev - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.neonai.dev All nameservers failed to answer the query.
[08:53:15] <icinga-miraheze>	 PROBLEM - rdb151 APT on rdb151 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (1 critical updates).
[08:53:45] <icinga-miraheze>	 PROBLEM - db181 APT on db181 is CRITICAL: APT CRITICAL: 60 packages available for upgrade (1 critical updates).
[08:53:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.37, 5.13, 4.39
[08:53:58] <icinga-miraheze>	 PROBLEM - cp51 APT on cp51 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (1 critical updates).
[08:54:07] <icinga-miraheze>	 PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 103 packages available for upgrade (1 critical updates).
[08:54:08] <icinga-miraheze>	 PROBLEM - eventgate181 APT on eventgate181 is CRITICAL: APT CRITICAL: 45 packages available for upgrade (1 critical updates).
[08:54:08] <icinga-miraheze>	 PROBLEM - cp37 APT on cp37 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (1 critical updates).
[08:54:31] <icinga-miraheze>	 PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 54 packages available for upgrade (1 critical updates).
[08:54:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.28, 19.91, 17.03
[08:54:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.87, 19.22, 16.68
[08:54:44] <icinga-miraheze>	 PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 59 packages available for upgrade (1 critical updates).
[08:54:53] <icinga-miraheze>	 PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 51 packages available for upgrade (1 critical updates).
[08:54:53] <icinga-miraheze>	 PROBLEM - swiftobject161 APT on swiftobject161 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (1 critical updates).
[08:54:53] <icinga-miraheze>	 PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 75 packages available for upgrade (1 critical updates).
[08:55:12] <icinga-miraheze>	 PROBLEM - phorge171 APT on phorge171 is CRITICAL: APT CRITICAL: 54 packages available for upgrade (1 critical updates).
[08:55:27] <icinga-miraheze>	 PROBLEM - db161 APT on db161 is CRITICAL: APT CRITICAL: 59 packages available for upgrade (1 critical updates).
[08:55:34] <icinga-miraheze>	 PROBLEM - os151 APT on os151 is CRITICAL: APT CRITICAL: 54 packages available for upgrade (1 critical updates).
[08:56:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.61, 20.79, 20.03
[08:56:18] <icinga-miraheze>	 PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: APT CRITICAL: 53 packages available for upgrade (1 critical updates).
[08:56:21] <icinga-miraheze>	 PROBLEM - cloud15 APT on cloud15 is CRITICAL: APT CRITICAL: 110 packages available for upgrade (1 critical updates).
[08:56:29] <icinga-miraheze>	 PROBLEM - bots171 APT on bots171 is CRITICAL: APT CRITICAL: 63 packages available for upgrade (1 critical updates).
[08:56:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[08:56:39] <icinga-miraheze>	 PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 58 packages available for upgrade (1 critical updates).
[08:56:49] <icinga-miraheze>	 PROBLEM - matomo151 APT on matomo151 is CRITICAL: APT CRITICAL: 63 packages available for upgrade (1 critical updates).
[08:56:59] <icinga-miraheze>	 PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 66 packages available for upgrade (1 critical updates).
[08:57:20] <icinga-miraheze>	 PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: APT CRITICAL: 41 packages available for upgrade (1 critical updates).
[08:58:18] <icinga-miraheze>	 PROBLEM - wiki.wildport.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.wildport.com All nameservers failed to answer the query.
[08:58:23] <icinga-miraheze>	 PROBLEM - graphite151 APT on graphite151 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (1 critical updates).
[08:58:28] <icinga-miraheze>	 PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 66 packages available for upgrade (1 critical updates).
[08:58:28] <icinga-miraheze>	 PROBLEM - wiki.minigamemadness.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.minigamemadness.net All nameservers failed to answer the query.
[08:58:33] <icinga-miraheze>	 PROBLEM - wiki.lefrenchmelee.fr - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.lefrenchmelee.fr All nameservers failed to answer the query.
[08:58:41] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.05, 18.19, 17.01
[08:58:44] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 12.83, 16.70, 16.30
[08:59:23] <icinga-miraheze>	 PROBLEM - db171 APT on db171 is CRITICAL: APT CRITICAL: 59 packages available for upgrade (1 critical updates).
[08:59:48] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[09:00:23] <icinga-miraheze>	 PROBLEM - kafka181 APT on kafka181 is CRITICAL: APT CRITICAL: 38 packages available for upgrade (1 critical updates).
[09:00:27] <icinga-miraheze>	 PROBLEM - db182 APT on db182 is CRITICAL: APT CRITICAL: 60 packages available for upgrade (1 critical updates).
[09:00:33] <icinga-miraheze>	 PROBLEM - mem161 APT on mem161 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (1 critical updates).
[09:00:42] <icinga-miraheze>	 PROBLEM - mem151 APT on mem151 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (1 critical updates).
[09:01:00] <icinga-miraheze>	 PROBLEM - db151 APT on db151 is CRITICAL: APT CRITICAL: 60 packages available for upgrade (1 critical updates).
[09:01:14] <icinga-miraheze>	 PROBLEM - ldap171 APT on ldap171 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (1 critical updates).
[09:01:47] <icinga-miraheze>	 PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: APT CRITICAL: 41 packages available for upgrade (1 critical updates).
[09:02:12] <icinga-miraheze>	 PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 56 packages available for upgrade (1 critical updates).
[09:02:20] <icinga-miraheze>	 PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 65 packages available for upgrade (1 critical updates).
[09:02:23] <icinga-miraheze>	 PROBLEM - jobchron171 APT on jobchron171 is CRITICAL: APT CRITICAL: 62 packages available for upgrade (1 critical updates).
[09:02:52] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[09:03:11] <icinga-miraheze>	 PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 63 packages available for upgrade (1 critical updates).
[09:03:15] <icinga-miraheze>	 PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 65 packages available for upgrade (1 critical updates).
[09:03:18] <icinga-miraheze>	 PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 65 packages available for upgrade (1 critical updates).
[09:03:26] <icinga-miraheze>	 PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 65 packages available for upgrade (1 critical updates).
[09:03:35] <icinga-miraheze>	 PROBLEM - swiftobject171 APT on swiftobject171 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (1 critical updates).
[09:03:48] <icinga-miraheze>	 PROBLEM - cloud17 APT on cloud17 is CRITICAL: APT CRITICAL: 102 packages available for upgrade (1 critical updates).
[09:03:55] <icinga-miraheze>	 PROBLEM - swiftproxy171 APT on swiftproxy171 is CRITICAL: APT CRITICAL: 55 packages available for upgrade (1 critical updates).
[09:04:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.12, 19.63, 20.08
[09:04:15] <icinga-miraheze>	 PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 63 packages available for upgrade (1 critical updates).
[09:05:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.75, 3.58, 3.97
[09:07:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 3.33, 3.89, 4.06
[09:10:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.35, 20.86, 20.33
[09:14:25] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[09:18:34] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[09:20:30] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.060 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[09:22:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.62, 23.31, 21.52
[09:23:27] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query.
[09:26:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.57, 22.70, 21.84
[09:29:51] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[09:31:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.37, 3.15, 3.89
[09:32:02] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:36:12] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001258313656 secs
[09:37:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.70, 3.22, 3.61
[09:39:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.48, 3.06, 3.52
[09:41:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.32, 3.94, 3.79
[09:47:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.93, 3.59, 3.78
[09:48:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.73, 23.22, 21.96
[09:52:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.02, 21.67, 21.64
[09:53:26] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found
[09:53:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.92, 3.81, 3.79
[09:54:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.30, 24.28, 22.60
[09:54:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.73, 20.61, 16.90
[09:55:08] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.63, 19.08, 14.23
[09:55:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 29.69, 22.25, 18.19
[09:55:19] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 29.88, 23.38, 18.70
[09:55:36] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.02, 21.90, 18.53
[09:56:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.70, 23.94, 22.74
[09:56:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[09:56:44] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.99, 18.46, 16.57
[09:57:08] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.24, 17.24, 14.15
[09:57:13] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.42, 22.15, 18.81
[09:57:35] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.11, 19.93, 18.21
[09:59:07] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.58, 19.71, 18.31
[09:59:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.26, 21.79, 19.06
[09:59:16] <botifico>	 [02mw-config] 07Reception123 closed pull request 03#5600: T12262: Add TimezoneConverter - 13https://github.com/miraheze/mw-config/pull/5600
[09:59:17] <botifico>	 [02mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9e04c07f3e99...c3020e57b877
[09:59:19] <botifico>	 [02mw-config] 07BlankEclair 03c3020e5 - T12262: Add TimezoneConverter (#5600)
[09:59:26] <MirahezeLSBot>	 !log [reception@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[09:59:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:59:43] <MirahezeLSBot>	 !log [reception@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 17s
[09:59:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:00:18] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[10:01:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.64, 19.87, 18.71
[10:01:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:06:57] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'config': True} to all
[10:07:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:07:10] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 12s
[10:07:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:09:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.36, 2.91, 3.81
[10:13:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.10, 4.22, 4.07
[10:16:14] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:16:39] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[10:17:55] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[10:18:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.53, 22.56, 21.94
[10:20:19] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[10:21:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:22:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.03, 23.71, 22.58
[10:22:51] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[10:23:44] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[10:23:45] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[10:23:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:24:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.73, 24.31, 22.89
[10:24:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:24:41] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:25:37] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.27, 20.85, 18.17
[10:26:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:27:32] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.58, 18.99, 17.80
[10:29:04] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 2.403 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[10:31:01] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[10:31:52] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[10:36:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.72, 23.37, 23.83
[10:36:20] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[10:36:30] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:37:30] <icinga-miraheze>	 RECOVERY - www.samafia.org - reverse DNS on sslhost is OK: SSL OK - www.samafia.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:37:39] <icinga-miraheze>	 RECOVERY - swests.org - reverse DNS on sslhost is OK: SSL OK - swests.org reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[10:37:39] <icinga-miraheze>	 RECOVERY - wiki.wubbygame.com - reverse DNS on sslhost is OK: SSL OK - wiki.wubbygame.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:38:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.24, 24.61, 24.19
[10:38:26] <icinga-miraheze>	 RECOVERY - wiki.ventistudio.fr - reverse DNS on sslhost is OK: SSL OK - wiki.ventistudio.fr reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:38:29] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[10:38:32] <icinga-miraheze>	 RECOVERY - www.dovearchives.wiki - reverse DNS on sslhost is OK: SSL OK - www.dovearchives.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:39:28] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:39:54] <icinga-miraheze>	 PROBLEM - wiki.minkyu.kim - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.minkyu.kim All nameservers failed to answer the query.
[10:40:41] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001109689474 secs
[10:41:26] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[10:41:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:42:10] <icinga-miraheze>	 RECOVERY - www.istpcomputing.com - reverse DNS on sslhost is OK: SSL OK - www.istpcomputing.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:42:24] <icinga-miraheze>	 PROBLEM - wiki.beergeeks.co.il - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.beergeeks.co.il All nameservers failed to answer the query.
[10:42:40] <icinga-miraheze>	 RECOVERY - wiki.gamergeeked.us - reverse DNS on sslhost is OK: SSL OK - wiki.gamergeeked.us reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:42:42] <icinga-miraheze>	 PROBLEM - wiki.lifesprogress.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.lifesprogress.com All nameservers failed to answer the query.
[10:42:58] <icinga-miraheze>	 PROBLEM - wiki.artmechanicum.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.artmechanicum.com All nameservers failed to answer the query.
[10:43:29] <icinga-miraheze>	 RECOVERY - wiki.sukotuze.com - reverse DNS on sslhost is OK: SSL OK - wiki.sukotuze.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:44:01] <icinga-miraheze>	 PROBLEM - wiki.case-clicker.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.case-clicker.com All nameservers failed to answer the query.
[10:44:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.64, 23.68, 23.96
[10:44:13] <icinga-miraheze>	 RECOVERY - wiki.veloren.net - reverse DNS on sslhost is OK: SSL OK - wiki.veloren.net reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:44:34] <icinga-miraheze>	 PROBLEM - ru-teirailway.f5.si - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - ru-teirailway.f5.si All nameservers failed to answer the query.
[10:44:54] <icinga-miraheze>	 PROBLEM - wiki.mostly.vet - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mostly.vet All nameservers failed to answer the query.
[10:46:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.88, 25.71, 24.68
[10:46:10] <icinga-miraheze>	 PROBLEM - unhingedwiki.paralleldv.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - unhingedwiki.paralleldv.org All nameservers failed to answer the query.
[10:46:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:46:35] <icinga-miraheze>	 RECOVERY - guia.esporo.net - reverse DNS on sslhost is OK: SSL OK - guia.esporo.net reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:46:48] <icinga-miraheze>	 PROBLEM - ru.countryhumans.polandball.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - ru.countryhumans.polandball.wiki All nameservers failed to answer the query.
[10:46:57] <icinga-miraheze>	 PROBLEM - wiki.pixlies.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.pixlies.net All nameservers failed to answer the query.
[10:47:02] <icinga-miraheze>	 RECOVERY - wiki.rosestulipsandliberty.com - reverse DNS on sslhost is OK: SSL OK - wiki.rosestulipsandliberty.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:47:11] <icinga-miraheze>	 RECOVERY - dlfm-wiki.top - reverse DNS on sslhost is OK: SSL OK - dlfm-wiki.top reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:48:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.53, 22.73, 23.72
[10:48:12] <icinga-miraheze>	 RECOVERY - small.polandballwiki.com - reverse DNS on sslhost is OK: SSL OK - small.polandballwiki.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:48:21] <icinga-miraheze>	 PROBLEM - wiki.ivy.cm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.ivy.cm All nameservers failed to answer the query.
[10:48:30] <icinga-miraheze>	 PROBLEM - wiki.cubestudios.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.cubestudios.xyz All nameservers failed to answer the query.
[10:48:40] <icinga-miraheze>	 PROBLEM - wiki.thunis.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.thunis.eu All nameservers failed to answer the query.
[10:48:45] <icinga-miraheze>	 RECOVERY - wiki.scvo.org - reverse DNS on sslhost is OK: SSL OK - wiki.scvo.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:48:50] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.070 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[10:49:42] <icinga-miraheze>	 PROBLEM - es.countryhumans.polandball.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - es.countryhumans.polandball.wiki All nameservers failed to answer the query.
[10:49:46] <icinga-miraheze>	 PROBLEM - wiki.gowashtenaw.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gowashtenaw.org All nameservers failed to answer the query.
[10:51:09] <icinga-miraheze>	 RECOVERY - wiki.neonai.dev - reverse DNS on sslhost is OK: SSL OK - wiki.neonai.dev reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:52:02] <icinga-miraheze>	 RECOVERY - wiki.triplescripts.org - reverse DNS on sslhost is OK: SSL OK - wiki.triplescripts.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:52:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.72, 18.92, 18.35
[10:52:37] <icinga-miraheze>	 PROBLEM - ao90.pinho.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - ao90.pinho.org All nameservers failed to answer the query.
[10:52:43] <icinga-miraheze>	 PROBLEM - wiki.stag.lol - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.stag.lol All nameservers failed to answer the query.
[10:54:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.28, 23.58, 23.38
[10:55:21] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.alathramc.com All nameservers failed to answer the query.
[10:56:00] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 18.17, 19.13, 18.68
[10:56:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.40, 23.08, 23.23
[10:56:17] <icinga-miraheze>	 RECOVERY - wiki.wildport.com - reverse DNS on sslhost is OK: SSL OK - wiki.wildport.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:56:23] <icinga-miraheze>	 RECOVERY - wiki.minigamemadness.net - reverse DNS on sslhost is OK: SSL OK - wiki.minigamemadness.net reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:56:32] <icinga-miraheze>	 RECOVERY - wiki.lefrenchmelee.fr - reverse DNS on sslhost is OK: SSL OK - wiki.lefrenchmelee.fr reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:57:21] <icinga-miraheze>	 PROBLEM - annapolishistorywiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'annapolishistorywiki.org' expires in 15 day(s) (Fri 06 Sep 2024 10:49:16 AM GMT +0000).
[10:57:31] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6d2ef0e2d07d...fd1e450284b0
[10:57:33] <botifico>	 [02ssl] 07WikiTideSSLBot 03fd1e450 - Bot: Update SSL cert for annapolishistorywiki.org
[10:58:10] <icinga-miraheze>	 PROBLEM - wiki.orivium.io - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.orivium.io All nameservers failed to answer the query.
[10:59:10] <icinga-miraheze>	 PROBLEM - www.winenjoy.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.winenjoy.net All nameservers failed to answer the query.
[10:59:41] <icinga-miraheze>	 PROBLEM - wiki.artmechanicum.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[10:59:53] <icinga-miraheze>	 RECOVERY - wiki.kirbygang.com - reverse DNS on sslhost is OK: SSL OK - wiki.kirbygang.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:59:53] <icinga-miraheze>	 PROBLEM - wiki.eastsidegamers.no - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.eastsidegamers.no All nameservers failed to answer the query.
[11:00:20] <icinga-miraheze>	 PROBLEM - wiki.tulpa.info - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.tulpa.info All nameservers failed to answer the query.
[11:01:02] <icinga-miraheze>	 PROBLEM - wiki.denby.tech - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.denby.tech All nameservers failed to answer the query.
[11:01:13] <icinga-miraheze>	 RECOVERY - correypedia.org - reverse DNS on sslhost is OK: SSL OK - correypedia.org reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[11:01:23] <icinga-miraheze>	 PROBLEM - wiki.macc.nyc - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.macc.nyc All nameservers failed to answer the query.
[11:02:18] <icinga-miraheze>	 PROBLEM - legacygt.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - legacygt.wiki All nameservers failed to answer the query.
[11:02:25] <icinga-miraheze>	 RECOVERY - zh.tardis.wiki - reverse DNS on sslhost is OK: SSL OK - zh.tardis.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK
[11:02:48] <icinga-miraheze>	 RECOVERY - wiki.18t.rip - reverse DNS on sslhost is OK: SSL OK - wiki.18t.rip reverse DNS resolves to cp36.wikitide.net - CNAME OK
[11:03:40] <icinga-miraheze>	 PROBLEM - data.nonbinary.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - data.nonbinary.wiki All nameservers failed to answer the query.
[11:03:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.54, 3.33, 3.98
[11:03:48] <icinga-miraheze>	 RECOVERY - wiki.chevrine.com - reverse DNS on sslhost is OK: SSL OK - wiki.chevrine.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[11:04:01] <icinga-miraheze>	 PROBLEM - archive.a2b2.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - archive.a2b2.org All nameservers failed to answer the query.
[11:04:12] <icinga-miraheze>	 RECOVERY - wiki.sadboyzpod.com - reverse DNS on sslhost is OK: SSL OK - wiki.sadboyzpod.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[11:04:30] <icinga-miraheze>	 PROBLEM - archive.stellurgists.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - archive.stellurgists.wiki All nameservers failed to answer the query.
[11:06:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.56, 22.01, 22.29
[11:08:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.12, 21.74, 22.18
[11:13:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.75, 3.41, 3.58
[11:19:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.20, 3.97, 3.82
[11:20:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.66, 23.00, 22.37
[11:21:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:21:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.24, 4.90, 4.18
[11:22:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.79, 21.09, 21.79
[11:22:28] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:23:06] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[11:24:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.60, 22.95, 22.35
[11:24:26] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[11:25:19] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['damon.ns.cloudflare.com.', 'gigi.ns.cloudflare.com.'], 'CNAME': 'wiki.alathra.com.'}
[11:26:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.67, 23.04, 22.47
[11:26:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:27:07] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.075 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[11:27:11] <icinga-miraheze>	 RECOVERY - annapolishistorywiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'annapolishistorywiki.org' will expire on Tue 19 Nov 2024 09:58:55 AM GMT +0000.
[11:27:13] <icinga-miraheze>	 RECOVERY - wiki.orivium.io - reverse DNS on sslhost is OK: SSL OK - wiki.orivium.io reverse DNS resolves to cp36.wikitide.net - CNAME OK
[11:27:48] <icinga-miraheze>	 RECOVERY - www.winenjoy.net - reverse DNS on sslhost is OK: SSL OK - www.winenjoy.net reverse DNS resolves to cp36.wikitide.net - CNAME OK
[11:28:19] <icinga-miraheze>	 PROBLEM - wiki.wildport.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.wildport.com All nameservers failed to answer the query.
[11:28:28] <icinga-miraheze>	 PROBLEM - wiki.minigamemadness.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.minigamemadness.net All nameservers failed to answer the query.
[11:28:36] <icinga-miraheze>	 PROBLEM - wiki.lefrenchmelee.fr - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.lefrenchmelee.fr All nameservers failed to answer the query.
[11:29:16] <icinga-miraheze>	 RECOVERY - wiki.eastsidegamers.no - reverse DNS on sslhost is OK: SSL OK - wiki.eastsidegamers.no reverse DNS resolves to cp36.wikitide.net - CNAME OK
[11:29:53] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:30:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.72, 23.83, 22.79
[11:30:03] <icinga-miraheze>	 RECOVERY - wiki.denby.tech - reverse DNS on sslhost is OK: SSL OK - wiki.denby.tech reverse DNS resolves to cp36.wikitide.net - CNAME OK
[11:30:39] <icinga-miraheze>	 RECOVERY - wiki.macc.nyc - reverse DNS on sslhost is OK: SSL OK - wiki.macc.nyc reverse DNS resolves to cp36.wikitide.net - CNAME OK
[11:31:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:31:30] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[11:31:40] <icinga-miraheze>	 PROBLEM - wiki.kirbygang.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.kirbygang.com All nameservers failed to answer the query.
[11:31:53] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:31:58] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0001856684685 secs
[11:32:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.52, 23.16, 22.69
[11:32:56] <icinga-miraheze>	 RECOVERY - archive.a2b2.org - reverse DNS on sslhost is OK: SSL OK - archive.a2b2.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[11:33:09] <icinga-miraheze>	 PROBLEM - correypedia.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - correypedia.org All nameservers failed to answer the query.
[11:33:59] <icinga-miraheze>	 RECOVERY - archive.stellurgists.wiki - reverse DNS on sslhost is OK: SSL OK - archive.stellurgists.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK
[11:36:06] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[11:36:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:38:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.73, 23.48, 22.57
[11:39:43] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.075 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[11:41:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:42:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.02, 21.92, 22.31
[11:46:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.52, 24.66, 23.30
[11:48:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.48, 23.26, 22.98
[11:48:22] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[11:49:42] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:51:26] <icinga-miraheze>	 PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway
[11:51:32] <icinga-miraheze>	 PROBLEM - phorge171 php-fpm on phorge171 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2'
[11:52:14] <icinga-miraheze>	 PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time
[11:55:20] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.alathramc.com All nameservers failed to answer the query.
[11:57:36] <icinga-miraheze>	 RECOVERY - wiki.wildport.com - reverse DNS on sslhost is OK: SSL OK - wiki.wildport.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[11:57:51] <icinga-miraheze>	 RECOVERY - wiki.minigamemadness.net - reverse DNS on sslhost is OK: SSL OK - wiki.minigamemadness.net reverse DNS resolves to cp36.wikitide.net - CNAME OK
[11:58:02] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[11:58:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.12, 22.91, 22.46
[11:58:05] <icinga-miraheze>	 RECOVERY - wiki.lefrenchmelee.fr - reverse DNS on sslhost is OK: SSL OK - wiki.lefrenchmelee.fr reverse DNS resolves to cp36.wikitide.net - CNAME OK
[11:58:24] <icinga-miraheze>	 PROBLEM - wiki.orivium.io - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.orivium.io All nameservers failed to answer the query.
[12:00:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.67, 22.70, 22.48
[12:00:38] <icinga-miraheze>	 PROBLEM - wiki.eastsidegamers.no - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.eastsidegamers.no All nameservers failed to answer the query.
[12:01:18] <icinga-miraheze>	 RECOVERY - wiki.kirbygang.com - reverse DNS on sslhost is OK: SSL OK - wiki.kirbygang.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[12:01:58] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[12:02:24] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:03:01] <icinga-miraheze>	 RECOVERY - correypedia.org - reverse DNS on sslhost is OK: SSL OK - correypedia.org reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[12:04:22] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[12:05:30] <icinga-miraheze>	 PROBLEM - archive.stellurgists.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - archive.stellurgists.wiki All nameservers failed to answer the query.
[12:06:02] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 2.767 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[12:08:17] <icinga-miraheze>	 RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19644 bytes in 2.617 second response time
[12:09:26] <icinga-miraheze>	 RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17718 bytes in 0.047 second response time
[12:09:32] <icinga-miraheze>	 RECOVERY - phorge171 php-fpm on phorge171 is OK: PROCS OK: 9 processes with command name 'php-fpm8.2'
[12:13:06] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:16:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:17:17] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[12:20:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.43, 22.17, 21.45
[12:21:39] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:23:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.52, 3.15, 3.93
[12:25:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.33, 4.14, 4.20
[12:28:00] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[12:29:10] <icinga-miraheze>	 PROBLEM - www.winenjoy.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.winenjoy.net All nameservers failed to answer the query.
[12:29:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.77, 3.39, 3.88
[12:31:02] <icinga-miraheze>	 PROBLEM - wiki.denby.tech - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.denby.tech All nameservers failed to answer the query.
[12:31:21] <icinga-miraheze>	 PROBLEM - wiki.macc.nyc - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.macc.nyc All nameservers failed to answer the query.
[12:31:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.11, 4.31, 4.16
[12:32:58] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[12:34:01] <icinga-miraheze>	 PROBLEM - archive.a2b2.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - archive.a2b2.org All nameservers failed to answer the query.
[12:34:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.73, 22.95, 23.30
[12:34:59] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 5.484 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[12:36:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.98, 24.31, 23.72
[12:36:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:37:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.51, 3.80, 4.00
[12:39:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.24, 3.99, 4.04
[12:40:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.43, 22.11, 23.06
[12:42:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.34, 23.26, 23.35
[12:43:35] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.20, 21.47, 18.94
[12:45:33] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.13, 20.44, 18.88
[12:46:54] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:48:26] <icinga-miraheze>	 PROBLEM - es.countryhumans.polandball.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[12:48:52] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[12:49:30] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.13, 20.16, 19.21
[12:49:38] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[12:52:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.87, 23.28, 23.91
[12:55:48] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[12:56:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.99, 23.18, 23.67
[12:56:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:59:05] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[13:00:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.30, 22.05, 23.23
[13:03:10] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 3.477 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[13:04:37] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:08:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.16, 23.51, 23.27
[13:09:28] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[13:10:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.27, 22.16, 22.81
[13:10:58] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[13:13:39] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 9.197 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[13:14:35] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[13:15:58] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 23 minutes ago with 0 failures
[13:18:07] <icinga-miraheze>	 PROBLEM - es.countryhumans.polandball.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - es.countryhumans.polandball.wiki All nameservers failed to answer the query.
[13:20:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.90, 24.00, 23.09
[13:21:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:22:55] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[13:25:31] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[13:25:35] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:28:02] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures
[13:29:33] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.995 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[13:29:46] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[13:29:51] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.00, 20.76, 18.66
[13:31:45] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 18.04, 19.57, 18.46
[13:39:22] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.16, 20.85, 19.40
[13:41:16] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.53, 18.95, 18.87
[13:47:16] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[13:48:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.63, 23.36, 23.93
[13:51:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.66, 20.42, 19.20
[13:52:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.37, 24.83, 24.32
[13:53:04] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.56, 19.69, 18.05
[13:53:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 19.43, 19.72, 19.08
[13:54:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.26, 23.40, 23.91
[13:55:00] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.56, 18.31, 17.76
[13:56:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:58:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.02, 24.16, 24.01
[13:59:31] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:01:28] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[14:01:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:02:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 14.95, 21.33, 23.08
[14:05:34] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[14:12:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.58, 17.06, 20.18
[14:13:14] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:14:18] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:15:11] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[14:20:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.97, 20.16, 20.28
[14:20:04] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.106 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[14:22:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.70, 18.92, 19.80
[14:25:19] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['damon.ns.cloudflare.com.', 'gigi.ns.cloudflare.com.'], 'CNAME': 'wiki.alathra.com.'}
[14:26:04] <icinga-miraheze>	 RECOVERY - wiki.orivium.io - reverse DNS on sslhost is OK: SSL OK - wiki.orivium.io reverse DNS resolves to cp36.wikitide.net - CNAME OK
[14:26:21] <botifico>	 [02mw-config] 07hurohukidaikon opened pull request 03#5641: Add UploadWizard configurations for kagagawiki in LocalWiki.php - 13https://github.com/miraheze/mw-config/pull/5641
[14:27:08] <icinga-miraheze>	 RECOVERY - www.winenjoy.net - reverse DNS on sslhost is OK: SSL OK - www.winenjoy.net reverse DNS resolves to cp36.wikitide.net - CNAME OK
[14:27:32] <icinga-miraheze>	 RECOVERY - wiki.artmechanicum.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.artmechanicum.com' will expire on Fri 11 Oct 2024 04:38:59 PM GMT +0000.
[14:27:40] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[14:27:53] <icinga-miraheze>	 RECOVERY - wiki.eastsidegamers.no - reverse DNS on sslhost is OK: SSL OK - wiki.eastsidegamers.no reverse DNS resolves to cp36.wikitide.net - CNAME OK
[14:28:10] <icinga-miraheze>	 RECOVERY - wiki.tulpa.info - reverse DNS on sslhost is OK: SSL OK - wiki.tulpa.info reverse DNS resolves to cp36.wikitide.net - CNAME OK
[14:28:18] <icinga-miraheze>	 PROBLEM - wiki.wildport.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.wildport.com All nameservers failed to answer the query.
[14:28:26] <icinga-miraheze>	 PROBLEM - wiki.minigamemadness.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.minigamemadness.net All nameservers failed to answer the query.
[14:28:34] <icinga-miraheze>	 PROBLEM - wiki.lefrenchmelee.fr - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.lefrenchmelee.fr All nameservers failed to answer the query.
[14:29:01] <icinga-miraheze>	 RECOVERY - wiki.denby.tech - reverse DNS on sslhost is OK: SSL OK - wiki.denby.tech reverse DNS resolves to cp36.wikitide.net - CNAME OK
[14:29:16] <icinga-miraheze>	 RECOVERY - wiki.macc.nyc - reverse DNS on sslhost is OK: SSL OK - wiki.macc.nyc reverse DNS resolves to cp36.wikitide.net - CNAME OK
[14:29:35] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.063 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[14:30:11] <icinga-miraheze>	 RECOVERY - legacygt.wiki - reverse DNS on sslhost is OK: SSL OK - legacygt.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK
[14:31:32] <icinga-miraheze>	 RECOVERY - data.nonbinary.wiki - reverse DNS on sslhost is OK: SSL OK - data.nonbinary.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK
[14:31:40] <icinga-miraheze>	 PROBLEM - wiki.kirbygang.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.kirbygang.com All nameservers failed to answer the query.
[14:39:07] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[14:41:07] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001043409109 secs
[14:41:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:42:48] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:43:17] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[14:44:53] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.47, 20.84, 19.83
[14:46:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:46:50] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.84, 19.88, 19.56
[14:47:22] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 3.821 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[14:49:00] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[14:55:20] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.alathramc.com All nameservers failed to answer the query.
[14:57:34] <icinga-miraheze>	 RECOVERY - wiki.wildport.com - reverse DNS on sslhost is OK: SSL OK - wiki.wildport.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[14:57:49] <icinga-miraheze>	 RECOVERY - wiki.minigamemadness.net - reverse DNS on sslhost is OK: SSL OK - wiki.minigamemadness.net reverse DNS resolves to cp36.wikitide.net - CNAME OK
[14:58:02] <icinga-miraheze>	 RECOVERY - wiki.lefrenchmelee.fr - reverse DNS on sslhost is OK: SSL OK - wiki.lefrenchmelee.fr reverse DNS resolves to cp36.wikitide.net - CNAME OK
[14:58:09] <icinga-miraheze>	 PROBLEM - wiki.orivium.io - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.orivium.io All nameservers failed to answer the query.
[14:59:10] <icinga-miraheze>	 PROBLEM - www.winenjoy.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.winenjoy.net All nameservers failed to answer the query.
[14:59:41] <icinga-miraheze>	 PROBLEM - wiki.artmechanicum.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[14:59:53] <icinga-miraheze>	 PROBLEM - wiki.eastsidegamers.no - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.eastsidegamers.no All nameservers failed to answer the query.
[15:00:00] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[15:00:20] <icinga-miraheze>	 PROBLEM - wiki.tulpa.info - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.tulpa.info All nameservers failed to answer the query.
[15:01:02] <icinga-miraheze>	 PROBLEM - wiki.denby.tech - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.denby.tech All nameservers failed to answer the query.
[15:01:18] <icinga-miraheze>	 RECOVERY - wiki.kirbygang.com - reverse DNS on sslhost is OK: SSL OK - wiki.kirbygang.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[15:01:21] <icinga-miraheze>	 PROBLEM - wiki.macc.nyc - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.macc.nyc All nameservers failed to answer the query.
[15:01:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:02:02] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 6.726 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[15:02:18] <icinga-miraheze>	 PROBLEM - legacygt.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - legacygt.wiki All nameservers failed to answer the query.
[15:03:40] <icinga-miraheze>	 PROBLEM - data.nonbinary.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - data.nonbinary.wiki All nameservers failed to answer the query.
[15:06:46] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:08:43] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[15:11:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:18:52] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[15:19:15] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:20:50] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.068 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[15:21:13] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[15:25:19] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['damon.ns.cloudflare.com.', 'gigi.ns.cloudflare.com.'], 'CNAME': 'wiki.alathra.com.'}
[15:27:13] <icinga-miraheze>	 RECOVERY - wiki.orivium.io - reverse DNS on sslhost is OK: SSL OK - wiki.orivium.io reverse DNS resolves to cp36.wikitide.net - CNAME OK
[15:27:48] <icinga-miraheze>	 RECOVERY - www.winenjoy.net - reverse DNS on sslhost is OK: SSL OK - www.winenjoy.net reverse DNS resolves to cp36.wikitide.net - CNAME OK
[15:28:51] <icinga-miraheze>	 PROBLEM - wiki.wildport.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.wildport.com All nameservers failed to answer the query.
[15:29:16] <icinga-miraheze>	 RECOVERY - wiki.eastsidegamers.no - reverse DNS on sslhost is OK: SSL OK - wiki.eastsidegamers.no reverse DNS resolves to cp36.wikitide.net - CNAME OK
[15:29:17] <icinga-miraheze>	 PROBLEM - wiki.minigamemadness.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.minigamemadness.net All nameservers failed to answer the query.
[15:29:36] <icinga-miraheze>	 PROBLEM - wiki.lefrenchmelee.fr - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.lefrenchmelee.fr All nameservers failed to answer the query.
[15:30:03] <icinga-miraheze>	 RECOVERY - wiki.denby.tech - reverse DNS on sslhost is OK: SSL OK - wiki.denby.tech reverse DNS resolves to cp36.wikitide.net - CNAME OK
[15:30:37] <icinga-miraheze>	 RECOVERY - wiki.macc.nyc - reverse DNS on sslhost is OK: SSL OK - wiki.macc.nyc reverse DNS resolves to cp36.wikitide.net - CNAME OK
[15:31:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:31:55] <icinga-miraheze>	 RECOVERY - archive.a2b2.org - reverse DNS on sslhost is OK: SSL OK - archive.a2b2.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[15:32:29] <icinga-miraheze>	 RECOVERY - archive.stellurgists.wiki - reverse DNS on sslhost is OK: SSL OK - archive.stellurgists.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK
[15:32:57] <icinga-miraheze>	 PROBLEM - wiki.kirbygang.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.kirbygang.com All nameservers failed to answer the query.
[15:33:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.68, 2.83, 3.85
[15:34:05] <icinga-miraheze>	 PROBLEM - correypedia.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - correypedia.org All nameservers failed to answer the query.
[15:35:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.36, 3.26, 3.85
[15:41:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.46, 3.59, 3.88
[15:43:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.79, 4.50, 4.19
[15:53:00] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[15:53:17] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:55:01] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 5.038 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[15:55:18] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[15:55:19] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.alathramc.com All nameservers failed to answer the query.
[15:58:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.65, 22.22, 20.00
[15:58:08] <icinga-miraheze>	 RECOVERY - wiki.wildport.com - reverse DNS on sslhost is OK: SSL OK - wiki.wildport.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[15:58:21] <icinga-miraheze>	 PROBLEM - wiki.orivium.io - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.orivium.io All nameservers failed to answer the query.
[15:58:40] <icinga-miraheze>	 RECOVERY - wiki.minigamemadness.net - reverse DNS on sslhost is OK: SSL OK - wiki.minigamemadness.net reverse DNS resolves to cp36.wikitide.net - CNAME OK
[15:59:04] <icinga-miraheze>	 RECOVERY - wiki.lefrenchmelee.fr - reverse DNS on sslhost is OK: SSL OK - wiki.lefrenchmelee.fr reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:00:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.57, 20.55, 19.70
[16:00:39] <icinga-miraheze>	 PROBLEM - wiki.eastsidegamers.no - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.eastsidegamers.no All nameservers failed to answer the query.
[16:02:35] <icinga-miraheze>	 RECOVERY - wiki.kirbygang.com - reverse DNS on sslhost is OK: SSL OK - wiki.kirbygang.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:03:35] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:03:57] <icinga-miraheze>	 RECOVERY - correypedia.org - reverse DNS on sslhost is OK: SSL OK - correypedia.org reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[16:04:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.71, 23.02, 20.79
[16:04:30] <icinga-miraheze>	 PROBLEM - archive.stellurgists.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - archive.stellurgists.wiki All nameservers failed to answer the query.
[16:05:41] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[16:06:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.86, 23.16, 21.15
[16:06:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:07:53] <icinga-miraheze>	 RECOVERY - wiki.minkyu.kim - reverse DNS on sslhost is OK: SSL OK - wiki.minkyu.kim reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:10:44] <icinga-miraheze>	 RECOVERY - wiki.lifesprogress.com - reverse DNS on sslhost is OK: SSL OK - wiki.lifesprogress.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:10:48] <icinga-miraheze>	 RECOVERY - wiki.artmechanicum.com - reverse DNS on sslhost is OK: SSL OK - wiki.artmechanicum.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:12:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.52, 23.42, 21.75
[16:12:06] <icinga-miraheze>	 RECOVERY - wiki.beergeeks.co.il - reverse DNS on sslhost is OK: SSL OK - wiki.beergeeks.co.il reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:12:44] <icinga-miraheze>	 RECOVERY - wiki.case-clicker.com - reverse DNS on sslhost is OK: SSL OK - wiki.case-clicker.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:14:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.10, 23.01, 21.82
[16:15:02] <icinga-miraheze>	 RECOVERY - wiki.pixlies.net - reverse DNS on sslhost is OK: SSL OK - wiki.pixlies.net reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:15:18] <icinga-miraheze>	 RECOVERY - wiki.mostly.vet - reverse DNS on sslhost is OK: SSL OK - wiki.mostly.vet reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:15:54] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:16:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.32, 23.39, 22.10
[16:16:15] <icinga-miraheze>	 RECOVERY - wiki.ivy.cm - reverse DNS on sslhost is OK: SSL OK - wiki.ivy.cm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:17:44] <icinga-miraheze>	 RECOVERY - wiki.gowashtenaw.org - reverse DNS on sslhost is OK: SSL OK - wiki.gowashtenaw.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:22:37] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:23:04] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.02, 19.98, 18.00
[16:24:38] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 8.842349052e-05 secs
[16:26:55] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.13, 18.70, 18.02
[16:27:32] <icinga-miraheze>	 RECOVERY - wiki.artmechanicum.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.artmechanicum.com' will expire on Fri 11 Oct 2024 04:38:59 PM GMT +0000.
[16:29:22] <icinga-miraheze>	 RECOVERY - wiki.tulpa.info - reverse DNS on sslhost is OK: SSL OK - wiki.tulpa.info reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:30:03] <icinga-miraheze>	 RECOVERY - wiki.eastsidegamers.no - reverse DNS on sslhost is OK: SSL OK - wiki.eastsidegamers.no reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:30:11] <icinga-miraheze>	 RECOVERY - legacygt.wiki - reverse DNS on sslhost is OK: SSL OK - legacygt.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:31:32] <icinga-miraheze>	 RECOVERY - data.nonbinary.wiki - reverse DNS on sslhost is OK: SSL OK - data.nonbinary.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:36:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:38:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.83, 22.86, 23.79
[16:39:54] <icinga-miraheze>	 PROBLEM - wiki.minkyu.kim - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.minkyu.kim All nameservers failed to answer the query.
[16:42:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.89, 23.18, 23.62
[16:42:26] <icinga-miraheze>	 RECOVERY - ru-teirailway.f5.si - reverse DNS on sslhost is OK: SSL OK - ru-teirailway.f5.si reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:44:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.40, 22.44, 23.35
[16:44:10] <icinga-miraheze>	 RECOVERY - unhingedwiki.paralleldv.org - reverse DNS on sslhost is OK: SSL OK - unhingedwiki.paralleldv.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:44:12] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[16:44:40] <icinga-miraheze>	 RECOVERY - ru.countryhumans.polandball.wiki - reverse DNS on sslhost is OK: SSL OK - ru.countryhumans.polandball.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:46:31] <icinga-miraheze>	 RECOVERY - wiki.cubestudios.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.cubestudios.xyz reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:46:36] <icinga-miraheze>	 RECOVERY - wiki.thunis.eu - reverse DNS on sslhost is OK: SSL OK - wiki.thunis.eu reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:47:25] <icinga-miraheze>	 PROBLEM - wiki.mostly.vet - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mostly.vet All nameservers failed to answer the query.
[16:47:38] <icinga-miraheze>	 RECOVERY - es.countryhumans.polandball.wiki - reverse DNS on sslhost is OK: SSL OK - es.countryhumans.polandball.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:50:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.13, 22.77, 23.06
[16:50:30] <icinga-miraheze>	 RECOVERY - ao90.pinho.org - reverse DNS on sslhost is OK: SSL OK - ao90.pinho.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:50:33] <icinga-miraheze>	 RECOVERY - wiki.stag.lol - reverse DNS on sslhost is OK: SSL OK - wiki.stag.lol reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:52:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.91, 22.35, 22.87
[16:54:06] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[16:55:19] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['damon.ns.cloudflare.com.', 'gigi.ns.cloudflare.com.'], 'CNAME': 'wiki.alathra.com.'}
[16:56:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:58:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.81, 24.02, 23.11
[16:58:17] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -4.440546036e-05 secs
[16:58:44] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.20, 21.05, 18.27
[16:59:08] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.94, 18.13, 14.73
[16:59:19] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw181
[16:59:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.01, 23.47, 19.55
[16:59:27] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 29.79, 23.66, 19.88
[16:59:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 29.24, 21.94, 17.57
[16:59:50] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw181
[17:00:03] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.89, 23.07, 19.32
[17:00:38] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.84, 20.71, 18.53
[17:01:08] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.46, 19.05, 15.55
[17:01:15] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.84, 22.04, 19.51
[17:01:19] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[17:01:26] <icinga-miraheze>	 PROBLEM - wiki.eastsidegamers.no - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.eastsidegamers.no All nameservers failed to answer the query.
[17:01:26] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.10, 22.12, 19.80
[17:01:28] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[17:01:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:01:32] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 17.23, 19.38, 17.14
[17:01:49] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[17:01:57] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.39, 22.09, 19.42
[17:02:32] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.40, 19.28, 18.28
[17:03:11] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.02, 20.06, 19.11
[17:03:24] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 15.28, 19.50, 19.11
[17:03:51] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.95, 20.03, 18.97
[17:04:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.34, 22.67, 23.14
[17:04:06] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[17:04:49] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:06:10] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures
[17:06:49] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001428812742 secs
[17:07:42] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 6.064 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[17:08:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.00, 23.70, 23.29
[17:08:47] <icinga-miraheze>	 RECOVERY - wiki.minkyu.kim - reverse DNS on sslhost is OK: SSL OK - wiki.minkyu.kim reverse DNS resolves to cp36.wikitide.net - CNAME OK
[17:10:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.02, 23.44, 23.27
[17:10:46] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:12:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.65, 24.46, 23.64
[17:12:05] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[17:12:43] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[17:13:31] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[17:14:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.14, 23.51, 23.37
[17:14:10] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 9.321 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[17:15:31] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001637130976 secs
[17:17:25] <icinga-miraheze>	 RECOVERY - wiki.mostly.vet - reverse DNS on sslhost is OK: SSL OK - wiki.mostly.vet reverse DNS resolves to cp36.wikitide.net - CNAME OK
[17:25:51] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[17:28:04] <MirahezeRelay>	 <bluemoon0332> !log restarted php-fpm8.2 on reports171
[17:28:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:30:48] <icinga-miraheze>	 RECOVERY - wiki.eastsidegamers.no - reverse DNS on sslhost is OK: SSL OK - wiki.eastsidegamers.no reverse DNS resolves to cp36.wikitide.net - CNAME OK
[17:30:51] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:31:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:32:01] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 2.687 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[17:32:49] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[17:33:01] <icinga-miraheze>	 RECOVERY - archive.stellurgists.wiki - reverse DNS on sslhost is OK: SSL OK - archive.stellurgists.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK
[17:36:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:42:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.33, 18.69, 20.40
[17:48:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.06, 20.63, 20.69
[17:50:31] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 50.53, 24.93, 9.99
[17:54:31] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 1.19, 11.49, 7.87
[17:55:19] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.alathramc.com All nameservers failed to answer the query.
[17:56:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:56:31] <icinga-miraheze>	 RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.32, 7.77, 6.95
[17:57:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.85, 3.19, 3.94
[17:58:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.32, 23.29, 21.98
[17:59:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.18, 3.19, 3.82
[18:00:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.72, 21.16, 17.41
[18:00:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 29.79, 22.71, 18.54
[18:01:05] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 2 backends are down. mw152 mw181
[18:01:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 29.18, 22.98, 19.19
[18:01:30] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:01:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.57, 20.90, 17.36
[18:01:57] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[18:02:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.76, 19.83, 17.42
[18:03:05] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[18:03:32] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 18.22, 19.74, 17.37
[18:03:53] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.255 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[18:04:31] <icinga-miraheze>	 PROBLEM - archive.stellurgists.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - archive.stellurgists.wiki All nameservers failed to answer the query.
[18:04:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 17.20, 20.79, 18.85
[18:05:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 18.61, 22.37, 20.00
[18:06:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.35, 23.68, 23.12
[18:06:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:06:41] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.56, 19.69, 18.67
[18:07:47] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[18:08:16] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[18:09:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.61, 20.28, 19.74
[18:09:54] <icinga-miraheze>	 PROBLEM - wiki.minkyu.kim - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.minkyu.kim All nameservers failed to answer the query.
[18:10:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.94, 23.48, 23.04
[18:10:12] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.068 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[18:11:00] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:11:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:12:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.91, 21.69, 22.42
[18:13:00] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0003772377968 secs
[18:16:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.56, 23.83, 23.10
[18:16:11] <icinga-miraheze>	 PROBLEM - unhingedwiki.paralleldv.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - unhingedwiki.paralleldv.org All nameservers failed to answer the query.
[18:18:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.70, 22.24, 22.61
[18:18:35] <icinga-miraheze>	 PROBLEM - guia.esporo.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.esporo.net could not be found
[18:20:04] <icinga-miraheze>	 PROBLEM - wiki.gowashtenaw.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gowashtenaw.org All nameservers failed to answer the query.
[18:21:37] <icinga-miraheze>	 PROBLEM - wiki.mostly.vet - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mostly.vet All nameservers failed to answer the query.
[18:24:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.63, 23.81, 22.91
[18:26:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.63, 22.92, 22.66
[18:26:31] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:28:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.73, 23.81, 23.02
[18:28:28] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[18:30:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.02, 21.08, 22.11
[18:31:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:31:34] <icinga-miraheze>	 PROBLEM - wiki.eastsidegamers.no - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.eastsidegamers.no All nameservers failed to answer the query.
[18:32:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.88, 23.94, 23.04
[18:32:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.02, 21.50, 19.53
[18:34:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.49, 21.99, 22.43
[18:34:30] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[18:34:41] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 14.07, 18.94, 18.85
[18:36:28] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 1.787 second response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[18:36:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:41:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:41:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.93, 3.07, 3.91
[18:42:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.14, 23.68, 22.72
[18:44:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.43, 21.86, 22.14
[18:45:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.54, 3.88, 3.99
[18:50:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.10, 21.26, 21.70
[18:50:34] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.71, 19.43, 18.59
[18:51:30] <botifico>	 [02mw-config] 07redbluegreenhat created branch 03replacetext-warning - 13https://github.com/miraheze/mw-config
[18:51:32] <botifico>	 [02mw-config] 07redbluegreenhat pushed 031 commit to 03replacetext-warning [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/d6fcd212b798
[18:51:34] <botifico>	 [02mw-config] 07redbluegreenhat 03d6fcd21 - Add a warning for Wiki Mechanics and Stewards about ReplaceText
[18:51:42] <botifico>	 [02mw-config] 07redbluegreenhat opened pull request 03#5642: Add a warning for Wiki Mechanics and Stewards about ReplaceText - 13https://github.com/miraheze/mw-config/pull/5642
[18:52:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.79, 22.07, 21.99
[18:52:28] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 19.10, 19.41, 18.69
[18:52:36] <botifico>	 [02mw-config] 07redbluegreenhat closed pull request 03#5642: Add a warning for Wiki Mechanics and Stewards about ReplaceText - 13https://github.com/miraheze/mw-config/pull/5642
[18:52:39] <botifico>	 [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c3020e57b877...7d4818bc93c7
[18:52:41] <botifico>	 [02mw-config] 07redbluegreenhat 037d4818b - Add a warning for Wiki Mechanics and Stewards about ReplaceText (#5642)
[18:52:41] <botifico>	 [02mw-config] 07redbluegreenhat deleted branch 03replacetext-warning
[18:52:42] <miraheze-github>	 miraheze/mw-config - redbluegreenhat the build passed.
[18:52:44] <botifico>	 [02mw-config] 07redbluegreenhat deleted branch 03replacetext-warning - 13https://github.com/miraheze/mw-config
[18:53:32] <miraheze-github>	 miraheze/mw-config - redbluegreenhat the build passed.
[19:03:20] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[19:05:08] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[19:05:17] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.933 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[19:06:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[19:07:15] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 10 minutes ago with 0 failures
[19:07:53] <icinga-miraheze>	 RECOVERY - wiki.minkyu.kim - reverse DNS on sslhost is OK: SSL OK - wiki.minkyu.kim reverse DNS resolves to cp36.wikitide.net - CNAME OK
[19:08:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.81, 18.80, 20.00
[19:11:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[19:11:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 0.19, 2.27, 3.91
[19:14:28] <icinga-miraheze>	 RECOVERY - unhingedwiki.paralleldv.org - reverse DNS on sslhost is OK: SSL OK - unhingedwiki.paralleldv.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[19:15:46] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.22, 1.13, 3.06
[19:18:00] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[19:20:01] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001550346613 secs
[19:22:53] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.19, 21.59, 20.27
[19:25:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.13, 20.45, 18.17
[19:25:19] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['damon.ns.cloudflare.com.', 'gigi.ns.cloudflare.com.'], 'CNAME': 'wiki.alathra.com.'}
[19:26:04] <icinga-miraheze>	 RECOVERY - wiki.orivium.io - reverse DNS on sslhost is OK: SSL OK - wiki.orivium.io reverse DNS resolves to cp36.wikitide.net - CNAME OK
[19:26:47] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.84, 23.45, 21.44
[19:27:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.93, 21.93, 19.01
[19:28:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 23.84, 24.03, 21.93
[19:29:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.20, 19.91, 18.61
[19:30:20] <icinga-miraheze>	 RECOVERY - wiki.eastsidegamers.no - reverse DNS on sslhost is OK: SSL OK - wiki.eastsidegamers.no reverse DNS resolves to cp36.wikitide.net - CNAME OK
[19:32:59] <icinga-miraheze>	 RECOVERY - archive.stellurgists.wiki - reverse DNS on sslhost is OK: SSL OK - archive.stellurgists.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK
[19:38:27] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.22, 22.51, 22.61
[19:39:52] <icinga-miraheze>	 PROBLEM - wiki.minkyu.kim - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.minkyu.kim All nameservers failed to answer the query.
[19:42:21] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.53, 23.33, 22.78
[19:44:18] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.46, 22.89, 22.65
[19:46:05] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:46:10] <icinga-miraheze>	 PROBLEM - unhingedwiki.paralleldv.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - unhingedwiki.paralleldv.org All nameservers failed to answer the query.
[19:55:20] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.alathramc.com All nameservers failed to answer the query.
[19:58:10] <icinga-miraheze>	 PROBLEM - wiki.orivium.io - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.orivium.io All nameservers failed to answer the query.
[19:58:41] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[20:00:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.31, 18.03, 20.24
[20:00:41] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001527607441 secs
[20:01:42] <icinga-miraheze>	 PROBLEM - wiki.eastsidegamers.no - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.eastsidegamers.no All nameservers failed to answer the query.
[20:05:12] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:07:13] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0002351403236 secs
[20:08:45] <icinga-miraheze>	 RECOVERY - wiki.minkyu.kim - reverse DNS on sslhost is OK: SSL OK - wiki.minkyu.kim reverse DNS resolves to cp36.wikitide.net - CNAME OK
[20:12:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.58, 19.38, 19.55
[20:14:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.01, 19.10, 19.45
[20:18:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.38, 23.15, 21.03
[20:19:04] <icinga-miraheze>	 RECOVERY - wiki.gowashtenaw.org - reverse DNS on sslhost is OK: SSL OK - wiki.gowashtenaw.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[20:22:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.04, 22.88, 21.44
[20:27:14] <icinga-miraheze>	 RECOVERY - wiki.orivium.io - reverse DNS on sslhost is OK: SSL OK - wiki.orivium.io reverse DNS resolves to cp36.wikitide.net - CNAME OK
[20:30:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.91, 24.42, 22.47
[20:32:24] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[20:34:24] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001732110977 secs
[20:34:31] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 9.60, 10.23, 4.75
[20:36:01] <icinga-miraheze>	 PROBLEM - db181 Puppet on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[20:36:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.31, 23.73, 22.99
[20:36:34] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 40.51, 19.92, 8.84
[20:38:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.01, 24.20, 23.24
[20:40:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.06, 23.96, 23.29
[20:41:35] <icinga-miraheze>	 PROBLEM - db181 PowerDNS Recursor on db181 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[20:43:16] <icinga-miraheze>	 PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[20:43:33] <icinga-miraheze>	 RECOVERY - db181 PowerDNS Recursor on db181 is OK: DNS OK: 0.061 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[20:43:37] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures
[20:43:47] <icinga-miraheze>	 RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[20:44:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 11.71, 15.75, 20.06
[20:44:30] <icinga-miraheze>	 RECOVERY - unhingedwiki.paralleldv.org - reverse DNS on sslhost is OK: SSL OK - unhingedwiki.paralleldv.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[20:45:17] <icinga-miraheze>	 RECOVERY - cp26 HTTPS on cp26 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3842 bytes in 2.200 second response time
[20:51:49] <icinga-miraheze>	 RECOVERY - wiki.mostly.vet - reverse DNS on sslhost is OK: SSL OK - wiki.mostly.vet reverse DNS resolves to cp36.wikitide.net - CNAME OK
[20:56:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:00:31] <icinga-miraheze>	 RECOVERY - wiki.eastsidegamers.no - reverse DNS on sslhost is OK: SSL OK - wiki.eastsidegamers.no reverse DNS resolves to cp36.wikitide.net - CNAME OK
[21:04:28] <miraheze-github>	 miraheze/mw-config - hurohukidaikon the build passed.
[21:04:31] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 0.46, 1.60, 10.99
[21:05:01] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[21:06:31] <icinga-miraheze>	 RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.32, 1.18, 9.70
[21:06:53] <botifico>	 [02mw-config] 07Universal-Omega reviewed pull request 03#5641 commit - 13https://github.com/miraheze/mw-config/pull/5641#discussion_r1725755455
[21:07:02] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001555621624 secs
[21:07:30] <icinga-miraheze>	 PROBLEM - archive.stellurgists.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - archive.stellurgists.wiki All nameservers failed to answer the query.
[21:16:10] <icinga-miraheze>	 PROBLEM - unhingedwiki.paralleldv.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - unhingedwiki.paralleldv.org All nameservers failed to answer the query.
[21:18:39] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.44, 4.28, 2.29
[21:19:18] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[21:20:05] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.76, 21.30, 19.87
[21:21:04] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[21:21:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:21:46] <botifico>	 [02TSPortal] 07Universal-Omega created branch 03migrate-underage - 13https://github.com/miraheze/TSPortal
[21:21:49] <botifico>	 [02TSPortal] 07Universal-Omega pushed 031 commit to 03migrate-underage [+1/-0/±0] 13https://github.com/miraheze/TSPortal/commit/9a5711ebea88
[21:21:51] <botifico>	 [02TSPortal] 07Universal-Omega 039a5711e - Add migration to change dpas underage field to TEXT rather than VARCHAR
[21:21:52] <botifico>	 [02TSPortal] 07Universal-Omega opened pull request 03#20: Add migration to change dpas underage field to TEXT rather than VARCHAR - 13https://github.com/miraheze/TSPortal/pull/20
[21:21:57] <botifico>	 [02TSPortal] 07coderabbitai[bot] commented on pull request 03#20: Add migration to change dpas underage field to TEXT rather than VARCHAR - 13https://github.com/miraheze/TSPortal/pull/20#issuecomment-2303035101
[21:23:05] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures
[21:23:20] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.084 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[21:23:55] <icinga-miraheze>	 PROBLEM - wiki.mostly.vet - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mostly.vet All nameservers failed to answer the query.
[21:26:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:28:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.89, 23.02, 21.01
[21:28:32] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.78, 3.80, 3.37
[21:30:31] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.88, 3.34, 3.26
[21:31:53] <icinga-miraheze>	 PROBLEM - wiki.eastsidegamers.no - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.eastsidegamers.no All nameservers failed to answer the query.
[21:32:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.64, 22.34, 21.34
[21:34:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.00, 23.68, 21.90
[21:34:28] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.89, 5.00, 3.93
[21:34:43] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[21:36:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.88, 23.42, 22.03
[21:36:22] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[21:36:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:36:48] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 9.246 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[21:37:01] <icinga-miraheze>	 RECOVERY - archive.stellurgists.wiki - reverse DNS on sslhost is OK: SSL OK - archive.stellurgists.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK
[21:38:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.60, 25.39, 22.93
[21:40:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.86, 23.23, 22.47
[21:41:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:42:08] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 27 minutes ago with 0 failures
[21:44:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 22.99, 24.31, 23.15
[21:46:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.86, 23.29, 22.91
[21:46:22] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.11, 3.44, 3.83
[21:48:20] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.97, 4.45, 4.15
[21:49:17] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[21:50:18] <icinga-miraheze>	 PROBLEM - wiki.gowashtenaw.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gowashtenaw.org All nameservers failed to answer the query.
[21:53:26] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 6.983 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[21:54:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.61, 23.23, 22.63
[21:54:15] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.31, 3.30, 3.79
[21:56:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.89, 23.25, 22.73
[21:56:15] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.10, 3.53, 3.78
[21:58:10] <icinga-miraheze>	 PROBLEM - wiki.orivium.io - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.orivium.io All nameservers failed to answer the query.
[21:58:14] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.25, 3.07, 3.59
[22:02:13] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.15, 4.14, 3.86
[22:03:00] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[22:04:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.88, 3.90, 3.83
[22:08:09] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.74, 3.88, 3.77
[22:09:07] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.072 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[22:10:08] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.99, 3.26, 3.54
[22:14:06] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.43, 3.97, 3.74
[22:16:04] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.45, 3.60, 3.65
[22:16:30] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:18:04] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.81, 4.36, 3.91
[22:20:03] <icinga-miraheze>	 RECOVERY - wiki.gowashtenaw.org - reverse DNS on sslhost is OK: SSL OK - wiki.gowashtenaw.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[22:20:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.30, 21.84, 21.13
[22:20:03] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.08, 3.47, 3.65
[22:22:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.54, 22.20, 21.38
[22:23:58] <icinga-miraheze>	 RECOVERY - wiki.mostly.vet - reverse DNS on sslhost is OK: SSL OK - wiki.mostly.vet reverse DNS resolves to cp36.wikitide.net - CNAME OK
[22:24:00] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.66, 3.92, 3.75
[22:24:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.51, 23.64, 21.99
[22:25:19] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['damon.ns.cloudflare.com.', 'gigi.ns.cloudflare.com.'], 'CNAME': 'wiki.alathra.com.'}
[22:26:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.91, 22.71, 21.86
[22:27:57] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.10, 3.90, 3.83
[22:30:01] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.37, 4.22, 3.91
[22:30:40] <icinga-miraheze>	 RECOVERY - wiki.eastsidegamers.no - reverse DNS on sslhost is OK: SSL OK - wiki.eastsidegamers.no reverse DNS resolves to cp36.wikitide.net - CNAME OK
[22:32:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.92, 23.33, 22.31
[22:34:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.52, 23.01, 22.33
[22:36:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.57, 24.39, 22.89
[22:39:54] <icinga-miraheze>	 PROBLEM - wiki.minkyu.kim - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.minkyu.kim All nameservers failed to answer the query.
[22:43:56] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:48:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.87, 23.79, 23.69
[22:48:11] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[22:50:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.47, 23.80, 23.69
[22:52:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.74, 23.28, 23.53
[22:55:09] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[22:55:19] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.alathramc.com All nameservers failed to answer the query.
[22:56:04] <icinga-miraheze>	 PROBLEM - wiki.mostly.vet - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mostly.vet All nameservers failed to answer the query.
[22:58:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.21, 23.57, 23.49
[23:00:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.14, 22.33, 23.05
[23:01:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:02:02] <icinga-miraheze>	 PROBLEM - wiki.eastsidegamers.no - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.eastsidegamers.no All nameservers failed to answer the query.
[23:03:58] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:05:36] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 8.304 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[23:05:55] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[23:06:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:14:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.43, 23.55, 23.04
[23:14:10] <icinga-miraheze>	 RECOVERY - unhingedwiki.paralleldv.org - reverse DNS on sslhost is OK: SSL OK - unhingedwiki.paralleldv.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[23:16:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:19:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.53, 3.09, 3.91
[23:23:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.30, 3.90, 4.01
[23:25:18] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['damon.ns.cloudflare.com.', 'gigi.ns.cloudflare.com.'], 'CNAME': 'wiki.alathra.com.'}
[23:25:21] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[23:26:04] <icinga-miraheze>	 RECOVERY - wiki.orivium.io - reverse DNS on sslhost is OK: SSL OK - wiki.orivium.io reverse DNS resolves to cp36.wikitide.net - CNAME OK
[23:26:05] <icinga-miraheze>	 RECOVERY - wiki.mostly.vet - reverse DNS on sslhost is OK: SSL OK - wiki.mostly.vet reverse DNS resolves to cp36.wikitide.net - CNAME OK
[23:27:17] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.074 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[23:31:25] <icinga-miraheze>	 RECOVERY - wiki.eastsidegamers.no - reverse DNS on sslhost is OK: SSL OK - wiki.eastsidegamers.no reverse DNS resolves to cp36.wikitide.net - CNAME OK
[23:31:30] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:35:35] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.54, 20.36, 18.69
[23:37:29] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 14.40, 18.36, 18.17
[23:45:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.85, 3.58, 4.00
[23:46:11] <icinga-miraheze>	 PROBLEM - unhingedwiki.paralleldv.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - unhingedwiki.paralleldv.org All nameservers failed to answer the query.
[23:47:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.70, 4.12, 4.15
[23:49:08] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[23:51:04] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.076 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[23:54:18] <icinga-miraheze>	 PROBLEM - wiki.gowashtenaw.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gowashtenaw.org All nameservers failed to answer the query.
[23:55:19] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.alathramc.com All nameservers failed to answer the query.
[23:58:10] <icinga-miraheze>	 PROBLEM - wiki.orivium.io - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.orivium.io All nameservers failed to answer the query.
[23:58:10] <icinga-miraheze>	 PROBLEM - wiki.mostly.vet - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mostly.vet All nameservers failed to answer the query.