[00:04:38] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 18929MiB (21% inode=98%); [00:12:44] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 54872MiB (12% inode=99%); [00:14:31] [02puppet] 07Universal-Omega left a file comment in pull request #4198 03a9078cc: Is there a reason we don't use the same format as grafana above? 13https://github.com/miraheze/puppet/pull/4198#discussion_r1970714102 [00:14:32] [02puppet] 07Universal-Omega left a file comment in pull request #4198 03a9078cc: Same as above 13https://github.com/miraheze/puppet/pull/4198#discussion_r1970714252 [00:16:36] [02puppet] 07paladox left a file comment in pull request #4198 03a9078cc: Yes, because we don't use: […] 13https://github.com/miraheze/puppet/pull/4198#discussion_r1970716625 [00:16:40] [02puppet] 07paladox left a file comment in pull request #4198 03a9078cc: see above 13https://github.com/miraheze/puppet/pull/4198#discussion_r1970716765 [00:40:58] [02puppet] 07Universal-Omega approved pull request #4198 13https://github.com/miraheze/puppet/pull/4198#pullrequestreview-2642764150 [00:41:04] [02puppet] 07Universal-Omega merged 07paladox's pull request #4198: wikitide-backup: Support backing up mattermost and puppetdb (03master...03paladox-patch-5) 13https://github.com/miraheze/puppet/pull/4198 [00:41:04] [02puppet] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/3057fed019c146fed56fe3208c4adf7adf45d846 [00:41:04] 02puppet/03master 07paladox 033057fed wikitide-backup: Support backing up mattermost and puppetdb (#4198)… [00:41:06] [02puppet] 07Universal-Omega 04deleted 03paladox-patch-5 at 03a9078cc 13https://github.com/miraheze/puppet/commit/a9078cc [01:02:35] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:16:31] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.57, 15.06, 7.61 [01:18:26] PROBLEM - mattermost1 Backups Mattermost DB on mattermost1 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/mattermost-backup/mattermost-db/mattermost-db.log [01:18:31] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 3.49, 10.44, 6.82 [01:18:39] PROBLEM - mattermost1 Backups Mattermost Data on mattermost1 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/mattermost-backup/mattermost-data/mattermost-data.log [01:52:17] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [02:18:21] PROBLEM - puppet181 Backups PuppetDB on puppet181 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/mattermost-backup/mattermost-data/mattermost-data.log [02:26:55] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importDump.php --wiki=spongebobfanonwiki dump.xml --no-updates (START) [02:26:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:32:50] miraheze/dns - MacFan4000 the build passed. [02:34:10] [02puppet] 07paladox opened pull request #4202: wikitide-backup: remove verbose mode from tar (03master...03paladox-patch-5) 13https://github.com/miraheze/puppet/pull/4202 [02:34:16] [02puppet] 07coderabbitai[bot] commented on pull request #4202: --- […] 13https://github.com/miraheze/puppet/pull/4202#issuecomment-2683744042 [02:34:32] [02puppet] 07paladox merged pull request #4202: wikitide-backup: remove verbose mode from tar (03master...03paladox-patch-5) 13https://github.com/miraheze/puppet/pull/4202 [02:34:32] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/8099c3c1d1c1786a14b550cf9289024832e11e13 [02:34:32] 02puppet/03master 07paladox 038099c3c wikitide-backup: remove verbose mode from tar (#4202) [02:34:33] [02puppet] 07paladox 04deleted 03paladox-patch-5 at 0361e3636 13https://github.com/miraheze/puppet/commit/61e3636 [02:43:05] [02puppet] 07paladox created 03paladox-patch-5 (+1 new commit) 13https://github.com/miraheze/puppet/commit/4ba654bd2d3b [02:43:05] 02puppet/03paladox-patch-5 07paladox 034ba654b wikitide-backup: allow setting whether use gateway [02:43:09] [02puppet] 07paladox opened pull request #4203: wikitide-backup: allow setting whether use gateway (03master...03paladox-patch-5) 13https://github.com/miraheze/puppet/pull/4203 [02:43:15] [02puppet] 07coderabbitai[bot] commented on pull request #4203: --- […] 13https://github.com/miraheze/puppet/pull/4203#issuecomment-2683753929 [02:44:43] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-5 13https://github.com/miraheze/puppet/commit/720eb546965f02dff8ef77e52a45127744981595 [02:44:43] 02puppet/03paladox-patch-5 07paladox 03720eb54 Update wikitide-backup.py.erb [02:45:10] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-5 13https://github.com/miraheze/puppet/commit/474e1c6da3f0ce847c60eeff2eb2b3f577a0a25f [02:45:11] 02puppet/03paladox-patch-5 07paladox 03474e1c6 Update backup.pp [02:45:27] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-5 13https://github.com/miraheze/puppet/commit/1bd632cb0fe64d06c5f9cba0f57846140e824815 [02:45:27] 02puppet/03paladox-patch-5 07paladox 031bd632c Update mattermost1.yaml [02:46:51] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-5 13https://github.com/miraheze/puppet/commit/426c3a784c6289f36f6ee8343cfb868e674f7a7d [02:46:52] 02puppet/03paladox-patch-5 07paladox 03426c3a7 Update backup.pp [02:47:20] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-5 13https://github.com/miraheze/puppet/commit/5c97d34103a4abb70bfd6fd44a1da8e28d82749b [02:47:20] 02puppet/03paladox-patch-5 07paladox 035c97d34 Update backup.pp [02:48:01] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-5 13https://github.com/miraheze/puppet/commit/b5de15624e62ac307bf0634bfc29840951eab638 [02:48:01] 02puppet/03paladox-patch-5 07paladox 03b5de156 Update wikitide-backup.py.erb [02:49:38] PROBLEM - mwtask161 Puppet on mwtask161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:49:44] PROBLEM - swiftobject151 Puppet on swiftobject151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:50:01] PROBLEM - mw183 Puppet on mw183 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:52:24] [02puppet] 07paladox merged pull request #4203: wikitide-backup: allow setting whether use gateway (03master...03paladox-patch-5) 13https://github.com/miraheze/puppet/pull/4203 [02:52:24] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/475f754006557360a7bbad6b99e05ad05f0c7345 [02:52:24] 02puppet/03master 07paladox 03475f754 wikitide-backup: allow setting whether use gateway (#4203) [02:52:26] [02puppet] 07paladox 04deleted 03paladox-patch-5 at 03b5de156 13https://github.com/miraheze/puppet/commit/b5de156 [02:55:38] RECOVERY - mw181.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000. [02:57:52] RECOVERY - mw173.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000. [03:01:55] RECOVERY - mw174.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000. [03:03:47] RECOVERY - mw172.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000. [03:04:17] RECOVERY - mw153.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000. [03:04:29] RECOVERY - mw152.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000. [03:04:41] RECOVERY - mw171.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000. [03:17:35] RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:18:13] RECOVERY - mw183 Puppet on mw183 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [03:18:51] RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:45:52] RECOVERY - mw183.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000. [06:18:58] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49868MiB (10% inode=99%); [06:29:18] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 20.64, 19.91, 14.59 [06:33:16] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 20.31, 20.12, 15.92 [07:01:21] [02statichelp] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/98d3669bae4edebc14231d86cc9ff4e979f2220a [07:01:21] 02statichelp/03main 07Universal-Omega 0398d3669 Auto-update Tech namespace pages 2025-02-26 06:59:58.585137 [07:55:40] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 21.62, 20.34, 16.30 [07:57:40] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 11.71, 17.66, 15.85 [08:03:39] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.07, 20.56, 17.72 [08:05:39] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 18.01, 20.03, 17.91 [09:02:29] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.25, 20.51, 13.92 [09:04:26] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 18.29, 19.75, 14.45 [09:08:18] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 20.43, 20.56, 15.98 [09:10:15] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 11.61, 17.72, 15.52 [09:45:39] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 28.88, 16.21, 10.96 [09:47:39] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 7.19, 13.08, 10.53 [11:25:39] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 21.46, 15.53, 8.37 [11:27:40] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 17.92, 16.72, 9.71 [11:47:04] PROBLEM - db181 NTP time on db181 is WARNING: NTP WARNING: Offset -0.1022686064 secs [11:49:09] RECOVERY - db181 NTP time on db181 is OK: NTP OK: Offset -0.05418145657 secs [12:08:37] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.04, 8.81, 4.26 [12:10:37] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 10.47, 9.22, 4.97 [12:10:41] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 4 backends are down. mw152 mw171 mw163 mw184 [12:10:48] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 5 backends are down. mw152 mw171 mw153 mw163 mw184 [12:12:37] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 13.25, 10.21, 5.80 [12:15:34] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:17:39] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4097 bytes in 7.362 second response time [12:19:36] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:20:37] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 6.27, 11.25, 8.84 [12:21:31] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4075 bytes in 0.674 second response time [12:22:37] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 9.44, 10.19, 8.70 [12:24:01] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:26:00] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4075 bytes in 0.053 second response time [12:27:46] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:32:37] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 12.93, 11.59, 9.60 [12:33:49] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4075 bytes in 8.358 second response time [12:36:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:38:41] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [12:38:48] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [12:42:41] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 7 backends are down. mw181 mw182 mw163 mw164 mw173 mw183 mw184 [12:42:48] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 7 backends are down. mw182 mw154 mw163 mw164 mw173 mw183 mw184 [12:45:05] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:45:57] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [12:46:37] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 6.80, 10.98, 11.45 [12:47:56] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 57% [12:49:15] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4075 bytes in 1.873 second response time [12:50:16] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [12:50:37] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 15.36, 11.27, 11.26 [12:51:53] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [12:52:37] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 10.16, 10.77, 11.08 [12:53:52] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 53% [12:54:37] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 21.41, 14.64, 12.45 [12:55:50] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 20% [12:56:00] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4097 bytes in 0.066 second response time [12:56:37] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 6.17, 11.55, 11.62 [12:56:41] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [12:56:48] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [13:00:37] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.02, 5.68, 9.18 [13:01:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:28:56] [02CreateWiki] 07BlankEclair created 03null-escape (+1 new commit) 13https://github.com/miraheze/CreateWiki/commit/e7209e23ec9c [13:28:56] 02CreateWiki/03null-escape 07BlankEclair 03e7209e2 Fix escaping NULL database records… [13:29:18] [02CreateWiki] 07BlankEclair opened pull request #676: Fix escaping NULL database records (03master...03null-escape) 13https://github.com/miraheze/CreateWiki/pull/676 [13:29:25] [02CreateWiki] 07coderabbitai[bot] commented on pull request #676: --- […] 13https://github.com/miraheze/CreateWiki/pull/676#issuecomment-2684962211 [13:31:40] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 23.01, 14.56, 7.00 [13:32:22] PROBLEM - cloud18 NTP time on cloud18 is WARNING: NTP WARNING: Offset -0.1047020853 secs [13:34:22] RECOVERY - cloud18 NTP time on cloud18 is OK: NTP OK: Offset -0.08543777466 secs [13:34:56] [02CreateWiki] 07paladox merged 07BlankEclair's pull request #676: Fix escaping NULL database records (03master...03null-escape) 13https://github.com/miraheze/CreateWiki/pull/676 [13:34:56] [02CreateWiki] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/CreateWiki/commit/d29ad82eb462cd5bf817777e177cf2d7718d928c [13:34:57] 02CreateWiki/03master 07Claire 03d29ad82 Fix escaping NULL database records (#676)… [13:34:59] [02CreateWiki] 07paladox 04deleted 03null-escape at 03e7209e2 13https://github.com/miraheze/CreateWiki/commit/e7209e2 [13:35:24] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': 'CreateWiki'} to all [13:35:27] !log [paladox@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [13:35:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:35:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:35:44] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.43', '1.44'], 'upgrade_extensions': 'CreateWiki'} to test151 [13:35:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:37:39] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 20.07, 19.06, 11.54 [13:40:42] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': ['CreateWiki', 'SimpleBlogPage']} to all [13:40:46] !log [paladox@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [13:40:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:40:54] !log [paladox@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [13:40:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:40:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:41:20] miraheze/CreateWiki - BlankEclair the build passed. [13:43:39] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.13, 21.02, 14.84 [13:46:12] PROBLEM - mwtask161 NTP time on mwtask161 is WARNING: NTP WARNING: Offset -0.100739032 secs [13:47:45] miraheze/CreateWiki - paladox the build passed. [13:48:16] RECOVERY - mwtask161 NTP time on mwtask161 is OK: NTP OK: Offset -0.09258756042 secs [13:48:25] PROBLEM - cloud18 NTP time on cloud18 is WARNING: NTP WARNING: Offset -0.1011312008 secs [13:48:40] PROBLEM - os162 NTP time on os162 is WARNING: NTP WARNING: Offset -0.1036320925 secs [13:49:39] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 24.05, 22.95, 17.65 [13:50:25] RECOVERY - cloud18 NTP time on cloud18 is OK: NTP OK: Offset -0.09452307224 secs [13:50:43] RECOVERY - os162 NTP time on os162 is OK: NTP OK: Offset -0.09431606531 secs [13:51:40] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 21.49, 22.53, 18.15 [13:53:39] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 25.33, 23.68, 19.11 [13:55:39] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 23.53, 23.46, 19.58 [13:57:39] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 24.09, 23.37, 20.01 [14:01:40] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 21.94, 23.03, 20.67 [14:05:39] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 3.62, 15.78, 18.56 [14:06:29] [02mw-config] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/934e8e6196a8 [14:06:29] 02mw-config/03paladox-patch-2 07paladox 03934e8e6 Fix install for simpleblogpage with NS [14:06:32] [02mw-config] 07paladox opened pull request #5834: Fix install for simpleblogpage with NS (03master...03paladox-patch-2) 13https://github.com/miraheze/mw-config/pull/5834 [14:07:32] [02mw-config] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/mw-config/commit/129a99d3d9346984da46680a209503003e108ea5 [14:07:33] 02mw-config/03paladox-patch-2 07paladox 03129a99d Update ManageWikiExtensions.php [14:07:41] miraheze/mw-config - paladox the build passed. [14:08:33] miraheze/mw-config - paladox the build passed. [14:55:39] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.50, 18.26, 11.81 [14:59:39] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 4.17, 13.86, 11.78 [15:15:31] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': ['CreateWiki', 'SimpleBlogPage']} to all [15:15:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:15:42] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.43', '1.44'], 'upgrade_extensions': 'CreateWiki'} to test151 [15:15:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:16:38] !log [paladox@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [15:16:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:19:11] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.43', '1.44'], 'upgrade_extensions': ['CreateWiki', 'SimpleBlogPage']} to test151 [15:19:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:19:27] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/c1dcaba135b9f4a096ba29a87cdf26213c3a092a [15:19:27] 02ssl/03master 07MacFan4000 03c1dcaba add abydos-jk.kro.kr and wiki.metacraft.nu [15:22:40] [02mw-config] 07paladox merged pull request #5834: Fix install for simpleblogpage with NS (03master...03paladox-patch-2) 13https://github.com/miraheze/mw-config/pull/5834 [15:22:40] [02mw-config] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/00ac8829d3d43f2ea55b899d662d9f580d6e1f80 [15:22:41] 02mw-config/03master 07paladox 0300ac882 Fix install for simpleblogpage with NS (#5834) [15:22:43] [02mw-config] 07paladox 04deleted 03paladox-patch-2 at 03129a99d 13https://github.com/miraheze/mw-config/commit/129a99d [15:23:37] miraheze/mw-config - paladox the build passed. [15:23:38] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': ['CreateWiki', 'SimpleBlogPage']} to all - SUCCESS in 487s [15:23:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:25:28] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.43', '1.44'], 'upgrade_extensions': ['CreateWiki', 'SimpleBlogPage']} to test151 - SUCCESS in 377s [15:25:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:27:58] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=trollpastawiki (START) [15:28:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:29:04] [02dns] 07Reception123 created 03revert-547-patch-34 (+1 new commit) 13https://github.com/miraheze/dns/commit/411416c1608d [15:29:04] 02dns/03revert-547-patch-34 07Reception123 03411416c Revert "Delete zones/9b9t.wiki (#547)"… [15:29:09] [02dns] 07Reception123 opened pull request #644: Revert "Delete zones/9b9t.wiki" (03master...03revert-547-patch-34) 13https://github.com/miraheze/dns/pull/644 [15:29:20] [02dns] 07Reception123 merged pull request #644: Revert "Delete zones/9b9t.wiki" (03master...03revert-547-patch-34) 13https://github.com/miraheze/dns/pull/644 [15:29:22] [02dns] 07Reception123 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/1fec562b073cab064fb4f690fc4e659afd353bb3 [15:29:24] [02dns] 07Reception123 04deleted 03revert-547-patch-34 at 03411416c 13https://github.com/miraheze/dns/commit/411416c [15:29:25] 02dns/03master 07Reception123 031fec562 Revert "Delete zones/9b9t.wiki" (#644)… [15:30:24] miraheze/dns - Reception123 the build passed. [15:30:59] miraheze/dns - Reception123 the build passed. [15:31:55] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/5b7a99a1e9b86c62546f860b209afa96a0cae328 [15:31:55] 02ssl/03master 07WikiTideSSLBot 035b7a99a Bot: Add SSL cert for 9b9t.wiki… [15:40:55] RECOVERY - mw162.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000. [15:46:36] RECOVERY - mw184.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000. [15:50:42] RECOVERY - mw163.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000. [15:51:15] RECOVERY - mw154.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000. [15:51:28] RECOVERY - mw164.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000. [15:52:05] RECOVERY - mw151.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000. [15:59:52] RECOVERY - mw182.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000. [16:02:01] RECOVERY - mw161.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000. [16:11:02] RECOVERY - www.christipedia.nl - Cloudflare on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Tue 27 May 2025 03:01:50 PM GMT +0000. [16:33:00] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/fe0095d7de1bf756c87c71dc5118af7d39789637 [16:33:00] 02ssl/03master 07MacFan4000 03fe0095d add wiki.smarthcm.cloud [17:10:42] !log reboot mattermost1 [17:10:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:53:40] [02puppet] 07Universal-Omega created 03mariadb-read_only (+1 new commit) 13https://github.com/miraheze/puppet/commit/e9a34bfdf9a5 [17:53:40] 02puppet/03mariadb-read_only 07CosmicAlpha 03e9a34bf mariadb: default to read_only on startup [17:53:45] [02puppet] 07Universal-Omega opened pull request #4204: mariadb: default to read_only on startup (03master...03mariadb-read_only) 13https://github.com/miraheze/puppet/pull/4204 [17:53:52] [02puppet] 07coderabbitai[bot] commented on pull request #4204: --- […] 13https://github.com/miraheze/puppet/pull/4204#issuecomment-2685790593 [18:00:19] [02puppet] 07Universal-Omega merged pull request #4204: mariadb: default to read_only on startup (03master...03mariadb-read_only) 13https://github.com/miraheze/puppet/pull/4204 [18:00:19] [02puppet] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/4f90c5e89d6291c572aeb783318e8b19047a2027 [18:00:19] 02puppet/03master 07CosmicAlpha 034f90c5e mariadb: default to read_only on startup (#4204)… [18:00:20] [02puppet] 07Universal-Omega 04deleted 03mariadb-read_only at 03e9a34bf 13https://github.com/miraheze/puppet/commit/e9a34bf [18:12:22] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importDump.php --wiki=spongebobfanonwiki dump.xml --no-updates (END - exit=2) [18:12:23] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=spongebobfanonwiki (START) [18:12:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:12:24] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=spongebobfanonwiki (END - exit=2) [18:12:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:12:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:15:13] [02puppet] 07Universal-Omega merged pull request #4192: Temp allow cloud ssh without bastion (03master...03cloud-ssh) 13https://github.com/miraheze/puppet/pull/4192 [18:15:14] [02puppet] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/1ffee00e0f6c9b01d26178d2d043480e76e155ea [18:15:14] 02puppet/03master 07CosmicAlpha 031ffee00 Temp allow cloud ssh without bastion (#4192)… [18:15:15] [02puppet] 07Universal-Omega 04deleted 03cloud-ssh at 03b6bc422 13https://github.com/miraheze/puppet/commit/b6bc422 [18:27:06] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=trollpastawiki (END - exit=0) [18:27:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:30:37] @Technology Team at this time please stop any maintenance script runs (imports, etc...) and hold any actions on servers, or github commits until the maintenance is complete to avoid interruptions etc.... Thank you! [18:30:55] Good thing mine just finished! [18:31:00] I was worried it wouldn't finish in time [18:31:16] Yep that's good! [18:42:44] Hi, I have questions/feedback regarding WX3270 latest version. Whom may I address? [18:43:17] WStanzl: I have no idea what WX3720 is? Can you expand? [18:44:20] 3270 Terminal emulator, which came out 2 days ago with a new version that seems to have a couple of problems [18:44:20] https://x3270.miraheze.org/wiki/Main_Page [18:45:00] WStanzl: we're just the host, we can't help you with technical issues with WX3720 [18:46:08] WStanzl: see https://x3270.miraheze.org/wiki/Reporting_issues when our maintenance is over [18:46:08] Ok, I see. there seems to be no other contact mentioned but this. [18:46:09] Yan you find out who owns x3270 ? [18:46:24] oh thanks [18:46:31] WStanzl: there is an email to contact on there [18:46:43] But our maintenance has just started [18:46:46] oh thanks I ws unaware of that [18:46:52] okay no prob [18:47:11] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 6878 bytes in 0.116 second response time [18:47:13] PROBLEM - db171 MariaDB Connections on db171 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connect [18:47:13] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 PROBLEM - db151 MariaDB Connections on db151 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db151.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connect [18:47:14] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db151.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 PROBLEM - db172 MariaDB Connections on db172 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db172.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connect [18:47:14] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db172.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4714 bytes in 0.078 second response time [18:47:21] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [18:47:28] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4714 bytes in 0.079 second response time [18:47:28] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4714 bytes in 0.082 second response time [18:47:35] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4714 bytes in 0.077 second response time [18:47:38] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 8191 bytes in 0.156 second response time [18:47:44] PROBLEM - db181 MariaDB Connections on db181 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connect [18:47:44] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 426 bytes in 0.039 second response time [18:47:47] PROBLEM - mw174 MediaWiki Rendering on mw174 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4714 bytes in 0.100 second response time [18:47:49] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 6878 bytes in 0.103 second response time [18:47:50] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4714 bytes in 0.087 second response time [18:47:52] PROBLEM - db161 MariaDB on db161 is CRITICAL: Can't connect to server on 'db161.wikitide.net' (115) [18:47:53] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 6878 bytes in 0.600 second response time [18:47:57] PROBLEM - db182 MariaDB Connections on db182 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connect [18:47:57] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 PROBLEM - db161 MariaDB Connections on db161 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db161.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connect [18:48:02] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db161.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 PROBLEM - db182 MariaDB on db182 is CRITICAL: Can't connect to server on 'db182.wikitide.net' (115) [18:48:03] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4714 bytes in 0.080 second response time [18:48:09] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [18:48:16] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 6880 bytes in 3.092 second response time [18:48:18] PROBLEM - mw164 MediaWiki Rendering on mw164 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4716 bytes in 3.086 second response time [18:48:21] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4716 bytes in 3.082 second response time [18:48:37] PROBLEM - mw184 MediaWiki Rendering on mw184 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4716 bytes in 3.088 second response time [18:48:39] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4716 bytes in 3.090 second response time [18:48:44] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4716 bytes in 3.079 second response time [18:48:47] PROBLEM - mw154 MediaWiki Rendering on mw154 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4716 bytes in 3.083 second response time [18:48:48] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4716 bytes in 3.091 second response time [18:48:52] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4716 bytes in 3.089 second response time [18:48:53] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4716 bytes in 3.086 second response time [18:48:53] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: connect to address 10.0.16.137 port 5666: No route to hostconnect to host 10.0.16.137 port 5666: No route to host [18:48:53] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4279 bytes in 2.021 second response time [18:49:03] PROBLEM - cp37 conntrack_table_size on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [18:49:03] PROBLEM - cp37 Nginx Backend for mwtask171 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [18:49:03] PROBLEM - cp37 Nginx Backend for mw171 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [18:49:03] PROBLEM - cp37 Nginx Backend for mwtask151 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [18:49:03] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp37.wikitide.net port 443 after 261 ms: Couldn't connect to server [18:49:04] PROBLEM - cp37 Nginx Backend for mon181 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [18:49:04] PROBLEM - cp37 Nginx Backend for mw174 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [18:49:05] PROBLEM - cp37 Nginx Backend for swiftproxy161 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [18:49:06] PROBLEM - cp37 Nginx Backend for mw164 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [18:49:06] PROBLEM - cp37 Nginx Backend for mw182 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [18:49:28] Well icinga went too now [18:49:37] At least it's quiet [18:58:25] Hey yeah [20:15:47] [02dns] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/73998adfeba9c36c859bf35a7756250a99e3b8f6 [20:15:47] 02dns/03master 07CosmicAlpha 0373998ad Update management interface IPs [20:16:45] miraheze/dns - Universal-Omega the build passed. [20:26:43] [02dns] 07Universal-Omega created 03cloud (+1 new commit) 13https://github.com/miraheze/dns/commit/29f27c8a969f [20:26:43] 02dns/03cloud 07CosmicAlpha 0329f27c8 Add cloud19 and cloud20 [20:26:48] [02dns] 07Universal-Omega opened pull request #645: Add cloud19 and cloud20 (03master...03cloud) 13https://github.com/miraheze/dns/pull/645 [20:28:44] miraheze/dns - Universal-Omega the build passed. [20:29:15] [02dns] 07Universal-Omega merged pull request #645: Add cloud19 and cloud20 (03master...03cloud) 13https://github.com/miraheze/dns/pull/645 [20:29:16] [02dns] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/0ef88ad442a4041b7e1dffa0b226ab4478aa560d [20:29:16] 02dns/03master 07CosmicAlpha 030ef88ad Add cloud19 and cloud20 (#645) [20:29:17] [02dns] 07Universal-Omega 04deleted 03cloud at 0329f27c8 13https://github.com/miraheze/dns/commit/29f27c8 [20:30:14] miraheze/dns - Universal-Omega the build passed. [20:41:37] PROBLEM - cp37 Nginx Backend for mw162 on cp37 is CRITICAL: connect to address localhost and port 8116: Connection refused [20:41:37] PROBLEM - cp37 Nginx Backend for mw151 on cp37 is CRITICAL: connect to address localhost and port 8113: Connection refused [20:41:37] RECOVERY - db181 MariaDB on db181 is OK: Uptime: 33 Threads: 1 Questions: 1 Slow queries: 0 Opens: 16 Open tables: 10 Queries per second avg: 0.030 [20:41:38] PROBLEM - cp37 Nginx Backend for reports171 on cp37 is CRITICAL: connect to address localhost and port 8205: Connection refused [20:41:38] PROBLEM - cp37 Nginx Backend for mw161 on cp37 is CRITICAL: connect to address localhost and port 8115: Connection refused [20:41:38] PROBLEM - cp36 Puppet on cp36 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:41:39] RECOVERY - db182 MariaDB Connections on db182 is OK: OK connection usage: 0.2%Current connections: 2 [20:41:39] RECOVERY - db182 MariaDB on db182 is OK: Uptime: 19 Threads: 2 Questions: 27 Slow queries: 0 Opens: 24 Open tables: 17 Queries per second avg: 1.421 [20:41:40] RECOVERY - db181 MariaDB Connections on db181 is OK: OK connection usage: 0.1%Current connections: 1 [20:41:41] PROBLEM - cp37 Nginx Backend for mw181 on cp37 is CRITICAL: connect to address localhost and port 8119: Connection refused [20:41:41] PROBLEM - cp37 Nginx Backend for test151 on cp37 is CRITICAL: connect to address localhost and port 8181: Connection refused [20:41:41] RECOVERY - db161 MariaDB Connections on db161 is OK: OK connection usage: 0.1%Current connections: 1 [20:41:43] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 mw184 mediawiki [20:41:55] PROBLEM - cp37 Nginx Backend for puppet181 on cp37 is CRITICAL: connect to address localhost and port 8204: Connection refused [20:41:58] PROBLEM - cp37 Nginx Backend for mw152 on cp37 is CRITICAL: connect to address localhost and port 8114: Connection refused [20:41:59] PROBLEM - cp37 Nginx Backend for mw153 on cp37 is CRITICAL: connect to address localhost and port 8121: Connection refused [20:42:02] PROBLEM - cp37 Nginx Backend for mw163 on cp37 is CRITICAL: connect to address localhost and port 8123: Connection refused [20:42:03] PROBLEM - cp37 Nginx Backend for mw184 on cp37 is CRITICAL: connect to address localhost and port 8128: Connection refused [20:42:04] PROBLEM - cp37 Nginx Backend for mw172 on cp37 is CRITICAL: connect to address localhost and port 8118: Connection refused [20:42:05] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.624 second response time [20:42:05] RECOVERY - cloud15 IPMI Sensors on cloud15 is OK: IPMI Status: OK [20:42:07] PROBLEM - cp37 Nginx Backend for matomo151 on cp37 is CRITICAL: connect to address localhost and port 8203: Connection refused [20:42:08] PROBLEM - db151 MariaDB on db151 is CRITICAL: Can't connect to server on 'db151.wikitide.net' (115) [20:42:09] PROBLEM - cp37 Nginx Backend for mw154 on cp37 is CRITICAL: connect to address localhost and port 8122: Connection refused [20:42:09] RECOVERY - cp37 conntrack_table_size on cp37 is OK: OK: nf_conntrack is 0 % full [20:42:11] PROBLEM - cp37 PowerDNS Recursor on cp37 is CRITICAL: Domain 'wikitide.net' was not found by the server [20:42:14] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.314 second response time [20:42:15] RECOVERY - db171 MariaDB Connections on db171 is OK: OK connection usage: 0.1%Current connections: 1 [20:42:17] RECOVERY - ping on cp37 is OK: PING OK - Packet loss = 0%, RTA = 0.33 ms [20:42:18] PROBLEM - cp37 Nginx Backend for phorge171 on cp37 is CRITICAL: connect to address localhost and port 8202: Connection refused [20:42:20] PROBLEM - cp37 Nginx Backend for mw173 on cp37 is CRITICAL: connect to address localhost and port 8125: Connection refused [20:42:21] PROBLEM - cp37 Nginx Backend for mw183 on cp37 is CRITICAL: connect to address localhost and port 8127: Connection refused [20:42:23] RECOVERY - db161 MariaDB on db161 is OK: Uptime: 77 Threads: 1 Questions: 9 Slow queries: 0 Opens: 20 Open tables: 14 Queries per second avg: 0.116 [20:42:24] RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.316 second response time [20:42:24] PROBLEM - cp37 Nginx Backend for swiftproxy171 on cp37 is CRITICAL: connect to address localhost and port 8207: Connection refused [20:42:25] PROBLEM - cp37 Nginx Backend for mwtask181 on cp37 is CRITICAL: connect to address localhost and port 8160: Connection refused [20:42:28] PROBLEM - cp37 NTP time on cp37 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o