[00:00:43] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:25] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/05c052a8878a...422e12ccccbc [00:02:27] [02WikiTideOrg/ssl] 07WikiTideSSLBot 03422e12c - Bot: Update SSL cert for sdiy.info [00:02:43] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:06:05] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 6.86, 6.85, 6.80 [00:14:01] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: LOAD OK - total load average: 6.42, 6.74, 6.79 [00:17:59] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 6.78, 6.81, 6.82 [00:19:59] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: LOAD OK - total load average: 6.33, 6.68, 6.77 [00:23:59] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 7.55, 7.08, 6.91 [00:38:38] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:36] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.145 second response time [02:49:59] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: LOAD CRITICAL - total load average: 8.02, 7.49, 7.28 [02:51:59] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 7.62, 7.48, 7.30 [02:59:50] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:01:52] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 5.190 second response time [03:11:59] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: LOAD CRITICAL - total load average: 8.15, 7.57, 7.41 [03:13:59] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 7.51, 7.44, 7.38 [04:39:59] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: LOAD CRITICAL - total load average: 8.12, 7.47, 7.30 [04:41:59] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 7.23, 7.32, 7.26 [05:05:33] [02WikiTideOrg/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/mediawiki-repos/compare/656cbe117b8f...2b852731eacb [05:05:34] [02WikiTideOrg/mediawiki-repos] 07Universal-Omega 032b85273 - ARCHIVED [05:07:09] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideMagic/compare/89b3619d5db0...d995a80d85c8 [05:07:11] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega 03d995a80 - ARCHIVED [05:07:21] [02WikiTideMagic] 07Universal-Omega closed pull request 03#9: Update mediawiki/mediawiki-codesniffer requirement from 42.0.0 to 43.0.0 - 13https://github.com/WikiTideOrg/WikiTideMagic/pull/9 [05:07:23] [02WikiTideMagic] 07dependabot[bot] commented on pull request 03#9: Update mediawiki/mediawiki-codesniffer requirement from 42.0.0 to 43.0.0 - 13https://github.com/WikiTideOrg/WikiTideMagic/pull/9#issuecomment-1925582774 [05:07:24] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-43.0.0 [05:07:27] [02WikiTideMagic] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-43.0.0 - 13https://github.com/WikiTideOrg/WikiTideMagic [05:08:06] [02WikiTideMagic] 07Universal-Omega closed pull request 03#8: Use miraheze/CreateWiki - 13https://github.com/WikiTideOrg/WikiTideMagic/pull/8 [05:08:09] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±6] 13https://github.com/WikiTideOrg/WikiTideMagic/compare/d995a80d85c8...ff517645bdd9 [05:08:12] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega 03ff51764 - Use miraheze/CreateWiki [05:08:15] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega deleted branch 03miraheze-CreateWiki [05:08:16] [02WikiTideMagic] 07Universal-Omega deleted branch 03miraheze-CreateWiki - 13https://github.com/WikiTideOrg/WikiTideMagic [05:11:15] [02WikiTideOrg/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ErrorPages/compare/d49f13c7e3fe...11f31718f572 [05:11:16] [02WikiTideOrg/ErrorPages] 07Universal-Omega 0311f3171 - ARCHIVED [05:12:52] [02WikiTideOrg/WikiTideDebug] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideDebug/compare/c2313aa06943...ffe47ab0eb8d [05:12:53] [02WikiTideOrg/WikiTideDebug] 07Universal-Omega 03ffe47ab - ARCHIVED [06:07:59] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: LOAD CRITICAL - total load average: 8.02, 7.55, 7.34 [06:09:59] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 7.51, 7.49, 7.35 [06:12:55] PROBLEM - jobrunner1 Disk Space on jobrunner1 is WARNING: DISK WARNING - free space: / 5540MiB (10% inode=77%); [06:26:55] RECOVERY - jobrunner1 Disk Space on jobrunner1 is OK: DISK OK - free space: / 5653MiB (11% inode=77%); [07:02:55] PROBLEM - jobrunner1 Disk Space on jobrunner1 is WARNING: DISK WARNING - free space: / 5540MiB (10% inode=77%); [07:33:50] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:52] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 5.092 second response time [12:06:38] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:36] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.152 second response time [13:07:59] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: LOAD CRITICAL - total load average: 8.13, 7.53, 7.33 [13:09:59] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 7.09, 7.30, 7.27 [14:25:59] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: LOAD CRITICAL - total load average: 8.26, 7.39, 7.22 [14:27:59] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 6.97, 7.26, 7.20 [16:09:50] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:11:47] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.200 second response time [17:10:14] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03remove-MatomoAnalytics [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/commit/423268ddc37c [17:10:16] [02WikiTideOrg/mw-config] 07Universal-Omega 03423268d - Remove MatomoAnalytics [17:10:18] [02mw-config] 07Universal-Omega created branch 03remove-MatomoAnalytics - 13https://github.com/WikiTideOrg/mw-config [17:10:21] [02mw-config] 07Universal-Omega opened pull request 03#32: Remove MatomoAnalytics - 13https://github.com/WikiTideOrg/mw-config/pull/32 [17:11:12] WikiTideOrg/mw-config - Universal-Omega the build passed. [17:12:00] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/d15aa4512b83...ad35e0a3060e [17:12:02] [02WikiTideOrg/mw-config] 07Universal-Omega 03ad35e0a - Disable RequestImportDump [17:12:50] WikiTideOrg/mw-config - Universal-Omega the build passed. [17:31:50] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:33:48] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.165 second response time [18:44:34] RECOVERY - cp1 PowerDNS Recursor on cp1 is OK: DNS OK: 4.895 seconds response time. wikitide.org returns 2001:41d0:601:1100::4062,63.141.240.6 [18:44:39] RECOVERY - swiftac1 PowerDNS Recursor on swiftac1 is OK: DNS OK: 1.989 second response time. wikitide.org returns 146.59.44.171 [18:44:47] RECOVERY - swiftobject1 PowerDNS Recursor on swiftobject1 is OK: DNS OK: 1.958 second response time. wikitide.org returns 146.59.44.171 [18:44:51] RECOVERY - cp4 PowerDNS Recursor on cp4 is OK: DNS OK: 3.357 seconds response time. wikitide.org returns 139.99.236.151,2001:41d0:601:1100::4062 [18:45:03] RECOVERY - cp5 PowerDNS Recursor on cp5 is OK: DNS OK: 3.841 seconds response time. wikitide.org returns 15.235.167.159,2402:1f00:8000:800::1b3d [18:45:07] RECOVERY - puppet1 PowerDNS Recursor on puppet1 is OK: DNS OK: 1.904 second response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:45:17] RECOVERY - cp2 PowerDNS Recursor on cp2 is OK: DNS OK: 1.912 second response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:45:26] RECOVERY - jobchron1 PowerDNS Recursor on jobchron1 is OK: DNS OK: 1.888 second response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:45:30] RECOVERY - matomo1 PowerDNS Recursor on matomo1 is OK: DNS OK: 0.409 seconds response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:45:31] RECOVERY - os1 PowerDNS Recursor on os1 is OK: DNS OK: 1.906 second response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:45:34] RECOVERY - ldap1 LDAP on ldap1 is OK: LDAP OK - 0.127 seconds response time [18:45:36] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.283 second response time [18:45:38] RECOVERY - db1 PowerDNS Recursor on db1 is OK: DNS OK: 1.910 second response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:45:47] RECOVERY - bots1 PowerDNS Recursor on bots1 is OK: DNS OK: 1.907 second response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:45:53] RECOVERY - db1 MariaDB Connections on db1 is OK: OK connection usage: 4.8%Current connections: 24 [18:45:54] RECOVERY - jobrunner1 PowerDNS Recursor on jobrunner1 is OK: DNS OK: 0.161 seconds response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:45:58] RECOVERY - test1 PowerDNS Recursor on test1 is OK: DNS OK: 1.918 second response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:46:11] RECOVERY - mem1 PowerDNS Recursor on mem1 is OK: DNS OK: 1.913 second response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:46:13] RECOVERY - services1 PowerDNS Recursor on services1 is OK: DNS OK: 1.894 second response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:46:16] PROBLEM - wc.wikitide.org on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 15 day(s) (Tue 20 Feb 2024 12:20:10 PM GMT +0000). [18:46:18] RECOVERY - ldap1 PowerDNS Recursor on ldap1 is OK: DNS OK: 1.925 second response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:46:23] RECOVERY - cp6 PowerDNS Recursor on cp6 is OK: DNS OK: 2.600 seconds response time. wikitide.org returns 139.99.236.151,2402:1f00:8100:400::1d23 [18:46:28] RECOVERY - mw2 PowerDNS Recursor on mw2 is OK: DNS OK: 0.183 seconds response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:46:32] RECOVERY - www.greatamerica.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'greatamerica.wiki' will expire on Fri 05 Apr 2024 11:24:34 PM GMT +0000. [18:46:32] RECOVERY - beta.sagan4.org - LetsEncrypt on sslhost is OK: OK - Certificate 'beta.sagan4.org' will expire on Fri 05 Apr 2024 11:51:59 PM GMT +0000. [18:46:33] RECOVERY - wiki.colleimadcat.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.colleimadcat.com' will expire on Fri 05 Apr 2024 11:23:01 PM GMT +0000. [18:46:37] RECOVERY - swiftproxy1 PowerDNS Recursor on swiftproxy1 is OK: DNS OK: 0.081 seconds response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:46:38] RECOVERY - mw1 PowerDNS Recursor on mw1 is OK: DNS OK: 0.161 seconds response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:46:39] RECOVERY - prometheus1 PowerDNS Recursor on prometheus1 is OK: DNS OK: 0.120 seconds response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:46:41] RECOVERY - phorge1 PowerDNS Recursor on phorge1 is OK: DNS OK: 0.311 seconds response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:46:42] RECOVERY - wiki.burning.art - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [18:46:58] RECOVERY - bast1 PowerDNS Recursor on bast1 is OK: DNS OK: 1.907 second response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:47:01] RECOVERY - graylog1 PowerDNS Recursor on graylog1 is OK: DNS OK: 0.408 seconds response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:47:04] RECOVERY - mon1 PowerDNS Recursor on mon1 is OK: DNS OK: 0.047 seconds response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [18:47:13] RECOVERY - db1 MariaDB on db1 is OK: Uptime: 3684127 Threads: 23 Questions: 2709314628 Slow queries: 101101661 Opens: 542999 Open tables: 16119 Queries per second avg: 735.402 [18:47:48] RECOVERY - alpha.sagan4.org - LetsEncrypt on sslhost is OK: OK - Certificate 'meta.sagan4.org' will expire on Fri 05 Apr 2024 11:29:24 PM GMT +0000. [18:47:49] RECOVERY - mlrpgspeedruns.com - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [18:48:16] RECOVERY - rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rosettacode.org' will expire on Fri 05 Apr 2024 11:35:24 PM GMT +0000. [18:50:17] RECOVERY - www.rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rosettacode.org' will expire on Fri 05 Apr 2024 11:35:24 PM GMT +0000. [18:50:18] RECOVERY - data.lophocmatngu.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'data.lophocmatngu.wiki' will expire on Fri 19 Apr 2024 06:22:59 PM GMT +0000. [18:54:45] RECOVERY - projects.dmvpetridish.com - LetsEncrypt on sslhost is OK: OK - Certificate 'projects.dmvpetridish.com' will expire on Thu 29 Feb 2024 03:18:01 PM GMT +0000. [18:55:59] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: LOAD OK - total load average: 5.98, 6.35, 6.74 [18:58:21] RECOVERY - valorantjp.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'valorantjp.wiki' will expire on Fri 05 Apr 2024 11:17:32 PM GMT +0000. [18:59:53] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/1f36ce479069...7c259acf8f25 [18:59:55] [02WikiTideOrg/puppet] 07Universal-Omega 037c259ac - Update recursor [19:00:02] RECOVERY - wiki.myehs.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.myehs.eu' will expire on Sat 23 Mar 2024 02:44:20 AM GMT +0000. [19:00:02] RECOVERY - www.lgbtqia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lgbtqia.wiki' will expire on Fri 05 Apr 2024 11:40:36 PM GMT +0000. [19:00:03] RECOVERY - wiki.chevrine.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.chevrine.com' will expire on Thu 29 Feb 2024 03:19:26 PM GMT +0000. [19:00:50] RECOVERY - wikitide.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Fri 12 Apr 2024 05:51:56 PM GMT +0000. [19:00:51] RECOVERY - nexttide.org - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [19:00:53] RECOVERY - mason.sagan4.org - LetsEncrypt on sslhost is OK: OK - Certificate 'beta.sagan4.org' will expire on Fri 05 Apr 2024 11:51:59 PM GMT +0000. [19:03:47] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:04:35] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [19:05:35] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 934 bytes in 0.107 second response time [19:05:43] RECOVERY - www.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'polandballwiki.com' will expire on Fri 05 Apr 2024 11:07:53 PM GMT +0000. [19:06:13] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:06:38] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:06:42] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:07:01] PROBLEM - graylog1 PowerDNS Recursor on graylog1 is CRITICAL: Domain 'wikitide.org' was not found by the server [19:07:15] RECOVERY - www.farthestfrontier.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'farthestfrontier.wiki' will expire on Fri 19 Apr 2024 06:29:37 PM GMT +0000. [19:07:15] RECOVERY - hsck.lophocmatngu.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'lophocmatngu.wiki' will expire on Fri 12 Apr 2024 11:08:03 PM GMT +0000. [19:07:34] RECOVERY - meta.sagan4.org - LetsEncrypt on sslhost is OK: OK - Certificate 'meta.sagan4.org' will expire on Fri 05 Apr 2024 11:29:24 PM GMT +0000. [19:07:35] PROBLEM - sdiy.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sdiy.info' expires in 8 day(s) (Tue 13 Feb 2024 01:26:39 PM GMT +0000). [19:07:40] PROBLEM - services1 PowerDNS Recursor on services1 is CRITICAL: Domain 'wikitide.org' was not found by the server [19:07:53] PROBLEM - mem1 PowerDNS Recursor on mem1 is CRITICAL: Domain 'wikitide.org' was not found by the server [19:07:59] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 7.45, 6.97, 6.78 [19:08:14] PROBLEM - jobrunner1 PowerDNS Recursor on jobrunner1 is CRITICAL: Domain 'wikitide.org' was not found by the server [19:09:10] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:09:29] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/7c259acf8f25...55e2db5fb24e [19:09:31] [02WikiTideOrg/puppet] 07Universal-Omega 0355e2db5 - Add IPv4 [19:11:14] RECOVERY - prometheus1 Puppet on prometheus1 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [19:11:26] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:11:40] PROBLEM - wikitide.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 15 day(s) (Tue 20 Feb 2024 12:20:10 PM GMT +0000). [19:12:02] RECOVERY - jobrunner1 PowerDNS Recursor on jobrunner1 is OK: DNS OK: 0.218 seconds response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [19:12:08] RECOVERY - phorge1 Puppet on phorge1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:12:13] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:13:03] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:13:13] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:13:15] RECOVERY - newcascadia.net - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [19:14:57] RECOVERY - swiftobject1 Puppet on swiftobject1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:15:47] RECOVERY - mem1 PowerDNS Recursor on mem1 is OK: DNS OK: 0.339 seconds response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [19:16:21] RECOVERY - jobchron1 Puppet on jobchron1 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [19:16:26] RECOVERY - bast1 Puppet on bast1 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [19:16:28] RECOVERY - matomo1 Puppet on matomo1 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [19:16:33] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [19:16:45] !log [@test1] starting deploy of {'config': True, 'versions': ['1.40']} to [test1] [19:16:46] !log [@test1] finished deploy of {'config': True, 'versions': ['1.40']} to [test1] - SUCCESS in 0s [19:16:47] RECOVERY - swiftac1 Puppet on swiftac1 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [19:16:56] RECOVERY - db1 Puppet on db1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:17:01] RECOVERY - graylog1 PowerDNS Recursor on graylog1 is OK: DNS OK: 0.334 seconds response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [19:17:05] RECOVERY - bots1 Puppet on bots1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:17:09] !log [@test1] starting deploy of {'folders': '1.42/extensions/WikiTideMagic', 'versions': ['1.40']} to [test1] [19:17:12] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [19:17:25] RECOVERY - services1 PowerDNS Recursor on services1 is OK: DNS OK: 0.327 seconds response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [19:17:28] RECOVERY - swiftproxy1 Puppet on swiftproxy1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:17:29] RECOVERY - cp1 Puppet on cp1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:17:33] RECOVERY - test1 JobChron Service on test1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [19:17:43] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:24:39] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.232 second response time [19:25:40] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 638 bytes in 0.083 second response time [19:26:56] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.170 second response time [19:59:59] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: LOAD OK - total load average: 5.32, 6.06, 6.66 [20:15:59] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 6.94, 6.39, 6.34 [20:19:59] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: LOAD OK - total load average: 6.71, 6.71, 6.49 [20:30:56] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 6.78, 6.86, 6.69 [20:32:55] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: LOAD OK - total load average: 6.58, 6.75, 6.67 [20:36:51] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 7.06, 6.89, 6.75 [20:44:47] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: LOAD OK - total load average: 5.54, 6.56, 6.71