[00:00:25] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [00:02:10] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/a8ea379666e7...b8f4e4b99603 [00:02:12] [02WikiTideOrg/puppet] 07Universal-Omega 03b8f4e4b - mwscript: use new syntax [00:03:25] WikiTideOrg/puppet - Universal-Omega the build has errored. [00:06:41] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/b8f4e4b99603...cb87b7a7294b [00:06:44] [02WikiTideOrg/puppet] 07Universal-Omega 03cb87b7a - Remove 3.12 for now [00:08:12] WikiTideOrg/puppet - Universal-Omega the build passed. [00:13:05] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/cb87b7a7294b...2dc653476c10 [00:13:08] [02WikiTideOrg/puppet] 07Universal-Omega 032dc6534 - mwdeploy: use new syntax [00:13:51] WikiTideOrg/puppet - Universal-Omega the build passed. [00:18:13] WikiTideOrg/puppet - Universal-Omega the build has errored. [00:22:14] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/2dc653476c10...0ffed8299282 [00:22:15] [02WikiTideOrg/puppet] 07Universal-Omega 030ffed82 - re-add 3.12 [00:37:22] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03py3.12 [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/commit/e6c04b710a3b [00:37:25] [02WikiTideOrg/puppet] 07Universal-Omega 03e6c04b7 - test fix [00:37:27] [02puppet] 07Universal-Omega created branch 03py3.12 - 13https://github.com/WikiTideOrg/puppet [00:37:29] [02puppet] 07Universal-Omega opened pull request 03#28: test fix - 13https://github.com/WikiTideOrg/puppet/pull/28 [00:37:33] WikiTideOrg/puppet - Universal-Omega the build has errored. [01:02:47] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [01:04:14] [02puppet] 07Universal-Omega closed pull request 03#28: test fix - 13https://github.com/WikiTideOrg/puppet/pull/28 [01:04:17] [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03py3.12 [01:04:18] [02puppet] 07Universal-Omega deleted branch 03py3.12 - 13https://github.com/WikiTideOrg/puppet [01:04:52] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/0ffed8299282...683e49b4d812 [01:04:53] [02WikiTideOrg/puppet] 07Universal-Omega 03683e49b - Remove again Doesn't seem to work [01:10:50] WikiTideOrg/puppet - Universal-Omega the build passed. [01:21:10] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03mwscript [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/commit/f33ac614eeae [01:21:12] [02WikiTideOrg/puppet] 07Universal-Omega 03f33ac61 - mwscript: support run.php for longscripts [01:21:13] [02puppet] 07Universal-Omega created branch 03mwscript - 13https://github.com/WikiTideOrg/puppet [01:21:14] [02puppet] 07Universal-Omega opened pull request 03#29: mwscript: support run.php for longscripts - 13https://github.com/WikiTideOrg/puppet/pull/29 [01:22:27] WikiTideOrg/puppet - Universal-Omega the build has errored. [01:24:45] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03mwscript [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/f33ac614eeae...e5ee46d7fdaa [01:24:48] [02WikiTideOrg/puppet] 07Universal-Omega 03e5ee46d - use syscheck [01:24:50] [02puppet] 07Universal-Omega synchronize pull request 03#29: mwscript: support run.php for longscripts - 13https://github.com/WikiTideOrg/puppet/pull/29 [01:25:58] WikiTideOrg/puppet - Universal-Omega the build has errored. [01:26:24] [02puppet] 07Universal-Omega synchronize pull request 03#29: mwscript: support run.php for longscripts - 13https://github.com/WikiTideOrg/puppet/pull/29 [01:26:27] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03mwscript [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e5ee46d7fdaa...1e34f7835260 [01:26:29] [02WikiTideOrg/puppet] 07Universal-Omega 031e34f78 - Fix [01:27:31] WikiTideOrg/puppet - Universal-Omega the build has errored. [01:30:19] [02puppet] 07Universal-Omega synchronize pull request 03#29: mwscript: support run.php for longscripts - 13https://github.com/WikiTideOrg/puppet/pull/29 [01:30:22] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03mwscript [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/1e34f7835260...1e58d6a1e4f4 [01:30:23] [02WikiTideOrg/puppet] 07Universal-Omega 031e58d6a - Fix [01:31:50] WikiTideOrg/puppet - Universal-Omega the build passed. [01:32:24] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [01:34:28] [02puppet] 07Universal-Omega closed pull request 03#29: mwscript: support run.php for longscripts - 13https://github.com/WikiTideOrg/puppet/pull/29 [01:34:30] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/WikiTideOrg/puppet/compare/683e49b4d812...593d10e997d9 [01:34:32] [02WikiTideOrg/puppet] 07Universal-Omega 03593d10e - mwscript: support run.php for longscripts [01:34:33] [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03mwscript [01:34:34] [02puppet] 07Universal-Omega deleted branch 03mwscript - 13https://github.com/WikiTideOrg/puppet [01:35:47] WikiTideOrg/puppet - Universal-Omega the build passed. [01:44:49] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [01:53:27] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki All nameservers failed to answer the query. [02:04:12] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.408 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [02:23:24] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [02:33:11] RECOVERY - Host puppet1 is UP: PING OK - Packet loss = 0%, RTA = 0.39 ms [02:33:18] RECOVERY - puppet1 conntrack_table_size on puppet1 is OK: OK: nf_conntrack is 0 % full [02:33:19] PROBLEM - puppet1 HTTPS on puppet1 is WARNING: HTTP WARNING: HTTP/2 403 - 113 bytes in 0.022 second response time [02:33:19] RECOVERY - puppet1 Backups SSLKeys on puppet1 is OK: FILE_AGE OK: /var/log/sslkeys-backup.log is 419594 seconds old and 52 bytes [02:33:23] RECOVERY - puppet1 puppetdb on puppet1 is OK: TCP OK - 0.001 second response time on 10.0.0.100 port 8081 [02:34:28] RECOVERY - puppet1 NTP time on puppet1 is OK: NTP OK: Offset 0.0006161034107 secs [02:34:28] RECOVERY - puppet1 puppetserver on puppet1 is OK: TCP OK - 0.001 second response time on 10.0.0.100 port 8140 [02:34:32] RECOVERY - puppet1 Current Load on puppet1 is OK: LOAD OK - total load average: 0.50, 0.64, 0.28 [02:34:33] RECOVERY - puppet1 Backups Private on puppet1 is OK: FILE_AGE OK: /var/log/private-backup.log is 430380 seconds old and 52 bytes [02:35:13] RECOVERY - puppet1 ferm_active on puppet1 is OK: OK ferm input default policy is set [02:35:18] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:36:51] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:37:04] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:37:09] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:38:02] PROBLEM - services1 mathoid on services1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:08] PROBLEM - graylog1 HTTPS on graylog1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [02:38:42] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:38:51] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:39:19] PROBLEM - mem1 memcached on mem1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:39:32] @agentisai is something wrong ^ [02:39:33] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [02:39:35] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw1 [02:39:39] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [02:39:45] has to be a puppet change [02:40:08] disable puppet on all remaining servers temporarily maybe [02:40:16] PROBLEM - cp1 Varnish Backends on cp1 is CRITICAL: 1 backends are down. mw1 [02:41:06] RECOVERY - prometheus1 Puppet on prometheus1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:41:07] I disabled it on mw2 [02:41:10] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [02:41:18] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [02:41:29] PROBLEM - mw1 HTTPS on mw1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10001 milliseconds [02:41:30] hmmm there is nothing that should be causing it go down... [02:41:53] oh dns? [02:42:00] maybe not [02:42:10] oh yes the server change [02:42:32] PROBLEM - prometheus1 Prometheus on prometheus1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:42:39] we haven't changed servers yet but broght puppet back online [02:42:47] @agentisai [02:42:50] RECOVERY - phorge1 Puppet on phorge1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:42:55] what exactly is wrong? [02:43:10] I don't know but everything is going down [02:43:22] As puppet runs [02:43:30] I think it's ferm [02:43:35] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [02:43:52] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:43:58] it shouldnt be unless IPs have changed. ferm was tested to work on everything but cloud [02:44:14] only thing I can see in syslog is ferm's config changing [02:44:19] Ferm was done before the last puppet run [02:44:35] So puppet would have pickdd up on it before [02:45:08] PROBLEM - ldap1 LDAP on ldap1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:45:11] RECOVERY - swiftobject1 Puppet on swiftobject1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:45:14] @agentisai https://github.com/WikiTideOrg/puppet/commit/dab2db36279401e8480075b7b9d4a9cf28bfecf6 ? [02:45:31] PROBLEM - mail1 SMTP on mail1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:45:37] PROBLEM - mail1 webmail.wikitide.net HTTPS on mail1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:45:54] ohhhh [02:45:59] yeah, that's why [02:46:00] PROBLEM - mail1 IMAP on mail1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:46:12] I didn't expect cloud1 to survive so long [02:46:16] RECOVERY - cp1 Varnish Backends on cp1 is OK: All 7 backends are healthy [02:46:19] yeah [02:46:58] PROBLEM - mail1 HTTPS on mail1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [02:47:09] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 7 backends are healthy [02:47:16] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.218 second response time [02:47:18] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [02:47:24] puppetserver is stopped [02:47:33] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [02:47:33] PROBLEM - swiftobject1 Swift Object Service on swiftobject1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:35] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [02:47:39] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [02:47:42] RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/2 200 - 362 bytes in 0.024 second response time [02:48:03] thanks [02:49:15] I thought puppetserver is restarted when its stopped though now? oh wait no I only made it start on boot. [02:49:38] PROBLEM - mw1 ferm_active on mw1 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly [02:50:03] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/WikiTideMagic/maintenance/unattachFromCentralAuth.php --wiki=harrypotterwikitide (END - exit=256) [02:50:06] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [02:50:08] PROBLEM - puppet1 puppetserver on puppet1 is CRITICAL: connect to address 10.0.0.100 and port 8140: Connection refused [02:50:55] @agentisai did you stop ferm on mw1 just wondering? [02:51:05] Also swift is down [02:54:13] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-nodes [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/commit/73b2b3859f92 [02:54:16] [02WikiTideOrg/puppet] 07Universal-Omega 0373b2b38 - Fix nodes A temporary solution [02:54:18] [02puppet] 07Universal-Omega created branch 03fix-nodes - 13https://github.com/WikiTideOrg/puppet [02:54:20] [02puppet] 07Universal-Omega opened pull request 03#30: Fix nodes - 13https://github.com/WikiTideOrg/puppet/pull/30 [02:54:47] @agentisai ^ [02:55:33] yes [02:56:31] Alright. Just was wondering, merge my PR if your happy with the temporary solution I have to go for a minute. [03:00:01] [02puppet] 07AgentIsai closed pull request 03#30: Fix nodes - 13https://github.com/WikiTideOrg/puppet/pull/30 [03:00:04] [02WikiTideOrg/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/593d10e997d9...2b354b027aa6 [03:00:06] [02WikiTideOrg/puppet] 07Universal-Omega 032b354b0 - Fix nodes (#30) A temporary solution [03:00:09] [02WikiTideOrg/puppet] 07AgentIsai deleted branch 03fix-nodes [03:00:10] [02puppet] 07AgentIsai deleted branch 03fix-nodes - 13https://github.com/WikiTideOrg/puppet [03:00:11] done [03:01:13] thanks [03:06:51] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:07:05] PROBLEM - mem1 Puppet on mem1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:07:09] PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:07:38] RECOVERY - mw1 ferm_active on mw1 is OK: OK ferm input default policy is set [03:07:52] RECOVERY - puppet1 puppetserver on puppet1 is OK: TCP OK - 0.001 second response time on 10.0.0.100 port 8140 [03:09:14] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:09:50] RECOVERY - prometheus1 Prometheus on prometheus1 is OK: TCP OK - 0.001 second response time on 10.0.0.118 port 9090 [03:10:16] PROBLEM - cp1 Varnish Backends on cp1 is CRITICAL: 1 backends are down. mw1 [03:11:13] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.321 second response time [03:12:16] RECOVERY - cp1 Varnish Backends on cp1 is OK: All 7 backends are healthy [03:12:32] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [03:12:42] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 1 minute ago with 2 failures. Failed resources (up to 3 shown): Package[cadvisor],Package[prometheus-nutcracker-exporter] [03:12:48] RECOVERY - mail1 SMTP on mail1 is OK: SMTP OK - 3.074 sec. response time [03:12:49] RECOVERY - ldap1 LDAP on ldap1 is OK: LDAP OK - 0.039 seconds response time [03:12:56] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.275 second response time [03:13:04] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:13:13] RECOVERY - mail1 webmail.wikitide.net HTTPS on mail1 is OK: HTTP OK: HTTP/1.1 200 OK - 6205 bytes in 0.035 second response time [03:13:19] RECOVERY - mem1 memcached on mem1 is OK: TCP OK - 0.000 second response time on 10.0.0.106 port 11211 [03:13:34] RECOVERY - mail1 IMAP on mail1 is OK: IMAP OK - 0.002 second response time on 10.0.0.110 port 143 [* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS LOGINDISABLED] Dovecot (Debian) ready.] [03:13:51] RECOVERY - swiftproxy1 Puppet on swiftproxy1 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [03:14:20] RECOVERY - mail1 HTTPS on mail1 is OK: HTTP OK: HTTP/2 301 - 227 bytes in 0.015 second response time [03:15:21] RECOVERY - swiftobject1 Swift Object Service on swiftobject1 is OK: TCP OK - 0.000 second response time on 10.0.0.121 port 6000 [03:15:35] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/opendkim/keys/mail.private],Exec[git_checkout_roundcubemail] [03:18:14] !log [@jobrunner1] starting deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] [03:18:16] !log [@jobrunner1] finished deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] - SUCCESS in 1s [03:18:17] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:18:20] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:18:20] RECOVERY - jobchron1 Puppet on jobchron1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:20:29] RECOVERY - bots1 Puppet on bots1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:23:20] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:25:26] RECOVERY - db1 Puppet on db1 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:26:43] RECOVERY - bast1 Puppet on bast1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:27:24] RECOVERY - os1 Puppet on os1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:28:04] RECOVERY - cp1 Puppet on cp1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:31:02] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [03:31:03] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [03:31:05] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:31:08] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:31:09] RECOVERY - swiftac1 Puppet on swiftac1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:35:09] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [03:35:49] RECOVERY - graylog1 HTTPS on graylog1 is OK: HTTP OK: HTTP/2 200 - 646 bytes in 0.016 second response time [03:36:51] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:36:57] RECOVERY - services1 mathoid on services1 is OK: TCP OK - 0.000 second response time on 10.0.0.115 port 10044 [03:44:43] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [04:13:48] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [04:42:52] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [05:12:35] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [05:18:37] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [05:42:34] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [06:16:45] RECOVERY - www.rosettacode.org - reverse DNS on sslhost is OK: SSL OK - www.rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [06:31:08] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [07:02:51] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [07:32:29] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [08:04:12] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [08:56:09] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [09:26:08] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [11:02:10] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [11:06:52] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:07:05] PROBLEM - mem1 Puppet on mem1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:07:09] PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:13:06] PROBLEM - prometheus1 Puppet on prometheus1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:14:48] PROBLEM - phorge1 Puppet on phorge1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:15:31] PROBLEM - swiftobject1 Puppet on swiftobject1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:15:52] PROBLEM - ldap1 Puppet on ldap1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:20:33] PROBLEM - jobchron1 Puppet on jobchron1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:22:29] PROBLEM - bots1 Puppet on bots1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:23:37] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:26:33] PROBLEM - db1 Puppet on db1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:28:03] PROBLEM - cp1 Puppet on cp1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:29:24] PROBLEM - os1 Puppet on os1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:30:08] PROBLEM - bast1 Puppet on bast1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:31:50] PROBLEM - swiftproxy1 Puppet on swiftproxy1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:33:09] PROBLEM - swiftac1 Puppet on swiftac1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:14:43] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [12:36:12] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.406 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [13:35:26] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [13:42:34] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [14:07:12] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [14:12:30] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [14:36:47] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [15:39:09] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [16:01:00] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [16:08:46] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [16:18:07] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/a36b7f46bedb...f6b03f1bf3d2 [16:18:08] [02WikiTideOrg/ssl] 07WikiTideSSL 03f6b03f1 - Bot: Adding SSL cert for projects.dmvpetridish.com Certificate committed by reception [16:19:32] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/f6b03f1bf3d2...2b2b2467e8fc [16:19:34] [02WikiTideOrg/ssl] 07WikiTideSSL 032b2b246 - Bot: Adding SSL cert for wiki.chevrine.com Certificate committed by reception [16:23:59] PROBLEM - valorantjp.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address valorantjp.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:31:20] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=sagan4alphawikitide --no-updates --username-prefix=mediawikiwiki /home/reception/sagan4alphawikitide-20231201031834.xml (START) [16:31:21] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=sagan4alphawikitide --no-updates --username-prefix=mediawikiwiki /home/reception/sagan4alphawikitide-20231201031834.xml (END - exit=256) [16:31:22] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=sagan4alphawikitide (START) [16:31:23] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [16:31:26] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [16:31:29] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [16:44:41] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [16:49:29] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [17:01:00] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [17:11:11] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [17:13:20] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.rosettacode.org All nameservers failed to answer the query. [17:13:44] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [17:23:34] RECOVERY - valorantjp.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'valorantjp.wiki' will expire on Sat 20 Jan 2024 06:52:59 AM GMT +0000. [17:55:33] PROBLEM - valorantjp.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address valorantjp.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:08:59] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/2b354b027aa6...aaed2603b0ec [18:09:01] [02WikiTideOrg/puppet] 07Universal-Omega 03aaed260 - test_mwscript: use syscheck [18:10:21] WikiTideOrg/puppet - Universal-Omega the build passed. [18:11:28] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [18:17:40] RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [18:19:38] PROBLEM - puppet1 Puppet on puppet1 is UNKNOWN: UNKNOWN: Failed to check. Reason is: failed_to_parse_summary_file [18:25:24] RECOVERY - valorantjp.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'valorantjp.wiki' will expire on Sat 20 Jan 2024 06:52:59 AM GMT +0000. [18:35:09] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [19:05:07] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [19:44:43] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [19:49:55] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [20:09:08] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [20:19:17] RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [20:41:02] RECOVERY - www.rosettacode.org - reverse DNS on sslhost is OK: SSL OK - www.rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [20:42:54] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [21:20:45] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03db-remove-ssl [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/commit/f88b673543a9 [21:20:48] [02WikiTideOrg/mw-config] 07Universal-Omega 03f88b673 - Remove SSL from DB [21:20:49] [02mw-config] 07Universal-Omega created branch 03db-remove-ssl - 13https://github.com/WikiTideOrg/mw-config [21:20:51] [02mw-config] 07Universal-Omega opened pull request 03#19: Remove SSL from DB - 13https://github.com/WikiTideOrg/mw-config/pull/19 [21:21:18] [02WikiTideOrg/mw-config] 07github-actions[bot] pushed 031 commit to 03db-remove-ssl [+0/-0/±5] 13https://github.com/WikiTideOrg/mw-config/compare/f88b673543a9...a4772e7dc42d [21:21:19] [02WikiTideOrg/mw-config] 07github-actions 03a4772e7 - CI: lint code to MediaWiki standards Check commit and GitHub actions for more details [21:21:21] [02mw-config] 07github-actions[bot] synchronize pull request 03#19: Remove SSL from DB - 13https://github.com/WikiTideOrg/mw-config/pull/19 [21:21:26] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03db-remove-ssl [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/a4772e7dc42d...858c1cfb718b [21:21:27] [02WikiTideOrg/mw-config] 07Universal-Omega 03858c1cf - Remove SSL from DB [21:21:30] [02mw-config] 07Universal-Omega synchronize pull request 03#19: Remove SSL from DB - 13https://github.com/WikiTideOrg/mw-config/pull/19 [21:21:49] WikiTideOrg/mw-config - Universal-Omega the build passed. [21:22:18] WikiTideOrg/mw-config - Universal-Omega the build passed. [21:26:07] !log run grants on db1 to remove REQUIRE SSL [21:26:10] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [21:32:48] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=sagan4alphawikitide (END - exit=0) [21:32:51] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [21:35:04] !log puppet1: restart puppetdb [21:35:07] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [21:37:08] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [21:37:11] [02puppet] 07Universal-Omega closed pull request 03#25: mariadb: remove SSL - 13https://github.com/WikiTideOrg/puppet/pull/25 [21:37:13] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±8] 13https://github.com/WikiTideOrg/puppet/compare/aaed2603b0ec...a228dd3d9996 [21:37:15] [02WikiTideOrg/puppet] 07Universal-Omega 03a228dd3 - mariadb: remove SSL [21:37:16] [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03mariadb-remove-ssl [21:37:16] [02puppet] 07Universal-Omega deleted branch 03mariadb-remove-ssl - 13https://github.com/WikiTideOrg/puppet [21:37:18] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03revert-25-mariadb-remove-ssl [+0/-0/±8] 13https://github.com/WikiTideOrg/puppet/commit/b0c7534a1e75 [21:37:20] [02WikiTideOrg/puppet] 07Universal-Omega 03b0c7534 - Revert "mariadb: remove SSL" This reverts commit a228dd3d9996e84d7bd0f6d647e3c87adc247872. [21:37:22] [02puppet] 07Universal-Omega created branch 03revert-25-mariadb-remove-ssl - 13https://github.com/WikiTideOrg/puppet [21:37:24] [02puppet] 07Universal-Omega opened pull request 03#31: Revert "mariadb: remove SSL" - 13https://github.com/WikiTideOrg/puppet/pull/31 [21:41:04] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [21:48:52] [02mw-config] 07Universal-Omega closed pull request 03#19: Remove SSL from DB - 13https://github.com/WikiTideOrg/mw-config/pull/19 [21:48:54] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±6] 13https://github.com/WikiTideOrg/mw-config/compare/2e4c77d998ce...14fda65a8e36 [21:48:55] [02WikiTideOrg/mw-config] 07Universal-Omega 0314fda65 - Remove SSL from DB [21:48:58] [02WikiTideOrg/mw-config] 07Universal-Omega deleted branch 03db-remove-ssl [21:49:01] [02mw-config] 07Universal-Omega deleted branch 03db-remove-ssl - 13https://github.com/WikiTideOrg/mw-config [21:49:42] WikiTideOrg/mw-config - Universal-Omega the build passed. [21:49:44] !log [universalomega@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] [21:49:45] !log [universalomega@jobrunner1] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [21:49:47] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [21:49:49] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [21:52:30] !log [universalomega@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] [21:52:33] !log [universalomega@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [21:52:41] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 915 bytes in 0.080 second response time [21:53:08] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 915 bytes in 0.107 second response time [21:53:41] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 915 bytes in 0.074 second response time [21:54:05] RECOVERY - jobrunner1 Disk Space on jobrunner1 is OK: DISK OK - free space: / 22080MiB (43% inode=77%); [22:01:44] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 3.150 second response time [22:02:41] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.136 second response time [22:03:08] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.222 second response time [22:05:03] [02puppet] 07Universal-Omega closed pull request 03#31: Revert "mariadb: remove SSL" - 13https://github.com/WikiTideOrg/puppet/pull/31 [22:11:30] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/a228dd3d9996...465634b0a6f0 [22:11:33] [02WikiTideOrg/puppet] 07Universal-Omega 03465634b - monitoring: remove ssl from DB [22:34:31] [02WikiTideOrg/mw-config] 07Universal-Omega created branch 03wiki-suffix 13https://github.com/WikiTideOrg/mw-config/commit/14fda65a8e36c6c6c89ccca0c3d39d3d4aec092e [22:34:33] [02mw-config] 07Universal-Omega created branch 03wiki-suffix - 13https://github.com/WikiTideOrg/mw-config [22:35:02] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03wiki-suffix [+0/-0/±2] 13https://github.com/WikiTideOrg/mw-config/compare/14fda65a8e36...46c59e6b5027 [22:35:03] [02WikiTideOrg/mw-config] 07Universal-Omega 0346c59e6 - Use wiki suffix for all new wiki creations [22:35:13] [02mw-config] 07Universal-Omega opened pull request 03#20: Use wiki suffix for all new wiki creations - 13https://github.com/WikiTideOrg/mw-config/pull/20 [22:36:10] WikiTideOrg/mw-config - Universal-Omega the build passed. [22:36:32] [02mw-config] 07Universal-Omega closed pull request 03#20: Use wiki suffix for all new wiki creations - 13https://github.com/WikiTideOrg/mw-config/pull/20 [22:36:33] [02WikiTideOrg/mw-config] 07Universal-Omega deleted branch 03wiki-suffix [22:36:35] [02mw-config] 07Universal-Omega deleted branch 03wiki-suffix - 13https://github.com/WikiTideOrg/mw-config [22:49:58] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [23:18:56] [02WikiTideOrg/puppet] 07Universal-Omega created branch 03mcrouter 13https://github.com/WikiTideOrg/puppet/commit/465634b0a6f0c2d469f04b7757c2d3faa28f4a74 [23:18:57] [02puppet] 07Universal-Omega created branch 03mcrouter - 13https://github.com/WikiTideOrg/puppet [23:19:17] RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [23:20:17] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03mcrouter [+7/-1/±5] 13https://github.com/WikiTideOrg/puppet/compare/465634b0a6f0...94f941f62a6b [23:20:19] [02WikiTideOrg/puppet] 07Universal-Omega 0394f941f - mediawiki: use mcrouter instead of nutcracker [23:20:26] [02puppet] 07Universal-Omega opened pull request 03#32: mediawiki: use mcrouter instead of nutcracker - 13https://github.com/WikiTideOrg/puppet/pull/32 [23:21:01] [02WikiTideOrg/puppet] 07github-actions[bot] pushed 031 commit to 03mcrouter [+0/-0/±4] 13https://github.com/WikiTideOrg/puppet/compare/94f941f62a6b...6a56c8df5b8b [23:21:02] [02WikiTideOrg/puppet] 07github-actions 036a56c8d - CI: lint puppet code to standards Check commit and GitHub actions for more details [23:21:04] [02puppet] 07github-actions[bot] synchronize pull request 03#32: mediawiki: use mcrouter instead of nutcracker - 13https://github.com/WikiTideOrg/puppet/pull/32 [23:26:17] RECOVERY - bast1 Puppet on bast1 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [23:26:24] RECOVERY - db1 Puppet on db1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:27:24] RECOVERY - os1 Puppet on os1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:29:51] RECOVERY - swiftproxy1 Puppet on swiftproxy1 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:30:45] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03mcrouter [+1/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/6a56c8df5b8b...4ac0baec57c2 [23:30:48] [02WikiTideOrg/puppet] 07Universal-Omega 034ac0bae - Add package [23:30:51] [02puppet] 07Universal-Omega synchronize pull request 03#32: mediawiki: use mcrouter instead of nutcracker - 13https://github.com/WikiTideOrg/puppet/pull/32 [23:31:01] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [23:31:02] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [23:31:03] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:31:05] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:31:09] RECOVERY - swiftac1 Puppet on swiftac1 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [23:33:38] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:35:04] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [23:35:09] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [23:36:52] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:37:23] PROBLEM - mon1 Check correctness of the icinga configuration on mon1 is CRITICAL: Icinga configuration contains errors [23:41:06] RECOVERY - prometheus1 Puppet on prometheus1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:41:52] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:42:48] RECOVERY - phorge1 Puppet on phorge1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:19] RECOVERY - swiftobject1 Puppet on swiftobject1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [23:44:42] [02WikiTideOrg/puppet] 07Universal-Omega created branch 03no-ssl 13https://github.com/WikiTideOrg/puppet/commit/465634b0a6f0c2d469f04b7757c2d3faa28f4a74 [23:44:43] [02puppet] 07Universal-Omega created branch 03no-ssl - 13https://github.com/WikiTideOrg/puppet [23:45:00] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03no-ssl [+0/-0/±3] 13https://github.com/WikiTideOrg/puppet/compare/465634b0a6f0...47637b665358 [23:45:01] [02WikiTideOrg/puppet] 07Universal-Omega 0347637b6 - Remove SSL from mariadb monitoring [23:45:09] [02puppet] 07Universal-Omega opened pull request 03#33: Remove SSL from mariadb monitoring - 13https://github.com/WikiTideOrg/puppet/pull/33 [23:45:26] [02puppet] 07Universal-Omega closed pull request 03#33: Remove SSL from mariadb monitoring - 13https://github.com/WikiTideOrg/puppet/pull/33 [23:45:27] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/WikiTideOrg/puppet/compare/465634b0a6f0...e7e8fe41cfd8 [23:45:29] [02WikiTideOrg/puppet] 07Universal-Omega 03e7e8fe4 - Remove SSL from mariadb monitoring [23:45:31] [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03no-ssl [23:45:32] [02puppet] 07Universal-Omega deleted branch 03no-ssl - 13https://github.com/WikiTideOrg/puppet [23:46:38] RECOVERY - db1 MariaDB Connections on db1 is OK: Usage: php check_mysql_connections.php --host=HOST --user=USER --password=PASSWORD --max-connections=MAX_CONNECTIONS --critical-threshold=CRITICAL_THRESHOLD --warning-threshold=WARNING_THRESHOLD [23:48:28] RECOVERY - jobchron1 Puppet on jobchron1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:49:23] RECOVERY - mon1 Check correctness of the icinga configuration on mon1 is OK: Icinga configuration is correct [23:49:42] PROBLEM - wiki.chevrine.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.digitalocean.com.', 'ns3.digitalocean.com.', 'ns2.digitalocean.com.'], 'CNAME': 'mw-lb.wikitide.org.'} [23:49:59]