[00:02:05] RECOVERY - cp1 Disk Space on cp1 is OK: DISK OK - free space: / 6374MiB (11% inode=98%); [02:33:58] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [02:51:18] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+3/-1/±2] 13https://github.com/WikiTideOrg/puppet/compare/399b254596c4...dc9b83dcd116 [02:51:20] [02WikiTideOrg/puppet] 07Universal-Omega 03dc9b83d - Add mathoid to services [03:37:34] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [03:56:37] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [04:37:02] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [04:56:07] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [05:08:53] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [06:24:47] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [06:32:16] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [06:53:30] RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [08:00:14] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [08:29:34] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [08:59:17] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [09:26:45] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki All nameservers failed to answer the query. [09:56:33] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [10:26:16] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [11:05:56] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [11:54:47] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [12:04:18] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [12:33:59] RECOVERY - www.rosettacode.org - reverse DNS on sslhost is OK: SSL OK - www.rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [12:52:40] RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [13:38:05] PROBLEM - cp1 Disk Space on cp1 is WARNING: DISK WARNING - free space: / 6074MiB (10% inode=98%); [14:26:21] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [14:43:11] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [15:25:52] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [17:31:10] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [18:18:10] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03mathoid-service [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/commit/f04ae34ecd7d [18:18:13] [02WikiTideOrg/mw-config] 07Universal-Omega 03f04ae34 - Use services1 for mathoid [18:18:15] [02mw-config] 07Universal-Omega created branch 03mathoid-service - 13https://github.com/WikiTideOrg/mw-config [18:18:17] [02mw-config] 07Universal-Omega opened pull request 03#14: Use services1 for mathoid - 13https://github.com/WikiTideOrg/mw-config/pull/14 [18:18:52] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03mathoid-service [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/f04ae34ecd7d...a2b47a381426 [18:18:54] [02WikiTideOrg/mw-config] 07Universal-Omega 03a2b47a3 - Add config [18:18:56] [02mw-config] 07Universal-Omega synchronize pull request 03#14: Use services1 for mathoid - 13https://github.com/WikiTideOrg/mw-config/pull/14 [18:19:08] WikiTideOrg/mw-config - Universal-Omega the build passed. [18:19:50] WikiTideOrg/mw-config - Universal-Omega the build passed. [18:32:54] [02mw-config] 07Universal-Omega synchronize pull request 03#14: Use services1 for mathoid - 13https://github.com/WikiTideOrg/mw-config/pull/14 [18:32:57] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03mathoid-service [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/a2b47a381426...17925292cec4 [18:32:58] [02WikiTideOrg/mw-config] 07Universal-Omega 031792529 - Update IP [18:33:50] WikiTideOrg/mw-config - Universal-Omega the build passed. [19:30:14] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [19:32:59] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [20:02:16] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [20:19:30] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+4/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/dc9b83dcd116...b16cf9af97d5 [20:19:32] [02WikiTideOrg/puppet] 07Universal-Omega 03b16cf9a - Add poolcounter module [20:51:16] [02Reports] 07Universal-Omega created tag 03v12 - 13https://github.com/WikiTideOrg/Reports [20:51:18] [02WikiTideOrg/Reports] 07Universal-Omega tagged 0374760f4 as 03v12 13https://github.com/WikiTideOrg/Reports/commit/74760f404aba8d692f5e2162c94f9a0ba410fbed [21:04:50] [02WikiTideOrg/puppet] 07Universal-Omega created branch 03reports 13https://github.com/WikiTideOrg/puppet/commit/b16cf9af97d50d03fdbaf05762323b13e8768c4e [21:04:52] [02puppet] 07Universal-Omega created branch 03reports - 13https://github.com/WikiTideOrg/puppet [21:05:23] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03reports [+5/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/b16cf9af97d5...a5849561f45f [21:05:26] [02WikiTideOrg/puppet] 07Universal-Omega 03a584956 - Add reports module [21:05:35] [02puppet] 07Universal-Omega opened pull request 03#18: Add reports module - 13https://github.com/WikiTideOrg/puppet/pull/18 [21:17:02] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/b16cf9af97d5...c397af72e7d5 [21:17:05] [02WikiTideOrg/puppet] 07Universal-Omega 03c397af7 - Remove RESTBase for now [21:18:32] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [21:22:16] PROBLEM - services1 Current Load on services1 is CRITICAL: LOAD CRITICAL - total load average: 13.33, 5.10, 2.19 [21:30:31] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/c397af72e7d5...d359e7300a97 [21:30:33] [02WikiTideOrg/puppet] 07Universal-Omega 03d359e73 - mathoid: use the same workers as there are CPU units [21:47:30] PROBLEM - services1 Current Load on services1 is WARNING: LOAD WARNING - total load average: 0.26, 0.60, 1.95 [21:51:30] RECOVERY - services1 Current Load on services1 is OK: LOAD OK - total load average: 0.75, 0.62, 1.64 [22:00:57] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [22:11:31] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±28] 13https://github.com/WikiTideOrg/puppet/compare/d359e7300a97...e38251194e7b [22:11:34] [02WikiTideOrg/puppet] 07Universal-Omega 03e382511 - Replace legacy facts in firewall rules [22:16:55] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e38251194e7b...73dfcdf72dba [22:16:56] [02WikiTideOrg/puppet] 07Universal-Omega 0373dfcdf - base::firewall: add monitoring and rules [22:18:41] [02WikiTideOrg/puppet] 07Universal-Omega pushed 035 commits to 03reports [+0/-0/±62] 13https://github.com/WikiTideOrg/puppet/compare/a5849561f45f...3370f8c4305a [22:18:42] [02WikiTideOrg/puppet] 07Universal-Omega 033370f8c - Merge branch 'master' into reports [22:18:44] [02puppet] 07Universal-Omega synchronize pull request 03#18: Add reports module - 13https://github.com/WikiTideOrg/puppet/pull/18 [22:19:34] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03reports [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/3370f8c4305a...5e4003e860e8 [22:19:35] [02WikiTideOrg/puppet] 07Universal-Omega 035e4003e - Add to firewall [22:19:37] [02puppet] 07Universal-Omega synchronize pull request 03#18: Add reports module - 13https://github.com/WikiTideOrg/puppet/pull/18 [22:30:40] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [22:54:05] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03ferm [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/commit/877da9bf0c8b [22:54:06] [02WikiTideOrg/puppet] 07Universal-Omega 03877da9b - Enable ferm [22:54:08] [02puppet] 07Universal-Omega created branch 03ferm - 13https://github.com/WikiTideOrg/puppet [22:54:09] [02puppet] 07Universal-Omega opened pull request 03#19: Enable ferm - 13https://github.com/WikiTideOrg/puppet/pull/19 [22:55:15] [02mw-config] 07Universal-Omega closed pull request 03#14: Use services1 for mathoid - 13https://github.com/WikiTideOrg/mw-config/pull/14 [22:55:18] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±2] 13https://github.com/WikiTideOrg/mw-config/compare/5e356d40224c...39388082eb6d [22:55:19] [02WikiTideOrg/mw-config] 07Universal-Omega 033938808 - Use services1 for mathoid [22:55:20] [02WikiTideOrg/mw-config] 07Universal-Omega deleted branch 03mathoid-service [22:55:23] [02mw-config] 07Universal-Omega deleted branch 03mathoid-service - 13https://github.com/WikiTideOrg/mw-config [22:56:11] WikiTideOrg/mw-config - Universal-Omega the build passed. [23:00:59] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [23:01:00] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [23:01:02] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:01:05] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:02:21] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [23:02:42] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03poolcounter [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/commit/74f380b62e7e [23:02:45] [02WikiTideOrg/mw-config] 07Universal-Omega 0374f380b - Add PoolCounter config [23:02:48] [02mw-config] 07Universal-Omega created branch 03poolcounter - 13https://github.com/WikiTideOrg/mw-config [23:02:50] [02mw-config] 07Universal-Omega opened pull request 03#15: Add PoolCounter config - 13https://github.com/WikiTideOrg/mw-config/pull/15 [23:03:49] WikiTideOrg/mw-config - Universal-Omega the build passed. [23:07:17] [02mw-config] 07Universal-Omega closed pull request 03#15: Add PoolCounter config - 13https://github.com/WikiTideOrg/mw-config/pull/15 [23:07:18] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/39388082eb6d...ca908b7e2bc5 [23:07:19] [02WikiTideOrg/mw-config] 07Universal-Omega 03ca908b7 - Add PoolCounter config [23:07:22] [02WikiTideOrg/mw-config] 07Universal-Omega deleted branch 03poolcounter [23:07:25] [02mw-config] 07Universal-Omega deleted branch 03poolcounter - 13https://github.com/WikiTideOrg/mw-config [23:07:26] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03revert-15-poolcounter [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/commit/433bdc443379 [23:07:27] [02WikiTideOrg/mw-config] 07Universal-Omega 03433bdc4 - Revert "Add PoolCounter config" This reverts commit ca908b7e2bc5c510dc483264ded6aec969ca7bdb. [23:07:28] [02mw-config] 07Universal-Omega created branch 03revert-15-poolcounter - 13https://github.com/WikiTideOrg/mw-config [23:07:29] [02mw-config] 07Universal-Omega opened pull request 03#16: Revert "Add PoolCounter config" - 13https://github.com/WikiTideOrg/mw-config/pull/16 [23:08:00] !log [universalomega@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] [23:08:02] !log [universalomega@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [23:08:03] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:08:06] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:08:07] WikiTideOrg/mw-config - Universal-Omega the build passed. [23:08:31] WikiTideOrg/mw-config - Universal-Omega the build passed. [23:08:40] [02mw-config] 07Universal-Omega closed pull request 03#16: Revert "Add PoolCounter config" - 13https://github.com/WikiTideOrg/mw-config/pull/16 [23:08:43] [02WikiTideOrg/mw-config] 07Universal-Omega deleted branch 03revert-15-poolcounter [23:08:46] [02mw-config] 07Universal-Omega deleted branch 03revert-15-poolcounter - 13https://github.com/WikiTideOrg/mw-config [23:11:44] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [23:31:37] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [23:31:38] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [23:31:39] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [23:31:41] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:31:44] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:43:51] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.