[00:09:47] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 50480MiB (11% inode=99%); [00:11:37] PROBLEM - cp38 Disk Space on cp38 is WARNING: DISK WARNING - free space: / 48566MiB (10% inode=99%); [00:13:37] RECOVERY - cp38 Disk Space on cp38 is OK: DISK OK - free space: / 54148MiB (11% inode=99%); [00:27:28] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/3b5db7aeb5dd01bd518ef120e685f295e18523e8 [00:27:29] 02puppet/03main 07CosmicAlpha 033b5db7a Add fsslc.wtnet versions of os and db nodes [00:41:33] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/601a6c96cdd116f8ee58701def408d6862b07730 [00:41:33] 02puppet/03main 07CosmicAlpha 03601a6c9 Remove wikitide.net versions of os nodes [00:45:03] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/ed1b3af4d832b3dda3dacc7fa0d5bd52e186259b [00:45:03] 02dns/03main 07CosmicAlpha 03ed1b3af wikitide.net: remove aliases for os servers [00:46:01] miraheze/dns - Universal-Omega the build passed. [00:46:27] PROBLEM - swiftobject181 PowerDNS Recursor on swiftobject181 is CRITICAL: Domain 'swiftobject181.wikitide.net' was not found by the server [00:46:31] RECOVERY - swiftobject181 PowerDNS Recursor on swiftobject181 is OK: DNS OK: 0.034 seconds response time. swiftobject181.wikitide.net returns 10.0.18.107 [00:58:07] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/da134273458198e49153f58829131a5e678373a4 [00:58:08] 02puppet/03main 07CosmicAlpha 03da13427 Remove wikitide.net versions of db nodes [01:05:34] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/60ac7539a6133e39f306cf6a78942d66b2b4a77f [01:05:34] 02dns/03main 07CosmicAlpha 0360ac753 wikitide.net: remove aliases for db servers [01:06:36] miraheze/dns - Universal-Omega the build passed. [03:35:42] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49860MiB (10% inode=99%); [03:51:46] PROBLEM - cp38 Disk Space on cp38 is WARNING: DISK WARNING - free space: / 49905MiB (10% inode=99%); [11:19:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749550760000&orgId=1&to=1749554393349 [11:24:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749551000000&orgId=1&to=1749554600000 [16:08:59] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': 'GoogleDocs4MW'} to all [16:09:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:09:27] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.43', '1.44'], 'upgrade_extensions': 'GoogleDocs4MW'} to test151 [16:09:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:10:47] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': 'GoogleDocs4MW'} to all - SUCCESS in 108s [16:10:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:19:36] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.43', '1.44'], 'upgrade_extensions': 'GoogleDocs4MW'} to test151 - SUCCESS in 608s [16:19:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:22:58] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/16f86a7802c6900ecfccd86f084bc0a57b401394 [16:22:58] 02puppet/03main 07CosmicAlpha 0316f86a7 Add fsslc.wtnet versions of rdb and mem nodes [16:33:10] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/c3cf1585f314e5edd0e1c6c90734afa992b3cc83 [16:33:10] 02puppet/03main 07CosmicAlpha 03c3cf158 Remove wikitide.net nodes for mem and rdb [16:41:16] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/00feaa318c9c2141af566c956bcacd03d9170fbf [16:41:16] 02dns/03main 07CosmicAlpha 0300feaa3 wikitide.net: remove aliases for mem and rdb servers [16:43:14] miraheze/dns - Universal-Omega the build passed. [16:44:04] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/5243c444d8b5283685c525876aeb6128a067ffff [16:44:04] 02puppet/03main 07CosmicAlpha 035243c44 Add fsslc.wtnet node for swiftac171 [16:51:06] [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/9aa643439e87f88dd3853e3161e28a1e38942aec [16:51:06] 02ssl/03main 07MacFan4000 039aa6434 add wikimoma.art [16:53:09] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/4bffad6b5c2c1e61ad49a069253d64d49fd18065 [16:53:09] 02puppet/03main 07CosmicAlpha 034bffad6 Remove wikitide.net node of swiftac171 [16:56:52] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/cce47b8dbbd94b74743895874fa97b97b6aac152 [16:56:52] 02dns/03main 07CosmicAlpha 03cce47b8 wikitide.net: remove alias for swiftac171 [16:58:40] miraheze/dns - Universal-Omega the build passed. [16:59:21] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/52be82d2e94c27d86230715cd14867fa846d1564 [16:59:21] 02puppet/03main 07CosmicAlpha 0352be82d Add fsslc.wtnet nodes for changeprop and eventgate [17:11:57] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/c72374c94e2b11ad254c7921be0c9e68ecb4502c [17:11:58] 02puppet/03main 07CosmicAlpha 03c72374c Remove wikitide.net nodes for changeprop and eventgate [17:18:18] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/fff3050b4b54de6605ff036e89c3001f003ebb10 [17:18:19] 02dns/03main 07CosmicAlpha 03fff3050 wikitide.net: remove aliases for changeprop and eventgate servers [17:20:29] miraheze/dns - Universal-Omega the build passed. [17:22:10] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/036cdf99b1108d21d434774ce8c9448e8f6b0950 [17:22:10] 02puppet/03main 07CosmicAlpha 03036cdf9 Add fsslc.wtnet node for swiftobject [17:27:22] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/b2a271499a1633fb2137f1d4a596b4747656dd7c [17:27:22] 02puppet/03main 07CosmicAlpha 03b2a2714 Update swiftobject151 to wtnet [17:33:30] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/56dd38f63b755703330ffb81e93b671ab812b2da [17:33:30] 02puppet/03main 07CosmicAlpha 0356dd38f Remove wikitide.net node for swiftobject [17:40:51] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/d738218c3e738a0d58674ebc40fbd5e9014222b2 [17:40:51] 02dns/03main 07CosmicAlpha 03d738218 wikitide.net: remove aliases for swiftobject servers [17:41:52] miraheze/dns - Universal-Omega the build passed. [17:46:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749573980000&orgId=1&to=1749577613348 [17:48:13] PROBLEM - ping on kafka181 is CRITICAL: CRITICAL - Host Unreachable (10.0.18.146) [17:48:26] PROBLEM - kafka181 APT on kafka181 is CRITICAL: connect to address 10.0.18.146 port 5666: No route to hostconnect to host 10.0.18.146 port 5666: No route to host [17:48:27] PROBLEM - kafka181 conntrack_table_size on kafka181 is CRITICAL: connect to address 10.0.18.146 port 5666: No route to hostconnect to host 10.0.18.146 port 5666: No route to host [17:48:40] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/5952b9d593bbd5044d08fefde8b23ac080acc183 [17:48:40] 02puppet/03main 07CosmicAlpha 035952b9d Update kafka181 IP [17:48:44] PROBLEM - Host kafka181 is DOWN: CRITICAL - Host Unreachable (10.0.18.146) [17:49:24] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/1bd8dcbecf46c58ae71c2937c458c1e64c717a34 [17:49:24] 02dns/03main 07CosmicAlpha 031bd8dcb Fix kafka181 IP [17:49:31] PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 6 backends are down. mw171 mw181 mw183 mw192 mw193 mw201 [17:49:44] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 7 backends are down. mw171 mw181 mw182 mw183 mw192 mw193 mw201 [17:49:52] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 7 backends are down. mw162 mw171 mw182 mw192 mw193 mw201 mw202 [17:51:08] miraheze/dns - Universal-Omega the build passed. [17:51:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749573980000&orgId=1&to=1749577913349[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749574160000&orgId=1&to=1749577913349 [17:54:42] RECOVERY - Host kafka181 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [17:55:57] RECOVERY - kafka181 APT on kafka181 is OK: APT OK: 64 packages available for upgrade (0 critical updates). [17:56:07] RECOVERY - ping on kafka181 is OK: PING OK - Packet loss = 0%, RTA = 0.20 ms [17:56:17] RECOVERY - kafka181 conntrack_table_size on kafka181 is OK: OK: nf_conntrack is 0 % full [17:57:33] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 31 backends are healthy [17:57:45] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 31 backends are healthy [17:57:52] RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy [17:59:09] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/9a1d2540e219adec9c22e4d5a9851a0743b91c2f [17:59:09] 02puppet/03main 07CosmicAlpha 039a1d254 Add fsslc.wtnet node for kafka181 [18:01:53] [Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749573980000&orgId=1&to=1749578390000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749574160000&orgId=1&to=1749578390000 [18:06:51] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/977415cf99024f153f8f32fba57eeef2947da922 [18:06:51] 02puppet/03main 07CosmicAlpha 03977415c Remove wikitide.net node of kafka181 [18:11:19] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/3aec062a3954764d29d198a01f10c55c8e45d368 [18:11:19] 02dns/03main 07CosmicAlpha 033aec062 wikitide.net: remove alias for kafka181 and reorganize [18:12:15] miraheze/dns - Universal-Omega the build passed. [20:44:29] [02dns] 07Universal-Omega created 03cp (+1 new commit) 13https://github.com/miraheze/dns/commit/24470bdf591e [20:44:29] 02dns/03cp 07CosmicAlpha 0324470bd Add new cp hostnames [20:44:34] [02dns] 07Universal-Omega opened pull request #689: Add new cp hostnames (03main...03cp) 13https://github.com/miraheze/dns/pull/689 [20:45:35] [02dns] 07Universal-Omega pushed 1 new commit to 03cp 13https://github.com/miraheze/dns/commit/850952e8dea670359ccac57a8f7860c1999a60ee [20:45:35] 02dns/03cp 07CosmicAlpha 03850952e Add [20:45:40] miraheze/dns - Universal-Omega the build passed. [20:46:35] [02dns] 07Universal-Omega pushed 1 new commit to 03cp 13https://github.com/miraheze/dns/commit/a8f451707e6f6380a4a02eae5b8e63b663c3fb63 [20:46:36] 02dns/03cp 07CosmicAlpha 03a8f4517 Update [20:47:30] miraheze/dns - Universal-Omega the build passed. [20:47:47] miraheze/dns - Universal-Omega the build passed. [20:49:04] [02dns] 07Universal-Omega merged pull request #689: Add new cp hostnames (03main...03cp) 13https://github.com/miraheze/dns/pull/689 [20:49:04] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/1874ef9de211b305cafd93540936f28c2382dace [20:49:04] 02dns/03main 07CosmicAlpha 031874ef9 Add new cp hostnames [20:49:05] [02dns] 07Universal-Omega 04deleted 03cp at 03a8f4517 13https://api.github.com/repos/miraheze/dns/commit/a8f4517 [20:51:38] miraheze/dns - Universal-Omega the build passed. [20:52:26] [02puppet] 07Universal-Omega created 03cp (+1 new commit) 13https://github.com/miraheze/puppet/commit/348d764a8c75 [20:52:26] 02puppet/03cp 07CosmicAlpha 03348d764 Add nodes for new cp hostnames [20:52:29] [02puppet] 07Universal-Omega opened pull request #4385: Add nodes for new cp hostnames (03main...03cp) 13https://github.com/miraheze/puppet/pull/4385 [20:52:36] [02puppet] 07coderabbitai[bot] commented on pull request #4385: --- […] 13https://github.com/miraheze/puppet/pull/4385#issuecomment-2960553937 [20:53:12] [02puppet] 07Universal-Omega pushed 1 new commit to 03cp 13https://github.com/miraheze/puppet/commit/7e30f1e7710e47a6b14d7236363b57312d8b87a9 [20:53:12] 02puppet/03cp 07CosmicAlpha 037e30f1e Add [20:55:52] [02puppet] 07Universal-Omega merged pull request #4385: Add nodes for new cp hostnames (03main...03cp) 13https://github.com/miraheze/puppet/pull/4385 [20:55:52] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/46d95727f669f6ea499f12b0ddb106e8661bc5ad [20:55:53] 02puppet/03main 07CosmicAlpha 0346d9572 Add nodes for new cp hostnames [20:55:55] [02puppet] 07Universal-Omega 04deleted 03cp at 037e30f1e 13https://api.github.com/repos/miraheze/puppet/commit/7e30f1e [21:07:43] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/0408863376df3dc633f28141050452f1c69ced1d [21:07:43] 02puppet/03main 07CosmicAlpha 030408863 Remove cp36 hiera… [21:08:40] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/faf3ff99dcd30db0adfef69fc29635835c2d7c23 [21:08:40] 02puppet/03main 07CosmicAlpha 03faf3ff9 Remove cp36 node… [21:19:36] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 28467MiB (6% inode=99%); [21:26:04] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/3d8ac7b9921d2fffa1ba169ca786ac9d530227c6 [21:26:04] 02puppet/03main 07CosmicAlpha 033d8ac7b Remove old cp nodes… [21:28:33] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 28471MiB (6% inode=99%); [21:35:18] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/cd42d0d84536abd645075a24b176559f05fd99ae [21:35:18] 02puppet/03main 07CosmicAlpha 03cd42d0d Update node names [21:38:59] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/e1bdbf720fc18b0cd326c5c194d56ff69bcec4f6 [21:38:59] 02mw-config/03main 07CosmicAlpha 03e1bdbf7 Update cp names in comment [21:39:51] miraheze/mw-config - Universal-Omega the build passed. [21:39:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749587960000&orgId=1&to=1749591593345 [21:40:55] [02dns] 07Universal-Omega created 03cp (+1 new commit) 13https://github.com/miraheze/dns/commit/1bc46fbd8397 [21:40:55] 02dns/03cp 07CosmicAlpha 031bc46fb Remove old cp hostnames [21:40:59] [02dns] 07Universal-Omega opened pull request #690: Remove old cp hostnames (03main...03cp) 13https://github.com/miraheze/dns/pull/690 [21:41:31] [02dns] 07Universal-Omega pushed 1 new commit to 03cp 13https://github.com/miraheze/dns/commit/d3dd03bd9051961bd5b8c57061d362a81bc08c0d [21:41:31] 02dns/03cp 07CosmicAlpha 03d3dd03b Remove [21:42:48] miraheze/dns - Universal-Omega the build passed. [21:43:01] miraheze/dns - Universal-Omega the build passed. [21:44:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749588260000&orgId=1&to=1749591860000 [21:52:15] [02dns] 07Universal-Omega merged pull request #690: Remove old cp hostnames (03main...03cp) 13https://github.com/miraheze/dns/pull/690 [21:52:17] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/911760b2ad86a856f23e3aa757fe673f66c0400f [21:52:18] [02dns] 07Universal-Omega 04deleted 03cp at 03d3dd03b 13https://api.github.com/repos/miraheze/dns/commit/d3dd03b [21:52:19] 02dns/03main 07CosmicAlpha 03911760b Remove old cp hostnames [21:54:08] miraheze/dns - Universal-Omega the build passed. [21:57:18] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/240fed356caa16069de54e3e3e73f5f820961d5d [21:57:18] 02dns/03main 07CosmicAlpha 03240fed3 Update PTR records for cp servers [21:59:25] miraheze/dns - Universal-Omega the build passed. [22:07:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749589640000&orgId=1&to=1749593273344 [22:22:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749590480000&orgId=1&to=1749594080000 [22:25:31] PROBLEM - cp171 Disk Space on cp171 is CRITICAL: DISK CRITICAL - free space: / 27217MiB (5% inode=99%); [22:33:25] PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 27224MiB (5% inode=99%); [22:41:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749591680000&orgId=1&to=1749595313344 [22:51:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749591680000&orgId=1&to=1749595880000 [23:38:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749595100000&orgId=1&to=1749598733345 [23:47:24] PROBLEM - swiftobject171 Puppet on swiftobject171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:48:09] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:48:49] PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:48:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749595460000&orgId=1&to=1749599060000 [23:49:32] PROBLEM - mw183 Puppet on mw183 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:50:03] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:51:02] PROBLEM - os191 Puppet on os191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:51:02] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:51:03] PROBLEM - mem191 Puppet on mem191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:51:15] PROBLEM - swiftobject201 Puppet on swiftobject201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:51:40] PROBLEM - mw193 Puppet on mw193 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:51:45] PROBLEM - ldap171 Puppet on ldap171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:52:58] PROBLEM - mw191 Puppet on mw191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:53:06] PROBLEM - mw163 Puppet on mw163 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:53:14] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:53:15] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:53:45] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [23:53:59] PROBLEM - mattermost1 Puppet on mattermost1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:54:05] PROBLEM - swiftproxy171 Puppet on swiftproxy171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:55:08] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:55:08] PROBLEM - kafka181 Puppet on kafka181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:55:10] PROBLEM - cloud20 Puppet on cloud20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:55:32] PROBLEM - mw201 Puppet on mw201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:55:45] RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:55:54] PROBLEM - os151 Puppet on os151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:55:59] RECOVERY - mattermost1 Puppet on mattermost1 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:57:08] RECOVERY - kafka181 Puppet on kafka181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:57:10] RECOVERY - cloud20 Puppet on cloud20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:58:58] RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:59:02] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:59:04] RECOVERY - mw163 Puppet on mw163 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [23:59:05] RECOVERY - mem191 Puppet on mem191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:59:08] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:59:33] RECOVERY - mw183 Puppet on mw183 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:59:35] RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:59:42] RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:59:56] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures