[00:00:00] !log [www-data@mwtask181] Began automatic backing up [00:00:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:01:09] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/ef9413525e6576130de212943e7a73082a035d5e [00:01:10] 02statichelp/03main 07WikiTideBot 03ef94135 Bot: Auto-update Tech namespace pages 2026-02-01 00:01:07 [00:02:55] PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 176 bytes in 0.009 second response time [00:10:54] [02mw-config] 07Universal-Omega created 03random-bast from 03main (+0 new commit) 13https://github.com/miraheze/mw-config/compare/random-bast [00:10:56] RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.516 second response time [00:14:23] [02mw-config] 07Universal-Omega pushed 1 new commit to 03random-bast 13https://github.com/miraheze/mw-config/commit/79bcf4e845bf88b95581c2aa091358c2cb0b8f4e [00:14:23] 02mw-config/03random-bast 07CosmicAlpha 0379bcf4e Randomly shift between bastion servers [00:15:24] [02mw-config] 07Universal-Omega opened pull request #6264: Randomly shift between bastion servers (03main...03random-bast) 13https://github.com/miraheze/mw-config/pull/6264 [00:16:18] miraheze/mw-config - Universal-Omega the build passed. [00:21:16] [02mw-config] 07Universal-Omega merged pull request #6264: Randomly shift between bastion servers (03main...03random-bast) 13https://github.com/miraheze/mw-config/pull/6264 [00:21:18] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/d8409d65d5d74e71105e650600d2ab81736c004d [00:21:20] [02mw-config] 07Universal-Omega 04deleted 03random-bast at 0379bcf4e 13https://api.github.com/repos/miraheze/mw-config/commit/79bcf4e [00:21:22] 02mw-config/03main 07CosmicAlpha 03d8409d6 Randomly shift between bastion servers [00:22:06] miraheze/mw-config - Universal-Omega the build passed. [00:22:32] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [00:22:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:02] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 30s [00:23:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:50] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 71959MiB (15.9% inode=100%); [00:33:49] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.54, 19.23, 13.07 [00:35:49] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 8.98, 15.22, 12.35 [00:43:30] RECOVERY - mwtask181 Check unit status of mediawiki_job_generate-sitemap-index on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_generate-sitemap-index [00:52:12] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/65ea97ec22d5 [00:52:12] 02puppet/03paladox-patch-1 07paladox 0365ea97e mariadb: Set innodb_snapshot_isolation to off… [00:52:16] [02puppet] 07paladox opened pull request #4733: mariadb: Set innodb_snapshot_isolation to off (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4733 [00:52:22] [02puppet] 07coderabbitai[bot] commented on pull request #4733: - [ ] 🔍 Trigger a full review […] 13https://github.com/miraheze/puppet/pull/4733#issuecomment-3829913730 [00:53:01] [02puppet] 07paladox merged pull request #4733: mariadb: Set innodb_snapshot_isolation to off (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4733 [00:53:01] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/7703d5077c9430a78303f3c7491f06add0f0eb4d [00:53:02] 02puppet/03main 07paladox 037703d50 mariadb: Set innodb_snapshot_isolation to off (#4733)… [00:53:03] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 0365ea97e 13https://api.github.com/repos/miraheze/puppet/commit/65ea97e [01:02:12] !log db: set global innodb_snapshot_isolation = OFF; [01:02:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:05:24] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 71582MiB (15.8% inode=100%); [01:06:32] [02mw-config] 07Universal-Omega closed pull request #6225: Randomly shift between bast161 and bast181 (03main...03paladox-patch-4) 13https://github.com/miraheze/mw-config/pull/6225 [01:06:32] [02mw-config] 07Universal-Omega commented on pull request #6225: Done with #6264 instead 13https://github.com/miraheze/mw-config/pull/6225#issuecomment-3829957721 [01:06:35] [02mw-config] 07Universal-Omega 04deleted 03paladox-patch-4 at 03c593a87 13https://api.github.com/repos/miraheze/mw-config/commit/c593a87 [02:01:35] [02TSPortal] 07renovate[bot] created 03renovate/lock-file-maintenance (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/88ac751afad4 [02:01:36] 02TSPortal/03renovate/lock-file-maintenance 07renovate[bot] 0388ac751 Lock file maintenance [11:27:08] !log ran the shell.php commands for https://meta.miraheze.org/wiki/Tech:Fixing_slot_roles_and_content_models on payloadrobloxwiki [11:27:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:08:15] [02TSPortal] 07renovate[bot] pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/88ac751afad4a12f5260dd5462e8fa4eca44c5ec [15:08:15] 02TSPortal/03main 07renovate[bot] 0388ac751 Lock file maintenance [15:08:16] [02TSPortal] 07renovate[bot] 04deleted 03renovate/lock-file-maintenance at 0388ac751 13https://api.github.com/repos/miraheze/TSPortal/commit/88ac751 [15:38:18] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/b372081b479b [15:38:19] 02puppet/03paladox-patch-1 07paladox 03b372081 mediawiki: increase max_execution_time to 210… [15:38:23] [02puppet] 07paladox opened pull request #4734: mediawiki: increase max_execution_time to 210 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4734 [15:38:30] [02puppet] 07coderabbitai[bot] commented on pull request #4734: - [ ] 🔍 Trigger a full review […] 13https://github.com/miraheze/puppet/pull/4734#issuecomment-3831223699 [15:38:43] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/c0e29d4560311fc10cd95085b19fd26aeb2ba306 [15:38:44] 02puppet/03paladox-patch-1 07paladox 03c0e29d4 Update mediawiki_beta.yaml [15:40:45] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/ef820441ddbc13859111eccae6d78ebfe70924df [15:40:45] 02puppet/03paladox-patch-1 07paladox 03ef82044 Update mediawiki_task.yaml [15:41:09] [02puppet] 07paladox merged pull request #4734: mediawiki: increase max_execution_time to 210 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4734 [15:41:09] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/877828739186a26f3715fb2531e3b26230cd9a60 [15:41:10] 02puppet/03main 07paladox 038778287 mediawiki: increase max_execution_time to 210 (#4734)… [15:41:12] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03ef82044 13https://api.github.com/repos/miraheze/puppet/commit/ef82044 [15:49:50] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 45270MiB (10.0% inode=100%); [15:55:16] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/c57928bab62832ba176c7c50c6cf486ac87f29a7 [15:55:16] 02ssl/03main 07WikiTideBot 03c57928b Bot: Auto-update domain lists [16:05:24] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45253MiB (10.0% inode=100%); [16:15:48] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/88e84ece83d7 [16:15:48] 02puppet/03paladox-patch-1 07paladox 0388e84ec nginx: Allow to add your own configs to nginx.conf [16:15:52] [02puppet] 07paladox opened pull request #4735: nginx: Allow to add your own configs to nginx.conf (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4735 [16:16:01] [02puppet] 07coderabbitai[bot] commented on pull request #4735: - [ ] 🔍 Trigger a full review […] 13https://github.com/miraheze/puppet/pull/4735#issuecomment-3831318121 [16:17:22] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/dca58b6f1bc3aefeee51865eae9195104b12b306 [16:17:22] 02puppet/03paladox-patch-1 07paladox 03dca58b6 Update nginx.conf.erb [16:17:54] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/cd8efb01dbcaebf07a7f34f1af0303096a442757 [16:17:54] 02puppet/03paladox-patch-1 07paladox 03cd8efb0 Update init.pp [16:18:39] [02puppet] 07paladox merged pull request #4735: nginx: Allow to add your own configs to nginx.conf (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4735 [16:18:40] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e4c3af1c93f271c0d32f95830d11c60fbf70cb53 [16:18:41] 02puppet/03main 07paladox 03e4c3af1 nginx: Allow to add your own configs to nginx.conf (#4735) [16:18:43] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03cd8efb0 13https://api.github.com/repos/miraheze/puppet/commit/cd8efb0 [16:21:11] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:22:02] PROBLEM - mw191 Puppet on mw191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:22:27] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/4a1b63c44c2d [16:22:27] 02puppet/03paladox-patch-1 07paladox 034a1b63c mediawiki: set nginx timeout configs [16:22:31] [02puppet] 07paladox opened pull request #4736: mediawiki: set nginx timeout configs (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4736 [16:22:37] [02puppet] 07coderabbitai[bot] commented on pull request #4736: - [ ] 🔍 Trigger a full review […] 13https://github.com/miraheze/puppet/pull/4736#issuecomment-3831327487 [16:22:41] @paladox please test changes on a single server and stop puppet if you're not sure about things [16:23:20] PROBLEM - mw153 Puppet on mw153 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:24:06] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/2a9b36c4d50c8edafda08e37bf2324c4e26b27ed [16:24:06] 02puppet/03main 07paladox 032a9b36c Fix syntax error [16:24:15] ok [16:24:28] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/6908dec63aa7a43e7998d1b055a9d6d748bc75aa [16:24:28] 02puppet/03paladox-patch-1 07paladox 036908dec Merge branch 'main' into paladox-patch-1 [16:25:09] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/9b880edf953dfedb179c7bede50fc1b3dd967a61 [16:25:09] 02puppet/03paladox-patch-1 07paladox 039b880ed Update mediawiki.yaml [16:26:37] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/0c38905e65b680de63cd0304d8d8152898a3f109 [16:26:37] 02puppet/03paladox-patch-1 07paladox 030c38905 Update mediawiki.conf.erb [16:27:13] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/49c3856a544290ccbf80752fbab5ede41175267e [16:27:13] 02puppet/03paladox-patch-1 07paladox 0349c3856 Update mediawiki-includes.conf.erb [16:27:50] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/7d32d40918ed1febe5ab38e3eb67cdcbe24cf7ef [16:27:50] 02puppet/03paladox-patch-1 07paladox 037d32d40 Update mediawiki_beta.yaml [16:28:06] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/af8d244d42b2e1959d703cacc49df74c3ef7b379 [16:28:06] 02puppet/03paladox-patch-1 07paladox 03af8d244 Update mediawiki_task.yaml [16:33:12] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/3d6448bc47e8cb9dcc050d997414c958aa536126 [16:33:12] 02puppet/03paladox-patch-1 07paladox 033d6448b Update mediawiki_task.yaml [16:33:50] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/b986c89e0c4b82481a92724f9e89028fb3f5995d [16:33:50] 02puppet/03paladox-patch-1 07paladox 03b986c89 Update mediawiki-includes.conf.erb [16:51:11] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:52:02] RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [16:53:01] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'FlexDiagrams'} to all [16:53:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:53:14] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'FlexDiagrams'} to test151 [16:53:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:53:20] RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:01:04] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'FlexDiagrams'} to all - SUCCESS in 483s [17:01:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:02:46] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'FlexDiagrams'} to test151 - SUCCESS in 572s [17:02:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:39:21] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'EmbedVideo'} to all [17:39:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:39:35] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'EmbedVideo'} to test151 [17:39:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:41:05] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'EmbedVideo'} to all - SUCCESS in 104s [17:41:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:44:56] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'EmbedVideo'} to test151 - SUCCESS in 321s [17:44:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:50:21] [02MirahezeMagic] 07paladox created 03paladox-patch-3 (+1 new commit) 13https://github.com/miraheze/MirahezeMagic/commit/bc1b75678507 [17:50:22] 02MirahezeMagic/03paladox-patch-3 07paladox 03bc1b756 SwiftDump: Fix "tar: Removing leading `/' from member names"… [17:50:26] [02MirahezeMagic] 07paladox opened pull request #627: SwiftDump: Fix "tar: Removing leading `/' from member names" (03main...03paladox-patch-3) 13https://github.com/miraheze/MirahezeMagic/pull/627 [17:50:35] [02MirahezeMagic] 07coderabbitai[bot] commented on pull request #627: - [ ] 🔍 Trigger a full review […] 13https://github.com/miraheze/MirahezeMagic/pull/627#issuecomment-3831593606 [17:56:36] miraheze/MirahezeMagic - paladox the build has errored. [17:57:29] [02ImportDump] 07paladox commented on pull request #164: We should maintain compatibility with MW 1.44 rather than waiting for us to move to MW 1.45. 13https://github.com/miraheze/ImportDump/pull/164#issuecomment-3831613011 [17:58:18] [02MirahezeMagic] 07paladox merged pull request #627: SwiftDump: Fix "tar: Removing leading `/' from member names" (03main...03paladox-patch-3) 13https://github.com/miraheze/MirahezeMagic/pull/627 [17:58:20] [02MirahezeMagic] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/628175deac3886d3f2b31656260dbd59f1267d51 [17:58:22] [02MirahezeMagic] 07paladox 04deleted 03paladox-patch-3 at 03bc1b756 13https://api.github.com/repos/miraheze/MirahezeMagic/commit/bc1b756 [17:58:24] 02MirahezeMagic/03main 07paladox 03628175d SwiftDump: Fix "tar: Removing leading `/' from member names" (#627)… [17:58:39] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'MirahezeMagic'} to test151 [17:58:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:58:51] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'MirahezeMagic'} to all [17:58:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:01:14] [02MirahezeMagic] 07paladox created 03paladox-patch-3 (+1 new commit) 13https://github.com/miraheze/MirahezeMagic/commit/c96d4d8848e7 [18:01:14] 02MirahezeMagic/03paladox-patch-3 07paladox 03c96d4d8 Fix… [18:01:17] [02MirahezeMagic] 07paladox opened pull request #628: Fix (03main...03paladox-patch-3) 13https://github.com/miraheze/MirahezeMagic/pull/628 [18:01:23] [02MirahezeMagic] 07coderabbitai[bot] commented on pull request #628: - [ ] 🔍 Trigger a full review […] 13https://github.com/miraheze/MirahezeMagic/pull/628#issuecomment-3831621587 [18:01:31] [02MirahezeMagic] 07paladox merged pull request #628: Fix (03main...03paladox-patch-3) 13https://github.com/miraheze/MirahezeMagic/pull/628 [18:01:31] [02MirahezeMagic] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/50eb8618d6beab9a544af3c30ae6ff61303ede31 [18:01:31] 02MirahezeMagic/03main 07paladox 0350eb861 Fix (#628)… [18:01:32] [02MirahezeMagic] 07paladox 04deleted 03paladox-patch-3 at 03c96d4d8 13https://api.github.com/repos/miraheze/MirahezeMagic/commit/c96d4d8 [18:02:18] [02ImportDump] 07Universal-Omega commented on pull request #164: @paladox […] 13https://github.com/miraheze/ImportDump/pull/164#issuecomment-3831623302 [18:02:33] !log [paladox@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [18:02:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:02:35] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'MirahezeMagic'} to all [18:02:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:02:37] !log [paladox@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [18:02:38] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'MirahezeMagic'} to test151 [18:02:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:02:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:07:15] miraheze/MirahezeMagic - paladox the build has errored. [18:07:21] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 285s [18:07:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:07:40] miraheze/MirahezeMagic - paladox the build has errored. [18:09:20] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'MirahezeMagic'} to test151 - SUCCESS in 402s [18:09:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:17:21] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 45312MiB (10.0% inode=100%); [18:25:42] [02ImportDump] 07paladox commented on pull request #164: It seems to be causing failures for some of the ext that have 1.45 in the CI 13https://github.com/miraheze/ImportDump/pull/164#issuecomment-3831707595 [18:34:45] [02puppet] 07Universal-Omega approved pull request #4736 13https://github.com/miraheze/puppet/pull/4736#pullrequestreview-3736460741 [18:38:05] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/b23e23cd9b0ea12920a21f7a8b580a26b5740997 [18:38:06] 02puppet/03paladox-patch-1 07paladox 03b23e23c Update nginx.conf.erb [18:38:45] PROBLEM - mw181 Puppet on mw181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [18:38:48] PROBLEM - mwtask161 Puppet on mwtask161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 26 minutes ago with 0 failures [18:39:04] PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 27 minutes ago with 0 failures [18:39:07] PROBLEM - mw162 Puppet on mw162 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 30 minutes ago with 0 failures [18:39:11] PROBLEM - mw171 Puppet on mw171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 18 minutes ago with 0 failures [18:39:20] PROBLEM - mw153 Puppet on mw153 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 18 minutes ago with 0 failures [18:39:33] PROBLEM - mw161 Puppet on mw161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 29 minutes ago with 0 failures [18:39:35] [02puppet] 07paladox commented on pull request #4736: I applied this to test and did some basic browsing of meta.mirabeta.org (nothing sophisticated) to make sure there was no breakage. 13https://github.com/miraheze/puppet/pull/4736#issuecomment-3831737561 [18:39:40] [02puppet] 07paladox merged pull request #4736: mediawiki: set nginx timeout configs (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4736 [18:39:40] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e57813ec055141171649f991e8a22a3199c41df5 [18:39:40] 02puppet/03main 07paladox 03e57813e mediawiki: set nginx timeout configs (#4736)… [18:39:41] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03b23e23c 13https://api.github.com/repos/miraheze/puppet/commit/b23e23c [18:39:41] PROBLEM - mw202 Puppet on mw202 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 30 minutes ago with 0 failures [18:39:46] PROBLEM - mwtask151 Puppet on mwtask151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 26 minutes ago with 0 failures [18:39:55] PROBLEM - mw172 Puppet on mw172 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 29 minutes ago with 0 failures [18:39:57] PROBLEM - mw173 Puppet on mw173 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 27 minutes ago with 0 failures [18:40:02] PROBLEM - mw191 Puppet on mw191 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 19 minutes ago with 0 failures [18:40:45] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:40:48] RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:41:04] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:41:07] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:41:11] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:41:20] RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:41:33] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:41:41] RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:41:45] RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:41:53] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [18:41:57] RECOVERY - mw173 Puppet on mw173 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [18:42:02] RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:45:11] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'Math'} to all [18:45:12] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'Math'} to test151 [18:45:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:45:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:47:30] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'Math'} to all - SUCCESS in 139s [18:47:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:51:18] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'Math'} to test151 - SUCCESS in 366s [18:51:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:24] [02mw-config] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/3de3b36ca93ca25b9b019815b0118aa110aeb263 [18:54:24] 02mw-config/03main 07paladox 033de3b36 Remove bast181 temp [18:54:36] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [18:54:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:55:05] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 28s [18:55:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:55:15] miraheze/mw-config - paladox the build passed. [18:59:04] PROBLEM - mw192 NTP time on mw192 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:05] PROBLEM - bast181 PowerDNS Recursor on bast181 is CRITICAL: Domain 'bast181.wikitide.net' was not found by the server [18:59:06] PROBLEM - mw153 NTP time on mw153 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:06] PROBLEM - phorge171 NTP time on phorge171 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:06] PROBLEM - db171 NTP time on db171 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:06] PROBLEM - mw173 NTP time on mw173 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:11] PROBLEM - os191 NTP time on os191 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:12] PROBLEM - db181 NTP time on db181 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:12] PROBLEM - mw152 NTP time on mw152 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:13] PROBLEM - mwtask181 NTP time on mwtask181 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:14] PROBLEM - mw191 NTP time on mw191 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:15] PROBLEM - kafka182 NTP time on kafka182 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:15] PROBLEM - mw193 NTP time on mw193 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:15] PROBLEM - swiftproxy171 NTP time on swiftproxy171 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:16] PROBLEM - mwtask161 NTP time on mwtask161 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:16] PROBLEM - cloud18 NTP time on cloud18 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:17] PROBLEM - cloud16 NTP time on cloud16 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:17] PROBLEM - llm191 NTP time on llm191 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:21] PROBLEM - os151 NTP time on os151 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:22] PROBLEM - mw181 NTP time on mw181 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:25] PROBLEM - mw151 NTP time on mw151 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:27] PROBLEM - ldap171 NTP time on ldap171 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:33] PROBLEM - mw201 NTP time on mw201 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:35] PROBLEM - swiftproxy161 NTP time on swiftproxy161 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:35] PROBLEM - cloud19 NTP time on cloud19 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:39] PROBLEM - swiftobject191 NTP time on swiftobject191 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:44] PROBLEM - db192 NTP time on db192 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:45] PROBLEM - swiftac171 NTP time on swiftac171 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:49] PROBLEM - test151 NTP time on test151 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:50] PROBLEM - swiftobject201 NTP time on swiftobject201 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:50] PROBLEM - mw203 NTP time on mw203 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:52] PROBLEM - mwtask151 NTP time on mwtask151 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:52] PROBLEM - cp191 NTP time on cp191 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:54] PROBLEM - reports171 NTP time on reports171 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:54] PROBLEM - mw183 NTP time on mw183 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:55] PROBLEM - mem201 NTP time on mem201 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:56] PROBLEM - os201 NTP time on os201 is UNKNOWN: NTP UNKNOWN: Offset unknown| [18:59:57] PROBLEM - mw182 NTP time on mw182 is CRITICAL: NTP CRITICAL: Offset unknown| [18:59:58] PROBLEM - rdb191 NTP time on rdb191 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:03] PROBLEM - bots171 NTP time on bots171 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:03] PROBLEM - mw172 NTP time on mw172 is CRITICAL: NTP CRITICAL: Offset unknown| [19:00:04] PROBLEM - cloud20 NTP time on cloud20 is CRITICAL: NTP CRITICAL: Offset unknown| [19:00:04] PROBLEM - changeprop202 NTP time on changeprop202 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:05] PROBLEM - graylog161 NTP time on graylog161 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:05] PROBLEM - mw161 NTP time on mw161 is CRITICAL: NTP CRITICAL: Offset unknown| [19:00:05] PROBLEM - cloud15 NTP time on cloud15 is CRITICAL: NTP CRITICAL: Offset unknown| [19:00:05] PROBLEM - os161 NTP time on os161 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:09] PROBLEM - db182 NTP time on db182 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:12] PROBLEM - os162 NTP time on os162 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:12] PROBLEM - mw162 NTP time on mw162 is CRITICAL: NTP CRITICAL: Offset unknown| [19:00:22] PROBLEM - puppet181 NTP time on puppet181 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:22] PROBLEM - mwtask171 NTP time on mwtask171 is CRITICAL: NTP CRITICAL: Offset unknown| [19:00:23] PROBLEM - cp201 NTP time on cp201 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:24] PROBLEM - eventgate182 NTP time on eventgate182 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:24] PROBLEM - mw163 NTP time on mw163 is CRITICAL: NTP CRITICAL: Offset unknown| [19:00:26] PROBLEM - cloud17 NTP time on cloud17 is CRITICAL: NTP CRITICAL: Offset unknown| [19:00:27] PROBLEM - matomo151 NTP time on matomo151 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:28] PROBLEM - swiftobject151 NTP time on swiftobject151 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:29] PROBLEM - db161 NTP time on db161 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:29] PROBLEM - os202 NTP time on os202 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:33] PROBLEM - mon181 NTP time on mon181 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:35] PROBLEM - db151 NTP time on db151 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:35] PROBLEM - mw163 Puppet on mw163 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [19:00:42] PROBLEM - db172 NTP time on db172 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:42] PROBLEM - prometheus151 NTP time on prometheus151 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:42] PROBLEM - db201 NTP time on db201 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:42] PROBLEM - mem161 NTP time on mem161 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:44] PROBLEM - cp161 NTP time on cp161 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:48] PROBLEM - swiftobject181 NTP time on swiftobject181 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:49] PROBLEM - swiftobject171 NTP time on swiftobject171 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:49] PROBLEM - cp171 NTP time on cp171 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:51] PROBLEM - mem151 NTP time on mem151 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:51] PROBLEM - mem191 NTP time on mem191 is UNKNOWN: NTP UNKNOWN: Offset unknown| [19:00:51] PROBLEM - bast181 NTP time on bast181 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o