[00:09:56] !log [paladox@mwtask181] [root@mwtask181:/srv/mediawiki/1.42/maintenance]# sudo -u www-data php cleanupTitles.php --wiki whitesidewiki [00:10:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:12:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:53:51] PROBLEM - db171 Backups SQL on db171 is CRITICAL: FILE_AGE CRITICAL: /var/log/db-backups/db-backups/db-backups.log is 1468918 seconds old and 1198964 bytes [10:42:45] PROBLEM - cloud18 Puppet on cloud18 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:10:45] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [12:23:49] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 141.81 ms [12:25:53] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 142.17 ms [14:54:47] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/deb23d714831 [14:54:47] 02puppet/03paladox-patch-1 07paladox 03deb23d7 puppetserver: bind to all network addresses for jmx [14:54:52] [02puppet] 07paladox opened pull request #4129: puppetserver: bind to all network addresses for jmx (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4129 [14:55:01] [02puppet] 07paladox merged pull request #4129: puppetserver: bind to all network addresses for jmx (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4129 [14:55:03] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/b8a18a615db8371e7051037d5643d3ee816df946 [14:55:04] 02puppet/03master 07paladox 03b8a18a6 puppetserver: bind to all network addresses for jmx (#4129) [14:55:06] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03deb23d7 13https://github.com/miraheze/puppet/commit/deb23d7 [14:55:55] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/910d398c1d6bfef847ab3afc5d46d7fa99306ccf [14:55:55] 02puppet/03master 07paladox 03910d398 puppetdb: bind to all network addresses for jmx [14:58:59] PROBLEM - swiftobject181 Puppet on swiftobject181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:59:18] PROBLEM - db161 Puppet on db161 is CRITICAL: CRITICAL: Puppet has 34 failures. Last run 2 minutes ago with 34 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[/etc/rsyslog.d],File[/etc/rsyslog.conf] [14:59:24] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:00:12] PROBLEM - db171 Puppet on db171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:05:51] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/df5f47272abe0be1a8d1845b8c62a8c52a6a9361 [15:05:51] 02puppet/03master 07paladox 03df5f472 Fix [15:09:38] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 2 minutes ago with 31 failures. Failed resources (up to 3 shown): File[/etc/nginx/fastcgi_params],File[/usr/local/bin/systemd-timer-mail-wrapper],File[/usr/local/bin/mediawiki-firejail-convert],File[/etc/firejail/mediawiki.local] [15:10:47] PROBLEM - jobchron171 Puppet on jobchron171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:24:59] [02puppet] 07coderabbitai[bot] commented on pull request #4129: --- […] 13https://github.com/miraheze/puppet/pull/4129#issuecomment-2558151110 [15:27:15] RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [15:27:18] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [15:27:24] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [15:28:12] RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:30:48] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/6531b887ce47 [15:30:48] 02puppet/03paladox-patch-1 07paladox 036531b88 puppet: Set reports to store [15:30:53] [02puppet] 07paladox opened pull request #4130: puppet: Set reports to store (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4130 [15:31:01] [02puppet] 07coderabbitai[bot] commented on pull request #4130: --- […] 13https://github.com/miraheze/puppet/pull/4130#issuecomment-2558152496 [15:31:08] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/b4025b7bf2e04767fef8a14b104fd46c7a587869 [15:31:08] 02puppet/03paladox-patch-1 07paladox 03b4025b7 Update puppet.conf.erb [15:31:17] [02puppet] 07paladox merged pull request #4130: puppet: Set reports to store (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4130 [15:31:19] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/58bb7e7bf8203ffbbb5c3c59d78b6c24b211aa66 [15:31:21] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03b4025b7 13https://github.com/miraheze/puppet/commit/b4025b7 [15:31:22] 02puppet/03master 07paladox 0358bb7e7 puppet: Set reports to store (#4130) [15:34:32] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:35:05] PROBLEM - os151 Puppet on os151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:36:29] PROBLEM - mw154 Puppet on mw154 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:37:56] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [15:38:47] RECOVERY - jobchron171 Puppet on jobchron171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:40:39] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/1669f28e7cec [15:40:39] 02puppet/03paladox-patch-1 07paladox 031669f28 opensearch: Fix "please use stdlib::merge instead" as .merge is deprecated [15:40:43] [02puppet] 07paladox opened pull request #4131: opensearch: Fix "please use stdlib::merge instead" as .merge is depre… (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4131 [15:40:47] [02puppet] 07paladox retitled PR #4131: "opensearch: Fix "please use stdlib::merge instead" as .merge is depre…" ➜ "opensearch: Fix "please use stdlib::merge instead" as .merge is deprecated" 13https://github.com/miraheze/puppet/pull/4131 [15:40:49] [02puppet] 07coderabbitai[bot] commented on pull request #4131: --- […] 13https://github.com/miraheze/puppet/pull/4131#issuecomment-2558154846 [15:41:55] [02puppet] 07paladox merged pull request #4131: opensearch: Fix "please use stdlib::merge instead" as .merge is deprecated (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4131 [15:41:57] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/4345c44a71fe121a7ba84ce55c03244c71b03625 [15:41:59] 02puppet/03master 07paladox 034345c44 opensearch: Fix "please use stdlib::merge instead" as .merge is deprecated (#4131) [15:42:01] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 031669f28 13https://github.com/miraheze/puppet/commit/1669f28 [15:43:05] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/535d63434564dd15568bc91fe1d71873f0246260 [15:43:05] 02puppet/03master 07paladox 03535d634 Fix [15:45:05] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:47:08] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/20828ba13f56dfc70d841523b53daf2673b6f68e [15:47:08] 02puppet/03master 07paladox 0320828ba Update connection.rb [15:48:43] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/02e83c46b34b903b25df0d5aa7f504d1f5c1f74e [15:48:43] 02puppet/03master 07paladox 0302e83c4 Fix deprecated [16:04:28] RECOVERY - mw154 Puppet on mw154 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:04:36] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:38:38] PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:40:35] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [17:23:55] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/4deed45b0313880a4db48024c4f5a760b8d35a38 [17:23:55] 02puppet/03master 07paladox 034deed45 cp37: reduce cp37 cache size [17:25:14] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/cc07c8ceb411faec9c5558c56b180ca34953a219 [17:25:15] 02puppet/03master 07paladox 03cc07c8c Fix [17:27:28] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 61650MiB (13% inode=99%); [20:49:48] [02puppet] 07paladox created 03migrate-query (+1 new commit) 13https://github.com/miraheze/puppet/commit/6cc5006162ca [20:49:48] 02puppet/03migrate-query 07paladox 036cc5006 Migrate from puppetdbquery to puppetdb::query_facts… [20:49:58] [02puppet] 07paladox opened pull request #4132: Migrate from puppetdbquery to puppetdb::query_facts (03master...03migrate-query) 13https://github.com/miraheze/puppet/pull/4132 [20:50:04] [02puppet] 07coderabbitai[bot] commented on pull request #4132: --- […] 13https://github.com/miraheze/puppet/pull/4132#issuecomment-2558237475 [20:50:12] [02puppet] 07paladox commented on pull request #4132: Not to be merged until deployment freeze is over. 13https://github.com/miraheze/puppet/pull/4132#issuecomment-2558237505 [20:50:53] miraheze/puppet - paladox the build has errored. [22:02:14] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:46:14] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:48:35] PROBLEM - decimatedrive.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'decimatedrive.wiki' expires in 14 day(s) (Sun 05 Jan 2025 10:46:22 PM GMT +0000). [22:48:46] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/30de6cd026529e937a94eaaaadac3fba5cae5533 [22:48:47] 02ssl/03master 07WikiTideSSLBot 0330de6cd Bot: Update SSL cert for decimatedrive.wiki [23:29:24] [02puppet] 07paladox 04force-pushed 03migrate-query from 036cc5006 to 03f97deee 13https://github.com/miraheze/puppet/commits/migrate-query [23:29:24] 02puppet/03migrate-query 07paladox 03f97deee Migrate from puppetdbquery to puppetdb::query_facts… [23:48:33] RECOVERY - decimatedrive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'decimatedrive.wiki' will expire on Fri 21 Mar 2025 09:50:10 PM GMT +0000.