[00:11:07] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 52235MiB (11% inode=99%); [00:12:45] RECOVERY - cp38 Disk Space on cp38 is OK: DISK OK - free space: / 51772MiB (11% inode=99%); [00:17:43] PROBLEM - www.ethene.wiki - Cloudflare on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [00:28:30] PROBLEM - ethene.wiki - Cloudflare on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [00:30:45] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/18aa4cc12f48d2fa38997909dbe0ce091ab43352 [00:30:45] 02puppet/03main 07CosmicAlpha 0318aa4cc Remove outdated comment [01:17:41] RECOVERY - www.ethene.wiki - Cloudflare on sslhost is OK: OK - Certificate 'www.ethene.wiki' will expire on Tue 05 Aug 2025 12:29:46 AM GMT +0000. [01:27:29] RECOVERY - ethene.wiki - Cloudflare on sslhost is OK: OK - Certificate 'ethene.wiki' will expire on Tue 22 Jul 2025 02:33:59 PM GMT +0000. [01:44:45] PROBLEM - cp38 Disk Space on cp38 is WARNING: DISK WARNING - free space: / 49915MiB (10% inode=99%); [02:06:14] PROBLEM - www.baldisbasics.wiki - Cloudflare on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [02:09:07] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49830MiB (10% inode=99%); [02:38:57] !log [universalomega@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_skins': 'Cosmos'} to all [02:38:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:39:20] !log [universalomega@mwtask181] finished deploy of {'versions': '1.43', 'upgrade_skins': 'Cosmos'} to all - SUCCESS in 23s [02:39:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:11:33] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:13:28] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4037 bytes in 0.277 second response time [10:55:28] PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 1 backends are down. mw153 [10:56:13] RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy [12:33:22] PROBLEM - db171 Backups SQL on db171 is WARNING: FILE_AGE WARNING: /var/log/db-backups/db-backups/db-backups.log is 1382522 seconds old and 4105116 bytes [13:41:54] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [13:44:04] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:44:54] huh [13:45:18] just ntp [13:45:22] shrug [13:46:04] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 7.700920105e-05 secs [14:14:18] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:16:21] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0001229047775 secs [14:49:57] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_3d2png] [15:17:57] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:32:25] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:34:28] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0001572966576 secs [15:57:53] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [15:59:56] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -3.844499588e-06 secs [16:10:09] [02puppet] 07Universal-Omega created 03safe-http-reuse (+1 new commit) 13https://github.com/miraheze/puppet/commit/6ff9137eb679 [16:10:09] 02puppet/03safe-http-reuse 07CosmicAlpha 036ff9137 haproxy: use safe http reuse for backends [16:10:13] [02puppet] 07Universal-Omega opened pull request #4365: haproxy: use safe http reuse for backends (03main...03safe-http-reuse) 13https://github.com/miraheze/puppet/pull/4365 [16:10:20] [02puppet] 07coderabbitai[bot] commented on pull request #4365: --- […] 13https://github.com/miraheze/puppet/pull/4365#issuecomment-2936156898 [17:40:10] PROBLEM - cp38 Puppet on cp38 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 16 minutes ago with 0 failures [17:58:11] RECOVERY - cp38 Puppet on cp38 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [19:39:47] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=vocaloidlyricswiki 138300 (END - exit=2) [19:39:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:19] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=vocaloidlyricswiki 214700 (START) [19:40:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:26] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=vocaloidlyricswiki 214700 (END - exit=2) [19:40:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:44:22] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=vocaloidlyricswiki 214700 (START) [19:44:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:45:51] this time it hung at the same state for hours [19:47:09] MacFan4000: why not use the bash script to shard it by page count [19:47:21] Not shard, chunk [19:48:32] oh, i've seen claire use somthing like that, but don't know where this is located [19:48:50] On the mediawiki.org doc for refreshLinks [19:56:00] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=vocaloidlyricswiki 214700 (END - exit=2) [19:56:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:06] !log [macfan@mwtask181] runRefreshLinksChunked: [1/1] Done \o/ [19:57:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:33] !log [macfan@mwtask181] runRefreshLinksChunked: [0/1] Handling vocaloidlyricswiki (start id: 138300) [20:02:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:16] i'm using the script in claire's homedir [20:05:31] Ok [20:07:07] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27191MiB (5% inode=99%); [20:24:45] PROBLEM - cp38 Disk Space on cp38 is CRITICAL: DISK CRITICAL - free space: / 27227MiB (5% inode=99%); [20:48:11] PROBLEM - cp38 Puppet on cp38 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 23 minutes ago with 0 failures [20:55:32] [02puppet] 07Universal-Omega created 03haproxy-balance (+1 new commit) 13https://github.com/miraheze/puppet/commit/d6fe2376f73d [20:55:32] 02puppet/03haproxy-balance 07CosmicAlpha 03d6fe237 haproxy: adjust balancing [20:55:36] [02puppet] 07Universal-Omega opened pull request #4366: haproxy: adjust balancing (03main...03haproxy-balance) 13https://github.com/miraheze/puppet/pull/4366 [20:55:42] [02puppet] 07coderabbitai[bot] commented on pull request #4366: --- […] 13https://github.com/miraheze/puppet/pull/4366#issuecomment-2937168299 [21:57:57] !log ran ALTER TABLE echo_event ADD event_variant VARBINARY(64) DEFAULT NULL; on morerandomstuffwikibeta [21:58:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:23:28] PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 1 backends are down. mw151 [23:27:28] RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy