[00:00:01] !log [www-data@test151] Began automatic backing up [00:00:04] !log [www-data@mwtask181] Began automatic backing up [00:00:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:00:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:01:45] RECOVERY - matomo151 Disk Space on matomo151 is OK: DISK OK - free space: / 7033MiB (39% inode=90%); [00:17:18] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 51561MiB (11% inode=99%); [00:25:13] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 12.18, 8.57, 4.87 [00:29:13] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 9.69, 10.26, 6.47 [00:31:13] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 5.78, 8.75, 6.38 [00:50:24] [02CreateWiki] 07AgentIsai pushed 1 new commit to 03master 13https://github.com/miraheze/CreateWiki/commit/402d714825e991d73767c65196d2b37a66f1d734 [00:50:24] 02CreateWiki/03master 07Agent Isai 03402d714 Properly process onhold requests [00:53:36] [02CreateWiki] 07AgentIsai pushed 1 new commit to 03master 13https://github.com/miraheze/CreateWiki/commit/ba39605d5f5e03cb8c1fcc4ade8b2f9cba0f184f [00:53:36] 02CreateWiki/03master 07Agent Isai 03ba39605 Fix unknown comment [01:05:09] miraheze/CreateWiki - AgentIsai the build passed. [01:07:16] Is this new [01:07:18] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49783MiB (10% inode=99%); [02:11:09] PROBLEM - wonderfuleveryday.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wonderfuleveryday.org' expires in 14 day(s) (Sun 16 Feb 2025 01:46:05 AM GMT +0000). [02:11:22] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/0d99dcd37481f9126229046b29d1d3a8c74f8998 [02:11:22] 02ssl/03master 07WikiTideSSLBot 030d99dcd Bot: Update SSL cert for wonderfuleveryday.org [02:59:47] [02mw-config] 07OAuthority merged 07BlankEclair's pull request #5793: T13130: Set some default options for ssbuniversewiki (07miraheze:03master...07BlankEclair:03T13130) 13https://github.com/miraheze/mw-config/pull/5793 [02:59:47] [02mw-config] 07OAuthority pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/79f1a1263ebb14a76fa2c7e6182ba638baf86504 [02:59:48] 02mw-config/03master 07Claire 0379f1a12 T13130: Set some default options for ssbuniversewiki (#5793)… [03:00:47] miraheze/mw-config - OAuthority the build passed. [03:01:08] [02mw-config] 07OAuthority merged 07BlankEclair's pull request #5792: T13129: Set $wgVectorNightMode for roguetown2ewiki (07miraheze:03master...07BlankEclair:03T13129) 13https://github.com/miraheze/mw-config/pull/5792 [03:01:08] [02mw-config] 07OAuthority pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/62b7330683df2135a73d71200c65aeeeffeeed86 [03:01:09] 02mw-config/03master 07Claire 0362b7330 T13129: Set $wgVectorNightMode for roguetown2ewiki (#5792)… [03:02:12] miraheze/mw-config - OAuthority the build passed. [03:09:09] RECOVERY - wonderfuleveryday.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wonderfuleveryday.org' will expire on Fri 02 May 2025 01:12:45 AM GMT +0000. [03:50:13] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/050be33bd31daf801245cf79566556784a1343b1 [03:50:15] 02ssl/03master 07MacFan4000 03050be33 Add avern.land and needlejuicerecords.wiki [03:53:11] [02ssl] 07MacFan4000 created 03MacFan4000-patch-1 (+1 new commit) 13https://github.com/miraheze/ssl/commit/e644b9cd1952 [03:53:11] 02ssl/03MacFan4000-patch-1 07MacFan4000 03e644b9c Update certs.yaml [03:53:28] [02ssl] 07MacFan4000 opened pull request #822: Remove non-pointed domains (03master...03MacFan4000-patch-1) 13https://github.com/miraheze/ssl/pull/822 [03:54:08] [02ssl] 07coderabbitai[bot] commented on pull request #822: The pull request involves the removal of the `isocasg.org` domain configuration from the `certs.yaml` file. This change eliminates the specific settings for this domain, including its URL, certificate authority (LetsEncrypt), and event status. The […] 13https://github.com/miraheze/ssl/pull/822#issuecomment-2628749782 [03:56:38] [02ssl] 07MacFan4000 pushed 1 new commit to 03MacFan4000-patch-1 13https://github.com/miraheze/ssl/commit/cdd591626cd72d02df05b235f80f92a4275464f0 [03:56:38] 02ssl/03MacFan4000-patch-1 07MacFan4000 03cdd5916 Update certs.yaml [03:57:18] [02ssl] 07MacFan4000 pushed 1 new commit to 03MacFan4000-patch-1 13https://github.com/miraheze/ssl/commit/d790afa1b03dbcd11bc226d851bd0e824785318b [03:57:19] 02ssl/03MacFan4000-patch-1 07MacFan4000 03d790afa Delete certificates/isocasg.org.crt [03:58:20] [02ssl] 07MacFan4000 merged pull request #822: Remove non-pointed domains (03master...03MacFan4000-patch-1) 13https://github.com/miraheze/ssl/pull/822 [03:58:20] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/98a231e68c5307e78052584181ca88ee175fc349 [03:58:20] 02ssl/03master 07MacFan4000 0398a231e Remove non-pointed domains (#822) [03:58:21] [02ssl] 07MacFan4000 04deleted 03MacFan4000-patch-1 at 03d790afa 13https://github.com/miraheze/ssl/commit/d790afa [04:36:43] [02puppet] 07BlankEclair opened pull request #4175: T13127: Add query parameters for /wiki -> /wiki/ redirect (07miraheze:03master...07BlankEclair:03fix-wiki-redirect) 13https://github.com/miraheze/puppet/pull/4175 [04:36:50] [02puppet] 07coderabbitai[bot] commented on pull request #4175: --- […] 13https://github.com/miraheze/puppet/pull/4175#issuecomment-2628765683 [04:37:25] [02mw-config] 07BlankEclair opened pull request #5803: T13127: Fix ?curid=... (07miraheze:03master...07BlankEclair:03T13127) 13https://github.com/miraheze/mw-config/pull/5803 [04:38:19] miraheze/mw-config - BlankEclair the build passed. [06:07:41] [02mw-config] 07BlankEclair opened pull request #5804: T13162: Add custom footer icons for farmwiki (07miraheze:03master...07BlankEclair:03T13162) 13https://github.com/miraheze/mw-config/pull/5804 [06:08:36] miraheze/mw-config - BlankEclair the build passed. [08:57:44] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.58, 6.90, 5.86 [08:59:43] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 6.37, 6.68, 5.90 [09:38:22] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.82, 7.11, 6.46 [09:42:20] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.64, 6.54, 6.40 [10:40:09] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27191MiB (5% inode=99%); [11:30:33] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 143.68 ms [11:34:38] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 143.36 ms [12:08:26] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 30.77, 17.77, 7.54 [12:11:54] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 22.98, 15.89, 8.42 [12:14:07] PROBLEM - mw184 Current Load on mw184 is WARNING: LOAD WARNING - total load average: 20.57, 17.25, 10.07 [12:15:54] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 16.38, 17.72, 10.98 [12:15:59] PROBLEM - ns2 Auth DNS on ns2 is CRITICAL: No response from DNS 2001:41d0:801:2000::4089 [12:16:07] RECOVERY - mw184 Current Load on mw184 is OK: LOAD OK - total load average: 15.60, 16.68, 10.74 [12:16:26] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 3.81, 10.24, 8.61 [12:17:14] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[gdnsd] [12:17:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is UNKNOWN: NRPE: Unable to read output [12:18:26] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 7.87, 9.47, 8.51 [12:22:26] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 21.14, 13.74, 10.33 [12:27:59] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.12, 20.44, 14.54 [12:29:55] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.48, 20.54, 15.28 [12:31:50] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 11.99, 17.76, 14.91 [12:32:26] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 3.50, 11.59, 11.91 [12:36:26] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 3.94, 6.70, 9.77 [12:42:26] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 22.76, 15.89, 12.67 [12:48:26] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 2.85, 9.73, 11.37 [12:50:26] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.58, 6.60, 10.02 [13:05:07] PROBLEM - kafka181 PowerDNS Recursor on kafka181 is CRITICAL: No response from DNS ::1 [13:05:08] PROBLEM - mw174 PowerDNS Recursor on mw174 is CRITICAL: No response from DNS ::1 [13:05:19] PROBLEM - mon181 PowerDNS Recursor on mon181 is CRITICAL: Domain 'wikitide.net' was not found by the server [13:05:23] PROBLEM - db181 PowerDNS Recursor on db181 is CRITICAL: Domain 'wikitide.net' was not found by the server [13:05:27] PROBLEM - mw184 PowerDNS Recursor on mw184 is CRITICAL: Domain 'wikitide.net' was not found by the server [13:05:31] PROBLEM - swiftobject171 PowerDNS Recursor on swiftobject171 is CRITICAL: No response from DNS ::1 [13:05:31] PROBLEM - dragdown.wiki - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [13:05:31] PROBLEM - mw183 PowerDNS Recursor on mw183 is CRITICAL: No response from DNS ::1 [13:05:34] PROBLEM - db151 PowerDNS Recursor on db151 is CRITICAL: Domain 'wikitide.net' was not found by the server [13:05:35] PROBLEM - www.journeytheword.wiki - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [13:05:37] PROBLEM - swiftproxy161 PowerDNS Recursor on swiftproxy161 is CRITICAL: No response from DNS ::1 [13:05:38] PROBLEM - bonesword.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:38] PROBLEM - magistro.co - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [13:05:38] PROBLEM - db151 MariaDB on db151 is UNKNOWN: check_mysql: Invalid hostname/address - db151.wikitide.netUsage: check_mysql [-d database] [-H host] [-P port] [-s socket] [-u user] [-p password] [-S] [-l] [-a cert] [-k key] [-C ca-cert] [-D ca-dir] [-L ciphers] [-f optfile] [-g group] [13:05:39] PROBLEM - db151 MariaDB Connections on db151 is UNKNOWN: PHP Warning: mysqli_real_connect(): php_network_getaddresses: getaddrinfo for db151.wikitide.net failed: Temporary failure in name resolution in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Warning: mysqli_real_connect(): php_network_getaddresses: getaddrinfo for db151.wikitide.net failed: Temporary failure in name resolution in /usr/lib/nagios/plugins/check_mysql [13:05:39] ns.php on line 66PHP Fatal error: Uncaught mysqli_sql_exception: php_network_getaddresses: getaddrinfo for db151.wikitide.net failed: Temporary failure in name resolution in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db151.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown [13:05:39] ib/nagios/plugins/check_mysql_connections.php on line 66Fatal error: Uncaught mysqli_sql_exception: php_network_getaddresses: getaddrinfo for db151.wikitide.net failed: Temporary failure in name resolution in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db151.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL [13:05:39] LL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [13:05:42] PROBLEM - db182 MariaDB on db182 is UNKNOWN: check_mysql: Invalid hostname/address - db182.wikitide.netUsage: check_mysql [-d database] [-H host] [-P port] [-s socket] [-u user] [-p password] [-S] [-l] [-a cert] [-k key] [-C ca-cert] [-D ca-dir] [-L ciphers] [-f optfile] [-g group] [13:05:42] PROBLEM - www.rothwell-leeds.co.uk - Cloudflare on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:45] PROBLEM - lotrmc.ru - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [13:05:47] PROBLEM - mw163 PowerDNS Recursor on mw163 is CRITICAL: Domain 'wikitide.net' was not found by the server [13:05:48] PROBLEM - theenglishlakes.uk - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [13:05:48] PROBLEM - mw162 PowerDNS Recursor on mw162 is CRITICAL: No response from DNS ::1 [13:05:50] PROBLEM - db171 MariaDB on db171 is UNKNOWN: check_mysql: Invalid hostname/address - db171.wikitide.netUsage: check_mysql [-d database] [-H host] [-P port] [-s socket] [-u user] [-p password] [-S] [-l] [-a cert] [-k key] [-C ca-cert] [-D ca-dir] [-L ciphers] [-f optfile] [-g group] [13:05:51] PROBLEM - mem161 PowerDNS Recursor on mem161 is CRITICAL: No response from DNS ::1 [13:05:53] PROBLEM - mw154 PowerDNS Recursor on mw154 is CRITICAL: Domain 'wikitide.net' was not found by the server [13:05:54] PROBLEM - ldap171 PowerDNS Recursor on ldap171 is CRITICAL: No response from DNS ::1 [13:05:54] PROBLEM - bast161 PowerDNS Recursor on bast161 is CRITICAL: No response from DNS ::1 [13:05:56] PROBLEM - cloud18 PowerDNS Recursor on cloud18 is CRITICAL: No response from DNS ::1 [13:05:56] PROBLEM - swiftac171 PowerDNS Recursor on swiftac171 is CRITICAL: No response from DNS ::1 [13:06:00] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [13:06:05] PROBLEM - db182 MariaDB Connections on db182 is UNKNOWN: PHP Warning: mysqli_real_connect(): php_network_getaddresses: getaddrinfo for db182.wikitide.net failed: Temporary failure in name resolution in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Warning: mysqli_real_connect(): php_network_getaddresses: getaddrinfo for db182.wikitide.net failed: Temporary failure in name resolution in /usr/lib/nagios/plugins/check_mysql [13:06:05] ns.php on line 66PHP Fatal error: Uncaught mysqli_sql_exception: php_network_getaddresses: getaddrinfo for db182.wikitide.net failed: Temporary failure in name resolution in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown [13:06:05] ib/nagios/plugins/check_mysql_connections.php on line 66Fatal error: Uncaught mysqli_sql_exception: php_network_getaddresses: getaddrinfo for db182.wikitide.net failed: Temporary failure in name resolution in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL [13:06:05] LL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [13:06:09] PROBLEM - reports171 PowerDNS Recursor on reports171 is CRITICAL: No response from DNS ::1 [13:06:13] PROBLEM - rdb151 PowerDNS Recursor on rdb151 is CRITICAL: No response from DNS ::1 [13:06:17] PROBLEM - db171 MariaDB Connections on db171 is UNKNOWN: PHP Warning: mysqli_real_connect(): php_network_getaddresses: getaddrinfo for db171.wikitide.net failed: Temporary failure in name resolution in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Warning: mysqli_real_connect(): php_network_getaddresses: getaddrinfo for db171.wikitide.net failed: Temporary failure in name resolution in /usr/lib/nagios/plugins/check_mysql [13:06:17] ns.php on line 66PHP Fatal error: Uncaught mysqli_sql_exception: php_network_getaddresses: getaddrinfo for db171.wikitide.net failed: Temporary failure in name resolution in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown [13:06:17] ib/nagios/plugins/check_mysql_connections.php on line 66Fatal error: Uncaught mysqli_sql_exception: php_network_getaddresses: getaddrinfo for db171.wikitide.net failed: Temporary failure in name resolution in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL [13:06:17] LL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [13:06:26] PROBLEM - ns1 NTP time on ns1 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o