[00:18:43] could somebody please merge ^ when they have a chance? [00:20:27] merge what? [00:21:16] 18:08 [dns] MacFan4000 opened pull request #522: Add annapolishistorywiki.org and terraforming.li zones - https://github.com/miraheze/dns/pull/522 [00:21:21] (DNS PR) [00:21:32] oh heh [00:22:03] not sure if that bot is still blocked in the relay [00:22:43] I think it is [00:31:38] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [00:31:39] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [00:38:16] could someone also see about merging my current cw pr so i can eventually work on some other stuff for it? [00:58:34] @reception123 [01:01:10] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Tue 27 Aug 2024 03:54:13 PM GMT +0000. [01:40:05] [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-33/±2] 13https://github.com/miraheze/ssl/compare/b26e711c5837...b7ea9707a627 [01:40:07] [02ssl] 07MacFan4000 03b7ea970 - Remove expired/non-pointed domains [01:43:11] PROBLEM - mem161 Current Load on mem161 is CRITICAL: LOAD CRITICAL - total load average: 6.59, 3.57, 1.52 [01:45:11] RECOVERY - mem161 Current Load on mem161 is OK: LOAD OK - total load average: 0.92, 2.41, 1.34 [01:46:36] PROBLEM - mem161 memcached on mem161 is CRITICAL: connect to address 10.0.16.131 and port 11211: Connection refused [01:54:51] [02dns] 07MacFan4000 opened pull request 03#523: remove zones for expired/non-pointed domains - 13https://github.com/miraheze/dns/pull/523 [01:58:36] RECOVERY - mem161 memcached on mem161 is OK: TCP OK - 0.000 second response time on 10.0.16.131 port 11211 [02:01:37] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:17:11] PROBLEM - mem151 memcached on mem151 is CRITICAL: connect to address 10.0.15.113 and port 11211: Connection refused [02:33:08] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.54, 20.70, 17.10 [02:34:42] RECOVERY - mem151 memcached on mem151 is OK: TCP OK - 0.000 second response time on 10.0.15.113 port 11211 [02:35:02] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 13.95, 18.58, 16.77 [02:51:42] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.55, 21.23, 18.07 [02:52:18] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.23, 17.37, 13.91 [02:53:42] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.62, 22.71, 19.05 [02:54:18] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 11.58, 15.14, 13.53 [02:57:42] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 13.84, 18.28, 18.11 [03:02:12] PROBLEM - mon181 Backups Grafana on mon181 is CRITICAL: FILE_AGE CRITICAL: /var/log/grafana-backup.log is 1209708 seconds old and 93 bytes [05:17:42] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.65, 21.51, 16.36 [05:21:42] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 10.88, 18.06, 16.35 [05:33:06] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:47:24] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [06:07:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:16:09] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp36.wikitide.net - CNAME OK [06:32:34] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [07:12:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:04:41] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:34:18] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:37:54] [02dns] 07redbluegreenhat closed pull request 03#523: remove zones for expired/non-pointed domains - 13https://github.com/miraheze/dns/pull/523 [08:37:55] [02dns] 07redbluegreenhat pushed 031 commit to 03master [+0/-13/±0] 13https://github.com/miraheze/dns/compare/e0068c030ed9...4ef8c037db07 [08:37:57] [02dns] 07MacFan4000 034ef8c03 - remove zones for expired/non-pointed domains (#523) [08:38:07] [02dns] 07redbluegreenhat closed pull request 03#522: Add annapolishistorywiki.org and terraforming.li zones - 13https://github.com/miraheze/dns/pull/522 [08:38:08] [02dns] 07redbluegreenhat pushed 031 commit to 03master [+2/-0/±0] 13https://github.com/miraheze/dns/compare/4ef8c037db07...0de1875388a6 [08:38:11] [02dns] 07MacFan4000 030de1875 - Add annapolishistorywiki.org and terraforming.li zones (#522) [08:38:14] MacFan4000: merged ^ [08:51:07] !log [alex@mwtask181] starting deploy of {'l10n': True, 'extension_list': True, 'versions': '1.41'} to all [08:51:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:52:47] !log [alex@mwtask181] finished deploy of {'l10n': True, 'extension_list': True, 'versions': '1.41'} to all - SUCCESS in 99s [08:52:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:55:50] [02mw-config] 07redbluegreenhat closed pull request 03#5570: T12146: Add CustomSubtitle - 13https://github.com/miraheze/mw-config/pull/5570 [08:55:51] [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d2e0bb72d8ca...87d8e37f1820 [08:55:54] [02mw-config] 07BlankEclair 0387d8e37 - T12146: Add CustomSubtitle (#5570) [08:56:22] !log [alex@mwtask181] starting deploy of {'files': '1.41/extensions/CustomSubtitle'} to all [08:56:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:56:37] !log [alex@mwtask181] finished deploy of {'files': '1.41/extensions/CustomSubtitle'} to all - SUCCESS in 14s [08:56:51] miraheze/mw-config - redbluegreenhat the build passed. [08:56:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:58:13] !log [alex@mwtask181] starting deploy of {'files': '../mediawiki-staging/1.41/extensions/CustomSubtitle'} to all [08:58:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:58:32] !log [alex@mwtask181] finished deploy of {'files': '../mediawiki-staging/1.41/extensions/CustomSubtitle'} to all - SUCCESS in 19s [08:58:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:00:42] !log [alex@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CustomSubtitle'} to all [09:00:43] !log [alex@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CustomSubtitle'} to all - SUCCESS in 1s [09:01:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:01:04] !log [alex@mwtask181] starting deploy of {'force': True, 'versions': '1.41', 'upgrade_extensions': 'CustomSubtitle'} to all [09:01:05] !log [alex@mwtask181] finished deploy of {'force': True, 'versions': '1.41', 'upgrade_extensions': 'CustomSubtitle'} to all - SUCCESS in 1s [09:01:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:01:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:01:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:02:14] !log [@mwtask181] starting deploy of {'config': True} to all [09:02:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:02:31] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 17s [09:02:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:02:36] !log [alex@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.41', 'upgrade_extensions': 'CustomSubtitle'} to all [09:02:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:02:49] !log [alex@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.41', 'upgrade_extensions': 'CustomSubtitle'} to all - SUCCESS in 12s [09:02:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:02:58] that one did the trick ^ [09:03:10] https://github.com/miraheze/puppet/blob/1d346d527cee748aa22e9d0bd3b3ec501ce4fbb7/modules/mediawiki/files/bin/mwdeploy.py#L449 [09:03:59] I couldn't just --world due to reasons, which is why I did it that way [09:06:02] !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [09:06:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:06:17] !log [alex@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 14s [09:06:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:07:14] !log [alex@test151] starting deploy of {'force_upgrade': True, 'versions': '1.42', 'upgrade_extensions': 'CustomSubtitle'} to test151 [09:07:15] !log [alex@test151] finished deploy of {'force_upgrade': True, 'versions': '1.42', 'upgrade_extensions': 'CustomSubtitle'} to test151 - SUCCESS in 1s [09:07:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:07:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:07:27] !log [@mwtask171] starting deploy of {'config': True} to all [09:07:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:07:36] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 9s [09:07:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:08:51] !log [alex@test151] starting deploy of {'extension_list': True, 'versions': '1.42'} to test151 [09:08:52] !log [alex@test151] finished deploy of {'extension_list': True, 'versions': '1.42'} to test151 - SUCCESS in 1s [09:08:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:09:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:09:57] !log [alex@test151] starting deploy of {'l10n': True, 'versions': '1.42'} to test151 [09:10:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:15:29] !log [alex@test151] finished deploy of {'l10n': True, 'versions': '1.42'} to test151 - SUCCESS in 332s [09:15:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:15:47] !log [alex@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [09:15:48] !log [alex@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s [09:15:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:15:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:17:43] !log [alex@mwtask181] starting deploy of {'l10n': True, 'versions': '1.41'} to all [09:17:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:24:31] !log [alex@mwtask181] finished deploy of {'l10n': True, 'versions': '1.41'} to all - SUCCESS in 407s [09:24:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:26:27] [02mw-config] 07redbluegreenhat commented on pull request 03#5570: T12146: Add CustomSubtitle - 13https://github.com/miraheze/mw-config/pull/5570#issuecomment-2155890702 [09:43:37] [02mw-config] 07BlankEclair commented on pull request 03#5570: T12146: Add CustomSubtitle - 13https://github.com/miraheze/mw-config/pull/5570#issuecomment-2155902680 [10:07:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:14:59] [02mw-config] 07BlankEclair commented on pull request 03#5570: T12146: Add CustomSubtitle - 13https://github.com/miraheze/mw-config/pull/5570#issuecomment-2155969811 [10:22:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:07:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:17:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:25:55] [02mw-config] 07redbluegreenhat commented on pull request 03#5570: T12146: Add CustomSubtitle - 13https://github.com/miraheze/mw-config/pull/5570#issuecomment-2155996791 [11:28:49] !log [alex@mwtask181] starting deploy of {'extension_list': True, 'versions': '1.41'} to all [11:28:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:29:01] !log [alex@mwtask181] finished deploy of {'extension_list': True, 'versions': '1.41'} to all - SUCCESS in 11s [11:29:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:33:39] [02mw-config] 07redbluegreenhat commented on pull request 03#5570: T12146: Add CustomSubtitle - 13https://github.com/miraheze/mw-config/pull/5570#issuecomment-2155998934 [11:35:21] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:49:23] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/b7ea9707a627...a9d7b7905e96 [11:49:26] [02ssl] 07WikiTideSSLBot 03a9d7b79 - Bot: Add SSL cert for annapolishistorywiki.org [11:51:08] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/a9d7b7905e96...4e2f1993ac7f [11:51:11] [02ssl] 07WikiTideSSLBot 034e2f199 - Bot: Add SSL cert for terraforming.li [12:02:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:04:54] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [12:16:28] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway [12:16:50] PROBLEM - phorge171 php-fpm on phorge171 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [12:17:39] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.013 second response time [12:36:29] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [12:38:18] RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17718 bytes in 0.039 second response time [12:38:42] RECOVERY - phorge171 php-fpm on phorge171 is OK: PROCS OK: 9 processes with command name 'php-fpm8.2' [12:38:43] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.42, 19.59, 16.44 [12:39:25] RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19091 bytes in 0.063 second response time [12:40:41] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.39, 17.52, 16.05 [12:55:37] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:34] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0001420378685 secs [13:05:29] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.82, 21.25, 19.20 [13:06:08] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:07:27] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.22, 19.39, 18.78 [13:14:56] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o