[00:01:17] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/0525bc3e639e2d22accc40ef368fabd291ba360b [00:01:18] 02statichelp/03main 07WikiTideBot 030525bc3 Bot: Auto-update Tech namespace pages 2025-10-10 00:01:15 [00:02:59] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760050790000&orgId=1&to=1760054390000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1760054579243[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1 [00:02:59] 43[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760050880000&orgId=1&to=1760054579243[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760050790000&orgId=1&to=1760054570000 [00:04:46] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.66, 6.54, 7.70 [00:07:59] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760051090000&orgId=1&to=1760054690000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1760054879244[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1 [00:07:59] 44[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760050880000&orgId=1&to=1760054660000 [00:08:47] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.56, 6.67, 7.39 [00:09:42] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:09:48] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:09:48] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 54551MiB (12% inode=99%); [00:11:18] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 55057MiB (12% inode=99%); [00:11:49] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 54326MiB (11% inode=99%); [00:12:12] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:12:21] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:14:42] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 9 minutes ago with 0 failures [00:14:43] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [00:15:49] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [00:15:55] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.054 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [00:17:59] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760051750000&orgId=1&to=1760055350000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1760055479245[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1 [00:17:59] 45[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760051180000&orgId=1&to=1760055410000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760051720000&orgId=1&to=1760055440000 [00:18:46] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 0.78, 5.44, 7.21 [00:20:46] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.22, 3.68, 6.35 [00:52:59] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1760057460000[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1760057460000 [03:25:27] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/c4849a9c4d596723d326d512ccff872574e6cdc3 [03:25:27] 02ssl/03main 07WikiTideBot 03c4849a9 Bot: Auto-update domain lists [03:41:49] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49814MiB (10% inode=99%); [03:51:48] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49848MiB (10% inode=99%); [04:15:18] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49899MiB (10% inode=99%); [04:28:08] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [04:36:07] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [05:01:49] [Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760068860000&orgId=1&to=1760072509754 [05:06:46] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:06:49] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760068860000&orgId=1&to=1760072809755[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760068860000&orgId=1&to=1760072809755[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760069060000&orgId=1&to=1760072780 [05:08:05] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:08:44] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.092 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [05:08:46] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 10.58, 7.54, 3.58 [05:11:49] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760068860000&orgId=1&to=1760073109755[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760068860000&orgId=1&to=1760073109756[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760069360000&orgId=1&to=1760073080 [05:11:50] a] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760069300000&orgId=1&to=1760072930000 [05:14:05] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [05:14:46] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.95, 6.98, 4.84 [05:16:46] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.90, 4.95, 4.36 [05:29:35] [02mw-config] 07songnguxyz opened pull request #6139: Removing conflict status for ParserPower (07miraheze:03main...07songnguxyz:03T13921) 13https://github.com/miraheze/mw-config/pull/6139 [05:31:29] miraheze/mw-config - songnguxyz the build passed. [05:32:12] @pskyechology hope you don't mind me removing conflict status for those who uses pageforms [05:32:22] #6139 [05:35:53] nah but I won't be able to deploy for a week anyway so someone else will have to do it [05:36:13] meh, fair [05:36:43] still hope someone get to it soon enough so my main wiki can also has it [05:36:56] -# I have Pageform enabled [05:51:49] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760068860000&orgId=1&to=1760075460000[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760068860000&orgId=1&to=1760075460000 [07:13:15] PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:15:15] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 126 packages available for upgrade (0 critical updates). [07:46:04] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/e47331dbf7f8f089b2314ae8070a95484fa3b31b [07:46:04] 02mw-config/03main 07Vũ Minh Nguyên 03e47331d Removing conflict status for ParserPower (#6139) [07:46:04] [02mw-config] 07SomeMWDev merged 07songnguxyz's pull request #6139: Removing conflict status for ParserPower (07miraheze:03main...07songnguxyz:03T13921) 13https://github.com/miraheze/mw-config/pull/6139 [07:46:38] !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [07:46:39] !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [07:46:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:46:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:46:59] miraheze/mw-config - SomeMWDev the build passed. [07:48:04] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [07:48:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:48:26] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s [07:48:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:49:06] deployed (using my phone) [09:55:35] man I don't know how you guys trust your phones to have SSH access tbh [09:56:15] I'd be paranoid if I had my SSH keys on my phone, I'd quite literally surgically attach it to my body or I wouldn't be able to sleep [10:43:21] i mean i guess you could do that with a yubikey lol [11:02:28] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760090480000&orgId=1&to=1760094148550 [11:07:28] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760090690000&orgId=1&to=1760094410000 [11:08:02] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760090690000&orgId=1&to=1760094482449 [11:08:46] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 14.06, 8.13, 3.74 [11:10:46] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.27, 6.14, 3.56 [11:13:02] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091110000&orgId=1&to=1760094782450[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760090990000&orgId=1&to=1760094680000 [11:17:00] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:18:02] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091290000&orgId=1&to=1760095010000 [11:18:46] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:18:46] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 10.63, 7.61, 5.00 [11:19:31] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:19:40] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:19:42] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:22:01] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [11:22:05] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 12 minutes ago with 0 failures [11:22:06] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [11:22:43] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [11:23:02] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091650000&orgId=1&to=1760095370000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091660000&orgId=1&to=1760095382450[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091660000&orgId=1&to=1 [11:23:02] 50[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091290000&orgId=1&to=1760095382450[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091770000&orgId=1&to=1760095382450 [11:23:12] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.065 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [11:24:46] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.61, 7.04, 5.90 [11:25:09] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.89, 21.34, 14.27 [11:28:46] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.50, 5.43, 5.59 [11:29:00] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 12.09, 19.41, 15.27 [11:33:02] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760092310000&orgId=1&to=1760095910000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091660000&orgId=1&to=1760095982451[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091660000&orgId=1&to=1 [11:33:02] 51[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760092130000&orgId=1&to=1760095970000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760092280000&orgId=1&to=1760095940000 [11:36:46] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 11.35, 7.67, 6.33 [11:38:02] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760092670000&orgId=1&to=1760096270000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091660000&orgId=1&to=1760096282452[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091660000&orgId=1&to=1 [11:38:02] 52[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760092400000&orgId=1&to=1760096282452[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760092670000&orgId=1&to=1760096282452 [11:38:46] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.45, 5.83, 5.81 [11:40:35] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.43, 20.31, 17.36 [11:43:01] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:43:02] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091660000&orgId=1&to=1760096582452[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091660000&orgId=1&to=1760096582452[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760092760000&orgId=1&to=1760096570 [11:43:02] a] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760092670000&orgId=1&to=1760096450000 [11:43:07] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:44:47] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 9.75, 7.83, 6.60 [11:45:04] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [11:45:30] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:45:39] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:46:26] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 7.13, 15.77, 16.68 [11:46:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 7.23, 7.87, 6.79 [11:47:10] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 3.646 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [11:48:02] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091660000&orgId=1&to=1760096882452[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091660000&orgId=1&to=1760096882452[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760092760000&orgId=1&to=1760096750 [11:48:02] a] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760093120000&orgId=1&to=1760096882453 [11:48:46] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.15, 7.94, 6.94 [11:49:20] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:51:32] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:53:02] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091660000&orgId=1&to=1760097182453[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091660000&orgId=1&to=1760097182453[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760093240000&orgId=1&to=1760097140 [11:53:02] a] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760093120000&orgId=1&to=1760096930000 [11:54:08] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:55:21] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [11:55:37] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.065 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [11:56:07] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [11:56:08] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [11:56:34] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 18 minutes ago with 0 failures [11:56:46] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.79, 6.98, 7.28 [11:58:46] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.36, 4.73, 6.42 [12:33:02] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091660000&orgId=1&to=1760099460000[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760091660000&orgId=1&to=1760099460000 [13:35:27] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/130dab3c5cfc6b2dc982bca11295944cfc929715 [13:35:27] 02ssl/03main 07WikiTideBot 03130dab3 Bot: Auto-update domain lists [14:03:35] [02mw-config] 07dependabot[bot] created 03dependabot/composer/phpunit/phpunit-12.4.1 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/a081109d9ccd [14:03:35] 02mw-config/03dependabot/composer/phpunit/phpunit-12.4.1 07dependabot[bot] 03a081109 Update phpunit/phpunit requirement from 12.4.0 to 12.4.1… [14:03:36] [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #6140 (Update phpunit/phpunit requirement from 12.4.0 to 12.4.1) 13https://github.com/miraheze/mw-config/pull/6140 [14:03:38] [02mw-config] 07dependabot[bot] added the label 'php' to pull request #6140 (Update phpunit/phpunit requirement from 12.4.0 to 12.4.1) 13https://github.com/miraheze/mw-config/pull/6140 [14:03:40] [02mw-config] 07dependabot[bot] opened pull request #6140: Update phpunit/phpunit requirement from 12.4.0 to 12.4.1 (03main...03dependabot/composer/phpunit/phpunit-12.4.1) 13https://github.com/miraheze/mw-config/pull/6140 [14:03:42] [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #6140 (Update phpunit/phpunit requirement from 12.4.0 to 12.4.1) 13https://github.com/miraheze/mw-config/pull/6140 [14:03:44] [02mw-config] 07dependabot[bot] added the label 'php' to pull request #6140 (Update phpunit/phpunit requirement from 12.4.0 to 12.4.1) 13https://github.com/miraheze/mw-config/pull/6140 [14:03:46] [02mw-config] 07dependabot[bot] created 03dependabot/composer/justinrainbow/json-schema-6.6.0 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/1c60741698b2 [14:03:48] [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #6141 (Update justinrainbow/json-schema requirement from 6.5.2 to 6.6.0) 13https://github.com/miraheze/mw-config/pull/6141 [14:03:50] [02mw-config] 07dependabot[bot] added the label 'php' to pull request #6141 (Update justinrainbow/json-schema requirement from 6.5.2 to 6.6.0) 13https://github.com/miraheze/mw-config/pull/6141 [14:03:52] [02mw-config] 07dependabot[bot] opened pull request #6141: Update justinrainbow/json-schema requirement from 6.5.2 to 6.6.0 (03main...03dependabot/composer/justinrainbow/json-schema-6.6.0) 13https://github.com/miraheze/mw-config/pull/6141 [14:03:54] [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #6141 (Update justinrainbow/json-schema requirement from 6.5.2 to 6.6.0) 13https://github.com/miraheze/mw-config/pull/6141 [14:03:56] [02mw-config] 07dependabot[bot] added the label 'php' to pull request #6141 (Update justinrainbow/json-schema requirement from 6.5.2 to 6.6.0) 13https://github.com/miraheze/mw-config/pull/6141 [14:03:58] 02mw-config/03dependabot/composer/justinrainbow/json-schema-6.6.0 07dependabot[bot] 031c60741 Update justinrainbow/json-schema requirement from 6.5.2 to 6.6.0… [14:04:30] miraheze/mw-config - dependabot[bot] the build passed. [14:04:33] miraheze/mw-config - dependabot[bot] the build passed. [15:02:59] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760104940000&orgId=1&to=1760108579808 [15:07:59] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760104940000&orgId=1&to=1760108630000 [17:02:37] [Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760115757531 [17:07:37] [Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760116057531[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112290000&orgId=1&to=1760116010000 [17:07:51] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [17:09:49] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.067 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [17:12:37] [Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760116357532[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112290000&orgId=1&to=1760116280000[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from= [17:12:37] 000&orgId=1&to=1760116357532 [17:17:37] [Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760116657533[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112920000&orgId=1&to=1760116657533[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from [17:17:37] 0000&orgId=1&to=1760116370000 [17:22:37] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760116957533[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760116957533[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113160000&orgId=1&to=1760116910 [17:28:16] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 11.68, 7.97, 5.47 [17:28:19] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [17:29:05] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:56] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:30:05] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:30:06] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:32:20] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [17:32:23] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 23 minutes ago with 0 failures [17:32:24] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.068 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [17:32:24] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [17:32:37] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113790000&orgId=1&to=1760117557535[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760117557535[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from [17:32:37] 0000&orgId=1&to=1760117557535[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113430000&orgId=1&to=1760117557535 [17:33:02] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [17:34:12] :ThinkerMH: [17:34:13] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.08, 7.71, 6.47 [17:37:19] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:37:37] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760114030000&orgId=1&to=1760117630000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760117857536[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1 [17:37:37] 36[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760114090000&orgId=1&to=1760117810000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760114000000&orgId=1&to=1760117780000 [17:38:10] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 9.58, 8.39, 6.96 [17:38:46] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [17:40:44] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.070 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [17:41:16] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [17:42:05] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 4.13, 7.86, 7.21 [17:42:37] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760114450000&orgId=1&to=1760118110000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760118157537[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1 [17:42:37] 37[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760114090000&orgId=1&to=1760118157537[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760114480000&orgId=1&to=1760118157537 [17:44:03] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 5.02, 6.60, 6.80 [17:47:03] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [17:47:37] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760114510000&orgId=1&to=1760118110000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760118457538[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1 [17:47:37] 38[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760114600000&orgId=1&to=1760118410000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760114480000&orgId=1&to=1760118290000 [17:47:59] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.70, 7.65, 7.17 [17:49:20] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:49:29] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:50:34] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:37] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760114990000&orgId=1&to=1760118757539[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760118757539[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from [17:52:37] 0000&orgId=1&to=1760118757539[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760114600000&orgId=1&to=1760118680000 [17:54:15] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [17:54:27] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 13 minutes ago with 0 failures [17:54:31] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [17:55:23] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.064 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [17:55:59] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.96, 7.63, 7.74 [17:57:37] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115290000&orgId=1&to=1760118890000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760119057540[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1 [17:57:37] 40[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760114600000&orgId=1&to=1760118980000[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115290000&orgId=1&to=1760119057540 [17:58:48] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:59:45] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [17:59:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 9.79, 8.60, 8.07 [18:01:12] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:01:21] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:02:37] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760119357541[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760119357541[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time p [18:02:37] s://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115380000&orgId=1&to=1760119357541[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115290000&orgId=1&to=1760119070000 [18:03:00] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:04:52] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [18:04:59] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [18:05:33] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/d142c888651ad44ea2b347ba175a0ef8f3b923dd [18:05:33] 02ssl/03main 07WikiTideBot 03d142c88 Bot: Auto-update domain lists [18:05:57] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.070 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [18:06:03] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [18:06:19] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 25 minutes ago with 0 failures [18:07:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760119657542[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115770000&orgId=1&to=1760119550000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [18:07:37] =1760119657542[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760119657542[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115380000&orgId=1&to=1760119640000[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/G [18:07:37] rom=1760115950000&orgId=1&to=1760119657542 [18:09:08] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:10:20] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [18:11:57] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:12:20] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 2.474 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [18:12:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760119957543[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760116310000&orgId=1&to=1760119957543[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/Gt [18:12:37] om=1760113260000&orgId=1&to=1760119957543[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760119957543[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115380000&orgId=1&to=1760119957543[Grafana] RESOLVED: DatasourceNoData http [18:12:37] a.wikitide.net/d/GtxbP1Xnk?from=1760115950000&orgId=1&to=1760119700000 [18:13:05] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [18:13:57] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [18:17:22] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:17:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760120257544[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760116310000&orgId=1&to=1760120030000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [18:17:37] =1760120257544[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760120257544[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760116490000&orgId=1&to=1760120210000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760116400000&orgId=1&to=1760120180000 [18:19:15] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [18:19:48] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.94, 6.87, 7.89 [18:22:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760120557545[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760116790000&orgId=1&to=1760120390000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [18:22:37] =1760120557545[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760120557545[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760116850000&orgId=1&to=1760120557545[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/ [18:22:37] from=1760116760000&orgId=1&to=1760120540000 [18:25:17] !log [void@changeprop201] update and restart redis [18:25:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:42] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 4.82, 5.03, 6.72 [18:25:58] RECOVERY - changeprop201 APT on changeprop201 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:26:14] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 72 packages available for upgrade (0 critical updates). [18:26:20] RECOVERY - rdb151 APT on rdb151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [18:26:25] !log [void@rdb151] update and restart redis [18:26:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:44] RECOVERY - test151 APT on test151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [18:26:57] !log [void@test151] update and restart redis [18:27:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:27:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760120857546[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760117030000&orgId=1&to=1760120630000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [18:27:37] =1760120857546[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760120857546[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760117090000&orgId=1&to=1760120810000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760117000000&orgId=1&to=1760120660000 [18:28:02] !log [void@matomo151] update and restart redis [18:28:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:28:35] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:28:46] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [18:29:37] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 11.27, 8.14, 7.57 [18:31:14] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:31:22] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:32:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760121157547[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760117450000&orgId=1&to=1760121157547[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/Gt [18:32:37] om=1760113260000&orgId=1&to=1760121157547[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760121157547[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760117090000&orgId=1&to=1760121080000 [18:32:51] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.107 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [18:33:12] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [18:33:22] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 20 minutes ago with 0 failures [18:34:36] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [18:35:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 6.35, 7.90, 7.87 [18:37:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.93, 8.35, 8.05 [18:37:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760121457547[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760117630000&orgId=1&to=1760121230000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [18:37:37] =1760121457548[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760121457548[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760117090000&orgId=1&to=1760121410000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760117630000&orgId=1&to=1760121380000 [18:38:52] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:39:11] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [18:39:59] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:40:17] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:41:47] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:42:36] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 30 minutes ago with 0 failures [18:42:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760121757548[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760118110000&orgId=1&to=1760121757548[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/Gt [18:42:37] om=1760113260000&orgId=1&to=1760121757548[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760121757548[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760117090000&orgId=1&to=1760121757548 [18:42:37] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [18:42:49] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [18:43:17] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.103 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [18:43:47] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [18:47:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760122057549[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760118110000&orgId=1&to=1760121830000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [18:47:37] =1760122057549[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760122057549[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760118290000&orgId=1&to=1760122057550[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/ [18:47:37] from=1760118200000&orgId=1&to=1760121980000 [18:49:25] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.19, 5.66, 7.47 [18:49:37] PROBLEM - db171 APT on db171 is CRITICAL: APT CRITICAL: 120 packages available for upgrade (1 critical updates). [18:49:59] PROBLEM - kafka181 APT on kafka181 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:51:40] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:51:55] PROBLEM - cloud17 APT on cloud17 is CRITICAL: APT CRITICAL: 185 packages available for upgrade (1 critical updates). [18:52:05] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:52:32] PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 185 packages available for upgrade (1 critical updates). [18:52:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760122357550[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760118470000&orgId=1&to=1760122070000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [18:52:37] =1760122357550[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760122357550[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760118590000&orgId=1&to=1760122310000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760118470000&orgId=1&to=1760122100000 [18:52:46] PROBLEM - db182 APT on db182 is CRITICAL: APT CRITICAL: 83 packages available for upgrade (1 critical updates). [18:53:17] PROBLEM - mw153 APT on mw153 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:53:22] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 9.43, 7.31, 7.71 [18:53:40] PROBLEM - mw191 APT on mw191 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:53:58] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [18:54:07] PROBLEM - cloud19 APT on cloud19 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:55:19] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.13, 5.32, 6.94 [18:56:06] PROBLEM - bots171 APT on bots171 is CRITICAL: APT CRITICAL: 116 packages available for upgrade (1 critical updates). [18:56:59] PROBLEM - rdb151 APT on rdb151 is CRITICAL: APT CRITICAL: 118 packages available for upgrade (1 critical updates). [18:57:14] PROBLEM - mw192 APT on mw192 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:57:17] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 5.73, 5.48, 6.80 [18:57:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760122657550[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760118890000&orgId=1&to=1760122490000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [18:57:37] =1760122657550[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760122657550[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760118950000&orgId=1&to=1760122657550[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/ [18:57:37] from=1760118860000&orgId=1&to=1760122640000 [18:57:39] PROBLEM - cp191 APT on cp191 is CRITICAL: APT CRITICAL: 110 packages available for upgrade (1 critical updates). [18:58:01] PROBLEM - changeprop201 APT on changeprop201 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:58:05] PROBLEM - mem191 APT on mem191 is CRITICAL: APT CRITICAL: 110 packages available for upgrade (1 critical updates). [18:58:09] PROBLEM - cp161 APT on cp161 is CRITICAL: APT CRITICAL: 120 packages available for upgrade (1 critical updates). [18:58:15] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:58:27] PROBLEM - os202 APT on os202 is CRITICAL: APT CRITICAL: 73 packages available for upgrade (1 critical updates). [18:58:29] PROBLEM - swiftobject171 APT on swiftobject171 is CRITICAL: APT CRITICAL: 74 packages available for upgrade (1 critical updates). [18:58:45] PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 112 packages available for upgrade (1 critical updates). [18:59:27] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 134 packages available for upgrade (3 critical updates). [18:59:36] PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 116 packages available for upgrade (1 critical updates). [18:59:58] PROBLEM - mw163 APT on mw163 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [19:00:09] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [19:01:20] PROBLEM - mattermost1 APT on mattermost1 is CRITICAL: APT CRITICAL: 113 packages available for upgrade (1 critical updates). [19:01:41] PROBLEM - db181 APT on db181 is CRITICAL: APT CRITICAL: 120 packages available for upgrade (1 critical updates). [19:01:59] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 115 packages available for upgrade (1 critical updates). [19:02:26] PROBLEM - cloud20 APT on cloud20 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (1 critical updates). [19:02:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760122957552[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760122957552[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1X [19:02:37] 60112060000&orgId=1&to=1760122957552[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760118950000&orgId=1&to=1760122880000[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760119190000&orgId=1&to=1760122957552 [19:02:43] PROBLEM - os191 APT on os191 is CRITICAL: APT CRITICAL: 110 packages available for upgrade (1 critical updates). [19:02:47] PROBLEM - reports171 APT on reports171 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (1 critical updates). [19:02:50] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [19:03:12] PROBLEM - swiftobject161 APT on swiftobject161 is CRITICAL: APT CRITICAL: 74 packages available for upgrade (1 critical updates). [19:03:12] PROBLEM - db151 APT on db151 is CRITICAL: APT CRITICAL: 128 packages available for upgrade (1 critical updates). [19:03:22] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 120 packages available for upgrade (3 critical updates). [19:03:23] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: APT CRITICAL: 74 packages available for upgrade (1 critical updates). [19:03:25] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 136 packages available for upgrade (3 critical updates). [19:03:51] PROBLEM - os161 APT on os161 is CRITICAL: APT CRITICAL: 110 packages available for upgrade (1 critical updates). [19:04:10] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [19:04:25] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:31] PROBLEM - mw201 APT on mw201 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [19:04:47] PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:04:50] PROBLEM - ldap171 APT on ldap171 is CRITICAL: APT CRITICAL: 110 packages available for upgrade (1 critical updates). [19:05:06] PROBLEM - bast181 APT on bast181 is CRITICAL: APT CRITICAL: 110 packages available for upgrade (1 critical updates). [19:05:11] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 14.43, 9.58, 7.99 [19:05:47] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [19:06:08] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [19:06:08] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.345 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [19:06:19] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [19:06:57] PROBLEM - mw183 APT on mw183 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [19:07:08] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.55, 7.25, 7.34 [19:07:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760123257553[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760123257553[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1X [19:07:37] 60112060000&orgId=1&to=1760123257553[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760119280000&orgId=1&to=1760123240000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760119550000&orgId=1&to=1760123240000 [19:08:13] PROBLEM - mem201 APT on mem201 is CRITICAL: APT CRITICAL: 118 packages available for upgrade (1 critical updates). [19:08:59] PROBLEM - matomo151 APT on matomo151 is CRITICAL: APT CRITICAL: 73 packages available for upgrade (1 critical updates). [19:09:23] PROBLEM - cp201 APT on cp201 is CRITICAL: APT CRITICAL: 81 packages available for upgrade (1 critical updates). [19:09:42] PROBLEM - mw202 APT on mw202 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [19:10:03] PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: APT CRITICAL: 127 packages available for upgrade (1 critical updates). [19:10:15] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: APT CRITICAL: 118 packages available for upgrade (1 critical updates). [19:10:36] PROBLEM - db172 APT on db172 is CRITICAL: APT CRITICAL: 93 packages available for upgrade (1 critical updates). [19:10:41] PROBLEM - eventgate181 APT on eventgate181 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:11:35] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [19:11:39] PROBLEM - swiftproxy171 APT on swiftproxy171 is CRITICAL: APT CRITICAL: 127 packages available for upgrade (1 critical updates). [19:12:02] PROBLEM - bast161 APT on bast161 is CRITICAL: APT CRITICAL: 110 packages available for upgrade (1 critical updates). [19:12:12] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 136 packages available for upgrade (3 critical updates). [19:12:37] PROBLEM - llm191 APT on llm191 is CRITICAL: APT CRITICAL: 47 packages available for upgrade (1 critical updates). [19:12:41] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [19:12:51] PROBLEM - swiftobject191 APT on swiftobject191 is CRITICAL: APT CRITICAL: 74 packages available for upgrade (1 critical updates). [19:12:51] PROBLEM - mw203 APT on mw203 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [19:13:05] PROBLEM - mem161 APT on mem161 is CRITICAL: APT CRITICAL: 118 packages available for upgrade (1 critical updates). [19:13:07] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [19:13:20] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 136 packages available for upgrade (3 critical updates). [19:13:29] PROBLEM - cp171 APT on cp171 is CRITICAL: APT CRITICAL: 110 packages available for upgrade (1 critical updates). [19:13:42] PROBLEM - swiftobject201 APT on swiftobject201 is CRITICAL: APT CRITICAL: 74 packages available for upgrade (1 critical updates). [19:14:35] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 136 packages available for upgrade (3 critical updates). [19:14:57] PROBLEM - phorge171 APT on phorge171 is CRITICAL: APT CRITICAL: 75 packages available for upgrade (1 critical updates). [19:15:00] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.64, 5.87, 6.68 [19:15:12] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 77 packages available for upgrade (1 critical updates). [19:15:16] PROBLEM - mw173 APT on mw173 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [19:15:53] PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: APT CRITICAL: 74 packages available for upgrade (1 critical updates). [19:16:02] PROBLEM - db161 APT on db161 is CRITICAL: APT CRITICAL: 128 packages available for upgrade (1 critical updates). [19:16:44] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: APT CRITICAL: 125 packages available for upgrade (1 critical updates). [19:18:02] PROBLEM - os201 APT on os201 is CRITICAL: APT CRITICAL: 82 packages available for upgrade (1 critical updates). [19:18:11] PROBLEM - mem151 APT on mem151 is CRITICAL: APT CRITICAL: 118 packages available for upgrade (1 critical updates). [19:18:25] PROBLEM - cloud16 APT on cloud16 is CRITICAL: APT CRITICAL: 185 packages available for upgrade (1 critical updates). [19:18:32] PROBLEM - cloud15 APT on cloud15 is CRITICAL: APT CRITICAL: 188 packages available for upgrade (1 critical updates). [19:18:37] PROBLEM - os151 APT on os151 is CRITICAL: APT CRITICAL: 112 packages available for upgrade (1 critical updates). [19:18:51] PROBLEM - mw193 APT on mw193 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [19:18:52] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:19:40] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 116 packages available for upgrade (1 critical updates). [19:20:45] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [19:22:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760124157554[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760124157554[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1X [19:22:37] 60112060000&orgId=1&to=1760124157554[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760120450000&orgId=1&to=1760124157554[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760120510000&orgId=1&to=1760124140000 [19:25:49] PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 27163MiB (5% inode=99%); [19:26:58] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:24] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [19:27:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760124457554[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760124457554[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1X [19:27:37] 60112060000&orgId=1&to=1760124457555[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760120600000&orgId=1&to=1760124410000 [19:29:46] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 11.38, 8.12, 6.78 [19:30:01] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:31:02] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:32:21] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 26 minutes ago with 0 failures [19:32:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760124757557[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760120990000&orgId=1&to=1760124757557[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/Gt [19:32:37] om=1760113260000&orgId=1&to=1760124757557[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760124757557[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760120600000&orgId=1&to=1760124757557 [19:32:59] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [19:33:02] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [19:33:36] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.069 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [19:33:42] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.83, 7.75, 7.10 [19:35:43] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 9.20, 7.93, 7.20 [19:37:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760125057557[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760121170000&orgId=1&to=1760124770000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [19:37:37] =1760125057557[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760125057557[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760121260000&orgId=1&to=1760125040000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760121170000&orgId=1&to=1760124950000 [19:37:41] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 5.71, 7.58, 7.19 [19:41:22] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:41:39] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 9.67, 7.91, 7.32 [19:42:49] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 19.12, 10.13, 5.14 [19:43:16] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [19:43:36] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 7.09, 7.92, 7.41 [19:43:48] PROBLEM - cp171 Disk Space on cp171 is CRITICAL: DISK CRITICAL - free space: / 27184MiB (5% inode=99%); [19:44:48] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 11.38, 9.77, 5.57 [19:45:35] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.70, 6.06, 6.78 [19:46:48] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.16, 6.77, 4.98 [19:49:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 7.30, 7.20, 7.08 [19:51:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 4.79, 5.88, 6.59 [19:52:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760125957558[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760125957558[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1X [19:52:37] 60112060000&orgId=1&to=1760125957558[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760121920000&orgId=1&to=1760125880000[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760122190000&orgId=1&to=1760125957559 [19:57:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760126257559[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760126257559[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1X [19:57:37] 60112060000&orgId=1&to=1760126257559[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760122550000&orgId=1&to=1760126257559[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760122190000&orgId=1&to=1760125970000 [19:59:42] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:00:20] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 10.94, 7.78, 7.03 [20:01:18] PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 27222MiB (5% inode=99%); [20:01:27] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:02:14] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:02:16] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:02:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760126557560[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760126557560[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1X [20:02:37] 60112060000&orgId=1&to=1760126557560[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760122550000&orgId=1&to=1760126540000 [20:04:19] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [20:04:22] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 26 minutes ago with 0 failures [20:05:24] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [20:05:55] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.112 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [20:07:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760126857560[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760123090000&orgId=1&to=1760126690000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [20:07:37] =1760126857560[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760126857560[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760122550000&orgId=1&to=1760126780000[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/G [20:07:38] rom=1760123090000&orgId=1&to=1760126857560 [20:09:40] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:10:17] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:11:16] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:12:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760127157560[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760127157560[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1X [20:12:37] 60112060000&orgId=1&to=1760127157560[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760123180000&orgId=1&to=1760127157560[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760123090000&orgId=1&to=1760126870000 [20:13:21] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:16:06] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [20:16:14] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 10 minutes ago with 0 failures [20:16:29] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.101 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [20:17:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760127457561[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760123570000&orgId=1&to=1760127410000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [20:17:37] =1760127457561[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760127457562[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760123180000&orgId=1&to=1760127457562[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by mo [20:17:37] 0 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760123810000&orgId=1&to=1760127457562 [20:17:49] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [20:20:51] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:22:05] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760127757562[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760123810000&orgId=1&to=1760127410000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [20:22:37] =1760127757562[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760127757562[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760123900000&orgId=1&to=1760127710000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760123810000&orgId=1&to=1760127590000 [20:25:04] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:25:09] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:27:20] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [20:27:22] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 21 minutes ago with 0 failures [20:27:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760128057563[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760124290000&orgId=1&to=1760128057563[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/Gt [20:27:37] om=1760113260000&orgId=1&to=1760128057563[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760128057563[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760123900000&orgId=1&to=1760128057563 [20:28:06] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [20:29:11] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.149 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [20:32:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760128357563[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760124470000&orgId=1&to=1760128070000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [20:32:37] =1760128357563[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760128357563[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760124590000&orgId=1&to=1760128357563[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/ [20:32:38] from=1760124470000&orgId=1&to=1760128280000 [20:33:53] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.66, 6.04, 7.56 [20:37:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760128657564[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760125010000&orgId=1&to=1760128610000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [20:37:37] =1760128657564[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760128657564[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760124860000&orgId=1&to=1760128610000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760124770000&orgId=1&to=1760128400000 [20:39:47] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 4.82, 5.03, 6.65 [20:42:18] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:42:32] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:43:43] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 9.91, 7.50, 7.28 [20:45:04] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:45:05] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:47:16] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [20:47:28] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 10 minutes ago with 0 failures [20:47:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760129257566[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760125550000&orgId=1&to=1760129150000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [20:47:37] =1760129257566[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760129257566[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760125100000&orgId=1&to=1760129180000[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/G [20:47:37] rom=1760125550000&orgId=1&to=1760129257566 [20:52:26] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [20:52:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760129557566[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760129557566[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1X [20:52:37] 60112060000&orgId=1&to=1760129557566[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760125670000&orgId=1&to=1760129557566[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760125940000&orgId=1&to=176 [20:52:59] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.069 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [20:53:35] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.41, 6.40, 7.29 [20:57:35] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.60, 5.32, 6.71 [20:57:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760129857567[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760126210000&orgId=1&to=1760129810000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [20:57:37] =1760129857567[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760129857567[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760126030000&orgId=1&to=1760129810000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760125940000&orgId=1&to=1760129720000 [21:02:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760130157568[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760126330000&orgId=1&to=1760129930000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [21:02:37] =1760130157568[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760130157568[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760126390000&orgId=1&to=1760130157568 [21:07:27] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 5.53, 7.12, 6.93 [21:07:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760130457569[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760130457569[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1X [21:07:37] 60112060000&orgId=1&to=1760130457569[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760126660000&orgId=1&to=1760130457569[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760126810000&orgId=1&to=176 [21:09:27] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.08, 6.96, 6.86 [21:11:25] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 5.69, 6.96, 6.90 [21:12:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760130757570[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760130757570[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1X [21:12:37] 60112060000&orgId=1&to=1760130757570[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760126930000&orgId=1&to=1760130757570[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760127080000&orgId=1&to=1760130740000 [21:13:24] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 6.50, 6.30, 6.63 [21:15:05] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:05] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [21:17:25] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 11.10, 8.31, 7.32 [21:19:02] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [21:19:11] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.115 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [21:19:23] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.79, 6.19, 6.68 [21:22:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760131357571[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760127470000&orgId=1&to=1760131130000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [21:22:37] =1760131357571[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760131357571[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760127590000&orgId=1&to=1760131357571[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by mo [21:22:37] 0 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760127740000&orgId=1&to=1760131357571 [21:27:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760131657571[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760128010000&orgId=1&to=1760131610000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [21:27:38] =1760131657571[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760131657571[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760127830000&orgId=1&to=1760131657571[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/ [21:27:38] from=1760128010000&orgId=1&to=1760131640000 [21:30:51] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [21:32:19] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:32:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760131957573[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760128010000&orgId=1&to=1760131610000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [21:32:37] =1760131957573[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760131957573[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760128100000&orgId=1&to=1760131910000 [21:33:03] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:33:05] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 12.38, 8.40, 7.11 [21:35:04] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:35:06] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 9.898 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [21:35:08] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 27 minutes ago with 0 failures [21:36:16] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [21:37:02] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 4.14, 8.00, 7.42 [21:37:08] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [21:37:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760132257574[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760128550000&orgId=1&to=1760132150000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [21:37:37] =1760132257574[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760132257574[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760128100000&orgId=1&to=1760132240000[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/G [21:37:37] rom=1760128550000&orgId=1&to=1760132257574 [21:39:01] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.15, 8.00, 7.47 [21:40:33] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:41:31] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [21:42:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760132557574[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760132557574[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1X [21:42:37] 60112060000&orgId=1&to=1760132557574[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760128640000&orgId=1&to=1760132540000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760128550000&orgId=1&to=1760132330000 [21:44:05] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:44:07] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:47:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760132857575[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760129030000&orgId=1&to=1760132857575[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/Gt [21:47:37] om=1760113260000&orgId=1&to=1760132857575[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760132857575[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760128640000&orgId=1&to=1760132810000 [21:52:18] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [21:52:20] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 15 minutes ago with 0 failures [21:52:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760133157575[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760129210000&orgId=1&to=1760132810000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [21:52:37] =1760133157576[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760133157576[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760129300000&orgId=1&to=1760133157576[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/ [21:52:37] from=1760129180000&orgId=1&to=1760132990000 [21:52:44] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [21:54:07] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.092 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [21:57:01] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:58:32] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [21:59:16] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:00:59] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [22:02:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760133757577[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760129930000&orgId=1&to=1760133530000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [22:02:37] =1760133757577[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760133757577[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760129660000&orgId=1&to=1760133650000[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/G [22:02:37] rom=1760130020000&orgId=1&to=1760133757577 [22:05:15] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:51] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.087 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [22:07:09] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [22:10:11] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [22:12:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760134357578[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760134357578[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1X [22:12:37] 60112060000&orgId=1&to=1760134357578[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760130440000&orgId=1&to=1760134340000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760130650000&orgId=1&to=1760134280000 [22:14:18] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:14:25] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:17:06] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:19:05] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:21:24] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [22:22:03] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 15 minutes ago with 0 failures [22:22:30] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [22:22:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760134957577[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131070000&orgId=1&to=1760134957577[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/Gt [22:22:37] om=1760113260000&orgId=1&to=1760134957577[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760134957577[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760130440000&orgId=1&to=1760134957577[Grafana] FIRING: The mediawiki JobQueue [22:22:37] increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131310000&orgId=1&to=1760134957578 [22:22:38] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.065 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [22:26:46] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:27:00] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:27:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760135257578[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131070000&orgId=1&to=1760134970000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [22:27:37] =1760135257578[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760135257578[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131400000&orgId=1&to=1760135210000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131310000&orgId=1&to=1760135090000 [22:28:58] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:29:58] !log c1: DROP DATABASE fallennwiki; [22:30:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:30:22] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:30:29] !log c4: DROP DATABASE fallennwiki; [22:30:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:32:27] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [22:32:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760135557579[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131790000&orgId=1&to=1760135557579[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/Gt [22:32:37] om=1760113260000&orgId=1&to=1760135557579[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760135557579[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131400000&orgId=1&to=1760135557579 [22:32:47] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [22:33:12] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.069 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [22:33:56] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 27 minutes ago with 0 failures [22:37:34] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:37:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760135857580[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131790000&orgId=1&to=1760135630000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [22:37:37] =1760135857580[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760135857580[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760132090000&orgId=1&to=1760135810000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760132000000&orgId=1&to=1760135780000 [22:39:32] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.087 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [22:42:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760136157580[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760132390000&orgId=1&to=1760135990000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [22:42:37] =1760136157580[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760136157580[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760132420000&orgId=1&to=1760136157580[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/ [22:42:37] from=1760132360000&orgId=1&to=1760136110000 [22:44:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.58, 5.83, 7.63 [22:47:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760136457581[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760132630000&orgId=1&to=1760136230000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [22:47:37] =1760136457581[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760136457581[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760132690000&orgId=1&to=1760136457581[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by mo [22:47:37] 0 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760132840000&orgId=1&to=1760136457581 [22:52:37] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760136757582[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760132870000&orgId=1&to=1760136470000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000& [22:52:37] =1760136757582[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=1760136757582[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760132930000&orgId=1&to=1760136757582[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/ [22:52:37] from=1760133110000&orgId=1&to=1760136740000 [22:52:48] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.25, 4.89, 6.60 [22:57:37] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760115710000&orgId=1&to=1760137010000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760113260000&orgId=1&to=1760137057583[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760112060000&orgId=1&to=176 [22:57:37] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760133200000&orgId=1&to=1760136950000 [22:59:26] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [23:00:47] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 10.62, 7.41, 6.84 [23:01:14] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:02:03] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:02:03] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:04:17] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [23:04:20] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 25 minutes ago with 0 failures [23:05:11] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [23:05:38] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.087 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [23:06:46] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 5.19, 7.67, 7.43 [23:07:46] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760133980000&orgId=1&to=1760137666514 [23:09:01] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.16, 0.03, 0.01 [23:10:20] !log rebooted prometheus151 as things seemed to be bad? high load, grafana not working with it etc and icinga errors about it [23:10:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:12:46] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760134190000&orgId=1&to=1760137790000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131260000&orgId=1&to=1760137966514[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131260000&orgId=1&to=1 [23:12:46] 14[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760134250000&orgId=1&to=1760137966514[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760134160000&orgId=1&to=1760137940000 [23:12:46] RECOVERY - mon181 APT on mon181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:13:21] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [23:14:54] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 9.30, 6.15, 2.67 [23:15:18] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.065 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [23:16:52] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.21, 4.81, 2.59 [23:17:46] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760134550000&orgId=1&to=1760138150000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131260000&orgId=1&to=1760138266516[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131260000&orgId=1&to=1 [23:17:46] 16[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760134490000&orgId=1&to=1760138210000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760134520000&orgId=1&to=1760138180000 [23:19:40] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [23:20:22] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:23:04] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:23:47] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 10.45, 8.33, 4.98 [23:24:52] !log increase prometheus151 ram to 10gb [23:24:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:25:11] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:27:11] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [23:27:21] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 12 minutes ago with 0 failures [23:27:42] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.67, 0.44, 0.15 [23:27:46] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760135210000&orgId=1&to=1760138866516[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131260000&orgId=1&to=1760138866516[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from [23:27:46] 0000&orgId=1&to=1760138866516[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760134640000&orgId=1&to=1760138866517 [23:27:51] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.057 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [23:28:18] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [23:32:46] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760135270000&orgId=1&to=1760138870000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131260000&orgId=1&to=1760139166521[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131260000&orgId=1&to=1 [23:32:46] 21[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760134640000&orgId=1&to=1760138960000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760135270000&orgId=1&to=1760138960000 [23:34:09] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/5fee6807a133588b9915ce04590e58c4f6a58c33 [23:34:09] 02puppet/03main 07paladox 035fee680 graylog: upgrade to 6.3.4 [23:52:46] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131260000&orgId=1&to=1760140260000[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131260000&orgId=1&to=1760140366523 [23:57:22] !log [paladox@mwtask181] starting deploy of {'config': True} to all [23:57:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:57:44] !log [paladox@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 22s [23:57:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log