[00:01:45] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/f5732c4f0361f39d841c1ab8ae6aba0ea257ee87 [00:01:47] 02statichelp/03main 07WikiTideBot 03f5732c4 Bot: Auto-update Tech namespace pages 2025-10-15 00:01:42 [00:04:27] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 5.16, 5.97, 9.64 [00:11:28] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 74260MiB (16% inode=99%); [00:13:17] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 74830MiB (16% inode=99%); [00:13:36] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 73841MiB (16% inode=99%); [03:49:46] !log [void@puppet181] reload nginx on mattermost, graylog, mon, and os servers [03:49:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:38:19] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [04:46:19] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [12:33:15] [02ssl] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/5225564480c9b4d606b9df3107a5488b8377724c [12:33:15] 02ssl/03main 07Skye 035225564 Remove pandorastale redirect [12:35:26] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/b18ef5ed6f511c5e2b717eccabaf45a4fd5e03ad [12:35:26] 02ssl/03main 07WikiTideBot 03b18ef5e Bot: Auto-update domain lists [14:08:17] PROBLEM - db182 Disk Space on db182 is WARNING: DISK WARNING - free space: / 52070MiB (10% inode=99%); [14:10:59] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/deleteBatch.php --wiki=regailiawiki --r '[[phab:T14395|Requested]]' /home/somerandomdeveloper/extra found templates 2.txt (START) [14:11:00] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/deleteBatch.php --wiki=regailiawiki --r '[[phab:T14395|Requested]]' /home/somerandomdeveloper/extra found templates 2.txt (END - exit=256) [14:11:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:11:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:11:22] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/deleteBatch.php --wiki=regailiawiki --r '[[phab:T14395|Requested]]' '/home/somerandomdeveloper/extra\ found\ templates\ 2.txt' (START) [14:11:23] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/deleteBatch.php --wiki=regailiawiki --r '[[phab:T14395|Requested]]' '/home/somerandomdeveloper/extra\ found\ templates\ 2.txt' (END - exit=256) [14:11:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:11:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:11:50] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/deleteBatch.php --wiki=regailiawiki --r '[[phab:T14395|Requested]]' /home/somerandomdeveloper/extraFoundTemplates2.txt (START) [14:11:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:12:24] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/deleteBatch.php --wiki=regailiawiki --r '[[phab:T14395|Requested]]' /home/somerandomdeveloper/extraFoundTemplates2.txt (END - exit=0) [14:12:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:26:30] miraheze/mw-config - SomeMWDev the build passed. [14:26:56] !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [14:27:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:27:06] !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 9s [14:27:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:27:38] !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151 [14:27:39] !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [14:27:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:27:43] miraheze/mw-config - SomeMWDev the build passed. [14:27:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:27:55] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [14:27:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:19] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s [14:28:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:45] !log [somerandomdeveloper@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php ManageWiki:PopulateWikiSettings --wiki=metawikibeta --setting=wgCommentStreamsEnableTalk --remove --all-wikis (END - exit=512) [14:28:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:58] !log [somerandomdeveloper@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php ManageWiki:PopulateWikiSettings --wiki=metawikibeta --setting=wgCommentStreamsEnableTalk --remove --all-wikis --execute (END - exit=0) [14:29:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:30:17] PROBLEM - db182 Disk Space on db182 is CRITICAL: DISK CRITICAL - free space: / 29273MiB (5% inode=99%); [14:38:55] miraheze/puppet - SomeMWDev the build passed. [15:00:32] RECOVERY - db182 Disk Space on db182 is OK: DISK OK - free space: / 133963MiB (26% inode=99%); [15:32:47] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php ManageWiki:PopulateWikiSettings --wiki=metawiki --setting=wgCommentStreamsEnableTalk --remove --all-wikis --execute (END - exit=0) [15:32:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:04:16] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.77, 17.94, 12.75 [16:04:58] !log increase db182 disk by 94gib [16:05:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:06:16] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 10.94, 16.45, 12.92 [16:17:59] [02puppet] 07Reception123 pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/444bb41d1c1836f4201a8471aaad045346554d9b [16:17:59] 02puppet/03main 07Reception123 03444bb41 set ai dryrun off (cw issue fixed) [16:39:46] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 87.81, 45.48, 20.16 [17:14:03] PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 176 bytes in 0.010 second response time [17:17:40] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 2.15, 6.84, 11.50 [17:18:03] PROBLEM - db182 MariaDB Connections on db182 is WARNING: Warning connection usage: 85.8%Current connections: 858 [17:20:04] PROBLEM - db182 MariaDB Connections on db182 is CRITICAL: Critical connection usage: 90%Current connections: 900 [17:21:40] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.08, 3.61, 9.10 [17:25:43] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/a73bcdc165c6389dfd9af8df49b2de62606cf643 [17:25:43] 02ssl/03main 07WikiTideBot 03a73bcdc Bot: Auto-update domain lists [17:26:03] PROBLEM - db182 MariaDB Connections on db182 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Too many connections in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.fsslc.wtn...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne [17:26:04] on line 66Fatal error: Uncaught mysqli_sql_exception: Too many connections in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.fsslc.wtn...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 PROBLEM - db182 MariaDB on db182 is CRITICAL: Too many connections [17:27:09] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4422 bytes in 0.023 second response time [17:28:17] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [17:37:13] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49794MiB (10% inode=99%); [17:59:40] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 7.53, 10.60, 7.21 [18:01:40] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.83, 7.37, 6.43 [18:10:03] RECOVERY - db182 MariaDB Connections on db182 is OK: OK connection usage: 2.1%Current connections: 21 [18:10:17] RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17668 bytes in 0.040 second response time [18:11:01] RECOVERY - db182 MariaDB on db182 is OK: Uptime: 1131576 Threads: 19 Questions: 3351807101 Slow queries: 7248 Opens: 1141897 Open tables: 2005 Queries per second avg: 2962.069 [18:11:09] RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19375 bytes in 0.066 second response time [18:11:10] RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.197 second response time [18:13:51] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49910MiB (10% inode=99%); [18:23:40] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49837MiB (10% inode=99%); [18:38:24] [02puppet] 07paladox created 03paladox-patch-3 (+1 new commit) 13https://github.com/miraheze/puppet/commit/026ad90d423c [18:38:24] 02puppet/03paladox-patch-3 07paladox 03026ad90 matomo: set enable_sql_optimize_queries to 0 (disable) [18:38:28] [02puppet] 07paladox opened pull request #4568: matomo: set enable_sql_optimize_queries to 0 (disable) (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4568 [18:38:53] [02puppet] 07coderabbitai[bot] commented on pull request #4568:
[…] 13https://github.com/miraheze/puppet/pull/4568#issuecomment-3407780734 [18:38:56] [02puppet] 07paladox merged pull request #4568: matomo: set enable_sql_optimize_queries to 0 (disable) (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4568 [18:38:56] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/eb2e46b6eb46fed1549aab41cb6b012b45bbf70e [18:38:58] 02puppet/03main 07paladox 03eb2e46b matomo: set enable_sql_optimize_queries to 0 (disable) (#4568)… [18:39:00] [02puppet] 07paladox 04deleted 03paladox-patch-3 at 03026ad90 13https://api.github.com/repos/miraheze/puppet/commit/026ad90 [18:47:26] [02puppet] 07paladox created 03paladox-patch-3 (+1 new commit) 13https://github.com/miraheze/puppet/commit/cbf0c9ec90b0 [18:47:26] 02puppet/03paladox-patch-3 07paladox 03cbf0c9e matomo: increase apc.shm_size to 512M [18:47:31] [02puppet] 07paladox opened pull request #4569: matomo: increase apc.shm_size to 512M (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4569 [18:47:39] [02puppet] 07paladox merged pull request #4569: matomo: increase apc.shm_size to 512M (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4569 [18:47:41] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/116726205b29e6b343f5a31eebdc8acc68c69e3d [18:47:43] [02puppet] 07paladox 04deleted 03paladox-patch-3 at 03cbf0c9e 13https://api.github.com/repos/miraheze/puppet/commit/cbf0c9e [18:47:44] 02puppet/03main 07paladox 031167262 matomo: increase apc.shm_size to 512M (#4569) [19:08:31] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [19:16:31] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [21:23:40] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.61, 13.07, 6.97 [21:25:29] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/91b92d1fe2913c26f05e28e5db7236502f5188ab [21:25:29] 02ssl/03main 07WikiTideBot 0391b92d1 Bot: Auto-update domain lists [21:25:40] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.84, 9.07, 6.24 [21:35:29] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/65d447c8a74995a86ed126cb13b076bb8d071c82 [21:35:29] 02ssl/03main 07WikiTideBot 0365d447c Bot: Auto-update domain lists [22:18:47] !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'VisualEditor'} to test151 [22:18:51] !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'VisualEditor'} to test151 - SUCCESS in 3s [22:19:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:19:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:21:40] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 7.54, 12.39, 8.27 [22:23:40] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 4.42, 10.68, 8.23 [22:25:40] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.76, 7.57, 7.38 [22:31:12] [02puppet] 07paladox created 03revert-4568-paladox-patch-3 (+1 new commit) 13https://github.com/miraheze/puppet/commit/83cbb4347e0f [22:31:12] 02puppet/03revert-4568-paladox-patch-3 07paladox 0383cbb43 Revert "matomo: set enable_sql_optimize_queries to 0 (disable) (#4568)"… [22:31:40] [02puppet] 07paladox opened pull request #4570: Revert "matomo: set enable_sql_optimize_queries to 0 (disable)" (03main...03revert-4568-paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4570 [22:31:50] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/383014dea1484cc57853fb162ef29edf1b43aed7 [22:31:52] 02puppet/03main 07paladox 03383014d Revert "matomo: set enable_sql_optimize_queries to 0 (disable)" (#4570)… [22:31:54] [02puppet] 07paladox merged pull request #4570: Revert "matomo: set enable_sql_optimize_queries to 0 (disable)" (03main...03revert-4568-paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4570 [22:31:56] [02puppet] 07paladox 04deleted 03revert-4568-paladox-patch-3 at 0383cbb43 13https://api.github.com/repos/miraheze/puppet/commit/83cbb43 [22:31:58] [02puppet] 07coderabbitai[bot] commented on pull request #4570:
[…] 13https://github.com/miraheze/puppet/pull/4570#issuecomment-3408535013 [22:36:42] PROBLEM - db182 Disk Space on db182 is WARNING: DISK WARNING - free space: / 58027MiB (9% inode=99%); [22:40:39] RECOVERY - db182 Disk Space on db182 is OK: DISK OK - free space: / 234678MiB (39% inode=99%); [22:55:26] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/81c32cd1078a7efc6f3ece1ed8a9c70c328034dd [22:55:27] 02ssl/03main 07WikiTideBot 0381c32cd Bot: Auto-update domain lists