[00:01:33] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/f39d28bd1b5d0f0281efc14d3ba8216d8411eb5e [00:01:33] 02statichelp/03main 07WikiTideBot 03f39d28b Bot: Auto-update Tech namespace pages 2025-10-11 00:01:30 [00:02:46] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760131260000&orgId=1&to=1760140860000[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760137130000&orgId=1&to=1760140760000 [00:06:08] !log [paladox@mwtask181] starting deploy of {'config': True} to all [00:06:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:06:31] !log [paladox@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 22s [00:06:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:11:18] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 55134MiB (12% inode=99%); [00:11:48] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 54571MiB (12% inode=99%); [00:11:49] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 54236MiB (11% inode=99%); [01:22:26] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.41, 18.63, 10.92 [01:28:26] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 10.79, 19.08, 14.12 [03:17:49] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49785MiB (10% inode=99%); [03:23:12] hey [03:23:16] I kinda needhelp [03:23:20] I wanna delete my wiki [03:23:53] uhh [03:23:56] ya [03:26:41] I need to delete my wiki [03:27:01] https://etherealendpoint.miraheze.org/wiki/Main_Page [03:35:48] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49818MiB (10% inode=99%); [03:39:20] [02puppet] 07Universal-Omega approved pull request #4556 13https://github.com/miraheze/puppet/pull/4556#pullrequestreview-3326589451 [03:40:40] [02puppet] 07Universal-Omega approved pull request #4558: Concept seems fine not sure on the potential issues. But overall looks good to me. 13https://github.com/miraheze/puppet/pull/4558#pullrequestreview-3326590995 [03:41:22] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/59615430bf923cb00df686bc6cb597abdb33272e [03:41:22] 02puppet/03main 07dependabot[bot] 035961543 build(deps-dev): bump rexml from 3.3.9 to 3.4.2 in /modules/graylog (#4526)… [03:41:22] [02puppet] 07Universal-Omega merged 07dependabot[bot]'s pull request #4526: build(deps-dev): bump rexml from 3.3.9 to 3.4.2 in /modules/graylog (03main...03dependabot/bundler/modules/graylog/rexml-3.4.2) 13https://github.com/miraheze/puppet/pull/4526 [03:41:23] [02puppet] 07Universal-Omega 04deleted 03dependabot/bundler/modules/graylog/rexml-3.4.2 at 031ff69c9 13https://api.github.com/repos/miraheze/puppet/commit/1ff69c9 [03:41:45] [02mw-config] 07Universal-Omega merged 07dependabot[bot]'s pull request #6140: Update phpunit/phpunit requirement from 12.4.0 to 12.4.1 (03main...03dependabot/composer/phpunit/phpunit-12.4.1) 13https://github.com/miraheze/mw-config/pull/6140 [03:41:46] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/a558f98219fb82406a5209d82a4b3a6e97482545 [03:41:46] 02mw-config/03main 07dependabot[bot] 03a558f98 Update phpunit/phpunit requirement from 12.4.0 to 12.4.1 (#6140)… [03:41:47] [02mw-config] 07Universal-Omega 04deleted 03dependabot/composer/phpunit/phpunit-12.4.1 at 03a081109 13https://api.github.com/repos/miraheze/mw-config/commit/a081109 [03:42:36] miraheze/mw-config - Universal-Omega the build passed. [03:42:46] [02mw-config] 07dependabot[bot] 04force-pushed 03dependabot/composer/justinrainbow/json-schema-6.6.0 from 031c60741 to 03936e182 13https://api.github.com/repos/miraheze/mw-config/commits/dependabot/composer/justinrainbow/json-schema-6.6.0 [03:42:46] 02mw-config/03dependabot/composer/justinrainbow/json-schema-6.6.0 07dependabot[bot] 03936e182 Update justinrainbow/json-schema requirement from 6.5.2 to 6.6.0… [03:43:32] [02mw-config] 07Universal-Omega merged 07dependabot[bot]'s pull request #6141: Update justinrainbow/json-schema requirement from 6.5.2 to 6.6.0 (03main...03dependabot/composer/justinrainbow/json-schema-6.6.0) 13https://github.com/miraheze/mw-config/pull/6141 [03:43:32] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/037da7a19113c76d7f17da457348f9a5800ebe20 [03:43:34] 02mw-config/03main 07dependabot[bot] 03037da7a Update justinrainbow/json-schema requirement from 6.5.2 to 6.6.0 (#6141)… [03:43:36] [02mw-config] 07Universal-Omega 04deleted 03dependabot/composer/justinrainbow/json-schema-6.6.0 at 03936e182 13https://api.github.com/repos/miraheze/mw-config/commit/936e182 [03:43:48] miraheze/mw-config - dependabot[bot] the build passed. [03:44:28] miraheze/mw-config - Universal-Omega the build passed. [03:53:18] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49915MiB (10% inode=99%); [08:45:54] PROBLEM - mem151 APT on mem151 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [08:47:57] PROBLEM - mem151 APT on mem151 is CRITICAL: APT CRITICAL: 118 packages available for upgrade (1 critical updates). [17:02:00] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760198480000&orgId=1&to=1760202120478 [17:08:05] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:08:46] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 10.36, 5.73, 2.62 [17:08:46] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [17:11:00] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:11:16] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:12:00] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760198720000&orgId=1&to=1760202710000 [17:12:07] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [17:12:32] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760199060000&orgId=1&to=1760202752513 [17:13:29] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [17:13:31] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 37 minutes ago with 0 failures [17:14:46] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 5.55, 7.76, 4.71 [17:14:58] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.064 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [17:16:46] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.92, 5.28, 4.17 [17:17:32] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760199290000&orgId=1&to=1760202890000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760199060000&orgId=1&to=1760203052516[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760199060000&orgId=1&to=1 [17:17:32] 16[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760198720000&orgId=1&to=1760202980000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760199290000&orgId=1&to=1760202980000 [17:19:49] PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 27106MiB (5% inode=99%); [17:37:48] PROBLEM - cp171 Disk Space on cp171 is CRITICAL: DISK CRITICAL - free space: / 27204MiB (5% inode=99%); [17:47:18] PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 27208MiB (5% inode=99%); [17:52:32] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760199060000&orgId=1&to=1760205060000[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760199060000&orgId=1&to=1760205060000 [18:11:29] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 9.46, 11.46, 5.89 [18:13:29] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.90, 7.90, 5.25 [19:55:21] !log [void@puppet181] Upgraded packages on bots171: libtiff6 [19:55:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:34] !log [void@puppet181] Upgraded packages on changeprop201: libtiff6 [19:55:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:47] !log [void@puppet181] Upgraded packages on db172: libtiff6 [19:55:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:58] RECOVERY - changeprop201 APT on changeprop201 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:56:01] !log [void@puppet181] Upgraded packages on cloud16: libtiff6 [19:56:05] RECOVERY - bots171 APT on bots171 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [19:56:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:14] !log [void@puppet181] Upgraded packages on cp161: libtiff6 [19:56:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:21] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 184 packages available for upgrade (0 critical updates). [19:56:27] !log [void@puppet181] Upgraded packages on cp191: libtiff6 [19:56:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:34] RECOVERY - db172 APT on db172 is OK: APT OK: 92 packages available for upgrade (0 critical updates). [19:56:42] !log [void@puppet181] Upgraded packages on cloud15: libtiff6 [19:56:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:56] !log [void@puppet181] Upgraded packages on cloud17: libtiff6 [19:56:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:09] !log [void@puppet181] Upgraded packages on db151: libtiff6 [19:57:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:20] RECOVERY - db151 APT on db151 is OK: APT OK: 127 packages available for upgrade (0 critical updates). [19:57:24] !log [void@puppet181] Upgraded packages on bast181: libtiff6 [19:57:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:35] RECOVERY - cp191 APT on cp191 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [19:57:38] !log [void@puppet181] Upgraded packages on cloud20: libtiff6 [19:57:40] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 184 packages available for upgrade (0 critical updates). [19:57:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:53] !log [void@puppet181] Upgraded packages on cp171: libtiff6 [19:57:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:04] !log [void@puppet181] Upgraded packages on cloud19: libtiff6 [19:58:05] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 187 packages available for upgrade (0 critical updates). [19:58:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:09] RECOVERY - cp161 APT on cp161 is OK: APT OK: 119 packages available for upgrade (0 critical updates). [19:58:20] !log [void@puppet181] Upgraded packages on cloud18: libtiff6 [19:58:23] RECOVERY - cloud20 APT on cloud20 is OK: APT OK: 188 packages available for upgrade (0 critical updates). [19:58:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:29] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 184 packages available for upgrade (0 critical updates). [19:58:35] !log [void@puppet181] Upgraded packages on db161: libtiff6 [19:58:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:50] !log [void@puppet181] Upgraded packages on db171: libtiff6 [19:58:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:03] RECOVERY - bast181 APT on bast181 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [19:59:06] !log [void@puppet181] Upgraded packages on db181: libtiff6 [19:59:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:12] RECOVERY - db171 APT on db171 is OK: APT OK: 119 packages available for upgrade (0 critical updates). [19:59:20] !log [void@puppet181] Upgraded packages on eventgate181: libtiff6 [19:59:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:33] !log [void@puppet181] Upgraded packages on cp201: libtiff6 [19:59:35] RECOVERY - db181 APT on db181 is OK: APT OK: 119 packages available for upgrade (0 critical updates). [19:59:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:44] RECOVERY - cp171 APT on cp171 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [19:59:47] !log [void@puppet181] Upgraded packages on db182: libtiff6 [19:59:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:54] RECOVERY - cloud19 APT on cloud19 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:59:59] RECOVERY - db161 APT on db161 is OK: APT OK: 127 packages available for upgrade (0 critical updates). [20:00:01] !log [void@puppet181] Upgraded packages on ldap171: libtiff6 [20:00:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:00:15] !log [void@puppet181] Upgraded packages on graylog161: libtiff6 [20:00:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:00:27] !log [void@puppet181] Upgraded packages on kafka181: libtiff6 [20:00:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:00:39] RECOVERY - eventgate181 APT on eventgate181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:00:40] !log [void@puppet181] Upgraded packages on llm191: libtiff6 [20:00:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:00:46] RECOVERY - db182 APT on db182 is OK: APT OK: 82 packages available for upgrade (0 critical updates). [20:00:55] !log [void@puppet181] Upgraded packages on bast161: libtiff6 [20:00:55] RECOVERY - ldap171 APT on ldap171 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [20:00:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:01:06] !log [void@puppet181] Upgraded packages on mattermost1: libtiff6 [20:01:06] RECOVERY - bast161 APT on bast161 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [20:01:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:01:18] !log [void@puppet181] Upgraded packages on matomo151: libtiff6 [20:01:19] RECOVERY - mattermost1 APT on mattermost1 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [20:01:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:01:22] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 72 packages available for upgrade (0 critical updates). [20:01:22] RECOVERY - cp201 APT on cp201 is OK: APT OK: 80 packages available for upgrade (0 critical updates). [20:01:31] !log [void@puppet181] Upgraded packages on mem161: libtiff6 [20:01:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:01:39] RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 114 packages available for upgrade (0 critical updates). [20:01:45] !log [void@puppet181] Upgraded packages on mem151: libtiff6 [20:01:47] RECOVERY - llm191 APT on llm191 is OK: APT OK: 46 packages available for upgrade (0 critical updates). [20:01:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:01:58] !log [void@puppet181] Upgraded packages on mem201: libtiff6 [20:01:58] RECOVERY - kafka181 APT on kafka181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:02:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:10] RECOVERY - mem201 APT on mem201 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [20:02:16] !log [void@puppet181] Upgraded packages on mw151: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:02:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:33] RECOVERY - mw151 APT on mw151 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [20:02:36] !log [void@puppet181] Upgraded packages on mw152: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:02:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:51] !log [void@puppet181] Upgraded packages on mem191: libtiff6 [20:02:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:03] RECOVERY - mem161 APT on mem161 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [20:03:14] !log [void@puppet181] Upgraded packages on mw162: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:03:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:26] RECOVERY - mem191 APT on mem191 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [20:03:33] RECOVERY - mem151 APT on mem151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [20:03:33] !log [void@puppet181] Upgraded packages on mw172: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:03:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:51] !log [void@puppet181] Upgraded packages on mw153: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:03:54] RECOVERY - mw162 APT on mw162 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:03:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:05] RECOVERY - mw152 APT on mw152 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:04:07] RECOVERY - mw172 APT on mw172 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:04:10] !log [void@puppet181] Upgraded packages on mw161: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:04:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:29] !log [void@puppet181] Upgraded packages on mw171: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:04:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:36] RECOVERY - mw153 APT on mw153 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:04:48] !log [void@puppet181] Upgraded packages on mw163: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:04:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:07] !log [void@puppet181] Upgraded packages on mw173: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:05:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:12] RECOVERY - mw173 APT on mw173 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:05:18] RECOVERY - mw161 APT on mw161 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:05:25] !log [void@puppet181] Upgraded packages on mw201: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:05:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:43] !log [void@puppet181] Upgraded packages on mw193: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:05:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:58] RECOVERY - mw163 APT on mw163 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:06:03] !log [void@puppet181] Upgraded packages on mw181: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:06:07] RECOVERY - mw193 APT on mw193 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:06:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:16] RECOVERY - mw171 APT on mw171 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:06:22] !log [void@puppet181] Upgraded packages on mw192: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:06:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:40] !log [void@puppet181] Upgraded packages on mw202: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:06:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:54] RECOVERY - mw201 APT on mw201 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:07:00] !log [void@puppet181] Upgraded packages on mw182: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:07:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:07:18] !log [void@puppet181] Upgraded packages on mw191: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:07:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:07:28] RECOVERY - mw202 APT on mw202 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:07:37] !log [void@puppet181] Upgraded packages on mw183: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:07:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:07:43] RECOVERY - mw191 APT on mw191 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:07:52] RECOVERY - mw181 APT on mw181 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:07:56] !log [void@puppet181] Upgraded packages on mwtask151: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:08:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:08:09] RECOVERY - mw192 APT on mw192 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:08:17] !log [void@puppet181] Upgraded packages on mw203: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:08:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:08:34] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 133 packages available for upgrade (0 critical updates). [20:08:36] !log [void@puppet181] Upgraded packages on mwtask161: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:08:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:08:47] RECOVERY - mw203 APT on mw203 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:08:52] RECOVERY - mw182 APT on mw182 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:08:53] RECOVERY - mw183 APT on mw183 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:08:55] !log [void@puppet181] Upgraded packages on mwtask171: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:08:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:09:14] !log [void@puppet181] Upgraded packages on ns1: libtiff6 [20:09:16] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 133 packages available for upgrade (0 critical updates). [20:09:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:09:32] !log [void@puppet181] Upgraded packages on mwtask181: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:09:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:09:44] !log [void@puppet181] Upgraded packages on phorge171: libtiff6 [20:09:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:09:58] !log [void@puppet181] Upgraded packages on os151: libtiff6 [20:10:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:13] !log [void@puppet181] Upgraded packages on os162: libtiff6 [20:10:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:27] !log [void@puppet181] Upgraded packages on os191: libtiff6 [20:10:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:37] RECOVERY - os162 APT on os162 is OK: APT OK: 111 packages available for upgrade (0 critical updates). [20:10:39] RECOVERY - os151 APT on os151 is OK: APT OK: 111 packages available for upgrade (0 critical updates). [20:10:40] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 133 packages available for upgrade (0 critical updates). [20:10:40] RECOVERY - os191 APT on os191 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [20:10:42] !log [void@puppet181] Upgraded packages on os201: libtiff6 [20:10:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:57] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 74 packages available for upgrade (0 critical updates). [20:10:57] !log [void@puppet181] Upgraded packages on rdb151: libtiff6 [20:11:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:10] !log [void@puppet181] Upgraded packages on prometheus151: libtiff6 [20:11:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:17] RECOVERY - ns1 APT on ns1 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [20:11:18] RECOVERY - os201 APT on os201 is OK: APT OK: 81 packages available for upgrade (0 critical updates). [20:11:23] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 133 packages available for upgrade (0 critical updates). [20:11:25] !log [void@puppet181] Upgraded packages on os161: libtiff6 [20:11:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:33] RECOVERY - rdb151 APT on rdb151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [20:11:44] !log [void@puppet181] Upgraded packages on os202: libtiff6 [20:11:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:57] !log [void@puppet181] Upgraded packages on reports171: libtiff6 [20:12:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:12:11] !log [void@puppet181] Upgraded packages on puppet181: libtiff6 [20:12:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:12:15] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [20:12:27] !log [void@puppet181] Upgraded packages on swiftac171: libtiff6 [20:12:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:12:41] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 124 packages available for upgrade (0 critical updates). [20:12:42] !log [void@puppet181] Upgraded packages on swiftobject161: libtiff6 [20:12:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:12:46] RECOVERY - reports171 APT on reports171 is OK: APT OK: 113 packages available for upgrade (0 critical updates). [20:12:57] !log [void@puppet181] Upgraded packages on swiftobject151: libtiff6 [20:13:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:13:01] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [20:13:12] !log [void@puppet181] Upgraded packages on swiftobject191: libtiff6 [20:13:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:13:23] RECOVERY - os202 APT on os202 is OK: APT OK: 72 packages available for upgrade (0 critical updates). [20:13:27] !log [void@puppet181] Upgraded packages on swiftobject201: libtiff6 [20:13:29] RECOVERY - os161 APT on os161 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [20:13:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:13:40] RECOVERY - swiftobject201 APT on swiftobject201 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [20:13:43] !log [void@puppet181] Upgraded packages on swiftobject171: libtiff6 [20:13:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:13:51] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [20:14:00] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [20:14:01] !log [void@puppet181] Upgraded packages on test151: libtiff-dev, libtiffxx6, libtiff6, ghostscript, libgs10, libgs10-common, and libgs-common [20:14:09] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 76 packages available for upgrade (0 critical updates). [20:14:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:14:18] !log [void@puppet181] Upgraded packages on swiftobject181: libtiff6 [20:14:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:14:32] RECOVERY - test151 APT on test151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [20:14:33] !log [void@puppet181] Upgraded packages on swiftproxy171: libtiff6 [20:14:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:14:50] !log [void@puppet181] Upgraded packages on swiftproxy161: libtiff6 [20:14:54] RECOVERY - swiftobject191 APT on swiftobject191 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [20:14:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:15:15] !log [void@puppet181] Upgraded packages on ns2: libtiff6 [20:15:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:15:36] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 126 packages available for upgrade (0 critical updates). [20:15:42] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 126 packages available for upgrade (0 critical updates). [20:15:47] RECOVERY - ns2 APT on ns2 is OK: APT OK: 114 packages available for upgrade (0 critical updates). [20:15:51] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [21:28:51] !log increase prometheus151 ram to 12gb [21:28:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:29:00] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760214500000&orgId=1&to=1760218139942 [21:33:08] Icinga: go away [21:34:00] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760214500000&orgId=1&to=1760218190000 [21:38:09] [02puppet] 07paladox created 03paladox-patch-8 (+1 new commit) 13https://github.com/miraheze/puppet/commit/921fdb65e8ef [21:38:09] 02puppet/03paladox-patch-8 07paladox 03921fdb6 prometheus::statsd_exporter: Add support for ttl… [21:38:14] [02puppet] 07paladox opened pull request #4561: prometheus::statsd_exporter: Add support for ttl (03main...03paladox-patch-8) 13https://github.com/miraheze/puppet/pull/4561 [21:38:25] [02puppet] 07coderabbitai[bot] commented on pull request #4561:

[…] 13https://github.com/miraheze/puppet/pull/4561#issuecomment-3393674956 [21:39:46] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-8 13https://github.com/miraheze/puppet/commit/6603a557e9dfa95b71d254ec250fab4cb381c9dd [21:39:46] 02puppet/03paladox-patch-8 07paladox 036603a55 Update statsd_exporter.pp [21:41:27] [02puppet] 07paladox merged pull request #4561: prometheus::statsd_exporter: Add support for ttl (03main...03paladox-patch-8) 13https://github.com/miraheze/puppet/pull/4561 [21:41:28] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/f67a2a8926993936f89fa34b42829324c69c9044 [21:41:29] 02puppet/03main 07paladox 03f67a2a8 prometheus::statsd_exporter: Add support for ttl (#4561)… [21:41:31] [02puppet] 07paladox 04deleted 03paladox-patch-8 at 036603a55 13https://api.github.com/repos/miraheze/puppet/commit/6603a55 [22:00:29] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760216390000&orgId=1&to=1760220029172 [22:05:29] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760216390000&orgId=1&to=1760220050000 [22:24:46] !log increase prometheus161 disk by 46gib [22:24:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:25:00] !log correction prometheus151 [22:25:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:14:49] !log reduce swiftproxy161 cpus to 4 to match swiftproxy171 [23:14:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:17:48] PROBLEM - swiftproxy171 Check unit status of swift_dispersion_stats on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift_dispersion_stats [23:25:48] RECOVERY - swiftproxy171 Check unit status of swift_dispersion_stats on swiftproxy171 is OK: OK: Status of the systemd unit swift_dispersion_stats [23:32:33] [02puppet] 07paladox created 03paladox-patch-8 (+1 new commit) 13https://github.com/miraheze/puppet/commit/6233214ace73 [23:32:33] 02puppet/03paladox-patch-8 07paladox 036233214 Lower role::prometheus::statsd_exporter::ttl to 360h… [23:32:37] [02puppet] 07paladox opened pull request #4562: Lower role::prometheus::statsd_exporter::ttl to 360h (03main...03paladox-patch-8) 13https://github.com/miraheze/puppet/pull/4562 [23:32:45] [02puppet] 07paladox merged pull request #4562: Lower role::prometheus::statsd_exporter::ttl to 360h (03main...03paladox-patch-8) 13https://github.com/miraheze/puppet/pull/4562 [23:32:47] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/7c7dcd8325e1ef5cdde01a644d3a19c7f368a927 [23:32:49] [02puppet] 07paladox 04deleted 03paladox-patch-8 at 036233214 13https://api.github.com/repos/miraheze/puppet/commit/6233214 [23:32:50] 02puppet/03main 07paladox 037c7dcd8 Lower role::prometheus::statsd_exporter::ttl to 360h (#4562)… [23:32:58] [02puppet] 07coderabbitai[bot] commented on pull request #4562:

[…] 13https://github.com/miraheze/puppet/pull/4562#issuecomment-3393733907 [23:33:45] [02puppet] 07paladox closed pull request #4532: mediawiki: set mediawiki::php::fpm::fpm_workers_multiplier to 2.5 (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4532 [23:33:47] [02puppet] 07paladox 04deleted 03paladox-patch-3 at 03fe5f0d2 13https://api.github.com/repos/miraheze/puppet/commit/fe5f0d2 [23:35:12] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/b7a4478a9f32e20c3bb7d573a799c7940eb0ea25 [23:35:13] 02puppet/03main 07paladox 03b7a4478 Fix typo