[00:01:22] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/a5629bb8970f3b1660117fcff88f40d330f85f13 [00:01:22] 02statichelp/03main 07WikiTideBot 03a5629bb Bot: Auto-update Tech namespace pages 2025-11-12 00:01:19 [00:05:47] [02CreateWiki] 07paladox left a file comment in pull request #787 03650e700: Need to add a use function at the top of file. 13https://github.com/miraheze/CreateWiki/pull/787#discussion_r2516231754 [00:08:18] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 65473MiB (14% inode=99%); [00:14:21] [02CreateWiki] 07SomeMWDev left a file comment in pull request #787 03650e700: Will do that tomorrow since it isn't necessary for it to work on prod. function_exists is not a clean solution here anyway 13https://github.com/miraheze/CreateWiki/pull/787#discussion_r2516242468 [00:17:42] [02mw-config] 07SomeMWDev created 03invalidate-opcache (+1 new commit) 13https://github.com/miraheze/mw-config/commit/bd3b8293076d [00:17:42] 02mw-config/03invalidate-opcache 07SomeRandomDeveloper 03bd3b829 Invalidate opcache after writing config cache [00:19:32] [02mw-config] 07SomeMWDev opened pull request #6174: Invalidate opcache after writing config cache (03main...03invalidate-opcache) 13https://github.com/miraheze/mw-config/pull/6174 [00:19:40] [02mw-config] 07SomeMWDev retitled PR #6174: "Invalidate opcache after writing config cache" ➜ "Invalidate opcache after writing to config cache" 13https://github.com/miraheze/mw-config/pull/6174 [00:20:29] miraheze/mw-config - SomeMWDev the build passed. [00:21:43] [02mw-config] 07paladox approved pull request #6174 13https://github.com/miraheze/mw-config/pull/6174#pullrequestreview-3450706216 [00:24:06] [02ManageWiki] 07SomeMWDev created 03invalidate-opcache (+1 new commit) 13https://github.com/miraheze/ManageWiki/commit/59ac14ca912d [00:24:06] 02ManageWiki/03invalidate-opcache 07SomeRandomDeveloper 0359ac14c Invalidate opcache after writing to cache file [00:24:21] [02ManageWiki] 07SomeMWDev opened pull request #750: Invalidate opcache after writing to cache file (03main...03invalidate-opcache) 13https://github.com/miraheze/ManageWiki/pull/750 [00:24:26] [02ManageWiki] 07coderabbitai[bot] commented on pull request #750:
[…] 13https://github.com/miraheze/ManageWiki/pull/750#issuecomment-3519308813 [00:24:38] [02mw-config] 07SomeMWDev merged pull request #6174: Invalidate opcache after writing to config cache (03main...03invalidate-opcache) 13https://github.com/miraheze/mw-config/pull/6174 [00:24:38] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/a7e56b0d35f076449aa642d833270048a11d07ca [00:24:38] 02mw-config/03main 07SomeRandomDeveloper 03a7e56b0 Invalidate opcache after writing to config cache (#6174) [00:24:39] [02mw-config] 07SomeMWDev 04deleted 03invalidate-opcache at 03bd3b829 13https://api.github.com/repos/miraheze/mw-config/commit/bd3b829 [00:25:04] !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [00:25:05] !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [00:25:07] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [00:25:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:25:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:25:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:25:29] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s [00:25:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:25:34] miraheze/mw-config - SomeMWDev the build passed. [00:27:07] !log [somerandomdeveloper@mwtask181] starting deploy of {'folders': '1.44/extensions/ManageWiki'} to all [00:27:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:28] !log [somerandomdeveloper@mwtask181] finished deploy of {'folders': '1.44/extensions/ManageWiki'} to all - SUCCESS in 21s [00:27:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:34] !log deploy ManageWiki commit 59ac14ca912d5cda8e14f5a14bce11ebaddc6b23 to prod [00:27:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:12] miraheze/ManageWiki - SomeMWDev the build has errored. [01:48:11] !log run sql of [[Tech:Fixing slot roles and content models]] on wingliscwiki [01:48:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:55:18] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/3811c699f2e1f812c2f24957fc111076a541eab3 [04:55:18] 02ssl/03main 07WikiTideBot 033811c69 Bot: Auto-update domain lists [10:06:05] PROBLEM - www.mwcosmos.com - Cloudflare on sslhost is WARNING: WARNING - Certificate 'www.mwcosmos.com' expires in 28 day(s) (Thu 11 Dec 2025 10:00:33 AM GMT +0000). [10:21:15] [02ManageWiki] 07SomeMWDev pushed 1 new commit to 03invalidate-opcache 13https://github.com/miraheze/ManageWiki/commit/c8b9b5d94a5e2bcd866184fdf4a471bba04131dd [10:21:15] 02ManageWiki/03invalidate-opcache 07SomeRandomDeveloper 03c8b9b5d Add function use statements [10:23:02] [02CreateWiki] 07SomeMWDev pushed 1 new commit to 03invalidate-dblist-opcache 13https://github.com/miraheze/CreateWiki/commit/31636cf1ade71c2c59f065077e2607bc8f30d32a [10:23:02] 02CreateWiki/03invalidate-dblist-opcache 07SomeRandomDeveloper 0331636cf Add function use statements [10:27:16] miraheze/ManageWiki - SomeMWDev the build passed. [10:28:52] [02CreateWiki] 07codecov[bot] commented on pull request #787: :white_check_mark: All modified and coverable lines are covered by tests. […] 13https://github.com/miraheze/CreateWiki/pull/787#issuecomment-3521210061 [10:35:06] [02CreateWiki] 07SomeMWDev retitled PR #787: "Invalidate opcache after writing to dblist" ➜ "Invalidate opcache for dblist after writing to it" 13https://github.com/miraheze/CreateWiki/pull/787 [10:35:24] [02ManageWiki] 07SomeMWDev retitled PR #750: "Invalidate opcache after writing to cache file" ➜ "Invalidate opcache for cache file after writing to it" 13https://github.com/miraheze/ManageWiki/pull/750 [10:45:45] miraheze/CreateWiki - SomeMWDev the build passed. [11:10:40] [02CreateWiki] 07SomeMWDev merged pull request #787: Invalidate opcache for dblist after writing to it (03main...03invalidate-dblist-opcache) 13https://github.com/miraheze/CreateWiki/pull/787 [11:10:42] [02CreateWiki] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/4f139551c296696489a07f2ea3c05327ae938a7b [11:10:44] 02CreateWiki/03main 07SomeRandomDeveloper 034f13955 Invalidate opcache for dblist after writing to it (#787)… [11:10:46] [02CreateWiki] 07SomeMWDev 04deleted 03invalidate-dblist-opcache at 0331636cf 13https://api.github.com/repos/miraheze/CreateWiki/commit/31636cf [11:10:52] [02ManageWiki] 07SomeMWDev merged pull request #750: Invalidate opcache for cache file after writing to it (03main...03invalidate-opcache) 13https://github.com/miraheze/ManageWiki/pull/750 [11:10:53] [02ManageWiki] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/aea96a1d7cfc29317ca4b4b07f5ccc99fdb36b0c [11:10:53] 02ManageWiki/03main 07SomeRandomDeveloper 03aea96a1 Invalidate opcache for cache file after writing to it (#750)… [11:10:54] [02ManageWiki] 07SomeMWDev 04deleted 03invalidate-opcache at 03c8b9b5d 13https://api.github.com/repos/miraheze/ManageWiki/commit/c8b9b5d [11:11:39] !log [somerandomdeveloper@mwtask181] starting deploy of {'folders': '1.44/extensions/ManageWiki,1.44/extensions/CreateWiki'} to all [11:11:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:12:19] !log [somerandomdeveloper@mwtask181] finished deploy of {'folders': '1.44/extensions/ManageWiki,1.44/extensions/CreateWiki'} to all - SUCCESS in 40s [11:12:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:13:13] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': ['CreateWiki', 'ManageWiki']} to test151 [11:13:16] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': ['CreateWiki', 'ManageWiki']} to test151 - SUCCESS in 2s [11:13:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:13:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:14:26] [02CreateWiki] 07SomeMWDev 04deleted 03DNM-dblist-debug-logging at 03fd5e9f2 13https://api.github.com/repos/miraheze/CreateWiki/commit/fd5e9f2 [11:16:44] miraheze/ManageWiki - SomeMWDev the build passed. [11:18:48] [02CreateWiki] 07SomeMWDev created 03debug-logging (+1 new commit) 13https://github.com/miraheze/CreateWiki/commit/9422417d6d5f [11:18:48] 02CreateWiki/03debug-logging 07SomeRandomDeveloper 039422417 CreateWikiDataStore: Log if mtime is invalid [11:29:29] !log [somerandomdeveloper@test151] starting deploy of {'folders': '1.45/extensions/CreateWiki'} to test151 [11:29:30] !log [somerandomdeveloper@test151] finished deploy of {'folders': '1.45/extensions/CreateWiki'} to test151 - SUCCESS in 0s [11:29:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:29:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:32:47] miraheze/CreateWiki - SomeMWDev the build passed. [11:43:14] [02CreateWiki] 07SomeMWDev pushed 1 new commit to 03debug-logging 13https://github.com/miraheze/CreateWiki/commit/a703dc560cf5c2cb95738f5ebaa5d17f8294a31e [11:43:15] 02CreateWiki/03debug-logging 07SomeRandomDeveloper 03a703dc5 Add debug logging for resetDatabaseLists [11:45:25] !log [somerandomdeveloper@test151] starting deploy of {'folders': '1.45/extensions/CreateWiki'} to test151 [11:45:26] !log [somerandomdeveloper@test151] finished deploy of {'folders': '1.45/extensions/CreateWiki'} to test151 - SUCCESS in 0s [11:45:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:45:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:53:02] !log [somerandomdeveloper@mwtask181] starting deploy of {'folders': '1.44/extensions/CreateWiki'} to all [11:53:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:53:23] !log [somerandomdeveloper@mwtask181] finished deploy of {'folders': '1.44/extensions/CreateWiki'} to all - SUCCESS in 21s [11:53:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:58:03] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.66, 16.97, 10.29 [12:00:03] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.98, 18.53, 11.72 [12:34:11] [02CreateWiki] 07SomeMWDev pushed 1 new commit to 03debug-logging 13https://github.com/miraheze/CreateWiki/commit/3c27e54b8dfc0c8f260589a87d8b59622bb33d4a [12:34:11] 02CreateWiki/03debug-logging 07SomeRandomDeveloper 033c27e54 Avoid regenerating dblists more than once per server [12:40:48] !log [somerandomdeveloper@test151] starting deploy of {'folders': '1.45/extensions/CreateWiki'} to test151 [12:40:49] !log [somerandomdeveloper@test151] finished deploy of {'folders': '1.45/extensions/CreateWiki'} to test151 - SUCCESS in 0s [12:40:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:40:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:43:38] !log [somerandomdeveloper@mwtask181] starting deploy of {'folders': '1.44/extensions/CreateWiki'} to all [12:43:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:43:59] !log [somerandomdeveloper@mwtask181] finished deploy of {'folders': '1.44/extensions/CreateWiki'} to all - SUCCESS in 20s [12:44:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:48:18] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49773MiB (10% inode=99%); [13:51:27] [02CreateWiki] 07SomeMWDev created 03local-dblist-cache-key (+1 new commit) 13https://github.com/miraheze/CreateWiki/commit/45586e7f0855 [13:51:27] 02CreateWiki/03local-dblist-cache-key 07SomeRandomDeveloper 0345586e7 Avoid regenerating dblists more than once per server… [13:51:45] [02CreateWiki] 07SomeMWDev opened pull request #788: Avoid regenerating dblists more than once per server (03main...03local-dblist-cache-key) 13https://github.com/miraheze/CreateWiki/pull/788 [13:51:53] [02CreateWiki] 07coderabbitai[bot] commented on pull request #788:
[…] 13https://github.com/miraheze/CreateWiki/pull/788#issuecomment-3522048749 [13:57:26] [02CreateWiki] 07codecov[bot] commented on pull request #788: :white_check_mark: All modified and coverable lines are covered by tests. […] 13https://github.com/miraheze/CreateWiki/pull/788#issuecomment-3522078904 [14:14:31] miraheze/CreateWiki - SomeMWDev the build passed. [15:46:23] !log upgrade grafana on mon181 [15:46:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:07:37] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 22.89, 15.34, 10.75 [16:13:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.97, 17.82, 11.47 [16:14:27] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 21.99, 18.38, 11.83 [16:15:32] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 21.17, 19.11, 12.69 [16:16:27] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 18.53, 18.45, 12.67 [16:17:02] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.08, 18.93, 12.93 [16:17:10] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.42, 18.71, 13.34 [16:17:32] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 16.09, 17.69, 12.94 [16:21:02] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 20.36, 20.03, 14.79 [16:35:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762961690000&orgId=1&to=1762965320043 [16:40:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762961810000&orgId=1&to=1762965410000 [16:45:25] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/a3ec3c828b6a3f65c88d2a1d8617d752d30980b4 [16:45:26] 02ssl/03main 07WikiTideBot 03a3ec3c8 Bot: Auto-update domain lists [17:01:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.63, 18.13, 15.20 [17:03:09] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.18, 18.10, 14.38 [17:05:02] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.56, 19.38, 15.67 [17:05:09] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 19.20, 18.47, 14.99 [17:12:27] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 21.76, 20.77, 17.49 [17:14:27] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 18.91, 20.30, 17.73 [17:17:02] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 16.12, 18.86, 17.85 [17:20:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762964390000&orgId=1&to=1762968020038 [17:29:10] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.57, 19.63, 19.76 [17:33:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.98, 21.18, 20.32 [17:37:10] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.07, 19.94, 20.08 [17:39:02] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [17:45:23] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/3ade134be2c2388abb7dc7ced2df7902943f14ce [17:45:23] 02ssl/03main 07WikiTideBot 033ade134 Bot: Auto-update domain lists [17:45:36] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 9.62, 9.69, 11.91 [17:47:02] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [17:55:36] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 12.52, 11.36, 11.62 [17:57:36] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 11.58, 11.37, 11.58 [17:59:36] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 14.06, 11.84, 11.69 [18:01:24] !log [somerandomdeveloper@mwtask181] starting deploy of {'folders': '1.44/extensions/DataDump'} to all [18:01:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:01:54] !log [somerandomdeveloper@mwtask181] finished deploy of {'folders': '1.44/extensions/DataDump'} to all - SUCCESS in 29s [18:02:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:09:36] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 3.14, 8.41, 10.93 [18:10:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762967390000&orgId=1&to=1762970990000 [18:11:36] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 3.19, 6.71, 10.01 [18:18:47] !log [somerandomdeveloper@mwtask181] starting deploy of {'folders': '1.44/extensions/DataDump'} to all [18:18:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:19:38] !log [somerandomdeveloper@mwtask181] finished deploy of {'folders': '1.44/extensions/DataDump'} to all - SUCCESS in 50s [18:19:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:21:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.77, 19.49, 17.81 [18:22:27] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 23.46, 18.76, 16.12 [18:24:27] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 18.13, 18.63, 16.42 [18:27:13] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.74, 19.97, 18.72 [18:33:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.11, 21.07, 19.43 [18:35:10] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.38, 19.92, 19.19 [18:43:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762969370000&orgId=1&to=1762973000038 [18:44:34] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49869MiB (10% inode=99%); [18:48:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762969550000&orgId=1&to=1762973150000 [19:18:14] [02DataDump] 07SomeMWDev created 03T14516 (+2 new commits) 13https://github.com/miraheze/DataDump/compare/d7dbbc468bee^...9a95aa54e96d [19:18:14] 02DataDump/03T14516 07SomeRandomDeveloper 03d7dbbc4 Get new DB connection to update status in the end [19:18:15] 02DataDump/03T14516 07SomeRandomDeveloper 039a95aa5 Add $wgDataDumpAllowRetries [19:19:01] [02DataDump] 07SomeMWDev opened pull request #111: Fix DB connection issue and add config option to disable retries (03main...03T14516) 13https://github.com/miraheze/DataDump/pull/111 [19:19:10] [02DataDump] 07coderabbitai[bot] commented on pull request #111:
[…] 13https://github.com/miraheze/DataDump/pull/111#issuecomment-3523531273 [19:20:01] [02DataDump] 07SomeMWDev pushed 1 new commit to 03T14516 13https://github.com/miraheze/DataDump/commit/6e0f92b9559a89704c9140e3b5c96a46cc580928 [19:20:01] 02DataDump/03T14516 07SomeRandomDeveloper 036e0f92b Add inheritDoc [19:21:45] [02mw-config] 07SomeMWDev created 03T14516 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/1e42bfd04b16 [19:21:46] 02mw-config/03T14516 07SomeRandomDeveloper 031e42bfd Disable data dump retries [19:24:52] miraheze/DataDump - SomeMWDev the build passed. [19:25:34] [02mw-config] 07SomeMWDev opened pull request #6175: Disable data dump retries (03main...03T14516) 13https://github.com/miraheze/mw-config/pull/6175 [19:25:52] [02DataDump] 07paladox merged 07SomeMWDev's pull request #111: Fix DB connection issue and add config option to disable retries (03main...03T14516) 13https://github.com/miraheze/DataDump/pull/111 [19:25:54] [02DataDump] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/DataDump/commit/31e2b2dc34cab3aee9b0ad4b290716b8ea1cfba7 [19:25:56] [02mw-config] 07SomeMWDev retitled PR #6175: "Disable data dump retries" ➜ "Disable DataDump retries" 13https://github.com/miraheze/mw-config/pull/6175 [19:25:58] [02DataDump] 07paladox 04deleted 03T14516 at 036e0f92b 13https://api.github.com/repos/miraheze/DataDump/commit/6e0f92b [19:25:59] 02DataDump/03main 07SomeRandomDeveloper 0331e2b2d Fix DB connection issue and add config option to disable retries (#111)… [19:26:25] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'DataDump'} to all [19:26:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:30] miraheze/mw-config - SomeMWDev the build passed. [19:26:48] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'DataDump'} to all - SUCCESS in 23s [19:26:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:00] [02mw-config] 07SomeMWDev merged pull request #6175: Disable DataDump retries (03main...03T14516) 13https://github.com/miraheze/mw-config/pull/6175 [19:27:00] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/c86aa33bb980a2222833413c983e9c70c62cd6a8 [19:27:00] 02mw-config/03main 07SomeRandomDeveloper 03c86aa33 Disable DataDump retries (#6175)… [19:27:01] [02mw-config] 07SomeMWDev 04deleted 03T14516 at 031e42bfd 13https://api.github.com/repos/miraheze/mw-config/commit/1e42bfd [19:27:31] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'DataDump'} to test151 [19:27:32] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'DataDump'} to test151 - SUCCESS in 1s [19:27:35] !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [19:27:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:36] !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [19:27:38] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:27:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:28:00] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s [19:28:03] miraheze/mw-config - SomeMWDev the build passed. [19:28:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:31:02] miraheze/DataDump - paladox the build passed. [20:01:18] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49820MiB (10% inode=99%); [20:03:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.96, 19.16, 17.16 [20:05:10] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.48, 18.78, 17.27 [20:37:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762976240000&orgId=1&to=1762979870037 [20:44:45] !log [void@puppet181] Upgraded packages on cp161: linux-libc-dev [20:44:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:45:00] !log [void@puppet181] Upgraded packages on cloud15: linux-libc-dev [20:45:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:45:12] !log [void@puppet181] Upgraded packages on changeprop201: linux-libc-dev [20:45:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:45:28] !log [void@puppet181] Upgraded packages on cloud20: linux-libc-dev [20:45:28] RECOVERY - cp161 APT on cp161 is OK: APT OK: 118 packages available for upgrade (0 critical updates). [20:45:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:45:43] !log [void@puppet181] Upgraded packages on bots171: linux-libc-dev [20:45:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:45:54] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 187 packages available for upgrade (0 critical updates). [20:45:55] !log [void@puppet181] Upgraded packages on cloud19: linux-libc-dev [20:45:58] RECOVERY - changeprop201 APT on changeprop201 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:45:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:46:05] RECOVERY - cloud20 APT on cloud20 is OK: APT OK: 188 packages available for upgrade (0 critical updates). [20:46:09] !log [void@puppet181] Upgraded packages on db172: linux-libc-dev [20:46:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:46:24] !log [void@puppet181] Upgraded packages on cloud16: linux-libc-dev [20:46:28] RECOVERY - db172 APT on db172 is OK: APT OK: 92 packages available for upgrade (0 critical updates). [20:46:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:46:41] !log [void@puppet181] Upgraded packages on cloud18: linux-libc-dev [20:46:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:46:57] !log [void@puppet181] Upgraded packages on cloud17: linux-libc-dev [20:47:01] RECOVERY - bots171 APT on bots171 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [20:47:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:47:11] !log [void@puppet181] Upgraded packages on eventgate181: linux-libc-dev [20:47:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:47:25] !log [void@puppet181] Upgraded packages on llm191: linux-compiler-gcc-12-x86, linux-kbuild-6.1, and linux-libc-dev [20:47:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:47:38] RECOVERY - cloud19 APT on cloud19 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [20:47:39] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 184 packages available for upgrade (0 critical updates). [20:47:40] !log [void@puppet181] Upgraded packages on bast161: linux-libc-dev [20:47:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:47:50] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762976240000&orgId=1&to=1762980260000 [20:47:50] RECOVERY - bast161 APT on bast161 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [20:47:57] RECOVERY - eventgate181 APT on eventgate181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:48:01] !log [void@puppet181] Upgraded packages on bast181: linux-libc-dev [20:48:05] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 184 packages available for upgrade (0 critical updates). [20:48:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:15] !log [void@puppet181] Upgraded packages on cp201: linux-libc-dev [20:48:20] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 184 packages available for upgrade (0 critical updates). [20:48:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:31] !log [void@puppet181] Upgraded packages on cp171: linux-libc-dev [20:48:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:42] RECOVERY - cp201 APT on cp201 is OK: APT OK: 79 packages available for upgrade (0 critical updates). [20:48:44] RECOVERY - llm191 APT on llm191 is OK: APT OK: 47 packages available for upgrade (0 critical updates). [20:48:46] !log [void@puppet181] Upgraded packages on cp191: linux-libc-dev [20:48:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:59] RECOVERY - cp191 APT on cp191 is OK: APT OK: 108 packages available for upgrade (0 critical updates). [20:49:00] !log [void@puppet181] Upgraded packages on db182: linux-libc-dev [20:49:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:13] !log [void@puppet181] Upgraded packages on kafka181: linux-libc-dev [20:49:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:29] !log [void@puppet181] Upgraded packages on ldap171: linux-libc-dev [20:49:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:45] !log [void@puppet181] Upgraded packages on db151: linux-libc-dev [20:49:48] RECOVERY - db182 APT on db182 is OK: APT OK: 82 packages available for upgrade (0 critical updates). [20:49:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:59] RECOVERY - bast181 APT on bast181 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [20:50:02] !log [void@puppet181] Upgraded packages on db171: linux-libc-dev [20:50:05] RECOVERY - cp171 APT on cp171 is OK: APT OK: 108 packages available for upgrade (0 critical updates). [20:50:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:50:17] !log [void@puppet181] Upgraded packages on db161: linux-libc-dev [20:50:22] RECOVERY - kafka181 APT on kafka181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:50:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:50:30] RECOVERY - db151 APT on db151 is OK: APT OK: 127 packages available for upgrade (0 critical updates). [20:50:34] !log [void@puppet181] Upgraded packages on graylog161: linux-libc-dev [20:50:36] RECOVERY - ldap171 APT on ldap171 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [20:50:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:50:50] !log [void@puppet181] Upgraded packages on db181: linux-libc-dev [20:50:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:50:55] RECOVERY - db171 APT on db171 is OK: APT OK: 119 packages available for upgrade (0 critical updates). [20:51:05] !log [void@puppet181] Upgraded packages on mem151: linux-libc-dev [20:51:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:51:20] !log [void@puppet181] Upgraded packages on mem161: linux-libc-dev [20:51:25] RECOVERY - db181 APT on db181 is OK: APT OK: 119 packages available for upgrade (0 critical updates). [20:51:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:51:29] RECOVERY - db161 APT on db161 is OK: APT OK: 127 packages available for upgrade (0 critical updates). [20:51:34] !log [void@puppet181] Upgraded packages on matomo151: linux-libc-dev [20:51:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:51:49] !log [void@puppet181] Upgraded packages on mem201: linux-libc-dev [20:51:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:52:00] RECOVERY - mem151 APT on mem151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [20:52:05] !log [void@puppet181] Upgraded packages on mem191: linux-libc-dev [20:52:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:52:11] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 72 packages available for upgrade (0 critical updates). [20:52:11] RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [20:52:16] !log [void@puppet181] Upgraded packages on mattermost1: linux-libc-dev [20:52:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:52:33] !log [void@puppet181] Upgraded packages on mw152: linux-libc-dev [20:52:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:52:47] !log [void@puppet181] Upgraded packages on mon181: linux-libc-dev [20:52:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:52:56] RECOVERY - mattermost1 APT on mattermost1 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [20:52:58] RECOVERY - mon181 APT on mon181 is OK: APT OK: 12 packages available for upgrade (0 critical updates). [20:53:01] RECOVERY - mem161 APT on mem161 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [20:53:03] !log [void@puppet181] Upgraded packages on mw151: linux-libc-dev [20:53:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:53:19] !log [void@puppet181] Upgraded packages on mw161: linux-libc-dev [20:53:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:53:28] RECOVERY - mem201 APT on mem201 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [20:53:36] !log [void@puppet181] Upgraded packages on mw153: linux-libc-dev [20:53:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:53:48] RECOVERY - mw152 APT on mw152 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:53:52] !log [void@puppet181] Upgraded packages on mw162: linux-libc-dev [20:53:54] RECOVERY - mw161 APT on mw161 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:53:56] RECOVERY - mw151 APT on mw151 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [20:53:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:54:06] RECOVERY - mem191 APT on mem191 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [20:54:09] !log [void@puppet181] Upgraded packages on mw171: linux-libc-dev [20:54:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:54:17] RECOVERY - mw162 APT on mw162 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:54:25] !log [void@puppet181] Upgraded packages on mw163: linux-libc-dev [20:54:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:54:42] !log [void@puppet181] Upgraded packages on mw172: linux-libc-dev [20:54:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:54:53] RECOVERY - mw171 APT on mw171 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:54:59] !log [void@puppet181] Upgraded packages on mw191: linux-libc-dev [20:55:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:55:16] !log [void@puppet181] Upgraded packages on mw173: linux-libc-dev [20:55:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:55:23] RECOVERY - mw153 APT on mw153 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:55:24] RECOVERY - mw191 APT on mw191 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:55:28] RECOVERY - mw172 APT on mw172 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:55:30] RECOVERY - mw163 APT on mw163 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:55:32] !log [void@puppet181] Upgraded packages on mw193: linux-libc-dev [20:55:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:55:49] !log [void@puppet181] Upgraded packages on mw182: linux-libc-dev [20:55:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:56:06] !log [void@puppet181] Upgraded packages on mw201: linux-libc-dev [20:56:10] RECOVERY - mw182 APT on mw182 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:56:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:56:16] RECOVERY - mw173 APT on mw173 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:56:23] !log [void@puppet181] Upgraded packages on mw192: linux-libc-dev [20:56:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:56:41] !log [void@puppet181] Upgraded packages on mw181: linux-libc-dev [20:56:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:56:58] !log [void@puppet181] Upgraded packages on mw202: linux-libc-dev [20:57:00] RECOVERY - mw192 APT on mw192 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:57:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:57:14] RECOVERY - mw202 APT on mw202 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:57:16] !log [void@puppet181] Upgraded packages on mw203: linux-libc-dev [20:57:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:57:21] RECOVERY - mw203 APT on mw203 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:57:34] !log [void@puppet181] Upgraded packages on mw183: linux-libc-dev [20:57:36] RECOVERY - mw193 APT on mw193 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:57:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:57:51] RECOVERY - mw201 APT on mw201 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:57:52] !log [void@puppet181] Upgraded packages on mwtask161: linux-libc-dev [20:57:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:58:04] RECOVERY - mw183 APT on mw183 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:58:11] !log [void@puppet181] Upgraded packages on mwtask151: linux-libc-dev [20:58:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:58:28] !log [void@puppet181] Upgraded packages on ns1: linux-libc-dev [20:58:30] RECOVERY - mw181 APT on mw181 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:58:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:58:35] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 133 packages available for upgrade (0 critical updates). [20:58:36] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 133 packages available for upgrade (0 critical updates). [20:58:44] RECOVERY - ns1 APT on ns1 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [20:58:47] !log [void@puppet181] Upgraded packages on mwtask171: linux-libc-dev [20:58:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:59:02] !log [void@puppet181] Upgraded packages on os161: linux-libc-dev [20:59:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:59:17] !log [void@puppet181] Upgraded packages on os162: linux-libc-dev [20:59:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:59:34] !log [void@puppet181] Upgraded packages on os191: linux-libc-dev [20:59:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:59:46] RECOVERY - os161 APT on os161 is OK: APT OK: 110 packages available for upgrade (0 critical updates). [20:59:52] !log [void@puppet181] Upgraded packages on mwtask181: linux-libc-dev [20:59:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:00:00] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 133 packages available for upgrade (0 critical updates). [21:00:06] !log [void@puppet181] Upgraded packages on rdb151: linux-libc-dev [21:00:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:00:16] RECOVERY - os191 APT on os191 is OK: APT OK: 110 packages available for upgrade (0 critical updates). [21:00:20] !log [void@puppet181] Upgraded packages on prometheus151: linux-libc-dev [21:00:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:00:29] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 133 packages available for upgrade (0 critical updates). [21:00:34] RECOVERY - os162 APT on os162 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [21:00:37] !log [void@puppet181] Upgraded packages on os151: linux-libc-dev [21:00:39] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 113 packages available for upgrade (0 critical updates). [21:00:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:00:51] !log [void@puppet181] Upgraded packages on phorge171: linux-libc-dev [21:00:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:01:06] !log [void@puppet181] Upgraded packages on puppet181: linux-libc-dev [21:01:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:01:21] !log [void@puppet181] Upgraded packages on reports171: linux-libc-dev [21:01:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:01:27] RECOVERY - rdb151 APT on rdb151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [21:01:27] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 74 packages available for upgrade (0 critical updates). [21:01:38] !log [void@puppet181] Upgraded packages on os201: linux-libc-dev [21:01:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:01:56] RECOVERY - os201 APT on os201 is OK: APT OK: 82 packages available for upgrade (0 critical updates). [21:01:57] !log [void@puppet181] Upgraded packages on ns2: linux-libc-dev [21:02:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:02:06] RECOVERY - os151 APT on os151 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [21:02:13] !log [void@puppet181] Upgraded packages on swiftobject161: linux-libc-dev [21:02:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:02:28] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 76 packages available for upgrade (0 critical updates). [21:02:31] !log [void@puppet181] Upgraded packages on swiftobject151: linux-libc-dev [21:02:36] RECOVERY - ns2 APT on ns2 is OK: APT OK: 114 packages available for upgrade (0 critical updates). [21:02:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:02:51] !log [void@puppet181] Upgraded packages on swiftac171: linux-libc-dev [21:02:53] RECOVERY - reports171 APT on reports171 is OK: APT OK: 113 packages available for upgrade (0 critical updates). [21:02:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:03:07] !log [void@puppet181] Upgraded packages on swiftobject191: linux-libc-dev [21:03:09] RECOVERY - swiftobject191 APT on swiftobject191 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [21:03:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:03:24] !log [void@puppet181] Upgraded packages on swiftobject201: linux-libc-dev [21:03:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:03:42] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [21:03:43] !log [void@puppet181] Upgraded packages on swiftobject181: linux-libc-dev [21:03:43] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 124 packages available for upgrade (0 critical updates). [21:03:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:01] !log [void@puppet181] Upgraded packages on swiftproxy161: linux-libc-dev [21:04:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:11] RECOVERY - swiftobject201 APT on swiftobject201 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [21:04:19] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [21:04:20] !log [void@puppet181] Upgraded packages on swiftobject171: linux-libc-dev [21:04:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:36] !log [void@puppet181] Upgraded packages on test151: linux-libc-dev [21:04:39] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=mashuptourneyswiki (END - exit=0) [21:04:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:43] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [21:04:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:54] !log [void@puppet181] Upgraded packages on swiftproxy171: linux-libc-dev [21:04:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:05:16] !log [void@puppet181] Upgraded packages on os202: linux-libc-dev [21:05:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:05:22] RECOVERY - test151 APT on test151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [21:05:23] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [21:05:40] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 126 packages available for upgrade (0 critical updates). [21:05:41] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 126 packages available for upgrade (0 critical updates). [21:05:44] RECOVERY - os202 APT on os202 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [23:42:03] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.34, 17.54, 13.38 [23:44:03] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.44, 18.29, 14.24 [23:58:50] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.18, 21.36, 16.73