[00:00:15] [02IncidentReporting] 07Universal-Omega merged 07dependabot[bot]'s pull request #99: Bump eslint-config-wikimedia from 0.28.2 to 0.29.0 (03master...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0) 13https://github.com/miraheze/IncidentReporting/pull/99 [00:00:15] [02IncidentReporting] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/IncidentReporting/commit/d32e49e2d5a8dccad28968a7047249728e4f383a [00:00:15] 02IncidentReporting/03master 07dependabot[bot] 03d32e49e Bump eslint-config-wikimedia from 0.28.2 to 0.29.0 (#99)… [00:00:16] [02IncidentReporting] 07Universal-Omega 04deleted 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0 at 0322521b5 13https://github.com/miraheze/IncidentReporting/commit/22521b5 [00:05:10] miraheze/IncidentReporting - Universal-Omega the build passed. [00:05:21] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 55577MiB (12% inode=99%); [00:13:44] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.62, 3.41, 2.87 [00:17:37] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 3.39, 3.39, 2.98 [01:18:32] [02RequestSSL] 07dependabot[bot] created 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0 (+1 new commit) 13https://github.com/miraheze/RequestSSL/commit/fbe41b1ef2ca [01:18:32] 02RequestSSL/03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0 07dependabot[bot] 03fbe41b1 Bump eslint-config-wikimedia from 0.28.2 to 0.29.0… [01:18:33] [02RequestSSL] 07dependabot[bot] opened pull request #67: Bump eslint-config-wikimedia from 0.28.2 to 0.29.0 (03master...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0) 13https://github.com/miraheze/RequestSSL/pull/67 [01:18:33] [02RequestSSL] 07dependabot[bot] added the label 'dependencies' to pull request #67 (Bump eslint-config-wikimedia from 0.28.2 to 0.29.0) 13https://github.com/miraheze/RequestSSL/pull/67 [01:18:35] [02RequestSSL] 07dependabot[bot] added the label 'javascript' to pull request #67 (Bump eslint-config-wikimedia from 0.28.2 to 0.29.0) 13https://github.com/miraheze/RequestSSL/pull/67 [01:18:39] [02RequestSSL] 07coderabbitai[bot] commented on pull request #67: --- […] 13https://github.com/miraheze/RequestSSL/pull/67#issuecomment-2680116098 [01:27:04] [02RequestSSL] 07Universal-Omega merged 07dependabot[bot]'s pull request #67: Bump eslint-config-wikimedia from 0.28.2 to 0.29.0 (03master...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0) 13https://github.com/miraheze/RequestSSL/pull/67 [01:27:05] [02RequestSSL] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/RequestSSL/commit/bcebd423d6a046c016fa4d3dd4dffca8569edb2f [01:27:06] 02RequestSSL/03master 07dependabot[bot] 03bcebd42 Bump eslint-config-wikimedia from 0.28.2 to 0.29.0 (#67)… [01:27:09] [02RequestSSL] 07Universal-Omega 04deleted 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0 at 03fbe41b1 13https://github.com/miraheze/RequestSSL/commit/fbe41b1 [01:30:02] miraheze/RequestSSL - dependabot[bot] the build passed. [01:39:08] miraheze/RequestSSL - Universal-Omega the build passed. [02:36:39] [02CreateWiki] 07Universal-Omega merged pull request #663: Expand usage of CreateWikiDatabaseUtils (03master...03databaseutils) 13https://github.com/miraheze/CreateWiki/pull/663 [02:36:41] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/CreateWiki/commit/e11e4a7f78b8b2de2d7f551de49048ec41f9184c [02:36:43] [02CreateWiki] 07Universal-Omega 04deleted 03databaseutils at 03fe68a56 13https://github.com/miraheze/CreateWiki/commit/fe68a56 [02:36:44] 02CreateWiki/03master 07CosmicAlpha 03e11e4a7 Expand usage of CreateWikiDatabaseUtils (#663) [02:49:11] miraheze/CreateWiki - Universal-Omega the build passed. [02:53:21] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49802MiB (10% inode=99%); [02:54:28] [02CreateWiki] 07Universal-Omega created 03databaseutils-perf (+1 new commit) 13https://github.com/miraheze/CreateWiki/commit/37bb216fbf1c [02:54:28] 02CreateWiki/03databaseutils-perf 07CosmicAlpha 0337bb216 Minor performance improvements in CreateWikiDatabaseUtils [02:54:32] [02CreateWiki] 07Universal-Omega opened pull request #675: Minor performance improvements in CreateWikiDatabaseUtils (03master...03databaseutils-perf) 13https://github.com/miraheze/CreateWiki/pull/675 [02:54:38] [02CreateWiki] 07coderabbitai[bot] commented on pull request #675: --- […] 13https://github.com/miraheze/CreateWiki/pull/675#issuecomment-2680275522 [02:57:48] miraheze/CreateWiki - Universal-Omega the build has errored. [03:00:11] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03databaseutils-perf 13https://github.com/miraheze/CreateWiki/commit/a04a4b60af02c020af256d318138c88583673915 [03:00:11] 02CreateWiki/03databaseutils-perf 07CosmicAlpha 03a04a4b6 Try [03:11:08] miraheze/CreateWiki - Universal-Omega the build has errored. [03:17:46] [02CreateWiki] 07Universal-Omega closed pull request #675: Minor performance improvements in CreateWikiDatabaseUtils (03master...03databaseutils-perf) 13https://github.com/miraheze/CreateWiki/pull/675 [03:17:49] [02CreateWiki] 07Universal-Omega 04deleted 03databaseutils-perf at 03a04a4b6 13https://github.com/miraheze/CreateWiki/commit/a04a4b6 [03:17:56] [02CreateWiki] 07Universal-Omega closed pull request #673: Apply SchemaVars when creating databases (03master...03schema-vars) 13https://github.com/miraheze/CreateWiki/pull/673 [03:17:58] [02CreateWiki] 07Universal-Omega 04deleted 03schema-vars at 0378d9ecf 13https://github.com/miraheze/CreateWiki/commit/78d9ecf [03:29:50] PROBLEM - mw161.wikitide.net SSL Check on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 30 day(s) (Fri 28 Mar 2025 03:25:53 AM GMT +0000). [03:33:26] PROBLEM - mw171.wikitide.net SSL Check on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 30 day(s) (Fri 28 Mar 2025 03:25:53 AM GMT +0000). [03:33:50] PROBLEM - mw151.wikitide.net SSL Check on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 30 day(s) (Fri 28 Mar 2025 03:25:53 AM GMT +0000). [03:34:40] PROBLEM - mw164.wikitide.net SSL Check on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 30 day(s) (Fri 28 Mar 2025 03:25:53 AM GMT +0000). [03:37:17] PROBLEM - mw172.wikitide.net SSL Check on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 30 day(s) (Fri 28 Mar 2025 03:25:53 AM GMT +0000). [03:38:17] PROBLEM - mw184.wikitide.net SSL Check on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 30 day(s) (Fri 28 Mar 2025 03:25:53 AM GMT +0000). [03:38:29] PROBLEM - mw183.wikitide.net SSL Check on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 30 day(s) (Fri 28 Mar 2025 03:25:53 AM GMT +0000). [03:39:48] PROBLEM - mw181.wikitide.net SSL Check on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 30 day(s) (Fri 28 Mar 2025 03:25:53 AM GMT +0000). [03:40:52] [02dns] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/cf75ac23d435621208f92a365c740fb49300c09a [03:40:53] 02dns/03master 07MacFan4000 03cf75ac2 add ACME challenges for wikitide.net [03:40:54] PROBLEM - mw174.wikitide.net SSL Check on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 30 day(s) (Fri 28 Mar 2025 03:25:53 AM GMT +0000). [03:41:49] miraheze/dns - MacFan4000 the build passed. [03:42:26] PROBLEM - mw152.wikitide.net SSL Check on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 30 day(s) (Fri 28 Mar 2025 03:25:53 AM GMT +0000). [03:44:12] [02dns] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/506fa3abe28c745121f3ac407a3f9b15c937ab1b [03:44:12] 02dns/03master 07MacFan4000 03506fa3a Update wikitide.net [03:44:44] PROBLEM - mw153.wikitide.net SSL Check on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 30 day(s) (Fri 28 Mar 2025 03:25:53 AM GMT +0000). [03:45:04] miraheze/dns - MacFan4000 the build passed. [03:47:08] PROBLEM - mw173.wikitide.net SSL Check on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 30 day(s) (Fri 28 Mar 2025 03:25:53 AM GMT +0000). [03:47:34] [02dns] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/3ce41d059cf0bd6653ee89c310ac35db147e79ab [03:47:34] 02dns/03master 07MacFan4000 033ce41d0 Update wikitide.net [03:48:08] PROBLEM - mw182.wikitide.net SSL Check on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 30 day(s) (Fri 28 Mar 2025 03:25:53 AM GMT +0000). [03:48:28] miraheze/dns - MacFan4000 the build passed. [03:48:58] PROBLEM - mw162.wikitide.net SSL Check on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 30 day(s) (Fri 28 Mar 2025 03:25:53 AM GMT +0000). [03:51:55] PROBLEM - mw154.wikitide.net SSL Check on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 30 day(s) (Fri 28 Mar 2025 03:25:53 AM GMT +0000). [03:53:30] PROBLEM - mw163.wikitide.net SSL Check on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 30 day(s) (Fri 28 Mar 2025 03:25:53 AM GMT +0000). [03:56:50] [02dns] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/88b0c56788ded9a5073d27d74a8876f7b62b3660 [03:56:50] 02dns/03master 07MacFan4000 0388b0c56 Update wikitide.net [03:58:00] miraheze/dns - MacFan4000 the build passed. [03:58:50] [02dns] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/031ad0e46adada159793a21af52f990eaabd7b03 [03:58:50] 02dns/03master 07MacFan4000 03031ad0e Update wikitide.net [03:59:54] miraheze/dns - MacFan4000 the build passed. [04:07:04] * MacFan4000 will have to try this again tomorrow after the rate limits reset [04:34:42] !log created BlankEclair's email login [04:34:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:21:13] [02puppet] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/07ed60645520c205eac6527743823fbd8156fa6b [05:21:13] 02puppet/03master 07CosmicAlpha 0307ed606 T13272: add BlankEclair icinga [05:22:53] [02puppet] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/42ba39fb678f8f87cb5e3cdb55e547636739c0ca [05:22:53] 02puppet/03master 07CosmicAlpha 0342ba39f Add BlankEclair [05:24:11] miraheze/puppet - Universal-Omega the build passed. [06:16:44] [02puppet] 07Universal-Omega created 03add-shell (+1 new commit) 13https://github.com/miraheze/puppet/commit/59cafb71caab [06:16:44] 02puppet/03add-shell 07CosmicAlpha 0359cafb7 Add BlankEclair shell [06:16:49] [02puppet] 07Universal-Omega opened pull request #4193: Add BlankEclair shell (03master...03add-shell) 13https://github.com/miraheze/puppet/pull/4193 [06:16:58] [02puppet] 07coderabbitai[bot] commented on pull request #4193: --- […] 13https://github.com/miraheze/puppet/pull/4193#issuecomment-2680744296 [06:17:04] [02puppet] 07Universal-Omega retitled PR #4193: "Add BlankEclair shell" ➜ "T13272: add BlankEclair shell" 13https://github.com/miraheze/puppet/pull/4193 [07:49:25] [02puppet] 07Universal-Omega merged pull request #4193: T13272: add BlankEclair shell (03master...03add-shell) 13https://github.com/miraheze/puppet/pull/4193 [07:49:26] [02puppet] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/bfa35b218b9a90a16443a3b56f2c6d4daf99b4ae [07:49:26] 02puppet/03master 07CosmicAlpha 03bfa35b2 T13272: add BlankEclair shell (#4193) [07:49:29] [02puppet] 07Universal-Omega 04deleted 03add-shell at 0359cafb7 13https://github.com/miraheze/puppet/commit/59cafb7 [07:49:33] hell yeah homosexuals [07:52:27] [02mw-config] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/7ecdaa3b609aba075285f31c49deca07bdbb1405 [07:52:28] 02mw-config/03master 07CosmicAlpha 037ecdaa3 T13272: Add BlankEclair to staffwiki [07:53:22] miraheze/mw-config - Universal-Omega the build passed. [08:05:22] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [08:05:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:05:43] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 20s [08:05:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:20:35] PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to matomo151.wikitide.net port 443 after 0 ms: Couldn't connect to server [08:26:35] RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.272 second response time [08:54:26] [[SAL]] [08:54:30] thanks wm-bot2 [08:54:30] Hey BlankEclair, you are welcome! [08:58:49] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importDump --wiki=hcmwiki /home/blankeclair/mediawiki-20250221-history.xml --no-updates --report 100 (START) [08:58:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:04:14] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importDump --wiki=hcmwiki /home/blankeclair/mediawiki-20250221-history.xml --no-updates --report 100 (END - exit=0) [09:04:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:10:56] PROBLEM - opendatascot.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'opendatascot.org' expires in 14 day(s) (Wed 12 Mar 2025 08:48:09 AM GMT +0000). [09:11:08] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/2bd84be6831c687e0fd1b14a707e61b78d68b401 [09:11:09] 02ssl/03master 07WikiTideSSLBot 032bd84be Bot: Update SSL cert for opendatascot.org [10:10:30] RECOVERY - opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Mon 26 May 2025 08:12:32 AM GMT +0000. [10:13:17] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importImages --wiki=hcmwiki /home/blankeclair/images --comment=Importing file per request ([[phorge:T13259|T13259]]) --search-recursively (START) [10:13:18] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importImages --wiki=hcmwiki /home/blankeclair/images --comment=Importing file per request ([[phorge:T13259|T13259]]) --search-recursively (END - exit=512) [10:13:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:13:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:13:37] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importImages --wiki=hcmwiki /home/blankeclair/images --comment='Importing file per request ([[phorge:T13259|T13259]])' --search-recursively (START) [10:13:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:13:40] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importImages --wiki=hcmwiki /home/blankeclair/images --comment='Importing file per request ([[phorge:T13259|T13259]])' --search-recursively (END - exit=0) [10:13:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:13:54] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php rebuildall --wiki=hcmwiki (START) [10:13:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:15:05] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php rebuildall --wiki=hcmwiki (END - exit=0) [10:15:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:09:41] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.73, 14.04, 7.53 [12:09:44] PROBLEM - mw174 Current Load on mw174 is WARNING: LOAD WARNING - total load average: 22.41, 14.20, 7.78 [12:10:16] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.26, 15.09, 8.07 [12:10:25] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.41, 15.92, 8.79 [12:10:51] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 21.71, 15.04, 8.36 [12:11:21] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.22, 16.15, 9.07 [12:11:25] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 22.38, 16.56, 9.26 [12:11:32] PROBLEM - mw154 Current Load on mw154 is WARNING: LOAD WARNING - total load average: 22.27, 17.17, 9.60 [12:11:38] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 16.12, 9.76, 4.74 [12:12:16] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.20, 18.16, 10.05 [12:12:25] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 19.55, 16.16, 9.70 [12:12:48] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 23.09, 17.81, 10.31 [12:12:51] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 17.62, 15.39, 9.30 [12:13:03] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw174 [12:13:21] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 15.89, 15.51, 9.69 [12:13:32] RECOVERY - mw154 Current Load on mw154 is OK: LOAD OK - total load average: 18.70, 17.27, 10.53 [12:13:44] RECOVERY - mw174 Current Load on mw174 is OK: LOAD OK - total load average: 14.27, 15.71, 9.99 [12:14:09] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw171 [12:14:16] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 20.35, 17.47, 10.69 [12:14:47] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 10.90, 15.25, 10.31 [12:15:25] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 17.79, 18.34, 11.70 [12:15:41] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 20.34, 19.24, 12.10 [12:16:08] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [12:17:03] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [12:17:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.06, 19.61, 12.80 [12:19:50] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:02] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 9 backends are down. mw151 mw171 mw172 mw182 mw153 mw163 mw173 mw183 mw184 [12:21:03] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 3 backends are down. mw181 mw164 mw174 [12:21:10] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 10.37, 16.89, 13.43 [12:23:55] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.556 second response time [12:24:08] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:27:38] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 5.77, 9.81, 9.82 [12:27:56] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4114 bytes in 0.058 second response time [12:32:05] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:34:04] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 41% [12:34:07] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:36:04] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4114 bytes in 0.260 second response time [12:38:06] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4114 bytes in 0.062 second response time [12:39:51] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [12:41:47] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 31% [12:42:05] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:44:07] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4114 bytes in 6.137 second response time [12:51:00] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp37.wikitide.net port 443 after 0 ms: Couldn't connect to server [12:51:03] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [12:51:21] PROBLEM - cp37 Nginx Backend for swiftproxy171 on cp37 is CRITICAL: connect to address localhost and port 8207: Connection refused [12:51:25] PROBLEM - cp37 Nginx Backend for mwtask181 on cp37 is CRITICAL: connect to address localhost and port 8160: Connection refused [12:51:28] PROBLEM - cp37 Nginx Backend for phorge171 on cp37 is CRITICAL: connect to address localhost and port 8202: Connection refused [12:52:55] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4114 bytes in 0.280 second response time [12:53:21] RECOVERY - cp37 Nginx Backend for swiftproxy171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8207 [12:53:25] RECOVERY - cp37 Nginx Backend for mwtask181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8160 [12:53:28] RECOVERY - cp37 Nginx Backend for phorge171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8202 [12:55:31] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [13:34:29] miraheze/ImportDump - dependabot[bot] the build passed. [13:49:27] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.99, 17.12, 10.42 [13:50:19] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [13:50:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:50:31] miraheze/mw-config - paladox the build passed. [13:50:37] !log [paladox@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [13:50:38] !log [paladox@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s [13:50:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:50:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:50:41] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s [13:50:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:50:57] miraheze/mw-config - paladox the build passed. [13:51:21] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 5.35, 12.76, 9.64 [14:52:45] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27204MiB (5% inode=99%); [15:24:36] PROBLEM - changeprop151 Current Load on changeprop151 is CRITICAL: LOAD CRITICAL - total load average: 13.56, 17.95, 9.33 [15:26:36] RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 2.56, 12.32, 8.31 [15:30:44] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 20.61, 18.82, 11.54 [15:32:44] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 3.85, 13.39, 10.46 [16:16:09] PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to matomo151.wikitide.net port 443 after 0 ms: Couldn't connect to server [16:22:09] RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.274 second response time [16:22:20] !log cache static.wikitide.net for 5 minutes; cache /1.43/ for 30 minutes [16:22:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:26:28] PROBLEM - swiftproxy161 HTTPS on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:27:44] PROBLEM - swiftproxy161 Swift NGINX SSL check on swiftproxy161 is CRITICAL: connect to address localhost and port 443: Connection refused [16:31:44] RECOVERY - swiftproxy161 Swift NGINX SSL check on swiftproxy161 is OK: OK - Certificate 'wikitide.net' will expire on Fri Mar 28 03:25:53 2025 +0000.TCP OK - 0.011 second response time on localhost port 443 [16:32:19] RECOVERY - swiftproxy161 HTTPS on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.022 second response time [16:50:56] PROBLEM - lotr.su - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lotr.su' expires in 14 day(s) (Wed 12 Mar 2025 04:28:42 PM GMT +0000). [16:51:11] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/058d02b90be928a12785140b6cb478b455c57972 [16:51:11] 02ssl/03master 07WikiTideSSLBot 03058d02b Bot: Update SSL cert for lotr.su [16:51:50] [02MatomoAnalytics] 07dependabot[bot] created 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0 (+1 new commit) 13https://github.com/miraheze/MatomoAnalytics/commit/988ebb3bb556 [16:51:50] 02MatomoAnalytics/03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0 07dependabot[bot] 03988ebb3 Bump eslint-config-wikimedia from 0.28.2 to 0.29.0… [16:51:52] [02MatomoAnalytics] 07dependabot[bot] added the label 'dependencies' to pull request #157 (Bump eslint-config-wikimedia from 0.28.2 to 0.29.0) 13https://github.com/miraheze/MatomoAnalytics/pull/157 [16:51:52] [02MatomoAnalytics] 07dependabot[bot] added the label 'javascript' to pull request #157 (Bump eslint-config-wikimedia from 0.28.2 to 0.29.0) 13https://github.com/miraheze/MatomoAnalytics/pull/157 [16:51:54] [02MatomoAnalytics] 07dependabot[bot] opened pull request #157: Bump eslint-config-wikimedia from 0.28.2 to 0.29.0 (03master...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0) 13https://github.com/miraheze/MatomoAnalytics/pull/157 [16:51:59] [02MatomoAnalytics] 07coderabbitai[bot] commented on pull request #157: --- […] 13https://github.com/miraheze/MatomoAnalytics/pull/157#issuecomment-2682645841 [16:56:49] miraheze/MatomoAnalytics - dependabot[bot] the build passed. [17:01:09] [02puppet] 07paladox created 03paladox-patch-5 (+1 new commit) 13https://github.com/miraheze/puppet/commit/300880a66c6e [17:01:09] 02puppet/03paladox-patch-5 07paladox 03300880a nginx: Change ssl_session_cache depending on how much ram [17:01:17] [02puppet] 07paladox opened pull request #4194: nginx: Change ssl_session_cache depending on how much ram (03master...03paladox-patch-5) 13https://github.com/miraheze/puppet/pull/4194 [17:01:25] [02puppet] 07coderabbitai[bot] commented on pull request #4194: --- […] 13https://github.com/miraheze/puppet/pull/4194#issuecomment-2682671956 [17:01:33] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/891182fe2f8116a9bfc6c4f80d983f5917ca13d9 [17:01:33] 02puppet/03master 07paladox 03891182f nginx: Change ssl_session_cache depending on how much ram (#4194) [17:01:35] [02puppet] 07paladox 04deleted 03paladox-patch-5 at 03300880a 13https://github.com/miraheze/puppet/commit/300880a [17:03:10] [02puppet] 07paladox merged pull request #4194: nginx: Change ssl_session_cache depending on how much ram (03master...03paladox-patch-5) 13https://github.com/miraheze/puppet/pull/4194 [17:06:42] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/643eb0f95b3345b991caf72326520c1a96b0809c [17:06:43] 02puppet/03master 07paladox 03643eb0f matomo151: lower php::fpm::fpm_workers_multiplier to 5.0 [17:19:01] [02ManageWiki] 07dependabot[bot] created 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0 (+1 new commit) 13https://github.com/miraheze/ManageWiki/commit/94835912d255 [17:19:03] 02ManageWiki/03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0 07dependabot[bot] 039483591 Bump eslint-config-wikimedia from 0.28.2 to 0.29.0… [17:19:05] [02ManageWiki] 07dependabot[bot] opened pull request #535: Bump eslint-config-wikimedia from 0.28.2 to 0.29.0 (03master...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0) 13https://github.com/miraheze/ManageWiki/pull/535 [17:19:07] [02ManageWiki] 07dependabot[bot] added the label 'dependencies' to pull request #535 (Bump eslint-config-wikimedia from 0.28.2 to 0.29.0) 13https://github.com/miraheze/ManageWiki/pull/535 [17:19:09] [02ManageWiki] 07dependabot[bot] added the label 'javascript' to pull request #535 (Bump eslint-config-wikimedia from 0.28.2 to 0.29.0) 13https://github.com/miraheze/ManageWiki/pull/535 [17:19:11] [02ManageWiki] 07coderabbitai[bot] commented on pull request #535: --- […] 13https://github.com/miraheze/ManageWiki/pull/535#issuecomment-2682719700 [17:20:18] RECOVERY - lotr.su - LetsEncrypt on sslhost is OK: OK - Certificate 'lotr.su' will expire on Mon 26 May 2025 03:52:34 PM GMT +0000. [17:23:34] [02puppet] 07paladox created 03paladox-patch-5 (+1 new commit) 13https://github.com/miraheze/puppet/commit/55e8661e16b4 [17:23:34] 02puppet/03paladox-patch-5 07paladox 0355e8661 matomo: run multiple queued process traffic requests [17:23:38] [02puppet] 07paladox opened pull request #4195: matomo: run multiple queued process traffic requests (03master...03paladox-patch-5) 13https://github.com/miraheze/puppet/pull/4195 [17:23:45] [02puppet] 07coderabbitai[bot] commented on pull request #4195: --- […] 13https://github.com/miraheze/puppet/pull/4195#issuecomment-2682731117 [17:24:06] miraheze/ManageWiki - dependabot[bot] the build passed. [17:24:15] [02puppet] 07github-actions[bot] pushed 1 new commit to 03paladox-patch-5 13https://github.com/miraheze/puppet/commit/2b2d925984f8feb7efb017f566a5de2e4d209342 [17:24:16] 02puppet/03paladox-patch-5 07github-actions 032b2d925 CI: lint puppet code to standards… [17:24:34] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-5 13https://github.com/miraheze/puppet/commit/f8c9684bf04dfee2e57fecdcb980e1ea8acbbf04 [17:24:35] 02puppet/03paladox-patch-5 07paladox 03f8c9684 Update init.pp [17:25:00] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-5 13https://github.com/miraheze/puppet/commit/a9032a267ec618f3f586ef0587bdef418a095de5 [17:25:01] 02puppet/03paladox-patch-5 07paladox 03a9032a2 Update init.pp [17:25:26] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-5 13https://github.com/miraheze/puppet/commit/bd402e43cf08d26c24cd0cd151d982403efffefb [17:25:26] 02puppet/03paladox-patch-5 07paladox 03bd402e4 Update init.pp [17:26:01] [02puppet] 07paladox merged pull request #4195: matomo: run multiple queued process traffic requests (03master...03paladox-patch-5) 13https://github.com/miraheze/puppet/pull/4195 [17:26:01] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/031e849874aafb330c11b4c2ffadfcb8b7c3540c [17:26:01] 02puppet/03master 07paladox 03031e849 matomo: run multiple queued process traffic requests (#4195)… [17:26:02] [02puppet] 07paladox 04deleted 03paladox-patch-5 at 03bd402e4 13https://github.com/miraheze/puppet/commit/bd402e4 [17:26:30] [02ManageWiki] 07paladox merged 07dependabot[bot]'s pull request #535: Bump eslint-config-wikimedia from 0.28.2 to 0.29.0 (03master...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0) 13https://github.com/miraheze/ManageWiki/pull/535 [17:26:31] [02ManageWiki] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/ManageWiki/commit/16d4104d2226718f55b10e6c0451f681adc73db6 [17:26:31] 02ManageWiki/03master 07dependabot[bot] 0316d4104 Bump eslint-config-wikimedia from 0.28.2 to 0.29.0 (#535)… [17:26:32] [02ManageWiki] 07paladox 04deleted 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0 at 039483591 13https://github.com/miraheze/ManageWiki/commit/9483591 [17:26:43] [02MatomoAnalytics] 07paladox merged 07dependabot[bot]'s pull request #157: Bump eslint-config-wikimedia from 0.28.2 to 0.29.0 (03master...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0) 13https://github.com/miraheze/MatomoAnalytics/pull/157 [17:26:43] [02MatomoAnalytics] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/MatomoAnalytics/commit/c97a9339b4f5564cccdceb26e535b637a6105ec0 [17:26:43] 02MatomoAnalytics/03master 07dependabot[bot] 03c97a933 Bump eslint-config-wikimedia from 0.28.2 to 0.29.0 (#157)… [17:26:49] [02MatomoAnalytics] 07dependabot[bot] 04deleted 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0 at 03988ebb3 13https://github.com/miraheze/MatomoAnalytics/commit/988ebb3 [17:27:01] [02ImportDump] 07paladox merged 07dependabot[bot]'s pull request #131: Bump eslint-config-wikimedia from 0.28.2 to 0.29.0 (03master...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0) 13https://github.com/miraheze/ImportDump/pull/131 [17:27:02] [02ImportDump] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/ImportDump/commit/e8e0e78f31692362976fdabcfeb8a274c9fd5124 [17:27:02] 02ImportDump/03master 07dependabot[bot] 03e8e0e78 Bump eslint-config-wikimedia from 0.28.2 to 0.29.0 (#131)… [17:27:03] [02ImportDump] 07paladox 04deleted 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0 at 035d159d8 13https://github.com/miraheze/ImportDump/commit/5d159d8 [17:27:50] [02puppet] 07paladox created 03paladox-patch-5 (+1 new commit) 13https://github.com/miraheze/puppet/commit/c1cb079bedc4 [17:27:50] 02puppet/03paladox-patch-5 07paladox 03c1cb079 matomo: remove absented systemd timer [17:27:55] [02puppet] 07paladox opened pull request #4196: matomo: remove absented systemd timer (03master...03paladox-patch-5) 13https://github.com/miraheze/puppet/pull/4196 [17:28:02] [02puppet] 07coderabbitai[bot] commented on pull request #4196: --- […] 13https://github.com/miraheze/puppet/pull/4196#issuecomment-2682744211 [17:31:29] miraheze/MatomoAnalytics - paladox the build passed. [17:31:36] miraheze/ManageWiki - paladox the build passed. [17:36:47] [02puppet] 07paladox merged pull request #4196: matomo: remove absented systemd timer (03master...03paladox-patch-5) 13https://github.com/miraheze/puppet/pull/4196 [17:36:48] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/0b0b1bddb3dbe36146e4b8b7762f8edbb78b2aee [17:36:48] 02puppet/03master 07paladox 030b0b1bd matomo: remove absented systemd timer (#4196) [17:36:48] [02puppet] 07paladox 04deleted 03paladox-patch-5 at 03c1cb079 13https://github.com/miraheze/puppet/commit/c1cb079 [17:39:38] miraheze/ImportDump - paladox the build passed. [18:33:17] PROBLEM - www.christipedia.nl - Cloudflare on sslhost is WARNING: WARNING - Certificate 'www.christipedia.nl' expires in 29 day(s) (Thu 27 Mar 2025 06:05:29 PM GMT +0000). [21:17:21] [02puppet] 07paladox created 03paladox-patch-5 (+1 new commit) 13https://github.com/miraheze/puppet/commit/160941c0c6d5 [21:17:21] 02puppet/03paladox-patch-5 07paladox 03160941c varnish: Set Cache-Control: no-transform header… [21:17:25] [02puppet] 07paladox opened pull request #4197: varnish: Set Cache-Control: no-transform header (03master...03paladox-patch-5) 13https://github.com/miraheze/puppet/pull/4197 [21:17:31] [02puppet] 07coderabbitai[bot] commented on pull request #4197: --- […] 13https://github.com/miraheze/puppet/pull/4197#issuecomment-2683313965 [21:17:48] [02puppet] 07paladox merged pull request #4197: varnish: Set Cache-Control: no-transform header (03master...03paladox-patch-5) 13https://github.com/miraheze/puppet/pull/4197 [21:17:48] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/e45e949390fe6c9dd846343fbbe1b74ad801dc42 [21:17:48] 02puppet/03master 07paladox 03e45e949 varnish: Set Cache-Control: no-transform header (#4197)… [21:17:49] [02puppet] 07paladox 04deleted 03paladox-patch-5 at 03160941c 13https://github.com/miraheze/puppet/commit/160941c [23:22:55] [02puppet] 07paladox created 03paladox-patch-5 (+1 new commit) 13https://github.com/miraheze/puppet/commit/ee4dd2a1e380 [23:22:55] 02puppet/03paladox-patch-5 07paladox 03ee4dd2a wikitide-backup: Support backing up mattermost and puppetdb [23:22:59] [02puppet] 07paladox opened pull request #4198: wikitide-backup: Support backing up mattermost and puppetdb (03master...03paladox-patch-5) 13https://github.com/miraheze/puppet/pull/4198 [23:23:05] [02puppet] 07coderabbitai[bot] commented on pull request #4198: --- […] 13https://github.com/miraheze/puppet/pull/4198#issuecomment-2683513846 [23:40:38] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9705MiB (10% inode=98%); [23:49:40] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-5 13https://github.com/miraheze/puppet/commit/ddd2fe9d26d10e87c1fe223d342aa9bdbce71745 [23:49:40] 02puppet/03paladox-patch-5 07paladox 03ddd2fe9 Update wikitide-backup.py.erb [23:50:16] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-5 13https://github.com/miraheze/puppet/commit/1d5e676097f6de044ef58f3317b4c75a6305905b [23:50:16] 02puppet/03paladox-patch-5 07paladox 031d5e676 Update wikitide-backup.py.erb [23:53:46] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-5 13https://github.com/miraheze/puppet/commit/568104e52d83afd315a9805f77516774e1f42b87 [23:53:46] 02puppet/03paladox-patch-5 07paladox 03568104e Update mattermost.pp [23:55:13] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-5 13https://github.com/miraheze/puppet/commit/a9078cc2388b5ae036c5c5d3ab4730675f8af914 [23:55:13] 02puppet/03paladox-patch-5 07paladox 03a9078cc Update init.pp