[00:01:11] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/affc349cb7c5528eeb7f8ac6a185431ed67d869a [00:01:11] 02statichelp/03main 07WikiTideBot 03affc349 Bot: Auto-update Tech namespace pages 2025-06-26 00:01:09 [00:05:21] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 58266MiB (12% inode=99%); [00:05:24] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 57840MiB (12% inode=99%); [00:05:46] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 57601MiB (12% inode=99%); [01:49:15] [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/8d37873a671434464997a27a845ea76911812dca [01:49:15] 02ssl/03main 07MacFan4000 038d37873 add wiki.spacestation13.pl and pixeldungeon.wiki [02:04:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750899860000&orgId=1&to=1750903493355 [02:34:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750901660000&orgId=1&to=1750905260000 [03:32:25] PROBLEM - db181 Backups SQL on db181 is WARNING: FILE_AGE WARNING: /var/log/db-backups/db-backups/db-backups.log is 1382584 seconds old and 4683001 bytes [06:05:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750914320000&orgId=1&to=1750917953357 [06:20:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750915100000&orgId=1&to=1750918700000 [07:30:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750919420000&orgId=1&to=1750923053355 [08:15:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750922120000&orgId=1&to=1750925720000 [09:31:03] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 49% [09:33:03] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 22% [10:55:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750931720000&orgId=1&to=1750935353360 [11:00:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750932020000&orgId=1&to=1750935620000 [11:38:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750934300000&orgId=1&to=1750937933352 [11:58:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750935440000&orgId=1&to=1750939040000 [12:12:28] [02DiscordNotifications] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/b87a3169ec6e29ba5e34295eacc12de3372ed870 [12:12:28] 02DiscordNotifications/03main 07translatewiki.net 03b87a316 Localisation updates from https://translatewiki.net. [12:12:29] [02CreateWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/4a86e3690ef6c8d7d300cf81570014fc1aa20f26 [12:12:29] 02CreateWiki/03main 07translatewiki.net 034a86e36 Localisation updates from https://translatewiki.net. [12:12:31] [02landing] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/landing/commit/96a3d9247e3cd98c155948f89d4f6587174988fc [12:12:33] 02landing/03main 07translatewiki.net 0396a3d92 Localisation updates from https://translatewiki.net. [12:12:35] [02ManageWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/d4b658bad0dc3f2cbda62ff8da3d52b1204cb4d6 [12:12:37] [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/a78db1b00a9aac642a4e8efb15e34543c88b1455 [12:12:38] 02ManageWiki/03main 07translatewiki.net 03d4b658b Localisation updates from https://translatewiki.net. [12:12:40] 02MirahezeMagic/03main 07translatewiki.net 03a78db1b Localisation updates from https://translatewiki.net. [12:13:25] miraheze/landing - translatewiki the build passed. [12:17:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750936640000&orgId=1&to=1750940273357 [12:18:05] miraheze/ManageWiki - translatewiki the build passed. [12:24:49] miraheze/MirahezeMagic - translatewiki the build passed. [12:31:17] miraheze/CreateWiki - translatewiki the build passed. [12:35:42] miraheze/DiscordNotifications - translatewiki the build passed. [12:47:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750938320000&orgId=1&to=1750941920000 [12:58:26] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49828MiB (10% inode=99%); [13:09:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750939760000&orgId=1&to=1750943393355 [13:30:21] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49869MiB (10% inode=99%); [13:57:17] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49927MiB (10% inode=99%); [13:59:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750942640000&orgId=1&to=1750946240000 [15:11:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 38.46.223.205/cpweb [15:13:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:49:39] [02puppet] 07SomeMWDev opened pull request #4430: T13893: Add m.mediawiki.org to the CSP (07miraheze:03main...07SomeMWDev:03patch-1) 13https://github.com/miraheze/puppet/pull/4430 [15:50:08] [02puppet] 07coderabbitai[bot] commented on pull request #4430:

[…] 13https://github.com/miraheze/puppet/pull/4430#issuecomment-3008958245 [15:54:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750949660000&orgId=1&to=1750953293352 [16:19:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750951100000&orgId=1&to=1750954700000 [16:37:40] [02puppet] 07SomeMWDev retitled PR #4430: "T13893: Add m.mediawiki.org to the CSP" ➜ "T13893: Add *.mediawiki.org to the CSP" 13https://github.com/miraheze/puppet/pull/4430 [16:40:46] [02puppet] 07RhinosF1 merged 07SomeMWDev's pull request #4430: T13893: Add *.mediawiki.org to the CSP (07miraheze:03main...07SomeMWDev:03patch-1) 13https://github.com/miraheze/puppet/pull/4430 [16:40:48] [02puppet] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/55c8c500a1e7d02243a8976bb0e00efbe5c99753 [16:40:49] 02puppet/03main 07SomeRandomDeveloper 0355c8c50 T13893: Add *.mediawiki.org to the CSP (#4430)… [17:00:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750953620000&orgId=1&to=1750957253349 [17:07:55] [02mw-config] 07Universal-Omega created 03chunk-xml (+1 new commit) 13https://github.com/miraheze/mw-config/commit/baa497d1d4e4 [17:07:55] 02mw-config/03chunk-xml 07CosmicAlpha 03baa497d Allow chunking XML dumps [17:07:59] [02mw-config] 07Universal-Omega opened pull request #6005: Allow chunking XML dumps (03main...03chunk-xml) 13https://github.com/miraheze/mw-config/pull/6005 [17:08:48] miraheze/mw-config - Universal-Omega the build passed. [17:20:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750954700000&orgId=1&to=1750958300000 [17:55:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750956920000&orgId=1&to=1750960553358 [18:15:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750956920000&orgId=1&to=1750961720000 [18:22:02] PROBLEM - cloud15 IPMI Sensors on cloud15 is WARNING: IPMI Status: Warning [1 system event log (SEL) entry present] [18:36:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750959380000&orgId=1&to=1750963013361 [18:42:02] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [2 system event log (SEL) entries present] [18:51:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750960100000&orgId=1&to=1750963700000 [18:53:51] [02puppet] 07paladox created 03paladox-patch-11 (+1 new commit) 13https://github.com/miraheze/puppet/commit/801dcd34f301 [18:53:52] 02puppet/03paladox-patch-11 07paladox 03801dcd3 Depool mem151/mem201 [18:53:55] [02puppet] 07paladox opened pull request #4431: Depool mem151/mem201 (03main...03paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4431 [18:54:25] [02puppet] 07coderabbitai[bot] commented on pull request #4431:

[…] 13https://github.com/miraheze/puppet/pull/4431#issuecomment-3009570348 [18:54:43] [02mw-config] 07paladox created 03paladox-patch-4 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/fe649593af37 [18:54:43] 02mw-config/03paladox-patch-4 07paladox 03fe64959 Depool c1 (db151) [18:54:47] [02mw-config] 07paladox opened pull request #6006: Depool c1 (db151) (03main...03paladox-patch-4) 13https://github.com/miraheze/mw-config/pull/6006 [18:55:49] miraheze/mw-config - paladox the build passed. [18:59:53] [02puppet] 07paladox merged pull request #4431: Depool mem151/mem201 (03main...03paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4431 [18:59:55] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/c567f98f6677f91df84cc194e7ebf864a7c485de [18:59:56] 02puppet/03main 07paladox 03c567f98 Depool mem151/mem201 (#4431) [18:59:58] [02puppet] 07paladox 04deleted 03paladox-patch-11 at 03801dcd3 13https://api.github.com/repos/miraheze/puppet/commit/801dcd3 [19:00:08] [02mw-config] 07paladox merged pull request #6006: Depool c1 (db151) (03main...03paladox-patch-4) 13https://github.com/miraheze/mw-config/pull/6006 [19:00:08] [02mw-config] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/bc47d351b74a15c29f760b0f130e8fc0f5e6194c [19:00:08] 02mw-config/03main 07paladox 03bc47d35 Depool c1 (db151) (#6006) [19:00:10] [02mw-config] 07paladox 04deleted 03paladox-patch-4 at 03fe64959 13https://api.github.com/repos/miraheze/mw-config/commit/fe64959 [19:00:22] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:00:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:00:32] !log shutting down all vms on cloud15 and cloud20. Shutting down both the cloud servers as well. [19:00:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:00:47] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [19:00:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:01:05] miraheze/mw-config - paladox the build passed. [19:02:34] PROBLEM - cp201 Disk Space on cp201 is CRITICAL: connect to address 10.0.20.166 port 5666: No route to hostconnect to host 10.0.20.166 port 5666: No route to host [19:02:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750960880000&orgId=1&to=1750964480000[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750960860000&orgId=1&to=1750964573364[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/G [19:02:53] rom=1750960880000&orgId=1&to=1750964573364 [19:03:08] PROBLEM - matomo151 Puppet on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [19:03:09] PROBLEM - mw152 APT on mw152 is CRITICAL: connect to address 10.0.15.115 port 5666: No route to hostconnect to host 10.0.15.115 port 5666: No route to host [19:03:13] PROBLEM - ping on os151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.111) [19:03:14] PROBLEM - mw151 PowerDNS Recursor on mw151 is CRITICAL: connect to address 10.0.15.114 port 5666: No route to hostconnect to host 10.0.15.114 port 5666: No route to host [19:03:18] PROBLEM - os151 PowerDNS Recursor on os151 is CRITICAL: connect to address 10.0.15.111 port 5666: No route to hostconnect to host 10.0.15.111 port 5666: No route to host [19:03:20] PROBLEM - matomo151 SSH on matomo151 is CRITICAL: connect to address 10.0.15.112 and port 22: No route to host [19:03:23] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw153.fsslc.wtnet port 443 after 2919 ms: Couldn't connect to server [19:03:23] PROBLEM - prometheus151 Prometheus on prometheus151 is CRITICAL: connect to address 10.0.15.116 and port 9090: No route to host [19:03:24] PROBLEM - ping on mw151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.114) [19:03:24] PROBLEM - mem151 Disk Space on mem151 is CRITICAL: connect to address 10.0.15.113 port 5666: No route to hostconnect to host 10.0.15.113 port 5666: No route to host [19:03:25] PROBLEM - ping on mem151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.113) [19:03:25] PROBLEM - mem151 SSH on mem151 is CRITICAL: connect to address 10.0.15.113 and port 22: No route to host [19:03:25] PROBLEM - os151 Disk Space on os151 is CRITICAL: connect to address 10.0.15.111 port 5666: No route to hostconnect to host 10.0.15.111 port 5666: No route to host [19:03:25] PROBLEM - os151 ferm_active on os151 is CRITICAL: connect to address 10.0.15.111 port 5666: No route to hostconnect to host 10.0.15.111 port 5666: No route to host [19:03:26] PROBLEM - db151 Check unit status of db-backups on db151 is CRITICAL: connect to address 10.0.15.110 port 5666: No route to hostconnect to host 10.0.15.110 port 5666: No route to host [19:03:28] PROBLEM - os151 Current Load on os151 is CRITICAL: connect to address 10.0.15.111 port 5666: No route to hostconnect to host 10.0.15.111 port 5666: No route to host [19:03:29] PROBLEM - ping on mw152 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.115) [19:03:30] PROBLEM - mem201 NTP time on mem201 is CRITICAL: connect to address 10.0.20.148 port 5666: No route to hostconnect to host 10.0.20.148 port 5666: No route to host [19:03:33] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: connect to address 10.0.15.140 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [19:03:34] PROBLEM - mem151 Current Load on mem151 is CRITICAL: connect to address 10.0.15.113 port 5666: No route to hostconnect to host 10.0.15.113 port 5666: No route to host [19:03:37] PROBLEM - mem151 ferm_active on mem151 is CRITICAL: connect to address 10.0.15.113 port 5666: No route to hostconnect to host 10.0.15.113 port 5666: No route to host [19:03:37] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: connect to address 10.0.20.166 port 5666: No route to hostconnect to host 10.0.20.166 port 5666: No route to host [19:03:39] PROBLEM - os151 APT on os151 is CRITICAL: connect to address 10.0.15.111 port 5666: No route to hostconnect to host 10.0.15.111 port 5666: No route to host [19:03:39] PROBLEM - ping on swiftobject151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.117) [19:03:40] PROBLEM - swiftobject151 conntrack_table_size on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [19:03:40] PROBLEM - swiftobject151 NTP time on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [19:03:40] PROBLEM - Host mem151 is DOWN: CRITICAL - Host Unreachable (10.0.15.113) [19:03:40] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 12 backends are down. mw151 mw152 mw161 mw171 mw172 mw182 mw153 mw163 mw191 mw201 mw202 mw203 [19:03:42] PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [19:03:42] PROBLEM - ping on prometheus151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.116) [19:03:42] PROBLEM - mw152 conntrack_table_size on mw152 is CRITICAL: connect to address 10.0.15.115 port 5666: No route to hostconnect to host 10.0.15.115 port 5666: No route to host [19:03:43] PROBLEM - mw152 Disk Space on mw152 is CRITICAL: connect to address 10.0.15.115 port 5666: No route to hostconnect to host 10.0.15.115 port 5666: No route to host [19:03:43] PROBLEM - swiftobject151 Current Load on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [19:03:43] PROBLEM - swiftobject151 ferm_active on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [19:03:43] PROBLEM - swiftobject151 Puppet on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [19:03:54] PROBLEM - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [19:03:54] PROBLEM - test151 ferm_active on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [19:03:55] PROBLEM - mw153 Disk Space on mw153 is CRITICAL: connect to address 10.0.15.140 port 5666: No route to hostconnect to host 10.0.15.140 port 5666: No route to host [19:03:55] PROBLEM - mw153 Current Load on mw153 is CRITICAL: connect to address 10.0.15.140 port 5666: No route to hostconnect to host 10.0.15.140 port 5666: No route to host [19:03:56] PROBLEM - mw153 Puppet on mw153 is CRITICAL: connect to address 10.0.15.140 port 5666: No route to hostconnect to host 10.0.15.140 port 5666: No route to host [19:03:56] PROBLEM - os201 Current Load on os201 is CRITICAL: connect to address 10.0.20.156 port 5666: No route to hostconnect to host 10.0.20.156 port 5666: No route to host [19:03:57] PROBLEM - Host os151 is DOWN: CRITICAL - Host Unreachable (10.0.15.111) [19:03:57] PROBLEM - Host prometheus151 is DOWN: CRITICAL - Host Unreachable (10.0.15.116) [19:03:58] PROBLEM - os161 Puppet on os161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Opensearch_template[graylog-internal] [19:03:58] PROBLEM - matomo151 Disk Space on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [19:03:59] PROBLEM - matomo151 Check unit status of matomo-optimize-last2 on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [19:03:59] PROBLEM - matomo151 NTP time on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [19:04:00] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [19:04:00] PROBLEM - swiftobject151 Swift Object Service on swiftobject151 is CRITICAL: connect to address 10.0.15.117 and port 6000: No route to host [19:04:01] PROBLEM - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [19:04:01] PROBLEM - test151 Check unit status of mediawiki_job_cleanup-upload-stash on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [19:04:12] PROBLEM - Host mem201 is DOWN: CRITICAL - Host Unreachable (10.0.20.148) [19:04:13] PROBLEM - test151 php-fpm on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [19:04:13] PROBLEM - mw202 SSH on mw202 is CRITICAL: connect to address 10.0.20.163 and port 22: No route to host [19:04:14] PROBLEM - Host mw202 is DOWN: CRITICAL - Host Unreachable (10.0.20.163) [19:04:14] PROBLEM - mw152 Puppet on mw152 is CRITICAL: connect to address 10.0.15.115 port 5666: No route to hostconnect to host 10.0.15.115 port 5666: No route to host [19:04:15] PROBLEM - test151 Puppet on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [19:04:15] PROBLEM - test151 APT on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [19:04:16] PROBLEM - os201 Puppet on os201 is CRITICAL: connect to address 10.0.20.156 port 5666: No route to hostconnect to host 10.0.20.156 port 5666: No route to host [19:04:16] PROBLEM - os201 SSH on os201 is CRITICAL: connect to address 10.0.20.156 and port 22: No route to host [19:04:17] PROBLEM - test151 Check unit status of mediawiki_job_purge-abusefilter on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [19:04:17] PROBLEM - Host db151 is DOWN: CRITICAL - Host Unreachable (10.0.15.110) [19:04:18] PROBLEM - db151 APT on db151 is CRITICAL: connect to address 10.0.15.110 port 5666: No route to hostconnect to host 10.0.15.110 port 5666: No route to host [19:04:18] PROBLEM - Host mw153 is DOWN: CRITICAL - Host Unreachable (10.0.15.140) [19:04:19] PROBLEM - mw152 NTP time on mw152 is CRITICAL: connect to address 10.0.15.115 port 5666: No route to hostconnect to host 10.0.15.115 port 5666: No route to host [19:04:19] PROBLEM - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [19:04:20] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [19:04:20] PROBLEM - changeprop201 Haproxy backend for localhost:9008 on changeprop201 is CRITICAL: connect to address 10.0.20.149 port 5666: No route to hostconnect to host 10.0.20.149 port 5666: No route to host [19:04:21] PROBLEM - cp201 Haproxy TLS backend for mw192 on cp201 is CRITICAL: connect to address 10.0.20.166 port 5666: No route to hostconnect to host 10.0.20.166 port 5666: No route to host [19:04:32] PROBLEM - test151 Check unit status of prometheus_jobqueue_stats on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [19:04:33] PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to test151.fsslc.wtnet port 443 after 2945 ms: Couldn't connect to server [19:04:34] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [19:04:34] PROBLEM - matomo151 Check unit status of matomo-queuedtracking-2 on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [19:04:34] PROBLEM - matomo151 Check unit status of matomo-queuedtracking-1 on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [19:04:35] PROBLEM - mw203 Disk Space on mw203 is CRITICAL: connect to address 10.0.20.165 port 5666: No route to hostconnect to host 10.0.20.165 port 5666: No route to host [19:04:37] PROBLEM - os201 Disk Space on os201 is CRITICAL: connect to address 10.0.20.156 port 5666: No route to hostconnect to host 10.0.20.156 port 5666: No route to host [19:04:38] PROBLEM - test151 SSH on test151 is CRITICAL: connect to address 10.0.15.118 and port 22: No route to host [19:04:39] PROBLEM - test151 memcached on test151 is CRITICAL: connect to address 10.0.15.118 and port 11211: No route to host [19:04:39] PROBLEM - test151 conntrack_table_size on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [19:04:39] PROBLEM - Host mw201 is DOWN: CRITICAL - Host Unreachable (10.0.20.162) [19:04:39] PROBLEM - matomo151 php-fpm on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [19:04:39] PROBLEM - matomo151 Redis Process on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [19:04:40] PROBLEM - matomo151 conntrack_table_size on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [19:04:40] PROBLEM - Host mw152 is DOWN: CRITICAL - Host Unreachable (10.0.15.115) [19:04:41] PROBLEM - mwtask151 mathoid on mwtask151 is CRITICAL: connect to address 10.0.15.150 and port 10044: No route to host [19:04:41] PROBLEM - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 7 - Failed to connect to 10.0.15.150 port 9005 after 294 ms: Couldn't connect to server [19:04:42] PROBLEM - test151 Check unit status of mediawiki_job_update-wikibase-sites-table on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [19:04:42] PROBLEM - Host test151 is DOWN: CRITICAL - Host Unreachable (10.0.15.118) [19:09:42] I muted icinga [19:45:59] i think there are no routes to host [19:46:00] !log upgrade proxmox-kernel-6.8 on cloud20 [19:52:11] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/7b58b04b1aac4dfe5e6ceb64c9140bb75ed4d196 [19:52:11] 02puppet/03main 07paladox 037b58b04 Revert "Depool mem151/mem201 (#4431)"… [19:52:33] [02mw-config] 07paladox created 03master (+1 new commit) 13https://github.com/miraheze/mw-config/commit/80e65f3df98b [19:52:33] 02mw-config/03master 07paladox 0380e65f3 Revert "Depool c1 (db151) (#6006)"… [19:52:42] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:52:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:13] !log [paladox@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw201 [19:53:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:31] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:53:33] !log [paladox@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw152 [19:53:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:40] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [19:53:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:54:25] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 45s [19:54:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:54:56] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [19:54:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:17] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 21s [19:55:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:56] [02mw-config] 07paladox created 03paladox-patch-4 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/0507979cd403 [19:58:56] 02mw-config/03paladox-patch-4 07paladox 030507979 Remove site notice [19:59:00] [02mw-config] 07paladox opened pull request #6007: Remove site notice (03main...03paladox-patch-4) 13https://github.com/miraheze/mw-config/pull/6007 [19:59:54] miraheze/mw-config - paladox the build passed. [20:00:02] [02mw-config] 07paladox merged pull request #6007: Remove site notice (03main...03paladox-patch-4) 13https://github.com/miraheze/mw-config/pull/6007 [20:00:04] [02mw-config] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/bbf4601272061fe197b419dccbc07607bf41f70c [20:00:05] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [20:00:06] [02mw-config] 07paladox 04deleted 03paladox-patch-4 at 030507979 13https://api.github.com/repos/miraheze/mw-config/commit/0507979 [20:00:07] 02mw-config/03main 07paladox 03bbf4601 Remove site notice (#6007) [20:00:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:00:27] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s [20:00:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:01:07] miraheze/mw-config - paladox the build passed. [20:13:26] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [20:13:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:13:47] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 21s [20:13:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:14:20] [02mw-config] 07paladox 04deleted 03master at 0380e65f3 13https://api.github.com/repos/miraheze/mw-config/commit/80e65f3 [20:14:47] [02mw-config] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/f5f264ad59a89d5eb16c45bd026bd77d08d7878c [20:14:48] 02mw-config/03main 07paladox 03f5f264a Revert "Depool c1 (db151) (#6006)"… [20:14:48] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [20:14:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:15:10] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 21s [20:15:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:15:43] miraheze/mw-config - paladox the build passed. [21:09:24] PROBLEM - cp201 APT on cp201 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:09:41] PROBLEM - mw202 APT on mw202 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:09:58] PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: APT CRITICAL: 71 packages available for upgrade (1 critical updates). [21:10:28] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:10:37] PROBLEM - eventgate181 APT on eventgate181 is CRITICAL: APT CRITICAL: 67 packages available for upgrade (1 critical updates). [21:11:08] PROBLEM - db172 APT on db172 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:11:11] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:11:24] PROBLEM - swiftproxy171 APT on swiftproxy171 is CRITICAL: APT CRITICAL: 71 packages available for upgrade (1 critical updates). [21:11:33] PROBLEM - mw203 APT on mw203 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:12:32] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:12:35] PROBLEM - bast161 APT on bast161 is CRITICAL: APT CRITICAL: 46 packages available for upgrade (1 critical updates). [21:12:38] PROBLEM - cp171 APT on cp171 is CRITICAL: APT CRITICAL: 47 packages available for upgrade (1 critical updates). [21:12:49] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:12:53] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:13:11] PROBLEM - swiftobject191 APT on swiftobject191 is CRITICAL: APT CRITICAL: 67 packages available for upgrade (1 critical updates). [21:13:28] PROBLEM - mem161 APT on mem161 is CRITICAL: APT CRITICAL: 66 packages available for upgrade (1 critical updates). [21:13:50] PROBLEM - swiftobject201 APT on swiftobject201 is CRITICAL: APT CRITICAL: 67 packages available for upgrade (1 critical updates). [21:14:09] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 54 packages available for upgrade (1 critical updates). [21:14:37] PROBLEM - mw173 APT on mw173 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:15:03] PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: APT CRITICAL: 67 packages available for upgrade (1 critical updates). [21:15:38] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:15:54] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [21:16:06] PROBLEM - phorge171 APT on phorge171 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (1 critical updates). [21:16:46] PROBLEM - os201 APT on os201 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (1 critical updates). [21:17:02] PROBLEM - cloud15 APT on cloud15 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (1 critical updates). [21:17:27] PROBLEM - db161 APT on db161 is CRITICAL: APT CRITICAL: 74 packages available for upgrade (1 critical updates). [21:17:29] PROBLEM - cloud16 APT on cloud16 is CRITICAL: APT CRITICAL: 129 packages available for upgrade (1 critical updates). [21:17:50] PROBLEM - os151 APT on os151 is CRITICAL: APT CRITICAL: 49 packages available for upgrade (1 critical updates). [21:18:39] PROBLEM - mw193 APT on mw193 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:19:12] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 59 packages available for upgrade (1 critical updates). [21:19:21] PROBLEM - kafka181 APT on kafka181 is CRITICAL: APT CRITICAL: 66 packages available for upgrade (1 critical updates). [21:19:40] PROBLEM - mem151 APT on mem151 is CRITICAL: APT CRITICAL: 66 packages available for upgrade (1 critical updates). [21:21:29] PROBLEM - db171 APT on db171 is CRITICAL: APT CRITICAL: 55 packages available for upgrade (1 critical updates). [21:21:40] PROBLEM - cloud17 APT on cloud17 is CRITICAL: APT CRITICAL: 129 packages available for upgrade (1 critical updates). [21:21:59] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:22:17] PROBLEM - mw191 APT on mw191 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:22:28] PROBLEM - db182 APT on db182 is CRITICAL: APT CRITICAL: 55 packages available for upgrade (1 critical updates). [21:22:35] PROBLEM - cloud19 APT on cloud19 is CRITICAL: APT CRITICAL: 131 packages available for upgrade (1 critical updates). [21:22:51] PROBLEM - mw153 APT on mw153 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:25:15] PROBLEM - bots171 APT on bots171 is CRITICAL: APT CRITICAL: 49 packages available for upgrade (1 critical updates). [21:25:24] PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 129 packages available for upgrade (1 critical updates). [21:26:18] PROBLEM - rdb151 APT on rdb151 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:27:00] PROBLEM - cp191 APT on cp191 is CRITICAL: APT CRITICAL: 47 packages available for upgrade (1 critical updates). [21:27:24] PROBLEM - swiftobject171 APT on swiftobject171 is CRITICAL: APT CRITICAL: 67 packages available for upgrade (1 critical updates). [21:27:27] PROBLEM - mem191 APT on mem191 is CRITICAL: APT CRITICAL: 45 packages available for upgrade (1 critical updates). [21:28:14] PROBLEM - mw192 APT on mw192 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:28:23] PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (1 critical updates). [21:28:23] PROBLEM - cp161 APT on cp161 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:28:32] !log [universalomega@puppet181] Upgraded packages on cp161: libicu72 [21:28:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:28:41] PROBLEM - changeprop201 APT on changeprop201 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [21:28:43] !log [universalomega@puppet181] Upgraded packages on bots171: libicu72 [21:28:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:28:54] !log [universalomega@puppet181] Upgraded packages on db172: libicu72 [21:28:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:29:07] RECOVERY - db172 APT on db172 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:29:09] !log [universalomega@puppet181] Upgraded packages on eventgate181: libicu72 [21:29:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:29:14] RECOVERY - bots171 APT on bots171 is OK: APT OK: 48 packages available for upgrade (0 critical updates). [21:29:20] !log [universalomega@puppet181] Upgraded packages on db171: libicu72 [21:29:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:29:26] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:29:28] RECOVERY - db171 APT on db171 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [21:29:33] !log [universalomega@puppet181] Upgraded packages on cloud17: libicu72 [21:29:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:29:39] PROBLEM - db181 APT on db181 is CRITICAL: APT CRITICAL: 54 packages available for upgrade (1 critical updates). [21:29:39] PROBLEM - mw163 APT on mw163 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:29:40] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 128 packages available for upgrade (0 critical updates). [21:29:45] !log [universalomega@puppet181] Upgraded packages on changeprop201: libicu72 [21:29:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:29:57] !log [universalomega@puppet181] Upgraded packages on cp171: libicu72 [21:29:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:30:05] PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 49 packages available for upgrade (1 critical updates). [21:30:08] !log [universalomega@puppet181] Upgraded packages on db161: libicu72 [21:30:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:30:20] !log [universalomega@puppet181] Upgraded packages on db151: libicu72 [21:30:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:30:23] RECOVERY - cp161 APT on cp161 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [21:30:32] !log [universalomega@puppet181] Upgraded packages on cloud19: libicu72 [21:30:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:30:38] RECOVERY - cp171 APT on cp171 is OK: APT OK: 46 packages available for upgrade (0 critical updates). [21:30:39] PROBLEM - mattermost1 APT on mattermost1 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (1 critical updates). [21:30:41] RECOVERY - cloud19 APT on cloud19 is OK: APT OK: 130 packages available for upgrade (0 critical updates). [21:30:42] RECOVERY - changeprop201 APT on changeprop201 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [21:30:44] !log [universalomega@puppet181] Upgraded packages on graylog161: libicu72 [21:30:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:30:57] !log [universalomega@puppet181] Upgraded packages on cloud16: libicu72 [21:31:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:31:00] RECOVERY - eventgate181 APT on eventgate181 is OK: APT OK: 66 packages available for upgrade (0 critical updates). [21:31:10] !log [universalomega@puppet181] Upgraded packages on kafka181: libicu72 [21:31:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:31:17] RECOVERY - kafka181 APT on kafka181 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [21:31:22] !log [universalomega@puppet181] Upgraded packages on cp191: libicu72 [21:31:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:31:32] RECOVERY - db161 APT on db161 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [21:31:34] !log [universalomega@puppet181] Upgraded packages on cloud15: libicu72 [21:31:35] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 65 packages available for upgrade (1 critical updates). [21:31:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:31:42] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 128 packages available for upgrade (0 critical updates). [21:31:46] !log [universalomega@puppet181] Upgraded packages on db181: libicu72 [21:31:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:31:57] !log [universalomega@puppet181] Upgraded packages on cp201: libicu72 [21:32:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:32:10] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 54 packages available for upgrade (3 critical updates). [21:32:16] !log [universalomega@puppet181] Upgraded packages on bast161: libicu72 [21:32:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:32:23] RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 51 packages available for upgrade (0 critical updates). [21:32:29] !log [universalomega@puppet181] Upgraded packages on cloud20: libicu72 [21:32:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:32:35] RECOVERY - bast161 APT on bast161 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [21:32:42] !log [universalomega@puppet181] Upgraded packages on cloud18: libicu72 [21:32:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:32:47] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:32:57] PROBLEM - os191 APT on os191 is CRITICAL: APT CRITICAL: 46 packages available for upgrade (1 critical updates). [21:32:58] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [21:33:01] !log [universalomega@puppet181] Upgraded packages on db182: libicu72 [21:33:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:33:04] RECOVERY - cp191 APT on cp191 is OK: APT OK: 46 packages available for upgrade (0 critical updates). [21:33:05] PROBLEM - os161 APT on os161 is CRITICAL: APT CRITICAL: 47 packages available for upgrade (1 critical updates). [21:33:12] PROBLEM - reports171 APT on reports171 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (1 critical updates). [21:33:13] !log [universalomega@puppet181] Upgraded packages on mem161: libicu72 [21:33:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:33:17] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: APT CRITICAL: 67 packages available for upgrade (1 critical updates). [21:33:21] RECOVERY - cp201 APT on cp201 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:33:27] !log [universalomega@puppet181] Upgraded packages on ldap171: libicu72 [21:33:28] RECOVERY - mem161 APT on mem161 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [21:33:28] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 128 packages available for upgrade (0 critical updates). [21:33:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:33:38] RECOVERY - db181 APT on db181 is OK: APT OK: 53 packages available for upgrade (0 critical updates). [21:33:39] !log [universalomega@puppet181] Upgraded packages on mem151: libicu72 [21:33:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:33:47] RECOVERY - mem151 APT on mem151 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [21:33:52] !log [universalomega@puppet181] Upgraded packages on mem191: libicu72 [21:33:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:34:11] !log [universalomega@puppet181] Upgraded packages on mon181: libicu72 [21:34:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:34:24] !log [universalomega@puppet181] Upgraded packages on mw152: libicu72 [21:34:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:34:37] !log [universalomega@puppet181] Upgraded packages on mw151: libicu72 [21:34:38] RECOVERY - db182 APT on db182 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [21:34:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:34:42] PROBLEM - mw201 APT on mw201 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:34:42] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:34:50] !log [universalomega@puppet181] Upgraded packages on mw153: libicu72 [21:34:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:35:00] PROBLEM - swiftobject161 APT on swiftobject161 is CRITICAL: APT CRITICAL: 67 packages available for upgrade (1 critical updates). [21:35:04] !log [universalomega@puppet181] Upgraded packages on mw201: libicu72 [21:35:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:35:18] !log [universalomega@puppet181] Upgraded packages on mw203: libicu72 [21:35:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:35:26] RECOVERY - mw151 APT on mw151 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [21:35:27] PROBLEM - bast181 APT on bast181 is CRITICAL: APT CRITICAL: 46 packages available for upgrade (1 critical updates). [21:35:27] RECOVERY - mem191 APT on mem191 is OK: APT OK: 44 packages available for upgrade (0 critical updates). [21:35:32] !log [universalomega@puppet181] Upgraded packages on mw202: libicu72 [21:35:33] RECOVERY - mw203 APT on mw203 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [21:35:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:35:41] !log [universalomega@puppet181] Upgraded packages on mattermost1: libicu72 [21:35:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:35:55] !log [universalomega@puppet181] Upgraded packages on mw171: libicu72 [21:35:58] RECOVERY - mw171 APT on mw171 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [21:35:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:36:03] RECOVERY - mw202 APT on mw202 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [21:36:09] !log [universalomega@puppet181] Upgraded packages on mw191: libicu72 [21:36:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:36:17] RECOVERY - mw191 APT on mw191 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [21:36:23] !log [universalomega@puppet181] Upgraded packages on mw192: libicu72 [21:36:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:36:30] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [21:36:37]