[00:01:13] I also think I'm going to reenable uploads and then redisable tomorrow again when I continue the other servers. Only doing swiftobject191 and 201 today. That will be communicated more though. [00:01:24] Got it [00:01:37] PROBLEM - os191 Current Load on os191 is CRITICAL: LOAD CRITICAL - total load average: 4.39, 3.88, 3.34 [00:03:34] PROBLEM - os191 Current Load on os191 is WARNING: LOAD WARNING - total load average: 3.18, 3.55, 3.28 [00:05:53] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 56231MiB (12% inode=99%); [00:06:38] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 56625MiB (12% inode=99%); [00:07:29] RECOVERY - os191 Current Load on os191 is OK: LOAD OK - total load average: 2.09, 2.98, 3.12 [00:14:37] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57040MiB (12% inode=99%); [00:20:07] PROBLEM - cp191 Current Load on cp191 is WARNING: LOAD WARNING - total load average: 0.58, 4.67, 15.46 [00:24:07] RECOVERY - cp191 Current Load on cp191 is OK: LOAD OK - total load average: 0.83, 2.42, 12.06 [00:34:21] !log backup finished (finally) [00:34:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:34:25] \o/ [00:35:01] now the actually work begins lol [00:35:06] *actual [00:35:24] wait, where are we backing up to [00:36:39] im just cloning the VM to the same cloud server so its cloned in the current state while I modify disks allowing to restore if needed. [00:37:40] oh ic [00:37:47] i wonder, how much free disk space do we have on the cloud* [00:37:59] 5TB or so lol [00:38:03] on each cloud [00:38:24] oh neat :3 [00:41:17] I hate seeing messages like "THIS IS POTENTIALLY DESTRUCTIVE ALL DATA MAY BE ERASED! PROCEED?" then I type Yes lol. Nothing really happens unless you well do it wrong lol. Thats what backup is for though... [00:41:45] in which CA wipes a cloud server by accident /j [00:44:53] [1/2] you won't believe the heart attack I almost had when doing swiftobject201. So the root disk was /dev/sda but when I added the new efi disk the root disk somehow became /dev/sdb and efi /dev/sda, well I try to login to shell at /dev/sda and get this giant message on a huge red screen that says no shell at this drive doesn't have a shell. Perhaps it is corrupted? Im like oh shit, t [00:44:53] [2/2] hen just for the sake of it try sdb and well relieved its there lol. [00:45:46] wait what [00:45:53] how [00:46:51] honestly no idea lol I didn't expect that at all. But it scared me half to death. [00:47:48] I guess it didn't say corrupted but that was my thought lol [00:48:17] https://cdn.discordapp.com/attachments/808001911868489748/1402453191798292665/chatgpt-1754431627484.jpg?ex=6893f7d1&is=6892a651&hm=e5ba6183ef8a7928324d756448d5797a2d4b531d8127a188aca64693361e65ba& [00:48:38] I took a picture of it lol [00:48:45] pic of screen [00:48:46] mmm yes [00:48:59] i suppose that you're not in fsslc? [00:49:21] hmm? [00:49:29] like, you're not physically at the dc, right? [00:49:35] oh no lol [00:49:42] why pic of screen :/ [00:50:05] Because I panicked and forgot there was a snipping tool or screenshot option lol [00:50:08] oh [00:50:45] also it was easier the terminal was full screen and my phone was right there lol [00:52:18] anyway I better get to finishing 191... let's hope all the steps I wrote down was accurate so I can do it fast this time lol [00:53:07] five hours later... [00:53:26] last one only took 4.5 hours lol [00:53:42] exactly :) [00:55:02] We should honestly probably move all VMs to efi at some point tbh also... [00:55:13] true [00:55:15] but like... [00:55:16] downtime [00:55:20] * BlankEclair blehs [00:55:25] (is bleh a verb?) [00:55:38] Not really except for db thats the only one that'll cause downtime. Everything else can really be depooled. [00:55:53] oh okay [00:56:02] phorge- [00:56:11] is the mattermost vps on efi? [00:56:29] Well true but thats not wiki downtime. [00:56:49] we ought to set up replicas [00:56:49] Im not sure. If it isn't we cant change it though (very easily anyway) [00:57:04] imagine if the cloud servers are on mbr [00:57:19] they are EFI lol I did check that. [00:57:38] ah okay [00:57:57] I actually didn't check what the host disk is though. Only that boot mode is EFI [00:58:07] unrelated, but woah, udp2raw seems cool [01:00:24] !log boot swiftobject191 into rescue mode [01:00:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:02:50] !log install gdisk on swiftobject191 [01:02:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:08:50] RECOVERY - Host swiftobject191 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [01:09:04] holy crap it breathes [01:10:18] RECOVERY - ping on swiftobject191 is OK: PING OK - Packet loss = 0%, RTA = 0.31 ms [01:19:41] RECOVERY - swiftobject191 APT on swiftobject191 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [01:19:44] RECOVERY - swiftobject191 Check unit status of disable-rsync on swiftobject191 is OK: OK: Status of the systemd unit disable-rsync [01:19:50] RECOVERY - swiftobject191 ferm_active on swiftobject191 is OK: OK ferm input default policy is set [01:19:51] PROBLEM - swiftobject191 NTP time on swiftobject191 is WARNING: NTP WARNING: Offset -0.1122650802 secs [01:19:56] and we are back that was fast lol [01:20:00] PROBLEM - swiftobject191 Disk Space on swiftobject191 is WARNING: DISK WARNING - free space: / 146632MiB (10% inode=86%); [01:20:03] RECOVERY - swiftobject191 PowerDNS Recursor on swiftobject191 is OK: DNS OK: 0.287 seconds response time. swiftobject191.fsslc.wtnet returns 10.0.19.120 [01:20:15] is this just a temporary re-enable? [01:20:15] RECOVERY - swiftobject191 SSH on swiftobject191 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [01:20:29] 2 servers are done 4 to go [01:20:35] got it [01:20:41] the othr 4 tomorow? [01:20:42] PROBLEM - swiftobject191 Puppet on swiftobject191 is WARNING: WARNING: Puppet last ran 2 hours ago [01:20:44] wait I forgot to expand 191 that will be easy though lol [01:20:48] RECOVERY - swiftobject191 conntrack_table_size on swiftobject191 is OK: OK: nf_conntrack is 0 % full [01:20:50] got ti lol [01:21:02] RECOVERY - swiftobject191 Swift Object Service on swiftobject191 is OK: TCP OK - 0.000 second response time on 10.0.19.120 port 6000 [01:21:39] RECOVERY - swiftobject191 Current Load on swiftobject191 is OK: LOAD OK - total load average: 0.67, 0.32, 0.12 [01:21:52] RECOVERY - swiftobject191 NTP time on swiftobject191 is OK: NTP OK: Offset -0.05421388149 secs [01:22:43] RECOVERY - swiftobject191 Puppet on swiftobject191 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [01:22:57] !log [universalomega@swiftobject191] uninstall grup-pc-bin and upgrade kernel and all packages and reboot [01:23:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:00] RECOVERY - swiftobject191 Disk Space on swiftobject191 is OK: DISK OK - free space: / 147097MiB (11% inode=86%); [01:24:50] !log [universalomega@swiftobject191] install cloud-guest-utils [01:24:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:25:24] !log [universalomega@swiftobject191] sudo growpart /dev/sda 1 [01:25:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:25:53] !log [universalomega@swiftobject191] sudo resize2fs /dev/sda1 [01:25:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:27] !log reboot swiftobject191 [01:26:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:27:49] swiftobject191 totally done now [01:29:18] I am just gonna try and get them all done tonight [01:30:06] all 4 remaining servers tonight? [01:30:29] yep [01:30:46] I can do each one in about 20 minutes now I think lets see lol [01:30:54] ahh ok! [01:30:56] hope I won't be seeing you when I wake up lol [01:31:05] and that'd be it? for maintenance? [01:31:11] yep [01:31:46] niiice!!! [01:32:01] !log installed dosfstools and grub-efi-amd64 on all swiftobject servers (or will) [01:32:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:33:34] !log [universalomega@swiftobject181] sudo apt install dosfstools grub-efi-amd64 gdisk cloud-guest-utils [01:33:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:34:16] !log shutdown swiftobject181 [01:34:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:34:36] !log do same to swiftobject181 as others [01:34:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:35:24] PROBLEM - swiftobject181 Disk Space on swiftobject181 is CRITICAL: connect to address 10.0.18.107 port 5666: No route to hostconnect to host 10.0.18.107 port 5666: No route to host [01:36:27] PROBLEM - ping on swiftobject181 is CRITICAL: CRITICAL - Host Unreachable (10.0.18.107) [01:36:33] PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: connect to address 10.0.18.107 port 5666: No route to hostconnect to host 10.0.18.107 port 5666: No route to host [01:36:41] PROBLEM - swiftobject181 NTP time on swiftobject181 is CRITICAL: connect to address 10.0.18.107 port 5666: No route to hostconnect to host 10.0.18.107 port 5666: No route to host [01:36:44] PROBLEM - swiftobject181 Swift Object Service on swiftobject181 is CRITICAL: connect to address 10.0.18.107 and port 6000: No route to host [01:36:58] PROBLEM - swiftobject181 Check unit status of disable-rsync on swiftobject181 is CRITICAL: connect to address 10.0.18.107 port 5666: No route to hostconnect to host 10.0.18.107 port 5666: No route to host [01:37:08] PROBLEM - swiftobject181 Puppet on swiftobject181 is CRITICAL: connect to address 10.0.18.107 port 5666: No route to hostconnect to host 10.0.18.107 port 5666: No route to host [01:37:11] PROBLEM - swiftobject181 Current Load on swiftobject181 is CRITICAL: connect to address 10.0.18.107 port 5666: No route to hostconnect to host 10.0.18.107 port 5666: No route to host [01:37:14] PROBLEM - Host swiftobject181 is DOWN: CRITICAL - Host Unreachable (10.0.18.107) [01:48:35] I'm around for a few hours if you need an extra set of hands for anything [01:49:26] RECOVERY - Host swiftobject181 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [01:50:07] !log [universalomega@swiftobject181] sudo growpart /dev/sda 1 [01:50:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:50:12] !log [universalomega@swiftobject181] sudo resize2fs /dev/sda1 [01:50:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:50:45] Thanks so far all good though. 3 left lol [01:50:53] RECOVERY - swiftobject181 Check unit status of disable-rsync on swiftobject181 is OK: OK: Status of the systemd unit disable-rsync [01:51:08] RECOVERY - swiftobject181 Current Load on swiftobject181 is OK: LOAD OK - total load average: 0.83, 0.22, 0.07 [01:51:13] RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [01:51:23] RECOVERY - swiftobject181 Disk Space on swiftobject181 is OK: DISK OK - free space: / 1484962MiB (55% inode=93%); [01:51:28] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 76 packages available for upgrade (0 critical updates). [01:51:38] RECOVERY - ping on swiftobject181 is OK: PING OK - Packet loss = 0%, RTA = 0.35 ms [01:51:43] RECOVERY - swiftobject181 Swift Object Service on swiftobject181 is OK: TCP OK - 0.000 second response time on 10.0.18.107 port 6000 [01:52:44] RECOVERY - swiftobject181 NTP time on swiftobject181 is OK: NTP OK: Offset -3.096461296e-05 secs [01:55:37] !log doing same on swiftobject171 [01:55:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:58:56] PROBLEM - swiftobject171 Disk Space on swiftobject171 is CRITICAL: connect to address 10.0.17.126 port 5666: No route to hostconnect to host 10.0.17.126 port 5666: No route to host [01:59:30] PROBLEM - swiftobject171 Puppet on swiftobject171 is CRITICAL: connect to address 10.0.17.126 port 5666: No route to hostconnect to host 10.0.17.126 port 5666: No route to host [01:59:33] PROBLEM - swiftobject171 NTP time on swiftobject171 is CRITICAL: connect to address 10.0.17.126 port 5666: No route to hostconnect to host 10.0.17.126 port 5666: No route to host [01:59:36] PROBLEM - swiftobject171 PowerDNS Recursor on swiftobject171 is CRITICAL: connect to address 10.0.17.126 port 5666: No route to hostconnect to host 10.0.17.126 port 5666: No route to host [01:59:37] PROBLEM - swiftobject171 Swift Object Service on swiftobject171 is CRITICAL: connect to address 10.0.17.126 and port 6000: No route to host [01:59:37] PROBLEM - swiftobject171 APT on swiftobject171 is CRITICAL: connect to address 10.0.17.126 port 5666: No route to hostconnect to host 10.0.17.126 port 5666: No route to host [02:00:18] PROBLEM - swiftobject171 Check unit status of disable-rsync on swiftobject171 is CRITICAL: connect to address 10.0.17.126 port 5666: No route to hostconnect to host 10.0.17.126 port 5666: No route to host [02:00:18] PROBLEM - swiftobject171 conntrack_table_size on swiftobject171 is CRITICAL: connect to address 10.0.17.126 port 5666: No route to hostconnect to host 10.0.17.126 port 5666: No route to host [02:00:27] PROBLEM - swiftobject171 ferm_active on swiftobject171 is CRITICAL: connect to address 10.0.17.126 port 5666: No route to hostconnect to host 10.0.17.126 port 5666: No route to host [02:00:34] PROBLEM - ping on swiftobject171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.126) [02:00:50] PROBLEM - swiftobject171 SSH on swiftobject171 is CRITICAL: connect to address 10.0.17.126 and port 22: No route to host [02:00:53] PROBLEM - swiftobject171 Current Load on swiftobject171 is CRITICAL: connect to address 10.0.17.126 port 5666: No route to hostconnect to host 10.0.17.126 port 5666: No route to host [02:01:16] PROBLEM - Host swiftobject171 is DOWN: CRITICAL - Host Unreachable (10.0.17.126) [02:07:15] RECOVERY - Host swiftobject171 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [02:07:31] RECOVERY - swiftobject171 Puppet on swiftobject171 is OK: OK: Puppet is currently enabled, last run 12 minutes ago with 0 failures [02:07:36] RECOVERY - swiftobject171 Swift Object Service on swiftobject171 is OK: TCP OK - 0.000 second response time on 10.0.17.126 port 6000 [02:07:37] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [02:07:51] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php rebuildall --wiki=newnovawiki (START) [02:07:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:08:14] RECOVERY - swiftobject171 conntrack_table_size on swiftobject171 is OK: OK: nf_conntrack is 0 % full [02:08:16] RECOVERY - swiftobject171 Check unit status of disable-rsync on swiftobject171 is OK: OK: Status of the systemd unit disable-rsync [02:08:24] RECOVERY - swiftobject171 ferm_active on swiftobject171 is OK: OK ferm input default policy is set [02:08:35] RECOVERY - ping on swiftobject171 is OK: PING OK - Packet loss = 0%, RTA = 0.27 ms [02:08:44] RECOVERY - swiftobject171 SSH on swiftobject171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [02:08:58] PROBLEM - swiftobject171 Disk Space on swiftobject171 is WARNING: DISK WARNING - free space: / 129963MiB (9% inode=86%); [02:08:59] RECOVERY - swiftobject171 Current Load on swiftobject171 is OK: LOAD OK - total load average: 0.56, 0.19, 0.07 [02:09:37] RECOVERY - swiftobject171 PowerDNS Recursor on swiftobject171 is OK: DNS OK: 0.054 seconds response time. swiftobject171.fsslc.wtnet returns 10.0.17.126 [02:09:47] RECOVERY - swiftobject171 NTP time on swiftobject171 is OK: NTP OK: Offset 0.09318500757 secs [02:09:51] !log [universalomega@swiftobject171] sudo growpart /dev/sdb 1 [02:09:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:09:57] !log [universalomega@swiftobject171] sudo resize2fs /dev/sdb1 [02:10:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:10:36] 2 more lol [02:10:43] woo! [02:10:47] you got this [02:10:57] RECOVERY - swiftobject171 Disk Space on swiftobject171 is OK: DISK OK - free space: / 1480817MiB (55% inode=93%); [02:11:08] I am probably going to take tomorrow off after today though lol [02:11:39] thanks for everything you do! [02:14:48] !log starting on swiftobject161 [02:14:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:17:55] PROBLEM - swiftobject161 Disk Space on swiftobject161 is CRITICAL: connect to address 10.0.16.134 port 5666: No route to hostconnect to host 10.0.16.134 port 5666: No route to host [02:19:08] PROBLEM - swiftproxy171 HTTP on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [02:19:24] PROBLEM - swiftproxy171 HTTPS on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [02:19:38] PROBLEM - ping on swiftobject161 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.134) [02:19:38] PROBLEM - Host swiftobject161 is DOWN: CRITICAL - Host Unreachable (10.0.16.134) [02:20:07] PROBLEM - swiftproxy161 HTTPS on swiftproxy161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [02:20:43] PROBLEM - swiftproxy161 HTTP on swiftproxy161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [02:25:07] RECOVERY - swiftproxy171 HTTP on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.021 second response time [02:25:37] !log [universalomega@swiftobject161] sudo growpart /dev/sda 1 [02:25:38] RECOVERY - Host swiftobject161 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms [02:25:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:25:42] !log [universalomega@swiftobject161] sudo resize2fs /dev/sda1 [02:25:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:25:53] PROBLEM - swiftobject161 ferm_active on swiftobject161 is CRITICAL: connect to address 10.0.16.134 port 5666: Connection refusedconnect to host 10.0.16.134 port 5666: Connection refused [02:25:53] PROBLEM - swiftobject161 Check unit status of disable-rsync on swiftobject161 is CRITICAL: connect to address 10.0.16.134 port 5666: Connection refusedconnect to host 10.0.16.134 port 5666: Connection refused [02:26:08] PROBLEM - swiftobject161 PowerDNS Recursor on swiftobject161 is CRITICAL: connect to address 10.0.16.134 port 5666: Connection refusedconnect to host 10.0.16.134 port 5666: Connection refused [02:26:08] RECOVERY - swiftobject161 Disk Space on swiftobject161 is OK: DISK OK - free space: / 1485474MiB (55% inode=93%); [02:26:18] PROBLEM - swiftobject161 NTP time on swiftobject161 is CRITICAL: NTP CRITICAL: Offset -0.635486573 secs [02:26:43] RECOVERY - swiftproxy161 HTTP on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.013 second response time [02:27:02] ONE MORE! [02:27:24] RECOVERY - swiftproxy171 HTTPS on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.022 second response time [02:27:38] RECOVERY - ping on swiftobject161 is OK: PING OK - Packet loss = 0%, RTA = 0.29 ms [02:27:45] RECOVERY - swiftobject161 Check unit status of disable-rsync on swiftobject161 is OK: OK: Status of the systemd unit disable-rsync [02:27:46] RECOVERY - swiftobject161 ferm_active on swiftobject161 is OK: OK ferm input default policy is set [02:28:00] RECOVERY - swiftobject161 PowerDNS Recursor on swiftobject161 is OK: DNS OK: 0.051 seconds response time. swiftobject161.fsslc.wtnet returns 10.0.16.134 [02:28:07] RECOVERY - swiftproxy161 HTTPS on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.021 second response time [02:28:17] RECOVERY - swiftobject161 NTP time on swiftobject161 is OK: NTP OK: Offset -9.799003601e-05 secs [02:29:09] !log starting on swiftobject151 [02:29:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:32:37] PROBLEM - swiftobject151 Disk Space on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [02:33:50] PROBLEM - swiftobject151 Check unit status of disable-rsync on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [02:34:01] PROBLEM - swiftobject151 SSH on swiftobject151 is CRITICAL: connect to address 10.0.15.117 and port 22: No route to host [02:34:01] PROBLEM - swiftobject151 conntrack_table_size on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [02:34:01] PROBLEM - swiftobject151 NTP time on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [02:34:24] PROBLEM - swiftobject151 Swift Object Service on swiftobject151 is CRITICAL: connect to address 10.0.15.117 and port 6000: No route to host [02:34:27] PROBLEM - swiftobject151 ferm_active on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [02:34:30] PROBLEM - swiftobject151 Puppet on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [02:34:33] PROBLEM - swiftobject151 Current Load on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [02:34:51] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [02:34:51] PROBLEM - ping on swiftobject151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.117) [02:35:13] PROBLEM - Host swiftobject151 is DOWN: CRITICAL - Host Unreachable (10.0.15.117) [02:36:34] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php initEditCount --wiki=newnovawiki (END - exit=0) [02:36:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:04] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php initSiteStats --wiki=newnovawiki --update (END - exit=0) [02:37:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:39:17] RECOVERY - Host swiftobject151 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [02:39:52] RECOVERY - swiftobject151 Check unit status of disable-rsync on swiftobject151 is OK: OK: Status of the systemd unit disable-rsync [02:39:56] RECOVERY - swiftobject151 SSH on swiftobject151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [02:40:00] RECOVERY - swiftobject151 conntrack_table_size on swiftobject151 is OK: OK: nf_conntrack is 0 % full [02:40:02] RECOVERY - swiftobject151 NTP time on swiftobject151 is OK: NTP OK: Offset 0.06689405441 secs [02:40:33] RECOVERY - swiftobject151 Current Load on swiftobject151 is OK: LOAD OK - total load average: 0.00, 0.00, 0.00 [02:40:38] PROBLEM - swiftobject151 Disk Space on swiftobject151 is WARNING: DISK WARNING - free space: / 139610MiB (10% inode=86%); [02:40:43] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [02:40:53] RECOVERY - ping on swiftobject151 is OK: PING OK - Packet loss = 0%, RTA = 0.25 ms [02:42:10] !log [universalomega@swiftobject151] sudo growpart /dev/sda 1 [02:42:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:42:20] !log [universalomega@swiftobject151] sudo resize2fs /dev/sda1 [02:42:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:42:26] RECOVERY - swiftobject151 ferm_active on swiftobject151 is OK: OK ferm input default policy is set [02:42:27] RECOVERY - swiftobject151 Swift Object Service on swiftobject151 is OK: TCP OK - 0.000 second response time on 10.0.15.117 port 6000 [02:42:30] RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures [02:42:34] PROBLEM - swiftobject151 Disk Space on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: Connection refusedconnect to host 10.0.15.117 port 5666: Connection refused [02:43:17] !log finished operations on all swiftobject servers [02:43:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:43:23] FINALLY!!!! [02:43:51] RECOVERY - swiftobject151 Disk Space on swiftobject151 is OK: DISK OK - free space: / 1490907MiB (55% inode=93%); [02:45:32] miraheze/mw-config - Universal-Omega the build passed. [02:45:45] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [02:45:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:46:07] miraheze/mw-config - Universal-Omega the build passed. [02:46:10] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [02:46:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:33:36] !log [macfan@test151] starting deploy of {'pull': 'errorpages', 'errorpages': True} to test151 [03:33:37] !log [macfan@test151] finished deploy of {'pull': 'errorpages', 'errorpages': True} to test151 - SUCCESS in 0s [03:33:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:33:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:34:01] RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4114 bytes in 0.056 second response time [04:05:29] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4215 bytes in 0.065 second response time [04:35:09] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4161 bytes in 0.066 second response time [04:37:53] miraheze/puppet - Universal-Omega the build has errored. [04:40:07] miraheze/puppet - Universal-Omega the build has errored. [04:43:39] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4161 bytes in 0.068 second response time [04:43:43] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4161 bytes in 0.068 second response time [06:08:15] miraheze/RequestSSL - AgentIsai the build has errored. [06:37:14] miraheze/RequestSSL - AgentIsai the build passed. [06:47:45] !log [agent@test151] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to test151 [06:47:46] !log [agent@test151] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to test151 - SUCCESS in 1s [06:47:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:47:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:50:41] miraheze/RequestSSL - AgentIsai the build has errored. [06:51:04] miraheze/RequestSSL - AgentIsai the build has errored. [06:56:48] !log [agent@test151] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to test151 [06:56:50] !log [agent@test151] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to test151 - SUCCESS in 1s [06:56:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:56:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:00:46] !log [agent@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [07:00:47] !log [agent@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s [07:00:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:00:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:01:31] miraheze/mw-config - AgentIsai the build passed. [07:07:48] !log [agent@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to all [07:07:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:08:37] !log [agent@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to all - SUCCESS in 48s [07:08:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:08:49] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [07:08:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:09:13] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s [07:09:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:09:43] !log [agent@mwtask181] starting deploy of {'l10n': True, 'versions': ['1.43', '1.44']} to all [07:09:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:17:41] !log [agent@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [07:17:42] miraheze/RequestSSL - AgentIsai the build passed. [07:17:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:18:03] !log [agent@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'MirahezeMagic'} to all [07:18:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:18:51] !log [agent@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 48s [07:18:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:20:07] !log [agent@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to all [07:20:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:20:52] !log [agent@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to all - SUCCESS in 45s [07:20:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:28:44] miraheze/MirahezeMagic - AgentIsai the build passed. [07:38:44] miraheze/RequestSSL - AgentIsai the build passed. [07:40:34] !log [agent@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'MirahezeMagic'} to all [07:40:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:41:19] !log [agent@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 45s [07:41:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:41:25] !log [agent@mwtask181] starting deploy of {'l10n': True, 'versions': ['1.43', '1.44']} to all [07:41:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:46:07] miraheze/dns - Universal-Omega the build passed. [07:46:51] miraheze/dns - Universal-Omega the build passed. [07:47:19] !log destory and recreate the unused os202 VM (preparing to use) using OVMF BIOS [07:47:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:47:46] destory interesting typo lol [07:53:45] miraheze/MirahezeMagic - AgentIsai the build passed. [07:56:29] !log [agent@mwtask181] finished deploy of {'l10n': True, 'versions': ['1.43', '1.44']} to all - SUCCESS in 903s [07:56:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:59:20] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49847MiB (10% inode=99%); [09:43:51] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49869MiB (10% inode=99%); [10:33:36] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49930MiB (10% inode=99%); [13:16:04] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php rebuildall --wiki=newnovawiki (END - exit=2) [13:16:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:16:16] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php rebuildall --wiki=newnovawiki (START) [13:16:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:16:46] hanged, not sure why [13:30:25] (probably my shenanigans with screen ngl) [13:53:56] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1754484800000&orgId=1&to=1754488436681 [14:03:56]