[00:07:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [00:37:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [01:07:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [01:19:15] PROBLEM Current Load is now: WARNING on bots-3 bots-3 output: WARNING - load average: 6.34, 8.38, 5.51 [01:27:05] PROBLEM Current Load is now: WARNING on dumps-nfs1 dumps-nfs1 output: WARNING - load average: 5.32, 6.75, 5.34 [01:29:15] RECOVERY Current Load is now: OK on bots-3 bots-3 output: OK - load average: 2.10, 3.47, 4.32 [01:31:44] huh [01:31:47] my crontab got deleted? [01:32:05] RECOVERY Current Load is now: OK on dumps-nfs1 dumps-nfs1 output: OK - load average: 4.41, 4.66, 4.77 [01:32:18] is there a way to restore it? [01:37:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [02:06:51] petan: whats up [02:07:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [02:37:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [03:01:55] PROBLEM Free ram is now: CRITICAL on puppet-lucid puppet-lucid output: Critical: 3% free memory [03:07:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [03:26:55] RECOVERY Free ram is now: OK on puppet-lucid puppet-lucid output: OK: 20% free memory [03:37:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [03:42:05] PROBLEM Current Load is now: WARNING on dumps-nfs1 dumps-nfs1 output: WARNING - load average: 4.17, 5.37, 5.11 [03:57:05] RECOVERY Current Load is now: OK on dumps-nfs1 dumps-nfs1 output: OK - load average: 3.83, 4.32, 4.75 [04:07:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [04:10:55] PROBLEM Free ram is now: WARNING on bots-3 bots-3 output: Warning: 12% free memory [04:15:55] RECOVERY Free ram is now: OK on bots-3 bots-3 output: OK: 58% free memory [04:37:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [05:07:27] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [05:37:28] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [05:48:38] PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge) [05:56:58] PROBLEM host: testing-puppet is DOWN address: testing-puppet CRITICAL - Host Unreachable (testing-puppet) [06:02:13] PROBLEM Current Load is now: CRITICAL on driver-dev-jumbo driver-dev-jumbo output: CHECK_NRPE: Socket timeout after 10 seconds. [06:02:13] PROBLEM Disk Space is now: CRITICAL on driver-dev-jumbo driver-dev-jumbo output: CHECK_NRPE: Socket timeout after 10 seconds. [06:05:08] PROBLEM Current Users is now: CRITICAL on driver-dev-jumbo driver-dev-jumbo output: CHECK_NRPE: Socket timeout after 10 seconds. [06:05:08] PROBLEM Free ram is now: CRITICAL on driver-dev-jumbo driver-dev-jumbo output: CHECK_NRPE: Socket timeout after 10 seconds. [06:05:08] PROBLEM Total Processes is now: CRITICAL on driver-dev-jumbo driver-dev-jumbo output: CHECK_NRPE: Socket timeout after 10 seconds. [06:05:13] PROBLEM dpkg-check is now: CRITICAL on driver-dev-jumbo driver-dev-jumbo output: CHECK_NRPE: Socket timeout after 10 seconds. [06:06:38] PROBLEM SSH is now: CRITICAL on driver-dev-jumbo driver-dev-jumbo output: CRITICAL - Socket timeout after 10 seconds [06:06:38] PROBLEM Current Users is now: CRITICAL on dumpster01 dumpster01 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:06:38] PROBLEM Disk Space is now: CRITICAL on dumpster01 dumpster01 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:07:28] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [06:08:28] PROBLEM Current Load is now: CRITICAL on dumpster01 dumpster01 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:08:28] PROBLEM Free ram is now: CRITICAL on dumpster01 dumpster01 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:08:28] PROBLEM Total Processes is now: CRITICAL on dumpster01 dumpster01 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:10:28] PROBLEM Current Load is now: WARNING on dumps-nfs2 dumps-nfs2 output: WARNING - load average: 3.15, 6.37, 5.28 [06:11:28] RECOVERY SSH is now: OK on driver-dev-jumbo driver-dev-jumbo output: SSH OK - OpenSSH_5.8p1 Debian-7ubuntu1 (protocol 2.0) [06:11:28] RECOVERY Current Users is now: OK on dumpster01 dumpster01 output: USERS OK - 0 users currently logged in [06:11:28] RECOVERY Disk Space is now: OK on dumpster01 dumpster01 output: DISK OK [06:11:58] PROBLEM Current Load is now: WARNING on driver-dev-jumbo driver-dev-jumbo output: WARNING - load average: 4.73, 11.94, 7.93 [06:11:58] RECOVERY Disk Space is now: OK on driver-dev-jumbo driver-dev-jumbo output: DISK OK [06:13:18] RECOVERY Current Load is now: OK on dumpster01 dumpster01 output: OK - load average: 1.13, 6.27, 4.20 [06:13:18] RECOVERY Free ram is now: OK on dumpster01 dumpster01 output: OK: 83% free memory [06:13:18] RECOVERY Total Processes is now: OK on dumpster01 dumpster01 output: PROCS OK: 77 processes [06:14:58] RECOVERY Current Users is now: OK on driver-dev-jumbo driver-dev-jumbo output: USERS OK - 0 users currently logged in [06:14:58] RECOVERY Free ram is now: OK on driver-dev-jumbo driver-dev-jumbo output: OK: 80% free memory [06:14:58] RECOVERY Total Processes is now: OK on driver-dev-jumbo driver-dev-jumbo output: PROCS OK: 161 processes [06:15:03] RECOVERY dpkg-check is now: OK on driver-dev-jumbo driver-dev-jumbo output: All packages OK [06:15:28] RECOVERY Current Load is now: OK on dumps-nfs2 dumps-nfs2 output: OK - load average: 4.54, 3.92, 4.42 [06:19:28] PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge) [06:21:58] RECOVERY Current Load is now: OK on driver-dev-jumbo driver-dev-jumbo output: OK - load average: 0.00, 1.63, 4.18 [06:27:57] PROBLEM host: testing-puppet is DOWN address: testing-puppet CRITICAL - Host Unreachable (testing-puppet) [06:37:37] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [06:49:42] PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge) [06:58:11] PROBLEM host: testing-puppet is DOWN address: testing-puppet CRITICAL - Host Unreachable (testing-puppet) [07:07:41] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [07:10:31] PROBLEM Current Load is now: WARNING on bots-sql3 bots-sql3 output: WARNING - load average: 4.25, 5.73, 5.04 [07:12:37] @new [07:20:41] PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge) [07:23:55] I've kind of screwed things up... [07:28:11] PROBLEM host: testing-puppet is DOWN address: testing-puppet CRITICAL - Host Unreachable (testing-puppet) [07:35:21] PROBLEM Current Load is now: WARNING on dumps-nfs1 dumps-nfs1 output: WARNING - load average: 13.30, 10.22, 5.66 [07:37:41] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [07:40:31] RECOVERY Current Load is now: OK on bots-sql3 bots-sql3 output: OK - load average: 1.72, 2.86, 4.49 [07:50:41] PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge) [07:53:31] PROBLEM Current Load is now: WARNING on bots-sql3 bots-sql3 output: WARNING - load average: 7.32, 6.84, 5.90 [07:58:21] PROBLEM host: testing-puppet is DOWN address: testing-puppet CRITICAL - Host Unreachable (testing-puppet) [08:07:41] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [08:14:02] !account [08:14:02] in order to get an access to labs, please type !account-questions and ask Ryan, or someone who is in charge of creating account on labs [08:14:07] !account-questio [08:14:08] !account-question [08:14:10] !account-question [08:14:11] !account-questions [08:14:11] I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your preferred email address. 3. Your SVN account name, or your preferred shell account name, if you do not have SVN access. [08:14:14] morning [08:14:25] Ryan_Lane: hi [08:14:29] morning. I believe I've broken things fairly badly [08:14:37] Ryan_Lane: I sent you a mail now [08:14:53] you did? [08:14:56] I don't see it [08:15:07] should arrive in a minute [08:15:23] ah ok [08:15:28] so... [08:15:30] we decide to make a web huggle and we want to host it on labs [08:15:40] so I need to create a new project and accounts for 2 devs [08:15:44] it's likely that any instance that reboots will die [08:15:48] oh [08:16:06] I'm working on fixing that, but I'm not totally sure how I'm going to do it [08:17:02] what you did? [08:17:15] I mean why it happens now [08:17:59] I screwed up the _base directory on the instances share [08:18:13] oh [08:18:16] THO|Cloud: hi [08:18:21] I'm recovering the files, but now the directory is fucked up [08:18:26] I can't re-create it [08:18:45] that's the folder where instance storage is? [08:18:56] well…. no [08:19:08] /var/lib/nova/instances is [08:19:14] _base is under that directory [08:19:19] ah [08:19:24] which data are in that [08:19:34] * is [08:19:38] when an instance is created, nova pulls an image from glance [08:19:43] it sticks it there [08:19:53] so it's the image of vm? [08:20:02] kind of [08:20:30] right, which fs it's using? most of unix fs should have not break since the fd is open [08:20:35] there's a cow2 image in /var/lib/nova/instances//disk.local [08:20:41] PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge) [08:20:45] problem is when instance die [08:20:54] but, it's based on the stuff in base [08:20:57] same with snapshots [08:21:08] so, I'm restoring _base from the fds [08:21:13] ok [08:21:17] but, I can't write it back into _base [08:21:19] does it affect the current instance data [08:21:30] because all of the instances are holding open files in the old _base [08:21:35] you likely need to shutdown the instances now [08:21:39] it shouldn't, no [08:21:59] those write into /var/lib/nova/instances/ [08:22:05] ah [08:22:11] the data in _base likely shouldn't change [08:22:19] I don't know that that is actually true, though [08:22:25] I need to ask the openstack people how that works [08:22:32] can you make a backup before putting it back [08:22:40] if I shut down the instances, it's possible they may not come back up [08:22:48] I think it's possible to recover the file just using the fd [08:23:01] yeah, that's what I'm doing [08:23:06] right [08:23:09] as long as the fd is there, I can just copy it [08:23:11] in that case instances must not die [08:23:15] indeed [08:23:28] I have a copy going based on lsof [08:23:35] ok [08:23:46] sounds cool [08:24:03] I wasn't really about to reboot stuff :) [08:24:04] I don't know how to handle the directory problem, though [08:24:21] hm, yes [08:24:24] creating instances is likely broken right now [08:24:36] you don't have a backup of that [08:24:44] oh what? [08:24:53] we don't keep backups of instances [08:24:53] to check how the directory was looking [08:24:57] ok [08:25:17] instances aren't intended to have data that needs to be backed up, for the most part [08:25:26] true [08:25:28] bots is an obvious exception [08:25:56] maybe the project storage could have a backup though [08:26:33] well, eventually it's supposed to go on the gluster shared storage [08:26:37] that isn't instance storage [08:26:50] ok [08:27:37] oh well. I'm off to bed [08:27:43] I'm going to finish fixing this tomorrow [08:27:47] ok [08:27:54] so no new instances today [08:28:13] actually I have one instance which can be removed so I can try to reboot it [08:28:19] we will see if it die [08:28:21] PROBLEM host: testing-puppet is DOWN address: testing-puppet CRITICAL - Host Unreachable (testing-puppet) [08:28:42] let's try it [08:30:21] RECOVERY Current Load is now: OK on dumps-nfs1 dumps-nfs1 output: OK - load average: 2.60, 2.75, 4.12 [08:34:02] sure [08:34:16] it's very, very likely to die [08:34:44] unless nova doesn't kill the process and recreate it [08:35:07] if it does it will definitely die, because it can't access the _base directory [08:36:41] PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1) [08:37:41] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [08:40:13] yes [08:40:16] it's gone [08:45:07] bah. I've been copying the same file over and over. heh [08:45:12] stupid incorrect script [08:50:41] PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge) [08:53:08] so, the _base directory holds a qcow2 copy of the image the instance is running. all changes to the qcow2 image are added to the filesystem in the instance's directory. [08:53:38] as long as the initial image exists it's possible to reboot the instances [08:56:49] but it doesn't exist [08:56:54] now [08:57:03] I can modify the basedir in nova's code for now [08:57:10] then I can reboot all of the instances [08:57:15] ok [08:57:17] of course, I'm going to test that ;) [08:57:21] not just do it [08:57:34] did you totally delete that instance, or left it broken? [08:57:38] left it [08:57:40] cause I can test using that [08:57:40] ok [08:57:42] which one is it? [08:57:48] turnkey-1 [08:57:52] ok [08:58:21] PROBLEM host: testing-puppet is DOWN address: testing-puppet CRITICAL - Host Unreachable (testing-puppet) [09:06:11] RECOVERY host: turnkey-1 is UP address: turnkey-1 PING OK - Packet loss = 0%, RTA = 0.52 ms [09:07:41] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [09:20:41] PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge) [09:28:21] PROBLEM host: testing-puppet is DOWN address: testing-puppet CRITICAL - Host Unreachable (testing-puppet) [09:37:41] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [09:50:41] PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge) [09:58:21] PROBLEM host: testing-puppet is DOWN address: testing-puppet CRITICAL - Host Unreachable (testing-puppet) [09:58:30] Ryan_Lane: creation is broken? [09:58:32] now [09:58:42] yeah, I told you it would be [09:58:46] ok [09:59:08] isn't it like 2 am where you are? [09:59:11] :D [09:59:14] yes [09:59:16] yay [09:59:22] I think we can wait [10:00:11] Most people do their best work at 2am :D [10:00:15] if there is anything I can help with, let me know [10:03:37] * Ryan_Lane nods [10:03:54] like rebooting all instances etc [10:04:06] please don't do that :) [10:05:46] ok [10:07:14] seems like that worked [10:07:30] yeah, so, this isn't going to be much fun :) [10:07:41] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [10:07:44] don't tell me we need to reinstall all [10:07:53] nah [10:08:21] we may need to shut down all instances and then restart them, though [10:08:31] RECOVERY host: testing-puppet is UP address: testing-puppet PING OK - Packet loss = 0%, RTA = 0.78 ms [10:08:44] ok [10:08:47] tell me when [10:08:54] well, I need to do it [10:08:57] ok [10:09:03] because it needs to be every single instance [10:09:04] I wanted to restart apaches one by one [10:09:09] so there is a short outage only [10:09:19] no. it needs to be all of them at the same time [10:09:22] ah [10:09:23] ok [10:09:32] because they need to release their handle on the directory [10:09:38] right [10:09:48] and when it gets shut down, it won't come back up until the directory is released [10:09:59] I guess I could just reboot all nodes [10:10:17] I think it's a good time to patch my bot [10:10:21] heh [10:10:42] maybe I could also install new kernels on all [10:10:51] I wonder if I can just suspend the instances [10:10:53] I bet I can [10:10:57] hm... [10:11:07] I think for bots it's better to shut down [10:11:11] why? [10:11:19] because some can't handle connection outage [10:11:31] ah [10:11:44] restart of process is better [10:11:52] but maybe it's just my case [10:12:05] I don't know the other bots but I bet cluebot will crash as well [10:12:10] Damianz: ^ [10:12:12] :o [10:12:48] are you sure you can suspend them? [10:12:57] well, it seems suspend keeps the process running [10:13:09] I think it needs to keep fd [10:13:20] otherwise it couldn't recover [10:13:28] depends [10:13:47] if there is a difference between the two images it was running from, it would crash [10:13:50] CB will just have a heart attack but supervise will bring it back up [10:13:50] I guess [10:13:58] if it was like hibernate, it would write changes to disk, then shut down the process [10:14:03] then reconnect to the file [10:14:06] when you resume [10:14:19] ok but if the file is changed while it's down [10:14:32] it could be problem a bit [10:14:41] maybe it keep the fd for that reason to prevent changes [10:14:42] why would the file change? [10:15:06] the files I restored never change [10:15:07] because as I understand it you copied the old image using fd and now you want to put it back [10:15:10] ah [10:15:24] I thought it's like a vd image [10:15:38] mounted as /dev/vda1 [10:15:43] ok [10:15:52] there's a base image and a disk image [10:16:02] the disk image only writes changes from the base [10:16:08] ah [10:16:22] that way, if we have 20 lucid images, it doesn't need to make 20 copies of the base [10:16:33] ok [10:19:11] Ryan_Lane: I forward that request to dzahn ok? my email [10:19:18] so that you don't need to handle it... [10:19:29] you can if you'd like [10:20:41] PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge) [10:20:47] mutante: ping [10:24:52] !log dumps Deleted instances dumps-6 & dumps-7 [10:24:54] Logged the message, Master [10:27:39] yep. no getting around it. I'm going to have to stop all instances [10:27:47] * Ryan_Lane sighs [10:28:39] ALL? [10:29:12] yep [10:29:31] the way in which I've fucked up only has one solution :D [10:29:52] you don't happen to mean all >100 instances [10:30:01] hey, it could be worse, we could have 1,000 instances [10:30:05] yes, I mean all of them [10:30:11] wait wait [10:30:17] * Hydriz goes and save work [10:30:22] I don't mean immediately [10:30:36] I'm going to send an email out, and give people till some time tomorrow. [10:30:49] phew [10:31:06] yeah, tomorrow then probably I have time [10:31:22] if right now then it would be scramble [10:31:29] but whats fucked? [10:31:41] I screwed up the filesystem somewhat [10:32:07] but how does it affect the instances? [10:32:09] and the only way to fix what I did is to make all of the instances drop their handles to the current files [10:32:29] Oh, I see what you did there [10:41:27] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [10:43:37] I'm going to shut them down on March 6th [10:43:59] which also means no instance creation [10:49:18] sent an email explaining it to labs-l [10:50:47] PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge) [10:51:52] * Ryan_Lane goes to sleep [10:51:56] * Damianz notes to take backup [10:55:56] bye [11:11:27] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [11:20:47] PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge) [11:41:27] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [11:50:47] PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge) [12:11:27] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [12:20:47] PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge) [12:41:27] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [12:50:47] PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge) [13:11:27] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [13:20:47] PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge) [13:41:27] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [13:49:26] 03/05/2012 - 13:49:26 - Creating a project directory for hugglewa [13:50:47] PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge) [13:59:33] mutante: here? [13:59:40] can you insert me to huggle wa too [14:01:34] petan|wk: Am I able to be added into deployment-prep? [14:01:49] hm... what are you going to do there? [14:02:03] probably going to just enable extensions for testing [14:02:12] (of course not cluster-wide or something) [14:02:15] that's exactly what people are not supposed to do there [14:02:26] zzz [14:02:34] deployment prep is a project where we test software before deploying it to cluster [14:02:37] if only labs allowed forking [14:02:52] then I would have forked this project and used it to enable extensions [14:02:57] that mean it's a place where approved sw is enabled to check it won't break the cluster [14:03:17] sw? [14:03:17] hm, it should be possible in future [14:03:27] Soft Ware [14:03:35] Ware soft! [14:04:39] <^demon> Wario Ware. [14:05:08] zzz [14:05:26] * Hydriz was just hoping to find out how the beta cluster is actually set up [14:05:43] weird [14:05:52] I would like to have someone to check and fix it [14:05:56] atm it's totally broken [14:06:10] if people won't start logging to channel [14:06:15] I will install audit daemon there [14:06:22] speaking of logging [14:06:31] I am currently running the bot from screen [14:06:42] ok [14:06:44] sudo service adminlogbot restart or something broke [14:06:48] permission issue [14:06:56] but it doesn't really matter who runs it anyway [14:06:57] which instance it is [14:07:06] sudo should run it as root [14:07:20] it does matter at some point [14:07:28] yes [14:07:34] I am wondering how root is getting permission issues [14:07:34] but somehow it broke in mid-air [14:07:43] ok let me fix it [14:07:50] !log bots restart logbot [14:07:51] Logged the message, Master [14:07:51] I think its starting the bot without sudo [14:08:01] letme kill it [14:08:22] logging in... [14:08:38] bad bot [14:08:39] !log killed :P [14:09:08] petan|wk: The bot was failing the other day on trying to write to a folder in /var that didn't exist [14:09:14] Also could do with moving to the labs instance. [14:09:22] it was moving? [14:09:30] thats good though [14:09:50] We have a bots-labs for like irc stuff as bots-2 randomly gets overloaded and kills like logbot. [14:10:01] Hydriz: you should not have kill it [14:10:06] but fine... [14:10:07] :O [14:10:15] * Hydriz is a murderer!!! [14:11:27] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [14:11:42] now it's runnig as root [14:11:55] loggie [14:12:00] not really [14:12:11] it doesn't run as root, but as its service user [14:12:40] !log bots Restarted as service user now... [14:12:41] Logged the message, Master [14:12:46] \o/ [14:13:22] but it doesn't matter who runs it, still [14:13:40] if it breaks again I believe anyone should just ssh in and restart it [14:13:47] as labslogbot [14:14:33] point of service is to be able to restart itself on fail [14:14:47] oh good [14:14:48] it shouldn't require us to restart it [14:14:52] it had better [14:16:14] hmm, whats audit daemon [14:16:29] stuff which track the changes to files [14:16:46] hyperon: hey [14:16:50] then how does logmsgbot work in -tech? [14:16:54] petan|wk: hey [14:17:02] what's up [14:17:07] can you move the log bot to bots-labs [14:17:15] I don't know where the .deb is [14:17:45] Hydriz: exactly as this one :o [14:18:10] Qn: What does "this one" refer to? [14:18:19] !log me [14:18:20]