[00:00:18] Short answer no, long answer use nohup/screen/double fork etc [00:00:39] If it's attached to the current session there's no easy way to detach it. [00:01:52] Damianz: okay so the best way to leave tasks running is to just use a cron tab? [00:02:16] is there a way to watch the status of a crontab? (i've set a couple up and they run fine - but can you monitor them whilst they are running?) [00:03:00] Cron is for running jobs on a schedule [00:03:35] It will email output by default, usualy you'd pipe the output into a log file or have the script write a log file for monitoring it. [00:09:34] Damianz: so each time it runs it should email me? [00:12:35] Check `mail` or /var/spool/mail/username and you might have output there. [00:16:38] first one says no mail [00:16:42] second one says no such directory [00:18:31] Hmm does it create any output? I assume these servers are running vixie cron [00:18:40] * Damianz -> Bed [02:44:44] PROBLEM Free ram is now: WARNING on puppet-lucid puppet-lucid output: Warning: 12% free memory [04:53:58] PROBLEM Current Load is now: CRITICAL on dumps-5 dumps-5 output: Connection refused by host [04:53:58] PROBLEM Current Load is now: CRITICAL on dumps-8 dumps-8 output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:54:38] PROBLEM Current Users is now: CRITICAL on dumps-5 dumps-5 output: Connection refused by host [04:54:48] PROBLEM Current Load is now: WARNING on dumps-nfs1 dumps-nfs1 output: WARNING - load average: 5.21, 5.58, 5.13 [04:55:18] PROBLEM Disk Space is now: CRITICAL on dumps-5 dumps-5 output: Connection refused by host [04:56:08] PROBLEM Free ram is now: CRITICAL on dumps-5 dumps-5 output: Connection refused by host [04:57:28] PROBLEM Total Processes is now: CRITICAL on dumps-5 dumps-5 output: Connection refused by host [04:58:18] PROBLEM dpkg-check is now: CRITICAL on dumps-5 dumps-5 output: Connection refused by host [04:58:58] RECOVERY Current Load is now: OK on dumps-8 dumps-8 output: OK - load average: 0.00, 0.60, 0.78 [05:09:48] RECOVERY Current Load is now: OK on dumps-nfs1 dumps-nfs1 output: OK - load average: 3.13, 3.80, 4.86 [05:15:42] !log dumps testing [05:16:45] !log bots Restarted morebots (again) [05:16:54] zzz [05:17:14] !log bots Restarted morebots (again) [05:17:23] ZZZ its freaking broken [05:33:18] RECOVERY dpkg-check is now: OK on dumps-5 dumps-5 output: All packages OK [05:33:58] RECOVERY Current Load is now: OK on dumps-5 dumps-5 output: OK - load average: 0.24, 0.15, 0.10 [05:34:38] RECOVERY Current Users is now: OK on dumps-5 dumps-5 output: USERS OK - 0 users currently logged in [05:35:18] RECOVERY Disk Space is now: OK on dumps-5 dumps-5 output: DISK OK [05:36:08] RECOVERY Free ram is now: OK on dumps-5 dumps-5 output: OK: 91% free memory [05:37:28] RECOVERY Total Processes is now: OK on dumps-5 dumps-5 output: PROCS OK: 80 processes [06:07:53] PROBLEM Current Load is now: WARNING on dumps-nfs1 dumps-nfs1 output: WARNING - load average: 7.35, 6.84, 5.45 [06:56:50] PROBLEM Current Load is now: WARNING on bots-sql3 bots-sql3 output: WARNING - load average: 6.76, 7.63, 5.61 [06:57:00] PROBLEM Disk Space is now: CRITICAL on dumpster01 dumpster01 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:57:00] PROBLEM Current Users is now: CRITICAL on dumpster01 dumpster01 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:57:00] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 1.56, 4.56, 3.12 [07:02:26] RECOVERY Disk Space is now: OK on dumpster01 dumpster01 output: DISK OK [07:02:26] RECOVERY Current Users is now: OK on dumpster01 dumpster01 output: USERS OK - 0 users currently logged in [07:12:23] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 0.43, 1.47, 2.32 [07:22:23] RECOVERY Current Load is now: OK on bots-sql3 bots-sql3 output: OK - load average: 0.61, 2.00, 4.57 [08:24:38] !log bots Restarted morebots [08:24:55] !log bots Restarted morebots [08:24:56] Logged the message, Master [08:25:04] W00t [09:57:52] PROBLEM Current Load is now: WARNING on dumps-nfs1 dumps-nfs1 output: WARNING - load average: 7.85, 6.62, 5.50 [10:22:42] RECOVERY Current Load is now: OK on dumps-nfs1 dumps-nfs1 output: OK - load average: 3.90, 4.18, 4.88 [10:30:42] PROBLEM Current Load is now: WARNING on dumps-nfs1 dumps-nfs1 output: WARNING - load average: 3.87, 5.16, 5.20 [11:04:28] !log deployment-prep Testing a switch to het deploy [12:02:45] hey petan [12:34:57] !log bots Lets hope I don't break again [12:36:00] !log bots Lets hope I don't break again [12:36:52] !log bots Lets hope I don't break again [12:37:04] ahh [12:38:00] !log bots Fixed issue with project.cache permissions [12:38:21] !log bots Lets hope I don't break again [12:38:37] !log bots Running with sudo now... [12:38:38] Logged the message, Master [12:38:42] FINALLY [12:38:49] :p [12:38:54] win [12:39:20] !log bots Fixed logbot. People: Please start logbot with sudo python adminlogbot.py now... [12:39:20] Logged the message, Master [12:39:47] debugging shows permission errors with project.cache [12:39:55] despite chmod-ding [12:41:26] hmm... [12:41:33] !log projectthatdoesnotexist Test [12:41:34] projectthatdoesnotexist is not a valid project. [12:41:46] labs-morebots: Be good now [12:44:53] Hydriz: I bet you it's a SELinux issue [12:45:06] hmm? [12:45:25] * Hydriz Googles... [12:46:01] okay... [12:46:10] * Hydriz didn't quite understand the Wikipedia article :( [12:46:42] I just chmod 755 the file [12:46:55] was thinking if I should chmod 777 instead [12:56:13] hmph [13:21:45] !log dumps Created more instances to upload dumps to the Internet Archive. [13:21:47] Logged the message, Master [13:41:42] PROBLEM Current Load is now: WARNING on dumps-nfs1 dumps-nfs1 output: WARNING - load average: 6.20, 6.21, 5.36 [13:46:42] RECOVERY Current Load is now: OK on dumps-nfs1 dumps-nfs1 output: OK - load average: 1.23, 3.61, 4.50 [13:57:58] !project Deployment-prep [13:57:58] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Deployment-prep [13:59:26] I wonder if I can have access to deployment-prep project [13:59:42] PROBLEM Current Load is now: WARNING on dumps-nfs1 dumps-nfs1 output: WARNING - load average: 8.29, 7.56, 6.11 [14:14:56] Hydriz: then you can add the extension yourself? :) [14:15:07] apparently it's a bit of wokr you need to create a new table or something [14:15:23] err, something like that... [14:15:47] And from the logs, you claimed that we need to edit LocalSettings.php to enable it [14:16:01] actually in this cluster, its done using InitialiseSettings.php [14:17:00] okay so it's not 'normal'? [14:17:13] yeah, kinda [14:17:42] this is why making site request changes in Wikimedia itself is hard :P [15:10:25] creating a new table isn't hard [15:24:50] RECOVERY Current Load is now: OK on dumps-nfs1 dumps-nfs1 output: OK - load average: 4.53, 3.74, 4.69 [15:32:50] PROBLEM Current Load is now: WARNING on dumps-nfs1 dumps-nfs1 output: WARNING - load average: 5.20, 6.35, 5.68 [15:47:50] RECOVERY Current Load is now: OK on dumps-nfs1 dumps-nfs1 output: OK - load average: 0.99, 3.09, 4.88 [16:22:37] who is playing with beta [16:22:52] !logging [16:22:52] To log a message, use the following format: !log [16:22:57] ^ [16:26:10] PROBLEM Disk Space is now: CRITICAL on driver-dev driver-dev output: Connection refused by host [16:26:10] PROBLEM Current Users is now: CRITICAL on driver-dev driver-dev output: Connection refused by host [16:26:10] PROBLEM Current Load is now: CRITICAL on driver-dev driver-dev output: Connection refused by host [16:26:10] PROBLEM Total Processes is now: CRITICAL on driver-dev driver-dev output: Connection refused by host [16:27:30] PROBLEM SSH is now: CRITICAL on driver-dev driver-dev output: Connection refused [16:27:30] PROBLEM Free ram is now: CRITICAL on driver-dev driver-dev output: Connection refused by host [16:27:30] PROBLEM dpkg-check is now: CRITICAL on driver-dev driver-dev output: Connection refused by host [16:31:05] PROBLEM Current Load is now: WARNING on bots-3 bots-3 output: WARNING - load average: 8.03, 7.71, 5.33 [16:36:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [16:41:05] RECOVERY Current Load is now: OK on bots-3 bots-3 output: OK - load average: 3.21, 4.19, 4.72 [16:48:59] ok, one more try... [16:54:54] dang [17:01:35] PROBLEM Free ram is now: WARNING on bots-3 bots-3 output: Warning: 19% free memory [17:07:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [17:37:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [18:07:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [18:37:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [18:58:33] https://labsconsole.wikimedia.org/wiki/Main_Page [18:58:36] addshore: ^ [18:58:38] cheers [18:58:57] labs is a bunch of projects, each project consist of vm's [18:59:10] you can get a project or join some existing, then you can work on it [18:59:15] huggle is one of projects there [18:59:25] you just need to have an account and that's all [18:59:33] once you have account you can create vm's using the console [18:59:38] linked [18:59:48] account need to be created by some wmf folk [18:59:55] it's sunday so they are not here [19:00:05] but tommorow, they will likely be [19:00:06] haha [19:00:17] !accountreq [19:00:17] in case you want to have an account on labs, please contact someone who is in charge of doing that: Ryan.Lane, m.utante or ssmolle.tt [19:00:23] mutante: hi [19:00:28] awesome, i have allot of other work on so probably wont get back to doing this till wednesday or thursday [19:00:30] !account-questions [19:00:30] I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your preferred email address. 3. Your SVN account name, or your preferred shell account name, if you do not have SVN access. [19:00:41] addshore: you just need to answer this [19:00:46] that's all :) [19:00:47] easy [19:01:20] !docs [19:01:20] View complete documentation at https://labsconsole.wikimedia.org/wiki/Help:Contents [19:01:34] labs is a beta we work on it [19:06:35] RECOVERY Free ram is now: OK on bots-3 bots-3 output: OK: 20% free memory [19:07:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [19:24:35] PROBLEM Free ram is now: WARNING on bots-3 bots-3 output: Warning: 19% free memory [19:25:00] petan: where is the huggle project wiki? [19:25:18] on labs? [19:25:54] ah found it [19:29:05] PROBLEM Current Load is now: WARNING on dumps-nfs1 dumps-nfs1 output: WARNING - load average: 4.74, 5.37, 5.13 [19:34:05] RECOVERY Current Load is now: OK on dumps-nfs1 dumps-nfs1 output: OK - load average: 3.51, 4.81, 4.99 [19:37:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [19:49:35] RECOVERY Free ram is now: OK on bots-3 bots-3 output: OK: 24% free memory [20:07:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [20:37:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [21:07:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [21:07:35] PROBLEM Free ram is now: WARNING on bots-3 bots-3 output: Warning: 18% free memory [21:37:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [22:07:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [22:12:35] PROBLEM Free ram is now: CRITICAL on bots-3 bots-3 output: Critical: 5% free memory [22:30:18] how do I download files in my own directory from labs? [22:30:28] it's on the bots-3 project, I'd like to have a look at the logs [22:37:19] sftp, or rsync [22:37:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [22:42:35] RECOVERY Free ram is now: OK on bots-3 bots-3 output: OK: 20% free memory [22:50:35] PROBLEM Free ram is now: WARNING on bots-3 bots-3 output: Warning: 18% free memory [23:07:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [23:33:19] MaxSem: what happened to http://hi.wikipedia.beta.wmflabs.org ? [23:33:49] ? [23:34:00] its forgotten twice in 30 secs that I'm logged in! [23:34:22] That used to be twice in 30 mins [23:34:42] no idea [23:35:03] MaxSem:You didn't get round to fixing the config? [23:35:10] no [23:35:23] * Sid-G sighs [23:35:58] Anyways, can someone (read Reedy) fix gadgets there? [23:37:21] what do you need them for? there were no edits for several days [23:37:25] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [23:38:03] MaxSem: I've been testing a twinkle module update for a month and I need to do some final tests before I deploy it to hi-wp [23:38:48] MaxSem:There've been no edits since 29th since gadgets aren't working since then [23:39:00] if you need it urgently, fix it yourself by making dummy edit [23:39:12] how? where? [23:39:42] MediaWiki:gadgets-definition [23:40:14] haven't you seen how Reedy fixed it last time? [23:40:24] no [23:40:46] I thought it had something to do with server-side stuff