[07:03:46] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [07:03:46] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [07:06:26] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [07:10:06] PROBLEM Puppet freshness is now: CRITICAL on deployment-web i-00000217 output: Puppet has not run in last 20 hours [07:16:36] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [07:21:32] New patchset: Dzahn; "add missing cron jobs for lx,re,ro,pa..." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6655 [07:21:46] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/6655 [07:22:27] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/6655 [07:22:29] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6655 [07:33:46] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [07:33:46] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [07:36:26] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [07:46:36] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [08:03:46] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [08:03:46] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [08:06:26] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [08:16:36] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [08:33:46] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [08:33:46] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [08:36:26] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [08:46:36] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [08:52:43] RECOVERY Free ram is now: OK on bots-2 i-0000009c output: OK: 26% free memory [09:00:36] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c output: Warning: 18% free memory [09:03:47] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [09:03:47] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [09:06:29] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [09:16:41] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [09:34:16] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [09:36:26] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [09:38:16] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [09:48:26] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [10:05:16] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [10:06:26] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [10:08:46] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [10:18:56] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [10:20:36] RECOVERY Free ram is now: OK on bots-2 i-0000009c output: OK: 20% free memory [10:36:16] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [10:36:26] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [10:38:46] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [10:50:26] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [11:06:26] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [11:06:26] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [11:08:46] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [11:15:26] PROBLEM HTTP is now: CRITICAL on deployment-web i-00000217 output: CRITICAL - Socket timeout after 10 seconds [11:20:16] PROBLEM HTTP is now: WARNING on deployment-web i-00000217 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.007 second response time [11:21:26] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [11:36:28] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [11:36:28] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [11:38:48] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [11:49:43] PROBLEM Disk Space is now: CRITICAL on nagios 127.0.0.1 output: DISK CRITICAL - free space: /home/dzahn 1561 MB (9% inode=81%): /home/petrb 1561 MB (9% inode=81%): [11:51:38] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [12:06:38] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [12:06:39] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [12:08:48] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [12:21:48] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [12:22:38] PROBLEM Disk Space is now: WARNING on labs-nfs1 i-0000005d output: DISK WARNING - free space: /export 986 MB (5% inode=81%): /home/SAVE 986 MB (5% inode=81%): [12:27:38] RECOVERY Disk Space is now: OK on labs-nfs1 i-0000005d output: DISK OK [12:36:38] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [12:36:38] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [12:38:48] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [12:51:48] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [13:06:38] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [13:06:38] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [13:08:48] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [13:21:48] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [13:35:38] PROBLEM Disk Space is now: WARNING on labs-nfs1 i-0000005d output: DISK WARNING - free space: /export 870 MB (5% inode=81%): /home/SAVE 870 MB (5% inode=81%): [13:36:38] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [13:36:38] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [13:38:48] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [13:53:44] PROBLEM Current Load is now: CRITICAL on incubator-bot2 i-00000252 output: Connection refused by host [13:53:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [13:54:24] PROBLEM Current Users is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:55:04] PROBLEM Disk Space is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:55:44] PROBLEM Free ram is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:56:54] PROBLEM Total Processes is now: CRITICAL on incubator-bot2 i-00000252 output: Connection refused by host [13:57:34] PROBLEM dpkg-check is now: CRITICAL on incubator-bot2 i-00000252 output: Connection refused by host [14:08:04] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [14:08:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [14:08:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [14:23:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [14:38:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [14:38:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [14:38:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [14:53:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [15:04:24] PROBLEM Current Load is now: CRITICAL on maps-test2 i-00000253 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:05:04] PROBLEM Current Users is now: CRITICAL on maps-test2 i-00000253 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:05:45] PROBLEM Disk Space is now: CRITICAL on maps-test2 i-00000253 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:06:14] PROBLEM Free ram is now: CRITICAL on maps-test2 i-00000253 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:07:34] PROBLEM Total Processes is now: CRITICAL on maps-test2 i-00000253 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:08:14] PROBLEM dpkg-check is now: CRITICAL on maps-test2 i-00000253 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:08:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [15:08:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [15:08:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [15:11:14] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap1 i-000000df output: Puppet has not run in last 20 hours [15:23:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [15:38:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [15:38:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [15:38:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [15:53:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [16:08:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [16:08:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [16:08:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [16:23:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [16:28:14] PROBLEM Puppet freshness is now: CRITICAL on mobile-feeds i-000000c1 output: Puppet has not run in last 20 hours [16:38:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [16:38:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [16:38:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [16:53:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [17:08:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [17:08:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [17:08:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [17:11:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-web i-00000217 output: Puppet has not run in last 20 hours [17:23:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [17:31:58] my screens seem to have deleted themselves on bots-3? [17:38:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [17:38:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [17:38:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [17:40:24] PROBLEM HTTP is now: CRITICAL on deployment-web i-00000217 output: CRITICAL - Socket timeout after 10 seconds [17:40:47] what happened on the 30th of April and the 23rd of April?? did we have some down time? my bot seems to have stopped editing on the 30th [17:41:09] crontab stopped running too [17:45:24] PROBLEM HTTP is now: WARNING on deployment-web i-00000217 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 3.345 second response time [17:53:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [17:55:24] PROBLEM HTTP is now: CRITICAL on deployment-web i-00000217 output: CRITICAL - Socket timeout after 10 seconds [18:08:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [18:08:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [18:08:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [18:21:24] PROBLEM host: deployment-web is DOWN address: i-00000217 CRITICAL - Host Unreachable (i-00000217) [18:23:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [18:26:34] RECOVERY host: deployment-web is UP address: i-00000217 PING OK - Packet loss = 0%, RTA = 0.64 ms [18:38:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [18:38:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [18:38:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [18:53:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [19:08:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [19:08:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [19:08:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [19:14:37] !logs [19:14:38] http://bots.wmflabs.org/~wm-bot/searchlog/index.php?action=search&channel=%23wikimedia-labs&query=$1 [19:15:18] @search log [19:15:18] Results (found 10): morebots, labs-morebots, credentials, logging, terminology, newgrp, initial-login, requests, hyperon, logs, [19:15:32] Thehelpfulone: read it [19:15:35] log [19:23:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [19:35:30] petan|wk: on the 30th? [19:38:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [19:38:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [19:38:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [19:48:24] PROBLEM HTTP is now: WARNING on deployment-web i-00000217 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 2.267 second response time [19:53:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [19:58:50] Thehelpfulone: ye [20:08:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [20:08:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [20:08:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [20:23:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [20:38:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [20:38:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [20:38:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [20:53:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [21:08:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [21:08:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [21:08:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [21:23:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [21:38:24] PROBLEM HTTP is now: CRITICAL on deployment-web i-00000217 output: CRITICAL - Socket timeout after 10 seconds [21:38:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [21:38:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [21:38:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [21:43:14] PROBLEM HTTP is now: WARNING on deployment-web i-00000217 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.005 second response time [21:53:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [21:58:24] PROBLEM HTTP is now: CRITICAL on deployment-web i-00000217 output: CRITICAL - Socket timeout after 10 seconds [21:59:38] jeremyb: can you access the mailman instance? [22:00:07] sorry, haven't tried yet [22:00:30] Thehelpfulone: see my hilight elsewhere ;) [22:01:30] clever you :P [22:01:49] i know! [22:08:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [22:08:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [22:08:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [22:09:22] * jeremyb renews request: no credentials found for your account. can we add rotating BZ quips to that message? [22:09:28] or a pony? [22:12:41] Thehelpfulone: look at the top of the mailman SAL. is that current? [22:13:53] SAL? [22:14:28] !sal [22:14:28] https://labsconsole.wikimedia.org/wiki/Server_Admin_Log see it and you will know all you need [22:14:37] oh [22:14:49] there's also https://labsconsole.wikimedia.org/wiki/Nova_Resource:Mailman which has a SAL embedded [22:14:55] ah nope that's not current [22:15:04] * Thehelpfulone updates [22:15:07] and https://labsconsole.wikimedia.org/wiki/Nova_Resource:Mailman/SAL [22:15:16] which is the actual source of the SAL [22:15:19] update how? [22:15:30] !help log [22:15:30] !documentation for labs !wm-bot for bot [22:15:38] hrmm [22:15:48] !log [22:15:53] !log x [22:15:54] Message missing. Nothing logged. [22:15:58] !log x x [22:15:58] x is not a valid project. [22:16:07] !log mailman hi Thehelpfulone [22:16:08] Logged the message, Master [22:16:31] !log mailman Bhartshorne added "adding an empty placeholder file for lighttpd/htdigest so that mailman will install in labs" [labs/private] (master) - https://gerrit.wikimedia.org/r/6566 on 05/03/2012 23:04 UTC [22:16:31] Logged the message, Master [22:16:42] @search log [22:16:42] Results (found 10): morebots, labs-morebots, credentials, logging, terminology, newgrp, initial-login, requests, hyperon, logs, [22:17:05] ok, i give up [22:17:14] !log mailman puppetd -tv ran without any problems after htdigest creation [22:17:15] Logged the message, Master [22:17:54] huh, gerrit says Not Found The page you requested was not found. [22:18:28] !log Damianz configured the lighttpd.conf at /etc/lighttpd/ to make the webui work [22:18:28] Damianz is not a valid project. [22:18:33] !log mailman Damianz configured the lighttpd.conf at /etc/lighttpd/ to make the webui work [22:18:34] Logged the message, Master [22:18:54] hah, Damianz ain't a project [22:19:12] :P [22:19:17] it should be! [22:19:30] Hello...I can't create a new page on labs using special:Newpages... http://en.wikipedia.beta.wmflabs.org/wiki/Special:NewPages [22:19:38] is something going on tha tI missed? [22:19:57] huh? why would you use [[special:newpages]] to create one? [22:20:13] because I am still learning :) [22:20:42] do you have any wiki editing experience tparveen? [22:20:56] some [22:21:08] ok so what are you trying to do? :) [22:21:27] create a page user:tp [22:21:45] go to http://en.wikipedia.beta.wmflabs.org/wiki/user:tp and click edit? [22:22:08] but there's no corresponding user [22:22:20] you should create an account with a name that matches your username [22:22:26] err [22:22:30] s/account/page/ [22:23:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [22:26:19] I have one - user:tparveen... I am trying to create another one...that is why I was using special:newpages - selected usertalk:tparveen [22:26:23] and it did not work [22:26:42] so I was wondering if that feature broke...I have used it before to create user:tparveen [22:26:48] why are you creating another one? [22:27:07] to do testing...I am testing TimedMediaHandler [22:27:18] sorry, I should have started with that.. [22:27:31] * jeremyb doesn't see why multiple accounts are needed [22:28:08] anyway, i'd make the account first and then the userpage afterward [22:29:45] !damianz is Damianz is not a valid project. [22:29:45] Key exist! [22:29:51] !damianz [22:29:51] some weirdo around here [22:30:17] hey petan [22:30:21] hi [22:30:28] did you see my !logs about deployment-prep? [22:30:35] no [22:30:42] when [22:30:51] because there are no new [22:31:13] so I saw them when u logged it XD [22:31:25] http://wikitech.wikimedia.org/index.php?title=Server_admin_log&action=historysubmit&diff=46576&oldid=46574 [22:31:25] probably [22:31:30] I read all logs [22:31:40] k [22:31:41] you mean that change of group [22:31:58] oh, no. that too. i meant the UDP stuff [22:32:09] ok [22:32:23] you mean the udp2log? [22:32:29] did you change anything on that [22:32:39] i logged exactly what i did [22:32:46] when it was [22:32:47] time [22:35:58] petan: look at `GIT_DIR=/usr/local/apache/.git git log -p --color` and my link above and what that links to [22:36:09] which is https://labsconsole.wikimedia.org/w/index.php?title=Nova_Resource:Deployment-prep/SAL&diff=3391&oldid=3366 [22:36:09] ok [22:38:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [22:38:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [22:38:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [22:39:15] btw, who is salt? [22:39:18] too vague i think [22:39:26] should be sodium chloride. or something [22:39:34] PROBLEM host: mailman-01 is DOWN address: i-00000235 CRITICAL - Host Unreachable (i-00000235) [22:39:59] hm [22:40:13] jeremyb: who do I ask to get an account on that wiki? I was reading a page and noticed a typo but couldn't fix it :( [22:40:37] Thehelpfulone: well i assume you have one on labsconsole [22:40:42] Thehelpfulone: so you mean wikitech? [22:40:46] yep [22:41:27] * jeremyb suggests a gander @ http://wikitech.wikimedia.org/index.php?title=Special%3ALog&type=newusers ;-P [22:42:18] not too active. 3 months since there was one created for a non-WMF person [22:42:28] but hey, the wiki itself isn't so active either! [22:46:19] ok, I'll poke someone about it :) [22:46:40] * Thehelpfulone pokes paravoid for something unrelated [22:48:14] Thehelpfulone: saturday! [22:48:36] same argument i had with nemo last night in #-tech i guess ;) [22:53:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [23:00:14] RECOVERY host: mailman-01 is UP address: i-00000235 PING OK - Packet loss = 0%, RTA = 0.66 ms [23:03:24] RECOVERY HTTP is now: OK on mailman-01 i-00000235 output: HTTP OK: HTTP/1.1 200 OK - 453 bytes in 0.060 second response time [23:06:29] !log mailman [mailman-01] aptitude update; aptitude upgrade; reboot (upgrade included a new kernel) [23:06:30] Logged the message, Master [23:08:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [23:08:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [23:08:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [23:16:47] jeremyb: :P [23:18:36] and I'm in agreement with Nemo, RT accounts for mortals would be nice - sure RT is for vulnerabilities but not everything in there needs to be private and for such an open organization, we should strive to have public tickets where possible, or failing that access to tickets for volunteers relating to non-sensitive issues [23:18:59] no, the argument about weekend vs. notweekend ;P [23:19:17] also, bugzilla is for vulnerabilities too [23:19:35] true, but then we have those developers that were and still are volunteers that work weekends like Reedy <3 [23:19:50] I'm just checking in online before I sleep [23:19:53] Up in less than 6 hours ;) [23:20:04] sure. i'm not saying weekend makes it impossible [23:20:14] just not going to have as much luck [23:20:30] enjoy your flight Reedy, who are you flying with? [23:20:37] BA [23:20:42] ah good airline :) [23:20:45] Terminal 5? [23:20:54] When I get to LHR, yeah [23:21:25] I get to fly in the wrong direction first [23:21:30] to then fly back the same way again [23:23:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [23:26:56] Reedy: i took the train from BOS -> EWR and then flew over BOS on the way to TXL. (SJ did the same thing except he flew instead of train) [23:28:06] likely because I'm in (counts) 7 wikimedia-labs chans. [23:28:26] * jeremyb follows not [23:28:42] jeremyb: what was that you were doing with mailman? [23:28:47] sorry, 7 wikimedia- chans [23:29:05] do we need to file an RT ticket (evil RT!) for the mailman dump or a bugzilla request? [23:29:17] RT or just ask someone [23:29:26] but hold on [23:29:27] Amgine: ok no problem, but how can we help? [23:29:33] before that we have to get mailman working at all [23:30:03] yeah, but mutante wants it without the mailman server jeremyb if possible so just the webserver interface [23:30:10] THO: other way around, likely. I was just invited here. [23:30:26] well at least that's what I think, as the changes I'm planning to make are all webui only [23:30:39] Amgine: by whom? :) [23:30:41] i don't follow at all [23:30:52] ^ [23:31:07] 05 22:45:01 * jeremyb wonders why Amgine is missing from the labs channel [23:31:13] 05 23:27:17 < Amgine> jeremyb: what is the labs channel? [23:31:23] jeremyb: me? or Amgine? [23:31:32] Thehelpfulone: you [23:31:40] heh okay [23:31:50] darn. I thought it was *my* job to be obscure... [23:32:06] Thehelpfulone: anyway, i'm working on it [23:32:13] ok [23:32:23] * Thehelpfulone pulls up his logs to see what was said exactly [23:38:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [23:38:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [23:38:54] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [23:44:28] Thehelpfulone: just saw your /msg now [23:44:52] Thehelpfulone: you mostly don't need a working MTA. but that's trivial [23:45:56] Thehelpfulone: that doesn't mean you don't need a real *mailman* server [23:47:09] yeah of course [23:47:20] hmm jeremyb http://mailman.wmflabs.org/mailman/listinfo was working before [23:47:35] oh, you got an IP?! [23:47:52] http://mailman.wmflabs.org/mailman/ took me to a mailman setup but no list info pages [23:48:07] yes yes, that was within a couple of days :D [23:48:48] and here's me thinking you actually read the SAL jeremyb :P [23:49:26] nope [23:50:10] so did you delete everything Damianz had done? ;) [23:50:22] i only did exactly what i logged [23:50:35] anyway, hold your horses [23:53:44] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce)