[00:00:19] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [00:08:20] there we go [00:18:29] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5 output: Critical: 4% free memory [00:30:19] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [00:38:29] RECOVERY Free ram is now: OK on bots-3 i-000000e5 output: OK: 41% free memory [01:00:19] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [01:30:19] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [02:00:19] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [02:30:19] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [02:35:19] PROBLEM Free ram is now: WARNING on incubator-bot1 i-00000251 output: Warning: 19% free memory [02:40:14] RECOVERY Free ram is now: OK on incubator-bot1 i-00000251 output: OK: 20% free memory [02:53:16] PROBLEM Free ram is now: WARNING on incubator-bot1 i-00000251 output: Warning: 19% free memory [03:00:26] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [03:32:12] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [03:36:12] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 16% free memory [03:46:22] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 15% free memory [03:46:52] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 15% free memory [03:48:12] RECOVERY Total Processes is now: OK on dumps-2 i-000002d8 output: PROCS OK: 86 processes [03:48:52] RECOVERY dpkg-check is now: OK on dumps-2 i-000002d8 output: All packages OK [03:49:32] RECOVERY Current Load is now: OK on dumps-2 i-000002d8 output: OK - load average: 0.29, 0.50, 0.37 [03:50:12] RECOVERY Current Users is now: OK on dumps-2 i-000002d8 output: USERS OK - 0 users currently logged in [03:51:12] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 5% free memory [03:51:22] RECOVERY Disk Space is now: OK on dumps-2 i-000002d8 output: DISK OK [03:51:32] RECOVERY Free ram is now: OK on dumps-2 i-000002d8 output: OK: 93% free memory [03:56:12] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 14% free memory [04:01:12] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:02:12] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [04:06:12] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 4% free memory [04:11:12] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 4% free memory [04:11:12] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 94% free memory [04:11:52] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 4% free memory [04:16:12] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 96% free memory [04:16:52] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [04:32:12] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [05:02:12] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [05:32:12] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [05:42:32] PROBLEM Free ram is now: WARNING on ganglia-test2 i-00000250 output: Warning: 19% free memory [06:02:12] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [06:03:12] PROBLEM Free ram is now: WARNING on test3 i-00000093 output: Warning: 10% free memory [06:08:12] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [06:31:25] PROBLEM Current Load is now: CRITICAL on deployment-apache30 i-000002d3 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:31:25] PROBLEM Disk Space is now: CRITICAL on deployment-apache30 i-000002d3 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:32:21] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [06:33:44] PROBLEM Total Processes is now: CRITICAL on e3 i-00000291 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:34:08] PROBLEM Total Processes is now: CRITICAL on ve-nodejs i-00000245 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:35:37] PROBLEM Current Load is now: CRITICAL on ve-nodejs i-00000245 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:35:37] PROBLEM Disk Space is now: CRITICAL on ve-nodejs i-00000245 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:35:37] PROBLEM Current Users is now: CRITICAL on ve-nodejs i-00000245 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:36:16] RECOVERY Current Load is now: OK on deployment-apache30 i-000002d3 output: OK - load average: 0.50, 1.99, 1.63 [06:36:16] RECOVERY Disk Space is now: OK on deployment-apache30 i-000002d3 output: DISK OK [06:36:31] PROBLEM Current Load is now: CRITICAL on mobile-wlm i-000002bc output: CHECK_NRPE: Socket timeout after 10 seconds. [06:36:31] PROBLEM Total Processes is now: CRITICAL on mobile-wlm i-000002bc output: CHECK_NRPE: Socket timeout after 10 seconds. [06:36:37] PROBLEM Free ram is now: CRITICAL on mobile-wlm i-000002bc output: CHECK_NRPE: Socket timeout after 10 seconds. [06:36:37] PROBLEM Current Users is now: CRITICAL on mobile-wlm i-000002bc output: CHECK_NRPE: Socket timeout after 10 seconds. [06:36:37] PROBLEM dpkg-check is now: CRITICAL on mobile-wlm i-000002bc output: CHECK_NRPE: Socket timeout after 10 seconds. [06:37:52] PROBLEM Current Users is now: CRITICAL on deployment-jobrunner05 i-0000028c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:38:59] RECOVERY Total Processes is now: OK on e3 i-00000291 output: PROCS OK: 88 processes [06:39:05] RECOVERY Total Processes is now: OK on ve-nodejs i-00000245 output: PROCS OK: 97 processes [06:40:30] RECOVERY Current Load is now: OK on ve-nodejs i-00000245 output: OK - load average: 0.34, 2.41, 1.78 [06:40:30] RECOVERY Disk Space is now: OK on ve-nodejs i-00000245 output: DISK OK [06:40:30] RECOVERY Current Users is now: OK on ve-nodejs i-00000245 output: USERS OK - 0 users currently logged in [06:41:30] RECOVERY Current Load is now: OK on mobile-wlm i-000002bc output: OK - load average: 2.78, 2.93, 1.85 [06:41:30] RECOVERY Total Processes is now: OK on mobile-wlm i-000002bc output: PROCS OK: 99 processes [06:41:35] RECOVERY Current Users is now: OK on mobile-wlm i-000002bc output: USERS OK - 0 users currently logged in [06:41:35] RECOVERY Free ram is now: OK on mobile-wlm i-000002bc output: OK: 79% free memory [06:41:35] RECOVERY dpkg-check is now: OK on mobile-wlm i-000002bc output: All packages OK [06:42:30] RECOVERY Current Users is now: OK on deployment-jobrunner05 i-0000028c output: USERS OK - 0 users currently logged in [06:44:44] PROBLEM dpkg-check is now: CRITICAL on incubator-bot0 i-00000296 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:36] PROBLEM Current Load is now: CRITICAL on maps-test2 i-00000253 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:36] PROBLEM Disk Space is now: CRITICAL on maps-test2 i-00000253 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:36] RECOVERY dpkg-check is now: OK on incubator-bot0 i-00000296 output: All packages OK [06:59:20] RECOVERY Current Load is now: OK on maps-test2 i-00000253 output: OK - load average: 1.26, 2.43, 1.83 [06:59:20] RECOVERY Disk Space is now: OK on maps-test2 i-00000253 output: DISK OK [07:03:10] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [07:33:10] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [08:03:10] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [08:33:10] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [09:03:10] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [09:33:10] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [09:42:21] PROBLEM Current Load is now: WARNING on bots-3 i-000000e5 output: WARNING - load average: 3.59, 6.33, 5.60 [09:49:27] New review: Dzahn; "per ticket" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11612 [09:49:29] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/11612 [09:52:21] RECOVERY Current Load is now: OK on bots-3 i-000000e5 output: OK - load average: 3.25, 4.19, 4.83 [10:03:39] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [10:33:59] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [11:04:01] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [11:14:01] RECOVERY Free ram is now: OK on ganglia-test2 i-00000250 output: OK: 86% free memory [11:36:20] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [12:06:20] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [12:36:20] PROBLEM host: wikistats-archive is DOWN address: i-000002d7 check_ping: Invalid hostname/address - i-000002d7 [12:45:12] Ryan_Lane: who is resposible for that? [12:45:14] ^^^ [12:45:35] no clue [12:46:12] why is it down [12:47:45] maybe it OOM'd? [12:48:27] it's been deleted [12:49:29] seem its wiki page wasn't [12:49:57] weird [12:50:03] it still shows in nova too [12:50:35] it's in dns too [12:51:46] well, I'm going to delete it [13:19:53] k [13:27:42] Ryan_Lane: can you make me operator of #wikimedia-dev? there is going to be outage of services in 30 minutes [13:27:54] so it's expected various trolls will cause troubles during that [13:28:07] that's why so many people are currently opped in all possible wm chans [13:28:52] you and Trevor are admins of that channel [13:30:11] it's just about typing /msg chanserv flags #wikimedia-dev petan +vAoti [13:36:24] op up! [13:41:53] Reedy: do you want op? [13:42:05] where? [13:42:08] here [13:42:09] I opped myself in -dev [13:42:12] ok [15:00:26] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [15:15:26] RECOVERY Free ram is now: OK on bots-sql2 i-000000af output: OK: 20% free memory [16:04:25] !log bots wmib: patching bot [16:04:26] Logged the message, Master [16:04:31] !log bots wmib: wm-bot [16:04:32] Logged the message, Master [16:06:46] @configure respond_message=true [16:06:47] This value can't be stored [16:07:10] @configure respond-message=true [16:07:11] Value true was stored into respond-message to config [16:07:14] wm-bot: hi [16:07:14] Hi petan, this is some error, I am a stupid bot and I am not intelligent enough to hold a conversation with you :-) [16:07:24] better [16:07:42] wm-bot: hi [16:07:45] even better [16:08:39] finally our bot is definitely better than one from #ubuntu [16:08:44] :D [16:09:05] wm-bot: bleh [16:09:15] lol [16:09:18] wm-bot: bleh [16:09:18] Hi petan, this is some error, I am a stupid bot and I am not intelligent enough to hold a conversation with you :-) [16:09:21] :) [16:09:27] 2 min timeout [16:09:30] to avoid spam [16:10:37] !ping [16:10:37] pong [16:22:18] !ping | test [16:22:18] test: pong [16:22:44] @configure infobot-trim-white-space-in-name=blah [16:22:44] Value of infobot-trim-white-space-in-name can't be blah, please use correct data type [16:22:51] !ping | test [16:22:51] test: pong [16:23:00] :o [16:35:46] !log bots petrb: fixing RC of wmib [16:35:48] Logged the message, Master [16:58:33] !log bots petrb: patching bot [16:58:35] Logged the message, Master [17:48:43] !project bots [17:48:43] https://labsconsole.wikimedia.org/wiki/Nova_Resource:bots [17:48:56] !log bots petrb: patching bot [17:48:58] Logged the message, Master [17:49:06] !project deployment-prep [17:49:06] https://labsconsole.wikimedia.org/wiki/Nova_Resource:deployment-prep [17:49:24] !project nagios [17:49:24] https://labsconsole.wikimedia.org/wiki/Nova_Resource:nagios [17:51:21] :x [17:51:22] !ping [17:51:22] pong [18:04:06] RECOVERY Disk Space is now: OK on ipv6test1 i-00000282 output: DISK OK [18:12:06] PROBLEM Disk Space is now: WARNING on ipv6test1 i-00000282 output: DISK WARNING - free space: / 67 MB (5% inode=57%): [18:46:17] Ryan_Lane: is there any plan how to merge console and wikitech [18:46:22] I would like to propose some [18:49:01] ok [18:49:09] we were going to mark content as "needs to be deleted" [18:49:15] then import the rest of it into labsconsole [18:49:45] we'll probably keep a dump of the content, just in case [18:51:52] I wouldn't mind reorganizing labsconsole some too [18:52:06] I don't like projects and instances being in the same namespace [18:52:12] I'd prefer projects be in the main namespace [18:53:00] either way, I'm leaving for now :) [19:13:28] Ryan_Lane: given the number of items that should be deleted I think it's easier to mark content that isn't to be deleted [19:13:29] :P [19:27:48] !log deployment-prep hashar: updating WikimediaMaintenance to get commits 1887339 913bcb8 [19:27:50] Logged the message, Master [22:09:06] PROBLEM Puppet freshness is now: CRITICAL on swift-be2 i-000001c8 output: Puppet has not run in last 20 hours [22:10:06] PROBLEM Puppet freshness is now: CRITICAL on deployment-nfs-memc i-000000d7 output: Puppet has not run in last 20 hours [22:11:06] PROBLEM Puppet freshness is now: CRITICAL on gerrit i-000000ff output: Puppet has not run in last 20 hours [22:20:06] PROBLEM Puppet freshness is now: CRITICAL on test2 i-0000013c output: Puppet has not run in last 20 hours [22:51:28] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [22:56:28] RECOVERY Free ram is now: OK on bots-sql2 i-000000af output: OK: 21% free memory