[00:02:28] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 22.48, 17.98, 16.27  
[00:07:48] <tsnag>	 s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7231.000000  
[00:15:36] <robla>	 @replag
[00:15:37] <tsbot>	 robla: s1-pri: 1m 23s [+0.01 s/s]; s1-sec: 1m 18s [+0.00 s/s]; s2/s5-pri-c: 2h 8m 27s [+1.00 s/s]; s3-rr: 54m 52s [+0.54 s/s]; s3-user: 54m 52s [+0.54 s/s]
[00:18:28] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[00:20:57] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[00:20:58] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[00:22:47] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[00:24:07] <tsnag>	 MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1946  
[00:24:27] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 13.03, 13.99, 14.90  
[00:26:48] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[00:32:29] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 15.08, 16.09, 15.55  
[00:34:28] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 11.68, 14.41, 14.98  
[01:02:39] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 20.13, 17.75, 15.24  
[01:07:58] <tsnag>	 s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10843.000000  
[01:18:38] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[01:19:39] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 13.56, 14.15, 14.98  
[01:21:08] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[01:21:09] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[01:22:38] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 13.85, 14.81, 15.15  
[01:22:57] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[01:24:08] <tsnag>	 MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2668  
[01:24:37] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 14.09, 14.49, 14.98  
[01:27:08] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[01:40:27] <msgbot>	 3(commented) [ACCAPP-451] Urgent Account Creation <10https://jira.toolserver.org/browse/ACCAPP-451>  (Cyberpower678)
[01:45:20] <msgbot>	 3(commented) [ACCAPP-451] Urgent Account Creation <10https://jira.toolserver.org/browse/ACCAPP-451>  (Michael Movchin)
[01:50:21] <msgbot>	 3(updated) [MNT-1191] Change master of s4 <10https://jira.toolserver.org/browse/MNT-1191>  (DaB.)
[01:56:22] <msgbot>	 3(commented) [ACCAPP-451] Urgent Account Creation <10https://jira.toolserver.org/browse/ACCAPP-451>  (Cyberpower678)
[01:58:22] <msgbot>	 3(commented) [ACCAPP-451] Urgent Account Creation <10https://jira.toolserver.org/browse/ACCAPP-451>  (Cyberpower678)
[02:01:40] <kylu>	 ...
[02:06:54] <wikirc_>	 [[User:Kylu]]  10https://wiki.toolserver.org/w/index.php?diff=6666&oldid=5553&rcid=8799 * Kylu * (-263) (ciao)
[02:08:08] <tsnag>	 s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8032.000000  
[02:18:38] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[02:21:09] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[02:21:09] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[02:22:08] <tsnag>	 s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3300.000000  
[02:23:08] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[02:24:08] <tsnag>	 MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3377  
[02:27:08] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[02:32:38] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 19.29, 17.31, 15.24  
[02:33:09] <tsnag>	 s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1774.000000  
[02:36:22] <msgbot>	 3(created) [ACCAPP-454] I want to share data from the Walters Museum with Wikimedia Commons; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-454>  (Dylan Kinnett)
[02:42:38] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 12.79, 14.57, 14.99  
[02:47:23] <msgbot>	 3(commented) [ACCAPP-454] I want to share data from the Walters Museum with Wikimedia Commons <10https://jira.toolserver.org/browse/ACCAPP-454>  (Jeremy Baron)
[02:49:24] <msgbot>	 3(commented) [ACCAPP-454] I want to share data from the Walters Museum with Wikimedia Commons <10https://jira.toolserver.org/browse/ACCAPP-454>  (aude)
[02:50:08] <tsnag>	 MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3630  
[02:52:08] <tsnag>	 MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3592  
[02:54:08] <tsnag>	 MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3621  
[02:57:38] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 14.50, 15.12, 14.90  
[02:58:38] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 13.41, 14.71, 14.77  
[03:06:18] <tsnag>	 MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3581  
[03:18:38] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[03:20:38] <tsnag>	 /sql on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 13874 MB (1% inode=99%):  
[03:21:18] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[03:21:18] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[03:23:08] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[03:26:18] <tsnag>	 MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4001  
[03:27:18] <tsnag>	 MySQL slave on z-dat-s3-a is OK: Uptime: 367793  Threads: 13  Questions: 301513644  Slow queries: 18226  Opens: 1229287  Flush tables: 1  Open tables: 16383  Queries per second avg: 819.791 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0  
[03:27:18] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[03:27:37] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 17.01, 14.27, 13.99  
[03:29:37] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 14.02, 14.09, 13.96  
[04:13:38] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 14.61, 17.80, 17.64  
[04:19:37] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[04:21:18] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[04:21:18] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[04:23:17] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[04:27:18] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[04:30:13] <wikirc_>	 [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=6667&oldid=6641&rcid=8800 * 91.198.174.202 * (+1) (updated page)
[04:48:39] <tsnag>	 Load avg. on nightshade is CRITICAL: CRITICAL - load average: 24.84, 21.94, 20.09  
[04:53:38] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 14.36, 19.45, 19.73  
[04:56:39] <tsnag>	 Load avg. on nightshade is CRITICAL: CRITICAL - load average: 24.51, 20.86, 20.14  
[05:19:38] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[05:21:18] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[05:22:18] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[05:23:18] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[05:27:18] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[05:57:38] <tsnag>	 Load avg. on nightshade is CRITICAL: CRITICAL - load average: 14.67, 18.33, 21.32  
[06:11:39] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 13.74, 17.67, 19.72  
[06:19:39] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[06:21:29] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[06:22:29] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[06:23:30] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[06:27:29] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[06:52:39] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 10.43, 13.80, 14.93  
[07:02:40] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 16.89, 15.93, 15.16  
[07:08:39] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 11.29, 14.59, 14.97  
[07:20:15] <DarkoNeko>	 nagios is red, the sysadmins turned blue ;  the infra is screwed, and so are you \o\
[07:20:20] <DarkoNeko>	 morning poetryyy
[07:20:39] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[07:21:29] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[07:21:38] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 23.53, 16.88, 14.80  
[07:22:29] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[07:24:29] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[07:27:29] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[07:30:40] <tsnag>	 Load avg. on nightshade is CRITICAL: CRITICAL - load average: 30.21, 20.98, 17.24  
[07:31:39] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 29.75, 22.60, 18.06  
[07:48:03] <reba>	 Ariel T. Glenn * Re: [Toolserver-l] What happened to pagecounts stats?
[07:54:39] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 11.12, 12.26, 14.97  
[07:56:29] <tsnag>	 /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43327 MB (10% inode=99%):  
[08:01:29] <tsnag>	 /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 44721 MB (11% inode=99%):  
[08:02:40] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 17.20, 16.61, 15.87  
[08:04:29] <tsnag>	 /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43035 MB (10% inode=99%):  
[08:09:39] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 9.76, 13.09, 14.66  
[08:15:23] <msgbot>	 3(created) [ACCAPP-455] Implement a tool to create statistical overviews; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-455>  (Sebastian Schulze)
[08:21:29] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[08:21:39] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[08:22:29] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[08:24:29] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[08:27:29] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[08:27:39] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 16.53, 14.88, 13.76  
[08:29:39] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 14.33, 14.73, 13.85  
[08:30:08] <tsnag>	 s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1806.000000  
[08:38:09] <tsnag>	 s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1788.000000  
[08:47:09] <tsnag>	 s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1878.000000  
[09:02:49] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 17.09, 15.59, 14.18  
[09:07:49] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 10.17, 14.06, 14.07  
[09:09:38] <tsnag>	 s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1908.000000  
[09:21:29] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[09:21:49] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[09:22:29] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[09:25:28] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[09:27:30] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[09:32:08] <tsnag>	 s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3631.000000  
[09:38:29] <tsnag>	 Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.348145/1.75, alarm hl:np_load_avg=0.397461/2.00, alarm hl:mem_free=238.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.348145/1.50, alarm hl:np_load_long=0.427246/1.75, alarm hl:mem_free=238.000000M/250M  
[09:41:30] <tsnag>	 Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK  
[09:47:29] <tsnag>	 Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.424316/1.75, alarm hl:np_load_avg=0.416992/2.00, alarm hl:mem_free=266.000000M/300M  
[10:09:50] <tsnag>	 s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3477.000000  
[10:13:49] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3605.000000  
[10:21:29] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[10:21:48] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[10:22:29] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[10:25:28] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[10:27:30] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[10:32:09] <tsnag>	 s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6073.000000  
[11:09:25] <msgbot>	 3(commented) [TS-1287] Database S5 extremely slow <10https://jira.toolserver.org/browse/TS-1287>  (Euku)
[11:13:59] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5235.000000  
[11:21:38] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[11:21:59] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[11:22:39] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[11:24:24] <msgbot>	 3(commented) [TS-1287] Database S5 extremely slow <10https://jira.toolserver.org/browse/TS-1287>  (Michael Movchin)
[11:25:29] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[11:27:39] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[11:32:08] <tsnag>	 s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6726.000000  
[11:43:09] <tsnag>	 /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 40692 MB (4% inode=99%):  
[11:57:59] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 16.11, 14.82, 13.19  
[12:07:59] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 9.90, 14.90, 14.81  
[12:13:59] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6911.000000  
[12:14:23] <msgbot>	 3(created) [DRTRIGON-115] Configurable edit summary for subster; DrTrigon: DrTrigonBot - pywikipedia; New Feature <10https://jira.toolserver.org/browse/DRTRIGON-115>  (drtrigon)
[12:19:23] <msgbot>	 3(created) [DRTRIGON-116] Possiblity to supress error output to pages for subster; DrTrigon: DrTrigonBot - pywikipedia; New Feature <10https://jira.toolserver.org/browse/DRTRIGON-116>  (drtrigon)
[12:21:39] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[12:21:59] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[12:22:23] <msgbot>	 3(created) [DRTRIGON-117] Tool for unshorten of whole pages (and more...); DrTrigon: DrTrigonBot - toolserver scripts; New Feature <10https://jira.toolserver.org/browse/DRTRIGON-117>  (drtrigon)
[12:22:39] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[12:25:39] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[12:26:26] <msgbot>	 3(closed) [DRTRIGON-117] Tool for unshorten of whole pages (and more...) <10https://jira.toolserver.org/browse/DRTRIGON-117>  (drtrigon)
[12:27:39] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[12:32:19] <tsnag>	 s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7872.000000  
[13:01:23] <msgbot>	 3(assigned) [TS-1287] Database S5 extremely slow <10https://jira.toolserver.org/browse/TS-1287>  (Marlen Caemmerer)
[13:01:29] <msgbot>	 3(commented) [TS-1287] Database S5 extremely slow <10https://jira.toolserver.org/browse/TS-1287>  (Marlen Caemmerer)
[13:13:59] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7355.000000  
[13:21:40] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[13:21:59] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[13:22:48] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[13:25:40] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[13:28:39] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[13:32:18] <tsnag>	 s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8186.000000  
[14:14:09] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8113.000000  
[14:22:09] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[14:22:39] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[14:22:49] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[14:26:39] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[14:28:39] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[14:32:19] <tsnag>	 s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9207.000000  
[14:34:28] <msgbot>	 3(commented) [ACCAPP-454] I want to share data from the Walters Museum with Wikimedia Commons <10https://jira.toolserver.org/browse/ACCAPP-454>  (Jeremy Baron)
[14:35:01] <jeremyb>	 i'm confused, is [[Account approval process]] protected or not? i see nothing in protection log
[14:36:26] <msgbot>	 3(assigned) [ACCAPP-445] Statistics summarizer for Wikimedia Deutschland <10https://jira.toolserver.org/browse/ACCAPP-445>  (Daniel Kinzler)
[14:44:22] <mmovchin>	 jeremyb:  09:09, 5 June 2009 Daniel (Talk | contribs) m (894 bytes) (Protected "Account approval" ([edit=editpolicy] (indefinite) [move=editpolicy] (indefinite) [delete=editpolicy] (indefinite)))
[14:44:28] <mmovchin>	 It's proteted
[14:44:45] <jeremyb>	 oh, page move?
[14:45:45] <jeremyb>	     2009-06-05T05:09:50 Daniel (Talk | contribs) moved Account approval to Account approval process ‎ (revert)
[14:47:10] * jeremyb  has now fixed his time prefs ;-P

[14:47:26] <jeremyb>	 those 2 log actions were 9 secs apart
[15:00:25] <msgbot>	 3(commented) [ACCAPP-445] Statistics summarizer for Wikimedia Deutschland <10https://jira.toolserver.org/browse/ACCAPP-445>  (Kai Nissen)
[15:04:24] <msgbot>	 3(updated) [ACCAPP-445] Statistics summarizer for Wikimedia Deutschland <10https://jira.toolserver.org/browse/ACCAPP-445>  (Kai Nissen)
[15:12:58] <tsnag>	 Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.501953/1.75, alarm hl:np_load_avg=0.403809/2.00, alarm hl:mem_free=215.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.501953/1.50, alarm hl:np_load_long=0.348633/1.75, alarm hl:mem_free=215.000000M/250M  
[15:14:09] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6894.000000  
[15:17:59] <tsnag>	 Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK  
[15:20:24] <jeremyb>	 hrmmm, why are there 2 dab accounts?
[15:20:38] <jeremyb>	 onwiki and presumably in LDAP too
[15:20:59] <tsnag>	 /sql on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 13882 MB (1% inode=99%):  
[15:21:23] <jeremyb>	 likewise 2 Daniel_WMDEs
[15:22:08] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[15:22:49] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[15:22:59] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[15:26:49] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[15:28:49] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[15:32:18] <tsnag>	 s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9086.000000  
[16:02:58] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 21.72, 15.69, 11.83  
[16:11:24] <msgbot>	 3(reopened) [DAB-57] TS-Thumb tool doesn't work anymore (HTTP Status 500 - Could not get a sql-connection because pool is empty. Please try later.) <10https://jira.toolserver.org/browse/DAB-57>  (Petr Kadlec)
[16:14:19] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6693.000000  
[16:20:59] <tsnag>	 Load avg. on nightshade is CRITICAL: CRITICAL - load average: 27.41, 23.68, 20.11  
[16:22:08] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[16:22:49] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[16:22:59] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[16:26:49] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[16:28:49] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[16:32:29] <tsnag>	 s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9512.000000  
[17:14:19] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7048.000000  
[17:20:25] <msgbot>	 3(resolved) [DAB-57] TS-Thumb tool doesn't work anymore (HTTP Status 500 - Could not get a sql-connection because pool is empty. Please try later.) <10https://jira.toolserver.org/browse/DAB-57>  (DaB.)
[17:21:09] <tsnag>	 Load avg. on nightshade is CRITICAL: CRITICAL - load average: 18.23, 20.35, 22.53  
[17:22:09] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[17:22:59] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[17:23:09] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[17:26:08] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 12.71, 15.84, 19.99  
[17:27:49] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[17:28:59] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[17:32:29] <tsnag>	 s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10894.000000  
[17:50:09] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 11.40, 13.68, 14.97  
[18:03:09] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 15.88, 15.28, 14.59  
[18:09:09] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 13.29, 14.79, 14.71  
[18:14:19] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8708.000000  
[18:22:09] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[18:22:59] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[18:23:10] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[18:26:02] <robla>	 @replag
[18:26:02] <tsbot>	 robla: s2-pri: 20s [+0.00 s/s]; s2/s5-pri-c: 3h 33m 15s [+0.63 s/s]; s3-rr: 19h 5m 17s [+1.00 s/s]; s3-user: 19h 5m 17s [+1.00 s/s]; s5-user: 2h 31m 52s [+0.59 s/s]
[18:26:26] <msgbot>	 3(commented) [TS-1287] Database S5 extremely slow <10https://jira.toolserver.org/browse/TS-1287>  (merl)
[18:27:49] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[18:28:59] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[18:32:29] <tsnag>	 s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12959.000000  
[19:14:19] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10510.000000  
[19:22:19] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[19:22:58] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[19:23:19] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[19:27:49] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[19:28:59] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[19:31:03] <reba>	 DaB. * [Toolserver-announce] Extraordinary hardware-maintenance 16 and 17 February
[19:32:29] <tsnag>	 s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14996.000000  
[19:50:19] <wikirc_>	 [[Admin:Datacenter-Tasks]]  10https://wiki.toolserver.org/w/index.php?diff=6668&oldid=6653&rcid=8801 * Dab * (+40) (+yarrow's card)
[19:52:52] <DaBPunkt>	 @replag
[19:52:53] <tsbot>	 DaBPunkt: s2-pri: 25m 47s [+0.29 s/s]; s2/s5-pri-c: 3h 48m 59s [+0.18 s/s]; s3-rr: 20h 32m 7s [+1.00 s/s]; s3-user: 20h 32m 7s [+1.00 s/s]; s5-user: 1h 39m 24s [-0.60 s/s]
[19:57:19] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 15.07, 13.18, 12.02  
[19:58:18] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 14.66, 13.40, 12.17  
[19:59:29] <tsnag>	 s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3306.000000  
[19:59:49] <tsnag>	 MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1926  
[20:00:19] <tsnag>	 s2 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1970.000000  
[20:02:19] <tsnag>	 Load avg. on nightshade is WARNING: WARNING - load average: 23.18, 18.38, 14.47  
[20:02:29] <tsnag>	 s5 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1748.000000  
[20:14:49] <tsnag>	 MySQL slave on daphne is OK: Uptime: 6902234  Threads: 30  Questions: 5556557515  Slow queries: 1012154  Opens: 36187968  Flush tables: 3  Open tables: 16355  Queries per second avg: 805.37 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1743  
[20:15:19] <tsnag>	 s2 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1626.000000  
[20:22:19] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[20:23:08] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[20:23:29] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[20:25:19] <tsnag>	 Load avg. on nightshade is OK: OK - load average: 10.89, 12.36, 14.79  
[20:28:50] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[20:28:59] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[20:32:29] <tsnag>	 s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8100.000000  
[20:36:23] <msgbot>	 3(commented) [TS-1287] Database S5 extremely slow <10https://jira.toolserver.org/browse/TS-1287> 
[20:51:32] <Danny_B|backup>	 do we have any cli jabber installed on nightshade?
[20:53:29] <tsnag>	 s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3544.000000  
[21:03:30] <tsnag>	 s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1377.000000  
[21:22:29] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[21:23:08] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[21:23:28] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[21:29:00] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[21:29:00] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[21:40:29] <tsnag>	 Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.466797/1.75, alarm hl:np_load_avg=0.361816/2.00, alarm hl:mem_free=1180.000000M/300M, error: no complex attribute for threshold available  
[21:42:28] <tsnag>	 Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK  
[22:04:29] <tsnag>	 Free Memory on turnera.esi is WARNING: WARNING - 9.6% (399848 kB) free!  
[22:05:28] <tsnag>	 Free Memory on turnera.esi is OK: OK - 10.9% (458040 kB) free.  
[22:22:29] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[22:23:08] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[22:23:39] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[22:29:09] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[22:29:09] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[23:03:28] <tsnag>	 Free Memory on turnera.esi is WARNING: WARNING - 9.3% (387172 kB) free!  
[23:05:28] <tsnag>	 Free Memory on turnera.esi is OK: OK - 10.7% (447172 kB) free.  
[23:22:29] <tsnag>	 Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output  
[23:23:08] <tsnag>	 SMF on damiana.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[23:23:39] <tsnag>	 SMF on turnera.esi is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[23:29:09] <tsnag>	 SSH on nightshade.mgmt is CRITICAL: Server answer:  
[23:29:09] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[23:43:29] <tsnag>	 /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 40193 MB (4% inode=99%):  
[23:44:29] <tsnag>	 Free Memory on turnera.esi is WARNING: WARNING - 9.9% (415532 kB) free!  
[23:45:29] <tsnag>	 Free Memory on turnera.esi is OK: OK - 10.4% (435456 kB) free.  
[23:49:07] <DaBPunkt>	 @replag
[23:49:08] <tsbot>	 DaBPunkt: s3-rr: 1d 28m 22s [+1.00 s/s]; s3-user: 1d 28m 22s [+1.00 s/s]
[23:49:16] <DaBPunkt>	 wtf?
[23:53:49] <tsnag>	 MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 95951