[00:02:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:04:35] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:04:36] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 240 bytes in 0.514 second response time [00:05:45] Free Memory on damiana is WARNING: WARNING - 5.1% (431504 kB) free! [00:10:25] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85754.000000 [00:10:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83607.000000 [00:11:25] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 82781.000000 [00:11:45] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 82820.000000 [00:17:45] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38772.000000 [00:17:45] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41279.000000 [00:17:45] Free Memory on damiana is OK: OK - 7.6% (634744 kB) free. [00:25:45] Free Memory on damiana is WARNING: WARNING - 5.3% (445224 kB) free! [00:27:25] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37072 [00:28:45] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14525 [00:28:45] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39799 [00:29:25] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28643 [00:36:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:36:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:36:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1309307.000000 [00:37:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:37:25] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.43, 25.81, 25.50 [00:37:25] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92494 MB (15% inode=99%): [00:38:25] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86050.000000 [00:38:25] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 135312 MB (2% inode=52%): [00:38:35] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 56589 [00:38:35] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:38:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56571.000000 [00:38:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:40:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:47:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85865.000000 [00:49:14] @replag [00:49:14] DaBPunkt: s1-rr-a: 9h 14m 38s [-3.94 s/s]; s1-rr-a-wd: 23h 10m 28s [+0.25 s/s]; s1-user: 15h 34m 56s [-1.41 s/s]; s1-user-c: 11h 6m 8s [-2.27 s/s]; s1-user-wd: 23h 51m 6s [+0.39 s/s]; s2-user: 12h 18m 20s [-2.54 s/s]; s2-user-c: 2w 1d 3h 30m 48s [-0.19 s/s]; s2-user-wd: 23h 55m 15s [+0.41 s/s] [00:49:15] DaBPunkt: s3-user: 6h 43m 42s [-4.80 s/s]; s4-user-wd: 23h 17m 44s [+0.26 s/s]; s5-user-c: 2w 1d 3h 30m 49s [-0.19 s/s]; s5-user-wd: 23h 55m 16s [+0.41 s/s]; s6-user: 10h 26m 14s [-1.72 s/s]; s6-user-wd: 23h 58m 36s [+0.46 s/s]; s7-user: 2h 6m 8s [-5.75 s/s]; s7-user-wd: 23h 10m 23s [+0.25 s/s] [00:57:46] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3381 [01:01:45] MySQL slave on z-dat-s7-a is OK: Uptime: 2453020 Threads: 4 Questions: 702844451 Slow queries: 45124 Opens: 1949914 Flush tables: 1 Open tables: 5082 Queries per second avg: 286.522 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1435 [01:02:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:04:35] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:04:35] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 241 bytes in 0.513 second response time [01:05:55] @replag [01:05:55] DaBPunkt: s1-rr-a: 8h 34m 14s [-]; s1-rr-a-wd: 23h 18m 28s [-]; s1-user: 15h 5m 57s [-]; s1-user-c: 10h 49m 55s [-]; s1-user-wd: 23h 52m 51s [-]; s2-rr: error; s2-user: 12h 5m 59s [-]; s2-user-c: 2w 1d 3h 12m 53s [-] [01:05:56] DaBPunkt: s2-user-wd: 23h 57m 12s [-]; s3-user: 5h 42m 26s [-]; s4-user-wd: 23h 25m 41s [-]; s5-user-c: 2w 1d 3h 12m 53s [-]; s5-user-wd: 23h 57m 12s [-]; s6-user: 9h 4m 10s [-]; s6-user-wd: 1d 18s [-]; s7-user-wd: 23h 18m 28s [-] [01:08:00] @replag [01:08:01] DaBPunkt: s1-rr-a: 8h 30m 43s [-1.68 s/s]; s1-rr-a-wd: 23h 19m 24s [+0.45 s/s]; s1-user: 15h 1m 58s [-1.90 s/s]; s1-user-c: 10h 45m 16s [-2.22 s/s]; s1-user-wd: 23h 53m 8s [+0.14 s/s]; s2-rr: error; s2-user: 12h 3m 43s [-1.08 s/s]; s2-user-c: 2w 1d 3h 9m 28s [-1.63 s/s] [01:08:02] DaBPunkt: s2-user-wd: 23h 57m 30s [+0.14 s/s]; s3-user: 5h 38m 58s [-1.66 s/s]; s4-user-wd: 23h 26m 45s [+0.51 s/s]; s5-user-c: 2w 1d 3h 9m 28s [-1.63 s/s]; s5-user-wd: 23h 57m 30s [+0.14 s/s]; s6-user: 8h 50m 50s [-6.37 s/s]; s6-user-wd: 1d 30s [+0.10 s/s]; s7-user-wd: 23h 19m 24s [+0.45 s/s] [01:10:36] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86462.000000 [01:10:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84483.000000 [01:11:25] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84060.000000 [01:11:45] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84069.000000 [01:17:45] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29498.000000 [01:17:46] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37949.000000 [01:19:11] @replag [01:19:11] DaBPunkt: s1-rr-a: 8h 9m 9s [-1.93 s/s]; s1-rr-a-wd: 23h 24m 53s [+0.49 s/s]; s1-user: 14h 36m 16s [-2.30 s/s]; s1-user-c: 10h 29m 52s [-1.38 s/s]; s1-user-wd: 23h 54m 8s [+0.09 s/s]; s2-rr: error; s2-user: 11h 44m 14s [-1.74 s/s]; s2-user-c: 2w 1d 2h 58m 44s [-0.96 s/s] [01:19:12] DaBPunkt: s2-user-wd: 23h 58m 29s [+0.09 s/s]; s3-user: 5h 3m 43s [-3.15 s/s]; s4-user-wd: 23h 32m 17s [+0.50 s/s]; s5-user-c: 2w 1d 2h 58m 45s [-0.96 s/s]; s5-user-wd: 23h 58m 30s [+0.09 s/s]; s6-user: 8h 25m 15s [-2.29 s/s]; s6-user-wd: 1d 3m 6s [+0.23 s/s]; s7-user: 12s [-] [01:19:13] DaBPunkt: s7-user-wd: 23h 24m 54s [+0.49 s/s] [01:19:58] @replag [01:19:58] DaBPunkt: s1-rr-a: 8h 7m 19s [-]; s1-rr-a-wd: 23h 25m 16s [-]; s1-user: 14h 35m 59s [-]; s1-user-c: 10h 28m 56s [-]; s1-user-wd: 23h 54m 15s [-]; s2-user: 11h 42m 31s [-]; s2-user-c: 2w 1d 2h 57m 57s [-]; s2-user-wd: 23h 58m 41s [-] [01:19:59] DaBPunkt: s3-user: 4h 57m 4s [-]; s4-user-wd: 23h 32m 37s [-]; s5-rr-a: 17s [-]; s5-user: 17s [-]; s5-user-c: 2w 1d 2h 57m 56s [-]; s5-user-wd: 23h 58m 41s [-]; s6-user: 8h 22m 59s [-]; s6-user-wd: 1d 3m 15s [-] [01:20:00] DaBPunkt: s7-user-wd: 23h 25m 17s [-] [01:20:21] @replag [01:20:22] DaBPunkt: s1-rr-a: 8h 6m 17s [-2.63 s/s]; s1-rr-a-wd: 23h 25m 29s [+0.55 s/s]; s1-user: 14h 36m 6s [+0.30 s/s]; s1-user-c: 10h 28m 13s [-1.83 s/s]; s1-user-wd: 23h 54m 16s [+0.04 s/s]; s2-user: 11h 41m 30s [-2.59 s/s]; s2-user-c: 2w 1d 2h 57m 31s [-1.11 s/s]; s2-user-wd: 23h 58m 43s [+0.09 s/s] [01:20:23] DaBPunkt: s3-user: 4h 55m 57s [-2.85 s/s]; s4-user-wd: 23h 32m 48s [+0.47 s/s]; s5-rr-a: 14s [-0.13 s/s]; s5-user: 14s [-0.13 s/s]; s5-user-c: 2w 1d 2h 57m 31s [-1.06 s/s]; s5-user-wd: 23h 58m 43s [+0.09 s/s]; s6-user: 8h 21m 17s [-4.34 s/s]; s6-user-wd: 1d 3m 18s [+0.13 s/s] [01:20:24] DaBPunkt: s7-user-wd: 23h 25m 29s [+0.51 s/s] [01:21:25] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 32.26, 23.20, 14.14 [01:22:25] Load avg. on ortelius is WARNING: WARNING - load average: 27.29, 23.62, 14.88 [01:24:39] nacht ts [01:27:25] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27759 [01:27:25] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 28.34, 25.45, 17.99 [01:35:46] um [01:35:50] did everything just die? [01:37:32] @replag [01:37:46] Thegratgrabber: the ts just went down [01:37:57] Wait a little while [01:37:59] then refresh [01:38:05] Networking stuff is going on in esams [01:38:20] ok [01:38:32] i cant ssh into nightshade nor willow right now :/ [01:38:41] still the same answer ;) [01:38:47] Thegratgrabber: s1-rr-a: 7h 3m 50s [-3.39 s/s]; s1-rr-a-wd: 23h 31m 18s [+0.32 s/s]; s1-user: 14h 12m 44s [-1.27 s/s]; s1-user-c: 10h 19m 44s [-0.46 s/s]; s1-user-wd: 1d 7m 22s [+0.71 s/s]; s2-rr: 2m 19s [-]; s2-user: 11h 21m 39s [-1.08 s/s]; s2-user-c: 2w 1d 2h 59m 31s [+0.11 s/s] [01:38:47] :P [01:38:48] Thegratgrabber: s2-user-wd: 1d 10m 2s [+0.61 s/s]; s3-user: 4h 2m 52s [-2.88 s/s]; s4-rr-a: 2m 1s [-]; s4-user: 2m 1s [-]; s4-user-wd: 23h 47m 10s [+0.78 s/s]; s5-rr-a: 1m 50s [+0.09 s/s]; s5-user: 1m 50s [+0.09 s/s]; s5-user-c: 2w 1d 2h 59m 31s [+0.11 s/s] [01:38:49] Thegratgrabber: s5-user-wd: 1d 10m 2s [+0.61 s/s]; s6-user: 6h 51m 31s [-4.87 s/s]; s6-user-wd: 1d 15m 7s [+0.64 s/s]; s7-user: 1m 52s [-]; s7-user-wd: 23h 31m 18s [+0.32 s/s] [01:38:54] * duh will be patient [01:40:24] awesome i'm in [01:47:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87240.000000 [02:02:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:03:35] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.739 second response time [02:04:35] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:10:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88317.000000 [02:10:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84544.000000 [02:11:25] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81881.000000 [02:11:45] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81846.000000 [02:17:46] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17765.000000 [02:17:46] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32263.000000 [02:24:26] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3549 [02:26:25] Load avg. on ortelius is WARNING: WARNING - load average: 28.23, 23.56, 15.99 [02:27:25] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16293 [02:28:25] MySQL slave on z-dat-s3-a is OK: Uptime: 1978275 Threads: 13 Questions: 1755885907 Slow queries: 91980 Opens: 13297584 Flush tables: 1 Open tables: 16384 Queries per second avg: 887.584 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1743 [02:28:45] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18370 [02:36:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:36:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1302490.000000 [02:37:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:37:25] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92260 MB (15% inode=99%): [02:37:25] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.56, 22.51, 23.31 [02:38:26] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88912.000000 [02:38:26] Load avg. on ortelius is OK: OK - load average: 3.74, 11.64, 14.25 [02:38:26] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 136603 MB (2% inode=52%): [02:38:35] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44147 [02:38:35] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:38:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44129.000000 [02:38:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:40:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:47:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87545.000000 [04:29:45] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.512 second response time [04:36:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1287351.000000 [04:37:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:37:35] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92074 MB (15% inode=99%): [04:37:35] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.54, 26.86, 26.83 [04:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:38:35] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79790.000000 [04:38:35] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 137063 MB (2% inode=52%): [04:38:35] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38840 [04:38:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:38:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38838.000000 [04:38:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:40:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:47:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78269.000000 [05:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:06:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:10:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80698.000000 [05:10:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65981.000000 [05:11:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60196.000000 [05:11:55] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60165.000000 [05:11:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:29:56] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.706 second response time [05:37:05] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1282540.000000 [05:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:37:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91990 MB (15% inode=99%): [05:37:36] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.34, 24.33, 25.23 [05:37:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:38:35] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 74359.000000 [05:38:35] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 136819 MB (2% inode=52%): [05:38:35] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36941 [05:38:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36823.000000 [05:40:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:47:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 71497.000000 [06:03:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:06:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:10:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73980.000000 [06:11:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54334.000000 [06:11:44] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60069.000000 [06:11:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:11:56] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54306.000000 [06:29:56] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.528 second response time [06:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:37:35] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91902 MB (15% inode=99%): [06:37:35] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.00, 23.63, 24.38 [06:37:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1279671.000000 [06:38:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:38:35] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 136649 MB (2% inode=52%): [06:38:35] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68235.000000 [06:38:36] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [06:38:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37903.000000 [06:40:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:47:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64974.000000 [07:03:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:06:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:10:46] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 67195.000000 [07:11:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51418.000000 [07:11:44] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55939.000000 [07:11:55] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51412.000000 [07:11:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:16:45] @replag [07:16:46] liangent: s1-rr-a-wd: 14h 14m 18s [-1.65 s/s]; s1-user: 11h 8m 40s [-0.54 s/s]; s1-user-wd: 17h 30m 12s [-1.18 s/s]; s2-rr: 34s [-0.01 s/s]; s2-user: 3h 10m 58s [-1.45 s/s]; s2-user-c: 2w 18h 50m 53s [-1.45 s/s]; s2-user-wd: 18h 2m 49s [-1.09 s/s]; s3-user: 33s [-0.72 s/s] [07:16:47] liangent: s4-user-wd: 15h 28m 5s [-1.48 s/s]; s5-user-c: 2w 18h 50m 53s [-1.45 s/s]; s5-user-wd: 18h 2m 49s [-1.09 s/s]; s6-user-wd: 18h 29m 4s [-1.02 s/s]; s7-user-wd: 14h 14m 18s [-1.65 s/s] [07:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.740 second response time [07:36:55] Free Memory on damiana is WARNING: WARNING - 6.8% (569588 kB) free! [07:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:37:36] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.29, 22.10, 22.39 [07:37:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91733 MB (15% inode=99%): [07:37:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1276610.000000 [07:38:35] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63899.000000 [07:38:36] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 136279 MB (2% inode=52%): [07:38:36] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [07:38:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [07:38:56] Free Memory on damiana is OK: OK - 7.4% (621436 kB) free. [07:39:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41504.000000 [07:40:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:40:29] [[Special:Log/newusers]] create 10 * WhiteyTrece850 * (New user account) [07:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:47:56] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60192.000000 [07:48:29] [[User talk:WhiteyTrece850]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7793&rcid=21496 * WhiteyTrece850 * (+5880) (Created page with "==Think Before You Click- Tips To Approach Social Media In A Responsible Way== For businesses, the world wide web is a huge market place. Knowing how people use the internet for...") [07:59:55] [[User talk:WhiteyTrece850]] ! 10https://wiki.toolserver.org/w/index.php?diff=7794&oldid=7793&rcid=21497 * Legoktm * (-5880) (spam) [08:00:31] ohmygod [08:00:33] so much spam [08:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:06:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:10:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62160.000000 [08:11:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46775.000000 [08:11:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53599.000000 [08:11:55] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46760.000000 [08:11:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:25:38] @replag [08:25:38] liangent: s1-rr-a-wd: 12h 43m 50s [-1.31 s/s]; s1-user: 12h 17m 32s [+1.00 s/s]; s1-user-wd: 16h 25m 8s [-0.94 s/s]; s2-user: 26m 57s [-2.38 s/s]; s2-user-c: 2w 18h 10m 35s [-0.59 s/s]; s2-user-wd: 16h 56m 36s [-0.96 s/s]; s3-user: 32s [-0.00 s/s]; s4-user-wd: 14h 43m 18s [-0.65 s/s] [08:25:39] liangent: s5-user-c: 2w 18h 10m 35s [-0.59 s/s]; s5-user-wd: 16h 56m 36s [-0.96 s/s]; s6-user-wd: 16h 56m 43s [-1.34 s/s]; s7-user-wd: 12h 43m 48s [-1.31 s/s] [08:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.690 second response time [08:36:04] @replag [08:36:04] liangent: s1-rr-a-wd: 12h 47m 12s [+0.32 s/s]; s1-user: 12h 27m 58s [+1.00 s/s]; s1-user-wd: 16h 16m 3s [-0.87 s/s]; s2-rr: 31s [-0.00 s/s]; s2-user: 24m 46s [-0.21 s/s]; s2-user-c: 2w 18h 2m 34s [-0.77 s/s]; s2-user-wd: 16h 55m 21s [-0.12 s/s]; s3-user: 33s [+0.00 s/s] [08:36:05] liangent: s4-user-wd: 14h 23m 30s [-1.90 s/s]; s5-user-c: 2w 18h 2m 34s [-0.77 s/s]; s5-user-wd: 16h 55m 21s [-0.12 s/s]; s6-user-wd: 16h 55m 11s [-0.15 s/s]; s7-user-wd: 12h 47m 12s [+0.33 s/s] [08:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:37:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91516 MB (15% inode=99%): [08:37:36] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.51, 23.16, 22.68 [08:37:56] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1274442.000000 [08:38:35] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60725.000000 [08:38:35] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 135848 MB (2% inode=52%): [08:38:35] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [08:38:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45103.000000 [08:39:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:40:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:47:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58186.000000 [09:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:06:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:10:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 59534.000000 [09:11:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43723.000000 [09:11:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49105.000000 [09:11:55] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43728.000000 [09:11:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:16:05] @replag [09:16:05] liangent: s1-rr-a-wd: 12h 4m 26s [-1.07 s/s]; s1-user: 13h 8m 0s [+1.00 s/s]; s1-user-wd: 15h 55m 34s [-0.51 s/s]; s2-rr: 11s [-0.01 s/s]; s2-user: 11s [-0.61 s/s]; s2-user-c: 2w 17h 43m 55s [-0.47 s/s]; s2-user-wd: 16h 38m 15s [-0.43 s/s]; s3-user: 16s [-0.01 s/s] [09:16:06] liangent: s4-user-wd: 13h 34m 51s [-1.22 s/s]; s5-user-c: 2w 17h 43m 55s [-0.47 s/s]; s5-user-wd: 16h 38m 15s [-0.43 s/s]; s6-user-wd: 16h 28m 32s [-0.67 s/s]; s7-user: 13s [-0.00 s/s]; s7-user-wd: 12h 4m 26s [-1.07 s/s] [09:17:55] Free Memory on damiana is WARNING: WARNING - 5.9% (497860 kB) free! [09:23:55] Free Memory on damiana is OK: OK - 7.1% (598064 kB) free. [09:29:54] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.647 second response time [09:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:37:35] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.09, 22.37, 22.47 [09:37:35] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91294 MB (15% inode=99%): [09:37:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1272694.000000 [09:38:35] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 59336.000000 [09:38:35] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 133949 MB (2% inode=52%): [09:38:35] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [09:38:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48703.000000 [09:39:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:40:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:47:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54929.000000 [09:52:56] /sql on cassia is WARNING: DISK WARNING - free space: /sql 118673 MB (10% inode=99%): [09:55:55] Free Memory on damiana is WARNING: WARNING - 5.2% (437636 kB) free! [09:56:55] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (413468 kB) free! [09:57:56] Free Memory on damiana is WARNING: WARNING - 5.3% (445336 kB) free! [10:03:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:06:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:10:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56275.000000 [10:11:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40331.000000 [10:11:44] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46732.000000 [10:11:55] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40311.000000 [10:11:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.517 second response time [10:33:55] Free Memory on damiana is WARNING: WARNING - 7.0% (582480 kB) free! [10:34:55] Free Memory on damiana is OK: OK - 7.9% (665204 kB) free. [10:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:37:35] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.75, 21.80, 21.94 [10:37:35] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91079 MB (14% inode=99%): [10:37:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1270907.000000 [10:38:35] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 133304 MB (2% inode=52%): [10:38:36] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56361.000000 [10:38:36] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [10:38:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52304.000000 [10:39:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:40:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:47:56] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53378.000000 [10:59:05] Silke Meyer * Re: [Toolserver-l] Adding more roots [11:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:06:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:10:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54257.000000 [11:11:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35916.000000 [11:11:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43888.000000 [11:11:55] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35899.000000 [11:11:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.517 second response time [11:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:37:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90822 MB (14% inode=99%): [11:37:36] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.46, 21.49, 21.68 [11:37:56] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1271673.000000 [11:38:35] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 132709 MB (2% inode=52%): [11:38:35] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [11:38:35] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54984.000000 [11:38:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55903.000000 [11:39:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:40:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:47:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49876.000000 [12:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:06:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:07:56] Free Memory on damiana is WARNING: WARNING - 6.4% (538028 kB) free! [12:10:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50526.000000 [12:11:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29965.000000 [12:11:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40146.000000 [12:11:55] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29923.000000 [12:11:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:15:05] Silke Meyer * Re: [Toolserver-l] Replication stopped [12:15:55] Free Memory on damiana is OK: OK - 7.5% (625084 kB) free. [12:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.508 second response time [12:33:05] Tim Landscheidt * Re: [Toolserver-l] Adding more roots [12:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:37:36] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.55, 24.29, 24.16 [12:37:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90502 MB (14% inode=99%): [12:37:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1271687.000000 [12:38:35] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 132212 MB (2% inode=52%): [12:38:35] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52827.000000 [12:38:35] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [12:38:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:39:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 59503.000000 [12:39:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:40:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:47:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47771.000000 [12:59:05] DaB. * [Toolserver-l] s2/s5 will be read-only at Saturday (was: Re: Split of s2/s5 (cassia)) [12:59:05] DaB. * [Toolserver-announce] s2/s5 will be read-only at Saturday (was: Re: [Toolserver-l] Split of s2/s5 (cassia)) [13:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:06:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:10:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47201.000000 [13:11:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27296.000000 [13:11:44] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34419.000000 [13:11:56] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27280.000000 [13:11:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:13:55] Free Memory on damiana is WARNING: WARNING - 6.0% (499500 kB) free! [13:22:55] Free Memory on damiana is OK: OK - 7.2% (601560 kB) free. [13:28:35] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 72217 MB (7% inode=99%): [13:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.632 second response time [13:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:37:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1269607.000000 [13:38:35] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90259 MB (14% inode=99%): [13:38:35] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 131807 MB (2% inode=52%): [13:38:35] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50524.000000 [13:38:35] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.48, 22.79, 23.05 [13:38:35] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [13:38:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:39:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63103.000000 [13:39:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:40:05] DaB. * Re: [Toolserver-l] Replication stopped [13:40:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:47:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44737.000000 [14:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:06:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:10:50] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42594.000000 [14:11:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24585.000000 [14:11:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31989.000000 [14:11:56] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24594.000000 [14:11:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:22:55] Free Memory on damiana is WARNING: WARNING - 6.7% (558736 kB) free! [14:23:55] Free Memory on damiana is OK: OK - 7.1% (593164 kB) free. [14:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.516 second response time [14:33:56] Free Memory on damiana is WARNING: WARNING - 5.5% (457788 kB) free! [14:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:37:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1268103.000000 [14:38:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89922 MB (14% inode=99%): [14:38:37] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [14:38:37] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47882.000000 [14:38:37] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 131124 MB (2% inode=52%): [14:38:37] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.34, 23.38, 23.33 [14:38:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66703.000000 [14:39:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:40:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:47:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40125.000000 [14:58:57] hello all [14:59:00] @replag [14:59:00] DaBPunkt: s1-rr-a-wd: 5h 37m 20s [-1.13 s/s]; s1-user: 18h 50m 54s [+1.00 s/s]; s1-user-wd: 10h 55m 27s [-0.88 s/s]; s2-rr: 20s [+0.00 s/s]; s2-user: 20s [+0.00 s/s]; s2-user-c: 2w 16h 6m 47s [-0.28 s/s]; s2-user-wd: 12h 48m 11s [-0.67 s/s]; s3-user: 21s [+0.00 s/s] [14:59:02] DaBPunkt: s4-user-wd: 8h 17m 27s [-0.93 s/s]; s5-user-c: 2w 16h 6m 47s [-0.28 s/s]; s5-user-wd: 12h 48m 11s [-0.67 s/s]; s6-user-wd: 10h 47m 4s [-1.00 s/s]; s7-user-wd: 5h 37m 20s [-1.13 s/s] [15:03:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:06:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:10:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38540.000000 [15:11:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19171.000000 [15:11:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29556.000000 [15:11:55] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19159.000000 [15:11:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:12:26] I will stop wikidata-replication at thyme for dumping [15:20:11] <^Mike> jira appears to be down: "Unable to create and acquire lock file for jira.home directory '/root/atlassian-jira-5.0.6-standalone'." [15:21:17] ^Mike: thanks for thenotification, but we know already :-). I will try to fix it during the weekend [15:21:27] <^Mike> okay [15:21:52] <^Mike> My account has expired, and I would like to have it re-enabled. Should I wait until Jira is back, and submit a new account request, or...? [15:23:06] ^Mike: We can do it here. Which login-name? [15:23:27] <^Mike> lifeguard [15:23:56] 1 moment please [15:24:45] ^Mike: ok, try to login [15:25:16] <^Mike> DaBPunkt: Great, thanks. [15:25:21] np [15:25:46] DaBPunkt: Is willow the only solaris user sever left? [15:26:10] hoo: no. [15:29:56] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.622 second response time [15:32:55] Free Memory on damiana is WARNING: WARNING - 6.3% (531220 kB) free! [15:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:37:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1265506.000000 [15:38:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.68, 23.33, 23.55 [15:38:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89657 MB (14% inode=99%): [15:38:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 67033 [15:38:46] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42552.000000 [15:38:46] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 126283 MB (2% inode=52%): [15:38:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:38:56] Free Memory on damiana is OK: OK - 7.1% (591152 kB) free. [15:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 67058.000000 [15:40:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:41:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:48:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36479.000000 [16:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:06:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:10:46] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31728.000000 [16:11:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13153.000000 [16:11:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19304.000000 [16:11:55] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21648.000000 [16:11:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.520 second response time [16:32:51] @replag [16:32:52] DaBPunkt: s1-rr-a: 14s [-0.47 s/s]; s1-rr-a-wd: 5h 25m 59s [-0.12 s/s]; s1-user: 18h 5m 42s [-0.48 s/s]; s1-user-wd: 8h 15m 9s [-1.71 s/s]; s2-rr: 36s [+0.00 s/s]; s2-user: 36s [+0.00 s/s]; s2-user-c: 2w 14h 59m 10s [-0.72 s/s]; s2-user-wd: 9h 51m 48s [-1.88 s/s] [16:32:53] DaBPunkt: s3-user: 1m 9s [+0.01 s/s]; s4-user-wd: 4h 21m 12s [-2.52 s/s]; s5-user-c: 2w 14h 59m 10s [-0.72 s/s]; s5-user-wd: 9h 51m 48s [-1.88 s/s]; s6-user: 12s [-0.46 s/s]; s6-user-wd: 7h 21m 47s [-2.19 s/s]; s7-user: 11s [-0.00 s/s]; s7-user-wd: 2h 56m 40s [-1.71 s/s] [16:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:37:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1263448.000000 [16:38:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89370 MB (14% inode=99%): [16:38:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35142.000000 [16:38:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 65146 [16:38:46] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.39, 23.34, 23.38 [16:38:46] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 126408 MB (2% inode=52%): [16:38:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65174.000000 [16:40:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:41:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:48:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27079.000000 [16:54:56] Free Memory on damiana is WARNING: WARNING - 6.7% (560296 kB) free! [17:01:56] Free Memory on damiana is OK: OK - 8.1% (677552 kB) free. [17:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:06:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:09:35] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3425.000000 [17:11:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18714.000000 [17:11:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10360.000000 [17:11:55] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15261.000000 [17:11:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:17:34] wikidata replag on z-dat-s7-a is OK: QUERY OK: SELECT ts_rc_age() returned 1658.000000 [17:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 244 bytes in 0.514 second response time [17:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:37:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1258808.000000 [17:38:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89090 MB (14% inode=99%): [17:38:46] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27018.000000 [17:38:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 62669 [17:38:46] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 125670 MB (2% inode=52%): [17:38:46] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.59, 22.69, 22.66 [17:38:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:39:45] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3423.000000 [17:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62544.000000 [17:40:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:41:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:46:37] hey guys! [17:46:51] is it ok to use a jquery from the google cdn for a toolserver project? [17:47:12] or are people paranoid that google with track them? [17:47:19] will track them [17:47:45] wikidata replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1606.000000 [17:47:56] Free Memory on damiana is CRITICAL: CRITICAL - 4.5% (376324 kB) free! [17:48:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19008.000000 [17:50:46] or is there i good way to load the exact same jquery that the wikimedia sites load [17:51:41] it is probably burried in some scriptloader crap [17:52:55] Free Memory on damiana is WARNING: WARNING - 5.2% (435980 kB) free! [18:02:55] Free Memory on damiana is OK: OK - 7.1% (590992 kB) free. [18:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:06:48] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:11:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9656.000000 [18:11:55] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4207.000000 [18:12:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:15:55] wikidata replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3425.000000 [18:24:56] wikidata replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1682.000000 [18:27:55] Free Memory on damiana is CRITICAL: CRITICAL - 3.5% (297360 kB) free! [18:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 244 bytes in 0.644 second response time [18:31:56] Free Memory on damiana is WARNING: WARNING - 5.3% (445288 kB) free! [18:37:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:37:56] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1250013.000000 [18:38:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 59817 [18:38:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88776 MB (14% inode=99%): [18:38:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20188.000000 [18:38:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.96, 22.95, 23.31 [18:38:45] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 125152 MB (2% inode=52%): [18:38:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:39:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 59723.000000 [18:40:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:40:45] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3499.000000 [18:41:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:41:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:48:02] no opinions? [18:48:06] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11398.000000 [18:48:36] Ok, I have added the Google CDN jquery to the development version of WMA, soon to go live [18:49:11] dschwen: WMA? [18:49:20] WikiMiniAtlas [18:49:29] * hoo slaps dschwen [18:49:37] ? [18:49:42] Read the privacy policy, no 3rd party inclusions -.- [18:49:43] no, please don't [18:49:46] wikidata replag on z-dat-s6-a is OK: QUERY OK: SELECT ts_rc_age() returned 1732.000000 [18:49:54] mhh [18:49:57] TFM [18:50:02] not R'ed [18:50:04] :-( [18:50:52] dschwen: Don't do this or I have to kill WMA and contact WMF legal :7 [18:51:04] Nobody wants that [18:51:18] who is hoo? [18:51:29] dschwen: You can just load jQuery locally. [18:51:30] yeah yeah, got it, no need for leagal threats [18:51:33] Jesus Christ! [18:51:57] dschwen: Just wanted to make it crystal clear, that WMF is no fun when it comes to privacy policy violations [18:52:06] We already deflagged sysops because of that etc. [18:52:08] https://wiki.toolserver.org/view/Rules [18:52:23] Loading JavaScript from Google exposes user data. [18:54:07] hoo: it depends on the case and who is responsible for such area [18:54:40] who are you, hoo? [18:55:00] do we have a shared jquery copy on the ts? [18:55:03] dschwen: How do you mean that? I'm known as hoo [18:55:11] one that is likely to be in cahce [18:55:23] I haven't seen you or dealt with you [18:55:29] Danny_B: By the time it somehow takes it's way onto the WMF cluster there's no mercy [18:55:31] which project? [18:55:41] dschwen: "Hoo man" on wiki?! [18:56:55] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (417364 kB) free! [18:57:01] * dschwen slaps hoo back [18:57:06] satisfaction [18:57:11] dschwen: :D [18:57:33] and to think that I tell my kids that hitting is not ok :-( [18:57:57] anyhow, all traces of goog cdn erased [18:58:27] it would be nice if I could use the jquery copy from the wikimedia sites, but that is wrapped in scriptloader cruft [18:58:32] jQuery is like one file. You can just download it yourself. [18:58:48] well, susan, thanks for the hint ;-) [18:58:53] :-) [18:58:56] Free Memory on damiana is WARNING: WARNING - 5.2% (439440 kB) free! [18:59:06] captain obvious [18:59:09] dschwen: The best is probably to fetch the google CDN version via cron eg. once a day and serve the local copy ;) [18:59:21] The point is to generate cache hits! [18:59:51] the google cdn version has a very high probability to be in the user's cache [19:00:04] I have been shipping my own copy all along [19:00:23] I've just been looking at ways to make the code download faster and smaller [19:00:27] It doesn't really matter whether the Google CDN version is in users' cache. [19:00:31] It's not usable. [19:00:37] oh dear god [19:00:40] yes [19:00:50] Heh, you seem on edge. [19:00:58] you are driving me there [19:01:08] A short drive, it seems. [19:01:12] it is perfectly clear to me now what the policy [19:01:14] is [19:01:21] Cool. :-) [19:01:34] I am trying to come up with a better alternative here [19:01:43] i.e. a central toolserver jquery [19:01:59] so that atleast between toolserver projects there could be cache hits [19:02:28] also avoiding for each webdev to keep their own copied [19:02:35] dschwen: Sounds sane... a MMP could handle that [19:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:04:28] Rage. [19:04:40] A "scripts" MMP or something, sure. [19:06:05] Susan: Dunno how much that really matters, jQuery isn't that large actually [19:06:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:06:49] I already said this. ;-) [19:06:51] It's like one file. [19:06:52] in fact having a shared libraries is a good idea and i am going to request such mmp and would like to be a member of it [19:07:09] But if people are really worried about cache hits... [19:08:04] hmm, but once the jira is running again... ;-) [19:08:55] Free Memory on damiana is OK: OK - 7.3% (609260 kB) free. [19:12:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:23:51] hoo, "Read the privacy policy, no 3rd party inclusions" where did you find that bit? [19:24:15] I'm memorizing the ts rules (should have done that all along) [19:24:34] This should probably be spelled out a bit clearer if it is the intention of the Rules [19:24:41] I linked them earlier. [19:24:47] dschwen: That's the WMF rules, dunno what TS says [19:24:53] https://wiki.toolserver.org/view/Rules [19:25:03] The TS says you have to comply with WMF's privacy policy. [19:25:05] as long as it appears within WMF sites you have to follow the WMF ones... [19:25:23] Yes, susan, I have seen your link, as I said I'm reading that page now [19:25:27] Right. The mini atlas might be slightly different because it's not really a tool at toolserver.org, exactly. [19:25:38] well, it is [19:25:42] why wouldn't it be? [19:25:52] Isn't it directly included in articles? [19:25:58] Or is that something else? Little globe icon? [19:26:02] reading http://wikimediafoundation.org/wiki/Privacy_policy now [19:26:09] little globe icon [19:26:15] Right. So there's local JS. [19:26:18] Susan: It appears on click... and there's no evidence to the user that it's external, as it appears on the WMF sites [19:26:18] it is included, yeah, but still served from ts [19:26:19] Which most tools don't have. [19:26:22] That was my point. [19:26:27] oh, ok, got it [19:26:45] Anyway, nobody's upset or anything. [19:27:20] yeah, me neither [19:27:37] but I'm trying to get to the bottom of this [19:27:59] I don't see the third party thing in the WMF privacy policy either [19:28:46] my primary goal is obviously to be policy compliant and be a good citizen of the ts (and wmf) [19:29:03] my secondary goal is to provide the best user experience i can [19:29:06] It's an old rule. [19:29:11] Not sure if it's in the privacy policy. [19:29:15] like 'tradition'? [19:29:17] ;-) [19:29:23] Heh, more than tradition. [19:29:32] Because it can get you into trouble if you violate it. [19:29:39] Like de-adminning, &c. [19:29:47] I know Google Analytics is banned. [19:29:55] Free Memory on damiana is WARNING: WARNING - 5.2% (434608 kB) free! [19:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 244 bytes in 0.517 second response time [19:30:03] a rule that can get you into trouble but is not written down with the other rules is kind of an ass-rule ;-) [19:30:03] The same rules apply to loading JS or other resources from outside WMF. [19:30:06] It has to be opt-in, I think. [19:30:37] well, the gzipped WMA script pack is onlu 88k [19:30:49] so that is not too bad [19:31:13] compared to the wmf js that is loaded with each wikipage [19:31:22] The local JS wouldn't load until the globe icon is clicked, preferably. [19:31:40] the ts js is not loaded until the globe is clicked [19:31:54] there is some js that adds the globe that has to be loaded [19:32:16] but that should result in cache hits quickly [19:32:18] Is the JS loaded on all pages, or just on pages that need the globe icon? [19:32:26] It's based on {{coords}} isn't it? [19:32:30] yeah [19:33:22] there is no way to know if the globe is needed without loading a logic that checks if the globe is needed [19:33:34] templates cannot trigger script loads [19:33:49] and the globe is not contingent on the coord template only [19:34:07] Okay. [19:35:09] anyhow, WMA gas WebGL (hardware accelerated) 3D buildings now. Check it out in Manhattan, or the Petronas Towers article [19:35:34] IMO it looks better than the wireframe buildings it had before [19:35:48] Neat. :-) [19:35:49] s/gas/has/ [19:35:55] Free Memory on damiana is OK: OK - 8.0% (670852 kB) free. [19:36:07] the wireframe is still there as a fallback for newer IEs [19:36:13] dschwen: I think the rule is more accurately something like "you can only load remote JS from places that also implement Wikimedia's privacy policy." [19:36:18] has canvas but not WebGL [19:36:21] So toolserver.org is fine. [19:36:31] But google-analytics.com or Google's CDN is not. [19:36:32] Or YouTube. [19:36:34] Or whatever. [19:36:36] Unless it's opt-in. [19:36:42] Whether any of this is documented, I have no idea. :-) [19:36:47] ok [19:36:48] Yep, TS is the only not WMF driven site that's ok [19:37:24] thanks for chiing in, it is not that I don;t trust your word, it is that this should be written down to avoid future confusion [19:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:37:45] Yeah, I'll e-mail wikimedia-l about it. [19:37:46] geez, what's with my typing today? s/chiing/chiming/ [19:37:49] thanks! [19:37:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1243352.000000 [19:37:59] I agree that it's kind of a nasty grey area currently. [19:38:03] Or at least I can't see where it's spelled out. [19:38:07] It may be buried on Meta-Wiki. [19:38:09] We'll find out. [19:38:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88475 MB (14% inode=99%): [19:38:46] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11146.000000 [19:38:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57796 [19:38:46] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 124586 MB (2% inode=52%): [19:38:46] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.66, 27.62, 26.56 [19:38:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:38:49] Susan: It's not spelled out explicitly I think... but the policy rest of the policy requires that [19:39:04] hoo: Which parts? [19:39:50] Susan: eg. "Aside from the above raw log data collected for general purposes, page visits do not expose a visitor's identity publicly. Sampled raw log data may include the IP address of any user, but it is not reproduced publicly. " [19:40:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57810.000000 [19:40:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:40:38] That is a promise by WMF [19:41:05] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3467.000000 [19:41:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:41:34] http://meta.wikimedia.org/wiki/Talk:Privacy_policy#Use_of_CDN_sites_for_script_delivery [19:41:43] but mailinlist is probably a better place [19:45:19] dschwen: http://lists.wikimedia.org/pipermail/wikimedia-l/2013-February/123917.html [19:45:22] There you go. [19:53:05] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1761.000000 [20:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:06:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:12:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:26:45] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3560.000000 [20:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 244 bytes in 0.514 second response time [20:35:33] @replag [20:35:34] Betacommand: s1-rr-a: 16s [+0.00 s/s]; s1-user: 15h 44m 30s [-0.58 s/s]; s1-user-c: 19m 18s [-0.53 s/s]; s1-user-wd: 15s [-2.04 s/s]; s2-user-c: 2w 7h 54m 52s [-1.75 s/s]; s2-user-wd: 32m 0s [-2.31 s/s]; s3-user: 13s [-0.00 s/s]; s5-user-c: 2w 7h 54m 52s [-1.75 s/s] [20:35:35] Betacommand: s5-user-wd: 32m 0s [-2.31 s/s]; s6-user-wd: 15s [-1.82 s/s]; s7-user-wd: 12s [-0.73 s/s] [20:36:45] wikidata replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1634.000000 [20:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:37:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1237833.000000 [20:38:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88220 MB (14% inode=99%): [20:38:46] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.76, 21.03, 21.35 [20:38:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 56575 [20:38:46] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 124087 MB (2% inode=52%): [20:38:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56569.000000 [20:40:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:41:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:55:55] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2103.000000 [21:03:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:06:05] thanks Susan McBride [21:06:19] Heh. [21:06:46] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:10:55] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1751.000000 [21:12:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:14:28] @replag [21:14:29] DaBPunkt: s1-rr-a-wd: 1m 5s [-1.15 s/s]; s1-user: 15h 30m 47s [-0.35 s/s]; s1-user-c: 26m 18s [+0.18 s/s]; s1-user-wd: 1m 8s [+0.02 s/s]; s2-user-c: 2w 7h 3m 5s [-1.33 s/s]; s2-user-wd: 1m 5s [-0.79 s/s]; s3-user: 19s [+0.00 s/s]; s4-user-wd: 1m 4s [-0.92 s/s] [21:14:30] DaBPunkt: s5-user-c: 2w 7h 3m 5s [-1.33 s/s]; s5-user-wd: 1m 5s [-0.79 s/s]; s6-user-wd: 1m 6s [+0.02 s/s]; s7-user-wd: 1m 6s [+0.02 s/s] [21:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 244 bytes in 0.661 second response time [21:37:34] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:37:56] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1232846.000000 [21:38:20] @replag [21:38:20] DaBPunkt: s1-rr-a-wd: 1m 19s [+0.01 s/s]; s1-user: 15h 5m 14s [-1.07 s/s]; s1-user-c: 2m 45s [-0.99 s/s]; s1-user-wd: 1m 18s [+0.01 s/s]; s2-user-c: 2w 6h 26m 33s [-1.53 s/s]; s2-user-wd: 1m 18s [+0.01 s/s]; s3-user: 29s [+0.01 s/s]; s4-user-wd: 1m 19s [+0.01 s/s] [21:38:21] DaBPunkt: s5-user-c: 2w 6h 26m 33s [-1.53 s/s]; s5-user-wd: 1m 18s [+0.01 s/s]; s6-user-wd: 1m 19s [+0.01 s/s]; s7-user-wd: 1m 19s [+0.01 s/s] [21:38:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87952 MB (14% inode=99%): [21:38:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.94, 21.76, 22.01 [21:38:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54322 [21:38:45] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 123573 MB (2% inode=52%): [21:38:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54308.000000 [21:40:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:48:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:52:55] /sql on cassia is WARNING: DISK WARNING - free space: /sql 118855 MB (10% inode=99%): [22:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:07:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:12:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 244 bytes in 0.528 second response time [22:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:37:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1227365.000000 [22:38:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87698 MB (14% inode=99%): [22:38:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 19.53, 19.80, 20.20 [22:38:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50554 [22:38:46] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 121856 MB (2% inode=52%): [22:38:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50559.000000 [22:40:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:48:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:49:45] Load avg. on willow is WARNING: WARNING - load average: 18.90, 19.59, 19.99 [22:50:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.89, 19.98, 20.10 [22:53:45] Load avg. on willow is WARNING: WARNING - load average: 19.28, 19.70, 19.97 [23:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:05:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.43, 20.25, 20.04 [23:07:04] @replag [23:07:10] DaBPunkt: s1-rr-a-wd: 60s [-0.00 s/s]; s1-user: 13h 39m 13s [-0.97 s/s]; s1-user-wd: 59s [-0.00 s/s]; s2-rr: 20s [-0.00 s/s]; s2-user: 42s [+0.00 s/s]; s2-user-c: 2w 4h 8m 41s [-1.55 s/s]; s2-user-wd: 1m 15s [-0.00 s/s]; s3-user: 35s [+0.00 s/s] [23:07:11] DaBPunkt: s4-user-wd: 1m 3s [-0.00 s/s]; s5-user-c: 2w 4h 8m 42s [-1.55 s/s]; s5-user-wd: 1m 17s [-0.00 s/s]; s6-user-wd: 1m 5s [-0.00 s/s]; s7-user-wd: 1m 1s [-0.00 s/s] [23:12:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:25:45] Load avg. on willow is WARNING: WARNING - load average: 20.74, 19.95, 19.88 [23:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 244 bytes in 0.703 second response time [23:31:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.59, 20.51, 20.07 [23:32:45] Load avg. on willow is WARNING: WARNING - load average: 19.94, 20.23, 20.00 [23:35:33] Hi guys [23:35:48] I am trying to follow the steps written on https://wiki.toolserver.org/view/Account_approval_process [23:35:58] but step 1 is... broken [23:36:20] Reedy: " [23:36:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:36:48] Yeah, JIRA is broken. [23:37:32] magioladitis: basically, the system that is used for tasks, including account requests, is broken, and no-one has had the time to fix it [23:37:34] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:37:58] Hi valhallasw. [23:38:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1221006.000000 [23:38:23] Susan: I figured out that only I need for AWB is to Get a toolserver account and then get access to the AWB mmt [23:38:31] MMP, not MMT. [23:38:33] But yeah, sounds about right. [23:38:40] AWB has its code in sourceforge. We only need access to upload release versions to the toolserver [23:38:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87442 MB (14% inode=99%): [23:38:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 47094 [23:38:45] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 120119 MB (2% inode=52%): [23:38:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:38:50] You may be able to e-mail ts-admins@toolserver.org directly to request an account. [23:38:57] Or you may need to wait until JIRA is fixed. Dunno. [23:39:10] if it's urgent, send an e-mail [23:39:11] Reedy may be able to help you. [23:39:17] Reedy: " [23:39:21] You can e-mail him (sam/reedyboy.net). [23:39:33] I am on skype with him right now [23:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47107.000000 [23:39:56] There's the technical side of it, and the approvals side [23:40:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:40:30] I can probably create the account (if I can find out how it's supposed to be done on TS), but I'm not sure if I'm authorised to approve an account [23:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:41:07] Someone from WMDE has to approve the account. [23:41:10] I believe. [23:41:35] https://wiki.toolserver.org/view/Accounts [23:41:55] WMDE = WikiMedia Deutschland? [23:41:59] Yes. [23:42:09] DaBPunkt: ^ ? [23:42:21] OK I know Tim Alder [23:42:24] I can ask him [23:42:39] Alles in Ordung :) [23:42:40] DaBPunkt: could you give magioladitis some more info about getting an account while JIRA is down? [23:43:41] valhallasw: I will fix JIRA during the weekend. He/she can than request a account. [23:44:15] DaBPunkt: OK I can wait :) [23:44:44] thanks! [23:45:19] np [23:45:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.17, 21.47, 20.40 [23:47:28] Night everyone! [23:48:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:50:07] uahh [23:50:25] tons of "internal server errors" on the ts [23:50:41] 500 and 503 (service unavailable) [23:51:12] are the servers on fire? [23:52:30] dschwen: taht is not a helpful error-description [23:53:05] what tool is failing? [23:53:28] WIWOSM [23:53:33] static file serving [23:53:42] http://toolserver.org/~dschwen/wma/tiles/mapnik/0/tile_0_4.png [23:53:44] I need an url [23:53:50] oh [23:54:06] http://toolserver.org/~dschwen/wma/tiles/mapnik/0/tile_0_2.png [23:54:12] DaBPunkt: Im getting that on several of my tools randomly [23:54:21] the first one failed with 503, the second one with 500 [23:54:23] Ill reload and it goes away [23:54:29] but as Betacommand said, it is rather randomly [23:54:31] looks like a problem with 1 of the nginx. I will check [23:54:33] yep [23:54:40] thanks! [23:56:08] looks good for me now [23:58:19] not for me [23:58:28] still the same issue [23:58:44] 500 error by nginx [23:58:55] Free Memory on damiana is WARNING: WARNING - 6.7% (557272 kB) free! [23:59:21] is it possible to add the server number or ip to the error message [23:59:38] that might be helpful in detecting which server to kick