[00:00:17] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.89, 21.03, 19.54 [00:06:40] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:10:38] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [00:19:39] MySQL slave on thyme is WARNING: No slaves defined [00:36:40] Load avg. on willow is WARNING: WARNING - load average: 15.11, 17.59, 19.77 [00:49:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1049636.000000 [00:50:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [00:53:42] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:56:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:57:07] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:57:40] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:58:17] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:58:40] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.620605/1.75, alarm hl:np_load_avg=2.515137/2.0, alarm hl:mem_free=200.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.620605/1.9, alarm hl:np_load_long=2.388184/2.25, alarm hl:mem_free=200.000000M/200M, alarm hl:available=1/0 [01:00:42] Load avg. on willow is CRITICAL: CRITICAL - load average: 43.24, 25.47, 21.06 [01:06:47] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:10:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [01:19:40] MySQL slave on thyme is WARNING: No slaves defined [01:44:39] Load avg. on willow is WARNING: WARNING - load average: 15.50, 18.59, 19.92 [01:45:38] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.83, 20.78, 20.58 [01:49:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1053244.000000 [01:51:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [01:54:38] Load avg. on willow is WARNING: WARNING - load average: 19.28, 19.46, 20.00 [01:56:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:57:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:58:07] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:58:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.395020/1.75, alarm hl:np_load_avg=2.546875/2.0, alarm hl:mem_free=409.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.395020/1.9, alarm hl:np_load_long=2.547363/2.25, alarm hl:mem_free=409.000000M/200M, alarm hl:available=1/0 [02:06:50] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:08:40] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:10:48] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [02:14:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 18.80, 21.95, 21.85 [02:20:37] MySQL slave on thyme is WARNING: No slaves defined [02:49:19] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1056847.000000 [02:51:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [02:56:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:57:38] Load avg. on willow is WARNING: WARNING - load average: 15.48, 18.33, 19.96 [02:57:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:58:08] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:59:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.701172/1.75, alarm hl:np_load_avg=2.164062/2.0, alarm hl:mem_free=305.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.701172/1.9, alarm hl:np_load_long=2.438477/2.25, alarm hl:mem_free=305.000000M/200M, alarm hl:available=1/0 [03:00:38] Load avg. on willow is CRITICAL: CRITICAL - load average: 37.54, 22.51, 21.02 [03:06:58] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:10:58] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [03:13:49] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.163086/1.10, alarm hl:np_load_long=0.781250/1.55, alarm hl:mem_free=21089.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.163086/1.00, alarm hl:np_load_long=0.781250/1.50, alarm hl:mem_free=21089.000000M/350M, alarm hl:available=1/0 [03:14:50] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [03:21:37] MySQL slave on thyme is WARNING: No slaves defined [03:37:39] Load avg. on willow is WARNING: WARNING - load average: 15.18, 18.18, 19.70 [03:40:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.29, 20.09, 20.02 [03:41:39] Load avg. on willow is WARNING: WARNING - load average: 19.09, 19.07, 19.66 [03:42:58] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.741211/1.10, alarm hl:np_load_long=0.881836/1.55, alarm hl:mem_free=20586.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.741211/1.00, alarm hl:np_load_long=0.881836/1.50, alarm hl:mem_free=20586.000000M/350M, alarm hl:available=1/0 [03:47:58] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [03:50:17] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1060510.000000 [03:50:58] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.681641/1.10, alarm hl:np_load_long=1.139649/1.55, alarm hl:mem_free=20892.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.681641/1.00, alarm hl:np_load_long=1.139649/1.50, alarm hl:mem_free=20892.000000M/350M, alarm hl:available=1/0 [03:52:38] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [03:56:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:57:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:58:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:58:59] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.899414/1.75, alarm hl:np_load_avg=2.161133/2.0, alarm hl:mem_free=349.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.899414/1.9, alarm hl:np_load_long=2.279785/2.25, alarm hl:mem_free=349.000000M/200M, alarm hl:available=1/0 [04:00:37] Load avg. on willow is CRITICAL: CRITICAL - load average: 38.41, 22.56, 19.93 [04:03:38] Load avg. on willow is WARNING: WARNING - load average: 19.48, 21.22, 19.90 [04:05:28] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 124837 MB (12% inode=99%): [04:06:59] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:10:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.94, 21.61, 20.16 [04:11:08] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [04:21:37] MySQL slave on thyme is WARNING: No slaves defined [04:30:09] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:30:37] Load avg. on willow is WARNING: WARNING - load average: 19.89, 17.25, 17.82 [04:33:09] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.660645/1.75, alarm hl:np_load_avg=1.965820/2.0, alarm hl:mem_free=345.000000M/350M, alarm hl:available=1/0 [04:50:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1064109.000000 [04:52:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [04:57:07] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:58:07] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:58:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:07:08] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:12:08] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [05:21:37] MySQL slave on thyme is WARNING: No slaves defined [05:30:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 34.23, 23.38, 19.36 [05:31:39] Load avg. on willow is WARNING: WARNING - load average: 24.23, 22.50, 19.30 [05:34:08] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.484863/1.75, alarm hl:np_load_avg=2.655762/2.0, alarm hl:mem_free=118.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.484863/1.9, alarm hl:np_load_long=2.406738/2.25, alarm hl:mem_free=118.000000M/200M, alarm hl:available=1/0 [05:44:59] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.514649/1.10, alarm hl:np_load_long=0.726562/1.55, alarm hl:mem_free=20565.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.514649/1.00, alarm hl:np_load_long=0.726562/1.50, alarm hl:mem_free=20565.000000M/350M, alarm hl:available=1/0 [05:46:59] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [05:50:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1067709.000000 [05:52:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [05:57:08] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:58:08] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:58:27] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:04:07] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:07:07] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.397461/1.75, alarm hl:np_load_avg=2.267090/2.0, alarm hl:mem_free=302.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.397461/1.9, alarm hl:np_load_long=2.228516/2.25, alarm hl:mem_free=302.000000M/200M, alarm hl:available=1/0 [06:07:17] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:12:08] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [06:18:08] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:21:47] MySQL slave on thyme is WARNING: No slaves defined [06:30:47] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.16, 19.32, 17.84 [06:31:47] Load avg. on willow is WARNING: WARNING - load average: 21.09, 18.71, 17.72 [06:50:28] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1071316.000000 [06:52:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [06:57:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:58:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:58:38] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:58:53] @replag [06:58:53] Firebolt: s1-rr-a: 1w 5d 9h 43m 49s [+1.00 s/s]; s1-user: 1w 5d 9h 43m 49s [+1.00 s/s]; s2-user: 19s [+0.00 s/s]; s3-rr-a: 44s [+0.00 s/s]; s3-user: 44s [+0.00 s/s] [07:07:29] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:12:28] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.562988/1.75, alarm hl:np_load_avg=2.472656/2.0, alarm hl:mem_free=227.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.562988/1.9, alarm hl:np_load_long=2.349609/2.25, alarm hl:mem_free=227.000000M/200M, alarm hl:available=1/0 [07:12:28] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [07:19:27] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [07:21:48] MySQL slave on thyme is WARNING: No slaves defined [07:23:29] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.960938/1.75, alarm hl:np_load_avg=1.978027/2.0, alarm hl:mem_free=407.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.960938/1.9, alarm hl:np_load_long=2.119629/2.25, alarm hl:mem_free=407.000000M/200M, alarm hl:available=1/0 [07:30:47] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.07, 21.56, 18.81 [07:31:46] Load avg. on willow is WARNING: WARNING - load average: 20.05, 19.91, 18.40 [07:36:28] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [07:38:38] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:42:28] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.580078/1.75, alarm hl:np_load_avg=1.803223/2.0, alarm hl:mem_free=310.000000M/350M, alarm hl:available=1/0 [07:50:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1074923.000000 [07:52:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [07:53:28] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [07:57:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:58:37] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:58:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:07:39] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:12:38] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [08:21:47] MySQL slave on thyme is WARNING: No slaves defined [08:23:49] Load avg. on willow is OK: OK - load average: 13.04, 14.00, 14.96 [08:28:16] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:32:47] Load avg. on willow is WARNING: WARNING - load average: 15.72, 16.91, 15.86 [08:39:48] Load avg. on willow is OK: OK - load average: 11.38, 13.66, 14.80 [08:50:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1078525.000000 [08:52:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [08:57:38] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:58:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:58:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:58:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:08:02] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:13:00] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [09:19:27] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.116211/1.10, alarm hl:np_load_long=0.610352/1.55, alarm hl:mem_free=21009.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.116211/1.00, alarm hl:np_load_long=0.610352/1.50, alarm hl:mem_free=21009.000000M/350M, alarm hl:available=1/0 [09:20:26] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [09:21:58] MySQL slave on thyme is WARNING: No slaves defined [09:26:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.273438/1.75, alarm hl:np_load_avg=2.413086/2.0, alarm hl:mem_free=640.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.273438/1.9, alarm hl:np_load_long=2.460938/2.25, alarm hl:mem_free=640.000000M/200M, alarm hl:available=1/0 [09:34:04] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [09:36:00] Load avg. on willow is WARNING: WARNING - load average: 15.67, 15.61, 17.43 [09:42:58] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.853027/1.75, alarm hl:np_load_avg=1.887207/2.0, alarm hl:mem_free=302.000000M/350M, alarm hl:available=1/0 [09:49:58] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [09:51:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1082143.000000 [09:53:00] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [09:53:59] Load avg. on willow is OK: OK - load average: 10.86, 12.95, 14.88 [09:58:01] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:58:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:59:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:59:16] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:02:38] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:03:08] Load avg. on willow is WARNING: WARNING - load average: 13.86, 15.62, 15.27 [10:03:37] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [10:04:08] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.327149/1.00, alarm hl:np_load_long=0.243164/1.50, alarm hl:mem_free=342.000000M/350M, alarm hl:available=1/0 [10:08:07] Load avg. on willow is OK: OK - load average: 11.66, 14.23, 14.86 [10:08:07] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:08:37] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:08] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [10:13:07] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [10:13:08] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:19:02] emijrp * [Toolserver-l] Where is my cron file? [10:22:08] MySQL slave on thyme is WARNING: No slaves defined [10:26:03] Toto Azéro * Re: [Toolserver-l] Where is my cron file? [10:28:08] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.172363/1.00, alarm hl:np_load_long=0.215820/1.50, alarm hl:mem_free=338.000000M/350M, alarm hl:available=1/0 [10:30:08] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [10:32:08] Load avg. on willow is WARNING: WARNING - load average: 15.97, 16.34, 15.09 [10:33:02] emijrp * Re: [Toolserver-l] Where is my cron file? [10:34:08] Load avg. on willow is OK: OK - load average: 11.29, 14.44, 14.53 [10:51:58] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1085810.000000 [10:53:07] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [10:58:08] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:59:07] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:59:08] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:59:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:00:09] Load avg. on willow is WARNING: WARNING - load average: 15.90, 14.89, 14.36 [11:08:08] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:13:07] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [11:14:08] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.397461/1.75, alarm hl:np_load_avg=2.626465/2.0, alarm hl:mem_free=153.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.397461/1.9, alarm hl:np_load_long=2.260742/2.25, alarm hl:mem_free=153.000000M/200M, alarm hl:available=1/0 [11:22:08] MySQL slave on thyme is WARNING: No slaves defined [11:52:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1089419.000000 [11:53:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [11:58:18] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:59:08] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:59:17] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:59:27] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:00:18] Load avg. on willow is WARNING: WARNING - load average: 22.12, 19.91, 18.33 [12:08:13] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:13:14] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [12:14:14] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.857910/1.75, alarm hl:np_load_avg=2.287598/2.0, alarm hl:mem_free=352.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.857910/1.9, alarm hl:np_load_long=2.348633/2.25, alarm hl:mem_free=352.000000M/200M, alarm hl:available=1/0 [12:20:14] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:22:14] MySQL slave on thyme is WARNING: No slaves defined [12:28:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.048828/1.75, alarm hl:np_load_avg=1.906738/2.0, alarm hl:mem_free=511.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.048828/1.9, alarm hl:np_load_long=2.023438/2.25, alarm hl:mem_free=511.000000M/200M, alarm hl:available=1/0 [12:31:13] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.14, 22.80, 18.95 [12:32:14] Load avg. on willow is WARNING: WARNING - load average: 26.15, 22.92, 19.25 [12:39:13] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:52:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1093021.000000 [12:53:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [12:59:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:59:14] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:59:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:00:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:08:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:14:13] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [13:23:13] MySQL slave on thyme is WARNING: No slaves defined [13:31:33] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.810059/1.75, alarm hl:np_load_avg=2.499512/2.0, alarm hl:mem_free=526.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.810059/1.9, alarm hl:np_load_long=2.326660/2.25, alarm hl:mem_free=526.000000M/200M, alarm hl:available=1/0 [13:32:32] Load avg. on willow is WARNING: WARNING - load average: 18.41, 19.32, 18.47 [13:37:44] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:38:54] 3(created) [JARRY-33] Support HTTPS; Jarry's Tools; Improvement <10https://jira.toolserver.org/browse/JARRY-33> (Krinkle) [13:41:43] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.152832/1.75, alarm hl:np_load_avg=2.143555/2.0, alarm hl:mem_free=636.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.152832/1.9, alarm hl:np_load_long=2.180176/2.25, alarm hl:mem_free=636.000000M/200M, alarm hl:available=1/0 [13:48:44] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:53:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1096681.000000 [13:54:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [13:56:02] Dennis Tobar * Re: [Toolserver-l] Where is my cron file? [13:59:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:59:44] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:59:52] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:00:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:08:42] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:10:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.701172/1.75, alarm hl:np_load_avg=2.600586/2.0, alarm hl:mem_free=402.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.701172/1.9, alarm hl:np_load_long=2.341797/2.25, alarm hl:mem_free=402.000000M/200M, alarm hl:available=1/0 [14:14:22] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [14:16:54] 3(closed) [JARRY-33] Support HTTPS <10https://jira.toolserver.org/browse/JARRY-33> (Jarry1250) [14:18:02] DaB. * Re: [Toolserver-l] Where is my cron file? [14:20:40] Jarry1250: I'm get a red HTTPS icon in Chrome. So you're still loading some files over HTTP. [14:20:54] Oh, what page? [14:21:02] I had a browse around in FF [14:21:29] All [14:21:38] and Firefox isn't turning blue [14:22:33] The background is still HTTP [14:22:48] Mmm, let's have a prod then. [14:23:04] grep "http://" * [14:23:23] MySQL slave on thyme is WARNING: No slaves defined [14:24:11] Dispenser: it's defined as background:url('maps.jpg') white repeat 0 0; [14:24:33] [14:25:02] [14:25:07] Hard reload? [14:26:02] Platonides * Re: [Toolserver-l] Where is my cron file? [14:29:20] Jarry1250: translatewiki has https, https://toolserver.org/~jarry/ is still broken [14:30:05] Dispenser: You're getting on that page? Because I'm not, which suggests a caching issue. [14:30:43] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.54, 20.44, 18.88 [14:31:00] Dispenser: Oh, I do if I use Chrome. Odd. [14:31:28] Definitely, I just checked using lynx on the Toolserver [14:31:43] Load avg. on willow is WARNING: WARNING - load average: 21.99, 19.87, 18.78 [14:36:44] Dispenser: create a new window and try there [14:36:56] Chrome has a weird https cache bug that remembers something within a window [14:49:43] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [14:53:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1100291.000000 [14:54:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [14:55:45] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.549316/1.75, alarm hl:np_load_avg=2.111816/2.0, alarm hl:mem_free=871.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.549316/1.9, alarm hl:np_load_long=2.113770/2.25, alarm hl:mem_free=871.000000M/200M, alarm hl:available=1/0 [14:59:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:59:44] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:59:49] There's a ton of http stuff all over toolserver [14:59:52] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:00:08] they can be fixed but in general I'd say, install HTTPS Everywhere in firefox/chrome and get done with it [15:00:35] Rules for WMF and Toolserver were added last summer [15:00:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:08:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:09:22] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 35678 MB (8% inode=99%): [15:12:22] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22860 MB (5% inode=99%): [15:14:44] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [15:15:43] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.00, 22.48, 20.01 [15:16:44] Load avg. on willow is WARNING: WARNING - load average: 21.43, 21.69, 19.89 [15:19:22] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 84675 MB (20% inode=99%): [15:20:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.03, 23.02, 20.64 [15:23:22] MySQL slave on thyme is WARNING: No slaves defined [15:28:43] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.301758/1.00, alarm hl:np_load_long=0.238769/1.50, alarm hl:mem_free=331.000000M/350M, alarm hl:available=1/0 [15:29:43] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [15:33:45] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.275391/1.10, alarm hl:np_load_long=0.243652/1.55, alarm hl:mem_free=122.000000M/300M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.275391/1.00, alarm hl:np_load_long=0.243652/1.50, alarm hl:mem_free=122.000000M/350M, alarm hl:available=1/0 [15:40:43] Load avg. on willow is WARNING: WARNING - load average: 18.08, 17.25, 18.15 [15:50:52] 3(commented) [MNT-1225] Growing replag on S1 due to a database migration at WMF <10https://jira.toolserver.org/browse/MNT-1225> (Marlen Caemmerer) [15:53:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1103894.000000 [15:55:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [15:56:45] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.959473/1.75, alarm hl:np_load_avg=2.010742/2.0, alarm hl:mem_free=230.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.959473/1.9, alarm hl:np_load_long=2.111816/2.25, alarm hl:mem_free=230.000000M/200M, alarm hl:available=1/0 [15:59:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:59:44] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [16:00:43] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.51, 20.24, 18.14 [16:00:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:00:52] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:01:44] Load avg. on willow is WARNING: WARNING - load average: 20.29, 18.89, 17.80 [16:04:02] emijrp * Re: [Toolserver-l] Where is my cron file? [16:05:32] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 122060 MB (12% inode=99%): [16:08:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:15:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [16:19:43] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:22:04] Merlijn van Deen * Re: [Toolserver-l] Where is my cron file? [16:22:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.876953/1.75, alarm hl:np_load_avg=1.912109/2.0, alarm hl:mem_free=193.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.876953/1.9, alarm hl:np_load_long=1.974609/2.25, alarm hl:mem_free=193.000000M/200M, alarm hl:available=1/0 [16:23:22] MySQL slave on thyme is WARNING: No slaves defined [16:28:43] Load avg. on willow is OK: OK - load average: 13.80, 13.95, 14.98 [16:32:44] Load avg. on willow is WARNING: WARNING - load average: 15.04, 16.05, 15.72 [16:36:52] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:38:52] Load avg. on willow is OK: OK - load average: 10.65, 13.79, 14.93 [16:42:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.978516/1.75, alarm hl:np_load_avg=1.827637/2.0, alarm hl:mem_free=636.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.978516/1.9, alarm hl:np_load_long=1.872559/2.25, alarm hl:mem_free=636.000000M/200M, alarm hl:available=1/0 [16:53:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1107507.000000 [16:55:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [16:59:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:59:53] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [17:00:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:01:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:09:02] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:16:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [17:16:02] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.446289/1.75, alarm hl:np_load_avg=2.035645/2.0, alarm hl:mem_free=234.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.446289/1.9, alarm hl:np_load_long=1.927246/2.25, alarm hl:mem_free=234.000000M/200M, alarm hl:available=1/0 [17:18:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:23:03] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.482422/1.10, alarm hl:np_load_long=0.831055/1.55, alarm hl:mem_free=20050.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.482422/1.00, alarm hl:np_load_long=0.831055/1.50, alarm hl:mem_free=20050.000000M/350M, alarm hl:available=1/0 [17:23:44] MySQL slave on thyme is WARNING: No slaves defined [17:28:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.590820/1.75, alarm hl:np_load_avg=1.727539/2.0, alarm hl:mem_free=318.000000M/350M, alarm hl:available=1/0 [17:28:03] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [17:29:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:46:02] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.124024/1.10, alarm hl:np_load_long=0.923828/1.55, alarm hl:mem_free=19993.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.124024/1.00, alarm hl:np_load_long=0.923828/1.50, alarm hl:mem_free=19993.000000M/350M, alarm hl:available=1/0 [17:47:03] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [17:50:52] Load avg. on willow is WARNING: WARNING - load average: 15.08, 14.81, 14.21 [17:51:53] Load avg. on willow is OK: OK - load average: 10.23, 13.46, 13.77 [17:53:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1111113.000000 [17:54:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.270508/1.75, alarm hl:np_load_avg=1.561524/2.0, alarm hl:mem_free=288.000000M/350M, alarm hl:available=1/0 [17:55:02] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:55:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [17:58:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.914062/1.75, alarm hl:np_load_avg=1.828613/2.0, alarm hl:mem_free=321.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.914062/1.9, alarm hl:np_load_long=1.761719/2.25, alarm hl:mem_free=321.000000M/200M, alarm hl:available=1/0 [18:00:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:00:03] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:01:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:01:22] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:02:54] Load avg. on willow is WARNING: WARNING - load average: 16.88, 16.94, 15.24 [18:08:02] Platonides * Re: [Toolserver-l] Where is my cron file? [18:09:12] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:16:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [18:19:12] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [18:24:42] MySQL slave on thyme is WARNING: No slaves defined [18:25:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.896973/1.75, alarm hl:np_load_avg=1.853516/2.0, alarm hl:mem_free=125.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.896973/1.9, alarm hl:np_load_long=1.915527/2.25, alarm hl:mem_free=125.000000M/200M, alarm hl:available=1/0 [18:28:03] Load avg. on willow is OK: OK - load average: 11.74, 13.73, 14.82 [18:33:03] Load avg. on willow is WARNING: WARNING - load average: 14.88, 15.42, 15.30 [18:34:12] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [18:35:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 127234 MB (20% inode=99%): [18:37:13] Load avg. on willow is OK: OK - load average: 11.43, 14.32, 14.93 [18:39:22] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.133789/1.10, alarm hl:np_load_long=0.876953/1.55, alarm hl:mem_free=20375.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.133789/1.00, alarm hl:np_load_long=0.876953/1.50, alarm hl:mem_free=20375.000000M/350M, alarm hl:available=1/0 [18:41:22] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [18:45:32] [[Special:Log/newusers]] create 10 * Rzuwig * (New user account) [18:53:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1114713.000000 [18:55:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [18:56:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.024902/1.75, alarm hl:np_load_avg=1.755371/2.0, alarm hl:mem_free=332.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.024902/1.9, alarm hl:np_load_long=1.770996/2.25, alarm hl:mem_free=332.000000M/200M, alarm hl:available=1/0 [18:58:23] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [19:00:23] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:00:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:01:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:01:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.961426/1.75, alarm hl:np_load_avg=2.197754/2.0, alarm hl:mem_free=175.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.961426/1.9, alarm hl:np_load_long=1.924316/2.25, alarm hl:mem_free=175.000000M/200M, alarm hl:available=1/0 [19:01:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:09:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:15:13] Load avg. on willow is WARNING: WARNING - load average: 19.95, 17.84, 16.82 [19:16:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [19:25:04] MySQL slave on thyme is WARNING: No slaves defined [19:33:23] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [19:36:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 127077 MB (20% inode=99%): [19:38:24] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.832520/1.75, alarm hl:np_load_avg=1.821289/2.0, alarm hl:mem_free=290.000000M/350M, alarm hl:available=1/0 [19:48:53] 3(moved) [TS-1345] Create (or re-activate) account <10https://jira.toolserver.org/browse/TS-1345> (DaB.) [19:50:03] DaBPunkt, once someone submits their public key/info, how long does it normally take to get an actual account on the ts? [19:50:52] 3(commented) [ACCAPP-441] SVN, Database access and storage <10https://jira.toolserver.org/browse/ACCAPP-441> (DaB.) [19:52:52] 3(commented) [ACCAPP-458] Run bots in ptwiki <10https://jira.toolserver.org/browse/ACCAPP-458> (DaB.) [19:52:54] 3(commented) [ACCAPP-469] Help development with CVN (and commit access to p_cvn), possibly run query services <10https://jira.toolserver.org/browse/ACCAPP-469> (DaB.) [19:53:13] Load avg. on willow is OK: OK - load average: 10.38, 13.12, 14.68 [19:54:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1118331.000000 [19:54:03] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36833 MB (9% inode=99%): [19:54:53] 3(commented) [ACCAPP-465] Request for new account <10https://jira.toolserver.org/browse/ACCAPP-465> (DaB.) [19:55:40] * Dispenser pokes DaBPunkt for another update [19:55:52] Firebolt: a few days. Normaly I would have created the accounts today, but the enwiki-import postpones that a little bit [19:56:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [19:56:15] Dispenser: nothing new. revision-table is still importing [19:56:17] ah, okay. Is the enwiki-import the thing causing the replag that i've heard about? [19:56:38] Firebolt: no, that's the thing that will fix the replag-thing [19:56:44] Ah [19:56:52] okay, cool [19:56:58] thank you [19:57:11] np [19:58:03] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 18464 MB (4% inode=99%): [20:00:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:00:23] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:00:44] DaBPunkt: it is just a performance thing, the waiting for the enwiki import before creating accounts thing, right? [20:01:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:02:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:02:13] Load avg. on willow is WARNING: WARNING - load average: 14.07, 15.25, 14.71 [20:03:13] Load avg. on willow is OK: OK - load average: 12.26, 14.46, 14.46 [20:06:08] chicocvenancio: the problem is that the account-creation-script would try to create access-rules on the db-server where enwiki is importing at the moment and I prefer to not disturb the server more than necessary [20:06:36] makes sense, I just asked to make sure [20:08:04] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 30016 MB (7% inode=99%): [20:09:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:16:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [20:17:24] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:18:43] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:20:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.163086/1.75, alarm hl:np_load_avg=1.772461/2.0, alarm hl:mem_free=340.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.163086/1.9, alarm hl:np_load_long=1.781738/2.25, alarm hl:mem_free=340.000000M/200M, alarm hl:available=1/0 [20:26:03] MySQL slave on thyme is WARNING: No slaves defined [20:29:23] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:32:24] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.223144/1.10, alarm hl:np_load_long=0.210449/1.55, alarm hl:mem_free=298.000000M/300M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.223144/1.00, alarm hl:np_load_long=0.210449/1.50, alarm hl:mem_free=298.000000M/350M, alarm hl:available=1/0 [20:33:23] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [20:36:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126965 MB (20% inode=99%): [20:36:25] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.221191/1.10, alarm hl:np_load_long=0.208984/1.55, alarm hl:mem_free=256.000000M/300M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.221191/1.00, alarm hl:np_load_long=0.208984/1.50, alarm hl:mem_free=256.000000M/350M, alarm hl:available=1/0 [20:53:16] @replag [20:53:17] Joan: s1-rr-a: 1w 5d 23h 38m 12s [+1.00 s/s]; s1-user: 1w 5d 23h 38m 12s [+1.00 s/s]; s3-rr-a: 14s [-0.00 s/s]; s3-user: 14s [-0.00 s/s] [20:53:24] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.524414/1.75, alarm hl:np_load_avg=1.691895/2.0, alarm hl:mem_free=186.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.524414/1.9, alarm hl:np_load_long=1.798340/2.25, alarm hl:mem_free=186.000000M/200M, alarm hl:available=1/0 [20:54:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1121934.000000 [20:56:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [20:57:23] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [21:00:24] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:00:24] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:01:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:02:23] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:02:24] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.081055/1.75, alarm hl:np_load_avg=1.985351/2.0, alarm hl:mem_free=183.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.081055/1.9, alarm hl:np_load_long=1.887695/2.25, alarm hl:mem_free=183.000000M/200M, alarm hl:available=1/0 [21:03:25] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [21:08:52] 3(updated) [ACCAPP-468] Rcsprinter <10https://jira.toolserver.org/browse/ACCAPP-468> (David Moon) [21:08:53] 3(updated) [ACCAPP-468] Rcsprinter <10https://jira.toolserver.org/browse/ACCAPP-468> (David Moon) [21:09:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:16:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [21:27:03] MySQL slave on thyme is WARNING: No slaves defined [21:33:34] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [21:37:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126855 MB (20% inode=99%): [21:37:53] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.815918/1.75, alarm hl:np_load_avg=1.854004/2.0, alarm hl:mem_free=259.000000M/350M, alarm hl:available=1/0 [21:54:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1125536.000000 [21:54:23] @replag [21:54:24] matthewrbowker: s1-rr-a: 1w 6d 39m 20s [+1.00 s/s]; s1-user: 1w 6d 39m 20s [+1.00 s/s]; s2-user: 12s [-0.00 s/s]; s6-rr-a: 11s [-0.07 s/s]; s6-user: 11s [-0.07 s/s] [21:56:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [21:56:37] ^ An extremely funny statement ;) [21:57:34] [[User authentication]] 10https://wiki.toolserver.org/w/index.php?diff=6980&oldid=4918&rcid=9187 * Krinkle * (+252) () [21:57:43] [[User authentication]] 10https://wiki.toolserver.org/w/index.php?diff=6981&oldid=6980&rcid=9188 * Krinkle * (-1) (fix) [22:00:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:00:54] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [22:01:55] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:02:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:09:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:16:43] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [22:19:54] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:26:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.547363/1.75, alarm hl:np_load_avg=1.586914/2.0, alarm hl:mem_free=283.000000M/350M, alarm hl:available=1/0 [22:27:04] MySQL slave on thyme is WARNING: No slaves defined [22:28:54] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:29:51] nacht ts [22:37:13] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126757 MB (20% inode=99%): [22:43:43] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:54:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1129136.000000 [22:56:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [23:00:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:00:54] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [23:01:14] Load avg. on willow is WARNING: WARNING - load average: 21.77, 16.50, 13.78 [23:01:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:03:14] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [23:03:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:03:53] 3(commented) [MNT-1227] Re-Import of enwiki <10https://jira.toolserver.org/browse/MNT-1227> (DaB.) [23:06:23] Load avg. on willow is OK: OK - load average: 12.06, 14.44, 13.71 [23:08:17] [[Special:Log/newusers]] create 10 * Fmriper * (New user account) [23:09:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.564453/1.75, alarm hl:np_load_avg=1.655762/2.0, alarm hl:mem_free=242.000000M/350M, alarm hl:available=1/0 [23:09:56] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:13:54] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:16:55] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [23:26:54] 3(created) [UTRS-91] Line breaks in the template view interface aren't shown; UTRS: Main Interface; Trivial Bug <10https://jira.toolserver.org/browse/UTRS-91> (Ben Kurtovic) [23:27:12] MySQL slave on thyme is WARNING: No slaves defined [23:27:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.207031/1.75, alarm hl:np_load_avg=1.380371/2.0, alarm hl:mem_free=332.000000M/350M, alarm hl:available=1/0 [23:28:53] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:32:23] Load avg. on willow is WARNING: WARNING - load average: 16.29, 13.75, 12.94 [23:33:24] Load avg. on willow is OK: OK - load average: 13.66, 13.49, 12.90 [23:37:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126637 MB (20% inode=99%): [23:54:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1132741.000000 [23:56:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p