[00:00:20] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55050 MB (5% inode=99%): [00:02:42] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=2.060059/1.75, alarm hl:np_load_avg=1.373535/2.00, alarm hl:mem_free=1069.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=2.060059/1.50, alarm hl:np_load_long=1.022461/1.75, alarm hl:mem_free=1069.000000M/250M [00:04:41] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [00:33:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [00:43:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:45:01] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:46:50] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:00:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54883 MB (5% inode=99%): [01:33:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [01:44:41] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:45:10] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:46:51] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:00:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54726 MB (5% inode=99%): [02:33:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [02:44:40] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:45:10] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:46:50] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:00:20] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54581 MB (5% inode=99%): [03:33:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [03:44:41] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:45:11] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:47:50] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:00:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54496 MB (5% inode=99%): [04:45:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:45:20] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:48:04] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:01:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54356 MB (5% inode=99%): [05:03:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:45:03] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:46:19] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:49:02] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:02:20] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54228 MB (5% inode=99%): [06:10:21] 3(created) [TS-1263] Please change my SSH public key; Toolserver: Accounts; Task <10https://jira.toolserver.org/browse/TS-1263> (Junaid PV) [06:13:20] 3(updated) [TS-1263] Please change my SSH public key <10https://jira.toolserver.org/browse/TS-1263> (Junaid PV) [06:33:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [06:45:09] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:47:20] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:49:02] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:02:20] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55028 MB (5% inode=99%): [07:06:01] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.416016/1.00, alarm hl:np_load_long=0.831055/1.50, alarm hl:mem_free=15363.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.416016/1.10, alarm hl:np_load_long=0.831055/1.75, alarm hl:mem_free=15363.000000M/300M [07:07:59] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [07:33:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [07:45:10] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:47:20] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:50:01] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:02:19] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54890 MB (5% inode=99%): [08:06:03] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 45656 MB (4% inode=99%): [08:33:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [08:39:42] What's the usual lead time for account creation? https://jira.toolserver.org/browse/ACCAPP-435 has been open for a while... [08:39:54] (And who'd processing those these days?) [08:46:09] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:47:19] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:50:01] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:03:24] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54767 MB (5% inode=99%): [09:46:10] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:47:20] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:48:49] s5 replag on daphne is CRITICAL: (Service Check Timed Out) [09:48:49] s4 replag on daphne is CRITICAL: (Service Check Timed Out) [09:48:49] s2 replag on daphne is CRITICAL: (Service Check Timed Out) [09:49:20] s2 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 498.000000 [09:49:20] s5 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 467.000000 [09:50:01] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:51:09] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 508.000000 [10:03:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [10:03:20] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54631 MB (5% inode=99%): [10:17:39] NTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:18:01] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:18:11] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:18:12] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:18:19] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:18:20] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:18:30] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:18:59] SMTP on z-dat-s4-a is OK: SMTP OK - 0.002 sec. response time [10:18:59] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:19:00] SMTP on z-dat-s7-a is OK: SMTP OK - 0.002 sec. response time [10:19:00] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 3642 MB (99% inode=99%): [10:19:10] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:19:10] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:19:29] NTP on hyacinth is OK: NTP OK: Offset 7.7e-05 secs [10:38:02] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:46:20] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:47:20] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:50:59] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:03:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [11:03:20] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54489 MB (5% inode=99%): [11:46:20] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:47:19] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:51:01] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:03:30] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54340 MB (5% inode=99%): [12:33:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [12:46:20] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:48:19] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:52:00] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:04:30] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54079 MB (5% inode=99%): [13:33:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [13:47:19] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:48:20] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:52:10] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:52:22] 3(resolved) [OSM-1] Install new version of Postgres and grant users access <10https://jira.toolserver.org/browse/OSM-1> (Marlen Caemmerer) [13:52:27] 3(assigned) [MNT-1173] Add the index on the hstore columns on osm_mapnik database again, that were forgotten during the re-import <10https://jira.toolserver.org/browse/MNT-1173> (Marlen Caemmerer) [14:04:29] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53891 MB (5% inode=99%): [14:33:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [14:47:21] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:48:20] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:52:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:04:29] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54677 MB (5% inode=99%): [15:33:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [15:45:10] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.442383/1.00, alarm hl:np_load_long=0.861328/1.50, alarm hl:mem_free=14971.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.442383/1.10, alarm hl:np_load_long=0.861328/1.75, alarm hl:mem_free=14971.000000M/300M [15:46:09] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [15:47:29] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:48:29] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:53:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:04:30] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54507 MB (5% inode=99%): [16:07:25] 3(commented) [MNT-1173] Add the index on the hstore columns on osm_mapnik database again, that were forgotten during the re-import <10https://jira.toolserver.org/browse/MNT-1173> (Kai Krueger) [16:12:24] 3(resolved) [TS-1263] Please change my SSH public key <10https://jira.toolserver.org/browse/TS-1263> (DaB.) [16:33:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [16:41:43] I thought login.toolserver.org was supposed not to offer password authentication? [16:47:29] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:49:28] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:53:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:04:29] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54359 MB (5% inode=99%): [17:10:48] https://blog.wikimedia.org/2012/01/16/wikipedias-community-calls-for-anti-sopa-blackout-january-18/ [17:10:48] Editing pages on English Wikipedia via the web service API will be [17:10:48] disabled for 24 hours beginning at 05:00 UTC on Wednesday, January 18, [17:10:51] as part of the anti-SOPA/PIPA blackout. [17:12:29] nosy: can you forward or publicize appropriately? [17:13:03] Sumana Harihareswara * [Toolserver-l] Editing en.wp via API to be disabled on 18 January [17:16:53] > Your message to Toolserver-announce awaits moderator approval [17:29:21] nosy: ping? [17:33:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [17:47:30] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:49:40] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:53:20] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:04:29] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54187 MB (5% inode=99%): [18:29:20] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=2.704102/1.75, alarm hl:np_load_avg=1.627441/2.00, alarm hl:mem_free=1817.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=2.704102/1.50, alarm hl:np_load_long=1.125976/1.75, alarm hl:mem_free=1817.000000M/250M [18:30:09] Load avg. on nightshade is WARNING: WARNING - load average: 29.38, 16.47, 10.46 [18:31:10] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 30.79, 20.39, 12.32 [18:32:09] Load avg. on nightshade is WARNING: WARNING - load average: 17.47, 18.48, 12.16 [18:35:20] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [18:47:40] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:48:52] * CloudDispenser wonders if they'll be any issues with the high replag on sql-s1 (enwiki) tonight [18:49:41] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:54:19] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:03:01] zzz [19:03:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [19:04:39] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54021 MB (5% inode=99%): [19:07:20] 3(created) [DRTRIGON-112] subster_irc bot forgets wiki login when accessing othe mediawiki project; DrTrigon: DrTrigonBot - pywikipedia; Bug <10https://jira.toolserver.org/browse/DRTRIGON-112> (drtrigon) [19:09:21] 3(updated) [DRTRIGON-112] subster_irc bot forgets wiki login when accessing other mediawiki project <10https://jira.toolserver.org/browse/DRTRIGON-112> (drtrigon) [19:09:22] 3(updated) [DRTRIGON-112] subster_irc bot forgets wiki login when accessing othe mediawiki project <10https://jira.toolserver.org/browse/DRTRIGON-112> (drtrigon) [19:47:50] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:49:04] Maarten Dammers * Re: [Toolserver-l] Editing en.wp via API to be disabled on 18 January [19:49:51] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:54:20] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:00:03] OQ * Re: [Toolserver-l] Editing en.wp via API to be disabled on 18 January [20:04:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53765 MB (5% inode=99%): [20:06:59] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 45616 MB (4% inode=99%): [20:19:03] Jeff Ferland * Re: [Toolserver-l] Editing en.wp via API to be disabled on 18 January [20:33:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [20:42:03] OQ * Re: [Toolserver-l] Editing en.wp via API to be disabled on 18 January [20:44:03] Erik Moeller * Re: [Toolserver-l] Editing en.wp via API to be disabled on 18 January [20:47:51] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:50:51] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:55:03] Maarten Dammers * Re: [Toolserver-l] Editing en.wp via API to be disabled on 18 January [20:55:19] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:04:03] Erik Moeller * Re: [Toolserver-l] Editing en.wp via API to be disabled on 18 January [21:04:50] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53692 MB (5% inode=99%): [21:48:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:51:00] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:55:19] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:03:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [22:05:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54456 MB (5% inode=99%): [22:16:21] 3(commented) [MNT-1173] Add the index on the hstore columns on osm_mapnik database again, that were forgotten during the re-import <10https://jira.toolserver.org/browse/MNT-1173> (Marlen Caemmerer) [22:34:20] 3(commented) [MNT-1173] Add the index on the hstore columns on osm_mapnik database again, that were forgotten during the re-import <10https://jira.toolserver.org/browse/MNT-1173> (Kai Krueger) [22:47:24] 3(commented) [MNT-1173] Add the index on the hstore columns on osm_mapnik database again, that were forgotten during the re-import <10https://jira.toolserver.org/browse/MNT-1173> (Marlen Caemmerer) [22:49:01] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:51:01] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:52:21] 3(commented) [MNT-1173] Add the index on the hstore columns on osm_mapnik database again, that were forgotten during the re-import <10https://jira.toolserver.org/browse/MNT-1173> (Kai Krueger) [22:54:14] nosy: Are you around? [22:55:20] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:03:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [23:05:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54279 MB (5% inode=99%): [23:27:07] DaB. * [Toolserver-announce] Fwd: [Toolserver-l] Editing en.wp via API to be disabled on 18 January [23:49:09] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:51:08] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:55:21] SSH on nightshade.mgmt is CRITICAL: Server answer: