[00:12:07] nacht ts [00:12:32] nn DaBPunkt [00:13:04] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.250000/1.10, alarm hl:np_load_long=0.805664/1.55, alarm hl:mem_free=15410.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.250000/1.00, alarm hl:np_load_long=0.805664/1.50, alarm hl:mem_free=15410.000000M/300M, alarm hl:available=1/0 [00:14:05] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [00:24:04] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 402266 MB (7% inode=41%): [00:44:25] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:48:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:48:46] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:51:24] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 33095 MB (3% inode=99%): [01:24:05] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 397679 MB (7% inode=40%): [01:29:25] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37591 MB (9% inode=99%): [01:30:23] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 61496 MB (15% inode=99%): [01:45:24] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:48:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:48:46] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:59:24] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36532 MB (8% inode=99%): [02:04:32] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 51966 MB (12% inode=99%): [02:09:31] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37413 MB (9% inode=99%): [02:25:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391682 MB (7% inode=40%): [02:26:52] 3(work stopped) [UTRS-64] System/Tool notice <10https://jira.toolserver.org/browse/UTRS-64> (Hersfold) [02:26:54] 3(commented) [UTRS-64] System/Tool notice <10https://jira.toolserver.org/browse/UTRS-64> (Hersfold) [02:45:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:48:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:49:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:13:20] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 15.775 second response time [03:14:10] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 272 bytes in 10.331 second response time [03:25:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 386744 MB (7% inode=40%): [03:26:30] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37236 MB (9% inode=99%): [03:28:30] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 59810 MB (14% inode=99%): [03:33:10] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.107422/1.10, alarm hl:np_load_long=0.869140/1.55, alarm hl:mem_free=15793.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.107422/1.00, alarm hl:np_load_long=0.869140/1.50, alarm hl:mem_free=15793.000000M/300M, alarm hl:available=1/0 [03:34:10] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [03:38:31] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40118 MB (9% inode=99%): [03:45:30] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:48:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:49:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:53:10] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.122070/1.10, alarm hl:np_load_long=0.924805/1.55, alarm hl:mem_free=15589.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.122070/1.00, alarm hl:np_load_long=0.924805/1.50, alarm hl:mem_free=15589.000000M/300M, alarm hl:available=1/0 [04:24:32] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39427 MB (9% inode=99%): [04:25:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 382298 MB (7% inode=39%): [04:27:31] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 58778 MB (14% inode=99%): [04:35:53] 3(commented) [UTRS-64] System/Tool notice <10https://jira.toolserver.org/browse/UTRS-64> (Hersfold) [04:42:31] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37775 MB (9% inode=99%): [04:43:31] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [04:43:53] ssahj [04:45:32] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:47:30] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 21502 MB (5% inode=99%): [04:48:51] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:49:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:55:31] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 59368 MB (14% inode=99%): [05:25:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 377638 MB (7% inode=39%): [05:29:20] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 272 bytes in 9.944 second response time [05:45:40] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:49:50] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:50:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:02:40] Load avg. on willow is WARNING: WARNING - load average: 15.82, 13.92, 10.80 [06:02:41] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=2.001465/1.8, alarm hl:np_load_avg=1.742676/2.3, alarm hl:mem_free=1113.000000M/300M, alarm hl:available=1/0: longrun@willow exceedes load threshold: alarm hl:np_load_short=2.001465/1.9, alarm hl:np_load_long=1.350586/2.25, alarm hl:mem_free=1113.000000M/200M, alarm hl:available=1/0 [06:03:41] Load avg. on willow is OK: OK - load average: 11.37, 12.97, 10.66 [06:03:42] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [06:26:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 373599 MB (7% inode=39%): [06:45:20] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.892578/1.10, alarm hl:np_load_long=0.983398/1.55, alarm hl:mem_free=15401.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.892578/1.00, alarm hl:np_load_long=0.983398/1.50, alarm hl:mem_free=15401.000000M/300M, alarm hl:available=1/0 [06:45:41] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:47:20] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [06:49:51] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:51:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:26:21] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 369488 MB (6% inode=39%): [07:43:20] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.387695/1.10, alarm hl:np_load_long=0.940430/1.55, alarm hl:mem_free=15287.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.387695/1.00, alarm hl:np_load_long=0.940430/1.50, alarm hl:mem_free=15287.000000M/300M, alarm hl:available=1/0 [07:46:40] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:47:21] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [07:47:51] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.801270/1.8, alarm hl:np_load_avg=1.857422/2.3, alarm hl:mem_free=1117.000000M/300M, alarm hl:available=1/0 [07:48:50] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [07:50:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:51:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:27:20] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 365713 MB (6% inode=38%): [08:45:20] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.148438/1.10, alarm hl:np_load_long=0.875000/1.55, alarm hl:mem_free=15644.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.148438/1.00, alarm hl:np_load_long=0.875000/1.50, alarm hl:mem_free=15644.000000M/300M, alarm hl:available=1/0 [08:46:19] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [08:46:51] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:51:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:52:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:28:20] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 361280 MB (6% inode=38%): [09:47:51] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:51:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:52:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:53:30] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38920 MB (9% inode=99%): [09:56:30] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 58877 MB (14% inode=99%): [10:04:31] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38094 MB (9% inode=99%): [10:12:54] 3(created) [DBQ-178] Statistics about abuse filters on all Wikimedia projects; Database Queries; Task <10https://jira.toolserver.org/browse/DBQ-178> (Federico Leva) [10:20:02] Danny B. * [Toolserver-l] Dumps maintenance this weekend [10:28:20] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 359055 MB (6% inode=38%): [10:43:31] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.142578/1.10, alarm hl:np_load_long=0.771485/1.55, alarm hl:mem_free=14526.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.142578/1.00, alarm hl:np_load_long=0.771485/1.50, alarm hl:mem_free=14526.000000M/300M, alarm hl:available=1/0 [10:45:31] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [10:47:51] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:51:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:53:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:01:29] has something just happened to /mnt/usr-store/dumps? [11:03:35] alkamid: yes, i'm doing the maintenance there. are you experiencing any issues? [11:04:16] Danny_B|backup, I can't access files in store/plwiktionary [11:04:37] alkamid: read or write? [11:04:43] Danny_B|backup, read [11:05:27] alkamid: try now [11:06:03] Danny_B|backup, thanks [11:06:22] sorry for inconvenience [11:13:57] typo... no preview on irc... ;-) [11:28:30] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 417837 MB (7% inode=42%): [11:47:52] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:51:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:53:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:13:33] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1927 [12:14:12] s4 replag on z-dat-s4-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1864.000000 [12:14:32] MySQL slave on z-dat-s4-a is OK: Uptime: 782747 Threads: 7 Questions: 30590780 Slow queries: 12703 Opens: 16466 Flush tables: 1 Open tables: 485 Queries per second avg: 39.81 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1710 [12:15:11] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 1400.000000 [12:28:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 414334 MB (7% inode=41%): [12:29:22] /sql on thyme is WARNING: DISK WARNING - free space: /sql 191196 MB (19% inode=99%): [12:47:52] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:51:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:51:32] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 33108 MB (3% inode=99%): [12:52:04] anybody here from group enwikt? [12:53:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:54:07] Joan: around? [13:26:22] /sql on thyme is OK: DISK OK - free space: /sql 214731 MB (22% inode=99%): [13:28:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 413247 MB (7% inode=41%): [13:32:22] seems willow got restarted yesterday, yet I saw no announcement [13:33:56] Platonides: that was unexpected - /tmp got full [13:34:18] ouch [13:35:18] I had expected, even if unplanned, something like an email saying that [13:35:27] thanks for the info, Danny_B [13:36:34] there is jira entry for that so far i know [13:48:02] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:51:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:54:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:29:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 407092 MB (7% inode=41%): [14:48:02] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:51:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:54:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:18:09] sigh [15:18:26] why so people move tools from expired account A to will-expire-sometime-in-the-future-account B? [15:18:36] s/so/do [15:19:24] Hi valhallasw , because MM projects are scary and you can't be a local hero [15:20:54] ah, yes, the local hero aspect [15:20:56] that explains [15:29:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 400275 MB (7% inode=41%): [15:30:42] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37081 MB (9% inode=99%): [15:32:42] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 58578 MB (14% inode=99%): [15:40:42] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40117 MB (9% inode=99%): [15:43:42] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.930664/1.10, alarm hl:np_load_long=0.993164/1.55, alarm hl:mem_free=14525.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.930664/1.00, alarm hl:np_load_long=0.993164/1.50, alarm hl:mem_free=14525.000000M/300M, alarm hl:available=1/0 [15:46:42] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [15:48:11] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:51:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:54:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:13:02] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.717773/1.8, alarm hl:np_load_avg=0.713379/2.3, alarm hl:mem_free=243.000000M/300M, alarm hl:available=1/0 [16:14:02] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [16:14:52] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.060547/1.00, alarm hl:np_load_long=0.864258/1.50, alarm hl:mem_free=14732.000000M/300M, alarm hl:available=1/0 [16:17:52] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [16:25:53] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=2.284180/1.10, alarm hl:np_load_long=1.086914/1.55, alarm hl:mem_free=14654.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=2.284180/1.00, alarm hl:np_load_long=1.086914/1.50, alarm hl:mem_free=14654.000000M/300M, alarm hl:available=1/0 [16:27:02] Sumana Harihareswara * [Toolserver-l] Submit a talk to Open Source Bridge by Mar 16th, get WMF subsidy [16:29:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 397387 MB (7% inode=40%): [16:33:23] Load avg. on ortelius is WARNING: WARNING - load average: 28.65, 18.15, 10.53 [16:34:22] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 30.52, 20.57, 11.87 [16:35:24] Load avg. on ortelius is WARNING: WARNING - load average: 22.86, 20.35, 12.36 [16:42:20] Danny_B|backup: I'm around. But I'm not from the enwikt group. [16:43:03] [[User:Dab/Debian-Packages]] 10https://wiki.toolserver.org/w/index.php?diff=6829&oldid=6827&rcid=9001 * Gifti * (+139) (my tcl dependencies) [16:43:05] Danny_B|backup: You're doing maintenance on user-store? Are you root now? [16:49:02] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:51:02] Sumana Harihareswara * Re: [Toolserver-l] Submit a talk to Open Source Bridge by Mar 16th, get WMF subsidy [16:51:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:54:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:55:29] ##@ [16:55:34] oops [17:11:52] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [17:14:52] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.077149/1.00, alarm hl:np_load_long=1.486328/1.50, alarm hl:mem_free=14664.000000M/300M, alarm hl:available=1/0 [17:18:52] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [17:30:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 442666 MB (8% inode=43%): [17:34:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37574 MB (9% inode=99%): [17:38:42] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 54961 MB (13% inode=99%): [17:44:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36700 MB (9% inode=99%): [17:49:11] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:51:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:52:22] Joan: nope, i'm not root [17:52:38] Danny_B|backup: What maintenance are you doing? [17:53:23] just systematic order [17:54:23] Moving files? [17:54:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:57:30] Joan: mostly [17:57:42] Joan: i've sent a notice to list [17:57:53] Hmmm, did you leave symlinks? [17:58:11] It's not as though nobody knew user-store was a mess. But breaking scripts might get you shot. [18:00:03] i'm leaving them, but they will eventually be deleted [18:00:27] but that will be noticed ahead, as i wrote in the post in list [18:01:04] i actually saw you have some files out of system, that's why i pinged you [18:07:31] [[Job scheduling]] ! 10https://wiki.toolserver.org/w/index.php?diff=6830&oldid=6828&rcid=9002 * Nemobis * (+133) (/* obligatory resources */ clarify, per merlissimo) [18:11:11] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.022949/1.8, alarm hl:np_load_avg=0.773438/2.3, alarm hl:mem_free=203.000000M/300M, alarm hl:available=1/0 [18:13:12] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [18:22:52] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.015625/1.00, alarm hl:np_load_long=0.944336/1.50, alarm hl:mem_free=14849.000000M/300M, alarm hl:available=1/0 [18:24:52] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [18:31:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 441370 MB (8% inode=43%): [18:45:52] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.036133/1.00, alarm hl:np_load_long=0.989258/1.50, alarm hl:mem_free=14900.000000M/300M, alarm hl:available=1/0 [18:46:51] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [18:49:12] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:51:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:55:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:11:52] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.108399/1.10, alarm hl:np_load_long=1.038086/1.55, alarm hl:mem_free=14757.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.108399/1.00, alarm hl:np_load_long=1.038086/1.50, alarm hl:mem_free=14757.000000M/300M, alarm hl:available=1/0 [19:13:11] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.868164/1.8, alarm hl:np_load_avg=0.831543/2.3, alarm hl:mem_free=268.000000M/300M, alarm hl:available=1/0 [19:13:52] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [19:14:12] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [19:31:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 449136 MB (8% inode=43%): [19:33:53] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=2.980469/1.10, alarm hl:np_load_long=1.183594/1.55, alarm hl:mem_free=14546.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=2.980469/1.00, alarm hl:np_load_long=1.183594/1.50, alarm hl:mem_free=14546.000000M/300M, alarm hl:available=1/0 [19:36:53] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [19:37:40] Danny_B|backup: I think several of the symbolic links you made under "dump" are incorrect, as they point to "dump" and not "dumps" [19:38:25] (and I suppose only "dumps" will stay after all) [19:40:14] jem-: there are some links to links to files... [19:40:19] not yet finished [19:42:07] Ok then [19:43:52] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.382812/1.10, alarm hl:np_load_long=1.170899/1.55, alarm hl:mem_free=14586.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.382812/1.00, alarm hl:np_load_long=1.170899/1.50, alarm hl:mem_free=14586.000000M/300M, alarm hl:available=1/0 [19:44:24] huh, what happened, there are some files i deleted and they reappeared ? [19:49:12] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:52:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:53:42] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41020 MB (10% inode=99%): [19:54:43] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 72611 MB (17% inode=99%): [19:55:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:57:58] nosy: around? [19:58:12] Danny_B|backup: yes [20:00:53] 3(resolved) [UTRS-64] System/Tool notice <10https://jira.toolserver.org/browse/UTRS-64> (Hersfold) [20:09:11] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.822754/1.8, alarm hl:np_load_avg=0.881348/2.3, alarm hl:mem_free=294.000000M/300M, alarm hl:available=1/0 [20:11:12] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [20:31:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 447629 MB (8% inode=43%): [20:45:51] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1818.000000 [20:46:52] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1755.000000 [20:49:12] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:53:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:56:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:05:42] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38255 MB (9% inode=99%): [21:10:42] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 59934 MB (14% inode=99%): [21:13:12] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.067383/1.8, alarm hl:np_load_avg=1.015625/2.3, alarm hl:mem_free=229.000000M/300M, alarm hl:available=1/0 [21:14:12] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [21:16:42] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37558 MB (9% inode=99%): [21:17:03] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=0.773438/1.10, alarm hl:np_load_long=1.604492/1.55, alarm hl:mem_free=14970.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=0.773438/1.00, alarm hl:np_load_long=1.604492/1.50, alarm hl:mem_free=14970.000000M/300M, alarm hl:available=1/0 [21:20:02] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [21:23:01] Joan: do you need the curltest.bz2 in user-store? [21:23:03] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1851 [21:24:03] MySQL slave on z-dat-s6-a is OK: Uptime: 815712 Threads: 9 Questions: 125047898 Slow queries: 55829 Opens: 1105398 Flush tables: 2 Open tables: 1834 Queries per second avg: 153.299 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1799 [21:28:03] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1978 [21:31:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 441022 MB (8% inode=43%): [21:43:12] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.565918/1.8, alarm hl:np_load_avg=0.663086/2.3, alarm hl:mem_free=220.000000M/300M, alarm hl:available=1/0 [21:49:12] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:53:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:56:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:59:12] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [22:03:13] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.803711/1.8, alarm hl:np_load_avg=0.753906/2.3, alarm hl:mem_free=223.000000M/300M, alarm hl:available=1/0 [22:07:33] Danny_B|backup: I dunno. Probably not. What is it? [22:07:44] And where is it? [22:14:05] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1929.000000 [22:14:05] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1930.000000 [22:19:22] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1904.000000 [22:21:12] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1687.000000 [22:22:22] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1983 [22:23:05] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [22:23:05] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [22:24:04] MySQL slave on z-dat-s6-a is OK: Uptime: 819313 Threads: 10 Questions: 125324809 Slow queries: 55966 Opens: 1108724 Flush tables: 2 Open tables: 1837 Queries per second avg: 152.963 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1716 [22:24:04] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [22:24:04] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1733.000000 [22:24:21] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [22:24:22] MySQL slave on rosemary is OK: Uptime: 2018068 Threads: 10 Questions: 529803321 Slow queries: 190933 Opens: 2354 Flush tables: 1 Open tables: 828 Queries per second avg: 262.529 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1662 [22:25:04] MySQL slave on thyme is OK: Uptime: 519404 Threads: 8 Questions: 155475996 Slow queries: 48690 Opens: 29612 Flush tables: 1 Open tables: 2557 Queries per second avg: 299.335 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1795 [22:25:04] MySQL slave on z-dat-s3-a is OK: Uptime: 910488 Threads: 15 Questions: 1168987950 Slow queries: 55380 Opens: 7820581 Flush tables: 1 Open tables: 16384 Queries per second avg: 1283.913 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1563 [22:25:04] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1785.000000 [22:25:22] MySQL slave on cassia is OK: Uptime: 913544 Threads: 19 Questions: 1184780323 Slow queries: 115554 Opens: 8233388 Flush tables: 2 Open tables: 12728 Queries per second avg: 1296.905 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [22:32:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 440866 MB (8% inode=43%): [22:41:03] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.023438/1.00, alarm hl:np_load_long=0.869140/1.50, alarm hl:mem_free=14860.000000M/300M, alarm hl:available=1/0 [22:42:03] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [22:44:52] Joan: /mnt/user-store [22:45:03] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.219726/1.10, alarm hl:np_load_long=0.993164/1.55, alarm hl:mem_free=14873.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.219726/1.00, alarm hl:np_load_long=0.993164/1.50, alarm hl:mem_free=14873.000000M/300M, alarm hl:available=1/0 [22:45:55] Danny_B|backup: Gone. [22:46:55] 3(created) [ACCAPP-470] Urgent: Account for Research on Controversy; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-470> (Shiri Dori-Hacohen) [22:48:12] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.900879/1.8, alarm hl:np_load_avg=1.015137/2.3, alarm hl:mem_free=241.000000M/300M, alarm hl:available=1/0 [22:49:13] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:50:12] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [22:50:46] Joan: awesome, thx [22:51:48] No problem. Thanks for cleaning up user-store. :-) [22:51:52] Hopefully it isn't too disruptive. [22:54:04] [[CVN API]] 10https://wiki.toolserver.org/w/index.php?diff=6831&oldid=5405&rcid=9003 * Krinkle * (+1397) () [22:54:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:57:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:58:12] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.693359/1.8, alarm hl:np_load_avg=0.783203/2.3, alarm hl:mem_free=188.000000M/300M, alarm hl:available=1/0: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.693359/1.9, alarm hl:np_load_long=0.888672/2.25, alarm hl:mem_free=188.000000M/200M, alarm hl:available=1/0 [23:12:53] i cannot write to directories in /mnt/user-store/dumps/store, is this intentional?! [23:14:06] Danny_B|backup? [23:14:45] What is store/? [23:15:24] as it seems the new dumps location, old directories symlink to there [23:15:41] Yes. Why? [23:15:55] They were orgnaized by database name inside dumps/. [23:16:04] must be the maintenance ;) [23:16:05] It needed another level? :-/ [23:16:35] Joan: you won't need the exact path in future [23:16:54] because all dumps will be downloaded automatically [23:17:00] great :) [23:17:09] I might not need the exact path, but my scripts will. [23:17:12] thekla: do you need to add another wiki? [23:17:38] Are you building some kind of API? [23:17:45] no, i want to download the newest pages-articles for dewiki [23:17:55] Where you can just say "current dump of enwiki pages" and get the exact file you want? [23:18:31] Joan: now there are symlinks and there will be notice to users to change their scripts and after some period of time all symlinks will be deleted and there will be only new clean system kept [23:18:37] Joan: yes, i do [23:18:59] thekla: i'll download it [23:19:03] Danny_B|backup: I was asking why you chose /mnt/user-store/dumps/store/enwiki instead of /mnt/user-store/dumps/enwiki [23:19:20] Are you going to put scripts inside of dumps/ or something? [23:19:38] Joan: yes exactly : 'you can just say "current dump of enwiki pages" and get the exact file you want' [23:19:46] Okay. :-) [23:19:54] I'd like that. [23:20:14] I've thought about doing something similar with symlinks and a -current.xml.bz2 synlink. [23:20:18] well currently there are symlinks in dumps/ for all latest dumps - check it [23:20:25] I saw the symlinks. [23:20:32] I'm asking what's going to go next to store? [23:20:39] thank you, i will temporarily need {page,redirect}.sql as well [23:21:39] Danny_B|backup: Could the pagecounts stats be added to that automatic download? [23:22:17] Joan: the deal is, that users should access dumps via symlinks which will guarantee them the last version without necessity to download it [23:22:58] jem-: in step 3, first i need to deal with wiki dumps [23:23:55] Ok :) That would be great, because schutz isn't updating them since Feb 14 [23:24:42] But there's no hurry by my side [23:26:36] Knowing that, I just won't complain again in the list about that or ask for a MMP [23:32:13] Joan jem- thekla do you guys have any dumps stored in your ~ ? [23:32:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 438981 MB (8% inode=43%): [23:33:03] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.118164/1.10, alarm hl:np_load_long=0.869140/1.55, alarm hl:mem_free=14615.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.118164/1.00, alarm hl:np_load_long=0.869140/1.50, alarm hl:mem_free=14615.000000M/300M, alarm hl:available=1/0 [23:34:01] I don't think I do. [23:35:05] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [23:37:38] Danny_B|backup: No, I don't [23:38:22] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.965332/1.8, alarm hl:np_load_avg=0.964844/2.3, alarm hl:mem_free=263.000000M/300M, alarm hl:available=1/0 [23:39:23] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [23:40:54] 3(created) [DRTRIGON-120] New bot script/part for opencv image recognition and classification; DrTrigon's tools: DrTrigonBot - pywikipedia; New Feature <10https://jira.toolserver.org/browse/DRTRIGON-120> (drtrigon) [23:43:03] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36016 MB (8% inode=99%): [23:44:53] 3(updated) [DRTRIGON-120] New bot script/part for opencv image recognition and classification <10https://jira.toolserver.org/browse/DRTRIGON-120> (drtrigon) [23:45:03] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.070312/1.00, alarm hl:np_load_long=0.880860/1.50, alarm hl:mem_free=14768.000000M/300M, alarm hl:available=1/0 [23:46:03] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 132371 MB (33% inode=99%): [23:47:23] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.809570/1.8, alarm hl:np_load_avg=0.943848/2.3, alarm hl:mem_free=181.000000M/300M, alarm hl:available=1/0: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.809570/1.9, alarm hl:np_load_long=0.993652/2.25, alarm hl:mem_free=181.000000M/200M, alarm hl:available=1/0 [23:49:24] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:51:46] Danny_B|backup: no … [23:54:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:57:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default