[00:02:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:15:01] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 25979 MB (6% inode=99%): [00:16:01] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24031 MB (5% inode=99%): [00:21:32] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [00:28:01] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 60045 MB (14% inode=99%): [00:34:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 633181 [00:34:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 63609 [00:38:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:39:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 453436 MB (8% inode=42%): [00:39:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:43:42] [[Special:Log/newusers]] create 10 * KarineLqn * (New user account) [00:46:02] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40689 MB (10% inode=99%): [00:46:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:48:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63754.000000 [00:55:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 633878.000000 [01:02:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:03:01] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24062 MB (5% inode=99%): [01:06:42] Load avg. on willow is WARNING: WARNING - load average: 16.38, 15.89, 14.46 [01:13:42] Load avg. on willow is OK: OK - load average: 12.49, 14.96, 14.65 [01:21:32] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [01:29:01] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 29270 MB (7% inode=99%): [01:34:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 635514 [01:34:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 63242 [01:36:02] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24283 MB (5% inode=99%): [01:38:15] [[Special:Log/newusers]] create 10 * AlanYdc * (New user account) [01:38:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:39:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 453352 MB (8% inode=42%): [01:39:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:46:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:47:01] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 52805 MB (13% inode=99%): [01:48:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63053.000000 [01:56:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 636296.000000 [02:02:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:20:42] /sql on thyme is WARNING: DISK WARNING - free space: /sql 121904 MB (12% inode=99%): [02:21:32] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [02:34:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 637610 [02:34:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61951 [02:38:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:39:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 451396 MB (8% inode=42%): [02:39:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:46:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:48:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61854.000000 [02:57:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 638375.000000 [03:02:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:22:32] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [03:34:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 639515 [03:34:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61330 [03:38:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:39:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 450852 MB (8% inode=42%): [03:39:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:46:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:48:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61516.000000 [03:57:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 640369.000000 [04:02:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:22:43] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [04:34:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 641300 [04:34:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 60869 [04:38:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:39:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 451297 MB (8% inode=42%): [04:39:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:46:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:48:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60897.000000 [04:58:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 642101.000000 [05:00:27] @replag [05:00:27] Earwig: s1-rr-a: 16h 54m 40s [-0.07 s/s]; s1-user: 1w 10h 22m 51s [+0.45 s/s]; s3-rr-a: 15s [-0.00 s/s]; s3-user: 15s [-0.00 s/s] [05:02:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:22:43] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [05:29:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 43829 MB (3% inode=98%): [05:34:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 643305 [05:35:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 60314 [05:39:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 451213 MB (8% inode=42%): [05:39:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:39:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:46:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:47:42] Load avg. on willow is WARNING: WARNING - load average: 13.61, 15.40, 14.28 [05:48:42] Load avg. on willow is OK: OK - load average: 11.76, 14.58, 14.06 [05:49:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60263.000000 [05:58:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 644186.000000 [06:03:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:23:43] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [06:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 645011 [06:35:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 59730 [06:39:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 451018 MB (8% inode=42%): [06:39:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:39:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:41:42] Load avg. on willow is WARNING: WARNING - load average: 25.97, 19.27, 15.49 [06:46:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:47:42] Load avg. on willow is OK: OK - load average: 9.99, 14.58, 14.91 [06:49:12] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 59521.000000 [06:58:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 645507.000000 [07:03:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:23:43] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [07:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 646794 [07:35:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 58362 [07:39:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 450950 MB (8% inode=42%): [07:39:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:39:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:46:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:49:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58172.000000 [07:58:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 647552.000000 [08:03:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:24:43] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [08:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 648625 [08:36:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 56756 [08:39:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 450845 MB (8% inode=42%): [08:39:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:39:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:43:02] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40596 MB (9% inode=99%): [08:47:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:49:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56690.000000 [08:58:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 649219.000000 [09:03:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:08:02] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24026 MB (5% inode=99%): [09:17:01] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 51027 MB (12% inode=99%): [09:25:43] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [09:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 650357 [09:36:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 55218 [09:39:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 450759 MB (8% inode=42%): [09:39:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:39:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:47:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:50:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55172.000000 [09:57:11] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:57:12] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:57:42] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 99635 MB (10% inode=98%): [09:57:42] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 99635 MB (10% inode=98%): [09:58:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 650969.000000 [10:03:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:09:11] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:11] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:26:42] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [10:28:42] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 99256 MB (10% inode=98%): [10:28:42] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 99242 MB (10% inode=98%): [10:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 652123 [10:36:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54192 [10:39:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 450661 MB (8% inode=41%): [10:39:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:40:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:47:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:50:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54161.000000 [10:56:53] @replag [10:56:53] JohannesK_wmde1: s1-rr-a: 15h 1m 5s [-0.32 s/s]; s1-user: 1w 13h 18m 25s [+0.49 s/s]; s3-rr-a: 30s [+0.00 s/s]; s3-user: 30s [+0.00 s/s] [10:58:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 652750.000000 [11:04:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:14:01] @replag [11:14:01] Dispenser: s1-rr-a: 14h 54m 44s [-0.37 s/s]; s1-user: 1w 13h 28m 50s [+0.61 s/s] [11:26:43] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [11:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 654342 [11:36:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 52983 [11:39:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 450453 MB (8% inode=41%): [11:39:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:40:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:47:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:50:21] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52864.000000 [11:53:47] https://simple.wikipedia.org/wiki/Special:Statistics says: [11:54:00] Page edits since Wikipedia was set up 4,398,065 [11:54:17] why SELECT COUNT(rev_timestamp) AS revisions FROM revision ; [11:54:37] gives me 3467170 [11:55:18] is there a latency in toolserver update? [11:58:30] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 655268.000000 [12:01:12] ok thanks [12:04:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:27:42] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [12:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 656440 [12:36:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 51911 [12:39:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 450350 MB (8% inode=41%): [12:39:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:40:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:47:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:50:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51838.000000 [12:58:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 657075.000000 [13:03:47] Anyone here who can help me with PuTTY? [13:04:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:05:28] [[PuTTY]] N 10https://wiki.toolserver.org/w/index.php?oldid=7463&rcid=10162 * Dispenser * (+46) (Redirected page to [[Logging in#Logging in with PuTTY]]) [13:06:12] Frigotoni|away: It should be all documented at https://wiki.toolserver.org/view/Logging_in [13:28:42] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [13:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 658072 [13:37:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 51104 [13:39:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 449937 MB (8% inode=41%): [13:39:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:40:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:47:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:50:30] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51045.000000 [13:58:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 658689.000000 [14:04:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:20:42] /sql on thyme is WARNING: DISK WARNING - free space: /sql 120632 MB (12% inode=99%): [14:28:43] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [14:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 659643 [14:37:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50664 [14:38:01] [[Special:Log/newusers]] create 10 * ReggieMH * (New user account) [14:40:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 449660 MB (8% inode=41%): [14:40:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:40:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:47:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:50:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50609.000000 [14:58:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 660381.000000 [15:04:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:28:10] what's the connection bandwidth of the toolserver? [15:29:42] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [15:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 661442 [15:37:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 49882 [15:40:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 479307 MB (9% inode=43%): [15:40:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:40:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:47:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:50:30] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49851.000000 [15:58:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 662057.000000 [16:04:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:29:42] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [16:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 663128 [16:38:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48674 [16:40:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 479188 MB (9% inode=43%): [16:40:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:40:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:47:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:50:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48520.000000 [16:58:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 663729.000000 [17:04:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:30:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 45202 MB (3% inode=98%): [17:30:43] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [17:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 664808 [17:38:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48338 [17:40:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:41:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 479120 MB (9% inode=43%): [17:41:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:47:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:50:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48294.000000 [17:58:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 665728.000000 [18:04:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:06:51] @replag [18:06:52] russblau: s1-rr-a: 13h 23m 0s [-0.22 s/s]; s1-user: 1w 17h 33s [+0.51 s/s]; s2-user: 12s [-0.00 s/s]; s3-rr-a: 11s [-0.00 s/s]; s3-user: 11s [-0.00 s/s] [18:31:43] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [18:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 667123 [18:38:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 47339 [18:40:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:41:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 478242 MB (9% inode=43%): [18:41:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:47:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:51:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47325.000000 [18:58:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 667962.000000 [19:04:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:31:43] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [19:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 669273 [19:38:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 45891 [19:40:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:41:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 479913 MB (9% inode=43%): [19:41:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:47:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:52:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45657.000000 [19:58:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 670198.000000 [20:05:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:31:42] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [20:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 671630 [20:39:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43069 [20:40:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:42:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 479793 MB (9% inode=43%): [20:42:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:48:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:52:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42511.000000 [20:58:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 672455.000000 [21:05:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:15:02] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40550 MB (9% inode=99%): [21:31:42] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [21:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 673787 [21:40:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:40:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39131 [21:41:02] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24141 MB (5% inode=99%): [21:43:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 479720 MB (9% inode=43%): [21:43:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:45:04] roots here? [21:45:46] liangent: what do you need? [21:46:15] Platonides: Probably a better mailing list question. [21:46:31] Betacommand: file recovery. I sent a request to ts-admins@ some days ago but no response [21:47:03] liangent: drop a note on the mailing list and file a jira ticket [21:47:53] are they handled more frequently than ts-admins@ ? [21:48:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:48:13] liangent: if you assign it to DaB it emails him [21:48:34] this way you are triple tagging the issue [21:49:17] Merlissimo: that is a bit heavy imho [21:49:19] ok but I don't want to use toolserver-l, or it's nagging everyone who're unable to do anything [21:49:21] oops [21:49:54] liangent: however it may catch the attention of one of the other TS roots [21:50:24] giftpflanze: ? [21:50:43] ich wollte dich nicht anpingen, sorry [21:52:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38367.000000 [21:55:01] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 52454 MB (12% inode=99%): [21:58:32] Betacommand: jira tickets filed, I'll wait for some more days before turning to toolserver-l [21:58:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 674633.000000 [21:59:04] liangent: you might not want to wait too long depending on how long they keep backups [21:59:05] It might be deleted by then [22:05:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:07:25] I sent one then [22:09:06] Liangent * [Toolserver-l] Help from roots for file undeletion wanted [22:19:13] @replag [22:19:14] Brooke: s1-rr-a: 10h 10m 29s [-0.76 s/s]; s1-user: 1w 19h 36m 26s [+0.62 s/s] [22:20:01] Load avg. on adenia is WARNING: WARNING - load average: 18.23, 15.98, 9.68 [22:29:42] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 99272 MB (10% inode=98%): [22:29:42] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 99265 MB (10% inode=98%): [22:31:02] Load avg. on adenia is OK: OK - load average: 12.78, 14.80, 12.79 [22:31:42] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [22:32:48] liangent: ts-roots do not have much time atm, so this may take a while [22:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 675969 [22:38:02] Load avg. on adenia is WARNING: WARNING - load average: 15.52, 15.92, 14.17 [22:40:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35330 [22:41:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:43:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 479623 MB (9% inode=43%): [22:43:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:48:02] Load avg. on adenia is OK: OK - load average: 14.32, 14.88, 14.55 [22:48:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:52:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35043.000000 [22:52:47] liangent, what did you store there? [22:52:55] seems to be placed on a git checkout [22:53:02] but that extensions/Toolserver is a bit odd [22:58:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 676820.000000 [23:01:34] Platonides: ~/code/gerrit/mediawiki/core/ is a git checkout, and Toolserver is an extension to allow it run on ts-replicated db [23:02:16] so I can run some maintenance scripts there, and some of my bots are running in this way [23:04:58] I developed that extension just on Toolserver, because it requires ts environment to work, and I don't have a local backup [23:05:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:06:22] why did it go away? [23:07:28] with `rm -Rf` and my mistake [23:08:25] oh, ok [23:09:13] and TS's quota, which required me to cleanup my files [23:09:34] ooh [23:32:42] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:UP: 1 int NOK : CRITICAL [23:35:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 678023 [23:40:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32753 [23:41:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:43:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 478212 MB (9% inode=43%): [23:43:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:48:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:52:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32377.000000 [23:58:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 678807.000000