[00:16:33] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 99532 MB (10% inode=99%): [00:31:32] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41291 MB (10% inode=99%): [00:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:35:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1079278.000000 [00:35:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1079281 [00:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:35:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [00:41:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:57:33] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24188 MB (5% inode=99%): [01:33:33] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 44821 MB (11% inode=99%): [01:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:35:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1080212 [01:35:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1080213.000000 [01:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:40:32] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 44409 MB (10% inode=99%): [01:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [01:41:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:42:33] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 45486 MB (11% inode=99%): [01:51:02] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:51:03] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:54:33] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 96018 MB (9% inode=98%): [01:54:33] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 96018 MB (9% inode=98%): [02:09:33] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 44397 MB (10% inode=99%): [02:15:32] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 45580 MB (11% inode=99%): [02:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:35:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1081011 [02:35:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1081016.000000 [02:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [02:41:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:54:03] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:54:03] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:56:33] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 95285 MB (9% inode=98%): [02:56:33] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 95280 MB (9% inode=98%): [03:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1081983 [03:35:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1081986.000000 [03:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [03:41:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1083377 [04:35:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1083379.000000 [04:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [04:41:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:50:53] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:55:03] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:55:03] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:56:33] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 93489 MB (9% inode=98%): [04:56:33] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 93489 MB (9% inode=98%): [05:06:02] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:06:03] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:25:52] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1084813 [05:35:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1084814.000000 [05:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:37:33] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43671 MB (10% inode=99%): [05:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [05:41:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:45:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 33983 MB (2% inode=98%): [05:53:33] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 94233 MB (9% inode=98%): [05:53:33] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 94233 MB (9% inode=98%): [06:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:35:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1086149 [06:35:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1086154.000000 [06:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:40:33] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24457 MB (5% inode=99%): [06:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [06:41:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1087035 [07:35:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1087037.000000 [07:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:40:52] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [07:41:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:55:03] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:56:33] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 93855 MB (9% inode=98%): [08:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1088118 [08:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:35:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1088124.000000 [08:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [08:41:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:35:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1089691 [09:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:35:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1089696.000000 [09:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [09:41:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:21:33] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 46142 MB (11% inode=99%): [10:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1091210 [10:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:35:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1091215.000000 [10:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:40:52] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [10:41:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:04:32] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 44335 MB (10% inode=99%): [11:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:35:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1093199 [11:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:35:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1093204.000000 [11:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [11:41:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:16:33] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 99050 MB (10% inode=99%): [12:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1094678 [12:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:35:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1094681.000000 [12:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:37:33] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 46042 MB (11% inode=99%): [12:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [12:41:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1095762 [13:35:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:35:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1095764.000000 [13:40:52] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [13:41:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1097035 [14:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:36:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1097044.000000 [14:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [14:41:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1098029 [15:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:36:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1098038.000000 [15:40:52] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [15:41:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1099022 [16:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:36:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1099030.000000 [16:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [16:41:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1100168 [17:35:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:36:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1100178.000000 [17:40:52] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [17:41:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:45:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 38270 MB (3% inode=98%): [17:53:42] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 95436 MB (9% inode=98%): [18:04:20] do fcgi processes get killed after a certain period? [19:27:50] valhallasw: In theory yes, but if your doing code refreshes you'll need to kill them manually [19:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1103092 [19:35:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:35:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:36:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1103110.000000 [19:40:52] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [19:41:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:56:32] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 95188 MB (9% inode=98%): [20:18:32] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41397 MB (10% inode=99%): [20:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1104553 [20:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:36:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1104572.000000 [20:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [20:41:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:45:32] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23925 MB (5% inode=99%): [20:46:33] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 26881 MB (6% inode=99%): [20:50:32] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24410 MB (5% inode=99%): [21:14:32] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 45090 MB (11% inode=99%): [21:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:35:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1106155 [21:35:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:35:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:36:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1106171.000000 [21:40:52] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [21:41:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:08:41] Dispenser: yes, I'm aware of the latter. Rather, I'm trying to understand why a process (or rather a spawned worker thread) does not finish, and doesn't emit an exception [22:09:31] multithreading in python? [22:11:10] Dispenser: yes [22:11:34] Dispenser: tsreports with a worker thread that runs the query instead of blocking the http response [22:12:11] it seems to work overall, but sometimes the query returns (or at least does not show up in SHOW PROCESSLIST anymore), but it does not get stored in the cache table & does not throw an exception [22:12:16] Do tell me if you manage to figure it out [22:13:07] Dispenser: multithreading in python in general or in fcgi specifically? [22:13:30] multithreading [22:14:12] https://github.com/valhallasw/tsreports/commit/54721ae861a8ec4830dbbfbd842f2d6bd0e0674d lines 108-113 [22:14:32] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41453 MB (10% inode=99%): [22:14:32] Is tsreports getting some love? [22:14:41] Brooke: yes, I'm working on it at the moment [22:14:44] It'd be awfully nice to have. :-) [22:14:51] This is River's old project, right? [22:14:54] yep [22:15:09] Mmmm. It needs pagination and page history, as I recal. [22:15:11] recall [22:15:12] I updated the caching; now it always shows a cached version and updates on the background [22:15:20] Those were the blockers. [22:15:30] page history should be relatively easy to implement; pagination does not add a lot, I think [22:15:35] And there wasn't a way to manipulate individual rows. [22:15:57] You favor truncating the output? [22:16:06] pagination is only useful for 'simple' queries that do not have to be cached [22:16:27] Pagination is useful for not crashing your browser. ;-) [22:16:28] and in the case of WM wiki's, there are not a lot of queries for which that is true [22:16:33] true :-) [22:16:50] There are plenty of database reports with a lot of data. [22:16:57] Brooke: feel free to clone the git repos and play with it [22:17:03] & to submit a pull request ;-) [22:17:25] I was hoping it would replace one day. [22:18:09] Brooke: that should be trivial [22:18:22] Which part? [22:18:29] except there is no support for 'run at the first of every month' at the moment [22:18:34] to replace those queries [22:18:37] There are a lot of reports. [22:18:48] Some of them I guess you'd want to keep on-wiki. [22:18:51] For the historical value. [22:19:10] oh, some of them are not pure SQL [22:19:16] those are more difficult, I guess [22:19:16] Most are. [22:19:24] Some are using their own special tracking table. [22:19:36] Or tracking file. [22:19:46] right [22:20:30] I actually have notes on a better database reports sytsem somewhere. [22:20:46] for these queries, is it important they are run on the first of every month specifically [22:20:57] or it just updating them roughly once a month? [22:21:12] Found it. [22:21:15] https://en.wikipedia.org/wiki/Wikipedia:Database_reports/Redesign [22:21:36] That was mostly with the idea of fixing up the current code, though. [22:21:39] Not switching systems. [22:21:49] valhallasw: Oh, no, just monthly. [22:21:55] Most aren't the first of the month. [22:22:01] I try to stagger them. [22:22:05] Brooke: right [22:22:15] https://en.wikipedia.org/wiki/Wikipedia:Database_reports/Configuration has the full crontab. [22:22:16] I think most of those queries can just be converted to a .report format [22:22:23] That'd be nice. :-) [22:22:26] and we can just plug it into ~reports [22:22:33] Supporting other wikis besides en.wiki would be cool. [22:22:38] I do some reports at Commons now. [22:22:41] That would be automagic ;-) [22:22:48] Is ~reports working? [22:22:50] yes [22:22:50] I thought it was down. [22:22:54] Oh, nice. [22:23:04] with improved caching [22:23:22] I just have to make sure queries get /* SLOW OK */ though :-) [22:23:22] So the results can be linked to? [22:23:44] yes [22:23:46] That was the other issue we hit: for rpeorts that update, there's no indication in the watchlist (for example) that hte report has updated. [22:23:57] reports * [22:24:03] http://wolfsbane.toolserver.org/~valhallasw/reports/?wiki=nlwiktionary_p&report=brokentemplates [22:24:04] I can't type today. [22:24:31] refresh it in ~60s or so [22:24:42] in my version, it has a cache at 60 seconds [22:25:01] http://wolfsbane.toolserver.org/~valhallasw/reports/?wiki=enwiki_p&report=mostrevisions [22:25:03] Heh. [22:25:23] I'll probably get an e-mail about the query being killed on that one ;-) [22:27:04] but before fiddling with that, I first want to make sure the entire system actually works well enough [22:27:17] without queries running but not getting stored, and stuff like that [22:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1107689 [22:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:36:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1107712.000000 [22:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [22:41:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:43:32] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24093 MB (5% inode=99%): [23:01:43] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 45227 MB (11% inode=99%): [23:08:33] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 35620 MB (8% inode=99%): [23:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1109499 [23:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:36:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1109519.000000 [23:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [23:41:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:49:33] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24376 MB (5% inode=99%):