[00:01:01] Danny_B|backup: the interesting question is: why is wikidata replag (also on s3) on all other servers low [00:01:05] Free Memory on turnera is WARNING: WARNING - 5.5% (460976 kB) free! [00:04:16] @replag [00:04:18] Danny_B|backup: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 1m 4s [-0.01 s/s]; s1-user-wd: 45s [-0.01 s/s]; s2-user: 16s [+0.00 s/s]; s2-user-wd: 45s [-0.01 s/s]; s3-rr-a: 1d 1h 20m 42s [+1.00 s/s]; s3-user: 1d 1h 20m 42s [+1.00 s/s] [00:04:18] Danny_B|backup: s4-user-wd: 46s [-0.01 s/s]; s5-user-wd: 46s [-0.01 s/s] [00:05:05] Free Memory on turnera is OK: OK - 7.1% (598308 kB) free. [00:27:43] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:29:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:31:35] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:34:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [00:35:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:27:45] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:29:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:31:35] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:35:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:35:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [02:27:45] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:29:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:31:34] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:35:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [02:36:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:28:43] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:30:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:32:33] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:36:33] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [03:37:05] Free Memory on turnera is WARNING: WARNING - 6.1% (508864 kB) free! [03:37:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:45:04] Free Memory on turnera is OK: OK - 7.3% (607788 kB) free. [04:28:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:30:10] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:32:39] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:37:20] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:37:29] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [05:29:01] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:30:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:32:40] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:37:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:37:31] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [06:29:11] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:30:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:32:40] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:37:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:37:30] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [07:20:11] Free Memory on turnera is WARNING: WARNING - 5.4% (449180 kB) free! [07:22:12] Free Memory on turnera is CRITICAL: CRITICAL - 4.7% (395712 kB) free! [07:24:12] Free Memory on turnera is WARNING: WARNING - 5.4% (450696 kB) free! [07:30:11] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:30:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:32:40] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:37:29] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [07:38:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:58:11] Free Memory on turnera is CRITICAL: CRITICAL - 3.4% (282228 kB) free! [07:59:12] Free Memory on turnera is OK: OK - 16.7% (1399828 kB) free. [08:30:11] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:30:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:32:40] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:37:41] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [08:38:20] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:18:00] aliasd on yarrow is CRITICAL: Connection refused [09:30:11] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:32:41] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:38:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:38:41] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [10:18:01] aliasd on yarrow is CRITICAL: Connection refused [10:31:11] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:31:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:32:40] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:38:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:38:41] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [11:19:00] aliasd on yarrow is CRITICAL: Connection refused [11:31:11] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:31:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:32:41] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:38:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:38:41] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [11:53:04] Danny B. * [Toolserver-l] S3 replication halted? [12:19:01] aliasd on yarrow is CRITICAL: Connection refused [12:31:11] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:31:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:32:41] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:38:20] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:38:41] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [13:19:01] aliasd on yarrow is CRITICAL: Connection refused [13:32:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:32:11] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:32:41] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:34:56] hello all [13:37:56] @replag [13:37:57] DaBPunkt: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 1m 52s [+0.00 s/s]; s1-user-wd: 30s [-0.00 s/s]; s2-user-wd: 30s [-0.00 s/s]; s3-rr-a: 1d 14h 54m 22s [+1.00 s/s]; s3-user: 1d 14h 54m 22s [+1.00 s/s]; s4-user-wd: 35s [-0.00 s/s] [13:37:57] DaBPunkt: s5-user-wd: 30s [-0.00 s/s]; s7-rr-a: 16s [+0.00 s/s]; s7-user: 16s [+0.00 s/s] [13:38:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:38:41] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [14:19:00] aliasd on yarrow is CRITICAL: Connection refused [14:22:21] replication stopped on s3 ? [14:24:51] @replag [14:24:51] DaBPunkt: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 9m 38s [+0.17 s/s]; s1-user-wd: 22s [-0.01 s/s]; s2-user: 19s [+0.00 s/s]; s2-user-wd: 23s [-0.01 s/s]; s3-rr-a: 1d 15h 39m 24s [+0.95 s/s]; s3-user: 1d 15h 39m 24s [+0.95 s/s] [14:24:52] DaBPunkt: s4-user-wd: 22s [-0.01 s/s]; s5-user-wd: 23s [-0.01 s/s] [14:26:04] DaB. * Re: [Toolserver-l] S3 replication halted? [14:27:04] DaB. * Re: [Toolserver-l] S3 replication halted? [14:32:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:32:12] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:32:49] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:38:20] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:39:41] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [15:19:00] aliasd on yarrow is CRITICAL: Connection refused [15:32:11] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:32:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:33:41] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:38:18] DaBPunkt: Hi. Would it be possible to do another update of mod_tile? [15:38:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:38:58] Given that the webservers are on the domain toolserver.org and the ptolemy tileserver is on a.wwww.toolserver.org, I needed to implement CORS to serve json tiles [15:39:40] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [15:49:53] apmon_: yes, but not now. I'm preparing mysql on thmye at the moment [15:50:09] OK, thanks, no problem [15:50:37] Do you know when you might have time? [15:51:29] Just so that I can try and be around to check things [16:18:57] @replag [16:18:58] DaBPunkt: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 12s [-0.08 s/s]; s1-user-c: 12s [+0.00 s/s]; s1-user-wd: 52s [+0.00 s/s]; s2-user: 13s [-0.00 s/s]; s2-user-c: 12s [+0.00 s/s]; s2-user-wd: 52s [+0.00 s/s] [16:18:59] DaBPunkt: s3-rr-a: 1d 2h 51m 57s [-6.73 s/s]; s3-user: 1d 2h 51m 57s [-6.73 s/s]; s4-rr-a: 12s [+0.00 s/s]; s4-user: 12s [+0.00 s/s]; s4-user-wd: 8m 3s [+0.07 s/s]; s5-rr-a: 12s [+0.00 s/s]; s5-user: 12s [+0.00 s/s]; s5-user-c: 12s [+0.00 s/s] [16:19:00] DaBPunkt: s5-user-wd: 52s [+0.00 s/s]; s6-rr-a: 11s [+0.00 s/s]; s6-user: 11s [+0.00 s/s]; s7-rr-a: 11s [-0.00 s/s]; s7-user: 11s [-0.00 s/s] [16:20:00] aliasd on yarrow is CRITICAL: Connection refused [16:21:00] aliasd on yarrow is OK: TCP OK - 0.013 second response time on port 984 [500 Not found.] [16:32:12] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:32:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:33:41] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:38:05] DaB. * [Toolserver-announce] SQL-S1 maintenance (was: Re: [Toolserver-l] sql-s1-rr will be switched this evening) [16:38:05] DaB. * [Toolserver-l] SQL-S1 maintenance (was: Re: sql-s1-rr will be switched this evening) [16:38:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:39:41] MySQL on thyme is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [16:52:38] afk [17:32:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:32:12] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:33:41] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:38:20] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:39:40] MySQL on thyme is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [18:32:12] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:32:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:33:40] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:38:20] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:39:41] MySQL on thyme is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [19:32:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:32:12] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:33:41] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:38:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:39:41] MySQL on thyme is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [19:54:05] hello [19:54:27] I will do the maintenance of wolfsbane in some minutes [20:32:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:33:40] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:33:57] @replag [20:33:58] Dereckson: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 2m 16s [+0.01 s/s]; s1-user-wd: 4h 15m 52s [+1.00 s/s]; s2-user: 12s [-0.00 s/s]; s2-user-wd: 4h 15m 52s [+1.00 s/s]; s4-user-wd: 4h 23m 3s [+1.00 s/s]; s5-user-wd: 4h 15m 52s [+1.00 s/s] [20:38:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:39:41] MySQL on thyme is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [20:46:41] MySQL on thyme is OK: Uptime: 16553 Threads: 2 Questions: 6959705 Slow queries: 10 Opens: 96 Flush tables: 1 Open tables: 89 Queries per second avg: 420.449 [21:32:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:33:41] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:36:45] @replag [21:36:46] DaBPunkt: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 8m 4s [+0.26 s/s]; s1-user-wd: 3h 55m 2s [-3.80 s/s]; s2-user: 13s [-0.11 s/s]; s2-user-c: 41s [+0.16 s/s]; s2-user-wd: 3h 50m 39s [-4.09 s/s]; s4-user-wd: 4h 1m 39s [-3.94 s/s] [21:36:47] DaBPunkt: s5-user-c: 41s [+0.16 s/s]; s5-user-wd: 3h 50m 39s [-4.09 s/s] [21:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:55:21] @replag [21:55:22] DaBPunkt: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 1m 26s [-0.36 s/s]; s1-user-wd: 3h 7m 32s [-2.55 s/s]; s2-user-wd: 3h 2m 1s [-2.62 s/s]; s3-rr-a: 41s [-0.01 s/s]; s3-user: 41s [-0.01 s/s]; s4-user-wd: 3h 10m 32s [-2.75 s/s] [21:55:23] DaBPunkt: s5-user-wd: 3h 2m 1s [-2.62 s/s] [22:08:52] DaBPunkt: Are you still around? [22:09:03] yes [22:10:36] I was away for a couple of hours and was wondering if you had mentioned if you have time today for another update of mod_tile, or if you are still busy. [22:10:51] we can do it now [22:10:57] great [22:11:43] The new one is again in ~osm/tools2/mod_tile.so [22:13:47] And it looks like some log file is growing without bounds since the last update. / is now at 97% full [22:16:46] apmon_: ok, try now [22:19:01] Looks good so far, thanks! [22:19:06] np [22:24:58] DaBPunkt: Could you create a directory /tiles/vtile? I don't have permission to create a directory in /tiles [22:25:57] DaBPunkt: https://jira.toolserver.org/browse/TS-1565 [22:26:03] apmon_: done [22:27:16] thanks [22:29:54] Ah, sweet. Vector rendering now works :-) http://toolserver.org/~osm/styles/kothicJS.html#17/40.00969/-105.27237 [22:32:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:33:41] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:39:20] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:39:46] apmon_: what's the difference? [22:40:30] One is rendered server side, whereas the other is transfered as geojson to the client and rendered in the browser [22:40:46] You have to switch to the Kothic layer [22:42:08] It is still early days for the software stack, but that should allow a much richer number of styles, as the server only has to cache one set of json tiles and then the style is applied on the client [22:43:13] It also potentially allows for interesting other things like arbitrary zoom levels or clickable POIs or highlighting of features [22:51:04] Platonides * Re: [Toolserver-l] SQL-S1 maintenance [22:56:05] @replag all [22:56:06] Brooke: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 26s [-0.02 s/s]; s1-user-c: 2s [+0.00 s/s]; s1-user-wd: 1h 35m 27s [-1.52 s/s]; s2-user: 1s [-0.00 s/s]; s2-user-c: 0s [-0.01 s/s]; s2-user-wd: 1h 23m 25s [-1.62 s/s] [22:56:07] Brooke: s3-rr-a: 52s [+0.00 s/s]; s3-user: 52s [+0.00 s/s]; s4-rr-a: 2s [+0.00 s/s]; s4-user: 3s [+0.00 s/s]; s4-user-wd: 1h 43m 54s [-1.43 s/s]; s5-rr-a: 2s [-0.00 s/s]; s5-user: 2s [-0.00 s/s]; s5-user-c: 1s [-0.01 s/s] [22:56:08] Brooke: s5-user-wd: 1h 23m 26s [-1.62 s/s]; s6-rr-a: 1s [+0.00 s/s]; s6-user: 1s [+0.00 s/s]; s7-rr-a: 2s [-0.00 s/s]; s7-user: 2s [-0.00 s/s] [23:05:04] DaB. * Re: [Toolserver-l] SQL-S1 maintenance [23:11:19] johang: added a quick&dirty-solution [23:32:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:33:41] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:39:20] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)