[00:00:52] Free Memory on turnera is WARNING: WARNING - 5.2% (434340 kB) free! [00:05:12] Environment IPMI on thyme is CRITICAL: Connection refused by host [00:07:01] SSH on thyme is CRITICAL: Connection refused [00:14:52] Free Memory on turnera is OK: OK - 7.1% (596884 kB) free. [00:16:52] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:27:31] aliasd on yarrow is CRITICAL: Connection refused [00:32:51] Free Memory on turnera is WARNING: WARNING - 5.2% (439024 kB) free! [00:33:50] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (415132 kB) free! [00:58:01] RAID on thyme is CRITICAL: Connection refused by host [01:05:11] Environment IPMI on thyme is CRITICAL: Connection refused by host [01:07:02] SSH on thyme is CRITICAL: Connection refused [01:14:51] Free Memory on turnera is WARNING: WARNING - 6.2% (518116 kB) free! [01:16:51] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:23:21] @replag [01:23:22] Riley: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 2h 6m 31s [+0.14 s/s]; s1-user-wd: 1m 37s [-0.00 s/s]; s2-user: 16s [+0.00 s/s]; s2-user-wd: 1m 37s [-0.00 s/s]; s3-rr-a: 1m 2s [-0.05 s/s]; s3-user: 1m 2s [-0.05 s/s] [01:23:23] Riley: s4-user-wd: 1m 37s [-0.00 s/s]; s5-user-wd: 1m 37s [-0.00 s/s]; s7-rr-a: 2d 5h 37m 56s [+1.00 s/s]; s7-user: 2d 5h 37m 56s [+1.00 s/s] [01:27:30] aliasd on yarrow is CRITICAL: Connection refused [01:31:52] Free Memory on turnera is OK: OK - 7.2% (603164 kB) free. [01:59:00] RAID on thyme is CRITICAL: Connection refused by host [02:05:12] Environment IPMI on thyme is CRITICAL: Connection refused by host [02:08:01] SSH on thyme is CRITICAL: Connection refused [02:16:51] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:27:31] aliasd on yarrow is CRITICAL: Connection refused [02:59:01] RAID on thyme is CRITICAL: Connection refused by host [03:05:12] Environment IPMI on thyme is CRITICAL: Connection refused by host [03:09:01] SSH on thyme is CRITICAL: Connection refused [03:16:51] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:28:31] aliasd on yarrow is CRITICAL: Connection refused [03:43:51] Free Memory on turnera is WARNING: WARNING - 6.0% (504616 kB) free! [03:45:51] Free Memory on turnera is OK: OK - 7.3% (607812 kB) free. [03:55:51] Free Memory on turnera is WARNING: WARNING - 6.2% (523096 kB) free! [04:00:01] RAID on thyme is CRITICAL: Connection refused by host [04:05:14] Environment IPMI on thyme is CRITICAL: Connection refused by host [04:09:14] SSH on thyme is CRITICAL: Connection refused [04:16:52] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:28:32] aliasd on yarrow is CRITICAL: Connection refused [04:31:52] Free Memory on turnera is WARNING: WARNING - 5.5% (464552 kB) free! [04:33:53] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (420876 kB) free! [04:35:52] Free Memory on turnera is WARNING: WARNING - 5.6% (466280 kB) free! [04:42:53] Free Memory on turnera is CRITICAL: CRITICAL - 4.2% (355704 kB) free! [05:00:16] RAID on thyme is CRITICAL: Connection refused by host [05:05:16] Environment IPMI on thyme is CRITICAL: Connection refused by host [05:09:16] SSH on thyme is CRITICAL: Connection refused [05:16:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:28:33] aliasd on yarrow is CRITICAL: Connection refused [06:00:17] RAID on thyme is CRITICAL: Connection refused by host [06:00:33] Is toolserver on its last legs? [06:06:16] Environment IPMI on thyme is CRITICAL: Connection refused by host [06:09:47] Tim1357: no, just having some issues [06:10:04] tyme needs a hard reboot, and we dont have anyone at the datacenter [06:10:12] Oh. [06:10:15] SSH on thyme is CRITICAL: Connection refused [06:10:24] So is it worth it to get my account re-activated. [06:10:28] Well. [06:10:30] yeah [06:10:31] I want my data. [06:10:37] So I'm going to be annoying and ask for it. [06:10:53] Tim1357: just file a JIRA ticket [06:11:06] Yeah. [06:11:12] This is the second time though. My pride... [06:11:18] DaB will get to it as soon as he can (he has a conference this week) [06:11:42] Tim1357: set your calendar and check your emails [06:11:51] you get like 4 nag emails [06:12:25] Well when I had it renewed the first time, my account was set to expire the next week. [06:13:20] Oo [06:14:09] Nope. [06:14:14] I just checked I had a month. [06:15:43] So beta are you moving your stuff to labs? [06:15:53] Free Memory on turnera is WARNING: WARNING - 5.9% (493500 kB) free! [06:15:56] Tim1357: not till they get stuff worked out [06:16:26] I don't understand why they're doing it. Does it save money? [06:16:28] Ive got extremely stable code [06:16:45] Tim1357: toolserver is WMDE while labs is WMF [06:16:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:16:55] They're separate? [06:17:01] yeah [06:17:16] WMDE is the german chapter [06:17:29] Will Labs have the same SQL replication? [06:17:33] That's the only thing I really need. [06:17:48] /want [06:18:11] Tim1357: at some point replication is on the todo list, wether or not it works as well is a different story [06:18:33] they where not planning on putting commons on all custers [06:18:48] == bad idea\ [06:18:52] That's silly. [06:19:18] Does this mean our data connection to the wikipedia servers will be better? Seeing that they'd theoretically be in the same building? [06:19:23] labs still will be about 2 years or so to catch up [06:19:44] Tim1357: they already are [06:19:58] oh [06:21:25] esams and knams are both in Amsterdam [06:22:29] Oh cool this is just like EC2 [06:22:36] The instance types and whatnot. [06:28:33] aliasd on yarrow is CRITICAL: Connection refused [06:34:55] Free Memory on turnera is CRITICAL: CRITICAL - 2.5% (206172 kB) free! [06:37:53] Free Memory on turnera is OK: OK - 12.1% (1014016 kB) free. [07:01:15] RAID on thyme is CRITICAL: Connection refused by host [07:06:16] Environment IPMI on thyme is CRITICAL: Connection refused by host [07:10:16] SSH on thyme is CRITICAL: Connection refused [07:16:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:28:33] aliasd on yarrow is CRITICAL: Connection refused [07:46:52] Free Memory on turnera is WARNING: WARNING - 6.2% (520088 kB) free! [07:49:54] Free Memory on turnera is OK: OK - 7.1% (596464 kB) free. [08:01:16] RAID on thyme is CRITICAL: Connection refused by host [08:04:53] Free Memory on turnera is WARNING: WARNING - 6.5% (545400 kB) free! [08:06:17] Environment IPMI on thyme is CRITICAL: Connection refused by host [08:11:16] SSH on thyme is CRITICAL: Connection refused [08:16:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:28:32] aliasd on yarrow is CRITICAL: Connection refused [08:46:53] Free Memory on turnera is WARNING: WARNING - 5.7% (476436 kB) free! [08:53:53] Free Memory on turnera is OK: OK - 7.1% (592004 kB) free. [09:01:16] RAID on thyme is CRITICAL: Connection refused by host [09:07:16] Environment IPMI on thyme is CRITICAL: Connection refused by host [09:11:16] SSH on thyme is CRITICAL: Connection refused [09:16:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:28:33] aliasd on yarrow is CRITICAL: Connection refused [09:56:54] Free Memory on turnera is WARNING: WARNING - 6.6% (556716 kB) free! [09:58:53] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (376788 kB) free! [10:00:53] Free Memory on turnera is WARNING: WARNING - 6.0% (506528 kB) free! [10:01:15] RAID on thyme is CRITICAL: Connection refused by host [10:05:57] @replag [10:08:15] Environment IPMI on thyme is CRITICAL: Connection refused by host [10:11:15] SSH on thyme is CRITICAL: Connection refused [10:16:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:19:16] DeltaQuad Wikipedia * [Toolserver-l] Public DB status down [10:28:33] aliasd on yarrow is CRITICAL: Connection refused [10:44:54] Free Memory on turnera is WARNING: WARNING - 6.4% (539688 kB) free! [10:45:53] Free Memory on turnera is OK: OK - 7.6% (637572 kB) free. [11:01:16] RAID on thyme is CRITICAL: Connection refused by host [11:03:15] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [11:08:17] Environment IPMI on thyme is CRITICAL: Connection refused by host [11:11:16] SSH on thyme is CRITICAL: Connection refused [11:17:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:21:15] MySQL on z-dat-s3-a is OK: Uptime: 1480 Threads: 6 Questions: 603 Slow queries: 0 Opens: 251 Flush tables: 1 Open tables: 243 Queries per second avg: 0.407 [11:23:04] @replag [11:29:33] aliasd on yarrow is CRITICAL: Connection refused [12:02:16] RAID on thyme is CRITICAL: Connection refused by host [12:09:16] Environment IPMI on thyme is CRITICAL: Connection refused by host [12:11:17] SSH on thyme is CRITICAL: Connection refused [12:17:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:29:34] aliasd on yarrow is CRITICAL: Connection refused [12:59:44] @replag [13:02:16] RAID on thyme is CRITICAL: Connection refused by host [13:09:15] Environment IPMI on thyme is CRITICAL: Connection refused by host [13:12:15] SSH on thyme is CRITICAL: Connection refused [13:17:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:29:33] aliasd on yarrow is CRITICAL: Connection refused [13:33:16] Carl (CBM) * Re: [Toolserver-l] Public DB status down [13:42:13] Thyme's been rosemary all Sunday morning http://munin.toolserver.org/Database/rosemary/io_busy_sd.html [13:42:30] Rosemary been thrashing Sunday morning http://munin.toolserver.org/Database/rosemary/io_busy_sd.html * [13:46:11] Strange things are happening with the s1 servers [13:47:31] rosemary is thrashing (it was thyme previously), restarting mysql-daemon seems like the only solution. http://munin.toolserver.org/Database/rosemary/io_busy_sd.html [13:50:16] DaB. * Re: [Toolserver-l] Public DB status down [13:56:21] hmm. now I can reconnect to sql-s1-user from command line, but when I try to kill the queries that are sitting there, they don't go away [14:02:15] RAID on thyme is CRITICAL: Connection refused by host [14:03:23] MySQL on rosemary is CRITICAL: Too many connections [14:04:24] MySQL on rosemary is OK: Uptime: 1928004 Threads: 501 Questions: 17005223612 Slow queries: 498799 Opens: 118485 Flush tables: 1 Open tables: 4257 Queries per second avg: 8820.118 [14:04:30] hello all [14:04:53] Free Memory on turnera is WARNING: WARNING - 5.3% (444068 kB) free! [14:05:01] hello, good afternoon [14:06:24] rosemary is swapping, I will restart the mysql-process [14:08:53] Free Memory on turnera is OK: OK - 7.1% (595948 kB) free. [14:09:16] Environment IPMI on thyme is CRITICAL: Connection refused by host [14:12:16] SSH on thyme is CRITICAL: Connection refused [14:14:23] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [14:14:31] 15% [14:18:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:29:33] aliasd on yarrow is CRITICAL: Connection refused [14:30:33] aliasd on yarrow is OK: TCP OK - 0.003 second response time on port 984 [500 Not found.] [14:33:23] MySQL on rosemary is OK: Uptime: 1461 Threads: 17 Questions: 6079 Slow queries: 22 Opens: 250 Flush tables: 1 Open tables: 240 Queries per second avg: 4.160 [15:03:15] RAID on thyme is CRITICAL: Connection refused by host [15:04:53] Free Memory on turnera is CRITICAL: CRITICAL - 3.5% (291372 kB) free! [15:09:16] Environment IPMI on thyme is CRITICAL: Connection refused by host [15:12:53] Free Memory on turnera is WARNING: WARNING - 5.1% (426976 kB) free! [15:13:16] SSH on thyme is CRITICAL: Connection refused [15:18:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:45:54] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (421060 kB) free! [15:47:53] Free Memory on turnera is WARNING: WARNING - 5.3% (444584 kB) free! [15:48:53] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (417700 kB) free! [16:03:15] RAID on thyme is CRITICAL: Connection refused by host [16:09:17] Environment IPMI on thyme is CRITICAL: Connection refused by host [16:11:53] Free Memory on turnera is WARNING: WARNING - 5.2% (432852 kB) free! [16:13:16] SSH on thyme is CRITICAL: Connection refused [16:13:54] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (420864 kB) free! [16:14:53] Free Memory on turnera is WARNING: WARNING - 5.1% (429780 kB) free! [16:18:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:28:53] Free Memory on turnera is OK: OK - 7.2% (601468 kB) free. [17:03:16] RAID on thyme is CRITICAL: Connection refused by host [17:10:16] Environment IPMI on thyme is CRITICAL: Connection refused by host [17:13:15] SSH on thyme is CRITICAL: Connection refused [17:19:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:37:53] Free Memory on turnera is WARNING: WARNING - 6.1% (508184 kB) free! [17:42:53] Free Memory on turnera is OK: OK - 7.2% (602976 kB) free. [17:56:53] Free Memory on turnera is WARNING: WARNING - 6.8% (570980 kB) free! [17:58:53] Free Memory on turnera is CRITICAL: CRITICAL - 4.0% (334716 kB) free! [18:04:15] RAID on thyme is CRITICAL: Connection refused by host [18:08:53] Free Memory on turnera is WARNING: WARNING - 5.1% (430324 kB) free! [18:11:16] Environment IPMI on thyme is CRITICAL: Connection refused by host [18:14:15] SSH on thyme is CRITICAL: Connection refused [18:19:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:40:53] Free Memory on turnera is OK: OK - 7.1% (596488 kB) free. [18:47:54] Free Memory on turnera is WARNING: WARNING - 6.8% (568020 kB) free! [18:53:54] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (404168 kB) free! [19:04:16] RAID on thyme is CRITICAL: Connection refused by host [19:11:16] Environment IPMI on thyme is CRITICAL: Connection refused by host [19:14:15] SSH on thyme is CRITICAL: Connection refused [19:18:53] Free Memory on turnera is WARNING: WARNING - 6.7% (558424 kB) free! [19:20:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:22:53] Free Memory on turnera is CRITICAL: CRITICAL - 4.2% (350200 kB) free! [19:30:53] Free Memory on turnera is WARNING: WARNING - 5.3% (441232 kB) free! [19:31:53] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (404468 kB) free! [20:05:15] RAID on thyme is CRITICAL: Connection refused by host [20:11:16] Environment IPMI on thyme is CRITICAL: Connection refused by host [20:14:17] SSH on thyme is CRITICAL: Connection refused [20:20:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:05:14] RAID on thyme is CRITICAL: Connection refused by host [21:05:49] DaBPunkt: Hi, the / disk partition on ptolemy has filled up again. [21:06:11] Do you know which (log) file is suddenly causing these run away disk usage? [21:06:37] let me look in a minute [21:08:40] thanks [21:11:15] Environment IPMI on thyme is CRITICAL: Connection refused by host [21:14:16] SSH on thyme is CRITICAL: Connection refused [21:14:40] apmon_: doesn't look like a logfile-problem to me [21:15:32] hmm, somehow the disk usage has been going up since we did the update of mod_tile [21:16:03] But a du as user osm can only account for 7Gb of the 20 Gb. So I can't find out where the space is going [21:16:23] I guess 20GB is just too small for all the stuff. I will ask nosy if it is possible to expand the partition. I will free some space now [21:17:49] looking at http://munin.toolserver.org/OSM/ptolemy/df-month.png it is an issue of unlimited growth though [21:18:23] It looks like it used to get automatically reset every day or so, but that cleanup process might no longer be running? [21:20:44] the complete var-partition is 3.4GB. 2.2GB of it is for the packages, the log-files are smaller than 500MB [21:20:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:22:16] It looks like it grew about 10Gb in 7 days. So the culprit should be in the order of about 10GB data. [21:23:47] If you run a du --max-depth=1 -x -h on / is there any directory that has more than 9GB in it? [21:24:43] 147M ./boot [21:24:45] 328M ./etc [21:24:46] 764M ./root [21:24:47] 1.2G ./usr [21:24:49] 2.8G ./var [21:24:51] 3.9G ./opt [21:26:46] I will stop the apache there for a moment [21:27:59] and 11GB back [21:28:21] something there is using disc-space in a unclean way [21:28:45] and it is not log files? [21:28:51] strange [21:37:16] DaB. * Re: [Toolserver-l] S1 [22:05:16] RAID on thyme is CRITICAL: Connection refused by host [22:11:15] Environment IPMI on thyme is CRITICAL: Connection refused by host [22:14:16] SSH on thyme is CRITICAL: Connection refused [22:20:52] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:05:15] RAID on thyme is CRITICAL: Connection refused by host [23:12:15] Environment IPMI on thyme is CRITICAL: Connection refused by host [23:15:15] SSH on thyme is CRITICAL: Connection refused [23:20:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:23:02] I can't access https://jira.toolserver.org/browse/ACCAPP-570 anymore. Does this mean my bot account request has been rejected? [23:23:59] Mailaender: no. the request was switched to private-mode so you can insert your real-bame [23:24:08] bame → name [23:25:03] I get "BERECHTIGUNGSVERLETZUNG" [23:25:36] are you logined? [23:27:00] yes [23:27:12] it says I have to login first [23:27:28] mm, I will switch the request back to normal mode than [23:27:44] try now [23:34:10] DaBPunkt: works [23:34:58] ok, I try to look after it tomorrow [23:35:20] thanks [23:38:20] npo [23:38:22] -o [23:38:47] nacht ts