[00:53:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:14:20] @replag [01:14:21] Alchimista: s1-rr-a-wd: 16s [+0.00 s/s]; s1-user-wd: 16s [-0.00 s/s]; s2-user: 20s [+0.00 s/s]; s2-user-wd: 16s [+0.00 s/s]; s3-rr-a: 4m 19s [-0.00 s/s]; s3-user: 4m 19s [-0.00 s/s]; s4-user-wd: 16s [+0.00 s/s]; s5-rr-a: 18s [+0.00 s/s] [01:14:22] Alchimista: s5-user: 18s [+0.00 s/s]; s5-user-wd: 15s [+0.00 s/s]; s7-rr-a: error; s7-user: error [01:53:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:53:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:53:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:53:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:53:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:53:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:53:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:53:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:53:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:53:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:53:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:53:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:33:33] DanielK_WMDE: do you still have root? [13:33:49] or anybody else here with root? [13:34:44] Danny_B|backup: i have root, but i have no idea what i'm doing ;) [13:53:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:53:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:16:32] hello all [15:16:35] @replag [15:16:36] DaBPunkt: s1-rr-a-wd: 29s [+0.01 s/s]; s1-user-wd: 29s [+0.01 s/s]; s2-user-wd: 23s [+0.00 s/s]; s3-rr-a: 7m 20s [+0.28 s/s]; s3-user: 7m 20s [+0.28 s/s]; s4-user-wd: 29s [+0.01 s/s]; s5-user-wd: 23s [+0.00 s/s]; s6-rr-a: 10m 3s [+0.42 s/s] [15:16:37] DaBPunkt: s6-user: 10m 3s [+0.42 s/s]; s7-rr-a: error; s7-user: error [15:19:10] hello DaB [15:53:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:41:03] @replag [16:41:03] jeremyb: s1-rr-a-wd: 30s [+0.00 s/s]; s1-user-wd: 26s [-0.00 s/s]; s2-user-wd: 30s [+0.00 s/s]; s3-rr-a: 14s [-0.08 s/s]; s3-user: 14s [-0.08 s/s]; s4-user-wd: 30s [+0.00 s/s]; s5-user-wd: 30s [+0.00 s/s]; s7-rr-a: error [16:41:04] jeremyb: s7-user: error [16:41:09] huh [16:53:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:01:19] DaBPunkt: It looks like the root partition on ptolemy is full again :-( [17:02:59] apmon_: wait a moment [17:03:42] " [17:03:44] root@ptolemy:~# svcadm disable network/ts/apache22 [17:03:45] svcadm: Unexpected libscf error on line 338: server has insufficient resources." [17:03:47] Great… [17:03:51] * Platonides imagines DaB running: ssh ptolemy rm -rf / [17:03:58] Sorry to bug you about this so often. But I don't know how to fix this correctly [17:06:19] ok, this time it is the access and the error-log [17:08:50] apmon_: ok, should work again [17:09:09] thanks [17:09:14] Is there something that should stop the logs from growing too much? [17:09:34] I will try to add them to solaris' logrotate [17:10:03] mm, they are already in the config… [17:13:45] /etc/opt/ts/apache/2.2/httpd.conf claims the LogLevel should be error. But the error logs contain a lot of notice level errors in them [17:23:19] "/opt/ts/apache/2.2/bin/apachectl graceful" seems to hand, what causes logadm zu hand [17:23:25] hand → hang [17:34:25] I wonder if mod_tile doesn't cleanly close all sockets, and so doesn't exit gracefully? [17:40:43] On a different note, is it possible to add IPv6 to {a.b.c}.www.toolserver.org? [17:43:44] hi. do you know whether s7 DB is ok now? i tried to access some data there and i can see there only the information schema [17:44:01] @replag [17:44:01] Danny_B|backup: s1-rr-a-wd: 33s [+0.00 s/s]; s1-user: 35s [+0.00 s/s]; s1-user-wd: 33s [+0.00 s/s]; s2-user-wd: 33s [+0.00 s/s]; s3-rr-a: 7m 28s [+0.11 s/s]; s3-user: 7m 28s [+0.11 s/s]; s4-user-wd: 33s [+0.00 s/s]; s5-user-wd: 33s [+0.00 s/s] [17:44:02] Danny_B|backup: s7-rr-a: error; s7-user: error [17:53:05] Eran Rosenthal * Re: [Toolserver-l] Defective database s7 [17:53:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:09:06] Marlen Caemmerer * Re: [Toolserver-l] Defective database s7 [18:53:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:55:05] Eran Rosenthal * Re: [Toolserver-l] Defective database s7 [19:06:05] Marlen Caemmerer * Re: [Toolserver-l] Defective database s7 [19:27:50] @replag [19:27:50] DaBPunkt: s1-rr-a-wd: 19m 27s [+0.48 s/s]; s1-user-wd: 20m 34s [+0.50 s/s]; s2-user-wd: 20m 43s [+0.49 s/s]; s4-user-wd: 47m 25s [+0.85 s/s]; s5-user-wd: 20m 43s [+0.49 s/s]; s7-rr-a: 1w 6h 7s [-5.46 s/s]; s7-user: 1w 6h 7s [-5.46 s/s] [19:32:32] @replag [19:32:32] jem-: s1-rr-a-wd: 23m 9s [+0.79 s/s]; s1-user: 14s [+0.01 s/s]; s1-user-wd: 23m 4s [+0.53 s/s]; s2-user-wd: 23m 21s [+0.56 s/s]; s4-user-wd: 51m 8s [+0.79 s/s]; s5-user-wd: 23m 21s [+0.56 s/s]; s7-rr-a: 1w 5h 31m 27s [-6.09 s/s]; s7-user: 1w 5h 31m 27s [-6.09 s/s] [19:33:00] Hooray, s7 is getting back! :) [19:53:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:05:16] @replag [20:05:16] DaBPunkt: s1-rr-a-wd: 54m 35s [-]; s1-user: 14s [-]; s1-user-wd: 40m 35s [-]; s2-user-wd: 41m 13s [-]; s4-user-wd: 1h 12m 49s [-]; s5-user-wd: 41m 13s [-]; s7-rr-a: 1w 3h 28m 20s [-]; s7-user: 1w 3h 28m 20s [-] [20:05:17] DaBPunkt: s7-user-wd: 54m 35s [-] [20:06:55] @replag [20:06:55] DaBPunkt: s1-rr-a-wd: 55m 23s [-]; s1-user-wd: 41m 20s [-]; s2-user-wd: 42m 2s [-]; s3-user: 1m 33s [-]; s4-user-wd: 1h 14m 12s [-]; s5-user-wd: 42m 2s [-]; s7-user: 1w 3h 16m 58s [-]; s7-user-wd: 55m 23s [-] [20:45:16] @replag [20:45:17] eranroz: s1-rr-a-wd: 1h 14m 47s [+0.51 s/s]; s1-user-wd: 57m 2s [+0.41 s/s]; s2-user-wd: 1h 7s [+0.47 s/s]; s4-user-wd: 1h 43m 55s [+0.77 s/s]; s5-user-wd: 1h 7s [+0.47 s/s]; s7-user: 6d 23h 50m 33s [-5.38 s/s]; s7-user-wd: 1h 17m 9s [+0.57 s/s] [20:46:35] 6 days and 23 hours replication lag instead of 1w and 3h. right direction :) [20:53:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:01:43] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [21:06:43] MySQL on z-dat-s6-a is OK: Uptime: 687 Threads: 3 Questions: 4734 Slow queries: 1 Opens: 92 Flush tables: 1 Open tables: 84 Queries per second avg: 6.890 [21:12:06] DaB. * Re: [Toolserver-l] Defective database s7 [21:53:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:07:30] @replag [22:07:31] DaBPunkt: s1-rr-a-wd: 13m 15s [-0.75 s/s]; s1-user: 21s [-]; s1-user-wd: 7m 50s [-0.60 s/s]; s2-user-wd: 7m 50s [-0.64 s/s]; s3-user: 16s [-0.01 s/s]; s4-user-wd: 2h 31m 37s [+0.58 s/s]; s5-user-wd: 7m 50s [-0.64 s/s]; s7-user: 6d 11h 41m 44s [-8.86 s/s] [22:07:32] DaBPunkt: s7-user-wd: 47m 33s [-0.36 s/s] [22:07:49] @replag [22:07:50] DaBPunkt: s1-rr-a-wd: 11m 55s [-4.18 s/s]; s1-user: 21s [-]; s1-user-wd: 8m 9s [+1.00 s/s]; s2-user-wd: 8m 9s [+1.00 s/s]; s3-user: 35s [+1.00 s/s]; s4-user-wd: 2h 31m 56s [+1.00 s/s]; s5-user-wd: 8m 9s [+1.00 s/s]; s7-user: 6d 11h 37m 55s [-12.04 s/s] [22:07:51] DaBPunkt: s7-user-wd: 46m 24s [-3.63 s/s] [22:23:05] @replag [22:23:06] DaBPunkt: s1-rr-a-wd: 30s [-0.75 s/s]; s1-user-wd: 10m 12s [+0.13 s/s]; s2-user-wd: 5m 55s [-0.15 s/s]; s4-user-wd: 2h 45m 22s [+0.88 s/s]; s5-user-wd: 5m 54s [-0.15 s/s]; s7-user: 6d 9h 48m 59s [-7.13 s/s]; s7-user-wd: 35m 25s [-0.72 s/s] [22:28:19] @replag [22:28:20] DaBPunkt: s1-rr-a-wd: 11s [-0.06 s/s]; s1-user-c: 14s [-]; s1-user-wd: 12s [-1.91 s/s]; s2-user-wd: 28s [-1.04 s/s]; s3-user: 44s [+0.01 s/s]; s4-user-wd: 2h 47m 36s [+0.43 s/s]; s5-user-wd: 28s [-1.04 s/s]; s7-user: 6d 9h 12m 48s [-6.92 s/s] [22:28:21] DaBPunkt: s7-user-wd: 14m 4s [-4.08 s/s] [22:33:47] @replag [22:33:47] DaBPunkt: s1-rr-a-wd: 21s [+0.03 s/s]; s1-user: 42s [+0.01 s/s]; s1-user-c: 1m 13s [+0.18 s/s]; s1-user-wd: 21s [+0.03 s/s]; s2-user: 17s [-]; s2-user-wd: 21s [-0.02 s/s]; s3-user: 1m 6s [+0.07 s/s]; s4-user-wd: 2h 50m 4s [+0.45 s/s] [22:33:48] DaBPunkt: s5-user-wd: 21s [-0.02 s/s]; s7-user: 6d 8h 38m 14s [-6.33 s/s]; s7-user-wd: 21s [-2.51 s/s] [22:41:30] @replag [22:41:30] DaBPunkt: s1-rr-a-wd: 26s [+0.01 s/s]; s1-user-wd: 26s [+0.01 s/s]; s2-user-wd: 26s [+0.01 s/s]; s3-user: 42s [-0.05 s/s]; s4-user-wd: 2h 47m 48s [-0.29 s/s]; s5-user-wd: 26s [+0.01 s/s]; s7-user: 6d 7h 45m 34s [-6.83 s/s]; s7-user-wd: 26s [+0.01 s/s] [22:49:40] @replag [22:49:40] DaBPunkt: s1-rr-a-wd: 12s [-0.03 s/s]; s1-user-wd: 12s [-0.03 s/s]; s2-user-wd: 11s [-0.03 s/s]; s4-user-wd: 2h 51m 11s [+0.41 s/s]; s5-user-wd: 11s [-0.03 s/s]; s7-user: 6d 6h 45m 3s [-7.40 s/s]; s7-user-wd: 12s [-0.03 s/s] [22:53:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:16:31] @replag [23:16:31] DaBPunkt: s1-rr-a-wd: 19s [+0.00 s/s]; s1-user-wd: 19s [+0.00 s/s]; s2-user-wd: 16s [+0.00 s/s]; s3-user: 1m 12s [+0.01 s/s]; s4-user-wd: 2h 10m 40s [-1.51 s/s]; s5-user-wd: 16s [+0.00 s/s]; s7-user: 6d 3h 45m 42s [-6.68 s/s]; s7-user-wd: 17s [+0.00 s/s] [23:42:11] @replag [23:42:11] DaBPunkt: s1-rr-a-wd: 26s [+0.00 s/s]; s1-user-wd: 28s [+0.01 s/s]; s2-user-wd: 28s [+0.01 s/s]; s3-user: 5m 23s [+0.16 s/s]; s4-user-wd: 38m 19s [-3.60 s/s]; s5-user-wd: 28s [+0.01 s/s]; s7-user: 6d 52m 33s [-6.75 s/s]; s7-user-wd: 28s [+0.01 s/s] [23:44:20] DaBPunkt: is it already safe to relaunch the bots reading data from s7? [23:44:33] Or should we wait until replag gets normal? [23:46:38] jem-: that depens if you can life with 7 days old data [23:46:54] Hmmmmm [23:47:25] I guess I could, but I'm afraid the editcount-addicts would not [23:47:37] Ok, I'll wait then :) [23:48:31] jem-: if you use sge, scripts will start if delay < 3h [23:49:34] Merlissimo: I have disabled it in order to not overwrite a warning that I added by hand to the rankings [23:49:57] So I prefer to be sure in order to reactivate [23:50:26] @replag [23:50:26] dschwen: s1-rr-a-wd: 32s [+0.01 s/s]; s1-user-wd: 24s [-0.01 s/s]; s2-user: 26s [+0.00 s/s]; s2-user-wd: 24s [-0.01 s/s]; s3-user: 1m 35s [-0.46 s/s]; s4-user-wd: 7m 30s [-3.73 s/s]; s5-user-wd: 24s [-0.01 s/s]; s7-user: 5d 23h 28m 47s [-10.15 s/s] [23:50:27] dschwen: s7-user-wd: 24s [-0.01 s/s] [23:50:55] hmm [23:51:05] is sql-s4 supposed to be up to date? [23:51:17] I'm banging my head against a wall here [23:51:24] @replag all [23:51:24] Merlissimo: s1-rr-a: 0s [-]; s1-rr-a-wd: 29s [-0.05 s/s]; s1-user: 2s [-0.01 s/s]; s1-user-c: 1s [-0.02 s/s]; s1-user-wd: 29s [+0.09 s/s]; s2-user: 14s [-0.21 s/s]; s2-user-c: 1s [-]; s2-user-wd: 29s [+0.09 s/s] [23:51:25] Merlissimo: s3-user: 18s [-1.33 s/s]; s4-rr-a: 1s [-]; s4-user: 1s [-]; s4-user-wd: 2m 53s [-4.78 s/s]; s5-rr-a: 4s [-]; s5-user: 4s [-]; s5-user-c: 1s [-]; s5-user-wd: 29s [+0.09 s/s] [23:51:26] Merlissimo: s6-user: 0s [-]; s7-user: 5d 23h 22m 24s [-6.61 s/s]; s7-user-wd: 29s [+0.09 s/s] [23:51:35] dschwen: ^^ [23:51:43] qicvic@sql-s4:commonswiki_p> select * from page where page_title='Valued_image_candidates/Domaine_Carneros_2011.jpg' and page_namespace=4; [23:51:43] Empty set (0.00 sec) [23:51:43] 4s [23:51:51] but the page exists! [23:51:59] Commons namespace [23:53:30] replication mistake [23:53:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:54:31] i on s2 it's there [23:54:45] select * from page where page_id=22526676; [23:54:47] select * from page where page_id=22526676; [23:54:50] 1 row in set (0.00 sec) [23:54:54] hmm [23:54:55] but not on s4 [23:55:11] so the replag does not tell the whole story [23:55:32] basically the commonswiki copy on s4 is inconsistent and not to be trusted [23:56:02] maybe s4 shouldnot be set as the default server for commonswiki_p then [23:56:21] only on s1 and s2/5, but not on any other server [23:57:03] commonswiki-p.rrdb.toolserver.org points to s4 [23:57:25] how am I supposed to refer to the db server then? [23:57:35] DaBPunkt: any know problems with commons on s3/s4/s6/s7? [23:57:37] shuld I hardcode it to s2? or s1? [23:58:43] Merlissimo: s3,s6 and s7 have no commons-copy [23:58:48] also because tsbot does not return a value for s3-user-c, s6-user-c or s7-user-c [23:59:37] this is about s4 !