[00:14:41] <DaBPunkt>	 nacht ts
[00:17:55] <tsnag>	 s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1710.000000  
[00:18:15] <tsnag>	 MySQL slave on thyme is OK: Uptime: 1072106  Threads: 7  Questions: 459126741  Slow queries: 150077  Opens: 48532  Flush tables: 1  Open tables: 2608  Queries per second avg: 428.247 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1662  
[00:43:25] <tsnag>	 /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 39576 MB (4% inode=99%):  
[00:43:35] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[00:43:47] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[00:43:56] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[00:44:17] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 81095  
[00:45:24] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[00:45:36] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[00:45:36] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[00:45:36] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[00:46:35] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[00:47:25] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[01:02:45] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 16.19, 15.54, 13.71  
[01:08:16] <tsnag>	 s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1874.000000  
[01:08:46] <tsnag>	 Load avg. on willow is OK: OK - load average: 13.95, 14.86, 14.05  
[01:12:46] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 13.95, 15.57, 14.61  
[01:44:25] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 85388  
[01:44:35] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[01:44:45] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[01:44:55] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[01:45:25] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[01:46:34] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[01:46:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[01:46:35] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[01:46:35] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[01:46:52] <msgbot>	 3(created) [MAGNUS-301] CatFood fails with message "something's wrong!"; Magnus' tools: Catfood; Bug <10https://jira.toolserver.org/browse/MAGNUS-301>  (rd232)
[01:46:55] <msgbot>	 3(closed) [UTRS-52] Find a way to add a notice that javascript is required, if it's disabled <10https://jira.toolserver.org/browse/UTRS-52>  (Hersfold)
[01:47:25] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[01:50:14] <tsnag>	 s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3603.000000  
[02:02:47] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 16.66, 15.72, 13.88  
[02:14:45] <tsnag>	 Load avg. on willow is OK: OK - load average: 12.81, 14.88, 14.57  
[02:27:47] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 15.15, 14.70, 14.50  
[02:29:46] <tsnag>	 Load avg. on willow is OK: OK - load average: 14.75, 14.91, 14.62  
[02:44:25] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 84913  
[02:44:46] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[02:44:56] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[02:45:25] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[02:45:35] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[02:46:35] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[02:46:35] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[02:46:35] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[02:46:35] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[02:47:25] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[02:50:15] <tsnag>	 s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5772.000000  
[03:06:19] <wikirc_>	 [[List of Wikimedia bots]] ! 10https://wiki.toolserver.org/w/index.php?diff=6715&oldid=6511&rcid=8870 * 111.91.137.36 * (+123) (/* {{#language:ko}} */ )
[03:06:55] <wikirc_>	 [[List of Wikimedia bots]] ! 10https://wiki.toolserver.org/w/index.php?diff=6716&oldid=6715&rcid=8871 * 111.91.137.36 * (-16) (/* {{#language:ko}} */ )
[03:09:45] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 10.60, 12.40, 15.74  
[03:17:46] <tsnag>	 Load avg. on willow is OK: OK - load average: 11.43, 12.98, 14.85  
[03:24:51] <msgbot>	 3(assigned) [UTRS-47] Notification of reply <10https://jira.toolserver.org/browse/UTRS-47>  (Andrew Pearson)
[03:24:55] <msgbot>	 3(resolved) [UTRS-47] Notification of reply <10https://jira.toolserver.org/browse/UTRS-47>  (Andrew Pearson)
[03:26:53] <msgbot>	 3(assigned) [UTRS-39] Send an email to users that have had their accounts approved to let them know that their account has been approved - also create one for when an account has been disabled (and allow the reason to show up too) <10https://jira.toolserver.org/browse/UTRS-39>  (Andrew Pearson)
[03:44:46] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[03:44:56] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[03:45:24] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88574  
[03:45:36] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[03:45:37] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[03:46:36] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[03:46:36] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[03:46:37] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[03:46:37] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[03:47:37] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[03:50:15] <tsnag>	 s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8280.000000  
[03:52:54] <msgbot>	 3(created) [MAGNUS-303] Undefined indices in fist.php; Magnus' tools; Bug <10https://jira.toolserver.org/browse/MAGNUS-303>  (Jeff G.)
[04:10:57] <msgbot>	 3(resolved) [UTRS-39] Send an email to users that have had their accounts approved to let them know that their account has been approved - also create one for when an account has been disabled (and allow the reason to show up too) <10https://jira.toolserver.org/browse/UTRS-39>  (Andrew Pearson)
[04:30:13] <wikirc_>	 [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=6717&oldid=6688&rcid=8872 * 91.198.174.202 * (+1) (updated page)
[04:35:52] <msgbot>	 3(closed) [UTRS-4] Allowing renaming of users by tool admins <10https://jira.toolserver.org/browse/UTRS-4>  (Hersfold)
[04:44:54] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[04:45:04] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[04:45:25] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 92175  
[04:45:36] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[04:45:45] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[04:46:45] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[04:47:36] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[04:47:36] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[04:47:36] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[04:48:36] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[04:50:15] <tsnag>	 s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11378.000000  
[05:29:45] <tsnag>	 /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40075 MB (9% inode=99%):  
[05:31:45] <tsnag>	 /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 66358 MB (16% inode=99%):  
[05:40:45] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 15.45, 15.24, 13.77  
[05:42:45] <tsnag>	 Load avg. on willow is OK: OK - load average: 12.03, 14.23, 13.58  
[05:45:05] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[05:45:05] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[05:45:25] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 95776  
[05:45:35] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[05:45:45] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[05:45:45] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 17.39, 16.00, 14.41  
[05:47:35] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[05:47:35] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[05:47:35] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[05:47:45] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[05:48:36] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[05:50:15] <tsnag>	 s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14356.000000  
[06:00:52] <msgbot>	 3(assigned) [UTRS-54] Replace expired IP data with one-way hash <10https://jira.toolserver.org/browse/UTRS-54>  (Andrew Pearson)
[06:02:55] <msgbot>	 3(resolved) [UTRS-54] Replace expired IP data with one-way hash <10https://jira.toolserver.org/browse/UTRS-54>  (Andrew Pearson)
[06:20:15] <tsnag>	 s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3586.000000  
[06:21:14] <tsnag>	 s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3620.000000  
[06:45:05] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[06:45:15] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[06:45:25] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 99376  
[06:45:45] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 20.72, 16.17, 14.87  
[06:45:46] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[06:46:35] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[06:47:46] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[06:47:47] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[06:47:55] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[06:48:35] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[06:48:45] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[06:51:45] <tsnag>	 Load avg. on willow is OK: OK - load average: 12.48, 14.72, 14.69  
[07:00:53] <Nirvanchik>	 is it possible to run java application on toolserver? to make this automatically by schedule?
[07:02:46] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 24.80, 19.22, 15.96  
[07:03:30] <Nirvanchik>	 why replication lag is so big in some servers(s2,s6) ?
[07:07:22] <whym>	 Nirvanchik: i have tried my java application from command line and it went well
[07:08:10] <Nirvanchik>	 whym: good
[07:08:25] <whym>	 Nirvanchik: as for scheduling you can use cronie (or crontab)
[07:08:39] <Nirvanchik>	 whym: what is java version there?
[07:09:33] <whym>	 java version "1.6.0_30"
[07:09:34] <whym>	 Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
[07:09:34] <whym>	 Java HotSpot(TM) Server VM (build 20.5-b03, mixed mode)
[07:09:49] <whym>	 Nirvanchik: java -version says above
[07:09:53] <Nirvanchik>	 :(
[07:10:17] <Nirvanchik>	 I would better need 1.7 :(
[07:10:18] <tsnag>	 s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3204.000000  
[07:10:48] <whym>	 Nirvanchik: you could request upgrading at jira.toolserver.org
[07:11:17] <Nirvanchik>	 whym: ok. But, if something, I can adapt my code to 1.6 anyway
[07:28:18] <tsnag>	 Load avg. on willow is OK: OK - load average: 11.25, 13.09, 14.95  
[07:36:18] <tsnag>	 s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1748.000000  
[07:42:50] <nosy>	 @replag all
[07:42:50] <tsbot>	 nosy: s1-rr-a: 2s [+0.00 s/s]; s1-rr-a-c: 4m 35s [-0.01 s/s]; s1-user: 2s [-0.09 s/s]; s2-user: 4d 11h 54m 45s [+0.50 s/s]; s2-user-c: 5s [+0.00 s/s]; s3-rr-a: 1m 34s [+0.00 s/s]; s3-user: 1m 34s [+0.00 s/s]; s4-rr-a: 5s [+0.00 s/s]
[07:42:51] <tsbot>	 nosy: s4-user: 5s [+0.00 s/s]; s5-rr-a: 2s [-0.00 s/s]; s5-user: 2s [-0.00 s/s]; s5-user-c: 5s [+0.00 s/s]; s6-rr-a: 1d 5h 25m 30s [+0.76 s/s]; s6-user: 1d 5h 25m 30s [+0.76 s/s]; s7-rr-a: 1s [-0.00 s/s]; s7-user: 1s [-0.00 s/s]
[07:45:18] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[07:45:27] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[07:45:27] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 102980  
[07:45:52] <msgbot>	 3(commented) [MNT-1192] s3 and s7 slaves were stopped <10https://jira.toolserver.org/browse/MNT-1192>  (Marlen Caemmerer)
[07:46:19] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[07:46:37] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[07:47:48] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[07:47:48] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[07:48:20] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[07:48:37] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[07:48:48] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[08:13:40] <Nirvanchik>	 s6 (fr/ja/ru.wikipedia.org): 1 days, 5 hours
[08:13:59] <Nirvanchik>	 what happened to s6? replication lag grows more and more
[08:14:45] <Nirvanchik>	 tsnag what about s6?
[08:16:17] <Nirvanchik>	 ok, I see: MySQL slave on z-dat-s6-a is CRITICAL
[08:17:00] <Nirvanchik>	 s6 has high replag because of shema-update
[08:18:05] <Nirvanchik>	 just to look a little lower, in status panel, haha:)
[08:40:22] <Daniel_WMDE>	 o_O
[08:40:39] <Daniel_WMDE>	 did sql-s2 just go away?
[08:41:57] <tsnag>	 MySQL on cassia is CRITICAL: Cant connect to MySQL server on cassia (146)  
[08:42:18] <tsnag>	 s5 replag on cassia is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on cassia (146)  
[08:42:18] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on cassia (146)  
[08:42:18] <tsnag>	 MySQL slave on cassia is CRITICAL: Cant connect to MySQL server on cassia (146)  
[08:43:57] <tsnag>	 MySQL on cassia is OK: Uptime: 253  Threads: 4  Questions: 42  Slow queries: 0  Opens: 29  Flush tables: 1  Open tables: 19  Queries per second avg: 0.166  
[08:44:17] <tsnag>	 MySQL slave on cassia is OK: Uptime: 273  Threads: 17  Questions: 236  Slow queries: 1  Opens: 79  Flush tables: 1  Open tables: 69  Queries per second avg: 0.864 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 272  
[08:44:26] <tsnag>	 s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 272.000000  
[08:44:37] <tsnag>	 s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 274.000000  
[08:45:18] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[08:45:27] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[08:45:37] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 103050  
[08:46:37] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[08:47:18] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[08:47:56] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[08:48:29] <Daniel_WMDE>	 nosy: are you poking around, or did sql-s2 just restart itself? seems to be read-only...
[08:48:37] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[08:48:47] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[08:49:18] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[08:49:47] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[08:50:44] <Daniel_WMDE>	 o_O
[08:52:18] <tsnag>	 MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146)  
[08:52:27] <tsnag>	 MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146)  
[08:52:37] <Daniel_WMDE>	 ...same thing just happened with sql-s3: connection lost in the middle of an explain statement ----^
[08:52:50] <Daniel_WMDE>	 am i hitting a mysql bug, crashing the server?
[09:08:56] <tsnag>	 /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36844 MB (9% inode=99%):  
[09:13:27] <tsnag>	 MySQL slave on z-dat-s3-a is OK: Uptime: 1411  Threads: 19  Questions: 143  Slow queries: 1  Opens: 68  Flush tables: 1  Open tables: 47  Queries per second avg: 0.101 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1414  
[09:13:57] <tsnag>	 /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22129 MB (5% inode=99%):  
[09:14:18] <tsnag>	 MySQL on z-dat-s3-a is OK: Uptime: 1456  Threads: 18  Questions: 11584  Slow queries: 6  Opens: 268  Flush tables: 1  Open tables: 245  Queries per second avg: 7.956  
[09:16:57] <tsnag>	 /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 81137 MB (20% inode=99%):  
[09:23:27] <tsnag>	 MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1912  
[09:30:27] <tsnag>	 MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146)  
[09:33:18] <tsnag>	 MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146)  
[09:37:53] <msgbot>	 3(created) [TS-1318] EXPLAIN involving a temporary table gets me locked out of mysql; Toolserver: Databases; Bug <10https://jira.toolserver.org/browse/TS-1318>  (Daniel Kinzler)
[09:39:57] <msgbot>	 3(commented) [TS-1318] EXPLAIN involving a temporary table gets me locked out of mysql <10https://jira.toolserver.org/browse/TS-1318>  (Daniel Kinzler)
[09:41:59] <msgbot>	 3(commented) [TS-1318] EXPLAIN involving a temporary table gets me locked out of mysql <10https://jira.toolserver.org/browse/TS-1318>  (Daniel Kinzler)
[09:45:18] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[09:45:37] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[09:45:38] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 106656  
[09:46:37] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[09:47:17] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[09:47:57] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[09:48:37] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[09:48:57] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[09:49:57] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[09:50:18] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[09:54:17] <tsnag>	 MySQL on z-dat-s3-a is OK: Uptime: 1439  Threads: 19  Questions: 2678  Slow queries: 0  Opens: 134  Flush tables: 1  Open tables: 122  Queries per second avg: 1.861  
[09:54:27] <tsnag>	 MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3307  
[09:54:58] <msgbot>	 3(commented) [TS-1318] EXPLAIN involving a temporary table gets me locked out of mysql <10https://jira.toolserver.org/browse/TS-1318>  (Daniel Kinzler)
[10:00:27] <tsnag>	 MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3607  
[10:03:19] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 21.03, 16.27, 13.13  
[10:06:17] <tsnag>	 Load avg. on willow is OK: OK - load average: 12.23, 14.56, 13.02  
[10:10:27] <tsnag>	 MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3531  
[10:18:53] <msgbot>	 3(assigned) [TS-1318] EXPLAIN involving a temporary table gets me locked out of mysql <10https://jira.toolserver.org/browse/TS-1318>  (Marlen Caemmerer)
[10:27:26] <tsnag>	 MySQL slave on z-dat-s3-a is OK: Uptime: 3434  Threads: 18  Questions: 2006052  Slow queries: 109  Opens: 16414  Flush tables: 1  Open tables: 9409  Queries per second avg: 584.173 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1630  
[10:36:59] <msgbot>	 3(commented) [TS-1318] EXPLAIN involving a temporary table gets me locked out of mysql <10https://jira.toolserver.org/browse/TS-1318>  (Marlen Caemmerer)
[10:45:17] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 16.31, 13.63, 12.11  
[10:45:47] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 110260  
[10:46:17] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[10:46:36] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[10:46:47] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[10:47:18] <tsnag>	 Load avg. on willow is OK: OK - load average: 13.72, 13.86, 12.40  
[10:47:18] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[10:47:57] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[10:48:47] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[10:48:58] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[10:49:57] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[10:51:18] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[10:52:22] <Nemo_bis>	 I don't understand how to use https://wiki.toolserver.org/view/SGE to execute a python script which needs to operate in my home directory and to be passed some parameters
[10:53:07] <Nemo_bis>	 I wrapped it in a shell script or in a shell script with screen as suggested in the docs, played with some other parameters etc., but it just dies after a few seconds without doing anything
[10:54:34] <Nemo_bis>	  Wallclock Time   = 00:00:02
[10:54:34] <Nemo_bis>	  CPU              = 00:00:00
[10:54:34] <Nemo_bis>	  Max vmem         = 16.648M
[10:54:34] <Nemo_bis>	  Exit Status      = 0
[10:56:19] <wikirc_>	 [[Job scheduling]] !M 10https://wiki.toolserver.org/w/index.php?diff=6718&oldid=6030&rcid=8873 * Nemobis * (+1) (/* Binaries */ typo)
[11:45:47] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 113860  
[11:46:18] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[11:46:37] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[11:46:46] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[11:47:18] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[11:47:57] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[11:48:48] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[11:49:57] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[11:49:57] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[11:52:18] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[12:36:57] <tsnag>	 /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39383 MB (9% inode=99%):  
[12:40:46] <Merlissimo>	 @replag
[12:40:46] <tsbot>	 Merlissimo: s1-rr-a: 13s [+0.00 s/s]; s1-user: 5m 46s [+0.02 s/s]; s2-user: 4d 13h 15m 18s [+0.27 s/s]; s3-rr-a: 57s [-0.00 s/s]; s3-user: 57s [-0.00 s/s]; s6-rr-a: 1d 9h 28m 6s [+0.81 s/s]; s6-user: 1d 9h 28m 6s [+0.81 s/s]
[12:43:48] <tsnag>	 /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 40127 MB (4% inode=99%):  
[12:44:56] <tsnag>	 /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23152 MB (5% inode=99%):  
[12:45:48] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 117460  
[12:46:18] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[12:46:47] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[12:47:18] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[12:47:37] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[12:47:57] <tsnag>	 /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 78443 MB (19% inode=99%):  
[12:48:07] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[12:48:47] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[12:50:06] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[12:50:06] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[12:52:18] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[13:18:24] <wikirc_>	 [[Job scheduling]] ! 10https://wiki.toolserver.org/w/index.php?diff=6719&oldid=6718&rcid=8874 * Nemobis * (+148) (/* Running jobs under screen */ debug)
[13:18:53] <msgbot>	 3(resolved) [TS-1318] EXPLAIN involving a temporary table gets me locked out of mysql <10https://jira.toolserver.org/browse/TS-1318>  (DaB.)
[13:24:05] <DaBPunkt>	 hello all
[13:24:12] * Damianz  waves

[13:36:29] <DaBPunkt>	 @replag
[13:36:29] <tsbot>	 DaBPunkt: s1-user: 1m 12s [-0.08 s/s]; s2-user: 4d 13h 12m 48s [-0.04 s/s]; s3-rr-a: 28s [-0.01 s/s]; s3-user: 28s [-0.01 s/s]; s6-rr-a: 1d 10h 23m 49s [+1.00 s/s]; s6-user: 1d 10h 23m 49s [+1.00 s/s]; s7-rr-a: 14s [+0.00 s/s]; s7-user: 14s [+0.00 s/s]
[13:46:19] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[13:46:47] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 121120  
[13:47:18] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[13:47:47] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[13:47:47] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[13:48:17] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[13:48:47] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[13:50:07] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[13:50:07] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[13:52:18] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[14:46:18] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[14:46:47] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 103762  
[14:47:47] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[14:47:47] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[14:48:18] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[14:48:18] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[14:48:48] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[14:50:06] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[14:50:06] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[14:52:18] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[15:02:18] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 15.04, 12.59, 11.01  
[15:03:19] <tsnag>	 Load avg. on willow is OK: OK - load average: 14.85, 12.94, 11.23  
[15:46:27] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[15:46:48] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 89622  
[15:47:48] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[15:47:48] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[15:48:27] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[15:48:48] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[15:49:17] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[15:50:07] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[15:50:07] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[15:50:47] <Daniel_WMDE>	 nosy: moin
[15:51:12] <nosy>	 Daniel_WMDE: moin
[15:52:27] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[15:52:40] <Daniel_WMDE>	 re TS-1318: i'd like to experiment some more before filing a bug with mysql. but i don't want to kill the mysqld all the time...
[15:52:48] <Daniel_WMDE>	 got an idea where/how to do that?
[15:53:20] <Daniel_WMDE>	 also: do we somewhere have documentation of where to find and how to access each database, as i mention in the bug report?
[15:54:29] <DaBPunkt>	 Daniel_WMDE: https://wiki.toolserver.org/view/Database_access
[15:54:38] <DaBPunkt>	 it also works with ssh
[15:55:30] <Daniel_WMDE>	 DaBPunkt: uh, does it? when i ssh sql-s2, don't get to a database server at all, i end up on the HA cluster.
[15:55:46] <nosy>	 i am afraid more documentation regarding host-database-relation wont be easy to maintain
[15:55:51] <nosy>	 thats true
[15:55:55] <Daniel_WMDE>	 so...
[15:56:01] <Daniel_WMDE>	 how do i find out which server to go to?
[15:56:13] <nosy>	 listing the hostname in the variable is the only way i can think of
[15:56:17] <Daniel_WMDE>	 and yea, that's why i suggested smq for this
[15:56:20] <DaBPunkt>	 rdab@hemlock:~$ ssh sql-s2.db.toolserver.org
[15:56:21] <DaBPunkt>	 [..]
[15:56:23] <DaBPunkt>	 rdab@damiana:/$
[15:56:33] <nosy>	 smw swm smq? 
[15:56:37] <nosy>	 wtf
[15:56:39] <nosy>	 ;)
[15:56:48] <Daniel_WMDE>	 semantic mediawiki
[15:56:51] <DaBPunkt>	 mm, wait a moment
[15:57:01] <Daniel_WMDE>	 damit kann man solche relationen im wiki direkt pflegen
[15:57:02] <Daniel_WMDE>	 naja
[15:57:25] <nosy>	 aber man muss es aktualisieren
[15:57:30] <nosy>	 ;)
[15:57:32] <Daniel_WMDE>	 ja, aber nur an einem ort
[15:57:37] <Daniel_WMDE>	 nicht auf x seiten
[15:57:39] <Daniel_WMDE>	 egal
[15:57:43] <Daniel_WMDE>	 welche variable meinst du denn?
[15:57:49] <nosy>	 moment
[15:57:55] <Daniel_WMDE>	 @@hoistname oder sowas?
[15:58:04] <Daniel_WMDE>	 geht nicht wenn der server  down ist :)
[15:58:15] <DaBPunkt>	 Daniel_WMDE: normalerweise SOLLEN die user gar nicht wisen, wo sie landen
[15:58:38] <DaBPunkt>	 sie sollen möglichst abstrakte Verweise benutzen
[15:58:46] <Daniel_WMDE>	 DaBPunkt: richtig. aber *ich* und *ihr*. wir müssen ja irgendwie rauskriegen können, auf welche kiste man gucken muss, wenn sql-s2 down ist
[15:59:00] <Daniel_WMDE>	 DaBPunkt: d.h. *irgendwo* muss das dokumentiert sein
[15:59:05] <Merlissimo>	 dns?
[15:59:12] <Daniel_WMDE>	 ich sehe aber gerade keine vernünftige möglichkeit, an diese info zu kommen
[15:59:27] <nosy>	 show variables like "hostname";
[15:59:28] <Daniel_WMDE>	 Merlissimo: nein. da ist 'nen proxy dazwischen. dns gibt mir nur die service-ip, das bringt mir nichts
[15:59:37] <Daniel_WMDE>	 (das ist damiana, so weit waren wir schon)
[15:59:40] <DaBPunkt>	 Daniel_WMDE: wir wissen es. Und wenn wir es nicht wissen, dann schauen wir in die Config des haproxies
[15:59:57] <nosy>	 Daniel_WMDE: show variables like "hostname";
[16:00:19] <Daniel_WMDE>	 nosy: das ist super solange mysql nicht gerade3 down ist. und dass es down war war ja der grund dass ich ans error log wollte :)
[16:00:29] <nosy>	 ok
[16:00:37] <DaBPunkt>	 sobald wir anfangen, eine Wiki-Seite zu pflegen,w erden die User anfangen sie zu benutzen…
[16:00:47] <nosy>	 lol
[16:00:54] <Daniel_WMDE>	 oh gott nein! user die was nutzen!"
[16:01:04] <nosy>	 inakzeptabel
[16:01:05] <Daniel_WMDE>	 DaBPunkt: ich weiß was du meinst
[16:01:07] <Merlissimo>	 nagios meldet doch, welcher server weg ist
[16:01:23] <DaBPunkt>	 Daniel_WMDE: wenn du dann die sources der user nachpflegst ;)
[16:01:28] <Daniel_WMDE>	 Merlissimo: tsnag hat zu der zeit *sehr* viele server gemeldet, das war entwas unübersichtlich
[16:01:29] <Merlissimo>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)
[16:01:46] <Daniel_WMDE>	 DaBPunkt: nee, selber schuld wenn man keinen alias verwendet
[16:01:53] <DaBPunkt>	 Merlissimo: BTW: Wie weit bist du mit dem Read-only-Nagios-Warning?
[16:01:56] <Daniel_WMDE>	 DaBPunkt: aber wegen mir muss das auch keine wiki-seite sein.
[16:02:17] <Daniel_WMDE>	 'ne dateiu in ~root. 'nen script "dbinfo". irgend was
[16:02:29] <Merlissimo>	 DaBPunkt: noch gar nicht, weil ich sge doku priorität gegeben habe
[16:02:45] <DaBPunkt>	 Daniel_WMDE: /global/misc/haproxy/haproxy.conf auf dem HA-nodes zeigt dir, was du wissen möchtest
[16:03:10] <nosy>	 Daniel_WMDE: oder ruf mich einfach an
[16:03:23] <DaBPunkt>	 Merlissimo: ah, ok :-)
[16:03:30] <Daniel_WMDE>	 Merlissimo: du hast recht. war mir aber nicht klar, dass ich auf die replag-warnung gucken muss
[16:03:52] <Daniel_WMDE>	 DaBPunkt: muss ich mir das jetzt merken?...
[16:04:05] <Daniel_WMDE>	 nosy: mach ich :)
[16:04:19] <nosy>	 :)
[16:04:19] <DaBPunkt>	 Daniel_WMDE: oder aufschreiben ;) (z.B. isn Wiki…)
[16:04:24] <Daniel_WMDE>	 DaBPunkt, nosy: was meint ihr, wo kann ich diesen bug am besten testen?
[16:04:39] <nosy>	 naja, daphne ist momentan nicht in produktion
[16:04:49] <DaBPunkt>	 Daniel_WMDE: du könntest dir daphne schnappen
[16:04:56] <nosy>	 DaBPunkt: ich wollte irgendwie die alten DBs wegwerfen und komplett neu aufbauen
[16:05:08] <nosy>	 sollen dann wieder perspektivisch s2 und s5 drauf?
[16:05:16] <nosy>	 oder was meinst du?
[16:05:23] <DaBPunkt>	 nosy: eher nur s5
[16:05:26] <DaBPunkt>	 und commons
[16:05:57] <DaBPunkt>	 nosy: aber ich dachte, du wolltest da das SAN-Setup testen?
[16:06:07] <nosy>	 DaBPunkt: priorisieren sie
[16:06:15] <nosy>	 also Daniel will jetzt erstmal testen
[16:06:37] <nosy>	 wir muessten alles wegwerfen und ne neue mysql-db anlegen und dann gehts wieder los
[16:06:58] <DaBPunkt>	 los?
[16:07:08] <nosy>	 na los mit der mysql auf daphne
[16:07:18] <DaBPunkt>	 ja
[16:07:21] <nosy>	 die startet ja sonst gar nicht richtig
[16:07:49] <DaBPunkt>	 ich setzte es schnell neu auf
[16:08:15] <Daniel_WMDE>	 DaBPunkt, nosy: ok, ich spiel jetzt mal 'ne stunde mit daphne.
[16:08:40] <nosy>	 Daniel_WMDE: machst du die mysql zuerst komplett neu?
[16:08:53] <DaBPunkt>	 nosy: ich mach schon
[16:08:58] <nosy>	 ja ok
[16:09:46] <Daniel_WMDE>	 danke
[16:10:51] <DaBPunkt>	 wird ein par Minuten dauern. Ich möchte gleich die Partion auf /sql einhängen, damit es einheitlich ist
[16:12:49] <Daniel_WMDE>	 ok, sag dann bescheid, bitte
[16:13:05] <Daniel_WMDE>	 ich guck derweil mal, ob ich's bei mir lokal auch reproduziert krieg
[16:19:37] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/7:DOWN:1 UP: OK  
[16:22:21] <Daniel_WMDE>	 nosy: gibt's nen core dump?
[16:23:37] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[16:32:14] <nosy>	 Daniel_WMDE: ich kann ihn nicht finden, obwohl das log ja sagt, es gaebe core
[16:35:21] <Daniel_WMDE>	 hrmpf
[16:35:35] <Daniel_WMDE>	 DaBPunkt: wie steht's mit daphne?
[16:35:42] <Daniel_WMDE>	 ich kann das problem bei mir lokal nicht reproduzieren.
[16:39:17] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 15.71, 13.59, 11.36  
[16:41:49] <DaBPunkt>	 Daniel_WMDE: mysql crasht sofort wieder, wenn man es startet
[16:42:17] <Daniel_WMDE>	 gah.
[16:42:47] <Daniel_WMDE>	 DaBPunkt: dann lag das probelm evtl etwas tiefer? kaputte daten?
[16:43:08] <Daniel_WMDE>	 aber gleich in zwei datenbanken, auf zwei servern?....
[16:43:15] <DaBPunkt>	 strange. daphne versucht tabellen zu reparieren, die es schon lange nicht mehr gibt…
[16:43:18] <tsnag>	 Load avg. on willow is OK: OK - load average: 14.57, 14.68, 12.41  
[16:43:25] <DaBPunkt>	 ich such weiter
[16:43:48] <Daniel_WMDE>	 DaBPunkt: ich hatte das problem erst mit sql-s2. der kam aber wieder hoch. dann mit sql-s3
[16:45:04] <DaBPunkt>	 Daniel_WMDE: egal was du tust: Bitte last s6 in Ruhe ;)
[16:45:34] <Daniel_WMDE>	 hehe
[16:46:19] <nosy>	 DaBPunkt: du musst den ganzen mysql-kram komplett loeschen und dann mit mysql_install_db alles neu machen
[16:46:28] <DaBPunkt>	 nosy: das hab ich schon!
[16:46:34] <nosy>	 oh
[16:46:39] <nosy>	 die innodb-files auch?
[16:46:45] <DaBPunkt>	 alles#
[16:46:48] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 68003  
[16:46:54] <Daniel_WMDE>	 ihr wollt die kiste komplett neu importieren?
[16:47:11] <nosy>	 haben wir eine wahl?
[16:47:16] <Daniel_WMDE>	 keine ahnung :)
[16:47:33] * Daniel_WMDE  wollte nur ganz unschulding 'ne query optimieren...

[16:47:35] <nosy>	 DaBPunkt guck mal in root@daphne:/mysql_fix
[16:47:37] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[16:47:47] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[16:47:55] <nosy>	 Daniel_WMDE: wir muessen das teil eh reparieren
[16:48:04] <DaBPunkt>	 nosy: was ist dort?
[16:48:06] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[16:48:28] <nosy>	 na innodb_data_file_path = /mysql_fix/innodb/ibdata1:50M:autoextend
[16:48:33] <nosy>	 ich hatte das ausgelagert
[16:48:38] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[16:48:41] <DaBPunkt>	 ah nosy…
[16:48:41] <nosy>	 wg zu wenig platz
[16:48:48] <DaBPunkt>	 *seufzÜ*
[16:49:07] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146)  
[16:49:17] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[16:50:07] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[16:50:07] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[16:54:37] <DaBPunkt>	 Daniel_WMDE: ok, du kannst jetzt
[16:55:05] <Daniel_WMDE>	 da binn ich ja mal gespannt, was ich genau kann
[16:56:15] <Daniel_WMDE>	 DaBPunkt:  Access denied for user 'root'@'willow.toolserver.org' (using password: YES)
[16:56:54] <Daniel_WMDE>	 ah, von localhost geht
[16:57:06] <DaBPunkt>	 mm, ihc hatte es von hemlock aus getestet…
[16:57:38] <Daniel_WMDE>	 evtl hatte ich mich auch nur vertippt.
[16:57:51] <DaBPunkt>	 Daniel_WMDE: nein, bei mir ging es auch nicht
[16:57:52] <Daniel_WMDE>	 ich überleg gerade, was ich da jetzt zum testen importiere
[16:58:07] <DaBPunkt>	 eigendlich soll von den userland-clients auch gar kein root-zugriff gehen
[16:58:25] <DaBPunkt>	 (in dem Fall ist es natürlich egal, weil keine daten auf daphne sind)
[16:58:37] <Daniel_WMDE>	 ich bin direkt als root drin
[16:58:54] <DaBPunkt>	 Daniel_WMDE: ja, andere User sind auch nicht angelegt
[16:59:12] <Daniel_WMDE>	 nein, ich meine, ich hab mich via ssh als root angemeldet
[16:59:19] <Daniel_WMDE>	 rdaniel geht nicht auf willow.
[16:59:53] <Daniel_WMDE>	 (fragt nach 'ne password statt nach der passphrase für den schlüssel)
[17:00:08] <DaBPunkt>	 Daniel_WMDE: du sollst ja auch hemlock benutzen ;)
[17:00:27] <Daniel_WMDE>	 kann sich doch kein mensch merken sowas
[17:00:45] <Daniel_WMDE>	 hemlock geht
[17:00:47] <Daniel_WMDE>	 naja
[17:01:00] <DaBPunkt>	 Daniel_WMDE: nun, wir versuchen es den usern einfach zu machen, nicht den root  ;)
[17:07:36] * DaBPunkt  macht nun Abendessen. Bis später

[17:10:13] <Daniel_WMDE>	 DaBPunkt, nosy: also, die selben queries gehen, wenn ich das auf 'ner leeren datenbank als root mache
[17:10:22] <Daniel_WMDE>	 also hängt entweder an den permissions oder an den daten
[17:10:34] <Daniel_WMDE>	 ist hanz schön schwer, das zu reproduzieren :(
[17:10:47] <Daniel_WMDE>	 ...und hier ist jetzt auch abendessen angesagt
[17:24:37] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[17:35:13] <Quentinv57>	 hello DaBPunkt :)
[17:35:33] <Quentinv57>	 do you have two minutes ? I need to disturb you... :P
[17:40:43] <Danny_B|backup>	 don't ask to disturb, just disturb...
[17:43:58] <Quentinv57>	 :D
[17:44:16] <Quentinv57>	 apparently, he's not here Danny_B|backup so I prefer to wait :)
[17:47:47] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[17:47:48] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27713  
[17:48:08] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES)  
[17:48:27] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[17:48:38] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[17:49:07] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES)  
[17:49:18] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[17:50:07] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[17:50:07] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[18:02:18] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 15.19, 14.26, 12.41  
[18:03:18] <tsnag>	 Load avg. on willow is OK: OK - load average: 14.38, 14.15, 12.49  
[18:21:54] <msgbot>	 3(commented) [DBQ-144] Query to create translated interlanguage links <10https://jira.toolserver.org/browse/DBQ-144>  (Stavros Giannou)
[18:25:38] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[18:33:37] <tsnag>	 s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1905.000000  
[18:39:28] <AMadman>	 Is something up with e-mail delivery? I've sent madman@toolserver.org a couple e-mails and none have gone through.
[18:41:18] <AMadman>	 Of course right when I say that, I get one of about seven test messages delivered.
[18:41:31] <AMadman>	 Presumably it was a problem with my .forward file, though there wasn't one before.
[18:47:47] <tsnag>	 MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12478  
[18:47:47] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[18:48:08] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES)  
[18:48:37] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[18:49:07] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES)  
[18:49:37] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[18:50:07] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[18:50:07] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[18:50:18] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[18:54:15] <AMadman>	 Oh, I know what I did wrong (I'm an idiot). Cheers!
[19:08:45] <DaBPunkt>	 @replag
[19:08:45] <tsbot>	 DaBPunkt: s1-rr-a-c: 50m 16s [+0.07 s/s]; s2-user: 4d 13h 43m 59s [+0.09 s/s]; s2-user-c: 19m 17s [+0.03 s/s]; s4-rr-a: 8m 46s [+0.01 s/s]; s4-user: 8m 46s [+0.01 s/s]; s5-user-c: 19m 17s [+0.03 s/s]; s6-rr-a: 1h 15m 57s [-5.98 s/s]; s6-user: 1h 15m 57s [-5.98 s/s]
[19:10:48] <tsnag>	 MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3555  
[19:19:38] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.475586/1.00, alarm hl:np_load_long=0.800781/1.50, alarm hl:mem_free=21855.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.475586/1.10, alarm hl:np_load_long=0.800781/1.50, alarm hl:mem_free=21855.000000M/300M, alarm hl:available=1/0  
[19:19:48] <tsnag>	 MySQL slave on z-dat-s6-a is OK: Uptime: 1820006  Threads: 6  Questions: 215611772  Slow queries: 109149  Opens: 2341793  Flush tables: 3  Open tables: 1865  Queries per second avg: 118.467 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1712  
[19:20:37] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK  
[19:25:38] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[19:33:38] <tsnag>	 s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3327.000000  
[19:47:17] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 16.66, 15.26, 14.07  
[19:47:48] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[19:48:08] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES)  
[19:48:37] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[19:49:38] <tsnag>	 s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3622.000000  
[19:49:49] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[19:50:07] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES)  
[19:50:07] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[19:50:07] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[19:51:17] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[19:52:17] <tsnag>	 Load avg. on willow is OK: OK - load average: 14.01, 14.97, 14.33  
[20:01:38] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.519531/1.00, alarm hl:np_load_long=0.681640/1.50, alarm hl:mem_free=21473.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.519531/1.10, alarm hl:np_load_long=0.681640/1.50, alarm hl:mem_free=21473.000000M/300M, alarm hl:available=1/0  
[20:02:18] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 20.15, 15.82, 14.55  
[20:02:37] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK  
[20:24:18] <tsnag>	 Load avg. on willow is OK: OK - load average: 12.70, 14.42, 15.00  
[20:25:48] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[20:47:28] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 17.09, 15.83, 15.08  
[20:47:57] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[20:48:07] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES)  
[20:48:37] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[20:49:37] <tsnag>	 s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5621.000000  
[20:50:08] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES)  
[20:50:17] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[20:50:18] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[20:50:58] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[20:51:27] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[20:54:28] <tsnag>	 Load avg. on willow is OK: OK - load average: 13.52, 14.69, 14.89  
[21:01:09] <wikirc_>	 [[Special:Log/newusers]] create 10 * Danhash *  (New user account)
[21:02:27] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 17.95, 17.08, 15.81  
[21:03:40] <wikirc_>	 [[User:Danhash]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6720&rcid=8876 * Danhash * (+28) (redirect to [[w:user:danhash|my wikipedia user page]])
[21:07:03] <Platonides>	 I seem to have the same problem as tsnag
[21:07:19] <Platonides>	 running the same script, I sometimes get Access denied for user 'platonides'@'damiana-ext-bge0.toolserver.org'
[21:07:22] <Platonides>	 and it sometimes works
[21:12:26] <wikirc_>	 [[User:Danhash]] ! 10https://wiki.toolserver.org/w/index.php?diff=6721&oldid=6720&rcid=8877 * Danhash * (+4) (change to soft redirect)
[21:18:33] <DarkoNeko>	 nighty~
[21:26:42] <Krinkle>	 ceradon:  http://toolserver.org/~krinkle/getTopRCusers.php works fine here
[21:26:46] <Krinkle>	 what error are you see in ?
[21:26:47] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[21:26:49] <Krinkle>	 seeing*
[21:27:00] <ceradon>	 dbConnect: ERROR: \nAccess denied for user 'krinkle'@'damiana-ext-bge0.toolserver.org' (using password: YES)
[21:27:45] <Krinkle>	 which wiki ?
[21:27:56] <Krinkle>	 commons ?
[21:28:36] <ceradon>	 yup
[21:30:06] <Krinkle>	 works here..
[21:30:11] <Krinkle>	 refresh ?
[21:30:24] <Krinkle>	 Sorry, I.. don't get the error. It looks wrong for sure
[21:30:36] <Krinkle>	 sounds like a problem at the Toolserver though
[21:30:39] <Krinkle>	 @replag
[21:30:40] <tsbot>	 Krinkle: s1-rr-a-c: 2h 2m 34s [+0.51 s/s]; s2-user: 4d 13h 32m 42s [-0.08 s/s]; s2-user-c: 20m 19s [+0.01 s/s]; s4-rr-a: 8m 47s [+0.00 s/s]; s4-user: 8m 47s [+0.00 s/s]; s5-user-c: 20m 19s [+0.01 s/s]
[21:30:48] <Krinkle>	 http://status.toolserver.org/
[21:30:52] <Krinkle>	 looks god
[21:30:53] <Krinkle>	 good
[21:38:12] <Krinkle>	 ceradon: Ah, now I'm getting it too
[21:38:14] <Krinkle>	 it's somewhat reandom
[21:38:15] <Krinkle>	 random
[21:38:40] <Krinkle>	 ceradon: http://wolfsbane.toolserver.org/~krinkle/getTopRCusers.php
[21:38:47] <Krinkle>	 does that work ? (http and wolfsbane)
[21:38:54] <Krinkle>	 this is a debug url, don't spread or save anywhere
[21:40:22] <ceradon>	 I got it, thanks
[21:41:05] <ceradon>	 the debug url is working but the public one is still down on my computer.
[21:41:28] <Platonides>	 is it this error? dbConnect: ERROR: Access denied for user 'krinkle'@'damiana-ext-bge0.toolserver.org' (using password: YES)
[21:41:32] <Krinkle>	 yes
[21:41:36] <Platonides>	 I mentioned it a bit ago
[21:41:51] <Platonides>	 I was having it 'randomly' on a script
[21:41:57] <Krinkle>	 tsnag is having it as well
[21:42:02] <Krinkle>	 <tsnag> s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES) 
[21:42:03] <Platonides>	 yes
[21:42:20] <Krinkle>	 Looks like there is a bad mysql slave in the mix
[21:42:21] <Platonides>	 I suspect it's either one server and load balancing making it random
[21:42:26] <Krinkle>	 yeah
[21:42:39] <Krinkle>	 DaBPunkt: 
[21:42:39] <Platonides>	 or that they all go to damiana and she sometimes gets too flooded
[21:48:07] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES)  
[21:48:07] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[21:48:37] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[21:49:37] <tsnag>	 s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7757.000000  
[21:50:07] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES)  
[21:50:18] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[21:50:18] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[21:51:16] <Merlissimo>	 Krinkle: daphne is down, thats why nagios cannot connect. but s2/s6 are available on cassia, so your error isn't related
[21:51:27] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[21:51:47] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[21:55:07] <Krinkle>	 Merlissimo: The load balancer knows that?
[21:55:23] <Krinkle>	 Merlissimo: Is tsnag bypassing the lb ?
[21:55:40] <Merlissimo>	 which domain are you connectiong to?
[21:56:05] <Krinkle>	 {wiki-name}-p.rrdb.toolserver.org'
[21:56:14] <Merlissimo>	 tsnag runs on each sql server, so it's not using the proxy
[21:56:30] <Krinkle>	 connect fails, msyql_error() returns: ERROR: Access denied for user 'krinkle'@'damiana-ext-bge0.toolserver.org' (using password: YES)
[21:56:38] <Platonides>	 mysql -h sql-s4-rr
[21:57:47] <Platonides>	 that will _sometimes_ fail
[21:58:37] <Platonides>	 even connecting by ip to 10.24.1.107 may fail :/
[21:59:07] <Merlissimo>	 i am having the same error and even root, too
[21:59:31] <Platonides>	 maybe that's the bug which needed a reboot?
[22:00:20] <Merlissimo>	 i thought s4-rr is rosemary
[22:01:25] <Platonides>	 maybe daphne is the other server with s4?
[22:01:28] * Platonides  checks

[22:01:52] <Merlissimo>	 no daphne is the old s2/s5
[22:02:23] <Merlissimo>	 and will be s2/s5 again after it is resetuped
[22:02:51] <Platonides>	 s4: cassia and hyacinth
[22:03:27] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 16.16, 16.46, 16.02  
[22:21:53] <msgbot>	 3(commented) [DBQ-144] Query to create translated interlanguage links <10https://jira.toolserver.org/browse/DBQ-144>  (Hoo man)
[22:26:57] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[22:41:54] <msgbot>	 3(commented) [DBQ-144] Query to create translated interlanguage links <10https://jira.toolserver.org/browse/DBQ-144>  (Stavros Giannou)
[22:48:18] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[22:48:38] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[22:49:07] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES)  
[22:49:38] <tsnag>	 s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9174.000000  
[22:50:08] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES)  
[22:50:17] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[22:50:18] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[22:51:37] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[22:51:57] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[22:53:27] <cyberpower678>	 I am trying to set up a cron in willow however, when I go to save the cron, I get no changes have been made and it ends up not installing.  Is it possible I am doing something wrong?
[22:55:00] <Merlissimo>	 cyberpower678:  yes,there must be a syntax error if cron is not accepting the entry. Can you post the line here?
[22:56:35] <cyberpower678>	 Your account will expire on Wednesday, 01 August 2012. cyberpower678@willow:~$ cronie -e no crontab for cyberpower678 - using an empty one cronie: no changes made to crontab cyberpower678@willow:~$
[22:57:07] <cyberpower678>	 Merlissimo: Here is the syntax I used:
[22:57:16] <cyberpower678>	 Your account will expire on Wednesday, 01 August 2012.
[22:57:29] <cyberpower678>	 cyberpower678@willow:~$ cronie -e
[22:57:48] <cyberpower678>	 no crontab for cyberpower678 - using an empty one
[22:58:00] <cyberpower678>	 no changes made to crontab
[22:58:07] <cyberpower678>	 cyberpower678@willow:~$
[22:58:24] <Merlissimo>	 so no editor opens?
[22:58:31] <cyberpower678>	 it does.
[22:58:37] <cyberpower678>	 hold on a sec.
[22:59:45] <cyberpower678>	 This is the syntax:*/5 * * * * cronsub -s enwiki-sandbox.php  $HOME/cyberpower678/bots/enwiki-sandbox.php 16 4 * * * cronsub -s meta-daily.php  $HOME/cyberpower678/bots/meta-daily.php */3 * * * * cronsub -s chu.php $HOME/cyberpower678/bots/chu.php 34 6 */2 * * cronsub -s afd-remove.php  $HOME/cyberpower678/bots/afd-remove.php 5 11 * * * cronsub -s uncat-remove.php  $HOME/cyberpower678/bots/uncat-remove.php 41 22 *
[23:00:26] <tsnag>	 Load avg. on willow is CRITICAL: CRITICAL - load average: 37.54, 19.60, 16.90  
[23:00:51] <cyberpower678>	 ?
[23:01:27] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 23.51, 18.89, 16.83  
[23:02:10] <cyberpower678>	 I still don't get it.  What does that mean?
[23:02:39] <Merlissimo>	 the first lines look ok, but "41 22 *" is wrong of course
[23:02:59] <Merlissimo>	 btw, you should install your cron on host submit, not on willow
[23:03:55] <cyberpower678>	 What's wrong with 41 22 */2 * *
[23:05:01] <Merlissimo>	 i think you post was cut
[23:05:09] <Merlissimo>	 by freenode
[23:05:31] <cyberpower678>	 Here's the remaining post: $HOME/cyberpower678/bots/uncat-remove.php 41 22 */2 * * cronsub -s bad-image.php  $HOME/cyberpower678/
[23:06:28] <Merlissimo>	 maybe wrong line delimiter?
[23:06:58] <Merlissimo>	 (i am only guessing)
[23:07:06] <cyberpower678>	 After I typed that syntax in, I pressed Ctrl+Z and hit y to save.  I was using cronie.  When it asked me for the directory, I placed it in my own page naming it .crontab.
[23:08:27] <Merlissimo>	 no, there should be already a default temporary name
[23:08:52] <msgbot>	 3(commented) [DBQ-144] Query to create translated interlanguage links <10https://jira.toolserver.org/browse/DBQ-144>  (Stavros Giannou)
[23:09:28] <cyberpower678>	 Then I get another error saying unable to install.
[23:09:36] <Merlissimo>	 something like  /tmp/crontab.XXXXXXXXXX
[23:11:27] <DaBPunkt>	 re
[23:11:43] <Merlissimo>	 DaBPunkt: sql-s4-rr is not working
[23:13:31] <DaBPunkt>	 should be fixed
[23:14:11] <Merlissimo>	 jepp, i getting rosemary only now.
[23:15:27] <cyberpower678>	 Should I try it again now with the default name?
[23:15:59] <Merlissimo>	 yes
[23:16:27] <cyberpower678>	 Alright.
[23:16:37] <cyberpower678>	 Activated editor.
[23:17:11] <Merlissimo>	 cron is making a local copy and open this using the defined editor and after you have edited is coping it back to the main crontab directory. that's why you have to save it to the prefiled temporary file
[23:18:00] <Merlissimo>	 but you should install it on host submit, not on willow, so that you cron is always executed even if e.g. willow is down
[23:20:48] <cyberpower678>	 I can always remove the cron when necessary.  Right now my main focus is getting the bot operational.
[23:22:27] <tsnag>	 Load avg. on willow is OK: OK - load average: 11.85, 13.09, 14.92  
[23:25:13] <cyberpower678>	 no crontab for cyberpower678 - using an empty one
[23:25:23] <cyberpower678>	 cronie: installing new crontab
[23:25:35] <cyberpower678>	 "/tmp/crontab.XXXXlFa4YR":2: bad minute
[23:25:47] <cyberpower678>	 errors in crontab file, can't install.
[23:26:06] <cyberpower678>	 Do you want to retry the same edit?
[23:26:17] <cyberpower678>	 No Go.
[23:26:58] <tsnag>	 FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL  
[23:27:27] <cyberpower678>	 What is this ^?
[23:27:33] <Merlissimo>	 cann you pastbin your crontab again? e.g. http://kpaste.net/
[23:28:22] <cyberpower678>	 You mean try again?
[23:28:33] <cyberpower678>	 I'm not sure I'm following.
[23:29:45] <Joan>	 Are you using cron or cronie?
[23:29:52] <Joan>	 cron doesn't support /2 syntax.
[23:29:53] <Platonides>	 cyberpower678, you may have more luck copying the cron in your own file, modifying there, and then installing that file
[23:30:36] <Joan>	 I've always found it's easiest to do cron shit in a text file like... crontab.txt. And then sync it as necessary using "crontab crontab.txt".
[23:30:44] <Joan>	 That way it gets backed up as well.
[23:31:19] <Merlissimo>	 Joan: at the post above he/she used cronie
[23:31:35] <Joan>	 Plus, if you do it in a text file, you don't have to worry about nano inserting stupid line breaks and you can edit it over SFTP or whatever.
[23:31:38] <Joan>	 It's great.
[23:31:43] <Joan>	 Merlissimo: Yeah, I see in scrollback now. :-)
[23:32:15] <cyberpower678>	 How do I manually install?
[23:32:17] <Joan>	 Based on how the lines were pasted in here, I'd almost be inclined to guess it's a line-endings problem. But that'd be unusual.
[23:34:40] <cyberpower678>	 How do I manually install crontab?
[23:35:07] <tsnag>	 /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37854 MB (9% inode=99%):  
[23:36:14] <Joan>	 cyberpower678: You can do what I described above.
[23:37:13] <cyberpower678>	 crontab crontab.txt where is it installed?
[23:41:08] <tsnag>	 /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23174 MB (5% inode=99%):  
[23:42:38] <cyberpower678>	 crontab crontab.txt.  Where and how do you use this command.  I can't open the file.
[23:43:10] <DaBPunkt>	 Joan: the question is for you AFAIS
[23:46:00] <wikirc_>	 [[Special:Log/newusers]] create 10 * Ueberwachung *  (New user account)
[23:46:20] <Joan>	 cyberpower678: You make a file called "crontab.txt" with your cronjobs in it.
[23:46:30] <cyberpower678>	 Did that.
[23:46:35] <Joan>	 Then sync it using "crontab crontab.txt" (assuming your in the same directory).
[23:46:44] <Joan>	 Or "crontab ~/path/to/crontab.txt".
[23:46:56] <Joan>	 Then you can use "crontab -l" to see your current crontab.
[23:47:04] <Merlissimo>	 Joan: crontab will not accept */2, thats why he used correctly cronie before
[23:47:13] <Joan>	 Oh, right.
[23:47:16] <Joan>	 Then same principle.
[23:47:20] <Joan>	 "cronie crontab.txt"
[23:47:46] <Joan>	 "crontab -r" should fix anything there.
[23:48:07] <tsnag>	 /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 74579 MB (18% inode=99%):  
[23:48:27] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[23:48:28] <Joan>	 I always found it bizarre that that didn't throw a warning.
[23:48:43] <Joan>	 I think there's some option that will throw one now. Still, should be -rf without a warning, surely.
[23:48:47] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[23:49:06] <tsnag>	 s2 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES)  
[23:49:27] <cyberpower678>	 No such file or directory
[23:49:33] <cyberpower678>	 Error: No such file or directory
[23:49:51] <cyberpower678>	 How do I type in the file directory?
[23:49:55] <Joan>	 Which is also why it's nice to have backups of the crontab.txt file. ;-)
[23:50:02] <Joan>	 cyberpower678: You're asking how to type?
[23:50:07] <tsnag>	 s5 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-ext-bge0.toolserver.org (using password: YES)  
[23:50:10] <cyberpower678>	 No.
[23:50:13] <Joan>	 Are you new to SSH/terminals?
[23:50:17] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[23:50:27] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[23:50:27] <cyberpower678>	 I am using PuTTY Terminal.
[23:50:37] <Joan>	 When did you install PuTTY?
[23:50:37] <tsnag>	 s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10577.000000  
[23:50:52] <msgbot>	 3(resolved) [MAGNUS-303] Undefined indices in fist.php <10https://jira.toolserver.org/browse/MAGNUS-303>  (Magnus Manske)
[23:51:38] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[23:51:38] <Joan>	 cyberpower678: Anyway, you should paste what command you ran and where.
[23:51:44] <cyberpower678>	 Putty was added 2/12/12
[23:51:44] <Joan>	 And then the output (what happened).
[23:51:50] <Joan>	 Okay. :-)
[23:51:55] <Joan>	 So that answers that.
[23:51:57] <tsnag>	 FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL  
[23:52:00] <Joan>	 Anyway, paste something like... 
[23:52:10] <Joan>	 cyberpower@willow:$ crontab ... blah blah
[23:52:14] <cyberpower678>	 cronie C:\downloads\new\crontab.txt C:downloadsnewcrontab.txt: No such file or directory
[23:52:28] <Joan>	 Oh.
[23:52:34] <Joan>	 You're not on the Toolserver.
[23:52:48] <Joan>	 C:\downloads\new\crontab.txt would be your local machine.
[23:52:54] <Joan>	 You want to be on the Toolserver.
[23:52:57] <cyberpower678>	 cyberpower678@willow:~$ cronie crontab.txt crontab.txt: No such file or directory
[23:53:01] <cyberpower678>	 Oh.
[23:53:02] <Joan>	 Right.
[23:53:07] <Joan>	 Move the text file to the server. :-)
[23:53:27] <cyberpower678>	 Hold on.  Let me send through SmartFTP.
[23:53:33] <Joan>	 You probably want an SFTP client like FileZil... or that.
[23:55:03] <cyberpower678>	 SmartFTP works fine.  It supports SFTP over SSH
[23:55:05] <cyberpower678>	 "/home/cyberpower678/.crontab":2: bad minute errors in crontab file, can't install.
[23:55:41] <Joan>	 Paste the command too.
[23:55:48] <Joan>	 Pasting just the result isn't as helpful.
[23:55:57] <cyberpower678>	 cyberpower678@willow:~$ cronie /home/cyberpower678/.crontab
[23:56:05] <Joan>	 Lovely.
[23:56:29] <Joan>	 So run "cat .crontab" and then paste that result to...
[23:56:43] <Joan>	 http://kpaste.net/
[23:56:53] <Joan>	 And then just give us the URL.
[23:56:59] <Joan>	 To see what your file looks like.
[23:57:28] <Joan>	 <input type="checkbox" name="A"/>I'm not a bot<br /><input type="submit" name="paste" value="Send"/>
[23:57:32] <Joan>	 That's cute.
[23:58:41] <Merlissimo>	 Joan: the name of the checkbox field changes weekly