[00:00:19] Change on 12mediawiki a page Developer access was modified, changed by Minnie87.eg link https://www.mediawiki.org/w/index.php?diff=605972 edit summary: /* User:Minnie87.eg */ [00:00:33] :) [00:01:00] It seems to be an user attracted by the outreach women program. [00:03:27] Dereckson: can labs projects have subdomains? [00:08:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:14:11] PiRSquared17 > currently, you have .wmflabs.org and you would like to be able to create a..wmflabs.org b..wmflabs.org? [00:14:38] Dereckson: no, I currently don't have .wmflabs.org but I would like to have that, yes [00:14:58] you already requested a public ip? [00:15:33] there's nothing about public IPs here: https://labsconsole.wikimedia.org/wiki/Help:Getting_Started [00:16:52] No, this is located at https://labsconsole.wikimedia.org/wiki/Help:Addresses and more targeted to network admins. [00:18:21] Dereckson: do you need that in order to have subdomains? [00:18:45] If you need your instance to be accessible from the outside world, yes you need that. [00:19:22] Ask mutante or RyanLane when you see them here or send a mail to the mailing list with your request. [00:19:33] ok [00:20:33] Dereckson: If I request a project, can it get subdomains after? [00:20:38] Yes. [00:22:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:39:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:40:48] Change on 12mediawiki a page Developer access was modified, changed by Dabneylyles link https://www.mediawiki.org/w/index.php?diff=605984 edit summary: [00:53:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:06:53] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 174 processes [01:09:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:11:53] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:24:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:41:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:50:25] Change on 12mediawiki a page Developer access was modified, changed by Dereckson link https://www.mediawiki.org/w/index.php?diff=605988 edit summary: /* User:Minnie87.eg */ [01:51:22] Change on 12mediawiki a page Developer access was modified, changed by Dereckson link https://www.mediawiki.org/w/index.php?diff=605989 edit summary: /* User:Minnie87.eg */ [01:52:37] Dereckson: "shell username. Shell account names should start with a lower-case letter, followed by 1 to 16 numbers, lower case ASCII letters or - (hyphen-minus) characters. (regex: [a-z_][-a-z0-9]*)" [01:52:44] So are dots allowed? [01:54:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:56:45] unrelated: I cannot log into deployment.wikimedia.beta.wmflabs.org [01:57:38] PiRSquared17: I use dots username with success for web accounts on FreeBSD and Debian server for 10 years but it's a very good idea to use [a-z][a-z0-9]* [01:57:45] (servers) [01:58:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [01:58:32] There are plenty of UNIX utilities which consider an username follows a particular format, without real convention [01:58:38] especially the older softwares. [02:11:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:24:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:40:32] RECOVERY Free ram is now: OK on swift-be2 i-000001c8.pmtpa.wmflabs output: OK: 20% free memory [02:41:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:50:21] 11/16/2012 - 02:50:21 - Updating keys for thibaultmarin at /export/keys/thibaultmarin [02:53:33] PROBLEM Free ram is now: WARNING on swift-be2 i-000001c8.pmtpa.wmflabs output: Warning: 16% free memory [02:54:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:11:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:23:30] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 220 processes [03:24:30] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:30:00] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 5.33, 5.82, 5.45 [03:30:50] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 7.15, 6.36, 5.49 [03:31:30] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 6.22, 6.14, 5.36 [03:38:31] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [03:43:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:46:33] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 5.20, 4.73, 4.91 [03:54:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:56:43] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.79, 3.81, 4.80 [03:59:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.72, 4.29, 4.86 [04:05:23] 11/16/2012 - 04:05:23 - Updating keys for thibaultmarin at /export/keys/thibaultmarin [04:13:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:24:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:43:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:54:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:13:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:24:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:34:52] PROBLEM Free ram is now: WARNING on aggregator1 i-0000010c.pmtpa.wmflabs output: Warning: 19% free memory [05:43:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:55:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:13:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:26:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:28:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 204 processes [06:29:23] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:34:23] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:34:33] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:43:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:44:33] RECOVERY Total processes is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS OK: 139 processes [06:46:45] !log [06:48:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [06:48:42] !log Wikidata-dev set ULS on test-repo to 8d5307852a2d25d327063fdcc630c52a83f99815 [06:48:47] is the bot here? [06:56:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:14:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:26:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:43:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 5.21, 4.95, 5.00 [07:44:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:51:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 4.68, 5.00, 5.06 [07:56:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:14:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:26:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:44:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:57:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:12:43] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 19% free memory [09:14:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:17:43] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 21% free memory [09:21:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.82, 4.76, 4.93 [09:27:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:44:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:58:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:13:52] PROBLEM dpkg-check is now: CRITICAL on master i-0000007a.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [10:14:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:29:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:44:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:59:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:14:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:23:52] RECOVERY dpkg-check is now: OK on master i-0000007a.pmtpa.wmflabs output: All packages OK [11:30:25] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:35:42] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 19% free memory [11:40:42] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 20% free memory [11:46:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:01:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:03:43] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 19% free memory [12:05:32] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 153 processes [12:10:33] RECOVERY Total processes is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS OK: 147 processes [12:16:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:31:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:33:33] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 153 processes [12:46:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:01:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:16:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:31:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:46:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:01:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:18:34] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:31:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:48:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:01:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:18:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:31:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:49:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:02:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:18:22] Change on 12mediawiki a page Developer access was modified, changed by Sucheta Ghoshal link https://www.mediawiki.org/w/index.php?diff=606208 edit summary: [16:19:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:32:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:49:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:50:22] 11/16/2012 - 16:50:21 - Updating keys for thibaultmarin at /export/keys/thibaultmarin [17:02:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:15:29] Could anybody with labs powers clean out NFS if that's still an issue? See https://bugzilla.wikimedia.org/show_bug.cgi?id=41008 [17:17:04] andre__: disk at 90% right now [17:18:35] andre__: marked as fixed :-] [17:18:45] hashar, oh, thanks a lot! [17:19:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:25:21] csteipp: Reedy any idea what is the beta branch for on beta labs ? [17:25:27] is that to pill in patchset pending merge ? [17:26:05] hashar: I think I made a branch (with a date in the name), that contained a bunch of uncommitted live hacks. [17:26:10] But that was a while back [17:26:17] csteipp: that is a good idea :-) [17:30:41] !log deployment-prep updating mediawiki-config [17:30:48] see bot does not like me [17:30:49] ping [17:31:32] mlitn: chrismcmahon did we have a bug for the redis/memcached issue on beta ? [17:31:50] * hashar creates an account [17:32:24] hashar: I don't think there was a bug for that [17:32:29] Login successful [17:32:35] fixed thanks to the awesome mlitn [17:32:37] \O/ [17:33:05] mlitn:+++ [17:33:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:39:00] I am out [17:39:04] have a good weekend [17:39:07] and thanks again mlitn [17:39:19] no problem, glad we sorted it out :) [17:39:24] have a nice weekend [17:40:17] thanks! [17:43:29] Change on 12mediawiki a page Developer access was modified, changed by Mitevam link https://www.mediawiki.org/w/index.php?diff=606235 edit summary: [17:49:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:03:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:19:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:34:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:51:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:04:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:18:42] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 19% free memory [19:21:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:34:04] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:50:13] PROBLEM Free ram is now: WARNING on mobile-solr2 i-00000507.pmtpa.wmflabs output: Warning: 8% free memory [19:51:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:52:05] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=606264 edit summary: /* User:Dabneylyles */ Sumana asking whether dabneylyles is ok [19:52:54] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=606265 edit summary: /* User:Sucheta Ghoshal */ done [19:53:38] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=606266 edit summary: /* User:Mitevam */ done [20:03:42] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 538% free memory [20:04:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:23:34] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:26:03] Ryan_Lane: welcome back! [20:26:10] thanks :) [20:26:14] everything going well? [20:26:21] no immediate fires? :D [20:26:33] yeah... wanted to ask you something a day or two ago but can't remember what [20:26:37] are you back in SF? [20:26:44] yeah [20:26:48] just got back in from airport, though [20:26:55] I'm trying not to work today [20:27:00] oh, wow [20:27:20] ok, for some reason 14th was stuck in my head as the day you were coming back. maybe that's wrong [20:27:39] nope. 16th [20:27:59] I didn't put the 14th in my email, eh? [20:28:00] heh [20:28:08] idk [20:28:29] yesterday gerrit was unresponsive by web for at least a few mins but i think ssh was working. fixed itself [20:28:37] (don't think anyone investigated) [20:28:39] * Ryan_Lane nods [20:32:22] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 155 processes [20:34:05] Ryan_Lane: oh, i have an idea of what it might be... maybe i wanted to link you to the ganeti talk. idk if you remember much about it or are interested. i haven't watched it myself yet but i hear it was very good [20:34:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:34:43] 15 05:07:39 < jeremyb> http://www.youtube.com/watch?v=Ktz3vONsrUI&feature=plcp [20:35:01] welcome back, Ryan_Lane [20:35:16] * jeremyb has been waiting for that talk to go up for a while (missed it in person) [20:35:29] Nice work not getting eaten by reef sharks. [20:35:30] andrewbogott: thanks :) [20:35:41] I didn't see a single shark while I was diving. boring. [20:35:42] :) [20:36:12] SCUBA? [20:37:01] yeah [20:38:18] It's not the sharks you see that you should be worried about. [20:38:28] * andrewbogott is pretty sure that's not at all true [20:38:50] heh. yeah. they told us that any shark we saw would be harmless [20:39:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 8.63, 7.49, 5.83 [20:40:24] Sea snakes? Or is that a different part of the world? [20:43:06] <^demon> Certain kinds of coral. Urchins. Man-O-Wars. [20:43:12] <^demon> Lots of fun things in the ocean. [20:53:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:04:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:13:42] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 9% free memory [21:18:43] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 23% free memory [21:23:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:34:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:53:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:02:23] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=606306 edit summary: /* User:Minnie87.eg */ done [22:05:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:07:54] !log deployment-prep log [22:07:58] !beta log [22:07:58] !log deployment-prep log [22:08:03] pooor bot [22:08:13] so hmm [22:08:21] I am migrating udp2log out of /home/ to /data/project [22:13:58] done :-] [22:20:37] dammit did the bot die again? [22:21:12] PROBLEM Disk Space is now: WARNING on sube i-000003d0.pmtpa.wmflabs output: DISK WARNING - free space: / 43 MB (3% inode=40%): [22:21:38] !log are you there, bot? It's me. [22:21:49] * andrewbogott sighs [22:22:23] andrewbogott: !log [projectname] message [22:22:31] I think it ignores you if you don't do that [22:22:49] RoanKattouw, nope, it tells you that it's an unknown project which is what i was testing :) [22:22:53] start-stop-daemon: warning: failed to kill 15343: No such process nom [22:22:54] Oh, ugh [22:23:03] !log are you there now? [22:23:04] are is not a valid project. [22:23:06] Yay! [22:23:14] you guys are using bots-labs, right? :) [22:23:25] Ryan_Lane, did you just restart the bot? [22:23:28] nah I thought we'd use hitler-blond instead [22:23:31] nope [22:23:41] andrewbogott: I did [22:23:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:24:11] It would help if it was actually in git with the prod hacks and lab hacks made into 1 version with I dunno... logging? [22:24:14] Ah, OK. my fingers were poised to restart the bot but it restarted before I hit and was suddenly worried that it had precog abilities [22:24:31] The server's just feeling intelligent today :D [22:24:44] Damianz: I added some logging to the version running on bots-2. At some point ^Demon is going to get the source into git and then I'll move my changes in. [22:24:54] We could totally just write a cronjob to check if the pid responds to kill -0 and if not restart it until it's fixed. [22:25:03] bots-2? bots-labs is current [22:25:14] Well, then I was sorely misled. [22:25:28] See - so many hacks and versions :P [22:25:47] Yep, without git we're doomed to repeat the same mistakes. [22:26:13] PROBLEM Disk Space is now: CRITICAL on sube i-000003d0.pmtpa.wmflabs output: DISK CRITICAL - free space: / 27 MB (2% inode=40%): [22:27:03] um… petan, when you return for your backscroll, please note that you and Damianz seem to disagree about where labs-morebots lives. [22:27:13] * Damianz wonders if Ryan_Lane is officially on holiday still or if he just skipped 2 weeks of his life without noticing again. [22:27:25] still on holiday :) [22:27:27] Don't shoot the messenger, it's all Ryan's fault [22:28:01] it definitely lived on bots-labs [22:28:03] not bots-2 [22:28:32] We probably should deleted the old, not in use versions out of init.d I guess. [22:29:33] All I'm saying is -- I spent half a day repairing the version on bots-2 because that's where petan told me it lived :( [22:29:45] And I narrowly prevented him from rewriting it in C# [22:30:14] He wants to rewrite everything in C#! [22:30:23] Let's write his C# in python! [22:30:35] Damianz: hello! [22:30:46] Hi kaldari. [22:31:24] Did you happen to get any of my pings about mysql on http://bots.wmflabs.org/? [22:31:42] Probably not - I've not yet read by scrollback from the past week or so. [22:31:53] it says that it currently isn't installed there [22:32:06] It's probably not - someone re-installed the server [22:32:07] although it used to work [22:32:23] who would I talk to about getting mysql installed? [22:32:37] it's pretty to me useless without it [22:32:45] er [22:32:46] Do you just need the php extension and client? [22:33:00] yep, that's all [22:33:01] andrewbogott: I don't see how a rewrite would help [22:33:02] Also I really, really need to get my puppet stuff merged in to sort this [22:33:22] there's some bug which makes the bot disappear and not reconnect occasionally, but that's it [22:33:23] Ryan_Lane: I didn't say /I/ wanted to rewrite it. [22:33:26] I have typing dyslexia I'm afraid [22:33:31] I haven't been able to track it down [22:33:35] andrewbogott: yeah [22:34:06] When I entered the scene, the bot was running on bots-2 and it crashed as soon as anyone logged to it. [22:34:13] But I don't know what kind of floundering preceeded that. [22:34:33] Probably it died on bots-labs and someone restarted the broken one on bots-2 by mistake [22:34:43] ah [22:34:46] that would do it [22:34:46] I'd say rewriting it could fix it - rewrite it to a proper module with a config, not the path hacks all over the place :D [22:34:55] it should probably be deleted from that box [22:35:12] Damianz: patches welcome ;) [22:35:40] I re-wrote most of it, but those things are from the original code [22:35:42] No they aren't! Not until it's in git [22:35:45] Ryan_Lane: Pending hacks from prod to be merged into the version in source. [22:35:49] here's where you tell me that it's been in git all along [22:35:52] andrewbogott: it is in git [22:35:52] * sumanah laughs [22:36:06] I pointed you to it in the email ;) [22:36:07] andrewbogott: Chad moved it the other day but apparently there's prod hacks missing from the version that was in svn. [22:36:12] operations/debs/adminbot [22:36:20] which prod hacks? [22:36:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:36:33] OK, so that makes me think it's in git twice, which is just as bad as it not being in git at all... [22:36:36] there shouldn't be any [22:36:41] it's in git twice? [22:36:43] ^demon|lunch can we sort this out? [22:36:46] what's the other location? [22:37:13] Ryan_Lane: Well, when you say it's in git assume you're not referring to the version that Chad imported while you were out of town... [22:37:13] ? [22:37:20] <^demon|lunch> I still hadn't gotten to it. [22:37:32] <^demon|lunch> So there's only 1 copy, unless someone imported it behind my back [22:37:40] Jesus, this is like Rashomon. [22:37:58] just the one copy :) [22:38:03] I have a solution [22:38:06] KILL SVN WITH FIRE [22:38:15] ^demon|lunch: I had you import this the week before I left :D [22:38:27] <^demon|lunch> What are we talking about now? [22:38:36] adminbot/morebots [22:38:41] <^demon|lunch> Ohhhh, herp derp. [22:38:44] <^demon|lunch> Yeah, I did that for Ryan [22:38:46] <^demon|lunch> I forgot. [22:38:46] hahaha [22:38:46] operations/debs/adminbot [22:38:49] <^demon|lunch> Too much Java in my life. [22:39:07] ok, so, how many repositories are there for this? 2 or more? [22:39:11] <^demon|lunch> I'VE BEEN WRITING ENTERPRISEY JAVA ALL WEEK. MY HEAD IS ABOUT TO 'SPLODE! [22:39:11] 1 [22:39:21] ^demon|lunch: public static THE ENDLESS VOID main? [22:39:55] ^demon|lunch: System.exit(2) [22:40:09] * sumanah 's Java joke #1 -- the problem with Java isn't the length of your code, it's the width [22:40:29] * sumanah 's Java joke #2 -- you have to use Eclipse because your verbosity BLOCKS OUT THE SUN [22:40:52] OK, if the bot is in git then I will do some cleanup in the next few days. [22:40:58] * sumanah stops [22:40:58] Including murdering the version on bots-2 [22:40:59] the docs for this are even up to date for where it's running: https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots#labs-morebots [22:41:00] :D [22:41:56] When I was working on the bots-2 version I was like, Man, this is so broken, there's now way this ever worked! I bet this isn't the real version. [22:42:40] ah. it's wrong in another spot, though [22:43:14] At least labs is a true reflection of production - stuff is throw all over the show and there's only really 1 person that knows the truth :D [22:43:45] Well, in this case it was documented, I just assumed that it wasn't. [22:44:18] heh [22:44:31] I just fixed the other incorrect reference [22:45:04] It would be lovely if the rc feed told us that (hint) [22:45:24] What we need is a bot that knows about all the bots [22:46:59] andrewbogott: that is nearing scary AI territory [22:48:46] AI and ANN is awesome [22:54:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:59:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 5.47, 4.00, 4.75 [23:00:59] kaldari: Did I tell you I'd installed php5-mysql and mysql-client on bots-apache01? I can't remember. [23:01:12] no, but thanks!! [23:01:19] looking... [23:03:07] Anyone want to review yucky puppet shizzle with a view to getting it merged into prod? [23:04:44] hmm: "Unknown MySQL server host 'bots-sql2'" [23:05:30] did the bots mysql hosts move elsewhere? [23:05:38] nope [23:05:42] try bots-sql2.pmtpa.wmflabs [23:06:11] Unknown MySQL server host 'bots-sql2.pmtpa.wmflabs' [23:06:13] We do actually need to move those sql servers but I've not had time yet and want to merge puppet stuff first [23:06:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:12:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 5.47, 5.59, 5.28 [23:18:15] Damianz: Working now. Thanks! [23:18:49] np [23:19:38] Damianz: Is that enwiki database on bots-sql2 mirrored yet, or just a dump? [23:19:56] No idea - it won't be a mirror though. [23:20:06] AFAIK we don't have any replication or federated database access setup [23:20:34] ok [23:24:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:37:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:54:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs)