[00:02:56] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 9.80, 7.63, 5.93 [00:03:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 7.59, 7.32, 5.89 [00:07:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:10:36] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 9.32, 8.75, 6.63 [00:12:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 6.34, 7.39, 6.16 [00:22:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:25:39] [bz] (8ASSIGNED - created by: 2Antoine "hashar" Musso, priority: 4Unprioritized - 6normal) [Bug 41133] beta all.dblist is a live hack - https://bugzilla.wikimedia.org/show_bug.cgi?id=41133 [00:37:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:52:06] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:07:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:09:53] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 176 processes [01:14:53] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:22:04] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:37:23] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 148 processes [01:37:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:37:52] PROBLEM Free ram is now: WARNING on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Warning: 17% free memory [01:40:35] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 2.18, 3.06, 4.97 [01:42:52] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 3.18, 3.67, 4.78 [01:52:06] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:00:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 9.87, 7.94, 6.00 [02:03:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 11.36, 9.52, 6.94 [02:08:26] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:22:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:39:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:40:22] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 154 processes [02:52:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:09:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:22:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:39:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:53:45] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:03:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.43, 3.50, 4.94 [04:07:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 5.77, 4.18, 4.87 [04:10:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:15:57] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 8.05, 6.98, 5.96 [04:16:58] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 6.90, 6.31, 5.60 [04:23:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:35:55] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 3.99, 4.54, 4.95 [04:40:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:48:32] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 3.24, 3.97, 4.80 [04:53:47] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:55:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.25, 3.74, 4.82 [04:56:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 2.52, 3.60, 4.72 [05:11:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:23:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:38:57] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 8.59, 7.87, 6.18 [05:39:56] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 7.64, 7.30, 5.70 [05:41:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:41:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 7.73, 7.27, 5.84 [05:43:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 7.29, 6.88, 5.67 [05:54:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:11:46] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:22:12] Change on 12mediawiki a page Developer access was modified, changed by Vishal0123 link https://www.mediawiki.org/w/index.php?diff=601874 edit summary: /* Request for developer access in wikimedia labs */ new section [06:24:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:31:23] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:42:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:54:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:56:22] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [07:03:49] Change on 12mediawiki a page Developer access was modified, changed by Jasper Deng link https://www.mediawiki.org/w/index.php?diff=601875 edit summary: /* Request for developer access in wikimedia labs */ rmv extraneous header [07:10:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 148 processes [07:12:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:24:17] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:42:45] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:54:24] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:11:38] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 2.85, 3.15, 4.64 [08:13:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:13:53] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 2.50, 3.00, 4.61 [08:14:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.02, 3.36, 4.80 [08:25:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:29:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 6.59, 6.31, 5.53 [08:32:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 6.36, 5.74, 5.25 [08:43:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:43:52] PROBLEM Current Load is now: CRITICAL on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: Connection refused by host [08:44:32] PROBLEM Current Users is now: CRITICAL on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: Connection refused by host [08:45:16] PROBLEM Disk Space is now: CRITICAL on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: Connection refused by host [08:46:03] PROBLEM Free ram is now: CRITICAL on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: Connection refused by host [08:47:23] PROBLEM Total processes is now: CRITICAL on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: Connection refused by host [08:47:53] PROBLEM dpkg-check is now: CRITICAL on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: Connection refused by host [08:52:52] RECOVERY dpkg-check is now: OK on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: All packages OK [08:53:52] RECOVERY Current Load is now: OK on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: OK - load average: 0.45, 1.08, 0.79 [08:54:34] RECOVERY Current Users is now: OK on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [08:54:34] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 2.77, 3.69, 4.57 [08:55:12] RECOVERY Disk Space is now: OK on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: DISK OK [08:56:03] RECOVERY Free ram is now: OK on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: OK: 524% free memory [08:56:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:57:22] RECOVERY Total processes is now: OK on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: PROCS OK: 84 processes [08:57:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 2.82, 3.50, 4.67 [09:08:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.89, 3.82, 4.66 [09:10:53] PROBLEM dpkg-check is now: CRITICAL on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: Connection refused by host [09:11:43] PROBLEM SSH is now: CRITICAL on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: Connection refused [09:11:53] PROBLEM Current Load is now: CRITICAL on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: Connection refused by host [09:12:33] PROBLEM Current Users is now: CRITICAL on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: Connection refused by host [09:13:17] PROBLEM Disk Space is now: CRITICAL on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: Connection refused by host [09:14:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:14:03] PROBLEM Free ram is now: CRITICAL on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: Connection refused by host [09:15:22] PROBLEM Total processes is now: CRITICAL on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: Connection refused by host [09:23:32] PROBLEM Current Users is now: WARNING on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in [09:26:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:31:42] RECOVERY SSH is now: OK on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [09:45:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:56:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:14:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:22:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.65, 4.74, 4.94 [10:26:57] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:35:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 5.31, 5.37, 5.21 [10:44:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:57:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:58:57] Silke_WMDE: Do you have fixed your problem= [10:59:25] Jan_Luca: Thanks for asking - yes! [10:59:57] Some variable foo that I omitted... [11:03:19] Silke_WMDE: Ok [11:15:25] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:15:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 11.74, 9.96, 6.34 [11:17:34] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 11.18, 9.49, 6.26 [11:17:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 10.76, 9.68, 6.26 [11:21:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 9.93, 9.46, 6.96 [11:27:04] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:28:35] PROBLEM Current Users is now: CRITICAL on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: Connection refused by host [11:33:34] RECOVERY Current Users is now: OK on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: USERS OK - 4 users currently logged in [11:45:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:57:26] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:58:53] PROBLEM Free ram is now: CRITICAL on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: Critical: 5% free memory [12:03:53] RECOVERY Free ram is now: OK on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: OK: 88% free memory [12:16:05] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:28:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:46:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:52:53] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 3.59, 3.81, 4.79 [12:58:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:02:33] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.41, 4.31, 4.98 [13:05:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [13:10:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [13:10:55] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.07, 4.51, 4.91 [13:16:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:26:53] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 4.64, 4.52, 4.91 [13:28:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:47:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:58:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:17:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:28:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:48:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:58:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:04:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [15:19:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:24:36] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [15:28:23] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 155 processes [15:29:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:49:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:50:55] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 3.99, 4.54, 4.84 [15:59:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:10:43] PROBLEM dpkg-check is now: CRITICAL on integration-jenkins i-00000363.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [16:15:44] RECOVERY dpkg-check is now: OK on integration-jenkins i-00000363.pmtpa.wmflabs output: All packages OK [16:19:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:29:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:32:53] PROBLEM Free ram is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Critical: 5% free memory [16:49:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:59:05] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:10:45] petan: I looked for obvious mistakes in the adminbot code but didn't find any. Can you tell me more about when it crashes and/or do you have any idea what the log looks like during crashes? [17:10:53] Change on 12mediawiki a page Developer access was modified, changed by Nick Wilson link https://www.mediawiki.org/w/index.php?diff=602085 edit summary: [17:18:23] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [17:19:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:29:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:40:00] Silke_WMDE_: I'm reading your patch, can't tell when/where your three new templates are used. Am I overlooking something or are they no longer needed? [17:40:50] Erm... yeah, actually I don't now why they were submitted. They are not used anymore. [17:40:56] * Silke_WMDE_ is a git beginner [17:41:05] That's great! +1 for simplicity. [17:41:31] Um… I can make helpful git suggestions or not, depending on your preference :) [17:42:12] hehe, how can I "unsubmit" those files? [17:43:00] Gerrit is pretty much one-way, but you can avoid including them in a future patch. [17:43:28] Are you creating your patch using 'git add' and a subsequent 'git commit'? Or are you doing something like git commit ? [17:46:25] git add, then git commit, then git review --amend [17:46:40] Ah, Ok, that's what I do too :) [17:47:08] Before submitting a patch, I also do 'git log' and then grab the shid of my patch... [17:47:19] and then do 'git show ' and that shows me everything that's going to get pushed to gerrit. [17:47:29] That would warn you about superflous files. [17:48:01] Oh, wait! I misread… I don't know what git review --amend does. [17:48:05] (btw with last week's feedback about tabs and spaces I studied the documentation and they "want" spaces.) [17:48:11] I use 'git commit --amend' to get my patch nice and square before submitting. [17:48:30] Really? Does that mean there's a puppet style-guide that I haven't read? [17:49:03] Silke_WMDE_: Does what I'm saying about 'commit --amend' and 'git show' make sense, or is it stuff you already do, or should I slow down? [17:49:39] no, I wrote bullshit [17:49:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:49:53] git commit --amend to signal it's a patchset [17:50:26] OK, great :) So then, I think my only advice is: look at the patch right before you submit it and make sure it looks the way you expected :) [17:51:28] ok [17:51:29] :) [17:52:03] 'git rm' is a little scary because it sometimes removes the actual file as well as removing it from the index… [17:52:11] I can't remember if 'git rm -f' does that or if 'git rm -f' doesn't do that... [17:52:16] So, proceed with caution. [17:52:25] ok [17:52:31] Although now those files are backed up in gerrit so you're safe. [17:52:38] Here's the puppet style guide: http://docs.puppetlabs.com/guides/style_guide.html [17:52:42] * andrewbogott hopes this is making sense [17:52:52] thanks [17:53:10] Hm… I don't know if wmf has an alternative styleguide or if we're supposed to be using that one. [17:53:38] But in any case, even if you do whitespace-edit a file, that should be part of a separate patch. Otherwise the whitespace changes make your patch look sort of crazy. [17:54:16] I tend to assume that 'follow the existing style of a given file' trumps any styleguide. And if you want to reformat a file entirely that should be a discrete patch (and that may be unpopular, depending.) [17:55:11] https://mediawiki.org/wiki/CC [17:55:14] andrewbogott: Style guide ^ [17:55:25] 11/07/2012 - 17:55:21 - Creating a home directory for harsh at /export/keys/harsh [17:55:38] (but the rest of what you said makes sense) [17:55:39] marktraceur: Nothing about puppet in there... [17:55:48] andrewbogott: No, but it's the general style [17:56:01] I've been swallowing my tab-rage for months, clearly that's now boiling over. [17:56:33] Oh, goddamn it, even the style guide says to use tab characters. *sigh* [17:57:05] andrewbogott: Yeah, we recommend tabs. Sorry! [17:58:31] So, does the general style guide apply to *everything*, also puppet stuff? [17:59:17] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:59:24] I guess. We probably need someone to add a puppet section to that page. [18:00:13] Silke_WMDE_: IIRC the guideline is "if all else fails, follow the PHP style guide where applicable", but I think you could wiggle around that. Probably we just need a puppet style guide. [18:00:25] 11/07/2012 - 18:00:21 - Updating keys for harsh at /export/keys/harsh [18:20:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:29:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:34:05] Change on 12mediawiki a page Developer access was modified, changed by Jbk malkum link https://www.mediawiki.org/w/index.php?diff=602104 edit summary: [18:51:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:59:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:07:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [19:21:46] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:28:55] Jan_Luca: Are you waiting on feedback from me, or are you and Faidon still in discussions? [19:30:19] andrewbogott: I wait for an answer of anyone but I think Faidon should answer because he has the problems with my change [19:30:28] yep, ok. [19:31:19] paravoid: Are you there, Faidon? [19:31:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:32:29] paravoid ^^ [19:39:54] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: OK - load average: 4.72, 4.75, 4.97 [19:42:04] Jan_Luca: paravoid keeps funny hours but he lives in Athens, so this is a bit late for him. [19:42:33] hey [19:46:34] paravoid: Hi [19:46:58] andrewbogott: I live in DE, I'm not very different to him [19:47:09] That's easy then [19:47:36] but I work at day so my Wikimedia-time is the late time ;-) [19:47:43] well, there's a slight difference, I do this as a dayjob :) [19:47:52] but fortunately (or unfortunately for me :) I work all kinds of hours [19:48:58] paravoid: I have answered you on Gerrit, maybe you can take a look but you can do it tomorrow as well [19:49:36] now is fine [19:50:02] okay, read that, it's basically the same argument [19:50:51] paravoid: I do not know how good you know the Labsconsole [19:51:25] but I think this classes are not very different in its use to a class webserver::php5 [19:51:43] I'm no expert, but I think I do know our puppet structure well [19:52:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:52:52] PROBLEM Free ram is now: WARNING on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Warning: 17% free memory [19:53:48] I still wonder what the use case is for that other than guesses on what people might find useful [19:55:29] paravoid: I still don't understand this discussion. Don't those classes need to be defined so that we can include them elsewhere, and aren't they already used elsewhere? [19:56:10] they need to be defined if we're to include them elsewhere, yes [19:56:15] andrewbogott: I have add some classes for use in other classes like the webserver::... ones and three classes for including in labs-interface [19:56:16] but I still not seeing where we would include them [19:56:32] the argument has been "labsconsole" but I don't agree that these are appropriate to labsconsole [19:56:54] they're certainly not appropriate for site.pp, and I don't see the two as *that* different [19:57:38] paravoid: So those that are there for the refactor of the webserver:: class you agree about? [19:57:40] The php::mysql, php::gd, php::pear classes can be used for example in webserver:: or mediawiki:: [19:58:18] I don't disagree with those [19:58:34] I disagree with the php::most_used (used by what? most of what?) [19:59:22] paravoid: I think he added most_used as a grouping because you were unhappy about each package having its own class :) [19:59:22] paravoid: I wanted to create two easy to install classes which setup a php installation [19:59:34] Oh, ok, maybe not... [19:59:56] andrewbogott: ... and reduce the number of little classes ;-) [20:00:46] I wasn't happy with having a class per package, and I counter-proposed php::extension { [ "foo", "bar" ]: ensure => installed } instead [20:01:03] then I realized that these were class because the intention was to go into labsconsole, which I also disagree with [20:01:22] and still, setup an installation of what? [20:01:44] an installation of php for e.g. bots, webtools, ... [20:01:55] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:02:17] why these can't have their own classes? [20:02:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 293 processes [20:03:03] that do all kinds of things, including but not limited to php packages? [20:04:12] because not every instance/project want to create a new puppet class, with some smaller ones they can select in Labsconsole the classes they need and puppet installs them [20:04:25] why not? [20:04:58] this isn't a vps environment, this is an attempt to integrate volunteers' work more into the wikimedia infrastructure [20:05:20] Because with this smaller ones they can setup an instance with checkboxes [20:05:37] what's the point of that? [20:07:05] projects need sysadmins. if a sysadmin has to do the same job over and over, then we can discuss what's the best way to automate this [20:07:25] it's also nice to see projects' work get more deeply integrated with our tools (puppet etc.) [20:07:28] Jan_Luca: I'm strongly in favor of you laying the groundwork to refactor webserver:: and mediawiki:: so maybe you should limit your php classes to that for the moment? Then when Ryan returns we can have a real discussion about whether we want labsconsole/puppet to be used as a GUI replacement for apt-get. [20:07:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [20:08:46] andrewbogott: That is a good idea, I can split up the change in two: one for refactoring and one about that we discuss [20:08:55] I'd also be in favor into such a refactoring [20:09:02] Jan_Luca: I think that's the best way to get you unblocked for the moment. [20:09:03] and it'd be much welcome [20:09:29] You can draft an email about the puppet vs. apt-get thing and save it in your 'drafts' folder until Ryan gets back on... [20:09:31] andrewbogott: what's your take on that issue btw? [20:09:33] a moment please, I split them up [20:09:34] * andrewbogott looks it up [20:10:22] Hah, ryan's calendar just says 11/2-11/? [20:10:51] paravoid: I think you're probably right that it's unneeded. But my mind will change as soon as 2 or more classes define the same php package :) [20:10:58] There's nothing wrong with waiting until there's evidence of repeated use. [20:11:09] oh indeed, same here [20:11:26] if patterns emerge, I'd be also happy to reconsider [20:11:37] but then most_used will be based on those patterns for example. [20:12:40] Jan_Luca: Ryan predicts return on the 16th. [20:13:16] Jan_Luca: But we seem to have an emerging consensus in the meantime :) [20:14:58] paravoid, andrewbogott: I have removed the classes with problems, please rereview: https://gerrit.wikimedia.org/r/#/c/29975/ [20:18:11] Change on 12mediawiki a page Developer access was modified, changed by Girlwithglasses link https://www.mediawiki.org/w/index.php?diff=602126 edit summary: [20:18:24] there you go [20:21:37] paravoid, andrewbogott: For the classes with discussion, I have uploaded a new change: https://gerrit.wikimedia.org/r/#/c/32263/ [20:22:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:23:11] paravoid: I think there should be a init.pp [20:23:41] hm, I have mixed feelings myself [20:24:22] This is my reference: http://docs.puppetlabs.com/puppet/3/reference/modules_fundamentals.html [20:24:47] apparently you don't need an init.pp doing anything, but generally from what I've seen there is allways one [20:25:09] oh you mean that we need an init.pp? no we don't [20:25:35] <3 autoloading [20:25:59] paravoid: The problem with php::extension is that there are some packages named php-... and some php5-... [20:26:30] yeah, that's a shame [20:27:25] I don't know what puppet do if a package does not exist [20:27:32] errors out [20:27:39] I can think of various ugly ways to handle that [20:27:55] like a whitelist or blacklist in the definition [20:28:03] any ideas? [20:28:27] or a package => "php5-foo" for the non-default case [20:28:47] thoughts? [20:28:58] andrewbogott: Any idea? [20:29:52] Um… not sure I understand the question yet. [20:30:58] andrewbogott: paravoid wants a class php::extension ( $name ) for things where is no special class [20:31:14] but there are packages php- and packages php5- [20:31:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:33:43] andrewbogott: See paravoid's comment: https://gerrit.wikimedia.org/r/#/c/29975/7 [20:34:52] Yeah… I can't think of a great solution. Short of enumerating every possible package... [20:35:22] 11/07/2012 - 20:35:21 - Updating keys for krenair at /export/keys/krenair [20:35:48] I think paravoid's suggesting may be best, have the class take either a shortened or a full-length package name, and presume that it's php- if the short name is specified. [20:35:56] Well, presuming there aren't packages called php-php. Dang. [20:36:23] Class could take a 'package prefix' arg that defaults to 'php'. [20:36:29] * andrewbogott still not totally sure he's talking about what you are talking about. [20:37:38] alternatively, we can ditch the packages altogether if we're not going to do anything else (configure their .ini etc.) with them [20:37:52] and just do package { "php5-foo": } in the callsites, it's not that difficult [20:41:26] paravoid: I would remove the php::wikimedia thing at moment and add a php::extension thing later [20:53:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:01:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:24:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:32:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:52:44] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=602152 edit summary: /* User:Vishal0123 */done [21:53:22] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=602154 edit summary: /* User:Nick Wilson */ done [21:53:53] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=602155 edit summary: /* User:Jbk malkum */ done [21:54:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:54:36] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=602156 edit summary: /* User:Girlwithglasses */ done! [22:02:05] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:02:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 293 processes [22:12:26] hashar- You around? [22:12:26] anomie: yup [22:12:31] anomie: in some hotel lobby right now [22:12:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [22:12:46] hashar- Traveling? [22:13:03] yeah we are having a continuous integration sprint in Utrecth, NL [22:13:13] with Krinkle and Zeljkov [22:13:30] hashar- Did you get a chance to see I posted some changesets on bug 41133? [22:13:56] anomie: nop. Taking a look. [22:14:44] ohhh [22:14:48] so hmm [22:16:09] that is nice [22:17:59] * AaronSchulz hasn't been in this channel for months :) [22:18:13] anomie: you did a ton of cleanup [22:18:13] awesome [22:18:28] anomie: I am unlikely to be able to review all of them [22:18:40] I would like to focus on CI for the next few days [22:19:11] anomie: Reedy is coming on saturday. So we can probably review the change together and get them merged over the weekend [22:19:21] So how do things look, and how do we test them besides throwing them on Beta and seeing what blows up? Or is that the plan? [22:19:37] blowing beta is fine to me :-] [22:19:58] though the wmf-beta-autoupdate script is doing a git submodule update --init of all extensions [22:20:04] and git pull of mediawiki/core [22:20:16] so it is tricky to apply your change in beta :-( [22:20:39] testing on beta is definitely a good idea [22:23:19] anomie: to disable the auto update: login on deployment-bastion [22:23:40] stop the auto update service: sudo service wmf-beta-autoupdate stop [22:23:52] might take a little bit to stop though [22:24:15] It looks like that script pulls mediawiki/core and mediawiki/extensions... Except for the changeset to WikimediaMaintenance, that shouldn't interfere with testing things anyway [22:24:23] then you could apply the changes using the command Gerrit offers under each patchset. Something like git fetch https://gerrit.wikimedia.org/r/mediawiki/extensions/WikimediaMaintenance refs/changes/69/32169/2 && git cherry-pick FETCH_HEAD [22:24:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:25:12] And for testing that bit, it seems they could probably just be copied into /home/wikipedia/common/bin or someplace? [22:25:42] Am I missing something? [22:26:12] anomie hashar +1 for blowing up beta. the more we use it, the more we learn about how to use it [22:26:33] hehe [22:26:40] I guess that is a nice staging area for such changes [22:26:56] anomie: definitely get some people that deploy often to review the change [22:27:07] maybe Roan, Aaron and Reedy [22:27:28] hashar- Did we ever figure out the problem with git status in beta? [22:27:44] anomie: probably related to the permission madness [22:27:56] anyway now we have to use depoyment-bastion [22:28:00] and run commands as mwdeploy user [22:28:23] sudo su --login mwdeploy -s /bin/bash [22:28:23] works well [22:28:29] so everything under /home/wikipedia/common should be chowned to mwdeploy? [22:28:31] the wmf-beta-autoupdate script runs as mwdeploy user as well [22:28:36] and on deployment-bastion [22:28:44] yeah everything should belong to mwdeploy:mwdeploy [22:32:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:34:42] hashar: sudo -u mwdeploy /bin/bash [22:35:13] Sorry, user hashar is not allowed to execute '/bin/bash' as mwdeploy on i-00000390.pmtpa.wmflabs. [22:35:14] ;--) [22:36:22] labs console has support for sudo policies, I have no idea how to properly set it up so only a handful of people would be allowed to run commands as the mwdeploy user [22:36:29] I drop a mail about that on labs-l [22:36:29] hashar: Which change is this? [22:36:45] !bug 41133 | RoanKattouw [22:36:45] RoanKattouw: https://bugzilla.wikimedia.org/41133 [22:36:51] that is 3 changes [22:37:16] we are hardcoding all.dblist everywhere [22:37:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [22:37:37] and beta has a different set of databases so we need to use a different "all" dblist such as all-labs.dblist [22:37:46] and anomie has been working on that ;-] [22:38:05] I should really start packing for tomorrow. Need to be down south and with internets before the wikidata meetin [22:38:13] hashar: hopefully, that'll be going away when wikivoyage is deployed ;) [22:38:35] OMG, triple dependency [22:38:54] Reedy: when do you arrive ? [22:39:13] RoanKattouw: scary ;-] [22:39:28] Depends how long it takes me to drive from Calais, to Gronau, and then back to Amsterdam on friday ;) [22:39:54] * RoanKattouw guesses 6-7 hours? [22:40:22] Google says 6 hours 40 [22:40:37]