[00:06:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:20:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:36:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:50:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:06:34] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 177 processes [01:06:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:11:32] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:20:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:34:43] PROBLEM Free ram is now: WARNING on aggregator2 i-000002c0.pmtpa.wmflabs output: Warning: 8% free memory [01:35:33] RECOVERY Disk Space is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: DISK OK [01:35:33] RECOVERY Current Users is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [01:35:33] RECOVERY Total processes is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: PROCS OK: 223 processes [01:36:13] RECOVERY SSH is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [01:36:23] RECOVERY Current Load is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: OK - load average: 0.23, 0.52, 0.68 [01:37:53] RECOVERY dpkg-check is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: All packages OK [01:38:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:51:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:04:42] PROBLEM Free ram is now: UNKNOWN on aggregator2 i-000002c0.pmtpa.wmflabs output: NRPE: Call to fork() failed [02:05:53] PROBLEM dpkg-check is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:08:33] PROBLEM Disk Space is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:08:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:08:33] PROBLEM Current Users is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:08:33] PROBLEM Total processes is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:09:13] PROBLEM SSH is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: Server answer: [02:09:23] PROBLEM Current Load is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:09:43] PROBLEM Free ram is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:21:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:38:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:38:43] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 21% free memory [02:51:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:08:34] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:11:42] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 15% free memory [03:21:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:38:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:51:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:08:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:10:52] RECOVERY dpkg-check is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: All packages OK [04:13:32] PROBLEM Disk Space is now: UNKNOWN on aggregator2 i-000002c0.pmtpa.wmflabs output: NRPE: Call to fork() failed [04:13:32] PROBLEM Current Users is now: UNKNOWN on aggregator2 i-000002c0.pmtpa.wmflabs output: NRPE: Call to fork() failed [04:13:32] PROBLEM Total processes is now: UNKNOWN on aggregator2 i-000002c0.pmtpa.wmflabs output: NRPE: Call to fork() failed [04:14:12] RECOVERY SSH is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [04:14:22] PROBLEM Current Load is now: UNKNOWN on aggregator2 i-000002c0.pmtpa.wmflabs output: NRPE: Call to fork() failed [04:14:42] PROBLEM Free ram is now: UNKNOWN on aggregator2 i-000002c0.pmtpa.wmflabs output: NRPE: Call to fork() failed [04:18:33] PROBLEM Disk Space is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:18:33] PROBLEM Current Users is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:18:33] PROBLEM Total processes is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:18:53] PROBLEM dpkg-check is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:19:23] PROBLEM Current Load is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:19:43] PROBLEM Free ram is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:22:13] PROBLEM SSH is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: Server answer: [04:22:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:38:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:52:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:08:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:23:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:38:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:53:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:02:54] Is there a labs admin in here that can give me ssh access? I really want it. [06:08:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:09:56] Reedy? [06:23:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:28:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 204 processes [06:36:34] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 152 processes [06:38:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:41:32] RECOVERY Total processes is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS OK: 146 processes [06:48:33] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 194 processes [06:54:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:08:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:23:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 7.73, 7.01, 5.52 [07:24:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:26:12] PROBLEM Disk Space is now: WARNING on sube i-000003d0.pmtpa.wmflabs output: DISK WARNING - free space: / 47 MB (3% inode=40%): [07:31:13] PROBLEM Disk Space is now: CRITICAL on sube i-000003d0.pmtpa.wmflabs output: DISK CRITICAL - free space: / 31 MB (2% inode=40%): [07:38:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:54:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:00:38] Change on 12mediawiki a page Developer access was modified, changed by יוסף שמח link https://www.mediawiki.org/w/index.php?diff=606884 edit summary: [08:08:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 3.01, 3.19, 4.71 [08:09:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:25:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:39:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:50:39] Change on 12mediawiki a page Developer access was modified, changed by Chihonglee link https://www.mediawiki.org/w/index.php?diff=606893 edit summary: [08:55:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:09:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:26:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:36:42] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 19% free memory [09:39:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:41:43] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 142% free memory [09:56:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:09:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:27:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:39:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:58:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:09:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:28:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:39:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:59:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:07:22] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 162 processes [12:09:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:10:02] PROBLEM Total processes is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: PROCS WARNING: 153 processes [12:15:02] RECOVERY Total processes is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: PROCS OK: 147 processes [12:22:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 146 processes [12:29:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:39:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:59:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:09:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:29:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:37:13] RECOVERY SSH is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [13:39:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:45:12] PROBLEM SSH is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: Server answer: [13:50:00] Hi! [13:51:37] Someone told me: Hi! I saw that you're interested in the technical side of Wikimedia sites, and I wanted to invite you to get developer access (to submit changes to MediaWiki and to Wikimedia site configuration, and to use the sandbox space of Wikimedia Labs). You're also invited to watch and participate in our live tech chats, which include video streaming. The next one is in two weeks and you'll be able to watch, live, [13:52:21] And I want to request for the Developer access [13:52:43] But I dont know how to join the MediaWiki development community [13:52:52] May anyone tell me how to do it? [13:59:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:02:13] Change on 12mediawiki a page Developer access was modified, changed by TK-999 link https://www.mediawiki.org/w/index.php?diff=606983 edit summary: [14:11:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:29:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:41:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:00:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:11:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:30:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:41:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:00:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:11:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:31:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:41:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:01:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:11:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:14:42] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 17% free memory [17:29:43] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 144% free memory [17:31:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:41:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:02:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:13:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:18:34] hello [18:32:19] How can a new project be created? [18:32:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:42:58] PiRQuad|absum: Hi - make a request on https://labsconsole.wikimedia.org/wiki/Special:FormEdit/New_Project_Request, then bug andrewbogott to read https://labsconsole.wikimedia.org/wiki/Special:Ask/-5B-5BCategory:New-20Project-20Requests-5D-5D-5B-5BIs-20Completed::No-5D-5D/-3FProject-20Name/-3FProject-20Justification/-3FModification-20date/format%3Dbroadtable/sort%3DModification-20date/order%3Dasc/h [18:43:34] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:43:54] PiRQuad|absum: I can create a project right now… what do you want it called and what's it for? [18:45:10] Oh hey andrewbogott, you're actually about heh [18:45:26] yeah, not for long [18:48:33] Change on 12mediawiki a page Developer access was modified, changed by Ilonakz link https://www.mediawiki.org/w/index.php?diff=607027 edit summary: [18:50:28] andrewbogott: https://labsconsole.wikimedia.org/wiki/New_Project_Request/Global_AbuseFilter_test [18:52:21] PiRSquared17 I need a project name that's all lower case no spaces [18:52:51] how about abusefilter-global ? [18:52:57] sure, ok. [18:55:04] 11/18/2012 - 18:55:03 - Creating a project directory for abusefilter-global [18:55:35] PiRQuad|absum: Getting mediawiki install and running on an instance is pretty straightforward. There's a howto guide here: https://labsconsole.wikimedia.org/wiki/Help:InstanceConfigMediawiki [18:55:59] PiRQuad|absum: But that page won't make much sense until you've digested the 'getting started' guide and fiddled around a bit. [18:56:23] Feel free to create temporary/disposable instances for learning and experimentation, just make sure you clean up the ones you aren't using. [18:58:22] can I ssh/log into it? [19:00:03] um... [19:00:40] PiRQuadratus: what do you mean by 'it'? [19:02:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:05:48] andrewbogott: the instance; so I need to Request Shell Access? (I was reading https://labsconsole.wikimedia.org/wiki/Getting_started#Project_Instances ) [19:06:16] Ah, I see. Yeah, if you haven't used labs before I need check a box, just a second... [19:07:43] PiRQuadratus: You should have access to bastion now. [19:13:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:15:23] 11/18/2012 - 19:15:22 - Creating a home directory for pirsquared at /export/keys/pirsquared [19:17:00] That doesn't give us many possible projects :( [19:20:21] 11/18/2012 - 19:20:20 - Updating keys for pirsquared at /export/keys/pirsquared [19:21:31] hm did I do that? [19:22:35] nope, crond did [19:23:00] or vixie-cron, not sure what's actually on that box [19:23:22] ah, automatically updated [19:25:30] * PiRQuadratus thought he typed something wrong for a second [19:27:29] I officially just turned 30days into 6 weeks <3 bug trackers time estimates [19:33:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:43:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:03:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:13:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:29:09] Labs uses cookies to log in users. You have cookies disabled. Please enable them and try again. [20:29:13] Cookies definitely aren't disabled... [20:30:15] Krenair: where? [20:30:31] Sorry, that's a separate issue I'm having logging into labsconsole [20:30:47] PiRQuadratus, so you're getting Permission denied (publickey) [20:31:02] yes [20:31:33] Windows? [20:32:30] Krenair: no --- I just got in actually [20:32:33] forgot to ssh-add [20:32:35] ah [20:33:13] In my windows .bashrc I put this: eval `ssh-agent` [20:33:14] ssh-add [20:33:28] That way it sets up for SSH whenever I launch Git Bash (I usually use SSH for git stuff you see) [20:33:42] But there are no instances [20:34:04] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:34:54] RECOVERY Free ram is now: OK on aggregator1 i-0000010c.pmtpa.wmflabs output: OK: 705% free memory [20:35:43] Yeah you'll need to set up an instance. I can't help you with that bit as I've never used labs [20:36:09] andrewbogott: what is the instance name? [20:36:45] PiRQuadratus: A project is a container for instances. If you haven't made any then you don't have any. [20:36:50] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Abusefilter-global shows you have no instances [20:43:23] Krenair: same error as you Labs uses cookies to log in users. You have cookies disabled. Please enable them and try again. [20:43:40] andrewbogott, any idea what that's about? ^ [20:43:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:43:57] but I definitely have cookies on because I'm logged in on Wikipedia [20:44:14] probably memcache falling over again [20:44:19] it's a pile of poo [20:44:21] I see it sometimes but have never investigated because I'm always able to get in eventually. [20:44:23] Lemme try. [20:45:36] Yep, happening to me too [20:46:57] And indeed memcached had crashed. PiRQuadratus, try now? [20:47:33] Can we replace it with redis? <3 redis [21:04:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:04:42] PROBLEM Total processes is now: CRITICAL on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [21:04:52] PROBLEM Free ram is now: CRITICAL on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [21:09:32] RECOVERY Total processes is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS OK: 147 processes [21:09:42] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 29% free memory [21:12:49] PiRQuadratus, have you tried again? [21:13:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:15:41] Krenair: yes [21:16:01] works [21:34:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:40:30] "You must have Sys Admin rights in order to create and manage instances" [21:41:48] 0.o I *am* a sysadmin [21:43:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:48:52] PROBLEM Current Load is now: CRITICAL on abusefilter-global-main i-00000511.pmtpa.wmflabs output: Connection refused by host [21:49:32] PROBLEM Current Users is now: CRITICAL on abusefilter-global-main i-00000511.pmtpa.wmflabs output: Connection refused by host [21:50:12] PROBLEM Disk Space is now: CRITICAL on abusefilter-global-main i-00000511.pmtpa.wmflabs output: Connection refused by host [21:51:02] PROBLEM Free ram is now: CRITICAL on abusefilter-global-main i-00000511.pmtpa.wmflabs output: Connection refused by host [21:52:22] PROBLEM Total processes is now: CRITICAL on abusefilter-global-main i-00000511.pmtpa.wmflabs output: Connection refused by host [21:52:52] PROBLEM dpkg-check is now: CRITICAL on abusefilter-global-main i-00000511.pmtpa.wmflabs output: Connection refused by host [21:56:42] So... did I break it? [21:57:22] RECOVERY Total processes is now: OK on abusefilter-global-main i-00000511.pmtpa.wmflabs output: PROCS OK: 84 processes [21:57:52] RECOVERY dpkg-check is now: OK on abusefilter-global-main i-00000511.pmtpa.wmflabs output: All packages OK [21:58:52] RECOVERY Current Load is now: OK on abusefilter-global-main i-00000511.pmtpa.wmflabs output: OK - load average: 0.12, 0.73, 0.58 [21:59:32] RECOVERY Current Users is now: OK on abusefilter-global-main i-00000511.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [22:00:12] RECOVERY Disk Space is now: OK on abusefilter-global-main i-00000511.pmtpa.wmflabs output: DISK OK [22:01:02] RECOVERY Free ram is now: OK on abusefilter-global-main i-00000511.pmtpa.wmflabs output: OK: 1056% free memory [22:01:59] DNS doesn't seem to work [22:03:16] yes PiRQuadratus? [22:03:34] Jasper_Deng: "PROBLEM Current Load is now: CRITICAL on abusefilter-global-main i-00000511.pmtpa.wmflabs output: Connection refused by host" [22:03:45] any idea how to fix? [22:04:05] idk [22:04:09] 'check firewall settings [22:04:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:13:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:14:04] PiRQuadratus: any luck? [22:14:29] I don't think so [22:15:35] is your Apache up? [22:20:38] andrewbogott: hello [22:20:49] I think I've made some progress [22:21:17] That's good! [22:21:48] but it still doesn't work on http [22:22:28] even though I added "role::mediawiki-install::labs", etc. [22:22:45] OK. And you set up a security group? [22:23:02] yes [22:23:57] Hm… I see that you have a web security group but it doesn't have any rules in it. [22:24:20] We don't need no rules [22:24:37] It also doesn't look like your instance is /in/ the web security group, although maybe that's a mistake on labsconsole… can you ssh in? [22:25:05] Oh, it is a labsconsole error, every instance shows as being in no security groups :( [22:25:16] andrewbogott: yes, I'm ssh-ed into this instance... how do I add the instance to the security group? [22:25:45] You can only set security groups when creating an instance. It's a list of checkboxes on the create instance page. [22:25:56] :( [22:26:00] But, if you added the instance to the 'web' group then you can add rules without having to rebuild the instance. [22:26:16] So I need to delete it and do it all over? [22:26:56] Presuming you didn't add it to the web group when you built it, then yes. [22:27:07] awww [22:27:22] But also since your security group doesn't have rules I'm worried that… more explanation is necessary :) [22:28:00] andrewbogott: "Web" and "default" or just "Web"? [22:28:11] Both. [22:28:20] 'default' allows ssh so you'll want every instance in that group. [22:29:50] ok, done [22:31:02] So, for background: 'default' is a special always-present security group. Any other group is completely arbitrary both in names and in rules. [22:31:22] So just naming a group 'web' doesn't do anything magical; you need a group that explicitly opens up http ports. [22:31:37] (Which, traditionally such group would be named 'web' but that's just for convenience.) [22:31:41] …does that make sense? [22:32:00] Yes, but it's a little confusing [22:32:25] how so? [22:32:31] We should come up with secret codewords to make it more confusing clearly. [22:35:31] "building" [22:35:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:37:09] andrewbogott: how long does "building" take? [22:37:15] PiRQuadratus, this may be useful: https://labsconsole.wikimedia.org/wiki/Help:Security_Groups#Examples [22:37:22] It varies, usually just a few minutes. [22:38:28] Is 10 minutes unusual? [22:39:43] not necessarily. [22:40:03] Once it gets going you can look at the console log to see what's happening. [22:40:10] the last one only took a minute or two [22:40:12] PROBLEM host: i-00000512.pmtpa.wmflabs is DOWN address: i-00000512.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000512.pmtpa.wmflabs) [22:40:22] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 156 processes [22:40:22] ^ that's it [22:43:52] RECOVERY host: i-00000512.pmtpa.wmflabs is UP address: i-00000512.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 4.89 ms [22:44:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:44:22] PROBLEM Total processes is now: CRITICAL on abusefilter-global-main i-00000512.pmtpa.wmflabs output: Connection refused by host [22:45:52] PROBLEM Current Load is now: CRITICAL on abusefilter-global-main i-00000512.pmtpa.wmflabs output: Connection refused by host [22:45:52] PROBLEM dpkg-check is now: CRITICAL on abusefilter-global-main i-00000512.pmtpa.wmflabs output: Connection refused by host [22:46:32] PROBLEM Current Users is now: CRITICAL on abusefilter-global-main i-00000512.pmtpa.wmflabs output: Connection refused by host [22:47:12] PROBLEM Disk Space is now: CRITICAL on abusefilter-global-main i-00000512.pmtpa.wmflabs output: Connection refused by host [22:48:02] PROBLEM Free ram is now: CRITICAL on abusefilter-global-main i-00000512.pmtpa.wmflabs output: Connection refused by host [22:51:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 5.35, 6.21, 5.14 [22:51:56] andrewbogott: when I tried to ssh in -> "WARNING: POSSIBLE DNS SPOOFING DETECTED" [22:52:16] also "IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!" [22:52:33] If your new instance has a different IP but the same name as one you previously registered with ssh, then... [22:52:38] then ssh is wise to warn you :) [22:52:53] I believe there's info in the warning message about where the known host is registered and you can remove that line. [22:54:23] see if we had sshfp records in place that wouldn't happen [22:54:23] :D [22:57:46] So how does one fix there "problem[s]" above? [22:57:50] *these [22:58:42] Which problems? [22:59:03] PROBLEM Free ram is now: CRITICAL on abusefilter-global-main i-00000512.pmtpa.wmflabs output: Connection refused by host [22:59:05] You mean the ssh thing, or the nagios warnings? [22:59:22] the nagios warnings [22:59:26] Oh, that's probably nothing to worry about. Nagios started doing status checks before the instance was up and running and ready to answer its stupid questions. [22:59:50] Although I would expect it to give all clears at some point... [23:00:07] What's next? [23:00:11] Anyway I would ignore them unless you are encountering actual tangible problems. Nagios tends to be a bit dramatic. [23:00:32] You're following this guide? https://labsconsole.wikimedia.org/wiki/Help:InstanceConfigMediawiki [23:00:46] I created an instance with a security group, enabled port 80 for that group, added role::mediawiki-install::labs [23:00:48] yes [23:00:51] seems to be done though [23:02:20] It looks to me like apache is up and running but mediawiki is not. [23:02:36] Did puppet report anything interesting when you refreshed? [23:02:49] ? [23:03:59] see console log/run puppetd -tv [23:05:57] NovaInstance?action=consoleoutput -> nothing [23:06:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:06:26] hmm [23:06:29] it built, right [23:07:00] PiRQuadratus: log into the instance, then do 'sudo puppetd -tv' [23:07:12] that will re-run puppet and we can see if there are error messages. [23:07:26] Also you will need to figure out about SOCKs proxying in order to get web access to the system. [23:07:42] pirsquared is not in the sudoers file. This incident will be reported. ...... [23:08:32] oh, that's for bastion1 [23:08:39] andrewbogott: I can't ssh into the instance [23:08:42] it's broken [23:08:58] could you previously? [23:09:07] yeah [23:09:14] now it says "@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @" etc. [23:09:22] RECOVERY Total processes is now: OK on abusefilter-global-main i-00000512.pmtpa.wmflabs output: PROCS OK: 97 processes [23:10:42] Wait, so… sorry. [23:10:53] RECOVERY Current Load is now: OK on abusefilter-global-main i-00000512.pmtpa.wmflabs output: OK - load average: 0.39, 1.26, 1.14 [23:10:53] RECOVERY dpkg-check is now: OK on abusefilter-global-main i-00000512.pmtpa.wmflabs output: All packages OK [23:10:54] To clarify -- when you say you could ssh in previously, you mean, you could ssh into the previous instance, right? [23:10:59] Have you ever sshd into /this/ instance? [23:11:33] RECOVERY Current Users is now: OK on abusefilter-global-main i-00000512.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [23:11:56] no [23:12:13] RECOVERY Disk Space is now: OK on abusefilter-global-main i-00000512.pmtpa.wmflabs output: DISK OK [23:12:25] OK, lemme scroll up and quote myself :) [23:12:39] "andrewbogott: I believe there's info in the warning message about where the known host is registered and you can remove that line." [23:13:03] RECOVERY Free ram is now: OK on abusefilter-global-main i-00000512.pmtpa.wmflabs output: OK: 690% free memory [23:13:19] Are you looking for "The RSA host key for abusefilter-global-main.pmtpa.wmflabs has changed, and the key for the corresponding IP address 10.4.1.38" ? [23:14:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:14:27] 690% free memory ? huh? [23:14:31] Doesn't the complete warning message include a bit like "Offending line: ~/.ssh/known_hosts: 68" or some such? [23:14:43] Offending key in /home/pirsquared/.ssh/known_hosts:1 [23:15:36] OK, so… do you understand what's going on? [23:15:48] I'm happy to explain if this is new. [23:15:52] it is [23:16:20] So when you first connected to the old instance, by name, ssh said something like "this is a new system you've never connected to are you sure you know what you're doing" [23:16:23] and you said 'yes' [23:16:30] So it made a note of that IP address and hostname. [23:16:43] Now you have created a new instance. It has a different IP address but a different hostname. [23:16:51] Ssh assumes, therefore, that you are being scammed. [23:16:57] hi, how do I get an account at https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots? It seems no one reads https://labsconsole.wikimedia.org/wiki/Nova_Resource_Talk:Bots [23:17:42] PiRQuadratus: Making sense so far? [23:18:05] andrewbogott: yes, but why don't you answer Mailaender first? -- I can wait some mroe [23:18:49] PiRQuadratus: Actually I don't have much to do with bots, I'm hoping someone else is lurking here and will answer Mailaender. Like maybe Damianz [23:19:01] okay [23:19:25] so do I delete this /home/pirsquared/.ssh/known_hosts line? [23:19:32] PiRQuadratus: So, ssh made a note of the old IP and name associated with your old instance. That note is at the filename:linenumber that it is reporting in that error. [23:19:42] so, yeah, if you delete that line, then ssh well be renewedly ignorant about that hostname. [23:19:50] No one does read there before we don't have a project access thing properly setup. [23:19:59] Which means that next when you try to connect you'll be back at the 'this is the first time blah blah blah' message. [23:20:08] I can add your account when I find my phone. [23:20:22] Damianz: Maybe y'all should remove that page and purge links to it if no one reads it? [23:20:31] Otherwise it seems like a trap [23:21:01] The FAQ only says: go find a project and join it, but no further explanation how and where. [23:21:26] oh damn this channel is logged, there goes my privacy... *argh* [23:22:45] andrewbogott: done, and works [23:22:51] cool! [23:22:51] but it still doesn't work on http [23:23:09] PiRQuadratus: OK, so now there are two remaining problems. [23:23:44] First of all, your browser can't connect to that host because it's not on a public network. It is essentially invisible… I'm not sure what URL you would use even. [23:23:49] Secondly: Mediawiki didn't install right :( [23:24:09] second seems more important [23:24:29] Well, without the first how will you know when it's working? And/or test anything ever? [23:24:39] hm that's true [23:24:43] andrewbogott: maybe we should have a structured documentation setup with workflows for noobs? Or an rc feed so people know when talk pages are edited? [23:25:23] Damianz: Maybe. In theory individual projects are supposed to be self-organizing I think. But it would be nice for us to provide better collaboration tools on labsconsole. [23:25:44] As it is I can just shrug and say "I know nothing of this project!" which is a big timesaver :) [23:26:09] Global documentation, especially as we move into more services is important. [23:26:40] PiRQuadratus, I use SOCKS to access labs machines. That's documented here: https://labsconsole.wikimedia.org/wiki/Help:Access#Accessing_web_services_using_a_SOCKS_proxy [23:27:02] It involves a firefox plugin and a running ssh tunnel on your local machine. I found it to be easy but others have disagreed :) [23:27:16] andrewbogott: can I set it up so other people can use it? [23:27:20] Without a proxy? [23:27:56] Yes, we can assign it a public IP if you want other people to visit. [23:28:17] though those are in very limited supply [23:28:22] Public IPs are a bit scarce so we don't use them for every single instance. But it's very useful if you're sharing a system with other non-developer non-sysadmin folks. [23:28:29] (unless ARIN gave us a new block and I didn't hear about it) [23:29:02] Jasper_Deng: how scarce? [23:29:21] Jasper_Deng: We got a few dozen more but still not enough for every instance. [23:29:41] PiRQuadratus: until that new allocation ^ [23:29:46] they were basically gone [23:29:50] :S [23:29:57] (at least for IPv4) [23:30:10] * Jasper_Deng wishes Labs started supporting IPv6.... [23:32:10] andrewbogott: can I enable the IP and remove it later? [23:33:13] I'd prefer that you figure out about proxying. It's a pretty necessary aspect of using labs. [23:33:45] Or so I imagine… Damianz, do you think that people mostly succeed in getting SOCKs set up, or mostly fail? [23:34:07] Probably fail [23:34:25]