[00:13:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:24:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [00:25:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:44:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:54:34] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [00:55:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:06:32] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 178 processes [01:11:33] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 98 processes [01:14:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:24:42] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [01:26:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:44:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:55:52] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [01:56:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:14:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:25:52] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [02:27:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:40:44] RECOVERY Free ram is now: OK on swift-be1 i-000001c7.pmtpa.wmflabs output: OK: 20% free memory [02:41:54] RECOVERY Free ram is now: OK on wikistream-1 i-0000016e.pmtpa.wmflabs output: OK: 25% free memory [02:41:54] RECOVERY Free ram is now: OK on sube i-000003d0.pmtpa.wmflabs output: OK: 41% free memory [02:44:44] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:49:53] PROBLEM Free ram is now: WARNING on wikistream-1 i-0000016e.pmtpa.wmflabs output: Warning: 12% free memory [02:55:54] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [02:57:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:08:42] PROBLEM Free ram is now: WARNING on swift-be1 i-000001c7.pmtpa.wmflabs output: Warning: 18% free memory [03:16:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:22:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 5.11, 5.20, 5.10 [03:26:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [03:28:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:31:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [03:36:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [03:37:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.79, 4.91, 4.98 [03:41:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [03:46:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:56:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [03:58:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:13:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 5.08, 5.09, 5.06 [04:16:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:26:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [04:28:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:46:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:56:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [04:58:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:16:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:27:22] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [05:28:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 5.43, 5.29, 5.31 [05:29:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:46:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:54:59] Change on 12mediawiki a page Developer access was modified, changed by Sixthhokage1 link https://www.mediawiki.org/w/index.php?diff=617192 edit summary: [05:57:24] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [05:59:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:03:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.61, 4.82, 5.00 [06:17:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:27:24] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [06:29:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:31:32] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:35:33] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 152 processes [06:47:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:47:43] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 157 processes [06:55:33] RECOVERY Total processes is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS OK: 145 processes [06:56:33] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:58:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [06:59:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:03:33] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 152 processes [07:17:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:28:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [07:29:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:32:42] PROBLEM Current Load is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: WARNING - load average: 6.83, 5.95, 5.93 [07:39:06] Change on 12mediawiki a page Developer access was modified, changed by Rjain link https://www.mediawiki.org/w/index.php?diff=617200 edit summary: /* Requests */ [07:48:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:51:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [07:58:52] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [08:00:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:18:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:25:55] anyone available to (stop and re)?start a series of bots on bots-3? [08:28:55] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [08:29:57] @labs-user Petrb [08:29:57] Petrb is member of 10 projects: Bastion, Bots, Configtest, Deployment-prep, Deployment-prepbackup, Gareth, Huggle, Hugglewa, Nagios, Openstack, [08:30:20] @labs-user Ryan Lane [08:30:20] Ryan Lane is member of 44 projects: Analytics, Bastion, Bots, Category-sorting, Cvresearch, Demo, Deployment-prep, Deployment-prepbackup, Dumps, Editor-engagement, Etherpad, Feeds, Ganglia, Gareth, Gerrit, Gluster, Integration, Juju, Linkcache, Lvs-labs, Mail, Mailman, Maps, Mapstory, Mediahandler, Mediawiki-bugfix, Membase, Mobile, Mobile-sms, Mobile-stats, Mwreview, Nagios, Nginx, Opengrok, Openstack, Otrs, Outreach, Patchtest, Pediapress, Phabricator, Project-proxy, Publicdata, Puppet, Puppet-cleanup, [08:30:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:30:33] @labs-project-info bastion [08:30:33] The project Bastion has 2 instances and 241 members, description: The project to allow Labs users to connect to other instances within Wikimedia Labs via tunneling/proxy. [08:30:43] @labs-project-users bastion [08:30:43] Following users are in this project (showing all241 members): Andre Engels, Aaron Schulz, Abartov, Abe.music, Addshore, Adminxor, Akhanna, Chughakshay16, Alejrb, Amgine, Amire80, Andrew Bogott, Anomie, Apmon, ArielGlenn, Arky, Asher, Dash1291, [08:31:03] mm [08:31:05] :/ [08:32:12] @labs-user billinghurst [08:32:12] That user is not a member of any project [08:32:17] shame [08:32:25] ;-) [08:33:06] one day I will find my labs account again, and get ssh installed and do other things to get off my arse [08:35:21] @labs-user Billinghurst [08:35:21] That user is not a member of any project [08:35:21] :P [08:35:41] @labs-project-users bastion [08:35:41] Following users are in this project (showing all241 members): Andre Engels, Aaron Schulz, Abartov, Abe.music, Addshore, Adminxor, Akhanna, Chughakshay16, Alejrb, Amgine, Amire80, Andrew Bogott, Anomie, Apmon, ArielGlenn, Arky, Asher, Dash1291, [08:35:58] um [08:35:58] something is wrong [08:39:00] !ping [08:39:01] pong [08:39:05] @labs-project-users bastion [08:39:05] Following users are in this project (displaying 18 of 241 total): Andre Engels, Aaron Schulz, Abartov, Abe.music, Addshore, Adminxor, Akhanna, Chughakshay16, Alejrb, Amgine, Amire80, Andrew Bogott, Anomie, Apmon, ArielGlenn, Arky, Asher, Dash1291, [08:39:09] better [08:40:59] @labs-project-users wikidata-dev [08:40:59] I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 9 seconds ago [08:41:25] :/ [08:45:33] @labs-project-users wikidata-dev [08:45:33] Following users are in this project (displaying 17 of 18 total): Aaron Schulz, Ataherivand, Aude, CSteipp, Daniel Kinzler, Demon, Hashar, Jens Ohlig, Ryan Lane, Novaadmin, Reedy, RobLa, Silke Meyer, Tim Starling, Denny Vrandecic, Anja Jentzsch, Andrew Bogott, [08:45:40] here we go [08:46:06] @labs-user Ryan Lane [08:46:06] Ryan Lane is member of 72 projects: Analytics, Bastion, Bots, Category-sorting, Cvresearch, Demo, Deployment-prep, Deployment-prepbackup, Dumps, Editor-engagement, Etherpad, Feeds, Ganglia, Gareth, Gerrit, Gluster, Integration, Juju, Linkcache, Lvs-labs, Mail, Mailman, Maps, Mapstory, Mediahandler, Mediawiki-bugfix, Membase, Mobile, Mobile-sms, Mobile-stats, Mwreview, Nagios, Nginx, Opengrok, Openstack, Otrs, Outreach, Patchtest, Pediapress, Phabricator, Project-proxy, Publicdata, Puppet, Puppet-cleanup, Queue, Reportcard, Selenium, Shop-analytics, Signwriting, Simplewiki, Statsgrokse, Storage, Sugarcrm, Swift, Swiftupgrade, Syslog-collection, Testing, Testlabs, Translation-memory, Turnkey-mediawiki, Video, Visualeditor, Webplatform, Webtools, Wiki-migration, Wikidata-dev, Wikifollower, Wikisource-dev, Wikistats, Wikistream, Wikitrust, Wlmjudging, [08:48:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:55:42] @labs-info bots-4 [08:55:42] [Name bots-4 doesn't exist but resolves to I-000000e8] I-000000e8 is Nova Instance with name: bots-4, host: virt6, IP: 10.4.0.64 of type: m1.small, with number of CPUs: 1, RAM of this size: 2048M, member of project: bots, size of storage: 30 and with image ID: lucid-server-cloudimg-amd64.img [08:58:54] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [09:00:08] petan, if you have time - I have a feeling that there is a HUGE file lingering somewhere on the bots-project, which prevents me from running bots - my quotum is full, but I have deleted all ' big' files in my tree [09:00:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:00:40] the disk is full [09:00:47] labs-nfs1 [09:01:00] it can be on any project [09:01:03] oh, it is just generally full [09:01:06] yes [09:01:09] :-( [09:01:14] * Beetstra deletes petan ;-) [09:01:17] on all projects the /home is 100% [09:01:35] != fun [09:02:56] why are you running bots from home anyway :P [09:03:06] I prefer using /mnt/share [09:03:11] local storage of each instance [09:03:28] you mean, I should dump the logs there .. [09:03:38] /home is small [09:04:20] you can just create /mnt/share/ [09:04:25] and move all files there [09:04:31] then run it from there [09:04:38] I don't see /mnt/share .. [09:04:41] (I am on bots-3 [09:04:44] ok sec [09:05:49] eh, someone deleteed it [09:05:52] it's back [09:06:21] that fs has more storage than /home itself [09:06:33] and ur the only user who use it [09:07:12] !home [09:08:07] !home is don't put big stuff in your ~ it's a small fs, if you are on bots project, you should use local instance storage at /mnt/share which has 20gb on each instance [09:08:07] Key was added [09:12:46] OK .. first three running from there [09:16:22] And fourth one as well [09:16:31] have to play a bit with the settings now, but this is nice [09:18:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:29:32] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [09:30:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:48:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:59:32] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [10:00:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:18:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:29:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [10:30:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:48:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:59:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [11:01:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:18:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:30:02] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [11:31:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:41:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 11% free memory [11:49:04] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:00:54] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [12:01:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:19:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:31:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [12:32:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:51:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:02:22] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [13:02:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:21:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:32:23] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [13:33:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:51:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:02:23] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [14:03:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:21:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:25:41] Change on 12mediawiki a page Developer access was modified, changed by Fomafix link https://www.mediawiki.org/w/index.php?diff=617272 edit summary: Remove mistake [14:33:02] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [14:34:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:39:53] PROBLEM Free ram is now: WARNING on sube i-000003d0.pmtpa.wmflabs output: Warning: 17% free memory [14:52:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:03:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [15:04:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:22:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:33:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [15:34:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:48:33] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 151 processes [15:50:56] hi all, there is a full partition on the bots-1 intstance [15:52:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:03:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [16:05:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:06:41] hi Guest39656 [16:06:59] i [16:07:02] hi [16:07:12] it's me dschwen [16:11:20] $ ssh bastion1.pmtpa.wmflabs 'df -i /home; df /home' 2>/dev/null [16:11:20] Filesystem Inodes IUsed IFree IUse% Mounted on [16:11:20] labs-nfs1:/export/home/bastion 1179648 599329 580319 51% /home [16:11:20] Filesystem 1K-blocks Used Available Use% Mounted on [16:11:20] labs-nfs1:/export/home/bastion 18578176 17660672 0 100% /home [16:11:54] andrewbogott_afk: paravoid: ^^^ [16:13:17] Guest39656: anyway, so this probably effects all instances. (on all projects) would be nice if people cleaned up there own homes but i think only ops could give an overview of who the worst offendors are [16:13:29] Guest39656: (also, only ops can add more space) [16:13:33] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [16:13:58] yeah, thanks anyways [16:22:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:25:37] Well it probably won't get more space [16:25:45] Been set to read only in like 15hours anyway [16:33:52] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [16:36:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:42:04] Damianz: and then how much space does it get at the new place? [16:42:26] Damianz: anyway, my points hold. mortals can't do much beyond delete their own stuff [16:43:31] Damianz: ops can find bad offenders and can decide for longer term (>12 hours) how much capacity to have [16:53:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:03:53] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [17:07:03] jeremyb: It's under project storage so like 300gb by default [17:07:18] And yeah - it's usually 1/2 bots logs or such that can be killed [17:07:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:18:02] Damianz: or compressed efficiently [17:18:28] nah [17:23:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:34:32] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [17:37:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:53:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:04:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [18:06:54] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [18:07:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:21:54] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [18:23:14] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:34:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [18:36:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [18:38:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:53:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:04:34] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [19:09:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:24:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:35:52] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [19:39:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:54:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:05:53] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [20:09:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:24:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:36:32] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [20:40:25] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:54:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:06:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [21:10:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:24:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:37:22] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [21:41:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:54:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:56:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [22:07:22] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [22:11:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:24:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:37:23] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [22:40:52] Hi Ryan_Lane [22:41:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:42:16] Krenair: howdy [22:42:45] nomnomnomnomnom [22:42:55] Ryan_Lane, did you see https://bugzilla.wikimedia.org/show_bug.cgi?id=43176 ? [22:45:17] nope, but it isn't actually broken [22:45:50] maybe I need to add username requirements to the creation page [22:46:07] I hate mediawiki sometimes [22:49:27] seems broken to me [22:54:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:55:15] Damianz: it does? [22:55:22] Damianz: what are you trying? [22:55:30] it won't tell you if a user name is in use [22:55:35] or if it's a bad username [22:55:37] well DamianZaremba_Test and damian_test both shouldn't be in use [22:55:49] Yet it still errored out [22:55:53] _ isn't valid [22:56:03] since when is _ not valid... [22:56:04] -.- [22:56:10] it's not valid in mediawiki [22:56:23] mediawiki converts _ to a space [22:56:45] this is why I hate mediawiki [22:56:45] I guess I'll add requirements to the page [22:56:48] yep [22:58:24] Django makes me happy, it's got like a proper signaling framework and does proper error handling on forms =D [23:00:17] ok, now the signup page lists the restrictions [23:00:34] yeah, did you see the bug I pointed at? [23:02:28] mhm [23:03:38] nice that your bugs also get ignored for years and not just the communities :D [23:03:42] yeah [23:03:50] to be fair, I was community at the time ;) [23:04:03] probably the issue [23:04:22] there we go [23:04:34] now the page also lists what the error means [23:04:41] though realistically, it could mean a number of things [23:04:49] like an actual issue talking to ldap [23:05:20] but realistically, that error means a bad shell account name 99.999% of the time [23:05:23] It's the most obscure error anyway that really means nothing in relation to the actual error [23:05:43] yes [23:05:49] it's terrible [23:07:23] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [23:09:18] seems webplatform.org is growing well: http://frozenice.de/webplatform/charts/ [23:12:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:26:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:28:17] I don't really like measuring views, it's a lame metric of engagement [23:28:27] Though upwards trends are allways nice [23:37:23] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [23:42:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:57:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100%