[00:01:02] RECOVERY Current Load is now: OK on rt-puppetdev3.pmtpa.wmflabs 10.4.0.195 output: OK - load average: 0.09, 0.62, 0.47 [00:01:42] RECOVERY Disk Space is now: OK on rt-puppetdev3.pmtpa.wmflabs 10.4.0.195 output: DISK OK [00:02:22] RECOVERY Free ram is now: OK on rt-puppetdev3.pmtpa.wmflabs 10.4.0.195 output: OK: 91% free memory [00:08:23] PROBLEM Free ram is now: CRITICAL on rt-puppetdev2.pmtpa.wmflabs 10.4.0.24 output: CHECK_NRPE: Socket timeout after 10 seconds. [01:10:02] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 175 processes [01:14:53] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 96 processes [03:05:53] PROBLEM Free ram is now: CRITICAL on bots-4.pmtpa.wmflabs 10.4.0.64 output: Critical: 4% free memory [03:15:52] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 8% free memory [04:09:18] Change on 12mediawiki a page OAuth was modified, changed by Superm401 link https://www.mediawiki.org/w/index.php?diff=638304 edit summary: [+41] /* Relevant bugs */ clarify, add title of main one [04:36:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 6.94, 6.40, 5.62 [04:39:33] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [04:52:32] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [04:57:23] PROBLEM Free ram is now: WARNING on stackfarm-sql2.pmtpa.wmflabs 10.4.1.23 output: Warning: 19% free memory [05:40:23] PROBLEM Current Load is now: WARNING on nagios-main.pmtpa.wmflabs 10.4.0.120 output: WARNING - load average: 4.47, 5.83, 5.18 [05:50:23] RECOVERY Current Load is now: OK on nagios-main.pmtpa.wmflabs 10.4.0.120 output: OK - load average: 3.62, 4.24, 4.74 [06:28:34] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 151 processes [06:30:23] PROBLEM Total processes is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS WARNING: 151 processes [06:30:53] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 154 processes [06:35:22] RECOVERY Total processes is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS OK: 146 processes [06:45:54] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 147 processes [06:58:42] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 145 processes [07:26:53] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.70, 4.90, 4.99 [08:37:33] RECOVERY Free ram is now: OK on stackfarm-sql2.pmtpa.wmflabs 10.4.1.23 output: OK: 23% free memory [08:37:33] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [08:55:32] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [09:10:33] PROBLEM Free ram is now: WARNING on stackfarm-sql2.pmtpa.wmflabs 10.4.1.23 output: Warning: 19% free memory [10:17:23] PROBLEM Total processes is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS WARNING: 152 processes [11:23:53] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 160 processes [11:42:41] !ping [11:42:41] pong [12:07:22] RECOVERY Total processes is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS OK: 146 processes [12:40:33] RECOVERY Free ram is now: OK on stackfarm-sql2.pmtpa.wmflabs 10.4.1.23 output: OK: 23% free memory [12:40:33] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [12:43:53] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 142 processes [12:48:33] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [13:08:32] PROBLEM Free ram is now: WARNING on stackfarm-sql2.pmtpa.wmflabs 10.4.1.23 output: Warning: 19% free memory [13:25:23] PROBLEM Total processes is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS WARNING: 152 processes [13:26:53] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 155 processes [14:19:29] anybody knows why this is broken? http://en.wikipedia.beta.wmflabs.org/ [14:21:01] sort-of [14:27:16] !ping [14:27:16] pong [14:27:24] :o [14:27:33] @infobot-detail ping [14:27:33] Info for ping: this key was created at N/A by N/A, this key was displayed 95 time(s), last time at 1/31/2013 2:27:16 PM (00:00:16.9032940 ago) this key is normal [14:27:43] mm, popular one [14:47:29] @labs-resolve deployment-apa [14:47:29] I don't know this instance - aren't you are looking for: I-0000031a (deployment-apache32), I-0000031b (deployment-apache33), [14:53:24] !log deployment-prep restarted squid and rebooted apache32 [14:53:26] Logged the message, Master [15:15:33] hashar ping [15:15:40] there is some problem on beta [15:15:42] network unreacheable [15:15:42] it's slow [15:15:45] :o [15:15:47] fill a bug :-D [15:15:50] where [15:15:55] oh [15:15:56] lol [15:16:00] I am about to leave so can't do anything yet [15:16:07] I mean your network is unreachable? how u on irc [15:16:14] ok [15:17:14] not even slow [15:17:17] it is just dead :-] [15:17:39] it's slow to me [15:17:44] I can do things there [15:18:04] got no clue what's uo [15:18:05] up [15:18:07] :/ [15:18:38] !log deployment-prep starting apache2 on -apache32 [15:18:40] Logged the message, Master [15:19:40] !log deployment-prep restarting squid process on deployment-squid [15:19:42] Logged the message, Master [15:22:34] [15:22:37] [15:22:42] getting http://en.wikipedia.beta.wmflabs.org/wiki/Choir [15:27:54] soo hmm [15:28:06] petan: the file are generated fastly by the apaches [15:28:16] but everything takes a long time to download [15:28:20] maybe that is my internet connection [15:28:21] or labs [15:29:19] no it's not [15:29:25] everyone has this problem [15:34:17] at least that is not the apaches [15:35:13] form deployment-squid I get the files properly [15:35:35] deployment-squid$ curl -x 10.4.0.166:80 http://en.wikipedia.beta.wmflabs.org/wiki/Guggenmusik [15:35:38] soo [15:35:46] that must be squid :-D [15:40:25] cache_dir ufs /data/project/squid1 8000MB 16 400 [15:40:28] maybe gluster [15:41:21] [2013-01-31 15:41:08.056843] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-deployment-prep-project-replicate-1: missing entry /squid1/00/8F/0000E029 on deployment-prep-project-client-3 [15:41:22] boooo [15:42:35] !log deployment-prep cleaned out deployment-squid:/mnt/ (add an old enwiki dump and some squid files [15:42:37] Logged the message, Master [15:46:01] !log deployment-prep stoping squid, migrating ufs cache from /data/project/squid1 (gluster) to /mnt/squid_cache [15:46:03] Logged the message, Master [15:46:29] hashar why u restore enwiki dump? just delete it all [15:46:38] it will create new cache [15:48:04] I am not resting it [15:48:12] I said "cleaned out" [15:48:13] :-D [15:48:38] you said add [15:48:46] ahh [15:48:50] s/add/had/ [15:48:51] sorry [15:48:54] oh lol [15:49:24] !log deployment-prep Deleting out /data/project/squid1 which has been migrated to /mnt/squid_cache. The gluster volume for data-project is corrupted on beta so we don't want to use it anymore. [15:49:26] Logged the message, Master [15:49:27] that should be fixed [15:49:30] brb [15:50:54] you guys should give me some feedback on a new skin I'm trying out [15:50:54] https://nova-precise2.pmtpa.wmflabs/wiki/Main_Page [15:51:29] Internet Explorer cannot display the webpage [15:51:33] XD [15:51:34] lol [15:51:35] socks proxy ;) [15:51:37] url [15:52:14] I know I need to move the login/create account links outside of the dropdown [15:53:11] baah that sucks [15:53:15] not the skin [15:53:23] I created a tunnel but it redirect me to port 80 [15:53:27] :D [15:53:28] and I opened it on another port lol [15:53:44] it should redirect to 443 [15:54:01] are you going to be at fosdem? [15:54:30] who [15:54:38] I am going to be on amsterdam meetup [15:54:43] no idea where is fosdem [15:54:48] neither when [15:54:49] belgium [15:54:52] ah [15:54:53] starting tomorrow [15:54:58] ok, I won't be there :D [15:55:26] ah. ok [15:55:32] well, ams meetup then :) [16:01:34] ok. boarding time [16:01:41] * Ryan_Lane waves [16:04:12] petan: ok seems solved :-] [16:04:16] ok [16:04:18] I am heading out of my coworking place *wave* [16:04:24] cya [16:04:27] there are still a few glitches though [16:04:30] will look at them later [16:26:52] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 147 processes [16:30:23] RECOVERY Total processes is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS OK: 146 processes [16:38:32] RECOVERY Free ram is now: OK on stackfarm-sql2.pmtpa.wmflabs 10.4.1.23 output: OK: 23% free memory [16:38:32] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [16:39:57] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 155 processes [16:43:22] PROBLEM Total processes is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS WARNING: 151 processes [16:46:32] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [17:06:32] PROBLEM Free ram is now: WARNING on stackfarm-sql2.pmtpa.wmflabs 10.4.1.23 output: Warning: 19% free memory [17:44:53] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 148 processes [18:08:23] RECOVERY Total processes is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS OK: 148 processes [19:01:33] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 153 processes [19:03:53] PROBLEM Current Load is now: CRITICAL on rt-puppetdev4.pmtpa.wmflabs 10.4.0.24 output: Connection refused by host [19:04:33] PROBLEM Disk Space is now: CRITICAL on rt-puppetdev4.pmtpa.wmflabs 10.4.0.24 output: Connection refused by host [19:05:14] PROBLEM Free ram is now: CRITICAL on rt-puppetdev4.pmtpa.wmflabs 10.4.0.24 output: Connection refused by host [19:08:54] RECOVERY Current Load is now: OK on rt-puppetdev4.pmtpa.wmflabs 10.4.0.24 output: OK - load average: 0.13, 0.61, 0.43 [19:09:34] RECOVERY Disk Space is now: OK on rt-puppetdev4.pmtpa.wmflabs 10.4.0.24 output: DISK OK [19:10:12] RECOVERY Free ram is now: OK on rt-puppetdev4.pmtpa.wmflabs 10.4.0.24 output: OK: 91% free memory [19:12:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 6.86, 6.65, 5.65 [19:38:13] PROBLEM Free ram is now: WARNING on rt-puppetdev4.pmtpa.wmflabs 10.4.0.24 output: Warning: 15% free memory [19:53:52] PROBLEM Current Load is now: CRITICAL on rt-puppetdev5.pmtpa.wmflabs 10.4.0.201 output: Connection refused by host [19:54:32] PROBLEM Disk Space is now: CRITICAL on rt-puppetdev5.pmtpa.wmflabs 10.4.0.201 output: Connection refused by host [19:55:13] PROBLEM Free ram is now: CRITICAL on rt-puppetdev5.pmtpa.wmflabs 10.4.0.201 output: Connection refused by host [19:56:33] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [19:58:53] RECOVERY Current Load is now: OK on rt-puppetdev5.pmtpa.wmflabs 10.4.0.201 output: OK - load average: 0.14, 0.60, 0.46 [19:59:33] RECOVERY Disk Space is now: OK on rt-puppetdev5.pmtpa.wmflabs 10.4.0.201 output: DISK OK [20:00:12] RECOVERY Free ram is now: OK on rt-puppetdev5.pmtpa.wmflabs 10.4.0.201 output: OK: 82% free memory [20:05:13] PROBLEM Free ram is now: WARNING on bots-3.pmtpa.wmflabs 10.4.0.59 output: Warning: 19% free memory [20:23:52] PROBLEM Current Load is now: CRITICAL on rt-puppetdev6.pmtpa.wmflabs 10.4.0.24 output: Connection refused by host [20:24:32] PROBLEM Disk Space is now: CRITICAL on rt-puppetdev6.pmtpa.wmflabs 10.4.0.24 output: Connection refused by host [20:25:12] PROBLEM Free ram is now: CRITICAL on rt-puppetdev6.pmtpa.wmflabs 10.4.0.24 output: Connection refused by host [20:26:42] PROBLEM Total processes is now: CRITICAL on rt-puppetdev6.pmtpa.wmflabs 10.4.0.24 output: Connection refused by host [20:27:22] PROBLEM dpkg-check is now: CRITICAL on rt-puppetdev6.pmtpa.wmflabs 10.4.0.24 output: Connection refused by host [20:28:52] RECOVERY Current Load is now: OK on rt-puppetdev6.pmtpa.wmflabs 10.4.0.24 output: OK - load average: 0.61, 0.91, 0.52 [20:29:32] RECOVERY Disk Space is now: OK on rt-puppetdev6.pmtpa.wmflabs 10.4.0.24 output: DISK OK [20:30:13] RECOVERY Free ram is now: OK on rt-puppetdev6.pmtpa.wmflabs 10.4.0.24 output: OK: 91% free memory [20:31:43] RECOVERY Total processes is now: OK on rt-puppetdev6.pmtpa.wmflabs 10.4.0.24 output: PROCS OK: 83 processes [20:32:23] RECOVERY dpkg-check is now: OK on rt-puppetdev6.pmtpa.wmflabs 10.4.0.24 output: All packages OK [20:40:12] RECOVERY Free ram is now: OK on bots-3.pmtpa.wmflabs 10.4.0.59 output: OK: 20% free memory [20:41:32] RECOVERY Free ram is now: OK on stackfarm-sql2.pmtpa.wmflabs 10.4.1.23 output: OK: 22% free memory [20:41:32] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [20:48:12] PROBLEM Free ram is now: WARNING on bots-3.pmtpa.wmflabs 10.4.0.59 output: Warning: 18% free memory [20:59:32] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [21:03:12] PROBLEM Free ram is now: WARNING on rt-puppetdev6.pmtpa.wmflabs 10.4.0.24 output: Warning: 10% free memory [21:04:32] PROBLEM Free ram is now: WARNING on stackfarm-sql2.pmtpa.wmflabs 10.4.1.23 output: Warning: 18% free memory [21:10:02] PROBLEM host: rt-puppetdev5.pmtpa.wmflabs is DOWN address: 10.4.0.201 CRITICAL - Host Unreachable (10.4.0.201) [21:13:53] PROBLEM Current Load is now: CRITICAL on rt-puppetdev7.pmtpa.wmflabs 10.4.0.195 output: Connection refused by host [21:14:33] PROBLEM Disk Space is now: CRITICAL on rt-puppetdev7.pmtpa.wmflabs 10.4.0.195 output: Connection refused by host [21:18:52] RECOVERY Current Load is now: OK on rt-puppetdev7.pmtpa.wmflabs 10.4.0.195 output: OK - load average: 0.09, 0.59, 0.48 [21:19:33] RECOVERY Disk Space is now: OK on rt-puppetdev7.pmtpa.wmflabs 10.4.0.195 output: DISK OK [22:22:52] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.84, 4.87, 4.97 [22:43:05] Looks like labsconsole blew up [22:43:27] http://paste.marktraceur.info/23 [22:43:28] <^demon> I saw something about memcached on virt0 going down :\ [22:43:44] Seems about right [22:44:05] And spagewmf just said something about scap being faster--maybe it crashed? [22:44:18] <^demon> memcached on virt0 wouldn't affect scap. [22:44:31] Hm. [22:45:01] I just restarted several things on virt0 but… no change. [22:46:35] andrewbogott: can you please take care of my account? [22:47:52] matanya: 'take care of'? [22:48:13] create an instance for me to use [22:51:14] matanya: We are in the midst of another catastrophe but yes, soon. [22:51:53] same as every day :) thanks, If I can help let me know. [22:53:11] Yay, catastrophe [22:56:04] I guess I should say, 'midst of an inconvenience' [23:11:02] andrewbogott: Any idea when it'll be back? [23:11:26] nope! I'm not having much luck troubleshooting… just rebooted, hoping it comes back up and is magically fixed :) [23:12:14] that never happens [23:13:26] * andrewbogott is an optimist, which is why he probably shouldn't have logins on any of these machines [23:14:03] !log rebooting virt0 in a fit of optimism and/or desperation [23:14:21] rebooting is not a valid project. [23:14:38] Heh. [23:15:02] Is Ryan not working today? [23:15:23] Is he going to FOSDEM perhaps? [23:15:30] Pretty much everyone is on jets to Belgium right now [23:16:03] Hah! fixed :) [23:16:07] For the moment [23:16:21] Sweet. [23:16:37] I didn't realize people had already left for FOSDEM, but that certainly explains it [23:16:57] magic [23:17:09] andrewbogott: I'll take that as a tip [23:18:27] OK, funky. [23:19:00] andrewbogott: The other reason I came to the labsconsole was to check on the orgchart instance, which apparently lost its public IP sometime between this morning and now. [23:19:18] matanya: In three or five minutes you should have an instance, 'nagios-dev'. With sudo. [23:19:43] marktraceur: /that/ shouldn't happen… is it currently listed as having a public IP? [23:19:44] thanks a lot andrewbogott [23:20:23] andrewbogott: Now it is, because I reassociated it. [23:20:41] Fair enough. No idea why it was lost (unless someone removed it) [23:21:11] andrewbogott: The funny thing is, the IP address said it was associated with orgchart, but orgchart didn't reciprocate. i.e. the instances page didn't connect the instance to the IP. [23:24:06] PROBLEM Current Load is now: CRITICAL on nagios-dev.pmtpa.wmflabs 10.4.0.201 output: Connection refused by host [23:24:24] marktraceur, I'm going to grap coffee… email me if things break again and you notice? [23:24:33] Righto [23:24:42] Probably someone else will notice first, but I'll forward it on. [23:24:50] 'grap'? [23:25:05] PROBLEM Disk Space is now: CRITICAL on nagios-dev.pmtpa.wmflabs 10.4.0.201 output: Connection refused by host [23:25:29] matanya: ^^ things like that are birthing pains, means the instance is coming up. [23:25:38] PROBLEM Free ram is now: CRITICAL on nagios-dev.pmtpa.wmflabs 10.4.0.201 output: Connection refused by host [23:25:46] good hint :) [23:29:00] andrewbogott: is their a reason I'm not listed at https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bastion as a member? [23:29:38] RECOVERY Current Load is now: OK on nagios-dev.pmtpa.wmflabs 10.4.0.201 output: OK - load average: 0.40, 0.82, 0.52 [23:30:43] RECOVERY Free ram is now: OK on nagios-dev.pmtpa.wmflabs 10.4.0.201 output: OK: 90% free memory [23:31:23] RECOVERY Disk Space is now: OK on nagios-dev.pmtpa.wmflabs 10.4.0.201 output: DISK OK [23:36:18] PROBLEM Free ram is now: WARNING on nagios-main.pmtpa.wmflabs 10.4.0.120 output: Warning: 16% free memory [23:40:38] PROBLEM Current Load is now: WARNING on nagios-main.pmtpa.wmflabs 10.4.0.120 output: WARNING - load average: 8.96, 9.38, 7.71 [23:41:17] RECOVERY Free ram is now: OK on nagios-main.pmtpa.wmflabs 10.4.0.120 output: OK: 36% free memory [23:55:41] RECOVERY Current Load is now: OK on nagios-main.pmtpa.wmflabs 10.4.0.120 output: OK - load average: 0.89, 1.30, 3.85 [23:58:42] andrewbogott: I can't log in