[00:02:24] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [00:32:52] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [00:34:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [01:02:53] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [01:08:52] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 168 processes [01:13:53] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 93 processes [01:17:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [01:32:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [01:32:53] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [02:04:32] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [02:15:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [02:34:32] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [02:40:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 23% free memory [02:40:52] RECOVERY Free ram is now: OK on swift-be1.pmtpa.wmflabs 10.4.0.107 output: OK: 20% free memory [02:53:53] PROBLEM Free ram is now: WARNING on swift-be1.pmtpa.wmflabs 10.4.0.107 output: Warning: 18% free memory [03:04:33] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [03:30:42] RECOVERY Free ram is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: OK: 48% free memory [03:35:53] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [03:49:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [03:59:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [04:05:53] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [04:32:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [04:36:12] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [04:43:22] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory [05:06:12] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [05:08:22] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 44% free memory [05:36:13] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [06:02:54] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [06:06:13] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [06:11:54] * Beetstra wonders if someone can help him getting COIBot and XLinkBot back online now bots-3 is dead [06:15:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [06:28:54] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 155 processes [06:30:52] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 152 processes [06:30:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [06:36:33] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [06:37:53] PROBLEM dpkg-check is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: DPKG CRITICAL dpkg reports broken packages [06:48:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [06:55:53] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 148 processes [07:06:52] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [07:23:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [07:36:52] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [08:06:53] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [08:33:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [08:36:52] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [08:46:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [09:01:39] !log wikidata-dev wikidata-dev-9: disabled memcached again on devrepo and devclient. Error message: "Could not acquire 'devclient:messages:en:status' lock." [09:01:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [09:06:52] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [09:36:52] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [09:54:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [10:02:38] !log bots petrb: started AfC bot on bots-1 [10:06:53] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [10:36:53] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [11:04:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [11:07:22] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [11:12:03] PROBLEM dpkg-check is now: CRITICAL on wikidata-puppet-repo.pmtpa.wmflabs 10.4.1.52 output: DPKG CRITICAL dpkg reports broken packages [11:22:02] RECOVERY dpkg-check is now: OK on wikidata-puppet-repo.pmtpa.wmflabs 10.4.1.52 output: All packages OK [11:22:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [11:32:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [11:37:22] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [11:44:05] ACKNOWLEDGEMENT host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79) [12:25:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [12:29:52] hello! I tried to register on wikilabs but captcha isn't shown for me in all browsers... how can it be avoided? [13:30:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [13:53:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [13:57:43] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [13:59:15] memcache seems down again [14:03:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [14:31:20] any puppet people here? [14:50:53] PROBLEM Current Load is now: CRITICAL on authorlinks.pmtpa.wmflabs 10.4.1.62 output: Connection refused by host [14:51:03] PROBLEM Free ram is now: CRITICAL on authorlinks.pmtpa.wmflabs 10.4.1.62 output: Connection refused by host [14:51:33] PROBLEM Current Users is now: CRITICAL on authorlinks.pmtpa.wmflabs 10.4.1.62 output: Connection refused by host [14:52:13] PROBLEM Disk Space is now: CRITICAL on authorlinks.pmtpa.wmflabs 10.4.1.62 output: Connection refused by host [14:52:23] PROBLEM Total processes is now: CRITICAL on authorlinks.pmtpa.wmflabs 10.4.1.62 output: Connection refused by host [14:52:53] PROBLEM dpkg-check is now: CRITICAL on authorlinks.pmtpa.wmflabs 10.4.1.62 output: Connection refused by host [14:53:46] giftpflanze memcache? [14:53:51] huh? [14:54:19] it's working again [14:54:26] I have no idea what you mean [14:54:30] memcache where? [14:54:36] virt0 [14:54:43] on labsconsole? [14:54:49] if it dies the wiki is damn slow [14:54:54] oh that [14:54:58] and captchas aren't shown [14:55:32] there are some memcache machines on labs too, so I wasn't sure which you mean [14:55:42] @labs-resolve memcache [14:55:42] I don't know this instance - aren't you are looking for: I-00000153 (memcache-puppet), I-0000053a (wikidata-dev-memcached), [14:55:53] ah, ok [14:56:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [15:00:52] RECOVERY Current Load is now: OK on authorlinks.pmtpa.wmflabs 10.4.1.62 output: OK - load average: 1.53, 1.18, 0.66 [15:01:02] RECOVERY Free ram is now: OK on authorlinks.pmtpa.wmflabs 10.4.1.62 output: OK: 683% free memory [15:01:07] 01/02/2013 - 15:01:06 - Updating keys for tpt at /export/keys/tpt [15:01:32] RECOVERY Current Users is now: OK on authorlinks.pmtpa.wmflabs 10.4.1.62 output: USERS OK - 1 users currently logged in [15:02:12] RECOVERY Disk Space is now: OK on authorlinks.pmtpa.wmflabs 10.4.1.62 output: DISK OK [15:02:22] RECOVERY Total processes is now: OK on authorlinks.pmtpa.wmflabs 10.4.1.62 output: PROCS OK: 88 processes [15:02:52] RECOVERY dpkg-check is now: OK on authorlinks.pmtpa.wmflabs 10.4.1.62 output: All packages OK [15:10:52] PROBLEM dpkg-check is now: CRITICAL on authorlinks.pmtpa.wmflabs 10.4.1.62 output: DPKG CRITICAL dpkg reports broken packages [15:22:42] I've an isse with the "Allocate IP" function. [15:22:44] When I try to add a new IP to wikisource-tools project I have the error "Failed to allocate new public IP address." [15:23:10] I've used the url https://labsconsole.wikimedia.org/w/index.php?title=Special:NovaAddress&action=allocate&project=wikisource-tools®ion=pmtpa to add the IP [15:57:23] bla bla... [15:57:39] ok it works [15:58:45] @seenrx mardetanha [15:58:46] petan: mardetanha is in here, right now (multiple results were found: mardetanha1) [15:59:39] petan: Ok. Thanks. I need it to allow access to a tool that search and output links between author pages of wikisource, commons, wikipedia and viaf. [16:00:37] ok, if only I remember where to ask, try filling in a ticket or send a mail [16:00:38] !mail [16:00:38] we have a mailing list labs-l@lists.wikimedia.org feel free to send a message there, don't forget to subscribe [16:01:40] petan: The doc says that ask on IRC is an option https://labsconsole.wikimedia.org/wiki/Help:Addresses#Request_a_Public_IP_address [16:01:48] aha [16:01:50] interesting [16:01:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [16:01:59] @search ops [16:01:59] No results were found, remember, the bot is searching through content of keys and their names [16:02:05] @searchrx ops [16:02:05] No results were found, remember, the bot is searching through content of keys and their names [16:02:12] @searchrx Ryan [16:02:12] No results were found, remember, the bot is searching through content of keys and their names [16:02:27] hmm [16:02:44] @seen Paravoid [16:02:44] petan: Paravoid is in here, right now [16:02:53] wtf my autocompletion is borked [16:03:02] ok in that case paravoid can handle this request [16:03:05] I guess [16:04:07] hm? [16:04:18] paravoid see request of Tpt [16:05:07] <^demon> Yay, incomplete apis. [16:24:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [17:43:43] Tpt, still need a public IP? [17:43:54] andrewbogott: yes [17:44:02] oh [17:44:03] sorry [17:44:13] forgot about it [17:44:15] my bad... [17:44:26] multitasking like crazy [17:44:38] paravoid/Tpt I'll do it [17:44:42] Tpt, project name? [17:45:05] andrewbogott: wikisource-tools [17:46:43] Tpt: Looks like there's one allocated already… you need a second? [17:46:52] (I don't object, just making sure everything gets used) [17:47:47] andrewbogott: Yes, I need a second one for a completely different tool. [17:47:53] 'k [17:48:10] Tpt: OK, should be all set [17:49:22] andrewbogott: It works fine. Thanks a lot! [18:25:34] andrewbogott: User account X! requested shell access. Do you think user X! is a valid username? [18:26:31] mike_want: Looks like it is: http://en.wikipedia.org/wiki/User:X! [18:26:49] But of course it won't work as a shell login [18:26:55] *mike_wang that is [18:28:09] X is a user, he does cbng stuff sometimes [18:28:56] Damianz: Yep, I'm just surprised (as, I suspect, is mike) that mediawiki doesn't choke on having a trailing ! in a username [18:29:11] It doesn't like _ which is annoying [18:31:48] 01/02/2013 - 18:31:46 - Creating a home directory for rubin at /export/keys/rubin [18:36:22] 01/02/2013 - 18:36:21 - Updating keys for rubin at /export/keys/rubin [18:38:06] hello once again :) can someone add me to bastion, please? [18:38:34] you should already be in [18:40:54] Silke_WMDE: Regarding your email… I don't know anything about how the client/repo relationship works with wikidata. If I set up a fresh unconfigured wiki from puppet would you expect me to get that same error, or did it only appear after you did some wikidata config stuff? [18:41:19] giftpflanze, really? I don't see myself in "List Projects" - bastion [18:42:19] where is that? [18:43:04] anyway, you was added to the shell group, so you should also be in the bastion project [18:43:18] https://labsconsole.wikimedia.org/wiki/Special:Ask/-5B-5BResource-20Type::project-5D-5D/-3F/-3FMember/-3FDescription/mainlabel%3D-2D/searchlabel%3Dprojects/offset%3D0 [18:43:30] and you are listed in Special:NovaProject [18:43:38] rubin16: I see you in the list, at the very bottom. [18:45:38] rubin16: Well, actually, I see you in the list giftpflanze linked to, not in yours. No idea what that's about. [18:45:43] But in any case you should have access. [18:45:58] ok, let's try, thanks [18:46:34] the nova resource wiki page doesn't get automatically updated when you're added to shell [18:46:46] and special:ask uses that [18:46:50] i think [18:53:52] PROBLEM Current Load is now: CRITICAL on mwreview-test.pmtpa.wmflabs 10.4.1.63 output: Connection refused by host [18:54:32] PROBLEM Current Users is now: CRITICAL on mwreview-test.pmtpa.wmflabs 10.4.1.63 output: Connection refused by host [18:55:12] PROBLEM Disk Space is now: CRITICAL on mwreview-test.pmtpa.wmflabs 10.4.1.63 output: Connection refused by host [18:56:02] PROBLEM Free ram is now: CRITICAL on mwreview-test.pmtpa.wmflabs 10.4.1.63 output: Connection refused by host [18:56:46] 01/02/2013 - 18:56:45 - Updating keys for rubin at /export/keys/rubin [18:57:22] PROBLEM Total processes is now: CRITICAL on mwreview-test.pmtpa.wmflabs 10.4.1.63 output: Connection refused by host [18:58:52] PROBLEM dpkg-check is now: CRITICAL on mwreview-test.pmtpa.wmflabs 10.4.1.63 output: Connection refused by host [19:03:53] RECOVERY Current Load is now: OK on mwreview-test.pmtpa.wmflabs 10.4.1.63 output: OK - load average: 1.91, 1.45, 0.93 [19:04:33] RECOVERY Current Users is now: OK on mwreview-test.pmtpa.wmflabs 10.4.1.63 output: USERS OK - 0 users currently logged in [19:05:13] RECOVERY Disk Space is now: OK on mwreview-test.pmtpa.wmflabs 10.4.1.63 output: DISK OK [19:06:03] RECOVERY Free ram is now: OK on mwreview-test.pmtpa.wmflabs 10.4.1.63 output: OK: 898% free memory [19:07:23] RECOVERY Total processes is now: OK on mwreview-test.pmtpa.wmflabs 10.4.1.63 output: PROCS OK: 84 processes [19:08:53] RECOVERY dpkg-check is now: OK on mwreview-test.pmtpa.wmflabs 10.4.1.63 output: All packages OK [19:30:10] andrewbogott: It's Wikidata related, with things that haven't been merged, yet. [19:31:05] Silke_WMDE: OK. Seems like the things you're seeing must be automatically generated by the mediawiki install script. Does that seem possible? [19:31:35] Silke_WMDE: figure out the load balancer stuff? [19:32:01] I dont't know. In the mediawiki install script, there is no wikiuser user, nor a "my_wiki" database [19:32:03] aude [19:32:07] yes, I did [19:32:11] Silke_WMDE: ok [19:32:14] :) [19:32:24] and aude happy new year! [19:32:29] you too! :) [19:32:33] :) [19:34:22] actually, I am eating with my people right now... Have to go back... andrewbogott I will play more with puppet tomorrow. Hopefully, I can find out how other files are related to the mediawiki installation. That puppet stuff is a dschungel. ;) [19:34:27] CU [19:34:39] ok! I'll email if I learn anything useful [19:36:13] 01/02/2013 - 19:36:12 - Updating keys for andrew at /export/keys/andrew [19:41:52] PROBLEM dpkg-check is now: CRITICAL on mwreview-test.pmtpa.wmflabs 10.4.1.63 output: DPKG CRITICAL dpkg reports broken packages [20:31:11] 01/02/2013 - 20:31:10 - Updating keys for laner at /export/keys/laner [21:13:42] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 152 processes [21:28:42] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 147 processes [21:29:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [21:35:52] PROBLEM dpkg-check is now: CRITICAL on mwreview-test.pmtpa.wmflabs 10.4.1.63 output: DPKG CRITICAL dpkg reports broken packages [21:37:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 17% free memory [22:47:42] PROBLEM Current Load is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:49:22] PROBLEM Current Users is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:50:02] PROBLEM Disk Space is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:50:02] PROBLEM dpkg-check is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:51:23] PROBLEM Total processes is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:52:43] RECOVERY Current Load is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK - load average: 0.91, 1.15, 0.97 [22:52:43] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 50% free memory [22:54:23] RECOVERY Current Users is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: USERS OK - 0 users currently logged in [22:55:12] RECOVERY Disk Space is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: DISK OK [22:55:12] RECOVERY dpkg-check is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: All packages OK [22:56:22] RECOVERY Total processes is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS OK: 123 processes [23:17:52] Ryan_Lane, did something just change with the firewall on bastion-restricted? My web proxy just stopped working. [23:18:02] nope [23:18:10] stopped working to where? [23:18:25] To two different labs machines at the same time. [23:18:46] it's working for me to nova-precise2 [23:18:58] though I'm also getting some errors [23:19:04] When I hit the proxy my tunnel says "channel 3: open failed: administratively prohibited: open failed" [23:19:10] Hm… works for me on nova-precise2 too. [23:19:33] without throwing any errors. So something else is happening. [23:20:18] well. it's throwing errors on my commandline [23:20:34] hm [23:20:41] I wonder if something is wrong with dns [23:20:44] I bet that's the case [23:21:07] hm. no. that's working fine [23:21:36] I wonder if it was failing temporarily [23:24:07] Ryan_Lane: working now [23:24:24] I'm betting there was a temporary failure in dns [23:24:57] we really need to do host limiting with the ssh proxies [23:26:13] bleh [23:26:28] PermitOpen requires literal domain names or addresses [23:32:32] PROBLEM dpkg-check is now: CRITICAL on building.pmtpa.wmflabs 10.4.0.97 output: DPKG CRITICAL dpkg reports broken packages [23:40:06] Ryan_Lane: Whatever just happened is happening again [23:42:07] andrewbogott: I'm getting connection refused for the server you are trying on port 80 [23:42:33] RECOVERY dpkg-check is now: OK on building.pmtpa.wmflabs 10.4.0.97 output: All packages OK [23:42:45] Yep, ok, I must've broken something. [23:42:45] andrewbogott: and nothing is running on port 80 on that system [23:46:09] apparently lighttpd sometimes says [ OK ] even when it doesn't start [23:46:37] ah [23:46:42] why lighttpd? [23:49:02] lightty sucks, use nginx [23:51:06] we do use lighttp for a couple things [23:51:13] but I think we should either use apache or nginx for everything [23:52:39] The existing puppet class uses lighttpd, I'm just trying to get it up on a labs instance [23:53:03] nginx all the way... as much as the fastcgi stuff in lighttpd is useful split all the things ftw [23:53:19] though nginx proxy stats suck... having to write a module for that [23:53:34] damn having to collorate stats from varnish, nginx and apache to figure out your total traffic [23:54:43] andrewbogott: oh, what service is this for? [23:55:36] rt [23:55:40] ahhh, ok