[00:00:28] <hedonil>	 Coren: even for a simple 304 event
[00:01:09] <hedonil>	 Coren: take a look to compare resources loaded from commons
[00:02:03] <Coren>	 hedonil: You may be hitting connection limits.
[00:02:25] <hedonil>	 Coren: raise 'em ;)
[00:03:04] <Coren>	 Ah, hm.   No, that's looks more like it's an issue with DNS.  Again.  Eff.
[00:03:08] <hedonil>	 Coren: thers *must* be a knot somewhere 
[00:15:45] <Coren>	 I hate having to do this, but I'm going to add all the tool hosts to the hosts file.
[00:17:07] <Betacommand>	 Coren: what does that do?
[00:17:30] <Coren>	 Betacommand: Bypasses DNS locally.
[00:18:08] <Coren>	 andrewbogott: FYI, the symptom is becoming increasingly frequent and I still have no idea what can possibly be causing it
[00:18:29] <andrewbogott>	 :(
[00:18:52] <Coren>	 Now I'm getting random delays, timeouts and (that's new) ECONNREFUSED
[00:18:58] <Coren>	 Host tools-exec-03.pmtpa.wmflabs not found: 5(REFUSED)
[00:19:20] <Coren>	 It looks like it's overloaded(?) but it's strictly internal.
[00:22:55] <Coren>	 andrewbogott: Holey sheets!  Have you seen how much DNS traffic reaches virt2?
[00:23:24] <andrewbogott>	 I don't think I would know where to look...
[00:23:28] <andrewbogott>	 is that on ganglia?
[00:23:57] <Coren>	 andrewbogott: I looked with tcpdump.
[00:25:01] <Betacommand>	 Coren: whats on vrit2 ?
[00:25:04] <andrewbogott>	 so just 'tcpdump port 53'?
[00:25:04] <Coren>	 I mean, it's surprisingly high since that should be labs-only.
[00:25:20] <Coren>	 andrewbogott: Yeah, on br103 (the virtual interface for the openstack network)
[00:25:34] <Coren>	 Betacommand: That runs the openstack network.
[00:26:17] <Coren>	 andrewbogott: 500 requests/secs it looks like, with some bursts.  I have no idea whether that silly dnsmasq thing can handle that.
[00:26:44] <Coren>	 (I should say that clearly it cannot)
[00:27:03] <Coren>	 828 packets captured
[00:27:03] <Coren>	 1037 packets received by filter
[00:27:03] <Coren>	 171 packets dropped by kernel
[00:27:19] <Coren>	 That's not good.
[00:27:46] <andrewbogott>	 OK, so, for example...
[00:27:47] <andrewbogott>	 00:26:50.202861 IP 10.4.1.91.48655 > 10.4.0.1.domain: 28971+ AAAA? www.archidiocesedelome.org. (44)
[00:28:01] <andrewbogott>	 is 'archidiocesedelome.org' the originator of the request?
[00:28:04] <andrewbogott>	 Or the content?
[00:28:13] * andrewbogott  has barely used this tool

[00:28:44] <Coren>	 andrewbogott: It's the request.  10.4.1.91 requested AAAA records for www.archi...
[00:30:01] <andrewbogott>	 so half the traffic is coming from 10.4.1.91
[00:31:26] <andrewbogott>	 which is… what?  dig doesn't know :(
[00:32:05] <Coren>	 It's not part of tools; lemme dig around.
[00:32:26] <Coren>	 dwl.pmtpa.wmflabs.
[00:33:37] <Coren>	 Wikipedia Dead Weblinks Checker Bot; contact: gifti@toolserver.org
[00:33:58] <Coren>	 giftpflanze: Looks like it might be your "fault" after all.
[00:34:20] <Coren>	 Although that dnsmasq thing is really shitty if it starts having trouble at <1k requests/s
[00:34:41] <andrewbogott>	 yeah.
[00:35:03] <andrewbogott>	 Nonetheless, it would be nice to check if that fixes the problem...
[00:35:05] <Coren>	 Thankfully, we're changing the networking in eqiad
[00:35:36] <Coren>	 It's running in a detached screen; I should be able to suspend it
[00:39:17] <andrewbogott>	 dns is suddenly much worse!
[00:39:30] <Coren>	 I've just suspended it.
[00:39:36] <Coren>	 Let's see if that help.
[00:39:52] <Coren>	 Most of its requests are for recursive "outside" queries, that might be much worse.
[00:40:48] * hedonil  rejoices: resource loading time (304 not modified): 0ms :-D  10.17s before. that's what you call a difference!

[00:41:34] <Coren>	 hedonil: I worked around the problem on tools with hosts file.
[00:41:36] <andrewbogott>	 I'm not getting timeouts anymore...
[00:41:45] <andrewbogott>	 Still lots of traffic from tools-webproxy, but that's to be expected...
[00:41:51] <Coren>	 andrewbogott: I'm still getting pretty random resolution delay.
[00:41:52] <hedonil>	 Coren: yeah. thx
[00:41:59] <Coren>	 But almost no timeouts anymore.
[00:42:24] <Coren>	 So yeah.  Limit reached on that thing.  Lemme see if it has a tunable we could tweak.
[00:48:39] <hedonil>	 Coren: hell yeah! this is how pages should perform. 
[00:49:52] <Coren>	 andrewbogott: I'm seeing absolutely no way to tune that thing to increase its connection pool or listen queue size.
[00:50:44] <andrewbogott>	 So… maybe we just need to adopt a "Don't do that, then" solution until the migration.
[00:50:56] <andrewbogott>	 It hasn't been much of a problem, historically...
[00:50:59] <Coren>	 We can alleviate the pressure somewhat, I'm guessing, by installing bind on the worst offenders.
[00:51:27] <Coren>	 We're still getting a couple timeouts even with the giftpflanze bot asleep.  (Though not nearly as much)
[00:54:13] <Coren>	 Also, no instance DNS caching whatsoever.
[00:55:02] * Coren  ponders.

[00:57:14] * Coren  enables nscd host cache on all tools instance

[01:02:52] <Coren>	 That reduced the pressure a bit too.
[01:45:25] <Coren>	 Wa da fu is enwp10 doing?
[02:00:51] <hedonil>	 !somethingtorelax is http://www.flickr.com/photos/110698835@N04/ 
[02:00:52] <wm-bot>	 Key was added
[02:00:57] <hedonil>	 lol
[02:13:31] <hedonil>	 Coren: priority: jfi: recurrent hiccups remain: https://tools.wmflabs.org/wikiviewstats/2013-12-07-030900-loading_page.png
[02:14:09] <hedonil>	 Coren: moved all img resources to commons now, better cache.
[06:59:48] <zhuyifei1999>	 Coren: ddos attack on tools-webserver-01?
[07:10:29] <zhuyifei1999>	 petan: ^^
[09:34:28] <giftpflanze>	 Coren: there?
[11:16:31] <Arnaugir>	 \join pywikipediabot
[15:09:34] <Coren>	 zhuyifei1999: Not a willful one.  It's enwp10 but I'll have to disable it for the moment.  I was hoping they were just running a job yesterday that woudl end relatively quickly, but they'll need some serious improvement.
[15:22:55] <giftpflanze>	 Coren: did i overload dns resolving in labs?
[15:23:38] <Coren>	 giftpflanze: You did, but it's not your fault so much as it is the extraordinarily crappy networking layer of openstack.
[15:24:01] <giftpflanze>	 so, what can we do?
[15:24:05] <Coren>	 Easiest way to fix: install a bind on your instance.
[15:24:15] <giftpflanze>	 what's that?
[15:24:18] <Coren>	 (Or any other DNS server that'll resolve and cache)
[15:24:31] <giftpflanze>	 mhm
[15:25:47] <giftpflanze>	 so i didn't access tools (too much)?
[15:50:13] <giftpflanze>	 Coren: do i have to do any configuration for bind?
[15:52:10] <Coren>	 https://help.ubuntu.com/community/BIND9ServerHowto#Caching_Server_configuration
[15:52:38] <giftpflanze>	 ok, what are the ip addresses?
[15:56:00] <giftpflanze>	 Coren: ^
[15:57:30] <Coren>	 giftpflanze: I was about to say you'll find them in /etc/resolv.conf; but avoiding that one is the whole point.  :-)  Use 208.80.152.131 and 208.80.152.132
[16:01:58] <giftpflanze>	 ok, thank you
[16:10:35] <giftpflanze>	 Coren: when labs has migrated to eqiad that won't be necessary anymore?
[16:11:23] <Coren>	 giftpflanze: Presumably not.  If the new version of openstack still relies on dnsmasq I'll put a real DNS box alongside it.
[16:11:48] <giftpflanze>	 ok
[16:17:42] <johang>	 is ipv6 supposed to work?
[16:17:44] <johang>	 johang@tools-dev:~$ ping6 en.wikipedia.org
[16:17:44] <johang>	 connect: Network is unreachable
[16:17:56] <CP678|iPad>	 Coren: http://ganglia.wmflabs.org/latest/?c=tools&h=tools-login&m=load_one&r=hour&s=by%20name&hc=4&mc=2
[16:21:46] <giftpflanze>	 something seems very wrong with s5 according to that page
[16:22:01] <Coren>	 johang: Not from labs, no.  The current version of the networking stack doesn't speak V6.  The new version we are deploying in eqiad should.
[17:18:47] <Cyberpower678>	 Coren, thanks
[17:39:54] <giftpflanze>	 Coren: i think dns caching made the bot a lot faster
[17:41:18] <Coren>	 giftpflanze: That wouldn't be very surprising; DNS roundtrips add up.
[17:55:51] <Steinsplitter>	 python speaking user around?
[17:59:37] <Danny_B>	 print "Hello, Steinsplitter!"
[18:01:20] <johang>	 Steinsplitter: do you mean the programming language or the snake?
[18:01:27] <giftpflanze>	 ^^
[18:01:41] <hedonil>	 lol
[18:01:53] <hedonil>	 !somethingtorelax
[18:01:54] <wm-bot>	 http://www.flickr.com/photos/110698835@N04/ 
[18:03:57] <dan-nl>	 anyone online happen to know how i can see if runJobs.php is run as a cron job on http://commons.wikimedia.beta.wmflabs.org/ ?
[18:04:43] <Steinsplitter>	 Python Programming Language :P
[18:05:38] <Steinsplitter>	 i am working since a hour on a error handler... but .... HMPF
[18:36:32] <hedonil>	 Steinsplitter: maybe pasting some lines will improve feedback rate  https://tools.wmflabs.org/paste/  maybe.
[19:06:05] <Betacommand>	 Steinsplitter: you need some python help?
[19:10:31] <valhallasw>	 Steinsplitter: have you fixed your error handling issue? For python questions, you can also try #python, which, in my experience, is very friendly and to the point.
[19:13:41] <Steinsplitter>	 re, oh thx :)   not resolved
[19:14:39] <Steinsplitter>	 pm? ther ar som bugs, so i dos not like to post it public? :)
[19:14:46] <valhallasw>	 Steinsplitter: sure.
[21:02:52] <mutante>	 !log wikistats (wx) - chapter wikis - fixed URLs due to site redesigns: wikimedia.de, wikimedia.org.ph, pa.us.wm -> pa-us.wm
[21:02:54] <labs-morebots>	 Logged the message, Master
[21:03:51] <mutante>	 !log wikistats (wx) - chapter wikis - removed because they all don't use mediawiki anymore (sigh!?) - wikimedia.fr, wikimedia.ch, wikimedia.org.ve, wikimedia.org.il, wikimedia.org.ar ..(less and less chapters actually use mw)
[21:03:53] <labs-morebots>	 Logged the message, Master
[21:05:37] <mutante>	 !log wikistats - various fixes with some broken updates in wiktionary and mediawiki table, remaining ones with error code "991" means the remote site has database issues, even if it looks like it works in browser, the script gets internal_api_error_DBConnectionError or similar
[21:05:38] <labs-morebots>	 Logged the message, Master
[21:07:49] <Coren>	 johang: Python: the language of choice for parseltongues!  :-)