[00:44:03] PROBLEM - Puppet errors on tools-exec-1440 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [00:55:40] 06Labs, 10Tool-Labs: IABot encountering DNS errors on labs. - https://phabricator.wikimedia.org/T164420#3233907 (10Cyberpower678) I'm pretty sure it's because the workers on different hosts. [00:56:15] 06Labs, 10Tool-Labs: IABot encountering DNS errors on labs. - https://phabricator.wikimedia.org/T164420#3233908 (10Cyberpower678) The 2 workers are actually working down a backlog right now. There are jobs piling up on the queue. [01:14:05] RECOVERY - Puppet errors on tools-exec-1440 is OK: OK: Less than 1.00% above the threshold [0.0] [03:32:49] PROBLEM - Puppet errors on tools-exec-1442 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [03:57:50] RECOVERY - Puppet errors on tools-exec-1442 is OK: OK: Less than 1.00% above the threshold [0.0] [04:19:16] PROBLEM - Puppet errors on tools-exec-1432 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [04:59:18] RECOVERY - Puppet errors on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [05:52:50] 06Labs, 05Security: Generate labsdb views for dtywiki, pawikisource, ptwikimedia, wbwikimedia - https://phabricator.wikimedia.org/T164103#3234047 (10Marostegui) I have done a FLUSH PRIVILEGES but it didn't work (as usual). What was the issue the last time? Was it also 1003? I remember Jaime and yourself talkin... [06:50:41] 06Labs, 05Security: Generate labsdb views for dtywiki, pawikisource, ptwikimedia, wbwikimedia - https://phabricator.wikimedia.org/T164103#3234095 (10jcrespo) I will try random stuff today until it works. [06:54:59] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:55:57] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1421 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:35:00] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [07:35:59] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1421 is OK: OK: Less than 1.00% above the threshold [0.0] [08:01:11] 06Labs, 10Labs-Infrastructure, 06Operations: Make all ldap users have a sane shell (/bin/bash) - https://phabricator.wikimedia.org/T86668#3234221 (10hashar) [08:03:55] 06Labs, 10Labs-Infrastructure, 10LDAP-Access-Requests, 06Operations: Make all ldap users have a sane shell (/bin/bash) - https://phabricator.wikimedia.org/T86668#3234234 (10hashar) There are only two account in LDAP with shells not being `/bin/bash`: ``` $ ldapsearch -LLL -x -b 'ou=people,dc=wikimedia,dc=o... [10:44:07] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 07Beta-Cluster-reproducible, 07Puppet: New instance have broken puppet configuration when using puppetmaster standalone - https://phabricator.wikimedia.org/T148929#3234740 (10hashar) [11:14:54] 06Labs, 06Operations, 10ops-eqiad, 13Patch-For-Review: (don't) decom promethium - https://phabricator.wikimedia.org/T164395#3232060 (10MoritzMuehlenhoff) @Andrew : Can you clarify, this is host running in the labs or production realm? [12:20:29] 06Labs, 06Operations, 13Patch-For-Review: (don't) decom promethium - https://phabricator.wikimedia.org/T164395#3235014 (10Cmjohnson) [12:34:21] 06Labs, 10Tool-Labs: pawikisource missing in meta_p database - https://phabricator.wikimedia.org/T164481#3235037 (10Phe) [12:50:14] 06Labs, 10Tool-Labs: pawikisource missing in meta_p database - https://phabricator.wikimedia.org/T164481#3235037 (10jcrespo) I will merge this onto the main task. [12:50:57] 06Labs, 05Security: Generate labsdb views for dtywiki, pawikisource, ptwikimedia, wbwikimedia - https://phabricator.wikimedia.org/T164103#3235122 (10jcrespo) [12:50:59] 06Labs, 10Tool-Labs: pawikisource missing in meta_p database - https://phabricator.wikimedia.org/T164481#3235124 (10jcrespo) [12:52:02] 06Labs, 05Security: Generate labsdb views for dtywiki, pawikisource, ptwikimedia, wbwikimedia - https://phabricator.wikimedia.org/T164103#3235125 (10chasemp) Forgot about meta_p, will run that today :) [13:16:04] Gerritbot is now ignoring my commits again [13:57:44] 06Labs, 10Tool-Labs: IABot encountering DNS errors on labs. - https://phabricator.wikimedia.org/T164420#3235319 (10Cyberpower678) FWIW, I found the option to enable DNS caching in CURL. I turned it on. [14:04:40] 06Labs, 06Operations: maintain-meta_p hands on connecting to wikimedia.org.uk - https://phabricator.wikimedia.org/T164490#3235339 (10chasemp) [14:04:57] 06Labs, 06Operations: maintain-meta_p hands on connecting to wikimedia.org.uk - https://phabricator.wikimedia.org/T164490#3235351 (10chasemp) p:05Triage>03Normal [14:05:34] 06Labs, 06Operations: maintain-meta_p hangs on connecting to wikimedia.org.uk - https://phabricator.wikimedia.org/T164490#3235339 (10chasemp) [14:18:16] 06Labs, 06Operations: maintain-meta_p hangs on connecting to wikimedia.org.uk - https://phabricator.wikimedia.org/T164490#3235339 (10jcrespo) I do not think we host that: ``` $ dig wikimedia.org.uk wikimedia.org.uk. 1847 IN A 37.188.117.184 $ whois 37.188.117.184 descr: Rackspac... [14:20:52] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 07Beta-Cluster-reproducible, 07Puppet: New instance have broken puppet configuration when using puppetmaster standalone - https://phabricator.wikimedia.org/T148929#2736876 (10madhuvishy) @hashar This seems like a known and documented... [14:24:15] 06Labs, 05Security: Generate labsdb views for dtywiki, pawikisource, ptwikimedia, wbwikimedia - https://phabricator.wikimedia.org/T164103#3235440 (10chasemp) select count(*) from meta_p.wiki; +----------+ | count(*) | +----------+ | 871 | +----------+ [14:42:33] Hi halfak [14:43:00] halfak: http://paste.ubuntu.com/24508465/ [14:43:17] o/ [14:43:33] Interesting. Log_title can be empty. [14:43:46] I'll handle that! [14:44:25] K [14:44:29] I'll also add some handling for text so I can start figuring out what it is. [14:44:37] Also, hundreds of elements… [14:45:11] Have to head out soon, but would love to restart the analyzer… where do I need to add the handler code? [14:45:38] OK I'll get you a new version ASAP that'll handle this. [14:46:04] K thanksa [14:46:09] Thanks* [14:49:06] codeofdusk, pip install mwxml==0.3.1 [14:49:51] 06Labs, 10Tool-Labs, 10DBA, 07Tracking: Certain tools users create multiple long running queries that take all memory from labsdb hosts, slowing it down and potentially crashing (tracking) - https://phabricator.wikimedia.org/T119601#3235540 (10jcrespo) [14:49:53] 06Labs, 10Tool-Labs, 06Discovery, 06Maps: p50380g50921 has 20+ open persistent connections to labsdb1001 & labsdb1003 - https://phabricator.wikimedia.org/T142356#3235538 (10jcrespo) 05Open>03Resolved a:03jcrespo [14:51:54] 06Labs, 05Security: Generate labsdb views for dtywiki, pawikisource, ptwikimedia, wbwikimedia - https://phabricator.wikimedia.org/T164103#3235551 (10jcrespo) I ran "maintain-views --databases wbwikimedia --replace-all --debug" and I think it works. The only thing I did was to delete the user and recreate it wi... [14:53:30] Halfak started up again, thanks! WIll let you know what happens... [15:14:29] 06Labs, 05Security: Generate labsdb views for dtywiki, pawikisource, ptwikimedia, wbwikimedia - https://phabricator.wikimedia.org/T164103#3235624 (10chasemp) 05Open>03Resolved a:03chasemp >>! In T164103#3232673, @chasemp wrote: > This is done on: > > labsdb1001 > labsdb1009 > labsdb1010 > labsdb1011 >... [15:19:17] chasemp, some sort of lag or not yet resolved [15:19:19] $ sql pawikisource [15:19:19] Could not find requested database [15:19:48] phe: ah that's a dns thing let me look [15:20:20] or phe where are you running that? I'm not entirely familiar with the short cuts for replica access [15:20:30] not entirely sure that would work or not [15:20:34] from bastion [15:20:39] bastion3* [15:21:08] same from a random exec node [15:22:26] phe: thanks I'm looking now [15:27:00] 06Labs, 05Security: Generate labsdb views for dtywiki, pawikisource, ptwikimedia, wbwikimedia - https://phabricator.wikimedia.org/T164103#3235649 (10chasemp) 05Resolved>03Open Need to add the DNS aliases [15:35:55] 06Labs, 07Puppet: Make changing puppetmasters for Labs instances more easy - https://phabricator.wikimedia.org/T152941#3235671 (10bd808) p:05Triage>03Lowest Patches are of course always welcome, but this seems like a pretty delicate operation to perform via Puppet for what is in reality a seldom used edge... [15:39:13] phe: pretty sure I know what's up but there is some other maintenance going on I don't want to muddy the waters for [15:39:18] may be a bit [15:47:48] phe: works for me [15:53:34] chasemp, for me too, ty [15:54:29] 06Labs, 05Security: Generate labsdb views for dtywiki, pawikisource, ptwikimedia, wbwikimedia - https://phabricator.wikimedia.org/T164103#3235721 (10chasemp) 05Open>03Resolved https://gerrit.wikimedia.org/r/#/c/351866/ [15:54:50] 06Labs, 05Security: Generate labsdb views for dtywiki, pawikisource, ptwikimedia, wbwikimedia - https://phabricator.wikimedia.org/T164103#3235723 (10chasemp) p:05Triage>03Normal [16:05:52] !log shinked Stopped ircecho for grid master backup [16:05:53] madhuvishy: Unknown project "shinked" [16:05:57] !log shinken Stopped ircecho for grid master backup [16:06:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL [16:16:13] !log tools Stopped gridengine-shadow on tools-grid-shadow.tools (service gridengine-shadow stop and kill -9 individual shadowd processes) [16:16:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:20:42] !log tools Shut off tools-grid-master.tools instance from horizon [16:20:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:21:57] !log tools Start instance tools-grid-master.tools from horizon [16:22:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:49:19] RECOVERY - Puppet errors on tools-grid-master is OK: OK: Less than 1.00% above the threshold [0.0] [16:50:23] !log shinken restarted ircecho and reenabled puppet [16:50:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL [17:10:48] Hi, how do I run a Flask app with non-standard path on Kubernetes? [17:13:17] 10Tool-Labs-tools-Pageviews, 13Patch-For-Review: Add ability to query for legacy pageviews for projects - https://phabricator.wikimedia.org/T149358#3236073 (10MusikAnimal) 05Open>03Resolved a:03MusikAnimal Deployed! Per above pagecounts are available as a separate metric, rather than placing them side by... [17:20:19] 06Labs, 06Operations, 13Patch-For-Review: (don't) decom promethium - https://phabricator.wikimedia.org/T164395#3236110 (10Dzahn) Afaict the host is exactly half-way between the realms in neverland, being "metal labs" and such, heh [17:30:06] 06Labs, 06Operations, 10hardware-requests: Codfw: (2) hardware access request for labtest [region 2] - https://phabricator.wikimedia.org/T161766#3236167 (10chasemp) [17:30:36] 06Labs, 06Operations, 10hardware-requests: Codfw: (2) hardware access request for labtest [region 2] - https://phabricator.wikimedia.org/T161766#3142263 (10chasemp) Updated description from: > Number of systems: 1 to the correct > Number of systems: 2 [17:33:40] 10Tool-Labs-tools-Pageviews, 13Patch-For-Review: Add ability to query for legacy pageviews for projects - https://phabricator.wikimedia.org/T149358#3236175 (10Nuria) Super thanks for the fast turnarround @MusikAnimal [18:04:30] madhuvishy: https://tools.wmflabs.org/wikidata-todo appears to be down. Maybe related to the work you just finished? [18:05:42] multichill: I doubt it, is the tool running? [18:05:58] andrewbogott hi, did you forget to use stretch here https://gerrit.wikimedia.org/r/#/c/351890/3/modules/labs_bootstrapvz/manifests/init.pp ? [18:05:59] it shows source => 'puppet:///modules/labs_bootstrapvz/labs-jessie.manifest.yaml', [18:06:24] yes, probably :) [18:07:10] oh :) [18:07:25] madhuvishy: Web service appears to be down. Can you give it a nudge? [18:08:02] multichill: ah are you not admin on the tool? [18:08:16] No, I'm not [18:08:24] alright, poking [18:09:28] Changes cause things to go down and sometimes stuff just dies.... [18:09:48] multichill: the webservice is on kubernetes [18:09:52] and seems to be running [18:10:01] all the grid tasks are also running [18:10:03] And it appears to be back [18:10:09] i didn't do anything [18:10:13] Excellent [18:10:30] Best way to solve an incident :P [18:11:26] TIL https://tools.wmflabs.org/wikidata-todo/ tho [18:11:29] neat tool [18:13:07] madhuvishy wins by doing absolutely nothing [18:13:42] He just looked really angry at the tool and it decided to start working again [18:13:51] She :) [18:14:52] andrewbogott hi, is there a process to update to https://horizon.wikimedia.org/project/images/f2ef20b4-0a51-4b0b-a4ea-2c7186fcb91c/ ? [18:15:22] paladox: this is all a work in progress, please don't touch anything to do with stretch right now [18:15:29] ok [18:16:25] madhuvishy: Sorry, She was probably as scary as https://www.youtube.com/watch?v=tAGlUUAmjxk [19:18:28] 06Labs, 06Operations, 10wikitech.wikimedia.org, 07HHVM: Move wikitech (silver) to HHVM - https://phabricator.wikimedia.org/T98813#3236727 (10Jdforrester-WMF) [19:18:38] 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP Warning: Memcached::touch(): touch is only supported with binary protocol - https://phabricator.wikimedia.org/T158613#3236729 (10Jdforrester-WMF) [19:19:03] 06Labs, 10wikitech.wikimedia.org, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#3145305 (10Jdforrester-WMF) Does this encompass {T98813} too? [19:23:58] !log tools Rebooting tools-grid-shadow [19:24:00] PROBLEM - SSH on tools-grid-shadow is CRITICAL: Connection refused [19:24:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:29:00] RECOVERY - SSH on tools-grid-shadow is OK: SSH OK - OpenSSH_6.9p1 Ubuntu-2~trusty1 (protocol 2.0) [20:32:41] 06Labs: role::labs::lvm::srv throws some errors on stretch - https://phabricator.wikimedia.org/T164534#3236982 (10Andrew) [20:36:39] 10Striker, 07Epic: Manage shared tool accounts via Striker - https://phabricator.wikimedia.org/T149458#3236997 (10bd808) Note: per {T86668} the `loginShell` attribute of newly created `posixAccount` records should be a valid shell and not `/usr/local/bin/sillyshell` [20:36:51] 06Labs, 10Labs-Infrastructure, 10LDAP-Access-Requests, 06Operations: Make all ldap users have a sane shell (/bin/bash) - https://phabricator.wikimedia.org/T86668#973669 (10bd808) >>! In T86668#3234234, @hashar wrote: > There are only two account in LDAP with shells not being `/bin/bash`: > ``` > $ ldapsear... [20:39:55] Zhaofeng_Li: the short answer is "you can't". The longer answer is that you just need $HOME/www/python/app.py to be the entry point and the rest of the code can live anywhere. [20:40:48] Zhaofeng_Li: This may be as easy as making that file do something like `from my_real_entrypoint import app` [20:42:12] oops I meant $HOME/www/python/src/app.py [20:43:18] Zhaofeng_Li: if you search https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/My_first_Django_OAuth_tool for "app.py" you can see an example of making the expected entry point for a django app. [20:44:58] there *might* (haven't read the code in a while) be a way to use a custom uwsgi.ini in the kubernetes setup too, but really making sure that $HOME/www/python/app.py exports an 'app' symbol that is your uwsgi entry point is the safest thing to do [20:45:57] * bd808 wishes these sorts of questions came on email more often than irc. these one-off conversations don't help future users much [20:46:52] bd808: paste the discussion into a mail, and send it? :o [20:47:20] I should just add it as a FAQ on the wiki :) [21:10:27] halfak: http://paste.ubuntu.com/24513211/ [21:28:00] bd808: Okay, thanks [21:31:02] By the way, one of my Flask routes streams a JSON, but the response isn't retrieved be the client until the whole body is complete [21:31:59] I suspect it's due to some configurations on the reverse proxy [21:34:11] Zhaofeng_Li: hmmm.. yeah there may be a bit of buffering there [21:36:42] Is there a way to work around that? I suspect the reverse proxy needs to be reconfigured, so it may not be something easy to do [21:37:41] I can do polling if streaming isn't possible, but it's kinda lame [21:38:25] (03PS1) 10Dzahn: add passwords::icinga with notsecret-fake password [labs/private] - 10https://gerrit.wikimedia.org/r/352043 [21:39:35] (03CR) 10Dzahn: [V: 032 C: 032] add passwords::icinga with notsecret-fake password [labs/private] - 10https://gerrit.wikimedia.org/r/352043 (owner: 10Dzahn) [21:52:57] Zhaofeng_Li: is it a lot of data that gets buffered? I'll go find the nginx config and we can look at what it is doing [21:55:21] Zhaofeng_Li: this is pretty much the bits of the nginx config that control the reverse proxy -- https://github.com/wikimedia/puppet/blob/production/modules/dynamicproxy/templates/urlproxy.conf#L146-L162 [21:55:58] $backend there should be the direct ip:port of your uwsgi service [21:57:48] Hello, Amitie 10g here [21:57:59] Zhaofeng_Li: do you know if your app is actually using a "Transfer-Encoding: chunked" header in the response [21:58:05] hello Amitie_10g [21:59:01] A question, Can? and How can I use the Labs infraestructure to compile a large program like HHVM (I need to compile a custom build)? [21:59:25] well... [22:00:07] we don't have a great place for things like that in Tool Labs. I know from experience that building HHVM from source is a pretty serious resource hog [22:00:51] if you were in another labs project then the answer would be to build an instance big enough to do the compile and just do it [22:01:18] Amitie_10g: are you hoping to run the resulting custom hhvm build in tool labs? [22:01:34] and if so, what problem are you trying to solve? [22:02:30] I'm trying to compile it in a virtual machine with Ubuntu 16.4 but got missing file error [22:02:55] i though labs only had debian jessie and ubuntu 14.04 [22:03:18] paladox: he didn't say the vm was in labs :) [22:03:26] oh woop [22:03:29] (lol) [22:03:30] Amitie_10g: what file are you missing? [22:04:19] Zhaofeng_Li: I may have found something for you to try. There is a "X-Accel-Buffering: no" that supposedly tells nginx not to buffer your response -- https://www.nginx.com/resources/wiki/start/topics/examples/x-accel/#x-accel-buffering [22:05:21] Let me tell the output of cmake [22:05:43] remember to put it in a paste somewhere and not to the irc channel [22:05:46] https://gist.github.com/Amitie10g/1f04db7c354f67a27b14b345063aa3b3 [22:05:48] Yes [22:05:52] There is [22:06:02] bd808: Just checked, and my response does not declare the chunked transfer encoding. I will give the accel buffering header a try [22:06:21] Zhaofeng_Li: cool. try it out and let me know if it helps [22:06:28] Amitie_10g: looking [22:07:55] Amitie_10g: is /home/davod/git/hhvm/third-party/proxygen/lib/http/HTTPCommonHeaders.h actually there? [22:08:23] if so it sounds like a cmake includes error [22:08:33] if not it sounds like a bug in the source files [22:09:04] Yes, since the file is located at third-party/proxygen/lib/http/HTTPHeaders.h [22:09:21] Inside the root directory I cloned [22:10:43] something isn't setting up the cmake environment as expected then [22:11:07] I haven't actually tried to build hhvm from source for a couple of years now [22:16:05] I'll try with a previous version [22:17:02] I know our ops team builds the deb on both 14.04 and debian jessie. those are the versions we have in our local apt repos [22:19:10] Have these Ubuntu/Debian installation specific packages in addition to those listed at the documentation? [22:19:21] https://docs.hhvm.com/hhvm/installation/building-from-source#ubuntu-14.04-trusty [22:20:31] I have a VM with Ubuntu 16.04 [22:22:03] debugging a cmake run of building hhvm seems like a topic for #hhvm. The Facebook HHVM team and other volunteers hang out there. [22:22:32] I'll ask there. So, does the team attempted to build a custom buildof HHVM with less modules? [22:23:09] bd808: Just tried, and it worked! Thanks a lot! [22:23:19] I don't understand "less modules". Can you rephrase the question? [22:23:42] Zhaofeng_Li: awesome. can you find somewhere to document that on wikitech? [22:23:59] Ideally in a place you would have looked for the answer :) [22:24:37] Excluding, for example MySQL support. [22:25:04] I want to build a statically-linked hhvm binary with only few modules [22:25:06] Amitie_10g: ah. no I don't think we turn off any built-in modules [22:26:57] bd808: Let me see... Is https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web a good place to do that? [22:27:11] Actually, is for another appliance [22:27:28] that onlye needs Proxygen (without FastCGI) [22:27:43] Zhaofeng_Li: yeah. somewhere on that page is where I'd expect to find info on streaming responses [22:28:56] Amitie_10g: unless your appliance is very space constrained you are probably best off not trying to disable the fastcgi hander. I doubt the size of the final binary will be a lot different. [22:29:31] Cool, I'll add that as a section. And thanks again for your help :) You saved my day [22:30:00] glad to help. and thanks for paying it forward my making the docs better [22:30:06] *by making [22:31:31] Exactly, it is space constrained. I'm talking at #hhvm [22:31:53] I ran this https://gist.github.com/Amitie10g/4c0dd61055ecb80a2d5e51e6c726f3d3 [22:44:36] Is there a good way to run generic long-lived processes on Kubernetes? In my case, it's a Celery worker that does not expose any web service. Do I need to make my own Docker image, or am I better off using the grid engine? [22:46:59] Zhaofeng_Li: its possible to use kubectrl directly to launch a pod using one of our base Dcoker images for something like that. stashbot's python irc bot runs as a pod on out kubernetes grid [22:47:34] I have a "todo" to write up how it's done, but I can point you to the source that does it now [22:49:17] Zhaofeng_Li: this is the deployment file that manages the pod -- https://phabricator.wikimedia.org/diffusion/LTST/browse/master/etc/deployment.yaml [22:49:42] Zhaofeng_Li: and this is the script I use to start and stop it -- https://phabricator.wikimedia.org/diffusion/LTST/browse/master/bin/stashbot.sh [22:51:49] Zhaofeng_Li: the upstream documentation on using Kubernetes is pretty nice. Most everything you read there about using kubectl applies in our environment. https://kubernetes.io/docs/home/ [22:55:17] Thanks, I'll check them out [23:17:44] 06Labs, 10Tool-Labs: IABot encountering DNS errors on labs. - https://phabricator.wikimedia.org/T164420#3237318 (10bd808) >>! In T164420#3235319, @Cyberpower678 wrote: > FWIW, I found the option to enable DNS caching in CURL. I turned it on. The last error I see in the logs is `2017-05-04T13:50:11 en.wikiped... [23:21:09] 06Labs, 10Tool-Labs: IABot encountering DNS errors on labs. - https://phabricator.wikimedia.org/T164420#3237327 (10Cyberpower678) Actually I have it set to 60 seconds.