[00:37:11] <chasemp>	 twentyafterfour: I thikn that may be a hiera issue
[00:37:14] <chasemp>	 look at that file
[00:37:16] <chasemp>	             $deployment_host = hiera('scap::deployment_server')
[00:37:16] <chasemp>	             $deployment_ip = ipresolve($deployment_host)
[00:37:26] <chasemp>	 are you sure you are setting that value in hiera for this project or host?
[00:37:50] <chasemp>	 but then also I think ipresolve has a 1 dns server param so 
[00:38:04] <chasemp>	 if 1 of the dns servers is being flaky ipresolve() in puppet is a place it can surface
[00:38:43] <chasemp>	 but yeah get a response from both
[00:38:51] <chasemp>	 twentyafterfour: make a task? this may take digging idk
[00:39:26] <twentyafterfour>	 chasemp: I'm sure the hiera value is set right, the error message is on the ipresolve not the hiera lookup
[00:39:35] <twentyafterfour>	 it's trying to resolve the right ip, just fails
[00:39:48] <chasemp>	 what dns server does it use?
[00:39:59] <twentyafterfour>	 Error: Could not retrieve catalog from remote server: Error 400 on SERVER: DNS lookup failed for phab-tin.phabricator.eqiad.wmflabs Resolv::DNS::Resource::IN::A at /etc/puppet/modules/scap/manifests/target.pp:98 on node phab-01.phabricator.eqiad.wmflabs
[00:40:14] <twentyafterfour>	 chasemp: I have no idea what server the puppetmaster uses. That's why I'm asking in here
[00:41:08] <twentyafterfour>	 from /etc/puppet/puppet.conf:
[00:41:10] <twentyafterfour>	 server = labs-puppetmaster-eqiad.wikimedia.org
[00:41:39] <twentyafterfour>	 the exact same thing works for deployment-prep but not for phabricator.
[00:42:03] <chasemp>	 but it would be resolving a different host?
[00:42:29] <twentyafterfour>	 yes and deployment-prep uses a project-local puppetmaster
[00:43:25] <twentyafterfour>	 hmm, ipresolve caches the results
[00:47:58] <chasemp>	 so
[00:48:09] <chasemp>	 it seems ipresolve uses system default in this case
[00:49:24] <chasemp>	 and the puppetmaster in question is
[00:49:29] <chasemp>	 labcontrol1001.wikimedia.org
[00:49:32] <chasemp>	 and labcontrol1001.wikimedia.org says
[00:49:39] <chasemp>	 root@labcontrol1001:~# host phab-tin.phabricator.eqiad.wmflabs
[00:49:39] <chasemp>	 Host phab-tin.phabricator.eqiad.wmflabs not found: 3(NXDOMAIN)
[00:49:45] <twentyafterfour>	 hmm
[00:49:52] <twentyafterfour>	 that'd be the problem
[00:49:57] <chasemp>	 I'm not sure that this ever worked in this fashion
[00:50:19] <twentyafterfour>	 works on deployment prep but not from labs puppetmaster so that might be why it works at all
[00:50:24] <chasemp>	 right
[00:50:41] <chasemp>	 I don't think is anything you are doing wrong, just maybe a model we don't see often
[00:50:58] <chasemp>	 and it should work as in it's reasonable to expect it to work
[00:51:13] <twentyafterfour>	 it's a bit of a hack anyway, I guess I could just hard code the IP of the instance. Is that likely to remain static or do instance ips change on a whim?
[00:51:20] <chasemp>	 but to fix this I would want to ask andrew why this dosn't work now
[00:51:29] <chasemp>	 no it will keep that ip for the life of the vm
[00:51:37] <chasemp>	 it's bizarro static dhcp
[00:51:46] <chasemp>	 so you can do hat but
[00:51:52] <chasemp>	 if you can toss this into a bug anyway
[00:51:55] <chasemp>	 I'll follow up
[00:52:48] <chasemp>	 I gotta grab dinner tho, good luck twentyafterfour!
[00:58:22] <twentyafterfour>	 chasemp: thanks
[01:01:26] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Phabricator, 07Puppet: puppet function ipresolve unable to look up instance on labs-puppetmaster - https://phabricator.wikimedia.org/T139011#2416717 (10mmodell)
[01:01:31] <twentyafterfour>	 https://phabricator.wikimedia.org/T139011
[02:00:01] <wikibugs>	 06Labs, 10Tool-Labs: Python can't be started at tools-bastion-02/03 - https://phabricator.wikimedia.org/T138990#2416791 (10bd808) 05Open>03Invalid
[02:00:34] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Phabricator, 07Puppet: puppet function ipresolve unable to look up instance on labs-puppetmaster - https://phabricator.wikimedia.org/T139011#2416717 (10AlexMonk-WMF) @chasemp, can you check other hosts that we know to work like `bastion-01.bastion.eqiad.wmflabs`?
[02:00:47] <wikibugs>	 06Labs, 10Phlogiston (Interrupt), 15User-bd808: Phlogiston-1 unresponsive - https://phabricator.wikimedia.org/T137736#2416794 (10bd808) a:05bd808>03None
[02:01:02] <wikibugs>	 06Labs, 10Phlogiston (Interrupt): Phlogiston-1 unresponsive - https://phabricator.wikimedia.org/T137736#2377044 (10bd808)
[02:46:37] <bd808>	 !log mwv-apt Added self (BryanDavis) as project admin
[02:46:44] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mwv-apt/SAL, Master
[07:54:44] <wikibugs>	 06Labs, 10Tool-Labs: uWSGI webservice terminating unexpectedly - https://phabricator.wikimedia.org/T139020#2416981 (10D3r1ck01)
[07:57:01] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Prnk28 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=702486 edit summary: 
[08:11:18] <wikibugs>	 06Labs, 10Tool-Labs: uWSGI webservice terminating unexpectedly - https://phabricator.wikimedia.org/T139020#2416981 (10valhallasw) Which tool is this, and is there any information in your uwsgi.log?
[08:18:45] <wikibugs>	 06Labs, 10Tool-Labs: uWSGI webservice terminating unexpectedly - https://phabricator.wikimedia.org/T139020#2417034 (10D3r1ck01) I am using the tool called "ifttt-testing", @valhallasw, I can add you so you can have access if its necessary. The uwsgi.log file doesn't have anything or information about the app b...
[08:35:11] <paladox>	 !log phabricator shutting down phab-02 for maintenance.
[08:35:14] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master
[08:36:09] <paladox>	 !log phabricator deleting phab-02 instance.
[08:36:13] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master
[08:37:14] <paladox>	 !log phabricator recreating phab-02 instance should fix ssh issues hopefully.
[08:37:17] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master
[08:37:50] <paladox>	 !log phabricator making phab-02 instance a medium size and running trusty.
[08:37:52] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master
[08:54:00] <paladox>	 !log phabricator waiting a few mins to recreate phab-02 since it seems to still be set on port 222 which is strange since other ones are set at port 22
[08:54:04] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master
[09:00:02] <grrrit-wm>	 (03CR) 10Jean-Frédéric: "Looks good to me. I ran the file through `Shellcheck` − could these be fixed ? (I should look into adding Shellcheck at CI level)." (033 comments) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296466 (https://phabricator.wikimedia.org/T138606) (owner: 10Lokal Profil)
[09:01:43] <DrTrigon>	 sDrewth: Hi, I'm trying to get wm-bot to log our gsoc channel but only get a "permission denied" - can you explain that?
[09:20:01] <paladox>	 !log phabricator phab-02 wont set ssh port correctly, using phab-04 as replacement recreating phab-04.
[09:20:05] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master
[09:20:55] <paladox>	 !log phabricator using phab-05 instead
[09:20:58] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master
[09:23:06] <paladox>	 !log phabricator i found what the problem is. Recreating phab-02 now.
[09:23:10] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master
[09:48:23] <paladox>	 !log phabricator phab-02 will need a while to be able to get rid of any settings. Using phab-05 as a replacement and will try again later using phab-02
[09:48:26] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master
[10:00:40] <paladox>	 !log installing apache2 and php5 and phabricator-deployment on phab-05
[10:00:41] <labs-morebots>	 installing is not a valid project.
[10:00:52] <paladox>	 !log phabricator installing apache2 and php5 and phabricator-deployment on phab-05
[10:00:57] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master
[10:12:50] <paladox>	 !log phabricator running ./bin/storage upgrade on phab-05
[10:12:54] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master
[10:23:57] <paladox>	 !log phabricator phab-05 all setup now. phab-02.wmflabs.org redirected to the phab-05 server. Url is the same just different server.
[10:24:00] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master
[10:35:29] <wikibugs>	 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 07Puppet: Implement role based hiera lookups for labs - https://phabricator.wikimedia.org/T120165#1847021 (10hashar)
[10:46:51] <wikibugs>	 06Labs, 10Labs-Infrastructure, 06Operations, 10Shinken, 07Graphite: Clean up labs graphite datapoints - https://phabricator.wikimedia.org/T111540#2417388 (10hashar)
[12:20:10] <wikibugs>	 10PAWS: Add possibility to run other users notebooks by copying to own folder - https://phabricator.wikimedia.org/T139036#2417613 (10Mattias_Ostmar-WMSE)
[12:25:15] <paladox>	 !log phabricator rebooting phabricator instance.
[12:25:19] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master
[12:34:54] <paladox>	 !log phabricator restarting phab-tin
[12:34:57] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master
[12:39:45] <wikibugs>	 10PAWS: Add possibility to run other users notebooks by copying to own folder - https://phabricator.wikimedia.org/T139036#2417703 (10Mattias_Ostmar-WMSE)
[13:20:42] <wikibugs>	 06Labs, 10Phabricator: https://phab-01.wmflabs.org returns a core exception - https://phabricator.wikimedia.org/T137270#2417775 (10Paladox) @krenair ive deleted phab-02 and recreated as phab-05. The url is still the same but just running under a different instance.  Reason is because phab-02 is using port 222...
[13:22:29] <wikibugs>	 06Labs, 10Phabricator: Login to phab-0[124].phabricator.eqiad.wmflabs is broken, even as root - https://phabricator.wikimedia.org/T130693#2417784 (10Paladox) phab-02 has been deleted. Reason is it kept setting it's ssh port as 222 which is wrong so I tried to recreate it and that didn't work. So I just created...
[14:01:15] <halfak>	 Hey folks.  I'm getting reports that labsdb is dropping connections often. 
[14:01:23] <halfak>	 Is this issue known?
[14:02:31] <halfak>	 I saw the email re. replica drift from yesterday. 
[14:02:36] <halfak>	 Not sure if that is related. 
[14:03:46] <yuvipanda>	 (Am answering question in -research)
[14:06:04] <paladox>	 yuvipanda i fixed phab's problem
[14:06:10] <paladox>	 yuvipanda but phab-02
[14:06:31] <paladox>	 is having problems ive deleted it and wanted to re create it. but it is always setting it's ssh port at 222
[14:06:36] <paladox>	 and im not sure why.
[14:06:43] <yuvipanda>	 I'm sorry paladox, I don't have the bandwidth to investigate that today, you should probably ask the releng / phab maintainers.
[14:07:12] <paladox>	 yuvipanda oh.
[14:11:09] <valhallasw`cloud>	 halfak: 'dropping connections'?
[14:11:54] <valhallasw`cloud>	 there are timeouts (on the order of a few minutes iirc), but those shouldn't be triggered while the server is actively doing something
[14:12:15] <valhallasw`cloud>	 but if you run a query, get a single row, wait for a long time, try to get another row, that might be problematic
[14:12:34] <wikibugs>	 06Labs, 10Phabricator: Login to phab-0[124].phabricator.eqiad.wmflabs is broken, even as root - https://phabricator.wikimedia.org/T130693#2418006 (10Paladox) 05Open>03Resolved a:03Paladox Closing as resolved since I fixed the problem by upgrading phab-01 to a large instance. I move phab-02 to phab-05 ins...
[14:15:58] <halfak>	 valhallasw`cloud, looks like these are long-running queries. 
[14:16:36] <halfak>	 Still we're going to have kjschiroo try some connection pooling :) 
[14:16:50] <valhallasw`cloud>	 don't do that.
[14:17:14] <yuvipanda>	 what, connection pooling?
[14:17:15] <valhallasw`cloud>	 idle connections still put load on the database server
[14:17:21] <valhallasw`cloud>	 (also, how would connection pooling solve the issue of disconnects?)
[14:17:31] <yuvipanda>	 ah, the sqlalchemy connection pooler deals with disconnects
[14:17:38] <yuvipanda>	 and I think it doesn't maintain idle connections
[14:17:47] <yuvipanda>	 it has a re-use timeout
[14:18:06] <valhallasw`cloud>	 but again, how does that solve the issue of disconnects that apparently happen /during/ a query?
[14:18:22] <yuvipanda>	 ah that doesn't, but I don't know if these are happening *during* a query.
[14:18:46] <yuvipanda>	 but most likely they might not be, since most of the time this happens it's just connection opened and idle too long...
[16:29:09] <grrrit-wm>	 (03PS3) 10Lokal Profil: Recreate source table prior to run [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296466 (https://phabricator.wikimedia.org/T138606) 
[16:30:16] <grrrit-wm>	 (03CR) 10Lokal Profil: "Nice. Didn't know about that one. Yes if we can get that into the CI it would be great." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296466 (https://phabricator.wikimedia.org/T138606) (owner: 10Lokal Profil)
[16:35:51] <grrrit-wm>	 (03PS1) 10Jean-Frédéric: Fix ShellCheck errors in bin scripts [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296774 
[16:40:28] <grrrit-wm>	 (03CR) 10Jean-Frédéric: [C: 032] ":)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296466 (https://phabricator.wikimedia.org/T138606) (owner: 10Lokal Profil)
[16:42:35] <grrrit-wm>	 (03Merged) 10jenkins-bot: Recreate source table prior to run [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296466 (https://phabricator.wikimedia.org/T138606) (owner: 10Lokal Profil)
[16:42:57] <grrrit-wm>	 (03CR) 10Lokal Profil: Fix ShellCheck errors in bin scripts (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296774 (owner: 10Jean-Frédéric)
[16:54:03] <grrrit-wm>	 (03PS2) 10Jean-Frédéric: Fix ShellCheck errors in bin scripts [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296774 
[16:56:28] <grrrit-wm>	 (03CR) 10Jean-Frédéric: "Aaaaah, was using the commandline ShellCheck (obtained through apt-get) which appears to be lagging behind the website. Good catch!" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296774 (owner: 10Jean-Frédéric)
[17:43:57] <grrrit-wm>	 (03PS1) 10Jean-Frédéric: Expand tatic HTML home page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296786 
[17:44:36] <grrrit-wm>	 (03PS2) 10Jean-Frédéric: Expand static HTML home page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296786 
[18:17:20] <Betacommand>	 jynus: the image link table is still showing drift on enwiki.
[18:17:54] <jynus>	 I agree
[18:22:04] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Phabricator, 07Puppet: puppet function ipresolve unable to look up instance on labs-puppetmaster - https://phabricator.wikimedia.org/T139011#2419186 (10Paladox) p:05Triage>03High Changing to high per @Dzahn   <mutante> and everybody should up the priority of that ticket
[18:37:53] <grrrit-wm>	 (03PS1) 10Jean-Frédéric: Change Intuition domain to heritage [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296795 
[18:39:36] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Phabricator, 07Puppet: puppet function ipresolve unable to look up instance on labs-puppetmaster - https://phabricator.wikimedia.org/T139011#2419238 (10chasemp) >>! In T139011#2416792, @AlexMonk-WMF wrote: > @chasemp, can you check other hosts that we know work like `bastio...
[18:41:39] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Phabricator, 07Puppet: puppet function ipresolve unable to look up instance on labs-puppetmaster - https://phabricator.wikimedia.org/T139011#2419246 (10AlexMonk-WMF) >>! In T139011#2419238, @chasemp wrote: >>>! In T139011#2416792, @AlexMonk-WMF wrote: >> @chasemp, can you c...
[18:43:04] <JeanFred>	 !log tools.heritage Deployed latest from Git: 795a396 (T136351), 4c8d9e3, e09e3d7 (T138519), 7840307, 4bdddd0 (T138606)
[18:43:06] <stashbot>	 T136351: Work out a local development environment for Heritage API - https://phabricator.wikimedia.org/T136351
[18:43:06] <stashbot>	 T138519: Add symlink to the toolbox - https://phabricator.wikimedia.org/T138519
[18:43:07] <stashbot>	 T138606: updating monuments_all fails due to wd_item - https://phabricator.wikimedia.org/T138606
[18:43:08] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master
[18:57:56] <grrrit-wm>	 (03PS1) 10Jean-Frédéric: Download Database dump to conf folder [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296798 
[19:07:00] <wikibugs>	 06Labs, 10Tool-Labs: Build containers for use with Tool Labs - https://phabricator.wikimedia.org/T130668#2419321 (10yuvipanda)
[19:07:02] <wikibugs>	 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 07Tracking: Goal: Allow using k8s instead of GridEngine as a backend for webservices (tracking) - https://phabricator.wikimedia.org/T129309#2419322 (10yuvipanda)
[19:07:04] <wikibugs>	 06Labs, 10Tool-Labs, 13Patch-For-Review: Define base Wikimedia Docker container - https://phabricator.wikimedia.org/T118446#2419319 (10yuvipanda) 05Open>03Resolved a:03yuvipanda
[19:45:11] <paladox>	 James_F hi, could you update these links to diffusion please
[19:45:13] <paladox>	 https://github.com/wikimedia/VisualEditor/blob/f05419b41b2e5fbfa7b8642b5d40dd9c6411258f/package.json#L15
[19:45:23] <paladox>	 and
[19:45:23] <paladox>	 https://github.com/wikimedia/VisualEditor/blob/f05419b41b2e5fbfa7b8642b5d40dd9c6411258f/package.json#L20
[19:45:24] <paladox>	 please
[19:45:46] <paladox>	 i think the first git.wikimedia.org link is a clone link. The clone link will look different in diffusion
[19:49:13] <paladox>	 Wrong place to even ask this sorry.
[19:49:25] <paladox>	 Oh wait never mind about first link
[19:55:36] <kaldari>	 matt_flaschen: phabricator-bug-status tool seems to be broken, just gives me a 500 server error
[19:58:38] <wikibugs>	 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 07Tracking: Goal: Allow using k8s instead of GridEngine as a backend for webservices (tracking) - https://phabricator.wikimedia.org/T129309#2419496 (10yuvipanda)
[19:58:40] <wikibugs>	 06Labs, 10Tool-Labs: Build containers for use with Tool Labs - https://phabricator.wikimedia.org/T130668#2419493 (10yuvipanda) 05Open>03Resolved a:03yuvipanda This is all done in the 'base' container, present in operations/docker-images/toollabs-images repository.
[20:17:35] <paladox>	 !log phabricator adding Danny_B as project and project admin of phabricator instance.
[20:17:39] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master
[20:17:40] <paladox>	 mutante ^^
[20:18:16] <mutante>	 Danny_B: ^ so the closest we have to prod is currently phab-03 we think
[20:18:50] <mutante>	 there are open tickets about issues that prevent the puppet roles from running
[20:19:10] <mutante>	 but the apache is there.. and then i copied 3 files manually from iridium
[20:19:42] <mutante>	 the 2 Apache sites, 50-git-wikimedia-org.conf  50-phabricator.conf in /etc/apache2/sites-*
[20:20:11] <mutante>	 and and /etc/apache/phabbanlist.conf
[20:20:27] <mutante>	 without that it got a syntax error
[20:20:36] <mutante>	 and that should normallhy be installed by puppet
[20:21:04] <paladox>	 :)
[20:21:29] <mutante>	 <insert link to ticket about labs issue?
[20:21:39] <mutante>	 that DNS lookup problem 
[20:22:36] <mutante>	 eh, i removed some IPs from that ban list file ..in labs
[20:22:48] <wikibugs>	 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Issues with 'webservice' kubernetes backend - https://phabricator.wikimedia.org/T139107#2419652 (10yuvipanda)
[20:22:48] <mutante>	 just there as placeholder so there is no error
[20:23:08] <wikibugs>	 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Issues with 'webservice' kubernetes backend - https://phabricator.wikimedia.org/T139107#2419665 (10yuvipanda)
[20:26:25] <mutante>	 !log phabricator phab-03 - updated /etc/apache2/sites-available/50-phabricator.conf with file from iridium to match production, restarted apache
[20:26:29] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master
[20:27:13] <mutante>	 Danny_B: paladox , and now i just updated it to match production 
[20:27:31] <paladox>	 :)
[20:27:40] <mutante>	 well, the webserver restarts without error
[20:28:01] <mutante>	 now i recommed to test something with curl on localhost and/or  try to install the apache-fast-test script there
[20:28:10] <mutante>	 which we have on tin in prod
[20:28:15] <mutante>	 and is in the puppet repo
[20:28:16] <paladox>	 :)
[20:28:27] <mutante>	 i used that to test it on iridium after merge
[20:28:41] <paladox>	 mutante we have git.wmflabs.org/
[20:28:48] <mutante>	 and the input for that tool was the list of URLs , one per line, that paladox has in pastebin 
[20:29:12] <mutante>	 the results i pasted on that ticket 
[20:29:32] <mutante>	 so we can compare that to the same in labs
[20:30:57] <mutante>	 paladox: wanna try that script on one of the labs instances? maybe another one
[20:31:21] <paladox>	 Yep
[20:31:26] <mutante>	 so that it doesnt have to run on localhost 
[20:31:37] <mutante>	 and also we dont touch that instance
[20:31:43] <mutante>	 with manuall installs
[20:32:16] <mutante>	 you could use one in the 'git" project
[20:32:16] <paladox>	 Ok
[20:32:19] <mutante>	 doesnt matter
[20:32:22] <paladox>	 Ok
[20:33:14] <mutante>	 modules/apache/manifests/helper_scripts.pp:    file  { '/usr/local/bin/apache-fast-test':
[20:33:17] <mutante>	 this file
[20:33:50] <mutante>	 just copy it, then see what is missing.. it will need Perl
[20:34:00] <paladox>	 Ok
[20:34:11] <paladox>	 Copy it ?
[20:34:14] <paladox>	 mutante ^^
[20:34:45] <mutante>	 i mean: find it in the puppet repo, copy from the local clone
[20:34:49] <mutante>	 to the labs instance
[20:34:55] <mutante>	 then run it on labs instance
[20:35:17] <mutante>	 using scp
[20:35:22] <mutante>	 or WinSCP
[20:35:56] <mutante>	 ..or git clone ops/puppet repo directly from labs instance
[20:36:08] <mutante>	 and then copy it around locally so you can run it
[20:36:23] <paladox>	 Ok
[20:37:17] <paladox>	 Im going to clone puppet
[20:37:45] <mutante>	 ./modules/apache/files/apache-fast-test
[20:38:01] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: graphite.wmflabs.org is very slow / flaky - https://phabricator.wikimedia.org/T127957#2419757 (10yuvipanda)
[20:38:03] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2419755 (10yuvipanda) 05Open>03Resolved I just puppetized the DocumentRoot and am going to call this 'done' since the other option is a rabbithole I do not have the time to dive int...
[20:38:29] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2419761 (10yuvipanda)
[20:38:46] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2383843 (10yuvipanda)
[20:39:12] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: graphite.wmflabs.org is very slow / flaky - https://phabricator.wikimedia.org/T127957#2059611 (10yuvipanda) 05Open>03Resolved Sufficiently fast now with SSDs.
[20:40:18] <paladox>	 mutante ive cloned the repo
[20:40:21] <paladox>	 what do i do next
[20:40:21] <paladox>	 ?
[20:40:23] <paladox>	 please
[20:40:44] <mutante>	 paladox: find the file i pasted above, and try to execute it
[20:41:01] <paladox>	 Ok do i execute it with /usr/local/bin/apache-fast-tes
[20:41:41] <mutante>	 no, eh. copy it to /usr/local/bin/
[20:41:47] <paladox>	 Oh
[20:41:49] <mutante>	 then just run it as "apache-fast-test"
[20:41:59] <mutante>	 (like it is on tin)
[20:42:07] <mutante>	 it's not bash, it's perl
[20:42:28] <paladox>	 mutante bash: apache-fast-test: command not found
[20:42:40] <mutante>	 root@tin:~# which apache-fast-test
[20:42:40] <mutante>	 /usr/local/bin/apache-fast-test
[20:42:52] <mutante>	 did you copy it there ?
[20:42:53] <paladox>	 Oh im using phab-03
[20:43:02] <mutante>	 from where you git cloned the files to
[20:43:08] <mutante>	 eh, dont
[20:43:08] <paladox>	  /srv
[20:43:11] <paladox>	 Oh
[20:43:12] <mutante>	 use another instance
[20:43:34] <paladox>	 Ok, im going to ssh into tin
[20:43:42] <paladox>	 phab-tin
[20:43:44] <mutante>	 i dont think you can
[20:43:47] <mutante>	 oh
[20:43:52] <mutante>	 eh, ok
[20:45:34] <paladox>	 Ok, ive cloned the rpeo
[20:45:35] <paladox>	 repo
[20:46:10] <paladox>	 root@phab-tin:/srv/puppet# apache-fast-test modules/apache/manifests/helper_scripts.pp
[20:46:10] <paladox>	 Can't locate LWP/UserAgent.pm in @INC (you may need to install the LWP::UserAgent module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.20.2 /usr/local/share/perl/5.20.2 /usr/lib/x86_64-linux-gnu/perl5/5.20 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.20 /usr/share/perl/5.20 /usr/local/lib/site_perl .) at /usr/local/bin/apache-fast-test line 27.
[20:46:10] <paladox>	 BEGIN failed--compilation aborted at /usr/local/bin/apache-fast-test line 27.
[20:46:15] <paladox>	 mutante ^^
[20:46:21] <mutante>	 ok, kind of expected
[20:46:26] <mutante>	 but the next step
[20:46:29] <paladox>	 Ok
[20:46:30] <mutante>	 you need perl
[20:46:35] <mutante>	 and modules
[20:46:36] <paladox>	 Ok, apt-get install perl
[20:46:43] <paladox>	 ?
[20:46:48] <mutante>	 yes
[20:46:55] <paladox>	 Ok
[20:47:12] <paladox>	 root@phab-tin:/srv/puppet# sudo apt-get install perl
[20:47:12] <paladox>	 Reading package lists... Done
[20:47:13] <paladox>	 Building dependency tree
[20:47:13] <paladox>	 Reading state information... Done
[20:47:13] <paladox>	 perl is already the newest version.
[20:47:13] <paladox>	 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
[20:47:15] <paladox>	 mutante ^^
[20:47:22] <wikibugs>	 06Labs: Delete gerrit-dev.wmflabs.org website - https://phabricator.wikimedia.org/T138101#2419841 (10yuvipanda) 05Open>03Resolved Done, with  ``` }yuvipanda@novaproxy-01:/srv$ curl -XDELETE localhost:5668/dynamicproxy-api/v1/gerrit/mapping/gerrit-dev.wmflabs.org. ```
[20:47:39] <mutante>	 paladox: type "cpan" 
[20:47:42] <paladox>	 Ok
[20:47:56] <paladox>	 cpan works
[20:47:57] <mutante>	 install LWP::UserAgent
[20:48:00] <mutante>	 in the cpan shell
[20:48:19] <paladox>	 Ok
[20:48:23] <paladox>	 its installing
[20:48:44] <mutante>	 ..but puppet should install the package
[20:48:52] <paladox>	 Oh
[20:48:58] <mutante>	 we want that to actualy happen
[20:49:10] <mutante>	 without us manually doing this and cpan
[20:49:16] <paladox>	 Oh, woops i did install LWP::UserAgent was that wrong
[20:49:19] <mutante>	 i just want to see this first
[20:49:44] <paladox>	 Ok
[20:49:45] <mutante>	 it's ok to test 
[20:49:49] <mutante>	 but we want to fix it
[20:49:51] <paladox>	 It's installed now
[20:50:03] <mutante>	 so that a puppet role does it next time
[20:50:09] <paladox>	 Ok
[20:50:24] <mutante>	 ok, try running the script again
[20:50:50] <paladox>	 Ok
[20:51:09] <paladox>	 apache-fast-test modules/apache/manifests/helper_scripts.pp
[20:51:14] <paladox>	 root@phab-tin:/srv/puppet# apache-fast-test modules/apache/manifests/helper_scripts.pp
[20:51:14] <paladox>	 Can't locate Net/DNS/Resolver.pm in @INC (you may need to install the Net::DNS::Resolver module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.20.2 /usr/local/share/perl/5.20.2 /usr/lib/x86_64-linux-gnu/perl5/5.20 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.20 /usr/share/perl/5.20 /usr/local/lib/site_perl .) at /usr/local/bin/apache-fast-test line 28.
[20:51:14] <paladox>	 BEGIN failed--compilation aborted at /usr/local/bin/apache-fast-test line 28.
[20:51:18] <paladox>	 mutante ^^
[20:51:58] <mutante>	 just run the command by itself, dont add that helper_scripts.pp there
[20:52:18] <paladox>	 mutante http://www.bbc.co.uk/news/blogs-trending-36663781
[20:52:20] <paladox>	 Ok
[20:52:45] <mutante>	 paladox: so it says it needs another module, and the name
[20:52:51] <paladox>	 yep
[20:52:51] <mutante>	 paladox: try installing it just like the other one
[20:52:52] <paladox>	 root@phab-tin:/srv/puppet# apache-fast-test modules/apache/manifests/
[20:52:52] <paladox>	 Can't locate Net/DNS/Resolver.pm in @INC (you may need to install the Net::DNS::Resolver module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.20.2 /usr/local/share/perl/5.20.2 /usr/lib/x86_64-linux-gnu/perl5/5.20 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.20 /usr/share/perl/5.20 /usr/local/lib/site_perl .) at /usr/local/bin/apache-fast-test line 28.
[20:52:52] <paladox>	 BEGIN failed--compilation aborted at /usr/local/bin/apache-fast-test line 28.
[20:52:55] <paladox>	 Ok
[20:53:02] <mutante>	 just keep track of which ones you needed somewhere
[20:53:32] <paladox>	 Ok
[20:53:36] <paladox>	 installing Net::DNS::Resolver
[20:54:06] <mutante>	 then we want the debian package names that provide these modules
[20:54:19] <paladox>	 Ok it is installed now
[20:54:27] <paladox>	 what do you mean by debian package name?
[20:54:49] <mutante>	 you can install these perl modules from distro package, with apt-get
[20:54:54] <mutante>	 instead of cpan
[20:55:08] <mutante>	 and it should be like that 
[20:55:12] <mutante>	 on tin
[20:56:01] <mutante>	 but you need to find which ones exactly
[20:56:13] <mutante>	 and it should be puppetized in a role that is on tin in prod
[20:56:22] <mutante>	 and ideally that is also on phab-tin if possible
[20:57:07] <paladox>	 https://www.theguardian.com/world/2016/jun/30/slovakia-foreign-minister-eu-support-measure-stop-brexit
[20:57:22] <paladox>	 mutante oh
[20:57:27] <paladox>	 not sure how to find the name
[20:57:35] <wikibugs>	 06Labs, 10Horizon, 13Patch-For-Review: Create puppet backend with REST api for labs instance configuration - https://phabricator.wikimedia.org/T133412#2419876 (10yuvipanda) @andrew can you tell me where this is running and how I can test this?
[20:58:18] <mutante>	 paladox: just get the script to run and show "usage" text for now
[20:58:28] <paladox>	 Ok
[20:59:17] <paladox>	 mutante
[20:59:17] <paladox>	 root@phab-tin:/srv/puppet# apache-fast-test modules/apache/manifests/
[20:59:18] <paladox>	 no urls found in modules/apache/manifests/
[20:59:29] <paladox>	 Appending installation info to /usr/local/lib/x86_64-linux-gnu/perl/5.20.2/perllocal.pod
[20:59:29] <paladox>	   NLNETLABS/Net-DNS-1.06.tar.gz
[20:59:29] <paladox>	   /usr/bin/make install  -- OK
[20:59:35] <paladox>	 theres ^^ from cpan
[20:59:57] <mutante>	 paladox: ok, cool. just run "apache-fast-test", do not append random pathes or filenames after it
[21:00:07] <paladox>	 Ok
[21:00:20] <mutante>	 it should show a "usage" text
[21:00:22] <paladox>	 root@phab-tin:/srv/puppet# apache-fast-test
[21:00:23] <paladox>	   usage: /usr/local/bin/apache-fast-test url_file [server spec]
[21:00:23] <paladox>	    url_file: (web or local file containing one http/https URL per line
[21:00:23] <paladox>	      ./urls.txt
[21:00:24] <paladox>	      http://somehost.eqiad.wmnet/test_urls.txt
[21:00:24] <paladox>	    server spec:
[21:00:26] <paladox>	      (none)      use staging server mw1017.eqiad.wmnet
[21:00:28] <paladox>	      pybal       fetch full webserver list from live pybal configuration
[21:00:29] <mutante>	 there you go
[21:00:30] <paladox>	      eqiad|codfw fetch full webserver list for a specific datacenter
[21:00:32] <paladox>	      host1 host2 host3
[21:00:35] <paladox>	 Yep :)
[21:00:47] <mutante>	 you dont need to paste the whole thing, but good
[21:00:58] <mutante>	 so remember when i asked you for the URL file
[21:01:06] <mutante>	 and i said one per line, no formatting etc
[21:01:17] <mutante>	 that was the "url_file" in the text above
[21:01:30] <mutante>	 get that from the pastebin
[21:01:42] <mutante>	 or just make a new text file with URls
[21:02:14] <mutante>	 maybe start with a very simple file, just 2 URLs, one per line
[21:02:27] <mutante>	 and then you run that as
[21:02:47] <mutante>	 apache-fast-test yourfile servername
[21:03:08] <mutante>	 and you replace "yourfile" with your actual file name and "servername" with the actual instance name that you are testing
[21:03:18] <mutante>	 such as phab-03.. but the full name
[21:03:28] <mutante>	 my laptop battery is shutting down.. be back soon..
[21:03:49] <paladox>	 Ok
[21:04:46] <paladox>	 mutante i will wait until you get back since i doint understand what to do next.
[21:20:36] <mutante>	 paladox: paste me the current error in PM
[21:20:47] <paladox>	 Ok
[21:21:01] <paladox>	 mutante i doint currently have error's but if i do i will
[21:33:26] <paladox>	 mutante
[21:33:27] <paladox>	 root@phab-tin:/srv# apache-fast-test rewrite_urls_one_per_line
[21:33:27] <paladox>	 testing 49 urls on 1 servers, totalling 49 requests
[21:33:28] <paladox>	 spawning threads.
[21:34:31] <mutante>	 paladox: there are 2 paramaters to the script, the file with the URLs in it, and the hostname you are testing
[21:34:33] <paladox>	 mutante
[21:34:34] <paladox>	 ..thread1 dropped 10.64.0.47, too many timeouts
[21:34:39] <paladox>	 Oh
[21:34:40] <mutante>	 you should add the second one 
[21:34:45] <mutante>	 default will not work 
[21:34:50] <paladox>	 Add the second one?
[21:35:03] <mutante>	 apache-fast-test filename servername
[21:35:09] <paladox>	 Oh
[21:35:18] <mutante>	 you just had filename there
[21:35:26] <mutante>	 and as servername use something like 
[21:35:33] <mutante>	 phab03.phabricator.eqiad.wmflabs
[21:35:46] <mutante>	 the name of another instance that runs Apache
[21:35:51] <mutante>	 that you are testing
[21:36:15] <paladox>	 Ok
[21:41:55] <wikibugs>	 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Issues with 'webservice' kubernetes backend - https://phabricator.wikimedia.org/T139107#2419652 (10Magnus) Tried this with my tool "wikidata-todo". Result: ``` Warning: include_once(php/common.php): failed to open stream: No such file or directory in /data/project/wiki...
[22:09:23] <mutante>	 we got it to work
[22:09:36] <mutante>	 apache-fast-test <urlfile> on phab-tin
[22:09:48] <mutante>	 the default staging server is phab-03
[22:13:40] <mutante>	 "got to work" = edited the default_staging_server inside the script
[22:13:56] <mutante>	 and then ran it with only one argument, the file with URLs
[22:14:25] <mutante>	 as long as we tried to manually supply the server name as 2nd argument it somehow fails
[22:14:34] <mutante>	 when making the DNS lookup (?)
[22:15:07] <mutante>	 on tin that's not an issue though.. not sure
[22:19:04] <mutante>	 Danny_B: ^ that stuff can be used for better testing now
[22:20:16] <mutante>	 on instance phab-tin, script apache-fast-test , with the only parameter being a plain text file with URLs, and the default server it tests is phab-03 
[22:51:24] <mutante>	 !log phabricator phab-tin now has apache-fast-test to test Apache changes. use phab-03 to change Apache and phab-tin to run the test script from
[22:51:29] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master
[23:03:41] <wikibugs>	 06Labs, 06Operations, 06Project-Admins: Archive old Incident-* projects - https://phabricator.wikimedia.org/T134624#2420338 (10Danny_B) Can #operations and #labs folks please check unclosed tasks in projects listed above? Thanks.