[00:18:38] <mutante>	 where's an example of the "virt-star.eqiad.wmnet" SSL cert being used
[00:18:44] <mutante>	 andrewbogott: ?
[00:19:01] <mutante>	 that's all i need and i am done with all the certs more or less
[00:32:37] <Krenair>	 modules/openstack/manifests/nova/compute.pp:            $certname = "labvirt-star.${site}.wmnet"
[00:32:37] <Krenair>	 modules/openstack/manifests/nova/compute.pp:            $certname = "virt-star.${site}.wmnet"
[00:32:53] <Krenair>	 mutante, ^
[00:34:35] <mutante>	 Krenair: ok, thanks! and re: ldap-mirror, see my revert https://gerrit.wikimedia.org/r/#/c/247951/
[00:35:03] <mutante>	 Krenair: so i guess a random "compute" box will do
[00:35:16] <mutante>	 is the protocol https?
[00:35:24] <Krenair>	 I'm not sure yet
[00:35:27] <mutante>	 'l
[00:35:29] <mutante>	 ok
[00:39:20] <Krenair>	 I can't figure it out, mutante 
[00:39:26] <Krenair>	 YuviPanda might know
[00:39:42] <mutante>	 ok, no worries, looking
[00:39:55] <mutante>	 virt1001.eqiad.wmnet
[00:40:31] <Krenair>	 yes, that box would have it
[00:42:14] <mutante>	 login is different from normal prod
[00:43:24] <Krenair>	 ?
[00:43:31] <mutante>	 i cant get on it yet
[00:43:49] <mutante>	 with either key
[00:46:52] <mutante>	 Krenair: node /^virt100[1-4].eqiad.wmnet/ {
[00:46:54] <mutante>	 vs.
[00:47:01] <mutante>	 Host virt1001.eqiad.wmnet not found: 3(NXDOMAIN)
[00:47:02] <mutante>	 :p
[00:47:19] <Krenair>	 Oh
[00:47:25] <Krenair>	 It might have become labvirt1001?
[00:47:39] <Krenair>	 or not
[00:47:45] <Krenair>	 commit c54b6c923c26662fbbb454f5e46322c23148543b
[00:47:45] <Krenair>	 Author: cmjohnson <cmjohnson@wikimedia.org>
[00:47:45] <Krenair>	 Date:   Thu Jul 30 14:11:51 2015 -0400
[00:47:46] <Krenair>	     Removing dns entries for virt1001-1009 including mgmt since these are going to be removed from the racks.
[00:48:06] <Krenair>	 so it sounds like that node block is redundant now
[00:48:19] <mutante>	 removed from DNS but not removed from puppet = meh
[00:48:43] <Krenair>	 also labvirt1000 doesn't exist
[00:48:47] <mutante>	 ok, so maybe we just dont need to check this cert then
[00:49:10] <mutante>	 i'll make a change to remote that from site.pp
[00:49:12] <mutante>	 remove
[00:52:51] <Krenair>	 did you figure out the labvirt cert mutante?
[00:54:32] <mutante>	 no, but if it was only used on compute nodes
[00:54:38] <mutante>	 and the compute nodes are not in DNS
[00:55:16] <mutante>	 looks at labvirt1001 now though
[00:55:21] <mutante>	 probably there
[00:55:53] <mutante>	 HTTP CRITICAL - Unable to open TCP socket
[00:57:22] <mutante>	 it's one of the other ports there
[00:57:41] <Krenair>	 the old virt* compute nodes are not in dns
[00:57:45] <Krenair>	 labvirt* ones are in use
[00:57:52] <mutante>	 i know
[00:58:06] <mutante>	 now what port is the service running on that uses the cert
[00:58:15] <mutante>	 there are a couple
[00:59:34] <Krenair>	 I ran nmap on a random  labs instance  and found a ton of open ports on labvirt1001
[00:59:42] <mutante>	 exactly, same here
[01:00:05] <mutante>	 so one of these, but i have something coming up in RL, need to continue later
[01:00:13] <Krenair>	 all 59xx
[01:01:45] <mutante>	 yep, saw the same
[01:03:04] <Krenair>	 I couldn't find anything using openssl s_client on these ports
[01:04:13] <mutante>	 hmm, ok. fine with me if we dont need it, then i can call the ticket resolved :p
[01:06:29] <Krenair>	 It's probably important
[01:15:00] <mutante>	 if so, then we should add some docs
[01:15:01] <mutante>	 laters
[03:49:50] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0]
[04:24:53] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0]
[06:26:51] <YuviPanda>	 Krenair: mutante I've no idea, that's all on andrewbogott :) (virt SSL cert)
[06:53:24] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[07:33:26] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:46:38] <shinken-wm>	 PROBLEM - Puppet staleness on tools-k8s-bastion-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]
[14:39:42] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations, 3labs-sprint-117: Labs test cluster in codfw - https://phabricator.wikimedia.org/T114435#1745441 (10mark) So we're down to just one system (32 GB) in warranty now?  This is Approved.
[15:08:59] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: add icinga::gsbmonitoring values in hiera [labs/private] - 10https://gerrit.wikimedia.org/r/248042 
[15:31:19] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] add icinga::gsbmonitoring values in hiera [labs/private] - 10https://gerrit.wikimedia.org/r/248042 (owner: 10Alexandros Kosiaris)
[15:56:32] <Luke081515>	 Hello, can someone help me with installing apache? I installed it, and started it, but I don't know, how I can configure a domain
[16:05:25] <chasemp>	 Luke081515: assuming you are not in tools you would most likely want to look at https://wikitech.wikimedia.org/wiki/Special:NovaProxy
[16:05:44] <chasemp>	 the majority of incoming http is reverse proxied and we don't hand out top level domains, you would be something.wmflabs.org
[16:06:27] <Luke081515>	 that's ok, but I want to set up some there now, and need Apche webserver for this. How can I configure this?
[16:07:18] <chasemp>	 Luke081515: it varies pretty widely for what you are doing but this is a basic setup http://code.tutsplus.com/articles/apache-2-basic-configuration-on-unix-like-systems--net-26607
[16:07:39] <Luke081515>	 chasemp: Thanks
[16:08:53] <Luke081515>	 chasemp: One question left: Do you know, which directory I have to choose for that config files? Is the home diretoy enough?
[16:08:57] <Luke081515>	 *directory
[16:10:07] <chasemp>	 if this is a normal vm w/ apache and I want a basic setup it would go somewhere like /etc/apache2/sites-enabled/mysite.conf
[16:10:38] <Luke081515>	 thanks!
[16:15:18] <Luke081515>	 chasemp: One problem left: Permission denied
[16:15:34] <Luke081515>	 I could not add a new file to this directory
[16:17:37] <Luke081515>	 do you know, how I can fix this?
[16:17:52] <wikibugs>	 6Labs, 10Tool-Labs, 10Tool-Labs-tools-Database-Queries: FULLTEXT not working - https://phabricator.wikimedia.org/T115940#1745669 (10Dispenser) 5Invalid>3Resolved Thank you for providing the solution and using utf8mb4.  I checked out your search engine claim: only [[http://dba.stackexchange.com/questions/...
[16:25:59] <Luke081515>	 I wonder, why this edit is not possible. I'm project admin, and created that instance. Can somebody help me?
[16:48:06] <chasemp>	 I think project admins have sudo
[16:48:15] <chasemp>	 sudo you try 'sudo <your command>'
[16:49:44] <Luke081515>	 -bash: httpd.conf: Permission denied
[16:49:48] <Luke081515>	 hm
[16:51:04] <Luke081515>	 chasemp: Sorry, but this does not help, I can't create it again
[16:52:11] <chasemp>	 put the full terminal output here https://etherpad.wikimedia.org/p/Luke081515
[16:52:50] <Luke081515>	 chasemp: Done
[16:54:25] <chasemp>	 this isn't a command that makes sense
[16:54:25] <chasemp>	  sudo cat > httpd.conf
[16:54:28] <ebernhardson|awy>	 which projects should i add requests for quota expansion in labs to?
[16:54:31] <ebernhardson|awy>	 (in phab)
[16:54:43] <chasemp>	 Luke081515: try "sudo nano httpd.conf"
[16:55:25] <wikibugs>	 6Labs: Increase cpu and disk quota for the 'search' group - https://phabricator.wikimedia.org/T116292#1745783 (10EBernhardson)
[16:55:27] <chasemp>	 ebernhardson: probably just #labs
[16:55:37] * ebernhardson is worried it will get lost and not triaged :P
[16:55:47] <Luke081515>	 chasemp: Thanks, works now
[16:56:00] <chasemp>	 ebernhardson: yeah...what's the ask?
[16:56:42] <ebernhardson>	 chasemp: mostly disk space, but possibly more cpu/memory as well. It's to build out our relevancy lab (https://phabricator.wikimedia.org/T115615, https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Relevance_Lab)
[16:57:36] <chasemp>	 it's pretty highly dependent on specifics (how much)
[16:57:51] <chasemp>	 we did a bit of talking on this last week and answer is basically, if we can fulfill it in labs now sanely ok
[16:58:00] <chasemp>	 beyond a certain point it has to be figured into budgets and needs lead time
[16:58:06] <chasemp>	 but it all boils down to specific asks
[16:58:19] <ebernhardson>	 yea, is there any way to have visibility into what is available in labs?
[16:58:28] <ebernhardson>	 i know i can look at ganglia to see the total size of the cluster, but not the utilization
[16:58:51] <ebernhardson>	 i don't know if it sits with half the memory used by VM's, or 95%
[16:59:16] <chasemp>	 I guess, not at all trolling here, but why does what is avail change what you need?
[16:59:41] <chasemp>	 the what is available question gets complicated w/ nodepool reserves and other teams next quarter expectations etc
[16:59:48] <ebernhardson>	 it means i can independantly think about our requirements, and either try and fit within the constraints or figure out if we need to get more stuff
[17:00:06] <ebernhardson>	 but if i have no clue what the constraints are, i just have to guess
[17:00:52] <ebernhardson>	 as in, i don't know if asking for 32G more memory and another TB of disk space is a big ask, or a tiny one
[17:01:02] <chasemp>	 educated minimum guess +20+ with an eye towards budget as it works out
[17:01:25] <chasemp>	 sure but you must know if you need 32G of memory vs 10G?
[17:01:43] <ebernhardson>	 well, we can trade off disk for memory
[17:01:52] <ebernhardson>	 well, disk iops to be precise
[17:01:55] <chasemp>	 yeah
[17:02:58] <chasemp>	 so per project quata should show up on https://horizon.wikimedia.org/project/
[17:03:10] <chasemp>	 and you could kind of play it that how many default project allocations are you talking
[17:03:25] <chasemp>	 is this project going to be fit within an normal project allocation or is it 3x
[17:03:31] <chasemp>	 that's a somewhat sane measuring stick at least
[17:03:57] <ebernhardson>	 we've tested on a current standard quota, and the answer is we can do a reasonable level of searching but we need 2-3x more disk space
[17:04:24] <chasemp>	 are iops a limitation?
[17:04:27] <ebernhardson>	 but then by using that much data, we will be using more iops, and since i've seen recent emails about disk space not being an issue, but disk iops can be, more memory would relieve that
[17:04:29] <chasemp>	 now I mean
[17:04:36] <ebernhardson>	 not yet
[17:04:52] <ebernhardson>	 but we havn't loaded as much data into it as we would like to, which might change the equation (but not too far)
[17:05:09] <chasemp>	 we have more asks disk space wise in the next 3 quarters than we could fulfill if you consider research
[17:05:34] <chasemp>	 but we are on teh cusp of allocating specific hardware and sometime in teh next few quarters making disk allocation more flexible
[17:06:50] <chasemp>	 3 projects worth of disk space sould be acheivable? It seems possible you are talking a general 3x project, but iops won't scale that way vertically
[17:07:01] <chasemp>	 all generalities though
[17:08:14] <ebernhardson>	 based on current vm sizes, of 8 cores and 16G memory, we can only fit two of those in a single project size, so i suppose technically we already have most of the memory we need available via quota, we just can't boot it because we don't have the vcpu's available to boot another large instance
[17:09:24] <chasemp>	 say you squeezed two more large instances into this same project + extra disk
[17:09:34] <chasemp>	 that sounds like about where you are at?
[17:09:43] <ebernhardson>	 that's where we would like to be, yea
[17:10:00] <ebernhardson>	 but again, if thats a big ask we could try on the current 2 instances with more disk, or with a 3rd instance and more disk
[17:10:09] <ebernhardson>	 the only real strict requirement is more disk :) 
[17:10:40] <ebernhardson>	 the more memory was mostly just an idea to relieve iops, i think the project would still work fine without it
[17:11:07] <chasemp>	 so if you proposed that way outlining what you need added quota wise to a current project and tag labs it should get looked at
[17:11:14] <chasemp>	 if not you can bug me 
[17:11:18] <ebernhardson>	 ok, thanks!
[17:13:59] <wikibugs>	 6Labs: Increase cpu and disk quota for the 'search' group - https://phabricator.wikimedia.org/T116292#1745847 (10EBernhardson) based on current vm sizes, of 8 cores and 16G memory with 160G of disk, we can only fit two of those in a single project size (bumps up against the 20 core maximum before we use up the m...
[17:17:11] <wikibugs>	 6Labs: Increase cpu and disk quota for the 'search' group - https://phabricator.wikimedia.org/T116292#1745854 (10EBernhardson)
[17:46:25] <Luke081515>	 chasemp: I created that file, and I can edit it, but with list, I can not see it. Do you know why?
[17:47:09] <chasemp>	 paste your terminal output https://etherpad.wikimedia.org/p/Luke081515 your question isn't making sense
[17:49:43] <Luke081515>	 chasemp: seems to be a confusing error a while ago. I can see the file know
[17:54:49] <wikibugs>	 6Labs: Puppet failures in labs if "Share home directories across instances" or "Create shared project storage" are unchecked - https://phabricator.wikimedia.org/T88420#1745957 (10chasemp)
[18:01:37] <wikibugs>	 6Labs: Increase quota's for search project in labs - https://phabricator.wikimedia.org/T109377#1745992 (10EBernhardson)
[18:01:38] <wikibugs>	 6Labs: Increase cpu and disk quota for the 'search' group - https://phabricator.wikimedia.org/T116292#1745993 (10EBernhardson)
[18:07:49] <wikibugs>	 6Labs, 10Wikimedia-Labs-General, 6operations, 7Database, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1746024 (10chasemp)
[18:32:49] <Luke081515>	 chasemp and other's who know that: Can you tell me, how I can enable mod_php and mod_rewrite at apache?
[18:40:14] <valhallasw`cloud>	 Luke081515: symlink from mods-available to mods-enabled and restart apache?
[18:40:45] <Luke081515>	 symlink?
[18:41:39] <Luke081515>	 valhallasw`cloud: How can I enable a symlink?
[18:41:53] <valhallasw`cloud>	 https://www.digitalocean.com/community/tutorials/how-to-install-configure-and-use-modules-in-the-apache-web-server suggests you should use a2enmod
[18:41:58] <Luke081515>	 ah, thanks
[20:16:16] <wikibugs>	 6Labs, 7Article-Recommendation: Investigate possible instances with 32G of RAM to test article-reccomendations - https://phabricator.wikimedia.org/T116321#1746391 (10yuvipanda) 3NEW
[20:20:09] <YuviPanda>	 andrewbogott: when you have a break from the new servers... https://phabricator.wikimedia.org/T116321?workflow=create
[22:00:05] <wikibugs>	 6Labs, 10Labs-Infrastructure, 6operations, 7Monitoring: monitor expiration of labvirt-star SSL cert - https://phabricator.wikimedia.org/T116332#1746787 (10Dzahn)
[22:01:21] <wikibugs>	 6Labs, 10Labs-Infrastructure, 6operations, 7Monitoring: monitor expiration of labvirt-star SSL cert - https://phabricator.wikimedia.org/T116332#1746798 (10Dzahn) a:5Dzahn>3Andrew
[22:04:21] <wikibugs>	 6Labs, 10Labs-Infrastructure, 6operations, 7Monitoring: monitor expiration of labvirt-star SSL cert - https://phabricator.wikimedia.org/T116332#1746802 (10Dzahn)  @neon:/usr/lib/nagios/plugins# ./check_http -I labvirt1001.eqiad.wmnet -p 5925 -S  CRITICAL - Cannot make SSL connection  ...error:140770FC:SSL...
[22:12:48] <wikibugs>	 6Labs, 10Labs-Infrastructure, 6operations, 7Monitoring: monitor expiration of labvirt-star SSL cert - https://phabricator.wikimedia.org/T116332#1746850 (10Dzahn)  <apergos> tcp        0      0 *:5906                  *:*                     LISTEN      8360/**kvm**
[22:20:04] <wikibugs>	 6Labs, 7Article-Recommendation: Investigate possible instances with 32G of RAM to test article-reccomendations - https://phabricator.wikimedia.org/T116321#1746867 (10Andrew) This is possible -- will this be a long-lived instance or a one-off?
[22:56:37] <kaldari>	 every time I try to run a query on a tool labs db it gives me the error: "SELECT command denied to user 'u1271'@'xx.xx.xx.xx'"
[22:56:48] <kaldari>	 it was working fine last time I tried
[22:57:42] <kaldari>	 YuviPanda: any idea what changed? ^
[22:57:45] <Krenair>	 what database are you trying to use exactly?
[22:57:58] <kaldari>	 I tried ptwiki most recently
[22:58:01] <Krenair>	 not  ptwiki_p?
[22:58:11] <Krenair>	 that'll cause you to be denied
[22:58:23] <kaldari>	 I mean ptwiki_p
[22:59:47] <Krenair>	 WFM.
[23:00:10] <Krenair>	 select @@hostname; ?
[23:01:12] <kaldari>	 nevermind, I was using the wrong dbname in the SQL (forgot the _p) :P
[23:02:03] <Krenair>	 ...
[23:02:12] <Krenair>	 I did bring up the _p thing for a reason :p
[23:03:59] <kaldari>	 yes, time for a trout-slap
[23:34:13] <wikibugs>	 6Labs: Puppet failures in labs if "Share home directories across instances" or "Create shared project storage" are unchecked - https://phabricator.wikimedia.org/T88420#1747092 (10yuvipanda) 5Open>3Invalid a:3yuvipanda Not true anymore since NFS is controlled from elsewhere!