[00:09:41] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1623267 (10yuvipanda) Ok, so if I set the token in .kube/config and then select it with --user... I get: ```yuvipanda@tools-k8s-master-01:~$ kubectl --insecure-skip-tls-verify -s https://tools-k8s-ma... [00:15:48] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1623292 (10yuvipanda) so if I add another line: ```{"user": "lolrrit-wm"}``` to the ABAC file it works. I now have to check if the namespace filter is available in the verison of kubernetes we're ru... [00:19:54] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1623296 (10yuvipanda) Ok, so the code definitely exists (and so do the docs) for namespace field ABAC in the version we are running. [00:28:20] I can't create or delete instances [00:28:29] is there some sort of maintenance ongoing? [00:29:18] duh, there even was a mail about it, which I missed somehow [00:29:22] don't mind me [02:42:07] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1623473 (10yuvipanda) Adding just a `{"namespace": "lolrrit-wm"}' doesn't help things [02:48:36] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1623475 (10yuvipanda) Hah, looks like I'm running into https://github.com/kubernetes/kubernetes/issues/13097 [03:32:13] Is there anything going on with Tool Labs right now? I can't connect to Tool Labs (though I can connect to my Wikimedia Labs instance) [03:40:04] I just logged in fine [03:40:17] how are you connecting? [03:40:22] ssh to tools-login? [03:40:42] Norepinephrine:~ jh$ ssh harej@tools-login.wmflabs.org [03:40:42] channel 0: open failed: connect failed: Connection timed out [03:40:42] ssh_exchange_identification: Connection closed by remote host [03:40:58] try it with -vvv? [03:42:11] It may be an issue with my config file [03:42:15] maybe [03:42:19] paste your .ssh/config? [03:42:21] https://www.irccloud.com/pastebin/9MPTh99x/ [03:43:40] how are you connecting to your other non-tools instance? [03:45:26] ssh harej@librarybase-reston-01.eqiad.wmflabs [03:46:18] harej: try now? [03:46:21] am tailing the log file [03:46:24] to see what's going on [03:49:15] anything insightful on your end? [03:52:22] also: [03:52:24] https://www.irccloud.com/pastebin/kqAJ6lVd/ [03:54:25] harej: no [03:55:38] harej: can you PM me your ip address? [03:58:26] Hmm [03:58:31] harej: hmm that IP isn't hitting the host [03:58:34] Shouldn't you be able to ping tools-login.wmflabs.org from bastion.wmflabs.org? [03:58:35] at least from what I can see [03:59:00] are you saying he's not hitting bastion or tools-login? [03:59:10] Well in any case, commenting out the config file makes things work again. [03:59:18] harej: I see it now [03:59:27] Yes, I logged in after commenting out the config file :P [03:59:31] harej, but then you can't connect to your instance? [03:59:38] harej: I think you need to get rid of the *.wmflabs.org from your Host [03:59:43] those you should connect directly [04:00:13] That will mean you have to always ssh with the correct remote username to *.wmflabs.org [04:00:22] that's already the case [04:00:27] no? [04:00:31] except for some hardcoded things [04:00:35] it's in puppet somewhere [04:00:43] no, because of "User harej" in the *.wmflabs.org host in his config [04:01:21] no I mean if you hit *.wmflabs.org from inside labs [04:01:25] you will get wildly varying results [04:06:28] YuviPanda, so what about bastion.wmflabs.org not being able to ping tools-login.wmflabs.org? [04:22:17] Krenair: labs can't hit public floating IPs from inside there [04:22:19] *inside labs [04:22:35] Krenair: https://phabricator.wikimedia.org/T95288 and related [05:00:49] 6Labs, 10Labs-Infrastructure: New database for designate pool manager - https://phabricator.wikimedia.org/T112041#1623557 (10Andrew) 3NEW a:3jcrespo [07:47:01] 6Labs, 10Tool-Labs: Setup a way to store secrets and access them from puppet inside the Tool Labs project - https://phabricator.wikimedia.org/T112005#1623746 (10valhallasw) Can we solve this using just ssh auth? We already have host-based auth infra in place, after all. Otoh, having a generic secure storage a... [07:59:26] 10Wikibugs: wikibugs - throttle output, don't get kicked for flooding - https://phabricator.wikimedia.org/T112032#1623766 (10valhallasw) We could do this by overriding IrcBot.send_line, although I'm not completely sure how to do this in an asyncio-friendly way. Obvious options are: * a simple yield asyncio.slee... [08:29:21] 6Labs, 10Labs-Infrastructure: New database for designate pool manager - https://phabricator.wikimedia.org/T112041#1623796 (10jcrespo) p:5Triage>3High [08:30:32] 6Labs, 10Labs-Infrastructure, 7Database: New database for designate pool manager - https://phabricator.wikimedia.org/T112041#1623557 (10jcrespo) [08:32:14] 6Labs, 7Tracking: New Labs Projects (TCB-Team) - https://phabricator.wikimedia.org/T112049#1623799 (10Addshore) 3NEW [08:32:56] 6Labs, 7Tracking: New Labs Projects (TCB-Team) - https://phabricator.wikimedia.org/T112049#1623799 (10Addshore) [09:19:27] 6Labs: New Labs Projects (TCB-Team) - https://phabricator.wikimedia.org/T112049#1624122 (10Revi) [09:48:03] 6Labs, 10Labs-Infrastructure, 7Database, 5Patch-For-Review: New database for designate pool manager - https://phabricator.wikimedia.org/T112041#1624254 (10jcrespo) 5Open>3Resolved Database created on m5-master for the user designate. In the future, I would try to avoid the underscore character, as it p... [13:51:49] Hello, is here somebody with admin access to phabricator? [13:52:27] Luke081515 you need andre [13:52:31] @seenrx andre [13:52:33] petan: Last time I saw andrewbogott they were talking in the channel, they are still in the channel. It was in #wikimedia-operations at 9/10/2015 1:44:10 PM (8m21s ago) (multiple results were found: andre__, andrewbogott_afk, jandresen, andrew_____, andrewbogott_ and 93 more results) [13:52:33] This search took too much time, please optimize query [13:52:45] o.O [13:52:50] what the fuck [13:52:53] I have never seen this [13:53:09] hm [13:53:11] it actually answers and then it complains it took too much time o.o [13:53:29] petan, where are you getting that? [13:53:36] on phab? [13:53:39] it's like "ok dude I will do that job you asked me to, but you know... fuck you" [13:53:50] jynus: getting what [13:53:54] sorry, petan [13:54:04] I meant, Luke081515 [13:54:17] oh, it is the bot [13:54:24] yes it's the bot [13:54:26] it started talking [13:54:29] for some reason [13:54:33] I want to ask somebody, if he can give me a copy of a configuration of, wait a moment [13:54:34] it's smarter than we think [13:54:44] well, I was interested if I can help Luke081515 [13:55:05] the config of maniphest.statuses [13:55:24] Luke081515, that is the phabricator team [13:55:54] if it is not a privacy issue, it should be on operations/puppet [13:56:13] no, it is nothing problematic [13:56:30] just the configuration of task statuses [13:56:35] but not my field of expertise, so I may not be very helpful there [13:57:12] this should be the URL where admins can read this config: https://phabricator.wikimedia.org/config/edit/maniphest.statuses/ [13:57:26] for normal users, there is just 403 [13:57:44] search here: https://github.com/wikimedia/operations-puppet/tree/production/modules/phabricator [13:57:57] ok, thanks for that link ;) [13:58:01] it is the only thing I can tell you that may be helpful :-) [13:58:33] hm, I think it is not there [13:58:47] 6Labs, 3labs-sprint-113: Separate scratch and tools NFS volumes to separate physical devices - https://phabricator.wikimedia.org/T111802#1625260 (10coren) 5Open>3Resolved This has completed succesfully. [13:59:21] I can tell you that some things on phav are kept private (only to people that has signed an NDA) for security reasons [13:59:40] but again, I am not your guy for those kind of things [14:00:23] yes, I know, the whole config is private (I got also an private instance of phab), but this entry just says, that you can choose between for example invalid and declined [14:02:02] andrewbogott: Are you here right now? [14:02:25] `*at the moment [14:02:26] andrewbogott is a bit busy at the moment [14:02:44] ok, it is not important [14:02:52] I am actually helping him doing an important task [14:03:08] ok, than, good luck ;) [14:03:27] maybe try later, when america wakes up completelly :-) [14:04:33] ok, I'm at MESZ, that's a bit away I thing ;). What time is it at your timezone? [14:05:10] Luke081515: what exactly are you looking for? [14:05:26] the config of maniphest.statuses at the WMF phabricator [14:05:41] I'm searching for transaction.icon entrys [14:06:07] Luke081515: https://github.com/wikimedia/operations-puppet/blob/52727eefd7142675a1e3eae43ffa578c38eab2a1/modules/phabricator/data/fixed_settings.yaml#L99 [14:06:24] JohnFLewis: Thanks a lot [14:07:13] Welcome. fyi jynus ^ resolved :) [14:07:36] thanks, as I said, phab was not my domain :-) [14:07:50] No problem ;) [14:08:03] I pointed to the right repo, though [14:08:12] :-) [14:08:13] It's hardly mine either :) just searching what you wanted gave the result however [14:08:33] ah, I found the right one ;) [14:19:45] 6Labs, 3labs-sprint-113: Evaluate gridengine's use of NFS and (possibly) move it to a different volume - https://phabricator.wikimedia.org/T111797#1625333 (10coren) a:3coren Right now, the only shared directory between gridengine nodes is `/var/lib/gridengine` which contains (a) the job spools, (b) the share... [14:34:22] 6Labs, 10Tool-Labs: Setup a way to store secrets and access them from puppet inside the Tool Labs project - https://phabricator.wikimedia.org/T112005#1625404 (10scfc) What's the purpose of this? If it's just a one-time job when setting up a new Kubernetes host, doing it manually seems acceptable to me (and mu... [14:37:38] 6Labs, 10Tool-Labs: Setup a way to store secrets and access them from puppet inside the Tool Labs project - https://phabricator.wikimedia.org/T112005#1625429 (10yuvipanda) 1. Adding user accounts - need to be added in the central apiserver and provision password on each user's homedir. Can't do it like we do f... [14:42:01] valhallasw`cloud: Coren andrewbogott ^ if you have opinions on that [14:44:08] * Coren checks. [14:44:48] 6Labs, 10Labs-Infrastructure, 7Database: fix m5 shard mysql issues with default encoding binary - https://phabricator.wikimedia.org/T112103#1625477 (10jcrespo) 3NEW [14:52:50] 6Labs, 10Tool-Labs: Setup a way to store secrets and access them from puppet inside the Tool Labs project - https://phabricator.wikimedia.org/T112005#1625522 (10scfc) As I wrote in T107993#1567549: > You don't need to involve NFS (directly). You can set up the certificate generator on any machine as a HTTP/wh... [14:55:10] 6Labs: New Labs Projects (TCB-Team) - https://phabricator.wikimedia.org/T112049#1625535 (10yuvipanda) Projects should not be 'team' scoped but 'project' scoped. Want to rename it to a particular thing you guys want to test? Creating new projects should be much quicker these days so can have a larger number of we... [14:58:08] 6Labs, 10Tool-Labs: Setup a way to store secrets and access them from puppet inside the Tool Labs project - https://phabricator.wikimedia.org/T112005#1625548 (10yuvipanda) That is a lot more complexity than just setting up a private puppetmaster :) we also shouldn't be doing any extra, non upstreamable pieces... [15:08:09] 6Labs, 10Tool-Labs: Setup a way to store secrets and access them from puppet inside the Tool Labs project - https://phabricator.wikimedia.org/T112005#1625638 (10coren) I'm not seeing a less bad way of doing it offhand. It might perhaps be possible to //actually// do it the replicas.my.cnf way by having lassto... [15:10:04] 6Labs, 10Tool-Labs: Setup a way to store secrets and access them from puppet inside the Tool Labs project - https://phabricator.wikimedia.org/T112005#1625645 (10yuvipanda) How would we be able to do it in labstore without k8s having nfs? The credentials will have to be generated somwehere (k8s master or labsto... [15:12:02] 6Labs, 10Labs-Infrastructure, 7Database: fix m5 shard mysql issues with default encoding binary - https://phabricator.wikimedia.org/T112103#1625656 (10hashar) #Nodepool uses [[ https://github.com/PyMySQL/PyMySQL PyMySQL ]] a pure python client. It connects to m5 with: `mysql+pymysql://user:pass@host/dbname`... [15:13:35] 6Labs, 10Tool-Labs: Reduce SGE NFS usage - https://phabricator.wikimedia.org/T111158#1625661 (10scfc) [15:13:36] 6Labs, 3labs-sprint-113: Evaluate gridengine's use of NFS and (possibly) move it to a different volume - https://phabricator.wikimedia.org/T111797#1625662 (10scfc) [15:19:04] 6Labs, 3labs-sprint-113: Evaluate gridengine's use of NFS and (possibly) move it to a different volume - https://phabricator.wikimedia.org/T111797#1625688 (10mark) >>! In T111797#1625333, @coren wrote: > Right now, the only shared directory between gridengine nodes is `/var/lib/gridengine` which contains (a) t... [15:32:56] 6Labs, 3labs-sprint-113: Evaluate gridengine's use of NFS and (possibly) move it to a different volume - https://phabricator.wikimedia.org/T111797#1625835 (10coren) @mark: The biggest disadvantage is the loss of history - right now, we can add and remove nodes on the grid freely without loosing logs and histor... [15:35:33] 6Labs, 3labs-sprint-113: Evaluate gridengine's use of NFS and (possibly) move it to a different volume - https://phabricator.wikimedia.org/T111797#1625863 (10mark) >>! In T111797#1625835, @coren wrote: > @mark: The biggest disadvantage is the loss of history - right now, we can add and remove nodes on the grid... [15:39:51] andrewbogott: would you happen to know why we don't expose ssh key pairs on the Ec2 metadata service ? http://169.254.169.254/latest/meta-data/public-keys/ does not exist :/ [15:40:11] we don’t use that kind of keypair at all, it’s all done via ldap [15:40:32] I mean, you’re welcome to, but labs doesn't [15:40:42] is that something we could enable for the contintcloud project? [15:41:07] turns out nodepool generates key pairs and upload them to openstack so cloud-init can create a 'debian' user that uses those credentials [15:41:26] let it then ssh as debian on the image to refresh it such as running puppet [15:41:30] I don’t know, don’t have time to look at it now [15:41:34] yeah I guess [15:48:07] 6Labs: New Labs Projects (TCB-Team) - https://phabricator.wikimedia.org/T112049#1625939 (10Addshore) TCB-AG [15:48:19] YuviPanda: ^^ totally forgot that you said that... hah [15:49:50] addshore: :) ok! must it be tcb-ab instead of just attributiongenerator or something? :) [15:49:59] * YuviPanda hates team names in things, and I think I told you why when I was there [15:50:33] YuviPanda: you could call it Lizenzverweisgenerator ;) [15:50:38] just kidding.... [15:50:43] too big :P [15:50:45] ..... you could call it "file-reuse" [15:50:49] oh? [15:51:00] https://github.com/wmde/file-reuse [15:51:02] I am ok calling it tcb-ag if that's the most identifable form of the name [15:51:10] s/name/project/ [15:51:13] but file-reuse sounds good to me too [15:51:21] yeh, go for file-reuse [15:51:31] 6Labs: New Labs Projects (TCB-Team) - https://phabricator.wikimedia.org/T112049#1625964 (10Addshore) "file-reuse" [15:52:16] 6Labs, 3labs-sprint-113: Evaluate gridengine's use of NFS and (possibly) move it to a different volume - https://phabricator.wikimedia.org/T111797#1625965 (10coren) (Just to clarify things, when I stated "neither particularly large nor especially high volume" above that applied to the logs and journals only, n... [15:54:38] 6Labs, 10Salt, 6operations: salt does not run reliably for toollabs / labs generally - https://phabricator.wikimedia.org/T99213#1625979 (10ArielGlenn) Authentication errors: finally found one at least of the causes. there's some cleanup script that deletes certain salt keys if they don't match a particular... [15:55:21] 10Wikibugs: wikibugs - throttle output, don't get kicked for flooding - https://phabricator.wikimedia.org/T112032#1625990 (10Legoktm) We could implement the rate limit on the wikibugs.py side and only push something into redis every second? Yay hacks. [15:57:13] 6Labs, 10Salt, 6operations: salt does not run reliably for toollabs / labs generally - https://phabricator.wikimedia.org/T99213#1626007 (10ArielGlenn) Note that reauth of all minions after key rotation takes a little time, this is by design. you don't want several hundred or a thousand hosts all trying to do... [15:58:47] 6Labs, 10Salt, 6operations: salt does not run reliably for toollabs / labs generally - https://phabricator.wikimedia.org/T99213#1626023 (10ArielGlenn) oh and in the long term we shouldn't be running such a script once a minute. let's delete salt keys on instance deletion etc [15:58:49] 6Labs, 10Tool-Labs: Setup a way to store secrets and access them from puppet inside the Tool Labs project - https://phabricator.wikimedia.org/T112005#1626024 (10coren) I should think it fairly easy to transfer credentials to the k8s master from labstore (rsync, or something else); but that's an extra piece of... [16:00:23] 6Labs, 10Tool-Labs: Setup a way to store secrets and access them from puppet inside the Tool Labs project - https://phabricator.wikimedia.org/T112005#1626029 (10yuvipanda) Indeed, it's 'another piece of home-spun software' that I hope we do not have. [16:03:52] 6Labs, 3labs-sprint-113: Evaluate gridengine's use of NFS and (possibly) move it to a different volume - https://phabricator.wikimedia.org/T111797#1626040 (10coren) Oh, wait, I just noticed a possible cause of miscommunication: I am speaking about the spool //DB//, and the document you refer to is talking abo... [16:06:29] 6Labs, 10Salt, 6operations: salt does not run reliably for toollabs / labs generally - https://phabricator.wikimedia.org/T99213#1626065 (10ArielGlenn) --rotate-aes-key this option to salt-key -d in the script will stop the rotations in case/when this comes up again. the quickie fix. [16:10:41] 6Labs, 10Salt, 6operations: salt does not run reliably for toollabs / labs generally - https://phabricator.wikimedia.org/T99213#1626084 (10akosiaris) OK, intrigued. Which script ? [16:28:05] 6Labs, 7Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#1626216 (10yuvipanda) [16:28:07] 6Labs: New Labs Projects (TCB-Team) - https://phabricator.wikimedia.org/T112049#1626212 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Done! [16:33:34] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1626240 (10yuvipanda) I've cherry-picked a fix to ^ and it's all good now. Ongoing discussions on the issue and pull request associated with it. Now to figure out the appropriate bits of ABAC require... [16:58:28] (03PS1) 10Alexandros Kosiaris: add maps private hieradata [labs/private] - 10https://gerrit.wikimedia.org/r/237420 [17:04:40] (03PS2) 10Alexandros Kosiaris: add maps private hieradata [labs/private] - 10https://gerrit.wikimedia.org/r/237420 [17:10:19] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] add maps private hieradata [labs/private] - 10https://gerrit.wikimedia.org/r/237420 (owner: 10Alexandros Kosiaris) [18:28:41] !log ores disabled restart crontab on ores-worker-01 under halfak's user, have set it to keep insane levels of logs to debug [18:28:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [19:15:31] 6Labs, 10Parsoid, 10VisualEditor, 10wikitech.wikimedia.org: Parsoid on wikitech fails - https://phabricator.wikimedia.org/T108776#1627379 (10Jdforrester-WMF) [19:40:27] 6Labs, 6Discovery, 7Elasticsearch: Replicate production elasticsearch indices to labs - https://phabricator.wikimedia.org/T109715#1627582 (10yuvipanda) @Ebernhardson so does this mean we can put some of the old lsearchd machines onto the labs-vnet and replicate our production cluster to them? Those boxes had... [19:51:10] 6Labs, 6Discovery, 7Elasticsearch: Replicate production elasticsearch indices to labs - https://phabricator.wikimedia.org/T109715#1627614 (10EBernhardson) how many boxes are there? the main concern would be if 2.5TB of data will fit. [19:51:50] 6Labs, 6Discovery, 7Elasticsearch: Replicate production elasticsearch indices to labs - https://phabricator.wikimedia.org/T109715#1627615 (10yuvipanda) @Ebernhardson we'd need to pick and choose from wikitech.wikimedia.org/wiki/Server_Spares and then justify it :) [19:56:28] 6Labs, 6Discovery, 7Elasticsearch: Replicate production elasticsearch indices to labs - https://phabricator.wikimedia.org/T109715#1627648 (10yuvipanda) We'd also need to make sure that deleted / revdelled content doesn't show up. [20:11:35] 6Labs, 6Discovery, 7Elasticsearch: Replicate production elasticsearch indices to labs - https://phabricator.wikimedia.org/T109715#1627717 (10EBernhardson) I have actually just written the code to send cirrussearch updates to multiple clusters in T109734. This would be a good test of that in addition to the... [20:12:22] 6Labs, 6Discovery, 7Elasticsearch: Replicate production elasticsearch indices to labs - https://phabricator.wikimedia.org/T109715#1627720 (10yuvipanda) Yeah, a simple reverse proxy seems easily doable. [21:03:44] legoktm: https://tools.wmflabs.org/forrestbot/log.txt seems OK but ReleaseTaggerBot hasn't added anything to wmf23 since Thursday - https://phabricator.wikimedia.org/project/feed/1472/ [21:03:53] legoktm: Oh, I see. [21:03:57] https://gerrit.wikimedia.org/r/#/c/235507/ [21:05:02] legoktm: How to fix? Manually delete the e-mail about that one from the queue? [21:05:48] James_F: yes. and complain to the patch author >.> [21:05:55] legoktm: The latter bit is done. [21:05:59] legoktm: How do I do the former? [21:06:44] James_F: the gmail credentials are in the config.json file on tool labs. Log in and delete it from the inbox [21:06:47] legoktm: However, why is RTB running on apps/android/wikipedia anyway? [21:06:56] OK. [21:07:56] James_F: it has to parse the email to figure out the project, so it also grabs the bug # at the same time [21:08:03] legoktm: Ooh. [21:08:04] * James_F nods. [21:13:08] Whee. [21:13:10] Thanks, legoktm. [21:15:36] !log tools.forrestbot Deleted e-mail about https://gerrit.wikimedia.org/r/#/c/235507/ which was breaking the bot and manually triggered a run. [21:15:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.forrestbot/SAL, Master [21:29:18] legoktm: should we have a voting job for all projects that validates the commit message? [21:29:33] re: forrestbot failures [21:30:05] bd808: hehe, James_F and I talked about doing that once. I think that would be a great idea [21:30:28] do we have a spec for a valid commit messasge? [21:31:02] https://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines [21:31:14] I always used to have svn pre-commit hooks for validations like this [21:31:54] I think we'd validate: Bug: is properly formatted, no line between headers and Change-Id, subject line length [21:33:29] seems reasonable [21:34:21] space between 'Bug:' and the ticket number [21:35:02] No "Task: xxxx" line [21:42:28] bd808: No "Fixes: " line either. :-) [21:45:09] I'm trying to create a new instance on labs, and I get "Failed to create instance." each time. [21:45:24] ragesoss: what project? [21:45:33] bd808: globaleducation [21:45:46] andrewbogott, ^ [21:46:06] ragesoss: there was an email to labs-l about that [21:46:11] ragesoss: should be fixed shortly [21:46:19] https://lists.wikimedia.org/pipermail/labs-l/2015-September/004020.html [21:47:24] ragesoss: meanwhile, maybe check that you’re not over quota? It’s been working for me most of the time... [21:47:55] I was trying to check that, but apparently you can only see quota for project you belong to? [21:48:30] ragesoss: look at https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProject&action=displayquotas&projectname=globaleducation and see if you have space for the instance [21:53:59] yeah, globaleducation is at 20/20 cores [21:55:36] okay, cool. so, I just need to delete some more stuff. [22:00:24] three instances are (deleting), but it's taking a while. that's normal? [22:12:31] it's been like 15 minutes, the instances still aren't deleted. [22:27:43] ragesoss: I don’t know… I’ll look. [22:28:13] Oh, you know, some of the virt nodes are depooled right now, I bet that means that deletion is deferred as well as creation. [22:28:24] If you’re still over quota I can just bump it up a bit. [22:29:10] ragesoss: you should be able to create a small instance now — is that enough [22:29:27] The deleted instances should clean themselves up when I upgrade the other compute nodes [22:30:01] andrewbogott: okay, no problem. I was going to createa four-core instance, so I'll just wait until they finish deleting. [22:30:14] thanks! [22:35:14] 6Labs, 10Beta-Cluster, 10Labs-Infrastructure, 6operations: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#1628650 (10Dzahn) one more reason we should have this is to avoid needing https://gerrit.wikimedia.org/r/237523 (T105794, T112195) [22:48:11] 6Labs, 10Labs-Infrastructure, 3Labs-sprint-112, 5Patch-For-Review, 3labs-sprint-113: Update remaining virt nodes to kilo - https://phabricator.wikimedia.org/T112200#1628743 (10Andrew) 3NEW a:3Andrew [22:48:33] 6Labs, 10Labs-Infrastructure, 3Labs-sprint-112, 5Patch-For-Review, 3labs-sprint-113: Update Horizon/Californium to Kilo - https://phabricator.wikimedia.org/T112201#1628755 (10Andrew) 3NEW a:3Andrew [23:59:37] 6Labs, 10Beta-Cluster, 10Labs-Infrastructure, 6operations: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#527164 (10Dzahn)