[13:04:16] does anybody know if the makevm cookbook to create a ganeti VM is ready for use? [13:04:28] from https://phabricator.wikimedia.org/T203963 it is not clear [13:04:36] the cookbook seems to be in spiecerack [13:05:14] err sorry in the cookbook repo [13:05:57] elukey: hey, give me a sec [13:06:59] sure sure I am reading https://gerrit.wikimedia.org/r/#/c/operations/cookbooks/+/496527/17/cookbooks/sre/ganeti/makevm.py [13:07:49] so IIRC all spicerack features are there and reployed, I'm checking if by any chance ther eis anything that we found that is not yet deployed [13:08:46] no, nothing, it should be good to go, but I'm not sure how much live testing was done (Cas knows better), so you might be a guinea pig ;) [13:09:34] I think that currently I cannot use it, since I need link=analytics [13:09:43] that I added to the makevm script [13:55:40] arturo: puppet is screaming in the traffic project at labs cause the client instances are not able to find /usr/local/bin/puppet-enc [13:55:53] arturo: any recent change that could explain this new behavior? [14:08:09] no changes on my side [14:08:29] vgutierrez: please ask in the cloud channel [16:29:06] this project to configure iptables & tc to simulate network losses has the most awesome name ever -- https://github.com/tylertreat/comcast [16:30:24] that is incredible [18:02:33] chaomodus: o/ - do you have a min? [18:05:19] (just found the answer :) [18:17:34] aha sorry i missed your message until just now [20:22:22] does anyone have handy the google doc link to the SRE summit notes? Or a suggestion as to how I might find it? [20:24:19] it comes right up for me if I go to docs.google.com on my WMF profile and search for [sre summit notes], but also, sent you a PM [20:25:18] this is a silly question to ask about google, but… how specifically are you doing that search? [20:25:35] * andrewbogott sees no magnifying glass [20:26:03] https://phabricator.wikimedia.org/F29674311 [20:28:08] Ah! It's visible if I don't already have a doc open [20:28:13] that will be very very useful [20:28:17] oh, ah, yeah sorry :) [20:28:27] both docs.google.com and drive.google.com have (slightly different) search interfaces on the front page [20:32:21] Is there a budget/req/whatever process for ganeti or can I just log on there and make myself some servers? (<- possibly a question for robh or akosiaris or wiki_willy ) [20:34:12] how many is "some"? [20:34:57] uhh... did we break the grafana dashboard for it recently? :\ [20:35:05] last I looked at this was months ago, but there used to be values there [20:35:49] drive vs docs is annoying :) [20:36:24] cdanis: three tiny little ones, for haproxies [20:36:35] awesome bar for the win. I find gdocs by just typing what I think I remember the name being and seeing what Firefox tells me [20:37:19] that's likely fine. I don't think there's any capacity planning process right now (there should be!) [20:37:22] bd808++ [20:37:38] andrewbogott: it seems like there was some light weight process you did when adding the ro-ldap nodes, but I don't remember what it was [20:38:01] I asked alex and he said 'yeah, that sounds fine' [20:38:04] I thought I remembered some advice being on https://wikitech.wikimedia.org/wiki/Ganeti but I'm not finding it now [20:38:13] (in terms of picking row and such) [20:39:12] do I remember correctly that there are only two ganeti clusters in eqiad atm? [20:39:20] or is there one in each row? [20:40:13] it's the same cluster, AIUI, but there are 'groups' of nodes within each cluster [20:40:28] I think so services that want to do HA for row failures can run a replica in each row? [20:40:47] cdanis: specifically I'm wondering if my vms can be spread over three different physical servers or only two [20:40:58] (for a three-system HA quorum) [20:41:22] that's beyond my knowledge [20:41:47] 'k [20:43:11] if you're worried about HA at the service level, though, you probably want a ganeti VM in each row available, and some strategy to do master election amongst them [20:44:14] yeah, exactly [20:44:19] docs say that I can select row a, b or c [20:44:21] so that's good [21:54:00] akosiaris: uhh [21:54:04] sorry, i meant to say [21:54:11] uhh i thought akosiaris still reviewed them [21:54:18] via the vm-requests project, but i could be wrong! [21:54:32] (it used to be this way for a long while but may have changed in last 12 months) [21:54:53] andrewbogott: so i'd file a request with that tag and ask alex [21:55:10] however you can specify that a the vms run on different servers iirc [21:55:14] robh: ah, vm-requests! Thanks [21:55:18] iirc we do that with the dns stuff we have in ganeti [21:55:53] but once he is aware, you can actually use the howto to create them [21:56:11] iirc its more of a alex needs to know how quickly we're using up our allocated ganeti nodes kind of thing [21:56:46] since ganeti nodes are really only requested by SRE folks and they know they have to justify not using cloudservices, they tend to be easy to approve from past lurking ;D [21:56:52] sorry, ganeti vms, not nodes. [21:57:18] andrewbogott: but if you asked alex and he said it sounds fine, seems like you dont need to ask him on the task directly =] [21:57:34] i'd be paranoid and file a task assigned to yourself documenting that it happened for ease of future reference, but im paranoid. [21:57:44] yeah, I'll make a ticket [21:58:25] also note the "makevm" script literally just moved from bash to spicerack [21:58:32] once you get to the part of creating them [21:59:09] did that cookbook get fixed today? [21:59:10] heh, ticket https://youtu.be/s0vNsH81YeA?t=94 [21:59:27] cdanis: i think there was a fix but a new bug after that [22:06:07] cdanis: yep, thanks to elu.key that played the guinea pig and fixed it while at it [22:07:47] * volans off [22:25:10] mutante: any chance of an update to https://wikitech.wikimedia.org/wiki/Ganeti#Create_a_VM before Jason starts making VMs? (Or should I make a task about that for someone else who knows what's what?) [22:26:40] andrewbogott: i assume the old script is not deleted yet. so i would say that can be used as as for now until the last couple bugs are solved in the new method [22:26:46] as is [22:26:49] ok :)