[08:34:35] FYI, cumin2001 reimage starting now [08:56:18] I'm about to enable thanos upload for prometheus ops in codfw/eqsin/ulsfo, no impact expected beside the prometheus restarts [10:36:50] argh, empty commit in the puppet private repo doesn't abort the commit [10:39:21] hnowlan: :cq ;) [10:39:37] that makes vim exit with non-zero and aborts the commit [10:41:49] ohhh I see the commit-msg hooks prepending the name makes it a commit either way. Not ideal [10:49:23] hnowlan: patches are always welcome :) we could check that there is actually something in the commit [10:50:08] yeah, looking at doing one! tbh I think we'd avoid my failure case by just checking for ^# or ^$ seeing as the hook is actually seeing the comments [10:50:26] but I wonder if there's a ordering way to make git itself see the comment before we prepend [11:01:01] vgutierrez: do you think we can merge the acme chief changes today? are you waiting Krenair +1 ? [11:20:33] I think so [11:25:24] I'll take care of that during the afternoon [11:30:25] ok, thanks! [12:06:50] does anyone have a way to test puppet changes on a laptop? [12:07:31] pcc is useful, but it requires commit + push to gerrit + run utils/pcc looking at the stdout of the puppet run, which is rather heavy [12:07:57] currently i'm trying to debug a hiera lookup that's failing [12:17:42] kormat: ./utils/hiera_lookup [12:18:26] and there is a run_ci_locally script as well. It does make a fair number of assumptions (like have docker locally) but if you can satisfy them, it works [12:18:42] but it won't help in this specific case I fear [12:19:23] `utils/hiera_lookup` just prints out `puppet now has this functionality built in. try the following on the puppet master` for me [12:20:38] still works though [12:21:36] akosiaris: not for me [12:22:55] ahh. i had misspelt the key [12:23:53] mm. it's pass/fail. if it doesn't work it doesn't give you any idea why [12:30:43] sometimes known as the hiera tar-pit 'where everything is possible but nothing of interest is easy' *g* [12:31:17] godog: yaaaay :) [12:33:16] kormat: seriously though, which key/lookup is not working ? [12:33:53] godog: https://gerrit.wikimedia.org/r/c/operations/puppet/+/605188/7/hieradata/common/profile/mariadb/multiinstance.yaml [12:35:05] that's what i added, but looking it up so far has failed [12:37:41] kormat: ack, looking it up from which part of the code ? [12:37:49] https://gerrit.wikimedia.org/r/c/operations/puppet/+/605188/7/modules/mariadb/functions/multiinstance_mariadb_port.pp [12:38:15] (this is my very first puppet code, be gentle ;) [12:40:11] haha! ok I'm not sure if looking up data within structures is a thing [12:40:36] puppet experts would know though [12:40:50] i believe i tried just looking up the structure itself (`rofile::mariadb::multiinstance`) and it didn't succeed [12:41:16] if i use run_ci_locally, does that mean i can get pcc to run locally? [12:42:38] run_ci_locally doesn't call pcc afaik no, getting pcc to run locally is possible I think but never done it [12:43:18] https://taylormertins.files.wordpress.com/2016/11/sigh.jpg [12:45:23] heheh indeed [12:46:12] I'll plug here what I've been experimenting with to shorten the local development -> try with a puppet master similar to production but beware that it comes without guarantees [12:46:16] https://wikitech.wikimedia.org/wiki/User:Filippo_Giunchedi/Pontoon [12:46:52] godog: at the moment i don't even care about making changes to any host. i just want to get puppet to tell me what it thinks [12:47:14] the equivelent of adding a "print("DEBUG: %s" % thing)" in the middle of a python script [12:47:53] I'm sure there's a way, but I'll yield (hah) to puppet experts [12:52:54] godog: who are these mysterious puppet experts btw? ;) [12:56:04] haha! not going to rat anyone out kormat [12:57:59] in seriousness though, the puppet equivalent of the print above would be notify or notice depending if on the master or the agent [12:58:24] godog: aye. but that seems to imply having the whole env set up [12:58:36] it sounds like i do need to follow something like your Pontoon approach [12:58:53] kormat: the name [12:59:01] profile::mariadb::multiinstance:__something__ [12:59:23] based solely on the name of the file [12:59:27] but let me see where you use it [12:59:49] oh god, right [13:00:17] I don't see a profile::mariadb::multiinstance class [13:02:08] and I think our styleguide doesn't really allow hiera lookups in functions, but I'm not 100% sure [13:03:09] kormat: the existing profile is profile::mariadb::core::multiinstance [13:03:30] right now my biggest priority is being able to test changes. when i can do that, i'm happy to rework anything/everything to fit the style guide [13:03:36] volans: this would be used by multiple profiles [13:03:59] then must be global [13:04:06] where would that live? [13:04:21] hieradata/common.yaml most likely [13:04:31] and yeah, it's too bad [13:04:39] if we can avoid it better [13:04:56] having some sort of profile::mariadb::multiinstance::common [13:05:06] that is included by other profiles [13:05:13] that are the specific ones with different behaviour [13:05:19] but depends how you need to use the data [13:06:36] don't worry, i'm quickly losing the will to continue ;) [13:08:16] going back to the standalone puppet master - should i be creating a new project in order to request a cloud vps? [13:08:45] I'm not one of those experts in puppet, but if you want we can chat a bit about what you want to do and what's the best way to do it [13:08:59] for the standalone master you can either do that or use an existing one [13:09:34] (either project to create a new vm or existing standalone master to play with) [13:10:17] I would happily land you mine, but it's too old and pending rebuilt with newer OS and puppet version [13:11:48] kormat: those are the existing ones in WMCS: https://openstack-browser.toolforge.org/puppetclass/role::puppetmaster::standalone [13:12:50] ok. sounds like a separate project is the way to go [13:19:12] kormat: re: Pontoon I realize it is quite the heavy handed approach for locally tested changes, maybe there's a middle ground there like a docker image for a puppet master to run locally that would run your local code (spitballing ideas) [13:19:40] kormat: since G.i.u.s.e.p.p.e. is on vacation there's about 50/50 odds that he implements pcc-on-Docker in the next two weeks [13:23:02] godog: there is https://hub.docker.com/r/puppet/puppetserver/, but i've no clue what i'd do with it [13:23:06] cdanis: hah ;) [14:09:44] kormat: it is possible to run pcc localy but its a bit of a pain i genraly only use it if im looking at the pcc code, run_ci_locally dose not run pcc (not sure if that got clarified). in relation to the style guide im not sure if its mentioned but there is definetly precedence added recently (see module/wmflib/functions) [14:13:30] in relation to genral debuggiug for simple case i genrally create a local manifest and run `puppet apply manifest.pp`. for more complext cases i rely a lot on spec test. you can also add `it { pp catalogue.resources }` to a spec test to dump the puppet catalouge which can be usefull [14:14:00] hiera though is a pain to debug [14:14:31] did you resolve your specific issue if not ping me and i can take a look [14:15:57] as to looking up subkeys using the dot notation i have a feeling that might require hiera 5 [14:18:01] puppetlabs hiera 3.2 documentations is now redirecting to https://github.com/puppetlabs/docs-archive/tree/master/hiera/3.2 which is a 404 :@ [14:23:54] the 3.1 docs mention support for subkeys so suspect it should work [14:32:20] however neither `puppet lookup` or `utils/hiera_lookup ` support it [14:45:53] jbond42: the specific issue is ~puppet~ :) [14:46:16] but volans has kindly lent me some cloud VPS resources, so i'll set up a standalone puppet master and see if that helps [14:48:27] lol cant help with that, i did have a look at the change and couldn't think of a better way to test then using a standalone [14:51:29] jbond42: yes, I explained the limitation of that too, but it might help him in general for puppet development [14:51:51] as for the problem at hand I suggested either to use hiera aliases or a common profile with the common stuff [14:51:59] but lmk if you disagree :) [14:52:14] > there is definetly precedence added recently (see module/wmflib/functions) [14:52:14] also do you agree we should avoid lookup() calls in functions? [14:52:24] jbond42: i didn't understand that bit [14:52:38] that is actully the bit volans is talking about [14:52:52] there is precedence for calling lookup function from functions [14:53:06] ahh [14:53:14] vindicated \o/ ;) [14:54:55] i dont know the context of where all the data would be used however the wmflib::service::fetch functions seems to be simlar to the function you are trying to create [14:55:50] true, forgot about that bit. But I would consider that mor of an exception as it's doing a pretty complex thing, but I might be convinced otherwise [14:55:51] kormat: will the data be used by anything other then the functions in mariadb [14:56:23] jbond42: there's talk about using it to generate a config file on disk for scripts to read, [14:56:31] so that we don't need to have the mapping defined in multiple places [14:56:56] so everything that needs the data would in an ideal world get to the data with theses functions? [14:57:40] everything _in puppet_, yes, i think so [14:58:16] in that case i would store the data in hieradat/common/mariadb and have the key named mariadb::multiinstance [14:58:24] volans: would you concur with that? [14:59:00] that sounds good to me [14:59:49] jbond42: hieradata/common/profile/mariadb ? [15:00:01] It's not clear to me then how to keep the namespace coherence when calling it [15:00:36] or if you're implying to make a small common profile [15:00:42] my reason is that it feels like private module data which shouldn't be overriden. if we had pupet version5 then it would be stoered in the module its self i.e. modules/mariadb/hieradata/ but we are not there yet and no idea how others would like that [15:01:30] yes agree that's "private" [15:03:19] jbond42: according to our styleguide (that ofc we can amend), would be ok to call lookup( mariadb::multiinstance) in say profile::mariadb::core::multiinstance ? [15:03:45] I was assuming that would not be much ok, but might be wrong :) [15:03:50] (that's the great thing about keeping standards in a wiki. anyone can 'fix' them ;) [15:04:58] ill start with i need to re-read the style guide. however i would say calling lookup( mariadb::multiinstance) from profile::mariadb::core::multiinstance is a bad thing. I also think calling lookup("profile::mariadb::multiinstance.${section}.mariadb", Integer, ) [15:05:02] [15:05:04] [15:05:13] from modules/mariadb/functions/multiinstance_mariadb_port.pp is wrong [15:06:22] my gut feeling is this specific example is not covered by the style guide but to me having modules/mariadb/functions/multiinstance_mariadb_port.pp call lookup( mariadb::multiinstance) and having anything elses that needs the data calling lookup("profile::mariadb::multiinstance.${section}.mariadb", Integer, ) feels like the nicer approach [15:06:59] this is simlar to how wmflib::service::fetch is implmented which is probably the only thing similar [15:08:21] sorry having everything elses that needs the data calling ` mariadb::multiinstance_mariadb_port()` [15:08:26] jbond42: should the function live in modules/mariadb, or in modules/profile/functions? [15:08:35] given that it's used at a profile-level [15:09:03] +1 that the style doesn't cover this case [15:12:03] kormat: i would say if the functions are generic enough to be used outside of WMF then its fine to have them in module if they are specific to us then putting them in profile is probably more correct [15:12:11] they're specific to us [15:12:57] then profile is probably a better place that also makes the descussion about where the data lives easier :) [15:14:53] on a side-note, i'm loving how my "nice easy starter projects" involve this amount of discussion from the people who understand the area. **shakes fist at marostegui** [15:15:15] kormat: yeah we're bad at those [15:15:17] kormat: your famous last words were: "I can easily set an alert on this with prometheus" [15:15:28] marostegui: damnit [15:15:35] lol [15:16:01] new hires are the best to trigger those discussions because of lack of context and that's great [15:16:18] because after few months to have made the context yours and you don't challenge it anymore [15:16:40] lol [15:18:30] I remember the meeting where we discussed that new alert as it bite us during a weekend or something I believe and kormat was like: yeah, that should be easy with prometheus... [15:18:43] 👀 [15:18:59] I had this discussion of testing changes locally with different people as well. I came to the conclusion that I should just submit the code to gerrit, run the pcc and then merge or abandon as required [15:22:05] sukhe: i think if i already knew puppet when i arrived, i could probably live with that. but i'm still learning basic things about syntax and semantics etc, where having a fast eval cycle is Really helpful [15:24:10] jbond42: "makes the descussion about where the data lives easier" -> what does this imply? [15:24:55] kormat: same here! I was (still am) new to puppet and was hesitant to push changes :P [15:24:58] the data would live in hierada/common/profile/mariadb/multiinstance and prefixed profile::mariadb::multiinstance [15:25:08] ok great, thanks :) [15:25:16] which is closer to the standard [15:28:35] kormat: for simple things like syntax i often create a local manifest and test with puppet apply. e.g. i created the following to see hwo imutable puppet variables are https://phabricator.wikimedia.org/P11500 and ran with `puppet apply ~/puppet_tests/mutable.pp` [15:29:12] its not great but its something [15:31:15] jbond42: i was trying something like that earlier, but hiera was completely non-functional [15:32:10] kormat: you make me want to scrape the gerrit API to find out what's the patch I have with the most patchsets [15:32:24] kormat: yes hiera is trickey to get working like that [15:32:33] i had won with 42 the other day [15:40:43] kormat: for hiera you need to have modules/wmflib in your module path. i set my codedir=/home/jbond/git/puppet in /etc/puppet/puppet.conf and have updated my hiera file like this https://phabricator.wikimedia.org/P11505 which i think works although is a bit hacky. ill try polish this config and something to a wiki artical later in the week [15:43:37] jbond42: i had been flailing around with that sort of approach, but couldn't get it working properly, so that would be great [15:44:16] kormat: ack ping on friday if i havn't sent something your way before hand :) [16:11:20] kormat: fyi my local testing suggest subkeys lookups with our hiera config is not supported this could be due to either nuyaml or the hiera version [16:33:07] jbond42: that's fine, it was just the first way i found that allowed me to access the keys. i guess i'll look up the hash index syntax :) [16:45:09] kormat: not sure what you mean by that, however my gut feeling is that anything you will need to pull in profile::mariadb::multiinstance and extract in puppet code. that said i do have a CR out to upgrade hiera to version 5. which should support subkeys i will let you know when that is pushed [17:02:57] jbond42: ah, what i meant was if i do `lookup(profile::mariadb::multiinstance)` i presumably get back some kind of map that i would need to index into to get the port for a given section, etc [17:04:38] ahh yes. you would do `$foo = lookup(profile::mariadb::multiinstance); $foo[$section]['mariadb'] [17:04:56] ok. that looks normal enough :) [17:04:58] that's a hash (dict if you want in python terms) [17:06:52] I see you too worried about puppet coding [17:06:55] lets meet tomorrow morning for planning? [17:07:25] and seeing examples of stuff already existing?