[07:27:24] reminder that cumin1001 will be reimaged later [07:27:52] marostegui, kormat, elukey: ^ as you're all currently logged in [07:28:03] moritzm: just left! [07:28:30] ack :-) [07:28:36] moritzm: unlogged [07:31:35] ack :-) [07:39:01] sorry I left a tmux opened! closed now [07:40:26] ack :-) [08:15:44] Could someone with a bit puppet/hieradata/cloudvps knowledge point me into the right direction of why lookup() is unable to find my heiradata "common" values for a profile? [08:16:19] I think the answer is hidden here https://wikitech.wikimedia.org/wiki/Help:Standalone_puppetmaster#Help!_Hiera_values_from_local_commits_don't_seem_to_apply! - but I don't get it :) [08:17:18] jayme: has godog told you about the glories of pontoon? :) [08:17:47] jayme: do you have a specific commit? [08:18:04] are you aware of the different hiera hierarchy between prod and wmcs? [08:18:48] I think I'm not (and that's why I don't get that paragraph :)) [08:19:19] 1 sec for the commit, it's not in gerrit currently [08:19:34] or just the value you're looking for [08:19:42] no need to push if you donw't want to [08:20:19] jayme: so this is the production hierarchy used: [08:20:19] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/puppetmaster/files/production.hiera.yaml [08:20:38] while this one is the one used in wmcs: [08:20:39] I don't mind, just did not had to. It's https://gerrit.wikimedia.org/r/c/operations/puppet/+/606956 [08:20:39] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/puppetmaster/files/labs.hiera.yaml [08:21:30] the first difference is that in wmcs you have also the values you set on horizon's UI for you instance/prefix/project [08:21:40] What I'm looking for are the values from "hieradata/common/profile/chartmuseum.yaml" [08:26:25] and what lookup are you doing? [08:26:55] actually, if you give me the FQDN of your puppemaster I can login and have a quicker look [08:27:18] sure jayme-standalone-puppetmaster.sre-sandbox.eqiad.wmflabs [08:27:48] the lookups that are failing are the ones in the profile: modules/profile/manifests/chartmuseum.pp [08:28:13] wow, you like to live dangerously, a standalone puppetmaster on the sre-sandbox project :) [08:29:07] jayme: do you have local modifications? I can't see them in /var/lib/git/operations/puppet/ [08:29:53] I thought was some new profile [08:30:15] I thought this is what the sre-sanbox project is for ... [08:30:29] nevermind, found your commit [08:30:36] yes, it is for any testing VMs [08:30:42] but htey live only 14 days [08:30:49] branch should be checked out in /var/lib/git/operations/puppet/ [08:31:02] while usually setting up a standalone puppetmaster is for something more long-lived given the hassle to set it up [08:31:34] ah, yea...I probably was naive thinking that two weeks is waaaay anough time :-) [08:31:39] ahahah [08:32:56] I't my first "puppet thing", so I wanted to see how all this works anyways [08:33:08] :) [08:33:18] it's easy, it doesn't ;0 [08:33:20] ;) [08:37:16] So I had set "role::chartmuseum" for jayme-chartmuseum.sre-sandbox.eqiad.wmflabs and ran puppet there and that faild (although I thought it should pick up the common/profile values) [08:38:40] yes, so, out of the top of my head I don't recall all the nitty gritty details and yes, is quite a mess, but the simplest thing is usually to set your values in horizon [08:39:02] in the puppet configuration tab of the jayme-chartmuseum instance [08:39:09] jayme: https://wikitech.wikimedia.org/wiki/User:Filippo_Giunchedi/Pontoon may be of interest to you [08:39:16] it's what i'm currently using to test puppet changes [08:40:12] kormat: oh..I thought that was some kind of weired joke I didn't get - sorry :D [08:40:28] it is an experiment but please LMK how/if it works for you :) [08:41:58] volans: yeah...thought about that as well but that did not seem like a valid test case (just adding values somewhere else, I mean) :) [08:42:18] it's a different setup for hiera [08:42:24] so it will not be the same of prod anyway [08:42:53] for that the puppet compiler will allow you to test them once you have a CR in gerrit [08:45:08] okay. Just wanted to make sure that I'm not missing something here. [08:45:44] jayme: your initial instinct that i'm making a weird joke is a pretty good one in general :) [08:46:53] jayme: I *think* if you put them in hieradata/cloud/eqiad1.yaml it **might** work too [08:47:01] but I usually go the horizon way to avoid the hassle [08:47:06] if is just for testing [08:47:27] and not something that will have to live/work in WMCS normally (like things that go into deployment-prep) [08:48:21] it's just for testing. And if I need to put them "somethere else" anyways, horizon is fine for me I guess :) [08:58:08] thanks volans (will take a look at pontoon as well kormat, godog :)) [08:58:55] jayme: yw, anytime, hope that worked [08:59:10] jayme: sounds good! [08:59:38] it did...failing on stuff I did wrong myself now ofc, but thats fine :) [08:59:50] great [10:21:48] ignore 5-minute average replication lag alerts on icinga [10:49:37] volans: hmm. are cumin aliases broken in cumin2001? `cumin db-all-eqiad` doesn't match any hosts, for example [10:49:55] kormat: checking [10:50:08] oh. is A: required? [10:50:18] yes [10:50:21] A:db-all-eqiad [10:50:24] it's the alias syntax [10:50:27] ahh. sorry, i forgot about that syntax [10:50:35] then i assume it's fine :) [10:50:39] fiuuu [11:00:57] I think I am going to send a patch with an "expanded version" for the uppercase letters [11:14:32] moritzm: moving here, [11:14:49] `puppet-merge` never prompted me for my private repo changes [11:15:13] oh, goddamnit. i see the problem [11:15:23] the puppet change and the private change depend on each other :/ [11:22:07] you could re-apply the hunk from https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/606984/ which defines the contact group, that should fix it? [11:37:59] Could I get access to the packaging project in horizon? [11:38:08] and/or should I ticket for that [12:32:29] moritzm: aye. i was in a bit too much of a hurry for that, as ~180 hosts were going to alert, and i was in a meeting, and lunch was soon :) in the end i reverted the private repo change too [12:32:44] i'll take a look at what went wrong and see if i can push a fixed version in a bit [12:37:23] sounds good! [12:37:59] and so continues my quest to discover all sharp corners by blindly running into them at speed. ;) [13:45:16] FYI, I need to disable puppet in codfw, I'd start in 5 mins unless that's an issue for anyone ATM [13:46:00] moritzm: if you can just leave it disabled permanently, even better ;) [13:46:39] hehe :-) [13:50:10] kormat: that wouldn't play well with our cp nodes :( [14:16:20] moritzm: i managed to change icinga config without breaking it (for once) \o/ [14:16:44] cool :-) [14:43:31] oh my. https://integration.wikimedia.org/ci/job/operations-puppet-tests-buster-docker/5556/console - having too long a Hosts: line breaks CI? [15:01:23] does... pcc not actually give you a diff? [15:03:03] it does, as long as the resource in question existed in both catalogs [15:03:19] otherwise it will just tell you that it was created or removed but not provide the contents [15:03:25] (but you can fetch it from the pson) [15:03:40] i've downloaded the 2 pson's, and diff on them is super-not-helpful [15:04:02] 12k line diff. almost all of which are some variation of: [15:04:10] - "file": "/srv/jenkins-workspace/puppet-compiler/23355/production/src/modules/profile/manifests/base/puppet.pp", [15:04:10] + "file": "/srv/jenkins-workspace/puppet-compiler/23355/change/src/modules/profile/manifests/base/puppet.pp", [15:04:35] yeah ofc [15:04:37] can you tell me what you're looking for? [15:04:46] sanity. i'm not finding it [15:04:52] kormat: you cane have multiple Hosts: lines in the commit msg [15:05:04] jbond42: ok, good to know, thanks :) [15:05:07] what you probably want to actually do is something like [15:05:07] you can also use a regex `Hosts: re:.* [15:05:13] curl -s https://puppet-compiler.wmflabs.org/compiler1003/22927/puppetmaster1002.eqiad.wmnet/change.puppetmaster1002.eqiad.wmnet.pson | jq '.resources[] | select(.title == "/usr/local/bin/puppet-merge") | .parameters.content' -r [15:05:24] which in that case extracted the 'compiled' script from the change pson [15:17:22] maybe i can get away with running sed on both files to make the paths the same [15:22:00] | grep -v /srv/jenkins-workspace/puppet-compiler/23355/ [15:22:01] ;) [15:22:42] `sed 's@/srv/jenkins-workspace/puppet-compiler/[^/]*/[^/]*@@'` and i finally have something readable [20:05:56] github: "An update to our nameservers has been rolled back. We are monitoring recovery." ya gotta feel for whoever's on duty [20:19:19] they probably have multiple DNS folks by now ;) [20:22:00] * RhinosF1 still can't get on [20:24:15] isn't it nice how we can still work and are not dependent on it [20:28:39] * RhinosF1 can't [20:38:42] Anyone around who knows things about bacula? I'm setting up a db backup job on each instance in a cluster but worried that that means they'll all run their mysqldump at exactly the same moment [20:53:34] andrewbogott: unless that's a special puppet class to do both, i don't think bacula is in the business of creating mysqldump files. that would be a separate cron or systemd timer and bacula will just look at the files it sees in the filesystem [20:53:57] so you can control the time they would run and do it like we start the puppet agent ..with splay [20:54:01] to randomize it [20:54:43] if you want to look at the time the actual bacula jobs run you can list them from "bconsole" on the backup server [21:10:10] mutante: that's a good point, I'm probably looking at the wrong bit :) [21:13:32] I'm trying to use backup::mysqlset, don't actually see any crons or timers in there so far [21:13:39] andrewbogott: looks like it _is_ a special defined type called "backup::mysqlset" [21:13:55] yep, and that generates a dump script... [21:14:03] but I can't figure out what calls it (unless bacula calls it) [21:14:07] and uses bacula::client::mysql_bpipe [21:14:14] ah, 'ClientRunBeforeJob' [21:14:31] I'm using predump 'cause that's what luca seemed to use when he last used this [21:15:48] yea, so backup/templates/mysql-predump.erb is a bash script [21:16:00] and then it is run whenever the bacula job starts [21:16:24] (one of the scrips that should not be .erb anymore) [21:17:41] I assume this is a side-issue, but… tell me more about 'should not be .erb anymore'? [21:18:59] yea, it's a side issue. we should not generate shell scripts from .erb templates per https://phabricator.wikimedia.org/T254480 https://phabricator.wikimedia.org/T200984 [21:19:22] ah, I see — I'll ignore that then :) [21:22:40] andrewbogott: here's what you can do: merge puppet change, wait a bit, ssh to backup1001, sudo bconsole, type "status", type "4" for scheduled jobs or "3" for client and pick one ... see the times it scheduled the jobs [21:23:22] that's reasonable, I guess I'm only applying this on a test cluster so far anyway [21:56:46] hm, now bacula won't start because of an error in a file that doesn't exist anymore [21:56:56] mutante: any idea how I can force it to clear its head? [21:57:38] oh, nm, maybe now it's failing for a real reason [21:59:05] I must be missing a piece, no matter what cadence I require it fails with 'Could not find config Resource "weekly-Sat-production" referenced on line 6 : JobDefs = "weekly-Sat-production"' [21:59:12] or the like [22:03:52] hm… possibly case mismatch [22:08:33] nope :( [22:18:42] andrewbogott: there is Hourly, Weekyly, Monthly.. just Daily isn't a thing [22:19:06] mutante: that's weird, right? [22:19:13] you probably have to create it using bacula::director::jobdefaults [22:19:21] Also, I see things like "hourly-saturday' which makes no sense to me [22:19:48] I'm going to try to get this working with Weekly before I break new ground with Daily :) [22:21:04] modules/backup/manifests/hourlyjobdefaults.pp: bacula::director::jobdefaults { "Hourly-${day}-${pool}": [22:21:07] modules/backup/manifests/weeklyjobdefaults.pp: bacula::director::jobdefaults { "Weekly-${day}-${pool}": [22:21:10] modules/backup/manifests/monthlyjobdefaults.pp: bacula::director::jobdefaults { "Monthly-1st-${day}-${pool}": [22:21:13] these 3 files generate the ones that are there [22:21:35] but no daily.. yea.. [22:22:46] mutante: what does hourly-${day} mean? Once an hour but only on Wednesdays? [22:23:25] Anyway, yes, Weekly (but not weekly) seems to be working [22:23:33] And it staggered the three hosts nicely [22:26:43] Anyway… I will try to implement Daily but not before dinner [22:27:08] andrewbogott: it seems something like "Hourly-Wed" means: Full backup every Wednesday, Incremental backup hourly [22:27:26] yikes [22:27:27] ok [22:27:28] thanks! [22:27:55] i looked at /etc/bacula/conf.d and there are various files starting with jobdefaults* and then those reference the files starting with schedule* [22:28:10] so schedule-Hourly-Wed.conf [22:28:24] 5 Run = Level=Full Wed at 02:05 [22:28:24] 6 Run = Level=Incremental hourly [22:31:12] the "Weekly" schedules only mention full backup once a week.. no incremental [22:32:02] the Hourly ones are like Weekly.. plus incremental every hour [22:33:39] Monthly is full backup on the 1st day that weekday happens in a month, differential 2 weeks later and incremental once daily.. afaict [22:33:59] Hm [22:34:10] I'm pretty sure I just want a plain old full daily backup [22:34:22] which should be simple enough to add now that I understand that it's not currently there :) [22:35:08] yea, i think go to modules/backup/manifests/ and copy one of the jobdefaults.pp and one of the schedule.pp [22:35:18] adjust and add j'aime as reviewer