[10:33:06] hey, I want to do a bit of cleanup for puppet catalog compiler (T279642), and there's a few issues to address, there's some nodes that don't exist anymore (ex. cloudceph2001-dev.wikimedia.org), anyone knows what's the process there? [10:33:07] T279642: [ceph] make puppet catalog compiler pass for all ceph hosts - https://phabricator.wikimedia.org/T279642 [10:34:57] dcaro: have you tried https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet-diffs#How_to_update_the_compiler's_facts?_(e.g._INFO:_Unable_to_find_facts_for_host_conf2001.codfw.wmnet,_skipping) ? [10:35:51] nope, it did not seem to be the same error, will try, found this also: https://wikitech.wikimedia.org/wiki/Puppet#node_cleanup [10:40:34] jbond42: that page is the most difficult one to find in the whole wikitech :) [10:40:45] dcaro: looking at the two nodes you mentioned the faliure is expected. for cloudcephosd2001-dev.codfw.wmnet we need to have the appropriate secrets in the puppet private repo. i dont think it makes senses to mock this as it could mask reall hidding values. for cloudceph2001-dev.wikimedia.org without a definition in site.pp pcc dosn;t no what catalouge to compile [10:41:26] *real missing values [10:41:32] XioNoX: lol i dont dissagree [10:42:52] jbond42: the thing is that cloudceph2001-dev.wikimedia.org does not exist [10:43:06] so pcc is pulling it from a stale entry on puppetdb (I think) [10:43:54] dcaro: ahh i see because you used a regex re:^cloud.*(wmnet|org) ok im with you now :) [10:44:16] im gussing it did exist at some point right? [10:44:23] I guess the same xd [10:45:32] maybe renamed cloudcephmon200X [10:45:41] yep, also the domain changed [10:45:57] wikimedia.org -> {eqiad,codfw}.wmnet [10:46:31] dcaro: leave it with me, im not sure of the best way to purge nodes from pcc, its likley a manually process on each compiler node but ill check [10:48:02] 👍 let me know if I can help [10:48:12] will do [10:55:06] for the missing secret, the secret 'bootsrap_keydata' does exist in the puppet secret repo (both for codfw and eqiad), so it should be getting it, unless we use a different 'fake' private repo (I remember something like that), in that case, what's the procedure to add a dummy entry there? [11:15:17] dcaro: https://gerrit.wikimedia.org/r/c/labs/private/+/677848 key is in the wrong location. this is due to the wmflib::expand_path hiera backend in production (let me know if you want me to expand on that) [12:55:49] jbond42: thanks. I'm curious :), is that documented anywhere I can read it? [12:57:57] dcaro: possibly ill take a look and can write something if not, failing anything elses there is the code for the hiera backend modules/wmflib/lib/puppet/functions/wmflib/expand_path.rb (although that is sparse on comments and assumes knowlage of hiera) [12:58:23] ack, thanks! I'll give it a look see if my hiera fu is enough [12:58:57] ack :) [13:00:15] beautiful :} https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/28960/console [13:05:40] dcaro: yes just responding on ticket :) [13:17:15] hiya apergos, yt? [13:17:23] yes [13:17:25] is there something more I have to do to get https://gerrit.wikimedia.org/r/c/operations/puppet/+/676922/ deployed on dumps.wm.org? [13:17:34] https://dumps.wikimedia.org/legal.html [13:17:43] uh [13:17:51] it has to be puppet-merged... has it not been? [13:17:58] it has [13:18:15] 2 days ago [13:19:06] yeah, I saw the +2 go by and figured that it was done [13:20:17] oh there are two legal.html files in the puppet repo [13:20:24] snapshot? [13:20:26] yeah I was just about to say [13:20:39] I wonder if the one for the labstore boxes is elsewhere [13:21:09] they are the same content? [13:21:17] puppet/modules/dumps/files/web/html/legal [13:21:24] apparently that's the one that gets used [13:21:57] hm not quitee [13:22:01] weird [13:22:04] that is, it gets used to the public [13:22:08] i wonder if the other shoul be removed? [13:22:20] the other one is used on the internal dumpsdata boxes, and it's hard to remove outright [13:22:46] it's a holdover from when dumps wrote directly to public-facing web servers, and we don't want to remove that capability I guess [13:23:02] let me see if we can rename it some nice way though, to avoid the mixup [13:23:10] or put a big fat README in the directory\ [13:23:30] can we just use the same file in both places? [13:23:40] should I revert the change I made to the snapshot one? [13:25:38] are we allowed to create draft patches on gerrit operations/puppet? (I'm getting a remote rejected, not sure if I should or not xd) [13:25:40] no need to revert, it's not very convenient to use the same file because these are two different modules [13:25:55] sounds like the module should be made generic? :) [13:26:06] and configured via hiera? :p [13:26:28] no, the modules are two very different things [13:26:31] https://gerrit.wikimedia.org/r/c/operations/puppet/+/677900 [13:26:32] one manages nfs servers [13:26:39] the other manages servers that produce dumps [13:27:13] ok ¯\_(ツ)_/¯ [13:27:19] this is the quick fix for now, can you make a task about the two legal.html's and put my name on it? I need to think about the best way to fix this for real [13:27:35] ok [13:28:45] so the module for snapshots for excample sets up a copy of the dumps repo, the c utils repo, mediawiki and all that. the nfs server one does none of those things, nor should it. the templates for the snapshot one go along with the dumps repo; thenfs/web servers just set up independent html files for any directory people want and serve data to the public... [13:28:52] anyways that's for me to fuss over [13:29:14] https://phabricator.wikimedia.org/T279661 [13:30:44] I've changed the tags, since I'm not on sre team now :-) thanks! [13:31:51] \ [13:32:08] sorry, that was an artifact of me cleaning the keyboard. [13:33:58] dcaro: i have added the description i put in the task to you to wikitech https://wikitech.wikimedia.org/wiki/Help:Puppet-compiler#Purging_nodes are you able to review and make sure it makes sense [13:34:15] XioNoX: https://wikitech.wikimedia.org/wiki/Help:Puppet-compiler#Updating_nodes hopefully a bit easier now [13:36:29] jbond42: thanks a lot! [13:36:45] np [13:44:48] looks good :), added some minor formatting (mostly for myself xd) [13:46:43] thanks [13:56:46] nice! [13:57:44] three cheers for sre.hosts.decommission! [14:31:22] are there any coverage metrics for puppet code? [14:31:34] dcaro: no [14:32:20] xd, okok, I'm trying to see how many modules will need some tests and such (would be great to have at least a 'compiles' test on each) [14:37:33] dcaro: i agree however as are modules ae all in the same repo and we dont use metadata.json, dependencies are a bit of a pain. this should now be a bit easier with the shared spec_helper ../../../../rake_modules/spec_helper, however there are other issues specificly some modules mock with mocha and other with rspec-mock (on my list of low prior things to fix). [14:38:57] ack, thanks [14:40:34] all that said i think this is mainy a problem for current modules i think fopr new modules it should be easy to add the following to anything wich dosn;t have a test https://phabricator.wikimedia.org/P15266 [14:41:00] s/current modules/ modules that currently have spec test/ [14:47:09] I do :) [14:47:24] it's helpful, specially for typos and such [14:48:17] cool [14:48:44] I need help debugging a pcc output, for some reason https://gerrit.wikimedia.org/r/c/operations/puppet/+/677911 adds a parameter to ceph::common (ceph_repository_component), but in the pcc output, the Ceph::Common class does not have that parameter: https://puppet-compiler.wmflabs.org/compiler1002/28962/cloudvirt1021.eqiad.wmnet/change.cloudvirt1021.eqiad.wmnet.pson [14:49:20] it's a stacked change, so maybe the job does not support those? [14:52:15] wait... it's an older patch xd, the patch #3 is not there... nor in zuul :/ [14:53:13] yep, that was the wrong patch, nm [14:59:09] dcaro: i have updated the puppet_hiera wiki in specifically added https://wikitech.wikimedia.org/wiki/Puppet_Hiera#wmflib::expand_path please give it a review and let me know if it make sense [15:00:43] ack [15:02:37] looks good :) [15:04:22] thanks [16:41:52] Question for moritzm and others: on a scale of "no freakin' way" to "no problem at all", how do we feel about armhf binaries in apt.wikimedia.org? [16:42:34] Performance team wants to replace our manually-configured Mac Mini which drives our phone testing lab with one or more Raspberry Pis whose configuration can be automated. [16:44:31] Most everything is architecture-independent, but there's a couple of tools which aren't in stock Debian, and would need to be cross-compiled. [17:01:12] dpifke: yay to switching from Macs to RPIs! are you planning on using vanilla Debian or raspbian? IIRC raspian's armhf is different than Debian's [17:16:05] there's some divergence, indeed [17:16:21] raspberry pi os, as it's called now, has more frequent firmware and kernel updates and such from the rpi foundation [17:19:49] Strong preference for stock Buster or Bullseye, but haven't figured out if I need anything from the RPi distro. [17:22:00] Right now I have a hacky script which merges a Docker container into an SD card image. I've been using Raspbian, but have to fight with some of the networking stuff. [17:22:26] (Purpose of using Docker is to be able to build and publish the SD card image using Blubber.) [17:22:48] (i.e. as part of a CI pipeline) [17:24:57] raspberry pis? oooohhh. where would they live? [17:26:36] SF office. The tentative plan is to set up a Pi + phone for each model we test, which just needs network + power. [17:26:40] https://usercontent.irccloud-cdn.com/file/MRiUb2C8/Render2.JPG https://usercontent.irccloud-cdn.com/file/qwv5MxvD/shelf.JPG [17:27:07] This is to replace the current "tangle of wires" approach. :) [17:27:15] legoktm: can we nudge https://gerrit.wikimedia.org/r/c/operations/puppet/+/675357 forwards? [17:27:37] oh yeah, let's do that now [17:27:46] people have had enough time to object [17:32:27] slick! [17:32:29] dpifke: reprepro should support it just fine, if it's just for some systems outside of prod, that should be fine. if the arm binaries should also be built/cross-compiled in prod, then it becomes more involved [17:32:53] can you open a task and tag it SRE? this can best be sorted out in Phab I guess [17:33:19] Awesome. Yeah, it'll be on the office network but not directly connected to anything prod. [17:33:33] Will do. [17:39:30] Majavah: done, I'm letting it rollout gradually across the fleet [17:39:43] did it manually on mwdebug and it worked correctly [17:39:46] Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-ubuntu]/ensure: removed [17:39:46] Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[ttf-ubuntu-font-family]/ensure: removed [19:43:23] quality use of raspis. [21:21:40] yahh it's conceptually neat. [22:27:01] 👋 anybody able to help me debug a low-priority puppet issue? I recently merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/678044/4/modules/profile/manifests/superset.pp#129 but now I'm seeing puppet error `Could not retrieve information from environment production source(s) puppet:///modules/superset/check_superset_http.sh` [22:27:27] I see other checks using this puppet:/// resource pattern so I'm not sure what's going wrong [22:46:06] Ok I think I see the issue! puppet:///modules/puppetmaster/filter_job_id.lua for example references file modules/puppetmaster/files/filter_job_id.lua /shrug [22:46:25] razzi: it is not intuitive but the correct place for that file -- and what that puppet:/// path resolves to -- is under modules/superset/files/ [22:46:37] jinx cdanis :) [22:46:42] aha yes :) sorry [23:09:52] hmmm.. should we have a linter rule for puppet patches that whines about things like that (files in the root of a module that are not README or LICENSE)? [23:59:28] bd808: I'm in favor [23:59:50] bd808: if you can point me to where linter rules like that would live, I'd be happy to write