[00:05:48] Majavah: thanks again for your work on the deployment-prep migrations. I hope you are having some fun along the way. :) [06:58:31] _joe_: so to be clear, the line in here https://wikitech.wikimedia.org/wiki/Conftool#The_tools that says you need either creds in your .etcdrc or to use sudo -i, that's no longer true, is that right? [06:58:37] I guess that's from https://gerrit.wikimedia.org/r/c/operations/software/conftool/+/534803 ? [07:06:25] well because I want to reclaim that tab, I have gone ahead and been bold and saved the edit, feel free to revert if it's wrong [07:07:11] <_joe_> apergos: correct [07:07:16] ty! [07:07:41] (I had well over ten tabs open in my browser so trying to clean them out today) [11:58:12] FYI I'll be looking loosely at IRC in the next days, ping me on VO if there is something urgent that needs my attention, or email/phab for non-urgent things [15:49:39] is there some convenient way of seeing the values of variables in puppet from a PCC run? [15:52:46] I've done various horrid things involving downloading the change catalog and running it through jq [15:53:30] i'm a simple woman. i'll settle for some way to 'debug printf' a variable [15:55:39] kormat: just open the catalog from PCC and look for your class and the parameeters are there [15:55:58] volans|away: these aren't parameters [15:56:10] params would show up in the normal pcc diff [15:57:55] for variables within your .pp files yes you have to log/use them in some template that is visible [15:58:12] log? [15:59:08] log/print/whatever [15:59:10] ;) [15:59:19] kormat: I haven't done it myself but https://www.puppetcookbook.com/posts/simple-debug-messages.html [15:59:19] volans|away: don't 'whatever'. what are these things? where do i find them? [15:59:27] you can also `assert` things that you want to be false [15:59:41] cdanis: will a notice() show up in pcc? [16:00:04] don't know but I think it will [16:00:12] in the "change errors/warnings" output [16:00:22] ok, that'd work. i'll give it a shot [16:04:12] nope 🤬 [16:04:36] ahh, wait. it shows up as a resource [16:28:19] <_joe_> kormat: notice() does show up in the output yes [16:29:58] kormat: there is a some what undocumented function one can use from one of the compiler100[123] hosts which can be usefull if you really need to dig into something (i mostly use it when making changes to pcc it self) `python -m puppet_compiler.debug_host` (https://phabricator.wikimedia.org/P14672) [16:36:13] _joe_: looks like pcc tries to filter out anything but errors (https://github.com/wikimedia/operations-software-puppet-compiler/blob/master/puppet_compiler/puppet.py#L69-L70) that said i often see warnings (wonder if they go to stderr) https://puppet-compiler.wmflabs.org/compiler1001/28446/pki2001.codfw.wmnet/change.pki2001.codfw.wmnet.err [16:56:57] jbond42: looks interesting re: debug_host, but i have no idea what it does :) [16:58:19] kormat: it esentially runs `puppet agent -t -d` against a specific change in a similar way to what how pcc runs puppet. sending the output to stdout [16:58:45] jbond42: mm, k. so i'd still need to use notice/notify to see the variable values? [17:04:01] kormat: yes probably [20:53:51] legoktm: curious if you have some obvious/easy things I could confirm/rule-out as probable cause for https://grafana.wikimedia.org/d/000000066/resourceloader?viewPanel=45&orgId=1&from=now-90d&to=now [20:54:07] Does Feb 1-10 align with buster roll out? [20:56:09] you can see the progress of reimaging https://phabricator.wikimedia.org/T245757, but yeah, that seems about right. [20:57:11] I guess you can't (or we didn't) fragment that metric by OS? [20:59:19] Krinkle: https://w.wiki/34qu [20:59:34] legoktm: it takes some doing :) [20:59:37] https://phabricator.wikimedia.org/T273312#6788233 [20:59:58] https://docs.google.com/presentation/d/1ONjYaTindhWB6X_Y00v_wGx9wg-S4vP-hK0pP-Buv0E/edit [21:06:51] legoktm: ack, so about a +20ms increase pretty much for all requests/precentiles on MW code [21:07:07] that seems to match load.php in-PHP backend timing as well [21:08:31] thx :) [21:08:52] cdanis: that node_uname_info graphs is very clever, nice one [21:14:00] Krinkle: there's a new metric now, not available previously as it's something I added rather than something built into node_exporter [21:14:45] Krinkle: https://w.wiki/34qy [21:15:23] ah, looks like I added that on the first