[07:45:37] moritzm, topranks: I see you're reinstalling install7001, I wonder if we have a bit of chicken and egg there and you might need to decom it first (or just remove it from puppetdb) in order for the reimage to not try to use install7001 to reimage itself ;) [07:46:56] heh that’s an interesting thought [07:47:12] I'm currently decommissioning it already [07:47:17] and then I'll reinstall [07:47:21] +1 :D [07:47:36] there were installer issues with prometheus7001 yesterday, which looked atftpd/DHCP related [07:47:59] and to remove the OS change from the equation I'm moving to bullseye like the rest [07:48:12] we can sort these out with a second bookworm install server in codfw eventually [07:49:12] ack [10:45:04] 10SRE-tools, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#9776962 (10MoritzMuehlenhoff) [10:58:36] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10Znuny: Clean up OTRS/Znuny addresses handles by gsuite - https://phabricator.wikimedia.org/T284145#9777028 (10LSobanski) [13:30:40] TIL https://docs.python.org/3.12/howto/perf_profiling.html [13:41:22] yeah that's nice but IIRC debian's python is not compiled with the right flags to enable it [13:41:49] and you might get better info from python-specific profilers, but as usual it depends on the use case [15:11:24] I've pointed the webproxy for magru back to install7001: https://gerrit.wikimedia.org/r/c/operations/dns/+/1028853 [15:26:02] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10Znuny: Clean up OTRS/Znuny addresses handles by gsuite - https://phabricator.wikimedia.org/T284145#9777785 (10LSobanski) [15:39:49] volans: ah snap :( I was thinking about it to debug Python pids running as containers on k8s [15:40:15] anyway, ms-fe1009 (swift proxy) is running with PKI [15:40:32] hopefully the rest will be done by middle of next week [15:43:45] nice! [15:44:06] eh, also as you have to enable it at PID start time, I'm not sure on the performance impact [15:46:51] yes yes [15:47:17] but surely less invasive than profiling etc.. (via python code I mean) [15:47:44] indeed [15:48:02] I already used perf a lot of times to debug Python issues on k8s but I was lucky that the culprit were C lib calls :) [15:48:16] eheheh [15:48:57] we could potentially have some "canary" process where it runs with it enabled, if we get python compiled with the correct flags [15:50:01] it is also handy to just inject the env variable and restart the pod (via helmfile or even manually) [15:50:05] if one doesn't one canaries [15:51:10] sure, or even have a signal handler that does sys.activate_stack_trampoline() with the proper signal [15:51:19] to enable/disable it at will [15:55:04] ah wow didn't think about it