[00:33:52] <bd808>	 andrewbogott: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1116888 is one possible solution for T385530
[00:33:53] <stashbot>	 T385530: Unable to persistently set fs.inotify.max_user_instances and fs.inotify.max_user_watches - https://phabricator.wikimedia.org/T385530
[00:34:16] <bd808>	 totally untested!
[00:36:04] <andrewbogott>	 ooh, that's very similar to what I just started to write.  I'm unsure about how the hiera lookup will behave though, I think it's regarded as bad from to not use explicit lookups.
[00:36:08] <andrewbogott>	 But I'll try it first :)
[00:37:57] <bd808>	 In the profiles I set the values directly without hiera
[00:38:51] <andrewbogott>	 yep, that part is definitely right
[00:38:58] <bd808>	 in classes we typically don't use the 'lookup' function and instead let Puppet do its normal thing, we just don't like that normal thing in prod config because someone decided that
[00:39:31] * bd808 knows who and when but that's irrelevant to the code convention
[00:41:29] <andrewbogott>	 yeah, as I feared setting "base::sysctl:;inotify::max_user_instances: 511" in hiera doesn't seem to do anything
[00:41:46] <andrewbogott>	 so it works perfectly as a ready-made pair of settings but we maybe want to add a way to configure via hiera...
[00:41:50] <andrewbogott>	 unless I'm missing something
[00:41:58] <bd808>	 if you have the ";" typo in there that's why
[00:42:20] <andrewbogott>	 ...
[00:42:38] <bd808>	 my first commit message had a typo and your paste just did too
[00:42:50] <andrewbogott>	 yep, fixing
[00:43:25] <andrewbogott>	 ok, minus the typo it does exactly what you say it does
[00:43:32] <bd808>	 I used hiera *a lot* in mediawiki-vagrant ;)
[00:43:50] <bd808>	 and did so actually for over a year before it was adopted in prod
[00:44:47] <bd808>	 The thing that is less useful about that method is that you can only apply one setting for the whole server via hiera no matter how many times the class is reused.
[00:45:00] <bd808>	 but for this that should be perfectly fine
[00:50:18] <andrewbogott>	 yeah, I think it's just what we need. adding Janis for a rubber stamp
[01:04:29] * andrewbogott stopping for the night
[08:22:30] <XioNoX>	 arturo: great work on https://wikitech.wikimedia.org/wiki/News/2025_Cloud_VPS_VXLAN_IPv6_migration !
[08:51:30] <arturo>	 XioNoX: thanks, hopefully we will restart work on that at the end of february, in a few weeks
[09:09:08] <arturo>	 ---> T385553 seems concerning
[09:09:08] <stashbot>	 T385553: Cloud VPS puppet breakage on 2025-02-04 related to puppet-enc - https://phabricator.wikimedia.org/T385553
[09:12:57] <taavi>	 is that "java unattended-upgrade without a puppetserver restart" bug again?
[09:13:09] <arturo>	 good theory, let me check
[09:15:25] <arturo>	 taavi: that was it
[09:15:52] <arturo>	 `sudo systemctl restart puppetserver.service` in the puppetserver fixes the problem
[09:39:51] <arturo>	 taavi: wdyt? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1117122
[09:43:41] <taavi>	 does that prevent manually upgrading the package?
[09:52:31] <arturo>	 taavi: I'm not sure. I know one way or two to override any apt preferences by hand (like `install package=version`)
[12:44:37] <arturo>	 andrewbogott: I think we are ready to go wrt. the cloudgw operation
[12:44:59] <andrewbogott>	 great! I assume there's nothing for me to do other than stand on the sidelines and cheer :)
[12:45:36] <arturo>	 mmm
[12:45:42] <arturo>	 there are 2 reimage operations to perform
[12:45:56] <arturo>	 to change puppet roles from cloudgw -> insetup and insetup -> cloudgw
[12:46:04] <arturo>	 I can do insetup -> cloudgw and you do the other?
[12:46:24] <arturo>	 cloudgw1004 is changing from insetup to the cloudgw role
[12:46:32] <arturo>	 cloudgw1002 is changing from the cloudgw role to insetup
[12:46:46] <arturo>	 this patch, basically https://gerrit.wikimedia.org/r/c/operations/puppet/+/1114998
[12:50:16] <andrewbogott>	 arturo: you want to reimage rather than just apply puppet and let it do its work?
[12:50:30] <arturo>	 yes, I prefer a full reimage
[12:50:56] <arturo>	 in neither of the two role transitions, puppet is prepared for proper leftover cleanup
[12:51:04] <andrewbogott>	 ok!
[12:51:06] <topranks>	 folks I don't think I can make the cloudgw call... but any issues just ping me 
[12:51:14] <andrewbogott>	 ok, thanks topranks 
[12:51:37] <andrewbogott>	 arturo: should we just dcom 1002 rather than go to ->insetup?
[12:51:52] <arturo>	 topranks: that's OK! I don't think we are doing the videocall anyway, the calendar invite was mostly a time slot reservation
[12:52:10] <arturo>	 andrewbogott: I believe ->insetup is == decom
[12:52:27] <andrewbogott>	 I guess it's a step on the road, I usually skip it but that's fine.
[12:52:32] <andrewbogott>	 Any reason for me not to start that right now?
[12:52:38] <arturo>	 do it!
[12:53:36] <andrewbogott>	 hm, this patch also touches 1001, will it be upset about 1004 not being ready yet?
[12:53:48] <arturo>	 hopefully not
[12:53:53] <arturo>	 they can survive without each other
[12:53:55] <andrewbogott>	 ...and stashbot dies, I hope we aren't having a toolforge outage now
[12:54:01] <arturo>	 we aren't
[12:54:05] <andrewbogott>	 ok :)
[12:54:28] <arturo>	 I'm checking with
[12:54:30] <arturo>	 while true ; do sudo cookbook wmcs.openstack.network.tests --cluster-name eqiad1 ; done
[12:54:33] <arturo>	 on cloudcumin1001
[12:57:04] <andrewbogott>	 and... just doublechecking, we're sure that cloudgw1001 is currently the active gw right? :D
[12:57:10] <arturo>	 yes
[12:57:28] <arturo>	 I failovered them by hand a few minutes ago
[12:57:28] <andrewbogott>	 ok, here we go
[13:00:29] <Reedy>	 ah
[13:00:40] <Reedy>	 I guess that might explain all the bots on tools disappearing from IRC
[13:00:54] <arturo>	 Reedy: yeah
[13:01:20] <arturo>	 our particular CG-NAT does not really play well with the IRC bots
[13:02:35] <arturo>	 andrewbogott: I noticed I selected 'bookworm' for the new OS of cloudgw1004, whereas cloudgw1002. So if I don't cancel the reimage, this is not only replacing hardware but the OS as well. Wdyt?
[13:02:53] <arturo>	 whereas cloudgw1002/1001 are bullseye
[13:02:56] <arturo>	 *
[13:03:23] <arturo>	 I don't think we should pair the two things together
[13:03:41] <andrewbogott>	 I don't think we should, but on the other hand it saves us another future outage if we skip ahead...
[13:03:43] <arturo>	 and at very least, we should exercise the cloudgw puppet role on codfw1dev first
[13:03:51] <andrewbogott>	 Oh, well, that's a good point.
[13:03:52] <andrewbogott>	 Hm
[13:03:57] <andrewbogott>	 So I guess bullseye for today :(
[13:04:09] <arturo>	 ok, will re-start the reimage
[13:06:22] <arturo>	 andrewbogott: the cookbook is asking whether this puppet role is already in puppet7
[13:06:42] <andrewbogott>	 let me check on 1001
[13:07:12] <andrewbogott>	 yes, it's on 7
[13:07:34] <arturo>	 ok, thanks
[13:12:56] <andrewbogott>	 arturo: belt + suspenders:  https://gerrit.wikimedia.org/r/c/operations/puppet/+/1117189
[13:14:25] <arturo>	 andrewbogott: maybe leave the file hieradata/hosts/cloudgw1003.yaml for when we move it
[13:14:59] <arturo>	 or, specify the setting on the role level, rather than per host?
[13:15:40] <andrewbogott>	 I think I'm generally confused about this, let me do some more tests
[13:21:00] <andrewbogott>	 arturo: it's already set in the role hiera so I'll just abandon my patch, all is well
[13:21:36] <arturo>	 ok great
[13:29:14] <arturo>	 so far the cloudgw1004 reimage is going fine. First puppet run started
[13:31:35] <andrewbogott>	 1002 reimage is wrapping up
[13:37:59] <arturo>	 1004 first puppet run completed, now rebooting and another puppet run
[13:46:12] <arturo>	 reimage finished
[13:46:16] <arturo>	 I'm spot checking a few things
[13:46:24] <arturo>	 then we will failover to the new box and see what happens?
[13:50:19] <andrewbogott>	 scary but, yes, I guess that's the plan :)
[13:55:14] <arturo>	 andrewbogott: ok, then doing it now!
[13:55:27] <andrewbogott>	 ok!
[13:55:49] <arturo>	 done
[13:56:31] <arturo>	 network tests look green
[13:56:37] * andrewbogott trying to not say anything that's bad luck
[13:58:37] <andrewbogott>	 I guess we need to change the dashboards to show the new node...
[13:58:47] <arturo>	 I'm doing so now
[13:59:38] <arturo>	 done
[14:00:22] <andrewbogott>	 whoops I'll drop my pending changes
[14:00:23] <arturo>	 cloudgw1004 is now sustaining almost 2Gbps traffic. I think we can declare the operation over, and a success :-P
[14:00:50] <andrewbogott>	 agreed, everything looks good to me.
[14:01:13] <andrewbogott>	 So we'll give this a couple of weeks and then do the other switchover.
[14:01:22] <arturo>	 ok
[14:01:25] <andrewbogott>	 Which we don't need to schedule since it'll be a passive server being changed
[14:01:29] <andrewbogott>	 nice work!
[14:01:50] <arturo>	 we would need to failover as well, for the sake of testing the new box, but yeah
[14:01:55] <arturo>	 thanks for assisting
[14:01:58] <andrewbogott>	 true
[14:03:24] <andrewbogott>	 arturo: want to send the 'all clear' email?
[14:03:39] <arturo>	 you do please, I'm about to lunch
[14:04:08] <andrewbogott>	 ok
[16:25:15] <arturo>	 topranks: there was something weird happening wrt. DNS while we did the cloudgw migration, see T385600 Not impacting anything, but definitely a mystery
[16:25:16] <stashbot>	 T385600: Cloud DNS: investigate weird graph during cloudgw replacement operation - https://phabricator.wikimedia.org/T385600
[16:41:55] <arturo>	 topranks: thanks for confirming :-) I'm a bit paranoid about DNS problems lately
[16:43:24] <topranks>	 arturo: np 
[16:43:42] <topranks>	 we can change that if you want, configure bird on the hosts to send a community with the router which will affect routing 
[16:44:07] <topranks>	 basically tag the routes as high/low priority 
[16:44:20] <arturo>	 nah, I'm fine with how things are today