[11:42:56] <Emperor>	 Hi folks - a decomm cookbook just failed at the deploying DNS stage
[11:43:34] <Emperor>	 Exception: Command python3 ./utils/zone_validator.py --error --zones_dir /tmp/dns-check.m6ylcsyq/zones --ignores MISSING_ASSET_TAG,MISSING_MG
[11:43:34] <Emperor>	 MT_FOR_NAME,TOO_FEW_MGMT_NAMES failed with exit code 1, stderr:                                                                              
[11:43:34] <Emperor>	 zone-validator[INFO] PARSE STATISTICS | Files:28, Origins:526, Domain origins with records:24, Pointer origins with records:450, Skipped orig
[11:43:34] <Emperor>	 ins:273, IPs:13424, PTRs:9181, Names from direct records:12882, Names from pointer records:11391, Skipped records:213                        
[11:43:37] <Emperor>	  
[11:44:06] <Emperor>	 Ah, I think the problem may relate to dborch1003
[11:44:20] <Emperor>	 E101|MULTIPLE_IPS_FOR_NAME: Found 4 IPs for name 'dborch1003.eqiad.wmnet.', expected 1:                                                      
[11:44:20] <Emperor>	     netbox/eqiad.wmnet:1040 dborch1003.eqiad.wmnet. A 10.64.0.20                                                                             
[11:44:20] <Emperor>	     netbox/eqiad.wmnet:1041 dborch1003.eqiad.wmnet. A 10.64.0.46                                                                             
[11:44:20] <Emperor>	     netbox/eqiad.wmnet:1042 dborch1003.eqiad.wmnet. AAAA 2620:0:861:101:10:64:0:20                                                           
[11:44:23] <Emperor>	     netbox/eqiad.wmnet:1043 dborch1003.eqiad.wmnet. AAAA 2620:0:861:101:10:64:0:46                                                           
[11:44:26] <Emperor>	  
[11:44:42] <Emperor>	 federico3: I noticed you had an expired lock on DNS updates, were you in the middle of something here?
[11:47:32] <Emperor>	 I'm also not sure how to proceed - AFAICT most of the decomm is done, so is it OK to "skip" this step, or will that leave a bunch of DNS updates missing?
[12:08:01] <Emperor>	 any ideas? My cookbook run is blocked, as presumably are any further DNS updates :-/
[12:10:26] <Emperor>	 [going for lunch, biab]
[12:22:54] <federico3>	 Emperor: the makevm cookbook failed 
[12:23:14] <federico3>	 not it aborted 
[12:23:34] <federico3>	 *it aborted, I'm not sure if it left something broken around
[12:43:53] <Emperor>	 federico3: were you trying to make/re-make dborch1003? It looks like it's left it with two sets of IP addresses, and that's broken DNS updates
[12:48:39] <federico3>	 in the logs it said in deleted them but evidently not enough :) I'm asking around to clarify
[12:49:31] <Emperor>	 thanks, it'd be good to get this unstuck :)
[15:46:42] <sukhe>	 _joe_: Amir1: rolling out the final leg of IPv6 support for the authdns servers. we will know if something breaks.
[15:47:21] <Amir1>	 Break a leg! (Or a server)
[15:47:44] <sukhe>	 the registrar update goes in later but (some) traffic should start flowing towards the v6 addresses already
[15:51:28] <jynus>	 I will be doing some validation of new hardware on backup1015, for that I will be transfering large amounts of data from backup1006 (eqiad C3 -> C5) 
[15:53:07] <jynus>	 I will give a heads up to next shift of on call just in case
[15:54:42] <jynus>	 for context, I an rsyncing HDs, it should be almost impossible to staturate the network
[15:55:16] <jynus>	 ~80 MB/s
[15:55:26] <volans>	 is this CI failure in puppet known? seems unrelated to my changes: https://integration.wikimedia.org/ci/job/operations-puppet-tests-bullseye/23857/console
[15:56:29] <sukhe>	 Amir1: all good, we are seeing traffic to the v6 already https://w.wiki/Hynj
[15:56:32] <sukhe>	 you can rest easy now :)
[15:57:09] <sukhe>	 [note: the bulk of the traffic will move when the registrar updates the records on their end but this indicates that nothing is broken so far]
[15:57:33] <volans>	 sukhe: if you need a v6 tester I'm here
[15:57:37] <volans>	 (dual stack)
[15:59:06] <sukhe>	 volans: thanks <3 your recursor may not try the v6 just yet until the Markmonitor update. but if you see something broken, please let us know here
[15:59:17] <volans>	 sure
[16:07:56] <volans>	 arnaudb: might be related to any of your recent gerrit changes? ^^^ (the CI failure above)
[16:08:18] <arnaudb>	 checking
[16:08:30] <volans>	 I've jsut rebased and re-tried but re-failed
[16:09:24] <arnaudb>	 we changed the topology so the checks might be a bit off
[16:12:04] <arnaudb>	 fix implemented
[16:12:10] <arnaudb>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1243168
[16:12:22] <arnaudb>	 cc hashar ↑
[16:13:06] <hashar>	 arnaudb: +1 ):
[16:13:07] <hashar>	 :)
[16:13:09] <volans>	 thx
[16:13:10] <arnaudb>	 thanks!
[16:13:51] <hashar>	 and I imagine it did not fail because the profile specs are not run when solely hieradata are changed
[16:15:25] <arnaudb>	 merged!
[16:56:09] <_joe_>	 sukhe: I've been non-stop in meetings since you pinged us, osrry for not getting back to you earlier :)
[16:56:43] <sukhe>	 no worries. mostly for awareness, if I broke it, I own it (in this case :)
[18:24:25] <sukhe>	 we are on IPv6 on all our nameservers, so for users with IPv6 connectivity, no more NAT64 lookup for this bit at least
[18:24:54] <sukhe>	 I couldn't find anything better https://internet.nl/site/wikimedia.org/
[19:04:03] <inflatador>	 {◕ ◡ ◕}
[19:59:24] <XioNoX>	 congrats!
[21:38:53] <Squee-D>	 Hi vgutierrez, it looks like the ubuntu mirror is stale. https://mirrors.wikimedia.org/ubuntu/dists/jammy-updates/Release shows 02/feb/2026. https://us.archive.ubuntu.com/ubuntu/dists/jammy-updates/Release shows 24/feb/2026
[21:57:03] <jhathaway>	 thanks Squee-D, I'll take a look at the service
[22:10:25] <jhathaway>	 should now be up to date