[03:24:05] * dhinus paged: ProjectProxyMainProxyDown project-proxy (proxy-04 main-nginx-https page wmcs)
[03:29:11] <dhinus>	 a similar alert triggered last thursday for the other proxy (proxy-03) and auto-resolved after 5 minutes. this time it's still firing after 15 mins
[03:29:31] <dhinus>	 checking the proxy-* vm state in horizon
[03:29:55] <dhinus>	 both proxy-03 and proxy-04 are shown as "running"
[03:31:11] <dhinus>	 I can ssh to "proxy-04"
[03:32:15] <dhinus>	 unit "nginx.service" is failed, among a few others
[03:35:46] <dhinus>	 nginx: [emerg] invalid number of arguments in "resolver" directive in /etc/nginx/sites-enabled/catalyst-qte-wmcloud-org:19
[03:37:13] <dhinus>	 restarting the unit fails with the same error
[03:40:03] <dhinus>	 I created an incident doc at https://docs.google.com/document/d/1VBLbLgg5dt_A_g_jHwxgyRMw9amINri4DPe6jc6bshc/edit?tab=t.0#heading=h.m4uq0l1dvtgq
[03:42:47] <dhinus>	 I am the IC for now
[03:48:07] <dhinus>	 the similar alert from last thursday seems unrelated: the nginx logs for proxy-03 don't include any error on Thursday
[03:54:05] <dhinus>	 the error is caused by a missing IP in /etc/nginx/sites-enabled/catalyst-qte-wmcloud-org:19
[03:54:31] <dhinus>	 I compared that file between proxy-03 and proxy-04, in proxy-03 the line is "resolver 172.20.255.1;", in proxy-04 it is "resolver ;"
[03:55:07] <dhinus>	 forcing a puppet run fails with "getaddrinfo: Temporary failure in name resolution"
[03:55:39] <dhinus>	 this sounds like another instance of T379927
[03:55:40] <stashbot>	 T379927: Puppet removed "nameserver" line from /etc/resolv.conf - https://phabricator.wikimedia.org/T379927
[03:56:24] <dhinus>	 I edited /etc/resolv.conf manually and restarted puppet
[03:57:01] <dhinus>	 that fixed the issue with puppet AND the issue with nginx
[04:12:25] <dhinus>	 the alert is no longer firing in alertmanager, but it was not auto-resolved in victorops. I'm resolving it manually
[04:13:58] <dhinus>	 I am resolving the incident.
[04:14:08] <dhinus>	 I have reopened T379927 and added a comment there.
[04:14:09] <stashbot>	 T379927: Puppet removed "nameserver" line from /etc/resolv.conf - https://phabricator.wikimedia.org/T379927
[04:15:26] <dhinus>	 back to sleep, see you later :)
[08:52:54] <dcaro>	 thanks for handling the page!
[09:36:08] <arturo>	 heads up, I'm about to deploy some new network settings into eqiad1 related to VXLAN and IPv6. No impact expected, so please report if you see anything weird
[10:19:26] <topranks>	 arturo: cool, let me know when you are done 
[10:19:42] <topranks>	 the faulty link from E4 to D5 we had last week has been clean since the optic change in E4 on Friday 
[10:19:43] <topranks>	 https://phabricator.wikimedia.org/T380503#10352059
[10:19:44] <arturo>	 topranks: thanks
[10:19:59] <topranks>	 So my feelings are to put traffic back on it, but I'll wait till you're done 
[10:51:01] <arturo>	 there seems to be a network outage in eqiad1
[10:51:10] <arturo>	 most likely related to the IPv6 changes, I'll try to undo them
[10:51:21] <dcaro>	 that might be why toolsbeta-harbor stopped answering xd (/me was sshing)
[10:51:35] <dcaro>	 let me know if you need me to do any test or anything
[10:51:47] <blancadesal>	 ...and why functional tests are failing on toolsbeta :)
[11:16:04] <topranks>	 yeah we didn't get anything similar 
[11:16:34] <topranks>	 the networks are all separate /21s so in theory it all should be fine, worst I'd have expected is the *new* networks not to work right for some reason 
[11:18:14] <topranks>	 tricky to know how to move forward with it right now 
[11:18:49] <topranks>	 arturo, dcaro: anyway - I will go ahead and "re-pool" that link from D5 to E4 unless anyone objects?
[11:19:01] <arturo>	 topranks: ok for me
[11:19:02] <dcaro>	 topranks: sounds good to me
[11:19:10] <topranks>	 ok cool, I'll monitor of course 
[11:19:16] <dcaro>	 thanks!
[11:19:25] <arturo>	 yeah, I don't know how to move forward at the moment, I'll need to think for a bit
[11:21:30] * dcaro lunch
[11:24:55] <topranks>	 link looks ok still, 100kpps on it and no errors so far 
[11:25:08] <topranks>	 I'll keep an eye on it over the day and close the task if it remains clean 
[12:22:40] <dhinus>	 heads up I'm migrating the designate zone db.svc.eqiad.wmflabs to the cloudinfra project T380491
[12:22:40] <stashbot>	 T380491: Migrate "db.svc.eqiad.wmflabs." DNS zone to cloudinfra project - https://phabricator.wikimedia.org/T380491
[12:23:04] <dhinus>	 this will simplify the toolsdb DNS change I have to perform as part of the toolsdb upgrade
[12:23:41] <arturo>	 ack
[12:27:20] <dhinus>	 the zone transfer was successful
[12:56:09] <arturo>	 dhinus: there are pending plan changes in tofu-infra, you may have forgotten to run tofu apply
[12:56:23] <arturo>	 I can do myself, if that's OK
[12:56:28] <dhinus>	 sure thanks
[12:56:41] <arturo>	 ok
[13:44:16] <dhinus>	 arturo: updating the DNS for toolsdb resulted in designate not resolving the hostname anymore
[13:44:26] <dhinus>	 maybe because i swapped a cname for an A record
[13:44:31] <arturo>	 :-(
[13:44:36] <dhinus>	 horizon looks good, but dig does not resolve from the bastion
[13:44:53] <arturo>	 how are you calling dig?
[13:44:58] <dhinus>	 is there any way to restart/refresh designate?
[13:45:04] <dhinus>	  dig tools.db.svc.wikimedia.cloud
[13:45:20] <arturo>	 yes, with the restart cookbook
[13:45:24] <dhinus>	 great
[13:45:28] <dhinus>	 let me try that
[13:45:55] <arturo>	 I doubt it will be that, but a restart should not cause any harm
[13:47:53] <dhinus>	 running the cookbook
[13:48:14] <dhinus>	  sudo cookbook wmcs.openstack.restart_openstack --designate --cluster-name eqiad1
[13:48:17] <arturo>	 in my laptop I can resolve it
[13:48:19] <arturo>	 arturo@nostromo:~ $ dig +short tools.db.svc.wikimedia.cloud
[13:48:19] <arturo>	 172.16.0.168
[13:48:28] <dhinus>	 interesting
[13:48:42] <arturo>	 mmm
[13:48:44] <dhinus>	 true from my laptop as well
[13:48:46] <arturo>	 I know what's happening
[13:48:51] <arturo>	 the FQDN changed zone, right?
[13:49:02] <arturo>	 well, I have a hunch
[13:49:03] <dhinus>	 no
[13:49:08] <dhinus>	 it only changed from CNAME to A
[13:49:10] <dhinus>	 same zone
[13:49:19] <arturo>	 mmm ok
[13:50:09] <dhinus>	 the cookbook restart did not fix it
[13:50:25] <arturo>	 dig shows SERVFAIL
[13:50:34] <arturo>	 which likely means invalid DNS config somehow
[13:51:09] <arturo>	 Nov 25 13:51:02 cloudservices1005 pdns-recursor[2773294]: msg="Sending SERVFAIL during resolve" error="got a CNAME referral (from cache) that causes a loop" subsystem="syncres" level="0" prio="Notice" tid="3" ts="1732542662.251" ecs="" mtid="102916031" proto="udp" qname="tools.db.svc.wikimedia.cloud" qtype="A" remote="185.15.56.63:39724"
[13:51:13] <arturo>	 dhinus: ^^^
[13:52:21] <dhinus>	 we are deleting the old CNAME
[13:52:26] <dhinus>	 let's see if it improves things
[13:53:20] <arturo>	 maybe a restart of the recursor will clear the cache and that's all we need
[13:53:23] <arturo>	 want me to do that?
[13:53:53] <dhinus>	 yes please
[13:54:01] <arturo>	 ok, doing
[13:54:17] <dhinus>	 yes now it works
[13:54:17] <arturo>	 done on cloudservices1005
[13:54:25] <arturo>	 🎉
[13:54:32] <dhinus>	 thanks arturo 
[13:55:36] <arturo>	 happy to help
[13:56:30] <arturo>	 (also done on cloudservices1006 for completeness)
[14:07:16] <arturo>	 topranks: there seems to be some netbox dns changes that are not being accepted by the cookbook because missing PTR zone for one of the new IPv6 subnets, T380746
[14:07:16] <stashbot>	 T380746: dns: add PTR support for 2a02:ec80:a000:: - https://phabricator.wikimedia.org/T380746
[14:10:16] <topranks>	 hahaha 
[14:10:34] <topranks>	 sorry I shouldn't laugh... my bad dude I shouldn't have left you walk into that one
[14:10:43] <topranks>	 forgot when I said add the IPs it'd happen 
[14:11:22] <arturo>	 that's ok :-)
[14:11:47] <topranks>	 let me get a patch ready 
[14:11:55] <arturo>	 thanks!
[14:12:02] <arturo>	 I need to step out for a bit, be back later
[14:12:13] <topranks>	 ok I'll get it fixed up 
[15:21:50] <topranks>	 arturo: just fyi I removed those dns names for now in netbox, CI confused me and it was blocking the magru work, I'll re-add them and fix it once magru stuff is done 
[15:23:50] <arturo>	 topranks: ok, thanks, no problem
[15:37:15] <dcaro>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1097390 quick review
[15:57:25] <bd808>	 taavi: I think I remember some discussion, but I don't remember if anyone tried to follow up on dropping .labsdb or even trying to figure out what tools might be impacted. I have an unsupported hunch that a number of very old tools/bot/scripts would fail, but they should also be pretty easy to fix.
[15:58:20] <bd808>	 For those unaware of the ancient history here, we deprecated .labsdb as part of the 2017 Wiki Replicas rebuild. https://phabricator.wikimedia.org/phame/post/view/70/new_wiki_replica_servers_ready_for_use/
[16:33:19] <dhinus>	 oh wow I didn't even know about "tools.labsdb" :) should we add ".labsdb" to the list of legacy domains at https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/DNS ?
[16:36:03] <arturo>	 dhinus: +1
[16:58:42] <dhinus>	 arturo: added
[16:59:14] <dhinus>	 I'm not sure where the records for .labsdb are stored, I don't see them in openstack
[17:00:26] <arturo>	 thanks
[17:00:28] * arturo offline
[17:05:17] <bd808>	 dhinus: modules/profile/files/openstack/base/pdns/recursor/labsdb.zone in ops/puppet.git
[17:28:34] <dcaro>	 anyone has ever seen a server install hang when doing mkfs?
[17:28:44] <dcaro>	 https://www.irccloud.com/pastebin/Vlg8IZtY/
[17:28:58] <dcaro>	 the mkfs stays in 'D' state (uninterruptible sleep)
[18:04:58] * dcaro off
[18:17:15] <andrewbogott>	 dhinus: do we still support the list of hostnames option with cumin on cloudcumin hosts?  e.g. "cumin F{"./bustedresolv.txt"} hostname"?
[18:17:21] <andrewbogott>	 Either we don't or my syntax is wrong
[18:53:35] <dhinus>	 andrewbogott: good question, I have never tried that syntax, does it return an error or an empty list?
[18:54:14] <dhinus>	 if an empty list, it could be because it's matching on prod hosts instead of cloudvps hosts
[18:57:23] <dhinus>	 can you try adding "--backend openstack"? if that doesn't fix it, I would open a Phab as it could be a bug we didn't discover when setting up cloudcumins
[18:58:26] <dhinus>	 F{} is a separate "backend" and maybe it's not enabled
[19:19:51] <andrewbogott>	 dhinus: T380789, not urgent.
[19:19:52] <stashbot>	 T380789: Revive the HostFile backend on cloudcuminXXXX - https://phabricator.wikimedia.org/T380789
[20:23:09] <Rook>	 Are codfw1dev projects managed in tofu atm?
[21:08:54] <arturo>	 Rook: yes
[21:09:07] <Rook>	 Neat, where is that?
[21:09:28] <arturo>	 https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/OpenTofu docs https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/ repo
[21:10:21] <Rook>	 Thanks!
[21:10:55] <arturo>	 np
[21:13:16] <arturo>	 perhaps Admin/tofu-infra would be a better name for the docs page