[00:52:13] <bd808>	 I am working on some stuff for the offsite and realized that https://k8s-status.toolforge.org/ is not on https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Ownership. Should I WP:BOLD'ly add it?
[00:53:16] <bd808>	 https://wikitech.wikimedia.org/wiki/Tool:Fourohfour isn't on there either
[01:19:28] <andrewbogott>	 bd808: yes please!
[03:11:46] <andrewbogott>	 For anyone who is curious: here is a quick report I ran about membership of cloud-vps projects.  https://docs.google.com/spreadsheets/d/1L0ZjH1FUH5XJK-bgs_rp6nPZLGymb_m2WtdP0E5eMgo/edit?usp=sharing  I wouldn't put those stats on a slide without closer analysis though!
[10:42:17] <blancadesal>	 arturo: thank you for all the work on the incident doc and follow-up items!
[10:42:29] <arturo>	 you are welcome!
[12:49:14] <arturo>	 please carefully review this https://gerrit.wikimedia.org/r/c/operations/puppet/+/1098498 this is one of the causes of the outage the other day
[12:53:07] <arturo>	 is T380833 in our radar?
[12:53:08] <stashbot>	 T380833: [harbor] some artifacts and projects seems to have gone missing - https://phabricator.wikimedia.org/T380833
[12:59:06] <dcaro>	 that's not one of the causes of the outage
[12:59:22] <dcaro>	 it was surfaced during it, but it's unrelated afaik
[12:59:45] <dcaro>	 oh, wait, you mean the patch, not the task?
[12:59:52] <dcaro>	 I'm confused xd
[13:00:28] <dcaro>	 Raymond_Ndibe: started looking into the task, that is unrelated to the outage (afaik), looking into the other patch (that is related to the outage)
[13:03:44] <arturo>	 yeah, the radar question was related to the harbor. I just liked the ticket to  the outage, because the description says so, but I can unlink if they are not really related
[13:07:43] <dcaro>	 it's ok, we noticed during the outage and it prevented some tools from restarting correctly, seems reasonable to link the tasks (even if it was not the cause)
[13:07:46] * dcaro lunch
[13:10:06] <arturo>	 patch merged and deployed (without outage!)
[13:23:26] <blancadesal>	 yay!
[13:48:14] <blancadesal>	 what was this recent spike in failed deployments? https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview?orgId=1&var-cluster=tools&from=now-1h&to=now&var-cluster_datasource=prometheus-tools (looks like just a blip but I'm curious)
[13:53:48] <arturo>	 I have no idea!
[13:57:57] * arturo food
[13:59:59] <blancadesal>	 looking at some of the runbooks, there seems to be a few outdated links. e.g. https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/JobsApiUpMetricUnknown, the links to Karma UI (is this something we still use) and both tools/toolsbeta prometheus
[14:00:31] <blancadesal>	 should those be links to grafana?
[14:03:25] <dcaro>	 no, prometheus is what behind grafana, it points there as the source of truth for prometheus metrics is there, from the prometheus you can see the scrapes (if they failed), and the alerts that are defined
[14:03:46] <dcaro>	 that alert is about the metric not being there, so you have to check why prometheus did not add it
[14:04:25] <blancadesal>	 ok, both of those links are broken though. what are the current ones?
[14:04:44] <dcaro>	 oh, should be https://tools-prometheus.wmflabs.org/tools one of them
[14:05:32] <dcaro>	 hmm, shouldn't wmcloud.org be the right domain?
[14:07:34] <dcaro>	 I'll add that one as it's the new domain we should be using in the proxies
[14:07:45] <blancadesal>	 and karma UI link should be this? https://alerts.wikimedia.org/
[14:07:55] <dcaro>	 wait, this works https://prometheus.svc.toolforge.org
[14:08:08] <dcaro>	 yep, that's the one we use (prod one)
[14:10:10] <blancadesal>	 the svc one leads me to the apache2 debian default page
[14:10:54] <dcaro>	 all do, unless you pass the extra /tools, updated the pages
[14:11:02] <dcaro>	 (envvars, jobs and builds api)
[14:11:24] <dcaro>	 we could fix that in the apache config
[14:12:45] <blancadesal>	 aha, we were editing at the same time 
[14:13:12] <dcaro>	 sorry xd
[14:13:24] <dcaro>	 feel free to write on top of my changes
[14:13:33] <blancadesal>	 I might have :)
[14:14:34] <dcaro>	 np
[14:15:14] <blancadesal>	 we need a bot that alerts when links are broken on wikitech :))
[14:15:56] <dcaro>	 I added an 'exploration' section to the incident wiki page, I plan on adding there more details on getting the causality of the incident (from the 3 main issues, doing a "this was caused by" noting if that link is strong or a guess), feel free to add/change/etc. with your notes
[14:16:18] <dcaro>	 it kinda rings a bell
[14:16:20] <dcaro>	 (the bot)
[14:21:46] * dhinus paged Cloudvirt node cloudvirt1061 is down
[14:23:34] <dhinus>	 arturo: this is probably related to the maintenance mentioned in T380673
[14:23:35] <stashbot>	 T380673: Kernel error Server cloudvirt1061 may have kernel errors - https://phabricator.wikimedia.org/T380673
[14:24:26] <dhinus>	 I will add a downtime with "sre.hosts.downtime --days 14"
[14:25:51] <dhinus>	 I think this is the first situation where the new "kernel error" alert has prevented a wider outage
[14:26:56] <dhinus>	 that server had a kernel error yesterday, it triggered the alert and it was manually depooled
[14:37:12] <andrewbogott>	 there's a link about the missing harbor image thing in the backscroll but no response as far as I can see... is anyone investigating that? (Or, alternatively, can someone explain to me why it isn't extremely serious?)
[14:37:17] <andrewbogott>	 (also, good morning!)
[14:40:42] <dcaro>	 andrewbogott: Raymond_Ndibe started looking into it, it's not a generalized issue (affecting <10 tools), you can always rebuild your image (worst case scenario)
[14:41:06] <dcaro>	 (I think I replied :/)
[14:43:22] <andrewbogott>	 dcaro: oh, you did, there was a : in your response so I scrolled past it as not directed to me :)
[14:43:26] <andrewbogott>	 <10 tools is reassuring
[14:59:24] <arturo>	 dhinus: thanks for T380673. Yes, a good catch by the detector
[14:59:25] <stashbot>	 T380673: Kernel error Server cloudvirt1061 may have kernel errors - https://phabricator.wikimedia.org/T380673
[14:59:50] <arturo>	 I used a cookbook to drain the HV, so maybe I did not use the right one, the one that should include the downtime
[15:10:29] <andrewbogott>	 I think you used the right cookbook, I got the same alert when I rebooted a different one on Monday. Something is a bit wrong with the timing in the cookbook I think
[15:34:31] <dhinus>	 the drain cookbook did not alert, but today DCops rebooted it again because we told them it was depooled, and that caused the alert
[15:44:09] <dhinus>	 new DNS failures today: T380991
[15:44:10] <stashbot>	 T380991: Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991
[16:07:18] <bd808>	 Fresh resolver issues in the integration project -- T380991
[16:07:19] <stashbot>	 T380991: Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991
[16:09:14] <bd808>	 100% guess, but it feels reasonable to me that CI does a lot of DNS lookups and people pay attention to failed jobs so this is less likely to be integration project specific and more likely that resolver failures are noticed there and reported.
[16:23:38] <dcaro>	 aahh, it's the same ticket dhinus shared, yep, I agree it's probably not project-specific 
[16:28:28] <hashar>	 maybe the recursors can be offloaded with a local cache?
[16:28:41] <hashar>	 I have long lost track of how DNS caching is managed
[16:29:23] <dhinus>	 I would really like to find the root cause of recursor issues, it seems they are getting more frequent, and even if we work around them, the will bite us later
[16:29:43] <hashar>	 +1
[16:29:45] <dhinus>	 it's possible it's a more general network issues, and recursor issues are the symptom
[16:30:23] <arturo>	 there are no clear or obvious indications in the logs or in the metrics that the DNS recursors are malfunctioning
[16:30:37] <arturo>	 at least I don't see anyt
[16:30:46] <dhinus>	 maybe the requests are not even getting to the recursors?
[16:31:58] <arturo>	 but the servers get 2K/reqs, why would some arbitrary requests not get there?
[16:32:11] <arturo>	 and I assume clients would retry anyway
[16:32:32] <dhinus>	 no idea...
[16:32:44] <arturo>	 so, assuming a client retries 3 times before reporting a timeout, why would 3 requests in a row not get to the server?
[16:38:13] <andrewbogott>	 The recursors could move to VMs and we could have a lot more of them,  if it turns out to be a capacity issue. I'd like to rule out networking issues first though.
[16:39:43] <taavi>	 T305834 would be an easy way to reduce the recursor query volume by a large percentage
[16:39:44] <stashbot>	 T305834: Cloud VPS: drop wmflabs names from profile::resolving::domain_search - https://phabricator.wikimedia.org/T305834
[16:40:37] <andrewbogott>	 I think there's still a handful of vms with wmflabs domains but we're close to rid of them
[16:40:47] * andrewbogott starts with 'watch -e dig gerrit.wikimedia.org'
[16:40:57] <dcaro>	 xd
[16:41:01] <taavi>	 there's one shutdown that we could delete anytime
[16:41:18] <taavi>	 (and a bunch of related patches to remove support for those that I'd like reviews for)
[16:41:32] <arturo>	 how old are wmflabs VMs?
[16:42:04] <arturo>	 more of a statement than a question: how old are wmflabs VMs! :-P
[16:42:15] <taavi>	 the last .wmflabs VMs date from just after the buster release
[16:44:05] <andrewbogott>	 taavi: do you already have a command handy for spitting out all existing .wmflabs domains?  If not I'll invent one.
[16:44:19] <taavi>	 os record list 114f1333-c2c1-44d3-beb4-ebed1a91742b --sudo-project-id noauth-project
[16:44:39] <taavi>	 (that's everything expect .svc.eqiad.wmflabs)
[16:44:52] <andrewbogott>	 thx
[16:45:24] <taavi>	 deleting deployment-cumin is T380678
[16:45:25] <stashbot>	 T380678: Re-create deployment-cumin - https://phabricator.wikimedia.org/T380678
[16:47:02] <taavi>	 basically the only problem here is ensuring the 'tools-db' and 'tools-redis' short names keep working in toolforge
[16:47:43] <andrewbogott>	 deployment-cumin has been shut off since Nov. 24, 2024, 12:47 p.m, that seems promising (maybe you did it?)
[16:47:57] <taavi>	 yes :-)
[16:48:01] <andrewbogott>	 great!
[16:50:57] <bd808>	 taavi: do you have a gut feel for how many tools might fail hard if we break resolution on those ancient tools-db and tools-redis bare hostnames? I know I have been an advocate of dragging around legacy aliases pretty much forever, but I can also see that sometimes breaking things for a bit is the better option.
[16:55:22] <andrewbogott>	 could we potentially grep tools nfs and actually find uses?
[16:55:47] <taavi>	 probably too many :( https://codesearch.wmcloud.org/search/?q=tools-%28db%7Credis%29%28%24%7C%5B%5E%5C.%5D%29&files=&excludeFiles=&repos= is showing a non-zero number of hits, and I suspect there's a far greated number in older tools that don't use git hosts that codesearch indexes
[17:01:42] <bd808>	 ugh, yeah
[17:04:17] <hashar>	 andrewbogott: `watch -e dig` would most probably not catch it :/
[17:04:34] <andrewbogott>	 hashar: have a thought about something on the cli that might?
[17:04:40] <hashar>	 I mean
[17:04:49] <hashar>	 watch defaults to running the command every two seconds
[17:04:55] <andrewbogott>	 btw, it did just catch it! Maybe
[17:04:59] <andrewbogott>	 https://www.irccloud.com/pastebin/JO40gywa/
[17:05:02] <hashar>	 OHHHHHHHHHHHHHHHHHHHhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
[17:05:09] <hashar>	 ahah
[17:05:17] <dcaro>	 andrewbogott: I added a not to the task, our k8s api pods are also seeing the issue
[17:05:20] * hashar files a lottery ticket
[17:05:23] <dcaro>	 (in tools)
[17:05:34] <hashar>	 andrewbogott: I stand corrected! :b
[17:06:56] <hashar>	 https://phabricator.wikimedia.org/P69462
[17:07:00] <hashar>	 that is what I came up with last time
[17:07:06] <arturo>	 I truly can't explain why 3 timeouts in a row. There are no packet drops reported anywhere, no?
[17:08:08] <hashar>	 that is 8 threads doing gethostbyname() with a 250ms sleep and  assumes that it would throw an ,exception on error
[17:08:48] <andrewbogott>	 hashar: thanks, that might be quicker
[17:10:11] <hashar>	 I don't think it caught anything though.  Then maybe python 3.9.2 socket.gethostbyname()  retries by itself/has a long timeout or whatever
[17:11:52] <hashar>	 Lock to allow python interpreter to continue, but only allow one
[17:11:52] <hashar>	    thread to be in gethostbyname or getaddrinfo */
[17:11:52] <hashar>	 #if defined(USE_GETHOSTBYNAME_LOCK) || defined(USE_GETADDRINFO_LOCK)
[17:11:52] <hashar>	 static PyThread_type_lock netdb_lock;
[17:11:52] <hashar>	 #endif
[17:12:06] <bd808>	 socket.gethostbyname should end up using the system's resolver cache, likely sssd in Cloud VPS
[17:12:08] <hashar>	 mouahah there is a global lock... so my threads are all waiting on each other
[17:12:20] <hashar>	 (maybe)
[17:14:45] <taavi>	 please review: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1098571
[17:15:23] <bd808>	 andrewbogott: That paste catching a failure with `watch dig` reminded me of this quip from Moriel about debugging a mediawiki-vagrant problem that was vexing here -- https://bash.toolforge.org/quip/AVteW6igQMK9DA-FLEiM :)
[17:21:05] <andrewbogott>	 as long as it's happening at all, I'm  happy that it's happening a lot
[17:21:49] <andrewbogott>	 taavi: you already added a hiera override for toolforge?
[17:22:16] <taavi>	 not needed, the "%{::wmcs_project}.eqiad.wmflabs" entry (not touched in that patch) keeps the tools-db/redis names working there for now
[17:22:25] <hashar>	 feel free to quip my claim of "that is not going to work" only to be proven wrong a minute later :b
[17:22:38] <andrewbogott>	 taavi: oh sure, ok
[17:23:32] <bd808>	 hashar: :) that was the best kind of being wrong
[17:24:01] <andrewbogott>	 hashar: right as you were saying that I was thinking '2 seconds is way too slow, I need to shorten that' and then it failed
[17:25:24] <hashar>	 OR
[17:25:38] <bd808>	 I just had a random thought about how amazing it would be to be able to search all of the toolforge tool log output for things like these failures. #someday
[17:25:39] <hashar>	 so yeah lot of luck :b
[17:26:02] <hashar>	 I assume dig does a direct query over the network, so at least that shows up there was a timeout
[17:26:19] <hashar>	 that can probably be pasted to the task for later reference
[17:29:03] <bd808>	 I am kind of wondering what constitutes a time out when dig should be using UDP for the lookups per it's man page
[17:29:28] <hashar>	 timeout waiting for a response?*
[17:29:37] <andrewbogott>	 yeah, the watch output says ' SERVER: 172.20.255.1#53(172.20.255.1) (UDP)'
[17:29:50] <bd808>	 I guess a "I sent a packet but never got a matching datagram back"
[17:30:15] <dcaro>	 I'd have the same guess yes (could be that the packet was lost, or ignored)
[17:30:48] <hashar>	 which can be the recursor never repliying
[17:30:52] <hashar>	 or one of the packet being lsot
[17:30:53] <hashar>	 lost
[17:30:54] <andrewbogott>	 there are a lot of things between dig and that recursor :/
[17:31:07] <bd808>	 "This option sets the timeout for a query to T seconds.  The  default  timeout  is 5 seconds. An attempt to set T to less than 1 is silently set to 1."
[17:31:14] <andrewbogott>	 And I think I just got lucky the first time, trying again with .1s interval and no failure yet.
[17:32:36] <andrewbogott>	 ah, there it goes
[17:32:49] * hashar awards the barn star of reproducibility
[17:35:05] <andrewbogott>	 probably unrelated but the recursor log just threw out a storm of
[17:35:06] <andrewbogott>	 cloudservices1005 pdns-recursor[122165]: msg="Ignoring empty (qdcount == 0) query on server socket!" subsystem="in" level="0" prio="Error" tid="3" ts="1732728865.038" proto="udp" remote="172.16.6.46:33794"
[17:35:36] <taavi>	 that remote is integration-agent-docker-1040.integration.eqiad1.wikimedia.cloud
[17:35:42] <andrewbogott>	 yeah
[17:36:07] <andrewbogott>	 it didn't cause my watch to fail though
[17:38:23] <hashar>	 that might be me
[17:39:00] <hashar>	 yeah sorry, I have tried emitting a bunch of empty udp packets to check whether there were packet loss or latency spikes
[17:39:03] <hashar>	 that was dumb
[17:39:28] <andrewbogott>	 np!
[17:39:31] <andrewbogott>	 Now I see
[17:39:32] <andrewbogott>	  cloudservices1005 pdns-recursor[122165]: msg="Sending SERVFAIL during resolve" error="Too much time waiting for www.t-ds.info|AAAA, timeouts: 5, throttles: 1, queries: 6, 7513msec" subsystem="syncres" level="0" prio="Notice" tid="1" ts="1732729156.698" ecs="" mtid="13153531" proto="udp" qname="www.t-ds.info" qtype="AAAA" remote="172.16.6.122:46225"
[17:39:46] <andrewbogott>	 oh that's someone trying to resolve a fake domain, normal
[17:43:26] <dcaro>	 yep, there's a bunch of those in the logs, but did not find any for gerrit (for example) last time I checked
[17:43:34] <dcaro>	 (couple days ago)
[17:44:03] <dcaro>	 there were some errors about some response from upstream DNS having the wrong size/format or something (it was a bit cryptic)
[17:45:25] <andrewbogott>	 taavi/arturo can you tell me more about how 172.20.255.1 balances between the two backends, and if there's a way for me to track when/if it flips from one to the other?
[17:45:49] <dcaro>	 getg, good luck
[17:45:52] * dcaro off
[17:46:06] <taavi>	 basically the in the same rack as the active cloudgw node gets all the traffic
[17:47:26] <andrewbogott>	 so it only flips if cloudgw also flips
[17:47:33] <taavi>	 yeah
[17:54:59] <dcaro>	 so if one goes down, it will not flip unless the cloudgw flips?
[17:54:59] * andrewbogott waiting to see if it still fails after https://gerrit.wikimedia.org/r/c/operations/puppet/+/1098571
[17:56:22] <dhinus>	 sorry for hijacking the conversation for something (hopefully) easier: do you know why horizon is not showing me the floating ip assignments for "project-proxy", but the CLI is?
[17:56:43] <dhinus>	 the "Action" buttons are also not visible, they work in project "tools", but not in "project-proxy"
[18:03:55] <andrewbogott>	 dhinus: are you a project member or just observer?
[18:04:09] <andrewbogott>	 hashar: this is a long-shot but any change the dns issues date all the way back to https://gerrit.wikimedia.org/r/c/operations/puppet/+/956419 ?
[18:04:20] <dhinus>	 andrewbogott: I've just added myself as member, but it didn't make a difference. Maybe I need to log out though.
[18:04:31] <andrewbogott>	 just switch to a different project and back
[18:05:33] <andrewbogott>	 *any chance*
[18:06:25] <dhinus>	 andrewbogott: slightly better, I can see the buttons now. I cannot see the value for one of them though, maybe because it's not matching any instance?
[18:07:02] <andrewbogott>	 dhinus: what panel are you looking at?
[18:07:07] <dhinus>	 https://horizon.wikimedia.org/project/floating_ips/
[18:07:12] <dhinus>	 project: "project-proxy"
[18:07:29] <dhinus>	 CLI tells me both are assigned: wmcs-openstack floating ip list --project project-proxy
[18:08:07] <andrewbogott>	 ok, I see one mapped and one unmapped in the ui
[18:08:25] <andrewbogott>	 yes, if it's mapped to a VM that doesn't exist anymore it probably doesn't know what to show you there
[18:08:50] <dhinus>	 the fun part is I can use that floating IP, i.e. I can ping it or curl it
[18:09:12] <dhinus>	 unless I'm missing something, it's the main floating IP for the web proxy
[18:09:14] <taavi>	 one of them is mapped to a neutron VIP, one to an instance
[18:09:28] <dhinus>	 taavi: ah-ha, where is that mapping?
[18:09:56] <taavi>	 the CLI command you pasted shows it
[18:10:04] <andrewbogott>	 so that's probably a smallish horizon bug, it should show the IP even if it can't attribute it to anything...
[18:10:41] <dhinus>	 I mean, how do I configure the "neutron VIP"? I was just trying to find out which proxy is active
[18:11:05] <dhinus>	 and the CLI tells me it maps to 172.16.5.140, where do I go next? :)
[18:11:07] <taavi>	 https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Keepalived
[18:11:13] <dhinus>	 thanks!
[18:12:42] <dhinus>	 that's the wikitech page I was missing. I'll add a link from https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Web_proxy
[18:25:08] <hashar>	 andrewbogott: the first report I was aware of is https://phabricator.wikimedia.org/T374830 which is from September
[18:25:19] <hashar>	 and I don't think I heard of it before that
[18:26:16] <hashar>	 maybe it is a rate limiting :)
[18:28:29] <dhinus>	 andrewbogott: I raised T381021
[18:28:30] <stashbot>	 T381021: [horizon] Floating IP pointing to Neutron VIP is not displayed - https://phabricator.wikimedia.org/T381021
[18:40:21] <dhinus>	 taavi: I added some info to https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Web_proxy but I'm a complete noob with keepalived, feel free to suggest better ways
[18:40:34] <dhinus>	 I assume there's some bits in puppet that might be worth linking but I haven't looked them up yet
[18:43:21] <andrewbogott>	 hashar: ok, so that's a whole year later than the switch to bgp
[18:43:27] <andrewbogott>	 dhinus: ok!
[18:43:57] <hashar>	 maybe the network part can be tested by having a daemon listening on an UDP port that would replyto incoming packet
[18:44:19] <hashar>	 then from a WMCS instance send packets to the cloudservices machine and see whether there is some packet loss
[18:44:31] * dhinus logging off
[18:44:36] <hashar>	 but I imagine that would show up somewhere in graphs
[18:45:26] <andrewbogott>	 hashar: my watch test is no longer failing, which suggests that it may be just a load issue. Can you please let me know if/when you see that failure again?
[18:45:47] <andrewbogott>	 (on T380991 if you remember)
[18:45:48] <stashbot>	 T380991: Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991
[18:45:57] <hashar>	 I don't seem them, but devs would ping the task or file one when it happens :)
[18:46:50] <taavi>	 https://grafana.wikimedia.org/d/000000044/pdns-recursor-stats?orgId=1&from=now-6h&to=now&viewPanel=3 shows a significant drop in load after that one patch was merged :-)
[18:47:51] <dcaro>	 also two days ago :/ weird
[18:52:56] <hashar>	 andrewbogott: https://phabricator.wikimedia.org/P71234
[18:53:14] <hashar>	 those are the Quibble build logs that match 'Could not resolve host'
[18:53:19] <hashar>	 over the last 3 / 4 days
[18:54:00] <hashar>	 maybe there is a rate limiter
[18:54:05] <hashar>	 packet throttling per project
[18:54:12] <andrewbogott>	 shouldn't be
[18:54:18] <andrewbogott>	 if that's from just now then it looks like things are better?
[18:54:42] <hashar>	 last one at 17:09
[18:55:03] <andrewbogott>	 90+ minutes ago, right?
[18:55:05] <hashar>	 so yeah that is "solved"
[18:55:07] <hashar>	 yeah
[18:55:08] <hashar>	 UTC
[18:55:12] <hashar>	 (sorry timezones are a pain)
[18:55:42] <hashar>	 last time that got "solved" by switching the service from a cloudservices to another one
[18:55:57] <hashar>	 and I think the DRAC / kernel hhave been updated
[18:56:03] <hashar>	 but the root cause was not found
[18:56:19] <hashar>	 which honestly is fine to me. It is not like those kind of issues are easy  to find
[18:56:56] <andrewbogott>	 oh yeah, I'm not saying it's fixed for good, just thinking it might be a load issue rather than network flaking (which is what I was thinking until an hour ago)
[18:57:12] <hashar>	 yup
[18:57:51] <andrewbogott>	 taavi: what are your thoughts about selectively removing - "%{::wmcs_project}.eqiad.wmflabs" from search (e.g. everywhere but tools or alternately just in integration + deployment-prep)?
[18:57:57] <andrewbogott>	 Of course tools is probably 90% of the load
[19:15:02] <hashar>	 the only thing I notice is packets being dropped on cloudgw1002 on enp101s0f0np0
[19:15:07] <hashar>	 https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=cloudgw1002&var-datasource=thanos&var-cluster=wmcs&from=now-6h&to=now&viewPanel=11
[19:17:02] <hashar>	 which started on sunday
[19:17:05] <hashar>	 so maybe that correlates
[19:17:14] <hashar>	 aka something is off in the network / cloudgw
[19:17:19] <hashar>	 causes the udp packets to be dropped
[19:18:10] <hashar>	 the packets emitted by the instances never reach the server
[19:18:13] <hashar>	 or the response is never sent
[19:18:19] <andrewbogott>	 you're right, that doesn't look great
[19:18:32] <hashar>	 the client side eventually stop waiting and flag a timeout
[19:18:44] <hashar>	 doesn't explain why the retry would not get it
[19:20:58] <hashar>	 ah https://phabricator.wikimedia.org/T374830#10206549
[19:21:14] <hashar>	 so the cloudgw server had some network card issue
[19:21:27] <hashar>	 traffic got moved to cloudgw1001
[19:21:38] <hashar>	 and things got resolved
[19:21:47] <hashar>	 so I am inclined to state that is the same trouble happening again
[19:22:03] <hashar>	 the task was https://phabricator.wikimedia.org/T376589
[19:22:37] <andrewbogott>	 topranks: still around?
[19:22:42] <andrewbogott>	 pretty sure arturo is not
[19:26:15] <topranks>	 I'm here yeah 
[19:26:22] <topranks>	 what's up?
[19:26:25] <topranks>	 packet drops
[19:26:28] * topranks reading scrollback 
[19:27:49] <andrewbogott>	 It doesn't look as serious to me as T376589 but still not great
[19:27:49] <stashbot>	 T376589: cloudgw1002: network interface problem - https://phabricator.wikimedia.org/T376589
[19:28:09] <hashar>	 so yeah last time there was:  [Mon Oct  7 07:17:27 2024] NETDEV WATCHDOG: enp101s0f0np0 (bnxt_en): transmit queue 4 timed out
[19:28:16] <hashar>	 Arturo rebooted the server
[19:28:25] <hashar>	 and had the idrac updated
[19:28:29] <hashar>	 so maybe it is a kernel bug
[19:30:00] <hashar>	 then when in the trace there is `asm_exc_invalid_op`  that sounds terrible :b
[19:33:21] <topranks>	 there are some receive errors showing on that NIC but they aren't incrementing at the moment 
[19:33:25] <topranks>	 https://www.irccloud.com/pastebin/JQpgimCl/
[19:36:16] <topranks>	 sry 1002 is the active one right now 
[19:44:33] <topranks>	 there are some small increments there 
[19:46:06] <topranks>	 tbh not seeing any smoking gun as such 
[19:46:28] <hashar>	 topranks: is there anything in dmesg?
[19:46:45] <topranks>	 the system may just be approaching limits in terms of packet processing, not sure how much optimisation has been done on it for high throughput 
[19:46:53] <topranks>	 let me see
[19:47:07] <andrewbogott>	 topranks: do  you mean nothing to be concerned about, or just nothing to worry about?  I was only worrying because https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=cloudgw1002&var-datasource=thanos&var-cluster=wmcs&from=now-7d&to=now&viewPanel=11 shows new behavior as of a few days ago
[19:47:15] <topranks>	 yeah there are these 
[19:47:16] <topranks>	 [3354093.871467] bnxt_en 0000:65:00.0 enp101s0f0np0: Received firmware debug notification, data1: 0xdd03, data2: 0x0
[19:49:53] <hashar>	 earlier in October Arturo found: NETDEV WATCHDOG: enp101s0f0np0 (bnxt_en): transmit queue 4 timed out
[19:50:49] <topranks>	 yeah so I'd need to look in more detail but that could just be cpu unable to keep up with the arrival rate 
[19:51:11] <topranks>	 *however* it does look worse in recent days, and the throughput looks relatively the same as it was prior to that 
[19:51:41] <hashar>	 even though the CPU usage looks flat?
[19:57:22] <topranks>	 well that's kind of what I mean, it's dropping packets now but throughput seems much the same as does CPU 
[19:59:50] <hashar>	 the only thing I see is the softirq metric https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=cloudgw1002&var-datasource=thanos&var-cluster=wmcs&from=now-7d&to=now&viewPanel=3
[20:00:45] <hashar>	 which was at 20/25% this morning between 9:00 UTC and noon UTC 
[20:00:50] <hashar>	 with spikes here and there
[20:01:08] <hashar>	 but if all those come from the network card and are bound to a single CPU
[20:01:12] <hashar>	 and the machine has multiple CPU
[20:01:27] <hashar>	 then then 20%/25% would be a single CPU saturating on those soft IRQ requests
[20:01:33] <hashar>	 something like that
[20:03:16] <hashar>	 but that does not align with the build failure encountered by CI https://phabricator.wikimedia.org/P71234 
[20:03:17] <hashar>	 anyway
[20:03:38] <hashar>	 it is 9pm, it is about time to have dinner :]
[20:05:10] <topranks>	 yeah I gotta log off also 
[20:05:21] <topranks>	 usage definitely seems to be growing looking at past few weeks 
[20:05:45] <topranks>	 the ingress packets should be distributed to multiple cores, based on a hash of packet and the nic driver setup 
[20:05:54] <hashar>	 I wish we had the details of those softirqs
[20:06:03] <hashar>	 or some graph of the network card business/queues etc
[20:06:31] <hashar>	 thanks topranks ! 
[20:06:32] <topranks>	 but as you say whether or not things are optimally tuned for the number of cores and what CPU is handling is rx queue is the thing 
[20:06:40] <hashar>	 yeah
[20:06:41] <topranks>	 there are also NUMA considerations at this kind of bandwidth 
[20:06:45] <topranks>	 https://usercontent.irccloud-cdn.com/file/ISDY3Gzp/image.png
[20:07:00] <hashar>	 I remember in 2017 when b.black explained me a challenging issue with IRQ usage and network
[20:07:04] <topranks>	 I messed with the graph to get a better sense, but we have more spikes since previous weeks 
[20:07:07] <hashar>	 on some LVS machine that was saturating
[20:07:46] <topranks>	 yeah, softirq will max out due to packet processing 
[20:08:07] <hashar>	 and NUMA is the bus between the CPU cores ?
[20:08:15] <topranks>	 exactly yeah 
[20:08:23] <topranks>	 the NIC is connected to one NUMA node 
[20:08:25] <hashar>	 ahah
[20:08:34] <topranks>	 if a CPU on the other one reads/writes a packet it has to cross the "numa bridge"
[20:08:42] <topranks>	 which is fine at lower IO rates 
[20:08:45] <hashar>	 so that course Brandon gave me some years ago paid off in the future!
[20:08:51] <topranks>	 but as it rises that's an internal bottleneck 
[20:08:52] <topranks>	 haha yeah
[20:09:03] <hashar>	 I am so glad we have experts at the foundation
[20:09:08] <topranks>	 and you're fully correct the irq's may only look small, but if it's maxing one cpu we could be in trouble 
[20:09:13] <hashar>	 at most org, people would just spin up a larger AWS instance
[20:09:16] <topranks>	 as in.... the message you posted mentioned queue 4 
[20:09:19] <topranks>	 haha yeah 
[20:09:25] <topranks>	 this was very busy this morning 
[20:09:27] <topranks>	 https://usercontent.irccloud-cdn.com/file/illvm1wi/image.png
[20:09:35] <hashar>	 (or worse add a cronjob to reboot every day)
[20:09:56] <topranks>	 hahaha 
[20:10:08] <topranks>	 I also note if I look at 'htop' the conntrack is using a lot of CPU 
[20:10:25] <hashar>	 which dashboard is rendering those?
[20:10:44] <topranks>	 I was messing with the host dashboard but I don't want to save it 
[20:10:49] <hashar>	 ah ok
[20:11:11] <topranks>	 i basically mutliplied by 8 to see the bits/sec and changed a few view settings 
[20:11:27] <hashar>	 is conntrack involved for UDP packets as well?
[20:11:27] <andrewbogott>	 topranks, hashar, thanks for looking but also you should go eat dinner if there's no immediate cause for alarm :)
[20:11:37] <hashar>	 yeah true
[20:11:39] <hashar>	 you are wise
[20:11:54] <topranks>	 yeah it is - and in some ways worse for udp as they just sit there until the box hasn't seen a packet for X mins (configurable)
[20:12:31] <hashar>	 <INSERT THE EMOJI OF JAWS DROPPING OUT OF FEAR AND BASED ON A FAMOUS PAINTING>
[20:13:54] <hashar>	 https://fr.wikipedia.org/wiki/Le_Cri#/media/Fichier:Edvard_Munch,_1893,_The_Scream,_oil,_tempera_and_pastel_on_cardboard,_91_x_73_cm,_National_Gallery_of_Norway.jpg
[20:14:27] <topranks>	 lol
[20:14:41] <topranks>	 total conntracks through the box are relatively steady, despite the increased throughput 
[20:15:11] <topranks>	 I'll have a chat with arturo in the morning, one thing for certain we are at traffic levels where considering some of that tuning is probably needed 
[20:15:28] <topranks>	 though I can't for sure say what the cause is 
[20:15:45] <hashar>	 to be continued
[20:15:54] <hashar>	 meanwhile, I am going to have my crepes ( https://fr.wikipedia.org/wiki/Galette_de_sarrasin_(Haute-Bretagne)#/media/Fichier:Galette_de_sarrasin_compl%C3%A8te_bretonne.jpg )
[20:16:01] <hashar>	 it is about time. Thank you andrewbogott and topranks !
[20:16:10] <topranks>	 enjoy!
[20:17:17] * andrewbogott never totally convinced by buckwheat crêpes
[20:19:45] * andrewbogott should probably learn to like buckwheat in preparation for the coming famine
[21:39:57] <quiddity>	 Hiyo. I'm wondering if we need to have a Tech News (newsletter) entry, for the WMCS Network Problems incident? (especially if there are any lingering after-effects that tool-users might be experiencing,  or follow-up actions needed from tool-maintainers, beyond what you've gotten from the wikitech-l/cloud-announce emails.) 
[21:39:57] <quiddity>	 If you don't think an entry is needed, that's fine, just let me know. 
[21:39:57] <quiddity>	 If you do think an entry is needed, please help me by drafting some wording, either here or directly in https://meta.wikimedia.org/wiki/Tech/News/2024/49 -- It could be purely informational (e.g. "Last week, some Toolforge tools were unavailable for a few hours because of DNS resolution issues. The errors have been resolved and investigation is ongoing.") or could include follow-up/feedback requests ("If you notice any 
[21:39:57] <quiddity>	 DNS-related issues, please report them in [[phab:T380844|the Phabricator task]]"). Thanks!
[21:39:58] <stashbot>	 T380844: 2024-11-26 Toolforge DNS incident - https://phabricator.wikimedia.org/T380844
[21:44:17] <andrewbogott>	 quiddity: there aren't lingering after-effects as far as we know, so I don't think an entry is probably needed.
[21:44:44] <quiddity>	 Sounds good, thanks for confirming :)