[07:32:25] taavi: about? [07:32:41] Can you (or someone else?) give me the access again to approve toolforge requests? [07:40:31] cc dcaro lucaswerkmeister (maybe) [07:40:37] based on https://toolsadmin.wikimedia.org/tools/id/admin [08:19:02] Reedy: hi! when did you lose your access? [08:30:26] dcaro: 1 year 2 months ago apparently :P [08:32:01] I’m guessing https://phabricator.wikimedia.org/T249774 more or less covers the procedural side? [08:32:13] did you lose it for any particular reason? ^^ [08:32:28] Probably because I hadn't used it in a while at that time ;) [08:36:52] did someone reach out before removing you or something? Seems weird that we'd remove you without notice [08:38:00] Taavi probably did at the time, but it's long enough ago that I probably wouldn't remember [08:38:20] done [08:38:33] (hopefully) [08:40:39] yup, awesome thanks [08:40:41] thanks lucaswerkmeister, I confirm I see Reedy in the list [08:40:41] morning [08:43:04] Reedy: you are not part of the wmcs-roots though (ssh access to cloud vps nodes, that as I understand, the above task granted you), is that something you want restored too? [08:47:30] Not specifically something I need atm [08:47:44] Basically I just need to be able to approve people to get toolforge access for the team challenges for the wikimania [08:47:46] ack [08:47:48] wikimania event [09:43:30] lucaswerkmeister: if you're access please do ot properly following https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolforge_roots_and_Toolforge_admins instead of only partially in a way that leaves the different list with an inconsistent list of people [10:48:08] * dcaro lunch [12:08:16] toolforge is down? [12:09:03] dcaro: are you using wikidough as a dns resolver by any chance? [12:09:48] nope xd [12:10:05] haproxies seem unstable [12:10:26] huh [12:10:33] (alerts + requests take some time to load) [12:10:37] requests seem ok now though [12:10:53] oh, no slow again [12:11:27] I am and can't resolve our DNS names via it but it works via any other resolver or querying our name servers directly [12:14:32] I think it's not DNS resolving (not directly though, my laptop resolves ok) [13:46:23] taavi: this change works but I'm interested in if you have a different way you'd like it done https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/335 (also, that's why tofu-infra is alerting at the moment) [13:52:43] andrewbogott: sigh.. should've thought of that previously. [13:53:17] I went through a brief period of "we should just have two repos" before writing that change. [13:53:42] so probably we should plumb the config values as variables and declare the module in the root main.tf or something [13:55:34] ok, so that would look like the version of the mr from a few versions ago [14:21:00] taavi: this what you were thinking? https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/336 [14:21:38] no [14:21:51] again, the code under modules/.../ should not have knowledge of this "datacenter" concept [14:22:02] or "region" or whatever we are calling it today [14:23:21] ... [14:30:49] :wikihug: xd [14:37:55] taavi: ok, this? https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/336/diffs [14:44:08] no, this is what I had in mind: https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/new/diffs?merge_request%5Bsource_branch%5D=taavi%2Ffix-magnum-templates [14:55:53] ...I see that that resembles the previous pattern in the code but I also find it almost impossible to follow. So we're defining modules, but those modules are defined in /resources, and those modules don't actually implement the templates but only define variables, and are then dereferenced when we actually define the templates. Is that about right? [14:57:30] yes, main thing i'm trying to avoid is having multiple different patterns of doing things in the same repo because that'll be just even more confusing [14:57:56] that's reasonable. But just for argument's sake... is the existing pattern a good one and I just need to get used to it? [14:59:19] I would describe it as okay, it works and I can't immediately think of something that's clearly better but I don't love it either [14:59:26] ok [14:59:49] I think I would accept some cruder branching and duplication in order avoid the confusion but I'm sure not going to rewrite things today [15:00:02] Shall I go ahead and make a mr out of your branch? [15:00:15] sure [15:10:17] godog: regarding the wikitech-static monitoring... your changes are a good way to make sure that we're running a recent container build but iirc we previously monitored whether the actual content in the container is up to date. Will we still be doing that? (Or am I misremembering and we already don't do that anymore?) [15:12:01] andrewbogott: my understanding is that if content can't be acquired the container fails to build and therefore the metric/timestamp is not updated, in other words the idea is to cover the pipeline end to end [15:12:51] Hm... yes, that seems right. I'm trying to think if there's any failure case that would result in a new container with old content. [15:13:06] I can't really think of one other than some kind of phantom cache which we probably don't need to monitor for :) [15:13:24] ok! [15:14:05] In general I like the idea of monitoring the thing that it's actually /for/ rather than just monitoring that its inner workings are still working. But... that's just a gut feeling, I can't think of a good reason in this case. [15:28:21] we can also extend the same idea with more metrics, e.g. last updated time of a specific page, number of pages, etc [20:18:18] andrewbogott: on bookworm and newer, systemd-networkd is the DHCP client [20:19:08] yeah, I'm gradually discovering that [20:20:10] and I guess it could be that the reason starting dhcpd fixed my problem isn't that systemd-networkd was asleep at the switch but just that it was waiting for a lease to expire... [20:20:16] in which case the VM would've healed itself in <24 hours [20:23:04] no, systemd-networkd is _the_ DHCP client, there should not be a separate process or server or package or whatever doing it [20:23:21] which VM was this on? [20:25:03] andrewbogott: I am starting to suspect you're seeing the IPv4 equivalent of T392611 for whatever reason [20:25:04] T392611: VMs with ferm host-level firewall do not permit DHCPv6 responses - https://phabricator.wikimedia.org/T392611 [20:25:19] Let [20:25:48] taavi: gitlab-1002 was the host where problems were seen today [20:26:29] Yeah, it had no IP assigned and no network available. Starting dhclient (which is apparently still installed everywhere despite being unused) caused it to recover [20:26:44] I now (as of 10 minutes ago) think that probably systemd-networkd would've done the same thing eventually [20:26:56] but maybe not until the lease expired which would've been a while [20:27:10] And we remain with 0 idea of why the VM forgot its IP [20:27:50] taavi: if you want to find the explanation that I could not find in the logs, it's gitlab-1002.devtools.eqiad1.wikimedia.cloud [20:27:52] well, at least its firewall is dropping inbound DHCP packets https://phabricator.wikimedia.org/P95029 [20:28:47] (why is this suddenly a problem only now??) [20:29:14] oooh does systemd-networkd use ipv6 for the request and dhclient use ipv4? [20:29:25] no [20:30:11] maybe something in the latest round of debian patches changed behavior? [20:31:13] taavi: then why could dhclient do what systemd-networkd could not? Or am I misunderstanding what you're saying? [20:31:31] according to the apt history file, networkd or any other systemd component hasn't gotten updates in the last few weeks [20:32:21] andrewbogott: when you explicitely started a DHCP client, it presumably sent out DISCOVER packets which got saved in the conntrack table and so allowed the responses past that firewall [20:32:34] I would imagine restarting systemd-networkd would do the same [20:33:27] would systemd-networkd also reach out after the dhcp lease expires? [20:33:47] it should [20:33:58] ok. So /probably/ this would have healed after a while. [20:34:17] But the firewall thing doesn't explain why it would forget its IP does it? [20:35:19] "firewall blocking DHCP responses from arriving" seems like a good explanation for a host losing its dhcp-assigned address to me [20:37:27] sorry, you're going to have to walk me through this. [20:37:30] We have an IP [20:37:33] Things are working. [20:37:36] What happens next? [20:38:05] something happens with the DHCP renewal [20:38:07] lease expires [20:38:17] --> we no longer have an IP [20:38:43] So you think systemd-networkd first turns off the network before trying to renew the lease? [20:39:20] And then can't renew the lease [20:40:07] I agree that this seems suspicious! But I'm back to my previous question "why could dhclient do what systemd-networkd could not" [20:40:19] Are we assuming that systemd-networkd can or can't renew leases? [20:40:24] Or that it sometimes can and sometimes can't? [20:40:37] I think systemd-networkd removes the IP address from the interface when it considers the lease for that address to be expired [20:41:01] ok. That's reasonable behavior. [20:41:48] dhclient works because you explicitely trigger a request for a new lease and the firewall sees the outgoing request packet [20:42:22] Great! This is what I'm misunderstanding. when systemd-networkd renews a lease... it /doesn't/ do that? [20:42:29] It doesn't request a new lease? [20:43:10] Is renewing a lease fundamentally different from requesting a new lease, from a network standpoint? [20:43:57] (btw, log shows a lot of "systemd-networkd-wait-online[3987846]: Timeout occurred while waiting for network connectivity." which fits) [20:45:06] requesting a completely new lease involves first sending a DISCOVER to find the server and then a REQUEST to get an address, a renewal is just a REQUEST [20:46:29] also the initial request is done using 0.0.0.0/255.255.255.255 addresses, while renewals can (but AIUI don't always) use the normally assigned unicast addresses [20:48:11] ok. I still only 80% understand but I'm convinced. [20:48:19] So that still leaves us with 'why is this only happening now' [20:50:37] the other thing in the logs I see is this https://phabricator.wikimedia.org/P95030 [20:50:43] it tried to do _something_ yesterday and failed [20:50:48] what that something was, I've no clue [20:51:49] and since then, the interface is in 'State: routable (failed)' and I wouldn't be surprised if that 'failed' means it's not processing renewals properly [20:52:29] if there's a reliable way to reproduce this, or even a VM where it happens semi-regularly, running networkd with debug logging enabled would be my next suggestion [20:52:45] It has the ferm rule though &RD_SERVICE(udp, 546, (fe80::/10), (fe80::/10)); [20:53:02] We don't have a way to reproduce it so far, but I'm watching to see if/when it happens again. [20:54:26] fe80::/10 is an ipv6 rule, doesn't explain v4 addressing [20:55:29] I thought we were still talking about T392611 [20:55:31] T392611: VMs with ferm host-level firewall do not permit DHCPv6 responses - https://phabricator.wikimedia.org/T392611 [20:56:30] I said that maybe we're seeing a similar issue of that but for IPv4 DHCP renewals, not that we're seeing a regression of that [20:57:40] ok [20:59:12] but at this point I think https://phabricator.wikimedia.org/P95030 is the more interesting mystery since that happens first and would explain the lack of renewal attempts (which then explains the rest of what we're seeing) [21:06:26] check out logging-logstash-04.logging.eqiad1.wikimedia.cloud, we might have a repeat underway there [21:07:50] andrewbogott: that host has both a v4 and a v6 address, did you do something to it already? [21:08:14] I rebooted it a couple of days ago, nothing since [21:09:11] I think that one is in the state where systemd-networkd can't renew, but the ipv4 hasn't actually expired yet [21:09:22] well, that's seemingly also timed out doing something and so probably about to fail? https://phabricator.wikimedia.org/P95031 [21:09:31] right, exactly [21:09:37] it failed a few days ago, and apparently is about to do so again [21:09:55] huh [21:10:06] so we have a VM where this happens regularly [21:10:07] I'm doing a fleet-wide search for ''Could not set route: Connection timed out" to see if it's happening all over the place :/ [21:10:33] (preview answer: not all over the place) [21:10:57] andrewbogott: I'm going to restart systemd-networkd there but with SYSTEMD_LOG_LEVEL=debug to have more information what exactly it times out doing [21:11:09] sgtm [21:11:50] done [21:13:48] (for reference: systemctl edit systemd-networkd.service; add a 'Environment=SYSTEMD_LOG_LEVEL=debug' in the [Service] block) [21:18:05] Am I reading the logs correctly that after the restart it is getting dhcp updates again? So we need to wait for to happen again? [21:19:19] yeah [21:20:34] a bit of duckduckgo-ing finds https://github.com/systemd/systemd/issues/25441 which might be our [21:23:09] They're mostly talking about an overloaded client, not a slow server -- that's not what I expected from the topic [21:24:48] the error in that issue matches exactly with ours, https://phabricator.wikimedia.org/P95030 [21:25:01] anyway, I guess the time will tell assuming logging-logstash-04 fails again [21:25:36] yeah, it's a close fit. [21:26:02] And the solutions proposed are somewhat discouraging. "Never let your system get busy" or "just check if the network is broken and restart it" [21:29:01] https://github.com/systemd/systemd/pull/29390 is in systemd v256 and newer only sadly (bullseye has v252), but bumping the timeout from 25 seconds to, say, an hour would probably make the problem go away [21:29:19] s/bullseye/bookworm/g [21:29:35] anyway, I really should not be here at this time, ttyl [21:29:44] * andrewbogott waves