[00:19:26] I talked through this with c.danis and we think the answer is, actually because this is an ingress service, nothing needs to be restarted anyplace (since no LVS service is affected) and the only effect of changing sites is to fix the probe. we'd have to fix the CNAME separately, if the CNAME were wrong, but we don't even have one for the staging service so there's nothing broken there. awesome [08:49:49] 06Traffic: the configuration file for check_rp_filter_disabled can be too large for icinga - https://phabricator.wikimedia.org/T443771#12526060 (10dkertesz) we're now collecting the `sysctl_net_ipv4_conf_rp_filter` metric on lvs nodes in relevant sites (eqiad, codfw) and we have [[ https://gerrit.wikimedia.org/r... [09:52:39] hey traffic team, we had an alert realting to dns7001 [09:54:03] it is not announcing any routes in BGP since ~15:30 yesterday? [09:54:05] https://w.wiki/gKdf [09:54:20] that's fine if it makes sense wanted to flag otherwise, I can create a task [09:58:43] actually this is true for dns7002 as well [09:58:46] mmh the timing seems to match sukh.e experiments "testing DNS admin cookbook changes" [09:58:53] which makes for a not-so-good-situation [09:59:12] we have no DNS now in magru, we are sending requests to codfw/eqiad (both internal and users connecting to ns2) [09:59:26] dkertesz: hmm ok [09:59:41] could it be as simple as he tested the depool option and forgot to re-enable? [09:59:59] I'll flag this to the team and try to take a look, but I'm probably too new to fix it 😅 [10:00:17] according to SAL it was re-pooled, but maybe the computers lied [10:01:03] checking [10:01:45] https://www.irccloud.com/pastebin/B4saHtSk/ [10:02:11] that's what happens when managers do technical stuff [10:02:13] anycast-healthchecker service (which controls whether the routes are announced) seems to say it is configured as down which would explain it [10:02:22] vgutierrez: you said it :D [10:05:24] jokes aside [10:05:27] 15:28 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified] [10:06:02] bird conf on these 2 hosts misses some ips [10:06:03] so maybe a bug in the cookbook? it didn't repool when it should have? [10:06:20] fabfur: how do you mean? [10:06:46] I'm just naively checked /etc/bird/anycast-prefixes.conf [10:06:53] yeah that makes sense [10:07:28] they aren't there because the anycast-healthchecker service isn't putting them there, in its logs (above) it's reporting the reason as the host is not pooled for dns [10:07:45] yep [10:07:52] so if the confctl (?) status of pooling for the host changes it should pick it up and add them back [10:08:03] wondering if re-running the repool cookbook helps [10:08:17] can't hurt I guess [10:08:59] no, the cookbook doesn't repool and it's in the instructions :) [10:09:08] https://www.irccloud.com/pastebin/n8TrS5P3/ [10:09:52] From Sukhbir's emails: [10:09:52] "Note that you have to pool the hosts manually after you pool a site since the cookbook does not (cannot) do that for you. (The cookbook reminds you to do that as part of the depool operation.)" [10:10:00] confctl confirms no host is pooled on magru [10:10:07] I'll proceed repooling them manually? [10:10:17] ssh to the host then run `pool` on them I guess [10:10:29] I'll do w/ confctl from cumin [10:10:40] XioNoX: what do you mean? [10:10:50] the cookboiok help says: "cookbook sre.dns.admin pool magru # [pools magru for everything]" [10:11:04] * volans|off curious why the cookbook (cannot) do that, but I'm not here today :D [10:11:05] topranks: not for DNS [10:11:13] for *almost* everything :) [10:11:32] topranks: see Sukhbir's email [10:11:57] alright I believe you [10:11:57] {{done}} [10:12:38] so he felt on his own documented gotcha? [10:12:42] confirmed with bird configuration on the 2 boxes [10:12:44] lol [10:12:49] https://www.irccloud.com/pastebin/SDcZkkwp/ [10:12:50] https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1335049 [10:13:07] for the patch iself volans|reallyoffandabsolutelynothere [10:13:24] hahahahaha I like the new nick [10:13:55] "we do not pool the host again as we have no way of discovering why the host was depooled and if we should pool it again. That's for the operator to do" [10:14:10] I really can't understand why someone would add a --depool-dns-hosts toggle with no equivalent --pool-dns-hosts one [10:14:17] I can only refer to Valentin's first comment above [10:14:53] "we have no way of discovering why the host was depooled and if we should pool it again" lol [10:15:00] yeah, I'd rather have a --pool-dns-hosts-i-know-what-im-doing [10:15:44] that's the --bblack flag [10:16:58] I'm wondering if the cookbook shouldn't warn the user *before* the depool if the host is already depooled [10:17:34] like "host is already depooled, you might not want to repool it after the maintenance" kind of message [10:17:56] I can't see how we trust someone enough to switch something off, but then say they can't switch it back on again [10:18:05] but the best would be like the puppet disable, to tie a disable to a "unique identifier" [10:18:07] I'd have thought the knowledge requirements were similar for both [10:58:59] 06Traffic, 10FY26-27 ST6.4 - Logged-in performance, 06MediaWiki-Core-Platform-Team: Implement Varnish URL Rewriting for Internal ESI Fragment Endpoints - https://phabricator.wikimedia.org/T443703#12526542 (10Vgutierrez) 05Open→03In progress a:03Vgutierrez [11:04:47] 06Traffic, 10FY26-27 ST6.4 - Logged-in performance, 06MediaWiki-Core-Platform-Team: Implement Varnish URL rewriting for internal ESI fragment endpoint - https://phabricator.wikimedia.org/T443703#12526556 (10Krinkle) [11:25:38] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12526580 (10cmooney) [11:30:02] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12526611 (10cmooney) [11:30:09] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad row A/B switch refresh configuration - https://phabricator.wikimedia.org/T439535#12526613 (10cmooney) [11:41:14] 10netops, 06Infrastructure-Foundations: Netbox: add validator to prevent interfaces set to access mode with no untagged_vlan - https://phabricator.wikimedia.org/T443893#12526650 (10cmooney) I tidied up a few bits there to have things consistent. I think for now the way to proceed is to ignore the virtual ones... [11:51:22] topranks: I realize the disparity and I am not happy about it [11:51:29] but the main reason we cannot pool hosts back is what vg pointed out in the comments [11:51:45] we have no way to tie a "state" of a DNS host in confctl or the cookbook [11:52:03] which means if a DNS host was depooled for a valid/prior reason to the cookbook run, we have no way of tying it to the DNS admin cookbok [11:52:24] and therefore if we allow the user to pool the DNS hosts, they may pool a host that was depooled for another reason [11:52:41] I couldn't think of a way to fix that but maybe there is, so let's discuss that [11:52:44] in the immediate case: [11:53:32] - I intentionally did not pool the hosts immediately after the cookbook -- thus ignoring my own warning -- as magru DNS hosts are running pdns-rec 5.4.x (the only two in the fleet) [11:54:05] there is a message in the journal about caching DNSSEC related records when obviously we don't enable it, so I wanted to explore that a bit further [11:54:15] and then yes, I absolutely forgot to repool them later after I was done [11:54:46] so I think I am having my own doubts with this approach in the absence of sufficient alerting [11:56:18] we already alert when confctl says a host is pooled but we are not advertising the VIP. but that of course did not kick in because the host was depooled [11:56:43] I think we can probably alert for 0 rps or something after an hour period, that's the only thing that comes to my mind right now [11:57:08] anyway, back to day off for me, and sorry for the trouble folks. happy to discuss more next week :) [11:58:12] sukhe: we had alerting on the networking side at least, that's how we caught it, (and a junos bug preventing us to move that alerting to the host itself in prometheus, but at least netops gets it) [11:59:09] sukhe: and I also suggest that the cookbook alerts if a dns node is already depooled before running the cookbook, to help the operator to know if they should repool it or not [12:04:41] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Netbox: add validator to prevent interfaces set to access mode with no untagged_vlan - https://phabricator.wikimedia.org/T443893#12526734 (10ayounsi) 05Open→03Resolved a:03ayounsi Deployed for physical interfaces [12:31:48] "if we allow the user to pool the DNS hosts, they may pool a host that was depooled for another reason" - sure [12:32:03] but you allow the user to depool the DNS hosts - they do that to all of them and wikipedia is offline [12:35:58] I get the problem - it was already depooled for some reason and an operator repools when they shouldn't [12:37:09] but just making it _slightly harder_ to repool doesn't prevent them doing so, I don't think this solves the issue [12:37:24] to me it makes more sense either the cookbook can set/unset this property. or it can't affect it. [13:04:57] 07:32:03 < topranks> but you allow the user to depool the DNS hosts - they do that to all of them and wikipedia is offline [13:05:05] fwiw, this can't happen through the cookbook at least [13:05:21] haha ok fair enough [13:05:24] the concept of depooling a DNS host is tied to depooling of sites, and there are safeguards in the cookbook to prevent that [13:05:40] https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/dns/admin.py#L88 [13:05:43] 07:37:24 < topranks> to me it makes more sense either the cookbook can set/unset this property. or it can't affect it. [13:05:52] this is the main point and yeah, I agree, it deserves a revisit [13:06:02] 06:59:09 < XioNoX> sukhe: and I also suggest that the cookbook alerts if a dns node is already depooled before running the cookbook, to help the operator to know if they should repool it or not [13:06:23] this was discussed -- I think my concern at least is that the admin cookbook is critical enough that when the operator is running it, they shouldn't have to figure this out themselves [13:06:40] but yeah, I won't revert this for now since --depool-dns-hosts is disabled by default [13:06:54] let's discuss it next week, if we can't find something suitable, I will revert [13:07:15] I think Arzhel’s suggestion is a simple and good one [13:52:46] 06Traffic, 06Data-Persistence, 06MediaWiki-Media-Platform-Team, 13Patch-For-Review, 07User-notice: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#12527151 (10Ladsgroup) There are a lot more follow up work like always but the main piece is done. Just on... [13:52:53] 06Traffic, 06Data-Persistence, 06MediaWiki-Media-Platform-Team, 13Patch-For-Review, 07User-notice: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#12527152 (10Ladsgroup) 05In progress→03Resolved a:03Ladsgroup [18:33:34] 06Traffic, 13Patch-For-Review: Fetching mediawiki GPG keys fail with error "No data" due to User-Agent requirement - https://phabricator.wikimedia.org/T405165#12528696 (10matmarex) I encountered this today while preparing the next MW release candidate. My idea was to change the recommended command: https://ger... [18:44:06] 06Traffic, 13Patch-For-Review: Fetching mediawiki GPG keys fail with error "No data" due to User-Agent requirement - https://phabricator.wikimedia.org/T405165#12528716 (10ArielGlenn) version 2.5.9 of GnuPG added the user-agent option with the default string of GnuPG/2.6, see https://github.com/gpg/gnupg/blob/7...