[08:50:51] Hi folks - I'm in the middle of a reimage and move-vlan for T421719 and the DNS update is failing - zone_validator is failing to find /tmp/dns-check.o_2fil3o/zones/netbox/0.0.e.f.3.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa (obviously a different tmppath on each DNS system). [08:50:52] T421719: Data persistance: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421719 [08:51:43] Emperor: looking [08:53:30] XioNoX: thanks. FWIW the host is ms-be1082, v6 moving from 2620:0:861:103:10:64:32:52 to 2620:0:861:135:10:64:175:12 (I think from scrollback) [08:53:43] Emperor: it's absolutely not related to you [08:53:51] That's a relief :) [08:56:06] https://netbox.wikimedia.org/extras/changelog/?request_id=33800599-ee1c-4861-8952-189f8d6c9d4a deleted the IPs from netbox so we need https://gerrit.wikimedia.org/r/c/operations/dns/+/1288473 to go too [08:59:15] same with https://netbox.wikimedia.org/extras/changelog/?request_id=6896b470-bb3a-4a80-adcc-c7cb07534d12 [08:59:36] Emperor: do you want to give a quick review on https://gerrit.wikimedia.org/r/c/operations/dns/+/1288473 ? [09:01:13] LGTM, +1 [09:04:22] Emperor: all deployed [09:04:36] dunno if you can resume your cookbook or if you need to start it over or what [09:05:56] cezmunsta: Am I Ok to merge your puppet-change? `mariadb: Decommission db2152 (99c92299e8)` [09:06:07] XioNoX: I'll have a go at retrying the failed step [09:06:31] btullis: please can you hold on a moment, I am just waiting for something to finish running? [09:06:40] Ack. Will do. [09:06:44] ty [09:08:17] XioNoX: VLAN move went through OK, thanks for your help :) [09:08:24] nice! [09:09:03] btullis: OK, you can go ahead now, sorry for the holdup [09:09:24] Done. Thanks. [13:34:37] brouberol: o/ noticed https://gerrit.wikimedia.org/r/c/operations/puppet/+/1270771 (only now) and am a bit confused by your commit message. the last octet is different between the service ipv4 and the host ipv4 as well.. which one is that rule intended to catch? or both? [13:36:04] I think I was tired and somehow missed out on the fact that these IPs were in fact different. Let me re-read what I wrote (which is wrong) [13:37:38] yep, ok, so dumps.wikimedia.org is now resolving to dumps-lb instead of directly resolving to the hosts IPs. IIRC there were IPs on both sets that only differed by a single number and I missed it, so my brain came up with a convoluted justification [13:38:13] I think we can clean this by a) removing the comment and b) removing the hosts IPs addresses from the external service policy [13:40:14] and by "we" I mean "I" [13:40:40] seems reasonable [14:11:05] oncallers: I'm going to do some Cassandra restarts on sessionstore, no impact expected. [14:47:20] and {{done}} [17:15:52] Our emails to yahoo.com are getting bounced, due to an apparent high spam rate, if someone has a moment could they review this pull request, https://gerrit.wikimedia.org/r/c/operations/dns/+/1288909?tab=comments [17:16:12] this verifies the wikimedia.org domain for yahoo's spam monitoring tool