[04:56:40] <marostegui>	 In 4 minutes we are starting the wikidata master switchover
[11:19:18] <cmjohnson1>	 Happy Reminder!  Today is PDU swap in B6
[12:40:59] <cmjohnson1>	 b6 pdu swap is over
[14:50:59] <ottomata>	 akosiaris:  look ok to you? https://gerrit.wikimedia.org/r/c/operations/dns/+/535221
[14:51:10] <ottomata>	 nodes have been reracked and we are renaming them.
[14:54:22] <akosiaris>	 why presto and not pesto? so I can at least dream of a nice spaghetti al pesto everytime I see those machines?
[14:54:39] <ottomata>	 al-pesto1001
[14:54:50] <akosiaris>	 there you go
[14:55:43] <ottomata>	 heheh :)
[14:55:57] <ottomata>	 presto-change-oh1001
[15:16:51] <jynus>	 did someone spot production impact?
[15:17:06] <andrewbogott>	 One of my hosts which alerted as down and back up is still claiming 43 days of uptime
[15:17:20] <andrewbogott>	 so I suspect monitoring issues unless someone has found a host that actually went down
[15:17:28] <jynus>	 I think all hosts down were only mgmt, only redundancy lost on actual hosts
[15:18:19] <jynus>	 andrewbogott: I don't think so, ipmi complained about lack of redundancy
[15:18:45] <jynus>	 hosts up again
[18:03:07] <ottomata>	 bblack:  o/  is there anything special I should know about decomming an LVS service?
[18:03:18] <ottomata>	 re T232122
[18:03:26] <stashbot>	 T232122: Decomission eventlogging-service-eventbus and clean up related configs and code - https://phabricator.wikimedia.org/T232122
[18:35:57] <bblack>	 ottomata: it's pretty straightforward.  the actual IPVS defs won't go away until a future manual cleanup or reboot, but that can wait
[18:36:37] <ottomata>	 so,  just remove stuff that I can from the service servers, and references in puppet?
[18:36:58] <bblack>	 yes basically
[18:37:19] <bblack>	 remove the LVS service before the service servers I guess, so we don't get alerts about all the healthchecks from LVS failing?
[18:38:12] <ottomata>	 hm ok
[18:38:22] <ottomata>	 from the hiera configs you mean, right?
[18:50:27] <ottomata>	 bblack:  if you have a moment: https://gerrit.wikimedia.org/r/c/operations/puppet/+/535669
[19:28:20] <bblack>	 ottomata: can you split discovery from the rest? I don't know if we've ever decommed a discovery service from discovery, but it may require some careful ordering with the parts in ops/dns repo as well
[20:17:26] <ottomata>	 ok
[20:17:35] <ottomata>	 shoudl discovery decom come first?
[20:17:37] <ottomata>	 bblack:  ?
[20:20:14] <bblack>	 ottomata: probably.  just split it off separately and I'll look at it in the AM, I'm about tapped out on complexity for the day :)
[20:33:53] <ottomata>	 ok thanks
[20:34:04] <ottomata>	 i am probably in the same state, will update patches in my morn tomorrow
[20:35:10] <XioNoX>	 ottomata: hijacking your conversation, could you check that everything is good with T232456 ? when you have some time
[21:20:53] <ottomata>	 hm XioNoX  i can't tell, archiva.wikimedia.org itself doesn't seem to be responding for me...
[21:23:07] <ottomata>	 XioNoX:  i didn't investigate this ticket at all before
[21:23:18] <ottomata>	 so I don't know if this was the status before
[21:23:26] <ottomata>	 but https://archiva.wikimedia.org should load
[21:23:50] <ottomata>	 it seems to not respond, just times out...
[21:29:36] <XioNoX>	 ottomata: doesn't seem to be related to that task
[21:35:10] <ottomata>	 huh, yeah i just restarted archiva service
[21:35:53] <ottomata>	 i think its working XioNoX
[21:36:03] <XioNoX>	 cool, thx!