[14:42:49] o/ [15:46:05] Trey314159: inflatador: I have two meetings conflicting with “Talk to Search”. I’d stick around and see if anyone joins and would hop on then. [15:46:54] pfischer yeah, I can join and see if anyone appears if that's what you'd like [15:46:59] pfischer: we can cover [15:52:32] Thanks! [19:57:19] ryankemper if you have time when you get in, could you take a look at https://gerrit.wikimedia.org/r/c/operations/alerts/+/1236766 ? I have to leave pairing early and wanted to cross off my list earlier if possible [19:58:11] Yeah can look in 30’ [20:14:25] eileen-m_: hey, we are mostly out on an offsite this week. I didn't realize that gets cutoff with the rendering, we should fix that. Try using the options `--reindexAndRemoveOk --indexIdentifier now` [20:16:22] alternatively, i tend to delete the whole mediawiki/elasticsearch instances and recreate it which creates a new index as part of mediawiki setup [20:16:49] i don't know if it's externally that useful, but i use the envs from our integration test runner: https://gitlab.wikimedia.org/repos/search-platform/cirrus-integration-test-runner [20:22:06] ryankemper cool, one more once you're in if you're able: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1236818 . This is related to a latency investigation [21:26:45] inflatador: https://gerrit.wikimedia.org/r/c/operations/alerts/+/1236766 do you know what makes the linting fail sometimes? that's the only thing i'm unsure about [21:29:46] ryankemper I'm not 100% sure, but my general take is that these kind of checks should be in CI instead of creating post-hoc alerts [21:30:45] I asked in 0lly IRC this morning and it's probably because the regex is matching for CODFW and EQIAD IPs and without those site tags, the alerts are also comparing against the other DCs where the labels don't exist [21:32:25] Zooming out a bit, we aren't getting value from these alert linting alerts, so I'm trying to shut them off [21:33:00] > I asked in 0lly IRC this morning and it's probably because the regex is matching for CODFW and EQIAD IPs and without those site tags, the alerts are also comparing against the other DCs where the labels don't exist [21:33:13] gotcha, yeah this is part of my motivation is trying to figure out if the site tags already addressed the issue without needing the pint [21:33:18] The root cause is less important than making sure the alerts go away for good [21:34:20] yeah i'll +1 [21:34:38] thanks, sorry if I sound edgy, I had this discussion once today already ;) [21:36:50] nah I didn't interpret it that way [21:36:55] i should catch up on the o11y context tho, one sec [21:48:47] np, my thought process is that the linting should be enforced in CI instead of silently creating noisy alerts that other teams have to respond to. I'm sure this wasn't anyone's intention and maybe it's the best bad option ATM. Regardless, I know for a fact these alerts work because I've seen them fire, so we're not losing anything [22:00:58] in this case it's probably hard for CI to know that the data only existed in eqiad/codfw, it's probably a lot more suited to finding errors in the definitions themselves [22:01:01] anyway +1'd the other patch too: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1236818 [22:01:19] i always have to refresh myself on the fact that `state: production` doesn't actually do anything scary for non-lvs-backed services :P [22:01:49] inflatador: feeding dog then joining pairing [22:01:57] ryankemper ACK, thanks!