[00:34:05] now at 3 hours of 5xx errors - can T442462 get attention / page? bblack herron [00:34:05] T442462: 2026-10-26 stream.wikimedia.org 5xx errors - https://phabricator.wikimedia.org/T442462 [01:38:40] thank you herron [01:38:43] thanks JJMC89 should be sorted now I've just put an update on the task [01:38:55] whoops updates crossed. no problem! [01:39:40] in case it helps, T427839 was also an OOM issue [01:39:41] T427839: 502/503 for mediawiki.page_change.v1 stream - https://phabricator.wikimedia.org/T427839 [01:43:42] ah good to know, worth a mention or add as related on the task for sure [05:40:10] so eventstreams have just paged again and i think it has been showing errors for a few hours [05:43:04] OOM errors? its doing better than before - my tool hasn't seen any 5xx errors since 03:43 [05:43:14] JJMC89: https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-3c-3e-origin-servers-overview?orgId=1&viewPanel=12&var-site=drmrs&var-cluster=text&var-origin=eventstreams.discovery.wmnet&from=now-24h&to=now&timezone=utc [05:43:27] i was trying to post that on the task but phab is also not helping [05:47:41] phab is having a bad time [05:48:15] yes I am aware [05:48:21] anyone knows how this https://phabricator.wikimedia.org/T442462#12475695 was done? [05:48:22] I guess I'm just luckily not the one getting them [05:50:22] JJMC89: I am getting them, but I am dealing with the eventstreams first [05:50:33] I'd guess phab is someone scrapping again [05:51:08] I meant I'm not seeing the 5xx from eventstreams right now - last one for me at 03:43 [05:51:37] JJMC89: my graph above shows them [05:51:40] and we got a p4ge [05:51:45] and I can see an increase again [05:53:12] it just auto resolved but I am not sure this is indeed resolved [06:17:09] seems the last bit is dc specific compared to before when it was all dcs [06:22:51] JJMC89: I am unfortunately a bit blind here because, we had no handover about this and the task also doesn't give much details on how this was solved other than increasing memory. The graph shows it is way better now, I've posted on the task that this will come back I am sure, but I don't want to phone herro.n as it is quite late for them and this doesn't seem to be on fire now [06:22:56] godog: any other idea? ^ [06:23:43] not really atm no, I silenced the page in drmrs only because as you said it is bound to come back while we debug [06:23:53] yeah [06:24:08] I also reached out to tchin on slack re: eventstream apply in SAL [06:24:29] I see no new commits on deployments-charts so I wonder what that was about [11:37:07] heads-up: we are trialing a new host-down alert. This will be used in parallel to the existing icinga one but we're hoping to move away from that one in time https://gerrit.wikimedia.org/r/c/operations/alerts/+/1351431 [11:37:21] Currently we use ping as the icinga classifier for a host being down [11:38:02] very cool, thank you for the heads up hnowlan [11:38:15] this alert uses an absence of prometheus metrics. This will hopefully give us some flexibility around grouping etc and silences which would have helped during some of the last big outages [11:38:42] for now nothing changes with the original alert but adding the new one might make for some alert noise [11:38:46] ack [11:39:10] hnowlan: that will be such a big step towards getting rid of icinga when it's done <3 [11:39:28] couldn't be a moment too soon, icinga's demise that is [11:39:38] we're getting closer :D [11:40:04] is that alert keeping in mind if prometheus is down/overloaded etc and cannot process metrics? [11:40:07] godog: *nods vigorously* [11:45:42] marostegui: if prometheus is down, that metric won't even be generated in most cases - the `up`ness comes from prometheus, not the agent. [11:45:51] But there's definitely some hardening we could add [11:46:27] we could stop the alert from firing on a certain percentage of hosts being down all at once [11:46:41] the external meta monitoring would catch prometheus being down before the host down alerts would fire though [12:04:52] There's a bunch of admin_ng changes pending on almost all clusters, I'm gonna go ahead and deploy them [12:07:13] (trying to clear out as much k8s noise as possible before the switchover) [15:03:03] We will be starting the depooling of eqiad as part of the DC switch over. This will take a little longer than normal due to an additional step [15:03:14] gl! [15:03:17] See: https://phabricator.wikimedia.org/T439675 [15:07:22] sukhe: we are getting errors on dns500X, which is expected we reckon ? [15:51:06] effie: that's probably because of the eqsin network issues I presume [15:51:09] both hosts? or just one? [15:51:32] it is ok just making sure [15:53:14] ok, please let me know if one of them is particularly unhappy [15:53:18] in theory we can depool both as well [15:53:27] traffic will be served by the ulsfo hosts [15:54:25] so far it is 5003 returning resouce unavailable [15:56:52] what is the full error? [15:57:10] resource temporary unavailable [16:00:13] and 5004 [16:00:53] effie: as part of which process though I guess? [16:01:05] yes yes [16:01:16] depooling A/A services [16:02:40] looking but I think it's pretty clear it's a connectivity thing but double checking [16:04:48] yeah they are both unhappy [16:04:50] hmmm [16:05:03] I am going to depool them in eqsin I guess [16:05:13] otherwise you will get all sorts of transient errors and inconsistencies [16:05:31] done [16:05:36] 12:05:29 <+logmsgbot> !log sukhe@puppetserver1001 conftool action : set/pooled=no; selector: dc=eqsin,cluster=dnsbox [16:09:29] Hello, apologies if this is not the right place but which SRE team should I contact (and how) if I am getting blocked from accessing Gerrit despite being a human who is just using standard Google Chrome? The error message I received mentioned bot-traffic@ and I did email it but it seems like that email address is used by Enterprise and intended for [16:09:30] use by people doing actual crawling. [16:10:32] are you blocked if logged in? [16:11:15] I can't login in now and probably wasn't either. [16:11:40] effie: let me know if you see any other issue [16:14:41] you are probably being caught by a filtering rule, this channel is a fine place to ask. [16:16:33] sukhe: the HE transport between eqsin and eqiad is having issues, I'm draining it - https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1359323 [16:16:44] not sure if relevant to your issues [16:16:52] XioNoX: yep, related I am guessing [16:17:09] I will pool eqsin once you drain this [16:17:20] for DNS [16:18:39] apergos: I doubt it's a filtering rule because I am getting blocked no matter what path I try to access (including the r/c//+/ path) [16:19:49] we have a range of rules, some not dependent on url paths for certain services [16:20:12] Oh [16:20:47] iirc someone got caught in a similar way yesterday, also for gerrit [16:21:00] sukhe: change pushed, let me know if it's better [16:21:28] Redmin: if you can give us the hex code in the error message you get, SREs can more easily find what rule you might be being impacted by [16:21:37] yeah looks all green [16:22:40] hnowlan: it's 9b7d648 [16:33:00] Redmin: can you try again? Are you still affected by the block? [16:33:48] jelto: Yep, it works now. Thanks! [16:34:16] great thanks for the quick confirmation and sorry for the trouble [16:35:55] No problem; thanks for fixing it. [16:46:35] eqiad depooling is done. We'll pick up next step of the switchover tomorrow. Thank you everyone :-) [16:46:56] thanks slyngs ! great job o> [16:51:14] \o/ ggwp! [16:52:01] \i/ [16:54:18] \m/ [17:27:19] well done Simon! [23:11:07] going to get the dog out for his afternoon stroll. back in a little.