[06:09:09] hello people, one qs - almost every (EU) morning around this time I see a lot of unknowns in icinga for check_trafficserver_log_fifo_purge_backend [06:09:14] is it known? [06:09:42] for most of the cp, no dc specific afaics [07:55:22] elukey: yeah, those seem to be all upload nodes. We have changed the icinga check from "run lsof and see if the pipe is opened properly" to "read from the socket and see if logging actually works". On upload (luckily) we do not have an insane amount of purges, so there are times when the check gives up trying to read a purge and that results in UNKNOWNs [07:57:05] we could play with the timeout, but that just means changing the amount of 'purge silence' time after which unknowns happen [07:58:39] or we could stop monitoring purge logging and be satisfied with checking regular logs [08:20:24] ema: ahhh okok.. I asked because from an "ignorant" point of view seeing all those unknows raises some questions [08:54:24] I agree, they also clutter the UI [11:21:19] 10netops, 10Operations: OSPF metrics - https://phabricator.wikimedia.org/T200277 (10faidon) Interesting idea! Couple of notes: - What do you mean by "virtual links" and Netbox not supporting them? Is that VLANs for our transports over the PtMP VPLS? - What do you envision the difference to be between "primary"... [11:45:15] elukey, XioNoX: fixed :) [11:46:35] thx! [12:24:11] <3 [12:25:15] 10Traffic, 10Operations, 10Wikimedia-Incident: cp1083: ats-tls and varnish-fe crashed due to insufficient memory - https://phabricator.wikimedia.org/T241593 (10fgiunchedi) Untagging observability for now since there doesn't seem to be any action [12:45:40] ema: suggestion for rewording "Scripted requests from your IP have been blocked, please contact... and " to "See ... or [or maybe otherwise]" to avoid extra emails? [12:46:24] assuming those are rate-based, wording seems to suggest human intervention is needed, which probably isn't in most cases [12:47:40] jynus: you're right [12:48:17] maybe "blocked automatically", but that may be not always the case or desired [12:48:36] just for my understanding, those are just kicking rate-limits, right [12:48:50] with no user agent, or no good one, right? [12:49:32] jynus: it depends. "python-requests|Go-http-client/2.0|CInetHttp/1.0" are just blocked with 403 without any rate limiting [12:49:38] ok ok [12:49:44] so it is more complicated than that [12:49:52] will let you answer when people write [12:51:12] or point to a wiki page with the details? [12:51:24] he does that already [12:51:41] but the msg suggest to both read the wiki AND contact by email [12:51:49] yeah, which isn't necessary actually [12:51:56] when in most cases, may be just overhead for us [12:52:11] so maybe put the email in the wiki page so we're sure people read it? :) [12:52:12] I think an email is nice [12:52:16] XioNoX: ha ha [12:52:36] I still think the email is nice, just not as the first/mandatory option 0:-D [12:53:10] "if it works again, don't emails us" :-D [12:53:18] how about this? [12:53:20] > Scripted requests from your IP have been blocked, please see https://meta.wikimedia.org/wiki/User-Agent_policy. In case of further questions, please contact noc@wikimedia.org. [12:53:25] +1 [12:53:47] some people will email anyway, but you cannot do anything about it [12:54:59] yeah but at least we probably will receive less emails from those whose bot works fine :D [12:56:42] it is funny because I had to ask people to not copy and paste from a browser on a forum [12:56:48] browser UA [12:57:10] they didn't do it with malign intent, just because they didn't have a lot of understanding on how that worked [13:31:24] have them mail OTRS ? [14:41:55] that's a good idea I think as long as otrs people have a good runbook associated to it [15:13:32] 10netops, 10Operations, 10observability: replace check_ripe_atlas Python script with a check_prometheus backed by atlasexporter data - https://phabricator.wikimedia.org/T251155 (10CDanis) [15:13:45] 10netops, 10Operations, 10observability: replace check_ripe_atlas Python script with a check_prometheus backed by atlasexporter data - https://phabricator.wikimedia.org/T251155 (10CDanis) p:05Triage→03Low [15:14:00] 10netops, 10Operations, 10observability: replace check_ripe_atlas Python script with a check_prometheus backed by atlasexporter data - https://phabricator.wikimedia.org/T251155 (10CDanis) [15:16:09] 10netops, 10Operations, 10observability: add traceroute measurements to RIPE Atlas prometheus data - https://phabricator.wikimedia.org/T251156 (10CDanis) p:05Triage→03Low [15:16:13] 10netops, 10Operations, 10observability: add traceroute measurements to RIPE Atlas prometheus data - https://phabricator.wikimedia.org/T251156 (10CDanis) [16:54:27] 10Traffic, 10Wikimedia-Apache-configuration, 10Operations: Stop advertising webmaster@wikimedia.org in apache configs - https://phabricator.wikimedia.org/T251005 (10crusnov) p:05Triage→03Medium [16:57:59] 10Traffic, 10Wikimedia-Apache-configuration, 10Operations: Stop advertising webmaster@wikimedia.org in apache configs - https://phabricator.wikimedia.org/T251005 (10Dzahn) Should be replaced with noc@wikimedia.org. The ones above are the special cases. Almost everything else uses noc@ as ServerAdmin if you... [18:18:39] 10Traffic, 10Operations: Statistics on a CN banner - https://phabricator.wikimedia.org/T251177 (10Ciell) [18:26:53] 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10Patch-For-Review: Stop advertising webmaster@wikimedia.org in apache configs - https://phabricator.wikimedia.org/T251005 (10Reedy) >>! In T251005#6086077, @Dzahn wrote: > .. except the apache and httpd modules are also used in cloud VPS projects... [19:44:01] 10Traffic, 10MediaWiki-Cache, 10Operations, 10serviceops, and 4 others: Stop sending purges for `action=history` for linked pages. - https://phabricator.wikimedia.org/T250261 (10Krinkle) a:05Krinkle→03daniel [20:34:40] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Move netflow to TLS encryption/authentication via librdkafka - https://phabricator.wikimedia.org/T248980 (10Nuria) 05Open→03Resolved