[08:42:55] <jynus>	 for what I see, linux 4.19.181-1 update is a regular bugfixing upgrade? https://metadata.ftp-master.debian.org/changelogs//main/l/linux-signed-amd64/linux-signed-amd64_4.19.181+1_changelog
[09:17:02] <ema>	 jynus: yeah, it's the LTS kernel 4.19.181 (not really the latest and greatest, which would be 4.19.183)
[09:18:39] <jynus>	 that's ok, I am just happy it is not a "remote vuln fix" kind of update :-)
[09:19:12] <ema>	 indeed! no, it's not :)
[12:03:37] <kormat>	 do we keep syslog/systemd logs longterm anywhere?
[12:03:48] <kormat>	 i'm interested in logs for a given machine from jan 21
[12:05:23] <kormat>	 ah hah. `centrallog1001:/srv/syslog/`
[12:05:57] <godog>	 yeah that's it! jan is definitely still there
[12:07:00] <volans>	 kormat: they are in kibana too (syslog dashboard) you can filter by host and by absolute dates
[12:07:13] <kormat>	 volans: i've never dared to try to _use_ kibana
[12:08:43] <godog>	 it isn't so bad, or maybe it is already too late for me and I've been assimilated
[12:17:32] <kormat>	 godog: the ui is impenetrable to me
[12:17:44] <kormat>	 e.g. i've managed to select the host and date range i want
[12:17:54] <kormat>	 but i cannot figure out how to search for messages containing 'mysqld'
[12:18:09] <kormat>	 add filter -> message -> ... there's no "contains" option
[12:20:15] <godog>	 kormat: mhh my next try would be sth like *mysqld* in the top bar, however not all syslog messages are in kibana/logstash
[12:21:06] <kormat>	 'no results found' https://logstash.wikimedia.org/goto/dc94071c79add1897c0443f20fd0f0c4
[12:21:29] <kormat>	 oh, great. it mangled my time window
[12:21:43] <godog>	 yeah I don't think we have mysqld logs in there
[12:22:38] <kormat>	 godog: so we send syslog to ELK, but _filter out some services_?
[12:22:53] <kormat>	 (because on debian /var/log/syslog contains the mysqld messages)
[12:23:35] <godog>	 the other way around, only some "program"s from the syslog firehose are sent to ELK
[12:23:49] <godog>	 centrallog has the full firehose though
[12:24:43] <kormat>	 where is this filtering done?
[12:25:49] <godog>	 by the host's rsyslog via 'lookup_table_output.json'
[12:29:19] <godog>	 FWIW the "syslog experience" now is obviously far from ideal, there's sort of a tracking task at https://phabricator.wikimedia.org/T254605
[12:42:02] <kormat>	 godog: thanks for the info 👍
[13:53:28] <Majavah>	 one more cloud-specific patch, https://gerrit.wikimedia.org/r/675802 . this one affects non-deployment-prep servers but I think removing that profile is safe since it doesn't remove the LVM volume from old VMs
[14:48:08] <Majavah>	 in addition to that, got a few beta-only hiera patches: https://gerrit.wikimedia.org/r/c/operations/puppet/+/675807 https://gerrit.wikimedia.org/r/c/operations/puppet/+/675814 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/675815
[14:51:18] <jynus>	 kormat, Andrew B. proposed a patch recently and this were the current blockers: https://gerrit.wikimedia.org/r/c/operations/puppet/+/664678/1#message-5ec6d5c03f25b5e3b383d5e79515bb3f23dc80c4
[14:52:04] <kormat>	 jynus: 👍
[14:52:48] <jynus>	 not sure what is the best way to overcome them a) fixing mariadb or b) creating a private logstash, B seems more feasable in the short term 0:-)