[12:20:18] \o I have been changing some a username (see https://gerrit.wikimedia.org/r/c/labs/private/+/1163739) for Swift. Ok for me to roll-restart the Thanos Swift FEs as indicated by https://wikitech.wikimedia.org/wiki/Swift/How_To#Create_a_new_swift_account_(Thanos_Cluster) ? [12:21:28] Or are things different now due to MOSS? [12:39:47] klausman: I'd say so, though please inquire data-persistence as the owners of thanos-swift [12:42:25] roger [14:54:35] FIRING: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [15:04:35] RESOLVED: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [15:29:35] FIRING: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [15:34:35] FIRING: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [15:39:35] RESOLVED: ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [15:44:35] FIRING: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [15:49:35] RESOLVED: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [15:59:35] FIRING: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [16:04:35] RESOLVED: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [16:10:35] FIRING: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [16:44:50] RESOLVED: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [16:46:05] FIRING: ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [16:51:05] FIRING: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [17:06:05] RESOLVED: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [17:22:35] FIRING: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [17:27:35] RESOLVED: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [17:37:35] FIRING: ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [17:42:35] RESOLVED: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [17:47:35] FIRING: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [17:57:35] RESOLVED: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [18:03:35] FIRING: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [18:08:35] RESOLVED: [2x] ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [18:14:35] FIRING: ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [18:19:35] RESOLVED: ThanosSidecarDropQueries: Thanos Sidecar is dropping large queries. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/b19644bfbf0ec1e108027cce268d99f7/thanos-sidecar - https://alerts.wikimedia.org/?q=alertname%3DThanosSidecarDropQueries [18:34:55] I wonder if thanos sidecar is dropping large queries [18:35:21] is this one a matter of tuning the threshold since we recently adjusted the sidecar limits?