[00:03:40] FIRING: [15x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [00:08:40] FIRING: [17x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [00:13:40] FIRING: [17x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [00:18:40] FIRING: [25x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [00:23:40] FIRING: [27x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [00:28:40] FIRING: [31x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [00:38:40] FIRING: [19x] VarnishHighThreadCount: Varnish's thread count on cp1102:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [00:43:40] FIRING: [16x] VarnishHighThreadCount: Varnish's thread count on cp1102:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [00:48:40] RESOLVED: [14x] VarnishHighThreadCount: Varnish's thread count on cp1102:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [08:55:30] 06Traffic, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: Benthos loses messages when under high load - https://phabricator.wikimedia.org/T364379#9782635 (10Fabfur) Little addendum: the [[ https://github.com/benthosdev/benthos/releases/tag/v4.27.0 | latest Benthos release ]] introduced th... [09:02:19] 06Traffic, 10Observability-Logging, 13Patch-For-Review: Use definitive field naming during grok parsing instead of renaming later (in mapping) - https://phabricator.wikimedia.org/T363420#9782637 (10Fabfur) 05Open→03Resolved p:05Triage→03Low [09:02:49] 06Traffic, 06Data-Engineering, 10Observability-Logging, 10Event-Platform, 13Patch-For-Review: Remove extra fields currently sent to Kafka - https://phabricator.wikimedia.org/T360642#9782643 (10Fabfur) 05Open→03Resolved [09:03:37] 06Traffic, 06Data Products, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: Change mtail configuration to ignore new fields in HAProxy logs - https://phabricator.wikimedia.org/T358107#9782645 (10Fabfur) 05Open→03Resolved [13:17:00] 06Traffic, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: Benthos loses messages when under high load - https://phabricator.wikimedia.org/T364379#9783228 (10CDanis) I was thinking about this while I was falling asleep last night, and I think it's actually desirable to drop some messages u... [13:32:41] 06Traffic, 10observability: support stdin/stdout/stderr in benthos unit file - https://phabricator.wikimedia.org/T364543 (10Fabfur) 03NEW [16:23:54] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 10ops-codfw: Create (or teach Andrew how to create) private connections+dns entries for new cloudcontrols - https://phabricator.wikimedia.org/T364559 (10Andrew) 03NEW [16:33:55] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Create (or teach Andrew how to create) private connections+dns entries for new cloudcontrols - https://phabricator.wikimedia.org/T364559#9783893 (10cmooney) Hey Andrew, Yeah this is on me, I'd not completed the work to ma... [16:58:16] 06Traffic, 06Movement-Insights, 13Patch-For-Review: Disable Chrome Private Prefetch Proxy - https://phabricator.wikimedia.org/T364126#9784006 (10ssingh) We have merged this change today and are serving the `/.well-known/traffic-advice` file with the content noted in the documentation to disable this feature. [17:31:25] 06Traffic, 06Movement-Insights, 13Patch-For-Review: Disable Chrome Private Prefetch Proxy - https://phabricator.wikimedia.org/T364126#9784114 (10KOfori) a:05KOfori→03OSefu-WMF [17:32:41] 06Traffic, 06Movement-Insights: Disable Chrome Private Prefetch Proxy - https://phabricator.wikimedia.org/T364126#9784122 (10KOfori) Hi @OSefu-WMF, @ssingh has completed the change. Will leave this task for you to verify and close as you see fit. [17:34:24] 06Traffic, 06Movement-Insights: Disable Chrome Private Prefetch Proxy - https://phabricator.wikimedia.org/T364126#9784127 (10OSefu-WMF) 05Stalled→03In progress [18:27:32] 06Traffic, 06Data-Platform-SRE, 13Patch-For-Review: LVS hosts: Monitor/alert on when pooled nodes are outside broadcast domain - https://phabricator.wikimedia.org/T363702#9784278 (10CodeReviewBot) cmooney closed https://gitlab.wikimedia.org/repos/search-platform/sre/lvs_l2_checker/-/merge_requests/2 Source... [19:24:04] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Create (or teach Andrew how to create) private connections+dns entries for new cloudcontrols - https://phabricator.wikimedia.org/T364559#9784420 (10Andrew) Reimaging cloudcontrol2006-dev works now, thanks! Bonus points: I... [19:59:13] 06Traffic, 06Data-Platform-SRE, 13Patch-For-Review: LVS hosts: Monitor/alert on when pooled nodes are outside broadcast domain - https://phabricator.wikimedia.org/T363702#9784540 (10CodeReviewBot) bking merged https://gitlab.wikimedia.org/repos/search-platform/sre/lvs_l2_checker/-/merge_requests/3 Source ba... [20:00:32] 06Traffic, 06Data-Platform-SRE, 13Patch-For-Review: LVS hosts: Monitor/alert on when pooled nodes are outside broadcast domain - https://phabricator.wikimedia.org/T363702#9784538 (10CodeReviewBot) bking opened https://gitlab.wikimedia.org/repos/search-platform/sre/lvs_l2_checker/-/merge_requests/3 Source ba... [20:06:50] 06Traffic, 06Data-Platform-SRE, 13Patch-For-Review: LVS hosts: Monitor/alert when pooled nodes are outside broadcast domain - https://phabricator.wikimedia.org/T363702#9784565 (10bking) [20:11:44] 06Traffic, 06Data-Platform-SRE, 13Patch-For-Review: LVS hosts: Monitor/alert when pooled nodes are outside broadcast domain - https://phabricator.wikimedia.org/T363702#9784575 (10CodeReviewBot) bking opened https://gitlab.wikimedia.org/repos/search-platform/sre/lvs_l2_checker/-/merge_requests/4 Output statu... [21:11:09] 06Traffic, 06Data-Platform-SRE, 13Patch-For-Review: LVS hosts: Monitor/alert when pooled nodes are outside broadcast domain - https://phabricator.wikimedia.org/T363702#9784659 (10CodeReviewBot) cmooney closed https://gitlab.wikimedia.org/repos/search-platform/sre/lvs_l2_checker/-/merge_requests/4 Output sta... [22:10:51] 10Acme-chief: acme-chief: add support for serving individual files over the puppet file system api - https://phabricator.wikimedia.org/T364589 (10jhathaway) 03NEW [22:15:49] 10Acme-chief: acme-chief: add support for serving individual files over the puppet file system api - https://phabricator.wikimedia.org/T364589#9784799 (10jhathaway)