[02:45:40] FIRING: [2x] VarnishHighThreadCount: Varnish's thread count on cp1104:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [02:50:40] FIRING: [3x] VarnishHighThreadCount: Varnish's thread count on cp1104:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [02:55:40] RESOLVED: [3x] VarnishHighThreadCount: Varnish's thread count on cp1104:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [10:38:01] 10netops, 06Traffic, 06Discovery-Search, 06Infrastructure-Foundations, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11993251 (10jijiki) [10:40:32] 10netops, 06Traffic, 06Discovery-Search, 06Infrastructure-Foundations, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11993265 (10jijiki) >>! In T427357#11958744, @jijiki wrote: > @ayounsi `mc2055` and `mc-gp2004` are on A4, and that is by accident. `mc-gp2004` is work... [11:54:30] 10netops, 06Infrastructure-Foundations: Create public vlans in eqiad and codfw - https://phabricator.wikimedia.org/T422043#11993668 (10ayounsi) [12:11:54] 10netops, 06Infrastructure-Foundations: Create public vlans in eqiad and codfw - https://phabricator.wikimedia.org/T422043#11993710 (10ayounsi) [12:12:32] 10netops, 06Infrastructure-Foundations: Create public vlans in eqiad and codfw - https://phabricator.wikimedia.org/T422043#11993711 (10ayounsi) 05Open→03Resolved a:03ayounsi All possible vlans have been created. We will add eqiad A3 and C1 once the new switches are ready. [13:58:18] Cuthead: certainly CN as well, per some of our own traffic data. [14:03:18] Cuthead: https://ooni.org/post/2019-china-wikipedia-blocking/ [15:27:40] FIRING: [12x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:32:40] FIRING: [20x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:37:40] FIRING: [22x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:42:40] FIRING: [22x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:47:40] FIRING: [22x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:52:40] FIRING: [18x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:57:40] RESOLVED: [9x] VarnishHighThreadCount: Varnish's thread count on cp1110:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [19:13:31] 06Traffic, 07SEO: Bing can't search images from Commons, is Wikimedia denying their requests? - https://phabricator.wikimedia.org/T425850#11995928 (10ssingh) Update: we are still waiting to get clarity on what is happening here. We are pursuing this and I will update this when things change. [20:06:40] FIRING: VarnishHighThreadCount: Varnish's thread count on cp3069:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=esams&var-instance=cp3069 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:11:40] FIRING: [18x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:12:39] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Moving switches to make space for the refreshed switches. - https://phabricator.wikimedia.org/T428195#11996138 (10VRiley-WMF) 05Open→03Resolved All of the switches in row A and most of row B have been shifted in order to make s... [20:16:40] FIRING: [22x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:21:40] FIRING: [27x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:25:54] ^will work on stopping that noise [20:26:40] FIRING: [27x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:31:40] FIRING: [24x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:36:40] FIRING: [18x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:41:40] RESOLVED: [16x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:44:30] 06Traffic, 06SRE, 13Patch-For-Review: WE5.2.13 Dumps UA enforcement - https://phabricator.wikimedia.org/T427836#11996319 (10BCornwall) 05Open→03In progress [21:06:54] 06Traffic, 06Data-Platform-SRE, 10Wikidata, 10WMDE Analytics, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11996368 (10Ahoelzl) a:03Ahoelzl [21:15:18] 06Traffic, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#11996409 (10Ahoelzl) a:05xcollazo→03Ahoelzl