[00:31:11] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11965905 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cp5032.eqsin.wmnet with OS... [01:47:39] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11965987 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cp5032.eqsin.wmnet with OS trix... [01:52:01] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11965992 (10Papaul) @BCornwall re-image done on cp5032. The node is now on the new private1-604-eqsin vlan. The DHCP issue I was having,... [01:53:25] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11965995 (10BCornwall) Sweet, thanks! I'll re-pool tomorrow. [01:54:44] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11965996 (10Papaul) [02:20:10] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11966000 (10Papaul) Please see below steps before re-imaging a node into the new vlan - Netbox 1- Search for the node in Netbox 2- Click... [06:33:28] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11966120 (10cmooney) Awesome work @papaul! I think possibly you can just reimage with the —move-vlan flag for the cookbook? We should... [07:05:40] FIRING: [4x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [07:07:55] 06Traffic, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): Validate pybal config in CI - https://phabricator.wikimedia.org/T394789#11966157 (10Gehel) [07:10:40] FIRING: [24x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [07:15:40] FIRING: [30x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [07:20:40] FIRING: [46x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [07:30:40] FIRING: [43x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [07:35:40] FIRING: [31x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [07:40:40] RESOLVED: [23x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [08:41:25] 10netops, 06Infrastructure-Foundations, 06SRE: cr2-drmrs unexpected reboot - https://phabricator.wikimedia.org/T427600 (10cmooney) 03NEW p:05Triage→03Medium [09:47:01] hi, I'm preparing to remove lvs configuration for eventstreams-internal https://gerrit.wikimedia.org/r/1295409, can I have some help? [10:35:08] for this diff https://gerrit.wikimedia.org/r/1295410 what cumin group I should test? [10:39:33] testing on lvs1019.eqiad.wmnet [11:01:44] atsukoito: Sorry, the team is a little short staffed today. [11:04:16] sukhe should be around a little later. I'm not totally comfortable with the lvs setup yet, so I'm not much help :-( [11:24:57] atsukoito: happy to take a look when I am online a bit later [12:03:37] slyngs: thanks! sukhe: i'm back online [12:14:30] * atsukoito talked to sukhe, will do the lvs removal on Monday [13:19:41] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11967201 (10Papaul) @cmooney thank you, yes move-vlan flag cookbook will also work we need to test that. I don't think we have done any i... [13:26:18] 06Traffic: x-ua-contact: recognize pywikibot user-agent strings - https://phabricator.wikimedia.org/T427491#11967267 (10CDanis) >>! In T427491#11965123, @daniel wrote: > Oh, I think I know what happened. There are log entries for `category_redirect (commons:commons; User:RussBot) Pywikibot/10.2.0 (g19732) re... [13:33:16] 06Traffic: x-ua-contact: recognize pywikibot user-agent strings - https://phabricator.wikimedia.org/T427491#11967382 (10daniel) >>! In T427491#11967267, @CDanis wrote: > How sharp of a corner case is this for you? It's not needed for the rate limiting to work, but it's useful when reading logs and investiga... [14:21:41] 06Traffic, 06SRE: netconsole being used for cache hosts? - https://phabricator.wikimedia.org/T427646 (10MoritzMuehlenhoff) 03NEW [15:04:32] 06Traffic, 07SEO: Bing can't search images from Commons, is Wikimedia denying their requests? - https://phabricator.wikimedia.org/T425850#11967865 (10Maryana) Hi @AlexisJazz – I'm also not able to reproduce this issue (accessing Bing from the US without a VPN). I see that others have reported issues with Bing... [15:05:11] 06Traffic, 07SEO: Bing can't search images from Commons, is Wikimedia denying their requests? - https://phabricator.wikimedia.org/T425850#11967866 (10Maryana) 05Open→03Resolved [15:09:23] 06Traffic, 06SRE: netconsole being used for cache hosts? - https://phabricator.wikimedia.org/T427646#11967877 (10ssingh) Yeah it's a good question and predates me so I don't have good answers. But, it doesn't seem that it is enabled for upload? The upload role does say `include profile::netconsole::client` but... [15:16:09] 06Traffic, 06SRE: netconsole being used for cache hosts? - https://phabricator.wikimedia.org/T427646#11967900 (10ssingh) I meant we set `profile::netconsole::client::ensure: absent` in `hieradata/role/common/cache/upload.yaml` so it should not be enabled there. [15:22:39] 06Traffic, 06Data-Persistence: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#11967928 (10MatthewVernon) A couple of thoughts. I am broadly in favour of handling thumbs and originals separately :) First - our [[ https://gerrit.wikimedia.org/r/plugins/gitiles/ope... [15:47:22] 06Traffic, 06Data-Persistence: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#11968047 (10Ladsgroup) >>! In T427465#11967928, @MatthewVernon wrote: > A couple of thoughts. I am broadly in favour of handling thumbs and originals separately :) > > First - our [[ h... [16:13:34] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE: Investigate hardware RAID usage in codfw LVS hosts - https://phabricator.wikimedia.org/T426912#11968120 (10ssingh) >>! In T426912#11964942, @BCornwall wrote: > @ssingh @BBlack Okay with me switching write-back to write-through slowly through the codfw cluster or sha... [19:18:36] 06Traffic, 07SEO: Bing can't search images from Commons, is Wikimedia denying their requests? - https://phabricator.wikimedia.org/T425850#11968618 (10AlexisJazz) 05Resolved→03Open The request was simple. Look in the server logs. I tried to reverse search https://upload.wikimedia.org/wikipedia/commons/6/63... [19:39:43] 06Traffic, 07SEO: Bing can't search images from Commons, is Wikimedia denying their requests? - https://phabricator.wikimedia.org/T425850#11968680 (10ssingh) >>! In T425850#11968618, @AlexisJazz wrote: > The request was simple. Look in the server logs. I did try to see if I could spot something, either for th... [20:40:00] 06Traffic, 07SEO: Bing can't search images from Commons, is Wikimedia denying their requests? - https://phabricator.wikimedia.org/T425850#11968939 (10ssingh) Update: We are looking into this -- on how to update the list of Bing IPs -- and will update the thread.