[05:31:32] 10netops, 06Infrastructure-Foundations, 06SRE: Update esams network pop diagrams - https://phabricator.wikimedia.org/T368084#11657121 (10Papaul) @ayounsi please see below for the IP layer diagram let me know if we need to modify it thanks {F72443222} [12:09:11] 06Traffic, 10Diff-blog, 10Technical Blog: Redirect techblog.wikimedia.org to diff.wikimedia.org - https://phabricator.wikimedia.org/T417940#11658057 (10A_smart_kitten) [12:19:43] FIRING: [7x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [12:24:43] FIRING: [47x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [12:34:43] FIRING: [47x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [12:39:43] RESOLVED: [47x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [13:17:34] 06Traffic, 10API Platform, 10MediaWiki-User-login-and-signup, 06MediaWiki-Platform-Team (Q3 Kanban Board), and 2 others: Login with `action=login` and bot password does not create a JWT session cookie - https://phabricator.wikimedia.org/T415007#11658252 (10Tgr) This might ave broken bot passwords; there ar... [14:12:24] 06Traffic, 10API Platform, 10MediaWiki-User-login-and-signup, 06MediaWiki-Platform-Team (Q3 Kanban Board), and 2 others: Login with `action=login` and bot password does not create a JWT session cookie - https://phabricator.wikimedia.org/T415007#11658459 (10Tgr) Probably it's from [[https://gerrit.wikimedia... [14:14:00] 06Traffic, 10API Platform, 10MediaWiki-User-login-and-signup, 06MediaWiki-Platform-Team (Q3 Kanban Board), and 2 others: Login with `action=login` and bot password does not create a JWT session cookie - https://phabricator.wikimedia.org/T415007#11658470 (10Tgr) It's a weekend so let's just disable for now. [15:02:09] 06Traffic, 10API Platform, 10MediaWiki-User-login-and-signup, 06MediaWiki-Platform-Team (Q3 Kanban Board), and 2 others: Login with `action=login` and bot password does not create a JWT session cookie - https://phabricator.wikimedia.org/T415007#11658639 (10Tgr) The logs are not down. I tested and bot passw... [15:43:59] 06Traffic, 10API Platform, 10MediaWiki-User-login-and-signup, 06MediaWiki-Platform-Team (Q3 Kanban Board), and 2 others: Login with `action=login` and bot password does not create a JWT session cookie - https://phabricator.wikimedia.org/T415007#11658751 (10Tgr) ` from /srv/mediawiki/php-1.46.0-wmf.17/inclu... [15:48:09] 06Traffic, 06MW-Interfaces-Team, 06MediaWiki-Platform-Team (Radar), 07OKR-Work, 13Patch-For-Review: haproxy: strip x-wmf-* headers from responses - https://phabricator.wikimedia.org/T417781#11658766 (10Vgutierrez) could I suggest using a dedicated prefix for API/REST gateway headers? We already use inter... [15:51:09] 06Traffic, 06MW-Interfaces-Team, 06MediaWiki-Platform-Team (Radar), 07OKR-Work, 13Patch-For-Review: haproxy: strip x-wmf-* headers from responses - https://phabricator.wikimedia.org/T417781#11658772 (10Joe) I don't want us to make anything under `x-wmf` be stripped automatically. It limits ourselves in t... [16:22:06] 06Traffic, 10API Platform, 10MediaWiki-User-login-and-signup, 06MediaWiki-Platform-Team (Q3 Kanban Board), and 2 others: Login with `action=login` and bot password does not create a JWT session cookie - https://phabricator.wikimedia.org/T415007#11658869 (10Tgr) So apparently this happens for anonymous sess... [16:40:15] 06Traffic, 06MW-Interfaces-Team, 06MediaWiki-Platform-Team (Radar), 07OKR-Work, 13Patch-For-Review: haproxy: strip x-wmf-* headers from responses - https://phabricator.wikimedia.org/T417781#11658995 (10daniel) The gateway is adding x-wmf-ratelimit-class and x-wmf-ratelimit-policy and x-wmf-user-id to req... [16:47:15] 06Traffic, 10API Platform, 10MediaWiki-User-login-and-signup, 06MediaWiki-Platform-Team (Q3 Kanban Board), and 2 others: Login with `action=login` and bot password does not create a JWT session cookie - https://phabricator.wikimedia.org/T415007#11659002 (10Tgr) Looking up the affected users, they are all b... [16:59:35] 06Traffic, 10API Platform, 10MediaWiki-User-login-and-signup, 06MediaWiki-Platform-Team (Q3 Kanban Board), and 2 others: Login with `action=login` and bot password does not create a JWT session cookie - https://phabricator.wikimedia.org/T415007#11659054 (10Tgr) Bot password JWTs were disabled today at 14:4... [20:00:31] 06Traffic, 10API Platform, 10MediaWiki-User-login-and-signup, 06MediaWiki-Platform-Team (Q3 Kanban Board), and 2 others: Login with `action=login` and bot password does not create a JWT session cookie - https://phabricator.wikimedia.org/T415007#11659539 (10Tgr) The log entries didn't fully stop around 18:4... [20:25:58] 06Traffic, 13Patch-For-Review: Reimage cp20[43-58] to Trixie - https://phabricator.wikimedia.org/T418161#11659608 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp2054.codfw.wmnet with OS trixie [20:26:59] 06Traffic, 13Patch-For-Review: Reimage cp20[43-58] to Trixie - https://phabricator.wikimedia.org/T418161#11659621 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp2055.codfw.wmnet with OS trixie [20:27:37] 06Traffic, 13Patch-For-Review: Reimage cp20[43-58] to Trixie - https://phabricator.wikimedia.org/T418161#11659623 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp2056.codfw.wmnet with OS trixie [20:55:14] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11659679 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp2057.codfw.wmnet with OS trixie [20:57:28] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11659681 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp2058.codfw.wmnet with OS trixie [21:08:38] 06Traffic: Reimage cp20[43-58] to Trixie - https://phabricator.wikimedia.org/T418161#11659735 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp2054.codfw.wmnet with OS trixie completed: - cp2054 (**PASS**) - Removed from Puppet and PuppetDB if present and del... [21:11:29] 06Traffic: Reimage cp20[43-58] to Trixie - https://phabricator.wikimedia.org/T418161#11659754 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp2055.codfw.wmnet with OS trixie completed: - cp2055 (**PASS**) - Removed from Puppet and PuppetDB if present and del... [21:12:15] 06Traffic: Reimage cp20[43-58] to Trixie - https://phabricator.wikimedia.org/T418161#11659755 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp2056.codfw.wmnet with OS trixie completed: - cp2056 (**PASS**) - Removed from Puppet and PuppetDB if present and del... [21:36:54] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11659810 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp2057.codfw.wmnet with OS trixie completed: - cp2057 (**PASS**) - Removed from Puppet and PuppetDB if present and d... [21:39:13] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11659816 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp2058.codfw.wmnet with OS trixie completed: - cp2058 (**PASS**) - Removed from Puppet and PuppetDB if present and d... [21:39:15] 06Traffic, 10API Platform, 10MediaWiki-User-login-and-signup, 06MediaWiki-Platform-Team (Q3 Kanban Board), and 2 others: Login with `action=login` and bot password does not create a JWT session cookie - https://phabricator.wikimedia.org/T415007#11659814 (10Tgr) One way to handle this is {T418475} - instead... [21:51:06] trying to track down whether HA Proxy does URL normalization before logging to Kafka (as I believe Varnish used to do). I see the log format here uses %HPO: [21:51:07] https://gerrit.wikimedia.org/g/operations/puppet/+/5bb8a7d2cb6bfe4e45b43b3a9d320491c6d5fc77/modules/profile/templates/cache/haproxy.cfg.erb#71 [21:51:50] I'm reading the docs https://github.com/haproxy/haproxy/blob/master/doc/configuration.txt and trying to understand [21:52:38] so like, if a request came in for /wiki/barack%20Obama, would %HPO be /wiki/Barack_Obama? [22:26:13] 06Traffic: Reimage cp20[43-58] to Trixie - https://phabricator.wikimedia.org/T418161#11659926 (10CDobbins) [22:26:16] 06Traffic: Reimage cp20[43-58] to Trixie - https://phabricator.wikimedia.org/T418161#11659927 (10CDobbins) 05In progress→03Resolved