[07:09:14] 10Traffic, 10Operations, 10Operations-Software-Development, 10Pybal, 10Patch-For-Review: Unhandled pybal error causing services to be depooled in etcd but not in lvs - https://phabricator.wikimedia.org/T134893#3485048 (10Volans) The added `PyBal IPVS diff check` is flapping a bit with UNKNOWN for some ho... [11:18:16] 10Traffic, 10Operations: OCSP update failed for /etc/update-ocsp.d/globalsign-2016-ecdsa-unified.conf - https://phabricator.wikimedia.org/T172101#3485695 (10ema) [11:18:25] 10Traffic, 10Operations: OCSP update failed for /etc/update-ocsp.d/globalsign-2016-ecdsa-unified.conf - https://phabricator.wikimedia.org/T172101#3485707 (10ema) p:05Triage>03Normal [11:56:27] 10Traffic, 10Operations: IPVS issues with UDP services, pybal depooling strategy - https://phabricator.wikimedia.org/T172103#3485845 (10ema) [11:56:43] 10Traffic, 10Operations, 10Pybal: IPVS issues with UDP services, pybal depooling strategy - https://phabricator.wikimedia.org/T172103#3485857 (10ema) p:05Triage>03Normal [13:34:32] bd808: that wikitech link on IPSec is probably largely accurate, not much has changed since then other minor cleanups and resiliency fixes that happened fairly early on [13:38:50] 10Traffic, 10Operations, 10Operations-Software-Development, 10Pybal, 10Patch-For-Review: Unhandled pybal error causing services to be depooled in etcd but not in lvs - https://phabricator.wikimedia.org/T134893#3486123 (10BBlack) >>! In T134893#3485048, @Volans wrote: > The added `PyBal IPVS diff check` i... [13:49:56] 10Traffic, 10Operations, 10Pybal: IPVS issues with UDP services, pybal depooling strategy - https://phabricator.wikimedia.org/T172103#3486193 (10BBlack) +1. There are a number of tricky things here to get to these simple goals, though, and since the sysctls affect all services, we have to have the TCP cases... [13:51:06] 10Traffic, 10Operations, 10Pybal: Backport ipvsadm - https://phabricator.wikimedia.org/T171850#3486199 (10BBlack) [13:51:09] 10Traffic, 10Operations, 10Pybal: IPVS issues with UDP services, pybal depooling strategy - https://phabricator.wikimedia.org/T172103#3486198 (10BBlack) [13:56:42] 10Traffic, 10Operations: OCSP update failed for /etc/update-ocsp.d/globalsign-2016-ecdsa-unified.conf - https://phabricator.wikimedia.org/T172101#3486214 (10BBlack) 05Open>03Resolved a:03BBlack Ran it again and it's ok now. [14:09:03] 10Traffic, 10Operations: Improve OCSP fetching and monitoring strategies - https://phabricator.wikimedia.org/T172116#3486266 (10BBlack) [14:16:58] 10Traffic, 10Operations: Improve OCSP fetching and monitoring strategies - https://phabricator.wikimedia.org/T172116#3486297 (10BBlack) Hmm I wrote that backwards above. The OCSP file-freshness checks look at age-of-mtime, not the timestamp within. In any case, we can still move them to crit=~3d and warn=~2d. [14:25:50] bblack: good to know. I found it listed on https://wikitech.wikimedia.org/wiki/Special:LonelyPages and saw a 2 year old draft so I jumped to conclusions. :) [15:05:44] 10HTTPS, 10Traffic, 10Operations, 10Wikimedia-Blog: Change automatic shortlink in blog theme - https://phabricator.wikimedia.org/T165511#3486463 (10Volker_E) The code provided by WordPress VIP has been merged into the repo on 20 Jun and got deployed shortly after. One would need to look into the FB redirec... [15:18:00] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Add support for setting weight=0 when depooling - https://phabricator.wikimedia.org/T86650#3486487 (10BBlack) [15:18:03] 10Traffic, 10Operations, 10Pybal: IPVS issues with UDP services, pybal depooling strategy - https://phabricator.wikimedia.org/T172103#3486486 (10BBlack) [15:26:02] 10Traffic, 10Android-app-feature-Compilations, 10Operations, 10Wikipedia-Android-App-Backlog, 10Reading-Infrastructure-Team-Backlog (Kanban): Determine how to upload Zim files to Swift infrastructure - https://phabricator.wikimedia.org/T172123#3486510 (10Fjalapeno) [15:26:56] 10Traffic, 10Android-app-feature-Compilations, 10Operations, 10Wikipedia-Android-App-Backlog, 10Reading-Infrastructure-Team-Backlog (Kanban): Determine how to upload Zim files to Swift infrastructure - https://phabricator.wikimedia.org/T172123#3486510 (10Fjalapeno) [15:27:23] 10Traffic, 10Android-app-feature-Compilations, 10Operations, 10Reading-Infrastructure-Team-Backlog, 10Wikipedia-Android-App-Backlog: Determine how to upload Zim files to Swift infrastructure - https://phabricator.wikimedia.org/T172123#3486510 (10Fjalapeno) [15:27:26] 10Traffic, 10Operations, 10Pybal: PyBal Feature: progressive depooling strategy for monitored failures - https://phabricator.wikimedia.org/T172124#3486529 (10BBlack) [15:27:51] 10Traffic, 10Operations, 10Pybal: Backport ipvsadm - https://phabricator.wikimedia.org/T171850#3486545 (10BBlack) [15:27:54] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Add support for setting weight=0 when depooling - https://phabricator.wikimedia.org/T86650#3486546 (10BBlack) [15:27:57] 10Traffic, 10Operations, 10Pybal: PyBal Feature: progressive depooling strategy for monitored failures - https://phabricator.wikimedia.org/T172124#3486529 (10BBlack) [15:28:28] 10Traffic, 10Android-app-feature-Compilations, 10Operations, 10Reading-Infrastructure-Team-Backlog, 10Wikipedia-Android-App-Backlog: Determine how to upload Zim files to Swift infrastructure - https://phabricator.wikimedia.org/T172123#3486547 (10Fjalapeno) [15:30:56] 10Traffic, 10Operations, 10Pybal: PyBal Feature: progressive depooling strategy for monitored failures - https://phabricator.wikimedia.org/T172124#3486555 (10BBlack) It's also an interesting thought to consider progressively scaling the weight. For example, you could make the strategy configurable such that... [15:41:28] 10Traffic, 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, 10User-Elukey: Encrypt Kafka traffic, and restrict access via ACLs - https://phabricator.wikimedia.org/T121561#3486631 (10mforns) [15:46:03] 10Traffic, 10Operations, 10Operations-Software-Development, 10Pybal, 10Patch-For-Review: Unhandled pybal error causing services to be depooled in etcd but not in lvs - https://phabricator.wikimedia.org/T134893#3486672 (10ema) >>! In T134893#3486123, @BBlack wrote: > That it's happening often enough to re... [16:10:52] 10Traffic, 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, 10User-Elukey: Encrypt Kafka traffic, and restrict access via ACLs - https://phabricator.wikimedia.org/T121561#3486779 (10mforns) [16:39:34] yeah so copying over from -ops: regardless of what actually happened with ES, it's clear we've got one minor design issue with the ipvs diff check, which is that it will confusingly alert when the "cannot depool because too many down" stuff happens (because at that point pybal+ipvs are intentionally un-synced by pybal) [16:39:55] right [16:39:56] so we'll probably need some creative solution there, probably export some new state bit about that condition from the pybal http interface? [16:42:33] honestly I'm not even sure what current pybal does about even tracking that state, I know it was a bug issue in the past [16:42:48] (that it would kinda lose track in those scenarios that it was internal-state-failed/down, but up in ipvs) [16:57:13] 10Traffic, 10Operations, 10Mobile, 10Need-volunteer, and 2 others: URLs with title query string parameter and additional query string parameters do not redirect to mobile site - https://phabricator.wikimedia.org/T154227#2904582 (10BBlack) It seems reasonable to relax the regex in question a bit (to allow a... [17:14:08] 10Traffic, 10Operations, 10Mobile, 10Need-volunteer, and 3 others: URLs with title query string parameter and additional query string parameters do not redirect to mobile site - https://phabricator.wikimedia.org/T154227#3486936 (10Jdlrobson) Something like this maybe? ``` @@ -23,8 +23,13 @@ sub mobile_re... [18:44:13] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team (Kanban): Verify that the codfw lvs is configured correctly for Phabricator - https://phabricator.wikimedia.org/T168699#3487227 (10mmodell) [19:48:23] 10Traffic, 10Android-app-feature-Compilations, 10Operations, 10Reading-Infrastructure-Team-Backlog, 10Wikipedia-Android-App-Backlog: Determine URL paths for Zim files - https://phabricator.wikimedia.org/T172148#3487493 (10Fjalapeno) [19:48:55] 10Traffic, 10Android-app-feature-Compilations, 10Operations, 10Wikipedia-Android-App-Backlog, 10Reading-Infrastructure-Team-Backlog (Kanban): Determine URL paths for Zim files - https://phabricator.wikimedia.org/T172148#3487493 (10Fjalapeno) [21:52:23] 10Traffic, 10Android-app-feature-Compilations, 10Operations, 10Wikipedia-Android-App-Backlog, 10Reading-Infrastructure-Team-Backlog (Kanban): Determine where to host zim files for the Android app - https://phabricator.wikimedia.org/T170843#3487861 (10Tbayer) [22:45:58] 10Traffic, 10Operations, 10RESTBase, 10RESTBase-API, 10Services (next): RESTBase support for www.wikimedia.org missing - https://phabricator.wikimedia.org/T133178#3487965 (10mobrovac) >>! In T133178#3482880, @GWicke wrote: > This sounds reasonable to me. Any objections against going with www.wikimedia.or... [23:27:01] 10HTTPS, 10Traffic, 10Operations, 10Wikimedia-Blog: Change automatic shortlink in blog theme - https://phabricator.wikimedia.org/T165511#3488092 (10EdErhart-WMF) @Volker_E Facebook's debugging tool allows you to see what their scraper sees on our blog. ([[ https://developers.facebook.com/tools/debug/echo/?...