[10:48:11] 10netops, 10Operations: PyBal BGP group prefix-limit 50 teardown - https://phabricator.wikimedia.org/T246110 (10fgiunchedi) +1 to bumping the limit, although the snipped above has `20` not `200` as the limit for pybal if I'm reading correctly [11:13:28] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission lvs2006.codfw.wmnet - https://phabricator.wikimedia.org/T246329 (10Vgutierrez) [11:14:25] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission lvs2006.codfw.wmnet - https://phabricator.wikimedia.org/T246329 (10Vgutierrez) a:03Vgutierrez [11:47:53] 10Traffic, 10DC-Ops, 10Operations, 10decommission, and 2 others: decommission lvs2006.codfw.wmnet - https://phabricator.wikimedia.org/T246329 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by vgutierrez@cumin2001 for hosts: `lvs2006.codfw.wmnet` - lvs2006.codfw.wmnet (**PASS**) - Downt... [11:58:31] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission lvs2006.codfw.wmnet - https://phabricator.wikimedia.org/T246329 (10Vgutierrez) a:05Vgutierrez→03Papaul [11:59:38] 10Traffic, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10Vgutierrez) @Papaul lvs2006 is all yours, I've filed T246329 [12:00:51] 10Traffic, 10Operations: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10Vgutierrez) [12:08:33] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission lvs2003.codfw.wmnet - https://phabricator.wikimedia.org/T246334 (10Vgutierrez) [12:20:16] 10Traffic, 10DC-Ops, 10Operations, 10decommission, and 2 others: decommission lvs2003.codfw.wmnet - https://phabricator.wikimedia.org/T246334 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by vgutierrez@cumin2001 for hosts: `lvs2003.codfw.wmnet` - lvs2003.codfw.wmnet (**PASS**) - Downt... [12:39:06] 10netops, 10Operations: PyBal BGP group prefix-limit 50 teardown - https://phabricator.wikimedia.org/T246110 (10ayounsi) The syntax is not obvious, `maximum 1000 teardown 20` means shutdown the session at 1000 but start sending warning logs at 20% of the 1000. [12:45:22] 10Traffic, 10DC-Ops, 10Operations, 10decommission, and 2 others: decommission lvs2003.codfw.wmnet - https://phabricator.wikimedia.org/T246334 (10Vgutierrez) a:05Vgutierrez→03Papaul [12:49:04] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10Vgutierrez) @Papaul same for lvs2003: T246334 Regarding lvs2007 and lvs2008, please update the NICs FW to the same versions as you did for lvs2009 and lvs... [12:51:09] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10Vgutierrez) [12:52:15] 10netops, 10Operations: PyBal BGP group prefix-limit 50 teardown - https://phabricator.wikimedia.org/T246110 (10ayounsi) 05Open→03Resolved Done. [13:35:30] 10netops, 10Operations: Add graceful-restart to cr2-esams - https://phabricator.wikimedia.org/T246338 (10ayounsi) p:05Triage→03Medium [14:35:20] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` lvs4007.ulsfo.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/20... [15:03:38] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['lvs4007.ulsfo.wmnet'] ` and were **ALL** successful. [15:17:56] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` lvs4006.ulsfo.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/20... [15:44:42] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['lvs4006.ulsfo.wmnet'] ` and were **ALL** successful. [16:20:36] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` lvs4005.ulsfo.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/20... [16:46:45] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['lvs4005.ulsfo.wmnet'] ` and were **ALL** successful. [17:05:25] vgutierrez: another back-burner thing we should probably get back onto: using the LE unified somewhere [17:05:37] I was originally thinking eqsin as a test [17:05:46] so.. we got the wikishop thingie everywhere [17:05:53] dunno if you turned the DNS knob already [17:06:01] but (a) globalsign has gotten expensive in terms of bytes [17:06:27] (because they do a pointlessly-large staple, and now require an extra intermedate because the root they use is so new) [17:06:59] well let's skip the other bits [17:07:08] but the point is, even in the US latency is hurting a little just from the GS cert [17:07:31] maybe we keep digicert where it is now, and transition the US ones that are using GS to the LE unified, starting with ulsfo? [17:08:12] bblack: on Monday? or today? [17:08:19] then we have our basic "two live certs for known-working redundancy", and the GS cert is basically a warm-ish backup that's not currently deployed [17:08:31] vgutierrez: either is fine, it's not urgent, just don't want to keep forgetting [17:08:44] ack, I'll do it on Monday then [17:08:48] sounds awesome [17:11:00] and yeah, fixing up the wikiworkshop situation, apparently I missed a ticket update there asking us to go ahead [17:11:32] in terms of monitoring wikiworkshop.org looked sane since it was deployed [17:11:42] so it should be the same for unified [17:11:46] (last famous words) [17:13:03] :) [17:18:43] 10Traffic, 10netops, 10Operations, 10ops-codfw: switch port configuration for lvs200[7-10] - https://phabricator.wikimedia.org/T196946 (10Papaul) |Servers|NIC1|NIC2|NIC3|NIC4|Note| |lvs2007| |lvs2008|asw-b2 xe-2/0/45|'A7': xe-7/0/45|C2': xe-2/0/45|D2': xe-2/0/46| using the same cables lvs2006 was using... [17:19:10] 10Traffic, 10Operations, 10Patch-For-Review: Provide an easy way of picking the traffic serving TLS certificate used by ATS - https://phabricator.wikimedia.org/T234803 (10Vgutierrez) 05Stalled→03Resolved a:03Vgutierrez [17:19:12] 10Acme-chief, 10Traffic, 10Operations: Decide/document criteria needed to serve acme-chief LE issued unified certificate to end users - https://phabricator.wikimedia.org/T230687 (10Vgutierrez) [17:27:53] bblack: https://gerrit.wikimedia.org/r/c/operations/puppet/+/575305 everything looks sane after setting public_tls_unified_cert_vendor to "lets-encrypt" [17:45:43] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['lvs5003.eqsin.wmnet'] ` and were **ALL** successful. [17:47:19] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10Vgutierrez) [17:52:14] 10Traffic, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10Papaul)