[00:22:41] 10Traffic, 10netops, 10Operations: Reporting en.wikipedia is down - https://phabricator.wikimedia.org/T246040 (10Reedy) 05Open→03Invalid [00:36:05] 10Traffic, 10Cleanup, 10Operations, 10fixcopyright.wikimedia.org, and 4 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10Jdforrester-WMF) [00:53:55] 10Traffic, 10Cleanup, 10Operations, 10fixcopyright.wikimedia.org, and 4 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10Jdforrester-WMF) [01:05:16] 10Traffic, 10Cleanup, 10Operations, 10fixcopyright.wikimedia.org, and 4 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10Jdforrester-WMF) [01:06:31] 10Traffic, 10Cleanup, 10Operations, 10fixcopyright.wikimedia.org, and 4 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10Jdforrester-WMF) [01:07:39] 10Traffic, 10Cleanup, 10Operations, 10fixcopyright.wikimedia.org, and 4 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10Jdforrester-WMF) [01:09:07] 10Traffic, 10Cleanup, 10Operations, 10fixcopyright.wikimedia.org, and 4 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10Jdforrester-WMF) [01:12:36] 10Traffic, 10Cleanup, 10Operations, 10Release-Engineering-Team-TODO, and 3 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10CCicalese_WMF) Thank you for all of your work on this, @Jdforrester-WMF! [01:12:50] 10Traffic, 10Cleanup, 10Operations, 10Release-Engineering-Team-TODO, and 3 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10Jdforrester-WMF) a:05Jdforrester-WMF→03None This is now done as much as we can at RelEng's side. Assigning back over to CPT for the task tr... [01:15:40] 10Traffic, 10Cleanup, 10Operations, 10Release-Engineering-Team-TODO, and 3 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10CCicalese_WMF) a:03CCicalese_WMF [07:48:19] 10Traffic, 10Operations: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10Vgutierrez) [08:03:56] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` lvs2010.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/20... [08:26:38] 10Traffic, 10Operations: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['lvs2010.codfw.wmnet'] ` Of which those **FAILED**: ` ['lvs2010.codfw.wmnet'] ` [12:35:21] sukhe: I guess that you're aware but.. https://blog.mozilla.org/netpolicy/2020/02/25/the-facts-mozillas-dns-over-https-doh/ :) [12:54:49] vgutierrez: ha, no, this one is new! their last update was that they had enabled it for 1% of US (geolocation) users [12:59:24] "We do not have plans to roll out the feature in Europe" -> I thought their concern was just the UK because of its parental filters (which I guess is "not Europe" now?) so that's an interesting update [13:41:50] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` lvs2010.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/20... [13:42:14] sukhe: err.. it's still Europe, but not UE anymoe [13:42:17] *anymore [13:42:35] sukhe: our legislators aren't able to mess with the continents (yet) [13:47:35] vgutierrez: ... for now, don't underestimate the ambition of Boris Johnson. I don't put it past him to (want to) "rearrange" the British Isles. [13:47:52] lol [13:48:39] Alternatively, if trump is still offering Greenland to the market in Denmark's behalf, Boris might buy that and move our citizens there [14:01:42] vgutierrez: I meant "not" as in not in theory :) [14:09:56] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['lvs2010.codfw.wmnet'] ` and were **ALL** successful. [14:37:19] 10netops, 10Operations: PyBal BGP group prefix-limit 50 teardown - https://phabricator.wikimedia.org/T246110 (10ayounsi) p:05Triage→03High [15:37:21] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` lvs2009.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/20... [15:53:17] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=icinga1001&service=cache_text%3A+Varnishkafka+statsv+Delivery+Errors+per+second+-esams- [16:04:53] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['lvs2009.codfw.wmnet'] ` and were **ALL** successful. [16:05:48] 10Traffic, 10Operations, 10ops-eqiad: cp1088 - https://phabricator.wikimedia.org/T245645 (10Cmjohnson) 05Open→03Resolved @RobH cp1088 does not have a front LCD display but the front LED is working. For good measure, I did a racreset and cleared the syslog [16:07:15] 10Traffic, 10Operations, 10ops-eqiad: cp1088 - https://phabricator.wikimedia.org/T245645 (10RobH) Cool, it was an odd error I've never seen before, so if it doesn't happen twice it didn't happen! [16:35:12] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10Vgutierrez) @BBlack @Papaul lvs2009 and lvs2010 are now online as secondary load balancers and ready to take over lvs2003 and lvs2006 respectively. This should be enough... [16:36:08] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade load balancers to buster - https://phabricator.wikimedia.org/T245984 (10Vgutierrez) [16:37:35] bblack: please feel free to double check lvs2009 and lvs2010 whenever you have the chance, but those two are looking good IMHO [16:38:01] awesome news [16:38:08] (up and running with buster, and ready to take over 2003 and 2006 [16:40:26] 10netops, 10Operations: PyBal BGP group prefix-limit 50 teardown - https://phabricator.wikimedia.org/T246110 (10Vgutierrez) Option number three looks good, but IMHO I'd decrease the teardown percentage. [16:46:39] FYI: Due to the other ongoing esams items, I've held off touching cp firmware yesterday (and likely today and tomorrow, as the other outstanding items in esams will be handled tomorrow) [16:47:24] it just seemed prudent to not futz with it. (echoing here in case i am incorrect and today is the ideal day due to the other issues) [16:50:16] so the obvious problem is that varnish-fe won't get warmed, but after a few days depooled I don't know if that's going to really matter [17:11:59] bblack: don't want to distract from other work, but a short TLDR update on https://phabricator.wikimedia.org/T238494 would be good to have if there's anything perf can to do help, or just to bounce ideas :) [17:12:48] last three comments capture the latest data [17:12:59] subtask(s) last updated ~ 9 days ago [17:20:26] Krinkle: so I already know why you didn't see the boost described on T244538 after getting rid of DNS resolution on ats-tls [17:20:26] T244538: ats-tls performance issues under production load - https://phabricator.wikimedia.org/T244538 [17:20:59] Krinkle: I'm afraid performance degrades over time for some reason [17:24:24] 10Traffic, 10Operations, 10Patch-For-Review: ats-tls performance issues under production load - https://phabricator.wikimedia.org/T244538 (10Vgutierrez) it looks like ats-tls performance gets degraded over time, cp1075 was showing similar values to the ones exhibited before disabling DNS on ats-tls, a servic... [17:40:46] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10Papaul) @Vgutierrez thanks for the update I will wait on @BBlack when he is done decommissioning lvs2003 and lvs2006 to proceed [17:47:23] hmm or it's just a load issue and eqiad is idle enough during EU mornings [19:54:45] 10netops, 10Operations, 10Wikimedia-Incident: Investigate Juniper storm control - https://phabricator.wikimedia.org/T245192 (10Papaul) Next step is to create a single profile named "wmf-mgmt-storm", configure a storm control bandwidth of 15,000Kbps and add all the interfaces except the interface connected t...