[00:37:18] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10matmarex) @bblack @ema @Vgutierrez Did you get a response on this from ATS developers? Did we file an issue or... [03:52:33] 10Traffic, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10Papaul) a:05BBlack→03Papaul [03:54:16] 10Traffic, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10Papaul) Firmware upgrade on lvs2008 Before BIOS Version 1.3.7 iDRAC Firmware Version 3.15.17.15 After BIOS Version 2.4.8 iDRAC Firmware Version 4.00.00 [10:20:44] 10Traffic, 10Operations, 10Pybal: Minor fixes in pybal checks - https://phabricator.wikimedia.org/T246431 (10jijiki) [10:58:43] 10Traffic, 10DBA, 10Operations, 10Patch-For-Review: Framework to transfer files over the LAN - https://phabricator.wikimedia.org/T156462 (10jcrespo) [10:58:50] 10HTTPS, 10Traffic, 10DBA, 10Operations, and 4 others: dbtree loads third party resources (from jquery.com and google.com) - https://phabricator.wikimedia.org/T96499 (10jcrespo) [12:09:00] Traffic, be ready: https://lists.gt.net/nanog/users/209454#209454 [12:09:28] "It's okay though, because we freed up UDP/53 by moving DNS to TCP/443, so then we can move HTTPS to UDP/53. " [12:15:53] what is wrong with HTTP/3 over IPv4? [12:26:20] I think providers blindly rate limiting UDP [12:52:56] lol [12:53:37] wait till DNS finally catches up and realizes that DoT should transport over QUIC, too :) [12:55:47] but seriously, in the big picture, I wish standards-pushers would stop cowing to whatever bad behavior of the month ISPs are doing, and hacking up RFC decisions to match current broken-ness... [12:59:51] lol [16:42:18] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10Papaul) [16:44:16] 10Traffic, 10Operations, 10ops-codfw: (OoW) lvs2006 crashed into (what it seems) an unrecoverable state - https://phabricator.wikimedia.org/T209337 (10Papaul) 05Stalled→03Declined Server is decommissioned in T246329 [16:44:57] 10Traffic, 10DC-Ops, 10Operations, 10ops-codfw: (OoW) lvs2006 Embedded Flash/SD-CARD iLO errors - https://phabricator.wikimedia.org/T192082 (10Papaul) 05Open→03Declined Server is decommissioned in T246329 [19:52:24] 10Traffic, 10netops, 10Operations, 10ops-codfw: switch port configuration for lvs200[7-10] - https://phabricator.wikimedia.org/T196946 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` lvs2008.codfw.wmnet ` The log can be found in `/var/log/wmf-auto... [20:34:37] 10Traffic, 10netops, 10Operations, 10ops-codfw: switch port configuration for lvs200[7-10] - https://phabricator.wikimedia.org/T196946 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['lvs2008.codfw.wmnet'] ` and were **ALL** successful. [20:35:41] 10Traffic, 10netops, 10Operations, 10ops-codfw: switch port configuration for lvs200[7-10] - https://phabricator.wikimedia.org/T196946 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` lvs2007.codfw.wmnet ` The log can be found in `/var/log/wmf-auto... [20:43:40] 10Traffic, 10netops, 10Operations, 10ops-codfw: switch port configuration for lvs200[7-10] - https://phabricator.wikimedia.org/T196946 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['lvs2007.codfw.wmnet'] ` Of which those **FAILED**: ` ['lvs2007.codfw.wmnet'] ` [20:45:17] 10Traffic, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10Papaul) @Vgutierrez lvs2008 is ready for service will can on lvs2007 on Monday