[01:11:47] 10Traffic, 10SRE, 10vm-requests: Please create two Ganeti VMs for Wikidough - https://phabricator.wikimedia.org/T283192 (10Dzahn) 05Open→03Resolved VMs have been created, added to site.pp with "insetup", added to DHCP and partma. OS has been installed (buster) and puppet certs signed. You can now SSH to... [08:24:33] 10netops: Lumen 10G Wave BDFS2448 (cr2-eqiad to cr2-esams) Down - https://phabricator.wikimedia.org/T283227 (10cmooney) p:05Triage→03Medium a:03cmooney [08:28:44] 10netops: Lumen 10G Wave BDFS2448 (cr2-eqiad to cr2-esams) Down - https://phabricator.wikimedia.org/T283227 (10cmooney) Traffic has moved to GTT VPLS service, now running from cr1-eqiad xe-4/2/2.13 to cr3-knams xe-0/1/5.13 and seems stable. As a result it does not appear any services are adversely affected othe... [08:30:44] 10netops: Lumen 10G Wave BDFS2448 (cr2-eqiad to cr2-esams) Down - https://phabricator.wikimedia.org/T283227 (10cmooney) Light level on cr2-eqiad for the record: {F34461498} [08:33:20] 10netops: Lumen 10G Wave (cr2-eqiad to cr2-esams) Down - https://phabricator.wikimedia.org/T283227 (10cmooney) [11:21:11] 10netops, 10SRE: routinator: create gabage collection job - https://phabricator.wikimedia.org/T282469 (10jbond) >>! In T282469#7099149, @ayounsi wrote: > @jbond would the upcoming changes in https://github.com/NLnetLabs/routinator/releases/tag/v0.9.0-rc1 solve that issue by using a database instead of the file... [11:31:06] 10Traffic, 10SRE, 10vm-requests: Please create two Ganeti VMs for Wikidough - https://phabricator.wikimedia.org/T283192 (10ssingh) >>! In T283192#7099749, @Dzahn wrote: > VMs have been created, added to site.pp with "insetup", added to DHCP and partma. > > OS has been installed (buster) and puppet certs sig... [12:42:09] 10Traffic, 10SRE: OpenSSL < 1.1.0 compatibility issues with new LE issuance chain - https://phabricator.wikimedia.org/T283165 (10BBlack) bump for testing purposes [12:46:59] :) [15:24:32] 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10faidon) p:05Triage→03High Given a) this was linked during budgeting in the context of of our cross-DC... [15:33:28] 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10jcrespo) One of the things I raised to my manager is that this limitation means that, in the event of a c... [16:16:26] 10netops, 10SRE: Lumen 10G Wave (cr2-eqiad to cr2-esams) Down - https://phabricator.wikimedia.org/T283227 (10cmooney) Came back up approx 40 mins ago: ` May 20 15:37:52 re0.cr2-eqiad mib2d[13184]: SNMP_TRAP_LINK_UP: ifIndex 660, ifAdminStatus up(1), ifOperStatus up(1), ifName xe-4/1/3 ` ` cmooney@re0.cr2-eqia... [19:42:29] 10Traffic, 10SRE, 10Patch-For-Review: Offer Wikidough as an anycasted service - https://phabricator.wikimedia.org/T283027 (10ssingh) [21:27:20] 10Traffic, 10SRE, 10Performance-Team (Radar): Strip new X-Request-Id header from non-debug responses - https://phabricator.wikimedia.org/T283291 (10Krinkle)