[01:49:01] 10netops, 10SRE, 10ops-codfw: ripe-atlas-codfw is down - https://phabricator.wikimedia.org/T267714 (10Papaul) @faidon do we have some documentation on the console configuration for the RIPE? - console baud rate - Type of cable to use to connect to the console I tried to use a Cisco console cable and a DB9 t... [01:54:53] 10netops, 10SRE, 10ops-codfw: ripe-atlas-codfw is down - https://phabricator.wikimedia.org/T267714 (10CDanis) I'm pretty sure the baud rate is 19200 Not sure about the cable type [01:59:35] 10netops, 10SRE, 10ops-codfw: ripe-atlas-codfw is down - https://phabricator.wikimedia.org/T267714 (10Papaul) I tried both 9600 and 19200 on both cable it didn't work [08:01:41] 10Traffic, 10SRE: Sudden surge of requests to https://wikipedia.org/ from Telus customers - https://phabricator.wikimedia.org/T276213 (10elukey) [08:06:34] 10Traffic, 10SRE: Sudden surge of requests to https://wikipedia.org/ from Telus customers - https://phabricator.wikimedia.org/T276213 (10elukey) > There is a similar spike at around the same time during the Pacific night, between 8 and 9 UTC on January 7th, but I'm not sure if we can still verify what was in t... [08:54:43] 10Traffic, 10Desktop Improvements, 10SRE, 10Bengali-Sites, and 4 others: CDN cache revalidation on several wikis for desktop improvements deployment pt 2 - https://phabricator.wikimedia.org/T274784 (10ovasileva) >>! In T274784#6875627, @BBlack wrote: > @ovasileva Yes, that plan seems reasonable! > > Just... [10:59:42] 10Traffic, 10netops, 10Data-Persistence-Backup, 10SRE, and 2 others: Depool codfw swift cluster - https://phabricator.wikimedia.org/T267338 (10jcrespo) [11:00:20] 10Traffic, 10netops, 10Data-Persistence-Backup, 10SRE, and 2 others: Depool codfw swift cluster - https://phabricator.wikimedia.org/T267338 (10jcrespo) [11:43:49] 10netops, 10SRE, 10ops-codfw: ripe-atlas-codfw is down - https://phabricator.wikimedia.org/T267714 (10faidon) I believe the Atlas is a PCEngines APU, so you'll need a null modem cable or adapter (RXD->TXD, TXD->RXD, etc.) If this is a Cisco rollover cable, it would do the trick, but your DB9<->RJ45 adapter s... [13:14:05] 10Traffic, 10netops, 10Data-Persistence-Backup, 10SRE, and 2 others: Depool codfw swift cluster - https://phabricator.wikimedia.org/T267338 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This has happened! We're back to swift active/active, tentatively resolving [14:52:22] https://blog.benjojo.co.uk/post/ping-with-loss-latency-split [15:47:00] 10netops, 10SRE, 10ops-codfw: ripe-atlas-codfw is down - https://phabricator.wikimedia.org/T267714 (10Papaul) Thank you for the information. I will try to work on it again when i am back on site tomorrow. [16:00:08] 10Traffic, 10Desktop Improvements, 10SRE, 10Bengali-Sites, and 4 others: CDN cache revalidation on several wikis for desktop improvements deployment pt 2 - https://phabricator.wikimedia.org/T274784 (10Vito-Genovese) Could this possibly turn out to be a solution for the issue described at T119366? [16:45:21] 10Traffic, 10SRE, 10Patch-For-Review: Remove SLAAC IPs from Ganeti hosts - https://phabricator.wikimedia.org/T265904 (10jbond) I have applied the changes in the import script and tested against against [[ https://netbox.wikimedia.org/dcim/devices/2552/ | ganeti224]] and i now see the Primary IPv6: 2620:0:860... [17:14:00] 10Traffic, 10DNS, 10SRE, 10Abstract Wikipedia (Phase δ): Establish wikifunctions.org - https://phabricator.wikimedia.org/T275904 (10DVrandecic) [17:20:37] 10Traffic, 10SRE, 10Patch-For-Review: Remove SLAAC IPs from Ganeti hosts - https://phabricator.wikimedia.org/T265904 (10Volans) Nice! I think that the script deletes ifaces not present in PuppetDB but doesn't do that for IPs... maybe we should also do that. Thoughts? [22:09:16] 10netops, 10SRE: migrate services from bast1002 to bast1003 - https://phabricator.wikimedia.org/T276399 (10RobH) p:05Triage→03Medium [22:09:33] 10netops, 10SRE: migrate services from bast1002 to bast1003 - https://phabricator.wikimedia.org/T276399 (10RobH)