[08:30:15] 10Traffic, 10Analytics, 10SRE: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10hashar) That is still hitting us from time to time, specially when the CI cache is not warmed up or has been wiped. [08:49:49] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10ema) Looking at average p75 over the last two days, all varnish versions in text@esams now perform very similarly: |**host**... [10:27:44] 10netops, 10SRE: Lumen 10G Wave (cr2-eqiad to cr2-esams) Down - https://phabricator.wikimedia.org/T283227 (10cmooney) Lumen confirmed fix, we have been stable since. Closing task. __________________________________________________________________________________ From: No-Reply via Ops Maintenance [mailto:ops... [10:28:40] 10netops, 10SRE: Lumen 10G Wave (cr2-eqiad to cr2-esams) Down - https://phabricator.wikimedia.org/T283227 (10cmooney) 05Open→03Resolved [12:00:49] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10Gilles) Thank you very much for the thorough investigation and explanations, I agree that now the issue is well understood and... [13:34:13] 10Traffic, 10Analytics, 10SRE: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10Ottomata) Moving back to incoming for re-triage [14:12:34] 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10cmooney) I've been looking into this issue a litte, and propose to do some tests Monday/Tuesday AM (Europ... [14:12:50] 10Traffic, 10SRE, 10Patch-For-Review: Offer Wikidough as an anycasted service - https://phabricator.wikimedia.org/T283027 (10ssingh) [14:33:44] 10netops, 10SRE, 10observability: Create RIPE Atlas probes against our public DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10CDanis) [14:34:16] 10netops, 10SRE, 10observability: Create RIPE Atlas measurements against our public DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10CDanis) [14:34:24] 10netops, 10SRE, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10CDanis) [14:47:10] 10Traffic, 10SRE: OpenSSL < 1.1.0 compatibility issues with new LE issuance chain - https://phabricator.wikimedia.org/T283165 (10Vgutierrez) As mentioned on the issue description, debian backported the fix for OpenSSL as it can be seen on a current debian jessie container: ` root@69310d82543d:~# cat /etc/debia... [15:01:50] 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10jcrespo) Thank you very much for your comments. > Due to this I would like to test the performance, usin... [15:06:20] 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10cmooney) Ok @jcrespo sounds like a plan. And thanks for the extra info, indeed it does seem to rule out... [15:12:18] 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10jcrespo) >>! In T274234#7103617, @cmooney wrote: > In terms of the WAN links I wouldn't rule them out 100... [15:34:46] 10Traffic, 10netops, 10SRE: Unable to load en.wikipedia.org from 84.19.61.192/26 - https://phabricator.wikimedia.org/T279503 (10cmooney) Is this issue still ongoing? A brief test reveals we appear to be able to route back to this network from the datacentre in question: ` cmooney@lvs3005:~$ mtr --address 9...