[08:30:15] <wikibugs>	 10Traffic, 10Analytics, 10SRE: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10hashar) That is still hitting us from time to time, specially when the CI cache is not warmed up or has been wiped.
[08:49:49] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10ema) Looking at average p75 over the last two days, all varnish versions in text@esams now perform very similarly:  |**host**...
[10:27:44] <wikibugs>	 10netops, 10SRE: Lumen 10G Wave (cr2-eqiad to cr2-esams) Down - https://phabricator.wikimedia.org/T283227 (10cmooney) Lumen confirmed fix, we have been stable since.  Closing task.  __________________________________________________________________________________ From: No-Reply via Ops Maintenance [mailto:ops...
[10:28:40] <wikibugs>	 10netops, 10SRE: Lumen 10G Wave (cr2-eqiad to cr2-esams) Down - https://phabricator.wikimedia.org/T283227 (10cmooney) 05Open→03Resolved
[12:00:49] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10Gilles) Thank you very much for the thorough investigation and explanations, I agree that now the issue is well understood and...
[13:34:13] <wikibugs>	 10Traffic, 10Analytics, 10SRE: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10Ottomata) Moving back to incoming for re-triage
[14:12:34] <wikibugs>	 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10cmooney) I've been looking into this issue a litte, and propose to do some tests Monday/Tuesday AM (Europ...
[14:12:50] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review: Offer Wikidough as an anycasted service - https://phabricator.wikimedia.org/T283027 (10ssingh)
[14:33:44] <wikibugs>	 10netops, 10SRE, 10observability: Create RIPE Atlas probes against our public DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10CDanis)
[14:34:16] <wikibugs>	 10netops, 10SRE, 10observability: Create RIPE Atlas measurements against our public DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10CDanis)
[14:34:24] <wikibugs>	 10netops, 10SRE, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10CDanis)
[14:47:10] <wikibugs>	 10Traffic, 10SRE: OpenSSL < 1.1.0 compatibility issues with new LE issuance chain - https://phabricator.wikimedia.org/T283165 (10Vgutierrez) As mentioned on the issue description, debian backported the fix for OpenSSL as it can be seen on a current debian jessie container: ` root@69310d82543d:~# cat /etc/debia...
[15:01:50] <wikibugs>	 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10jcrespo) Thank you very much for your comments.  > Due to this I would like to test the performance, usin...
[15:06:20] <wikibugs>	 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10cmooney) Ok @jcrespo sounds like a plan.  And thanks for the extra info, indeed it does seem to rule out...
[15:12:18] <wikibugs>	 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10jcrespo) >>! In T274234#7103617, @cmooney wrote: > In terms of the WAN links I wouldn't rule them out 100...
[15:34:46] <wikibugs>	 10Traffic, 10netops, 10SRE: Unable to load en.wikipedia.org from 84.19.61.192/26 - https://phabricator.wikimedia.org/T279503 (10cmooney) Is this issue still ongoing?  A brief test reveals we appear to be able to route back to this network from the datacentre in question:  ` cmooney@lvs3005:~$ mtr --address 9...