[07:16:04] 10netops, 10SRE, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10Marostegui) p:05Triage→03Medium [07:38:02] 10netops, 10SRE, 10User-jbond: Sporatic RST drops in the ulogd logs - https://phabricator.wikimedia.org/T238823 (10Marostegui) [07:47:31] 10netops, 10SRE, 10User-jbond: Sporadic RST drops in the ulogd logs - https://phabricator.wikimedia.org/T238823 (10Nemo_bis) [09:53:10] 10Traffic, 10Performance-Team (Radar): Strip new X-Request-Id header from non-debug responses - https://phabricator.wikimedia.org/T283291 (10Marostegui) p:05Triage→03Medium [10:09:23] 10Traffic, 10SRE: Let's Encrypt issuance chains update - https://phabricator.wikimedia.org/T283164 (10Marostegui) p:05Triage→03Medium [11:39:24] 10Traffic: OpenSSL < 1.1.0 compatibility issues with new LE issuance chain - https://phabricator.wikimedia.org/T283165 (10Marostegui) [11:41:42] 10Traffic, 10observability, 10User-fgiunchedi: Port traffic/netops grafana alerts to AM - https://phabricator.wikimedia.org/T282806 (10Marostegui) [11:47:13] 10Traffic, 10Patch-For-Review: Make Netbox Active/Active - https://phabricator.wikimedia.org/T234997 (10Marostegui) p:05Triage→03Medium [13:37:37] 10Traffic, 10netops: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10ssingh) [13:48:23] 10Traffic, 10netops, 10SRE: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10ssingh) [13:48:26] 10Traffic, 10SRE, 10Patch-For-Review: Offer Wikidough as an anycasted service - https://phabricator.wikimedia.org/T283027 (10ssingh) [14:41:09] 10Traffic, 10netops, 10SRE: Unable to load en.wikipedia.org from 84.19.61.192/26 - https://phabricator.wikimedia.org/T279503 (10A189605) This is still very much an issue unfortunately. I have connected my laptop into a layer 2 switch connected to our ISPs CPE and attempted to access en.wikipedia.org. Still s... [15:30:25] 10netops, 10SRE, 10ops-codfw: Multiple host down alerts from rack C2 - https://phabricator.wikimedia.org/T279457 (10Papaul) 05Open→03Resolved Received the new switch. Resolving this task {F34466158} [15:54:05] 10netops, 10SRE, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10jbond) > Is simple query success + rtt enough? If so, the built-in functionality of atlas_exporter is sufficient. Definitely for the first inst... [15:56:36] 10Traffic, 10netops, 10SRE: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10cmooney) Hey Sukhbir, Can you confirm "doh2001" and "doh2002" are the VMs which will Anycast the Wikidough IP? I'm not sure using a static route is the best idea. It's bette... [15:58:06] 10Traffic, 10netops, 10SRE: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10Marostegui) p:05Triage→03Medium [16:12:20] 10Traffic, 10Privacy Engineering, 10SRE: Create and document Wikidough's privacy policy - https://phabricator.wikimedia.org/T275409 (10JFishback_WMF) [16:15:41] 10Traffic, 10netops, 10SRE: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10ssingh) >>! In T283503#7107926, @cmooney wrote: > Hey Sukhbir, > > Can you confirm "doh2001" and "doh2002" are the VMs which will Anycast the Wikidough IP? Sorry, I should hav... [16:48:07] 10Traffic, 10netops, 10SRE: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10cmooney) I spoke with Arzhel about it and he's pointed me in the right direction. I think your change is good to merge, however we don't want to have a huge time gap between m... [17:00:01] 10Traffic, 10netops, 10SRE: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10ssingh) >>! In T283503#7108144, @cmooney wrote: > I spoke with Arzhel about it and he's pointed me in the right direction. > > I think your change is good to merge, however we... [17:15:14] 10Traffic, 10netops, 10SRE: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10cmooney) Change created: https://gerrit.wikimedia.org/r/c/operations/homer/public/+/693923 [17:45:24] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10Gilles) 05Open→03Resolved [17:59:29] 10netops, 10SRE, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10cmooney) Was looking at the NSID stuff based on John's suggestion, indeed it is returned by our authdns at the moment: ` root@nyc2:~# for i in {... [18:09:09] 10netops, 10SRE, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10CDanis) Thanks very much for all the helpful comments! * I'll definitely enable the NSID bit, it will be very useful especially for mapping the... [19:43:31] 10netops, 10SRE, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10jbond) >>! In T283359#7108488, @CDanis wrote: > Thanks very much for all the helpful comments! > > * I'll definitely enable the NSID bit, it wi... [20:59:19] 10netops, 10SRE, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10CDanis) Some very initial results running against the anycast NS at https://atlas.ripe.net/frames/measurements/30366093#!probes Thanks again fo...