[07:20:45] <wikibugs>	 06Traffic, 06DBA, 06Infrastructure-Foundations: Move orchestrator (dborch) to private ipaddrs + CDN - https://phabricator.wikimedia.org/T317179#11822878 (10ayounsi) It's great to see progress on this ! Unless there are other blockers I'm unaware off, CDN seems better than LVS as it doesn't require to use a n...
[08:08:15] <wikibugs>	 06Traffic, 06DBA, 06Infrastructure-Foundations: Move orchestrator (dborch) to private ipaddrs + CDN - https://phabricator.wikimedia.org/T317179#11823008 (10FCeratto-WMF) @ssingh the main point is to minimize the attack surface, not exposing orchestrator directly on a public ipaddr.
[08:30:08] <wikibugs>	 06Traffic, 06collaboration-services, 10Gerrit, 06Release-Engineering-Team, and 2 others: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11823059 (10ABran-WMF) Verbose logging has been enabled on Envoy on `gerrit2003`: ` curl -i -X POST 'http://localhost:96...
[08:38:58] <wikibugs>	 10netops, 06Infrastructure-Foundations: Investigate internal rejected prefixes - https://phabricator.wikimedia.org/T423384 (10ayounsi) 03NEW p:05Triage→03Low
[08:39:48] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10observability, 10Prod-Kubernetes, and 5 others: Increase visibility of kubernetes network status - https://phabricator.wikimedia.org/T356877#11823153 (10ayounsi) > Regarding the count of "prefixes received by the switch but not accepted", I think as a first step it...
[10:44:21] <wikibugs>	 06Traffic, 10Data-Services, 10Datasets-General-or-Unknown, 06tools-infrastructure-team, 13Patch-For-Review: Migrate clouddumps https/rsync interfaces behind LVS - https://phabricator.wikimedia.org/T422040#11823716 (10taavi) 05Open→03Resolved
[13:17:35] <wikibugs>	 06Traffic, 06DBA, 06Infrastructure-Foundations: Move orchestrator (dborch) to private ipaddrs + CDN - https://phabricator.wikimedia.org/T317179#11824211 (10ssingh) >>! In T317179#11823008, @FCeratto-WMF wrote: > @ssingh the main point is to minimize the attack surface, not exposing orchestrator directly on a...
[13:17:57] <wikibugs>	 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: hardware troubleshooting: NVMe errors on cp1115.eqiad.wmnet - https://phabricator.wikimedia.org/T421007#11824216 (10ssingh) Hi @VRiley-WMF: any updates from Dell's side? Thanks!
[14:08:11] <wikibugs>	 06Traffic, 06collaboration-services, 10Gerrit, 06Release-Engineering-Team, and 2 others: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11824498 (10ABran-WMF) `quibble-with-gated-extensions-vendor-mysql-php83/27335` also failed during the clone phase with...
[14:53:19] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: No not announce OSPF routes in unicast BGP on Nokia SR-Linux - https://phabricator.wikimedia.org/T423430 (10cmooney) 03NEW p:05Triage→03Low
[14:53:23] <wikibugs>	 10netops, 06Infrastructure-Foundations: Investigate internal rejected prefixes - https://phabricator.wikimedia.org/T423384#11824787 (10cmooney)
[14:53:26] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: No not announce OSPF routes in unicast BGP on Nokia SR-Linux - https://phabricator.wikimedia.org/T423430#11824786 (10cmooney)
[14:59:11] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Don't announce OSPF routes in unicast BGP on Nokia SR-Linux - https://phabricator.wikimedia.org/T423430#11824827 (10cmooney)
[16:21:30] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Standardize management routers interfaces - https://phabricator.wikimedia.org/T421674#11825509 (10VRiley-WMF) I have plugged in 1 QFX-SFP-1GE-T into mr1-eqiad ge-0/0/7
[16:25:00] <wikibugs>	 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: hardware troubleshooting: NVMe errors on cp1115.eqiad.wmnet - https://phabricator.wikimedia.org/T421007#11825529 (10VRiley-WMF) Hey @ssingh thanks for checking. We just got the part in today (it was supposed to be here yesterday). I will be swapping it shortly. I wi...
[16:29:17] <wikibugs>	 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: hardware troubleshooting: NVMe errors on cp1115.eqiad.wmnet - https://phabricator.wikimedia.org/T421007#11825535 (10VRiley-WMF) 05Open→03In progress
[16:54:29] <wikibugs>	 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: hardware troubleshooting: NVMe errors on cp1115.eqiad.wmnet - https://phabricator.wikimedia.org/T421007#11825695 (10VRiley-WMF) Part has been installed and it should be good to go. I checked it in iDRAC and it sees both of the drives. Would you be able to check it o...
[18:11:57] <wikibugs>	 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: hardware troubleshooting: NVMe errors on cp1115.eqiad.wmnet - https://phabricator.wikimedia.org/T421007#11826508 (10BCornwall) 05In progress→03Resolved I can confirm it's behaving properly now! Reimage worked just fine and I don't have any kernel errors any...
[19:49:43] <wikibugs>	 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: mr1-eqiad: move from OSPF to BGP - https://phabricator.wikimedia.org/T421238#11826839 (10cmooney) 05Resolved→03Open a:05Papaul→03cmooney Re-opening this so we can look at the automation changes that are needed to remove tthe OSPF stuff (see...