[07:49:21] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443#11458860 (10JAllemandou) In the meatime I'm going to reenable the indexation job with more resources, allowing it not to...
[09:50:54] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443#11459188 (10cmooney) >>! In T412443#11458860, @JAllemandou wrote: > Either sampling more, or keeping one week of data fo...
[10:23:45] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443#11459310 (10ayounsi) >>! In T412443#11459188, @cmooney wrote: >>>! In T412443#11458860, @JAllemandou wrote: >> Either sa...
[13:33:48] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443#11460057 (10JAllemandou) I've been spending time on this, here's my latest finding: By reducing retention to 30 days, we...
[13:48:28] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443#11460139 (10cmooney) >>! In T412443#11460057, @JAllemandou wrote: > I'll keep the conf as is (30 days retention), and wi...
[13:57:22] <wikibugs>	 06Traffic, 10MediaWiki-Debug-Logger, 06MediaWiki-Platform-Team: Pass through information about the client from the CDN to MediaWiki to Logstash - https://phabricator.wikimedia.org/T412396#11460238 (10Tgr)
[15:14:59] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade  Decom Asw Switches in Rows C & D - https://phabricator.wikimedia.org/T412525#11460664 (10Jclark-ctr) p:05Triage→03Medium a:03Jclark-ctr
[15:15:16] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade  Decom Asw Switches in Rows C & D - https://phabricator.wikimedia.org/T412525#11460667 (10Jclark-ctr) @cmooney  i have moved all Scs cables from nokia switches to Juniper switches in rows c/d
[15:15:27] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade  Decom Asw Switches in Rows C & D - https://phabricator.wikimedia.org/T412525#11460668 (10Jclark-ctr) a:05Jclark-ctr→03cmooney
[15:22:55] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Decom Juniper EX/QFX switches in eqiad rows C/D - https://phabricator.wikimedia.org/T412271#11460722 (10Jclark-ctr) 05Open→03Resolved a:03Jclark-ctr Alll 4 links have been removed physically and deleted from netbox.     Scs c...
[15:36:56] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143#11460805 (10ayounsi) p:05Triage→03Medium
[15:37:34] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443#11460806 (10ayounsi) p:05Triage→03Medium
[15:39:04] <wikibugs>	 06Traffic, 13Patch-For-Review: x-provenance header: identify WMCS - https://phabricator.wikimedia.org/T411503#11460815 (10SLyngshede-WMF) p:05Triage→03Low a:03SLyngshede-WMF
[15:39:15] <wikibugs>	 06Traffic, 13Patch-For-Review: x-provenance header: identify WMCS - https://phabricator.wikimedia.org/T411503#11460817 (10LSobanski)
[16:21:47] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733 (10cmooney) 03NEW p:05Triage→03Medium
[16:22:00] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#11461053 (10cmooney)
[16:23:49] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#11461064 (10cmooney)
[16:23:55] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#11461066 (10cmooney)
[16:24:04] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#11461068 (10cmooney)
[16:24:20] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#11461070 (10cmooney)
[16:39:43] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443#11461141 (10JAllemandou) a:03JAllemandou
[17:11:46] <wikibugs>	 06Traffic, 06MW-Interfaces-Team, 06serviceops, 07Epic, and 3 others: Epic: Enforce API rate limits (WE5.1.3c) - https://phabricator.wikimedia.org/T412585#11461265 (10Clement_Goubert) >>! In T412585#11458421, @matmarex wrote: > I haven't been able to find this in the related tasks: what does the error respo...
[18:04:08] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: mr1-codfw: add second uplink to lsw1-a2-codfw - https://phabricator.wikimedia.org/T410717#11461432 (10Jhancock.wm) @cmooney i checked what we had in stock. We don't have that SFP. and could use a new 5 meter fiber for this task.
[19:44:39] <wikibugs>	 06Traffic, 06MW-Interfaces-Team, 06serviceops, 07Epic, and 3 others: Epic: Enforce API rate limits (WE5.1.3c) - https://phabricator.wikimedia.org/T412585#11461635 (10daniel) >>! In T412585#11458421, @matmarex wrote: > is there some way to "force" an error response to test client libraries?  Hm... That is u...
[23:49:49] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 06SRE: Improve port-utilisation alerting to take QoS into account - https://phabricator.wikimedia.org/T384052#11462541 (10cmooney) This has come up again in terms of the pages we have been getting of late, and we may take some action to chan...