[02:27:04] 06Traffic, 06Data-Persistence, 06SRE, 10SRE-swift-storage, and 6 others: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914#10742000 (10MikhailRyazanov) By the way, are there any reasons, besides historical, to specify image sizes in “pixels” (which nowadays often don't corresp... [03:58:20] 06Traffic, 10Citoid, 06Editing-team, 10RESTBase Sunsetting, and 3 others: Switch from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10742054 (10Ryasmeen) [06:09:00] 10netops, 06Infrastructure-Foundations: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#10742262 (10ayounsi) More vulns : https://supportportal.juniper.net/s/article/2025-04-Security-Bulletin-Junos-OS-and-Junos-OS-Evolved-A-specific-CLI-command-will-cause-a-RPD-crash-when-rib-sh... [07:58:00] FIRING: [11x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [08:03:00] FIRING: [16x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [08:03:40] FIRING: [3x] VarnishHighThreadCount: Varnish's thread count on cp5018:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [08:08:00] RESOLVED: [32x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [08:08:40] RESOLVED: [3x] VarnishHighThreadCount: Varnish's thread count on cp5018:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [08:38:01] FIRING: [15x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [08:43:01] FIRING: [16x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [08:48:01] RESOLVED: [28x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [09:27:00] FIRING: [2x] PurgedHighEventLag: High event process lag with purged on cp5018:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [09:32:00] RESOLVED: [32x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [10:52:25] 06Traffic, 13Patch-For-Review: varnish 7.1.1 crash - https://phabricator.wikimedia.org/T391334#10742962 (10Vgutierrez) [11:03:02] 06Traffic, 06[Archived]Wikidata Dev Team, 10Prod-Kubernetes, 06SRE, and 4 others: Frequent 500 Errors and Timeouts When Adding Statements to New Item or Lexeme-typed Properties - https://phabricator.wikimedia.org/T374230#10742996 (10Ifrahkhanyaree_WMDE) [11:04:39] 06Traffic, 06Data-Persistence, 06SRE, 10SRE-swift-storage, and 6 others: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914#10743021 (10Ladsgroup) >>! In T355914#10738719, @hgzh wrote: > I tried an onwiki answer, so thank you for the reply here. But IMO this could have been ann... [11:07:14] 06Traffic, 13Patch-For-Review: varnish 7.1.1 crash - https://phabricator.wikimedia.org/T391334#10743039 (10Vgutierrez) [12:41:32] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting: Migrate network icinga alerts to gNMI/prometheus - https://phabricator.wikimedia.org/T388641#10743419 (10ayounsi) For OSPF it looks like the interface states are there, but not the neighbor states: {P75019} That's subscribing to `/network-ins... [13:04:42] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 13Patch-For-Review: Migrate network icinga alerts to gNMI/prometheus - https://phabricator.wikimedia.org/T388641#10743565 (10ayounsi) [13:24:00] 06Traffic: haproxy should set x-cache-status to int-tls even in tls frontend - https://phabricator.wikimedia.org/T391967 (10Fabfur) 03NEW [13:49:21] 06Traffic: Test ESI feasibility with current Varnish installation - https://phabricator.wikimedia.org/T308799#10743774 (10Jdforrester-WMF) Can the ESI test code now be removed from test2.wikipedia.org? [14:39:01] 06Traffic, 13Patch-For-Review: varnish 7.1.1 crash - https://phabricator.wikimedia.org/T391334#10744024 (10Vgutierrez) [15:31:14] 10Wikimedia-Apache-configuration, 10DNS, 06SRE: Unconfigured subdomains of wikimedia.org should display an error page rather than the wikimedia.org homepage - https://phabricator.wikimedia.org/T391016#10744291 (10Joe) 05Open→03Declined This was never the behaviour of our servers, as far back as I can... [15:34:14] 06Traffic, 06Data-Persistence, 06SRE, 10SRE-swift-storage, and 6 others: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914#10744312 (10TheDJ) >>! In T355914#10742000, @MikhailRyazanov wrote: > By the way, are there any reasons, besides historical, to specify image sizes in “pi... [16:20:40] 10netops, 06Infrastructure-Foundations, 06SRE: Create alerting for saturation on sub-rated interfaces - https://phabricator.wikimedia.org/T374614#10744550 (10cmooney) >>! In T374614#10707267, @cmooney wrote: >>>! In T374614#10147994, @ayounsi wrote: >> Short term I think if you add `[4Gbps]` to the interface... [16:29:37] 06Traffic: haproxy should set x-cache-status to int-tls even in tls frontend - https://phabricator.wikimedia.org/T391967#10744606 (10Fabfur) [18:02:40] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 10ops-eqiad: eqiad: second frack parent tracking task - https://phabricator.wikimedia.org/T392006 (10RobH) 03NEW p:05Triage→03High [18:04:52] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 10ops-eqiad: eqiad: determine second frack - https://phabricator.wikimedia.org/T392007 (10RobH) 03NEW [18:11:51] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 10ops-eqiad: eqiad: determine second frack - https://phabricator.wikimedia.org/T392007#10744966 (10RobH) [18:14:10] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 10ops-eqiad: eqiad: determine second frack - https://phabricator.wikimedia.org/T392007#10744975 (10RobH) @ayounsi & @cmooney: Per our conversation today in our codfw/eqiad buildout meetings, this was brought up and I've created th... [18:14:35] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 10ops-eqiad: eqiad: determine second frack - https://phabricator.wikimedia.org/T392007#10744979 (10RobH) @Jclark-ctr & @VRiley-WMF Per today's meeting, one of the action items was to have an eqiad onsite detrmine how many free cro... [18:28:48] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 10ops-eqiad: eqiad: determine second frack - https://phabricator.wikimedia.org/T392007#10745024 (10cmooney) >>! In T392007#10744966, @RobH wrote: > Please detail via comment specifically how using D6 would cause a network imbalance... [18:30:36] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: eqiad: determine second frack - https://phabricator.wikimedia.org/T392007#10745029 (10RobH) [18:31:26] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: eqiad: determine second frack - https://phabricator.wikimedia.org/T392007#10745033 (10RobH) [18:49:57] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: eqiad: second frack parent tracking task - https://phabricator.wikimedia.org/T392006#10745097 (10RobH) [18:49:59] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Migrate non-fundraising hosts out of eqiad D6 - https://phabricator.wikimedia.org/T390240#10745098 (10RobH) [18:50:28] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: eqiad: second frack parent tracking task - https://phabricator.wikimedia.org/T392006#10745104 (10RobH) Please note I've tied original task T390240 to this for ease of tracking. If rack D6 is not selected (likely wont b... [18:53:59] 06Traffic, 13Patch-For-Review: varnish 7.1.1 crash - https://phabricator.wikimedia.org/T391334#10745111 (10Vgutierrez) 05Open→03Stalled [18:55:55] 06Traffic, 13Patch-For-Review: varnish 7.1.1-1.1~bpo11+wmf1 crash - https://phabricator.wikimedia.org/T391334#10745118 (10Vgutierrez) [18:59:23] 06Traffic, 13Patch-For-Review: haproxy should set x-cache-status to int-tls even in tls frontend - https://phabricator.wikimedia.org/T391967#10745131 (10Fabfur) [19:12:26] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: eqiad: determine second frack - https://phabricator.wikimedia.org/T392007#10745165 (10Jclark-ctr) @RobH we have 1 free cross connect circuit id 21996480 [19:13:12] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: eqiad: determine second frack - https://phabricator.wikimedia.org/T392007#10745171 (10Jclark-ctr) [19:27:13] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: eqiad: determine second frack - https://phabricator.wikimedia.org/T392007#10745240 (10RobH) [20:03:08] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: eqiad: determine second frack - https://phabricator.wikimedia.org/T392007#10745411 (10Jclark-ctr) [20:09:26] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: eqiad: determine second frack - https://phabricator.wikimedia.org/T392007#10745517 (10Jclark-ctr) [21:28:30] 10Domains, 06Traffic, 06SRE, 13Patch-For-Review: Acquire enwp.org - https://phabricator.wikimedia.org/T332220#10745745 (10BCornwall) 05Open→03Stalled Indeed.... too bad. Hopefully we'll hear back sooner rather than later! [22:47:38] FIRING: [4x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.112:443 @ cp4047 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS [22:52:38] RESOLVED: [4x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.112:443 @ cp4047 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS