[01:47:06] 06Traffic, 06Data-Platform: 4030 on http://dumps.wikimedia.org - https://phabricator.wikimedia.org/T379990 (10Platonides) 03NEW [01:48:21] 06Traffic, 06Data-Platform: 403 on http://dumps.wikimedia.org - https://phabricator.wikimedia.org/T379990#10325103 (10Pppery) [12:46:09] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: CloudVPS: IPv6 in codfw1dev - https://phabricator.wikimedia.org/T245495#10326497 (10aborrero) [12:48:48] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: openstack: work out IPv6 and designate integration - https://phabricator.wikimedia.org/T374715#10326506 (10aborrero) 05Stalled→03Resolved a:03aborrero this was done by means of {T378192} [12:51:53] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: dns: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10326522 (10aborrero) 05In progress→03Resolved [12:52:19] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: CloudVPS: IPv6 in codfw1dev - https://phabricator.wikimedia.org/T245495#10326524 (10aborrero) [12:53:57] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: CloudVPS: IPv6 in codfw1dev - https://phabricator.wikimedia.org/T245495#10326527 (10aborrero) 05Open→03Resolved a:03aborrero I think we can consider IPv6 to be fully working on codfw1dev. [13:03:48] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Add Dell switches support to Homer/Cookbooks - https://phabricator.wikimedia.org/T320638#10326595 (10ayounsi) 05Stalled→03Declined Because of the various limitations listed in {T340045} we're not going to proceed any further on Del... [13:04:53] 10netops, 06Infrastructure-Foundations, 06SRE: Put Dell SONiC switches in production - https://phabricator.wikimedia.org/T335028#10326604 (10ayounsi) 05Stalled→03Declined Because of the various limitations listed in {T342673} (plus the ones from pygnmi) we're not going to proceed any further on Dell... [14:16:56] 06Traffic, 10Beta-Cluster-Infrastructure: Beta cluster is down, 2024-11-15 edition - https://phabricator.wikimedia.org/T380056#10326896 (10hashar) //The issue seems related to Apache Traffic server on `deployment-cache-text08.deployment-prep.eqiad1.wikimedia.cloud` and I guess #traffic team might be able to as... [14:18:58] 06Traffic, 10Beta-Cluster-Infrastructure: Beta cluster is down, 2024-11-15 edition - https://phabricator.wikimedia.org/T380056#10326903 (10ssingh) Thanks for the task. We are aware of this issue and trying some fixes, including having reported this upstream https://github.com/apache/trafficserver/issues/11869. [14:32:02] 06Traffic, 10Beta-Cluster-Infrastructure: Beta cluster is down, 2024-11-15 edition - https://phabricator.wikimedia.org/T380056#10326994 (10hashar) Looking at `/var/log/apache2/other_vhosts_access.log`, on `deployment-mediawiki13` the last entry is at 6:21, but on `deployment-mediawiki14` there are accesses bei... [14:34:08] 06Traffic, 10Beta-Cluster-Infrastructure: Beta cluster is down, 2024-11-15 edition - https://phabricator.wikimedia.org/T380056#10326997 (10ssingh) >>! In T380056#10326994, @hashar wrote: > Looking at `/var/log/apache2/other_vhosts_access.log`, on `deployment-mediawiki13` the last entry is at 6:21, but on `depl... [14:34:17] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: Decom prod infra side of the ulsfo-office link - https://phabricator.wikimedia.org/T379778#10326998 (10RobH) > "Created by: cbissell Cross connect has been disconnected, per our policy it will be removed after 48 hours" I'... [14:41:06] 06Traffic, 10Beta-Cluster-Infrastructure: Beta cluster is down, 2024-11-15 edition - https://phabricator.wikimedia.org/T380056#10327016 (10hashar) The #beta-cluster-infrastructure uses unattended upgrade and packages uploaded to apt.wikimedia.org are thus installed automatically at the next 6:20am. That was d... [14:42:44] 06Traffic, 10Beta-Cluster-Infrastructure: Beta cluster is down, 2024-11-15 edition - https://phabricator.wikimedia.org/T380056#10327019 (10ssingh) >>! In T380056#10327016, @hashar wrote: > The #beta-cluster-infrastructure uses unattended upgrade and packages uploaded to apt.wikimedia.org are thus installed aut... [15:47:04] 10netops, 06Infrastructure-Foundations, 06SRE: Manange fundraising network elements from Netbox - https://phabricator.wikimedia.org/T377996#10327189 (10cmooney) [15:47:06] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 06SRE: Manage frack switches with Netbox - https://phabricator.wikimedia.org/T268802#10327190 (10cmooney) [16:02:27] 06Traffic, 10Beta-Cluster-Infrastructure: Beta cluster is down, 2024-11-15 edition - https://phabricator.wikimedia.org/T380056#10327305 (10ssingh) This should now be fixed. Thanks for filing the task and sorry about the change; I didn't expect beta to be updated (and I guess we just got lucky in the past)... [16:02:59] 06Traffic, 10Beta-Cluster-Infrastructure: Beta cluster is down, 2024-11-15 edition - https://phabricator.wikimedia.org/T380056#10327307 (10ssingh) 05Open→03Resolved a:03ssingh [16:12:38] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10327369 (10RobH) They confirmed receipt of the message/ticket but did not confirm the date/time specifically so I've sent a followup just now: > Suppo... [17:23:28] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10327708 (10RobH) Window confirmed! > Comentário gerado em Smart Hands: Hi Robert, good afternoon; > We can assist you at these times. [19:56:53] 06Traffic, 10Observability-Alerting, 06SRE: PuppetFailure alert is not being fired for host(s) where agent has failed - https://phabricator.wikimedia.org/T379807#10328186 (10ssingh) I will test this on Monday and then mark this as resolved but I think this should be fixed. Much thanks to @colewhite for the...