[08:13:44] 10netops, 10Operations, 10ops-eqsin: cr3-eqsin to production - https://phabricator.wikimedia.org/T255766 (10ayounsi) 05Open→03Resolved a:03ayounsi Netbox updated. [08:14:43] 10netops, 10DC-Ops, 10Operations, 10ops-eqsin: (Need By: TBD) rack/setup/install cr3-eqsin.wikimedia.org - https://phabricator.wikimedia.org/T253246 (10ayounsi) 05Open→03Resolved I believe there is nothing left to do here, see T255766. [08:14:46] 10netops, 10Operations, 10ops-eqsin: cr3-eqsin to production - https://phabricator.wikimedia.org/T255766 (10ayounsi) [08:52:11] 10Traffic, 10Operations, 10Patch-For-Review, 10User-notice: several purgeds badly backlogged (> 10 days) - https://phabricator.wikimedia.org/T256444 (10ema) >>! In T256444#6268449, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operations), href=https://sal.toolforge.org/log/Q38pBnMBv... [08:56:29] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, 10SRE-swift-storage: Cached thumbnails and originals are sometimes not being purged correctly/quickly - https://phabricator.wikimedia.org/T256313 (10ema) p:05Triage→03Medium [09:12:31] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, 10SRE-swift-storage: Cached thumbnails and originals are sometimes not being purged correctly/quickly - https://phabricator.wikimedia.org/T256313 (10ema) The cause is most probably T256444. I'm saying this based on two important pieces of i... [09:17:27] 10Traffic, 10Analytics, 10Operations: Remove ganglia leftovers from ops/puppet - https://phabricator.wikimedia.org/T253555 (10ema) [09:19:11] 10Traffic, 10Analytics, 10Operations: Remove ganglia leftovers from ops/puppet - https://phabricator.wikimedia.org/T253555 (10ema) @fgiunchedi: the `puppetmaster` module still has some ganglia-related things such as `prometheus-ganglia-gen`. Is that still needed? [09:22:53] 10Traffic, 10Operations: Certain links being rejected by caching if opened in Internet Explorer with a HTTP 400 error - https://phabricator.wikimedia.org/T256302 (10ema) p:05Triage→03Medium [09:26:28] 10Traffic, 10Operations: Certain links being rejected by caching if opened in Internet Explorer with a HTTP 400 error - https://phabricator.wikimedia.org/T256302 (10ema) Interesting, I'd be inclined to think that the issue here cannot be simply the user agent or we would know. :-) - Is the problem reproducibl... [09:59:16] 10netops, 10Operations, 10Patch-For-Review, 10Sustainability (Incident Prevention): Juniper HA audit - https://phabricator.wikimedia.org/T191667 (10ayounsi) [10:01:53] 10Traffic, 10Analytics, 10Operations: Remove ganglia leftovers from ops/puppet - https://phabricator.wikimedia.org/T253555 (10fgiunchedi) >>! In T253555#6273943, @ema wrote: > @fgiunchedi: the `puppetmaster` module still has some ganglia-related things such as `prometheus-ganglia-gen`. Is that still needed?... [11:31:54] 10Traffic, 10Operations, 10Patch-For-Review, 10Sustainability (Incident Prevention): monitoring & alerting for purged - https://phabricator.wikimedia.org/T256446 (10ema) [11:37:31] 10Traffic, 10Operations, 10Patch-For-Review, 10Sustainability (Incident Prevention): monitoring & alerting for purged - https://phabricator.wikimedia.org/T256446 (10ema) @cdanis all done except for `rdkafka_consumer_topics_partitions_consumer_lag`, there's silence on grafana.wikimedia.org/explore when look... [11:50:46] 10netops, 10Operations, 10Patch-For-Review, 10Sustainability (Incident Prevention): Juniper HA audit - https://phabricator.wikimedia.org/T191667 (10ayounsi) [12:17:49] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad, 10cloud-services-team (Hardware): (Need By: 2020-06-12) rack/setup/install WMCS 10G switches - https://phabricator.wikimedia.org/T251632 (10ayounsi) @Jclark-ctr or @Cmjohnson some information are missing in Netbox about the cabling: See https://netbox.wikime... [12:37:18] 10Traffic, 10Analytics, 10Operations, 10Patch-For-Review: Remove ganglia leftovers from ops/puppet - https://phabricator.wikimedia.org/T253555 (10ema) [12:37:25] 10Traffic, 10Analytics, 10Operations, 10Patch-For-Review: Remove ganglia leftovers from ops/puppet - https://phabricator.wikimedia.org/T253555 (10ema) 05Open→03Resolved [13:58:55] 10Traffic, 10Operations, 10Sustainability (Incident Prevention): monitoring & alerting for purged - https://phabricator.wikimedia.org/T256446 (10CDanis) 05Open→03Resolved a:03CDanis I was thinking about this metric: `kafka_burrow_partition_lag{topic=~".*\\.resource-purge",group=~"cp.*"}` -- grafana exp... [13:58:58] 10Traffic, 10Operations, 10Patch-For-Review, 10User-notice: several purgeds badly backlogged (> 10 days) - https://phabricator.wikimedia.org/T256444 (10CDanis) [13:59:05] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review, and 2 others: Make CDN purges reliable - https://phabricator.wikimedia.org/T133821 (10CDanis) [14:52:16] 10Traffic, 10Operations, 10Services (watching), 10Sustainability (MediaWiki-MultiDC): Create HTTP verb and sticky cookie DC routing in VCL - https://phabricator.wikimedia.org/T91820 (10Krinkle) [17:22:15] 10netops, 10Analytics, 10Operations: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10faidon) [17:23:35] 10netops, 10Analytics, 10Operations: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10faidon) So - how do we make progress here? Any thoughts on who/how? :) Some of these features could really make a tremendous amount of difference to our network operati... [19:14:47] 10Traffic, 10Operations: Block hotlinking - https://phabricator.wikimedia.org/T152091 (10Nemo_bis) 05Open→03Declined For now I'm marking this declined for clarity, given the pro-hotlinking policy has been standing for 12 years unchallenged. Should the Wikimedia Foundation start a radical reconsideration of... [21:30:48] 10netops, 10Operations, 10ops-codfw: (Need by: End of July-2020 ) codfw:rack/setup/new management switches - https://phabricator.wikimedia.org/T253154 (10Papaul)