[00:03:50] jouncebot: nowandnext [00:03:50] No deployments scheduled for the next 5 hour(s) and 56 minute(s) [00:03:50] In 5 hour(s) and 56 minute(s): MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260501T0600) [00:05:05] (03PS1) 10Dzahn: zuul: rename zuul-nodepool systemd template to zuul-launcher, adjust it [puppet] - 10https://gerrit.wikimedia.org/r/1280829 (https://phabricator.wikimedia.org/T424879) [00:06:30] (03CR) 10Zabe: [C:03+2] Add script to fix fr_deleted drifts [extensions/WikimediaMaintenance] (wmf/1.46.0-wmf.26) - 10https://gerrit.wikimedia.org/r/1280417 (https://phabricator.wikimedia.org/T424553) (owner: 10Zabe) [00:07:28] 10SRE-swift-storage, 06Data-Persistence, 10MediaViewer, 10Thumbor, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11877620 (10Snaevar) >>! In T414805#11874323, @Nux wrote: > Things get deprecated for __years__ in traditional programming before the... [00:07:38] (03CR) 10Dzahn: [C:03+2] zuul: rename zuul-nodepool systemd template to zuul-launcher, adjust it [puppet] - 10https://gerrit.wikimedia.org/r/1280829 (https://phabricator.wikimedia.org/T424879) (owner: 10Dzahn) [00:09:21] (03Merged) 10jenkins-bot: Add script to fix fr_deleted drifts [extensions/WikimediaMaintenance] (wmf/1.46.0-wmf.26) - 10https://gerrit.wikimedia.org/r/1280417 (https://phabricator.wikimedia.org/T424553) (owner: 10Zabe) [00:09:39] RESOLVED: [2x] TransitBGPDown: Transit BGP session down between cr2-codfw and Hurricane Electric (2001:504:61::1b1b:0:1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [00:09:52] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1280417|Add script to fix fr_deleted drifts (T424553)]] [00:11:35] !log zabe@deploy1003 zabe: Backport for [[gerrit:1280417|Add script to fix fr_deleted drifts (T424553)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [00:12:26] (03PS1) 10Dzahn: zuul: remove nodepool-related code [puppet] - 10https://gerrit.wikimedia.org/r/1280832 (https://phabricator.wikimedia.org/T424879) [00:13:10] !log zabe@deploy1003 zabe: Continuing with deployment [00:13:45] FIRING: CirrusConsumerRerenderFetchErrorRate: cirrus_streaming_updater_consumer_cloudelastic_eqiad in eqiad (k8s): ... [00:13:45] fetch error (rerenders) rate too high - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=eqiad+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusConsumerRerenderFetchErrorRate [00:16:25] RESOLVED: SystemdUnitFailed: wmf_auto_restart_prometheus-blazegraph-exporter-wdqs-blazegraph.service on wdqs1018:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:16:45] RESOLVED: CirrusConsumerFetchErrorRate: cirrus_streaming_updater_consumer_cloudelastic_eqiad in eqiad (k8s): fetch error rate too high - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=eqiad+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusConsumerFetchErrorRate [00:16:57] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1280417|Add script to fix fr_deleted drifts (T424553)]] (duration: 07m 05s) [00:18:45] RESOLVED: CirrusConsumerRerenderFetchErrorRate: cirrus_streaming_updater_consumer_cloudelastic_eqiad in eqiad (k8s): ... [00:18:45] fetch error (rerenders) rate too high - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=eqiad+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusConsumerRerenderFetchErrorRate [00:19:08] (03CR) 10Dzahn: [C:03+2] zuul: remove nodepool-related code [puppet] - 10https://gerrit.wikimedia.org/r/1280832 (https://phabricator.wikimedia.org/T424879) (owner: 10Dzahn) [00:20:45] FIRING: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-cloudelastic@eqiad is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [00:26:51] (03PS1) 10Dzahn: zuul: update zuul-launcher version to 14.2.0-1 [puppet] - 10https://gerrit.wikimedia.org/r/1280840 (https://phabricator.wikimedia.org/T424879) [00:27:16] (03CR) 10Dzahn: [C:03+2] zuul: update zuul-launcher version to 14.2.0-1 [puppet] - 10https://gerrit.wikimedia.org/r/1280840 (https://phabricator.wikimedia.org/T424879) (owner: 10Dzahn) [00:28:12] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1008 is CRITICAL: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Max retries exceeded with url: /_cluster/health (Caused by NewConnectionError(urllib3.connection.HTTPConnection object at 0x7fbe45059550: Failed to establish a new connection: [Errno 111] Connection refused)) https://wikitec [00:28:12] dia.org/wiki/Search%23Administration [00:29:23] cccccbukvgbcrllddgftvetdjnkhrknevnnvcthvhlcg [00:30:25] FIRING: SystemdUnitFailed: opensearch_2@cloudelastic-chi-eqiad.service on cloudelastic1008:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:31:50] 10SRE-swift-storage, 06Data-Persistence, 10MediaViewer, 10Thumbor, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11877645 (10AntiCompositeNumber) > This process started with T211661 in 2022 with giving new thumbnails TTL (time to live). This was... [00:35:45] RESOLVED: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-cloudelastic@eqiad is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [00:36:45] FIRING: CirrusStreamingUpdaterFlinkJobUnstable: cirrus_streaming_updater_consumer_cloudelastic_eqiad in eqiad (k8s) is unstable - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=eqiad+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterFlinkJobUnsta [00:39:12] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1007 is CRITICAL: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Max retries exceeded with url: /_cluster/health (Caused by NewConnectionError(urllib3.connection.HTTPConnection object at 0x7f2a7f8d1550: Failed to establish a new connection: [Errno 111] Connection refused)) https://wikitec [00:39:12] dia.org/wiki/Search%23Administration [00:39:18] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1012 is CRITICAL: CRITICAL - elasticsearch inactive shards 333 threshold =0.15 breach: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 745, active_shards: 1200, relocating_shards: 0, initializing_shards: 19, unassigned_shards: [00:39:18] layed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 78.27788649706457 https://wikitech.wikimedia.org/wiki/Search%23Administration [00:39:45] FIRING: CirrusStreamingUpdaterRateTooLow: CirrusSearch update rate from flink-app-consumer-cloudelastic is critically low - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterRateTooLow [00:40:12] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1009 is CRITICAL: CRITICAL - elasticsearch inactive shards 316 threshold =0.15 breach: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 745, active_shards: 1217, relocating_shards: 0, initializing_shards: 16, unassigned_shards: [00:40:12] layed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 79.38682322243966 https://wikitech.wikimedia.org/wiki/Search%23Administration [00:40:14] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1011 is CRITICAL: CRITICAL - elasticsearch inactive shards 316 threshold =0.15 breach: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 745, active_shards: 1217, relocating_shards: 0, initializing_shards: 16, unassigned_shards: [00:40:14] layed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 79.38682322243966 https://wikitech.wikimedia.org/wiki/Search%23Administration [00:40:14] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1010 is CRITICAL: CRITICAL - elasticsearch inactive shards 316 threshold =0.15 breach: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 745, active_shards: 1217, relocating_shards: 0, initializing_shards: 16, unassigned_shards: [00:40:14] layed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 79.38682322243966 https://wikitech.wikimedia.org/wiki/Search%23Administration [00:40:25] RESOLVED: SystemdUnitFailed: opensearch_2@cloudelastic-chi-eqiad.service on cloudelastic1008:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:41:55] FIRING: [2x] SystemdUnitFailed: opensearch_2@cloudelastic-chi-eqiad.service on cloudelastic1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:46:45] RESOLVED: CirrusStreamingUpdaterFlinkJobUnstable: cirrus_streaming_updater_consumer_cloudelastic_eqiad in eqiad (k8s) is unstable - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=eqiad+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterFlinkJobUns [00:48:14] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1010 is CRITICAL: CRITICAL - elasticsearch inactive shards 270 threshold =0.15 breach: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 745, active_shards: 1263, relocating_shards: 0, initializing_shards: 7, unassigned_shards: [00:48:14] ayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 82.38747553816047 https://wikitech.wikimedia.org/wiki/Search%23Administration [00:51:55] FIRING: [3x] SystemdUnitFailed: opensearch-disable-readahead-cloudelastic-chi-eqiad.service on cloudelastic1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:53:12] RECOVERY - OpenSearch health check for shards on 9200 on cloudelastic1007 is OK: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 752, active_shards: 1313, relocating_shards: 0, initializing_shards: 8, unassigned_shards: 212, delayed_unassigned_ [00:53:12] 0, number_of_pending_tasks: 4, number_of_in_flight_fetch: 6, task_max_waiting_in_queue_millis: 1984, active_shards_percent_as_number: 85.64905414220483 https://wikitech.wikimedia.org/wiki/Search%23Administration [00:53:12] RECOVERY - OpenSearch health check for shards on 9200 on cloudelastic1009 is OK: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 752, active_shards: 1316, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 207, delayed_unassigned [00:53:12] 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 6, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 85.84474885844749 https://wikitech.wikimedia.org/wiki/Search%23Administration [00:53:14] RECOVERY - OpenSearch health check for shards on 9200 on cloudelastic1011 is OK: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 752, active_shards: 1316, relocating_shards: 0, initializing_shards: 13, unassigned_shards: 204, delayed_unassigned [00:53:14] 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 6, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 85.84474885844749 https://wikitech.wikimedia.org/wiki/Search%23Administration [00:53:14] RECOVERY - OpenSearch health check for shards on 9200 on cloudelastic1010 is OK: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 752, active_shards: 1316, relocating_shards: 0, initializing_shards: 13, unassigned_shards: 204, delayed_unassigned [00:53:14] 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 6, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 85.84474885844749 https://wikitech.wikimedia.org/wiki/Search%23Administration [00:53:18] RECOVERY - OpenSearch health check for shards on 9200 on cloudelastic1012 is OK: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 753, active_shards: 1318, relocating_shards: 0, initializing_shards: 13, unassigned_shards: 202, delayed_unassigned [00:53:18] 0, number_of_pending_tasks: 2, number_of_in_flight_fetch: 12, task_max_waiting_in_queue_millis: 470, active_shards_percent_as_number: 85.97521200260925 https://wikitech.wikimedia.org/wiki/Search%23Administration [00:54:14] RECOVERY - OpenSearch health check for shards on 9200 on cloudelastic1008 is OK: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 755, active_shards: 1358, relocating_shards: 0, initializing_shards: 13, unassigned_shards: 162, delayed_unassigned [00:54:14] 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 88.58447488584474 https://wikitech.wikimedia.org/wiki/Search%23Administration [00:56:55] FIRING: [3x] SystemdUnitFailed: opensearch-disable-readahead-cloudelastic-chi-eqiad.service on cloudelastic1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:58:45] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [00:58:51] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [00:59:02] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [00:59:08] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [01:09:59] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1280875 [01:09:59] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1280875 (owner: 10TrainBranchBot) [01:10:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:12:20] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1008 is CRITICAL: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4) https://wikitech.wikimedia.org/wiki/Search%23Administration [01:13:16] RECOVERY - OpenSearch health check for shards on 9200 on cloudelastic1008 is OK: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: yellow, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 766, active_shards: 1434, relocating_shards: 0, initializing_shards: 14, unassigned_shards: 85, delayed_unassign [01:13:16] s: 0, number_of_pending_tasks: 5, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 613, active_shards_percent_as_number: 93.54207436399217 https://wikitech.wikimedia.org/wiki/Search%23Administration [01:18:45] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [01:18:51] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [01:19:02] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [01:19:08] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [01:20:52] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1280875 (owner: 10TrainBranchBot) [01:21:55] RESOLVED: SystemdUnitFailed: opensearch-disable-readahead-cloudelastic-chi-eqiad.service on cloudelastic1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:28:26] FIRING: [16x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:37:45] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [01:37:51] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [01:37:53] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [01:37:59] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [01:42:45] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [01:42:45] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [01:43:02] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [01:43:08] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [01:43:21] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [01:43:27] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [01:43:29] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [01:43:35] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [01:48:15] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [01:48:15] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [01:48:32] RESOLVED: Outbound discards: Device asw2-a-eqiad.mgmt.eqiad.wmnet recovered from Outbound discards - https://alerts.wikimedia.org/?q=alertname%3DOutbound+discards [01:49:15] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [01:49:15] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [02:00:35] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:07:16] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 06m 41s) [02:09:20] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:33:00] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [02:33:00] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [02:33:00] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [02:33:05] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [02:33:40] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [02:33:46] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [02:34:31] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [02:34:36] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [02:34:41] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:39:15] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [02:39:21] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [02:43:00] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [02:43:00] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [02:55:15] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [02:55:21] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [02:55:23] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [02:55:29] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [02:58:33] 10ops-ulsfo, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11877657 (10Papaul) @cmooney please see below for all the DNS names for IPV6 needed. Thanks irb0-411.asw1-22-ulsfo.wikimedia.org irb0-421.asw1-22-ulsf... [03:15:15] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [03:15:15] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [03:16:15] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [03:16:15] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [03:21:15] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [03:21:15] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [03:22:15] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [03:22:15] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [03:26:21] !log akhatun@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [03:26:29] !log akhatun@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [03:32:45] 10SRE-swift-storage, 06Data-Persistence, 10MediaViewer, 10Thumbor, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11877666 (10Ladsgroup) The main gain from bucketing and standardizing thumbnail steps back then was to improve performance. That was... [03:45:15] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [03:45:21] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [03:51:15] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [03:51:21] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [04:06:15] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [04:06:21] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [04:07:15] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [04:07:21] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [04:12:15] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [04:12:21] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [04:12:32] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [04:12:38] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [04:13:15] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [04:13:20] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [04:13:30] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [04:13:30] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [04:34:12] (03PS1) 10AKhatun: alerts: update runbook link for mw-page-html-feature-counts-change-enrich [alerts] - 10https://gerrit.wikimedia.org/r/1281017 (https://phabricator.wikimedia.org/T424225) [04:36:18] (03CR) 10AKhatun: "I changed my mind... Adding both streams in a single doc was becoming confusing. And given we are going to expand the feature counts strea" [alerts] - 10https://gerrit.wikimedia.org/r/1281017 (https://phabricator.wikimedia.org/T424225) (owner: 10AKhatun) [04:37:35] (03CR) 10AKhatun: "I changed my mind... Adding both streams in a single doc was becoming confusing. And given we are going to expand the feature counts strea" [alerts] - 10https://gerrit.wikimedia.org/r/1281017 (https://phabricator.wikimedia.org/T424225) (owner: 10AKhatun) [04:38:15] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [04:38:21] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [04:39:45] FIRING: CirrusStreamingUpdaterRateTooLow: CirrusSearch update rate from flink-app-consumer-cloudelastic is critically low - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterRateTooLow [04:42:15] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [04:42:15] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [04:45:45] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [04:45:51] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [04:46:08] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [04:46:14] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [05:10:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:15:45] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [05:15:51] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [05:16:02] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [05:16:08] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [05:16:21] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [05:16:27] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [05:17:15] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [05:17:21] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [05:28:26] FIRING: [16x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [05:32:30] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 339895472 and 39 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [05:34:30] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 317376 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [05:41:00] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [05:41:00] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [05:41:17] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [05:41:23] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [05:42:04] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [05:45:45] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [05:45:45] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [05:45:53] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [05:45:59] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [05:50:43] FIRING: [4x] CoreBGPDown: Core BGP session down between cr1-drmrs and cr2-eqiad (185.15.58.138) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [05:53:30] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 320483216 and 38 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [05:54:30] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 3381960 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260501T0600) [06:07:50] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING WARNING - Packet loss = 0%, RTA = 616.14 ms [06:12:44] FIRING: KubernetesDeploymentUnavailableReplicas: ... [06:12:44] Deployment function-evaluator-python-evaluator in wikifunctions at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=wikifunctions&var-deployment=function-evaluator-python-evaluator - ... [06:12:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [06:15:45] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [06:15:51] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [06:15:53] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [06:15:59] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [06:19:18] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [06:24:15] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [06:24:21] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [06:24:22] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING WARNING - Packet loss = 33%, RTA = 943.20 ms [06:24:23] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [06:24:29] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [06:55:40] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [06:57:44] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [06:57:44] Deployment function-evaluator-python-evaluator in wikifunctions at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=wikifunctions&var-deployment=function-evaluator-python-evaluator - ... [06:57:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [07:00:04] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260501T0700) [07:04:15] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [07:04:21] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [07:05:15] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [07:05:20] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [07:05:54] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 259.55 ms [07:14:15] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [07:14:15] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [07:15:15] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [07:15:15] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [07:23:34] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [07:28:36] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 474.64 ms [07:30:45] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [07:30:45] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [07:30:53] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [07:30:59] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [07:33:13] jouncebot: nowandnext [07:33:13] For the next 23 hour(s) and 26 minute(s): No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260501T0700) [07:33:13] In 3 hour(s) and 26 minute(s): GitLab version upgrades (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260501T1100) [07:33:56] but surely if we're all at the hackathon we *could* do a *small* change... :D [07:45:04] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [07:50:45] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [07:50:45] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [07:51:02] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [07:51:08] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [07:51:29] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [07:51:35] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [07:51:38] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [07:51:44] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [07:57:22] (03PS1) 10Clément Goubert: rest-gateway: Allowlist Milan Hackathon IPs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1281391 (https://phabricator.wikimedia.org/T425009) [07:58:02] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2002 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [07:59:06] 10SRE-SLO, 06ServiceOps new, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 07Essential-Work, and 2 others: IPoid: Define service level indicators and service level objectives - https://phabricator.wikimedia.org/T348935#11878010 (10OKryva-WMF) [08:04:35] (03PS2) 10Clément Goubert: rest-gateway: Allowlist Milan Hackathon IPs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1281391 (https://phabricator.wikimedia.org/T425009) [08:07:08] (03CR) 10Giuseppe Lavagetto: [C:03+1] rest-gateway: Allowlist Milan Hackathon IPs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1281391 (https://phabricator.wikimedia.org/T425009) (owner: 10Clément Goubert) [08:07:42] (03CR) 10Clément Goubert: [C:03+2] rest-gateway: Allowlist Milan Hackathon IPs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1281391 (https://phabricator.wikimedia.org/T425009) (owner: 10Clément Goubert) [08:08:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr2-codfw and Hurricane Electric (2001:504:61::1b1b:0:1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [08:09:46] (03Merged) 10jenkins-bot: rest-gateway: Allowlist Milan Hackathon IPs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1281391 (https://phabricator.wikimedia.org/T425009) (owner: 10Clément Goubert) [08:10:50] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 269.41 ms [08:12:49] !log cgoubert@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [08:13:01] !log cgoubert@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [08:13:09] !log cgoubert@deploy1003 helmfile [codfw] START helmfile.d/services/rest-gateway: apply [08:13:29] !log cgoubert@deploy1003 helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [08:13:37] !log cgoubert@deploy1003 helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [08:13:58] !log cgoubert@deploy1003 helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [08:15:39] (03CR) 10Jforrester: [C:03+1] Add Wikifunctions' evaluator ingress endpoints to service.yaml [puppet] - 10https://gerrit.wikimedia.org/r/1280433 (https://phabricator.wikimedia.org/T424193) (owner: 10Elukey) [08:15:49] (03CR) 10Jforrester: [C:03+1] Turn Wikifunctions evaluator endpoints to production state [puppet] - 10https://gerrit.wikimedia.org/r/1280434 (https://phabricator.wikimedia.org/T424193) (owner: 10Elukey) [08:16:07] (03CR) 10Jforrester: [C:03+1] profile::services_proxy::envoy: add wikifunctions eval endpoints [puppet] - 10https://gerrit.wikimedia.org/r/1280435 (https://phabricator.wikimedia.org/T424193) (owner: 10Elukey) [08:23:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr2-codfw and Hurricane Electric (2001:504:61::1b1b:0:1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [08:28:39] RESOLVED: [2x] TransitBGPDown: Transit BGP session down between cr2-codfw and Hurricane Electric (2001:504:61::1b1b:0:1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [08:29:03] (03CR) 10Dreamy Jazz: "I'd recommend using PS1 or maybe make a new DB list that defines which wikis have securepoll at all. `$wmgUseSecurePoll` is defined to ena" [puppet] - 10https://gerrit.wikimedia.org/r/1279281 (https://phabricator.wikimedia.org/T419309) (owner: 10Novem Linguae) [08:37:11] (03CR) 10Atsuko: [C:03+2] dse-k8s: deploy additional opensearch clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1280515 (https://phabricator.wikimedia.org/T424248) (owner: 10Atsuko) [08:37:16] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 66%, RTA = 4731.43 ms [08:39:35] (03Merged) 10jenkins-bot: dse-k8s: deploy additional opensearch clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1280515 (https://phabricator.wikimedia.org/T424248) (owner: 10Atsuko) [08:40:00] FIRING: CirrusStreamingUpdaterRateTooLow: CirrusSearch update rate from flink-app-consumer-cloudelastic is critically low - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterRateTooLow [08:41:15] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [08:41:15] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [08:41:32] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [08:41:38] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [08:42:18] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 222.79 ms [08:47:15] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [08:47:21] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [08:47:32] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [08:47:38] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [08:48:30] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 39657216 and 4 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [08:49:30] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 2398864 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [08:54:39] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST events) on k8s-dse@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s-dse&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [08:54:50] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [08:58:02] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2002 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [09:05:04] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 222.73 ms [09:10:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:13:30] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 289534088 and 34 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [09:15:13] FIRING: BFDdown: BFD session down between cr2-eqdfw and fe80::a6e1:1a00:1a6f:d3a3 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqdfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:16:30] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 2866000 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [09:17:15] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [09:17:15] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [09:17:23] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [09:17:29] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [09:17:42] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [09:17:50] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [09:17:56] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [09:19:30] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [09:19:30] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [09:20:13] RESOLVED: BFDdown: BFD session down between cr2-eqdfw and fe80::a6e1:1a00:1a6f:d3a3 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqdfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:22:15] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [09:22:15] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [09:22:23] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [09:22:29] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [09:23:15] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [09:23:15] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [09:23:23] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [09:23:29] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [09:24:39] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST events) on k8s-dse@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s-dse&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [09:28:26] FIRING: [16x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:28:54] (03PS1) 10Samtar: Switch watchstar from Popover to Dialog [core] (wmf/1.46.0-wmf.26) - 10https://gerrit.wikimedia.org/r/1281423 (https://phabricator.wikimedia.org/T417847) [09:29:31] jouncebot: nowandnext [09:29:31] For the next 21 hour(s) and 30 minute(s): No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260501T0700) [09:29:31] In 1 hour(s) and 30 minute(s): GitLab version upgrades (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260501T1100) [09:30:21] (03PS1) 10Urbanecm: Update the interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1281426 (https://phabricator.wikimedia.org/T239173) [09:30:41] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1281426 (https://phabricator.wikimedia.org/T239173) (owner: 10Urbanecm) [09:31:37] (03Merged) 10jenkins-bot: Update the interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1281426 (https://phabricator.wikimedia.org/T239173) (owner: 10Urbanecm) [09:32:02] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1281426|Update the interwiki cache (T239173)]] [09:32:04] T239173: gewikimedia's w interwiki links to (nonexistent) gewiki - https://phabricator.wikimedia.org/T239173 [09:32:38] fyi I am going to be deploying https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1281423 which fixes an urgent usability bug. We're at the hackathon and have checked its okay to do (after the deployment above is done) [09:33:21] TheresNoTime: feel free to +2 to save on CI [09:33:28] ack [09:35:45] (03CR) 10Samtar: [C:03+2] Switch watchstar from Popover to Dialog [core] (wmf/1.46.0-wmf.26) - 10https://gerrit.wikimedia.org/r/1281423 (https://phabricator.wikimedia.org/T417847) (owner: 10Samtar) [09:36:56] (03PS1) 10Gkyziridis: eventstreams: Configure new stream for revertrisk-multilingual model. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1281431 (https://phabricator.wikimedia.org/T415892) [09:38:07] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1281426|Update the interwiki cache (T239173)]] (duration: 06m 05s) [09:38:10] T239173: gewikimedia's w interwiki links to (nonexistent) gewiki - https://phabricator.wikimedia.org/T239173 [09:38:15] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [09:38:15] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [09:38:15] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [09:38:16] TheresNoTime: i'm done, over to you [09:38:21] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [09:38:22] thanks! [09:39:00] (03CR) 10TrainBranchBot: [C:03+2] "Approved by samtar@deploy1003 using scap backport" [core] (wmf/1.46.0-wmf.26) - 10https://gerrit.wikimedia.org/r/1281423 (https://phabricator.wikimedia.org/T417847) (owner: 10Samtar) [09:50:31] (03Merged) 10jenkins-bot: Switch watchstar from Popover to Dialog [core] (wmf/1.46.0-wmf.26) - 10https://gerrit.wikimedia.org/r/1281423 (https://phabricator.wikimedia.org/T417847) (owner: 10Samtar) [09:50:46] !log samtar@deploy1003 Started scap sync-world: Backport for [[gerrit:1281423|Switch watchstar from Popover to Dialog (T417847)]] [09:50:48] T417847: Add labels field to watchstar popup - https://phabricator.wikimedia.org/T417847 [09:50:58] FIRING: [4x] CoreBGPDown: Core BGP session down between cr1-drmrs and cr2-eqiad (185.15.58.138) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [09:52:27] !log samtar@deploy1003 samtar: Backport for [[gerrit:1281423|Switch watchstar from Popover to Dialog (T417847)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [09:53:44] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 337.39 ms [09:53:45] !log samtar@deploy1003 samtar: Continuing with deployment [09:54:45] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [09:54:45] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [09:55:02] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [09:55:08] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [09:57:35] !log samtar@deploy1003 Finished scap sync-world: Backport for [[gerrit:1281423|Switch watchstar from Popover to Dialog (T417847)]] (duration: 06m 49s) [09:57:38] T417847: Add labels field to watchstar popup - https://phabricator.wikimedia.org/T417847 [10:09:45] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [10:09:45] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [10:09:53] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [10:09:59] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [10:10:21] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [10:10:27] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [10:10:38] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [10:10:44] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [10:11:40] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2014.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [10:11:40] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2015.codfw.wmnet, wdqs2012.codfw.wmnet, wdqs2014.codfw.wmnet, wdqs2007.codfw.wmnet, wdqs2022.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [10:12:38] (03PS3) 10Novem Linguae: purge_securepoll: don't exclude private wikis [puppet] - 10https://gerrit.wikimedia.org/r/1279281 (https://phabricator.wikimedia.org/T419309) [10:13:20] (03CR) 10CI reject: [V:04-1] purge_securepoll: don't exclude private wikis [puppet] - 10https://gerrit.wikimedia.org/r/1279281 (https://phabricator.wikimedia.org/T419309) (owner: 10Novem Linguae) [10:13:34] (03PS4) 10Novem Linguae: purge_securepoll: don't exclude private wikis [puppet] - 10https://gerrit.wikimedia.org/r/1279281 (https://phabricator.wikimedia.org/T419309) [10:14:17] (03CR) 10Novem Linguae: "PS2 has the same challenges that we're trying to solve, e.g. there's a danger of forgetting to update the dblist and then some SecurePolls" [puppet] - 10https://gerrit.wikimedia.org/r/1279281 (https://phabricator.wikimedia.org/T419309) (owner: 10Novem Linguae) [10:14:46] (03PS5) 10Novem Linguae: purge_securepoll: don't exclude private wikis [puppet] - 10https://gerrit.wikimedia.org/r/1279281 (https://phabricator.wikimedia.org/T419309) [10:15:05] (03CR) 10Dreamy Jazz: [C:03+1] purge_securepoll: don't exclude private wikis [puppet] - 10https://gerrit.wikimedia.org/r/1279281 (https://phabricator.wikimedia.org/T419309) (owner: 10Novem Linguae) [10:15:15] jouncebot: nowandnext [10:15:15] For the next 20 hour(s) and 44 minute(s): No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260501T0700) [10:15:15] In 0 hour(s) and 44 minute(s): GitLab version upgrades (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260501T1100) [10:15:53] (Just wanted a link to the calendar, don't plan on deploying things) [10:20:13] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 185.15.58.139 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:21:40] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [10:21:40] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [10:25:00] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [10:25:00] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [10:25:08] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [10:25:14] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [10:28:31] 06SRE, 06Data-Platform-SRE: archiva1002 has stale jobs in /var/cache/archiva that uses all the disk space - https://phabricator.wikimedia.org/T425083#11878642 (10atsuko) 05Open→03In progress p:05Triage→03Low a:03atsuko [10:29:09] 06SRE, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): archiva1002 has stale jobs in /var/cache/archiva that uses all the disk space - https://phabricator.wikimedia.org/T425083#11878649 (10atsuko) [10:29:40] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2013.codfw.wmnet, wdqs2015.codfw.wmnet, wdqs2007.codfw.wmnet, wdqs2010.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [10:29:40] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2013.codfw.wmnet, wdqs2015.codfw.wmnet, wdqs2007.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [10:31:40] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [10:31:40] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [10:31:45] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [10:31:51] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [10:31:53] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [10:31:59] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [10:34:50] 10ops-ulsfo, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11878675 (10cmooney) @Papaul thanks. I see most of those don't exist even for IPv4, nor are there any IPv6 addresses listed, so I'm not sure exactly what migh... [10:42:35] 06SRE, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): archiva1002 has stale jobs in /var/cache/archiva that uses all the disk space - https://phabricator.wikimedia.org/T425083#11878700 (10atsuko) 05In progress→03Stalled a:05atsuko→03None I manually cleaned up `/var/cache/archiva` more than wanted, so I... [10:55:52] 06SRE, 10observability: Observability: Re-IP codfw private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T422816#11878768 (10cmooney) >>! In T422816#11876405, @herron wrote: > @ayounsi @cmooney would you be able to have a look to see if anything stands out from #netops persp... [11:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260501T0700) [11:00:05] jelto, arnoldokoth, mutante, and arnaudb: #bothumor My software never has bugs. It just develops random features. Rise for GitLab version upgrades. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260501T1100). [11:06:32] 10ops-eqiad, 06DC-Ops: Q3 :rack/setup/install cloudvirt refresh - https://phabricator.wikimedia.org/T425088 (10Jclark-ctr) 03NEW [11:07:47] 10ops-eqiad, 06DC-Ops: Q3 :rack/setup/install cloudvirt refresh - https://phabricator.wikimedia.org/T425088#11878787 (10Jclark-ctr) @Andrew Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.... [11:18:29] 06SRE, 10observability: Observability: Re-IP codfw private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T422816#11878805 (10elukey) These are the last occurrences of IOException on 2001: ` [2026-04-30 16:56:35,442] WARN [ReplicaFetcher replicaId=2001, leaderId=2005, fetche... [11:21:46] 06SRE, 10observability: Observability: Re-IP codfw private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T422816#11878806 (10cmooney) >>! In T422816#11878805, @elukey wrote: > It doesn't seem a propagation error related to ferm, but it should be something related to it if I... [11:26:09] 10ops-eqiad, 06DC-Ops: Q3 :rack/setup/install cloudvirt refresh - https://phabricator.wikimedia.org/T425088#11878813 (10Jclark-ctr) @Andrew i am assuming these will be cloudvirt1077-1080 and they are replacing 6 servers that are all located in D5 so i have for the time Racked these all in D5 [11:26:45] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [11:26:45] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [11:26:45] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [11:26:51] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [11:26:55] (03PS1) 10Atsuko: wmnet: add additional opensearch clusters [dns] - 10https://gerrit.wikimedia.org/r/1281462 (https://phabricator.wikimedia.org/T424248) [11:26:57] 10ops-eqiad, 06DC-Ops: Q3 :rack/setup/install cloudvirt refresh - https://phabricator.wikimedia.org/T425088#11878830 (10Jclark-ctr) [11:31:45] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [11:31:45] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [11:31:45] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [11:31:51] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [11:36:45] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [11:36:45] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [11:36:53] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [11:36:59] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [11:41:05] 06SRE, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): archiva1002 has stale jobs in /var/cache/archiva that uses all the disk space - https://phabricator.wikimedia.org/T425083#11878867 (10atsuko) [11:42:27] 06SRE: archiva1002 - disk 98% full - https://phabricator.wikimedia.org/T391904#11878871 (10atsuko) [12:02:08] 10SRE-swift-storage, 06Data-Persistence, 10MediaViewer, 10Thumbor, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11878954 (10Nux) I appreciated the detailed write-up. The circumstances are unfortunate because editing JS is now hard, and that, I k... [12:05:45] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [12:05:51] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [12:05:53] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [12:05:59] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [12:13:18] (03CR) 10Btullis: [C:03+1] "Looks good to me." [dns] - 10https://gerrit.wikimedia.org/r/1281462 (https://phabricator.wikimedia.org/T424248) (owner: 10Atsuko) [12:24:06] (03PS1) 10Zabe: Close Wikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1281479 (https://phabricator.wikimedia.org/T421796) [12:25:53] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [12:25:59] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [12:26:02] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [12:26:08] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [12:26:21] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [12:26:27] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [12:26:38] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [12:26:44] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [12:26:49] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST events) on k8s-dse@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s-dse&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [12:31:00] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [12:31:06] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [12:31:17] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [12:31:23] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [12:31:39] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST events) on k8s-dse@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s-dse&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [12:33:30] !log jgiannelos@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [12:33:59] !log jgiannelos@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [12:34:05] !log jgiannelos@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [12:34:36] !log jgiannelos@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [12:40:00] FIRING: CirrusStreamingUpdaterRateTooLow: CirrusSearch update rate from flink-app-consumer-cloudelastic is critically low - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterRateTooLow [12:42:23] (03PS1) 10Zabe: Drop some unneeded wikinews configs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1281485 (https://phabricator.wikimedia.org/T421796) [12:49:37] (03CR) 10Bugreporter: "See also T423578, but we need to depopulate existing user groups yet." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1281485 (https://phabricator.wikimedia.org/T421796) (owner: 10Zabe) [12:55:40] 10ops-eqiad, 06SRE, 06DC-Ops: Q3 :rack/setup/install cloudvirt refresh - https://phabricator.wikimedia.org/T425088#11879199 (10Jclark-ctr) [13:00:41] !log jclark@cumin1003 START - Cookbook sre.dns.netbox [13:04:30] !log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1077 to eqiad - jclark@cumin1003" [13:04:36] !log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1077 to eqiad - jclark@cumin1003" [13:04:36] !log jclark@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:05:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:05:38] (03PS1) 10Zabe: Remove custom user groups from Wikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1281491 (https://phabricator.wikimedia.org/T423578) [13:05:45] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [13:05:45] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [13:05:45] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [13:05:51] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [13:06:00] !log jclark@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1080.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:06:09] !log jclark@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:06:23] !log jclark@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:06:34] !log jclark@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1079.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:07:52] !log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1080.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:07:58] !log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:08:42] !log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:08:42] 10ops-eqiad, 06SRE, 06DC-Ops: Q3 :rack/setup/install cloudvirt refresh - https://phabricator.wikimedia.org/T425088#11879303 (10Jclark-ctr) these servers are failing to provision. @elukey Supermicro... ` Connecting to the BMC as user root (wmf_root_mgmt) Testing Redfish API connection to cloudvirt1077 (... [13:08:48] !log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1079.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:08:57] !log jclark@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1077 [13:09:00] !log jclark@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1078 [13:09:03] !log jclark@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1079 [13:09:05] !log jclark@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1080 [13:09:07] !log jclark@cumin1003 END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host cloudvirt1077 [13:09:13] !log jclark@cumin1003 END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host cloudvirt1079 [13:09:15] !log jclark@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1078 [13:09:23] !log jclark@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1080 [13:24:32] <_Gerges> !log WikiMonitor setup [13:24:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:28:26] FIRING: [16x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:28:44] (03CR) 10Ottomata: [C:03+1] eventstreams: Configure new stream for revertrisk-multilingual model. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1281431 (https://phabricator.wikimedia.org/T415892) (owner: 10Gkyziridis) [13:29:16] (03CR) 10Ottomata: [C:03+1] "Feel free to merge and deploy. Just the usual wikikube staging, codfw, eqiad helmfile apply process 😊" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1281431 (https://phabricator.wikimedia.org/T415892) (owner: 10Gkyziridis) [13:34:03] 10ops-eqiad, 06SRE, 06DC-Ops: Q3:rack/setup/install cloudcephosd1054 - https://phabricator.wikimedia.org/T416395#11879347 (10Jclark-ctr) a:03Jclark-ctr [13:35:45] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [13:35:45] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [13:36:02] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [13:36:08] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [13:36:21] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [13:36:26] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [13:36:38] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [13:36:44] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [13:39:10] 10ops-ulsfo, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11879352 (10Papaul) @cmooney ok thank you. [13:44:48] !log zabe@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-experimental: apply [13:45:17] !log zabe@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply [13:50:58] FIRING: [4x] CoreBGPDown: Core BGP session down between cr1-drmrs and cr2-eqiad (185.15.58.138) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:51:17] (03PS1) 10Mmartorana: Email confirmation banner: Remove obsolete arm_b variant [core] (wmf/1.46.0-wmf.26) - 10https://gerrit.wikimedia.org/r/1281501 (https://phabricator.wikimedia.org/T421366) [13:54:51] (03PS1) 10Mmartorana: Use js promise for email confirmation banner [extensions/WikimediaEvents] (wmf/1.46.0-wmf.26) - 10https://gerrit.wikimedia.org/r/1281504 (https://phabricator.wikimedia.org/T420007) [14:03:36] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, May 04 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [core] (wmf/1.46.0-wmf.26) - 10https://gerrit.wikimedia.org/r/1281501 (https://phabricator.wikimedia.org/T421366) (owner: 10Mmartorana) [14:03:59] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, May 04 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [extensions/WikimediaEvents] (wmf/1.46.0-wmf.26) - 10https://gerrit.wikimedia.org/r/1281504 (https://phabricator.wikimedia.org/T420007) (owner: 10Mmartorana) [14:11:32] (03PS1) 10Zabe: Disable FlaggedRevs on wikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1281506 (https://phabricator.wikimedia.org/T423577) [14:20:29] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 185.15.58.139 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:21:00] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [14:21:06] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [14:21:08] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [14:21:14] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [14:21:36] 10ops-eqiad, 06SRE, 06DC-Ops: Q3:rack/setup/install cloudcephosd1053 - https://phabricator.wikimedia.org/T416394#11879450 (10Jclark-ctr) [14:21:41] 10ops-eqiad, 06SRE, 06DC-Ops: Q3:rack/setup/install cloudcephosd1054 - https://phabricator.wikimedia.org/T416395#11879451 (10Jclark-ctr) [14:22:17] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#11879453 (10Jclark-ctr) [14:22:40] 10ops-eqiad, 06SRE, 06DC-Ops: Q3:rack/setup/install cloudcephosd1053 - https://phabricator.wikimedia.org/T416394#11879458 (10Jclark-ctr) 05Open→03Resolved Moving this racking task to T419892 [14:22:53] 10ops-eqiad, 06SRE, 06DC-Ops: Q3:rack/setup/install cloudcephosd1054 - https://phabricator.wikimedia.org/T416395#11879461 (10Jclark-ctr) 05Open→03Resolved Moving this racking ticket to T419892 [14:25:45] (03PS1) 10Andrew Bogott: wmfkeystonehooks: add more explicit logging around ldap group sync [puppet] - 10https://gerrit.wikimedia.org/r/1281512 (https://phabricator.wikimedia.org/T379550) [14:26:11] 06SRE, 10observability: Observability: Re-IP codfw private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T422816#11879484 (10herron) Thanks for looking @cmooney @elukey! I think we can certainly rule out a network issue. Having a look this morning with fresh eyes, 2005 doe... [14:26:30] (03CR) 10Andrew Bogott: [C:03+2] wmfkeystonehooks: add more explicit logging around ldap group sync [puppet] - 10https://gerrit.wikimedia.org/r/1281512 (https://phabricator.wikimedia.org/T379550) (owner: 10Andrew Bogott) [14:27:15] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [14:27:15] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [14:27:23] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [14:27:29] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [14:35:09] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps new, 10ServiceOps-Upgrades-Hardware: Q3:rack/setup/install wikikube-worker23[57-74] - https://phabricator.wikimedia.org/T418925#11879520 (10Jhancock.wm) [14:42:15] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [14:42:21] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [14:42:23] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [14:42:29] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [14:44:45] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [14:44:45] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [14:44:53] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [14:44:59] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [14:47:22] !log herron@cumin1003 START - Cookbook sre.hosts.reimage for host kafka-logging2004.codfw.wmnet with OS trixie [14:47:50] !log herron@cumin1003 START - Cookbook sre.hosts.move-vlan for host kafka-logging2004 [14:49:45] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [14:49:50] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [14:50:02] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [14:50:08] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [14:51:03] (03PS1) 10Herron: kafka-logging: update kafka-logging2004 IP addresses [puppet] - 10https://gerrit.wikimedia.org/r/1281517 (https://phabricator.wikimedia.org/T422816) [14:52:12] RECOVERY - WMF Cloud -Omega Cluster- - Public Internet Port - SSL Expiry on cloudelastic.wikimedia.org is OK: OK - Certificate cloudelastic.wikimedia.org will expire on Sun 05 Jul 2026 07:49:09 AM GMT +0000. https://wikitech.wikimedia.org/wiki/Search%23Administration [14:52:12] RECOVERY - WMF Cloud -Chi Cluster- - Public Internet Port - SSL Expiry on cloudelastic.wikimedia.org is OK: OK - Certificate cloudelastic.wikimedia.org will expire on Sun 05 Jul 2026 07:49:09 AM GMT +0000. https://wikitech.wikimedia.org/wiki/Search%23Administration [14:52:12] RECOVERY - WMF Cloud -Chi Cluster- - Prod MW AppServer Port - SSL Expiry on cloudelastic.wikimedia.org is OK: OK - Certificate cloudelastic.wikimedia.org will expire on Sun 05 Jul 2026 07:49:09 AM GMT +0000. https://wikitech.wikimedia.org/wiki/Search%23Administration [14:52:12] RECOVERY - WMF Cloud -Omega Cluster- - Public Internet Port - HTTPS on cloudelastic.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 750 bytes in 0.010 second response time https://wikitech.wikimedia.org/wiki/Search%23Administration [14:52:12] RECOVERY - WMF Cloud -Chi Cluster- - Prod MW AppServer Port - HTTPS on cloudelastic.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 746 bytes in 0.013 second response time https://wikitech.wikimedia.org/wiki/Search%23Administration [14:52:12] RECOVERY - WMF Cloud -Chi Cluster- - Public Internet Port - HTTPS on cloudelastic.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 746 bytes in 0.010 second response time https://wikitech.wikimedia.org/wiki/Search%23Administration [14:52:50] herron@cumin1003 reimage (PID 3281907) is awaiting input [14:54:40] (03PS1) 10Andrew Bogott: wmfkeystonehooks: more narration when ldap is adding a user to a project group [puppet] - 10https://gerrit.wikimedia.org/r/1281519 (https://phabricator.wikimedia.org/T379550) [14:56:58] (03CR) 10Andrew Bogott: [C:03+2] wmfkeystonehooks: more narration when ldap is adding a user to a project group [puppet] - 10https://gerrit.wikimedia.org/r/1281519 (https://phabricator.wikimedia.org/T379550) (owner: 10Andrew Bogott) [14:57:45] !log herron@cumin1003 START - Cookbook sre.dns.netbox [15:03:33] herron@cumin1003 reimage (PID 3281907) is awaiting input [15:03:55] !log dancy@deploy1003 Installing scap version "4.258.0" for 2 host(s) [15:05:46] !log dancy@deploy1003 Installation of scap version "4.258.0" completed for 2 hosts [15:07:53] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [15:07:59] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [15:08:02] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [15:08:08] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [15:11:17] !log herron@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2004 - herron@cumin1003" [15:11:22] !log herron@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2004 - herron@cumin1003" [15:11:22] !log herron@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [15:11:23] !log herron@cumin1003 START - Cookbook sre.dns.wipe-cache kafka-logging2004.codfw.wmnet 38.16.192.10.in-addr.arpa 8.3.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [15:11:26] !log herron@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-logging2004.codfw.wmnet 38.16.192.10.in-addr.arpa 8.3.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [15:11:27] !log herron@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging2004 [15:14:19] !log herron@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging2004 [15:14:19] !log herron@cumin1003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging2004 [15:19:53] (03PS1) 10SomeRandomDeveloper: Replace use of $wgRequest [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1281526 (https://phabricator.wikimedia.org/T336703) [15:26:03] FIRING: KafkaUnderReplicatedPartitions: Under replicated partitions for Kafka cluster logging-codfw in codfw - https://wikitech.wikimedia.org/wiki/Kafka/Administration - https://grafana.wikimedia.org/d/000000027/kafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-kafka_cluster=logging-codfw - https://alerts.wikimedia.org/?q=alertname%3DKafkaUnderReplicatedPartitions [15:26:31] FIRING: Traffic on tunnel link: Alert for device cr1-drmrs.wikimedia.org - Traffic on tunnel link - https://alerts.wikimedia.org/?q=alertname%3DTraffic+on+tunnel+link [15:28:52] (03PS1) 10AikoChou: ml-services: update staging image for RRLA and revscoring model [deployment-charts] - 10https://gerrit.wikimedia.org/r/1281528 (https://phabricator.wikimedia.org/T416384) [15:30:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker1065:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1065 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [15:30:51] !log herron@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging2004.codfw.wmnet with reason: host reimage [15:31:31] RESOLVED: Traffic on tunnel link: Device cr1-drmrs.wikimedia.org recovered from Traffic on tunnel link - https://alerts.wikimedia.org/?q=alertname%3DTraffic+on+tunnel+link [15:34:49] !log herron@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging2004.codfw.wmnet with reason: host reimage [15:35:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1065:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1065 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [15:37:45] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [15:37:45] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [15:38:02] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [15:38:08] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [15:38:40] (03CR) 10AikoChou: [C:03+2] ml-services: update staging image for RRLA and revscoring model [deployment-charts] - 10https://gerrit.wikimedia.org/r/1281528 (https://phabricator.wikimedia.org/T416384) (owner: 10AikoChou) [15:41:48] (03Merged) 10jenkins-bot: ml-services: update staging image for RRLA and revscoring model [deployment-charts] - 10https://gerrit.wikimedia.org/r/1281528 (https://phabricator.wikimedia.org/T416384) (owner: 10AikoChou) [15:44:45] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [15:44:45] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [15:45:02] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [15:45:08] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [15:45:19] !log dancy@deploy1003 Installing scap version "4.258.1" for 2 host(s) [15:47:10] !log dancy@deploy1003 Installation of scap version "4.258.1" completed for 2 hosts [15:49:45] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [15:49:45] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [15:49:53] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [15:49:59] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [15:55:29] !log jmm@cumin2002 START - Cookbook sre.hosts.reboot-single for host sretest2001.codfw.wmnet [15:59:03] !log aikochou@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [16:00:50] !log jmm@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2001.codfw.wmnet [16:02:12] !log jmm@cumin2002 START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet [16:08:23] !log jmm@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet [16:09:20] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:09:33] (03PS1) 10Dduvall: zuul: Mount /etc/zuul and /etc/zuul-launcher for zuul-launcher [puppet] - 10https://gerrit.wikimedia.org/r/1281536 (https://phabricator.wikimedia.org/T424879) [16:15:03] (03CR) 10Dzahn: "You are faster than me. I wanted to make this patch as well, was still in meeting :)" [puppet] - 10https://gerrit.wikimedia.org/r/1281536 (https://phabricator.wikimedia.org/T424879) (owner: 10Dduvall) [16:15:38] (03CR) 10Dzahn: [C:03+2] zuul: Mount /etc/zuul and /etc/zuul-launcher for zuul-launcher [puppet] - 10https://gerrit.wikimedia.org/r/1281536 (https://phabricator.wikimedia.org/T424879) (owner: 10Dduvall) [16:15:55] mutante: :) ty! [16:16:39] dduvall: you just did what I thought "will do after the coffee"):) [16:17:24] deploying [16:17:38] haha. i'm on my second cup since i was on kiddo drop-off duty today :) [16:18:31] [zuul1001:~] $ sudo docker ps [16:18:31] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES [16:18:35] a44d270ff760 docker-registry.wikimedia.org/repos/releng/zuul/zuul/zuul-launcher:14.2.0-1 "/usr/bin/dumb-init …" 26 seconds ago Up 25 seconds [16:18:39] there it is [16:19:43] nice. no errors in the logs either [16:19:58] dduvall: :) btw, do you think we need https://gerrit.wikimedia.org/r/c/operations/puppet/+/1271042 [16:20:17] probably yes, but as the last step? [16:21:00] oh good question. i'm not totally sure [16:21:22] ok, don't worry for now [16:23:16] i can't think of a reason zuul would need standard ssh access to gerrit hosts. there's nothing in zuul.conf that suggests a need either [16:23:46] ok. as far as I remember this was created as a reminder that we might need it .. in a meeting with Tyler a while ago [16:24:03] will figure it out sooner or later [16:24:10] sounds good [16:29:53] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [16:29:59] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [16:30:02] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [16:30:08] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [16:30:29] !log ebernhardson@deploy1003 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [16:30:35] !log ebernhardson@deploy1003 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [16:34:20] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:34:28] !log cdobbins@cumin2002 conftool action : get/pooled; selector: name=cp5024.eqsin.wmnet [16:36:35] 06SRE: Please add Google Search Console domain verification for wikimediafoundation.org - https://phabricator.wikimedia.org/T424976#11880027 (10Dzahn) There already is a TXT entry in the template for wikimediafoundation.org for Google Search Console. It has a comment linking it to task T404974. [16:37:30] 06SRE, 06Traffic: [Search Console Verification DNS Request] - {{wikimediafoundation.org}} - https://phabricator.wikimedia.org/T404974#11880031 (10Dzahn) T424976 requests for this TXT record to be added. I thought "didn't we have this already" and found the code comment links over here. [16:39:49] 06SRE, 10LDAP-Access-Requests: Requesting logstash-access LDAP group access for HakanIST - https://phabricator.wikimedia.org/T424812#11880034 (10Dzahn) 05Open→03In progress [16:40:00] FIRING: CirrusStreamingUpdaterRateTooLow: CirrusSearch update rate from flink-app-consumer-cloudelastic is critically low - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterRateTooLow [16:41:38] (03PS3) 10Kamila Součková: WIP: add tracing support [software/spicerack] - 10https://gerrit.wikimedia.org/r/1120500 (owner: 10Volans) [16:41:38] (03CR) 10Kamila Součková: "<3" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1120500 (owner: 10Volans) [16:43:44] (03CR) 10CI reject: [V:04-1] WIP: add tracing support [software/spicerack] - 10https://gerrit.wikimedia.org/r/1120500 (owner: 10Volans) [16:53:29] (03PS1) 10JHathaway: Remove access for derick [puppet] - 10https://gerrit.wikimedia.org/r/1281553 [17:04:37] (03PS1) 10Herron: kafka-logging2004: use trixie jvm settings [puppet] - 10https://gerrit.wikimedia.org/r/1281529 (https://phabricator.wikimedia.org/T417001) [17:11:03] RESOLVED: KafkaUnderReplicatedPartitions: Under replicated partitions for Kafka cluster logging-codfw in codfw - https://wikitech.wikimedia.org/wiki/Kafka/Administration - https://grafana.wikimedia.org/d/000000027/kafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-kafka_cluster=logging-codfw - https://alerts.wikimedia.org/?q=alertname%3DKafkaUnderReplicatedPartitions [17:15:44] !log herron@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging2004.codfw.wmnet with OS trixie [17:19:45] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [17:19:51] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [17:20:02] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [17:20:08] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [17:21:15] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [17:21:20] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [17:21:32] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [17:21:38] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [17:26:15] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [17:26:21] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [17:26:32] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [17:26:38] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [17:28:18] !log herron@cumin1003 START - Cookbook sre.hosts.reimage for host kafka-logging2003.codfw.wmnet with OS trixie [17:28:26] FIRING: [16x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:28:46] !log herron@cumin1003 START - Cookbook sre.hosts.move-vlan for host kafka-logging2003 [17:31:49] herron@cumin1003 reimage (PID 3302269) is awaiting input [17:32:50] (03PS1) 10Herron: kafka-logging2003: update IP and prep for trixie [puppet] - 10https://gerrit.wikimedia.org/r/1281562 (https://phabricator.wikimedia.org/T422816) [17:33:10] !log herron@cumin1003 START - Cookbook sre.dns.netbox [17:38:51] herron@cumin1003 reimage (PID 3302269) is awaiting input [17:40:33] !log herron@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2003 - herron@cumin1003" [17:40:38] !log herron@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2003 - herron@cumin1003" [17:40:38] !log herron@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [17:40:38] !log herron@cumin1003 START - Cookbook sre.dns.wipe-cache kafka-logging2003.codfw.wmnet 24.32.192.10.in-addr.arpa 4.2.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [17:40:42] !log herron@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-logging2003.codfw.wmnet 24.32.192.10.in-addr.arpa 4.2.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [17:40:42] !log herron@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging2003 [17:41:02] !log herron@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging2003 [17:41:02] !log herron@cumin1003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging2003 [17:50:58] FIRING: [4x] CoreBGPDown: Core BGP session down between cr1-drmrs and cr2-eqiad (185.15.58.138) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [17:52:03] FIRING: KafkaUnderReplicatedPartitions: Under replicated partitions for Kafka cluster logging-codfw in codfw - https://wikitech.wikimedia.org/wiki/Kafka/Administration - https://grafana.wikimedia.org/d/000000027/kafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-kafka_cluster=logging-codfw - https://alerts.wikimedia.org/?q=alertname%3DKafkaUnderReplicatedPartitions [17:56:58] PROBLEM - Host mr1-magru.oob is DOWN: PING CRITICAL - Packet loss = 100% [18:00:42] !log herron@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging2003.codfw.wmnet with reason: host reimage [18:03:45] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [18:03:45] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [18:04:02] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [18:04:08] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [18:04:27] (03PS2) 10JHathaway: Remove access for derick [puppet] - 10https://gerrit.wikimedia.org/r/1281553 [18:04:32] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1281553 (owner: 10JHathaway) [18:04:51] !log herron@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging2003.codfw.wmnet with reason: host reimage [18:07:13] RECOVERY - Host mr1-magru.oob is UP: PING OK - Packet loss = 0%, RTA = 117.18 ms [18:08:11] (03PS1) 10Eevans: airflow-main: remove obsolete hosts (from commented entry) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1281587 (https://phabricator.wikimedia.org/T412830) [18:08:14] (03PS1) 10Eevans: revise-tone-task-generator: updated list of aqs cassandra nodes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1281588 (https://phabricator.wikimedia.org/T412830) [18:08:16] (03PS1) 10Eevans: _aqs2-common_: updated aqs node list [deployment-charts] - 10https://gerrit.wikimedia.org/r/1281589 (https://phabricator.wikimedia.org/T412830) [18:09:18] (03CR) 10CDanis: [C:03+1] Remove access for derick [puppet] - 10https://gerrit.wikimedia.org/r/1281553 (owner: 10JHathaway) [18:09:37] (03CR) 10JHathaway: [C:03+2] Remove access for derick [puppet] - 10https://gerrit.wikimedia.org/r/1281553 (owner: 10JHathaway) [18:13:45] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [18:13:45] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [18:13:53] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [18:13:59] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [18:14:45] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [18:14:45] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [18:15:15] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [18:15:15] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [18:19:45] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [18:19:45] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [18:20:15] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [18:20:15] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [18:20:29] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 185.15.58.139 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [18:22:03] RESOLVED: KafkaUnderReplicatedPartitions: Under replicated partitions for Kafka cluster logging-codfw in codfw - https://wikitech.wikimedia.org/wiki/Kafka/Administration - https://grafana.wikimedia.org/d/000000027/kafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-kafka_cluster=logging-codfw - https://alerts.wikimedia.org/?q=alertname%3DKafkaUnderReplicatedPartitions [18:26:57] !log herron@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging2003.codfw.wmnet with OS trixie [18:31:46] (03PS1) 10Dzahn: zuul: add launcher_connection Hiera key to executor role [puppet] - 10https://gerrit.wikimedia.org/r/1281595 (https://phabricator.wikimedia.org/T395938) [18:32:06] (03PS2) 10Dzahn: zuul: add launcher_connection Hiera key to executor role [puppet] - 10https://gerrit.wikimedia.org/r/1281595 (https://phabricator.wikimedia.org/T395938) [18:32:32] (03CR) 10Dzahn: [C:03+2] zuul: add launcher_connection Hiera key to executor role [puppet] - 10https://gerrit.wikimedia.org/r/1281595 (https://phabricator.wikimedia.org/T395938) (owner: 10Dzahn) [18:32:34] !log herron@cumin1003 START - Cookbook sre.hosts.reimage for host kafka-logging2002.codfw.wmnet with OS trixie [18:33:03] !log herron@cumin1003 START - Cookbook sre.hosts.move-vlan for host kafka-logging2002 [18:34:17] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [18:34:23] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [18:35:15] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [18:35:21] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [18:35:35] (03PS1) 10Herron: kafka-logging2002: update IP and prep for trixie [puppet] - 10https://gerrit.wikimedia.org/r/1281596 (https://phabricator.wikimedia.org/T422816) [18:36:06] herron@cumin1003 reimage (PID 3310785) is awaiting input [18:36:55] !log herron@cumin1003 START - Cookbook sre.dns.netbox [18:40:49] !log herron@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2002 - herron@cumin1003" [18:40:55] !log herron@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging2002 - herron@cumin1003" [18:40:55] !log herron@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [18:40:55] !log herron@cumin1003 START - Cookbook sre.dns.wipe-cache kafka-logging2002.codfw.wmnet 50.16.192.10.in-addr.arpa 0.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [18:40:59] !log herron@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-logging2002.codfw.wmnet 50.16.192.10.in-addr.arpa 0.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [18:41:00] !log herron@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging2002 [18:41:13] !log herron@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging2002 [18:41:13] !log herron@cumin1003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging2002 [18:42:09] (03CR) 10Eevans: "Once merged, anyone who might find themselves deploying one of the services that use this should be made aware to expect the diff, and b) " [deployment-charts] - 10https://gerrit.wikimedia.org/r/1281589 (https://phabricator.wikimedia.org/T412830) (owner: 10Eevans) [18:43:44] !log jhathaway@cumin1003 DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Alangi Derick out of all services on: 2442 hosts [18:46:51] FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx errors from eventgate-logging-external.discovery.wmnet in codfw #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=codfw&var-cluster=text&var-origin=eventgate-logging-external.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [18:47:45] looking... [18:49:26] o/ [18:50:01] o/ [18:50:03] jhathaway: that's likely related to the kafka-logging codfw trixie updates I'm working on, wonder if I've missed an ACL as the hosts are moving vlan/ip at the same time [18:50:17] I'm not sure off hand exatly where that's defined [18:51:06] thanks herron, anything we should do? [18:52:17] I see a lot of very informative ""message":"message timed out","origin":"local","stack":"Error: Local: Message timed out","stack_trace":"Error: Local: Message timed out","type":"LibrdKafkaError"},"log.level":"error"" on the pods [18:52:28] herron: o/ did you move another node? [18:52:54] elukey: yeah actually almost done with trixie reimages in codfw, 2001 is the only one remaining and 2002 is in flight right now [18:53:03] FIRING: KafkaUnderReplicatedPartitions: Under replicated partitions for Kafka cluster logging-codfw in codfw - https://wikitech.wikimedia.org/wiki/Kafka/Administration - https://grafana.wikimedia.org/d/000000027/kafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-kafka_cluster=logging-codfw - https://alerts.wikimedia.org/?q=alertname%3DKafkaUnderReplicatedPartitions [18:53:21] I'm guessing there's an acl between eventgate and kafka-logging that needs the new IPs [18:53:44] herron: qq - when you changed the IP, did you roll out the ferm ip change on all nodes first? I mean, running puppet on all of them to refresh the list etc.. [18:53:57] ah yeah external services in k8s! [18:54:04] lemme check [18:54:22] about to get on a plane, but was just about to ask ^ [18:54:48] external-services probably needs an update for the network policy to sync [18:55:07] ahh interesting ok [18:55:08] !log elukey@deploy1003 helmfile [codfw] START helmfile.d/admin 'sync'. [18:55:11] fixing now [18:55:24] (03PS1) 10Eevans: cumin: use aqs1016 as canary alias [puppet] - 10https://gerrit.wikimedia.org/r/1281602 (https://phabricator.wikimedia.org/T412830) [18:55:31] can you DM me the command you're running and I'll include that as I roll through the remaining hosts [18:55:42] !log elukey@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'sync'. [18:56:05] sure, I can paste them here so others can see [18:56:56] on deploy1003: kube-env admin codfw; cd /srv/deployment-charts/helmfile.d/admin_ng; helmfile -e codfw -lname=external-services diffl helmfile -e codfw -lname=external-services sync [18:57:30] basically we have a chart called external-services that is responsible to use puppet defined clusters to create egress network policies for k8s [18:57:38] to have a dry config [18:57:48] in your case, I think all the IPs were the old ones [18:57:55] so eventgate failed to contact any of the newer ones [18:57:56] sweet, makes sense thanks. is that ever refreshed automatically? [18:58:13] once puppet runs on deploy1003, but you need to apply it manually [18:58:22] ahh got it, ok [18:58:31] jhathaway: the graph seems recovering [18:58:38] nice work [18:59:00] thanks elukey jhathaway, sorry for the p-word [19:00:42] herron: for eqiad it is not necessary that you do it after every vlan move, since eventgate will retry other ips etc.. maybe half way through is ok, if you do them in a short timeframe [19:01:42] ok [19:01:51] RESOLVED: ATSBackendErrorsHigh: ATS: elevated 5xx errors from eventgate-logging-external.discovery.wmnet in codfw #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=codfw&var-cluster=text&var-origin=eventgate-logging-external.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHi [19:02:20] * swfrench-wmf files away another data point for "we need to devise a safe mechanism to automatically deploy external-services diffs" [19:03:04] swfrench-wmf: +1 [19:03:32] herron: on the bright side, first kafka prod cluster on trixie + jdk 21! [19:04:12] true! [19:08:04] (03PS1) 10Eevans: Update aqs host list [software/logstash-logback-encoder] - 10https://gerrit.wikimedia.org/r/1281605 (https://phabricator.wikimedia.org/T412830) [19:14:32] (03PS1) 10Ahmon Dancy: scap.cfg.erb: Remove unused canary_service setting [puppet] - 10https://gerrit.wikimedia.org/r/1281606 [19:19:45] 10ops-eqiad, 06SRE, 06DC-Ops: Q3 :rack/setup/install cloudvirt refresh - https://phabricator.wikimedia.org/T425088#11880438 (10elukey) @Jclark-ctr I didn't find the BMC passwords in our shared sheet, could you please send them to me so I can try to run provision? [19:25:45] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [19:25:51] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [19:25:53] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [19:25:59] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [19:34:51] !log dancy@deploy1003 Installing scap version "4.259.0" for 2 host(s) [19:36:42] !log dancy@deploy1003 Installation of scap version "4.259.0" completed for 2 hosts [19:37:28] !log dancy@deploy1003 Started scap sync-world: testing T317405 [19:37:30] T317405: Add failure rate triggered rollback to scap - https://phabricator.wikimedia.org/T317405 [19:40:51] !log dancy@deploy1003 Finished scap sync-world: testing T317405 (duration: 03m 23s) [19:40:57] (03PS1) 10Bvibber: Enable LCStoreStaticArray on beta for live performance testing [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1281609 (https://phabricator.wikimedia.org/T99740) [19:44:17] (03PS5) 10Jasmine: kafka-main: set main-codfw cluster brokers to Confluent distro 77 (3.7) [puppet] - 10https://gerrit.wikimedia.org/r/1278832 (https://phabricator.wikimedia.org/T419216) [19:47:26] (03CR) 10Jasmine: "Adjusted to start with codfw rather, Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1278832 (https://phabricator.wikimedia.org/T419216) (owner: 10Jasmine) [19:47:36] (03CR) 10Jasmine: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1278832 (https://phabricator.wikimedia.org/T419216) (owner: 10Jasmine) [19:49:53] !log herron@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging2002.codfw.wmnet with reason: host reimage [19:49:56] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1269440 (https://phabricator.wikimedia.org/T414338) (owner: 10Krinkle) [19:49:56] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1269441 (https://phabricator.wikimedia.org/T414338) (owner: 10Krinkle) [19:50:52] (03Merged) 10jenkins-bot: Enable wgTrackMediaRequestProvenance on wikidata.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1269440 (https://phabricator.wikimedia.org/T414338) (owner: 10Krinkle) [19:50:57] (03Merged) 10jenkins-bot: Enable wgTrackMediaRequestProvenance on Commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1269441 (https://phabricator.wikimedia.org/T414338) (owner: 10Krinkle) [19:51:14] !log krinkle@deploy1003 Started scap sync-world: Backport for [[gerrit:1269440|Enable wgTrackMediaRequestProvenance on wikidata.org (T414338)]], [[gerrit:1269441|Enable wgTrackMediaRequestProvenance on Commons (T414338)]] [19:51:17] T414338: FY25-26 WE5.4.12: Identify the provenance of image requests - https://phabricator.wikimedia.org/T414338 [19:52:57] !log krinkle@deploy1003 krinkle: Backport for [[gerrit:1269440|Enable wgTrackMediaRequestProvenance on wikidata.org (T414338)]], [[gerrit:1269441|Enable wgTrackMediaRequestProvenance on Commons (T414338)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [19:54:42] !log herron@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging2002.codfw.wmnet with reason: host reimage [19:55:53] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [19:55:59] CirrusSearch consumer-cloudelastic@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [19:56:02] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [19:56:08] CirrusSearch consumer-cloudelastic@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow