[00:39:30] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1251604 [00:39:31] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1251604 (owner: 10TrainBranchBot) [00:52:31] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1251604 (owner: 10TrainBranchBot) [01:09:28] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1251609 [01:09:28] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1251609 (owner: 10TrainBranchBot) [01:11:39] (03PS1) 10C. Scott Ananian: Turn on postprocessing cache for all Parsoid parses [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251610 (https://phabricator.wikimedia.org/T348255) [01:26:40] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1251609 (owner: 10TrainBranchBot) [01:34:59] FIRING: [2x] HelmReleaseBadStatus: Helm release kserve/kserve on k8s-mlserve@codfw in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [02:00:47] !log mwpresync@deploy2002 Started scap build-images: Publishing wmf/next image [02:08:39] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:08:47] !log mwpresync@deploy2002 Finished scap build-images: Publishing wmf/next image (duration: 08m 00s) [02:11:03] (03CR) 10Pppery: Uninstall AbuseFilter from closed wikis with no AbuseFilter logs (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251582 (https://phabricator.wikimedia.org/T420063) (owner: 10Dreamy Jazz) [02:33:39] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:34:59] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461) {#3909}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [02:42:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:34:59] FIRING: [2x] HelmReleaseBadStatus: Helm release kserve/kserve on k8s-mlserve@codfw in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [05:35:55] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 26957288 and 3 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [05:36:55] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 3069592 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [06:29:39] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2002 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [06:34:59] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461) {#3909}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [06:42:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:29:39] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2002 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [07:59:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2014.codfw.wmnet, wdqs2013.codfw.wmnet, wdqs2015.codfw.wmnet, wdqs2007.codfw.wmnet, wdqs2022.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [07:59:17] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2014.codfw.wmnet, wdqs2015.codfw.wmnet, wdqs2007.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [08:05:17] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [08:06:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [08:08:17] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2014.codfw.wmnet, wdqs2012.codfw.wmnet, wdqs2022.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [08:09:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2007.codfw.wmnet, wdqs2015.codfw.wmnet, wdqs2011.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [08:10:17] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [08:11:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [08:13:17] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2021.codfw.wmnet, wdqs2007.codfw.wmnet, wdqs2008.codfw.wmnet, wdqs2010.codfw.wmnet, wdqs2013.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [08:14:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2007.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [08:15:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [08:15:17] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [08:18:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2008.codfw.wmnet, wdqs2013.codfw.wmnet, wdqs2022.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [08:18:17] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2021.codfw.wmnet, wdqs2014.codfw.wmnet, wdqs2012.codfw.wmnet, wdqs2013.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [08:20:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [08:20:17] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [08:37:56] (03CR) 10Dreamrimmer: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251193 (https://phabricator.wikimedia.org/T419105) (owner: 10Codename Noreste) [08:40:57] (03CR) 10Dreamrimmer: idwiki: Remove unused user groups on Indonesian Wikipedia (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251193 (https://phabricator.wikimedia.org/T419105) (owner: 10Codename Noreste) [08:49:11] (03CR) 10Ladsgroup: Uninstall GlobalBlocking from closed wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251589 (https://phabricator.wikimedia.org/T420062) (owner: 10Dreamy Jazz) [08:50:57] (03PS5) 10Dreamy Jazz: Uninstall GlobalBlocking from closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251589 (https://phabricator.wikimedia.org/T420062) [08:51:43] (03CR) 10Dreamy Jazz: Uninstall GlobalBlocking from closed wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251589 (https://phabricator.wikimedia.org/T420062) (owner: 10Dreamy Jazz) [08:51:47] (03PS6) 10Dreamy Jazz: Uninstall GlobalBlocking from closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251589 (https://phabricator.wikimedia.org/T420062) [08:53:59] (03CR) 10Ladsgroup: [C:03+1] Uninstall AbuseFilter from closed wikis with no AbuseFilter logs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251582 (https://phabricator.wikimedia.org/T420063) (owner: 10Dreamy Jazz) [08:54:30] (03PS1) 10Elukey: java: add java-21-security erb template [puppet] - 10https://gerrit.wikimedia.org/r/1251836 (https://phabricator.wikimedia.org/T420083) [08:55:15] (03PS2) 10Elukey: java: add java-21-security erb template [puppet] - 10https://gerrit.wikimedia.org/r/1251836 (https://phabricator.wikimedia.org/T420083) [08:55:15] (03CR) 10Dreamy Jazz: Uninstall AbuseFilter from closed wikis with no AbuseFilter logs (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251582 (https://phabricator.wikimedia.org/T420063) (owner: 10Dreamy Jazz) [08:56:05] (03CR) 10Elukey: java: add java-21-security erb template (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1251836 (https://phabricator.wikimedia.org/T420083) (owner: 10Elukey) [08:58:23] (03CR) 10Elukey: "LGTM, will roll it out on monday!" [puppet] - 10https://gerrit.wikimedia.org/r/1251539 (https://phabricator.wikimedia.org/T420034) (owner: 10Majavah) [08:58:35] (03CR) 10Elukey: "LGTM, will roll it out on monday!" [puppet] - 10https://gerrit.wikimedia.org/r/1251540 (https://phabricator.wikimedia.org/T420034) (owner: 10Majavah) [09:34:59] FIRING: [2x] HelmReleaseBadStatus: Helm release kserve/kserve on k8s-mlserve@codfw in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [09:42:57] (03PS1) 10Dreamy Jazz: Disable CheckUser on closed wikis where no checks were ever made [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251848 (https://phabricator.wikimedia.org/T420062) [10:05:20] FIRING: [2x] CirrusSearchPoolCounterRejectionTooHigh: ... [10:05:26] MediaWiki CirrusSearch failing to obtain a token from the pool counter at a very high rate - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Pool_Counter_rejections_(search_is_currently_too_busy) - https://grafana.wikimedia.org/d/qrOStmdGk/elasticsearch-pool-counters?viewPanel=4&orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchPoolCounterRejectionTooHigh [10:05:30] FIRING: [2x] PoolcounterFullQueues: Full queues for poolcounter1006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [10:10:20] RESOLVED: [2x] CirrusSearchPoolCounterRejectionTooHigh: ... [10:10:20] MediaWiki CirrusSearch failing to obtain a token from the pool counter at a very high rate - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Pool_Counter_rejections_(search_is_currently_too_busy) - https://grafana.wikimedia.org/d/qrOStmdGk/elasticsearch-pool-counters?viewPanel=4&orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchPoolCounterRejectionTooHigh [10:10:30] RESOLVED: [2x] PoolcounterFullQueues: Full queues for poolcounter1006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [10:18:00] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1251863 [10:18:30] (03CR) 10Ladsgroup: [C:03+1] Disable CheckUser on closed wikis where no checks were ever made [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251848 (https://phabricator.wikimedia.org/T420062) (owner: 10Dreamy Jazz) [10:18:49] (03CR) 10Ladsgroup: [C:03+1] Uninstall GlobalBlocking from closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251589 (https://phabricator.wikimedia.org/T420062) (owner: 10Dreamy Jazz) [10:33:41] (03PS1) 10Dreamy Jazz: Uninstall SecurePoll from closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251865 (https://phabricator.wikimedia.org/T420062) [10:34:59] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461) {#3909}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [10:35:12] (03PS7) 10Dreamy Jazz: Uninstall GlobalBlocking from closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251589 (https://phabricator.wikimedia.org/T420062) [10:35:25] (03PS2) 10Dreamy Jazz: Disable CheckUser on closed wikis where no checks were ever made [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251848 (https://phabricator.wikimedia.org/T420062) [10:35:25] (03PS2) 10Dreamy Jazz: Uninstall SecurePoll from closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251865 (https://phabricator.wikimedia.org/T420062) [10:42:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:37:32] (03PS1) 10Dreamy Jazz: DiscussionTools: Uninstall from closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) [11:37:57] (03PS2) 10Dreamy Jazz: DiscussionTools: Uninstall from closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) [11:48:01] (03CR) 10A smart kitten: "Would this stop [comment permalinks](https://www.mediawiki.org/wiki/Help:DiscussionTools#Talk_pages_permalinking) from working on these wi" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) (owner: 10Dreamy Jazz) [11:54:07] (03CR) 10Bartosz Dziewoński: "Yes, I was going to say the same thing. It would be fine to remove DiscussionTools from wikis that have been closed before we invented Dis" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) (owner: 10Dreamy Jazz) [12:08:20] (03CR) 10Dreamy Jazz: [C:04-1] "Per the comments" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) (owner: 10Dreamy Jazz) [12:14:51] (03PS8) 10Dreamy Jazz: Uninstall GlobalBlocking from closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251589 (https://phabricator.wikimedia.org/T420062) [12:15:03] (03PS3) 10Dreamy Jazz: Disable CheckUser on closed wikis where no checks were ever made [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251848 (https://phabricator.wikimedia.org/T420062) [12:15:03] (03PS3) 10Dreamy Jazz: Uninstall SecurePoll from closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251865 (https://phabricator.wikimedia.org/T420062) [12:15:03] (03PS3) 10Dreamy Jazz: DiscussionTools: Uninstall from closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) [12:16:03] (03CR) 10A smart kitten: "(I mean, FWIW, it seems possible that someone might've copied a link to a DiscussionTools permalink on a wiki after that wiki was closed. " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) (owner: 10Dreamy Jazz) [12:20:46] (03CR) 10Ladsgroup: [C:03+1] Uninstall SecurePoll from closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251865 (https://phabricator.wikimedia.org/T420062) (owner: 10Dreamy Jazz) [12:26:05] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/8276/co" [puppet] - 10https://gerrit.wikimedia.org/r/1251494 (https://phabricator.wikimedia.org/T407485) (owner: 10Btullis) [12:27:25] (03CR) 10Majavah: [V:03+1 C:03+1] "PCC shows that this will leave behind a bunch of stuff relating to monitoring and logging, +1 if you're fine with that (or will clean it u" [puppet] - 10https://gerrit.wikimedia.org/r/1251494 (https://phabricator.wikimedia.org/T407485) (owner: 10Btullis) [12:28:56] (03PS4) 10Reedy: Specify class in IRC RCFeed setup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251912 (owner: 10Lewis Cawte) [12:30:57] (03PS5) 10Lewis Cawte: CommonSettings: Specify class in IRC RCFeed setup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251912 [12:35:53] (03CR) 10Bartosz Dziewoński: "It's possible, yes, but IMO not really worth worrying about." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) (owner: 10Dreamy Jazz) [12:41:59] (03CR) 10Reedy: [C:03+2] CommonSettings: Specify class in IRC RCFeed setup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251912 (owner: 10Lewis Cawte) [12:42:58] (03Merged) 10jenkins-bot: CommonSettings: Specify class in IRC RCFeed setup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251912 (owner: 10Lewis Cawte) [12:44:55] !log reedy@deploy2002 Started scap sync-world: Backport for [[gerrit:1251912|CommonSettings: Specify class in IRC RCFeed setup]] [12:46:51] !log reedy@deploy2002 reedy, lcawte: Backport for [[gerrit:1251912|CommonSettings: Specify class in IRC RCFeed setup]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:47:14] !log reedy@deploy2002 reedy, lcawte: Continuing with sync [12:47:34] deployment time \o/ [12:47:44] you turn next? :P [12:51:14] !log reedy@deploy2002 Finished scap sync-world: Backport for [[gerrit:1251912|CommonSettings: Specify class in IRC RCFeed setup]] (duration: 06m 19s) [12:57:08] (03PS4) 10Dreamy Jazz: DiscussionTools: Uninstall wikis closed before the tool was created [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) [12:57:43] (03PS5) 10Dreamy Jazz: DiscussionTools: Uninstall wikis closed before the tool was created [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) [12:59:35] (03PS6) 10Dreamy Jazz: DiscussionTools: Uninstall wikis closed before the tool was created [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) [12:59:41] (03PS7) 10Dreamy Jazz: DiscussionTools: Uninstall wikis closed before the tool was created [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) [13:02:11] (03CR) 10Dreamy Jazz: "Yeah, I would agree not worth worrying about that." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) (owner: 10Dreamy Jazz) [13:02:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:04:54] (03PS8) 10Dreamy Jazz: DiscussionTools: Uninstall from wikis closed before the tool was created [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) [13:34:59] FIRING: [2x] HelmReleaseBadStatus: Helm release kserve/kserve on k8s-mlserve@codfw in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [13:44:28] (03CR) 10Bartosz Dziewoński: "You could use 2024 as the cutoff date if you wanted to disable it on more wikis. Deployment ticket: T302011" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) (owner: 10Dreamy Jazz) [13:56:07] (03PS9) 10Dreamy Jazz: DiscussionTools: Uninstall wikis closed before the tool was created [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) [13:56:43] (03CR) 10Dreamy Jazz: "Thanks, I've updated it to January 2024" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) (owner: 10Dreamy Jazz) [13:58:30] (03PS3) 10Dreamy Jazz: Uninstall AbuseFilter from closed wikis with no AbuseFilter logs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251582 (https://phabricator.wikimedia.org/T420063) [13:58:30] (03PS9) 10Dreamy Jazz: Uninstall GlobalBlocking from closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251589 (https://phabricator.wikimedia.org/T420062) [13:58:31] (03PS4) 10Dreamy Jazz: Disable CheckUser on closed wikis where no checks were ever made [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251848 (https://phabricator.wikimedia.org/T420062) [13:58:31] (03PS4) 10Dreamy Jazz: Uninstall SecurePoll from closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251865 (https://phabricator.wikimedia.org/T420062) [13:58:32] (03PS10) 10Dreamy Jazz: DiscussionTools: Uninstall wikis closed before the tool was created [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) [14:03:48] (03PS2) 10Reedy: CommonSettings: Set class in $wgCentralAuthRC [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251941 [14:03:57] (03CR) 10Dreamy Jazz: Uninstall AbuseFilter from closed wikis with no AbuseFilter logs (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251582 (https://phabricator.wikimedia.org/T420063) (owner: 10Dreamy Jazz) [14:07:04] (03CR) 10Reedy: [C:03+2] CommonSettings: Set class in $wgCentralAuthRC [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251941 (owner: 10Reedy) [14:08:18] (03Merged) 10jenkins-bot: CommonSettings: Set class in $wgCentralAuthRC [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251941 (owner: 10Reedy) [14:10:03] !log reedy@deploy2002 Started scap sync-world: Backport for [[gerrit:1251941|CommonSettings: Set class in $wgCentralAuthRC]] [14:11:52] !log reedy@deploy2002 reedy: Backport for [[gerrit:1251941|CommonSettings: Set class in $wgCentralAuthRC]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:12:21] !log reedy@deploy2002 reedy: Continuing with sync [14:16:20] !log reedy@deploy2002 Finished scap sync-world: Backport for [[gerrit:1251941|CommonSettings: Set class in $wgCentralAuthRC]] (duration: 06m 17s) [14:16:54] Suprised pikachu face [14:34:59] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461) {#3909}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [15:10:11] 10SRE-tools, 06Infrastructure-Foundations, 06serviceops-radar: Add --min-uptime to cookbooks - https://phabricator.wikimedia.org/T419967#11710143 (10Aklapper) [15:44:09] (03CR) 10A smart kitten: "Potentially just as a note to myself rather than specifically anyone else — if I understand correctly, some of the data stored by Discussi" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) (owner: 10Dreamy Jazz) [15:56:01] (03CR) 10A smart kitten: Uninstall SecurePoll from closed wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251865 (https://phabricator.wikimedia.org/T420062) (owner: 10Dreamy Jazz) [16:04:35] (03CR) 10A smart kitten: Uninstall GlobalBlocking from closed wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251589 (https://phabricator.wikimedia.org/T420062) (owner: 10Dreamy Jazz) [16:08:02] (03CR) 10Dreamy Jazz: Uninstall SecurePoll from closed wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251865 (https://phabricator.wikimedia.org/T420062) (owner: 10Dreamy Jazz) [16:08:40] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:10:21] (03PS10) 10Dreamy Jazz: Uninstall GlobalBlocking from closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251589 (https://phabricator.wikimedia.org/T420062) [16:10:27] (03CR) 10Dreamy Jazz: Uninstall GlobalBlocking from closed wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251589 (https://phabricator.wikimedia.org/T420062) (owner: 10Dreamy Jazz) [16:10:52] (03PS11) 10Dreamy Jazz: DiscussionTools: Uninstall wikis closed before permalinks were deployed [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) [16:11:57] (03PS12) 10Dreamy Jazz: DiscussionTools: Uninstall wikis closed before permalinks were deployed [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) [16:12:02] (03CR) 10Dreamy Jazz: DiscussionTools: Uninstall wikis closed before permalinks were deployed (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) (owner: 10Dreamy Jazz) [16:12:08] (03PS4) 10Dreamy Jazz: Uninstall AbuseFilter from closed wikis with no AbuseFilter logs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251582 (https://phabricator.wikimedia.org/T420063) [16:12:08] (03PS11) 10Dreamy Jazz: Uninstall GlobalBlocking from closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251589 (https://phabricator.wikimedia.org/T420062) [16:12:08] (03PS5) 10Dreamy Jazz: Disable CheckUser on closed wikis where no checks were ever made [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251848 (https://phabricator.wikimedia.org/T420062) [16:12:08] (03PS5) 10Dreamy Jazz: Uninstall SecurePoll from closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251865 (https://phabricator.wikimedia.org/T420062) [16:12:09] (03PS13) 10Dreamy Jazz: DiscussionTools: Uninstall wikis closed before permalinks were deployed [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251888 (https://phabricator.wikimedia.org/T420052) [16:33:40] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:34:59] FIRING: [2x] HelmReleaseBadStatus: Helm release kserve/kserve on k8s-mlserve@codfw in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [18:13:05] (03PS1) 10Elukey: Disable notifications for db1253 [puppet] - 10https://gerrit.wikimedia.org/r/1252002 (https://phabricator.wikimedia.org/T420041) [18:34:59] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461) {#3909}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [20:16:57] FIRING: ProbeDown: Service wikifeeds:4101 has failed probes (http_wikifeeds_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wikifeeds:4101 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:21:57] RESOLVED: ProbeDown: Service wikifeeds:4101 has failed probes (http_wikifeeds_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wikifeeds:4101 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:34:59] FIRING: [2x] HelmReleaseBadStatus: Helm release kserve/kserve on k8s-mlserve@codfw in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [21:49:41] PROBLEM - Host titan1002 is DOWN: PING CRITICAL - Packet loss = 100% [21:49:53] RECOVERY - Host titan1002 is UP: PING WARNING - Packet loss = 0%, RTA = 1281.83 ms [21:50:53] PROBLEM - Host titan1002 is DOWN: PING CRITICAL - Packet loss = 100% [21:52:03] FIRING: ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:53:39] RECOVERY - Host titan1002 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [21:57:03] RESOLVED: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:34:59] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461) {#3909}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown