[00:34:17] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#11726122 (10Dzahn) [00:50:01] 10Beta-Cluster-Infrastructure: Some production sites on the beta cluster were unable to be selected as import targets - https://phabricator.wikimedia.org/T410109#11726143 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/refs/heads/master/wmf-config/CommonSettings-labs.php co... [00:56:32] 10Beta-Cluster-Infrastructure: Some production sites on the beta cluster were unable to be selected as import targets - https://phabricator.wikimedia.org/T410109#11726164 (10bd808) >>! In T410109#11726141, @bd808 wrote: > @TTO do you have any objections to ripping this config experiment out and going back to a s... [01:00:12] 10Beta-Cluster-Infrastructure: Some production sites on the beta cluster were unable to be selected as import targets - https://phabricator.wikimedia.org/T410109#11726167 (10bd808) 05Openโ†’03In progress p:05Triageโ†’03Medium a:03bd808 [01:37:18] 10Beta-Cluster-Infrastructure: Some production sites on the beta cluster were unable to be selected as import targets - https://phabricator.wikimedia.org/T410109#11726226 (10TTO) Thanks for the ping, @bd808. The idea was to allow any WMF wiki to import from any other WMF wiki without requiring a config change ev... [06:47:57] 10Gerrit, 06Release-Engineering-Team, 14Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with repo... - https://phabricator.wikimedia.org/T246763#11726482 [09:16:48] 06Release-Engineering-Team (Doing ๐Ÿ˜Ž), 06WMF-NDA-Requests: NDA task access - ABreault-WMF - https://phabricator.wikimedia.org/T420541#11726663 (10Aklapper) 05Openโ†’03Resolved a:03Aklapper Done. Bonus points for [linking your LDAP account to your Phabricator account](https://phabricator.wikimedia.org/se... [09:17:26] 06Release-Engineering-Team (Doing ๐Ÿ˜Ž), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T413811#11726672 (10Aklapper) 05Openโ†’03Resolved This looks fine. ๐Ÿ”ฅ [10:17:18] 10GitLab (Project Migration), 10Wikispeech-Jobrunner, 10Wikispeech-Text-to-Speech: Move Speechoid components to Gitlab - https://phabricator.wikimedia.org/T360758#11726806 (10Viktoria_Hillerud_WMSE) I will start on wikispeech_server [11:35:13] 06Gerrit-Privilege-Requests: Requesting access to +2 on operations/deployment-charts for trueg and lerickson - https://phabricator.wikimedia.org/T420568#11727023 (10taavi) [11:36:34] 06Gerrit-Privilege-Requests: Requesting access to +2 on operations/deployment-charts for trueg and lerickson - https://phabricator.wikimedia.org/T420568#11727027 (10taavi) 05Openโ†’03Resolved a:03taavi Added you to the wmf-deployment Gerrit group as you are in the corresponding Unix group. [12:28:54] (03open) 10viktoriahillerudwmse: Add Wikispeech_server to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/156 [13:10:51] (03update) 10viktoriahillerudwmse: Add Wikispeech_server to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/156 [14:09:31] FIRING: ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:09:40] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T420523#11727747 (10phaultfinder) [14:14:31] RESOLVED: ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:20:55] 06Release-Engineering-Team, 10MediaWiki-extensions-WikimediaMaintenance, 10Wiki-Setup (Create): addWiki.php fails with CannotReplaceActiveServiceException for DBLoadBalancerFactory - https://phabricator.wikimedia.org/T420531#11727816 (10Jdforrester-WMF) OK, I've quickly read through all calls I could find to... [14:29:21] (03PS1) 10Sbisson: Zuul: [mediawiki/extensions/ArticleGuidance]: Add TestKitchen as phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1255746 (https://phabricator.wikimedia.org/T420385) [14:34:18] 06Release-Engineering-Team, 10MediaWiki-extensions-WikimediaMaintenance, 10Wiki-Setup (Create): addWiki.php fails with CannotReplaceActiveServiceException for DBLoadBalancerFactory - https://phabricator.wikimedia.org/T420531#11727900 (10Jdforrester-WMF) Tried running interactively: ` jforrester@deploy2002:~... [14:46:35] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11727954 (10ABran-WMF) The `MaxRequestWorkers` bump will also help for [[ h... [14:47:05] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - Service gerrit2003:443 has failed probes (http_gerrit_tls_ip6) - https://phabricator.wikimedia.org/T420523#11727956 (10ABran-WMF) p:05Triageโ†’03Medium [14:50:42] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - Service gerrit2003:443 has failed probes (http_gerrit_tls_ip6) - https://phabricator.wikimedia.org/T420523#11727981 (10ABran-WMF) 05Openโ†’03Resolved a:03ABran-WMF gerrit was scraped while a load increase happened, all available httpd wo... [14:50:56] 06Release-Engineering-Team, 10MediaWiki-extensions-WikimediaMaintenance, 10Wiki-Setup (Create): addWiki.php fails with CannotReplaceActiveServiceException for DBLoadBalancerFactory - https://phabricator.wikimedia.org/T420531#11727985 (10Jdforrester-WMF) Shorter version, without even calling addWiki: ` jforr... [15:03:48] (03merge) 10dancy: Add Wikispeech_server to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/156 (owner: 10viktoriahillerudwmse) [15:04:46] 10Beta-Cluster-Infrastructure: beta-scap-sync-world tries to communicate with deployment-parsoid14.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420509#11728031 (10dancy) 05Openโ†’03Resolved Repaired. [15:15:53] 06Release-Engineering-Team, 06collaboration-services: gerrit: Throttle traffic to gitiles - https://phabricator.wikimedia.org/T420595 (10ABran-WMF) 03NEW [15:16:24] 06Release-Engineering-Team (Doing ๐Ÿ˜Ž), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Fix access check for multiple env deletion endpoint - https://phabricator.wikimedia.org/T420596 (10jnuche) 03NEW [15:16:45] 06Release-Engineering-Team (Doing ๐Ÿ˜Ž), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Fix access check for multiple env deletion endpoint - https://phabricator.wikimedia.org/T420596#11728104 (10jnuche) [15:16:49] 06Release-Engineering-Team, 06collaboration-services: gerrit: Throttle traffic to gitiles - https://phabricator.wikimedia.org/T420595#11728105 (10ABran-WMF) 05Openโ†’03In progress p:05Triageโ†’03Medium [15:17:39] 06Release-Engineering-Team, 06collaboration-services: gerrit: Throttle traffic to gitiles - https://phabricator.wikimedia.org/T420595#11728114 (10ABran-WMF) [15:17:41] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit behind CDN - https://phabricator.wikimedia.org/T411895#11728115 (10ABran-WMF) [15:21:17] (03CR) 10Krinkle: fresh-install: fix fresh-node24 checksum and install validation (031 comment) [fresh] - 10https://gerrit.wikimedia.org/r/1227819 (https://phabricator.wikimedia.org/T414756) (owner: 10Prashant_32194) [15:21:43] (03open) 10dancy: add-project.py: get_projects_for_runner: Add pagination support [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/157 [15:21:46] (03update) 10dancy: add-project.py: get_projects_for_runner: Add pagination support [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/157 [15:23:21] (03merge) 10dancy: add-project.py: get_projects_for_runner: Add pagination support [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/157 [15:26:15] (03open) 10dancy: add-project.py: get_projects_for_runner: Report errors [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/158 [15:26:18] (03update) 10dancy: add-project.py: get_projects_for_runner: Report errors [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/158 [15:27:35] (03update) 10dancy: add-project.py: get_projects_for_runner: Report errors [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/158 [15:28:03] (03merge) 10dancy: add-project.py: get_projects_for_runner: Report errors [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/158 [15:28:41] 06Release-Engineering-Team (Doing ๐Ÿ˜Ž), 06WMF-NDA-Requests: NDA task access - ABreault-WMF - https://phabricator.wikimedia.org/T420541#11728184 (10ABreault-WMF) Thanks [15:31:02] (03PS5) 10Krinkle: Prepare v25.09.1 release [fresh] - 10https://gerrit.wikimedia.org/r/1186537 (owner: 10Jforrester) [15:31:48] (03CR) 10Krinkle: Prepare v25.09.1 release (031 comment) [fresh] - 10https://gerrit.wikimedia.org/r/1186537 (owner: 10Jforrester) [15:32:05] (03PS6) 10Krinkle: Prepare v25.09.1 release [fresh] - 10https://gerrit.wikimedia.org/r/1186537 (owner: 10Jforrester) [15:32:15] (03PS7) 10Krinkle: Prepare v25.09.1 release [fresh] - 10https://gerrit.wikimedia.org/r/1186537 (https://phabricator.wikimedia.org/T414756) (owner: 10Jforrester) [15:32:51] (03open) 10dancy: add-project.py: get_projects_for_runner: Fix pagination query [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/159 [15:32:55] (03update) 10dancy: add-project.py: get_projects_for_runner: Fix pagination query [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/159 [15:34:01] (03update) 10dancy: add-project.py: get_projects_for_runner: Fix pagination query [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/159 [15:34:13] (03PS1) 10Krinkle: Release v25.09.1 [fresh] - 10https://gerrit.wikimedia.org/r/1255761 [15:34:54] (03merge) 10dancy: add-project.py: get_projects_for_runner: Fix pagination query [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/159 [15:35:00] (03CR) 10Krinkle: "Amended to update README as well (now documented in CONTRIBUTING.md)" [fresh] - 10https://gerrit.wikimedia.org/r/1186538 (owner: 10Jforrester) [15:35:13] (03Abandoned) 10Krinkle: Release v25.09.1 [fresh] - 10https://gerrit.wikimedia.org/r/1255761 (owner: 10Krinkle) [15:35:24] (03PS4) 10Krinkle: Release v25.09.1 [fresh] - 10https://gerrit.wikimedia.org/r/1186538 (owner: 10Jforrester) [15:35:52] (03PS5) 10Krinkle: Release v25.09.1 [fresh] - 10https://gerrit.wikimedia.org/r/1186538 (owner: 10Jforrester) [15:36:44] 10Fresh: Update Fresh's version of Node 18 from 18.20.2 to 18.20.4 - https://phabricator.wikimedia.org/T373938#11728263 (10Krinkle) 05Openโ†’03Resolved a:03MusikAnimal [15:39:18] (03PS8) 10Krinkle: Prepare v25.09.1 release [fresh] - 10https://gerrit.wikimedia.org/r/1186537 (https://phabricator.wikimedia.org/T414756) (owner: 10Jforrester) [15:39:18] (03PS6) 10Krinkle: Release v25.09.1 [fresh] - 10https://gerrit.wikimedia.org/r/1186538 (owner: 10Jforrester) [15:39:36] 10Fresh, 13Patch-For-Review: fresh-node24 is not installed by fresh-install due to bad checksum - https://phabricator.wikimedia.org/T414756#11728285 (10Krinkle) p:05Triageโ†’03Medium a:03Krinkle [15:40:33] 10Beta-Cluster-Infrastructure, 06Content-Transform-Team: Maps don't display on the beta cluster - https://phabricator.wikimedia.org/T420299#11728295 (10bd808) The maps-beta.wmflabs.org proxy points at the [[https://openstack-browser.toolforge.org/server/deployment-maps-master02.deployment-prep.eqiad1.wikimedia... [15:41:02] 10Beta-Cluster-Infrastructure, 06Content-Transform-Team: Maps don't display on the beta cluster; tile server offline on deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420299#11728297 (10bd808) [15:43:16] 10Phabricator, 10Bitu, 06cloud-services-team, 06Infrastructure-Foundations, and 3 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599 (10Dondersmooi) 03NEW [15:43:19] Could a gerrit admin edit the "labs-tools-guc" group and either delete/archive it or edit its description in a way that says "ARCHIVED: Use tools-guc instead". https://gerrit.wikimedia.org/r/admin/groups/255123622003e17680e62aa8fc6a032fd0d79906 [15:43:32] I didn't realize that group still existed, but apparently it does. Afaik it isn't used. [15:43:37] per https://gerrit.wikimedia.org/r/admin/repos/labs/tools/guc,access [15:46:30] Krinkle: {{done}} [15:48:46] 10Beta-Cluster-Infrastructure, 06Content-Transform-Team: Maps don't display on the beta cluster; tile server offline on deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420299#11728350 (10awight) I don't have any clues, but +1 to bd808's suggested next stop! [15:49:25] taavi: Thanks <3 [15:49:38] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikimediaEvents, 06Test Kitchen: Make WikimediaEvents depend on TestKitchen - https://phabricator.wikimedia.org/T419679#11728358 (10KReid-WMF) [15:50:09] !log Create labs/xtools repo (branch: main, parent: labs, owner: labs-xtools), ref T402086 [15:50:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:50:13] T402086: Move XTools from Github to Gerrit - https://phabricator.wikimedia.org/T402086 [15:52:50] 10Phabricator, 10Bitu, 06cloud-services-team, 06Infrastructure-Foundations, 10Striker: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11728389 (10JJMC89) [15:52:59] 06Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week), 10MediaWiki Train Development Environment, 05Train Deployments: Train Enquiry - https://phabricator.wikimedia.org/T420602 (10Pakistanyes) 03NEW [15:53:26] (03CR) 10Jforrester: [C:04-1] "We should probably point to different releases at this point, 6 months later?" [fresh] - 10https://gerrit.wikimedia.org/r/1186537 (https://phabricator.wikimedia.org/T414756) (owner: 10Jforrester) [15:56:09] 06Release-Engineering-Team, 06MediaWiki-Engineering, 10MediaWiki-extensions-WikimediaMaintenance, 10Wiki-Setup (Create): addWiki.php fails with CannotReplaceActiveServiceException for DBLoadBalancerFactory - https://phabricator.wikimedia.org/T420531#11728432 (10Jdforrester-WMF) p:05Triageโ†’03Unbreak! [15:57:13] 10Phabricator: Phabricator does not show archived milestones as archived - https://phabricator.wikimedia.org/T420605 (10taavi) 03NEW [16:00:09] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikimediaEvents, 06Test Kitchen: Make WikimediaEvents depend on TestKitchen - https://phabricator.wikimedia.org/T419679#11728517 (10KReid-WMF) [16:01:04] !log Hoist l10n-bot rights from labs/tools parent to labs parent to reduce duplication in other labs/ repos [16:01:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:09:25] (03CR) 10CI reject: [V:04-1] Release v25.09.1 [fresh] - 10https://gerrit.wikimedia.org/r/1186538 (owner: 10Jforrester) [16:22:13] (03PS1) 10Krinkle: zuul: Define Jenkins jobs for xtools repo [integration/config] - 10https://gerrit.wikimedia.org/r/1255777 (https://phabricator.wikimedia.org/T402086) [16:29:06] (03CR) 10Krinkle: [C:03+2] zuul: Define Jenkins jobs for xtools repo [integration/config] - 10https://gerrit.wikimedia.org/r/1255777 (https://phabricator.wikimedia.org/T402086) (owner: 10Krinkle) [16:30:41] (03Merged) 10jenkins-bot: zuul: Define Jenkins jobs for xtools repo [integration/config] - 10https://gerrit.wikimedia.org/r/1255777 (https://phabricator.wikimedia.org/T402086) (owner: 10Krinkle) [16:33:04] 06Release-Engineering-Team, 06MediaWiki-Engineering, 10MediaWiki-extensions-WikimediaMaintenance, 10Wiki-Setup (Create): addWiki.php fails with CannotReplaceActiveServiceException for DBLoadBalancerFactory - https://phabricator.wikimedia.org/T420531#11728712 (10Jdforrester-WMF) 05Openโ†’03Resolved a:... [16:54:44] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1255777 [16:54:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:17:01] 10Phabricator-Bot-Requests: Create mcp-phabricator bot account - https://phabricator.wikimedia.org/T420446#11728957 (10Aklapper) > Another approach might be to look at not requiring an API token at all for searching. This doesn't seem like that big of a stretch, given that we allow searching via the web interfac... [17:19:31] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#11728963 (10Dzahn) [17:28:12] 10Beta-Cluster-Infrastructure: Some production sites on the beta cluster were unable to be selected as import targets - https://phabricator.wikimedia.org/T410109#11729016 (10bd808) `lang=shell-session bd808@deployment-deploy04:~$ mwscript shell.php zhwiki Psy Shell v0.12.20 (PHP 8.3.30 โ€” cli) by Justin Hileman >... [17:57:26] (03CR) 10Jforrester: "-1 Why is this not testing in PHP 8.5?" [integration/config] - 10https://gerrit.wikimedia.org/r/1255777 (https://phabricator.wikimedia.org/T402086) (owner: 10Krinkle) [17:58:49] (03CR) 10Krinkle: [C:03+2] "Because I'm importing the repo and it uses php82-84 on github. Changing that is a post-import proposal that you can pose to MusikAnimal on" [integration/config] - 10https://gerrit.wikimedia.org/r/1255777 (https://phabricator.wikimedia.org/T402086) (owner: 10Krinkle) [18:11:18] (03approved) 10jhuneidi: image: Derive exported variable names from `CI_JOB_NAME_SLUG` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/165 (owner: 10dduvall) [18:11:53] (03merge) 10jhuneidi: image: Derive exported variable names from `CI_JOB_NAME_SLUG` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/165 (owner: 10dduvall) [18:12:34] (03CR) 10Jforrester: "We're not adding new repos without full CI testing, as each one adds hours of follow-up Phab creation, chasing, and eventually just doing " [integration/config] - 10https://gerrit.wikimedia.org/r/1255777 (https://phabricator.wikimedia.org/T402086) (owner: 10Krinkle) [18:19:31] FIRING: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:19:42] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T420625 (10phaultfinder) 03NEW [18:20:29] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#11729260 (10Dzahn) [18:21:58] (03open) 10jhuneidi: Version 2.16.0 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/166 [18:22:54] (03approved) 10dduvall: Version 2.16.0 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/166 (owner: 10jhuneidi) [18:24:31] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:25:36] (03merge) 10dduvall: Version 2.16.0 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/166 (owner: 10jhuneidi) [18:38:31] FIRING: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:38:45] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T420625#11729392 (10phaultfinder) [18:43:31] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:54:32] FIRING: InstanceDown: Project deployment-prep instance deployment-cirrussearch12 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:54:38] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cirrussearch12 is down - https://phabricator.wikimedia.org/T420642 (10wmcs-alerts) 03NEW [18:59:32] FIRING: [5x] InstanceDown: Project deployment-prep instance deployment-cache-upload08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:03:28] Project mediawiki-core-doxygen build #18729: 04FAILURE in 33 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18729/ [19:04:32] RESOLVED: [5x] InstanceDown: Project deployment-prep instance deployment-cache-upload08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:09:31] FIRING: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:09:45] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T420625#11729650 (10phaultfinder) [19:14:31] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:15:46] Project beta-code-update-eqiad build #592355: 04FAILURE in 2 min 45 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/592355/ [19:16:31] FIRING: ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:16:36] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T420625#11729673 (10phaultfinder) [19:21:31] FIRING: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:21:34] Project mediawiki-core-doxygen build #18730: 04STILL FAILING in 3 min 30 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18730/ [19:21:59] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T420625#11729681 (10phaultfinder) [19:23:36] Project beta-code-update-eqiad build #592356: 04STILL FAILING in 35 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/592356/ [19:35:25] Yippee, build fixed! [19:35:25] Project beta-code-update-eqiad build #592357: 09FIXED in 2 min 24 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/592357/ [19:36:31] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:39:31] FIRING: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:40:32] (03open) 10dduvall: image: Remove redundant dashes from exported variable names [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/167 [19:44:31] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:45:05] (03update) 10dduvall: image: Remove redundant dashes from exported variable names [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/167 [19:48:42] Hello, would anyone be able to assist me on why I'm no longer able to ssh into the prod deploy server? I have been able to `ssh deployment.eqiad.wmnet` or `ssh deployment.codfw.wmnet` from 'my location', `bast4005.wikimedia.org` but now I receive: `ssh: Could not resolve hostname bast4005.wikimedia.org: nodename nor servname provided, or not known [19:48:42] Connection closed by UNKNOWN port 65535` when I try shelling in [19:49:22] Project beta-code-update-eqiad build #592358: 04FAILURE in 2.1 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/592358/ [19:50:35] ecarg: do any of the other bastions work? [19:50:48] ecarg: in your ssh config, try replacing bast4005 with one of "bast1004", "bast2003", "bast5004" [19:50:49] ecarg: see the email that was sent to the ops list earlier this month, bast4005 is down for maintenance and bast1003/2003 should be used in the meantime [19:53:15] Project beta-code-update-eqiad build #592359: 04STILL FAILING in 14 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/592359/ [19:54:31] FIRING: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:54:43] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T420625#11729720 (10phaultfinder) [19:55:49] RhinosF1 had not tried any of the other bastions, mutante + taavi thank you, I used 1003; also I did not receive that email (perhaps bc I'm not in ops?) is there a way I can be added to the mailing address? [19:59:13] (03update) 10dduvall: image: Remove redundant dashes from exported variable names [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/167 [19:59:18] (03update) 10dduvall: image: Remove redundant dashes from exported variable names [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/167 [19:59:36] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:00:11] ecarg: sending you a link in PM [20:03:31] FIRING: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:03:42] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T420625#11729742 (10phaultfinder) [20:05:32] (03update) 10jhuneidi: image: Remove redundant dashes from exported variable names [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/167 (owner: 10dduvall) [20:05:33] (03approved) 10jhuneidi: image: Remove redundant dashes from exported variable names [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/167 (owner: 10dduvall) [20:08:31] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:09:23] Yippee, build fixed! [20:09:23] Project beta-code-update-eqiad build #592360: 09FIXED in 6 min 22 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/592360/ [20:17:16] 10GitLab: GitLab Private Repository Request for: MediaWiki JS CSS Scan Signatures, MediaWiki JS CSS Scan Tool - https://phabricator.wikimedia.org/T420646 (10sbassett) 03NEW [20:17:37] 10GitLab: GitLab Private Repository Request for: MediaWiki JS CSS Scan Signatures, MediaWiki JS CSS Scan Tool - https://phabricator.wikimedia.org/T420646#11729774 (10sbassett) [20:33:25] Yippee, build fixed! [20:33:25] Project mediawiki-core-doxygen build #18731: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18731/ [20:44:08] Project beta-code-update-eqiad build #592364: 04FAILURE in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/592364/ [20:44:31] FIRING: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:55:22] Project beta-code-update-eqiad build #592365: 04STILL FAILING in 2 min 21 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/592365/ [20:59:31] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:05:26] Yippee, build fixed! [21:05:26] Project beta-code-update-eqiad build #592366: 09FIXED in 2 min 25 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/592366/ [22:21:48] Project mediawiki-core-doxygen build #18733: 04FAILURE in 3 min 44 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18733/ [22:23:31] FIRING: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:28:31] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [23:32:56] Yippee, build fixed! [23:32:56] Project mediawiki-core-doxygen build #18734: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18734/