[00:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:08:03] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications, 07Wikimedia-production-error: Wikitech notifications failing to load cross-wiki - https://phabricator.wikimedia.org/T376305#10198412 (10Etonkovidova) [00:12:24] 10Cloud-VPS (Project-requests), 07affects-Miraheze: Request creation of createwikitest VPS project - https://phabricator.wikimedia.org/T375454#10198422 (10Xaloria) Ok got it I will ensure the thing you said in the future. [00:31:26] FIRING: SystemdUnitDown: The systemd unit rsync_enterprise_htmldumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [03:26:57] RECOVERY - Host cloudcephosd1025 is UP: PING WARNING - Packet loss = 50%, RTA = 3.03 ms [03:33:00] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for Waldir Pimenta (Waldyrious) - https://phabricator.wikimedia.org/T375110#10198603 (10taavi) >>! In T375110#10196324, @bd808 wrote: > The next step on https://wikitech.wikimedia.org/wiki/Volunteer_NDA seems to be "After that, ask in the Phabr... [03:33:21] PROBLEM - Host cloudcephosd1025 is DOWN: PING CRITICAL - Packet loss = 100% [03:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:31:26] FIRING: SystemdUnitDown: The systemd unit rsync_enterprise_htmldumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:28:00] 10Tool-translatetagger: Add support for wiki links - https://phabricator.wikimedia.org/T376364 (10Gopavasanth) 03NEW [06:56:43] 10Tool-translatetagger, 06Indic-TechCom, 10Technical-Tool-Request: Tool to convert Wikitext into translatable wiki-text - https://phabricator.wikimedia.org/T372243#10198677 (10Gopavasanth) [07:09:08] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for moswiki - https://phabricator.wikimedia.org/T375568#10198693 (10ABran-WMF) [08:02:56] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10198750 (10hashar) p:05Unb... [08:20:50] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10198785 (10Joe) Please next... [08:26:58] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10198804 (10Joe) Also, havin... [08:27:27] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10198805 (10hashar) From the... [08:31:26] FIRING: SystemdUnitDown: The systemd unit rsync_enterprise_htmldumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:53:48] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10198877 (10dcaro) Hmm, it's... [09:17:20] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10198891 (10hashar) I went o... [09:42:24] 10Tool-yearinreview: Add Support for URL Parameters to Enable Sharable Links - https://phabricator.wikimedia.org/T376371 (10Gopavasanth) 03NEW [09:56:42] 06cloud-services-team, 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114#10198967 (10jijiki) Thanks @taavi for pointing it out, we'll try to find a temporary bandaid to keep dumps running... [10:03:56] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10198973 (10dcaro) >>! In T3... [10:08:13] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10198979 (10aborrero) The cl... [11:00:09] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10199043 (10cmooney) As of n... [11:22:38] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10199083 (10aborrero) I noti... [11:23:00] 10Tool-toolwatch: Redesign Tool Details Navigation and Button Text - https://phabricator.wikimedia.org/T376173#10199086 (10Kiyohan) a:03Kiyohan [11:24:19] 10Tool-toolwatch: Redesign Tool Details Navigation and Button Text - https://phabricator.wikimedia.org/T376173#10199087 (10Kiyohan) 05Open→03In progress raised a PR at https://github.com/gopavasanth/ToolWatch/pull/16 [11:28:41] (03open) 10aborrero: tofu-infra: rename data/ dir to templates/ [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/83 (https://phabricator.wikimedia.org/T375283) [11:37:15] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] (slavina/wrap-all-responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [11:38:18] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] (slavina/wrap-all-responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [11:44:05] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), and 2 others: Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10199126 (10cmooney) >>! In T374830#10199083, @abor... [11:56:33] 10Tool-toolwatch, 07good first task: Create a stats Page for ToolWatch - https://phabricator.wikimedia.org/T375967#10199144 (10MahimaSinghal) {F57585777} [11:58:38] 10Tool-toolwatch, 07good first task: Create a stats Page for ToolWatch - https://phabricator.wikimedia.org/T375967#10199148 (10Gopavasanth) @ MahimaSinghal could you please raise the PR too and link here? Thanks! [12:25:11] (03open) 10aborrero: dns: track the svc.project.deploy.wikimedia.cloud zone via tofu-infra [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/84 (https://phabricator.wikimedia.org/T376110) [12:25:52] (03update) 10aborrero: dns: track the svc.project.deploy.wikimedia.cloud zone via tofu-infra [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/84 (https://phabricator.wikimedia.org/T376110) [12:27:35] (03update) 10aborrero: dns: track the svc.project.deploy.wikimedia.cloud zone via tofu-infra [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/84 (https://phabricator.wikimedia.org/T376110) [12:30:57] (03update) 10aborrero: dns: track the svc.project.deploy.wikimedia.cloud zone via tofu-infra [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/84 (https://phabricator.wikimedia.org/T376110) [12:31:26] FIRING: SystemdUnitDown: The systemd unit rsync_enterprise_htmldumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [12:36:24] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), and 2 others: Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10199225 (10aborrero) >>! In T374830#10199126, @cmo... [12:38:51] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: CloudVPS: IPv6 in codfw1dev - https://phabricator.wikimedia.org/T245495#10199231 (10aborrero) [12:41:31] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: netbox: create IPv6 entries for Cloud VPS - https://phabricator.wikimedia.org/T374712#10199227 (10aborrero) 05Open→03Resolved [12:41:37] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: Cloud IPv6 subnets - https://phabricator.wikimedia.org/T187929#10199229 (10aborrero) [12:42:25] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10199233 (10aborrero) please @Jhancock.wm try again with this one after the patch I merged yesterday. [12:48:21] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: cloudsw: codfw: enable IPv6 - https://phabricator.wikimedia.org/T374713#10199236 (10aborrero) Created: * https://netbox.wikimedia.org/ipam/prefixes/1085/ * https://netbox.wikimedia.org/ipam/prefixes/1086/ * https://netbox.... [12:50:54] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops, 13Patch-For-Review: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10199240 (10Ladsgroup) This probably needs a ticket to undeploy the extension altogether fro... [12:57:09] 10Tool-toolwatch, 07good first task: Create a stats Page for ToolWatch - https://phabricator.wikimedia.org/T375967#10199249 (10Gopavasanth) 05Open→03Resolved a:03MahimaSinghal Thanks Mahima :-) Changes are live now: https://tool-watch.toolforge.org/ [13:05:51] 10wikitech.wikimedia.org, 10TimedMediaHandler, 07Wikimedia-production-error: Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'labswiki.transcode' doesn't existFunction: MediaWiki\TimedMediaHandler\WebVideoTranscode\WebVideoTranscode::getTranscodeStateQuery: SELECT *... - https://phabricator.wikimedia.org/T376382 [13:11:54] 10Tool-toolwatch: Improvise pagination experience - https://phabricator.wikimedia.org/T376175#10199320 (10MahimaSinghal) https://github.com/gopavasanth/ToolWatch/pull/19 {F57585971} [13:16:01] (03update) 10raymond-ndibe: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 (owner: 10dcaro) [13:23:25] 10wikitech.wikimedia.org, 10TimedMediaHandler, 07Wikimedia-production-error: Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'labswiki.transcode' doesn't existFunction: MediaWiki\TimedMediaHandler\WebVideoTranscode\WebVideoTranscode::getTranscodeStateQuery:... - https://phabricator.wikimedia.org/T376382#10199350 [13:23:31] 10wikitech.wikimedia.org, 10TimedMediaHandler, 07Wikimedia-production-error: Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'labswiki.transcode' doesn't existFunction: MediaWiki\TimedMediaHandler\WebVideoTranscode\WebVideoTranscode::getTranscodeStateQuery:... - https://phabricator.wikimedia.org/T376382#10199348 [13:34:42] 10wikitech.wikimedia.org: Cookie “WMF-Last-Access-Global” has been rejected for invalid domain. - https://phabricator.wikimedia.org/T376384 (10Reedy) 03NEW [13:41:04] 10Tool-toolwatch: Improvise pagination experience - https://phabricator.wikimedia.org/T376175#10199409 (10Gopavasanth) a:03MahimaSinghal [13:50:24] (03approved) 10dcaro: all: upgrade to tekton 0.59.X LTS [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/111 (https://phabricator.wikimedia.org/T374908) [13:50:28] (03merge) 10dcaro: all: upgrade to tekton 0.59.X LTS [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/111 (https://phabricator.wikimedia.org/T374908) [13:50:31] (03approved) 10dcaro: tekton: upgrade to v0.59.3 [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/61 [13:50:35] (03merge) 10dcaro: tekton: upgrade to v0.59.3 [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/61 [13:52:07] (03PS1) 10Klausman: hiera: add pseudosecret for S3 access from ml-lab machines [labs/private] - 10https://gerrit.wikimedia.org/r/1077720 [13:52:11] (03CR) 10Klausman: [V:03+2 C:03+2] "check experimental" [labs/private] - 10https://gerrit.wikimedia.org/r/1077720 (owner: 10Klausman) [13:52:27] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-builder: bump to 0.0.121-20241003135043-98c2199c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/546 [13:57:34] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-api: bump to 0.0.173-20241003135043-0a8f9093 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/547 (https://phabricator.wikimedia.org/T374908) [13:59:16] (03close) 10dcaro: builds-builder: bump to 0.0.121-20241003135043-98c2199c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/546 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [13:59:34] (03close) 10dcaro: builds-api: bump to 0.0.173-20241003135043-0a8f9093 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/547 (https://phabricator.wikimedia.org/T374908) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [14:00:43] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [14:03:56] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 (https://phabricator.wikimedia.org/T374908) [14:10:43] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 (https://phabricator.wikimedia.org/T374908) [14:47:31] (03merge) 10aborrero: projects: create 'test-project-creation-delete-me' project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/82 [14:47:46] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [14:48:18] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [14:52:28] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [14:53:46] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [14:54:54] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [14:55:27] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [14:59:41] (03approved) 10fnegri: tofu-infra: rename data/ dir to templates/ [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/83 (https://phabricator.wikimedia.org/T375283) (owner: 10aborrero) [15:00:19] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10199752 (10jeremyb) |**Wikitech account/LDAP:**| Jeremyb | |**SUL account**| Jeremyb | |**Account linked on IDM** |Y| |**I have read [[ https://wikitech.wikimedia.org/wiki/MediaWiki:Loginpro... [15:00:42] (03approved) 10fnegri: dns: track the svc.project.deploy.wikimedia.cloud zone via tofu-infra [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/84 (https://phabricator.wikimedia.org/T376110) (owner: 10aborrero) [15:18:01] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10199846 (10Ladsgroup) Can you try to reset your password in wikitech. Set it a temp password since it'll be thrown away after SUL unification? [15:19:36] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10199851 (10jeremyb) sure, of course, I just thought we were trying to unify without that? (or else docs were confusing) and once I've done that then there's no going back to test the curren... [15:24:36] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10199866 (10Jdforrester-WMF) [15:36:30] (03approved) 10dcaro: api-gateway: enable components-api on local [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/545 [15:36:33] (03merge) 10dcaro: api-gateway: enable components-api on local [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/545 [15:58:02] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10199971 (10hashar) > There... [16:07:32] (03update) 10sstefanova: api: wrap errors in ApiResponse [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/15 [16:09:06] 06cloud-services-team, 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114#10199975 (10jijiki) 05Open→03In progress p:05Triage→03Unbreak! [16:13:39] (03update) 10sstefanova: storage: add k8s storage [repos/cloud/toolforge/components-api] (slavina/wrap-all-responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 (owner: 10dcaro) [16:20:31] 06cloud-services-team, 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114#10200015 (10jijiki) 05Resolved→03In progress [16:20:34] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10200018 (10brennen) [16:20:35] 10wikitech.wikimedia.org, 10TimedMediaHandler, 07Wikimedia-production-error: Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'labswiki.transcode' doesn't existFunction: MediaWiki\TimedMediaHandler\WebVideoTranscode\WebVideoTranscode::getTranscodeStateQuery:... - https://phabricator.wikimedia.org/T376382#10200017 [16:20:47] 06cloud-services-team, 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114#10200019 (10jijiki) p:05Unbreak!→03Low [16:24:58] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10200023 (10Amire80) |**Wikitech account/LDAP:**|amire80| |**SUL account**|amire80| |**Account linked on [[ https://idm.wikimedia.org/ | IDM ]]** |Y| |**I have visited [[ https://wikitech.wik... [16:27:43] 06cloud-services-team, 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114#10200009 (10jijiki) 05In progress→03Resolved a:03jijiki @Ladsgroup and I believe this... [16:29:44] (03update) 10dcaro: DONOTMERGE components-api: bump to 0.0.29-20241002095441-cd2060f1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/544 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:30:37] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10200045 (10matmarex) @Amire80 I requested a password reset for you, you should get it at your email and should be able to proceed with account unification from there. [16:31:26] FIRING: SystemdUnitDown: The systemd unit rsync_enterprise_htmldumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:31:29] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10200059 (10Amire80) >>! In T376267#10200045, @matmarex wrote: > @Amire80 I requested a password reset for you, you should get it at your email and should be able to proceed with account unif... [16:33:28] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops, 13Patch-For-Review: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10200066 (10Pppery) {T376097} was already created. [17:03:52] (03approved) 10dcaro: api: wrap errors in ApiResponse [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/15 (owner: 10sstefanova) [17:07:55] 10Tool-toolwatch, 06Indic MediaWiki Developers UG, 06Indic-TechCom: Sort tools based on tool Title - https://phabricator.wikimedia.org/T353579#10200225 (10MahimaSinghal) {F57586498} I have added an option for users to sort the tools based on tool title. This is something you were expecting, right? [17:22:52] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10200334 (10dcaro) @Ladsgroup Hi, following up on https://wikitech.wikimedia.org/wiki/Wikitech:Rename_requests#c-DCaro_(WMF)-20241001151800-David_Caro (as I can't reply there) I tried using... [17:22:54] 10Tool-toolwatch, 06Indic MediaWiki Developers UG, 06Indic-TechCom: Sort tools based on tool Title - https://phabricator.wikimedia.org/T353579#10200335 (10MahimaSinghal) a:03MahimaSinghal [17:38:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on clouddumps1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [17:39:41] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10200422 (10Ladsgroup) >>! In T376267#10200334, @dcaro wrote: > @Ladsgroup Hi, following up on https://wikitech.wikimedia.org/wiki/Wikitech:Rename_requests#c-DCaro_(WMF)-20241001151800-David_... [17:43:48] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on clouddumps1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [17:48:54] 06cloud-services-team, 10Data-Services, 05Goal: [toolsdb] Upgrade to MariaDB 10.6 - https://phabricator.wikimedia.org/T352206#10200447 (10Slst2020) [17:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:53:48] FIRING: [3x] PuppetZeroResources: Puppet has failed generate resources on clouddumps1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [18:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:49:20] 06cloud-services-team, 10Toolforge: Support jdk21 on toolforge - https://phabricator.wikimedia.org/T346477#10200693 (10Don-vip) With the power of buildpacks I was even able to update very easily to Java 23 :) https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/commit/c00aee97338ec2cb0af4051e2d82555aa8033e8f [19:10:18] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: cloudsw: codfw: enable IPv6 - https://phabricator.wikimedia.org/T374713#10200735 (10cmooney) >>! In T374713#10199236, @aborrero wrote: > Created: Thanks! I've made some minor edits to them in Netbox btw, just some things... [19:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:31:27] FIRING: SystemdUnitDown: The systemd unit rsync_enterprise_htmldumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [21:53:48] FIRING: [3x] PuppetZeroResources: Puppet has failed generate resources on clouddumps1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [22:47:30] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in 21d 23h 58m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [23:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks