[00:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:31:27] FIRING: SystemdUnitDown: The systemd unit rsync_enterprise_htmldumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [01:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:53:48] FIRING: [3x] PuppetZeroResources: Puppet has failed generate resources on clouddumps1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:59:34] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for Waldir Pimenta (Waldyrious) - https://phabricator.wikimedia.org/T375110#10201441 (10Pppery) @KFrancis has typically been the person managing volunteer NDAs. It feels like some team (SRE clinic duty?) should have processed this ticket but i... [04:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:31:27] FIRING: SystemdUnitDown: The systemd unit rsync_enterprise_htmldumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:53:48] FIRING: [3x] PuppetZeroResources: Puppet has failed generate resources on clouddumps1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:52:34] 10PAWS: PAWS replace doesn't work any more - https://phabricator.wikimedia.org/T376448 (10T._Wirbitzki) 03NEW [06:59:51] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:04:34] 06cloud-services-team, 10Toolforge: Support jdk21 on toolforge - https://phabricator.wikimedia.org/T346477#10201595 (10Slst2020) 05Open→03Resolved a:03Slst2020 >>! In T346477#10200693, @Don-vip wrote: > With the power of buildpacks I was even able to update very easily to Java 23 :) https://gitlab.wi... [07:04:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:07:33] 10PAWS: PAWS replace doesn't work any more - https://phabricator.wikimedia.org/T376448#10201600 (10T._Wirbitzki) 05Open→03Invalid [07:12:19] 10PAWS: PAWS replace doesn't work any more - https://phabricator.wikimedia.org/T376448#10201606 (10T._Wirbitzki) The environment works fine. "Execution time: Script terminated successfully." means that the search didn't find anything, which is correct in my examples. I just had to learn again what I forgot... [07:12:26] 10Tool-toolwatch: Implementing alert system to notify maintainers of downtime - https://phabricator.wikimedia.org/T368816#10201605 (10Gopavasanth) {F57588190} Source: https://tool-watch.toolforge.org/ It has been observed that out of a total of 1925 tools, only 543 are currently up and running, while 1382 tools... [07:59:45] (03update) 10sstefanova: storage: add k8s storage [repos/cloud/toolforge/components-api] (slavina/wrap-all-responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 (owner: 10dcaro) [08:05:17] 10Tool-toolwatch: Toolwatch incorrectly reports non-web tools as unavailable - https://phabricator.wikimedia.org/T376451 (10JJMC89) 03NEW [08:06:42] 10Tool-toolwatch: Implementing alert system to notify maintainers of downtime - https://phabricator.wikimedia.org/T368816#10201684 (10JJMC89) {T376451} is a blocker to doing any kind of notifications. [08:13:42] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for madwiktionary - https://phabricator.wikimedia.org/T375023#10201710 (10Gehel) [08:13:48] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for moswiki - https://phabricator.wikimedia.org/T375568#10201714 (10Gehel) p:05Triage→03High [08:14:28] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for shnwikinews - https://phabricator.wikimedia.org/T375432#10201718 (10Gehel) p:05Triage→03High [08:14:35] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for gorwikiquote - https://phabricator.wikimedia.org/T375094#10201719 (10Gehel) p:05Triage→03High [08:14:44] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for madwiktionary - https://phabricator.wikimedia.org/T375023#10201720 (10Gehel) p:05Triage→03High [08:14:51] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for kgewiki - https://phabricator.wikimedia.org/T374814#10201721 (10Gehel) p:05Triage→03High [08:15:08] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for kgewiki - https://phabricator.wikimedia.org/T374814#10201712 (10Gehel) [08:15:20] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for shnwikinews - https://phabricator.wikimedia.org/T375432#10201706 (10Gehel) [08:15:36] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for gorwikiquote - https://phabricator.wikimedia.org/T375094#10201708 (10Gehel) [08:17:32] (03PS1) 10Klausman: hiera: move S3 pseudo-secrets for ml-lab::gpu to the right place [labs/private] - 10https://gerrit.wikimedia.org/r/1077896 [08:17:54] 10Tool-toolwatch: Implementing alert system to notify maintainers of downtime - https://phabricator.wikimedia.org/T368816#10201723 (10Slst2020) There would need to be a way to differentiate between tools that are down because they are "unhealthy" vs. tools that are "intentionally" down. As for notifications, I... [08:31:27] RESOLVED: SystemdUnitDown: The systemd unit rsync_enterprise_htmldumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:37:19] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10201774 (10aborrero) >>! In... [08:39:03] 06cloud-services-team, 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Prepare and check storage layer for moswiki - https://phabricator.wikimedia.org/T375568#10201775 (10Gehel) [08:39:05] 06cloud-services-team, 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Prepare and check storage layer for shnwikinews - https://phabricator.wikimedia.org/T375432#10201776 (10Gehel) [08:39:10] 06cloud-services-team, 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Prepare and check storage layer for gorwikiquote - https://phabricator.wikimedia.org/T375094#10201777 (10Gehel) [08:39:11] 06cloud-services-team, 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Prepare and check storage layer for madwiktionary - https://phabricator.wikimedia.org/T375023#10201778 (10Gehel) [08:39:18] 06cloud-services-team, 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Prepare and check storage layer for kgewiki - https://phabricator.wikimedia.org/T374814#10201779 (10Gehel) [08:41:26] (03update) 10sstefanova: api: wrap errors in ApiResponse [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/15 [08:43:44] (03update) 10sstefanova: api: wrap errors in ApiResponse [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/15 [08:43:58] (03merge) 10sstefanova: api: wrap errors in ApiResponse [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/15 [08:44:02] (03update) 10sstefanova: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 (owner: 10dcaro) [08:45:46] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: DONOTMERGE components-api: bump to 0.0.29-20241002095441-cd2060f1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/544 [08:45:51] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: DONOTMERGE components-api: bump to 0.0.29-20241002095441-cd2060f1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/544 [08:56:39] (03update) 10sstefanova: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 (owner: 10dcaro) [09:13:26] 06cloud-services-team, 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Prepare and check storage layer for moswiki - https://phabricator.wikimedia.org/T375568#10201827 (10BTullis) a:03BTullis [09:13:28] 06cloud-services-team, 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Prepare and check storage layer for shnwikinews - https://phabricator.wikimedia.org/T375432#10201828 (10BTullis) a:03BTullis [09:13:33] 06cloud-services-team, 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Prepare and check storage layer for gorwikiquote - https://phabricator.wikimedia.org/T375094#10201829 (10BTullis) a:03BTullis [09:13:34] 06cloud-services-team, 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Prepare and check storage layer for madwiktionary - https://phabricator.wikimedia.org/T375023#10201830 (10BTullis) a:03BTullis [09:13:41] 06cloud-services-team, 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Prepare and check storage layer for kgewiki - https://phabricator.wikimedia.org/T374814#10201831 (10BTullis) a:03BTullis [09:22:57] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/125 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:22:59] (03approved) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/125 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:23:07] (03merge) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/125 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:25:39] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.337-20241004092316-2f3bd09d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/548 [09:25:43] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/41 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:34:16] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/41 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:51:58] (03approved) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/41 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:52:07] (03merge) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/41 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:53:48] FIRING: [3x] PuppetZeroResources: Puppet has failed generate resources on clouddumps1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:54:03] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.46-20241004095219-90664382 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/549 [09:54:08] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.46-20241004095219-90664382 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/549 [09:57:10] (03PS1) 10Brouberol: Provision dummy keys for cephosd mds servers [labs/private] - 10https://gerrit.wikimedia.org/r/1077926 [09:58:29] (03PS2) 10Brouberol: Provision dummy keys for cephosd mds servers [labs/private] - 10https://gerrit.wikimedia.org/r/1077926 (https://phabricator.wikimedia.org/T376402) [10:01:39] (03CR) 10Btullis: [C:03+1] Provision dummy keys for cephosd mds servers [labs/private] - 10https://gerrit.wikimedia.org/r/1077926 (https://phabricator.wikimedia.org/T376402) (owner: 10Brouberol) [10:03:30] (03CR) 10Brouberol: [C:03+2] Provision dummy keys for cephosd mds servers [labs/private] - 10https://gerrit.wikimedia.org/r/1077926 (https://phabricator.wikimedia.org/T376402) (owner: 10Brouberol) [10:03:33] (03CR) 10Brouberol: [V:03+2 C:03+2] Provision dummy keys for cephosd mds servers [labs/private] - 10https://gerrit.wikimedia.org/r/1077926 (https://phabricator.wikimedia.org/T376402) (owner: 10Brouberol) [10:06:37] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10202010 (10hashar) I could use some help. I have followed the instruction and given my username (`Hashar`) is the same, I went to https://wikitech.wikimedia.org/wiki/Special:MergeAccount I... [10:30:35] 06cloud-services-team, 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Prepare and check storage layer for moswiki - https://phabricator.wikimedia.org/T375568#10202034 (10BTullis) This is now complete. ` btullis@tools-bastion-13:~$ sql moswiki Reading table information for completion... [10:34:09] 06cloud-services-team, 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Prepare and check storage layer for moswiki - https://phabricator.wikimedia.org/T375568#10202037 (10BTullis) 05Open→03Resolved [10:37:33] 06cloud-services-team, 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Prepare and check storage layer for shnwikinews - https://phabricator.wikimedia.org/T375432#10202039 (10BTullis) 05Open→03Resolved This is complete.` btullis@tools-bastion-13:~$ sql shnwikinews Reading tab... [10:45:13] 06cloud-services-team, 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Prepare and check storage layer for madwiktionary - https://phabricator.wikimedia.org/T375023#10202071 (10BTullis) 05Open→03Resolved This is now complete. ` btullis@tools-bastion-13:~$ sql madwiktionary Re... [10:46:27] 06cloud-services-team, 10wikitech.wikimedia.org: Reimage eqiad cloudweb hosts to bookworm - https://phabricator.wikimedia.org/T376277#10202083 (10jijiki) [10:46:32] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: Cleanup: Wikitech code leftovers - https://phabricator.wikimedia.org/T371378#10202084 (10jijiki) [10:46:47] 06cloud-services-team, 10wikitech.wikimedia.org: Reimage eqiad cloudweb hosts to bookworm - https://phabricator.wikimedia.org/T376277#10202086 (10jijiki) [10:46:49] 06cloud-services-team, 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Prepare and check storage layer for gorwikiquote - https://phabricator.wikimedia.org/T375094#10202062 (10BTullis) 05Open→03Resolved This is now complete. ` btullis@tools-bastion-13:~$ sql gorwikiquote Read... [10:47:02] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: ☂ Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10202087 (10jijiki) [10:47:23] 06cloud-services-team, 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Prepare and check storage layer for kgewiki - https://phabricator.wikimedia.org/T374814#10202075 (10BTullis) 05Open→03Resolved This is now complete. ` btullis@tools-bastion-13:~$ sql kgewiki Reading table... [10:49:15] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: mediawiki-config: consolidate labswiki - https://phabricator.wikimedia.org/T371374#10202066 (10jijiki) 05Stalled→03Resolved Any other cleanup work will be attached under T371378, marking this as done [10:49:17] 06cloud-services-team, 10Horizon, 10Striker, 10wikitech.wikimedia.org: consider eliminating labweb/cloudweb hardware servers - https://phabricator.wikimedia.org/T305233#10202088 (10jijiki) [10:49:24] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: ☂ Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10202089 (10jijiki) [10:52:30] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: dns/netbox: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462 (10aborrero) 03NEW [11:04:05] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: dns/netbox: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10202145 (10cmooney) This patch covers the delegation for the openstack-managed ranges, I think it's correct? https://g... [11:14:24] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: dns: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10202157 (10aborrero) [11:16:06] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: dns: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10202161 (10cmooney) To be more clear, you need to make sure these two zones are working on the openstack authdns: ` 0.0.0.0.0.... [11:21:01] (03open) 10aborrero: codfw1dev: dns: add reverse zones for 2a02:ec80:a100::/48 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/85 (https://phabricator.wikimedia.org/T376462) [11:22:18] (03update) 10aborrero: codfw1dev: dns: add reverse zones for 2a02:ec80:a100::/48 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/85 (https://phabricator.wikimedia.org/T376462) [11:25:34] (03open) 10aborrero: zones: import only if they have the import_id attribute set [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/86 (https://phabricator.wikimedia.org/T374338) [11:26:12] (03update) 10aborrero: codfw1dev: dns: add reverse zones for 2a02:ec80:a100::/48 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/85 (https://phabricator.wikimedia.org/T376462) [11:28:27] (03update) 10aborrero: zones: import only if they have the import_id attribute set [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/86 (https://phabricator.wikimedia.org/T374338) [11:29:46] (03update) 10aborrero: codfw1dev: dns: add reverse zones for 2a02:ec80:a100::/48 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/85 (https://phabricator.wikimedia.org/T376462) [11:31:30] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [11:32:59] (03merge) 10aborrero: zones: import only if they have the import_id attribute set [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/86 (https://phabricator.wikimedia.org/T374338) [11:34:12] (03update) 10aborrero: tofu-infra: rename data/ dir to templates/ [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/83 (https://phabricator.wikimedia.org/T375283) [11:34:31] (03merge) 10aborrero: tofu-infra: rename data/ dir to templates/ [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/83 (https://phabricator.wikimedia.org/T375283) [11:34:45] (03update) 10aborrero: Draft: dns: track the svc.project.deploy.wikimedia.cloud zone via tofu-infra [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/84 (https://phabricator.wikimedia.org/T376110) [11:35:12] (03update) 10aborrero: Draft: dns: track the svc.project.deploy.wikimedia.cloud zone via tofu-infra [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/84 (https://phabricator.wikimedia.org/T376110) [11:35:25] (03update) 10aborrero: codfw1dev: dns: add reverse zones for 2a02:ec80:a100::/48 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/85 (https://phabricator.wikimedia.org/T376462) [11:35:33] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:35:42] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [11:36:13] (03update) 10aborrero: codfw1dev: dns: add reverse zones for 2a02:ec80:a100::/48 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/85 (https://phabricator.wikimedia.org/T376462) [11:37:05] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [11:38:10] (03open) 10aborrero: locals: fix reference to old data/ directory [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/87 [11:38:16] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [11:38:56] (03merge) 10aborrero: locals: fix reference to old data/ directory [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/87 [11:39:03] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:39:28] (03update) 10aborrero: codfw1dev: dns: add reverse zones for 2a02:ec80:a100::/48 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/85 (https://phabricator.wikimedia.org/T376462) [11:39:40] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [11:42:06] (03update) 10sstefanova: jobs-api: bump to 0.0.337-20241004092316-2f3bd09d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/548 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:42:48] (03update) 10aborrero: codfw1dev: dns: add reverse zones for 2a02:ec80:a100::/48 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/85 (https://phabricator.wikimedia.org/T376462) [11:43:22] (03update) 10sstefanova: api-gateway: bump to 0.0.46-20241004095219-90664382 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/549 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:44:31] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [11:44:48] (03approved) 10sstefanova: jobs-api: bump to 0.0.337-20241004092316-2f3bd09d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/548 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:44:53] (03merge) 10sstefanova: jobs-api: bump to 0.0.337-20241004092316-2f3bd09d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/548 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:45:46] (03update) 10sstefanova: api-gateway: bump to 0.0.46-20241004095219-90664382 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/549 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:45:58] 10wikitech.wikimedia.org: MABot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376222#10202209 (10MarcoAurelio) 05Open→03In progress p:05Triage→03Medium a:03MarcoAurelio [11:46:26] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [11:47:28] 10wikitech.wikimedia.org: MABot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376222#10202220 (10MarcoAurelio) I've attached the Wikitech account to the global one through Special:MergeAccount. The bot seems able to edit after that. [11:47:47] (03update) 10aborrero: codfw1dev: dns: add reverse zones for 2a02:ec80:a100::/48 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/85 (https://phabricator.wikimedia.org/T376462) [11:50:33] (03update) 10aborrero: codfw1dev: dns: add reverse zones for 2a02:ec80:a100::/48 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/85 (https://phabricator.wikimedia.org/T376462) [11:50:47] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [11:51:15] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [11:52:00] (03merge) 10aborrero: codfw1dev: dns: add reverse zones for 2a02:ec80:a100::/48 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/85 (https://phabricator.wikimedia.org/T376462) [11:52:16] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:52:52] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [11:54:48] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, and 2 others: dns: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10202237 (10aborrero) before and after merging the tofu-infra patch above: `lang=shell-session arturo@nostromo:~ $ dig SO... [11:57:17] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [11:58:55] 10wikitech.wikimedia.org: MABot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376222#10202248 (10MarcoAurelio) 05In progress→03Resolved [12:00:14] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, and 2 others: dns: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10202238 (10aborrero) 05Open→03In progress p:05Triage→03Medium [12:01:33] 10Tool-toolwatch: Implementing alert system to notify maintainers of downtime - https://phabricator.wikimedia.org/T368816#10202256 (10Sophivorus) Hi! I very much like the idea, and wouldn't mind to receive an email warning me if one of my tool is down. That being said, when I opened the tool, first thing I did... [12:11:30] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10202260 (10Ladsgroup) @hashar Please try https://wikitech.wikimedia.org/wiki/Special:PasswordReset and reset your password. https://idm.wikimedia.org username and password is now completely... [12:12:52] (03approved) 10sstefanova: api-gateway: bump to 0.0.46-20241004095219-90664382 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/549 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [12:12:55] (03merge) 10sstefanova: api-gateway: bump to 0.0.46-20241004095219-90664382 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/549 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [12:34:42] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10202286 (10hashar) One can... [12:35:05] (03update) 10aborrero: projects: refactor back into a single file [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/88 (https://phabricator.wikimedia.org/T375283) [12:35:11] (03open) 10aborrero: projects: refactor back into a single file [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/88 (https://phabricator.wikimedia.org/T375283) [13:20:27] (03open) 10sstefanova: api: add auth [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/16 [13:49:28] (03update) 10sstefanova: builds-builder: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 (https://phabricator.wikimedia.org/T374908) (owner: 10dcaro) [13:49:45] (03update) 10sstefanova: builds-builder: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 (https://phabricator.wikimedia.org/T374908) (owner: 10dcaro) [13:53:49] FIRING: [3x] PuppetZeroResources: Puppet has failed generate resources on clouddumps1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [14:09:16] (03CR) 10Hashar: "It is not urgent, I would just like code search to be moved back to the Gerrit replica to cut the logging spam it causes on the primary Ge" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/920243 (https://phabricator.wikimedia.org/T336710) (owner: 10Hashar) [14:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:16:15] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10202748 (10aborrero) I have... [15:27:30] (03update) 10aborrero: projects: refactor back into a single file [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/88 (https://phabricator.wikimedia.org/T375283) [15:30:45] (03update) 10aborrero: projects: refactor back into a single file [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/88 (https://phabricator.wikimedia.org/T375283) [15:41:42] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10202834 (10Dominic3203) |**Wikitech account/LDAP:**| Dominic3203| |**SUL account**| Dominic3203| |**Account linked on [[ https://idm.wikimedia.org/ | IDM ]]** |Y| |**I have visited [[ https:... [16:07:24] 10Tool-toolwatch: Toolwatch incorrectly reports non-web tools as unavailable - https://phabricator.wikimedia.org/T376451#10202942 (10TheresNoTime) a:03TheresNoTime https://github.com/gopavasanth/ToolWatch/pull/20 [16:08:49] FIRING: [4x] PuppetZeroResources: Puppet has failed generate resources on cloudcephosd1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [16:18:49] FIRING: [4x] PuppetZeroResources: Puppet has failed generate resources on cloudcephosd1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [16:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:35:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:15:27] 10Tool-toolwatch: Toolwatch incorrectly reports non-web tools as unavailable - https://phabricator.wikimedia.org/T376451#10203225 (10Gopavasanth) Thanks for the quick patch, @TheresNoTime!! PR is merged and will deploy this change at the earliest possible :) [17:19:09] (03CR) 10Dzahn: "Gotcha! Ok, good. I was wondering if it was in any way a follow-up to the outage." [labs/codesearch] - 10https://gerrit.wikimedia.org/r/920243 (https://phabricator.wikimedia.org/T336710) (owner: 10Hashar) [17:42:38] (03CR) 10Majavah: "Which errors are you seeing? I can't seem to be able to reproduce. Also we should try updating the images (Keystone to Caracal/2024.1 whic" [labs/striker] - 10https://gerrit.wikimedia.org/r/1077474 (owner: 10BryanDavis) [17:57:49] 06cloud-services-team, 10Toolforge, 10Elasticsearch: Add access control for Toolforge Elasticsearch - https://phabricator.wikimedia.org/T348943#10203324 (10RoySmith) @bd808 following up to our conversation earlier today, it turns out this ticket already exists (and I had forgotten about it). I'll just add,... [18:50:20] 06cloud-services-team, 10Toolforge, 10Elasticsearch: Add access control for Toolforge Elasticsearch - https://phabricator.wikimedia.org/T348943#10203511 (10bd808) I'm going to be bold and suggest that we rewrite the root task here to be about deploying OpenSearch with [[https://opensearch.org/docs/latest/sec... [18:50:21] 10Tool-toolwatch: Implementing alert system to notify maintainers of downtime - https://phabricator.wikimedia.org/T368816#10203509 (10Soylacarli) Hi, This is an awesome idea/tool! I recently started mapping some tools myself and encountered quite a few that were no longer available, so this will definitely help.... [18:51:56] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for Waldir Pimenta (Waldyrious) - https://phabricator.wikimedia.org/T375110#10203512 (10KFrancis) Hi @waldyrious, please email your full name, mailing address, and email to kfrancis@wikimedia.org and I'll put the NDA agreement together for you... [19:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:53:16] 10Toolforge (Quota-requests): Request increased quota for video-answer-tool Toolforge tool - https://phabricator.wikimedia.org/T376505 (10derenrich) 03NEW [20:05:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:18:49] FIRING: [3x] PuppetZeroResources: Puppet has failed generate resources on clouddumps1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:04:38] (03CR) 10BryanDavis: "I would have to do some piddling locally to recreate the issue, but it was generally with both images that one of the official apt repos w" [labs/striker] - 10https://gerrit.wikimedia.org/r/1077474 (owner: 10BryanDavis) [21:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:47:31] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in 20d 23h 58m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [23:07:30] 10Tool-toolwatch: Implementing alert system to notify maintainers of downtime - https://phabricator.wikimedia.org/T368816#10204182 (10Fnielsen) I am running Ordia at https://ordia.toolforge.org/. When I view the status at https://tool-watch.toolforge.org/search?search=ordia it says Unavailable, but the tool is a...