[01:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:58:16] 10Tool-bullseye: replace bullseye's Shodan key - https://phabricator.wikimedia.org/T382563 (10AntiCompositeNumber) 03NEW [05:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:44:45] 10Tool-translatetagger, 06Indic-TechCom, 10Technical-Tool-Request: Tool to convert Wikitext into translatable wiki-text - https://phabricator.wikimedia.org/T372243#10417963 (10Aafi) A test-page to help improve created at: https://meta.wikimedia.org/wiki/User:Aafi/Test_for_Bhubaneswar_Hackathon [07:01:48] (03PS1) 10Harsv567: Support for Type-Hinting [labs/tools/bookreader] - 10https://gerrit.wikimedia.org/r/1105829 [09:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:26:24] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 06DBA: Prepare and check storage layer for tigwiki - https://phabricator.wikimedia.org/T381378#10418296 (10fnegri) 05In progress→03Resolved [10:28:10] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for idwikivoyage - https://phabricator.wikimedia.org/T381079#10418300 (10fnegri) 05In progress→03Resolved [10:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:40:02] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10418689 [14:14:20] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10418771 [14:15:24] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10418774 [14:21:56] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install cloudcephosd2004-dev - https://phabricator.wikimedia.org/T378825#10418786 (10Jhancock.wm) @Andrew so I thought i fixed it. turns out its not working. I was having the same issue after running reimage. After it ran once, i... [14:50:22] (03update) 10sstefanova: openapi: add description to all operations [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/50 [14:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:51:28] (03merge) 10sstefanova: openapi: add description to all operations [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/50 [14:56:43] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: envvars-api: bump to 0.0.64-20241220145144-0b848d21 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/647 [15:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:29:12] 06cloud-services-team, 10Data-Services, 10XTools: 500: Internal Server Error - when accessing https://xtools.wmcloud.org/globalcontribs/~aanzx - https://phabricator.wikimedia.org/T382564#10418899 (10Bugreporter) [15:37:43] 06cloud-services-team, 10Data-Services, 10XTools: 500: Internal Server Error - when accessing https://xtools.wmcloud.org/globalcontribs/~aanzx - https://phabricator.wikimedia.org/T382564#10418921 (10Bugreporter) This specific issue will be resolved since T381079#10418299 is completed, but fixing xtools to ig... [15:43:29] 10Tool-translatetagger, 06Indic-TechCom, 10Technical-Tool-Request: Tool to convert Wikitext into translatable wiki-text - https://phabricator.wikimedia.org/T372243#10418927 (10Aafi) We have had some good improvements today, during the first day of Indic MediaWiki Hackathon in Bhubhaneswar. For the sake of t... [16:11:59] 10Cloud-Services, 06Data-Platform-SRE, 10Wikidata-Query-Service: Object storage quota increase request for wikidata-query project - https://phabricator.wikimedia.org/T382601 (10bking) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikim... [16:13:03] 10Cloud-Services, 06Data-Platform-SRE, 10Wikidata-Query-Service: Object storage quota increase request for wikidata-query project - https://phabricator.wikimedia.org/T382601#10418993 (10bking) [16:13:58] 10Cloud-VPS (Quota-requests), 06Data-Platform-SRE, 10Wikidata-Query-Service: Object storage quota increase request for wikidata-query project - https://phabricator.wikimedia.org/T382601#10418994 (10bking) [16:46:47] 10Cloud-VPS (Quota-requests), 06Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service: Object storage quota increase request for wikidata-query project - https://phabricator.wikimedia.org/T382601#10419053 (10rook) +1 [17:09:46] 06cloud-services-team, 06serviceops, 06SRE: Modernise memcached systemd unit / sync, and make it presentable - https://phabricator.wikimedia.org/T273950#10419111 (10fnegri) [17:25:30] 10Cloud-VPS (Quota-requests), 06Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service: Object storage quota increase request for wikidata-query project - https://phabricator.wikimedia.org/T382601#10419160 (10Andrew) a:03Andrew Due to a sad but well-known upstream issue (https://lists.ceph.io/hyperkitty... [17:40:00] 06cloud-services-team, 10Toolforge: Deleting an envvar breaks ReplicaSet driven automatic restarts of a Pod (CreateContainerConfigError) - https://phabricator.wikimedia.org/T365048#10419214 (10bd808) 05Open→03Invalid >>! In T365048#10218850, @bd808 wrote: > I will try to verify my reproduction case lat... [17:40:47] 10Cloud-VPS (Quota-requests), 06Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service: Object storage quota increase request for search project - https://phabricator.wikimedia.org/T382601#10419216 (10bking) [17:42:23] 10Cloud-VPS (Quota-requests), 06Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service: Object storage quota increase request for search project - https://phabricator.wikimedia.org/T382601#10419221 (10bking) [17:42:29] 10Cloud-VPS (Quota-requests), 06Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service: Object storage quota increase request for search project - https://phabricator.wikimedia.org/T382601#10419222 (10bking) Thanks for the update @Andrew . Per IRC conversation, I've changed this request from `wikidata-que... [17:54:15] 10Cloud-VPS (Quota-requests), 06Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service: Object storage quota increase request for search project - https://phabricator.wikimedia.org/T382601#10419239 (10Andrew) 05Open→03Resolved Done (for 'search') [18:03:31] 10Cloud Services Proposals, 06cloud-services-team, 06Data-Platform-SRE: Decision request - running wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607 (10fnegri) 03NEW [18:03:38] 10Cloud Services Proposals, 06cloud-services-team, 06Data-Platform-SRE: Decision request - running wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10419267 (10fnegri) p:05Triage→03Medium [18:04:29] 10Cloud Services Proposals, 06cloud-services-team, 06Data-Platform-SRE: Decision request - who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10419269 (10fnegri) [18:05:15] 10Cloud Services Proposals, 06cloud-services-team, 06Data-Platform-SRE: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10419270 (10fnegri) [18:29:10] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T382609 (10Curb_Safe_Charmer) 03NEW [18:32:04] 10Cloud Services Proposals, 06cloud-services-team, 06Data-Platform-SRE: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10419338 (10fnegri) [18:32:51] 10Cloud Services Proposals, 06cloud-services-team, 06Data-Platform-SRE: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10419339 (10fnegri) [18:34:24] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T382609#10419342 (10Curb_Safe_Charmer) a:03TheresNoTime I've tried restarting the webservice but that hasn't helped. Sammy, are you around to take a look? [20:53:35] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install cloudcephosd2004-dev - https://phabricator.wikimedia.org/T378825#10419601 (10Andrew) I will have another go!