[00:00:09] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcontrol2010-dev - https://phabricator.wikimedia.org/T393102#10831747 (10Jhancock.wm) [00:10:59] 06cloud-services-team, 10Toolforge: [build-service] remove legacy fagiani/apt 0.2.5 builder from `--use-latest-versions` stack - https://phabricator.wikimedia.org/T394466#10831753 (10bd808) >>! In T394466#10828809, @dcaro wrote: > Thanks @bd808, I've added a preliminar things to test in the parent task, if you... [00:24:22] (03update) 10raymond-ndibe: Draft: [envvars-api] return custom message for invalid EnvvarName [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T360147) [00:31:18] (03update) 10raymond-ndibe: Draft: [envvars-api] return custom message for invalid EnvvarName [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T360147) [00:37:54] (03update) 10raymond-ndibe: Draft: [envvars-api] return custom message for invalid EnvvarName [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T360147) [00:59:07] (03update) 10raymond-ndibe: Draft: [envvars-api] return custom message for invalid EnvvarName [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T360147) [00:59:24] (03update) 10raymond-ndibe: [envvars-api] return custom message for invalid EnvvarName [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T360147) [00:59:29] (03update) 10raymond-ndibe: [envvars-api] return custom message for invalid EnvvarName [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T360147) [00:59:34] (03update) 10raymond-ndibe: [envvars-api] return custom message for invalid EnvvarName [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T360147) [01:46:38] (03open) 10raymond-ndibe: [envvars-cli] print error string and not dict [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/81 (https://phabricator.wikimedia.org/T360147) [01:47:25] (03update) 10raymond-ndibe: [envvars-cli] print error string and not dict [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/81 (https://phabricator.wikimedia.org/T360147) [01:58:46] (03update) 10raymond-ndibe: [envvars-cli] print error string and not dict [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/81 (https://phabricator.wikimedia.org/T360147) [02:03:42] (03update) 10raymond-ndibe: [envvars-cli] print error string and not dict [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/81 (https://phabricator.wikimedia.org/T360147) [05:05:41] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:15:41] RESOLVED: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:35:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.994% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:07:27] 10Tool-yearinreview, 06Indic MediaWiki Developers UG, 06Indic-TechCom: Year in Review tool doesn't work for Wikidata and Mediawiki - https://phabricator.wikimedia.org/T381681#10832137 (10KCVelaga) p:05Triage→03Medium [09:07:36] 10Tool-yearinreview, 06Indic MediaWiki Developers UG, 06Indic-TechCom: Check number of thanks stats in year in review tool - https://phabricator.wikimedia.org/T381413#10832140 (10KCVelaga) p:05Triage→03Medium [09:07:44] 10Tool-yearinreview, 06Indic MediaWiki Developers UG, 06Indic-TechCom: it should be a day instead of edits in yearinreview tool - https://phabricator.wikimedia.org/T377669#10832142 (10KCVelaga) p:05Triage→03Medium [09:35:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.681% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:50:34] RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.93% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [10:17:12] 10Tool-campwiz-nxt: Migration of CampWiz NXT to toolforge - https://phabricator.wikimedia.org/T394515#10832165 (10Nokib_Sarkar) [10:27:26] 10Tool-campwiz-nxt, 06WMF-Legal: Is Using sentry for error monitoring against wikimedia cloud privacy policy? - https://phabricator.wikimedia.org/T394577 (10Nokib_Sarkar) 03NEW As the #WMF-Legal project tag was added to this task, some general information to avoid wrong expectations: Please note that public... [10:33:31] 10Tool-campwiz-nxt, 06WMF-Legal: Is Using sentry for error monitoring against wikimedia cloud privacy policy? - https://phabricator.wikimedia.org/T394577#10832180 (10Nokib_Sarkar) Sent to `answers@wikimedia.org` [10:33:37] 10Tool-campwiz-nxt, 06WMF-Legal: Is Using sentry for error monitoring against wikimedia cloud privacy policy? - https://phabricator.wikimedia.org/T394577#10832183 (10Nokib_Sarkar) 05Open→03Stalled [10:35:54] 10Tool-campwiz-nxt: Migration of CampWiz NXT to toolforge - https://phabricator.wikimedia.org/T394515#10832184 (10Nokib_Sarkar) [10:36:35] 10Tool-campwiz-nxt, 06WMF-Legal: Is Using sentry for error monitoring against wikimedia cloud privacy policy? - https://phabricator.wikimedia.org/T394577#10832189 (10Nokib_Sarkar) [10:36:36] 10Tool-campwiz-nxt: Migration of CampWiz NXT to toolforge - https://phabricator.wikimedia.org/T394515#10832190 (10Nokib_Sarkar) [10:38:02] 10Tool-campwiz-nxt: Migration of CampWiz NXT to toolforge - https://phabricator.wikimedia.org/T394515#10832191 (10Nokib_Sarkar) [11:20:53] 10Cloud-Services, 06WMF-Legal: Is Using sentry for error monitoring against wikimedia cloud privacy policy? - https://phabricator.wikimedia.org/T394577#10832226 (10Peachey88) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/pr... [13:35:55] 06cloud-services-team, 10Toolforge, 06WMF-Legal: Is Using sentry for error monitoring against wikimedia cloud privacy policy? - https://phabricator.wikimedia.org/T394577#10832317 (10Nokib_Sarkar) [15:20:28] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [15:21:39] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [15:25:28] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [17:05:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:15:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:33:55] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcontrol2010-dev - https://phabricator.wikimedia.org/T393102#10832429 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudcontrol2010-dev.codfw.wmnet with OS boo... [18:10:28] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:11:39] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [20:01:56] (03open) 10raymond-ndibe: api.metrics: add deprecation metrics [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/166 (https://phabricator.wikimedia.org/T390137) [20:02:12] (03update) 10raymond-ndibe: api.metrics: add deprecation metrics [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/166 (https://phabricator.wikimedia.org/T390137) [20:35:41] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:45:41] RESOLVED: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:41:24] 10Tool-campwiz-nxt: Migration of CampWiz NXT to toolforge - https://phabricator.wikimedia.org/T394515#10832576 (10Nokib_Sarkar)