[00:32:29] 10Tools: zoomviewer uses an unreasonable amount of disk space - https://phabricator.wikimedia.org/T395020#10894205 (10tstarling) >>! In T395020#10891317, @taavi wrote: > The `/data/project/zoomviewer` directory has grown by about a terabyte in the two weeks since I filed this task. Is the cleanup script working... [04:06:19] FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [04:16:19] RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [06:06:46] 10Quarry: Quarry down - web service unreachable - https://phabricator.wikimedia.org/T395201#10894364 (10Liz) 05Resolved→03Open It's happening again! The exact same thing is happening again. [06:10:29] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [06:10:39] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [06:14:57] 10Quarry: [bug] Quarry down? - https://phabricator.wikimedia.org/T396295#10894368 (10SD0001) →14Duplicate dup:03T395201 [06:14:58] 10Quarry: Quarry down - web service unreachable - https://phabricator.wikimedia.org/T395201#10894370 (10SD0001) [07:50:28] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:50:39] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [09:34:28] 10Cloud-VPS (Quota-requests): Pixel project "disk40" flavor, and perhaps a few more cores? - https://phabricator.wikimedia.org/T395837#10894776 (10fnegri) Documentation about Cinder volumes is here: https://wikitech.wikimedia.org/wiki/Help:Adding_disk_space_to_Cloud_VPS_instances [09:53:21] 10Toolforge (Quota-requests): Elasticsearch credential request for sdzerobot - https://phabricator.wikimedia.org/T396237#10894857 (10taavi) +1 [09:55:49] 10Toolforge (Quota-requests): Elasticsearch credential request for sdzerobot - https://phabricator.wikimedia.org/T396237#10894861 (10fnegri) 05Open→03In progress a:03fnegri [10:07:16] 10Toolforge (Quota-requests): Elasticsearch credential request for sdzerobot - https://phabricator.wikimedia.org/T396237#10894871 (10fnegri) 05In progress→03Resolved @SD0001 the credentials are available in your tool in the env vars TOOL_ELASTICSEARCH_USER and TOOL_ELASTICSEARCH_PASSWORD. [10:08:50] 10Toolforge (Quota-requests): Elasticsearch credential request for sdzerobot - https://phabricator.wikimedia.org/T396237#10894874 (10SD0001) Thanks! [10:49:03] 06cloud-services-team, 10Cloud-VPS, 10Beta-Cluster-Infrastructure: Set up an https://github.com/knyar/phalerts instance in metricsinfra (@wmcs-alerts Phab account) - https://phabricator.wikimedia.org/T394446#10894976 (10Aklapper) [11:10:30] (03CR) 10Eugene233: [C:03+2] Restructure README file [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1139001 (https://phabricator.wikimedia.org/T392670) (owner: 10Eugene233) [11:10:37] (03CR) 10Eugene233: [V:03+2 C:03+2] Restructure README file [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1139001 (https://phabricator.wikimedia.org/T392670) (owner: 10Eugene233) [11:46:28] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [11:56:28] RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [12:00:30] 06cloud-services-team, 10Striker: Rotate StrikerBot GitLab PAT before it expires on 2025-07-29 - https://phabricator.wikimedia.org/T395694#10895090 (10taavi) >>! In T395694#10872450, @bd808 wrote: > The password may be in pwstore? I definitely have it in my personal password manager as I authed with it earlier... [12:03:43] (03CR) 10FNegri: [C:03+1] vps: Drop support for .wmflabs hostnames [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1137738 (owner: 10Majavah) [12:05:33] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10895105 (10fnegri) 05Open→03In progress [12:09:33] (03CR) 10Majavah: [C:03+2] vps: Drop support for .wmflabs hostnames [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1137738 (owner: 10Majavah) [12:13:13] (03Merged) 10jenkins-bot: vps: Drop support for .wmflabs hostnames [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1137738 (owner: 10Majavah) [12:23:50] (03PS1) 10Majavah: toolforge: Add cookbook to mirror Loki-related images [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1154802 (https://phabricator.wikimedia.org/T386480) [12:27:37] 06cloud-services-team, 10Toolforge, 10Pywikibot, 06SRE: pywikibot.org landing page is not updated - https://phabricator.wikimedia.org/T396338#10895144 (10Xqt) [12:27:37] (03CR) 10CI reject: [V:04-1] toolforge: Add cookbook to mirror Loki-related images [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1154802 (https://phabricator.wikimedia.org/T386480) (owner: 10Majavah) [12:28:11] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10895147 (10fnegri) [12:29:16] 06cloud-services-team, 10Toolforge, 10Pywikibot, 06SRE: pywikibot.org landing page is not updated - https://phabricator.wikimedia.org/T396338#10895148 (10Xqt) [12:33:52] (03update) 10taavi: logging: Add basic rate limiting and retention config [repos/cloud/toolforge/toolforge-deploy] (main-If8f503514316703ce91f966fb6ad40b04ef8fdd0) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/807 (https://phabricator.wikimedia.org/T386480) [12:33:56] (03PS2) 10Majavah: toolforge: Add cookbook to mirror Loki-related images [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1154802 (https://phabricator.wikimedia.org/T386480) [12:33:58] (03update) 10taavi: logging: Add basic rate limiting and retention config [repos/cloud/toolforge/toolforge-deploy] (main-If8f503514316703ce91f966fb6ad40b04ef8fdd0) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/807 (https://phabricator.wikimedia.org/T386480) [12:34:01] (03update) 10taavi: logging: Init component [repos/cloud/toolforge/toolforge-deploy] (main-Icb012f1ad81b582b65a569bb493095e12d3fbd72) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/796 (https://phabricator.wikimedia.org/T386480) [12:34:51] 06cloud-services-team, 10Toolforge, 10Pywikibot: pywikibot.org landing page is not updated - https://phabricator.wikimedia.org/T396338#10895151 (10taavi) [12:37:12] (03CR) 10CI reject: [V:04-1] toolforge: Add cookbook to mirror Loki-related images [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1154802 (https://phabricator.wikimedia.org/T386480) (owner: 10Majavah) [12:38:07] 06cloud-services-team, 10Toolforge, 10Pywikibot: pywikibot.org landing page is not updated - https://phabricator.wikimedia.org/T396338#10895193 (10taavi) How is that command being run? [12:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:38:42] (03PS3) 10Majavah: toolforge: Add cookbook to mirror Loki-related images [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1154802 (https://phabricator.wikimedia.org/T386480) [12:44:35] 06cloud-services-team, 10Toolforge, 10Pywikibot: pywikibot.org landing page is not updated - https://phabricator.wikimedia.org/T396338#10895223 (10Xqt) There is a crontab: ` --- - name: pre-nightly command: ./pre_nightly image: bookworm schedule: "0 23,11 * * *" # mem: 1Gi emails: onfailure - name:... [13:14:43] 06cloud-services-team, 10Toolforge, 10Pywikibot: pywikibot.org landing page is not updated - https://phabricator.wikimedia.org/T396338#10895267 (10taavi) I re-started the nightly job manually with: `lang=shell-session tools.pywikibot@tools-bastion-12:~$ toolforge jobs restart nightly ` It completed successfu... [13:21:51] 06cloud-services-team, 10Toolforge, 10Pywikibot: pywikibot.org landing page is not updated - https://phabricator.wikimedia.org/T396338#10895379 (10Xqt) I had removed the error logs because they were very huge. There was another problem with deleting the `public_html.old` folder. @taavi: could you start a sec... [13:41:27] 06cloud-services-team, 10Cloud-VPS, 10Data-Engineering (Q4 2025 April 1st - June 30th), 07IPv6: Add new WMCS IP ranges to analytics - https://phabricator.wikimedia.org/T392468#10895435 (10taavi) May I ask what's the status of this? The task itself is open and the checkboxes in the description are still mar... [13:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:48:45] (03PS2) 10NkwadaNora: rearrange the location of some files [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1152117 [13:49:36] 06cloud-services-team, 10Toolforge, 10Pywikibot: pywikibot.org landing page is not updated - https://phabricator.wikimedia.org/T396338#10895455 (10Xqt) [13:52:42] 06cloud-services-team, 10Toolforge, 10Pywikibot: pywikibot.org landing page is not updated - https://phabricator.wikimedia.org/T396338#10895463 (10taavi) That part seems like a duplicate of {T375279}? [13:55:34] 06cloud-services-team, 10Toolforge, 10Pywikibot: pywikibot.org landing page is not updated - https://phabricator.wikimedia.org/T396338#10895472 (10Xqt) >>! In T396338#10895463, @taavi wrote: > That part seems like a duplicate of {T375279}? Ah, yes! Didn't find it before. [14:21:14] (03merge) 10chuckonwumelu: Adding warning message for beta [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/78 (https://phabricator.wikimedia.org/T394277) [14:23:45] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.112-20250609142128-f12ef86d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/805 (https://phabricator.wikimedia.org/T394273 https://phabricator.wikimedia.org/T394277) [14:23:50] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.112-20250609142128-f12ef86d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/805 (https://phabricator.wikimedia.org/T394273 https://phabricator.wikimedia.org/T394277) [14:31:59] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10895604 (10fnegri) [14:33:43] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcephosd200[567] - https://phabricator.wikimedia.org/T393614#10895609 (10Jhancock.wm) [14:35:53] (03PS1) 10Eugene233: Add CI config for WdTmCollab Tool [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1154830 [14:35:53] (03PS1) 10Eugene233: Add support for ci tasks on tool [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1154831 (https://phabricator.wikimedia.org/T396357) [14:36:42] 06cloud-services-team, 10Toolforge: toolsbeta paging - https://phabricator.wikimedia.org/T396038#10895638 (10fnegri) p:05Triage→03High [14:37:48] (03CR) 10Eugene233: [C:03+2] Add support for ci tasks on tool [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1154831 (https://phabricator.wikimedia.org/T396357) (owner: 10Eugene233) [14:37:50] (03CR) 10Eugene233: [V:03+2 C:03+2] Add support for ci tasks on tool [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1154831 (https://phabricator.wikimedia.org/T396357) (owner: 10Eugene233) [14:38:47] (03CR) 10Eugene233: [C:03+2] Add CI config for WdTmCollab Tool [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1154830 (owner: 10Eugene233) [14:38:49] (03CR) 10Eugene233: [V:03+2 C:03+2] Add CI config for WdTmCollab Tool [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1154830 (owner: 10Eugene233) [14:41:43] 06cloud-services-team, 10Toolforge, 10Pywikibot: pywikibot.org landing page is not updated - https://phabricator.wikimedia.org/T396338#10895698 (10Xqt) →14Duplicate dup:03T375279 [14:41:47] 06cloud-services-team, 10Toolforge, 10Pywikibot: unexpected root-owned files in /data/project/pywikibot/public_html - https://phabricator.wikimedia.org/T375279#10895700 (10Xqt) [14:56:50] !log chuckonwumelu@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [15:03:02] (03CR) 10Eugene233: "recheck" [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1152117 (owner: 10NkwadaNora) [15:03:14] (03CR) 10Jforrester: "recheck" [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1154831 (https://phabricator.wikimedia.org/T396357) (owner: 10Eugene233) [15:03:18] (03CR) 10CI reject: [V:04-1] rearrange the location of some files [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1152117 (owner: 10NkwadaNora) [15:03:29] (03CR) 10CI reject: [V:04-1] Add support for ci tasks on tool [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1154831 (https://phabricator.wikimedia.org/T396357) (owner: 10Eugene233) [15:13:08] !log chuckonwumelu@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [15:13:14] 06cloud-services-team, 10Toolforge: [toolsbeta,tofu,infra] There's some discrepancy between the volumes in toolsbeta and tofu - https://phabricator.wikimedia.org/T396276#10895907 (10taavi) > toolsbeta-prometheus-a volume is not in tofu https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/blob... [15:19:53] (03merge) 10bd808: gitlab: Ignore Gerritlab artifacts in target branch [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/54 (owner: 10taavi) [15:23:55] (03reopen) 10bd808: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 (https://phabricator.wikimedia.org/T90594) [15:24:14] (03close) 10bd808: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 (https://phabricator.wikimedia.org/T90594) [15:25:52] 06cloud-services-team, 10Phabricator, 07Developer Productivity, 10Release-Engineering-Team (Seen), 07SecTeam-Processed: Some very specific Maniphest search queries by RelEng, Sec Team and WMCS are global and shown for all users - https://phabricator.wikimedia.org/T214579#10895948 (10sbassett) >>! In T214... [15:26:49] 10Wikibugs, 13Patch-For-Review: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#10895966 (10bd808) test [15:32:29] (03CR) 10BryanDavis: [C:04-2] "test" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [15:52:49] 06cloud-services-team, 10Phabricator, 07Developer Productivity, 10Release-Engineering-Team (Seen), 07SecTeam-Processed: Some very specific Maniphest search queries by RelEng, Sec Team and WMCS are global and shown for all users - https://phabricator.wikimedia.org/T214579#10896100 (10bd808) I know I made... [15:54:09] 06cloud-services-team, 10Phabricator, 07Developer Productivity, 10Release-Engineering-Team (Seen), 07SecTeam-Processed: Some very specific Maniphest search queries by RelEng, Sec Team and WMCS are global and shown for all users - https://phabricator.wikimedia.org/T214579#10896101 (10bd808) >>! In T214579... [16:03:46] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10896236 (10fnegri) clouddb1015 is upgraded and repooled! I will continue with clouddb1019 tomorrow. [16:09:34] !log chuckonwumelu@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [16:13:16] !log chuckonwumelu@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [17:15:14] 10Wikibugs: Wikibugs not reporting Phabricator activity to #wikimedia-zuul as hoped - https://phabricator.wikimedia.org/T396387 (10bd808) 03NEW [17:32:50] (03approved) 10bd808: Don’t show tool name in element [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/13 (owner: 10lucaswerkmeister) [18:06:49] 06cloud-services-team, 10decommission-hardware: decommission cloudcontrol2004-dev.codfw.wmnet - https://phabricator.wikimedia.org/T396396 (10Andrew) 03NEW [18:07:00] 06cloud-services-team, 10decommission-hardware: decommission cloudcontrol2004-dev.codfw.wmnet - https://phabricator.wikimedia.org/T396396#10896748 (10Andrew) a:03Andrew [18:22:56] 10Tools: ipcheck 504 Gateway Time-out - https://phabricator.wikimedia.org/T387947#10896835 (10MusikAnimal) 05Open→03Resolved [18:59:26] 06cloud-services-team, 10decommission-hardware, 13Patch-For-Review: decommission cloudcontrol2004-dev.codfw.wmnet - https://phabricator.wikimedia.org/T396396#10896985 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by andrew@cumin1002 for hosts: `cloudcontrol2004-dev.codfw.wmnet` - cloudcont... [19:03:50] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [19:03:55] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [19:05:00] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [19:05:02] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [19:05:35] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [19:05:38] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [19:05:48] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [19:05:53] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [19:06:57] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [19:06:59] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [19:07:38] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [19:07:40] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [19:07:43] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [19:07:45] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [19:08:41] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [19:08:45] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [19:08:51] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [19:08:56] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [19:09:02] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,keystone [19:09:12] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,keystone [19:09:16] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate [19:09:19] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,designate [19:09:39] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate [19:11:16] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,designate [19:13:09] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate [19:13:10] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment codfw1dev for service: project,designate [19:13:14] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [19:15:57] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [19:20:21] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [19:23:29] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [19:39:16] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [19:42:13] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [20:09:52] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [20:10:03] 06cloud-services-team, 10Data-Services, 10MediaWiki-extensions-SecurePoll, 06Trust and Safety Product Team: Add Extension:SecurePoll SQL tables to wiki replicas - https://phabricator.wikimedia.org/T396408 (10Novem_Linguae) 03NEW [20:11:52] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [20:21:01] (03update) 10lucaswerkmeister: Use toolforge envvars if available [toolforge-repos/python-toolforge] - 10https://gitlab.wikimedia.org/toolforge-repos/python-toolforge/-/merge_requests/27 (https://phabricator.wikimedia.org/T339940) [20:22:42] 06cloud-services-team, 10Data-Services, 10MediaWiki-extensions-SecurePoll, 06Trust and Safety Product Team: Add Extension:SecurePoll SQL tables to wiki replicas - https://phabricator.wikimedia.org/T396408#10897271 (10A_smart_kitten) Potentially a duplicate of {T381197}? [20:26:47] 06cloud-services-team, 10Data-Services, 10MediaWiki-extensions-SecurePoll, 06Trust and Safety Product Team: Add Extension:SecurePoll SQL tables to wiki replicas - https://phabricator.wikimedia.org/T396408#10897303 (10Novem_Linguae) →14Duplicate dup:03T381197 [20:26:51] 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 06Privacy Engineering: Create views for SecurePoll db tables in Toolforge replicas - https://phabricator.wikimedia.org/T381197#10897305 (10Novem_Linguae) [22:06:23] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-codfw: decommission cloudcontrol2004-dev.codfw.wmnet - https://phabricator.wikimedia.org/T396396#10897568 (10Andrew) a:05Andrew→03None