[00:44:55] (03update) 10don-vip: Draft: Update to Spring Boot 4.0.0 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/14 [01:33:30] (03update) 10don-vip: Draft: Update to Spring Boot 4.0.0 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/14 [04:37:46] 06cloud-services-team, 10PAWS: Paws Server Failed - https://phabricator.wikimedia.org/T410833#11398744 (10Pppery) [04:38:17] 06cloud-services-team, 10PAWS: Paws Server Failed - https://phabricator.wikimedia.org/T410833#11398747 (10Pppery) Can't reproduce. [05:43:12] 06cloud-services-team, 10PAWS: Paws Server Failed - https://phabricator.wikimedia.org/T410833#11398772 (10Sadeiiw67) Thats okay but why it is happening? [07:01:27] 06cloud-services-team, 10PAWS: [bug] - https://phabricator.wikimedia.org/T410712#11398775 (10Sadeiiw67) Hello there, Can you tell me that If your problem is solved or not Till now? Waiting for response......... [07:51:26] 06cloud-services-team, 10PAWS: [bug] - https://phabricator.wikimedia.org/T410712#11398780 (10Anoop) [07:51:29] 06cloud-services-team, 10PAWS: Paws Server Failed - https://phabricator.wikimedia.org/T410833#11398782 (10Anoop) →14Duplicate dup:03T410712 [07:53:04] 06cloud-services-team, 10PAWS: Paws Server Failed - https://phabricator.wikimedia.org/T410833#11398784 (10Anoop) 05Duplicate→03Open p:05Unbreak!→03Triage working for me , merged this task into similar one [07:53:26] 06cloud-services-team, 10PAWS: Paws Server Failed - https://phabricator.wikimedia.org/T410833#11398787 (10Anoop) 05Open→03Resolved a:03Anoop [07:53:57] 06cloud-services-team, 10PAWS: Paws Server Failed - https://phabricator.wikimedia.org/T410833#11398789 (10Anoop) a:05Anoop→03None [07:57:52] 06cloud-services-team, 10PAWS: [bug] - https://phabricator.wikimedia.org/T410712#11398790 (10Anoop) [07:57:53] 06cloud-services-team, 10PAWS: Paws Server Failed - https://phabricator.wikimedia.org/T410833#11398792 (10Anoop) →14Duplicate dup:03T410712 [08:25:37] 10Cloud-Services, 06cloud-services-team, 10PAWS: PAWS: User servers failing to spawn (timeout 30s) across multiple accounts - https://phabricator.wikimedia.org/T410837 (10Sadeiiw67) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimed... [08:27:06] 06cloud-services-team, 10PAWS: PAWS: User servers failing to spawn (timeout 30s) across multiple accounts - https://phabricator.wikimedia.org/T410837#11398812 (10Sadeiiw67) [08:31:35] 06cloud-services-team, 10PAWS, 10Toolforge: [bug] - https://phabricator.wikimedia.org/T410712#11398813 (10Sadeiiw67) p:05Triage→03High [08:36:05] 06cloud-services-team, 10PAWS, 10Toolforge: [bug] PAWS: User servers failing to spawn (timeout 30s) - https://phabricator.wikimedia.org/T410712#11398816 (10Sadeiiw67) [08:37:12] 10PAWS: [bug] PAWS: User servers failing to spawn (timeout 30s) - https://phabricator.wikimedia.org/T410712#11398818 (10JJMC89) [08:38:56] 06cloud-services-team, 10PAWS: PAWS: User servers failing to spawn (timeout 30s) across multiple accounts - https://phabricator.wikimedia.org/T410837#11398819 (10Sadeiiw67) 05Open→03Resolved [08:41:06] 10PAWS: [bug] PAWS: User servers failing to spawn (timeout 30s) - https://phabricator.wikimedia.org/T410712#11398820 (10Sadeiiw67) a:05Lollovand→03fnegri [09:24:02] 10PAWS, 10Toolforge: [bug] PAWS: User servers failing to spawn (timeout 30s) - https://phabricator.wikimedia.org/T410712#11398826 (10Sadeiiw67) p:05High→03Unbreak! [09:40:28] 10PAWS: [bug] PAWS: User servers failing to spawn (timeout 30s) - https://phabricator.wikimedia.org/T410712#11398843 (10Anoop) p:05Unbreak!→03Triage a:05fnegri→03None @Sadeiiw67 , per https://www.mediawiki.org/wiki/Bug_management/Phabricator_etiquette you were already informed not to assign task to someo... [10:39:51] 10PAWS: [bug] PAWS: User servers failing to spawn (timeout 30s) - https://phabricator.wikimedia.org/T410712#11398881 (10Sadeiiw67) For me it's not working what can I do now? @Anoop [13:05:06] (03update) 10don-vip: Draft: Update to Spring Boot 4.0.0 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/14 [13:54:07] (03update) 10don-vip: Update to Spring Boot 4.0.0 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/14 [13:54:16] (03merge) 10don-vip: Update to Spring Boot 4.0.0 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/14 [13:55:46] (03update) 10don-vip: Draft: Add NIH BioArt [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/10 [13:57:44] (03update) 10don-vip: Draft: Update to new NASA Photojournal [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/8 [14:07:12] 10PAWS: [bug] PAWS: User servers failing to spawn (timeout 30s) - https://phabricator.wikimedia.org/T410712#11398912 (10Anoop) You don't have to do anything, please wait for someone who are can handle this request to respond [14:29:08] (03CR) 10Urbanecm: [C:03+2] [tox] Remove comments from tox.ini, where they are no longer expected [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1208405 (owner: 10Urbanecm) [14:29:09] (03CR) 10Urbanecm: [C:03+2] Fix deployment script [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1208404 (owner: 10Urbanecm) [14:29:33] (03Merged) 10jenkins-bot: [tox] Remove comments from tox.ini, where they are no longer expected [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1208405 (owner: 10Urbanecm) [14:29:33] (03Merged) 10jenkins-bot: Fix deployment script [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1208404 (owner: 10Urbanecm) [15:11:48] (03update) 10don-vip: Draft: Add NIH BioArt [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/10 [15:48:43] (03update) 10don-vip: Draft: Add NIH BioArt [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/10 [16:06:48] (03update) 10don-vip: Draft: Add NIH BioArt [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/10 [16:07:02] (03update) 10don-vip: Add NIH BioArt [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/10 [16:22:45] (03update) 10don-vip: Add NIH BioArt [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/10 [16:31:46] (03update) 10don-vip: Add NIH BioArt [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/10 [16:41:20] (03merge) 10don-vip: Add NIH BioArt [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/10 [18:31:09] (03merge) 10legoktm: Fix range error in [A-z] [toolforge-repos/logo-test] - 10https://gitlab.wikimedia.org/toolforge-repos/logo-test/-/merge_requests/3 (owner: 10hartman) [19:08:31] 06cloud-services-team, 10PAWS: [bug] PAWS: User servers failing to spawn (timeout 30s) - https://phabricator.wikimedia.org/T410712#11399059 (10Pppery) (Re-adding tag per T410712#11395036/H449) [19:23:11] 06cloud-services-team: wmcs Trixie kernel reboots - https://phabricator.wikimedia.org/T410846 (10Andrew) 03NEW [19:23:57] 06cloud-services-team, 10Cloud-VPS: wmcs Trixie kernel reboots - https://phabricator.wikimedia.org/T410846#11399074 (10Andrew) [19:27:29] 06cloud-services-team, 10Cloud-VPS: cloudcephosd bios upgrades - https://phabricator.wikimedia.org/T410784#11399075 (10Andrew) [19:28:01] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2005-dev.codfw.wmnet' [19:31:12] PROBLEM - Host cloudcephosd1035 is DOWN: PING CRITICAL - Packet loss = 100% [19:31:29] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt2005-dev.codfw.wmnet' [19:33:14] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt2005-dev.codfw.wmnet}' [19:34:20] ACKNOWLEDGEMENT - SSH on cloudcephosd1035 is CRITICAL: CRITICAL - Socket timeout after 10 seconds Andrew Bogott rebooting for T410784 https://wikitech.wikimedia.org/wiki/SSH/monitoring [19:34:20] ACKNOWLEDGEMENT - Host cloudcephosd1035 is DOWN: PING CRITICAL - Packet loss = 100% Andrew Bogott rebooting for T410784 [19:35:47] FIRING: NodeDown: Node cloudcephosd1035 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1035 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [19:36:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [19:40:40] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt2005-dev.codfw.wmnet}' [19:41:28] RECOVERY - Host cloudcephosd1035 is UP: PING WARNING - Packet loss = 33%, RTA = 0.41 ms [19:46:38] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt2006-dev.codfw.wmnet}' [19:47:07] 06cloud-services-team, 10Cloud-VPS: wmcs Trixie kernel reboots - https://phabricator.wikimedia.org/T410846#11399080 (10Andrew) [19:52:17] RESOLVED: NodeDown: Node cloudcephosd1035 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1035 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [19:53:11] (03open) 10jaredblumer: Add Server Object Rulesets and Corresponding Specs [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/3 [19:55:35] (03update) 10jaredblumer: Add Server Object Rulesets and Corresponding Specs [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/3 [19:57:17] FIRING: [2x] NodeDown: Node cloudcephosd1035 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [19:57:52] (03update) 10jaredblumer: Add Server Object Rulesets and Corresponding Specs [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/3 [19:59:17] (03update) 10jaredblumer: Add Server Object Rulesets and Corresponding Specs [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/3 [20:07:18] andrew@cloudcumin1001 safe_reboot (PID 1296976) is awaiting input [20:08:17] FIRING: NodeDown: Node cloudcephosd1037 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1037 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:13:17] RESOLVED: NodeDown: Node cloudcephosd1037 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1037 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:14:48] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt2006-dev.codfw.wmnet}' [20:19:17] FIRING: [2x] NodeDown: Node cloudcephosd1037 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:21:32] RESOLVED: NodeDown: Node cloudcephosd1037 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1037 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:22:14] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1042.eqiad.wmnet}' [20:25:15] 06cloud-services-team, 10Cloud-VPS: cloudcephosd bios upgrades - https://phabricator.wikimedia.org/T410784#11399094 (10Andrew) [20:35:17] FIRING: [2x] NodeDown: Node cloudcephosd1038 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:40:17] RESOLVED: [2x] NodeDown: Node cloudcephosd1038 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:47:42] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt1042.eqiad.wmnet}' [20:47:48] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1041.eqiad.wmnet}' [20:48:32] FIRING: [2x] NodeDown: Node cloudcephosd1039 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:53:32] RESOLVED: [2x] NodeDown: Node cloudcephosd1039 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:58:17] FIRING: [2x] NodeDown: Node cloudcephosd1040 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:59:44] 06cloud-services-team, 10Cloud-VPS: cloudcephosd bios upgrades - https://phabricator.wikimedia.org/T410784#11399113 (10Andrew) 05Open→03Resolved [21:03:17] RESOLVED: [2x] NodeDown: Node cloudcephosd1040 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [21:04:19] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt1041.eqiad.wmnet}' [21:19:39] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [21:20:20] (03update) 10jaredblumer: Add Server Object Rulesets and Corresponding Specs [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/3 [21:27:45] (03PS2) 10Ladsgroup: Switch off netbox-exports from diffusion [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1208445 (https://phabricator.wikimedia.org/T410624) [21:28:08] (03CR) 10Ladsgroup: "Thanks for the context. I added that as note." [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1208445 (https://phabricator.wikimedia.org/T410624) (owner: 10Ladsgroup) [21:38:55] 10Tool-toolwatch: Migrate tool to gitlab - https://phabricator.wikimedia.org/T410850 (10Reputation22) 03NEW [21:58:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance maps-proxy-6 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [22:32:25] 10Tool-toolwatch, 10GitLab (Project Migration): Migrate tool to gitlab - https://phabricator.wikimedia.org/T410850#11399170 (10A_smart_kitten) [23:38:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [23:43:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors