[12:45:57] We got a mail from someone at ftr.com asking us to reset a BGP session. Did someone networky notice it amidst the sea of spam? :) [15:44:45] !log cdepooling db1248 to clone a new db [15:44:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:09:56] Emperor or elukey have you seen this error when provisioning ms-be nodes? [16:10:03] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resour [16:10:05] ce Statement, Duplicate declaration: Exec[mountpoint-root-/srv/swift-storage/objects0] is already declared at (file: /srv/puppet_code/environments/production/modules/swift/manifests/mount_filesystem.pp [16:10:07] , line: 21); cannot redeclare (file: /srv/puppet_code/environments/production/modules/swift/manifests/mount_filesystem.pp, line: 21) (file: /srv/puppet_code/environments/production/modules/swift/manife [16:10:09] sts/mount_filesystem.pp, line: 21, column: 5) (file: /srv/puppet_code/environments/production/modules/profile/manifests/swift/storage/configure_disks.pp, line: 53) on node ms-be2088.codfw.wmnet [16:10:27] I can dig in and figure out the cause, but wanted to ask first [16:19:37] jhathaway: o/ I can't recall a duplicate declaration like that, but it sound as if the regular number of partitions weren't created properly [16:20:27] nod thanks elukey, probably some artifact of the re-image [16:25:31] elukey: fyi the issue was having the virtual media connected, the code blew up on the attached usb cdrom drive [16:26:40] nice :D [16:35:07] I've seen issues like that with the virutal media on Dell's before. It can re-arrange the order of hardware as the OS sees it [16:35:51] the virtual hardware stays there even after you've disconnected from the console, gotta make sure to do "disconnect virutal-media" after you use it [16:36:15] nightmare to troubleshoot any time I hit it, nice find Jesse! [16:43:47] thanks [21:08:12] I have a virtual machine, doc2002.codfw.wmnet, that says it's on Ganeti cluster codfw and I can connect to it just fine. Then I am told that ganeti2033.codfw.wmnet is the master node for "this cluster" but a gnt-instance list says "Instance not found". [21:08:27] is there now a separate master for each row? [21:09:59] yea, I see a couple VMs there but not mine. Trying to figure out how to determine the right master per row.. [21:12:41] I did try netbox, but that for example tells me about ganeti2011.codfw.wmnet/ ganeti2012 as servers in row C.. but then they are offline [21:15:17] ok, I found it. gaenti2032. but it was more trying random ones until 2040 told me to go to 2032 :) [22:49:02] you can log into any ganeti node, the motd prints out the current master [23:06:29] yea, that's how I found 2032. 2040 told me to go to 2032. but before that 2034 had told me to go to 2033 and that's a different cluster even though both are codfw. [23:16:42] no alerts during entire on-call week. logging off.