Ghost Solution Suite

I've created an image of server running VMware ESX 4.1 using standalone Ghost executable. Since Ghost does not understand VMware partitions (in particular, it can not process VMFS), I expected to get sector-by sector copy and used "Image Disk" option. The image was successfully created, but now I can't do anything with it - when I try to start restore or even check operation on that image, Ghost crashes in few seconds with error 36000.

The image was created using DOS boot disk and Symantec Ghost 11.0.2. For check/restore, I've tried versions 11.0.2 and most recent 11.5.1 (purchased an upgrade), DOS-based ghost.exe and ghost32.exe running under Windows Server 2003. Also attempted to use "-wa" command line option, all to no success - it always crashes. One more note: image file is quite big, slightly over 1TB.

I've opened a case at Symantec support and soon had a call, but the only outcome of the conversation was a statement that "ESX is not supported" and a link to this article: http://www.symantec.com/business/support/index?page=content&id=TECH110404 (The situation described in the article is very different from mine - I do not run Ghost from virtual machine, I do not use Console and do not use network, but the guy from support said that sentece about lack of ESX support still applies)

I understand that Ghost "has not been tested", but in sector-by-sector mode I expect it to succeed irregardless of what OS/filesystems are on disk partitions.

Any ideas? That image is not deadly important for me (fortunately, I had another backup made differently), but it is still desirable to extract some data from it. One more thing I will try - restore individual partitions from the image, but I'm rather skeptical.

Another user has reported that PCDOS falls over when it reaches around 26Gb image size.  I think you are being unreasonably optimistic in expecting PCDOS to image a 1Tb volume when it has no concept of disks with block counts exceeding that which can be handled by a 32 bit field.

You need to think about using WinPE as your boot environment and run Ghost from there. This article can get you started : https://www-secure.symantec.com/connect/articles/readyadventures-winpe or you can use the WinPE boot image creation tools in 11.5.1.2266.  By creating an ISO image (as you would then use to burn a CD) you can mount this ISO in VMWare and boot off it. This should give you access to the virtual disk and enable a backup to be made to a USB device which is connected to the VM.

Equally, by defining a VM with a suitably large hard disk, then booting WinPE, you should be able to reload an image back into your VMWare environment and then boot from it, if you first prepare the virtual hard disk using Diskpart, etc.

Thanks for your comments, but:

I think you are being unreasonably optimistic in expecting PCDOS to image a 1Tb volume when it has no concept of disks with block counts exceeding that which can be handled by a 32 bit field.

32 bit field is sufficient to address 2TB disk if block is 512-byte sector. Image was created, after all. If it is corrupted, I expect Ghost to report a corruption warning when working later with the image instead of crashing. That's what we have "Check image" command for, right?

By creating an ISO image (as you would then use to burn a CD) you can mount this ISO in VMWare and boot off it. This should give you access to the virtual disk and enable a backup to be made to a USB device which is connected to the VM.

Equally, by defining a VM with a suitably large hard disk, then booting WinPE, you should be able to reload an image back into your VMWare environment and then boot from it, if you first prepare the virtual hard disk using Diskpart, etc.

I wasn't mounting ISO under VMware virtual machine, and I don't need it (I did it successfully in past for other tasks). I booted _physical_ machine from Ghost boot CD and wanted to backup whatever is on it's disk. The image does not contain a disk of single VM - it contains entire disk of ESX host toghether with some special ESX partitions, VMFS partition containing several VMs, etc.

And, as I said, I already tried to do restore by starting Ghost from W2K3 which should not be worse than WinPE. It didn't help either.

Sorry, I presented the 32 bit argument incorrectly. The limitations of the ATA interface under PCDOS (as there is no SATA support) are as follows:

At most 65536 cylinders (numbered 0-65535), 16 heads (numbered 0-15), 255 sectors/track (numbered 1-255), for a maximum total capacity of 267386880 sectors (of 512 bytes each), that is, 136902082560 bytes (137 GB).  This limit also applied to the original release of XP, and it was not fixed until SP1 when additional extensions to INT13 were released. I recall at the time updating a Promise controller from 32 bit to 48 bit addressing in order to support a 200Gb disk. SATA now uses 48 bit addressing natively.

The "Check Image" command has some limitations which Nigel Bree has covered in the past.

At most 65536 cylinders (numbered 0-65535), 16 heads (numbered 0-15), 255 sectors/track (numbered 1-255), for a maximum total capacity of 267386880 sectors (of 512 bytes each), that is, 136902082560 bytes (137 GB).

These may be limitations of (PC)DOS, but I'm not sure how Gost accesses disks under DOS - via DOS calls or via BIOS. If it direclty calls INT 13h Extensions functions (why not?), then since there are 8 bytes for sector number it can access as much as 2^64 sectors = 2^73 bytes with 512-byte sectors. That's REALLY much :-)

See: http://en.wikipedia.org/wiki/INT_13H#INT_13h_AH.3D42h:_Extended_Read_Sectors_From_Drive

This limit also applied to the original release of XP, and it was not fixed until SP1 when additional extensions to INT13 were released.

INT 13 functions are provided by system BIOS and/or add-on disk controller card ROMs. XP SPs can not change it. And why XP (and 2000 and even NT) should depend on INT13 or it's extensions (except OS loader time)? Windows drivers communicate directly with disk controller via ports/IRQs/DMA, they don't use BIOS. I guess 137GB was a limitation of how XP used ATA protocol, but it wasn't INT 13 problem as such.

http://www.symantec.com/docs/TECH139493
 

Yep, I know it can't handle VMFS - that's why I was using sector-by-sector copy which should be irrelevant to file systems being copied. It should be an uneffective but reliable way.

Ok, I've tried restoring separate partitions from the image instead of a whole disk and, to my great surprise, it worked. I've successfully extracted VMFS partition (the most important one) onto a free disk, mounted it under ESX and copied my data. So image wasn't corrupted.

Looks like it is partition structure what kills Ghost. On this machine I have two identical RAID volumes constructed of similar disks. One volume contains ESX partitions, another one is emty. I start Ghost, select "Partition" -> "From image", select my image file and partition from it. Then Ghost shows list of available target disks, including these two RAID volumes. If I select volume which already contains ESX, Ghost crashes with error 36000, it doesn't even show me list of partitions on target disk. If I select clean volume, then it proceeds and successfully extracts source partition from the image.

That's all with Ghost32.exe 11.5.1.

Just a quick check - are you up to build 2266 via LiveUpdate?

Also, a search on 36000 threw up this KB article as one of 69 hits:  http://www.symantec.com/business/support/index?page=content&id=TECH106950&locale=en_US

Just to remind you Ed, any case which causes a 36000 error *will* be due to a fault in the code, so these always have to go to the dev team to get fixed. That's one of the catch-all codes derived from the x86 processor exception, usually a null pointer dereference or something similar, and from a development perspective any time it happens there's a code fault - while it's sometimes possible from a specific fault to devise a workaround, the full solution *always* involves a code change to detect the error properly.

So, the first port of call for any report like that is to just go to support. It's quite possible that this has already been reported by someone else - customers with ESX tend to be the kind of folks who buy support contracts, and so if it's been escalated that way there could be a fix in hand already.

The problem there is that for customers who don't have maintenance, getting problems escalated from L1 support to L2 and then getting L2 to escalate to development is a crapshoot; even when it happens, it can take three months or so to actually get through the layers of bureacracy and then you'd never know what it would be escalated as. Numerous times, problems reported here on the forums that I'd pick up and just get fixed immediately would appear as escalations some 2-3 months later, and they'd be almost unrecognizable - we'd only know *what* the case was actually referring to because I'd talked to the customer directly, because the escalation would be some kind of strange word salad.

In fact, it became a kind of a game for us when we'd fix things based on forum reports, seeing just *how* divergent the eventual escalation would be if one ever came through. One of the most common sources of mangling would be some completely unrelated aspect of the customer's environment morphing into the chief complaint, via the mechanism Raymond Chen notes:

While it's probably true that you don't burn a support request just for idle curiosity, I've seen support requests turn into idle curiosity. Once the customers got the answer to their question, they realized that since they had already burned a support request, they may as well pile on with follow-up questions that are just idle curiosity. That was not the case here, because the customer led with the question

Nigel - thanks for that - I'd missed the fact that 36000 was always a code fault. I will bear that in mind for the future.

Hello Nigel,

you are not the only IT company who has that kind of problems with customer support :-) I once tried to contact Adaptec - that was a nightmare.

Returning to the case - yes, I have active support contract (purchased together with GSS upgrade). As I said, I already has contacted support, but it didn't get any further "it was not tested / ESX is not supported". Little bit funny, but I had to explain to support guy that ESX is a kind of OS itself and that it does not run under any other OS. Additional complexity in talking to support arises from the fact that I'm not a native English-speaking person, and while I'm able to sufficiently good communicate via reading and writing, it's much harder for me to understand somebody speaking English, especially by phone. So e-mail or forum posts are much better.

I don't know, are you interested in obtaining my Ghost dump files? I can provide them, and can reliably reproduce the crash.

Yes, I'm on 11.5.1.2266 (but not via LiveUpdate, I downloaded it from licensing.symantec.com).

I don't know, are you interested in obtaining my Ghost dump files? I can provide them, and can reliably reproduce the crash.

If I still worked there, I'd have been all over this getting you to e-mail me from the get-go. Now there is only a small dev/QA team in India who take the most serious cases escalated from L2; they are fine folks, by the way, doing a demanding job in difficult circumstances, but the rules of engagement they operate under are pretty restrictive and their funding even more limited than the (frankly laughable) budgets we were given. So they really can't just initiate work on an issue until they can justify it by having it referred to them - the folks who Get Things Done need at least some kind of political cover, especially after a few years when Symantec has been utterly ruthless in jettisoning any engineering talent that poked its head over the parapets. Being seen to be Following The Process(tm) is unfortunately part of playing the game now :-(

Since you have a maintenance contract, I'd suggest the best result should come if you push harder against your frontline rep to get him to escalate this. The fact that it's ESX shouldn't matter; a 36000 is a code fault, and it needs to be fixed or a workaround devised. Something else you can use is that VMDK support is an advertised feature, even though VMWare were playing some games and keeping important details of the ESX variant of the VMDK format undocumented (something we didn't know about when 2.5 was released, but...).

[ Some more ammunition (and to be fair to VMWare): the other senior engineer in the team kept on for awhile after the GSS shutdown - the former lead on the cloning engine - was working on better ESX support during mid-2010 in cooperation with folks at VMware. That was not long before he and I were laid off, so that work probably isn't in a released build, but it's entirely possible that a fix to your issue is one of the various things related to Ghost that is sitting fallow in the source archives. ]

If you *still* don't get any satisfaction, you're welcome to e-mail me and bring me into the conversation with your support rep, and I'll try and do what I can to assist and facilitate.

Additional complexity in talking to support arises from the fact that I'm not a native English-speaking person, and while I'm able to sufficiently good communicate via reading and writing, it's much harder for me to understand somebody speaking English, especially by phone. So e-mail or forum posts are much better.

I understand completely; and whatever other faults it may have, Symantec right from top to bottom generally deals *extremely* well with both internationalization and access issues. No matter where you are in the world, your accent or language or primary culture, that's something Symantec can and generally will happily cope with. You may just need to prompt your support rep to work with you via e-mail instead of via telephone; that should be no problem at all, once you make your front-line contact aware of your preferences - you just have to make that preference known explictly, and Symantec has the diversity and global depth to cope.

Ghost was always developed outside North America and we always did most of our real business by e-mail with our global customers (and the folks in the support organization in North America). Symantec doesn't just talk a good game about diversity and culture, as many North American firms do while actually being very culturally chauvinistic; it's actually something that has been a part of the company culture for a long time. The support organization and escalation does happen in North America but the folks at the top of Ghost's support there are great, you just need to get past the front-line to the folks who are real product specialists.