Last month we received delivery of a shiny new Dell Optiplex 990 for ratification. It's arrival heralded the start of a process which is vital to ensure new hardware integrates seemlessly into our environment. My side of this process is to give prospective hardware the 'all clear' from the Altiris desktop management point of view. As many Altiris Admins will be undergoing similar processes out there, I thought it might be of interest to let you know how we go about it.
Please note that this is NOT a document which tells you how to incoorporate the Dell Optiplex 990 into your environment! This is just me hanging up our washing so you can see what our process entails. Having said that, you might still find the Optiplex 990 Driver Pack attached to this article useful.
So to begin, the very first thing we check is that the equipment delivered is what we ordered -we really don't want to go through all the following effort to ratify a machine which has been delivered with the wrong internals. Once we're happy we've got the right model, our Altiris hardware approval consists of the following four core tasks;
- Upgrade Imaging Environment
- Upgrade Image Mass Storage Support
- Gather Plug'n'Play Drivers
- Hardware Burn-in & Performance Test
With focus, this process should generally take no more than a day. Often however, as there are many tugs on an Altiris Admin's time this can take a week if it's just an hour here and there. Complications can also occur which extend this ratification process -for example a model might fail to ratify pending a vendor update to resolve issues. Reasons for delay can include,
- Lack of mass-storage support within Linux automation meaning that drivers have to be fudged
- Model BIOS/Driver issues requiring extended vendor correspondance to resolve.These are normally PXE related problems, or blue-screens due to firmware/driver incompatabilities.
- Internal processes to reduce the vendors plethora of upgrade options to a reasonable and supportable subset. The idea is to create a subset of options for users which cover all business use cases for the device.
1. Upgrade Imaging Environment
The first port of call is testing that our imaging environment of choice can see the disks and network card. We use Linux as it's fast to boot and get to the business task of imaging our machines. A quick boot from our Linux automation USB sticks quickly showed we had the neccessary disk support across all of the storage controller modes (ATA, AHCI & IRRT) for the Optiplex 990 so big sighs of relief there. However support for the new Intel ethernet controller was missing.
A quick search of Connect showed that JMLAN had already posted the new Linux e1000e.ko kernel module for the new intel chipsets. So all we had to do was download and build them into our automation USB stick.
With DS6.9 this is very quick.
- Add the kernel module into the Linux drivers folder
Use Windows explorer to locate the Bootwiz folder structure on the eXpress share,
In the above you can see I've highlighted the e1000e kernel module we downloaded from Connect in the folder .\Bootwiz\platforms\Linux\x86\DRIVERS\CUSTOM\lib\modules\_ver_\kernel\opt\bdc
- Regenerate your Linux Automation Files
Open up Bootdisk Creator. We select our target Linux confuration and select Edit Configuration.
As we are using USB sticks, we select from here the option to create files rather than ISOs. This might seem odd as there is a separate USB option, but our process of using the raw files means IT staff can create their automation images without using bootdisk creator. Always remember at this point to take note of the destination folder where these files are about to be written.
- Update the USB Drive
The file we're interested in that's generated by the above process is the updated compressed root filesystem rootfs.gz. So we grab it from the file dump created by bootdisk creator,
And then copy this across to the USB Drive to overwrite the previous root filesystem archive that our SysLinux powered USB sticks use. Incidently, SysLinux is our tool of choice here as it's menu system permits the booting of mutliple automation configurations (both LinPE & WinPE).
The final step is to test that the new USB stick works as expected -a simple image deployment task is the general ratification test here.
2. Update Image Mass-Storage Support
One problem when deploying sysprep'd images to new machines is that you have to have bullt all you mass-storage support into the image at sysprep time. This means that when we deliver our images to new models (hosting new disk controllers) we pretty much find that our current sysprep images blue screen with the infamous STOP 7B error.
At this stage if you don't have a pucker HII solution in place you need to rebuild your images and sysprep them again to add support for the new controller. This can be very time consuming if you manage many images -although you can completely automate the process it generally takes at least a full day per image. It can also be very frustrating when a month after running through all such rebuilds new hardware arrives which forces you to run through the process all over again....
To remove the image rebuild overhead required when adding disk controller support, we use an offline update technique. This reduces that day's worth of work to just a few minutes. This offline process consists of a script which mounts our image and injects the appropriate controller data into the image. This is documented in the Connect download Offline Updating of Mass-Storage Controllers in XP.
Before we can inject the controller data into the image we first need to establish exactly which controller our hardware hosts and download the appropriate drivers. To get all the data we need here we run an automation data dump. This is a script which utilises the dmesg buffer, kernel module data and pci device listing to summarise the hardware.
The output for the Optiplex with the mass-storage contoller configured in AHCI mode is as follows,
Hardware
========
Model: 0VNP2H
Product Name: OptiPlex 990
Manufacturer: Dell Inc.
Serial Number: *******
Computer Name: DELL-990
Environment
===========
Linux 782bcb91ed2d 2.6.27.7 #1 SMP Wed Mar 10 19:04:29 MST 2010 i686 unknown unknown GNU/Linux
ALTIRIS_VER=DS6.9
ALTIRIS_BDC=BDC6.9.9269.402
ALTIRIS_PROCESSOR=x86
Mass-Storage Controller Data
============================
00:1f.2 0106: 8086:1c02 (rev 04)
00:1f.2 SATA controller: Intel Corporation Cougar Point 6 port SATA AHCI Controller (rev 04)
Loaded Kernel Modules
=====================
filename: /lib/modules/2.6.27.7/kernel/opt/bdc/ata_piix.ko
version: 2.12-ILA2010b
srcversion: 8FD45116A15E1BB5ED5AE1E
vermagic: 2.6.27.7 SMP mod_unload 686
filename: /lib/modules/2.6.27.7/kernel/drivers/ata/libata.ko
version: 3.00
vermagic: 2.6.27.7 SMP mod_unload 686
srcversion: DD763F10D771C0C4236FE9C
filename: /lib/modules/2.6.27.7/kernel/opt/bdc/ahci.ko
version: 3.0-ILA2010a
srcversion: 065DA0B7348EFA85E082AFC
vermagic: 2.6.27.7 SMP mod_unload 686
This tells us we have an Intel Corporation Cougar Point SATA controller which has device ID of 1C02. With the Intel mass-storage controllers, disk drivers can be found in both the chipset download and in the separate Intel Rapid Restore Technology (IRRT) driver download. As IRRT RAID implemention is layered on top of AHCI, it is here we find our required mass-storage support.
The steps for the Optiplex 990 were therefore as follows,
- Download the IRRT drivers from Dell
As a picture tells a thousand words.....
- Extract the Driver Files
Take note of where the drive files are located. Here we've diverted them to the temp folder on my machine on the C:\ Drive as this is a quick'n'dirty dumping ground
- Search INF files for the Controller Device ID
The XP puppy is brilliant for this. Just search the folder for any INF file containing the text 1C02. This should quickly tell us if we have support for our controller. In this case the iaAHCI.inf file comes up trumps.
- Extract Device Lines from INF File
Now we open up the file in notepad, and find the device line for this controller by searching for the string 1C02 in the file. Copy this out and keep it in a safe place.
%PCI\VEN_8086&DEV_1C02&CC_0106.DeviceDesc% = iaStor_Install, PCI\VEN_8086&DEV_1C02&CC_0106
- Update Sysprep.inf
In order to ensure that future images natively contain support for this controller, we next update the driver store on the server and our generic sysprep file by adding the appropriate device line to the [SysprepMassStorage] section. Below you can see we've added a new entry for the 990 using the format above. This entry references the iaahci.inf in c:\sysprep\MassStorage\ICH folder -this is copied from the server to the client before sysprep runs.
[SysprepMassStorage]
; Dell Optiplex 990
PCI\VEN_8086&DEV_1C02&CC_0106=C:\sysprep\MassStorage\ICH\iaahci.inf
; Dell D630C Latitude
PCI\VEN_8086&DEV_2829&CC_0106=C:\sysprep\MassStorage\ICH\iaahci.inf
; HP Eliteboot 6930c - Intel ICH9 Mobile SATA AHCI Controller
; Lenovo R400 - Intel ICH9 Mobile SATA AHCI Controller
; Dell Latitude E4300- Intel ICH9 Mobile SATA AHCI Controller
PCI\VEN_8086&DEV_2929&CC_0106=C:\sysprep\MassStorage\ICH\iaahci.inf
; Dell 760 & HP dc7900 - Intel(R) ICH10 SATA AHCI Controller
PCI\VEN_8086&DEV_3A02&CC_0106=C:\sysprep\MassStorage\ICH\iaahci.inf
- Perform Offline Updates on Existing Images
Now we've done the tricky bits, it's time to roll-in this new controller support into our existing image real estate. For this we use our offline mass-storage driver update script. The steps are simple. First we modify the script header to include the drive update details by setting the following variables,
1. Img_Path : the folder to the image (note that write access is required!)
2. Img_File : the image file name
3. Volume_Name : the volume index in the image to be mounted (typically the first volume is the system volume)
4. Driver : the path to the updated storage controller driver
5. Vendor: the vendor ID for the driver
6. Device: the device ID for the driver
7. Service: the service name used by the storage driver
Once these have been entered, the top of the mass-storage injection script looks like this for the Optiplex 990 SATA injection,
-
REM ###############################################################
REM # #
REM # --------- Start of user-editable Section --------- #
REM # #
REM ###############################################################
SET Img_Path=S:\Windows\XP\2009r5
SET Img_File=Sysprep-XP.img
SET Volume_Name=1
SET Image_Explorer="C:\Program Files\Altiris\Express\Deployment Server\ImgExpl.exe"
SET Driver=S:\Deploy\XP\sysprep\MassStorage\ICH\iaStor.sys
SET Vendor=8086
SET Device=1C02
SET Service=iaStor
Now we're ready to run it. This typically takes just a few seconds,
From here, it's just a matter of repeating this process on the other images which require the Dell Optiplex 990 support.
Now it's time to test an image deployment and see if the mini-setup begins. If it does, that means that the mass-storage injection has been successful, and we can move forward to the next stage of gathering the Plug'n'Play drivers.
3. Gather Plug'n'Play Device Drivers
This is the most tedious stage of the hardware ratification process. In order to get product support from our vendors, we have to ensure driver isolation. That is to say, our Hardware Independent Imaging process must target hardware with drivers as supplied by the vendor for that model. Our answer to this problem was to build a driver structure where we name the driver folder after each hardware's product name.
Each of the driver folders follow a specific structure which we store an example of in the Template folder,
Within each of the above driver folders, the drivers are immediately present so that when mini-setup tests the OEMPnPDriversPath string (provided in the Sysprep file) it immediately finds the drivers it needs (subfolder navigation is not supported in XP mini-setup). Below is an extract from our sysyprep.inf which leads to the above folder structure.
[Unattended]
OemSkipEula=Yes
InstallFilesPath=C:\sysprep\i386
OEMPnPDriversPath = "Drivers\HECI;Drivers\CARDREADER;Drivers\SOL;Drivers\SOLinf;Drivers\LAN;Drivers\LAN2;Drivers\Chipset;Drivers\Chipset2;Drivers\VGA;Drivers\VGA2;Drivers\Audio;Drivers\Audio2;Drivers\TPM;Drivers\modem;Drivers\modem2;Drivers\bluetooth;Drivers\Monitor;Drivers\Monitor2;Drivers\MISC;Drivers\MISC2;Drivers\MISC3;Drivers\MISC4;Drivers\CARDREADER"
DriverSigningPolicy = "Ignore"
UpdateInstalledDrivers=yes
In summary the PnP Drivers process is as follows,
- Downloading all the drivers from the vendor site
- Extract the drivers to folders
- Manually test that the drivers work on your freshly built mass-storage injected machine
- Dump the drivers into the driver library into the appropriate folders
- Test
Once we've tested that our images can be deployed with all drivers in place (a device manager check sorts this out) then you can be happy that everything is good to proceed to the hardware burn-in stage. Incidentally, we also have a HII verification script which runs after deploying our managed images. This emails us if we've delivered an image to a machine which results in a device being left without a driver. If this happens after a model has been ratififed, it's normally due to a hardware revision in the model... a situation which can usually be resolved in a few minutes with an update to the driver library.
The result of this driver scavenge is attached to this article.
4. Hardware Burn-in and Performance Test
This is the rewarding part of the testing. Here we use PassMark PerformanceTest and BurnInTest to,
- Perform a benchmark of the system
- Perform a minimum of a 5 hour hardware burn-in
The resulting reports provide us with that warm confidence that the system is stable, and will perform well for the end user. For example, the Burn-in test automatically generated this certificate for us,
And the Performance Test summary is,
Although these numbers can seem a little daunting, such records can prove most useful later when determining how well a particular model is performing. For example, is the model performing as expected for it's specification or is the performance degraded. Without such benchmarking of it's initial performance you can be left a little stuck.
Summary
What I've tried to illustrate today is the main steps we take when ratifying our desktop and laptop hardware for deployment with Altiris. Throughout this process I should also say that we document progress on a helpdesk ticket. As we move through each phase of our testing, we update this ticket to keep our purchasing team informed as to our progress.
The focus for our hardware approval process has been to form a standard which can be adhered to for integrating hardware in an auditable fashion into our hardware library. There are some technical painpoints however,
- Ensuring Linux Automation functions correctly for the new hardware
- Tracking down and unpacking the vendor drivers for HII integration
Linux drivers can be a pain. In some cases drivers are simply not available from the vendor which means you have to modify existing drivers at source yourself and compile (see How To Fudge Linux AHCI & ATA Drivers for ICH10 Controllers) . I have toyed seriously with using WinPE to deploy our images in the hope that this painpoint would be replaced with a less taxing WinPE variant. WinPE however has a frustrating boot time (especially over PXE) and that frustration keeps me coming back to Linux.
The second problem of tracking down the vendor drivers would also be eased by lessening our requirement for driver isolation. I'm not keen on this though, as a single driver update could potentially affect many models in unpredictable ways. Our main vendor also requires that only their drivers are delivered when imaging. Consequently, for us to obtain support this process remains a necessary evil.
And remember how I said that some machines might be delayed in their ratification pending a vendor update? Well it turns out that if you have a keyboard with an in-built USB hub, these Optiplex 990's can't see them in Power-On Self-Test phase of boot. Thanks to our master tester/breaker here on the team Dell are looking into it...
I think that concludes hope this has been helpful to those starting out or even reviewing their own Altiris ratification processes.
Kind Regards,
Ian./