Introduction
You can read what follows as a rant, or you can read it as helpful information to save other people from wasted time and effort.
My goal here was to do a quick update of the BIOS on a Dell R210 II server. Quick however, it was anything but.
I wanted to do this update remotely as it would be quicker than a round trip to the data centre to do probably 20mins work, or so I thought.
Modern Dell servers incorporate iDRAC web based console, power control and virtual media – this is only in the enterprise iDRAC but well worth the extra investment IMO. They have some clever stuff in the way of virtual media, you can attach a local floppy or ISO image on your laptop and present it as if its a local drive on the server, you can also attach an ISO file over CIFS/SMB/NFS.
Word of note, if you are going to try and attach a large ISO file over broadband with slow upload this will likely not work, it’s just too much data and the server can’t usually cope with the lag of waiting for the data to come across in my experience. Small files work fine, larger files work fine if you’re on ethernet local to the server or I presume have good broadband upload speed.
With the 11th generation (11G) servers you have approximately three methods of performing BIOS updates, I ended up working through all three in the following order, update via OS, update via DOS image, update through Lifecycle controller and finally succeeded with an update via DOS image.
OS Update – Linux
The first method of BIOS update is by using an operating system utility on either Windows or Linux, in this case the server is running Debian 6.0/squeeze so I downloaded the updater binary. The only Linux updater Dell offer says it is for Redhat, with some reservation I executed the binary and was presented with a screen full of ‘set’ errors. Thanks Dell, really helpful.
I know why it IS this way, Redhat is the commercially supported operating system for Dell.
What I don’t understand is why it HAS to be this way.
Most modern Linux distributions have IMO standardised libc, bash etc to a degree so I don’t think it would be too hard to build an updater that would work on say Redhat X, Debian X or newer, Ubuntu X or newer. Having a binary that depends on Redhat when it doesn’t need to, from a technical point of view, is quite restrictive. Thanks Dell, really helpful.
So we have to move on to another method…
Repository Manager
I promised myself I would never use Repository Manager again after the last time I had to do battle with it, yet here we are. RM is a utility to build SUU packages that can be updated through the Lifecycle controller, more on this later.
The first time I used this was a while ago and the particular install was on a Vista VM, so being slightly au fait with it I thought I would just install the latest version, and as laborious as the process is, build the SUU.
Firstly you need to install .NET 4 and by install I mean the whole package, you can’t just use the client package from Windows update so you’ll need to google and download the installer. Once this was done I installed RM 1.5 and then the fun began.
To understand how RM works a few things to note.. Dell maintain a master catalog on the Dell ftp server, this is like an index of all their hardware products and corresponding drivers and firmware updates. You can build your own local copy (repository) from this by selecting the subset of hardware you have (eg. R200 and R210 servers) and it will download the applicable updates into your local repository. I would suggest selecting as few hardware profiles as possible unless you have lots of time and lots of bandwidth.
You then select what components you want to update and build an SUU from it.
Now here’s the thing, to build an SUU you need to install the SUU plug-in, and I say plug-in like it’s some snappy little thing that plugs right in, in reality I’m talking about approximately 300 MB of cab file, yes really.
My problem was that however I installed 1.5 and the latest 720 cab file, the cab file would NEVER actually install and RM would not by itself cache the download, so yes on each attempt it would re-download that 300 MB file again, and again, until I used the option to download and store it, and load it from the local directory.
This installation was on XP because that’s all I had in way of a VM, the problem as far as I can see seemed to be something to do with certificate validation, Dell sign part of these updates/catalog presumably to stop any neferious updates getting in. Only this appeared to be broken in some way, because it kept popping up saying that action was required and I needed to approve the certificate, only the certificate window was entirely empty. I think the problem might be XP, but I don’t know as google didn’t find anyone else experiencing this problem.
More google research suggested that in the past they have had problems with a revoked certificate in the chain somewhere and the suggestion was to disable revocation checks in IE (yes because IE is so tied into the core OS). This however did not seem to solve the problem for me.
In the end I solved this by installing RM 1.4 and a 651 cab file. It still seemed very finicky, occasionally saying the plugin was not installed and having to reinstall it, I’m not sure whether it was disabling the certificate revocation check, using an older RM or using an older SUU cab that fixed the problem. But by now, I was tearing my hair out so I didn’t much care.
Great! So to build an SUU for Lifecycle controller you select the latest bundle of updates for Windows, yes this is slighty odd.. Lifecycle controller is not Windows and there is nothing to indiciate what you actually have to do, but take it from me a Windows bundle is what you need and in fact a Lifecycle controller screen says it will only handle Windows bundles in the latest releases. Some time passes and we are presented with a 1.2 GB data set to up date our BIOS.
Wuh huh.
Okay okay I don’t really need to update every bit of firmware, nor all the Windows drivers since we’re on a Debian system.
So I build a custom bundle containing JUST the BIOS update. We’re now presented with a 400 MB data set.
What. The. Fuck. 400 MB of data to load an 11 MB BIOS data file? Seriously. Are you kidding?
This would not represent a problem I suppose except that I am trying to do this update remotely, you know, where modern IT systems are designed so that you can do stuff without having to go on site to the data centre.
Aside, I had a poke around in this data and found the culprit.. there is a java directory using most of that space. Uh.
Amusingly, during the course of trying to work through all this I downloaded a PDF called “Dell Repository Manager Tutorial” and it was subtitled with the following:
“A simple and Efficient Way to Manage the Dell Update files”
Clearly written by someone who has never actually used this product. Or they were making a sneaky joke.
To be honest this product would be best renamed Suppository Manager because by the time you’ve finished using it you’ll feel like someone has bent you over and…
Anyhow, the good news is that now we have our SUU we can crack on and get the updates installed..
Lifecycle controller / Dell Unified Server Configurator (USC)
Like that klunky name do you? This is a hint of things to come.
Now I don’t know why, but generally this part of the system is referred to as Lifecycle controller however it also pops up with the title Unified Server Configurator (is configurator actually even a word?) and seems to be referenced as such in various places so I’m going to take a guess that marketing got hold of this product late in the game and decided that Unified Server Configurator wasn’t snappy enough and didn’t really sell its worth so it got renamed. I’ll further refer to it as USC for the purposes of this post.
During boot you hit F10 for system services and up pops (eventually, it’s s-l-o-w) USC.
From here you can do lots of things but we want platform update, and then select our source media.
I should point out that I had to make the trip to the data centre, because remember how big that update is? 400 MB for the BIOS only SUU and 1.2 GB for the full SUU.
So we kick off the process and we are presented with “Catalog file not authenticated correctly. Do you want to proceed?” Oh well that sounds kind of worrying but we’ll select Yes and continue..
More time passes..
“The updates you are trying to apply are not Dell-authorized updates.”
Greeeeeeat. Is this something to do with those certificate issues? Who knows really, it’s all a bit vague isn’t it? Thanks Dell, really helpful.
Google provides the answer, older versions of USC will not apply updates, um, why? This is what USC is for surely? You know, the new modern system for applying all your updates, except when it won’t.
Thankfully Dell provide a way of easily updating the USC firmware, you can do it through iDRAC as if you were updating the iDRAC firmware.
You simply need to download the Lifecycle Controller Repair Package.
Hmm, repair package you say, for a firmware? Curious. This is actually another hint of things to come.
Once this is done we can repeat the procedure, one oddity I note is that now the USC presents the list of components to update and while BIOS is ticked by default it is also greyed out, no indication of what on earth this might mean? With some trepidation I attempt to proceed and it lets me… hurrah, the BIOS starts updating.. we’re almost there now!
And up pops a message “bioswrapper.efl – return code mismatch” or something. I say something because you’d want a pen and paper handy, or your fingers near the screen capture option because this is accompanied by a 10second countdown before the system reboots.
Yeah because no-one would need to make a note of this pretty critical sounding error message would they? Thanks Dell, really helpful.
Gah. That’s it, I’m done with USC for this now.
Before I move on, I’ll throw in some other USC observations.
Firstly, USC seems to maintain “state” somehow about what you’ve been doing, at least you can have rebooted and the server will pop back into USC without you having hit F10. Why, I don’t know, but I don’t like it, *I* should be deciding when that happens or not.
I also got the following reassuring screen upon attempting to enter USC “MAS001 partition not found! Unable to launch System Services image. System halted!” I power cycled and have not seen this screen since. Good to know that this flakey thing is not doing anything mission critical like I don’t know, fiddling with my device firmware?
Remember when I was talking about the USC repair package, I quote:
“The Unified Server Configurator (USC) repair package restores embedded tools and utilities in the event of a hardware failure or flash memory corruption. Such failures can occur for many reasons, including power loss or interruption of an update process.”
How many other firmware based tools have you come across that specifically need a repair package? The above makes no sense, firmware doesn’t get corrupted unless you are updating it and interrupt the process. My assertion is that USC is flakey and gets its knickers in a knot often enough that it needs a dedicated generally available repair option. Inspiring, not.
I’m taking a guess that USC is written in Java considering that it is so slow, the general visual look of it and there were a humungous amount of Java files in the SUU package, perhaps, maybe..
One final note is that since actually completing the BIOS update (more on that later) the USC will no longer read my existing SUU sources, I tried a CD ROM copy and also the ISO mounted via RFS – the image that had previously been working fine. Um, I have no idea why, I might try to re-upload the USC repair image, I guess? Sigh.
“There was an internal error while trying to retrieve information on the updates in your system. Perform an A/C power cycle and retry the operation.”
The USC trouble shooting resource doesn’t even list this error:
http://support.euro.dell.com/support/edocs/software/smusc/smlc/lc_1_5/usclce/en/ug/html/faq.htm
All in all I’m left feeling that USC or LCC or whatever it’s called is.. klunky, slow, unpredictable and scary.
Let’s not forget this is a production system, and I’m having to take it off-line at every attempt to complete this update, it’s just not up to par.
I’m sure it must work for some people, and I know that if you have a large MS deployment it ties into Openmanage and vCenter or something and is possibly quite useful.
From my experiences I hope that the life of Lifecycle controller is quite short. Alas I fear that Dell have committed to this horrible thing for the medium term.
DOS
Performing a BIOS update from a DOS image was actually my second choice before doing battle with RM and USC but I just could not get a DOS image to load remotely. Dell actually supply a utility for creating images, unfortunately I can’t remember what this is called.
It will let you create a floppy image and a hard disk image, so I looked at the 11 MB standalone DOS BIOS updater and knew that a floppy image would be no good, so I elected to create a hard disk image.
I was presented with a 4 MB image file. Thanks Dell, really helpful.
In the end I found somewhere a Dell DOS image and I managed to get the BIOS update file into that and on to a memory stick but it did not work, in fact the updater just hung. Not knowing what “DOS” this was based on I reverted to a standard (Windows ME) based DOS image on a USB stick and finally(!!!) the BIOS updated.
In the process of doing this I had a quick look at the readme from the DOS updater package to see how you invoke the updater, if it needs any command line flags, what you might expect to see on screen or for it to ask you, etc.
Needless to say it does not really give you any information in this regard. Thanks Dell, really helpful.
It was at this point that something raised an eyebrow:
“Note 2: In order to flash to BIOS version 2.2.3 or newer using USC, you must first flash to BIOS version 2.1.2, if you are on an older version of BIOS. Other BIOS update methods do not have this prerequisite.”
In case the penny hasn’t dropped, I’ll spell it out.
All my efforts in using RM and USC to update the BIOS have been: A. Complete. Fucking. Waste. Of. Time.
Given that I was on BIOS 1.3.1 it was technically IMPOSSIBLE to ever have updated to BIOS 2.2.3 through USC. EVER.
You know USC, that new modern system that is the way forward for managing your updates? Yeah.
Thanks Dell, really helpful.
I made a fundamental school boy error here, I just trusted that I could install the latest BIOS update via USC and I hadn’t read the BIOS release notes, well actually I had, but I must’ve read them earlier and somehow not picked up on this critical nugget of information.
You remember that BIOS component that was greyed out in USC before it failed to actually update the BIOS? I guess now we know why it was greyed out, USC obviously knew there was no way you could apply it. Why not warn you of this or prevent you? Thanks Dell, really helpful.
Oh, and by the by, the Dell site (at least the route where you enter your asset tag and have it offer you all the relevent BIOS and drivers for that hardware) never did offer me BIOS 2.1.2 at all anyway so I don’t even know where you get that from. Thanks Dell, really helpful.
Conclusion
The Linux updater method is no use to anyone unless you are running Redhat.
The Lifecycle controller is no use to anyone.
The DOS updater is ESSENTIAL if you can find an appropriately sized and compatible DOS image.
You may be wondering why on earth I didn’t just persue the DOS updater method from the off, you know I am wondering that myself after all this nonsense.
A couple of things steered me away from this. Firstly I associate DOS images with floppy drives and no-one uses them any more. Secondly DOS itself is antiquated and no-one uses it any more.
Lifecycle controller is supposed to be the modern way forward for managing your updates, so I leant away from the DOS approach. Clearly if you want relatively quick and hassle free BIOS updates then a DOS USB drive is the way to go.
A few thoughts…
Why can’t there be a multi distribution capable Linux updater? It’s not *that* hard surely.
Why can’t RM have more intelligence and warn you about updates it can’t apply?
Why can’t RM warn you about Lifecycle firmware versions?
Why can’t Dell offer sensible sized useful DOS images?
Why can’t I just bloody hit an option in the BIOS and get it to read a firmware file off a USB drive?
Lifecycle controller was developed to reduce the lifecycle of the person using it.
One amusing though, I did ask a Dell engineer once about BIOS updates and mentioned Lifecycler controller “oh I never use that, just a quick DOS update”…
I think they may have changed things a little in the 12G servers, but anyhow my final words on this debacle are, as ever..
Thanks Dell, really helpful.
Leave a Reply