{"id":15,"date":"2016-09-17T23:19:02","date_gmt":"2016-09-17T22:19:02","guid":{"rendered":"http:\/\/www.xciv.org\/blog\/?p=15"},"modified":"2016-10-29T16:09:21","modified_gmt":"2016-10-29T15:09:21","slug":"updating-a-dell-server-bios-or-a-very-long-tale-of-woe","status":"publish","type":"post","link":"https:\/\/www.xciv.org\/blog\/2016\/09\/17\/updating-a-dell-server-bios-or-a-very-long-tale-of-woe\/","title":{"rendered":"Updating a Dell server BIOS – or – a (very) long tale of woe"},"content":{"rendered":"

Introduction<\/strong><\/p>\n

You can read what follows as a rant, or you can read it as helpful information to save other people from wasted time and effort.<\/p>\n

My goal here was to do a quick update of the BIOS on a Dell R210 II server.\u00a0 Quick however, it was anything but.<\/p>\n

I wanted to do this update remotely as it would be quicker than a round trip to the data centre to do probably 20mins work, or so I thought.<\/p>\n

Modern Dell servers incorporate iDRAC web based console, power control and virtual media – this is only in the enterprise iDRAC but well worth the extra investment IMO.\u00a0 They have some clever stuff in the way of virtual media, you can attach a local floppy or ISO image on your laptop and present it as if its a local drive on the server, you can also attach an ISO file over CIFS\/SMB\/NFS.<\/p>\n

Word of note, if you are going to try and attach a large ISO file over broadband with slow upload this will likely not work, it’s just too much data and the server can’t usually cope with the lag of waiting for the data to come across in my experience.\u00a0 Small files work fine, larger files work fine if you’re on ethernet local to the server or I presume have good broadband upload speed.<\/p>\n

With the 11th generation (11G) servers you have approximately three methods of performing BIOS updates, I ended up working through all three in the following order, update via OS, update via DOS image, update through Lifecycle controller and finally succeeded with an update via DOS image.<\/p>\n

OS Update – Linux<\/strong><\/p>\n

The first method of BIOS update is by using an operating system utility on either Windows or Linux, in this case the server is running Debian 6.0\/squeeze so I downloaded the updater binary.\u00a0 The only Linux updater Dell offer says it is for Redhat, with some reservation I executed the binary and was presented with a screen full of ‘set’ errors.\u00a0 Thanks Dell, really helpful.<\/p>\n

I know why it IS this way, Redhat is the commercially supported operating system for Dell.<\/p>\n

What I don’t understand is why it HAS to be this way.<\/p>\n

Most modern Linux distributions have IMO standardised libc, bash etc to a degree so I don’t think it would be too hard to build an updater that would work on say Redhat X, Debian X or newer, Ubuntu X or newer.\u00a0 Having a binary that depends on Redhat when it doesn’t need to, from a technical point of view, is quite restrictive.\u00a0 Thanks Dell, really helpful.<\/p>\n

So we have to move on to another method…<\/p>\n

Repository Manager<\/strong><\/p>\n

I promised myself I would never use Repository Manager again after the last time I had to do battle with it, yet here we are.\u00a0 RM is a utility to build SUU packages that can be updated through the Lifecycle controller, more on this later.<\/p>\n

The first time I used this was a while ago and the particular install was on a Vista VM, so being slightly au fait with it I thought I would just install the latest version, and as laborious as the process is, build the SUU.<\/p>\n

Firstly you need to install .NET 4 and by install I mean the whole package, you can’t just use the client package from Windows update so you’ll need to google and download the installer.\u00a0 Once this was done I installed RM 1.5 and then the fun began.<\/p>\n

To understand how RM works a few things to note..\u00a0 Dell maintain a master catalog on the Dell ftp server, this is like an index of all their hardware products and corresponding drivers and firmware updates.\u00a0 You can build your own local copy (repository) from this by selecting the subset of hardware you have (eg. R200 and R210 servers) and it will download the applicable updates into your local repository.\u00a0 I would suggest selecting as few hardware profiles as possible unless you have lots of time and lots of bandwidth.<\/p>\n

You then select what components you want to update and build an SUU from it.<\/p>\n

Now here’s the thing, to build an SUU you need to install the SUU plug-in, and I say plug-in like it’s some snappy little thing that plugs right in, in reality I’m talking about approximately 300 MB of cab file, yes really.<\/p>\n

My problem was that however I installed 1.5 and the latest 720 cab file, the cab file would NEVER actually install and RM would not by itself cache the download, so yes on each attempt it would re-download that 300 MB file again, and again, until I used the option to download and store it, and load it from the local directory.<\/p>\n

This installation was on XP because that’s all I had in way of a VM, the problem as far as I can see seemed to be something to do with certificate validation, Dell sign part of these updates\/catalog presumably to stop any neferious updates getting in.\u00a0 Only this appeared to be broken in some way, because it kept popping up saying that action was required and I needed to approve the certificate, only the certificate window was entirely empty.\u00a0 I think the problem might be XP, but I don’t know as google didn’t find anyone else experiencing this problem.<\/p>\n

More google research suggested that in the past they have had problems with a revoked certificate in the chain somewhere and the suggestion was to disable revocation checks in IE (yes because IE is so tied into the core OS).\u00a0 This however did not seem to solve the problem for me.<\/p>\n

In the end I solved this by installing RM 1.4 and a 651 cab file.\u00a0 It still seemed very finicky, occasionally saying the plugin was not installed and having to reinstall it, I’m not sure whether it was disabling the certificate revocation check, using an older RM or using an older SUU cab that fixed the problem.\u00a0 But by now, I was tearing my hair out so I didn’t much care.<\/p>\n

Great!\u00a0 So to build an SUU for Lifecycle controller you select the latest bundle of updates for Windows, yes this is slighty odd.. Lifecycle controller is not Windows and there is nothing to indiciate what you actually have to do, but take it from me a Windows bundle is what you need and in fact a Lifecycle controller screen says it will only handle Windows bundles in the latest releases.\u00a0 Some time passes and we are presented with a 1.2 GB data set to up date our BIOS.<\/p>\n

Wuh huh.<\/p>\n

Okay okay I don’t really need to update every bit of firmware, nor all the Windows drivers since we’re on a Debian system.<\/p>\n

So I build a custom bundle containing JUST the BIOS update.\u00a0 We’re now presented with a 400 MB data set.<\/p>\n

What. The. Fuck.\u00a0 400 MB of data to load an 11 MB BIOS data file?\u00a0 Seriously.\u00a0 Are you kidding?<\/p>\n

This would not represent a problem I suppose except that I am trying to do this update remotely, you know, where modern IT systems are designed so that you can do stuff without having to go on site to the data centre.<\/p>\n

Aside, I had a poke around in this data and found the culprit.. there is a java directory using most of that space.\u00a0 Uh.<\/p>\n

Amusingly, during the course of trying to work through all this I downloaded a PDF called “Dell Repository Manager Tutorial” and it was subtitled with the following:<\/p>\n

“A simple and Efficient Way to Manage the Dell Update files”<\/p>\n

Clearly written by someone who has never actually used this product.\u00a0 Or they were making a sneaky joke.<\/p>\n

To be honest this product would be best renamed Suppository Manager because by the time you’ve finished using it you’ll feel like someone has bent you over and…<\/p>\n

Anyhow, the good news is that now we have our SUU we can crack on and get the updates installed..<\/p>\n

Lifecycle controller \/ Dell Unified Server Configurator (USC)<\/strong><\/p>\n

Like that klunky name do you?\u00a0 This is a hint of things to come.<\/p>\n

Now I don’t know why, but generally this part of the system is referred to as Lifecycle controller however it also pops up with the title Unified Server Configurator (is configurator actually even a word?) and seems to be referenced as such in various places so I’m going to take a guess that marketing got hold of this product late in the game and decided that Unified Server Configurator wasn’t snappy enough and didn’t really sell its worth so it got renamed.\u00a0 I’ll further refer to it as USC for the purposes of this post.<\/p>\n

During boot you hit F10 for system services and up pops (eventually, it’s s-l-o-w) USC.<\/p>\n

From here you can do lots of things but we want platform update, and then select our source media.<\/p>\n

I should point out that I had to make the trip to the data centre, because remember how big that update is?\u00a0 400 MB for the BIOS only SUU and 1.2 GB for the full SUU.<\/p>\n

So we kick off the process and we are presented with “Catalog file not authenticated correctly.\u00a0 Do you want to proceed?” Oh well that sounds kind of worrying but we’ll select Yes and continue..<\/p>\n

More time passes..<\/p>\n

“The updates you are trying to apply are not Dell-authorized updates.”<\/p>\n

Greeeeeeat.\u00a0 Is this something to do with those certificate issues?\u00a0 Who knows really, it’s all a bit vague isn’t it?\u00a0 Thanks Dell, really helpful.<\/p>\n

Google provides the answer, older versions of USC will not apply updates, um, why?\u00a0 This is what USC is for surely?\u00a0 You know, the new modern system for applying all your updates, except when it won’t.<\/p>\n

Thankfully Dell provide a way of easily updating the USC firmware, you can do it through iDRAC as if you were updating the iDRAC firmware.<\/p>\n

You simply need to download the Lifecycle Controller Repair Package.<\/p>\n

Hmm, repair package you say, for a firmware?\u00a0 Curious.\u00a0 This is actually another hint of things to come.<\/p>\n

Once this is done we can repeat the procedure, one oddity I note is that now the USC presents the list of components to update and while BIOS is ticked by default it is also greyed out, no indication of what on earth this might mean?\u00a0 With some trepidation I attempt to proceed and it lets me… hurrah, the BIOS starts updating..\u00a0 we’re almost there now!<\/p>\n

And up pops a message “bioswrapper.efl – return code mismatch” or something.\u00a0 I say something because you’d want a pen and paper handy, or your fingers near the screen capture option because this is accompanied by a 10second countdown before the system reboots.<\/p>\n

Yeah because no-one would need to make a note of this pretty critical sounding error message would they?\u00a0 Thanks Dell, really helpful.<\/p>\n

Gah.\u00a0 That’s it, I’m done with USC for this now.<\/p>\n

Before I move on, I’ll throw in some other USC observations.<\/p>\n

Firstly, USC seems to maintain “state” somehow about what you’ve been doing, at least you can have rebooted and the server will pop back into USC without you having hit F10.\u00a0 Why, I don’t know, but I don’t like it, *I* should be deciding when that happens or not.<\/p>\n

I also got the following reassuring screen upon attempting to enter USC “MAS001 partition not found! Unable to launch System Services image. System halted!”\u00a0 I power cycled and have not seen this screen since.\u00a0 Good to know that this flakey thing is not doing anything mission critical like I don’t know, fiddling with my device firmware?<\/p>\n

Remember when I was talking about the USC repair package, I quote:<\/p>\n

“The Unified Server Configurator (USC) repair package restores embedded tools and utilities in the event of a hardware failure or flash memory corruption. Such failures can occur for many reasons, including power loss or interruption of an update process.”<\/p>\n

How many other firmware based tools have you come across that specifically need a repair package?\u00a0 The above makes no sense, firmware doesn’t get corrupted unless you are updating it and interrupt the process.\u00a0 My assertion is that USC is flakey and gets its knickers in a knot often enough that it needs a dedicated generally available repair option.\u00a0 Inspiring, not.<\/p>\n

I’m taking a guess that USC is written in Java considering that it is so slow, the general visual look of it and there were a humungous amount of Java files in the SUU package, perhaps, maybe..<\/p>\n

One final note is that since actually completing the BIOS update (more on that later) the USC will no longer read my existing SUU sources, I tried a CD ROM copy and also the ISO mounted via RFS – the image that had previously been working fine.\u00a0 Um, I have no idea why, I might try to re-upload the USC repair image, I guess?\u00a0 Sigh.<\/p>\n

“There was an internal error while trying to retrieve information on the updates in your system. Perform an A\/C power cycle and retry the operation.”<\/p>\n

The USC trouble shooting resource doesn’t even list this error:
\nhttp:\/\/support.euro.dell.com\/support\/edocs\/software\/smusc\/smlc\/lc_1_5\/usclce\/en\/ug\/html\/faq.htm<\/a><\/p>\n

All in all I’m left feeling that USC or LCC or whatever it’s called is.. klunky, slow, unpredictable and scary.<\/p>\n

Let’s not forget this is a production system, and I’m having to take it off-line at every attempt to complete this update, it’s just not up to par.<\/p>\n

I’m sure it must work for some people, and I know that if you have a large MS deployment it ties into Openmanage and vCenter or something and is possibly quite useful.<\/p>\n

From my experiences I hope that the life of Lifecycle controller is quite short.\u00a0 Alas I fear that Dell have committed to this horrible thing for the medium term.<\/p>\n

DOS<\/strong><\/p>\n

Performing a BIOS update from a DOS image was actually my second choice before doing battle with RM and USC but I just could not get a DOS image to load remotely.\u00a0 Dell actually supply a utility for creating images, unfortunately I can’t remember what this is called.<\/p>\n

It will let you create a floppy image and a hard disk image, so I looked at the 11 MB standalone DOS BIOS updater and knew that a floppy image would be no good, so I elected to create a hard disk image.<\/p>\n

I was presented with a 4 MB image file.\u00a0 Thanks Dell, really helpful.<\/p>\n

In the end I found somewhere a Dell DOS image and I managed to get the BIOS update file into that and on to a memory stick but it did not work, in fact the updater just hung.\u00a0 Not knowing what “DOS” this was based on I reverted to a standard (Windows ME) based DOS image on a USB stick and finally(!!!) the BIOS updated.<\/p>\n

In the process of doing this I had a quick look at the readme from the DOS updater package to see how you invoke the updater, if it needs any command line flags, what you might expect to see on screen or for it to ask you, etc.<\/p>\n

Needless to say it does not really give you any information in this regard.\u00a0 Thanks Dell, really helpful.<\/p>\n

It was at this point that something raised an eyebrow:<\/p>\n

“Note 2: In order to flash to BIOS version 2.2.3 or newer using USC, you must first flash to BIOS version 2.1.2,\u00a0 if you are on an older version of BIOS. Other BIOS update methods do not have this prerequisite.”<\/p>\n

In case the penny hasn’t dropped, I’ll spell it out.<\/p>\n

All my efforts in using RM and USC to update the BIOS have been:\u00a0 A. Complete. Fucking. Waste. Of. Time.<\/p>\n

Given that I was on BIOS 1.3.1 it was technically IMPOSSIBLE to ever have updated to BIOS 2.2.3 through USC.\u00a0 EVER.<\/p>\n

You know USC, that new modern system that is the way forward for managing your updates?\u00a0 Yeah.<\/p>\n

Thanks Dell, really helpful.<\/p>\n

I made a fundamental school boy error here, I just trusted that I could install the latest BIOS update via USC and I hadn’t read the BIOS release notes, well actually I had, but I must’ve read them earlier and somehow not picked up on this critical nugget of information.<\/p>\n

You remember that BIOS component that was greyed out in USC before it failed to actually update the BIOS?\u00a0 I guess now we know why it was greyed out, USC obviously knew there was no way you could apply it.\u00a0 Why not warn you of this or prevent you?\u00a0 Thanks Dell, really helpful.<\/p>\n

Oh, and by the by, the Dell site (at least the route where you enter your asset tag and have it offer you all the relevent BIOS and drivers for that hardware) never did offer me BIOS 2.1.2 at all anyway so I don’t even know where you get that from.\u00a0 Thanks Dell, really helpful.<\/p>\n

Conclusion<\/strong><\/p>\n

The Linux updater method is no use to anyone unless you are running Redhat.<\/p>\n

The Lifecycle controller is no use to anyone.<\/p>\n

The DOS updater is ESSENTIAL if you can find an appropriately sized and compatible DOS image.<\/p>\n

You may be wondering why on earth I didn’t just persue the DOS updater method from the off, you know I am wondering that myself after all this nonsense.<\/p>\n

A couple of things steered me away from this.\u00a0 Firstly I associate DOS images with floppy drives and no-one uses them any more.\u00a0 Secondly DOS itself is antiquated and no-one uses it any more.<\/p>\n

Lifecycle controller is supposed to be the modern way forward for managing your updates, so I leant away from the DOS approach.\u00a0 Clearly if you want relatively quick and hassle free BIOS updates then a DOS USB drive is the way to go.<\/p>\n

A few thoughts…
\nWhy can’t there be a multi distribution capable Linux updater?\u00a0 It’s not *that* hard surely.
\nWhy can’t RM have more intelligence and warn you about updates it can’t apply?
\nWhy can’t RM warn you about Lifecycle firmware versions?
\nWhy can’t Dell offer sensible sized useful DOS images?
\nWhy can’t I just bloody hit an option in the BIOS and get it to read a firmware file off a USB drive?
\nLifecycle controller was developed to reduce the lifecycle of the person using it.<\/p>\n

One amusing though, I did ask a Dell engineer once about BIOS updates and mentioned Lifecycler controller “oh I never use that, just a quick DOS update”…<\/p>\n

I think they may have changed things a little in the 12G servers, but anyhow my final words on this debacle are, as ever..<\/p>\n

Thanks Dell, really helpful.<\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

Introduction You can read what follows as a rant, or you can read it as helpful information to save other people from wasted time and effort. My goal here was to do a quick update of the BIOS on a Dell R210 II server.\u00a0 Quick however, it was anything but. I wanted to do this […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"twitterCardType":"","cardImageID":0,"cardImage":"","cardTitle":"","cardDesc":"","cardImageAlt":"","cardPlayer":"","cardPlayerWidth":0,"cardPlayerHeight":0,"cardPlayerStream":"","cardPlayerCodec":"","footnotes":""},"categories":[7,3],"tags":[4,6],"class_list":["post-15","post","type-post","status-publish","format-standard","hentry","category-computing","category-technology","tag-computing","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.xciv.org\/blog\/wp-json\/wp\/v2\/posts\/15","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.xciv.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.xciv.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.xciv.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.xciv.org\/blog\/wp-json\/wp\/v2\/comments?post=15"}],"version-history":[{"count":5,"href":"https:\/\/www.xciv.org\/blog\/wp-json\/wp\/v2\/posts\/15\/revisions"}],"predecessor-version":[{"id":20,"href":"https:\/\/www.xciv.org\/blog\/wp-json\/wp\/v2\/posts\/15\/revisions\/20"}],"wp:attachment":[{"href":"https:\/\/www.xciv.org\/blog\/wp-json\/wp\/v2\/media?parent=15"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.xciv.org\/blog\/wp-json\/wp\/v2\/categories?post=15"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.xciv.org\/blog\/wp-json\/wp\/v2\/tags?post=15"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}