What not to do when you upgrade PureDisk
How it started
I've been delaying this PureDisk upgrade from 6.2 to 18.104.22.168.
There were simply many other tasks at my hands and also the upgrade did not simply look easy and it made me hesitate.
What if it goes wrong?
I thought "It will take too much time to create a full backup of the content router storage!" (well, later I found I did not have to do full content router backup anyway)
The most scray thing is that it's actually an OS upgrade and an application upgrade at the same time.
Both PDOS and PureDisk had to be upgraded together, on 3 nodes storage pool - SPA and MBE and CR.
I was telling myself, later I will do it, later I will do it ... and that "later" finally came.
Some other project was adding 3 more nodes (MBE CR CR) in to this storage pool and I had to upgrade existing pool before I add those 3 in.
What goes wrong comes wrong
Although my company has all the right systems to prevent people from making silly mistakes, awkward moments do occur.
I've seen it many times, and I never imagined I'd have one of such moments myself too.
It all started with a simple and serious mistake - I copied PDOS CD contents to /cdrom directory instead of loading the physical CD on CD ROM drive.
I thought, "What's the matter? Physical CD or copied on HDD, they are files and directories, same permission anyway, copied using tar command syntax."
Well, of course that's wrong!
It's an OS CD, used to boot the system!
I must have been too much ... uh ... in a relaxed mode to say, to make such a simple mistake mindlessly.
So that's what I did wrong, which went out.
What came back wrong then?
The system did have a PDOS CD loaded on CD drive - yes an older 6.1 version one...!!!
Do the right thing
I did not even notice that I actually "downgraded" the PDOS until I was "finished" with the (what I think was an) OS upgrade on SPA.
Seeing no error messages from the "upgrade which was in fact a downgrade" on SPA, I moved on to MBE to do OS upgrade.
MBE did not have CD loaded on the drive, only the CD contents I copied to /cdrom directory was there.
Naturally, when I rebooted, it did not come up.
I consoled in, then I noticed something was wrong.
And there came the tide of "what have I done to the SPA node then ?!?!?!" kind of feeling.
Keep this in mind
I managed to put the right CD in, and did upgrade (which was in fact an upgrade this time) and somehow it completed successfully.
But I learned a lesson.
It's not enough to just RTFM - if you read the manual, then do it properly as it says.
What's the use of RingTFM if you just throw it away after?
1. Take full backup of system when you make a change - especially when you upgrade OS!
2. These days too many things are virtualized and remote-accessible. Double/Triple check everything before you start.
3. Better to do it as a team, not alone. That'll reduce risk of making silly mistakes.
4. You are a genius, yes, but geniuses usually make more mistakes than others. So don't mindlessly do things - always be cautious.
I learned a lesson.
How long will it last?
I don't know.
But what I know for sure is,
next time I'll be more careful.