swft
Needs a life
Full throttle!
Posts: One MEEEEEELLION
|
posted October 01, 2002 05:47 PM
What a day...
Customer experienced a triple-double disk raid failure. That's three double disk raid failures, each one capable of rendering the entire volume of data unavailable. It would be the equivalent of being shot, stabbed and beheaded, all at the same time! Got onsite and evaluated the error messages. Shot up the bigass batman flare and engaged our bigbrain sustaining engineering folks. Brought in new drives, and stripped the PCB controllers off them, and replaced the old drive controllers, then put everything back together and was able to access the formerly shot,stabbed and beheaded drives. Engineering wrote me a custom kernel to do a bit copy of a single drive (pretty good considering you are talking about a single drive in a raid 4 array) and we used it to read the data off the formerly dead drives and copy it to the new drives. That let us get the three raid groups up to a degraded state, and start rebuilding them from parity. After that, we ran the equivalent of fsck, which will be complete in about 30 hours. The filesystem is a 6TB bouncing baby boy, with about 30 million customer's datafiles on it, so it's just a LITTLE bit important. 
  
|
slug
Pro
Out in search of my mind...
Posts: 1433
|
posted October 01, 2002 07:14 PM
double disk raid?
is that like mirror? or something else?
striping?
sorry, never heard of that level of raid or that type ;P
   
|
swft
Needs a life
Full throttle!
Posts: One MEEEEEELLION
|
posted October 01, 2002 07:51 PM
Actually, it's RAID 4 - data striped across multiple disks, with a seperate parity drive. You can lose a single drive, but two is a no-no.
|
Megabyte
Pro
Posts: 1047
|
posted October 01, 2002 08:11 PM
Two years ago we upgraded our NOS and our hardware. We run 10 disk RaiD 5 arrays w/2 hot spares on 35 Netware 5.1 Servers, Backup everything daily, and haven't lost a system yet. I've had drives fail, but they are hot swappable so I just plug in a new drive and send the bad one in for repair. I monitor everything with Compaq's Insight Manager, which immediately alerts me if there is a problem, only way to go....
____________
We First make our habits and then our habits make us.
|
DaveInDaytona
Pro
Posts: 1696
|
posted October 01, 2002 08:11 PM
Each entire block is written onto a data disk. Parity for same rank blocks is generated on Writes, recorded on the parity disk and checked on Reads.
It's all magic, and sometimes a magic trick goes wrong.
____________
DaytonaSportbikes Forum
|
ZHooligan
Moderator
Post Whore Extraordinaire!
Posts: 3829
|
posted October 01, 2002 09:37 PM
Very technical and very boring!!!!!!!!!!!!!!
____________
To those who do not count their life in years, but in how life
has touched them in the past and how much it can hold in the
future; -- Youth is forever.
|
swft
Needs a life
Full throttle!
Posts: One MEEEEEELLION
|
posted October 02, 2002 06:45 AM
 Edited By: swft on 2 Oct 2002 07:46
What I am recommending to the customer is synchronous mirroring with cluster failover. The heads are seperated by up to 500meters. Each head has it's own file system and a synchronous copy of it's partner's file system. If you lose a datacenter, the partner picks up and serves data for the down head as well as itself.
|
dougmeyer
Needs a job
moderated
Posts: 2713
|
posted October 02, 2002 06:52 AM
Yeah, I agree. But how many CC's is it??
____________
It's not that I think you're dumb, it's just that so much of what you know isn't true....
|
kawachan
Pro
Posts: 1031
|
posted October 02, 2002 07:00 AM
Dude, I'm lost?? I can understand upgrading the NOS, but then......!!!!!
____________
RED NINJAS RULE!!
|
ZHooligan
Moderator
Post Whore Extraordinaire!
Posts: 3829
|
posted October 02, 2002 08:21 AM
Wouldn't it be be faster if you doubled the voltage, increased the amperage by a factor of 6 to the ninth power and then sprinkled it with a light mist of saltwater?
____________
To those who do not count their life in years, but in how life
has touched them in the past and how much it can hold in the
future; -- Youth is forever.
|
your car is slow
Needs a job
Fuck Nitrous...Got Boost?
Posts: 4089
|
posted October 02, 2002 08:23 AM
NOS...muhahaha...whats your bottle pressure?
____________
Do not taunt happy fun ball!
|
12RPilot
Pro
Posts: 1094
|
posted October 02, 2002 08:44 AM
Uh...I have a backup USB hub. Pretty cool, huh?
____________
If you aren't an AMA member, you're part of the problem.
NESBA #209
http://www.bikepics.com/members/12rpilot/04zx10r/
|
your car is slow
Needs a job
Fuck Nitrous...Got Boost?
Posts: 4089
|
posted October 02, 2002 09:18 AM
Way cool..You win an AOL Setup Disk. 
____________
Do not taunt happy fun ball!
|
Rubber Pants
Zone Head
Posts: 798
|
posted October 02, 2002 09:36 AM
Get a Mac!! It's the Kaw of computers (Rock Solid) Non Crashable?!?!? Fast etc. ............I should know I have 3 as well as a bunch of PC's!
|
jonwright
Needs a job
Posts: 2416
|
posted October 02, 2002 11:28 AM
Dood: Get a Symmetrix, mirror the volumes, run BCV's then SRDF the BCV volumes off site. Cluster to the SRDF volumes off site. Easy.
'course, I am a little biased seems as how I work for EMC.
j
|
DaveInDaytona
Pro
Posts: 1696
|
posted October 02, 2002 12:30 PM
EMC ? We resell your gear, got the tour of your place up by Boston a few weeks ago. Good time was had by all.
Beware of Sharks.
____________
DaytonaSportbikes Forum
|
vozizm
Needs a job
Got Nothing Witty To Say
Posts: 4417
|
posted October 02, 2002 03:08 PM
which EMC JONWRIGHT... i'm a contrator at the apex plant in NC
|
Megabyte
Pro
Posts: 1047
|
posted October 02, 2002 03:24 PM
How much band width do you have between the heads?
quote:
What I am recommending to the customer is synchronous mirroring with cluster failover. The heads are seperated by up to 500meters. Each head has it's own file system and a synchronous copy of it's partner's file system. If you lose a datacenter, the partner picks up and serves data for the down head as well as itself.
____________
We First make our habits and then our habits make us.
|
swft
Needs a life
Full throttle!
Posts: One MEEEEEELLION
|
posted October 02, 2002 04:32 PM
Data doesn't flow between the heads, just heartbeat and write log activity. Since each head has access to the other head's disks (fiber channel drives are dual channel, dontchaknow) the heads don't have to exchange data to keep track of the other's filesystem. The only data sent between the two is uncommitted write activity, which is stored in NVRAM.
|
frEEk
Administrator
ummm... yeah
Posts: 9660
|
posted October 04, 2002 12:06 AM
drooooolll......
anyone wanna buy a sun 1010 raid array? huh? Huh? damnit i gotta unload this thing! it woudl help if it didnt have 2G drives 
onebay a week ago or so, some bankruptcy [sp] group was sellin off a whack of 5TB sun raid arrays. well, full cabinets of raid arrays that is. u immediately came to mind swft. made me drool. the arrays, not u. well, mostly at least 
|
swft
Needs a life
Full throttle!
Posts: One MEEEEEELLION
|
posted October 05, 2002 01:18 AM
Well, the saga is finally over. Zero data loss by the customer, complete hardware replacement so we can do failure analysis on the old one...Everyone's happy and I get to go to bed before 7am for a change.
|
ZHooligan
Moderator
Post Whore Extraordinaire!
Posts: 3829
|
posted October 05, 2002 09:08 AM
And yes the internet is safe again.
____________
To those who do not count their life in years, but in how life
has touched them in the past and how much it can hold in the
future; -- Youth is forever.
|
frEEk
Administrator
ummm... yeah
Posts: 9660
|
posted October 05, 2002 11:56 AM
thanks to ....SUPER SWFT!!!!!!!!!!!!
|
swft
Needs a life
Full throttle!
Posts: One MEEEEEELLION
|
posted October 05, 2002 09:31 PM
Oh no, I'm just VERY ARROGANT.
|
ZHooligan
Moderator
Post Whore Extraordinaire!
Posts: 3829
|
posted October 05, 2002 09:46 PM
Don't you mean aromatic?
____________
To those who do not count their life in years, but in how life
has touched them in the past and how much it can hold in the
future; -- Youth is forever.
|
|
|
HOME
NEW TOPIC