My search for data storage
Posted by mjsouth on February 3, 2008
My job at Monolithic has been one of the greatest adventures in my life, and has given me opportunities to participate in many different projects. There is one part of my job that not too many people know about, but that is the job of IT manager.
David has always believed in keeping our office computer systems up to date, and state of the art, and that has definitely contributed to the design, manufacturing, and construction of Monolithic Domes.
Our website is one of our biggest assets. Our website got 6,225,837 hits last month and we hope to keep that number growing everyday.
Our office only uses Apple computers. We have iMacs for our draftsman, Mac Minis, a beefed up Mac Pro (see video) that has two dual core processors and 16 gb of ram for finite element analysis, MacBooks, MacBook Pros, and recently a new MacBook Air for my boss David.
All of the computers, except for the laptops, are ran using an xServe that maintains our Directory Services. That means, that most of the computers are just shells, and all our user data comes from one xServe. Well, that xServe is getting very full, so it was my job to figure out our data storage plan for the future. I started by looking at what Apple had to offer, and this is called an xRaid.
The xRaid is a box that can hold up to 14 hard drives, and combine them to make one massive place to store your data. The first thing I looked at was the price, between 6 and 12 thousand dollars, I said its time to look somewhere else.
After looking around at a lot of different options, way too many to post here, I am back to the xRaid. Here’s why: #1 This is a permanent solution. Once we get over the pain of buying the thing, we won’t have to buy anything like it for a long time. When we need more space, we just go drop in another drive. #2 The xRaid is built to last. The xRaid has two power supplies, two cooling systems, and can be implemented with any kind of raid that you want, so even if do lose a drive, you will not lose data. #3 Apple’s Drive Modules. I was very surprised to learn that Apple’s drives go through 350 hours of testing vs. the 5 minutes for regular drives. Their drives also have a .02% failure rate vs. the 20% failure rate of an off the shelf drive. #4 It is completely manageable by remote. Which in my job is important, since I am out on the site half the time.
*The xRaid also works very well in Windows, Linux, and even mixed environments.
So Mike, where do you learn all this stuff? Well, my brother Dave is the person who continues to teach me, and he has pointed me to another place. We have been buying all of our equipment through a company called Expercom. My favorite thing about Expercom, is that when you have troubles or need advice, they are right there to help. They have a wonderful customer service department and have literally saved us thousands of dollars on computer equipment.
February 3, 2008 at 6:01 pm
Sounds like a great storage solution.
Personally I build my own file servers. It is pretty inexpensive, and very flexible.
Recently, I bought 4 hard drives. I found one went bad after being powered for 11 hours. The second one went bad after 57 hours. They were replaced under warrantly.
The third one, though it passed all of the manufacturers tests, still read back data incorrectly about once ever 4 gigabytes. I have sent it back for replacement. It sure would be nice to buy drives that have been more throughly burned-in, like the Apple
Drive Modules. It took me many hours to track down these bad hard drives. In the past I have had very few hard drives fail.
On another note, fileservers have nothing to do with backing up data.
I recommend backing up data often, and storing the backups either in a fireproof,
media safe, or offsight, or both.
February 3, 2008 at 11:23 pm
Well, I really haven’t gone into a lot detail about exactly how I have set up my computer system for security reasons, but this is what I can tell you.
We are going to start backing our server up with Leopards built in tool called Time Machine. Time Machine does an incremental backup once an hour for 24 hours, every day for a month, and once a week for as many weeks as the drive can hold. The only thing I wish Time Machine would do is allow me to change the every hour backup to be once a day, but again, its so easy to use that we are going to give it a shot.
I have also been involved with setting up some of our own Linux servers, but when you build your own server, you own the server. My goal with this setup is to make a server that is so easy to manage, and that I can easily teach some of the people in my office to do the maintenance. If I can train others to manage the upkeep, then I can focus my energy into other aspects of the company.
You have also confirmed what I was saying about off the shelf hard drives. Thanks for the comment.
February 6, 2008 at 1:23 am
Mike,
Greetings! I noticed the sharp point to the airform. Is this a new design?
Mike Carr
February 6, 2008 at 3:19 am
No, this picture was taken very early, probably 20 min after inflation, and we hadn’t put it up to 2 inches of pressure yet. The airform starts out like that so when it stretches it looks normal. I am going up there tomorrow and I will take some more pictures.
February 7, 2008 at 6:10 pm
The XServe RAID is a good choice. We did a study of their performance for a government agency and were generally quite pleased. At the time they had performance approaching that of (much) more expensive devices. We opted not to use them but that had to do with the need for offsite block level realtime replication of data. I’ve personally used “personal NAS” devices that hold 4 drives whih put them in a RAID array attached to an Ethernet. Performance just isn’t there. I’ve also built my own. Again, you can save money with a roll-it-yourself or another vendor but for OS X integration and performance the XServe RAID fits a sweet spot.
Time machine and portable home directories really make an easy environment to administer. I still use both since PHDs give a very quick “oops, laptop is dead… walk over to the mac mini and get back to work instantly (well, save time for the system to automatically copy my home directory)” Time machine has already saved me once when I inadvertently deleted the wrong files, purged the trash, and didn’t realize it for a few days.
February 12, 2008 at 8:12 pm
Nice blog, Mike!
February 17, 2008 at 4:23 pm
Hi Mike,
I recently built a workstation and discovered that there are actually different classes/grades of hard drives, consumer and enterprise. Consumer drives are not intended to be run 24/7 while enterprise drives are specifically designed and engineered to be used 24/7. They are more tolerant of vibration and environmental temperatures. I’d guess that the xRaid server is equipped with the enterprise class hard drives. Good Luck!
TerryG
February 24, 2008 at 11:15 pm
I know a tiny amount about Time Machine. It isn’t a real backup solution. It is a ‘oops I deleted some files I really didn’t mean to’ solution. A backup solution will copy your data elsewhere. If your machine fail, the backup is separate. If the hard drives fail, the backup is separate. You need something that holds the data, that you can pick up and move elsewhere. Personally, I store my backups in my fireproof safe. Think about what you would do to secure your data if someone stole your xServe. Think about the computers in the twin towers at 9/11. The companies that didn’t have offsite backup, all went out of business.
One solution, is to buy another xServe, and copy all the data from one to another, and store the second one in a safe location. Not the cheapest solution, but perhaps the
simplest one.
jeff
February 25, 2008 at 12:40 am
Like I said, I haven’t gone into a lot of detail about my plans for many reasons. But I will tell you that you are wrong about Time Machine. Time machine isn’t limited to internal or external drives, it can also backup to network shares (as long as both computers are running Leopard). My plan is to have our server use Time Machine to do an incremental, fully restorable backup, to a computer that is on our network but out of our office. Backing up to an off site location, like you said, is pretty much the only protection from fire, theft, etc.
In our office, we have 5 MacBooks, a MacBook Air, and a MacBook Pro. All of these laptops are using Time Machine, and are backing up to a Mac Mini with 2 500Gb drives hooked to it. We bought our lead draftsman a new MacBook Pro and it took it about 2 hours to restore about 30 Gb of data over our Gigabit network. When it was done, it was exactly like he had left it. Time Machine works.
One more quick note: The reason I am so excited about Time Machine, is that if my server room burned to the ground, it would be simple to restore. Just buy a new server, turn it on, point it to the network shared Time Machine Backup, and let it do the work. If I were to restore it from a standard backup, I would be able to restore all the data, but the preferences, users, groups, shares, etc. would have to be restored by hand.
Jeff, you have triggered a lot of questions in my mind about exactly how well the xServe would restore. I got a new xServe yesterday, and I am going to do some Time Machine tests and post them online. Then we can see if my theory is right or if I will run into some other problems. Thanks for the comment.