I know that Framestore-CFC adopted Lustre/CFS way back in the v1.4 days (about 4-5 years ago, I think), and published a press release about it. Isilon should probably qualify as a distributed file system for the purposes of this discussion, as would Panassas I believe, but most of the other parallel file systems (CFS, PVFS, GFS, GPFS, etc.) are still largely the domain of HPC research nerds working science and high-tech, I don't know that they're that common in production houses (but they probably should be!)

Klaus

On 9/1/09 11:44 PM, "Derrick MacPherson" derrick@packetsafe.net etched on stone tablets:

lol.. I wondered that and had emailed Sean off list to ask but hadn't heard back... i should've waited so i was sure i was on topic.

is anyone out there using a distributed file system at all?

On 1-Sep-09, at 11:36 PM, Klaus Steden wrote:

Hi Derrick,

I think we're actually talking about different products which have the same name and float around in the same HPC soup ... there's Lustre the grading product from Autodesk, and Lustre the distributed parallel file system from Sun now Oracle a/k/a CFS. Both are good, but very different. :-)

My posting was about CFS, the distributed parallel file system. If anyone's interested ... http://en.wikipedia.org/wiki/Lustre_(file_system)

cheers, Klaus

On 9/1/09 11:01 PM, "Derrick MacPherson" derrick@packetsafe.net etched on stone tablets:

Yup, all that plus can handle Red footage in real time with the latest hardware - I might be mistaken on that, talk to a sales droid - but i'm pretty sure of that. I'll be talking to Autodesk support (my preference instead of sales guys) later this week and will confirm and update the list if I'm wrong on that..

On 1-Sep-09, at 10:57 PM, Klaus Steden wrote:

I did a fair bit of work with Lustre over 2007-2008. It?s pretty sweet technology, but whether or not something like this backblaze would work with it would depend a great deal on how the host OS presents storage. Lustre is just another Linux file system, so it looks for LUNs upon which to construct Lustre file system objects (metadata storage, object storage, etc.), so if these things act as NAS devices, it would be a no-go. The one thing I really like about Lustre is that it supports a variety of network interconnects, so you can build a clustered SAN using GigE, IB, 10 GigE, etc. and it?s all transparent to the client. It?s fast as hell, too. I don?t get too jazzed about file systems and storage, but Lustre is definitely pretty interesting technology.

Klaus

On 9/1/09 8:34 PM, "Sean Laverty" SLaverty@blizzard.com etched on stone tablets:

Hm.

I?m wondering if a collection of backblaze systems could, if organized into a Lustre file system and front-ended by an adequate Linux Samba/NFS file server, provide a useable and relatively high- density, low-cost solution for multi-hundred TB online archives.

Has anyone on the list been working with Lustre?

• Sean

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com ] On Behalf Of Daniel Roizman Sent: Tuesday, September 01, 2009 8:07 PM To: discuss@studiosysadmins.com Subject: Re: Question about network deleting.

wow, great that they are sharing their design. one thing I wonder about is PSU failure. from the way he outlined the connections, the raid is spread across PSU's so if one PSU goes down, more than 2 drives will fail which would potentially corrupt the raid, no?

with google sharing some of their server specs and blackblaze sharing their storage, i wonder how long it'll be before we start seeing some opensource hardware configurations tightly tuned with linux distros.

On Tue, Sep 1, 2009 at 5:06 PM, Klaus Steden klaus-s@moving-picture.com wrote:

You could try one of these ...

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-ch ea p-cloud-storage/

Klaus

On 8/31/09 12:43 PM, "Greg Whynott" Greg.Whynott@oicr.on.ca etched on stone tablets: Another free non rsync approach is a ?copy on write? file system, such as ZFS. I took a look at a snapshot solution on linux using XFS at a previous job which worked as expected. I?m sure there are other filesystems out there that support it. I don?t think you have to spend a lot of money to get file level protection, its more a matter of available capacity.

-g

On 8/28/09 12:20 PM, "Jorg-Ulrich Mohnen, M.Sc., MBA" jorg.mohnen@wavgen.com wrote: Just followed this thread and wanted to give my two cents on this. Perhaps someone already mentioned it.

Its really a 0 or a 1 here. If your company has no money, then setup a cheap RAIDed solution as Jesse C suggests, and do an rsync every night from 12 midnight to 6am on the folders you need rsynced. Its really quite easy under linux /etc

I have setup cheap Linux RAIDs for this type of nightly back up on Linux for under $2000 and with a more stable and secure MAC OSX server and XRAID for under $5000. The RAIDed linux software solution would yield about 4TB of usable space, and the MAC Server XRAID solution would yield around 6TB of usable space. Keep in mind that Apple only delivered IDE XRAIDs with maximum 500GB IDE disks. I bought the 750GB disks from CDW (OEM) and they work just fine under non-warranty conditions.

This solution guarantees a 24 hour ~quasi~ snap shot at times of low to minimal production. And make this KNOWN to the artists and folks that there are only 24 hour back ups as there is no money for a more expensive solution.

The more expensive hardware like Isolons and NetApps and especially BlueArcs have more elegant solutions and they will actually give you scripts. In Blue Arcs case, they actually have prewritten stuff as part of the deal. SUN MIcroSystems is a class in itself and are excellent in this type of support. If you pay $70,000 for a small 9TB usable volume (16TB raw), you might expect them to bend over backwards to help you out. And in the three companies I have recently worked for since SONY, they did (Jim Henson Co, YuCo, and StudioGPU). I rewrote some stuff elegantly and the snap shots approached runtime and were not ~quasi~ snapshots or nightly backups, but actual snapshots.

J.

On Aug 28, 2009, at 9:02 AM, Jesse C wrote: We're currently snapshotting between 1.8TB and 2.3TB of production files with 2.7TB of snapshot disk. We snapshot every 6 hours and keep snapshots for a month by default, but we've got cron scripts that go through and remove older snapshots if we start running out disk space (which can happen if there has been ridiculous amounts of churn in the file system).

We could easily increase the size of our snapshot if we needed to. We've got 5x750GB drives right now in the RAID. If we had to, we could expand out to 6 drives in the RAID and move up to 2TB drives, which would give us close to 10TB. Not quite what you guys need, but there are motherboards out there with 10xSATA ports on them, which when combined with 2TB drives, would get you more into your ballpark.

On Fri, Aug 28, 2009 at 9:44 AM, Curtis Linstead curtis@rodeofx.com wrote:

Jesse,

Thanks for the info! I will definitely look further into this as a potential option for us.

How large was the pool you were doing the snapshot on and how large was the RAID you built?

From what I?m reading I would need the size of one full backup

plus a little more and my current data pool can hit upwards of 16TB.

Not sure how easily I could configure that on a montherboard based raid heh, but it definitely interests me.

Thanks again!

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com

http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discus>>>>> s >>>>> >>>>> >>>>> Jorg-Ulrich Mohnen, M.Sc. MBA >>>>> WavGen / Terratracer Incorporated >>>>> 901 North Curson Avenue >>>>> West Hollywood, CA 90046 >>>>> Tel 888-654-5615 >>>>> Email jorg.mohnen@wavgen.com >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _ >>>>> StudioSysAdmins-Discuss mailing list >>>>> StudioSysAdmins-Discuss@studiosysadmins.com >>>>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discus>>>>> s >>>>> >>>>> >>>>> __ >>>>> StudioSysAdmins-Discuss mailing list >>>>> StudioSysAdmins-Discuss@studiosysadmins.com >>>>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discus>>>>> s >>>>> >>>>> >>>>> __ >>>>> StudioSysAdmins-Discuss mailing list >>>>> StudioSysAdmins-Discuss@studiosysadmins.com >>>>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discus>>>>> s >>>> >>>> __ >>>> StudioSysAdmins-Discuss mailing list >>>> StudioSysAdmins-Discuss@studiosysadmins.com >>>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >>> >>> __ >>> StudioSysAdmins-Discuss mailing list >>> StudioSysAdmins-Discuss@studiosysadmins.com >>> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss >> >> __ >> StudioSysAdmins-Discuss mailing list >> StudioSysAdmins-Discuss@studiosysadmins.com >> http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss > > __ > StudioSysAdmins-Discuss mailing list > StudioSysAdmins-Discuss@studiosysadmins.com > http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

---

StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss