I forgot to mention Arista, apparently they have some good kit and are gaining some ground. I've not used or know of anyone who has, but you may want to check them out too.

www.aristanetworks.com

g

I think he is hitting a resource limit within the switch. you shouldn't see many if any TX errors. when you do it sometimes is an indication of a buffer depletion issue.

HP 8212's are great devices for HPC when you consider the cost per port and chassis capacity. Extreme and Force10 are both having a sale on 24 port 10Gbit switches this month. Personally I prefer Force10 over Extreme.

I' am in contact wiht many of North America's largest HPC admins, non of them that I am aware of have Fourndry deployed. nor could I recommend them. Especially now since they were bought by the kings of gouging. Soon you'll have to license your ethernet ports individualy, and pay a prenium of you want to use any feature, such as QoS, or trunking.

-g

When I was doing HPC storage research (Lustre, specifically), I found that Extreme switches (Summit series) consistently outperformed the competition (Cisco, Juniper, and others) by sometimes as much as 5-10%, and the Extreme was able to reliably get me about 118 MBps every time, and was giving me as much as 95% of theoretical max without any tweaking.

As always, YMMV, but I've found Extreme to be extremely reliable in terms of switch throughput, and it comes without the "Cisco" sticker premium. There are other great performers out there as well (Foundry is one of them), but I didn't get a chance to punish one the same way so unfortunately I don't have numbers handy.

Klaus

On 3/11/10 3:23 PM, "Tom Taylor" taylor@the-mill.com etched on stone tablets:

Everything is in an HP 2900 - 2 x 1Gb trunks into each cluster. (Soon to be Brocade or Extreme 10Gb)

Seems like the interfaces are not happy.

Port Total Bytes Total Frames Errors Rx Drops Tx Ctrl Limit ------- -------------- -------------- ------------ ------------ ----- ------ 1-Trk1 2,512,407,710 2,727,621,561 86,495 11,466,568 off 0 2-Trk1 3,123,561,267 1,198,521,278 18,014 8,707,104 off 0 3-Trk2 798,322,066 982,377,621 49,155 11,695,269 off 0 4-Trk2 53,405,790 2,926,838,905 163,569 14,461,588 off 0 5-Trk3 553,600,861 3,618,989,149 82,960 7,622,094 off 0 6-Trk3 1,887,102,210 4,292,551,184 60,317 6,679,635 off 0 7-Trk4 1,137,517,211 3,081,662,500 43,361 8,464,187 off 0 8-Trk4 3,503,697,323 3,465,088,585 85,276 9,135,323 off 0 9-Trk5 2,296,075,992 1,128,920,646 72,032 7,489,051 off 0 10-Trk5 2,879,550,610 2,600,747,860 54,721 8,322,167 off 0 11-Trk6 2,748,755,020 3,165,756,973 24,034 183,152,905 off 0 12-Trk6 1,611,690,357 2,354,340,713 9430 64,319,051 off 0

Thanks everyone for the test results!

Cheers, Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of bill@yuco.com Sent: Thursday, March 11, 2010 6:19 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

Out of curiosity, what type of switches are you using?

-----Original Message----- From: Tom Taylor [mailto:taylor@the-mill.com] Sent: Thursday, March 11, 2010 01:10 PM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

I've got a quick one, I'd like to test out our Isilon cluster.

It seems as if I'm getting low performance at the moment and I was wondering if anyone else out there would be able to run a simple DD to compare.

Here's my output from 2 different machines.

[root@flame-09-ny-courtney Tom]# dd if=/dev/zero of=test3.2gb bs=1M count=2096 2197815296 bytes (2.2 GB) copied, 47.8699 seconds, 45.9 MB/s

[root@ny3d-24 Tom]# dd if=/dev/zero of=ny3d.4gb bs=1M count=20096 21072183296 bytes (21 GB) copied, 432.756 s, 48.7 MB/s

I'm currently running a 6 node 6000x cluster on version 5.5.4.21.

If anyone else could send me their output I'd appreciate it.

In related news, they've just announced a product called InsightIQ. It's a VM that shows incredible statistics about the cluster. I suggest chatting to your Isilon rep and getting a demo.

Cheers, Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Sven Nielsen Sent: Tuesday, March 09, 2010 6:25 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

This is one thing I've been asked for a few times; the problem is that NAS systems have their own filesystem on the NAS head, then present the files of that filesystem out via NFS or CIFS (or whatever file- based protocol).

A fiber channel interface only knows how to speak blocks (via SCSI commands); so there has to be a driver on top of that to implement a filesystem (ext3, NTFS, CVFS, etc). Additionally, while the NAS head manages competing requests for the same blocks, there's nothing like that when you talk to a disk via Fiber Channel -- the disk will happily take a write from one system that overwrites a change just made by a different system. If one system decides to put a file where the previous system just extended a directory entry, then your filesystem is corrupt.

Even NDMP over fiber channel is similar; the NDMP server (which understands the filesystem's layout) is telling the NDMP tape device to read blocks in a particular order from the disk.

The only way for BlueArc, NetApp, etc. to present the same view of a volume via fiber channel or iSCSI that they do via NFS or CIFS would be if they: a) released a windows/linux/mac/whatever driver that implements WFS or WAFL, their on-disk filesystem type, and b) set up these drivers and the NAS head to know how to communicate to make sure that only one system is modifying a disk block at any one time (which CVFS/StorNext handles for you).

It gets to be a can of worms, and (at least for now) if you need fiber channel access to disk for multiple clients, StorNext seems to be the best combination of FCP-and-network access, since you can connect an FCP client to the filesystem, and have it re-serve the filesystem via NFS, CIFS, or StorNext LAN client. There's also filesystem software like (I think) GPFS, Lustre, and others, that I think do similar things to StorNext/CVFS, but I haven't explored those too much.

If there are other and better options, I'd love to hear about them -- anything to make the bits move faster. }:>

-Sven

On Mar 8, 2010, at 3:18 PM, J. J. Franzen wrote:

Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct FC access to their volumes? I know Blue Arc does not and am curious if anyone else does. I would so love to do away with NDMP for my backups and be able to do true incremental backups like in the good ol days...

J^2

On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss- bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with > their > sales-pitch. Can anyone share their experiences with Isilon > products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

v1 http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&a mp;mt=8 If you experience problems downloading, please use the alternative link below: http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&a mp;mt=8

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

__ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss __ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

The chipset they have been using was recently bought by Broadcom.

Not sure what the roadmap is.

Strong backers and interesting product.

Seven Group - Calgary 106-919 Centre Street N.W
Calgary, Alberta
T2A 2P6

1-877-462-1777 x 102

"Data Management with Integrity"

----- Original Message ----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com studiosysadmins-discuss-bounces@mailman.studiosysadmins.com To: discuss@studiosysadmins.com discuss@studiosysadmins.com Sent: Thu Mar 11 16:27:03 2010 Subject: OFFTOPIC: switches. RE: Researching: Isilon and alternatives

I forgot to mention Arista, apparently they have some good kit and are gaining some ground. I've not used or know of anyone who has, but you may want to check them out too.

www.aristanetworks.com

g

I think he is hitting a resource limit within the switch. you shouldn't see many if any TX errors. when you do it sometimes is an indication of a buffer depletion issue.

HP 8212's are great devices for HPC when you consider the cost per port and chassis capacity. Extreme and Force10 are both having a sale on 24 port 10Gbit switches this month. Personally I prefer Force10 over Extreme.

I' am in contact wiht many of North America's largest HPC admins, non of them that I am aware of have Fourndry deployed. nor could I recommend them. Especially now since they were bought by the kings of gouging. Soon you'll have to license your ethernet ports individualy, and pay a prenium of you want to use any feature, such as QoS, or trunking.

-g

When I was doing HPC storage research (Lustre, specifically), I found that Extreme switches (Summit series) consistently outperformed the competition (Cisco, Juniper, and others) by sometimes as much as 5-10%, and the Extreme was able to reliably get me about 118 MBps every time, and was giving me as much as 95% of theoretical max without any tweaking.

As always, YMMV, but I've found Extreme to be extremely reliable in terms of switch throughput, and it comes without the "Cisco" sticker premium. There are other great performers out there as well (Foundry is one of them), but I didn't get a chance to punish one the same way so unfortunately I don't have numbers handy.

Klaus

On 3/11/10 3:23 PM, "Tom Taylor" taylor@the-mill.com etched on stone tablets:

Everything is in an HP 2900 - 2 x 1Gb trunks into each cluster. (Soon to be Brocade or Extreme 10Gb)

Seems like the interfaces are not happy.

Port Total Bytes Total Frames Errors Rx Drops Tx Ctrl Limit ------- -------------- -------------- ------------ ------------ ----- ------ 1-Trk1 2,512,407,710 2,727,621,561 86,495 11,466,568 off 0 2-Trk1 3,123,561,267 1,198,521,278 18,014 8,707,104 off 0 3-Trk2 798,322,066 982,377,621 49,155 11,695,269 off 0 4-Trk2 53,405,790 2,926,838,905 163,569 14,461,588 off 0 5-Trk3 553,600,861 3,618,989,149 82,960 7,622,094 off 0 6-Trk3 1,887,102,210 4,292,551,184 60,317 6,679,635 off 0 7-Trk4 1,137,517,211 3,081,662,500 43,361 8,464,187 off 0 8-Trk4 3,503,697,323 3,465,088,585 85,276 9,135,323 off 0 9-Trk5 2,296,075,992 1,128,920,646 72,032 7,489,051 off 0 10-Trk5 2,879,550,610 2,600,747,860 54,721 8,322,167 off 0 11-Trk6 2,748,755,020 3,165,756,973 24,034 183,152,905 off 0 12-Trk6 1,611,690,357 2,354,340,713 9430 64,319,051 off 0

Thanks everyone for the test results!

Cheers, Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of bill@yuco.com Sent: Thursday, March 11, 2010 6:19 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

Out of curiosity, what type of switches are you using?

-----Original Message----- From: Tom Taylor [mailto:taylor@the-mill.com] Sent: Thursday, March 11, 2010 01:10 PM To: discuss@studiosysadmins.com Subject: RE: Researching: Isilon and alternatives

I've got a quick one, I'd like to test out our Isilon cluster.

It seems as if I'm getting low performance at the moment and I was wondering if anyone else out there would be able to run a simple DD to compare.

Here's my output from 2 different machines.

[root@flame-09-ny-courtney Tom]# dd if=/dev/zero of=test3.2gb bs=1M count=2096 2197815296 bytes (2.2 GB) copied, 47.8699 seconds, 45.9 MB/s

[root@ny3d-24 Tom]# dd if=/dev/zero of=ny3d.4gb bs=1M count=20096 21072183296 bytes (21 GB) copied, 432.756 s, 48.7 MB/s

I'm currently running a 6 node 6000x cluster on version 5.5.4.21.

If anyone else could send me their output I'd appreciate it.

In related news, they've just announced a product called InsightIQ. It's a VM that shows incredible statistics about the cluster. I suggest chatting to your Isilon rep and getting a demo.

Cheers, Tom

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@mailman.studiosysadmins.com] On Behalf Of Sven Nielsen Sent: Tuesday, March 09, 2010 6:25 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

This is one thing I've been asked for a few times; the problem is that NAS systems have their own filesystem on the NAS head, then present the files of that filesystem out via NFS or CIFS (or whatever file- based protocol).

A fiber channel interface only knows how to speak blocks (via SCSI commands); so there has to be a driver on top of that to implement a filesystem (ext3, NTFS, CVFS, etc). Additionally, while the NAS head manages competing requests for the same blocks, there's nothing like that when you talk to a disk via Fiber Channel -- the disk will happily take a write from one system that overwrites a change just made by a different system. If one system decides to put a file where the previous system just extended a directory entry, then your filesystem is corrupt.

Even NDMP over fiber channel is similar; the NDMP server (which understands the filesystem's layout) is telling the NDMP tape device to read blocks in a particular order from the disk.

The only way for BlueArc, NetApp, etc. to present the same view of a volume via fiber channel or iSCSI that they do via NFS or CIFS would be if they: a) released a windows/linux/mac/whatever driver that implements WFS or WAFL, their on-disk filesystem type, and b) set up these drivers and the NAS head to know how to communicate to make sure that only one system is modifying a disk block at any one time (which CVFS/StorNext handles for you).

It gets to be a can of worms, and (at least for now) if you need fiber channel access to disk for multiple clients, StorNext seems to be the best combination of FCP-and-network access, since you can connect an FCP client to the filesystem, and have it re-serve the filesystem via NFS, CIFS, or StorNext LAN client. There's also filesystem software like (I think) GPFS, Lustre, and others, that I think do similar things to StorNext/CVFS, but I haven't explored those too much.

If there are other and better options, I'd love to hear about them -- anything to make the bits move faster. }:>

-Sven

On Mar 8, 2010, at 3:18 PM, J. J. Franzen wrote:

Actually, do any of the big guys (Netapp, Isilon, etc.) allow direct FC access to their volumes? I know Blue Arc does not and am curious if anyone else does. I would so love to do away with NDMP for my backups and be able to do true incremental backups like in the good ol days...

J^2

On Mar 8, 2010, at 3:18 PM, Ian Haskin wrote:

We're a small TV-commercial shop (3D/compositing, 20 people, with 50 or so machines, incl. 4 flames) that is outgrowing our home-built Linux NAS. I'm looking for something scalable with high-bandwidth links for our compositing nodes, with a tie-in to the rest of our 3D 1Gb-ethernet network.

-----Original Message----- From: studiosysadmins-discuss-bounces@mailman.studiosysadmins.com [mailto:studiosysadmins-discuss- bounces@mailman.studiosysadmins.com] On Behalf Of Barry Robison Sent: March-08-10 5:18 PM To: discuss@studiosysadmins.com Subject: Re: Researching: Isilon and alternatives

On Tue, Mar 9, 2010 at 9:00 AM, Ian Haskin ian@topixfx.com wrote: > I've been looking into Isilon storage and I'm quite impressed with > their > sales-pitch. Can anyone share their experiences with Isilon > products or any > other products that compete/compare with them? >

Hi Ian,

What is your budget and expected workload?

We evaluated both Isilon and BlueArc, with BlueArc winning our business. For our expected artist count (200) and farm size ( ~6k cores ), we liked BA better. We were sold on the data tiering, disaster recovery, and throughput the hardware architecture allows. From other shops we heard that BA was the only thing saturating 10GbE. We had issues getting Isilon working on 10GbE at all...

At a previous facility I worked at we used a small Isilon cluster ( 4 nodes ) quite happily. I think when you get up to the number of nodes ( and accelerator nodes ) you need for a large cluster, some problems can be expected.

Good luck, -Barry _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

v1 http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&a mp;mt=8 If you experience problems downloading, please use the alternative link below: http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=318704758&a mp;mt=8

_ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

__ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss __ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

On 12/03/2010, at 1:27 PM, Greg Whynott wrote: > I forgot to mention Arista, apparently they have some good kit and
> are gaining some ground. I've not used or know of anyone who has,
> but you may want to check them out too. > > www.aristanetworks.com >

We've got a demo of a 10G Arista and so far it's really nice.

Arista use the same Fulcrum chipset as just about all the other 10G
switches out there but apparently they do make their own mainboards.
Have a look at the Fulcrum Monaco switch and then the Juniper EX2500
or the Blade Networks Rackswitch...

The Brocade Turboiron is a Woven which is now sold by Fortinet...

So it really comes down to which software feature set you need. And
price. Arista is doing some really interesting stuff with their
software at the moment.

One thing to keep in mind is that most of the new 10G stackable
switches are cut-through which is how they get low latency. But that
means that they have tiny buffers so they don't deal well with with
oversubscribed links. They'll just drop everything they can't handle.
For the most part you still need to go to a chassis switch for good
buffering.

Thanks, Matt

That Juniper EX2500 does not run JUNOS, FWIW. I think they licensed
someone else's, for the sake of having a top-of-rack 10G option. I
hope to see a proper Juniper 10G ToR switch that runs JUNOS some day.

Also take a look at Fujitsu's line of ToR 10G switches. They're
decent, fast, and mature (not Johnny-come-lately).

For full disclosure, my employer reps both Juniper and Fujitsu, but
I'm an engineer, not a sales dude...

---Ken

On Mar 11, 2010, at 7:42 PM, Matt Provost mprovost@wetafx.co.nz wrote:

On 12/03/2010, at 1:27 PM, Greg Whynott wrote: > I forgot to mention Arista, apparently they have some good kit and
> are gaining some ground. I've not used or know of anyone who has,
> but you may want to check them out too. > > www.aristanetworks.com >

We've got a demo of a 10G Arista and so far it's really nice.

Arista use the same Fulcrum chipset as just about all the other 10G
switches out there but apparently they do make their own mainboards.
Have a look at the Fulcrum Monaco switch and then the Juniper EX2500
or the Blade Networks Rackswitch...

The Brocade Turboiron is a Woven which is now sold by Fortinet...

So it really comes down to which software feature set you need. And
price. Arista is doing some really interesting stuff with their
software at the moment.

One thing to keep in mind is that most of the new 10G stackable
switches are cut-through which is how they get low latency. But that
means that they have tiny buffers so they don't deal well with with
oversubscribed links. They'll just drop everything they can't
handle. For the most part you still need to go to a chassis switch
for good buffering.

Thanks, Matt _ StudioSysAdmins-Discuss mailing list StudioSysAdmins-Discuss@mailman.studiosysadmins.com http://mailman.studiosysadmins.com/mailman/listinfo/studiosysadmins-discuss

On 12/03/2010, at 5:46 PM, Douglas C. Atkinson wrote: > Junipers EX2500 are oem blade networks.

Yes that's the point I was trying to make sorry I wasn't clearer.
There are a bunch of companies out there (Juniper, Blade Networks,
etc) reselling the Fulcrum reference design. And most of the rest
(Arista, Force10) are using the Fulcrum chipset and maybe doing
something more clever with it. So the hardware is generally about the
same and it really comes down to features and price.

Thanks, Matt

I was doing reading on these switches, one of the 'geeky' interesting things with the Arista line is they run linux and apparently you can drop to a bash shell. The reversable fans are a neat idea too, front to back or back to front cooling. -g

On 12/03/2010, at 5:46 PM, Douglas C. Atkinson wrote: > Junipers EX2500 are oem blade networks.

Yes that's the point I was trying to make sorry I wasn't clearer. There are a bunch of companies out there (Juniper, Blade Networks, etc) reselling the Fulcrum reference design. And most of the rest (Arista, Force10) are using the Fulcrum chipset and maybe doing something more clever with it. So the hardware is generally about the same and it really comes down to features and price.

Thanks, Matt

On 12/03/2010, at 7:08 PM, Greg Whynott wrote: > I was doing reading on these switches, one of the 'geeky'
> interesting things with the Arista line is they run linux and
> apparently you can drop to a bash shell. The reversable fans are a
> neat idea too, front to back or back to front cooling.

Indeed: