Sponsors
Sponsor Products
Metadata controller options?
posted by Perry Paolantonio  on March 27, 2018, 10:29 a.m. (3 months, 23 days ago)
4 Responses     1 Plus One's     3 Comments  

We're a small studio (3 users, about a dozen workstations), that specializes in film scanning, color correction, restoration. We're dealing with very big files - a combo of large image sequences and large containerized files (ProRes). We have a 40GbE network that has been up and running for a few years, and all machines are connected over SMB. Each workstation has a large local RAID, so we've been able to limp along by reading stuff over the network from one shared local RAID, doing something with the files, and writing to that workstation's local RAID. But this has obvious limitations, so over the past few months, I built an iSCSI SAN. It has two 60TB hardware RAID6 pools, which we've divided up into 12-16TB iSCSI targets.

The hardware has been up and running for a while, but we're using it the way we were local drives: A workstation will mount an iSCSI target, and then use SMB to share that with others that may need occasional access. This kind of works, but what I want is to have multiple machines with read access on a target that one machine has read/write access on. SMB is a performance bottleneck, so I want to be able to connect directly over iSCSI.

So I am looking for a metadata controller for iSCSI on a 40GbE network, software only, that will work with our existing hardware installation. So far I've found a grand total of two options:

TigerStore (used to be MetaSAN) - about $1000/server but price is tied to storage size. More space = more costs, so we're probably looking at $2k to start, $3k by the end of the year. This seems an arbitrary way to price, and it rubs me the wrong way. Also, I haven't heard from them after trying to contact them a few weeks ago with questions, so that's not encouraging. 

iSANmp (Storage Network Solutions) - not a metadata server, but a peer-to-peer system that requires the software be installed on each workstation, at $200/seat. Probably in the $2500 range to set this up. Also, each machine uses a USB dongle, so adding new seats is a little bit of a pain.

*all* I want is a software solution. I don't need a turnkey system. We can build the server or use one of our existing servers for this purpose. I dont care if the metadata controller is running on Windows or Linux, but we do need support on the workstation side for Mac, Windows 7 and Linux (CentOS 6 and 7).

Is an inexpensive software-only solution to this problem a pipe dream? I'm hoping to make a list of stuff to look at at NAB so any suggestions are welcome.

Thanks!


Thread Tags:
  storage 

Response from Anonymous @ March 27, 2018, 12:40 p.m.
Ive been using TigerStore for several years now (with iSCSI/Infiniband SRP storage) and its been rock solid. Support has been quick and helpful when Ive had concerns, and they've even provided custom builds of the client software when one workstation was having some unique issues.
The only frustrating point is that the TigerStore server and all the clients have to be on the same version of the software. As a result I do have to schedule downtime to update the server and every single client when I need support for a newer OS, linux kernel, etc. It doesnt happen often, but it would be great to only have to update the affected client in the case of OS compatibility. I dont have too much experience with other MDCs, however, and for all I know this might be the norm.
The storage-based pricing structure is relatively new (announced at NAB last year I think?) - pricing used to be based on the number of SAN clients. Were still on the old client-based licensing structure, but will be moving to the new storage-based one eventually. In my situation, the latter makes more sense as Im adding workstations more often than expanding the storage. Sometimes I set up test stations and dont want to buy another license just for SAN access.

On March 27, 2018 at 10:46:45, julian firminger (justdigitalfilm@gmail.com) wrote:

An inexpensive stable and reliable software only solution is indeed a pipe dream. You should care *a lot* about where and on what the MDCs run. If you lose your MDCs you lose your whole operation and all the data....
SMB does not need to be a performance bottleneck with 3.1. I've seen pretty healthy workloads on 40G.
I cant speak to those you have mentioned but I would worry deeply about putting a "cheap" overlay on top of your infra. It'd be a bit like putting a cheap roll cage in a 100K sports car you want to race on weekends. have you considered GPFS (cue others on this list)

Julian Firminger

Senior Systems Administrator United Broadcast Facilities Amsterdam, The Netherlands

On Tue, Mar 27, 2018 at 4:29 PM, Perry Paolantonio <content@studiosysadmins.com> wrote:

We're a small studio (3 users, about a dozen workstations), that specializes in film scanning, color correction, restoration. We're dealing with very big files - a combo of large image sequences and large containerized files (ProRes). We have a 40GbE network that has been up and running for a few years, and all machines are connected over SMB. Each workstation has a large local RAID, so we've been able to limp along by reading stuff over the network from one shared local RAID, doing something with the files, and writing to that workstation's local RAID. But this has obvious limitations, so over the past few months, I built an iSCSI SAN. It has two 60TB hardware RAID6 pools, which we've divided up into 12-16TB iSCSI targets.

The hardware has been up and running for a while, but we're using it the way we were local drives: A workstation will mount an iSCSI target, and then use SMB to share that with others that may need occasional access. This kind of works, but what I want is to have multiple machines with read access on a target that one machine has read/write access on. SMB is a performance bottleneck, so I want to be able to connect directly over iSCSI.

So I am looking for a metadata controller for iSCSI on a 40GbE network, software only, that will work with our existing hardware installation. So far I've found a grand total of two options:

TigerStore (used to be MetaSAN) - about $1000/server but price is tied to storage size. More space = more costs, so we're probably looking at $2k to start, $3k by the end of the year. This seems an arbitrary way to price, and it rubs me the wrong way. Also, I haven't heard from them after trying to contact them a few weeks ago with questions, so that's not encouraging.

iSANmp (Storage Network Solutions) - not a metadata server, but a peer-to-peer system that requires the software be installed on each workstation, at $200/seat. Probably in the $2500 range to set this up. Also, each machine uses a USB dongle, so adding new seats is a little bit of a pain.

*all* I want is a software solution. I don't need a turnkey system. We can build the server or use one of our existing servers for this purpose.I dont care if the metadata controller is running on Windows or Linux, but we do need support on the workstation side for Mac, Windows 7 and Linux (CentOS 6 and 7).

Is an inexpensive software-only solution to this problem a pipe dream? I'm hoping to make a list of stuff to look at at NAB so any suggestions are welcome.

Thanks!


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

1 Plus One's     1 Comments  
   

Response from Perry Paolantonio @ July 10, 2018, 9:19 a.m.

Thought I'd follow up on this.

As it turns out there is no linux client for SNS's iSANmp, unless you have their storage appliances. I went to NAB and was told by three separate people from SNS that it would work for each of the scenarios I set out. Then after two weeks of trying to get demo software from them, i was told it wouldn't work unless we bought their storage hardware. I was pretty cranky at that point because we desperately needed to get the SAN online for a project that was starting. 

Went back to TigerStore and it turned out to be more affordable than I had originally understood. We went with our own server (a simple Windows 7 machine, nothing fancy or expensive, built from some old components that were sitting on the shelf). The storage server serves up iSCSI targets, which this Windows 7 machine mounts locally. TigerStore then serves that up to the workstations with file-level locking, so it's much nicer than block level. We opted for the 128TB limit with unlimited clients, and by the end of the year we'll upgrade that to 256T, which should hold us for quite some time. Total cost was about $2000 for the software and it's been rock solid for almost 2 months. For slightly more money, we could have done unlimited space with a 5-workstation limit, but after assessing the storage needs, it made more sense to go with unlimited seats. I can access all the SAN volumes from my lowly iMac on my desk over the 1 gigbit network, which is great for QC'ing stuff or uploading to our external file sharing system. 

I'm very happy with this setup so far. 


0 Plus One's     0 Comments  
   

Response from Jean-Francois Panisset @ March 27, 2018, 11:10 p.m.
Quantum StorNext is a pretty widely used solution in M&E, and will work with iSCSI LUNs. When I've used it in the past it typically wanted 2 standalone metadata controller servers to implement failover.

From your use of SMB I'm guessing you are Windows based? If you are running Server 2016 on those machines you could consider setting up Storage Spaces Direct, which lets you aggregate local storage into a large, shared storage pool, although at this point it might be more of a science experiment than anything else (the use case for Storage Spaces Direct tends to be Hyper-V VM storage for virtualization clusters rather than M&E applications). Still could be intriguing:

https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/storage-spaces-direct-overview

Finally, these days SMB 3.1 performance over a high performance network is quite high. You said "I built an iSCSI SAN. It has two 60TB hardware RAID6 pools, which we've divided up into 12-16TB iSCSI targets." What kind of storage shares those LUNs over iSCSI? Is it a Windows box, Linux, or dedicated iSCSI arrays? If it's a Linux or Windows server, you could consider creating a local filesystem and sharing it via SMB to your 3 clients, you may in fact have enough performance for your needs without the complication of a clustered filesystem, or having to worry about an additional software layer for your storage.

JF


On Tue, Mar 27, 2018 at 9:35 AM, Trevor Murray <trevor@criterion.com> wrote:
Ive been using TigerStore for several years now (with iSCSI/Infiniband SRP storage) and its been rock solid. Support has been quick and helpful when Ive had concerns, and they've even provided custom builds of the client software when one workstation was having some unique issues.
The only frustrating point is that the TigerStore server and all the clients have to be on the same version of the software. As a result I do have to schedule downtime to update the server and every single client when I need support for a newer OS, linux kernel, etc. It doesnt happen often, but it would be great to only have to update the affected client in the case of OS compatibility. I dont have too much experience with other MDCs, however, and for all I know this might be the norm.
The storage-based pricing structure is relatively new (announced at NAB last year I think?) - pricing used to be based on the number of SAN clients. Were still on the old client-based licensing structure, but will be moving to the new storage-based one eventually. In my situation, the latter makes more sense as Im adding workstations more often than expanding the storage. Sometimes I set up test stations and dont want to buy another license just for SAN access.

On March 27, 2018 at 10:46:45, julian firminger (justdigitalfilm@gmail.com) wrote:

An inexpensive stable and reliable software only solution is indeed a pipe dream. You should care *a lot* about where and on what the MDCs run. If you lose your MDCs you lose your whole operation and all the data....
SMB does not need to be a performance bottleneck with 3.1. I've seen pretty healthy workloads on 40G.
I cant speak to those you have mentioned but I would worry deeply about putting a "cheap" overlay on top of your infra. It'd be a bit like putting a cheap roll cage in a 100K sports car you want to race on weekends. have you considered GPFS (cue others on this list)

Julian Firminger

Senior Systems Administrator United Broadcast Facilities Amsterdam, The Netherlands

On Tue, Mar 27, 2018 at 4:29 PM, Perry Paolantonio <content@studiosysadmins.com> wrote:

We're a small studio (3 users, about a dozen workstations), that specializes in film scanning, color correction, restoration. We're dealing with very big files - a combo of large image sequences and large containerized files (ProRes). We have a 40GbE network that has been up and running for a few years, and all machines are connected over SMB. Each workstation has a large local RAID, so we've been able to limp along by reading stuff over the network from one shared local RAID, doing something with the files, and writing to that workstation's local RAID. But this has obvious limitations, so over the past few months, I built an iSCSI SAN. It has two 60TB hardware RAID6 pools, which we've divided up into 12-16TB iSCSI targets.

The hardware has been up and running for a while, but we're using it the way we were local drives: A workstation will mount an iSCSI target, and then use SMB to share that with others that may need occasional access. This kind of works, but what I want is to have multiple machines with read access on a target that one machine has read/write access on. SMB is a performance bottleneck, so I want to be able to connect directly over iSCSI.

So I am looking for a metadata controller for iSCSI on a 40GbE network, software only, that will work with our existing hardware installation. So far I've found a grand total of two options:

TigerStore (used to be MetaSAN) - about $1000/server but price is tied to storage size. More space = more costs, so we're probably looking at $2k to start, $3k by the end of the year. This seems an arbitrary way to price, and it rubs me the wrong way. Also, I haven't heard from them after trying to contact them a few weeks ago with questions, so that's not encouraging.

iSANmp (Storage Network Solutions) - not a metadata server, but a peer-to-peer system that requires the software be installed on each workstation, at $200/seat. Probably in the $2500 range to set this up. Also, each machine uses a USB dongle, so adding new seats is a little bit of a pain.

*all* I want is a software solution. I don't need a turnkey system. We can build the server or use one of our existing servers for this purpose.I dont care if the metadata controller is running on Windows or Linux, but we do need support on the workstation side for Mac, Windows 7 and Linux (CentOS 6 and 7).

Is an inexpensive software-only solution to this problem a pipe dream? I'm hoping to make a list of stuff to look at at NAB so any suggestions are welcome.

Thanks!


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     1 Comments  
   

Response from Julian Firminger @ March 27, 2018, 10:50 a.m.
An inexpensive stable and reliable software only solution is indeed a pipe dream. You should care *a lot* about where and on what the MDCs run. If you lose your MDCs you lose your whole operation and all the data....
SMB does not need to be a performance bottleneck with 3.1. I've seen pretty healthy workloads on 40G.
I cant speak to those you have mentioned but I would worry deeply about putting a "cheap" overlay on top of your infra. It'd be a bit like putting a cheap roll cage in a 100K sports car you want to race on weekends. have you considered GPFS (cue others on this list)

Julian Firminger

Senior Systems AdministratorUnited Broadcast FacilitiesAmsterdam, The Netherlands

On Tue, Mar 27, 2018 at 4:29 PM, Perry Paolantonio <content@studiosysadmins.com> wrote:

We're a small studio (3 users, about a dozen workstations), that specializes in film scanning, color correction, restoration. We're dealing with very big files - a combo of large image sequences and large containerized files (ProRes). We have a 40GbE network that has been up and running for a few years, and all machines are connected over SMB. Each workstation has a large local RAID, so we've been able to limp along by reading stuff over the network from one shared local RAID, doing something with the files, and writing to that workstation's local RAID. But this has obvious limitations, so over the past few months, I built an iSCSI SAN. It has two 60TB hardware RAID6 pools, which we've divided up into 12-16TB iSCSI targets.

The hardware has been up and running for a while, but we're using it the way we were local drives: A workstation will mount an iSCSI target, and then use SMB to share that with others that may need occasional access. This kind of works, but what I want is to have multiple machines with read access on a target that one machine has read/write access on. SMB is a performance bottleneck, so I want to be able to connect directly over iSCSI.

So I am looking for a metadata controller for iSCSI on a 40GbE network, software only, that will work with our existing hardware installation. So far I've found a grand total of two options:

TigerStore (used to be MetaSAN) - about $1000/server but price is tied to storage size. More space = more costs, so we're probably looking at $2k to start, $3k by the end of the year. This seems an arbitrary way to price, and it rubs me the wrong way. Also, I haven't heard from them after trying to contact them a few weeks ago with questions, so that's not encouraging.

iSANmp (Storage Network Solutions) - not a metadata server, but a peer-to-peer system that requires the software be installed on each workstation, at $200/seat. Probably in the $2500 range to set this up. Also, each machine uses a USB dongle, so adding new seats is a little bit of a pain.

*all* I want is a software solution. I don't need a turnkey system. We can build the server or use one of our existing servers for this purpose.I dont care if the metadata controller is running on Windows or Linux, but we do need support on the workstation side for Mac, Windows 7 and Linux (CentOS 6 and 7).

Is an inexpensive software-only solution to this problem a pipe dream? I'm hoping to make a list of stuff to look at at NAB so any suggestions are welcome.

Thanks!


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     1 Comments