Sponsors
Sponsor Products
Render Node Server Recommendations
posted by Brent Slater  on May 24, 2016, 10:08 p.m. (2 years ago)
13 Responses     0 Plus One's     1 Comments  

Hi Guys,

We are looking at increasing the amount of render nodes that we are using for a Deadline Render Farm.

We reached out to our supplier and they provided these specs which we think as a bit of overkill.

DELL Poweredge Server R630

PowerEdge R630 Motherboard MLK

Intel Xeon Dual E5-2670 v3 2.3GHz,30M

Cache,9.60GT/s QPI,Turbo,HT,12C/24T

Chassis with up to 8, 2.5" Hard Drives, 3xPCIe

8x16GB RDIMM, 2133 MT/s, Dual Rank, x4 Data Width

Dell Remote Access Controller, Enterprise

2 x300GB 15K RPM SAS 12Gbps 2.5in

PERC H730 Integrated RAID Controller, 1GB Cache

120W Heatsink for PowerEdge R630

DVD+/-RW, SATA, Internal

Dual, Hot-plug, Redundant Power Supply (1+1), 750W

Broadcom 5720 QP 1Gb Network Daughter Card

ReadyRails Sliding Rails With Cable Management Arm

RAID 1 for H330/H730/H730P (2 HDDs or SSDs)

Windows Server 2012R2 Standard Edition,

Factory Installed, 2 Socket, 2 VMs, No CALs

Windows Server 2012R2 Standard Media

FI Standard Ed Downgrade image, Eng

3Yr ProSupport: NBD Service (Parts + Labour)

Limited Warranty: Year 1-3 (NBD)

Limited Warranty: Year 1-3 (POW)

ProSupport: 7x24 TS & Assistance: 3Yr

 

Is there any other servers in the Dell rack mount range that anyone would suggest.

We are in need of these for a large show launch in the near future so any response would be great.

 

Cheers

 

Brent


Thread Tags:
  render 

Response from Brian Krusic @ July 18, 2016, 4:06 p.m.
Well, perhaps branch out from Dell and try bell.
Just get a couple 56 core units from William@Bell; rent to own.
Load em with SSD and 256GB RAM so you can stack renders in parallel.
The SSD comes in awfully handy for swap.
Our tests have shown them to be faster then the usual 64 core units in Maya, Houdini, Nuke.
The 56 core units;E5-2690 v4 @ 2.60GHz
The 64 core units;E5-2698 v3 @ 2.30GHz
We noticed that artists dont mind waiting for renders as long as they actually do get CPU time.  Its an interesting psychology.
- Brian
A good day is when no one shows up... and you don't have to go anywhere."
On May 24, 2016, at 7:08 PM, Brent Slater <content@studiosysadmins.com> wrote:

Hi Guys,

We are looking at increasing the amount of render nodes that we are using for a Deadline Render Farm.

We reached out to our supplier and they provided these specs which we think as a bit of overkill.

DELL Poweredge Server R630

PowerEdge R630 Motherboard MLK

Intel Xeon Dual E5-2670 v3 2.3GHz,30M

Cache,9.60GT/s QPI,Turbo,HT,12C/24T

Chassis with up to 8, 2.5" Hard Drives, 3xPCIe

8x16GB RDIMM, 2133 MT/s, Dual Rank, x4 Data Width

Dell Remote Access Controller, Enterprise

2 x300GB 15K RPM SAS 12Gbps 2.5in

PERC H730 Integrated RAID Controller, 1GB Cache

120W Heatsink for PowerEdge R630

DVD+/-RW, SATA, Internal

Dual, Hot-plug, Redundant Power Supply (1+1), 750W

Broadcom 5720 QP 1Gb Network Daughter Card

ReadyRails Sliding Rails With Cable Management Arm

RAID 1 for H330/H730/H730P (2 HDDs or SSDs)

Windows Server 2012R2 Standard Edition,

Factory Installed, 2 Socket, 2 VMs, No CALs

Windows Server 2012R2 Standard Media

FI Standard Ed Downgrade image, Eng

3Yr ProSupport: NBD Service (Parts + Labour)

Limited Warranty: Year 1-3 (NBD)

Limited Warranty: Year 1-3 (POW)

ProSupport: 7x24 TS & Assistance: 3Yr

 

Is there any other servers in the Dell rack mount range that anyone would suggest.

We are in need of these for a large show launch in the near future so any response would be great.

 

Cheers

 

Brent

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Peter Devlin @ July 18, 2016, 4:06 p.m.
My 0.02 worth.
First off the Dell R630 is overkill in this scenario. That's a lot of lettuce for all of those extras (Windows server OS, RAID card, management card). Don't be tempted.
As noted already, your render profile is important. For Maya and V-ray I'd take a closer look at the Dell R430 form factor instead. Chassis, RAM and CPU numbers should be adjusted to suit that render profile. Depending on how you use Deadline, local on-node asset caching may benefit from SSD over HDD. Caveat is that you'll get a good price point for dual Xeons but it won't be the most space efficient in terms of CPU dies per rack U or indeed cabling and network ports required.
Specs-wise I'm in the UK so my sweet spot for CPU bang for buck may differ from USA or others. We've found that the v3 CPUs in the 2650v3 were the best, and we reckon that the 2630v4 will be the best in the new iteration (the 2650v4 is a huge jump in price in the UK). Do you find that your V-ray renders scale linearly with number of cores? If so then look to the higher core count CPUs and adjust RAM accordingly.
Note also that Dell absolutely will not sell you render nodes with Windows 7 64 bit. Do you need Windows at all? if you do you'll have to source it yourself via VLK or similar.
If you are not unbreakably wedded to Dell, or indeed if you wish some negotiating leverage with the Dell drones, cost out a Supermicro 2UTwin2 chassis with similar RAM and CPU per mobo. The savings may surprise you ;)


--
Thanks,

Peter Devlin
Head of ITTel+44 (0)141 572 2802




A X I S

Axis Productions Limited

7.1 Skypark 1, Elliot Place

Glasgow, G3 8EP

axisanimation.com


-----------------------------------------------------------------------

Axis Animation (Axis Productions Ltd)
Registered in Scotland: SC306712

Registered Office: Suite 7-1, The Skypark, 8 Elliot Place, Glasgow G3 8EP

0 Plus One's     0 Comments  
   

Response from Ali Poursamadi @ July 18, 2016, 4:05 p.m.

Depending on your rendering strategy, If you are not localizing your data on render nodes before rendering, the storage on that node is overkill, can easily be replaced with a 512 SSD ( could be NVME as well ) .
-Ali
On Tue, May 24, 2016 at 7:09 PM, Brent Slater <content@studiosysadmins.com> wrote:

Just to add to that.

We are currently only using the farmf for Maya & V-ray rendering

Cheers


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from William Sandler @ July 18, 2016, 4:05 p.m.
I would get one of those in for demo and a similar system with a CPU like the E5-2620 V3 and compare how fast they render your scenes. That CPU is about 3x the cost of the the 2620V3, but gives you double the cores. So it depends if your goals are server room density or best price per scene rendered. Of course you also have to factor in power costs.
Agreed with Ali, SSDs over spinning rust any day.



William SandlerAll Things Media, LLCOffice:201.818.1999 Ex 158.william.sandler@allthingsmedia.com
On Tue, May 24, 2016 at 10:09 PM, Brent Slater <content@studiosysadmins.com> wrote:

Just to add to that.

We are currently only using the farmf for Maya & V-ray rendering

Cheers


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Jean-Francois Panisset @ July 18, 2016, 4:05 p.m.
For a render farm application, a "full fat" server like the R630 may not be the most cost effective approach, you are paying a lot of money for things like individual iDRAC Enterprise, redundant PSU... which are important if you have a few servers running important workloads, but for render farm applications you don't really care if some small percentage of your servers are down at any one time. I would take a look at high density SuperMicro stuff like the TwinBlade chassis which gets you 20 x 2P servers in 7U, and all that comes out the back is 4 power cords, a 10GbE uplink and a Cat6 for management (having to individually rack and plumb 1U servers with both network and IPMI is annoying). And the SuperMicro stuff has just enough management functionality to remotely configure and manage your blades.

You could also take a look at the Dell PowerEdge FX stuff, which can fit 8 x 2P servers in 2U: again you can leverage shared infrastructure between the blades, and maybe you can get good pricing to get close to SuperMicro level.

The E5-26xx v4 CPUs are starting to ship, and you typically get 2 more cores per CPU for a given pricing level. Given that Intel typically doesn't discount their previous generation parts, I would suggest you look at v4 parts. So for instance instead of the E5-2670 v3 (120W, 12 core, 2.3GHz), you could get something like the E5-2680 v4 (120W, 14 core, 2.4GHz).

Unless you are running an entirely "free" rendering stack (say CentOS + free render scheduler + Mantra), don't forget the cost of your software in deciding what CPU to get. Modern renderers like V-Ray and Arnold typically scale quite while with the number of cores, so I would suggest getting the fastest CPUs your money and power/cooling budgets can afford, fewer/faster machines will typically get you more "frames per dollar" than more slower machines.

JF



On Tue, May 24, 2016 at 7:20 PM, William Sandler <william.sandler@allthingsmedia.com> wrote:
I would get one of those in for demo and a similar system with a CPU like the E5-2620 V3 and compare how fast they render your scenes. That CPU is about 3x the cost of the the 2620V3, but gives you double the cores. So it depends if your goals are server room density or best price per scene rendered. Of course you also have to factor in power costs.
Agreed with Ali, SSDs over spinning rust any day.



William SandlerAll Things Media, LLCOffice:201.818.1999 Ex 158.william.sandler@allthingsmedia.com
On Tue, May 24, 2016 at 10:09 PM, Brent Slater <content@studiosysadmins.com> wrote:

Just to add to that.

We are currently only using the farmf for Maya & V-ray rendering

Cheers


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Ben Tucker @ June 10, 2016, 9 p.m.

I'm working through a similar problem Drew. We're predominantly a tabletop food / live action company but we are getting more CG and VFX jobs. Rental costs are starting to add up so here's a build list I've ordered, hoping to test the two processors in the coming days. The v1 proc you can get clean pulled for $110 ea. and Newegg lists a patching pair for $190. The newest proc is obviously more expensive so we're curious if the added cost is worth it or if we build these blades with the v1 proc, and swap out processors & RAM as projects require. 

Suoermicro 1028TR-T Link $1,500
Intel E5 2670 v1 Link $110
Intel E5 2650 v3 Link $1,150
8GB DD4 ECC Sticks Link $60
SSD Boot Link $90

Our rendering is dispatched through Deadline and predominantly Maya / VRay & Nuke, though playing a bit with Krakatoa and other volumetric particle based rendererers. I'm just waiting on the Supermicro chasis so I'll update here once we get it up and running and have a window to run some tests on it. Hope this helps your search some.


0 Plus One's     0 Comments  
   

Response from Devin Conn @ June 10, 2016, 2:17 p.m.

Ditto to Drew. FX2s have a nice quad blade form factor. Plus you can manage multiple chassis with their chassis control module, and Dell's remote management tool, iDRAC. 


0 Plus One's     0 Comments  
   

Response from Drew Rosen @ May 25, 2016, 11:22 p.m.

Mantra Report

We have had good success and feedback renting out the Dell FX2 series. Works well and the processor is middle of the line, so it's good on a price per pound ratio.

 

Rental Server Dell PowerEdge FX2 FC630

comes with (4)PowerEdge FC630 Server Nodes - (4)2xE5-2600v3, 2.3Ghz, 2.5MB Cache (10 Core) - (16)16GB

RDIMM - (4)500GB HDD - (2)Power Cord (C13-C14)

 

But with the new v4 chips out, it's a new ball game!

 

Drew Rosen

vfxnow.com

 


0 Plus One's     0 Comments  
   

Response from Drew Rosen @ May 25, 2016, 11:22 p.m.

Mantra Report

We have had good success and feedback renting out the Dell FX2 series. Works well and the processor is middle of the line, so it's good on a price per pound ratio.

 

Rental Server Dell PowerEdge FX2 FC630

comes with (4)PowerEdge FC630 Server Nodes - (4)2xE5-2600v3, 2.3Ghz, 2.5MB Cache (10 Core) - (16)16GB

RDIMM - (4)500GB HDD - (2)Power Cord (C13-C14)

 

But with the new v4 chips out, it's a new ball game!

 

Drew Rosen

vfxnow.com

 


0 Plus One's     0 Comments  
   

Response from Rob Tomson @ May 25, 2016, 12:47 p.m.

I've been looking at Dell's FC430 blades. You can cram 8 of them in 2U of rack space with the same specs as what you posted. 

For your processor, take a look at the Intel E5-2650v4 instead of the E5-2670v3. Yes, it's .1Ghz slower but it's $436 cheaper, has the same core count, uses less power, and allows for faster memory speeds (2400MHz vs 2133MHz).

Let us know how it goes!

 

Rob


0 Plus One's     0 Comments  
   

Response from Kareem.Yassih @ May 25, 2016, 12:30 a.m.

Hey Brent,

I am sure many on this list can sympathize with the challenges your about to face having done this exercise many times myself. Bjorn is absolutly right ! if you have the luxury to do so, profile the shows needs first and then find the right solution that fits those needs...

I usually start with what is the budget ? ( Can be a blessing and/or a curse ! ) If your buying 1 node GO for the biggest SOB you can afford install Ganglia on that bad boy and sit back and watch utilization graphs ! :-)

If your looking at purchasing > 1 render node then eliminating the bells and whistles that are not needed can reduce your costs significantly when it comes to render farms. From experience Vendors never throw in anything for "Free" everything has a cost so try and breakup your item list into $$ figures and ask yourself do I really need that ? 

Take the time to profile your shows needs : If you are able to run some test renders to profile a shows actual usage it goes a long way to narrowing your field of requirements.

  • how well do your renders utilize cores (threading)
  • How much memory does the render need ? ( always add a buffer ) especially if the next question is not that good !
  • Optimization culture ? Historically does the studio focus on optimization of renders ? nothing more frustrating then crunch time blowouts !
  • Render Expectations ? Is the expectation all renders to finish overnight or can they cook ? ( usually a combination of both )
  • Electricity and Hosting costs will determine if you should go the low power wattage / double density route.

In today's render farm climate we are spoiled for choice .. If Dell is your preferred supplier then that will help narrow your focus to Dell offerings 

I really hope this helps a little ... I really hate leaving such open ended advice .. I wish i could say BUY X and you should be good as gold but unfortunately this IMHO is not one of those cases ! 

Good luck 


0 Plus One's     0 Comments  
   

Response from Bjorn Leffler @ May 24, 2016, 11:08 p.m.

Before you buy something, do you know what you actually need?

1. How many cores do your renders typically use?

2. How much ram do your renders typically use?

3. How much IO do you need? Do you make heavy use of local disk? Do you store everything on a central file server.

4. What is the licensing scheme for your software? Some software is licensed by machine, which is great if you have very large machines. Others are licensed by process. It might be cheaper to buy more smaller machines, than fewer larger machines.

 

As for the vendor's suggestion, it sure looks overkill. And expensive.

5. Do you need 128G of ram and 16 physical cores?

6. I'd say SAS drives are out of fashion these days. SATA is cheaper. SSD has more iops. Combine the two for better performance and price than SAS.

7. Two SAS drives? How much do you use local disk on the render nodes.

8. Do you really need these features on a render node? Ask yourself what would happen if a machine died. Would you just restart the render on another node? 

- Physical RAID controller. Both Linux and Windows can do software raid these days.

- Dual hot swap power supplies. Do you have multiple power circuits?

- Extra 1G network card.

- DVD R/W.

- Windows license.

- 24/7 support. If a machine breaks, is it OK to wait a day or two for spare parts?

 


0 Plus One's     0 Comments  
   

Response from Brent Slater @ May 24, 2016, 10:09 p.m.

Just to add to that.

 

We are currently only using the farmf for Maya & V-ray rendering

 

Cheers


0 Plus One's     1 Comments