Sponsors
Sponsor Products
zfs record sizes for vfx work flows and other zfs optimisations
posted by Ben De Luca  on Jan. 1, 2018, 7:20 p.m. (2 years, 6 months, 13 days ago)
9 Responses     0 Plus One's     0 Comments  
Hi, I wondered if any one had done the RND on record size for VFX workflows? I was looking at optimising one particular workflow with backups and virtual tapes and record size was hugely significant. I am guessing it will be for vfx work flows also, but wondered if any one had actually RND'd it.


Thread Tags:
  discuss-at-studiosysadmins 

Response from Ben De Luca @ Jan. 4, 2018, 5:55 a.m.
in my mind experiments, I think sync always with fast SSD would be a performance bonus
On 3 January 2018 at 23:19, greg whynott <greg.whynott@gmail.com> wrote:
any data in the ARC/L2ARC should already be on stable storage, did you mean ZIL?

We have Zeus drives to "backup" the ZIL, in the event of a power fail, the FS remains consistent and any pending writes happen when the system comes back up.

I was pondering if block size "matters" as much if all your writes are being cached, collated then laid down to disk in a disk friendly manner (when sync is set to always).
I'd not use sync=always if we didn't have the Zeus drives, that would be cowboyish for sure. :)

-greg








On Wed, Jan 3, 2018 at 4:53 PM, Brian Krusic <brian@krusic.com> wrote:
Hi,
Well Ive disabled sync as in my experience it's not needed. I imagine that you are cringing about now :)
However Ive tested my servers live by unplugging power to actually see what happens and playing dumb about it. Of course during times of non delivery and at the beginning of a shows.
I also have replicated hosts and test disaster every few months w/o telling any one about it.
ZFS is COW (copy on write) so the only thing lost during this condition is data in ARC that hasn't committed to disk.
I believe one can loose the FS only if sufficient disks in RAIDZ# are not avalyaibe.
Ive even run them 100% w/o reliability issues or too much slow down in IO.
At any rate to your question, I have no idea.
- Brian
Every day is a gift Thats why they call it the present"
On Jan 3, 2018, at 4:21 PM, greg whynott <greg.whynott@gmail.com> wrote:
Thank you -

I wonder how much the different sizes would contribute when you have a sufficient amount of safe ZIL configured with sync=always (asynchronous) set..

-g


On Wed, Jan 3, 2018 at 8:07 AM, Brian Krusic <brian@krusic.com> wrote:
Hi,
Ive kept it default with my VMs which range from DB, web, mail to lic servers. Ive read that 8K sizes are good for EXSI and 4K for KVM but I have't run any tests.
The 1M runs very well in VFX workflows.
- Brian
Every day is a gift Thats why they call it the present"
On Jan 2, 2018, at 10:13 PM, greg whynott <greg.whynott@gmail.com> wrote:
Hi Brian and Dan, are you using ZFS for VM storage with those settings too or are you speaking VFX pipeline related workloads? I assume you are addressing the original question but just curious if that is the default you use.

On Tue, Jan 2, 2018 at 8:39 PM, Dan Mons <dmons@cuttingedge.com.au> wrote:
On 3 January 2018 at 04:38, Brian Krusic <brian@krusic.com> wrote:
Ive had very good results with 1M record size and 1M NFS r/w size.

Ditto.

-Dan

Thisemailis confidential and solely for the use of the intended recipient.If you have received thisemailin error please notify the author and delete it immediately. Thisemailis not to be distributed without the author's written consent.Unauthorised forwarding, printing, copying or use is strictly prohibited and may be a breach of copyright. Any views expressed in thisemailare those of the individual sender unless specifically stated to be the views ofCutting EdgePost Pty Ltd (Cutting Edge). Although thisemailhas been sent in the belief that it is virus-free, it is the responsibility of the recipient to ensure that it is virus free. No responsibility is accepted byCutting Edgefor any loss or damage arising in any way from receipt or use of thisemail. Thisemailmay contain legally privileged information and privilege is not waived if you have received thisemailin error.

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Greg Whynott @ Jan. 3, 2018, 5:25 p.m.
any data in the ARC/L2ARC should already be on stable storage, did you mean ZIL?

We have Zeus drives to "backup" the ZIL, in the event of a power fail, the FS remains consistent and any pending writes happen when the system comes back up.

I was pondering if block size "matters" as much if all your writes are being cached, collated then laid down to disk in a disk friendly manner (when sync is set to always).
I'd not use sync=always if we didn't have the Zeus drives, that would be cowboyish for sure. :)

-greg







On Wed, Jan 3, 2018 at 4:53 PM, Brian Krusic <brian@krusic.com> wrote:
Hi,
Well Ive disabled sync as in my experience it's not needed. I imagine that you are cringing about now :)
However Ive tested my servers live by unplugging power to actually see what happens and playing dumb about it. Of course during times of non delivery and at the beginning of a shows.
I also have replicated hosts and test disaster every few months w/o telling any one about it.
ZFS is COW (copy on write) so the only thing lost during this condition is data in ARC that hasn't committed to disk.
I believe one can loose the FS only if sufficient disks in RAIDZ# are not avalyaibe.
Ive even run them 100% w/o reliability issues or too much slow down in IO.
At any rate to your question, I have no idea.
- Brian
Every day is a gift Thats why they call it the present"
On Jan 3, 2018, at 4:21 PM, greg whynott <greg.whynott@gmail.com> wrote:
Thank you -

I wonder how much the different sizes would contribute when you have a sufficient amount of safe ZIL configured with sync=always (asynchronous) set..

-g


On Wed, Jan 3, 2018 at 8:07 AM, Brian Krusic <brian@krusic.com> wrote:
Hi,
Ive kept it default with my VMs which range from DB, web, mail to lic servers. Ive read that 8K sizes are good for EXSI and 4K for KVM but I have't run any tests.
The 1M runs very well in VFX workflows.
- Brian
Every day is a gift Thats why they call it the present"
On Jan 2, 2018, at 10:13 PM, greg whynott <greg.whynott@gmail.com> wrote:
Hi Brian and Dan, are you using ZFS for VM storage with those settings too or are you speaking VFX pipeline related workloads? I assume you are addressing the original question but just curious if that is the default you use.

On Tue, Jan 2, 2018 at 8:39 PM, Dan Mons <dmons@cuttingedge.com.au> wrote:
On 3 January 2018 at 04:38, Brian Krusic <brian@krusic.com> wrote:
Ive had very good results with 1M record size and 1M NFS r/w size.

Ditto.

-Dan

Thisemailis confidential and solely for the use of the intended recipient.If you have received thisemailin error please notify the author and delete it immediately. Thisemailis not to be distributed without the author's written consent.Unauthorised forwarding, printing, copying or use is strictly prohibited and may be a breach of copyright. Any views expressed in thisemailare those of the individual sender unless specifically stated to be the views ofCutting EdgePost Pty Ltd (Cutting Edge). Although thisemailhas been sent in the belief that it is virus-free, it is the responsibility of the recipient to ensure that it is virus free. No responsibility is accepted byCutting Edgefor any loss or damage arising in any way from receipt or use of thisemail. Thisemailmay contain legally privileged information and privilege is not waived if you have received thisemailin error.

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Greg Whynott @ Jan. 3, 2018, 4:25 p.m.
Thank you -

I wonder how much the different sizes would contribute when you have a sufficient amount of safe ZIL configured with sync=always (asynchronous) set..

-g


On Wed, Jan 3, 2018 at 8:07 AM, Brian Krusic <brian@krusic.com> wrote:
Hi,
Ive kept it default with my VMs which range from DB, web, mail to lic servers. Ive read that 8K sizes are good for EXSI and 4K for KVM but I have't run any tests.
The 1M runs very well in VFX workflows.
- Brian
Every day is a gift Thats why they call it the present"
On Jan 2, 2018, at 10:13 PM, greg whynott <greg.whynott@gmail.com> wrote:
Hi Brian and Dan, are you using ZFS for VM storage with those settings too or are you speaking VFX pipeline related workloads? I assume you are addressing the original question but just curious if that is the default you use.

On Tue, Jan 2, 2018 at 8:39 PM, Dan Mons <dmons@cuttingedge.com.au> wrote:
On 3 January 2018 at 04:38, Brian Krusic <brian@krusic.com> wrote:
Ive had very good results with 1M record size and 1M NFS r/w size.

Ditto.

-Dan

Thisemailis confidential and solely for the use of the intended recipient.If you have received thisemailin error please notify the author and delete it immediately. Thisemailis not to be distributed without the author's written consent.Unauthorised forwarding, printing, copying or use is strictly prohibited and may be a breach of copyright. Any views expressed in thisemailare those of the individual sender unless specifically stated to be the views ofCutting EdgePost Pty Ltd (Cutting Edge). Although thisemailhas been sent in the belief that it is virus-free, it is the responsibility of the recipient to ensure that it is virus free. No responsibility is accepted byCutting Edgefor any loss or damage arising in any way from receipt or use of thisemail. Thisemailmay contain legally privileged information and privilege is not waived if you have received thisemailin error.

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Ben De Luca @ Jan. 3, 2018, 6 a.m.
My limited non real world testing suggested 1M is much better multiple times performance increase for reads that are not in the cache.
On 3 January 2018 at 06:06, Dan Mons <dmons@cuttingedge.com.au> wrote:
On 3 January 2018 at 13:13, greg whynott <greg.whynott@gmail.com> wrote:
Hi Brian and Dan, are you using ZFS for VM storage with those settings too or are you speaking VFX pipeline related workloads?

Both, although our VM workloads are very light (all our high CPU/IO workloads are on bare metal).
-Dan


Thisemailis confidential and solely for the use of the intended recipient.If you have received thisemailin error please notify the author and delete it immediately. Thisemailis not to be distributed without the author's written consent.Unauthorised forwarding, printing, copying or use is strictly prohibited and may be a breach of copyright. Any views expressed in thisemailare those of the individual sender unless specifically stated to be the views ofCutting EdgePost Pty Ltd (Cutting Edge). Although thisemailhas been sent in the belief that it is virus-free, it is the responsibility of the recipient to ensure that it is virus free. No responsibility is accepted byCutting Edgefor any loss or damage arising in any way from receipt or use of thisemail. Thisemailmay contain legally privileged information and privilege is not waived if you have received thisemailin error.



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Greg Whynott @ Jan. 2, 2018, 10:15 p.m.
Hi Brian and Dan, are you using ZFS for VM storage with those settings too or are you speaking VFX pipeline related workloads? I assume you are addressing the original question but just curious if that is the default you use.

On Tue, Jan 2, 2018 at 8:39 PM, Dan Mons <dmons@cuttingedge.com.au> wrote:
On 3 January 2018 at 04:38, Brian Krusic <brian@krusic.com> wrote:
Ive had very good results with 1M record size and 1M NFS r/w size.

Ditto.

-Dan

Thisemailis confidential and solely for the use of the intended recipient.If you have received thisemailin error please notify the author and delete it immediately. Thisemailis not to be distributed without the author's written consent.Unauthorised forwarding, printing, copying or use is strictly prohibited and may be a breach of copyright. Any views expressed in thisemailare those of the individual sender unless specifically stated to be the views ofCutting EdgePost Pty Ltd (Cutting Edge). Although thisemailhas been sent in the belief that it is virus-free, it is the responsibility of the recipient to ensure that it is virus free. No responsibility is accepted byCutting Edgefor any loss or damage arising in any way from receipt or use of thisemail. Thisemailmay contain legally privileged information and privilege is not waived if you have received thisemailin error.



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Greg Whynott @ Jan. 2, 2018, 1:45 p.m.
https://www.youtube.com/watch?v=SJB1cJfcjYI

might be worth a watch Ben. Its not the video I was looking for but has some good points/considerations. I'll keep looking, there was a video by one of the Sun devs talking explicitly about record sizes tuning..

0g



On Tue, Jan 2, 2018 at 1:38 PM, Brian Krusic <brian@krusic.com> wrote:
Hi,
Ive had very good results with 1M record size and 1M NFS r/w size.
Well, I mean as good as it can be. All in all nothing is ever really fast enough :)
- Brian
Every day is a gift Thats why they call it the present"
On Jan 2, 2018, at 1:35 PM, greg whynott <greg.whynott@gmail.com> wrote:
default/choose.. words. ;)

On Tue, Jan 2, 2018 at 12:05 PM, Ben De Luca <bdeluca@gmail.com> wrote:
zfs doesnt choose, its set on the file system and defaults to 128k
On 2 January 2018 at 17:24, greg whynott <greg.whynott@gmail.com> wrote:
No R&D but for what it is worth...

Do your own IO profile. What you guys are doing in your shop may not be the same elsewhere and certainly not everyone is in the same boat or uses storage verbatim. Best bet is to monitor for a period of time and build your shops own profile.

It is also a good idea if possible to build FS's for different use cases. Example, a FS for your database servers might have a record size of 8k, whereas a FS with big files on it (over 1 megabyte) may be set at 128k.

My final thought would be in most cases with VFX related work, you are dealing with big files, a 128k record size may be the best bet, or let ZFS choose itself.

greg





On Mon, Jan 1, 2018 at 7:16 PM, Ben De Luca <bdeluca@gmail.com> wrote:
Hi, I wondered if any one had done the RND on record size for VFX workflows? I was looking at optimising one particular workflow with backups and virtual tapes and record size was hugely significant. I am guessing it will be for vfx work flows also, but wondered if any one had actually RND'd it.


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Greg Whynott @ Jan. 2, 2018, 1:40 p.m.
default/choose.. words. ;)

On Tue, Jan 2, 2018 at 12:05 PM, Ben De Luca <bdeluca@gmail.com> wrote:
zfs doesnt choose, its set on the file system and defaults to 128k
On 2 January 2018 at 17:24, greg whynott <greg.whynott@gmail.com> wrote:
No R&D but for what it is worth...

Do your own IO profile. What you guys are doing in your shop may not be the same elsewhere and certainly not everyone is in the same boat or uses storage verbatim. Best bet is to monitor for a period of time and build your shops own profile.

It is also a good idea if possible to build FS's for different use cases. Example, a FS for your database servers might have a record size of 8k, whereas a FS with big files on it (over 1 megabyte) may be set at 128k.

My final thought would be in most cases with VFX related work, you are dealing with big files, a 128k record size may be the best bet, or let ZFS choose itself.

greg





On Mon, Jan 1, 2018 at 7:16 PM, Ben De Luca <bdeluca@gmail.com> wrote:
Hi, I wondered if any one had done the RND on record size for VFX workflows? I was looking at optimising one particular workflow with backups and virtual tapes and record size was hugely significant. I am guessing it will be for vfx work flows also, but wondered if any one had actually RND'd it.


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Ben De Luca @ Jan. 2, 2018, 12:10 p.m.
zfs doesnt choose, its set on the file system and defaults to 128k
On 2 January 2018 at 17:24, greg whynott <greg.whynott@gmail.com> wrote:
No R&D but for what it is worth...

Do your own IO profile. What you guys are doing in your shop may not be the same elsewhere and certainly not everyone is in the same boat or uses storage verbatim. Best bet is to monitor for a period of time and build your shops own profile.

It is also a good idea if possible to build FS's for different use cases. Example, a FS for your database servers might have a record size of 8k, whereas a FS with big files on it (over 1 megabyte) may be set at 128k.

My final thought would be in most cases with VFX related work, you are dealing with big files, a 128k record size may be the best bet, or let ZFS choose itself.

greg





On Mon, Jan 1, 2018 at 7:16 PM, Ben De Luca <bdeluca@gmail.com> wrote:
Hi, I wondered if any one had done the RND on record size for VFX workflows? I was looking at optimising one particular workflow with backups and virtual tapes and record size was hugely significant. I am guessing it will be for vfx work flows also, but wondered if any one had actually RND'd it.


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Greg Whynott @ Jan. 2, 2018, 11:25 a.m.
No R&D but for what it is worth...

Do your own IO profile. What you guys are doing in your shop may not be the same elsewhere and certainly not everyone is in the same boat or uses storage verbatim. Best bet is to monitor for a period of time and build your shops own profile.

It is also a good idea if possible to build FS's for different use cases. Example, a FS for your database servers might have a record size of 8k, whereas a FS with big files on it (over 1 megabyte) may be set at 128k.

My final thought would be in most cases with VFX related work, you are dealing with big files, a 128k record size may be the best bet, or let ZFS choose itself.

greg





On Mon, Jan 1, 2018 at 7:16 PM, Ben De Luca <bdeluca@gmail.com> wrote:
Hi, I wondered if any one had done the RND on record size for VFX workflows? I was looking at optimising one particular workflow with backups and virtual tapes and record size was hugely significant. I am guessing it will be for vfx work flows also, but wondered if any one had actually RND'd it.


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments