Sponsors
Sponsor Products
Isilon Continuous Availability bug + Disabling SMB3 on Isilon
posted by Isaac Guenard  on July 13, 2017, 1:50 p.m. (13 days ago)
6 Responses     0 Plus One's     0 Comments  
Ugh. Very well timed warning. Thanks for the heads up Julian.


Isaac Guenard
Head of Pipeline
1751 Rue Richardson, 4400 Montreal
C: +1 514 746-9078


On Thu, Jul 13, 2017 at 1:34 PM, Zorion Terrell <Zorion.Terrell@dhxmedia.com> wrote:

This is good to know as we are an SMB shop planning the upgrade to OneFS 8

Any idea if its been patched in the OneFS 8.1 variant running on the Gen6 hardware?

Zorion Terrell

IT Manager | DHX Studios

e: zorion.terrell@dhxmedia.com

t: 604-684-2363 | m: 604-562-5148

380 West 5th Ave

Vancouver, BC Canada V5Y 1J5

Email Signature_DHX_Media

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of julian firminger
Sent: July 11, 2017 2:53 AM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: [SSA-Discuss] Isilon Continuous Availability bug + Disabling SMB3 on Isilon

Isilon-peeps,

We've been heavily effected by what appears to be a bug in Isilon OFSv8.0.0.x implementation of SMB3-CA extension where Win2012R2 and later servers will fail to release ephemeral socket connections to the Isilon, eventually leading to a network stack collapse on the server. - I will post more on this if/when we know it.

Out of this came some useful tooling around handling SMB3 that I though I would share:

To disable SMB3 multichannel support:

# isi smb settings global modify --support-multichannel=no

This will preserve the SMB3 dialect negotiation, authentication, CA and piping but disable support for cli/serv NIC RSS. Persisting sessions will continue to be multichannel until dropped / re-established.

To prevent Continuous Availability session setup:

# isi_for_array s /usr/likewise/bin/lwsm stop lwwit

This will disable the Isilon's file-witness service (on each node). Clients will still negotiate up to SMB3 dialect and if CA capable will attempt a CA session setup but this will fail when the client cannot find a file-witness. This will effectively leave CA turned on (on enabled shares) but prevent clients from using it. This will force reset any persisting SMB3-CA sessions.

To disable SMB3 entirely:

# isi_gconfig registry.Services.lwio.Parameters.Drivers.srv.MaxSmb2DialectVersion=2.10

This will stop the Isilon offering SMB3 dialect on session setup so the client side will have only 2.1, 2.0 or 1.0 to choose from. Persisting sessions will continue to be SMB3 until dropped / re-established.

The last two are USE AT OWN PERIL. And I have been advised that these changes should be done by an EMC engineer. Especially the lwwit stop. This can have drastic and immediate effects across all SMB sessions.

Julian Firminger

Senior Systems Administrator

United Broadcast Facilities

Amsterdam, The Netherlands


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


Thread Tags:
  discuss-at-studiosysadmins 

Response from Ben De Luca @ July 21, 2017, 7:30 a.m.
for example.id est I.E.


On 21 July 2017 at 09:48, julian firminger <justdigitalfilm@gmail.com> wrote:
IE?
On 21 Jul. 2017 06:48, "Ben De Luca" <bdeluca@gmail.com> wrote:
Continuous Availability? IE Working?



On 13 July 2017 at 21:01, julian firminger <justdigitalfilm@gmail.com> wrote:
Gents. I will fwd you the KBs as soon as I can. Do NOT go to SMB3-CA for heavy performance servers on 2012R2 or 2016 under OFS8.0.0.x It will break badly.
On 13 Jul. 2017 19:49, "Isaac Guenard" <isaac.guenard@primefocusworld.com> wrote:
Ugh. Very well timed warning. Thanks for the heads up Julian.


Isaac Guenard
Head of Pipeline
1751 Rue Richardson, 4400 Montreal
C: +1 514 746-9078


On Thu, Jul 13, 2017 at 1:34 PM, Zorion Terrell <Zorion.Terrell@dhxmedia.com> wrote:

This is good to know as we are an SMB shop planning the upgrade to OneFS 8

Any idea if its been patched in the OneFS 8.1 variant running on the Gen6 hardware?

Zorion Terrell

IT Manager | DHX Studios

e: zorion.terrell@dhxmedia.com

t: 604-684-2363 | m: 604-562-5148

380 West 5th Ave

Vancouver, BC Canada V5Y 1J5

Email Signature_DHX_Media

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of julian firminger
Sent: July 11, 2017 2:53 AM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: [SSA-Discuss] Isilon Continuous Availability bug + Disabling SMB3 on Isilon

Isilon-peeps,

We've been heavily effected by what appears to be a bug in Isilon OFSv8.0.0.x implementation of SMB3-CA extension where Win2012R2 and later servers will fail to release ephemeral socket connections to the Isilon, eventually leading to a network stack collapse on the server. - I will post more on this if/when we know it.

Out of this came some useful tooling around handling SMB3 that I though I would share:

To disable SMB3 multichannel support:

# isi smb settings global modify --support-multichannel=no

This will preserve the SMB3 dialect negotiation, authentication, CA and piping but disable support for cli/serv NIC RSS. Persisting sessions will continue to be multichannel until dropped / re-established.

To prevent Continuous Availability session setup:

# isi_for_array s /usr/likewise/bin/lwsm stop lwwit

This will disable the Isilon's file-witness service (on each node). Clients will still negotiate up to SMB3 dialect and if CA capable will attempt a CA session setup but this will fail when the client cannot find a file-witness. This will effectively leave CA turned on (on enabled shares) but prevent clients from using it. This will force reset any persisting SMB3-CA sessions.

To disable SMB3 entirely:

# isi_gconfig registry.Services.lwio.Parameters.Drivers.srv.MaxSmb2DialectVersion=2.10

This will stop the Isilon offering SMB3 dialect on session setup so the client side will have only 2.1, 2.0 or 1.0 to choose from. Persisting sessions will continue to be SMB3 until dropped / re-established.

The last two are USE AT OWN PERIL. And I have been advised that these changes should be done by an EMC engineer. Especially the lwwit stop. This can have drastic and immediate effects across all SMB sessions.

Julian Firminger

Senior Systems Administrator

United Broadcast Facilities

Amsterdam, The Netherlands


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Julian Firminger @ July 21, 2017, 2:50 a.m.
IE?
On 21 Jul. 2017 06:48, "Ben De Luca" <bdeluca@gmail.com> wrote:
Continuous Availability? IE Working?



On 13 July 2017 at 21:01, julian firminger <justdigitalfilm@gmail.com> wrote:
Gents. I will fwd you the KBs as soon as I can. Do NOT go to SMB3-CA for heavy performance servers on 2012R2 or 2016 under OFS8.0.0.x It will break badly.
On 13 Jul. 2017 19:49, "Isaac Guenard" <isaac.guenard@primefocusworld.com> wrote:
Ugh. Very well timed warning. Thanks for the heads up Julian.


Isaac Guenard
Head of Pipeline
1751 Rue Richardson, 4400 Montreal
C: +1 514 746-9078


On Thu, Jul 13, 2017 at 1:34 PM, Zorion Terrell <Zorion.Terrell@dhxmedia.com> wrote:

This is good to know as we are an SMB shop planning the upgrade to OneFS 8

Any idea if its been patched in the OneFS 8.1 variant running on the Gen6 hardware?

Zorion Terrell

IT Manager | DHX Studios

e: zorion.terrell@dhxmedia.com

t: 604-684-2363 | m: 604-562-5148

380 West 5th Ave

Vancouver, BC Canada V5Y 1J5

Email Signature_DHX_Media

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of julian firminger
Sent: July 11, 2017 2:53 AM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: [SSA-Discuss] Isilon Continuous Availability bug + Disabling SMB3 on Isilon

Isilon-peeps,

We've been heavily effected by what appears to be a bug in Isilon OFSv8.0.0.x implementation of SMB3-CA extension where Win2012R2 and later servers will fail to release ephemeral socket connections to the Isilon, eventually leading to a network stack collapse on the server. - I will post more on this if/when we know it.

Out of this came some useful tooling around handling SMB3 that I though I would share:

To disable SMB3 multichannel support:

# isi smb settings global modify --support-multichannel=no

This will preserve the SMB3 dialect negotiation, authentication, CA and piping but disable support for cli/serv NIC RSS. Persisting sessions will continue to be multichannel until dropped / re-established.

To prevent Continuous Availability session setup:

# isi_for_array s /usr/likewise/bin/lwsm stop lwwit

This will disable the Isilon's file-witness service (on each node). Clients will still negotiate up to SMB3 dialect and if CA capable will attempt a CA session setup but this will fail when the client cannot find a file-witness. This will effectively leave CA turned on (on enabled shares) but prevent clients from using it. This will force reset any persisting SMB3-CA sessions.

To disable SMB3 entirely:

# isi_gconfig registry.Services.lwio.Parameters.Drivers.srv.MaxSmb2DialectVersion=2.10

This will stop the Isilon offering SMB3 dialect on session setup so the client side will have only 2.1, 2.0 or 1.0 to choose from. Persisting sessions will continue to be SMB3 until dropped / re-established.

The last two are USE AT OWN PERIL. And I have been advised that these changes should be done by an EMC engineer. Especially the lwwit stop. This can have drastic and immediate effects across all SMB sessions.

Julian Firminger

Senior Systems Administrator

United Broadcast Facilities

Amsterdam, The Netherlands


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

0 Plus One's     0 Comments  
   

Response from Ben De Luca @ July 21, 2017, 12:50 a.m.
Continuous Availability? IE Working?



On 13 July 2017 at 21:01, julian firminger <justdigitalfilm@gmail.com> wrote:
Gents. I will fwd you the KBs as soon as I can. Do NOT go to SMB3-CA for heavy performance servers on 2012R2 or 2016 under OFS8.0.0.x It will break badly.
On 13 Jul. 2017 19:49, "Isaac Guenard" <isaac.guenard@primefocusworld.com> wrote:
Ugh. Very well timed warning. Thanks for the heads up Julian.


Isaac Guenard
Head of Pipeline
1751 Rue Richardson, 4400 Montreal
C: +1 514 746-9078


On Thu, Jul 13, 2017 at 1:34 PM, Zorion Terrell <Zorion.Terrell@dhxmedia.com> wrote:

This is good to know as we are an SMB shop planning the upgrade to OneFS 8

Any idea if its been patched in the OneFS 8.1 variant running on the Gen6 hardware?

Zorion Terrell

IT Manager | DHX Studios

e: zorion.terrell@dhxmedia.com

t: 604-684-2363 | m: 604-562-5148

380 West 5th Ave

Vancouver, BC Canada V5Y 1J5

Email Signature_DHX_Media

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of julian firminger
Sent: July 11, 2017 2:53 AM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: [SSA-Discuss] Isilon Continuous Availability bug + Disabling SMB3 on Isilon

Isilon-peeps,

We've been heavily effected by what appears to be a bug in Isilon OFSv8.0.0.x implementation of SMB3-CA extension where Win2012R2 and later servers will fail to release ephemeral socket connections to the Isilon, eventually leading to a network stack collapse on the server. - I will post more on this if/when we know it.

Out of this came some useful tooling around handling SMB3 that I though I would share:

To disable SMB3 multichannel support:

# isi smb settings global modify --support-multichannel=no

This will preserve the SMB3 dialect negotiation, authentication, CA and piping but disable support for cli/serv NIC RSS. Persisting sessions will continue to be multichannel until dropped / re-established.

To prevent Continuous Availability session setup:

# isi_for_array s /usr/likewise/bin/lwsm stop lwwit

This will disable the Isilon's file-witness service (on each node). Clients will still negotiate up to SMB3 dialect and if CA capable will attempt a CA session setup but this will fail when the client cannot find a file-witness. This will effectively leave CA turned on (on enabled shares) but prevent clients from using it. This will force reset any persisting SMB3-CA sessions.

To disable SMB3 entirely:

# isi_gconfig registry.Services.lwio.Parameters.Drivers.srv.MaxSmb2DialectVersion=2.10

This will stop the Isilon offering SMB3 dialect on session setup so the client side will have only 2.1, 2.0 or 1.0 to choose from. Persisting sessions will continue to be SMB3 until dropped / re-established.

The last two are USE AT OWN PERIL. And I have been advised that these changes should be done by an EMC engineer. Especially the lwwit stop. This can have drastic and immediate effects across all SMB sessions.

Julian Firminger

Senior Systems Administrator

United Broadcast Facilities

Amsterdam, The Netherlands


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Brett Dellandre @ July 18, 2017, 5:29 p.m.

Following up here, can you reference a particular KB article? I do see one: KB000486147 but its more about resolution of an SPN outside the system zone and is addressed in 8.0.0.2 

SMB3.0 CA should be very selectively implemented. There is a known performance hit (which we are addressing in future OneFS releases) and long file open handles <GB files do put a lot of pressure on the lwio (Isilon SMB) process to be able to hanlde the open file handles and backend witnesses so we typically don't reccomened it unless your Isilon CPU's can handle it.

Its a cool feature and it has its use cases, home directories, corp file shares, Deadline repository (!!) but not so much with a projects/assets type share (because of the large files). 


0 Plus One's     0 Comments  
   

Response from Brett Dellandre @ July 18, 2017, 5:29 p.m.

Following up here, can you reference a particular KB article? I do see one: KB000486147 but its more about resolution of an SPN outside the system zone and is addressed in 8.0.0.2 

SMB3.0 CA should be very selectively implemented. There is a known performance hit (which we are addressing in future OneFS releases) and long file open handles <GB files do put a lot of pressure on the lwio (Isilon SMB) process to be able to hanlde the open file handles and backend witnesses so we typically don't reccomened it unless your Isilon CPU's can handle it.

Its a cool feature and it has its use cases, home directories, corp file shares, Deadline repository (!!) but not so much with a projects/assets type share (because of the large files). 


0 Plus One's     0 Comments  
   

Response from Julian Firminger @ July 13, 2017, 2:05 p.m.
Gents. I will fwd you the KBs as soon as I can. Do NOT go to SMB3-CA for heavy performance servers on 2012R2 or 2016 under OFS8.0.0.x It will break badly.
On 13 Jul. 2017 19:49, "Isaac Guenard" <isaac.guenard@primefocusworld.com> wrote:
Ugh. Very well timed warning. Thanks for the heads up Julian.


Isaac Guenard
Head of Pipeline
1751 Rue Richardson, 4400 Montreal
C: +1 514 746-9078


On Thu, Jul 13, 2017 at 1:34 PM, Zorion Terrell <Zorion.Terrell@dhxmedia.com> wrote:

This is good to know as we are an SMB shop planning the upgrade to OneFS 8

Any idea if its been patched in the OneFS 8.1 variant running on the Gen6 hardware?

Zorion Terrell

IT Manager | DHX Studios

e: zorion.terrell@dhxmedia.com

t: 604-684-2363 | m: 604-562-5148

380 West 5th Ave

Vancouver, BC Canada V5Y 1J5

Email Signature_DHX_Media

From: studiosysadmins-discuss-bounces@studiosysadmins.com [mailto:studiosysadmins-discuss-bounces@studiosysadmins.com] On Behalf Of julian firminger
Sent: July 11, 2017 2:53 AM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: [SSA-Discuss] Isilon Continuous Availability bug + Disabling SMB3 on Isilon

Isilon-peeps,

We've been heavily effected by what appears to be a bug in Isilon OFSv8.0.0.x implementation of SMB3-CA extension where Win2012R2 and later servers will fail to release ephemeral socket connections to the Isilon, eventually leading to a network stack collapse on the server. - I will post more on this if/when we know it.

Out of this came some useful tooling around handling SMB3 that I though I would share:

To disable SMB3 multichannel support:

# isi smb settings global modify --support-multichannel=no

This will preserve the SMB3 dialect negotiation, authentication, CA and piping but disable support for cli/serv NIC RSS. Persisting sessions will continue to be multichannel until dropped / re-established.

To prevent Continuous Availability session setup:

# isi_for_array s /usr/likewise/bin/lwsm stop lwwit

This will disable the Isilon's file-witness service (on each node). Clients will still negotiate up to SMB3 dialect and if CA capable will attempt a CA session setup but this will fail when the client cannot find a file-witness. This will effectively leave CA turned on (on enabled shares) but prevent clients from using it. This will force reset any persisting SMB3-CA sessions.

To disable SMB3 entirely:

# isi_gconfig registry.Services.lwio.Parameters.Drivers.srv.MaxSmb2DialectVersion=2.10

This will stop the Isilon offering SMB3 dialect on session setup so the client side will have only 2.1, 2.0 or 1.0 to choose from. Persisting sessions will continue to be SMB3 until dropped / re-established.

The last two are USE AT OWN PERIL. And I have been advised that these changes should be done by an EMC engineer. Especially the lwwit stop. This can have drastic and immediate effects across all SMB sessions.

Julian Firminger

Senior Systems Administrator

United Broadcast Facilities

Amsterdam, The Netherlands


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

0 Plus One's     0 Comments