Sponsors
Sponsor Products
gridFTP
posted by Greg Whynott  on Feb. 15, 2017, 11:10 p.m. (6 months, 6 days ago)
15 Responses     0 Plus One's     0 Comments  
Thought this was worth sharing, at a geek level.

Black line is from server to DMZ VM, red line is from DMZ VM to Korea.

The VM is on a large array and the server is a HPC node with a single 7200 RPM drive, working on a job at the same time. Likely the bottle neck here is the drive/system itself, but still, its amusing it took less time to transfer the file to the other side of earth.


side note, if you have to compress large files or sets, check out pigz if you are not aware of it. will use every core, even on single files. it splits the file into chunks equal to the core count and works on them in parallel. Its so fast..






take good care,

_g



Thread Tags:
  discuss-at-studiosysadmins 

Response from Greg Whynott @ Feb. 21, 2017, 1:50 p.m.
`official` word from a Globus team member:
"hard links are read and created as normal files bythe Globus tools."

so looks like you would need to make that wrapper/pre-processor.
_g


On Sat, Feb 18, 2017 at 9:38 PM, greg whynott <greg.whynott@gmail.com> wrote:
Tried it and it does copy 3 files. ;( I asked on the user list, see if anyone has some input there who is more familiarwith the kit.
Writing a pre-flight staging wrapper might work, have a function find the hard links, generate a list of files to xfer, feed that to gridftp (or whatever you are using), then create the hardlinks on the other end. gridftp allows you to send commands to the other end just like you would with ssh, mkdir, ln, ls, du etc...
http://opendedup.org might have something useful, depending on the data set sizes you are working with.
Thankfully we are only sending data to our remote studios and have access on both ends to setup things. If I had to send to many autonomous 3rd party clients, I'd be thinking about Aspera. ;) Good luck with things, let me/us know what you end up doing to resolve the problem.
_g



On Thu, Feb 16, 2017 at 6:28 PM, greg whynott <greg.whynott@gmail.com> wrote:
Hi Tom,
I'll give it a go later on tonight. Just hitting the road and won't be home for a few hours yet. I would speculate it'll also copy it as an autonomous file without passing any options/methods. I would also think there is a way to handle this out on the webs, GridFTP has been in deployment for a few decades now.
_g


On Thu, Feb 16, 2017 at 6:08 PM, Tom Taylor <taylor@themill.com> wrote:

My response from support was;


The hard link gets transferred as a file. So, async transfers three files:

/toSync/img1.jpg
/toSync/additional/img1.jpg
/toSync/img2.jpg (whose contents are img1.jpg).

I just tested it myself and it created 3 files with separate inodesrather than 2 with 1 hardlink

async -N Test -x -d /tmp/tosync -r root@X:/tmp/tosync -a fair -l 30000 --dedup=hardlink -k md5



On destination server:

# ls -i img1.jpg

791455 img1.jpg

# ls -i img2.jpg

791457 img2.jpg



On original server:

# ls -i img1.jpg

1966821 img1.jpg

# ls -i img2.jpg

1966821 img2.jpg



Greg, based on the above we're using UDT/UDR for now which works but I'm interested to find out if GridFTP behaves the same.

TomTaylor
HeadofEngineering,Chicago
T+13126058900
The Mill
TheMill1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com|@millchannel|facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 4:54 PM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **

Should do if you add the --dedup=hardlink option to async ... Haven't tested this myself in a while but that's the idea...


Michelle


On 2/16/17 2:50 PM, Tom Taylor wrote:

Michelle, could you tell me if this is possible; If so, fantastic.


cp /originals/img1.jpg /toSync/img1.jpg
cp /originals/img1.jpg /toSync/additional/img1.jpg
ln /toSync/img1.jpg /toSync/img2.jpg

After syncing /toSync from one computer to another, will the resulting directories at the remote end contain two real files called img1.jpg and one hard link called img2.jpg ?





TomTaylor
HeadofEngineering,Chicago
T +13126058900
The Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 4:46 PM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **

Guys,

Hardlinks are preserved (copied from source to destination) in async when you run with the --dedup option as far as I know;

man async

--dedup[=MODE]
MODE=hardlink(default),copy Hard link/copy files with the same content

Michelle


On 2/16/17 2:41 PM, greg whynott wrote:
Hi Tom,
what do you mean when you say it "expands hard-links"? I'm trying to figure out what you mean exactly.
thanks much, greg


On Thu, Feb 16, 2017 at 1:47 PM, Tom Taylor <taylor@themill.com> wrote:

Michelle, it doesn't, it expands hard-links.


At least that is what I was told from your supportdepartment, if theyhave given me the wrong info feel free to hit me up off list.


-Tom




TomTaylor
HeadofEngineering,Chicago
T +13126058900
The Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 11:10 AM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **
Know we like to keep commercials to a minimum but Aspera sync does all of these things.
Let me know if you would like more info.
Michelle

Sent from my iPhone
On Feb 16, 2017, at 10:24 AM, Tom Taylor <taylor@themill.com> wrote:

But.... does it do hard-links? <OutlookEmoji-.png>


I've been searching high and low for something that'll let me;


1. View the status of current transfers and change/edit priority (Pause, stop, play, schedule)

2. Supports hard-links


It appears to be the golden goose of accelerated transfer, seems it's going to have to be something custom.


-TT


TomTaylor
HeadofEngineering,Chicago
T +13126058900
The
                                                          Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of greg whynott <greg.whynott@gmail.com>
Sent: Wednesday, February 15, 2017 10:07:00 PM
To: studiosysadmins-discuss
Subject: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **
Thought this was worth sharing, at a geek level.

Black line is from server to DMZ VM, red line is from DMZ VM to Korea.

The VM is on a large array and the server is a HPC node with a single 7200 RPM drive, working on a job at the same time. Likely the bottle neck here is the drive/system itself, but still, its amusing it took less time to transfer the file to the other side of earth.


side note, if you have to compress large files or sets, check out pigz if you are not aware of it. will use every core, even on single files. it splits the file into chunks equal to the core count and works on them in parallel. Its so fast..






take good care,

_g


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe




0 Plus One's     0 Comments  
   

Response from Greg Whynott @ Feb. 18, 2017, 9:40 p.m.
Tried it and it does copy 3 files. ;( I asked on the user list, see if anyone has some input there who is more familiarwith the kit.
Writing a pre-flight staging wrapper might work, have a function find the hard links, generate a list of files to xfer, feed that to gridftp (or whatever you are using), then create the hardlinks on the other end. gridftp allows you to send commands to the other end just like you would with ssh, mkdir, ln, ls, du etc...
http://opendedup.org might have something useful, depending on the data set sizes you are working with.
Thankfully we are only sending data to our remote studios and have access on both ends to setup things. If I had to send to many autonomous 3rd party clients, I'd be thinking about Aspera. ;) Good luck with things, let me/us know what you end up doing to resolve the problem.
_g



On Thu, Feb 16, 2017 at 6:28 PM, greg whynott <greg.whynott@gmail.com> wrote:
Hi Tom,
I'll give it a go later on tonight. Just hitting the road and won't be home for a few hours yet. I would speculate it'll also copy it as an autonomous file without passing any options/methods. I would also think there is a way to handle this out on the webs, GridFTP has been in deployment for a few decades now.
_g


On Thu, Feb 16, 2017 at 6:08 PM, Tom Taylor <taylor@themill.com> wrote:

My response from support was;


The hard link gets transferred as a file. So, async transfers three files:

/toSync/img1.jpg
/toSync/additional/img1.jpg
/toSync/img2.jpg (whose contents are img1.jpg).

I just tested it myself and it created 3 files with separate inodesrather than 2 with 1 hardlink

async -N Test -x -d /tmp/tosync -r root@X:/tmp/tosync -a fair -l 30000 --dedup=hardlink -k md5



On destination server:

# ls -i img1.jpg

791455 img1.jpg

# ls -i img2.jpg

791457 img2.jpg



On original server:

# ls -i img1.jpg

1966821 img1.jpg

# ls -i img2.jpg

1966821 img2.jpg



Greg, based on the above we're using UDT/UDR for now which works but I'm interested to find out if GridFTP behaves the same.

TomTaylor
HeadofEngineering,Chicago
T+13126058900
The Mill
TheMill1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com|@millchannel|facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 4:54 PM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **

Should do if you add the --dedup=hardlink option to async ... Haven't tested this myself in a while but that's the idea...


Michelle


On 2/16/17 2:50 PM, Tom Taylor wrote:

Michelle, could you tell me if this is possible; If so, fantastic.


cp /originals/img1.jpg /toSync/img1.jpg
cp /originals/img1.jpg /toSync/additional/img1.jpg
ln /toSync/img1.jpg /toSync/img2.jpg

After syncing /toSync from one computer to another, will the resulting directories at the remote end contain two real files called img1.jpg and one hard link called img2.jpg ?





TomTaylor
HeadofEngineering,Chicago
T +13126058900
The Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 4:46 PM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **

Guys,

Hardlinks are preserved (copied from source to destination) in async when you run with the --dedup option as far as I know;

man async

--dedup[=MODE]
MODE=hardlink(default),copy Hard link/copy files with the same content

Michelle


On 2/16/17 2:41 PM, greg whynott wrote:
Hi Tom,
what do you mean when you say it "expands hard-links"? I'm trying to figure out what you mean exactly.
thanks much, greg


On Thu, Feb 16, 2017 at 1:47 PM, Tom Taylor <taylor@themill.com> wrote:

Michelle, it doesn't, it expands hard-links.


At least that is what I was told from your supportdepartment, if theyhave given me the wrong info feel free to hit me up off list.


-Tom




TomTaylor
HeadofEngineering,Chicago
T +13126058900
The Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 11:10 AM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **
Know we like to keep commercials to a minimum but Aspera sync does all of these things.
Let me know if you would like more info.
Michelle

Sent from my iPhone
On Feb 16, 2017, at 10:24 AM, Tom Taylor <taylor@themill.com> wrote:

But.... does it do hard-links? <OutlookEmoji-.png>


I've been searching high and low for something that'll let me;


1. View the status of current transfers and change/edit priority (Pause, stop, play, schedule)

2. Supports hard-links


It appears to be the golden goose of accelerated transfer, seems it's going to have to be something custom.


-TT


TomTaylor
HeadofEngineering,Chicago
T +13126058900
The
                                                          Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of greg whynott <greg.whynott@gmail.com>
Sent: Wednesday, February 15, 2017 10:07:00 PM
To: studiosysadmins-discuss
Subject: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **
Thought this was worth sharing, at a geek level.

Black line is from server to DMZ VM, red line is from DMZ VM to Korea.

The VM is on a large array and the server is a HPC node with a single 7200 RPM drive, working on a job at the same time. Likely the bottle neck here is the drive/system itself, but still, its amusing it took less time to transfer the file to the other side of earth.


side note, if you have to compress large files or sets, check out pigz if you are not aware of it. will use every core, even on single files. it splits the file into chunks equal to the core count and works on them in parallel. Its so fast..






take good care,

_g


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



0 Plus One's     0 Comments  
   

Response from Serban Simu @ Feb. 16, 2017, 6:30 p.m.
Quick demo - As currently released 'async' doesn't check files that are processed in parallel (it processes up to several thousands files in parallel). Once a file is 'syncd' then all its hardlinks will be properly detected.
This is one quick demo showing this by first syncing then creating the hardlinks and re-syncing.

serban@kub:~$ ls -li L/total 132082622009 -rwxr-xr-x 1 serban serban 6760104 Feb 16 14:53 file.12622010 -rwxr-xr-x 1 serban serban 6760109 Feb 16 15:12 file.2

serban@kub:~$ async -N hardlink-demo -d /home/serban/L -r 10.0.109.1:/home/serban/R --dedup -k sha1Password:/ SYNCHRONIZED/file.1 SYNCHRONIZED/file.2 SYNCHRONIZED[stopped ] Found/Syncd/Pending/Error/Conflict=3/3/0/0/0Completed. Synchronized 3 paths (12.89 MB transferred) in 22.27 seconds

serban@kub:~$ ln L/file.1 L/link.1
serban@kub:~$ ln L/file.2 L/link.2

serban@kub:~$ async -N hardlink-demo -d /home/serban/L -r 10.0.109.1:/home/serban/R --dedup -k sha1
Password:/link.1 SYNCHRONIZED(ddp)/link.2 SYNCHRONIZED(ddp)[stopped ] Found/Syncd/Pending/Error/Conflict=5/5/0/0/0Completed. Synchronized 2 paths (0 bytes transferred) in 11.73 seconds

serban@kub:~$ ls -li /home/serban/L
total 264162622009 -rwxr-xr-x 2 serban serban 6760104 Feb 16 14:53 file.12622010 -rwxr-xr-x 2 serban serban 6760109 Feb 16 15:12 file.22622009 -rwxr-xr-x 2 serban serban 6760104 Feb 16 14:53 link.12622010 -rwxr-xr-x 2 serban serban 6760109 Feb 16 15:12 link.2

serban@kub:~$ ls -li /home/serban/Rtotal 264165111860 -rwxr-xr-x 2 serban serban 6760104 Feb 16 15:14 file.15111862 -rwxr-xr-x 2 serban serban 6760109 Feb 16 15:14 file.25111860 -rwxr-xr-x 2 serban serban 6760104 Feb 16 15:14 link.15111862 -rwxr-xr-x 2 serban serban 6760109 Feb 16 15:14 link.2

-- Serban Simu




On Thu, Feb 16, 2017 at 3:12 PM, Michelle Munson <michelle@asperasoft.com> wrote:

Tom - Can you tell me the version of async you are running (async --version) and a little about the file system (what is it) on either end?

I'll work it down for you.

Michelle


On 2/16/17 3:08 PM, Tom Taylor wrote:

My response from support was;


The hard link gets transferred as a file. So, async transfers three files:

/toSync/img1.jpg
/toSync/additional/img1.jpg
/toSync/img2.jpg (whose contents are img1.jpg).

I just tested it myself and it created 3 files with separate inodesrather than 2 with 1 hardlink

async -N Test -x -d /tmp/tosync -r root@X:/tmp/tosync -a fair -l 30000 --dedup=hardlink -k md5



On destination server:

# ls -i img1.jpg

791455 img1.jpg

# ls -i img2.jpg

791457 img2.jpg



On original server:

# ls -i img1.jpg

1966821 img1.jpg

# ls -i img2.jpg

1966821 img2.jpg



Greg, based on the above we're using UDT/UDR for now which works but I'm interested to find out if GridFTP behaves the same.

TomTaylor
HeadofEngineering,Chicago
T +13126058900
The Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner
                                  Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 4:54 PM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **

Should do if you add the --dedup=hardlink option to async ... Haven't tested this myself in a while but that's the idea...


Michelle


On 2/16/17 2:50 PM, Tom Taylor wrote:

Michelle, could you tell me if this is possible; If so, fantastic.


cp /originals/img1.jpg /toSync/img1.jpg
cp /originals/img1.jpg /toSync/additional/img1.jpg
ln /toSync/img1.jpg /toSync/img2.jpg

After syncing /toSync from one computer to another, will the resulting directories at the remote end contain two real files called img1.jpg and one hard link called img2.jpg ?





TomTaylor
HeadofEngineering,Chicago
T +13126058900
The
                                            Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 4:46 PM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **

Guys,

Hardlinks are preserved (copied from source to destination) in async when you run with the --dedup option as far as I know;

man async

--dedup[=MODE]
MODE=hardlink(default),copy Hard link/copy files with the same content

Michelle


On 2/16/17 2:41 PM, greg whynott wrote:
Hi Tom,
what do you mean when you say it "expands hard-links"? I'm trying to figure out what you mean exactly.
thanks much, greg


On Thu, Feb 16, 2017 at 1:47 PM, Tom Taylor <taylor@themill.com> wrote:

Michelle, it doesn't, it expands hard-links.


At least that is what I was told from your supportdepartment, if theyhave given me the wrong info feel free to hit me up off list.


-Tom




TomTaylor
HeadofEngineering,Chicago
T +13126058900
The
                                                          Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 11:10 AM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **
Know we like to keep commercials to a minimum but Aspera sync does all of these things.
Let me know if you would like more info.
Michelle

Sent from my iPhone
On Feb 16, 2017, at 10:24 AM, Tom Taylor <taylor@themill.com> wrote:

But.... does it do hard-links? <OutlookEmoji-.png>


I've been searching high and low for something that'll let me;


1. View the status of current transfers and change/edit priority (Pause, stop, play, schedule)

2. Supports hard-links


It appears to be the golden goose of accelerated transfer, seems it's going to have to be something custom.


-TT


TomTaylor
HeadofEngineering,Chicago
T +13126058900
The
                                                          Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of greg whynott <greg.whynott@gmail.com>
Sent: Wednesday, February 15, 2017 10:07:00 PM
To: studiosysadmins-discuss
Subject: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **
Thought this was worth sharing, at a geek level.

Black line is from server to DMZ VM, red line is from DMZ VM to Korea.

The VM is on a large array and the server is a HPC node with a single 7200 RPM drive, working on a job at the same time. Likely the bottle neck here is the drive/system itself, but still, its amusing it took less time to transfer the file to the other side of earth.


side note, if you have to compress large files or sets, check out pigz if you are not aware of it. will use every core, even on single files. it splits the file into chunks equal to the core count and works on them in parallel. Its so fast..






take good care,

_g


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Greg Whynott @ Feb. 16, 2017, 6:30 p.m.
Hi Tom,
I'll give it a go later on tonight. Just hitting the road and won't be home for a few hours yet. I would speculate it'll also copy it as an autonomous file without passing any options/methods. I would also think there is a way to handle this out on the webs, GridFTP has been in deployment for a few decades now.
_g


On Thu, Feb 16, 2017 at 6:08 PM, Tom Taylor <taylor@themill.com> wrote:

My response from support was;


The hard link gets transferred as a file. So, async transfers three files:

/toSync/img1.jpg
/toSync/additional/img1.jpg
/toSync/img2.jpg (whose contents are img1.jpg).

I just tested it myself and it created 3 files with separate inodesrather than 2 with 1 hardlink

async -N Test -x -d /tmp/tosync -r root@X:/tmp/tosync -a fair -l 30000 --dedup=hardlink -k md5



On destination server:

# ls -i img1.jpg

791455 img1.jpg

# ls -i img2.jpg

791457 img2.jpg



On original server:

# ls -i img1.jpg

1966821 img1.jpg

# ls -i img2.jpg

1966821 img2.jpg



Greg, based on the above we're using UDT/UDR for now which works but I'm interested to find out if GridFTP behaves the same.

TomTaylor
HeadofEngineering,Chicago
T+13126058900
The Mill
TheMill1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com|@millchannel|facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 4:54 PM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **

Should do if you add the --dedup=hardlink option to async ... Haven't tested this myself in a while but that's the idea...


Michelle


On 2/16/17 2:50 PM, Tom Taylor wrote:

Michelle, could you tell me if this is possible; If so, fantastic.


cp /originals/img1.jpg /toSync/img1.jpg
cp /originals/img1.jpg /toSync/additional/img1.jpg
ln /toSync/img1.jpg /toSync/img2.jpg

After syncing /toSync from one computer to another, will the resulting directories at the remote end contain two real files called img1.jpg and one hard link called img2.jpg ?





TomTaylor
HeadofEngineering,Chicago
T +13126058900
The Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 4:46 PM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **

Guys,

Hardlinks are preserved (copied from source to destination) in async when you run with the --dedup option as far as I know;

man async

--dedup[=MODE]
MODE=hardlink(default),copy Hard link/copy files with the same content

Michelle


On 2/16/17 2:41 PM, greg whynott wrote:
Hi Tom,
what do you mean when you say it "expands hard-links"? I'm trying to figure out what you mean exactly.
thanks much, greg


On Thu, Feb 16, 2017 at 1:47 PM, Tom Taylor <taylor@themill.com> wrote:

Michelle, it doesn't, it expands hard-links.


At least that is what I was told from your supportdepartment, if theyhave given me the wrong info feel free to hit me up off list.


-Tom




TomTaylor
HeadofEngineering,Chicago
T +13126058900
The Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 11:10 AM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **
Know we like to keep commercials to a minimum but Aspera sync does all of these things.
Let me know if you would like more info.
Michelle

Sent from my iPhone
On Feb 16, 2017, at 10:24 AM, Tom Taylor <taylor@themill.com> wrote:

But.... does it do hard-links? <OutlookEmoji-.png>


I've been searching high and low for something that'll let me;


1. View the status of current transfers and change/edit priority (Pause, stop, play, schedule)

2. Supports hard-links


It appears to be the golden goose of accelerated transfer, seems it's going to have to be something custom.


-TT


TomTaylor
HeadofEngineering,Chicago
T +13126058900
The
                                                          Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of greg whynott <greg.whynott@gmail.com>
Sent: Wednesday, February 15, 2017 10:07:00 PM
To: studiosysadmins-discuss
Subject: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **
Thought this was worth sharing, at a geek level.

Black line is from server to DMZ VM, red line is from DMZ VM to Korea.

The VM is on a large array and the server is a HPC node with a single 7200 RPM drive, working on a job at the same time. Likely the bottle neck here is the drive/system itself, but still, its amusing it took less time to transfer the file to the other side of earth.


side note, if you have to compress large files or sets, check out pigz if you are not aware of it. will use every core, even on single files. it splits the file into chunks equal to the core count and works on them in parallel. Its so fast..






take good care,

_g


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Michelle Munson @ Feb. 16, 2017, 6:15 p.m.

Tom - Can you tell me the version of async you are running (async --version) and a little about the file system (what is it) on either end?

I'll work it down for you.

Michelle


On 2/16/17 3:08 PM, Tom Taylor wrote:

My response from support was;


The hard link gets transferred as a file. So, async transfers three files:

/toSync/img1.jpg
/toSync/additional/img1.jpg
/toSync/img2.jpg (whose contents are img1.jpg).

I just tested it myself and it created 3 files with separate inodesrather than 2 with 1 hardlink

async -N Test -x -d /tmp/tosync -r root@X:/tmp/tosync -a fair -l 30000 --dedup=hardlink -k md5



On destination server:

# ls -i img1.jpg

791455 img1.jpg

# ls -i img2.jpg

791457 img2.jpg



On original server:

# ls -i img1.jpg

1966821 img1.jpg

# ls -i img2.jpg

1966821 img2.jpg



Greg, based on the above we're using UDT/UDR for now which works but I'm interested to find out if GridFTP behaves the same.

TomTaylor
HeadofEngineering,Chicago
T +13126058900
The Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner
                                  Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 4:54 PM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **

Should do if you add the --dedup=hardlink option to async ... Haven't tested this myself in a while but that's the idea...


Michelle


On 2/16/17 2:50 PM, Tom Taylor wrote:

Michelle, could you tell me if this is possible; If so, fantastic.


cp /originals/img1.jpg /toSync/img1.jpg
cp /originals/img1.jpg /toSync/additional/img1.jpg
ln /toSync/img1.jpg /toSync/img2.jpg

After syncing /toSync from one computer to another, will the resulting directories at the remote end contain two real files called img1.jpg and one hard link called img2.jpg ?





TomTaylor
HeadofEngineering,Chicago
T +13126058900
The
                                            Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 4:46 PM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **

Guys,

Hardlinks are preserved (copied from source to destination) in async when you run with the --dedup option as far as I know;

man async

--dedup[=MODE]
MODE=hardlink(default),copy Hard link/copy files with the same content

Michelle


On 2/16/17 2:41 PM, greg whynott wrote:
Hi Tom,
what do you mean when you say it "expands hard-links"? I'm trying to figure out what you mean exactly.
thanks much, greg


On Thu, Feb 16, 2017 at 1:47 PM, Tom Taylor <taylor@themill.com> wrote:

Michelle, it doesn't, it expands hard-links.


At least that is what I was told from your supportdepartment, if theyhave given me the wrong info feel free to hit me up off list.


-Tom




TomTaylor
HeadofEngineering,Chicago
T +13126058900
The
                                                          Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 11:10 AM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **
Know we like to keep commercials to a minimum but Aspera sync does all of these things.
Let me know if you would like more info.
Michelle

Sent from my iPhone
On Feb 16, 2017, at 10:24 AM, Tom Taylor <taylor@themill.com> wrote:

But.... does it do hard-links? <OutlookEmoji-.png>


I've been searching high and low for something that'll let me;


1. View the status of current transfers and change/edit priority (Pause, stop, play, schedule)

2. Supports hard-links


It appears to be the golden goose of accelerated transfer, seems it's going to have to be something custom.


-TT


TomTaylor
HeadofEngineering,Chicago
T +13126058900
The
                                                          Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of greg whynott <greg.whynott@gmail.com>
Sent: Wednesday, February 15, 2017 10:07:00 PM
To: studiosysadmins-discuss
Subject: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **
Thought this was worth sharing, at a geek level.

Black line is from server to DMZ VM, red line is from DMZ VM to Korea.

The VM is on a large array and the server is a HPC node with a single 7200 RPM drive, working on a job at the same time. Likely the bottle neck here is the drive/system itself, but still, its amusing it took less time to transfer the file to the other side of earth.


side note, if you have to compress large files or sets, check out pigz if you are not aware of it. will use every core, even on single files. it splits the file into chunks equal to the core count and works on them in parallel. Its so fast..






take good care,

_g


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Anonymous @ Feb. 16, 2017, 6:10 p.m.

My response from support was;


The hard link gets transferred as a file. So, async transfers three files:

/toSync/img1.jpg
/toSync/additional/img1.jpg
/toSync/img2.jpg (whose contents are img1.jpg).

I just tested it myself and it created 3 files with separate inodes rather than 2 with 1 hardlink

async -N Test -x -d /tmp/tosync -r root@X:/tmp/tosync  -a fair -l 30000 --dedup=hardlink -k md5



On destination server:

# ls -i img1.jpg 

791455 img1.jpg

# ls -i img2.jpg 

791457 img2.jpg



On original server:

# ls -i img1.jpg 

1966821 img1.jpg

# ls -i img2.jpg 

1966821 img2.jpg



Greg, based on the above we're using UDT/UDR for now which works but I'm interested to find out if GridFTP behaves the same. 

Tom Taylor
Head of Engineering, Chicago
+1 312 605 8900
The Mill
The Mill 1000 West Fulton Market, Suite 250, Chicago,  IL 60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 4:54 PM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
  ** WARNING: This mail is from an external source **

Should do if you add the --dedup=hardlink option to async ... Haven't tested this myself in a while but that's the idea...


Michelle


On 2/16/17 2:50 PM, Tom Taylor wrote:

Michelle, could you tell me if this is possible; If so, fantastic. 


cp /originals/img1.jpg /toSync/img1.jpg
cp /originals/img1.jpg /toSync/additional/img1.jpg
ln /toSync/img1.jpg /toSync/img2.jpg

After syncing /toSync from one computer to another, will the resulting directories at the remote end contain two real files called img1.jpg and one hard link called img2.jpg ?





Tom Taylor
Head of Engineering, Chicago
+1 312 605 8900
The Mill
The Mill  1000 West Fulton Market, Suite 250, Chicago,  IL 60607
themill.com  |  @millchannel  |  facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 4:46 PM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
  ** WARNING: This mail is from an external source **

Guys,

Hardlinks are preserved (copied from source to destination) in async when you run with the --dedup option as far as I know;

man async

--dedup[=MODE]
           MODE=hardlink(default),copy Hard link/copy files with the same content

Michelle


On 2/16/17 2:41 PM, greg whynott wrote:
Hi Tom,
what do you mean when you say it "expands hard-links"?   I'm trying to figure out what you mean exactly.  
thanks much, greg


On Thu, Feb 16, 2017 at 1:47 PM, Tom Taylor <taylor@themill.com> wrote:

Michelle, it doesn't, it expands hard-links. 


At least that is what I was told from your support department, if they have given me the wrong info feel free to hit me up off list. 


-Tom




Tom Taylor
Head of Engineering, Chicago
+1 312 605 8900
The Mill
The Mill  1000 West Fulton Market, Suite 250, Chicago,  IL 60607
themill.com  |  @millchannel  |  facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 11:10 AM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
  ** WARNING: This mail is from an external source **
Know we like to keep commercials to a minimum but Aspera sync does all of these things.
Let me know if you would like more info.
Michelle

Sent from my iPhone
On Feb 16, 2017, at 10:24 AM, Tom Taylor <taylor@themill.com> wrote:

But.... does it do hard-links? <OutlookEmoji-.png> 


I've been searching high and low for something that'll let me;


1. View the status of current transfers and change/edit priority (Pause, stop, play, schedule) 

2. Supports hard-links 


It appears to be the golden goose of accelerated transfer, seems it's going to have to be something custom. 


-TT


Tom Taylor
Head of Engineering, Chicago
+1 312 605 8900
The
                                                          Mill
The Mill  1000 West Fulton Market, Suite 250, Chicago,  IL 60607
themill.com  |  @millchannel  |  facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of greg whynott <greg.whynott@gmail.com>
Sent: Wednesday, February 15, 2017 10:07:00 PM
To: studiosysadmins-discuss
Subject: [SSA-Discuss] gridFTP
  ** WARNING: This mail is from an external source **
Thought this was worth sharing,  at a geek level.  

Black line is from server to DMZ VM,   red line is from DMZ VM to Korea. 

 

The VM is on a large array and the server is a HPC node with a single 7200 RPM drive,  working on a job at the same time.  Likely the bottle neck here is the drive/system itself,  but still,  its amusing it took less time to transfer the file to the other side of earth.  


side note,  if you have to compress large files or sets,  check out pigz if you are not aware of it.   will use every core,  even on single files.  it splits the file into chunks equal to the core count and works on them in parallel. Its so fast.. 

 






take good care,

_g


 

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Anonymous @ Feb. 16, 2017, 5:55 p.m.

Michelle, could you tell me if this is possible; If so, fantastic. 


cp /originals/img1.jpg /toSync/img1.jpg
cp /originals/img1.jpg /toSync/additional/img1.jpg
ln /toSync/img1.jpg /toSync/img2.jpg

After syncing /toSync from one computer to another, will the resulting directories at the remote end contain two real files called img1.jpg and one hard link called img2.jpg ?





Tom Taylor
Head of Engineering, Chicago
+1 312 605 8900
The Mill
The Mill 1000 West Fulton Market, Suite 250, Chicago,  IL 60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 4:46 PM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
  ** WARNING: This mail is from an external source **

Guys,

Hardlinks are preserved (copied from source to destination) in async when you run with the --dedup option as far as I know;

man async

--dedup[=MODE]
           MODE=hardlink(default),copy Hard link/copy files with the same content

Michelle


On 2/16/17 2:41 PM, greg whynott wrote:
Hi Tom,
what do you mean when you say it "expands hard-links"?   I'm trying to figure out what you mean exactly.  
thanks much, greg


On Thu, Feb 16, 2017 at 1:47 PM, Tom Taylor <taylor@themill.com> wrote:

Michelle, it doesn't, it expands hard-links. 


At least that is what I was told from your support department, if they have given me the wrong info feel free to hit me up off list. 


-Tom




Tom Taylor
Head of Engineering, Chicago
+1 312 605 8900
The
                                          Mill
The Mill  1000 West Fulton Market, Suite 250, Chicago,  IL 60607
themill.com  |  @millchannel  |  facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 11:10 AM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
  ** WARNING: This mail is from an external source **
Know we like to keep commercials to a minimum but Aspera sync does all of these things.
Let me know if you would like more info.
Michelle

Sent from my iPhone
On Feb 16, 2017, at 10:24 AM, Tom Taylor <taylor@themill.com> wrote:

But.... does it do hard-links? <OutlookEmoji-��.png> 


I've been searching high and low for something that'll let me;


1. View the status of current transfers and change/edit priority (Pause, stop, play, schedule) 

2. Supports hard-links 


It appears to be the golden goose of accelerated transfer, seems it's going to have to be something custom. 


-TT


Tom Taylor
Head of Engineering, Chicago
+1 312 605 8900
The Mill
The Mill  1000 West Fulton Market, Suite 250, Chicago,  IL 60607
themill.com  |  @millchannel  |  facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of greg whynott <greg.whynott@gmail.com>
Sent: Wednesday, February 15, 2017 10:07:00 PM
To: studiosysadmins-discuss
Subject: [SSA-Discuss] gridFTP
  ** WARNING: This mail is from an external source **
Thought this was worth sharing,  at a geek level.  

Black line is from server to DMZ VM,   red line is from DMZ VM to Korea. 

 

The VM is on a large array and the ‘server’ is a HPC node with a single 7200 RPM drive,  working on a job at the same time.  Likely the bottle neck here is the drive/system itself,  but still,  it’s amusing it took less time to transfer the file to the other side of earth.  ��


side note,  if you have to compress large files or sets,  check out pigz if you are not aware of it.   will use every core,  even on single files.  it splits the file into chunks equal to the core count and works on them in parallel. Its so fast.. 

 






take good care,

_g


 

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Michelle Munson @ Feb. 16, 2017, 5:55 p.m.

Should do if you add the --dedup=hardlink option to async ... Haven't tested this myself in a while but that's the idea...


Michelle


On 2/16/17 2:50 PM, Tom Taylor wrote:

Michelle, could you tell me if this is possible; If so, fantastic.


cp /originals/img1.jpg /toSync/img1.jpg
cp /originals/img1.jpg /toSync/additional/img1.jpg
ln /toSync/img1.jpg /toSync/img2.jpg

After syncing /toSync from one computer to another, will the resulting directories at the remote end contain two real files called img1.jpg and one hard link called img2.jpg ?





TomTaylor
HeadofEngineering,Chicago
T +13126058900
The Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 4:46 PM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **

Guys,

Hardlinks are preserved (copied from source to destination) in async when you run with the --dedup option as far as I know;

man async

--dedup[=MODE]
MODE=hardlink(default),copy Hard link/copy files with the same content

Michelle


On 2/16/17 2:41 PM, greg whynott wrote:
Hi Tom,
what do you mean when you say it "expands hard-links"? I'm trying to figure out what you mean exactly.
thanks much, greg


On Thu, Feb 16, 2017 at 1:47 PM, Tom Taylor <taylor@themill.com> wrote:

Michelle, it doesn't, it expands hard-links.


At least that is what I was told from your supportdepartment, if theyhave given me the wrong info feel free to hit me up off list.


-Tom




TomTaylor
HeadofEngineering,Chicago
T +13126058900
The Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 11:10 AM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **
Know we like to keep commercials to a minimum but Aspera sync does all of these things.
Let me know if you would like more info.
Michelle

Sent from my iPhone
On Feb 16, 2017, at 10:24 AM, Tom Taylor <taylor@themill.com> wrote:

But.... does it do hard-links? <OutlookEmoji-.png>


I've been searching high and low for something that'll let me;


1. View the status of current transfers and change/edit priority (Pause, stop, play, schedule)

2. Supports hard-links


It appears to be the golden goose of accelerated transfer, seems it's going to have to be something custom.


-TT


TomTaylor
HeadofEngineering,Chicago
T +13126058900
The
                                                          Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of greg whynott <greg.whynott@gmail.com>
Sent: Wednesday, February 15, 2017 10:07:00 PM
To: studiosysadmins-discuss
Subject: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **
Thought this was worth sharing, at a geek level.

Black line is from server to DMZ VM, red line is from DMZ VM to Korea.

The VM is on a large array and the server is a HPC node with a single 7200 RPM drive, working on a job at the same time. Likely the bottle neck here is the drive/system itself, but still, its amusing it took less time to transfer the file to the other side of earth.


side note, if you have to compress large files or sets, check out pigz if you are not aware of it. will use every core, even on single files. it splits the file into chunks equal to the core count and works on them in parallel. Its so fast..






take good care,

_g


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Michelle Munson @ Feb. 16, 2017, 5:50 p.m.

Guys,

Hardlinks are preserved (copied from source to destination) in async when you run with the --dedup option as far as I know;

man async

--dedup[=MODE]
MODE=hardlink(default),copy Hard link/copy files with the same content

Michelle


On 2/16/17 2:41 PM, greg whynott wrote:
Hi Tom,
what do you mean when you say it "expands hard-links"? I'm trying to figure out what you mean exactly.
thanks much, greg


On Thu, Feb 16, 2017 at 1:47 PM, Tom Taylor <taylor@themill.com> wrote:

Michelle, it doesn't, it expands hard-links.


At least that is what I was told from your supportdepartment, if theyhave given me the wrong info feel free to hit me up off list.


-Tom




TomTaylor
HeadofEngineering,Chicago
T +13126058900
The
                                          Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 11:10 AM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **
Know we like to keep commercials to a minimum but Aspera sync does all of these things.
Let me know if you would like more info.
Michelle

Sent from my iPhone
On Feb 16, 2017, at 10:24 AM, Tom Taylor <taylor@themill.com> wrote:

But.... does it do hard-links? <OutlookEmoji-.png>


I've been searching high and low for something that'll let me;


1. View the status of current transfers and change/edit priority (Pause, stop, play, schedule)

2. Supports hard-links


It appears to be the golden goose of accelerated transfer, seems it's going to have to be something custom.


-TT


TomTaylor
HeadofEngineering,Chicago
T +13126058900
The Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of greg whynott <greg.whynott@gmail.com>
Sent: Wednesday, February 15, 2017 10:07:00 PM
To: studiosysadmins-discuss
Subject: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **
Thought this was worth sharing, at a geek level.

Black line is from server to DMZ VM, red line is from DMZ VM to Korea.

The VM is on a large array and the server is a HPC node with a single 7200 RPM drive, working on a job at the same time. Likely the bottle neck here is the drive/system itself, but still, its amusing it took less time to transfer the file to the other side of earth.


side note, if you have to compress large files or sets, check out pigz if you are not aware of it. will use every core, even on single files. it splits the file into chunks equal to the core count and works on them in parallel. Its so fast..






take good care,

_g


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Greg Whynott @ Feb. 16, 2017, 5:45 p.m.
Hi Tom,
what do you mean when you say it "expands hard-links"? I'm trying to figure out what you mean exactly.
thanks much,greg


On Thu, Feb 16, 2017 at 1:47 PM, Tom Taylor <taylor@themill.com> wrote:

Michelle, it doesn't, it expands hard-links.


At least that is what I was told from your supportdepartment, if theyhave given me the wrong info feel free to hit me up off list.


-Tom




TomTaylor
HeadofEngineering,Chicago
T+13126058900
The Mill
TheMill1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com|@millchannel|facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 11:10 AM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **
Know we like to keep commercials to a minimum but Aspera sync does all of these things.
Let me know if you would like more info.
Michelle

Sent from my iPhone
On Feb 16, 2017, at 10:24 AM, Tom Taylor <taylor@themill.com> wrote:

But.... does it do hard-links? <OutlookEmoji-.png>


I've been searching high and low for something that'll let me;


1. View the status of current transfers and change/edit priority (Pause, stop, play, schedule)

2. Supports hard-links


It appears to be the golden goose of accelerated transfer, seems it's going to have to be something custom.


-TT


TomTaylor
HeadofEngineering,Chicago
T +13126058900
The Mill
TheMill 1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of greg whynott <greg.whynott@gmail.com>
Sent: Wednesday, February 15, 2017 10:07:00 PM
To: studiosysadmins-discuss
Subject: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **
Thought this was worth sharing, at a geek level.

Black line is from server to DMZ VM, red line is from DMZ VM to Korea.

The VM is on a large array and the server is a HPC node with a single 7200 RPM drive, working on a job at the same time. Likely the bottle neck here is the drive/system itself, but still, its amusing it took less time to transfer the file to the other side of earth.


side note, if you have to compress large files or sets, check out pigz if you are not aware of it. will use every core, even on single files. it splits the file into chunks equal to the core count and works on them in parallel. Its so fast..






take good care,

_g


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Michelle Munson @ Feb. 16, 2017, 3:20 p.m.
Not in dedup mode... I will get you the details

Sent from my iPhone
On Feb 16, 2017, at 1:47 PM, Tom Taylor <taylor@themill.com> wrote:

Michelle, it doesn't, it expands hard-links. 


At least that is what I was told from your support department, if they have given me the wrong info feel free to hit me up off list. 


-Tom




Tom Taylor
Head of Engineering, Chicago
+1 312 605 8900
The Mill
The Mill 1000 West Fulton Market, Suite 250, Chicago,  IL 60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 11:10 AM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
  ** WARNING: This mail is from an external source **
Know we like to keep commercials to a minimum but Aspera sync does all of these things.
Let me know if you would like more info.
Michelle

Sent from my iPhone
On Feb 16, 2017, at 10:24 AM, Tom Taylor <taylor@themill.com> wrote:

But.... does it do hard-links? <OutlookEmoji-.png> 


I've been searching high and low for something that'll let me;


1. View the status of current transfers and change/edit priority (Pause, stop, play, schedule) 

2. Supports hard-links 


It appears to be the golden goose of accelerated transfer, seems it's going to have to be something custom. 


-TT


Tom Taylor
Head of Engineering, Chicago
+1 312 605 8900
The Mill
The Mill  1000 West Fulton Market, Suite 250, Chicago,  IL 60607
themill.com  |  @millchannel  |  facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of greg whynott <greg.whynott@gmail.com>
Sent: Wednesday, February 15, 2017 10:07:00 PM
To: studiosysadmins-discuss
Subject: [SSA-Discuss] gridFTP
  ** WARNING: This mail is from an external source **
Thought this was worth sharing,  at a geek level.  

Black line is from server to DMZ VM,   red line is from DMZ VM to Korea. 

 

The VM is on a large array and the server is a HPC node with a single 7200 RPM drive,  working on a job at the same time.  Likely the bottle neck here is the drive/system itself,  but still,  its amusing it took less time to transfer the file to the other side of earth.  


side note,  if you have to compress large files or sets,  check out pigz if you are not aware of it.   will use every core,  even on single files.  it splits the file into chunks equal to the core count and works on them in parallel. Its so fast.. 

 






take good care,

_g


 

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe
To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

0 Plus One's     0 Comments  
   

Response from Anonymous @ Feb. 16, 2017, 1:50 p.m.

Michelle, it doesn't, it expands hard-links. 


At least that is what I was told from your support department, if they have given me the wrong info feel free to hit me up off list. 


-Tom




Tom Taylor
Head of Engineering, Chicago
+1 312 605 8900
The Mill
The Mill 1000 West Fulton Market, Suite 250, Chicago,  IL 60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of Michelle Munson <michelle@asperasoft.com>
Sent: Thursday, February 16, 2017 11:10 AM
To: studiosysadmins-discuss@studiosysadmins.com
Subject: Re: [SSA-Discuss] gridFTP
  ** WARNING: This mail is from an external source **
Know we like to keep commercials to a minimum but Aspera sync does all of these things.
Let me know if you would like more info.
Michelle

Sent from my iPhone
On Feb 16, 2017, at 10:24 AM, Tom Taylor <taylor@themill.com> wrote:

But.... does it do hard-links? <OutlookEmoji-��.png> 


I've been searching high and low for something that'll let me;


1. View the status of current transfers and change/edit priority (Pause, stop, play, schedule) 

2. Supports hard-links 


It appears to be the golden goose of accelerated transfer, seems it's going to have to be something custom. 


-TT


Tom Taylor
Head of Engineering, Chicago
+1 312 605 8900
The Mill
The Mill  1000 West Fulton Market, Suite 250, Chicago,  IL 60607
themill.com  |  @millchannel  |  facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of greg whynott <greg.whynott@gmail.com>
Sent: Wednesday, February 15, 2017 10:07:00 PM
To: studiosysadmins-discuss
Subject: [SSA-Discuss] gridFTP
  ** WARNING: This mail is from an external source **
Thought this was worth sharing,  at a geek level.  

Black line is from server to DMZ VM,   red line is from DMZ VM to Korea. 

 

The VM is on a large array and the ‘server’ is a HPC node with a single 7200 RPM drive,  working on a job at the same time.  Likely the bottle neck here is the drive/system itself,  but still,  it’s amusing it took less time to transfer the file to the other side of earth.  ��


side note,  if you have to compress large files or sets,  check out pigz if you are not aware of it.   will use every core,  even on single files.  it splits the file into chunks equal to the core count and works on them in parallel. Its so fast.. 

 






take good care,

_g


 

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

0 Plus One's     0 Comments  
   

Response from Greg Whynott @ Feb. 16, 2017, 12:25 p.m.
Hi Tom,
It seems to move hard linked files, but I may be missing what you are actually asking. :)
[globus@tor-dmz-xfer greg]$ ls -ltrtotal 97580-rw-r--r-- 1 globus globus 199918260 Feb 16 11:58 chunk8.bz2[globus@tor-dmz-xfer greg]$ mv chunk8.bz2 /tmp/[globus@tor-dmz-xfer greg]$ ln /tmp/chunk8.bz2 .[globus@tor-dmz-xfer greg]$ ln -s /tmp/chunk8.bz2 chunk8.soft[globus@tor-dmz-xfer greg]$ globus-url-copy -cd -vb -p 8 file:/opt/xfer/greg/* sshftp://kor-srv-xfer:/opt/xfer/greg/linktest/Source: file:/opt/xfer/greg/Dest: sshftp://kor-srv-xfer:/opt/xfer/greg/linktest/ chunk8.bz2
Source: file:/opt/xfer/greg/82.08 MB/sec avg 85.08 MB/sec instDest: sshftp://kor-srv-xfer:/opt/xfer/greg/linktest/ chunk8.soft
199918260 bytes 90.43 MB/sec avg 91.23 MB/sec inst[globus@tor-dmz-xfer greg]$

If you have a more specific test case scenario I'd be happy to try it for you. Aspera can throttle down connections, but I don't think what I'm using can. But you can set limits, so stopping and restarting (it'll pick up where it left off) with new limits may be workable.
take care,_g




On Thu, Feb 16, 2017 at 10:24 AM, Tom Taylor <taylor@themill.com> wrote:

But.... does it do hard-links?


I've been searching high and low for something that'll let me;


1. View the status of current transfers and change/edit priority (Pause, stop, play, schedule)

2. Supports hard-links


It appears to be the golden goose of accelerated transfer, seems it's going to have to be something custom.


-TT


TomTaylor
HeadofEngineering,Chicago
T+13126058900
The Mill
TheMill1000WestFultonMarket,Suite250,Chicago,IL60607
themill.com|@millchannel|facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of greg whynott <greg.whynott@gmail.com>
Sent: Wednesday, February 15, 2017 10:07:00 PM
To: studiosysadmins-discuss
Subject: [SSA-Discuss] gridFTP
** WARNING: This mail is from an external source **
Thought this was worth sharing, at a geek level.

Black line is from server to DMZ VM, red line is from DMZ VM to Korea.

The VM is on a large array and the server is a HPC node with a single 7200 RPM drive, working on a job at the same time. Likely the bottle neck here is the drive/system itself, but still, its amusing it took less time to transfer the file to the other side of earth.


side note, if you have to compress large files or sets, check out pigz if you are not aware of it. will use every core, even on single files. it splits the file into chunks equal to the core count and works on them in parallel. Its so fast..






take good care,

_g



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Michelle Munson @ Feb. 16, 2017, 12:15 p.m.
Know we like to keep commercials to a minimum but Aspera sync does all of these things.
Let me know if you would like more info.
Michelle

Sent from my iPhone
On Feb 16, 2017, at 10:24 AM, Tom Taylor <taylor@themill.com> wrote:

But.... does it do hard-links? <OutlookEmoji-.png> 


I've been searching high and low for something that'll let me;


1. View the status of current transfers and change/edit priority (Pause, stop, play, schedule) 

2. Supports hard-links 


It appears to be the golden goose of accelerated transfer, seems it's going to have to be something custom. 


-TT


Tom Taylor
Head of Engineering, Chicago
+1 312 605 8900
The Mill
The Mill 1000 West Fulton Market, Suite 250, Chicago,  IL 60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of greg whynott <greg.whynott@gmail.com>
Sent: Wednesday, February 15, 2017 10:07:00 PM
To: studiosysadmins-discuss
Subject: [SSA-Discuss] gridFTP
  ** WARNING: This mail is from an external source **
Thought this was worth sharing,  at a geek level.  

Black line is from server to DMZ VM,   red line is from DMZ VM to Korea. 

 

The VM is on a large array and the server is a HPC node with a single 7200 RPM drive,  working on a job at the same time.  Likely the bottle neck here is the drive/system itself,  but still,  its amusing it took less time to transfer the file to the other side of earth.  


side note,  if you have to compress large files or sets,  check out pigz if you are not aware of it.   will use every core,  even on single files.  it splits the file into chunks equal to the core count and works on them in parallel. Its so fast.. 

 






take good care,

_g


 

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

0 Plus One's     0 Comments  
   

Response from Anonymous @ Feb. 16, 2017, 10:30 a.m.

But.... does it do hard-links?  


I've been searching high and low for something that'll let me;


1. View the status of current transfers and change/edit priority (Pause, stop, play, schedule) 

2. Supports hard-links 


It appears to be the golden goose of accelerated transfer, seems it's going to have to be something custom. 


-TT


Tom Taylor
Head of Engineering, Chicago
+1 312 605 8900
The Mill
The Mill 1000 West Fulton Market, Suite 250, Chicago,  IL 60607
themill.com | @millchannel | facebook.com/millchannel
Banner Image


From: studiosysadmins-discuss-bounces@studiosysadmins.com <studiosysadmins-discuss-bounces@studiosysadmins.com> on behalf of greg whynott <greg.whynott@gmail.com>
Sent: Wednesday, February 15, 2017 10:07:00 PM
To: studiosysadmins-discuss
Subject: [SSA-Discuss] gridFTP
  ** WARNING: This mail is from an external source **
Thought this was worth sharing,  at a geek level.  

Black line is from server to DMZ VM,   red line is from DMZ VM to Korea. 

 

The VM is on a large array and the server is a HPC node with a single 7200 RPM drive,  working on a job at the same time.  Likely the bottle neck here is the drive/system itself,  but still,  its amusing it took less time to transfer the file to the other side of earth.  


side note,  if you have to compress large files or sets,  check out pigz if you are not aware of it.   will use every core,  even on single files.  it splits the file into chunks equal to the core count and works on them in parallel. Its so fast.. 

 






take good care,

_g


 


0 Plus One's     0 Comments