Sponsors
Sponsor Products
FW: Colo Reccomendation in Northern NJ or Manhattan Area?
posted by Brittany Coulson  on Aug. 5, 2015, 4:35 p.m. (4 years, 11 months, 29 days ago)
0 Responses     0 Plus One's     0 Comments  
For Google Cloud Storage:gsutil rsyncis the default answer here. The more youcan break up the data across multiple machines, the more you'llbe able to saturate yourconnection and move the data faster.

Let me know if you have further questions, I'll be at Siggraph next week and happy to talk.
Thanks,Brittany

On Wednesday, August 5, 2015, Saker Klippsten <sakerk@gmail.com> wrote:
I'm a big fan of compartmentalization. We have specific machines (VMs) working as transfer machines. These machines run our custom code and scripts for moving data around facilities. They mount our various Storage Silos via NFS. From there we run our code to do various tasks. They run cron jobs etc . Or receive jobs to do things like backup to tape etc..or move data to an aspera server... So if we use google to potentially archive long term they would be used in this manner.
It's safer than running code on the file serves.
Also from a security aspect it's safer to put ACLs in place and have an in between machine than running or exposing the server..


On Aug 5, 2015, at 12:37 PM, William Sandler <william.sandler@allthingsmedia.com> wrote:

Yes it's a Solaris 11 box.
I also thought about having a Linux Box in the middle but would like to avoid an extra point of failure if possible. I suppose I could even export the pool and do a ZOL but that isn't too feasible.

William SandlerAll Things Media, LLCOffice:201.818.1999 Ex 158.william.sandler@allthingsmedia.com
On Wed, Aug 5, 2015 at 3:31 PM, Saker Klippsten <sakerk@gmail.com> wrote:
Is this because your filer is Solaris? How fast is your net connection to google?Why not just have a Linux Box in the middle, NFS mount your filer or storage server and act as your transfer box to google.


-S


On Wed, Aug 5, 2015 at 12:26 PM, William Sandler <william.sandler@allthingsmedia.com> wrote:
I wonder if I could get gsutil to work on Solaris since it's Python based. Their site seems to intimate that it's Linux only.

William SandlerAll Things Media, LLCOffice:201.818.1999 Ex 158.william.sandler@allthingsmedia.com
On Wed, Aug 5, 2015 at 3:23 PM, Saker Klippsten <sakerk@gmail.com> wrote:
Sorry hit return and it sent!ack!

https://cloud.google.com/storage/transfer/index

https://cloud.google.com/storage/docs/gsutil?hl=en

https://cloud.google.com/files/MigratingtoGoogleCloudStorageNearlineFromAmazonGlacier.pdf
This talks about going form Glacier to GN but references some gutil commands to

cloud$gsutil-mrsync-r-cbatch-log.txt\ /data/test-vaultgs://nearline-migration/test-vault The -moption instructs gsutil to use multiple threads to copy, which will significantly increase performance if there are many files to copy. The -roption tells gsutil to recurse into any subdirectories below /data/test-vault. The -coption tells gsutil to continue to copy even if there is an error copying one of the files. Note that we did not specify the -d option, which means no objects will be deleted from the destination bucket.


Also quick google search. Looks interesting I think there are linux versions as well
http://www.cloudberrylab.com/

http://www.cloudberrylab.com/free-google-storage-explorer.aspx

seems to have commandline support.



On Wed, Aug 5, 2015 at 11:42 AM, William Sandler <william.sandler@allthingsmedia.com> wrote:
Had a few talks with Google sales guys. It seems the only way to get the data into Nearline is to setup a web server and have them download it.
Does anybody know if there is another way that the Sales guys don't know about? Something like Rsync or FTP? I'd rather not have to install Apache on our backup server and turn it into an external facing web server.




William SandlerAll Things Media, LLCOffice:201.818.1999 Ex 158.william.sandler@allthingsmedia.com
On Wed, Jul 29, 2015 at 10:09 PM, William Sandler <william.sandler@allthingsmedia.com> wrote:
I hope to never deal with tape in my career. I've somehow avoided it thus far.
Wow I really like those Google specs. I had incorrectly assumed it was like Glacier and was slow and meant for archive, not backup. I wonder what the retrieval bandwidth is. From a brief look, it seems like 4MB/s but you can pay for faster speeds. Assuming true disaster, with the Colo at least I can always physically pick up the server and bring it locally to recover data at gigabit speeds or higher. (Our WAN is no where near that).
Just to compare costs: I had spec'd out a self-built ZFS server (RAID-Z2) at around $25k and $1,500 a month at the Colo. Additional costs over time would just be adding more back-end.
Nobody ever wants to delete anything these days as they don't want to be the one responsible for ordering a deletion and then a client from 8 years ago comes back asking us to make some edits to ancient project.
Thanks for the info. Just once I wish google wasn't the answer. Eggs, basket.



William SandlerAll Things Media, LLCOffice:201.818.1999 Ex 158.william.sandler@allthingsmedia.com
On Wed, Jul 29, 2015 at 6:08 PM, Saker Klippsten <sakerk@gmail.com> wrote:
This is just a brain dump..
I have be researching COLO's for a bit for our NY Office. You might want to reach out toMatt Easton @ Internap, measton@internap.comThey have one of their COLO in Secauscus NJ. Hop Skip and Jump from the City.Besides just rack space.. they offer many other Services. Cloud Compute, backup etc..I have not pulled the trigger yet with them but compared to anything on the Island its much less expensive.

Currently we use a mixture of Isilon NL108 and 400 Nodes for up to the min DR and Snapshots and upon project completion we do final project archive using Spectra Logic 380 Libraries and LTO-6 Media.DR for Vancouver is in LA and Vice-Versa. Now that we have NY looking into adding another potential replica site or even moving LA to NY ( Thanks to the New Yorker Article ) hahah

As far as a cloud argument?I was just reading up on googles new storage-nearline offering with 3ms retrieval time and 1cent per GB.
https://cloud.google.com/storage-nearline/
1cent per GB per month so that's about $1500 a month for 150TB. I am sure there are taxes.. as well.
vs. Colo with Tape.
All #'s off the top of my head very rough..
Tape: Cost would be about once time cost of $3000 ( 60 tapes with 2.5TB per tape @ $50each ) one set or copy of the dataLibrary: Cost at say 50k for a 60 tape loader with 1 drive.Software: Cost 3500? if using BakboneColo Cost: Half Rack just for Robot , Server and switch? ~1500/month? with Connectivity and Power? Taxes etc..Offiste tape storage cost for second set of tapes? I am not putting this into the equation..
~74K with a potential 56k up front cost.
*It would take you about 49 Months ( ~4 years ) to see that $ back vs the cost of Google Cloud Archive and just another $1500 to pull down that 150TB when needed
*How long do you need to hold on to that data? Indefinitely? Or will that 150TB be updated periodically and purged out..will that 150TB grow ? if so how much?
* If you were to add another 150TB over the course of a year. thats another 21k ( cost of tape plus monthly colo. vs google cloud depending upon how fast you uploaded 150tb you could spend as much as 18ki if you uploaded in January or as little as $1500 if you uploaded 150TB in December..
* Its really all about understanding how long your data needs to live. This will really dictate the cost benefit of rolling your own on prem or colo vs cloud.
* There is also the convenience factor and simplicity of the cloud I did not take into account. Not having to drive or deal with a colo to change tapes or just dealing with tapes.

Decisions.... you always get screwed somehow right?


-S




On Wed, Jul 29, 2015 at 1:52 PM, William Sandler <william.sandler@allthingsmedia.com> wrote:
Any favorites?
Or if anyone has a convincing argument to go with managed cloud storage instead to backup ~150TB.
What are you guys using for off-site backups of your storage appliances?


William SandlerAll Things Media, LLCOffice:201.818.1999 Ex 158.william.sandler@allthingsmedia.com
To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


--
________________Brittany Coulson | Cloud Specialist | Google Inc. |bcoulson@google.com| C:(508)768-7567


Thread Tags:
  discuss-at-studiosysadmins