Amazon S3 Photo/Video Backup Solution

Far far away, someone, somewhere said:

“There are two kinds of people, those who back up their data and those who have never lost all their data.”

To be the lucky one, it's important to do backups regularly. Fortunately there is no need to backup the whole machine, since the operating system can be re-installed in minutes; most apps can be installed through single command for Mac or Win; projects can be synced through Github. The only key things that require backed up are important files like personal photo collection and videos.

Backup strategy in a nutshell

We suggest to kept all of the photos/videos in a well organized collection, ordered chronologically by year and by session / event. It is normally not an enormous amount of data (around few hundred GB) but the sentimental value that it holds could be immense.

Normally this collection can be stored on two physical devices. It can be on the computer’s hard drive, an external flash disc, or a NAS server.

The challenge here is, as photo/video size grows, the disks could run out of storage quickly, and moving files around to allocate space is a headache. Furthermore, things happen! Disks fail, people rob, rivers flood, comets fall. In case any of that occurs we need one copy in the cloud as the final rescue. We have tested multiple solutions and services over the past few years and finally we feel that the backup to cloud is more mature now (in terms of cost and convenience) and here we are to help you out.

What could be a good cloud backup and things we don't care about

There are plenty of services that offer cloud storage for amateur and professional photographers or youtubers. Dropbox, Google Drive, Box, OneDrive or Backblaze just to name a few.

There are a few key things that should be considered for the cloud backup solution.

security first - no one wants anyone else to look at their personal pictures/videos
auto-scaling and unlimited storage - the collection will grow over time
stay alive - we don't want to investing time in solutions that may not be around tomorrow.
price - cost effective

The providers listed above usually offer tons of things that we don’t care much about. We don’t need a fancy app with tons of bells and whistles. We don’t need a constant live sync and seamless integration with our OS. It is a last resort backup - the file structure is probably never going to change. We just add more stuff over time.

Here we are not to compare the available options or convince you to use one over the other. We spent years looking for a solution that suits this picture/video backup needs and we would like to share it with you.

Say hello to AWS Simple Storage Service (S3)

AWS (Amazon Web Services) is a platform that offers a number of things that your business or you, as an individual, may need. From computational power, through to database storage, content delivery networks to machine learning and IoT (Internet of things) related products. A storage solution is one of the many services that AWS has to offer. It is well established and proven by the mile-long list of clients like: Adobe, AirBnb, Netflix, NASA, CapitalOne, Canon, GoPro… The list goes on and on.

You may have heard the opinion that AWS is complicated to use. In reality it is crazy complicated but to be in a band you don’t have to play all the instruments — just master a single one. Storage is what we need. In details, the thing that we need is a container of data stored within an S3 bucket and its seamless transition to the Glacier/Deep Archive class.

What is S3 and how it works

Amazon S3 is a simple storage solution that offers a range of classes designed for specific use cases. For frequently used, general storage use S3 Standard. Infrequent Access works best for files that you don’t have to access very often but still keep them accessible whenever you need them. For archiving purposes, Glacier/Deep Archive is the best option. Each of these categories comes with pros and cons and each of them suits different needs. The main differences between them are price and waiting time to access objects (photos in our case). For those that are curious I would direct you to Marc Trimuschats presentation from the AWS Summit 2017. Deep Dive on Object Storage tells you everything that you need.

Amazon S3 storage classes

Essentially, files stored in the hot storage (S3 Standard) are accessible immediately but they will cost you a fortune ($0.023 / GB). Cold storage (Glacier) on the other hand is extremely cheap ($0.004 per GB) but a file restoration can take from 1 minute up to 12 hours. Deep code storage (Deep Archive) is even cheaper ($0.00099 per GB) but a file restoration can take within 12 hours. You will be charged for each GB retrieved from the cold storage cluster too. The pricing may vary a bit depending on the region of your S3 “bucket”.

Privacy of files is something that we can easily control with S3. If you want to make a file public or private, no more than a single click is needed.

How to

Start with creating a free AWS account. This process requires you to add a credit card to your account and authorize it by a phone call you will receive from Amazon’s bot. It is worth mentioning that you are eligible to use a Free Tier that gives you access to a snippet of AWS features totally for free. You can end this process here but if you want sophisticated security credential handling, we suggest to look at the IAM (Identity and Access Management) best practices. i.e., “root” account is just for billing purposes and to manage users. An IAM user shall be created with sufficient permissions to handle the backup tasks.

Once the account is ready to use, it is time to connect to the S3 storage and create the first storage “bucket” for the S3 storage. Here we use CrossFTP as the tool to show how to do the backups.

Create a new site in CrossFTP, and choose Protocol as S3/HTTPS

Fill in the Access Key field and Secret field. Press connect to connect to the S3 site. Your Access key and Secret key can also be found at this page.

Select default storage class for your backup at the S3 tab in site manager.

For Glacier storage class, select "GLACIER" (~$0.004 per GB)
For Deep Archive class, select "DEEP_ARCHIVE" (~$0.00099 per GB)

Connect to S3 storage, and create the first storage “bucket” under the S3. Use a unique name for your bucket and choose a location of interest. Make a wise decision at this point because you won’t be able to change those details later on.

Go into the "bucket" you created, and drag & drop your files to the S3 pane to do the backup.

How to change storage class

If you want to change existing files' storage class to Glacier/Deep Archive, you can select the files, and choose "Change Storage Class" from the popup menu.

How to restore data

When you need to restore data form S3 folder to your local download folder (e.g., /Download), open CrossFTP, and navigate the local panel to the local download folder (e..g, /Download), and navigate the remote panel to the S3's folder (e.g., /crossftp-backup/20200908), and select restore to restore & download data. An estimation of the restore retrieve fee will be provided on the UI.

Happy backing up

This solution works for us really well. We managed to reduce the cost of our digital backups from $10 per month to around $1. We like to hear your feedback and hope this works for you as well.

References

Amazon S3 storage classes and pricing
Paweł Grzybek, My Amazon S3 photo backup solution
CrossFTP, Amazon S3 client for Windows, Mac, and Linux.

CrossFTP, FTP and Amazon S3 Client

Monday, September 28, 2020

CrossFTP 1.99.6 Released

Sunday, September 13, 2020