It goes without saying that, as a professional photographer, my entire archive will not fit on my laptops internal SSD. But every now and again, I need to go through a shoot from a year ago or two. And since I can’t be carrying around ten external hard drives everywhere I go, I’ve been experimenting a bunch with different cloud workflows. Here’s what I’ve come up with.
First some key take-aways
This is NOT backup. That’s a totally different process separate from my cloud workflow.
You’ll need a NAS setup.
You’ll need a BitTorrent sync client such as Resilio or Syncthing.
When you need to access your files, they’ll download to your local hard drive. Any changes you make to these will automatically get synced back to the cloud in real time. When you’re done, you can just remove them from your local HDD.
Put your files online
Get a NAS. Or maybe even two.
NAS stands for Network Attached Storage. Basically a tiny little low wattage server with a bunch of storage.
I’m using a simple 4-bay QNAP with 4x8TB HDDs. Which gives me 32TB, this capacity is then sliced into half since the drives are configured in a Raid 10 setup, which means that only two of the drives are acting as storage. The two other drives are only there to mirror the main drives in case the main one fails. This means that I have 16TB to play with. Not exactly future proof, but at least I’m good for another year or two.
This is going to run you a little over $1000 with todays hard drive prices, which might seem like a heavy investment for something as boring as digital asset management. But after doing some math, it seems like it’s still the only viable way for large archives. Right now, there are no cloud services that will be able to provide you with unthrottled 10TB+ for a reasonable price. And this workflow has made my life a whole lot easier.
You then have to dump all of those external hard drives lying around onto the NAS and get your archive online.
Make them accessible
Get a sync client
So now that your archive is in one place and online, how do you access it?
Although you might be able to work directly from your NAS by mounting it as a network drive. This would only be possible on a very good local network. But it’s definitely not gonna happen if you’re somewhere else and try to access the NAS remotely, which is kinda the whole point with getting on the cloud.
This is where Resilio Sync (I’m using the Home License) comes in. Resilio is a sync utility that you install on your computer and NAS, much like the dropbox/One Drive/Google Drive client you already have installed. There’s also Syncthing which seems pretty similar, but I haven’t tried it myself.
You then choose which folders you want synced, and point the client to where the corresponding folder is located on the NAS.
Anyway, Resilio not a perfect piece of software by any means. It has its quirks, and it takes some time before you figure everything out. But once you have it set up, it’s going to take care of it self pretty well without you babysitting it. And there are two things I really like about it.
Tiny place-holder files
Your entire archive in 2 megabytes.
The first thing I love is this feature they call “Selective Sync”. What this does is it creates place-holder files that doesn’t take up any space on your computer. This means you can have the folder structure of your entire archive on your laptop, and it only takes up a couple of megabytes.
These folders make up about 7 TB together, but the place holders only take up 2,3 MB.
And when you need the actual raw-files, you just download them, straight from your computers file browser. Anything you do to these files will then sync back in real time to your NAS, including if you delete it or corrupt it, which is also why this is NOT a substitute for backup. Always keep an offline copy of your files as well.
Just click “Sync to this device”, and the files will start to download to your local hard drive. Then “Remove from this device” once no longer need them on your computer.
Speed and reliability
Pros of the BitTorrent Protocol
The second thing I love about it is that it relies on the Bittorrent protocol, which is very fast and stable. Especially if you have more than one node in the network. I for example have two NAS setups. One at my home office in New York City, and another one at my brothers place in Stockholm. These two devices mirror each other 24/7 for additional redundancy. This also means that I get greater stability and speed when I need to access my files since I have two independent setups with their own internet connection, and thus “double” the bandwidth. This will of course get potentially bottlenecked by whatever down-speed you have at the moment. But at least the up-speed wont be a problem.
Below is a screenshot of Resilio downloading an old shoot I had to work on while sitting at my favourite neighbourhood café. A pretty decent speed overall.
This also shows how trash the New York ISPs are compared to Sweden. The bottom one is the NAS located in Stockholm.
Once your files are fully synced to your computer they will show up as online in Lightroom and you can start working on them as if they were always there.