Inspiration for swarm storage

Imagine a new way of data storage of private and public data, independent on any single point of failure or trust of them. Hold that thought and have a look at what there is today.

Storage

During the last couple of years I have seen new technologies and services that all work with storing and transporting data. These have all given me some ideas that I will try to extract here.

Your own hard drive is the most accessible storage when you are at your computer. It has benefits of speed, size and you trust it with your private data. Since it’s very much used it risks loosing files by mistake or bad intentions from others. Usually this is the device being backed up.

Discs(CD/DVD) and external USB hard drives can be considered trusted as long as you don’t loose them. They are fast, have large storage. As a permanent storage they can be considered safer against mistakes and ill intent. Their mobility allows them to be more accessible but It can also be used for safer storage such as in another location than your computer.

Having several computers or other local network storage solutions, allows one to keep current copies of you data using rsync . This gives fast and trusted storage. Mistakes does though replicate quite easily onto the other devices.

Some storage solutions don’t give you the physical control and thus the trust will be less.
Amazon S3 and almost every webhost there is gives you the ability to store any kind of data. These services can be accessed from any internet connection. The confidentiality depend on how much you trust the company but usually the availability is high.
A friends computer can also be considered as a network storage. Trust of this solution can be higher or lower than comercial services depending on your friend. Usually though availability is less both in network speed and uptime.

Transport

So far I have mentioned physical transport and the internet where I assume single point to point communication such as ftp or rsync. There are some more interesting ones.

“Bittorrent”:http://en.wikipedia.org/wiki/BitTorrent_%28protocol%29 is sometimes as a joke described as a very good backup solution. It is a protocol used for spreading files between many users. It allows the files to be stored in chunks on many computers giving high speed access and redundancy. The drawbacks is that you are dependent on other users to hold and share the data which is less likely for your private files. Also bittorrent is not designed in any way for confidentiality, the files are accessible to anyone.

OneSwarm is described as a friend to friend file sharing protocol. The communication model looks interesting where you access the others storag and if not already done it could be extended to allow pushing files onto eachother rather than just pulling. It also introduces a service transparency where you don’t have to know exactly where the files are stored, they could be stored both on the computer you are talking to but it could also redirect the traffic in a way that start to look like somewhat related to the way TOR allows communication to be anonymous.

Finally there is a producet/service
Baqup , it’s website has very little information but from a third party it seems to be a backup solution where the data is stored on friends and families computers as well as hosted services.
They have come quite close to what I’m looking for, although that will not prevent me from reaching further.

To come

With this knowledge of storage options I would not look at the solutions but my own files. Talking about storage and backup I would classify them using the following scales.

Confidentiality, do I want the file to be accesible by only me, my friends or anyone. Here I see a general model of encryption used to secure data as well on DVDs as on remote storage.

Availability, considering backup there is no problem if the files are stored on friends computers or on a DVD stored in a remote location. Daily usage files might be more interesting to have more available. Network speed might also affect the choice of what files to store remotely or not.

I see all options of storage mentioned here as possibillities. They could be from service providers, friends or even unknown people. The last one is interesting because even though you cannot trust it to 100% be available you could still in practice get fully satisfactory result with a fallback on local CDs.

With this I tried to summarize what I have seen so far and that has inspired me in thinking of new backup, storage and distribution solutions. Later I will write a more specific description about what to come but for now I gladly accept your interpretations or other crazy ideas.

Update

Friend backup: http://www.cucku.com/

You can leave a response, or trackback from your own site.

One Response to “Inspiration for swarm storage”

  1. [...] previous post Inspiration for swarm storage, was the initial inspiration for this design. Some of the goals from that post have been [...]

Leave a Reply