The Proxy Solution

The answer to these need is writing a new http proxy intended for personal use. The main goal of the proxy is a modular filtering design. These filters can be applied on both requests and responses. They can modify the requests before being sent and also totally block them.

Using a proxy makes it easier in the future for me to change browser and keep my filtering with minor setup. Even running multiple browsers at the same time using the same live settings would be possible.

By writing this program myself I will make it easier for me to later add one hour hacks that I want to try out.

Program design

I want the program to be easy to use for average users, probably a single executable where all configuration is done in the web interface.

To achieve this simplicity the bar for making additions to the program is raised. Filters are described as modules but they are build into the binary. New modules means a new binary, that will work by simply replacing the binary executable. For other developers this mean that when you write a new module it must be included into the source tree.
This stand of “module” design  is untested and I will gladly accept any ideas on this. Practically there could be a more external modules design using dll or similar.

Personal Proxy

The project is named “Personal Proxy” and is available at BitBucket.org under phq/personal-proxy.

The program i written in C# and developed in Monodevelop but it works in VisualStudio as welll.

Source code can be downloaded using the download links on the project page. However if your want to conrtibute I suggest you use mercurial and get the source code from the bitbucket repo.

hg clone https://bitbucket.org/phq/personal-proxy

If you already know the basics in http you can skip to the privacy pad part.

What your browser reveal about itself

For every page you visit, your browser must first download the page. This is done by sending a request to the server that usually looks like this:

GET / HTTP/1.1
Host: notes.endnode.se
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.2pre) Gecko/20100130 Ubuntu/9.10 (karmic) Namoroka/3.6.2pre
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.7,sv;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: UTF-8,*
Keep-Alive: 115
Connection: keep-alive


This is sent with every request for each html page, css-stylesheet and image that is requested. Especially the User-Agent line is mostly addressed since it is not only complex in contents but also that this varies a lot between different computers.

These headers can be combined into a fingerprint that the browser will reveal with every request.

Inverse Privacy Pad

The inverse privacy pad is an online notepad where you can write some notes and save them for later. Each visitor is presented a notepad based on their browsers fingerprint. This way every visitor will see a different text in their where their browser configuration differs.

Notes saved can only be accessed by others using the exact same configuration(or faking the same headers). Therefore the more unique your browser is, the easier you will be to track but you can be sure that there is a less chance for others to read you notes on this page.

Try it out yourself at the inverse privacy pad.

First of all, this is not a typical example of writing a tcp server in python. As a beginner you might still be able to get some knowledge, but to be able to understand the whole example you must understand threads and python decorators.

I want a program that listens on a tcp port. When a remote connection is established it will be connected to the terminal so that when I type on the terminal it is sent to the remote machine and when the remote machine send text it is printed on the terminal. Think of it as a reversed telnet client.

My intuitive solution

To make this work I need one function that listens for new connections, waitConnect(). When I get this new connection I initiate two new functions. First socket_reader() which read all incoming data and print it to the screen, stdout.write() and then terminal_reader() which will read what I type and send it to the connection, con.send(). Here is what it looks like:

Simple TCP server

The code

A simplified version, without error checking, looks like this:

def terminal_reader(con):
	while True:
		line = sys.stdin.readline()
		con.send(line.encode('utf-8'))
def socket_reader(con):
	while True:
		data = con.recv(1)
		sys.stdout.write(data.decode('utf-8'))
def waitConnect(s):
	while True:
		connection, address = s.accept() #
		sr = socket_reader(connection)
		tr = terminal_reader(connection)

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("127.0.0.1", 8082))
s.listen(5)
waitConnect(s)

Simple right?

Everyone who is somewhat familiar with programming have already noticed that there is something missing. Ordinary functions would not work alone.

Many of the internal function calls within our functions are blocking, such as accept(), readline() and recv(). Without threads it is impossible to use blocking functions and still have the program responsive to both tcp input and terminal input at the same time.

Event magic

To make this possible we will do some slight modification to our program. We add the decorator @ed.function to all our three functions and in the end of our code we add ed.join(). Finally we must also add an import ed in the beginning of the code.

The ed is a library I wrote in python to accomplish something similar to the example above. It initiates a queue and a number of worker threads. The @ed.function decoration modifies the functions so that instead of being called immediately they will be queued. The worker threads will then run each function call. Finally the ed.join() will wait until the queue is empty and then exit.

What we have now is events in the form of function calls.

This is a proof-of-concept with only the basic functions as presented here. The code can be downloaded using “git clone http://src.endnode.se/git/event-demo“.

Currently the main goal is to have a system for encrypted off-site backup allowing for incremental backups.

This will be a slightly technical post, but remember that anything, even the goals, can change.

The goals in more detail:

• Encrypted – the backup might be stored on less trusted locations
• Effective storage – incremental backup
• Redundancy – have the backup stored on multiple locations
• Traffic efficiency – allow to share the data between storage servers to minimize the traffic to the local storage
• Local storage servers as well as remote, friends computers, removable disks or purchased networked storage
• Share data among users, once the backup is already on a friends computer, sharing it will be much faster
• Data instead of connection focused – the “protocol” should work offline without direct connection between server and client.

Server – Client design overview

The design will be separated into two parts. The local client and the storage server.

The client will handle the filesystem, determine what to backup, preparing the data to be backed-up, keeping the cryptographic keys as well as deciding how the backups should be distributed.

The server will not work with files as the client, instead it will work with smaller units of data, called chunks, that the client prepares. The main task for the server will be to receive and store these units as well as distribute them when requested by the client.

Therefore the communication between the server and client is only about chunks, no files, therefore the system might be used to store other kind of data other than files.

Local file-system

Snapshot

This is partly specific to the client. Other implementations might function differently and still be able to work with the same server.

One or more folders are selected to be backed-up. These will be indexed into a traditional file tree. Each file will be stored in a specific file structure. This tree with all its file meta-data will be combined into a snapshot. This snapshot will itself be in the form of a chunk that will be described later.

The client can further have other features such as automatic monitoring and backup.

File storage

File structure

A file will have the most basic attributes such as name and change date.

The more specific part here is that the file will be split into several chunks. These will be the fundamental storage unit in the final backup system.

Thus the file structure only contain the ID to fnd these chunks, not their data.

Chunk

Chunk

Each chunk is a piece of data from the file that is encrypted before being sent to the remote backup.

There is two parts identifying the data. First the ID that is an encrypted hash of the cleartext data. The second part is a hash of the encrypted data. There is also an encrypted key that follows each chunk that is used to decrypt the data.

The ID and decryption key comes in pairs for each chunk. Every new pair represents a new local key-change or a new user that is given access to the data.

The shared ID principle is used to make it possible for a single server to identify identical chunks of data that does not have to be transferred. One problem with current design is that M might upload false data with a specific chunk id that correlates to some data that A might later upload. When A is going to upload the correct data the server will believe it already has got the data and stop the upload. Thus the true data will not be backed-up. So far there is no way for the server to verify the ID hash since it does not have access to the data. Comments on this is very appreciated.

Order

Since the design will work with both online and offline transports we introduce the order.

An order is a document signed by one user. The document states what chunks a specific server will store or release.

Order

The figure is currenlty missing target server to which the order applies.

Once a server receives an order it will start filling up with the chunks. They can come from the client computer or another server. Since the order verifies the data there is no need to further verify from where the data comes. Therefore an order with its chunks payload may be transferred on a removable disk and later uploaded to the server from an insecure machine.

The main operation can be one of the following:

• Put – store data on the server
• Get – retrieve data from the server
• Update – add new keys and ID:s to existing chunks

Some additional conditions could be included in the order such as expire date.

In the end the order contain a list of Chunk ID:s that indicate what chunks this order affects.

Put

A put order authorizes that the server may store the listed chunks on the server. The server may get these from the users local machine but could also retrieve them from other servers.

Get

A get order authorized retrieval of data. This indicates that one user or server is allowed to retrieve the chunks from that server.

Although you must have the private key to decrypt the data, it is still of interest to limit ones access to this data.

Update

This order updates the chunks with more pairs of key and ID from new users.

Key handling

We have not been specific in this first draft about what keys there are, but we still have a few thoughts.

The system will be setup so that a secret key for decryption can be stored elsewhere. This key will be the only one necessary to restore the backup from a new computer.

There could be only one pair of asymmetric keys for both encryption and hash signing, or they could be two different.

Similar protocols

The previous post Inspiration for swarm storage, was the initial inspiration for this design. Some of the goals from that post have been met.

So far the server has been described as an active one with a specific software that verifies signatures and minimize the traffic. Another option is still to use an existing service such as a ftp account. This would then need ftp support into the client.

Furthermore the snapshot might have more similarities with the bittorrent protocol and could thus be modified to be able to use existing trackers to initiate communication between client and servers.

It is done using tcpdump to collect the data and a program I have written to extract the response time.

This will monitor the delay between the clients request and the first response from the webserver. It works with other services as well as long as they follow the same request-response pattern as in http.

Recording

The recoding on the server is done using tcpdump. Run the following shell command as root to store the result into web.log.

tcpdump -tt -p -n "port 80 and tcp[13] & 8 != 0" > web.log

This will listen for packets on the first interface, if you want to specify which one, use “-i eth0″.

-tt will print the timestamp unformated.

-p makes sure the interface is not in promiscuous mode, which we don’t need since we know exactly what we are listening for.

-n will output all addressas in number format, no dns lookups.

Finally comes the filter that tells tcpdump what packages to record.

port 80 - only capture package to or from port 80 - our web server is at that port.


tcp[13] & 8 != 0 - only capture packages with payload/data. This will discard all other packages used to initiate the connection and ACK.


Extraction

The program called webresponse will parse the log using the following command:

cat response.log | mono webresponse.exe 123.123.123.123 > delay.log

The output looks something like this:

1263241029.252353 1.05
1263241030.639528 0.924
1263241031.623903 2.212
1263241032.659558 118.8014
1263241032.974350 12.255
1263241034.054304 1.254
1263241036.442082 1.404

First comes the unix time followed by the response delay in milliseconds.

Source

WebResponse source code can be downloaded using git from http://src.endnode.se/git/webresponse

git clone http://src.endnode.se/git/webresponse