Here is my shot at it:

• This is a new world
• This is a different world
• Where listening and watching is copying
• Copies that are modified and copied again by others
• And no one may stop anyone from doing it

The OpenVZ parts still work but will not collect any information unless you run it under an OpenVZ system.

New since last post is another network traffic visualization, remote server ping monitoring, per user cpu and memory usage monitoring. Most important feature though is the automatic detection of users and network interfaces, you no longer need to configure which network card and users to monitor, now they all are recorded.

Network visualization

I am running a TOR node which generates a lot of traffic that usually hides all other traffic. My solution here is a diff plot that visualizes the difference in input and output traffic.

As can be seen around the 19th and 20th the yellow spikes show output traffic when the symmetric traffic is subtracted. The total assymetric output was 3.78 GB.

Another issue that I cannot truly say I have solved is the spikes of transfer that sometimes occur when I transfer large amounts of data between servers. Since the upper bounds in the graph is determined automatically from the data, a short time hig bandwidth transfer will scale the graph so that the normal level traffic is barely noticeable.

My attempt to solve this is by using a logarithmic scale. As this kind of scale does not work with negative numbers I designed another graph with the sum yellow and below the diff overlay. The diff is represented by color where magenta represents the extra output traffic compared to the input, and cyan the other way around.

I don’t like this representation very much for two reasons. One the log scale fail to give a relevant visual representation of this kind of data, the output overlap looks like it is half the traffic of the total traffic, but since we know it is logarithmic we also know that it’s not the fact. Two, it is very hard to notice any irregularities such as spikes in the data transfer.

Remote server ping

The latest contibution is the remote ping monitoring. It is a simple ping rtt(round trip time) monitoring that came from the need to know for how long a server has been unreachable. It is also good to monitor local internet connection quality.

With the current configuration, each measurement is done using three ping requests and the average rtt is recorded.

This feature must be configured in the config.py file, you can use the template file config-dist.py.

User CPU/memory usage

As with the network monitoring the per user memory usage is now also automatic, there is no need to configure which users to monitor – all is monitored.

As an addition we now also monitor the CPU usage per user. If you run on a multicore server 100 equals full utilization of one core.

How to get it

Download it from the git repo:

git clone http://src.endnode.se/git/csm

The Proxy Solution

The answer to these need is writing a new http proxy intended for personal use. The main goal of the proxy is a modular filtering design. These filters can be applied on both requests and responses. They can modify the requests before being sent and also totally block them.

Using a proxy makes it easier in the future for me to change browser and keep my filtering with minor setup. Even running multiple browsers at the same time using the same live settings would be possible.

By writing this program myself I will make it easier for me to later add one hour hacks that I want to try out.

Program design

I want the program to be easy to use for average users, probably a single executable where all configuration is done in the web interface.

To achieve this simplicity the bar for making additions to the program is raised. Filters are described as modules but they are build into the binary. New modules means a new binary, that will work by simply replacing the binary executable. For other developers this mean that when you write a new module it must be included into the source tree.
This stand of “module” design  is untested and I will gladly accept any ideas on this. Practically there could be a more external modules design using dll or similar.

Personal Proxy

The project is named “Personal Proxy” and is available at BitBucket.org under phq/personal-proxy.

The program i written in C# and developed in Monodevelop but it works in VisualStudio as welll.

Source code can be downloaded using the download links on the project page. However if your want to conrtibute I suggest you use mercurial and get the source code from the bitbucket repo.

hg clone https://bitbucket.org/phq/personal-proxy

If you already know the basics in http you can skip to the privacy pad part.

What your browser reveal about itself

For every page you visit, your browser must first download the page. This is done by sending a request to the server that usually looks like this:

GET / HTTP/1.1
Host: notes.endnode.se
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.2pre) Gecko/20100130 Ubuntu/9.10 (karmic) Namoroka/3.6.2pre
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.7,sv;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: UTF-8,*
Keep-Alive: 115
Connection: keep-alive


This is sent with every request for each html page, css-stylesheet and image that is requested. Especially the User-Agent line is mostly addressed since it is not only complex in contents but also that this varies a lot between different computers.

These headers can be combined into a fingerprint that the browser will reveal with every request.

Inverse Privacy Pad

The inverse privacy pad is an online notepad where you can write some notes and save them for later. Each visitor is presented a notepad based on their browsers fingerprint. This way every visitor will see a different text in their where their browser configuration differs.

Notes saved can only be accessed by others using the exact same configuration(or faking the same headers). Therefore the more unique your browser is, the easier you will be to track but you can be sure that there is a less chance for others to read you notes on this page.

Try it out yourself at the inverse privacy pad.

First of all, this is not a typical example of writing a tcp server in python. As a beginner you might still be able to get some knowledge, but to be able to understand the whole example you must understand threads and python decorators.

I want a program that listens on a tcp port. When a remote connection is established it will be connected to the terminal so that when I type on the terminal it is sent to the remote machine and when the remote machine send text it is printed on the terminal. Think of it as a reversed telnet client.

My intuitive solution

To make this work I need one function that listens for new connections, waitConnect(). When I get this new connection I initiate two new functions. First socket_reader() which read all incoming data and print it to the screen, stdout.write() and then terminal_reader() which will read what I type and send it to the connection, con.send(). Here is what it looks like:

Simple TCP server

The code

A simplified version, without error checking, looks like this:

def terminal_reader(con):
	while True:
		line = sys.stdin.readline()
		con.send(line.encode('utf-8'))
def socket_reader(con):
	while True:
		data = con.recv(1)
		sys.stdout.write(data.decode('utf-8'))
def waitConnect(s):
	while True:
		connection, address = s.accept() #
		sr = socket_reader(connection)
		tr = terminal_reader(connection)

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("127.0.0.1", 8082))
s.listen(5)
waitConnect(s)

Simple right?

Everyone who is somewhat familiar with programming have already noticed that there is something missing. Ordinary functions would not work alone.

Many of the internal function calls within our functions are blocking, such as accept(), readline() and recv(). Without threads it is impossible to use blocking functions and still have the program responsive to both tcp input and terminal input at the same time.

Event magic

To make this possible we will do some slight modification to our program. We add the decorator @ed.function to all our three functions and in the end of our code we add ed.join(). Finally we must also add an import ed in the beginning of the code.

The ed is a library I wrote in python to accomplish something similar to the example above. It initiates a queue and a number of worker threads. The @ed.function decoration modifies the functions so that instead of being called immediately they will be queued. The worker threads will then run each function call. Finally the ed.join() will wait until the queue is empty and then exit.

What we have now is events in the form of function calls.

This is a proof-of-concept with only the basic functions as presented here. The code can be downloaded using “git clone http://src.endnode.se/git/event-demo“.