0.9.28 has better enforcement of DHT upload rate limit for non-locally generated requests.

the limit applies to: DHT replies, refreshes of stored items, checking for status/tracker and proxy server.

local requests (eg. dhtgets from the UI) are excluded from this limit, so user experience is not affected.
in other words: we limit only the band contributed back to twister network for maintenance tasks.

this should improve resilience against some sorts of denial-of-service attacks and also
prevents saturating the uplink as observed recently.

I’d like to share some thoughts on stuff i’d like to implement or improve in twister…

Following the recent introduction of “getmentions” api in twister-core 0.9.27, i’d like to explore the possibilities for creating new user following “types”.

Currently we have only two following types: public and private.

1) Public: the username is stored in public DHT and twister-core starts a torrent to follow this user. Public following are synchronized between computers (twister-html read back the listing from DHT)

2) Private: twister-core starts the torrent, but username is not announced to DHT. Therefore the posts appear in timeline but people don’t know you are following him.

The interface for “Private” is actually awkward / poorly designed: there is no option to follow privately from the start, one has to public follow (propagates the name to DHT) and then change it to private (removes the name from DHT). So for a brief time window one will know about that.

So one obvious improvement here is to change the interface. Perhaps opening a box to inquire the following type would be the least layout disrupting, also allowing for more types:

3) DM/Mentions only: this is a kind of “mute”, that is, the user might be too lousy so you want to follow him form mentions and DMs but not to polute your timeline. This is actually orthogonal to public/private setting.

4) Restrict charset: i’m not sure this should be a per-user setting or global one. The idea is to limit showing only latin, or russian or chinese posts.

Any other suggestions for following or per-user settings?

The above ideas only require HTML/Javascript coding. If someone wants to help it will more than welcomed

Then there is something i’d like to explore in twister-core: bandwidth limiting.

I’ve being noticing that sometimes twisterd saturates my upload link and it seems to be due to high DHT traffic. I still need to investigate if this is caused by some bug in other twisterd nodes (requesting too much) or if someone is aggressivelly crawling the network.

Either way, i was thinking about creating a new setting to limit the amount of bw that twisterd may use to serve the other nodes. This also improves / works as DDoS protection.

My idea is that DHT requests originated on your own host would always be allowed to go out, while replies to other nodes would be subject to bandwidth accounting.

Comments?

Yes, twister is already decentralized. However there are two areas where we may improve twister to make it more resilient:

If you prefer to contact me by email its also mfreitas, but on gmail.com domain.

A new feature is available from latest twister repository (both twister-core 0.9.26 and twister-html must be updated), it is called “Direct Message copy-to-self”.

DM’s copy-to-self will allow users to safely store the private messages they send and also sync these messages between twister instances.

Originally, the sent DMs were discarded, only a local copy were kept in order maintain the user interface sane… However, if one erases his own local twister directory, the sent DMs are lost. Only the recipient would be able to decrypt them back from torrent.

The new idea, which was suggested and partially implemented by @denis (github @dryabov) fixes this by sending two copies of the message: one is encrypted with recipient’s public key and another one is encrypted with our own public key. Because the encrypted data preserved by twister network, it may be recovered later.

For users who want to revert to old behavior should just go to options page and disable “Direct Message’s copy to self”.

We just got another great theme contribution merged into twister-html: the ‘Nin’ theme by @myleneb! Thanks!

twister project is very lucky to receive such talented contributions as both Nin and Calm (by Hedgehog) are so beautifully designed. I will add some screenshots to our twister’s main site, but in the meantime one may check their github pages for some images:

https://github.com/myleneb/twister-nin

https://github.com/myleneb/twister-nin

Now we should promote a poll among twister users:

- Which theme should we set as default? Original, Calm or Nin?

A pre-configured VirtualBox Gitian image based on Ubuntu 12.04 LTS x86_64 + gitian + twister sources is now available for download (3.2 GB file):

ubuntu_12.04_amd64_gitian_twister.ova (download from Mega)

ubuntu_12.04_amd64_gitian_twister.ova (download from My Air Bridge)

ubuntu_12.04_amd64_gitian_twister.ova (download from Dropbox)

ubuntu_12.04_amd64_gitian_twister.ova (download from Google Drive)

The md5sum of the file is:

48f62fce1ba105cdcd172b8de6a2466a ubuntu_12.04_amd64_gitian_twister.ova

The Ubuntu desktop contains a simple README text file with very basic gitian instructions on building both 32 and 64-bits native Windows executables (using mingw, not cygwin).

One just have to boot the VirtualBox and execute a couple of commands to create an updated twisterd.exe from twister-core github repository.

For more information on the building process refer to Gitian documentation.

For instructions on configuring Gitian on an existing Ubuntu check this: Build native Windows client using Gitian

We have just reached an important milestone: twister is now compatible with Tor!

/

As some people may be aware, the blocking issue from Tor compatibility was the DHT part. Because our DHT requires UDP and Tor doesn’t support it (and I’d guess it may probably never implement such thing) that was pretty much a showstopper.

Still, running twister on top of Tor was a long time goal. As discussed in the white paper ( /?attachment_id=355 ), the Tor support allows a far more interesting threat model.

The solution was to implement the routing of the DHT traffic through other twister peers. As we already have an unstructured TCP-based P2P overlay network derived from the Bitcoin protocol, I have just extended this protocol with the new DHT relaying commands.

So, a few minutes ago, I’ve been able to successfully run twister over Tor for the first time, with all functionality. That includes downloading the blockchain, using DHT to obtain posts, profiles, avatars and the swarm peer lists from trackers, and also synchronizing post’s torrents. All the 3 network overlays were tunneled by the Tor socks5 proxy.

In order to try this feature just add the “-proxy=127.0.0.1″ argument to twisterd command line (assuming your Tor daemon is running on localhost with port 9050).

It might take some time to reach a few relaying nodes though. This is probably going to improve over the next few days as people update their twister clients.

I’m about to commit a patch that implements tunneling of the DHT traffic over TCP. The motivation is to make twister compatible with TOR (they do not support UDP, required by DHT).

So how this thing works?

The premise is that most of people will still be running twister outside TOR, thus with full internet connectivity including UDP. These nodes will behave as “DHT proxies” to nodes without UDP, relaying the traffic back and forward.

The TCP connection used is the same as the registration network, that is, the Bitcoin P2P protocol, now extended with DHT requests and replies commands.

Of course, some limits will be enforced, like maximum number of simultaneous DHT requests and total requests/second. If node misbehave it will be banned (Bitcoin closes connection).

Nodes requiring DHT proxy do not rely on any particular nodes for that service: they will randomly choose four connections to use for each DHT request they made. This should help to balance the load and also adds redundancy.

So far I have only tested it locally, so you might guess this is pretty much experimental.

Besides, I don’t think we are tor-ready yet: the DHT traffic used for profiles, avatars etc has been tunneled but not the internal DHT requests from libtorrent (ie. tracker requests).

But we are getting there…

A new twister-core version 0.9.21 is now available in github!

Now we have an interesting new feature: RSS feed support

https://github.com/miguelfreitas/twister-core/pull/204

Feedback is appreciated!

Also, there are 3 important bugfixes i’d like to mention:

1) wait twister threads to finish before destroying libtorrent session (also known as “on stopping daemon: SIGABRT, throwing an instance of ‘libtorrent::libtorrent_exception’, what(): session is closing”)

https://github.com/miguelfreitas/twister-core/issues/200

2) Don’t assume the fastest dht node to be the most up-to-date on a given resource.
(that is, now we also enable multiple replies collecting for non-multi entries. default parameter is a very small timeout for additional replies: that’s a tradeoff between probability of getting an older value x slowing the UI.)

https://github.com/miguelfreitas/twister-core/commit/5eb40d9ffd1c83d8dd29a1ce2127f3f6bf65e8d5

Note: This is a general improvement, previously it was not uncommon to see old avatars, profiles or outdated status.

3) getposts memory leakage (also known as “when twister-html started: twisterd grabs too much RAM, continously”)

https://github.com/miguelfreitas/twister-html/issues/161

Enjoy!