Scripting example

New twister features may be implemented on top of JSON RPC interface using any language of choice.

Consider, for example, the problem of searching users for their partial name, location or email. Because the information we need is completely distributed on the network (each profile is stored in a different DHT resource so the full dataset is scattered all over the world) it is very difficult for the twister daemon to provide such functionality.

However a twister crawler can be easily implemented in python, as shown in twister repository file contrib/usernameCrawler.py below. Once we have the entire data combined in a single place then it becomes easily searcheable. In fact, any node of the network is technically able to host and provide such service for twister users.

#!/usr/bin/python
#
# This sample script is a username crawler: it will obtain all known usernames
# from block chain and then try to download avatar and profiles for all of
# them. The report is shown as an html file.
#
# Downloaded data is cached in a python pickle file, so it may be executed
# again and it won't need to get everything all over again (you may run it
# from cron scripts, for example)

import sys, cPickle, time

dbFileName = "usernameCrawler.pickle"
htmlFileName = "userlist.html"
cacheTimeout = 24*3600

try:
    from bitcoinrpc.authproxy import AuthServiceProxy
except ImportError as exc:
    sys.stderr.write("Error: install python-bitcoinrpc (https://github.com/jgarzik/python-bitcoinrpc)\n")
    exit(-1)

serverUrl = "http://user:pwd@127.0.0.1:28332"
if len(sys.argv) > 1:
    serverUrl = sys.argv[1]

twister = AuthServiceProxy(serverUrl)

class User:
    avatar = ""
    fullname = ""
    location = ""
    updateTime = 0

class MyDb:
    lastBlockHash = 0

try:
    db = cPickle.load(open(dbFileName))
    nextHash = db.lastBlockHash
except:
    db = MyDb()
    db.usernames = {}
    nextHash = twister.getblockhash(0)

while True:
    block = twister.getblock(nextHash)
    db.lastBlockHash = block["hash"]
    print str(block["height"]) + "\r",
    usernames = block["usernames"]
    for u in usernames:
        if not db.usernames.has_key(u):
            db.usernames[u] = User()
    if block.has_key("nextblockhash"):
        nextHash = block["nextblockhash"]
    else:
        break

now = time.time()
for u in db.usernames.keys():
    if db.usernames[u].updateTime + cacheTimeout < now:

        print "getting avatar for", u, "..."
        d = twister.dhtget(u,"avatar","s")
        if len(d) == 1 and d[0].has_key("p") and d[0]["p"].has_key("v"):
            db.usernames[u].avatar = d[0]["p"]["v"]

        print "getting profile for", u, "..."
        d = twister.dhtget(u,"profile","s")
        if len(d) == 1 and d[0].has_key("p") and d[0]["p"].has_key("v"):
            db.usernames[u].fullname = d[0]["p"]["v"]["fullname"]
            db.usernames[u].location = d[0]["p"]["v"]["location"]

        db.usernames[u].updateTime = now

cPickle.dump(db,open(dbFileName,"w"))


from HTML import HTML
from cgi import escape
def outputHtmlUserlist(fname, db, keys):
    h = HTML()
    head = h.head("")
    with h.body(""):
        with h.table(border='1', newlines=True):
            with h.colgroup:
                h.col(span="1", style="width: 64px;")
                h.col(span="1", style="width: 130px;")
                h.col(span="1", style="width: 250px;")
                h.col(span="1", style="width: 250px;")
            with h.tr:
                h.th("avatar")
                h.th("username")
                h.th("fullname")
                h.th("location")
            for u in keys:
                with h.tr:
                    with h.td():
                        h.img('',src=escape(db.usernames[u].avatar), width="64", height="64")
                    h.td(u)
                    h.td(escape(db.usernames[u].fullname))
                    h.td(escape(db.usernames[u].location))
    open(fname, "w").write(str(h))

print "Generating", htmlFileName, "..."

keys = db.usernames.keys()
keys.sort() # sorted by username
outputHtmlUserlist(htmlFileName, db, keys)

5 comments on “Scripting example
  1. Daniel Smith says:

    can i ask the reason why you used python in your example?
    any specific reason?
    the thing i am wondering, and i have been looking around to
    see the programming language twister is created in, is wouldn’t it be better to write
    in C++, since that’s what libtorrent is in?
    just learning,
    thanks

    • mfreitas says:

      Daniel, python is much easier to write simple scripting examples like that. In fact, python is probably easier for almost anything ;-)

      Just because the core is written in C++ it doesn’t mean it would be easier to use. To use C++, for example, one would need to link against some library providing RPC+JSON+HTTP client etc. In python we still require an external RPC lib, but this is much easier to install, and everything else is already provided.

  2. Hi,
    I came across your project, while working on something a bit similar, see http://torlus.github.io/2014/05/08/mqtt-dht/

    The “partial username” search issue is quite close to the MQTT topic search issue that I want to address.
    In your case, usernames are stored in the blockchain, which might be applicable to topics.

    But I thought also about something in the likes of a “hashtag search”.
    Correct me if I’m wrong but it seems that your trending hashtags are stored in the DHT.
    I’ve been thinking about using these: http://en.wikipedia.org/wiki/Locality-sensitive_hashing
    Such hashes could be distributed across the DHT the same way the Node IDs are. However their more deterministic nature could help in finding the closest matches, which should theorically be close to each other.

    What do you think about it ?
    Regards,
    Greg

    • mfreitas says:

      I think the comments area is not the appropriate space for technical discussions ;-)

      If you want to discuss twister development ideas, please join twister-dev group.

  3. You’re right :)
    I did it and posted my comment. Please delete these ones if you want to.
    Regards,
    Greg

Leave a Reply to mfreitas Cancel reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>