Storj

It’s been awhile since I had a chance to write. Looks like having a kid has this effect and I don’t mind :) But now as I’m writing this text I am thinking about the past few months and what they were about. I changed my job and joined a DevOps team, I also got neck deep into the whole blockchain/crypto currency world. So maybe it’s time to add a crypto section in here!

And the first entry of this section starts right about now (even so I did kind of added the Playing with Blockchain one too :P). This story is all about Storj. I’m not very deeply involved with it but it’s something that’s on my mind lately so I’ll share why.

Storj is a blockchain based product that will compete with products like the Amazon Web Services S3 buckets. It’s essentially a distributed data storage. Within the blockchain network the nodes hold data and users use it. If you throw a file in there you’re also paying a small fee as long as it is kept by the network. It’s completely decentralized and as more people comes into the network and share their disk space more people can enjoy their service. So in that regard we have two types of people here:

  • People who share their disk space and earn some kind of value (the Storj crypto coin but potentially other kinds of currencies as well)
  • People who use the shared space and the security of having their data encrypted and in enough places (redundancy) so it is always available

If you have ever encountered services like the AWS S3 you probably enjoyed the advantage of throwing your data in the buckets and use it as object storage. For example if you serve a web application that takes a lot of photos you can store those photos in a bucket, you can store your frontend in another and even parts of you CDN libraries in another… You get the point. The whole thing can be done via code which enables easy application integrations. For example using Ruby the example looks like this:

require 'aws-sdk-s3'  # v2: require 'aws-sdk'
require 'json'

profile_name = 'david'
region = "us-east-1"
bucket = 'doc-sample-bucket'
my_bucket = 'david-cloud'

# S3

# Configure SDK
s3 = Aws::S3::Client.new(profile: profile_name, region: region)

# Display a List of Amazon S3 Buckets
resp = s3.list_buckets
resp.buckets.each do |bucket|
  puts bucket.name
end

# Create a S3 bucket from S3::client
s3.create_bucket(bucket: bucket)

# Upload a file to s3 bucket, directly putting string data
s3.put_object(bucket: bucket, key: "file1", body: "My first s3 object")

# Check the file exists
resp = s3.list_objects_v2(bucket: bucket)
resp.contents.each do |obj|
  puts obj.key
end

# Copy files from bucket to bucket
s3.copy_object(bucket: bucket,
               copy_source: "#{my_bucket}/test_file",
               key: 'file2')
s3.copy_object(bucket: bucket,
               copy_source: "#{my_bucket}/test_file1",
               key: 'file3')

# Delete multiple objects in a single HTTP request
s3.delete_objects(
  bucket: 'doc-sample-bucket',
  delete: {
    objects: [
      {
        key: 'file2'
      },
      {
        key: 'file3'
      }
    ]
  }
)

# Verify objects now have been deleted
resp = s3.list_objects_v2(bucket: bucket)
resp.contents.each do |obj|
  puts obj.key
end

You can find that example here.

Even without much explanation it’s pretty easy to navigate this code. It uses a gem library called ‘aws-sdk-s3’ and configures its client with some settings like the AWS region. It then lists the buckets, creates a bucket, upload files, etc. You can enrich this configuration with your API keys to authenticate. If you ever go that far I suggest to have a good look also at AWS IAM. You can configure an EC2 node to have roles that are authorizing it without the need to throw API keys in your code. That might be a safer way in many cases but anyway - back to Storj.

How does a code using Storj looks like? Here it is:

require 'libuv'
require 'ffi'

module Example
  extend FFI::Library
  # load built shared object from C/C++ using ruby ffi
  ffi_lib '/path/to/libexample.so'

  # NB: if `libexample` is installed in the usual system location*
  # ffi_lib 'example'

  # expose C functions as members of the `Example` module
 attach_function('queue_work', 'example_queue_work', [:pointer, :int], :int)
end

handle = FFI::Function.new(...) {...}
cb     = FFI::Function.new(...) {...}

reactor do |reactor|
  reactor.work do
    Example.queue_work handle, cb
  end
end

Yeah, ffi… I know. For those of you who don’t get the example I’ll just say that it’s using C++ library and calls its functions via C binding interface. It’s not very convenient since it’s not really a Ruby library. Keep in mind that Storj is in a very early stage of its development. It’s a child even by the standards of the Crypto community but it’s one of the successful examples of what Blockchain could give to the world. Think about it - you can access cheap storage from within your application not caring about servers. Just like a service from a cloud provider but cheaper. On the other hand people share their space and earn money. I mean - how cool is that?

I’m very invested in the idea of people sharing. The micro transactions of value enable such services. I looked at the white paper of Storj and I saw that the micro transactions happen every time the client (the guy who is using the space) gets assurance that his data is available and can be retrieved. The nodes keeping the data and the ability to send it at any moment are getting paid automatically. When I have more time I might try and explain some of the technicalities that I see in the white paper but for now I’ll just leave a link to it. It’s here:

Storj White Paper

For those who are curious to know how does it feel to be on the “sharing side” I will show you an output of a Javascript library used for the sharing. It’s also making your node a part of the Storj Blockchain network.

┌─[kstaykov@phoenix]─[~]
└──╼ $ ~/node_modules/storjshare-daemon/bin/storjshare.js status

┌─────────────────────────────────────────────┬─────────┬──────────┬──────────┬─────────┬───────────────┬─────────┬──────────┬───────────┬──────────────┐
│ Share                                       │ Status  │ Uptime   │ Restarts │ Peers   │ Allocs        │ Delta   │ Port     │ Shared    │ Bridges      │
├─────────────────────────────────────────────┼─────────┼──────────┼──────────┼─────────┼───────────────┼─────────┼──────────┼───────────┼──────────────┤
│ 28f50a37a07902a585a2001f077b5c1bfded5681    │ running │ 1d 5h 4… │ 0        │ 32      │ 46            │ -1ms    │ 4955     │ 324B      │ connected    │
│   → /vms/storj                              │         │          │          │         │ 2 received    │         │ (Tunnel) │ (0%)      │              │
└─────────────────────────────────────────────┴─────────┴──────────┴──────────┴─────────┴───────────────┴─────────┴──────────┴───────────┴──────────────┘
┌─[kstaykov@phoenix]─[~]
└──╼ $

In this example you can see that my node is sharing a directory called /vms/storj for more than a day. It has some peers connected and it shares 324B. Yup, that’s not much but as I said the network is not yet in a production state. The people using it are mostly testers, people who were part of the ICO funding and some like me who wanted to share some space and see what happens. If you want to join me in that regard head on to https://storj.io/share.html and follow the instructions.

Have fun! Until next time… :)

Categories:

Updated: