Hack @ EF, TALKIEN.IO is born

Hack @ EF, TALKIEN.IO is born

Recently I had an opportunity to participate on a hackathon organized by Enterpreneur First. I really want to thank the guys at EF for the opportunity and great weekend! For those who don't know, EF is an investor who invests in talents, entrepreneurs. I really recommend everybody who is into startup world applying for their next cohort. 

It was also a great weekend because I met Miika, Pyry and Pavel. We formed a team to create something cool in 24 hours. It was a huge fun and an honour to work with these guys!

Pavel, me, Miika, Pyry @ EF

We created a browser based application which listens to you while you speak and shows relevant pictures and information instantly.

As we really enjoyed working with each other, we continued the work and published it, so here is TALKIEN.IO. 

We are still searching for a good use case, so any feedback is appreciated! :)

Creating a graph of software technologies with Docker, Neo4j, NodeJS and D3.js Part 2

The plan

In the previous part we created a running Neo4j instance inside a Docker container.
Now we can create a middleware which will call the database's REST API, on the bottom of it's layer and expose an API to the client on the top of it's layer. As we did with the database instance, we will put the Node(s) into containers as well.

Ok, so let's create a Docker image for Node first.

The Node image

The good news is that there is an official Node image on Docker Hub. (Be aware, that does not mean we are super secure, it just means that this image should work properly). Let's check it's Dockerfile.
What we can see is that NPM is installed, so we can manage Node packages easily. Also if we run this image, the Node interpreter will be executed and will be waiting for input.
We are able to create our own image with a running Node + the packages we need.

We could put lots of "RUN npm install [package]" commands inside a Dockerfile, but instead of doing that, we will create package.json file and run "npm install" on it. It will be an external resource. If we run npm install, a node_modules directory will be created. It could be in a separate Docker image, and we could build a new image whenever we change the package.json, but I don't think it makes it too robust by putting the Node instance and the packages together into the same Docker image.

Ok let's do it.

Create the package.json file first.

I am using Express.js as a middleware to help me with the routing and HTTP requests, responses. There are loads of other middlewares to try out, but Express suits perfectly for now. So we will put Express into our package.json as a dependency.
I am also using the request Node module to make POST requests to our Neo4j instance.

Ok, now an app.js file.

I'll leave it empty for now.

Now next to the package.json let's create the Dockerfile.
So while building an image from this Dockerfile, a package.json and an app.js file will be added, and npm install will be called. It will create a node_modules directory and download into it the needed packages.

So, let's build it!

You can see it from the log that it downloaded what we need. Coolio!

The code

Now let's create a simple application which will handle adding a new technology, adding a new connection between two technologies and getting the adjacent technologies of a particular tech. Later we can add some logic for setting weights for connections to visualize how strong the connection is, and we can add weights to the techs as well to show how popular they are.

These are the API endpoints

POST:
/api/tech/new?name=[techname]

POST:
/api/connection/new?A=[Atechname]&B=[Btechname]&connection=[connection]

GET:
/api/tech/all

Let's open our app.js empty file and add our code.

The code is simple. Of course we would like to put the API endpoints, the function's and the configuration into separate files (modules) later, but in this stage we want to have simple solutions.

Ok. Now we need to rebuild our Docker image.

And run our web application. (Don't forget to stop-remove the old container.)

Notice that we are using --link neo4j:db. This creates some environment variables for us that we can use in our webapp for networking purpose.

Just because we are curious let's check these variables in our running webapp container.

Cool! We see that now we don't have to care what is the exact IP address of our Neo4j container. Docker will handle this for us, we just need to use the env variable.

Now we can create some technology nodes with curl or alternative tool for doing HTTP requests. I will execute these requests just to try it out:

1. POST: http://0.0.0.0:3000/api/tech/new?name=HTML
2. POST: http://0.0.0.0:3000/api/tech/new?name=FLASH
3. POST: http://0.0.0.0:3000/api/connection/new?A=HTML&B=FLASH&RELATION=IS_RELATED_TO
4. GET: http://0.0.0.0:3000/api/tech/all?technology=HTML

The last query returns  this JSON:

{"results":[{"columns":["(n1)-[*1]-(n2)"],"data":[{"row":[[[{"name":"HTML"},{},{"name":"FLASH"}]]]},{"row":[[[{"name":"HTML"},{},{"name":"FLASH"}]]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]}]}],"errors":[]}

In the next part we will add the client code and actually draw our graph.

Meetups: Selling to schools

 Selling to schools
Jan 2015

I wanted to learn about how to approach a school with a product (software application).

The key things I learned and did not forget :) :

• teachers, schools live their own life, so don't expect they will find always time doing online chats with you, probably you will have to travel
• getting 10-100 schools on board is not easy at all but manageable, above 100 it is extremely hard
• the UX has to be perfect, don't try to save energy on it
• the online marketing through e-mails is still (40 times) more accurate then using social networks
• schedule e-mails in early mornings between Tue-Thu
• use celebs, famous people to do tweets for you
• ask the teachers for feedback, quotes, they won't give you without asking
• involve the users ASAP after you have something working and STABLE
• be patient and kind!

Anyone who is interested more check out this meetup, and the presenters. They are doing this professionally, really recommended.

Check out the meetup here.

http://www.meetup.com/EdTech-Developers-Meetup/events/218818932/

Thanks to the organizers!

Meetups: Docker London

 Docker London
Jan 2015

I've never been on a Docker meetup, so I was curious and so were lots of others. 

The organizers created a good event. The order of the presentations was perfect. I wish all the meetup organizers would think about how to max out the opportunity in presentations by finding a balance. (So people won't be bored, but won't start loosing attention because of too many details.)

The order was good because Andrew's presentation was a good intro and sum how can Docker fit into  a continuous delivery pipeline and how it is done in their company. Then Johan's presentation was more a demo about GCloud tool  (+ Kubernetes) and the Google Cloud platform which I think was good for those who already knew about tools around Docker, but did not try out Kubernetes. 

In the end Dan Williams's talk about his trip to found out how container ships work in real life was really great, and was really needed for the end of the day to chill out.

All of these talks were about things already done, and existing. I really missed talks about new ideas or possible security issues like the issues with Docker pull, but that is maybe more for a conference talk.

Check the meetup and the agenda here:

http://www.meetup.com/Docker-London/events/218940249/

And thanks to the organizers for creating the event! I really recommend it!

Creating a graph of software technologies with Docker, Neo4j, NodeJS and D3.js Part 1

Feedback highly appreciated

Okey! So what would I like to do?

I would like to experiment with some technologies, which were in my mind nowadays, and have fun. I might later swap some of the pieces (maybe Node to JVM platform or to another client side JS framework). The idea is to visualize the connections between software technologies using this stack.

The imagined problem and solution is the following. (It might turn up to be a good thing, but maybe it will just remain as a prototype.)

Usually we are interested in particular technologies and their relations to others so we start googling. We do this because we know what we want to achieve, but we don't know what tools do we have or we know some of the tools, but we want to see if there are comparable alternatives. We select usually by checking if the technology is free or not, is it mature enough for our purpose, is there a community behind it or maybe a known company, is it maintained frequently and we also sometimes need to predict for how long this technology will be around before a new comes. 

To visualize the technologies I would like to create a web application.

• It will run in a browser so I need some JavaScript libs for sure. I always wanted to try out the powerful D3.js. 
• On the server side I will go with Node.js for now as a middleware  (might swap to JVM platform later). It will provide an API for the client application. 
• The data, as the whole model is basically a graph, will be stored in a graph database, for now, let's go with Neo4j.
• I would like to put the server side pieces in lightweight containers. Docker will be perfect for this.
• I will need to manage the containers and I don't want to do this manually, so I might use Fig or Flocker or Kubernetes or all :).

Let's start and see what happens :)

What types of connections we should consider?

Implements

It means that A technology (e.g. library, framework) implements the B specification or protocol.

Uses

This connection says that A technology uses the B. This is a transitive relation.

Extends

A technology extends B technology. This is a transitive relation.

Relates to

A technology is related to B technology. This is a symmetric relation.

A is an alternative of B

A technology is created for a similar purpose like B. This is a symmetric relation.

Later we can consider the inverted connections like contains, specifies etc. We have to do this carefully as it can speed up our queries because of more specific types, but it can also slow down in some cases. It really depends on the use cases and size. For now this is enough. To learn more about this, the Graph Databases book is a great reference.

We can model these relationships with a property model graph.

What do we need?

(Note: Installed Docker is required and some Linux distribution. Windows and Mac users have to use boot2docker and set up the port forwarding for their boot2docker controlled VM. Another sibling for Windows users is to use Spoon but that is not a Docker based platform.)
Lets build our stack from down to top.

We need a running Neo4j instance. Why not run it in a Docker container? We could move our instance whenever we want or reuse the image for staging environments, integration tests, or for adding new nodes. Unfortunately there is no official Neo4j Docker image on Docker Hub, but there is one which is quite popular created by tpires. Could be a good fit, but first let's check it's Dockerfile to see how the image was built.

It is based on dockerfile/java which is based on an ubuntu image.

Looks good!

To just try it if it works (and to get the dependencies) let's run what the author says:

We are saying here, hey Docker run tpires/neo4j image in a container please, and bind the host machine's 7474 port to this container's 7474 port. 

Great. Let's check if its running.

Yup!

Neo4j runs a webserver for us, so let's open our browser and type localhost:7474.

It works!

It would be good to separate the data from the functionality into two containers, not to loose the portability, so let's create a data-only container.

Create a Dockerfile.

Let's reuse ubuntu image and add a volume.

Build the image.

Run the image.

Bind the volume to our Neo4j container and run it. (Don't forget to stop it.)

Nice. 

Ok, so now we have a running database. Of course it is not production ready, but for this prototype it is enough.

Next we will add a middleware based on Node platform, which will call the Neo4j's REST API and add some business logic. We will do this in the next part.

(Note: If we would want, we could extend the Neo4j's REST API with writing extensions in Java using JAX-RS annotations.) 

Feedback highly appreciated