Creating a graph of software technologies with Docker, Neo4j, NodeJS and D3.js Part 2

The plan

In the previous part we created a running Neo4j instance inside a Docker container.
Now we can create a middleware which will call the database's REST API, on the bottom of it's layer and expose an API to the client on the top of it's layer. As we did with the database instance, we will put the Node(s) into containers as well.

Ok, so let's create a Docker image for Node first.

The Node image

The good news is that there is an official Node image on Docker Hub. (Be aware, that does not mean we are super secure, it just means that this image should work properly). Let's check it's Dockerfile.
What we can see is that NPM is installed, so we can manage Node packages easily. Also if we run this image, the Node interpreter will be executed and will be waiting for input.
We are able to create our own image with a running Node + the packages we need.

We could put lots of "RUN npm install [package]" commands inside a Dockerfile, but instead of doing that, we will create package.json file and run "npm install" on it. It will be an external resource. If we run npm install, a node_modules directory will be created. It could be in a separate Docker image, and we could build a new image whenever we change the package.json, but I don't think it makes it too robust by putting the Node instance and the packages together into the same Docker image.

Ok let's do it.

Create the package.json file first.

	ogi@ubuntu:~/techgraph$ mkdir webapp
	ogi@ubuntu:~/techgraph$ cd webapp/
	ogi@ubuntu:~/techgraph/webapp$ touch package.json

view raw touchPackagejson hosted with ❤ by GitHub

I am using Express.js as a middleware to help me with the routing and HTTP requests, responses. There are loads of other middlewares to try out, but Express suits perfectly for now. So we will put Express into our package.json as a dependency.
I am also using the request Node module to make POST requests to our Neo4j instance.

	{
	"name": "techgraph",
	"description": "A graph of technology connections",
	"version": "0.0.1",
	"author": {
	"name": "Ognjen Bubalo",
	"email": "ognjen.bubalo@gmail.com"
	},
	"dependencies": {
	"express": "^4.10.4",
	"request": "^2.51.0"
	}
	}

view raw nodePackageJson hosted with ❤ by GitHub

Ok, now an app.js file.

ogi@ubuntu:~/techgraph/webapp$ touch app.js

view raw touchAppjs hosted with ❤ by GitHub

I'll leave it empty for now.

Now next to the package.json let's create the Dockerfile.

	FROM node:0.10-onbuild
	ADD package.json /
	ADD app.js /
	RUN npm install
	EXPOSE 3000

view raw nodeDockerfile hosted with ❤ by GitHub

So while building an image from this Dockerfile, a package.json and an app.js file will be added, and npm install will be called. It will create a node_modules directory and download into it the needed packages.

So, let's build it!

	ogi@ubuntu:~/techgraph/webapp$ sudo docker build -t webapp .
	[sudo] password for ogi:
	[info] POST /v1.15/build?rm=1&t=webapp
	[60fa92d5] +job build()
	Sending build context to Docker daemon 5.632 kB
	Sending build context to Docker daemon
	Step 0 : FROM node:0.10-onbuild
	# Executing 3 build triggers
	Trigger 0, COPY package.json /usr/src/app/
	Step 0 : COPY package.json /usr/src/app/
	Trigger 1, RUN npm install
	Step 0 : RUN npm install
	---> Running in a5b81efe0636
	[info] No non localhost DNS resolver found in resolv.conf and containers can't use it. Using default external servers : [8.8.8.8 8.8.4.4]
	[60fa92d5] +job allocate_interface(a5b81efe0636ab568bd93bbcbda64c6d1c2ff2a8c7baebf9b6572a95294093df)
	[60fa92d5] -job allocate_interface(a5b81efe0636ab568bd93bbcbda64c6d1c2ff2a8c7baebf9b6572a95294093df) = OK (0)
	[60fa92d5] +job log(start, a5b81efe0636ab568bd93bbcbda64c6d1c2ff2a8c7baebf9b6572a95294093df, 6e61fea2084d)
	[60fa92d5] -job log(start, a5b81efe0636ab568bd93bbcbda64c6d1c2ff2a8c7baebf9b6572a95294093df, 6e61fea2084d) = OK (0)
	npm WARN package.json techgraph@0.0.1 No repository field.
	npm WARN package.json techgraph@0.0.1 No README data
	express@4.10.4 node_modules/express
	├── merge-descriptors@0.0.2
	├── utils-merge@1.0.0
	├── fresh@0.2.4
	├── cookie@0.1.2
	├── escape-html@1.0.1
	├── range-parser@1.0.2
	├── cookie-signature@1.0.5
	├── finalhandler@0.3.2
	├── vary@1.0.0
	├── media-typer@0.3.0
	├── methods@1.1.0
	├── parseurl@1.3.0
	├── serve-static@1.7.1
	├── content-disposition@0.5.0
	├── path-to-regexp@0.1.3
	├── depd@1.0.0
	├── qs@2.3.3
	├── on-finished@2.1.1 (ee-first@1.1.0)
	├── debug@2.1.0 (ms@0.6.2)
	├── etag@1.5.1 (crc@3.2.1)
	├── proxy-addr@1.0.4 (forwarded@0.1.0, ipaddr.js@0.1.5)
	├── send@0.10.1 (destroy@1.0.3, ms@0.6.2, mime@1.2.11)
	├── type-is@1.5.3 (mime-types@2.0.3)
	└── accepts@1.1.3 (negotiator@0.4.9, mime-types@2.0.3)
	[60fa92d5] +job log(die, a5b81efe0636ab568bd93bbcbda64c6d1c2ff2a8c7baebf9b6572a95294093df, 6e61fea2084d)
	[60fa92d5] -job log(die, a5b81efe0636ab568bd93bbcbda64c6d1c2ff2a8c7baebf9b6572a95294093df, 6e61fea2084d) = OK (0)
	[60fa92d5] +job release_interface(a5b81efe0636ab568bd93bbcbda64c6d1c2ff2a8c7baebf9b6572a95294093df)
	[60fa92d5] -job release_interface(a5b81efe0636ab568bd93bbcbda64c6d1c2ff2a8c7baebf9b6572a95294093df) = OK (0)
	Trigger 2, COPY . /usr/src/app
	Step 0 : COPY . /usr/src/app
	---> b87d168baa02
	Removing intermediate container 796ba98d68fb
	Removing intermediate container a5b81efe0636
	Removing intermediate container e03e21971d9d
	Step 1 : ADD package.json /
	---> f70e4073b657
	Removing intermediate container 9ae8b0c3f5ed
	Step 2 : ADD app.js /
	---> d78c048a2199
	Removing intermediate container 4149681ef68b
	Step 3 : RUN npm install
	---> Running in b169fd1ed87c
	[info] No non localhost DNS resolver found in resolv.conf and containers can't use it. Using default external servers : [8.8.8.8 8.8.4.4]
	[60fa92d5] +job allocate_interface(b169fd1ed87c3be1793217ce6eb71a31d9cdbbcf00d8f84d33c4d8eff693eced)
	[60fa92d5] -job allocate_interface(b169fd1ed87c3be1793217ce6eb71a31d9cdbbcf00d8f84d33c4d8eff693eced) = OK (0)
	[60fa92d5] +job log(start, b169fd1ed87c3be1793217ce6eb71a31d9cdbbcf00d8f84d33c4d8eff693eced, d78c048a2199)
	[60fa92d5] -job log(start, b169fd1ed87c3be1793217ce6eb71a31d9cdbbcf00d8f84d33c4d8eff693eced, d78c048a2199) = OK (0)
	npm WARN package.json techgraph@0.0.1 No repository field.
	npm WARN package.json techgraph@0.0.1 No README data
	[60fa92d5] +job log(die, b169fd1ed87c3be1793217ce6eb71a31d9cdbbcf00d8f84d33c4d8eff693eced, d78c048a2199)
	[60fa92d5] -job log(die, b169fd1ed87c3be1793217ce6eb71a31d9cdbbcf00d8f84d33c4d8eff693eced, d78c048a2199) = OK (0)
	[60fa92d5] +job release_interface(b169fd1ed87c3be1793217ce6eb71a31d9cdbbcf00d8f84d33c4d8eff693eced)
	[60fa92d5] -job release_interface(b169fd1ed87c3be1793217ce6eb71a31d9cdbbcf00d8f84d33c4d8eff693eced) = OK (0)
	---> 78cc6e097fc1
	Removing intermediate container b169fd1ed87c
	Step 4 : EXPOSE 3000
	---> Running in 1cdc1ed198a5
	---> 1eac9601fffe
	Removing intermediate container 1cdc1ed198a5
	Successfully built 1eac9601fffe
	[60fa92d5] -job build() = OK (0)

view raw webappDockerBuild hosted with ❤ by GitHub

You can see it from the log that it downloaded what we need. Coolio!

The code

Now let's create a simple application which will handle adding a new technology, adding a new connection between two technologies and getting the adjacent technologies of a particular tech. Later we can add some logic for setting weights for connections to visualize how strong the connection is, and we can add weights to the techs as well to show how popular they are.

These are the API endpoints

POST:
/api/tech/new?name=[techname]

POST:
/api/connection/new?A=[Atechname]&B=[Btechname]&connection=[connection]

GET:
/api/tech/all

Let's open our app.js empty file and add our code.

The code is simple. Of course we would like to put the API endpoints, the function's and the configuration into separate files (modules) later, but in this stage we want to have simple solutions.

	var http = require('http');
	var express = require('express');
	var querystring = require('querystring');
	var request = require('request');
	var app = express();
	app.get('/', function (req, res) {
	res.send('Hello World!')
	});
	app.post('/api/tech/new', function (req, res) {
	var options = {
	uri: 'http://' + process.env.DB_PORT_7474_TCP_ADDR + ':7474' + '/db/data/transaction/commit',
	method: 'POST',
	json: {statements : [ {statement : 'CREATE (n:Technology { name:"'+ req.param("name") + '" });' } ]}
	};
	request(options, function (error, response, body) {
	if (!error && response.statusCode == 200) {
	res.send(200);
	} else {
	res.send(500);
	}
	});
	});
	app.post('/api/connection/new', function (req, res) {
	var options = {
	uri: 'http://' + process.env.DB_PORT_7474_TCP_ADDR + ':7474' + '/db/data/transaction/commit',
	method: 'POST',
	json: {statements : [ {statement : 'MATCH (n1:Technology) WHERE n1.name = "'
	+ req.param("A")
	+'" MATCH (n2:Technology) WHERE n2.name = "'
	+ req.param("B") + '" CREATE (n1)-[:' + req.param("connection") + ']->(n2);' } ]}
	};
	request(options, function (error, response, body) {
	if (!error && response.statusCode == 200) {
	res.send(200);
	} else {
	res.send(500);
	}
	});
	});
	app.get('/api/tech/all', function (req, res) {
	var options = {
	uri: 'http://' + process.env.DB_PORT_7474_TCP_ADDR + ':7474' + '/db/data/transaction/commit',
	method: 'POST',
	json: {statements : [ {statement : 'MATCH (n1:Technology) WHERE n1.name="'+ req.param("technology") + '" MATCH (n2:Technology) RETURN (n1)-[*1]-(n2);' } ]}
	};
	request(options, function (error, response, body) {
	if (!error && response.statusCode == 200) {
	res.send(response.body);
	} else {
	res.send(500);
	}
	});
	});
	var server = app.listen(3000, function () {
	var host = server.address().address
	var port = server.address().port
	console.log('Example app listening at http://%s:%s', host, port)
	})

view raw webappJS hosted with ❤ by GitHub

Ok. Now we need to rebuild our Docker image.

	ogi@ubuntu:~/techgraph/webapp$ sudo docker build -t webapp .
	Sending build context to Docker daemon 6.144 kB
	Sending build context to Docker daemon
	Step 0 : FROM node:0.10-onbuild
	# Executing 3 build triggers
	Trigger 0, COPY package.json /usr/src/app/
	Step 0 : COPY package.json /usr/src/app/
	---> Using cache
	Trigger 1, RUN npm install
	Step 0 : RUN npm install
	---> Using cache
	Trigger 2, COPY . /usr/src/app
	Step 0 : COPY . /usr/src/app
	---> 3fa0ef35fbcb
	Removing intermediate container ec0171e665fe
	Step 1 : ADD package.json /
	---> 5f7b89b4965c
	Removing intermediate container b9edbb543490
	Step 2 : ADD app.js /
	---> 813c42f3fd7d
	Removing intermediate container 5967a6cd24ad
	Step 3 : RUN npm install
	---> Running in 006e852a5774
	npm WARN package.json techgraph@0.0.1 No repository field.
	npm WARN package.json techgraph@0.0.1 No README data
	---> 87d30e82b064
	Removing intermediate container 006e852a5774
	Step 4 : EXPOSE 3000
	---> Running in 474b6698c7cb
	---> 8ad13a3bc1f3
	Removing intermediate container 474b6698c7cb
	Successfully built 8ad13a3bc1f3
	ogi@ubuntu:~/techgraph/webapp$

view raw rebuildDockerImage hosted with ❤ by GitHub

And run our web application. (Don't forget to stop-remove the old container.)

	ogi@ubuntu:~/techgraph/webapp$ sudo docker run --link neo4j:db -i -t -d -p 3000:3000 --name webapp webapp node app
	3b65c41ec751256dc3eefdc3afb1fa092566afa3b72c873a4471496b08110e80

view raw runWebapp hosted with ❤ by GitHub

Notice that we are using --link neo4j:db. This creates some environment variables for us that we can use in our webapp for networking purpose.

Just because we are curious let's check these variables in our running webapp container.

Cool! We see that now we don't have to care what is the exact IP address of our Neo4j container. Docker will handle this for us, we just need to use the env variable.

Now we can create some technology nodes with curl or alternative tool for doing HTTP requests. I will execute these requests just to try it out:

	ogi@ubuntu:~/techgraph/webapp$ sudo docker exec -i -t webapp bash
	root@3b65c41ec751:/usr/src/app# printenv
	NODE_VERSION=0.10.33
	DB_PORT_1337_TCP_PROTO=tcp
	DB_PORT_7474_TCP_PORT=7474
	HOSTNAME=3b65c41ec751
	DB_NAME=/webapp/db
	DB_ENV_JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
	TERM=xterm
	DB_PORT=tcp://172.17.0.25:1337
	DB_PORT_7474_TCP_ADDR=172.17.0.25
	DB_PORT_1337_TCP_ADDR=172.17.0.25
	DB_PORT_1337_TCP_PORT=1337
	PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
	PWD=/usr/src/app
	DB_PORT_1337_TCP=tcp://172.17.0.25:1337
	NPM_VERSION=2.1.9
	DB_PORT_7474_TCP_PROTO=tcp
	SHLVL=1
	DB_PORT_7474_TCP=tcp://172.17.0.25:7474
	_=/usr/bin/printenv
	root@3b65c41ec751:/usr/src/app#

view raw checkEnvVariables hosted with ❤ by GitHub

1. POST: http://0.0.0.0:3000/api/tech/new?name=HTML
2. POST: http://0.0.0.0:3000/api/tech/new?name=FLASH
3. POST: http://0.0.0.0:3000/api/connection/new?A=HTML&B=FLASH&RELATION=IS_RELATED_TO
4. GET: http://0.0.0.0:3000/api/tech/all?technology=HTML

The last query returns  this JSON:

{"results":[{"columns":["(n1)-[*1]-(n2)"],"data":[{"row":[[[{"name":"HTML"},{},{"name":"FLASH"}]]]},{"row":[[[{"name":"HTML"},{},{"name":"FLASH"}]]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]},{"row":[[]]}]}],"errors":[]}

In the next part we will add the client code and actually draw our graph.