Build 4 nodes elasticsearch cluster with Docker for Windows

Install Docker for Windows

From my understanding, Docker is a lightweight virtualization platform which hosts lightweight linux virtual machines. Individual system components can be packaged into multiple containers (VM's), read more details from Docker.

Download Docker for Windows installer and follow the instruction, keep clicking the Next button.

After installation, 2 icons created on Desktop

What have been installed

• Oracle VirtualBox
• MSYS-git (Git for Windows)
• Boot2Docker management tool
• boot2docker Linux ISO

Oracle VirtualBox

This is only used for virtualizing the boot2docker Linux ISO. I believe Docker does not use it to virtualize docker containers.

Program Path

%PROGRAMFILES%\Oracle\VirtualBox

MSYS-git

Program Path

%PROGRAMFILES(x86)%\Git

I have installed Git before installing Docker, now I have 2 Git context menu extension. I think there was an option to unselect Git in the installer.

Boot2Docker

Program Path

%PROGRAMFILES%\Boot2Docker for Windows

The desktop Boot2Docker Start icon points to start.sh

#!/bin/bash.exe

# convert backslash paths to forward slash (yes, really, sometimes you get either)
B2DPATH=${0//\\/\//}
# remove the script-name
B2DPATH=${B2DPATH%/*}
# convert any C:/ into /c/ as MSYS needs this form
B2DPATH=${B2DPATH//C:\//\/c/}
# simplify by adding the program dir to the path
PATH="$B2DPATH:$PATH"

ISO="$USERPROFILE/.boot2docker/boot2docker.iso"

if [ ! -e "$ISO" ]; then
 echo "copying initial boot2docker.iso (run 'boot2docker.exe download' to update"
 mkdir -p "$USERPROFILE/.boot2docker"
 cp "$B2DPATH/boot2docker.iso" "$ISO"
fi

echo "initializing..."
boot2docker.exe init -v
echo "starting..."
boot2docker.exe start
echo "connecting..."
boot2docker.exe ssh

read

The shell script copy boot2docker.iso (boot2docker Linux ISO) from Docker folder to %USERPROFILE%/.boot2docker folder. When Boot2Docker start, it starts the boot2docker-vm from Oracle VM VirtualBox. Double-click on Oracle VM Virtual Desktop icon launches Oracle VM VirtualBox Manager

The default Base Memory setting on the VM is 2GB, it can be adjusted by stopping the boot2docker-vm at the windows command prompt, NOT docker console.

%PROGRAMFILES%\Boot2Docker for Windows\boot2docker down

Then, right click on boot2docker-vm, select Settings... on the context menu. Somehow it does not let me max out all physical memory.

I also changed the default VM Folder location, window menu: File > Preferences... (Ctrl+G)

boot2docker Linux ISO (Console)

Build elasticsearch image

Method 1: docker pull (quick)

At Docker Console

docker pull dockerfile/elasticsearch

Method 2: docker build (can be customized)

At Docker Console

docker build -t="dockerfile/elasticsearch" github.com/dockerfile/elasticsearch

"dockerfile/elasticsearch" can be replace with other namespace/name

See ElasticSearch Dockerfile at GitHub for detail

Run elasticsearch images

docker run -d -P dockerfile/elasticsearch

See Working with Containers for other run command options.

-P option will map exposed port (9200, 9300) automatically, especially necessary for multiple instances in the same docker. See Linking Containers Together for detail.

Issue the command 3 more times to create 4 elasticsearch nodes

Connecting to elasticsearch cluster via ssh tunnel

The following sections apply to elasticsearch cluster server located behind firewall without NAT admin access. Access to a server with sshd is required. I have setup a ubuntu vm at Microsoft Windows Azure, running with minimum resources, as my ssh proxy server.

Setup sshd proxy server in the cloud

Install OpenSSH Server

Generate key

ssh man page

ssh-keygen -b 2048 -t rsa -f <keyfile>

Add public key to ~/.ssh/authorized_keys

cat <keyfile>.pub >> ~/.ssh/authorized_key

Copy the keyfile from cloud server to docker host and home PC

Either use cat and copy the content to clipboard or use scp

Setup SSH Reverse Proxy to elasticsearch Cluster

Find out which port to forward from Docker Console

docker ps

CONTAINER ID        IMAGE               COMMAND                CREATED
   STATUS              PORTS                                              NAMES
4587da9458db        hcst/es:latest      /elasticsearch/bin/e   2 days ago
   Up 2 days           0.0.0.0:49161->9200/tcp, 0.0.0.0:49162->9300/tcp   es4

dc18e9eb1ed1        hcst/es:latest      /elasticsearch/bin/e   2 days ago
   Up 2 days           0.0.0.0:49159->9200/tcp, 0.0.0.0:49160->9300/tcp   es3

698ba8ed7162        hcst/es:latest      /elasticsearch/bin/e   2 days ago
   Up 2 days           0.0.0.0:49155->9200/tcp, 0.0.0.0:49156->9300/tcp   es2

8b6479e41713        hcst/es:latest      /elasticsearch/bin/e   2 days ago
   Up 2 days           0.0.0.0:49153->9200/tcp, 0.0.0.0:49154->9300/tcp   es1

Select any single node and forward the port

ssh -R 9200:localhost:49153 -i <keyfile> -N -f user@server.cloud.com

Just in case outgoing port 22 is blocked by firewall, use other common port which allow out going traffic, such as 443 or 80 ... Just need extra step to map port 443 to 22 of the proxy server.

ssh -p 443 -R 9200:localhost:49153 -i <keyfile> -N -f user@server.cloud.com

Connect to elasticsearch Cluster via SSH Proxy Tunnel from Home PC

Bind a gateway port at Home PC

ssh -L 9200:localhost:9200 -i <keyfile> -N -f user@server.cloud.com

If proxy server port 22 map to other port, such as 443

ssh -p 443 -L 9200:localhost:9200 -i <keyfile> -N -f user@server.cloud.com

Test the connection

curl "http://localhost:9200"