My first GitHub Page

Objective

I wanted to do a experiment with a web development workflow using static page generator Hugo and other front end web development tools (Yeoman, Bower, Grunt, Gulp) to construct a static site with dynamic data.  The web dev environment will be dockerized in a Docker Container.  The final goal is pushing the assets to GitHub to build GitHub page.

Setting up Hugo from Source

Hugo binary, single executable, is running on a physical ubuntu box.

Pull Go Container

Pull from Official Repo

sudo docker pull golang

Start Go Container

• -it = Interactive Terminal
• --rm = remove container when exit
• -v = mount data volume from host directory /media/data/volumes/projects/go/ to container $GOPATH directory /go
• -w = working directory is /go

Compile Hugo

See install instruction for detail

$go get -v github.com/spf13/hugo
$exit

After the command is executed, 3 folders created: bin, pkg, src. The hugo runtime executable is outputted at bin folder. exit to terminate go session.

Add go bin to $PATH (System-wise)

Add the bin path to /etc/environment file.

Create a site with Hugo

Official Quick Start Guide

My Quicker Start Recipe

Scaffolding

$ cd /to/the/project/site/directory
$ hugo new site demo

Install Themes

$ git clone --recursive https://github.com/spf13/hugoThemes themes

Generate

$ hugo

Starting Hugo web server for preview

$ hugo server

Comment

After trying out all of its stock themes, none of the stock theme fit my need. I will start the workflow from the "traditional" toolset and then integrate Hugo workflow with dynamic content. I want to push a generic home page to GitHub ASAP

Create a WebDev Container with Yeoman, Grunt, Gulp, Bower

Dockerfile at GitHub
The build file is a bash script for the build command.

$ docker build -t hcsoftech/webdev .

It saved a few typing strokes when composing and trying out the Dockerfile.

Starting the WebDev Container

• -it = Interactive Terminal
• --rm = remove container when exit
• -v = mount data volume from host project src directory to container directory: /src
• --name = name the container as webdev
• --expose 9000 = expose Container port 9000 to host, which is the port for grunt server
• -p 80:9000 = map host port 80 to container port 9000

Scaffolding

$yo webapp

Answer a few questions, then generate a standard webapp structure.

/app               bower.json    /node_modules  /test
/bower_components  Gruntfile.js  package.json

Update Gruntfile.js

• Replace localhost with 0.0.0.0, so that grunt server will bind to all network interface.
• Comment out 'test' step from default task, since there is access right problem with phantomJS running within the container.

    grunt.registerTask('default', [
      'newer:jshint',
  //    'test',
      'build'
    ]);

Preview the site

$ grunt server

Grunt start web server with liveload on http://0.0.0.0:9000 in container, port 9000 is map to host port 80. Therefore, starting a browser point to the host ip receive page from container web server.

Generate the distribution

$ grunt

The generated files are written in the dist directory.

Setup a GitHub Page

1. Setup a User or organization site
2. Setup a custom domain with GitHub Pages
3. Publish content from dist directory to GitHub
4. Here is my sample site.

Mix Hugo with Grunt workflow ... not simple!

My ideal workflow would be grunt > hugo > publish; where grun handles all design and relatively permanent artifacts, hugo handles generation of dynamic data into static html. However, hugo generate index.html at root, which ignore the root index.html from grunt. As of now, I will use the yo-bower-grunt workflow to create a site, and then think of some way to integrate with hugo.

Storing docker images on a separate volume

Docker is consuming my boot volume quickly, I wanted to dedicated a hard drive just for docker images.

Provision ext4 volume

1. Start gdisk program (new hard disk: /dev/sdb)
   

   sudo gdisk /dev/sdb

2. Remove all partition from the hard disk

   d delete a partition

3. Create a new partition

   n add a new partition
   Command (? for help): n
   Partition number (1-128, default 1): 
   First sector (34-209715166, default = 2048) or {+-}size{KMGTP}: 
   Last sector (2048-209715166, default = 209715166) or {+-}size{KMGTP}: 
   Current type is 'Linux filesystem'
   Hex code or GUID (L to show codes, Enter = 8300): 
   Changed type of partition to 'Linux filesystem'
   

4. Save the partition on disk

   w write table to disk and exit
   Command (? for help): w
   
   Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
   PARTITIONS!!
   
   Do you want to proceed? (Y/N): y
   OK; writing new GUID partition table (GPT) to /dev/sdb.
   The operation has completed successfully.
   admiral@LorientK2:~$ sudo mkfs -t ext4 /dev/sdb1 
   mke2fs 1.42.9 (4-Feb-2014)
   Discarding device blocks: done                            
   Filesystem label=
   OS type: Linux
   Block size=4096 (log=2)
   Fragment size=4096 (log=2)
   Stride=0 blocks, Stripe width=0 blocks
   6553600 inodes, 26214139 blocks
   1310706 blocks (5.00%) reserved for the super user
   First data block=0
   Maximum filesystem blocks=4294967296
   800 block groups
   32768 blocks per group, 32768 fragments per group
   8192 inodes per group
   Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424, 20480000, 23887872
   
   Allocating group tables: done                            
   Writing inode tables: done                            
   Creating journal (32768 blocks): done
   Writing superblocks and filesystem accounting information: done

5. Create a new ext4 fs on the partition

   sudo mkfs -t ext4 /dev/sdb1

Mounting the new partition on boot

1. Create a mounting point (e.g. /media/data)

   sudo mkdir /media/data

2. Edit /etc/fstab

   sudo vi /etc/fstab

3. Add line to the end of file

   /dev/sdb1       /media/data     ext4    defaults        0       2

4. Reload mount

   sudo mount -a

Bind /var/lib/docker to the new partition

1. Create docker directory in the new partition

   sudo mkdir /media/data/docker

2. Stop docker service

   sudo service docker stop

3. Copy existing docker content to new directory

   sudo rsync -aXS /var/lib/docker/. /media/data/docker/

4. Compare directories

   sudo diff -r /var/lib/docker /media/data/docker

5. Edit /etc/fstab

   sudo vi /etc/fstab

6. Add line to the end of file

   /media/data/docker      /var/lib/docker none    bind    0       0

7. Reload mount

   sudo mount -a

8. Restart Docker

   sudo service docker start

Reference

• Install a new hard drive on Ubuntu documentation
• Moving docker images location to different partition 

Dockerization

Background

There is a good laptop, Dell Latitude E6410 (Early 2010) i7, laying around and doing nothing at home.  I started a project this weekend to breathe some new air into the sleeping laptop.  It is my perfect machine for using as home server because it uses very low electricity - around 10W when idle, as much as lighting a 60W equal LED light bulb.  Full throttle costs 30W extra, home server sitting idle most of the time anyway.

Intended Application

• File Server
• Running Crashplan cloud backup
• Running elasticsearch
• Running redis

Setup Server

Download Ubuntu Server then burn the iso to CD or DVD.

Minimal Install

• SSHd for remote access

Update packages to latest

$ sudo apt-get update && sudo apt-get upgrade

Enable github

Generate SSH key for github

Setup reverse ssh tunnel connection

If your home volume is encrypted, cron cannot access to the volume without password login at least once.  So an unencrypted worker account is needed.

sudo adduser worker

Login to the worker account after it is created.

Setup environment variables

Edit .pam_environment
System-wide environment variables at /etc/environment

SSH_SOCKS_PORT=12380
SSH_REDIR_PORT=12322

Setup reverse ssh tunnel

Generate a key set WITHOUT passphrase, for corn job to connect to the cloud server.

ssh-keygen -t rsa -b 4096 -f cloud_sshd 

Setup ~/.ssh/config

HOST cloud_sshd
    HostName sshd.atcloud.com
    Port 22
    User receptionist
    IdentityFile ~/.ssh/cloud_sshd.key

Save the script in worker home, crontab -e under worker user account

chmod 700 start_rssh_tunnel.sh

Setup postfix for cron to log error

sudo apt-get install postfix

Error log deliver to /var/mail/worker

Enable key only ssh authorization

• How do I force SSH to only allow uses with a key to log in?

  edit /etc/ssh/sshd_config, set PassowrdAuthentication no

• Adding SSH Keys to Authorized_Keys

  ssh-keygen -t rsa -b 4096 -f keyfile 

  cat keyfile.pub >> /etc/ssh/%u/authorized_keys

  See Troubleshooting - Encrypted Home Directory section of Ubuntu SSH/OpenSSH/Keys

Install CIFS (Optional)

Windows Share

sudo apt-get install cifs-utils

Install Docker

Docker Ubuntu Doc
Follow the Docker-maintained Package Installation, which install the latest version.  For people who is as lazy as me, I used the shortcut.

$ curl -sSL https://get.docker.com/ubuntu/ | sudo sh

DONE!

Pull Images to local repository

ubuntu

$ sudo docker pull ubuntu:latest

phusion/baseimage

$ sudo docker pull phusion/baseimage

redis

$ sudo docker pull redis

dockerfile/elasticsearch

$ sudo docker pull dockerfile/elasticsearch

golang

$ sudo docker pull golang

Install dnsmasq

Ubuntu Official Dnsmasq

$ sudo apt-get install dnsmasq
$ sudo vim /etc/dnsmasq.conf

• Don't need to uncomment (x) #listen-address= to listen-address=127.0.0.1, if dnsmasq listen on all address binding.
• Uncomment (x) conf-dir=/etc/dnsmasq.d, create a file docker_$name.conf.  After starting a container, write the container host record to the file in this format: host-record=$name,$ip

$ sudo vim /etc/dhcp/dhclient.conf

Uncomment (x) prepend domain-name-servers 127.0.0.1;

Restart the service

$ sudo service dnsmasq restart

Testing DNSmasq

1. Start a ubuntu instance with interactive bash session
   

   $ sudo docker run -t -i --dns 172.17.42.1 --name test ubuntu:latest /bin/bash

   where 172.17.42.1 is the default ip of host docker0 network interface
2. grep the container ip address
   

   $ sudo docker inspect test | grep IPAddress

3. Copy the ip address
4. Add entry to /etc/dnsmasq.d/docker_test.conf

   host-record=test,172.17.0.2

5. Restart dnsmasq

   sudo service dnsmasq restart

6. Lookup the name

dig test

Container startup script

DNS and Docker Container

#!/bin/bash
container=$1
#echo "$container"
ip=$(docker inspect $container | grep IPAddress | cut -f4 -d'"')
#echo "$ip"
echo "host-record=$container,$ip" > /etc/dnsmasq.d/docker_$container.conf

#reset to the next argument to be processed
OPTIND=2
while getopts "r" opt; do
  case $opt in
    r)
      service dnsmasq restart
      ;;
   \?)
      echo "Invalid option: -$OPTARG" >&2
      ;;
  esac
done

Usage:

sudo ./regdock.sh $container_name -r