My first GitHub Page

Objective

I wanted to do a experiment with a web development workflow using static page generator Hugo and other front end web development tools (Yeoman, Bower, Grunt, Gulp) to construct a static site with dynamic data.  The web dev environment will be dockerized in a Docker Container.  The final goal is pushing the assets to GitHub to build GitHub page.

Setting up Hugo from Source

Hugo binary, single executable, is running on a physical ubuntu box.

Pull Go Container

Pull from Official Repo

sudo docker pull golang

Start Go Container

• -it = Interactive Terminal
• --rm = remove container when exit
• -v = mount data volume from host directory /media/data/volumes/projects/go/ to container $GOPATH directory /go
• -w = working directory is /go

Compile Hugo

See install instruction for detail

$go get -v github.com/spf13/hugo
$exit

After the command is executed, 3 folders created: bin, pkg, src. The hugo runtime executable is outputted at bin folder. exit to terminate go session.

Add go bin to $PATH (System-wise)

Add the bin path to /etc/environment file.

Create a site with Hugo

Official Quick Start Guide

My Quicker Start Recipe

Scaffolding

$ cd /to/the/project/site/directory
$ hugo new site demo

Install Themes

$ git clone --recursive https://github.com/spf13/hugoThemes themes

Generate

$ hugo

Starting Hugo web server for preview

$ hugo server

Comment

After trying out all of its stock themes, none of the stock theme fit my need. I will start the workflow from the "traditional" toolset and then integrate Hugo workflow with dynamic content. I want to push a generic home page to GitHub ASAP

Create a WebDev Container with Yeoman, Grunt, Gulp, Bower

Dockerfile at GitHub
The build file is a bash script for the build command.

$ docker build -t hcsoftech/webdev .

It saved a few typing strokes when composing and trying out the Dockerfile.

Starting the WebDev Container

• -it = Interactive Terminal
• --rm = remove container when exit
• -v = mount data volume from host project src directory to container directory: /src
• --name = name the container as webdev
• --expose 9000 = expose Container port 9000 to host, which is the port for grunt server
• -p 80:9000 = map host port 80 to container port 9000

Scaffolding

$yo webapp

Answer a few questions, then generate a standard webapp structure.

/app               bower.json    /node_modules  /test
/bower_components  Gruntfile.js  package.json

Update Gruntfile.js

• Replace localhost with 0.0.0.0, so that grunt server will bind to all network interface.
• Comment out 'test' step from default task, since there is access right problem with phantomJS running within the container.

    grunt.registerTask('default', [
      'newer:jshint',
  //    'test',
      'build'
    ]);

Preview the site

$ grunt server

Grunt start web server with liveload on http://0.0.0.0:9000 in container, port 9000 is map to host port 80. Therefore, starting a browser point to the host ip receive page from container web server.

Generate the distribution

$ grunt

The generated files are written in the dist directory.

Setup a GitHub Page

1. Setup a User or organization site
2. Setup a custom domain with GitHub Pages
3. Publish content from dist directory to GitHub
4. Here is my sample site.

Mix Hugo with Grunt workflow ... not simple!

My ideal workflow would be grunt > hugo > publish; where grun handles all design and relatively permanent artifacts, hugo handles generation of dynamic data into static html. However, hugo generate index.html at root, which ignore the root index.html from grunt. As of now, I will use the yo-bower-grunt workflow to create a site, and then think of some way to integrate with hugo.

Storing docker images on a separate volume

Docker is consuming my boot volume quickly, I wanted to dedicated a hard drive just for docker images.

Provision ext4 volume

1. Start gdisk program (new hard disk: /dev/sdb)
   

   sudo gdisk /dev/sdb

2. Remove all partition from the hard disk

   d delete a partition

3. Create a new partition

   n add a new partition
   Command (? for help): n
   Partition number (1-128, default 1): 
   First sector (34-209715166, default = 2048) or {+-}size{KMGTP}: 
   Last sector (2048-209715166, default = 209715166) or {+-}size{KMGTP}: 
   Current type is 'Linux filesystem'
   Hex code or GUID (L to show codes, Enter = 8300): 
   Changed type of partition to 'Linux filesystem'
   

4. Save the partition on disk

   w write table to disk and exit
   Command (? for help): w
   
   Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
   PARTITIONS!!
   
   Do you want to proceed? (Y/N): y
   OK; writing new GUID partition table (GPT) to /dev/sdb.
   The operation has completed successfully.
   admiral@LorientK2:~$ sudo mkfs -t ext4 /dev/sdb1 
   mke2fs 1.42.9 (4-Feb-2014)
   Discarding device blocks: done                            
   Filesystem label=
   OS type: Linux
   Block size=4096 (log=2)
   Fragment size=4096 (log=2)
   Stride=0 blocks, Stripe width=0 blocks
   6553600 inodes, 26214139 blocks
   1310706 blocks (5.00%) reserved for the super user
   First data block=0
   Maximum filesystem blocks=4294967296
   800 block groups
   32768 blocks per group, 32768 fragments per group
   8192 inodes per group
   Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424, 20480000, 23887872
   
   Allocating group tables: done                            
   Writing inode tables: done                            
   Creating journal (32768 blocks): done
   Writing superblocks and filesystem accounting information: done

5. Create a new ext4 fs on the partition

   sudo mkfs -t ext4 /dev/sdb1

Mounting the new partition on boot

1. Create a mounting point (e.g. /media/data)

   sudo mkdir /media/data

2. Edit /etc/fstab

   sudo vi /etc/fstab

3. Add line to the end of file

   /dev/sdb1       /media/data     ext4    defaults        0       2

4. Reload mount

   sudo mount -a

Bind /var/lib/docker to the new partition

1. Create docker directory in the new partition

   sudo mkdir /media/data/docker

2. Stop docker service

   sudo service docker stop

3. Copy existing docker content to new directory

   sudo rsync -aXS /var/lib/docker/. /media/data/docker/

4. Compare directories

   sudo diff -r /var/lib/docker /media/data/docker

5. Edit /etc/fstab

   sudo vi /etc/fstab

6. Add line to the end of file

   /media/data/docker      /var/lib/docker none    bind    0       0

7. Reload mount

   sudo mount -a

8. Restart Docker

   sudo service docker start

Reference

• Install a new hard drive on Ubuntu documentation
• Moving docker images location to different partition 

Dockerization

Background

There is a good laptop, Dell Latitude E6410 (Early 2010) i7, laying around and doing nothing at home.  I started a project this weekend to breathe some new air into the sleeping laptop.  It is my perfect machine for using as home server because it uses very low electricity - around 10W when idle, as much as lighting a 60W equal LED light bulb.  Full throttle costs 30W extra, home server sitting idle most of the time anyway.

Intended Application

• File Server
• Running Crashplan cloud backup
• Running elasticsearch
• Running redis

Setup Server

Download Ubuntu Server then burn the iso to CD or DVD.

Minimal Install

• SSHd for remote access

Update packages to latest

$ sudo apt-get update && sudo apt-get upgrade

Enable github

Generate SSH key for github

Setup reverse ssh tunnel connection

If your home volume is encrypted, cron cannot access to the volume without password login at least once.  So an unencrypted worker account is needed.

sudo adduser worker

Login to the worker account after it is created.

Setup environment variables

Edit .pam_environment
System-wide environment variables at /etc/environment

SSH_SOCKS_PORT=12380
SSH_REDIR_PORT=12322

Setup reverse ssh tunnel

Generate a key set WITHOUT passphrase, for corn job to connect to the cloud server.

ssh-keygen -t rsa -b 4096 -f cloud_sshd 

Setup ~/.ssh/config

HOST cloud_sshd
    HostName sshd.atcloud.com
    Port 22
    User receptionist
    IdentityFile ~/.ssh/cloud_sshd.key

Save the script in worker home, crontab -e under worker user account

chmod 700 start_rssh_tunnel.sh

Setup postfix for cron to log error

sudo apt-get install postfix

Error log deliver to /var/mail/worker

Enable key only ssh authorization

• How do I force SSH to only allow uses with a key to log in?

  edit /etc/ssh/sshd_config, set PassowrdAuthentication no

• Adding SSH Keys to Authorized_Keys

  ssh-keygen -t rsa -b 4096 -f keyfile 

  cat keyfile.pub >> /etc/ssh/%u/authorized_keys

  See Troubleshooting - Encrypted Home Directory section of Ubuntu SSH/OpenSSH/Keys

Install CIFS (Optional)

Windows Share

sudo apt-get install cifs-utils

Install Docker

Docker Ubuntu Doc
Follow the Docker-maintained Package Installation, which install the latest version.  For people who is as lazy as me, I used the shortcut.

$ curl -sSL https://get.docker.com/ubuntu/ | sudo sh

DONE!

Pull Images to local repository

ubuntu

$ sudo docker pull ubuntu:latest

phusion/baseimage

$ sudo docker pull phusion/baseimage

redis

$ sudo docker pull redis

dockerfile/elasticsearch

$ sudo docker pull dockerfile/elasticsearch

golang

$ sudo docker pull golang

Install dnsmasq

Ubuntu Official Dnsmasq

$ sudo apt-get install dnsmasq
$ sudo vim /etc/dnsmasq.conf

• Don't need to uncomment (x) #listen-address= to listen-address=127.0.0.1, if dnsmasq listen on all address binding.
• Uncomment (x) conf-dir=/etc/dnsmasq.d, create a file docker_$name.conf.  After starting a container, write the container host record to the file in this format: host-record=$name,$ip

$ sudo vim /etc/dhcp/dhclient.conf

Uncomment (x) prepend domain-name-servers 127.0.0.1;

Restart the service

$ sudo service dnsmasq restart

Testing DNSmasq

1. Start a ubuntu instance with interactive bash session
   

   $ sudo docker run -t -i --dns 172.17.42.1 --name test ubuntu:latest /bin/bash

   where 172.17.42.1 is the default ip of host docker0 network interface
2. grep the container ip address
   

   $ sudo docker inspect test | grep IPAddress

3. Copy the ip address
4. Add entry to /etc/dnsmasq.d/docker_test.conf

   host-record=test,172.17.0.2

5. Restart dnsmasq

   sudo service dnsmasq restart

6. Lookup the name

dig test

Container startup script

DNS and Docker Container

#!/bin/bash
container=$1
#echo "$container"
ip=$(docker inspect $container | grep IPAddress | cut -f4 -d'"')
#echo "$ip"
echo "host-record=$container,$ip" > /etc/dnsmasq.d/docker_$container.conf

#reset to the next argument to be processed
OPTIND=2
while getopts "r" opt; do
  case $opt in
    r)
      service dnsmasq restart
      ;;
   \?)
      echo "Invalid option: -$OPTARG" >&2
      ;;
  esac
done

Usage:

sudo ./regdock.sh $container_name -r

Building an eCommerce marketplace, like shopify, using the hippest tech

Background Story

My friend, Stephen, is a small business owner selling aftermarket auto performance part and accessories.  He used Magento for his online store, but Magento will stop his product by Feb 2015.  He asked me to build a eCommerce web site, I told him to use Shopify and help him to setup his store in a weekend.

Shopify - Cheap to start, expensive to make sales

$14/month - Starter Plan

25 Items

The package just provide enough feature to have the brand display 25 items online.

NO real time carrier shipping

Unless most products are small and light items which can easily absorb shipping cost into its sales price (Free Shipping), most buyers abandon shopping carts when seeing high flat rate shipping fee. 

2.9% + 30¢ credit card rate

When buyers abandon shopping carts, the above rate is meaningless.

$29/month - Basic Plan

Unlimited items

If most visitors are window shoppers, this package will allow the brand to disclose the entire product catalog for price comparison.

Still NO real time carrier shipping

For auto performance part and accessories, items size and weight varies.  Free shipping for big and heavy items lose money; while flat rate shipping makes shoppers feel unfair and dishonest - abandon shopping carts!!

2.9% + 30¢ credit card rate

A lots of window shoppers ...

Discount Code Engine

I have not use it, so I don't know how complicate a rule can be setup.  To be discovered ...

Fraud analysis tools

These tools show the risk level of each transaction base on the credit card information.  Oversea credit card has a higher risk level than domestic ...  Base on the information, seller can do additional verification or just refund the buyer before the card owner charge back.  This feature is only available for transaction using shopify gateway, at the basic plan tier.

$79/month - Professional Plan

Still NO real time carrier shipping, really!

$50 premium and still no real time carrier shipping, what a rip off.  Easypost only charges 5 cents per shipping label.  However, there is no option to integrate external shipping calculator with shopify checkout workflow.  Let's see what are the other useful features offset the $50 premium.

2.5% + 30¢ credit card rate

Without real time carrier shipping, there are still lots of abandon carts all over cache.

Gift Cards

Gift card strategy does not apply to many merchants, especially small online retailer.  One requirement is the product can be given as gift.

Creating gift card and take 2.5% + 30¢ profit from seller, for a $100 gift card, shopify profit $2.80 from seller and seller received $97.20 cash and $100 liability.  I like this business model.

Professional reports

I have not try the report feature, so no comment and to be discovered.

Abandoned cart recovery

This is a must have feature ☺ when abandoned shopping carts are all over the cache after seeing unfair shipping rate.  Anyway, I have yet to see how this feature work, to be discovered.

$179/month - Unlimited Plan

2.25% + 30¢ credit card rate

Some merchants has already negotiate a better rate, less than 2%, with local bank directly.  Some banks use authorize.net or other third party payment gateways.  I have setup authorize.net with Shopify for Steven's shop.

Advanced report builder

Sure, it is really useful ... for high volume sellers.  Anyway, to be discovered ...

Real time carrier shipping, finally

Let's assume all additional features above Basic Plan are unessential, the cost for getting real time carrier shipping is $150/month.  Base on EasyPost rate, a merchant need to ship over 3000 packages to offset the difference.

There are $100 difference when comparing with Professional Plan, a merchant need to ship over 2000 packages to break even.  I would care less about $100 when I am shipping 2000 packages a month, but what does 2000 packages a month really means?

Online retail store opens 24/7, assume there are 30 days month, the site need to have a average 66.666... sales per day volume to justify upgrading from Professional to Unlimited; and 100 sales per day volume to justify upgrading from Basic to Unlimited.

However, for some online retailer, to make a successful profitable conversion, it is necessary to provide real time carrier shipping to customer; and making the 7 - 10 daily online sales volume is already very successful.

New Project - A small business marketplace that make sense and money

I want to help small businesses, like Stephen's auto parts store, to operate a low cost eCommerce channel to archive better profit margin.  I will leverage the project on cloud platforms from IDE to deployment to realize my idea of how a web site should be built.  

My 2 cents on the future (today and beyond) of web development

I was asked my opinion on the future of web development.  I blasted out my not well thought opinion, only from the development angle.  I think I owe myself a well organized response to express my point of view better next time.

AngularJs + RESTful data service + pre-composed HTML

I believe a web framework/system should handle/serve data and static assets separately.  The job of composing the UI html DOM will be handled by javascript framework - AngularJs.  If there is a need for the home page to display near real time "dynamic content" in a fast and robust fashion, the home page html will be pre-composed by a worker process at the server side and save the html to the CDN servers.

Better Customer Experience

There are many pages, especially home page and SEO pages, serving the same "dynamic content" over a long period of time - P, where P is the length of time the "dynamic content" is updated.  

I think the worst user experience are getting 500 error or a nice error page.  This happen when sever side scripting logic is having uncaught exception to compose the html of a requesting page.  Serving static html with the pre-composed "dynamic content" can effectively reduce the 500 error, and improve customer experience and performance.

When the traditional GET/POST round trip is encountering error, browser is redirect to the error page == customers yell WTF.  UX building around ajax or websocket interaction, provide better workflow experience because both browser and its user are in the same continuous context, rather than redirect, load and rebuild context. 

Same level of Security

The same security protocol that is used to protect a MVC web server can be used to protect the RESTful data service.  After all, RESTful data service is just another MVC web server using the same HTTP protocol.

Better Availability

When 500 responses are reduced, the web site availability increase.

With the separation of static and data server, scaling can be better tailored and customized to the I/O bounded and CPU bounded strategy, in contrast to the server side scripting web system.  For example, static file server, CDN, can adopt caching strategy with less CPU resource, while more CPU/memory budget can be invested into data servers, or even adding more instances to host the heavy weight data service.

Effective scaling improve availability as well.

Better Performance

In terms of composing static HTML, it is simply O(1) vs O(n); the static content needs to be composed once and serve n requests; while server side scripting compose n times to serve n requests.

In terms of constructing dynamic data, the data service only need to construct and return the data payload in an ajax call; while the server side scripting need to construct and return the entire html. 

Complete Separation of Concern

Modern web site heavily depends on javascript scripting.  I remember 10 years ago, web design/development still needed to consider the case of browser with javascript turn off, while today browser having javascript disabled == who cares, not my customer.

Base on the software system design principal - separation of concern, I believe all UI manipulation logic be contained within the same code base as much as possible.  When javascript DOM manipulation is a requirement, the UI logic/assets should be contained in client side code base as much as possible.

Even with the best practice of MVC on server side scripting development, the view and view manipulation logic are sharing responsibility with the client side javascript UI code.  Embedding javascript in the view and in-lining dynamic value in javascript when composing at server side causes more confusion and inconsistency in the system.

Easier for Testing

Since decoupling presentation and data logic into 2 sub system, testing on each sub system is simpler.

For client side testing, a data stub can be easily injected to mock the data coming from data service under AngularJs framework,  The UI testing parameters can be easily setup on client side scripts.

For data service testing, since it is a RESTful web service and the return payload is JSON string, the tests can be easily scripted and automated.  Validating a JSON string is much easier than validating a HTML string.  For lazy developer like myself, I use Chrome App - Postman to do minimum sanity check.

For pre-composed HTML testing, a test step can be added to the end of automated HTML composing process.  The end result HTML can be loaded into headless browser, like PhantomJs or simply be parsed and validated.

Build 4 nodes elasticsearch cluster with Docker for Windows

Install Docker for Windows

From my understanding, Docker is a lightweight virtualization platform which hosts lightweight linux virtual machines. Individual system components can be packaged into multiple containers (VM's), read more details from Docker.

Download Docker for Windows installer and follow the instruction, keep clicking the Next button.

After installation, 2 icons created on Desktop

What have been installed

• Oracle VirtualBox
• MSYS-git (Git for Windows)
• Boot2Docker management tool
• boot2docker Linux ISO

Oracle VirtualBox

This is only used for virtualizing the boot2docker Linux ISO. I believe Docker does not use it to virtualize docker containers.

Program Path

%PROGRAMFILES%\Oracle\VirtualBox

MSYS-git

Program Path

%PROGRAMFILES(x86)%\Git

I have installed Git before installing Docker, now I have 2 Git context menu extension. I think there was an option to unselect Git in the installer.

Boot2Docker

Program Path

%PROGRAMFILES%\Boot2Docker for Windows

The desktop Boot2Docker Start icon points to start.sh

#!/bin/bash.exe

# convert backslash paths to forward slash (yes, really, sometimes you get either)
B2DPATH=${0//\\/\//}
# remove the script-name
B2DPATH=${B2DPATH%/*}
# convert any C:/ into /c/ as MSYS needs this form
B2DPATH=${B2DPATH//C:\//\/c/}
# simplify by adding the program dir to the path
PATH="$B2DPATH:$PATH"

ISO="$USERPROFILE/.boot2docker/boot2docker.iso"

if [ ! -e "$ISO" ]; then
 echo "copying initial boot2docker.iso (run 'boot2docker.exe download' to update"
 mkdir -p "$USERPROFILE/.boot2docker"
 cp "$B2DPATH/boot2docker.iso" "$ISO"
fi

echo "initializing..."
boot2docker.exe init -v
echo "starting..."
boot2docker.exe start
echo "connecting..."
boot2docker.exe ssh

read

The shell script copy boot2docker.iso (boot2docker Linux ISO) from Docker folder to %USERPROFILE%/.boot2docker folder. When Boot2Docker start, it starts the boot2docker-vm from Oracle VM VirtualBox. Double-click on Oracle VM Virtual Desktop icon launches Oracle VM VirtualBox Manager

The default Base Memory setting on the VM is 2GB, it can be adjusted by stopping the boot2docker-vm at the windows command prompt, NOT docker console.

%PROGRAMFILES%\Boot2Docker for Windows\boot2docker down

Then, right click on boot2docker-vm, select Settings... on the context menu. Somehow it does not let me max out all physical memory.

I also changed the default VM Folder location, window menu: File > Preferences... (Ctrl+G)

boot2docker Linux ISO (Console)

Build elasticsearch image

Method 1: docker pull (quick)

At Docker Console

docker pull dockerfile/elasticsearch

Method 2: docker build (can be customized)

At Docker Console

docker build -t="dockerfile/elasticsearch" github.com/dockerfile/elasticsearch

"dockerfile/elasticsearch" can be replace with other namespace/name

See ElasticSearch Dockerfile at GitHub for detail

Run elasticsearch images

docker run -d -P dockerfile/elasticsearch

See Working with Containers for other run command options.

-P option will map exposed port (9200, 9300) automatically, especially necessary for multiple instances in the same docker. See Linking Containers Together for detail.

Issue the command 3 more times to create 4 elasticsearch nodes

Connecting to elasticsearch cluster via ssh tunnel

The following sections apply to elasticsearch cluster server located behind firewall without NAT admin access. Access to a server with sshd is required. I have setup a ubuntu vm at Microsoft Windows Azure, running with minimum resources, as my ssh proxy server.

Setup sshd proxy server in the cloud

Install OpenSSH Server

Generate key

ssh man page

ssh-keygen -b 2048 -t rsa -f <keyfile>

Add public key to ~/.ssh/authorized_keys

cat <keyfile>.pub >> ~/.ssh/authorized_key

Copy the keyfile from cloud server to docker host and home PC

Either use cat and copy the content to clipboard or use scp

Setup SSH Reverse Proxy to elasticsearch Cluster

Find out which port to forward from Docker Console

docker ps

CONTAINER ID        IMAGE               COMMAND                CREATED
   STATUS              PORTS                                              NAMES
4587da9458db        hcst/es:latest      /elasticsearch/bin/e   2 days ago
   Up 2 days           0.0.0.0:49161->9200/tcp, 0.0.0.0:49162->9300/tcp   es4

dc18e9eb1ed1        hcst/es:latest      /elasticsearch/bin/e   2 days ago
   Up 2 days           0.0.0.0:49159->9200/tcp, 0.0.0.0:49160->9300/tcp   es3

698ba8ed7162        hcst/es:latest      /elasticsearch/bin/e   2 days ago
   Up 2 days           0.0.0.0:49155->9200/tcp, 0.0.0.0:49156->9300/tcp   es2

8b6479e41713        hcst/es:latest      /elasticsearch/bin/e   2 days ago
   Up 2 days           0.0.0.0:49153->9200/tcp, 0.0.0.0:49154->9300/tcp   es1

Select any single node and forward the port

ssh -R 9200:localhost:49153 -i <keyfile> -N -f user@server.cloud.com

Just in case outgoing port 22 is blocked by firewall, use other common port which allow out going traffic, such as 443 or 80 ... Just need extra step to map port 443 to 22 of the proxy server.

ssh -p 443 -R 9200:localhost:49153 -i <keyfile> -N -f user@server.cloud.com

Connect to elasticsearch Cluster via SSH Proxy Tunnel from Home PC

Bind a gateway port at Home PC

ssh -L 9200:localhost:9200 -i <keyfile> -N -f user@server.cloud.com

If proxy server port 22 map to other port, such as 443

ssh -p 443 -L 9200:localhost:9200 -i <keyfile> -N -f user@server.cloud.com

Test the connection

curl "http://localhost:9200"

SSH Raspberry Pi with SSH key

Raspberry Pi Documentation

Generate Key at host that remote from

ssh-keygen -t rsa -C <name>@<hostname>

Copy public key to remote host

Create a .ssh folder at Raspberry Pi, if not exists

cat ~/.ssh/id_rsa.pub | ssh <USERNAME>@<IP-ADDRESS> 'cat >> .ssh/authorized_keys'

Connect

ssh <USERNAME>@<IP-ADDRESS>