Building an eCommerce marketplace, like shopify, using the hippest tech

Background Story

My friend, Stephen, is a small business owner selling aftermarket auto performance part and accessories.  He used Magento for his online store, but Magento will stop his product by Feb 2015.  He asked me to build a eCommerce web site, I told him to use Shopify and help him to setup his store in a weekend.

Shopify - Cheap to start, expensive to make sales

$14/month - Starter Plan

25 Items

The package just provide enough feature to have the brand display 25 items online.

NO real time carrier shipping

Unless most products are small and light items which can easily absorb shipping cost into its sales price (Free Shipping), most buyers abandon shopping carts when seeing high flat rate shipping fee. 

2.9% + 30¢ credit card rate

When buyers abandon shopping carts, the above rate is meaningless.

$29/month - Basic Plan

Unlimited items

If most visitors are window shoppers, this package will allow the brand to disclose the entire product catalog for price comparison.

Still NO real time carrier shipping

For auto performance part and accessories, items size and weight varies.  Free shipping for big and heavy items lose money; while flat rate shipping makes shoppers feel unfair and dishonest - abandon shopping carts!!

2.9% + 30¢ credit card rate

A lots of window shoppers ...

Discount Code Engine

I have not use it, so I don't know how complicate a rule can be setup.  To be discovered ...

Fraud analysis tools

These tools show the risk level of each transaction base on the credit card information.  Oversea credit card has a higher risk level than domestic ...  Base on the information, seller can do additional verification or just refund the buyer before the card owner charge back.  This feature is only available for transaction using shopify gateway, at the basic plan tier.

$79/month - Professional Plan

Still NO real time carrier shipping, really!

$50 premium and still no real time carrier shipping, what a rip off.  Easypost only charges 5 cents per shipping label.  However, there is no option to integrate external shipping calculator with shopify checkout workflow.  Let's see what are the other useful features offset the $50 premium.

2.5% + 30¢ credit card rate

Without real time carrier shipping, there are still lots of abandon carts all over cache.

Gift Cards

Gift card strategy does not apply to many merchants, especially small online retailer.  One requirement is the product can be given as gift.

Creating gift card and take 2.5% + 30¢ profit from seller, for a $100 gift card, shopify profit $2.80 from seller and seller received $97.20 cash and $100 liability.  I like this business model.

Professional reports

I have not try the report feature, so no comment and to be discovered.

Abandoned cart recovery

This is a must have feature ☺ when abandoned shopping carts are all over the cache after seeing unfair shipping rate.  Anyway, I have yet to see how this feature work, to be discovered.

$179/month - Unlimited Plan

2.25% + 30¢ credit card rate

Some merchants has already negotiate a better rate, less than 2%, with local bank directly.  Some banks use authorize.net or other third party payment gateways.  I have setup authorize.net with Shopify for Steven's shop.

Advanced report builder

Sure, it is really useful ... for high volume sellers.  Anyway, to be discovered ...

Real time carrier shipping, finally

Let's assume all additional features above Basic Plan are unessential, the cost for getting real time carrier shipping is $150/month.  Base on EasyPost rate, a merchant need to ship over 3000 packages to offset the difference.

There are $100 difference when comparing with Professional Plan, a merchant need to ship over 2000 packages to break even.  I would care less about $100 when I am shipping 2000 packages a month, but what does 2000 packages a month really means?

Online retail store opens 24/7, assume there are 30 days month, the site need to have a average 66.666... sales per day volume to justify upgrading from Professional to Unlimited; and 100 sales per day volume to justify upgrading from Basic to Unlimited.

However, for some online retailer, to make a successful profitable conversion, it is necessary to provide real time carrier shipping to customer; and making the 7 - 10 daily online sales volume is already very successful.

New Project - A small business marketplace that make sense and money

I want to help small businesses, like Stephen's auto parts store, to operate a low cost eCommerce channel to archive better profit margin.  I will leverage the project on cloud platforms from IDE to deployment to realize my idea of how a web site should be built.  

My 2 cents on the future (today and beyond) of web development

I was asked my opinion on the future of web development.  I blasted out my not well thought opinion, only from the development angle.  I think I owe myself a well organized response to express my point of view better next time.

AngularJs + RESTful data service + pre-composed HTML

I believe a web framework/system should handle/serve data and static assets separately.  The job of composing the UI html DOM will be handled by javascript framework - AngularJs.  If there is a need for the home page to display near real time "dynamic content" in a fast and robust fashion, the home page html will be pre-composed by a worker process at the server side and save the html to the CDN servers.

Better Customer Experience

There are many pages, especially home page and SEO pages, serving the same "dynamic content" over a long period of time - P, where P is the length of time the "dynamic content" is updated.  

I think the worst user experience are getting 500 error or a nice error page.  This happen when sever side scripting logic is having uncaught exception to compose the html of a requesting page.  Serving static html with the pre-composed "dynamic content" can effectively reduce the 500 error, and improve customer experience and performance.

When the traditional GET/POST round trip is encountering error, browser is redirect to the error page == customers yell WTF.  UX building around ajax or websocket interaction, provide better workflow experience because both browser and its user are in the same continuous context, rather than redirect, load and rebuild context. 

Same level of Security

The same security protocol that is used to protect a MVC web server can be used to protect the RESTful data service.  After all, RESTful data service is just another MVC web server using the same HTTP protocol.

Better Availability

When 500 responses are reduced, the web site availability increase.

With the separation of static and data server, scaling can be better tailored and customized to the I/O bounded and CPU bounded strategy, in contrast to the server side scripting web system.  For example, static file server, CDN, can adopt caching strategy with less CPU resource, while more CPU/memory budget can be invested into data servers, or even adding more instances to host the heavy weight data service.

Effective scaling improve availability as well.

Better Performance

In terms of composing static HTML, it is simply O(1) vs O(n); the static content needs to be composed once and serve n requests; while server side scripting compose n times to serve n requests.

In terms of constructing dynamic data, the data service only need to construct and return the data payload in an ajax call; while the server side scripting need to construct and return the entire html. 

Complete Separation of Concern

Modern web site heavily depends on javascript scripting.  I remember 10 years ago, web design/development still needed to consider the case of browser with javascript turn off, while today browser having javascript disabled == who cares, not my customer.

Base on the software system design principal - separation of concern, I believe all UI manipulation logic be contained within the same code base as much as possible.  When javascript DOM manipulation is a requirement, the UI logic/assets should be contained in client side code base as much as possible.

Even with the best practice of MVC on server side scripting development, the view and view manipulation logic are sharing responsibility with the client side javascript UI code.  Embedding javascript in the view and in-lining dynamic value in javascript when composing at server side causes more confusion and inconsistency in the system.

Easier for Testing

Since decoupling presentation and data logic into 2 sub system, testing on each sub system is simpler.

For client side testing, a data stub can be easily injected to mock the data coming from data service under AngularJs framework,  The UI testing parameters can be easily setup on client side scripts.

For data service testing, since it is a RESTful web service and the return payload is JSON string, the tests can be easily scripted and automated.  Validating a JSON string is much easier than validating a HTML string.  For lazy developer like myself, I use Chrome App - Postman to do minimum sanity check.

For pre-composed HTML testing, a test step can be added to the end of automated HTML composing process.  The end result HTML can be loaded into headless browser, like PhantomJs or simply be parsed and validated.

Build 4 nodes elasticsearch cluster with Docker for Windows

Install Docker for Windows

From my understanding, Docker is a lightweight virtualization platform which hosts lightweight linux virtual machines. Individual system components can be packaged into multiple containers (VM's), read more details from Docker.

Download Docker for Windows installer and follow the instruction, keep clicking the Next button.

After installation, 2 icons created on Desktop

What have been installed

• Oracle VirtualBox
• MSYS-git (Git for Windows)
• Boot2Docker management tool
• boot2docker Linux ISO

Oracle VirtualBox

This is only used for virtualizing the boot2docker Linux ISO. I believe Docker does not use it to virtualize docker containers.

Program Path

%PROGRAMFILES%\Oracle\VirtualBox

MSYS-git

Program Path

%PROGRAMFILES(x86)%\Git

I have installed Git before installing Docker, now I have 2 Git context menu extension. I think there was an option to unselect Git in the installer.

Boot2Docker

Program Path

%PROGRAMFILES%\Boot2Docker for Windows

The desktop Boot2Docker Start icon points to start.sh

#!/bin/bash.exe

# convert backslash paths to forward slash (yes, really, sometimes you get either)
B2DPATH=${0//\\/\//}
# remove the script-name
B2DPATH=${B2DPATH%/*}
# convert any C:/ into /c/ as MSYS needs this form
B2DPATH=${B2DPATH//C:\//\/c/}
# simplify by adding the program dir to the path
PATH="$B2DPATH:$PATH"

ISO="$USERPROFILE/.boot2docker/boot2docker.iso"

if [ ! -e "$ISO" ]; then
 echo "copying initial boot2docker.iso (run 'boot2docker.exe download' to update"
 mkdir -p "$USERPROFILE/.boot2docker"
 cp "$B2DPATH/boot2docker.iso" "$ISO"
fi

echo "initializing..."
boot2docker.exe init -v
echo "starting..."
boot2docker.exe start
echo "connecting..."
boot2docker.exe ssh

read

The shell script copy boot2docker.iso (boot2docker Linux ISO) from Docker folder to %USERPROFILE%/.boot2docker folder. When Boot2Docker start, it starts the boot2docker-vm from Oracle VM VirtualBox. Double-click on Oracle VM Virtual Desktop icon launches Oracle VM VirtualBox Manager

The default Base Memory setting on the VM is 2GB, it can be adjusted by stopping the boot2docker-vm at the windows command prompt, NOT docker console.

%PROGRAMFILES%\Boot2Docker for Windows\boot2docker down

Then, right click on boot2docker-vm, select Settings... on the context menu. Somehow it does not let me max out all physical memory.

I also changed the default VM Folder location, window menu: File > Preferences... (Ctrl+G)

boot2docker Linux ISO (Console)

Build elasticsearch image

Method 1: docker pull (quick)

At Docker Console

docker pull dockerfile/elasticsearch

Method 2: docker build (can be customized)

At Docker Console

docker build -t="dockerfile/elasticsearch" github.com/dockerfile/elasticsearch

"dockerfile/elasticsearch" can be replace with other namespace/name

See ElasticSearch Dockerfile at GitHub for detail

Run elasticsearch images

docker run -d -P dockerfile/elasticsearch

See Working with Containers for other run command options.

-P option will map exposed port (9200, 9300) automatically, especially necessary for multiple instances in the same docker. See Linking Containers Together for detail.

Issue the command 3 more times to create 4 elasticsearch nodes

Connecting to elasticsearch cluster via ssh tunnel

The following sections apply to elasticsearch cluster server located behind firewall without NAT admin access. Access to a server with sshd is required. I have setup a ubuntu vm at Microsoft Windows Azure, running with minimum resources, as my ssh proxy server.

Setup sshd proxy server in the cloud

Install OpenSSH Server

Generate key

ssh man page

ssh-keygen -b 2048 -t rsa -f <keyfile>

Add public key to ~/.ssh/authorized_keys

cat <keyfile>.pub >> ~/.ssh/authorized_key

Copy the keyfile from cloud server to docker host and home PC

Either use cat and copy the content to clipboard or use scp

Setup SSH Reverse Proxy to elasticsearch Cluster

Find out which port to forward from Docker Console

docker ps

CONTAINER ID        IMAGE               COMMAND                CREATED
   STATUS              PORTS                                              NAMES
4587da9458db        hcst/es:latest      /elasticsearch/bin/e   2 days ago
   Up 2 days           0.0.0.0:49161->9200/tcp, 0.0.0.0:49162->9300/tcp   es4

dc18e9eb1ed1        hcst/es:latest      /elasticsearch/bin/e   2 days ago
   Up 2 days           0.0.0.0:49159->9200/tcp, 0.0.0.0:49160->9300/tcp   es3

698ba8ed7162        hcst/es:latest      /elasticsearch/bin/e   2 days ago
   Up 2 days           0.0.0.0:49155->9200/tcp, 0.0.0.0:49156->9300/tcp   es2

8b6479e41713        hcst/es:latest      /elasticsearch/bin/e   2 days ago
   Up 2 days           0.0.0.0:49153->9200/tcp, 0.0.0.0:49154->9300/tcp   es1

Select any single node and forward the port

ssh -R 9200:localhost:49153 -i <keyfile> -N -f user@server.cloud.com

Just in case outgoing port 22 is blocked by firewall, use other common port which allow out going traffic, such as 443 or 80 ... Just need extra step to map port 443 to 22 of the proxy server.

ssh -p 443 -R 9200:localhost:49153 -i <keyfile> -N -f user@server.cloud.com

Connect to elasticsearch Cluster via SSH Proxy Tunnel from Home PC

Bind a gateway port at Home PC

ssh -L 9200:localhost:9200 -i <keyfile> -N -f user@server.cloud.com

If proxy server port 22 map to other port, such as 443

ssh -p 443 -L 9200:localhost:9200 -i <keyfile> -N -f user@server.cloud.com

Test the connection

curl "http://localhost:9200"

SSH Raspberry Pi with SSH key

Raspberry Pi Documentation

Generate Key at host that remote from

ssh-keygen -t rsa -C <name>@<hostname>

Copy public key to remote host

Create a .ssh folder at Raspberry Pi, if not exists

cat ~/.ssh/id_rsa.pub | ssh <USERNAME>@<IP-ADDRESS> 'cat >> .ssh/authorized_keys'

Connect

ssh <USERNAME>@<IP-ADDRESS>

Building new Raspberry Pi Model B+

Buying the Raspberry Pi Model B+

I ordered the 512MB Raspberry Pi Model B+ Project Board on 7/14/2014, I was lucky somehow I got free shipping code by default, otherwise it cost $9.99 to ship to California with their "Super Saver Shipping" option.  I got the package on the 19th in a padded brown envelope.

Unpacking the Raspberry Pi Model B+

Raspberry Pi box front

Raspberry Pi box back

Raspberry Pi unwrapped

As small as the iPhone5 screen

4xUSB + Fast Ethernet

Micro USB for power supply, HDMI, AV jack

Micro SD card

Building my case with spare LEGO

Lego case base

Fitting the board into lego case

Finished Lego Case

Install NOOBS on SD

Getting Started with NOOBS

Hooking it up

• Keyboard (Dusty DELL generic 101-keyboard>
• Power supply (Apple iPhone/iPod power supply fit the requirement*)
• Micro USB cable as power cable
• Ethernet cable
• HDMI cable
• Micro SD card (8GB+)

* Power supply rated at 5V dc, and a minim current of 600-1800mA.

Turn it on the first time

Raspberry Pi Software Configuration Tool (raspi-config) will be launched automatically after the first boot. It can be launched manually by the command

sudo raspi-config

Change Local

4 > I1 > en_US.UTF-8 UTF-8

Change Timezone

4 > I2 > US > Pacific Ocean

Change Keyboard Layout

4 > I3 > Dell 101-key PC > English (US) > default > No compose key > Ctrl+Alt+Backspace = terminate the X server

Set hostname

8 > A2 > [newhostname]

Enable SSH

8 > A4 > <Enable>

Change Password

2 > [new password]

Update

8 > A7

Hands on compare Go IDEs: Eclipse vs LiteIDE

Trying to start practicing Go, downloaded the latest Go 1.3 and tried to find the best IDE for the job.  I know Go can be written with vi, but it would be difficult for someone using Visual Studio everyday, over 10 years.  Anyway, I google on "Go IDE Comparison" and went with the first result: Comparison of IDEs for Google Go - GeekMonkey.  There were 9 editors but only 2 caught my attention: Eclipse and LiteIDE.  Both IDE has debugger capability and able to run on Mac and PC.

I used LiteIDE on Windows and felt that it was too lite, so I started exploring Eclipse.  Most of my time spent on figuring out how to get the right Java version on my Mac.  I am not a java fan, so I wanted to make the java footprint as small as possible.  I downloaded jre 1.7u60 and 1.8u3, both of them installed properly, but Eclipse failed to start because it could only to find the Apple java 1.6.

$ /usr/libexec/java_home -V only returned "Java SE 6".  It seems like jre only good for running applet on browser.  Then I installed the jdk 1.7 SE, it worked finally, wasted 2 hours of my life.  After starting Eclipse, then I install GoClipse plugin for Go programming, wrote the hello world, started debug, oops ... debug failed, gdb not found.  I had wasted enough time on configuration, move on to LiteIDE.

LiteIDE does not require any runtime environment configuration, it just work out of the box.  I wrote the hello world, it builded and ran fine, then put some break points and tried debug, oops ...  debug failed, gdb not found.  gdb was removed from the new OSX, google and found a detail article on Installing GDB on OS X Mavericks.  Follow the instruction and install gdb from homebrew.  It built but failed to add symbolic link, /usr/local/bin/gdb, permission problem on /usr/local/share/man/man5.  To fix the permisson:

$ sudo chown -R myself /usr/local/share/man/man5
$ brew link gdb
$ sudo chown -R root /usr/local/share/man/man5

Go back to LiteIDE and keep Go-ing.