Azure Cluster Setup

INFRASTRUCTURE – Azure

Microsoft Azure Virtual Machines allow you to deploy a wide range of computing
solutions in an agile way. With Virtual Machines, you can deploy nearly
instantaneously and you pay only by the minute. With Windows, Linux, SQL
Server, Oracle, IBM, SAP, and BizTalk, you can deploy any workload, any
language, on nearly any operating system.

Virtual
Machine

Optimize Azure deployment

Managing and deploying clusters in Azure is a delicate task. Microsoft Azure is
managed through the webportal or API, the latter having many supported plugins.
We use the Azure CLI XPLAT for
managing our cluster, and have our deployment configuration based on
Errordeveloper – Kubernetes on Azure with CoreOS and
Weave

Our goal is to maximize the Azure credits offered by the Microsoft
Bizspark
. The program offers a total of
$750,- per month to spend on Azure, based on 5 account with each $150,- of
credits. We want to optimize the utilization of these accounts, by running 2
Standard D1 instances per subscription, and interconnecting them using Weave
Net. Providing the basis for our Docker container cluster.

Each subscription hosts 2 Standard D1 computing instances, that are linked to a
Cloud Service, in a private VNET, attached to Azure storage for VHD storage,
and are offered a high performance local SSD of 50GB. See our other blog post
for optimizing utilization of this local SSD with Docker.

The Cloud Service and local VNET, save bandwith for local instance to instance
communication. One instance is exposed from the Cloud Service, so it can be
accessed by the other subscriptions to cloudservice.cloudapp.net. We expose
port TCP 6783 and UDP 6783 for Weave to connect the subscriptions.

Install Azure CLI XPLAT

You will need to have Node.js installed on you
machine. If you have previously used Azure CLI, you should have it already

sudo apt-get install nodejs-legacy
sudo apt-get install npm

You install the Azure CLI with

sudo npm install -g azure-cli

Authenticat with Azure

Authentication with Azure can be tricky to start with, as you need to create a
domain account, make this account subscription admin, and use this account to
login to the portal and for api access. We found a great guide for setting this
up at How to connect to your Azure
subscription

Next you will need to download your .publishsettings file, for authentication
with the API.

azure account download

This will return the link to
download the .publishsettings file.

info:    Executing command account download
info:    Launching browser to http://go.microsoft.com/fwlink/?LinkId=254432
help:    Save the downloaded file, then execute the command
help:      account import <file>
info:    account download command OK

Deployment to Azure

We use configuration files per subscription cluster. This configuration file is
formatted in YAML. The reason we use these config files, is for
reviewability and as documentation. We use these config files to create,
deploy, redeploy, remove, and destroy the cluster. Allowing us to move a
configuration between subscriptions.

For our first subscription cluster, we use the following configuration file.
> Because this is a development deployment, we store passwords in the
> configuration file. Second switching subscriptions, sometimes requires
> logging in, as well as the .publishsettings file.

#deploy-config

name: "cluster-01"

ssh_key:
  key: "../cert/ssh.key"
  pem: "../cert/ssh.pem"

account:
  login:
    user: "USER@DOMAIN.onmicrosoft.com"
    password: "PASSWORD"
  subscription:
    id: "12357234-12473485-1234852"
    publish_file: "../conf/azure/cluster-01.publishsettings"
    name: "BizSpark"
    user:
      - name: "USER@DOMAIN.onmicrosoft.com"
        type: "user"
        password: "PASSWORD"

  affinity_group:
    name: "cluster-01-ag"
    label: "cluster-01-ag"
    location: "West Europe"

storage:
  account:
    name: "clusterstorage"
    label: "clusterstorage"
    affinity_group: "cluster-01-ag"
    location: "West Europe"
    type: "LRS"

network:
  vnet:
    name: "clustervnet"
    address_space: "172.16.1.0"
    cidr: "24"
    location: "West Europe"
    affinity_group: "cluster-01-ag"

  reserved_ip:
    name: "cluster-01-ip"
    label: "cluster-01-ip"
    location: "West Europe"

virtual_machines:
  - name: "azure-prd-01"
    image: "2b171e93f07c4903bcad35bda10acf22__CoreOS-Beta-723.3.0"
    size: "Standard_D1"
    service: "cluster-01"
    location: "West Europe"
    affinity_group: "cluster-01-ag"
    ssh_port: "2201"
    ssh_cert: "../cert/ssh.pem"
    vnet: "clustervnet"
    # IPs 1-10 are reserved in Azure
    ip: "172.16.1.10"
    reserved_ip: "cluster-01-ip"
    custom_data: "../conf/coreos/cluster-01.yaml"

    endpoints:
      - name: "Weave-TCP"
        port: "6783"
        protocol: "TCP"

      - name: "Weave-UDP"
        port: "6783"
        protocol: "UDP"

  - name: "azure-prd-02"
    image: "2b171e93f07c4903bcad35bda10acf22__CoreOS-Beta-723.3.0"
    #image: "2b171e93f07c4903bcad35bda10acf22__CoreOS-Alpha-752.1.0"
    size: "Standard_D1"
    service: "cluster-01"
    location: "West Europe"
    affinity_group: "cluster-01-ag"
    ssh_port: "2202"
    ssh_cert: "../cert/ssh.pem"
    vnet: "clustervnet"
    # IPs 1-10 are reserved in Azure
    ip: "172.16.1.11"
    reserved_ip: "cluster-01-ip"
    custom_data: "../conf/coreos/cluster-01.yaml"

Deploy the subscription cluster using the following command.

./deploy.js deploy ../conf/azure/cluster-01.yaml

Deployment file

  Usage: deploy [options] [command]


  Commands:

    create [options] <config>    Create the config deployment
    destroy [options] <config>   Destroy the deployed config
    deploy [options] <config>    Deploy machines to the config deployment
    remove [options] <config>    Remove machines from the deployed config
    redeploy [options] <config>  Re-deploy machines to the config deployment

  Options:

    -h, --help           output usage information
    -V, --version        output the version number
    -c, --config <path>  set config path.
    -v, --verbose        Verbose

The deployment file offers you different options. To setup a fresh cluster use
the following commands.

Create:

./deploy.js create ../conf/azure/cluster-01.yaml
./deploy.js deploy ../conf/azure/cluster-01.yaml

To destroy the cluster, use the following commands.

./deploy.js remove ../conf/azure/cluster-01.yaml
./deploy.js detroy ../conf/azure/cluster-01.yaml

Azure Local SSD optimization

INFRASTRUCTURE – Azure Local SSD

Special for Azure D# instances, these instances are equiped with a local SSD
drive. This local SSD drive needs to be mounted to be used by Docker and
Kubernetes.

The local SSD drive offers a huge performance boost, compared to attached
disks. The internal SSD is not limited on IO and throughput. See the
D-Series Performance
Expectations

blog post for more information about the Azure Standard D1 performance.

> One side note, the local disk is ephemeral, so on host reboot it can be
> wiped, and should therfor only be used as temp storage.

Second we use OverlayFS, as compared to BTFRS for Docker. Which also provides
us with a huge performance boost. See the blog post Overview Storage
Scalability
Docker

for more information.

Deployment

We deploy the Azure Standard D1 host with the following #cloud-config. First
the tmp drive /dev/sdb is unmounted, wiped, formatted as ext4. Second it is
mounted as docker storage /var/lib/docker.

<br />#cloud-config

coreos:
  units:
    # AZURE
    - name: docker.service
      drop-ins:
        - name: 10-overlayfs.conf
          content: |
            [Service]
            Environment='DOCKER_OPTS="--storage-driver=overlay"'

    - name: format-ephemeral.service
      command: start
      content: |
        [Unit]
        Description=Format Ephemeral Volume
        Documentation=https://coreos.com/os/docs/latest/mounting-storage.html
        Before=docker.service var-lib-docker.mount
        After=dev-sdb.device
        Requires=dev-sdb.device
        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/bin/bash -c '/usr/bin/umount -f /mnt/resource || /bin/true'
        ExecStart=/bin/bash -c '/usr/bin/umount -A /dev/sdb || /bin/true'
        ExecStart=/usr/bin/rm -rf /mnt/resource
        ExecStart=/usr/sbin/wipefs -f /dev/sdb
        ExecStart=/usr/sbin/mkfs.ext4 -F /dev/sdb
        [Install]
        RequiredBy=var-lib-docker.mount

    - name: var-lib-docker.mount
      enable: true
      content: |
        [Unit]
        Description=Mount /var/lib/docker
        Documentation=https://coreos.com/os/docs/latest/mounting-storage.html
        Before=docker.service
        After=format-ephemeral.service
        Requires=format-ephemeral.service
        [Install]
        RequiredBy=docker.service
        [Mount]
        What=/dev/sdb
        Where=/var/lib/docker
        Type=ext4

Kubernetes Cluster Setup

INFRASTRUCTURE – Kubernetes

Manage a cluster of Linux containers as a single system to accelerate Dev and
simplify Ops.

Kubernetes is an open source orchestration system
for Docker containers. It handles scheduling onto nodes in a compute cluster
and actively manages workloads to ensure that their state matches the users
declared intentions. Using the concepts of “labels” and “pods”, it groups the
containers which make up an application into logical units for easy management
and discovery.

Kubernetes
Overview

We use Kubernetes for orchestrating our cluster. Kubernetes allows us to deploy
Docker images to hosts in our cluster, and allows us to monitor these
deployments. The main advantage for us, is not to worry about how and where
the instance is deployed, and ensures the instance is relaunched on crash.

Please mind, that you need to follow The Twelve-factor
App
methodology. Enabling your container instances to be
stateless, and can operate together in a cluster.

> A instance should be designed in a way, it can crash and relaunched, without
effecting other instances in the cluster.

Deploying Kubernetes

Kubernetes is based on master – node methodology. Having the nodes being
managed by the master Kubernetes instance.

We spawn one Kubernetes master on our azure-prd-01 host. Because this host is
a high available cloud vm, launched on Azure. Our nodes consist of
azure-prd-02, azure-prd-03, azure-prd-04 and, ded-prd-01.

Our goal is to spawn high-available instances on the Azure hosts, and
processing intensive instances on the ded-prd-01 host.

The Docker images we use are hosted on Quay.io, providing
us with a private image registry. The login details for Quay can be placed in a
.dockercfg file, that can be deployed in the #cloud-config, replace PASSWORD
with your access token.

We use the following #cloud-config for deployment of our Kubernetes cluster
on CoreOS, with Weave Net.

Setting up Kubernetes

#cloud-config

write_files:
  - path: /opt/bin/wupiao
    permissions: '0755'
    content: |
      #!/bin/bash
      # [w]ait [u]ntil [p]ort [i]s [a]ctually [o]pen
      [ -n "$1" ] && 
        until curl -o /dev/null -sIf http://${1}; do 
          sleep 1 && echo .;
        done;
      exit $?

  # Kubernetes TARBALL location
  - path: /etc/kubernetes.env
    permissions: 0644
    owner: root
    content: |
      KUBE_RELEASE_TARBALL=https://github.com/GoogleCloudPlatform/kubernetes/releases/download/v1.0.1/kubernetes.tar.gz

  #Docker config
  - path: /var/lib/kubelet/.dockercfg
    owner: core:core
    permissions: 0644
    content: |
      {
        "quay.io": {
          "auth": "PASSWORD",
          "email": ""
        }
      }

  #Docker config
  - path: /home/core/.dockercfg
    owner: core:core
    permissions: 0600
    content: |
      {
        "quay.io": {
          "auth": "PASSWORD",
          "email": ""
        }
      }

coreos:
  units:
    - name: kubernetes.master.target
      enable: true
      command: start
      content: |
        [Unit]
        Description=Kubernetes Cluster Master
        Documentation=http://kubernetes.io/
        RefuseManualStart=no
        After=weave-network.target etcd2.service
        Requires=weave-network.target etcd2.service
        ConditionHost=azure-prd-01
        Wants=kubernetes.download.service
        Wants=kubernetes.apiserver.service
        Wants=kubernetes.scheduler.service
        Wants=kubernetes.controller-manager.service
        [Install]
        WantedBy=multi-user.target

    - name: kubernetes.node.target
      enable: true
      command: start
      content: |
        [Unit]
        Description=Kubernetes Cluster Node
        Documentation=http://kubernetes.io/
        RefuseManualStart=no
        ConditionHost=!azure-prd-01
        After=weave-network.target etcd2.service
        Requires=weave-network.target etcd2.service
        Wants=kubernetes.download.service
        Wants=kubernetes.proxy.service
        Wants=kubernetes.kubelet.service
        [Install]
        WantedBy=multi-user.target

    - name: kubernetes.download.service
      enable: true
      content: |
        [Unit]
        Description=Download Kubernetes Binaries
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes
        After=network-online.target systemd-networkd-wait-online.service
        Requires=network-online.target systemd-networkd-wait-online.service
        [Service]
        EnvironmentFile=/etc/kubernetes.env
        ExecStartPre=/bin/mkdir -p /opt/bin/
        ExecStart=/bin/bash -c "curl --silent --location $KUBE_RELEASE_TARBALL | tar xzv -C /tmp/"
        ExecStart=/bin/tar xzvf /tmp/kubernetes/server/kubernetes-server-linux-amd64.tar.gz -C /opt
        ExecStartPost=/usr/bin/chmod o+rx -R /opt/kubernetes
        ExecStartPost=/bin/ln -sf /opt/kubernetes/server/bin/kubectl /opt/bin/
        ExecStartPost=/bin/rm -rf /tmp/kubernetes
        RemainAfterExit=yes
        Type=oneshot

Kubernetes Master

Specific to the Kubernetes master are the following services:
– Api
– Scheduler
– Controller-Manager

The API server exposes port 8080, allowing nodes to interact with the master
server. Because we are using a private bridge network, using Weave Net, we do
not use TLS, this is possible and we will look into this in the future.

The following #cloud-config will be used for the Kubernetes Master on
azure-prd-01.

coreos:
  units:
    # Kubernetes Master on azure-prd-01
    - name: kubernetes.apiserver.service
      enable: true
      content: |
        [Unit]
        Description=Kubernetes API Server
        Documentation=http://kubernetes.io/
        ConditionFileIsExecutable=/opt/kubernetes/server/bin/kube-apiserver
        Before=kubernetes.controller-manager.service kubernetes.scheduler.service
        After=etcd2.service kubernetes.download.service
        Wants=etcd2.service kubernetes.download.service
        ConditionHost=azure-prd-01
        [Service]
        ExecStart=/opt/kubernetes/server/bin/kube-apiserver \
            --address=0.0.0.0 \
            --port=8080 \
            --kubelet-https=true \
            --secure-port=6443 \
            --service-cluster-ip-range=10.100.0.0/16 \
            --etcd-servers=http://10.1.0.1:2379,http://10.1.0.2:2379 \
            --cloud-provider=vagrant \
            --logtostderr=true \
            --v=3
        Restart=always
        RestartSec=10
        [Install]
        WantedBy=kubernetes.master.target

    - name: kubernetes.scheduler.service
      enable: true
      content: |
        [Unit]
        Description=Kubernetes Scheduler
        Documentation=http://kubernetes.io/
        ConditionFileIsExecutable=/opt/kubernetes/server/bin/kube-scheduler
        After=kubernetes.apiserver.service
        Wants=kubernetes.apiserver.service
        ConditionHost=azure-prd-01
        [Service]
        ExecStartPre=/opt/bin/wupiao 10.1.0.1:8080
        ExecStart=/opt/kubernetes/server/bin/kube-scheduler \
            --logtostderr=true \
            --master=127.0.0.1:8080
        Restart=always
        RestartSec=10
        [Install]
        WantedBy=kubernetes.master.target

    - name: kubernetes.controller-manager.service
      enable: true
      content: |
        [Unit]
        Description=Kubernetes Controller Manager
        Documentation=http://kubernetes.io/
        ConditionFileIsExecutable=/opt/kubernetes/server/bin/kube-controller-manager
        After=kubernetes.apiserver.service
        Wants=kubernetes.apiserver.service
        ConditionHost=azure-prd-01
        [Service]
        ExecStartPre=/bin/bash -x -c 'result=`wget --retry-connrefused --tries=5 127.0.0.1:8080/healthz -O -` && test -n "$${result}" && test "$${result}" = ok'
        ExecStart=/opt/kubernetes/server/bin/kube-controller-manager \
            --cloud-provider=vagrant \
            --master=127.0.0.1:8080 \
            --logtostderr=true
        Restart=always
        RestartSec=10
        [Install]
        WantedBy=kubernetes.master.target

Kubernetes Node

Kubernetes nodes host the Kubelet service, and the proxy service. Allowing
communication and performing orchestration on the host.

We use the following #cloud-config on the node hosts.

coreos:
  units:
    - name: kubernetes.kubelet.service
      enable: true
      content: |
        [Unit]
        Description=Kubernetes Kubelet
        Documentation=http://kubernetes.io/
        ConditionFileIsExecutable=/opt/kubernetes/server/bin/kubelet
        After=kubernetes.download.service
        Wants=kubernetes.download.service
        ConditionHost=!azure-prd-01
        [Service]
        ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests/
        ExecStart=/opt/kubernetes/server/bin/kubelet 
            --address=0.0.0.0 
            --port=10250 
            --api-servers=http://10.1.0.1:8080 
            --logtostderr=true 
            --config=/etc/kubernetes/manifests/ 
            --register-node
        # Hostname override allows refering kubelets on master by hostname
        #    --hostname-override=%H 
        #    --cluster-dns=10.1.0.3 
        #    --cluster-domain=kube.local 
        Restart=always
        RestartSec=10
        [Install]
        WantedBy=kubernetes.node.target

    - name: kubernetes.proxy.service
      enable: true
      content: |
        [Unit]
        Description=Kubernetes Proxy
        Documentation=http://kubernetes.io/
        ConditionFileIsExecutable=/opt/kubernetes/server/bin/kube-proxy
        After=kubernetes.download.service
        Wants=kubernetes.download.service
        ConditionHost=!azure-prd-01
        [Service]
        ExecStart=/opt/kubernetes/server/bin/kube-proxy 
             --master=http://10.1.0.1:8080 
             --logtostderr=true
        Restart=always
        RestartSec=10
        [Install]
        WantedBy=kubernetes.node.target

Manual node registration

It is possible to manually register nodes to the master, including extra
information, as labels, on the node. This might also be usefull in debugging
problems with registering nodes.

Make sure the –register-node=false on the kubelet service, for manual node
registration.

Use the following #cloud-config for manual node registration.

write_files:
  # Optional for registering nodes with the Kubernetes master
  - path: /opt/bin/register_node.sh
    permissions: '0755'
    owner: root
    content: |
      #!/bin/sh -xe
      node_id="${1}"
      master_url="${2}"
      env_label="${3}"
      until healthcheck=$(curl --fail --silent "${master_url}/healthz")
      do sleep 2
      done
      test -n "${healthcheck}"
      test "${healthcheck}" = "ok"
      printf '{
        "id": "%s",
        "kind": "Minion",
        "apiVersion": "v1beta1",
        "labels": { "environment": "%s", "host": "azure", "name": "%s" }
        }' "${node_id}" "${env_label}" "${node_id}" 
        | /opt/bin/kubectl create -s "${master_url}" -f -
      if [ $? -ne 0 ]
        then echo "Failed registering node, already registered?"
        else echo "Successfully registered node"
      fi

coreos:
  units:
    - name: kubernetes.register-node.service
      enable: true
      content: |
        [Unit]
        Description=Kubernetes Create Node
        Documentation=http://kubernetes.io/
        ConditionFileIsExecutable=/opt/bin/kubectl
        ConditionFileIsExecutable=/opt/bin/register_node.sh
        After=kubernetes.download.service
        Before=kubernetes.proxy.service kubernetes.kubelet.service
        Wants=kubernetes.download.service
        ConditionHost=!azure-prd-01
        [Service]
        ExecStart=/opt/bin/register_node.sh %H http://10.0.1.0:8080 production
        Type=oneshot
        [Install]
        WantedBy=kubernetes.node.target

CoreOS Cluster setup

INFRASTRUCTURE – CoreOS

Setting up CoreOS

CoreOS is designed for security, consistency, and reliability. Instead of
installing packages via yum or apt, CoreOS uses Linux containers to manage your
services at a higher level of abstraction. A single service’s code and all
dependencies are packaged within a container that can be run on one or many
CoreOS machines.

We will use CoreOS as operating system for hosting our containers. As the focus
of CoreOS is for cluster configurations. CoreOS will be our host, that will run
Docker, on which we can deploy our containers.

The deployment of CoreOS is easy, for bare-metal using CoreOS
Install

or to Azure using Azure CLI
install
.

CoreOS is bootstrapped with a #cloud-config file, in which you can configure
the services on the machine. We use this file to bootstrap Weave and Kubernetes
on our hosts. The #cloud-config files will be reloaded on host reboot,
refreshing the configuration of the host, while files will be left unchanged.

The deployed #cloud-config file can be found in:
– Bare metal
/var/lib/coreos-install/user-data
– Azure
/var/lib/waagent/CustomData

CoreOS offers various integrated services, from which we require
etcd a highly-available key value store for
shared configuration and service discovery. Etcd will be used by
Kubernetes for node discovery and key-value store.

Configuration

#cloud-config

A stripped down version of our #cloud-config file, including the etcd2
service configuration. This configuration file will be included to deploy our
host machines.

The configuration file below will be used for the first cluster azure-prd-01
and azure-prd-02. This cluster will use a Weave network bridge, to create a
network bridge between our clusters. Etcd will use the Weave Bridge
ip-addresses.

<br />#cloud-config

write_files:
  - path: /etc/etcd2.env
    permissions: 0644
    owner: root
    content: |
      ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
      ETCD_INITIAL_CLUSTER="azure-prd-01=http://10.1.0.1:2380,azure-prd-02=http://10.1.0.2:2380,azure-prd-03=http://10.1.0.3:2380,azure-prd-04=http://10.1.0.4:2380"
      ETCD_INITIAL_CLUSTER_STATE="new"

  - path: /etc/etcd2.azure-prd-01.env
    permissions: 0644
    owner: root
    content: |
      ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.1.0.1:2380"
      ETCD_LISTEN_PEER_URLS="http://10.1.0.1:2380,http://10.1.0.1:7001"
      ETCD_LISTEN_CLIENT_URLS="http://10.1.0.1:2379,http://10.1.0.1:4001,http://127.0.0.1:2379"
      ETCD_ADVERTISE_CLIENT_URLS="http://10.1.0.1:2379"

  - path: /etc/etcd2.azure-prd-02.env
    permissions: 0644
    owner: root
    content: |
      ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.1.0.2:2380"
      ETCD_LISTEN_PEER_URLS="http://10.1.0.2:2380,http://10.1.0.2:7001"
      ETCD_LISTEN_CLIENT_URLS="http://10.1.0.2:2379,http://10.1.0.2:4001,http://127.0.0.1:2379"
      ETCD_ADVERTISE_CLIENT_URLS="http://10.1.0.2:2379"

coreos:
  update:
    group: beta
    reboot-strategy: off

  etcd2:
    heartbeat-interval: 500
    election-timeout: 2500

  units:
    - name: etcd2.service
      drop-ins:
        - name: 50-environment-variables.conf
          content: |
            [Service]
            Environment=ETCD_DATA_DIR=/var/lib/etcd2
            Environment=ETCD_NAME=%H
            EnvironmentFile=-/etc/etcd2.env
            EnvironmentFile=-/etc/etcd2.%H.env

Running our host with the above configuration, will spawn 2 host machines, both
running etcd2 and using the CoreOS beta image.

Secondary clusters

The second cluster, hosting azure-prd-03 and azure-prd-04 will use a
similar configuration file.

<br />#cloud-config

write_files:
  - path: /etc/etcd2.env
    permissions: 0644
    owner: root
    content: |
      ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
      ETCD_INITIAL_CLUSTER="azure-prd-01=http://10.1.0.1:2380,azure-prd-02=http://10.1.0.2:2380,azure-prd-03=http://10.1.0.3:2380,azure-prd-04=http://10.1.0.4:2380"
      ETCD_INITIAL_CLUSTER_STATE="new"

  - path: /etc/etcd2.azure-prd-03.env
    permissions: 0644
    owner: root
    content: |
      ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.1.0.3:2380"
      ETCD_LISTEN_PEER_URLS="http://10.1.0.3:2380,http://10.1.0.3:7001"
      ETCD_LISTEN_CLIENT_URLS="http://10.1.0.3:2379,http://10.1.0.3:4001,http://127.0.0.1:2379"
      ETCD_ADVERTISE_CLIENT_URLS="http://10.1.0.3:2379"

  - path: /etc/etcd2.azure-prd-04.env
    permissions: 0644
    owner: root
    content: |
      ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.1.0.4:2380"
      ETCD_LISTEN_PEER_URLS="http://10.1.0.4:2380,http://10.1.0.4:7001"
      ETCD_LISTEN_CLIENT_URLS="http://10.1.0.4:2379,http://10.1.0.4:4001,http://127.0.0.1:2379"
      ETCD_ADVERTISE_CLIENT_URLS="http://10.1.0.4:2379"

coreos:
  update:
    group: beta
    reboot-strategy: off

  etcd2:
    heartbeat-interval: 500
    election-timeout: 2500

  units:
    - name: etcd2.service
      drop-ins:
        - name: 50-environment-variables.conf
          content: |
            [Service]
            Environment=ETCD_DATA_DIR=/var/lib/etcd2
            Environment=ETCD_NAME=%H
            EnvironmentFile=-/etc/etcd2.env
            EnvironmentFile=-/etc/etcd2.%H.env

Dedicated machine

The dedicated machine we will include in this cluster, requires a different
#cloud-config file, as this will be deployed on bare-metal. Deployments on
Azure or other cloud providers, include default #cloud-config settings, as
the host-name, ssh-keys, and system settings. For dedicated machines these need
to be included in the #cloud-config. This dedicated server will not run etcd
, as we are running etcd on our high-available machines.

Below is our #cloud-config file for the dedicated machine.

<br />#cloud-config
ssh_authorized_keys:
  - ssh-rsa ###KEY###

hostname: ded-prd-01

coreos:
  update:
    group: beta
    reboot-strategy: 'off'

Deployment

Deployment of our hosts to Azure and the bare-metal machine, are done using the
following commands.

Azure

Deployment to Azure will be done with the Azure CLI
XPLAT

written in node.js. The following command can be used for deployment to Azure.

azure vm create --custom-data=cloud-config.yaml --vm-size=Small --ssh=22
--ssh-cert=path/to/cert --no-ssh-password --vm-name=node-1 --location="West US"
my-cloud-service 2b171e93f07c4903bcad35bda10acf22__CoreOS-Beta-723.3.0 core

For our production deployment, we use a customized node.js Azure deployment,
based on the implementation of
Errordeveloper.
That allows us to define storage, network, reserved-ip and cloud-service.
This gives more flexibility in the implementation, as well as better
documentation of the implementation. A follow-up blog post will be dedicated to
this.

Dedicated

Our dedicated server is hosted at Hetzner a German
dedicated server supplier, offering great performance for a good price.

You have to boot into the rescue
mode
of your server
, this will allow you to install CoreOS using the following commands.

wget https://raw.github.com/coreos/init/master/bin/coreos-install
chmod +x ./coreos-install
./coreos-install -d /dev/sda -C beta -c cloud-config.yaml

Weave Cluster Setup

Setting up Weave

Weave Net makes it as easy as possible for developers to create a network of
Docker containers. Weave Net consists of a network layer, dns service, and
docker plugin. Weave provides a total solution for your container network, as
well for the host machines it operates on.

We run our Docker cluster on CoreOS. We want to run our Docker instances in
combination with a Weave network. This allows us to connect Docker instances
over multiple hosts, and even multiple clouds. We use Weave IPAM for automatic
ip allocation to Docker instances, and Weave DNS to allow containers looking up
other containers through simple DNS queries.

Our network is divided in multiple layers, we have one container network, one
host bridge network, and optional local cloud networks. Our focus is connecting
the host machines for cluster orchestration, and a container network for a
internal container network, separated from external networks. Cluster
orchestration will be managed by Google Kubernetes, which also offers container
service discovery for exposing containers to external services.

Following a diagram of the cluster setup, including 2 Microsoft Azure Bizspark
subscriptions, and one dedicated server. There are 3 layers of network,
internal virtual network (172.16.1.10/24), internal Weave bridge network
(10.1.0.0/16), and Weave container network (10.2.0.0/16). The Docker container
subnet is 10.2.0.0/16, where we use Weave IPAM for automatic IP address
allocation to Docker instances.

Weave Network

TL;DR; Final Weave network config

Default cloud-config file

#cloud-config

write_files:
  - path: /etc/weave.env
    permissions: 0644
    owner: root
    content: |
      WEAVE_PASSWORD="PASSWORD"
      WEAVE_BREAKOUT_ROUTE="10.1.0.0/16"
      WEAVE_IPRANGE="10.2.0.1/16"

coreos:
  units:
    # WEAVE
    - name: weave-network.target
      enable: true
      content: |
        [Unit]
        Description=Weave
        Documentation=man:systemd.special(7)
        RefuseManualStart=no
        After=network-online.target
        Requires=install-weave.service weave-create-bridge.service weave.service
        Requires=weavedns.service weaveproxy.service
        [Install]
        WantedBy=multi-user.target
        WantedBy=kubernetes-master.target
        WantedBy=kubernetes-node.target

    - name: 10-weave.network
      runtime: false
      content: |
        [Match]
        Type=bridge
        Name=weave*
        [Network]

    - name: install-weave.service
      enable: true
      content: |
        [Unit]
        Description=Install Weave
        Documentation=http://docs.weave.works/
        Before=weave.service
        After=network-online.target docker.service
        Requires=network-online.target docker.service
        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStartPre=/bin/mkdir -p /opt/bin/
        ExecStartPre=/usr/bin/wget -O /opt/bin/weave 
          https://github.com/weaveworks/weave/releases/download/latest_release/weave
        ExecStartPre=/usr/bin/chmod +x /opt/bin/weave
        ExecStartPre=/opt/bin/weave --local setup
        ExecStart=/bin/echo Weave Installed
        [Install]
        WantedBy=weave-network.target

    - name: weave-create-bridge.service
      enable: true
      content: |
        [Unit]
        Description=Weave Create Bridge
        Documentation=http://docs.weave.works/
        Before=weave.service
        After=install-weave.service
        Requires=install-weave.service
        [Service]
        Type=oneshot
        RemainAfterExit=yes
        EnvironmentFile=/etc/weave.env
        EnvironmentFile=/etc/weave.%H.env
        ExecStart=/opt/bin/weave --local create-bridge
        # Workaround for rebooting
        ExecStart=/bin/bash -c '/usr/bin/ip addr add dev weave $WEAVE_BRIDGE_ADDRESS || /bin/true'
        ExecStart=/bin/bash -c '/usr/bin/ip route add $WEAVE_BREAKOUT_ROUTE dev weave scope link || /bin/true'
        # Multicast routing
        ExecStart=/bin/bash -c '/usr/bin/ip route add 224.0.0.0/4 dev weave || /bin/true'
        [Install]
        WantedBy=weave-network.target

    # http://docs.weave.works/weave/latest_release/features.html
    - name: weave.service
      enable: true
      content: |
        [Unit]
        Description=Weave Net
        Documentation=http://docs.weave.works/
        After=docker.service install-weave.service
        Requires=docker.service install-weave.service
        [Service]
        TimeoutStartSec=0
        EnvironmentFile=-/etc/weave.env
        EnvironmentFile=-/etc/weave.%H.env
        ExecStartPre=/opt/bin/weave launch -iprange $WEAVE_IPRANGE $WEAVE_PEERS
        ExecStart=/usr/bin/docker attach weave
        Restart=on-failure
        ExecStop=/opt/bin/weave stop
        [Install]
        WantedBy=weave-network.target

    # http://docs.weave.works/weave/latest_release/weavedns.html
    - name: weavedns.service
      enable: true
      content: |
        [Unit]
        Description=Weave Run - DNS
        Documentation=http://docs.weave.works/
        After=weave.service
        Requires=weave.service
        [Service]
        TimeoutStartSec=0
        EnvironmentFile=-/etc/weave.env
        EnvironmentFile=-/etc/weave.%H.env
        ExecStartPre=/opt/bin/weave launch-dns
        ExecStart=/usr/bin/docker attach weavedns
        Restart=on-failure
        ExecStop=/opt/bin/weave stop-dns
        [Install]
        WantedBy=weave-network.target

    # http://docs.weave.works/weave/latest_release/proxy.html
    - name: weaveproxy.service
      enable: true
      content: |
        [Unit]
        Description=Weave Run - PROXY
        Documentation=http://docs.weave.works/
        After=weavedns.service
        Requires=weavedns.service
        [Service]
        TimeoutStartSec=0
        EnvironmentFile=-/etc/weave.env
        EnvironmentFile=-/etc/weave.%H.env
        ExecStartPre=/opt/bin/weave launch-proxy --with-dns
        ExecStart=/usr/bin/docker attach weaveproxy
        Restart=on-failure
        ExecStop=/opt/bin/weave stop-proxy
        [Install]
        WantedBy=weave-network.target

Cluster-01 – azure-prd-01 and azure-prd-02

Configuration settings for azure-prd-01 and azure-prd-02.

#cloud-config

write_files:
  - path: /etc/weave.azure-prd-01.env
    permissions: 0644
    owner: root
    content: |
      WEAVE_PEERS=""
      WEAVE_BRIDGE_ADDRESS="10.1.0.1/16"

  - path: /etc/weave.azure-prd-02.env
    permissions: 0644
    owner: root
    content: |
      WEAVE_PEERS="azure-prd-01"
      WEAVE_BRIDGE_ADDRESS="10.1.0.2/16"

Cluster-02 – azure-prd-03 and azure-prd-04

Configuration settings for azure-prd-03 and azure-prd-04.

#cloud-config

write_files:
  - path: /etc/weave.azure-prd-03.env
    permissions: 0644
    owner: root
    content: |
      WEAVE_PEERS="cluster-01.cloudapp.net"
      WEAVE_BRIDGE_ADDRESS="10.1.0.3/16"

  - path: /etc/weave.azure-prd-04.env
    permissions: 0644
    owner: root
    content: |
      WEAVE_PEERS="azure-prd-03"
      WEAVE_BRIDGE_ADDRESS="10.1.0.4/16"

Cluster-03 – ded-prd-01

Configuration settings for the dedicated server ded-prd-01.

#cloud-config

write_files:
  - path: /etc/weave.azure-prd-03.env
    permissions: 0644
    owner: root
    content: |
      WEAVE_PEERS="cluster-01.cloudapp.net"
      WEAVE_BRIDGE_ADDRESS="10.1.0.5/16"

We use system environment variables per host, for defining the fixed host ip in
the Weave bridge network, and the Weave settings as password and container
subnet.

We start Weave with weave launch -iprange $WEAVE_SUBNET $WEAVE_PEERS where
the $WEAVE_SUBNET is 10.2.0.0/16 and $WEAVE_PEERS are the other hosts
Weave should connect to. Weave will automatically discover the other hosts in
the network and establish connections to them if it can (in order to avoid
unnecessary multi-hop routing). For more information about the subnet, see
Application Isolation

#TODO: Review if fixed ip is better
For DNS operations in the cluster and between instances, we use Weave DNS. For
control over DNS instances, we use fixed IP addresses for our Weave DNS
instances. These need to be provided with a CIDR, in our case we use
10.254.0.1/24

By using Weave Proxy service, Weave will automatically integrate with Docker.
This is a better option than using a bridge for Docker. See reference blog post
Bridge over troubled Weavers

Using the Weave bridge, we can communicate through this bridge to all machines
in the cluster. We use fixed ip-addresses for each machine in the cluster.
This allowes Kubernetes to orchestrate between cloud environments.