
I've been working lately with ETL workflows, so I'd like to take some time to explain how to build a simple one from scratch. Note that you will need an AWS account for this tutorial.

## What is an ETL

ETL is acronym for Extract, Tranform, Load and it is used as a procedure to copy data from a source to a destination with potentially different data structure, often used if data warehousing.

There are a lot of different use cases for ETL:
* You might have data in a database and want to do a migration to a differnt one, for example because an adquisition was made by a bigger company and the data needs to be transformed to a new format.
* You might have historical data that you want to do BI queries on them, so a data warehouse is the best storage format.

## Architecture

> `Quotes Website` -> **Quotes Scrapy** -> `S3` -> `SQS` -> **Quotes ETL** -> `S3`

**Quotes Scrapy**: For our quick ETL example we need some sort of data flowing our way. We'll be doing some scrapping of https://quotes.toscrape.com/ and retrieving some quotes in JSON format and store them in an S3 bucket `quotes-scrapy`.

**SQS**: In order to be able to trigger actions when a file is uploaded to the S3 bucket, we'll create an Amazon SQS (Simple Queue Service) and configure the S3 bucket to send a notification to the queue whenever a new file is stored.

**Quotes ETL**: Now that we have the files in the bucket and the messages in the queue, we'll create an ETL using Python that reads the messages from the SQS and for each message it reads the input file, transforms it and writes the output to a separate S3 bucket `quotes-scrapy-warehouse`.

## Step 1: Quotes Scrapy

For the first step we'll use [scrapy](https://scrapy.org/) which is a framework for extracting data from websites.

We'll be using the same example they are documenting in their [Scrapy Tutorial](https://docs.scrapy.org/en/latest/intro/tutorial.html) with simple adjustment so that we get JSON files written directly to our S3 bucket on each execution.

Create project

```sh
scrapy startproject quotes_scrapy
```

create a spider `quotes_scrapy/quotes_scrapy/spiders/quotes_spider.py`:

```python
import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
    ]

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('span small::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }

        next_page = response.css('li.next a::attr(href)').get()
        if next_page is not None:
            yield response.follow(next_page, callback=self.parse)
```

Run our spider:

```sh
scrapy crawl quotes
```

Configure output in `tutorial/settings.py`:

```python
# append at the end of file

FEEDS={
    'items.json': {
        'format': 'json',
        'encoding': 'utf8',
        'store_empty': False,
        'fields': None,
        'indent': 4,
        'item_export_kwargs': {
           'export_empty_fields': True,
        },
    },
    's3://quotes-scrapy/items-%(time)s.json': {
        'format': 'json',
        'encoding': 'utf8',
        'store_empty': False,
        'fields': None,
        'indent': 4,
        'item_export_kwargs': {
           'export_empty_fields': True,
        },
    },
}
```

Now run the crawl again and it will generate an `items.json` locally and a file with timestamp as suffix on the S3 bucket. This way for each execution we'll get a different JSON file written to the S3 bucket.

Here is an example of the JSON entries so far:

```json
{
    "text": "“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”",
    "author": "Albert Einstein",
    "tags": [
        "change",
        "deep-thoughts",
        "thinking",
        "world"
    ]
},
```

And the S3 bucket should look like that after a few executions:

```sh
$ aws s3 ls quotes-scrapy
2021-07-31 10:03:33      24402 items-2021-07-31T07-03-23.json
2021-07-31 10:05:04      24402 items-2021-07-31T07-04-57.json
2021-07-31 11:04:27      24402 items-2021-07-31T08-04-18.json
2021-07-31 11:05:37      24402 items-2021-07-31T08-05-13.json
2021-07-31 12:07:22      24402 items-2021-07-31T09-07-12.json
```

Next step, let's create a message queue to get notified when new files are stored in the bucket.

## Step 2. Quotes SQS

Go to Amazon SQS [Create Queue](https://us-west-2.console.aws.amazon.com/sqs/v2/home#/create-queue). You can leave everything on default option. An important detail is the type of queue we want to create. Standard queue means that the message ordering is not guarranteed and the messages may be delivered more than once. Both are fine for our case, so we'll use standard type.

For Access Policy we need to specifically grant the S3 bucket access to write messages to the SQS so use the advanced tab and configure it with (as documented [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-notification-config-to-bucket.html#notification-walkthrough-summary):

```json
{
 "Version": "2012-10-17",
 "Id": "example-ID",
 "Statement": [
  {
   "Sid": "example-statement-ID",
   "Effect": "Allow",
   "Principal": {
    "Service": "s3.amazonaws.com"  
   },
   "Action": [
    "SQS:SendMessage"
   ],
   "Resource": "SQS-queue-ARN",
   "Condition": {
      "ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:awsexamplebucket1" },
      "StringEquals": { "aws:SourceAccount": "bucket-owner-account-id" }
   }
  }
 ]
}
```

and replace `SQS-queue-ARN`, `awsexamplebucket1` and `bucket-owner-account-id` with the actual values.

Once finished, go to the S3 bucket > Properties and Create an event notification with destination the freshly created SQS.

Now run the crawl again and the new uploaded file should create a message on our SQS.

## Step 3. Quotes ETL

Now for the last part we'll need to create a python app that reads each SQS message, downloads the related S3 bucket file, transforms it and uploads it to a separate output bucket.

The good part now is that since the messages are in the queue we can run this process whenever we want, e.g. once every day and consume thoses messages. Once the message is processed ok it is deleted.

Here is the ETL process:

```python
import json
import logging
from s3_client import s3_write
from transform_etl import transform_etl
from s3_client import s3_read
import pandas as pd

from sqs_client import sqs_delete_message, sqs_receive_message

logger = logging.getLogger(__name__)
QUEUE_URL = '***'
DESTINATION_BUCKET = "***"

def run():
    while True:
        parse_message()

def parse_message():
    # get next message from SQS
    logger.info("receiving next message")
    msgs = sqs_receive_message(QUEUE_URL)

    # exit if there are no more messages
    if (len(msgs)==0):
        logger.info("no messages found")
        exit(0)

    for msg in msgs:
        # parse msg body to read filename and bucket
        msg_body = json.loads(msg["Body"])
        for record in msg_body["Records"]:
            bucket = record["s3"]["bucket"]["name"]
            filename = record["s3"]["object"]["key"]

            # read file from origin S3
            logger.info("downloading file s3://%s/%s", bucket, filename)
            file_body = json.loads(s3_read(bucket, filename))

            # do some file transformation using
            transformed_body = transform_etl(file_body)

            # write file to destination S3
            logger.info("uploading file s3://%s/%s", DESTINATION_BUCKET, filename)
            s3_write(DESTINATION_BUCKET, filename, transformed_body)

        # remove message from queue
        sqs_delete_message(QUEUE_URL, msg['ReceiptHandle'])
        logger.info("message deleted")

def transform_etl(data: any) -> any:
    df = pd.DataFrame(data)
   
    # rename "text" column to "message"
    df.rename(columns = {"text": "message"}, inplace = True)
    # drop "author" column
    df.drop(columns =["author"], inplace = True)

    return df.to_json(orient='records')
```

For more details you can check the full source code [here](https://github.com/dkapanidis/etl-tutorial).

Now running the ETL I get the following whenever a new file is uploaded in the S3 bucket:

```sh
❯ python quotes_etl/
2021-08-01 13:46:46,447 - quotes_etl - INFO - receiving next message
2021-08-01 13:46:46,873 - quotes_etl - INFO - downloading file s3://quotes-scrapy/items-2021-08-01T10-46-36.json
2021-08-01 13:46:47,376 - quotes_etl - INFO - uploading file s3://quotes-scrapy-output/items-2021-08-01T10-46-36.json
2021-08-01 13:46:48,233 - quotes_etl - INFO - message deleted
2021-08-01 13:46:48,233 - quotes_etl - INFO - receiving next message
2021-08-01 13:46:48,438 - quotes_etl - INFO - no messages found
```

and let's check the output of the 2nd S3 bucket:

```sh
$ aws s3 ls quotes-scrapy-output
2021-07-31 20:40:02      18222 items-2021-07-31T17-39-49.json
2021-08-01 13:23:00      18222 items-2021-08-01T10-07-29.json
2021-08-01 13:37:09      18222 items-2021-08-01T10-36-19.json
2021-08-01 13:46:48      18222 items-2021-08-01T10-46-36.json
```

The entries on these files are shown below. Note that the `text` field is renamed to `message` and that the `author` is dropped in comparison with the original entry shown above:

```json
{
  "message": "“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”",
  "tags": [
    "change",
    "deep-thoughts",
    "thinking",
    "world"
  ]
},
```

## Summary

This was a simple ETL process, where we took some JSON entries, transformed them and stored them in a separate bucket. By using SQS message queue we decouple the upload of the original files with the transformation process and keep tabs of the pending files using the SQS queue.

In next posts I'll take it a step further using [Redshift](https://aws.amazon.com/redshift/) to store the data after the transformation instead of simple JSON files.


Building an ETL Workflow from scratch


I have talked recently about building docker images for [Golang] and [Python] projects, today we'll be focusing on building a Docker image for a React project.

[Golang]: /build-docker-image-for-golang-projects
[Python]: /build-docker-image-for-python-projects

## Requirements

Before continuing make sure to have [Node] and [Docker] installed on your machine:

```shellsession
$ node -v
v15.4.0

$ yarn -v
1.22.10

$ docker -v
Docker version 20.10.6, build 370c289
```

[Node]: https://nodejs.org/
[Docker]: https://www.docker.com/

## Simple React App

First, let's bootstrap our project using `create-react-app`, I'm using the typescript template as an example:

```shellsession
$ npx create-react-app react-sample --template typescript
```

The app can be started with:

```shellsession
$ cd react-sample
$ yarn start
```

React app is built and served on port `:3000` locally.

![react-starter](/posts/build-docker-image-for-react-projects/react-starter.png)


Let's create a minimal `Dockerfile`:

```dockerfile
# Dockerfile

FROM node:15.13.0-alpine
WORKDIR /app
COPY package.json yarn.lock ./
RUN yarn install
COPY . .
EXPOSE 3000
CMD ["yarn", "start"]
```

* `FROM`: We start with a node image based on alpine OS.
* `WORKDIR`: Set the current directory from this point to be /app
* `COPY`: Copy the package.json & yarn.lock file inside the container image.
* `RUN`: Install dependencies using `yarn install`.
* `COPY`: Copy source code.
* `EXPOSE`: Inform docker which port the server will be exposed to.
* `CMD`: define which command gets executed on startup.

Also let's add a `.dockerignore` file to avoid adding our local dependencies inside the container image:

```sh
#  .dockerignore

node_modules/
```

To build the docker image, start and test it:

```shellsession 
$ docker build -t react-sample .
[+] Building 0.8s (10/10) FINISHED
...

$ docker run -d -p3000:3000 react-sample
effa6ecd4d320677b1055cefa82fe32b0eddd136ad17e6e50730e95d91838a25

# wait for server to start
$ docker logs effa6ecd

# open on browser http://localhost:3000

$ docker stop effa6ecd
```

## Multi-Stage & Optimized Production Build

We started our react app on our previous example but you probably noticed that it takes some time on startup to load, this is because the compilation of our source code is done during container startup instead of the build stage. Also, we have started our react app in development mode, which includes stuff such as watching for file changes and hot-reload.

Instead we want a production build which in react can be done by running `yarn build` and placing the output in a static web server such as an `nginx`.

We can do that by updating our Dockerfile:

```dockerfile
# Dockerfile

# build environment
FROM node:15.13.0-alpine as build
WORKDIR /app
COPY package.json yarn.lock ./
RUN yarn install
COPY . .
RUN yarn build

# runtime environment
FROM nginx:stable-alpine
COPY --from=build /app/build /usr/share/nginx/html
COPY --from=build /app/nginx/ /etc/nginx/conf.d/
```

Now our build is done on our build image, and then we use a separate final image which is based on `nginx` and we include the output of the previous build and some default config under `conf.d/default.conf`:

```nginx
# /conf.d/default.conf

server {

  listen 80;

  location / {
    root   /usr/share/nginx/html;
    index  index.html index.htm;
    try_files $uri $uri/ /index.html;
  }

  error_page   500 502 503 504  /50x.html;

  location = /50x.html {
    root   /usr/share/nginx/html;
  }

}
```

As added benefit since we use multi-stage builds the source code is no longer added in the final image, just the minified production build.

## Conclusion

If you're interested in deploying React alongside other apps inside a Kubernetes cluster, this is hopefully a great starting point to prepare your Dockerfile for your React.


Progressively Build an Optimized Docker Image for React Projects


Using Python and want to run your projects in containers? Let's take a close look on how to build an optimized Dockerfile for Python projects.

Previously we built an optimized docker image for [Golang Projects], now we'll do the same process for Python.

[Golang Projects]: /build-docker-image-for-golang-projects

## Requirements

Before continuing make sure to have [Python 3] and [Docker] installed on your machine:

[Python 3]: https://www.python.org/
[Docker]: https://www.docker.com/

```shellsession
$ python3 --version
Python 3.7.7

$ docker -v
Docker version 20.10.6, build 370c289
```

## Simple Web Server

Let's start with a simple hello world server in Python:

Write the following in `app.py`

```python
# app.py

import http.server
import socketserver

PORT = 5000

class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        self.wfile.write("Hello world".encode('utf-8'))

with socketserver.TCPServer(("", PORT),  Handler) as httpd:
    print("serving at port", PORT)
    httpd.serve_forever()
```

The server can be started with:

```shellsession
$ python3 app.py
```

Python is an interpreted language, so there is no need for compilation step, instead it runs directly the code. The app will listen on port `:5000` locally and when it receives a GET request it will respond with `Hello world`. Leave the app running in a console and on a separate console use curl:

```shellssesion
$ curl localhost:5000
Hello world
```

Let's create a minimal `Dockerfile`:

```dockerfile
# Dockerfile

FROM python:3-alpine
WORKDIR /app
COPY app.py /app/
EXPOSE 5000
CMD [ "python3", "-u", "app.py"]
```

* `FROM`: We start with a python 3 base image based on alpine OS.
* `WORKDIR`: Set the current directory from this point to be `/app`
* `COPY`: Copy the `app.py` file inside the container image.
* `EXPOSE`: Inform docker which port the server will be exposed to.
* `CMD`: define which command gets executed on startup. `-u` param is used for unbuffered output.

To build the docker image, start the container and test it:

```shellsession
$ docker build -t python-sample .
[+] Building 1.5s (8/8) FINISHED
...

$ docker run -d -p5000:5000 python-sample
43bed4c8d55083a7cae901c0d5cbea95c4996010ff45ffea2b168c62f5fdc08f

$ curl localhost:5000
Hello world

$ docker stop 43bed4c8
```

## Multi-stage build

Multi-stage builds can separate build and runtime environments, but in interpreted languages such as Python I don't actually see much of a benefit. As an example of such multi-stage I leave below how to use alpine image for `build` and `distroless` for runtime.

```dockerfile
# Dockerfile

FROM python:3-alpine AS build
WORKDIR /app
COPY app.py /app/

FROM gcr.io/distroless/python3
COPY --from=build /app /app
WORKDIR /app
EXPOSE 5000
CMD ["app.py"]
```

Checking the output of both images:

| Base Image                | Base Image Size | Final Size |
| ------------------------- | --------------- | ---------- |
| python:3-alpine           | 44.9MB          | 44.9MB     |
| gcr.io/distroless/python3 | 52.2MB          | 52.2MB     |

Of course, the base image size and final size are identical since we only added our `app.py` file in between. Alpine image is smaller but distroless should be considered more secure by design.

## Web Server using Flask

Let's use now Flask to run a web server instead.

First we install Flask and write the `requirements.txt` file:

```shellsession
$ pip3 install Flask
Collecting Flask
  Downloading Flask-2.0.0-py3-none-any.whl (93 kB)
     |████████████████████████████████| 93 kB 1.6 MB/s
Collecting click>=7.1.2
  Downloading click-8.0.0-py3-none-any.whl (96 kB)
     |████████████████████████████████| 96 kB 4.3 MB/s
Collecting Jinja2>=3.0
  Downloading Jinja2-3.0.0-py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 8.6 MB/s
Collecting Werkzeug>=2.0
  Downloading Werkzeug-2.0.0-py3-none-any.whl (288 kB)
     |████████████████████████████████| 288 kB 8.5 MB/s
Collecting itsdangerous>=2.0
  Downloading itsdangerous-2.0.0-py3-none-any.whl (18 kB)
Collecting MarkupSafe>=2.0.0rc2
  Downloading MarkupSafe-2.0.0-cp37-cp37m-macosx_10_9_x86_64.whl (13 kB)
Installing collected packages: click, MarkupSafe, Jinja2, Werkzeug, itsdangerous, Flask
Successfully installed Flask-2.0.0 Jinja2-3.0.0 MarkupSafe-2.0.0 Werkzeug-2.0.0 click-8.0.0 itsdangerous-2.0.0

$ pip3 freeze | grep Flask > requirements.txt
```

Now let's update our `app.py` with a Flask version:

```python
# app.py

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello world'
```

To start the app do:

```shellsession
$ python3 -m flask run
```

To test it is working open on a browser `http://localhost:5000`.

Let's update our `Dockerfile` to be able to use dependencies:

```dockerfile
# Dockerfile

FROM python:3-alpine
WORKDIR /app
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
COPY . .
EXPOSE 5000
CMD [ "python3", "-m" , "flask", "run", "--host=0.0.0.0"]
```

This will copy the `requirements.txt` file that defines our dependencies and install all dependencies before we even add the source code. If the requirements file changes the dependencies layer will be invalidated and run again to fetch the new dependencies, but if it stays the same the cached layer will be reused saving us time during development which is nice.

Let's build the image again and run it to check it works ok:

```shellsession
$ docker build -t python-sample .
[+] Building 0.1s (10/10) FINISHED
...

$ docker run -d -p5000:5000 python-sample
4a0e0e8cbdefd9ebcbcec7f8477305cb027dd08daa03ac4a8ecadd004ba39f1d

$ curl localhost:5000
Hello world

$ docker stop 4a0e0e8cb
```

## Python Including Unit Tests

In order to get a quality gateway on our resulting images we can make sure no images are generated that fail the unit tests by executing them during the build process.

If you're interested in learning more about unit testing in Python there is a nice post [here](https://realpython.com/python-testing/) that describes nicely how to start testing your python app.

I will be using `nose2` test runner on the example so let's install it first:

```shellsession
$ pip3 install nose2
Collecting nose2
  Using cached nose2-0.10.0-py2.py3-none-any.whl (141 kB)
Requirement already satisfied: coverage>=4.4.1 in /usr/local/lib/python3.7/site-packages (from nose2) (5.5)
Requirement already satisfied: six>=1.7 in /usr/local/lib/python3.7/site-packages (from nose2) (1.12.0)
Installing collected packages: nose2
Successfully installed nose2-0.10.0

$ pip3 freeze | grep nose2 >> requirements.txt

$ cat requirements.txt
Flask==2.0.0
nose2==0.10.0
```

Ok, let's write two simple passing tests in `test_example.py`:

```python
# test_example.py

def test_sum():
    assert sum([1, 2, 3]) == 6, "Should be 6"

def test_sum_tuple():
    assert sum((1, 2, 3)) == 6, "Should be 6"
```

Test it locally:

```shellssesion
$ python -m nose2
..
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK
```

Let's add this now during our build process:

```dockerfile
# Dockerfile

FROM python:3-alpine
WORKDIR /app
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
COPY . .
RUN python -m nose2
EXPOSE 5000
CMD [ "python3", "-m" , "flask", "run", "--host=0.0.0.0"]
```

If we change the test to fail we can see the build process now stops:

```diff
-    assert sum((1, 2, 3)) == 6, "Should be 6"
+    assert sum((1, 2, 2)) == 6, "Should be 6"
```

```shellsession
$  docker build -t python-sample .
[+] Building 1.0s (10/10) FINISHED
 => [internal] load build definition from Dockerfile                                                             0.0s
 => => transferring dockerfile: 37B                                                                              0.0s
 => [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 2B                                                                                  0.0s
 => [internal] load metadata for docker.io/library/python:3-alpine                                               0.0s
 => [1/6] FROM docker.io/library/python:3-alpine                                                                 0.0s
 => [internal] load build context                                                                                0.0s
 => => transferring context: 884B                                                                                0.0s
 => CACHED [2/6] WORKDIR /app                                                                                    0.0s
 => CACHED [3/6] COPY requirements.txt requirements.txt                                                          0.0s
 => CACHED [4/6] RUN pip3 install -r requirements.txt                                                            0.0s
 => [5/6] COPY . .                                                                                               0.0s
 => ERROR [6/6] RUN python -m nose2                                                                              0.8s
------
 > [6/6] RUN python -m nose2:
#10 0.790 .F
#10 0.791 ======================================================================
#10 0.791 FAIL: test_example.test_sum_tuple
#10 0.791 ----------------------------------------------------------------------
#10 0.791 Traceback (most recent call last):
#10 0.791   File "/app/test_example.py", line 5, in test_sum_tuple
#10 0.791     assert sum((1, 2, 2)) == 6, "Should be 6"
#10 0.791 AssertionError: Should be 6
#10 0.791
#10 0.791 ----------------------------------------------------------------------
#10 0.791 Ran 2 tests in 0.000s
#10 0.791
#10 0.792 FAILED (failures=1)
------
executor failed running [/bin/sh -c python -m nose2]: exit code: 1
```

## Conclusion

In this tutorial you created a simple python web server using Flask, packaged it inside a container image, optimize build time by taking advantage of the cache for dependencies and added unit testing during the build process to avoid broken code to be delivered in production.

You are now ready for your next python 🐍&nbsp; project to be delivered 📦&nbsp; inside containers 🙌️


Progressively Build an Optimized Docker Image for Python Projects


Learning the basics of python and how it works in less than 10 mins. Python is a great language for web development, data science and scripting.

## Hello World

Let's just print a simple hello world message:

```python
# hello.py
print('hello world')
```

Type `python hello.py` to execute.

## Primitive Types

Python has four primitive types: int, float, bool, str.

```python
# primitives.py
ten = 10            # int
pi = 3.14159265359  # float
yes = True          # bool
no = False          # bool
name = "John Wick"  # str
print(ten, pi, yes, no, name, len(name))
```

```shellsession
$ python primitives.py
10 3.14159265359 True False John Wick 9
```

## Data Types

Apart from the primitive types python has the following data types:

- Sequence Types: list, tuple, range
- Mapping Types: dict
- Set Types: set, frozenset

```python
fruitsList = ["apple", "banana", "cherry"]
fruitsTuple = ("apple", "banana", "cherry")
fruitsSet = {"apple", "banana", "cherry"}
fruitsFrozenSet = frozenset({"apple", "banana", "cherry"})
x = range(6)
print ("list", fruitsList)
print ("tuple", fruitsTuple)
print ("set", fruitsSet)
print ("frozenSet", fruitsFrozenSet)
print ("range", x)
```

## Operations

```python
add = 3+2
print (add)

subtract = 3-2
print (subtract)

multiply = 3*2
print (multiply)

division = 3/2
print (division)

modulus = 3%2
print (modulus)

lessThan = 3<2
print (lessThan)

greaterThan = 3>2
print(greaterThan)

equals = 3==3
print (equals)

logicalAnd = (2==2) and (3==3) and (4==4)
print (logicalAnd)

logicalOr = (2==1) or (2==2) or (2==3)
print (logicalOr)

logicalNot = not (3==2)
print (logicalNot)
```

## Loops

```python
fruits = ["apple", "banana", "cherry"]
for x in fruits:
  print(x)

# 0,1,2,3,4,5 
for x in range(6):
  print(x)

# 5,6,7,8,9,10
for x in range(5,11):
  print(x)
```

## Decision Making

```python
x = int(input("Enter a number: "))
if x > 0:
    print(x)
else:
    print(-x)
```

## Functions

```python
def printme( str ):
   "This prints a passed string into this function"
   print(str)
   return

printme("hello")
```


Learn Python in 10 minutes


If you're getting started with a brand new golang project and want to package it inside a docker image, this is a good starting point for you.

I tend to switch programming languages often depending on the project I'm in and in order to help me with the quirks of each language I keep handy cheat-sheets, here is one I have for Golang that includes a quick walkthrough of the [golang basics in less than 10 mins](/learn-go-in-10-mins).

## Requirements

Before continuing make sure to have [Go] and [Docker] installed on your machine:

[Go]: https://golang.org/
[Docker]: https://www.docker.com/

```shellsession
$ go version
go version go1.15.5 darwin/amd64

$ docker -v
Docker version 20.10.6, build 370c289
```

## Simple Web Server

Let's start by writting a simple hello world web server in Go:

Write the following in `main.go`

```go
// main.go

package main

import (
	"fmt"
	"net/http"
)

func main() {
	http.HandleFunc("/", handler)
	http.ListenAndServe(":8080", nil)
}

func handler(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "Hello world")
}
```

The server can be started with:

```shellsession
$ go run main.go
```

which will compile the source code and run it. The app will listen on port `:8080` locally and when it receives a GET request it will respond with `Hello world`. Leave the app running in a console and on a separate console use curl:

```shellsession
$ curl localhost:8080
Hello world
```

Let's start with a minimal `Dockerfile`:

```dockerfile
# Dockerfile

FROM golang:1.14.2-alpine
WORKDIR /app
COPY main.go .
RUN go build -o ./out/app .
EXPOSE 8080
CMD ["/app/out/app"]
```

* `FROM`: We start with a golang base image based on alpine OS.
* `WORKDIR`: Set the current directory from this point to to be `/app`
* `COPY`: Copy the `main.go` file inside the container image.
* `RUN`: Build the binary from source code.
* `EXPOSE`: Inform docker which port the service will be exposed to.
* `CMD`: define which command gets executed on startup.

To build the docker image, start the container and test it:

```shellsession
$ docker build -t golang-sample .
[+] Building 3.6s (9/9) FINISHED
...

$ docker run -d -p8080:8080 golang-sample
7d087298199b917a1413740a0b92e55d2e37b5cf7e6c6a4763bbfd89af1d25dc

$ curl localhost:8080
Hello world

$ docker stop 7d087298
```

## Multi-stage build

Our example is working great, but along with our binary we also deliver our source code this way:

```shellsession
$ docker run -it --rm golang-sample cat /app/main.go
package main

import (
	"fmt"
	"net/http"
)

func main() {
	http.HandleFunc("/", handler)
	http.ListenAndServe(":8080", nil)
}

func handler(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "Hello world")
}
```

This is because our final image also includes all previous build layers and since we instructed to `COPY` the source code inside the container image it still resides in the final image.

In order to resolve this we'll split the build process in two images:

```dockerfile
# Dockerfile

# Build image
FROM golang:1.14.2-alpine AS build
WORKDIR /app
COPY main.go .
RUN go build -o ./out/app .

# Runtime image
FROM alpine:3.9
COPY --from=build /app/out/app /usr/local/bin/app
EXPOSE 8080
CMD ["/usr/local/bin/app"]
```

Let's do the build, run & test cycle again now:

```shellsession
$ docker build -t golang-sample .
[+] Building 3.6s (9/9) FINISHED
...

$ docker run -d -p8080:8080 golang-sample
f62c06302497d3a13ed036f98009dd054c506c1b1adcf0bdd4f92514a013f64a

$ curl localhost:8080
Hello world

$ docker stop f62c0630
```

It works and as added bonus the source code is not included now on the final image:

```shellsession
$ docker run -it --rm golang-sample ls /app
ls: /app: No such file or directory
```

## Web Server using Gin

Let's expand our example now to be able to include multiple files and module dependencies.

First of all we need to generate our own `go.mod`:

```shellsession
$ go mod init example.com/golang-sample
go: creating new go.mod: module example.com/golang-sample

$ cat go.mod
module example.com/golang-sample

go 1.15
```

Let's use [gin](https://github.com/gin-gonic/gin) as HTTP web framework:


```shellsession
$ go get -u github.com/gin-gonic/gin
```

Now let's update the `main.go` with the following:

```go
// main.go

package main

import "github.com/gin-gonic/gin"

func main() {
	r := gin.Default()
	r.GET("/", func(c *gin.Context) {
		c.String(200, "hello world")
	})
	r.Run("0.0.0.0:8080")
}
```

Follow the same steps as before to run it locally and test it, it performs the same but uses a module dependency.

In order to package it though, we need to add more files than simply the `main.go`, so let's update our Dockerfile:

```dockerfile
# Dockerfile

# Build image
FROM golang:1.14.2-alpine AS build
WORKDIR /app
COPY . .
RUN go mod download
RUN go build -o ./out/app .

# Runtime image
FROM alpine:3.9
COPY --from=build /app/out/app /usr/local/bin/app
EXPOSE 8080
CMD ["/usr/local/bin/app"]
```

This will copy all files from our app in the container image, download the module deps and then build our binary before sending it to the runtime image.

But this is a bit inefficient, since every time we change something on our code the build process will have to re-download all modules deps again and again. So let's rephrase that code a bit differently:

```dockerfile
# Dockerfile

# Build image
FROM golang:1.14.2-alpine AS build
WORKDIR /app
COPY go.mod go.sum .
RUN go mod download
COPY . .
RUN go build -o ./out/app .

# Runtime image
FROM alpine:3.9
COPY --from=build /app/out/app /usr/local/bin/app
EXPOSE 8080
CMD ["/usr/local/bin/app"]
```

That's a lot better now, we first copy `go.mod` and `go.sum` and download the deps, if the dependencies change these layers will be invalidated and rerun, otherwise they can be reused for consecutive builds and save some time.

## Web Server project layout

Our example until now is a simple `main.go` file. Let's test if we can use a proper [project layout](https://github.com/golang-standards/project-layout). Create file `pkg/apis/hello.go`:

```go
// pkg/apis/hello.go

package apis

import (
	"net/http"

	"github.com/gin-gonic/gin"
)

func Hello(c *gin.Context) {
	c.String(http.StatusOK, "hello world")
}
```

Now update `main.go` with:

```go
// main.go

package main

import (
	"example.com/golang-sample/pkg/apis"
	"github.com/gin-gonic/gin"
)

func main() {
	r := gin.Default()
	r.GET("/", apis.Hello)
	r.Run("0.0.0.0:8080")
}
```

Let's build the image again and run it to check it works ok:

```shellsession
$ docker build -t golang-sample .
[+] Building 30.5s (14/14) FINISHED
...

$ docker run -d -p8080:8080 golang-sample
c7fd4fe0823dc91bc567e18b413b7432acffdad40e510d91be5d2b6a731d69a3

$ curl localhost:8080
hello world

$ docker stop c7fd4fe0823d
c7fd4fe0823d
```

## Go Including Unit Tests

What about unit tests? We can add them during our build cycle to make sure we don't get broken tests in production.

Let's write a simple math function in `pkg/math/math.go`:

```go
// pkg/math/math.go

package math

// Add is our function that sums two integers
func Add(x, y int) (res int) {
	return x + y
}

// Subtract subtracts two integers
func Subtract(x, y int) (res int) {
	return x - y
}
```

and let's write some unit tests in `pkg/math/math_test.go`:

```go
// pkg/math/math_test.go

package math

import "testing"

func TestAdd(t *testing.T) {
	got := Add(4, 6)
	want := 10

	if got != want {
		t.Errorf("got %d, wanted %d", got, want)
	}
}

func TestSubtract(t *testing.T) {
	got := Subtract(6, 2)
	want := 4

	if got != want {
		t.Errorf("got %d, wanted %d", got, want)
	}
}
```

Test it locally:

```shellsession
$ go test ./pkg/math -v
=== RUN   TestAdd
--- PASS: TestAdd (0.00s)
=== RUN   TestSubtract
--- PASS: TestSubtract (0.00s)
PASS
ok  	example.com/golang-sample/pkg/math	0.876s
```

Let's add this now during our build process:

```dockerfile
# Dockerfile

# Build image
FROM golang:1.14.2-alpine AS build
WORKDIR /app
COPY go.mod go.sum .
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go test -v ./...
RUN go build -o ./out/app .

# Runtime image
FROM alpine:3.9
COPY --from=build /app/out/app /usr/local/bin/app
EXPOSE 8080
CMD ["/usr/local/bin/app"]
```

If we change the test to fail we can see the build process now stops:

```diff
// pkg/math/math_test.go

func TestSubtract(t *testing.T) {
	got := Subtract(6, 2)
-	want := 4
+	want := 42

	if got != want {
		t.Errorf("got %d, wanted %d", got, want)
	}
}
```

```shellsession
$ docker build -t golang-sample .
[+] Building 43.3s (12/14)
 => [internal] load build definition from Dockerfile                     0.1s
 => => transferring dockerfile: 37B                                      0.0s
 => [internal] load .dockerignore                                        0.1s
 => => transferring context: 2B                                          0.0s
 => [internal] load metadata for docker.io/library/alpine:3.9            0.6s
 => [internal] load metadata for docker.io/library/golang:1.14.2-alpine  0.0s
 => [stage-1 1/2] FROM docker.io/library/alpine:3.9@sha256:414e0518bb92  0.0s
 => [build 1/7] FROM docker.io/library/golang:1.14.2-alpine              0.0s
 => [internal] load build context                                        0.0s
 => => transferring context: 338B                                        0.0s
 => CACHED [build 2/7] WORKDIR /app                                      0.0s
 => CACHED [build 3/7] COPY go.mod go.sum .                              0.0s
 => CACHED [build 4/7] RUN go mod download                               0.0s
 => CACHED [build 5/7] COPY . .                                          0.0s
 => ERROR [build 6/7] RUN CGO_ENABLED=0 go test -v ./...                42.1s
------
 > [build 6/7] RUN CGO_ENABLED=0 go test -v ./...:
#12 41.79 ?     example.com/golang-sample       [no test files]
#12 41.79 ?     example.com/golang-sample/pkg/apis      [no test files]
#12 41.79 === RUN   TestAdd
#12 41.79 --- PASS: TestAdd (0.00s)
#12 41.79 === RUN   TestSubtract
#12 41.79     TestSubtract: math_test.go:19: got 4, wanted 42
#12 41.79 --- FAIL: TestSubtract (0.00s)
#12 41.79 FAIL
#12 41.79 FAIL	example.com/golang-sample/pkg/math	0.008s
#12 41.79 FAIL
------
```

The build process is now taking longer (in my machine it was 27 secs without tests and 45 secs with), but normally I offload the build process to a CI/CD server so it is build async and it can notify me if I missed some unit tests during my dev by failing the build.

Note that we only talk about unit tests here, since integration tests include other services and are better suited in the CI/CD workflow instead of embedding them inside the image build process.

## Reduce Image Size

Let's take a look at the final image size:

```shellsession
$ docker images golang-sample
REPOSITORY      TAG       IMAGE ID       CREATED        SIZE
golang-sample   latest    9a44e42117a8   23 hours ago   22.8MB
```

It's not bad, but we can still do better. We currently used `alpine` as base image, but go generates [mostly](https://oddcode.daveamit.com/2018/08/16/statically-compile-golang-binary/) static binaries so we can also use [distroless] static flavor or even [scratch]:

* [alpine] contains a full OS including a shell to exec into. Use this if you want to enter and debug the container.
* [distroless] static contains ca-certificates, /etc/password entry for a root user, /tmp directory, and timezone data. Use this if your app needs any of the above.
* [scratch] is an empty container image, nothing is included as base.

[alpine]: https://hub.docker.com/_/alpine
[distroless]: https://github.com/GoogleContainerTools/distroless
[scratch]: https://hub.docker.com/_/scratch

In order to make sure the final binaries are statically linked we need to add `CGO_ENABLED=0` to the build process:

```diff
-RUN go build -o ./out/app .
+RUN CGO_ENABLED=0 go build -o ./out/app .
```

| Runtime Image                     | Runtime Image Size | Final Size |
| --------------------------------- | ------------------ | ---------- |
| alpine:3.9                        | 5.61 MB            | 22.8 MB    |
| gcr.io/distroless/static-debian10 | 1.82 MB            | 19.1 MB    |
| scratch                           |    0 MB            | 17.3 MB    |

Now let's take a look at the Go binary itself. We can reduce it's own size by adding ldflags that strip debug info from the binaries, and we can also use `upx` to further compress the final binary.

| Parameters         | Compressed with UPX | Final Size |
| ------------------ | ------------------- | ---------- |
| None               | No                  | 15.0 MB    |
| `-ldflags="-s -w"` | No                  | 12.0 MB    |
| None               | Yes                 |  7.2 MB    |
| `-ldflags="-s -w"` | Yes                 |  4.1 MB    |

So let's put this to work in our Dockerfile

```dockerfile
# Dockerfile

# Build image
FROM golang:1.14.2-alpine AS build
RUN apk add upx
WORKDIR /app
COPY go.mod go.sum .
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go test -v ./...
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o ./out/app .
RUN upx ./out/app

# Runtime image
FROM scratch
COPY --from=build /app/out/app /usr/local/bin/app
EXPOSE 8080
CMD ["/usr/local/bin/app"]
```

Let's see the final result!

```shellsession
$ docker images golang-sample
REPOSITORY      TAG       IMAGE ID       CREATED         SIZE
golang-sample   latest    461827301dda   3 seconds ago   4.77MB
```

## Conclusion

In this tutorial you created a simple golang app and gradually packaged it inside a container image, following best practices to avoid injecting the source code in the final build, optimize build time by taking advantage of the cache for dependencies, added unit testing during the build process to avoid delivering broken code to production and optimized the final image size.

Now for your next Go project you know how to fine-tune your `Dockerfile` 🙌️.

Progressively Build an Optimized Docker Image for Golang Projects

The Seven Steps to build a Cloud Native CI/CD for GitHub repos using Tekton


![Banner](/posts/demystifying-stateful-apps-on-kubernetes/banner.png)

## Stateless apps

A stateless app is an application program that does not save client data generated in one session for use in the next session with that client.

As an example a REST API backend service that performs requests on a database is stateless, though the database itself is stateful.

![Stateless](/posts/demystifying-stateful-apps-on-kubernetes/stateless.gif "Stateless apps")

One of the main benefit of stateless apps is that their instances can be scaled on multiple replicas without issues. On the diagram above we can see that a deployment is scaled to two replicas, so it generates two pods. a service is connecting the active pods under a service IP that can be exposed internally or externally. If a pod dies, the deployment is responsible for generating a fresh identical replica of the pod, when that is active it will be attached to the service.

## Start with Stateless

If you’re starting out your Kubernetes cluster and either migrating pre-existing instances or starting your new pet project, don’t start out with stateful solutions!

As a rule-of-thumb, give yourself at least 3 months of running stateless solutions on Kubernetes and then take on stateful solutions. Of course 3 months is arbitrary and depends myriads of other variables, but getting hands-on experience with stateless apps will give you a foothold so that you can work your way up the ladder with stateful apps.

So, if for example you’re starting out and running a cluster on GKE don’t try to fit your database and backend to kubernetes, instead use a managed db and run your stateless backend to kubernetes first. Once you get a feeling, then you can evaluate migrating stateful apps such as a db to the cluster.

## Stateful apps

A stateful app is a program that saves client data from the activities of one session for use in the next session. The data that is saved is called the application’s state.

In order to get state we need a disk to store data. In kubernetes the disks are represented by [PersistentVolumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) (PV) and a claim of a disk with specific characteristics can be done with [PersistentVolumeClaims](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) (PVC). Normally during deploy of an app we create a PVC and a [StorageClass](https://kubernetes.io/docs/concepts/storage/storage-classes/) will generate the PV and attach it to our PVC.

![Stateful](/posts/demystifying-stateful-apps-on-kubernetes/stateful.png "Stateful app (1 instance)")

On the diagram above we can see a [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) that generates a Pod which mounts a disk provided by the PVC/PV pair. If you run a single instance database server, that would fit the above example. But in case the instance goes down we will loose service, in order to get a stateful app in high-availability (HA) mode we need more than one instance.

There are different patterns to accomplish HA on stateful apps, we’ll use as example how [etcd](https://etcd.io/) distributed key-value store works.

![Stateful2](/posts/demystifying-stateful-apps-on-kubernetes/stateful2.png "Stateful app (3 instances)")

When running etcd on HA, all instances form an etcd cluster. each instance has its own local disk (PVC/PV pair). The etcd cluster decides in unison a leader using [raft](https://raft.github.io/) protocol, if the leader goes down the absolute majority will vote a new leader, thus the odd number of cluster nodes, in our example 3 instances. Each instance can receive read/write requests. the read requests are responded by the local data, while the write requests are forwarded to the leader which in turn broadcast the changes to the rest.

## Deploying an etcd cluster

Now that we went over the theory let’s do an example by deploying an etcd cluster using StatefulSet inside Kubernetes.

### Create Kubernetes Cluster

In case you already have a kubernetes cluster you can skip this.

We’ll use [Google Cloud Platform](https://cloud.google.com/) to run our cluster as GKE. There is a free tier which you can use to run the example.

```sh
# create tutorial cluster
gcloud container clusters create tutorial --num-nodes 1

# get credentials of the cluster
gcloud container clusters get-credentials tutorial
```

Once the cluster is ready you should be able to test the connectivity:

```sh
❯ kubectl get nodes
NAME                                      STATUS   ROLES    AGE    VERSION
gke-tutorial-default-pool-ca31e247-bagb   Ready    <none>   2m2s   v1.16.15-gke.4901
```

### Create StorageClass

[Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine) (GKE) by default is providing pd-standard disks with the default storageclass:

```sh
$ kubectl get storageclass -o=custom-columns=NAME:.metadata.name,TYPE:.parameters.type
NAME       TYPE
standard   pd-standard
```

Let’s create the ssd-storageclass.yaml:

```yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ssd
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
```

And apply it to the cluster:

```sh
❯ kubectl create -f ssd-storageclass.yaml
storageclass.storage.k8s.io/ssd created
```

### Deploy etcd cluster

We can now deploy the etcd cluster, which will be a StatefulSet with 3 replicas and two Services, one for internal peer communication between the instances as headless service and another for accessing the cluster externally through the API.

```yaml
apiVersion: v1
kind: Service
metadata:
  name: etcd-client
spec:
  type: LoadBalancer
  ports:
    - name: etcd-client
      port: 2379
      protocol: TCP
      targetPort: 2379
  selector:
    app: etcd
---
apiVersion: v1
kind: Service
metadata:
  name: etcd
spec:
  clusterIP: None
  ports:
    - port: 2379
      name: client
    - port: 2380
      name: peer
  selector:
    app: etcd
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: etcd
  labels:
    app: etcd
spec:
  serviceName: etcd
  selector:
    matchLabels:
      app: etcd
  replicas: 3
  template:
    metadata:
      name: etcd
      labels:
        app: etcd
    spec:
      containers:
        - name: etcd
          image: quay.io/coreos/etcd:latest
          ports:
            - containerPort: 2379
              name: client
            - containerPort: 2380
              name: peer
          volumeMounts:
            - name: data
              mountPath: /var/run/etcd
          command:
            - /bin/sh
            - -c
            - |
              PEERS="etcd-0=http://etcd-0.etcd:2380,etcd-1=http://etcd-1.etcd:2380,etcd-2=http://etcd-2.etcd:2380"
              exec etcd --name ${HOSTNAME} \
                --listen-peer-urls http://0.0.0.0:2380 \
                --listen-client-urls http://0.0.0.0:2379 \
                --advertise-client-urls http://${HOSTNAME}.etcd:2379 \
                --initial-advertise-peer-urls http://${HOSTNAME}:2380 \
                --initial-cluster-token etcd-cluster-1 \
                --initial-cluster ${PEERS} \
                --initial-cluster-state new \
                --data-dir /var/run/etcd/default.etcd
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        storageClassName: ssd
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi
```

Let’s apply it on the cluster:

```sh
❯ kubectl create -f etcd-statefulset.yaml
service/etcd-client created
service/etcd created
statefulset.apps/etcd created
```

You should now have an etcd cluster running:

```sh
❯ kubectl get pod
NAME     READY   STATUS    RESTARTS   AGE
etcd-0   1/1     Running   0          49s
etcd-1   1/1     Running   0          38s
etcd-2   1/1     Running   0          18s
```

To verify that the cluster is connected properly run:

```sh
❯ kubectl exec -it etcd-0 -- etcdctl cluster-health
member 2e80f96756a54ca9 is healthy: got healthy result from http://etcd-0.etcd:2379
member 7fd61f3f79d97779 is healthy: got healthy result from http://etcd-1.etcd:2379
member b429c86e3cd4e077 is healthy: got healthy result from http://etcd-2.etcd:2379
cluster is healthy
```

### Interacting with etcd

Now that etcd cluster is setup we can use it either with REST API or with the etcdctl CLI. First let’s check the external connectivity with REST API:

```sh
❯ kubectl get svc
NAME          TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)             AGE
etcd          ClusterIP      None         <none>          2379/TCP,2380/TCP   11m
etcd-client   LoadBalancer   10.62.3.17   34.76.137.212   2379:30917/TCP      11m
```

Use the external IP of etcd-client to send a request to store msg=hello-world.

```sh
# set msg
❯ curl http://34.76.137.212:2379/v2/keys/msg -XPUT -d value=hello-world
{"action":"set","node":{"key":"/msg","value":"hello-
world","modifiedIndex":21,"createdIndex":21}}
# get msg
❯ curl http://34.76.137.212:2379/v2/keys/msg
{"action":"get","node":{"key":"/msg","value":"hello-world","modifiedIndex":21,"createdIndex":21}}
```

Since the requests are passing through the service etcd-client they are balanced between all instances.

If we want to test each of the instances separately we can do the same request from inside the cluster with the internal service:

```sh
# get msg from inside the cluster with REST API (etcd-0.etcd)
❯ kubectl exec -it etcd-0 -- wget -qO- http://etcd-0.etcd:2379/v2/keys/msg
{"action":"get","node":{"key":"/msg","value":"hello-world","modifiedIndex":21,"createdIndex":21}}

# get msg from inside the cluster with REST API (etcd-1.etcd)
❯ kubectl exec -it etcd-0 -- wget -qO- http://etcd-1.etcd:2379/v2/keys/msg
{"action":"get","node":{"key":"/msg","value":"hello-world","modifiedIndex":21,"createdIndex":21}}

# get msg from inside the cluster with REST API (etcd-2.etcd)
❯ kubectl exec -it etcd-0 -- wget -qO- http://etcd-2.etcd:2379/v2/keys/msg
{"action":"get","node":{"key":"/msg","value":"hello-world","modifiedIndex":21,"createdIndex":21}}
```

This works because of the [headless service](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services) etcd. The DNS entries `etcd-0.etcd, etcd-1.etcd, etcd-2.etcd` resolve to the respective pod IPs. This is useful so that discovery between peers can happen easily.

Lastly, we can use etcdctl to check the value on each instance locally:

```sh
❯ kubectl exec -it etcd-0 -- etcdctl get msg
hello-world

❯ kubectl exec -it etcd-1 -- etcdctl get msg
hello-world

❯ kubectl exec -it etcd-2 -- etcdctl get msg
hello-world
```

### Testing High Availability

To test that the etcd cluster works on high availability, let’s kill the leader of the cluster and see how the re-election works.

First let’s find out who is the leader:

```sh
❯ kubectl exec -it etcd-0 -- etcdctl member list
2e80f96756a54ca9: name=etcd-0 peerURLs=http://etcd-0.etcd:2380 clientURLs=http://etcd-0.etcd:2379 isLeader=true
7fd61f3f79d97779: name=etcd-1 peerURLs=http://etcd-1.etcd:2380 clientURLs=http://etcd-1.etcd:2379 isLeader=false
b429c86e3cd4e077: name=etcd-2 peerURLs=http://etcd-2.etcd:2380 clientURLs=http://etcd-2.etcd:2379 isLeader=false
```

Now let’s kill the leader and do some tests:

```sh
# kill the leader
❯ kubectl delete pod etcd-0
pod "etcd-0" deleted

# check cluster health
❯ kubectl exec -it etcd-1 -- etcdctl cluster-health
failed to check the health of member 2e80f96756a54ca9 on http://etcd-0.etcd:2379: Get http://etcd-0.etcd:2379/health: dial tcp 10.64.2.64:2379: i/o timeout
member 2e80f96756a54ca9 is unreachable: [http://etcd-0.etcd:2379] are all unreachable
member 7fd61f3f79d97779 is healthy: got healthy result from http://etcd-1.etcd:2379
member b429c86e3cd4e077 is healthy: got healthy result from http://etcd-2.etcd:2379
cluster is degraded
command terminated with exit code 5

# check service is still working (read)
❯ curl http://34.76.137.212:2379/v2/keys/msg
{"action":"get","node":{"key":"/msg","value":"hello-world","modifiedIndex":21,"createdIndex":21}}

# check service is still working (write)
❯ curl http://34.76.137.212:2379/v2/keys/msg -XPUT -d value=hello-again
{"action":"set","node":{"key":"/msg","value":"hello-again","modifiedIndex":33,"createdIndex":33},"prevNode":{"key":"/msg","value":"hello-world","modifiedIndex":31,"createdIndex":31}}

# check cluster is healthy after a while
❯ kubectl exec -it etcd-1 -- etcdctl cluster-health
member 2e80f96756a54ca9 is healthy: got healthy result from http://etcd-0.etcd:2379
member 7fd61f3f79d97779 is healthy: got healthy result from http://etcd-1.etcd:2379
member b429c86e3cd4e077 is healthy: got healthy result from http://etcd-2.etcd:2379
cluster is healthy

# check the leader has changed
❯ kubectl exec -it etcd-1 -- etcdctl member list
2e80f96756a54ca9: name=etcd-0 peerURLs=http://etcd-0.etcd:2380 clientURLs=http://etcd-0.etcd:2379 isLeader=false
7fd61f3f79d97779: name=etcd-1 peerURLs=http://etcd-1.etcd:2380 clientURLs=http://etcd-1.etcd:2379 isLeader=false
b429c86e3cd4e077: name=etcd-2 peerURLs=http://etcd-2.etcd:2380 clientURLs=http://etcd-2.etcd:2379 isLeader=true
```

### Conclusion

Making a Stateful app to work in Kubernetes is not difficult, but making it work so that it is resilient and in high-availability mode depends on the architecture of the app itself, etcd is a battle-tested example in Kubernetes, but other solutions such as relational databases are a bit more complicated.

Hopefully this post has helped in providing some perspective on the different aspects that are important on setting up stateful apps.


Demystifying stateful apps on Kubernetes by deploying an etcd cluster


![Banner](/posts/awesome-cli-tools-2020/banner.png)

I like to keep my terminal lean and productive, here is a list of some awesome tools to help you with your daily work

* Brew
* Alfred
* iTerm2
* Zsh
* Powerlevel10k
* Meslo Nerd Font
* z
* Kubectl
* Kubectx
* Helm
* Stern
* Kubernetic

## 1. Brew package manager (Mac)

I use [Brew](https://brew.sh/) for installing everything on my Mac. To install brew itself simply run:

```shell
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
```

## 2. Alfred Productivity App (Mac)

Alfred is an application launcher for Mac which is not directly terminal related but I use it to start my terminal, so worth the mention.

![Alfred](/posts/awesome-cli-tools-2020/alfred.png)

Keyboard shortcut just the one and only:

* `Cmd+Space`: Open Alfred app


* Installation: `brew install alfred`
* Alternatives: [QuickSilver](https://qsapp.com/), [Launchbar](https://www.obdev.at/products/launchbar/), Spotilight.

## 3. iTerm2 (Mac)

As a terminal emulator I prefer to use iTerm2, which integrates great with other tools in this list. Some of my favourite features are spawning multiple terminals to work with.

![Iterm2](/posts/awesome-cli-tools-2020/iterm2.png)

Keyboard shortcuts I use daily:

* `Cmd+Enter`: Fullscreen mode.
* `Cmd+D`: Split horizontally.
* `Cmd+[` or `Cmd+]`: Next or previous terminal.
* `Cmd+t`: New tab terminal.
* `Cmd+Alt+i`: Broadcast input to all panes in current tab.

The last one I'm not using daily, but it super helpful in case you want to perform a repetitive work in multiple folders | servers | containers | clusters.

* Installation: `brew install iterm2`
* Alternatives: [Alacritty](https://github.com/alacritty/alacritty) (Mac, Linux, Win), [Kitty](https://sw.kovidgoyal.net/kitty/) (Mac, Linux)

## 4. Zsh Shell

Choosing a shell is something of a personal preference, I chose some time ago zsh and have sticked to it since then.

* Installation: `brew install zsh`
* Alternatives: [Fish](https://fishshell.com/), [Bash](https://www.gnu.org/software/bash/)

## 5. Powerlevel10k theme for Zsh

Powerlevel10k (or p10k) is a zsh theme that greatly improves the UX of the CLI itself. Once installed do `p10k configure` and it will interactively prompt you to customize it as you want.

![P10k](/posts/awesome-cli-tools-2020/p10k.png)

Some of my favourite features here are:

* **Instant Prompt**: It provides you with an interactive prompt before finish loading the shell configuration itself, this makes it super fast to startup.
* **Show On Command**: It displays command related info (e.g. the kubernetes context & namespace). I have as alias `k` for `kubectl` so it appears the moment I strike `k` on my keyboard:

![ShowOnCommand](/posts/awesome-cli-tools-2020/show-on-command.gif "show on command the kubernetes context and namespace")

* Installation: `brew install p10k; p10k configure`
* Alternatives: [oh-my-zsh](https://github.com/ohmyzsh/ohmyzsh)

## 6. Meslo Nerd Font

P10k prompts if you want automatic installation of `Meslo Nerd Font` during `p10k configure` process. As a console font is good enough for me, so I'm sticking with that one.

* Installation: `p10k configure` 
* Alternatives: [Fira Code](https://github.com/tonsky/FiraCode), [Incosolata](https://fonts.google.com/specimen/Inconsolata), [Victor Mono](https://rubjo.github.io/victor-mono/)
 
## 7. z

Z is one of those small tools that really make a difference! Its description is "jump around" and is used as a direct replacement of `cd` to switch directories.

Instead of jumping by using absolute or relative paths between directories, it tracks your most frequent paths and jumps to the best match. So instead of doing for example `cd ~/src/private/harbur/docs/playbook` I simply do `z play` and it picks it up.

![Z](/posts/awesome-cli-tools-2020/z.gif "z - jump around")

* Installation: `brew install z`

## 8. Kubectl

Cloud native is almost synonym these days with kubernetes, so kubectl on the CLI is a must. There are different ways to install the binary, for the sake of uniformity we'll install it with brew.

* Installation: `brew install kubernetes-cli`

## 9. Minikube

Either if you're starting out or you're an experienced user of kubernetes one of the best ways to start a Kubernetes cluster locally is using [minikube](https://github.com/kubernetes/minikube).

![Minikube](/posts/awesome-cli-tools-2020/minikube.gif "starting kubernetes cluster with minikube locally")

* Installation: `brew install minikube`

## 9. kubectx

If you work with many clusters or many namespaces it s(h)aves a loooot of time to avoid passing those as parameters to each command, instead you can switch your active context or namespace to be the one you want to use.

For that you can use [kubectx](https://github.com/ahmetb/kubectx). It installs both `kubectx` and `kubens` for switching context or namespace respectively, combined with the "Show on command" feature of p10k at the right side of the screen it really shines:

![Kubectx](/posts/awesome-cli-tools-2020/kubectx.gif "switching kubernetes context fast with kubectx")

![Kubens](/posts/awesome-cli-tools-2020/kubens.gif "switching kubernetes namespace fast with kubens")

* Installation: `brew install kubectx`

## 10. Helm

[Helm](https://helm.sh/) is a package manager for kubernetes, it helps you organize multiple resources and install them together as a unified app, it also manages the lifecycle of the releases providing the ability to do upgrades and rollbacks.

![Helm](/posts/awesome-cli-tools-2020/helm.gif "listing helm releases on cert-manager namespace")

Make sure to use v3, as helm v2 has reached its end-of-life.

* Installation: `brew install helm`

## 11. Stern

[Stern](https://github.com/wercker/stern) is a multi pod and container log tailing for Kubernetes, so for example if you want to get the logs of a deployment with two pods you simply do `stern nginx` and it will tail the logs of both pods on screen:

![Stern](/posts/awesome-cli-tools-2020/stern.gif "using stern to tail logs of multiple pods")

* Installation `brew install stern`
* Alternatives: [kubetail](https://github.com/johanhaleby/kubetail), [kail](https://github.com/boz/kail)

## 12. Kubernetic Desktop client

[Kubernetic](http://kubernetic.com/) is a desktop client for kubernetes focused on productivity. While CLI is great for productivity and automation when needed, Kubernetic is best for feature discoverability, provides a great learning platform for newcomers, and is actually faster than working with the CLI for experienced users.

With Kubernetic you can manage multiple contexts & namespaces, see resources usage consumption, do port-forwarding of services, execute commands in containers, craft resources without extensive YAML knowledge and much much more.

![Kubernetic](/posts/awesome-cli-tools-2020/kubernetic.gif "create an nginx deployment with 2 replicas and do port-forward with Kubernetic")

* Installation: `brew install kubernetic`
* Disclaimer: Kubernetic is a product of Harbur Cloud Solutions, the consulting company where I currently work.


12 Awesome CLI tools to supercharge your Cloud Native terminal

Refactoring landing page with React, NextJS & TailwindCSS


Learning the basics of Go and how it works. I chose Go as a starting language here because of its ease of use and performance.

## Why Go

Go is a language that was designed at 2009 at Google by Robert Griesemer, Rob Pike, and Ken Thompson. It was created in order to address performance issues of other languages and to take advantage of the multi-core CPUs by providing concurrency out-of-the-box.

It is used to code backend components that are small, efficient with concurrency in mind. Almost all components in the Cloud-native ecosystem are written in Go (Docker, Kubernetes, Istio, Helm, Prometheus, etcd, Knative to name a few) which makes it a perfect fit for integrating with those components.

## Code Style Formatting

Go language has built-in code style [formatting](https://blog.golang.org/go-fmt-your-code).

VSCode has built-in integration of Go which supports formatting on-save.

## Quick Language Overview

For quick language overview check this [Go Programming Language Tutorial](https://www.youtube.com/watch?v=Q0sKAMal4WQ) 1h video that explains all basic concepts of Go.

Below are some of those concepts:

### Hello World

```go
package main

import "fmt"

func main() {
    fmt.Println("Hello World!")
}
```

Type `go run hello.go` to execute.


### Variables & Constants

```go
package main

import "fmt"

func main() {
    // A constant is a variable with a value that can't be changed
    const pi float64 = 3.14159265359
    // You can declare multiple variables like this
    var (
        varA = 2
        varB = 3
    )
    fmt.Println(varA, varB)

    // Strings are a series of characters between " or `
    var Name string = "John Wick"

    // Get string length
    fmt.Println(len(Name))
}
```

### Data Types

* Numeric
* int (uint8, uint16, uint32, uint64, int8, int16, int32, int64)
* float
* String
* Boolean
* true
* false
* Derived
* Pointer
* Array
* Structure
* Map
* Interface

```go
package main

import "fmt"

func main() {
    // Variable assignment with specific type
    var a int = 5
    var b float32 = 4.32
    const pi float64 = 3.1415139475

    // Variable assignment with auto type
    c := 1

    // Variable reassignment (should be same type as first assignment)
    c = 5

    // Multiple variables assignment
    var (
        d="hello"
        e="there"
    )
    f,g:=14,15

    fmt.Println(a)
    fmt.Println(b)
    fmt.Println(pi)
    fmt.Println(c)
    fmt.Println(d)
    fmt.Println(e)
    fmt.Println(f)
    fmt.Println(g)
}
```

### Operators

* Arithmetic
* addition (+)
* subtraction (-)
* multiplication (*)
* division (/)
* modulus (%)
* Relational
* greater than (>)
* lesser than (<)
* greater than or equal (>=)
* equivalence (==)
* not equals (!=)
* Logical
* and (&&)
* or (||)
* negation (!)

```go
package main

import "fmt"

func main() {
    // Arithmetic
    x,y:=5,6
    fmt.Println("x + y =", x + y)
    fmt.Println("x - y =", x - y)
    fmt.Println("x * y =", x * y)
    fmt.Println("x / y =", x / y)
    fmt.Println("x % y =", x % y)

    isBool := true
    hate := false

    fmt.Println(isBool && hate)
    fmt.Println(isBool || hate)
    fmt.Println(!isBool)
}
```

### Pointers

```go
package main

import "fmt"

func main() {
	x := 5

	fmt.Println(x)
	fmt.Println(&x)
}
```

Pass by value:

```go
package main

import "fmt"

func main() {
	x := 10

    changeValue(x) // pass by value
    fmt.Println(x) // Prints 10
}

func changeValue(x int) {
    x=7
}
```

Pass by reference:

```go
package main

import "fmt"

func main() {
	x := 10

	changeValue(&x) // pass by reference
	fmt.Println(x) // prints 7
}

func changeValue(x *int) {
	*x = 7
}
```

Printf

```go
package main

import "fmt"

func main() {
	var name string = "John Wick"
	const pi float64 = 3.16412732
	x := 5
	isbool := true

	fmt.Printf("%f \n", pi)     // print float: 3.16412732
	fmt.Printf("%.3f \n", pi)   // print float: 3.164
	fmt.Printf("%T \n", name)   // print type: string
	fmt.Printf("%t \n", isbool) // print boolean: true
	fmt.Printf("%d \n", x)      // print int: 5
	fmt.Printf("%b \n", 25)     // print binary: 11001
	fmt.Printf("%c \n", 33)     // print character: !
	fmt.Printf("%x \n", 15)     // print hexcode: f
}
```

### Loops

normal for loop:

```go
package main

import "fmt"

func main() {
	for i := 1; i < 10; i++ {
		fmt.Println(i)
    }
}
```

while style loop:

```go
package main

import "fmt"

func main() {
	i := 1
	for i < 10 {
		fmt.Println(i)
		i++
	}
}
```

nested loops:

```go
package main

import "fmt"

func main() {
	for i := 1; i < 10; i++ {
        for j := 1; j < i; j++ {
            fmt.Printf("*")
        }
        fmt.Println()
	}
}
```


### Decision Making

```go
package main

import "fmt"

func main() {
	age := 18

	if age >= 18 {
		fmt.Println("you can van vote")
	} else {
		fmt.Println("you cannot vote")
	}
}
```

```go
package main

import "fmt"

func main() {
	age := 18

	switch age {
	case 16:
		fmt.Println("Prepare for college")
	case 18:
		fmt.Println("Don't run after girls")
	case 20:
		fmt.Println("Get yourself a job!")
	default:
		fmt.Println("Are you even alive?")
	}
}
```

### Arrays

```go
package main

import "fmt"

func main() {
	var EvenNum [5]int
	EvenNum[0] = 0
	EvenNum[1] = 2
	EvenNum[2] = 4
	EvenNum[3] = 6
	EvenNum[4] = 8

	fmt.Println(EvenNum[2]) // prints: 4
}
```

```go
package main

import "fmt"

func main() {
	EvenNum := [5]int{0, 2, 4, 6, 8}
	fmt.Println(EvenNum[2]) // prints: 4
}
```

```go
package main

import "fmt"

func main() {
	EvenNum := [5]int{0, 2, 4, 6, 8}
	for _, value := range EvenNum {
		fmt.Println(value)
	}
}
```

slices:

```go
package main

import "fmt"

func main() {
	EvenNum := [5]int{0, 2, 4, 6, 8}
	for i, value := range EvenNum {
		fmt.Println(value, i)
	}

	numSlice := []int{5, 4, 3, 2, 1}
	sliced := numSlice[3:5]
	fmt.Println(sliced)        // prints: [2, 1]
	fmt.Println(numSlice[3:])  // prints: [2, 1]
	fmt.Println(numSlice[:])   // prints: [5, 4, 3, 2, 1]
	fmt.Println(numSlice[0:1]) // prints: [5]

	slice2 := make([]int, 5, 10)
	copy(slice2, numSlice) // copies numSlice to slice2
	fmt.Println(slice2)    // prints [5, 4, 3, 2, 1]

	slice3 := append(numSlice, 3, 0, -1)
	fmt.Println(slice3) // prints [5, 4, 3, 2, 1, 3, 0, -1]
}
```

### Maps

```go
package main

import "fmt"

func main() {
	StudentAge := make(map[string]int)

	StudentAge["John"] = 23
	StudentAge["Bob"] = 27
	StudentAge["Anna"] = 21
	StudentAge["Alice"] = 19
	StudentAge["Carl"] = 42
	StudentAge["Maria"] = 22

	fmt.Println(StudentAge["maria"]) // not found (case-sensitive) returns: 0
	fmt.Println(StudentAge["Maria"]) // found returns: 22
	fmt.Println(len(StudentAge))     // returns 6 (number of elements)
}
```

map in a map:

```go
package main

import "fmt"

func main() {
	superhero := map[string]map[string]string{
		"Superman": map[string]string{
			"RealName": "Clar Kent",
			"City":     "Metropolis",
		},
		"Batman": map[string]string{
			"RealName": "Bruce Wayne",
			"City":     "Gotham City",
		},
	}

	if temp, hero := superhero["Superman"]; hero {
		fmt.Println(temp["RealName"], temp["City"], hero)
	}
}
```

### Functions

```go
package main

import "fmt"

func main() {
	x, y := 5, 6
	fmt.Println(add(x, y)) // prints: 11
}

func add(num1, num2 int) int {
	return num1 + num2
}
```

recursion:

```go
package main

import "fmt"

func main() {
	x := 5
	fmt.Println(factorial(x)) // ( factorial(5) = 5x4x3x2x1 ) prints: 120
}

func factorial(num int) int {
	if num == 0 {
		return 1
	}
	return num * factorial(num-1)
}
```

### Defer, Recover, Panic

Control Structure:

* Defer: Defer statement defers the execution of a function until the surrounding function returns. (e.g. cleanup resources)
* Recover: Helps regain ther normal flow of execution after a panic.
* Panic: Panic is similar to throwing exception like other language.

defer:

```go
package main

import "fmt"

func main() {
	defer firstRun() // It will execute last, after secondRun() function
	secondRun()
}

func firstRun() {
	fmt.Println("I executed first")
}

func secondRun() {
	fmt.Println("I executed second")
}
```

panic & recover:

```go
package main

import "fmt"

func main() {
	fmt.Println(div(3, 0))
	fmt.Println(div(5, 3))
	demPanic()
}

func div(num1, num2 int) int {
	defer func() {
		fmt.Println("div Recovered: ", recover())
	}()
	solution := num1 / num2 // (This will panic if num2 is 0)
	return solution
}

func demPanic() {
	defer func() {
		fmt.Println("demPanic Recovered:", recover())
	}()
	panic("Paniced")
}
```

### Structures & Interfaces

structures:

```go
package main

import "fmt"

func main() {
	rect1 := Rectangle{height: 10, width: 5}
	rect2 := Rectangle{10, 5}

	fmt.Println(rect1.height) // returns: 10
	fmt.Println(rect1.width)  // returns: 5
	fmt.Println(rect2)        // returns {10, 5}
	fmt.Println(rect1.area()) // returns: 50
}

// Rectangle a rectangle with dimensions
type Rectangle struct {
	height float64
	width  float64
}

func (rect *Rectangle) area() float64 {
	return rect.height * rect.width
}
```

interfaces:

```go
package main

import (
	"fmt"
	"math"
)

func main() {
	rect1 := Rectangle{50, 60}
	circ := Circle{7}

	fmt.Println(getArea(rect1)) // returns: 3000
	fmt.Println(getArea(circ))  // returns: 52.1415926535898
}

// Shape interface of a shape
type Shape interface {
	area() float64
}

// Rectangle a rectangle with dimensions
type Rectangle struct {
	height float64
	width  float64
}

// Circle a circle
type Circle struct {
	radius float64
}

func (r Rectangle) area() float64 {
	return r.height * r.width
}

func (c Circle) area() float64 {
	return math.Pi + math.Pow(c.radius, 2)
}

func getArea(shape Shape) float64 {
	return shape.area()
}
```

### File I/O

```go
package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"os"
)

func main() {
	file, err := os.Create("sample.txt")
	if err != nil {
		log.Fatal(err)
	}

	file.WriteString("Hi my name is John Wick and this file was created using GO!!")
	file.Close()

	stream, err := ioutil.ReadFile("sample.txt")
	if err != nil {
		log.Fatal(err)
	}
	s1 := string(stream)
	fmt.Println(s1)
}
```

### Webserver

```go
package main

import (
	"fmt"
	"net/http"
)

func main() {
	http.HandleFunc("/", handler)
	http.HandleFunc("/hello", handler2)
	http.ListenAndServe(":8080", nil)
}

func handler(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "welcome to my home page")
}

func handler2(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "Hello world")
}
```

Posts about Terminal

Tips & tricks on how to tune the terminal to boost your productivity.

12 Awesome CLI tools to supercharge your Cloud Native terminal

Posts by Topics 👨‍💻

Go

React

Terminal

Coding

Kubernetes

Videos 📹

React in Docker & Kubernetes walkthrough: Multistage build, push & deploy in Kubernetes

Building Slack clone using React, TailwindCSS & Supabase

Live coding a Slack clone from scratch using React, TailwindCSS & Firebase

Subscribe to the Newsletter!