I'm going to run all those in my local env. So change the hostnames or FQDN for your domain.

Create KeyCloak Client

Properties of Keycloak client must be like following;

• Client Type: OpenID Connect
  Client ID: grafana-oauth
• Authentication: On
  Authentication Flow: Standart Flow Enabled
  Direct access grants Enabled
• Root URL: Grafana's root url
  Home URL: Grafana's root url
  Valid Redirect URIs: Grafana's root url + /login/generic_oauth
  Web Origins: Grafana's root url

After creating the client, go to client details page and click Credentials tab to obtain client secret.

Grafana Compose File and OAuth Configuration

What to replace the file in below;

• Your realm name and Client secret
• Your keycloak hostname. (There are two domains. Localhost is sent to browser. Host.docker.internal is for Grafana access to Keycloak.

version: '3.8'

services:
  grafana:
    image: docker.io/grafana/grafana-oss:12.0.2
    container_name: grafana
    restart: unless-stopped
    environment:
    - GF_SERVER_ROOT_URL=http://localhost:3000/
    - GF_PLUGINS_PREINSTALL=grafana-clock-panel
    - GF_SECURITY_ADMIN_USER=admin
    - GF_SECURITY_ADMIN_PASSWORD=admin
    - GF_AUTH_GENERIC_OAUTH_ENABLED=true
    - GF_AUTH_GENERIC_OAUTH_NAME=Keycloak-OAuth
    - GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
    - GF_AUTH_GENERIC_OAUTH_CLIENT_ID=grafana-oauth
    - GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=[SECRET]
    - GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile offline_access roles
    - GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH=email
    - GF_AUTH_GENERIC_OAUTH_LOGIN_ATTRIBUTE_PATH=username
    - GF_AUTH_GENERIC_OAUTH_NAME_ATTRIBUTE_PATH=full_name
    - GF_AUTH_GENERIC_OAUTH_AUTH_URL=http://localhost:8080/realms/local/protocol/openid-connect/auth
    - GF_AUTH_GENERIC_OAUTH_TOKEN_URL=http://host.docker.internal:8080/realms/local/protocol/openid-connect/token
    - GF_AUTH_GENERIC_OAUTH_API_URL=http://host.docker.internal:8080/realms/local/protocol/openid-connect/userinfo
    - GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=contains(roles[*], 'admin') && 'Admin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer'

    ports:
     - '3000:3000'
    volumes:
     - 'grafana_storage:/var/lib/grafana'
volumes:
  grafana_storage: 

You would see you can log in to Grafana via your Keycloak user. One important note here. In the localhost; you can't map Keycloak roles to Grafana users. OAUTH_ROLE_ATTRIBUTE_PATH defines mapping. So normally you should assign that role to user. It's happening due to localhost and host.docker.internal domains' differences. The JWT token couldn't be verified by Grafana so it doesn't fetch it. But it works in production. And make sure, the path matches with the key in your JWT token. Adjust it by the token. Or modify client scope details in Keycloak.

Install requirements in below

Requirements
- uvicorn, fastapi, pydantic, pydantic-settings, python-keycloak, pyjwt, python-multipart

KeyCloak Realm and Client Configuration

• First create a realm in KeyCloak named local and select it.
• Under clients , create a new client with below specs
  Client type: OpenID Connect
  Client ID: local-api
• Click next and enable below capabilities
  Client Authentication: On
  Direct Access Grants: On
  Standart Flow: On
• Click next and configure login settings
  Root URL: http://localhost:8000
  Home URL: http://localhost:8000
  Valid Redirect URL: http://localhost:8000/*
  Web Origins: http://localhost:8000
• Click save to create the client

Obtain Client Credentials

• Go to client details page of local-api client and select credentials tab.
• Copy client secret, it will be used by FastAPI app

Create a Test User in Keycloak

• Click Users tab, and Click Create new user or Add user button
• Fill the form and click create button.
• In the user details page, click credentials tab and set a new password. Don't forget to disable Temporary password toggle to keep password permanent.

Create settings.py

I'm going to use pydantic-settings to load env variables based on the runtime env. If you set RUNTIME_ENV as local or unset, then application will look for .local.env in the app directory to load env variables. You must set KEYCLOAK variables in .local.env file. An example .env file would look like this.

RUNTIME_ENV=local
KEYCLOAK_SERVER_URL=http://localhost:8080/auth
KEYCLOAK_CLIENT_ID=local-api
KEYCLOAK_REALM_NAME=local
KEYCLOAK_CLIENT_SECRET_KEY=<secret that you get from keycloak>

from pydantic_settings import BaseSettings, SettingsConfigDict
import os 


class CommonSettings():
    VERSION: str = "0.1.0"
    
    KEYCLOAK_SERVER_URL: str
    KEYCLOAK_CLIENT_ID: str
    KEYCLOAK_REALM_NAME: str 
    KEYCLOAK_CLIENT_SECRET_KEY: str


class LocalSettings(CommonSettings, BaseSettings):
    model_config = SettingsConfigDict(env_file='.local.env', env_file_encoding='utf-8')
    RUNTIME_ENV: str = "local"


class ProdSettings(CommonSettings, BaseSettings):
    model_config = SettingsConfigDict(env_file='.env', env_file_encoding='utf-8')
    RUNTIME_ENV: str = "prod"


runtime_env = os.environ.get("RUNTIME_ENV", "local")
settings = LocalSettings() if runtime_env == "local" else ProdSettings()

Create security.py

In this module, we implement functions to use later and initiliaze and configure keycloak class for connecting to keycloak server.

from keycloak import KeycloakOpenID
from .settings import settings as s


idp = KeycloakOpenID(
    s.KEYCLOAK_SERVER_URL,
    s.KEYCLOAK_REALM_NAME,
    s.KEYCLOAK_CLIENT_ID,
    s.KEYCLOAK_CLIENT_SECRET_KEY
)

Create auth.py and Login endpoint

In this module we implement login endpoint and issue a token from Keycloak by using credentials provided by user. That's why you need to enable Direct Access Grants in client capabilities to exchange token with username and password. We return access and refresh tokens to user.

from typing import Annotated
from fastapi import APIRouter, Depends, HTTPException
from fastapi.security import OAuth2PasswordRequestForm
from pydantic import BaseModel


from .settings import settings as s
from .security import idp
from keycloak.exceptions import KeycloakAuthenticationError, KeycloakError


class OpenIdToken(BaseModel):
    token_type: str
    access_token: str 
    refresh_token: str
    expires_in: int 
    refresh_expires_in: int 
    

app = FastAPI(title="Code Reference API",
              description="API for code reference",
              version=s.VERSION)


@app.post("/login")
async def login(form: Annotated[OAuth2PasswordRequestForm, Depends()]):
    try: 
        resp = idp.token(form.username, form.password, scope="openid profile email")
        return OpenIdToken(**resp)

    except KeycloakAuthenticationError as e:
        raise HTTPException(401, "Username or password is incorrect!") from e

    except KeycloakError as e:
        raise HTTPException(400, "Request malformed!") from e

By here, we have managed to login through Keycloak. Now you must be able to get access token from /login endpoint. Give it a try!

Protect an Endpoint and Validate Access Tokens

Now it's time to make JWT tokens mandatory for protected endpoints. Let's decode and validate JWT tokens.

Add below lines to security.py

• oauth_token gives information where to obtain token from for swagger UI.
• decode_token calls keycloak decode_token function to decode and verify token by fetching public key from KeyCloak. (I'm going to show how to verify it without fetching key every time)
  Function also handles the exceptions and returns proper error messages.

from fastapi.security import OAuth2PasswordBearer


oauth_token = OAuth2PasswordBearer("/login")


async def decode_token(token: str) -> dict:
    try: 
        return await idp.a_decode_token(token, validate=True)
        
    except JWTExpired as e:
        raise HTTPException(401, "Token has expired") from e 
    
    except InvalidJWSSignature as e: 
        raise HTTPException(400, "Token signature couldn't be verified") from e 
    
    except JWException as e: 
        raise HTTPException(400, "Token is malformed") from e 
    
    except Exception as e:
        print(e)
        raise HTTPException(500, "Error while decoding token") from e 

Then we need to implement a dependency function to mark Authorization header as required and inject it into endpoint.

from typing import Annotated
from fastapi import Depends
from .security import oauth_token, decode_token


async def get_user(token: Annotated[str, Depends(oauth_token)]):
    return await decode_token(token)

@app.post("/me")
async def me(user: Annotated[dict, Depends(get_user)]):
    return token

When you visit the swagger UI, you will see an lock icon next to /me endpoint and login button in top-right corner. Login and try to send a request to me endpoint. You should see the issued token by Keycloak.

How to Verify without fetching key every time?

On the first startup, we can load the key and store it in the memory. While decoding the token, we use the from memory instead of fetching it from Keycloak. It reduces the load on keycloak and latency, but it comes with its consequences such as how to handle key rotation? Just keep in mind.

We're going to use lifespan feature in FastAPI. We load the key on app startup and save it to settings variable to make it available everywhere.
Don't forget to add JWT_KEY to your settings class.

from contextlib import asynccontextmanager
from jwcrypto import jwk
from .settings import settings as s 
from .security import idp


@asynccontextmanager
async def lifespan(app: FastAPI):
    key = (
        "-----BEGIN PUBLIC KEY-----\n"
        + await idp.a_public_key()
        + "\n-----END PUBLIC KEY-----"
    )
    s.JWT_KEY = jwk.JWK.from_pem(key.encode("utf-8"))
    yield 


app = FastAPI(version=s.VERSION,
              lifespan=lifespan)

Now we should provide the key to Keycloak class to use it, instead of fetching it. Find and change decode_token function content.

return await idp.a_decode_token(token, validate=True, key=s.JWT_KEY)

Expectations

Usually, we expect the below criteria from an image registry.

• Push and pull container images
• Delete container images
• Persistent storage for images, preferably object storage like S3
• Authentication layer for security

Extras

• Mirroring on Docker hub or another container registry
• Security vulnerability scans on images
• Container image signing and licence checking
• Automatic retaining images by defined rules

Zot support all of the above features.

Configuration

This is the version I used: ghcr.io/project-zot/zot:v2.1.2-rc3
The entry point command is zot and I mount the configuration file below.

docker run -d -p 8000:8000 --name zot -v `pwd`/config.json:/etc/zot/config.json ghcr.io/project-zot/zot:v2.1.2-rc3 serve /etc/zot/config.json

{
    "distSpecVersion": "1.0.1",
    "storage": {
        "rootDirectory": "/tmp/zot",
        "commit": true,
        "dedupe": false,
        "gc": true,
        "gcDelay": "2h",
        "gcInterval": "1h",
        "storageDriver": {
            "name": "s3",
            "region": "eu-west-1",
            "bucket": "harver-zot-private-registry",
            "secure": true,
            "skipverify": false
        },
        "retention": {
            "dryRun": false,
            "delay": "24h",
            "policies": [
                {
                    "repositories": ["infra/**", "base/**"],
                    "keepTags": [{
                        "patterns": [".*"]
                    }]
                },
                {
                    "keepTags": [{
                        "patterns": [".*"]
                    }]
                }
            ]
        }
    },
    "http": {
        "address": "0.0.0.0",
        "port": "8000"
    },
    "extensions": {
        "metrics": {
            "enable": true,
            "prometheus": {
                "path": "/metrics"
            }
        },
        "sync": {
            "downloadDir": "/tmp/mirror",
            "enable": true,
            "registries": [
                {
                    "urls": ["https://docker.io"],
                    "content": [
                        {
                            "prefix": "**",
                            "destination": "/docker"
                        }
                    ],
                    "onDemand": true,
                    "tlsVerify": true
                }
            ]
        },
        "search": {
            "enable": true
        },
        "scrub": {},
        "lint": {},
        "trust": {},
        "ui": {
            "enable": true
        }
    },
    "log": {
        "level": "debug"
    }
}

Container images can be built for different architectures. Then you can combine them into a single image tag. Actually, this is called image manifest file. That file has references for the regarding architectures.

Build Image from Dockerfile - buildx

Let's build a container image for linux/amd64 and linux/arm64 architectures. We can use buildx for that. First we need to install it

Buildx and buildkit is already enabled in Docker Desktop

Buildx has builder instances for the builds. Each build is executed inside them. The advantage of them, they provide isolated environments. You can even set up a build farm with a set of builders. You can switch between the builders.

For building multi-platform images; follow below

docker run --privileged --rm tonistiigi/binfmt --install all

docker buildx create --use --name multi-arch node-amd64
docker buildx create --amend --name multi-arch node-arm64

# List and switch contexts
docker buildx ls
docker buildx use --builder multi-arch

The builder instances are ready to use. We can build container images in two architectures on the same host.
--push parameter pushes the image to the repository. If you want to keep in local pass --load parameter.

Build Multi-Arch images

docker buildx build \
	--platform linux/arm64,linux/amd64 \
	--push \
	--tag [image_tag] \
	.

UUID

UUID (Universal Unique Identifier) is a 128-bit value used to uniquely identify an object or entity on the internet.
A real world example; When you visit a news website, you see something following in the address bar of the web browser.
https://newssite.example/category/economy/1
As you can imagine, the 1 indicates that it's the first article. If you increase the number you view the second article. It's predictable and the people can copy your content easily, just by increasing the numbers. So hiding your content's identical number to visitors can be crucial for web services. To be able to hide it you can start to implement UUID instead of regular numbers. Predicting UUID is difficult but implementing it is easy. Because an UUID looks like below;
291ad1cd-c352-4220-b8f8-5ee876da5390
If you can guess the next article's ID, don't wait and play lottery :)

The best place to put UUID field is the Base class. We ensure all resources in database has UUID type by defining there. And it is good for DRY (don't repeat yourself) principle.

As default parameter, we provide uuid.uuid4 function's itself. SQLAlchemy will call the function to generate UUID values. Both way is shown in below.

from sqlalchemy.orm import as_declarative
from sqlalchemy import Column, Integer
from sqlalchemy.dialects.postgresql import UUID
import uuid

class_registry: dict = {}

@as_declarative(class_registry=class_registry)
class Base():
    # id = Column('id', Integer, primary_key=True, index=True)
    uuid = Column('uuid', UUID(as_uuid=True), primary_key=True, index=True, default=uuid.uuid4)

Auto-Generated Table Names

Instead of typing __tablename__ in each model classes, you can follow below approach in Base class. You can use this decorator for more of course, like creating Mix-in classes. (in below it's explained)

from sqlalchemy.orm import as_declarative, declared_attr
from sqlalchemy import Column
from sqlalchemy.dialects.postgresql import UUID
import uuid

class_registry: dict = {}

@as_declarative(class_registry=class_registry)
class Base():
    @declared_attr
    def __tablename__(cls):
    	return f"{cls.__name__.lower()}s"

    uuid = Column('uuid', UUID(as_uuid=True), primary_key=True, index=True, default=uuid.uuid4)

Common Date Time Mix-in Classes

For auditing purposes and keeping everything under record, we should keep the creation and update dates. The below mix-in class is our abstract class. We will use them in classes which needs Date Time records.

Creation date and update date can be set automatically. Be aware of that we use server_default in created_at with func.now()
But we use onupdate in updated_at again with func.now() The reason is explained in this thread.

from sqlalchemy import DateTime, func
from sqlalchemy.orm import declared_attr

class DateTimeMixin:
    @declared_attr
    def created_at(cls):
        return Column(DateTime(timezone=True), server_default=func.now())
        
    @declared_attr
    def updated_at(cls):
        return Column(DateTime(timezone=True), onupdate=func.now())
        

class User(Base, DateTimeMixin):
    __tablename__ = "users"
    
    username = Column(String, index=True)
    password = Column(String)

Install required packages

poetry add sqlalchemy alembic

fooapi/ # Main package
    fooapi/
    	__init__.py
        main.py
    tests/
    poetry.lock
    pyproject.toml
    db/
    	__init__.py
        base_class.py
        base.py
        session.py

SQLAlchemy

We should create a new sub-package called db. We store the base classes, session objects and configurations for database connection. The structure is like below.

We define our base class first. We inherit new model classes from the base class. We can define the common columns in base class. It effects every model classes that is inherited from it. So that you don't have to type commun columns in each model classes.

from sqlalchemy.orm import as_declarative
from sqlalchemy import Column, Integer

class_registry: dict = {}

@as_declarative(class_registry=class_registry)
class Base:
	id = Column('id', Integer, primary_key=True, index=True)

class_registry dictionary is used for tracking the which classes are used creating database tables. It's mapping. as_declarative means that we declare the attributes and structure of table. We added an id column which is common for all of our tables.

Now we just defined our Base class. We need to initiate a Session to connect the database.  SQLAlchemy is able to connect to the different database engines with almost same code or with small changes.
We keep database connection string in a variable. We need to create an engine. ORM gets the target engine from connection string. We can pass some extra parameters here. For instance, sqlite doesn't support multi-thread applications. FastAPI creates threads for requests because it's based on starlette (ASGI) which works async. We define that don't check is the connection coming from same thread, or not.
sessionmaker functions returns Session object which is used actually to connect and initiate a session in database. This object is callable. If you want to connect to the db, you call this object. SessionLocal()

from sqlalchemy.orm import sessionmaker
from sqlalchemy import create_engine

SQL_URI = "sqlite:///app.db"

engine = create_engine(SQL_URI, connect_args={"check_same_thread": False})
SessionLocal = sessionmaker(engine, autoflush=False, autocommit=False)

We create another module called base.py I'll explain that in alembic section. This is how it looks like.

from .base_class import Base

Alembic

We need to initate alembic in project. Go to directory next to pyproject.toml and run alembic init alembic It creates a directory named alembic and alembic.ini file next to pyproject.toml This is last directory setup.

fooapi/ # Main package
    fooapi/
    	__init__.py
        main.py
    tests/
    poetry.lock
    pyproject.toml
    db/
    	__init__.py
        base_class.py
        base.py
        session.py
    # NEW files below
    alembic.ini 
    alembic/
    	versions/
        env.py

Find and comment out sqlalchemy.url line in alembic.ini, because we set this value by getting from session.py Make below changes in env.py

from fooapi.db.base import Base
target_metadata = Base.metadata

from fooapi.db.session import SQL_URI
def get_url() -> str:
	return SQL_URI

target_metadata is required to run alembic properly. It finds Base class get its metadata with table-class mapping that we defined in base_class.py
We imported SQL_URI to point out the target database.

In addition to that, we must change the run_migrations_offline function. We commented out the sqlalchemy.url in alembic.ini. Update the value of url parameter in context.configure method.

context.configure(
    url=get_url,
    target_metadata=target_metadata,
    literal_binds=True,
    dialect_opts={"paramstyle": "named"},
)

We have to make some updates in run_migrations_online function as well. Because we load connection string from session.py We need to set the connection string in this configuration too.

configuration = config.get_section(config.config_ini_section, {})
configuration['sqlalchemy.url'] = get_url()
connectable = engine_from_config(
    configuration,
    prefix="sqlalchemy.",
    poolclass=pool.NullPool,
    )

Alembic Revision and Apply

Alembic creates migration scripts and they're versioned. To create a new revision and apply it, you can give this command

alembic revision --autogenerate -m "first revision"
alembic upgrade head

Most probably it's gonna create anything in database, because we didn't define any model classes that are mapped to real database tables. At least we tested our integration in this step.

Create DB Model

Create a new sub-package called models under the fooapi package. Create a new module named user.py We define the user model class in this module.
It must be inherited from Base class. We have only 2 columns for this table. We have to set the table name as well.

from fooapi.db.base_class import Base
from sqlalchemy import Column, String


class User(Base):
    __tablename__ = "users"
    
    username = Column(String, index=True)
    password = Column(String)

Now go to the base.py under the db sub-package, import this model class, otherwise alembic isn't going to detect that model class is available.

from .base_class import Base

from fooapi.models.user import User

This part is really important. You must have base.py and base_class.py both.
Alembic needs base.py. Model classes need base_class.py.
Alembic needs Model classes as well, that's why we import model classes in base.py
If you didn't define your Base class in base_class.py you would cause circular dependency. That's why you must define this in separated files. If you get an error about that, ensure you imported Base class from the correct module.

Model Classes;
- Import Base class from base_class.py
- Import Module class within base.py for alembic to discover it.

Alembic;
- Import base.py
- base.py must import Base class from base_class.py
- Make sure you imported all defined Model classes within base.py

To be able to create the table for User model. Give below commands

alembic revision --autogenerate -m "User model added"
alembic upgrade head 


alembic downgrade -1  # To revert changes

Now we have created a table by assist of Alembic. We store the changes in our repository and able to revert and provision versions in database with less-effort.

shoin-posts/scheduled_queries.py at master · xsetra/shoin-posts

Contains devops scripts to re-use. Contribute to xsetra/shoin-posts development by creating an account on GitHub.

GitHubxsetra

Workflow

• The script simply reads the query list from query-catalog-file. There is a file  in repo named query-catalog.json that shows the structure.
  name , schedule and pubsub_topic is optional fields.
• It creates or deletes the queries according to operation parameter.
• The query must be provided within the catalog file.
• While deleting the queries, the generate_name function takes a big role. Because the query name is composed with a parent name that includes project, location, dataset_id. The script finds the correct name, then deletes the query.

usage: scheduled_queries.py 
    project_id 
    query-catalog-file 
    operation
    [-h] 
    [--service-account-name SERVICE_ACCOUNT_NAME] 
    [--pubsub-topic-id PUBSUB_TOPIC_ID] 
    [--default-schedule DEFAULT_SCHEDULE] 
    [--location LOCATION] 

Which one wins the race? File or Parameter?

Answer is file. For instance if you don't specify the name in file, the script will generate a name for it. If you specify, the file name is chosen. You can find the name rules in generate_name function.

Optional Parameters

• Service Account Name
  It's associated to bigquery scheduled query. You have to grant correct permissions according to query requirements. For instance, bigquery user.
• PubSub Topic Id
  When scheduled query executed, the result of query will be send to this topic.
  It's required to monitor the queries. Take a look monitoring solution.
• Default Schedule
  If you don't specify the schedule in catalog-file, this value will be used.
• Location
  What is the location of dataset or query execution location.

Example Usage

./scheduled_queries.py gcp-project query-catalog.json create

• VPC endpoint provides access to public AWS services for resources that don't have public IP or where NAT gateway isn't deployed.
• There are gateway (s3, dynamodb) and interface (most aws services) endpoints.

Gateway Endpoints

• Uses routing.
• They are present in route table via prefix lists which are represent the CIDR of service. Prefix lists are updated by AWS.
• They can have associated policies which defines who can access.
• They are highly available for all AZs in a Region.

Interface Endpoints

• Provisions a networking object, ENI.
• The provisioned ENI is your interface endpoint to connect the service.
• That ENI has got ip/dns pair, Security group. NACL works in subnet level.
• You can provision that ENI in multiple subnets, one per AZ. The applications will use DNS resolve. By doing that endpoints will be highly available.
• Enable the Private DNS to override public default name of service. Also, you can give a different names for AWS services, because you will have private route 53 zone.

VPC Peering - Layer 3

• VPC peering is a way to link or connect two VPCs together without using any additional non AWS services.
• You can connect the services via private IP while VPC peering span AWS accounts, regions with limitations.
• Data is encrypted and transits via the Global backbone with lower latency.
• It's scalable and highly performant way.
• Use Case:
  - Sharing database with other VPC and access to Database.
  - Security auditors can be connect your VPC and performs tests.
  - Vendor provided service, it should be a web API
  - Splitted application for blast radius
• Peering connection is a gateway like NAT and IGW.
• VPC overlap is the limitation for that. There is requester and accepter.
• Adjust the route tables in both VPC side. Remote CIDR
• NACLs and SGs can be used to control access because you will have an ENI in your VPC. If VPCs are in same region, you can reference SG id.
• IPv6 support is available for cross-region
• DNS Resolution to private IPs can be enabled. It's a setting needed to adjust both sides. It prevents the traffic leaves AWS.
• Transitive Routing isn't supported. Let's say A-B and B-C are peered. It doesn't mean you can reach to C from A. You must create another peering between A-C

Isolated Network Blast Radius
Let's say something happens in VPC-A such as malware infection or an attack. When you isolate your network, the other VPCs won't be affected from issue in VPC-A. It will decrease you blast radius.

VPC and Subnets

If you have a fresh AWS account, the default VPC will be created for you. The properties of default VPC like following;

• 172.31.0.0/16 and /20 subnet in each AZ with public IP enabled.
• Internet Gateway with a configured main route table and DHCP option set
• NACL: All allow in/out , Security Group: All from itself (ingress) , All egress

VPC has some limitations such as max subnet range could be /16 and min /28 for ipv4. but ipv6 max is /56.

The VPC is software defined network, but these definitions would effect the hardware such as router, gateway, firewalls. AWS offers two way of tenancy to consume network devices.
Default: Shared network. The underlying devices are shared with other tenancies. It could be changed after creation.
Dedicated: Locks this VPC to dedicated hardware and can't be changed later.

In addition to above limitations; you must to be ensure that ip overlapping doesn't exist with other accounts and partners, while designing corporate network.

VPC Routes

To deliver packets from a subnet to other or internet, we need routing.

• Internet gateway has public IP and performs Static NAT. It translates the private ip addresses with public IP.
• Route table manages the VPC router. It can route propagation over BGP when you have DirectConnect or VPN connection.
• Route table is associated with subnets. Specific IP in route table has high priority.

VPC Security

AWS offers different services which they can work in different network levels.

NACL - Layer 4

• Controls data traffic across subnets. It is consisted list of rules.
• Impacts only traffic crossing boundary of subnet.
• It contains explicitly deny or allow rules. Protocol, IP Range, Port for source and destination.
• Rules are processed in number order. Lowest first. When a match found, process stops.
• * rule is default. Last processed and implicit deny.
• NACLs are stateless, you must add your rule ingress and egress appropriately for ephemeral ports.

Security Groups - Layer 5

• Think like firewall rules. Thanks to Layer 5 capabilities, it stores the session. That means you have stateful firewall.
• Security groups can be associated with AWS resources such as EC2 instance, EFS mount point. You can associate with each resource that has ENI
• SG can not deny traffic explicitly. Insert allowed sources and protocols. If you want to explicitly deny, use NACL.

NAT Gateway

If instances or resources inside a VPC don't need incoming internet access, don't give them IPv4 public ip. IPv4 pool is out of space. Use NAT Gateway.

NAT Gateway is used to access public world from private network with a static IP address. It manages the egress traffic. It's suit for your private subnet's resources.
NAT Gateway run in subnet. Subnet is present in AZ. You may provision NAT Gateways for each subnet. Create route tables for each subnet that has been configured to consume NAT Gateways. By doing that your system will be resilient and highly available.

Cross VPC Access

VPC offers the ability of cross access or communication between your VPCs or 3rd part VPCs. I mentioned about the requirement the isolation of VPCs.

If you want to consume 3rd party service which is developed in AWS without public internet access, the only thing that you need is AWS PrivateLink. There will be another post.

Job Common Properties

• First of all visit dataflow service page. Click the create job from template button. You should create a dataflow job from template. The template name is Bulk Delete Entities in Datastore.
• Give a name to your job.
• Select an endpoint. Job metadata will be stored in here. Select same with datastore region.

Required Parameters

• Type your GQL query. Simply SELECT * FROM [kind_name]
• Type your read_project_id
• Type your delete_project_id. In this point you should ask, why I had given two project id. Dataflow performs read operations from your read_project_id, then deletes the entities from the delete_project_id.
• Temporary location is required to store some metadata and log variety things. Type a bucket path. gs://temp-dataflow-delete/path/

You shouldn't need to define any additional parameters theoretically. Because you defined required parameters, right? But there is more.

• For example; we haven't defined the namespace? It will delete but from where?
  Click Show Optional Parameters to define.
• Type your datastore namespace to first field.
• UDF GCS path doesn't required at this point, leave it blank.
• Same for UDF function name, leave it blank.
• Max workers. It is most important point. However GCP didn't pay enough attention to this. As you can imagine, it limits the worker count.
  Dataflow provisions VM instances in your account to perform read and delete operations. This parameter set the maximum number of workers within a node group. Specify as you wish.
• Number of workers defines the initial number of workers. For example 1. Node group will scales the instances up to the Max workers.
• Select the worker region and zone.  If you select same region with your datastore, it would be good. So you don't pay inter zone and region  data transfer price.
• If you would like to associate a service account to workers, type the email address.
• Machine type is another important point. However they didn't pay enough attention to this parameter as always.
• Additional experiments isn't required. Leave it blank.
• Worker IP address configuration is important. If you don't have a special case, I suggest you to choose private. If you select the public, you will be charged. Because GCP suppose that it's giving a service to customer application or 3rd party. If you enabled Private Google Access in your VPC, then you will have an secure and high performant connection. Also you won't be charged. Yes again they didn't pay attention. :) I wonder why? :)
• You can specify the VPC network
• Also subnetwork.

That's it. You can run your job safely from price perspective. Also I suggest to use the burstable instance types which are f1-micro and g1-small. If you don't have performance concern, because they are cheaper :)

How works?

• It checks the availability of services regularly by using the health checks.
• The health checks are like the regular requests coming from clients.
• In addition you can specify the sources of health checks. If your customers are based in Europe, you can define health checks based on European cities.
• It updates the DNS records to another working service which is predefined IP or host when it determines an outage.
• Simply you define a primary/secondary system.

Advantages

• It forwards the traffic to always running system.
• Thanks to DNS failover your customers don't face with an issue or minimal while connecting your services. So you don't lose customers.
• You can cover the SLA as much as possible. And business happy :)

Use Cases

• The master servers of SQL database management systems.
• Web services for a location
• Network services like dns, smtp, ldap.