Encryption

In our Kubernetes clusters, we leverage Cilium as the primary networking solution, configured with WireGuard for node-to-node encryption. This setup ensures that all communication between Kubernetes nodes is encrypted, thereby protecting data in transit within the cluster. This document outlines the encryption protocols and mechanisms employed, highlighting compliance with security standards.

Node-to-Node Encryption

Node-to-node encryption in Kubernetes clusters is critical for ensuring that traffic traversing the network between nodes is protected against interception or tampering. By using WireGuard integrated with Cilium, we ensure that sensitive traffic remains secure, even in environments where underlying network infrastructure might be untrusted.

Encryption Details

Protocol and Cipher Suite

Cilium utilizes WireGuard, a modern VPN protocol known for its simplicity and performance. WireGuard employs ChaCha20-Poly1305 for authenticated encryption, providing both confidentiality and data integrity. Specifically:

ChaCha20 is a stream cipher that encrypts the data.

Poly1305 is a cryptographic message authentication code (MAC) used to verify the authenticity of the data.

This combination ensures:

Confidentiality: Unauthorized parties cannot read the data as it travels between nodes.

Integrity: Any tampering with the data will be detected, and the data will be rejected.

Key Exchange

WireGuard uses a static key exchange mechanism based on the Curve25519 elliptic curve. The keys are generated on each node, and WireGuard facilitates secure communication by using a pre-shared public key for each peer-to-peer connection. The key exchange mechanism ensures forward secrecy, meaning that if one key is compromised, it cannot decrypt past communications.

Authentication

WireGuard uses pre-shared public keys to authenticate peers. Each Kubernetes node is assigned a unique public/private key pair, and nodes exchange public keys securely. Only traffic between authenticated nodes is encrypted, ensuring that unauthorized nodes cannot inject or intercept traffic.

Data Encryption in Transit

All communication between Kubernetes nodes, including API requests, data replication, and service traffic, is encrypted by default when node-to-node encryption is enabled. This covers:

Pod-to-pod traffic across nodes

Encryption at Rest: Methods & Options

Encryption at rest is essential for protecting sensitive data stored on disk or other storage systems.

Below are three methods companies can use to encrypt data at rest when working with WitnessAI, providing varying levels of security, control, and complexity.

We use postgreSQL-16.0 and elasticsearch-8.0 for all our application/observability data.

Key Management Options

Witness-Hosted KMS Key: Simplifies encryption management, but Witness holds the KMS keys.

Witness-Hosted KMS with CloudHSM: Adds enhanced security with hardware-backed key management, while still being hosted by Witness.

Company-Hosted KMS/KMS-HSM: Provides full control and flexibility, allowing companies to manage their own encryption keys while granting Witness the ability to use them for encryption tasks.

Each method provides strong encryption at rest, with increasing levels of control and responsibility as you progress from option 1 to option 3. Choose based on your organization's security needs, compliance requirements, and desire for control over encryption keys.

1. Witness-Hosted KMS Key

In this approach, a Key Management Service (KMS) key is hosted by Witness. This key is unique to your company and is used to encrypt data across multiple components:

Data Stores: All databases and file systems that store data on disk are encrypted using this KMS key.

Applications: Any application storing sensitive data is also protected by this encryption method.

Cluster-Level Encryption: The cluster that runs your applications is encrypted, ensuring that data at rest within the cluster is protected.

Secrets Management: Any secrets stored in the system, such as passwords, API keys, or other sensitive configuration data, are encrypted using this key.

This method simplifies the process since Witness manages the encryption for you, while still providing strong security guarantees tailored to your organization.

2. Witness-Hosted KMS Key Backed by CloudHSM

This option is similar to the first but with a higher level of control and security. Witness hosts the KMS key, but it is backed by a CloudHSM (Hardware Security Module). In this scenario:

Higher Security with HSM: CloudHSM allows your company to manage its own encryption keys in hardware, adding an extra layer of security. CloudHSM ensures that encryption keys are generated and stored in hardware modules, which makes them much more difficult to extract or tamper with.

Use Your Own Keys: This gives the company the ability to bring their own encryption keys (BYOK), while Witness can still leverage these keys to encrypt data at rest. The HSM ensures that even if the system is compromised, the key itself remains secure.

This approach is beneficial for companies with strict compliance requirements or those who need full control over encryption keys.

3. Company-Hosted KMS/KMS-HSM

In this method, the company takes full control by hosting their own KMS or KMS backed by HSM. In this setup:

Company-Controlled Encryption: The company is responsible for hosting and managing their own KMS (or KMS-HSM), giving them complete control over the encryption process. Witness is given access to use the keys by assuming a specific role with predefined permissions.

Flexible Key Management: This option provides the highest level of flexibility and control, as the company can configure their own encryption policies, rotations, and audits.

Witness Integration: Witness will be granted access to read from the company-hosted KMS keys via an IAM role, enabling it to use the keys for encryption at rest while leaving the control of the key lifecycle in the hands of the company.

This method is ideal for companies that have specific regulatory or security needs that require them to manage their encryption keys completely independently of Witness.

AWS KMS Key Generation Instructions

This is a step-by-step guide to generate an AWS KMS key that can be used as a Bring Your Own Key (BYOK) for encrypting your data in your Witness AI tenant:

Prerequisites

AWS account with appropriate IAM permissions

AWS CLI installed and configured, or access to AWS Management Console

Step 1: Create the KMS Key

Using AWS Console:

Navigate to AWS Key Management Service (KMS) in the AWS Console

Click "Create key"

Select "Symmetric" key type

Choose "Encrypt and decrypt" for key usage

Select "Customer managed key"

Click "Next"

Using AWS CLI:

aws kms create-key \

--description "BYOK key for [Your Service Name]" \

--key-usage ENCRYPT_DECRYPT \

--key-spec SYMMETRIC_DEFAULT

Step 2: Configure Key Policy

Set up the key policy to allow WitnessAI to use the key, while you maintain control:


{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "Enable customer account permissions",

"Effect": "Allow",

"Principal": {

"AWS": "arn:aws:iam::<CUSTOMER ACCOUNT ID>:root"

},

"Action": "kms:*",

"Resource": "*"

},

{

"Sid": "Enable customer account permissions",

"Effect": "Allow",

"Principal": {

"AWS": "arn:aws:iam::<CUSTOMER ACCOUNT ID>:role/<CUSTOMER ROLE NAME>"

},

"Action": "kms:*",

"Resource": "*"

},

{

"Sid": "Allow service provider access",

"Effect": "Allow",

"Principal": {

"AWS": "arn:aws:iam::971422685605:root"

},

"Action": [

"kms:Encrypt",

"kms:Decrypt",

"kms:ReEncrypt*",

"kms:GenerateDataKey*",

"kms:DescribeKey",

"kms:ListAliases",

"kms:CreateGrant",

"kms:ListGrants",

"kms:RevokeGrant"

],

"Resource": "*"

}

]

}

Variable Descriptions

<CUSTOMER ACCOUNT ID> in the first block would be customer account ID specifying use of root role. This should be filled in automatically if you are using the AWS GUI.

The second <CUSTOMER ACCOUNT ID> is optional but recommended. This allows you to manage the key when logged in to AWS with a different defined role. For example, in case you don’t wish to have to log in as root to manage the key.

"Allow service provider access" Sid should have the role of root. You can obtain the <Witness_Account_ID> from your WitnessAI account team.

Step 3: Create Key Alias (Optional but Recommended)


aws kms create-alias \

--alias-name alias/byok-[your-service-name] \

--target-key-id [KEY_ID]

Step 4: Cross-Account Access Setup

If the key will be used across AWS accounts, configure cross-account access:

Add WitnesAI’s account to the key policy (shown above)

The WitnessAI must create an IAM role with permissions to use the key

Establish trust relationship between accounts

Step 5: Provide Key Information

Share the following information with your WitnessAI account team:

Key ID or Key ARN: Found in the KMS console or CLI output

AWS Region: Where the key was created

Key Alias: If created

Example Key ARN format:


arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012

Step 6: Test Key Access

Verify the key works by testing encryption/decryption:


# Test encryption

aws kms encrypt \

--key-id [KEY_ID] \

--plaintext "test message" \

--output text \

--query CiphertextBlob

# Test decryption

aws kms decrypt \

--ciphertext-blob [ENCRYPTED_OUTPUT] \

--output text \

--query Plaintext | base64 --decode

Once you've completed these steps, provide the Key ARN to WitnessAI to begin using your customer-managed encryption key for data protection.

Important Security Considerations

Key Rotation: Enable automatic key rotation for enhanced security

Monitoring: Set up CloudTrail logging to monitor key usage

Access Control: Regularly review and audit key policies

Backup: Consider key backup strategies for disaster recovery

Key Management Best Practices

Use descriptive names and tags for easy identification

Implement least-privilege access principles

Regularly rotate access keys and review permissions

Monitor key usage through CloudWatch and CloudTrail

Document key purposes and authorized users

Troubleshooting Common Issues

Access Denied: Check IAM permissions and key policies

Key Not Found: Verify key ID/ARN and region

Cross-Account Issues: Ensure proper trust relationships are established