Stop Guessing Your Traffic: Analyze AWS VPC Flow Logs with Confidence

1. Introduction

In my experience working with AWS environments, I've found that many teams—despite having Flow Logs enabled—struggle to answer a surprisingly simple question: how much of our traffic is going to the internet, and to what destinations? AWS doesn't provide built-in metrics to answer this directly, and while VPC Flow Logs record granular details about IP traffic in and out of your VPC, turning that raw data into something actionable isn't straightforward.

Flow Logs are powerful for uncovering unexpected traffic patterns, identifying security risks, troubleshooting connectivity issues, and supporting compliance—but without the right tools and approach, they often remain an underutilized resource.

In this article, I'll walk you through how to go from enabling Flow Logs to extracting real insight:

• How to enable and store VPC Flow Logs
• Choosing the right destination: CloudWatch Logs vs Amazon S3
• Running real-time queries using CloudWatch Log Insights
• Performing large-scale historical analysis using Amazon Athena
• When to use Log Insights vs Athena

2. Setting Up VPC Flow Logs: Destinations & Storage Options

Enabling VPC Flow Logs is relatively straightforward, but the configuration choices you make will determine how effectively you can use the logs—and how much you'll spend storing and analyzing them.

2.1 Select the Monitoring Scope

VPC Flow Logs are most commonly enabled at the VPC level, which captures traffic from all Elastic Network Interfaces (ENIs) within that VPC. This provides broad visibility across your network and is the recommended starting point.

For more focused use cases, you also have the option to enable logging at the Subnet or ENI level.

2.2 Choose the Log Format

You can choose between:

• Default format – includes basic fields like source/destination IP, ports, protocol, traffic action (ACCEPT/REJECT), and byte/packet counts.
• Custom format – allows you to add fields such as tcp-flags, flow-direction, pkt-srcaddr, and more, giving you greater flexibility for filtering and analysis.

2.3 Assign an IAM Role

To deliver logs to your selected destination, you'll need to associate an IAM role with appropriate permissions:

• You can use the AWS-managed policy AmazonVPCFlowLogs, which grants the necessary access to write logs to CloudWatch Logs or S3.
• Alternatively, you can define a custom IAM role with more granular permissions.

2.4 Choosing a Destination: CloudWatch Logs vs. Amazon S3

After setup, you'll need to decide where the Flow Logs should be delivered. Both CloudWatch Logs and Amazon S3 are valid options, and the best choice depends on your analysis needs, expected log volume, and budget.

If you go with S3, structure your bucket using date and region-based prefixes like:

s3://your-bucket/AWSLogs/{account-id}/vpcflowlogs/{region}/YYYY/MM/DD/

This makes it easier to partition your data in Athena and optimize query performance.

VPC Flow Logs Architecture and Destinations

The following diagram illustrates the VPC Flow Logs architecture and the two main analysis paths:

VPC Flow Logs Destination Comparison

| Feature | CloudWatch Logs | Amazon S3 | |---------|-----------------|-----------| | Best For | Real-time analysis, quick queries | Historical analysis, cost optimization | | Query Tool | CloudWatch Log Insights | Amazon Athena | | Cost Model | Pay per GB ingested + queries | Pay per GB stored + queries | | Retention | Configurable (1 day to never expire) | Unlimited with lifecycle policies | | Integration | Native CloudWatch integration | Requires Athena setup |

3. Analyzing Flow Logs with CloudWatch & Athena

Once Flow Logs are flowing into CloudWatch or S3, it's time to extract insights. AWS offers two main tools for this: CloudWatch Log Insights for near real-time investigation, and Amazon Athena for large-scale, SQL-based analysis.

3.1 CloudWatch Log Insights

Best for: Fast, ad-hoc queries and live troubleshooting.

Getting Started

To get started:

1. Go to the CloudWatch Console
2. In the Logs -> Log groups section, select the ENI or VPC log group you want to analyze
3. Click "View in Logs Insights" from the top right to open the query editor

Practical Query Examples

As an example, if you're analyzing the ENI attached to a NAT instance, here are some practical queries to gain visibility into outbound internet traffic:

Top Internet Destinations (by bytes)

filter interfaceId = "eni-xxxxxxxxxxxxxxxxx" and not dstAddr like /^10\./
| stats sum(bytes) as total_bytes by dstAddr
| sort total_bytes desc
| limit 10

Top Internal Sources Using the NAT

filter interfaceId = "eni-xxxxxxxxxxxxxxxxx" and srcAddr like /^10\./
| stats sum(bytes) as total_bytes by srcAddr
| sort total_bytes desc
| limit 10

Top External Endpoints by IP and Port

filter interfaceId = "eni-xxxxxxxxxxxxxxxxx" and not dstAddr like /^10\./
| stats sum(bytes) as total_bytes by dstAddr, dstPort
| sort total_bytes desc
| limit 10

Traffic by Protocol

filter interfaceId = "eni-xxxxxxxxxxxxxxxxx"
| stats sum(bytes) as total_bytes by protocol
| sort total_bytes desc

Rejected Traffic Analysis

filter interfaceId = "eni-xxxxxxxxxxxxxxxxx" and action = "REJECT"
| stats count() as reject_count by srcAddr, dstAddr, dstPort
| sort reject_count desc
| limit 20

Important Note: Since NAT performs IP translation, Flow Logs reflect post-NAT addresses. You'll see the NAT's source IP and public destinations, but not the original private source IPs unless you're logging both ends of the connection.

3.2 Amazon Athena

Best for: Historical analysis, cost optimization, and large-scale traffic exploration.

Athena allows you to run SQL queries over VPC Flow Logs stored in S3. It's ideal for investigating trends over time or integrating Flow Logs with broader data pipelines.

Prerequisites

• Ensure Flow Logs are delivered to an S3 bucket
• Use a partition-friendly prefix structure such as: s3://your-bucket/AWSLogs/{account-id}/vpcflowlogs/{region}/YYYY/MM/DD/

Step 1: Define the Athena Table

CREATE EXTERNAL TABLE vpc_flow_logs (
  version int,
  account_id string,
  interface_id string,
  srcaddr string,
  dstaddr string,
  srcport int,
  dstport int,
  protocol int,
  packets bigint,
  bytes bigint,
  start bigint,
  end bigint,
  action string,
  log_status string
)
PARTITIONED BY (region string, dt string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ' '
LOCATION 's3://your-bucket/AWSLogs/{account-id}/vpcflowlogs/';

Step 2: Load Partitions

MSCK REPAIR TABLE vpc_flow_logs;

Alternatively, you can manually add partitions for better control:

ALTER TABLE vpc_flow_logs ADD PARTITION (region='us-east-1', dt='2025-05-07')
LOCATION 's3://your-bucket/AWSLogs/{account-id}/vpcflowlogs/us-east-1/2025/05/07/';

Step 3: Example Queries

Top Internet Destinations

SELECT 
    dstaddr, 
    sum(bytes) AS total_bytes,
    sum(packets) AS total_packets
FROM vpc_flow_logs
WHERE dt = '2025-05-07'
  AND NOT dstaddr LIKE '10.%'
  AND NOT dstaddr LIKE '172.%'
  AND NOT dstaddr LIKE '192.168.%'
GROUP BY dstaddr
ORDER BY total_bytes DESC
LIMIT 10;

Top Internal Sources Using the NAT

SELECT 
    srcaddr, 
    sum(bytes) AS total_bytes,
    count(*) AS connection_count
FROM vpc_flow_logs
WHERE dt = '2025-05-07'
  AND srcaddr LIKE '10.%'
GROUP BY srcaddr
ORDER BY total_bytes DESC
LIMIT 10;

Potential Port Scans

SELECT 
    srcaddr, 
    count(DISTINCT dstport) AS unique_ports,
    count(*) AS total_attempts
FROM vpc_flow_logs
WHERE dt = '2025-05-07'
GROUP BY srcaddr
HAVING unique_ports > 50
ORDER BY unique_ports DESC;

Traffic Trends Over Time

SELECT 
    dt,
    sum(bytes) AS total_bytes,
    sum(packets) AS total_packets,
    count(*) AS flow_count
FROM vpc_flow_logs
WHERE dt >= '2025-05-01'
  AND dt <= '2025-05-07'
GROUP BY dt
ORDER BY dt;

Cost Analysis: Top Data Transfer Destinations

SELECT 
    dstaddr,
    sum(bytes) / (1024.0 * 1024.0 * 1024.0) AS total_gb,
    sum(bytes) * 0.09 / (1024.0 * 1024.0 * 1024.0) AS estimated_cost_usd
FROM vpc_flow_logs
WHERE dt = '2025-05-07'
  AND NOT dstaddr LIKE '10.%'
  AND action = 'ACCEPT'
GROUP BY dstaddr
ORDER BY total_gb DESC
LIMIT 20;

Protocol Distribution

SELECT 
    CASE protocol
        WHEN 6 THEN 'TCP'
        WHEN 17 THEN 'UDP'
        WHEN 1 THEN 'ICMP'
        ELSE CAST(protocol AS VARCHAR)
    END AS protocol_name,
    sum(bytes) AS total_bytes,
    sum(packets) AS total_packets
FROM vpc_flow_logs
WHERE dt = '2025-05-07'
GROUP BY protocol
ORDER BY total_bytes DESC;

Athena excels when you need to run deep historical analysis, join data sets, or build dashboards around long-term traffic patterns.

4. Choosing the Right Tool

Both CloudWatch Log Insights and Amazon Athena are valuable tools for analyzing VPC Flow Logs—but they serve different needs. Understanding when to use each, and how to configure them effectively, can significantly improve both visibility and cost-efficiency.

When to Use CloudWatch Log Insights

Use CloudWatch Log Insights when:

• You need real-time visibility into network behavior
• You want to visualize patterns or set up alerts using CloudWatch metrics and dashboards
• Your log volume is moderate, and cost is not a primary concern
• You're troubleshooting immediate connectivity issues
• You need quick answers to ad-hoc questions

Advantages: * Fast query execution (typically seconds) * Native integration with CloudWatch alarms and dashboards * No additional setup required if logs are already in CloudWatch * Easy to use query interface

Limitations: * CloudWatch is great for quick answers and operational monitoring, but its pricing can escalate with high-volume or long-retention use cases * Limited to querying logs within the retention period * Less efficient for large-scale historical analysis

When to Use Amazon Athena

Use Athena when:

• You need to analyze large-scale logs across many days, weeks, or months
• You're investigating trends, cost optimization, or security patterns
• You want to join Flow Logs with other datasets (e.g., billing, CloudTrail, DNS logs)
• You require cost-effective storage and querying for long-term retention
• You need to build automated reporting or dashboards
• You're performing compliance audits or security investigations

Advantages: * Cost-effective for large-scale analysis (pay per query, not per GB stored) * Unlimited retention with S3 lifecycle policies * SQL-based queries (familiar to most developers) * Can join with other data sources * Better for batch processing and scheduled reports

Considerations: * Requires initial setup (table creation, partition management) * Query execution time depends on data volume * S3 storage costs need to be considered separately

Cost Optimization Tips

1. Use S3 Lifecycle Policies: Move older logs to cheaper storage classes (IA, Glacier) to reduce costs
2. Partition Your Data: Proper partitioning in Athena significantly reduces query costs and improves performance
3. Set Retention Policies: Configure CloudWatch log retention to avoid unnecessary storage costs
4. Use Athena for Historical Analysis: For queries spanning weeks or months, Athena is typically more cost-effective than CloudWatch
5. Compress Logs: Enable compression for S3 logs to reduce storage costs

5. Advanced Use Cases

5.1 Security Monitoring

Flow Logs can help identify security threats:

-- Identify potential DDoS patterns
SELECT 
    dstaddr,
    count(*) AS connection_count,
    sum(bytes) AS total_bytes
FROM vpc_flow_logs
WHERE dt = '2025-05-07'
  AND action = 'ACCEPT'
GROUP BY dstaddr
HAVING connection_count > 10000
ORDER BY connection_count DESC;

5.2 Cost Optimization

Identify high-cost data transfer patterns:

-- Find top data transfer destinations outside AWS
SELECT 
    dstaddr,
    sum(bytes) / (1024.0 * 1024.0 * 1024.0) AS total_gb
FROM vpc_flow_logs
WHERE dt >= '2025-05-01'
  AND dt <= '2025-05-07'
  AND NOT dstaddr LIKE '10.%'
  AND NOT dstaddr LIKE '172.%'
  AND NOT dstaddr LIKE '192.168.%'
GROUP BY dstaddr
ORDER BY total_gb DESC
LIMIT 50;

5.3 Compliance and Auditing

Track all outbound connections for compliance:

-- Complete audit trail of outbound connections
SELECT 
    dt,
    srcaddr,
    dstaddr,
    dstport,
    protocol,
    sum(bytes) AS total_bytes,
    count(*) AS connection_count
FROM vpc_flow_logs
WHERE dt >= '2025-05-01'
  AND action = 'ACCEPT'
  AND NOT dstaddr LIKE '10.%'
GROUP BY dt, srcaddr, dstaddr, dstport, protocol
ORDER BY dt, total_bytes DESC;

Final Thoughts

VPC Flow Logs are one of the most underutilized tools in AWS when it comes to understanding real network behavior. Simply enabling them isn't enough—it's how you store, query, and act on the data that makes the difference.

Whether you're debugging connectivity issues, monitoring for unusual traffic, or looking to cut data transfer costs, combining CloudWatch Log Insights for real-time analysis and Amazon Athena for scalable, long-term querying can give you the clarity you need.

The key is to start with CloudWatch for immediate visibility, then leverage Athena for deeper historical analysis and cost optimization. With the right queries and approach, Flow Logs can transform from raw data into actionable insights that drive better decisions about your network architecture and spending.

If your focus is specifically on NAT traffic analysis, consider implementing a self-managed NAT solution for enhanced visibility and cost savings.