VPC Flow Logs and VPC Traffic Mirroring

Important points 

A few important points related to VPC flow logs are listed below:

• It helps monitor and troubleshoot our network issues.

• VPC flow logs can be stored in Amazon S3, CloudWatch, and Kinesis Firehose.

• The EC2 instance to which an elastic network interface (ENI) is attached must have a public IP address.

• In VPC peering, we can create flow logs for the peered VPC only if that VPC belongs to our AWS account. 

• Once the VPC flow log is created, we can’t modify its settings, fields, or formats. We must delete it and create a new one with the required configurations.

Flow logs at the VPC level

When the flow log is configured at a VPC level, it monitors all the network interfaces in all the subnets attached to the resources. Configuring flow logs at the VPC level is useful when resources are deployed in different subnets, and we want to monitor traffic flow between them. The diagram below shows the flow log configured for the VPC and stores the logs in Amazon S3. 

Flow logs at the subnet level

Sometimes, we only want to monitor resources in a specific subnet. Then, we don’t need to configure flow logs at the VPC level. Monitoring the network interfaces in that particular subnet brings more relevant information. The following diagram shows the flow logs configuration at the subnet level and stores the logs in Amazon S3.

Flow logs at the network interface level

The most fine-grained level of monitoring is to capture the traffic of the exact network interfaces in a subnet. There can be multiple EC2 instances in a subnet, and we can configure the flow log on a particular instance to monitor its traffic. The following diagram shows the flow logs configuration at the interface level in a subnet and stores the logs in Amazon S3.

Available fields in flow log records

Flow logs are captured and stored in regular intervals which are known as aggregation intervals. The default value of the aggregation interval for S3 is 10 minutes, and for CloudWatch, it is 5 minutes. The lower value of the aggregation interval means that a higher number of logs will be generated. Let’s look at some of the important fields in the table below:

VPC Traffic Mirroring

AWS offers a Traffic Mirroring service to capture and inspect traffic within a VPC. It captures the traffic from the network interface of the source and transfers it to the network interfaces of the target EC2 instances or a network load balancer. Traffic Mirroring also allows us to filter traffic and truncate packets that are not required. 

Traffic Mirroring is used for content inspection, threat monitoring, and troubleshooting by mirroring the traffic to the monitoring appliances running on separate instances. The following illustration shows that the source_Instance communicates with the internet traffic, and some security appliance applications run on EC2 instances managed by the Auto Scaling group. A network load balancer is attached to the instances.

Traffic Mirroring is configured between source_Instance and Auto Scaling group of appliance applications to mirror the traffic. We can filter traffic and process it as required.

VPC Traffic Mirroring vs. VPC Flow Logs

VPC flow logs capture the IP traffic flowing in and out of the elastic network interfaces in our VPC. VPC flow logs collect logs without getting in the way of your traffic route, so the network throughput or latency is not disturbed.

Whereas, Traffic Mirroring is a feature that copies the inbound and outbound traffic from the source ENI to the target ENIs. Traffic Mirroring takes bandwidth from the instances, and if the network gets choked, then priority will be given to the actual traffic flow, and the traffic mirroring is dropped.

We’ve studied VPC flow logs and their importance in monitoring and troubleshooting network issues in our VPC. Flow logs can be configured at three levels and the logs can be stored in Amazon S3, Amazon CloudWatch, or Kinesis Firehose. We also discussed some important fields from the flow logs.