‘White hat’ penetration testing, which allows for better understanding and mitigation of system vulnerabilities, starts with full insight into your organization’s publicly accessible systems. Across multiple accounts, services, and regions, there is no standard report available in AWS to quickly and efficiently provide this information.
We created a public IP inspector, PIPPIN, to address this challenge using the AWS Application Programming Interface (API). This post walks through our process of discovering and cataloging an organization’s publicly accessible systems and our results with PIPPIN.
Defining your Public IP Landscape
You first need to collect AWS Account IDs, a twelve digit identifier that every AWS account receives upon creation. Enterprises may have dozens or hundreds of AWS Account IDs, depending on their history of AWS usage and the degree of centralization. The IDs need to be entered into a ‘System of Record (SoR)’ that PIPPIN can query and use for future processing. As new AWS Accounts are created, the Account ID needs to be added into the SoR. We typically recommend that customers use Amazon DynamoDB as the SoR for the Account IDs.
You then need to establish an IAM Role within each AWS Account. PIPPIN will utilize this role to run the necessary queries. Depending on the number of AWS Accounts in an organization, it may take some time to log in to each AWS account with root credentials and create the needed IAM Role. Once the IAM Role is established, PIPPIN will use the AWS Security Token Service API to request temporary credentials that have the permissions to execute the queries.
Next, you need to determine which AWS services can provision publicly available resources, either by IP address or domain name. While AWS updates its services frequently, a partial list as of this writing includes the following:
- Amazon EC2
- Amazon RDS
- Amazon API Gateway
- Amazon Elastic Load Balancer (ELB)
- Amazon Lightsail
- Amazon Redshift
- Amazon Elasticsearch Service
- Amazon Cloudfront
Implementation
So what is the best way to query and store IP addresses and domain names? The most obvious course of action is to write a single program that reads the list of Account IDs, queries each service for public IP addresses, and stores it in the master account. However, each AWS account has sixteen commercially available regions (soon to be 20) and a typical enterprise customer will be interested in multiple services from the bulleted list above. For example, an enterprise with 20 accounts querying all 8 services would need to execute 2560 queries [20 accounts * 8 services * 16 regions ]. If we assume 0.5 seconds per query, this results in overall processing time of 20 minutes, which is too long for most enterprises.
As opposed to a monolithic, single threaded program, we decided to use AWS Lambda in asynchronous mode. This allows us to query the regions within each account in parallel, dramatically reducing the runtime of the system.
Below is a PIPPIN diagram that illustrates the use of AWS Lambda to execute queries against four services.
The AWS Cloud account on the left is the master account and the AWS Cloud account on the right represents each of twenty accounts to be queried (the target accounts). Each service requires its own inspector. As new services are added, we write a new inspector, plug it into PIPPIN, and run it in parallel with other services.
Workflow and Data Formats
The PIPPIN workflow is as follows:
- A scheduled Cloudwatch Event Rule invokes PIPPIN (daily).
- PIPPIN reads Account IDs from the SoR.
- For each Account ID, PIPPIN obtains temporary credentials from the Security Token Service within the target account.
- For each service, PIPPIN invokes the inspector for the service with the Invocation Type set to ‘Event.’
- PIPPIN obtains a list of all the regions.
- For each region, PIPPIN executes the query and appends the results to a list.
- PIPPIN sends the results to a predefined Amazon SQS queue.
- At a different time of day, a scheduled Cloudwatch Event Rule invokes PIPPIN to flush the queue and send the results to Amazon S3.
A representative record for an EC2 public instance discovered by PIPPIN is as follows:
[‘111122223333’, ‘us-east-1’, ‘EC2’,
‘ec2-45-3-87-4.compute-1.amazonaws.com’, ‘45.3.87.4’, ‘p3.2xlarge’]
The SQS queue stores messages from each AWS Account-Service. For example, each EC2 instance discovered within account 111122223333 will have a record within a single SQS message, and SQS messages can have one or more records.
The record composition for each Account-Service is shown below for EC2, RDS, API Gateway, and ELB.
| Inspector | Account | Region | Service | Domain | IP Address | Instance Type |
| EC2 | YES | YES | ‘EC2’ | MAYBE | YES | YES |
| RDS | YES | YES | ‘RDS’ | YES | YES | YES |
| API Gateway | YES | YES | ‘API Gateway’ | YES | YES | NO |
| ELB | YES | YES | ‘ELB’ | YES | YES | NO |
For API Gateway and ELB records, a single resource will have multiple records because those resources are supported by multiple computing instances for service high availability. Instance Type will be omitted since that is abstracted by the service.
EC2 records may contain a domain value since assignment of domain names to EC2 instances is dependent upon the DNS Hostname attribute of the VPC.
A view of the results stored in S3 is shown below.
Files can be downloaded individually or in bulk for import into business productivity applications and shared with white hat testers for analysis.
PIPPIN Outcomes
Runtime. For the example given above, PIPPIN takes about 10 seconds to query the SoR, assume the IAM Role in the target accounts, and kick off the 160 inspectors (20 accounts * 8 services per account). Each inspector, running in parallel, takes about 10 seconds to iterate across sixteen regions, run the query, and send the results to the SQS queue. Therefore, the overall runtime of PIPPIN is approximately 20 seconds – much better than the 20 minutes that would otherwise be required!
Scalability. The PIPPIN runtime is driven by the time to query the SoR and asynchronously invoke the inspectors. 20 accounts equates to 10 seconds, 40 accounts equates to 20 seconds, 80 accounts equates to 40 seconds, 160 accounts equates to 80 seconds, and so forth. These times are well within the runtime boundary of AWS Lambda. As an enterprise’s number of AWS Accounts grow, it can raise its soft limit of concurrent Lambdas to ensure that its inspector Lambda functions are not throttled.
Summary
Overall, PIPPIN efficiently and cost-effectively queries an enterprise’s AWS infrastructure and reports out the publicly-available resources for white hat testing. It’s a great complement to the other cloud capabilities that Novetta offers to enterprise customers.
