Part 1: Setting Up Initial AWS Infrastructure for the Intrusion Detection System with Terraform

Introduction

This article is the first in a series where I’ll build an Intrusion Detection System(IDS) that classifies network traffic into normal and malicious traffic using AWS, Terraform, Flask, NextJS, and the other frameworks and tools.

In this part, I’ll focus on creating the foundational AWS infrastructure to manage the Terraform state files and the S3 bucket that will handle the uploaded dataset storage.

NOTE: This is not a tutorial. The project summaries for each part will be posted here, the highlights, problems faced and their respective solutions.

Highlights of the setup

Terraform Backend

This is where the state files of the main infrastructure will be stored. A separate remote backend ensures the project’s infrastructure management is seamless and collaborative, thus I provisioned:

• An S3 bucket for storing the Terraform state file

• A DynamoDB table to lock the state file and prevent simultaneous conflicting operations.

Although I did this alone and could have had the Terraform backend local, I believe having it remote on S3 is best practice.

Terraform Main Infrastructure

Now to the main infrastructure, with the backend already provisioned, this main infrastructure is set to use that backend. Both are separate, with the Backend independent and the Main dependent on the backend.

The first service I provisioned was an S3 bucket. This bucket will hold all the dataset uploads by the user and it will be versioned and encrypted.

An S3 bucket

Verifying the setup

Once I created my Terraform backend I went to my AWS management console to check the resources that have been made, and everything was. From the backend S3 Bucket and DynamoDB Table, to the main infrastructures S3 bucket, everything was provisioned accordingly.

Challenges Faced

Backend and Main Infrastructure Together: Initially, I defined both the remote backend (S3 and DynamoDB) and the main infrastructure (e.g., S3 file storage bucket) within the same Terraform configuration. When I ran terraform destroy to clean up resources, the backend itself was deleted, preventing me from re-provisioning the infrastructure because there was no backend to manage the state file.

I resolved this by separating the backend setup into its own Terraform configuration, independent of the main infrastructure.

Lessons Learned

1. Separate Terraform configurations: I understood the importance of having the remote backend separate from the main infrastructure to prevent inadvertent deletions when i try to clean up my resources, ensuring my backend remains intact.

2. Debugging Terraform: The terraform validate and terraform plan commands are invaluable for catching configuration errors early

Conclusion

In this article, I outlined the initial setup of AWS infrastructure for our intrusion detection project. I configured a remote backend with S3 and DynamoDB and deployed an S3 bucket for file storage.

What’s Next?

In the next part, we’ll build the Next.js component that allows users to upload files to the S3 bucket via an API route in our Flask backend. Stay tuned!