The Logs ingested to ELK is the flaws.cloud AWS CloudTrail dataset to identify unusual patterns and detect potential security threats.

CloudTrail Log Dataset Summary

The Log Dataset used in this lab is the Public dataset of Cloudtrail logs from flaws.cloud which can be accessed Here.

You can Read through the blog post Here. md5: 4f0481c6be700ffc7610dbbf8cee5578 Size: ~240MB of log data

• Download the CloudTrail zip file from HERE into a folder called s3_logs.

  

  • Decompress files using 7zip or in-build decompresser in mac

  

• Navigate to your project file and use the command below to download the Helper Script aws-cloudtrail2sof-elk.py from the SOF-ELK repository.

  wget <https://raw.githubusercontent.com/philhagen/sof-elk/main/supporting-scripts/aws-cloudtrail2sof-elk.py>
  

  

• Ensure Python3 is Installed. If not use sudo apt-get install python3 on Ubuntu or on Mac use the command brew install python3 once Homebrew is installed .

  Use the command to check Python3 is installed:

  python3 --version
  

  

• Next, run the following command to create a single JSON file from the CloudTrail logs located in the ./s3_logs/ directory:

  NB: Make sure you are in the directory that the Flaws_cloud CloudTrail logs (Here,s3_logs) are stored

  python3 aws-cloudtrail2sof-elk.py -f -r ./s3_logs/ -w FlawsReadyToIngest.json
  

  • f: Tells the script to force overwrite any existing output file.

  • r ./s3_logs/: Specifies the directory where your CloudTrail logs are stored.

  • w ReadyToIngest.json: Specifies the output file (FlawsReadyToIngest.json) that will contain all the logs in JSON format, ready for ingestion.

    

• The File Processed Successfully

  

Unfortunately, the File upload restriction to ELK is 100MB, so the FlawsReadyToIngest.json file can’t be uploaded. So, therefore, there is a need to split the data into multi-files for easy upload. A Python program has been created to separate the FlawsReadyToIngest.json into chunks of 50MB files.

• The python code split_json.py is executed

  

# To execute the code, run the command 
python3 split_json.py -f -r ./FlawsReadyToIngest.json -w ./flaws_split_files/

• This command created a folder with the slitted files of FlawsReadyToIngest.json

  

Ingest the CloudTrail Log Files to ELK