AWS Lambda Function that will create 7zip archive from the files in S3 Bucket (S3 Event as a trigger)

devops-aws

This Lambda function is triggered by S3 object events and rebuilds an index.7z archive from objects in the same folder. It skips existing archive files, creates a temporary working directory, and uploads the resulting archive back to S3.

Workflow Summary

  • Read bucket/key from S3 event record
  • Resolve folder prefix from incoming object key
  • Download folder files to unique /tmp path
  • Create 7z archive via /opt/bin/7z
  • Upload as index.7z and cleanup temporary files

Lambda Handler (Python)

import os
import boto3
import subprocess
import shutil
import logging

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()

s3_client = boto3.client('s3')

def lambda_handler(event, context):
    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']

        # Check folder where index.zip is located
        if '/' in key:
            prefix = key.rsplit('/', 1)[0] + '/'
        else:
            prefix = ''

        # List all objects
        response = s3_client.list_objects_v2(Bucket=bucket, Prefix=prefix)
        if 'Contents' not in response:
            logger.info(f"No objects found in {prefix}")
            continue

        # Unique TMP dir
        work_dir = f"/tmp/game_folder_{context.aws_request_id}"
        if os.path.exists(work_dir):
            shutil.rmtree(work_dir)
        os.makedirs(work_dir)

        # Download files except index.zip, index.7z, and directories
        for obj in response['Contents']:
            obj_key = obj['Key']
            if (obj_key.endswith("index.zip") or
                obj_key.endswith("index.7z") or
                obj_key.endswith('/')):
                continue

            relative_path = obj_key[len(prefix):]
            local_file_path = os.path.join(work_dir, relative_path)
            local_dir = os.path.dirname(local_file_path)
            if not os.path.exists(local_dir):
                os.makedirs(local_dir)

            logger.info(f"Downloading s3://{bucket}/{obj_key} to {local_file_path}")
            try:
                s3_client.download_file(bucket, obj_key, local_file_path)
            except Exception as e:
                logger.error(f"Error downloading {obj_key}: {e}")
                raise

        # Define the path of the archive
        if not os.path.exists('/opt/bin/7z'):
            logger.error("7z binary not found at /opt/bin/7z")
            raise FileNotFoundError("7z binary missing")
        local_archive = f"/tmp/index_{context.aws_request_id}.7z"
        try:
            subprocess.check_call(
                ['/opt/bin/7z', 'a', local_archive, '.'],
                cwd=work_dir
            )
        except subprocess.CalledProcessError as e:
            logger.error(f"Error during archive creation: {e}")
            raise e

        # Upload the 7z archive to S3
        new_key = prefix + "index.7z"
        logger.info(f"Uploading archive to s3://{bucket}/{new_key}")
        s3_client.upload_file(local_archive, bucket, new_key)

        # Cleanup
        shutil.rmtree(work_dir)
        if os.path.exists(local_archive):
            os.remove(local_archive)

    return {"statusCode": 200, "body": "index.7z created and uploaded successfully."}

Operational Notes

  • Package 7z into a Lambda layer and verify binary path /opt/bin/7z.
  • Set timeout/memory high enough for archive size.
  • Ensure IAM policy allows s3:ListBucket, s3:GetObject, and s3:PutObject.
  • Consider object versioning for safer overwrite behavior of index.7z.

Official Documentation

Leave a Reply

Your email address will not be published. Required fields are marked *

โ˜• Support us ยท ๐Ÿ’ณ Monobank