How to delete the objects in S3 bucket if versioning is enabled?

When versioning is enabled in an Amazon S3 bucket, deleting an object doesn’t actually remove it from the bucket. Instead, it adds a new “delete marker” version to indicate that the object has been deleted. The original object remains in the bucket with its version ID, and the delete marker version becomes the current version.

If you want to permanently delete an object and all of its versions from an S3 bucket with versioning enabled, you need to follow these steps:

1. Use the AWS Management Console, AWS CLI, SDKs, or API to delete the object. This will create a new version, which is a delete marker.
2. To permanently remove the object and all its versions, you have a couple of options:

• Use the AWS Management Console: Go to the bucket, select the “Versions” tab, locate the delete marker version, and select “Delete” to remove it.
• Use the AWS CLI or SDKs: Use the aws s3api delete-object command with the --version-id parameter to specify the delete marker version ID.
• Use Life cycle policy : Use 2 lifecycle policy one for Object deletion for current and concurrent version of object and another for delete marker.

Once the delete marker version is removed, the object and its associated versions will be permanently deleted from the bucket.

It’s important to exercise caution when deleting objects in versioned S3 buckets, as the versioning feature is designed to help retain data integrity and recovery options. Always ensure you have backups or copies of critical data before performing deletions.

💡 Learn more about the AWS Well-Architected Framework:

From Good to Great: Using AWS Well-Architected Tool for Cloud Excellence

What is deletion marker?

In Amazon S3, a delete marker is a version-specific object that indicates the deletion of a specific version of an object in a versioned bucket. When versioning is enabled for an S3 bucket, each object modification, including deletions, results in the creation of a new version.

When you delete an object in a versioned bucket, rather than physically removing the object, S3 adds a delete marker as a new version to the bucket. This delete marker serves as a record of the deletion, indicating that the object with that particular version ID has been deleted.

The presence of a delete marker has several implications:

1. Data Integrity: The use of delete markers helps maintain data integrity. Deleted objects can be restored by deleting the delete marker, which effectively “undoes” the deletion and makes the original object version accessible again.
2. Versioning History: Each delete marker version contributes to the versioning history of the bucket, showing when an object was deleted and providing a way to track object changes over time.
3. Permanently Deleting Objects: To permanently delete an object and its versions from a versioned S3 bucket, you need to remove both the delete marker and the object versions associated with it.
4. Object Listing: When listing objects in a versioned bucket, the delete markers are included in the list of objects returned. This helps maintain consistency in object version listing.
5. REST API Interaction: When interacting with the S3 API, you can specify the version ID of a delete marker or a specific object version to perform actions such as retrieval or deletion.

It’s important to note that while delete markers maintain data integrity, they do consume storage space in your S3 bucket. If you plan to permanently remove objects and their delete markers to reclaim storage space, ensure you follow proper procedures to prevent unintended data loss.

Pre-requisite:

• AWS Account
• AWS CLI
• IAM Permissions
• S3 bucket versioning
• AWS S3 bucket
• S3 lifecycle policy
• JQ installation

By using AWS Console:

Create S3 bucket and enable the versioning.

Now upload the object and select it to delete.

Write delete to confirm the deletion.

You will see the object is deleted.

Now toggle the button to see the version of this object.

you will see here 2 objects one is non current version object and other is delete marker associated with that object.

To permanently delete the object you will have to delete the non current version object.

If you want to restore the object you can delete the delete marker first and then the objects gets restored. Let’s try it out.

Now again delete this object.

Now let’s delete the non-current version of the object.

You will see an option of permanent delete.

Now you just have delete marker left. Lets delete that as well.

Now you see the below object is deleted.

By using AWS CLI:

Step 1 :

Delete the objects based on versions to delete the current and non current version.(With Prefix we will be able to delete all the objects under specific folder)

aws s3api list-object-versions --bucket <BucketName> --prefix <prefix/> --query 'Versions[].[Key, VersionId]' | jq -r '.[] | "--key '\''" + .[0] + "'\'' --version-id " + .[1]' |  xargs -L1 aws s3api delete-object --bucket <BucketName>

Step 2 :

Now after executing the above command successfully we need to delete the expired marker to delete it permanently so use below command.

aws s3api list-object-versions --bucket <BucketName> --prefix prifix/ --query 'DeleteMarkers[].[Key, VersionId]' | jq -r '.[] | "--key '\''" + .[0] + "'\'' --version-id " + .[1]' |  xargs -L1 aws s3api delete-object --bucket <BucketName>

By using lifecycle policy:

To use the lifecycle policy we need to use 2 policy here.

Policy 1 :-

Give the name of lifecycle rule and select 2 rules action

• expire current versions of object
• permanently delete noncurrent version of object.

Now give the days to expire current version of object and noncurrent version object and click on create rule.

Now we need to add the rule for deletion of delete marker after deletion of current and noncurrent version.

Select

• Delete expired object delete markers or incomplete multipart uploads

Select the 2 options and add number of days for multipart uploads and click on create rule.

Now we see that two rules are added.

This will delete the objects permanently.

Conclusion:

We explored three distinct methods for effectively managing object deletion within versioned Amazon S3 buckets. The first method involved the user-friendly interface of the AWS Management Console, which provides an intuitive graphical environment for interacting with S3 objects.The second method showcased leveraging the power of the AWS Command Line Interface (CLI), offering a command-driven approach for quick and precise deletions.

Lastly, the strategic use of lifecycle policies, allowing for automated and rule-based removal of objects based on defined criteria. We delved into each method and it has its unique strengths, catering to different user preferences and operational needs. By providing a comprehensive overview of these approaches.

Please follow me for more such innovative blogs.

Thank you for being awesome!