Recursively delete files from Google Storage bucket

recently I started working with the Google SDK for .NET, just to realize that just like any other “Google Code”, it’s not gonna be easy.

Google Storage uses flat namespace when storing files in a bucket, which makes it hard to recursive delete if you don’t know the names of all the files that share the same “directory-like” root folder.

gsutil hands it pretty easy with a single line of command:

gsutil rm gs://[bucketname]/[foldername]/*

but the SDK don’t offer this flexibility and only allow the deletion of one object at a time.

So what can we do?

The function receives an initialized StorageService, the name of the bucket and the name of the “ROOT” folder.

then we initialize a new ListRequest class object with 4 important properties

1. Prefix – the name of the “ROOT” folder followed by a slash.
2. Fields – what fields should be returned as a representation of the objects in the bucket.
3. Versions – set to True so no objects will be left behind.
4. Projection – set to Full ( means all objects including the ones with no ACL ).

We execute the List into a new Objects class that will store the response.
If there is a match then a foreach loop will iterate over the items in the Objects class and execute a DELETE call on each file. Of Course this can be done with a Parallel.ForEach for a much faster result.

One thing that helped me understand the SDK much better is to know that although the code is wrapped using lots of classes, eventually the code makes a call to the JSON API.

2 Tips for you:
1.Going over the [Google Storage API Docs] is pretty mandatory.
Google also supplies an APIs Explorer where you can understand it more thoroughly.
2. If you have gsutil installed you can run each command with the “-DD” option to debug and watch what the tool does.

3 thoughts on “Recursively delete files from Google Storage bucket

  1. Hi,

    I used your code to delete a bucket with its contents but sometimes I get objects that are not present in the bucket then I get Exception (not found object).
    Hope, you can clarify what I don’t understand.

    • This code is very old and does not align with the current API, so maybe that is the reason?
      If you are using this old API, you can just wrap the delete command with a try catch.

      • Hi Shai,

        Firstly thanks to you for this answer.
        Could you share with me the new code ?

        Thanks you very much

Leave a Reply

Your email address will not be published. Required fields are marked *