Unstructured Blob Storage (Building Real-World Cloud Apps with Azure)

by Rick Anderson, Tom Dykstra

Download Fix It Project or Download E-book

The Building Real World Cloud Apps with Azure e-book is based on a presentation developed by Scott Guthrie. It explains 13 patterns and practices that can help you be successful developing web apps for the cloud. For information about the e-book, see the first chapter.

In the previous chapter we looked at partitioning schemes and explained how the Fix It app stores images in the Azure Storage Blob service, and other task data in Azure SQL Database. In this chapter we go deeper into the Blob service and show how it's implemented in Fix It project code.

What is Blob storage?

The Azure Storage Blob service provides a way to store files in the cloud. The Blob service has a number of advantages over storing files in a local network file system:

  • It's highly scalable. A single Storage account can store hundreds of terabytes, and you can have multiple Storage accounts. Some of the biggest Azure customers store hundreds of petabytes. Microsoft SkyDrive uses blob storage.
  • It's durable. Every file you store in the Blob service is automatically backed up.
  • It provides high availability. The SLA for Storage promises 99.9% or 99.99% uptime, depending on which geo-redundancy option you choose.
  • It's a platform-as-a-service (PaaS) feature of Azure, which means you just store and retrieve files, paying only for the actual amount of storage you use, and Azure automatically takes care of setting up and managing all of the VMs and disk drives required for the service.
  • You can access the Blob service by using a REST API or by using a programming language API. SDKs are available for .NET, Java, Ruby, and others.
  • When you store a file in the Blob service, you can easily make it publicly available over the Internet.
  • You can secure files in the Blob service so they can accessed only by authorized users, or you can provide temporary access tokens that makes them available to someone only for a limited period of time.

Anytime you're building an app for Azure and you want to store a lot of data that in an on-premises environment would go in files -- such as images, videos, PDFs, spreadsheets, etc. -- consider the Blob service.

Creating a Storage account

To get started with the Blob service you create a Storage account in Azure. In the portal, click New -- Data Services -- Storage -- Quick Create, and then enter a URL and a data center location. The data center location should be the same as your web app.

Create a storage acct

You pick the primary region where you want to store the content, and if you choose the geo-replication option, Azure creates replicas of all your data in a different data center in another part of the country/region. For example, if you choose the Western US data center, when you store a file it goes to the Western US data center, but in the background Azure also copies it to one of the other US data centers. If a disaster happens in one part of the country/region, your data is still safe.

Azure won't replicate data across geo-political boundaries: if your primary location is in the U.S., your files are only replicated to another region within the U.S.; if your primary location is Australia, your files are only replicated to another data center in Australia.

Of course, you can also create a Storage account by executing commands from a script, as we saw earlier. Here's a Windows PowerShell command to create a Storage account:

# Create a new storage account
New-AzureStorageAccount -StorageAccountName $Name -Location $Location -Verbose

Once you have a Storage account, you can immediately start storing files in the Blob service.

Using Blob storage in the Fix It app

The Fix It app enables you to upload photos.

Create a Fix It task

When you click Create the FixIt, the application uploads the specified image file and stores it in the Blob service.

Set up the Blob container

In order to store a file in the Blob service you need a container to store it in. A Blob service container corresponds to a file system folder. The environment creation scripts that we reviewed in the Automate Everything chapter create the Storage account, but they don't create a container. So the purpose of the CreateAndConfigure method of the PhotoService class is to create a container if it doesn't already exist. This method is called from the Application_Start method in Global.asax.

public async void CreateAndConfigureAsync()
{
    try
    {
        CloudStorageAccount storageAccount = StorageUtils.StorageAccount;

        // Create a blob client and retrieve reference to images container
        CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
        CloudBlobContainer container = blobClient.GetContainerReference("images");

        // Create the "images" container if it doesn't already exist.
        if (await container.CreateIfNotExistsAsync())
        {
            // Enable public access on the newly created "images" container
            await container.SetPermissionsAsync(
                new BlobContainerPermissions
                {
                    PublicAccess =
                        BlobContainerPublicAccessType.Blob
                });

            log.Information("Successfully created Blob Storage Images Container and made it public");
        }
    }
    catch (Exception ex)
    {
        log.Error(ex, "Failure to Create or Configure images container in Blob Storage Service");
    }
}

The storage account name and access key are stored in the appSettings collection of the Web.config file, and code in the StorageUtils.StorageAccount method uses those values to build a connection string and establish a connection:

string account = CloudConfigurationManager.GetSetting("StorageAccountName");
string key = CloudConfigurationManager.GetSetting("StorageAccountAccessKey");
string connectionString = String.Format("DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}", account, key);
return CloudStorageAccount.Parse(connectionString);

The CreateAndConfigureAsync method then creates an object that represents the Blob service, and an object that represents a container (folder) named "images" in the Blob service:

CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference("images");

If a container named "images" doesn't exist yet -- which will be true the first time you run the app against a new storage account -- the code creates the container and sets permissions to make it public. (By default, new blob containers are private and are accessible only to users who have permission to access your storage account.)

if (await container.CreateIfNotExistsAsync())
{
    // Enable public access on the newly created "images" container
    await container.SetPermissionsAsync(
        new BlobContainerPermissions
        {
            PublicAccess =
                BlobContainerPublicAccessType.Blob
        });

    log.Information("Successfully created Blob Storage Images Container and made it public");
}

Store the uploaded photo in Blob storage

To upload and save the image file, the app uses an IPhotoService interface and an implementation of the interface in the PhotoService class. The PhotoService.cs file contains all of the code in the Fix It app that communicates with the Blob service.

The following MVC controller method is called when the user clicks Create the FixIt. In this code, photoService refers to an instance of the PhotoService class, and fixittask refers to an instance of the FixItTask entity class that stores data for a new task.

[HttpPost]
[ValidateAntiForgeryToken]
public async Task<ActionResult> Create([Bind(Include = "FixItTaskId,CreatedBy,Owner,Title,Notes,PhotoUrl,IsDone")]FixItTask fixittask, HttpPostedFileBase photo)
{
    if (ModelState.IsValid)
    {
        fixittask.CreatedBy = User.Identity.Name;
        fixittask.PhotoUrl = await photoService.UploadPhotoAsync(photo);
        await fixItRepository.CreateAsync(fixittask);
        return RedirectToAction("Success");
    }

    return View(fixittask);
}

The UploadPhotoAsync method in the PhotoService class stores the uploaded file in the Blob service and returns a URL that points to the new blob.

public async Task<string> UploadPhotoAsync(HttpPostedFileBase photoToUpload)
{            
    if (photoToUpload == null || photoToUpload.ContentLength == 0)
    {
        return null;
    }

    string fullPath = null;
    Stopwatch timespan = Stopwatch.StartNew();

    try
    {
        CloudStorageAccount storageAccount = StorageUtils.StorageAccount;

        // Create the blob client and reference the container
        CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
        CloudBlobContainer container = blobClient.GetContainerReference("images");

        // Create a unique name for the images we are about to upload
        string imageName = String.Format("task-photo-{0}{1}",
            Guid.NewGuid().ToString(),
            Path.GetExtension(photoToUpload.FileName));

        // Upload image to Blob Storage
        CloudBlockBlob blockBlob = container.GetBlockBlobReference(imageName);
        blockBlob.Properties.ContentType = photoToUpload.ContentType;
        await blockBlob.UploadFromStreamAsync(photoToUpload.InputStream);

        // Convert to be HTTP based URI (default storage path is HTTPS)
        var uriBuilder = new UriBuilder(blockBlob.Uri);
        uriBuilder.Scheme = "http";
        fullPath = uriBuilder.ToString();

        timespan.Stop();
        log.TraceApi("Blob Service", "PhotoService.UploadPhoto", timespan.Elapsed, "imagepath={0}", fullPath);
    }
    catch (Exception ex)
    {
        log.Error(ex, "Error upload photo blob to storage");
    }

    return fullPath;
}

As in the CreateAndConfigure method, the code connects to the storage account and creates an object that represents the "images" blob container, except here it assumes the container already exists.

Then it creates a unique identifier for the image about to be uploaded, by concatenating a new GUID value with the file extension:

string imageName = String.Format("task-photo-{0}{1}",
    Guid.NewGuid().ToString(),
    Path.GetExtension(photoToUpload.FileName));

The code then uses the blob container object and the new unique identifier to create a blob object, sets an attribute on that object indicating what kind of file it is, and then uses the blob object to store the file in blob storage.

CloudBlockBlob blockBlob = container.GetBlockBlobReference(imageName);
blockBlob.Properties.ContentType = photoToUpload.ContentType;
blockBlob.UploadFromStream(photoToUpload.InputStream);

Finally, it gets a URL that references the blob. This URL will be stored in the database and can be used in Fix It web pages to display the uploaded image.

fullPath = String.Format("http://{0}{1}", blockBlob.Uri.DnsSafeHost, blockBlob.Uri.AbsolutePath);

This URL is stored in the database as one of the columns of the FixItTask table.

public class FixItTask
{
    public int FixItTaskId  { get; set; }
    public string CreatedBy { get; set; }
    [Required]
    public string Owner     { get; set; }
    [Required]
    public string Title     { get; set; }
    public string Notes     { get; set; }
    public string PhotoUrl  { get; set; }
    public bool IsDone      { get; set; } 
}

With only the URL in the database, and images in Blob storage, the Fix It app keeps the database small, scalable, and inexpensive, while the images are stored where storage is cheap and capable of handling terabytes or petabytes. One storage account can store hundreds of terabytes of Fix It photos, and you only pay for what you use. So you can start off small paying 9 cents for the first gigabyte, and add more images for pennies per additional gigabyte.

Display the uploaded file

The Fix It application displays the uploaded image file when it displays details for a task.

Fix It task details with photo

To display the image, all the MVC view has to do is include the PhotoUrl value in the HTML sent to the browser. The web server and the database are not using cycles to display the image, they are only serving up a few bytes to the image URL. In the following Razor code, Model refers to an instance of the FixItTask entity class.

<fieldset>
<legend>@Html.DisplayFor(model => model.Title)</legend>
<dl>
    <dt>Opened By</dt>
    <dd>@Html.DisplayFor(model => model.CreatedBy)</dd>
                <br />
    <dt>@Html.DisplayNameFor(model => model.Notes)</dt>
    <dd>@Html.DisplayFor(model => model.Notes)</dd>
                <br />
                @if(Model.PhotoUrl != null) {
        <dd><img src="@Model.PhotoUrl" title="@Model.Title" /></dd>
                }
</dl>
</fieldset>

If you look at the HTML of the page that displays, you see the URL pointing directly to the image in blob storage, something like this:

<fieldset>
<legend>Brush the dog again</legend>
<dl>
    <dt>Opened By</dt>
    <dd>Tom</dd>
                <br />
    <dt>Notes</dt>
    <dd>Another dog brushing task</dd>
                <br />
    <dd>
<img src="http://storageaccountname.blob.core.windows.net/images/task-photo-312dd635-ba87-4542-8b15-767032c55f4e.jpg" 
           title="Brush the dog again" />
    </dd>
</dl>
</fieldset>

Summary

You've seen how the Fix It app stores images in the Blob service and only image URLs in the SQL database. Using the Blob service keeps the SQL database much smaller than it otherwise would be, makes it possible to scale up to an almost unlimited number of tasks, and can be done without writing a lot of code.

You can have hundreds of terabytes in a storage account, and the storage cost is much less expensive than SQL Database storage, starting at about 3 cents per gigabyte per month plus a small transaction charge. And keep in mind that you're not paying for the maximum capacity but only for the amount you actually store, so your app is ready to scale but you're not paying for all that extra capacity.

In the next chapter we'll talk about the importance of making a cloud app capable of gracefully handling failures.

Resources

For more information see the following resources: