113

Configuring and Using Microsoft Azure Blob Storage

View non-AMP version at androidpolice.com

Course Overview

Course Overview

Hi everyone. My name is Neil Morrissey, and welcome to my course on Configuring and Using Microsoft Azure Blob Storage. I'm a solutions architect, and I'm really excited to introduce you to the blob service in Azure Storage and tell you all about the newest features. Blob stands for binary large object, and blobs can be any kind of file, really, from documents and media files to logs and website files to VM disks and even databases. Storing and managing all that data is a challenge for every organization. Cost-effective storage is a very compelling reason in itself for an organization to move to the cloud, and Azure blob storage delivers on that, but also adds a lot of value on top, with a whole host of features to helps you move data to the cloud, secure that data, and allow users and applications to access that data. And Microsoft Azure Blob Storage integrates tightly with other Azure services to help you build powerful, storage-based solutions. In this course, we're going to examine the features of the blob service. Some of the major topics we'll cover include the basics of blob storage, including the different kinds of blobs and how to upload and secure them in the service. You'll see the latest features of blob storage to make management easier, like soft delete and authentication to the blob service using Azure AD, as well as features that can also save you money, like Azure blob lifecycle management and static website hosting. We'll look at how Azure Blob Storage integrates with Microsoft's content delivery network to reduce latency when serving blobs to users around the world, and we'll go in-depth on how blob storage can be indexed by Azure Search, giving you the ability to search the contents of a variety of different file types, and that can help you create rich search experiences in your applications. The end of this course, you'll know how to set up and use blob storage, how to interact with it from the Azure CLI and from code, and how to configure other Azure services to leverage blob storage. So I hope you'll join me on this journey to learn all about Configuring and Using Microsoft Azure Blob Storage, here on Pluralsight.

Understanding and Configuring Blob Storage

Azure Storage Accounts

Let's start by talking about azure storage accounts, which is the parent container for billing and configuration that a blob services created within depending on the type of storage account chosen, it can contain blob storage, file storage, disk storage, table storage and que storage. Let's quickly go over these services. Blob Storage is an object storage solution that's optimized for storing massive amounts of unstructured data like documents or binary data like image files, video and even virtual machine images. Blob storage is used to store managed and unmanaged disks used by virtual machines, so disk storage is actually a part of blob storage that uses a certain type of flop format called a page blob. We'll talk more about blob types in the next clip. File storage is similar to blob storage in that you can store various types of unstructured data. But file storage supports the SMB Protocol, which means you can attach this storage to virtual machines for read and write access and then file storage, then shows up with a Docker letter on the virtual machine. As a file share. Table storage is a no SQL data store that allows you to store massive amounts of structured, non relational data, so it can be a good choice for web apps, address books, device information or any type of structured information that you're app might need. Que storage is used to store and retrieve messages so it helps in building asynchronous reliable applications that error based on messages. All of these different services share features of the parent storage account and are billed as a group via the storage account at the storage account level. You also make configuration decisions about replication of the data across data centers and even across regions, and this allows you to configure high availability and disaster recovery for your storage account. Data is always replicated three times in the Primary Data Center, and that's called locally redundant storage. In some regions, you can choose own redundant storage to replicate the data within three availability zones in the primary region. And each availability zone is a separate physical location with its own independent power cooling and networking. If you're concerned about the possibility of a region becoming unavailable, maybe due to a natural disaster, you can replicate across two regions using geo redundant storage, Microsoft pairs regions and initiates fail over when the primary region is unavailable. If you want access to the secondary region to always be able to read the data you can choose read access geo redundant storage. And there are also redundancy options that combines own redundant storage with geo redundant storage as well as with read access. Do redundant storage so you have a lot of options when it comes to replication. All the different services in the storage account exposed rest based endpoints to the internet for accessing data and administering the different services. It's through the rest endpoints that the storage client libraries communicate with the blob service and other services. Other clients, like PowerShell and the Azure Lai, also use the rest services as well as tools like Azure Storage Explorer and Ese Copy. You confined the you or else to the end points for a particular storage account from the Properties page for the storage account. There are currently five different types of storage accounts. They could be grouped into standard and premium storage account types. Let's talk about the standard types. First. General Purpose V. Want to counts are the original storage account type. It's an older version that doesn't include all the latest features, but does support blob storage, file storage, table storage and Q storage. It also supports deploying with the classic deployment model, as opposed to the resource Manager model. So you might need this type of storage account in order to integrate with virtual networks and virtual machines that were deployed using the classic model. Blob. Storage accounts only include the blob service and don't include file table and que services. They also have something called Access Tears, which allows you to store blobs in hot, cool or archive tears in order to save on the cost of storing infrequently used data. We'll talk more about this feature later in this module. The General Purpose V two storage account type supports all the latest features for blob storage file table and que storage, including blob access. Tears be to storage Accounts also have lower storage costs than be one accounts, and you can easily upgrade a G P V one account to a GP V to account in the portal or using PowerShell or the Azure CLI G P V two storage accounts are the Microsoft recommended choice when creating a new storage account. G P V one and V P V two storage accounts support blob file table and Q services and storing virtual machine discs. And they support those services on a standard performance here, which means they're backed by hard disk drives. There is a performance to error offered for G P V one and V P V two accounts, which is backed by solid state drives. But this is only available for storing unmanaged virtual machine discs in 2019 to new storage account types were introduced that allow you to store blobs and files on premium storage. A block blob storage account allows the storing of block and upend blobs on premium storage. We'll talk more about the different types of blobs in the next clip, but this provides better performance for scenarios with high transaction rates, using smaller objects or requiring consistently low storage latency. Blog blob storage accounts don't support blob access tears so you can take advantage of the cost savings of cool and archive tears. In terms of storage account replication only locally redundant and zone redundant storage are supported. File storage is another new type of storage account that supports file shares backed by premium SSD storage premium file shares, provide consistently low latency and offer increased performance through new features like IOPS bursting. There's an entire course in this path on configuring Microsoft azure file storage. If you'd like to know more now, in terms of the amount of data you can put into these storage accounts, this is an area that continues to change. So I encourage you to consult Docker dot Microsoft dot com. If you have concerns about hitting the limits of your storage account at time of filming the maximum storage account capacity for G P V, one G P V two blob storage and block blob storage accounts is five petabytes. Notice that it says right here that standard storage accounts can support higher capacity limits. So if you have concerns about hitting those limits, you can contact azure support to discuss increasing your capacity limit. That support ticket might be worth the effort before you start breaking your solution into multiple storage accounts unnecessarily. So in this module you've already learned about azure storage accounts, which are the parent container for the blob service. Next we'll discuss the different types of blobs that you can store, and then you'll learn about blob access tears, which provide you a way to save costs on storing infrequently accessed blobs. After that, we'll discuss the containers that store blobs within the blob services, and then you'll see how to create those containers in the portal and using the azure CLI. Next, we'll explore features of blob storage in the azure portal. Then you'll learn about change feed logs and a way to make blobs immutable so they can't be modified after creation, which you may want for compliance reasons. After that, you learn about blob Lifecycle management, which works with blog access tears to move blobs across the access tears in order to save you costs in an automated way. Then we'll talk about soft elite to provide a recycle bin for your blob service. And after that you'll see how you can host static HTML type websites directly in blob storage so you can save on the compute charges associated with other services like azure app service. And finally, we'll talk about Azure Data Lake Storage Gen two, which is a big Data analytics offering built on top of blob storage. So next let's talk about the different types of blobs supported in the blob storage service

Blob Blobs, Append Blobs, Page Blobs

Now that you understand the storage account that a blob service gets created within, let's talk about the different types of blobs that gets stored in the blob's service. The term blob is an acronym for binary large object, so a blob itself could be any type of file, a document, an image or video, a virtual machine desk or even an entire database. The file extension doesn't matter to Azure. It's just a bunch of bites. But when uploading a file, you do need to categorize the blob as one of three distinct types. And that does affect how Azure treats the blob. Once the blob has been created in Azure, you can't change its type, and it can only be updated using operations appropriate for its blob type. The three blob types are blocked blobs, upend blobs and page blobs, block blobs, error blobs that error broken down into smaller units called blocks. Each block has an ID, and you can create, read or modify individual blocks within a block blob. Now you're probably used to moving an entire file around, and that's typically how you still think about it at the logical level. But the fact that the blob is broken down into blocks really matters when it comes to uploading large blobs efficiently because it's broken down into block sizes and you can define those sizes, by the way, tools like azure Storage Explorer, ese copy and storage client libraries can upload blocks in parallel, and azure will assemble them into the final block. Blob. A block can only have 50,000 blocks, though, so you need to choose your block size accordingly in order to maximize theological size for large files. But breaking a blob down into smaller blocks can help with low bandwidth Internet connections, so it's good to consider the options when you're planning your solution. Append blobs are also composed of blocks but upend blobs error optimized for adding to the blob only. In other words, you can only add blocks to the end of the blob. Updating and leading of existing blocks isn't supported, so why would you need this? Well, logging is the best example. If you've got a log file that's constantly being written to. Storing it as an append blob is not only more efficient, but it also ensures that no one could modify the file so that might help with regulatory compliance. But of course, a person with the Wright permissions could still delete the file. A little later in this module, I'll talk about a feature that prevents blobs from being deleted using policies. It's called immutable storage, and one of its purposes is to satisfy regulatory compliance for archiving data. The last blob type is the page Blob. A page blob is a collection of 512 bite pages that error optimized for random, read and write operations so you can add or update contents by writing pages into the blob page blobs error, actually the foundation of azure infrastructure as a service Virtual machine discs. They're also used to store the data bases for azure SQL database. And again, this is because page blob support fast access to random locations in the Blob. This blob type also makes it easy to migrate. Your existing VM disks to the cloud and page blobs could be stored in either standard storage or premium storage. Premium storage, as I mentioned in the previous clip, uses solid state drives, which makes performance and throughput even better for low latency needs like database storage. Let's briefly talk about maximum file sizes for the different types of blobs. This information has changed over the different versions of this course, so it's good to get the information from the subscription and service limits on docks. Stop Microsoft dot com Until late 2016 the maximum file size for a block blob was 195 gigabytes. That was increased to 4.75 terabytes. Now there's a preview that allows a block blob to be up to 190.7 terabytes in size, and the maximum allowable size of a block within the Blob is currently 100 megabytes. But the same preview allows blocks up to 4000 megabytes. The maximum size oven, a panda blob is 195 gigabytes, and for page blobs, it's eight terabytes. There are features that error common to all blob types. When you update a blob, the changes take effect immediately. You can lease any blob for exclusive right access, like checking it out in a version control system. Once it's least you have to pass in the current lease. I'd to the app I in order to modify the blob or the blocks within the blob. An example of leasing is the way that an Azure virtual machine leases the underlying page blob for its disc in order to ensure that it's managed by a single VM, and you can also create snapshots of blobs so you can make a point in time copy of a blob. And you can even promote a snapshot to overwrite the current version of the Blob so you can keep a history with snapshots and go back to a certain point in time before the blob was further modified. This is a manual process, though. You need to create snapshots yourself, but there's now the ability to enable blob versioning at the storage account level. Azure will automatically create copies of your blobs to capture their state before they get modified. And, of course, you can restore a previous version of the Blob to overrate the current state. Next, let's talk about a feature I touched upon earlier that could help you save on storage costs, and that's blob access tears

Cost Savings with Blob Access Tiers

blob access Tears are a feature of G, P, V two and blob storage accounts that allow you to store blob data in the most cost effective way, depending on how often you need to use that data. Blob access Tears used to be called blob storage tiers, so if you see storage tiers referenced anywhere, they're the same thing. Pricing for azure storage depends on a few factors. There's the cost of actually storing the data at rest, which is called capacity in the Azure calculator. And then there's the cost of app I a parade Asians to read, write and modify the data as well as listing and retrieving the data. There's also egress charges when data is read or downloaded from azure. And that's true for all azure services. Depending on the redundancy options you've chosen for the storage account, you may be storing data in multiple regions also, and there are data replication charges associated with that, too. You can check out the azure pricing calculator to see the most current costs for all these charges, and to estimate your overall storage costs for different scenarios. Glob Access Tears let you organize your data based on the frequency of access and planned retention period in order to save money on storage. Use Cases for data can vary when developing applications to solve business problems. Data might be accessed regularly or data might get uploaded and used for a while and then not read or modified for a long period of time, if ever, or the data could be uploaded right away for archival purposes and left alone until it's ever needed. Blob access Tears give you flexibility in how you store your data so you can maximize cost savings. In azure storage. There are three tiers that you can move blobs between, depending on how your data is being accessed, the hot, cool and archive tears. There's actually 1/4 performance tear, but it's only available in premium block blob storage accounts. The hot access tear has higher storage costs than cool and archive tears, but the cost of data access is lower. You might choose this tier for data that's actively read and written to or staging data that needs to be processed but can later be archived. The cool access tear has higher access costs but lower storage costs. It's for data that you don't plan to access is often, but you still want to have available. Maybe it's longer term data that won't be used for a while or back up and disaster recovery data that you don't expect to use but still want to make sure it's available if you do need it. These archive accessed here has the lowest storage cost but the highest data retrieval cost, and you need to be able to tolerate it, taking up to a few hours to retrieve the data. The data is actually offline when it's in the archive accessed here, but you can still read the metadata, the archive tier confused for long term back apps. Maybe you need to keep raw data after it's been processed, in case it's needed later, like original uncut media files or source data that's gone through a Net l process. Or maybe you need it for compliance, like storing old X rays for health care organizations or old security camera footage. And because the data is actually offline, it could take up to 15 hours to restore it. But there's also a high priority option available when rehydrating blobs from the archived here. This option prioritizes requests over the standard default hydration option, so the rehydration to the hot or cool accessed here could take under an hour, depending on the blob size and the current demand. High priority Rehydration does cost more, though, so you'll usually only want to use it in an emergency situation. Now let's talk briefly about the premium blob storage here. This is a high performance access here that's only available for premium block blob storage account types. Remember, we talked about the different types of storage accounts at the beginning of this module. This storage account type gives you the best performance for workloads that require very fast response times and high transaction rates. It can only store block and depend blobs. It doesn't store page blobs. And if you want to move blobs from hot or cool access tears in a G P V two or blob storage account into the premium blob storage tear, you need to copy the blobs into a premium block blob storage account. The same is true in reverse. You can't tear down from premium storage to standard storage. You need to copy the blobs manually. I know it's a little confusing to talk about premium blob storage alongside the hot, cool and archive tears because the hot, cool and archive tears exist in a G P V two storage account type or a blob storage account type, and the premium blob storage could only exist in a separate type of storage account the block blob storage account type. But it's useful to see all the options for performance and cost. Here in one place, blobs in the hot, cool and archived tears can coexist in the same storage account. They don't actually get moved anywhere when they accessed here has changed. The default blob access tier can be set at the storage account level. When the account is created or afterwards by the configuration screen, you can set it to either hot or cool, with hot being the default and then blobs get their access to your set to the default. When they're uploaded, we can changed the storage tear for a blob right at the blob level, including moving the blob to the archive storage. Here. This is called blob level tearing, blob level tearing, and the archived here are only supported for block blobs. Keep in mind that moving blobs between access tears does incur charges. If you do, move your data to cool or archive tears and then decide to change the data back to the hot access tear you'll be charged for read operations on all that data as well as for data retrieval per gigabyte. Similarly, if you change from hot to cool, you'll be charged for the right operations to move the data. Just something to remember when planning out how to store your data based on usage patterns, you can manage the access tears for blobs in the portal as well as programmatically with PowerShell and the azure CLI. You can also control blob level tearing and archive storage. Using this storage client libraries for .net JavaScript was, but all these approaches still require deliberate effort to change the access to your for a blob. I'll show you later in this module a feature called Azure Blob Storage Life Cycle Management that lets you set retention policies to automatically move blobs from hot to cool and archive tears. And you can even configure a policy to move a blob directly to archive right when it's uploaded

Blob Containers and Public Access Levels

So far, we've talked about storage accounts and blobs, but there's one other construct you need to understand. And that's the container. A storage account. Blob services made up of containers and those containers store blobs. It's similar to a folder in a file system, but containers have their own properties and features. You can't create containers within containers, but you can create folders within a container. In that case, the folder is really just a part of. The path to the blob. Storage account can contain unlimited number of containers and a container construe or unlimited number of blobs. Container names need to be unique within a storage account, but different storage accounts can have containers with the same name. The name needs to be all lowercase and be between three and 63 characters long. It can only have letters and numbers and dashes, but can't start with a dash. And these restrictions error because the container name is part of the URL to the blob. When it's referenced from the internet, containers have properties to and metadata. You can settle lease on a container as well as access policies. But the main feature I want to talk about that specific to containers is the public access level. This has to do with allowing anonymous access to the blobs in the container. By default, the container is set to private, meaning everyone needs to authenticate with valid credentials in order to access the blobs in the container. We'll talk more about security and authentication in the next module, but you may also have scenarios where you want people or applications to access the data in your container over the Internet without having to authenticate. The public access level is actually influenced by two settings, one on the storage account itself and then a setting on each individual container. At the storage account level, you can configure whether or not blob public access is enabled or disabled. This setting overrides any settings at the container level. If you enable public access on the storage account, though more fine grained access control is possible at the container level, there are three public access levels available when you create a container, and they can also be set on existing containers to the private level. I already mentioned is the default, and it allows no anonymous access. The blob access level allows for public read access to blobs only, so the blobs in the container could be read anonymously. But information about the container itself can't be read, and the anonymous clients can't enumerate the list of blobs within the container, so they need to have a URL directly to the blob itself. The third access level is called container, and it allows anonymous readers to view the blobs but also to enumerate the contents of the container. But it's still specific to that container. The client can't enumerates all the containers in the storage account. With this access level, you can set the public access level in the portal or using PowerShell or the azure CLI, or by using one of the storage client libraries or directly via the rest app.

Creating a Storage Account and Blob Containers Using the Azure Portal

Now let's create a storage account and containers using the azure command line interface. The Azure CLI is Microsoft's cross platform cmdlet experience for managing azure resources. You can still do all of this with PowerShell, of course, but in general, the Azure CLI tends to use shorter, easier to remember commands and is easier for most users to pick up than PowerShell. There are a few ways you can use these azure lai. You can download it, which is what we're gonna do shortly. But you can also access the azure command line interface right within the azure portal by clicking this little icon at the top. This opens up a window within the browser, and this is actually called the Azure cloud shell, because I haven't used this before. Within the subscription, it says, I need an azure file share. I could associate an existing file share using the advanced options here, but let's close this and just let Azure create a new storage account for us to use with the azure cloud shell, and it takes a few seconds to create a new storage account and file share. The purpose of the file share is so the azure cloud shell can persist files across sessions. It actually stores a disk image with a home directory that you can access on subsequent sessions. Now, it says, my account has been initialized for Cloud Shell, and it's connecting to a terminal. But sometimes this will hang for a while. So the easiest way to fix this is to click the power button at the top of the cloud shell. Here. This restarts the cloud shell, and the cloud shell will connect you again at the top left, there's a drop-down where we could change the shell from the default PowerShell interface to a bash shell. We'll have to confirm this, and we get brought to a Bash shell version of the cloud shell. So if you're familiar with bash commands, this is a way to integrate those. Let's switch back to the PowerShell interface, and I'll confirm this notice. It says authenticating to Azure. Once we open the azure cloud shell were actually authenticated using the logged in account, so there's no need to re authenticate within the shell here. Let's try running a command just as simple command to list the resource groups in the subscription all azure CLI commands begin with a Z and then the command group, which in this case is called Group for Resource Groups and then the actual command We want to run, which is list I can. See a detailed list of all the resource groups in this subscription. So the azure cloud shell offers you a way to use these Azure CLI or PowerShell directly within azure. But you might prefer to run the commands from your local workstation. Maybe you've got a management workstation that's only accessible by administrators. Or maybe you just prefer working within a desktop interface. In this case, you can download and install the azure Lai on your local machine, so let's do that. First, let's download the Azure CLI, and we'll do that from Doc. Stop Microsoft dot com. As I mentioned, it's available on a variety of platforms, but since I'm on Windows, I'm going to choose that, and I'm taken to the page where I can download the MSC installer. Once it's downloaded, I'll go ahead and say yes to the license agreement and click install. It'll take a minute here for the product to install while it copies all the new files. These azure CLI gets integrated into your command line or bash shell. So once it's installed, we can, access it by opening a new command line. We can. See what commands error available related to azure storage. By using the command ese storage and add the help argument, you can see the different subgroups of commands and, under each of these error, even more sub commands to do things like create containers, upload files and manage the different services. But before we start issuing commands against our subscription, we need to log in. There are a few ways to do that. You can enter your credentials programmatically or use a service principle and on resources configured for manage identities. For azure resources, you can sign in using the managed identity. But let's use the interactive approach by just entering. Ese log in a browser window opens automatically because I'm on Windows 10 here, I can choose an account that's already been used, but let's use another account. I'll use my Microsoft account that's a global administrator for this subscription. I've logged in with this account previously and allowed the browser to store my password, which, of course, isn't recommended for an admin account like this. But I've actually set this account up for multi factor authentication. So after I click sign in, Azure is sending a push notification to my mobile device, where I've got the Microsoft authenticator app installed. I missed the pop up at the top, so let's open up the Microsoft Authenticator app. And here's the sign in approval that was pushed to the app. I'll click a proof and the browser page says, I've logged into Microsoft Azure. So we can close this browser and back in the command prompt window, I'm logged into the Azure Lai, and it's showing information about the subscription my account has access to. It's a good idea to enable multi factor authentication for your administrator accounts, and if you'd like to know how to set that up, I've got an entire course in the Pluralsight Library on implementing and managing Microsoft Azure multi factor authentication. Okay, now let's start using the Azure CLI with Azure storage. I'm just going to clear this green, and now let's begin by creating a resource group. Of course, you could use an existing resource group. When you create a storage account, let's create a new one. Just go over the commands we do this with the ese group, Create command in the parameter name. We give it the name we want to call our resource group and for the parameter location, we have to choose these as your region that we want to create this resource group in. I'm going to choose East us. But you can use any of the available azure regions. And by the way, if you want to know the names of the supported regions for your subscription, you could get them with the command ese account list locations. Now that we've created a resource group, let's create a storage account and we do that with the easy storage account. Create command for the location parameter will put in the same region east us for the kind parameter we're going to choose. The General Purpose V two storage account type will assign the storage account of name using the name parameter and specify to have it created in that resource group that we just created. The skew will be standard lRS and this has to do with replication. In this case, we're going to create it within a single region, just like we did in the portal in the previous clip. It'll take a minute for this to get created, and once it does, we get all the properties of the storage account back. You can see the girls for each of the storage services appear of the top. Next. Let's see how we can use environment variables to store connection and authorization information. This isn't required, but it can save some time by minimizing the amount of parameters needed in each request. We could store the storage account name and Keon Environment variables, but let's get the connection string for the storage account and will use it to authenticate to our storage account in subsequent requests. And we do that with a Z storage account show Connection String, providing the name of the storage account and the resource group. We get back some Jason with the connection string in it, and we need to copy that. We're gonna assign this connection string to an environment variable so that going forward we won't need to adit to each of the commands on Linux. We would do this with the export keyword, but since I'm on windows, I use the set command and the environment Variable were setting is called azure Storage Connection String, so let's paste in that connection string. Next, let's create a container using the azure CLI. We do that with the ese storage container, Create command and for the name parameter, we'll just call a container one. If we didn't have the environment variable set, we would need to type the account name and specify the Auth mode as log in in order to leverage the azure a D role based authentication. But let's just let the environment variable. Take care of the connection and authorization information, and we get back Jason indicating that it's been created. Next, let's create another container and we'll call this one container to, and we'll set the public access level during creation, this time using this parameter, and we'll set that public access level to blob. Now let's go back to that first container and let's change the public access level. We do that with the easy storage container set permission command. We give it the name of the container and the public access level that we want to change it to. In this case, we're gonna change to the container access level, and you can see we get back the last modified date along with any tags, so this indicates that it was successful. Now let's go back into the portal and take a look at what we just created. I'll start by opening the resource group that was just created. Open up the new storage account and let's go into the blob services. So we've got the two containers I created, and you can see the public access level on both of them. So that's how you create a storage account and containers using the azure CLI.

Creating Containers Using the Azure CLI

let's create a new storage account in the azure portal. We'll start by opening all services and searching for storage. I'll click on storage accounts to open the list of existing storage accounts. You might not have anything in your list yet, so let's click. Add to create a new storage account. First, we'll need a resource group that will contain the storage account I could select from the existing resource groups in my subscription. But let's create a new one. All we need to do is give it a name, and it will be created in the same region that we select for the storage account below. Now let's give this storage account a name, and this name needs to be unique across azure because it will be part of the URL to the rest endpoints for the storage account as your verifies that the name of typed is unique. So now let's choose a location for the storage account. When choosing a region, think about what other services might be accessing the storage account. If you've got VMS in the east US region that need to access the storage account, then you'll probably want to create the storage account in that same region to keep latency to a minimum when accessing it. Let's choose Canada Central, because that's my closest data center. The performance Tear defaults to Standard, which means our storage account will be backed by standard hard disk drives with that option selected we can. Choose general purpose fee to G P V one or two. Create a blob storage account and for any of those storage account types, we can. Choose the replication type as locally redundant. Do redundant read access Geo redundant. And because this region supports it, we can also choose zone redundant storage. If I change the performance. Tier two Premium. We're told at the bottom that the G P V two storage account kind will only support page blobs and in the account kind drop-down, we no longer have the blob storage account type, but we have the newer premium account types available, block, blob storage and file storage. And with the premium performance, tear selected we can only enable replication within the azure region we've chosen. So let's leave the performance tear as standard and create a new G P V two storage account with locally redundant replication, and we'll leave the default accessed here for uploaded blobs as hot as we expect these blobs to get access frequently on the next tab, we need to choose the connectivity method. The default is for the storage account to be available to the internet, but you can, of course, still require callers to authenticate public endpoints. Restricts the access to only azure virtual networks that you specify here. And this is to only allow virtual machines and other resources on your veena. To access the storage account, a private endpoint creates a private I P address for your storage account on your veena. Using azure private link. Traffic then travels over the Microsoft backbone, eliminating exposure to the internet. At the bottom are network routing preferences. This is currently in preview and not available in this region. This feature allows you to specify how network traffic is routed to your account from clients over the Internet. On the next tab, there are options to configure point in time restore for containers. This feature relies on soft elite change feed and blob versioning, all of which we'll talk about later in this module, there are options to enable soft elite for blobs and file shares as well as a great out option here for containers, which is currently in preview. We talked about blob versioning earlier in the module, and there's a clip on the change feed blog later in this module on the advanced tab we can disable Secure transfer by default. The storage account endpoints for the different services are only available over https. But we can allow http from here we can disable blob public access, which means we won't be able to specify the access level on the individual containers that will be overridden for the entire storage account. From here. We can specify the TLS version for accessing the storage account and at the bottom is the option to enable hierarchical name space support in order to use this blob service for Azure Data Lake Storage, Gentoo. We'll talk about that later in this module to on the next tab. We can attach tags to this storage account, and these can help with billing and automation scripting later. But these kanno also be added after the storage account is created. So let's go to the review page and click create the storage account takes a few seconds to get created, so I'll speed things up by pausing the recording. And now that the storage account is created, I'll click Go to Resource. But we could navigate their from the storage account list, like at the beginning of this clip. Inside the storage account on the overview page, you can see the location, the storage account type, the replication type and the performance and access tiers below that are the services in this storage account and we can get to those from the menu on the left also, so let's scroll down to the Blob Service grouping and click on containers. We don't have any containers yet to store the blobs, so let's create one. I'll give this container a name and I can choose a public access level during creation here, but let's leave. It is private for now. Under the advanced section, you can choose an encryption scope for this container. We'll take a look at encryption scopes in the upcoming clip on exploring blob features, but basically this lets you keep the data from different customers in the same storage account by encrypting different containers with their own encryption key. I don't have any encryption scopes created here yet, though, so this container will just use the encryption key for the storage account. I'll click create, and this container gets created. And once it's created we can, modify the properties from the drop down list on the right and I can modify the access level from here. Also, by clicking change access level. Let's change the access level to Blob, and I'll okay, this change. You can see the public access level was updated in the containers list. Let's open up this container. We don't have any blobs in here yet, of course, but we'll be uploading some shortly. Next, let's see how to create containers using the azure CLI.

Exploring Blob Features

Now let's upload a blob using the portal interface and explore some of the features of blob storage. I'm inside the storage account we created earlier. I won't go into the configuration options on the storage account because I have an entire course on that subject. But one thing I do want to show you is the encryption tab under settings. This is where you can specify how you want the blob and file services encrypted. You can either use Microsoft managed keys, which is the default or we can. Specify customer managed keys. Those keys get created and stored inside an azure key vault that you control. Another option for encryption is to create an encryption scope. This lets you keep data from different customers separated by specifying the encryption key used to encrypt individual containers and even individual blobs. I have a couple of scopes I've already created, and they could be disabled and hidden, and it's easy to create a new scope. And here again, you have the option to use an encryption key that's managed by Microsoft or manage your own key in azure key vault. Instance within your subscription. Okay, let's close this and let's go down to the blob containers and let's open up this container we created earlier notice near the top. It says these authentication method we're using to access the storage account is with the storage account access key. These Azure Pluralsight requests to Azure storage using your log in credentials. So this interface is telling us that we're authenticating with the access key of the storage account and because the administrator account I'm using has privileges to read that key. That's why I can see the contents of the blob container. It says I can switch to using the privileges of my azure a D user account. I can do that because I've assigned a certain role to my account. Otherwise I would get an error. Let's look at the access control tab for the container. In order to use Azure 80 account access, your account needs at least reader privileges to the storage account or higher, which I have. But you also need one of the new storage account specific roles. I've granted my azure idea Count Storage blob. Data owner and I've also granted this account the storage Account contributor role, which allows this account Theobald ity to access the storage account access key. If I close this and add a new role assignment, you can see the storage blob data roles. You'll need storage blob, data contributor, owner or reader in order to access the blob data. If you choose to use this azure aid method instead of the account key, so why would you want to use this approach over just accessing the account with storage account key? Basically, it's about operating on the principle of least privilege with Azure A D account access. You're giving a user access to this resource and not the whole storage account, which happens when they have access to the storage account key. And that access isn't just for the azure portal. It extends to azure storage Explorer, the azure, CLI and PowerShell. Basically, anything that accesses the storage account rest APIs. Of course, you can limit the scope of access using shared access signatures. But assigning these new roles to azure AD FS APIs these access to a particular user and allows you to revoke that access in an easier way to now. Let's look at managing the blobs in this blob container. There's a variety of programmatic ways to upload blobs, but you can also just do it simply from the azure portal interface here by clicking upload at the top of the screen. So I'll choose a file on my local hard Docker, and we could upload This is a block blob without doing anything else. But let's take a look at what's available in the advanced section. And again we can choose these authentication type, which I'll just leave as Thea's your a D user account. We can choose whether we want this file to be a block, blob and upend blob or a page blob. We covered what those different blob types mean earlier in this module, but it's important to know that no matter what the source file, you can control how azure storage treats it. By making this selection yourself for block blobs and a pen blobs, you can choose the block size block. Size can help you manage the upload of larger files over networks by uploading multiple smaller blocks in parallel to decrease the upload time. But remember that a block can only contain 50,000 blocks, so if you've got a large file, you may want to use a larger block size in order to maximize the file size. That's possible. I'll just leave the default because this is a pretty small file and doesn't really require that kind of consideration. I have the ability to add blob index tags here during upload. This is a feature that lets you add name value pairs of strings to blobs so you can categorize and search for the blobs later using these tags. This feature is currently in preview, and I've created this storage account in a region that supported for this feature during the preview, we're going to look at blob index tags in detail later in this course, and at the bottom, you can choose an encryption scope for this blob. The default is to use the same encryption scope as the parent container, but you can use a different scope for a particular blob, so you might need to encrypt data from different customers within the same container, and this allows you to provide a security boundary using encryption. I'll go ahead and upload this document, and now let's take a look at some of the options in the drop down list for this blob we can view or edit the Blob, which for block blobs allows us to see the contents, and I'll show you that in a minute. We can Download the file or look at its properties. Let's take a look at the properties first. We've got a URL to the blob at the top. So depending on the public access level of the container and the authentication of the caller we can expose this blob to the internet. There's some audit information like the created in last modified times. At the bottom are the metadata for the blob, and the blob. Index tags will examine both of these features later in the course, When we look at searching blobs at the top, we can see versions of the Blob. This is a feature you can turn on at the storage account level, and this allows the Blob service to automatically create versions of your blob when it changes. Another option is to manage versions yourself. Using a feature called Snapshots. We'll look at snapshots shortly for simple blobs like this text file. You can actually edit the contents of the blob right here in the portal. Of course, for things like binary objects, this isn't possible, and from here in the blob properties, you can generate a shared access signature to provide fine grained access control on the Blob without granting an azure a D roll to the user. We'll look at this in the next module we can change the storage access tier of the Blob and we can. Choose from hot, cool or archive tears. If we know this file is going to be accessed less often, this is an opportunity to save storage costs. For now, let's close out of the screen and go back to the blob list. Let's create a snapshot of the blob and let's open up the view of all the snapshots that have been created. I mentioned earlier that you can actually edit blobs from this portal screen, so let's do that and I'll save the changes now. If anyone were to open this blob, they would see this text because this is the published version of the document. But if I go back to the snapshot that I created before these changes and choose to promote it, I've just made the snapshot into the current version of the document. And then if I open the blob for editing again, you can see my previous changes, error gone and it's being reverted back to the copy that was created as a snapshot earlier. So those error, some basic features of blobs. You could manage all of these from the azure CLI or PowerShell, too. But I wanted to illustrate the concepts by showing you here in the portal.

Blob Change Feed Logs

blog Change Feed is a feature that provides transaction logs of all the changes that occur to blobs and the blob metadata in your storage account. It's a read only log that's ordered durable and mutable, meaning that the logs can't be altered once they're written. The logs error stored is upend blobs in a container within the storage account. Now, Azure storage accounts already have extensive logging features through something called Storage Analytics. You can turn on storage analytics for the Blob service, as well as for tables and queues, storage analytics, blog reads, modifications, deletions and even failed requests. The logs error stored in text files in a container within the storage account itself. The text files are just CSV files. In my course on creating and configuring Microsoft Azure storage accounts, I show you how you can download the storage analytics logs using PowerShell, convert them into Jason and upload them into a log Analytics workspace blog change Feed logs are different than storage analytics. They also get stored is blobs within a container in the storage account. But the main differences are that they error guaranteed to be ordered, whereas logs and storage analytics don't guarantee ordering change. Feed logs also only log, blob creation, modification and deletion, so you don't have to filter out all the read operations or failed requests that you'd see in storage. Analytics logs change feed. Logs are also read only, So new entries kanno only be appended to a log change. Feed logs error designed and optimized for application development, where you want to see chronological changes to your blobs and change feed logs. Error. Stored in the Apache Avro format, this is a compact binary format that provides rich data structures within line schema. The format is widely used in the Hadoop ecosystem, so it's understood by Stream Analytics and Azure Data. Factory Analytics. Applications like Apache Drill or Apache Spark can also consume Afro files directly. You can also access the logs using the change feed process er sdk, which is a new get package for DOT Net. There are also client libraries available for JavaScript and python. The SDK allows you to read the logs without having to parse the Avro formatted files. Now something to keep in mind. Changes to your blobs aren't logged instantaneously to the change feed. It could take up to a few minutes for logging to take place. So if your application needs to respond quicker than this, you could use blob storage events instead. Blob storage events error pushed to azure event grid. Once the event is picked up by event grid, you could respond with built in connectors in azure functions or logic apps. Or build your own handler with blob change. Feed change records get appended within a few minutes of the change. It's more about being able to store, audit and analyze changes to your objects over time for security or compliance reasons. You can also use change feed logs to extract business analytics, insights and metrics about the changes to your blobs or for more content management related scenarios like updating a secondary index to the blob stored somewhere else or synchronizing with a cash or a search engine, you enable change feed, blog support in the portal or using PowerShell. You could enable change feed during account creation or on an existing storage account from the data protection tab. Under the Blob's service menu. There's one change feed in each storage account, and it's stored in a container that gets created called blob change feed all creates updates and elites for the entire blob services are captured. The client applications error responsible for filtering out what they don't need. Only G, P V two and blob storage accounts. Error supported for change. Feed and storage accounts with hierarchical name space enabled aren't supported either. Those error accounts that error configured for Azure Data Lake Storage Gentoo, which we'll talk about later in this module. So change feed logs are immutable, meaning you can't modify the files. Next. Let's talk about another feature of blob storage that lets you make any blob container in your storage account, immutable so data can't be modified or deleted.

Immutable Blob Storage

some industries error required by regulations to keep data in immutable storage, meaning it can't be modified or erased for a certain retention period. It's particularly common in the financial services sector, but also needed in health care, legal insurance and public safety sectors. This type of storage is called immutable storage. It's also known as write Once, read many or goes under the acronym Worm. Even if you're not subject to regulations requiring it, immutable storage can prevent modification or deletion of any data you want protected within your organization. You configure this at the container level. There are two types of configuration. Time based retention, which allows you to specify the period of time that the data is immutable for and legal hold retention, which keeps the data immutable until the hold is cleared. There's also an audit logs stored that shows when the hold was set and by whom there's no additional charge to enable or use immutable storage. It's just a feature that's been added to blob storage. Immutable storage is available in all azure regions for G P V one GP V two blob storage and block blob storage account types and immutable storage is available in all blob tears. Hot, cool and archive. Let's look at how to enable it within the azure portal. I've got a G P V two storage account that I created earlier, so I'll open up the Blob services and there's a container here. If I pull down the menu on the right for the container and choose access policy, this menu pops up. There's a section here for configuring, immutable blob storage. So let's add a policy. You can see we've got two options in the policy type drop-down list. Legal hold to lock the container contents until the hold is cleared and time based retention, which lets us set the number of days that the data will be immutable for for legal holds. We can also attach tags to the data. But let's create a time based policy and set it so the data can be updated or deleted for 30 days. Now our policy is created, but because we chose the time based policy, it's not actually effective until we lock the policy and we do that from the drop down list for the policy. I'm not gonna actually lock it because there's no way to undo that until the policy expires. But let's look at the policy audit here. It shows the retention period that was set as well as the date. We can also edit the policy in order to extend to the retention period if it's set to expire. So that's a feature of azure blob storage that lets you mark data as immutable and can help you with regulatory compliance.

Azure Blob Storage Lifecycle Management

Earlier, we discussed blob access tears, which allow you to put blobs that error frequently accessed in a hot storage tear where storages more expensive. But the cost of accessing the data is lowest, and then, when the data is less frequently used, you can choose to move it to the cool tear or even the archived here, where storage is the least expensive but access costs more. It's a great feature that can help you manage costs in azure storage, but it requires you to initiate that change of access here for your blobs. Azure Blob Storage Life cycle management lets you define rules about when your blobs should be moved between access, tears and even when those blobs should be deleted. Azure Blob storage Life cycle management can be applied to both G P V two and blob storage accounts, and the rules comply to containers or to a subset of blobs by using prefixes as filters to identify the blobs that should be affected or by leveraging blob index tags for filtering. We'll look at blob index tags in detail later in this course, when we examine searching blobs, the rules error executed once a day at the storage account level. Let's see how to configure this in the portal. I'm inside a storage account. If I scroll down to the blob storage section of the menu, I'll click on life cycle management. There are two options here. You can view rules in the list view or use the code view. You can edit the rules in Jason Format here, but let's switch back to list view, and let's create a rule using the designer First, you need to give the ruling name, and then you specify whether the rules should apply to all the blobs in your storage account or to specific blobs by using filters. And we'll specify those filters on another tab. Then you choose whether the rule applies to block and or upend blobs and whether the rule applies to the base blobs or also to the snapshots. On the next tab, you create the rule. The rule is an if then statement. Based on the last modified date of the blob, you can choose to move the blob to cool storage, to archive storage or delete the blob. Let's choose to move the impacted blobs to the cool access here you can add additional actions here also like the leading the Blob after another set number of days. Let's remove this, though, and just keep the action to move the blobs to the cool access Here. On the next tab, you can add filters based on a prefix like a file path to the blob. Remember, folders error religious part of the path name to the blob. You can also use a blob index tag to filter blobs that this rule will apply to you. Attach blob index tags during upload or after blobs have been created. Let's add this rule now that the rule has been created. If you select it, you can see the Jason representation of the rule in the code view, and you can modify it here also. So that's how to create life cycle management policies to move blobs between blob access tears and delete blobs based on rules

Soft Delete for Blobs

soft delete is a feature of blob storage that protects your data from being accidentally or intentionally deleted. It's basically a recycle bin for your blobs. You consider retention period for deleted blobs, during which time they could be under leet id back into the storage account. The retention period can be from 1 to 365 days after the retention period expires, the blob gets permanently deleted. Soft elite applies to block blobs, page blobs and depend blobs. It's available for unmanaged discs, which error just page blobs, but it's not available for managed disks. And soft elite is available for G P V two storage accounts, G P V, want to counts and blob storage accounts. There's also currently a feature in preview that provides soft elite at the container level. This gives you the same functionality for containers and all of their contents. Soft elite for blobs is disabled by default, so let's see how to enable it for an azure storage account. I'm inside a storage account, and if I scroll down the menu to the Blob's service section, I'll click on data protection. This is where you can enable soft elite for blobs When you check the box, you need to specify your attention. Period. Let's set this for 14 days, after which the blobs will be permanently deleted. Okay, I'll save this. Now let's go into containers and open a container, and there's a folder here containing some blobs, and let's select a blob and delete it. I just need to confirm the deletion. Now the blob is removed from the list, but you can view deleted blobs that error still within the retention period by toggle ing this box that shows the blob with a red status icon to indicate that it's in a deleted state. If I click on the Blob, the overview shows that the blob status is deleted, the date and time that it was deleted and the number of days until it gets permanently deleted, which is defined by the retention period that we set. You can also bundle eat the block from here, so that's how to configure soft delete for blobs in your storage account.

Static Website Hosting in Azure Storage

this last feature I want to show you is these ability to host static websites within azure storage so you can host HTML, JavaScript, CSS and images basically any files that don't require any kind of server side processing. What this means is that for static websites, you get highly available websites without having to incur compute charges by hosting an azure app service or on an eye as virtual machine. When you enable static website hosting, a default container gets created for you in the blob storage of your storage account. You need to specify a default document and an error page. Static websites integrate with azure cdn and traffic manager. So if your storage account is set up for do redundant storage, there will be a secondary web endpoint also, and you can even configure accustomed domain to point to the static website using the existing support within azure storage for custom domains. So let's take a look at how to set this up first. Let's look at the blob's service I have in my storage account. You can see there are currently no containers. Now let's enable static website hosting by scrolling down the menu and clicking on static website. We need to enable it and then specify a default document as well as an error document. These documents don't exist yet will need to upload our own. So let's save these changes and then copy this URL for the static website so that we can test it shortly. Now I'll go back to the root of my storage account and select the Blob services. There's still no containers showing, so let's refresh this. And there's the new dollar sign web container that was automatically created. Let's open that up. And as I said, those default documents aren't automatically created, of course, because this is your website. So let's upload some simple HTML pages. I've got a couple of pages I already created for a really simple example of a website, so I'll choose to upload these HTML files, and I'm also gonna upload that error page and I indicated on the previous screen that it would be in a sub folder so we can create that folder during the upload by naming it here. And that's really all there is to setting up the static website. So let's open up a new tab and navigate to the URL that azure storage created for us. Notice that this isn't the URL for the blob service itself. We still have that original end point. This is a new URL specifically created for the static website that's in the dollar sign web container. I'll paste that you are Ln and here's my super Simple Website. Remember, you can upload any kind of website as long as you don't need to host any server side processing like a s p dot net or an asp dot net core website. Those still need to be hosted in app services error on a VM. But for something like a single page application that calls out to another back end service for retrieving data, for example, this is a really great way to host your site without incurring compute charges. Before we finish with this, I just want to show you how you can set up a custom domain for this site. This feature has actually existed for a while for azure storage and we can leverage it with static websites too. So let's go back into the portal and scroll down to custom domain in the Blob services. This is where we enter the custom domain that we own, but DNA's needs to be set up with our domain register. Our first there's actually a couple ways you could do this here. The first way is to just create a C name record with the registrar and enter the domain name here, and that can incur some downtime, while Azure verifies that you actually own the domain name. The other way is to create a C name record to a _____ sub domain that starts with as verify and that lets Azure validate the ownership of your domain before you actually cut over your DNA's. We'll just go ahead with the first method here, though, as it's a little simpler, I'll switch over to my DNA's registrar and scroll down to where I can enter a C name record. I'll create a sub domain on my example load dot com domain and call it test dot example load dot com. Then I need to point it at the URL that azure provided that I still have copied on my clipboard. I'll remove the https and the trailing forward slash, and I need to add a period at the end because we're redirecting to another domain so I'll go ahead and save these changes and now I can switch back to the Azure portal and enter that sub domain here. Now, when I save the changes, I get a confirmation that the changes have taken effect. You might get an error here because the DNS is not propagated throughout the internet right away. If so, just wait a little bit and try again. Now let's open up a new browser window and navigate to the custom domain name test, dot example load dot com and were taken to the website. Something to keep in mind is that azure storage currently doesn't support certificates for custom domains, so we can't serve up. The site is https yet, but we can actually accomplish this by using azure cdn in front of the storage account URL. We'll take a look at Azure Cdn later in this course

Azure Data Lake Storage Gen2

Azure Data Lake Storage. Gen two is a set of capabilities for Big Data Analytics, and it's built on top of azure blob storage. You've probably heard the term, but what is a data lake? It's simply a large repository of data stored in its natural form. The data can be in any file format. It might contain structured data from relational databases, semi structured data like CSV XML and Jason Files, or unstructured data like documents, pdf's and emails, as well as binary data like images, audio and video files. A data lake allows you to store and manage massive amounts of data in a single location. A data lake can be a source for Advanced analytics as well is used for reporting visualizations and even is the data source for machine learning. So the Data Lake can contain both raw data as well as data that's being transformed for purposes like machine learning. There was and still is Gen one version of Azure Data Lake storage that doesn't use blob storage as its underlying file storage mechanism. Both versions have features that make Big Data Analytics possible. Mainly, they use a hierarchy of directories for efficient access by analysis tools, along with directory and file level security. By building Gentoo on top of azure blob storage, it gets the benefits of low cost tiered storage with built in availability and disaster recovery capabilities. When you're creating a storage account, you can choose to enable Azure Data Lake Storage Gentoo, and that's done by enabling hierarchical name space. This is really the key to plain old blob storage becoming azure Data Lake storage. Gentoo, enabling hierarchical name space support, organizes the objects and files into a hierarchy of directories for efficient data access. Blob storage normally uses slashes in the path to the blob name to mimic a directory structure. If you look at the blobs in the portal, you see containers and folders and even sub folders. But these error just implied from the URL path to the blob by inserting the directory names between slashes in the path. When you enable hierarchical name space support on your storage account, this structure becomes a real directory structure. What that means is that operations, like renaming or deleting a directory, become a single atomic operation with regular non hierarchical blob storage. The directory structure is just implied by the slashes in the path of the blob, so it would essentially mean renaming all of the individual blobs in order to change a folder name that's not very efficient. And it's not an atomic operation. It requires a numerator and processing all the objects that shared the same prefix for the directory. So this improves performance when dealing with massive amounts of data. It also makes management easier because you could manipulate multiple files through directories and subdirectories, and it makes security more enforceable because you could define permissions on directories or individual files. These permissions are called PAS ax permissions, which stands for portable operating system interface. PAS X is a family of standards for compatibility between operating systems, and it's traditionally used by UNIX based systems. It's an alternative to access control lists typically used in Windows systems, but a C L's error still supported. Since big data solutions like Hadoop have traditionally been built on UNIX or Linux. Using posits permissions for azure Data Lake storage, Gentoo lets you use those tools on your blob data. In fact, you can manage in access data just as you would with a Hadoop distributed file system. There's a new driver for accessing Gentoo storage, called a BFF's or the Azure Blob file system driver. It's available in all Apache Hadoop environments, including Azure, HD Insight, azure data bricks and Azure synapse analytics. So all of those analytics tools can be used to operate on the data in Azure Data Lake Storage, Gentoo. There are a new set of rest endpoints to your storage account that error specific to Data Lake Storage jen to the new rest. APIs are surfaced at the name of your storage account suffix with D efs core that windows dot net. But you can also use these existing blah breast APIs to operate on the same data in the storage account that has hierarchical name space enabled. This is called multi protocol support, and it means existing tools and applications that use the blob app. I can get the benefits of Data Lake Storage Jen to like directory operations and posits compliant access control lists. And those existing applications don't need to be modified. Now. You may be thinking, Why not just turn on hierarchical name space support for all my GP V two storage accounts? Well, it's because not all of the features of blob storage are available for storage accounts with hierarchical name space turned on. But more features are becoming supported all the time. There's a page published in Docker Microsoft dot com that outlines what blob features are supported and which ones aren't. Some here are in preview, like static websites, immutable storage and container soft elite. And some still aren't supported yet at the time of filming, like blob soft elite, customer provided encryption keys and custom domains on blob storage endpoints. So before you decide to enable hierarchical name space support on your storage account in order to turn it into Azure Data Lake storage Gentoo you might want to consult this page to make sure there aren't any features here unsupported that you need for your solution. There's another page here that shows you what Azure services Inter operate with Data Lake storage. Gentoo. You can see here that azure data factory has supported, along with event grid logic apps I O T hub Power BI I and Azure Cognitive search. But a time of filming these Azure content delivery network is still not supported. Later in this course will be looking at the azure cdn to publish blob data for high performance cashed access all around the world. So be aware you won't be able to do that if hierarchical name space support is turned on within your storage account. If you're creating a storage account to use as a data lake, these limitations error probably fine. You can always create another storage account to use for other business solutions where these features and interoperability are needed. Just be aware that hierarchical name space support isn't just a feature to add on to every storage account. There are tradeoffs involved in turning it on within your storage account. Big Data Analytics is a topic unto itself, so there won't be much more information on Data Lake storage. Gentoo In this course, I just want you to know what it is, why it uses blob storage and how turning it on affect all the features that we discussed in this course.

Module Summary

So in this module you learned about the main features of blob storage in azure storage accounts. First we discussed azure storage accounts. Then you learned about the different types of blobs supported. Next, we talked about blob access tears, then containers. You saw how to create containers in the portal and using the azure CLI. Then we explored features of blobs in the portal. And then you learned more about individual features of blob storage like change feed logs, immutable storage, life cycle management, soft elite and static website hosting. And finally, we talked about enabling hierarchical name space support for Azure Data Lake Storage Jen to in the next module, we're going to look a uploading content to azure blob storage. This will include the different authentication options, as well as tools available to optimize up loading and different programmatic approach. Is that error available?

Uploading Content to Azure Blob Storage

Introduction

when it comes to upload ing blobs to azure blob storage, there are a variety of scenarios that are supported by the free Microsoft tooling In the last module, we uploaded some blobs using the interface in the azure portal. When you need to upload files regularly or upload large image files or delegate uploading to another user or even an application, you don't want to be creating credentials in the azure portal for those purposes. So in this module we're gonna look at the various options that are available for uploading content to azure blob storage. We'll start out with an overview of the different authentication options available. If you're not making the blobs available for anonymous access using the public access levels we examine in the last module, then you need a way to authenticate users and there are a few different approaches. Then we'll look at the tools and libraries available for the simplest experience. There's a graphical user interface called Azure Storage Explorer. When I say it's a simplest, I just mean that it doesn't require any scripting or programming. It's actually quite powerful and flexible as your storage explorers a free download and is available for Windows, Mac and Lennox. It's a desktop tool that provides a window into your azure storage account and lets you interact with content in the blob file table and cue service is, as an administrator, you can do some limited management of the service's to button area. Where it's really useful is in allowing limited access for non administrative users. You can provide them with a shared access signature that gives them limited privileges within the blob service in terms of which containers they can access, whether they can just right or just read and how long the access is good for. So this could be really useful to allow clients to upload blobs manually as part of a business process. For more advanced scenarios, there's a command line interface tool called a Z copy that's available for Windows, Mac and Lenox. This tool lets you upload files from a source directory or move files between storage account locations within Azur. Besides being optimized for uploading blobs, a synchronously ese copy has additional features available using command line parameters for an azure storage administrator, easy copies and must have tool. Of course, you may need to upload is part of your scripting activities, so we'll look at how to upload using the azure sea ally. I didn't include a clip on Azure Power Shell, but there is a Power Shell script in the course downloads that will show you how to upload files to blob storage. The last technology I'll show you in this module is one that allows you to integrate uploading into custom applications by using the storage client libraries, thes code libraries, rap calls to the azure storage, rest a p I and provide a layer of abstraction that makes developing applications easier. There are storage client libraries available for a number of different programming languages, including Java, Python, Go, Ruby Know JS, php, JavaScript and, of course dot net. I'll show you how to use the dot net storage client library in a Web app that uploads files to a blob container and will use storage account keys and shared access signatures in that demo. Then I'll explain how you can leverage Asher A D to authenticate your application code to azure storage, and you'll see a demo of how to use a managed identity for azure resources to get access to azure storage. Using the identity of the APP service without having to store any credentials. This is a great new feature that makes your code more secure and portable. Before we look at examples of uploads, though, let's get started with a review of the different options available for authenticating to the azure storage blob service.

Authenticating to Azure Blob Storage

when I uploaded blobs in the last module using the azure portal, I had access to the management plain, meaning I could access the overall storage account from an administrator perspective, which allowed me full control over the administrative activities because I created the storage account. I inherited permission within the portal to access the data, but only from within the portal. And I could only access the capabilities that the portal has to offer. When we're trying to access data in the storage account from Tools and SD case outside of the portal, we're actually working against what's called the data plane, and we access the data plane with tools like Azure Storage Explorer Easy Copy Power Shell, the azure sea ally in the storage client libraries thes require a different set of credentials. There are three ways to authenticate to azure blob storage. You can use one of the two storage account keys. You can use a shared access signature, or there's a newer approach that allows you to use role based access control with identities in azure Active directory are back with Azure A D allows you to assign roles to Azure 80 accounts to allow access to azure storage blobs and cues. This feature became generally available in March of 2019. Let's look at each of these individually. We'll start with the storage account keys, which are the master credentials for a storage account. Each storage account has to storage account keys, thes air, 512 bits, strings that air the master keys to the account, and each key individually provides all the permissions a user needs to manage the data and have administrative control of the storage account. There are two strings because you may need to regenerate thumb if an administrator leaves your organization, for example, or if the keys become compromised. For some reason, having two keys allows you to limit the impact on users and applications using one of the keys while you transition everyone to the newly generated storage account keys. You can use these keys to get full control over the storage count, which may be necessary for some administrative tasks. But typically you wouldn't want to hand these out unless absolutely necessary. When it comes to security, you typically want to work on the principle of least privilege, meaning that you only give a user or application the access they need to accomplish necessary tasks, and only for the time period that they need that access for. For that purpose, you can generate shared access signatures. A shared access signature or a sass token is a string that contains a security token. You can use that string and calls to the service from the azure CLI power shell and the client libraries where he can actually upend the SAS token onto the end of the U. R L. To the blob and azure will parse the token and provide access. In the last module, you saw how to enable anonymous access to blobs in containers using the public access level. But if you set the public access level, the private, then you need to pass credentials. You could still access a protected blob in the browser by a pending a valid, shared access signature onto the girl to the blob. SAS tokens let you specify access to a particular storage service like the blob service, but not the file table or Q service is, and you can scope the access down to a container within the blob's service or even to a particular blob. You can also specify what permissions air granted, like only allowing the token toe list, container contents or only read of love, or to be able to write two blobs and elite gloves. And you can even embed a validity period into the token, so it only provides access for the period of time that you specify. I won't go into much more detail on shared access signatures in this course because I have an entire module on configuring security and access. In my other course in the Ascher storage path, on creating and configuring Microsoft azure storage accounts, there are a couple of their futures I'd like to mention. There are two types of shared access signatures. The service SAS delegates access to resources within a storage account. But there is another type called on account sass, and it encompasses the same capabilities but also allows access across more than one service within the storage account so it could allow a caller to access the blob and file service is, and the cue service is, for example, on account. SAS also allows the caller to perform administrative tasks like setting service properties and retrieving statistics about the service. The last feature I'd like to mention about shared access signatures is that you can package up groups of parameters into something called a stored access policy. This lets you set up a group of permissions within azure, and then the shared access signatures can reuse those permissions by referring to the stored policy and adding any additional parameters. Using stored access policies makes revoking tokens easier to because revoking the stored access policy makes all the shared access signatures that refer to it invalid. The last type of authentication is relatively new. It's the ability to grant access to the blob service by using credentials stored in azure A. D. So this is more like the traditional type of access you typically associate with a file system where access is granted directly to a specific user or Windows Principle as you're a de authentication is available for blobs and cues so you can now use role based access control to grant specific permissions to users, groups and applications that are registered with azure A D down to the scope of an individual blob container. You can also leverage a feature called managed identity for Asher Resources, which was formerly called managed service identity you can enable this feature for virtual machines, containers and as your APP service is, so they have a managed service principal identity in Azure A. D and administrators can assign roles to these identities. What this means is that you no longer have to store account keys or shared access signatures in code. Instead, you're just requesting an O off 2.0 access token from Azure A. D has your A D, then handles authentication of the security principle and returns the token to the APP. And the APP can then use the token to authenticate to azure storage. Access could be scoped to the subscription, the resource group, the storage account or to an individual blob container. And you can assign access rights using the azure portal, the command line tools or the management AP eyes. It's all done using, built in or custom are back rolls. The Azure Blob Service offers three built in roles. The Storage Blob Data Contributor role, which provides, read and write permissions, the storage Blob data reader role for read only access and the Storage Blob Data Owner role, which allows you to set ownership and access control for Asher data Lake Storage Jen, too. The version of the azure storage service rest a p I that works with Azure. A de authentication is the 2017 11 09 version or higher, and you have to use https to call azure storage operations. Now that we understand the different authentication options toe access data in the azure storage blob service, let's look at some examples using the different tooling provided by Microsoft.

Using Azure Storage Explorer to Upload Blobs

I'm on the page to download and install Azure Storage Explorer. As I mentioned in the introduction, this is a free tool from Microsoft that's available for Windows, Mac and Lennox. There's nothing special about the install, so we'll skip that, and I've already installed it on my virtual machine here. So let's open it up. When Azure Storage Explorer opens, it prompts me to connect to azure storage. This is where we can choose the authentication option, and that can impact exactly what we have access to. I could add an azure storage account here, and the account would need at least read access at the storage account level in orderto list the containers. This is because the permission is applied to the management layer. Recall that we talked about the management layer versus the data layer in the authentication overview. In order to see the contents of the containers, the user would need at least the contributor role, which provides access to the storage account keys. If you want to give the user more fine grained access to the contents of the storage account using Azure 80 roles, then there's a new approach using azure E D credentials that will show you later in this clip. For now, let's close this dialogue and take a more basic approach and log in using the storage account keys. Remember, this is the key that provides full access to a single storage account. But it's also a key that can be regenerated if need be, so the credential can essentially be revoked from everyone who has it. Remember that a storage account name is unique across all of Azure. Of course, I don't have the storage account key memorized, so let's go into the azure portal and copy it. I already have the storage account blade open, so we'll just go down to access keys and copy the key. I could remember the storage account name because it's pretty simple. Then we'll go back to Azure Storage Explorer, type in the account name and paste in that account key string and give the account of display name. Then I'll click next and connect this error. Adding a connection is unknown error in the current version of Storage Explorer, so we can safely ignore this. The storage account has been added, so let's clear this and open up the list at the top left. My storage account shows up under this list with the connection type in brackets, and I could add as many of these types of connections to different storage accounts is, I want so you might have connections to accounts in different subscriptions. Like if your organization chooses to break up Dev You 80 and production environments by subscription, for example, I'll open up the Blob Container folder and Here's Container, one that we created earlier and uploaded some files into I've Got a Block blob, a page blob in a pen blob here, along with a folder. Before we upload anything new, though, let's take a quick look at what you can do at the Blob's service level within Azure Storage Explorer. So I can create a new container, configure across origin resource sharing, configure the soft elite policy and search the contents of the blob service. And at the container level, I can delete and rename the blob container and do many of the same operations I did in the portal, but without having to have administrator credentials within the portal. One of the things I can do is set the public access level of the container like we did in the previous module. I can also create stored access policies and generate a shared access signature, and I'll show you that in a minute. But first, let's look at what we can do with existing blobs. There are a number of operations available across the top of the window, like creating folders within the container, copying and renaming blobs and managing snapshots. If I right click on an existing blob, I've got some of the same things available, plus the ability to edit properties, which includes metadata about the Blob. So let's go ahead and upload a new blob. I've got an audio file on my virtual machine, so I'll select that and you can see we've got the same options available on this upload dialogue that we had on the advanced menu within the portal. We can choose what kind of club this file will be categorized as as well as having the ability to create a folder during the upload. I'll go ahead and upload this file at the bottom. I've got a list of activities, so if we were uploading multiple files, we could monitor the progress here. In this case, there was just the one activity for the single file. That's all there is to uploaded a file in Azure Storage Explorer. But where this tool really shines is in delegating tasks to other users. I've logged in here with the storage account key, which provides a lot of privileges, and you might not wanna hand that out. Let's say you've got a business client who needs to upload a file as part of a business process, and this client might not have an identity in Azure Active Directory. In this case, you could generate a shared access signature and provided to them, along with Azure Storage Explorer. Let's look at how to generate that token with my storage account key log In credentials, I can generate a sass token at the container level and in the dialogue that opens. I could choose to have the token reuse, a set of permissions created as a stored access policy. But I don't have any of those created yet, so I'll just create this SAS token with the defaults, which is a shared access signature that's valid for one day, and it will only allow the user to read blobs, enlist the container contents I want to leave these permissions so I can show you the experience in the tool. Let's go ahead and create this token. Now I'll copy the URL and close the dialogue, and now I'm going to detach from the storage account that I logged into with the storage account key. So essentially, I'm creating a fresh instance of Azure Storage Explorer, where I have no log in credentials. So we'll open up the connect menu again as if I'm a different user. And this time I'll choose to use a shared access signature as my credential, and you can see that azure storage Explorer is parsing the your eye and sees that it's valid for a specific container in this case, container one. I'll click next, and I can see the expiration date and the permissions list so I'll go ahead and connect in the container opens under attached to containers in the menu on the left. Remember, there was a second container that was visible when I connected using this storage account key, but we can't see that anymore. And if I right click on the container, I have a limited set of operations I can do here Now let's take a look at a different way to allow user to access a blob container when that user has an account in Asher Active Directory. First, let's go back into the portal and down to the blobs menu and open up container, too. I'll open the access control blade, and this is scoped to just the container level now, and I'll add a role assignment. I'm going to choose one of the new storage blob data rolls. I'll give this user the storage blob data contributor role, and now I'll search for the user I want to give the access to. Let's save these changes and I'll open up the role assignments. At the bottom is the user and their new role in this container. Now let's go back to Storage Explorer, you know, add a new account. We're going to be accessing the container using the option. Add a resource via azure active directory. But before we can do that, we need to log in using an Asher 80 account. So click next with that option selected and I'll enter. The users log in credentials. Now the user is logged in, but we can't expand the contents for this user because they don't have a role that gives them access at the management layer of any of the storage accounts. So we need to add the particular resource this user has access to. Let's add a new connection and this time choose at a resource via Asher Active Directory. I'll click next, and this user account is the only one logged in. This is why we need to log in first so the user shows up in this list because you can't log in from here. I leave the default subscription choice because that's where the container is stored and click next. And now I need to choose whether we want to connect to a blob container or an azure Data Lake storage jen to container. I leave the default, and now I need the U R L to the container end point. Let's go back into the portal and get that. I'll go to the Properties menu inside the storage account blob container and copy this year L to the container. Now let's paste that in and I leave the default display name. Click next and connect. Now the container shows up under attached to containers user name is next to the container. If I right click on the container, this user has access to manage the container because of the storage Blob Data contributor role that was assigned. So this is a way to provide scoped access to users within your organization. Azure Storage Explorer provides a powerful view into your storage account data that can be used in a variety of scenarios where a graphical user interface is preferred. Next, let's take a look at the command line tool called a Z copy.

Understanding the AzCopy Command Line Utility

Ese copy is a command line utility designed for high performance copying of data to and from azure storage. Easy Copy is a free download from Microsoft, and it's available for Windows, Lennox and Mac OS clients. Before we get into some examples, let's talk about the capabilities of a Z copy at the time of filming the current version of ese copies. Version 10 Swell discussed the features of this release. There's no limit on the number of files you can copy or transfer with a Z copy, so it's great for large batch jobs. You can use wild cards to upload based on file extensions, and you can also upload files a recursive lee. So it'll upload all the files in the sub folders that match the criteria, and it will even create the sub folders in the destination blob container. You can synchronize files from your local file system to azure based on the file name and the last modified time stamp, and you can use a flag to delete files and azure that no longer exist in your source directory, and you can do the reverse. Also, sink the files and azure down to a local directory. You can move data from your local network to azure, or vice versa. You can move data between storage accounts, and it does that using the server to server AP eyes. So data is copied directly in azure and ese. Copy will also allow you to copy data from Amazon s three buckets to azure. When you submit a command to transfer files, ese copy creates a job plan, which lists all the files that were identified for processing. If a transfer fails for some reason, you can look at the job's list and choose to resume the transfer. In the latest version of a Z copy, which is version 10 ese copy will attempt to transfer all the files that are listed in the plan file, which weren't already transferred. And it does this at the file level so it won't restart transferring blocks in a file. It will restart the entire file. You can change the file name during the upload, and you can also set the content type of the destination file. This is useful for media files that you want to be able to open from the browser if they don't have their content typeset than the browser won't know how to play them as media files. You can set the blob access to your during upload in case you'd like the blobs to be in the cool or archived here right away. By default, ese copy will attempt to upload concurrently in order to maximize upload performance. But for low band with networks, this can cause failures because a large number of concurrent operations could overwhelm the network connection and keep up loads from completing. So you can throttle the number of concurrent requests using an environment variable. And you can also cap the throughput data rate so easy copy doesn't use all of your network bandwidth. You can generate over Bos Log file to track all the data operations and the times that they occurred in terms of authentication. You can connect to asher file storage using a shared access signature only. But for blob storage, you can use either a shared access signature or use azure Edie credentials with the latest storage blob data rolls. If you use a Z copy as part of scripted automation, you could even use the service principal identity and either leverage the client secret of the associative Appa registration and azure during log in or upload a certificate to the AP registration, and then use the certificate the law again. If you enable a managed identity for V, M and Azure, you could also grant that identity access to the storage account and run a Z copy from the VM using the managed identity. I'll talk more about managed identities later in this module. Ese copy provides performance and reliability through a scalable design where concurrency is scaled up according to the number of the machines. Logical course. You can also leverage the performance of a Z copy within Azure Storage Explorer by going to the preview menu and enabling it for use behind the scenes when uploading and downloading blobs. Next, let's look at some of the features of a Z copy in a demo.

Uploading Blobs with AzCopy

you can download a Z copy from Doc. Stop microsoft dot com. It's available for Windows, Lennox and Mac, and it's actually just a standalone execute herbal so you can put it anywhere on the file system that you like. If you're on Windows, I recommend creating environment variables to the file location so you won't have to prefix the execute herbal name with the path I've already downloaded ese copy and created environment variables to the folder I copied it to on my local file system. Ese copy integrates with Windows Command prompt, so I'll open that up by searching for it at the bottom here. First, let's type in ese copy help. This gives us a list of all the commands that are available, and most of these commands have various flags available for additional functionality. You can dig into the documentation for each command by typing ese. Copy the commander's name. I'll use the log in command and then two dashes and help for the log in command. Here, you can see the various flags that can be used to specify the type of log in. If we planned on using a sass token, we wouldn't actually log in. We would just depend the SAS token onto the U R L to the storage container with each operation, but we're going to leverage as your a D and log in with user account. So first, let's go back to the browser where I've got the Azure portal open and it's open to all the storage accounts in the subscription. I've already created a new storage account for this demo, so let's open that up. Let's give a user account permissions to this storage account. We do that from the access control tab, and I'll add a role assignment. We need to use the new storage blob data rolls, so I'll give this user the storage blob data contributor roll. If we just wanted the user to be able to download files, we could assign this storage blob data reader role. Next, I'll search for the user. I want to grant access to and click save to assign the role. Now we're going to be logging into a specific azure a D tenant. So let's open up azure active directory from the menu on the left and go to the Properties tab and copy the directory. I D now back in the command prompt window, I'll type ese copy log in two dashes than tenant Dash I D and Paste. In the idea of the ashtray de tenant that the user belongs to. The message says to navigate to microsoft dot com forward slash device Log in and paste in this code, so I'll copy the code, go to the browser and open a new tab and navigate to the girl. I'll paste in the code I copied, and I want to log in using the account that we granted access to on the storage account. So I'll enter the user's credentials. And now it says, I'm signed into easy copy. Back in the command prompt window, it says the log in succeeded. Okay, let's go back to the browser, and I'll close that tab and get back into the azure portal, and I want to go back to the storage account so we can see what effect are easy copy commands or having I won't put up the Blob's service in the storage account, and there's no containers yet, so let's create one using a Z copy. We do that with a Z copy make and then the path to the storage account, which always ends in blob dot core got windows dot net and then a forward slash and the name of the container we want to create. Remember, the container name needs to be all lower case. It says the container was created successfully, so let's go into the portal and refresh the Blob's service page. And there's the container. Okay. Next, let's upload a file from the local file system. This is really simple. You just type ese copy than the command name copy than the source file location on your local network and the destination file location in Azure. I'm going to change the name of the file. We're putting an azure during the upload and let's add a flag on the end to set the blob accessed here. During the upload, I'll type two dashes than block dash, blob, dash, tear and let's set. This blob is cool, so it'll be cheaper to store but more expensive to access. You can see how long the transfer took and how many files completed. Let's go into the azure portal and open the container we created, and there's the file that was uploaded downloading is just Azizi. I've got a folder here on the file system that's empty. So let's go back to a Z copy and I'll copy the location of the file we just uploaded into Azure. And to download the file, you type a Z, copy the command name copy and just reverse the source and destination. So the source location is now the file in Azure, and the destination is the location. I want to put the file in on the local file system. If I open up that folder, there's the file downloaded for Mazur. Now let's see how we can copy multiple files and let's only copy files in this local directory that have the file extension T X t. So there are two files here that our text files we want a copy. Only those I'll type ese copy, copy. Then, for the source directory, I'll add a backslash the asterisk wildcard character dot t x T. So this will select all the files with the T X T file extension in that directory. Then I'll type the Earl to the Blob container and let's create a folder during the upload that will store these files. It says the transfers completed successfully, so let's go into the portal and we're inside the blob container already. So let's refresh the contents. And there's the folder that was created. An insider, the to text files. Let's see how the list the contents of the blob container using a Z copy, you just type ese copy list and the path to the blob container. There's the files and their file size. If you want to copy all the files in the directory, including files in any sub folders, you can do that by using the directory name as the source location and after the destination in Asher used the flag recursive. The interface shows us the throughput for the upload, and there's a large audio file in the folder, so the transfer takes a little longer. Now let's go into the portal and refresh the blob container and a folder God created that has the same name as the Source folder on the file system. And inside are all the files in the sub folder also, so those are some of the ways the upload files to azure using the ese copy command line utility. Next, let's see how we can upload files using the Ascher cli

Using the Azure CLI to Upload Blobs

I'm inside the same storage account I used in the A Z copy demo. And I have a user here with the storage blob. Data contributor Role assigned. Well, log into the Ascher, see Ally with this user account and upload some blobs into this storage account. I'm using the latest version of the Ascher. See Ally and you convey. Verify that by running a Z two dashes version. It says the C ally is up to date. Now let's log in using ese log in and this opens a browser to the azure log in page. I want to log in with the user account. We gave the storage blob data contributor role too. And now I'm logged in soil. Return to the command prompt window where it shows that I'm logged into this subscription. Now let's upload a single file to a container. The command for that is a Z storage blob upload, then the file parameter and the location of the file on the local system. Then the storage account, name the container, name the name. We want to give the uploaded file the type of blob which in this case will be a block blob and then the off mode parameter, which will set toe log in so it uses the logged in as your A D credentials for this user, it says, the upload finished. So let's go back into the Ascher portal and inside the storage account, I'll navigate to the Blob service and let's open up Container one and there's the blob we just uploaded. Next, let's see how to upload multiple blobs at once. Let's go back to the command prompt, and I've got a local directory that has some files in it. There are two audio files, so let's upload just these files. I'll type ese storage, blob, upload, dash batch, then the location of the source file, the location of the destination container and azure, the account name of the storage account. This parameter pattern allows us to use a wild card to specify that only files ending with the wave file extension should be included in the upload. Then we use off mode with the log in mode, and finally, if you just want to see a summary of the operations without actually uploading the files, you can use this parameter dry run. Let's see what this does. We get back some Jason showing the files were found, but they weren't uploaded yet. So let's run this again without the dry run parameter. Now the files show a last modified date along with an e tag, and it says the upload finished. Let's make sure by looking in the azure portal, I'll just refresh the contents of the container. And there's the audio files we uploaded. So that's how you can upload files to blob storage using the azure sea ally.

Uploads Using the .Net Storage Client Library

in this clip, I'm going to show you the basics of uploading blobs to a storage account using one of the storage client libraries. In this case, it's the dot net core storage client library, but the principles error the same with other languages. This example will demonstrate using the storage account keys for authentication and then a more limited access scenario using a shared access signature. This is kind of a worst case scenario in terms of authentication, because it's possible to use Azure A D authentication to authenticate the calling application. And I'll show you that later in this module. But this clip will demonstrate what used to be the only way to authenticate to azure storage, and it's still a valid use case. I'm inside a storage account, and I've created a new container in the blob service called Library Container. There are no blobs in here right now, so let's open Visual Studio 2019 and build a simple web app to upload files to this container. When visual studio opens, I'll choose to create a new project, and then I'll select the ASP Net core web application template. I'll give this project a name and I'll a pendant with 12 because we're going to use Version 12 of the client library and you'll be able to find the completed project in the course downloads, by the way. Next I'll leave the defaults for dot net Core and SPD Net core 3.1, which is the latest framework at the time of filming. This works with earlier versions, though, as well as with the full dot net framework. And I'll use the Web application template, which uses razor pages. I don't have any authentication set because I want to keep this as simple as possible. We'll be authenticating to Azure storage and code. I'll click, create and speed up the video because it will take a few seconds for visual studio to create the project. Okay, now we've got a basic web application created, so let's add a reference to the client libraries we need to connect to Azure Storage. I'll open up solution Explorer and right-click on the project file and select manage new get packages. This opens up new get package manager and shows all the packages installed. So let's browse for all the available packages, and I'll do a search for Azure dot storage dot blobs, So I'll select that and click install. This package will load some dependencies also, so I need to okay that and accept the license terms for the packages. The packages get downloaded and installed. Now let's close these windows and clean things up a bit. Okay, now, back in Solution Explorer, we're just going to modify one page the index dot CS html page that opens by default. We'll update the presentation CSS html page first and then the code behind. So let's open up the CSS html page. I'm going to paste in some code, and you can find this code in the course downloads. This code just contains an HTML form with a post method, and that post will get initiated by the submit button. Here. The Post will get handled by a method we're going to add to the code behind. There's an input with the type file, so this will allow us to choose multiple files from the local system for upload to Azure storage. Now let's open up the code behind for this page, which is index dot CS html dot CS, I'll paste in the code here first, the using statements at the top. Now let's get the code for the index model class. First, I'll paste in this code that I'm going to use to pass messages to the CSS HTML page. Then I'll paste in a method that's the handler for the submit button on the page. I just need to resolve some references here. Some of these error specific to the storage client library, and some are just for the file and asp. Net functionality. I've got code here to use different types of authentication to blob storage. First, there's code that takes the connection string to the blob storage account, and you can get that from the azure portal. Then there's this block of code that uses the name of the storage account and the storage account key. And those error used to create a storage shared key credential object that's passed into the constructor of a blob service client object, along with the dry to the blob's service endpoint. We'll be filling in this information soon. There's commented code to use a shared access signature instead of the storage account key, and we'll go through this shortly. In both instances, the blob service client object is used to create a blob container client object that scope to the container we want upload to further down the blob container client object is used to create a blob client object, and this is what's used to upload the files. This, for each loop will iterated through all the files that error being uploaded. Now let's get the storage account key from the Azure portal. Let's close out of the container and scroll up to the properties and let's copy the URL to the Blob's service and back in Visual studio. Let's paste this into the variable that will get past the Blob's service client constructor. Now let's go back to the portal and go to access keys and let's copy the storage account name and paste that back into the variable in the code. And let's copy the storage account key and paste that into the code. Also, that's all we need for this code. So let's set a break point and run the code. When the page opens, I'll choose a few files from my local file system on this VM these to text files error fine and then I'll just click upload that causes the break point to get hit in the code behind. So let's step through that. I'll just create some space here and then hitting F 10 in visual studio we can step through the lines of code. First, the storage shared key credential gets created using the account name and the account key, and that gets passed into the constructor for the Blob's service client object and that's used to create a blob container client object. Next, we iterate through the files that error being uploaded. For each file, we create a blob client object and read the file into a stream object and passed that stream into the upload, a sync method on the Blob client. The first connection takes a second, and then we could just loop through for the second file and let's click. Continue to finish debugging the screen, says the files error uploaded. But let's go into azure storage and make sure I'll scroll down to blobs and open up the library container that we referenced in code. And there are the two files I chose on my VM. I'll just click on one of these blobs, and since it's a text file, we can see the contents by clicking edit. Okay, good. Let's just delete these files for now because we're going to use this container again. So that was how to authenticate using the storage account key. But giving that key out means giving developers and applications full control over the storage account. Where possible, you should limit their access. So next let's comment out the code that uses the storage account key and let's uncommon the code that uses a shared access signature here. We just need the URL to the container with the shared access signature appended on the end that will be used to create a blob service client object, and the rest of the code is the same. So let's go back into the azure portal and create a shared access signature that only allows access to this particular blob container. I'll close out of the container and go up to the Storage Explorer menu. This is a new feature in the portal that has some of the same behavior as Azure storage Explorer. You can right-click on the container and choose get shared access signature. Let's give this token permission to ad and right to the blob container, and I'll leave the default validity period of one day, I'll click create, and we get a URL that has the full path to the container. With the SAS Token upended on the end and back in visual studio, I'll paste it into this variable. Okay, everything else is the same, so let's run the code. When the page opens, I'll choose a couple of files and click upload. We get sent to the break point and I'll click. Continue to end the debugging. It says the two files were uploaded, so let's verify that in Azure I don't have to go to the Blob services we can. Just close this pop out and refresh the container view within the storage Explorer blade here, and there are the two files we just uploaded. So that's the basics of how to upload files to azure storage

Authenticating Code Using Azure AD

you've seen in this course how you congrats storage account access to an individual user account using the storage blob data roles. And that's because the user account is represented by a user principal object in azure Active directory. It's also possible to create a service principal object for your applications and grant those apps access to azure storage. That way, you don't need to store the storage account key or even a shared access signature in code. There are two ways you can do this. You can create an app registration in azure aid that represents your application. A service principal gets created behind the scene, and you can then grant that service principal access to azure storage. When you create an app registration, you need a way for the calling code to verify itself. So you can either upload a certificate that will be used by the caller or create a secret key that the caller will send to verify its identity. The secret key is just a string, and you can choose how long you want it to be valid for these e idea of the app. Registration and the secret key get stored in your code You can then use Microsoft client libraries to call azure app passing in those values. This provides a clean way of providing access, and it works regardless of where your code is deployed. It could be within an app service in azure, on a VM, in azure or on premises on an I s server on your own network. But this approach still requires storing credentials. Just in this case, the credentials error, the idea of the app registration and to secret key. Those credentials could still be shared or compromised, and they might get checked into your source code repository by developers. So there's a great solution to this problem if you're deploying your code on to Azure apps Service or onto a virtual machine in azure infrastructure as a service managed identity for azure resources is a feature of Azure that provides Azure services like app services and BMS with a managed identity in azure aid. In fact, it used to be called managed service identity, or MSC. It's simple to turn on this feature within app service from the identity menu, and you can do the same thing with an azure virtual machine, and then you grant the resource access to other azure resources like azure storage, key vault or even azure SQL. On the code side, you just need to add a reference to the azure dot identity library. Then it's just a line of code to create a default azure credential. The library automatically picks up the managed service identity of the host that the code is running on. And if a managed identity isn't available, the library automatically supports the use of azure aid user credentials. That means that during development, it can fall back to using the developers user account. If you grant that account access to the storage account in Azure in your development environment, for example, the developer can run code against that environment from their local workstation and be able to use azure storage. Then, when they deploy the code to the app service in Azure, the code will use these identity of the app service without having to make any code changes. Next, let's see how we can leverage managed identity for azure resources. In a web app

Managed Identity for Azure Resources with .NET Code

Now let's see how we can authenticate to blob storage using managed identity for azure resources. First, we'll connect using the credentials of the user that's logged into visual studio on the local VM. Then we'll upload the code to an azure app service and use the managed identity of that app service. In both cases, these identities have been granted permissions on the storage account. But the code itself doesn't need to change from local development to the production deployment. The correct managed identity is picked up by the framework. I'm using the same SPD net core project that uses the Azure storage Blob Client library. I've already added the azure identity library that we need in order to use a managed identity. Let's go to the code behind for the index page. I've commented out all the authentication methods you saw earlier and added this line to create a credential using default azure credential. This uses these Azure identity package. Then that credential gets passed to the constructor for the Blob Service client library. This code hasn't changed. We're just adding a different credential Now. All the rest of the code is the same. A S P net core will create this credential. Depending on the environment, the code is running in when it's running in azure in app service, it will try to use the managed identity of the app service. When the code runs here locally in visual studio, it will use the credentials of the user that's logged into visual studio. You can choose that identity from the Tools menu under options than azure service authentication you can select from any of your logged in accounts here. I've chosen a user stored in my azure active directory tenant. Let's go over to the azure portal and up to the access control menu in the storage account. And let's view the permissions on the storage account. These er logged individual studio has been granted the storage blob data owner role on the storage account. So let's go back to visual studio and run this code. I'll select a file on my local computer and click upload. It says one file was uploaded. So let's go over to the portal and go to containers and open up the library container and there's the file. Okay, I have another tab open here, and I've created an APP service already. There's no code deployed here yet, though, so let's deploy the solution from visual studio. First, I'll stop debugging and in Solution Explorer, let's right-click on the project and choose published. I'll create a new published profile and select an existing app service. When I click create profile, I get a list of resource groups, so I'll open the one that contains these app service and select that Now just okay and publish. While that's publishing, let's go back to the portal and in the app service, I'll scroll down to the identity menu and change the status to on. I'll just save this and confirm it says the managed identity was successfully registered with Azure Active Directory. Now let's go to the storage account that's still open on this tab, and I'll close the container and scroll up to the access control menu. Let's add a role assignment for the app service managed identity. I'll choose the role, and the one we want is storage blob data contributor. Now let's search for the name of the app service, which is app service blob test. You can actually see it on the other tab that's open. It's being found So let's select it and click Save. And now let's go back to the app service. We already deployed the code, so let's go up to the overview tab and click browse that opens these app. In another tab. Let's choose a file and click upload. It says the file was uploaded. So let's just verify that back in the storage account, I'll just open up the container and there's the file. So we were able to develop locally against the storage account, using the identity of the user in visual studio, and then deploy the application to Azure without making any code changes. The framework handled selecting the correct identity. And best of all, we didn't have to store any credentials in code or in configuration files. It was all handled by the Azure identity library and managed identity for azure resources.

Module Summary

this module looked at the different ways to upload blobs into Asher Blob storage. We started with a review of the different ways to authenticate to azure storage. Then we looked at Azure Storage Explorer, which is a graphical user interface that gives you a window into your storage account. Then we went in depth on the A Z copy command line utility. Then you saw how to upload blobs using the azure sea ally. And finally we looked at the Storage Client library for dot Net. First, we authenticated the calls to azure storage, using an access key and a shared access signature. Then I explained how the leverage azure Edie for authentication, and you saw a demo of using managed identity for azure resources. In the next module, we're gonna look at how to configure azure content delivery network end points to expose azure blob storage

Configuring CDN Network Endpoints

Introduction

cashing is the process of storing data locally so that it can be provided more quickly when it's requested again in the future. A content delivery network is a distributed network of servers all around the world that store cash data in order to minimize latency by providing the data to a user from the closest source, as well as offloading traffic from the source web or storage servers where the data originates, the data is typically static data like images, fonts, videos, HTML pages, client side scripts and really any kind of static file like a word document or a pdf Azure Cdn can connect to a number of azure services to cash, web and media data, and it also caches blob data by integrating with azure blob storage in azure cdn terminology. The blob service the Cdn connects to is called the Origin Server, and the CDN servers providing the data to users are called edge servers. Besides minimizing latency and providing better performance for users, as your cdn also provides large scaling to handle loads and it reduces user traffic that would normally go directly to your blob service. You pay for users to access the azure Cdn. Of course, costs associated with accessing azure storage are only incurred when azure cdn needs to refresh data from the origin servers. When a request for a file comes to an azure cdn endpoint, Azure cdn will route the request to the edge server that's geographically closest to the user. If no edge servers have the file in their cash, the Edge server retrieves the file from the Origin Server, which is the blob's service. In this case, the Edge server cashes the file and returns it to the user who requested it. The file remains on the edge server until the time to live for the file expires. As long as the file hasn't expired on the edge server, meaning it hasn't outlasted its time to live, it keeps getting served to users from the edge server. You can specify how long of files should be cached for by setting the cash control property directly on blobs. You could do this in azure storage explorer or programmatically, but you can also create cdn cashing rules within Azure Cdn, which override any cash control settings on the blobs themselves. And these rules could be scoped globally to all files or be applied to specific file paths or file extensions in order to provide coverage all around the world. Microsoft offers different plans with azure cdn, some of which work with partners like Verizon and Akemi. And there's also a standard Microsoft offering these different plans error called pricing tiers. When you create an azure cdn profile, something to keep in mind is that there are different features offered by the different pricing tiers, so just make sure you understand which features you want to utilize and choose your plan accordingly. Has your cdn is made up of profiles and endpoints? A profile is a collection of endpoints and azure. Cdn is priced at the profile level, so you select a pricing tier per profile. A Cdn endpoint is essentially a URL that provides access to a blob service within an azure storage account and represents a specific configuration of content, delivery, behavior and access. One of those configuration options is the cashing behavior. Another configuration option is accustomed. Domain name by default. The host name of an endpoint is Suffolk's by azure edge dot net, but you can specify accustomed domain for the endpoint. You can also enable https and either use an azure certificate or supply your own. You can configure an endpoint to apply file compression to file types that you choose, and there are other Optimization is available for delivering larger files. You can also configure a feature called Geo filtering that lets you allow or block content to users in specific countries. When azure CDN request data from the azure blob service, azure cdn edge servers error essentially acting as servers on the Internet and retrieving data from your storage account over the Internet. So if your blob data is not publicly available to the internet, in other words, the public access level of the blob container is set to private. Then the CDN servers need to authenticate using a shared access signature. There are configuration options for this, depending on which pricing tier you choose as your cdn also collects Usage Metrics and Corr analytics that you can export to blob storage to Event-Hubs or to azure log analytics so you can monitor things like the number of requests, How many requests were served from the cache and how many required retrieving the data from the origin servers. These amount of outbound data transfer in gigabytes and the http status codes that were returned to the collar. There's also additional reporting available if you're using the premium Verizon pricing tier. So in this module we'll look at how to enable Azure cdn to cash data from your blob service. We'll explore some of the configuration options available on the Cdn endpoint. Then we'll look at some security considerations that impact how your data can be accessed from the browser. And this especially matters when you're integrating your blobs in to web pages. Then you'll see how you can restrict anonymous users from accessing your blob containers by setting the public access level to private but still allowing azure cdn to retrieve the blobs for cashing and in the demo. For that feature, I'll show you some of the advanced features available with the Verizon premium pricing tier. Let's get started by creating an azure cdn endpoint for the Blob services

Enabling Azure CDN for Blob Storage

There are a couple of different places you can start from to enable Azure CDN for your blob storage. You could create a CDN profile and configure it against the Blob service in a storage account, or you can actually enable it right from within the Blob service. So let's do it the second way. I'll open up this storage account and navigate to the Blob service. I've got a container here, and inside there's a single blob that I want to be able to cache using Azure CDN. So let's go down the menu on the left side for the storage account into the Blob service options and select Azure CDN. This menu lists all the endpoints that have been created against the Blob service. You can multiple endpoints if you need different configurations with regards to caching, compression, geo-filtering, and the other options. Down at the bottom we can create a new endpoint. As part of the process, we have to specify a CDN profile, which is the parent of the endpoint and controls the features available by selecting the pricing tier. Let's create a new CDN profile, and we'll use the Standard Verizon pricing tier. The endpoint name is the name that will be available on the internet, so I'll give it a name that's unique across Azure. And the Origin hostname is already filled in because we're creating this endpoint from within the storage account. And it'll actually take a minute or so, so I'm just going to pause the recording. And now the CDN profile is created and the endpoint is showing here in the list. Let's go into that endpoint and copy the endpoint hostname. And I'll open a browser window and paste in the endpoint name. I'll add on the container name, as well as the name of the file I want to access. And you can see at the bottom the browser has downloaded the file. So I'll just open up the file to make sure it can be accessed. And that's how to set up Azure CDN to work with Blob Storage.

Exploring Azure CDN Features

Now that we have an Azure CDN endpoint created for our Blob service, let's take a look at some of the features in the portal. Under settings we've got information about the origin, which in our case is Azure Storage. We could change the storage account here if we wanted. For blob storage the host header has to match the origin hostname, so we wouldn't want to modify that unless we change the hostname. And if we wanted to scope this endpoint down to a specific container, we could do that with the origin path. We can also choose whether to disable HTTP or HTTPS. We can set up a custom domain here, so our users would see our domain for the Azure CDN endpoint, instead of our endpoint name plus azureedge. net. We can fine tune the compression settings, to add or remove which file types we want Azure CDN to compress when the files are served to the user. We can enable geo-filtering, which lets us restrict whether specific content is blocked for users from certain countries. And there's a big list of countries here to choose from. Optimization options depend on the type of source content and the pricing plan we've selected. General web delivery is used for media streaming and large file download, but there are other options available in some plans that are specific to media streaming, video on-demand streaming and dynamic site acceleration, which is used to accelerate responses for web apps that aren't static. On the Diagnostics tab we can choose to send logs to a storage account, an event hub or to Azure Log Analytics. And the last feature I want to talk about is the caching rules. We have global caching rules here that apply to all the content handled by the endpoint. The caching behavior lets us turn off caching altogether by choosing Bypass cache. This also overrides any cache directive headers set on the blobs themselves in the origin storage account. Override means Azure CDN will ignore any cache directive headers retrieved with the blob, and it'll use the cache duration we specify here instead. And Set if missing will honor any cache directive headers set on the source blob if they exist; otherwise Azure CDN will use the cache duration set here. You can set those cache directive headers on the blobs using Azure Storage Explorer or programmatically. Below that you can set the cache duration, and below that you choose how query strings on the requests should be handled. You can ignore query strings, which means Azure CDN will pass through the query string to the Blob service on the first request to retrieve the blob, then it will ignore query strings when that blob is requested again by any user. The same blob will be served from the cache until the time to live expires and it gets refreshed from the Blob service. Bypass caching for query strings means that if there's a query string on the blob request, it'll be retrieved from the Blob service every time and won't be cached at all. And Cache every unique URL means each URL with a query string is treated as a unique asset with its own cache, so if 10 people request a blob with different query string parameters, that blob gets cached 10 times, once for each unique URL. This option is useful for cross-origin resource sharing, as well as for accessing containers that have their public access level set to private by appending a Shared access signature as a querystring parameter. We'll talk about both of those scenarios later in this module. Below the global cache settings, we can set up custom caching rules that override the global settings. So you can specify cache durations for specific file extensions or paths, meaning containers and folders within the Blob service. You can set up specific caching behaviors and cache durations to the content matching the condition. So that's a quick tour of the typical options for an Azure CDN endpoint.

CORS and SSL Considerations

There are a few considerations with Azure CDN and blob storage when it comes to web browser security. If you're using the CDN to display content in the browser as part of a website that's being served using HTTPS, you'll want to make sure that your Azure COULDN'T endpoint is also served using HTTPS; otherwise users will get a warning in the browser about mixed content. If you're using the standard endpoint provided by Azure CDN, HTTPS is enabled by default and there's no additional cost. You can also configure HTTPS on a custom domain, and you can have Azure CDN create and manage your certificate through DigiCert, and it's actually free. You just need to be aware that there is an email verification step that relies on the who is information for your domain name, so if that's not publicly available, you may need to contact Microsoft directly to complete the process. Or if you're on the Microsoft Standard pricing tier, you can use your own certificate and it gets configured through integration with Key Vault. Another security consideration is cross-origin resource sharing, or CORS. It's is an HTTP feature that enables a web application running under one domain to access resources in another domain. Web browsers implement a security restriction known as same-origin policy that prevents a web page from calling APIs in a different domain; CORS provides a way to allow a webpage served up from one domain to call APIs in another domain. You configure a CORS policy on the origin server, in other words in the Azure Blob service. CORS isn't enabled by default. CORS on Azure CDN will work automatically with no additional configuration when the Access-Control-Allow-Origin header is set to wildcard or a single origin. The CDN will cache the first response, and subsequent requests will use the same header. If you need more advanced control over setting multiple origins, you can either enable query string caching for the standard pricing tiers, which requires you to use a unique query string for request from each allowed domain. You'll end up with multiple copies of the same file in the CDN though, so it's not ideal. A better approach is to use the Premium Verizon pricing tier, which lets you create a rule that sets the Access-Control-Allow-Origin header sent back to the browser, so the browser will allow the content. In this way, Azure CDN handles the CORS policy. The next thing I want to talk to you about also has to do with security and how you can let Azure CDN access containers whose public access level is set to private.

Controlling Access Using Shared Access Signatures

When you configure Azure CDN to cache content from your Blob service, it needs to be able to access the content over the internet, so your containers need to have their public access level set to blob or container, which allows anyone who has the URL to be able to access the blob in the Blob service directly, bypassing the Azure CDN. Depending on your requirements, that might be fine, but you can also restrict access by setting the container's public access level to private, and create a shared access signature to allow clients and the Azure CDN to retrieve the blobs from the Blob service. There are three options for doing this, and which one you use depends on which pricing tier you select. All of the pricing tiers support using a single SAS token and passing it through to the Blob service. The client