While the Log Analytics Workspace is an excellent tool for monitoring and analyzing logs in Azure, storing logs in a Storage Account provides a more cost-effective and flexible solution for long-term retention and external access. This setup allows organizations to store logs for extended periods and export them for integration with other tools or services.
Why Export Logs to a Storage Account?
There are several benefits to exporting tenant logs and other Azure logs to a Storage Account:
Long-Term Retention: You can define a retention policy to keep logs for months or years, depending on compliance and operational requirements.
Cost Efficiency: Compared to storing everything in a Log Analytics Workspace, which is more costly for extensive data, Storage Accounts offer lower-cost alternatives for long-term log retention.
Accessibility: Logs stored in a storage account can be accessed through APIs, or via tools like Azure Storage Explorer, allowing easy download, transfer, and external analysis.
However, each organization must balance storage needs with costs, as larger volumes of data will increase storage costs over time.
-------------------------------------------------------------------------------------------------------------
Steps to Export Tenant Logs to a Storage Account
Step 1: Set Up Diagnostic Settings to Export Logs
Navigate to Diagnostic Settings:
In the Azure portal, search for Azure Active Directory and select it.
Under the Monitoring section, select Diagnostic settings.
Create a New Diagnostic Setting:
Click Add diagnostic setting.
Name your setting (e.g., "TenantLogStorageExport").
Select Log Categories:
Choose the logs you want to export, such as Audit Logs, Sign-in Logs, and Provisioning Logs.
Select Destination:
Choose Archive to a storage account and select the storage account where the logs will be stored.
Confirm and save the settings.
Once configured, the selected logs will start streaming into the specified storage account.
-------------------------------------------------------------------------------------------------------------
Accessing Logs with Azure Storage Explorer
Azure Storage Explorer is a free, graphical tool that allows you to easily access and manage data in your storage accounts, including logs stored as blobs.
Using Azure Storage Explorer:
Download and Install: Install Azure Storage Explorer on your local machine from here.
Connect to Your Azure Account:
Launch Storage Explorer and sign in with your Azure credentials.
Browse to your storage account and locate the blobs where your logs are stored (e.g., insights-logs-signinlogs).
View and Download Logs:
Use the explorer interface to view the logs. You can download these blobs to your local machine for offline analysis, or even automate log retrieval using tools like AzCopy or Python scripts.
Logs are typically stored in a hierarchical structure, with each log file containing valuable data in JSON or CSV formats.
Examples of Log Types in Storage Accounts
Here are some common logs that you might store in your storage account:
insights-logs-signinlogs: Logs of all user and service sign-in activities.
insights-logs-auditlogs: Logs of administrative changes such as adding or removing users, apps, or roles.
insights-logs-networksecuritygrouprulecounter: Tracks network security group rules and counters.
insights-logs-networksecuritygroupflowevent: Monitors NSG traffic flows.
These logs are stored as blobs, while certain logs (e.g., OS logs) might be stored in tables within the storage account.
-------------------------------------------------------------------------------------------------------------
Sending Logs to Event Hub for External Systems
If you need to export tenant logs or other logs to a non-Azure system, Event Hub is a great option. Event Hub is a real-time data ingestion service that can process millions of events per second and is often used to feed external systems such as SIEMs (Security Information and Event Management).
How to Configure Event Hub Export:
Create Event Hub:
Set up an Event Hub within Azure Event Hubs service.
Configure Diagnostic Settings:
Just as you did for the storage account, go to Diagnostic settings for Azure Active Directory and select Stream to an event hub as the destination. Enter the namespace and event hub name.
This setup allows you to forward Azure logs in real-time to any system capable of receiving data from Event Hub, such as a SIEM or a custom log analytics platform.
-------------------------------------------------------------------------------------------------------------
Leveraging Microsoft Graph API for Log Retrieval
In addition to Storage Accounts and Event Hubs, Azure also supports the Microsoft Graph API for retrieving tenant logs programmatically. This API allows you to pull log data directly from Azure and Microsoft 365 services.
The Graph API supports many programming languages, including Python, C#, and Node.js, making it highly flexible. It’s commonly used to integrate Azure logs into custom applications or third-party systems.
-------------------------------------------------------------------------------------------------------------
All Above logs were part of the tenant logs: Lets start with second log category called
Subscription Logs
What are Subscription Logs?
Subscription logs track and log all activities within your Azure subscription. They record changes made to resources, providing a clear audit trail and insight into tenant-wide services. The primary information recorded includes details on operations, identities involved, success or failure status, and IP addresses.
Accessing Subscription Logs
Subscription logs are available under the Activity log in the Azure portal. You can use the logs in multiple ways:
View them directly in the Azure portal for a quick, interactive inspection.
Store them in a Log Analytics workspace for advanced querying and long-term retention.
Archive them in a storage account, useful for maintaining a long-term log history.
Forward them to a SIEM (Security Information and Event Management) solution via Azure Event Hub for enhanced security monitoring and correlation.
To access the logs in the Azure portal, use the search bar to look for Activity log. This will provide a quick summary view of activities within the portal.
-------------------------------------------------------------------------------------------------------------
Key Elements of the Subscription Log Schema
Each activity log entry has several key fields that can help in monitoring and troubleshooting. When an action, such as creating a new virtual machine (VM), is logged, the following fields provide detailed information:
resourceId: This is a unique identifier for the resource that was acted upon, allowing precise tracking of the specific VM, storage account, or network security group.
operationName: Specifies the action taken on the resource. For example, creating a VM might appear as MICROSOFT.COMPUTE/VIRTUALMACHINES/WRITE.
resultType and resultSignature: These fields show whether the operation succeeded, failed, or was canceled, with additional error codes or success indicators in resultSignature.
callerIpAddress: The IP address from which the action originated, identifying the source of the request.
correlationId: A unique GUID that ties together all sub-operations in a single request, allowing you to trace a sequence of actions as part of a single change or request.
claims: Contains identity details of the principal making the change, including any associated authentication data. This can include fields from an identity provider like Azure AD, giving insight into the user or service making the request.
Each log entry captures critical details that aid in understanding who, what, when, and where changes were made.
-------------------------------------------------------------------------------------------------------------
Subscription Log Access Options
Azure offers different access and filtering methods for subscription logs. Here’s a breakdown of how you can utilize the portal effectively:
Azure Portal:
The Azure portal offers a quick, visual way to explore logs. You can select a subscription, set the event severity level (e.g., Critical, Error, Warning, Informational), and define a timeframe for the log entries you need.
The Export Activity Logs option on the top menu or the Diagnostic Settings on the left allows you to set up data export or view diagnostic logs.
Log Analytics Workspace:
The Log Analytics workspace offers a more robust and flexible environment for log analysis. By sending your logs here, you can perform advanced queries, create dashboards, and set up alerts.
This workspace enables centralized log management, making it an ideal choice for larger organizations or those with specific compliance requirements.
Programmatic Access:
Using the PowerShell cmdlet Get-AzLog or the Azure CLI with az monitor activity-log, you can query the activity logs programmatically. This is useful for automated scripts or integrating logs into third-party solutions.
Event Hub Integration:
For real-time analysis, integrate subscription logs with Event Hub and forward them to a SIEM for security insights and anomaly detection. This setup is beneficial for organizations that require constant monitoring and incident response.
-------------------------------------------------------------------------------------------------------------
Subscription Logs in Log Analytics workspace
For Detailed analysis, it's best to set up a Log Analytics workspace. This enables centralized log storage and querying capabilities, combining subscription logs with other logs (such as Azure Active Directory logs(Entra ID Logs)) for a comprehensive view.
The setup process is identical to the one for the tenant logs: select the log categories you wish to save and the Log Analytics workspace to send them to.
Subscription Log Categories
The main log categories available are:
Administrative: Tracks actions related to resources, such as creating, updating, or deleting resources via the Azure Resource Manager.
Security: Logs security alerts generated by Azure Security Center.
Service Health: Reports incidents affecting the health of Azure services.
Alert: Logs triggered alerts based on predefined metrics, such as high CPU usage.
Recommendation: Records Azure Advisor recommendations for resource optimization.
Policy: Logs policy events for auditing and enforcing subscription-level policies.
Autoscale: Contains events from the autoscale feature based on usage settings.
Resource Health: Provides resource health status, indicating whether a resource is available, degraded, or unavailable.
-------------------------------------------------------------------------------------------------------------
Querying Subscription Logs in Log Analytics
The logs are stored in the AzureActivity table in Log Analytics. Here are some example queries:
Identify Deleted Resources:
AzureActivity
| where OperationNameValue contains "DELETE"This query is useful for investigating deletions, such as a scenario where a malicious actor deletes a resource group, causing all contained resources to be deleted.
Track Virtual Machine Operations:
AzureActivity
| where OperationNameValue contains "COMPUTE"
| distinct OperationNameValueThis query lists unique operations related to virtual machines, helpful for getting an overview of VM activity.
Count VM Operations:
AzureActivity
| where OperationNameValue contains "COMPUTE"
| summarize count() by OperationNameValue
By counting operations, this query provides insights into the volume of VM activities, which can reveal patterns such as frequent VM creation or deletion.
-------------------------------------------------------------------------------------------------------------
Archiving and Streaming Logs
To save logs for long-term storage or send them to a SIEM:
Configure diagnostic settings to specify the storage account or Event Hub for archiving and real-time streaming.
Logs stored in a storage account appear in a structured format, often in JSON files within deeply nested directories, which can be accessed and processed using tools like Azure Storage Explorer.
By effectively leveraging subscription logs and these configurations, Azure administrators can enhance monitoring, identify security issues, and ensure accountability in their environments.
-----------------------------------------------------------------------------------------------------------
Further we will talk in next blog, Until than stay safe and keep learning
Akash Patel
Comments