Microsoft has announced the general availability of ingestion-time transformation for Azure Firewall logs in Log Analytics. This powerful feature allows you to filter, shape, and transform logs before they are ingested, meaning you can significantly reduce your logging costs while improving the quality and relevance of the data you keep.

In this post, I’ll explain what this feature means, why it matters, and give you practical steps to implement it for cost savings, compliance, and security monitoring.

Why This Update Matters

For many organizations, firewall logs are among the biggest contributors to Azure Log Analytics costs. Until now, every log generated by Azure Firewall would be ingested and stored — even if much of it was “noise.”

With ingestion-time transformations, you can:

Reduce costs by filtering out unnecessary logs.
Focus on security by only storing suspicious or high-value traffic.
Improve compliance by routing logs to the correct audit/reporting systems.
Speed up incident response with more relevant logs.
Enable smarter monitoring with targeted dashboards and alerts in Azure Monitor.

Key Scenarios

Security monitoring – Store only denied or suspicious traffic for easier detection of threats.
Compliance – Apply Data Collection Rules (DCRs) to send specific logs to secure storage for audits.
Incident response – Get to the critical data faster by avoiding noise.
Cost optimization – Ingest only the logs that matter.

How to Use Ingestion-Time Transformation in Log Analytics

Here’s a step-by-step guide to help you get started:

Step 1: Identify Your Logging Requirements

Review your current Log Analytics costs.
Identify which categories of logs are truly valuable (e.g., denied traffic, specific subnets, or suspicious IPs).

Step 2: Create or Update a Data Collection Rule (DCR)

Go to the Azure Portal → Monitor → Data Collection Rules.
Create a new DCR or edit an existing one.
Select Azure Firewall diagnostic logs as a data source.

Step 3: Define Transformations

Add a Kusto query transformation to filter or shape data before ingestion.
Example: where Action == "Deny" and Protocol == "TCP" This will only ingest denied TCP traffic.

Step 4: Test and Validate

Send some test traffic through your firewall.
Confirm that only the expected logs are showing up in your Log Analytics workspace.

Step 5: Build Dashboards and Alerts

Use Azure Monitor to create custom alerts based on the filtered logs.
Build dashboards to track trends over time, such as spikes in denied traffic.

Step 6: Monitor and Optimize

Continuously review costs and storage consumption.
Refine transformations as your security and compliance needs evolve.

Best Practices

Start small – Begin with one transformation rule and expand.
Document your rules – Helps with audits and troubleshooting.
Align with compliance – Ensure logs meet regulatory requirements before filtering.
Automate – Use ARM templates or Bicep to deploy and maintain transformations across environments.

Learn More

Sample Transformation Rule Library for Azure Firewall Logs

Below are some common scenarios with example ingestion-time transformation rules. You can copy them into your Log Analytics DCRs to filter or shape logs before ingestion.

1. Cost Optimization: Ingest Only Denied Traffic

where Action == "Deny"

Purpose: Only store logs where traffic was denied, reducing unnecessary log storage.

2. Security Monitoring: Track Suspicious IPs

where RemoteIPCountry !in ("US", "CA", "UK") and Action == "Deny"

Purpose: Only log denied traffic from countries outside your typical business footprint.

3. Compliance: Capture All Audit Events

where Category == "AzureFirewallApplicationRule" or Category == "AzureFirewallNetworkRule"

Purpose: Ensure all firewall rules are logged for auditing purposes while ignoring non-critical telemetry.

4. Incident Response: High Severity Alerts

where ThreatLevel in ("High", "Critical") or Action == "Deny"

Purpose: Prioritize ingestion for high-severity events to speed up investigation.

5. Custom Dashboarding: Log Only Specific Protocols

where Protocol in ("TCP", "UDP") and Action == "Deny"

Purpose: Focus on relevant protocols for visualization in Azure Monitor dashboards.

6. Regional Filtering: Only Ingest Logs for Specific VNets

where VirtualNetwork == "ProductionVNet" or VirtualNetwork == "CriticalVNet"

Purpose: Limit logs to critical networks, reducing unnecessary ingestion from development or test VNets.

How to Apply These Rules

Navigate to your Data Collection Rule in the Azure Portal.
Add a transformation under the “Transformation” section.
Paste your KQL rule.
Save and apply the DCR to your Azure Firewall diagnostic logs.
Test your rules with sample traffic to verify only the desired logs are ingested.

Linkedin Profile Http://nl.linkedin.com/in/robertsmit

Robert Smit MVP Blog

Message History

Add new keyword