How Syslog Really Works with Azure Log Analytics: Flow Analysis

Straight to the point, lets understand how Syslog logs ingestion flow looks like.

Overview

Diagram 1.0 shows the high-level flow of Syslog

Diagram for reference in your journey reading this.

Log Sources to Log Collector (Rsyslog)

Log sources like Palo Alto Firewalls can be config to send Syslogs to the collector via TCP/UDP port 514. For CEF-capable devces, logs sent in CEF format are ingested into the CommonSecurityLog table (otherwise raw syslog will be ingested into Syslog table). To enable this, two things are required:

1. Configure the firewall to send logs (the configuration specific to the network devices)
2. Configure the Rsyslog on the collector to listen on port 514 by uncommenting these lines in /etc/rsyslog.conf:

module(load="imudp")
input(type="imudp" port="514")

module(load="imtcp")
input(type="imtcp" port="514")

# Disable any local loggins like *cron/*auth/*priv

1. Port use — 514
2. Protocol — TCP/UDP

So to verify if rsyslog is receiving logs from firewall (or other log sources) over TCP/UDP 514, use the following command in the log collector:

For UDP (default Syslog):

sudo tcpdump -i any udp port 514 -A

For TCP:

sudo tcpdump -i any tcp port 514 -A

And to specifically check for CEF logs, you can do:

#For capturing CEF logs
sudo tcpdump -i any tcp port 514 -A | grep "CEF"

#For capturing logs from certain host
sudo tcpdump -i any tcp port 514 -A | grep "hostname or keywords"

Which if logs are arriving, you will see the logs entries:

20:50:43.758909 IP 172.25.8.38.37406 > ptsg-3sen0004UQ.shell: Flags [P.], seq 4054:5070, ack 1, win 251, options [nop,nop,TS val 1110545853 ecr 3045084786], length 1016
E..,.6@.@......&...z.....@..,........N.....
B1....Nr<166>May 15 2025 12:50:43 172.25.8.38 CEF:0|Zscaler|ZPA|1.0|open|OPEN_OR_ACTIVE_CONNECTION....(logs continue)

What above output prove? There are packets being sent from log sources to rsyslog which means the communication between log sources and Rsyslog is ESTABLISHED.

rsyslog to Azure Monitor Agent (AMA)

One of the mistakes in my understanding was to treat log collector as a whole instead, log collector consists of two distinct components:

1. Rsyslog — Receives logs from network devices
2. Azure Monitor Agent (AMA) — Forward logs to Log Analytics Workspace

These components communicate internally and when AMA is installed, it creates a configuration file at /etc/rsyslog.d/10-azuremonitoragent-omfwd.conf. This configures Rsyslog to forward logs to AMA via TCP on localhost (127.0.0.1:28330) since AMA and Rsyslog run on the same host with AMA acting as the intermediary to Log Analytics Workspace

*.* action(type="omfwd"
template="AMA_RSYSLOG_TraditionalForwardFormat"
queue.type="LinkedList"
queue.filename="omfwd-azuremonitoragent"
queue.maxFileSize="32m"
quenetstue.maxDiskSpace="1g"
action.resumeRetryCount="-1"
action.resumeInterval="5"
action.reportSuspension="on"
action.reportSuspensionContinuation="on"
queue.size="25000"
queue.workerThreads="100"
queue.dequeueBatchSize="2048"
queue.saveonshutdown="on"
target="127.0.0.1" Port="28330" Protocol="tcp")

And to confirm there are ESTABLISHED connection between Rsyslog -> AMA, run netstat command:

sudo netstat -tnp | grep 28330

Which the expected output will be something like this:

tcp   0   0 127.0.0.1:28330    0.0.0.0:*    LISTEN      <PID>/azuremonitoragent
tcp   0   0 127.0.0.1:<PORT>   127.0.0.1:28330  ESTABLISHED <PID>/rsyslogd

This confirms that rsyslog is successfully forwarding packets to the Azure Monitor Agent. To verify this connection, you can run tcpdump on port 28330 (the AMA listening port) instead of port 514, using a similar command to what we used previously:

sudo tcpdump -i any tcp port 28330 -A
sudo tcpdump -i any tcp port 28330 -A | grep "keywords to search for the logs"

AMA to Log Analytics Workspace

AMA communicates with Log Analytics Workspace via HTTPS.

To monitor AMA log forwarding status, use:

sudo cat /var/opt/azuremonitoragent/log/mdsd.info | grep "keywords to search for the logs"

With the expected output of:

2025-05-15T11:58:50.6702370Z: ODSUploader::UploadFixedTypeLogs(const string&, const string&, const std::function<void(bool, long unsigned int, long unsigned int)>&, uint64_t)::<lambda(pplx::task<web::http::http_response>)> (.../mdsd/ODSUploader.cc +861) Successfull uploaded to ODS, Datatype: LINUX_SYSLOGS_BLOB, RequestId: <request-id>

By default, mdsd.info only displays heartbeat logs being sent to the Log Analytics workspace. To enable debugging for additional log types such as CEF and Syslog, you need to add the trace flag “-T 0x2002” to the MDSD_OPTIONS parameter in the azuremonitoragent configuration file (line 5), as shown below:

sudo nano /etc/default/azuremonitoragent

#Change the MDSD_OPTIONS
export MDSD_CONFIG_DIR=/etc/opt/microsoft/azuremonitoragent
export MDSD_LOG_DIR=/var/opt/microsoft/azuremonitoragent/log
export MDSD_ROLE_PREFIX=/run/azuremonitoragent/default
export MDSD_SPOOL_DIRECTORY=/var/opt/microsoft/azuremonitoragent
export MDSD_OPTIONS="-A -R -c /etc/opt/microsoft/azuremonitoragent/mdsd.xml -d -r $MDSD_ROLE_PREFIX -S $MDSD_SPOOL_DIRECTORY/eh -L $MDSD_SPOOL_DIRECTORY/events -T 0x2002"
export MDSD_USE_LOCAL_PERSISTENCY=true
export MDSD_TCMALLOC_RELEASE_FREQ_SEC=1
export MONITORING_USE_GENEVA_CONFIG_SERVICE=false
export ENABLE_MCS=true
export SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt
export PA_GIG_BRIDGE_MODE=true
export PA_FLUENT_SOCKET_PORT=13005
export PA_DATA_PORT=13005

This modification will display detailed logging in mdsd.info. Now you can see if the logs fail from AMA to Log Analytics Workspaces

Why are these important?

While high-level knowledge suffices for basic implementation, especially since many configurations are automated, deep understanding becomes crucial when things go wrong. When Syslog messages fail to appear in Log Analytics Workspace, where do we start? Log Sources to Rsyslog? Rsyslog to AMA? AMA to Log Analytics Workspace?

This detailed understanding transforms troubleshooting from guesswork into a systematic process. See ya in another post :D