Share via

RedHat Linux - AMA agent Rsyslog issue

George OCAK 70 Reputation points
2025-05-08T15:32:20.78+00:00

Hi All,

We have the AMA agent installed on our host and confirmed that CEF events are being received. However, they are not being forwarded to port 28330 as expected. The firewall is disabled, SELinux is disabled, and the JSON configuration files for the DCR, rsyslog.conf, and the AMA agent config all exist and appear correct.

The Microsoft Sentinel troubleshooting script flagged only one issue: the root directory disk has less than 1GB of space remaining.

This disk is mounted to the host, but it is not partitioned or explicitly assigned to / or /var/log.

Could the insufficient disk space or the way the disk is mounted/partitioned be preventing log forwarding to port 28330, even though logs are being received?

If not, what else might block the AMA agent from forwarding CEF logs?

Note that port is open and host listen the port 28330 but not all the times i have seen the error message below.

rsyslogd[61418]: cannot connect to 127.0.0.1:28330: Connection refused [v8.2102.0-15.el8_10.1 try https://www.rsyslog.com/e/2027 ]

But when i restart the rsyslog error was gone.

Thanks

Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
3,587 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Vinod Pittala 1,830 Reputation points Microsoft External Staff Moderator
    2025-05-09T00:13:31.9766667+00:00

    Hello George OCAK,

    Insufficient disk space can indeed affect the Azure Monitor Agent (AMA) from forwarding logs. The AMA buffers events to /var/opt/microsoft/azuremonitoragent/events prior to ingestion, and if the disk is nearly full, it may prevent the agent from processing and forwarding logs effectively. and the agent might need to create temporary files during the forwarding process, and lack of space can hinder this.

    And you mentioned that the JSON configuration files for the Data Collection Rule (DCR), rsyslog.conf, and the AMA agent config all exist and appear correct. However, it’s essential to double-check the configurations for any potential issues.

    Additionally, even though the port 28330 is open and the host is listening on it, other factors could block the forwarding of CEF logs. These might include:

    1. Configuration Issues: Ensure the DCR is correctly set up to forward CEF logs. The rsyslog.conf file is correctly configured to forward logs to port 28330. and the AMA agent configuration is correct and matches the intended setup.
    2. Rsyslog Daemon State: If the rsyslog daemon is not running properly or encounters errors, it may not forward logs as expected. Restarting the rsyslog service, as you noted, resolved the error temporarily, which suggests that there might be intermittent issues with the daemon.
    3. Network Issues: Although you mentioned that the firewall is disabled, ensure that there are no network-related issues affecting the communication between the rsyslog and the AMA agent.
    4. Resource Limits: Check for any resource limits (like file descriptors) that may be imposed on the rsyslog or AMA processes.

    The error message you mentioned could indicate the connection to port 28330 is being refused. Restarting rsyslog resolves the issue temporarily, suggesting a potential configuration or resource issue. So, Investigate the rsyslog configuration and logs to identify any recurring issues. As said, ensure that rsyslog is correctly configured to forward logs to port 28330 and that there are no resource constraints affecting its operation.

    You can refer to the below useful documents.

    1. https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-troubleshoot-linux-vm-rsyslog
    2. https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-troubleshoot-linux-vm

    If the problem still persists post addressing these potential causes, you can enable debugging logs for the AMA agent to gather more insights into what might be going wrong during log forwarding.

    If you have any further questions, please do not hesitate to reach out. I am here to assist you.

    If you found the above comment helpful, kindly click "Upvote it".

    Thank you

    0 comments

Your answer