Check Data Collector Logs for Syslog Traffic

Requirements:

• Console or SSH access to the FMOS Data Collector CLI
• CLI Credentials

Summary:

This article will outline the procedure to review data collector logs to see if it is receiving syslog messages from a specific device.  This can be used to assist in troubleshooting logging issues.

Procedure:

Set the DataCollector log level to Debug

1. Log into the Server Control Panel at https://<DataCollectorServerIP>:55555
2. On the toolbar, click SecMgr > DC.  
3. Set the Data Collector Root Log Level field to DEBUG.  
4. Click Stage Changes and then Apply Configuration.

Reminder: Once you are done troubleshooting you’ll want to disable debugging mode.  Set the Data Collector Root Log Level field back to (Default)INFO, then click Stage Changes and then Apply Configuration.

General Syslog Ingestion Flow

If you can view results while tailing the log and grepping for the device, look for the following messages:

• FireMonUdpServer - shows any incoming UDP Traffic
• SyslogObserverList  - attempts to match syslog message to a specific device in FireMon

It uses two methods to do this:

Compares the Source IP address of the syslog message to the Central Syslog IP addresses configured in the Administration application.

• If there is a match between Source IP and Central Syslog IP then we compare the syslog to several Central Syslog Regex patterns.
• If there is a match we then compare secondary information from the message like Syslog Match Name (which may be a S/N or IP address) and/or VSYS/VDOM name associated with the Regex that was matched.

Compares the Source IP address of the syslog message to the Management IP addresses of the devices in SIP. If there is a match then we know what device the syslog message belongs to. Once a specific device is matched:

• Identify device type, which then tells us what device pack to utilize.  
• Compare the syslog message to the regex's within that device pack's Collections Configurations page.
• If there is a match in the Usage Collection REGEX and if the syslog message has a unique rule identifier, we increment the hit count for that rule and increment any object hit counts that the syslog message includes.
• We use the source/destination IP, user, application, service, etc. and search through that device’s entire rule base until we find a matching rule.  If found, we increment the hit count for that rule, we'll also increment any object hit counts that the syslog message includes. (Internally we refer to this as "Legacy Logging", it is not the norm, it's extremely resource-intensive, an example would be Cisco's 302013 and 302015 traffic messages).
• If there is a match to the Change Collection REGEX, then the DC performs a retrieval on the device.

Considerations:

• Some devices do not generate syslog messages regularly, it’s not uncommon to not see any output from the tail command for several minutes.  To speed this up you can generate traffic through your device.
• If you are not able to get any output to generate this indicates logs are not making it to the Data Collector.  Please ensure the device is configured to log to the Data Collector correctly, ensure there are no routing issues and make certain any firewalls on route have rules allowing the syslog traffic through.
• If you are seeing logs but there is an issue (Critical Health status) in the FMOS GUI then there may be an issue with the syslogs being sent or a device pack may need to be updated.
• If the logs are not sourced directly from the device, but rather sent to a 3rd party collector or a Device Management Station (such as Panorama, Fortimanager, FortiAnalyzer) then a representation of the Central Syslog Server must be created in the Administration Settings menu using the IP address of the new log source.   Then within the Device Properties in the Administration Device menu, the Central Syslog Server must be assigned, and the Syslog Match Name field must contain an identifier for the device logs. This can be the device IP address, hostname, serial number, vdom name, etc... as long as it is unique to the device and appears in each of the log messages forwarded for the device.