MDaemon Technologies, Ltd.

Wing

10/31/24, 1:16 AM

Hello!

I'm searching for the automated log parsing solution to achieve next goal - ability to view, filter and search for undelivered e-mails. something like dashboard.

I progressed along this way and wrote complex powershell script to parse SMTP-out log file , store and update the results at MS SQL table with several rows such as Date, SessionID, Sender, Recipient, Subject and, the most important for me - Delivery Result. And the next step was very clear for me - SQL View and simple ASPX/HTML with grid.

but I'm faced with Delivery Result problem: delivery result string is not always before the last "--->QUIT" string, it's not always begin with "<---- 2.5.0", it can be several lines and start with "5.5.0" or "4.5.2" or something else. I don't have such strong posh and regexp skills to rule them all.

then I look deeper to the manual (yep, RTFM first!) and found Mdstats.exe. I was happy like a child - no need to reinvite the wheel! Ok, I can see some information at the GUI and Delivery Result as Yes or No, but not the reason of delivery failure! got some command line options https://help.altn.com/mdaemon/en/queuestats_mdstats_command_line_parameter.html and did not achieve any success with CMD start - any combination just start mdstat.exe GUI application.

So at the end of the longread above I have several questions:

1. Does Mdstats still have cmd-line workflow or it's depricated?

2. Is there any other solution to achive my goal?

any help will be much appricated.

5
Posts

39
Views

A

Arron Staff

10/31/24, 10:32 AM

I have some ideas that I think will help.

MDaemon has two options for statistics, you can write the data to a CSV file or to a database. The options can be found Setup / Server Settings / Logging / Statistics Log. Whether you use the CSV file or the DB, you will likely still need to parse log files to obtain specific details about undeliverable message.

The statistics log can be found in the MDaemon\Logs directory and is named according to your logging settings. For example, MDaemon-2024-10-31-Statistics.log. It contains details on every message including the session ID and formatted as a CSV file with the following columns:

string StartTime ; YYMMDDHHMMSS (hour is in military format)
string EndTime ; YYMMDDHHMMSS (hour is in military format)
string Type ; SMTPI, SMTPO, POP, DPOP, MPOP, or RAS
int ID ; Session ID number
string Sender ; Email address of the message sender (if any)
string Recipient ; Email address of the message recipient (if any)
string List ; Email address of the mailing list (if list message)
string Gateway ; Gateway domain name (if gateway message)
string PeerIP ; IP address of other side of connection (if any)
string VirusName ; Name of first virus encountered in message (if any)
string RBL ; RBL host names of matches (space separated, if any)
float SpamScore ; Heuristic score message received (if any)
long Bytes ; Total bytes in/out including protocol (if any)
boolean SSL ; 1 if SSL was used, 0 otherwise

The Statistics DB can be found in the MDaemon\StatsDB\MDaemon-statistics.db3 file and includes the result and reason.

For the Statistics table, the Type column values are:

SMTP Inbound = 0
SMTP Outbound = 1
RAS = 2
IMAP = 3
POP = 4
MultiPOP = 5
DomainPOP = 6

The Result column values are:

Accepted = 0
Quarantined = 1
Rejected = 2

The Reason column values are:

None = 0
Spam = 1
Virus = 2

The statistics include the start time and end time of the session, along with the session ID. You should be able to use this data to find the actual reason the message failed, but you'd have to parse the log files. My suggestion would be to use the Session ID to get all lines from the log for that session, and then I'd parse the logs for SMTP response codes. As you already know parsing the logs is not going to be easy, but limiting it to a single session should help. For successful outbound messags look for this sequence:

Transfer Complete
<-- 250 2.0.0 OK

This basically says, MD has sent the entire message and the recipient has received and accepted the entire message.

If there is a delivery error, look for 4xx or 5xx errors. 4xx errors are temporary, 5xx errors are permanent. MDaemon should retry delivery when a 4xx error is retruned. MDaemon should not retry delivery when a 5xx error is returned.

Inbound SMTP sessions will probably be more dififcult as sending servers can do all kinds of things. A successful receipt of a message will look like this:

--> 250 2.6.0 Ok, message saved
<-- QUIT
--> 221 2.0.0 See ya in cyberspace

The 4xx and 5xx SMTP response codes apply here as well.

We are looking into expanding the stats collection in a future version of MDaemon to allow for better reporting for sending failures.

An easier solution would be to purchase and install our SecurityGateway product in front of MDaemon. It already does what I think you are trying to do. Here is an example of the Message Log from the SecurityGateway User Interface on my test server.

If you want more information on SecurityGateway, please take a look at https://mdaemon.com/pages/security-gateway and let us know if you have any questions.

W

Wing

11/1/24, 12:38 AM

Thank you Arron!

It looks like a complete challenge for me even with Claude =)

it's quite surprisingly for me that MD still has such tiny and weak stats, just as the 6.xx version (it was my first 😅 )

The 4xx and 5xx SMTP response codes apply here as well.

I read RFC but it's not so simple - there are many e-mail servers that responds with multiple 4xx or 5xx strings, gmail is for an example. they really responds with a poem about how they will reeject a mail message.

W

Wing

11/1/24, 12:42 AM

basicly I'm searching for "mailbox is full" or "no such user here" or something like this. but those answers aren't strictly defined with RFC and that's why any MTA sounds different.

A

Arron Staff

11/1/24, 7:11 AM

You are correct, it is extremely complicated.

Log Parsing and MDstats.exe