MDaemon Technologies, Ltd.

Scott

6/16/23, 9:57 AM

Well, correction was short lived. It seems after a system reboot the system is now back to its same behavior as shown in a previous post in this thread:

How can traffic like this be configured to use the default bound IP address?

Wed 2023-05-31 10:35:12.548: [02495910] REMOTE message: pd3501000270355.msg
Wed 2023-05-31 10:35:12.548: [02495910] * Session 02495910; child 0001
Wed 2023-05-31 10:35:12.548: [02495910] * From: xxxx@xxxxx.com
Wed 2023-05-31 10:35:12.548: [02495910] * To: xxxxx@xxx.com
Wed 2023-05-31 10:35:12.548: [02495910] * Subject: General Information
Wed 2023-05-31 10:35:12.548: [02495910] * Message-ID: <eme7ddca38-7d8c-434c-be84-1ac325c62cc9@486de2c9.com>
Wed 2023-05-31 10:35:12.548: [02495910] * Size: 169535; <j:\mdaemon\queues\remote\pd3501000270355.msg>
Wed 2023-05-31 10:35:12.548: [02495910] * DNSSEC service requested
Wed 2023-05-31 10:35:12.555: [02495910] Resolving MX record for stoweattorneys.com (DNS Server: 1.0.0.1)...
Wed 2023-05-31 10:35:12.671: [02495910] * P=010 S=001 D=stoweattorneys.com TTL=(5) MX=[aspmx.l.google.com]
Wed 2023-05-31 10:35:12.671: [02495910] * P=020 S=002 D=stoweattorneys.com TTL=(5) MX=[alt2.aspmx.l.google.com]
Wed 2023-05-31 10:35:12.671: [02495910] * P=020 S=005 D=xxx.com TTL=(5) MX=[alt1.aspmx.l.google.com]
Wed 2023-05-31 10:35:12.671: [02495910] * P=030 S=000 D=xxx.com TTL=(5) MX=[aspmx2.googlemail.com]
Wed 2023-05-31 10:35:12.671: [02495910] * P=030 S=003 D=xxx.com TTL=(5) MX=[aspmx5.googlemail.com]
Wed 2023-05-31 10:35:12.671: [02495910] * P=030 S=004 D=xxxx.com TTL=(5) MX=[aspmx3.googlemail.com]
Wed 2023-05-31 10:35:12.672: [02495910] * P=030 S=006 D=xxxx.com TTL=(5) MX=[aspmx4.googlemail.com]
Wed 2023-05-31 10:35:12.672: [02495910] Attempting SMTP connection to aspmx.l.google.com
Wed 2023-05-31 10:35:12.675: [02495910] Resolving A record for aspmx.l.google.com (DNS Server: 1.0.0.1)...
Wed 2023-05-31 10:35:12.693: [02495910] * D=aspmx.l.google.com TTL=(3) A=[142.251.111.26]
Wed 2023-05-31 10:35:12.693: [02495910] Attempting SMTP connection to 142.251.111.26:25
Wed 2023-05-31 10:35:12.695: [02495910] Waiting for socket connection...
Wed 2023-05-31 10:35:12.696: [02495910] * Connection established 127.0.0.1:39904 --> 142.251.111.26:25

Previously you mentioned:

Get-NetIPAddress | ft IPAddress, InterfaceAlias, SkipAsSource

Below is the current output:

172.10.0.4 thernet 2 False
192.168.21.251 Ethernet False
192.168.21.250 Ethernet True
192.168.21.249 Ethernet True
192.168.21.239 Ethernet True
192.168.21.186 Ethernet True
192.168.21.185 Ethernet True
192.168.21.184 Ethernet True
192.168.21.169 Ethernet False

The result wasn't what was expected. No traffic coming in from 192.168.21.251 (clustered virual ip for mail) was passing to 192.168.21.169, so all client connection failed.

Reverting back the setting so 192.168.21.169 SkipAsSource was set to FALSE traffic starting flowing again.

Firewall trace still shows traffic coming in from192.168.21.169 and not the virual IP 192.168.21.251 so the following SMTP (out) conditions still remain.

"Connection established 127.0.0.1:39904 --> 142.251.111.26:25"

--AND--

SSL negotiation successful (TLS 1.2, 255 bit key exchange, 128 bit AES encryption)
SSL certificate is not valid (CRYPT_E_NO_REVOCATION_CHECK)

Considering we are bound to 192.168.21.251 within Mdaemon is a clustered environment causing problems for Mdaemon?

A

Arron Staff

6/16/23, 10:04 AM

We have been unable to find any issues with the way MDaemon is binding to IP addresses. If you use the utility to rebuild virtual and physical network cards does it fix the issue again?

S

Scott

6/16/23, 10:38 AM

The utilities were used previously and did appear to have 'effected' the problem but has not corrected the problem permanently.

Certainly, binding to a network card normally works (I’m sure), but it is looking more and more that overlaying the cluster ip is causing problems. By manipulating the ‘SkipAsSource’ for the local network card stops all communications with the virtual address underscore its dependence.

We know:

“netstat -anof | findstr 192.168.21.251”

* Netstat command (above) clearly shows mdaemon is bound to our virtual ip address

* All clients can connect remotely to the virtual ip

* When smtp (out) traffic on port 25 and certificate verification on port 80 appear to use 127.0.0.0 address which is displayed in the Mdaemon smtp (out) logfile. The same traffic appears to originate from 192.168.21.168 which is displayed on the firewall capture.

NOTE: if binding is occurring as you mentioned, then why is a loopback address displayed in Mdaemon's logfiles and not the bound ip address of 192.168.21.251?

A

Arron Staff

6/16/23, 11:50 AM

MDaemon is getting the IP that it displays in the logs from windows. It is using this windows function to retrieve the IP that is being used for the socket.

https://learn.microsoft.com/en-us/windows/win32/api/winsock/nf-winsock-getsockname

There is no way for a socket that is bound to localhost to connect to a public IP address. A socket bound to localhost can only connect to localhost. So it can't actually be bound to localhost or the connection would fail completely.

If you capture the traffic using WireShark, what does it show is happening?

If you use a 3rd party tool such as curl, that can bind to an outbound IP, does it work? What does wireshark show is happening?

S

Scott

6/16/23, 1:11 PM

Simply stated, when Mdaemon smtp (out) log shows:

Connection established 127.0.0.1:39904 --> 142.251.111.26:25

Our firewall display the bound virtual IP address given to Mdaemon in a clustered environment (192.168.21.251)

We have a packet-c.pcapng file available (if necessary).

"There is no way for a socket that is bound to localhost to connect to a public IP address. A socket bound to localhost can only connect to localhost. So it can't actually be bound to localhost or the connection would fail completely. "

I'm sure MS must have an enhanced library to capture the ip address or sockname associated with application in a clustered environment, because the packet capture tells a different story, the ip address is there.

A

Arron Staff

6/16/23, 2:31 PM

You had previously reported that the firewall sees "(127.0.0.1) as 192.168.21.178 "

The latest screen shot, which I think is from the firewall, shows the source IP as 192.168.21.251, so does that mean its working?

Are you still see issues with the (CRYPT_E_NO_REVOCATION_CHECK) error? If you turn of MTA STS and RequireTLS is MDaemon able to deliver the email? Security / Security Settings / SSL & TLS / SMTP Extensions /

Our developoment team is not aware of any other windows functions that can provide the local IP of the socket.

S

Scott

6/19/23, 7:45 AM

"The latest screen shot, which I think is from the firewall, shows the source IP as 192.168.21.251, so does that mean its working?

What is reported by the firewall is certainly a step in the right direction. This indicates traffic is flowing from Mdaemon to the firewall using our virtual ip address assigned to the application. However Mdaemon SMTP (out) still shows:

Connection established 127.0.0.1:61105 --> 35.85.199.61:25

Additionally, we are still seeing the following, even after we enable MTA STS and Require TLS:

SSL negotiation successful (TLS 1.2, 255 bit key exchange, 256 bit AES encryption)
SSL certificate is not valid (CRYPT_E_NO_REVOCATION_CHECK)

A

Arron Staff

6/19/23, 8:08 AM

If the firewall is showing the traffic coming from the correct IP, I wouldn't worry about the IP that is in MDaemon's log. We are using the only Windows function we are aware of to get the local IP of the socket when it connects.

even after we enable MTA STS and Require TLS

Sorry, I had a typo in my last message, can you disable MTA-STS and RequireTLS?

If you are still seeing the "SSL certificate is not valid (CRYPT_E_NO_REVOCATION_CHECK)" error after disabling them please send me a copy of the entire SMTP log. You can send it to arron.caruth @ mdaemon.com.

S

Scott

6/19/23, 8:37 AM

"can you disable MTA-STS and RequireTLS?"

Both had previously been disabled. I'll certainly pull together today's SMTP (out) log file and email it out ASAP.

A

Arron Staff

6/19/23, 11:49 AM

MDaemon is using Windows API functions for the certificate checks.

CertGetCertificateChain

CertVerifyCertificateChainPolicy

MDaemon always logs the errors that are returned, but in this case the error is ignored by MDaemon and it continues delivering the message via the TLS encrypted session.

Microsoft documents the CRYPT_E_NO_REVOCATION_CHECK error here: https://learn.microsoft.com/en-us/windows/win32/api/wincrypt/ns-wincrypt-cert_chain_policy_status

The only information they provide is "The revocation function was unable to check revocation for the certificate."

It seems to indicate that something in the environment is preventing the revocation checks for the certificate from being executed, but there is very little information available to assist with troubleshooting it.

S

Scott

6/19/23, 12:40 PM

So, it appears from your last message, there's nothing further we can check, or parameters to change to effect this problem -- which is unfortunate.

Thank you for your time on this issue.

A

Arron Staff

6/19/23, 1:08 PM

I have been able to find very little information on the CRYPT_E_NO_REVOCATION_CHECK error. Something in the enviroment is preventing the revocation check from working. Other than MDaemon logging the error that is returned, its not really an MDaemon issue.

I did find this post:

https://github.com/steffengy/schannel-rs/issues/13

which seems to indicate that if a certifcate does not have a revocation URL or if the revocation URL cannot be reached the error can be returned.

This page has some good information about certificate revocation urls, https://www.digicert.com/kb/util/utility-test-ocsp-and-crl-access-from-a-server.htm.

You may be able to use WireShark to capture the traffic generated by the revocation check and get a better idea of what is happening.

S

Scott

6/22/23, 8:03 AM

Update.

I've identified what was causing the problems with the "connection established" not returning the correct network card ip address and the systems inability to verify the "SSL Certificate" (noted in yellow below)

Thu 2023-06-22 08:38:41.698: [02643668] Randomly picked 205.139.110.242 from list of possible hosts
Thu 2023-06-22 08:38:41.698: [02643668] Attempting SMTP connection to 205.139.110.242:25
Thu 2023-06-22 08:38:41.700: [02643668] Waiting for socket connection...
Thu 2023-06-22 08:38:41.715: [02643668] * Connection established 192.168.21.251:63263 --> 205.139.110.242:25
Thu 2023-06-22 08:38:41.715: [02643668] Waiting for protocol to start...
Thu 2023-06-22 08:38:41.731: [02643668] <-- 220 us-smtp-1.mimecast.com ESMTP; Thu, 22 Jun 2023 08:38:41 -0400
Thu 2023-06-22 08:38:41.732: [02643668] --> EHLO mail.hsmc-ul.com
Thu 2023-06-22 08:38:41.747: [02643668] <-- 250-us-smtp-1.mimecast.com Hello [216.57.120.170]
Thu 2023-06-22 08:38:41.747: [02643668] <-- 250-STARTTLS
Thu 2023-06-22 08:38:41.747: [02643668] <-- 250 HELP
Thu 2023-06-22 08:38:41.747: [02643668] --> STARTTLS
Thu 2023-06-22 08:38:41.757: [02643668] <-- 220 Starting TLS [nyxstMFeMrm0MjA5i5Ptyg.us352]
Thu 2023-06-22 08:38:41.866: [02643668] SSL negotiation successful (TLS 1.2, 255 bit key exchange, 256 bit AES encryption)
Thu 2023-06-22 08:38:41.873: [02643668] SSL certificate is valid (matches us-smtp-inbound-1.mimecast.com and is signed by recognized CA)

The bottom line. We found that an updated version of Avast Business Antivirus inserted a driver like application that sits between the network card and the OS.

Although we’ve used this application for several years, we found a recent automatic update to a module in the application was found to have caused the issue. We demonstrated to an Avast representative over a remote session, how simply disabling a parameter within the module causes the problem to disappear, then reappear when the parameter was retuned to its normal setting.

Needless to say, a ticket has been generated with the vendor so they can identify the root cause and create a patch in the near future.

Thank you again for your time on this issue.

Your support and personal time spent on this issue was certainly appreciated.

A

Arron Staff

6/22/23, 8:31 PM

Wow!!! I'm so glad you were able to find the cause of the issue and thank you for sharing with us!

SMTP (out) communication